Derive the backpropagation recursion

ProofIntermediate⏱ 30 minDeep Learningbackpropagationchain ruleadjoint

Problem

For a feedforward network $z^{(\ell)} = W^{(\ell)}a^{(\ell-1)}+b^{(\ell)}$ , $a^{(\ell)} = \rho(z^{(\ell)})$ , derive

\delta^{(\ell)} = \big((W^{(\ell+1)})^\top\delta^{(\ell+1)}\big)\odot\rho'(z^{(\ell)}), \qquad \frac{\partial\mathcal J}{\partial W^{(\ell)}} = \delta^{(\ell)}\big(a^{(\ell-1)}\big)^\top ,

and state the time and memory cost.

Write a complete argument. Every inequality you use should be one you could state precisely.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
XOR is not linearly separable	Deep Learning	Proof	Foundation	15 min
Gradient magnitude through a deep chain	Deep Learning	Proof	Intermediate	30 min
Implement backpropagation from scratch run	Deep Learning	Coding	Intermediate	3 hours
Width needed to represent a piecewise linear function	Deep Learning	Proof	Advanced	1 hour