Gradient magnitude through a deep chain

ProofIntermediate⏱ 30 minDeep Learningvanishing gradientjacobianinitialisationresidual

Theory: Approximation Theory and Backpropagation →

Problem

Show that the input-layer adjoint is a product of layer Jacobians, and that under random initialisation with i.i.d. weights of variance $\sigma_w^2$ , the expected squared gradient norm scales like $(\sigma_w^2 n_{\text{in}}\E[\rho'^2])^{L}$ . Deduce the He initialisation and explain the effect of a residual connection.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
XOR is not linearly separable	Deep Learning	Proof	Foundation	15 min
Derive the backpropagation recursion	Deep Learning	Proof	Intermediate	30 min
Implement backpropagation from scratch run	Deep Learning	Coding	Intermediate	3 hours
The RNN gradient is a repeated matrix product	Deep Learning	Proof	Intermediate	30 min
The softmax Jacobian as a variance, and saturation	Deep Learning	Proof	Intermediate	30 min