The LSTM cell-state Jacobian

ProofAdvanced⏱ 30 minDeep Learninglstmgatingjacobianconstant error carousel

Theory: Convolution, Recurrence and Normalisation →

Problem

Compute $\partial c_t/\partial c_{t-1}$ for the LSTM along the cell-state path, and explain why replacing a product of matrices with a product of diagonal entries in $(0,1)$ makes long-range gradients survivable. Relate to the GRU and to residual connections.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Receptive field growth with depth and dilation	Deep Learning	Numerical	Beginner	15 min
Convolution diagonalises in the Fourier basis	Deep Learning	Proof	Intermediate	30 min
Gradient magnitude through a deep chain	Deep Learning	Proof	Intermediate	30 min
The RNN gradient is a repeated matrix product	Deep Learning	Proof	Intermediate	30 min
The softmax Jacobian as a variance, and saturation	Deep Learning	Proof	Intermediate	30 min