Deep Learning Mathematics
Approximation theory, backpropagation as reverse-mode differentiation, convolutions, recurrence and transformers.
Level 2 — Advanced
- Approximation Theory and Backpropagation10 min
Universal approximation with its exact hypotheses, why it says nothing about learnability, and backpropagation as reverse-mode differentiation on a computational graph.
A single linear threshold unit cannot represent XORUniversal approximation theoremWhat universal approximation does not give - Convolution, Recurrence and Normalisation13 min
Equivariance as the defining property of convolution, the Fourier view, the vanishing-gradient computation for RNNs, and the LSTM gating equations derived from it.
Translation-equivariant linear maps are exactly convolutionsConvolution theoremReceptive field growth
Level 3 — Graduate
Problems (18)
Open in the browser| Solved | Problem | Type | Difficulty | Time |
|---|---|---|---|---|
| XOR is not linearly separable | Proof | 15 min | ||
| Receptive field growth with depth and dilation | Numerical | 15 min | ||
| When does attention dominate the MLP? | Numerical | 30 min | ||
| Attention is permutation equivariant | Proof | 30 min | ||
| Convolution diagonalises in the Fourier basis | Proof | 30 min | ||
| Derive the backpropagation recursion | Proof | 30 min | ||
| Gradient magnitude through a deep chain | Proof | 30 min | ||
| Implement backpropagation from scratch run | Coding | 3 hours | ||
| Implement multi-head attention from scratch run | Coding | 3 hours | ||
| The RNN gradient is a repeated matrix product | Proof | 30 min | ||
| The softmax Jacobian as a variance, and saturation | Proof | 30 min | ||
| Translation-equivariant linear maps are convolutions | Proof | 30 min |