Implement backpropagation from scratch

CodingIntermediate⏱ 3 hoursDeep Learningbackpropagationimplementationgradient check

Problem

Implement a small MLP with forward and backward passes by hand (no autodiff), and verify it.

Check the analytic gradient against central finite differences.
Measure the wall-clock ratio of backward to forward passes.
Demonstrate vanishing gradients with a deep sigmoid network, and fix it with He initialisation plus ReLU.
Add a residual connection and measure the gradient norm at the first layer as depth grows.

Implement it, then check the implementation against something you can verify independently.

Solve it here5 tests · Python runs in your browser

⌘/Ctrl + ↵ to run · limit 30s · draft saved in this browser

5 tests will run against your code. The first run downloads the Python runtime — several megabytes, once per browser.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
XOR is not linearly separable	Deep Learning	Proof	Foundation	15 min
Derive the backpropagation recursion	Deep Learning	Proof	Intermediate	30 min
Gradient magnitude through a deep chain	Deep Learning	Proof	Intermediate	30 min
Implement kernel ridge regression run	Classical ML	Coding	Intermediate	1 hour
Implement multi-head attention from scratch run	Deep Learning	Coding	Intermediate	3 hours