Implement gradient descent and verify all four rates
CodingAdvanced⏱ 3 hoursOptimizationimplementationratesverificationoptimisation
Problem
Implement gradient descent and empirically verify each of the four theorems:
- Convex smooth: on (separable data removed so a minimiser exists — add a tiny ridge).
- Strongly convex smooth: linear rate on a quadratic, and check the measured contraction factor against and against .
- Non-convex smooth: decaying like on a non-convex objective.
- Non-smooth convex: the last iterate oscillating and the average converging on .
Implement it, then check the implementation against something you can verify independently.
Write it
Solve it here4 tests · Python runs in your browser
⌘/Ctrl + ↵ to run · limit 30s · draft saved in this browser
4 tests will run against your code. The first run downloads the Python runtime — several megabytes, once per browser.
Solution
Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.
Related problems
All problems| Solved | Problem | Field | Type | Difficulty | Time |
|---|---|---|---|---|---|
| The exact step-size threshold on a quadratic | Optimization | Proof | 15 min | ||
| Implement backpropagation from scratch run | Deep Learning | Coding | 3 hours | ||
| Implement kernel ridge regression run | Classical ML | Coding | 1 hour | ||
| Implement multi-head attention from scratch run | Deep Learning | Coding | 3 hours | ||
| Implement power iteration and verify the convergence rate run | Foundations | Coding | 1 hour |