Implement SGD and locate the noise floor

CodingAdvanced⏱ 3 hoursSGDsgdimplementationnoise floorschedules

Problem

Implement SGD on a strongly convex quadratic with an explicit Gaussian gradient oracle.

Verify the two-phase behaviour: geometric decay followed by a plateau.
Verify that the plateau height is proportional to $\eta\sigma^2/\mu$ by sweeping $\eta$ and $\sigma$ .
Compare constant, $1/\sqrt t$ and $1/t$ schedules; fit the asymptotic slopes.
Add mini-batching and confirm the floor scales as $1/B$ .

Implement it, then check the implementation against something you can verify independently.

Solve it here4 tests · Python runs in your browser

⌘/Ctrl + ↵ to run · limit 40s · draft saved in this browser

4 tests will run against your code. The first run downloads the Python runtime — several megabytes, once per browser.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
Implement backpropagation from scratch run	Deep Learning	Coding	Intermediate	3 hours
Implement kernel ridge regression run	Classical ML	Coding	Intermediate	1 hour
Implement multi-head attention from scratch run	Deep Learning	Coding	Intermediate	3 hours
Implement power iteration and verify the convergence rate run	Foundations	Coding	Intermediate	1 hour
Implement random projection and measure the distortion run	High-Dim Probability	Coding	Intermediate	1 hour