Skip to content
nonconvexcard

Implement gradient descent and verify all four rates

CodingAdvanced⏱ 3 hoursOptimizationimplementationratesverificationoptimisation

Problem

Implement gradient descent and empirically verify each of the four theorems:

  1. Convex smooth: O(1/t)O(1/t) on f(x)=∑ilog⁡(1+e−yiai⊤x)f(x) = \sum_i\log(1+e^{-y_ia_i^\top x}) (separable data removed so a minimiser exists — add a tiny ridge).
  2. Strongly convex smooth: linear rate on a quadratic, and check the measured contraction factor against 1−1/κ1-1/\kappa and against 1−2/(κ+1)1-2/(\kappa+1).
  3. Non-convex smooth: min⁡s≤t∥∇f∥\min_{s\le t}\|\nabla f\| decaying like t−1/2t^{-1/2} on a non-convex objective.
  4. Non-smooth convex: the last iterate oscillating and the average converging on f(x)=∥x∥1f(x)=\|x\|_1.

Implement it, then check the implementation against something you can verify independently.

Write it

Solve it here4 tests · Python runs in your browser
⌘/Ctrl + ↵ to run · limit 30s · draft saved in this browser
4 tests will run against your code. The first run downloads the Python runtime — several megabytes, once per browser.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems
SolvedProblemFieldTypeDifficultyTime
The exact step-size threshold on a quadraticOptimizationProofBeginner15 min
Implement backpropagation from scratch runDeep LearningCodingIntermediate3 hours
Implement kernel ridge regression runClassical MLCodingIntermediate1 hour
Implement multi-head attention from scratch runDeep LearningCodingIntermediate3 hours
Implement power iteration and verify the convergence rate runFoundationsCodingIntermediate1 hour