Skip to content
nonconvexcard

The Ω(1/t²) lower bound for first-order methods

ProofOlympiad / Extreme⏱ 1 dayAdvanced Optimizationlower boundaccelerationoptimalityworst case

Problem

Prove that for every first-order method whose iterates satisfy

xt∈x0+span⁡{∇f(x0),…,∇f(xt−1)},x_t \in x_0 + \operatorname{span}\{\nabla f(x_0),\dots,\nabla f(x_{t-1})\},

and for every t≤(d−1)/2t \le (d-1)/2, there exists a convex LL-smooth function ff on Rd\R^d with

f(xt)−f⋆ ≥ 3L∥x0−x⋆∥2232(t+1)2.f(x_t) - f^\star \ \ge\ \frac{3L\|x_0-x^\star\|_2^2}{32(t+1)^2}.

Conclude that Nesterov's method is optimal in this class up to a constant.

Write a complete argument. Every inequality you use should be one you could state precisely.

Hints

Hints — 0 of 3 revealed

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems
SolvedProblemFieldTypeDifficultyTime
Simulate and compare convergence rates runAdvanced OptimizationCodingIntermediate1 hour
Single-stream decode latency floorsLLM OpsNumericalIntermediate30 min
The irreducible loss is the conditional entropyLLMsProofIntermediate30 min
The volumetric lower bound on covering numbersHigh-Dim ProbabilityProofIntermediate30 min
The no-free-lunch theoremLearning TheoryProofAdvanced1 hour