Open: why does Adam beat SGD on transformers?
Problem
On convex problems, and in the worst-case non-convex analyses, adaptive methods have no asymptotic advantage over well-tuned SGD; several analyses give them strictly worse constants. On transformer language models the gap is large, reproducible, and survives careful tuning of SGD.
Explain it. Give a property of the objective — provable or at least measurable — that implies a separation between Adam and SGD, and verify it holds for transformer training.
This is an open problem. Nobody has a solution, and none is hidden below — what is recorded is the state of the argument. Progress on it would be publishable.
What is known
This problem is open. What follows is an account of what has been proved, what has only been measured, and where the argument stops — not a solution, because there is not one.
Related problems
All problems| Solved | Problem | Field | Type | Difficulty | Time |
|---|---|---|---|---|---|
| Derive Adam's bias correction | SGD | Proof | 15 min | ||
| Will this fine-tune fit? | Fine-Tuning | Numerical | 30 min | ||
| Why AdamW differs from Adam with L2 | SGD | Proof | 30 min | ||
| The AdaGrad summation inequality | SGD | Proof | 30 min | ||
| The SVRG control variate | SGD | Proof | 1 hour |