Why Does Deep Learning Optimization Work?

They deliver an approximate stationary point. They do not say that point is a local minimum, that it has low loss, or that it generalises. A neural network loss has exponentially many stationary points, most of them useless. So the assumption-free theory is compatible with training producing nothing of value — and the fact that it does not is precisely what needs explaining.

Note also that neural network losses are typically not $L$-smooth globally (ReLU has unbounded curvature at the kink locus in a suitable sense; the loss surface has regions of very large curvature), so even these theorems apply only under a hypothesis that is not verified in practice.

Four separate problems with reading the NTK theorem as an explanation of deep learning:

1. The width requirement is astronomical. $m = \Omega(n^6/\lambda_0^4)$ with $n = 10^6$ is beyond any conceivable model. Sharper analyses have reduced the exponent, but not to the regime of real networks.
2. In the NTK regime the network does not learn features. By construction the parameters barely move, so the tangent features are those at initialisation: the model is a linear model in random features, and the "deep" part is inert. Chizat, Oyallon and Bach call this lazy training and show it is a consequence of the scaling, not of depth.
3. NTK-regime networks generalise worse than real ones. Empirically, the NTK-limit kernel machine underperforms the trained finite network on standard benchmarks, often substantially. So whatever the finite network is doing, it is not what the NTK describes.
4. The kernel demonstrably drifts. Measurements of $\Theta_t$ during real training show large changes, especially early. The hypothesis of the theorem is observably false in practice.

The mean-field / $\mu$P line of work studies alternative scalings under which features do move in the infinite-width limit, and is the more promising direction. It has produced genuine results for two-layer networks and useful hyperparameter-transfer prescriptions; a comparable theory for deep networks is open.

This is arguably the central technical gap. If the implicit bias were characterised, the generalisation question would reduce to asking whether the induced complexity measure is small for the data — a question of the kind learning theory knows how to answer. Without it, we do not even know what function class the algorithm is effectively searching.

Note also that the discrete/continuous distinction is not cosmetic: gradient descent with a finite step has a different implicit bias from gradient flow (there are extra terms of order $\eta$ that penalise gradient norm), and the SGD noise adds further terms. Results proved for flow do not transfer automatically.

It gives an approximate second-order stationary point, which rules out strict saddles but not: degenerate saddles with $\lambda_{\min}=0$, poor local minima, or plateaus. And deep network losses are known to have degenerate critical points in abundance — every permutation of hidden units gives a symmetry, and every dead ReLU gives a flat direction. The theorem is progress on one obstruction, not on all of them.

The correlation is real and reproducible. The causal claim is not established, and there is a sharp objection. Dinh et al. showed that for a ReLU network the map $(W_1,W_2)\mapsto(\alpha W_1,\alpha^{-1}W_2)$ leaves the function — hence the test error — exactly unchanged while changing the Hessian spectrum by factors of $\alpha^{\pm2}$. So "sharpness" is not a property of the learned function; it is a property of the parameterisation, and can be made arbitrarily large or small without touching generalisation.

Responses have been to define reparameterisation-invariant sharpness measures (adaptive sharpness, information-theoretic or PAC-Bayes flatness), and some of these correlate with generalisation more robustly. But no theorem currently shows that flatness in any such sense implies generalisation for realistic networks. Treat "flat minima generalise" as a well-supported empirical heuristic with a known counterexample to its naive form.

The observation is striking because $2/\eta$ is exactly the classical stability threshold of the quadratic analysis: gradient descent on a quadratic with curvature above $2/\eta$ diverges. So training operates at the boundary of where the standard theory applies, and beyond the regime where descent-lemma arguments are valid.

This has a direct consequence: convergence proofs assuming $\eta \le 1/L$ with $L$ the true smoothness constant do not describe what happens, because the effective $L$ adapts to $\eta$ rather than the reverse. Theory for the edge-of-stability regime exists for specific simplified models and is an active area; there is no general account.

Several mechanisms have been proposed: a slow transition from a memorising to a generalising circuit driven by weight decay; a competition between a fast-to-find memorisation solution and a slower-to-find but lower-norm generalising one; a delayed alignment of representations. Each has supporting evidence on specific tasks, and mechanistic-interpretability work has reverse-engineered the learned circuit in the modular-addition case in convincing detail.

What is not established is a general theory: which tasks grok, how long the delay is, or whether the phenomenon occurs in large-scale training at all (the evidence is largely from small models on algorithmic data). It is an unusually clean experimental handle on the memorisation-to-generalisation transition, and that is its main value.

This is among the most useful empirical regularities in the field — it is what makes large training runs planable — and it is not derived. There are suggestive arguments connecting power-law loss curves to power-law data spectra, to intrinsic dimension of the data manifold, and to nonparametric-rate calculations of the kind on the nearest neighbours page. None of them predicts the observed exponents from properties of the data, and none explains the functional form's remarkable stability across modalities and architectures.

Two further cautions. The fitted form is an extrapolation and has been revised when refit with better methodology — the Kaplan-to-Chinchilla correction moved the recommended data-to-parameter ratio by more than an order of magnitude, from a fit to the same functional family. And the laws describe loss, not capability; the relationship between loss and downstream behaviour is itself not a smooth function and is much less well characterised.