Skip to content
nonconvexcard

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Arthur Jacot, Franck Gabriel, Clément Hongler · 2018 · NeurIPS

DL Theoryntkinfinite widthkernelgradient flowlazy training
Read the paper

The problem

Gradient descent on a non-convex neural network loss has no general convergence theory, and there was no tractable limit in which the dynamics could be solved.

The contribution

In the infinite-width limit with a particular scaling, the tangent kernel becomes deterministic and constant in time, so gradient flow on the network is exactly kernel gradient flow — a linear, solvable system.

What problem does this paper solve?

It supplies a regime in which training dynamics can be computed exactly, turning a question about non-convex optimisation into a question about a linear system.

What mathematical objects are introduced?

The neural tangent kernel

Θ(L)(x,x′)=⟨∇θf(x;θ),∇θf(x′;θ)⟩,\Theta^{(L)}(x,x') = \ip{\nabla_\theta f(x;\theta)}{\nabla_\theta f(x';\theta)} ,

and its infinite-width limit, which the paper shows is deterministic (independent of the random initialisation) and computable by an explicit layerwise recursion.

What assumptions are made?

The NTK parameterisation (each layer's output scaled by 1/width1/\sqrt{\text{width}}), widths tending to infinity in all layers, and gradient flow rather than discrete steps. The scaling is essential and is not the one used in practice.

What equations matter?

Under gradient flow the network outputs on the training set evolve as

u˙t=−Θ (ut−y),\dot u_t = -\Theta\,(u_t - y),

a linear ODE. Since Θ⪰0\Theta\succeq0 this converges, and the rate is governed by λmin⁡(Θ)\lambda_{\min}(\Theta) — which is exactly a PL inequality, as the theory page shows.

What theorem or result is important?

As width →∞\to\infty: (i) Θ\Theta converges in probability to a deterministic limit; (ii) Θ\Theta stays constant during training; (iii) consequently the trained network equals kernel regression with kernel Θ\Theta. Du et al. and others subsequently turned this into finite-width convergence theorems with explicit width requirements.

What proof techniques are used?

Law-of-large-numbers arguments over the width for each layer, with a careful induction on depth to control the recursion; concentration of the Gram matrices of the intermediate representations.

What prior mathematics is required?

Kernels and RKHS; concentration for sums of independent random vectors; gradient flow as an ODE.

Limitations

The regime is lazy: parameters move by O(1/width)O(1/\sqrt{\text{width}}), so no features are learned and the "deep" structure is inert. The width requirements in the finite-width follow-ups are far beyond practical models. Empirically the NTK kernel machine underperforms the finite trained network, and the kernel is observed to drift substantially in real training. So the theory describes a regime that exists mathematically and is not the regime practitioners are in.

Why it mattered

It is the first setting in which deep-network training dynamics were solved exactly, and it made precise what overparameterisation buys and what it does not. It also framed the central question sharply: the interesting behaviour of real networks is outside the lazy regime, and the mean-field and μ\muP lines of work exist to reach it.

What came next