NTK convergence is a PL inequality

ProofGraduate⏱ 1 hourDL Theoryntkpl inequalitygradient flowmechanism

Theory: Why Does Deep Learning Optimization Work? →

Problem

Derive the gradient-flow dynamics $\dot u_t = -\Theta_t(u_t-y)$ for the network outputs on the training set, show that $\lambda_{\min}(\Theta_t)\ge\lambda_0/2$ implies exponential convergence of the squared loss, and identify the resulting statement as a Polyak–Łojasiewicz inequality. State precisely what remains to be proved to turn this into a theorem.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Implicit bias of gradient descent on underdetermined least squares	DL Theory	Proof	Intermediate	30 min
Strong convexity implies the Polyak–Lojasiewicz inequality	Optimization	Proof	Intermediate	30 min
What fitting random labels rules out	DL Theory	Proof	Intermediate	30 min
Fit a scaling law and derive the compute-optimal allocation run	DL Theory	Coding	Advanced	3 hours
Sharpness is not a property of the function	DL Theory	Proof	Advanced	30 min