Sharpness is not a property of the function

ProofAdvanced⏱ 30 minDL Theoryflat minimasharpnessreparameterisationcounterexample

Theory: Why Does Deep Learning Optimization Work? →

Problem

For a two-layer ReLU network $f(x) = W_2\rho(W_1x)$ , show that $(W_1,W_2)\mapsto(\alpha W_1,\alpha^{-1}W_2)$ leaves $f$ unchanged, and determine how the Hessian eigenvalues scale. Conclude that naive sharpness cannot causally explain generalisation.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Pairwise independence does not imply independence	Probability	Proof	Foundation	15 min
A variable with all moments finite that is not sub-Gaussian	High-Dim Probability	Proof	Intermediate	30 min
All directional derivatives exist, yet the function is not differentiable	Foundations	Proof	Intermediate	30 min
Implicit bias of gradient descent on underdetermined least squares	DL Theory	Proof	Intermediate	30 min
Separating the four modes of convergence	Probability	Proof	Intermediate	30 min