Why AdamW differs from Adam with L2

ProofIntermediate⏱ 30 minSGDadamwweight decaypreconditionerregularisation

Theory: Adaptive Methods and Variance Reduction →

Problem

Write the update of (a) Adam applied to $f(\theta)+\frac\lambda2\|\theta\|_2^2$ and (b) Adam with decoupled weight decay. Show the two coincide iff the preconditioner is a multiple of the identity, and explain why the $L^2$ version applies weaker shrinkage to high-variance coordinates.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Derive Adam's bias correction	SGD	Proof	Beginner	15 min
Bounding how far the adapted model moves	Fine-Tuning	Proof	Intermediate	30 min
The soft-margin SVM is regularised hinge-loss minimisation	Classical ML	Proof	Intermediate	30 min
Normalisation makes weights scale-invariant	Deep Learning	Proof	Advanced	30 min
The AdaGrad summation inequality	SGD	Proof	Advanced	30 min