Derive Adam's bias correction

ProofBeginner⏱ 15 minSGDadambias correctionmoving average

Theory: Adaptive Methods and Variance Reduction →

Problem

Show that the exponential moving average $m_t = \beta m_{t-1}+(1-\beta)g_t$ started from $m_0 = 0$ satisfies $\E m_t = (1-\beta^t)\E g$ under a stationary gradient distribution, and deduce the correction factor. Quantify the size of the bias at $t=1$ and $t=100$ for $\beta = 0.999$ .

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Will this fine-tune fit?	Fine-Tuning	Numerical	Beginner	30 min
Why AdamW differs from Adam with L2	SGD	Proof	Intermediate	30 min
The AdaGrad summation inequality	SGD	Proof	Advanced	30 min
The SVRG control variate	SGD	Proof	Advanced	1 hour
Open: why does Adam beat SGD on transformers? abyss	SGD	Proof	Research	Research