Mini-batching and the compute budget

NumericalIntermediate⏱ 30 minSGDminibatchvariancecomputescaling

Problem

Show that averaging $B$ conditionally i.i.d. unbiased gradients divides the variance by $B$ .
Substituting into the non-convex rate, show that the bound depends on $B$ and $T$ only through $BT$ when the step is retuned.
What does this predict about the value of increasing batch size at fixed total compute? What does it fail to predict?

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
Derive the attention crossover length	LLM Ops	Proof	Beginner	15 min
The Δ²/12 noise variance	Quantization	Proof	Beginner	15 min
Migrating quantization difficulty	Quantization	Numerical	Intermediate	30 min
Reading the Robbins–Monro conditions off the proof	SGD	Proof	Intermediate	30 min
SGD on convex Lipschitz objectives	SGD	Proof	Intermediate	30 min