Optimal checkpointing frequency

ProofIntermediate⏱ 30 minFine-Tuninggradient checkpointingoptimisation

Problem

Prove that storing every $k$ -th layer gives peak activation storage $L/k + k$ and that $k^\star=\sqrt L$ minimises it.
Compute the saving for $L = 32$ and $L = 80$ .
Suppose recomputation costs a fraction $\theta$ of a full training step. Write total cost as a function of $k$ under a memory constraint and say when checkpointing is not worth it.

Write a complete argument. Every inequality you use should be one you could state precisely.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
When activations dominate	Fine-Tuning	Numerical	Beginner	30 min
Will this fine-tune fit?	Fine-Tuning	Numerical	Beginner	30 min
Choosing the draft length	LLM Ops	Numerical	Advanced	30 min
Implement gradient descent and verify all four rates run	Optimization	Coding	Advanced	3 hours
Optimal clipping for a Laplace-distributed tensor	Quantization	Proof	Advanced	1 hour