When activations dominate

NumericalBeginner⏱ 30 minFine-Tuningactivationssequence lengthmemory

Theory: The Memory Arithmetic of Training →

Problem

For a model with $L=32$ , $d=4096$ , using 20 bytes of activation per layer per token per unit width divided by $d$ — that is, $20Ld$ bytes per token:

Compute activation memory for batch 8 at sequence lengths 512, 4096 and 32768.
Compare each with the 13.4 GB of BF16 weights.
At which sequence length do activations exceed LoRA's entire persistent state?

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
KV cache sizing with MHA and GQA	LLM Ops	Numerical	Beginner	30 min
Product quantisation budget arithmetic	RAG	Numerical	Beginner	30 min
When does attention dominate the MLP?	Deep Learning	Numerical	Beginner	30 min
Will this fine-tune fit?	Fine-Tuning	Numerical	Beginner	30 min
How many users fit on one GPU	LLM Ops	Numerical	Intermediate	30 min