KV cache sizing with MHA and GQA

NumericalBeginner⏱ 30 minLLM Opskv cachegqamemory

Theory: KV Cache, Batching and Little's Law →

Problem

Model A is 7B with $L = 32$ , $d = 4096$ , 32 attention heads, multi-head attention. Model B is 70B with $L = 80$ , $d = 8192$ , 64 query heads and 8 key/value heads (GQA). Both store the cache in BF16.

Compute bytes of cache per token for each.
Compute the cache for one 8192-token conversation in each.
Compute what model B's cache would have been with full MHA, and state the GQA saving.

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Implement the cluster sizing formula run	AI Systems	Coding	Beginner	30 min
Product quantisation budget arithmetic	RAG	Numerical	Beginner	30 min
When activations dominate	Fine-Tuning	Numerical	Beginner	30 min
When does attention dominate the MLP?	Deep Learning	Numerical	Beginner	30 min
Will this fine-tune fit?	Fine-Tuning	Numerical	Beginner	30 min