How many users fit on one GPU
NumericalIntermediate⏱ 30 minLLM Opsbatchingmemorycapacity
Problem
An 80 GB GPU serves the 7B MHA model from the previous problem in BF16 (13.4 GB of weights), with 5 GB of framework and activation overhead.
- Compute the maximum concurrent sequences at context lengths 2k, 8k and 32k.
- At each, state whether decoding is compute-bound, taking the ridge-point batch as 295.
- Compute the context length at which capacity falls to a single sequence.
Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.
Solution
Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.
Related problems
All problems| Solved | Problem | Field | Type | Difficulty | Time |
|---|---|---|---|---|---|
| KV cache sizing with MHA and GQA | LLM Ops | Numerical | 30 min | ||
| Product quantisation budget arithmetic | RAG | Numerical | 30 min | ||
| Ridge point and the batch size needed to saturate | LLM Ops | Numerical | 15 min | ||
| When activations dominate | Fine-Tuning | Numerical | 30 min | ||
| When does attention dominate the MLP? | Deep Learning | Numerical | 30 min |