Mixture-of-experts sparsity accounting

NumericalIntermediate⏱ 30 minLLMsmixture of expertsflopsmemoryload balancing

Problem

A model routes each token to $k=2$ of $E=64$ experts, each expert having $P = 2\times10^8$ parameters, with $S = 10^9$ shared parameters.

Compute the total parameter count and the per-token active parameter count.
Compute the FLOPs per token and compare with a dense model of the same total size.
Compute the memory required and comment on the binding constraint.

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
Count the FLOPs of a forward pass	LLM Ops	Numerical	Foundation	15 min
Derive the attention crossover length	LLM Ops	Proof	Beginner	15 min
KV cache sizing with MHA and GQA	LLM Ops	Numerical	Beginner	30 min
LoRA parameter accounting	LLMs	Numerical	Beginner	15 min
Product quantisation budget arithmetic	RAG	Numerical	Beginner	30 min