When does attention dominate the MLP?
NumericalBeginner⏱ 30 minDeep Learningcomplexityflopslong contextmemory
Problem
For a transformer layer with width , sequence length , heads and MLP expansion :
- Write the FLOP counts for the attention and MLP sublayers.
- Find the at which they are equal.
- Evaluate for at and .
- Compute the attention matrix memory in both cases and comment.
Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.
Solution
Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.
Related problems
All problems| Solved | Problem | Field | Type | Difficulty | Time |
|---|---|---|---|---|---|
| Count the FLOPs of a forward pass | LLM Ops | Numerical | 15 min | ||
| Derive the attention crossover length | LLM Ops | Proof | 15 min | ||
| KV cache sizing with MHA and GQA | LLM Ops | Numerical | 30 min | ||
| Product quantisation budget arithmetic | RAG | Numerical | 30 min | ||
| The 6ND FLOP estimate | LLMs | Numerical | 15 min |