LoRA parameter accounting

NumericalBeginner⏱ 15 minLLMsloraparameter countfine-tuning

Problem

For a transformer with $d = 4096$ , $L = 32$ layers, adapting $W_Q$ and $W_V$ in each layer with rank $r$ :

Count the trainable parameters as a function of $r$ .
Compare with full fine-tuning of those matrices, and with the full model at $7$ B parameters.
Compute the checkpoint size for $100$ tasks at $r = 8$ in fp16.

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
Adapter parameter counts	Fine-Tuning	Numerical	Foundation	15 min
The 6ND FLOP estimate	LLMs	Numerical	Beginner	15 min
Will this fine-tune fit?	Fine-Tuning	Numerical	Beginner	30 min
Derive the compute-optimal allocation	LLMs	Proof	Intermediate	30 min
Mixture-of-experts sparsity accounting	LLMs	Numerical	Intermediate	30 min