Adapter parameter counts

NumericalFoundation⏱ 15 minFine-Tuningloracounting

Problem

A model has 32 layers, each with four attention projections of size $4096\times4096$ .

Count full fine-tuning parameters for those projections.
Count LoRA parameters at $r = 8$ and at $r = 64$ , applied to all four projections in every layer.
Give the reduction factors, and the rank at which LoRA stops saving anything.

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
Count the FLOPs of a forward pass	LLM Ops	Numerical	Foundation	15 min
LoRA parameter accounting	LLMs	Numerical	Beginner	15 min
Will this fine-tune fit?	Fine-Tuning	Numerical	Beginner	30 min
The α/r scaling	Fine-Tuning	Proof	Intermediate	30 min
Why exactly one factor starts at zero	Fine-Tuning	Proof	Intermediate	30 min