Will this fine-tune fit?

NumericalBeginner⏱ 30 minFine-Tuningmemoryadamloraqlora

Problem

You have one 80 GB GPU and a 13B-parameter model.

Compute persistent state for full fine-tuning with mixed-precision Adam. Does it fit?
Compute it for LoRA with 20M trainable parameters and a BF16 frozen base.
Compute it for QLoRA with a 4-bit frozen base.
In case 2, how much is left for activations, and what does that allow at $L=40$ , $d=5120$ , assuming 20 bytes of activation per layer per token per unit of width?

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
Adapter parameter counts	Fine-Tuning	Numerical	Foundation	15 min
Derive Adam's bias correction	SGD	Proof	Beginner	15 min
KV cache sizing with MHA and GQA	LLM Ops	Numerical	Beginner	30 min
LoRA parameter accounting	LLMs	Numerical	Beginner	15 min
Product quantisation budget arithmetic	RAG	Numerical	Beginner	30 min