Fine-Tuning Mathematics

Adapting a pretrained model: the low-rank hypothesis behind LoRA, the memory arithmetic that decides what fits on your GPU, and what is actually known about forgetting.

3 topics11 problems (4% of the database)assumes LLMs assumes Optimization

Level 2 — Advanced

Low-Rank Adaptation11 min
LoRA's parameterisation, why B starts at zero and the update is scaled by α/r, and the low-rank hypothesis that the method rests on but does not prove.
Trainable parameter countZero initialisation preserves the pretrained functionThe α/r scaling makes updates rank-invariant in magnitude
The Memory Arithmetic of Training9 min
Sixteen bytes per trainable parameter, the activation bill, and why checkpointing every √L layers is exactly the right thing to do.
Sixteen bytes per trainable parameterActivation memory scales with batch, sequence and depthOptimal uniform checkpointing uses √L segments

Level 3 — Graduate

Forgetting and Regularisation11 min
Where the quadratic penalty in elastic weight consolidation comes from, why the Fisher information is the right curvature, and what the derivation assumes that deep networks violate.
Information matrix equalityThe elastic-weight-consolidation objectiveLow rank bounds how far the function can move

Problems (11)

Open in the browser

Problem	Type	Difficulty	Time
Adapter parameter counts	Numerical	Foundation	15 min
When activations dominate	Numerical	Beginner	30 min
Will this fine-tune fit?	Numerical	Beginner	30 min
Bounding how far the adapted model moves	Proof	Intermediate	30 min
Optimal checkpointing frequency	Proof	Intermediate	30 min
The α/r scaling	Proof	Intermediate	30 min
Why exactly one factor starts at zero	Proof	Intermediate	30 min
Derive the EWC objective	Proof	Advanced	1 hour
Implement LoRA and measure the spectrum run	Coding	Advanced	3 hours
Prove Eckart–Young–Mirsky abyss	Proof	Advanced	1 hour
The information matrix equality abyss	Proof	Graduate	1 hour