Low-Rank Adaptation

LoRA's parameterisation, why B starts at zero and the update is scaled by α/r, and the low-rank hypothesis that the method rests on but does not prove.

Fine-TuningAdvanced⏱ 1 hour11 min readloralow rankpefteckart-youngadapters

5 problems →

AssumesLinear Algebra for Machine Learning · Approximation Theory and Backpropagation · RLHF, Bradley–Terry and DPO

The parameterisation

DefinitionLow-rank adapter

For a frozen pretrained matrix $W_0 \in \R^{d\times k}$ , a rank- $r$ adapter replaces it with

W = W_0 + \frac{\alpha}{r} BA, \qquad B \in \R^{d\times r},\quad A \in \R^{r\times k},\quad r \ll \min(d,k),

where $A$ and $B$ are trained and $W_0$ is not. The scalar $\alpha$ is a fixed hyperparameter.

PropositionTrainable parameter count

The adapter has $r(d+k)$ trainable parameters against $dk$ for the full matrix, a reduction factor of

\frac{dk}{r(d+k)}.

For a square matrix $d = k$ this is $d/(2r)$ .

Why $B$ starts at zero

PropositionZero initialisation preserves the pretrained function

Initialise $B = 0$ and $A$ with i.i.d. entries of variance $\sigma^2 > 0$ . Then at initialisation $W = W_0$ , so the adapted network computes exactly the pretrained function; yet the gradient with respect to $B$ is generally non-zero, so training is not stuck.

Key idea

The asymmetry is the point. Initialising both factors at zero would leave both gradients zero, a genuine stationary point. Initialising both randomly would perturb the pretrained function before any learning, discarding some of what you are paying to keep. Exactly one factor at zero gives you the pretrained function at step zero and a live gradient.

The $\alpha/r$ scaling

PropositionThe α/r scaling makes updates rank-invariant in magnitude

Let $B$ and $A$ have i.i.d. mean-zero entries with variances $\sigma_B^2$ and $\sigma_A^2$ respectively, independent of each other. Then

\E\left[\left\|\frac{\alpha}{r}BA\right\|_F^2\right] = \frac{\alpha^2}{r}\,dk\,\sigma_B^2\sigma_A^2 ,

so the expected squared magnitude falls as $1/r$ rather than growing linearly with $r$ as it would without the scaling.

What a rank constraint can and cannot represent

TheoremEckart–Young–Mirsky

Let $M \in \R^{d\times k}$ have singular values $\sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_{\min(d,k)}$ and SVD $M = \sum_i \sigma_i u_i v_i^\top$ . Then for any $r$ , the truncation $M_r = \sum_{i\le r}\sigma_i u_iv_i^\top$ satisfies

\min_{\operatorname{rank}(N)\le r}\|M - N\|_F^2 = \|M-M_r\|_F^2 = \sum_{i&gt;r}\sigma_i^2,

and the analogous statement in spectral norm with value $\sigma_{r+1}$ .

Key idea

Eckart–Young tells you the price of a rank budget, given the matrix: it is the tail of the singular value spectrum. So LoRA at rank $r$ can represent the ideal full fine-tuning update well exactly when that update has rapidly decaying singular values. The theorem converts the question "is LoRA enough?" into an empirical question about a spectrum — and notice that it does not answer it.

The hypothesis underneath

ConjectureThe low-rank update hypothesis

Conjecture — believed, supported by evidence, not proved.

For a pretrained model of sufficient capability and a downstream task in its neighbourhood, there exists a weight update $\Delta W^\star$ achieving near-full-fine-tuning performance whose singular values decay fast enough that $\operatorname{rank}_\eps(\Delta W^\star) \ll \min(d,k)$ .

Common mistake

This is the load-bearing assumption of every low-rank adaptation method, and it is not a theorem. What exists is evidence:

Measurements that fine-tuning reaches strong performance when optimisation is confined to a random low-dimensional subspace, with the required dimension shrinking as pretrained models get better.
Direct reports that LoRA at modest rank matches full fine-tuning on many downstream tasks.
Observations that the spectrum of measured $\Delta W$ from full fine-tuning does decay quickly.

What does not exist is a proof, a characterisation of which tasks satisfy it, or a way to predict the needed rank before training. There are also documented failure cases — notably when fine-tuning must inject substantial new knowledge rather than re-weight existing capability, where reported gaps to full fine-tuning are larger. Treat "LoRA is as good as full fine-tuning" as a claim with a scope that has to be checked per task, not as an established fact.

What this does not say

Rank is not the only thing that matters. Which matrices carry adapters — attention projections only, or the MLP too — changes results substantially and is not addressed by anything above. The propositions here concern a single matrix in isolation; the composition of many adapted layers, and the interaction with a frozen quantized base as in QLoRA, are outside what has been proved here.

References

PaperE. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
PaperA. Aghajanyan, L. Zettlemoyer, S. Gupta (2020). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
PaperT. Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

PropositionTrainable parameter countproof includedA rank-r adapter on a d×k matrix trains r(d+k) parameters instead of dk, a reduction by dk/(r(d+k)).
PropositionZero initialisation preserves the pretrained functionproof includedInitialising B = 0 makes the adapted model identical to the pretrained one at step zero while keeping the gradient non-zero.
PropositionThe α/r scaling makes updates rank-invariant in magnitudeproof includedUnder standard initialisation, scaling by α/r keeps the expected size of the update independent of r, so the learning rate need not be retuned per rank.
TheoremEckart–Young–Mirskyproof includedThe best rank-r approximation in Frobenius or spectral norm is given by truncating the SVD, with error determined by the discarded singular values.
ConjectureThe low-rank update hypothesisThat the weight change required to adapt a pretrained model to a downstream task is well approximated by a low-rank matrix — assumed by LoRA, supported empirically, not proved.

Problems (5)

All problems