Implement LoRA and measure the spectrum

CodingAdvanced⏱ 3 hoursFine-Tuningloraimplementationspectrumeckart-young

Problem

Implement a LoRA layer wrapping a frozen linear layer, with correct initialisation and $\alpha/r$ scaling, and verify it computes the base function at step 0.
Fine-tune a small model fully, and measure the singular value spectrum of the resulting $\Delta W$ per layer.
Use Eckart–Young to compute, per layer, the rank needed to capture 90% of $\|\Delta W\|_F^2$ .
Train LoRA at that rank and at half of it, and compare against full fine-tuning.

Implement it, then check the implementation against something you can verify independently.

Write it

Solve it here8 tests · Python runs in your browser

import numpy as np

# Conventions, so the tests and your code agree:
#   W has shape (d_out, d_in) and a batch x has shape (batch, d_in),
#   so the base layer computes x @ W.T.
#   A has shape (r, d_in) and B has shape (d_out, r).

def init_lora(d_in, d_out, r, seed=0):
    """Return the adapter pair (A, B) at initialisation.

A is random with entries of scale about 1/sqrt(d_in); B is exactly zero.
    That asymmetry is the whole trick: the product B @ A vanishes at step 0,
    so the adapted layer starts as the base layer, but the gradient with
    respect to B does not vanish, so training can leave that point.
    """
    raise NotImplementedError

def lora_delta(A, B, alpha, r):
    """The effective weight update, (alpha / r) * B @ A, shape (d_out, d_in).

The alpha/r factor is what lets you change r without re-tuning the
    learning rate: it keeps the size of the update roughly independent of r.
    """
    raise NotImplementedError

def lora_forward(x, W, A, B, alpha, r):
    """The adapted layer: the frozen base plus the adapter, x @ (W + delta).T."""
    raise NotImplementedError

def rank_for_energy(delta_w, fraction=0.9):
    """Smallest k whose top-k singular values carry `fraction` of ||delta_w||_F^2.

By Eckart-Young the best rank-k approximation keeps exactly the top k
    singular values, so the squared error of the best rank-k fit is the tail
    sum of squared singular values. Return the smallest k with

sum_{i<k} s_i^2  >=  fraction * sum_i s_i^2.
    """
    raise NotImplementedError

⌘/Ctrl + ↵ to run · limit 60s · draft saved in this browser

8 tests will run against your code. The first run downloads the Python runtime — several megabytes, once per browser.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
Adapter parameter counts	Fine-Tuning	Numerical	Foundation	15 min
LoRA parameter accounting	LLMs	Numerical	Beginner	15 min
Will this fine-tune fit?	Fine-Tuning	Numerical	Beginner	30 min
Implement backpropagation from scratch run	Deep Learning	Coding	Intermediate	3 hours
Implement kernel ridge regression run	Classical ML	Coding	Intermediate	1 hour

Problem

Write it

Solution

Related problems