Quantization Mathematics
Number formats, rounding error and its propagation, the Hessian-weighted objective behind GPTQ, and why a handful of outlier features breaks naive INT8.
Level 2 — Advanced
Level 3 — Graduate
- Hessian-Aware Rounding14 min
Rounding each weight to the nearest grid point is not optimal — the layer-wise reconstruction objective has a closed-form correction, and that is what GPTQ computes.
The layer-wise objective is a quadratic form in the weight errorOptimal update under a fixed roundingThe cost of quantizing a coordinate - Outlier Features and Scale Migration8 min
A handful of activation channels with huge magnitudes wrecks per-tensor quantization — and an exact diagonal invariance lets you move the difficulty somewhere it hurts less.
Systematic outlier features in large transformersOutliers cost bits directlyDiagonal scale migration is exact
Problems (11)
Open in the browser| Solved | Problem | Type | Difficulty | Time |
|---|---|---|---|---|
| The round-trip error bound | Proof | 15 min | ||
| The true bit width of a group-quantized format | Numerical | 15 min | ||
| Effective bits lost to dynamic range | Numerical | 15 min | ||
| Six decibels per bit, and what outliers cost | Numerical | 30 min | ||
| The Δ²/12 noise variance | Proof | 15 min | ||
| Migrating quantization difficulty | Numerical | 30 min | ||
| Which weight to quantize next | Proof | 30 min | ||
| Derive the optimal weight update abyss | Proof | 1 hour | ||
| Implement Hessian-aware rounding run | Coding | 1 day | ||
| Optimal clipping for a Laplace-distributed tensor | Proof | 1 hour | ||
| The rank-one inverse update abyss | Proof | 1 hour |