Uniform Quantization and Its Error

Affine quantizers, the exact error bound, the additive-noise model that is not a theorem, and the clipping–resolution trade-off.

QuantizationAdvanced⏱ 1 hour12 min readquantizationroundingsqnrdynamic rangeint8

5 problems →

AssumesLinear Algebra for Machine Learning · Expectation, Variance and Conditional Expectation

The affine quantizer

DefinitionAffine quantizer

For a target integer range $\lbrace 0,1,\dots,2^n-1 \rbrace$ and a clipping interval $[\alpha,\beta]$ , define the scale and zero point

\Delta = \frac{\beta-\alpha}{2^n-1}, \qquad z = -\operatorname{round}\!\left(\frac{\alpha}{\Delta}\right),

and the quantize / dequantize pair

Q(x) = \operatorname{clamp}\left(\operatorname{round}\!\left(\frac{x}{\Delta}\right) + z,\ 0,\ 2^n-1\right), \qquad \widehat{x} = \Delta\,\big(Q(x) - z\big).

The quantizer is symmetric when $z=0$ , which requires $\alpha = -\beta$ .

PropositionAffine quantization round-trip error

If $x \in [\alpha,\beta]$ , so that no clamping occurs, then

\big|\widehat{x} - x\big| \ \le\ \frac{\Delta}{2}.

This is a deterministic bound — it holds for every input, with no probabilistic assumption. Keep it separate in your mind from what follows.

Error as a random variable

PropositionExact error law on a single cell

Let $X \sim \mathrm{Unif}\big([(k-\tfrac12)\Delta,\ (k+\tfrac12)\Delta]\big)$ for an integer $k$ , so that $X$ lies in the cell rounding to $k\Delta$ . Then the error $E = \widehat X - X$ satisfies

E \sim \mathrm{Unif}\left[-\frac{\Delta}{2}, \frac{\Delta}{2}\right], \qquad \E[E] = 0, \qquad \Var(E) = \frac{\Delta^2}{12}.

HeuristicThe additive white-noise model

Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.

For a general input distribution, one models quantization as

\widehat X = X + E, \qquad \E[E]=0,\quad \Var(E) = \frac{\Delta^2}{12},\quad E \perp X,

with $E$ uncorrelated across coordinates.

Common mistake

This is not a theorem, and the distinction matters. The error is a deterministic function of the input, $E = \Delta\operatorname{round}(X/\Delta) - X$ , so it cannot be independent of $X$ — for instance, $X$ already on the grid forces $E = 0$ . The model is accurate in the high-resolution regime, where $\Delta$ is small relative to the scale over which the density of $X$ varies, so that the density is nearly constant across each cell and the previous proposition applies cell by cell. There is an exact version (Widrow's quantizing theorem) requiring the characteristic function of $X$ to vanish outside $(-\pi/\Delta, \pi/\Delta)$ , a condition no real distribution satisfies.

At the bit widths that matter for LLMs — 4 bits, sometimes 3 — the high-resolution regime is plainly not in force: there are 16 levels covering the whole range of a weight group. Predictions derived from this model should be treated as orientation, not as bounds. The deterministic $|E| \le \Delta/2$ remains valid throughout.

Signal-to-quantization-noise ratio

PropositionSix decibels per bit

Assume the noise model, a symmetric $b$ -bit quantizer with clipping range $[-A, A]$ so that $\Delta = 2A/2^{b}$ , and a signal with variance $\sigma^2$ and negligible clipping. Then

\mathrm{SQNR}_{\mathrm{dB}} = 10\log_{10}\frac{\sigma^2}{\Var(E)} = 6.02\,b + 4.77 - 20\log_{10}\!\left(\frac{A}{\sigma}\right).

Key idea

Two readings, both useful:

Each bit buys about 6 dB, a factor of 4 in noise power. Going from 8-bit to 4-bit costs 24 dB — a 250× increase in quantization noise power, which is why 4-bit needs machinery that 8-bit does not.
The loading factor $A/\sigma$ enters as a penalty. If one outlier forces $A$ to be $50\sigma$ , the term $20\log_{10}50 \approx 34$ dB eats nearly six bits of precision. The grid is being spent on a range almost no value occupies.

That second reading is the central practical fact about quantizing transformers, and the reason per-tensor scaling is inadequate for activations.

Clipping versus resolution

PropositionClipping and resolution trade off

Let $X$ have density $f$ , symmetric about $0$ , and quantize with $b$ bits over $[-A,A]$ . Under the noise model for the unclipped part, the mean squared error decomposes as

\mathrm{MSE}(A) = \underbrace{\frac{A^2}{3\cdot 2^{2b}}\,\P(|X| \le A)}_{\text{rounding}} \;+\; \underbrace{2\int_A^\infty (x-A)^2 f(x)\,dx}_{\text{clipping}} .

The first term is increasing in $A$ and the second decreasing, so if $X$ is unbounded the minimiser is attained at some finite $A^\star > 0$ .

The formats in use

Open problem

Open problem — no proof and no counterexample is known to the author of this page.

There is no satisfactory theory predicting end-task degradation from quantization error. The chain from weight-level MSE, through layer output perturbation, to next-token distribution shift, to downstream accuracy is not analytically controlled for networks of this depth: layer-wise error bounds compose multiplicatively through Lipschitz constants that are far too loose to be informative. Quantization methods are selected by measurement, and claims that a format "preserves quality" are empirical claims about particular benchmarks, not theorems.

References

SurveyR. M. Gray, D. L. Neuhoff (1998). QuantizationThe standard reference for the theory, including when the additive-noise model is and is not valid.
PaperT. Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

PropositionAffine quantization round-trip errorproof includedWith step size Δ and no clipping, the dequantized value differs from the original by at most Δ/2, deterministically.
PropositionExact error law on a single cellproof includedIf the input is uniform on one quantization cell, the error is exactly uniform on [−Δ/2, Δ/2] with variance Δ²/12.
HeuristicThe additive white-noise modelTreating quantization error as zero-mean noise of variance Δ²/12, independent of the signal, is an approximation that fails at low bit widths.
PropositionSix decibels per bitproof includedUnder the noise model, each additional bit adds 6.02 dB of signal-to-quantization-noise ratio.
PropositionClipping and resolution trade offproof includedTotal distortion splits into a clipping term that decreases with range and a rounding term that grows with it, so an interior optimum exists.

Problems (5)

All problems