Outlier Features and Scale Migration

A handful of activation channels with huge magnitudes wrecks per-tensor quantization — and an exact diagonal invariance lets you move the difficulty somewhere it hurts less.

QuantizationGraduate⏱ 1 hour8 min readoutlierssmoothquantawqper-channelactivation quantization

2 problems →

AssumesUniform Quantization and Its Error · The Mathematics of Transformers

The observation

Empirical observationSystematic outlier features in large transformers

Empirical observation — a regularity seen in experiments. No theorem on this page establishes it in general.

In transformer language models above roughly 6–7 billion parameters, a small set of hidden dimensions — on the order of tens out of thousands — carries activation magnitudes one to two orders of magnitude above the median channel. The affected dimensions are largely consistent across inputs and across layers, and zeroing them degrades the model sharply.

Why a few large values are so expensive

PropositionOutliers cost bits directly

Consider a symmetric $b$ -bit quantizer whose range is set by the tensor maximum, $A = \max_i|x_i|$ . Let $\tau$ be a typical magnitude, say the median of $|x_i|$ , and $R = A/\tau$ the outlier ratio. Then the number of quantization levels covering $[-\tau,\tau]$ is

2^{b}/R,

so ordinary values are represented as if with

b_{\text{eff}} = b - \log_2 R

bits.

At $R = 50$ this costs $\log_2 50 \approx 5.6$ bits. An 8-bit quantizer is then behaving like a 2.4-bit one for the values that make up almost the whole tensor — which is why naive per-tensor INT8 activation quantization of a large model collapses, while the same scheme on a small model is fine.

Key idea

Finer granularity attacks $R$ directly. Per-channel or per-group scales mean an outlier inflates $R$ only within its own group, leaving every other group at full effective precision. For weights this is straightforward. For activations it is awkward: the reduction axis of the matrix multiply is the one you would need to scale along, and a per-token scale must be computed at run time.

Moving the problem

PropositionDiagonal scale migration is exact

Let $X \in \R^{n\times d}$ , $W \in \R^{d \times m}$ , and let $S = \operatorname{diag}(s_1,\dots,s_d)$ with all $s_j \ne 0$ . Then

XW \;=\; \big(XS^{-1}\big)\big(SW\big).

The transformation rescales column $j$ of $X$ by $1/s_j$ and row $j$ of $W$ by $s_j$ , and changes the layer's output not at all.

Trivial as mathematics, and the basis of the main practical technique: the function is invariant, but the quantization error is not, because the two factors are quantized after the rescaling. Pick $s_j$ large for channels where activations are large, and the activation outlier is damped while the corresponding weight row grows.

Protecting the channels that matter

A second use of the same invariance: rather than equalising ranges, scale up the weight channels that the activations actually excite.

PropositionScaling up a salient channel reduces its relative quantization error

Consider group-wise symmetric quantization with step $\Delta = 2A/2^b$ where $A$ is the maximum magnitude in the group. Multiply one channel's weights by $s > 1$ and divide the matching activations by $s$ , per the invariance. If the group maximum is unchanged — that is, the scaled channel is still not the largest in its group, so the step stays $\Delta$ — then the contribution of that channel to the layer output has quantization error scaled by exactly $1/s$ .

Assumptions

The hypothesis "the group maximum is unchanged" is doing all the work and is the reason the technique is applied to a small fraction of channels. Scaling many channels, or scaling one past the group maximum, raises $A$ and hence $\Delta$ for the whole group — which increases the error on every other channel in it. The gain is a redistribution, not a free lunch: it is worth taking only when the protected channels matter disproportionately to the output, which is what "activation-aware" means in practice.

What this does not say

None of this bounds end-to-end degradation. The propositions above control error in one channel's contribution to one layer's pre-activation, under stated conditions. They say nothing about how that perturbation propagates through the remaining layers, nor about generation quality. As in the Hessian-aware setting, the case that these methods preserve model quality is empirical.

Open problem

Open problem — no proof and no counterexample is known to the author of this page.

Why outlier features exist at all is open. Candidate accounts — that they implement a soft attention-sink or no-op mechanism, that they are a consequence of layer normalisation interacting with softmax saturation, that they are an artefact of Adam's per-coordinate scaling — each explain part of the evidence and none predicts which dimensions will be affected in a model not yet trained. An account that did would likely also suggest how to train models that do not develop them, which would make most of this page unnecessary.

References

PaperT. Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at ScaleReports the emergence of systematic outlier features and their effect on INT8 accuracy.
PaperG. Xiao et al. (2023). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
PaperJ. Lin et al. (2024). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

Empirical observationSystematic outlier features in large transformersAbove a few billion parameters, a small fixed set of hidden dimensions carries activations one to two orders of magnitude larger than the rest.
PropositionOutliers cost bits directlyproof includedA per-tensor quantizer whose range is set by an outlier loses log2(max/typical) bits of effective resolution on ordinary values.
PropositionDiagonal scale migration is exactproof includedFor any invertible diagonal S, XW equals (XS⁻¹)(SW), so quantization difficulty can be moved between activations and weights without changing the function.
PropositionScaling up a salient channel reduces its relative quantization errorproof includedIf a channel's scale-up does not move the group maximum, its relative error falls by exactly the scale factor.

Problems (2)

All problems