Hessian-Aware Rounding

Rounding each weight to the nearest grid point is not optimal — the layer-wise reconstruction objective has a closed-form correction, and that is what GPTQ computes.

QuantizationGraduate⏱ 3 hours14 min readgptqobqoptimal brain surgeonhessianlayer-wise

4 problems →

AssumesUniform Quantization and Its Error · Linear Algebra for Machine Learning · Lagrangian Duality and the KKT Conditions

The layer-wise objective

Post-training quantization methods almost never optimise the end-to-end loss — that would need labels, backpropagation through the whole network, and far more compute. Instead they fix each layer in turn and ask that its output be preserved on a small calibration set.

PropositionThe layer-wise objective is a quadratic form in the weight error

Let a linear layer have weights $W \in \R^{m \times d}$ and let $X \in \R^{d \times n}$ hold $n$ calibration inputs as columns. Then

\big\|WX - \widehat W X\big\|_F^2 \;=\; \sum_{i=1}^{m} \delta_i^\top \big(XX^\top\big)\, \delta_i, \qquad \delta_i = \big(\widehat W - W\big)_{i,:}^\top ,

so the problem decouples across the $m$ output rows, each governed by the same matrix $H := 2XX^\top$ , the Hessian of the row objective.

The optimal correction

TheoremOptimal update under a fixed rounding

Let $H \succ 0$ , let $w \in \R^d$ be the current row, and fix coordinate $q$ to the quantized value $\operatorname{quant}(w_q)$ . Among all $\delta \in \R^d$ satisfying that constraint, the one minimising $\tfrac12\delta^\top H \delta$ is

\boxed{\;\delta^\star \;=\; -\,\frac{w_q - \operatorname{quant}(w_q)}{\big[H^{-1}\big]_{qq}}\;H^{-1}e_q\;}

where $e_q$ is the $q$ -th standard basis vector.

Assumptions

$H \succ 0$ , so the objective is strictly convex and $H^{-1}$ exists; the constraint is the single linear equation $e_q^\top \delta = \operatorname{quant}(w_q) - w_q$ . No assumption is made about the other coordinates — they are all free to move, which is precisely the point.

Derivation

Write $c = \operatorname{quant}(w_q) - w_q$ and form
$\mathcal L(\delta,\lambda) = \tfrac12\delta^\top H\delta + \lambda\big(e_q^\top\delta - c\big).$
A strictly convex quadratic under one linear equality constraint; KKT reduces to stationarity of the Lagrangian.
$\nabla_\delta \mathcal L = H\delta + \lambda e_q = 0 \implies \delta = -\lambda H^{-1}e_q$ .
Stationarity in δ.
$e_q^\top \delta = -\lambda\, e_q^\top H^{-1} e_q = -\lambda\big[H^{-1}\big]_{qq} = c$ , hence $\lambda = -c/[H^{-1}]_{qq}$ , which is well defined because $[H^{-1}]_{qq} > 0$ for $H \succ 0$ .
Impose the constraint to identify the multiplier.
$\delta^\star = \frac{c}{[H^{-1}]_{qq}}H^{-1}e_q = -\frac{w_q - \operatorname{quant}(w_q)}{[H^{-1}]_{qq}}H^{-1}e_q .$
Strict convexity makes this stationary point the unique global minimiser. $\blacksquare$
Substitute back, and rewrite c in terms of the rounding residual.

Key idea

Read the formula: $H^{-1}e_q$ is the $q$ -th column of the inverse Hessian, so the correction spreads over every remaining coordinate, in proportion to how strongly each is coupled to $q$ through the input covariance. Round-to-nearest is the special case where you refuse to apply this correction at all.

CorollaryThe cost of quantizing a coordinate

The minimum objective value incurred by fixing coordinate $q$ is

\mathcal E_q \;=\; \tfrac12 (\delta^\star)^\top H \delta^\star \;=\; \frac{\big(w_q - \operatorname{quant}(w_q)\big)^2}{2\big[H^{-1}\big]_{qq}} .

So the greedy rule is: quantize next the coordinate with the smallest $(w_q - \operatorname{quant}(w_q))^2 / [H^{-1}]_{qq}$ — small rounding residual, or a direction the data barely constrains.

Doing it for the whole row without $d$ inversions

Naively, after fixing $q$ you would re-invert the Hessian restricted to the remaining coordinates. That would be $O(d^4)$ per row. It is unnecessary.

LemmaInverse Hessian after deleting a coordinate

Let $H \succ 0$ and let $F$ index its coordinates with $q \in F$ . Writing $H_{-q}$ for $H$ with row and column $q$ removed,

\big(H_{-q}\big)^{-1} \;=\; \Big[H^{-1} - \frac{1}{[H^{-1}]_{qq}}\,H^{-1}e_q\,e_q^\top H^{-1}\Big]_{-q},

a rank-one correction followed by deletion.

Algorithm — Hessian-aware rounding of one layer

Accumulate $H = 2XX^\top$ over the calibration set and damp: $H \leftarrow H + \lambda I$ .
Compute $H^{-1}$ once — shared by all $m$ rows, since the proposition showed they share $H$ .
For $q = 1,\dots,d$ in a fixed order, and for all rows in parallel: a. Set $\widehat w_q = \operatorname{quant}(w_q)$ . b. Apply $\delta^\star$ to the not-yet-quantized coordinates. c. Update $H^{-1}$ by the rank-one deletion above.
Emit the quantized weights and the per-group scales.

Is the correction actually needed?

PropositionRound-to-nearest is suboptimal whenever inputs are correlated

Suppose $H$ is not diagonal. Then there exist a weight vector $w$ and a quantization grid for which round-to-nearest incurs strictly larger layer-wise error than the corrected procedure.

The gap is governed by $1-\rho^2$ : the more correlated the inputs, the more there is to gain. Since the input activations of a transformer layer are strongly correlated across channels, this is the regime that applies.

What this does not say

The objective optimised here is layer-wise output reconstruction on a calibration set, not the model's loss. Three gaps follow, and they are all real:

Error accumulates across layers. Each layer is optimised assuming its input is the unquantized model's input; once earlier layers are quantized, that assumption is false. Sequential variants that feed quantized activations forward address this partially.
Calibration set dependence. $H$ is estimated from a few hundred sequences; a distribution shift at deployment invalidates it, and nothing in the derivation bounds that.
Reconstruction is a proxy. Nothing here shows small layer-wise MSE implies small degradation in generation quality. The literature reports that it usually does; that is an empirical regularity, not a consequence of any of the results above.

References

PaperY. LeCun, J. Denker, S. Solla (1989). Optimal Brain Damage
PaperB. Hassibi, D. Stork (1992). Second Order Derivatives for Network Pruning: Optimal Brain SurgeonThe constrained-quadratic argument reproduced below originates here, for pruning.
PaperE. Frantar, S. Ashkboos, T. Hoefler, D. Alistarh (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

PropositionThe layer-wise objective is a quadratic form in the weight errorproof includedReconstruction error decouples across output rows and equals a quadratic form with Hessian 2XXᵀ.
TheoremOptimal update under a fixed roundingproof includedFixing one weight to its quantized value, the error-minimising correction to the remaining weights has a closed form involving one column of the inverse Hessian.
CorollaryThe cost of quantizing a coordinateproof includedThe incurred error is the squared rounding residual divided by twice the corresponding diagonal entry of the inverse Hessian.
LemmaInverse Hessian after deleting a coordinateproof includedRemoving one coordinate updates the inverse Hessian by a rank-one correction, so the whole sweep costs one inversion rather than one per weight.
PropositionRound-to-nearest is suboptimal whenever inputs are correlatedproof includedIf the input covariance is not diagonal, rounding each weight independently is strictly worse than the corrected update for some weight vector.

Problems (4)

All problems