Skip to content
nonconvexcard

Six decibels per bit, and what outliers cost

NumericalBeginner⏱ 30 minQuantizationsqnrloading factoroutliers

Problem

  1. Compute SQNR for a symmetric 8-bit quantizer with loading factor A/σ=4A/\sigma = 4.
  2. Repeat at 4 bits with the same loading factor.
  3. A tensor has one value at 50σ50\sigma. Compute the SQNR at 8 bits when AA is set to the maximum, and state how many bits are effectively wasted.

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems
SolvedProblemFieldTypeDifficultyTime
The round-trip error boundQuantizationProofFoundation15 min
The true bit width of a group-quantized formatQuantizationNumericalFoundation15 min
Effective bits lost to dynamic rangeQuantizationNumericalBeginner15 min
The Δ²/12 noise varianceQuantizationProofBeginner15 min
Optimal clipping for a Laplace-distributed tensorQuantizationProofAdvanced1 hour