Six decibels per bit, and what outliers cost
NumericalBeginner⏱ 30 minQuantizationsqnrloading factoroutliers
Problem
- Compute SQNR for a symmetric 8-bit quantizer with loading factor .
- Repeat at 4 bits with the same loading factor.
- A tensor has one value at . Compute the SQNR at 8 bits when is set to the maximum, and state how many bits are effectively wasted.
Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.
Solution
Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.
Related problems
All problems| Solved | Problem | Field | Type | Difficulty | Time |
|---|---|---|---|---|---|
| The round-trip error bound | Quantization | Proof | 15 min | ||
| The true bit width of a group-quantized format | Quantization | Numerical | 15 min | ||
| Effective bits lost to dynamic range | Quantization | Numerical | 15 min | ||
| The Δ²/12 noise variance | Quantization | Proof | 15 min | ||
| Optimal clipping for a Laplace-distributed tensor | Quantization | Proof | 1 hour |