Migrating quantization difficulty

NumericalIntermediate⏱ 30 minQuantizationsmoothquantscalingmigration

Theory: Outlier Features and Scale Migration →

Problem

A channel $j$ has activation maximum $|X_j|_{\max} = 60$ and weight-row maximum $|W_j|_{\max} = 0.1$ . Typical channels have activation max $2$ and weight max $0.1$ .

Compute the migration scale $s_j$ at $\alpha = 0.5$ and the resulting maxima.
Repeat at $\alpha = 0.75$ .
Compute the activation and weight dynamic ranges before and after in each case, and say which $\alpha$ you would choose given that weights are quantized per channel to INT8 and activations per tensor to INT8.

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Derive the attention crossover length	LLM Ops	Proof	Beginner	15 min
Effective bits lost to dynamic range	Quantization	Numerical	Beginner	15 min
Mini-batching and the compute budget	SGD	Numerical	Intermediate	30 min
The α/r scaling	Fine-Tuning	Proof	Intermediate	30 min
Why attention divides by the square root of d_k	Deep Learning	Proof	Intermediate	30 min