Why attention divides by the square root of d_k

ProofIntermediate⏱ 30 minDeep Learningattentionscalingvariancesoftmax

Theory: The Mathematics of Transformers →

Problem

Compute the mean and variance of $\langle q,k\rangle$ for independent random $q,k\in\R^{d_k}$ with i.i.d. entries of variance $\sigma^2$ . Deduce the scaling, and quantify the effect on the softmax at $d_k = 64$ with and without it.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Derive the attention crossover length	LLM Ops	Proof	Beginner	15 min
The cross-entropy gradient is the prediction error	LLMs	Proof	Beginner	15 min
The Δ²/12 noise variance	Quantization	Proof	Beginner	15 min
When does attention dominate the MLP?	Deep Learning	Numerical	Beginner	30 min
Attention is permutation equivariant	Deep Learning	Proof	Intermediate	30 min