The softmax Jacobian is a covariance

ProofIntermediate⏱ 30 minFoundationssoftmaxjacobianvariancepsd

Problem

Let $p = \operatorname{softmax}(z)$ for $z \in \R^K$ . Prove that

\frac{\partial p}{\partial z} = \operatorname{diag}(p) - pp^\top,

that this matrix is positive semidefinite with $(\operatorname{diag}(p)-pp^\top)\mathbf 1 = 0$ , and that for any $v$ ,

v^\top\big(\operatorname{diag}(p)-pp^\top\big)v = \operatorname{Var}_{i\sim p}(v_i).

Deduce the rank of the Jacobian when $p$ has full support.

Write a complete argument. Every inequality you use should be one you could state precisely.

Hints — 0 of 3 revealed

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
The cross-entropy gradient is the prediction error	LLMs	Proof	Beginner	15 min
The Δ²/12 noise variance	Quantization	Proof	Beginner	15 min
Tightness of the descent lemma	Foundations	Proof	Beginner	15 min
All directional derivatives exist, yet the function is not differentiable	Foundations	Proof	Intermediate	30 min
Gradient magnitude through a deep chain	Deep Learning	Proof	Intermediate	30 min