Maximum likelihood is KL minimisation

ProofBeginner⏱ 15 minLLMsmlekl divergencecross entropy

Theory: Language Modelling, Cross Entropy and Perplexity →

Problem

Prove $\mathcal L(\theta) = H(\hat p) + \mathrm{KL}(\hat p\|q_\theta)$ and deduce that minimising cross entropy is minimising the forward KL. Explain the mode-covering consequence.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Perplexity is not comparable across tokenisers	LLMs	Numerical	Beginner	15 min
The cross-entropy gradient is the prediction error	LLMs	Proof	Beginner	15 min
The irreducible loss is the conditional entropy	LLMs	Proof	Intermediate	30 min
Temperature monotonically controls entropy	LLMs	Proof	Advanced	30 min
The KL divergence of best-of-n abyss	Agents	Proof	Graduate	1 hour