Derive the DPO objective

ProofGraduate⏱ 1 hourLLMsdporeparameterisationbradley-terryderivationflagship

Problem

Starting from the closed-form KL-regularised optimum and the Bradley–Terry model, derive

\mathcal L_{\mathrm{DPO}} = -\E\log\sigma\Big(\beta\log\tfrac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \beta\log\tfrac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}\Big),

showing explicitly where the intractable partition function cancels and why that cancellation is not accidental.

Write a complete argument. Every inequality you use should be one you could state precisely.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
Derive Hoeffding's inequality from Hoeffding's lemma	High-Dim Probability	Proof	Intermediate	30 min
Derive the dual of a quadratic program	Advanced Optimization	Proof	Intermediate	30 min
Rewards are identifiable only up to a prompt-dependent shift	LLMs	Proof	Intermediate	30 min
Prove the O(1/t) rate for gradient descent abyss	Optimization	Proof	Advanced	1 hour
Sharpness is not a property of the function	DL Theory	Proof	Advanced	30 min