The DPO gradient is a weighted contrastive step

ProofAdvanced⏱ 30 minLLMsdpogradienthard examplescontrastive

Problem

Compute $\nabla_\theta\mathcal L_{\mathrm{DPO}}$ and interpret the two factors. Explain why the weighting gives automatic hard-example emphasis, and what happens as training progresses.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
The cross-entropy gradient is the prediction error	LLMs	Proof	Beginner	15 min
Rewards are identifiable only up to a prompt-dependent shift	LLMs	Proof	Intermediate	30 min
The softmax Jacobian as a variance, and saturation	Deep Learning	Proof	Intermediate	30 min
Why exactly one factor starts at zero	Fine-Tuning	Proof	Intermediate	30 min
Closed form of the KL-regularised RL optimum abyss	LLMs	Proof	Graduate	1 hour