RLHF, Bradley–Terry and DPO

The preference model, the KL-regularised RL objective, its closed-form optimum, and the algebra that turns it into a supervised loss.

LLMsResearch⏱ 1 day12 min readrlhfbradley-terryppodpokl regularisation

4 problems →

AssumesLanguage Modelling, Cross Entropy and Perplexity

Preferences to rewards

DefinitionBradley–Terry preference model

Given a prompt $x$ and two responses $y_1,y_2$ , assume there is a latent reward $r^\star(x,y)$ with

\P\big(y_1\succ y_2\mid x\big) = \frac{\exp r^\star(x,y_1)}{\exp r^\star(x,y_1)+\exp r^\star(x,y_2)} = \sigma\big(r^\star(x,y_1)-r^\star(x,y_2)\big) .

Given preference data $\Dcal = \{(x,y_w,y_l)\}$ (winner, loser), the reward model $r_\phi$ is fit by maximum likelihood:

\mathcal{L}_R(\phi) = -\E_{(x,y_w,y_l)\sim\Dcal}\Big[\log\sigma\big(r_\phi(x,y_w)-r_\phi(x,y_l)\big)\Big].

PropositionThe reward is identifiable only up to a prompt-dependent shift

For any $c:\Xcal\to\R$ , the rewards $r$ and $r' = r + c(x)$ induce identical preference probabilities, hence identical likelihood. Consequently the reward learned from preference data alone is determined only up to such a shift.

The RLHF objective and its optimum

TheoremClosed form of the KL-regularised RL optimum

Consider

\max_{\pi}\ \E_{x\sim\Dcal,\ y\sim\pi(\cdot\mid x)}\big[r(x,y)\big] - \beta\,\mathrm{KL}\big(\pi(\cdot\mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot\mid x)\big) .

The maximiser is

\pi^\star(y\mid x) = \frac{1}{Z(x)}\,\pi_{\mathrm{ref}}(y\mid x)\,\exp\!\left(\frac{r(x,y)}{\beta}\right), \qquad Z(x) = \sum_{y}\pi_{\mathrm{ref}}(y\mid x)\exp\!\left(\frac{r(x,y)}{\beta}\right).

Key idea

Two observations from the proof. First, the KL-regularised RL problem has a closed-form solution — a Gibbs/Boltzmann reweighting of the reference policy. The reason RLHF is implemented with PPO rather than by writing down this formula is purely computational: $Z(x)$ is a sum over all sequences and is intractable. Second, the prompt-dependent shift $c(x)$ from the identifiability proposition changes $r$ and $Z$ by matching factors, so $\pi^\star$ is unchanged — the unidentifiability is harmless here.

DPO

TheoremThe DPO objective

Under the Bradley–Terry model and the KL-regularised objective, the reward is recoverable from the optimal policy:

r(x,y) = \beta\log\frac{\pi^\star(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)} + \beta\log Z(x),

and substituting into the Bradley–Terry likelihood cancels $Z(x)$ , giving the objective

\mathcal{L}_{\mathrm{DPO}}(\theta) = -\E_{(x,y_w,y_l)}\left[\log\sigma\!\left( \beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)} -\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right)\right],

which involves no reward model and no sampling from $\pi_\theta$ .

Key idea

The whole of DPO is: the reward is a reparameterisation of the policy, and Bradley–Terry only ever uses reward differences, so the intractable normaliser drops out. No new optimisation theory, no new statistical assumption beyond those already in RLHF — a change of variables. It is a good example of how far algebra can go before any analysis is needed.

The DPO gradient, and what it does

$\Delta = \beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)} - \beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)} .$
Differentiate the logistic loss. Write the implicit reward margin as Δ.
$\nabla_\theta\mathcal{L}_{\mathrm{DPO}} = -\beta\,\E\Big[\sigma(-\Delta)\Big(\nabla_\theta\log\pi_\theta(y_w\mid x) - \nabla_\theta\log\pi_\theta(y_l\mid x)\Big)\Big].$
The derivative of −log σ(Δ) is −σ(−Δ) times the derivative of Δ.
The update is a contrastive step on the two responses.
Read the two factors. The bracket raises the log-probability of the winner and lowers that of the loser — an ordinary supervised signal.
This is an automatic hard-example weighting, with no scheduling.
The weight σ(−Δ) is large exactly when the current model ranks the pair wrongly, so mis-ranked pairs dominate the gradient and already-correct pairs contribute little.

Empirical observationWhat DPO does and does not replace

Empirical observation — a regularity seen in experiments. No theorem on this page establishes it in general.

DPO removes the separately-trained reward model and the on-policy RL loop, and is markedly simpler and cheaper. It is not equivalent to RLHF in practice.

What this does not say

The derivation is exact, but it establishes the equivalence at the optimum, for the exact Bradley–Terry model, with unlimited preference data. Several things differ in practice:

Off-distribution behaviour. RLHF's reward model is queried on samples from the current policy, so it supplies a signal on responses that the preference dataset never contained. DPO only ever sees the fixed pairs, so the implicit reward is unconstrained off the preference distribution, and the optimised policy can drift into regions where nothing pins it down.
The Bradley–Terry assumption is inherited, not removed. Both methods assume preferences are generated by a single latent scalar reward with a logistic link. Non-transitive or heterogeneous human preferences violate this, and no amount of data fixes it.
The KL constraint is enforced differently. In RLHF it is an explicit penalty on samples; in DPO it is implicit in the $\log\pi_\theta/\pi_{\mathrm{ref}}$ parameterisation, and the effective constraint is weaker on unseen responses.
Empirically the two give different behaviour, with DPO sometimes reducing output diversity more sharply and being more sensitive to the quality of $\pi_{\mathrm{ref}}$ . Which is better depends on the setting, and the literature has not converged.

HeuristicReward hacking and the role of the KL term

Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.

As $\beta\to0$ the optimal policy concentrates on $\argmax_y r(x,y)$ — the maximiser of the learned reward, which is a finite-sample estimate. That maximiser is typically an adversarial input to the reward model rather than a good response.

What this does not say

This is Goodhart's law with a mathematical form. The learned $r_\phi$ approximates $r^\star$ well on the preference distribution, and the optimiser's job is to find points where they differ most — precisely the points where the approximation is untested. The KL term is a proximity constraint keeping the policy inside the region where $r_\phi$ was fit.

What is not available: a theorem bounding the true reward of the optimised policy in terms of the reward model's error and $\beta$ . Such a bound would need a uniform error guarantee for $r_\phi$ over the region reachable at a given KL budget, and the available uniform convergence tools are vacuous at this scale. Choosing $\beta$ is therefore empirical, and reward hacking is diagnosed after the fact rather than prevented by design.

Problems

Closed form of the KL-regularised RL optimumProof · Graduate · 1 hour Derive the DPO objectiveProof · Graduate · 1 hour Rewards are identifiable only up to a prompt-dependent shiftProof · Intermediate · 30 min The DPO gradient is a weighted contrastive stepProof · Advanced · 30 min

References

PaperRalph A. Bradley and Milton E. Terry (1952). Rank Analysis of Incomplete Block Designs
PaperPaul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg and Dario Amodei (2017). Deep Reinforcement Learning from Human Preferences
PaperLong Ouyang and collaborators (2022). Training language models to follow instructions with human feedback
PaperRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon and Chelsea Finn (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
PaperJohn Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford and Oleg Klimov (2017). Proximal Policy Optimization Algorithms

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

DefinitionBradley–Terry preference modelThe probability that one response is preferred is the logistic function of the reward difference.
PropositionThe reward is identifiable only up to a prompt-dependent shiftproof includedBradley-Terry depends on reward differences, so any function of the prompt alone can be added without changing the likelihood.
TheoremClosed form of the KL-regularised RL optimumproof includedThe optimal policy is the reference policy reweighted by the exponentiated reward.
TheoremThe DPO objectiveproof includedInverting the closed-form optimum expresses the reward in terms of the policy, turning RLHF into a supervised classification loss.
Empirical observationWhat DPO does and does not replaceDPO removes the reward model and the RL loop but inherits the Bradley-Terry assumption and behaves differently off the preference distribution.
HeuristicReward hacking and the role of the KL termThe KL penalty is what keeps the optimised policy inside the region where the learned reward is trustworthy.

Problems (4)

All problems