Skip to content
nonconvexcard

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray and others · 2022 · NeurIPS

LLMsrlhfinstruction tuningalignmentreward modelppo
Read the paper

The problem

A language model trained to predict the next token is optimising a proxy that does not match what users want, so it follows the corpus rather than the instruction.

The contribution

A three-stage pipeline — supervised fine-tuning on demonstrations, a reward model fit to human preference comparisons, and PPO optimisation of the policy against that reward with a KL penalty — which made a 1.3B model preferred to a 175B base model.

What problem does this paper solve?

Objective mismatch. Maximum likelihood on web text optimises KL(p^ ∥ qθ)\mathrm{KL}(\hat p\,\|\,q_\theta) against the empirical text distribution, which is not the objective "be helpful and follow the instruction". The paper's contribution is a way to optimise something closer to the latter.

What mathematical objects are introduced?

A reward model rϕ(x,y)r_\phi(x,y) fit by Bradley–Terry maximum likelihood on preference pairs, and the KL-regularised RL objective

max⁡π Ex,y∼π[rϕ(x,y)]−β KL(π ∥ πSFT)+γ Epretrain[log⁡π],\max_\pi\ \E_{x,y\sim\pi}\big[r_\phi(x,y)\big] - \beta\,\mathrm{KL}\big(\pi\,\|\,\pi_{\mathrm{SFT}}\big) + \gamma\,\E_{\text{pretrain}}\big[\log\pi\big],

where the last term mitigates capability regression on the pretraining distribution.

What assumptions are made?

That human preferences are generated by a single latent scalar reward with a logistic link (Bradley–Terry), that the labelers' preferences represent the intended objective, and that the KL budget keeps the policy in the region where rϕr_\phi is accurate.

What equations matter?

The Bradley–Terry likelihood and the KL-regularised objective above, whose optimum has the closed form π⋆∝πrefer/β\pi^\star\propto\pi_{\mathrm{ref}}e^{r/\beta} — intractable to compute directly, which is why PPO is used.

What theorem or result is important?

No theorem. The headline result: outputs of the 1.31.3B InstructGPT model were preferred by human labelers to those of the 175175B GPT-3 base model, a 100×100\times effective parameter saving from alignment alone.

What proof techniques are used?

None; the pipeline is engineering and the evaluation is human preference judgment.

What prior mathematics is required?

KL divergence and its variational properties; policy-gradient methods; the identifiability of Bradley–Terry rewards.

Limitations

Reward hacking as β→0\beta\to0, with no bound on the true reward of the optimised policy in terms of rϕr_\phi's error. An "alignment tax" — degradation on some capability benchmarks. Dependence on labeler demographics and instructions, which the paper is explicit about. And the complexity of the pipeline (three models, an RL loop) which motivated DPO.

Why it mattered

It is the paper that made language models usable as assistants, and every deployed chat model descends from this recipe. It also established preference data as the central training resource for the final stage of model development.

What came next