Rewards are identifiable only up to a prompt-dependent shift

ProofIntermediate⏱ 30 minLLMsidentifiabilitybradley-terryinvariance

Problem

Show that $r$ and $r+c(x)$ induce identical Bradley–Terry likelihoods, and that they induce identical KL-regularised optimal policies. What does this imply about interpreting reward-model scores?

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
The DPO gradient is a weighted contrastive step	LLMs	Proof	Advanced	30 min
Closed form of the KL-regularised RL optimum abyss	LLMs	Proof	Graduate	1 hour
Derive the DPO objective abyss	LLMs	Proof	Graduate	1 hour