Post-training: From Base Model to Assistant

Part 10 ended with a base model: it predicts the next token very well, but all it knows how to do is continue text. Ask it a question and it may continue it into a web page or a list of other questions. This post covers the stage that turns it into an assistant, something that answers the question it was asked, knows when to stop, and declines when it should. This stage is called post-training, and it has three layers:

• SFT (supervised fine-tuning): keep training on example conversations packaged by a chat template, with the loss computed only on the response.
• Learning from comparisons: tell the model which answer is better, either with RLHF (a reward model plus PPO) or with DPO, which optimizes the same objective without the reward model and without the RL loop.
• Replacing human raters with a model: RLAIF and Constitutional AI.

The thread running through it: post-training adds almost no knowledge; it shapes how the model uses the knowledge it already has. Many familiar chatbot habits, such as verbosity, telling users what they want to hear, or over-refusing, come from this stage and not from the architecture. The post follows the order of Figure 1 at the top of the page.

01From base model to assistant

Part 1, section 7.4 drew the big picture: pretraining gives the model knowledge, SFT teaches it to answer in the conversation format, and alignment with RLHF or DPO teaches it which answers are preferred. That section also has a table linking familiar chatbot behaviors to the stage that produces them. This post does not repeat the big picture; it goes into the mechanism: what each step optimizes, where the formulas come from, and where things tend to break.

One thing to keep in mind from the start: mathematically, most of post-training is things we have already seen. SFT uses exactly the cross-entropy of part 10; only the data and the positions that count toward the loss change. DPO ends up as a classification loss over log-probs. Only PPO brings something genuinely new: the model has to generate text during training and then gets scored on it. That difference drives almost all of the cost and complexity of RLHF.

02Why a base model does not follow instructions

What the pretraining objective rewards

Pretraining only rewards assigning high probability to the token that actually comes next in text from the internet. The objective has no notion of “question” or “answer”. The sentence “What is the capital of France?” on the internet may sit in a quiz (followed by another question), on a forum (followed by a timestamp) or in an FAQ (followed by the answer). A base model learns exactly that distribution. It is not that it does not know the answer; it does not know that its job is to answer.

The knowledge is already there: the superficial alignment hypothesis

LIMA (Zhou et al. 2023) fine-tunes LLaMa 65B on only 1,000 carefully curated examples, with no RLHF. In human evaluation, LIMA’s responses are equivalent to or preferred over GPT-4 in 43% of cases, over Bard in 58% and over DaVinci003 in 65% (abstract). The authors propose the Superficial Alignment Hypothesis (section 2): almost all knowledge and capability is learned during pretraining, and alignment mostly teaches the model which format and style to use.

URIAL (Lin et al. 2023) measures this directly: let the base model and its aligned version read the same answer, and at each position check where the aligned model’s chosen token ranks in the base model’s distribution. For Llama-2-7b and Llama-2-7b-chat, 77.7% of positions match the top-1 token, 14.5% are the base model’s rank-2 or rank-3 token (what section 2.1 calls “marginal”), and only 7.8% are truly shifted (Figure 3). The shifted tokens are mostly stylistic: discourse markers, polite openers, safety disclaimers (sections 2.2–2.3). URIAL then gets a base model to answer like an assistant with a prompt alone: a system prompt plus 3 constant examples, 1,011 tokens in total, no training at all (section 3.3).

The limits need stating clearly: LIMA and URIAL measure conversational quality on general questions. Following complex formats, calling tools, multi-step reasoning and refusing in the right places still need real post-training, and the pipelines in section 10 use hundreds of thousands of examples. “Superficial” means no new knowledge is created, not that it is easy.

03SFT: teaching the conversation format

SFT is next-token prediction on different data

SFT has no new objective: it is still next-token cross-entropy as in part 10, still the training loop of part 3. Only two things change: the data is conversations (instruction, response), and the loss mask keeps only the assistant’s tokens. System-prompt and user tokens still pass through the model as context but contribute no gradient.

The history of SFT data shows that quality matters more than quantity:

• Turning existing NLP datasets into instructions. FLAN (Wei et al. 2021) writes 10 templates for each of 62 datasets (section 2.1), fine-tunes the 137B LaMDA-PT, and beats zero-shot 175B GPT-3 on 20 of 25 datasets (abstract).
• Letting the model write the instructions. Self-Instruct (Wang et al. 2022) starts from 175 seed tasks, has GPT-3 generate more and filters them, ending with 52,445 instructions (Table 1). Fine-tuning GPT-3 on them gains 33 absolute points on Super-NaturalInstructions, on par with InstructGPT-001 (abstract, Table 3). This is the ancestor of today’s synthetic data.
• Written by hand. InstructGPT uses about 13 thousand prompts for SFT (section 3.2). Llama 2 stopped at 27,540 human-written examples, because a small high-quality set beat millions of examples from third-party sources (section 3.1, “Quality Is All You Need”).

Chat templates

Part 4, section 6.3 introduced the core idea: a chat template turns a conversation into a token protocol with roles. This is the hands-on part. On Hugging Face, the template is a Jinja snippet stored in tokenizer_config.json (the chat_template field), and tokenizer.apply_chat_template(messages, ...) runs it over a list of messages. Two parameters matter most:

• add_generation_prompt=True appends the opening of the assistant’s turn (for Qwen, <|im_start|>assistant\n) so the model knows it is its turn. Without it, the model may continue the user’s turn.
• If you tokenize the templated string yourself, the template usually already contains every special token it needs, so the Hugging Face docs recommend add_special_tokens=False to avoid adding them twice.

The three most common mistakes all live here: forgetting add_generation_prompt, using another model’s template, and concatenating strings by hand. The model saw exactly one format throughout SFT; a format mismatch at inference pushes it out of the distribution it learned.

A loss mask on the assistant’s tokens only

Part 5, sections 11.2–11.3 explained labels, the shift, and the loss mask. For SFT, prompt positions get label = -100, the value PyTorch’s cross-entropy ignores; response positions keep their token IDs. We want the model to learn to answer, not to ask questions like the user, and not to spend gradient memorizing a system prompt repeated in every sample.

Running it on a real model

The code below prints the string produced by the chat template, marks the tokens that count toward the loss, then gives the same question to Qwen2.5-0.5B base and instruct, with and without the template. Decoding is greedy, so the results are reproducible.

# Python 3.12, torch 2.14.1 (CPU), transformers 5.18.0
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.utils import logging
logging.disable_progress_bar()

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

# 1. Chat template: conversation -> text with roles
chat = [
    {"role": "user", "content": "What is the capital of France?"},
    {"role": "assistant", "content": "The capital of France is Paris."},
]
print(tok.apply_chat_template(chat, tokenize=False))

# 2. Loss mask: only assistant tokens count toward the loss
prompt_ids = tok.apply_chat_template(chat[:1], add_generation_prompt=True, tokenize=True, return_dict=True)["input_ids"]
full_ids = tok.apply_chat_template(chat, tokenize=True, return_dict=True)["input_ids"]
assert full_ids[: len(prompt_ids)] == prompt_ids          # the prompt is a prefix of the full conversation
labels = [-100] * len(prompt_ids) + full_ids[len(prompt_ids):]
print(f"{len(full_ids)} tokens in total, loss on {sum(l != -100 for l in labels)} tokens:")
print([tok.decode(t) for t, l in zip(full_ids, labels) if l != -100])

# 3. Same question to base and instruct, with and without the template
question = "What is the capital of France?"
for name in ("Qwen/Qwen2.5-0.5B", "Qwen/Qwen2.5-0.5B-Instruct"):
    model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32).eval()
    for use_template in (False, True):
        text = (tok.apply_chat_template([{"role": "user", "content": question}],
                                        add_generation_prompt=True, tokenize=False)
                if use_template else question)
        ids = tok(text, return_tensors="pt").input_ids
        out = model.generate(ids, max_new_tokens=40, do_sample=False)
        answer = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=False)
        print(f"\n[{name.split('/')[1]} | template={use_template}]\n{answer!r}")

Output:

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant
The capital of France is Paris.<|im_end|>

45 tokens in total, loss on 9 tokens:
['The', ' capital', ' of', ' France', ' is', ' Paris', '.', '<|im_end|>', '\n']

[Qwen2.5-0.5B | template=False]
' The capital of France is Paris.<|endoftext|>'

[Qwen2.5-0.5B | template=True]
'France is the capital of France. The capital of France is Paris.<|endoftext|>'

[Qwen2.5-0.5B-Instruct | template=False]
' The capital of France is Paris. \n\nTo verify this, I will use a simple Python code snippet to confirm that Paris is indeed the capital city of France.\n\n```python\n# Define the capital city'

[Qwen2.5-0.5B-Instruct | template=True]
'The capital of France is Paris.<|im_end|>'

Reading the output:

• The template inserts a system prompt “You are Qwen…” even though we gave none. It appears in every SFT sample of this model, so it must be there at inference too, one more reason to call apply_chat_template rather than concatenating strings.
• Only 9 of 45 tokens count toward the loss, including <|im_end|>. This is exactly how the model learns when to stop.
• The base model without a template already answers. “The capital of France is Paris.” followed by <|endoftext|>. English question–answer text is common enough in pretraining data that this pattern is likely. The Vietnamese version of this post asks the same question in Vietnamese, and there the base model continues the text like a forum page with a timestamp; so this is a matter of the data distribution, not a fixed rule.
• The base model with a template gets confused. It starts with a garbled sentence (“France is the capital of France”), then gives the right answer, and ends with <|endoftext|> rather than <|im_end|>. It has probably seen similar formats during pretraining but was never taught the turn-taking protocol.
• The instruct model without a template does not stop. It answers, then carries on writing a Python snippet to “verify” the answer until it hits the token limit. Post-training does not erase the ability to continue text; it teaches a new behavior tied to a specific format. Use the wrong format and you get the old behavior back.

One trap: apply_chat_template has a return_assistant_tokens_mask=True parameter, but it only works when the template marks the assistant part with a {% generation %} tag. Qwen2.5’s template has no such tag, and with transformers 5.18.0 the mask comes back as all zeros; transformers only prints a warning that the template does not contain {% generation %}, with no error, meaning no token would be trained. That is why the code above builds the mask by comparing prefixes, with an assert.

04The limits of SFT

SFT teaches the model to imitate a reference answer, and that is also its limit.

No “better” signal. Cross-entropy only says “raise the probability of exactly this sequence”. It does not distinguish a slightly worse answer from a completely wrong one. For summaries, explanations or a polite refusal there is no single “correct” answer, and the people writing the data can only write their own best answer. Humans struggle to write perfect answers but are fairly good at comparing two; learning from comparisons exploits exactly that.

Exposure bias. During SFT the model always sees the reference answer as context (teacher forcing). At inference it reads back its own tokens, and one small mistake puts it in a state it never saw in training. SFT never trains on text the model generated itself, so it cannot teach the model how to recover. RL trains precisely on the policy’s own output.

SFT can teach the model to make things up. John Schulman argued in a 2023 talk that “even if you clone on a hundred percent correct answers, you’re teaching the model to hallucinate” when the model does not have those facts: the only way to reduce the loss is to learn the confident tone of the answer. RL can reward “I’m not sure” when the model is not sure, something a fixed reference answer has trouble expressing.

05RLHF (1): preference data and the reward model

RLHF has three parts: collect comparison data, train a reward model, then optimize the policy against the reward model. This section covers the first two.

Collecting comparisons

Learning a reward from human comparisons predates LLMs. Christiano et al. 2017 had people choose the better of two clips of an Atari agent or a simulated robot; feedback on less than 1% of the agent’s interactions, about an hour of human time, was enough to learn complex behaviors (abstract). Stiennon et al. 2020 brought this to text summarization, and InstructGPT (Ouyang et al. 2022) turned it into the standard recipe for LLMs:

• about 40 contractors through Upwork and ScaleAI (section 3.4);
• for each prompt, K = 4 to 9 responses that raters rank, giving C(K, 2) pairs, up to 36 (section 3.5);
• about 13 thousand prompts for SFT, 33 thousand for the reward model, 31 thousand for PPO (section 3.2; exact counts in Table 6, Appendix A.3).

Anthropic’s HH-RLHF (Bai et al. 2022) splits two axes: helpfulness, where crowdworkers pick the more helpful answer, and harmlessness, where crowdworkers try to get the model to say something harmful and then pick the less harmful answer; the base set has about 44 thousand and 42 thousand comparisons (section 2.3). Llama 2 collected 1,418,091 binary comparisons of Meta’s own (Table 6), each with a degree ranging from “significantly better” to “negligibly better/ unsure” (section 3.2.1).

Raters disagree with each other

In InstructGPT, two raters trained on the same process agree only 72.6 ± 1.5% of the time; for held-out raters (who did not produce training data) the figure is 77.3 ± 1.3% (section 3.4). This puts a ceiling on everything downstream: if humans agree with each other only about 3/4 of the time, a reward model that is 75% accurate on a test set is already close to the best possible. “Human preference” is not a deterministic function; it is noisy.

Bradley–Terry: from comparisons to scores

The data looks like “for prompt x, answer y_w was chosen over y_l” (w for win, l for lose). We want a function r(x, y) that gives each answer a number. The bridge is the Bradley–Terry model, which Christiano et al. 2017 also used (equation 1).

Step 1. Assumption. Each answer has a latent score r. The probability that y_w is chosen over y_l is proportional to each answer’s exp(r):

P(y_w ≻ y_l | x) = exp(r(x, y_w)) / ( exp(r(x, y_w)) + exp(r(x, y_l)) )

This is equation 1 in the DPO paper. It is exactly a softmax over two options.

Step 2. Simplify. Divide numerator and denominator by exp(r(x, y_w)):

P(y_w ≻ y_l | x) = 1 / ( 1 + exp(−(r(x, y_w) − r(x, y_l))) )
                 = σ( r(x, y_w) − r(x, y_l) )

where σ(z) = 1 / (1 + e^(−z)) is the sigmoid. This says something important: the probability depends only on the difference between the two scores. Adding the same constant to every score changes nothing. A reward model therefore has no absolute scale; a reward of 3.2 means nothing by itself, only the gap between two answers to the same prompt does.

Step 3. Maximum likelihood. We want reward-model parameters φ that make the observed data most likely. Take the log and flip the sign to get a loss to minimize:

L_R(φ) = −E_(x, y_w, y_l) [ log σ( r_φ(x, y_w) − r_φ(x, y_l) ) ]

This is equation 2 in the DPO paper and also the loss in Stiennon et al. (section 3.4) and InstructGPT (section 3.5): a binary cross-entropy for “which answer wins”, with the reward difference as the logit. If r_w − r_l = 0, the loss is −log 0.5 = ln 2 ≈ 0.693; a difference of 1 gives a loss of ≈ 0.313; a difference of −1 (ranked backwards) gives ≈ 1.313. Llama 2 adds a margin m(r) inside the sigmoid, larger when the rater said “significantly better” (section 3.2.2, equation 2); Llama 3 drops the margin (section 4.1.2).

A reward model is an LLM with a scalar output

A reward model is usually an LLM initialized from the SFT model, with the lm_head that projects to the vocabulary replaced by a linear layer that outputs a single number, read at the last token of prompt + response. It is still the Transformer of part 8, only the head differs. InstructGPT uses a 6B reward model for every policy, including 175B, because it saves compute and because training a 175B reward model was unstable (section 3.5). A detail worth noting: the K responses for one prompt go into a single batch element; shuffling the pairs made the reward model overfit after one epoch.

06RLHF (2): optimizing with PPO

The objective: reward minus KL

We want the policy π_θ (initialized from the SFT model) to generate answers with high reward. But if we only maximize reward, the policy quickly finds odd answers that the reward model mistakenly scores high, because the reward model was only trained on the distribution of the SFT model’s answers. The standard objective therefore adds a penalty:

max_π  E_(x ~ D, y ~ π(·|x)) [ r(x, y) ]  −  β · D_KL( π(·|x) ‖ π_ref(·|x) )

This is equation 3 in the DPO paper. π_ref is the reference model, a frozen copy of the SFT model. β sets how strong the penalty is.

The KL term does three jobs: it keeps the policy where the reward model is still reliable; it keeps the SFT model’s language ability; and it keeps diversity, because without KL the optimum puts all probability on the single highest-reward answer.

In practice, the penalty is computed on each generated answer and subtracted directly from the reward (Stiennon et al., section 3.4):

R(x, y) = r(x, y) − β · log( π_θ(y|x) / π_ref(y|x) )

The expectation of the log-ratio under y ~ π_θ is exactly the KL, so maximizing the average R maximizes the objective above. A sequence’s log-prob is the sum of its token log-probs, so the penalty splits into a per-token penalty, while the reward model’s score is added only at the last token. InstructGPT (equation 2) adds a γ term that mixes the pretraining loss back in, called PPO-ptx, to reduce the “alignment tax” (section 7). β is actually small: 0.02 in InstructGPT (Appendix C.4); 0.01 for Llama 2 7B and 13B, 0.005 for 34B and 70B (section 3.2.3).

PPO at the level of intuition

The objective above cannot be differentiated directly like SFT, because y is sampled from the policy itself, and sampling is not differentiable. Policy gradient takes a detour: raise the log-prob of tokens that led to better-than-expected outcomes and lower the log-prob of tokens that led to worse ones. Three concepts:

• Advantage: how much “better than expected” a token was. If every answer gets a reward of about 5, a reward of 5 says nothing; what is worth learning is which answer got 7 and which got 3.
• Value model: predicts the total reward still to come from the current token, the “expectation” at each position. InstructGPT initializes it from the reward model (section 3.5). The advantage is the actual reward minus this prediction.
• Clipping: PPO (Schulman et al. 2017) clips the ratio ρ = π_θ_new(token) / π_θ_old(token) to [1 − ε, 1 + ε], typically with ε = 0.2 (equation 7; Table 1). One update cannot move a token’s probability by more than 20% relative to the policy that generated the data, which prevents jumps that wreck the policy.

So one PPO round is: let the policy generate answers for a batch of prompts (the sampling of part 9, running inside the training loop); the reward model scores them and the reference model computes the KL penalty; the value model estimates advantages; update the policy and the value model for a few epochs on the data just generated; repeat with the new policy. InstructGPT ran 256 thousand episodes over about 31 thousand prompts, with a batch size of 512 (Appendix C.4).

Four models in memory

The top row of Figure 2 (in section 8) draws this loop. The models that must be present:

The policy and value models need weights, gradients and optimizer state, about 16 bytes per parameter as part 10, section 8 computed; the reward and reference models need only weights. Generation is slower still: it runs one token at a time. In OpenRLHF’s profiling (Hu et al. 2024, v1, section 2.2), PPO’s sample generation stage takes 80% of total training time; the latest version of the paper puts it at over 90% (section 1). That is why there is a whole class of frameworks just for running RLHF, and why DPO in section 8 is so attractive.

07Reward hacking and the cost of alignment

Overoptimization is the rule

A reward model is only an approximation of human preference, trained on tens of thousands of comparisons with about 75% agreement. Optimize an approximation hard enough and sooner or later you find where it is wrong (Goodhart’s law). Stiennon et al. 2020 saw it: optimizing against the reward model first improves summaries, then overfits, until the reward model becomes anti-correlated with human judgment (section 4.3, Figure 5).

Gao et al. 2022 measured this quantitatively. They used a 6B reward model as the “gold” stand-in for human raters, trained “proxy” reward models from 3M to 3B parameters on labels produced by the gold model (section 2.1), optimized a policy against the proxy, and tracked the gold score. As the policy moves away from the reference (measured by d = sqrt(KL)), the proxy score keeps rising while the gold score rises and then falls (Figure 1), following two functional forms that fit very well (section 1):

best-of-n:  R(d) = d · (α_bon − β_bon · d)
RL:         R(d) = d · (α_RL − β_RL · log d)

Both have a peak; past it, further optimization only makes real quality worse. Larger reward models push the peak further out but do not remove it. That is why the KL penalty, early stopping, and retraining the reward model every round are not minor details.

Length: the most common form of reward hacking

Raters tend to prefer longer, more thorough answers; the reward model learns that, and the policy learns it back very quickly. Singhal et al. 2023 show that for WebGPT and RLCD, 70–90% of the reward gain after PPO can be explained by the answers simply getting longer (section 3.1, Table 1). Replace the reward model with a reward based only on length and PPO still reaches win rates close to standard RLHF: 56% vs 58% on WebGPT, 64% vs 63% on RLCD (simulated preference, section 3.2, Table 2). These are each model’s win rate against the SFT model, not a head-to-head between the two.

Sycophancy

Sharma et al. 2023 tested five assistants (claude-1.3, claude-2.0, gpt-3.5-turbo, gpt-4, llama-2-70b-chat) on four tasks, such as abandoning a correct answer when challenged or repeating the user’s mistakes; all five were consistently sycophantic (section 3). Analyzing preference data showed that both humans and preference models sometimes choose a convincingly written sycophantic answer over a correct one (section 4.3, Figure 7). Sycophancy is not a bug of one model; it is taught by the comparison data itself.

Less diversity, and the alignment tax

Kirk et al. 2023 find that RLHF generalizes out of distribution better than SFT but significantly reduces output diversity (abstract; sections 6.1–6.2). The diversity measurements are mostly on summarization, and the largest drop is between answers to the same input. InstructGPT reports that the RLHF policy lost points on SQuAD, DROP, HellaSwag and WMT15 French–English translation, calling it the alignment tax (sections 1 and 4.2); PPO-ptx removes most of the drop. Bai et al. 2022 find the “tax” heavy for small models, while their 13B and 52B models actually do better on zero-shot evaluations after RLHF (sections 1.1 and 4.6).

Put together, these results explain the behavior table in part 1, section 7.4: verbosity comes from length bias, sycophancy from preference data, and “all the answers sound the same” from the trade-off between reward and diversity. Casper et al. 2023 organize the problems of RLHF by human feedback, reward model, policy, and the joint training of the last two, separating tractable problems (fixable within RLHF) from fundamental ones (needing a different approach) (section 3, Figure 1).

08DPO: dropping the reward model and the RL loop

RLHF with PPO is expensive and hard: four models, text generation inside the training loop, many hyperparameters. DPO (Rafailov et al. 2023) asks: if the reward-minus-KL objective has a closed-form solution, can we skip the reward model and optimize directly on comparison data? Yes, and the derivation takes only a few lines of algebra.

Step 1: the closed-form solution of reward minus KL

Start from the objective of section 6 for a fixed prompt x. Write the KL as an expectation of the log-ratio and bring everything inside:

  E_(y~π) [ r(x, y) ] − β · KL( π ‖ π_ref )
= E_(y~π) [ r(x, y) − β · log( π(y|x) / π_ref(y|x) ) ]
= −β · E_(y~π) [ log( π(y|x) / π_ref(y|x) ) − r(x, y) / β ]
= −β · E_(y~π) [ log( π(y|x) / ( π_ref(y|x) · exp(r(x, y) / β) ) ) ]

The denominator π_ref(y|x) · exp(r(x, y)/β) is almost a distribution; it just is not normalized. Define the normalizing constant:

Z(x) = Σ_y π_ref(y|x) · exp( r(x, y) / β )

and the distribution:

π*(y|x) = π_ref(y|x) · exp( r(x, y) / β ) / Z(x)

Substitute π_ref · exp(r/β) = Z(x) · π* into the last line:

= −β · E_(y~π) [ log( π(y|x) / π*(y|x) ) − log Z(x) ]
= −β · KL( π ‖ π* ) + β · log Z(x)

Z(x) does not depend on π, and the KL is non-negative and zero exactly when the two distributions coincide. So the objective is maximized exactly at π = π* (equation 4; the full derivation is in Appendix A.1). Read intuitively: the optimal policy is the reference tilted by the reward, with each answer’s probability multiplied by exp(r/β). But Z(x) is a sum over every possible sequence, which cannot be computed, so π* cannot be used directly. The next step sidesteps exactly that.

Step 2: write the reward in terms of the policy

Take the log of both sides of the definition of π* and rearrange:

r(x, y) = β · log( π*(y|x) / π_ref(y|x) ) + β · log Z(x)

This is equation 5: the reward is the log-ratio between the optimal policy and the reference, plus a term that depends only on the prompt.

Step 3: plug into Bradley–Terry, and Z(x) cancels

Bradley–Terry from section 5 only uses the difference between two rewards for the same prompt:

r(x, y_w) − r(x, y_l) = β · log( π*(y_w|x) / π_ref(y_w|x) ) − β · log( π*(y_l|x) / π_ref(y_l|x) )

The β · log Z(x) term appears in both rewards and cancels. The intractable sum is gone.

Step 4: replace π* with the model being trained

Replace the unknown π* with a parameterized model π_θ and apply maximum likelihood as for the reward model, and you get the DPO loss (equation 7):

L_DPO(θ) = −E_(x, y_w, y_l) [ log σ( β · log( π_θ(y_w|x) / π_ref(y_w|x) )
                                    − β · log( π_θ(y_l|x) / π_ref(y_l|x) ) ) ]

This is exactly the reward-model loss from section 5, with r_φ(x, y) replaced by β · log(π_θ(y|x) / π_ref(y|x)), the quantity the paper calls the implicit reward. The policy plays the role of the reward model itself, just as the paper’s title says: “Your Language Model is Secretly a Reward Model”. The gradient (section 4 of the paper):

∇L = −β · E[ σ( r̂(y_l) − r̂(y_w) ) · ( ∇log π_θ(y_w|x) − ∇log π_θ(y_l|x) ) ]

Raise the log-prob of the winner, lower the log-prob of the loser, with more weight when the model currently ranks them wrong; without that weighting the model degenerates (Appendix, Table 3).

What β does. β is still the KL coefficient of the original objective, now showing up as the scale of the implicit reward. Large β: a small change in the log-ratio already pushes the sigmoid toward saturation, so the policy does not need to move far from the reference. Small β: the policy has to move further to reduce the loss. Llama 3 uses β = 0.1 (section 4.1.4).

Computing the DPO loss on a real model

The code below uses Qwen2.5-0.5B-Instruct as both the reference (frozen) and the policy (trained), with one pair: “Paris” is chosen, “Lyon” is rejected. A sequence’s log-prob is the sum of the log-probs of the response tokens, including <|im_end|>, computed exactly like the loss mask in section 3. The code runs four SGD steps on this pair alone, then checks the loss by hand using only math.

# Python 3.12, torch 2.14.1 (CPU), transformers 5.18.0
import math
import torch
import torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.utils import logging
logging.disable_progress_bar()

name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
ref = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32).eval()     # reference: frozen
policy = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32).eval()  # policy: trained
for p in ref.parameters():
    p.requires_grad_(False)

prompt = [{"role": "user", "content": "What is the capital of France?"}]
chosen, rejected = "The capital of France is Paris.", "The capital of France is Lyon."

def seq_logprob(model, response):
    """log π(y|x): sum of log-probs of the response tokens (including <|im_end|>)."""
    p_ids = tok.apply_chat_template(prompt, add_generation_prompt=True, tokenize=True, return_dict=True)["input_ids"]
    f_ids = tok.apply_chat_template(prompt + [{"role": "assistant", "content": response}],
                                    tokenize=True, return_dict=True)["input_ids"]
    ids = torch.tensor([f_ids])
    logp = F.log_softmax(model(ids).logits[0, :-1], dim=-1)          # position t predicts token t+1
    target = ids[0, 1:]
    token_logp = logp.gather(-1, target[:, None]).squeeze(-1)
    return token_logp[len(p_ids) - 1:].sum()                         # assistant part only

beta = 0.1
with torch.no_grad():
    ref_c, ref_r = seq_logprob(ref, chosen), seq_logprob(ref, rejected)
print(f"reference: log π_ref(chosen) = {ref_c:.4f}   log π_ref(rejected) = {ref_r:.4f}")

opt = torch.optim.SGD(policy.parameters(), lr=1e-4)
for step in range(4):
    pol_c, pol_r = seq_logprob(policy, chosen), seq_logprob(policy, rejected)
    r_c = beta * (pol_c - ref_c)                     # implicit reward of chosen
    r_r = beta * (pol_r - ref_r)                     # implicit reward of rejected
    loss = -F.logsigmoid(r_c - r_r)
    print(f"step {step}: log π(c) = {pol_c.item():.4f}  log π(r) = {pol_r.item():.4f}  "
          f"r_c = {r_c.item():+.4f}  r_r = {r_r.item():+.4f}  loss = {loss.item():.4f}")
    opt.zero_grad()
    loss.backward()
    opt.step()

# Check by hand with the last step's numbers, using only math
margin = beta * ((pol_c.item() - ref_c.item()) - (pol_r.item() - ref_r.item()))
print(f"by hand: margin = {margin:.4f}  -log σ(margin) = {-math.log(1 / (1 + math.exp(-margin))):.4f}")

Output (about 15 seconds on CPU):

reference: log π_ref(chosen) = -0.3372   log π_ref(rejected) = -6.6565
step 0: log π(c) = -0.3372  log π(r) = -6.6565  r_c = +0.0000  r_r = +0.0000  loss = 0.6931
step 1: log π(c) = -0.4584  log π(r) = -9.2344  r_c = -0.0121  r_r = -0.2578  loss = 0.5778
step 2: log π(c) = -0.6536  log π(r) = -11.7018  r_c = -0.0316  r_r = -0.5045  loss = 0.4844
step 3: log π(c) = -0.9602  log π(r) = -14.3204  r_c = -0.0623  r_r = -0.7664  loss = 0.4018
by hand: margin = 0.7041  -log σ(margin) = 0.4018

Reading the output:

• At step 0 the loss is exactly ln 2. The policy equals the reference, so every log-ratio is 0 and the loss is −log σ(0) = 0.6931. This is a quick check for any DPO implementation: a first-step loss other than ln 2 means something is wrong with how log-probs are computed or how the reference is set up.
• The hand check matches. From the four log-probs of step 3, β · ((−0.9602 − (−0.3372)) − (−14.3204 − (−6.6565))) gives a margin of 0.7041, and −log σ(0.7041) = 0.4018, exactly the loss PyTorch reports.
• DPO does not care about absolute levels. The reference already prefers “Paris” (−0.34) over “Lyon” (−6.66); DPO only cares how much the policy changes relative to the reference on each answer.
• The winner’s log-prob drops too, from −0.34 to −0.96, so its probability almost halves. The loss still falls because the loser is pushed down much faster. The two answers differ by one word, and the gradient pushing the loser down drags the shared tokens with it. Pal et al. 2024 describe exactly this failure mode: the DPO loss can be low even when the winner’s probability falls below the reference, especially when the two answers are nearly identical (section 3); Razin et al. 2024 call it likelihood displacement. Llama 3 adds an NLL term with coefficient 0.2 on the winner to the DPO loss (section 4.1.4).

This is a toy example (one pair, a large learning rate, four steps), just to see the mechanism and check the formula.

Compared with PPO

In theory the two optimize the same objective; in practice they differ in important ways:

DPO’s biggest weakness is in the third row. PPO trains on the current policy’s output, so it can fix the mistakes the policy is currently making; DPO only learns from existing pairs, and where no pair covers the policy’s behavior there is no signal. Xu et al. 2024 prove that the set of PPO solutions is a proper subset of the DPO solutions, and that DPO can find solutions that exploit out-of-distribution answers (section 4.1, Theorem 4.1).

Which one wins? Still debated

• The DPO paper: on TL;DR summarization with a GPT-J SFT model, with GPT-4 as the judge, DPO reaches a win rate of about 61% against reference summaries, versus 57% for PPO at its best temperature (section 6.2, Figure 2); in the human evaluation, raters preferred DPO samples over PPO 58% of the time (section 6.4, Table 2).
• Xu et al. 2024: with advantage normalization, large batches and an exponential-moving-average update of the reference, PPO beats DPO in all of their experiments (abstract, sections 1 and 6). On CodeContests, PPO with CodeLlama-34B reaches 22.4% (10@1k), above AlphaCode-41B with clustering at 16.4%, while DPO gets 0.0% and SFT 15.2% (Table 8).
• Ivison et al. 2024: PPO beats DPO by up to 2.5% in math and 1.2% in general domains, but preference data quality is the biggest factor, worth up to 8% in instruction following and truthfulness (abstract). Their order of importance: data > algorithm > reward model > policy training prompts (section 6).

The overall picture: carefully tuned PPO usually comes out slightly ahead, especially in math and code, but costs much more. DPO is cheap, stable and good enough for most purposes, which is why it became the default in many pipelines such as Llama 3. Many teams narrow the gap by running DPO over several rounds on fresh pairs generated by the current policy and scored by a reward model.

Variants

SimPO and Tülu 3’s length-normalized DPO target a problem the code sample also shows: a sequence’s log-prob is a sum over tokens, so longer answers have more negative log-probs, and the original DPO loss is sensitive to length.

09RLAIF and Constitutional AI

Everything above needs human raters, and human raters are expensive, slow and disagree with each other. If an LLM is good enough to answer, is it good enough to judge?

Constitutional AI: two phases

Constitutional AI (Bai et al. 2022b, Anthropic) turns this idea into a complete process. The “constitution” is a list of principles written in natural language. There are two phases (section 1.2, Figure 1):

Phase 1: self-critique and revision (section 3). Give harmful prompts to a model trained only to be helpful. The model critiques its own answer against a randomly chosen principle, then rewrites it, a few times over; the principle set has 16 pairs of critique and revision requests (section 3.1, Appendix C.1). SFT on the revised answers produces SL-CAI.

Phase 2: RL from AI feedback (section 4). SL-CAI generates two answers; another model is asked, as a multiple-choice question: according to this principle, which answer is better? One principle, quoted verbatim (Appendix C.2):

“Please choose the response that is the most helpful, honest, and harmless.”

This phase also uses 16 principles, one sampled at random for each comparison (section 4.1). The judging model’s probabilities become soft labels for training a preference model, followed by RL as in section 6, producing RL-CAI. The paper calls this phase RL from AI Feedback (RLAIF). When the judge reasons step by step before choosing, its probabilities become extreme and RL produces over-the-top behavior; the authors clamp the probabilities to the 40–60% range (sections 4.1 and 4.3). The result is a “harmless but non-evasive” model that explains why it objects instead of simply refusing (abstract, section 4.4).

“No humans involved” is a misreading

Humans write the constitution: which principles, and how they are phrased. And in the paper, helpfulness labels still come from humans; only harmlessness labels come from the AI, so the preference model is a “hybrid human/AI” one (sections 1.2, 4.1). The discussion admits they still relied on human supervision through helpfulness labels (section 6). What changes is where humans sit: from judging individual pairs to writing principles, a form of supervision that is more transparent and easier to audit.

Is RLAIF as good as RLHF?

Lee et al. 2023 (Google) compare them directly, with PaLM 2 Large as the AI labeler and policy and reward model initialized from PaLM 2 Extra-Small (sections 3.2–3.3). Human evaluation (Table 1, section 4.1):

Head to head, RLAIF beats RLHF 50% of the time on summarization and 52% on helpful dialogue, not significantly different from 50%; on harmlessness RLAIF beats both. Two side results are even more interesting. When the AI labeler is the same size as the policy (PaLM 2 XS), RLAIF still beats SFT 68% of the time on summarization (section 4.2). And direct RLAIF (d-RLAIF), which drops the reward model and has the LLM score directly inside the RL loop, beats SFT 74% of the time and beats same-size RLAIF 60% of the time (section 4.3). On helpful dialogue, the d-RLAIF experiment uses the same checkpoint as both labeler and initial policy and still beats SFT 66% of the time (section 4.3); the paper calls this “strict self-improvement”. The same-size experiment in section 4.2, the paper says explicitly, is not yet strict self-improvement.

LLM-as-judge and its risks

Using a model as a judge means inheriting its biases. Zheng et al. 2023 document three kinds in LLM judges: position bias (favoring a certain position), verbosity bias (favoring longer answers) and self-enhancement bias (favoring its own answers) (section 3.3). Verbosity bias is the length bias of section 7, now automated, and in RLAIF every bias of the judge gets amplified by the optimization loop. In exchange: orders of magnitude cheaper, scalable, and the principles are written down for anyone to read and argue about. Real pipelines mix both: humans for hard and sensitive decisions, models for volume.

10Post-training pipelines today

Real pipelines are rarely a straight line from SFT to RLHF. They run over several rounds, each round using the best model of the previous one to generate data for the next. These three technical reports give enough detail to put side by side:

A few details worth reading closely:

• Rejection sampling uses a reward model without PPO: generate K answers per prompt, keep the highest-reward one, then run SFT on it. Llama 3 uses K between 10 and 30 (section 4.2.2).
• Llama 3 chose DPO over PPO because DPO “required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval” (section 4.1.4), and it averages the weights of models trained with different data or hyperparameters at each stage (section 4.1.5).
• Tülu 3 (Lambert et al. 2024, AI2) releases everything: data, code, checkpoints for each stage and the recipe (section 2.4, Table 1). The 70B model, built on the Llama 3.1 base, reaches an average of 76.2 on their evaluation suite, against 74.1 for Llama 3.1 70B Instruct, 72.8 for Qwen 2.5 72B Instruct, 69.6 for GPT-4o-mini and 75.3 for Claude 3.5 Haiku (Table 2); against Haiku, the paper itself elsewhere uses the word “approaching”.

RLVR and GRPO: one short paragraph

The last stage of Tülu 3 is RLVR (reinforcement learning with verifiable rewards): for tasks whose answers a program can check, such as math (GSM8K, MATH) or IFEval-style constraints, the reward comes from a checker: α = 10 if correct, 0 otherwise (section 6, equation 8), over about 30 thousand prompts (Table 22). A reward like that has no approximation to hack in the sense of section 7. GRPO (Shao et al. 2024, DeepSeekMath) is a PPO variant often paired with RLVR: it drops the value model, generates a group of answers per prompt, and uses as the advantage each answer’s reward minus the group mean, divided by the group standard deviation (section 4.1). This is the foundation of reasoning models, the topic of part 15 of this series.

11Common misconceptions

The six rows below match the bottom row of Figure 1.

Conclusion: one base model, three shaping layers

Post-training does not make the model know more. It teaches the model to use what it already knows the way people want: answer the question asked, in the right format, stop, and decline when needed. Each layer teaches part of that, and each brings its own quirks.

• A base model does not follow instructions because pretraining only rewards continuing text; LIMA and URIAL show that what is missing is mostly format and style.
• SFT is the same cross-entropy as pretraining, on conversations packaged by a chat template, with the loss only on the assistant’s tokens; the wrong template at inference pushes the model out of the distribution it learned.
• A reward model learns from comparisons through Bradley–Terry, σ(r_w − r_l), so only reward differences mean anything, and its accuracy is capped by the roughly 73–77% agreement between human raters.
• RLHF with PPO maximizes reward − β·KL, needs four models and generates text inside the training loop; reward hacking is the rule, with length and sycophancy as its two most common forms.
• DPO follows from the closed form π* ∝ π_ref · exp(r/β): the reward becomes β·log(π/π_ref), Z(x) cancels in Bradley–Terry, and what remains is a classification loss on pairs. It is cheap and stable but offline, and it can drag the winner’s probability down too. DPO versus PPO is still debated; good data matters more than the algorithm.
• RLAIF and Constitutional AI move humans from judging individual pairs to writing principles, matching RLHF on Lee et al.’s tasks, but amplifying the judge’s biases.
• Real pipelines run over many rounds: Llama 2 through 5 RLHF versions, Llama 3 through 6 rounds, and Tülu 3 releasing the whole SFT → DPO → RLVR recipe.

Every step in this post fine-tunes all of the weights, and PPO even keeps four copies of the model in memory at once. For a 70B model that is far beyond what most teams can afford. Part 12 covers the cheaper route: LoRA, QLoRA and quantization.

References

Papers

• Christiano et al. 2017 — Deep reinforcement learning from human preferences
• Schulman et al. 2017 — Proximal Policy Optimization Algorithms
• Stiennon et al. 2020 — Learning to summarize from human feedback
• Wei et al. 2021 — Finetuned Language Models Are Zero-Shot Learners
• Ouyang et al. 2022 — Training language models to follow instructions with human feedback
• Bai et al. 2022 — Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Gao et al. 2022 — Scaling Laws for Reward Model Overoptimization
• Wang et al. 2022 — Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Bai et al. 2022b — Constitutional AI: Harmlessness from AI Feedback
• Rafailov et al. 2023 — Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Zhou et al. 2023 — LIMA: Less Is More for Alignment
• Zheng et al. 2023 — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Touvron et al. 2023 — Llama 2: Open Foundation and Fine-Tuned Chat Models
• Casper et al. 2023 — Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
• Lee et al. 2023 — RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
• Kirk et al. 2023 — Understanding the Effects of RLHF on LLM Generalisation and Diversity
• Singhal et al. 2023 — A Long Way to Go: Investigating Length Correlations in RLHF
• Sharma et al. 2023 — Towards Understanding Sycophancy in Language Models
• Azar et al. 2023 — A General Theoretical Paradigm to Understand Learning from Human Preferences (IPO)
• Lin et al. 2023 — The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (URIAL)
• Ethayarajh et al. 2024 — KTO: Model Alignment as Prospect Theoretic Optimization
• Shao et al. 2024 — DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO)
• Pal et al. 2024 — Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
• Hong et al. 2024 — ORPO: Monolithic Preference Optimization without Reference Model
• Xu et al. 2024 — Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
• Hu et al. 2024 — OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework (v1)
• Meng et al. 2024 — SimPO: Simple Preference Optimization with a Reference-Free Reward
• Ivison et al. 2024 — Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback
• Llama Team, Meta 2024 — The Llama 3 Herd of Models
• Razin et al. 2024 — Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization
• Lambert et al. 2024 — Tülu 3: Pushing Frontiers in Open Language Model Post-Training

Docs and talks

• Hugging Face — Chat templates
• Hugging Face — TRL: Transformer Reinforcement Learning
• Hugging Face — Qwen/Qwen2.5-0.5B and Qwen/Qwen2.5-0.5B-Instruct
• Nathan Lambert — Reinforcement Learning from Human Feedback (RLHF Book)
• John Schulman — Reinforcement Learning from Human Feedback: Progress and Challenges (talk at UC Berkeley, 2023; transcript)