trainschool/07 · Alignment: RLHF & DPO‹ prevnext ›

Lesson 7 · 30 min

Alignment: RLHF & DPO

The training signal changes for the first time: not the next character, but a human's choice between two answers. Reward models, the RLHF pipeline, and DPO — which skips the reward model and trains the policy on the pairs directly, on a leash to a frozen copy of itself.

In this lesson

Every loss so far has scored one thing: the probability the model gave to the character that actually came next. That is the right signal for learning what text looks like and it is the wrong signal for learning what people want. A model that has read every answer ever written to a question has learned to write the typical answer, not the best one; the typical answer is sometimes wrong, sometimes rude, sometimes a refusal. Nobody has a file of "the best answer" to train on — but almost anyone can look at two answers and say which is better. Alignment is training on that judgement. This lesson builds the pipeline that turns a pile of A is better than B into a gradient, then runs it on the model you fine-tuned in Lesson 6.

A preference is a pair

One prompt, two answers, one pick1/4 · one prompt, two answers
0.0s
The lab's preference set: 4,000 sums, each with the right answer (its digits reversed, as in Lesson 6) and the same answer with one digit changed. A human — here, the dataset generator — marks which is which. The list on the right is drawn from the same format; the counter is the real size of the set.

A preference pair is a prompt, two candidate answers, and a bit saying which one a human preferred. It carries less information than a label — it does not say how much better, or whether either answer is any good — and that is exactly why it scales: rating an answer on a ten-point scale is slow and inconsistent between raters, but "this one or that one" takes a second and people mostly agree. The lab's preferences are generated rather than clicked, which keeps the lesson honest about what it is measuring: each pair is 47+38= with the chosen answer 580 (85, reversed) and the rejected answer 570 — the same string with one digit changed at random. The set has 4,000 pairs drawn from the 9,000 training sums of Lesson 6, and a held-out 128 pairs built from sums the model never trained on, which is where the default run’s margin is measured. New runs inherit the source checkpoint’s split, so changing the training seed does not move sums between train and validation. One quirk of the lab's pair format is worth knowing: each prompt is written bare, 47+38= at the very start of the context, where the fine-tune's training stream always has a newline first. Every accuracy figure in this lesson is measured on the fine-tune's own format — the lab's held-out prompt, at the start of a line — which is the format the sums are written in.

The main recorded curves below use seed 1337 and the same split as their source. Historical ranges across other seeds were measured before split membership was fixed; some of those exams overlap their ancestor’s training data. Keep them as historical observations, not a clean estimate of generalisation. The evaluation lab checks the full checkpoint ancestry and suppresses a comparison delta when that history is incompatible or missing.

RLHF: a reward model, then a policy

A second model learns to score1/4 · score every answer
0.0s
Stylised. A small reward model gives each answer a number; before training it has no preference (0.20 against 0.30). Two turns of the ordinary loop — score, loss, blame, turn — and the chosen answer scores 1.30 to the rejected one's −0.50. The loss is −log σ(gap); the gap between the two scores is the reward signal.

The classic pipeline, RLHF (reinforcement learning from human feedback), has two halves. First a reward model: a copy of the language model with its output layer swapped for a single number, trained on the pairs so that score(chosen) − score(rejected) is large. The loss is the same shape as every loss in this course: −log σ(r_chosen − r_rejected), the probability under a logistic model that the human's pick would win — the Bradley–Terry model of a two-horse race, and the loss starts at ln 2 ≈ 0.69 when the scorer is indifferent. Second, the policy: the language model itself is trained to produce answers the reward model scores highly, by sampling answers, scoring them, and pushing the knobs toward the high scorers with a reinforcement-learning algorithm (PPO, in the original recipe), while a penalty keeps its outputs from drifting too far from where they started. It works, and it is expensive: two extra models in memory (the reward model and the frozen original), a sampling loop inside the training loop, and a reward model that can be gamed — the policy is optimising a number, and any answer the scorer happens to over-rate is a hole the policy will find.

Train a two-knob reward model

toy · a two-knob reward modelw = [0.00, 0.00] · agrees with your picks —

pair 1 of 12 · click the answer you prefer

7+5 = ?

Each answer is two numbers: correct? and plausible? (the right number of digits). The reward is w · features, and every click is one step of logistic regression on −log σ(r_chosen − r_rejected) at lr 0.5: p = σ(w·Δf), w += lr·(1 − p)·Δf. The consistent human has the hidden weight w* = [2, 0.5].

the two knobsw correct0.00w plausible0.000

The reward model's whole job is to compress your clicks into a scoring rule. This one has two knobs — a weight on correct? and a weight on plausible? (the right number of digits) — and each click is one step of logistic regression on the pair you just judged: p = σ(w · (f_chosen − f_rejected)), then w += 0.5 · (1 − p) · (f_chosen − f_rejected). Label all twelve as a consistent human would (correctness first, plausibility second) and the weights land at 1.38 for correctness and 0.93 for plausibility, in the same order as the hidden preference that generated the clicks, and the model then agrees with all twelve of your picks. Now reset and be inconsistent — label eleven as before and prefer the wrong answer on the last pair — and the agreement drops to 11 of 12: a reward model can only be as coherent as the humans it learns from, and in a real pipeline those humans disagree with each other on a quarter to a third of the pairs.

DPO: skip the reward model

Direct preference optimisation1/5 · no reward model
0.0s
The bars are stylised probabilities of the two whole answers; the ghost behind each is a frozen copy of the model before DPO, the reference. Push the chosen answer up, the rejected one down, measure each move against the ghost, and the loss on this one pair is the real DPO loss at the lab's β = 0.1: from ln 2 to 0.556. The curve at the end is the recorded training loss of the run the lab launches by default: ln 2 to 0.32 in 300 steps.

DPO (direct preference optimisation) removes the reward model. Its insight is algebraic: for the RLHF objective with a KL penalty, the optimal policy and the reward are two views of one thing — the reward is β · log(π(answer) / π_ref(answer)), how much more likely the trained policy makes an answer than the frozen reference did, scaled by β. Substitute that into the Bradley–Terry loss and the reward model disappears:

loss = −log σ( β · [ (log π(chosen) − log π_ref(chosen)) − (log π(rejected) − log π_ref(rejected)) ] )

In words: raise the chosen answer's log-probability above where the reference had it, lower the rejected answer's below where the reference had it, and score the difference of those two moves with the same logistic loss the reward model used. No sampling, no second training loop, one gradient step per batch of pairs — the lab computes it in dpo.py with a masked sum over the answer tokens (the prompt is context, not target). At step 1 the policy is the reference, both moves are zero, and the loss is exactly ln 2; every step after that it falls. The reference is a copy of the model deep-copied and frozen before the first step. The β in front is the exchange rate between distance-from-the-reference and reward: at β = 0.1, a margin of 1.0 means the policy has moved the two answers' log-probabilities a total of 10 nats apart from where the reference had them. A larger β reports a larger margin for the same move and saturates the loss sooner.

The leash

Ten times the learning rate1/4 · too big a step
0.0s
The track is stylised; everything else is recorded. At lr 1e-3, ten times the default, the policy leaves the region where it still writes sums: its loss on ordinary sums goes from 0.98 to 13.0 and its real answer to 47+38= becomes a newline and two zeros. At the default lr 1e-4 the loss on sums ends at 1.67 and the model still writes digits — but its answer to the same prompt is now 5860, a well-formed wrong sum, and only 35 % of held-out sums are still right. Every prompt is sent as the lab's accuracy sends it, at the start of a line.

Nothing in the DPO loss says "stay a good language model". It rewards one thing: the gap between the two answers, relative to the reference. The reference and β are the leash — the loss is measured against where the model started and saturates once the gap is a few nats — but the leash is only as short as the steps are small. Take steps ten times too big and the model finds a shortcut: it can widen the gap by making both answers less likely, as long as the rejected one falls faster, and after 300 steps at lr 1e-3 it has stopped writing digits at all. Measured on the fine-tuned base: loss on sums from 0.98 to 13.0, accuracy from 100 % to 0 %, and — the tell — the held-out margin, which rose to 0.75 in the first 25 steps, collapsed to 0.03: the model no longer prefers either answer because it no longer produces either.

At the default rate the damage is quieter and more instructive. The held-out margin climbs steadily, 0 → 0.47 at 25 steps → 1.09 at 100 → 1.73 at 300, and the DPO loss falls from ln 2 to 0.32. But the model was already right: its held-out preference accuracy — the fraction of pairs where it puts more probability on the chosen answer than the rejected one — is 100 % at the first eval and stays there. DPO cannot teach it anything about sums; it can only sharpen a preference it already has, by pushing the rejected answers' probability down through the floor. And that is not free. Loss on sums creeps from 0.98 to 1.67 and accuracy on held-out sums falls from 100 % to 35 % — across three seeds, to 25–35 %, and from a Lesson 6 LoRA base to 54 %. The model still writes digits; two times in three they are the wrong ones. DPO does not raise accuracy on a task the model already does; here it takes two thirds of it away. Every extra step past the point where the margin is comfortably positive is drift bought with nothing: 600 steps take the loss on sums to 2.63 and the accuracy to 23 %. The Practice block is about stopping in time.

Run DPO yourself

Train it yourselfdpo · preferences

loading runs…

pick a run to start from

no run yet

waiting for the first step…

Pick your Lesson 6 fine-tuned run as the source — the full one or the LoRA one; both are plain checkpoints, and any finished run of the default shape will be accepted. The launcher plots the reward margin rather than the loss: the noisy line is the training batch's margin, the dots are the held-out margin every 25 steps. It should leave zero within the first few steps and be near 1.7 at the end, with the summary line showing margin and pref acc 100 %; the run takes about ten seconds. Then read the two numbers the plot does not show: val loss is the loss on ordinary sums, which should have crept up from about 0.98, and accuracy is the fraction of held-out sums the model still gets right. The margin says alignment worked; the other two say what it cost. The lab's DPO learning rate defaults to 1e-4 when the field is left alone (a tenth of the pretraining default) — lower it and the drift shrinks: at 1e-5 for 300 steps the margin still reaches 0.97, the loss on sums stays at 1.07 and 81 % of the sums survive.

What it costs

The compute line under a finished run prices the DPO step the way every other run is priced, 6·N·D, with D the answer-and-prompt tokens the policy actually processed: 32 pairs × 2 answers × 10 characters × 300 steps = 192,000 tokens, 9.2e11 FLOPs. That is one sixteenth of the Lesson 4 pretrain (1.5e13) and one forty-third of the Lesson 6 fine-tune (3.9e13), because the sequences are ten characters instead of 64 and the run is short. The line undercounts by a third: every step also runs the frozen reference forward over the same tokens (2·N per token, no backward), which the lab's counter does not include. At real scale the ratio is the same story as fine-tuning:

Pretraining Llama-2-7B — 2 T tokens6·N·D = 8.4e22 FLOPs
hardwaretimecost
RTX 4090 24GB40.4 y$106.1K–$194.4K
A100 80GB21.3 y$205.7K–$411.3K
H100 80GB SXM6.7 y$117.4K–$206.4K
B200 192GB3 y$90.7K–$155.6K

N = 7B params, D = 2T tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.

DPO on 100,000 pairs of 1,000 tokens — 200 M tokens6·N·D = 8.4e18 FLOPs
hardwaretimecost
RTX 4090 24GB35.4 h$10.61–$19.44
A100 80GB18.7 h$20.57–$41.13
H100 80GB SXM5.9 h$11.74–$20.64
B200 192GB2.6 h$9.07–$15.56

N = 7B params, D = 200M tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.

A production preference set is tens of thousands of pairs — Zephyr-7B was aligned on about 60,000 — and each pair is two answers, so a 100,000-pair DPO run sees 200 million tokens: 10,000 times fewer than the pretrain, about 6 GPU-hours on the H100 row against 59,000. The reward model of a full RLHF pipeline costs the same again to train, and the PPO loop that follows samples fresh answers every step, so RLHF runs several times the DPO bill; either way alignment is a rounding error next to pretraining. That is why every open base model has a dozen aligned variants and why the argument about alignment is never about compute.

Practice

quiz · Check · 1/4open

What is a preference pair?

predict · Predict, then runopen

Run DPO at ten times the learning rate (1e-3). What happens to the model's loss on ordinary sums?

task · Taskopen

Reach a held-out reward margin of at least 0.6 without letting the loss on ordinary sums rise above 1.2 (it starts near 0.98). The default — 300 steps — gets the margin and loses the model: measured, its loss on sums ends between 1.46 and 1.85 across three seeds and two bases, with 25–54 % of the held-out sums still right.

goal: reward_margin >= 0.6 within val_loss <= 1.2

Your attemptdpo · preferences

loading runs…

pick a run to start from

no run yet

waiting for the first step…
reflect · Why it worksopen
  1. Why train on pairs rather than asking people to score each answer?

  2. Why does a frozen reference stop the model collapsing, and why does it not stop it completely?

  3. When would you still use RLHF with a reward model instead of DPO?

levers · How to improveopen

Symptom: the reward margin climbs but the model's sums are now wrong. Order the levers you would pull, first to last.

  1. raise β (a tighter leash)
  2. more preference pairs
  3. lower the learning rate
  4. fewer DPO steps

Next

The model now prefers right answers to nearly-right ones, and you have watched the cost of that preference appear in a number the loss did not show. Next: evaluation — how to know whether a run made the model better, on what, with how much confidence, and why the eval you did not run is the one that bites.