Every loss so far has scored one thing: the probability the model gave to the character that actually came next. That is the right signal for learning what text looks like and it is the wrong signal for learning what people want. A model that has read every answer ever written to a question has learned to write the typical answer, not the best one; the typical answer is sometimes wrong, sometimes rude, sometimes a refusal. Nobody has a file of "the best answer" to train on — but almost anyone can look at two answers and say which is better. Alignment is training on that judgement. This lesson builds the pipeline that turns a pile of A is better than B into a gradient, then runs it on the model you fine-tuned in Lesson 6.
A preference is a pair
A preference pair is a prompt, two candidate answers, and a bit saying which one a human preferred. It carries less information than a label — it does not say how much better, or whether either answer is any good — and that is exactly why it scales: rating an answer on a ten-point scale is slow and inconsistent between raters, but "this one or that one" takes a second and people mostly agree. The lab's preferences are generated rather than clicked, which keeps the lesson honest about what it is measuring: each pair is 47+38= with the chosen answer 580 (85, reversed) and the rejected answer 570 — the same string with one digit changed at random. The set has 4,000 pairs drawn from the 9,000 training sums of Lesson 6, and a held-out 128 pairs built from sums the model never trained on, which is where the default run’s margin is measured. New runs inherit the source checkpoint’s split, so changing the training seed does not move sums between train and validation. One quirk of the lab's pair format is worth knowing: each prompt is written bare, 47+38= at the very start of the context, where the fine-tune's training stream always has a newline first. Every accuracy figure in this lesson is measured on the fine-tune's own format — the lab's held-out prompt, at the start of a line — which is the format the sums are written in.
The main recorded curves below use seed 1337 and the same split as their source. Historical ranges across other seeds were measured before split membership was fixed; some of those exams overlap their ancestor’s training data. Keep them as historical observations, not a clean estimate of generalisation. The evaluation lab checks the full checkpoint ancestry and suppresses a comparison delta when that history is incompatible or missing.
RLHF: a reward model, then a policy
The classic pipeline, RLHF (reinforcement learning from human feedback), has two halves. First a reward model: a copy of the language model with its output layer swapped for a single number, trained on the pairs so that score(chosen) − score(rejected) is large. The loss is the same shape as every loss in this course: −log σ(r_chosen − r_rejected), the probability under a logistic model that the human's pick would win — the Bradley–Terry model of a two-horse race, and the loss starts at ln 2 ≈ 0.69 when the scorer is indifferent. Second, the policy: the language model itself is trained to produce answers the reward model scores highly, by sampling answers, scoring them, and pushing the knobs toward the high scorers with a reinforcement-learning algorithm (PPO, in the original recipe), while a penalty keeps its outputs from drifting too far from where they started. It works, and it is expensive: two extra models in memory (the reward model and the frozen original), a sampling loop inside the training loop, and a reward model that can be gamed — the policy is optimising a number, and any answer the scorer happens to over-rate is a hole the policy will find.
Train a two-knob reward model
pair 1 of 12 · click the answer you prefer
7+5 = ?
Each answer is two numbers: correct? and plausible? (the right number of digits). The reward is w · features, and every click is one step of logistic regression on −log σ(r_chosen − r_rejected) at lr 0.5: p = σ(w·Δf), w += lr·(1 − p)·Δf. The consistent human has the hidden weight w* = [2, 0.5].
The reward model's whole job is to compress your clicks into a scoring rule. This one has two knobs — a weight on correct? and a weight on plausible? (the right number of digits) — and each click is one step of logistic regression on the pair you just judged: p = σ(w · (f_chosen − f_rejected)), then w += 0.5 · (1 − p) · (f_chosen − f_rejected). Label all twelve as a consistent human would (correctness first, plausibility second) and the weights land at 1.38 for correctness and 0.93 for plausibility, in the same order as the hidden preference that generated the clicks, and the model then agrees with all twelve of your picks. Now reset and be inconsistent — label eleven as before and prefer the wrong answer on the last pair — and the agreement drops to 11 of 12: a reward model can only be as coherent as the humans it learns from, and in a real pipeline those humans disagree with each other on a quarter to a third of the pairs.
DPO: skip the reward model
DPO (direct preference optimisation) removes the reward model. Its insight is algebraic: for the RLHF objective with a KL penalty, the optimal policy and the reward are two views of one thing — the reward is β · log(π(answer) / π_ref(answer)), how much more likely the trained policy makes an answer than the frozen reference did, scaled by β. Substitute that into the Bradley–Terry loss and the reward model disappears:
loss = −log σ( β · [ (log π(chosen) − log π_ref(chosen)) − (log π(rejected) − log π_ref(rejected)) ] )
In words: raise the chosen answer's log-probability above where the reference had it, lower the rejected answer's below where the reference had it, and score the difference of those two moves with the same logistic loss the reward model used. No sampling, no second training loop, one gradient step per batch of pairs — the lab computes it in dpo.py with a masked sum over the answer tokens (the prompt is context, not target). At step 1 the policy is the reference, both moves are zero, and the loss is exactly ln 2; every step after that it falls. The reference is a copy of the model deep-copied and frozen before the first step. The β in front is the exchange rate between distance-from-the-reference and reward: at β = 0.1, a margin of 1.0 means the policy has moved the two answers' log-probabilities a total of 10 nats apart from where the reference had them. A larger β reports a larger margin for the same move and saturates the loss sooner.
The leash
Nothing in the DPO loss says "stay a good language model". It rewards one thing: the gap between the two answers, relative to the reference. The reference and β are the leash — the loss is measured against where the model started and saturates once the gap is a few nats — but the leash is only as short as the steps are small. Take steps ten times too big and the model finds a shortcut: it can widen the gap by making both answers less likely, as long as the rejected one falls faster, and after 300 steps at lr 1e-3 it has stopped writing digits at all. Measured on the fine-tuned base: loss on sums from 0.98 to 13.0, accuracy from 100 % to 0 %, and — the tell — the held-out margin, which rose to 0.75 in the first 25 steps, collapsed to 0.03: the model no longer prefers either answer because it no longer produces either.
At the default rate the damage is quieter and more instructive. The held-out margin climbs steadily, 0 → 0.47 at 25 steps → 1.09 at 100 → 1.73 at 300, and the DPO loss falls from ln 2 to 0.32. But the model was already right: its held-out preference accuracy — the fraction of pairs where it puts more probability on the chosen answer than the rejected one — is 100 % at the first eval and stays there. DPO cannot teach it anything about sums; it can only sharpen a preference it already has, by pushing the rejected answers' probability down through the floor. And that is not free. Loss on sums creeps from 0.98 to 1.67 and accuracy on held-out sums falls from 100 % to 35 % — across three seeds, to 25–35 %, and from a Lesson 6 LoRA base to 54 %. The model still writes digits; two times in three they are the wrong ones. DPO does not raise accuracy on a task the model already does; here it takes two thirds of it away. Every extra step past the point where the margin is comfortably positive is drift bought with nothing: 600 steps take the loss on sums to 2.63 and the accuracy to 23 %. The Practice block is about stopping in time.
Run DPO yourself
Pick your Lesson 6 fine-tuned run as the source — the full one or the LoRA one; both are plain checkpoints, and any finished run of the default shape will be accepted. The launcher plots the reward margin rather than the loss: the noisy line is the training batch's margin, the dots are the held-out margin every 25 steps. It should leave zero within the first few steps and be near 1.7 at the end, with the summary line showing margin and pref acc 100 %; the run takes about ten seconds. Then read the two numbers the plot does not show: val loss is the loss on ordinary sums, which should have crept up from about 0.98, and accuracy is the fraction of held-out sums the model still gets right. The margin says alignment worked; the other two say what it cost. The lab's DPO learning rate defaults to 1e-4 when the field is left alone (a tenth of the pretraining default) — lower it and the drift shrinks: at 1e-5 for 300 steps the margin still reaches 0.97, the loss on sums stays at 1.07 and 81 % of the sums survive.
What it costs
The compute line under a finished run prices the DPO step the way every other run is priced, 6·N·D, with D the answer-and-prompt tokens the policy actually processed: 32 pairs × 2 answers × 10 characters × 300 steps = 192,000 tokens, 9.2e11 FLOPs. That is one sixteenth of the Lesson 4 pretrain (1.5e13) and one forty-third of the Lesson 6 fine-tune (3.9e13), because the sequences are ten characters instead of 64 and the run is short. The line undercounts by a third: every step also runs the frozen reference forward over the same tokens (2·N per token, no backward), which the lab's counter does not include. At real scale the ratio is the same story as fine-tuning:
| hardware | time | cost |
|---|---|---|
| RTX 4090 24GB | 40.4 y | $106.1K–$194.4K |
| A100 80GB | 21.3 y | $205.7K–$411.3K |
| H100 80GB SXM | 6.7 y | $117.4K–$206.4K |
| B200 192GB | 3 y | $90.7K–$155.6K |
N = 7B params, D = 2T tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.
| hardware | time | cost |
|---|---|---|
| RTX 4090 24GB | 35.4 h | $10.61–$19.44 |
| A100 80GB | 18.7 h | $20.57–$41.13 |
| H100 80GB SXM | 5.9 h | $11.74–$20.64 |
| B200 192GB | 2.6 h | $9.07–$15.56 |
N = 7B params, D = 200M tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.
A production preference set is tens of thousands of pairs — Zephyr-7B was aligned on about 60,000 — and each pair is two answers, so a 100,000-pair DPO run sees 200 million tokens: 10,000 times fewer than the pretrain, about 6 GPU-hours on the H100 row against 59,000. The reward model of a full RLHF pipeline costs the same again to train, and the PPO loop that follows samples fresh answers every step, so RLHF runs several times the DPO bill; either way alignment is a rounding error next to pretraining. That is why every open base model has a dozen aligned variants and why the argument about alignment is never about compute.
Practice
What is a preference pair?
Run DPO at ten times the learning rate (1e-3). What happens to the model's loss on ordinary sums?
Reach a held-out reward margin of at least 0.6 without letting the loss on ordinary sums rise above 1.2 (it starts near 0.98). The default — 300 steps — gets the margin and loses the model: measured, its loss on sums ends between 1.46 and 1.85 across three seeds and two bases, with 25–54 % of the held-out sums still right.
goal: reward_margin >= 0.6 within val_loss <= 1.2
Why train on pairs rather than asking people to score each answer?
Why does a frozen reference stop the model collapsing, and why does it not stop it completely?
When would you still use RLHF with a reward model instead of DPO?
Symptom: the reward margin climbs but the model's sums are now wrong. Order the levers you would pull, first to last.
- raise β (a tighter leash)
- more preference pairs
- lower the learning rate
- fewer DPO steps
Next
The model now prefers right answers to nearly-right ones, and you have watched the cost of that preference appear in a number the loss did not show. Next: evaluation — how to know whether a run made the model better, on what, with how much confidence, and why the eval you did not run is the one that bites.