You have trained the same model four times now: from scratch on Shakespeare, properly in Lesson 4, on sums in Lesson 6, and on preferences in Lesson 7. Every run ended with a number, and the numbers disagreed about what had happened. The Lesson 6 fine-tune's loss looked finished at step 1,000 while its accuracy was 11 %; the Lesson 7 DPO run's margin climbed the whole way while its accuracy on sums fell by two thirds. And the number this course reported for accuracy was, until this lesson, measured by an eval that asked the question wrong — a real defect, kept here as the worked example. This lesson is about the number you can trust: where it must come from, what it can and cannot say, how many examples it takes to mean anything, and why "better" is only ever a comparison.
The exam is the part it never saw
A model that has seen the answer is not being tested; it is being asked to recall. So before the first step of training, part of the data is cut off and hidden, and every number that claims to measure the model is computed on that part alone. The lab does this the same way for every dataset in the course: for the sums, it lists all 10,000 pairs of two-digit numbers, shuffles them with a fixed seed, and holds out the last 1,000. The 20,000 training lines are drawn from the other 9,000 pairs — with repeats, so the model sees most of them a couple of times — and two things are measured on the held-out 1,000: the loss on sums, over 2,000 lines drawn from them, and the accuracy, by prompting the model with a+b= at the start of a line — exactly as every training line is written — for the first 200 held-out pairs, taking its three characters, and checking the reversed number. The readout under every fine-tune run is that 200-sum score; the /eval endpoint this lesson's lab calls can score up to all 1,000.
The split only works if it is clean. Two-digit sums are a closed world — 10,000 pairs, each one either in the training set or out of it — so the cut is exact. A text corpus is not: the same paragraph appears on a thousand web pages with different footers, the same song lyric in a hundred forum posts, and a random 10 % cut leaves copies of the "held-out" text in the training set. That is why Lesson 3 put deduplication before the split. An eval set that leaked into training measures memory, not skill, and the number it produces is worse than no number, because it is confident. The lab's Shakespeare split is the crude version — the last 10 % of the play text, cut once, never shuffled — which is safe only because a play does not repeat its scenes.
Loss and the metric answer different questions
The loss is what training minimises: the average, over every character in every line, of −log p(the right character). It gives partial credit. A model that puts 40 % on the right digit and 60 % on the digit next to it scores 0.92 on that character instead of the 0 a certain model would score, and it collects that credit at every position, including the +, the = and the newline, which are free once the format is learned. That is why the loss on sums falls smoothly from 11.4 to 1.3 in the first thousand steps and then creeps toward its floor of about 0.98 — the floor is not zero because four characters in every line, the digits of the two numbers being added, are random and cost ln 10 ≈ 2.3 each whatever the model does.
The metric is what you actually want: a sum is right or it is not. It gives no partial credit, so it stays low while the model is nearly right on every line and exactly right on few, and it jumps when the last piece — the carry — clicks into place. The loss can fall from 11.4 to 1.31 while the accuracy goes from 0 % to 11 %, and then the accuracy can go from 23 % to 93 % while the loss moves from 1.10 to 1.03: seven tenths of the task arrive in seven hundredths of a nat. A run whose loss stops a fifth of a unit short of the floor — 1.20 instead of 0.98, which is what a 2,500-step schedule does from this base — has 7 % of the sums right. They are two answers to two different questions about the same knobs. Optimise the loss, because it is smooth and the gradient can follow it; decide on the metric, because it is the thing you promised.
One number the eval reports deserves its own sentence: perplexity is e raised to the loss, the number of equally likely characters the model is effectively choosing between at each position. The fine-tune's loss of 0.983 on sums is a perplexity of 2.67; the Lesson 4 base's 1.80 on Shakespeare is 6.06; a model that has learned nothing about a 97-character alphabet sits at 97. It is the same information as the loss on a scale people can picture, and it inherits every limitation: a perplexity of 2.67 on sums says nothing about whether a single sum is right.
Generalise or memorise
Two runs can report the same training loss and be different models. What separates them is the gap between the loss on the training data and the loss on the held-out data. A model that is learning the structure of the task lowers both; a model that has started to memorise lowers one. The tell is not the size of the gap but its direction over time: when the held-out loss stops falling and turns up while the training loss keeps going, every further step is making the model worse at the job while making the number you are watching better. Lesson 4's overfitting scene showed this on a small text corpus; the fix there was to stop at the best held-out checkpoint, and the deeper fix is more, and more varied, data.
The sums make the distinction concrete. The training set is 9,000 of the 10,000 possible pairs, so a model could pass the training set by memorising nine thousand answers and fail the thousand it never saw. The recorded fine-tune did not: its loss on training lines ended at 0.981 and its loss on held-out lines at 0.983, a gap of 0.002, and it answered every one of the 1,000 held-out sums. That is what learning the rule looks like from the outside — the held-out set is indistinguishable from the training set — and it is the only way to tell it apart from a lookup table.
How many sums is enough?
An evaluation is a sample, and a score on n examples carries an uncertainty that shrinks like 1/√n. The lab's readout, 200 sums, is enough to tell a broken model from a working one and not enough to tell two working ones apart: a measured 90 % on 200 sums means the truth is somewhere between 85.1 % and 93.4 %. The intervals in the scene and the toy are Wilson score intervals, which unlike the textbook ± 1.96√(p(1−p)/n) stay inside 0–100 % and are not empty when the score is exactly 100 %. That last case matters here, because it is the case the lab produces: 100 % on 200 sums means the true accuracy on two-digit sums in this format is at least 98.1 %, and 100 % on all 1,000 means at least 99.6 %. To claim 99 % or better from an all-correct eval you need every one of 381 sums right.
Set the true accuracy and the eval size and read the interval; then sample an eval and see where a real measurement lands. On ten sums the same 90 % model measures anywhere from 70 % to 100 % across a dozen draws; on a thousand it stays within 3 points of the truth. Lesson 7's DPO run is the worked example: its readout said 34.5 % on 200 sums, an interval of 28.3–41.3 %, and scored on all 1,000 it measured 37.7 % — inside the interval, and a reminder that the 200-sum number is a measurement with a width, not a fact.
The width is the smaller of the two things a small eval hides. The larger is what the eval set does not contain. The fine-tune's 100 % on 1,000 held-out sums is strong evidence about two-digit sums written as a+b= with both numbers zero-padded, because that is what the held-out set is drawn from. It is no evidence at all about anything else, and the model is happy to demonstrate: prompted with 147+238= it writes 240, with 7+8= (no padding) it writes 461, and with 47 + 38 = (spaces) it writes 580 — right, and right on four more spaced sums. None of those prompts are in the training distribution, so none of them were in the eval, so the 100 % never claimed them: the model generalises to spaces and not to a third digit, and the score could not have told you which. An eval measures the distribution it was drawn from, and the first question to ask of any score is what that distribution was.
The eval that lied
The harness's own prompt format is part of that distribution, and this course shipped the mistake. Until this lesson was written, the lab's accuracy prompted the model with the sum at the very start of the context — 47+38= as the whole input, nothing before the first digit. No training line ever sits there: the training stream is …\n47+38=580\n12+09=120\n…, so every sum the model has ever seen begins after a newline, and a sum at position 0 is an ambiguous context it never trained on. The metric was measuring how each checkpoint happened to handle that ambiguity, not whether it could add. Here is what it did to real checkpoints — the same weights, the same 200 held-out sums, scored by the shipped prompt and by the fixed one:
| run | loss on sums | bare prompt | start of a line |
|---|---|---|---|
| full fine-tune from a 300-step base, 4,000 steps | 0.99 | 2 % | 100 % |
| full fine-tune at 3e-3, seed 1 | 0.99 | 9.5 % | 100 % |
| full fine-tune at 3e-3, seed 1337 | 1.18 | 10.5 % | 16 % |
| full fine-tune at 1e-3 | 1.11 | 16.5 % | 47.5 % |
| full fine-tune at 3e-4 | 0.99 | 92 % | 100 % |
| the same recipe from random knobs | 0.98 | 58 % | 95 % |
| the preset run (Lesson 6's curve) | 0.98 | 100 % | 100 % |
| LoRA r=8 (the lab's default) | 0.98 | 100 % | 100 % |
| a 2,500-step schedule at 5e-4 | 1.20 | 9.5 % | 7 % |
| the Lesson 7 DPO run | 1.68 | 71.5 % | 34.5 % |
Read the loss column against the last one. Every run whose loss on sums reached the 0.98–0.99 floor adds — 95–100 % of the held-out sums when asked the way it was trained — and the shipped prompt scored those same runs anywhere from 2 % to 100 %. Two full fine-tunes were reported as failures at 2 % and 9.5 % that were in fact perfect; a learning rate was written off as "does not learn" on the strength of one of them; the DPO run, whose preference pairs are written bare at position 0 — the eval's prompt exactly, and not the fine-tune's format — scored 71.5 % on the prompt it had trained on and 34.5 % on the one the sums are written in. The runs the metric got roughly right are the ones that had not learned the task at all. How it was caught: the loss. A loss on sums at the floor with accuracy at 2 % is not a model that cannot add; the loss is averaged over the next-character predictions of 2,000 held-out lines and cannot sit at 0.99 unless the answer digits are being predicted almost perfectly. When the two numbers disagree that badly, one of them is measuring something else, and the first suspect is always the prompt. The fix was one character: the eval now prefixes every sum with the newline the training lines carry, and every number in Lessons 6, 7 and 8 was re-measured with it. Which is why the metric, not the loss, is the number this lesson trusts — and why the first thing to check about any metric is how, exactly, the harness asks, by printing one eval prompt and one training window side by side.
Better than what?
loading runs…
loading runs…
Both models take the same exam: run B’s held-out data. Choose a fine-tuned addition run as B to measure whether your base model learned the task.
"Better" is a comparison, and a comparison is only honest when both checkpoints sit the same exam. Pick your Lesson 4 base as run A and your Lesson 6 fine-tune as run B and evaluate both. The toy scores both checkpoints on B’s held-out data, so both now get an accuracy on sums, even though the base was trained on Shakespeare. The course's recorded base scored 0 % on this task; the fine-tune reached 100 %. These are measured examples, not a promise about every run.
Now pick the fine-tune as A and your Lesson 7 DPO run as B. The same comparison shows whether alignment preserved the task: in the recorded course runs, accuracy fell from 100 % to about 35 %. The result labels the actual number of sums scored separately from the windows sampled for loss. Models with different context widths can see different loss windows, so the toy suppresses the loss delta when their benchmark identities differ. New fine-tunes and DPO runs inherit their source's held-out split; historical experiments retain their original split. The comparison also checks every checkpoint ancestor: an exam is not held out if an earlier model in the chain already trained on it. Incompatible or missing ancestry suppresses the delta. To make a defensible comparison, keep the same task, split, prompt format, and scoring procedure.
Two rules fall out of this. Compare checkpoints on the same held-out set, scored the same way, or you have two numbers and no comparison. And keep the previous checkpoint's number next to the new one: a score of 35 % is a fine score for a model that could not add, and a poor one for a model that used to be right every time. The number by itself does not say which.
What it costs
The eval this lesson runs is cheap. The held-out set is 1,000 lines of ten characters — 10,000 tokens of text — and scoring it is forward passes only, 2·N per token instead of training's 6·N. The lab's /eval on all 1,000 sums does more than the minimum: it re-estimates the loss over 50 batches of training text and 50 of held-out text (204,800 tokens), then generates three characters for each of the 1,000 prompts without a cache, re-reading the growing seven-character prompt each time (24,000 token-positions). That is about 229,000 tokens through the model, 3.7e11 FLOPs, one percent of the fine-tune's 3.9e13, and under a second on this CPU when nothing else is training (25 s with a fine-tune running beside it). At real scale the ratio is the same story:
| hardware | time | cost |
|---|---|---|
| RTX 4090 24GB | 40.4 y | $106.1K–$194.4K |
| A100 80GB | 21.3 y | $205.7K–$411.3K |
| H100 80GB SXM | 6.7 y | $117.4K–$206.4K |
| B200 192GB | 3 y | $90.7K–$155.6K |
N = 7B params, D = 2T tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.
| hardware | time | cost |
|---|---|---|
| RTX 4090 24GB | 35.4 min | $0.18–$0.32 |
| A100 80GB | 18.7 min | $0.34–$0.69 |
| H100 80GB SXM | 5.9 min | $0.20–$0.34 |
| B200 192GB | 2.6 min | $0.15–$0.26 |
N = 7B params, D = 10M tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.
A serious evaluation suite for a language model — a few thousand questions from each of a dozen benchmarks, with their few-shot prompts — is on the order of ten million tokens per checkpoint. Forward-only, that is 1.4e17 FLOPs: about six minutes of one H100, 600,000 times less than the pretrain. Evaluating a hundred checkpoints on ten such suites is about 100 GPU-hours against the 59,000 of the pretrain, which is why evaluation is never the compute problem and always the attention problem: the expensive part is deciding what to measure, writing the examples nobody has trained on, and checking that they did not leak.
Practice
During your Lesson 6 fine-tune the loss on sums looked nearly done by step 1,000 — about 1.3 on the full fine-tune, against a floor of 0.98 — while the accuracy was still about 10 %. Why can they disagree like that?
Fine-tune the whole model from your base for 1,500 steps. The loss on sums will be well below 1.5 by then. What will the accuracy on held-out sums be?
This launcher runs the full-fine-tune preset on a 2,500-step schedule instead of its 4,000. Run it as it is first and read both numbers under the curve. Measured from the course's base: the loss on sums ends at 1.20 — a fifth of a unit above the 0.98 floor the 100 % model reaches, close enough to look finished on a curve — and the accuracy at 7 %. That fifth of a unit is the carry. Change what you must to reach at least 80 % on the lab's accuracy, in at most 3,000 steps.
goal: accuracy >= 0.8 within step <= 3000
Why is a single number never enough to say a model is good?
If the loss is what training optimises, why not decide on the loss too?
How many held-out sums would you want before claiming the model is 99 % accurate?
Symptom: the model scores 100 % on the eval set but fails in use. Order the levers you would pull, first to last.
- make the eval set harder and larger
- retrain with more data
- check the eval set leaked into training
- evaluate on the real input distribution
Next
You can now say, with a number and its width, whether a run made the model better at the task, and you know which of the numbers on the screen to believe. Next: serving the model — what one generated token costs in compute and memory, why the KV cache grows, and why the bill for using a model can dwarf the bill for training it.