A training run begins long before the first gradient step. Someone chooses the pages, code, transcripts, logs, or measurements; decides what counts as a duplicate; and reserves a slice the optimiser will never see. Those choices define what the model can learn and whether its score can be trusted.
From a corpus to a stream
Real corpora mix public web pages, licensed books and news, code, reference works, transcripts, and data made for a particular task. Each source brings its own coverage and its own gaps. A model trained mostly on English web pages will be much better at their language and conventions than at a rare dialect, an unpublished event, or an image it never received.
The files are cleaned and converted into the tokenizer's integers. Training does not preserve their page layout: it samples fixed-length windows from the resulting streams. A source that appears ten times therefore contributes about ten times as many updates unless the pipeline removes the copies.
Drop the copies
Exact compares every character. Normalised lowercases, removes punctuation, and collapses whitespace before comparing.
- 1The moon rose over the hill.
- 2A fox crossed the empty road.
- 3The moon rose over the hill.
- 4Rain tapped softly on the glass.
- 5A train arrived before dawn.
- 6A fox crossed the empty road.
- 7Birds gathered beside the river.
- 8Rain tapped softly on the glass.
- 9Lanterns glowed along the quay.
- 10A train arrived before dawn.
- 11the moon rose over the hill
- 12BIRDS GATHERED BESIDE THE RIVER!
The toy starts with 12 lines. Exact matching keeps 8: four later lines are byte-for-byte copies. Normalised matching keeps 6 because it also catches two lines that differ only in case or punctuation. Production systems compare larger blocks and use similarity methods as well, but the reason is the same. Copies waste compute, encourage memorisation, and can cause evaluation leakage if a benchmark answer occurs in both training and validation data. Normalisation is useful but risky: aggressive rules can collapse text whose punctuation or case carries meaning.
Reserve the evidence first
The lab makes its 90/10 split before shuffling training windows. That order matters. If you create overlapping windows first, shuffle them, and then split, almost identical passages can land on both sides. Validation would reward the model for recognising training text instead of predicting genuinely held-out text.
The training side changes the weights. The validation side never does; it is evidence about whether those changes generalise. A falling training loss with a rising validation loss means the model is learning particulars of the training set faster than patterns that transfer. More steps will usually widen that gap.
How much data is enough?
The Chinchilla scaling experiments found a useful rule of thumb for compute-optimal language-model training: budget about 20 tokens for every parameter. It balances spending more compute on a larger model against showing a smaller model more data. It is an empirical guide, not a law: data quality, architecture, repeated epochs, and the job the model must do all move the best ratio.
For the lab's 801,152 parameters, 20:1 recommends about 16 million tokens. Lesson 0 presented exactly 3,072,000 tokens: 1,500 steps × 32 sequences × 64 tokens. That is about 19% of the scaling-law target and roughly 2.8 passes over the 1.1-million-character Shakespeare corpus. The course model is under-trained on purpose so a run finishes in seconds and leaves room for you to see the curve move. Lesson 5 prices the "right" amount at larger scales.
Train on your own text
Paste text and watch the compute line update. The token count there is how many token positions the optimiser processes, batch size × context length × steps; it can be much larger than the pasted corpus because the lab samples the same stream repeatedly. The last tenth of what you paste is held out before any windows are sampled, so put enough representative text on both sides of that cut.
| hardware | time | cost |
|---|---|---|
| RTX 4090 24GB | 0.1 s | $0.0000–$0.0000 |
| A100 80GB | 0.1 s | $0.0000–$0.0000 |
| H100 80GB SXM | 0 s | $0.0000–$0.0000 |
| B200 192GB | 0 s | $0.0000–$0.0000 |
N = 801.2K params, D = 2M tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.
This box prices one 1,000-step lab run: about 2.05 million token presentations. Building a 15-trillion-token web crawl is a different cost. Fetching, storing, filtering, deduplicating, documenting provenance, and licensing that data all happen before accelerator compute begins. Repeating a cheap low-quality corpus does not turn it into 15 trillion useful tokens; at real scale, the data pipeline can cost as much organisational effort as the training cluster.
Practice
Why remove duplicates before training?
Train the full lab model for 600 steps on 1,202 characters: a repeated training sentence followed by a different held-out tail. What do the curves do?
With your own text, keep the final val − train gap under 0.3 without exceeding 1,000 steps. The automatic gate can compare only one metric, so it checks validation loss ≤ 2.2; inspect the two final losses to confirm the gap itself. Measured: 60,000 Shakespeare characters at 1,000 steps reached train 1.57 and val 1.70, a 0.13 gap.
goal: val_loss <= 2.2 within step <= 1000
Why can a duplicate-heavy corpus make a model look more confident than it should?
Why is 20 tokens per parameter a rule of thumb rather than a law?
What does an epoch mean when data is scarce?
Symptom: validation loss starts rising at step 400 while training loss keeps falling. Order the levers you would pull, first to last.
- shrink the model
- add more text
- stop at the best checkpoint
- dedup the corpus