trainschool/03 · Data: what you feed it‹ prevnext ›

Lesson 3 · 25 min

Data: what you feed it

A model can only learn from the stream it receives. Build that stream, remove repeats, hold out an honest test, and give each parameter enough tokens to learn from.

In this lesson

A training run begins long before the first gradient step. Someone chooses the pages, code, transcripts, logs, or measurements; decides what counts as a duplicate; and reserves a slice the optimiser will never see. Those choices define what the model can learn and whether its score can be trusted.

From a corpus to a stream

A corpus becomes tokens1/4 · a pile of text
0.0s
Text arrives as separate documents or shards. Exact repeats are removed, then the kept text is tokenised and concatenated into one stream of ids from which training windows are sampled.

Real corpora mix public web pages, licensed books and news, code, reference works, transcripts, and data made for a particular task. Each source brings its own coverage and its own gaps. A model trained mostly on English web pages will be much better at their language and conventions than at a rare dialect, an unpublished event, or an image it never received.

The files are cleaned and converted into the tokenizer's integers. Training does not preserve their page layout: it samples fixed-length windows from the resulting streams. A source that appears ten times therefore contributes about ten times as many updates unless the pipeline removes the copies.

Drop the copies

toy · dedupkept 8 of 12

Exact compares every character. Normalised lowercases, removes punctuation, and collapses whitespace before comparing.

  1. 1The moon rose over the hill.
  2. 2A fox crossed the empty road.
  3. 3The moon rose over the hill.
  4. 4Rain tapped softly on the glass.
  5. 5A train arrived before dawn.
  6. 6A fox crossed the empty road.
  7. 7Birds gathered beside the river.
  8. 8Rain tapped softly on the glass.
  9. 9Lanterns glowed along the quay.
  10. 10A train arrived before dawn.
  11. 11the moon rose over the hill
  12. 12BIRDS GATHERED BESIDE THE RIVER!

The toy starts with 12 lines. Exact matching keeps 8: four later lines are byte-for-byte copies. Normalised matching keeps 6 because it also catches two lines that differ only in case or punctuation. Production systems compare larger blocks and use similarity methods as well, but the reason is the same. Copies waste compute, encourage memorisation, and can cause evaluation leakage if a benchmark answer occurs in both training and validation data. Normalisation is useful but risky: aggressive rules can collapse text whose punctuation or case carries meaning.

Reserve the evidence first

Train on one side, judge on the other1/4 · the stream
0.0s
The lab cuts the token stream once at 90%. Optimiser updates sample only from the left side; validation loss is measured on the held-out last tenth.

The lab makes its 90/10 split before shuffling training windows. That order matters. If you create overlapping windows first, shuffle them, and then split, almost identical passages can land on both sides. Validation would reward the model for recognising training text instead of predicting genuinely held-out text.

The training side changes the weights. The validation side never does; it is evidence about whether those changes generalise. A falling training loss with a rising validation loss means the model is learning particulars of the training set faster than patterns that transfer. More steps will usually widen that gap.

How much data is enough?

Twenty tokens for each parameter1/4 · a model of N knobs
0.0s
The Chinchilla scaling result puts a compute-optimal run near 20 training tokens per parameter. For the lab's 801,152 parameters that is about 16 million tokens; Lesson 0 presented 3.07 million.

The Chinchilla scaling experiments found a useful rule of thumb for compute-optimal language-model training: budget about 20 tokens for every parameter. It balances spending more compute on a larger model against showing a smaller model more data. It is an empirical guide, not a law: data quality, architecture, repeated epochs, and the job the model must do all move the best ratio.

For the lab's 801,152 parameters, 20:1 recommends about 16 million tokens. Lesson 0 presented exactly 3,072,000 tokens: 1,500 steps × 32 sequences × 64 tokens. That is about 19% of the scaling-law target and roughly 2.8 passes over the 1.1-million-character Shakespeare corpus. The course model is under-trained on purpose so a run finishes in seconds and leaves room for you to see the curve move. Lesson 5 prices the "right" amount at larger scales.

Train on your own text

Train it yourselfpretrain · custom

paste some training text

no run yet

waiting for the first step…

Paste text and watch the compute line update. The token count there is how many token positions the optimiser processes, batch size × context length × steps; it can be much larger than the pasted corpus because the lab samples the same stream repeatedly. The last tenth of what you paste is held out before any windows are sampled, so put enough representative text on both sides of that cut.

Custom run · 1,000 steps6·N·D = 9.84e12 FLOPs
hardwaretimecost
RTX 4090 24GB0.1 s$0.0000–$0.0000
A100 80GB0.1 s$0.0000–$0.0000
H100 80GB SXM0 s$0.0000–$0.0000
B200 192GB0 s$0.0000–$0.0000

N = 801.2K params, D = 2M tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.

This box prices one 1,000-step lab run: about 2.05 million token presentations. Building a 15-trillion-token web crawl is a different cost. Fetching, storing, filtering, deduplicating, documenting provenance, and licensing that data all happen before accelerator compute begins. Repeating a cheap low-quality corpus does not turn it into 15 trillion useful tokens; at real scale, the data pipeline can cost as much organisational effort as the training cluster.

Practice

quiz · Check · 1/4open

Why remove duplicates before training?

predict · Predict, then runopen

Train the full lab model for 600 steps on 1,202 characters: a repeated training sentence followed by a different held-out tail. What do the curves do?

task · Taskopen

With your own text, keep the final val − train gap under 0.3 without exceeding 1,000 steps. The automatic gate can compare only one metric, so it checks validation loss ≤ 2.2; inspect the two final losses to confirm the gap itself. Measured: 60,000 Shakespeare characters at 1,000 steps reached train 1.57 and val 1.70, a 0.13 gap.

goal: val_loss <= 2.2 within step <= 1000

Your attemptpretrain · custom

paste some training text

no run yet

waiting for the first step…
reflect · Why it worksopen
  1. Why can a duplicate-heavy corpus make a model look more confident than it should?

  2. Why is 20 tokens per parameter a rule of thumb rather than a law?

  3. What does an epoch mean when data is scarce?

levers · How to improveopen

Symptom: validation loss starts rising at step 400 while training loss keeps falling. Order the levers you would pull, first to last.

  1. shrink the model
  2. add more text
  3. stop at the best checkpoint
  4. dedup the corpus