trainschool/04 · The pretraining loop‹ prevnext ›

Lesson 4 · 35 min

The pretraining loop

One step of training, phase by phase — forward, loss, backward, update — then the three dials that shape a run: the learning-rate schedule, the batch size, and when to stop. Ends with the run the rest of the course builds on.

In this lesson

Lesson 0 showed the loop from the outside: guess, measure, turn the knobs, repeat. This lesson opens one turn of it and names the four phases, then looks at the three things you actually get to choose when you pretrain — how big each step is over time, how much text goes into each step, and when to stop. At the end you train the model that Lessons 6 to 9 fine-tune, align, evaluate and serve. Keep that run.

One step, four phases

One training step1/6 · a batch goes in
0.0s
A batch of 32 sequences, 64 characters each, goes through four blocks of knobs and comes out as a guess for the next character at every position. The loss scores the guess; the backward pass hands every knob its share of the blame; the update nudges all 800,000 of them at once. The loss numbers are the default run's: 4.57 at step 1, about 1.81 at step 1,500.

Every step of every run in this course does exactly these four things, in this order:

  1. Forward. A batch of text goes in — 32 sequences of 64 characters, 2,048 tokens — and the model produces, at every one of the 2,048 positions, a probability for each of the 97 possible next characters. Nothing is learned here; it is a big evaluation of the current knobs.
  2. Loss. Each guess is scored against the character that actually came next: −log p(truth), averaged over the batch. At step 1 the knobs are random, every character gets about 1/97, and the loss is 4.57.
  3. Backward. Backpropagation walks the computation in reverse — output layer, block 4, block 3, block 2, block 1, embeddings — and computes, for every knob, the slope of the loss with respect to that knob. It does not compute new values; it computes which way each knob would push the loss and how hard. The whole backward pass costs about twice the forward pass, which is where the 6 in 6·N·D comes from: 2 for forward, 4 for backward, per parameter per token.
  4. Update. Every knob moves a little against its slope: knob −= lr × slope. The step is the same rule for all 800,000 knobs, each with its own slope. Then the next batch is drawn and the loop goes round again — 1,500 times in the default run.

The lab's optimiser is AdamW, the one nearly every language model uses. It keeps two running averages per knob — of the gradient (momentum, so a consistent push accumulates) and of the gradient squared — and divides one by the square root of the other, so every knob moves at about the same speed whatever the scale of its own gradient. The "W" is weight decay: each step also pulls every matrix knob slightly toward zero (the lab uses 0.1), a mild, constant pressure against fitting the batch's particulars. One more safety rail sits between backward and update: gradient clipping rescales the whole gradient whenever its length exceeds 1.0, so a single wild batch cannot throw the knobs — you will test how much it matters in the practice block.

The step size over time

The learning-rate schedule1/4 · warmup
0.0s
The exact curve the lab runs, from its own formula: a linear warmup from 0 to 1e-3 over the first 100 steps, then a cosine decay to 1e-4 at step 1,500. Halfway through the decay, at step 800, the rate is 5.5e-4.

The learning rate is not one number; it is a curve. The lab's has two parts, and both exist for a reason you can measure.

Warmup ramps the rate up from zero over the first 100 steps. At step 1 the knobs are random and AdamW's running averages are empty, so the first gradients are huge and badly estimated; taking full-size steps on them throws the knobs somewhere bad that the run then has to recover from. Warmup lets the averages fill in and the loss drop from 4.57 to about 2.6 before the steps reach full size. It matters more the bigger the peak: at the default peak of 1e-3, dropping the warmup costs little (1.95 against 1.93 at 1,000 steps); at a peak of 3e-3 it costs a lot (2.09 against 1.79).

Cosine decay brings the rate down smoothly to the floor, 1e-4 — a tenth of the default peak — by the last step. Big steps early cover ground; small steps late stop the knobs bouncing around the bottom of the valley, so the run settles on a minimum instead of orbiting it. On the lab's model at a peak of 3e-3 over 1,000 steps, cosine ends at 1.79 and a constant rate ends at 1.84 — the difference is the settling.

The peak itself is the number you tune. The default, 1e-3, is deliberately conservative so that everything in the course trains stably; this tiny model can take more. Try that in the toy first, then in the practice block for real.

Draw your own schedule

toy · the schedulelr at last step 1.00e-4 · loss at step 1,500 1.81

The lr curve is the lab's own formula. The loss curve is synthetic: calibrated so the default schedule ends at the measured 1.81, with noise proportional to the current lr.

01,50001.0e-3learning ratewarmup ends01,5001.54.57training stepsloss (stylised)

The top curve is the lab's lr_at formula, ported line for line, so what you draw here is what a run would use. The loss curve under it is stylised — it knows only how much learning rate has accumulated and how much the current rate jitters the estimate — and is calibrated so the default schedule ends at the measured 1.81. Push the peak to 1e-4 and the curve is still at 3.46 after 1,500 steps: too small a step, too slow. Push it to 1e-2 and it is below 2 by step 300, but the noise band is about ten times wider (±0.2 instead of ±0.02), which is the trade the real runs make too. Switch decay off and watch the jitter stay at full width to the end; that width is what cosine removes. The toy's trend line ignores the settling, so with decay off its final number is lower; the lab's is not (1.84 vs 1.79) — the jitter band is the part the toy gets right.

How much text per step

Batch size and noise1/3 · batch of 4
0.0s
Stylised: two runs on the same trend. A batch of 4 sequences (256 tokens) gives a jittery estimate of the loss and of the gradient; a batch of 32 (2,048 tokens) is far steadier. Same trend, less noise — for eight times the tokens in every step.

The loss the model minimises on any one step is measured on that step's batch, and a batch is a random sample of the text. A small batch is a noisy sample: its loss jumps from step to step, and so does its gradient, so the knobs get pushed a little sideways on every step. A bigger batch averages over more text, which steadies both — at the cost of more tokens, and so more compute, per step.

Measured on the lab, same 1,500 steps: batch 4 ends at val 2.27 in 15 s; batch 32 (the default) at 1.82 in 46 s; and batch 64 for 1,000 steps ends at 1.84 in 62 s where batch 32 for 1,000 steps ends at 1.93 in 31 s. Bigger batches buy a lower loss per step, never per token for free: the batch-64 run saw twice the text of the batch-32 run. On a GPU the trade is different, because a bigger batch keeps the hardware busier and so costs less time per token — one reason production runs use batches of millions of tokens.

When to stop

Overfitting1/4 · train and val together
0.0s
Stylised: a small dataset trained for too long. Red is loss on the training text, blue on held-out text. They fall together, part at step 800, and then training loss keeps falling while validation rises: the model is memorising the training text instead of learning the language. The checkpoint to keep is the one at the bottom of the blue curve.

Training loss on its own can always be made smaller: keep going and the model starts to memorise the exact text. The honest signal is the validation loss — measured every 100 steps on text the model never trains on — and specifically the gap between the two. While they fall together, the model is learning something general. When validation turns up while training keeps falling, every further step is memorisation, and the right move is to stop and keep the checkpoint with the lowest validation loss.

Your default run does not get there: on a megabyte of Shakespeare, after 1,500 steps, training loss is about 1.66 and validation about 1.82, and validation is still falling — a gap has opened but there is nothing to stop for yet. Lesson 1's task showed the other case: 6,000 characters of Shakespeare ended at 3.97 after 1,000 steps and 4.26 after 1,500, because the extra steps only memorised harder.

The sample texts the lab prints every 250 steps tell the same story from the outside. On the default run, step 250 is letters in about the right proportions arranged into word-shaped nonsense, with a capitalised speaker-name line; step 500 has real short words — you, the, and, well — in a stream; step 1,000 has speaker headings and mostly real words; step 1,500 produces lines that scan and occasionally mean something. None of it is Shakespeare. All of it is the loss falling.

Now do it properly

Train it yourselfpretrain · shakespeare

no run yet

waiting for the first step…

This is the run the rest of the course builds on — leave it finished. Lesson 6 fine-tunes this exact checkpoint on addition, Lesson 7 aligns it, Lesson 8 evaluates it against the fine-tuned one, Lesson 9 serves it. Run it once on the defaults (4 layers, 128 wide, 1,500 steps, peak 1e-3, warmup 100, cosine, batch 32) and it will reach a validation loss of about 1.8 in under a minute; then experiment as much as you like in the practice block, where the runs are separate. The cost box under the finished run prices this exact amount of compute on real GPUs.

Practice

quiz · Check · 1/4open

What does the backward pass compute?

predict · Predict, then runopen

The quick preset (2 layers, 64 wide, 300 steps) ends at a validation loss of 2.48 with the default learning rate of 1e-3. Run it at ten times that, 1e-2, with gradient clipping on at 1.0. What happens?

task · Taskopen

You have 1,000 steps, not 1,500. The default schedule reaches a validation loss of 1.93 in that budget. Reach 1.88 or lower by changing the schedule or the batch — the step count is fixed and the model's shape is not on the table.

goal: val_loss <= 1.88 within step <= 1000

Your attemptpretrain · shakespeare

no run yet

waiting for the first step…
reflect · Why it worksopen
  1. Why is the update written per knob — knob −= lr × slope — rather than as one move of the whole model?

  2. Why does decaying the learning rate help at the end of a run?

  3. A run's loss curve is flat from the first step. What do you look at first?

levers · How to improveopen

Symptom: loss falls for 200 steps, then explodes to NaN. Order the levers you would pull, first to last.

  1. reduce the batch size
  2. check that gradient clipping is on
  3. lower the peak learning rate
  4. add or lengthen the warmup

Next

You now have a pretrained model and a run you can point at. Next: what it would cost to do this at the scale of a real model — FLOPs, GPU-hours and dollars, from the same 6·N·D that priced your run.