trainschool/06 · Fine-tuning: make it 100 % better at one task‹ prevnext ›

Lesson 6 · 35 min

Fine-tuning: make it 100 % better at one task

The course's title question, answered: take the Shakespeare model that cannot add, run the same loop on 20,000 sums, and watch accuracy go from 0 % to 100 % — with every knob, or with one knob in ten (LoRA).

In this lesson

The model you pretrained in Lesson 4 writes lines that scan like Shakespeare and cannot add two numbers. Ask it 47+38= and it answers ulle. This lesson makes it 100 % accurate at exactly that — two-digit sums, scored on sums it never trained on — without changing a line of the training loop. That is fine-tuning: the loop from Lessons 0 and 4, a new dataset, and a starting point that is not random.

The same loop, new data

From the base to the task1/5 · a pretrained box
0.0s
Lesson 0's box of knobs, already set by pretraining, meets a new dataset: 20,000 lines like 47+38=580 (the answer's digits are written back to front, so the first digit out is the easiest one). The loop is unchanged. The curves are the recorded eval points of a full fine-tune from the Lesson 4 base — the finetuneFull preset this lesson's Predict launches; the lab below defaults to LoRA — loss on sums from 11.4 to 1.0, accuracy on held-out sums from 0 % to 100 %.

Nothing about the algorithm changes. A batch of text goes in, the model guesses the next character at every position, the loss scores the guesses, backpropagation blames every knob, the optimiser nudges them. The only differences from Lesson 4 are the text — sums instead of plays — and the starting knobs — the pretrained checkpoint instead of random numbers. The lab passes the checkpoint through init_from: the fine-tune run inherits the base run's shape (4 layers, 128 wide; the shape fields disappear from the launcher once a source is picked), loads its weights, and starts stepping on the new data.

The sums are written as a+b= followed by the answer with its digits reversed and padded to three: 47+38=580 means 47 + 38 = 085. The reversal is Karpathy's trick from the original adder experiment: the first digit the model has to emit is the units digit, which depends only on the two units digits it can see, so the easy part of the task is learned first and the carries are learned on top of it. The training stream is 20,000 lines drawn from 9,000 of the 10,000 possible pairs; the other 1,000 pairs are held out, and accuracy is measured by prompting the model with a+b= from that held-out set, taking its three characters, and checking the reversed number.

Why the base matters

The base model has never seen a sum — tiny Shakespeare has 27 digits in 1.1 million characters and not a single + or = — and its first loss on the new text is 11.4, far above the 4.57 of a random model, because a Shakespeare model is confident that a 4 is followed by a letter. Fine-tuning has to undo that confidence before it can build anything. What the base does bring is everything below the surface: the character embedding table it shares with the new data (no resizing, because every dataset in this course uses the same 97-symbol alphabet), very likely attention heads that already copy and align characters at fixed offsets, and LayerNorms tuned to keep activations in range. Trained from random knobs on the same 4,000 steps of sums at the same rate, the same architecture gets its loss to the same floor but its accuracy did not settle in one run — 100 % at 3,250 and 3,750 steps, 95 % at 4,000; from the base it is 100 % from step 2,500 on and stays there.

Loss and accuracy answer different questions

Watch the two curves in the scene. Loss on sums falls from 11.4 to about 1.3 within the first 1,000 steps, and accuracy is still low: 11 % at step 1,000. Loss gives partial credit — it rewards the model for putting probability on the right digit at every position, and several characters in every line (+, =, the newline) are free. Accuracy gives none: a sum is right only when all three digits are. The carry is the last thing learned, and until it is, the model is nearly right on almost every line and exactly right on almost none. Then, between 1,500 and 2,000 steps, the last piece clicks into place and accuracy climbs a cliff: 23 % at 1,500, 93 % at 1,750, 98 % at 2,000, 100 % from 2,500. The floor the loss settles on, about 1.0, is not zero because four of the ten characters in every line — the digits of a and b — are random and cost ln 10 ≈ 2.3 each whatever the model does.

LoRA: train one knob in ten

LoRA: a low-rank detour1/5 · a weight matrix W
0.0s
One of the model's 128 × 128 weight matrices, drawn at 16 × 16. Freeze it; add two thin matrices beside it — B is 128 × 8 and starts at zero, A is 8 × 128 — so the layer computes W·x + B·A·x (the lab scales the detour by α/r = 2); train only B and A; then add B·A into W so the served model is one plain matrix again. In the lab's recipe, adapters go on the four projections of every block and the token embedding and LayerNorms stay trainable: 80,256 of 801,152 knobs, 10 %.

Full fine-tuning moves every one of the 801,152 knobs, and for every one the optimiser keeps a gradient and two running averages. LoRA (low-rank adaptation) keeps the big matrices frozen and adds, beside each one, a detour of two thin matrices: W·x becomes W·x + B·A·x, where A is r × 128 and B is 128 × r for a small rank r. The product B·A is a full-size 128 × 128 change to W, but it is built from 2 · 128 · r numbers instead of 128²: at rank 8 that is 2,048 knobs standing in for 16,384. B starts at zero, so at step 0 the detour contributes nothing and the model is exactly the base; gradients reach only A and B; and when training is done the lab adds B·A into W and saves a plain checkpoint, so nothing downstream — generation, evaluation, the DPO run in Lesson 7 — knows LoRA was used.

The lab's recipe is measured, not assumed. Adapters on the attention projections alone — the textbook minimum — cannot learn this task from this base: 0 % at ranks 4, 8 and 16. Addition needs the MLPs to change too, so the lab puts adapters on all four projections of every block (c_attn, c_proj, mlp.fc, mlp.proj), and, following the modules_to_save idiom from the PEFT library, trains the token embedding and every LayerNorm in full. The embedding matters because the digits' rows were nearly unused during pretraining and need to move a long way; LayerNorms are cheap (256 numbers each) and let every block rescale its activations for the new distribution. The count comes to 4 · 16 · 128 · r + 97 · 128 + 9 · 256: 80,256 at rank 8, 47,488 at rank 4.

What LoRA buys

toy · full vs LoRAtrainable 10.0 % · memory 2.7 MB vs 12.8 MB

The lab's recipe: rank-r adapters on the four projections of every block, plus the token embedding and every LayerNorm trained in full. Memory is 16 bytes per trainable knob (weights, gradient, two Adam averages, fp32 master) and 2 bytes per frozen one.

knobs that trainfull801,152 · every knobLoRA80,256 · 10.0 %optimiser memoryfull12.8 MB · 16 B × 801,152LoRA2.7 MB · 2 B × every knob + 14 B × trainable

What LoRA saves is memory, not compute. Every training step still runs the full forward pass and still backpropagates through every frozen matrix to reach the adapters below it; the only work skipped is the gradient with respect to the frozen weights themselves, about a third of a full step, and the lab's cost box prices both at the same 6·N·D. But the optimiser state — gradient, two Adam averages, fp32 master copy, 14 bytes per knob on top of the 2-byte weight — exists only for the trainable subset. On the lab's model that is 2.7 MB against 12.8 MB, which nobody cares about; the toy lets you scale the same formula up. At 12 layers and 512 wide (38 M knobs, a small production model's shape), rank 8 trains 2.3 % of the knobs and the optimiser needs 88 MB instead of 605 MB. For a 7-billion-parameter model, full fine-tuning needs 112 GB of weights and optimiser state before a single activation is stored — more than an 80 GB H100 holds — while LoRA at 0.5 % trainable needs about 14.5 GB and fits on a 24 GB card.

The second thing it buys is many variants of one base. A LoRA adapter is a few megabytes; a full fine-tune is a copy of the model. A product that needs a model per customer, per language or per task keeps one base and a shelf of adapters, and merges whichever one it is serving. The price in steps, on this task, is one eval interval: compare the two runs below.

Full against LoRA

Full fine-tune against LoRA1/4 · two runs, same task
0.0s
Two recorded runs from the same Lesson 4 base on the same sums. Full: every knob moves at a learning rate of 5e-4; accuracy crosses 90 % at step 1,750 and reads 100 % at 2,500. LoRA at rank 8: 80,256 knobs move at 3e-3; it crosses 90 % at 2,000 and reads 100 % at 2,500. Same accuracy, one knob in ten, one eval later.

Both runs land in the same place. The difference that matters is not the accuracy but the learning rate that got there, and it is the one thing about fine-tuning that is easy to get wrong. The LoRA run uses 3e-3, three times the pretraining peak; the full run uses 5e-4, half of it. Swap them and each goes wrong in its own way. A full fine-tune at 3e-3 is a coin toss: on one seed its loss drops to 1.2 in 1,500 steps and then sits there, ending at 16 % accuracy after 4,000 steps — it learned which characters are digits and lost the machinery it needed to add them — and on two other seeds it reaches 100 %, one of them only at step 3,750. 1e-3 lands in between (48 %, loss 1.11). Go the other way and 1e-4 is too slow (6 % at 4,000, loss still 1.30). Between them, 3e-4 reaches 100 % and 5e-4 reaches 100 % on two of three seeds with the third ending at 94 %, every seed's loss on the floor — the rate that gets there on every seed, which is why the preset uses it.

The asymmetry is the whole argument for adapters. With every knob free, a large step can overwrite what the base knew faster than the new data can teach it; the safe rate for a full fine-tune sits below the pretraining rate, which is why instruction-tuning a 7 B model runs at 1e-5 or 2e-5. A LoRA detour starts at zero and can only move the model along 80,256 directions out of 801,152, so the base survives a rate six times higher, and needs it, because the thin matrices have less leverage per step: rank 4 at 1e-3 is still short of 90 % after 6,000 steps. Given its rate, the detour is barely slower than the full run — its cliff comes one eval later, 90 % at 2,000 instead of 1,750, and both read 100 % at 2,500.

Fine-tune it yourself

Train it yourselffinetune · addition

loading runs…

pick a run to start from

no run yet

waiting for the first step…

Pick your Lesson 4 run as the source — or the Lesson 0 one; any finished Shakespeare pretrain of the default shape at the default rate works; Lesson 8 measures which bases learn the sums, and a lower Shakespeare loss bought with a hotter pretraining rate is not a better base. Every count in this lesson assumes that shape, 4 layers × 128 wide, which the picker shows as 4×128; a run you widened in Lesson 0's task has more knobs, more of them trainable, and different numbers from the ones quoted here. Leave max_iters at 4,500 the first time and watch the accuracy figure in the readout under the curve rather than the loss: the loss will look done by step 1,000 and the accuracy will not move until about 1,500, then jump. The run takes a little over two minutes on this CPU. The compute line under a finished run shows trainable next to params: about 80K next to 801K at the default rank. Switch method to full to move every knob instead, and lower the learning rate to 5e-4 when you do: a full fine-tune at LoRA's rate is not reliable here (measured at 3e-3 after 4,000 steps: 16 % on one seed, 100 % on two; at 5e-4: 100 %, 100 % and 94 %).

What it costs

The cost box under your finished run prices the fine-tune's compute exactly as Lesson 4 priced the pretrain: 6·N·D with N the model's 801,152 parameters — all of them, because the forward pass and the backward pass through the frozen matrices run regardless of how many knobs the optimiser updates — and D the tokens seen, 4,500 steps × 2,048 = 9.2 M. That is 4.4e13 FLOPs, three times the pretrain's 1.5e13, because the pretrain ran for only 1,500 steps. At this scale the ratio is upside down; at real scale it is the whole point of fine-tuning:

Pretraining Llama-2-7B — 2 T tokens6·N·D = 8.4e22 FLOPs
hardwaretimecost
RTX 4090 24GB40.4 y$106.1K–$194.4K
A100 80GB21.3 y$205.7K–$411.3K
H100 80GB SXM6.7 y$117.4K–$206.4K
B200 192GB3 y$90.7K–$155.6K

N = 7B params, D = 2T tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.

Fine-tuning the same model — 100 M tokens6·N·D = 4.2e18 FLOPs
hardwaretimecost
RTX 4090 24GB17.7 h$5.30–$9.72
A100 80GB9.3 h$10.28–$20.57
H100 80GB SXM2.9 h$5.87–$10.32
B200 192GB1.3 h$4.54–$7.78

N = 7B params, D = 100M tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.

A production instruction-tuning set is on the order of 100,000 examples of a thousand tokens each, so the fine-tune sees 20,000 times fewer tokens than the pretrain and costs 20,000 times less: about 0.005 % of the pretraining bill — on the H100 row, 3 GPU-hours against 59,000. That is why every capability you have seen added to a base model in an afternoon — a new format, a new tool, a new language — was a fine-tune, and why the base, trained once, is the asset.

Practice

quiz · Check · 1/4open

Why is fine-tuning so much cheaper than pretraining?

predict · Predict, then runopen

Fine-tune the whole model, from your base, for only 1,000 steps. The loss on sums will be well under 1.5 by then. Accuracy on held-out sums?

task · Taskopen

Reach 90 % accuracy with at most 60,000 trainable parameters, starting from a default-shape base (4×128 in the picker; the counts below assume it). The default rank 8 trains 80,256 — over budget — so the rank has to come down, and a thinner detour settles later.

goal: accuracy >= 0.9 within trainable_params <= 60000

Your attemptfinetune · addition

loading runs…

pick a run to start from

no run yet

waiting for the first step…
reflect · Why it worksopen
  1. Why does the same loop that learned Shakespeare learn addition when you only change the data?

  2. Why is the accuracy curve a cliff rather than a slope like the loss curve?

  3. For a real product built on an open base model, what would you fine-tune and what would you leave frozen?

levers · How to improveopen

Symptom: a LoRA fine-tune's accuracy stalls at 60 % after 5,000 steps. Order the levers you would pull, first to last.

  1. more steps
  2. lower the learning rate
  3. raise the LoRA rank or unfreeze more
  4. check the base run actually trained (val ≈ 1.8)

Next

The model now does one thing it could not do before, and you can measure exactly how well. Next: making it behave — alignment, where the training signal is a preference between two answers rather than the next character, and the loop changes for the first time in the course.