Your tokenizer has turned text into ids. A transformer turns those ids into a prediction by moving a row of vectors through the same two operations again and again: attention mixes information between positions, and an MLP computes on each position separately. The residual stream carries the result through the whole stack.
From ids to vectors
An id is only an address. The embedding table has one row for each of the lab's 97 possible characters and n_embd numbers in each row. Looking up id 85 means taking row 85; no arithmetic relation between 85 and 73 matters. Training changes the numbers in those rows until characters used in similar ways acquire useful representations.
The same character can mean something different at a different place, so the model adds a position embedding. Token 85 at position 0 gets token[85] + position[0]; token 85 at position 20 gets the same token row plus a different position row. The sum is the first state in the residual stream.
Attention: a weighted lookup
Attention answers one narrow question for every position: which earlier tokens matter for this one? Each input vector is projected three ways. Its query describes what it is looking for, its key describes what it can match, and its value carries the information to retrieve. A dot product scores every query–key pair. Softmax turns one row of scores into non-negative attention weights that sum to one, and those weights take a weighted average of the value vectors.
During next-token training the answer must not leak in from the right. The causal mask sets every score for a future position to an impossible value before softmax, so the weight above the diagonal is zero. At sat, the head may use the, cat, and sat; it cannot inspect the words that follow.
n_head says how many of these lookups happen in parallel. With n_embd=128 and n_head=4, each head works in 32 dimensions, learns its own attention pattern, and the four outputs are joined back into one 128-number vector. More heads divide the same width rather than making the model wider by themselves.
Open your model's attention
loading runs…
First, pick your finished Lesson 0 run and try the cat sat on the. Choose a layer and compare its heads. Rows are the tokens doing the looking, columns are the tokens they may use, and brighter cells carry more weight. The blank upper triangle is the causal mask. A trained head may focus on the previous character, spaces, repeated letters, or a broader pattern; the explorer is the lab for this lesson because it runs the actual checkpoint and returns its actual attention matrices.
The residual stream
The residual stream is the sequence of vectors flowing through the model. Attention reads the stream and adds its result back with x = x + attention(x). Then the MLP does the same: x = x + mlp(x). Because both are additions, a later block can preserve an earlier feature, strengthen it, or cancel it. Information has a direct path from the embeddings to the final prediction instead of being rewritten from scratch at every layer.
Attention is the between-token operation. The MLP is per-token computation: the same small network is applied independently at every position. In the lab it expands each n_embd vector to four times that width, applies a non-linearity, and projects it back. Attention gathers relevant context; the MLP transforms what each position has gathered.
Width, depth, and the knob count
Depth is n_layer: how many times the model can gather context and compute on it. Width is n_embd: how much state every token carries. A wider vector makes nearly every matrix wider in both directions, so the dominant term grows with n_embd²; adding a layer repeats that collection of matrices once. That is why width changes the parameter count quadratically while depth changes it roughly linearly.
For this lab, the reported count excludes the position table. With context length 64 and width 128, that table adds 8,192 learned parameters: the default model trains 809,344 parameters in total. The course uses the reported count consistently in its approximate cost calculations. That reported count is vocab·n + L·(12n² + 4n) + 2n. The quick 2×64 model has 105,152 parameters. The Lesson 0 model, 4×128, has 801,152. An 8×256 model has 6,324,992, about 6.3 million. n_head does not appear in the formula: splitting a 128-wide projection into four heads changes how attention is organised, while the projection matrices remain 128 by 128.
| hardware | time | cost |
|---|---|---|
| RTX 4090 24GB | 1.7 s | $0.0001–$0.0003 |
| A100 80GB | 0.9 s | $0.0003–$0.0006 |
| H100 80GB SXM | 0.3 s | $0.0002–$0.0003 |
| B200 192GB | 0.1 s | $0.0001–$0.0002 |
N = 6.4M params, D = 3M tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.
The token budget here is the same 1,500 steps as Lesson 0: about 3 million tokens from 1,500 × 32 × 64. The model is about eight times larger, so the estimated training compute is about eight times larger too. Lesson 5 will turn that relationship into the rule 6·N·D: six units of forward-and-backward work for each of N parameters over each of D training tokens.
Practice
What are attention weights?
Halve the depth (1 layer instead of 2), same 300 steps. Validation loss?
Beat validation loss 2.25 within 300 steps by changing the architecture only. Keep the Shakespeare data and training recipe fixed.
goal: val_loss <= 2.25 within step <= 300
Why is attention a lookup rather than a fixed rule?
Why can stacking blocks do more than one very wide block?
What does the reported count of 801,152 include?
Symptom: loss is fine but the model can't use context from 40 characters back. Order the levers you would pull, first to last.
- raise block_size
- add heads
- add layers
- raise the learning rate
Next
You now know the machine's shape. Next comes the material it learns from: how a dataset is split, batched, cleaned, and allowed to shape every one of those knobs.