trainschool/02 · Inside the transformer‹ prevnext ›

Lesson 2 · 30 min

Inside the transformer

How token ids become vectors, attention gathers context, residual paths carry it, and repeated blocks turn width and depth into a model.

In this lesson

Your tokenizer has turned text into ids. A transformer turns those ids into a prediction by moving a row of vectors through the same two operations again and again: attention mixes information between positions, and an MLP computes on each position separately. The residual stream carries the result through the whole stack.

From ids to vectors

Embedding lookup1/4 · ids in
0.0s
The ids 85, 73 and 70 select three rows from a 97 × 8 embedding table. A second learned table supplies positions 0, 1 and 2. Adding the two gives each token both an identity and a place in the sequence.

An id is only an address. The embedding table has one row for each of the lab's 97 possible characters and n_embd numbers in each row. Looking up id 85 means taking row 85; no arithmetic relation between 85 and 73 matters. Training changes the numbers in those rows until characters used in similar ways acquire useful representations.

The same character can mean something different at a different place, so the model adds a position embedding. Token 85 at position 0 gets token[85] + position[0]; token 85 at position 20 gets the same token row plus a different position row. The sum is the first state in the residual stream.

Attention: a weighted lookup

One attention head1/5 · queries, keys, values
0.0s
For each token, a query scores every key. The causal mask first excludes future scores. Softmax then turns the allowed scores into weights whose row sums to 1, and the row's weights mix the value vectors into one output.

Attention answers one narrow question for every position: which earlier tokens matter for this one? Each input vector is projected three ways. Its query describes what it is looking for, its key describes what it can match, and its value carries the information to retrieve. A dot product scores every query–key pair. Softmax turns one row of scores into non-negative attention weights that sum to one, and those weights take a weighted average of the value vectors.

During next-token training the answer must not leak in from the right. The causal mask sets every score for a future position to an impossible value before softmax, so the weight above the diagonal is zero. At sat, the head may use the, cat, and sat; it cannot inspect the words that follow.

n_head says how many of these lookups happen in parallel. With n_embd=128 and n_head=4, each head works in 32 dimensions, learns its own attention pattern, and the four outputs are joined back into one 128-number vector. More heads divide the same width rather than making the model wider by themselves.

Open your model's attention

toy · what each head looks at

loading runs…

First, pick your finished Lesson 0 run and try the cat sat on the. Choose a layer and compare its heads. Rows are the tokens doing the looking, columns are the tokens they may use, and brighter cells carry more weight. The blank upper triangle is the causal mask. A trained head may focus on the previous character, spaces, repeated letters, or a broader pattern; the explorer is the lab for this lesson because it runs the actual checkpoint and returns its actual attention matrices.

The residual stream

The residual stream1/4 · the stream
0.0s
A block does not replace its input. Attention adds a contextual update to the stream, then the MLP adds a per-token update. The unchanged path keeps earlier information available while each block contributes something new.

The residual stream is the sequence of vectors flowing through the model. Attention reads the stream and adds its result back with x = x + attention(x). Then the MLP does the same: x = x + mlp(x). Because both are additions, a later block can preserve an earlier feature, strengthen it, or cancel it. Information has a direct path from the embeddings to the final prediction instead of being rewritten from scratch at every layer.

Attention is the between-token operation. The MLP is per-token computation: the same small network is applied independently at every position. In the lab it expands each n_embd vector to four times that width, applies a non-linearity, and projects it back. Attention gathers relevant context; the MLP transforms what each position has gathered.

Width, depth, and the knob count

Stacking transformer blocks1/4 · one block
0.0s
The lab repeats the same attention-plus-MLP shape four times. At width 128, embeddings, four blocks and the final normalisation contain 801,152 reported parameters, excluding the separately learned position table.

Depth is n_layer: how many times the model can gather context and compute on it. Width is n_embd: how much state every token carries. A wider vector makes nearly every matrix wider in both directions, so the dominant term grows with n_embd²; adding a layer repeats that collection of matrices once. That is why width changes the parameter count quadratically while depth changes it roughly linearly.

For this lab, the reported count excludes the position table. With context length 64 and width 128, that table adds 8,192 learned parameters: the default model trains 809,344 parameters in total. The course uses the reported count consistently in its approximate cost calculations. That reported count is vocab·n + L·(12n² + 4n) + 2n. The quick 2×64 model has 105,152 parameters. The Lesson 0 model, 4×128, has 801,152. An 8×256 model has 6,324,992, about 6.3 million. n_head does not appear in the formula: splitting a 128-wide projection into four heads changes how attention is organised, while the projection matrices remain 128 by 128.

the same 1,500 steps on the 8×256 model6·N·D = 1.15e14 FLOPs
hardwaretimecost
RTX 4090 24GB1.7 s$0.0001–$0.0003
A100 80GB0.9 s$0.0003–$0.0006
H100 80GB SXM0.3 s$0.0002–$0.0003
B200 192GB0.1 s$0.0001–$0.0002

N = 6.4M params, D = 3M tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.

The token budget here is the same 1,500 steps as Lesson 0: about 3 million tokens from 1,500 × 32 × 64. The model is about eight times larger, so the estimated training compute is about eight times larger too. Lesson 5 will turn that relationship into the rule 6·N·D: six units of forward-and-backward work for each of N parameters over each of D training tokens.

Practice

quiz · Check · 1/4open

What are attention weights?

predict · Predict, then runopen

Halve the depth (1 layer instead of 2), same 300 steps. Validation loss?

task · Taskopen

Beat validation loss 2.25 within 300 steps by changing the architecture only. Keep the Shakespeare data and training recipe fixed.

goal: val_loss <= 2.25 within step <= 300

Your attemptpretrain · shakespeare

no run yet

waiting for the first step…
reflect · Why it worksopen
  1. Why is attention a lookup rather than a fixed rule?

  2. Why can stacking blocks do more than one very wide block?

  3. What does the reported count of 801,152 include?

levers · How to improveopen

Symptom: loss is fine but the model can't use context from 40 characters back. Order the levers you would pull, first to last.

  1. raise block_size
  2. add heads
  3. add layers
  4. raise the learning rate

Next

You now know the machine's shape. Next comes the material it learns from: how a dataset is split, batched, cleaned, and allowed to shape every one of those knobs.