The model you trained in Lesson 0 never saw a letter. It saw a stream of integers, and the loss it started from — 4.57 — was the cost of guessing uniformly across the 97 integers it could choose between. This lesson is about that first step: which numbers, why those, and what the choice costs.
Why numbers
A transformer is arithmetic: matrix multiplications, additions, a softmax. Nothing in it can take a string. So before training, every piece of text is turned into a list of integers, and the very first layer of the model is a table that maps each integer to a row of numbers — the embedding. From there on the model has no idea whether id 85 was a t, a pixel, or a chord. The tokenizer is the contract that says what the integers mean.
This course uses one tokenizer everywhere: a fixed alphabet of 97 symbols — newline, then the 95 printable ASCII characters from space to ~, then one catch-all <unk> for anything else. "the cat" becomes 85 73 70 1 68 66 85. It is deliberately boring. The output layer of a model has one row per symbol, and the embedding table one row per symbol, so a model's shape is tied to its vocabulary. Keeping the vocabulary fixed across every lesson means the model you pretrain in Lesson 4 can be loaded and fine-tuned in Lesson 6 without resizing a single matrix.
Merging pairs: BPE
Real models use a bigger vocabulary than 97, and they build it from data with byte-pair encoding. Start with the 256 byte values as the vocabulary. Count every adjacent pair of tokens in the training text, take the pair that occurs most often, and give it a new id — 256 for the first merge, 257 for the next. Replace every occurrence of that pair with the new id, then count again. Each merge shortens the text and adds one entry to the vocabulary; stop when the vocabulary reaches the size you asked for. Encoding new text replays the merges in the order they were learned. That is the whole algorithm — under a hundred lines of Python in lab/trainschool_lab/tokenizers.py, and the same idea, at 100,000 merges instead of three, behind GPT-4's tokenizer.
The trade-off
A bigger vocabulary buys shorter sequences. That matters twice over: attention costs grow with the square of the sequence length, and every token of context is a token the model can look back at, so a 4× shorter sequence sees 4× more text in the same window. The price is paid per token: the embedding table and the output layer each have one row per vocabulary entry, so a 100,000-entry vocabulary makes the output softmax a 100,000-way choice at every position, and each token's embedding has to carry the meaning of a whole word fragment rather than a letter. Character-level is the other extreme — a tiny table, a trivial output layer, and a model that must spend its layers learning how letters make words before it can learn anything about words.
Tokenize anything
Start with bytes, count neighbouring pairs, then watch each merge. The animation updates after you finish typing.
Preparing your animation…
Short examples are easiest to follow (up to 512 characters). Spaces appear as ␣, newlines as ⏎; unfinished Unicode characters stay visible as bytes.
Type a word or a short sentence. Learn BPE merges animates the lab's actual merge loop on your text: start with bytes, count adjacent pairs, choose the most frequent pair, and join it. Only the current step is shown. Pause, use the arrows to move one step at a time, or scrub back to replay a merge. The final frame reports the ratio "chars / tokens". A bigger requested vocabulary allows more merges, until the example has only one token left.
Switch to Character IDs to follow each character into the course's 97-symbol alphabet, or GPT-4 tokens to reveal the pieces and IDs from cl100k_base, its already-trained 100,277-entry vocabulary. Those two views perform a lookup with an existing vocabulary; they do not learn new merges from your input. A leading space can be part of a token: " cat" and "cat" have different IDs. Editing the text resets the animation so every step belongs to your current example.
One thing to try: paste a rocket emoji. It is a single character but four bytes in UTF-8 (F0 9F 9A 80), and cl100k_base has no single token for it, so it splits into a two-byte token and two single-byte tokens — three tokens for one character. Text in scripts that were rare in the tokenizer's training data pays the same price on every character, which is why the same sentence costs more in some languages than in others.
What tokens cost
| hardware | time | cost |
|---|---|---|
| RTX 4090 24GB | 7.4 d | $53.03–$97.22 |
| A100 80GB | 3.9 d | $102.83–$205.66 |
| H100 80GB SXM | 29.5 h | $58.69–$103.22 |
| B200 192GB | 13 h | $45.37–$77.78 |
N = 7B params, D = 1B tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.
| hardware | time | cost |
|---|---|---|
| RTX 4090 24GB | 44.2 h | $13.26–$24.31 |
| A100 80GB | 23.4 h | $25.71–$51.42 |
| H100 80GB SXM | 7.4 h | $14.67–$25.80 |
| B200 192GB | 3.2 h | $11.34–$19.44 |
N = 7B params, D = 250M tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.
Training compute is about 6 FLOPs per parameter per token, and the tokenizer decides how many tokens a corpus is. The boxes price a real-scale case — a 7-billion-parameter model on a billion characters of text — because the lab's 800,000-knob model costs fractions of a cent either way. A billion characters are a billion tokens under the character alphabet and about 250 million under GPT-4's tokenizer, which averages four characters per token on English. Same model, same text, one quarter of the FLOPs: on an H100 that is 29.5 GPU-hours against 7.4, and roughly $60–100 against $15–26. Serving is billed the same way: an API charges per token, so a tokenizer that compresses your language well makes every call cheaper.
Practice
Why does the model need tokens at all?
Train the lab's default model for 300 steps on one sentence repeated 300 times: 7,200 characters of "the cat sat on the mat." Where does validation loss end?
Paste at least 5,000 characters of text you know well and reach a validation loss of 2.0 or lower within 1,500 steps. The last tenth of whatever you paste is held out as validation, so the model has to predict text it never trained on. Six thousand characters of Shakespeare is not enough — find out why, then fix it.
goal: val_loss <= 2 within step <= 1500
Why does a fixed vocabulary let one model be reused across the whole course?
Why does GPT-4 need a vocabulary of 100,000 tokens rather than 97?
What does a token count tell you about cost?
Symptom: your BPE tokenizer turns a 1,000-word paragraph into 6,000 tokens. Order the levers you would pull, first to last.
- switch to character-level
- raise the vocab size
- train BPE on more representative text
- check the text isn't mostly digits or code
Next
The integers are chosen. Next: what the model does with them — the embedding lookup, attention as a weighted lookup over earlier tokens, and how the layers stack.