trainschool/01 · Tokens: how text becomes numbers‹ prevnext ›

Lesson 1 · 20 min

Tokens: how text becomes numbers

A model does arithmetic, so text has to become numbers first. How a tokenizer chooses those numbers decides how long every sequence is — and how much every run costs.

In this lesson

The model you trained in Lesson 0 never saw a letter. It saw a stream of integers, and the loss it started from — 4.57 — was the cost of guessing uniformly across the 97 integers it could choose between. This lesson is about that first step: which numbers, why those, and what the choice costs.

Why numbers

Characters become ids1/4 · a sentence
0.0s
Every character has a fixed slot in a 97-symbol alphabet: newline is 0, space is 1, the printable ASCII characters follow in order, and anything else is 96. The model receives the slot numbers and nothing else.

A transformer is arithmetic: matrix multiplications, additions, a softmax. Nothing in it can take a string. So before training, every piece of text is turned into a list of integers, and the very first layer of the model is a table that maps each integer to a row of numbers — the embedding. From there on the model has no idea whether id 85 was a t, a pixel, or a chord. The tokenizer is the contract that says what the integers mean.

This course uses one tokenizer everywhere: a fixed alphabet of 97 symbols — newline, then the 95 printable ASCII characters from space to ~, then one catch-all <unk> for anything else. "the cat" becomes 85 73 70 1 68 66 85. It is deliberately boring. The output layer of a model has one row per symbol, and the embedding table one row per symbol, so a model's shape is tied to its vocabulary. Keeping the vocabulary fixed across every lesson means the model you pretrain in Lesson 4 can be loaded and fine-tuned in Lesson 6 without resizing a single matrix.

Merging pairs: BPE

Byte-pair encoding1/5 · count adjacent pairs
0.0s
The text starts as bytes. Count every adjacent pair, replace the commonest with a new id, repeat. Three merges take aaabdaaabac from 11 tokens to 5 and add three entries to the vocabulary. The counts are the lab's own algorithm on the same string.

Real models use a bigger vocabulary than 97, and they build it from data with byte-pair encoding. Start with the 256 byte values as the vocabulary. Count every adjacent pair of tokens in the training text, take the pair that occurs most often, and give it a new id — 256 for the first merge, 257 for the next. Replace every occurrence of that pair with the new id, then count again. Each merge shortens the text and adds one entry to the vocabulary; stop when the vocabulary reaches the size you asked for. Encoding new text replays the merges in the order they were learned. That is the whole algorithm — under a hundred lines of Python in lab/trainschool_lab/tokenizers.py, and the same idea, at 100,000 merges instead of three, behind GPT-4's tokenizer.

The trade-off

Vocabulary size against sequence length1/5 · same text, three tokenizers
0.0s
One 60-character sentence through three tokenizers, measured with the lab: the course's 97-character alphabet gives 60 tokens; a BPE vocabulary of 512 trained on Shakespeare gives 28; GPT-4's 100,277-entry vocabulary gives 14. Each cell is one token.

A bigger vocabulary buys shorter sequences. That matters twice over: attention costs grow with the square of the sequence length, and every token of context is a token the model can look back at, so a 4× shorter sequence sees 4× more text in the same window. The price is paid per token: the embedding table and the output layer each have one row per vocabulary entry, so a 100,000-entry vocabulary makes the output softmax a 100,000-way choice at every position, and each token's embedding has to carry the meaning of a whole word fragment rather than a letter. Character-level is the other extreme — a tiny table, a trivial output layer, and a model that must spend its layers learning how letters make words before it can learn anything about words.

Tokenize anything

Text → tokens, one step at a timeTry your own example

Start with bytes, count neighbouring pairs, then watch each merge. The animation updates after you finish typing.

Preparing your animation…

Short examples are easiest to follow (up to 512 characters). Spaces appear as ␣, newlines as ⏎; unfinished Unicode characters stay visible as bytes.

Type a word or a short sentence. Learn BPE merges animates the lab's actual merge loop on your text: start with bytes, count adjacent pairs, choose the most frequent pair, and join it. Only the current step is shown. Pause, use the arrows to move one step at a time, or scrub back to replay a merge. The final frame reports the ratio "chars / tokens". A bigger requested vocabulary allows more merges, until the example has only one token left.

Switch to Character IDs to follow each character into the course's 97-symbol alphabet, or GPT-4 tokens to reveal the pieces and IDs from cl100k_base, its already-trained 100,277-entry vocabulary. Those two views perform a lookup with an existing vocabulary; they do not learn new merges from your input. A leading space can be part of a token: " cat" and "cat" have different IDs. Editing the text resets the animation so every step belongs to your current example.

One thing to try: paste a rocket emoji. It is a single character but four bytes in UTF-8 (F0 9F 9A 80), and cl100k_base has no single token for it, so it splits into a two-byte token and two single-byte tokens — three tokens for one character. Text in scripts that were rare in the tokenizer's training data pays the same price on every character, which is why the same sentence costs more in some languages than in others.

What tokens cost

1 token per character — 1 B chars, 7 B params6·N·D = 4.2e19 FLOPs
hardwaretimecost
RTX 4090 24GB7.4 d$53.03–$97.22
A100 80GB3.9 d$102.83–$205.66
H100 80GB SXM29.5 h$58.69–$103.22
B200 192GB13 h$45.37–$77.78

N = 7B params, D = 1B tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.

GPT-4 tokenizer (≈4 chars/token) — same corpus6·N·D = 1.05e19 FLOPs
hardwaretimecost
RTX 4090 24GB44.2 h$13.26–$24.31
A100 80GB23.4 h$25.71–$51.42
H100 80GB SXM7.4 h$14.67–$25.80
B200 192GB3.2 h$11.34–$19.44

N = 7B params, D = 250M tokens. Assumes 40% MFU and on-demand cloud prices as of 2026-09.

Training compute is about 6 FLOPs per parameter per token, and the tokenizer decides how many tokens a corpus is. The boxes price a real-scale case — a 7-billion-parameter model on a billion characters of text — because the lab's 800,000-knob model costs fractions of a cent either way. A billion characters are a billion tokens under the character alphabet and about 250 million under GPT-4's tokenizer, which averages four characters per token on English. Same model, same text, one quarter of the FLOPs: on an H100 that is 29.5 GPU-hours against 7.4, and roughly $60–100 against $15–26. Serving is billed the same way: an API charges per token, so a tokenizer that compresses your language well makes every call cheaper.

Practice

quiz · Check · 1/4open

Why does the model need tokens at all?

predict · Predict, then runopen

Train the lab's default model for 300 steps on one sentence repeated 300 times: 7,200 characters of "the cat sat on the mat." Where does validation loss end?

task · Taskopen

Paste at least 5,000 characters of text you know well and reach a validation loss of 2.0 or lower within 1,500 steps. The last tenth of whatever you paste is held out as validation, so the model has to predict text it never trained on. Six thousand characters of Shakespeare is not enough — find out why, then fix it.

goal: val_loss <= 2 within step <= 1500

Your attemptpretrain · custom

paste some training text

no run yet

waiting for the first step…
reflect · Why it worksopen
  1. Why does a fixed vocabulary let one model be reused across the whole course?

  2. Why does GPT-4 need a vocabulary of 100,000 tokens rather than 97?

  3. What does a token count tell you about cost?

levers · How to improveopen

Symptom: your BPE tokenizer turns a 1,000-word paragraph into 6,000 tokens. Order the levers you would pull, first to last.

  1. switch to character-level
  2. raise the vocab size
  3. train BPE on more representative text
  4. check the text isn't mostly digits or code

Next

The integers are chosen. Next: what the model does with them — the embedding lookup, attention as a weighted lookup over earlier tokens, and how the layers stack.