trainschool/09 · Inference & serving cost‹ prevnext ›

Lesson 9 · 25 min

Inference & serving cost

Training chooses the knobs once. Inference runs those knobs for every answer: one forward pass per token, with caches, batches and smaller weight formats deciding how much service fits on each machine.

In this lesson

You have followed one model through the whole course. Lesson 4 trained its base, Lesson 6 taught it sums, Lesson 7 changed its preferences, and Lesson 8 measured what survived. Now the loop is over. The knobs stop moving and the model begins the job it may do millions of times: take a prompt, predict one token, append it, and run forward again. That repeated forward pass is inference.

Training happens once per checkpoint. Serving happens once per user, per answer, per generated token. This is why a model whose training bill made the headlines can still spend most of its lifetime budget after training.

One token costs 2·N

One token: 2·N1/4 · one token out
0.0s
For a decoder with a KV cache, the dense matrix work for one new token is about 2 operations per counted parameter: 1,602,304 FLOPs for the lesson's N = 801,152. The lab endpoint has no cache and reruns its current context, so its actual work is higher. Training's dense estimate was 6·N because backward added another 4·N.

The model does not write an answer in one pass. It predicts a distribution for the next token, samples one token, appends it to the context, and repeats. With a KV cache, the dense weight matrices run once for each new token, giving the cached-decoder dense-matrix approximation of 2·N FLOPs per generated token. For the lesson model, N = 801,152, so that estimate is 1,602,304 FLOPs per token and about 320 million for a 200-token answer.

That N follows the lab's reported non-position-embedding count. It includes the tied token embedding but excludes the 8,192 learned positional-embedding weights (64 positions × 128 wide). The estimate also omits the attention work that reads the growing cache. It is a useful dominant-term approximation for production decoders, not an exact count of the lab endpoint: the actual lab generator has no KV cache and reruns up to all 64 positions in its context window for every new token.

Training estimated 6·N of dense work per token: 2·N forward and 4·N backward. A cached production decoder never computes parameter gradients or updates an optimiser, so its basic dense work is about one third of the training estimate for the same number of tokens. That comparison is about dominant operations, not wall-clock speed: generation has to wait for token 1 before it can begin token 2, while training processes many known tokens together.

Remember the past instead of recomputing it

The KV cache1/5 · token 1
0.0s
Without a cache, every new token would rebuild the attention keys and values for all earlier tokens. A KV cache stores those tensors once and reads them again. Compute falls, while memory grows with context length and with the number of active requests.

Self-attention needs information from every earlier token. The naive generator feeds the whole growing string back through the model and recomputes those earlier attention keys and values every time. Token 100 would rebuild work already done for tokens 1 through 99.

A KV cache keeps each layer's keys and values after their first pass. The next token computes only its new key and value, then attends to the stored shelf. The trade is memory for compute: cache memory grows linearly with context length, layers, hidden width and the number of active requests. The model's weights can be shared across a batch, but each request has its own prompt and therefore its own KV cache.

Caching does not make attention free. Every new query still reads the growing cache, so very long contexts become dominated by memory movement. Cache precision, grouped-query attention, paged allocation and eviction policy all change the real number. The lesson's throughput toy uses one explicit representative allowance rather than pretending model size alone determines it.

Batching fills idle hardware

Batching requests1/4 · one request
0.0s
An idealised utilisation sketch: one request leaves much of a GPU idle, while eight independent requests expose enough parallel work to approach the same fixed compute ceiling. Aggregate throughput rises; each request still waits for sequential tokens, and queueing can raise latency.

A single request has a thin, sequential decode loop. Large accelerators have far more parallel arithmetic than that loop can feed. A serving system can place several requests in one batch, multiply their matrices together, and use more of the same device at once.

That raises aggregate throughput: the server finishes more tokens per second across all users. It does not multiply the GPU's fixed-MFU FLOP ceiling by the batch size. The diagram is an idealised utilisation curve, not a benchmark or an unconditional 8× promise: this lesson assumes utilisation fills linearly through batch 8 and then stops. Real saturation can arrive earlier or later.

Latency answers a different question. Each request receives only its share of aggregate throughput, and a batcher may hold a prompt briefly while it gathers peers. Continuous batching can replace finished rows, but a larger batch can still make one user wait longer even while it lowers cost per token for the service.

Fewer bits per knob

Quantisation1/4 · fp16: 2 bytes a knob
0.0s
The weight tile shrinks from fp16 to int8 to 4-bit. The +0.03 loss is an illustrative toy delta, not a measured result or a guarantee: real quality and speed depend on the model, calibration, kernels and hardware.

The capacity toy assumes fp16 weights at 2 bytes per parameter, so a 7 B model needs about 14 GB just for weights. The actual lab model runs in float32 on CPU, 4 bytes per weight, and its generator does not quantise them. Int8 uses 1 byte per parameter; 4-bit uses half a byte. Smaller weights can fit on a smaller GPU and move through memory with less traffic.

Those formats are approximations. Rounding can change the output distribution, and some layers are more sensitive than others. A quantised model can be faster when the hardware has suitable kernels and memory bandwidth is the bottleneck; it can also be the same speed or slower after conversion overhead. Quality loss and speed gain are not guaranteed. Measure both on the prompts and hardware that matter.

Throughput and the serving bill

The toy below is a planning model, not a live hardware quote. Its arithmetic lives in the same cost library as the course calculator:

  • the compute ceiling is GPU peak BF16 × 30 % MFU ÷ (2·N) aggregate tokens/s;
  • batch recovers utilisation up to that fixed 30 % MFU compute ceiling at batch 8, then aggregate throughput stays flat;
  • capacity counts shared fp16 weights once (2 bytes·N) plus 1 GiB per active request for a representative 2,048-token KV cache;
  • if the weights plus one cache do not fit on one GPU, it says does not fit on one GPU and shows no numeric throughput or bill;
  • per-request tokens/s is aggregate throughput divided by the active batch;
  • the dollar range uses the stored September 2026 price snapshot, not a live quote.

The 1 GiB cache allowance is roughly a 32-layer, width-4,096 decoder with fp16 K and V at a 2,048-token context. It is deliberately visible because it is only an approximation. Architecture, grouped-query attention, cache precision and context length can move it sharply. The linear fill to batch 8 is likewise an idealised utilisation curve; it prevents batching from creating compute that the GPU does not have.

toy · fill the GPU

aggregate 21,193 tok/s

per request 2,649 tok/s

per 1M tokens $0.03–$0.05

8 active requests; weights are shared.

Classroom estimate: 2·N FLOPs/token, 30 % MFU, batch 8 to fill the compute ceiling, fp16 weights, and 1 GiB KV cache per active 2k-context request. Real kernels, model shapes and contexts differ. GPU prices are the stored 2026-09 snapshot.

At the toy's 30 % MFU ceiling, a 7 B model on the H100 row is about 21,193 aggregate tokens/s and the pure FLOP bill is $0.026–$0.046 per 1 million tokens. A real 7 B serving stack might deliver around 2,000 batched tokens/s after memory traffic, kernels, scheduling and sampling; treat that as an illustrative operational number, not a benchmark of this lab. Keeping one H100 occupied at 2,000 tokens/s would put the same stored price range around $0.28–$0.49 per million tokens. The gap is the price of assumptions the FLOP estimate leaves out.

The lab model is a different scale and a different machine. Its CPU generator is around 800 decoding tokens/s on this box for a 200-token sample. It does not implement a KV cache, request batching or quantisation; it keeps float32 weights and reruns up to all 64 positions in its context window for every new token. Its speed demonstrates this lab's sequential generation loop rather than the production optimisations in the toy.

Serve your checkpoints

Generate 200 tokens from your Lesson 4 base and your Lesson 6 fine-tune. Read the decoding tokens/s under each answer. Try temperature 0.2 and 1.5 with the same prompt and seed, then move top-k from 1 to 97.

lab · sample from your model

loading runs…

Temperature rescales the logits before sampling. At 0.2 the largest scores become much more dominant; at 1.5 the retained alternatives become closer together. Top-k keeps only the k highest-scoring tokens before sampling: 1 is greedy, 40 is the default, and 97 admits the lab's whole character vocabulary. Seed fixes the random draws, so changing one control at a time makes the comparison readable.

This CPU toy implements those sampling controls. It does not implement the KV cache or quantisation shown above, and it handles one request at a time. The tokens/s it prints are decoding tokens/s. The Predict and Task below launch training runs and report training tokens/s over already-known batches. They are different workloads and cannot be compared as one benchmark.

Where the lifetime money goes

Pretraining pays a huge bill once to choose the weights. Fine-tuning pays smaller bills to revise them. Serving keeps paying while the product is used. One million users receiving 200 new tokens means 200 million token predictions; batching may combine many predictions into one device call, but their dense and attention work remains. Repeat that volume every day and the lifetime inference tokens quickly outrun a one-time experiment.

This changes the useful levers. Training asks how to finish a learning run efficiently. Serving asks how to avoid repeated work with a KV cache, keep the device occupied with batches, reduce bytes moved with quantisation, and choose the smallest model that still passes the evaluation. Every lever has a boundary: caches consume memory, batching trades latency against throughput, quantisation can move quality, and a smaller model may fail the task.

Practice

quiz · Check · 1/4open

About how many FLOPs does one generated token cost for an N-parameter decoder?

predict · Predict, then runopen

Make the quick training model 4× wider, from 64 to 256. Its training throughput in tokens/s?

task · Taskopen

Build the fastest training model you can. Reach at least 390,000 training tokens/s on this box. This checks training batches; it does not estimate serving speed.

goal: tokens_per_s >= 390000

Your attemptpretrain · shakespeare

no run yet

waiting for the first step…
reflect · Why it worksopen
  1. Why can serving dominate a model's lifetime cost even when one inference token is cheaper than one training token?

  2. What would you quantise first, and what would you measure before keeping the change?

  3. How can batching improve throughput while making an individual request slower?

levers · How to improveopen

Symptom: each user waits 4 s for a 100-token answer. Order the levers you would pull, first to last.

  1. quantise to int8
  2. use a smaller model
  3. enable the KV cache
  4. batch concurrent requests

The loop now closes: data became tokens, tokens moved through a transformer, loss moved its knobs, scale set the bill, fine-tuning and preferences changed its behaviour, evaluation checked the result, and inference turns that checkpoint into a service. Every number in the service still comes back to the same three questions: how much work does one token require, how well does the hardware stay occupied, and what memory must remain resident while users wait?