trainschool/05 · Scaling laws → GPU-hours → dollars‹ prevnext ›

Lesson 5 · 25 min

Scaling laws → GPU-hours → dollars

Turn model size and training tokens into FLOPs, turn FLOPs into GPU-hours with model FLOPs utilisation, and turn GPU-hours into a dated dollar estimate. Then choose where to sit relative to the Chinchilla frontier.

In this lesson

Every run in the course ends with three compute numbers: parameters, tokens and FLOPs. This lesson connects them to time and money. There is no training lab here; the calculator is the lab. Its prices are a recorded September 2026 snapshot, not a live quote, and every answer is an estimate whose assumptions stay visible.

The rectangle behind the bill

Why training costs about 6ND1/5 · N knobs
0.0s
A forward pass does about two arithmetic operations per parameter per token; its backward pass does about four. Multiply the N-by-D rectangle by six. For GPT-3's published 175 billion parameters and 300 billion training tokens, that estimate is 3.15e23 FLOPs.

Start with N parameters. For one token, the forward pass uses each matrix weight in roughly one multiply and one add, about 2N floating-point operations. Run that over D training tokens and the forward pass is about 2ND.

Training also needs the backward pass. It computes gradients for the inputs and for the weights, roughly two forward passes' worth of work, or 4ND. Add the two pieces:

forward 2ND + backward 4ND = training 6ND

This is an accounting approximation, not an instruction count. Embeddings, attention's context term, optimiser updates, recomputation and communication all bend it. It remains useful because the large matrix multiplications dominate ordinary transformer training and because it states the two levers plainly: more knobs and more token presentations both grow the rectangle.

Peak hardware is not training throughput

Model FLOPs utilisation1/4 · peak
0.0s
An H100 SXM is rated at 989 dense BF16 tensor TFLOPS. The 40% amber bar is a planning assumption, not a benchmark: it budgets 396 useful model TFLOPS after data movement, kernels, synchronisation and idle gaps.

A specification-sheet peak says how much arithmetic the chip can perform under the right instruction mix. A training loop never feeds its tensor cores perfectly. It moves weights and activations through memory, launches kernels, synchronises devices, prepares batches, saves checkpoints and sometimes waits.

Model FLOPs utilisation (MFU) divides the model FLOPs you actually process each second by the hardware peak. The calculator defaults to 40% because it is an honest planning assumption for a well-run but imperfect job, not because every H100 delivers exactly that. A small model or batch can be much lower; fused kernels, large batches and careful parallelism can be higher. Measure MFU on the workload when a budget matters.

The time estimate is therefore:

GPU-hours = training FLOPs ÷ (peak FLOPs/s × MFU) ÷ 3,600

At 40% MFU, one H100 in this snapshot contributes about 396 trillion useful model FLOPs per second. Asking for 100% would erase every systems cost from the estimate.

GPU-hours, wall-clock and dollars

From GPU-hours to a bill1/5 · hours
0.0s
The calculator's estimate for 175B × 300B on an H100 at 40% MFU: 221,183 GPU-hours. At the snapshot's $1.99–$3.50 per GPU-hour that is $440k–$774k. Divide those GPU-hours by 1,024 GPUs, then by 24 hours per day, for nine days of wall-clock time.

GPU-hours add work across devices. One GPU for 24 hours and 24 GPUs for one hour are both 24 GPU-hours if scaling is perfect. Wall-clock time divides that work by the number of GPUs. It changes when the result arrives; it does not make the arithmetic disappear.

GPT-3's 3.15e23 FLOPs become about 221,183 GPU-hours on the calculator's H100 and 40% MFU assumptions. Divide by 1,024 GPUs and by 24 hours per day and the ideal answer is almost exactly nine days. Real clusters communicate, retry work and lose capacity, so actual wall-clock and GPU-hours are higher. The scene's nine days is the clean arithmetic floor.

The stored H100 range is $1.99–$3.50 per GPU-hour, producing about $440,000–$774,000 for that arithmetic estimate. Cloud prices depend on region, provider, date, commitment and availability. On-demand capacity buys flexibility; reserved or committed capacity often lowers the rate in exchange for a longer promise. The calculator labels its date and shows the recorded range rather than pretending either end is a guaranteed offer today. This is accelerator rental only, before engineering, storage, networking, failed runs and data work.

Put your own run in the calculator

training FLOPs (6·N·D)7.44e18
GPU-hours5.2 h
wall-clock on 8 GPUs39.2 min
cost (on-demand, low–high)$10.40–$18.28
training memory (16 B/param), GB2
GPUs just to hold the training state1
Chinchilla-optimal tokens (20·N)2.5B

Prices as of 2026-09. Wall-clock ignores multi-GPU communication overhead, which is real above a few hundred GPUs.

Choose my lab run after completing Lesson 4. It loads that run's measured parameter and token counts. The default Lesson 4 run has 801,152 parameters and processes 3,072,000 tokens, so 6ND is 1.5e13 FLOPs. Under the calculator's H100, 40% MFU and stored price range, that is about $0.00003 of theoretical accelerator time, roughly $0.00002–$0.00004. The same work took 42 seconds in the recorded Lesson 0 laptop CPU run. Different hardware, software and fixed overheads make those clocks incomparable; 6ND is the shared amount of model arithmetic.

Change one input at a time. Doubling tokens doubles FLOPs. Doubling parameters also doubles this estimate, although a wider matrix can change real hardware efficiency. More GPUs divide ideal wall-clock but leave ideal GPU-hours and dollars unchanged. MFU changes both GPU-hours and dollars because better utilisation finishes the same arithmetic in fewer rented hours.

The Chinchilla frontier

Parameters or tokens?1/4 · more knobs or more tokens?
0.0s
With N on the horizontal axis and D on the vertical, the dashed line is D = 20N. GPT-3 is below it and under-trained by this rule; Llama 3 8B is far above it and over-trained. The labels describe position relative to the compute-optimal line, not data quality.

If a fixed training budget buys a larger model or more tokens, where should it go? The Chinchilla scaling study fitted empirical loss curves and found a useful rule near 20 training tokens per parameter. “Chinchilla-optimal” means loss-optimal for a fixed training-compute budget under that study's assumptions. It does not mean best for latency, memory, serving cost or every dataset.

Read the axes before saying above or below: N on the horizontal axis and D on the vertical. A point above D = 20N used more than 20 tokens per parameter and is over-trained relative to that compute-optimal frontier. A point below it used fewer and is under-trained. GPT-3 sits below the line at about 1.7 tokens per parameter. Llama 3 8B sits far above it at about 1,875 tokens per parameter. Chinchilla 70B sits on the rule by construction in this comparison.

Modern open models often train smaller models on far more tokens than the loss-optimal training frontier recommends. The training bill rises, but inference gets cheaper: generated-token compute and memory scale strongly with parameter count, and a model may be served millions of times after it is trained once. Spending extra training compute on data can buy a smaller model that is easier to deploy. The frontier answers a training-loss question; product economics can rationally choose another point.

The course lab is below the line too: about 3.8 tokens per parameter instead of 20. That is deliberate. Its job is to make the whole loop visible on a laptop, not to set a record for loss per FLOP.

Practice

quiz · Check · 1/4open

What makes up the 6 in the training estimate 6ND?

predict · Predict, then runopen

The recorded quick preset is 2 layers × 64 wide and runs at 249,056 tokens/s on this idle 8-thread CPU. Double only the width to 128. What happens to training throughput?

task · Taskopen

Spend at least 1e14 FLOPs in one run. The measured 4 layers × 256 wide, 3,000-step configuration took 247.7 seconds (4m08s) on the idle 8-thread CPU and spent 1.1705e14 FLOPs, so budget about four minutes for the direct route.

goal: flops >= 100000000000000

Your attemptpretrain · shakespeare

no run yet

waiting for the first step…
reflect · Why it worksopen
  1. Why did doubling width not divide tokens/s by exactly four on the CPU?

  2. Why can the same number of FLOPs cost more on a slower GPU?

  3. What would you change first to halve a training bill?

levers · How to improveopen

Symptom: the run is projected to take 3 days on one GPU. Order the levers you would pull, first to last.

  1. train fewer tokens
  2. use more GPUs (wall-clock, same bill)
  3. shrink the model
  4. raise MFU (bigger batch, fused kernels)