llm-inference-arithmetic

Run the numbers yourself.

Five calculations that decide what an LLM deployment costs and how fast it feels. Everything below runs in this page — no server, no GPU, no model download. The same arithmetic is in the Python package, and its test suite asserts these figures, so if the two ever disagree one of them is wrong.

1. Are you memory-bound?

Peak arithmetic rate divided by memory bandwidth gives the ridge point: the operations-per-byte below which you are paying to move bytes rather than to multiply them. In decode, your arithmetic intensity is roughly your batch size.

ridge point
your arithmetic intensity
achievable

Intensity is batch × (2 ÷ bytes-per-weight), so it equals the batch only at 2-byte weights. Peak figures are dense: vendors often publish the with-sparsity number, which is double.

2. What one conversation costs

Two, times the layers whose cache grows, times the key-value heads, times the head dimension, times the bytes per number. Every value is in the model's config.json.

per token

Read layer_types before you trust the layer count. Models that interleave sliding-window attention only grow the cache on their full-attention layers. gpt-oss-120b has 36 layers and 18 of them grow, so using 36 doubles your answer.

3. Does your target concurrency fit?

Total memory, minus the weights, divided by what one conversation costs. This belongs in every capacity plan and is almost never in one.

KV budget
per conversation
conversations that fit

Uses the cache size from calculator 2 above, so change the model there and this follows. Utilisation defaults to vLLM's gpu_memory_utilization of 0.92, and 5 GiB is held back for activations and CUDA graphs.

4. What is that kernel speedup actually worth?

A kernel that runs twice as fast only helps in proportion to how much of your time was spent in it. The speedup you were quoted was measured on somebody else's workload.

end to end
ceiling, if that part became free
speedup needed for 1.5× overall

5. Can the benchmark prove its own claim?

Convert the percentages back into items before you believe anything. A dozen items is not a finding, however large the percentage gap looks.

counts
difference
95% interval, baseline
95% interval, comparison
two-proportion p

Overlapping intervals prove nothing. Non-overlap implies significance; overlap does not imply its absence. If both models were scored on the same items the correct test is McNemar's, which needs to know which items each one got wrong — and a published percentage never carries that.