Arithmetic intensity

Two speed limits

\displaystyle \text{compute: } \approx 989 \text{ TFLOP/s (BF16, dense)}

\displaystyle \text{memory: } 3.35 \text{ TB/s}

An NVIDIA H100 SXM, from its datasheet (as of 2026). Every operation needs some math and some data read from memory; whichever takes longer sets its speed.

\displaystyle I = \frac{\text{FLOPs}}{\text{bytes moved}}

\displaystyle \text{ridge point} = \frac{989 \times 10^{12}}{3.35 \times 10^{12}} \approx 295 \text{ FLOPs/byte}

An operation that does fewer than about 295 FLOPs per byte it reads is memory-bound on this GPU: the math units sit idle waiting for data. Above that, it’s compute-bound.

Write

I

= FLOPs / bytes. Below the ridge (≈ 295 on an H100): memory-bound.

Matrix × vector: 1 FLOP per byte

\displaystyle xW,\ W\!: d \times d \text{ in BF16}

\displaystyle \text{FLOPs} = 2d^2, \quad \text{bytes} = 2d^2 \;\Rightarrow\; I = 1

Each weight is read once (2 bytes) and used once (a multiply-add, 2 FLOPs). Generating one token for one sequence is all matrix × vector products, so it’s about 300× below the ridge.

Write

Decode at batch 1:

I \approx 1

FLOP/byte. Memory-bound.

Batching shares each weight read

\displaystyle XW,\ X\!: B \times d

\displaystyle \text{FLOPs} = 2Bd^2, \quad \text{bytes} \approx 2d^2 \;\Rightarrow\; I \approx B

$B$ sequences decoding together reuse every weight $B$ times per read. Intensity grows with the batch, until around a few hundred it reaches the ridge. That’s why serving systems batch many users together. Prefill (processing a prompt of $T$ tokens at once) has $I \approx T$ and is compute-bound.

Write

Batch

B

:

I \approx B

. Prefill of

T

tokens:

I \approx T

.

A decode speed limit

\displaystyle \text{tokens/s} \le \frac{\text{bandwidth}}{\text{bytes read per token}}

\displaystyle \frac{3.35 \times 10^{12}}{16.06 \times 10^{9}} \approx 209 \text{ tokens/s}

Llama 3 8B in BF16 reads 16 GB of weights per generated token (plus its KV cache). One H100 can’t generate faster than about 209 tokens per second for a single sequence, however many FLOPs it has.

Write

Decode ceiling ≈ bandwidth / (weight bytes + KV bytes).

✏️ Your turn

Compute a ceiling

\displaystyle \text{weights: } 4 \text{ GB (INT4)}

\displaystyle \text{bandwidth: } 3.35 \text{ TB/s}

Roughly how many tokens per second at most, for one sequence? (Ignore the KV cache.)

Work it out on paper, then go to the next card to check.

Answer: about 840

\displaystyle \frac{3.35 \times 10^{12}}{4 \times 10^9} \approx 838

Four times the BF16 ceiling: fewer bytes per weight is directly more tokens per second when you’re memory-bound.

What this explains

In a transformer

Quantizing weights speeds up decoding even if the math is no faster.
The KV cache (Part 5) is read every step too, so long contexts slow decoding down.
Speculative decoding and batching both turn matrix × vector work into matrix × matrix work, raising intensity.

📌 Recap

Arithmetic intensity in one card

$I$ = FLOPs per byte; compare with the GPU’s ridge (≈ 295 for an H100 in BF16).
Matrix × vector: $I \approx 1$ . Batch $B$ : $I \approx B$ .
Decode speed ≤ bandwidth / bytes per token.

Write

The three lines above.

Sources

NVIDIA H100 Tensor Core GPU datasheet: H100 SXM, 80 GB HBM3, 3.35 TB/s; BF16 Tensor Core 1,979 TFLOPS with sparsity, so about 989 TFLOPS dense.