Count a model — Linear algebra for LLMs

One classic block: $12d^2$

\displaystyle \underbrace{4d^2}_{\text{attention}} + \underbrace{8d^2}_{\text{MLP}} = 12d^2

Multi-head attention has four $d \times d$ matrices; a $4d$ MLP has two $d \times 4d$ ones. Biases and norm gains add only a few thousand per block, small enough to ignore.

Write

Classic block (MHA,

4d

MLP):

12d^2

parameters.

The whole model

\displaystyle N \approx 12\,L\,d^2 + V d \;(+\, V d \text{ if not tied})

$L$ blocks, plus the embedding table, plus a separate unembedding when the weights aren’t tied. (GPT-2 also has a learned position table, $T_{\max} \times d$ .)

Write

N \approx 12Ld^2 + Vd

(tied).

Check it on GPT-2 small

\displaystyle 12 \times 12 \times 768^2 = 84{,}934{,}656

\displaystyle 50{,}257 \times 768 = 38{,}597{,}376

\displaystyle 1024 \times 768 = 786{,}432

\displaystyle \text{total} \approx 124.3 \text{ million}

$L = 12$ , $d = 768$ , tied weights, 1,024 positions. GPT-2 small is known as the 124M model; counting every bias and gain exactly gives 124,439,808.

Llama 3 8B, one layer

\displaystyle \text{attention: } 2 \cdot 4096^2 + 2 \cdot 4096 \cdot 1024 = 41{,}943{,}040

\displaystyle \text{MLP: } 3 \cdot 4096 \cdot 14336 = 176{,}160{,}768

\displaystyle \text{two RMSNorm gains: } 2 \cdot 4096 = 8{,}192

\displaystyle \text{per layer: } 218{,}112{,}000

$W_Q$ and $W_O$ are $4096 \times 4096$ ; with 8 KV heads of 128, $W_K$ and $W_V$ are $4096 \times 1024$ (GQA, Part 5). No biases.

Llama 3 8B, the whole model

\displaystyle 32 \times 218{,}112{,}000 = 6{,}979{,}584{,}000

\displaystyle +\; 2 \times 128{,}256 \times 4096 = 1{,}050{,}673{,}152

\displaystyle +\; 4096 = 8{,}030{,}261{,}248

32 layers, an embedding table and a separate output matrix (not tied), and a final norm: 8.03 billion parameters, which is why it’s called 8B. About 13% of it is the two vocabulary matrices.

Write

Llama 3 8B:

32 \times 218.1\text{M} + 2 \times 525.3\text{M} \approx 8.03\text{B}

.

FLOPs per token

\displaystyle \text{forward} \approx 2N \text{ FLOPs per token}

\displaystyle \text{attention over } T \text{ tokens} \approx 4\,L\,T\,d

From Part 2: one multiply-add (2 FLOPs) per weight per token. Llama 3 8B: about 16 GFLOPs per token. Attention adds the $QK^\top$ and $AV$ work, $2Td$ each per layer: at $T = 8192$ that’s $4 \cdot 32 \cdot 8192 \cdot 4096 \approx 4.3$ GFLOPs more.

Write

Forward:

\approx 2N

FLOPs/token, plus

4LTd

for attention over

T

tokens.

✏️ Your turn

Size a model

\displaystyle L = 24, \quad d = 2048, \quad V = 50{,}000, \quad \text{tied}

Estimate $N$ with $12Ld^2 + Vd$ , and the forward FLOPs per token (ignore attention).

Work it out on paper, then go to the next card to check.

Answer: about 1.3B

\displaystyle 12 \times 24 \times 2048^2 = 1{,}207{,}959{,}552

\displaystyle +\; 50{,}000 \times 2048 = 102{,}400{,}000

\displaystyle N \approx 1.31 \times 10^9, \quad 2N \approx 2.6 \text{ GFLOPs per token}

📌 Recap

Count a model in one card

Classic block: $12d^2$ . Model: $12Ld^2 + Vd$ (tied).
GPT-2 small: about 124M. Llama 3 8B: 8,030,261,248, counted matrix by matrix.
Forward pass: $\approx 2N$ FLOPs per token, plus $4LTd$ for attention.

Write

The three lines above.