What a matmul costs — Linear algebra for LLMs

One entry: $k$ multiplies and $k$ adds

\displaystyle C_{ij} = a_{i1}b_{1j} + \dots + a_{ik}b_{kj}

One entry of $C = AB$ , with $A\,(m \times k)$ and $B\,(k \times n)$ , takes $k$ multiplications and $k - 1$ additions. The standard count rounds this to $2k$ floating-point operations (FLOPs): GPUs do a multiply and an add together as one fused multiply-add, which counts as 2 FLOPs.

Write

One entry

\approx 2k

FLOPs (

k

multiply-adds).

The whole product: $2mkn$

\displaystyle \text{FLOPs}(AB) = 2\,m\,k\,n

\displaystyle (2 \times 3)(3 \times 2): \; 2 \cdot 2 \cdot 3 \cdot 2 = 24

There are $m \times n$ entries and each costs $2k$ . Memorize this one; every cost estimate for a transformer is built from it.

Write

(m \times k)(k \times n)

costs

2mkn

FLOPs.

A transformer-sized product

\displaystyle (2048 \times 4096)(4096 \times 4096)

\displaystyle 2 \cdot 2048 \cdot 4096 \cdot 4096 \approx 6.9 \times 10^{10}

Projecting 2,048 tokens of width 4,096 through one $4096 \times 4096$ weight takes 68,719,476,736 FLOPs, about 69 billion. That’s a single weight in a single layer.

✏️ Your turn

One token through one weight

\displaystyle (1 \times 4096)(4096 \times 4096)

How many FLOPs? And how many parameters does the weight have?

Work it out on paper, then go to the next card to check.

Answer: 2 FLOPs per parameter

\displaystyle \text{FLOPs} = 2 \cdot 4096^2 \approx 3.4 \times 10^{7}

\displaystyle \text{params} = 4096^2 \approx 1.7 \times 10^{7}

33,554,432 FLOPs and 16,777,216 parameters: exactly 2 FLOPs per parameter per token, one multiply and one add for each weight.

In a transformer

Add this up over every weight in the model and you get the rule of thumb forward pass $\approx 2N$ FLOPs per token for $N$ parameters (Part 6).

Write

One token through a weight: 2 FLOPs per parameter. Forward pass

\approx 2N

FLOPs per token.

Order matters for cost: $(AB)x$ vs $A(Bx)$

$A$ and $B$ are $n \times n$ , $x$ is a vector, $n = 4096$ .

Order	Work	FLOPs
$(AB)x$	a matrix product, then matrix × vector	$2n^3 + 2n^2 \approx 1.37 \times 10^{11}$
$A(Bx)$	two matrix × vector products	$4n^2 \approx 6.71 \times 10^{7}$

Same answer, about $(n + 1)/2 \approx 2000$ times the work. Associativity lets you choose the cheap order.

Write

Same result, different cost: multiply by the vector first.

Where cheap ordering shows up

In a transformer

LoRA stores a weight update as two thin matrices instead of one $d \times d$ matrix: $A$ maps $d$ features down to $r$ , and $B$ maps $r$ back up to $d$ , with $r$ small (say 8). Passing a token through $A$ , then $B$ , costs $2dr + 2rd = 4dr$ FLOPs. Multiplying the two matrices together first would build a full $d \times d$ matrix: $2d^2 r$ FLOPs to make, then $2d^2$ per token. For $d = 4096$ and $r = 8$ that’s about 131 thousand FLOPs per token against 34 million. (At inference you can instead merge the update into the weight once; Part 7.)

📌 Recap

What a matmul costs in one card

$(m \times k)(k \times n)$ costs $2mkn$ FLOPs: a multiply-add is 2 FLOPs.
One token through a weight: 2 FLOPs per parameter, so a forward pass $\approx 2N$ FLOPs per token.
$(AB)x = A(Bx)$ , but $A(Bx)$ is far cheaper: choose the order.

Write

The three lines above.