Composition — Linear algebra for LLMs

One map after another

\displaystyle B(Ax) = (BA)\,x

Apply $A$ , then apply $B$ to the result. The product $BA$ does both in one step. Read it right to left: the matrix nearest $x$ acts first.

Write

B(Ax) = (BA)x

:

A

first, then

B

.

Two maps to combine

turn 90°

stretch the width ×2

$R$ is the quarter turn from the last deck. $S$ doubles every first coordinate.

Turn, then stretch: $SR$

SR =

$e_1$ turns to $[0, 1]$ , and the stretch leaves it alone (its first coordinate is 0). $e_2$ turns to $[-1, 0]$ , then stretches to $[-2, 0]$ .

Stretch, then turn: $RS$

RS =

Now $e_1$ stretches to $[2, 0]$ first, then turns to $[0, 2]$ . A different map: $SR \ne RS$ . The same matrices in a different order do different things.

Write

SR \ne RS

: the order of maps matters.

✏️ Your turn

Compose

P = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}

R = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix}

Write the matrix for “shear with $P$ , then turn with $R$ ”. Which order do you multiply in?

Work it out on paper, then go to the next card to check.

Answer: $RP$

RP =

$P$ acts first, so it sits on the right: $RP$ . Check a column: $P$ sends $e_2$ to $[1, 1]$ , and $R$ turns that to $[-1, 1]$ , which is column 2 of $RP$ .

Stacked linear layers collapse

\displaystyle W_2(W_1 x) = (W_2 W_1)\,x

In a transformer

Two linear layers in a row are one linear layer: the product $W_2 W_1$ is a single matrix. A transformer’s MLP has a nonlinearity (GELU or SiLU) between its two matrices for exactly this reason. Without it, the extra matrix would add parameters but no new kinds of function.

Write

No nonlinearity between them: two layers = one matrix

W_2 W_1

.

In code, read left to right

y = x @ W1 @ W2        # row-vector layout: W1 acts first
y = x @ (W1 @ W2)      # the same map, one matrix

With row vectors the order flips: $xW_1W_2$ applies $W_1$ first. It’s the same composition, transposed: $(W_2^\top W_1^\top x^\top)^\top$ .

📌 Recap

Composition in one card

$B(Ax) = (BA)x$ : column layout reads right to left.
Order matters: $SR \ne RS$ .
Linear layers with nothing between them collapse into one matrix; activations prevent that.
Code layout $xW_1W_2$ reads left to right.

Write

The four lines above.