Four ways to see AB — Linear algebra for LLMs

One product, four readings

=

This is the product from the last deck. The numbers won’t change in this deck; only how you read them. Each reading explains a different part of a transformer.

We’ll write $A_{i,:}$ for row $i$ of $A$ and $A_{:,j}$ for column $j$ , the same slicing as code: A[i, :] and A[:, j].

Write

C = AB

. Row

i

:

A_{i,:}

. Column

j

:

A_{:,j}

.

1. Every entry is a dot product

=

$C_{ij} = A_{i,:} \cdot B_{:,j}$ : row $i$ of $A$ dotted with column $j$ of $B$ . This is the view you compute with.

Write

C_{ij} = A_{i,:} \cdot B_{:,j}

Dot products of every pair: $XX^\top$

=

With $B = X^\top$ , the columns of $B$ are the rows of $X$ , so entry $(i, j)$ of $XX^\top$ is $x_i \cdot x_j$ : every pair of rows, dotted. It’s called a Gram matrix, and it’s symmetric because $x_i \cdot x_j = x_j \cdot x_i$ .

In a transformer

$QK^\top$ is the same idea with two different matrices: every query dotted with every key.

Write

(XX^\top)_{ij} = x_i \cdot x_j

. Symmetric.

2. Each column of $C$ mixes $A$ 's columns

2

+

1

+

4

=

Column $j$ of $AB$ is $A$ times column $j$ of $B$ . Column 1 of $B$ is $[2, 1, 4]$ , so column 1 of $C$ is $2A_{:,1} + A_{:,2} + 4A_{:,3} = [4, 9]$ . It’s matrix × vector, once per column.

Write

C_{:,j} = A\,B_{:,j}

: a mix of

A

’s columns.

3. Each row of $C$ mixes $B$ 's rows

1

+

2

+

0

=

Row $i$ of $AB$ is row $i$ of $A$ times $B$ . Row 1 of $A$ is $[1, 2, 0]$ , so row 1 of $C$ is $1\,[2, 1] + 2\,[1, 0] + 0\,[4, -1] = [4, 1]$ .

In a transformer

This is the token view. In $XW$ , each output row depends only on the same row of $X$ : a linear layer treats every token independently, with the same weights.

Write

C_{i,:} = A_{i,:}\,B

: a mix of

B

’s rows.

4. A sum of outer products: the first piece

=

A column times a row is an outer product $uv^\top$ : every entry of $u$ times every entry of $v$ . Here column 1 of $A$ times row 1 of $B$ . Every row of the result is a multiple of $[2, 1]$ .

Write

Outer product:

(uv^\top)_{ij} = u_i v_j

The second and third pieces

=

Column 2 of $A$ with row 2 of $B$ , column 3 of $A$ with row 3 of $B$ . There’s one piece for each value of the inner index $k$ .

Add the pieces

+

=

$AB = \sum_k A_{:,k}\,B_{k,:}$ : column $k$ of $A$ times row $k$ of $B$ , added up over $k$ . The same $\begin{bmatrix} 4 & 1 \\ 9 & 2 \end{bmatrix}$ .

Write

AB = \sum_k A_{:,k}\,B_{k,:}

: one outer product per value of

k

.

Why the outer-product view matters

Every outer product has rank 1: all its rows are multiples of one vector. So a product with inner size $k$ is a sum of $k$ rank-1 pieces, and has rank at most $k$ .

In a transformer

LoRA (Part 7) adds $BA$ to a weight, with a small inner size $r$ : a sum of $r$ rank-1 pieces, cheap to train. And an attention head’s output $AV$ (Part 5) is one outer product per position: $AV = \sum_j A_{:,j}\,V_{j,:}$ .

✏️ Your turn

Outer product

Write the outer product and its shape.

Work it out on paper, then go to the next card to check.

Answer

=

$(2 \times 1)(1 \times 3) = 2 \times 3$ . Row 2 is twice row 1: rank 1.

📌 Recap

Four ways to see $AB$ in one card

View	$C = AB$	Shows up as
Dot products	$C_{ij} = A_{i,:} \cdot B_{:,j}$	$QK^\top$ , Gram matrices
Columns	$C_{:,j} = A\,B_{:,j}$	the column space (Part 3)
Rows	$C_{i,:} = A_{i,:}\,B$	$XW$ : every token on its own
Outer products	$C = \sum_k A_{:,k}\,B_{k,:}$	rank, LoRA (Part 7)

Write

The four views in the table.