Matrix × vector — Linear algebra for LLMs

A matrix times a vector

=

$A$ is $2 \times 3$ and $x$ is a column of 3 numbers. The result has one entry per row of $A$ : 2 entries.

Write

A\,(2 \times 3)

times

x\,(3 \times 1)

gives

y\,(2 \times 1)

.

Row view: entry 1

=

Entry 1 of $y$ is row 1 of $A$ dotted with $x$ : $1 \cdot 2 + 2 \cdot 1 + 0 \cdot 4 = 4$ .

Row view: entry 2

=

Entry 2 is row 2 dotted with $x$ : $3 \cdot 2 + (-1) \cdot 1 + 1 \cdot 4 = 9$ . One dot product per row.

Write

Row view:

y_i = (\text{row } i \text{ of } A) \cdot x

.

y = [4, 9]

The shape rule

\displaystyle (2 \times \underline{3})\,(\underline{3} \times 1) = (2 \times 1)

Each row of $A$ must be as long as $x$ , so the inner sizes have to match. The outer sizes are what’s left: one output per row of $A$ .

Column view: a mix of $A$ 's columns

2

+

1

+

4

=

Now read it by columns. $x$ ’s entries say how much of each column of $A$ to take: 2 of column 1, 1 of column 2, 4 of column 3. $[2, 6] + [2, -1] + [0, 4] = [4, 9]$ . The same answer.

Write

Column view:

Ax = x_1 (\text{col } 1) + x_2 (\text{col } 2) + x_3 (\text{col } 3)

Two views, one product

View	$Ax$ is	Good for
Rows	a list of dot products, one per row	computing by hand; “each output scores the input”
Columns	a mix of $A$ ’s columns, weighted by $x$	seeing what outputs are possible at all

Every output $Ax$ is some combination of $A$ ’s columns. Those combinations are $A$ ’s column space (Part 3).

✏️ Your turn

Multiply

Write the product, using the row view. Then check it with the column view.

Work it out on paper, then go to the next card to check.

Answer

=

1

+

2

=

Rows: $2 \cdot 1 + 0 \cdot 2 = 2$ , $1 \cdot 1 + 1 \cdot 2 = 3$ , $0 \cdot 1 + 3 \cdot 2 = 6$ . Columns: 1 of the first column plus 2 of the second. A $3 \times 2$ matrix turns 2 numbers into 3.

The code layout: a row vector times a matrix

=

2

+

1

With the vector on the left as a row, everything flips: $xW$ is a mix of $W$ ’s rows, weighted by $x$ ’s entries. $2\,[1, 0, 2] + 1\,[3, 1, -1] = [5, 1, 3]$ . Entry $j$ is also $x$ dotted with column $j$ of $W$ .

Write

Row vector:

xW = x_1 (\text{row } 1 \text{ of } W) + x_2 (\text{row } 2 \text{ of } W) + \dots

What a linear layer does to one token

In a transformer

For one token $x\,(1 \times d_{\text{in}})$ and a weight $W\,(d_{\text{in}} \times d_{\text{out}})$ , the layer outputs $xW\,(1 \times d_{\text{out}})$ . Each output feature $j$ is $x \cdot W_{:,j}$ : column $j$ of $W$ acts as a detector, scoring how much $x$ points its way. One head’s slice of $W_Q$ is $4096 \times 128$ in a Llama 3 8B-sized model.

y = x @ W        # x: (d_in,), W: (d_in, d_out) → y: (d_out,)

📌 Recap

Matrix × vector in one card

Shapes: $(m \times n)(n \times 1) = (m \times 1)$ . Inner sizes match.
Row view: each output is a row of $A$ dotted with $x$ .
Column view: $Ax$ mixes $A$ ’s columns, weighted by $x$ .
Code layout: $xW$ mixes $W$ ’s rows; output $j$ = $x \cdot W_{:,j}$ .

Write

The four lines above.