Add, scale, combine — Linear algebra for LLMs

Adding vectors

+

=

Add entry by entry: first with first ( $3 + 1 = 4$ ), second with second ( $1 + 2 = 3$ ). Both vectors must have the same length.

Write

a + b = [3, 1] + [1, 2] = [4, 3]

Adding is putting arrows tip to tail

Draw $a$ , then draw $b$ starting where $a$ ends. $a + b$ goes from the start of $a$ to the end of $b$ : the point $(4, 3)$ .

Scaling a vector

Multiplying by a number scales every entry: $2a = [6, 2]$ is twice as long, $-a = [-3, -1]$ points the opposite way, $0.5a$ is half as long. The direction only changes when the number is negative.

Write

2a = [6, 2]

,

-a = [-3, -1]

A linear combination

2a - b

=

−

=

Scale some vectors, then add them up: that’s a linear combination. Almost everything in this course is one: a matrix times a vector, attention’s weighted average, the residual stream.

Write

A linear combination:

c_1 v_1 + c_2 v_2 + \dots

(scale, then add).

2a - b = [5, 0]

Every vector is a combination of the axes

=

3

+

1

$e_1$ and $e_2$ are the standard basis: one 1, the rest 0. A vector’s entries are exactly how much of each basis vector it contains.

✏️ Your turn

Combine these

3

−

2

Write the result.

Work it out on paper, then go to the next card to check.

Answer

−

=

Scale first (3 times each entry, 2 times each entry), then subtract entry by entry.

The residual stream is a running sum

embedding+

attention writes+

MLP writes=

In a transformer

Each layer of a transformer reads the current vector $x$ , computes something, and adds it back: $x \leftarrow x + f(x)$ . The final vector is the embedding plus every layer’s contribution. This sum is called the residual stream (Part 6).

Write

x \leftarrow x + f(x)

. Final

x

= embedding + all the layers’ outputs.

Averaging vectors

→

The mean adds the rows and divides by how many there are: $[3, 4, 1, 2] / 3$ . Many sentence-embedding models get one vector for a whole text this way, called mean pooling.

Write

\bar x = \frac{1}{T} \sum_{t=1}^{T} x_t

: the sum of the rows over

T

.

In PyTorch

a + b            # elementwise sum (shapes must match, or broadcast)
2 * a            # scale every entry
2 * a - b        # a linear combination
X.mean(dim=0)    # average the rows of X: (T, d) → (d,)

dim=0 means “collapse axis 0”, the rows. Part 4 covers axes properly.

📌 Recap

Add, scale, combine in one card

$a + b$ adds entry by entry; as arrows, tip to tail.
$ca$ scales every entry; a negative $c$ flips the direction.
A linear combination is $c_1 v_1 + c_2 v_2 + \dots$ : scale, then add.
The residual stream: $x \leftarrow x + f(x)$ , a running sum of every layer’s output.
Mean pooling: $\bar x = \frac{1}{T} \sum_t x_t$ .

Write

The five lines above.