Linear maps — Linear algebra for LLMs

A matrix is a function

\displaystyle x \;\mapsto\; Mx

Feed a vector in, get a vector out. In this part we draw the maps the way geometry books do, with column vectors and $Mx$ . Code does the same thing with rows and $xW$ ; only the layout is transposed.

Write

A matrix

M

is the function

x \mapsto Mx

.

Watch where the basis vectors go

$Me_1 = [2, 0]$ is column 1 of $M$ (blue). $Me_2 = [1, 1]$ is column 2 (pink). The whole grid follows them: every square becomes the same slanted parallelogram.

Write

Column

j

of

M

= where

e_j

lands:

Me_j = M_{:,j}

.

Linear means: follow the basis vectors

\displaystyle M(a + b) = Ma + Mb

\displaystyle M(c\,x) = c\,Mx

\displaystyle M\begin{bmatrix} 3 \\ 1 \end{bmatrix} = 3\,Me_1 + 1\,Me_2

\displaystyle = 3\begin{bmatrix} 2 \\ 0 \end{bmatrix} + \begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 7 \\ 1 \end{bmatrix}

A linear map keeps sums and scalings. So once you know where $e_1$ and $e_2$ go, you know where everything goes: $x = x_1 e_1 + x_2 e_2$ lands on $x_1 Me_1 + x_2 Me_2$ . That’s the column view of $Mx$ from Part 2.

Write

Linear:

M(a + b) = Ma + Mb

,

M(cx) = c\,Mx

. So

Mx = x_1 Me_1 + x_2 Me_2

.

Scale

A diagonal matrix stretches each axis by its own factor: twice as wide, half as tall. No direction mixes with another.

Shear

$e_1$ stays where it is; $e_2$ leans over to $[1, 1]$ . Each point slides sideways by its height. Areas don’t change.

Reflect

Flipping the sign of one column mirrors the plane: here across the horizontal axis. The plane comes out “inside out”, which the determinant will report as a minus sign.

Project

$e_2$ lands on $0$ , so the whole plane collapses onto the horizontal axis. Every point keeps its first coordinate and loses its second, for good. A map that loses information can’t be undone.

✏️ Your turn

What does this matrix do?

Write $Re_1$ and $Re_2$ , then describe the map in words.

Work it out on paper, then go to the next card to check.

Answer: a quarter turn

$Re_1 = [0, 1]$ and $Re_2 = [-1, 0]$ : both basis vectors turn 90° anticlockwise, so everything does. The next deck but one is all about rotations.

Every layer is a linear map

In a transformer

A weight $W$ of shape $d_{\text{in}} \times d_{\text{out}}$ maps one feature space to another, exactly like these 2D pictures but with thousands of axes. The interesting part of a transformer is what each map keeps, mixes or throws away.

y = x @ W                 # the same map, code layout
y = (W.T @ x.T).T         # the math layout, transposed back

📌 Recap

Linear maps in one card

A matrix is a function $x \mapsto Mx$ .
Column $j$ of $M$ is where $e_j$ lands; linearity gives everything else.
Diagonal: scale. Off-diagonal: shear. Negative column: reflect. Zero column: project and lose information.

Write

The three lines above.