Special matrices — Linear algebra for LLMs

The identity matrix

Ones on the diagonal, zeros everywhere else. $IA = AI = A$ for any $A$ of the right shape: it’s the “1” of matrix multiplication.

Write

I

: 1s on the diagonal, 0s elsewhere.

IA = AI = A

A diagonal matrix scales each feature

=

$D = \operatorname{diag}(2, 0.5, -1)$ . Multiplying by it scales feature 1 by 2, feature 2 by 0.5 and feature 3 by $-1$ . Nothing mixes between features.

…which is just elementwise multiplication

⊙

=

$x\,\operatorname{diag}(\gamma) = x \odot \gamma$ . Code never builds the $d \times d$ diagonal matrix; it multiplies entry by entry.

In a transformer

The learnable gain $\gamma$ in LayerNorm and RMSNorm is exactly this: one scale per feature (Part 6).

Write

x\,\operatorname{diag}(\gamma) = x \odot \gamma

(scale each feature).

$\odot$ and `@` are different products

⊙

=

@

=

$\odot$ (the Hadamard product) multiplies matching entries: $1 \cdot 5$ , $2 \cdot 6$ , $3 \cdot 7$ , $4 \cdot 8$ . @ is the matrix product: rows times columns. In PyTorch, * is $\odot$ and @ is the matrix product.

Write

A \odot B

(code A * B): entry by entry.

AB

(code A @ B): rows times columns.

✏️ Your turn

Both products

Write $R \odot S$ and $RS$ .

Work it out on paper, then go to the next card to check.

Answer

R \odot S =

RS =

With square matrices both run without an error, so mixing up * and @ gives wrong numbers silently. A shape check won’t catch it.

Triangular matrices

A lower-triangular matrix has only zeros above the diagonal. Read row $i$ as “token $i$ may look at tokens 1 to $i$ , never later ones”.

In a transformer

This is the shape of the causal mask in a decoder: token $i$ attends only to itself and earlier tokens. Part 5 builds it with $-\infty$ instead of 0.

torch.tril(torch.ones(4, 4))   # the matrix above

Write

Lower-triangular: zeros above the diagonal. Causal mask shape.

Block matrices multiply block by block

\displaystyle X \,\big[\, W_1 \;\big|\; W_2 \,\big] = \big[\, XW_1 \;\big|\; XW_2 \,\big]

Split a matrix into blocks and multiply the blocks as if they were numbers. Splitting $W$ ’s columns into two blocks splits the output’s columns the same way.

In a transformer

Multi-head attention uses this. $W_Q$ ( $d \times d$ ) is $h$ column blocks of width $d_h$ , one per head, so one big $XW_Q$ computes every head’s queries at once. Part 4 shows how to cut the result back into heads.

Write

X\,[\,W_1 \mid W_2\,] = [\,XW_1 \mid XW_2\,]

. One matmul, all heads.

📌 Recap

Special matrices in one card

$IA = AI = A$ .
$\operatorname{diag}(\gamma)$ scales each feature: $x\,\operatorname{diag}(\gamma) = x \odot \gamma$ .
$\odot$ (*) is entry by entry; @ is the matrix product. Square shapes hide the mix-up.
Lower-triangular = the causal mask’s shape.
Blocks multiply like numbers: $X\,[\,W_1 \mid W_2\,] = [\,XW_1 \mid XW_2\,]$ .

Write

The five lines above.