Cheat sheet — Linear algebra for LLMs

Everything from the ten parts on one page. Each heading links to the part’s first deck.

Vectors

Dot product: $a \cdot b = \sum_i a_i b_i = \|a\|\,\|b\| \cos\theta$
Norm: $\|a\| = \sqrt{a \cdot a}$ ; cosine: $\cos\theta = \dfrac{a \cdot b}{\|a\|\,\|b\|}$
Random directions in $d$ dimensions: $\cos\theta \approx 0 \pm 1/\sqrt{d}$
Unit-variance entries: $q \cdot k$ has standard deviation $\sqrt{d_k}$

Shapes: $(m \times k)(k \times n) = m \times n$ ; inner sizes must match
$(AB)_{ij} = A_{i,:} \cdot B_{:,j}$ ; $AB$ = sum of $k$ outer products $A_{:,k} B_{k,:}$
$(AB)^\top = B^\top A^\top$ ; $AB \ne BA$ in general
Cost: $2mkn$ FLOPs; group products so the vectors stay vectors
* is elementwise ( $\odot$ ), @ is the matrix product

Column $j$ of $M$ is where $e_j$ lands
Rotation: $\begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}$ , $R^\top R = I$
$\det$ = area scale; $\det = 0$ ⇔ not invertible ⇔ a non-zero null space
$\operatorname{rank}(uv^\top) = 1$ ; $\operatorname{rank}(AB) \le \min(\operatorname{rank} A, \operatorname{rank} B)$
Projection onto unit $u$ : $(x \cdot u)\,u$ , matrix $uu^\top$

Broadcasting: align from the right; each axis equal or 1
view keeps memory order; split heads with view(B, T, h, d_h).transpose(1, 2)
Reductions remove the axis; keepdim=True keeps it as size 1
einsum('bhqd,bhkd->bhqk', q, k) = $QK^\top$ per batch and head

$\operatorname{softmax}(z)_i = e^{z_i - \max z} / \sum_j e^{z_j - \max z}$
$\operatorname{Attn} = \operatorname{softmax}\!\left(\dfrac{QK^\top}{\sqrt{d_k}} + \text{mask}\right) V$ ; mask $-\infty$ above the diagonal; softmax along rows
Heads: $h \cdot d_h = d$ ; GQA shares each K/V head across $h / h_{kv}$ query heads
KV cache: $2 \cdot L \cdot h_{kv} \cdot d_h \cdot \text{bytes}$ per token

nn.Linear(i, o): weight $(o, i)$ , $y = xW^\top + b$
Block: $x \mathrel{+}= \operatorname{Attn}(\operatorname{norm}(x))$ , then $x \mathrel{+}= \operatorname{MLP}(\operatorname{norm}(x))$
RMSNorm: $x / \operatorname{rms}(x) \odot g$ , no mean-centering, no bias
SwiGLU: $(\operatorname{SiLU}(xW_g) \odot xW_u) W_d$ , $3 d\, d_{ff}$ parameters
RoPE: rotate pair $i$ by $p\,\theta_i$ , $\theta_i = b^{-2i/d_h}$ ; scores depend on $p - q$
Classic block: $12d^2$ parameters; forward ≈ $2N$ FLOPs per token

$Av = \lambda v$ ; symmetric $\Rightarrow$ real $\lambda$ , orthogonal eigenvectors
SVD: $A = U \Sigma V^\top = \sum_i \sigma_i u_i v_i^\top$ ; $\sigma_i^2$ = eigenvalues of $A^\top A$
Best rank- $r$ : keep the top $r$ terms; error $\sqrt{\sigma_{r+1}^2 + \dots}$ (Frobenius)
LoRA: $W_0 + \tfrac{\alpha}{r} BA$ , $r(d_{\text{in}} + d_{\text{out}})$ parameters, $B = 0$ at init
Spectral norm $\|A\|_2 = \sigma_1$ ; condition number $\sigma_1 / \sigma_n$

A gradient has its parameter’s shape; descent: $w \leftarrow w - \eta\, \partial L/\partial w$
$Y = XW$ : $\partial L/\partial W = X^\top G$ , $\partial L/\partial X = GW^\top$ ; bias: sum $G$ ’s rows
Softmax + cross-entropy: $\partial \mathcal{L}/\partial z = p - \operatorname{onehot}(y)$
Training ≈ $6ND$ FLOPs; Adam in mixed precision ≈ 16 bytes per parameter

Bytes: FP32 4, BF16 2, FP8 and INT8 1, INT4 ½; BF16 has FP32’s range
Intensity $I$ = FLOPs / bytes; matrix × vector $I \approx 1$ , batch $B$ gives $I \approx B$
Decode ≤ bandwidth / bytes per token; prefill ≈ $2NT + 4LT^2 d$
INT8: $s = \max|x| / 127$ , $q = \operatorname{round}(x/s)$ , error $\le s/2$
Online softmax: $\ell' = \ell\, e^{m - m'} + \sum e^{z_j - m'}$