Glossary

Alphabetical. Each entry links to the deck that explains it.

Arithmetic intensity: FLOPs done per byte read from memory. Below the GPU’s ridge point an operation is memory-bound. Deck

Basis: a set of independent vectors whose combinations reach every vector in the space. Deck

BF16: 16-bit float with FP32’s 8 exponent bits and 7 mantissa bits: wide range, low precision. Deck

Broadcasting: stretching size-1 axes (aligned from the right) so tensors of different shapes combine elementwise. Deck

Causal mask: $-\infty$ added to scores above the diagonal so no token attends to later tokens. Deck

Condition number: $\sigma_1 / \sigma_n$, how much a matrix can amplify relative errors. Deck

Cosine similarity: the dot product of two vectors divided by both lengths; the cosine of the angle between them. Deck

Cross-entropy: $-\log p_y$, the loss on the correct token’s probability; its gradient on the logits is $p - \operatorname{onehot}(y)$. Deck

Determinant: the factor by which a square matrix scales area or volume; zero means it collapses a dimension. Deck

Dot product: $\sum_i a_i b_i$, one number measuring how aligned two vectors are. Deck

Eigenvector: a non-zero $v$ with $Av = \lambda v$: the matrix only stretches it, by the eigenvalue $\lambda$. Deck

einsum: a notation that names each axis and sums over the letters missing from the output. Deck

Embedding: the vector a token id looks up, one row of the embedding table. Deck

FLOP: one floating-point operation; a multiply-add counts as 2. Deck

FlashAttention: exact attention computed tile by tile with an online softmax, never storing the $T \times T$ matrix. Deck

GQA (grouped-query attention): several query heads share one key/value head, shrinking the KV cache. Deck

Gradient: the derivatives of the loss with respect to a parameter, shaped like the parameter. Deck

Head: one of $h$ independent attention computations, each on a $d_h$-wide slice. Deck

Jacobian: the matrix of every output’s derivative with respect to every input. Deck

KV cache: the keys and values of earlier tokens, stored so each new token computes only its own. Deck

LayerNorm / RMSNorm: per-token normalization; RMSNorm drops the mean-centering and the bias. Deck

Logits: the unnormalized scores over the vocabulary, one row per position. Deck

LoRA: fine-tuning a frozen $W_0$ by training a low-rank update $BA$. Deck

Null space: the vectors a matrix sends to zero. Deck

Outer product: $uv^\top$, a rank-1 matrix. Deck

PCA: finding the directions of greatest variance, the top right singular vectors of the centred data. Deck

Prefill / decode: processing the prompt all at once (compute-bound) vs. generating one token at a time (memory-bound). Deck

Projection: the closest point to $x$ in a subspace; onto a unit $u$, $(x \cdot u)\,u$. Deck

Quantization: storing numbers as scaled low-bit integers. Deck

Rank: the number of independent columns (or rows): the dimension of the output space. Deck

Residual stream: the running vector each block reads from and adds to. Deck

RoPE: rotary position embedding; rotates pairs of query and key entries by position-dependent angles. Deck

Singular value decomposition (SVD): $A = U\Sigma V^\top$, any matrix as rotate, scale, rotate. Deck

Softmax: exponentiate and normalize, turning scores into positive weights that sum to 1. Deck

SwiGLU: the gated MLP $(\operatorname{SiLU}(xW_g) \odot xW_u) W_d$. Deck

Tensor: an array with any number of axes; its shape lists their sizes. Deck

Transpose: swapping rows and columns, $(A^\top)_{ij} = A_{ji}$. Deck