High dimensions — Linear algebra for LLMs

In 2D, a random angle can be anything

Pick two random directions in the plane and the angle between them is anywhere from 0° to 180°. Their cosine is often close to $1$ or $-1$ . In thousands of dimensions, something very different happens.

In high dimensions, random vectors are nearly orthogonal

$d$	typical $\cos\theta$ of two random vectors
2	$\pm 0.71$
64	$\pm 0.13$
768	$\pm 0.036$
4,096	$\pm 0.016$

The cosine of two random directions has an average of $0$ and a spread (standard deviation) of $1/\sqrt{d}$ . At $d = 4096$ two random embeddings are almost exactly at 90°.

Write

Random vectors in

\mathbb{R}^d

:

\cos\theta \approx 0 \pm 1/\sqrt{d}

. Nearly orthogonal.

Why: the products cancel

\displaystyle a \cdot b = a_1 b_1 + a_2 b_2 + \dots + a_d b_d

Each product $a_i b_i$ is as likely to be negative as positive, so most of them cancel. What’s left grows only like $\sqrt{d}$ , while the lengths $|a|$ and $|b|$ each grow like $\sqrt{d}$ too. Divide and you get about $\sqrt{d} / d = 1/\sqrt{d}$ .

The size of a random dot product

\displaystyle q_i, k_i\!: \text{ mean } 0, \text{ variance } 1

\displaystyle \operatorname{Var}(q \cdot k) = d

\displaystyle \text{typical } |q \cdot k| \approx \sqrt{d}

A sum of $d$ independent terms, each with variance 1, has variance $d$ . Its typical size, the standard deviation, is $\sqrt{d}$ : 8 for $d = 64$ , about 11.3 for $d = 128$ .

Write

Random

q, k

with unit-variance entries:

\operatorname{Var}(q \cdot k) = d

, size

\approx \sqrt{d}

.

✏️ Your turn

How big?

\displaystyle d_k = 100

Query and key entries are random with mean 0 and variance 1. Roughly how big is a typical $q \cdot k$ ?

Work it out on paper, then go to the next card to check.

Answer: about 10

\displaystyle \sqrt{100} = 10

Scores of $\pm 10$ are a problem, as the next card shows.

Big scores make softmax pick one winner

\operatorname{softmax}

=

\operatorname{softmax}

=

Softmax (Part 5) turns scores into weights that sum to 1. With scores around $\pm 10$ , almost all the weight lands on one key, and the gradients for the others nearly vanish. Divided by 10, the same scores give a useful spread.

In a transformer

This is why attention computes $QK^\top / \sqrt{d_k}$ : dividing by $\sqrt{d_k}$ brings random scores back to a typical size of about 1, whatever the head width.

Write

Attention divides scores by

\sqrt{d_k}

so they stay

\approx 1

in size.

Room for many more directions than dimensions

Directions at…	How many fit in $\mathbb{R}^d$
exactly 90°	at most $d$
nearly 90°	exponentially many in $d$

$\mathbb{R}^d$ has room for only $d$ directions at exactly 90° to each other. But if “nearly 90°” is enough (say $|\cos\theta| < 0.1$ for every pair), the number of directions you can fit grows exponentially with $d$ .

In a transformer

One explanation of how models store more concepts than they have dimensions, called superposition, relies on this: features get nearly orthogonal directions, so they interfere only a little (Anthropic, Toy Models of Superposition, 2022).

Try it in PyTorch

a, b = torch.randn(2, 4096)          # two random vectors, entries ~ N(0, 1)
F.cosine_similarity(a, b, dim=0)     # typically within ±0.03
(a @ b) / 4096 ** 0.5                # typically between −2 and 2

Run it a few times: the cosine hugs 0, and the scaled dot product behaves like a draw from $\mathcal{N}(0, 1)$ .

📌 Recap

High dimensions in one card

Two random vectors in $\mathbb{R}^d$ are nearly orthogonal: $\cos\theta \approx 0 \pm 1/\sqrt{d}$ .
With unit-variance entries, $q \cdot k$ has variance $d$ and typical size $\sqrt{d}$ .
Big scores make softmax put nearly all the weight on one key, so attention divides by $\sqrt{d_k}$ .
Exactly orthogonal: $d$ directions. Nearly orthogonal: exponentially many.

Write

The four lines above.