Tokens to vectors — Linear algebra for LLMs

Token ids

“the cat sat”→

A tokenizer turns text into integer ids, positions in a vocabulary of size $V$ . Our toy vocabulary from Part 1 has $V = 5$ : the, cat, sat, on, mat.

A token as a one-hot vector

e_{\text{cat}} =

A one-hot vector has a 1 in the token’s position and 0 everywhere else: length $V$ . It’s a way to write a token as a vector without giving it any meaning yet.

Write

One-hot: 1 at the token’s id, 0 elsewhere. Length

V

.

One-hot times E picks a row

=

Row view of $xE$ (Part 2): a mix of $E$ ’s rows weighted by $x$ . Only one weight is 1, so you get exactly row 1, cat’s embedding. An embedding lookup is a matrix product.

Write

e_{\text{id}}^\top E = E_{\text{id},:}

: one-hot × table = that row.

A whole sequence at once

=

Stack the one-hot rows into a $T \times V$ matrix and multiply: $(T \times V)(V \times d) = T \times d$ . That’s $X$ , the input to the first layer.

Code indexes instead

X = E[ids]                 # what the code does: pick rows
X = F.one_hot(ids, V).float() @ E   # the same numbers, the slow way

In a transformer

The product would cost $2TVd$ FLOPs and multiply almost entirely by zeros, so frameworks just index. But thinking of it as a product explains two things: the gradient reaches only the rows that were used, and the output layer can reuse $E$ transposed (weight tying, Part 6).

Where does position come in?

Attention compares every token with every other, and nothing in it knows which token came first: shuffle the input rows and the outputs shuffle the same way. Order has to be added.

GPT-2 adds a learned position vector to each embedding: $X = E[\text{ids}] + P[{:}T]$ .
Llama and most current LLMs add nothing to $X$ . They rotate the queries and keys instead (RoPE, Part 6).

Write

Attention alone ignores order. GPT-2: add

P

. Llama: rotate

q, k

(RoPE).

✏️ Your turn

Look up “sat”

Write the product.

Work it out on paper, then go to the next card to check.

Answer

Row 2 of $E$ : $[2, 1, 0, 1]$ .

📌 Recap

Tokens to vectors in one card

A token id becomes a one-hot vector of length $V$ .
One-hot × $E$ = the token’s row: a lookup is a matrix product.
A sequence: $(T \times V)(V \times d) = X\,(T \times d)$ .
Position comes from added vectors (GPT-2) or rotations (RoPE).

Write

The four lines above.