Vectors and embeddings

$v$ has 2 numbers, so it lives in 2 dimensions. We write $v \in \mathbb{R}^2$: $v$ is a point in the 2-dimensional real space.

Start at the origin, go 3 along the first axis and 1 along the second. The arrow and its tip, the point (3, 1), are two pictures of the same vector.

$u$ has 4 numbers, so $u \in \mathbb{R}^4$. You can’t draw 4 axes, but every rule in this course works the same way in any number of dimensions. Real models use hundreds or thousands: GPT-2 small has $d = 768$ and Llama 3 8B has $d = 4096$.

A model gives every token in its vocabulary one vector, its embedding, stored as a row of the table $E$. With $V$ tokens and width $d$, $E$ has shape $V \times d$. These toy numbers are small integers so you can copy them quickly.

“cat” is token 1, so its embedding is row 1 of $E$: $[0, 3, -1, 2]$. A lookup is just picking a row. In code, E[1].

“the cat sat” is the token ids [0, 1, 2]. Stack their rows and you have $X$, one row per token: shape $T \times d$. This is the matrix every later layer works on.

The token ids are [3, 0, 4]. Write $X$ and its shape.

Rows 3, 0 and 4 of $E$, in the order of the tokens. Order matters: the same tokens in another order give the same rows in another order.

emb = torch.nn.Embedding(num_embeddings=V, embedding_dim=d)
emb.weight.shape   # (V, d): the table E
X = emb(ids)       # ids of shape (T,) → X of shape (T, d)

• A vector in $\mathbb{R}^d$ is a list of $d$ numbers: an arrow, or a point.
• The embedding table $E$ is $V \times d$: one row per token.
• Embedding a token = picking its row. A sequence of $T$ tokens becomes $X$, shape $T \times d$.