Q, K and V — Linear algebra for LLMs

Three projections of the same X

\displaystyle Q = XW_Q \qquad K = XW_K \qquad V = XW_V

\displaystyle (T \times d)(d \times d_k) = T \times d_k

Every token’s row is projected three times, with three different weights. Each result has one row per token. For the rest of Part 5 we follow one head with $T = 3$ , $d = 4$ and $d_k = 2$ .

Write

Q = XW_Q

,

K = XW_K

,

V = XW_V

: each

T \times d_k

.

The input

Three tokens, four features each. Small 0/1 entries so every product below is quick to write.

Write

X = \begin{bmatrix} 1 & 0 & 1 & 0 \\ 0 & 1 & 0 & 1 \\ 1 & 1 & 0 & 0 \end{bmatrix}

Queries: $Q = XW_Q$

=

Row “the” is $[1, 0, 1, 0]\,W_Q$ : rows 1 and 3 of $W_Q$ added, $[1, 0] + [1, 0] = [2, 0]$ .

Write

Q = \begin{bmatrix} 2 & 0 \\ 0 & 2 \\ 1 & 1 \end{bmatrix}

Keys: $K = XW_K$

=

Row “cat” picks rows 2 and 4 of $W_K$ : $[0, 1] + [1, 1] = [1, 2]$ .

Write

K = \begin{bmatrix} 1 & 0 \\ 1 & 2 \\ 1 & 1 \end{bmatrix}

Values: $V = XW_V$

=

Write

V = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 1 & 0 \end{bmatrix}

What each one is for

Matrix	Row $i$ is token $i$ ’s…	Used to
$Q$	query: what it is looking for	score against every key
$K$	key: what it can be found by	be scored by every query
$V$	value: what it passes on	be mixed into the outputs

Queries and keys only ever meet in dot products, so they must have the same width $d_k$ . Values can have their own width $d_v$ (usually equal).

Write

Q, K

: matched by dot products.

V

: what gets mixed.

✏️ Your turn

One row by hand

Write the query for “sat”.

Work it out on paper, then go to the next card to check.

Answer: $[1, 1]$

\displaystyle [1, 1, 0, 0]\,W_Q = [1, 0] + [0, 1] = [1, 1]

Rows 1 and 2 of $W_Q$ , added. It matches row 3 of $Q$ .

In PyTorch

q_proj = nn.Linear(d, d_k, bias=False)   # weight (d_k, d): out, in
q = q_proj(x)                            # x @ q_proj.weight.T

In a transformer

Llama’s attention projections have no bias, so each one is a single matrix multiply. In a real model one q_proj computes every head’s queries at once (Part 2’s block matrices); the next decks use one head to keep the numbers small.

📌 Recap

Q, K and V in one card

$Q = XW_Q$ , $K = XW_K$ , $V = XW_V$ , each $T \times d_k$ .
Queries and keys are compared; values are mixed.
Our example: $Q = [2\ 0;\ 0\ 2;\ 1\ 1]$ , $K = [1\ 0;\ 1\ 2;\ 1\ 1]$ , $V = [1\ 1;\ 0\ 1;\ 1\ 0]$ .

Write

The three lines above, with the example matrices.