The dot product

The dot product takes two vectors with the same number of entries and gives back one number.

First with first, second with second, third with third: $1 \times 4 = 4$, $2 \times (-1) = -2$, $3 \times 2 = 6$.

That’s the whole definition: multiply matching entries, add the products.

Write the products, then the sum.

A zero entry contributes nothing, and a negative product pulls the total down.

Here $u \cdot w = 3 + 2 = 5$, $|u| = \sqrt{10}$ and $|w| = \sqrt{5}$, so $\cos\theta = 5 / \sqrt{50} \approx 0.707$: $\theta = 45^\circ$. The algebra (sum of products) and the geometry (lengths and angle) always agree.

Against $e = [2, 0]$: $[1, 2]$ gives $+2$, $[0, 2]$ gives $0$, $[-1, 2]$ gives $-2$.

When $\hat u$ has length 1, $a \cdot \hat u$ is the length of $a$’s shadow on $\hat u$’s direction: here $[2, 3] \cdot [1, 0] = 2$. For any $b$, the shadow length is $a \cdot b / |b|$.

a @ b              # dot product of two 1-D tensors
torch.dot(a, b)    # the same
(a * b).sum()      # the same, written out: multiply, then add
K @ q              # one dot product per row of K: all the scores

• $a \cdot b = \sum_i a_i b_i$: multiply matching entries, add. One number out.
• $a \cdot b = |a|\,|b| \cos\theta$. Positive: same way; zero: orthogonal; negative: opposite.
• With $|\hat u| = 1$, $a \cdot \hat u$ is the length of $a$’s shadow on $\hat u$.
• Attention scores are dot products between a query and each key.