Norms and cosine similarity

Pythagoras: square the entries, add, take the square root. This is the $L_2$ norm, written $|a|$ or $\|a\|_2$.

A neat way to remember it: $|a|^2 = a \cdot a$. The length squared is the vector dotted with itself.

$L_1$ adds the sizes of the entries: the distance walking along the grid instead of in a straight line. It shows up in regularization and sparsity; “norm” on its own almost always means $L_2$.

Divide a vector by its length and you get a unit vector: the same direction, length 1. Check: $0.6^2 + 0.8^2 = 0.36 + 0.64 = 1$. This is called normalizing.

Write $|a|$, then $\hat a$.

Every entry divided by 3. The direction didn’t change; only the length did.

Rearrange $a \cdot b = |a||b|\cos\theta$ and you get the cosine: how aligned two vectors are, from $-1$ (opposite) through $0$ (orthogonal) to $1$ (same direction), whatever their lengths.

With $q = [1, 0]$, the dot product prefers $k_1$ because it’s long. Cosine similarity prefers $k_2$ because it points almost exactly along $q$. Which one you want depends on whether length should count.

torch.linalg.vector_norm(a)          # |a|, the L2 norm
torch.linalg.vector_norm(a, ord=1)   # the L1 norm
F.normalize(a, dim=-1)               # a / |a|, along the last axis
F.cosine_similarity(a, b, dim=-1)    # a·b / (|a| |b|)

F is torch.nn.functional.

• $|a| = \sqrt{\textstyle\sum_i a_i^2}$, and $|a|^2 = a \cdot a$. $L_1$: $\sum_i |a_i|$.
• $\hat a = a / |a|$ is a unit vector: same direction, length 1.
• $\cos\theta = \frac{a \cdot b}{|a|\,|b|}$, from $-1$ to $1$, ignoring length.
• For unit vectors, $a \cdot b = \cos\theta$.