Span, basis and rank

The span of some vectors is every linear combination of them. $a = [1, 2]$ and $b = [2, -1]$ point different ways, so $c_1 a + c_2 b$ reaches every point of the plane.

$c = -1.5\,a$, so mixing $a$ and $c$ never leaves the dashed line. The second vector added nothing new.

Vectors are linearly independent when none of them is a mix of the others. A basis is an independent set that spans the whole space. Every basis of $\mathbb{R}^d$ has exactly $d$ vectors; that number is the dimension.

In $\mathbb{R}^2$:

• $[1, 2]$ and $[2, -1]$: independent, span the plane. A basis.
• $[1, 2]$ and $[-1.5, -3]$: dependent, span a line.
• $[1, 0]$, $[0, 1]$ and $[1, 1]$: dependent (three vectors in 2D), span the plane.

Every output $Ax$ is a mix of $A$’s columns, so the outputs form the span of the columns: the column space. Its dimension is the rank: the number of independent columns. (It always equals the number of independent rows too.)

Row 2 is twice row 1, and every column is a multiple of $[1, 2]$. Three columns, but only one direction: rank 1. Every rank-1 matrix is an outer product $uv^\top$, and every output $Ax$ lands on the line through $[1, 2]$.

An $m \times n$ matrix can’t have more independent columns than it has columns, or rows. And a product can’t have more rank than either factor: whatever passes through a narrow middle can’t widen again. A product $(m \times k)(k \times n)$ has rank at most $k$.

What is $\operatorname{rank}(B)$? Look for a row that’s a mix of the others.

Row 3 = row 1 + row 2, so only two rows are independent. Rows 1 and 2 aren’t multiples of each other, so the rank is exactly 2. The same holds for columns: column 3 = column 1 + column 2.

torch.linalg.matrix_rank(A)      # rank, from the singular values
torch.outer(u, v)                # a rank-1 matrix u vᵀ

Real weight matrices are almost never exactly rank-deficient; what matters in practice is how much of them is captured by a few directions. Part 7 measures that with the SVD.

• Span: all combinations. Independent: none is a combination of the others. Basis: both.
• Column space = all outputs $Ax$; rank = its dimension.
• Rank 1 ⇔ outer product $uv^\top$.
• $\operatorname{rank}(AB) \le \min(\operatorname{rank} A, \operatorname{rank} B)$: one head’s $W_Q W_K^\top$ has rank $\le d_h$.