Inverse, determinant, null space — Linear algebra for LLMs

The determinant is an area factor

\displaystyle \det \begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bc

\displaystyle \det \begin{bmatrix} 2 & 1 \\ 0 & 1 \end{bmatrix} = 2 \cdot 1 - 1 \cdot 0 = 2

The unit square maps to a parallelogram whose area is $|\det|$ . The shear-and-stretch from the first deck of this part doubles every area. A negative determinant means the map also flips the plane over (the reflection had $\det = -1$ ).

Write

\det = ad - bc

= area scale factor (sign: flipped or not).

Determinant zero: the plane collapses

$1 \cdot 4 - 2 \cdot 2 = 0$ . Both columns lie on one line, so the whole plane lands on that line: area 0. Rank 1, not 2.

Write

\det A = 0

⇔ columns dependent ⇔ rank < n.

Null space: what gets destroyed

=

The null space is every $x$ with $Ax = 0$ . Here it’s the line through $[2, -1]$ . Anything along that direction is wiped out, so two inputs that differ by it give the same output. Rank + nullity = number of columns: $1 + 1 = 2$ .

Write

Null space:

\{x : Ax = 0\}

. rank + dim(null space) = n.

The inverse undoes the map

\displaystyle A^{-1} A = A A^{-1} = I

\displaystyle \begin{bmatrix} a & b \\ c & d \end{bmatrix}^{-1} = \frac{1}{ad - bc} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix}

Swap the diagonal, negate the off-diagonal, divide by the determinant. That division is why $\det = 0$ means no inverse: a collapsed plane can’t be un-collapsed.

Write

2×2 inverse: swap

a, d

; negate

b, c

; divide by

ad - bc

.

Check an inverse

=

$\det A = 2 - 1 = 1$ , so $A^{-1} = \begin{bmatrix} 1 & -1 \\ -1 & 2 \end{bmatrix}$ . Multiplying back gives the identity.

✏️ Your turn

Invert this

Write $\det B$ and $B^{-1}$ , then check that $BB^{-1} = I$ .

Work it out on paper, then go to the next card to check.

Answer

\det B = 3 - 2 = 1, \quad B^{-1} =

Check row 1 of $B$ against the columns: $3 \cdot 1 + 1 \cdot (-2) = 1$ and $3 \cdot (-1) + 1 \cdot 3 = 0$ .

Orthogonal matrices invert for free

\displaystyle Q^\top Q = I \;\Rightarrow\; Q^{-1} = Q^\top

Rotations and reflections keep lengths, so undoing them is just a transpose: no division, no rounding trouble. Their determinant is $\pm 1$ .

In practice

In a transformer

You rarely invert a matrix in ML work. Most weights aren’t square (a $4096 \times 14336$ up-projection has no inverse), and when you do need to solve $Ax = b$ , torch.linalg.solve is faster and more accurate than forming $A^{-1}$ . What carries over is the idea: a map with a null space throws information away for good.

torch.linalg.det(A)
torch.linalg.solve(A, b)     # x with A x = b, without forming A⁻¹
torch.linalg.pinv(W)         # pseudo-inverse, for non-square W

📌 Recap

Inverse, determinant, null space in one card

$\det \begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bc$ : the area factor.
$\det = 0$ ⇔ rank < n ⇔ a null space ⇔ no inverse.
$A^{-1} = \frac{1}{ad - bc}\begin{bmatrix} d & -b \\ -c & a \end{bmatrix}$ for 2×2.
Orthogonal: $Q^{-1} = Q^\top$ .

Write

The four lines above.