Will Ahmad

Linear algebra for LLMsPart 7: Decompositions and low rank

Eigenvectors

7 min

Directions that only stretch

Av=λv\displaystyle Av = \lambda v

Most vectors change direction when a matrix acts on them. An eigenvector vv doesn’t: it comes out as a multiple of itself. The factor λ\lambda is its eigenvalue: how much that direction is stretched (or flipped, if negative).

Write

Av=λvAv = \lambda v: eigenvector vv, eigenvalue λ\lambda.

Find them for a 2×2

S=[2112]\displaystyle S = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}
S[11]=[33]=3[11]S[1−1]=[1−1]\displaystyle S\begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 3 \\ 3 \end{bmatrix} = 3\begin{bmatrix} 1 \\ 1 \end{bmatrix} \qquad S\begin{bmatrix} 1 \\ -1 \end{bmatrix} = \begin{bmatrix} 1 \\ -1 \end{bmatrix}

[1,1][1, 1] is stretched by 3; [1,−1][1, -1] by 1 (unchanged). In general, the eigenvalues solve det⁡(S−λI)=0\det(S - \lambda I) = 0: here (2−λ)2−1=0(2 - \lambda)^2 - 1 = 0, so λ=3\lambda = 3 or 11.

Write

det⁡(A−λI)=0\det(A - \lambda I) = 0 gives the eigenvalues. SS: λ=3\lambda = 3 on [1,1][1, 1], λ=1\lambda = 1 on [1,−1][1, -1].

See it on the plane

Me₁Me₂v₁v₂

The grid is stretched 3× along the [1,1][1, 1] diagonal and left alone along [1,−1][1, -1]. Those two lines are the only directions that don’t turn.

Symmetric matrices are the nice case

S=S⊤  ⇒  S=QΛQ⊤\displaystyle S = S^\top \;\Rightarrow\; S = Q \Lambda Q^\top
Q=12[111−1],Λ=[3001]\displaystyle Q = \tfrac{1}{\sqrt 2}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}, \quad \Lambda = \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix}

The spectral theorem: a real symmetric matrix has real eigenvalues and orthogonal eigenvectors. Put the unit eigenvectors in the columns of QQ and the eigenvalues on the diagonal of Λ\Lambda: rotate into the eigenvector axes, stretch each axis, rotate back. Covariance matrices and X⊤XX^\top X are symmetric, so this always applies to them.

Write

Symmetric: S=QΛQ⊤S = Q\Lambda Q^\top, real λ\lambda, orthogonal eigenvectors.

Multiply again and again

Start anywhere and keep applying SS. The part along [1,1][1, 1] grows like 3k3^k, the rest like 1k1^k, so the direction (right column, normalized) swings toward [1,1]/2≈[0.707,0.707][1, 1]/\sqrt2 \approx [0.707, 0.707]. This is power iteration: the largest eigenvalue’s direction wins.

Write

SkxS^k x → direction of the largest ∣λ∣|\lambda|, size ∼∣λmax⁡∣k\sim |\lambda_{\max}|^k.

Why deep products explode or vanish

In a transformer

Pushing a signal (or a gradient) through many layers multiplies it by many matrices. If the important eigenvalues are above 1, it grows exponentially; below 1, it shrinks to nothing. Residual connections, normalization and careful initialization all exist to keep these products near 1.

✏️ Your turn

Find the eigenvalues

What are the eigenvalues and eigenvectors of BB? (Look before you compute.)

Work it out on paper, then go to the next card to check.

Answer

λ=4 on [1,0],λ=−2 on [0,1]\displaystyle \lambda = 4 \text{ on } [1, 0], \qquad \lambda = -2 \text{ on } [0, 1]

A diagonal matrix’s eigenvectors are the axes and its eigenvalues are the diagonal. The negative one flips e2e_2 as well as doubling it.

Where eigendecomposition stops

Only square matrices have eigenvalues, and a non-symmetric one may have complex eigenvalues (a rotation turns every direction) or too few eigenvectors (a shear has only one direction). Weight matrices are usually not square at all. The SVD, next, works for every matrix.

torch.linalg.eigh(S)    # symmetric: real eigenvalues, orthonormal eigenvectors
torch.linalg.eig(A)     # general square: may be complex

📌 Recap

Eigenvectors in one card

  • Av=λvAv = \lambda v: a direction that only stretches; det⁡(A−λI)=0\det(A - \lambda I) = 0.
  • Symmetric: S=QΛQ⊤S = Q\Lambda Q^\top with orthogonal eigenvectors.
  • Repeated multiplication: the largest ∣λ∣|\lambda| dominates; products grow or shrink like ∣λ∣k|\lambda|^k.
  • Non-square or non-symmetric matrices need the SVD.

Write

The four lines above.