The SVD

Any $m \times n$ matrix factors this way. $V^\top$ turns the input (an orthogonal matrix: rotation or reflection), $\Sigma$ stretches each axis by a singular value $\sigma_i \ge 0$, and $U$ turns the result into place. Unlike eigenvectors, the SVD always exists, for every shape.

$Ae_1 = [3, 4]$ and $Ae_2 = [0, 5]$. It’s not a rotation or a pure stretch. The SVD will split it into exactly those pieces.

$A^\top A = V\Sigma^2 V^\top$ is symmetric, so the last deck applies: its eigenvalues are the $\sigma_i^2$ and its eigenvectors are the columns of $V$. Here those are $[1, 1]/\sqrt2$ ($\lambda = 45$) and $[-1, 1]/\sqrt2$ ($\lambda = 5$).

Each right singular vector $v_i$ lands on $\sigma_i$ times a unit vector $u_i$. The $u_i$ come out orthogonal too: $(1)(-3) + (3)(1) = 0$.

$V^\top$ rotates by $-45^\circ$, $\Sigma$ stretches the axes by 6.708 and 2.236, $U$ rotates by about $71.6^\circ$. Multiplied back, they give $A$. Check: $\det A = 15 = \sigma_1 \sigma_2$.

• $\sigma_1$ is the most $A$ can stretch any unit vector: $\max_{|x| = 1} |Ax| = \sigma_1$, reached at $x = v_1$.
• $\sigma_{\min}$ is the least.
• The number of non-zero $\sigma_i$ is the rank.
• A unit circle maps to an ellipse with half-axes $\sigma_1 u_1$ and $\sigma_2 u_2$.

Compute $A^\top A$ and the singular values.

Singular values are never negative, even though the matrix has a $-3$: signs and swaps live in $U$ and $V$.

U, S, Vh = torch.linalg.svd(W, full_matrices=False)
# W ≈ U @ torch.diag(S) @ Vh;  S is sorted largest first

• $A = U\Sigma V^\top$ for every matrix: rotate, stretch, rotate.
• $\sigma_i^2$ = eigenvalues of $A^\top A$; $Av_i = \sigma_i u_i$.
• $\sigma_1$ = largest stretch; rank = number of non-zero $\sigma_i$.
• Example: $\begin{bmatrix} 3 & 0 \\ 4 & 5 \end{bmatrix}$ has $\sigma = \sqrt{45}, \sqrt5$.