Matrix norms and conditioning — Linear algebra for LLMs

Frobenius norm: all the entries

\displaystyle \|A\|_F = \sqrt{\textstyle\sum_{ij} a_{ij}^2} = \sqrt{\textstyle\sum_i \sigma_i^2}

\displaystyle \left\| \begin{bmatrix} 3 & 0 \\ 4 & 5 \end{bmatrix} \right\|_F = \sqrt{9 + 0 + 16 + 25} = \sqrt{50} \approx 7.07

Treat the matrix as one long vector and take its length. It also equals the length of the singular-value list: $\sqrt{45 + 5}$ . Weight decay penalizes this (squared).

Write

\|A\|_F = \sqrt{\sum a_{ij}^2} = \sqrt{\sum \sigma_i^2}

.

Spectral norm: the largest stretch

\displaystyle \|A\|_2 = \max_{|x| = 1} |Ax| = \sigma_1

\displaystyle \|A\|_2 = \sqrt{45} \approx 6.71

How much can the matrix lengthen a vector at most? Exactly $\sigma_1$ , reached along $v_1$ . It bounds the output: $|Ax| \le \sigma_1 |x|$ for every $x$ .

Write

\|A\|_2 = \sigma_1

, so

|Ax| \le \sigma_1|x|

.

Condition number

\displaystyle \kappa(A) = \frac{\sigma_{\max}}{\sigma_{\min}} = \frac{\sqrt{45}}{\sqrt5} = 3

The ratio of the largest to the smallest stretch. When $\kappa$ is large, the matrix nearly collapses some direction, and solving $Ax = b$ can turn a tiny error in $b$ into a large error in $x$ . Orthogonal matrices have $\kappa = 1$ , the best possible.

Write

\kappa = \sigma_{\max}/\sigma_{\min}

. Large

\kappa

: errors amplified.

✏️ Your turn

Compute the norms

Its singular values are 3 and 2 (last deck but one). Write the Frobenius norm, the spectral norm and the condition number.

Work it out on paper, then go to the next card to check.

Answer

\displaystyle \|A\|_F = \sqrt{4 + 9} = \sqrt{13} \approx 3.61

\displaystyle \|A\|_2 = 3, \qquad \kappa = 3/2 = 1.5

Why it matters in training

In a transformer

A layer’s spectral norm limits how much it can amplify its input, so stacks of layers with large $\sigma_1$ blow activations up. Some methods control it directly: spectral normalization divides a weight by $\sigma_1$ , and some recent optimizers (Muon, for example) replace an update matrix by its orthogonal factor $UV^\top$ , whose singular values are all 1. Mixed-precision training also watches conditioning, since BF16 has few digits to spare (Part 9).

torch.linalg.matrix_norm(A)            # Frobenius
torch.linalg.matrix_norm(A, ord=2)     # spectral: the largest singular value
torch.linalg.cond(A)                   # condition number

📌 Recap

Matrix norms in one card

Frobenius: $\sqrt{\sum a_{ij}^2} = \sqrt{\sum\sigma_i^2}$ .
Spectral: $\sigma_1$ , the largest stretch; $|Ax| \le \sigma_1|x|$ .
Condition number $\kappa = \sigma_{\max}/\sigma_{\min}$ ; orthogonal matrices have $\kappa = 1$ .

Write

The three lines above.