Quantization

Symmetric INT8 quantization: pick one scale so the largest value lands on $\pm 127$, store the rounded integers plus the scale, and multiply back when needed.

The largest magnitude, 1.27, sets $s = 0.01$. Every value is stored as a whole number of hundredths; 0.031 becomes 3, so it comes back as 0.03. Rounding never moves a value by more than $s/2$.

With levels $-7 \dots 7$ the steps are 18× coarser: 0.42 comes back as 0.363, and 0.031 vanishes to 0. INT4 halves the memory again, at a real cost in accuracy unless it’s done carefully.

The 10 forces a big step, so the small values get one to three levels each: 0.1 comes back as 0.079, a 21% error. LLM activations and some weights do have such outliers.

Give each output channel, or each group of 128 consecutive weights, its own scale. An outlier then only coarsens its own group. A 16-bit scale per 128 four-bit weights adds an eighth of a bit per weight.

With one scale per output row, the dot product can run on the integers and be rescaled once at the end. That’s how quantized kernels avoid converting every weight back to floating point first.

Use $s = \max|x|/7$. Write $s$, $q$ and $\hat x$.

$0.62/0.2 = 3.1 \to 3$ and $0.25/0.2 = 1.25 \to 1$. Both errors are under $s/2 = 0.1$.

• $s = \max|x|/q_{\max}$, $q = \operatorname{round}(x/s)$, $\hat x = sq$; error $\le s/2$.
• INT4 has 15 levels: much coarser than INT8’s 255.
• Outliers coarsen a shared scale; per-channel or per-group scales contain them.
• Per-row scales factor out of the dot product.