LoRA

LoRA (low-rank adaptation) keeps the pretrained $W$ fixed and trains two thin matrices. Their product $BA$ has $W$’s shape but rank at most $r$ (Part 3), with $r$ small: 8, 16 or 64.

In nn.Linear layout: $A$ maps the input down to $r$ numbers, $B$ maps them back up. The $\text{out} \times \text{in}$ product is never formed during training (Part 2’s cheap ordering).

$A$ starts random and $B$ starts at zero, so training begins from exactly the pretrained model and the update grows from nothing. $\alpha/r$ scales the update so that changing $r$ needs less retuning of the learning rate.

0.39% of the matrix. Gradients and optimizer state are only needed for these, which is where most of LoRA’s memory saving comes from.

How many LoRA parameters if you adapt $W_Q$ and $W_V$ in every layer? Roughly what fraction of 8.03B is that?

Under a tenth of a percent of the model. The original LoRA paper found that adapting these two projections alone worked well.

After training, add the update into the weight once. The merged layer is an ordinary matrix of the same shape: no extra parameters or FLOPs at inference. (Or keep it separate, to swap many adapters on one base model.)

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=8, alpha=16):
        super().__init__()
        self.base, self.scale = base, alpha / r
        self.A = nn.Parameter(torch.randn(r, base.in_features) / r)
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        base.weight.requires_grad_(False)

    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T) @ self.B.T

A sketch of the idea; libraries such as PEFT add dropout, initialization choices and merging.

• $W' = W + \frac{\alpha}{r}BA$, $B\!: (\text{out}, r)$, $A\!: (r, \text{in})$, $W$ frozen.
• $B = 0$ at the start, so training begins at the pretrained model.
• $r(\text{in} + \text{out})$ parameters per adapted matrix.
• Merge after training: no inference cost.