Notation

• A scalar is one number. Lowercase italic: $a$.
• A vector is a list of numbers. Lowercase: $x$ (some books use bold x).
• A matrix is a grid of numbers. Uppercase: $W$.

W has 2 rows and 3 columns. Its shape is 2 × 3, rows first, always. In PyTorch, W.shape is torch.Size([2, 3]).

$W_{23}$ is the entry in row 2, column 3: here −1. In code the same entry is W[1, 2], because Python counts from 0.

Most math books and many papers treat a vector as a column and put the matrix on the left: $y = Wx$. W maps 3 inputs to 2 outputs, so its shape is (outputs × inputs).

Code stores one example (one token) per row, so the vector comes first: $y = xW$, with W shaped (inputs × outputs). It’s the same map, transposed: $(Wx)^\top = x^\top W^\top$. Same numbers: 7 and −1.

A sequence of T tokens is a matrix X with one row per token and one column per feature, so its shape is $T \times d$. A layer with weight W computes $XW$: every row passes through W at once.

lin = torch.nn.Linear(in_features=4, out_features=2)
lin.weight.shape          # torch.Size([2, 4])  (out, in)
y = x @ lin.weight.T + lin.bias   # what lin(x) computes

PyTorch keeps the paper’s (out × in) layout for the weight and transposes it in the forward pass: $y = xW^\top + b$. So a checkpoint’s q_proj.weight of shape [4096, 4096] reads as (out, in).

The example sizes are a Llama 3 8B-style model (except T and B, which you choose).

• Scalars $a$, vectors $x$, matrices $W$. Shapes are (rows, columns).
• Math indexes from 1 ($W_{23}$), code from 0 (W[1, 2]).
• Papers: $y = Wx$ with W as (out × in). Code: $y = xW$ with W as (in × out).
• This course: one row per token, $X$ is $T \times d$, a layer computes $XW$.
• nn.Linear stores (out, in) and computes $xW^\top + b$.