Transpose

$A$ has 2 rows and 3 columns, so its shape is $2 \times 3$.

The transpose, written $A^\top$, takes the first row of $A$ (blue) and stands it up as its first column.

The second row (pink) becomes the second column. $A^\top$ has 3 rows and 2 columns: the shape flipped from $2 \times 3$ to $3 \times 2$.

The entry in row $i$, column $j$ of $A^\top$ is the entry in row $j$, column $i$ of $A$. Entries on the diagonal (row = column) stay where they are: here 1 and 5.

Write $B^\top$ and its shape.

Each column of $B$ became a row of $B^\top$: $3 \times 2$ became $2 \times 3$.

Flipping over the diagonal twice puts every entry back where it started.

A square matrix equal to its own transpose is symmetric: $S_{ij} = S_{ji}$, a mirror image across the diagonal. Covariance matrices and every matrix of the form $XX^\top$ are symmetric (you’ll see why in Four ways to see AB).

Transposing a row vector ($1 \times 3$) gives a column vector ($3 \times 1$). Papers write a dot product as $x^\top y$: a row times a column gives a $1 \times 1$ matrix, which is a single number.

A.T                   # a 2-D matrix: swap rows and columns
k.transpose(-2, -1)   # a batch of matrices: swap only the last two axes
k.mT                  # the same, shorter

In PyTorch, .T and .transpose don’t copy any numbers. They return a view of the same memory with the axes swapped (Part 4).

• $A^\top$ turns rows into columns: $(A^\top)_{ij} = A_{ji}$.
• The shape flips: $m \times n \to n \times m$. The diagonal stays put.
• $(A^\top)^\top = A$. Symmetric means $S^\top = S$.
• A row vector transposed is a column vector, so $x^\top y$ is a dot product.
• Attention uses $K^\top$ so that every query meets every key: $QK^\top$ is $T \times T$.