Llama 3 8B, end to end

Everything on the following cards follows from these seven numbers (from the model’s config and Meta’s model card). Copy them at the top of a fresh page; you’ll use every one.

Each id picks one row of the embedding table (Part 1). No arithmetic, just a lookup. This vector starts the residual stream (Part 6).

RMSNorm divides each row by its root mean square and applies a gain (Part 6). Then three matrix products. $W_K$ and $W_V$ are narrower because of GQA: 8 key/value heads of 128 instead of 32 (Part 5).

view then transpose(1, 2), never a bare reshape (Part 4). RoPE then rotates each pair of entries in $Q$ and $K$ by an angle that depends on the position (Part 6), so the scores will depend on relative position. Each K/V head serves 4 query heads.

A dot product for every (query, key) pair in every head, divided by $\sqrt{d_h} \approx 11.3$ so the scores don’t saturate the softmax (Part 1). The causal mask puts $-\infty$ above the diagonal, and the softmax runs along each row (Part 5).

Each output row is a weighted average of value rows (Part 5). Transpose back, merge the heads into one 4096-wide row, multiply by $W_O$, and add to the residual stream.

Normalize again, widen to 14,336 twice, gate one with the other elementwise, and project back down (SwiGLU, Part 6). Add to the residual stream. That’s one block; there are 32.

A final norm, then one score per vocabulary token. When generating, only the last row matters: its softmax is the distribution the next token is sampled from. In training, every row is scored against the actual next token with cross-entropy (Part 8).

Total parameters? (You did this in Part 6.)

Exactly 8,030,261,248, counting the final norm.

Weights plus KV cache, in GB?

Fits on one 80 GB GPU with plenty of room for a batch: each extra full-length sequence adds only 1 GiB of cache, thanks to GQA. Without GQA (32 K/V heads) it would be 4 GiB each.

What’s the most tokens per second for that one sequence?

Every token reads all the weights and the whole cache once (Part 9). Batching 32 sequences reads the weights once for all of them, so total throughput rises far more than the per-sequence speed falls.

Processing an 8,192-token prompt: the weight matmuls cost $2N$ per token. Scores $QK^\top$ and mixing $AV$ each cost $2T^2 d$ per layer, on top. At full context, attention adds about 27% (half of it is masked, but simple kernels compute it anyway); at 1,000 tokens, about 3%. About $1.7 \times 10^{14}$ FLOPs in all: at 989 TFLOP/s, at least 0.17 s.

Estimate the training FLOPs. Meta reports 1.3 million H100 GPU-hours; what sustained FLOP/s per GPU does that imply?

About 150 TFLOP/s of counted work per GPU, roughly 15% of the dense BF16 peak. $6ND$ leaves out attention and everything that isn’t a training step, and Meta’s token count is “over 15T”, so read this as an order-of-magnitude check, not a utilization figure.

Part 7 put adapters on $W_Q$ and $W_V$. Now adapt the three MLP matrices instead. How many trainable parameters? What fraction of the model?

$r(d_{\text{in}} + d_{\text{out}})$ per adapted matrix, the same for $W_{\text{down}}$ since only the sum of the sides matters. About 4× Part 7’s attention-only adapter, still tiny: 57 MB in BF16, small enough to keep one per customer and swap at serving time.

• Lookup → 32 × (norm, QKV, RoPE, scores, mask, softmax, mix, $W_O$, add; norm, SwiGLU, add) → norm → logits.
• 8.03B parameters; 16 GB in BF16; KV 128 KiB per token.
• Decode ≈ bandwidth / bytes per token; prefill ≈ $2NT + 4LT^2 d$; training ≈ $6ND$.
• LoRA: $r(d_{\text{in}} + d_{\text{out}})$ per matrix.

• Meta Llama 3 model card (github.com/meta-llama/llama3, MODEL_CARD.md): 8B and 70B, context 8k, GQA in all models, pretrained on over 15 trillion tokens, 1.3M H100-80GB GPU-hours for 8B.
• Meta-Llama-3-8B config.json: hidden size 4,096, 32 layers, 32 attention heads, 8 key/value heads, intermediate size 14,336, vocabulary 128,256.
• NVIDIA H100 datasheet: 3.35 TB/s (SXM), about 989 TFLOPS dense BF16.