The LLM lifecycle: pre-training to production

Field notes on the five phases that take a pile of GPUs to a model people actually use: pre-training, post-training, adaptation, reliability/evals, and scale engineering. Same standard as before: intuition first, then the math worked through a concrete example, then the production code a real team would actually run, then the tradeoffs, then real numbers from named systems. Nothing here is “just follow the list” -- every chapter below has what the original outline had, plus what frontier teams actually need that the outline left out, called out explicitly so the gap is visible rather than silently patched.

00 · Why these five phases are one pipeline, and what the outline left out#

❓ Why this framing. Each phase produces the input the next phase depends on and inherits the debt the previous phase left unpaid. Pre-training’s data contamination becomes post-training’s inflated eval scores becomes adaptation’s false confidence becomes a reliability incident nobody can root-cause because nobody logged which phase introduced the problem. Treating these as five independent chapters to check off misses the thing that actually breaks in production: the seams between them.

$$\text{final system quality} = f(\text{data quality}) \times g(\text{alignment quality}) \times h(\text{retrieval/tool quality}) \times k(\text{eval coverage}) \times m(\text{serving reliability})$$

It’s a product, not a sum -- a frontier-grade pre-trained base model with sloppy post-training data still ships a mediocre assistant, and a perfectly aligned model with no eval harness ships regressions nobody catches until users complain. Every phase can zero out the others.

What the original outline gets right#

The five-phase structure (pre-train → post-train → adapt → evaluate → scale) is the correct top-level shape, and the “production things to learn” split alongside “subtopics” in each chapter is exactly the right instinct -- frontier engineering is not the algorithm, it’s the algorithm plus everything that breaks when the algorithm meets real data, real traffic, and real failure modes.

What’s missing, added explicitly rather than silently folded in#

• Scaling laws (Kaplan et al., Hoffmann et al./Chinchilla) -- how to decide how big a model and how much data before spending a training budget at all. This is arguably the single highest-leverage decision in the entire pipeline and isn’t in the original list at all. Added to ch. 1.
• Direct preference optimization and its variants (DPO, IPO, KTO, ORPO) -- the outline says “preference optimization” and “RLHF” as if RLHF-with-PPO is still the default; in practice DPO-family methods are now the more common production starting point precisely because they remove the reward-model-plus-RL-loop complexity. Added to ch. 2 with full derivation.
• Model distillation and model merging -- compressing a frontier model into a cheaper one to actually serve, and combining multiple fine-tuned checkpoints without retraining. Missing entirely from the post-training list. Added to ch. 2.
• The long-context-vs-RAG decision -- the outline’s adaptation chapter lists RAG as a given without asking when adaptation should reach for a bigger context window instead (this document’s companion piece on long-context architectures is the other half of that decision). Added to ch. 3.
• Agentic/multi-step workflows (ReAct, tool-use loops) as a first-class adaptation pattern, not folded silently into “tool calling.” Added to ch. 3.
• LLM-as-judge methodology, and its own well-documented biases -- the outline’s evals chapter lists “offline vs online evals” but not the specific mechanism (a second LLM scoring the first) that now underlies most production eval pipelines, nor its failure modes. Added to ch. 4.
• Statistical rigor for evals -- sample-size and confidence-interval reasoning, without which “eval went from 82% to 83%” is often noise, not signal. Added to ch. 4.
• Cost modeling and SLO/error-budget math -- the outline’s scale-engineering chapter lists “cost/performance tradeoffs” as a bullet with no actual cost formula. Added to ch. 5, alongside a pointer to this document’s companion piece for the KV-cache/serving/quantization depth that chapter would otherwise duplicate.
• Data licensing and contamination-as-a-legal-not-just-technical problem -- briefly, because it changes real engineering decisions (what you’re allowed to train on, what you must be able to prove you didn’t train on) even though it isn’t a modeling technique. Added to ch. 1.

Each addition below is folded into its natural chapter rather than bolted on as an afterthought, with the same math-plus-code-plus-real-numbers treatment as everything the original list asked for.

01 · Pre-training#

❓ Why this topic. Everything downstream -- alignment, adaptation, evals, serving -- inherits whatever the base model actually learned, and cannot add knowledge or capability the base model never saw. Pre-training is also the phase with the least room for error: a mistake discovered at step 200,000 of a single 3-month run cannot be patched, only restarted from a checkpoint, at a direct cost of real GPU-hours and real money. Tokenization, core architecture, optimization (AdamW, LR schedules, gradient clipping), mixed precision, and distributed-training mechanics (FlashAttention, ZeRO, Megatron-style tensor/pipeline parallelism) are covered in full in the companion transformer field notes (chapters 0–1, 5–6) -- this chapter doesn’t re-derive them, it covers what that document didn’t: how much to train, on what, deduplicated how, and what happens when a GPU dies at 3am on day 40 of a run.

Scaling laws: deciding how big before you spend the budget#

The problem this solves. Given a fixed compute budget $C$ (in FLOPs), how should it be split between model size $N$ (parameters) and data size $D$ (tokens)? Get this wrong and you either train a model too large for the data it saw (undertrained, GPT-3’s actual mistake in hindsight) or too small to use the data well.

Kaplan et al. (2020) first fit power laws for loss as a function of $N$, $D$, and $C$ independently, training a large grid of transformers ranging from under 1,000 to over 1 billion parameters, and reported the power-law fit holding smoothly across more than seven orders of magnitude of compute -- their specific, and at the time surprising, additional finding was that loss depends almost entirely on scale ($N$, $D$, $C$) and only very weakly on architectural shape at fixed scale (their own ablations varied network width-to-depth ratio substantially with little effect on the fitted loss curve), which is the empirical result that legitimized “just make it bigger” as a research strategy in the first place. Hoffmann et al. (2022, “Chinchilla”) re-fit this more carefully -- varying both $N$ and $D$ jointly across over 400 training runs ranging from 70M to 16B parameters -- and found the earlier laws had implicitly assumed a fixed, suboptimal data-to-parameter ratio. The Chinchilla-form loss law:

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$

$E$ is an irreducible-loss floor (entropy of natural language itself, unreachable by any model), the two power-law terms are the loss added by finite model size and finite data respectively. Minimizing $L$ subject to a fixed compute budget $C \approx 6ND$ (the standard FLOPs-per-token-per-parameter approximation for a transformer forward+backward pass) gives closed-form optimal scaling:

$$N_{\text{opt}}(C) \propto C^{a}, \qquad D_{\text{opt}}(C) \propto C^{b}, \qquad a + b = 1$$

Hoffmann et al.’s fitted result: $a \approx b \approx 0.5$ -- compute-optimal training scales model size and data size at roughly the same rate, i.e. a 10x compute budget increase should go roughly 3.2x into more parameters and 3.2x into more tokens, not overwhelmingly into one or the other.

Worked example, the number that mattered in practice. GPT-3 was trained with $N \approx 175\text{B}$ parameters on $D \approx 300\text{B}$ tokens -- a ratio of roughly 1.7 tokens per parameter. Applying the Chinchilla-optimal ratio to GPT-3’s actual training compute budget instead implies a compute-optimal model would have been roughly 4x smaller (~67-70B parameters) trained on roughly 4x more data (~1.4T tokens), for the same training FLOPs. Hoffmann et al.’s own Chinchilla model (70B parameters, 1.4T tokens) was trained on the same compute budget as DeepMind’s own 280B-parameter Gopher and outperformed it on every evaluated downstream benchmark -- direct empirical proof that Gopher (and by the same logic, GPT-3) had spent its compute budget on parameters it didn’t have enough data to justify.

Why that’s not a subtle result -- same money, better product. Think of the training compute budget as a fixed amount of cash to spend on a training run, and $N$ (model size) and $D$ (data seen) as two things you’re buying with it -- bigger model costs more per token processed, so for the same total spend, a bigger model necessarily means seeing fewer tokens, and a smaller model means you can afford to show it far more tokens. GPT-3 spent that budget mostly on making the model bigger (175B params, only 300B tokens -- under 2 tokens of data per parameter, quite thin). Chinchilla, given the identical dollar amount, spent it differently: a model four times smaller, shown four times more data -- 20 tokens per parameter instead of fewer than 2. Same bill at the end of training. And the smaller-but-better-fed model simply performed better on every benchmark tested against the bigger-but-thinner-fed Gopher. The one-sentence takeaway: past a certain point, a bigger model that hasn’t seen enough text is worse than a smaller model that’s read more, even though the smaller one is strictly cheaper to run afterward too.

$$\text{tokens per parameter (Chinchilla-optimal)} \approx 20 \qquad \text{tokens per parameter (GPT-3, actual)} \approx 1.7$$

The production consequence. Every frontier lab’s post-2022 training recipe shifted toward far higher token-to-parameter ratios than GPT-3’s -- Llama 2’s 70B model trained on 2T tokens (~28 tokens/parameter), Llama 3’s 8B model trained on 15T tokens (~1,875 tokens/parameter, deliberately far past the Chinchilla-optimal point). This latter choice is itself a documented, deliberate deviation: Meta’s own Llama 3 report explains that Chinchilla-optimal sizing minimizes training compute for a target loss, but a smaller model trained past that point on more data can still be improving on downstream tasks well after the “optimal” stopping point, and a smaller model is far cheaper to serve at scale for years afterward -- so the real production decision is compute-optimal for training cost vs. compute-optimal for total cost of ownership including inference, and every frontier lab now deliberately overtrains small models for exactly this reason.

In plain terms: picture opening a bakery on a fixed budget split between an oven and flour. Buy an oven far bigger than the flour you can afford, and most of that oven sits empty -- that’s what GPT-3 did, an oven-sized model without enough flour-sized data to fill it. Chinchilla’s finding is just that for a fixed budget there’s a sweet spot between oven size and flour, and a smaller oven with more flour bakes more usable bread per dollar than the biggest oven you can afford. Frontier labs then made a second, separate choice: buy a smaller oven than the sweet spot anyway, and just buy even more flour than “optimal” -- because a smaller oven is cheaper to keep running (serving) for years afterward, even if baking day itself (training) took a bit more flour to reach the same quality loaf.

# Fitting a (toy) Chinchilla-style scaling law from a grid of small runs, and solving for N_opt, D_opt
import numpy as np
from scipy.optimize import curve_fit

def chinchilla_loss(X, E, A, alpha, B, beta):
    N, D = X
    return E + A / N**alpha + B / D**beta

# N, D, and observed final loss from a sweep of small training runs (illustrative shape only)
Ns = np.array([1e8, 3e8, 1e9, 3e9, 1e9, 3e9])
Ds = np.array([1e10, 1e10, 1e10, 1e10, 3e10, 3e10])
losses = np.array([3.4, 3.1, 2.9, 2.75, 2.7, 2.55])   # replace with real measured eval loss per run

popt, _ = curve_fit(chinchilla_loss, (Ns, Ds), losses, p0=[1.7, 1.0, 0.34, 1.0, 0.28], maxfev=20000)
E, A, alpha, B, beta = popt

def solve_optimal_allocation(C, E, A, alpha, B, beta):
    # C ≈ 6*N*D; minimize L(N,D) s.t. compute constraint via Lagrange multiplier (closed form ratio)
    a = beta / (alpha + beta)          # fraction of compute-scaling exponent going to N
    b = alpha / (alpha + beta)         # fraction going to D
    N_opt = (C / 6) ** a
    D_opt = (C / 6) ** b
    return N_opt, D_opt

N_opt, D_opt = solve_optimal_allocation(C=1e21, E=E, A=A, alpha=alpha, B=B, beta=beta)
print(f"For 1e21 FLOPs: N_opt ≈ {N_opt:.2e} params, D_opt ≈ {D_opt:.2e} tokens")

Data collection and filtering: the funnel, with real numbers#

Raw Common Crawl is enormous and mostly unusable as-is: boilerplate, spam, non-natural-language text, and duplicate content dominate by volume. Every frontier pretraining corpus is the output of a filtering funnel, not the raw crawl itself.

$$\text{raw crawl} \xrightarrow{\text{language ID}} \xrightarrow{\text{quality filter}} \xrightarrow{\text{dedup (ch. below)}} \xrightarrow{\text{PII/toxicity filter}} \text{final training corpus}$$

Real published funnel: the FineWeb report (Penedo et al., 2024, Hugging Face) documents starting from roughly 96 Common Crawl snapshots and applying language identification (fastText classifier), heuristic quality filters (adapted from Gopher’s and C4’s rulesets -- e.g. reject documents with excessive symbol-to-word ratio, too few sentences, too much boilerplate), then deduplication (below), then a quality classifier trained to imitate curated data (a lightweight classifier trained on OpenWebText-style “was this ever linked from Reddit with enough upvotes” positive examples, scoring every remaining document and keeping only the highest-scoring fraction) -- the whole pipeline discarding the large majority of raw crawled text by volume while measurably improving downstream benchmark performance per token trained on, evidence that raw volume and training-value are not the same axis at all.

# A minimal quality-filter pass, in the spirit of the Gopher/C4/FineWeb heuristic rules
import re

def passes_heuristic_quality_filter(text: str) -> bool:
    words = text.split()
    if not (50 <= len(words) <= 100_000):                       # too short: boilerplate; too long: likely garbage dump
        return False
    symbol_ratio = sum(c in "#*{}[]<>" for c in text) / max(len(text), 1)
    if symbol_ratio > 0.1:                                        # markup/code-dump heuristic
        return False
    mean_word_len = sum(len(w) for w in words) / len(words)
    if not (3 <= mean_word_len <= 10):                            # gibberish or token-spam heuristic
        return False
    if len(re.findall(r"[.!?]", text)) < len(words) / 50:         # too few sentence boundaries -> not prose
        return False
    return True

Deduplication: MinHash, worked by hand#

Why it matters more than it looks. Lee et al. (2022, “Deduplicating Training Data Makes Language Models Better”) showed duplicate documents in pretraining data are not a harmless minor inefficiency: near-duplicate content (boilerplate templates, syndicated news articles, scraped mirror sites) gets memorized disproportionately relative to its actual information content, measurably increasing verbatim regurgitation of training data at generation time and wasting training compute re-learning the same text repeatedly.

Exact-match dedup (hash every document, drop exact repeats) catches only identical copies. Real web text has enormous amounts of near-duplicate content -- the same article with a different ad banner, a mirrored page with a different header -- which needs a similarity measure, and comparing every document pairwise ($O(n^2)$) is infeasible at web scale. MinHash solves this by turning set similarity into a cheap, approximate, hashable signature.

The math, worked on a tiny example. Represent each document as a set of overlapping word shingles (n-grams):

doc A: "the quick brown fox jumps" -> shingles (n=3): {the quick brown, quick brown fox, brown fox jumps}
doc B: "the quick brown fox runs"  -> shingles (n=3): {the quick brown, quick brown fox, brown fox runs}

Jaccard similarity of these two sets: $J(A,B) = |A \cap B| / |A \cup B| = 2/4 = 0.5$. Computing exact Jaccard similarity for every pair of documents in a billion-document corpus is intractable. MinHash’s trick: apply $k$ independent random hash functions to every shingle in a set, keep only the minimum hash value each function produces across the whole set (one integer per hash function, per document) -- this small MinHash signature has a remarkable property:

$$P(\text{minhash}_i(A) = \text{minhash}_i(B)) = J(A, B)$$

The probability that two documents’ MinHash signatures agree on a randomly chosen hash function exactly equals their true Jaccard similarity -- so averaging agreement across $k$ independent hash functions gives an unbiased, cheap estimate of Jaccard similarity without ever computing the true set intersection. Locality-sensitive hashing (LSH) then buckets documents by bands of their MinHash signature so only documents likely to be near-duplicates (same bucket) are ever compared directly, turning an $O(n^2)$ problem into a near-linear one.

In plain terms: imagine judging whether two strangers in a crowd of a million people are dressed similarly, without time to examine every stitch of every outfit. Instead, everyone agrees in advance on a handful of quick spot-checks -- hat color, shoe color, jacket style -- and only those get compared. Two people who match on most spot-checks are probably dressed alike overall, even though nobody ever compared full outfits thread-by-thread. MinHash does exactly this with a document’s words instead of an outfit’s items: a handful of cheap “spot-checks” (hash functions) stand in for a full, expensive comparison.

# Real, runnable near-duplicate detection using datasketch's MinHashLSH
from datasketch import MinHash, MinHashLSH

def shingles(text, n=3):
    words = text.split()
    return {" ".join(words[i:i+n]) for i in range(len(words) - n + 1)}

def make_minhash(text, num_perm=128):
    m = MinHash(num_perm=num_perm)
    for shingle in shingles(text):
        m.update(shingle.encode("utf8"))
    return m

lsh = MinHashLSH(threshold=0.8, num_perm=128)   # threshold: Jaccard similarity above which docs are "near-duplicate"

documents = {"doc_1": "the quick brown fox jumps over the lazy dog",
             "doc_2": "the quick brown fox runs over the lazy dog",     # near-duplicate of doc_1
             "doc_3": "completely unrelated content about astronomy"}

minhashes = {}
for doc_id, text in documents.items():
    mh = make_minhash(text)
    minhashes[doc_id] = mh
    duplicates = lsh.query(mh)          # near-duplicates already inserted, found in near-constant time
    if duplicates:
        print(f"{doc_id} is a near-duplicate of {duplicates}, dropping from training corpus")
        continue
    lsh.insert(doc_id, mh)

Exact-substring dedup, the complementary technique. MinHash catches whole-document near-duplication; it misses a long exact passage duplicated inside otherwise-different documents (a quoted news wire story embedded in many different articles). Lee et al.’s exact-substring method instead builds a suffix array over the entire concatenated corpus and finds all repeated substrings above a minimum length (their paper uses 50 tokens), removing those spans specifically rather than dropping whole documents -- real production pretraining pipelines (Llama, GPT-NeoX/Pile, RefinedWeb) run both MinHash-style near-dedup and exact-substring dedup, since they catch different failure modes.

Contamination control: is your eval already in your training data?#

Why this is a distinct problem from dedup. Deduplication is about redundancy within the training corpus. Contamination is about overlap between the training corpus and the benchmarks you’ll later evaluate on -- if a benchmark’s test questions (or close paraphrases) appear in pretraining data, every downstream eval score in chapter 4 becomes partly a memorization test, not a capability test.

The standard detection method is n-gram overlap: for every benchmark example, check whether a sufficiently long, sufficiently rare n-gram (13-gram is GPT-3’s own published choice) from that example appears anywhere in the training corpus. GPT-3’s own paper documented measurable contamination across several benchmarks this way and re-reported “clean” scores on the decontaminated subset specifically because of it -- contamination auditing has been standard practice at every frontier lab since. The engineering reality: this check must run before the expensive training run, not after, because a contaminated corpus found post-training means either accepting inflated benchmark numbers or a full, costly retrain.

Distributed training: FSDP, tensor parallelism, and pipeline parallelism, with real memory math#

The transformer notes (ch. 6) cover ZeRO/FSDP and Megatron-style tensor parallelism at the concept level. The memory arithmetic that decides which strategy a real run actually needs:

Per-GPU memory for full (unsharded) training of an $N$-parameter model, mixed precision, Adam:

$$\text{memory} \approx \underbrace{2N}_{\text{bf16 weights}} + \underbrace{2N}_{\text{bf16 grads}} + \underbrace{4N + 4N}_{\text{fp32 Adam } m, v} + \underbrace{4N}_{\text{fp32 master weights}} = 16N \text{ bytes}$$

Worked example. A 70B-parameter model needs roughly $16 \times 70 \times 10^9 \approx 1.12\text{ TB}$ of optimizer+weight+gradient state -- on a GPU with 80GB of HBM, that alone is 14 GPUs’ worth of memory before a single activation is stored, which is exactly why no real frontier model trains on a single GPU, or even without sharding this state across many.

FSDP (Fully Sharded Data Parallel) shards this $16N$-byte state itself across $P$ data-parallel ranks -- each GPU holds only $16N/P$ bytes at rest, all-gathering the full parameter shard from every other rank just before it’s needed for a forward or backward computation on a given layer, then discarding it again. The real cost is communication: every layer’s forward and backward pass now requires an all-gather (assembling that layer’s full weights) and later a reduce-scatter (redistributing that layer’s gradient), so FSDP trades memory for network bandwidth -- it only wins when the interconnect (NVLink within a node, InfiniBand across nodes) is fast enough that this constant communication doesn’t dominate compute time.

In plain terms: think of FSDP as several roommates sharing one bookshelf instead of each buying a full copy of every book. Each roommate stores only their assigned fraction of the books, and borrows whichever book a coworker needs at the exact moment it’s needed, then hands it right back. This saves a lot of shelf space (memory) -- but only works smoothly if handing books back and forth is fast; if the roommates lived in different cities and had to mail books to each other, the constant mailing (slow network) would eat up all the time saved.

# torch FSDP2 (per-parameter sharding), the current production API
import torch
from torch.distributed.fsdp import fully_shard, FSDPModule
from torch.distributed.device_mesh import init_device_mesh

mesh = init_device_mesh("cuda", (8,))          # 8-way data-parallel shard group, e.g. one node of 8 GPUs
model = build_transformer_model(...)             # any nn.Module

for layer in model.layers:                        # shard at transformer-block granularity, not the whole model at once
    fully_shard(layer, mesh=mesh)
fully_shard(model, mesh=mesh)                     # shard the remaining top-level params (embeddings, final norm)

optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)
# forward/backward look identical to single-GPU code; FSDP transparently all-gathers/reduce-scatters per layer

Tensor parallelism (Megatron-style) takes the opposite approach for a single layer: instead of sharding which layers live where, it splits an individual weight matrix (e.g. the FFN’s up-projection) column-wise across GPUs, so each GPU computes a slice of the same matmul and an all-reduce combines partial results before the next operation. This needs far more frequent, smaller communication than FSDP’s per-layer all-gather, which is why it’s only used within a node over NVLink (hundreds of GB/s) and essentially never across the slower interconnect between nodes.

Pipeline parallelism instead assigns different layers to different GPUs entirely (GPU 1 holds layers 1–10, GPU 2 holds layers 11–20, …), passing activations forward and gradients backward between them. Its distinctive cost is the pipeline bubble: GPU 2 sits idle until GPU 1 finishes its first microbatch, and GPU 1 sits idle again at the end waiting for the backward pass to return -- the fraction of idle time shrinks as the number of microbatches per pipeline stage grows, but never reaches zero.

$$\text{bubble fraction} \approx \frac{p - 1}{m}$$

where $p$ is the number of pipeline stages and $m$ is the number of microbatches -- a 4-stage pipeline with only 4 microbatches wastes 3/4 of its time in bubbles; the same pipeline with 32 microbatches wastes under 10%, which is why real production configs push microbatch count as high as memory allows.

Real numbers: how it composes at frontier scale. Llama 3’s 405B model training (Dubey et al., 2024) combined all three simultaneously -- 4D parallelism (data, tensor, pipeline, and context parallelism, the sequence-dimension sharding from Ring Attention, ch. 1 of the companion long-context document) across a cluster the paper reports at over 16,000 H100 GPUs, precisely because no single parallelism strategy alone scales to that size: tensor parallelism is capped by intra-node NVLink bandwidth, pipeline parallelism’s bubble cost grows with depth, and pure data parallelism alone would need each GPU to hold the entire $16N$-byte optimizer state, which doesn’t fit.

Checkpointing and fault tolerance: real numbers on how often things actually break#

Why this is not paranoia. At the GPU-count and multi-week duration of a frontier training run, hardware failure isn’t an edge case, it’s an expected, recurring event. Meta’s own Llama 3 report is unusually candid about this: across a 54-day period of the 405B model’s pretraining on their ~16,384-GPU H100 cluster, the paper documents 466 job interruptions, the majority attributed to hardware issues (GPU failures alone accounting for a substantial fraction, alongside host and networking failures), averaging roughly one interruption every three hours across the whole run. A training pipeline that can’t checkpoint and resume quickly at that failure rate simply cannot complete a run of this scale within any reasonable wall-clock budget.

What real recovery requires. Frequent, fast, distributed checkpointing (every model shard’s local state, written in parallel, not funneled through a single node) and equally fast, automatic resume -- Llama 3’s own reported checkpoint-write throughput was high enough that even at that failure cadence, GPU utilization stayed above 90% for the run, because the time lost per interruption was small relative to the interruption interval.

In plain terms: this is exactly why video games autosave every few minutes instead of only at the very end. If a crash is going to happen roughly every three hours regardless, the only real question is how much progress you lose when it does -- a fast, frequent save means a crash costs you a few minutes of replaying, not the whole session.

# torch.distributed.checkpoint -- the current production API for sharded, parallel checkpoint save/load
import torch.distributed.checkpoint as dcp
from torch.distributed.checkpoint.state_dict import get_state_dict, set_state_dict

def save_checkpoint(model, optimizer, step, ckpt_dir):
    model_state, optim_state = get_state_dict(model, optimizer)
    dcp.save({"model": model_state, "optim": optim_state, "step": step},
              checkpoint_id=f"{ckpt_dir}/step_{step}")   # every rank writes its own shard in parallel

def resume_checkpoint(model, optimizer, ckpt_dir):
    model_state, optim_state = get_state_dict(model, optimizer)
    state_dict = {"model": model_state, "optim": optim_state, "step": 0}
    dcp.load(state_dict, checkpoint_id=ckpt_dir)         # each rank reads only its own shard back
    set_state_dict(model, optimizer, model_state_dict=state_dict["model"], optim_state_dict=state_dict["optim"])
    return state_dict["step"]

# elastic, fault-tolerant launch: torchrun automatically handles a subset of workers restarting mid-job
torchrun --nnodes=1:64 --nproc_per_node=8 --max_restarts=10 \
    --rdzv_id=job123 --rdzv_backend=c10d --rdzv_endpoint=$HOSTNAME:29500 \
    train.py --resume-from-checkpoint /checkpoints/latest

Production tooling: the rest of the list, at real depth#

task	real tool / method	the thing that actually matters
Data validation / anomaly detection	schema and distribution checks on token-length histograms, language-ID distribution, and duplicate-rate per data source, run before a shard enters the training mix	a silent upstream scraper bug (e.g. one data source suddenly emitting 90% boilerplate) will train a measurably worse model with no error thrown anywhere unless caught here
Job orchestration	Slurm (sbatch, srun) for on-prem/HPC clusters, Ray or Kubernetes-based schedulers for cloud-native setups	the orchestrator, not the training script, is what handles the 466-interruptions-per-54-days reality above -- automatic requeue on node failure is a hard requirement, not a nice-to-have
GPU profiling	NVIDIA Nsight Systems/Compute for kernel-level bottleneck analysis, torch.profiler for framework-level traces	model FLOPs utilization (MFU) -- actual achieved FLOPs divided by hardware peak FLOPs -- is the single number that tells you whether you’re compute-bound (good) or stalled on data loading/communication (bad); PaLM’s own reported 46.2% MFU (companion long-context document, ch. 6) is a realistic, not disappointing, frontier number
Experiment tracking	Weights & Biases, MLflow, or an internal equivalent, logging loss curves, gradient norms, LR schedule, and MFU per step	the gradient-norm and loss-spike monitoring from the transformer notes (ch. 5) is only actionable if it’s actually being logged and alerted on in real time, not reconstructed after the fact from scattered logs
Reproducibility	pinned data-shard ordering (seeded shuffling, deterministic sharding across ranks), pinned library/CUDA versions, logged exact hyperparameters and data mixture per run	without deterministic data order, “rerun this exact experiment” is not actually possible even with the same seed, because distributed data loading order depends on how the corpus was sharded across workers that run
Cost modeling	$\text{training cost} \approx \dfrac{6ND}{\text{achieved FLOPs/sec}} \times \text{hourly GPU cost}$	DeepSeek-V3’s own reported ~2.79M H800 GPU-hours (companion long-context document, ch. 6) is the real-world instantiation of exactly this formula -- it’s why scaling laws (above) are a cost decision as much as a quality decision

The part of this list that isn’t a modeling technique at all#

Data licensing and contamination as a legal question, not just a technical one. What a corpus is legally permitted to be trained on is a live, actively litigated question (e.g. The New York Times Co. v. OpenAI, ongoing as of this writing), and regulatory regimes are starting to require disclosure of training data sources and processes (the EU AI Act’s transparency obligations for general-purpose AI models being the most concrete current example). This changes real engineering decisions -- which data sources are even eligible for the corpus, what provenance and consent metadata must be tracked per document, and what a lab must be able to prove (or disprove) about what it did or didn’t train on -- independent of any technical filtering or dedup pipeline. It belongs in this chapter because production data pipelines now build licensing/provenance tracking in alongside quality and dedup filtering, not as a separate afterthought.

📚 Further reading

• Kaplan et al., 2020 -- Scaling Laws for Neural Language Models
• Hoffmann et al., 2022 -- Training Compute-Optimal Large Language Models (Chinchilla)
• Dubey et al., 2024 -- The Llama 3 Herd of Models (data pipeline, 4D parallelism, checkpointing/failure numbers)
• Penedo et al., 2024 -- The FineWeb Datasets
• Lee et al., 2022 -- Deduplicating Training Data Makes Language Models Better
• Broder, 1997 -- On the Resemblance and Containment of Documents (MinHash)
• Rae et al., 2021 -- Scaling Language Models: Methods, Analysis & Insights from Training Gopher

02 · Post-training / alignment#

❓ Why this topic. A base model is a next-token predictor over whatever text it saw -- it has no notion of “answer the question,” “refuse this,” or “be concise,” because none of those are things next-token prediction on raw web text directly optimizes for. Post-training is the phase that turns “predicts plausible continuations” into “acts like a helpful assistant,” and it is where the majority of a frontier lab’s user-visible quality differentiation now happens -- the base model landscape has converged enough that alignment quality, not raw pretraining scale, is often the deciding factor between two similarly-sized models.

Supervised fine-tuning: the loss-masking detail that actually matters#

What it does. Continue training the base model with ordinary next-token cross-entropy, but on curated (prompt, response) pairs instead of raw web text, formatted through the model’s chat template (transformer notes, ch. 1).

The detail the original outline’s “loss masking and response-only training” bullet is really about. If you compute cross-entropy loss over the entire sequence (prompt tokens included), the model spends gradient updates learning to predict the prompt -- text it didn’t generate and never needs to generate at inference time -- which dilutes the signal that actually shapes response behavior and can measurably slow convergence toward good response quality. The fix: mask prompt tokens’ labels to a sentinel value (-100 in PyTorch’s cross-entropy convention, which the loss function is instructed to ignore) so gradients flow only from tokens the model is actually responsible for generating.

$$\mathcal{L}_{\text{SFT}} = -\frac{1}{|\{i : \text{label}_i \neq -100\}|} \sum_{i:\, \text{label}_i \neq -100} \log p_\theta(y_i \mid y_{<i})$$

# Loss masking, made concrete: labels for prompt tokens set to -100, response tokens kept as-is
def build_labels(input_ids, prompt_len):
    labels = input_ids.clone()
    labels[:prompt_len] = -100        # cross-entropy skips these positions entirely, no gradient from them
    return labels

# real production path: TRL's SFTTrainer does this automatically given a chat-formatted dataset
from trl import SFTTrainer
from datasets import load_dataset

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-7B",
    train_dataset=load_dataset("trl-lib/Capybara", split="train"),   # prompt/response pairs, response-only loss by default
)
trainer.train()

Reward modeling: the Bradley-Terry math, worked#

The problem. You have pairs of responses to the same prompt, one marked preferred by a human rater, one not -- but no scalar “quality score” for either individually. The Bradley-Terry model converts pairwise comparisons into an implied scalar reward function $r_\phi(x, y)$ by assuming the probability of preferring $y_w$ (winner) over $y_l$ (loser) is a logistic function of their reward difference:

$$P(y_w \succ y_l \mid x) = \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big) = \frac{1}{1 + e^{-(r_\phi(x,y_w) - r_\phi(x,y_l))}}$$

Training the reward model is then ordinary binary cross-entropy on this probability, maximizing the log-likelihood of the observed human preferences:

$$\mathcal{L}_{\text{RM}} = -\mathbb{E}_{(x, y_w, y_l)}\Big[\log \sigma\big(r_\phi(x,y_w) - r_\phi(x,y_l)\big)\Big]$$

Worked numeric example. Suppose the reward model currently scores a preferred response $r_\phi(x,y_w) = 2.1$ and the dispreferred one $r_\phi(x,y_l) = 1.4$. The model’s implied probability that a human would have preferred $y_w$: $\sigma(2.1 - 1.4) = \sigma(0.7) \approx 0.668$. Since the human did prefer $y_w$ (label = 1), the loss is $-\log(0.668) \approx 0.403$ -- a real, nonzero penalty pushing the reward gap wider, because 66.8% confidence in the correct preference still leaves room to be more confident. If instead the model had scored them backwards ($r_\phi(x,y_w) = 1.0$, $r_\phi(x,y_l) = 2.0$), $\sigma(1.0 - 2.0) = \sigma(-1.0) \approx 0.269$, and the loss $-\log(0.269) \approx 1.313$ -- over 3x the penalty, correctly punishing the model harder for confidently getting the ranking backwards.

The same numbers, without needing to know what $\sigma$ or $\log$ mean. Reward scores here are just arbitrary point totals the model assigns -- bigger means “the model liked it more.” In the first case, the preferred response scored 2.1 and the loser scored 1.4, so the model gave the actual winner the higher score, but only by 0.7 points -- a fairly close call. Plugging that 0.7-point gap through the sigmoid (a tool that converts “how big is the gap” into “how confident is that, on a 0-to-1 scale”) turns it into roughly 67% confidence. Since 67% confidence in the correct answer is decent but not great, the training signal (“loss”) pushes the model to widen that gap further -- it’s not being punished hard, but it’s being nudged. In the second case, the model got the ranking backwards: it gave the loser the higher score (2.0 vs. 1.0). That negative 1.0-point gap runs through the same sigmoid and comes out as only 27% confidence in the correct answer -- meaning the model was more than 70% confident in the wrong ranking. That’s a much bigger mistake, and the resulting penalty (1.313 vs. 0.403 -- over three times larger) reflects exactly that: getting it backwards with confidence is punished far harder than being right but timid.

In plain terms: this is the same idea as a chess rating system. You never measure a player’s “true skill” directly -- you infer it purely from who beats whom, and a bigger rating gap between two players should predict a more lopsided win probability. The reward model is doing the same thing with two AI responses instead of two chess players: it never sees a “true quality score,” only who a human preferred, and it’s penalized more the more confidently it gets that preference backwards.

import torch, torch.nn.functional as F

def bradley_terry_loss(reward_chosen, reward_rejected):
    return -F.logsigmoid(reward_chosen - reward_rejected).mean()

# real production path: TRL's RewardTrainer expects a dataset with chosen/rejected columns
from trl import RewardTrainer, RewardConfig
from datasets import load_dataset

trainer = RewardTrainer(
    model="Qwen/Qwen2.5-0.5B",
    args=RewardConfig(output_dir="reward-model"),
    train_dataset=load_dataset("trl-lib/ultrafeedback_binarized", split="train"),
)
trainer.train()

⚠️ Reward hacking / overoptimization. A reward model is itself an imperfect proxy for actual human preference, and optimizing a policy hard against it (RLHF’s PPO step, below) eventually exploits whatever the reward model gets systematically wrong rather than what humans actually want -- Gao et al. (2023) measured this directly, using a clever methodology to get around the fact that “true” human preference isn’t a number you can plot on an x-axis at scale: they trained a very large synthetic “gold” reward model to stand in for ground-truth human preference, then trained much smaller proxy reward models against it and optimized policies against those proxies via both RL and best-of-$n$ sampling. The result: policy reward under the proxy reward model keeps climbing smoothly (fit well by their own reported functional form) well past the point where reward under the gold model peaks and then declines -- and, notably, they found the KL-divergence budget at which this decline begins scales predictably with proxy reward model size, meaning a bigger, better proxy reward model buys a larger safe optimization budget before overoptimization sets in, not immunity from it. This is a real, quantified instance of Goodhart’s law (Goodhart, 1975: “when a measure becomes a target, it ceases to be a good measure”) specific to RLHF, and the direct empirical reason production RLHF pipelines cap the KL penalty (the $\beta$ term in the objective below) rather than optimizing reward without bound.

RLHF with PPO: the objective, and why it’s expensive#

The objective. Maximize expected reward while staying close (in KL divergence) to the reference (SFT) policy, so the model doesn’t drift into degenerate, reward-hacking outputs the reward model happens to score highly but no human actually wants:

$$\max_\theta \; \mathbb{E}_{x \sim D,\, y \sim \pi_\theta(\cdot|x)}\Big[r_\phi(x,y) - \beta \cdot \text{KL}\big(\pi_\theta(\cdot|x) \,\|\, \pi_{\text{ref}}(\cdot|x)\big)\Big]$$

PPO (Schulman et al., 2017) optimizes this via a clipped surrogate objective that prevents any single update from moving the policy too far from where it collected its current batch of experience -- necessary because RLHF’s data is on-policy (generated by the current model, then scored), and stale, overly large updates on stale data are unstable:

$$\mathcal{L}_{\text{PPO}} = \mathbb{E}\Big[\min\big(\rho_t \hat{A}_t,\; \text{clip}(\rho_t, 1{-}\epsilon, 1{+}\epsilon)\hat{A}_t\big)\Big], \qquad \rho_t = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$$

Why this is operationally expensive relative to what’s below. A single PPO training step requires: generating rollouts from the current policy (an expensive decode pass), scoring them with the reward model (a second model’s forward pass), computing advantages (often via a separate learned value model, a third model), and only then taking a gradient step -- four models resident in memory simultaneously (policy, reference, reward, value) for a method whose data collection step is itself sequential autoregressive generation. This operational cost, more than any quality ceiling, is the direct motivation for DPO below.

In plain terms: picture training a dog with treats, but also keeping it on a leash so it doesn’t wander off and do something bizarre purely to get more treats. The reward model is the treat-giver, the KL penalty is the leash, and the “reference model” is simply a memory of how the dog used to behave before any treats were involved, so you can tell if the leash is being pulled too hard. PPO needs a treat-giver, a leash-checker, a memory of old behavior, and a fourth helper (the value model) that tries to predict how many treats a given action will earn before it’s even tried -- four separate things to keep track of and pay for, just to train one dog.

The real result this objective produced: InstructGPT (Ouyang et al., 2022), the paper that made this whole pipeline standard. Applying exactly the objective above -- SFT, then a reward model trained on ranked human comparisons, then PPO against it with a KL penalty back to the SFT policy -- OpenAI’s own reported numbers are worth reading directly rather than paraphrased: on their held-out prompt distribution, 175B InstructGPT outputs were preferred to 175B GPT-3 outputs 85% ± 3% of the time, and preferred 71% ± 4% of the time even against a few-shot-prompted 175B GPT-3 (i.e., PPO-tuning beat the best available prompting trick on the un-tuned base model, not just the un-prompted one). The headline scale result: their 1.3B-parameter InstructGPT model was preferred by human labelers over the 175B GPT-3 base model, despite having over 100x fewer parameters -- the single clearest piece of published evidence that alignment quality and raw scale are different axes, not the same axis measured two ways. On TruthfulQA, the tuned model’s hallucination rate on adversarial prompts dropped from roughly 41% down to about 21%. None of this was free: the paper documents a measurable alignment tax -- small regressions on some public NLP benchmark scores relative to plain GPT-3 -- which they mitigate by mixing in a fraction of the original pretraining loss during the PPO step itself (a variant they call PPO-ptx, with the pretraining-loss mixing coefficient set to $\gamma = 27.8$ after their own sweep), directly trading a little of the RLHF objective’s pull away from the base model’s original distribution for smaller regressions elsewhere -- the KL term in the objective above is the general mechanism, PPO-ptx’s extra pretraining-loss term is InstructGPT’s own specific, published refinement of it. One more detail worth carrying forward: the human preference labels themselves weren’t unanimous -- the paper reports roughly a 27% disagreement rate between labelers on which of two outputs was better, a concrete, quantified instance of the inter-rater-agreement issue chapter 4 raises for human feedback pipelines in general, present in the very paper that established this whole methodology.

DPO: the derivation that removes the reward model and the RL loop entirely#

The key insight (Rafailov et al., 2023). The KL-constrained RLHF objective above has a closed-form optimal solution -- you can write down exactly what $\pi_\theta$ should look like at the objective’s maximum, without ever running RL:

$$\pi^*(y|x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y|x) \exp\!\left(\frac{1}{\beta} r_\phi(x,y)\right), \qquad Z(x) = \sum_y \pi_{\text{ref}}(y|x)\exp\!\left(\frac{1}{\beta}r_\phi(x,y)\right)$$

Solve this expression for the reward function instead of the policy -- algebraically invert it:

$$r_\phi(x,y) = \beta \log\frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta \log Z(x)$$

The step that makes this useful in practice: substitute this expression for $r_\phi$ back into the Bradley-Terry preference model from the reward-modeling section above. Because Bradley-Terry only ever needs the difference $r_\phi(x,y_w) - r_\phi(x,y_l)$, and $\beta \log Z(x)$ depends only on the prompt $x$ (not on which of $y_w, y_l$ you’re looking at), the intractable partition function $Z(x)$ cancels exactly:

$$P(y_w \succ y_l \mid x) = \sigma\!\left(\beta \log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)$$

This is now a loss computable directly from the policy model’s own log-probabilities -- no separate reward model, no rollout generation, no RL algorithm at all. Training becomes ordinary supervised learning on static (prompt, chosen, rejected) triples:

$$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x,y_w,y_l)}\left[\log \sigma\!\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right]$$

Worked example. Suppose for a given prompt, the current policy assigns log-probability $-8.0$ to the chosen response and the frozen reference model assigns it $-9.0$ (the policy has already moved toward liking this response more than the reference did, a good sign) -- and for the rejected response, policy log-prob $-6.0$ versus reference $-6.2$ (barely moved). With $\beta = 0.1$: the policy-vs-reference log-ratio for chosen is $-8.0 - (-9.0) = 1.0$, scaled by $\beta$: $0.1$. For rejected: $-6.0-(-6.2) = 0.2$, scaled: $0.02$. The DPO score is $\sigma(0.1 - 0.02) = \sigma(0.08) \approx 0.520$ -- barely past chance, so the loss $-\log(0.520) \approx 0.654$ is still substantial, correctly signaling the model hasn’t yet separated chosen from rejected nearly enough, even though it moved in the right direction on the chosen response.

Reading that same example one line at a time, no symbols required. Log-probability is just a way of writing “how likely the model thought this response was” where less negative is more likely -- think of it as a confidence score where $-6$ beats $-8$.

1. Chosen response: the policy now rates it $-8.0$; the old (reference) model rated it $-9.0$. Since $-8.0$ is less negative than $-9.0$, the policy grew more confident in the chosen response than the reference model was. Subtracting gives $1.0$ -- that’s “how much more confident,” in raw log terms.
2. Rejected response: the policy rates it $-6.0$; the reference rated it $-6.2$. Barely any change: subtracting gives $0.2$.
3. Multiply both by $\beta=0.1$ (a dial that controls how aggressively the model is allowed to move away from the reference at all) -- chosen becomes $0.1$, rejected becomes $0.02$.
4. Subtract rejected’s number from chosen’s number: $0.1 - 0.02 = 0.08$. This single number is “how much more the model has shifted toward the good response than toward the bad one” -- the bigger this is, the more DPO considers the update to be going in the right direction.
5. Feed that $0.08$ into the sigmoid function ($\sigma$), which is just a squashing tool that turns any number into something between 0 and 1 -- 0 means “definitely wrong,” 1 means “definitely right,” 0.5 means “no better than a coin flip.” Here it comes out to about 0.520, i.e. barely better than a coin flip.
6. The loss is $-\log$ of that number -- a bookkeeping trick so that “close to 1” (good) produces a small loss and “close to 0” (bad) produces a large one. $-\log(0.520)$ comes out to about 0.654, a fairly large loss, which is the model correctly being told: “you moved in the right direction, but not nearly enough -- keep pushing.”

In plain terms: DPO’s whole trick is realizing you don’t need to hire a food critic (the reward model) to know whether a chef has improved. You can just compare how much more the chef now leans toward one dish versus another, relative to how they used to cook before any coaching -- and that shift, on its own, tells you everything the critic would have told you, without ever paying for a critic at all.

# TRL's DPOTrainer -- this entire derivation, in three lines of usage
from datasets import load_dataset
from trl import DPOTrainer, DPOConfig

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")   # (prompt, chosen, rejected) triples
trainer = DPOTrainer(
    model="Qwen/Qwen3-0.6B",              # policy, initialized from the SFT checkpoint
    args=DPOConfig(beta=0.1),              # the same beta from the derivation above
    train_dataset=dataset,
    # ref_model omitted: TRL creates a frozen copy of the initial policy automatically when not provided
)
trainer.train()

DPO’s variants, and what each one is actually fixing#

method	what it changes vs. plain DPO	the specific problem it targets
IPO (Azar et al., 2023)	replaces the logistic loss with a squared-error form on the same log-ratio quantity	DPO can, in the unregularized limit, push chosen/rejected probability arbitrarily far apart even past the point of usefully separating them (a known overfitting mode on deterministic/low-diversity preference data); IPO’s bounded loss shape resists this
KTO (Ethayarajh et al., 2024)	trains on unpaired binary desirable/undesirable labels instead of matched (chosen, rejected) pairs, motivated by prospect theory (loss aversion)	paired preference data is expensive and often unavailable at scale; KTO can use the far more common “thumbs up/down” signal directly
ORPO (Hong et al., 2024)	folds an odds-ratio preference penalty directly into the SFT loss itself, no separate reference model or separate stage at all	removes an entire training stage (reference-model-based preference tuning) by combining SFT and preference alignment into one pass

Synthetic data and distillation: teaching a small model what a large one knows#

Synthetic data generation for post-training. Human-written instruction/response pairs are expensive and slow to collect at the scale modern SFT/DPO datasets need. The now-standard alternative: use a strong existing model to generate candidate instructions and/or responses (Self-Instruct, Wang et al., 2022; Alpaca, Taori et al., 2023, distilling from a proprietary frontier model into a small open one), or to generate critiques and revisions of a model’s own outputs against a written set of principles rather than human preference labels at all -- Constitutional AI (Bai et al., 2022, Anthropic) is the clearest published version of this: a model critiques and revises its own responses against a set of written principles, and that self-critique data trains the model further, substituting AI-generated preference/critique signal (RLAIF) for a large fraction of what would otherwise require human labelers, though Anthropic’s own paper is explicit that human oversight remains part of the pipeline, not a full replacement for it.

Distillation: training a smaller model to match a larger one’s output distribution, not just its hard labels:

$$\mathcal{L}_{\text{distill}} = \text{KL}\big(p_{\text{teacher}}(\cdot|x) \,\|\, p_{\text{student}}(\cdot|x)\big)$$

matching the full soft probability distribution over the vocabulary carries more signal than matching only the single correct next token, because the teacher’s relative confidence across incorrect tokens (“this is almost as plausible as the right answer” vs. “this is nonsense”) is itself informative.

The real, current example worth knowing. DeepSeek’s own R1 report (DeepSeek-AI, 2025) documents distilling a large reasoning model’s long chain-of-thought outputs directly into much smaller dense models (from 1.5B up to 70B parameters, based on Qwen and Llama checkpoints) purely via SFT on the larger model’s generated reasoning traces -- no RL step needed for the smaller models at all -- and reports these distilled small models outperforming RL-trained models of similar or even larger size on reasoning benchmarks, a concrete, recent demonstration that distilling a strong teacher’s behavior can beat training a same-size model from scratch with the “proper” full pipeline, at a fraction of the training cost.

Model merging: combining checkpoints without any further training#

Model soups (Wortsman et al., 2022): simply average the weights of several independently fine-tuned checkpoints (same base model, different hyperparameters or data orders) -- $\theta_{\text{merged}} = \frac{1}{k}\sum_i \theta_i$ -- and the averaged model frequently matches or beats the best individual checkpoint on held-out evaluation, at zero additional training cost, because independently-trained fine-tunes often land in different-but-compatible regions of a wide loss basin rather than truly conflicting ones. The paper’s own headline number: their “greedy soup” recipe (start from the single best checkpoint in a hyperparameter sweep, then greedily add each additional checkpoint only if it doesn’t hurt held-out accuracy) applied to a ViT-G model pretrained on JFT and fine-tuned on ImageNet produced 90.94% top-1 ImageNet accuracy -- a new state of the art at publication, beating every individual model in the sweep it was built from, using zero additional training and zero additional inference cost over any single one of those models -- the direct empirical case for “don’t just throw away the checkpoints that didn’t win the sweep.” Task arithmetic (Ilharco et al., 2022) generalizes this to combining capabilities: compute a “task vector” as the weight difference between a fine-tuned checkpoint and its base ($\tau = \theta_{\text{finetuned}} - \theta_{\text{base}}$), and add or subtract these vectors from a base model to add or remove a capability -- adding a “coding” task vector and a “safety” task vector to the same base model is a real, used technique for combining two independently-developed fine-tunes without retraining on their combined data, though it degrades as task vectors become less related or more numerous, and doesn’t substitute for a real multi-task training run when tasks strongly conflict.

LoRA and QLoRA: the math and the real memory savings#

LoRA’s core idea (Hu et al., 2021): freeze the pretrained weight matrix $W_0$ entirely, and learn only a low-rank update:

$$W' = W_0 + \Delta W = W_0 + \frac{\alpha}{r} BA, \qquad B \in \mathbb{R}^{d\times r},\ A \in \mathbb{R}^{r\times d},\ r \ll d$$

Worked parameter-count example. A single attention projection matrix in a 7B-class model might be $d = 4096$ square, i.e. $4096^2 \approx 16.8\text{M}$ parameters if fully fine-tuned. A LoRA adapter at rank $r=16$ instead adds only $B$ ($4096 \times 16$) plus $A$ ($16 \times 4096$) $= 2 \times 4096 \times 16 \approx 131{,}072$ parameters -- about 0.78% of the full matrix’s parameter count, for that one projection. Applied across all attention and FFN projections in a real 7B model, LoRA typically trains well under 1% of total parameters while recovering most of full-fine-tuning’s task performance in practice, and -- the detail that matters for serving -- because $BA$ is the same shape as $W_0$, it can be merged directly into the base weights after training ($W' = W_0 + \frac{\alpha}{r}BA$), so a LoRA-tuned model runs at exactly the same inference speed as the untuned base model, with zero added latency.

In plain terms: LoRA is like correcting a massive textbook with a thin, removable sticky-note insert instead of reprinting the whole book. The original textbook (the frozen weights) never gets touched or reprinted; you only need to write and store the small sticky note (the low-rank update), and when you’re done, you can even glue the sticky note permanently into the page so reading the book afterward is no slower than reading the original.

QLoRA (Dettmers et al., 2023) goes further: load the frozen base model in 4-bit (NF4, the same quantization format from the companion long-context document’s chapter 8) and train LoRA adapters in bf16 on top of that frozen 4-bit base -- the adapter itself still trains at full precision, but the (far larger) frozen base weights need only a quarter of the memory a bf16 base would, making it possible to fine-tune a 65B-parameter model on a single 48GB GPU, which the QLoRA paper demonstrated directly.

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.3", torch_dtype="bfloat16")
full_params = sum(p.numel() for p in base_model.parameters())

lora_config = LoraConfig(
    r=16, lora_alpha=32,                      # alpha/r = 2.0, the scaling factor from the W' formula above
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
)
peft_model = get_peft_model(base_model, lora_config)
trainable = sum(p.numel() for p in peft_model.parameters() if p.requires_grad)
print(f"trainable: {trainable:,} / {full_params:,} = {trainable/full_params:.3%}")   # typically well under 1%

# QLoRA: 4-bit frozen base + LoRA adapters trained in bf16 on top
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                                  bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-70b-hf", quantization_config=bnb_config, device_map="auto")
model = prepare_model_for_kbit_training(model)     # casts norms to fp32, enables gradient checkpointing, etc.
model = get_peft_model(model, LoraConfig(r=64, lora_alpha=16, task_type="CAUSAL_LM"))

Production tooling: the rest of the list, at real depth#

task	real tool / method	the thing that actually matters
Dataset versioning and audit trails	DVC, or a data lakehouse with immutable snapshots per training run, tagged with the exact commit of filtering/curation code that produced it	”which exact data mixture produced checkpoint X” must be answerable months later, both for debugging a regression and for compliance/audit purposes
Training run comparison	W&B/MLflow run comparison dashboards, tracking eval-suite scores (ch. 4) alongside loss curves per run	a lower training loss does not imply a better aligned model -- DPO/RLHF loss curves are notoriously uninformative about downstream quality compared to actual eval scores
Safe rollback of bad checkpoints	blue/green model deployment, keeping the previous production checkpoint hot and ready behind a feature flag	reward hacking (above) or a bad data batch can pass loss-curve sanity checks and still produce a measurably worse assistant only visible in downstream evals or live traffic -- rollback needs to be a one-flag operation, not a redeploy
Human feedback pipelines	structured labeling platforms (Surge AI, Scale AI, or in-house tooling) with inter-rater agreement tracking	low agreement between human raters on a preference task is itself a signal the task or instructions are ambiguous, not just noise to average away -- measure it, don’t ignore it
Policy safety checks	a fixed suite of known-hard refusal/safety prompts run against every candidate checkpoint before promotion	catches an alignment regression (a model that got more helpful on borderline requests than intended) before it reaches production, not after
Automated eval gates for release	CI-style pipelines that block a model promotion unless it clears a fixed score threshold on the eval suite from chapter 4	without an automated gate, “ship it, it looks good” is a subjective per-release judgment call instead of a reproducible bar

📚 Further reading

• Ouyang et al., 2022 -- Training Language Models to Follow Instructions with Human Feedback (InstructGPT)
• Schulman et al., 2017 -- Proximal Policy Optimization Algorithms
• Rafailov et al., 2023 -- Direct Preference Optimization
• Azar et al., 2023 -- A General Theoretical Paradigm to Understand Learning from Human Preferences (IPO)
• Ethayarajh et al., 2024 -- KTO: Model Alignment as Prospect Theoretic Optimization
• Hong et al., 2024 -- ORPO: Monolithic Preference Optimization without Reference Model
• Gao et al., 2023 -- Scaling Laws for Reward Model Overoptimization
• Bai et al., 2022 -- Constitutional AI: Harmlessness from AI Feedback (Anthropic)
• Hu et al., 2021 -- LoRA: Low-Rank Adaptation of Large Language Models
• Dettmers et al., 2023 -- QLoRA: Efficient Finetuning of Quantized LLMs
• Wortsman et al., 2022 -- Model Soups
• Ilharco et al., 2022 -- Editing Models with Task Arithmetic
• DeepSeek-AI, 2025 -- DeepSeek-R1: Incentivizing Reasoning Capability via Reinforcement Learning

03 · Adaptation#

❓ Why this topic. Pre-training and post-training produce a general-purpose assistant; adaptation is what makes it your product -- grounded in your documents, able to call your tools, constrained to your output format, and able to say “I don’t know” instead of confidently guessing. This is also the phase with the fastest iteration loop (a retrieval index or a system prompt can change in minutes, a model weight cannot), which is exactly why it’s usually the first lever pulled when a production system underperforms, and why getting its failure modes wrong is so easy to miss.

The decision this chapter’s outline skipped: RAG vs. long context#

Before reaching for retrieval at all, the real first question is whether retrieval is even the right tool. The companion long-context document covers this from the model-capability side (chapter 1’s context-length techniques, chapter 2’s KV-cache cost of holding a long context resident); the adaptation-side framing:

	RAG (retrieve, then generate)	long-context (stuff everything in the prompt)
Cost per query	cheap: embed query + vector search + small generation prompt	expensive: full prompt reprocessed (or cached, ch. 2 of the companion doc) every query
Freshness	trivial -- update the index, no retraining or re-prompting design needed	requires re-including updated content in every future prompt
Precision on huge corpora	scales to millions of documents via approximate nearest-neighbor search	bounded by the model’s actual context window, and by lost-in-the-middle degradation (companion doc, ch. 1) well before that limit
Precision on a small, fixed corpus that fits in context	retrieval can miss relevant chunks (embedding similarity is imperfect)	the model sees everything; no retrieval-miss failure mode at all
Auditability	easy -- you can show exactly which chunks were retrieved and cite them	harder -- no clean way to say which part of a huge prompt the answer came from

The practical rule. If the relevant corpus is small enough to fit in context and mostly static, long-context wins on precision (no retrieval-miss possible) and simplicity. If the corpus is large, changes frequently, or needs per-query filtering (permissions, freshness, source attribution), RAG wins on cost and freshness. Most production systems at real scale end up doing both: RAG to narrow millions of documents down to a relevant handful, then stuffing those into a long context window for the actual generation -- not a binary choice.

What the paper that named this pattern actually reported. Lewis et al. (2020), the paper that coined “Retrieval-Augmented Generation,” combined a dense retriever (encoding query and Wikipedia passages into the same vector space, retrieving the top-5) with a BART generator conditioning on the retrieved passages, trained end-to-end. Their reported numbers, against the strongest non-retrieval baselines available at publication: 44.5 exact-match on Natural Questions, 56.8 on TriviaQA, and 68.0 on WebQuestions -- state-of-the-art open-domain QA results at the time, from a generator that never saw these facts during its own pretraining, purely by conditioning on retrieved text. On a harder, generation-oriented test (open-ended Jeopardy-style question generation), human evaluators judged RAG’s outputs more factual than a parametric-only BART baseline’s in 42.7% of cases, versus BART being judged more factual in only 7.1% -- a direct, measured demonstration of retrieval reducing hallucination rather than just improving a multiple-choice-style accuracy number. One more detail worth carrying into chapter 3’s hybrid-search section above: the paper found dense retrieval outperformed BM25 on every task except FEVER (fact verification), where FEVER’s entity-centric claims specifically rewarded exact keyword overlap -- a real, paper-level data point for why hybrid search (rather than dense-only) remains the safer production default even after fifteen years of steady dense-retrieval improvement.

Embeddings and retrieval: the math, worked#

Cosine similarity, the standard measure of “how related are these two pieces of text” once both are embedded as vectors:

$$\text{sim}(a, b) = \frac{a \cdot b}{\|a\|\|b\|} = \cos(\theta)$$

Worked example. Two 4-dimensional (toy-sized, real embedding models use 384–3072 dimensions) embeddings for “the cat sat on the mat” and “a dog lay on the rug”: $a = [0.8, 0.1, 0.3, 0.5]$, $b = [0.7, 0.2, 0.4, 0.4]$. Dot product: $0.8{\times}0.7 + 0.1{\times}0.2 + 0.3{\times}0.4 + 0.5{\times}0.4 = 0.56+0.02+0.12+0.20 = 0.90$. Norms: $\|a\| = \sqrt{0.64+0.01+0.09+0.25} = \sqrt{0.99}\approx0.995$, $\|b\| = \sqrt{0.49+0.04+0.16+0.16}=\sqrt{0.85}\approx0.922$. $\text{sim}(a,b) \approx 0.90/(0.995\times0.922) \approx 0.981$ -- high similarity, correctly reflecting that both sentences describe an animal resting on a surface, despite sharing zero words.

In plain terms: each number in a vector like $a$ or $b$ is just one “aspect” the model measured about the sentence’s meaning (nothing you’d recognize by name -- not “is it about animals,” just some learned dimension). The dot-product step is literally “multiply matching aspects together and add them up” -- if two sentences are similar, their matching aspects tend to point the same direction, so the products come out positive and the sum is large. The square-root steps are just a way of measuring each vector’s own “length,” so that a sentence described in unusually strong or unusually weak terms doesn’t unfairly score higher or lower just because its numbers happen to be bigger or smaller overall -- dividing by both lengths puts every comparison on the same 0-to-1-ish scale. The end result, 0.981, is close to the maximum possible score of 1.0, which is the model’s way of saying “these two sentences are about nearly the same thing” -- even though a keyword search comparing the actual words “cat” and “dog” would have found zero overlap at all.

In plain terms: cosine similarity is like judging how similar two people’s movie tastes are by the direction their preferences point -- both love action-comedies and dislike documentaries -- rather than by how enthusiastically each one rates things on a 1-to-10 scale. Two reviewers can disagree on how many stars to give and still clearly like the same kinds of movies; embeddings capture that shared “direction of taste” even when the exact words used are completely different.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-base-en-v1.5")     # 768-dim, a widely used open embedding model
query_emb = model.encode("How do I reset my password?", normalize_embeddings=True)
doc_embs = model.encode(["To reset your password, go to Settings > Security...",
                          "Our refund policy allows returns within 30 days..."], normalize_embeddings=True)
scores = query_emb @ doc_embs.T          # normalized embeddings -> dot product IS cosine similarity

Chunking: the tradeoff that determines retrieval quality more than the embedding model does#

Splitting documents into retrievable units trades off two failure modes directly: chunks too large dilute a single relevant sentence with irrelevant surrounding text (hurting the embedding’s specificity and wasting context budget once retrieved); chunks too small lose the surrounding context needed to make the chunk meaningful on its own (a sentence fragment that only makes sense next to the paragraph before it). Real production defaults sit around 256–512 tokens per chunk with 10–20% overlap between consecutive chunks (so information straddling a chunk boundary isn’t lost to either chunk), though semantic chunking (splitting at detected topic-shift boundaries via embedding-similarity drops between consecutive sentences, rather than a fixed token count) measurably improves retrieval precision on heterogeneous documents at the cost of a preprocessing pass to compute those boundaries.

Hybrid search: combining keyword and semantic retrieval with Reciprocal Rank Fusion#

Why neither alone is enough. Dense embedding search (cosine similarity above) captures semantic meaning but can miss exact keyword/identifier matches (a product SKU, an error code, a proper noun) that a simple keyword index would catch trivially. BM25 (Robertson & Zaragoza, 2009, a classical, still-standard keyword-ranking function) scores documents by term frequency, weighted to penalize very common terms and reward rare, distinctive ones -- exactly what dense embeddings are weakest at. The actual scoring function, term by term:

$$\text{BM25}(D,Q) = \sum_{q \in Q} \text{IDF}(q) \cdot \frac{f(q,D)\cdot(k_1+1)}{f(q,D) + k_1\left(1 - b + b\cdot\frac{|D|}{\text{avgdl}}\right)}, \qquad \text{IDF}(q) = \ln\!\left(\frac{N - n(q) + 0.5}{n(q)+0.5}+1\right)$$

where $f(q,D)$ is how many times term $q$ appears in document $D$, $|D|/\text{avgdl}$ is the document’s length relative to the corpus average (long documents get a length-normalization penalty via $b$, commonly 0.75, so a term isn’t over-credited just because the document is long), and $k_1$ (commonly 1.2–2.0) controls how quickly additional occurrences of the same term stop adding score -- term frequency saturates rather than growing linearly, which is what stops a document that just repeats one keyword fifty times from dominating the ranking. Worked example. A 3-term query where term $q_1$ appears in 10 of a corpus’s 10,000 documents ($\text{IDF}(q_1) = \ln(\frac{10000-10+0.5}{10+0.5}+1)\approx6.62$, a rare, distinctive term) versus term $q_2$ appearing in 5,000 of 10,000 documents ($\text{IDF}(q_2) = \ln(\frac{10000-5000+0.5}{5000+0.5}+1)\approx0.69$, a common term contributing far less to the score) -- this is the concrete mechanism behind “BM25 rewards rare, distinctive terms”: a document matching the rare term $q_1$ once outscores, term-for-term, a document matching the common term $q_2$ several times over, exactly the behavior a product-SKU or error-code lookup needs and dense cosine similarity alone doesn’t reliably guarantee.

Same idea, no formula required. “IDF” stands for inverse document frequency, and all it’s really asking is: how many other documents in the whole collection also contain this word? If a word shows up in almost every document (like “the” or, in this example, a common term appearing in half the corpus), finding it in your result tells you almost nothing about relevance -- everything has it. If a word shows up in only a handful of documents out of 10,000 (a rare, specific term -- an error code, a part number), finding it is a strong, specific signal. The formula above just converts “rare” and “common” into numbers you can add up: in this example, the rare term scores about 6.62 “relevance points” just for being rare, while the common term scores only about 0.69 -- nearly ten times less. That’s why one single mention of a distinctive keyword can outrank several mentions of a generic one: rarity, not raw repetition, is what BM25 is actually rewarding.

In plain terms: BM25 works like a detective who weighs a rare, distinctive clue (a specific fingerprint found at only one other crime scene) far more heavily than a common one (everyone at the scene happened to wear black shoes). A term that shows up almost everywhere in a corpus is weak evidence of relevance; a term that shows up almost nowhere except in the right document is strong evidence -- that’s the entire intuition behind BM25’s IDF weighting.

Reciprocal Rank Fusion (RRF) combines two independently-ranked result lists without needing their scores to be on the same scale (BM25 scores and cosine similarities aren’t comparable numbers at all):

$$\text{RRF}(d) = \sum_{\text{ranker} \in \{\text{BM25}, \text{dense}\}} \frac{1}{k + \text{rank}_{\text{ranker}}(d)}$$

with $k$ (commonly 60) a smoothing constant that dampens the effect of a single ranker’s extreme top rank.

Worked example. Document $d$ ranks 1st under BM25 and 8th under dense search; document $e$ ranks 5th under BM25 and 2nd under dense search, with $k=60$: $\text{RRF}(d) = \frac{1}{61} + \frac{1}{68} \approx 0.0164 + 0.0147 = 0.0311$. $\text{RRF}(e) = \frac{1}{65}+\frac{1}{62} \approx 0.0154+0.0161=0.0315$ -- $e$ narrowly outranks $d$ in the fused result despite neither being the single best result under either method alone, exactly the point: RRF rewards documents that rank reasonably well across both signals over documents that rank perfectly under one and poorly under the other.

In plain terms: each document gets one score per ranking method, computed as “1 divided by (roughly) its position in that ranking” -- so 1st place is worth a lot, 8th place is worth much less, and the gap between 1st and 2nd matters more than the gap between 8th and 9th. Add those two small fractions together for each document, and whichever document has the bigger total wins the combined ranking. Document $d$ was brilliant on one search method (1st place) but mediocre on the other (8th) -- a bit of a one-trick performer. Document $e$ was never the single best on either method, but was solidly good on both (5th and 2nd). Adding the fractions up, $e$‘s “solid on both” profile just barely beats $d$‘s “amazing on one, weak on the other” profile -- which is exactly the behavior you want from a system meant to reward documents that are genuinely relevant by more than one measure, not just a specialist that got lucky on one.

In plain terms: imagine combining a close friend’s restaurant tips with a big review site’s star rankings. A restaurant your friend raves about and that ranks reasonably well on the review site is probably a safer bet than one that’s the review site’s #1 pick but that your friend has never once mentioned -- being liked by more than one source, even imperfectly, beats being loved by only one.

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[str]:
    # rankings: one ranked doc-id list per retrieval method (e.g. [bm25_ranked_ids, dense_ranked_ids])
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

# Qdrant's real hybrid-search API: dense + sparse (BM25-like) vectors in one query, fused server-side
from qdrant_client import QdrantClient, models

client = QdrantClient(":memory:")
client.create_collection(
    collection_name="docs",
    vectors_config={"dense": models.VectorParams(size=768, distance=models.Distance.COSINE)},
    sparse_vectors_config={"bm25": models.SparseVectorParams()},
)
# after upserting both dense and sparse vectors per document:
results = client.query_points(
    collection_name="docs",
    prefetch=[
        models.Prefetch(query=dense_query_vector, using="dense", limit=20),
        models.Prefetch(query=sparse_query_vector, using="bm25", limit=20),
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),   # Qdrant's built-in RRF, same math as above
    limit=10,
)

Reranking: why a second, slower pass is worth it#

Bi-encoders vs. cross-encoders. The embedding search above is a bi-encoder: query and document are embedded independently, so similarity is a single dot product computable in advance for every document, letting search scale to millions of documents via approximate nearest-neighbor indices. A cross-encoder instead feeds the query and a candidate document together into one model, letting attention directly compare query tokens against document tokens rather than compressing each into one fixed vector first -- meaningfully more accurate at judging true relevance, but $O(n)$ per query (must run the model once per candidate document, cannot precompute), so it’s only tractable as a second-stage reranker over the top-k candidates a cheap bi-encoder retrieval already narrowed down, not as the retrieval mechanism itself.

In plain terms: this is exactly how resume screening works at scale. A fast first pass just checks keyword matches on paper for every applicant (the bi-encoder, cheap enough to run on thousands of resumes at once) -- then only the shortlisted top candidates get a slower, careful interview where they’re compared directly, side by side, against each other and the actual job (the cross-encoder). Nobody interviews all ten thousand applicants directly; that’s the whole reason the cheap first pass exists.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
candidates = ["To reset your password, visit Settings.", "Refunds are processed within 5 business days."]
scores = reranker.predict([("How do I reset my password?", c) for c in candidates])
reranked = [c for _, c in sorted(zip(scores, candidates), reverse=True)]

Retrieval quality: the metrics that actually predict downstream answer quality#

• Recall@k: of all truly relevant documents for a query, what fraction appear in the top $k$ retrieved? The floor metric -- if the relevant chunk isn’t retrieved at all, no amount of generation quality recovers it.
• MRR (Mean Reciprocal Rank): $\frac{1}{|Q|}\sum_q \frac{1}{\text{rank of first relevant result for } q}$ -- rewards relevant results appearing early, not just present somewhere in the top-k.
• nDCG (normalized Discounted Cumulative Gain): like MRR but accounts for graded relevance (some documents more relevant than others, not just binary relevant/irrelevant) and discounts relevance found lower in the ranking logarithmically rather than by strict rank position alone.

The production point. These are measurable before generation ever runs, which matters because a downstream “wrong answer” eval failure (chapter 4) is ambiguous -- was retrieval’s fault (didn’t fetch the right chunk) or generation’s fault (had the right chunk, answered wrong anyway)? Tracking retrieval metrics separately from end-to-end answer-quality metrics is what makes that root-cause question answerable at all instead of a guess.

Structured outputs: constrained decoding, not just prompting-and-hoping#

The problem with “just ask nicely for JSON.” A model can be well-aligned and still occasionally emit malformed JSON, an extra trailing comma, or a field name that doesn’t match the schema -- costly in any pipeline that parses the output programmatically downstream.

The real mechanism: grammar-constrained decoding. Rather than hoping the model’s free-form output happens to be valid, build a finite-state automaton (or a more general context-free grammar) representing exactly which tokens are valid next given the schema and everything generated so far, and mask the logits at every decoding step so only grammar-valid tokens can be sampled at all -- this is a direct application of the masking mechanism from the transformer notes (ch. 3): valid-continuation tokens keep their logits, everything else gets set to $-\infty$ before softmax, exactly as a causal mask blocks future tokens.

# Outlines: regex/JSON-schema-constrained generation, enforced at the logit level every step
import outlines
from pydantic import BaseModel

class SupportTicket(BaseModel):
    category: str
    priority: int
    summary: str

model = outlines.models.transformers("meta-llama/Llama-3.1-8B-Instruct")
generator = outlines.generate.json(model, SupportTicket)   # builds the constraining automaton from the schema
ticket = generator("Customer says the app crashes every time they upload a photo.")
# ticket is guaranteed parseable as SupportTicket -- malformed JSON is not a reachable output at all

In plain terms: grammar-constrained decoding is like being handed a form with boxes that only accept a number here, only accept one of three checkboxes there -- you physically cannot write a name where a number belongs, because the form itself blocks it. The model isn’t being asked nicely to produce valid JSON, it’s handed a fill-in-the-blank form where the invalid options were never on the page to begin with.

# instructor: the same idea, layered on top of a real chat-completion API instead of raw logit access
import instructor
from openai import OpenAI
from pydantic import BaseModel

client = instructor.from_openai(OpenAI())

class SupportTicket(BaseModel):
    category: str
    priority: int

ticket = client.chat.completions.create(
    model="gpt-4o", response_model=SupportTicket,
    messages=[{"role": "user", "content": "Customer says the app crashes on photo upload."}],
)   # instructor retries with the validation error fed back to the model if parsing fails, rather than crashing

Tool calling and agentic loops (ReAct)#

Tool calling mechanics. The model is given a set of tool schemas (name, description, parameter types) in its prompt or a dedicated API field; instead of generating a free-form answer, it can generate a structured tool-call request (constrained exactly as above), the calling code executes the real function, and the result is fed back into the conversation for the model to continue from.

ReAct (Yao et al., 2022, “Reasoning + Acting”) is the standard pattern for multi-step tool use: interleave explicit reasoning (“I need to check the user’s order history first”) with tool calls and their results, in a loop, until the model decides it has enough information to answer -- rather than a single tool call, this is what lets a model handle a task that genuinely requires several dependent steps (look up the order, check the refund policy, then decide). The paper’s own reported numbers, worth citing directly: on HotpotQA and FEVER, giving the model access to a simple Wikipedia-search API and letting it interleave reasoning with lookups reduced the fact-hallucination and error-propagation failures the authors specifically documented in pure chain-of-thought prompting (which has no way to correct a wrong “fact” it invented three reasoning steps earlier, since it never checks anything against the world). On the two interactive decision-making benchmarks tested (ALFWorld, a simulated household task-completion environment, and WebShop, a web-navigation shopping task), ReAct -- using only one or two in-context examples, no task-specific training at all -- outperformed imitation-learning and reinforcement-learning baselines trained on roughly 100,000 task instances each, by an absolute success-rate margin of 34 percentage points on ALFWorld and 10 points on WebShop. That comparison is the concrete case for the ReAct pattern’s actual value: a few-shot-prompted loop beating methods trained on five orders of magnitude more task-specific data, because the interleaved reasoning-and-acting structure, not scale, was what the task needed.

# a minimal ReAct-style loop: reason, call a tool, observe, repeat, until the model stops calling tools
def react_loop(model, tools: dict, user_query: str, max_steps: int = 6):
    messages = [{"role": "user", "content": user_query}]
    for _ in range(max_steps):
        response = model.chat(messages, tools=list(tools.values()))
        if response.tool_calls:
            for call in response.tool_calls:
                result = tools[call.name](**call.arguments)          # execute the real function
                messages.append({"role": "tool", "tool_call_id": call.id, "content": str(result)})
        else:
            return response.content                                   # model decided it has enough to answer
    return "Max reasoning steps exceeded without a final answer."

⚠️ The real production failure mode. Agentic loops can silently run away -- a model that keeps calling a tool with slightly different arguments hoping for a different result, never converging. Production agent frameworks cap steps (as above), track per-run cost, and log the full reasoning trace, precisely because a runaway loop is a real cost and reliability incident, not a hypothetical.

In plain terms: an uncapped agent loop is like an intern who keeps re-trying the same broken approach to a task, slightly tweaked each time, convinced the next attempt will finally work -- burning hours and expenses with nothing to show for it unless someone sets a hard limit on how many attempts they’re allowed before stopping to ask for help.

Routing between models#

Not every query needs the largest, most expensive model. A lightweight classifier (or even a small model prompted to rate query complexity) routes simple queries to a small, cheap, fast model and complex ones to a larger one -- real production systems (e.g. commercial routing layers built on top of multiple hosted models) report meaningful cost reduction with minimal quality loss on the (large fraction of) queries that never needed the expensive model at all.

def route(query: str, complexity_classifier, cheap_model, expensive_model):
    complexity = complexity_classifier.predict(query)     # e.g. a small trained classifier, or a fast small-LLM judge
    return cheap_model if complexity < 0.5 else expensive_model

Confidence and abstention#

Entropy-based abstention (companion long-context document, ch. 10’s entropy discussion applies directly here) is the cheapest signal, but in an adaptation context the more useful signal is often groundedness: does the generated answer actually overlap with what was retrieved, or did the model answer from its own prior despite having relevant context available (the exact failure mode context-aware decoding, companion document ch. 4, targets directly). A production abstention policy typically combines both: low retrieval-confidence (nothing scored above a similarity threshold) and low output-token confidence together are a much stronger “don’t answer” signal than either alone.

Production tooling: the rest of the list, at real depth#

task	real tool / method	the thing that actually matters
Vector databases	Qdrant, Pinecone, Weaviate, or pgvector inside an existing Postgres deployment	pgvector is the right default when you already run Postgres and don’t need purpose-built vector-DB scale; a dedicated vector DB earns its complexity past tens of millions of vectors or when hybrid search / filtering at scale matters
Prompt/version management	prompt templates checked into version control alongside code, with the exact template hash logged per production request	”which prompt version produced this output” must be answerable during an incident, exactly like dataset versioning in chapter 2
Guardrails and policy enforcement	Llama Guard (a purpose-trained classifier model for input/output safety screening) or NeMo Guardrails (a programmable rail/policy layer) run as a pre- or post-processing step around the main model	a guardrail is a separate small model or ruleset, not a prompt instruction to the main model -- prompt-only “don’t say X” instructions are exactly the kind of instruction jailbreak techniques target and bypass
Latency optimization	caching (exact-match and semantic caching of repeated queries), and everything in the companion long-context document’s chapters 2–3	a retrieval+rerank+generate pipeline has three sequential latency contributors; profile each separately before assuming the LLM call itself is the bottleneck
Observability for agent/tool workflows	structured tracing (e.g. OpenTelemetry-style spans) per reasoning step and tool call, not just a final input/output log	debugging why a multi-step agent gave a wrong answer requires seeing which step went wrong, not just the final output
User-feedback loops	explicit thumbs up/down tied back to the exact retrieval set, prompt version, and model version that produced the response	feedback that isn’t tied to the specific system configuration that produced it can’t be used to fix that configuration later

📚 Further reading

• Robertson & Zaragoza, 2009 -- The Probabilistic Relevance Framework: BM25 and Beyond
• Cormack et al., 2009 -- Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods
• Yao et al., 2022 -- ReAct: Synergizing Reasoning and Acting in Language Models
• Lewis et al., 2020 -- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Willard & Louf, 2023 -- Efficient Guided Generation for Large Language Models (Outlines)

04 · Reliability & evals#

❓ Why this topic. A benchmark score is a single number computed under conditions that may not resemble production traffic at all -- different prompt distributions, different failure costs, different notions of “correct.” This chapter is about the gap between “the eval suite passed” and “the system is actually reliable,” which is exactly the gap real incidents live in.

Offline vs. online evals: different questions, both necessary#

Offline evals run against a fixed, held-out test set before a model or prompt change ships -- reproducible, cheap to rerun, but only as representative of production as the test set was constructed to be. Online evals measure real production traffic after a change ships -- genuinely representative by definition, but confounded by everything else that changed at the same time (seasonality, a concurrent feature launch, a shift in user population) and inherently after the fact: an online eval catching a regression means it already happened to real users. Neither substitutes for the other; a mature pipeline gates releases on offline evals and continues monitoring the same metrics online after release specifically to catch what the offline test set didn’t cover.

Calibration: the math, worked with real bins#

The finding that made this a standard metric at all (Guo et al., 2017). Before this paper, the implicit assumption was that a more accurate network is, if anything, also better calibrated. Guo et al. measured reliability diagrams (binned confidence vs. binned accuracy, exactly the table below) across a range of vision architectures and found the opposite trend as networks got deeper and more accurate: their own comparison of a small, older LeNet against a modern (at the time) deep ResNet on the same benchmark showed LeNet’s confidence closely tracked its actual accuracy across bins, while the far more accurate ResNet was substantially overconfident -- higher accuracy paired with worse calibration, not better. The paper traces this specifically to modern training practices (batch normalization, weight decay choices, and simply more capacity than needed to fit the training set) rather than to depth itself, and proposes temperature scaling -- dividing logits by a single learned scalar $T > 1$ before softmax, fit on a held-out set after training -- as a simple, nearly-free fix, precisely because it doesn’t change the network’s ranking of classes (and therefore doesn’t change accuracy at all) while flattening the softmax enough to bring stated confidence back in line with actual correctness.

Expected Calibration Error (ECE) measures whether a model’s stated confidence matches its actual accuracy -- a model that says “90% confident” should be right roughly 90% of the time across many such predictions, not 60% or 99%.

$$\text{ECE} = \sum_{b=1}^{B} \frac{|S_b|}{N} \left| \text{acc}(S_b) - \text{conf}(S_b) \right|$$

Worked example. Bin predictions by stated confidence into, say, 3 bins from a batch of 100 predictions:

bin	confidence range	predictions in bin	actual accuracy	mean stated confidence	contribution
low	0.5–0.7	20	0.55	0.60	$\frac{20}{100}
mid	0.7–0.9	50	0.78	0.82	$\frac{50}{100}
high	0.9–1.0	30	0.95	0.97	$\frac{30}{100}

$\text{ECE} = 0.010+0.020+0.006 = 0.036$ -- a well-calibrated model in this example, small but nonzero systematic overconfidence in every bin. As the transformer notes (ch. 10) cover, RLHF/instruction-tuning measurably degrades calibration relative to the base model (OpenAI’s own GPT-4 report documents this directly) -- this metric is exactly how that degradation gets quantified rather than just asserted.

In plain terms: think of a weather forecaster who says “80% chance of rain” every time. If it actually rains on roughly 80% of those days, the forecaster is well-calibrated -- trustworthy in a very specific, checkable way. If it only rains on 50% of the days they said “80%,” they’re overconfident, no matter how often their forecasts get the yes/no call right overall. ECE is just this same check applied to a model’s stated confidence instead of a forecaster’s.

import numpy as np

def expected_calibration_error(confidences, correct, n_bins=10):
    bins = np.linspace(0, 1, n_bins + 1)
    ece = 0.0
    for lo, hi in zip(bins[:-1], bins[1:]):
        mask = (confidences > lo) & (confidences <= hi)
        if mask.sum() == 0:
            continue
        bin_acc = correct[mask].mean()
        bin_conf = confidences[mask].mean()
        ece += (mask.sum() / len(confidences)) * abs(bin_acc - bin_conf)
    return ece

Statistical rigor: is “82% vs. 83%” even real#

The problem. Eval suites are finite samples. A 1-point difference between two model versions on a 100-example benchmark is very often statistical noise, not a real capability difference -- and shipping (or reverting) a change based on noise is a real, avoidable production mistake.

The Wilson score interval gives a confidence interval for a proportion (accuracy) that stays well-behaved even at small sample sizes or extreme accuracy values (unlike the naive normal approximation, which can produce nonsensical intervals like “confidence interval extends below 0% accuracy”):

$$\hat{p} \pm z\sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}} \Big/ \left(1 + \frac{z^2}{n}\right), \qquad \text{centered at } \frac{\hat{p} + \frac{z^2}{2n}}{1+\frac{z^2}{n}}$$

Worked example. Model A scores 82/100 (82%), model B scores 83/100 (83%) on the same 100-example benchmark. At 95% confidence ($z=1.96$), model A’s Wilson interval is roughly [73.1%, 88.5%] and model B’s is roughly [74.2%, 89.3%] -- the intervals overlap almost entirely, meaning this single-point difference gives essentially no evidence that B is actually better than A. The same 1-point gap measured on a 10,000-example benchmark would have a far narrower interval and could be a real, meaningful signal -- sample size, not just the raw score gap, determines whether a difference is trustworthy.

What that interval is actually saying, in everyday terms. “82/100” feels precise, but it’s really a small sample telling you a rough range of what the true accuracy probably is if you tested this model on far more than 100 examples. The Wilson interval turns that one number into an honest range: “given only 100 tries and 82 successes, the real accuracy is most likely somewhere between about 73% and 89%” -- a wide range, because 100 examples just isn’t that much evidence. Model B’s range, from its 83/100, comes out to roughly 74%–89% -- almost the exact same range as model A’s. Since the two ranges overlap almost completely, you genuinely cannot tell these two models apart from this test alone -- the “83 beats 82” you’d read off a leaderboard could easily flip the other way if you reran the same test with 100 different questions. The fix isn’t a smarter formula, it’s more examples: run the same comparison on 10,000 questions instead of 100, and the range around each score tightens dramatically, to the point where a 1-point gap can become genuinely trustworthy instead of noise.

In plain terms: if you flip a coin 10 times and get 6 heads, that tells you almost nothing about whether the coin is biased. If you flip it 1,000 times and get 600 heads, that’s real evidence. The exact same “60%” result means completely different things depending on how many flips it’s based on -- and a benchmark score works exactly the same way.

from scipy.stats import norm

def wilson_interval(successes, n, confidence=0.95):
    z = norm.ppf(1 - (1 - confidence) / 2)
    p_hat = successes / n
    denom = 1 + z**2 / n
    center = (p_hat + z**2 / (2 * n)) / denom
    margin = z * ((p_hat * (1 - p_hat) / n + z**2 / (4 * n**2)) ** 0.5) / denom
    return center - margin, center + margin

print(wilson_interval(82, 100))   # (0.731..., 0.885...)
print(wilson_interval(83, 100))   # (0.742..., 0.893...) -- overlaps heavily with the above

LLM-as-judge: the methodology, and its documented biases#

The mechanism. Instead of exact-match scoring (which fails on open-ended generation -- there’s no single “correct” string for “summarize this article”), use a second, usually stronger LLM to score or compare outputs against a written rubric -- this is now the dominant methodology behind most production eval pipelines for open-ended tasks, and behind widely-used public leaderboards (e.g. Chatbot Arena’s win-rate methodology relies on a related pairwise-comparison idea, though with human rather than LLM judges).

The specific number that legitimized this whole methodology. Zheng et al.’s own MT-Bench experiment measured GPT-4-as-judge against expert human raters on 80 multi-turn questions spanning writing, reasoning, math, coding, knowledge, and roleplay, and reported GPT-4/human agreement of 85% (excluding ties) -- which the paper points out is higher than the 81% agreement measured between two human raters on the same questions. This is the number that turned “have an LLM grade your eval” from a convenience shortcut into a methodology people could actually trust: the judge isn’t matching some lower, “good enough” bar, it’s matching or exceeding the ceiling set by how consistently humans agree with each other on the same open-ended judgment.

⚠️ The well-documented biases, not hypothetical. Zheng et al. (2023, “Judging LLM-as-a-Judge”) measured several systematic biases directly: position bias (judges favor whichever response is presented first, regardless of content -- mitigated by scoring both orderings and averaging, or randomizing order across the eval set), verbosity bias (judges systematically favor longer responses even when a shorter one is equally or more correct), and self-enhancement bias (a model judging outputs, including its own family’s outputs, tends to score its own family’s style/outputs somewhat more favorably than an independent judge would). None of these make LLM-as-judge unusable -- they make it require the same rigor as any other measurement instrument: known biases, controlled for explicitly, not trusted blindly.

def pairwise_judge(judge_model, prompt, response_a, response_b, rubric):
    # position-bias control: score both orderings and require agreement, or average
    verdict_1 = judge_model.judge(prompt, response_a, response_b, rubric)   # A shown first
    verdict_2 = judge_model.judge(prompt, response_b, response_a, rubric)   # B shown first
    if verdict_1 == "A" and verdict_2 == "B":       # consistent preference for response_a regardless of order
        return "A"
    if verdict_1 == "B" and verdict_2 == "A":
        return "B"
    return "tie"   # inconsistent across orderings -> position bias is likely dominating, don't trust either verdict

Regression testing and safety/jailbreak testing#

Treat prompt and model changes exactly like code changes: a fixed regression suite of known-hard cases (previous production failures, edge cases, adversarial jailbreak attempts) run automatically on every candidate change, with a required pass threshold before promotion -- this is the direct evals-side analog of chapter 2’s automated eval gates, and the reason a previously-fixed jailbreak or previously-fixed hallucination case reappearing is caught by CI rather than by a user screenshot.

Production tooling: the rest of the list, at real depth#

task	real tool / method	the thing that actually matters
Eval harness design	lm-evaluation-harness for standardized academic benchmarks, a custom harness for product-specific tasks	standardizing on one harness (as the companion long-context document’s chapter 8 flags for lm-eval-harness specifically) matters because small scoring-implementation differences swing reported numbers by several points -- apples-to-apples comparison requires the same harness version, not just the same benchmark name
Continuous regression suites	CI-integrated eval runs on every model/prompt change, blocking merge on failure	the regression-testing point above, operationalized as infrastructure rather than a manual step someone might skip under deadline pressure
Canary releases for models	route a small percentage of real traffic to a candidate model/prompt version, compare online metrics against the incumbent before full rollout	catches issues offline evals didn’t cover, at a bounded blast radius instead of a full-traffic surprise
Error analysis workflows	structured, sampled review of failure cases, clustered by root cause (retrieval miss vs. generation error vs. tool failure, ch. 3) rather than aggregate pass/fail	an aggregate score tells you that quality dropped; clustered error analysis tells you why, which is what actually gets fixed
Monitoring drift over time	tracking input-distribution shift (e.g. Population Stability Index between this week’s and last month’s query embeddings) alongside output-quality metrics	a model can stay unchanged while the traffic it receives drifts away from what it was evaluated and aligned on -- this is a real, gradual failure mode distinct from any single bad deploy
Incident review for model failures	blameless postmortems that trace a failure back to a specific phase (bad training data? a reward-hacking artifact? a retrieval miss? a prompt regression?) using the versioning/logging infrastructure from chapters 1–3	without cross-phase logging, “why did the model say that” often can’t be answered at all months after the fact

# Population Stability Index -- a standard, simple drift detector between two distributions of a score/embedding bucket
import numpy as np

def population_stability_index(expected, actual, bins=10):
    breakpoints = np.percentile(expected, np.linspace(0, 100, bins + 1))
    e_perc, a_perc = np.histogram(expected, breakpoints)[0] / len(expected), np.histogram(actual, breakpoints)[0] / len(actual)
    e_perc, a_perc = np.clip(e_perc, 1e-6, None), np.clip(a_perc, 1e-6, None)
    return np.sum((a_perc - e_perc) * np.log(a_perc / e_perc))
# PSI < 0.1: no significant shift. 0.1-0.25: moderate shift, investigate. > 0.25: major shift, likely needs action.

# lm-evaluation-harness, the standardized academic-benchmark path referenced above and in the companion document
lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct \
    --tasks mmlu,gsm8k,truthfulqa,hellaswag --num_fewshot 5 --batch_size 8 \
    --output_path results/llama-3.1-8b-instruct/

📚 Further reading

• Guo et al., 2017 -- On Calibration of Modern Neural Networks
• Wilson, 1927 -- Probable Inference, the Law of Succession, and Statistical Inference
• Zheng et al., 2023 -- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• OpenAI, 2023 -- GPT-4 Technical Report (calibration-degradation findings)

05 · Scale engineering#

❓ Why this topic, and why this chapter is shorter than the others. Throughput vs. latency, KV cache management, quantization, batching/scheduling, distributed inference, and model/kernel-level optimization -- the core of the original outline’s scale-engineering list -- are covered in full, dedicated depth (with math, worked examples, and real production code across vLLM, SGLang, TensorRT-LLM, and mamba_ssm) in the companion document on long-context architectures, inference, decoding, and state-space models. Repeating that material here would either shortchange it or duplicate it; this chapter instead covers what that document doesn’t: the cost, reliability, and operational math that sits above the serving stack rather than inside it.

Cost modeling: the formula that turns architecture decisions into a dollar figure#

$$\text{cost per 1M tokens} = \frac{\text{GPU \$/hour}}{\text{tokens/sec} \times 3600} \times 10^6$$

Worked example. An H100 costs roughly $2–3/hour on-demand from a cloud provider (a real, if provider-dependent, current figure); a 70B-parameter model served with GQA, FP8 KV cache, and continuous batching (companion document, ch. 2–3) achieving, say, 2,000 output tokens/sec aggregate throughput per GPU under realistic concurrent load: \frac{2.50}{2000\times3600}\times10^6 \approx \0.347$ per million output tokens. The same model without those optimizations, achieving perhaps 400 tokens/sec (a realistic naive-serving number given the companion document’s fragmentation and batching-efficiency figures), costs five times more per token served -- this single formula is the direct dollar translation of every technique in that document’s chapters 2–3, which is exactly why those techniques get engineering priority in any team actually paying a cloud bill.

def cost_per_million_tokens(gpu_dollars_per_hour, tokens_per_sec):
    return (gpu_dollars_per_hour / (tokens_per_sec * 3600)) * 1_000_000

optimized = cost_per_million_tokens(2.50, 2000)     # ~$0.35 / 1M tokens
naive     = cost_per_million_tokens(2.50, 400)      # ~$1.74 / 1M tokens -- 5x more expensive for identical model

SLOs and error budgets: turning “reliable” into a number you can page on#

Service Level Objectives state a target as a measurable threshold (e.g. “p99 time-to-first-token under 500ms,” “99.9% of requests succeed”) rather than a vague aspiration. The error budget is what that target implies is allowed to fail:

$$\text{error budget} = (1 - \text{SLO}) \times \text{total requests in period}$$

Worked example. A 99.9% availability SLO over a 30-day month with 10M requests allows $(1-0.999)\times10{,}000{,}000 = 10{,}000$ failed requests before the SLO is breached. If an incident burns 3,000 of that budget in a single afternoon, that’s a 30% burn rate in hours instead of weeks -- the standard “burn rate alert” pattern (page if burning the budget more than, say, 10x faster than the sustainable linear rate) is exactly this ratio, computed continuously, and is what lets an on-call engineer distinguish “a blip that’s still within budget” from “we need to act right now.”

In plain terms: an error budget works like a household’s monthly grocery budget. Overspending a little on one bad week is fine -- you have a whole month to average it out. Blowing through the entire month’s budget in a single afternoon is a completely different situation, even though “we spent money” is technically true in both cases. Burn rate is just: how fast are we spending the budget right now, compared to the pace that would let it last the whole month.

def burn_rate(errors_in_window, window_hours, total_budget, budget_period_hours):
    actual_rate = errors_in_window / window_hours
    sustainable_rate = total_budget / budget_period_hours
    return actual_rate / sustainable_rate    # >1 means burning budget faster than sustainable; page if too high

Autoscaling: reacting to queue depth, not just raw CPU/GPU utilization#

Why naive utilization-based autoscaling fails for LLM serving. GPU utilization can look high (near 100%) even while requests queue and latency degrades, because a saturated batch is “fully utilized” in the compute sense while still accumulating a backlog -- the more directly useful signal is queue depth and time-in-queue, not raw utilization.

$$\text{target replicas} = \left\lceil \frac{\text{current arrival rate (req/sec)} \times \text{target p99 latency}}{\text{max concurrent requests per replica (companion doc, ch. 3)}} \right\rceil$$

This is Little’s Law applied directly to inference serving: the number of requests that must be “in flight” simultaneously to hit a latency target is arrival rate times target latency, and that number divided by how many requests one replica can hold in its continuous batch (companion document, ch. 3) gives the replica count needed -- real autoscalers (Kubernetes HPA with a custom metric, or a purpose-built controller) drive off exactly this queue-depth-derived signal rather than CPU/GPU percent alone.

In plain terms: a restaurant deciding how many tables it needs shouldn’t just look at whether its current tables are “busy” -- a fully-occupied restaurant can still be turning customers away at the door. What actually matters is how fast customers are arriving and how long they stay: more arrivals, or longer stays, both mean more tables are needed, regardless of how “utilized” the existing tables already look. Serving infrastructure has the identical blind spot if it only watches GPU utilization instead of the queue.

Fault tolerance and chaos engineering for serving#

Pre-training’s fault tolerance (checkpointing, elastic restart, ch. 1) has a serving-side analog: a request in flight when a GPU dies should be retried on a healthy replica, not silently dropped, and a serving fleet should be regularly tested against deliberately injected failures (killing a replica mid-request, simulating a slow network partition) rather than only discovering failure-handling gaps during a real incident -- the same “test the failure path, not just the happy path” discipline chaos engineering brought to general distributed systems, applied specifically to inference fleets.

GPU profiling, in practice#

# torch.profiler -- the standard first tool to reach for when MFU (companion doc) is lower than expected
import torch
from torch.profiler import profile, ProfilerActivity

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True) as prof:
    model(input_batch)

print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
# the top rows answer the question the companion document's roofline model (ch. 6) poses in the abstract:
# is time actually going to compute-bound matmuls, or to memory-bound ops / kernel-launch overhead?

NVIDIA Nsight Systems gives the same answer at a lower level (actual kernel-by-kernel GPU timeline, including gaps where the GPU sits idle waiting on data or communication) -- the standard escalation path is torch.profiler first to find which op is slow, Nsight Systems next to find out why, at the kernel and hardware-counter level, once the op-level view isn’t enough.

The rest of the original list, cross-referenced rather than duplicated#

task	where it’s covered in full	what’s genuinely new here
Throughput vs. latency	companion document, ch. 3 (prefill vs. decode, continuous batching)	the cost-per-token formula above turns that tradeoff into a dollar figure
KV cache management	companion document, ch. 2 (PagedAttention, RadixAttention, quantization)	--
Quantization	companion document, ch. 2, 8 (KIVI, AWQ, GPTQ, bitsandbytes, FP8)	--
Batching and scheduling	companion document, ch. 3 (continuous batching, disaggregated prefill/decode)	the autoscaling formula above is the layer above per-replica batching -- how many replicas, not how one replica schedules requests
Distributed inference / model parallelism	companion document, ch. 1 (Ring Attention), ch. 3 (Splitwise/DistServe) and ch. 1 of this document (FSDP/TP/PP, training-side)	--
Kernel-level optimization	companion document, ch. 6 (FlashAttention, kernel fusion, roofline model)	--
Fault tolerance	ch. 1 of this document (training-side: checkpointing, elastic restart)	this chapter’s serving-side analog: retry-on-failure and chaos testing, above

📚 Further reading

• The companion document: Long context, fast inference, smart decoding, and life beyond attention (chapters 2, 3, 6 especially)
• Beyer et al. (eds.), 2016 -- Site Reliability Engineering (Google) -- SLOs and error budgets
• Basiri et al., 2016 -- Chaos Engineering (Netflix)

06 · Sequencing the whole pipeline: what a frontier lab actually does, in order#

The five chapters above read linearly, but the real pipeline has feedback loops the linear reading hides. Laid out the way a frontier lab actually sequences and re-sequences it:

1. Pre-train (ch. 1), sized by a scaling-law budget decision made before the run starts, not adjusted mid-run -- a compute-optimal or deliberately-overtrained-for-serving-cost choice (ch. 1’s Llama 3 example) locked in up front, because changing model size mid-run means restarting.

2. Post-train (ch. 2) in stages, not one pass: SFT first (teaches format and basic instruction-following on curated or synthetic data), then preference optimization (DPO or RLHF) on top of the SFT checkpoint, often across multiple rounds -- generate outputs from the current model, collect fresh preference data specifically on this model’s current failure modes (not the previous round’s), retrain, repeat. This iterative-rounds detail is why “post-training” is really a loop, not a single stage, in every published frontier recipe (InstructGPT, Llama 2/3, Constitutional AI all describe multiple rounds explicitly).

3. Build the adaptation layer (ch. 3) largely in parallel with, not strictly after, post-training -- retrieval indices, tool schemas, and guardrails are typically developed against an early post-trained checkpoint so the eval suite (next step) can exercise the full system, not the bare model.

4. Gate every candidate -- base, post-trained, and adapted -- through the eval suite (ch. 4) before it touches real traffic: offline evals first (cheap, fast iteration), then canary release to a small traffic percentage (catches what offline evals didn’t cover), then full rollout, with automated eval gates (ch. 2’s production table) blocking promotion at each step rather than relying on a human’s judgment call under deadline pressure.

5. Serve it (ch. 5) behind SLOs, with the cost model from ch. 5 informing which serving-stack choices (quantization, speculative decoding, disaggregation -- companion document) are actually worth their engineering cost for this specific traffic pattern, not applied uniformly because a benchmark said so.

6. Feed production signal back to step 2, not just step 4. Real user feedback, logged failure cases (ch. 4’s error analysis), and drift-monitoring alerts (ch. 4) become the next round’s post-training preference data -- closing the loop from “detected a problem in production” back to “fixed in the next alignment pass,” which is the actual mechanism by which a deployed model improves over time, distinct from a full pre-training redo.

The note-taking template, extended#

The original outline’s closing instruction -- for each chapter, capture what the phase does, main algorithms, main failure modes, main production systems involved, metrics to track, common bugs, and real-world engineering tradeoffs -- is the right instinct and applies cleanly to everything above. Two additions worth making explicit before using it: track provenance across phases (which pre-training data mixture, which post-training round, which retrieval index version, which eval gate result produced this specific production behavior -- chapters 1–4’s versioning tables exist because this question is unanswerable without them), and track the cost figure alongside the quality figure for every phase (ch. 1’s compute cost, ch. 2’s training cost, ch. 5’s serving cost per token) -- a technique that improves quality at a cost the product can’t sustain isn’t a real option, and the note-taking template should make that tradeoff visible every time, not just at the end in a scale-engineering chapter.

07 · Real training logs: debugging GPT-3, Llama, Mistral, and DeepSeek’s own papers, number by number#

❓ Why this chapter. Every hyperparameter table in a textbook or tutorial repo (warmup=2000 steps, batch=256, cosine-to-10%) is a convention, copied forward from paper to paper because it’s a safe default -- not evidence of what actually happened inside a frontier run. The real papers report something more specific and more useful: the exact schedule, the exact point training got unstable, the exact moment a capability emerged mid-run. This chapter reads those numbers directly rather than the textbook version of them.

Pre-training: the batch-size and learning-rate schedules nobody’s tutorial matches#

GPT-3 (Brown et al., 2020), the primary-source numbers. Adam with $\beta_1=0.9$, $\beta_2=0.95$, $\epsilon=10^{-8}$; global gradient norm clipped at 1.0; cosine decay of the learning rate down to 10% of peak over 260 billion tokens (after which training continues at that floor, not zero); linear LR warmup over the first 375 million tokens (not “2000 steps” -- tied to tokens seen, which matters because it means the warmup duration in steps changes with batch size); and, the detail almost every reproduction skips: the batch size itself is not constant even at the start -- it ramps linearly from 32k tokens up to the model’s full batch size (3.2M tokens for the 175B model) over the first 4–12 billion tokens of training, with the exact ramp length depending on model size.

Llama 3 405B (Dubey et al., 2024), the primary-source numbers. AdamW, peak LR $8\times10^{-5}$, linear warmup over 8,000 steps, cosine decay down to $8\times10^{-7}$ over 1.2 million steps -- three orders of magnitude lower than GPT-3’s 10%-of-peak floor, reflecting how much further into the loss landscape a 15T-token run travels. The batch-size and sequence-length ramp, reported explicitly as a stability measure: 4M-token batches at 4,096 sequence length for the first 252M tokens, then doubled to 8M tokens at 8,192 sequence length through 2.87T tokens, then doubled again to 16M tokens for the remainder of the ~15T-token run. Context length itself was extended in six separate stages from 8K to 128K tokens, using roughly 800B of the total training tokens specifically for that extension -- long context is not a property baked in from step zero, it’s a late-training-phase intervention with its own token budget.

Why the ramp exists at all: the gradient noise scale. McCandlish et al. (2018) give the actual mechanism: a gradient estimated from a mini-batch is a noisy estimate of the true full-dataset gradient, and the gradient noise scale $B_{\text{noise}}$ is the batch size at which that noise and the true gradient signal are comparable in magnitude -- training below it wastes compute on noise-dominated steps, training far above it wastes compute on redundant samples that barely reduce noise further per additional token processed:

$$B_{\text{noise}} \approx \frac{\text{tr}(\Sigma)}{\|G\|^2}$$

where $G$ is the true gradient and $\Sigma$ its per-example covariance. Early in training, loss curvature is steep and the useful signal-to-noise ratio is achieved at a smaller batch size; as training progresses and the loss landscape flattens, $B_{\text{noise}}$ grows, and a fixed-batch-size run either wastes early compute (batch too large for the early noise scale) or later trains less stably than it could (batch too small once the noise floor has risen) -- the empirical ramp schedules above are exactly the largest labs’ practical answer to this, tuned by observation of loss-curve stability rather than computing $B_{\text{noise}}$ analytically at scale.

The same idea as a political poll, no calculus required. Imagine trying to estimate a country’s opinion on some issue by polling people -- a “batch” is just how many people you poll before computing your estimate. Poll only 10 people (a small batch) and your estimate bounces around wildly depending on who happened to answer -- mostly noise, barely any real signal about the true national opinion. Poll 10 million people (a huge batch) and your estimate barely moves whether you use 10 million or 11 million respondents -- you’re paying for a lot of extra phone calls to shave off an already-tiny amount of remaining noise. Somewhere in between is a “sweet spot” batch size -- big enough that noise has mostly washed out, but not so big that you’re wasting effort squeezing out the last drops of it. That sweet spot is exactly what the gradient noise scale formula is computing, except instead of polling people it’s averaging a training signal (the gradient) across training examples. And the sweet spot isn’t fixed throughout training: very early on, when the model’s understanding is still crude and every example teaches it something dramatically different, the “true opinion” it’s trying to sense is itself shifting fast, so a smaller poll (smaller batch) is already good enough to catch the signal cheaply. Later, once the model is fine-tuning small details, the useful signal is more subtle and buried in more noise, so a bigger poll (bigger batch) is worth its extra cost. That’s the entire justification, in plain terms, for why GPT-3 and Llama 3 both start training with a smaller batch size and grow it over time rather than picking one fixed number for the whole run.

# The GPT-3 batch-size ramp and cosine LR schedule, reconstructed from the paper's exact numbers
import math

def gpt3_batch_size(tokens_seen, min_bs=32_000, max_bs=3_200_000, ramp_end_tokens=8e9):
    if tokens_seen >= ramp_end_tokens:
        return max_bs
    frac = tokens_seen / ramp_end_tokens
    return int(min_bs + frac * (max_bs - min_bs))

def gpt3_lr(tokens_seen, peak_lr=6e-5, warmup_tokens=375e6, decay_tokens=260e9, min_frac=0.10):
    if tokens_seen < warmup_tokens:
        return peak_lr * (tokens_seen / warmup_tokens)                    # linear warmup, by tokens not steps
    if tokens_seen >= decay_tokens:
        return peak_lr * min_frac                                          # floor: 10% of peak, held indefinitely
    progress = (tokens_seen - warmup_tokens) / (decay_tokens - warmup_tokens)
    cosine_frac = min_frac + (1 - min_frac) * 0.5 * (1 + math.cos(math.pi * progress))
    return peak_lr * cosine_frac

# Llama 3 405B's own schedule, same shape, very different numbers
def llama3_405b_lr(step, peak_lr=8e-5, warmup_steps=8_000, decay_steps=1_200_000, floor_lr=8e-7):
    if step < warmup_steps:
        return peak_lr * (step / warmup_steps)
    if step >= decay_steps:
        return floor_lr
    progress = (step - warmup_steps) / (decay_steps - warmup_steps)
    return floor_lr + (peak_lr - floor_lr) * 0.5 * (1 + math.cos(math.pi * progress))

Pre-training backend: DualPipe and FP8, the parts of the DeepSeek-V3 paper that aren’t the architecture#

The cost number, and where it actually came from. DeepSeek-V3 (DeepSeek-AI, 2024) reports 2.788M H800 GPU-hours total -- split explicitly as 2.664M GPU-hours for the main pre-training on 14.8T tokens, plus 119K GPU-hours for context-length extension, plus 5K GPU-hours for post-training -- at an assumed $2/GPU-hour, a $5.576M total figure. The paper is explicit that this excludes prior research and ablation compute; it’s the cost of the one final training run, not the program that produced it.

DualPipe: why a custom pipeline scheduler was worth building. Standard pipeline parallelism’s bubble (ch. 1’s $\frac{p-1}{m}$ formula) is one source of idle time; a second, distinct source at DeepSeek-V3’s scale is cross-node expert-parallel communication for its MoE layers -- every token’s routed experts may live on a different node, and that all-to-all communication doesn’t automatically overlap with useful compute the way a well-scheduled pipeline’s forward/backward compute can. DualPipe’s specific contribution: split each pipeline chunk into forward and backward halves and schedule them so that, for any given micro-batch, one direction’s computation runs concurrently with another micro-batch’s communication -- the paper’s own reported result is close to full overlap between the MoE all-to-all communication and pipeline compute, which is the actual mechanism behind training a 671B-parameter (37B-active) MoE model without communication becoming the bottleneck it would otherwise be at this scale.

FP8 training: why it’s not just “cast everything to 8 bits.” The transformer notes and companion document cover FP8/INT8 inference quantization; DeepSeek-V3’s contribution is doing this during training itself, where the numerical stability requirements are stricter (training accumulates quantization error over millions of steps, not just one forward pass). The specific recipe reported: fine-grained, block-wise quantization rather than one scale factor per tensor -- activations quantized in $1\times128$ blocks and weights in $128\times128$ blocks, each block getting its own scale factor, precisely the same “don’t use one global scale on an outlier-heavy distribution” lesson the companion document’s KV-cache-quantization chapter describes for inference, now applied to the training forward and backward pass itself. The paper’s own framing: this is, to their knowledge, the first validated demonstration that FP8 training is feasible and effective at this parameter scale, not merely a proposal.

Post-training: Llama 2’s real RLHF progression, version by version#

The detail almost every summary skips: PPO was not used until the fifth and final round. Llama 2-Chat (Touvron et al., 2023) went through five successive RLHF versions, RLHF-V1 through RLHF-V5 -- and PPO, the “actual RL” step, was applied only in RLHF-V5. RLHF-V1 through V4 used rejection sampling fine-tuning only: sample $K$ candidate responses per prompt from the current 70B model, score all $K$ with the (separately trained, iteratively improved) reward model, keep only the highest-scoring response per prompt, and fine-tune on that curated set -- ordinary supervised learning on self-selected best-of-$K$ data, not policy-gradient RL at all. The paper’s own stated reason: the reward model’s own quality was accumulating gradually as more human preference data arrived, and running full PPO against an unreliable reward model risks the reward-hacking failure mode chapter 2 of this document already covers -- so the team deliberately delayed PPO until the reward model was judged reliable enough to optimize against directly. Rejection sampling was performed only on the largest (70B) model, with all smaller Llama 2-Chat variants fine-tuned on data sampled from the 70B model -- a real, explicit case of the distillation pattern (ch. 2) applied inside the alignment pipeline itself, not just at the pretraining-to-small-model stage. Two separate reward models were trained throughout -- one for helpfulness, one for safety -- specifically because the paper found a single blended reward model let helpfulness and safety trade off against each other in ways that were hard to control; scoring them independently and combining at the PPO objective level kept that tension visible and tunable rather than silently resolved inside one scalar.

Post-training: GRPO, the algorithm and DeepSeek-R1-Zero’s actual numbers#

Why GRPO exists. Chapter 2’s PPO objective needs a learned value model to compute the advantage estimate $\hat{A}_t$ -- a fourth model (alongside policy, reference, and reward) that itself needs training and holds real GPU memory. GRPO (Shao et al., 2024, introduced for DeepSeekMath, used again for R1) removes it: sample a group of $G$ outputs for the same prompt, and use the group’s own reward statistics as the baseline instead of a learned value function.

$$\hat{A}_i = \frac{r_i - \text{mean}(r_1, \ldots, r_G)}{\text{std}(r_1,\ldots,r_G)}, \qquad \mathcal{L}_{\text{GRPO}} = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G} \min\!\big(\rho_i \hat{A}_i,\ \text{clip}(\rho_i, 1{-}\epsilon, 1{+}\epsilon)\hat{A}_i\big)\right] - \beta\, \text{KL}(\pi_\theta \| \pi_{\text{ref}})$$

same clipped-surrogate shape as PPO, but $\hat{A}_i$ now comes from comparing outputs to each other within the group, not from a separately trained critic.

DeepSeek-R1-Zero’s actual reported hyperparameters (DeepSeek-AI, 2025): learning rate $3\times10^{-6}$, KL coefficient $\beta = 0.001$, GRPO clip ratio $\epsilon$ reported at 10 (an unusually wide clip range relative to PPO’s typical 0.1–0.2, consistent with GRPO’s group-normalized advantage already being on a different, self-normalized scale), sampling temperature 1.0 for rollouts, $G=16$ sampled outputs per question, 32 unique questions per training step giving a batch size of 512 rollouts per step, and a maximum generation length of 32,768 tokens -- and, distinctively, no SFT stage before RL at all: R1-Zero trains directly from the DeepSeek-V3-Base checkpoint with pure RL, specifically to test whether reasoning capability could emerge without any human-written reasoning demonstrations.

Worked numeric example, using the real reward-and-group shape above. Suppose for one math question, $G=16$ sampled responses get binary correctness rewards (rule-based: 1 if the final boxed answer matches, else 0) with 5 correct and 11 incorrect: mean $= 5/16 = 0.3125$, and treating rewards as Bernoulli, std $\approx \sqrt{0.3125 \times 0.6875} \approx 0.463$. A correct response ($r_i=1$) gets advantage $\hat{A}_i = (1-0.3125)/0.463 \approx 1.485$ -- a strong positive push. An incorrect response ($r_i=0$) gets $\hat{A}_i = (0-0.3125)/0.463 \approx -0.675$ -- a real but proportionally smaller negative push, because with only 5/16 correct, being wrong is the majority outcome and carries less surprising-failure signal than being one of the rare correct responses carries reward. This asymmetry -- correct-and-rare responses getting pushed harder than incorrect-and-common ones -- is exactly the mechanism that, aggregated over the paper’s reported thousands of RL steps, drove AIME 2024 pass@1 from 15.6% to 71.0% (86.7% with majority-vote/cons@16), with no supervised reasoning data involved at any point.

Unpacking that without the statistics vocabulary. The model attempts the same math problem 16 separate times (that’s the “group”). Each attempt gets graded pass/fail -- 1 point for a correct final answer, 0 for wrong -- no partial credit, no human judgment, just “did the number match.” In this example, 5 of the 16 attempts were correct and 11 were wrong, so the average score across the group is $5/16 \approx 0.31$ -- think of this as “31% of attempts succeeded, so 0.31 is the going rate for this problem right now.” The “std” (standard deviation) number, 0.463, is just a measure of how spread-out the results were -- with a roughly-70/30 pass/fail split, individual results swing noticeably above and below that 0.31 average, so this number ends up moderately large. Now the key step: instead of just saying “correct = good, incorrect = bad,” GRPO asks how far above or below the group’s own average each result landed, then divides by that spread number to keep the scale sensible. A correct attempt (score 1) sits $1 - 0.31 = 0.69$ points above the group average; dividing by the spread (0.463) stretches that into roughly +1.49, a strong “do more of this” signal. An incorrect attempt (score 0) sits $0 - 0.31 = -0.31$ points below average, which divided the same way becomes roughly -0.68 -- a real but smaller “do less of this” signal. In plain terms: because most attempts on this problem failed, being one of the rare successes is treated as unusually good news and rewarded hard, while being one of the common failures is treated as unremarkable and penalized more gently -- the model learns fastest from its rare wins, not by being hammered for its common losses.

In plain terms: GRPO grades on a curve, literally. Instead of hiring and training a separate professional critic to judge every essay against some fixed, absolute standard (the value model PPO needs), it just has the whole class write essays on the same question and rewards each student for beating their own classmates’ average that round. Being one of the rare students who gets it right when most of the class doesn’t earns a bigger reward than being wrong along with the majority -- no external critic ever needs to be hired or trained.

The “aha moment,” and the honest caveat about it. The R1 paper reports a qualitative shift partway through training where the model begins visibly re-evaluating its own initial approach mid-response (“Wait, let me reconsider…”) before continuing -- accompanied by a measured, sustained increase in average response length over the course of RL training, interpreted as the model learning to allocate more “thinking” tokens to harder problems because doing so is what the reward signal rewards. Independent replication studies (Sea AI Lab, 2025; Liu et al., 2025, “Understanding R1-Zero-Like Training”) complicate the framing somewhat: they report that self-reflection-like text patterns can already appear in some base models before any RL at all, and that the response-length increase is a real, robust, reproducible RL effect while the specifically qualitative “sudden emergence” framing is less clearly a discrete moment than the original paper’s narrative implies -- worth knowing both because it’s the honest state of replication evidence and because it’s a useful caution against over-narrativizing a training curve.

# GRPO's group-relative advantage and clipped objective, with the exact R1-Zero hyperparameters
import torch

def grpo_loss(logprobs_new, logprobs_old, logprobs_ref, rewards, clip_eps=10.0, kl_coef=0.001):
    # rewards, logprobs_*: shape (G,) -- one group of G=16 sampled outputs for a single prompt
    advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-8)     # group-relative, no value model needed
    ratio = torch.exp(logprobs_new - logprobs_old)
    clipped = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps)
    policy_loss = -torch.min(ratio * advantages, clipped * advantages).mean()
    kl = (torch.exp(logprobs_ref - logprobs_new) - (logprobs_ref - logprobs_new) - 1).mean()  # KL estimator, k3 form
    return policy_loss + kl_coef * kl

# real production path: TRL's GRPOTrainer implements exactly this, with a rule-based reward function
from trl import GRPOTrainer, GRPOConfig
from datasets import load_dataset

def math_correctness_reward(completions, ground_truth, **kwargs):
    return [1.0 if extract_boxed_answer(c) == gt else 0.0 for c, gt in zip(completions, ground_truth)]

trainer = GRPOTrainer(
    model="deepseek-ai/deepseek-math-7b-base",
    reward_funcs=math_correctness_reward,
    args=GRPOConfig(learning_rate=3e-6, beta=0.001, num_generations=16, max_completion_length=32_768),
    train_dataset=load_dataset("your-math-dataset", split="train"),
)
trainer.train()

Adaptation: the honest state of public, paper-level numbers#

Unlike pre-training and post-training, frontier labs publish comparatively little granular, paper-level detail about their production RAG/tool-use adaptation layers -- these are treated as product infrastructure, iterated on continuously, and rarely frozen into a citable configuration the way a training run’s hyperparameters are. What is publicly measurable comes from open benchmarks rather than frontier labs’ own internals: embedding model quality is tracked on MTEB (the Massive Text Embedding Benchmark) rather than any single lab’s paper, and real HNSW-index tradeoff numbers (the approximate-nearest-neighbor structure underlying most production vector search) show recall/latency curves that shift substantially with the M (graph connectivity) and ef_construction/ef_search (search-width) parameters -- real benchmarks commonly show recall in the high-90s% achievable within single-digit-millisecond query latency at these tuned settings on million-scale vector collections, with the tradeoff curve, not a single fixed setting, being the actual production decision every deployment has to make for its own latency budget and corpus size. This is worth stating plainly rather than inventing a false sense of paper-level precision: adaptation is the phase where “read the production benchmark and tune it yourself” replaces “read the paper’s reported hyperparameter.”

📚 Further reading

• Brown et al., 2020 -- Language Models are Few-Shot Learners (GPT-3), Appendix B
• McCandlish et al., 2018 -- An Empirical Model of Large-Batch Training
• Dubey et al., 2024 -- The Llama 3 Herd of Models
• Touvron et al., 2023 -- Llama 2: Open Foundation and Fine-Tuned Chat Models
• DeepSeek-AI, 2024 -- DeepSeek-V3 Technical Report
• Shao et al., 2024 -- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO)
• DeepSeek-AI, 2025 -- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Liu et al., 2025 -- Understanding R1-Zero-Like Training: A Critical Perspective

08 · Backend internals frontier engineers rarely see#

❓ Why this chapter. Everything above this point is “what technique, and roughly why.” This chapter is the layer most tutorials skip entirely: the actual memory arithmetic, network bandwidth numbers, and data-pipeline mechanics that decide whether a training run physically fits on the hardware at all, before any question of quality comes up.

ZeRO’s three stages, the real per-GPU memory numbers#

Chapter 1’s $16N$-byte-per-parameter formula (bf16 weights + bf16 grads + fp32 Adam $m,v$ + fp32 master weights) is what FSDP (ZeRO stage 3) shards completely. The intermediate stages shard less, and knowing exactly what each one saves is what decides which stage a real team actually reaches for:

stage	shards	per-GPU memory (70B model, $P$-way shard, before activations)
ZeRO-1	optimizer states only ($m, v$, fp32 master weights $= 12N$ bytes)	$\dfrac{12N}{P} + 4N$ (weights + grads stay replicated in full on every GPU)
ZeRO-2	+ gradients (adds the $4N$ bf16-grad term to the shard)	$\dfrac{16N}{P}$… but weights are still replicated in full: $\dfrac{12N}{P} + 2N$
ZeRO-3 (= FSDP)	+ parameters themselves	$\dfrac{16N}{P}$ -- nothing left replicated at rest

Worked example, 70B params, $P=8$ (one node of 8 GPUs). ZeRO-1: $\frac{12\times70\times10^9}{8} + 4\times70\times10^9 \approx 105\text{GB} + 280\text{GB} = 385\text{GB}$ per GPU before any activation memory -- already far past a single H100’s 80GB, meaning ZeRO-1 alone cannot fit a 70B model on 8 GPUs no matter how activations are handled. ZeRO-3/FSDP: $\frac{16\times70\times10^9}{8} = 140\text{GB}$ per GPU -- still over budget on 8 GPUs at this size, which is exactly why real 70B-class training runs combine FSDP with tensor and/or pipeline parallelism (ch. 1) rather than relying on data-parallel sharding alone; the memory arithmetic, not a preference, forces the combination.

What’s actually happening here, without tracking the fractions. Picture the model’s full “state” (weights, gradients, and the optimizer’s own bookkeeping numbers) as a pile of boxes that has to live somewhere in GPU memory, and picture 8 GPUs as 8 shelves that can share the load. ZeRO-1 only lets the optimizer’s boxes be split across the 8 shelves -- the weights and gradients still get a full, undivided copy sitting on every single shelf, unshared. That’s why even after dividing, each shelf is still stuck holding two entire, un-split copies of the “weights + gradients” pile on top of its one-eighth share of the optimizer pile -- and the two full copies alone (280GB) already blow past what an H100 shelf can hold (80GB), regardless of how well the smaller, split pile is divided. ZeRO-3 (FSDP) is the version where nothing gets a free, undivided copy -- every single box, weights included, is split eight ways. That drops the total from 385GB down to 140GB per shelf -- real progress -- but 140GB is still nearly double what one H100 actually has. The conclusion isn’t “use a smarter formula,” it’s “you need more shelves, arranged more cleverly” -- which is exactly why real training runs stack FSDP together with the tensor- and pipeline-parallelism tricks from chapter 1, rather than expecting one sharding trick alone to solve a problem this size.

In plain terms: ZeRO’s three stages are just three different answers to “how much of the shared bookshelf does each roommate carry around personally, versus fetch from a coworker only when needed.” Stage 1 only shares the optimizer’s private notes; stage 2 also shares the day’s to-do list (gradients); stage 3 shares everything, including the books themselves (the weights). The more you share, the less any one roommate has to personally carry -- but even sharing everything, a big enough model still needs more roommates (more parallelism strategies) than one small apartment (one node) can hold.

def zero_stage_memory_per_gpu(N, P, stage):
    bytes_per_param = {"weights_bf16": 2, "grads_bf16": 2, "adam_m_fp32": 4, "adam_v_fp32": 4, "master_fp32": 4}
    optimizer_state = bytes_per_param["adam_m_fp32"] + bytes_per_param["adam_v_fp32"] + bytes_per_param["master_fp32"]
    if stage == 1:
        return (optimizer_state * N) / P + (bytes_per_param["weights_bf16"] + bytes_per_param["grads_bf16"]) * N
    if stage == 2:
        return (optimizer_state * N) / P + bytes_per_param["weights_bf16"] * N
    if stage == 3:
        return ((optimizer_state + bytes_per_param["weights_bf16"] + bytes_per_param["grads_bf16"]) * N) / P

N = 70e9
for stage in (1, 2, 3):
    print(f"ZeRO-{stage}, P=8: {zero_stage_memory_per_gpu(N, 8, stage)/1e9:.1f} GB/GPU")

Network topology: why tensor parallelism cannot cross a node boundary#

The bandwidth gap that decides this, in real numbers. Intra-node GPU-to-GPU links (NVLink, on an H100 node) deliver on the order of 900 GB/s bidirectional per GPU. Inter-node links (InfiniBand, at a common frontier-cluster configuration of multiple NDR 400Gb/s NICs per node) deliver on the order of tens of GB/s effective per GPU once accounting for how many GPUs share each node’s NICs -- often more than an order of magnitude slower than NVLink. Tensor parallelism (ch. 1) requires an all-reduce every layer, for every forward and backward pass -- at that communication frequency, running it over InfiniBand instead of NVLink doesn’t just slow training, it can make communication the dominant cost entirely, which is why every major published training recipe (Megatron-LM’s own guidance included) confines tensor-parallel groups to fit within a single node’s NVLink domain, and uses the slower inter-node links only for the much less frequent communication patterns -- FSDP’s per-layer all-gather/reduce-scatter, and pipeline parallelism’s boundary activation hand-off -- that can tolerate the extra latency.

In plain terms: this is the difference between chatting with the person sitting right next to you versus shouting a message down a long hallway into another room. A rapid back-and-forth conversation works fine with your neighbor; try to have that same rapid back-and-forth down the hallway and it grinds to a halt. So you keep the chatty, constant conversation (tensor parallelism) confined to people sitting right next to each other, and only send the occasional, less frequent message (FSDP’s sharding, pipeline hand-offs) down the slower hallway.

$$\text{time lost to TP communication} \approx \frac{2 \times (\text{tensor size})}{\text{bandwidth}} \times (\text{layers}) \times (\text{forward+backward passes})$$

is why this specific placement rule -- TP within a node, everything else across nodes -- appears identically across essentially every frontier training report rather than being independently rediscovered each time: the bandwidth ratio makes it close to the only viable choice, not one option among several.

Activation checkpointing: the memory/compute trade, with real numbers#

The problem it solves. Storing every layer’s activations for the backward pass costs memory linear in depth ($O(L)$). At 405B-parameter, 126-layer scale, this can exceed available memory even after FSDP shards the weights/optimizer/gradients -- activations are a separate memory pool FSDP doesn’t touch.

The trade. Activation checkpointing (Chen et al., 2016) stores only a subset of layers’ activations (commonly every $\sqrt{L}$-th layer) and recomputes the rest during the backward pass by re-running the forward pass for just that segment -- trading memory for extra compute:

$$\text{memory} \propto O(\sqrt{L}) \quad \text{instead of} \quad O(L), \qquad \text{extra compute} \approx +33\%\text{ (one extra forward pass per checkpointed segment)}$$

Worked example. A 126-layer model with full activation storage might need, say, 40GB/GPU just for activations at a given batch size -- with $\sqrt{126}\approx11$ checkpoint segments instead, activation memory drops to roughly $40/\sqrt{126}\times\sqrt{126}$… more concretely: only 11 segments’ activations are kept resident instead of 126 layers’ worth, a roughly 11x reduction in activation memory, at the cost of re-running forward compute for the other ~115 layers during backward -- a real, deliberate ~33% increase in total FLOPs for the run, accepted specifically because the alternative (not fitting in memory at all) isn’t a training run, it’s an OOM crash.

In plain terms: this is like taking detailed notes only every ten pages of a lecture instead of every single sentence, and reconstructing the missing detail later by quickly re-reading the last ten pages when you actually need it. You carry far less paper around (memory) at the cost of some extra re-reading later (compute) -- worth it when carrying the full, complete notes the whole time simply wouldn’t fit in your bag at all.

# torch's real activation-checkpointing API -- trades this exact tradeoff, per-layer, with one wrapper
from torch.utils.checkpoint import checkpoint

class TransformerBlockWithCheckpointing(torch.nn.Module):
    def forward(self, x):
        return checkpoint(self._forward_impl, x, use_reentrant=False)   # recomputed on backward, not stored
    def _forward_impl(self, x):
        return self.mlp(self.attn(self.norm1(x)) + x)

PPO’s real memory bill at 70B scale, and what GRPO actually removes#

The four-model problem, in GB, not the abstract. Running PPO-based RLHF at 70B scale needs, resident simultaneously: the policy model (trainable, needs the full $16N$-byte-per-parameter cost from ch. 1’s formula -- roughly 1.12TB at 70B), the reference model (frozen, inference-only, needs just bf16 weights $= 2N \approx 140\text{GB}$), the reward model (frozen during the PPO step, similarly $\approx 140\text{GB}$ if a similarly-sized architecture), and the value model (trainable -- if it’s a similarly-sized network, another full $16N \approx 1.12\text{TB}$ optimizer-inclusive footprint, though in practice value models are often smaller than the policy). Even in the more favorable case (a much smaller value head), this is on the order of 1.4TB+ of resident model state before a single activation or KV cache (for the generation/rollout step, which needs its own inference-time memory, companion document ch. 2) is counted -- the direct, concrete reason PPO-based RLHF is treated as an infrastructure-heavy undertaking at frontier scale, not merely an algorithmically-more-complex one.

What GRPO removes, in the same units. Dropping the learned value model removes that second trainable-model memory bill entirely -- at the illustrative 70B scale above, roughly 1.12TB of GPU memory that no longer needs to exist, alongside removing an entire model’s forward/backward compute from every training step. This is the concrete, GB-denominated version of “GRPO simplifies RLHF” -- not a vaguer claim about algorithmic elegance, but a specific, large resident-memory footprint that a team no longer has to provision hardware for.

In plain terms: removing the value model from PPO is like realizing you don’t need to keep a second, fully-staffed department on payroll (predicting how good an answer will turn out to be, before it’s even finished) when you can just compare finished answers to each other directly. GRPO doesn’t make that department cheaper to run -- it lays the whole department off, and the resident memory that used to be reserved for it is memory that simply doesn’t need to exist anymore.

Data mixture: the fractions that actually went into a frontier model#

Llama 3’s own report describes tuning its final pretraining data mixture via smaller-scale ablation runs before committing the full-scale budget to one mixture, converging on a blend the paper describes as roughly half general-knowledge web text, about a quarter mathematical/reasoning content, a substantial code fraction, and a smaller multilingual fraction -- the exact point being that these fractions were measured and tuned, not assumed: the team ran scaled-down training experiments varying the mixture and evaluated downstream benchmark impact before locking in a ratio for the full, unrepeatable 15T-token run, applying the same “don’t guess, measure at small scale first” discipline chapter 1’s scaling-law section applies to model/data size, now applied to data composition instead of data quantity.

📚 Further reading

• Rajbhandari et al., 2020 -- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Chen et al., 2016 -- Training Deep Nets with Sublinear Memory Cost (activation checkpointing)
• Shoeybi et al., 2019 -- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Dubey et al., 2024 -- The Llama 3 Herd of Models (data-mix ablation methodology, section 3)

---

Zero → Frontier Engineering, log 03. A practical reference, not a tutorial: assumes logs 01–02 and gets denser from there. Named production incidents, papers, and numbers throughout are cited inline per chapter under “further reading.”