Don't read these codebases linearly, trace one training step through all of them instead

Everything in this series so far, DDP’s replicated state, FSDP’s sharded state, tensor and pipeline parallelism’s split layers and depth, context and expert parallelism’s split sequences and routing, NCCL’s four primitives, and deterministic checkpoint recovery, was built as a derivation: a mental model, a formula, a worked example, checked against a small reference case. That’s necessary and it isn’t sufficient. The actual skill of opening a codebase that implements all of it at once, finding the fifteen lines that do the thing you just derived, and trusting that you’re reading it correctly, is a different, separate skill, and it has to be built deliberately rather than assumed to follow automatically from understanding the theory.

Read one execution path, not one file at a time#

The wrong instinct is starting at a repository’s root and reading files roughly in the order a file browser lists them. The useful instinct is picking a single training step and tracing it: launch, process group formation, model partitioning, forward, backward, gradient sync, checkpointing, metrics, in that order, following whichever files that specific path actually touches and ignoring everything it doesn’t. A codebase read this way teaches you its actual design decisions; the same codebase read top to bottom teaches you file names.

PyTorch’s own distributed examples: start here for the mechanics, not the scale#

Before anything more ambitious, the official PyTorch distributed examples repository and tutorials are worth reading first specifically because they track current PyTorch semantics directly, rather than a third-party snippet whose API usage quietly drifted out of date. init_process_group, the DDP wrapper, DistributedSampler, rank-local device placement, rank-aware checkpoint saving, all the mechanics built from scratch earlier in this series, show up here in their minimal, canonical form, uncomplicated by a large model’s actual architecture. Reading a production-scale codebase before this is reading the advanced version of a mechanism you haven’t seen the basic version of yet.

TorchTitan: the reference for how all of it gets composed into one job#

TorchTitan is the clearest current answer to “how does a real job actually combine FSDP2, tensor parallel, pipeline parallel, context parallel, Float8, and checkpointing into one coherent training script,” because composing them is the entire point of the project rather than a side effect. A useful reading order, each file corresponding directly to a mechanism already derived in this series:

File	What it is	Maps onto
train.py	The actual training loop, where every technique below gets wired together into one step	The full arc of this series, in executable form
models/llama3/parallelize_llama.py	Where ColwiseParallel/RowwiseParallel and fully_shard actually get applied to a real model’s real layers	Tensor and pipeline parallelism, FSDP2’s wrap granularity
models/llama3/infra/pipeline.py	Model splitting and schedule selection for pipeline parallelism	The SplitPoint/ScheduleGPipe mechanics from the same post
components/checkpoint.py	Sharded save/load, the async staging path	Deterministic resume, async sharded checkpointing
components/quantization/float8.py	FP8 scaling recipes applied during training	Loss scaling and FP8 training

Worth being honest about a real constraint here rather than presenting file paths as permanent: an actively developed repository reorganizes its own internal structure over time, models/llama3/parallelize.py and distributed/pipeline_parallel.py are both names this exact functionality has gone by at different points, so treat the table above as “what to search the repository for,” not a guarantee that a specific path is still current by the time you’re reading it. For every process group the code creates, draw its rank membership by hand before moving to the next file. A DeviceMesh with dimensions for data-parallel, tensor-parallel, and pipeline-parallel axes is easy to read past without noticing which physical GPUs actually end up in which group, and that mapping, not the API calls that construct it, is what a real topology decision or a real incident diagnosis actually depends on.

Megatron-LM and Megatron Core: where the architecture and the parallelism were co-designed#

The column-parallel-then-row-parallel MLP derived earlier, the placement decision that gets a transformer block down to one AllReduce instead of one per matmul, originates here, and it’s worth reading Megatron-LM specifically to see that the architecture and the parallelization strategy were designed together, not a generic model with parallelism bolted on afterward. Beyond the linear-layer split, this is also the reference for sequence parallelism as a memory optimization riding on the TP process group, pipeline schedule implementations, vocabulary-parallel loss computation, and, in Megatron Core specifically, MoE token dispatch and grouped GEMM, the actual expert-parallel machinery behind the toy router built in this series’ context-and-expert-parallelism post.

DeepSpeed: a different engineering philosophy solving the same memory problem#

FSDP and DeepSpeed’s ZeRO stages implement the same underlying sharding idea; reading DeepSpeed’s own engine is valuable specifically for what it does that FSDP doesn’t emphasize as heavily: CPU and NVMe offload as a first-class path, spilling optimizer state past GPU memory entirely rather than only sharding it across GPUs, and a more monolithic, integrated engine model as opposed to PyTorch-native composition through a shared DeviceMesh. The value of reading both isn’t picking a favorite, it’s being able to recognize which philosophy a given production codebase already committed to and reason inside that choice rather than trying to import the other one’s mental model wholesale.

NCCL-tests: the independent baseline that has to exist before any model-level number means anything#

Every collective-timing number this series has discussed, whether an AllReduce is bandwidth- or latency-bound, whether a job’s actual throughput matches what the hardware should deliver, is only interpretable against a baseline established independently of any training code. NVIDIA’s nccl-tests repository, all_reduce_perf, all_gather_perf, reduce_scatter_perf, alltoall_perf, swept across message sizes from kilobytes to multiple gigabytes, both intra-node and inter-node, is that baseline. Without it, “the AllReduce is slow” has no reference point to be slow relative to; a training job’s own collective timing, read in isolation, cannot distinguish a genuinely underperforming network from a network performing exactly as its physical topology allows.

Transformer Engine: the FP8 recipes underneath the numbers already covered#

The BF16-versus-FP8 tradeoffs and DeepSeek-V3’s FP8 training are worth grounding in NVIDIA’s Transformer Engine specifically because it’s the concrete, checkable implementation of the scaling recipes involved, delayed scaling, per-tensor and block scaling, which operations stay in BF16 versus which drop to FP8, rather than a description of FP8 training as a single, uniform switch to flip.

TorchFT: read this only after checkpoint-restart correctness is actually proven#

TorchFT explores fault-tolerant DDP and hybrid-sharded execution, health checks, quorum decisions, reinitializable communication groups, peer-based recovery that reduces how often a failure has to fall all the way back to a full checkpoint restore. It’s deliberately placed last in this list, not because it’s less important, but because everything it builds on top of assumes deterministic, validated checkpoint-restart already works correctly. Reading TorchFT before that assumption is actually verified in your own system is reading an advanced answer to a question your own infrastructure hasn’t earned the right to ask yet.

DCGM: the hardware-health layer underneath every diagnosis in this series#

Every incident walked through in this series, a straggler, a correlated thermal slowdown, a dead rank, ultimately gets correlated against real hardware telemetry, ECC error rates, GPU temperature, power draw, per-link NVLink bandwidth. NVIDIA DCGM and DCGM Exporter are the standard tooling this correlation actually runs against in production, and reading their metric surface directly is what turns “this rank is slow” into “this rank is slow because of X” rather than a guess.

The paper order, and reproducing one number from each rather than trusting the abstract#

Reading a paper without reproducing at least one of its concrete claims is exactly the failure mode this series has flagged repeatedly: a number gets quoted, then requoted, and the requoting is where the error creeps in. A working order, each entry paired with the one thing worth deriving or checking directly rather than accepting on faith:

1. PyTorch DDP (“Experiences on Accelerating Data Parallel Training”) — derive the bucketing and overlap mechanism built earlier in this series from the paper’s own description, not just the summary.
2. ZeRO — rederive the three-stage memory table already established in this series from the paper’s own accounting, and confirm it matches.
3. PyTorch FSDP — read for the production lessons on rate-limiting, wrapping, and parameter materialization behind the lifecycle diagrammed in this series.
4. Megatron-LM — derive the column/row-parallel transformer layer independently, then compare against the worked matrix example in this series.
5. Efficient Large-Scale LM Training on GPU Clusters Using Megatron-LM — the combined DP+TP+PP paper; check its reported configuration’s dimensions multiply out to its stated GPU count, the same arithmetic check this series ran against Llama 3’s Table 4.
6. Reducing Activation Recomputation in Large Transformer Models — sequence parallelism and selective recomputation, the technique distinguished from context parallelism in this series.
7. Ring Attention and DeepSpeed Ulysses — compare the two directly against the point-to-point-versus-all-to-all framing built in this series.
8. MegaScale — algorithm-system co-design and large-cluster stability at a scale this series has referenced but not derived from the primary source directly.
9. The Llama 3 Herd of Models — read the infrastructure and parallelism sections specifically, not the benchmark tables; Table 4 is the section this series drew directly from, and it rewards checking the row arithmetic yourself rather than trusting a paraphrase.
10. The OPT-175B logbook — restarts, loss-spike firefighting, and host replacement as they actually happened, day by day, the least sanitized account of what a long training run really looks like operationally.
11. DeepSeek-V3 — FP8, MoE expert parallelism, and the bias-based load balancing worked through as a case study in this series, read against the primary report rather than a secondary summary of it.
12. TorchTitan — the paper, read last, as the account of how a PyTorch-native team composed everything above into one system, now that every individual piece it composes has already been read on its own.

Common mistakes#

Reading a paper’s abstract and results section while skipping the method section that contains the actual mechanism: the number without the derivation behind it is exactly what gets miscited three sources later, the failure this series has caught more than once.

Treating a third-party reproduction or blog summary as equivalent to the official repository or the primary paper: API surfaces and file layouts drift, and a summary written against an older version silently inherits that drift without flagging it.

Reading TorchFT, or any fault-tolerance layer, before the deterministic-resume correctness it depends on has actually been verified in your own system: the more advanced mechanism inherits every bug in the foundation underneath it.

Skimming a large repository top to bottom instead of tracing one execution path through it: file-by-file reading teaches naming conventions; tracing launch through checkpointing teaches the actual design.

What this takes to be frontier-job-ready#

The technical axis is exactly what tracing one execution path through a real codebase demonstrates: the ability to find where a specific, previously-derived mechanism actually lives in someone else’s production code, not just recognize it in a diagram.

The operational axis is stated directly in real hiring language for this class of role: Mistral’s own Senior/Staff research-engineering requirements list, verbatim, having “contributed to a large codebase used by many” as a named, standalone requirement, not folded into a generic “strong engineering” line, alongside publications and research breadth. Reading a codebase closely enough to trace one execution path through it is the direct prerequisite to eventually contributing to one at that level, not a separate, lesser activity.

The autonomy axis: nobody hands you a curated file list for a production incident. The reading order in this post is a starting scaffold, not a permanent map, exactly because these repositories reorganize themselves over time, and the actual skill being built is picking the right file to open next based on what the previous one just told you, not memorizing a fixed path.

Try it yourself#

Beginner. In the official PyTorch DDP example, list every environment variable read between process launch and the first collective call, and note which ones torchrun supplies automatically versus which a hand-built Slurm launcher would have to compute itself.

Intermediate. Open TorchTitan’s Llama 3 parallelization file and map every parallelize_module call you find back to which post in this series first derived that specific technique, noting anywhere the production code diverges from the simplified version built here and why.

Advanced. Pick one paper from the list above whose headline number this series already cited, and reproduce that number, the ZeRO memory formula, the GPipe bubble fraction, the DDP overlap percentage, from the paper’s own method section rather than from this series’ restatement of it, and note anywhere your derivation disagrees.

---

The one-sentence version: read launch through checkpointing as one traced path rather than a pile of files, start with PyTorch’s own examples for mechanics before TorchTitan for composition, treat Megatron-LM and DeepSpeed as two different philosophies solving the same memory problem rather than competitors, save TorchFT for after checkpoint correctness is actually proven, and reproduce one number from each paper rather than trusting the citation chain that’s already let one error propagate in this series. Every mental model built across this entire arc, from DDP’s promise about the global batch to a 671B-parameter router’s all-to-all traffic, exists in one of these codebases as actual, runnable lines; the last step was always going to be reading them. Reading the code is not the same claim as being ready to run it in production, though, and there’s a specific, six-question test for that difference.