Clusters & Reliability
Schedulers, collectives, checkpoints, staying up at scale.
13 logs in this box-
OPT-175B's real training logbook has 40+ restarts and a cloud provider that deleted the whole cluster, and it's where half this series' practices actually came from
Meta's actual, publicly released metaseq chronicles, TP=8 and FSDP=124 with no PP, CP, or EP because 2021 didn't need them yet, a documented numerical-instability lineage that's the real source of the loss-spike-recovery pattern already cited in this series, and a silent one-line dependency bump that changed training numerics without anyone understanding why for weeks.
-
Mixtral 8x22B used tensor parallelism where DeepSeek-V3 didn't, and a 900-versus-400 GB/s NVLink gap is the entire reason why
NVIDIA's real MoE Parallel Folding paper and its actual Table 3/4/5 numbers, the exact Megatron-Core CLI flags that let attention layers and MoE layers use two different parallelism meshes on the same GPUs, a measured 46.3-to-49.3% MFU improvement from one specific architectural change, and the precise hardware fact, H100's 900 GB/s NVLink against H800's 400 GB/s, that explains why the same TP reasoning gave two frontier labs opposite answers.
-
DeepSeek-V3 trained 671 billion parameters without tensor parallelism at all, and that omission is the real production lesson
TorchTitan's actual, currently-checked-in DeepSeek-V3 671B config, DeepSeek's own disclosed 2048-GPU mesh (PP=16, EP=64, DP=2, TP=1 by deliberate choice), the real fake_backend and local_tensor debug modes engineers use before ever touching a cluster, a full incident-driven iteration loop, and the production-readiness gate applied to real, disclosed training numbers instead of an invented scenario.
-
Production-ready is a pass/fail test, not a feeling, and every answer in this series is the evidence
Six questions worth being able to answer cold before scaling up, why 'add more GPUs' is almost never the correct response to a stalled job, a per-rank phase-timing breakdown that turns observability from a feeling into a table, and why every incident in this series was really the same six-question test in disguise.
-
Don't read these codebases linearly, trace one training step through all of them instead
Why launch, process groups, model partitioning, backward, checkpointing, and metrics is the one path worth tracing through TorchTitan, Megatron-LM, and DeepSpeed instead of reading top to bottom, the specific files that map onto every mechanism derived earlier in this series, and the order that gets each codebase's own design philosophy instead of a random walk through source.
-
Ring Attention and Ulysses shard a sequence two different ways, and MoE routing turns a router's output into a networking problem
Ring Attention's point-to-point KV rotation against Ulysses' all-to-all layout swap, why context parallelism and sequence parallelism solve different problems despite the shared name, the exact all-to-all dispatch/combine cycle expert parallelism runs, and DeepSeek-V3's node-limited routing and bias-based load balancing as the real production answer to a 671B-parameter router.
-
Tensor parallelism splits a matmul, pipeline parallelism splits a network, and only one of them scales past a single node
The column-then-row-parallel MLP that needs exactly one AllReduce instead of one per matmul, why TP=8 shows up in Llama 3's real config because that's the NVLink domain and not a round number, the GPipe bubble formula worked at two microbatch counts, and Table 4's actual numbers with a citation worth double-checking.
-
FSDP2's parameter lifecycle, and the wrap-policy bug that turns one AllGather into ten
Why 16 bytes per parameter is a planning heuristic and not a memory ledger, the exact AllGather-materialize-reshard-ReduceScatter lifecycle FSDP2 runs per module, the wrap-granularity bug already flagged as an incident earlier in this series worked properly this time, and when FSDP loses to plain DDP.
-
Deterministic resume, and the checkpoint interval formula that reproduces Llama 3's actual number
Why async checkpointing alone doesn't eliminate the stall, the physical reason NFS can't do fast sharded writes, the exact RNG state list a deterministic resume needs, and a derived formula that lands on the same 20-30 minute interval real frontier labs actually use.
-
Goodput, DCGM, and the correlated failure your straggler detector can't see
Why a peer-relative straggler check is structurally blind to a datacenter-wide cooling problem, the exact auto-cordon sequence and why each step exists, and the goodput math behind Llama 3 hitting 90%+ utilization despite an interruption every three hours.
-
Four NCCL incidents, walked the way you'd actually work them
AllReduce, AllGather, ReduceScatter, and All-to-All as the four primitives every distributed technique reduces to, why NCCL has no default timeout, and four full diagnostic walkthroughs distinguishing a rendezvous failure from a dead rank from a network flow collision from a config bug that all look identical from a dashboard.
-
Containers are the process illusion, repackaged, and why Kubernetes doesn't gang-schedule by default
Namespaces and cgroups as os_linux's virtual-memory trick extended to the whole process, why a Kubernetes OOMKilled pod is the exact same OOM killer at a different boundary, and the gang-scheduling gap that silently deadlocks 512-GPU jobs on vanilla Kubernetes.
-
Partial failure is the default: stragglers, deadlocks, and coordinating GPUs that can't see each other
Why one 12%-slow GPU slows every GPU, the fail-stop vs fail-slow distinction that decides whether checkpoint-and-restart helps or hurts, the four ranked mitigation strategies, and the three-axis bar a frontier lab actually hires against.