Production-ready is a pass/fail test, not a feeling, and every answer in this series is the evidence
Six questions worth being able to answer cold before scaling up, why 'add more GPUs' is almost never the correct response to a stalled job, a per-rank phase-timing breakdown that turns observability from a feeling into a table, and why every incident in this series was really the same six-question test in disguise.
A training job that completes one lucky run end to end has told you almost nothing. Meta’s Llama 3 405B run hit 419 unexpected interruptions over 54 days, roughly one every three hours, for nearly two months straight, and that’s the documented baseline at frontier scale, not an unlucky outlier. A system is production-ready when it survives that operating condition, not when it happens to finish once under ideal ones. Which means “is this ready” can’t be a feeling, or a demo that worked yesterday, it has to be a fixed set of questions you can answer cold, with evidence, before you scale up rather than after something breaks and you find out the hard way which questions you never actually had answers to.
The six questions, and where the evidence for each one already lives
1. Can you explain why this parallelism layout was chosen over at least one real alternative? Not “we used FSDP because the model is big,” the actual comparison. Llama 3’s own published configuration is the concrete version of this answer: TP fixed at specifically because that’s the NVLink domain, not a round number, and CP moved from to specifically to fund a K-token context, paid for by shrinking DP from to rather than by adding more GPUs, at a measured, accepted MFU cost from to . That’s what “why this layout, not another” looks like as a real, checkable answer instead of a justification invented after the fact.
2. Can you predict the bottleneck before running the job? The FSDP memory ledger, persistent state at bytes per parameter versus the transient row that actually sets peak memory, and the GPipe bubble formula, , both exist specifically so the answer to “will this OOM” or “how much idle time will this pipeline pay” is a number computed before the job runs, not a discovery made an hour into it. A team that can’t do this arithmetic on paper is going to spend real cluster time discovering it empirically instead, at a real dollar cost per hour of GPU time.
3. Can you recover from a worker loss, a slow rank, and a bad checkpoint, as three genuinely different problems? They are not the same failure wearing different symptoms, and conflating them is exactly the mistake the four-incident NCCL walkthrough was built to prevent: a rendezvous failure, a mid-run collective hang, a network flow collision, and a misconfigured wrap policy all present differently and get diagnosed differently, and reaching for the same tool, restarting the job, or checking the flight recorder regardless of symptom, is precisely the failure mode that costs hours instead of minutes. Auto-cordon’s drain, wait, checkpoint, resize, resume sequence is the actual, worked answer for the worker-loss case specifically, and it’s a sequence, not a single action, because skipping any one step trades a five-minute recovery for a worse failure than the one being recovered from.
4. Can you prove checkpoint validity and resume consistency, rather than assume them? The three-step validation protocol, verify every expected shard exists, verify byte counts to catch a truncated write, then attempt an actual test load to catch silent bit-flip corruption the first two checks miss, exists because a checkpoint that merely exists on disk is not the same claim as a checkpoint that’s valid, and the difference between those two claims is worth exactly the roughly \144{,}000$ an 8,000-GPU job loses falling back six hours if it turns out the answer to this question was wrong.
5. Can you tell whether a throughput win is real, or just noise? This is the specific trap the goodput post’s straggler blind spot was built around: a peer-relative check compares each rank against the current run’s own median, which is structurally blind to a correlated, fleet-wide degradation, a shared cooling problem throttling 200 GPUs together looks like “everything’s a little slow” rather than triggering any single alert. Telling a real change from noise needs an absolute baseline against healthy historical runs, not just a relative one against the current run’s own peers, and the roughly 30% MFU recoverable from properly overlapping communication with compute is exactly the size of win that’s easy to either miss entirely or falsely credit to the wrong change if you’re only checking one of the two.
6. Can you show observability for collectives, memory, topology, and storage, on demand, not after building it under incident pressure? This is the one question in this series that hasn’t had a single, concrete artifact built for it yet, so here’s one:
Per-step phase breakdown, per rank, not aggregated into one "step time" number:
data_wait -- time blocked on the DataLoader, not compute
host_to_device -- time moving a batch from CPU to GPU memory
forward -- forward pass compute
backward_compute -- backward pass compute
exposed_collective_time -- collective time NOT hidden behind compute (the number overlap is fighting to shrink)
optimizer -- optimizer step
checkpoint_staging -- device-to-host staging for async checkpointing, when active
checkpoint_write -- background write time, when it blocks the next save
total_step -- sum of the above, the only number a dashboard shows by defaultA dashboard that logs only total_step cannot distinguish a data-starved job from a communication-bound one from a job paying an unnecessary checkpoint stall, and that exact confusion, mistaking a healthy-looking GPU utilization graph for evidence of progress when a rank is actually blocked inside a collective, is precisely how Incident 2 in the NCCL post cost real diagnosis time before the phase decomposition made the actual stuck rank obvious. The question isn’t “do you have logs,” it’s “does your logging already distinguish these eight things, per rank, before an incident forces you to add the distinction under pressure.”
Why “add more GPUs” is almost never the right response
Every one of the six answers above is a mechanism for turning a vague symptom into a specific, falsifiable claim, this rank, this step, this collective, this checkpoint shard. Throwing more hardware at a system that can’t yet answer these questions just makes the blast radius of the next unexplained failure bigger, at a higher dollar cost per hour, while doing nothing to close the actual gap. The correct response to “the job is slow” or “the job hung” when one of the six questions doesn’t have a confident answer is almost always one of three things, in this order: simplify the decomposition (is the parallelism layout actually forced by the model, or assumed), improve the instrumentation (can the phase table above actually be produced right now, for this job, or does it need to be built first), or switch strategies entirely (is FSDP alone enough, or does the memory ledger actually justify TP; is EP earning its all-to-all cost, or was it adopted for the parameter-count headline). Scaling up is what you do once those three are already true, not a substitute for making them true.
The hiring language across this series was asking the same six questions the whole time
It’s worth noticing, now that all six are laid out together, that Mistral’s “you don’t panic when you see OOM errors or when NCCL feels like not wanting to talk” is question 3 stated as a temperament. Google DeepMind’s “experience training large-scale models on accelerators in a distributed environment” is questions 1 and 2 stated as a résumé line. Mistral’s “contributed to a large codebase used by many” is the prerequisite for ever getting to answer question 6 for a system you didn’t build alone. None of the quotes gathered across this series were really about a specific framework or a specific number, they were all pointing at the same underlying test: can this person tell you, with evidence, what’s actually happening, rather than what should be happening.
Common mistakes
Treating one successful run as proof of readiness: a job that finishes without incident under ideal conditions has been tested against exactly one operating point, and 419 interruptions over 54 days is the actual operating condition production systems are built for.
Reaching for “add more GPUs” as a default response to an unexplained slowdown: it increases blast radius and cost without closing the gap in any of the six questions, and the three correct responses, simplify, instrument, or switch strategy, are all cheaper and more likely to actually fix the problem.
Building observability under incident pressure instead of before: the per-rank phase table above is trivial to add before a job starts and genuinely difficult to reconstruct after the fact from an aggregate total_step number and a vague memory of what happened.
Answering these six questions once and considering them permanently answered: a topology change, a wrap-policy edit, a new parallelism dimension, or a scaled-up world size can invalidate any of the six independently, and each is worth re-checking specifically when one of those changes, not on a fixed calendar schedule.
Try it yourself
Beginner. Take a training job you’ve actually run, personal project scale is fine, and answer all six questions honestly. For each one you can’t answer with real evidence, write one sentence naming exactly what you’d need to build or measure to answer it.
Intermediate. Implement the per-step phase-timing table above for a real or toy distributed job, and identify which single phase dominates total_step. Then explain what that dominant phase implies about which of the three responses, simplify, instrument, or switch strategy, would actually help.
Advanced. Design a chaos drill that combines two failure classes from this series at once, a slow rank and a checkpoint corrupted mid-write, arriving within the same recovery window, and walk through whether auto-cordon’s drain-wait-checkpoint-resize-resume sequence handles the compound case correctly or silently assumes only one failure happens at a time.
The one-sentence version: production-ready is not a feeling a system gives you after a run that happened to finish, it’s six specific questions, why this layout, what’s the bottleneck, can you recover from three different kinds of failure, can you prove a checkpoint is actually valid, can you tell a real win from noise, can you show observability on demand, and every incident, formula, and worked number across this entire series was the evidence for one of those six the whole time. DDP’s opening promise, that every rank trains on a distinct, disjoint slice of the same global shuffle, was never really about set_epoch(), it was the first, smallest instance of the same underlying discipline this whole arc has been building toward: don’t assume the system is doing what you think it’s doing, check. Applied once, in full, against a real 671B-parameter production mesh, that discipline is also what explains why a frontier lab shipped without one of the six techniques entirely.