Goodput, DCGM, and the correlated failure your straggler detector can't see
Why a peer-relative straggler check is structurally blind to a datacenter-wide cooling problem, the exact auto-cordon sequence and why each step exists, and the goodput math behind Llama 3 hitting 90%+ utilization despite an interruption every three hours.
One failing GPU drags 15,999 others down with it. Because training is synchronous, every rank has to reach the same collective operation before any of them proceed, so a single slow or dead machine doesn’t cost th of throughput. It can cost all of it, for as long as the rest of the fleet sits waiting. That’s the entire reason cluster operations exists as its own discipline rather than something a training job just handles: the blast radius of one machine failing at this scale is the whole cluster, and the entire economic value of the run rides on how fast that failure gets detected, diagnosed, and routed around.
The baseline worth internalizing before any of this looks like anything other than paranoid over-engineering: Meta’s Llama 3 run hit 419 unexpected interruptions over 54 days, roughly one every three hours, for nearly two months straight, and that’s the documented baseline, not an unusually unlucky run. ByteDance’s MegaScale reports needing to repair and recover their training process over 100 times across a comparable run. At this scale, failure isn’t an edge case you defend against, it’s the ambient operating condition you build for.
Topology-aware placement and gang scheduling, getting the physical layout right before a job even starts, and the flight recorder and per-rank step-time telemetry for catching a straggler mid-run, were both covered earlier in this series. What follows is what wasn’t: how you actually find out why a flagged rank is slow, the specific blind spot in the detection method already covered, the precise auto-cordon sequence and why each step in it exists, and the single number that tells you whether all of this machinery is actually working.
Straggler detection tells you who. DCGM tells you why.
NVIDIA DCGM (Data Center GPU Manager) tracks the underlying hardware signals, ECC error rates, GPU temperature, power draw, and per-link NVLink bandwidth, continuously, in the background, on every GPU in the fleet. This matters as a distinct system from straggler detection rather than being folded into it, because straggler detection tells you a rank is slow; DCGM correlation tells you why. A rank that’s slow with elevated ECC error counts is a hardware failure in progress, the GPU’s memory is producing correctable errors that are slowing computation and will likely progress to an uncorrectable one. A rank that’s slow with clean ECC and normal thermals but degraded bandwidth on one specific NVLink points at a cable or connector issue on that link specifically, not the GPU itself. That distinction, hardware failure versus network-link issue versus something else entirely, determines the entire remediation path, and you only get it by correlating the symptom against the underlying health telemetry, never from the step-time signal alone.
The blind spot: what happens when everyone is degraded together
Here’s the failure mode that catches people who’ve only implemented the naive version of straggler detection. Thermal throttling isn’t always an isolated single-GPU problem. Data center temperature swings, a cooling system operating near capacity, an unusually hot day affecting ambient intake temperature, can cause correlated throttling across many nodes simultaneously, not just one, and this has been identified as a real production cause in both the Llama 3 and MegaScale reports.
If straggler detection only looks for a single outlier rank relative to the rest of the fleet, a scenario where 200 GPUs across a rack are all throttling together because of a shared cooling issue can look like “everything is a little slow” rather than triggering any individual straggler alert at all, because the ratio-to-median calculation compares each rank against its peers, and if the peers are also degraded, the ratio never crosses the threshold. A relative check is structurally blind to a correlated, fleet-wide degradation by construction, not by an oversight in tuning the threshold. Catching this needs a different kind of check entirely: an absolute comparison against historical baseline step times from healthy runs, not just against the current run’s peer median. Running both checks side by side, one relative and one absolute, is what closes the gap, and it’s a genuinely non-obvious requirement that a well-intentioned, correctly-implemented relative-only system will simply never surface.
The exact auto-cordon sequence, and why each step is load-bearing
The mechanism that converts a hardware failure from a multi-hour, all-hands incident into a five-minute blip is worth knowing as an exact sequence, not a one-line summary, because each step solves a specific problem the step before it doesn’t handle on its own.
A health check, straggler detection, DCGM correlation, or an explicit hardware fault, flags a specific node as bad. That node is marked as draining, it won’t accept new work, but it isn’t ripped out instantly. The system waits for the in-flight training step to complete, killing a step mid-computation risks leaving the job in an inconsistent state that’s harder to cleanly recover from than just letting the step finish. Once the step completes, the job checkpoints, the safety net, whatever state exists right now is durably saved before anything else changes. The NCCL communicator is then resized to exclude the bad node, a genuinely nontrivial step, the collective communication group every remaining rank belongs to has to be reformed with updated membership, you can’t just remove a member from an existing communicator, you have to rebuild it. Training then resumes on the remaining healthy GPUs.
Skip the wait-for-in-flight-step and you risk a corrupted or unrecoverable intermediate state. Skip the checkpoint and there’s nothing durable to fall back to if something goes wrong during the resize itself. Skip the proper communicator resize and you don’t have a working distributed job anymore, you have a stale reference to a rank that no longer exists, exactly the kind of thing that causes the silent NCCL hangs this discipline exists to prevent in the first place.
Without this sequence, one dead GPU means all 16,000 sit idle while on-call manually diagnoses the failure, identifies the bad node, removes it, and restarts the job, historically on the order of three hours. With it, the same failure costs roughly five minutes, the time for the health check to fire, the in-flight step to finish, the checkpoint to complete, and the communicator to reform. The goal was never eliminating failures. It was making each one cost minutes instead of hours.
Goodput: the one number that tells you if any of this is working
Of all the time a job has been “running” from a clock’s perspective, what fraction was actually spent doing useful training computation, as opposed to time lost to checkpointing overhead, failure recovery, straggler-induced slowdowns, or idle waiting. The target that anchors this in reality isn’t an arbitrary round number: greater than 90% is the figure Meta reported maintaining across Llama 3’s full 54-day run, despite 419 separate interruptions. An interruption roughly every three hours, and still better than 90% of all wall-clock time spent on genuinely useful computation. That combination is only possible because of everything above, fast detection, fast diagnosis, fast auto-cordon-and-resume, each interruption costing minutes rather than hours is precisely what makes a >90% goodput figure achievable even at that failure rate.
Production systems track compute time, checkpoint time, recovery time, and idle time as separate line items, not folded into one aggregate percentage, because the breakdown is what tells you where to actually invest engineering effort next. A genuinely useful, worth-internalizing finding from real production systems: most goodput loss traces back to hardware failures, not software issues. That’s real operational intelligence, it means the highest-leverage place to invest isn’t necessarily more code review or better training scripts, it’s faster hardware failure detection and faster recovery mechanics, because that’s empirically where the time is actually being lost.
The number that makes checkpoint reliability itself a hard dependency, not a nice-to-have. Falling back to a checkpoint from six hours ago on an 8,000-GPU run, at roughly 144,000 in wasted compute for that single incident. The entire “resume from last checkpoint” step in auto-cordon is worthless if the checkpoint itself might be corrupted or non-deterministic on resume, and a number that large is exactly why checkpoint validation sits underneath cluster operations as a hard dependency rather than downstream polish.
The standing checklist this actually reduces to
Topology constraints enforced at scheduling time, not checked after the fact, tensor-parallel groups physically verified to land on NVLink-connected nodes before a job launches, since catching this after the run has started means restarting to fix it. Gang scheduling enabled and correctly configured, so a distributed job never partially starts and sits blocked at NCCL init waiting for the rest of its allocation. Per-rank step-time telemetry running from step one, with both the relative-ratio-to-median check and an absolute historical-baseline check running side by side, specifically because the relative check alone is structurally blind to correlated, fleet-wide degradation. DCGM or equivalent health monitoring wired directly into the straggler-alert pipeline, so a flagged straggler comes with an immediate hardware correlation rather than requiring a human to separately go look it up. Auto-cordon implemented as the full sequence, drain, wait, checkpoint, resize, resume, not a partial version that skips the checkpoint or attempts to hot-swap the communicator without properly reforming it. Goodput tracked continuously and broken down by category, not just as one aggregate percentage. And a checkpoint-recovery system trustworthy enough for auto-cordon to actually depend on, because $144,000 is what a single bad checkpoint costs when that trust turns out to be misplaced.
Sourcing worth being precise about
Two claims about this exact category of hiring keep resurfacing in slightly different phrasing across sources, and it’s worth checking them directly rather than repeating either. A Terraform, OpenTofu, and Ansible skill combination gets attributed to Poolside’s infrastructure hiring in more than one place. Checked directly against Poolside’s actual posted requirements: the confirmed language is “strong programming skills in Python, as well as systems languages like Go, C, C++, Rust, or Scala,” “experience with extreme scale (100s of TB to multi-PB) distributed systems,” and “previous hands-on experience shipping and maintaining Kubernetes-based critical infrastructure.” No Terraform, OpenTofu, or Ansible anywhere in it. Similarly, a version of “respond to on-call incidents that can’t wait for tomorrow” gets attributed to Anthropic’s Pretraining Scaling posting; the actual confirmed language there is that the work is “highly operational, you’ll be deeply involved in keeping our production models training smoothly,” with the team working “extended hours” and responding “to issues on evenings and weekends” during launches. Close in substance, not verbatim, and worth using the real wording rather than a paraphrase dressed as a quote, especially in a topic where “trust the primary source, not the secondhand summary” is the entire operational discipline being taught.
What does check out directly: DeepSeek’s own hiring language describes reinforcing “the teams responsible for infrastructure required to train and run large language models: specialists in AI computing centers, distributed storage, networking, and training platforms,” confirming this is a live, actively growing hiring category at a frontier lab, not a mature, fully staffed function people rarely get hired into.
Common mistakes
Treating a clean straggler dashboard as proof the cluster is healthy: a peer-relative check can’t see a correlated, fleet-wide slowdown by construction, and it takes a separate absolute-baseline check to catch a shared cooling problem instead of an isolated bad GPU.
Skipping steps in the auto-cordon sequence to make recovery faster: each step, especially the checkpoint before the communicator resize, exists specifically to prevent a worse failure than the one being recovered from.
Watching goodput as a single aggregate percentage instead of its breakdown: the aggregate number tells you something is being lost, only the compute/checkpoint/recovery/idle breakdown tells you where to actually spend the next engineering hour.
Treating checkpoint reliability as separate from, and lower priority than, cluster operations: the entire auto-cordon mechanism is worthless the moment a checkpoint turns out to be corrupted, and the dollar cost of that failure is concrete enough to price directly.
Try it yourself
Beginner. Using the goodput formula, compute the goodput for a 24-hour run that spent 21.6 hours on successful compute, 1 hour on checkpointing, and the remainder on recovery from two incidents, and compare it against the greater-than-90% target.
Intermediate. Design the absolute historical-baseline check described above: what data would you need to collect from healthy runs, and what statistic (not just a raw threshold) would make the check robust to normal day-to-day variance in step time across different workloads.
Advanced. Walk through what would go wrong, precisely, if auto-cordon’s NCCL communicator resize step were skipped and the system instead tried to keep the existing communicator with the bad rank simply removed from future collective calls, connecting your answer back to why a communicator has to be rebuilt rather than patched.
The one-sentence version: DCGM turns “this rank is slow” into “this rank is slow because of X,” a peer-relative straggler check is structurally blind to a cooling problem that degrades every rank in a rack together, auto-cordon’s drain-wait-checkpoint-resize-resume sequence exists precisely because skipping any single step trades a five-minute recovery for a much worse failure, and goodput above 90% despite an interruption every three hours is only possible because every one of those mechanisms is doing its job in the background, all the time, not just when something visibly breaks.