Partial failure is the default: stragglers, deadlocks, and coordinating GPUs that can't see each other

The networking post covered how a gradient physically crosses the wire. This one is about what it means to coordinate many machines toward one goal when any of them can be slow or dead at any moment, and none of them can directly see what any other is doing. Four terms are worth distinguishing precisely before anything else, because sloppiness here is the root of most confusion later:

• Concurrency: multiple tasks make progress in an interleaved way, taking turns, without necessarily running at the same instant.
• Parallelism: multiple tasks run simultaneously, on genuinely separate hardware.
• Distributed system: independent machines connected by an unreliable network, with no shared clock and no shared memory. Everything one machine knows about another, it knows only because a message arrived, and that message might never arrive, might arrive late, or might arrive out of order.
• Partial failure: some components fail while others keep running. And this is the load-bearing idea of the whole post: at large scale, partial failure is the default state of the system, not an exception to it.

Why one barrier makes the whole job fragile#

Synchronous SGD, the standard way large models train, requires one giant barrier per training step: every participating GPU must finish before the collective AllReduce can complete. A barrier is a rendezvous with no partial credit, every worker calls barrier(), and none proceed past that point until all of them have. AllReduce is a barrier in disguise: it equals barrier plus data aggregation, and since every round of the reduce-scatter/all-gather ring needs every GPU to have arrived, the whole multi-round operation is a barrier repeated $N-1$ times.

The consequence is unforgiving. One slow or dead GPU stalls every other GPU in the job, not most of them, because the barrier has no notion of “close enough,” only “everyone or nobody.” This is why the numbers are as bad as they are. In a study of 27 large-scale (512 to 1024 GPU) jobs, 16 hit slow-failures that delayed average completion time by 34.59%; 20% of jobs were delayed more than 50%; the mean slow-failure lasted 72 minutes. The engineering goal, stated plainly, is not zero failures. It’s making each failure cost 5 minutes instead of 3 hours.

The straggler: the failure that hides in the averages#

A straggler is a GPU that runs slower than its peers without dying. One GPU running 12% slower, from thermal throttling, say, makes every training step 12% slower, not 12% of steps, because the barrier waits for the slowest participant every single time. In a concrete measured case, a single straggler dropped MFU (Model FLOPs Utilization, the ratio of achieved compute throughput to the hardware’s theoretical peak) from 44% to 37%.

The insidious part is how it’s detected: per-rank step-time monitoring, not aggregate cluster metrics. Aggregate metrics average the straggler’s slowness across all ranks and hide it entirely, so the dashboard looks healthy while the job bleeds throughput. This is the same lesson as the OOM killer being silent to your process but loud to the kernel: the signal exists, just not where you’d first look.

One subtlety worth holding onto, because it’s a trap: temperature is a correlate, not a diagnosis. In that same study, a GPU running 20% slow was also running near 70°C, but only about 0.5% of elevated-temperature cases actually degraded performance. Rising temperature is a thing to check, not a thing to conclude from.

Straggler vs deadlock: identical symptom, opposite fix#

A deadlock is different in kind from a straggler, even though from the outside both present identically as “the job appears frozen, no error.” A straggler means everyone eventually finishes, just some slower. A deadlock means worker A waits for something only B can provide while B simultaneously waits for something only A can provide, and neither proceeds, ever, without intervention. The difference decides the fix: a straggler resolves itself or responds to mitigation; a deadlock never resolves on its own no matter how long you wait. Confusing the two means either restarting a job that would have recovered, or waiting forever on one that won’t.

Fail-stop vs fail-slow: the distinction that makes checkpoint-and-restart backfire#

Reliability issues split into two categories that demand different responses. Fail-stop is a crash: the component is gone. Fail-slow is a still-functioning but slow straggler. The common industry practice is to manually detect fail-slows and just treat them as fail-stops, using checkpoint-and-restart, which is labor-intensive and, more importantly, can be actively counterproductive.

Here’s the arithmetic that makes that clear: dumping a GPT2-100B model takes nearly 100 minutes, longer than the mean fail-slow duration of 72 minutes in that cluster. If your fix for a slowdown is checkpoint-and-restart, and the average slowdown would have resolved in 72 minutes, but checkpointing alone takes 100 minutes, your fix costs more than the problem. That single comparison is the entire reason fail-slow detection deserves to be its own discipline rather than a special case of crash recovery.

The four mitigation strategies, ranked#

Strategy	What it does	Fixes	Overhead
S1: Do nothing	Hope the straggler self-resolves	Neither type	None
S2: Adjust micro-batch distribution	Redistribute work away from slow GPUs, proportional to measured speed	Computation stragglers only	Low (~30s even at 512 groups)
S3: Adjust parallelism topology	Reroute congested links, consolidate stragglers into fewer pipeline stages	Both computation and communication stragglers	Medium (~1 min, in-memory param swap)
S4: Checkpoint-and-restart	Treat as fail-stop, replace the component	Eliminates it completely	High, potentially longer than the problem itself

The optimal choice is the classical ski-rental problem: start with the cheap strategy and escalate only when accumulated slowdown cost exceeds the next strategy’s overhead, reacting to cost that has already accrued rather than trying to predict how long the slowdown will last in advance. Measured at scale, a 64-GPU run with 8 injected computation and 2 injected communication slow-failures dropped to 14.8 iterations/min from a healthy 17.1 when unmitigated; with mitigation it recovered to 16.2, a 60.1% reduction in the slowdown’s impact.

Failure modes, each traced to its mechanism#

Failure	Mechanism	Cost blind	Cost with tooling
NCCL hang: rank 7 crashes before AllReduce	Rank 0 waits at the barrier forever	Completely blind to which rank, which collective	~5 min with the NCCL flight recorder
Thermal-throttling straggler	One GPU silently slower every step, barrier waits every time	Invisible in aggregate metrics	Per-rank step-time monitoring
Network-congestion straggler	Congestion-notification packet surge correlates with SM-utilization drops, GPUs stay healthy	Looks like a GPU problem	Correlate NIC congestion with SM utilization
Deadlock	A waits for B, B waits for A	Identical symptom to a straggler	Needs tooling that distinguishes; deadlock never self-resolves

These aren’t rare. ByteDance’s MegaScale reports 100+ such events in a single multi-week production run, routine rather than exceptional. And the finding generalizes across organizations: slow-failures have been reported in Meta’s Llama training and ByteDance’s MegaScale, and engineers report the same on single-tenant clusters of over 10,000 GPUs. The consensus is that slow-failures are hard to detect at scale, three independent organizations converging on the same conclusion, not one team’s bad luck.

Diagnostic commands worth knowing#

# NCCL flight recorder -- turns "completely blind" into ~5 minutes to diagnose
export TORCH_NCCL_DUMP_ON_TIMEOUT=1
export TORCH_NCCL_ASYNC_ERROR_HANDLING=1

# Per-rank step-time monitoring -- catches a straggler aggregate metrics would hide
# (illustrative; real instrumentation is framework-specific)
for rank in $(seq 0 $((WORLD_SIZE-1))); do
    echo "rank $rank last step time: $(tail -1 /tmp/step_times/rank_$rank.log)"
done

The technique researchers use to test mitigation systems is worth knowing too, because it tells you exactly what a real slow-failure looks like: lock the GPU SM frequency with nvidia-smi -lgc to mimic performance degradation, and start side-channel communication jobs to create network-bandwidth contention for a communication slow-failure. If you want to build straggler tooling, that’s how you generate ground truth to test it against.

What “frontier-job-ready” actually means, on three axes not one#

It’s tempting to define readiness for this work as “can diagnose these failures.” That’s necessary and it’s not sufficient, and being honest about the gap matters more than sounding complete. The actual hiring bar at frontier labs runs along at least three independent axes.

Technical and diagnostic fluency. This is the axis everything above targets. Given “the job appears frozen,” distinguish which of three genuinely different causes it is before acting: a dead rank (an OOM kill), a network partition (a link flap), or a genuine deadlock, because the fix for each is completely different and they look identical from outside. Given “aggregate utilization looks fine but training is slow,” know that aggregate metrics specifically hide stragglers. Given a slowdown, ask “fail-stop or fail-slow” before reaching for checkpoint-and-restart, because of the 100-minutes-versus-72-minutes arithmetic. Anthropic’s own pretraining-scaling role phrases this as one competency, not three: debug and resolve complex issues “across the full stack, from hardware errors and networking to training dynamics and evaluation infrastructure.”

Operational resilience. A different requirement, and not a technical one. The verified language from actual postings is blunt about it: the work is “highly operational,” teams “respond to issues on evenings and weekends” during launches, and you’re expected to “thrive in controlled chaos… rather than overwhelmed, when juggling multiple urgent priorities.” Mistral’s version: “You don’t panic when you see OOM errors or when NCCL feels like not wanting to talk.” Every straggler, deadlock, and NCCL hang in this post is precisely the kind of event that temperament requirement is written for. It’s a baseline, not an aspiration.

Autonomy and ownership. The axis most easily left out of a technical writeup, and stated as a hard disqualifier at least one place. OpenAI names “people who need a well-defined roadmap” as people who won’t succeed, and asks for the ability to “own ambiguous problems end-to-end without needing a tightly specified roadmap.” Mistral: “You don’t need roadmaps: you just do. You don’t need a manager: you just ship.” Diagnosing a straggler at 2 AM during a launch is not a scripted procedure with someone assigning the next step, and the escalation logic in the mitigation table above (do nothing → micro-batch → topology → restart) is itself a judgment call under uncertainty, made in real time. The skill this axis names is making that call correctly under pressure, and no amount of additional technical depth substitutes for it, because it isn’t a technical requirement.

There’s a fourth axis a notes document simply can’t close: credentials. DeepMind research-scientist roles normally require a PhD and some list it verbatim; Anthropic states a bachelor’s minimum, OpenAI and Poolside list none. Worth knowing the fork exists before assuming any single profile is a universal ticket, but nothing you read makes it true.

Common mistakes#

Treating a frozen job as one kind of problem: dead rank, network partition, and deadlock present identically and are fixed completely differently, so “the job hung” is the start of diagnosis, not the diagnosis.

Trusting aggregate GPU utilization as a health signal: it averages stragglers away by construction, so a job can look fully healthy on the dashboard while a single throttled GPU taxes every step.

Reaching for checkpoint-and-restart on any slowdown reflexively: if the checkpoint takes longer than the slowdown would have lasted, you’ve spent more than you saved, which is why the fail-stop/fail-slow distinction is operational, not academic.

Concluding a hot GPU is the cause: only about 0.5% of elevated-temperature cases actually degrade performance, so temperature is a lead to follow, not a verdict.

Defining your own readiness purely by diagnostic skill: the postings ask, equally, whether you can operate calmly under launch pressure and own ambiguous problems without a roadmap, and those are separate axes that technical depth alone never fills.

Try it yourself#

Beginner. A job runs 256 GPUs; one is thermally throttled to 88% of the others’ speed. By what percentage does each training step slow down, and why is the answer not “1/256 of the steps”?

Intermediate. Given a 72-minute mean slow-failure and a 100-minute checkpoint dump, compute the break-even slow-failure duration below which checkpoint-and-restart is strictly worse than doing nothing, and describe where S2 (micro-batch redistribution, ~30s overhead) fits relative to that break-even.

Advanced. Frame the S1→S2→S3→S4 escalation as a ski-rental problem explicitly: define the “rent” (per-step slowdown cost) and the “buy” (each strategy’s overhead), and write the decision rule for when accumulated rent justifies escalating to the next strategy. Then explain why this reacts to accumulated cost rather than predicting slow-failure duration up front, and why that’s the safer design given that duration is unknowable in advance.

---

The one-sentence version: in a synchronous distributed job the barrier has no notion of “close enough,” so one slow or dead GPU taxes all of them, stragglers hide in aggregate metrics while deadlocks masquerade as stragglers, the fail-stop-versus-fail-slow distinction decides whether checkpoint-and-restart saves time or wastes it, and being ready to work on this at a frontier lab means clearing three separate bars, diagnosing the failure, staying calm while it’s on fire, and owning the fix without being told how, not just the first one. That same barrier is also why losing one pod out of a 512-GPU Kubernetes job takes down all 512, not just the one that got preempted.