Four NCCL incidents, walked the way you'd actually work them

Every distributed training job, data parallel, FSDP, tensor parallel, MoE, is underneath all of it a small set of collective communication primitives running over a network, at a scale where the network is a first-class production concern rather than an implementation detail. NCCL is what implements those primitives over NVLink and InfiniBand plus RDMA, and it’s worth taking seriously for one blunt fact: NCCL hangs are the number one cause of frozen training jobs in production, and at 16,000-GPU scale, a frozen job means millions of dollars of hardware sitting idle while someone works out which one of 16,000 GPUs is the problem.

Four primitives, not four unrelated facts to memorize#

AllReduce: every GPU holds a tensor, the operation sums them all and gives every GPU the same, fully summed result. This is gradient synchronization in data-parallel training, derived precisely, with its exact bandwidth cost, earlier in this series: after backward, every rank has a different gradient computed on its own batch shard, and AllReduce makes every rank end up with the same, globally correct gradient before the optimizer step.

AllGather: every GPU holds a different shard of a tensor, and the operation gives every GPU the concatenation of all shards, no reduction, just collection. This is FSDP’s parameter reconstruction: each rank holds $1/N$ of a layer’s weights, and before that layer’s forward pass, AllGather reconstructs the full layer on every rank.

ReduceScatter: the inverse structure of AllGather. Every GPU holds a full tensor, the operation sums them together, but instead of every GPU getting the full sum, each GPU gets only its own shard of the summed result. This is FSDP’s gradient synchronization: after backward, you don’t need every rank to hold the full gradient, only the shard it’s responsible for updating.

All-to-All: the most structurally different of the four. Every GPU sends different data to every other GPU and receives different data from each in turn. This is MoE expert routing: GPU $i$ holds tokens that need to go to experts scattered across many GPUs, and All-to-All ships each token to whichever GPU hosts its assigned expert, then ships the results back.

AllReduce and ReduceScatter both involve a reduction. AllGather and All-to-All don’t reduce, they redistribute. AllGather and ReduceScatter are literally a matched pair, not a coincidence, FSDP uses one on the way into a layer and the other on the way out, the same underlying sharding idea applied in each direction.

Topology isn’t a fixed choice, it’s auto-detected#

Ring AllReduce’s derivation already showed the bandwidth-optimal result, roughly $2\times$ tensor size per GPU, independent of $N$, and where tree algorithms take over past roughly 64 nodes for their $\log_2 N$ round count instead of ring’s $N-1$. Worth adding here: NCCL doesn’t hardcode either. It auto-detects the actual physical topology, NVLink versus PCIe versus InfiniBand, how nodes are physically connected, and picks between ring and tree, tuning buffer sizes accordingly, with the choice overridable via environment variables for tuning or debugging. Assuming “NCCL always does ring” is exactly the kind of gap that shows up when a collective behaves differently on two clusters with different physical interconnects for reasons that have nothing to do with the training code.

All-to-All’s traffic pattern is a genuinely harder networking problem than the other three, because it’s every-node-to-every-node rather than a structured ring or tree. DeepSeek-V3 built a custom multi-plane network topology specifically for MoE All-to-All traffic, giving each expert-parallel group its own dedicated network plane, specifically to prevent the congestion that occurs when many GPUs simultaneously route tokens to many different experts across the fleet. A lab training a trillion-parameter MoE model at the frontier found the generic network topology wasn’t sufficient for this specific collective’s traffic shape, and built custom infrastructure because of it. That’s the concrete answer to why the specific collective operation matters for infrastructure decisions, not an abstract point.

The silent hang, in full#

This is the single most important operational fact in this area, worth the exact causal chain rather than the summary: NCCL has no timeout by default.

Rank 7 crashes, commonly because the Linux OOM killer silently terminates it, with no warning to the rest of the job, before it reaches an AllReduce call every other rank is already waiting at. Every surviving rank calls AllReduce and blocks at the barrier, waiting for rank 7 to arrive. Rank 7 never arrives, because it’s dead. With no default timeout, every surviving rank waits forever. The job doesn’t crash and doesn’t log an error. It just stops making progress, silently, while every other GPU sits idle waiting for a rank that will never respond. From the outside this looks identical to a job taking a long step, until enough time passes that someone notices the step counter hasn’t moved.

This is why collective timeout configuration is mandatory production setting, not an optional tuning knob: set NCCL_TIMEOUT explicitly, or run a separate watchdog that kills the job after N seconds of collective inactivity. Without one, a single dead rank silently costs you the wall-clock time until a human notices the dashboard hasn’t moved, historically hours, not minutes.

The NCCL flight recorder turns this from a multi-hour mystery into a fast diagnosis. Enabled via NCCL_ENABLE_FLIGHT_RECORDER=1, it logs the last $N$ collective operations per rank into a circular buffer. After a hang, dumping the buffer shows exactly which collective each rank was in when things stopped, and critically, which rank simply never showed up. Without it, 16,000 GPUs are frozen and there’s no way to know whether the problem is rank 7, rank 11,203, or a network switch between two racks. With it, diagnosis time drops to roughly 5 minutes.

Worth being precise about the actual Llama 3 numbers here, because a specific claim about this is worth double-checking rather than repeating: some accounts cite network and NCCL-related issues as causing roughly 59 to 60% of Llama 3 405B’s 419 unexpected interruptions. That figure doesn’t match Meta’s actual reported breakdown, GPU failures including NVLink issues plus HBM3 memory failures together were about 58.7%, and network switches and cables specifically caused 35 interruptions, 8.4% of the total. The ~60% figure appears to be the same mis-citation surfacing again from a different source repeating it, not a new, independently confirmed number, worth flagging again rather than letting a repeated claim quietly gain credibility through repetition. What does check out: Meta built NCCLX, an extended, hardened NCCL, and integrated the flight recorder directly into their training infrastructure in direct response to the real pattern of repeated hangs burning engineering time. ByteDance’s MegaScale took the same lesson further, building millisecond-resolution monitoring tracking NCCL metrics at multiple levels, because by the time training runs at this scale, checking a dashboard after something looks wrong isn’t fast enough.

The operational response, once you’ve diagnosed a hang#

Per-rank step-time telemetry, the same straggler-detection discipline covered earlier in this series, catches problems before they become a full silent hang: a rank running more than roughly 15% slower than the median gets flagged, its GPU health metrics get pulled automatically, and if the pattern persists across steps, on-call gets paged. If a rank has actually died, the response is auto-cordon: remove the failed node, resize the job to the remaining healthy GPUs, and resume from the last checkpoint, the mechanism that converts one dead GPU freezing 16,000 others for hours into one dead GPU costing the job about 5 minutes.

The optimization that actually recovers MFU: overlapping communication with compute#

Everything above treats a collective as something that happens, finishes, then compute resumes. In production, collectives aren’t run naively in sequence with compute, they’re overlapped, and the overlap is where a meaningful chunk of real-world throughput comes from. While a GPU is still computing the backward pass for layer $i$, it can simultaneously kick off the AllReduce for layer $i-1$‘s already-completed gradient, rather than waiting for all backward computation to finish before starting any communication. This requires careful scheduling, Megatron-LM implements it automatically rather than leaving it hand-tuned, and done correctly, overlap is worth up to roughly 30% MFU gain compared to running compute and communication strictly sequentially. The bandwidth math tells you how much data moves; whether that movement costs any wall-clock time at all depends entirely on whether it’s scheduled to overlap with compute that was going to happen anyway.

Four incidents, walked the way you’d actually work them#

Knowing that NCCL can hang isn’t the skill. The skill is recognizing which kind of problem you’re looking at from the symptoms alone, because these four look completely different from the outside and get misdiagnosed as each other constantly by people who haven’t seen enough of them.

Incident 1: the job never starts. A 512-GPU job is submitted. GPU utilization across all nodes sits at 0%. No training log output, no error, ten minutes in. The instinct is to suspect a training bug, bad config, a launch script typo, a stuck data loader, and it’s wrong here, which is exactly why this one’s worth walking through: the job hasn’t reached training code at all. The first real question is whether all 512 ranks have even found each other and agreed to form a collective group, rendezvous, which happens via NCCL_INIT_METHOD before any communicator, let alone any training step, exists. Rendezvous is all-or-nothing by design, the group can’t form partially, so a single rank that failed to launch means it never completes. Root cause here: one node had a stale container image, a version pin drifted, and that node was running a different NCCL version than the other 511. NCCL version mismatches at init time present as nothing happening, not a crash, because the mismatch prevents the ranks from ever agreeing to form the group. Fix: kill the job, fix the image pin, resubmit, exactly why container image pinning is treated as core to run reproducibility. The lesson: a hang with zero symptoms, no partial progress, no logs, pattern-matches to initialization, not a mid-training collective. Don’t reach for the flight recorder here, it logs collective operations, and if rendezvous never completed, there were none to log yet.

Incident 2: the job ran fine for six hours, then froze completely. Step count climbed steadily, then stopped. GPU utilization on most nodes still shows high, the counter just hasn’t moved in twenty minutes. The instinct is a slow data loader or an unusually long straggler step, both common and boring, and usually correct, which is why the line between “unusually long” and “actually stuck” matters here. GPUs showing utilization while the step counter is frozen isn’t a contradiction, a rank blocked at a barrier inside AllReduce, waiting for a peer that will never respond, still shows as active, because the GPU is executing the wait. The real check is whether any rank has moved to the next step. Pull the flight recorder: it shows every rank except rank 203 sitting inside the same AllReduce call, and rank 203 isn’t there. Its own logs show it was OOM-killed eleven minutes ago by an unrelated process on the same host consuming system RAM, with no signal sent to the rest of the job. The other 511 ranks are exactly where you’d expect, waiting at a barrier for a rank that no longer exists, with no default timeout to release them. Fix: because a watchdog was configured, it kills the job after threshold, auto-cordon removes the bad node, the job resizes, training resumes from the last checkpoint, costing roughly the time since the last checkpoint interval, not the hours it would take to notice manually. The lesson: GPU utilization looking fine is not evidence of progress, a rank blocked inside a collective looks busy. The step counter is the ground truth.

Incident 3: no hang, no crash, just quietly bad throughput. The job is running, the step counter moving, nothing frozen, but MFU has sat at roughly 60% of expected for hours, no stragglers flagged, no thermal issues, no recovery events logged. This is the one where people burn the most time, because the instinct is to look at compute, a bad kernel, a library regression, a slightly bigger model, all reasonable and all wrong here, because the problem is in the network layer underneath a collective that’s completing correctly, just slowly. Since nothing is hanging or crashing, the flight recorder shows nothing wrong, every collective completes, just late, which pushes the investigation down a level to actual network-layer inspection. A specific, known pattern: an ECMP hash collision, a load balancer splitting what should be a single, ordered RDMA flow across two physical paths with slightly different latencies, so packets arrive out of order, triggering retransmission, and a retransmit storm on an RDMA flow can cost roughly 40% of bandwidth, precisely consistent with an MFU number sitting at a fraction of expected with no compute-side explanation. Root cause: a recent network fabric change altered flow-hashing behavior, splitting a subset of RDMA flows that hadn’t been split before. Fix: reconfigure flow hashing to pin each RDMA flow to a single physical path, a network configuration change, not training code, not an NCCL setting. The lesson: a throughput problem with no hang, no crash, and no straggler signal is a strong hint to look below NCCL, at the physical network layer, rather than re-reading training code.

Incident 4: training works, but communication cost is 10x higher than it should be. FSDP-based training runs, completes steps, checkpoints fine, but MFU is roughly a third of what comparable hardware should get, and it’s been that way since the run started, not something that degraded. The instinct is to suspect hardware, bad GPUs, bad interconnect, because “slow since the start” pattern-matches to a hardware allocation problem rather than a configuration problem. But hardware faults tend to be intermittent or degrade over time, from thermal or ECC causes, and consistent slowness from the very start looks structural instead. Check the actual communication volume against what the architecture should require: how many AllGather calls is FSDP issuing per step, and does that match the model’s layer structure. Root cause: the FSDP auto_wrap_policy was set to wrap parameters at the individual-module level rather than the TransformerBlock level, so FSDP issues one AllGather per linear layer inside each transformer block instead of one AllGather for the entire block, a roughly 10x increase in collective calls for architecturally identical work. Fix: correct the wrap policy to operate at TransformerBlock granularity, matching FSDP’s communication pattern to the model’s actual architectural boundaries. The lesson: this failure never announces itself as a communication bug, nothing in “low MFU” points specifically at NCCL, and it’s tempting to go looking in compute-side places, kernels, precision, batch size, before checking whether the volume of communication issued matches what the architecture should actually require.

The pattern across all four#

The fix is almost never restart and hope. In each case the actual work was matching a symptom shape, zero progress, frozen after progress, slow but running, slow since the start, to the right diagnostic layer, rendezvous, mid-run collective, physical network, application-level configuration, and only then reaching for the specific tool that layer requires. The skill being exercised isn’t knowing a lot of NCCL facts. It’s holding several plausible explanations in mind, picking the cheapest check that would distinguish between them, and not committing to a root cause until the evidence actually points there. That discipline, cheap falsifying checks before an expensive fix, is what real on-call experience with this class of problem teaches, and it transfers to the next incident even when it isn’t exactly one of these four.

Where this actually gets tested#

“Derive the bandwidth cost of Ring AllReduce and explain why it’s bandwidth-optimal” tests the two-phase derivation landing on the $2\times$ tensor-size, $N$-independent result, not just reciting “it’s efficient.” “A job shows 0% GPU utilization ten minutes after submission versus a job that ran fine for six hours and then froze, do you diagnose these the same way” wants no, and the reasoning is exactly Incidents 1 and 2, rendezvous failure versus mid-run collective deadlock, testing whether the flight recorder gets reached for reflexively or the tool actually gets matched to the failure phase. “MFU is 60% of expected, nothing is hanging, nothing is crashing, where do you look” is Incident 3, and the answer showing real judgment is the network layer underneath NCCL, precisely because the absence of a hang or crash rules out the layers people check first. “Why does NCCL need both a ring algorithm and a tree algorithm, why not always use the bandwidth-optimal one” tests the ring-versus-tree latency tradeoff and that topology auto-detection is choosing based on real physical constraints, not a fixed default. “A MoE training run has fine AllReduce performance but terrible throughput during token routing, where do you look” wants All-to-All-specific network congestion, potentially topology work like DeepSeek-V3’s multi-plane design, not a generic “check the network.”

What this takes to be frontier-job-ready#

The technical axis is the four-primitive taxonomy itself, being able to say which collective a given distributed technique actually uses and why, not just that “communication happens.”

The operational axis is stated close to verbatim by at least one lab: comfort with the reality that NCCL can simply stop responding, for reasons that take real investigation to uncover, described directly in hiring language as staying calm and methodical rather than escalating into panic. A large fraction of the actual day-to-day experience of running large distributed training is exactly this, a job that’s supposed to be making progress, isn’t, and the reason isn’t obvious from outside, and the engineers who do well here have internalized that this is routine, not a crisis each time.

The autonomy axis is the four-incident pattern itself: no fixed script tells you which of rendezvous, dead rank, network flow, or config bug you’re facing, only the discipline of picking the cheapest check that distinguishes between plausible explanations before committing to a fix.

Common mistakes#

Reaching for the flight recorder regardless of symptom: it only helps once a job has successfully initialized and entered a collective, and it’s useless for a rendezvous failure where no collective has run yet.

Trusting GPU utilization as evidence of training progress: a rank blocked inside a collective call shows as busy while contributing zero forward progress, and the step counter, not the utilization graph, is the actual ground truth.

Repeating a network-failure percentage without checking it against the primary report: the same roughly-60% Llama 3 figure has now surfaced from more than one secondary source, and it still doesn’t match Meta’s own reported breakdown.

Assuming consistent low throughput since a run’s very start means bad hardware: hardware faults tend to be intermittent or to degrade over time, and a structural cause like a misconfigured FSDP wrap policy produces exactly the same symptom without touching the hardware at all.

Try it yourself#

Beginner. For each of the four collective operations, name which of the distributed techniques covered in this series, data-parallel training, FSDP, MoE routing, actually issues it, and explain in one sentence why that technique needs that specific operation and not one of the other three.

Intermediate. Given a job with 0% GPU utilization and no logs ten minutes after submission, write out the specific sequence of checks you’d run before concluding it’s a rendezvous failure, in the order you’d actually run them from cheapest to most expensive.

Advanced. Explain why an ECMP hash collision produces a throughput regression rather than a hang, in terms of what RDMA’s ordering guarantees require, and why the same underlying network change (altered flow hashing) could instead cause a full hang under a different collective’s traffic pattern.

---

The one-sentence version: AllReduce, AllGather, ReduceScatter, and All-to-All are the four primitives every distributed technique in this series reduces to, NCCL has no default timeout so one dead rank freezes every survivor at a barrier forever unless a watchdog is configured, and the actual skill this whole area demands isn’t memorizing failure modes, it’s matching a symptom shape, zero progress, frozen after progress, slow but running, slow from the start, to the right diagnostic layer before reaching for any specific tool at all. Diagnosing an incident is only half the job; the automated machinery that turns a diagnosis into a five-minute recovery instead of a three-hour one is the other half.