How gradients cross the wire: TCP, RDMA, and why one dead link hangs 16,000 GPUs

The previous post assumed everything a program needs lives on one machine. The moment training needs more GPUs than a single machine holds, that assumption breaks: every gradient computed on machine A has to physically travel to machines B, C, D and back before the next training step can begin. Networking is the study of how that travel happens, how fast, and what breaks it.

Two numbers matter, and they are not the same number. Latency is how long until the first byte arrives once you ask. Bandwidth is how many bytes per second the link sustains once data is flowing. They don’t trade off predictably: a link can have enormous bandwidth and still be useless for small, frequent messages if its latency is high. Gradient synchronization is exactly the kind of workload that punishes getting this wrong, because it’s both large (whole-model gradients) and frequent (every single step).

The standard network stack, and why it’s too slow here#

The ordinary stack is application → TCP/UDP → IP → Ethernet, and each layer adds processing the CPU has to do: touch every packet, copy data between buffers, manage the TCP connection’s state (acknowledgments, retransmission, congestion control). Standard TCP over 100GbE Ethernet carries high CPU overhead, roughly 10μs latency, and a 100Gbps peak. That’s fine for serving web pages. It’s too slow for synchronizing gradients across dozens of machines every step, where the CPU touching every packet becomes the bottleneck long before the wire does.

RDMA and InfiniBand: taking the CPU out of the path#

RDMA (Remote Direct Memory Access) lets a GPU’s own DMA engine read and write a remote GPU’s HBM directly, without the CPU touching the data at all. GPUDirect RDMA extends this across the entire path, GPU → NIC → wire → NIC → GPU, with no CPU involvement anywhere in the chain. InfiniBand is the specialized physical fabric built for it: 200 to 400 Gbps at roughly 1μs latency.

Put together, InfiniBand plus RDMA delivers 400Gbps at about 1μs latency with zero CPU overhead, a 4 to 10x improvement in collective-communication throughput over standard TCP/Ethernet. This is the difference between a training run that’s compute-bound (good, the GPUs are the bottleneck) and one that’s communication-bound (bad, the GPUs sit idle waiting on the network). Without understanding this layer, you can’t diagnose why a run is communication-bound, or why one failed switch can hang thousands of GPUs at once.

Ring AllReduce, derived#

This is the actual algorithm that makes multi-machine gradient synchronization work, and it’s worth understanding precisely rather than as a named term. It runs in two phases across $N$ GPUs arranged in a ring.

Phase 1, reduce-scatter. Each GPU starts holding its own full gradient. Over $N-1$ rounds, each GPU sends a $1/N$-sized chunk to its ring neighbor and receives another chunk, summing as it goes. After $N-1$ rounds, every GPU holds exactly $1/N$ of the fully summed gradient, its assigned slice of the combined total, not its own gradient anymore.

Phase 2, all-gather. Over another $N-1$ rounds, those summed slices get passed around the same ring, so that by the end every GPU holds the complete, fully-summed gradient.

The total data each GPU moves is:

$\text{Data per GPU} \approx 2 \times \frac{N-1}{N} \times \text{gradient size}$

As $N$ grows, $(N-1)/N \to 1$, so this converges to about 2x the gradient size, independent of $N$. That’s the precise meaning of “bandwidth-optimal”: adding more GPUs to the ring does not increase how much data any single GPU has to move, only how many rounds it takes. That independence from $N$ is exactly why ring AllReduce became the standard for data-parallel training in the first place.

Here’s the number that makes this concrete: for a 70B-parameter model, an all-reduce over 100GbE can take roughly 19.6 seconds per step, against a typical training step of 20 to 40 seconds. Communication isn’t a small tax there, it’s potentially most of the step. Move that same all-reduce onto InfiniBand with RDMA and it drops into a range where the GPUs go back to being the bottleneck, which is where you want them.

Where ring stops being optimal#

Ring AllReduce’s bandwidth cost stays flat as $N$ grows, but its latency cost, the number of rounds, is $N-1$ and grows linearly. At small $N$ that round count is negligible next to the data-transfer time. Past roughly 64 nodes it starts to dominate, even though each round moves very little data. Tree-based algorithms trade a small bandwidth penalty for a $\log_2(N)$ round count instead of $N-1$: at 16,384 GPUs, that’s the difference between about 16,383 sequential rounds and about 14. This is why frontier labs training on tens of thousands of GPUs don’t use pure ring AllReduce uniformly, and why NCCL selects between ring and tree algorithms depending on scale and message size. The algorithm choice is itself a tunable lever, not a fixed constant.

Regime	Round count	Best algorithm
2 to 32 nodes	$N-1$, small enough to ignore	Ring (bandwidth-optimal, uniform link use)
64+ nodes	$N-1$ rounds start dominating latency	Tree ($\log_2 N$ depth), or a hybrid

Failure modes, each traced to its mechanism#

Failure	Mechanism	Why it cascades
InfiniBand link flap (physical cable issue)	One rank loses connectivity mid-AllReduce	The collective requires every rank in every round, so one missing rank means it never completes, and the entire job hangs, not just that rank
ECMP hash collision	Load balancer splits one RDMA flow across two paths	Packets arrive out of order, triggering a retransmission storm and a reported ~40% bandwidth drop
MTU mismatch	Jumbo frames (9000B) set on one hop but not the next	Every packet crossing that boundary gets fragmented, and performance collapses

The unifying pattern is the same one from every prior systems post: a synchronous collective (all ranks must finish together) means a single failure anywhere in the ring or tree stalls the entire job. The failure doesn’t stay local, it propagates to every participant by design, because that’s what “collective” means. A frozen 16,000-GPU cluster is very often one dead rank or one flapping cable, not a software bug, and knowing that is the difference between grepping the right log and burning a day.

A worked correction, because the source got it wrong#

The knowledge-graph node this was built from claimed network failures caused “~60% of 419 training interruptions” during Llama 3’s run. That doesn’t match Meta’s actual reported breakdown, and it’s worth showing the correction rather than quietly fixing it, because verifying named statistics against the primary report is the job.

Llama 3 405B trained on 16,384 H100 GPUs over 54 days, with 419 unexpected interruptions (plus 47 planned, 466 total). Of the 419: 148 (30.1%) were GPU failures including NVLink issues, and 72 (17.2%) were HBM3 memory failures, so GPU and memory causes together were about 58.7%, not network. Network switches and cables specifically caused 35 interruptions, 8.4% of the total, real but far smaller than “~60%.” The rest (software bugs, cables, adapters) made up the remainder, and network alone is a subset of that.

What the node got right, and what checks out: Meta did build NCCLX (an extended NCCL) and integrate an NCCL flight recorder in direct response to these interruptions, capturing collective-communication metadata and stack traces to speed up diagnosis. DeepSeek-V3’s custom multi-plane network topology, engineered for MoE All-to-All traffic by giving each expert-parallel group a dedicated network plane to prevent congestion, is a real, documented architectural choice. And ByteDance’s MegaScale (NSDI 2024) built millisecond-resolution network monitoring specifically for anomaly detection at scale. The tooling and architecture details were accurate; only the causal percentage was overstated.

Diagnostic commands worth knowing by hand#

# Confirm you're actually on NVLink/InfiniBand, not silently falling back to Ethernet
nccl-tests/build/all_reduce_perf -b 8 -e 256M -f 2 -g 8

# Check NCCL's own view of which network paths it selected
NCCL_DEBUG=INFO python train.py 2>&1 | grep -i "NET/IB\|NET/Socket"

# Force specific adapters if one isn't showing active despite being configured
NCCL_IB_HCA=mlx5_0:1,mlx5_1:1 NCCL_DEBUG=INFO python train.py

The most common quiet failure here is the first one: a misconfiguration that leaves NCCL falling back to TCP sockets over Ethernet instead of using the InfiniBand fabric you paid for, turning a 1μs path into a 10μs one and a compute-bound run into a communication-bound one, with no error anywhere. The NET/IB versus NET/Socket line in the debug output is how you catch it.

Common mistakes#

Reading per-GPU utilization as if it were step efficiency: a GPU can show high utilization while the actual bottleneck is the AllReduce it’s waiting on before the next step can start. Step time is the honest metric, not per-GPU busy percentage.

Assuming a hung cluster means a code bug: in a synchronous collective, one missing participant, a flapped link or a stalled rank, is sufficient to freeze thousands of healthy GPUs, and that’s the first thing to check, not the last.

Treating a scale-dependent slowdown as a hardware problem: a slowdown that only appears past a certain cluster size may be an algorithmic boundary, ring’s linear round count, not a failing component, and the fix is NCCL’s algorithm selection rather than a hardware swap.

Trusting a named training-run statistic from a secondary summary without checking the primary report: this post’s own source material got the Llama 3 network figure wrong, and the only reason it’s right here is that it got checked.

Try it yourself#

Beginner. Using the ring AllReduce data formula, compute the data moved per GPU for a 7B model (14GB of gradients in FP16) at $N=8$ and again at $N=512$, and confirm the per-GPU figure barely changes, that’s bandwidth-optimality made concrete.

Intermediate. Given the 19.6-second all-reduce-over-100GbE figure for a 70B model against a 20 to 40 second step, estimate what fraction of a training run’s wall-clock time is pure communication, then estimate how that fraction changes if InfiniBand cuts the all-reduce time by 5x.

Advanced. At 16,384 GPUs, work out the round counts for ring ($N-1$) versus tree ($\log_2 N$), and explain why the tree’s small per-round bandwidth penalty is still worth paying at that scale but not at $N=16$. Connect this back to why NCCL chooses between them by message size as well as node count.

---

The one-sentence version: standard TCP is too slow for gradient sync because the CPU touches every packet, RDMA over InfiniBand fixes that by moving data GPU-to-GPU with no CPU in the path, ring AllReduce moves a fixed ~2x the gradient size per GPU no matter how many GPUs you add (until its linear round count makes tree algorithms win at scale), and the defining hazard of this whole layer is that collectives are synchronous, so one dead link doesn’t fail one GPU, it hangs all of them at once. That synchronous barrier is also what makes a single slow GPU tax every other GPU in the job, which is where partial failure stops being an edge case and becomes the default.