Ring Attention and Ulysses shard a sequence two different ways, and MoE routing turns a router's output into a networking problem

Tensor and pipeline parallelism answer two walls: a layer too large for one GPU, and a model too deep for one GPU’s slice of it. Two more walls remain, and both are more specific than the first two, worth reaching for only when the architecture actually forces the issue rather than by default. A sequence long enough that attention’s own memory cost, not parameter memory, becomes the bottleneck is context parallelism’s problem. A model that’s architecturally sparse, a mixture of experts where each token only visits a handful of the model’s total parameters, is expert parallelism’s problem. They don’t share a solution, but they do share a primitive: All-to-All, the most structurally different of the four collectives, every rank sending different data to every other rank, shows up as the actual traffic pattern underneath both, in two different guises.

Context parallelism shards the sequence itself, not just its normalization#

Megatron’s sequence parallelism, covered alongside tensor parallelism, reclaims activation memory for LayerNorm and dropout inside an existing TP group, a memory optimization riding on TP’s own process group. Context parallelism is a different technique solving a different problem: it shards the actual input sequence for attention’s real computation, so that no single device ever holds the full sequence’s queries, keys, and values at once. Each device owns a chunk of the context and, because every token’s attention computation needs to see the entire sequence’s keys and values, not just its own chunk, has to obtain the rest of that information from other ranks somehow. How it obtains it is where the two dominant approaches diverge.

Ring Attention: point-to-point rotation, and an online softmax that never sees the whole row at once#

Each rank keeps its own queries fixed in place and rotates key/value blocks around a ring of ranks, one hop per round:

Round 0: Q0 attends K0, V0   (local, no communication)
Round 1: Q0 attends K1, V1   (received from the previous rank in the ring)
Round 2: Q0 attends K2, V2   (received from two hops away)
...

The genuinely nontrivial part isn’t the rotation, it’s that softmax normally needs the entire row of attention scores before it can normalize any of them, and no rank ever has the entire row here, only whatever KV block just arrived. The fix is the exact same online softmax FlashAttention already uses to avoid materializing the full attention matrix in the first place: a running max and normalizer updated incrementally as each new KV block arrives, rescaling the partial output computed so far every time a larger score is seen, so the final result is identical to having computed softmax over the full row at once despite never holding that full row anywhere simultaneously. Ring Attention and FlashAttention solve two different memory walls, one across ranks, one across HBM and SRAM, with literally the same numerical trick. Because each rank only needs the next KV block in the ring, not the whole sequence, the communication is point-to-point sends between ring neighbors, and because the next block’s transfer can be issued while the current block’s attention computation is still running, that communication has real room to overlap with compute rather than sitting exposed on the critical path.

The update rule itself, on one rank, working through however many KV blocks arrive:

import numpy as np

def ring_attention_step(q, k_blocks, v_blocks):
    running_max = np.full(q.shape[0], -np.inf)
    running_sum = np.zeros(q.shape[0])
    output = np.zeros((q.shape[0], v_blocks[0].shape[1]))

    for k, v in zip(k_blocks, v_blocks):          # one iteration per KV block received from the ring
        scores = q @ k.T
        block_max = scores.max(axis=1)
        new_max = np.maximum(running_max, block_max)
        correction = np.exp(running_max - new_max)  # rescales everything accumulated so far
        p = np.exp(scores - new_max[:, None])
        running_sum = running_sum * correction + p.sum(axis=1)
        output = output * correction[:, None] + p @ v
        running_max = new_max

    return output / running_sum[:, None]

q = np.random.randn(4, 8)
k_full, v_full = np.random.randn(12, 8), np.random.randn(12, 8)

# Reference: ordinary softmax attention over the whole sequence at once
ref_scores = q @ k_full.T
ref = (np.exp(ref_scores - ref_scores.max(axis=1, keepdims=True))
       / np.exp(ref_scores - ref_scores.max(axis=1, keepdims=True)).sum(axis=1, keepdims=True)) @ v_full

# Ring version: the same K, V split into 3 blocks of 4, arriving one at a time
ring = ring_attention_step(q, np.split(k_full, 3), np.split(v_full, 3))
print(np.allclose(ref, ring, atol=1e-6))   # True -- identical result, never holding all 12 keys' scores at once

The correction term is the entire mechanism: every time a new block’s scores contain a value larger than anything seen so far, both the running sum and the partial output computed from all previous blocks get rescaled by $e^{\text{old max} - \text{new max}}$ before the new block’s contribution is added, which is exactly what keeps the final result identical to a single, full-row softmax despite building it up incrementally.

Ulysses: an all-to-all relayout instead of a rotation#

Ulysses takes a structurally different approach: rather than rotating KV blocks past a fixed set of queries, it uses an All-to-All to transform the layout itself, exchanging a sequence-sharded representation for a head-sharded one, so that each device ends up holding a different subset of attention heads, each with access to the full sequence for those specific heads, rather than a subset of the sequence across all heads. Attention then runs locally and completely within each device’s assigned heads, no further communication needed until the next layout swap. Where Ring Attention pays for point-to-point transfers proportional to sequence length divided by ring size, Ulysses pays for two All-to-All relayouts per attention block, transfers whose cost scales with how the framework batches head and sequence dimensions rather than with ring hops. Neither is universally faster; which one wins depends on head count relative to context-parallel degree and on whether the network handles many-point-to-point traffic or full all-to-all traffic more gracefully on the specific fabric in use.

Reading Llama 3’s own CP dimension correctly, now that the mechanism is built#

Llama 3 405B’s published configuration showed CP moving from $1$ at 8K context to $16$ at 131K context, funded by shrinking data-parallel width from $64$ to $4$, at a measured MFU cost from $43\%$ down to $38\%$. That MFU drop is now something concretely explainable rather than just observed: it’s the real, paid cost of the ring rotations or all-to-all relayouts derived above, communication that a shorter-context configuration with $CP=1$ never has to issue at all, because at $CP=1$ every rank already holds the entire sequence and there’s nothing to rotate or relayout in the first place. Reach for context parallelism specifically when FlashAttention and activation checkpointing, already sufficient at moderate sequence lengths, stop being enough, not as a default add-on to increase GPU count.

Expert parallelism: routing turns a model architecture choice into a network traffic pattern#

A mixture-of-experts layer replaces one dense feed-forward block with many smaller expert blocks and a router that sends each token to only a handful of them. The systems consequence is that “running the model” now means physically moving tokens to wherever their assigned experts live:

Tokens on every rank
        │
        ▼
Router computes top-k expert assignment per token
        │
        ▼
Tokens packed and permuted by destination expert
        │
        ▼
All-to-All dispatch: each token physically sent to its expert's GPU
        │
        ▼
Expert GEMMs run, locally, on whichever tokens arrived
        │
        ▼
All-to-All combine: expert outputs sent back to each token's origin rank
        │
        ▼
Tokens unpermuted back into original sequence order

What the router actually hands the dispatch step, worked on a small batch so the irregularity is visible directly:

import numpy as np
np.random.seed(0)

n_tokens, n_experts, k = 8, 4, 2
router_logits = np.random.randn(n_tokens, n_experts)
topk_experts = np.argsort(-router_logits, axis=1)[:, :k]   # each token's top-k expert choices

dispatch = {e: [] for e in range(n_experts)}
for token_idx, experts in enumerate(topk_experts):
    for e in experts:
        dispatch[e].append(token_idx)

for expert, tokens in dispatch.items():
    print(f"expert {expert}: {len(tokens)} tokens -> {tokens}")
# expert 0: 6 tokens   expert 1: 3 tokens   expert 2: 5 tokens   expert 3: 2 tokens
# 8 tokens x top-2 = 16 total assignments, split 6/3/5/2 across 4 experts -- nobody chose this distribution, the router did

This dispatch dictionary is what All-to-All physically has to move: each entry is a distinct, GPU-specific payload whose size wasn’t known until router_logits was computed for this exact batch, which is the concrete version of “the traffic pattern is data-dependent” below. A fixed, evenly-sharded collective like AllReduce never has this problem, every rank always contributes and receives the same amount; an expert that drew 5 tokens this batch and 1 the next is where load imbalance and the need for node-limited routing actually come from.

Unlike TP’s or DDP’s collectives, this traffic pattern is data-dependent: how many tokens land on any given GPU depends on what the router decided for this specific batch, not on a fixed, predictable tensor shape known in advance. That’s what makes expert parallelism a genuinely different networking problem rather than just another sharding scheme: the fan-out pattern itself is irregular and has to be handled as irregular, not assumed away.

DeepSeek-V3: a 671B-parameter router, engineered as a networking problem from the start#

DeepSeek-V3 is $671$B total parameters with only $37$B activated per token, $256$ routed experts plus a shared expert, top-$8$ routing selecting which of the $256$ each token actually visits. Run naively, this is exactly the irregular all-to-all fan-out described above, at a scale where communication time can exceed 50% of total training time without deliberate engineering against it. Three specific mechanisms are the answer, each targeting a different part of that cost:

Node-limited routing caps how far a token’s routing can spread: DeepSeek constrained active experts for any given token to never span more than four nodes, $32$ GPUs, out of the full fleet. This bounds the worst-case fan-out of the all-to-all directly, rather than allowing any token to potentially route to any of hundreds of GPUs scattered arbitrarily across the cluster, and it’s the training-side analog of the custom multi-plane network topology DeepSeek-V3 built specifically for this traffic, giving expert-parallel groups dedicated network capacity because generic topology wasn’t sufficient for this specific collective’s shape.

A two-hop dispatch path sends tokens across nodes via InfiniBand first, then forwards them among the GPUs within the destination node over NVLink, rather than treating the all-to-all as one flat, undifferentiated transfer, matching the actual physical bandwidth hierarchy, fast intra-node, slower inter-node, instead of ignoring it.

DualPipe is the piece that actually recovers the throughput the other two make possible: a bidirectional pipeline schedule that feeds microbatches in from both ends of the pipeline simultaneously and splits each chunk into attention, all-to-all dispatch, MLP, and all-to-all combine phases explicitly, so the dispatch and combine communication for one chunk overlaps with the attention and MLP compute of another. Even with communication-to-computation running at roughly a $1{:}1$ ratio, near-zero exposed all-to-all overhead is achievable, the same overlap discipline already established for AllReduce and AllGather, applied to the one collective pattern, All-to-All, that’s structurally hardest to hide because its shape isn’t known until the router has already run.

Load balancing without an auxiliary loss fighting the real objective#

A router that isn’t actively balanced tends toward collapse, sending most tokens to a small set of experts while the rest sit undertrained, the MoE equivalent of a straggler problem, except the imbalance is in compute assignment rather than hardware health. The traditional fix is an auxiliary loss term added to the training objective specifically to encourage balanced routing, but that term competes with the actual language-modeling objective for gradient signal, a real trade-off, not a free correction. DeepSeek-V3’s alternative adds a learnable bias term to each expert’s routing score used only for the top-$k$ selection decision itself, excluded from the gating weights that actually scale each expert’s output, and adjusts that bias dynamically during training, upward for an under-utilized expert, downward for an over-utilized one, based on observed load. Balance gets enforced directly on the routing decision, without ever touching the primary gradient the way a competing loss term does.

When EP is the right call, and when it’s a self-inflicted networking problem#

Expert parallelism earns its complexity only for a genuinely sparse architecture where the quality-per-active-FLOP advantage over a dense model of comparable active size is already established, and only on a network that can sustain irregular all-to-all fan-out, with real per-expert load and dropped-token observability in place before the run starts, not added after a training job has already shown mysterious throughput problems. A dense model that already meets quality and cost targets, or a network that’s oversubscribed even for regular traffic patterns, is a strong signal to leave MoE and expert parallelism alone rather than adopt them for their own sake.

What this takes to be frontier-job-ready#

The technical axis: recognizing that CP and EP aren’t general-purpose scaling knobs the way DP is, both are specific answers to specific architectural pressure, long context or genuine sparsity, and reaching for either without that pressure actually present adds real communication cost for no corresponding benefit.

The operational axis, worth being precise about the difference between a verbatim requirement and an informed inference rather than treating them the same way: Poolside’s reinforcement-learning engineering role states directly, in the job description itself, “you will have access to thousands of GPUs in this team,” real language about operating at a scale where CP and EP stop being theoretical. Moonshot AI, by contrast, publishes no public job description for its Kimi K2 team; what can honestly be said is only an inference from the model’s own published technical choices: a trillion-parameter MoE architecture implies the team maintaining it needs real fluency in MoE training, expert parallelism, and load balancing, and a 256K-token context window implies the same for context parallelism and long-context training specifically. That’s a meaningfully weaker claim than a quoted requirement, and this series has tried throughout not to blur the two together.

The autonomy axis: DeepSeek’s own published engineering choices, node-limited routing, a bandwidth-aware two-hop dispatch path, a bias-based balancer, are the output of a team that diagnosed one specific networking problem and built three separate, targeted mechanisms against it rather than reaching for one generic fix. That diagnostic instinct, not memorized familiarity with any single one of the three, is what the exercises below are actually testing.

Common mistakes#

Treating context parallelism and Megatron sequence parallelism as the same technique because both shard “along the sequence”: one shards LayerNorm and dropout activation memory inside a TP group, the other shards attention’s actual computation across a long context, and confusing them means reaching for the wrong fix when either one’s specific problem shows up.

Assuming All-to-All traffic behaves like AllReduce or AllGather for capacity planning purposes: its cost is data-dependent on the router’s actual decisions for a given batch, not a fixed, predictable tensor shape, and node-limited routing exists specifically because unconstrained fan-out is a real, not hypothetical, network risk.

Reaching for an auxiliary balancing loss as the only option: it directly competes with the language-modeling gradient, and a bias-based approach that only touches the selection decision, not the gating weights, is a real, shipped alternative rather than a theoretical one.

Adopting MoE and expert parallelism for the parameter-count headline without the network and observability to support irregular all-to-all traffic: the complexity is worth paying only once the quality-per-active-FLOP case is actually established.

Try it yourself#

Beginner. For a batch of 4 tokens routed top-2 across 4 experts living on 4 different GPUs, write out one possible token-to-expert assignment and draw the exact all-to-all dispatch pattern (which GPU sends how many tokens to which other GPU) it produces.

Intermediate. Implement the online-softmax update rule (running max, running sum, rescaling on a new maximum) over three KV blocks arriving in sequence, and confirm the final normalized result matches computing softmax over all three blocks concatenated at once.

Advanced. Explain precisely why DeepSeek-V3’s bias-based load balancing, which touches only the top-$k$ selection score, cannot on its own guarantee two different tokens both destined for an over-loaded expert get redirected identically, and what an actual token-dropping or capacity-limit policy has to additionally provide that a routing-score bias alone does not.

---

The one-sentence version: Ring Attention pays for long context with point-to-point rotations and an online softmax that never needs the full row at once, Ulysses pays for it with an all-to-all layout swap instead, and expert parallelism turns a router’s per-token decision into a genuinely data-dependent all-to-all traffic pattern that DeepSeek-V3 tames with node-limited routing, a bandwidth-aware two-hop dispatch path, and a pipeline schedule built to hide that specific collective’s cost. Every dimension in this series, DDP’s replicated state, FSDP’s sharded state, tensor and pipeline parallelism’s split layers and split depth, and context and expert parallelism’s split sequences and split routing, is the same underlying question asked about a different resource: what doesn’t fit, and which collective is the honest price of making it fit anyway. Every one of those answers exists as real, runnable lines in a handful of production codebases, and reading them properly is its own skill, worth building on top of the derivations rather than assuming it follows automatically from them.