Mixtral 8x22B used tensor parallelism where DeepSeek-V3 didn't, and a 900-versus-400 GB/s NVLink gap is the entire reason why

DeepSeek-V3’s real, disclosed mesh skipped tensor parallelism entirely, a genuine, verified case of a frontier lab deciding TP’s per-layer collective cost wasn’t worth what it bought. That result is easy to over-generalize into “TP isn’t worth it for MoE models,” which is wrong, and a real, differently-configured MoE model is the fastest way to see exactly where that generalization breaks. NVIDIA’s Megatron-Core team published a paper, MoE Parallel Folding, benchmarking Mixtral 8x22B with tensor parallelism very much in the mesh, alongside expert, pipeline, context, and data parallelism, all six of this series’ techniques genuinely active at once, and the paper’s own tables are precise enough to show exactly why the same reasoning that eliminated TP for one model kept it for another.

The real comparison the paper actually reports#

Table 3 of the paper gives both the naive composition and the improved one, on the identical model and GPU count, which is the cleanest kind of tradeoff evidence this series has cited yet, an ablation, not just a single reported number:

Configuration	Attention mesh	MoE mesh	MFU
MCore, baseline	CP=1, TP=2, PP=8	EP=4 (shares attention’s TP)	46.3%
MCore, w/ Parallel Folding	CP=1, TP=2, PP=8	EP=8, ETP=1 (independent mesh)	49.3%

Same 128 GPUs, same model, same sequence length. The only change is letting the MoE layers use a different parallelism mesh than the attention layers, and that one change is worth 3 percentage points of MFU, a 6.5% relative improvement, on a real, measured benchmark. Qwen2-57B-A14B shows the same pattern at a smaller scale, 64 GPUs, 35.3% baseline against 39.0% folded, a 10.5% relative gain. Neither number is a simulation or a projection; both are what the paper reports actually running.

What “folding” means, in the arithmetic the paper’s own config implies#

The baseline configuration forces MoE’s expert-parallel degree to share the same process groups attention’s tensor parallelism already claimed, EP=4 sitting inside a mesh attention built for its own purposes. Folding’s actual contribution is decoupling the two: attention keeps its own mesh, $CP{=}1 \times TP{=}2 \times PP{=}8 = 16$, leaving a data-parallel width of $128/16=8$ for the attention parameters, while the MoE layers get an independent mesh entirely, $EP{=}8 \times ETP{=}1 = 8$, leaving an expert-data-parallel width of $128/8=16$ for the routed-expert parameters specifically. Both meshes span the same 128 physical GPUs; they just slice that shared hardware two different ways depending on which parameters are being addressed.

Neither mesh is “the real one” with the other layered on top; they’re two independent, co-existing views of the same physical allocation, chosen because attention’s compute pattern and MoE’s compute pattern have different needs, and forcing one mesh to serve both, the baseline row in the table above, is precisely what costs the 3 points of MFU.

The real code, both versions#

Megatron-Core’s actual CLI arguments, --tensor-model-parallel-size, --pipeline-model-parallel-size, --context-parallel-size, --expert-model-parallel-size, and --expert-tensor-parallel-size, are exactly what turn the table above into a real launch command. The baseline row, EP implicitly sharing attention’s tensor-parallel group:

torchrun --nproc_per_node=8 --nnodes=16 pretrain_gpt.py \
  --tensor-model-parallel-size 2 \
  --pipeline-model-parallel-size 8 \
  --context-parallel-size 1 \
  --expert-model-parallel-size 4 \
  --sequence-parallel

The folded row, adding exactly one flag that gives MoE its own tensor-parallel degree instead of inheriting attention’s:

torchrun --nproc_per_node=8 --nnodes=16 pretrain_gpt.py \
  --tensor-model-parallel-size 2 \
  --pipeline-model-parallel-size 8 \
  --context-parallel-size 1 \
  --expert-model-parallel-size 8 \
  --expert-tensor-parallel-size 1 \
  --sequence-parallel

That’s the entire diff, one flag, --expert-tensor-parallel-size 1 added, --expert-model-parallel-size doubled from 4 to 8 now that it’s free to move independently. Megatron-LM’s own argument validation sets expert_tensor_parallel_size to the value of tensor_model_parallel_size automatically whenever it isn’t passed explicitly, which is exactly why the baseline command above doesn’t need the flag at all, and exactly why omitting it silently reproduces the lower-MFU row instead of raising any error. A one-flag difference between a 46.3% and a 49.3% MFU run, with no error on either side to tell you which one you actually launched, is worth internalizing as its own lesson: the flag that’s silently defaulted is sometimes the one doing the most work.

Why TP earns its place here, and a precise number for why it didn’t there#

DeepSeek-V3’s own stated reason for skipping TP was that the NVLink bandwidth actually available didn’t clear the cost of TP’s per-layer collective. That reasoning wasn’t a preference, it was arithmetic against a specific number, and the specific number is where these two real systems diverge: DeepSeek-V3 trained on H800 GPUs, NVLink capped at 400 GB/s, an export-compliant variant with its interconnect bandwidth deliberately reduced below the H100’s; this benchmark ran on H100 GPUs, NVLink at 900 GB/s, better than double. The same per-layer-collective cost that didn’t clear the bar at 400 GB/s clears it comfortably at 900 GB/s, at a smaller model besides, 141B total parameters against DeepSeek-V3’s 671B, meaning less total communication volume to move across whichever link is available in the first place. Two frontier-adjacent teams ran the identical cost-benefit calculation TP always requires and landed on opposite answers, not because one of them was wrong, but because the actual number on the right-hand side of that inequality was different by more than a factor of two.

The scaling table, and what it’s honest about#

Table 4 reports MFU for Mixtral 8x22B with folding across four real GPU counts, not one:

GPUs	128	256	512	1024
MFU	52.2%	50.7%	48.9%	44.9%

Doubling the cluster four times over costs a bit more than 7 points of MFU, a real, measured, monotonic decline, not a flat line, because more GPUs means more participants in every collective and more opportunity for a small per-step overhead to compound. This is worth reading against the goodput-versus-MFU distinction already established in this series: a declining MFU curve as GPU count grows is not automatically a bug to chase down, it’s frequently the honest cost of scale itself, and the actual engineering question is whether the absolute throughput gain from more GPUs still outpaces that decline, not whether the percentage stays flat.

Table 5 reports the equivalent trade against context length instead of GPU count, at fixed 128-to-1024 GPU allocations:

Sequence length	16K	~131K
MFU	47.6%	42.9%

The same shape already measured in Llama 3’s real Table 4, $43\%$ at 8K tokens down to $38\%$ at 131K, a roughly 5-point cost for extending context by that much, shows up again here, on a completely different model, a different framework, and different hardware, at a similar roughly 5-point cost. That’s not a coincidence worth shrugging past: two independently reported, real benchmarks landing on close to the same-sized context-parallelism tax is exactly the kind of cross-checkable pattern this series has tried to build the habit of looking for, rather than trusting either number in isolation.

Common mistakes#

Generalizing DeepSeek-V3’s real decision to skip TP into “MoE models don’t need TP”: the decision was arithmetic against a specific, disclosed NVLink bandwidth number, and a different bandwidth or a different model size changes the answer, as this benchmark demonstrates directly.

Treating a parallelism flag’s silent default as equivalent to explicitly reasoning about it: expert_tensor_parallel_size quietly inheriting tensor_model_parallel_size is exactly how a run ends up on the lower-MFU baseline row without anyone choosing that outcome on purpose.

Reading a declining MFU-versus-scale curve as a bug by default: Table 4’s monotonic decline from 52.2% to 44.9% across a 8x GPU increase is the disclosed, expected cost of scale in this real system, not evidence something is misconfigured.

Try it yourself#

Beginner. Using Table 3’s baseline row, confirm the attention mesh’s arithmetic, $CP \times TP \times PP \times DP = 128$, by hand, then do the same for the MoE mesh in the folded row, $EP \times ETP \times EDP = 128$, and state in one sentence why both meshes are allowed to use the same total despite having different individual factors.

Intermediate. Write the single-flag diff between the baseline and folded launch commands above as an actual diff, and explain precisely what happens, mechanically, to a token’s routing path through the MoE layer when expert_model_parallel_size doubles from 4 to 8 while expert_tensor_parallel_size drops from an inherited 2 to an explicit 1.

Advanced. Using the real 900 GB/s and 400 GB/s NVLink figures for H100 and H800, and the AllReduce bandwidth model already derived in this series, estimate the relative wall-clock cost of one TP AllReduce on each GPU generation for a fixed tensor size, and use that estimate to argue, quantitatively rather than qualitatively, why DeepSeek-V3’s real decision to skip TP and this benchmark’s real decision to keep it are both correct for their respective hardware.

What this takes to be frontier-job-ready#

The technical axis is holding two real, contradictory-looking decisions as consistent instances of the same reasoning rather than as a rule and its exception: TP is not “on” or “off” as a matter of model architecture, it’s the output of a specific inequality, and being able to name the actual variables in that inequality, NVLink bandwidth, model size, GPU count, is what separates citing a result from being able to rederive it for a system neither DeepSeek nor this paper ever published.

The operational axis is this paper itself: a production framework team publishing an ablation table, baseline against improvement, on the same hardware, same model, same GPU count, is the real evidence-producing discipline this series’ production-readiness gate has been asking for throughout, not a single before-and-after anecdote but a controlled comparison someone could rerun.

The autonomy axis: the one-flag diff between a 46.3% and a 49.3% MFU run, with no error to distinguish them, is exactly the kind of silent default a real production engineer has to know to check for by habit, not because a linter or a config validator will ever catch it.

---

The one-sentence version: the same tensor-parallelism cost-benefit arithmetic that made DeepSeek-V3 skip TP entirely on 400 GB/s H800 NVLink kept TP firmly in Mixtral 8x22B’s real, measured mesh on 900 GB/s H100 NVLink, MoE Parallel Folding’s entire contribution is letting attention and MoE layers answer that arithmetic independently on the same physical GPUs instead of forcing one mesh to serve both, and the 3-point MFU gap between doing that and not doing it is a real, rerunnable number, not a claim taken on faith. Two real systems, two opposite TP decisions, one shared discipline: never generalize a real number into a rule without checking whether the variables that produced it are still the same ones in front of you. Both of these systems had the mature tooling to make that arithmetic clean; the team that didn’t yet have it is where most of that tooling actually came from.