DeepSeek-V3 trained 671 billion parameters without tensor parallelism at all, and that omission is the real production lesson

Every post in this series so far derived one technique at a time: DDP’s promise, FSDP’s lifecycle, tensor parallelism’s placement decision, pipeline parallelism’s bubble, context and expert parallelism’s traffic patterns. The real job, the one a staff engineer actually does, isn’t picking one of those. It’s staring at a real model registry entry, a real GPU allocation, and deciding which subset of everything derived so far the architecture and the hardware actually force, and which subset is a trap that burns cluster time for nothing. This post does that once, in full, on a real artifact: DeepSeek-V3’s actual production mesh, as disclosed in its own technical report, walked through against TorchTitan’s real, currently-checked-in deepseek_v3_671b() config. Not an invented scenario built to hit all six techniques for the sake of coverage. The real one, including the part where a frontier lab deliberately skipped one of the six entirely, because that decision is the actual lesson.

The artifact you actually inherit#

Nobody designs a training job from a blank file. You inherit a model registry entry that someone else wrote, and the first job is separating what’s intrinsic to the model from what’s a property of the specific cluster you’ve been handed. Here is TorchTitan’s real deepseek_v3_671b() function, fetched directly from the pytorch/torchtitan repository’s main branch, not paraphrased:

def deepseek_v3_671b() -> Trainer.Config:
    compile_config = CompileConfig(enable=True, components=["loss"])
    model_spec = model_registry(
        "671B",
        attn_backend="flex",
        converters=[
            Float8LinearConverter.Config(
                filter_fqns=["output", "router.gate"],
                model_compile_enabled=True,
            ),
            Float8GroupedExpertsConverter.Config(model_compile_enabled=True),
        ],
    )
    return Trainer.Config(
        loss=ChunkedLossWrapper.Config(
            loss_fn=CrossEntropyLoss.Config(global_vocab_size=decoder_vocab_size(model_spec)),
        ),
        hf_assets_path="./assets/hf/DeepSeek-V3.1-Base",
        model_spec=model_spec,
        dataloader=HuggingFaceTextDataLoader.Config(dataset="c4"),
        optimizer=default_adamw(lr=2.2e-4),
        lr_scheduler=LRSchedulersContainer.Config(
            warmup_steps=2000, decay_ratio=0.8, decay_type="cosine", min_lr_factor=0.1,
        ),
        training=TrainingConfig(local_batch_size=4, seq_len=4096, steps=10000),
        parallelism=ParallelismConfig(
            pipeline_parallel_schedule="Interleaved1F1B",
            expert_parallel_degree=2,
        ),
        checkpoint=CheckpointManager.Config(interval=500),
        activation_checkpoint=SelectiveAC.Config(),
        compile=compile_config,
    )

Notice exactly what’s not set: tensor_parallel_degree, pipeline_parallel_degree, context_parallel_degree, and data_parallel_replicate_degree are all absent, left at their framework defaults. That’s not an oversight, it’s a design decision worth internalizing on its own: FP8 recipe, learning rate, warmup, sequence length, and the base expert-parallel degree are model-intrinsic, they don’t change based on how many GPUs you happen to have today. TP, PP, CP, and the outer DP width are cluster-intrinsic, they get supplied as CLI overrides at launch, because they depend on an allocation this config file has no way of knowing in advance. TorchTitan’s real run_train.sh confirms the pattern directly: python3 -m torchtitan.train --module ${MODULE} --config ${CONFIG} "$@", where "$@" is exactly where those cluster-shape flags get appended. Conflating the two categories, hardcoding a cluster-specific degree into a model-intrinsic config, is a real, recurring source of configs that work on the cluster they were written for and silently misbehave on the next one.

The real cluster, and the mesh DeepSeek actually disclosed#

DeepSeek’s own technical report states the training cluster directly: 2,048 NVIDIA H800 GPUs, 256 nodes of 8 GPUs each, NVLink and NVSwitch inside a node, InfiniBand across nodes, eight 400Gbps IB NICs per node sustaining over 40GB/s of all-to-all traffic. That’s the real allocation this design has to fit. And the report discloses the real mesh chosen for it: pipeline parallelism at 16, using DualPipe’s asymmetric stage layout; expert parallelism at 64, distributing all 256 routed experts across that many GPUs, four experts per GPU on average; and, worth sitting with rather than skimming past, tensor parallelism was not used during training at all. The report’s own stated reason: TP’s cost is a collective at every single layer boundary, and under the NVLink bandwidth actually available at their target throughput, that cost wasn’t worth what it bought.

$16 \times 64 \times 2 = 2048$

That arithmetic is the whole mesh: PP claims $16$-way depth splitting first, which leaves $2048/16 = 128$ GPUs, sixteen nodes, per pipeline stage. Within one stage’s $128$ GPUs, expert parallelism claims $64$ of them to hold that stage’s complete share of the $256$ experts, four per GPU, a full, self-contained set, since EP is what does the actual sharding of the expert weights and there’s nothing left to sub-divide once all $256$ are placed. What the remaining factor of $2$ is doing to that already-complete $64$-GPU group is necessarily replication, not further sharding, the same product-check discipline this series ran against Llama 3’s Table 4 holds here too: $16 \times 64 \times 2$ lands exactly on $2048$, not approximately, and that arithmetic is what makes this mesh a coherent, checkable claim rather than four numbers that happen to be mentioned in the same paragraph.

Each small box is one 8-GPU node. The dashed boundary is the actual constraint DeepSeek’s paper names directly: a single token’s routing, even though its expert-parallel group spans all 64 GPUs across 8 nodes, is never allowed to touch more than 4 of those nodes, 32 GPUs. That’s the training-side counterpart to the custom multi-plane network topology already covered for this exact traffic pattern: bounding the worst-case fan-out of a data-dependent All-to-All before it ever has a chance to congest the fabric, rather than hoping the router behaves.

Reasoning through all six, including the one that got skipped#

Tensor parallelism: the deliberate non-choice. The reasoning that pins TP at 8 in Llama 3’s real config, because that’s the NVLink domain and TP fires a collective at every layer boundary, is the exact same reasoning DeepSeek’s team ran, taken one step further: if the achievable NVLink bandwidth at your target throughput doesn’t clear the cost of that per-layer collective even at the smallest useful TP degree, the correct answer isn’t “use less TP,” it’s “use none.” This is worth sitting with specifically because it contradicts a reflexive assumption: a 671B-parameter model with individual weight matrices plenty large enough to justify splitting across GPUs on memory grounds alone still shipped without TP, because memory-fits-the-layer and communication-is-worth-it are two different questions, and this is a real, disclosed case where the first was true and the second wasn’t.

Pipeline parallelism at 16, and what 61 layers over 16 stages actually implies. DeepSeek-V3 has 61 transformer layers total. Sixteen stages doesn’t divide that evenly, and the report is explicit that the layout is asymmetric, not a naive even split, because the embedding layer, the loss computation, and the mix of dense-versus-MoE layers near the front of the network don’t cost the same amount of compute per layer. The bubble formula derived earlier in this series, $\frac{P-1}{M+P-1}$, assumes reasonably balanced stages to begin with; an unbalanced stage doesn’t just cost its own idle time, it becomes the slowest stage every other stage waits on, every microbatch, which is exactly why DualPipe’s bidirectional scheduling, feeding microbatches in from both ends of the pipeline simultaneously, matters here specifically: it’s a direct answer to a 16-stage pipeline that can’t be made perfectly even by layer count alone.

Expert parallelism at 64, the load-bearing dimension for a 671B model. 37B of those 671B parameters activate per token; the other 634B sit sharded across the expert-parallel dimension, contributing zero compute to most tokens’ forward pass, which is the entire premise MoE trades on. EP=64 is what makes holding 256 experts, four per GPU on average, and the node-limited routing constraint diagrammed above is what keeps the resulting All-to-All from becoming the greater-than-50%-of-training-time communication tax already documented for this exact traffic pattern when left unconstrained.

The remaining factor of 2, and the trap in treating it as one number. This is the part worth getting exactly right rather than glossing over, because the honest answer is that “DP=2” means two different things depending on which parameters you’re asking about, and collapsing them into one number is exactly the kind of mistake this reasoning is prone to if the expert and dense parameters aren’t separated explicitly. For the expert weights specifically, EP=64 has already placed all 256 experts, four per GPU, as a complete set, there’s nothing smaller left to shard, so the factor of $2$ on top of it can only be data_parallel_replicate_degree, two full, independent copies of that already-complete 64-GPU expert-parallel group. For the dense parameters, attention, embeddings, norms, the router gate itself, none of that logic applies, they were never claimed by EP at all, and the same 16-bytes-per-parameter accounting already derived in this series means a real model’s dense-parameter optimizer state still needs real sharding to fit on 80GB H800s. TorchTitan’s ParallelismConfig exposes exactly this distinction as two separate fields, data_parallel_shard_degree (FSDP-style partitioning, defaulting to -1, meaning “auto-infer from whatever’s left”) and data_parallel_replicate_degree (full duplication, defaulting to 1), specifically because a real mesh composing EP with FSDP needs both answers at once, not one shared number standing in for two different memory strategies.

Context parallelism: present, but not in the number above. TorchTitan’s real registry entry trains at seq_len=4096, and $16 \times 64 \times 2$ already accounts for the full 2048-GPU budget with no room left for an independent CP dimension. DeepSeek’s own report separates pretraining from context-length extension as two distinct phases with two separate GPU-hour budgets, 2,664K for the main run against 119K for extension, which is the honest place to say plainly: what mesh that second, smaller-scale phase used for extending context isn’t the same disclosed detail as the main run’s mesh, and stating a specific CP degree for it here would be inventing a number this report doesn’t give. What the extension phase would need, Ring Attention’s point-to-point KV rotation or Ulysses’ all-to-all layout swap, is exactly what that earlier post derives; which one, at what degree, for this specific model, isn’t public.

The launch command, adapted from the real one#

TorchTitan’s actual multi-node SLURM script already wraps training in dcgmi profiling start/stop and configures NCCL for AWS EFA. Adapted with the real degrees derived above, supplied exactly where run_train.sh expects cluster-shape flags to be appended:

#!/bin/bash
#SBATCH --job-name=deepseek-v3-671b
#SBATCH --nodes=256
#SBATCH --gpus-per-task=8
#SBATCH --partition=training

nodes=( $(scontrol show hostnames "$SLURM_JOB_NODELIST") )
head_node_ip=$(srun --nodes=1 --ntasks=1 -w "${nodes[0]}" hostname --ip-address)
export NCCL_DEBUG=WARN

srun torchrun \
  --nnodes 256 --nproc_per_node 8 \
  --rdzv_id 671 --rdzv_backend c10d --rdzv_endpoint "$head_node_ip:29500" \
  -m torchtitan.train --module deepseek_v3 --config deepseek_v3_671b \
  --parallelism.pipeline_parallel_degree=16 \
  --parallelism.expert_parallel_degree=64 \
  --parallelism.data_parallel_replicate_degree=2

Everything above the srun torchrun line is cluster plumbing this series already built from first principles, rendezvous, rank identity, one process per GPU. data_parallel_shard_degree is deliberately left off this command entirely, not forgotten: its real default is -1, TorchTitan’s own sentinel for “shard the dense parameters across whatever’s left once the other dimensions are accounted for,” which is the correct, idiomatic choice here precisely because that width isn’t a number worth hand-deriving, it’s whatever the other four flags don’t already claim.

Before the cluster, the two debug modes that cost nothing#

TorchTitan’s run_train.sh ships two real, named debug modes worth using before a single real GPU-hour is spent: fake_backend, which validates the entire config, does the parallelism math check itself, and constructs the model, using fake process groups with no real communication and no torchrun, on a single GPU or even none; and local_tensor, which simulates the full multi-GPU communication and computation pattern on one shared GPU, for catching bugs in the distributed logic specifically, before those bugs get to hide inside real network variance. Running NGPU=2048 COMM_MODE="fake_backend" ./run_train.sh --config deepseek_v3_671b --parallelism.pipeline_parallel_degree=16 --parallelism.expert_parallel_degree=64 costs a few seconds on a laptop and would have caught the entire class of mistake this whole post has been warning about, a cluster-shape flag that doesn’t multiply out to the real GPU count, before it ever became a 256-node incident.

The iteration loop: what a bad result actually looks like, and where you go next#

Every branch in that flowchart already has a full, worked answer somewhere earlier in this series; none of it is new machinery invented for this scenario. What’s new here is applying it to this specific mesh: a NaN at step 4,000 on this job is worth checking against the FP8 recipe specifically, Float8LinearConverter excludes output and router.gate from quantization for a reason, and a config that accidentally quantized the router gate would produce exactly this symptom, invisible in the config diff unless you know to look for that specific exclusion list. An MFU that’s fine on the dense early layers but drops once the pipeline reaches its MoE-heavy stages points straight at the EP all-to-all rather than at PP or FSDP, because the phase table’s exposed_collective_time line will show exactly which collective is failing to hide behind compute, and on this mesh specifically, that collective is far more likely to be the expert dispatch/combine than anything TP-related, since there is no TP.

What running that loop actually looks like, in sequence. DeepSeek’s own paper doesn’t publish an attempt-by-attempt log, so what follows is this series’ own worked walkthrough of the flowchart above, illustrative rather than DeepSeek’s literal internal history, but built from exactly the failure modes this whole series has already established as real, not invented for effect:

1. Before any GPU is touched: NGPU=2048 COMM_MODE="fake_backend" ./run_train.sh --config deepseek_v3_671b --parallelism.pipeline_parallel_degree=16 --parallelism.expert_parallel_degree=64 --parallelism.data_parallel_replicate_degree=2 catches a transposed digit, expert_parallel_degree=46 instead of 64, which fails the mesh’s own arithmetic check instantly. Cost: seconds, on a laptop, not a cluster.
2. First real multi-node run, step 340: loss goes to NaN. The flowchart’s leftmost branch says check the FP8 recipe first. It turns out filter_fqns=["output", "router.gate"] was copied without the second entry, so the router gate’s logits were quantized to FP8 along with everything else, and a gate with insufficient precision to make a stable top-8 selection is exactly consistent with a NaN this specific and this early. Fixed by restoring the full exclusion list. Cost: real, but small, because per-rank phase telemetry flagged the divergence at step 340, not at step 3,400.
3. Second real run, stable past step 2,000, but MFU sitting well under what the disclosed 180K-GPU-hour-per-trillion-token figure implies: exposed_collective_time is elevated specifically on the MoE-heavy stages, not uniformly. The node-limited-routing constraint diagrammed above turns out not to be enforced by the router implementation actually deployed, tokens are landing across all 8 nodes of an EP group instead of staying within the intended 4-node boundary, exactly the unconstrained-fan-out cost already documented for this traffic pattern. Fixed by correcting the routing constraint. MFU recovers.
4. Third run, throughput now healthy, but goodput dips for six hours starting around day 4: DCGM correlation on the flagged ranks shows elevated temperatures concentrated in one rack, not one GPU, exactly the correlated-degradation blind spot a peer-relative-only straggler check misses. Auto-cordon isn’t triggered because no single rank crosses the relative threshold; an absolute historical-baseline check catches it instead. The rack gets cooled, the job resumes from the last validated checkpoint, and goodput recovers.

Four distinct failures, four different diagnostic layers, and every one of them was already a named failure mode somewhere earlier in this series before this specific job ever ran. That’s the actual claim worth taking from this sequence: the loop isn’t “keep trying things,” it’s “match the symptom to the layer, then reach for the tool that layer already has.”

What “correctly deployed” actually means here, against real numbers#

DeepSeek’s report gives real, disclosed outcome numbers, worth treating as the actual acceptance evidence rather than a marketing figure: 180K H800 GPU-hours per trillion tokens, 3.7 days per trillion tokens on the full 2,048-GPU cluster, 2,664K GPU-hours for the complete pretraining stage, 2.788M total GPU-hours including context extension and post-training, at roughly $5.576M total cost assuming $2 per GPU-hour. Applied against the six-question production-readiness gate already built in this series: a mesh whose degrees multiply out exactly to the allocated GPU count is question 1 and 2 answered before the job starts; a report that separately accounts for pretraining, context extension, and post-training GPU-hours rather than one aggregate number is exactly the goodput-style breakdown question 5 asks for; and a specific, named exclusion list in the FP8 recipe, rather than “we used FP8,” is what question 6’s observability bar actually looks like applied to a precision decision instead of a collective.

The number worth being honest about not having: DeepSeek’s report doesn’t publish a Llama-3-style Table 4 with a directly reported BF16/FP8 MFU percentage for this specific run the way Meta’s paper does. Stating one here would be inventing a figure this source doesn’t provide, exactly the mistake this series has caught in secondary sources more than once already. What’s real and disclosed is the GPU-hours-per-trillion-tokens efficiency figure above; treat that as the actual throughput evidence, and treat an MFU percentage for this specific run as an unconfirmed number until a primary source states one directly.

Common mistakes#

Hardcoding a cluster-specific parallelism degree into a model-intrinsic config file: it works on the cluster it was written for and produces a silent mismatch, not necessarily a crash, the moment it runs on a different allocation.

Assuming a large model always needs tensor parallelism because its layers are large: DeepSeek-V3’s real, disclosed choice was skipping TP entirely, because memory-fits-the-layer and communication-is-worth-it are different questions, and this run answered them differently.

Treating TorchTitan’s own registered hyperparameters (learning rate, warmup, batch size) as if they were DeepSeek’s original disclosed values: they’re two different sources, one is a reproduction recipe checked into a PyTorch-native framework, the other is the primary technical report, and conflating them misattributes a claim to the wrong source.

Stating an MFU percentage for this run because one exists for a different, better-documented run: the honest answer, when a primary source doesn’t disclose a number, is that it isn’t disclosed, not a plausible-sounding estimate presented as fact.

Try it yourself#

Beginner. Confirm the mesh arithmetic yourself: given $2048$ total GPUs, $\text{PP}=16$, and $\text{EP}=64$, compute the remaining DP-shard degree by hand, and state in one sentence why that number being small implies ZeRO-style sharding rather than full replication across it.

Intermediate. Using the debug-mode pattern described above, write out the exact NGPU=... COMM_MODE="fake_backend" command you’d run to validate a different hypothetical mesh, say $\text{PP}=8$, $\text{EP}=128$, on the same 2048-GPU allocation, and confirm the product still equals 2048 before ever writing the SLURM script.

Advanced. Using the bubble formula derived earlier in this series, explain why an asymmetric 16-stage layout across 61 layers with a mix of dense and MoE compute per layer cannot be evaluated with the plain $\frac{P-1}{M+P-1}$ formula as-is, and describe, in terms of stage-level compute time rather than layer count, what the correct generalization would need to account for.

What this takes to be frontier-job-ready#

This entire case study is the direct, concrete version of a hiring line already gathered earlier in this series: DeepSeek’s own stated hiring language describes reinforcing “the teams responsible for infrastructure required to train and run large language models: specialists in AI computing centers, distributed storage, networking, and training platforms.” Every noun in that sentence has a literal referent in this post, AI computing centers is the 256-node H800 allocation, distributed storage is the checkpoint validation this mesh depends on just as much as any smaller job, networking is the node-limited-routing boundary drawn in the diagram above, and training platforms is the difference between a model-intrinsic config file and a cluster-intrinsic launch command that this whole post turned on. The technical bar isn’t reciting that DeepSeek-V3 used PP=16 and EP=64. It’s being able to look at a real, unfamiliar model registry entry on a real, unfamiliar cluster and reconstruct that same reasoning cold, including the discipline to say “not disclosed” about the one number this report doesn’t give.

---

The one-sentence version: a real, currently-checked-in 671B-parameter config splits cleanly into what’s intrinsic to the model and what’s intrinsic to the cluster running it, DeepSeek’s own disclosed mesh, $16 \times 64 \times 2 = 2048$, with tensor parallelism deliberately absent, is proof that composing every technique this series has covered was never the goal, using exactly the subset the hardware and architecture actually force is, and every branch of what to do when a real result looks wrong on that mesh already has a worked answer sitting in an earlier post in this series, waiting to be applied rather than reinvented. The six-question test that decides whether any of this is actually ready to ship was never abstract, it’s the exact gate this real job has to clear. Skipping TP here was correct for this hardware specifically, not a universal rule, and a real, differently-configured MoE model on different NVLink bandwidth is the fastest way to see exactly where that generalization stops holding.