OPT-175B's real training logbook has 40+ restarts and a cloud provider that deleted the whole cluster, and it's where half this series' practices actually came from

DeepSeek-V3 and Mixtral 8x22B are both mature-tooling case studies, real meshes, real launch commands, real measured MFU, produced by teams with FSDP2, TorchTitan, and Megatron-Core already built and battle-tested underneath them. OPT-175B is the opposite kind of real: Meta’s metaseq repository contains the actual, publicly released operational logbook, written in real time, from a 2021 training run where a meaningful fraction of this series’ own production practices didn’t exist yet as settled discipline, they were being invented, under real pressure, in response to exactly the failures documented in that logbook. Reading it isn’t nostalgia. It’s the closest thing available to watching the discipline this series has been building get forged.

The real, minimal mesh, and why it’s minimal#

OPT-175B trained on 992 A100 GPUs using tensor parallelism at 8 and FSDP for the remaining data-parallel width, $992/8=124$ FSDP-sharded groups. No pipeline parallelism, no context parallelism, no expert parallelism, and worth being precise about why each is absent rather than assuming an oversight: it’s a dense 175B-parameter model, so there’s no router and nothing for EP to shard; the training context length was modest by this series’ later standards, so CP’s entire memory problem never came up; and FSDP alone, at 124-way sharding, was already sufficient to fit the model without needing PP’s added depth-splitting complexity on top. TP=8 matching the 8-GPU node is the identical NVLink-domain reasoning already derived for Llama 3 and DeepSeek-V3, holding true across three real systems built years apart on different hardware generations. What’s genuinely notable historically: this was one of the first large-scale production proofs that FSDP, the exact lifecycle this series built out in full, worked at all. The logbook is partly the story of finding that out in real time.

The numerical-instability lineage, and where a real anecdote already in this series actually comes from#

The 8-separate-learning-rate-adjustment, skip-the-bad-batches pattern already cited earlier in this series is a compressed summary of a much longer, much messier real sequence, worth unpacking properly here rather than leaving it as a one-line anecdote. Starting from experiment 11.4, the team hit “grad norm explosions / loss explosions / nans after a couple hundred updates after each restart,” with loss scale collapsing so far it started massively underflowing. What actually happened next wasn’t one fix, it was a documented sequence of real, individually-motivated interventions: learning rate reduced multiple times, gradient-clipping threshold tightened from 2.5 down through 1.5 to 1.0 on the L2 gradient norm, weight decay raised from 0.05 to 0.1, Adam’s beta2 lowered from 0.98 to 0.95, and, by experiment 11.10, ReLU swapped for GeLU alongside a more numerically stable multi-head-attention calculation. Buried in the same lineage: the learned positional embeddings, initialized with a sinusoidal scheme, had effectively frozen in place, a real, subtle architecture-initialization bug with nothing to do with the optimizer at all, sitting alongside genuine optimizer instability and easy to misattribute to the wrong cause if you weren’t tracking both independently.

By experiment 12.x, the team pivoted outright: adopted Megatron and GPT-3’s own settings directly, removed Normformer, switched to standard embedding initialization. Later, at run 12.43, activation and gradient norms started drifting again, learning rate cut by 10%; at 12.45.2, a large grad-norm spike pushed the learning rate down to two-thirds of what GPT-3 itself used. The gradient-clipping mechanism and the loss-scale-hits-minimum-threshold detection this series already derived are the exact instruments that made each of these interventions detectable at all, not abstract theory applied after the fact, the actual, contemporaneous tools this team was watching in real time to know an intervention was needed.

The infrastructure chaos, and the checkpoint discipline it justifies#

The logbook counts 40-plus restarts in the roughly two-and-a-half weeks between the 10% and 27% updates alone, caused by hardware failures compounding with an inability to provision enough buffer nodes to swap failures out fast enough. The specific failure modes read like a direct, real-world instance of this series’ own fault taxonomy: ECC memory errors, NCCL and InfiniBand connectivity failures, missing GPUs on instances, training hanging nondeterministically after a new node joined, CUDA errors, partial checkpoint uploads after a hardware error. One detail worth sitting with specifically: node replacement through the cloud provider’s interface routinely took hours and often returned the exact same defective hardware, a real, absurd, and completely undocumented-in-any-textbook failure mode that no amount of correct training code could have prevented.

The checkpoint problem, concretely. At 1.6TB per checkpoint, split across 992 files of roughly 1.7GB each, downloads from blob storage would “hang nondeterministically.” This is the exact, real-world justification for the checkpoint-validation protocol this series derived in the abstract, existence, then byte count, then an actual test load, because a checkpoint that’s supposed to exist and silently doesn’t finish transferring is precisely the failure mode “did the file get created” alone cannot catch. A team without that three-step discipline in place, resuming from a checkpoint they assumed was fine because a file with the right name showed up, is exactly the team that discovers the problem hours into a resumed run instead of in the seconds a test load would have taken.

The worst single incident: on December 21, the cloud provider accidentally deleted the entire cluster while replenishing the buffer node pool. Recovery was fast, but 16 of the restored machines failed the team’s own infrastructure checks and had to be excluded; the buffer pool wasn’t fully replenished again until December 23. A five-host outage on December 6, and roughly two machines failing per day as ordinary background attrition, round out the picture: at this scale, in 2021, hardware failure wasn’t an edge case anyone was defending against reactively, it was the baseline condition the entire operation had to be built to survive, the same “ambient operating condition, not an edge case” framing this series has stated directly, reported here from inside the incident rather than as a retrospective lesson.

The bug nobody could explain, and why that’s the actually important part#

Somewhere in the run, an upgrade from Megatron 2.4 to 2.6, adopted for reasons unrelated to numerics, silently changed the model’s activation-norm trajectory while also improving throughput by 2%. The team traced the effect to one line: torch._C._jit_set_profiling_executor(False), a JIT compiler setting, and the logbook is honest that they never fully understood the mechanism behind why flipping it changed training dynamics at all. This is worth taking seriously as the single most transferable lesson in the whole logbook, more than any specific hyperparameter value: a dependency upgrade adopted for an unrelated reason can silently change numerics, with no error, no warning, and a real, measurable throughput improvement that makes the change look purely beneficial. This is the identical risk already flagged for a FlashAttention version bump elsewhere in this series, a core-math dependency changing behavior across a version boundary, and OPT-175B’s logbook is the real, dated, primary-source proof that the risk isn’t hypothetical or specific to attention kernels, it’s a general property of any dependency sitting underneath the actual math.

What the modern version of each real problem looks like#

Every failure mode documented above has a direct, named answer built earlier in this series, worth stating explicitly rather than leaving the connection implicit. Forty-plus restarts from slow, manual hardware triage is what auto-cordon’s drain-wait-checkpoint-resize-resume sequence exists to compress into minutes. A cloud provider silently returning the same defective hardware on “replacement” is exactly what GPU burn-in and health-check validation before a node re-enters the pool is built to catch. A 1.6TB checkpoint hanging nondeterministically mid-download is what the three-step existence/byte-count/test-load validation protocol exists specifically to never trust blindly. And a silent numerics change from an unrelated dependency bump is what treating any core-math library upgrade as a token-by-token comparison problem, not a green-loss-curve one, exists to catch before it ships. None of this is a coincidence: the practices in this series exist because problems exactly like these, documented in public, in real time, by a real team, made them necessary.

Common mistakes#

Treating “40+ restarts” as evidence of a badly run project rather than the documented, expected condition of frontier-scale training in 2021: the same order-of-magnitude interruption rate shows up in Llama 3’s later, better-tooled run, and the actual measure of success was never zero interruptions, it was surviving them.

Attributing every instability to the optimizer: the frozen positional-embedding bug sitting inside the same 11.x lineage as genuine gradient instability is a real example of two unrelated root causes producing overlapping symptoms, and conflating them would have wasted real debugging time on the wrong half of the problem.

Assuming a beneficial-looking change (2% faster throughput) needs no further scrutiny: the Megatron 2.4-to-2.6 upgrade improved throughput and silently changed numerics in the same commit, and only one of those two effects was the one anyone was looking for.

Try it yourself#

Beginner. Using the real 992-GPU, TP=8 figures, compute the FSDP-sharded data-parallel width by hand, and explain in one sentence why this run needed no pipeline parallelism at a scale where a later, larger model like DeepSeek-V3 needed PP=16.

Intermediate. Given the documented gradient-clipping progression (2.5 → 1.5 → 1.0) and Adam beta2 progression (0.98 → 0.95), explain what each individual change was doing to training dynamics, and why tightening both at once, rather than one at a time, makes it harder to know afterward which change actually fixed the instability.

Advanced. Design the specific check that would have caught the Megatron 2.4-to-2.6 numerics change before it trained on real data for weeks: what would you diff, at what granularity, and how would you distinguish “acceptable numerical noise from a minor version bump” from “a real behavioral change” without re-running the entire training job twice.

What this takes to be frontier-job-ready#

The technical axis: recognizing that a two-technique mesh, TP and FSDP only, was the objectively correct choice for this model and era, not a limitation to apologize for, the same discipline DeepSeek-V3’s real decision to skip TP entirely already established in this series, applied at the opposite end of the technique-count spectrum.

The operational axis is this entire post: a public, dated, warts-and-all engineering logbook is about as close as this series can get to raw, unfiltered evidence of what the day-to-day reality behind “operational excellence” and “staying calm under a stalled job” actually looks like, not the polished summary either a technical report or a blog post about the run tends to produce afterward.

The autonomy axis: the Megatron version-bump incident is a real example of a team that noticed an anomaly nobody asked them to look for, a throughput number that improved for a reason they didn’t yet understand, and investigated it anyway rather than accepting a free win without asking why.

---

The one-sentence version: OPT-175B’s real logbook is two techniques, TP and FSDP, genuinely sufficient for what the model needed, running into forty-plus restarts, a deleted cluster, a documented numerical-instability lineage, and a silent one-line dependency change nobody could fully explain, and every one of those real, dated, publicly available incidents is the actual origin of a specific practice this series has spent this whole arc building out in full. Three real systems, three different technique counts, one shared discipline running underneath all of them: use exactly what the model and the hardware force, instrument everything, and never trust a result, or a checkpoint, or a beneficial-looking dependency bump, until you’ve actually checked it.