Deterministic resume, and the checkpoint interval formula that reproduces Llama 3's actual number
Why async checkpointing alone doesn't eliminate the stall, the physical reason NFS can't do fast sharded writes, the exact RNG state list a deterministic resume needs, and a derived formula that lands on the same 20-30 minute interval real frontier labs actually use.
Every mechanism in cluster operations, straggler detection, auto-cordon, turning a 3-hour outage into a 5-minute one, has one silent, load-bearing assumption underneath it: that resuming from a checkpoint gives back a correct, working model, indistinguishable from one that never failed. If that assumption is wrong, none of the recovery machinery above it matters. Auto-cordon can detect the bad node in two minutes and resize the communicator in thirty seconds, and if the checkpoint it just resumed from is silently corrupted or subtly non-deterministic, nothing has actually been recovered. The failure just gets found out later, more expensively, mixed in with days of subsequent training built on a bad foundation.
That’s why checkpoint recovery is its own discipline rather than the last line item of cluster ops. It has to satisfy four requirements simultaneously, each of which independently fails in production if skipped: async (training never stalls waiting for a write to finish), sharded (writing hundreds of gigabytes from one coordinator is a non-starter at scale), validated (a partially-written checkpoint is worse than no checkpoint, because it looks like it exists), and deterministic (resuming has to reproduce exactly what would have happened if the failure never occurred, not something reasonable, exactly that).
Why naive checkpointing fails before a single hardware failure even happens
A 70B-parameter model in BF16 is 140GB of weights alone, before optimizer state, which for Adam typically doubles the parameter memory with momentum and variance terms, so the actual state persisted per checkpoint is closer to 400 to 500GB total, not just the raw weights. The naive approach, stopping every GPU, gathering the complete model state onto rank 0, and writing the whole thing from a single coordinator, stalls the entire cluster for 90-plus seconds per checkpoint. At a 30-minute interval, that’s a fixed, recurring 5% goodput loss from checkpointing alone, before a single hardware failure has occurred. That’s the first problem, and it alone is enough to rule the naive approach out at frontier scale, because it’s a tax paid every time, regardless of whether anything ever actually breaks.
Async: overlapping the write, not eliminating it
The training loop should never wait for a checkpoint write to finish. Each rank hands its local state off to a background I/O thread and immediately continues to the next step, streaming to storage concurrently with ongoing computation. In PyTorch, torch.distributed.checkpoint.save() with a StorageWriter is the native sharded checkpoint API: each rank writes its own local shard, parameters, optimizer state, RNG state, to a separate file path, with a background thread handling the actual I/O while the main loop proceeds without blocking. The write path uses async I/O with an fsync on completion, the specific mechanism guaranteeing data is durably on disk rather than sitting in an OS write buffer a subsequent crash could lose.
Here’s the tradeoff worth being precise about: async checkpointing doesn’t eliminate the write cost, it overlaps it with useful compute, but the write still has to complete before the next checkpoint can start. If writes are slow relative to the checkpoint interval, a synchronous pause happens anyway. Checkpoint every 30 minutes with each write taking 4 minutes and there’s a real, quantifiable goodput loss stacking up even with async enabled, if the interval and write time aren’t tuned against each other. Async checkpointing is necessary but not sufficient; the interval itself, derived below, has to be chosen with the real write time in mind.
Sharded: why the filesystem underneath this matters as much as the code
A single-writer approach means one write path moving 140 to 500GB. Even with async I/O, one writer streaming that much data is slow, and it’s a single point of failure and a single bandwidth bottleneck. The actual production approach: each of, say, 64 ranks writes only its own local shard, roughly 2.2GB per rank for a 140GB model, and all 64 writes happen simultaneously, in parallel, on a filesystem built to support exactly this. Total wall-clock time for the write becomes roughly the time to write one 2.2GB shard, not the full 140GB sequentially, a direct 64x reduction from parallelism alone, not from writing less data.
This is where the storage layer covered earlier in this series becomes a hard prerequisite rather than an afterthought. This pattern requires a filesystem that genuinely supports many simultaneous writers without becoming a bottleneck itself. NFS cannot do this: it has a single metadata server, so even if the actual data writes could theoretically go in parallel, every one of those 64 ranks still negotiates through the same single metadata bottleneck, reintroducing exactly the serialization the sharded pattern was supposed to eliminate. Lustre, GPFS, and BeeGFS distribute metadata across many servers specifically so thousands of ranks can write simultaneously without contention. “We use Lustre” is a real engineering decision forced by physics, not a preference: the sharded-write pattern is architecturally impossible to get right on NFS no matter how well the checkpointing code is written, because the bottleneck sits one layer below the code, in the filesystem’s own metadata architecture.
Resharding matters the moment GPU count changes between save and load. torch.distributed.checkpoint supports a checkpoint saved with 64 ranks being correctly loaded by a job resuming with a different rank count, 48, or 80, whatever an auto-cordon-adjusted job now has. When a bad node gets cordoned out, the job resumes with fewer GPUs than it checkpointed with, and a format that hard-codes exactly 64 shards for exactly 64 ranks either fails outright or silently loads incorrectly at 63. Resharding support is what makes cordon-and-resume actually work in practice rather than being a clean mechanism on paper that breaks the moment rank count changes, and it’s the same underlying problem as reforming an NCCL communicator after a rank disappears, two halves of “the job just got smaller.”
Validated: a checkpoint that exists on disk might still be worthless
A crash, power failure, OOM kill, hardware fault, can happen mid-write, leaving a directory with files in it but not all of them, or not all of them complete. 47 of 64 shards written, then the crash. From a casual glance, a checkpoint exists, there’s a directory with files. It’s completely unusable, and worse than no checkpoint at all, because a system that doesn’t check will find that directory, assume it’s valid, attempt to resume, and fail in a way far more confusing to diagnose than “no checkpoint exists.”
The actual protocol: list the expected files, since the valid shard count is known upfront, verify every expected file exists, verify each file’s byte count matches the expected size, catching a truncated write that technically created a file but didn’t finish writing to it, then attempt an actual test load, the step that catches what the first two checks miss, a file that exists, has the right size, but contains bytes that don’t deserialize correctly, a bit-flip during write. Only after all three pass does the system mark the checkpoint valid. The ordering, existence, then byte count, then test load, is deliberately cheapest-check-first, the same discipline as picking the cheapest check that distinguishes between plausible explanations before committing to a fix: don’t reach for the most expensive verification step until the cheap ones have failed to rule the problem out.
Deterministic: the requirement that fails without ever producing an error
This is the hardest requirement to get right, precisely because getting it wrong doesn’t produce an error. It produces a model that trains, looks completely normal, and quietly ends up somewhere different than it would have if the failure never happened, discovered as a confusing eval regression days later with no obvious link back to a resume that happened three days prior.
Deterministic resume requires a complete list, not a best-effort one: Python’s own random module state, NumPy’s random state, PyTorch’s CPU RNG state, PyTorch’s CUDA RNG state per device, not just once, every GPU has its own, and missing even one device’s state means that GPU’s subsequent random operations, dropout masks among them, diverge from what they would have been, and the DataLoader’s exact iterator position, precisely which batch it had reached when the failure occurred. Omit any single one and the run continues, produces plausible-looking loss curves, and is nonetheless on a different trajectory, with no error message anywhere, because nothing has gone wrong in a sense any system could detect. The divergence is semantic, not mechanical.
If DataLoader position specifically is the piece that’s wrong, the model either sees repeated data, a real overfitting risk since those examples now get more total gradient updates than everything else in the corpus, or skips data entirely, a distribution shift, since some slice of the training corpus the model was supposed to see never gets trained on. Either way it’s invisible at the moment it happens, and shows up, if it shows up at all, as an eval regression roughly three days later, long enough that a checkpoint resume three days back is no longer the first place anyone thinks to look, because three days of subsequent training now sit on top of it. The list has to be exhaustive rather than “the ones that seem to matter most” precisely because there’s no way to know in advance which specific piece of state a given training step will need, dropout hits CUDA RNG, shuffling hits DataLoader position and possibly Python’s random, augmentation hits NumPy’s state. Skipping the ones that seem unlikely is exactly how a resume ends up deterministic 95% of the time and silently wrong the other 5%, arguably worse than being wrong consistently, because it’s far harder to catch in testing.
The interval question, derived, not quoted
Every checkpoint has a cost, the write itself, and every interval has a risk, compute lost if a failure lands before the next checkpoint. This has an actual derived answer.
Let interval be the time between checkpoints, save_time the duration of each write, MTBF the mean time between failures. Checkpointing every interval and paying save_time each time means the fraction of time spent checkpointing is save_time / interval, shrinking the interval raises this cost. If a failure can land anywhere within an interval with equal probability, on average half of that interval’s progress is lost when one hits, sometimes almost nothing if the failure lands right after a checkpoint, sometimes almost the whole interval if it lands right before the next one, averaging to half. At failure rate 1/MTBF, the expected fraction of time lost to failures is interval / (2 × MTBF), shrinking the interval lowers this cost, since less unsaved work is at risk at any moment.
Two costs moving in opposite directions as interval changes, exactly the shape that produces an interior minimum. Differentiating and setting to zero:
Solving:
Not a memorized formula, the direct result of minimizing checkpoint overhead plus expected failure loss.
Sanity check against Llama 3’s actual numbers. With an MTBF around 3 hours, consistent with 419 interruptions over 54 days, hours divided by 419 is close to 3 hours per interruption, and a save time around 2 minutes: minutes. That lands squarely in the 20 to 30 minute range actually observed in production. The formula isn’t a theoretical exercise disconnected from practice, running the real numbers through it reproduces the real interval a frontier lab used, genuine confirmation the model captures the right tradeoff rather than a coincidentally plausible range. It’s also actionable going forward: double checkpoint write speed, or double the hardware failure rate, and the formula says exactly how the interval should move, rather than leaving “how often should we checkpoint” as a guess re-made from scratch every time conditions change.
Rollback: why you keep three checkpoints, not one
The naive assumption is that validation makes a single kept checkpoint sufficient, if it’s ever bad, that’s what validation catches. But validation catches structural corruption, missing shards, wrong byte counts, a failed test load. It doesn’t catch a checkpoint that’s structurally perfectly valid but reflects a training run that’s gone subtly wrong, a bug that doesn’t become visible until two or more checkpoints after it was introduced. Keep only the single most recent checkpoint, and if that one is already downstream of the bug, there’s nowhere clean left to roll back to.
The actual production practice is keeping the last three. If checkpoint is where a problem becomes visible, in eval numbers, in loss curve shape, however it’s first noticed, and the actual root cause was introduced back at , having available is what lets you roll back to a point that’s actually clean, rather than being stuck between a most-recent checkpoint that validation can’t flag as bad and the earliest one still lying around. A cheap insurance policy, a few hundred extra gigabytes of storage, against a scenario that otherwise costs potentially days of retraining because every fallback option was already downstream of the actual problem.
Real production stakes
OPT-175B’s operational history: 35-plus manual restarts and 70-plus automatic restarts over two months of training, every one of those 105-plus restarts requiring a valid checkpoint to resume from, roughly every 12 to 13 hours on average across the full run. Checkpoint recovery wasn’t an occasional safety net invoked a handful of times, it was in continuous, routine use, more than once a day on average, for the entire run. And Llama 3’s greater-than-90% goodput despite 419 separate interruptions is described as achievable specifically because async checkpointing was in place, not a nice-to-have layered on top of an already-working system but a load-bearing requirement without which that target would have been unreachable given the observed failure rate.
Where this actually gets tested
“Walk me through why NFS can’t support fast sharded checkpointing, even with correctly written checkpointing code” tests whether the single-metadata-server bottleneck is understood as a physical, architectural limitation of the filesystem itself, not something code can route around. “Derive the optimal checkpoint interval given a 3-hour MTBF and a 2-minute save time” tests whether can actually be applied to land in the right range, versus only being able to cite “Llama 3 checkpointed every 20 to 30 minutes” as a memorized fact with no idea where it came from. “A checkpoint exists on disk, files are there, but resuming from it fails, what do you check and in what order” wants the validation protocol run backward as a diagnostic, existence, then byte counts, then a test load, cheapest first, and specifically knowing a populated directory is not sufficient evidence a checkpoint is usable. “Your model’s eval score regressed for no apparent reason three days after a routine checkpoint resume, where do you look” wants non-deterministic resume, specifically whether every RNG state and the DataLoader position were correctly restored, not a re-examination of the last three days of training code. “Why keep three checkpoints instead of just the most recent valid one” wants the specific reasoning, a bug can take two-plus checkpoints to become visible, not a vague “just in case.”
What this takes to be frontier-job-ready
The technical axis is holding all four requirements as genuinely independent failure modes rather than one blob called “checkpointing works”: async, sharding, validation, and determinism each fail differently, and a system that gets three of the four right still has a real, undetected gap.
The operational axis is treating checkpoint validation as a continuously running safety system rather than a one-off check performed manually after something already looks wrong. One thing worth being honest about: a specific quote describing a dedicated test-environment training-pipeline validation system gets attributed to an Anthropic ML Systems role in some accounts of this work, and I couldn’t independently confirm that exact framing or role title against Anthropic’s own posted job descriptions, so treat the underlying practice, continuous automated validation rather than manual spot-checks, as the well-supported part, and the specific attribution as unconfirmed rather than settled.
The autonomy axis shows up in the interval-tuning judgment call: the derived formula tells you where the optimum sits given your current save time and failure rate, but deciding when those inputs have changed enough to warrant re-tuning the interval, rather than leaving it at whatever was set once, is exactly the kind of ongoing, self-directed maintenance this series has repeatedly pointed at as the real differentiator.
Common mistakes
Treating async checkpointing as eliminating the write cost rather than overlapping it: if write time exceeds what the chosen interval can absorb, a synchronous stall happens anyway, async alone doesn’t guarantee it away.
Assuming any parallel filesystem handles sharded writes equally well: NFS’s single metadata server is a specific, physical bottleneck that no amount of well-written checkpointing code works around.
Checking only that checkpoint files exist rather than running the full validation protocol: a directory full of files can still be a completely unusable checkpoint, and it’s a more confusing failure to debug than a missing one.
Restoring model weights and optimizer state but treating RNG and DataLoader position as optional extras: skipping any single piece of the deterministic-resume list produces a run that looks fine and silently diverges, with the eventual eval regression showing up days later and pointing nowhere obvious.
Keeping only the most recent checkpoint because validation exists: validation catches structural corruption, not a structurally valid checkpoint sitting downstream of a subtly wrong training run, which is exactly why the last three, not one, get kept.
Try it yourself
Beginner. Using the derived formula, compute the optimal checkpoint interval for a save time of 90 seconds and an MTBF of 4 hours, and compare it against the Llama 3 sanity check above.
Intermediate. Given a checkpoint interval of 20 minutes and a save time that has crept up to 5 minutes due to a growing model size, compute the actual goodput loss from checkpointing overhead alone, and determine whether the interval should be adjusted using the derived formula.
Advanced. Design the specific fields a deterministic-resume checkpoint needs to store for RNG state (Python, NumPy, PyTorch CPU, PyTorch CUDA per device) and DataLoader position, and describe a test that would actually verify determinism end to end, comparing a run that never failed against a run that failed and resumed at the identical step, checking for bit-identical outputs rather than just plausible-looking ones.
The one-sentence version: a checkpoint has to be async so training never stalls waiting on it, sharded because writing hundreds of gigabytes from one coordinator is architecturally impossible to make fast on the wrong filesystem, validated because a directory full of files is not the same claim as a usable checkpoint, and deterministic down to every last RNG state and DataLoader position, because the one failure mode in this entire pipeline that never throws an error is also the one that costs you three days of training built on a foundation that was already wrong.