Reproducibility on a GPU has two different meanings that get conflated constantly, forcing PyTorch's actually-deterministic mode costs a real, measured 2 to 5x slowdown, and Chinchilla's own paper names the exact way an ablation quietly becomes invalid

Every technique this series has derived so far, from Chinchilla’s compute-optimal ratio to DeepSeek-V3’s parallelism mesh to the choice between DPO and GRPO, was arrived at by comparing experiments against each other. None of that comparison is trustworthy for free. Two separate, easy-to-conflate problems sit underneath it: whether a single run is even reproducible on the hardware you ran it on, and whether two different runs actually isolated the one variable you meant to test. Both fail silently. A non-deterministic run doesn’t crash, it just gives you a slightly different number next time and no signal about why. A confounded ablation doesn’t crash either, it gives you a clean, confident, wrong answer about which change caused which effect. This is the discipline underneath every number cited elsewhere in this series, and it’s mundane enough that most treatments skip it entirely, right up until it’s the reason a real result doesn’t replicate.

Two different guarantees hiding under one word#

“Reproducible” gets used for two genuinely different claims, and mixing them up wastes real debugging time. The weak guarantee: run the same code, same seed, same data, and get the same final loss and eval numbers, useful for regression testing, “did this refactor silently change behavior,” and CI. The strong guarantee: every intermediate tensor, every step, is bitwise identical between two runs, needed when you’re bisecting exactly which step or which line introduced a divergence, a NaN, or a numerics regression after a dependency upgrade, the exact scenario this series already flagged for a FlashAttention version bump. The weak guarantee is nearly free. The strong guarantee has a real, specific, and non-trivial cost, and knowing which one you actually need before you pay for it is the first real decision here.

The real cost of true determinism#

GPU training is non-deterministic by default for a specific, nameable reason, not general floating-point fuzziness: several common operations use atomicAdd-based reduction kernels, whose parallel execution order isn’t fixed, and floating-point addition isn’t associative, so summing the same values in a different order produces a slightly different result. PyTorch documents the affected operators directly: scatter_add_, index_add_, bincount in the forward pass, and the backward kernels for embedding_bag, ctc_loss, interpolate, and several pooling and padding operations. None of these are exotic, ordinary transformer training touches several of them.

torch.use_deterministic_algorithms(True) forces PyTorch to use deterministic implementations everywhere one exists, and throws a runtime error outright if an operation you’re using has no deterministic version at all, which is itself useful information, it tells you exactly where your pipeline depends on non-deterministic ops. The real, measured cost of turning this on: throughput typically drops by a factor of two to five. That’s not a rounding cost, it’s the difference between a training run finishing in two weeks or finishing in up to two months, which is exactly why nobody runs full-scale pretraining with it enabled by default. The correct, real practice is narrower and cheaper: enable strict determinism only when actively bisecting a specific reproducibility bug, on a small model and a short run, then turn it back off. Paying a 2 to 5x tax on every training run to get a guarantee you need for maybe one debugging session a quarter is solving the wrong problem at the wrong scope.

What a real ablation actually isolates, and the exact way it silently doesn’t#

Chinchilla’s own isoFLOP methodology, already derived in full in this series, is worth returning to specifically as the canonical example of what a rigorous, expensive-to-run ablation looks like in production: hold compute fixed, vary the size-versus-data tradeoff across a family of small, cheap proxy models, fit a power law, extrapolate to the real budget. That post already names the exact failure mode that invalidates it, worth restating here because it’s the general case, not a Chinchilla-specific quirk: a proxy model trained on a different data mix or architecture family than the actual target run makes the extrapolation invalid, because you fit a curve on one distribution and are using it to predict outcomes on a different one. Stated abstractly this is obvious. In practice it’s the single most common way a real ablation quietly produces a confidently wrong answer: someone runs the cheap proxy sweep against a convenient, slightly stale data snapshot, or a data source that’s about to be added but isn’t in yet, changing two things (the variable under test, and the underlying data) while believing they changed one.

The general rule this generalizes to: a valid comparison changes exactly one thing between the control and the treatment, and everything else, code version, data snapshot, hardware, library versions, has to be pinned identically or the result is confounded. This sounds trivial stated as a rule and is genuinely hard to guarantee in practice, because “everything else” includes things that don’t feel like variables until they silently differ, a data loader’s cache serving a slightly older snapshot, a library patch version that changed a kernel’s numerics, a config file edited after the “final” run was already kicked off and logged.

One seed is a sample size of one#

Henderson et al.’s 2018 “Deep Reinforcement Learning that Matters” is the real, citable evidence for a fact that’s easy to state and easy to ignore in practice: two runs of the identical algorithm, identical code, differing only in random seed, can produce statistically significantly different final performance. The paper’s broader point, backed by systematic comparison across common RL benchmarks, is that a substantial fraction of claimed algorithmic improvements in published results don’t clear the bar of seed-to-seed variance in the baseline itself, meaning the reported “improvement” could just as easily be noise from which seed happened to get reported. The discipline this implies is unglamorous but non-negotiable at real scale: run a config across multiple seeds, report a mean and a spread, not a single best-looking run, and treat a comparison between two techniques evaluated at one seed each as exactly as strong as it is, which is not very. This is expensive, seeds multiply your compute bill directly, which is precisely why cutting this corner is tempting and precisely why it’s the corner that produces false conclusions when cut.

The system of record: what tracking infrastructure is actually for#

Tools in this space, Weights & Biases is the most widely adopted, with MLflow as a common open-source alternative, exist to solve a specific, unglamorous problem: automatically capturing the full context a result needs to mean anything later, hyperparameters, exact code version (git commit), data version, and the resulting metrics, together, at the moment a run actually executes, not reconstructed from memory afterward. Reported adoption for W&B specifically includes production use at OpenAI, Meta, NVIDIA, and Microsoft among others, which is less interesting as a brand-name list than as confirmation that this is treated as production infrastructure at exactly the labs this series has been citing throughout, not an optional nicety for smaller teams.

The real, recurring failure mode this tooling exists to prevent is config drift: someone kicks off a logged run, then edits the launch script or a config value before the next run without updating what got recorded, and the tracked metadata for that first run now silently lies about what code actually produced its results. Nothing errors. The dashboard shows a clean, plausible-looking config next to a real metric, and the two have quietly stopped corresponding to each other. The fix isn’t more discipline, it’s removing the human step: log the config automatically from the actual running process’s resolved state, including the exact git commit hash, at the moment training starts, not from a file someone remembers to update.

Common mistakes#

Enabling torch.use_deterministic_algorithms(True) for a full production training run “to be safe”: the real, measured 2 to 5x throughput cost is a bad trade for a guarantee you only need during targeted debugging, not as a permanent default.

Running an ablation’s proxy experiments on a conveniently available but different data snapshot than the target run will use: Chinchilla’s own paper names this exact failure as what invalidates an isoFLOP extrapolation, and it’s a general risk for any ablation, not specific to scaling laws.

Reporting a single seed’s result as if it were the algorithm’s performance: Henderson et al.’s real finding is that seed-to-seed variance alone can be statistically significant, meaning a one-seed comparison between two techniques hasn’t actually established which one is better yet.

Try it yourself#

Beginner. List three PyTorch operations that use atomicAdd-based kernels and are therefore non-deterministic by default, and explain in one sentence why floating-point addition’s lack of associativity, not a bug, is the root cause.

Intermediate. You’re comparing two training runs to evaluate a new data-mixing ratio. Run A uses the old mix on last week’s code checkpoint; Run B uses the new mix on this week’s checkpoint, which also includes an unrelated bug fix in the data loader. Identify the confound, and describe the minimal additional run needed to isolate the mixing ratio’s real effect.

Advanced. Design a seed-variance study for a training configuration you believe improves over a baseline: how many seeds per configuration would you run, what statistic would you report beyond the mean, and using Henderson et al.’s finding as your prior, what effect size would you consider large enough to trust without further replication?

What this takes to be frontier-job-ready#

OpenAI’s own listing for Research Engineer/Scientist, RL/Reasoning states the temperament this entire post has been building toward almost word for word: “You value principled approaches, simple experiments in tightly-controlled settings, and reaching trustworthy conclusions which stand the test of time.” That’s not soft-skill language, it’s a direct description of running one-variable-at-a-time ablations, tracking them so they’re reconstructable later, and treating a single seed’s result with appropriate suspicion, exactly the discipline this post derives concretely rather than gestures at.

The operational reality is the 2-to-5x determinism cost and the multi-seed compute bill together: neither is optional at the point a result needs to ship or a paper needs to hold up, and both are the kind of real, disclosed cost a team has to budget for explicitly rather than discover after committing to a launch date built on a single lucky run.

---

The one-sentence version: “reproducible” quietly means two different things, same-final-answer and bitwise-identical-every-step, with the second costing a real, measured 2 to 5x slowdown that’s only worth paying during targeted debugging; a valid ablation changes exactly one variable while pinning everything else, the exact discipline Chinchilla’s own paper names the failure mode for when it’s skipped; and Henderson et al.’s real finding, that random seed alone produces statistically significant performance differences, is the concrete reason a single run was never a result, it was a sample size of one. Every number this series has cited from a real lab, Chinchilla’s fitted constants, DeepSeek-V3’s disclosed GPU-hours, Llama 3’s proxy-experiment schedule, exists because a team somewhere paid this exact tax before publishing it.