Chinchilla, compute-optimal training, and why GPT-3 was trained wrong

Scaling laws are the difference between “we’re spending $50M on this training run because it feels right" and "we're spending$50M because we ran $50K of experiments and have a fitted equation predicting the outcome.” Before this existed as a rigorous practice, models were trained by intuition: pick a size that feels ambitious, pick a token count that feels sufficient, spend the compute, see what happens. GPT-3 is the textbook example of what that produces. 175 billion parameters, 300 billion tokens, and roughly 4x undertrained. The same compute budget, spent instead on a roughly 35B-parameter model with 1.4 trillion tokens, would have outperformed it. That’s not a marginal inefficiency. A frontier lab spent tens of millions of dollars on compute and left a better model on the table because nobody had yet worked out the equation telling them the correct ratio.

The equation, derived, not quoted#

Pretraining loss follows a power law in two variables, model size $N$ and dataset size $D$:

$L(N, D) \approx \frac{A}{N^\alpha} + \frac{B}{D^\beta} + L_\infty$

Each term is worth reading for what it represents, because the shape of this equation is the whole story. $A/N^\alpha$ is the “you didn’t have enough parameters” term, shrinking toward zero as $N$ grows, capturing loss paid for a model too small to represent what it needs to. $B/D^\beta$ is the “you didn’t see enough data” term, shrinking as $D$ grows, capturing loss paid for a model that hasn’t seen enough tokens to have learned what it’s capable of learning. $L_\infty$ is the irreducible floor: even with infinite parameters and infinite data, loss doesn’t reach zero, because language itself has an entropy floor, multiple valid next tokens exist for almost any context. “Just scale infinitely” was never going to produce a perfect model. It asymptotes toward a floor, not toward zero.

The engineering insight buried in the equation: loss is being paid down from two independent, competing directions, and with a fixed compute budget, growing $N$ and growing $D$ trade off against each other, a bigger model costs more per token processed, more tokens cost more compute regardless of model size. The entire compute-optimal question is: for a fixed compute budget $C$, what split between $N$ and $D$ minimizes $L$?

Compute-optimal allocation, actually worked through#

Total training compute for a dense transformer is well-approximated as $C \approx 6ND$, 6 FLOPs per parameter per token, 2 for the forward pass, 4 for the backward pass, standard convention across the field. This is what links three otherwise-independent numbers, model size, data seen, and compute spent, the moment a budget is fixed.

Minimizing $L(N,D)$ subject to $6ND = C$ is a constrained optimization problem: substitute $D = C/(6N)$ into the loss equation, differentiate with respect to $N$, set it to zero, and solve. Worth doing by hand once rather than memorizing the result, it’s a straightforward Lagrangian. It lands on:

$N^* = G \cdot (C/6)^{0.5}, \qquad D^* = G^{-1} \cdot (C/6)^{0.5}$

where $G = \sqrt{A\alpha / (B\beta)}$, a constant derived from the fitted $A, \alpha, B, \beta$ values. The result worth being able to say cold: compute-optimal training has $N$ scaling as $C^{0.5}$ and $D$ scaling as $C^{0.5}$, both at the same rate. That symmetry is the actual Chinchilla finding, and it’s exactly what GPT-3-style training got wrong: earlier practice scaled $N$ much faster than $D$ as compute grew, which is precisely why GPT-3 ended up roughly 4x undertrained relative to the compute-optimal point.

The isoFLOP sweep: what this actually looks like as a job#

Nobody derives $A$, $\alpha$, $B$, $\beta$ from theory. They come from a specific, repeatable experimental design, and understanding this procedure is the difference between someone who can quote “N scales as the square root of compute” and someone who could actually run the experiment that produces that constant for a new architecture or data mix, which is most of what this work actually is day to day.

Fix a compute budget, say $10^{21}$ FLOPs. Vary the $(N,D)$ pair while holding $6ND \approx C$, training a family of models that all cost the same compute but trade off size against data differently, a big model on fewer tokens, a small model on many tokens, and points between. Train each to convergence at its target $D$, and plot final loss against $N$. The result is a U-shaped curve, models too small underfit, models too big and undertrained for their size also underperform, and there’s a minimum somewhere in the middle, the compute-optimal $(N,D)$ pair for that budget. Repeat the entire sweep at several different compute budgets, $10^{19}$, $10^{20}$, $10^{21}$, $10^{22}$. Fit $\log(L) = -\alpha\log(N) + \text{const}$ across the minima found at each budget, that’s where $\alpha$ actually comes from, and the equivalent procedure for $D$ gives $\beta$. Extrapolate the fitted equation to the actual target budget, $10^{24}$ FLOPs, say, and read off the predicted optimal $N$ and $D$, and the loss to expect.

That’s the whole game: spend maybe $50K running a grid of small, cheap models, fit the power law, and use the fitted curve to predict what a$50M run will produce before spending the $50M. That’s the literal mechanism by which training a frontier model went from something closer to alchemy to something that’s actually engineering, a falsifiable prediction checked against the real run afterward.

The failure mode that catches people who’ve only read the paper, not run the experiment. A 1B-parameter proxy model has to use the same data mix and architecture family as the actual target model, or the extrapolation is simply invalid, you fit a power law on one distribution and are trying to predict outcomes on a different one. Stated plainly this sounds obvious, but it’s the single most common way isoFLOP sweeps quietly produce wrong predictions: someone runs the proxy sweep on a convenient, slightly different data mix, an older snapshot, a missing data source that’s about to be added, and the extrapolated compute-optimal point is confidently wrong for the actual target run.

Chinchilla answers half the question a production lab actually has#

Chinchilla-optimal is a training-time answer. It says nothing about inference cost, because inference cost scales with $N$, you pay for every parameter every single time someone queries the model, while training cost is a one-time expenditure. If a model is going to serve hundreds of millions of users over its production lifetime, the true total cost, training plus every inference call across the model’s deployed lifespan, is minimized by a smaller model trained on more tokens than pure Chinchilla-optimal suggests, willingly accepting a slightly suboptimal training-compute tradeoff in exchange for a permanently cheaper model to run. This is exactly why smaller, more heavily trained models are so often the right production call even when a bigger model would hit a lower training loss for the same training compute: training compute is spent once, inference compute is spent every call for the rest of the model’s life. Anyone doing pretraining scaling work at a frontier lab runs this inference-adjusted version of the calculation, not the textbook Chinchilla-optimal one, because the textbook version was only ever answering half the actual cost question.

Two failure modes that break the clean story#

Scaling laws predict smooth, continuous improvement in loss as $N$ and $D$ grow. Certain capabilities, multi-step reasoning, chain-of-thought, have been observed to appear discontinuously at specific scale thresholds rather than gradually. A smooth power-law fit on loss doesn’t capture a phase transition in capability, because loss and capability aren’t the same measurement, a model’s aggregate next-token loss can be improving smoothly while a specific downstream skill it needs, correctly chaining three reasoning steps, either isn’t there yet or suddenly is. Watching only the loss curve can mean missing the exact moment a capability turns on, because the loss curve was never designed to tell you that.

The constants are not universal, and using the wrong ones is a real production mistake, not a hypothetical one. $A$, $\alpha$, $B$, $\beta$ are fitted per data distribution and per architecture family. Code, math, multilingual text, and general English web text each produce different fitted constants. Applying English-web-text-derived scaling constants to plan a run whose data mix is heavily code or heavily multilingual gives a wrong compute allocation, under- or over-investing in model size relative to data volume, because the extrapolation came from a curve never fit on the actual training distribution. A commonly cited entry exercise for getting into this work is exactly this: derive Chinchilla laws for a given setting, then see how they differ for dense versus MoE architectures. The point isn’t reproducing the original Chinchilla numbers, it’s internalizing that the procedure generalizes but the constants absolutely do not, and they have to be refit for the actual setting every time.

MoE: where scaling laws stop being a training-only story#

Everything above assumes a dense model, where FLOPs scale directly with parameter count. Mixture of Experts breaks that assumption in a way that’s become one of the central architectural bets in the field, specifically because of what it does to the scaling-law cost curve.

In a dense model, FLOPs are proportional to parameters, every parameter gets used on every token, so parameter count and inference cost are effectively the same number. In an MoE model, only $k$ experts out of $N$ total activate per token, so FLOPs are proportional to active parameters, that’s $k/N$ of the total, not total parameters. DeepSeek-V3 is the concrete, public example: 671 billion total parameters, but only 37 billion active per token, 8 of 257 experts routed per token. You get the representational capacity of a 671B-parameter model while paying the inference cost of a 37B-parameter model.

This is the architectural bet the field is increasingly making to escape the scaling-law cost curve. Chinchilla-style scaling laws, derived for dense models, say better quality costs proportionally more compute, both to train and to run. MoE structurally decouples those two things: total parameter count, and therefore capacity, can scale up by 10x without scaling per-token inference FLOPs anywhere near 10x. It doesn’t repeal the scaling law, it changes which variable, total params or active params, the law’s $N$ term should actually refer to, and getting that distinction wrong is exactly the kind of mistake the dense-versus-MoE exercise above is designed to catch.

What a failed run actually costs, and why the math is the operations#

A failed 54-day training run wastes upward of $10M in compute, a run of roughly the exact length and scale Llama 3 405B’s actual training took, 16,384 H100 GPUs over 54 days. That number explains why every decision on a real pretraining run, data mix, learning rate schedule, architecture choice, parallelism layout, gets validated with cheap proxy experiments first, using exactly the isoFLOP-sweep-and-extrapolate methodology above, before committing to the full-scale run. Scaling laws aren’t run once at a project’s start and forgotten. They’re the standing methodology by which every major decision on a run gets pressure-tested against a multi-million-dollar downside before it’s locked in.

This is also, concretely, why the people who fit these power laws are not separated from the people on call when a production run breaks. Anthropic’s own Pretraining Scaling role states this directly, and it’s worth reading in its actual words rather than a paraphrase: the work is highly operational, engineers stay deeply involved in keeping production models training smoothly, and during launches the team often works extended hours and may need to respond to issues on evenings and weekends. Separately, the same job posting names exactly this kind of proxy-experiment work as a literal example of the role’s scope, optimizing pretraining data mixes and comparing the compute efficiency of different datasets. The math and the operations are the same job, not two adjacent ones.

Where this actually gets tested#

“Why was GPT-3 undertrained, and what would compute-optimal have looked like for the same compute budget” tests whether the actual finding is known, roughly 35B parameters and 1.4T tokens would have beaten 175B parameters and 300B tokens at the same compute, or just the name “Chinchilla.” “Walk me through how you’d design the experiment to fit a scaling law for a new data mix” is the isoFLOP sweep procedure above; answering “I’d look up the Chinchilla constants” is the wrong answer, the entire point is that constants get refit per distribution, never looked up. “Your isoFLOP sweep says one thing, but the full-scale run comes in noticeably worse than predicted, where do you look first” points at proxy-model validity, did the small proxy models actually share architecture and data mix with the target run, or did something drift between the cheap experiment and the expensive one. “How does compute-optimal change once you account for inference cost across the model’s deployed lifetime” is the inference-adjusted question above, and it’s the one separating someone who’s read the Chinchilla paper from someone who’s actually planned a model that has to serve users for years after training ends.

What this takes to be frontier-job-ready#

The technical axis is the isoFLOP methodology itself, treated as something you can actually execute, not just cite: design a sweep, fit the power law, and know precisely what would make the extrapolation invalid before trusting it.

The operational axis is stated directly in the same job posting quoted above: this work is highly operational, and the team that fits these curves is the same team that responds to production incidents during launches, not a research function sitting apart from the operational side of running training.

The autonomy axis shows up in the non-universal-constants failure mode. Refitting scaling constants for a new architecture or data mix, rather than reaching for published numbers that were never fit on your actual distribution, is a judgment call with no textbook answer, and it’s precisely the kind of ambiguous, self-directed work this series has repeatedly pointed at as the real differentiator.

Common mistakes#

Treating Chinchilla’s $N \propto C^{0.5}$, $D \propto C^{0.5}$ result as a universal constant rather than the output of a specific fit: the exponents and coefficients are refit per architecture and data distribution, and reusing someone else’s fitted values on a different setting is a real, common source of a wrong compute allocation.

Optimizing purely for training loss when the model will actually be served: training-optimal and inference-adjusted-optimal are different points on the curve, and ignoring the difference means overpaying for inference for the entire deployed lifetime of the model to save training compute spent exactly once.

Watching the loss curve as the sole signal for whether a capability has emerged: loss and downstream capability are different measurements, and a capability can appear discontinuously while the loss curve keeps improving smoothly through the transition.

Applying a dense-model scaling law directly to an MoE model without adjusting which parameter count the law’s $N$ refers to: total parameters and active parameters answer different questions, and conflating them misprices exactly the tradeoff MoE was adopted to exploit.

Try it yourself#

Beginner. Using $C \approx 6ND$, compute the training compute for GPT-3’s actual configuration (175B params, 300B tokens) and for the compute-optimal alternative (roughly 35B params, 1.4T tokens), and confirm both consume approximately the same compute budget.

Intermediate. Sketch an isoFLOP sweep design for a fixed budget of $10^{20}$ FLOPs: choose five $(N,D)$ pairs satisfying $6ND \approx 10^{20}$, spanning from a large model on few tokens to a small model on many tokens, and describe what shape you’d expect the resulting loss-versus-$N$ curve to take.

Advanced. For a DeepSeek-V3-style MoE model with 671B total parameters and 37B active parameters, work out how the $C \approx 6ND$ compute formula needs to change, and explain why a scaling law fit using total parameters instead of active parameters would misestimate the model’s actual training and inference compute costs.

---

The one-sentence version: a $50K isoFLOP sweep exists to predict what a$50M training run will produce before you spend the $50M, GPT-3’s 4x-undertrained result is the concrete proof of what happens without one, compute-optimal training scales model size and data size at the identical square-root-of-compute rate, that answer changes once inference cost across a model’s deployed lifetime enters the accounting, and MoE doesn’t repeal the law, it just changes which parameter count the law was ever actually about. None of that math means anything if what you fed the model in the first place was wrong.