Loss scaling, FP8, and the failure that never crashes

The bit layouts for FP32, BF16, FP16, and FP8 were covered precisely earlier in this series: sign, exponent, mantissa, and the range-versus-precision trade-off each format makes. What wasn’t covered is the actual engineering discipline of using mixed precision safely in a real training run, specifically loss scaling, the technique that makes FP16 trainable at all despite its narrow exponent range, and the specific failure signatures that show up when precision choices go wrong at frontier scale.

The triangle: throughput, memory, and stability#

Each halving of bits used roughly doubles tensor core throughput and halves memory footprint. BF16 versus FP32 is one 2x/2x step that’s essentially free, because BF16 keeps FP32’s full 8-bit exponent and only shrinks the mantissa, so there’s no range cost. FP8 versus BF16 is another 2x/2x step, but this one does cost stability, because FP8’s exponent shrinks too, 4 bits for E4M3 or 5 for E5M2, both narrower than BF16’s 8. Skipping mixed precision entirely leaves 2 to 4x of performance on the table, and this connects directly to the roofline model: precision sets $P_{\text{peak}}$ directly, so choosing a lower precision literally moves the ridge point, changing which operations are compute-bound versus memory-bound without touching a single line of the algorithm itself.

The standard pattern, and the one format that needs a safety net#

Forward and backward passes run in BF16, fast and half the memory, while gradients get accumulated and master weights get kept in FP32, stable, full precision for values updated repeatedly over thousands of steps. torch.autocast handles the forward-pass casting automatically, deciding per-operation which precision to run in without the practitioner manually casting every tensor.

FP16 is the one format in this mix that needs an extra mechanism, because its exponent is only 5 bits wide, far narrower than BF16’s 8. Small gradient values that would be perfectly representable in BF16 can underflow to exactly zero in FP16. GradScaler exists to fix exactly this.

Loss scaling, derived from the chain rule you already have#

Because differentiation is linear, scaling the loss by a constant scales every downstream gradient by that same constant, a direct consequence of the chain rule, not a new fact to memorize:

$L_{\text{scaled}} = S \times L, \qquad g_{\text{scaled}} = \frac{\partial L_{\text{scaled}}}{\partial W} = S \times \frac{\partial L}{\partial W} = S \times g$

Multiply the loss by a scale factor $S$ before calling backward, and every single gradient in the entire backward pass gets multiplied by that same $S$, pushing small values up out of FP16’s underflow range. After backward, divide every gradient by $S$ again, in FP32, before the optimizer step:

$g_{\text{recovered}} = \frac{g_{\text{scaled}}}{S}$

recovering the true gradient magnitude. If any gradient comes back as Inf or NaN, meaning $S$ was too large and something overflowed instead, the step is skipped entirely and $S$ is halved. If steps go cleanly for a while, every 2000 steps in the standard implementation, $S$ is doubled instead, continuously searching for the largest safe scale. This is an additive-increase-on-success, halve-on-failure control loop, the same shape as TCP congestion control’s slow start: search upward for the most aggressive safe setting, back off immediately the moment it proves too aggressive.

Worked example: a gradient that would have simply vanished#

A gradient of $10^{-8}$, cast directly to FP16, becomes exactly 0.0, below FP16’s smallest representable subnormal value of about $5.96\times10^{-8}$, this value has no representation at all except zero. A weight relying on this gradient literally never updates, step after step, with no error thrown anywhere.

Scaled by $S=1024$ before casting, the value becomes approximately $1.025\times10^{-5}$, comfortably within FP16’s representable range. After the backward pass, dividing by 1024 in FP32 recovers approximately $1.001\times10^{-8}$, close to the true $10^{-8}$, not zero. The gradient survived, and the weight update actually happens.

import numpy as np

def demonstrate_gradient_underflow_and_scaling(gradient_value, scale):
    fp32_grad = np.float32(gradient_value)
    unscaled_fp16 = np.float16(fp32_grad)
    scaled_fp16 = np.float16(fp32_grad * scale)
    recovered_fp32 = np.float32(scaled_fp16) / scale
    return unscaled_fp16, scaled_fp16, recovered_fp32

unscaled, scaled, recovered = demonstrate_gradient_underflow_and_scaling(1e-8, 1024.0)
print(unscaled, scaled, recovered)
# 0.0   1.0251998901367188e-05   1.0011717677116394e-08

np.float16(fp32_grad) performs a real IEEE 754 half-precision cast using NumPy’s actual binary representation, not a simulation, so the 0.0 is a genuine underflow, the same one that occurs inside a real training run.

class SimpleGradScaler:
    def __init__(self, init_scale=1024.0, growth_factor=2.0, growth_interval=2000):
        self.scale = init_scale
        self.growth_factor = growth_factor
        self.growth_interval = growth_interval
        self.clean_steps = 0

    def step(self, gradients):
        if any(np.isinf(g) or np.isnan(g) for g in gradients):
            self.scale /= 2.0
            self.clean_steps = 0
            return None, "skipped (Inf/NaN detected, scale halved)"
        recovered = [g / self.scale for g in gradients]
        self.clean_steps += 1
        if self.clean_steps >= self.growth_interval:
            self.scale *= self.growth_factor
            self.clean_steps = 0
        return recovered, "applied"

With growth_interval=3 to make growth visible quickly, and an injected Inf at step 4:

Step 1: scale=1024.0, status=applied
Step 2: scale=1024.0, status=applied
Step 3: scale=2048.0, status=applied   <- 3 clean steps reached, scale doubled
Step 4: scale=1024.0, status=skipped (Inf/NaN detected, scale halved)
Step 5: scale=1024.0, status=applied

Every branch of the control loop fires at least once here: growth at step 3, skip-and-halve at step 4, normal operation resuming immediately after.

FP8 needs a finer-grained fix than a single scale factor#

The naive approach applies one scale factor across an entire weight matrix. It underperforms for the same reason naive per-tensor INT8 quantization does: most values cluster in a narrow band while the single scale has to accommodate the few largest outliers, wasting FP8’s already-limited representable range on headroom the typical value never needs. DeepSeek-V3’s actual approach, which is more specific than “per-tensor scaling,” is tile-wise $1\times128$ scaling for activations and block-wise $128\times128$ scaling for weights, many separate, small scale factors instead of one global one, directly analogous to per-channel outlier handling generalized to a finer grid.

There’s a second, genuinely separate problem hiding underneath the storage format itself. NVIDIA’s FP8 tensor cores accumulate results in a 14-bit register internally, not enough precision for a long chain of additions inside a large matmul. DeepSeek-V3’s kernels periodically promote to FP32 accumulation, roughly every 4 tensor-core instructions, specifically to prevent this internal accumulator’s limited precision from silently corrupting a long summation. This is a hardware-level accumulation constraint that has nothing to do with how values are stored, and it’s easy to miss entirely if precision is only thought about at the storage level rather than the summation level. Two separate precision problems, solved by two separate techniques, both required simultaneously for FP8 training to actually work at this scale.

Real production incidents, with the sourcing corrected#

OPT-175B’s team observed a correlation between loss divergence, their dynamic loss scalar crashing to 0, and the L2-norm of the final layer’s activations spiking, and specifically chose restart points where the loss scalar was still in a “healthy” state, at or above 1.0. Worth being precise about sourcing here: that description is a paraphrase of the paper’s actual findings, not a verbatim quotation, even though every specific technical claim in it, the scalar crashing to 0, the activation norm spiking, the 1.0 threshold, checks out completely against the primary source. Getting the substance right and the framing wrong are different failures, and it’s worth naming the difference rather than letting an accurate paraphrase pass as something it isn’t.

DeepSeek-V3 is confirmed as the first validated large-scale FP8 training run, by DeepSeek’s own paper and corroborated by independent analysis. The actual engineering achievement is more specific than “per-tensor scaling,” it’s the combination of fine-grained tile and block-wise scaling, avoiding the outlier-driven range-wasting problem above, and periodic FP32 accumulation, avoiding the 14-bit accumulator’s precision limit, two separate fixes for two separate problems, both necessary at once.

Failure modes, ranked by how loudly they announce themselves#

Failure	Mechanism	Danger level
FP16 without loss scaling	Gradients silently underflow to 0; weights stop updating	The most dangerous failure here: no NaN, no crash, training looks normal while a fraction of the model learns nothing
BF16 overflow	Rare, but very large activations (e.g., embedding tables) can still overflow despite matching FP32’s exponent width	Uncommon but real: “same range as FP32” is a strong guarantee, not an absolute one for every activation pattern
FP8 without calibration	Entire layers produce Inf without proper tile/block scaling	Loud: training halts outright, far easier to catch than silent underflow
Wrong cast order	Accumulating a chain of FP16 values directly instead of upcasting to FP32 first	Silent precision loss: each intermediate addition loses precision a single final upcast would have preserved

The single thing worth internalizing above everything else in this post: FP16 underflow without loss scaling is uniquely dangerous precisely because it fails silently. Every other row in that table announces itself, an Inf, a NaN, a crash. This one doesn’t, and a model can burn a full training budget making zero progress on some fraction of its weights with nothing in the logs to flag it.

Common mistakes#

Assuming “the loss isn’t NaN” means training is healthy: a collapsed loss scalar sitting near zero, or gradients silently underflowing, produce completely normal-looking loss curves while real learning has partially stopped.

Treating FP8 as a drop-in replacement for BF16 with just a smaller scale factor: the storage-format underflow problem and the 14-bit accumulator problem are separate failure modes requiring separate fixes, and solving only one leaves the other free to silently corrupt a long summation.

Applying one global scale to an entire tensor for FP8 the way naive INT8 quantization does: it’s the same outlier-driven range-wasting mistake in a different numeric format, and it’s exactly why DeepSeek-V3 moved to tile and block-wise granularity instead.

Reporting a paraphrase as a verbatim quotation: the substance can be entirely correct while the framing misrepresents the source, and the two are worth catching separately rather than treating “the facts check out” as clearing the citation too.

What this takes to be frontier-job-ready#

The technical axis: given “training loss looks fine but the model isn’t improving,” know to check the loss scalar’s trajectory and whether it’s collapsed toward zero before assuming the data or architecture is at fault, precisely the diagnostic OPT-175B’s own team used.

The operational axis: monitoring a loss scalar for early warning signs and deciding in real time whether to skip a step or reduce the scale is exactly the kind of steady-under-pressure competence this series has already named as an explicit hiring bar. Precision instability is one of the concrete things that requirement is actually describing, not an abstract soft skill.

The autonomy axis: choosing calibration granularity for FP8, per-tensor versus tile-wise versus block-wise, is a judgment call made against a specific model’s own activation statistics. DeepSeek-V3’s team didn’t follow a universal recipe, they measured that per-tensor scaling underperformed for their specific case and engineered a finer-grained alternative. There’s no fixed script for this decision, only the willingness to measure and adjust rather than assume the standard answer is the right one.

Try it yourself#

Beginner. Using demonstrate_gradient_underflow_and_scaling, find the smallest scale factor that lifts a gradient of $10^{-8}$ above FP16’s minimum subnormal threshold, and confirm it recovers correctly after unscaling.

Intermediate. Extend SimpleGradScaler to track and print the number of skipped steps over a run, then simulate 10,000 steps with a 0.1% random Inf-injection rate and report the final converged scale.

Advanced. Explain why the 14-bit accumulator problem cannot be fixed by choosing a different FP8 storage format (E4M3 versus E5M2): work through why the accumulation precision limit is a property of the tensor core’s internal register, not of how the operands themselves are stored, and why periodic FP32 promotion is a summation-level fix rather than a storage-level one.

---

The one-sentence version: BF16 versus FP32 is a free 2x because only the mantissa shrinks, FP8 versus BF16 costs real stability because the exponent shrinks too, loss scaling recovers FP16’s underflowing gradients by exploiting the same chain-rule linearity from earlier in this series, and DeepSeek-V3’s actual FP8 breakthrough required fixing two independent precision problems at once, an outlier-driven scaling problem borrowed from quantization and a hardware accumulator limit borrowed from nowhere else, neither of which shows up if you only think about precision as a storage format instead of a full arithmetic pipeline. That Inf-or-NaN check GradScaler runs every step is the exact same signal autograd’s own exploding-gradient failure mode produces on its own, for a completely different reason.