Autograd is your hand-derived backward pass, automated, and where it silently lies to you

The manual backward pass through a linear layer and through attention was derived and checked against finite differences to within $10^{-6}$ earlier in this series. Autograd is the automation of exactly that work, nothing more mysterious than that. A Tensor is an array like every other one used throughout this series, carrying extra metadata: which device it lives on, its dtype, and whether operations on it should be tracked for gradient computation at all. The value of having derived the manual version first is specific: when autograd does something surprising, you know what it’s actually computing instead of treating it as a black box you can only restart and hope fixes itself.

Two real gaps, filled before building on them#

Broadcasting is the one that trains a model wrong without ever telling you. When two arrays of different shapes combine, NumPy and PyTorch both silently expand the smaller one to match, following fixed rules, with no error even when the result is semantically wrong. Subtracting a shape-(3,) array from a shape-(4,3) array succeeds silently, producing a shape-(4,3) result. If that (3,) array was meant to hold one value per example, which needs shape (4,) or (4,1), the computation just subtracted a per-feature value from every row instead of a per-example value from each row. The model still trains. The loss still goes down. The computation is wrong the entire time, and nothing in the code path ever raises an exception to tell you.

Vanishing and exploding gradients are what the chain rule looks like at 96 layers deep. Through $n$ layers, $\frac{dL}{dW_1}$ is a product of $n$ local derivatives, the multivariable chain rule again, just multiplied across many stages instead of two. If each local derivative has magnitude under 1, the product shrinks exponentially with depth: $0.9^{96} \approx 4.05\times10^{-5}$, meaning a 96-layer network with no mitigation sees its first layer receive a gradient roughly 25,000 times smaller than its last, effectively never learning. If each local derivative exceeds 1, the product grows instead: $1.1^{96} \approx 9412$, still large enough to overflow toward Inf or NaN in practice.

Worth pausing on that second number specifically. One version of this material in circulation states $1.1^{96} \approx 3.5$ million, off by roughly 372x from the actual value. It’s worth verifying arithmetic claims in source material rather than repeating them, the same discipline this series has applied to citations now applies to a plain calculation: $\ln(1.1) \times 96 \approx 9.15$, and $e^{9.15} \approx 9412$, not 3.5 million. Both numbers, corrected or not, support the same qualitative point, exponential growth is dangerous, but only one of them is actually correct.

Residual connections are the direct, mechanical fix. A layer computing $x + F(x)$ instead of just $F(x)$ has derivative $1 + \frac{dF}{dx}$. That leading $1$ means the gradient always has a path straight through, unmultiplied, no matter how small $\frac{dF}{dx}$ gets. This is exactly why every modern transformer is built from residual blocks rather than stacked layers directly, it’s a structural answer to the exponential-shrinkage problem just derived, not a heuristic that happens to help.

The computation graph, built exactly as the manual derivation anticipated#

Setting tensor.requires_grad=True means every subsequent operation on that tensor gets recorded, not just the result, but which function produced it and what its inputs were. This builds a DAG dynamically, one node per operation, as the forward pass actually executes, which is why PyTorch’s approach is called “define-by-run”: the graph isn’t specified in advance, it’s recorded as a side effect of running ordinary Python code.

loss.backward() is a topological sort followed by reverse traversal. Starting from the loss, a single scalar, PyTorch sorts the DAG topologically, then walks it in reverse, calling each recorded operation’s backward function in turn, each one exactly one of the formulas already derived by hand, the transpose rule for linear layers, the diagonal-minus-outer-product rule for softmax. The result of each backward function accumulates into the .grad attribute of the tensors that fed into it.

If a single tensor feeds into two different downstream operations, as in a residual connection or any branching computation, its gradient is the sum of the gradients flowing back from each path:

$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial f}\cdot\frac{\partial f}{\partial x} + \frac{\partial L}{\partial g}\cdot\frac{\partial g}{\partial x}$

$x$ affects $L$ through two independent routes, and the total sensitivity is the sum of the sensitivities along each route, the multivariable chain rule, applied automatically rather than by hand.

Worked example. $z = xy + x$ at $x=3, y=5$. Two paths from $x$ to $z$: through the $xy$ term (local derivative $y=5$) and through the bare $x$ term (local derivative $1$). Total: $\frac{dz}{dx} = y + 1 = 6$.

def z_func(x, y):
    return x * y + x

def numerical_partial_x(x, y, h=1e-6):
    return (z_func(x + h, y) - z_func(x, y)) / h

x_val, y_val = 3.0, 5.0
analytical = y_val + 1
numerical = numerical_partial_x(x_val, y_val)
print(f"Analytical: {analytical}, Numerical: {numerical:.6f}")
# Analytical: 6.0, Numerical: 6.000000

The independent finite-difference check agrees to six decimal places, confirming the accumulation rule isn’t just plausible, it’s precisely what a from-scratch numerical check produces.

The bugs everyone hits exactly once#

import torch

x = torch.tensor(3.0, requires_grad=True)
y = torch.tensor(5.0, requires_grad=True)
z = x * y + x            # builds the DAG as this line executes
z.backward()               # topological sort + reverse traversal
print(x.grad)               # tensor(6.) -- matches the hand derivation exactly

The missing-zero_grad() bug, reproduced conceptually:

for step in range(3):
    loss = (x * y + x)     # rebuild the graph each step
    loss.backward()          # accumulates INTO x.grad, does not overwrite it
    print(f"step {step}: x.grad = {x.grad}")
    # x.grad grows to 6, 12, 18 across steps, never reset between them

.grad accumulates by default rather than resetting. Calling backward() repeatedly without zero_grad() between calls silently multiplies the effective gradient magnitude by the number of steps since the last reset, which is functionally identical to secretly multiplying your learning rate by that same count. Nothing crashes. The loss curve just looks strange in a way that’s easy to blame on the model instead of one missing line.

The in-place-on-a-leaf bug:

leaf = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
# leaf[0] = 5.0
# RuntimeError: a leaf Variable that requires grad is being used in an
# in-place operation -- autograd cannot safely track this modification

A leaf tensor is one created directly by the user rather than the output of some other tracked operation. Modifying it in place would silently invalidate any already-recorded graph nodes that reference its original value, which is exactly why PyTorch refuses outright rather than risk producing silently wrong gradients, the one failure mode in this post that announces itself loudly instead of hiding.

The full loop, four lines that do all the real work:

model_weight = torch.tensor(0.5, requires_grad=True)
optimizer = torch.optim.SGD([model_weight], lr=0.01)

for step in range(100):
    optimizer.zero_grad()                  # reset .grad -- prevents the bug above
    prediction = model_weight * 2.0
    loss = (prediction - 10.0) ** 2          # target: model_weight should reach 5.0
    loss.backward()                           # populate .grad via the DAG traversal
    optimizer.step()                           # apply the update rule already derived

zero_grad() sits at the start specifically to prevent the accumulation bug demonstrated above. Every other line is exactly the sequence already built: forward pass, DAG-recorded backward pass, optimizer update.

Where autograd connects to everything else already covered#

torch.no_grad() disables graph construction entirely for a block of code, used for inference, where no backward pass will ever be called and tracking a graph would waste memory and compute for nothing.

torch.compile traces the computation graph and emits optimized, fused CUDA kernels rather than executing operations one at a time, connecting directly to the arithmetic-intensity framing from the roofline post: fusing operations reduces redundant HBM round-trips between them, which is exactly how a kernel moves closer to the ridge point from below. One caveat worth stating honestly rather than asserting as settled: torch.compile is commonly described as falling back to eager mode, silently, when it hits an unsupported custom CUDA op, at a real cost in runtime with no warning printed. That’s consistent with the framework’s general graceful-degradation design philosophy, but exact fallback behavior is the kind of detail that shifts across versions, so treat it as plausible rather than confirmed without checking against whatever version you’re actually running.

FSDP and DDP add hooks that fire automatically as .grad gets populated during the backward pass, triggering the reduce-scatter-then-all-gather AllReduce collective to synchronize gradients across every participating GPU. Distributed training’s communication isn’t a separate step bolted onto the training loop, it’s woven directly into the same backward-pass mechanism already built here. That’s also why one dead rank hangs the entire job: the AllReduce hook fires from inside .grad population, so a rank that never reaches backward() never fires the hook, and every other rank waits at that barrier forever, the identical mechanism as everywhere else this barrier has shown up in this series.

Failure modes, ranked by how much they announce themselves#

Failure	Mechanism	Fix
Missing zero_grad()	.grad accumulates across steps instead of resetting	Call zero_grad() once per iteration, consistently
In-place op on a leaf tensor	Would invalidate graph nodes referencing the tensor’s original value	PyTorch raises an error; the one failure here that isn’t silent
Vanishing gradients	Chain of local derivatives under 1, multiplied across many layers	Residual connections, the derived $1+\frac{dF}{dx}$ fix
Exploding gradients	Chain of local derivatives over 1; $1.1^{96}\approx9412$ is large enough to overflow	Gradient clipping and loss scaling, already covered
Broadcasting error	Shape mismatch resolved silently instead of raising an error	No automatic fix; explicit shape assertions before the operation

Real production practice, not just the mechanics#

A widely echoed piece of advice for breaking into frontier-lab work is coding a small transformer from scratch by hand, specifically to internalize matrix multiplication and gradients directly rather than through a framework, the same discipline this whole series has followed. Andrej Karpathy’s llm.c is the concrete, checkable version of what that looks like taken to its extreme: it reproduces GPT-2 training in raw C/CUDA in roughly 4,000 lines, with no autograd at all, meaning every gradient in it is computed by exactly the manual backward-pass logic already derived in this series, written out by hand instead of automated. That’s worth sitting with: the “automated” version and the “from scratch” version compute the identical mathematics, llm.c just makes every one of autograd’s hidden steps visible as actual code you can read top to bottom.

Common mistakes#

Assuming a smoothly decreasing loss curve means the computation is correct: a broadcasting bug produces exactly this, a plausible-looking curve built on a silently wrong per-feature-instead-of-per-example computation.

Forgetting zero_grad() and mistaking the resulting instability for a learning-rate problem: the fix isn’t a smaller learning rate, it’s the one missing reset call, and lowering the learning rate to compensate just delays the same failure.

Treating vanishing and exploding gradients as the same problem with opposite signs: they have different fixes, residual connections for one, clipping and loss scaling for the other, and neither fix reliably helps the other failure.

Repeating an arithmetic claim from a source without checking it: the $1.1^{96}$ correction above exists because verifying a plain calculation costs one line of Python and catches an error three orders of magnitude off, the same standard this series holds citations to.

What this takes to be frontier-job-ready#

The technical axis: given “training loss is behaving strangely,” distinguish the specific, distinguishable signatures already built here, gradient accumulation from a missing zero_grad() (the effective learning rate seems to be climbing), vanishing gradients in early layers (near-zero updates deep in the network), or exploding gradients (Inf/NaN, connecting straight to the loss-scaling post). Each has a different fix, and conflating them wastes real debugging time.

The operational axis: implementing a stable, fast version of a new training algorithm someone else proposed is standard PyTorch-implementation work at a frontier lab, and doing it correctly requires exactly the DAG and accumulation understanding built here, not just familiarity with the high-level API surface.

The autonomy axis: the from-scratch coding exercise recommended above is itself a test of this axis. The value is explicitly in doing the derivation yourself rather than being handed one, the same principle this entire series has followed since its first post.

Try it yourself#

Beginner. Using the four-line training loop above, remove optimizer.zero_grad() and run 5 steps, printing model_weight.grad each time. Confirm it grows rather than resets, and explain why this behaves like a learning rate that’s secretly increasing.

Intermediate. Construct a shape-(4,) per-example array and a shape-(4,3) per-feature array, and deliberately broadcast them the wrong way (subtracting the wrong axis). Confirm NumPy raises no error, then write an explicit shape assertion that would have caught the mistake before it silently ran.

Advanced. Build a 50-layer toy network where each layer multiplies its input by a fixed scalar $c$, without residual connections. Compute the ratio between the first and last layer’s gradient magnitude analytically for $c=0.9$ and $c=1.1$, then verify both numerically, and explain precisely why adding $x + c \cdot x$ instead of $c \cdot x$ at each layer eliminates the exponential blowup in the $c=1.1$ case.

---

The one-sentence version: autograd builds a DAG as your forward pass runs and walks it backward doing exactly the transpose-rule and softmax-Jacobian arithmetic you already derived by hand, gradients at a branch point sum rather than average, and its two most expensive failure modes, a forgotten zero_grad() and a silent broadcasting mismatch, both look like a healthy, decreasing loss curve right up until someone checks the actual numbers underneath it. That DAG walks backward at all is itself a choice, and it’s the only choice that actually works at this scale.