What a language model's loss actually measures: cross-entropy, KL divergence, and perplexity

When a language model predicts the next token, it doesn’t output one answer, it outputs a full probability distribution: a number for every token in its vocabulary, all of them nonnegative, all of them summing to exactly 1. “The cat sat on the ___” might get mat: 0.7, chair: 0.2, floor: 0.08, moon: 0.02, across every token in a 100,000-word vocabulary. Training a language model is adjusting that distribution, over and over, so it puts more weight on whatever token actually came next in real text. Softmax isn’t a probability function by loose analogy, its output genuinely satisfies the definition, nonnegative entries summing to one, which is exactly why it’s the standard way to turn a model’s raw scores into something that can be scored against reality.

Cross-entropy: a penalty derived, not assumed#

Say the true next token is “mat,” which means the correct distribution is $1$ on “mat” and $0$ everywhere else, a one-hot distribution: all belief concentrated on a single outcome. The model assigns “mat” some probability $p$, and you need a number that’s small when $p$ is close to $1$ and large when $p$ is close to $0$. $-\log(p)$ does exactly this: as $p \to 1$, $-\log(p) \to 0$, no penalty for a confident correct answer; as $p \to 0$, $-\log(p) \to \infty$, unboundedly large penalty for being confidently wrong. It isn’t an arbitrary choice among equally good options, it’s the function that makes the gradient below come out clean, which is the actual reason it’s used everywhere rather than some other penalty shape.

For a one-hot true distribution $y$ and predicted distribution $\hat y$ (softmax’s output), cross-entropy collapses to a single term, since every other entry of $y$ is zero:

$L = -\sum_i y_i \log(\hat y_i) = -\log(\hat y_c)$

where $c$ is the correct class. In language modeling this is almost always the one-hot case, so cross-entropy reduces to something simpler than the general sum suggests: just $-\log$ of the probability the model assigned to the actual right answer.

The gradient of this loss with respect to the pre-softmax logits $z$ is where it gets genuinely elegant. Chaining together the softmax Jacobian from backprop through attention (diagonal terms $\alpha_i(1-\alpha_i)$, off-diagonal terms $-\alpha_i\alpha_j$) with the derivative of $-\log(\hat y_c)$, every messy term cancels, leaving:

$\frac{\partial L}{\partial z_i} = \hat y_i - y_i$

Predicted probability minus true probability, entry by entry. This clean cancellation is exactly why softmax and cross-entropy are paired together almost universally rather than mixed and matched with other loss functions: the messy Jacobian terms from one and the messy log-derivative from the other were never going to cancel this cleanly with some other pairing.

Worked example. Vocabulary [mat, chair, floor, moon], true token “mat” so $y=[1,0,0,0]$, predicted $\hat y = [0.7, 0.2, 0.08, 0.02]$. Loss: $-\log(0.7) \approx 0.357$. Gradient: $[0.7-1, 0.2, 0.08, 0.02] = [-0.3, 0.2, 0.08, 0.02]$, negative at the correct token (push its probability up), positive everywhere else (push them down). The gradient’s magnitude is quite literally “how wrong was this prediction.”

import numpy as np

def softmax(z):
    z = z - np.max(z)
    e = np.exp(z)
    return e / np.sum(e)

def cross_entropy(y_true_onehot, y_pred):
    return -np.log(y_pred[np.argmax(y_true_onehot)])

z = np.array([1.5, 0.5, -0.3, -1.8])
y_true = np.array([1, 0, 0, 0])
y_pred = softmax(z)

print(cross_entropy(y_true, y_pred))    # 0.357...
print(y_pred - y_true)                  # the clean gradient: [-0.3, 0.2, ...]

Checking this gradient against an independent finite-difference estimate (perturb each logit, measure the loss change, divide) confirms the two agree to about $10^{-6}$: the cancellation is a real algebraic fact, not a hand-wave.

This is not a toy example scaled down. Cross-entropy between a model’s predicted next-token distribution and the actual next token in the training corpus is the training objective for essentially every pretrained LLM: millions of applications of this exact $\hat y - y$ gradient, one per token, flowing backward through the attention and linear-layer machinery underneath.

KL divergence: the more general object cross-entropy hides inside#

$D_{KL}(P \| Q) = \sum_i P(i) \log\frac{P(i)}{Q(i)}$

KL divergence measures how much probability mass gets “wasted” describing a system actually governed by $P$ using an approximation $Q$ instead. It is not symmetric, $D_{KL}(P\|Q) \ne D_{KL}(Q\|P)$ in general, and that asymmetry isn’t a flaw, it’s meaningful: penalizing $Q$ for being confident somewhere $P$ isn’t is a different failure than penalizing $Q$ for being unconfident somewhere $P$ is. RLHF exploits this directly. Every standard RLHF objective penalizes a policy model’s KL divergence from a reference model, not the reverse, specifically to control one of those two failure modes and not the other.

Entropy: the floor cross-entropy can never beat#

Define the “surprise” of an outcome with probability $p$ as $-\log(p)$, the same function cross-entropy was built from. Entropy is the expected surprise of a distribution under itself:

$H(P) = -\sum_i P(i)\log P(i)$

A coin that’s 100% heads has zero entropy, no surprise is possible. A fair coin has maximum entropy for two outcomes, every flip is a genuine toss-up. For a model finishing “The capital of France is ___,” low entropy is the right answer; for a model at the start of a story, high entropy is.

The relationship between entropy, cross-entropy, and KL divergence is worth stating precisely rather than leaving implicit:

$\text{CE}(y, \hat y) = H(y) + D_{KL}(y \| \hat y)$

Cross-entropy, the actual training loss, equals the data’s own irreducible uncertainty plus a nonnegative penalty for the model being imperfect. Training drives that KL term toward zero, but it can never push cross-entropy below $H(y)$, no matter how good the model gets, because $H(y)$ is a property of the data, not the model. Language has genuine irreducible uncertainty: usually several genuinely plausible next words exist, there isn’t one deterministic right answer, and the entropy term is where that fact lives mathematically. For a one-hot true distribution specifically, $H(y) = 0$ and $D_{KL}(y\|\hat y)$ becomes numerically identical to cross-entropy itself, which is exactly why the one-hot case in language modeling makes KL divergence feel almost redundant with cross-entropy: in that case, it is.

def entropy(p, eps=1e-12):
    p = np.clip(p, eps, 1.0)
    return -np.sum(p * np.log(p))

def kl_divergence(y_true, y_pred, eps=1e-12):
    y_true_c, y_pred_c = np.clip(y_true, eps, 1.0), np.clip(y_pred, eps, 1.0)
    return np.sum(y_true * np.log(y_true_c / y_pred_c))

H = entropy(y_true.astype(float))                 # 0.0, one-hot has no uncertainty
CE = cross_entropy(y_true, y_pred)                 # 0.357
KL = kl_divergence(y_true.astype(float), y_pred)   # 0.357, identical to CE here
print(H, KL, H + KL, CE)                           # H + KL == CE, exactly

Perplexity: the number everyone actually reports#

Perplexity is just entropy re-expressed in a more intuitive unit: $2^H$ (or $e^H$, matching whatever log base was used). It isn’t a separate concept, it’s a unit conversion, chosen because “the model was as uncertain as if guessing uniformly among 20 options” lands more intuitively than “cross-entropy was 3.0 nats,” even though the two numbers carry identical information. When a paper reports “perplexity 15 on this benchmark,” that’s the average cross-entropy over a large held-out corpus, exponentiated back out of log-space; lower means the model was less surprised by real text on average, and it’s the standard cheap sanity check run before any more expensive downstream evaluation.

Concept	Question it answers	Formula
Entropy	How uncertain is this one distribution, on its own?	$-\sum P\log P$
Cross-entropy	How surprised is a predicted distribution by reality?	$-\sum y \log \hat y$
KL divergence	How much of that surprise is the model’s fault, beyond irreducible entropy?	$\text{CE} - H$
Perplexity	Cross-entropy, rescaled to “effective number of equally likely options”	$2^H$ or $e^H$
Mutual information	How much does knowing $Y$ reduce uncertainty about $X$?	$H(X) - H(X\mid Y)$

Mutual information, $I(X;Y) = H(X) - H(X\mid Y)$, is the one entry in that table worth a separate word because it’s easy to invoke loosely. If $Y$ tells you nothing about $X$, $H(X\mid Y) = H(X)$ and $I(X;Y)=0$; if $Y$ fully determines $X$, $H(X\mid Y)=0$ and $I(X;Y)=H(X)$, the maximum possible reduction. Papers and blog posts that say an attention head “encodes information about” something are sometimes making this precise, computed claim, and sometimes just speaking loosely; the two get blurred constantly, and it’s worth noticing which one you’re actually reading.

Common mistakes#

Treating softmax’s output as merely “probability-like” instead of a genuine probability distribution: the whole reason cross-entropy applies to it at all is that it satisfies the actual mathematical definition, not an approximation of one.

Assuming a well-trained model’s cross-entropy should approach zero: it approaches $H(y)$, the data’s own irreducible entropy, which is nonzero for essentially all real language. A loss that’s stopped dropping isn’t necessarily a broken run, it might be near the actual floor.

Letting $\hat y_c$ underflow to exactly $0.0$ in low-precision floating point, sending $-\log(0)$ to infinity or NaN. This is the same range-limit failure mode as everywhere else in this series, and the standard fix, a small epsilon before the log or working in log-space throughout, exists because this is common enough to need one.

Using “information” in the loose, intuitive sense and the rigorous mutual-information sense interchangeably: they aren’t the same claim, and conflating them is a real, documented source of imprecise statements in interpretability writing.

Try it yourself#

Beginner. For $\hat y = [0.5, 0.3, 0.2]$ with true token at index 1, compute cross-entropy by hand, then the gradient $\hat y - y$, and check that the loss isn’t tiny (0.3 isn’t a confident correct prediction).

Intermediate. Construct a soft label, $y = [0.6, 0.4, 0, 0]$, representing genuine ambiguity in the true answer rather than one-hot certainty. Show $H(y) > 0$ in this case, and verify $H + D_{KL} = \text{CE}$ still holds.

Advanced. Derive the $\hat y - y$ result from scratch: combine the softmax Jacobian’s diagonal and off-diagonal terms with the derivative of $-\log(\hat y_c)$ with respect to $\hat y_c$, and show the messy pieces cancel into the clean subtraction, rather than trusting the finite-difference check above as sufficient proof.

---

The one-sentence version: a model’s output is a genuine probability distribution, cross-entropy measures how surprised that distribution is by the actual next token in a way that happens to produce a remarkably clean gradient, $\hat y - y$, and every number derived from it, KL divergence, entropy, perplexity, is a different way of asking the same question: how much of that surprise is irreducible in the data itself, and how much of it is the model’s own fault. That $\hat y - y$ gradient is also exactly what gradient descent and Adam consume on every single training step, millions of times over.