A hallucination isn't a black box: a linear probe on one mid-layer hidden state can already tell a claim is false before the model finishes the sentence

The frame this series has used for hallucination up to now, that a model “doesn’t know it doesn’t know,” is close but not quite right, and the imprecision matters. It implies the information isn’t there. It usually is. The real, more useful question is narrower and more operational: when a model states something false with total fluency, what do you actually do about it, mechanically, not “prompt it better” or “add a disclaimer,” but where in the forward pass does the wrongness live, and what can you attach to that location.

Why “the model doesn’t know” is the wrong frame#

A pretrained model’s only job is minimizing $-\log p(\text{correct token})$, one token at a time, over trillions of tokens. Nothing in that objective ever rewards the token “I don’t know.” So when a query lands somewhere with weak or absent evidence, an obscure fact, a name close to several real ones, a question past the training cutoff, the residual stream still has to produce something at the final layer. What it produces is whichever direction the feedforward layers’ learned priors push hardest, a fluent, specific, plausible continuation, because that’s what next-token prediction was ever trained to do. Not a gap in knowledge. A gap in what the training objective ever asked the model to do with uncertainty.

That reframing has a testable consequence: if the wrongness is a late-stage decision about what to say, not an early absence of any relevant signal, then something upstream of the final logits should already carry evidence the answer doesn’t check out. That’s a claim you can go looking for directly, not just theorize about.

The probe: reading truth off a single hidden state#

Azaria & Mitchell (2023, “The Internal State of an LLM Knows When It’s Lying”) went looking for exactly that. They took hidden states from a single mid-layer of a pretrained model, generated for a set of true and false factual statements spanning several unrelated topics, and trained a small logistic-regression classifier on nothing but that one vector per statement. The probe predicted true-versus-false well above chance, including in their cross-topic generalization tests, where the probe was trained on statements about one subject and evaluated on statements about a subject it had never seen labeled. A linear classifier on one hidden state, nowhere near the output layer, already separating true from false.

That’s the direct evidence for the mid-layer framing this series has gestured at before without pinning down: whatever mechanism produces a hallucination, the “this doesn’t check out” signal is frequently sitting in the residual stream well before it reaches the unembedding matrix. The output logits, and the confidence a user actually sees, are downstream of a representation that was arguably more honest a few layers earlier.

Attention diffusion: a signal you already paid for#

The probe needs a trained classifier. There’s a cruder, nearly-free signal sitting in the same forward pass: how concentrated the attention pattern is at the position generating a claim. Scaled dot-product attention is a similarity match between a query and every key in context, and when nothing in context strongly matches, the softmax doesn’t collapse onto one position, it spreads thin across many weakly relevant ones. A diffuse attention pattern at the exact position where a claim is being generated correlates with “no single strong source exists for what’s about to be said,” and unlike the probe above, it costs nothing extra to compute, the weights are already sitting in memory from the forward pass that just ran.

def confidence_signal(logits, attn_weights, entropy_threshold=3.0, attn_peak_threshold=0.15):
    probs = logits.softmax(dim=-1)
    entropy = -(probs * probs.clamp_min(1e-12).log()).sum(-1)
    peak_attn = attn_weights.max(dim=-1).values.mean()   # how concentrated is attention, right now
    confident = (entropy.mean() < entropy_threshold) and (peak_attn > attn_peak_threshold)
    return confident, entropy.mean().item(), peak_attn.item()

Neither signal alone is a hallucination detector. Low output-token entropy doesn’t mean correct, a model can be fluently, confidently wrong with a near-one-hot distribution over the wrong token, which is exactly the failure case that matters most. But entropy and attention diffusion are cheap, already-computed, and worth logging by default, the probe above is the one that’s actually discriminative, and it costs a forward hook and a few hundred labeled examples to train.

The logit lens: watching the wrong answer lock in#

If the mid-layer signal is real, there should be a visible moment where the model’s best guess, read out layer by layer, settles on the wrong answer. The logit lens (nostalgebraist, 2020) makes that literal: take the hidden state at any intermediate layer, skip straight to the model’s own unembedding matrix, and read off what token it would predict right now, as if that layer were the last one.

def logit_lens(model, hidden_states_per_layer, layer_idx):
    h = model.final_norm(hidden_states_per_layer[layer_idx])   # models expect this before unembedding
    logits = h @ model.unembed.weight.T
    return logits.argmax(dim=-1)

for l, h in enumerate(hidden_states_per_layer):
    guess = logit_lens(model, hidden_states_per_layer, l)
    print(l, tokenizer.decode(guess[0, -1]))

Run this across every layer for a hallucinated span and the eventual wrong answer typically doesn’t appear early. Earlier layers often show a genuinely diffuse distribution across several candidates, sometimes including the correct one, before the representation collapses onto a single, wrong, confident answer somewhere in the back third of the stack. That’s the same result as the probe, from a different angle: the commitment to the wrong answer is a late-stage event you can watch happen, not a property baked in from layer one. Watching where it locks in, not just what it ends at, is what actually localizes which layers are doing the damage in a specific case.

Inference-Time Intervention: steering the direction, not retraining the model#

If a truthfulness direction is linearly readable from a hidden state, per Azaria & Mitchell’s probe, the same direction should be usable in the other direction, to push the model toward it. That’s exactly what Li et al. (NeurIPS 2023, “Inference-Time Intervention”) did: fit linear probes per attention head on labeled true and false statements, identify the heads whose probes are most predictive, and at inference time shift each of those heads’ activations along the learned truthfulness direction by a small fixed amount, no gradient update, no retraining, applied once per generated token. The paper reports a large improvement on TruthfulQA, roughly doubling the fraction of answers judged both truthful and informative relative to the unmodified model, from a purely inference-time intervention on a small number of heads.

That’s activation steering with a specific direction and a specific target, not a general demonstration that concepts are linear in activation space. It’s the same mechanism the “Golden Gate Claude” demo used for a landmark, applied here to something that actually matters operationally: given that the direction is findable and the shift is cheap, there’s no excuse for a production system to leave that lever unused once the probe already exists as a byproduct of debugging.

Five places this actually goes in code#

None of the above is useful sitting in a paper. Here’s where each piece attaches to a real generation path.

# 1. retrieval-gated generation: only answer from evidence actually retrieved
def retrieval_gated_generate(model, query, retriever, min_score=0.4):
    docs, scores = retriever.search(query)
    if max(scores, default=0) < min_score:
        return "I don't have reliable information to answer that."
    context = "\n".join(docs)
    return model.generate(prompt=f"Context:\n{context}\n\nQuestion: {query}")

# 2. confidence gate, from the entropy + attention-diffusion signal above
def should_abstain(logits, attn_weights):
    confident, _, _ = confidence_signal(logits, attn_weights)
    return not confident

# 3. instrument the mid-layer probe directly, as a forward hook
activations = {}
model.blocks[len(model.blocks) // 2].register_forward_hook(
    lambda module, inp, out: activations.__setitem__("mid_layer", out.detach())
)
# after the forward pass:
truth_score = truthfulness_probe(activations["mid_layer"][:, -1, :])   # the Azaria & Mitchell probe

# 4. groundedness check: does the answer actually overlap the retrieved evidence
def groundedness_score(answer_tokens, retrieved_tokens):
    overlap = set(answer_tokens.tolist()) & set(retrieved_tokens.tolist())
    return len(overlap) / max(1, len(set(answer_tokens.tolist())))

# 5. penalize unsupported continuations directly in preference optimization
# chosen = grounded answer, rejected = fluent-but-unsupported answer, same prompt
loss = dpo_loss(policy_logps_chosen, policy_logps_rejected,
                 ref_logps_chosen, ref_logps_rejected, beta=0.1)

Points 1 and 5 are the two that touch training or serving infrastructure most directly, and they target the mechanism from the first section head-on: 1 gives the model real evidence to compete against the FFN prior in-context, 5 trains the preference for grounded answers directly instead of hoping abstention emerges on its own, because as established above, nothing in pretraining ever taught it to.

The debate that isn’t settled#

Everything above treats truthfulness as something to detect and steer after the fact. There’s a second, harder question this series has already touched from two different angles without fully reconciling them: OpenAI’s own GPT-4 report documents that RLHF measurably degrades calibration relative to the pretrained base model, the tuned model sounds more certain without being more correct, because raters preferred confident-sounding answers during training, not because confidence started tracking reality better. Separately, DeepSeek-R1-Zero shows that outcome-only reinforcement learning, correctness reward and format reward only, no step-level supervision, no human-labeled reasoning trace, no probe or steering vector anywhere in the loop, produces self-verification and backtracking behavior on its own.

Put those next to each other and the open question is real: is mechanistic instrumentation, probes, logit lens, steering, actually the path to better-calibrated models at the frontier, or is it a debugging and post-hoc-correction toolkit for a problem that outcome-only RL at sufficient scale might route around entirely, the way R1-Zero appears to have arrived at reasoning behaviors nobody hand-specified. Both are true simultaneously right now. Neither side of that has won yet, and presenting it as settled in either direction is the tell that someone stopped reading a year before the field did.

Where this actually gets tested#

This doesn’t get asked as “define hallucination.” It shows up as “your RAG system’s retrieval missed, what does the model do, and how would you have caught that in an eval before a user did,” which tests whether groundedness checking is a designed system property or an afterthought. It shows up as “walk me through what you’d instrument to catch a hallucination before it’s shown to a user, not after,” which has a real wrong answer, output-only heuristics like a low top-1 probability, and a real right answer, something upstream of the final logits, because that’s specifically what the probe evidence above demonstrates output confidence can miss. And it shows up as “is more RLHF the fix for hallucination,” where the correct answer engages with the calibration-degradation finding directly instead of assuming more post-training is strictly better.

What this takes to be frontier-job-ready#

Google DeepMind’s own Research Engineer definition describes the role as “a critical bridge between theory and implementation, designing, building, and scaling complex systems to test and evaluate new ideas.” A forward hook that turns a 2023 interpretability paper’s method into a running probe on a production model’s activations is precisely that bridge, not a research exercise kept separate from shipping code.

Mistral’s Applied Scientist JD states the operational bar in blunt, verbatim terms: “you don’t panic when you see OOM errors or when NCCL feels like not wanting to talk… you don’t need roadmaps: you just do.” Deciding, without being told, that a hallucination bug needs a layer-by-layer trace rather than a prompt tweak is exactly that kind of self-directed diagnostic instinct, applied to a model-behavior bug instead of an infrastructure one.

OpenAI’s stated bar for who succeeds is people who “own ambiguous problems end-to-end without needing a tightly specified roadmap.” The debate in the section above, whether mechanistic tooling or outcome-only RL is the real long-run answer to calibration, has no roadmap yet. Forming a defensible position from the actual evidence on both sides, not from whichever paper was read first, is the specific skill that bar is naming.

Common mistakes#

Treating low output-token entropy as proof of correctness: it’s evidence the model committed to an answer, not evidence the answer is right, and confident hallucination is a near-one-hot distribution over the wrong token, the exact case entropy alone can’t catch.

Building a hallucination detector entirely from output-layer signals when the probe evidence says the more honest signal sits upstream: a probe on a mid-layer hidden state outperforms output-confidence heuristics precisely because it’s reading a representation before the model has committed to sounding certain.

Treating retrieval grounding as sufficient on its own: RAG gives the model evidence to compete against the FFN prior, it doesn’t force the model to use that evidence, which is why groundedness scoring against the retrieved spans is a separate, necessary check, not a redundant one.

Presenting RLHF as a strictly-improving post-training step for this specific problem: the GPT-4 report’s own calibration finding says otherwise, and repeating “just RLHF it more” without that caveat is the tell described in the section above.

Try it yourself#

Beginner. Using the confidence_signal function above, run a small model on five prompts with clear, well-known answers and five prompts about obscure or fictional facts it can’t know, and check whether entropy and attention-peak actually separate the two groups on average, then find at least one case where they don’t, a confident wrong answer.

Intermediate. Register a forward hook at three different layers, early, middle, and late, and implement a version of logit_lens that prints the top-3 predicted tokens at each of the three layers for a single hallucinated generation, to see directly whether the wrong answer is present early or only appears late.

Advanced. Using a small open model, construct a labeled dataset of 40 to 60 true and false one-sentence factual statements across at least three unrelated topics, extract a single mid-layer hidden state per statement, and train a logistic-regression probe, then test it on a held-out topic it never saw during training, the actual cross-topic generalization test Azaria & Mitchell ran.

---

The one-sentence version: a hallucination isn’t the absence of a signal, it’s a fluent commitment the model makes after a signal that the claim doesn’t check out was already linearly readable in a mid-layer hidden state, which means the honest fix isn’t a better prompt, it’s a probe, a logit-lens trace, or a steering vector attached to the exact layer where that commitment happens. The retrieval side of this same problem and the post-training side are both already covered in this series; this is the part that lives inside the forward pass itself.