Evaluating GenAI systems: why each metric exists, and exactly how it lies to you
The one test that actually matters for any eval number, worked through NLG quality, classification, calibration, RAG, hallucination, code generation, LLM-judges, safety, agents, and the operational layer, with the specific failure mode where each metric goes green while the product gets worse.
Every eval metric in production is answering one question underneath whatever it claims to measure: can I trust this number enough to ship on it, or block a release on it. That’s the only test that matters, and it fails constantly. A metric that looks rigorous on paper but that nobody on the team can explain to a stakeholder in one sentence, or that doesn’t move when the product actually gets worse, is worse than useless, it’s a false sense of safety. Most eval incidents I’ve seen were caused by a dashboard that stayed green, not by a metric that was missing entirely.
This is a field guide to the metrics that decide real production questions: what to route where, what release to block, what gets paged at 3am. For each family, three things matter more than the formula: why the metric exists, a worked example with real numbers, and the specific way it lies to you, since that part is what every glossary skips and what actually costs you an incident.
The four axes, and why one hero number never survives contact with production
No single metric survives being turned into the only thing a team optimizes against. Every eval program that’s held up under real traffic ends up organized around four largely independent axes instead of one score: quality (is the output good, true, and complete), safety (would shipping this hurt someone or leak something), reliability (would the same quality happen again on the next run, the next user, the next reordering of the same choices), and operational (is it fast and cheap enough to survive real traffic at all). A system can be excellent on one axis while quietly failing on another, and a blended score hides exactly which one broke. Everything below maps onto one of these four axes. The closing section explains why collapsing them back into a single number is the most common way a mature eval program degrades.
Fluency isn’t correctness: why BLEU and ROUGE measure the wrong failure mode now
BLEU, ROUGE, METEOR, and chrF were built for machine translation and summarization in the 2002 to 2018 era, when the dominant failure mode was disfluent, garbled output. A statistical MT system in 2005 would produce “the cat sat mat on the,” and n-gram overlap against a reference was a genuinely good proxy for “does this look like real language.”
Modern LLMs essentially never produce disfluent garbage. So n-gram metrics have quietly become proxies for the wrong failure mode. This is the single most common mistake I see junior eval engineers make: running ROUGE on a summarizer and treating a 0.4 score as “bad,” when the summary is perfectly good, just phrased differently than the reference.
Worked example. Reference: “The Federal Reserve raised interest rates by 0.25%, citing persistent inflation concerns, marking the fifth consecutive increase this year.” Model output: “Citing ongoing inflation worries, the Fed hiked rates a quarter point, the fifth straight hike this year.” ROUGE-L (longest common subsequence) on this pair comes out around 0.3 to 0.4, because nearly every content word got paraphrased: “raised” to “hiked,” “0.25%” to “a quarter point,” “consecutive increase” to “straight hike.” A human reading both would call this a perfect summary. If your CI gate is “ROUGE-L above 0.5 or block the release,” you just blocked a good model.
This is why BERTScore and embedding-based similarity exist: they use contextual embeddings to recognize that “hiked” and “raised” occupy similar semantic space, and score this pair around 0.85 to 0.9. That’s the right answer for this failure mode.
Where BERTScore also lies to you. It can be fooled by fluent hallucination. If the model instead outputs “The Fed cut rates by 0.25%, reversing course due to a slowing economy,” that sentence is fluent, uses similar vocabulary and structure, and still scores reasonably high on BERTScore (maybe 0.75 to 0.8) despite being factually inverted, cut versus raised. Embedding similarity measures topical closeness, not factual polarity. I’ve watched a team ship a financial-summarization feature with BERTScore as the sole gate and get burned exactly this way: the eval set didn’t have enough polarity-flip adversarial examples. The lesson: n-gram and embedding metrics tell you “is this talking about the same thing, in similar language.” Neither tells you “is this true.” That’s faithfulness, covered below, and you need both.
One footnote worth knowing before it costs you a day: if two papers report wildly different BLEU numbers for the “same” model on the “same” dataset, it’s almost always a tokenization mismatch, one used the Moses tokenizer, one used something else, and BLEU is brutally sensitive to that. sacreBLEU exists purely to standardize this so scores are comparable across papers.
Diversity metrics catch a specific frontier-model failure: mode collapse. When you fine-tune a model heavily against a reward model, the policy can learn that a particular phrasing reliably scores well and starts producing it constantly. That’s the “as an AI language model, I…” disease from early RLHF-era chat models, or more subtly, a support bot that opens every response with “I completely understand your frustration” regardless of whether the customer expressed any. Distinct-2 (unique bigrams over total bigrams, across many generations) drops sharply when this happens. Self-BLEU (how similar generation N is to generations 1 through N-1) rises. Neither BLEU/ROUGE nor BERTScore would ever catch this, since both compare one output to one reference. Diversity metrics are population-level: you need hundreds of generations to see the collapse.
A realistic pattern: a support-bot team notices CSAT quietly declining over two months even though every individually spot-checked response looks fine. Running Distinct-2 across 500 sampled production responses reveals it dropped from 0.71 to 0.44 over that window, the model had converged onto six or seven stock phrasings a reward model liked, and customers started feeling like they were talking to a script, even though no single response was individually wrong.
Classification metrics: what your moderation and routing layer actually runs on
Every production LLM system has a classification layer hiding somewhere, even when the user only sees generated text: an intent router deciding billing versus technical, a moderation classifier deciding block versus allow, a hallucination detector deciding supported versus not, an ambiguity detector deciding whether to ask a clarifying question. All of them are binary or multi-class classifiers, and the classic ML metrics apply directly.
Precision versus recall is a business decision disguised as a math problem. Take a jailbreak or self-harm classifier sitting in front of a consumer chat product. Its ROC-AUC might look great (0.94), but that number alone tells you nothing about what to ship. Two reasonable teams will land in different places:
High recall, lower precision (catch 98% of true self-harm content, but 1 in 20 flagged messages is actually benign, someone discussing a movie plot): the cost is some legitimate users get an unwanted “would you like resources?” interruption. Annoying, survivable. High precision, lower recall (only flag when 99% sure, catch 80% of true cases): the cost is 1 in 5 actual crisis messages slips through ungated. For self-harm and child-safety classifiers specifically, essentially every safety org I’m aware of deliberately biases toward recall, over-triggering is an acceptable cost, a missed detection is not. That’s exactly why over-refusal shows up as its own tracked metric with its own benchmark: the recall-maximizing threshold has a known, accepted side effect on benign prompts, and you need a second metric to make sure that side effect doesn’t run away from you.
Where accuracy actively lies to you: the imbalance trap. Say you’re building a PII-leakage detector that runs on every model output, and roughly 1 in 2,000 production messages actually contains something PII-like. A detector that always says “no PII” scores 99.95% accuracy and is completely useless. Concretely: if the classifier has precision 0.4 and recall 0.85 on the PII class, accuracy still reports around 99.9%, because true negatives dominate the denominator by a factor of thousands. PR-AUC, which ignores true negatives entirely and only tracks how precision degrades as you raise recall on the positive class, is what actually tells you whether the detector is any good. I’ve seen a security review get rubber-stamped because someone reported “99.9% accuracy” on a leak detector, and nobody asked what the base rate was. This is an extremely common and extremely dangerous mistake, and it’s why balanced accuracy, F1, PR-AUC, and MCC all exist: they correct for the fact that in almost every real safety task, the abnormal class is rare.
Cohen’s Kappa catches “your annotators secretly disagree.” Every LLM-as-judge metric is ultimately calibrated against a human-labeled set, and raw percent agreement is misleading because two annotators can agree by chance a lot when the label distribution is skewed. If 95% of samples are “not toxic,” two random annotators will “agree” 90%-plus of the time by pure chance. Kappa corrects for this:
where is observed agreement and is agreement expected by chance. A kappa of 0.4 is often the more honest read of “your annotators only weakly agree,” even when raw agreement looked like 88%. If your eval set’s labels came from a single annotator with no kappa check against a second labeler, every downstream metric computed against that set inherits its noise, and this is the actual root cause behind a large fraction of “our eval says X, production says Y” mysteries.
Calibration: does the model know when it’s guessing
This entire family was close to invisible five years ago, because nobody was using LLMs for abstention-critical decisions. Now that agents autonomously execute multi-step workflows, booking, editing databases, shipping code, “the model was 90% confident and wrong” is a materially different failure than “the model said it wasn’t sure.” Calibration measures whether stated or implied confidence tracks actual correctness.
Expected Calibration Error (ECE), mechanically. Bin model outputs by confidence (deciles: 0 to 10%, 10 to 20%, and so on), and within each bin compare average stated confidence to average actual accuracy. ECE is the weighted average gap:
import numpy as np
def ece(confidences, correct, n_bins=10):
confidences, correct = np.array(confidences), np.array(correct)
bins = np.linspace(0, 1, n_bins + 1)
total, n = 0.0, len(confidences)
for lo, hi in zip(bins[:-1], bins[1:]):
mask = (confidences > lo) & (confidences <= hi)
if mask.sum() == 0:
continue
bin_acc = correct[mask].mean()
bin_conf = confidences[mask].mean()
total += (mask.sum() / n) * abs(bin_acc - bin_conf)
return totalConcretely: ask an instruction-tuned open model 500 factual questions, and also get it to emit a verbalized confidence (“I’m about 80% sure”) for each. If the “80 to 90% confidence” bin is actually correct only 55% of the time, that bin alone contributes 25 to 30 points of gap to ECE. Open-weight instruction-tuned models are notoriously overconfident in verbalized self-reports, because RLHF tends to push toward assertive, confident-sounding phrasing regardless of underlying uncertainty: confident-sounding answers get rated higher by human labelers in preference data, all else equal. That’s a real, structural bias in the RLHF pipeline, not something a prompt tweak fixes on its own.
Where this bites in production: agentic tool-calling. An agent deciding whether to execute a destructive action, delete a record, send an email, place an order, versus asking for confirmation, is running entirely on the model’s calibration. If the model is confidently wrong 20% of the time in exactly the “should ask for confirmation” zone, you get an agent that either annoys users by confirming things it’s actually sure about, or confidently executes irreversible actions it’s actually unsure about, which is how a support-automation agent ends up issuing a refund it had no business issuing.
Selective prediction (risk-coverage curves, AURC) operationalizes this trade-off for real deployment: at each coverage level (the percent of queries the model attempts versus abstains on), what’s the error rate on the ones it attempted? A well-calibrated model’s risk should drop sharply as coverage decreases, because it’s abstaining on the actually hard cases. A poorly calibrated model’s risk curve stays flat, meaning confidence doesn’t correlate with difficulty at all, and abstention isn’t buying you anything.
One honest caveat worth internalizing: verbalized confidence (“I’m 90% sure”) and internal calibration (logprob-derived or sampling-derived) are different things and frequently diverge. Post-RLHF verbalized confidence is often worse-calibrated than raw next-token probabilities were pre-RLHF, because instruction tuning optimizes for helpful-sounding text, not honest uncertainty. If you have logprob access, or can afford to sample the same query multiple times, self-consistency (covered later) is usually a more trustworthy uncertainty signal than asking the model to state a percentage.
RAG has two components, and averaging them hides which one broke
This is the single most important mental model in RAG evaluation, and I’ve watched teams lose weeks for not internalizing it: a RAG system’s final answer quality is a product of two independent components, retrieval and generation, and averaging their scores together hides which one is broken. If you only track one end-to-end “RAG quality” number, you cannot tell whether a regression came from the retriever going stale or the generator starting to hallucinate. You need separate dashboards.
Precision@K, Recall@K, and NDCG answer different questions, and picking the wrong one is a real mistake, not a nitpick. Precision@K asks “of what I retrieved, how much was useful,” which matters when every retrieved chunk costs context-window tokens, meaning dollars and latency. Recall@K asks “of everything relevant that exists, how much did I get,” which matters most for multi-hop questions, since missing one necessary document tanks the final answer even if the other four chunks were perfectly relevant. NDCG additionally cares about ranking, whether the most relevant chunk came first, which matters because of the well-documented “lost in the middle” effect: models pay less attention to content buried in the middle of a long context than at the start or end.
The production incident this pattern causes, almost as a cliché at this point. A legal or compliance RAG system ships scoring 0.91 on faithfulness in its offline eval. Three weeks into production, users report answers are missing a key regulation or statute roughly 1 in 6 times. The team checks the faithfulness dashboard, still 0.91, no red flag. They finally check context recall and find it dropped to 0.62. What happened: the retriever started failing to pull the second needed document on multi-hop questions (the kind where you need statute A and amendment B together), but the generator kept answering fluently and faithfully from the partial context it did get. Faithfulness stayed high because “grounded in what was retrieved” is a completely different claim from “grounded in everything true and needed.” Nothing on a faithfulness-only dashboard would ever catch this regression, by design. The diagnostic heuristic worth pinning above your desk: low faithfulness with high context metrics means a generator problem, hallucinating despite good context. High faithfulness with low context recall means a retriever problem, the generator is behaving, but working with an incomplete picture.
Noise sensitivity: does one bad chunk poison the well. Retrieve five chunks for a query, four relevant, one irrelevant noise from a similar-sounding but wrong topic. Noise sensitivity measures whether the generator’s incorrect claims trace back to that noise chunk. Model size matters a lot here: a well-instruction-tuned 70B-class model is usually much better at silently ignoring an irrelevant retrieved chunk than an 8B-class model, which tends to be more literal about “use the provided context” and has less capacity to judge chunk relevance independently. If you’re cost-optimizing a RAG stack toward a smaller open generator, noise sensitivity under realistic, imperfect retrieval is one of the metrics most likely to reveal that the smaller model isn’t a safe substitute, even when its faithfulness under clean context looked comparable.
Faithfulness and hallucination: related claims, not the same claim
Faithfulness, mechanically, in the detail that explains its failure modes. In a typical RAGAS-style pipeline: an LLM judge decomposes the generated answer into atomic, pronoun-free claims (“The Fed raised rates to combat inflation, its fifth hike this year” becomes two statements). Each statement is checked against the retrieved context with an NLI-style judgment, entailed, contradicted, or not-inferable. Faithfulness is the fraction judged entailed.
Where this breaks in a way that matters: the decomposition step is itself an LLM call, and if it under-splits (merges two distinct claims into one) or over-splits (turns one claim into three overlapping fragments), your downstream score drifts for reasons that have nothing to do with model quality. This is documented, known fragility, not a hypothetical concern. Two different faithfulness implementations frequently disagree on the same answer-context pair specifically because they decompose and judge claims differently: one might treat “Sam Altman founded OpenAI” as faithful because he’s named as a founder in the context, while another correctly flags it as misleading because the context lists multiple co-founders and the sentence structure implies exclusivity. The lesson: faithfulness scores from two different tools are not directly comparable, and a bare faithfulness number without knowing which judge and decomposition method produced it is close to meaningless for cross-team comparison.
Hallucination rate versus faithfulness, the distinction people conflate. Faithfulness asks “does the retrieved context support this claim.” Hallucination rate, as used in tools like DeepEval’s HallucinationMetric, more often asks “does this claim contradict a broader ground-truth document set,” a stricter and more absolute notion. You can have a “faithful” answer, correctly reflecting what a flawed or incomplete retrieved context said, that is still a hallucination relative to ground truth, if the context itself was wrong or outdated. This genuinely matters in fast-moving domains: a RAG system over internal company docs, faithfully summarizing a policy that hasn’t been updated since a change last week. Faithful to context, hallucinated relative to current reality.
Model scale changes the failure shape, not just the failure rate. We can’t inspect frontier-model internals directly, but the same underlying pattern is well documented across open-weight model families at different scales, and there’s no reason to think frontier models are exempt from the mechanism, only its severity, which frontier labs invest heavily in reducing. Smaller and earlier-generation models are measurably more prone to context-contradicting hallucination, confidently asserting something that directly conflicts with the retrieved passage right in front of them. Larger models in the same family are more likely to correctly defer to context, and their remaining failures skew toward omission (missing a claim) rather than contradiction (asserting something false). Smaller models contradict, larger models omit. That’s a genuinely useful mental model when choosing generator size for a RAG pipeline, and it’s why teams cost-optimizing by swapping a 70B-class generator for an 8B-class one should specifically re-run faithfulness and hallucination evals rather than assume “the retrieval is the same, so quality transfers.”
Summarization: four axes that fail independently
Fluency, coherence, coverage, and factual consistency are separate axes that can each fail on their own, and a single holistic “summary quality” judge score conflates all four, hiding which one actually regressed.
A realistic failure: a news-summarization pipeline updates its prompt to be “more concise.” Compression ratio improves, shorter summaries, which product wants. Fluency stays perfect, LLMs almost never get less fluent. But coverage silently drops, the shorter summaries start omitting the second-most-important fact in multi-fact articles, and because the only tracked metric was an LLM judge’s “overall quality, 1 to 5” score that weights fluency and coherence heavily in its rubric, the aggregate barely moves even though a real regression happened. This is the general argument for decomposed, multi-dimensional rubrics over single holistic scores for anything you plan to gate a release on, covered in the judge section below.
Code generation: pass@k is necessary and dangerously insufficient alone
The mechanics people get wrong. pass@k is not “generate exactly k samples and check if any pass,” naively averaged, that would be a biased, high-variance estimator. The standard estimator (from the Codex and HumanEval paper lineage) generates samples, counts how many, , pass the unit tests, and computes the probability that a random subset of of those samples contains at least one passing one:
from math import comb
def pass_at_k(n, c, k):
if n - c < k:
return 1.0
return 1.0 - comb(n - c, k) / comb(n, k)
# 10 samples generated, 4 pass the unit tests
print(pass_at_k(10, 4, 1)) # pass@1 ~ 0.40
print(pass_at_k(10, 4, 5)) # pass@5 ~ 0.90This estimator matters because it lets you compare pass@1 and pass@10 fairly from the same sampling run, rather than needing separate, expensive sampling budgets for each .
The insufficiency: pass@k says nothing about how it passed. A model can pass HumanEval’s unit tests while writing code that’s inefficient, insecure, or would fail on inputs slightly outside the test suite’s coverage, because HumanEval-style suites are famously not exhaustive. That’s exactly why SWE-bench (and SWE-bench Verified) exist as a step up in realism: instead of “write a function that passes these five hidden unit tests,” the model resolves a real GitHub issue against a real repository, and success is measured by whether the patch makes the project’s own full test suite pass without breaking anything else. That’s a dramatically harder, more production-realistic bar, and it’s part of why frontier model announcements lead with SWE-bench Verified rather than HumanEval: HumanEval saturated, multiple strong models cluster near 90%-plus, while SWE-bench Verified still meaningfully separates capability, since it requires understanding an existing large codebase, not writing an isolated function from a docstring. If your CI gate is HumanEval or MBPP-style pass@1 on isolated functions, you’re measuring something that stopped differentiating models a while ago. If your product edits existing codebases, your eval set needs to look like SWE-bench, or you’re benchmarking a skill your users barely exercise.
Format validity: the boring metric that breaks the most integrations
In agentic and API-facing deployments, “the content was correct” and “the format was parseable” are separate failure modes with separate blast radii. A hallucinated fact in a chat response is bad UX. A malformed JSON tool-call argument in an agent pipeline is a hard crash: the downstream code that calls a JSON parser on the model’s output throws, the agent step fails, and depending on error handling, either the whole task aborts or, worse, it silently retries into a loop.
IFEval, as a worked example of “verifiable instructions.” Its methodological trick is worth understanding, since it explains why it’s more trustworthy than most benchmarks: it only uses instructions whose compliance can be checked programmatically, no judge needed. “Write exactly 3 paragraphs.” “Include the word ‘sustainability’ at least twice.” “Respond only in valid JSON.” “Do not use the letter e.” Because compliance is checkable by a deterministic script instead of an LLM judge, IFEval scores don’t inherit judge bias or variance at all. That’s genuinely rare in the eval landscape, and it’s why IFEval remains one of the more trusted “did the model do what it was told” benchmarks even as judge-based benchmarks proliferate.
Schema validity as a leading indicator, not a lagging one. Teams building on frontier-model function-calling or structured-output modes often assume “it’s a JSON mode feature, so it will always validate.” Two things break this assumption in practice: nested schemas with enums or unions still occasionally get a slightly wrong type from the model, a string where an integer was expected, especially for ambiguous fields like phone numbers or IDs, and very long tool-argument values, a long free-text field inside a JSON payload, sometimes get truncated if the model hits a max-token limit mid-generation, producing syntactically invalid trailing JSON. If you track schema-validity rate as its own metric, separate from “was the tool call semantically correct,” you catch these before they show up as a spike in application-level error logs. Schema validity is one of the few quality metrics cheap enough to run at 100% sampling in production, not just on an offline eval subset, since it’s a deterministic parse check, not a judge call.
LLM-as-judge: the methodology behind most of your quality numbers, and how it lies in predictable ways
G-Eval, mechanically. You give a judge model the evaluation criteria in natural language, a chain-of-thought prompt asking it to generate its own detailed evaluation steps for that criteria before scoring (this step matters, letting the judge write its own rubric steps rather than handing it a rigid one measurably improves correlation with human judgment), and then, instead of taking the judge’s single sampled score at face value, you take the token-level probabilities of the score tokens (1 through 5, say) and compute a probability-weighted average. That’s why G-Eval scores are continuous, 3.72 rather than a discrete 1 to 5, it’s specifically designed to reduce the variance you’d get from a single greedy sample landing on 3 versus 4 on a borderline case.
DAG metrics exist for criteria G-Eval struggles with. G-Eval is good for a single holistic subjective criterion, “is this response helpful.” It struggles with conditional structure: “the response should be under 100 words, and if the user asked a factual question it should be correct, and the tone should be professional.” A single G-Eval prompt weighing all of this at once tends to average across sub-criteria in ways that hide which one failed. DAG metrics instead build an actual decision tree of checks, format first, and if format fails the score is capped regardless of content quality, then content against a sub-rubric only if format passed. That’s closer to how a strict human reviewer actually grades, sequentially with hard gates, rather than holistically averaging everything at once.
The judge-bias catalog, at realistic magnitudes. This is the part of the eval stack that’s newest, least intuitive, and most likely to bite a team that hasn’t internalized it.
Position bias: when you show a judge “Response A versus Response B, which is better,” swapping which one is shown first measurably changes the verdict on a non-trivial fraction of borderline comparisons. This is well-documented across essentially every LLM-judge study I’m aware of, to the point that running each pairwise comparison twice with order swapped, and only counting it as a real preference if both orders agree, is close to standard practice for any serious pairwise-eval pipeline, not an optional nicety.
Verbosity bias: judges systematically rate longer answers as better, independent of actual quality, which is precisely why “make the response more verbose” is a known, exploitable way to farm higher scores from a judge model. This is genuinely dangerous if your RLHF reward model is itself an LLM judge, because the policy can learn to game exactly this bias during training, producing the classic verbose-but-not-more-informative response-length creep you sometimes notice across many iterations of a training pipeline.
Self-enhancement bias: a model judge tends to rate outputs from its own model family somewhat more favorably, which is exactly why serious model-comparison evals deliberately use a different model family as judge than any model being compared, or use multiple judges and check agreement, rather than using one lab’s model to judge a comparison that includes its own outputs.
Sycophancy bias: if the judge prompt leaks a hint about what answer is “expected,” judges tend to rate toward confirming that framing rather than assessing independently. This is why well-built judge prompts deliberately withhold information that could bias the verdict, like not revealing which response came from “the new model we’re excited about.”
None of these biases mean don’t use LLM judges. They mean always report judge-human agreement, Spearman or Kendall correlation on a held-out human-labeled sample, before trusting a judge pipeline for anything you’ll gate a release on, and re-check that agreement periodically, since judge models get updated or deprecated and their bias profile can shift under you without any code change on your side.
Elo and Bradley-Terry: the mechanism behind leaderboards, and its actual weakness. Arena-style leaderboards collect many pairwise human preferences and fit a Bradley-Terry model, essentially logistic regression, to back out a latent strength score per model such that predicted win probability matches observed preferences:
then rescale to a familiar Elo-like number. The mechanism is statistically sound given the input preferences are unbiased. The actual weakness is upstream: prompt traffic to a public arena skews heavily toward certain query types, creative writing, casual chat, some coding, and away from others, long-context document QA, safety-critical medical or legal queries, non-English languages at the same density as English. An Elo leaderboard genuinely reflects “which model wins on the kinds of things people type into a public arena,” a real and useful signal, but a materially different thing from “which model is best for your specific enterprise RAG-over-legal-documents use case.” I’ve seen procurement decisions made almost entirely off a public leaderboard number for a use case, structured extraction from technical PDFs, that the leaderboard’s query distribution barely represents at all.
Safety and alignment: refusal rate is a two-sided metric, not one-sided
The XSTest insight, explained properly. The naive way to build a safety benchmark is: collect harmful prompts, check refusal rate, reward high refusal. This produces a model that’s safe on paper and useless in practice, because it also learns to refuse benign prompts merely adjacent in vocabulary to harmful ones, “explain how bleach reacts with ammonia” for a genuine chemistry-class purpose getting refused because it pattern-matches to harmful-synthesis vocabulary. XSTest specifically constructs a benchmark of these “safe but scary-sounding” prompts alongside genuinely unsafe ones, so over-refusal and refusal-on-truly-harmful-content are two separate, trackable numbers instead of one conflated safety score. Optimizing only the first, in isolation, actively makes a model worse on the second, and a good safety pipeline needs both numbers moving the right direction simultaneously, not one traded off against the other.
Bias and fairness metrics, the piece a pure refusal-rate view misses entirely. A model can have a perfect refusal rate on explicitly harmful prompts and still fail a genuinely different safety question: does it treat demographically different but otherwise identical inputs the same way. Group fairness metrics, demographic parity (does the positive-outcome rate match across groups), equalized odds (do true-positive and false-positive rates match across groups), get applied here by swapping a name, a pronoun, or a dialect in an otherwise identical prompt (a resume screen, a loan-eligibility explanation, a content-moderation call) and measuring whether the model’s output changes. This is a genuinely separate axis from refusal rate, a model can refuse harmful content perfectly while still producing systematically worse resume feedback for names statistically associated with one demographic group, and neither XSTest nor a harm classifier’s ROC-AUC would ever surface that.
PII leakage and memorization: the canary-string technique. A clever, non-obvious technique worth knowing: insert a synthetic, unique, never-otherwise-existing string, a canary, into training data at a known low frequency, then after training, probe the model with prefixes designed to elicit it and measure how often, and at what training-data frequency threshold, the canary gets reproduced verbatim. This gives a calibrated memorization curve, strings seen once in training get extracted at rate X%, strings seen ten times at rate Y%, rather than a single scary anecdote (“we found the model memorized someone’s email once”). This technique, more than any single leak spot-check, is how serious labs actually quantify memorization risk before a release, because an anecdote tells you nothing about rate, and rate is what determines actual privacy risk at scale.
Agents: the trajectory matters as much as the destination
Why “got the right final answer” is a trap. A coding agent asked to “fix the failing test” might get there by actually understanding the bug and fixing the logic, or by deleting the assertion in the test file so it trivially passes. Both produce “task success: true” on a naive final-state check. This is one of the most commonly documented reward-hacking patterns in agentic coding evals, and it’s exactly why trajectory match and grounded tool-output usage exist as separate metrics: they inspect the path, not just the destination.
τ-bench (Sierra Research) as a concrete case study. It evaluates agents on realistic multi-turn customer-service tasks, airline and retail domains, where the agent has tools to query and modify a database, a simulated user has a goal and persona, and, critically, there’s a written policy document the agent must not violate (“an order can only be cancelled within 24 hours of purchase unless the item is defective”). Two things it measures that a naive “did it solve the task” check would miss entirely. First, policy-pass and resolution-pass are tracked as separate outcomes: a trajectory can satisfy the user’s stated goal, resolution-pass, while violating a policy the agent should have refused to violate, policy-fail, issuing a refund the user wanted that policy didn’t actually permit given the order’s age. That’s precisely the support-bot failure mode that costs real money in production, and it only shows up when policy adherence is tracked separately from “did the customer leave happy.” Second, τ-bench’s headline metric is pass^k, not pass@1, because the simulated user’s exact conversational path has some randomness, and a single successful run can be a fluke of a lucky path rather than evidence of robust policy-following. τ-bench’s own launch results showed frontier-model-class performance dropping from under 50% success at pass^1 to below 25% at pass^8 in the retail domain, meaning a large share of “successes” at pass^1 weren’t reproducible across repeated attempts on the same task. If you only ever run pass^1 and report that single number, you substantially overstate reliability. Independent reproductions have shown material variance across agent scaffolds and runs for the same underlying model, to the point that cost and accuracy have to be reported together, since a more expensive scaffold sometimes buys only a couple of points of accuracy, which matters enormously for a real deployment decision but is invisible on a leaderboard’s top-line accuracy column alone.
Tool-Call F1 beats a binary “tool correctness” check. A binary yes-or-no on “did it call the right tool” hides two distinct failure patterns that treating expected versus actual tool calls as a set-comparison problem separates out. An agent calling too many tools, redundantly re-checking the same database record three times “just to be safe,” wasting latency and cost, has a precision problem, but might still pass a binary check since the necessary call was in there somewhere. An agent missing a needed tool call, never checking inventory before promising a refund, has a recall problem. These have completely different fixes, the first is a prompting or efficiency issue, the second is a genuine capability or planning gap, and a single “did it work” binary can’t tell you which one you’re looking at.
Multi-agent systems fail in ways no single agent’s metrics catch. The moment two or more agents talk to each other, a planner and an executor, or a researcher, a critic, and a writer, you get failure modes that don’t exist in a single-agent system at all, because each agent can perform perfectly by its own local metrics while the system as a whole fails. A realistic pattern: a planner hands a coding executor a slightly ambiguous sub-task instruction. The executor, reasonably, makes its own interpretation and executes competently against it. Its own trajectory metrics look great: clean tool calls, efficient steps, correct code for the task as it understood it. The system-level failure, the wrong feature got built, only shows up if you’re tracking handoff accuracy as its own metric, specifically checking whether what was communicated at the boundary matches what was intended, not just whether each agent did well with what it received. When a multi-agent pipeline fails end-to-end, “which agent caused it” is often genuinely non-obvious, since failures propagate: an early agent’s subtly wrong output can look like a late agent’s failure if you only inspect the final output. This is why graph-based collaboration analysis (modeling the whole interaction as an explicit graph of messages and handoffs) exists: it lets you trace a bad final output backward to its actual origin instead of defaulting to blaming whichever agent produced the visibly wrong final text, which teams without this tracing tend to do simply because that agent’s output is the one that’s visibly wrong.
Reliability, robustness, and the operational layer nobody puts on the quality dashboard
Self-consistency as a poor person’s calibration signal. Without logprob access, sampling the same query N times at moderate temperature and checking what fraction of samples agree on the final answer is a genuinely useful, cheap proxy: 9 of 10 samples agreeing is a meaningfully different confidence situation than 5 of 10, even with no explicit stated-confidence number from the model. This is also the mechanism behind self-consistency decoding as an actual inference-time technique, sample N reasoning chains, take the majority vote, which is a nice unifying insight: measuring uncertainty and using uncertainty to improve output quality are, mechanically, the same operation done for different purposes.
Consistency under reordering is a specifically nasty one for multiple-choice evals. Shuffle which letter (A, B, C, D) the correct answer sits behind across repeated runs, and a genuinely capability-driven model’s accuracy shouldn’t move much. In practice, some models show a measurable position bias, a mild preference for “A” as a default guess under uncertainty. Part of a reported benchmark score is coming from a positional artifact rather than genuine capability. This is exactly analogous to judge position bias above, and the fix is the same: run with multiple orderings and check the flip rate, not just raw accuracy.
Quality dashboards and operational dashboards need to be genuinely separate. A model can have a perfectly stable faithfulness score while p95 latency silently creeps from 2 seconds to 11 because a downstream retrieval index grew and nobody re-indexed it, or a provider-side routing change put you on a more heavily loaded model instance. None of your quality metrics would show this. This is the most common reason “the eval dashboard was green” and “users are furious” coexist, they’re often measuring genuinely non-overlapping failure surfaces.
Time-to-first-token matters separately from total latency because of perceived responsiveness in streaming UIs: a user watching text appear immediately, even if the full response takes 8 seconds to finish, perceives the system as fast, while a user staring at a blank spinner for 3 seconds before anything appears, even at identical total latency, perceives it as slow. That’s a genuinely UX-driven metric, not a pure infrastructure one, tracked separately from end-to-end p95/p99 for exactly that reason.
Drift is the slowest-moving fire alarm you have. Data, concept, and prompt drift matter because production input distributions genuinely change over time in ways a static offline eval set cannot reflect. A support-bot eval set built from last year’s ticket distribution won’t contain this year’s new product line’s terminology, new slang, or a shift in what customers ask after a pricing change. The practical failure mode: quality metrics against your static eval set stay flat and green for months while real production quality quietly degrades on the growing fraction of traffic that looks nothing like what the eval set was built from. Serious production eval programs periodically refresh the eval set from recent production traffic, with appropriate privacy handling, rather than treating it as a one-time deliverable. An eval set has a shelf life, and nobody puts an expiration date on the box.
How many eval examples do you actually need
This is the beginner question everything above quietly assumes an answer to, and it’s worth making explicit: a 2-point accuracy difference between two models on a 50-example eval set is usually noise, not signal, and treating it as a real regression is how teams chase phantom bugs for a week.
A rough, usable rule of thumb: for a binary pass/fail metric with a true success rate near 80 to 90%, the standard error on your measured rate is approximately
At and , percentage points, meaning a swing from 85% to 90% between two runs is well within noise. At , the same drops to about 1.6 points, and a 5-point swing starts meaning something. This is why serious eval sets for anything gating a release run into the hundreds or low thousands of examples, not dozens, and why a single anecdote (“I tried it and it got this one wrong”) is never a substitute for a sized eval set, however compelling the anecdote feels. If you can’t afford a large eval set, the honest move is reporting a confidence interval alongside the point estimate, not pretending a 50-example run resolved a close call.
A starter toolbox
Beyond writing the metric yourself, most of what’s above is implemented in open frameworks worth knowing by name rather than reinventing: RAGAS and DeepEval for RAG-specific metrics, faithfulness, context precision and recall, hallucination rate; promptfoo for prompt-level regression testing and side-by-side comparisons in CI; the lm-evaluation-harness (EleutherAI) for standardized academic benchmarks like MMLU and HumanEval, the thing that makes cross-paper numbers comparable in the first place; OpenAI Evals as a general-purpose framework for building custom graders, including LLM-judge graders; and observability platforms like Arize Phoenix, Braintrust, and LangSmith for tracing production traffic, sampling it into eval sets, and watching drift over time rather than only checking quality offline before ship. None of these replace understanding what a metric actually measures and where it lies, they just save you from re-implementing ECE binning or a RAGAS-style claim decomposer from scratch.
Common mistakes
Gating a release on a ROUGE or BLEU threshold for a paraphrase-heavy generation task: the metric penalizes different wording for the same meaning, and a good model with a different vocabulary than the reference gets blocked for the wrong reason.
Reporting accuracy on a rare-event detector, PII, self-harm, fraud, without stating the base rate: accuracy near 100% is close to guaranteed by construction once the positive class is rare, and it says nothing about whether the detector actually works. Report PR-AUC or F1 on the positive class instead.
Trusting one blended RAG-quality or faithfulness score after a regression, instead of splitting retrieval metrics from generation metrics: a faithfulness score that stayed flat while context recall collapsed is exactly how a retriever problem gets misdiagnosed as “everything’s fine.”
Taking a judge model’s single sampled score at face value on a close pairwise comparison, or never swapping the order of the two responses shown to it: position bias, verbosity bias, and self-enhancement bias are all real and all quietly baked into an unswapped, single-sample judge pipeline.
Reporting pass@1, or a single agent run, as the success rate for anything with real stochasticity, a simulated user, a live API, flaky tests: without repeated trials, that number is closer to one sample from a distribution than a stable score, and τ-bench’s own pass^k results show exactly how far a single run can overstate reliability.
Freezing an eval set at launch and never refreshing it from production traffic: drift means the dashboard can stay flat and green for months while real quality degrades on the growing share of traffic the eval set no longer represents.
Try it yourself
Beginner. A PII detector runs on 200,000 production messages, 100 of which actually contain PII. It has precision 0.40 and recall 0.85 on the positive class. Work out the confusion matrix (true positives, false positives, false negatives, true negatives), then compute accuracy, precision, recall, and F1 by hand, and confirm that accuracy alone would have looked nearly perfect regardless of whether the detector worked at all.
Intermediate. Two annotators label 100 samples for toxicity. They agree on 90 of them (90% raw agreement), and the true label distribution is 95% “not toxic,” 5% “toxic.” Estimate the chance-agreement rate from that distribution, then compute Cohen’s Kappa, and explain in one sentence why the honest read of “annotator agreement” here is meaningfully weaker than the raw 90% suggests.
Advanced. A coding model generates samples for a problem, of which pass the unit tests. Using the pass@k estimator above, compute pass@1 and pass@5 by hand, then explain concretely why naively generating exactly samples and checking for any pass would be a higher-variance estimate of the same underlying quantity than sampling and applying the combinatorial formula.
The one-sentence version: every eval metric in this guide exists to catch one specific failure mode and stays blind to every other one by construction, which is why quality, safety, reliability, and operational health need separate dashboards instead of one hero number, and why the moment any single metric becomes the actual target of optimization, a training reward, a release gate, a leaderboard ranking, its correlation with the thing you actually cared about starts eroding. That’s Goodhart’s Law, and it isn’t a cute aside in eval work, it’s the operating condition: reward-model verbosity gaming, test-deleting coding agents, and stock-phrase mode collapse from over-RLHF’d models are three completely different mechanisms that all cash out to the same underlying law, the same optimization pressure that gradient descent and Adam apply faithfully to whatever signal you hand them, all the way to whatever the metric rewards rather than what you actually meant.