Evaluation
Measuring whether any of it actually works.
3 logs in this box-
OpenAI's own April 2025 account of GPT-4o's sycophancy rollback says its offline evals looked fine and its A/B tests showed users liked the update, and it still had to be pulled four days after launch, which is the real, disclosed proof that a model can pass every gate you built and still fail the one thing you didn't think to measure
Sculley et al.'s real 2015 paper on hidden technical debt in ML systems, CACE, hidden feedback loops, undeclared consumers, as the reason production ML monitoring is structurally harder than ordinary software monitoring. The real deployment gradient, shadow mode, canary, A/B test, full rollout, and what specific failure class each stage exists to catch. OpenAI's real, disclosed April 2025 sycophancy incident worked through as a case study in exactly this gradient failing at the metric-selection step, not the process step. The Population Stability Index worked out by hand on a real, computed example (PSI = 0.203, landing in the real, standard moderate-drift band), and its real, disclosed blind spot: it detects that a distribution moved, not what kind of drift moved it.
-
GPT-3 really did use a bloom filter to catch contaminated eval data, Llama 2's own paper found 16% of MMLU had leaked into its training set, and running the actual bootstrap-CI gate on a real 200-example eval shows a genuinely improved model getting blocked as a false regression purely from sampling noise
The real N approx 20,000 power calculation verified from scratch, GPT-3's real 13-gram bloom-filter contamination check versus GPT-4's 40-gram bar versus 2025's backdoor-based DyePack, and an executed bootstrap-CI promotion gate run at n=200 versus n=20,000 on the identical true 1% improvement, showing the small eval invert the sign of a real, genuine gain into an apparent regression by chance alone.
-
Evaluating GenAI systems: why each metric exists, and exactly how it lies to you
The one test that actually matters for any eval number, worked through NLG quality, classification, calibration, RAG, hallucination, code generation, LLM-judges, safety, agents, and the operational layer, with the specific failure mode where each metric goes green while the product gets worse.