TL;DR: A new preprint suggests that simple sliding-window attention (SWA) with sinks significantly outperforms linear attention variants on long-context reasoning benchmarks, offering a more efficient alternative for LLMs.
Summary: A recent arXiv preprint by Alexia Jolicoeur-Martineau et al. claims that sliding-window attention (SWA) with sinks, a straightforward solution to the quadratic cost problem in LLMs, achieves 2 to 10 times higher performance than linear attention on long-context reasoning tasks like Needle-in-a-Haystack and BABILong. The authors recommend switching to SWA, noting it requires no post-training, runs fast, and maintains low memory usage.
Why it matters: This finding challenges the current focus on post-training linear models, suggesting a simpler, more performant architectural choice for long-context LLMs. AI builders should investigate SWA as a potentially superior and more efficient attention mechanism for their models, especially when dealing with extended contexts.
Source: reddit