TL;DR: A new attention system, Adaptive Learnable Hierarchical Routing (ALHR), claims O(NlogN) complexity while maintaining high accuracy for long contexts, offering significant memory and VRAM scaling improvements.
Summary: ALHR is a static binary tree-based attention system that uses learnable functions to reduce the number of keys processed. This approach aims to decrease memory usage and improve VRAM scaling with increased token counts, reportedly retaining 97% accuracy over long contexts. The developer has tested it with Tiny Stories, showing promising early results.
Why it matters: This development could enable more efficient processing of very long sequences in AI models, reducing computational costs and VRAM requirements. AI builders should monitor its progress for potential applications in large language models and other sequence-to-sequence tasks.
Source: reddit