TL;DR: A budget-matched head-to-head at ~15M params found accuracy indistinguishable between a gated linear state-space backbone and standard attention, but the linear model was 2.37× faster at 8k context and roughly 6–7× faster to fine-tune.
Summary: An independent developer trained two decision-only (non-generative) models of ~15.09M parameters each — one using a Gated Segmented State Space linear recurrent backbone, the other standard Transformer attention — with an identical decision head, on a 2k–8k character retrieval-decision task (2000 train / 500 val / 500 test). Test accuracy was a coin flip at that scale: 26.8% for the linear model vs 28.2% for the Transformer, against a 25% chance baseline. The measurable differences were efficiency: at 8192 tokens the linear model ran 2.37× faster (108.6 vs 257.7 ms), and fine-tuning took about 30 minutes for six epochs versus 40–45 minutes per epoch for the Transformer. The author notes that earlier, shorter decision tasks in the same repo had the linear model winning outright on accuracy (50.4 vs 46.2 and 66.2 vs 61.0).
Why it matters: It is a small but concrete data point that at tiny parameter counts linear/SSM backbones buy large long-context latency and training-cost wins without a clear accuracy penalty — and may actually help on short decision tasks. Worth reproducing the repo's notebooks on your own decision-head workloads before defaulting to attention at this scale, while treating the accuracy result as unresolved rather than settled.
Source: reddit