TL;DR: llama.cpp merged native inference support for Muse Spark Open, a latent-space MoE model handling text, vision, and audio in one architecture.
Summary: A llama.cpp PR (b11671) adds the Muse Spark Open architecture, a latent-space MoE where the residual stream stays at 8192 dimensions while 256 experts operate in 4096 with down/up projections and top-8 sigmoid routing plus a correction bias. Attention interleaves three sliding-window layers at 2048 tokens with one global no-rope layer, using a sigmoid output gate and weightless QK-norm; vision reuses the Muse Glimmer tower and audio uses a four-stage causal convolutional pyramid with one embedding per 80ms of 24kHz. The quantizer keeps latent projections and the attention gate at Q8_0 (just over 1% of weights) to avoid repetition-loop degradation.
Why it matters: Multimodal MoE models with unusual latent-space routing now run in llama.cpp, so builders can test text/vision/audio inference in a single local stack. Watch for GGUF quant releases and check whether low-bit quantizing of the projection layers degrades output.
Source: github_releases