TL;DR: llama.cpp updated its MoE implementation to use per-expert-token scaling for down projections, enhancing efficiency and enabling tensor parallelism.
Summary: The latest llama.cpp release (b11785) introduces a significant improvement for Mixture-of-Experts (MoE) models. It replaces global amax rescaling with a per-expert-token factor for MoE down projections, computed via a POOL_1D max reduction. This change also enables tensor parallelism by handling POOL_1D in Meta split-state logic.
Why it matters: This optimization can lead to more efficient inference for MoE models on consumer hardware, potentially reducing memory usage and increasing speed. AI builders should explore how this impacts performance for MoE models running on llama.cpp, especially for local deployments.
Source: github_releases