TL;DR: llama.cpp release b10701 fixes a critical bug where NVFP4 draft models failed to pass necessary scales, significantly improving speculative decoding performance.
Summary: The latest llama.cpp release, b10701, addresses a bug in DFlash2 NVFP4 draft models. Previously, Q, K, V, and output projection scales were not passed to attention operations, causing almost no speculative tokens to be accepted. This fix ensures proper scale passing, enhancing the efficiency of speculative decoding.
Why it matters: This fix is crucial for developers using NVFP4 models with llama.cpp, as it directly improves the performance and reliability of speculative decoding. Builders should update to b10701 to leverage these performance gains in their local LLM applications.
Source: github_releases