TL;DR: New llama.cpp release accelerates large-batch prompt processing of IQ-quantized models on AVX2 CPUs with batched GEMM kernels and vectorized IQ panel decode.
Summary: llama.cpp v10726 (PR #27402) adds a batched GEMM kernel (ggml_gemm_iqp_8x8_q8_K_p4) for grid IQ quantizations, improving large-batch prompt processing on AVX2 hardware. It also vectorizes IQ panel decode, lowers the speedup threshold, and includes NUMA fallback plus expanded IQP test coverage. This is an incremental performance-focused release for CPU inference.
Why it matters: For AI builders running IQ-quantized models locally, this is a free throughput win on consumer CPUs. Benchmark b10726 with large batch sizes to see the speedup before upgrading your deployment.
Source: github_releases