TL;DR: llama.cpp now supports Q2_0 quantization on Vulkan, improving performance for low-bit model inference on compatible GPUs.
Summary: The latest llama.cpp release (b10058) introduces support for Q2_0 quantization on Vulkan. This update, specifically PR #25430, enhances the efficiency of mat-vec-mul operations by optimizing rows per workgroup, leading to better performance compared to Q2_K in initial tests.
Why it matters: This enables more efficient execution of quantized LLMs on a wider range of hardware with Vulkan support, especially for low-power devices. Developers should explore Q2_0 for their Vulkan-enabled inference pipelines to potentially reduce memory footprint and increase speed.
Source: github_releases