TL;DR: llama.cpp now supports Q2_0 quantization on Apple Metal GPUs, enabling even smaller and faster local LLM inference on Apple hardware.
Summary: The latest llama.cpp release (b9994) introduces Q2_0 quantization support for Metal. This enhancement allows for more efficient processing of large language models on Apple's Metal-enabled devices, including macOS and iOS, by reducing model size and memory footprint.
Why it matters: AI developers targeting Apple ecosystems can now run LLMs with significantly reduced memory and potentially improved speed using Q2_0 quantization. Experiment with this new quantization level to optimize your on-device AI applications for Apple users.
Source: github_releases