TL;DR: llama.cpp now supports Hadamard operations and improved quantization for Qualcomm Hexagon DSPs, enhancing on-device LLM performance.
Summary: The latest llama.cpp release (b11717) introduces support for Hadamard MUL_MAT and GET_ROW operations on Qualcomm Hexagon DSPs. This update specifically targets flat and tiled Q1_0 and Q2_0 quantization, alongside reducing packet counts in q8_1 quantizer and vec_dot_q1. These optimizations are designed to improve the efficiency of running LLMs on Hexagon architectures.
Why it matters: This significantly boosts the performance of llama.cpp on devices powered by Qualcomm Hexagon DSPs, enabling more efficient on-device AI. Developers should explore leveraging these new capabilities for deploying LLMs on mobile and edge devices with Hexagon processors.
Source: github_releases