TL;DR: A new post-training quantization (PTQ) method, ExTernD, achieves near arbitrary accuracy for ternary LLMs by decomposing matrices, offering significant efficiency gains.
Summary: ExTernD introduces an Expanded-Rank Ternary Decomposition method for post-training quantization (PTQ) of large language models. It decomposes matrices into two ternary matrices and an inner diagonal scaling matrix, allowing for an arbitrarily large inner rank. This approach enables ternary quantization to achieve accuracy levels comparable to any quantization scheme, while requiring only slightly more VRAM than current methods.
Why it matters: This breakthrough could make highly efficient, low-bit LLMs more practical for edge devices and resource-constrained environments. AI builders should investigate ExTernD for deploying powerful models with reduced memory footprints and faster inference.
Source: reddit