TL;DR: Ollama's latest release, v0.40.6, now supports the GGUF q2_0 tensor and file types, enhancing compatibility and efficiency for local LLM inference.
Summary: Ollama released version 0.40.6, which includes the addition of q2_0 tensor and file types for GGUF. This update, contributed by @jmorganca, improves the framework's ability to handle more quantized model formats. The GGUF format is a standard for storing and distributing large language models for local inference.
Why it matters: This update expands the range of quantized models that can be run efficiently on Ollama, potentially enabling more powerful models on resource-constrained hardware. AI builders should explore models utilizing the q2_0 quantization for improved performance and reduced memory footprint.
Source: github_releases