TL;DR: llama.cpp now supports up to 256 single-token option markers for 'Granite switch decision' models, improving model output probability distribution.
Summary: A recent update to ggerganov/llama.cpp (b11707) introduces support for 256 single-token, semantic option markers for 'Granite switch decision' models. This feature enhances the model's ability to identify and map a wider range of unique single-token sequences to their respective logits for output probability distribution. The discovery of these model-specific single-token markers is performed during the conversion process.
Why it matters: This technical improvement allows for more nuanced and precise control over model outputs, especially for tasks requiring discrete choices. AI builders working with llama.cpp or similar local inference engines should explore how this expanded option space can be leveraged for more complex decision-making or structured output generation.
Source: github_releases