TL;DR: llama.cpp's server now intelligently saves only restorable slot states to its prompt cache, improving efficiency by preventing non-restorable task states from filling the cache.
Summary: The latest llama.cpp server update (b11642) optimizes prompt cache usage by saving only 'restorable' slot states. This change prevents states from tasks like reranking, embedding, decision, and infill from being stored in the cache, as these tasks do not restore from it. The system now tracks cache_pending for each slot to ensure only relevant completion task states are cached.
Why it matters: This optimization reduces memory footprint and improves performance for llama.cpp users running various server tasks, especially those with mixed workloads. Developers should observe more efficient prompt caching and potentially faster context switching in their local LLM deployments.
Source: github_releases