TL;DR: llama.cpp now reports compute graph reallocations during inference, providing deeper insights into memory management and performance.
Summary: ggerganov/llama.cpp introduced an optional ggml_backend_sched_realloc_callback to notify users when inference graph buffers are insufficient and require reallocation. This callback reports changes in backend assignments and graph size, with llama_context registering it to track expected and unexpected reallocations.
Why it matters: This feature helps developers optimize memory usage and debug performance bottlenecks in local LLM inference. Builders should monitor these reallocation reports to fine-tune model execution and resource allocation.
Source: github_releases