TL;DR: llama.cpp now supports CUDA Virtual Devices, enhancing GPU resource management for local LLM inference.
Summary: The latest llama.cpp release (b10043) introduces support for CUDA Virtual Devices. This feature allows for more flexible and efficient utilization of NVIDIA GPUs, particularly in environments where multiple processes or users need to share GPU resources for running LLMs.
Why it matters: This update improves resource allocation and performance for AI builders deploying local LLMs on NVIDIA GPUs. Experiment with virtual devices to optimize multi-user or multi-model inference setups.
Source: github_releases