TL;DR: A user achieved high inference speeds for the Qwen 3.8 27B LLM on consumer-grade hardware using a custom GGUF quantization and optimized llama.cpp settings.
Summary: A developer successfully ran the Qwen 3.8 27B language model at an average of 75 tokens per second (t/s) on an NVIDIA RTX 5080 with 16GB VRAM. This was accomplished using the 'Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller' model, a hybrid quantization designed for strict VRAM budgets, combined with specific llama.cpp server arguments.
Why it matters: This demonstrates that powerful LLMs can achieve impressive performance on consumer hardware with careful model quantization and inference engine tuning. Indie developers should explore custom GGUF quantizations and optimized llama.cpp settings to deploy large models efficiently on limited VRAM.
Source: reddit