TL;DR: A Qwen 3.8 27B model achieved 50 tokens/second with a 100,000 token context window on a consumer 16GB GPU using beellama.cpp and specific quantization.
Summary: A user successfully ran the Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller model, a custom hybrid quantization, on an RTX 4070 Ti SUPER (16GB VRAM). This setup achieved 50 tokens/second inference speed with a 100,000 token context window. The key enabler was beellama.cpp, which supports kvarn KV cache types, along with optimized cache settings and a specific chat template.
Why it matters: This demonstrates a significant advancement in running large language models with extensive context on consumer hardware. AI builders can explore beellama.cpp and kvarn cache types to optimize their local LLM deployments for long-context applications.
Source: reddit