TL;DR: Xolotl, a fork of Strata, demonstrates running the 552B DeepSeek-V4.1-Flash model on a single gaming PC with an RTX 4090, achieving impressive inference speeds through advanced offloading techniques.
Summary: Indras-Mirror's Xolotl (formerly Strata-DSV4.1) project showcases the ability to run the 552B DeepSeek-V4.1-Flash model on a single consumer-grade gaming PC equipped with an RTX 4090 and 90 GB RAM. It achieves approximately 27 tokens per second by tiering MoE experts across VRAM, pinned RAM, and NVMe storage. The project is a heavily extended fork of Strata, also supporting GLM-5.3-Flash and Qwen models.
Why it matters: This development significantly lowers the hardware barrier for experimenting with large MoE models locally, enabling indie developers to run powerful LLMs without enterprise-grade infrastructure. Builders should explore Xolotl's offloading strategies to optimize local inference for other large models.
Source: github_topics