TL;DR: Picchio is a new C-based inference engine enabling large Mixture-of-Experts (MoE) models to run on consumer hardware by streaming data from SSD.
Summary: benmaster82 released Picchio, a streaming Mixture-of-Experts (MoE) inference engine written in pure C. It allows GPT-OSS models (20B and 120B) to run on consumer hardware, even if the model size exceeds available RAM, by leveraging SSD for data streaming.
Why it matters: This lowers the barrier for local deployment and experimentation with large MoE models. AI builders should explore Picchio for memory-efficient inference of large language models on commodity hardware.
Source: github_topics