TL;DR: Ninnix released q36, an open-source inference engine that runs Qwen3 models on Vulkan and Metal, bringing local LLMs to Apple Silicon and other GPUs.
Summary: Ninnix/q36 is a local inference engine designed for Qwen3.8-27B and Qwen3.6-35B-A3B, leveraging Vulkan and Metal APIs. It supports GGUF quantization, MoE, SSD streaming, and an OpenAI-compatible server, targeting Apple Silicon, BC-250, and C-based deployments.
Why it matters: With CUDA-free inference, builders can deploy Qwen3 models on personal Apple Silicon Macs and Vulkan-capable GPUs, potentially lowering hardware barriers. Try it if you need a portable, OpenAI-compatible local LLM runtime for edge devices.
Source: github_topics