TL;DR: A new configuration for Qwen3.8-27B on DGX Spark (GB10) demonstrates high inference speeds using SGLang, NVFP4, and DFlash2, setting a new benchmark for local LLM deployment.
Summary: The hasso5703/dgx-spark-qwen38 project reports a configuration for the Qwen3.8-27B model on NVIDIA DGX Spark (GB10) achieving a median greedy inference speed of 50 tokens/second. This setup leverages SGLang, NVFP4, and DFlash2, offering deterministic boots and high throughput, reaching 148 tok/s at 8 streams and 258 tok/s at 32 streams.
Why it matters: This benchmark provides a reference for optimizing large language model inference on powerful local hardware, indicating the potential for high-performance, cost-effective local AI applications. AI builders should investigate SGLang, NVFP4, and DFlash2 for accelerating their own LLM deployments on similar hardware.
Source: github_topics