Ternary Bonsai 2 27B
Compresses a 27B multimodal model to run locally on consumer hardware with near-lossless performance for coding, vision, and agentic tasks.
The gist
Ternary Bonsai 2 27B is a highly compressed multimodal foundation model from PrismML. It is designed to solve the challenge of deploying large 27B-class models on local devices by reducing the memory footprint by over 9x. This allows powerful reasoning, coding, and vision capabilities to run efficiently on consumer-grade hardware like laptops, where memory and power are limited.
What it does
- Compresses a 27B parameter model to a 5.9GB footprint using ternary quantization.
- Retains 98.2% of the original full-precision model's performance across benchmarks.
- Processes multimodal text and image inputs within a 262K-token context window.
- Runs on NVIDIA GPUs via CUDA and Apple Silicon devices via MLX.
- Demonstrates strong capabilities in coding, vision, reasoning, and agentic tool use.
How it works
The model uses ternary {-1, 0, +1} weights and FP16 scaling, resulting in an effective 1.76 bits per weight. It is deployed using custom low-bit kernels for CUDA and MLX, which are required for inference. The model weights are released under the open-source Apache 2.0 license and are available on Hugging Face for developers to integrate into applications.
Best for
This model is ideal for developers building AI applications that require powerful, 27B-class intelligence but must run locally due to privacy, latency, or cost constraints.
Watch out for
Running the model requires custom low-bit kernels provided by PrismML; it is not a standard full-precision model that works out-of-the-box with all common inference frameworks.