TL;DR: Korean company Motif released a 13B active, 314B total Mixture-of-Experts (MoE) model with custom architectural innovations, achieving performance comparable to larger models.
Summary: Motif, a Korean AI company, has open-sourced a new Mixture-of-Experts (MoE) model. This model features a 13B active parameter count within a 314B total parameter architecture. It incorporates novel research, including a per-expert activation function called Polynorm and a variant of differential attention (GDLA), alongside modified mHC.
Why it matters: This release provides a powerful, openly available MoE model that indie developers can leverage for efficient, high-performance applications. Builders should explore its custom architectural components for potential integration or inspiration in their own model designs.
Source: x_com