~314B total / ~13.2B active per token (sparse MoE)
262,144 tokens
MIT License
Final MIT-licensed release of Motif Technologies' fully in-house large-scale sparse Mixture-of-Experts language model, following the July Motif-3-Beta checkpoint. Motif-3 is a 53-layer decoder-only architecture with 2 dense layers followed by 51 MoE layers, 384 routed experts with top-8 activation plus 1 shared expert, a 4,096-dimensional hidden state, and approximately 13.2B of its 314B parameters active per token. Its custom architecture centers on Grouped Differential Latent Attention (GDLA), which combines grouped differential attention with the compressed key-value representation of Multi-head Latent Attention and a query-dependent output gate. It also introduces Expert-Specific PolyNorm activations, modified manifold-constrained hyper-connections that mix four parallel residual streams, and a one-layer Multi-Token Prediction head that supports self-speculative decoding. The model was pretrained on approximately 12.5T tokens spanning web documents, STEM, code, mathematics, multilingual and domain-specialized corpora, with added emphasis on Korean, reasoning-intensive, legal, and financial data. Post-training combines general supervised fine-tuning, six reinforcement-learning specialist teachers, a separately trained software-engineering teacher, and Multi-teacher On-Policy Distillation to consolidate reasoning, coding, professional work, calibrated abstention, instruction following, and long-horizon tool use into one model. Motif reports scores including 38.7 on GDPVal v2, 35.3 on tau3-Banking, 51.5 on ITBench, 76.2 on SWE-Bench Verified, 74.9 on Terminal-Bench 2.1, 83.4 on GPQA Diamond, 37.0 on Humanity's Last Exam, 72.3 on AA-LCR, and 78.2 on IFBench. Official artifacts include the post-trained BF16 checkpoint, an unaligned base checkpoint intended for continued training and research, and an NVFP4 quantization validated for serving on two NVIDIA B200 GPUs. Motif reports that NVFP4 remains within approximately 0.15 average points of its Block-FP8 deployment across five accuracy evaluations. The release supports native 256K context, automatic tool selection with Motif-specific reasoning and tool-call parsers, online Block-FP8 quantization, expert-parallel vLLM deployment, and built-in MTP self-speculative decoding.