16B total / 2.8B active per token (MoE)
Up to 64K tokens (4K before the long-context Base stage)
Research-Only RAIL Model License (academic and research purposes only)
Fully disclosed Mixture-of-Experts language-model family trained from scratch by AMD, grouping six downloadable checkpoints spanning Pretrain, Midtrain, long-context Base, supervised fine-tuning, direct preference optimization, and reinforcement-learned Think stages. The 27-layer decoder has a 2,048-dimensional hidden state, 16 attention heads, 64 routed experts, two shared experts, and six routed experts selected per token, giving it 16 billion total parameters with approximately 2.8 billion active. Its architecture adds a learned output gate to Multi-head Latent Attention, called Gated MLA, and uses FarSkip-Collective connectivity to overlap expert-parallel communication with computation; AMD reports a 12.7% pretraining speedup and up to a 39.2% reduction in time to first token under expert-parallel SGLang serving. Pretraining covered 7.1 trillion tokens of openly identified web, mathematics, code, science, and instruction-style data on AMD Instinct MI300X and MI325X accelerators using ROCm and Primus, with a Multi-Token Prediction objective during the Pretrain and Midtrain stages. The Base checkpoint extends context from 4K to 64K using YaRN, document-boundary masking, and Longmino 100B data, after which AMD provides SFT, DPO, and Think checkpoints for instruction following and chain-of-thought reasoning. The final RL process uses the Miles framework, 1,400 steps of instruction-following-specialized training, and Multi-Teacher On-Policy Distillation to incorporate instruction-following gains while preserving math, coding, and general capabilities. AMD reports a 76.7 average for the Base checkpoint across its standard evaluation suite, including 86.5 on WinoGrande and 65.7 on HumanEval+, plus 41.5 on HELMET and 79.4 on RULER at long context. The Think checkpoint reaches a reported 73.22 aggregate score, 83.70 on IFEval, 82.50 on AGIEval, and 73.40 on AIME25. In addition to every major weight checkpoint, AMD released training configurations, data-mixture documentation, intermediate artifacts, tokenizer assets, and inference and training code. The model weights are downloadable but restricted by AMD's Research-Only RAIL license to academic and research purposes, with mandatory responsible-use conditions.