7.9B total / 1.3B active (Tiny); 124B total / 5.1B active (Flash)
262,144 tokens
MIT
Research-oriented release of six unaligned Ling-3.0 base-model checkpoints spanning two sparse Mixture-of-Experts scales and three stages of the same training recipe. For both Ling-3.0-tiny and Ling-3.0-flash, inclusionAI provides a `-base-30T` checkpoint taken after large-scale pretraining but before mid-training, learning-rate decay or WSM merging, and post-training; a `-base-midtrain` checkpoint taken after mid-training but before merging or post-training; and an unsuffixed `-base` checkpoint produced from the mid-trained model through Warmup-Stable and Merge, or WSM. WSM replaces a conventional learning-rate decay phase with weighted checkpoint merging, making the resulting models more suitable for continual pretraining and allowing researchers to explore different effective decay profiles offline without rerunning the corresponding training schedules. Tiny and Flash share one training recipe so that continued-pretraining, fine-tuning, and post-training strategies can be developed on the smaller model before being scaled to Flash. Ling-3.0-tiny contains 7.9B total and 1.3B active parameters across 24 transformer layers arranged as 18 Kimi Delta Attention and 6 gated Multi-Head Latent Attention layers in a 3:1 pattern. It has one initial dense layer, 128 routed experts, and activates eight routed experts plus one shared expert per token. Ling-3.0-flash contains 124B total and 5.1B active non-embedding parameters across 42 layers arranged as 35 Kimi Delta Attention and 7 gated Multi-Head Latent Attention layers in a 5:1 pattern. It has two initial dense layers, 512 routed experts, and activates eight routed experts plus one shared expert per token. Both architectures use native hybrid linear attention, a 157,184-token vocabulary, and configurations supporting 262,144-token contexts. None of the six checkpoints has undergone supervised or reinforcement-learning post-training; inclusionAI recommends them for continued pretraining, mid-training, supervised fine-tuning, preference optimization, RL, distillation, and long-context or MoE research rather than direct end-user deployment.