125B total / 6B active MoE backbone, with 51B N-gram embeddings and 4B MTP parameters
262,144 tokens native; extensible to 1,000,000 tokens with YaRN
Qwen Community License 1.0
Experimental open-weight multimodal preview of the architecture being developed for Qwen4, distinct from the forthcoming production Qwen3.8-Flash managed service. The post-trained model accepts text, images, and video and produces text, using a 125-billion-parameter sparse backbone with approximately six billion parameters activated per token, 512 experts with ten routed experts plus one shared expert selected per token, 51 billion parameters of host-resident bigram and trigram embeddings, and four billion Multi-Token Prediction parameters. Its 48 language-model layers comprise 12 repetitions of three Gated DeltaNet-plus-MoE layers followed by one Qwen Sparse Attention-plus-MoE layer. Additional architectural experiments include four-branch Gated Residual connections, a 20-million-entry N-gram embedding table inserted at layer two, QSA retrieval over 2,048 selected tokens or 512 blocks, and Muon optimization for two-dimensional linear parameters. The technical report attributes an approximately ninefold reduction in training FLOPs relative to its 397B-A17B predecessor to using roughly one-third as many active parameters and one-third as many training tokens; it also reports that QSA's attention kernel is 7.6 times faster during one-million-token prefill and 4.9 times faster during decoding than dense attention. The model supports thinking and non-thinking operation, xhigh, medium, and low reasoning-effort levels, and preserve_thinking enabled by default so that historical reasoning can be retained across multi-turn and tool-calling workflows. Published evaluations report 58.7 on DeepSWE 1.1, 62.5 on SWE-bench Pro, 81.0 on SWE-bench Multilingual, 73.9 on CoWorkBench, 91.7 on GPQA Diamond, 91.9 on LiveCodeBench v6, 84.5 on AndroidWorld, 64.0 on Vision2Web, and 95.7 on MathVision with computer interaction. The approximately 360 GB release is distributed across 131 BF16 safetensors shards. Qwen separately announced that the forthcoming production Qwen3.8-Flash service will provide default one-million-token context, built-in tools, and pricing of $0.16 per million input tokens and $0.47 per million output tokens. The Qwen Community License requires prominent Qwen attribution for products exceeding 100 million monthly active users or $20 million in monthly revenue and requires a separate commercial license for third-party Model-as-a-Service or AI work-assistant businesses, excluding internal use.