505B total / 18B active (MoE)
524,288 tokens
OpenPangu Model License Agreement Version 2.0
Ascend-native open-weight Mixture-of-Experts language model designed for long-context agentic reasoning, with unified thinking and non-thinking behavior, structured tool calling, and a native 512K-token context window. Huawei specifies 505B total parameters with approximately 18B active per token, while the Hugging Face repository indexes 541.49B stored tensor parameters across the shipped checkpoint, which includes three multi-token-prediction layers and auxiliary components. The 50-layer architecture begins with three dense layers followed by 47 MoE layers, using 384 routed experts plus one shared expert and selecting eight routed experts per token. Its configuration specifies a 5,120-dimensional hidden state, 64 attention heads, a 1,792-dimensional expert intermediate state, and a 151,552-token vocabulary. openPangu-2.0-Pro builds on Multi-Head Latent Attention with a repeating 1:2 mixture of DeepSeek Sparse Attention and Sliding-Window Attention. Sliding-window layers use a 512-token local window, while sparse global layers use a lightweight indexer to select 2,048 relevant historical tokens. Additional attention mechanisms include 128 learned parametric attention-sink entries per layer and local-context ModAttn convolutions around global attention. The model replaces a single residual stream with four-stream Manifold-Constrained Hyper-Connections, adds sandwich and periodic block normalization for stability, and was pretrained with the Muon optimizer on approximately 34 trillion tokens without reported loss spikes. Three MTP heads predict three additional tokens for self-speculative decoding. Post-training consists of unified supervised fine-tuning, parallel reinforcement-learning specialists for reasoning, general tasks, agents, and coding, followed by multi-expert on-policy distillation to combine those capabilities. Huawei reports thinking-mode scores of 95.4 on AIME 2026, 86.2 on HMMT February 2026, 87.9 on GPQA-Diamond, 27.1 on Humanity's Last Exam, 81.1 on TAU2-Bench, 65.7 on BrowseComp, 85.7 on LiveCodeBench v6, and 68.5 on SWE-bench Verified. Its Ascend-optimized omni-infer deployment stack includes specialized mHC, sparse-attention, communication, cache-management, prefix-caching, tokenization, and quantization optimizations; the technical report claims 9.55 ms time per output token at ultra-low latency and 1,326 generated tokens per second per NPU at 20 ms TPOT with 128K context. The OpenPangu Model License Agreement Version 2.0 prohibits accessing, downloading, installing, running, deploying, integrating, modifying, or otherwise using the model within the European Union and requires specified openPangu attribution when distributing products or services based on the model.
https://huggingface.co/openpangu/openPangu-2.0-Pro/blob/main/openPangu-2.0%20Tech%20Report.pdf