550B total / 55B active (MoE)
1M tokens
OpenMDW-1.1
Official open-weight Nemotron 3 Ultra release family covering the base BF16 checkpoint, the post-trained BF16 checkpoint, and an inference-optimized NVFP4 quantized checkpoint for frontier-scale agentic reasoning. NVIDIA positions Nemotron 3 Ultra as the largest and most capable model in the Nemotron 3 line, built for long-running autonomous agents across coding, research, multi-step planning, tool use, synthesis, verification, and recovery workflows. The technical report describes a hybrid Mamba-Attention Mixture-of-Experts architecture with 108 total layers, 8,192 model dimension, 64 query heads, 2 KV heads, 512 experts per layer with top-22 activation, LatentMoE routing, and two shared-weight Multi-Token Prediction heads that enable native speculative decoding. NVIDIA says the model was pretrained in NVFP4 on 20T text tokens, then extended to 1M context, and post-trained with a pipeline combining supervised fine-tuning, unified RLVR across reasoning and agentic environments, and Multi-teacher On-Policy Distillation, with reasoning budget control exposed at inference time. Official release materials emphasize both accuracy and efficiency, including claims of 5.9x, 4.8x, and 1.6x higher inference throughput than GLM-5.1-754B-A40B, Kimi-K2.6-1T-A32B, and Qwen-3.5-397B-17B respectively on an 8K input / 64K output setting while maintaining on-par benchmark accuracy, plus open release of the checkpoints, training data, RL environments, and model recipes.
https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technical-Report.pdf