3.98B physical parameters (looped dense model)
262,144 tokens (256K)
OpenMDW License Agreement, version 1.1
Small text-only reasoning and agent model built around a looped, or latent-recurrent, normalized-GPT architecture. Dory divides its network into eight input layers, a shared 20-layer recurrent block, and eight output layers, reinjecting the input into each recurrence. The same checkpoint can execute the recurrent block one, two, or four times without retraining; NVIDIA recommends two loops as the default balance between compute and accuracy. Its attention pattern interleaves sliding-window and full-attention layers. Per-loop KV caching retains separate caches for each recurrence, while the optional last-loop mode overwrites them so KV-cache memory remains at approximately the one-loop footprint regardless of loop count. The BF16 repository contains exactly 3,975,229,440 parameters in one 7.95 GB safetensors file, with official MXFP8, NVFP4, and Q5_K_M GGUF releases also available. NVIDIA pretrained Dory on 6T tokens, using native NVFP4 for approximately the first 4.5T tokens and BF16 for the remainder, then performed a 300B-token context-extension phase, supervised fine-tuning, and asynchronous multi-environment GRPO across mathematics, coding, science, instruction following, structured output, multi-step tool use, and multi-turn conversation. The released chat template supports reasoning and tool calling. NVIDIA reports SWE-bench Verified scores of 50.3, 59.5, and 61.8 and Terminal-Bench 2.1 scores of 12.5, 18.1, and 22.2 at one, two, and four loops respectively; these are publisher evaluations under NVIDIA's stated harness. Native 256K inference is documented through vLLM, but Dory support remains under review upstream in vLLM and llama.cpp at release time.