1B
4096
Apache-2.0
Official open-weight 1B-parameter base language model built on Sapient Intelligence's Hierarchical Reasoning Model architecture. The release frames HRM-Text as an efficiency-focused from-scratch pretraining effort using a dual-timescale recurrent design with separate high-level and low-level transformer stacks that are reused across nested recurrence steps, aiming to increase effective compute depth without scaling parameter count in the usual way. Official model details describe PrefixLM training, gated attention with a per-head sigmoid output gate, parameterless Pre-RMSNorm, RoPE positional encoding, SwiGLU activations, a 65,536-token vocabulary, and a training run over 40B unique tokens. The model card explicitly positions this checkpoint as pre-alignment rather than an assistant model: it is non-chat, non-instruction-tuned, and intended as a starting point for further SFT or RL. Release materials also emphasize unusually low reported pretraining cost for the reference runs, with a 0.6B model trained on 8 H100s in about 50 hours and a 1B model on 16 H100s in about 46 hours, alongside benchmark claims including 84.7 on GSM8K, 56.5 on MATH, 82.3 on DROP, and 60.7 on MMLU for the 1B reference run.