0.6B and 1.2B physical parameters; three recurrent passes producing 54 and 72 effective layers
4,096 tokens
Apache 2.0
Fully open family of pretrained recurrent-depth language models introducing Attention Residuals for looped Transformers. Rather than repeatedly accumulating every intermediate representation into a single residual stream, each effective sublayer uses a learned query to selectively attend over representations produced at earlier depths. Residual updates are grouped into blocks to control storage and computation. ARLoop retains direct access to the original input embedding while limiting depth attention to a sliding window of the three most recent residual blocks; controlled experiments found this local depth window matched or exceeded full-history attention at greater recurrent depths. The released ARLoop-0.6B contains 18 physical Transformer layers shared across three passes for an effective depth of 54 layers, with a 1,536-dimensional hidden state and 4,608-dimensional SwiGLU intermediate layers. ARLoop-1.2B contains 24 physical layers shared across three passes for an effective depth of 72 layers, with a 2,048-dimensional hidden state and 5,632-dimensional intermediate layers. Both models use grouped-query attention with 16 query heads and four key/value heads, QK normalization, alternating shared value embeddings, tied token embeddings and output heads, a 49,152-token SmolLM2 vocabulary, RoPE with a 50,000 base, and a 4,096-token context window. ARLoop-0.6B was pretrained on approximately 180 billion tokens, while ARLoop-1.2B was pretrained on 500 billion tokens. Their three-stage training mixture progressively introduces higher-quality mathematics, reasoning, and code data drawn from FineWeb-Edu, StackExchange, FineMath, InfiWebMath, OpenMathReasoning, Natural Reasoning, OpenMathInstruct2, Dolmino Math, OpenCodeReasoning, Kaggle notebooks, The Stack v2, GitHub Issues, and Stack-Edu. Training uses AdamW, cosine learning-rate decay, BF16 mixed precision, and packed 4,096-token sequences. Developer-reported ARLoop-1.2B scores include 56.3 on ARC-Challenge, 71.5 on HellaSwag, 66.8 on WinoGrande, 45.5 on OpenBookQA, 63.1 on GSM8K, 37.8 on MATH, 52.1 on MBPP, and 33.1 on HumanEval. Under the authors' common evaluation protocol, the 1.2B model exceeds both Huginn-3.5B and Qwen3.5-2B across all eight reported tasks despite using only 500 billion pretraining tokens compared with Qwen3.5-2B's reported 36 trillion. At the 0.6B scale, Attention Residuals improve over an otherwise matched PreNormLoop baseline particularly on mathematics and coding: GSM8K rises from 35.7 to 40.9, MATH from 20.9 to 26.3, MBPP from 33.1 to 36.2, and HumanEval from 17.6 to 20.1. The release includes a separately loadable PreNormLoop-0.6B baseline, controlled 140M- and 370M-parameter experiments spanning one through five recurrences, complete data and training configurations, evaluation scripts, and intermediate checkpoint revisions throughout the 180B- and 500B-token runs. The final ARLoop-0.6B checkpoint contains approximately 2.33 GB of stored FP32 weights, while ARLoop-1.2B contains approximately 4.83 GB; the repositories load them as BF16 for inference while retaining FP32 routing and normalization statistics internally. These are English-predominant base pretrained models intended for text completion, continued pretraining, mid-training, and post-training; they do not ship as instruction-following chat assistants.