35,072,768
2,048 tokens
Apache License 2.0
Compact English-centric decoder-only base language model designed to maximize capability per parameter and released as a native PyTorch completion model rather than an instruction-tuned chat model. BarunLM-35M uses 12 dense Transformer layers with a 448-dimensional hidden state, seven query heads, one shared key-value head, a 1,228-dimensional feed-forward layer, and tied input/output embeddings over a 16,384-token byte-level BPE vocabulary. Its hybrid attention schedule repeats three local-attention layers with a 256-token window followed by one full-attention layer, providing periodic global information exchange while limiting most KV-cache growth. Additional architectural features include grouped-query attention, 50% partial RoPE, query-key normalization, sigmoid-gated attention outputs, activation-bounded SwiGLU, and a learned convex residual selector after every four-layer block that can preserve the block input when its transformation is unhelpful. The model was pretrained on 5,699,985,408 tokens, approximately 162.5 tokens per parameter, from a deduplicated mixture of FineWeb-Edu, Cosmopedia v2, FineMath-4+, DCLM-Baseline, CodeSearchNet Python, and CodeParrot Clean. Its final four-billion-token continuation used Muon with a peak learning rate of 1e-4, 0.1 weight decay, a batch size of 48, and 40,690 optimizer steps on one NVIDIA H200. Harshal Singh reports 41.01% macro accuracy across a fixed nine-task zero-shot LM Evaluation Harness suite comprising ARC-Challenge, ARC-Easy, BoolQ, HellaSwag, LAMBADA OpenAI, OpenBookQA, PIQA, SciQ, and WinoGrande. Under the same decontaminated protocol, BarunLM reportedly exceeds LFM2.5-230M-Base by 1.81 percentage points while using 6.55 times fewer parameters; the published 10,000-resample bootstrap interval for that difference is +0.92 to +2.71 points. Evaluation excludes 1,854 samples detected through a correctness-blind exact 13-token scan of the complete training history. The release includes the checkpoint, tokenizer, architecture implementation, generation script, checksums, configuration, and detailed benchmark evidence under Apache 2.0. Its creator cautions that the model's small capacity, short context, lack of instruction tuning, and English-centric training limit factual recall, multi-step reasoning, instruction following, and high-stakes use.