50M
1,024 tokens
Apache 2.0
Official open-weight Supra-50M family release covering both a base causal language model and an instruction-tuned variant, positioned by SupraLabs as the first model in its Scaling Up Plan. The base model card describes Supra-50M as a decoder-only LlamaForCausalLM architecture trained from scratch on 20B tokens from HuggingFaceFW/fineweb-edu, with a 32K custom byte-level BPE tokenizer, 12 layers, 512 hidden size, 1,408 intermediate size, 8 attention heads, 4 key-value heads with GQA, and 1,024 maximum position embeddings. Release materials claim the 50M model is competitive with or better than substantially larger small models on several benchmarks, including reported gains over GPT-2 124M on BLiMP, SciQ, ARC-Easy, and HellaSwag, while remaining CPU-friendly and tiny-model oriented. The instruct variant is built from the same pretrained base and supervised fine-tuned for four epochs on yahma/alpaca-cleaned, giving the family a chat-capable release alongside the raw base checkpoint. Both cards describe Apache 2.0 licensing and standard Transformers compatibility, with the family framed as a compact educational-web-text-trained English text-generation line under Project Chimera.