0.4B dense (421,772,288 serialized parameters)
16,384 tokens
MIT License
Small English base language model using the Transformers GPT2LMHeadModel architecture with the cl100k_base tokenizer associated with GPT-4-era models. Its configuration contains 24 transformer layers, 16 attention heads, 1,024-dimensional embeddings, and a 100,277-token vocabulary. The developer reports training it for 24 hours on a Google Colab G4 using approximately 30GB of USENET posts, web scrapes, and stories, reaching a training loss of 2.5916 and validation loss of 2.7335. The released checkpoint uses FP32 weights and supports a 16,384-token context window. It is a raw text-completion model rather than an instruction-following assistant, and its model card explicitly recommends supervised fine-tuning before practical use.