3B (dense)
8,192 tokens
GPL-2.0-only
Open-weight preservation release of Clio, NovelAI's first fully original language model pretrained from scratch, ahead of its eventual retirement from the NovelAI service. Clio was originally released on May 23, 2023 after training on NovelAI's Shoggy H100 cluster. NovelAI reports approximately 1.6 trillion training tokens, representing roughly 6 TB of text, drawn from its custom Nerdstash dataset and encoded with Nerdstash Tokenizer V1. The released checkpoint uses a 28-layer dense causal-language-model architecture compatible with Hugging Face's StableLmForCausalLM class, with a 2,816-dimensional hidden state, 22 attention and key/value heads, a 7,552-dimensional feed-forward layer, parallel residual connections, partial rotary embeddings, and an 8,192-token context window. NovelAI reports that Clio reached approximately 73% LAMBADA accuracy during pretraining and exceeded 74% during fine-tuning, while its writing performance surpassed the substantially larger Euterpe and Krake models. The Hugging Face release occupies approximately 6.36 GB across two primarily BF16 Safetensors shards, includes the original tokenizer assets, and converts NovelAI's customization modules into PEFT format. NovelAI found an existing Transformers model class that fits Clio once an additional configuration flag is enabled, so the preservation release requires no proprietary model implementation. The checkpoint and modules were published for nostalgia, posterity, and historic preservation.