21B (dense; GPT-NeoX-20B fine-tune)
2,048 tokens
GPL-3.0
Open-weight preservation release of NovelAI's legacy Krake V2 storytelling model ahead of its eventual retirement from the NovelAI service. Krake is a fine-tune of EleutherAI's GPT-NeoX-20B and was originally introduced on March 11, 2022, with its V2 update following on April 29, 2022. The released configuration uses a 44-layer dense GPT-NeoX causal-language-model architecture with a 6,144-dimensional hidden state, 64 attention heads, a 24,576-dimensional feed-forward layer, parallel residual connections, partial rotary embeddings, and a 2,048-token context window. NovelAI describes Krake as a temperamental but versatile storytelling model that was historically restricted to its Opus subscription tier because of its size and serving cost. For this preservation release, NovelAI converted the model from its internal format into nine FP16 Hugging Face Safetensors shards totaling approximately 41.1 GB and converted its legacy customization modules into Hugging Face PEFT format. The standard GPTNeoXForCausalLM implementation permits use through Transformers without a proprietary runtime. NovelAI says it released the checkpoint and modules for nostalgia, posterity, and historic preservation so that Krake remains available after leaving the hosted service.