80B total / 3B active (MoE; 81,286,433,408 stored BF16 parameters)
262,144 tokens
Apache 2.0
Official open-weight, text-only pretrained foundation model developed and trained entirely from scratch by Yandex. AliceAI-Foundation-80B-A3B-Base uses a highly sparse 48-layer hybrid causal architecture arranged as 12 repetitions of three KDA linear-attention layers followed by one gated full-attention layer, with a Mixture-of-Experts feed-forward block attached to every layer. The model has a 2,048-dimensional hidden state, a 129,024-token vocabulary, and approximately 80B marketed parameters while activating about 3B per token; the published safetensors metadata counts exactly 81,286,433,408 BF16 parameters. Each MoE layer contains 512 routed experts, selects ten experts per token, and additionally activates one shared expert, with 512-dimensional expert intermediate states, sigmoid routing, and router-bias correction. Its 36 KDA layers use 32 query, key, and value heads with 128-dimensional heads and a four-token convolution kernel, while the 12 gated-attention layers use 16 query heads, two KV heads, 256-dimensional heads, and partial rotary position encoding. The architecture also contains a single Multi-Token Prediction layer for speculative decoding and supports a native 262,144-token context window. Yandex assembled a new Russian-and-English training corpus containing data for factual knowledge, complex reasoning, coding, and tool use. The final pretraining-token count is not disclosed, but Yandex says key architectural and hyperparameter decisions were validated through multiple independent training runs from scratch, each consuming two trillion tokens. This release represents the pretraining stage and deliberately does not define a default chat template; raw generation should not be treated as instruction-following behavior. The repository instead includes the agentic conversation format encountered during training, a separate Jinja template for preparing supervised fine-tuning data, and a four-80GB-GPU FSDP2 LoRA example. Vendor evaluations were conducted through Yandex's internal infrastructure using vLLM. At deterministic temperature zero, Yandex reports 86.5 on WikiWebFacts, 67.9 on HardMultiQA, 86.5 on CultCat, 74.2 on EduBench Russian, 90.5 on the Russian Unified State Exam benchmark, 66.8 on MMLU-Pro, 44.3 on SuperGPQA, 48.3 on BigCodeBench, 50.5 on LiveCodeBench v5-6 with one-shot chain of thought, 74.1 on FinQA 128K, and 64.6 on LongMemEval 128K. In higher-temperature pass-at-k evaluations, it reports 96.7 pass@32 on AIME 2026, 96.9 pass@32 on HMMT February 2026, 88.7 pass@8 on IMO AnswerBench, 60.4 pass@1 and 82.9 pass@8 on LiveCodeBench v5-6, and 68.9 pass@8 on Codeforces C++. The release is particularly strong on Yandex's Russian factual and educational evaluations, while its English TriviaQA and several general-knowledge results remain behind larger-active-parameter baselines. Yandex simultaneously released the WikiWebFacts and HardMultiQA Russian factual-knowledge datasets and their evaluation protocols. The BF16 checkpoint is distributed across 49 safetensors shards totaling approximately 162.58 GB. Transformers loading uses bundled custom AliceAI model code and therefore requires trust_remote_code=True, while the reference vLLM launch enables the included one-token MTP speculative decoder.