8.4B total / 760M active (MoE)
Unknown
Apache 2.0
Official open-weight Zyphra release for compact high-efficiency reasoning, math, and coding workloads, published alongside public cloud access on Zyphra Cloud. The Hugging Face model card describes ZAYA1-8B as a small mixture-of-experts language model trained end-to-end by Zyphra, with under 1B active parameters and strong long-form reasoning performance relative to much larger open-weight models. Zyphra's release post says it is the first MoE model pretrained, midtrained, and supervised fine-tuned on an AMD Instinct MI300 stack, using a 1,024-node MI300X cluster with AMD Pensando Pollara interconnect on infrastructure built with IBM. Release materials attribute its efficiency to Compressed Convolutional Attention, an MLP-based expert router, and learned residual scaling, then describe a five-stage post-training pipeline covering SFT, reasoning warmup, RLVE-Gym training, large-scale math-and-code RL, and a final RLHF/RLAIF pass. Zyphra also introduces Markovian RSA as a test-time-compute method co-trained with the model, claiming major gains on difficult math benchmarks under larger reasoning budgets and positioning ZAYA1-8B as competitive with substantially larger reasoning models on AIME'26, HMMT Feb.'26, LiveCodeBench v6, GPQA-Diamond, and MMLU-Pro.