35B total / approximately 3B active (MoE)
262,144 tokens
Apache License 2.0
Custom-packed additive compression of Qwen3.6-35B-A3B, released in a text-focused repository and a same-day multimodal variant that adds the base model's BF16 vision tower to the byte-identical compressed language payload. The underlying 40-layer Mixture-of-Experts architecture contains 256 routed experts, selects eight experts per token, and repeats three linear-attention layers followed by one full-attention layer. The published configuration specifies a 2,048-dimensional hidden state, 16 attention heads, two key-value heads, and a 262,144-token context window. Syzygy's manifest reports a 7.5335 GB text payload averaging 1.6978 bits per weight, including 6.2083 GB of packed experts. Expert matrices use integer L1-ball trellis codes with per-wavefront scales and continuous FP16 side streams; the non-expert spine uses a 64-level integer lattice, while the language-model head uses grouped INT5 and the embedding uses INT4. Syzygy describes the representation as enabling add/subtract-only weight matrix multiplications. The release drops the source model's multi-token-prediction head but retains its reasoning-oriented chat template and structured tool-calling format. A bundled standalone NumPy decoder documents and verifies reconstruction of the custom weight representation, although the packed files are not directly loadable as an ordinary Transformers Safetensors checkpoint. Syzygy reports 95.0% mean score retention relative to the BF16 Qwen3.6-35B-A3B teacher across 12 benchmarks in its evaluation harness. Published absolute scores include 89.58 on AIME26, 98.00 on MATH-500, 87.50 on AIME25, 94.44 on MBPP+, 92.68 on HumanEval+, 83.75 on IFEval, 68.97 on BFCL-v3, 71.58 on tau2-bench, and 54.08 on IFBench. The multimodal variant adds the uncompressed 27-layer Qwen vision tower; Syzygy states that its compressed language weights are otherwise byte-identical to the text release.