4.11B dense
1,048,576 tokens
Apache-2.0
Apache-2.0 compact general-purpose language model for local and on-device agents. Spark-X2.6-4B retains exactly the same 4.11B-parameter architecture as Spark-X2.5-4B and attributes its improvements to a new post-training recipe rather than additional parameters or architectural changes. The 36-layer dense Transformer uses a 2,560-dimensional hidden state, 10,240-dimensional feed-forward layers, 16 query heads, 4 key-value heads, 256-dimensional attention heads, tied input and output embeddings, a 131,072-token vocabulary, and headwise sigmoid attention-output gates. Its hybrid attention pattern repeats three 512-token sliding-window layers followed by one full-attention layer, giving nine full-attention layers across the network while natively supporting a 1,048,576-token context window. Full-attention layers use 25% partial rotary embeddings with a 5,000,000 RoPE base, while sliding-window layers apply full rotary embeddings with a 10,000 base. Post-training targets coding, software engineering, multi-step planning, tool use, personal-assistant workflows, web research, multilingual behavior, reasoning, and effective use of extremely long contexts. The chat interface exposes three thinking budgets: none, low, and high. In publisher evaluations conducted in thinking mode with Spark-X2.6 set to high, it attains a 63.3 average across the reported suite versus 52.1 for Spark-X2.5-4B, 49.1 for Qwen3.5-9B, 43.2 for Qwen3.5-4B, 47.0 for Gemma4-12B, and 29.7 for Gemma4-E4B. Reported individual scores include 71.8 on BFCL-V4, 86.9 on tau2-bench, 34.7 on tau3-bench, 62.4 on MCP-Atlas, 53.1 on BrowseComp, 74.1 on PAgentBench, 54.3 on SWE-bench Pro, 62.8 on SWE-bench Verified, 64.7 on SWE-bench Multilingual, 91.2 on AIME 2026, 36.9 on HLE with tools, and 67.3 on AA-LCR v1.1. Long-context results include 92.2 on Graphwalks BFS at 256K, 34.2 at 1M, 65.6 on OpenAI MRCR v2 from 256K to 512K, and 26.0 from 512K to 1M. Some comparison-model values are copied from their published cards rather than reproduced by XHToken. The BF16 checkpoint occupies approximately 8.22 GB across five safetensors shards. Official deployment variants include approximately 4.45 GB FP8 and INT8 checkpoints, a 3.41 GB INT4 checkpoint, and GGUF releases in BF16, Q8_0, and Q4_K_M. The release supports Transformers, SGLang, vLLM, llama.cpp, Spark-MLX-LLM, Ollama, Unsloth Studio, NVIDIA GPUs, Apple silicon, and Huawei Ascend NPUs.