Description
K2-Horizon-MoVA-36B-A4B is a sophisticated Mixture-of-Experts model that incorporates Mixture-of-Values attention, designed to handle complex tasks with efficiency. It is part of the K2-Horizon family and is notable for its sparse architecture, which allows it to store 36 billion parameters while running 4 billion per token. This model has been released with open access to its checkpoints, data, and training code, supporting the democratization of artificial intelligence through open-source and open science initiatives.
Read More
The model achieves frontier-class results, particularly excelling in agentic and reasoning benchmarks. It outperforms open-weight dense models of approximately 30 billion in size and MoE models up to 15 times its size, while also competing effectively against closed frontier models. With a native context window of 524,288 tokens, K2-Horizon-MoVA-36B-A4B supports extensive context processing from midtraining stages onward.
Intermediate checkpoints are available, allowing researchers to study capability changes across different training stages. The model's training data and recipe, along with the training code, are fully open, promoting transparency and collaboration in AI research.
K2-Horizon-MoVA-36B-A4B is validated with Transformers 5.15.0, PyTorch 2.13.0, and Safetensors 0.8.0. It supports multiple tool-call formats, including JSON, XML, and XML-typed, with XML as the default. The model is designed to be used with high reasoning effort settings, ensuring optimal performance across various applications.
The model is suitable for a range of applications, including agentic tool use, terminal use, scientific coding, and graduate-level science QA. It is particularly effective in long-context reasoning and maintaining factual accuracy, making it a valuable tool for researchers and developers in the AI community.
- CategoryAI Models & LLMs
- Pricing ModelOpen source
- DeploymentCloud
- PlatformsWeb
K2-Horizon-MoVA-36B-A4B Highlights
Mixture-of-Experts architecture
Mixture-of-Values attention
36 billion parameters
4 billion active parameters
Open-source checkpoints and data
512K context window
Intermediate checkpoints available
Validated with Transformers and PyTorch
Getting Started with K2-Horizon-MoVA-36B-A4B
Access model page: Visit Hugging Face
Authenticate: Log in to your account
Configure: Set reasoning_effort to high
Send prompts: Use API with recommended settings
Fine-tune: Utilize available checkpoints
K2-Horizon-MoVA-36B-A4B's Use Cases
- Agentic Tool Use
- Terminal Use
- Scientific Coding
- Graduate-Level QA
- Long-Context Reasoning
