Description
MiMo-V2.5-Pro is a cutting-edge open-source Mixture-of-Experts (MoE) language model with a total of 1.02 trillion parameters and 42 billion active parameters. It is designed to handle the most demanding agentic, complex software engineering, and long-horizon tasks. The model utilizes a hybrid attention architecture, interleaving Sliding Window Attention (SWA) and Global Attention (GA) with a 6:1 ratio, which significantly reduces KV-cache storage while maintaining long-context performance. This architecture is complemented by three lightweight Multi-Token Prediction (MTP) modules that enhance output speed during inference.
The model is pre-trained on 27 trillion tokens using FP8 mixed precision and supports a context window of up to 1 million tokens. Post-training, MiMo-V2.5-Pro employs Supervised Fine-Tuning (SFT), large-scale agentic Reinforcement Learning (RL), and Multi-Teacher On-Policy Distillation (MOPD) to achieve superior performance in complex tasks. It excels in sustaining complex trajectories over a 1M-token context window, making it highly effective for tasks requiring strong instruction following and coherence.
MiMo-V2.5-Pro's architecture addresses the quadratic complexity of long contexts by integrating Local Sliding Window Attention and Global Attention. Its training process includes a three-stage post-training paradigm that begins with SFT to build foundational skills, followed by Domain-Specialized Training with diverse teacher models, and culminating in MOPD for dynamic on-policy RL.
The model's deployment is supported by the SGLang and vLLM communities, with recommended configurations for optimal performance. MiMo-V2.5-Pro is ideal for industries requiring advanced language processing capabilities, such as software engineering and multilingual applications.
MiMo-V2.5-Pro Highlights
1.02T total parameters
42B active parameters
Hybrid attention architecture
Multi-Token Prediction (MTP)
Efficient pre-training on 27T tokens
1M tokens context length
Supervised Fine-Tuning (SFT)
Multi-Teacher On-Policy Distillation (MOPD)
Sliding Window Attention (SWA)
Global Attention (GA)
Getting Started with MiMo-V2.5-Pro
Access page: Visit the Hugging Face model page
Load model: Download MiMo-V2.5-Pro
Configure environment: Set up with recommended parameters
Integrate: Implement in your application
Fine-tune: Adjust model for specific tasks
MiMo-V2.5-Pro's Use Cases
- Complex Software Engineering
- Agentic Tasks
- Multilingual Applications
- Long-Context Reasoning
- Reinforcement Learning
