SmolLM3 Language Model

Description

SmolLM3 is a 3 billion parameter language model developed to advance the capabilities of small models in artificial intelligence. It is designed to support dual mode reasoning and is multilingual, natively supporting six languages: English, French, Spanish, German, Italian, and Portuguese. The model is fully open, with open weights and comprehensive training details, including public data mixtures and training configurations.

The architecture of SmolLM3 is a decoder-only transformer using GQA and NoPE with a 3:1 ratio. It was pretrained on 11.2 trillion tokens using a staged curriculum that included web, code, math, and reasoning data. Post-training involved midtraining on 140 billion reasoning tokens, followed by supervised fine-tuning and alignment through Anchored Preference Optimization (APO).

SmolLM3 is optimized for hybrid reasoning and supports long context processing, trained on a 64k context and capable of handling up to 128k tokens using YARN extrapolation. The model can be deployed using vLLM and SGLang, compatible with OpenAI format APIs. It also supports tool calling, allowing integration with various tools through XML or Python function calls.

For local inference, SmolLM3 can be used with llama.cpp, ONNX, MLX, MLC, and ExecuTorch. Quantized checkpoints are available for efficient deployment. The model's performance has been evaluated across various benchmarks, demonstrating strong results in multilingual Q&A, competitive programming, and instruction following tasks.

SmolLM3 is a valuable tool for developers and researchers looking to leverage AI for multilingual and complex reasoning tasks. However, users should be aware that the generated content may not always be factually accurate or free from biases present in the training data.

SmolLM3 Language Model Highlights

• 3B parameter language model

• Supports dual mode reasoning

• Multilingual: 6 languages

• Long context processing up to 128k tokens

• Open model with open weights

• Instruct model optimized for hybrid reasoning

• Tool calling support

• Compatible with vLLM and SGLang

Getting Started with SmolLM3 Language Model

1. 1

   Access page: Visit the Hugging Face model page

2. 2

   Load model: Use transformers v4.53.0 or latest vllm

3. 3

   Configure environment: Set sampling parameters and context length

4. 4

   Integrate: Use tool calling with XML or Python functions

5. 5

   Fine-tune: Apply supervised fine-tuning and alignment

SmolLM3 Language Model's Use Cases

• Multilingual Q&A
• Hybrid Reasoning
• Tool Integration
• Long Context Processing
• Instruction Following

FAQ from SmolLM3 Language Model

• What is SmolLM3?

  SmolLM3 is a 3B parameter language model designed for small model efficiency, supporting dual mode reasoning and multilingual capabilities.

• How many languages does SmolLM3 support?

  SmolLM3 natively supports six languages: English, French, Spanish, German, Italian, and Portuguese.

• What is the context length capability of SmolLM3?

  SmolLM3 supports long context processing, trained on 64k context and capable of handling up to 128k tokens using YARN extrapolation.

• Is SmolLM3 an open model?

  Yes, SmolLM3 is a fully open model with open weights and detailed training configurations available.

• What is the architecture of SmolLM3?

  SmolLM3 is a decoder-only transformer using GQA and NoPE with a 3:1 ratio.

• Can SmolLM3 be used for tool calling?

  Yes, SmolLM3 supports tool calling through XML or Python function calls.

• What frameworks are compatible with SmolLM3?

  SmolLM3 can be deployed using vLLM, SGLang, llama.cpp, ONNX, MLX, MLC, and ExecuTorch.

• What are the main use cases for SmolLM3?

  SmolLM3 is suitable for multilingual tasks, complex reasoning, and hybrid reasoning applications.