Description
NVIDIA NemotronLabs VoiceChat is an advanced AI model designed for real-time conversational interactions. It is a full-duplex speech model that integrates streaming speech understanding and generation within a single architecture. This model eliminates the need for multiple models or API handoffs, significantly reducing end-to-end latency and enhancing the natural flow of conversations.
Read More
The model operates on audio signals, which are encoded using a fast conformer module. These audio tokens are then processed by the Nemotron Nano V2 9B LLM backbone to predict text tokens, which are subsequently fed into a TTS decoder to generate the agent's speech. A separate output channel is utilized for predicting tool-calling scripts, making it the first open full-duplex model to support tool calling while maintaining a natural conversation flow.
NVIDIA NemotronLabs VoiceChat is particularly beneficial for researchers, developers, and professionals in the fields of natural language processing and speech technology. It is suitable for applications such as automatic speech recognition, text-to-speech synthesis, and voice assistant development. The model is optimized to run on NVIDIA GPU-accelerated systems, leveraging NVIDIA's hardware and software frameworks to achieve faster training and inference times.
The model's architecture is a hybrid Mamba/Transformer with 11 billion parameters, offering a remarkable trade-off between intelligence and latency. It supports global deployment and is governed by the OpenMDW License Agreement, version 1.1. The model was released on August 3, 2026, and has been trained on a diverse set of datasets, including both real and synthetic speech data.
Benchmarking results highlight its capabilities, with the model achieving second place among open full-duplex models on VoiceBench and Full-Duplex-Bench 1.0. It also excels in tool selection accuracy on Full-Duplex-Bench v3. Despite its advanced capabilities, users should be aware of known limitations identified through extensive internal testing.
- CategoryAI Models & LLMs
- DeploymentCloud
- PlatformsLinux
NVIDIA NemotronLabs VoiceChat Highlights
Real-time full-duplex speech model
Integrated speech understanding and generation
Single architecture reduces latency
Supports tool calling with natural flow
Optimized for NVIDIA GPU systems
11 billion parameters
Hybrid Mamba/Transformer architecture
Global deployment support
Getting Started with NVIDIA NemotronLabs VoiceChat
Access model: Visit Hugging Face page
Authenticate: Use API key
Set up environment: Configure NVIDIA GPU
Integrate via API: Use WebSocket interface
Optimise: Leverage CUDA libraries
NVIDIA NemotronLabs VoiceChat's Use Cases
- Speech Recognition
- Voice Assistants
- Text-to-Speech
- Tool Calling
- Real-Time Interaction
