Description
SoundStorm is an advanced AI model designed for highly efficient and non-autoregressive audio generation. It takes semantic tokens from AudioLM as input and utilizes bidirectional attention along with confidence-based parallel decoding to generate the tokens for a neural audio codec. This approach significantly accelerates audio production, achieving speeds two orders of magnitude faster than AudioLM's autoregressive generation.
Read More
This model excels in producing audio of equivalent quality and superior consistency in voice and acoustic conditions compared to its predecessors. A key capability is its ability to generate 30 seconds of audio in just 0.5 seconds on a TPU-v4. SoundStorm demonstrates scalability for longer audio sequences, enabling the synthesis of high-quality, natural dialogue segments. This is achieved by conditioning on a transcript annotated with speaker turns and a short voice prompt from the desired speaker.
When integrated with text-to-semantic modeling stages like SPEAR-TTS, SoundStorm can synthesize natural-sounding dialogues. Users can precisely control the spoken content through transcripts, the speaker's voice via short prompts, and the flow of conversation through speaker turn annotations. For instance, synthesizing 30-second dialogue segments takes only 2 seconds on a single TPU-v4, even for text and speakers unseen during training.
SoundStorm also offers capabilities for both unprompted and prompted audio generation. In the unprompted scenario, it samples different speakers, while in the prompted case, it maintains the speaker's voice with high fidelity. This makes it a powerful tool for various audio synthesis tasks. Compared to baseline methods like AudioLM and greedy decoding, SoundStorm offers improved acoustic consistency and better voice preservation from prompts, alongside higher audio quality.
The model's efficient generation pipeline makes research in audio generation more accessible. While the ability to mimic voices raises concerns about potential misuse, such as impersonation, safeguards are being developed. SoundStorm's generated audio has been shown to be detectable by dedicated classifiers, mitigating some risks. The research aims to advance audio generation technology responsibly.
- CategoryAI Models & LLMs
- DeploymentCloud
- PlatformsWeb
SoundStorm Highlights
Efficient non-autoregressive audio generation
Generates audio two orders of magnitude faster than AudioLM
Maintains high consistency in voice and acoustic conditions
Synthesizes natural dialogue segments with speaker control
Accepts semantic tokens from AudioLM as input
Utilizes bidirectional attention and confidence-based parallel decoding
Scalable to longer audio sequences
Supports synthesis of dialogue with control over content, voice, and speaker turns
Can generate audio from text-to-semantic models
Demonstrates high acoustic consistency and voice preservation in prompted generation
Produces higher quality audio compared to greedy decoding baselines
Getting Started with SoundStorm
Access model: Obtain access to the SoundStorm model and its associated components.
Set up environment: Configure the necessary software and hardware environment for running the model.
Input semantic tokens: Provide the semantic tokens generated by AudioLM to SoundStorm.
Integrate via API: Utilize the model's API for seamless integration into audio generation pipelines.
Provide conditioning information: Supply transcripts, speaker turn annotations, and voice prompts for controlled synthesis.
Generate audio: Execute the generation process to produce neural audio codec representations.
Decode audio: Convert the generated representations into audible sound using a neural audio codec.
SoundStorm's Use Cases
- Dialogue Synthesis
- Voice Cloning
- Fast Audio Generation
- Speech Synthesis
- Audio Content Creation
- Accessibility Tools
- Research Acceleration
