DreamTalk

Description

DreamTalk is an official implementation of a diffusion-based audio-driven expressive talking head generation framework. This tool is designed to produce high-quality talking head videos that accurately reflect diverse speaking styles and emotions. Its core capability lies in its robust performance across a wide array of inputs, including standard speech, songs, multilingual audio, and even noisy audio signals. Furthermore, DreamTalk demonstrates resilience when processing out-of-domain portraits, making it a versatile solution for various applications.

The framework leverages diffusion probabilistic models to achieve its expressive and realistic video generation. Users can install the project using conda and pip, following specific version requirements for PyTorch, torchvision, torchaudio, and other dependencies. The installation process involves creating a dedicated conda environment and then installing the necessary packages from a requirements file.

For users interested in obtaining the pre-trained checkpoints, public download access has been ceased due to social impact considerations. Interested parties must request checkpoints via email, explicitly consenting to their use solely for academic research purposes. Once obtained, checkpoints should be placed in the designated 'checkpoints' folder.

Inference with DreamTalk involves running a Python script with several key parameters. These include paths to the input audio file (supporting various formats like wav, mp3, m4a, and mp4), a reference style clip, a head pose sequence, and the input portrait image. Additional parameters like 'cfg_scale' control the intensity of speaking styles, while 'max_gen_len' sets the maximum video generation duration. The output video is saved in the 'output_video' folder, with intermediate results in a temporary folder.

DreamTalk also offers ad-hoc solutions to improve video resolution. Two methods are suggested: CodeFormer for up to 1024x1024 resolution, though it is slower and may have temporal inconsistency issues, and the Temporal Super-Resolution Model from MetaPortrait for 512x512 resolution with faster performance and temporal coherence, though it might reduce facial emotion intensity. The project acknowledges and builds upon preceding works in the field, citing relevant research and providing a citation entry for academic use.

DreamTalk's Core Features

• Diffusion-based audio-driven talking head generation

• High-quality video output with diverse speaking styles

• Robust performance with various audio inputs (speech, songs, noisy audio)

• Handles out-of-domain portraits

• Supports multiple languages

• Provides official implementation code

• Includes inference script for video generation

• Offers ad-hoc solutions for resolution enhancement (CodeFormer, MetaPortrait)

• Adjustable parameters for style intensity and generation length

• Supports CPU inference

Getting Started with DreamTalk

1. 1

   Clone: Clone the DreamTalk repository from GitHub.

2. 2

   Install dependencies: Create a conda environment and install required packages using the provided requirements.txt.

3. 3

   Download Checkpoints: Request checkpoints via email for academic research and place them in the 'checkpoints' folder.

4. 4

   Prepare Inputs: Gather input audio, reference style clips, head pose sequences, and portrait images.

5. 5

   Configure Inference: Specify input paths and parameters like cfg_scale and max_gen_len in the inference script.

6. 6

   Execute Inference: Run the inference script (e.g., python inference_for_demo_video.py) to generate talking head videos.

7. 7

   Improve Resolution (Optional): Apply CodeFormer or MetaPortrait for enhanced video resolution.

8. 8

   Utilize for Research: Employ the generated videos for academic research purposes.

DreamTalk's Use Cases

• Expressive Talking Head Generation
• Audio-Visual Synthesis
• Research in AI Animation
• Cross-lingual Talking Heads
• Style Transfer for Faces
• Handling Noisy Audio

FAQ from DreamTalk

• What is DreamTalk?

  DreamTalk is an official implementation of a diffusion-based framework for generating expressive talking head videos from audio, designed for academic research.

• What kind of audio inputs does DreamTalk support?

  DreamTalk supports a diverse range of audio inputs, including speech in multiple languages, songs, and noisy audio.

• Can DreamTalk generate talking heads from any portrait image?

  Yes, DreamTalk exhibits robust performance with diverse inputs, including out-of-domain portraits, though frontal portraits with faces directly facing forward generally yield better results.

• How can I install DreamTalk?

  Installation involves creating a conda environment, activating it, and then installing dependencies using pip install -r requirements.txt, along with specific PyTorch and other library versions.

• How do I obtain the pre-trained checkpoints for DreamTalk?

  Public download access to checkpoints has been ceased. You must request them via email to mayf18@mails.tsinghua.edu.cn, agreeing to use them solely for academic research.

• What parameters are important for running the inference script?

  Key parameters include paths to audio, style clip, pose, and source image, as well as 'cfg_scale' for style intensity and 'max_gen_len' for video duration.

• Can DreamTalk be run on a CPU?

  Yes, you can run DreamTalk on a CPU by adding the '--device=cpu' argument to the command line.

• What are the options for improving the resolution of generated videos?

  Ad-hoc solutions include using CodeFormer for up to 1024x1024 resolution or the Temporal Super-Resolution Model from MetaPortrait for 512x512 resolution.

• What is the intended use of DreamTalk?

  This method is intended for RESEARCH/NON-COMMERCIAL USE ONLY.

• Where can I find sample inputs for DreamTalk?

  Sample inputs are provided in the 'data' folder within the repository, though specific songs used in samples cannot be included due to copyright.