Make AI videos with audio of anyone. Free & offline.
Recent years have witnessed significant progress in audio-driven human animation. However, critical challenges remain in:
1) Generating highly dynamic videos while preserving character consistency,
2) Achieving precise emotion alignment between characters and audio,
3) Enabling multi-character audio-driven animation.
To address these challenges, we propose HunyuanVideo-Avatar, a multimodal diffusion transformer (MM-DiT)-based model capable of simultaneously generating dynamic, emotion-controllable, and multi-character dialogue videos. Concretely, HunyuanVideo-Avatar introduces three key innovations:
1) A character image injection module is designed to replace the conventional addition-based character conditioning scheme, eliminating the inherent condition mismatch between training and inference. This ensures the dynamic motion and strong character consistency.
2) An Audio Emotion Module (AEM) is introduced to extract and transfer the emotional cues from an emotion reference image to the target generated video, enabling fine-grained and accurate emotion style control.
3) A Face-Aware Audio Adapter (FAA) is proposed to isolate the audio-driven character with latent-level face mask, enabling independent audio injection via cross-attention for multi-character scenarios.
These innovations empower HunyuanVideo-Avatar to surpass state-of-the-art methods on benchmark datasets and a newly proposed wild dataset, generating realistic avatars in dynamic, immersive scenarios.
https://hunyuanvideo-avatar.github.io
https://github.com/Tencent-Hunyuan/HunyuanVideo-Avatar
https://github.com/deepbeepmeep/Wan2GP
https://git-scm.com/downloads
https://www.anaconda.com/docs/getting-started/miniconda/install
00:00 - Hunyuan Video Avatar
01:12 - Official demos
03:06 - How to use online
04:28 - Personal demos
11:37 - Veo3 and alternatives
13:52 - Vidu AI video generator
15:10 - How to install HunyuanVideo Avatar locally
17:33 - Git
18:28 - Installation continued
19:34 - Conda
22:15 - Installation continued
25:28 - How to use HunyuanVideo Avatar locally