Open Avatar Chat

中文 | English

A modular interactive digital human conversation implementation.

🤗 Demo  |   Demo  |  💬 WeChat

• Multimodal language model: Supports multimodal language models including text, audio, video, etc.
• Modular design: Uses modular design, allowing flexible component replacement to achieve different function combinations.

• [2025.08.19] ⭐️⭐️⭐️ Version 0.5.1 Released:
  • LiteAvatar support multi-session. Please refer to LiteAvatar configuration part below.
  • Add support for the Qwen-Omni multimodal model, using the Qwen-Omni-Realtime API service of BaiLian. For the configuration file, please refer to this config.
• [2025.08.12] ⭐️⭐️⭐️ Version 0.5.0 Released:
  • Modified to a separated frontend and backend version. The frontend repositoryOpenAvatarChat-WebUIhas been added to facilitate custom frontend interfaces and expand interactions.
  • Added basic support for calling dify, currently only supporting the chatflow version.
• [2025.06.12] ⭐️⭐️⭐️ Version 0.4.1 Released:
  • Added support for MuseTalk, including customizable videos for personalized avatars.
  • Released 50 new LiteAvatar styles featuring a variety of professional roles. Please refer to LiteAvatarGallery.
• [2025.04.18] ⭐️⭐️⭐️ Version 0.3.0 Released:
  • 🎉🎉🎉 Congratulations to the LAM team on their paper being accepted to SIGGRAPH 2025! 🎉🎉🎉
  • Added support for LAM in digital humans, enabling concurrent configuration when LAM is selected. TTS now supports edge_tts and BaiLian CosyVoice.
  • Updated dependency management approach based on UV and handler modules, supporting direct execution or using Docker.
  • CSS responsive layout updated.
• [2025.04.14] ⭐️⭐️⭐️ Version 0.2.2 released:
  • 100 new avatars released, visit LiteAvatarGallery
  • Run LiteAvatar use GPU by default
• [2025.04.07] ⭐️⭐️⭐️ Version 0.2.1 released:
  • Added support for history logging
  • Support for text input
  • Camera requirement removed at startup
  • Optimized modular loading method
• [2025.02.20] ⭐️⭐️⭐️ Version 0.1.0 released:
  • Modular real-time interactive digital human
  • Supports MiniCPM-o as a multimodal language model with cloud API options

• Refine documentation and video tutorials
• Integrate Live2D digital humans
• Integrate 3D digital humans

We have deployed a demo service on ModelScope and 🤗 HuggingFace . The audio part is implemented using SenseVoice + Qwen-VL + CosyVoice. Now you can switch between LiteAvatar and LAM. Feel free to try it out.

• Wechat Group

Frequently asked questions encountered during the course of the project can be found at link

• 🔥 Core Highlights
• 📢 News
  • Changelog
  • Todo List
• Demo
  • Try it Online
  • Demo Video
• Community
• 🚨 FAQ
• Overview
  • Introduction
  • Requirements
  • Performance
  • Component Dependencies
  • Pre-set Modes
• 🚀 Get Started
  • Select a config
    • chat_with_lam.yaml
    • chat_with_qwen_omni.yaml
    • chat_with_openai_compatible.yaml
    • chat_with_openai_compatible_edge_tts.yaml
    • chat_with_openai_compatible_bailian_cosyvoice.yaml
    • chat_with_openai_compatible_bailian_cosyvoice_musetalk.yaml
    • chat_with_minicpm.yaml
  • Local Execution
    • UV Installation
    • Dependency Installation
      • Install all dependencies
      • Install dependencies for the required mode only
    • Run
  • Docker Execution
    • Docker Compose
• Handler Dependencies Installation Notes
  • Server Rendering RTC Client Handler
  • LAM Client Rendering Handler
    • Select the Avatar Asset
  • OpenAI Compatible LLM Handler
  • Qwen-Omni Speech2Speech Handler
  • MiniCPM Omni Speech2Speech Handler
    • Models used
  • Bailian CosyVoice Handler
  • CosyVoice Local Inference Handler
  • Edge TTS Handler
  • LiteAvatar Avatar Handler
    • Model Dependencies
  • Dify Chatflow Handler
    • Configuration
  • LAM Avatar Driver Handler
    • Models used
  • MuseTalk Avatar Handler
    • Model Dependencies
    • Configuration
    • Run
• Optional Deployment
  • Prepare ssl certificates
  • TURN Server
    • Local Installation
    • Docker Installation
• Configuration
• Community Thanks
• Star History
• Citation

Open Avatar Chat is a modular interactive digital human dialogue implementation that can run full functionality on a single PC. It currently supports MiniCPM-o as a multimodal language model or using cloud-based APIs to replace the conventional ASR + LLM + TTS setup. The architecture of these two modes is illustrated in the diagram below. For more pre-set modes, see below.

• Python version >=3.10, <3.12
• CUDA-enabled GPU. The CUDA version supported by the NVIDIA driver needs to be >= 12.4.
• The unquantized multimodal language model MiniCPM-o requires more than 20GB of VRAM.
• The digital human component can perform inference using GPU/CPU. The test device is an i9-13980HX CPU, achieving up to 30 FPS for CPU inference.

In our tests, using a PC equipped with an i9-13900KF processor and Nvidia RTX 4090 graphics card, we recorded the response delay. After ten tests, the average delay was about 2.2 seconds. The delay time is the interval from the end of the user's speech to the start of the digital human's speech, including RTC two-way data transmission time, VAD (Voice Activity Detection) stop delay, and the entire process computation time.

Type	Open Source Project	GitHub Link	Model Link
RTC	HumanAIGC-Engineering/gradio-webrtc
WebUI	HumanAIGC-Engineering/OpenAvatarChat-WebUI
VAD	snakers4/silero-vad
LLM	OpenBMB/MiniCPM-o		🤗
LLM-int4	OpenBMB/MiniCPM-o		🤗
Avatar	HumanAIGC/lite-avatar
TTS	FunAudioLLM/CosyVoice
Avatar	aigc3d/LAM_Audio2Expression		🤗
	facebook/wav2vec2-base-960h		🤗
Avatar	TMElyralab/MuseTalk

The functionalities of OpenAvatarChat will follow the config specified during startup. We provided several sample config files under the config folder.

This config uses LAM generated gaussion splatting asset as client-side rendered avatar. With api based openai compatible llm and tts from Bailian platform, only vad and asr handlers are run locally, so this is the lightest config choice, which supports multiple connection on single service.

Local speech-to-speech dialogue generation is implemented using Qwen-Omni, with the Qwen-Omni-Realtime API (from Alibaba Cloud BaiLian).

This config use openai-compatible api as llm provider and CosyVoice as local tts model.

This config use Edge TTS, it does not need an API Key of Bailian.

Both LLM and TTS are provided by API, it is the lightest config for LiteAvatar.

Both LLM and TTS are provided by API, while the 2D digital human uses MuseTalk for inference. By default, it uses GPU for inference and CPU inference is not currently supported.

Use MiniCPM-o-2.6 as audio2audio chat model, it need enough VRAM and GPU computaion power.

sudo apt install git-lfs
git lfs install 

git submodule update --init --recursive

It is recommended to install UV, using UV for local environment management.

Official standalone installer

# On Windows.
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
# On macOS and Linux.
curl -LsSf https://astral.sh/uv/install.sh | sh

PyPI installation

# With pip.
pip install uv
# Or pipx.
pipx install uv

uv sync --all-packages

uv venv --python 3.11.11

uv pip install setuptools pip

uv run install.py --uv --config <absolute path to config file>.yaml

./scripts/post_config_install.sh --config <absolute path to config file>.yaml

uv run src/demo.py --config <absolute path to config file>.yaml

./build_and_run.sh --config <relative path to config file>.yaml

# Clone the project repository
git clone https://github.com/HumanAIGC-Engineering/OpenAvatarChat.git

# Navigate to the project directory
cd OpenAvatarChat

# Download all submodules
git submodule update --init --recursive --depth 1

# Download models required for LiteAvatar
# The script uses ModelScope to download models by default. 
# If ModelScope is not installed locally, install it first with: pip install modelscope
bash scripts/download_liteavatar_weights.sh

# Download models required for LAM
git clone --depth 1 https://www.modelscope.cn/AI-ModelScope/wav2vec2-base-960h.git ./models/wav2vec2-base-960h
wget https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/aigc3d/data/LAM/LAM_audio2exp_streaming.tar -P ./models/LAM_audio2exp/
tar -xzvf ./models/LAM_audio2exp/LAM_audio2exp_streaming.tar -C ./models/LAM_audio2exp && rm ./models/LAM_audio2exp/LAM_audio2exp_streaming.tar

# Download models required for MuseTalk
bash scripts/download_musetalk_weights.sh

# Build the Docker image (with CUDA 12.8)
bash build_cuda128.sh

# (Optional) If using the Bailian API:
# Create a .env file in the project root and add your API key
touch .env 
# Edit the .env file manually to add: DASHSCOPE_API_KEY=sk-xxxxx

# Run the Docker container
# Replace the config file with your desired one (example below)
bash run_docker_cuda128.sh --config config/chat_with_openai_compatible_bailian_cosyvoice_musetalk.yaml

Supports using Docker Compose to start the openavatarchat service along with a coturn service launched via Docker image in one go.

# Start services
docker compose up

# Stop services
docker compose down

Currently there is no extra dependency or essential configs.

Client rendering handler is derived from Server Rendering RTC Client Handler. It supports multi-connection. Client avatar asset can be selected in handler config.

LAM avatar asset can be generated by the LAM project (The ready-to-use generation pipeline is not ready yet. Stay tunned!). OpenAvatarChat provides 4 sample asset. They can be found under src/handlers/client/h5_rendering_client/lam_samples. The selected asset should be set to the asset_path field in the handler config. You can use one of the sample asset, a your own asset that created by LAM, please refer to the follow handler config sample:

LamClient:
  module: client/h5_rendering_client/client_handler_lam
  asset_path: "lam_samples/barbara.zip"
  concurrent_limit: 5

Local llm handler has relatively high startup requirements. If you already have an available LLM api_key, you can start it this way to experience interactive digital humans. Modify the corresponding config, such as the LLMOpenAICompatible configuration in config/chat_with_openai_compatible.yaml. The invocation method in the code uses the standard OpenAI approach, which should theoretically be compatible with similar setups.

LLMOpenAICompatible: 
  model_name: "qwen-plus"
  system_prompt: "You are an AI digital human. Respond to my questions briefly and insert punctuation where appropriate."
  api_url: 'https://dashscope.aliyuncs.com/compatible-mode/v1'
  api_key: 'yourapikey' # default=os.getenv("DASHSCOPE_API_KEY")

client = OpenAI(
      api_key= self.api_key, 
      base_url=self.api_url,
  )
completion = client.chat.completions.create(
    model=self.model_name,
    messages=[
       self.system_prompt,
        {'role': 'user', 'content': chat_text}
    ],
    stream=True
    )

The capabilities of Qwen-Omni are integrated via Alibaba Cloud BaiLian's API. Currently, only the manual mode is supported. Voice Activity Detection (VAD) is executed by the local SileroVad model. Additionally, due to the poor quality and unreliability of input_transcription results in manual mode, an extra SenseVoice module has been added exclusively for echoing conversation records. For the complete configuration file, please refer to chat_with_qwen_omni.yaml. Among them, the avatar module supports a choice between AvatarMusetalk and LiteAvatar.

In this project, MiniCPM-o-2.6 can be used as a multimodal language model to provide dialogue capabilities for digital humans. Users can download the relevant model as needed from Huggingface or Modelscope. It is recommended to directly download the model to /models/. The default configuration points to this path, so if the model is placed elsewhere, you need to modify the configuration file. There is a corresponding model download script in the scripts directory, which can be used in a Linux environment. Please run the script in the project root directory:

scripts/download_MiniCPM-o_2.6.sh

scripts/download_MiniCPM-o_2.6-int4.sh

Bailian provides CosyVoice API, it can be used as an alternative to local tts inference handler. Though it requires an Bailian API Key, it reduces quite amount of system requirments. Sample handler config looks like this:

CosyVoice:
  module: tts/bailian_tts/tts_handler_cosyvoice_bailian
  voice: "longxiaocheng"
  model_name: "cosyvoice-v1"
  api_key: 'yourapikey' # default=os.getenv("DASHSCOPE_API_KEY")

Same as OpenAI Compatible LLM Handler, api_key can be set in the handler config or from environment variables.

When using CosyVoice locally as TTS on Windows, it is necessary to combine Conda and UV for installation. The specific dependency installation and execution process are as follows:

1. Install Anaconda or Miniconda

conda create -n openavatarchat python=3.10
conda activate openavatarchat
conda install -c conda-forge pynini==2.1.6

2. Set the environment variable indexed by UV to the Conda environment

# cmd
set VIRTUAL_ENV=%CONDA_PREFIX%
# powershell 
$env:VIRTUAL_ENV=$env:CONDA_PREFIX

3. When installing dependencies and running with UV, add the --active parameter to prioritize the use of the activated virtual environment

# Install dependencies
uv sync --active --all-packages
# Install required dependencies only
uv run --active install.py --uv --config config/chat_with_openai_compatible.yaml
# Run CosyVoice 
uv run --active src/demo.py --config config/chat_with_openai_compatible.yaml

OpenAvatarChat integrated Microsoft Edge TTS, it is inference on the cloud and api key is not esstential, the sample handler config looks like:

Edge_TTS:
  module: tts/edgetts/tts_handler_edgetts
  voice: "zh-CN-XiaoxiaoNeural"

LiteAvatar is integarted to provide 2D avatar feature. Currenty, 100 avatar assets are provided on modelscope project LiteAvatarGallery, please refer to this project for detail.

Model weights have to be downloaded before you use LiteAvatar, LiteAvatar source code includes a model download script. For convenience, a script for Linux enviroments also provided in the scripts directory of this repo. You can call this script under project root:

bash scripts/download_liteavatar_weights.sh

The project currently integrates Dify's Chatflow. Users can create a Chatflow in Dify, and after filling in the generated Chatflow application's api_url and api_key, they can use Dify's Chatflow for conversation.

 Dify:
      enabled: True
      module: llm/dify/llm_handler_dify
      enable_video_input: False # Allow camera input, ensure application supports vision and accepts file inputs
      api_key: '' #your dify api key
      api_url: 'http://localhost/v1' # your dify api url
 

LiteAvatar can be run on CPU as well as GPU. If other GPU heavy handlers are used, let liteavatar run on cpu may be a good choice.

Sample handler config looks like:

LiteAvatar:
  module: avatar/liteavatar/avatar_handler_liteavatar
  avatar_name: 20250408/sample_data
  fps: 25
  use_gpu: true

LiteAvatar supports multiple sessions on a single machine. To enable this feature, refer to config/chat_with_openai_compatible_bailian_cosyvoice.yaml and set the default.chan_engine.concurrent_limit parameter. By configuring this parameter, you predefine the maximum number of concurrent sessions supported at startup.

Please note that running multiple sessions significantly increases system resource demands. When LiteAvatar runs on a GPU, each concurrent session consumes approximately 3GB of GPU memory. Setting concurrent_limit too high may lead to out-of-memory errors. Please adjust the number of concurrent sessions according to your machine's hardware specifications.

• facebook/wav2vec2-base-960h 🤗
  • Download from huggingface, ensure lfs is installed properly，run following cmd under root of the project:

  git clone --depth 1 https://huggingface.co/facebook/wav2vec2-base-960h ./models/wav2vec2-base-960h
  

  • Download from modelscope, ensure lfs is installed properly，run following cmd under root of the project:

  git clone --depth 1 https://www.modelscope.cn/AI-ModelScope/wav2vec2-base-960h.git ./models/wav2vec2-base-960h
  

• LAM_audio2exp 🤗
  • Download form huggingface, ensure lfs is installed properly，run following cmds under root of the project:

  wget https://huggingface.co/3DAIGC/LAM_audio2exp/resolve/main/LAM_audio2exp_streaming.tar -P ./models/LAM_audio2exp/
  tar -xzvf ./models/LAM_audio2exp/LAM_audio2exp_streaming.tar -C ./models/LAM_audio2exp && rm ./models/LAM_audio2exp/LAM_audio2exp_streaming.tar
  

  • If huggingface is unreachable, it can also be downloaded from oss, run following cmds under root of the project:

  wget https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/aigc3d/data/LAM/LAM_audio2exp_streaming.tar -P ./models/LAM_audio2exp/
  tar -xzvf ./models/LAM_audio2exp/LAM_audio2exp_streaming.tar -C ./models/LAM_audio2exp && rm ./models/LAM_audio2exp/LAM_audio2exp_streaming.tar
  

The project currently integrates the latest MuseTalk 1.5 (previous versions are not tested). This version supports custom avatars, which can be selected by modifying the avatar_video_path parameter.

• MuseTalk source code includes a model download script. To keep the directory structure consistent, a modified script is provided in the scripts directory for Linux environments. The original MuseTalk code uses relative paths for loading; although adaptations have been made, some code cannot be configured via input parameters. Do not change the model download location. Run the script from the project root:

  bash scripts/download_musetalk_weights.sh
  

• The MuseTalk source code will download a model s3fd-619a316812.pth on first startup, which is not included in the download script. The initial download might be slow.

By setting avatar_video_path, you can customize the base video for the digital human. To facilitate users without digital human material, we provide a tool that allows MuseTalk users to use digital human materials provided by LiteAvatar. The script file is scripts/download_avatar_model.py, and the model list can be viewed at LiteAvatarGallery.

Usage Method:

# 1. View help information
python scripts/download_avatar_model.py --help

# 2. Download the specified digital human model
python scripts/download_avatar_model.py -m "20250612/P1rcvIW8H6kvcYWNkEnBWPfg"

# 3. View the list of downloaded models
python scripts/download_avatar_model.py -d

# Output example:
# Downloaded Models List:
# avatar_name（for LiteAvatar config）    avatar_video_path（for Musetalk config）
# --------------------------------------------------------------------------------
# 20250612/P1rcvIW8H6kvcYWNkEnBWPfg       resource/avatar/liteavatar/20250612/P1rcvIW8H6kvcYWNkEnBWPfg/bg_video_silence.mp4

• Avatar selection: MuseTalk source includes two default avatars. You can select by modifying the avatar_video_path parameter. The system will prepare data on first load and cache it for subsequent runs. You can force regeneration by setting force_create_avatar: true. The avatar_model_dir parameter specifies where to save avatar data (default: models/musetalk/avatar_model).
• Frame rate: Although MuseTalk documentation claims 30fps on V100, our adaptation (referencing realtime_inference.py) does not reach this in practice. We recommend fps: 20, but you can adjust based on your GPU. If you see the warning [IDLE_FRAME] Inserted idle during speaking in logs, it means actual inference fps is lower than set fps.
• Batch size: Increasing batch_size can improve throughput, but too large a batch may slow first-frame response. The minimum batch_size for inference is 2. If you set it to 1, an error will appear in the log[IDLE_FRAME] 1 validation error for AvatarMuseTalkConfig，batch_size - Input should be greater than or equal to 2 [type=greater_than_equal, input_value=1, input_type=int]

Sample config:

Avatar_MuseTalk:
  module: avatar/musetalk/avatar_handler_musetalk
  fps: 20  # Video frame rate
  batch_size: 2  # Batch processing frame count, must be greater than 2
  avatar_video_path: "src/handlers/avatar/musetalk/MuseTalk/data/video/sun.mp4"  # Initialization video path
  avatar_model_dir: "models/musetalk/avatar_model"  # Default avatar model directory
  force_create_avatar: false  # Whether to force regenerate digital human data
  debug: false  # Whether to enable debug mode
  ... # See AvatarMuseTalkConfig for more parameters

• Docker

bash build_cuda128.sh

bash run_docker_cuda128.sh --config config/chat_with_openai_compatible_bailian_cosyvoice_musetalk.yaml

• Local deployment

The order of commands for installing dependencies locally is as follows:

uv venv --python 3.11.11Add commentMore actions

./scripts/pre_config_install.sh --config config/chat_with_openai_compatible_bailian_cosyvoice_musetalk.yaml

uv run install.py --uv --config config/chat_with_openai_compatible_bailian_cosyvoice_musetalk.yaml

./scripts/post_config_install.sh --config config/chat_with_openai_compatible_bailian_cosyvoice_musetalk.yaml

Note: The mmcv installed by uv by default may report an error "No module named 'mmcv._ext'" during actual runtime. Refer to MMCV-FAQ. The solution is:

uv pip uninstall mmcv
uv run mim install mmcv==2.2.0 --force

When running the MuseTalk source code for the first time, it will automatically download a model called s3fd-619a316812.pth. This model is now integrated into the download script. It has already been mapped when starting with Docker. However, when running locally, you need to manually perform the mapping again.

# linux
ln -s $(pwd)/models/musetalk/s3fd-619a316812/* ~/.cache/torch/hub/checkpoints/

To start the program:

uv run src/demo.py --config config/chat_with_openai_compatible_bailian_cosyvoice_musetalk.yaml

Since we use rtc to stream the video and audio, if not linked from localhost an ssl certificates is needed, user can put exist ones into the ssl_certs folder and config them in the config file or create a new self signed one with the provided script. Run the script under project root to put the result into proper position.

scripts/create_ssl_certs.sh

If you encounter a continuous waiting state after clicking "Start Conversation", it may be due to NAT traversal issues in your deployment environment (such as deployment on cloud machines). In this case, data relay is required. On Linux systems, you can use coturn to set up a TURN server.

Follow these steps to install, start, and configure coturn on the same machine:

• Run the installation script

$ chmod 777 scripts/setup_coturn.sh
# scripts/setup_coturn.sh

• Modify the config file, add the following configuration and start the service

default:
  chat_engine:
    handler_configs:
      RtcClient: # If using Lam, this config should be LamClient
        turn_config:
          turn_provider: "turn_server"
          urls: ["turn:your-turn-server.com:3478", "turns:your-turn-server.com:5349"]
          username: "your-username"
          credential: "your-credential"

• Ensure that the firewall (including cloud machine security group policies) opens the ports required by coturn

You can use the Dockerized coturn service. For details, please refer to the docker compose section to start all services together.

The default parameter will load config from <project_root>/configs/chat_with_minicpm.yaml. Config can be loaded from other file by add the --config parameter.

uv run src/demo.py --config <absolute-path-to-the-config>.yaml

Configurable parameters are listed here：

Current implemented handler provide following configs:

• VAD

• LLM

ASR FunASR Model

LLM Plain Text Model

TTS CosyVoice Model

LiteAvatar Digital Human

• Grateful to the volunteers, “十字鱼”, for sharing a video on Bilibili featuring a one-click installation package, along with the download link. (The extraction code is included in the video description—take a close look!) One-click package
• Grateful to the volunteers, "W&H", for the Quark one-click package windows version: extract code a79V and linux version: extract code: E8Kq
• Grateful to the volunteers, "W&H", for the source code zipQuark:extract code 9iNy and BaiDu:extract code xrxr

If you found OpenAvatarChat helpful in your research/project, we would appreciate a Star⭐ and citation✏️

@software{avatarchat2025,
  author = {Gang Cheng, Tao Chen, Feng Wang, Binchao Huang, Hui Xu, Guanqiao He, Yi Lu, Shengyin Tan},
  title = {OpenAvatarChat},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/HumanAIGC-Engineering/OpenAvatarChat}
}