The gist

Open-LLM-VTuber is an open-source project for creating a voice-interactive AI companion with a live 2D avatar. Developed by its GitHub community, it allows users to build a personalized virtual character that can run completely offline on Windows, macOS, or Linux. The project aims to provide an open-source alternative to closed-source AI VTubers, enabling users to customize their companion's appearance, persona, and voice for interactive conversations and desktop companionship.

What it does

Run completely offline using local models for LLM, ASR, and TTS functions.
Perceive visual input from cameras, screen recordings, or screenshots.
Support voice interruption for more natural conversation flow.
Customize the avatar's appearance with Live2D models and persona via prompts.
Operate in a 'desktop pet' mode with a transparent, click-through background.
Integrate with a wide range of models, including Ollama, OpenAI-compatible APIs, and local GGUF files.

How it works

Users interact with the AI companion by speaking into a microphone. The application uses a speech recognition model to transcribe the audio, an LLM to generate a response, and a text-to-speech model to voice the reply. The output is delivered through a Live2D avatar in either a web interface or a desktop client. As an open-source project, it is free to download and run on a local machine, supporting a wide variety of pluggable AI models.

Best for

This tool is best for developers and AI hobbyists who want to build and customize their own interactive, self-hosted VTuber or virtual AI companion using open-source models.

Watch out for

The project is in its early stages and undergoing a complete rewrite for v2.0, with the current v1 branch only receiving bug fixes. Initial setup can be technical, requiring configuration for features like remote access.