QwenAudio/qwen-audio-agent

★ 3,138⑂ 332

A realtime voice runtime that keeps Agents talking, working, and present. Real-time Voice Runtime for AI Agents

About QwenAudio/qwen-audio-agent

QwenAudio/qwen-audio-agent is an open-source project on GitHub, mainly written in JavaScript. A realtime voice runtime that keeps Agents talking, working, and present. Real-time Voice Runtime for AI Agents It currently holds 3,138 stars and 332 forks with 16 open issues, and was last pushed on 2026-11-04 (repository created 2026-07-27).

Project Overview

Git Homed tracks it on the Audio Trending board and on the AI Audio Trending list.

GitHub Repository Details

Repository QwenAudio/qwen-audio-agent · default branch main · size 67805 KB · watchers 20 · source: GitHub REST API and repository README

README

Qwen Audio Agent

中文 | English | User Guide | Quickstart | Paper

CI npm node license arXiv WeChat

Agent Presence

Real conversation should not leave you waiting after a single sentence, nor should it grind to a halt just because the Agent is looking something up, calling a tool, or working on a task.

Conversation should keep flowing, and the Agent should always be present.

That is why we built qwen-audio-agent—a realtime voice runtime that keeps Agents talking, working, and present. Whether chatting with you, thinking through a problem, or working on a task, your Agent remains in the conversation. It listens, responds, and when the task is complete, naturally tells you:

"It's ready."

News

🎙️ Adds Qwen Audio 3.1 Realtime and unified frontend configuration; 🧩 supports updating running tasks and continuing completed work; 🎭 adds a runnable dual-connection digital-human example with replaceable rendering services; 📷 adds photo capture to WebUI. 🏗️ Rebuilt the orchestration runtime with a unified client protocol and ACP / A2A backend integration; 🎙️ expanded voice and video model support; 🧠 enhanced frontend tools, memory, and knowledge libraries; 💬 added a desktop conversation panel and remote mobile access; 🧩 added customer service, smart cockpit, digital human, and other examples; 📄 published the technical report. 🧩 Adds embeddable Gateway and Realtime Provider extensions; 🛠️ supports installing and managing Agent Skills; 📎 adds multimodal input to the TUI; 🎨 links pet animations to runtime states. 🧩 Desktop task cards show live Agent progress; 🔎 backend Agent selection is clearer and searchable; 🎙️ supports Qwen3.5-Omni Realtime frontend integration. 🎨 The orb opens up custom skins — import your own look, compatible with pet packs from the Awesome Codex Pet community gallery; 🪟 improved Windows backend Agent startup. ⏰ Adds scheduled reminders and progress reporting; 🗣️ adds the voice wake word ("你好千问"); 🐧 desktop build support for Linux; the desktop app now uses a data directory isolated from the CLI. 🎙️ Adds 🤗 speech-to-speech frontend integration, supporting fully local VAD, STT, LLM, and TTS. 🚀 First stable release, introducing a macOS desktop app with a built-in Gateway. 🌍 Project officially open-sourced; backend Agents unified under the ACP architecture.

Conversation Continues, Tasks Too

Conversation doesn't stop for background tasks; when a task completes, the result naturally returns to the current conversation:

Office Smart Cockpit

Core Features

Architecture

https://github.com/QwenAudio/qwen-audio-agent/blob/HEAD/qwen-audio-agent architecture https://github.com/QwenAudio/qwen-audio-agent/blob/HEAD/qwen-audio-agent reference architecture

Questions that can be answered directly are answered immediately; when tools or sustained processing are needed, the task is delegated to the backend Agent. Throughout, the user always faces the same assistant.

For the full design and module breakdown, see the architecture document.

Frontend and Backend Support

The voice frontend handles realtime conversation; the backend Agent executes tasks. They integrate independently and can be combined as needed.

Voice Frontends

| Voice frontend | Deployment | Setup | Features | | --- | --- | --- | --- | | Qwen Audio 3.0/3.1 Realtime | Cloud | Bailian API Key | Duplex voice, tool calling | | GPT-Live / OpenAI Realtime | Cloud | OpenAI API Key | — | | Google Gemini Live | Cloud | Google API Key | Live video input | | Qwen3.5-Omni Realtime | Cloud | Bailian API Key | Live video input | | Qwen3.8 Omni Flash Realtime | Cloud | Bailian API Key + workspace-specific endpoint | Live video input | | Doubao Seeduplex 3.0 Realtime | Cloud | Volcengine Speech API Key | — | | StepAudio 3 Realtime | Cloud | StepFun API Key | — | | Hugging Face Speech-to-Speech | Local | Start the service and set its URL | Configurable STT / LLM / TTS | | MiniCPM-o 4.5 | Local or cloud | Compatible service URL | Live video input, no tool calling |

Frontend connections share the same provider, model, API Key, service URL, and voice settings. Supported models and protocol parameters are maintained by the framework; see frontend configuration.

To connect another voice service, implement the Realtime Provider interface without changing the Gateway's core voice-session or backend-task logic.

Backend Agents

| Backend Agent | Integration | Setup | Rating | | --- | --- | --- | --- | | None | N/A | Frontend-only mode, no backend config needed | ★★★★★ | | Bare LLM | Built-in Chat Completions | OpenAI-compatible API key, URL, and model | Experimental | | Qwen Code | Native ACP | One-click install, user config required | ★★★★★ | | OpenCode | Native ACP | One-click install + Bailian config | ★★★★★ | | OpenClaw | Built-in ACP bridge | One-click install + Bailian config | ★★★★★ | | Qoder | Native ACP | One-click install, user config required | ★★★★★ | | MiniMax Code | Native ACP | One-click install, user config required | ★★★★☆ | | Kimi Code | Native ACP | One-click install, user config required | ★★★★★ | | Hermes | Native ACP | One-click install, user config required | ★★★★☆ | | CodeBuddy | Native ACP | One-click install, user config required | ★★★★☆ | | Codex | External ACP adapter | One-click install (base + adapter), user config required | ★★★★☆ | | Claude Code | External ACP adapter | One-click install (base + adapter), user config required | ★★★★☆ | | DeepSeek Harness | Native ACP | One-click install, DeepSeek API key required | ★★★★☆ | | Pi | External ACP adapter | One-click install (base + adapter), user config required | ★★★★☆ | | Muse Code | Native MSP adapter | Install Muse and its optional SDK on demand; user config required | ★★★☆☆ |

Ratings reflect current integration completeness, compatibility, and verification level: five stars indicate a thoroughly tested recommended integration; four stars indicate active development or not yet fully verified. For detailed configuration and capability boundaries, see the backend Agent documentation and configuration guide.

Installation

Requires Node.js 22.22.2+ or 24.15.0+, npm 10+. One-click install (recommended):

npm install -g qwen-audio-agent

For building from source, installing from GitHub, and obtaining a DashScope API Key, see the installation guide.

Quick Start

1. Create your config and fill in the API Key:

qwenaudio config
QWEN_AUDIO_REALTIME_API_KEY=your-key

Voice frontend model: optional, defaults to Qwen Audio 3.1 Realtime Plus

QWEN_AUDIO_REALTIME_MODEL=qwen-audio-3.1-realtime-plus

Backend Agent: optional, leave empty or set to none for frontend-only mode

AGENT_PROTOCOL=openclaw

Backend model: optional; explicit values use standard ACP, empty reuses Agent config

QWEN_AUDIO_AGENT_BACKEND_MODEL=qwen3.7-max

Before starting, create a key from the Bailian API Key page. Eligible new users can review the new-user free quota and check remaining usage on the model usage page. Quota and billing rules are subject to the current official Bailian documentation.

The example above uses the default DashScope voice frontend. See
Voice Frontends for other cloud and self-hosted options.

Backend credentials remain in the Agent's own configuration. For managed Bailian setup, see backend configuration.

With a visual-capable Realtime frontend, WebUI can explicitly stream bounded camera frames alongside live audio. See Realtime frontend configuration.

2. Start the Gateway, then open another terminal to start the TUI (or use qwenaudio webui for the browser UI):

qwenaudio        # Terminal 1: Gateway
qwenaudio tui    # Terminal 2: TUI

For full configuration options, local voice frontend setup, and TUI platform notes, see quick start, voice frontends, and TUI notes.

Desktop App

The desktop app provides a persistent floating voice orb with a built-in Gateway, automatic idle sleep, local voice wake, and customizable appearance. Download the installer for your platform from the releases page, or build from source:

npm run desktop:build:local      # macOS
npm run desktop:build:win        # Windows
npm run desktop:build:linux      # Linux (AppImage + deb, no signing)

For visuals, orb behavior, and build instructions, see the desktop documentation.

Examples and Scenario Expansion

The current qwen-audio-agent framework focuses on desktop productivity: users can keep talking with the Agent in realtime while delegating tool use, file work, code changes, and long-running tasks to the backend Agent.

This "foreground conversation + background task" design is not limited to desktop use. It can also expand to more scenarios where the Agent can both chat naturally and get real work done.

| Scenario | Description | Link | Status | | --- | --- | --- | --- | | Desktop | Voice chat, progress follow-up, tools, and background tasks. | [Docs][desktop-docs] | Available | | Smart cockpit | Vehicle control, navigation, music, weather, and services. | [Example][smart-cockpit-example] | Available | | X-Omni | Visual conversation, on-demand capture, optional observation and narration. | Example | Available | | AI Passport | Qwen Voice Bean on a hardware card, with voice conversation and backend tasks. Currently half-duplex only. | [Example][ai-passport-example] | Available | | Customer Service | Voice customer service for retail and airline scenarios. | Example | Available | | Embodied intelligence | Voice commands, action execution, inspection, and exception feedback. | TBD | Planned | | Livestream assistant | Audience interaction, product explanation, coupons, and risk reminders. | TBD | Planned |

[desktop-docs]: docs/desktop/overview.md [smart-cockpit-example]: examples/smart-cockpit [ai-passport-example]: examples/ai-passport

Community

You can start discussions directly in GitHub Issues.

For users in China, scan the QR codes below to join the WeChat group. If the group QR code is full or expired, scan either maintainer's personal QR code to be invited.

| WeChat Group | Personal | Personal | | :---: | :---: | :---: | | https://github.com/QwenAudio/qwen-audio-agent/blob/HEAD/WeChat group QR code | https://github.com/QwenAudio/qwen-audio-agent/blob/HEAD/Li Xu personal WeChat QR code | https://github.com/QwenAudio/qwen-audio-agent/blob/HEAD/Pigeon.Dan personal WeChat QR code |

Contributing and Security

License

Apache License 2.0

GitHub Stars & Activity

3,138Stars
332Forks
16Open issues
JavaScriptLanguage

GitHub Popularity

GitHub stars3,138
Forks332
Open issues16
Primary languageJavaScript
LicenseApache-2.0
Stars gained today0
Created2026-07-27
Last pushed2026-11-04

Trending History

Trending statusnot on today's boards

Related GitHub Projects

1

fspecii / ace-step-ui

JavaScript★ 5,231⑂ 0
→
2

techjarves / Portable-Local-Studio

JavaScript★ 2,040⑂ 0
→
3

huggingface / transformers

Python★ 169,134⑂ 0
→
4

debpalash / VoiceStudio

Python★ 112,289⑂ 0
→
5

RVC-Boss / GPT-SoVITS

Python★ 65,245⑂ 0
→
6

ggml-org / whisper.cpp

C++★ 55,265⑂ 0
→
7

coqui-ai / TTS

Python★ 46,240⑂ 0
→
8

OpenBMB / VoxCPM

Python★ 42,576⑂ 0
→

More Trending Repositories