mirror of
https://github.com/78/xiaozhi-esp32.git
synced 2026-07-21 02:05:52 +00:00
8755a658a8
* Enhance Otto Robot camera support by adding configuration for OV3660. Updated config.h to define camera types and GPIO settings, modified config.json to include new camera options, and refactored otto_robot.cc for improved camera detection and initialization logic. * fix: 移除 OttoEmojiDisplay 构造函数中的 SetTheme 调用以修复 LoadProhibited 崩溃 Made-with: Cursor * refactor: improve audio service error handling and codec timeout management - Updated AudioService to prevent input task termination on read timeout, introducing a delay instead. - Enhanced NoAudioCodec to implement a read timeout for I2S channel reads. - Adjusted WebSocketControlServer to set a control port for improved socket management. - Added manufacturer information to the config.json for waveshare ESP32-Touch-LCD-3.5. * fix(otto): WebSocket direct clients not receiving MCP responses When a browser connects directly to the WebSocket control server (port 8080) and sends a JSON-RPC request, the MCP response was routed through Application::SendMcpMessage -> protocol_->SendMcpMessage, which sends it to the cloud protocol channel. As a result, the direct WebSocket client never received the response, while the WeChat mini-program could because it communicates via the cloud. Fix: - Add BroadcastMessage() to WebSocketControlServer, using httpd_queue_work + httpd_ws_send_frame_async to asynchronously send responses back to all connected clients on port 8080 - Add RegisterMcpBroadcastCallback() to Application, allowing an additional MCP send callback to be registered; SendMcpMessage() now invokes it alongside the cloud protocol - Register the broadcast callback in OttoRobot after the WebSocket server starts successfully Also add WebSocket direct-connect API documentation to README.md with complete JSON-RPC 2.0 command examples. * fix(otto-robot): migrate camera backend and set safe dark default theme - Migrate `otto-robot` camera backend from `EspVideo` to `Esp32Camera` to improve capture stability after reboot/power cycle. - Keep runtime sensor detection for both OV2640 and OV3660, and rename PID macros with `OTTO_` prefix to avoid symbol conflicts. - Configure camera output as `RGB565 + FRAMESIZE_240X240` to match the 240x240 display. - Rotate OV2640 output by 180 degrees (`VFlip + HMirror`) for correct orientation. - Simplify `otto-robot` camera sdkconfig options by keeping only: - `CONFIG_CAMERA_OV2640=y` - `CONFIG_CAMERA_OV3660=y` - Move Otto default dark theme setup into `OttoEmojiDisplay::SetupUI()` (after base UI init) to avoid boot-time LVGL null-object crash caused by calling `SetTheme()` too early. * feat(motion): smooth action ending and adaptive homing on otto/electron-bot Improve servo motion transitions to reduce abrupt returns to home pose. Replace linear interpolation with ease-out movement, make homing duration adaptive to angle delta, and skip intermediate homing when queued actions are pending to keep multi-action sequences fluid. * fix(electron-bot): show chat subtitle on display ElectronEmojiDisplay overrode SetupChatLabel() to delete the parent's chat_message_label_ (which lives inside bottom_bar_) and recreate it on the bottom-layer container_ without alignment. As a result the real subtitle label was hidden behind emoji_box_/top_bar_, while SetChatMessage() only toggled the now-empty bottom_bar_, so chat subtitles were never visible. Reuse the parent label created in bottom_bar_ (matching otto-robot) and only switch to the dark theme in SetupUI(), removing the custom SetupChatLabel(). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(electron-bot): restore status icons on standby and clean up display - SetStatus() did not re-show network_label_/battery_label_ when returning to STANDBY, so the WiFi and battery icons stayed hidden after listening or speaking. Restore them on standby to match otto-robot. - Remove the unused InitializeElectronEmojis() method and its declaration. - Drop unused includes (vector, assets.h, emoji_collection.h, lvgl_image.h) and add the missing trailing newline in the header. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
An MCP-based Chatbot
Introduction
👉 Handcraft your AI girlfriend, beginner's guide【bilibili】
As a voice interaction entry, the XiaoZhi AI chatbot leverages the AI capabilities of large models like Qwen / DeepSeek, and achieves multi-terminal control via the MCP protocol.
Version Notes
The current v2 version is incompatible with the v1 partition table, so it is not possible to upgrade from v1 to v2 via OTA. For partition table details, see partitions/v2/README.md.
All hardware running v1 can be upgraded to v2 by manually flashing the firmware.
The stable version of v1 is 1.9.2. You can switch to v1 by running git checkout v1. The v1 branch will be maintained until February 2026.
Features Implemented
- Wi-Fi / ML307 Cat.1 4G
- Offline voice wake-up ESP-SR
- Supports two communication protocols (Websocket or MQTT+UDP)
- Uses OPUS audio codec
- Voice interaction based on streaming ASR + LLM + TTS architecture
- Speaker recognition, identifies the current speaker 3D Speaker
- OLED / LCD display, supports emoji display
- Battery display and power management
- Multi-language support (Chinese, English, Japanese)
- Supports ESP32-C3, ESP32-S3, ESP32-P4 chip platforms
- Device-side MCP for device control (Speaker, LED, Servo, GPIO, etc.)
- Cloud-side MCP to extend large model capabilities (smart home control, PC desktop operation, knowledge search, email, etc.)
- Customizable wake words, fonts, emojis, and chat backgrounds with online web-based editing (Custom Assets Generator)
Hardware
Breadboard DIY Practice
See the Feishu document tutorial:
👉 "XiaoZhi AI Chatbot Encyclopedia"
Breadboard demo:
Supports 70+ Open Source Hardware (Partial List)
- LiChuang ESP32-S3 Development Board
- Espressif ESP32-S3-BOX3
- M5Stack CoreS3
- M5Stack AtomS3R + Echo Base
- Magic Button 2.4
- Waveshare ESP32-S3-Touch-AMOLED-1.8
- LILYGO T-Circle-S3
- XiaGe Mini C3
- CuiCan AI Pendant
- WMnologo-Xingzhi-1.54TFT
- SenseCAP Watcher
- ESP-HI Low Cost Robot Dog
Software
Firmware Flashing
For beginners, it is recommended to use the firmware that can be flashed without setting up a development environment.
The firmware connects to the official xiaozhi.me server by default. Personal users can register an account to use the Qwen real-time model for free.
👉 Beginner's Firmware Flashing Guide
Development Environment
- Cursor or VSCode
- Install ESP-IDF plugin, select SDK version 5.4 or above
- Linux is better than Windows for faster compilation and fewer driver issues
- This project uses Google C++ code style, please ensure compliance when submitting code
Developer Documentation
- Custom Board Guide - Learn how to create custom boards for XiaoZhi AI
- MCP Protocol IoT Control Usage - Learn how to control IoT devices via MCP protocol
- MCP Protocol Interaction Flow - Device-side MCP protocol implementation
- MQTT + UDP Hybrid Communication Protocol Document
- A detailed WebSocket communication protocol document
Large Model Configuration
If you already have a XiaoZhi AI chatbot device and have connected to the official server, you can log in to the xiaozhi.me console for configuration.
👉 Backend Operation Video Tutorial (Old Interface)
Related Open Source Projects
For server deployment on personal computers, refer to the following open-source projects:
- xinnan-tech/xiaozhi-esp32-server Python server
- joey-zhou/xiaozhi-esp32-server-java Java server
- AnimeAIChat/xiaozhi-server-go Golang server
- hackers365/xiaozhi-esp32-server-golang Golang server
Other client projects using the XiaoZhi communication protocol:
- huangjunsen0406/py-xiaozhi Python client
- TOM88812/xiaozhi-android-client Android client
- 100askTeam/xiaozhi-linux Linux client by 100ask
- 78/xiaozhi-sf32 Bluetooth chip firmware by Sichuan
- QuecPython/solution-xiaozhiAI QuecPython firmware by Quectel
Custom Assets Tools:
- 78/xiaozhi-assets-generator Custom Assets Generator (Wake words, fonts, emojis, backgrounds)
About the Project
This is an open-source ESP32 project, released under the MIT license, allowing anyone to use it for free, including for commercial purposes.
We hope this project helps everyone understand AI hardware development and apply rapidly evolving large language models to real hardware devices.
If you have any ideas or suggestions, please feel free to raise Issues or join our Discord or QQ group: 994694848