furyhawk/xiaozhi-esp32
1
0
Fork 0
mirror of https://github.com/78/xiaozhi-esp32.git synced 2026-07-20 09:45:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

Migrate to xiaozhi-fonts 2.0.0 (#2125)

Browse Source 
* feat: migrate firmware to xiaozhi-fonts 2.0.0

* fix: restart scrolling subtitles on text change

* docs: add AGENTS.md for project overview, architecture, rules, and commands

- Introduced AGENTS.md to document the XiaoZhi voice-assistant firmware architecture, required rules for development, board configuration process, and validation steps.
- Included detailed sections on project structure, commands for building and testing, and authoritative documentation references.

* feat: finalize glyph push protocol

* fix: remove stale emoji font include

* build: use xiaozhi-fonts 2.0.0

* fix: decouple custom fonts from glyph push

* refactor: improve wake word invocation handling

- Introduced BeginWakeWordInvoke method to streamline the wake word processing flow.
- Updated HandleWakeWordDetectedEvent to utilize the new method, ensuring proper state transitions and scheduling.
- Enhanced error handling to prevent the device from getting stuck in the connecting state.
- Refactored audio channel management to improve responsiveness during wake word detection.

---------

Co-authored-by: Xiaoxia <terrence.huang@tenclass.com>
This commit is contained in:
Xiaoxia
2026-07-18 21:29:00 +08:00
committed by GitHub
co-authored by Xiaoxia
parent 0f6c435f66
commit 3dea6e7f84
60 changed files with 2513 additions and 1364 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+94
View File
@@ -0,0 +1,94 @@
# AGENTS.md
## Project
XiaoZhi is an ESP-IDF C/C++ voice-assistant firmware supporting many chips, boards, displays, audio devices, and network transports. A build selects exactly one board implementation.
Use ESP-IDF v6.0.2 when possible. IDF 5.5.x is retained only for documented legacy boards.
## Architecture
- `main/application.*`: main event loop, protocol lifecycle, and high-level behavior.
- `main/device_state_machine.*`: legal runtime state transitions.
- `main/boards/common/`: board interfaces and reusable hardware/network helpers.
- `main/boards/**/`: board-specific pins, initialization, and build variants.
- `main/audio/`: codecs, audio tasks, engines, wake words, and queues.
- `main/protocols/`: transport-neutral API plus WebSocket and MQTT/UDP.
- `main/display/` and `main/led/`: reusable UI implementations.
- `main/mcp_server.*`: common device-side MCP tools and dispatch.
- `main/Kconfig.projbuild`: board and feature configuration.
- `main/CMakeLists.txt`: source, board, locale, font, and asset selection.
- `scripts/release.py`: canonical board/variant build entry point.
Read the closest existing implementation before adding a new one. Prefer the narrowest owning layer; do not put board-specific behavior into core modules.
## Required Rules
- Preserve unrelated worktree changes and keep patches focused.
- A build must export exactly one board factory through `DECLARE_BOARD(...)`.
- Never alter an existing board's pins to support different hardware. Add a uniquely named board or release variant; board identity affects OTA compatibility.
- Core code depends on `Board` interfaces, never a concrete board class or board `config.h`.
- Treat camera, backlight, display, LED, battery, and similar capabilities as optional.
- Change runtime state through `Application::SetDeviceState()` and the state machine.
- Callbacks may run outside the main task. Schedule application mutations with `Application::Schedule()` or event bits.
- Do not block the main event loop or audio tasks. Avoid unbounded queues and repeated large allocations in audio paths.
- Keep shared message semantics in `Protocol`; verify both transports when changing its contract.
- Validate network input and preserve `cJSON` ownership. NVS keys are persistent API and require migration when changed.
- Guard target-specific features with Kconfig/component rules. Do not assume every target has PSRAM or S3/P4 resources.
- Do not manually edit generated/vendor output: `build/`, `releases/`, `managed_components/`, `components/`, `sdkconfig*`, `main/assets/lang_config.h`, or generated mmap headers.
- Format only touched C/C++ files with the repository `.clang-format`; avoid unrelated mass formatting.
## Boards and Configuration
Board selection is a coupled chain:
`config.json` -> `scripts/release.py` -> `main/Kconfig.projbuild` -> `main/CMakeLists.txt` -> board source and `config.h`.
When adding a board or variant, update every relevant link in that chain. Include a unique board identity, correct chip target, flash/partition settings, exactly one `DECLARE_BOARD`, and board documentation. Follow `docs/custom-board.md`.
## Commands
Source the intended ESP-IDF environment first:
```sh
source /path/to/esp-idf/export.sh
idf.py --version
```
```sh
# Discover exact board and variant names
python3 scripts/release.py --list-boards
# Canonical variant build
python3 scripts/release.py <board-directory> --name <variant-name>
# Host-side release tests
python3 -m unittest discover -s scripts/tests -v
# Format/check touched files
clang-format -i <files>
clang-format --dry-run -Werror <files>
```
The release script changes local `sdkconfig` and build state. Do not assume the build directory still represents a previous target.
## Validation
- Board-only change: build affected variants and smoke-test changed hardware.
- Core, common-board, audio, protocol, display, dependency, Kconfig, or CMake change: run host tests and build representative affected chip/network paths.
- Protocol changes: verify WebSocket and MQTT/UDP when shared behavior changes.
- Audio changes: verify capture, playback, wake/VAD, interruption, reconnect, and applicable AEC modes.
- UI/assets changes: verify applicable no-display/OLED/LVGL paths and partition size.
- Always report what was tested and what still needs physical hardware. A successful build is not hardware validation.
## Authoritative Documentation
- Overview and SDK policy: `README.md`
- SDK compatibility: `docs/esp-idf-6-migration.md`
- Board guide: `docs/custom-board.md`
- Audio design: `main/audio/README.md`
- Code style: `docs/code_style.md`
- Protocols: `docs/websocket.md`, `docs/mqtt-udp.md`, `docs/mcp-protocol.md`
- CI matrix: `.github/workflows/build.yml`
Keep detailed or fast-changing information in those files, not here. Add a nested `AGENTS.md` only when a subsystem needs specialized instructions.
docs/glyph-push.md
+185
View File
@@ -0,0 +1,185 @@
# Dynamic Text Glyph Push Extension
This document defines version 1 of the `glyph_push` protocol extension. The extension lets a server
send bitmap glyphs that are missing from a device's installed text font. It applies equally to the
WebSocket and MQTT/UDP transports because capability advertisement and incoming JSON handling are
implemented in the shared protocol layer.
The extension supplements text rendering only. It does not change the text, TTS audio, or STT
semantics of the containing message.
## 1. Capability advertisement
The device advertises support in its client `hello` message:
```json
{
"type": "hello",
"version": 1,
"features": {
"mcp": true,
"glyph_push": true
},
"text_font": {
"bundle": "noto-v1",
"charset": "common",
"size": 20,
"bpp": 4
}
}
```
`features.glyph_push` indicates support for this extension. A server must treat a missing or false
value as unsupported. The `v` field in each pushed payload carries the extension version.
The `text_font` object describes the exact font data installed on the device:
| Field | Type | Meaning |
|---|---|---|
| `bundle` | string | Explicit font bundle identifier. It changes when glyph metrics, rendering behavior, character sets, or wire compatibility change. |
| `charset` | string | Installed character set. Version 1 devices report `basic` or `common`. |
| `size` | number | Text font pixel profile used by the firmware. |
| `bpp` | number | Bits per pixel of the text font bitmap, currently `1` or `4`. |
`basic` is the font linked into the firmware. The standard XiaoZhi assets report `common` after
loading their common font from the assets partition. The server must use the values from each
device's hello message rather than inferring them from the board model.
An OTA assets package may replace the text font with a different size, bpp, character set, or font
family. The firmware still loads any structurally valid CBIN font. When the package also provides
complete `text_font_meta` fields, the device advertises those active runtime values and validates
glyph pushes against them. A legacy or custom package without compatible glyph metadata continues
to use its custom font, emoji, colors, and background, but advertises `glyph_push: false` and omits
`text_font`. This prevents incompatible fallback glyphs without restricting theme customization.
## 2. Server glyph payload
The server may attach a `glyph_push` object to either of these server-to-device messages:
- a TTS message with `"type": "tts"` and `"state": "sentence_start"`;
- an STT message with `"type": "stt"`.
Example:
```json
{
"type": "tts",
"state": "sentence_start",
"text": "𠮷野家",
"glyph_push": {
"v": 1,
"bundle": "noto-v1",
"size": 20,
"bpp": 4,
"glyphs": [
{
"codepoint": 134071,
"adv_w": 320,
"box_w": 20,
"box_h": 20,
"ofs_x": 0,
"ofs_y": 0,
"bitmap": "<base64-encoded bitmap>"
}
]
}
}
```
The payload header must match the device capability exactly:
| Field | Requirement |
|---|---|
| `v` | Must be `1`. |
| `bundle` | Must equal `text_font.bundle`. |
| `size` | Must equal `text_font.size`. |
| `bpp` | Must equal `text_font.bpp`. |
| `glyphs` | The partial glyph batch being pushed, containing at most 64 entries. |
Each item uses the LVGL native bitmap-font metrics:
| Field | Meaning |
|---|---|
| `codepoint` | Unicode code point from `1` through `0x10FFFF`. |
| `adv_w` | Horizontal advance in LVGL fixed-point units with four fractional bits (one pixel is 16 units). |
| `box_w`, `box_h` | Bitmap dimensions in pixels. Each dimension must be from 0 through 64. |
| `ofs_x`, `ofs_y` | Signed 16-bit glyph offsets relative to the text baseline and cursor position. |
| `bitmap` | Base64 encoding of the uncompressed LVGL plain bitmap. |
The decoded bitmap length must be exactly:
```text
ceil(box_w * box_h * bpp / 8)
```
The bitmap must use the same plain, zero-stride layout as the matching Noto full-bundle CBIN font.
Servers should extract and forward the bitmap and metrics directly from that CBIN profile instead of
rasterizing an unrelated font at request time.
The sum of decoded bitmap lengths in one payload must not exceed 64 KiB. If any header, glyph, or
bitmap is invalid, the device rejects the entire glyph payload but still displays the message text
using its installed fonts. A PSRAM device may also use fallback glyphs cached by earlier messages.
## 3. Server selection algorithm
For every connection that advertises `glyph_push: true`, the server should:
1. Resolve the full font bundle identified by `text_font.bundle`.
2. Select the CBIN profile matching `text_font.size` and `text_font.bpp`.
3. Decode the message text into Unicode code points.
4. Remove control characters, duplicates, and code points already present in
`text_font.charset`.
5. Extract the remaining glyphs from the full bundle.
6. Apply the per-message limits and attach one `glyph_push` object to the text message.
7. Omit `glyph_push` when no missing glyph is available.
The installed text font is searched before the dynamic fallback font. Pushed glyphs therefore fill
missing code points; they do not override glyphs in `basic` or `common`.
The full font bundle may be shared by all device connections in a server process. Per-connection
work is limited to using the capability tuple `(bundle, charset, size, bpp)` to select which glyphs
are missing and which profile to read.
## 4. Device cache behavior
All glyphs in one message are inserted first, followed by a single fallback-font rebuild. The device
never rebuilds the font once per glyph.
On a device with initialized PSRAM:
- bitmap, cmap, descriptor, and cache-entry storage is allocated in PSRAM;
- glyphs are retained across messages;
- the cache holds at most 256 glyphs and 64 KiB of decoded bitmap data;
- the least recently inserted or updated entries are evicted when a limit is exceeded.
On a device without PSRAM:
- storage uses internal RAM;
- only the current message's glyph batch is retained;
- the next text message replaces or clears the previous batch.
This distinction does not affect the protocol. A server can send the glyphs needed by each message
without knowing whether the device has PSRAM.
## 5. Compatibility and versioning
The server must not send glyphs when any of these conditions is true:
- `features.glyph_push` is absent or not supported by the server;
- the server does not have the advertised bundle;
- no full-font profile matches the advertised size and bpp;
- the glyph data cannot satisfy the version 1 validation rules.
Fallback is automatic: messages without `glyph_push`, and messages whose glyph payload is rejected,
are still processed normally.
When the font generator changes metrics, bitmap layout, source fonts, character sets, or rendering
behavior, publish a new explicit bundle identifier. Do not serve glyphs from one bundle under
another bundle's identifier even if their size and bpp happen to match.
## 6. Security requirements
Glyph payloads are untrusted network input. Implementations must validate the complete payload before
mutating a live font, bound both item count and decoded size, verify base64 decoded length, and reject
invalid code points or metrics. Servers should also bound their own per-message work and avoid
sending glyphs already covered by the advertised charset.
docs/glyph-push_zh.md
+174
View File
@@ -0,0 +1,174 @@
# 动态文字 Glyph Push 扩展
本文档定义 `glyph_push` 协议扩展的版本 1。服务器可以通过该扩展向设备下发本地文字
字库中缺失的位图 glyph。能力声明和 JSON 消息处理位于公共协议层，因此 WebSocket 与
MQTT/UDP 使用完全相同的扩展格式。
该扩展只补充文字渲染能力，不改变消息中的文本、TTS 音频或 STT 语义。
## 1. 能力声明
设备在客户端 `hello` 消息中声明能力：
```json
{
"type": "hello",
"version": 1,
"features": {
"mcp": true,
"glyph_push": true
},
"text_font": {
"bundle": "noto-v1",
"charset": "common",
"size": 20,
"bpp": 4
}
}
```
`features.glyph_push` 表示设备支持该扩展。字段缺失或为 false 时，服务器必须视为设备不支持。
每次推送 payload 中的 `v` 字段负责表达扩展版本。
`text_font` 描述设备实际安装的文字字库：
| 字段 | 类型 | 含义 |
|---|---|---|
| `bundle` | string | 显式的字体 bundle 标识。glyph 度量、渲染方式、字符集或线格式变化时需要更换。 |
| `charset` | string | 已安装字符集。版本 1 设备报告 `basic` 或 `common`。 |
| `size` | number | 固件使用的文字字体像素规格。 |
| `bpp` | number | 字体位图的每像素位数，目前为 `1` 或 `4`。 |
`basic` 是链接进固件的字库。标准小智 assets 从分区加载 common 字库后会报告 `common`。
服务器必须使用每个连接在 hello 中报告的实际值，不能根据板型推测。
OTA assets 可以把文字字体替换为不同字号、bpp、字符集或字体家族的 CBIN 字体。固件仍会
加载任何结构有效的 CBIN 字体；如果 assets 同时提供完整的 `text_font_meta`，设备会声明当前
实际使用的运行时字体参数，并据此校验 glyph push。旧版或自定义 assets 缺少兼容的 glyph
metadata 时，自定义字体、表情、颜色和背景仍然正常使用，但设备会声明 `glyph_push: false`
并省略 `text_font`，从而只禁止不兼容的 fallback glyph，不限制主题定制能力。
## 2. 服务器下发格式
服务器可以在以下消息中附加 `glyph_push` 对象：
- `"type": "tts"`、`"state": "sentence_start"` 的 TTS 消息；
- `"type": "stt"` 的 STT 消息。
示例：
```json
{
"type": "tts",
"state": "sentence_start",
"text": "𠮷野家",
"glyph_push": {
"v": 1,
"bundle": "noto-v1",
"size": 20,
"bpp": 4,
"glyphs": [
{
"codepoint": 134071,
"adv_w": 320,
"box_w": 20,
"box_h": 20,
"ofs_x": 0,
"ofs_y": 0,
"bitmap": "<base64 编码的位图>"
}
]
}
}
```
payload 头必须与设备能力完全匹配：
| 字段 | 要求 |
|---|---|
| `v` | 必须为 `1`。 |
| `bundle` | 必须等于 `text_font.bundle`。 |
| `size` | 必须等于 `text_font.size`。 |
| `bpp` | 必须等于 `text_font.bpp`。 |
| `glyphs` | 本次增量推送的 glyph 数组，最多 64 项。 |
每个 item 使用 LVGL 原生位图字体度量：
| 字段 | 含义 |
|---|---|
| `codepoint` | `1` 到 `0x10FFFF` 的 Unicode code point。 |
| `adv_w` | 带 4 位小数的 LVGL 定点水平 advance，16 个单位等于 1 像素。 |
| `box_w`、`box_h` | 位图宽高，单项范围为 0 到 64 像素。 |
| `ofs_x`、`ofs_y` | 相对于文字基线和光标位置的有符号 16 位偏移。 |
| `bitmap` | 未压缩 LVGL plain 位图的 Base64 编码。 |
解码后的位图长度必须严格等于：
```text
ceil(box_w * box_h * bpp / 8)
```
位图必须与对应 Noto full bundle CBIN 字体使用相同的 plain、零 stride 布局。服务器应直接
提取相同 profile 的 CBIN 位图和度量，不应临时使用其他字体重新光栅化。
单条 payload 的解码位图总长度不得超过 64 KiB。任意头字段、glyph 或位图无效时，设备会
拒绝整批 glyph，但仍使用本地字体显示消息文本；有 PSRAM 的设备还可能使用此前缓存的
fallback glyph。
## 3. 服务器选择流程
对于声明 `glyph_push: true` 的每个连接，服务器应：
1. 根据 `text_font.bundle` 找到对应 full 字体 bundle。
2. 根据 `text_font.size` 和 `text_font.bpp` 选择 CBIN profile。
3. 将消息文本解码为 Unicode code point。
4. 去掉控制字符、重复字符及 `text_font.charset` 已包含的字符。
5. 从 full bundle 提取剩余 glyph。
6. 执行单消息限制，并将一个 `glyph_push` 对象附加到文字消息。
7. 没有可用的缺失 glyph 时省略 `glyph_push`。
设备先查询本地文字字体，再查询动态 fallback。因此 glyph push 只补充缺字，不会覆盖
`basic` 或 `common` 中已有的 glyph。
同一服务器进程中的所有设备连接可以共享 full 字体 bundle 服务。每个连接只需根据
`(bundle, charset, size, bpp)` 能力组合判断缺字并选择 profile。
## 4. 设备缓存行为
同一消息中的全部 glyph 会先加入缓存，随后只执行一次 fallback 字体 rebuild，不会每加入
一个 glyph 就 rebuild 一次。
设备有已初始化的 PSRAM 时：
- bitmap、cmap、descriptor 和缓存条目存放在 PSRAM；
- glyph 跨消息保留；
- 缓存上限为 256 个 glyph 和 64 KiB 解码位图；
- 超限时淘汰最早插入或更新的条目。
设备没有 PSRAM 时：
- 数据使用内部 RAM；
- 只保留当前消息的 glyph 批次；
- 下一条文字消息会替换或清空上一批 glyph。
该差异不改变协议。服务器无需知道设备是否有 PSRAM，可以为每条消息发送它所需的 glyph。
## 5. 兼容与版本管理
以下任一条件成立时，服务器不得发送 glyph：
- `features.glyph_push` 缺失、为 false，或服务器不支持该扩展；
- 服务器没有设备声明的 bundle；
- 找不到匹配 size 和 bpp 的 full 字体 profile；
- glyph 数据不满足版本 1 的校验要求。
兼容降级是自动的：没有 `glyph_push`，或 glyph payload 被拒绝时，设备仍会正常处理消息。
字体生成器的度量、位图布局、源字体、字符集或渲染方式变化时，应发布新的显式 bundle
标识。即使 size 和 bpp 相同，也不能用一个 bundle 的标识下发另一个 bundle 的 glyph。
## 6. 安全要求
glyph payload 是不可信网络输入。实现必须先完整校验 payload，再修改正在使用的字体；限制
item 数量和解码总长度；验证 Base64 解码长度；拒绝非法 code point 或度量。服务器也应限制
单消息处理量，避免重复发送设备声明字符集已经包含的 glyph。
docs/mqtt-udp.md
+10 -1
View File
@@ -80,7 +80,14 @@ The device connects to the broker using:
"transport": "udp",
"features": {
"mcp": true,
"aec": true
"aec": true,
"glyph_push": true
},
"text_font": {
"bundle": "noto-v1",
"charset": "common",
"size": 20,
"bpp": 4
},
"audio_params": {
"format": "opus",
@@ -92,6 +99,8 @@ The device connects to the broker using:
```
`features.mcp` is always set; `features.aec` is set when `CONFIG_USE_SERVER_AEC` is enabled.
`features.glyph_push` and `text_font` advertise the shared dynamic text-glyph extension described in
[Dynamic Text Glyph Push Extension](glyph-push.md).
#### 3.2.2 Server -> Device
docs/mqtt-udp_zh.md
+12 -2
View File
@@ -78,7 +78,14 @@ sequenceDiagram
"version": 3,
"transport": "udp",
"features": {
"mcp": true
"mcp": true,
"glyph_push": true
},
"text_font": {
"bundle": "noto-v1",
"charset": "common",
"size": 20,
"bpp": 4
},
"audio_params": {
"format": "opus",
@@ -89,6 +96,9 @@ sequenceDiagram
}
```
`features.glyph_push` 和 `text_font` 声明 WebSocket 与 MQTT/UDP 共用的动态文字扩展，详见
[动态文字 Glyph Push 扩展](glyph-push_zh.md)。
#### 3.2.2 服务器响应 Hello
```json
@@ -390,4 +400,4 @@ MQTT + UDP 混合协议通过以下设计实现高效的音视频通信：
- **自动恢复**：支持连接断开后的自动重连
- **性能优化**：UDP 传输保证音频数据的实时性
该协议适用于对实时性要求较高的语音交互场景，但需要在网络复杂度和传输性能之间做出权衡。
该协议适用于对实时性要求较高的语音交互场景，但需要在网络复杂度和传输性能之间做出权衡。
docs/websocket.md
+9 -1
View File
@@ -27,7 +27,14 @@ This document describes the WebSocket communication protocol between the device
"version": 1,
"features": {
"mcp": true,
"aec": true
"aec": true,
"glyph_push": true
},
"text_font": {
"bundle": "noto-v1",
"charset": "common",
"size": 20,
"bpp": 4
},
"transport": "websocket",
"audio_params": {
@@ -39,6 +46,7 @@ This document describes the WebSocket communication protocol between the device
}
```
- `features` is optional and generated from compile-time configuration. For example, `"mcp": true` means the device supports MCP, and `"aec": true` is emitted when `CONFIG_USE_SERVER_AEC` is enabled.
- `"glyph_push": true` and `text_font` advertise the optional dynamic text-glyph extension. See [Dynamic Text Glyph Push Extension](glyph-push.md).
- `frame_duration` matches `OPUS_FRAME_DURATION_MS` (typically 60 ms).
4. **Server replies with "hello"**
docs/websocket_zh.md
+9 -1
View File
@@ -26,7 +26,14 @@
"type": "hello",
"version": 1,
"features": {
"mcp": true
"mcp": true,
"glyph_push": true
},
"text_font": {
"bundle": "noto-v1",
"charset": "common",
"size": 20,
"bpp": 4
},
"transport": "websocket",
"audio_params": {
@@ -38,6 +45,7 @@
}
```
- 其中 `features` 字段为可选，内容根据设备编译配置自动生成。例如：`"mcp": true` 表示支持 MCP 协议。
- `"glyph_push": true` 和 `text_font` 声明可选的动态文字 glyph 扩展，详见[动态文字 Glyph Push 扩展](glyph-push_zh.md)。
- `frame_duration` 的值对应 `OPUS_FRAME_DURATION_MS`（例如 60ms）。
4. **服务器回复 "hello"**
main/CMakeLists.txt
+437 -412
View File
File diff suppressed because it is too large Load Diff
main/application.cc
+201 -183
View File
@@ -1,25 +1,24 @@
#include "application.h"
#include "assets.h"
#include "assets/lang_config.h"
#include "audio_codec.h"
#include "board.h"
#include "display.h"
#include "system_info.h"
#include "audio_codec.h"
#include "mqtt_protocol.h"
#include "websocket_protocol.h"
#include "assets/lang_config.h"
#include "mcp_server.h"
#include "assets.h"
#include "mqtt_protocol.h"
#include "settings.h"
#include "system_info.h"
#include "text_glyph_payload.h"
#include "websocket_protocol.h"
#include <cstring>
#include <esp_log.h>
#include <cJSON.h>
#include <driver/gpio.h>
#include <esp_log.h>
#include <arpa/inet.h>
#include <font_awesome.h>
#include <cJSON.h>
#include <cstring>
#define TAG "Application"
Application::Application() {
event_group_ = xEventGroupCreate();
@@ -33,16 +32,16 @@ Application::Application() {
aec_mode_ = kAecOff;
#endif
esp_timer_create_args_t clock_timer_args = {
.callback = [](void* arg) {
Application* app = (Application*)arg;
xEventGroupSetBits(app->event_group_, MAIN_EVENT_CLOCK_TICK);
},
.arg = this,
.dispatch_method = ESP_TIMER_TASK,
.name = "clock_timer",
.skip_unhandled_events = true
};
esp_timer_create_args_t clock_timer_args = {.callback =
[](void* arg) {
Application* app = (Application*)arg;
xEventGroupSetBits(app->event_group_,
MAIN_EVENT_CLOCK_TICK);
},
.arg = this,
.dispatch_method = ESP_TIMER_TASK,
.name = "clock_timer",
.skip_unhandled_events = true};
esp_timer_create(&clock_timer_args, &clock_timer_handle_);
}
@@ -54,9 +53,7 @@ Application::~Application() {
vEventGroupDelete(event_group_);
}
bool Application::SetDeviceState(DeviceState state) {
return state_machine_.TransitionTo(state);
}
bool Application::SetDeviceState(DeviceState state) { return state_machine_.TransitionTo(state); }
void Application::Initialize() {
auto& board = Board::GetInstance();
@@ -104,7 +101,7 @@ void Application::Initialize() {
// Set network event callback for UI updates and network state handling
board.SetNetworkEventCallback([this](NetworkEvent event, const std::string& data) {
auto display = Board::GetInstance().GetDisplay();
switch (event) {
case NetworkEvent::Scanning:
display->ShowNotification(Lang::Strings::SCANNING_WIFI, 30000);
@@ -144,13 +141,16 @@ void Application::Initialize() {
display->SetStatus(Lang::Strings::DETECTING_MODULE);
break;
case NetworkEvent::ModemErrorNoSim:
Alert(Lang::Strings::ERROR, Lang::Strings::PIN_ERROR, "triangle_exclamation", Lang::Sounds::OGG_ERR_PIN);
Alert(Lang::Strings::ERROR, Lang::Strings::PIN_ERROR, "warning",
Lang::Sounds::OGG_ERR_PIN);
break;
case NetworkEvent::ModemErrorRegDenied:
Alert(Lang::Strings::ERROR, Lang::Strings::REG_ERROR, "triangle_exclamation", Lang::Sounds::OGG_ERR_REG);
Alert(Lang::Strings::ERROR, Lang::Strings::REG_ERROR, "warning",
Lang::Sounds::OGG_ERR_REG);
break;
case NetworkEvent::ModemErrorInitFailed:
Alert(Lang::Strings::ERROR, Lang::Strings::MODEM_INIT_ERROR, "triangle_exclamation", Lang::Sounds::OGG_EXCLAMATION);
Alert(Lang::Strings::ERROR, Lang::Strings::MODEM_INIT_ERROR, "warning",
Lang::Sounds::OGG_EXCLAMATION);
break;
case NetworkEvent::ModemErrorTimeout:
display->SetStatus(Lang::Strings::REGISTERING_NETWORK);
@@ -169,28 +169,20 @@ void Application::Run() {
// Set the priority of the main task to 10
vTaskPrioritySet(nullptr, 10);
const EventBits_t ALL_EVENTS =
MAIN_EVENT_SCHEDULE |
MAIN_EVENT_SEND_AUDIO |
MAIN_EVENT_WAKE_WORD_DETECTED |
MAIN_EVENT_VAD_CHANGE |
MAIN_EVENT_CLOCK_TICK |
MAIN_EVENT_ERROR |
MAIN_EVENT_NETWORK_CONNECTED |
MAIN_EVENT_NETWORK_DISCONNECTED |
MAIN_EVENT_TOGGLE_CHAT |
MAIN_EVENT_START_LISTENING |
MAIN_EVENT_STOP_LISTENING |
MAIN_EVENT_ACTIVATION_DONE |
MAIN_EVENT_STATE_CHANGED |
MAIN_EVENT_PLAYBACK_DRAINED;
const EventBits_t ALL_EVENTS =
MAIN_EVENT_SCHEDULE | MAIN_EVENT_SEND_AUDIO | MAIN_EVENT_WAKE_WORD_DETECTED |
MAIN_EVENT_VAD_CHANGE | MAIN_EVENT_CLOCK_TICK | MAIN_EVENT_ERROR |
MAIN_EVENT_NETWORK_CONNECTED | MAIN_EVENT_NETWORK_DISCONNECTED | MAIN_EVENT_TOGGLE_CHAT |
MAIN_EVENT_START_LISTENING | MAIN_EVENT_STOP_LISTENING | MAIN_EVENT_ACTIVATION_DONE |
MAIN_EVENT_STATE_CHANGED | MAIN_EVENT_PLAYBACK_DRAINED;
while (true) {
auto bits = xEventGroupWaitBits(event_group_, ALL_EVENTS, pdTRUE, pdFALSE, portMAX_DELAY);
if (bits & MAIN_EVENT_ERROR) {
SetDeviceState(kDeviceStateIdle);
Alert(Lang::Strings::ERROR, last_error_message_.c_str(), "circle_xmark", Lang::Sounds::OGG_EXCLAMATION);
Alert(Lang::Strings::ERROR, last_error_message_.c_str(), "cancel",
Lang::Sounds::OGG_EXCLAMATION);
}
if (bits & MAIN_EVENT_NETWORK_CONNECTED) {
@@ -238,7 +230,8 @@ void Application::Run() {
// stall the Opus codec task (it waits for queue space), which in
// turn deadlocks the whole audio input pipeline, as no new
// MAIN_EVENT_SEND_AUDIO event would ever be triggered again.
while (audio_service_.PopPacketFromSendQueue());
while (audio_service_.PopPacketFromSendQueue())
;
break;
}
}
@@ -268,7 +261,7 @@ void Application::Run() {
clock_ticks_++;
auto display = Board::GetInstance().GetDisplay();
display->UpdateStatusBar();
// Print debug info every 10 seconds
if (clock_ticks_ % 10 == 0) {
SystemInfo::PrintHeapStats();
@@ -291,12 +284,14 @@ void Application::HandleNetworkConnectedEvent() {
return;
}
xTaskCreate([](void* arg) {
Application* app = static_cast<Application*>(arg);
app->ActivationTask();
app->activation_task_handle_ = nullptr;
vTaskDelete(NULL);
}, "activation", 4096 * 2, this, 2, &activation_task_handle_);
xTaskCreate(
[](void* arg) {
Application* app = static_cast<Application*>(arg);
app->ActivationTask();
app->activation_task_handle_ = nullptr;
vTaskDelete(NULL);
},
"activation", 4096 * 2, this, 2, &activation_task_handle_);
}
// Update the status bar immediately to show the network state
@@ -307,7 +302,8 @@ void Application::HandleNetworkConnectedEvent() {
void Application::HandleNetworkDisconnectedEvent() {
// Close current conversation when network disconnected
auto state = GetDeviceState();
if (state == kDeviceStateConnecting || state == kDeviceStateListening || state == kDeviceStateSpeaking) {
if (state == kDeviceStateConnecting || state == kDeviceStateListening ||
state == kDeviceStateSpeaking) {
ESP_LOGI(TAG, "Closing audio channel due to network disconnection");
protocol_->CloseAudioChannel();
}
@@ -373,7 +369,7 @@ void Application::CheckAssetsVersion() {
ESP_LOGW(TAG, "Assets partition is disabled for board %s", BOARD_NAME);
return;
}
Settings settings("assets", true);
// Check if there is a new assets need to be downloaded
std::string download_url = settings.GetString("download_url");
@@ -383,27 +379,29 @@ void Application::CheckAssetsVersion() {
char message[256];
snprintf(message, sizeof(message), Lang::Strings::FOUND_NEW_ASSETS, download_url.c_str());
Alert(Lang::Strings::LOADING_ASSETS, message, "cloud_arrow_down", Lang::Sounds::OGG_UPGRADE);
Alert(Lang::Strings::LOADING_ASSETS, message, "cloud_download", Lang::Sounds::OGG_UPGRADE);
// Wait for the audio service to be idle for 3 seconds
vTaskDelay(pdMS_TO_TICKS(3000));
SetDeviceState(kDeviceStateUpgrading);
board.SetPowerSaveLevel(PowerSaveLevel::PERFORMANCE);
display->SetChatMessage("system", Lang::Strings::PLEASE_WAIT);
bool success = assets.Download(download_url, [this, display](int progress, size_t speed) -> void {
char buffer[32];
snprintf(buffer, sizeof(buffer), "%d%% %uKB/s", progress, speed / 1024);
Schedule([display, message = std::string(buffer)]() {
display->SetChatMessage("system", message.c_str());
bool success =
assets.Download(download_url, [this, display](int progress, size_t speed) -> void {
char buffer[32];
snprintf(buffer, sizeof(buffer), "%d%% %uKB/s", progress, speed / 1024);
Schedule([display, message = std::string(buffer)]() {
display->SetChatMessage("system", message.c_str());
});
});
});
board.SetPowerSaveLevel(PowerSaveLevel::LOW_POWER);
vTaskDelay(pdMS_TO_TICKS(1000));
if (!success) {
Alert(Lang::Strings::ERROR, Lang::Strings::DOWNLOAD_ASSETS_FAILED, "circle_xmark", Lang::Sounds::OGG_EXCLAMATION);
Alert(Lang::Strings::ERROR, Lang::Strings::DOWNLOAD_ASSETS_FAILED, "cancel",
Lang::Sounds::OGG_EXCLAMATION);
vTaskDelay(pdMS_TO_TICKS(2000));
SetDeviceState(kDeviceStateActivating);
return;
@@ -413,13 +411,13 @@ void Application::CheckAssetsVersion() {
// Apply assets
assets.Apply();
display->SetChatMessage("system", "");
display->SetEmotion("microchip_ai");
display->SetEmotion("robot_2");
}
void Application::CheckNewVersion() {
const int MAX_RETRY = 10;
int retry_count = 0;
int retry_delay = 10; // Initial retry delay in seconds
int retry_delay = 10; // Initial retry delay in seconds
auto& board = Board::GetInstance();
while (true) {
@@ -435,27 +433,30 @@ void Application::CheckNewVersion() {
}
char error_message[128];
snprintf(error_message, sizeof(error_message), "code=%d, url=%s", err, ota_->GetCheckVersionUrl().c_str());
snprintf(error_message, sizeof(error_message), "code=%d, url=%s", err,
ota_->GetCheckVersionUrl().c_str());
char buffer[256];
snprintf(buffer, sizeof(buffer), Lang::Strings::CHECK_NEW_VERSION_FAILED, retry_delay, error_message);
Alert(Lang::Strings::ERROR, buffer, "cloud_slash", Lang::Sounds::OGG_EXCLAMATION);
snprintf(buffer, sizeof(buffer), Lang::Strings::CHECK_NEW_VERSION_FAILED, retry_delay,
error_message);
Alert(Lang::Strings::ERROR, buffer, "cloud_off", Lang::Sounds::OGG_EXCLAMATION);
ESP_LOGW(TAG, "Check new version failed, retry in %d seconds (%d/%d)", retry_delay, retry_count, MAX_RETRY);
ESP_LOGW(TAG, "Check new version failed, retry in %d seconds (%d/%d)", retry_delay,
retry_count, MAX_RETRY);
for (int i = 0; i < retry_delay; i++) {
vTaskDelay(pdMS_TO_TICKS(1000));
if (GetDeviceState() == kDeviceStateIdle) {
break;
}
}
retry_delay *= 2; // Double the retry delay
retry_delay *= 2; // Double the retry delay
continue;
}
retry_count = 0;
retry_delay = 10; // Reset retry delay
retry_delay = 10; // Reset retry delay
if (ota_->HasNewVersion()) {
if (UpgradeFirmware(ota_->GetFirmwareUrl(), ota_->GetFirmwareVersion())) {
return; // This line will never be reached after reboot
return; // This line will never be reached after reboot
}
// If upgrade failed, continue to normal operation
}
@@ -507,29 +508,29 @@ void Application::InitializeProtocol() {
protocol_ = std::make_unique<MqttProtocol>();
}
protocol_->OnConnected([this]() {
DismissAlert();
});
protocol_->OnConnected([this]() { DismissAlert(); });
protocol_->OnNetworkError([this](const std::string& message) {
last_error_message_ = message;
xEventGroupSetBits(event_group_, MAIN_EVENT_ERROR);
});
protocol_->OnIncomingAudio([this](std::unique_ptr<AudioStreamPacket> packet) {
if (GetDeviceState() == kDeviceStateSpeaking) {
audio_service_.PushPacketToDecodeQueue(std::move(packet));
}
});
protocol_->OnAudioChannelOpened([this, codec, &board]() {
board.SetPowerSaveLevel(PowerSaveLevel::PERFORMANCE);
if (protocol_->server_sample_rate() != codec->output_sample_rate()) {
ESP_LOGW(TAG, "Server sample rate %d does not match device output sample rate %d, resampling may cause distortion",
protocol_->server_sample_rate(), codec->output_sample_rate());
ESP_LOGW(TAG,
"Server sample rate %d does not match device output sample rate %d, "
"resampling may cause distortion",
protocol_->server_sample_rate(), codec->output_sample_rate());
}
});
protocol_->OnAudioChannelClosed([this, &board]() {
board.SetPowerSaveLevel(PowerSaveLevel::LOW_POWER);
Schedule([this]() {
@@ -538,12 +539,19 @@ void Application::InitializeProtocol() {
SetDeviceState(kDeviceStateIdle);
});
});
protocol_->OnIncomingJson([this, display](const cJSON* root) {
// Parse JSON data
auto type = cJSON_GetObjectItem(root, "type");
if (!cJSON_IsString(type)) {
ESP_LOGW(TAG, "Incoming JSON message has no type");
return;
}
if (strcmp(type->valuestring, "tts") == 0) {
auto state = cJSON_GetObjectItem(root, "state");
if (!cJSON_IsString(state)) {
return;
}
if (strcmp(state->valuestring, "start") == 0) {
Schedule([this]() {
aborted_ = false;
@@ -562,8 +570,15 @@ void Application::InitializeProtocol() {
} else if (strcmp(state->valuestring, "sentence_start") == 0) {
auto text = cJSON_GetObjectItem(root, "text");
if (cJSON_IsString(text)) {
std::vector<TextGlyph> glyphs;
uint8_t bpp = 0;
if (!TextGlyphPayload::Parse(root, glyphs, bpp)) {
glyphs.clear();
}
ESP_LOGI(TAG, "<< %s", text->valuestring);
Schedule([display, message = std::string(text->valuestring)]() {
Schedule([display, message = std::string(text->valuestring),
glyphs = std::move(glyphs), bpp]() {
display->AddTextGlyphs(glyphs, bpp);
display->SetChatMessage("assistant", message.c_str());
});
}
@@ -571,8 +586,15 @@ void Application::InitializeProtocol() {
} else if (strcmp(type->valuestring, "stt") == 0) {
auto text = cJSON_GetObjectItem(root, "text");
if (cJSON_IsString(text)) {
std::vector<TextGlyph> glyphs;
uint8_t bpp = 0;
if (!TextGlyphPayload::Parse(root, glyphs, bpp)) {
glyphs.clear();
}
ESP_LOGI(TAG, ">> %s", text->valuestring);
Schedule([display, message = std::string(text->valuestring)]() {
Schedule([display, message = std::string(text->valuestring),
glyphs = std::move(glyphs), bpp]() {
display->AddTextGlyphs(glyphs, bpp);
display->SetChatMessage("user", message.c_str());
});
}
@@ -594,9 +616,7 @@ void Application::InitializeProtocol() {
ESP_LOGI(TAG, "System command: %s", command->valuestring);
if (strcmp(command->valuestring, "reboot") == 0) {
// Do a reboot if user requests a OTA update
Schedule([this]() {
Reboot();
});
Schedule([this]() { Reboot(); });
} else {
ESP_LOGW(TAG, "Unknown system command: %s", command->valuestring);
}
@@ -606,7 +626,8 @@ void Application::InitializeProtocol() {
auto message = cJSON_GetObjectItem(root, "message");
auto emotion = cJSON_GetObjectItem(root, "emotion");
if (cJSON_IsString(status) && cJSON_IsString(message) && cJSON_IsString(emotion)) {
Alert(status->valuestring, message->valuestring, emotion->valuestring, Lang::Sounds::OGG_VIBRATION);
Alert(status->valuestring, message->valuestring, emotion->valuestring,
Lang::Sounds::OGG_VIBRATION);
} else {
ESP_LOGW(TAG, "Alert command requires status, message and emotion");
}
@@ -615,9 +636,10 @@ void Application::InitializeProtocol() {
auto payload = cJSON_GetObjectItem(root, "payload");
ESP_LOGI(TAG, "Received custom message: %s", cJSON_PrintUnformatted(root));
if (cJSON_IsObject(payload)) {
Schedule([this, display, payload_str = std::string(cJSON_PrintUnformatted(payload))]() {
display->SetChatMessage("system", payload_str.c_str());
});
Schedule(
[this, display, payload_str = std::string(cJSON_PrintUnformatted(payload))]() {
display->SetChatMessage("system", payload_str.c_str());
});
} else {
ESP_LOGW(TAG, "Invalid custom message format: missing payload");
}
@@ -626,7 +648,7 @@ void Application::InitializeProtocol() {
ESP_LOGW(TAG, "Unknown message type: %s", type->valuestring);
}
});
protocol_->Start();
}
@@ -635,32 +657,27 @@ void Application::ShowActivationCode(const std::string& code, const std::string&
char digit;
const std::string_view& sound;
};
static const std::array<digit_sound, 10> digit_sounds{{
digit_sound{'0', Lang::Sounds::OGG_0},
digit_sound{'1', Lang::Sounds::OGG_1},
digit_sound{'2', Lang::Sounds::OGG_2},
digit_sound{'3', Lang::Sounds::OGG_3},
digit_sound{'4', Lang::Sounds::OGG_4},
digit_sound{'5', Lang::Sounds::OGG_5},
digit_sound{'6', Lang::Sounds::OGG_6},
digit_sound{'7', Lang::Sounds::OGG_7},
digit_sound{'8', Lang::Sounds::OGG_8},
digit_sound{'9', Lang::Sounds::OGG_9}
}};
static const std::array<digit_sound, 10> digit_sounds{
{digit_sound{'0', Lang::Sounds::OGG_0}, digit_sound{'1', Lang::Sounds::OGG_1},
digit_sound{'2', Lang::Sounds::OGG_2}, digit_sound{'3', Lang::Sounds::OGG_3},
digit_sound{'4', Lang::Sounds::OGG_4}, digit_sound{'5', Lang::Sounds::OGG_5},
digit_sound{'6', Lang::Sounds::OGG_6}, digit_sound{'7', Lang::Sounds::OGG_7},
digit_sound{'8', Lang::Sounds::OGG_8}, digit_sound{'9', Lang::Sounds::OGG_9}}};
// This sentence uses 9KB of SRAM, so we need to wait for it to finish
Alert(Lang::Strings::ACTIVATION, message.c_str(), "link", Lang::Sounds::OGG_ACTIVATION);
for (const auto& digit : code) {
auto it = std::find_if(digit_sounds.begin(), digit_sounds.end(),
[digit](const digit_sound& ds) { return ds.digit == digit; });
[digit](const digit_sound& ds) { return ds.digit == digit; });
if (it != digit_sounds.end()) {
audio_service_.PlaySound(it->sound);
}
}
}
void Application::Alert(const char* status, const char* message, const char* emotion, const std::string_view& sound) {
void Application::Alert(const char* status, const char* message, const char* emotion,
const std::string_view& sound) {
ESP_LOGW(TAG, "Alert [%s] %s: %s", emotion, status, message);
auto display = Board::GetInstance().GetDisplay();
display->SetStatus(status);
@@ -680,21 +697,15 @@ void Application::DismissAlert() {
}
}
void Application::ToggleChatState() {
xEventGroupSetBits(event_group_, MAIN_EVENT_TOGGLE_CHAT);
}
void Application::ToggleChatState() { xEventGroupSetBits(event_group_, MAIN_EVENT_TOGGLE_CHAT); }
void Application::StartListening() {
xEventGroupSetBits(event_group_, MAIN_EVENT_START_LISTENING);
}
void Application::StartListening() { xEventGroupSetBits(event_group_, MAIN_EVENT_START_LISTENING); }
void Application::StopListening() {
xEventGroupSetBits(event_group_, MAIN_EVENT_STOP_LISTENING);
}
void Application::StopListening() { xEventGroupSetBits(event_group_, MAIN_EVENT_STOP_LISTENING); }
void Application::HandleToggleChatEvent() {
auto state = GetDeviceState();
if (state == kDeviceStateActivating) {
SetDeviceState(kDeviceStateIdle);
return;
@@ -718,9 +729,7 @@ void Application::HandleToggleChatEvent() {
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
// Schedule to let the state change be processed first (UI update)
Schedule([this, mode]() {
ContinueOpenAudioChannel(mode);
});
Schedule([this, mode]() { ContinueOpenAudioChannel(mode); });
return;
}
SetListeningMode(mode);
@@ -743,6 +752,9 @@ void Application::ContinueOpenAudioChannel(ListeningMode mode) {
if (!protocol_->IsAudioChannelOpened()) {
if (!protocol_->OpenAudioChannel()) {
// Return to idle so the device is not stuck in the connecting
// state (not every failure path reports a network error)
SetDeviceState(kDeviceStateIdle);
return;
}
}
@@ -752,7 +764,7 @@ void Application::ContinueOpenAudioChannel(ListeningMode mode) {
void Application::HandleStartListeningEvent() {
auto state = GetDeviceState();
if (state == kDeviceStateActivating) {
SetDeviceState(kDeviceStateIdle);
return;
@@ -766,14 +778,12 @@ void Application::HandleStartListeningEvent() {
ESP_LOGE(TAG, "Protocol not initialized");
return;
}
if (state == kDeviceStateIdle) {
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
// Schedule to let the state change be processed first (UI update)
Schedule([this]() {
ContinueOpenAudioChannel(kListeningModeManualStop);
});
Schedule([this]() { ContinueOpenAudioChannel(kListeningModeManualStop); });
return;
}
SetListeningMode(kListeningModeManualStop);
@@ -785,7 +795,7 @@ void Application::HandleStartListeningEvent() {
void Application::HandleStopListeningEvent() {
auto state = GetDeviceState();
if (state == kDeviceStateAudioTesting) {
audio_service_.EnableAudioTesting(false);
SetDeviceState(kDeviceStateWifiConfiguring);
@@ -808,24 +818,12 @@ void Application::HandleWakeWordDetectedEvent() {
ESP_LOGI(TAG, "Wake word detected: %s (state: %d)", wake_word.c_str(), (int)state);
if (state == kDeviceStateIdle) {
audio_service_.EncodeWakeWord();
auto wake_word = audio_service_.GetLastWakeWord();
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
// Schedule to let the state change be processed first (UI update),
// then continue with OpenAudioChannel which may block for ~1 second
Schedule([this, wake_word]() {
ContinueWakeWordInvoke(wake_word);
});
return;
}
// Channel already opened, continue directly
ContinueWakeWordInvoke(wake_word);
BeginWakeWordInvoke(wake_word);
} else if (state == kDeviceStateSpeaking || state == kDeviceStateListening) {
AbortSpeaking(kAbortReasonWakeWordDetected);
// Clear send queue to avoid sending residues to server
while (audio_service_.PopPacketFromSendQueue());
while (audio_service_.PopPacketFromSendQueue())
;
if (state == kDeviceStateListening) {
protocol_->SendStartListening(GetDefaultListeningMode());
@@ -844,6 +842,30 @@ void Application::HandleWakeWordDetectedEvent() {
}
}
void Application::BeginWakeWordInvoke(const std::string& wake_word) {
// Must run in the main task with the device in idle state
audio_service_.EncodeWakeWord();
// Always pass through the connecting state, even if the audio channel is
// already opened. ContinueWakeWordInvoke() rejects any other state, so
// skipping this transition would silently drop the wake word invocation.
if (!SetDeviceState(kDeviceStateConnecting)) {
// Wake word detection was stopped by the detection itself; restore it
// so the device does not become unresponsive to wake words.
audio_service_.EnableWakeWordDetection(true);
return;
}
if (!protocol_->IsAudioChannelOpened()) {
// Schedule to let the state change be processed first (UI update),
// then continue with OpenAudioChannel which may block for ~1 second
Schedule([this, wake_word]() { ContinueWakeWordInvoke(wake_word); });
return;
}
// Channel already opened, continue directly
ContinueWakeWordInvoke(wake_word);
}
void Application::ContinueWakeWordInvoke(const std::string& wake_word) {
// Check state again in case it was changed during scheduling
if (GetDeviceState() != kDeviceStateConnecting) {
@@ -856,7 +878,10 @@ void Application::ContinueWakeWordInvoke(const std::string& wake_word) {
if (!protocol_->IsAudioChannelOpened()) {
if (!protocol_->OpenAudioChannel()) {
audio_service_.EnableWakeWordDetection(true);
// Return to idle so the device is not stuck in the connecting
// state (not every failure path reports a network error), and
// wake word detection is re-enabled by the idle state handler.
SetDeviceState(kDeviceStateIdle);
return;
}
}
@@ -889,13 +914,13 @@ void Application::HandleStateChangedEvent() {
auto display = board.GetDisplay();
auto led = board.GetLed();
led->OnStateChanged();
switch (new_state) {
case kDeviceStateUnknown:
case kDeviceStateIdle:
display->SetStatus(Lang::Strings::STANDBY);
display->ClearChatMessages(); // Clear messages first
display->SetEmotion("neutral"); // Then set emotion (wechat mode checks child count)
display->ClearChatMessages(); // Clear messages first
display->SetEmotion("neutral"); // Then set emotion (wechat mode checks child count)
audio_service_.EnableVoiceProcessing(false);
audio_service_.EnableWakeWordDetection(true);
break;
@@ -1025,7 +1050,8 @@ bool Application::UpgradeFirmware(const std::string& url, const std::string& ver
}
ESP_LOGI(TAG, "Starting firmware upgrade from URL: %s", upgrade_url.c_str());
Alert(Lang::Strings::OTA_UPGRADE, Lang::Strings::UPGRADING, "download", Lang::Sounds::OGG_UPGRADE);
Alert(Lang::Strings::OTA_UPGRADE, Lang::Strings::UPGRADING, "download",
Lang::Sounds::OGG_UPGRADE);
vTaskDelay(pdMS_TO_TICKS(3000));
SetDeviceState(kDeviceStateUpgrading);
@@ -1047,17 +1073,19 @@ bool Application::UpgradeFirmware(const std::string& url, const std::string& ver
if (!upgrade_success) {
// Upgrade failed, restart audio service and continue running
ESP_LOGE(TAG, "Firmware upgrade failed, restarting audio service and continuing operation...");
audio_service_.Start(); // Restart audio service
board.SetPowerSaveLevel(PowerSaveLevel::LOW_POWER); // Restore power save level
Alert(Lang::Strings::ERROR, Lang::Strings::UPGRADE_FAILED, "circle_xmark", Lang::Sounds::OGG_EXCLAMATION);
ESP_LOGE(TAG,
"Firmware upgrade failed, restarting audio service and continuing operation...");
audio_service_.Start(); // Restart audio service
board.SetPowerSaveLevel(PowerSaveLevel::LOW_POWER); // Restore power save level
Alert(Lang::Strings::ERROR, Lang::Strings::UPGRADE_FAILED, "cancel",
Lang::Sounds::OGG_EXCLAMATION);
vTaskDelay(pdMS_TO_TICKS(3000));
return false;
} else {
// Upgrade success, reboot immediately
ESP_LOGI(TAG, "Firmware upgrade successful, rebooting...");
display->SetChatMessage("system", "Upgrade successful, rebooting...");
vTaskDelay(pdMS_TO_TICKS(1000)); // Brief pause to show message
vTaskDelay(pdMS_TO_TICKS(1000)); // Brief pause to show message
Reboot();
return true;
}
@@ -1069,25 +1097,18 @@ void Application::WakeWordInvoke(const std::string& wake_word) {
}
auto state = GetDeviceState();
if (state == kDeviceStateIdle) {
audio_service_.EncodeWakeWord();
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
// Schedule to let the state change be processed first (UI update)
Schedule([this, wake_word]() {
ContinueWakeWordInvoke(wake_word);
});
return;
}
// Channel already opened, continue directly
ContinueWakeWordInvoke(wake_word);
} else if (state == kDeviceStateSpeaking) {
Schedule([this]() {
AbortSpeaking(kAbortReasonNone);
if (state == kDeviceStateIdle) {
// May be called from outside the main task (e.g. board button
// callbacks), so schedule the invocation instead of running it here
Schedule([this, wake_word]() {
if (GetDeviceState() == kDeviceStateIdle) {
BeginWakeWordInvoke(wake_word);
}
});
} else if (state == kDeviceStateListening) {
} else if (state == kDeviceStateSpeaking) {
Schedule([this]() { AbortSpeaking(kAbortReasonNone); });
} else if (state == kDeviceStateListening) {
Schedule([this]() {
if (protocol_) {
protocol_->CloseAudioChannel();
@@ -1119,7 +1140,7 @@ void Application::RegisterMcpBroadcastCallback(std::function<void(const std::str
void Application::SendMcpMessage(const std::string& payload) {
// Always schedule to run in main task for thread safety
Schedule([this, payload](){
Schedule([this, payload]() {
if (protocol_) {
protocol_->SendMcpMessage(payload);
}
@@ -1135,18 +1156,18 @@ void Application::SetAecMode(AecMode mode) {
auto& board = Board::GetInstance();
auto display = board.GetDisplay();
switch (aec_mode_) {
case kAecOff:
audio_service_.EnableDeviceAec(false);
display->ShowNotification(Lang::Strings::RTC_MODE_OFF);
break;
case kAecOnServerSide:
audio_service_.EnableDeviceAec(false);
display->ShowNotification(Lang::Strings::RTC_MODE_ON);
break;
case kAecOnDeviceSide:
audio_service_.EnableDeviceAec(true);
display->ShowNotification(Lang::Strings::RTC_MODE_ON);
break;
case kAecOff:
audio_service_.EnableDeviceAec(false);
display->ShowNotification(Lang::Strings::RTC_MODE_OFF);
break;
case kAecOnServerSide:
audio_service_.EnableDeviceAec(false);
display->ShowNotification(Lang::Strings::RTC_MODE_ON);
break;
case kAecOnDeviceSide:
audio_service_.EnableDeviceAec(true);
display->ShowNotification(Lang::Strings::RTC_MODE_ON);
break;
}
// If the AEC mode is changed, close the audio channel
@@ -1156,9 +1177,7 @@ void Application::SetAecMode(AecMode mode) {
});
}
void Application::PlaySound(const std::string_view& sound) {
audio_service_.PlaySound(sound);
}
void Application::PlaySound(const std::string_view& sound) { audio_service_.PlaySound(sound); }
void Application::ResetProtocol() {
Schedule([this]() {
@@ -1170,4 +1189,3 @@ void Application::ResetProtocol() {
protocol_.reset();
});
}
main/application.h
+1
View File
@@ -160,6 +160,7 @@ private:
void HandleActivationDoneEvent();
void HandleWakeWordDetectedEvent();
void ContinueOpenAudioChannel(ListeningMode mode);
void BeginWakeWordInvoke(const std::string& wake_word);
void ContinueWakeWordInvoke(const std::string& wake_word);
void StartListeningAudio();
void ConfigureWakeWordForListening();
main/assets.cc
+139 -80
View File
@@ -1,33 +1,37 @@
#include "assets.h"
#include "application.h"
#include "board.h"
#include "display.h"
#include "application.h"
#include "lvgl_theme.h"
#include "emote_display.h"
#include "expression_emote.h"
#include "lvgl_theme.h"
#if HAVE_LVGL
#include "display/lcd_display.h"
#include <spi_flash_mmap.h>
#include "display/lcd_display.h"
#include "display/lvgl_display/lvgl_display.h"
#endif
#include <esp_heap_caps.h>
#include <esp_log.h>
#include <esp_timer.h>
#include <esp_heap_caps.h>
#include <cbin_font.h>
#include <noto_font_bundle.h>
#include <cstring>
#define TAG "Assets"
#define PARTITION_LABEL "assets"
struct mmap_assets_table {
char asset_name[32]; /*!< Name of the asset */
uint32_t asset_size; /*!< Size of the asset */
uint32_t asset_offset; /*!< Offset of the asset */
uint16_t asset_width; /*!< Width of the asset */
uint16_t asset_height; /*!< Height of the asset */
char asset_name[32]; /*!< Name of the asset */
uint32_t asset_size; /*!< Size of the asset */
uint32_t asset_offset; /*!< Offset of the asset */
uint16_t asset_width; /*!< Width of the asset */
uint16_t asset_height; /*!< Height of the asset */
};
Assets::Assets() {
UseBuiltInTextFontCapability();
#if HAVE_LVGL
strategy_ = std::make_unique<Assets::LvglStrategy>();
#else
@@ -37,12 +41,11 @@ Assets::Assets() {
InitializePartition();
}
Assets::~Assets() {
UnApplyPartition();
}
Assets::~Assets() { UnApplyPartition(); }
bool Assets::FindPartition(Assets* assets) {
assets->partition_ = esp_partition_find_first(ESP_PARTITION_TYPE_ANY, ESP_PARTITION_SUBTYPE_ANY, PARTITION_LABEL);
assets->partition_ = esp_partition_find_first(ESP_PARTITION_TYPE_ANY, ESP_PARTITION_SUBTYPE_ANY,
PARTITION_LABEL);
if (assets->partition_ == nullptr) {
ESP_LOGI(TAG, "No assets partition found");
return false;
@@ -59,11 +62,24 @@ bool Assets::InitializePartition() {
}
void Assets::UnApplyPartition() {
UseBuiltInTextFontCapability();
if (strategy_) {
strategy_->UnApplyPartition(this);
}
}
void Assets::UseBuiltInTextFontCapability() {
text_font_capability_ = {
.glyph_push = true,
.bundle = NOTO_FONT_BUNDLE_ID,
.charset = "basic",
.size = TEXT_FONT_SIZE,
.bpp = TEXT_FONT_BPP,
};
}
void Assets::DisableTextFontGlyphPush() { text_font_capability_ = {}; }
bool Assets::GetAssetData(const std::string& name, void*& ptr, size_t& size) {
return strategy_ ? strategy_->GetAssetData(this, name, ptr, size) : false;
}
@@ -140,11 +156,14 @@ bool Assets::LvglStrategy::InitializePartition(Assets* assets) {
ESP_LOGI(TAG, "The storage free size is %ld KB", storage_size / 1024);
ESP_LOGI(TAG, "The partition size is %ld KB", assets->partition_->size / 1024);
if (storage_size < assets->partition_->size) {
ESP_LOGE(TAG, "The free size %ld KB is less than assets partition required %ld KB", storage_size / 1024, assets->partition_->size / 1024);
ESP_LOGE(TAG, "The free size %ld KB is less than assets partition required %ld KB",
storage_size / 1024, assets->partition_->size / 1024);
return false;
}
esp_err_t err = esp_partition_mmap(assets->partition_, 0, assets->partition_->size, ESP_PARTITION_MMAP_DATA, (const void**)&mmap_root_, &mmap_handle_);
esp_err_t err =
esp_partition_mmap(assets->partition_, 0, assets->partition_->size, ESP_PARTITION_MMAP_DATA,
(const void**)&mmap_root_, &mmap_handle_);
if (err != ESP_OK) {
ESP_LOGE(TAG, "Failed to mmap assets partition: %s", esp_err_to_name(err));
return false;
@@ -157,7 +176,8 @@ bool Assets::LvglStrategy::InitializePartition(Assets* assets) {
uint32_t stored_len = *(uint32_t*)(mmap_root_ + 8);
if (stored_len > assets->partition_->size - 12) {
ESP_LOGD(TAG, "The stored_len (0x%lx) is greater than the partition size (0x%lx) - 12", stored_len, assets->partition_->size);
ESP_LOGD(TAG, "The stored_len (0x%lx) is greater than the partition size (0x%lx) - 12",
stored_len, assets->partition_->size);
return false;
}
@@ -167,7 +187,8 @@ bool Assets::LvglStrategy::InitializePartition(Assets* assets) {
ESP_LOGI(TAG, "The checksum calculation time is %d ms", int((end_time - start_time) / 1000));
if (calculated_checksum != stored_chksum) {
ESP_LOGE(TAG, "The calculated checksum (0x%lx) does not match the stored checksum (0x%lx)", calculated_checksum, stored_chksum);
ESP_LOGE(TAG, "The calculated checksum (0x%lx) does not match the stored checksum (0x%lx)",
calculated_checksum, stored_chksum);
return false;
}
@@ -175,10 +196,9 @@ bool Assets::LvglStrategy::InitializePartition(Assets* assets) {
for (uint32_t i = 0; i < stored_files; i++) {
auto item = (const mmap_assets_table*)(mmap_root_ + 12 + i * sizeof(mmap_assets_table));
auto asset = Asset{
.size = static_cast<size_t>(item->asset_size),
.offset = static_cast<size_t>(12 + sizeof(mmap_assets_table) * stored_files + item->asset_offset)
};
auto asset = Asset{.size = static_cast<size_t>(item->asset_size),
.offset = static_cast<size_t>(
12 + sizeof(mmap_assets_table) * stored_files + item->asset_offset)};
assets_[item->asset_name] = asset;
}
return checksum_valid_;
@@ -192,17 +212,19 @@ void Assets::LvglStrategy::UnApplyPartition(Assets* assets) {
}
checksum_valid_ = false;
assets_.clear();
(void)assets; // Unused parameter
(void)assets; // Unused parameter
}
bool Assets::LvglStrategy::GetAssetData(Assets* assets, const std::string& name, void*& ptr, size_t& size) {
bool Assets::LvglStrategy::GetAssetData(Assets* assets, const std::string& name, void*& ptr,
size_t& size) {
auto asset = assets_.find(name);
if (asset == assets_.end()) {
return false;
}
auto data = (const char*)(mmap_root_ + asset->second.offset);
if (data[0] != 'Z' || data[1] != 'Z') {
ESP_LOGE(TAG, "The asset %s is not valid with magic %02x%02x", name.c_str(), data[0], data[1]);
ESP_LOGE(TAG, "The asset %s is not valid with magic %02x%02x", name.c_str(), data[0],
data[1]);
return false;
}
@@ -228,7 +250,8 @@ bool Assets::LvglStrategy::Apply(Assets* assets, bool refresh_display_theme) {
cJSON* version = cJSON_GetObjectItem(root, "version");
if (cJSON_IsNumber(version)) {
if (version->valuedouble > 1) {
ESP_LOGE(TAG, "The assets version %d is not supported, please upgrade the firmware", version->valueint);
ESP_LOGE(TAG, "The assets version %d is not supported, please upgrade the firmware",
version->valueint);
return false;
}
}
@@ -244,15 +267,39 @@ bool Assets::LvglStrategy::Apply(Assets* assets, bool refresh_display_theme) {
std::string fonts_text_file = font->valuestring;
if (assets->GetAssetData(fonts_text_file, ptr, size)) {
auto text_font = std::make_shared<LvglCBinFont>(ptr);
if (text_font->font() == nullptr) {
ESP_LOGE(TAG, "Failed to load fonts.bin");
return false;
}
if (light_theme != nullptr) {
light_theme->set_text_font(text_font);
}
if (dark_theme != nullptr) {
dark_theme->set_text_font(text_font);
auto display = dynamic_cast<LvglDisplay*>(Board::GetInstance().GetDisplay());
if (text_font->font() == nullptr || display == nullptr ||
!display->SetTextFont(text_font)) {
ESP_LOGW(TAG, "Ignoring invalid text font asset %s", fonts_text_file.c_str());
} else {
assets->DisableTextFontGlyphPush();
cJSON* metadata = cJSON_GetObjectItem(root, "text_font_meta");
cJSON* charset = cJSON_GetObjectItem(metadata, "charset");
cJSON* font_size = cJSON_GetObjectItem(metadata, "size");
cJSON* font_bpp = cJSON_GetObjectItem(metadata, "bpp");
cJSON* bundle = cJSON_GetObjectItem(metadata, "bundle");
bool supports_glyph_push =
cJSON_IsString(charset) &&
(std::strcmp(charset->valuestring, "basic") == 0 ||
std::strcmp(charset->valuestring, "common") == 0) &&
cJSON_IsNumber(font_size) && font_size->valueint > 0 &&
font_size->valuedouble == font_size->valueint && cJSON_IsNumber(font_bpp) &&
font_bpp->valuedouble == font_bpp->valueint &&
(font_bpp->valueint == 1 || font_bpp->valueint == 4) &&
font_bpp->valueint == text_font->bpp() && cJSON_IsString(bundle) &&
bundle->valuestring[0] != '\0' && std::strlen(bundle->valuestring) <= 64;
if (supports_glyph_push) {
assets->text_font_capability_ = {
.glyph_push = true,
.bundle = bundle->valuestring,
.charset = charset->valuestring,
.size = font_size->valueint,
.bpp = font_bpp->valueint,
};
} else {
ESP_LOGW(TAG, "Loaded custom text font without compatible glyph push metadata");
}
}
} else {
ESP_LOGE(TAG, "The font file %s is not found", fonts_text_file.c_str());
@@ -269,12 +316,14 @@ bool Assets::LvglStrategy::Apply(Assets* assets, bool refresh_display_theme) {
cJSON* name = cJSON_GetObjectItem(emoji, "name");
cJSON* file = cJSON_GetObjectItem(emoji, "file");
cJSON* eaf = cJSON_GetObjectItem(emoji, "eaf");
if (cJSON_IsString(name) && cJSON_IsString(file) && (NULL== eaf)) {
if (cJSON_IsString(name) && cJSON_IsString(file) && (NULL == eaf)) {
if (!assets->GetAssetData(file->valuestring, ptr, size)) {
ESP_LOGE(TAG, "Emoji %s image file %s is not found", name->valuestring, file->valuestring);
ESP_LOGE(TAG, "Emoji %s image file %s is not found", name->valuestring,
file->valuestring);
continue;
}
custom_emoji_collection->AddEmoji(name->valuestring, new LvglRawImage(ptr, size));
custom_emoji_collection->AddEmoji(name->valuestring,
new LvglRawImage(ptr, size));
}
}
}
@@ -284,6 +333,7 @@ bool Assets::LvglStrategy::Apply(Assets* assets, bool refresh_display_theme) {
if (dark_theme != nullptr) {
dark_theme->set_emoji_collection(custom_emoji_collection);
}
Board::GetInstance().GetDisplay()->SetEmojiCollection(custom_emoji_collection);
}
cJSON* skin = cJSON_GetObjectItem(root, "skin");
@@ -297,12 +347,15 @@ bool Assets::LvglStrategy::Apply(Assets* assets, bool refresh_display_theme) {
light_theme->set_text_color(LvglTheme::ParseColor(text_color->valuestring));
}
if (cJSON_IsString(background_color)) {
light_theme->set_background_color(LvglTheme::ParseColor(background_color->valuestring));
light_theme->set_chat_background_color(LvglTheme::ParseColor(background_color->valuestring));
light_theme->set_background_color(
LvglTheme::ParseColor(background_color->valuestring));
light_theme->set_chat_background_color(
LvglTheme::ParseColor(background_color->valuestring));
}
if (cJSON_IsString(background_image)) {
if (!assets->GetAssetData(background_image->valuestring, ptr, size)) {
ESP_LOGE(TAG, "The background image file %s is not found", background_image->valuestring);
ESP_LOGE(TAG, "The background image file %s is not found",
background_image->valuestring);
return false;
}
auto background_image = std::make_shared<LvglCBinImage>(ptr);
@@ -318,12 +371,15 @@ bool Assets::LvglStrategy::Apply(Assets* assets, bool refresh_display_theme) {
dark_theme->set_text_color(LvglTheme::ParseColor(text_color->valuestring));
}
if (cJSON_IsString(background_color)) {
dark_theme->set_background_color(LvglTheme::ParseColor(background_color->valuestring));
dark_theme->set_chat_background_color(LvglTheme::ParseColor(background_color->valuestring));
dark_theme->set_background_color(
LvglTheme::ParseColor(background_color->valuestring));
dark_theme->set_chat_background_color(
LvglTheme::ParseColor(background_color->valuestring));
}
if (cJSON_IsString(background_image)) {
if (!assets->GetAssetData(background_image->valuestring, ptr, size)) {
ESP_LOGE(TAG, "The background image file %s is not found", background_image->valuestring);
ESP_LOGE(TAG, "The background image file %s is not found",
background_image->valuestring);
return false;
}
auto background_image = std::make_shared<LvglCBinImage>(ptr);
@@ -352,11 +408,11 @@ bool Assets::LvglStrategy::Apply(Assets* assets, bool refresh_display_theme) {
}
}
}
cJSON_Delete(root);
return true;
}
#endif // HAVE_LVGL
#endif // HAVE_LVGL
bool Assets::EmoteStrategy::InitializePartition(Assets* assets) {
assets->partition_valid_ = false;
@@ -371,12 +427,14 @@ bool Assets::EmoteStrategy::InitializePartition(Assets* assets) {
if (emote_display && emote_display->GetEmoteHandle() != nullptr) {
const emote_data_t data = {
.type = EMOTE_SOURCE_PARTITION,
.source = {
.partition_label = PARTITION_LABEL,
},
.flags = {
.mmap_enable = true, //must be true here!!!
},
.source =
{
.partition_label = PARTITION_LABEL,
},
.flags =
{
.mmap_enable = true, // must be true here!!!
},
};
ret = emote_mount_assets(emote_display->GetEmoteHandle(), &data);
} else {
@@ -392,16 +450,18 @@ void Assets::EmoteStrategy::UnApplyPartition(Assets* assets) {
if (emote_display && emote_display->GetEmoteHandle() != nullptr) {
emote_unmount_assets(emote_display->GetEmoteHandle());
}
(void)assets; // Unused parameter
(void)assets; // Unused parameter
}
bool Assets::EmoteStrategy::GetAssetData(Assets* assets, const std::string& name, void*& ptr, size_t& size) {
bool Assets::EmoteStrategy::GetAssetData(Assets* assets, const std::string& name, void*& ptr,
size_t& size) {
auto display = Board::GetInstance().GetDisplay();
auto* emote_display = dynamic_cast<emote::EmoteDisplay*>(display);
if (emote_display && emote_display->GetEmoteHandle() != nullptr) {
const uint8_t* data = nullptr;
size_t data_size = 0;
if (ESP_OK == emote_get_asset_data_by_name(emote_display->GetEmoteHandle(), name.c_str(), &data, &data_size)) {
if (ESP_OK == emote_get_asset_data_by_name(emote_display->GetEmoteHandle(), name.c_str(),
&data, &data_size)) {
ptr = const_cast<void*>(static_cast<const void*>(data));
size = data_size;
return true;
@@ -409,7 +469,7 @@ bool Assets::EmoteStrategy::GetAssetData(Assets* assets, const std::string& name
ESP_LOGE(TAG, "Failed to get asset data by name: %s", name.c_str());
return false;
}
(void)assets; // Unused parameter
(void)assets; // Unused parameter
return false;
}
@@ -425,7 +485,8 @@ bool Assets::EmoteStrategy::Apply(Assets* assets, bool refresh_display_theme) {
return true;
}
bool Assets::Download(std::string url, std::function<void(int progress, size_t speed)> progress_callback) {
bool Assets::Download(std::string url,
std::function<void(int progress, size_t speed)> progress_callback) {
ESP_LOGI(TAG, "Downloading new version of assets from %s", url.c_str());
auto network = Board::GetInstance().GetNetwork();
@@ -449,25 +510,24 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
}
if (content_length > partition_->size) {
ESP_LOGE(TAG, "Assets file size (%u) is larger than partition size (%lu)",
content_length, partition_->size);
ESP_LOGE(TAG, "Assets file size (%u) is larger than partition size (%lu)", content_length,
partition_->size);
return false;
}
constexpr size_t HEADER_SIZE = 12;
if (content_length < HEADER_SIZE) {
ESP_LOGE(TAG, "Content length (%u) is smaller than header size (%u)",
content_length, HEADER_SIZE);
ESP_LOGE(TAG, "Content length (%u) is smaller than header size (%u)", content_length,
HEADER_SIZE);
return false;
}
const size_t SECTOR_SIZE = esp_partition_get_main_flash_sector_size();
using BufferPtr = std::unique_ptr<char, decltype(&heap_caps_free)>;
BufferPtr buffer(
static_cast<char *>(heap_caps_malloc(SECTOR_SIZE, MALLOC_CAP_INTERNAL)),
&heap_caps_free);
BufferPtr buffer(static_cast<char*>(heap_caps_malloc(SECTOR_SIZE, MALLOC_CAP_INTERNAL)),
&heap_caps_free);
if (!buffer) {
ESP_LOGE(TAG, "Failed to allocate buffer");
@@ -482,8 +542,7 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
ESP_LOGI(TAG,
"Sector size: %u, content length: %u, "
"sectors to erase: %u, total erase size: %u",
SECTOR_SIZE, content_length,
sectors_to_erase, total_erase_size);
SECTOR_SIZE, content_length, sectors_to_erase, total_erase_size);
size_t total_written = 0;
size_t recent_written = 0;
@@ -529,17 +588,17 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
size_t sector_start = current_sector * SECTOR_SIZE;
size_t sector_end = sector_start + SECTOR_SIZE;
if (sector_end > partition_->size) {
ESP_LOGE(TAG, "Sector end (%u) exceeds partition size (%lu)",
sector_end, partition_->size);
ESP_LOGE(TAG, "Sector end (%u) exceeds partition size (%lu)", sector_end,
partition_->size);
erase_failed = true;
break;
}
ESP_LOGD(TAG, "Erasing sector %u (offset: %u, size: %u)",
current_sector, sector_start, SECTOR_SIZE);
ESP_LOGD(TAG, "Erasing sector %u (offset: %u, size: %u)", current_sector,
sector_start, SECTOR_SIZE);
esp_err_t err = esp_partition_erase_range(partition_, sector_start, SECTOR_SIZE);
if (err != ESP_OK) {
ESP_LOGE(TAG, "Failed to erase sector %u at offset %u: %s",
current_sector, sector_start, esp_err_to_name(err));
ESP_LOGE(TAG, "Failed to erase sector %u at offset %u: %s", current_sector,
sector_start, esp_err_to_name(err));
erase_failed = true;
break;
}
@@ -563,15 +622,13 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
}
// Calculate progress
if (esp_timer_get_time() - last_calc_time >= 1000000 || (header_collected + total_written) == content_length) {
if (esp_timer_get_time() - last_calc_time >= 1000000 ||
(header_collected + total_written) == content_length) {
size_t progress = (header_collected + total_written) * 100 / content_length;
size_t speed = recent_written;
ESP_LOGI(TAG, "Progress: %u%% (%u/%u), Speed: %u B/s, Sectors erased: %u",
progress,
(unsigned int)(header_collected + total_written),
(unsigned int)content_length,
(unsigned int)speed,
(unsigned int)current_sector);
ESP_LOGI(TAG, "Progress: %u%% (%u/%u), Speed: %u B/s, Sectors erased: %u", progress,
(unsigned int)(header_collected + total_written), (unsigned int)content_length,
(unsigned int)speed, (unsigned int)current_sector);
if (progress_callback) {
progress_callback(progress, speed);
@@ -602,7 +659,9 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
return false;
}
ESP_LOGI(TAG, "Header written, assets download completed, total written: %u bytes, total sectors erased: %u",
ESP_LOGI(TAG,
"Header written, assets download completed, total written: %u bytes, total sectors "
"erased: %u",
(unsigned int)(header_collected + total_written), (unsigned int)current_sector);
// Re-initialize the assets partition
@@ -612,4 +671,4 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
}
return true;
}
}
main/assets.h
+27 -10
View File
@@ -1,12 +1,12 @@
#ifndef ASSETS_H
#define ASSETS_H
#include <string>
#include <functional>
#include <memory>
#include <string>
#include <cJSON.h>
#include <esp_partition.h>
#include <cJSON.h>
#include <model_path.h>
#include <map>
#include <string>
@@ -20,6 +20,14 @@ struct Asset {
size_t offset;
};
struct TextFontCapability {
bool glyph_push = false;
std::string bundle;
std::string charset;
int size = 0;
int bpp = 0;
};
class Assets {
public:
static Assets& GetInstance() {
@@ -28,12 +36,14 @@ public:
}
~Assets();
bool Download(std::string url, std::function<void(int progress, size_t speed)> progress_callback);
bool Download(std::string url,
std::function<void(int progress, size_t speed)> progress_callback);
bool Apply(bool refresh_display_theme = true);
bool GetAssetData(const std::string& name, void*& ptr, size_t& size);
inline bool partition_valid() const { return partition_valid_; }
inline std::string default_assets_url() const { return default_assets_url_; }
inline TextFontCapability text_font_capability() const { return text_font_capability_; }
private:
Assets();
@@ -44,22 +54,27 @@ private:
void UnApplyPartition();
static bool FindPartition(Assets* assets);
static bool LoadSrmodelsFromIndex(Assets* assets, cJSON* root = nullptr);
void UseBuiltInTextFontCapability();
void DisableTextFontGlyphPush();
class AssetStrategy {
public:
virtual ~AssetStrategy() = default;
virtual bool Apply(Assets* assets, bool refresh_display_theme = true) = 0;
virtual bool InitializePartition(Assets* assets) = 0;
virtual void UnApplyPartition(Assets* assets) = 0;
virtual bool GetAssetData(Assets* assets, const std::string& name, void*& ptr, size_t& size) = 0;
virtual bool GetAssetData(Assets* assets, const std::string& name, void*& ptr,
size_t& size) = 0;
};
class LvglStrategy : public AssetStrategy {
public:
bool Apply(Assets* assets, bool refresh_display_theme = true) override;
bool InitializePartition(Assets* assets) override;
void UnApplyPartition(Assets* assets) override;
bool GetAssetData(Assets* assets, const std::string& name, void*& ptr, size_t& size) override;
bool GetAssetData(Assets* assets, const std::string& name, void*& ptr,
size_t& size) override;
private:
static uint32_t CalculateChecksum(const char* data, uint32_t length);
std::map<std::string, Asset> assets_;
@@ -67,15 +82,16 @@ private:
const char* mmap_root_ = nullptr;
bool checksum_valid_ = false;
};
class EmoteStrategy : public AssetStrategy {
public:
bool Apply(Assets* assets, bool refresh_display_theme = true) override;
bool InitializePartition(Assets* assets) override;
void UnApplyPartition(Assets* assets) override;
bool GetAssetData(Assets* assets, const std::string& name, void*& ptr, size_t& size) override;
bool GetAssetData(Assets* assets, const std::string& name, void*& ptr,
size_t& size) override;
};
// Strategy instance
std::unique_ptr<AssetStrategy> strategy_;
@@ -83,6 +99,7 @@ protected:
const esp_partition_t* partition_ = nullptr;
bool partition_valid_ = false;
std::string default_assets_url_;
TextFontCapability text_font_capability_;
srmodel_list_t* models_list_ = nullptr;
};
main/boards/atk-dnesp32s3-box3/atk_dnesp32s3_box3.cc
+2 -2
View File
@@ -30,8 +30,8 @@
#define TAG "atk_dnesp32s3_box3"
LV_FONT_DECLARE(font_puhui_20_4);
LV_FONT_DECLARE(font_awesome_20_4);
LV_FONT_DECLARE(font_noto_sans_basic_20_4);
LV_FONT_DECLARE(font_material_symbols_20_4);
class atk_dnesp32s3_box3 : public WifiBoard {
private:
main/boards/atoms3-echo-base/atoms3_echo_base.cc
+1 -1
View File
@@ -132,7 +132,7 @@ private:
display_->SetupUI();
display_->SetStatus(Lang::Strings::ERROR);
display_->SetEmotion("triangle_exclamation");
display_->SetEmotion("warning");
display_->SetChatMessage("system", "Echo Base\nnot connected");
while (1) {
main/boards/atoms3r-echo-base/atoms3r_echo_base.cc
+1 -1
View File
@@ -199,7 +199,7 @@ private:
display_->SetupUI();
display_->SetStatus(Lang::Strings::ERROR);
display_->SetEmotion("triangle_exclamation");
display_->SetEmotion("warning");
display_->SetChatMessage("system", "Echo Base\nnot connected");
while (1) {
main/boards/atoms3r-echo-pyramid/atoms3r_echo_pyramid.cc
+1 -1
View File
@@ -642,7 +642,7 @@ private:
display_->SetupUI();
display_->SetStatus(Lang::Strings::ERROR);
display_->SetEmotion("triangle_exclamation");
display_->SetEmotion("warning");
display_->SetChatMessage("system", "Echo Pyramid\nnot connected");
while (1) {
main/boards/common/ethernet_board.cc
+2 -2
View File
@@ -12,7 +12,7 @@
#include <esp_mac.h>
#include <esp_network.h>
#include <esp_netif.h>
#include <font_awesome.h>
#include <material_symbols.h>
#include <freertos/FreeRTOS.h>
#include <freertos/task.h>
#include <soc/soc_caps.h>
@@ -221,7 +221,7 @@ NetworkInterface* EthernetBoard::GetNetwork() {
}
const char* EthernetBoard::GetNetworkStateIcon() {
return connected_ ? FONT_AWESOME_SIGNAL_STRONG : FONT_AWESOME_SIGNAL_OFF;
return connected_ ? MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR : MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR_OFF;
}
std::string EthernetBoard::GetEthernetMacAddress() {
main/boards/common/ml307_board.cc
+8 -8
View File
@@ -7,7 +7,7 @@
#include <esp_timer.h>
#include <freertos/FreeRTOS.h>
#include <freertos/task.h>
#include <font_awesome.h>
#include <material_symbols.h>
#include <utility>
static const char *TAG = "Ml307Board";
@@ -146,23 +146,23 @@ NetworkInterface* Ml307Board::GetNetwork() {
const char* Ml307Board::GetNetworkStateIcon() {
if (modem_ == nullptr || !modem_->network_ready()) {
return FONT_AWESOME_SIGNAL_OFF;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR_OFF;
}
int csq = modem_->GetCsq();
if (csq == -1) {
return FONT_AWESOME_SIGNAL_OFF;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR_OFF;
} else if (csq >= 0 && csq <= 9) {
return FONT_AWESOME_SIGNAL_WEAK;
return MATERIAL_SYMBOLS_SIGNAL_CELLULAR_ALT_1_BAR;
} else if (csq >= 10 && csq <= 14) {
return FONT_AWESOME_SIGNAL_FAIR;
return MATERIAL_SYMBOLS_SIGNAL_CELLULAR_ALT_2_BAR;
} else if (csq >= 15 && csq <= 19) {
return FONT_AWESOME_SIGNAL_GOOD;
return MATERIAL_SYMBOLS_SIGNAL_CELLULAR_ALT;
} else if (csq >= 20 && csq <= 31) {
return FONT_AWESOME_SIGNAL_STRONG;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR;
}
ESP_LOGW(TAG, "Invalid CSQ: %d", csq);
return FONT_AWESOME_SIGNAL_OFF;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR_OFF;
}
std::string Ml307Board::GetBoardJson() {
main/boards/common/nt26_board.cc
+8 -8
View File
@@ -3,7 +3,7 @@
#include "application.h"
#include "audio_codec.h"
#include <esp_log.h>
#include <font_awesome.h>
#include <material_symbols.h>
#include <cJSON.h>
#define TAG "Nt26Board"
@@ -146,21 +146,21 @@ NetworkInterface* Nt26Board::GetNetwork() {
const char* Nt26Board::GetNetworkStateIcon() {
if (modem_ == nullptr || !modem_->IsInitialized()) {
return FONT_AWESOME_SIGNAL_OFF;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR_OFF;
}
int csq = modem_->GetSignalStrength();
if (csq == 99 || csq == -1) {
return FONT_AWESOME_SIGNAL_OFF;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR_OFF;
} else if (csq >= 0 && csq <= 9) {
return FONT_AWESOME_SIGNAL_WEAK;
return MATERIAL_SYMBOLS_SIGNAL_CELLULAR_ALT_1_BAR;
} else if (csq >= 10 && csq <= 14) {
return FONT_AWESOME_SIGNAL_FAIR;
return MATERIAL_SYMBOLS_SIGNAL_CELLULAR_ALT_2_BAR;
} else if (csq >= 15 && csq <= 19) {
return FONT_AWESOME_SIGNAL_GOOD;
return MATERIAL_SYMBOLS_SIGNAL_CELLULAR_ALT;
} else if (csq >= 20 && csq <= 31) {
return FONT_AWESOME_SIGNAL_STRONG;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR;
}
return FONT_AWESOME_SIGNAL_OFF;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR_OFF;
}
void Nt26Board::SetPowerSaveLevel(PowerSaveLevel level) {
main/boards/common/rndis_board.cc
+3 -3
View File
@@ -10,7 +10,7 @@
#include <esp_network.h>
#include <esp_log.h>
#include <utility>
#include <font_awesome.h>
#include <material_symbols.h>
#if CONFIG_IDF_TARGET_ESP32P4 || CONFIG_IDF_TARGET_ESP32S3
@@ -176,7 +176,7 @@ NetworkInterface* RndisBoard::GetNetwork() {
}
const char* RndisBoard::GetNetworkStateIcon() {
return FONT_AWESOME_SIGNAL_STRONG;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR;
}
std::string RndisBoard::GetBoardJson() {
@@ -244,4 +244,4 @@ std::string RndisBoard::GetDeviceStatusJson() {
cJSON_Delete(root);
return result;
}
#endif // CONFIG_IDF_TARGET_ESP32P4 || CONFIG_IDF_TARGET_ESP32S3
#endif // CONFIG_IDF_TARGET_ESP32P4 || CONFIG_IDF_TARGET_ESP32S3
main/boards/common/wifi_board.cc
+6 -6
View File
@@ -13,7 +13,7 @@
#include <esp_mac.h>
#include <utility>
#include <font_awesome.h>
#include <material_symbols.h>
#include <wifi_manager.h>
#include <wifi_station.h>
#include <ssid_manager.h>
@@ -259,19 +259,19 @@ const char* WifiBoard::GetNetworkStateIcon() {
auto& wifi = WifiManager::GetInstance();
if (wifi.IsConfigMode()) {
return FONT_AWESOME_WIFI;
return MATERIAL_SYMBOLS_WIFI;
}
if (!wifi.IsConnected()) {
return FONT_AWESOME_WIFI_SLASH;
return MATERIAL_SYMBOLS_WIFI_OFF;
}
int rssi = wifi.GetRssi();
if (rssi >= -65) {
return FONT_AWESOME_WIFI;
return MATERIAL_SYMBOLS_WIFI;
} else if (rssi >= -75) {
return FONT_AWESOME_WIFI_FAIR;
return MATERIAL_SYMBOLS_WIFI_2_BAR;
}
return FONT_AWESOME_WIFI_WEAK;
return MATERIAL_SYMBOLS_WIFI_1_BAR;
}
std::string WifiBoard::GetBoardJson() {
main/boards/esp-box-3/README.md
+3
View File
@@ -58,6 +58,9 @@ ESP-BOX-3 支持多种不同的 UI 显示风格，通过 menuconfig 配置选择
https://dl.espressif.com/AE/wn9_nihaoxiaozhi_tts-font_puhui_common_20_4-esp-box-3.bin
```
这是保留表情动画和模型的历史资源包；新版固件会忽略其中没有 Noto 元数据的旧 Puhui
字体，并使用内置 Noto basic 字体。待新的 Noto 资源包上传后可直接替换此 URL。
##### 默认消息风格 (Enable default message style)
- **配置选项**: `USE_DEFAULT_MESSAGE_STYLE` (默认)
- **特点**: 使用标准的消息显示界面
main/boards/esp-vocat/README.md
+4 -1
View File
@@ -49,6 +49,9 @@ ESP-VoCat 支持多种不同的 UI 显示风格，通过 menuconfig 配置选择
https://dl.espressif.com/AE/wn9_nihaoxiaozhi_tts-font_puhui_common_20_4-echoear.bin
```
这是保留表情动画和模型的历史资源包；新版固件会忽略其中没有 Noto 元数据的旧 Puhui
字体，并使用内置 Noto basic 字体。待新的 Noto 资源包上传后可直接替换此 URL。
##### 默认消息风格 (Enable default message style)
- **配置选项**: `USE_DEFAULT_MESSAGE_STYLE` (默认)
- **特点**: 使用标准的消息显示界面
@@ -79,4 +82,4 @@ idf.py build
```bash
idf.py flash
```
```
main/boards/m5stack-tab5/m5stack_tab5.cc
-1
View File
@@ -3,7 +3,6 @@
#include "display/lcd_display.h"
#include "esp_lcd_ili9881c.h"
#include "esp_lcd_st7123.h"
#include "font_emoji.h"
#include "application.h"
#include "button.h"
#include "config.h"
main/boards/quandong-s3-dev/HARDWARE_DIFF.md
+3 -3
View File
@@ -16,7 +16,7 @@
| 板载 LED | 无 | GPIO48 单色 |
| MCP 外设示例 | 无 | `LampController`（GPIO18） |
| 背光 | PWM on GPIO45 | OLED 无背光 |
| 字体 / emoji 资源 | `font_noto_basic_20_4` + `font_awesome_20_4` + `noto-emoji_128` | `font_puhui_basic_14_1` + `font_awesome_14_1`（无 emoji） |
| 字体 / emoji 资源 | `font_noto_sans_basic_20_4` + `font_material_symbols_20_4` + `noto-color-emoji_64` | `font_noto_sans_basic_14_1` + `font_material_symbols_14_1`（无 emoji） |
---
@@ -144,8 +144,8 @@ InitializeTools() // 注册 LampController（MCP 外设）
| 板子 | text font | icon font | emoji collection |
|---|---|---|---|
| quandong-s3-dev | `font_noto_basic_20_4` | `font_awesome_20_4` | `noto-emoji_128` |
| bread-compact-wifi | `font_puhui_basic_14_1` | `font_awesome_14_1` | （无） |
| quandong-s3-dev | `font_noto_sans_basic_20_4` | `font_material_symbols_20_4` | `noto-color-emoji_64` |
| bread-compact-wifi | `font_noto_sans_basic_14_1` | `font_material_symbols_14_1` | （无） |
→ quandong 走的是和 `lichuang-dev`、`esp-box-3` 等 240×320 LCD 板同档资源；OLED 板因为像素少，用 14 像素位图字体即可，emoji 也加载不动。
main/boards/sp-esp32-s3-1.28-box/sp-esp32-s3-1.28-box.cc
+2 -2
View File
@@ -29,8 +29,8 @@
#define TAG "Spotpear_ESP32_S3_1_28_BOX"
LV_FONT_DECLARE(font_puhui_16_4);
LV_FONT_DECLARE(font_awesome_16_4);
LV_FONT_DECLARE(font_noto_sans_basic_16_4);
LV_FONT_DECLARE(font_material_symbols_16_4);
class Cst816d : public I2cDevice {
main/boards/waveshare/esp32-s3-rgb-matrix/rgb_matrix_display.cc
+15 -15
View File
@@ -2,11 +2,12 @@
#include <esp_heap_caps.h>
#include <esp_log.h>
#include <esp_lvgl_port.h>
#include <font_awesome.h>
#include <material_symbols.h>
#include <stdio.h>
#include <time.h>
#include <cstring>
#include <string>
#include <utility>
#include "application.h"
#include "assets/lang_config.h"
#include "board.h"
@@ -15,12 +16,9 @@
#include "hub75.h"
// 声明中文字体
LV_FONT_DECLARE(font_puhui_14_1);
LV_FONT_DECLARE(font_noto_sans_basic_14_1);
LV_FONT_DECLARE(BUILTIN_ICON_FONT);
LV_FONT_DECLARE(font_awesome_30_4);
// 声明32x32 Emoji集合
class Twemoji32;
LV_FONT_DECLARE(font_material_symbols_30_4);
struct Hub75Context {
Hub75Driver driver;
@@ -41,7 +39,7 @@ bool UseBuiltInEmotionIcon(const char* emotion) {
if (emotion == nullptr) {
return false;
}
if (strcmp(emotion, "microchip_ai") == 0) {
if (strcmp(emotion, "robot_2") == 0) {
return true;
}
if (strcmp(emotion, "link") == 0) {
@@ -270,9 +268,6 @@ void CustomMatrixDisplay::SetupUI() {
}
Display::SetupUI();
// 初始化 Emoji 资源（用于 SetEmotion）
emoji_collection_ = std::make_shared<Twemoji32>();
const int ui_width_px = width_;
const int ui_height_px = height_;
@@ -301,7 +296,7 @@ void CustomMatrixDisplay::SetupUI() {
lv_label_set_long_mode(status_label_, LV_LABEL_LONG_CLIP);
lv_obj_set_style_text_align(status_label_, LV_TEXT_ALIGN_RIGHT, 0);
lv_obj_set_style_text_color(status_label_, lv_color_white(), 0);
lv_obj_set_style_text_font(status_label_, &font_puhui_14_1, 0);
lv_obj_set_style_text_font(status_label_, &font_noto_sans_basic_14_1, 0);
lv_obj_align(status_label_, LV_ALIGN_TOP_RIGHT, 0, -1);
status_text_ = "初始化";
RefreshStatusLabelLocked();
@@ -312,7 +307,7 @@ void CustomMatrixDisplay::SetupUI() {
lv_label_set_long_mode(message_label_, LV_LABEL_LONG_SCROLL_CIRCULAR);
lv_obj_set_style_text_align(message_label_, LV_TEXT_ALIGN_CENTER, 0);
lv_obj_set_style_text_color(message_label_, lv_color_white(), 0);
lv_obj_set_style_text_font(message_label_, &font_puhui_14_1, 0);
lv_obj_set_style_text_font(message_label_, &font_noto_sans_basic_14_1, 0);
lv_obj_align(message_label_, LV_ALIGN_BOTTOM_MID, 0, 0);
lv_label_set_text(message_label_, "hi 小智");
@@ -322,13 +317,18 @@ void CustomMatrixDisplay::SetupUI() {
lv_obj_add_flag(emoji_image_, LV_OBJ_FLAG_HIDDEN);
emotion_icon_label_ = lv_label_create(main_container_);
lv_label_set_text(emotion_icon_label_, FONT_AWESOME_MICROCHIP_AI);
lv_obj_set_style_text_font(emotion_icon_label_, &font_awesome_30_4, 0);
lv_label_set_text(emotion_icon_label_, MATERIAL_SYMBOLS_ROBOT_2);
lv_obj_set_style_text_font(emotion_icon_label_, &font_material_symbols_30_4, 0);
lv_obj_set_style_text_color(emotion_icon_label_, lv_color_white(), 0);
lv_obj_align(emotion_icon_label_, LV_ALIGN_CENTER, 0, -1);
lv_obj_remove_flag(emotion_icon_label_, LV_OBJ_FLAG_HIDDEN);
}
void CustomMatrixDisplay::SetEmojiCollection(std::shared_ptr<EmojiCollection> collection) {
DisplayLockGuard lock(this);
emoji_collection_ = std::move(collection);
}
void CustomMatrixDisplay::SetEmotion(const char* emotion) {
DisplayLockGuard lock(this);
@@ -358,7 +358,7 @@ void CustomMatrixDisplay::SetEmotion(const char* emotion) {
if (emotion_icon_label_ == nullptr) {
return;
}
lv_label_set_text(emotion_icon_label_, FONT_AWESOME_MICROCHIP_AI);
lv_label_set_text(emotion_icon_label_, MATERIAL_SYMBOLS_ROBOT_2);
SetObjectVisible(emotion_icon_label_, true);
}
main/boards/waveshare/esp32-s3-rgb-matrix/rgb_matrix_display.h
+1
View File
@@ -16,6 +16,7 @@ public:
void SetBrightness(uint8_t brightness_0_100);
virtual void SetEmotion(const char* emotion) override;
virtual void SetEmojiCollection(std::shared_ptr<EmojiCollection> collection) override;
virtual void SetChatMessage(const char* role, const char* content) override;
virtual void SetStatus(const char* status) override;
virtual void ShowNotification(const char* notification, int duration_ms = 3000) override;
main/boards/waveshare/esp32-s3-touch-lcd-7c/esp32_s3_touch_lcd_7c.cc
-1
View File
@@ -1,6 +1,5 @@
#include "wifi_board.h"
#include "display/lcd_display.h"
// #include "font_awesome_symbols.h"
#include "codecs/box_audio_codec.h"
#include "waveshare_audio_codec.h"
main/boards/xingzhi-abs-2.0/customlcddisplay.h
+3 -3
View File
@@ -4,8 +4,8 @@
#include "display/lcd_display.h"
#include "lvgl_theme.h"
LV_FONT_DECLARE(font_puhui_16_4);
LV_FONT_DECLARE(font_awesome_16_4);
LV_FONT_DECLARE(font_noto_sans_basic_16_4);
LV_FONT_DECLARE(font_material_symbols_16_4);
class CustomLcdDisplay : public SpiLcdDisplay {
private:
@@ -91,4 +91,4 @@ public:
}
};
#endif
#endif
main/boards/zhengchen-cam-ml307/zhengchen_cam_board_ml307.cc
+2 -2
View File
@@ -28,8 +28,8 @@
//控制器初始化函数声明
void InitializeMCPController();
LV_FONT_DECLARE(font_puhui_20_4);
LV_FONT_DECLARE(font_awesome_20_4);
LV_FONT_DECLARE(font_noto_sans_basic_20_4);
LV_FONT_DECLARE(font_material_symbols_20_4);
class Pca9557 : public I2cDevice {
public:
main/boards/zhengchen-cam/zhengchen_cam_board.cc
+2 -2
View File
@@ -28,8 +28,8 @@
//控制器初始化函数声明
void InitializeMCPController();
LV_FONT_DECLARE(font_puhui_20_4);
LV_FONT_DECLARE(font_awesome_20_4);
LV_FONT_DECLARE(font_noto_sans_basic_20_4);
LV_FONT_DECLARE(font_material_symbols_20_4);
class Pca9557 : public I2cDevice {
public:
main/display/display.cc
+13 -25
View File
@@ -1,30 +1,24 @@
#include <esp_log.h>
#include "display.h"
#include <esp_err.h>
#include <string>
#include <esp_log.h>
#include <cstdlib>
#include <cstring>
#include <font_awesome.h>
#include "display.h"
#include "board.h"
#include <string>
#include "application.h"
#include "audio_codec.h"
#include "settings.h"
#include "assets/lang_config.h"
#include "audio_codec.h"
#include "board.h"
#include "settings.h"
#define TAG "Display"
Display::Display() {
}
Display::Display() {}
Display::~Display() {
}
Display::~Display() {}
void Display::SetStatus(const char* status) {
ESP_LOGW(TAG, "SetStatus: %s", status);
}
void Display::SetStatus(const char* status) { ESP_LOGW(TAG, "SetStatus: %s", status); }
void Display::ShowNotification(const std::string &notification, int duration_ms) {
void Display::ShowNotification(const std::string& notification, int duration_ms) {
ShowNotification(notification.c_str(), duration_ms);
}
@@ -32,13 +26,9 @@ void Display::ShowNotification(const char* notification, int duration_ms) {
ESP_LOGW(TAG, "ShowNotification: %s", notification);
}
void Display::UpdateStatusBar(bool update_all) {
}
void Display::UpdateStatusBar(bool update_all) {}
void Display::SetEmotion(const char* emotion) {
ESP_LOGW(TAG, "SetEmotion: %s", emotion);
}
void Display::SetEmotion(const char* emotion) { ESP_LOGW(TAG, "SetEmotion: %s", emotion); }
void Display::SetChatMessage(const char* role, const char* content) {
ESP_LOGW(TAG, "Role:%s", role);
@@ -55,6 +45,4 @@ void Display::SetTheme(Theme* theme) {
settings.SetString("theme", theme->name());
}
void Display::SetPowerSaveMode(bool on) {
ESP_LOGW(TAG, "SetPowerSaveMode: %d", on);
}
void Display::SetPowerSaveMode(bool on) { ESP_LOGW(TAG, "SetPowerSaveMode: %d", on); }
main/display/display.h
+15 -15
View File
@@ -2,18 +2,21 @@
#define DISPLAY_H
#include "emoji_collection.h"
#include "text_glyph.h"
#ifndef CONFIG_USE_EMOTE_MESSAGE_STYLE
#define HAVE_LVGL 1
#include <lvgl.h>
#endif
#include <esp_timer.h>
#include <esp_log.h>
#include <esp_pm.h>
#include <esp_timer.h>
#include <string>
#include <chrono>
#include <cstdint>
#include <string>
#include <vector>
class Theme {
public:
@@ -21,6 +24,7 @@ public:
virtual ~Theme() = default;
inline std::string name() const { return name_; }
private:
std::string name_;
};
@@ -32,7 +36,7 @@ public:
virtual void SetStatus(const char* status);
virtual void ShowNotification(const char* notification, int duration_ms = 3000);
virtual void ShowNotification(const std::string &notification, int duration_ms = 3000);
virtual void ShowNotification(const std::string& notification, int duration_ms = 3000);
virtual void SetEmotion(const char* emotion);
virtual void SetChatMessage(const char* role, const char* content);
virtual void ClearChatMessages();
@@ -40,9 +44,10 @@ public:
virtual Theme* GetTheme() { return current_theme_; }
virtual void UpdateStatusBar(bool update_all = false);
virtual void SetPowerSaveMode(bool on);
virtual void SetupUI() {
setup_ui_called_ = true;
}
virtual bool AddTextGlyphs(const std::vector<TextGlyph>& glyphs, uint8_t bpp) { return false; }
virtual void ClearTextGlyphs() {}
virtual void SetEmojiCollection(std::shared_ptr<EmojiCollection>) {}
virtual void SetupUI() { setup_ui_called_ = true; }
inline int width() const { return width_; }
inline int height() const { return height_; }
@@ -60,27 +65,22 @@ protected:
virtual void Unlock() = 0;
};
class DisplayLockGuard {
public:
DisplayLockGuard(Display *display) : display_(display) {
DisplayLockGuard(Display* display) : display_(display) {
if (!display_->Lock(30000)) {
ESP_LOGE("Display", "Failed to lock display");
}
}
~DisplayLockGuard() {
display_->Unlock();
}
~DisplayLockGuard() { display_->Unlock(); }
private:
Display *display_;
Display* display_;
};
class NoDisplay : public Display {
private:
virtual bool Lock(int timeout_ms = 0) override {
return true;
}
virtual bool Lock(int timeout_ms = 0) override { return true; }
virtual void Unlock() override {}
};
main/display/lcd_display.cc
+227 -188
View File
@@ -1,18 +1,19 @@
#include "lcd_display.h"
#include "gif/lvgl_gif.h"
#include "settings.h"
#include "lvgl_theme.h"
#include "assets/lang_config.h"
#include "gif/lvgl_gif.h"
#include "lvgl_theme.h"
#include "settings.h"
#include <vector>
#include <algorithm>
#include <font_awesome.h>
#include <esp_log.h>
#include <esp_err.h>
#include <esp_log.h>
#include <esp_lvgl_port.h>
#include <esp_psram.h>
#include <cstring>
#include <material_symbols.h>
#include <noto_emoji.h>
#include <src/misc/cache/lv_cache.h>
#include <algorithm>
#include <cstring>
#include <vector>
#include "board.h"
@@ -20,12 +21,14 @@
LV_FONT_DECLARE(BUILTIN_TEXT_FONT);
LV_FONT_DECLARE(BUILTIN_ICON_FONT);
LV_FONT_DECLARE(font_awesome_30_4);
LV_FONT_DECLARE(font_material_symbols_30_4);
LV_FONT_DECLARE(font_noto_emoji_30_4);
void LcdDisplay::InitializeLcdThemes() {
auto text_font = std::make_shared<LvglBuiltInFont>(&BUILTIN_TEXT_FONT);
auto icon_font = std::make_shared<LvglBuiltInFont>(&BUILTIN_ICON_FONT);
auto large_icon_font = std::make_shared<LvglBuiltInFont>(&font_awesome_30_4);
auto large_icon_font = std::make_shared<LvglBuiltInFont>(&font_material_symbols_30_4);
auto emoji_font = std::make_shared<LvglBuiltInFont>(&font_noto_emoji_30_4);
// light theme
auto light_theme = new LvglTheme("light");
@@ -41,6 +44,7 @@ void LcdDisplay::InitializeLcdThemes() {
light_theme->set_text_font(text_font);
light_theme->set_icon_font(icon_font);
light_theme->set_large_icon_font(large_icon_font);
light_theme->set_emoji_font(emoji_font);
// dark theme
auto dark_theme = new LvglTheme("dark");
@@ -56,13 +60,15 @@ void LcdDisplay::InitializeLcdThemes() {
dark_theme->set_text_font(text_font);
dark_theme->set_icon_font(icon_font);
dark_theme->set_large_icon_font(large_icon_font);
dark_theme->set_emoji_font(emoji_font);
auto& theme_manager = LvglThemeManager::GetInstance();
theme_manager.RegisterTheme("light", light_theme);
theme_manager.RegisterTheme("dark", dark_theme);
}
LcdDisplay::LcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel, int width, int height)
LcdDisplay::LcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel, int width,
int height)
: panel_io_(panel_io), panel_(panel) {
width_ = width;
height_ = height;
@@ -77,10 +83,11 @@ LcdDisplay::LcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_
// Create a timer to hide the preview image
esp_timer_create_args_t preview_timer_args = {
.callback = [](void* arg) {
LcdDisplay* display = static_cast<LcdDisplay*>(arg);
display->SetPreviewImage(nullptr);
},
.callback =
[](void* arg) {
LcdDisplay* display = static_cast<LcdDisplay*>(arg);
display->SetPreviewImage(nullptr);
},
.arg = this,
.dispatch_method = ESP_TIMER_TASK,
.name = "preview_timer",
@@ -90,9 +97,9 @@ LcdDisplay::LcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_
}
SpiLcdDisplay::SpiLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel,
int width, int height, int offset_x, int offset_y, bool mirror_x, bool mirror_y, bool swap_xy)
int width, int height, int offset_x, int offset_y, bool mirror_x,
bool mirror_y, bool swap_xy)
: LcdDisplay(panel_io, panel, width, height) {
// draw white
std::vector<uint16_t> buffer(width_, 0xFFFF);
for (int y = 0; y < height_; y++) {
@@ -144,20 +151,22 @@ SpiLcdDisplay::SpiLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_h
.hres = static_cast<uint32_t>(width_),
.vres = static_cast<uint32_t>(height_),
.monochrome = false,
.rotation = {
.swap_xy = swap_xy,
.mirror_x = mirror_x,
.mirror_y = mirror_y,
},
.rotation =
{
.swap_xy = swap_xy,
.mirror_x = mirror_x,
.mirror_y = mirror_y,
},
.color_format = LV_COLOR_FORMAT_RGB565,
.flags = {
.buff_dma = 1,
.buff_spiram = 0,
.sw_rotate = 0,
.swap_bytes = 1,
.full_refresh = 0,
.direct_mode = 0,
},
.flags =
{
.buff_dma = 1,
.buff_spiram = 0,
.sw_rotate = 0,
.swap_bytes = 1,
.full_refresh = 0,
.direct_mode = 0,
},
};
display_ = lvgl_port_add_disp(&display_cfg);
@@ -171,13 +180,11 @@ SpiLcdDisplay::SpiLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_h
}
}
// RGB LCD implementation
RgbLcdDisplay::RgbLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel,
int width, int height, int offset_x, int offset_y,
bool mirror_x, bool mirror_y, bool swap_xy)
int width, int height, int offset_x, int offset_y, bool mirror_x,
bool mirror_y, bool swap_xy)
: LcdDisplay(panel_io, panel, width, height) {
// draw white
std::vector<uint16_t> buffer(width_, 0xFFFF);
for (int y = 0; y < height_; y++) {
@@ -201,42 +208,41 @@ RgbLcdDisplay::RgbLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_h
.double_buffer = true,
.hres = static_cast<uint32_t>(width_),
.vres = static_cast<uint32_t>(height_),
.rotation = {
.swap_xy = swap_xy,
.mirror_x = mirror_x,
.mirror_y = mirror_y,
},
.flags = {
.buff_dma = 1,
.swap_bytes = 0,
.full_refresh = 1,
.direct_mode = 1,
},
.rotation =
{
.swap_xy = swap_xy,
.mirror_x = mirror_x,
.mirror_y = mirror_y,
},
.flags =
{
.buff_dma = 1,
.swap_bytes = 0,
.full_refresh = 1,
.direct_mode = 1,
},
};
const lvgl_port_display_rgb_cfg_t rgb_cfg = {
.flags = {
.bb_mode = true,
.avoid_tearing = true,
}
};
const lvgl_port_display_rgb_cfg_t rgb_cfg = {.flags = {
.bb_mode = true,
.avoid_tearing = true,
}};
display_ = lvgl_port_add_disp_rgb(&display_cfg, &rgb_cfg);
if (display_ == nullptr) {
ESP_LOGE(TAG, "Failed to add RGB display");
return;
}
if (offset_x != 0 || offset_y != 0) {
lv_display_set_offset(display_, offset_x, offset_y);
}
}
MipiLcdDisplay::MipiLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel,
int width, int height, int offset_x, int offset_y,
bool mirror_x, bool mirror_y, bool swap_xy)
int width, int height, int offset_x, int offset_y, bool mirror_x,
bool mirror_y, bool swap_xy)
: LcdDisplay(panel_io, panel, width, height) {
ESP_LOGI(TAG, "Initialize LVGL library");
lv_init();
@@ -255,23 +261,23 @@ MipiLcdDisplay::MipiLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel
.vres = static_cast<uint32_t>(height_),
.monochrome = false,
/* Rotation values must be same as used in esp_lcd for initial settings of the screen */
.rotation = {
.swap_xy = swap_xy,
.mirror_x = mirror_x,
.mirror_y = mirror_y,
},
.flags = {
.buff_dma = true,
.buff_spiram =false,
.sw_rotate = true,
},
.rotation =
{
.swap_xy = swap_xy,
.mirror_x = mirror_x,
.mirror_y = mirror_y,
},
.flags =
{
.buff_dma = true,
.buff_spiram = false,
.sw_rotate = true,
},
};
const lvgl_port_display_dsi_cfg_t dpi_cfg = {
.flags = {
.avoid_tearing = false,
}
};
const lvgl_port_display_dsi_cfg_t dpi_cfg = {.flags = {
.avoid_tearing = false,
}};
display_ = lvgl_port_add_disp_dsi(&disp_cfg, &dpi_cfg);
if (display_ == nullptr) {
ESP_LOGE(TAG, "Failed to add display");
@@ -285,13 +291,13 @@ MipiLcdDisplay::MipiLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel
LcdDisplay::~LcdDisplay() {
SetPreviewImage(nullptr);
// Clean up GIF controller
if (gif_controller_) {
gif_controller_->Stop();
gif_controller_.reset();
}
if (preview_timer_ != nullptr) {
esp_timer_stop(preview_timer_);
esp_timer_delete(preview_timer_);
@@ -342,13 +348,9 @@ LcdDisplay::~LcdDisplay() {
}
}
bool LcdDisplay::Lock(int timeout_ms) {
return lvgl_port_lock(timeout_ms);
}
bool LcdDisplay::Lock(int timeout_ms) { return lvgl_port_lock(timeout_ms); }
void LcdDisplay::Unlock() {
lvgl_port_unlock();
}
void LcdDisplay::Unlock() { lvgl_port_unlock(); }
#if CONFIG_USE_WECHAT_MESSAGE_STYLE
void LcdDisplay::SetupUI() {
@@ -357,7 +359,7 @@ void LcdDisplay::SetupUI() {
ESP_LOGW(TAG, "SetupUI() called multiple times, skipping duplicate call");
return;
}
Display::SetupUI(); // Mark SetupUI as called
DisplayLockGuard lock(this);
@@ -395,7 +397,8 @@ void LcdDisplay::SetupUI() {
lv_obj_set_style_pad_left(top_bar_, lvgl_theme->spacing(4), 0);
lv_obj_set_style_pad_right(top_bar_, lvgl_theme->spacing(4), 0);
lv_obj_set_flex_flow(top_bar_, LV_FLEX_FLOW_ROW);
lv_obj_set_flex_align(top_bar_, LV_FLEX_ALIGN_SPACE_BETWEEN, LV_FLEX_ALIGN_CENTER, LV_FLEX_ALIGN_CENTER);
lv_obj_set_flex_align(top_bar_, LV_FLEX_ALIGN_SPACE_BETWEEN, LV_FLEX_ALIGN_CENTER,
LV_FLEX_ALIGN_CENTER);
lv_obj_set_scrollbar_mode(top_bar_, LV_SCROLLBAR_MODE_OFF);
// Left icon
@@ -411,7 +414,8 @@ void LcdDisplay::SetupUI() {
lv_obj_set_style_border_width(right_icons, 0, 0);
lv_obj_set_style_pad_all(right_icons, 0, 0);
lv_obj_set_flex_flow(right_icons, LV_FLEX_FLOW_ROW);
lv_obj_set_flex_align(right_icons, LV_FLEX_ALIGN_END, LV_FLEX_ALIGN_CENTER, LV_FLEX_ALIGN_CENTER);
lv_obj_set_flex_align(right_icons, LV_FLEX_ALIGN_END, LV_FLEX_ALIGN_CENTER,
LV_FLEX_ALIGN_CENTER);
mute_label_ = lv_label_create(right_icons);
lv_label_set_text(mute_label_, "");
@@ -435,7 +439,7 @@ void LcdDisplay::SetupUI() {
lv_obj_set_style_pad_bottom(status_bar_, lvgl_theme->spacing(2), 0);
lv_obj_set_scrollbar_mode(status_bar_, LV_SCROLLBAR_MODE_OFF);
lv_obj_set_style_layout(status_bar_, LV_LAYOUT_NONE, 0); // Use absolute positioning
lv_obj_align(status_bar_, LV_ALIGN_TOP_MID, 0, 0); // Overlap with top_bar_
lv_obj_align(status_bar_, LV_ALIGN_TOP_MID, 0, 0); // Overlap with top_bar_
notification_label_ = lv_label_create(status_bar_);
lv_obj_set_width(notification_label_, LV_HOR_RES * 0.8);
@@ -452,7 +456,7 @@ void LcdDisplay::SetupUI() {
lv_obj_set_style_text_color(status_label_, lvgl_theme->text_color(), 0);
lv_label_set_text(status_label_, Lang::Strings::INITIALIZING);
lv_obj_align(status_label_, LV_ALIGN_CENTER, 0, 0);
/* Content - Chat area */
content_ = lv_obj_create(container_);
lv_obj_set_style_radius(content_, 0, 0);
@@ -460,16 +464,17 @@ void LcdDisplay::SetupUI() {
lv_obj_set_flex_grow(content_, 1);
lv_obj_set_style_pad_all(content_, lvgl_theme->spacing(4), 0);
lv_obj_set_style_border_width(content_, 0, 0);
lv_obj_set_style_bg_color(content_, lvgl_theme->chat_background_color(), 0); // Background for chat area
lv_obj_set_style_bg_color(content_, lvgl_theme->chat_background_color(),
0); // Background for chat area
// Enable scrolling for chat content
lv_obj_set_scrollbar_mode(content_, LV_SCROLLBAR_MODE_OFF);
lv_obj_set_scroll_dir(content_, LV_DIR_VER);
// Create a flex container for chat messages
lv_obj_set_flex_flow(content_, LV_FLEX_FLOW_COLUMN);
lv_obj_set_flex_align(content_, LV_FLEX_ALIGN_START, LV_FLEX_ALIGN_START, LV_FLEX_ALIGN_START);
lv_obj_set_style_pad_row(content_, lvgl_theme->spacing(4), 0); // Space between messages
lv_obj_set_style_pad_row(content_, lvgl_theme->spacing(4), 0); // Space between messages
// We'll create chat messages dynamically in SetChatMessage
chat_message_label_ = nullptr;
@@ -487,32 +492,37 @@ void LcdDisplay::SetupUI() {
lv_obj_add_flag(low_battery_popup_, LV_OBJ_FLAG_HIDDEN);
emoji_image_ = lv_img_create(screen);
lv_obj_align(emoji_image_, LV_ALIGN_TOP_MID, 0, text_font->line_height + lvgl_theme->spacing(8));
lv_obj_align(emoji_image_, LV_ALIGN_TOP_MID, 0,
text_font->line_height + lvgl_theme->spacing(8));
// Display AI logo while booting
emoji_label_ = lv_label_create(screen);
lv_obj_center(emoji_label_);
lv_obj_set_style_text_font(emoji_label_, large_icon_font, 0);
lv_obj_set_style_text_color(emoji_label_, lvgl_theme->text_color(), 0);
lv_label_set_text(emoji_label_, FONT_AWESOME_MICROCHIP_AI);
lv_label_set_text(emoji_label_, MATERIAL_SYMBOLS_ROBOT_2);
}
#if CONFIG_IDF_TARGET_ESP32P4
#define MAX_MESSAGES 40
#define MAX_MESSAGES 40
#else
#define MAX_MESSAGES 20
#define MAX_MESSAGES 20
#endif
void LcdDisplay::SetChatMessage(const char* role, const char* content) {
if (!setup_ui_called_) {
ESP_LOGW(TAG, "SetChatMessage('%s', '%s') called before SetupUI() - message will be lost!", role, content);
ESP_LOGW(TAG, "SetChatMessage('%s', '%s') called before SetupUI() - message will be lost!",
role, content);
}
DisplayLockGuard lock(this);
if (content_ == nullptr) {
if (setup_ui_called_) {
ESP_LOGW(TAG, "SetChatMessage('%s', '%s') failed: content_ is nullptr (SetupUI() was called but container not created)", role, content);
ESP_LOGW(TAG,
"SetChatMessage('%s', '%s') failed: content_ is nullptr (SetupUI() was called "
"but container not created)",
role, content);
}
return;
}
// Check if message count exceeds limit
uint32_t child_count = lv_obj_get_child_cnt(content_);
if (child_count >= MAX_MESSAGES) {
@@ -531,21 +541,24 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
}
}
}
// Collapse system messages (if it's a system message, check if the last message is also a system message)
// Collapse system messages (if it's a system message, check if the last message is also a
// system message)
if (strcmp(role, "system") == 0) {
// Refresh child count to get accurate count after potential deletion above
child_count = lv_obj_get_child_cnt(content_);
if (child_count > 0) {
// Get the last message container
lv_obj_t* last_container = lv_obj_get_child(content_, child_count - 1);
if (last_container != nullptr && lv_obj_is_valid(last_container) && lv_obj_get_child_cnt(last_container) > 0) {
if (last_container != nullptr && lv_obj_is_valid(last_container) &&
lv_obj_get_child_cnt(last_container) > 0) {
// Get the bubble inside the container
lv_obj_t* last_bubble = lv_obj_get_child(last_container, 0);
if (last_bubble != nullptr && lv_obj_is_valid(last_bubble)) {
// Check if bubble type is system message
void* bubble_type_ptr = lv_obj_get_user_data(last_bubble);
if (bubble_type_ptr != nullptr && strcmp((const char*)bubble_type_ptr, "system") == 0) {
if (bubble_type_ptr != nullptr &&
strcmp((const char*)bubble_type_ptr, "system") == 0) {
// If the last message is also a system message, delete it
lv_obj_del(last_container);
}
@@ -558,7 +571,7 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
}
// Avoid empty message boxes
if(strlen(content) == 0) {
if (strlen(content) == 0) {
return;
}
@@ -574,16 +587,16 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
// Create the message text
lv_obj_t* msg_text = lv_label_create(msg_bubble);
lv_label_set_text(msg_text, content);
// Calculate bubble width constraints
lv_coord_t max_width = LV_HOR_RES * 85 / 100 - 16; // 85% of screen width
lv_coord_t min_width = 20;
lv_coord_t min_width = 20;
// Let LVGL calculate the natural text width first
lv_obj_set_width(msg_text, LV_SIZE_CONTENT);
lv_obj_update_layout(msg_text);
lv_coord_t text_width = lv_obj_get_width(msg_text);
// Ensure text width is not less than minimum width
if (text_width < min_width) {
text_width = min_width;
@@ -591,7 +604,7 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
// Constrain to max width
lv_coord_t bubble_width = (text_width < max_width) ? text_width : max_width;
// Set message text width
lv_obj_set_width(msg_text, bubble_width);
lv_label_set_long_mode(msg_text, LV_LABEL_LONG_WRAP);
@@ -607,14 +620,14 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
lv_obj_set_style_bg_opa(msg_bubble, LV_OPA_70, 0);
// Set text color for contrast
lv_obj_set_style_text_color(msg_text, lvgl_theme->text_color(), 0);
// Set custom attribute to mark bubble type
lv_obj_set_user_data(msg_bubble, (void*)"user");
// Set appropriate width for content
lv_obj_set_width(msg_bubble, LV_SIZE_CONTENT);
lv_obj_set_height(msg_bubble, LV_SIZE_CONTENT);
// Don't grow
lv_obj_set_style_flex_grow(msg_bubble, 0, 0);
} else if (strcmp(role, "assistant") == 0) {
@@ -623,14 +636,14 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
lv_obj_set_style_bg_opa(msg_bubble, LV_OPA_70, 0);
// Set text color for contrast
lv_obj_set_style_text_color(msg_text, lvgl_theme->text_color(), 0);
// Set custom attribute to mark bubble type
lv_obj_set_user_data(msg_bubble, (void*)"assistant");
// Set appropriate width for content
lv_obj_set_width(msg_bubble, LV_SIZE_CONTENT);
lv_obj_set_height(msg_bubble, LV_SIZE_CONTENT);
// Don't grow
lv_obj_set_style_flex_grow(msg_bubble, 0, 0);
} else if (strcmp(role, "system") == 0) {
@@ -639,36 +652,36 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
lv_obj_set_style_bg_opa(msg_bubble, LV_OPA_70, 0);
// Set text color for contrast
lv_obj_set_style_text_color(msg_text, lvgl_theme->system_text_color(), 0);
// Set custom attribute to mark bubble type
lv_obj_set_user_data(msg_bubble, (void*)"system");
// Set appropriate width for content
lv_obj_set_width(msg_bubble, LV_SIZE_CONTENT);
lv_obj_set_height(msg_bubble, LV_SIZE_CONTENT);
// Don't grow
lv_obj_set_style_flex_grow(msg_bubble, 0, 0);
}
// Create a full-width container for user messages to ensure right alignment
if (strcmp(role, "user") == 0) {
// Create a full-width container
lv_obj_t* container = lv_obj_create(content_);
lv_obj_set_width(container, LV_HOR_RES);
lv_obj_set_height(container, LV_SIZE_CONTENT);
// Make container transparent and borderless
lv_obj_set_style_bg_opa(container, LV_OPA_TRANSP, 0);
lv_obj_set_style_border_width(container, 0, 0);
lv_obj_set_style_pad_all(container, 0, 0);
// Move the message bubble into this container
lv_obj_set_parent(msg_bubble, container);
// Right align the bubble in the container
lv_obj_align(msg_bubble, LV_ALIGN_RIGHT_MID, -25, 0);
// Auto-scroll to this container
lv_obj_scroll_to_view_recursive(container, LV_ANIM_ON);
} else if (strcmp(role, "system") == 0) {
@@ -676,11 +689,11 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
lv_obj_t* container = lv_obj_create(content_);
lv_obj_set_width(container, LV_HOR_RES);
lv_obj_set_height(container, LV_SIZE_CONTENT);
lv_obj_set_style_bg_opa(container, LV_OPA_TRANSP, 0);
lv_obj_set_style_border_width(container, 0, 0);
lv_obj_set_style_pad_all(container, 0, 0);
lv_obj_set_parent(msg_bubble, container);
lv_obj_align(msg_bubble, LV_ALIGN_CENTER, 0, 0);
lv_obj_scroll_to_view_recursive(container, LV_ANIM_ON);
@@ -692,7 +705,7 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
// Auto-scroll to the message bubble
lv_obj_scroll_to_view_recursive(msg_bubble, LV_ANIM_ON);
}
// Store reference to the latest message label
chat_message_label_ = msg_text;
}
@@ -706,7 +719,7 @@ void LcdDisplay::SetPreviewImage(std::unique_ptr<LvglImage> image) {
if (image == nullptr) {
return;
}
auto lvgl_theme = static_cast<LvglTheme*>(current_theme_);
// Create a message bubble for image preview
lv_obj_t* img_bubble = lv_obj_create(content_);
@@ -714,21 +727,21 @@ void LcdDisplay::SetPreviewImage(std::unique_ptr<LvglImage> image) {
lv_obj_set_scrollbar_mode(img_bubble, LV_SCROLLBAR_MODE_OFF);
lv_obj_set_style_border_width(img_bubble, 0, 0);
lv_obj_set_style_pad_all(img_bubble, lvgl_theme->spacing(4), 0);
// Set image bubble background color (similar to system message)
lv_obj_set_style_bg_color(img_bubble, lvgl_theme->assistant_bubble_color(), 0);
lv_obj_set_style_bg_opa(img_bubble, LV_OPA_70, 0);
// Set custom attribute to mark bubble type
lv_obj_set_user_data(img_bubble, (void*)"image");
// Create the image object inside the bubble
lv_obj_t* preview_image = lv_image_create(img_bubble);
// Calculate appropriate size for the image
lv_coord_t max_width = LV_HOR_RES * 70 / 100; // 70% of screen width
lv_coord_t max_height = LV_VER_RES * 50 / 100; // 50% of screen height
lv_coord_t max_width = LV_HOR_RES * 70 / 100; // 70% of screen width
lv_coord_t max_height = LV_VER_RES * 50 / 100; // 50% of screen height
// Calculate zoom factor to fit within maximum dimensions
auto img_dsc = image->image_dsc();
lv_coord_t img_width = img_dsc->header.w;
@@ -736,44 +749,49 @@ void LcdDisplay::SetPreviewImage(std::unique_ptr<LvglImage> image) {
if (img_width == 0 || img_height == 0) {
img_width = max_width;
img_height = max_height;
ESP_LOGW(TAG, "Invalid image dimensions: %ld x %ld, using default dimensions: %ld x %ld", img_width, img_height, max_width, max_height);
ESP_LOGW(TAG, "Invalid image dimensions: %ld x %ld, using default dimensions: %ld x %ld",
img_width, img_height, max_width, max_height);
}
lv_coord_t zoom_w = (max_width * 256) / img_width;
lv_coord_t zoom_h = (max_height * 256) / img_height;
lv_coord_t zoom = (zoom_w < zoom_h) ? zoom_w : zoom_h;
// Ensure zoom doesn't exceed 256 (100%)
if (zoom > 256) zoom = 256;
if (zoom > 256)
zoom = 256;
// Set image properties
lv_image_set_src(preview_image, img_dsc);
lv_image_set_scale(preview_image, zoom);
// Add event handler to clean up LvglImage when image is deleted
// We need to transfer ownership of the unique_ptr to the event callback
LvglImage* raw_image = image.release(); // Release ownership of smart pointer
lv_obj_add_event_cb(preview_image, [](lv_event_t* e) {
LvglImage* img = (LvglImage*)lv_event_get_user_data(e);
if (img != nullptr) {
delete img; // Properly release memory by deleting LvglImage object
}
}, LV_EVENT_DELETE, (void*)raw_image);
LvglImage* raw_image = image.release(); // Release ownership of smart pointer
lv_obj_add_event_cb(
preview_image,
[](lv_event_t* e) {
LvglImage* img = (LvglImage*)lv_event_get_user_data(e);
if (img != nullptr) {
delete img; // Properly release memory by deleting LvglImage object
}
},
LV_EVENT_DELETE, (void*)raw_image);
// Calculate actual scaled image dimensions
lv_coord_t scaled_width = (img_width * zoom) / 256;
lv_coord_t scaled_height = (img_height * zoom) / 256;
// Set bubble size to be 16 pixels larger than the image (8 pixels on each side)
lv_obj_set_width(img_bubble, scaled_width + 16);
lv_obj_set_height(img_bubble, scaled_height + 16);
// Don't grow in flex layout
lv_obj_set_style_flex_grow(img_bubble, 0, 0);
// Center the image within the bubble
lv_obj_center(preview_image);
// Left align the image bubble like assistant messages
lv_obj_align(img_bubble, LV_ALIGN_LEFT_MID, 0, 0);
@@ -786,18 +804,18 @@ void LcdDisplay::ClearChatMessages() {
if (content_ == nullptr) {
return;
}
// Use lv_obj_clean to delete all children of content_ (chat message bubbles)
lv_obj_clean(content_);
// Reset chat_message_label_ as it has been deleted
chat_message_label_ = nullptr;
// Show the centered AI logo (emoji_label_) again
if (emoji_label_ != nullptr) {
lv_obj_remove_flag(emoji_label_, LV_OBJ_FLAG_HIDDEN);
}
ESP_LOGI(TAG, "Chat messages cleared");
}
#else
@@ -807,7 +825,7 @@ void LcdDisplay::SetupUI() {
ESP_LOGW(TAG, "SetupUI() called multiple times, skipping duplicate call");
return;
}
Display::SetupUI(); // Mark SetupUI as called
DisplayLockGuard lock(this);
LvglTheme* lvgl_theme = static_cast<LvglTheme*>(current_theme_);
@@ -840,7 +858,7 @@ void LcdDisplay::SetupUI() {
emoji_label_ = lv_label_create(emoji_box_);
lv_obj_set_style_text_font(emoji_label_, large_icon_font, 0);
lv_obj_set_style_text_color(emoji_label_, lvgl_theme->text_color(), 0);
lv_label_set_text(emoji_label_, FONT_AWESOME_MICROCHIP_AI);
lv_label_set_text(emoji_label_, MATERIAL_SYMBOLS_ROBOT_2);
emoji_image_ = lv_img_create(emoji_box_);
lv_obj_center(emoji_image_);
@@ -865,7 +883,8 @@ void LcdDisplay::SetupUI() {
lv_obj_set_style_pad_left(top_bar_, lvgl_theme->spacing(4), 0);
lv_obj_set_style_pad_right(top_bar_, lvgl_theme->spacing(4), 0);
lv_obj_set_flex_flow(top_bar_, LV_FLEX_FLOW_ROW);
lv_obj_set_flex_align(top_bar_, LV_FLEX_ALIGN_SPACE_BETWEEN, LV_FLEX_ALIGN_CENTER, LV_FLEX_ALIGN_CENTER);
lv_obj_set_flex_align(top_bar_, LV_FLEX_ALIGN_SPACE_BETWEEN, LV_FLEX_ALIGN_CENTER,
LV_FLEX_ALIGN_CENTER);
lv_obj_set_scrollbar_mode(top_bar_, LV_SCROLLBAR_MODE_OFF);
lv_obj_align(top_bar_, LV_ALIGN_TOP_MID, 0, 0);
@@ -882,7 +901,8 @@ void LcdDisplay::SetupUI() {
lv_obj_set_style_border_width(right_icons, 0, 0);
lv_obj_set_style_pad_all(right_icons, 0, 0);
lv_obj_set_flex_flow(right_icons, LV_FLEX_FLOW_ROW);
lv_obj_set_flex_align(right_icons, LV_FLEX_ALIGN_END, LV_FLEX_ALIGN_CENTER, LV_FLEX_ALIGN_CENTER);
lv_obj_set_flex_align(right_icons, LV_FLEX_ALIGN_END, LV_FLEX_ALIGN_CENTER,
LV_FLEX_ALIGN_CENTER);
mute_label_ = lv_label_create(right_icons);
lv_label_set_text(mute_label_, "");
@@ -906,7 +926,7 @@ void LcdDisplay::SetupUI() {
lv_obj_set_style_pad_bottom(status_bar_, lvgl_theme->spacing(2), 0);
lv_obj_set_scrollbar_mode(status_bar_, LV_SCROLLBAR_MODE_OFF);
lv_obj_set_style_layout(status_bar_, LV_LAYOUT_NONE, 0); // Use absolute positioning
lv_obj_align(status_bar_, LV_ALIGN_TOP_MID, 0, 0); // Overlap with top_bar_
lv_obj_align(status_bar_, LV_ALIGN_TOP_MID, 0, 0); // Overlap with top_bar_
notification_label_ = lv_label_create(status_bar_);
lv_obj_set_width(notification_label_, LV_HOR_RES * 0.75);
@@ -976,7 +996,8 @@ void LcdDisplay::SetupUI() {
lv_anim_set_delay(&a, 1000);
lv_anim_set_repeat_count(&a, LV_ANIM_REPEAT_INFINITE);
lv_obj_set_style_anim(chat_message_label_, &a, LV_PART_MAIN);
lv_obj_set_style_anim_duration(chat_message_label_, lv_anim_speed_clamped(60, 300, 60000), LV_PART_MAIN);
lv_obj_set_style_anim_duration(chat_message_label_, lv_anim_speed_clamped(60, 300, 60000),
LV_PART_MAIN);
lv_obj_add_flag(bottom_bar_, LV_OBJ_FLAG_HIDDEN); // Hide until there is content
#endif
@@ -986,7 +1007,7 @@ void LcdDisplay::SetupUI() {
lv_obj_align(low_battery_popup_, LV_ALIGN_BOTTOM_MID, 0, -lvgl_theme->spacing(4));
lv_obj_set_style_bg_color(low_battery_popup_, lvgl_theme->low_battery_color(), 0);
lv_obj_set_style_radius(low_battery_popup_, lvgl_theme->spacing(4), 0);
low_battery_label_ = lv_label_create(low_battery_popup_);
lv_label_set_text(low_battery_label_, Lang::Strings::BATTERY_NEED_CHARGE);
lv_obj_set_style_text_color(low_battery_label_, lv_color_white(), 0);
@@ -1032,15 +1053,20 @@ void LcdDisplay::SetPreviewImage(std::unique_ptr<LvglImage> image) {
void LcdDisplay::SetChatMessage(const char* role, const char* content) {
if (!setup_ui_called_) {
ESP_LOGW(TAG, "SetChatMessage('%s', '%s') called before SetupUI() - message will be lost!", role, content);
ESP_LOGW(TAG, "SetChatMessage('%s', '%s') called before SetupUI() - message will be lost!",
role, content);
}
DisplayLockGuard lock(this);
if (chat_message_label_ == nullptr) {
if (setup_ui_called_) {
ESP_LOGW(TAG, "SetChatMessage('%s', '%s') failed: chat_message_label_ is nullptr (SetupUI() was called but label not created)", role, content);
ESP_LOGW(TAG,
"SetChatMessage('%s', '%s') failed: chat_message_label_ is nullptr (SetupUI() "
"was called but label not created)",
role, content);
}
return;
}
lv_anim_delete(chat_message_label_, nullptr);
lv_label_set_text(chat_message_label_, content);
// Show bottom_bar_ only when there is content (and subtitle is not globally hidden)
if (bottom_bar_ != nullptr) {
@@ -1073,11 +1099,15 @@ void LcdDisplay::ClearChatMessages() {
void LcdDisplay::SetEmotion(const char* emotion) {
if (!setup_ui_called_) {
ESP_LOGW(TAG, "SetEmotion('%s') called before SetupUI() - emotion will not be displayed!", emotion);
ESP_LOGW(TAG, "SetEmotion('%s') called before SetupUI() - emotion will not be displayed!",
emotion);
}
if (emoji_image_ == nullptr) {
if (setup_ui_called_) {
ESP_LOGW(TAG, "SetEmotion('%s') failed: emoji_image_ is nullptr (SetupUI() was called but emoji image not created)", emotion);
ESP_LOGW(TAG,
"SetEmotion('%s') failed: emoji_image_ is nullptr (SetupUI() was called but "
"emoji image not created)",
emotion);
}
return;
}
@@ -1085,13 +1115,20 @@ void LcdDisplay::SetEmotion(const char* emotion) {
auto emoji_collection = static_cast<LvglTheme*>(current_theme_)->emoji_collection();
auto image = emoji_collection != nullptr ? emoji_collection->GetEmojiImage(emotion) : nullptr;
if (image == nullptr) {
const char* utf8 = font_awesome_get_utf8(emotion);
auto lvgl_theme = static_cast<LvglTheme*>(current_theme_);
const char* utf8 = noto_emoji_get_utf8(emotion);
const lv_font_t* emotion_font = lvgl_theme->emoji_font()->font();
if (utf8 == nullptr) {
utf8 = material_symbols_get_utf8(emotion);
emotion_font = lvgl_theme->large_icon_font()->font();
}
if (utf8 != nullptr && emoji_label_ != nullptr) {
DisplayLockGuard lock(this);
if (gif_controller_) {
gif_controller_->Stop();
gif_controller_.reset();
}
lv_obj_set_style_text_font(emoji_label_, emotion_font, 0);
lv_label_set_text(emoji_label_, utf8);
lv_obj_add_flag(emoji_image_, LV_OBJ_FLAG_HIDDEN);
lv_obj_remove_flag(emoji_label_, LV_OBJ_FLAG_HIDDEN);
@@ -1109,17 +1146,16 @@ void LcdDisplay::SetEmotion(const char* emotion) {
if (image->IsGif()) {
// Create new GIF controller
gif_controller_ = std::make_unique<LvglGif>(image->image_dsc());
if (gif_controller_->IsLoaded()) {
// Set up frame update callback
gif_controller_->SetFrameCallback([this]() {
lv_image_set_src(emoji_image_, gif_controller_->image_dsc());
});
gif_controller_->SetFrameCallback(
[this]() { lv_image_set_src(emoji_image_, gif_controller_->image_dsc()); });
// Set initial frame and start animation
lv_image_set_src(emoji_image_, gif_controller_->image_dsc());
gif_controller_->Start();
// Show GIF, hide others
lv_obj_add_flag(emoji_label_, LV_OBJ_FLAG_HIDDEN);
lv_obj_remove_flag(emoji_image_, LV_OBJ_FLAG_HIDDEN);
@@ -1142,7 +1178,7 @@ void LcdDisplay::SetEmotion(const char* emotion) {
gif_controller_->Stop();
gif_controller_.reset();
}
lv_obj_add_flag(emoji_image_, LV_OBJ_FLAG_HIDDEN);
lv_obj_add_flag(emoji_label_, LV_OBJ_FLAG_HIDDEN);
}
@@ -1151,9 +1187,9 @@ void LcdDisplay::SetEmotion(const char* emotion) {
void LcdDisplay::SetTheme(Theme* theme) {
DisplayLockGuard lock(this);
auto lvgl_theme = static_cast<LvglTheme*>(theme);
// Get the active screen
lv_obj_t* screen = lv_screen_active();
@@ -1183,13 +1219,13 @@ void LcdDisplay::SetTheme(Theme* theme) {
lv_obj_set_style_bg_image_src(container_, nullptr, 0);
lv_obj_set_style_bg_color(container_, lvgl_theme->background_color(), 0);
}
// Update top bar background color with 50% opacity
if (top_bar_ != nullptr) {
lv_obj_set_style_bg_opa(top_bar_, LV_OPA_50, 0);
lv_obj_set_style_bg_color(top_bar_, lvgl_theme->background_color(), 0);
}
// Update status bar elements
lv_obj_set_style_text_color(network_label_, lvgl_theme->text_color(), 0);
lv_obj_set_style_text_color(status_label_, lvgl_theme->text_color(), 0);
@@ -1207,10 +1243,11 @@ void LcdDisplay::SetTheme(Theme* theme) {
uint32_t child_count = lv_obj_get_child_cnt(content_);
for (uint32_t i = 0; i < child_count; i++) {
lv_obj_t* obj = lv_obj_get_child(content_, i);
if (obj == nullptr) continue;
if (obj == nullptr)
continue;
lv_obj_t* bubble = nullptr;
// Check if this object is a container or bubble
// If it's a container (user or system message), get its child as bubble
// If it's a bubble (assistant message), use it directly
@@ -1229,28 +1266,29 @@ void LcdDisplay::SetTheme(Theme* theme) {
// No child elements, might be other UI elements, skip
continue;
}
if (bubble == nullptr) continue;
if (bubble == nullptr)
continue;
// Use saved user data to identify bubble type
void* bubble_type_ptr = lv_obj_get_user_data(bubble);
if (bubble_type_ptr != nullptr) {
const char* bubble_type = static_cast<const char*>(bubble_type_ptr);
// Apply correct color based on bubble type
if (strcmp(bubble_type, "user") == 0) {
lv_obj_set_style_bg_color(bubble, lvgl_theme->user_bubble_color(), 0);
} else if (strcmp(bubble_type, "assistant") == 0) {
lv_obj_set_style_bg_color(bubble, lvgl_theme->assistant_bubble_color(), 0);
lv_obj_set_style_bg_color(bubble, lvgl_theme->assistant_bubble_color(), 0);
} else if (strcmp(bubble_type, "system") == 0) {
lv_obj_set_style_bg_color(bubble, lvgl_theme->system_bubble_color(), 0);
} else if (strcmp(bubble_type, "image") == 0) {
lv_obj_set_style_bg_color(bubble, lvgl_theme->system_bubble_color(), 0);
}
// Update border color
lv_obj_set_style_border_color(bubble, lvgl_theme->border_color(), 0);
// Update text color for the message
if (lv_obj_get_child_cnt(bubble) > 0) {
lv_obj_t* text = lv_obj_get_child(bubble, 0);
@@ -1272,18 +1310,18 @@ void LcdDisplay::SetTheme(Theme* theme) {
if (chat_message_label_ != nullptr) {
lv_obj_set_style_text_color(chat_message_label_, lvgl_theme->text_color(), 0);
}
if (emoji_label_ != nullptr) {
lv_obj_set_style_text_color(emoji_label_, lvgl_theme->text_color(), 0);
}
// Update bottom bar background color with 50% opacity
if (bottom_bar_ != nullptr) {
lv_obj_set_style_bg_opa(bottom_bar_, LV_OPA_50, 0);
lv_obj_set_style_bg_color(bottom_bar_, lvgl_theme->background_color(), 0);
}
#endif
// Update low battery popup
lv_obj_set_style_bg_color(low_battery_popup_, lvgl_theme->low_battery_color(), 0);
@@ -1294,14 +1332,15 @@ void LcdDisplay::SetTheme(Theme* theme) {
void LcdDisplay::SetHideSubtitle(bool hide) {
DisplayLockGuard lock(this);
hide_subtitle_ = hide;
// Immediately update UI visibility based on the setting
if (bottom_bar_ != nullptr) {
if (hide) {
lv_obj_add_flag(bottom_bar_, LV_OBJ_FLAG_HIDDEN);
} else {
// Only show if there is actual content to display
const char* text = (chat_message_label_ != nullptr) ? lv_label_get_text(chat_message_label_) : nullptr;
const char* text =
(chat_message_label_ != nullptr) ? lv_label_get_text(chat_message_label_) : nullptr;
if (text != nullptr && text[0] != '\0') {
lv_obj_remove_flag(bottom_bar_, LV_OBJ_FLAG_HIDDEN);
}
main/display/lcd_display.h
+16 -18
View File
@@ -1,24 +1,21 @@
#ifndef LCD_DISPLAY_H
#define LCD_DISPLAY_H
#include "lvgl_display.h"
#include "gif/lvgl_gif.h"
#include "lvgl_display.h"
#include <esp_lcd_panel_io.h>
#include <esp_lcd_panel_ops.h>
#include <font_emoji.h>
#include <atomic>
#include <memory>
#define PREVIEW_IMAGE_DURATION_MS 5000
class LcdDisplay : public LvglDisplay {
protected:
esp_lcd_panel_io_handle_t panel_io_ = nullptr;
esp_lcd_panel_handle_t panel_ = nullptr;
lv_draw_buf_t draw_buf_;
lv_obj_t* top_bar_ = nullptr;
lv_obj_t* status_bar_ = nullptr;
@@ -42,8 +39,9 @@ protected:
protected:
// Add protected constructor
LcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel, int width, int height);
LcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel, int width,
int height);
public:
~LcdDisplay();
virtual void SetEmotion(const char* emotion) override;
@@ -53,7 +51,7 @@ public:
virtual void SetupUI() override;
// Add theme switching function
virtual void SetTheme(Theme* theme) override;
// Set whether to hide chat messages/subtitles
void SetHideSubtitle(bool hide);
};
@@ -61,25 +59,25 @@ public:
// SPI LCD display
class SpiLcdDisplay : public LcdDisplay {
public:
SpiLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel,
int width, int height, int offset_x, int offset_y,
bool mirror_x, bool mirror_y, bool swap_xy);
SpiLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel, int width,
int height, int offset_x, int offset_y, bool mirror_x, bool mirror_y,
bool swap_xy);
};
// RGB LCD display
class RgbLcdDisplay : public LcdDisplay {
public:
RgbLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel,
int width, int height, int offset_x, int offset_y,
bool mirror_x, bool mirror_y, bool swap_xy);
RgbLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel, int width,
int height, int offset_x, int offset_y, bool mirror_x, bool mirror_y,
bool swap_xy);
};
// MIPI LCD display
class MipiLcdDisplay : public LcdDisplay {
public:
MipiLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel,
int width, int height, int offset_x, int offset_y,
bool mirror_x, bool mirror_y, bool swap_xy);
MipiLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel, int width,
int height, int offset_x, int offset_y, bool mirror_x, bool mirror_y,
bool swap_xy);
};
#endif // LCD_DISPLAY_H
#endif // LCD_DISPLAY_H
main/display/lvgl_display/dynamic_glyph_cache.cc
+181
View File
@@ -0,0 +1,181 @@
#include "dynamic_glyph_cache.h"
#include <esp_log.h>
#include <algorithm>
#include <iterator>
#define TAG "DynamicGlyphCache"
DynamicGlyphCache::DynamicGlyphCache() : retain_between_batches_(TextGlyphStorageUsesPsram()) {}
lv_font_t* DynamicGlyphCache::EnsureFont(const lv_font_t* base_font, uint8_t bpp) {
if (base_font == nullptr || (bpp != 1 && bpp != 4)) {
return nullptr;
}
bool needs_rebuild = !initialized_;
if (initialized_ && bpp_ != bpp) {
entries_.clear();
use_counter_ = 0;
needs_rebuild = true;
}
bpp_ = bpp;
font_.get_glyph_dsc = lv_font_get_glyph_dsc_fmt_txt;
font_.get_glyph_bitmap = lv_font_get_bitmap_fmt_txt;
font_.release_glyph = nullptr;
font_.line_height = base_font->line_height;
font_.base_line = base_font->base_line;
font_.subpx = LV_FONT_SUBPX_NONE;
font_.kerning = LV_FONT_KERNING_NONE;
font_.static_bitmap = 0;
font_.underline_position = base_font->underline_position;
font_.underline_thickness = base_font->underline_thickness;
font_.fallback = nullptr;
font_.user_data = nullptr;
font_.dsc = &dsc_;
initialized_ = true;
if (needs_rebuild) {
Rebuild();
}
return &font_;
}
size_t DynamicGlyphCache::BitmapBytes() const {
size_t total = 0;
for (const auto& entry : entries_) {
total += entry.bitmap.size();
}
return total;
}
bool DynamicGlyphCache::AddGlyphs(const std::vector<TextGlyph>& glyphs) {
if (!initialized_ || glyphs.empty()) {
return false;
}
bool changed = false;
if (!retain_between_batches_) {
changed = !entries_.empty();
entries_.clear();
use_counter_ = 0;
}
size_t bitmap_bytes = BitmapBytes();
for (const auto& glyph : glyphs) {
const size_t expected = (static_cast<size_t>(glyph.box_w) * glyph.box_h * bpp_ + 7) / 8;
if (glyph.codepoint == 0 || glyph.codepoint > 0x10FFFF || glyph.bitmap.size() != expected) {
ESP_LOGW(TAG, "Rejected glyph U+%04lX", static_cast<unsigned long>(glyph.codepoint));
continue;
}
auto existing = std::find_if(
entries_.begin(), entries_.end(),
[&glyph](const Entry& entry) { return entry.codepoint == glyph.codepoint; });
if (existing != entries_.end()) {
bitmap_bytes -= existing->bitmap.size();
} else {
entries_.emplace_back();
existing = std::prev(entries_.end());
}
existing->codepoint = glyph.codepoint;
existing->adv_w = glyph.adv_w;
existing->box_w = glyph.box_w;
existing->box_h = glyph.box_h;
existing->ofs_x = glyph.ofs_x;
existing->ofs_y = glyph.ofs_y;
existing->bitmap.assign(glyph.bitmap.begin(), glyph.bitmap.end());
existing->last_use = ++use_counter_;
bitmap_bytes += glyph.bitmap.size();
changed = true;
}
while (entries_.size() > kMaxGlyphs || bitmap_bytes > kMaxBitmapBytes) {
auto oldest = std::min_element(
entries_.begin(), entries_.end(),
[](const Entry& a, const Entry& b) { return a.last_use < b.last_use; });
if (oldest == entries_.end()) {
break;
}
bitmap_bytes -= oldest->bitmap.size();
entries_.erase(oldest);
}
if (changed) {
Rebuild();
}
return changed;
}
void DynamicGlyphCache::Clear() {
if (retain_between_batches_) {
entries_.clear();
} else {
TextGlyphVector<Entry>().swap(entries_);
TextGlyphVector<uint8_t>().swap(bitmap_blob_);
TextGlyphVector<uint16_t>().swap(unicode_list_);
TextGlyphVector<lv_font_fmt_txt_glyph_dsc_t>().swap(glyph_dsc_);
TextGlyphVector<lv_font_fmt_txt_cmap_t>().swap(cmaps_);
}
use_counter_ = 0;
Rebuild();
}
void DynamicGlyphCache::Rebuild() {
std::sort(entries_.begin(), entries_.end(),
[](const Entry& a, const Entry& b) { return a.codepoint < b.codepoint; });
bitmap_blob_.clear();
unicode_list_.clear();
glyph_dsc_.clear();
cmaps_.clear();
glyph_dsc_.push_back(lv_font_fmt_txt_glyph_dsc_t{});
struct Range {
uint32_t start;
uint32_t list_begin;
uint32_t last;
uint16_t count;
};
TextGlyphVector<Range> ranges;
for (const auto& entry : entries_) {
if (ranges.empty() || entry.codepoint - ranges.back().start > 0xFFFE) {
ranges.push_back(Range{entry.codepoint, static_cast<uint32_t>(unicode_list_.size()),
entry.codepoint, 0});
}
auto& range = ranges.back();
unicode_list_.push_back(static_cast<uint16_t>(entry.codepoint - range.start));
range.last = entry.codepoint;
++range.count;
lv_font_fmt_txt_glyph_dsc_t glyph_dsc{};
glyph_dsc.bitmap_index = bitmap_blob_.size();
glyph_dsc.adv_w = entry.adv_w;
glyph_dsc.box_w = entry.box_w;
glyph_dsc.box_h = entry.box_h;
glyph_dsc.ofs_x = entry.ofs_x;
glyph_dsc.ofs_y = entry.ofs_y;
glyph_dsc_.push_back(glyph_dsc);
bitmap_blob_.insert(bitmap_blob_.end(), entry.bitmap.begin(), entry.bitmap.end());
}
for (const auto& range : ranges) {
lv_font_fmt_txt_cmap_t cmap{};
cmap.range_start = range.start;
cmap.range_length = static_cast<uint16_t>(range.last - range.start + 1);
cmap.glyph_id_start = static_cast<uint16_t>(range.list_begin + 1);
cmap.unicode_list = unicode_list_.data() + range.list_begin;
cmap.glyph_id_ofs_list = nullptr;
cmap.list_length = range.count;
cmap.type = LV_FONT_FMT_TXT_CMAP_SPARSE_TINY;
cmaps_.push_back(cmap);
}
dsc_.glyph_bitmap = bitmap_blob_.empty() ? nullptr : bitmap_blob_.data();
dsc_.glyph_dsc = glyph_dsc_.data();
dsc_.cmaps = cmaps_.empty() ? nullptr : cmaps_.data();
dsc_.kern_dsc = nullptr;
dsc_.kern_scale = 0;
dsc_.cmap_num = cmaps_.size();
dsc_.bpp = bpp_;
dsc_.kern_classes = 0;
dsc_.bitmap_format = LV_FONT_FMT_TXT_PLAIN;
dsc_.stride = 0;
font_.dsc = &dsc_;
}
main/display/lvgl_display/dynamic_glyph_cache.h
+49
View File
@@ -0,0 +1,49 @@
#ifndef DYNAMIC_GLYPH_CACHE_H
#define DYNAMIC_GLYPH_CACHE_H
#include "display.h"
#include <lvgl.h>
#include <cstddef>
#include <cstdint>
#include <vector>
class DynamicGlyphCache {
public:
static constexpr size_t kMaxGlyphs = 256;
static constexpr size_t kMaxBitmapBytes = 64 * 1024;
DynamicGlyphCache();
lv_font_t* EnsureFont(const lv_font_t* base_font, uint8_t bpp);
bool AddGlyphs(const std::vector<TextGlyph>& glyphs);
void Clear();
private:
struct Entry {
uint32_t codepoint = 0;
uint32_t adv_w = 0;
uint16_t box_w = 0;
uint16_t box_h = 0;
int16_t ofs_x = 0;
int16_t ofs_y = 0;
TextGlyphVector<uint8_t> bitmap;
uint64_t last_use = 0;
};
void Rebuild();
size_t BitmapBytes() const;
bool initialized_ = false;
bool retain_between_batches_ = false;
uint8_t bpp_ = 0;
uint64_t use_counter_ = 0;
lv_font_t font_{};
lv_font_fmt_txt_dsc_t dsc_{};
TextGlyphVector<Entry> entries_;
TextGlyphVector<uint8_t> bitmap_blob_;
TextGlyphVector<uint16_t> unicode_list_;
TextGlyphVector<lv_font_fmt_txt_glyph_dsc_t> glyph_dsc_;
TextGlyphVector<lv_font_fmt_txt_cmap_t> cmaps_;
};
#endif
main/display/lvgl_display/emoji_collection.cc
-97
View File
@@ -1,8 +1,6 @@
#include "emoji_collection.h"
#include <esp_log.h>
#include <unordered_map>
#include <string>
#define TAG "EmojiCollection"
@@ -26,98 +24,3 @@ EmojiCollection::~EmojiCollection() {
}
emoji_collection_.clear();
}
// These are declared in xiaozhi-fonts/src/font_emoji_32.c
extern const lv_image_dsc_t emoji_1f636_32; // neutral
extern const lv_image_dsc_t emoji_1f642_32; // happy
extern const lv_image_dsc_t emoji_1f606_32; // laughing
extern const lv_image_dsc_t emoji_1f602_32; // funny
extern const lv_image_dsc_t emoji_1f614_32; // sad
extern const lv_image_dsc_t emoji_1f620_32; // angry
extern const lv_image_dsc_t emoji_1f62d_32; // crying
extern const lv_image_dsc_t emoji_1f60d_32; // loving
extern const lv_image_dsc_t emoji_1f633_32; // embarrassed
extern const lv_image_dsc_t emoji_1f62f_32; // surprised
extern const lv_image_dsc_t emoji_1f631_32; // shocked
extern const lv_image_dsc_t emoji_1f914_32; // thinking
extern const lv_image_dsc_t emoji_1f609_32; // winking
extern const lv_image_dsc_t emoji_1f60e_32; // cool
extern const lv_image_dsc_t emoji_1f60c_32; // relaxed
extern const lv_image_dsc_t emoji_1f924_32; // delicious
extern const lv_image_dsc_t emoji_1f618_32; // kissy
extern const lv_image_dsc_t emoji_1f60f_32; // confident
extern const lv_image_dsc_t emoji_1f634_32; // sleepy
extern const lv_image_dsc_t emoji_1f61c_32; // silly
extern const lv_image_dsc_t emoji_1f644_32; // confused
Twemoji32::Twemoji32() {
AddEmoji("neutral", new LvglSourceImage(&emoji_1f636_32));
AddEmoji("happy", new LvglSourceImage(&emoji_1f642_32));
AddEmoji("laughing", new LvglSourceImage(&emoji_1f606_32));
AddEmoji("funny", new LvglSourceImage(&emoji_1f602_32));
AddEmoji("sad", new LvglSourceImage(&emoji_1f614_32));
AddEmoji("angry", new LvglSourceImage(&emoji_1f620_32));
AddEmoji("crying", new LvglSourceImage(&emoji_1f62d_32));
AddEmoji("loving", new LvglSourceImage(&emoji_1f60d_32));
AddEmoji("embarrassed", new LvglSourceImage(&emoji_1f633_32));
AddEmoji("surprised", new LvglSourceImage(&emoji_1f62f_32));
AddEmoji("shocked", new LvglSourceImage(&emoji_1f631_32));
AddEmoji("thinking", new LvglSourceImage(&emoji_1f914_32));
AddEmoji("winking", new LvglSourceImage(&emoji_1f609_32));
AddEmoji("cool", new LvglSourceImage(&emoji_1f60e_32));
AddEmoji("relaxed", new LvglSourceImage(&emoji_1f60c_32));
AddEmoji("delicious", new LvglSourceImage(&emoji_1f924_32));
AddEmoji("kissy", new LvglSourceImage(&emoji_1f618_32));
AddEmoji("confident", new LvglSourceImage(&emoji_1f60f_32));
AddEmoji("sleepy", new LvglSourceImage(&emoji_1f634_32));
AddEmoji("silly", new LvglSourceImage(&emoji_1f61c_32));
AddEmoji("confused", new LvglSourceImage(&emoji_1f644_32));
}
// These are declared in xiaozhi-fonts/src/font_emoji_64.c
extern const lv_image_dsc_t emoji_1f636_64; // neutral
extern const lv_image_dsc_t emoji_1f642_64; // happy
extern const lv_image_dsc_t emoji_1f606_64; // laughing
extern const lv_image_dsc_t emoji_1f602_64; // funny
extern const lv_image_dsc_t emoji_1f614_64; // sad
extern const lv_image_dsc_t emoji_1f620_64; // angry
extern const lv_image_dsc_t emoji_1f62d_64; // crying
extern const lv_image_dsc_t emoji_1f60d_64; // loving
extern const lv_image_dsc_t emoji_1f633_64; // embarrassed
extern const lv_image_dsc_t emoji_1f62f_64; // surprised
extern const lv_image_dsc_t emoji_1f631_64; // shocked
extern const lv_image_dsc_t emoji_1f914_64; // thinking
extern const lv_image_dsc_t emoji_1f609_64; // winking
extern const lv_image_dsc_t emoji_1f60e_64; // cool
extern const lv_image_dsc_t emoji_1f60c_64; // relaxed
extern const lv_image_dsc_t emoji_1f924_64; // delicious
extern const lv_image_dsc_t emoji_1f618_64; // kissy
extern const lv_image_dsc_t emoji_1f60f_64; // confident
extern const lv_image_dsc_t emoji_1f634_64; // sleepy
extern const lv_image_dsc_t emoji_1f61c_64; // silly
extern const lv_image_dsc_t emoji_1f644_64; // confused
Twemoji64::Twemoji64() {
AddEmoji("neutral", new LvglSourceImage(&emoji_1f636_64));
AddEmoji("happy", new LvglSourceImage(&emoji_1f642_64));
AddEmoji("laughing", new LvglSourceImage(&emoji_1f606_64));
AddEmoji("funny", new LvglSourceImage(&emoji_1f602_64));
AddEmoji("sad", new LvglSourceImage(&emoji_1f614_64));
AddEmoji("angry", new LvglSourceImage(&emoji_1f620_64));
AddEmoji("crying", new LvglSourceImage(&emoji_1f62d_64));
AddEmoji("loving", new LvglSourceImage(&emoji_1f60d_64));
AddEmoji("embarrassed", new LvglSourceImage(&emoji_1f633_64));
AddEmoji("surprised", new LvglSourceImage(&emoji_1f62f_64));
AddEmoji("shocked", new LvglSourceImage(&emoji_1f631_64));
AddEmoji("thinking", new LvglSourceImage(&emoji_1f914_64));
AddEmoji("winking", new LvglSourceImage(&emoji_1f609_64));
AddEmoji("cool", new LvglSourceImage(&emoji_1f60e_64));
AddEmoji("relaxed", new LvglSourceImage(&emoji_1f60c_64));
AddEmoji("delicious", new LvglSourceImage(&emoji_1f924_64));
AddEmoji("kissy", new LvglSourceImage(&emoji_1f618_64));
AddEmoji("confident", new LvglSourceImage(&emoji_1f60f_64));
AddEmoji("sleepy", new LvglSourceImage(&emoji_1f634_64));
AddEmoji("silly", new LvglSourceImage(&emoji_1f61c_64));
AddEmoji("confused", new LvglSourceImage(&emoji_1f644_64));
}
main/display/lvgl_display/emoji_collection.h
+1 -12
View File
@@ -6,9 +6,8 @@
#include <lvgl.h>
#include <map>
#include <string>
#include <memory>
#include <string>
// Define interface for emoji collection
class EmojiCollection {
@@ -21,14 +20,4 @@ private:
std::map<std::string, LvglImage*> emoji_collection_;
};
class Twemoji32 : public EmojiCollection {
public:
Twemoji32();
};
class Twemoji64 : public EmojiCollection {
public:
Twemoji64();
};
#endif
main/display/lvgl_display/lvgl_display.cc
+134 -50
View File
@@ -1,29 +1,33 @@
#include <esp_log.h>
#include <esp_err.h>
#include <string>
#include <esp_log.h>
#include <material_symbols.h>
#include <cstdlib>
#include <cstring>
#include <font_awesome.h>
#include <string>
#include "lvgl_display.h"
#include "board.h"
#include "application.h"
#include "audio_codec.h"
#include "settings.h"
#include "assets/lang_config.h"
#include "audio_codec.h"
#include "board.h"
#include "dynamic_glyph_cache.h"
#include "jpg/image_to_jpeg.h"
#include "lvgl_display.h"
#include "lvgl_theme.h"
#include "settings.h"
#define TAG "Display"
LvglDisplay::LvglDisplay() {
dynamic_glyph_cache_ = std::make_unique<DynamicGlyphCache>();
// Notification timer
esp_timer_create_args_t notification_timer_args = {
.callback = [](void *arg) {
LvglDisplay *display = static_cast<LvglDisplay*>(arg);
DisplayLockGuard lock(display);
lv_obj_add_flag(display->notification_label_, LV_OBJ_FLAG_HIDDEN);
lv_obj_remove_flag(display->status_label_, LV_OBJ_FLAG_HIDDEN);
},
.callback =
[](void* arg) {
LvglDisplay* display = static_cast<LvglDisplay*>(arg);
DisplayLockGuard lock(display);
lv_obj_add_flag(display->notification_label_, LV_OBJ_FLAG_HIDDEN);
lv_obj_remove_flag(display->status_label_, LV_OBJ_FLAG_HIDDEN);
},
.arg = this,
.dispatch_method = ESP_TIMER_TASK,
.name = "notification_timer",
@@ -40,6 +44,73 @@ LvglDisplay::LvglDisplay() {
}
}
bool LvglDisplay::SetTextFont(std::shared_ptr<LvglFont> text_font) {
if (text_font == nullptr || text_font->font() == nullptr) {
return false;
}
DisplayLockGuard lock(this);
auto& theme_manager = LvglThemeManager::GetInstance();
auto light_theme = theme_manager.GetTheme("light");
auto dark_theme = theme_manager.GetTheme("dark");
if (light_theme == nullptr && dark_theme == nullptr) {
return false;
}
// LVGL styles keep raw lv_font_t pointers. Keep the previous font owners alive until the
// current theme has rebound every style to the new font.
auto previous_light_font = light_theme != nullptr ? light_theme->text_font() : nullptr;
auto previous_dark_font = dark_theme != nullptr ? dark_theme->text_font() : nullptr;
if (light_theme != nullptr) {
light_theme->set_text_font(text_font);
}
if (dark_theme != nullptr) {
dark_theme->set_text_font(text_font);
}
if (current_theme_ != nullptr) {
SetTheme(current_theme_);
}
previous_light_font.reset();
previous_dark_font.reset();
return true;
}
bool LvglDisplay::AddTextGlyphs(const std::vector<TextGlyph>& glyphs, uint8_t bpp) {
if (dynamic_glyph_cache_ == nullptr) {
return false;
}
if (glyphs.empty()) {
if (!TextGlyphStorageUsesPsram()) {
ClearTextGlyphs();
}
return false;
}
if (bpp != 1 && bpp != 4) {
return false;
}
DisplayLockGuard lock(this);
auto theme = dynamic_cast<LvglTheme*>(current_theme_);
if (theme == nullptr || theme->text_font() == nullptr) {
return false;
}
auto fallback = dynamic_glyph_cache_->EnsureFont(theme->text_font()->font(), bpp);
if (fallback == nullptr) {
return false;
}
theme->text_font()->SetFallback(fallback);
return dynamic_glyph_cache_->AddGlyphs(glyphs);
}
void LvglDisplay::ClearTextGlyphs() {
if (dynamic_glyph_cache_ == nullptr) {
return;
}
DisplayLockGuard lock(this);
dynamic_glyph_cache_->Clear();
}
LvglDisplay::~LvglDisplay() {
if (notification_timer_ != nullptr) {
esp_timer_stop(notification_timer_);
@@ -61,7 +132,7 @@ LvglDisplay::~LvglDisplay() {
if (battery_label_ != nullptr) {
lv_obj_del(battery_label_);
}
if( low_battery_popup_ != nullptr ) {
if (low_battery_popup_ != nullptr) {
lv_obj_del(low_battery_popup_);
}
if (pm_lock_ != nullptr) {
@@ -76,7 +147,10 @@ void LvglDisplay::SetStatus(const char* status) {
DisplayLockGuard lock(this);
if (status_label_ == nullptr) {
if (setup_ui_called_) {
ESP_LOGW(TAG, "SetStatus('%s') failed: status_label_ is nullptr (SetupUI() was called but label not created)", status);
ESP_LOGW(TAG,
"SetStatus('%s') failed: status_label_ is nullptr (SetupUI() was called but "
"label not created)",
status);
}
return;
}
@@ -87,18 +161,22 @@ void LvglDisplay::SetStatus(const char* status) {
last_status_update_time_ = std::chrono::system_clock::now();
}
void LvglDisplay::ShowNotification(const std::string &notification, int duration_ms) {
void LvglDisplay::ShowNotification(const std::string& notification, int duration_ms) {
ShowNotification(notification.c_str(), duration_ms);
}
void LvglDisplay::ShowNotification(const char* notification, int duration_ms) {
if (!setup_ui_called_) {
ESP_LOGW(TAG, "ShowNotification('%s') called before SetupUI() - message will be lost!", notification);
ESP_LOGW(TAG, "ShowNotification('%s') called before SetupUI() - message will be lost!",
notification);
}
DisplayLockGuard lock(this);
if (notification_label_ == nullptr) {
if (setup_ui_called_) {
ESP_LOGW(TAG, "ShowNotification('%s') failed: notification_label_ is nullptr (SetupUI() was called but label not created)", notification);
ESP_LOGW(TAG,
"ShowNotification('%s') failed: notification_label_ is nullptr (SetupUI() was "
"called but label not created)",
notification);
}
return;
}
@@ -125,7 +203,7 @@ void LvglDisplay::UpdateStatusBar(bool update_all) {
// Update icon if mute state changes
if (codec->output_volume() == 0 && !muted_) {
muted_ = true;
lv_label_set_text(mute_label_, FONT_AWESOME_VOLUME_XMARK);
lv_label_set_text(mute_label_, MATERIAL_SYMBOLS_VOLUME_OFF);
} else if (codec->output_volume() > 0 && muted_) {
muted_ = false;
lv_label_set_text(mute_label_, "");
@@ -134,7 +212,8 @@ void LvglDisplay::UpdateStatusBar(bool update_all) {
// Update time
if (app.GetDeviceState() == kDeviceStateIdle) {
if (last_status_update_time_ + std::chrono::seconds(10) < std::chrono::system_clock::now()) {
if (last_status_update_time_ + std::chrono::seconds(10) <
std::chrono::system_clock::now()) {
// Set status to clock "HH:MM"
time_t now = time(NULL);
struct tm* tm = localtime(&now);
@@ -156,17 +235,22 @@ void LvglDisplay::UpdateStatusBar(bool update_all) {
const char* icon = nullptr;
if (board.GetBatteryLevel(battery_level, charging, discharging)) {
if (charging) {
icon = FONT_AWESOME_BATTERY_BOLT;
icon = MATERIAL_SYMBOLS_BATTERY_ANDROID_FRAME_BOLT;
} else {
const char* levels[] = {
FONT_AWESOME_BATTERY_EMPTY, // 0-19%
FONT_AWESOME_BATTERY_QUARTER, // 20-39%
FONT_AWESOME_BATTERY_HALF, // 40-59%
FONT_AWESOME_BATTERY_THREE_QUARTERS, // 60-79%
FONT_AWESOME_BATTERY_FULL, // 80-99%
FONT_AWESOME_BATTERY_FULL, // 100%
MATERIAL_SYMBOLS_BATTERY_ANDROID_0,
MATERIAL_SYMBOLS_BATTERY_ANDROID_FRAME_1,
MATERIAL_SYMBOLS_BATTERY_ANDROID_FRAME_2,
MATERIAL_SYMBOLS_BATTERY_ANDROID_FRAME_3,
MATERIAL_SYMBOLS_BATTERY_ANDROID_FRAME_4,
MATERIAL_SYMBOLS_BATTERY_ANDROID_FRAME_5,
MATERIAL_SYMBOLS_BATTERY_ANDROID_FRAME_6,
MATERIAL_SYMBOLS_BATTERY_ANDROID_FRAME_FULL,
};
icon = levels[battery_level / 20];
int level_index = battery_level <= 0
? 0
: (battery_level >= 100 ? 7 : 1 + ((battery_level - 1) * 6 / 99));
icon = levels[level_index];
}
DisplayLockGuard lock(this);
if (battery_label_ != nullptr && battery_icon_ != icon) {
@@ -177,16 +261,16 @@ void LvglDisplay::UpdateStatusBar(bool update_all) {
// Check low battery popup only when clock tick event is triggered
// Because when initializing, the battery level is not ready yet.
if (low_battery_popup_ != nullptr && !update_all) {
if (strcmp(icon, FONT_AWESOME_BATTERY_EMPTY) == 0 && discharging) {
if (lv_obj_has_flag(low_battery_popup_, LV_OBJ_FLAG_HIDDEN)) { // Show if low battery popup is hidden
if (strcmp(icon, MATERIAL_SYMBOLS_BATTERY_ANDROID_0) == 0 && discharging) {
if (lv_obj_has_flag(low_battery_popup_,
LV_OBJ_FLAG_HIDDEN)) { // Show if low battery popup is hidden
lv_obj_remove_flag(low_battery_popup_, LV_OBJ_FLAG_HIDDEN);
app.Schedule([&app]() {
app.PlaySound(Lang::Sounds::OGG_LOW_BATTERY);
});
app.Schedule([&app]() { app.PlaySound(Lang::Sounds::OGG_LOW_BATTERY); });
}
} else {
// Hide the low battery popup when the battery is not empty
if (!lv_obj_has_flag(low_battery_popup_, LV_OBJ_FLAG_HIDDEN)) { // Hide if low battery popup is shown
if (!lv_obj_has_flag(low_battery_popup_,
LV_OBJ_FLAG_HIDDEN)) { // Hide if low battery popup is shown
lv_obj_add_flag(low_battery_popup_, LV_OBJ_FLAG_HIDDEN);
}
}
@@ -199,13 +283,11 @@ void LvglDisplay::UpdateStatusBar(bool update_all) {
// Don't read 4G network status during firmware upgrade to avoid occupying UART resources
auto device_state = Application::GetInstance().GetDeviceState();
static const std::vector<DeviceState> allowed_states = {
kDeviceStateIdle,
kDeviceStateStarting,
kDeviceStateWifiConfiguring,
kDeviceStateListening,
kDeviceStateActivating,
kDeviceStateIdle, kDeviceStateStarting, kDeviceStateWifiConfiguring,
kDeviceStateListening, kDeviceStateActivating,
};
if (std::find(allowed_states.begin(), allowed_states.end(), device_state) != allowed_states.end()) {
if (std::find(allowed_states.begin(), allowed_states.end(), device_state) !=
allowed_states.end()) {
icon = board.GetNetworkStateIcon();
if (network_label_ != nullptr && icon != nullptr && network_icon_ != icon) {
DisplayLockGuard lock(this);
@@ -218,8 +300,7 @@ void LvglDisplay::UpdateStatusBar(bool update_all) {
esp_pm_lock_release(pm_lock_);
}
void LvglDisplay::SetPreviewImage(std::unique_ptr<LvglImage> image) {
}
void LvglDisplay::SetPreviewImage(std::unique_ptr<LvglImage> image) {}
void LvglDisplay::SetPowerSaveMode(bool on) {
if (on) {
@@ -253,14 +334,17 @@ bool LvglDisplay::SnapshotToJpeg(std::string& jpeg_data, int quality) {
jpeg_data.clear();
// Use callback-based JPEG encoder to further save memory
bool ret = image_to_jpeg_cb((uint8_t*)draw_buffer->data, draw_buffer->data_size, draw_buffer->header.w, draw_buffer->header.h, V4L2_PIX_FMT_RGB565, quality,
[](void *arg, size_t index, const void *data, size_t len) -> size_t {
std::string* output = static_cast<std::string*>(arg);
if (data && len > 0) {
output->append(static_cast<const char*>(data), len);
}
return len;
}, &jpeg_data);
bool ret =
image_to_jpeg_cb((uint8_t*)draw_buffer->data, draw_buffer->data_size, draw_buffer->header.w,
draw_buffer->header.h, V4L2_PIX_FMT_RGB565, quality,
[](void* arg, size_t index, const void* data, size_t len) -> size_t {
std::string* output = static_cast<std::string*>(arg);
if (data && len > 0) {
output->append(static_cast<const char*>(data), len);
}
return len;
},
&jpeg_data);
if (!ret) {
ESP_LOGE(TAG, "Failed to convert image to JPEG");
}
main/display/lvgl_display/lvgl_display.h
+19 -12
View File
@@ -4,13 +4,17 @@
#include "display.h"
#include "lvgl_image.h"
#include <lvgl.h>
#include <esp_timer.h>
#include <esp_log.h>
#include <esp_pm.h>
#include <esp_timer.h>
#include <lvgl.h>
#include <string>
#include <chrono>
#include <memory>
#include <string>
class DynamicGlyphCache;
class LvglFont;
class LvglDisplay : public Display {
public:
@@ -19,35 +23,38 @@ public:
virtual void SetStatus(const char* status);
virtual void ShowNotification(const char* notification, int duration_ms = 3000);
virtual void ShowNotification(const std::string &notification, int duration_ms = 3000);
virtual void ShowNotification(const std::string& notification, int duration_ms = 3000);
virtual void SetPreviewImage(std::unique_ptr<LvglImage> image);
virtual void UpdateStatusBar(bool update_all = false);
virtual void SetPowerSaveMode(bool on);
virtual bool SnapshotToJpeg(std::string& jpeg_data, int quality = 80);
virtual bool AddTextGlyphs(const std::vector<TextGlyph>& glyphs, uint8_t bpp) override;
virtual void ClearTextGlyphs() override;
bool SetTextFont(std::shared_ptr<LvglFont> text_font);
protected:
esp_pm_lock_handle_t pm_lock_ = nullptr;
lv_display_t *display_ = nullptr;
lv_display_t* display_ = nullptr;
lv_obj_t *network_label_ = nullptr;
lv_obj_t *status_label_ = nullptr;
lv_obj_t *notification_label_ = nullptr;
lv_obj_t *mute_label_ = nullptr;
lv_obj_t *battery_label_ = nullptr;
lv_obj_t* network_label_ = nullptr;
lv_obj_t* status_label_ = nullptr;
lv_obj_t* notification_label_ = nullptr;
lv_obj_t* mute_label_ = nullptr;
lv_obj_t* battery_label_ = nullptr;
lv_obj_t* low_battery_popup_ = nullptr;
lv_obj_t* low_battery_label_ = nullptr;
const char* battery_icon_ = nullptr;
const char* network_icon_ = nullptr;
bool muted_ = false;
std::chrono::system_clock::time_point last_status_update_time_;
esp_timer_handle_t notification_timer_ = nullptr;
std::unique_ptr<DynamicGlyphCache> dynamic_glyph_cache_;
friend class DisplayLockGuard;
virtual bool Lock(int timeout_ms = 0) = 0;
virtual void Unlock() = 0;
};
#endif
main/display/lvgl_display/lvgl_font.h
+16 -5
View File
@@ -2,29 +2,40 @@
#include <lvgl.h>
class LvglFont {
public:
virtual const lv_font_t* font() const = 0;
virtual void SetFallback(const lv_font_t* fallback) = 0;
virtual ~LvglFont() = default;
};
// Built-in font
class LvglBuiltInFont : public LvglFont {
public:
LvglBuiltInFont(const lv_font_t* font) : font_(font) {}
virtual const lv_font_t* font() const override { return font_; }
LvglBuiltInFont(const lv_font_t* font) : font_(*font) {}
virtual const lv_font_t* font() const override { return &font_; }
virtual void SetFallback(const lv_font_t* fallback) override { font_.fallback = fallback; }
private:
const lv_font_t* font_;
lv_font_t font_{};
};
class LvglCBinFont : public LvglFont {
public:
LvglCBinFont(void* data);
virtual ~LvglCBinFont();
virtual const lv_font_t* font() const override { return font_; }
uint8_t bpp() const {
if (font_ == nullptr || font_->dsc == nullptr) {
return 0;
}
return static_cast<const lv_font_fmt_txt_dsc_t*>(font_->dsc)->bpp;
}
virtual void SetFallback(const lv_font_t* fallback) override {
if (font_ != nullptr) {
font_->fallback = fallback;
}
}
private:
lv_font_t* font_;
main/display/lvgl_display/lvgl_image.h
+3 -13
View File
@@ -2,7 +2,6 @@
#include <lvgl.h>
// Wrap around lv_img_dsc_t
class LvglImage {
public:
@@ -11,7 +10,6 @@ public:
virtual ~LvglImage() = default;
};
class LvglRawImage : public LvglImage {
public:
LvglRawImage(void* data, size_t size);
@@ -32,22 +30,14 @@ private:
lv_img_dsc_t* image_dsc_ = nullptr;
};
class LvglSourceImage : public LvglImage {
public:
LvglSourceImage(const lv_img_dsc_t* image_dsc) : image_dsc_(image_dsc) {}
virtual const lv_img_dsc_t* image_dsc() const override { return image_dsc_; }
private:
const lv_img_dsc_t* image_dsc_;
};
class LvglAllocatedImage : public LvglImage {
public:
LvglAllocatedImage(void* data, size_t size);
LvglAllocatedImage(void* data, size_t size, int width, int height, int stride, int color_format);
LvglAllocatedImage(void* data, size_t size, int width, int height, int stride,
int color_format);
virtual ~LvglAllocatedImage();
virtual const lv_img_dsc_t* image_dsc() const override { return &image_dsc_; }
private:
lv_img_dsc_t image_dsc_;
};
};
main/display/lvgl_display/lvgl_theme.h
+24 -11
View File
@@ -1,16 +1,15 @@
#pragma once
#include "display.h"
#include "lvgl_image.h"
#include "lvgl_font.h"
#include "emoji_collection.h"
#include "lvgl_font.h"
#include "lvgl_image.h"
#include <lvgl.h>
#include <memory>
#include <map>
#include <memory>
#include <string>
class LvglTheme : public Theme {
public:
static lv_color_t ParseColor(const std::string& color);
@@ -32,22 +31,36 @@ public:
inline std::shared_ptr<LvglFont> text_font() const { return text_font_; }
inline std::shared_ptr<LvglFont> icon_font() const { return icon_font_; }
inline std::shared_ptr<LvglFont> large_icon_font() const { return large_icon_font_; }
inline std::shared_ptr<LvglFont> emoji_font() const { return emoji_font_; }
inline int spacing(int scale) const { return spacing_ * scale; }
inline void set_background_color(lv_color_t background) { background_color_ = background; }
inline void set_text_color(lv_color_t text) { text_color_ = text; }
inline void set_chat_background_color(lv_color_t chat_background) { chat_background_color_ = chat_background; }
inline void set_chat_background_color(lv_color_t chat_background) {
chat_background_color_ = chat_background;
}
inline void set_user_bubble_color(lv_color_t user_bubble) { user_bubble_color_ = user_bubble; }
inline void set_assistant_bubble_color(lv_color_t assistant_bubble) { assistant_bubble_color_ = assistant_bubble; }
inline void set_system_bubble_color(lv_color_t system_bubble) { system_bubble_color_ = system_bubble; }
inline void set_assistant_bubble_color(lv_color_t assistant_bubble) {
assistant_bubble_color_ = assistant_bubble;
}
inline void set_system_bubble_color(lv_color_t system_bubble) {
system_bubble_color_ = system_bubble;
}
inline void set_system_text_color(lv_color_t system_text) { system_text_color_ = system_text; }
inline void set_border_color(lv_color_t border) { border_color_ = border; }
inline void set_low_battery_color(lv_color_t low_battery) { low_battery_color_ = low_battery; }
inline void set_background_image(std::shared_ptr<LvglImage> background_image) { background_image_ = background_image; }
inline void set_emoji_collection(std::shared_ptr<EmojiCollection> emoji_collection) { emoji_collection_ = emoji_collection; }
inline void set_background_image(std::shared_ptr<LvglImage> background_image) {
background_image_ = background_image;
}
inline void set_emoji_collection(std::shared_ptr<EmojiCollection> emoji_collection) {
emoji_collection_ = emoji_collection;
}
inline void set_text_font(std::shared_ptr<LvglFont> text_font) { text_font_ = text_font; }
inline void set_icon_font(std::shared_ptr<LvglFont> icon_font) { icon_font_ = icon_font; }
inline void set_large_icon_font(std::shared_ptr<LvglFont> large_icon_font) { large_icon_font_ = large_icon_font; }
inline void set_large_icon_font(std::shared_ptr<LvglFont> large_icon_font) {
large_icon_font_ = large_icon_font;
}
inline void set_emoji_font(std::shared_ptr<LvglFont> emoji_font) { emoji_font_ = emoji_font; }
private:
int spacing_ = 2;
@@ -70,12 +83,12 @@ private:
std::shared_ptr<LvglFont> text_font_ = nullptr;
std::shared_ptr<LvglFont> icon_font_ = nullptr;
std::shared_ptr<LvglFont> large_icon_font_ = nullptr;
std::shared_ptr<LvglFont> emoji_font_ = nullptr;
// Emoji collection
std::shared_ptr<EmojiCollection> emoji_collection_ = nullptr;
};
class LvglThemeManager {
public:
static LvglThemeManager& GetInstance() {
main/display/oled_display.cc
+51 -36
View File
@@ -1,36 +1,40 @@
#include "oled_display.h"
#include "assets/lang_config.h"
#include "lvgl_theme.h"
#include "lvgl_font.h"
#include "lvgl_theme.h"
#include <string>
#include <algorithm>
#include <string>
#include <esp_log.h>
#include <esp_err.h>
#include <esp_log.h>
#include <esp_lvgl_port.h>
#include <font_awesome.h>
#include <material_symbols.h>
#include <noto_emoji.h>
#define TAG "OledDisplay"
LV_FONT_DECLARE(BUILTIN_TEXT_FONT);
LV_FONT_DECLARE(BUILTIN_ICON_FONT);
LV_FONT_DECLARE(font_awesome_30_1);
LV_FONT_DECLARE(font_material_symbols_30_1);
LV_FONT_DECLARE(font_noto_emoji_30_1);
OledDisplay::OledDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel,
int width, int height, bool mirror_x, bool mirror_y)
int width, int height, bool mirror_x, bool mirror_y)
: panel_io_(panel_io), panel_(panel) {
width_ = width;
height_ = height;
auto text_font = std::make_shared<LvglBuiltInFont>(&BUILTIN_TEXT_FONT);
auto icon_font = std::make_shared<LvglBuiltInFont>(&BUILTIN_ICON_FONT);
auto large_icon_font = std::make_shared<LvglBuiltInFont>(&font_awesome_30_1);
auto large_icon_font = std::make_shared<LvglBuiltInFont>(&font_material_symbols_30_1);
auto emoji_font = std::make_shared<LvglBuiltInFont>(&font_noto_emoji_30_1);
auto dark_theme = new LvglTheme("dark");
dark_theme->set_text_font(text_font);
dark_theme->set_icon_font(icon_font);
dark_theme->set_large_icon_font(large_icon_font);
dark_theme->set_emoji_font(emoji_font);
auto& theme_manager = LvglThemeManager::GetInstance();
theme_manager.RegisterTheme("dark", dark_theme);
@@ -56,18 +60,20 @@ OledDisplay::OledDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handl
.hres = static_cast<uint32_t>(width_),
.vres = static_cast<uint32_t>(height_),
.monochrome = true,
.rotation = {
.swap_xy = false,
.mirror_x = mirror_x,
.mirror_y = mirror_y,
},
.flags = {
.buff_dma = 1,
.buff_spiram = 0,
.sw_rotate = 0,
.full_refresh = 0,
.direct_mode = 0,
},
.rotation =
{
.swap_xy = false,
.mirror_x = mirror_x,
.mirror_y = mirror_y,
},
.flags =
{
.buff_dma = 1,
.buff_spiram = 0,
.sw_rotate = 0,
.full_refresh = 0,
.direct_mode = 0,
},
};
display_ = lvgl_port_add_disp(&display_cfg);
@@ -86,7 +92,7 @@ void OledDisplay::SetupUI() {
ESP_LOGW(TAG, "SetupUI() called multiple times, skipping duplicate call");
return;
}
Display::SetupUI(); // Mark SetupUI as called
if (height_ == 64) {
SetupUI_128x64();
@@ -135,13 +141,9 @@ OledDisplay::~OledDisplay() {
lvgl_port_deinit();
}
bool OledDisplay::Lock(int timeout_ms) {
return lvgl_port_lock(timeout_ms);
}
bool OledDisplay::Lock(int timeout_ms) { return lvgl_port_lock(timeout_ms); }
void OledDisplay::Unlock() {
lvgl_port_unlock();
}
void OledDisplay::Unlock() { lvgl_port_unlock(); }
void OledDisplay::SetChatMessage(const char* role, const char* content) {
DisplayLockGuard lock(this);
@@ -153,6 +155,7 @@ void OledDisplay::SetChatMessage(const char* role, const char* content) {
std::string content_str = content;
std::replace(content_str.begin(), content_str.end(), '\n', ' ');
lv_anim_delete(chat_message_label_, nullptr);
if (content_right_ == nullptr) {
lv_label_set_text(chat_message_label_, content_str.c_str());
} else {
@@ -193,7 +196,8 @@ void OledDisplay::SetupUI_128x64() {
lv_obj_set_style_border_width(top_bar_, 0, 0);
lv_obj_set_style_pad_all(top_bar_, 0, 0);
lv_obj_set_flex_flow(top_bar_, LV_FLEX_FLOW_ROW);
lv_obj_set_flex_align(top_bar_, LV_FLEX_ALIGN_SPACE_BETWEEN, LV_FLEX_ALIGN_CENTER, LV_FLEX_ALIGN_CENTER);
lv_obj_set_flex_align(top_bar_, LV_FLEX_ALIGN_SPACE_BETWEEN, LV_FLEX_ALIGN_CENTER,
LV_FLEX_ALIGN_CENTER);
lv_obj_set_scrollbar_mode(top_bar_, LV_SCROLLBAR_MODE_OFF);
network_label_ = lv_label_create(top_bar_);
@@ -206,7 +210,8 @@ void OledDisplay::SetupUI_128x64() {
lv_obj_set_style_border_width(right_icons, 0, 0);
lv_obj_set_style_pad_all(right_icons, 0, 0);
lv_obj_set_flex_flow(right_icons, LV_FLEX_FLOW_ROW);
lv_obj_set_flex_align(right_icons, LV_FLEX_ALIGN_END, LV_FLEX_ALIGN_CENTER, LV_FLEX_ALIGN_CENTER);
lv_obj_set_flex_align(right_icons, LV_FLEX_ALIGN_END, LV_FLEX_ALIGN_CENTER,
LV_FLEX_ALIGN_CENTER);
mute_label_ = lv_label_create(right_icons);
lv_label_set_text(mute_label_, "");
@@ -225,7 +230,7 @@ void OledDisplay::SetupUI_128x64() {
lv_obj_set_style_pad_all(status_bar_, 0, 0);
lv_obj_set_scrollbar_mode(status_bar_, LV_SCROLLBAR_MODE_OFF);
lv_obj_set_style_layout(status_bar_, LV_LAYOUT_NONE, 0); // Use absolute positioning
lv_obj_align(status_bar_, LV_ALIGN_TOP_MID, 0, 0); // Overlap with top_bar_
lv_obj_align(status_bar_, LV_ALIGN_TOP_MID, 0, 0); // Overlap with top_bar_
notification_label_ = lv_label_create(status_bar_);
lv_obj_set_width(notification_label_, LV_HOR_RES);
@@ -258,7 +263,7 @@ void OledDisplay::SetupUI_128x64() {
emotion_label_ = lv_label_create(content_left_);
lv_obj_set_style_text_font(emotion_label_, large_icon_font, 0);
lv_label_set_text(emotion_label_, FONT_AWESOME_MICROCHIP_AI);
lv_label_set_text(emotion_label_, MATERIAL_SYMBOLS_ROBOT_2);
lv_obj_center(emotion_label_);
lv_obj_set_style_pad_top(emotion_label_, 8, 0);
@@ -282,7 +287,8 @@ void OledDisplay::SetupUI_128x64() {
lv_anim_set_delay(&a, 1000);
lv_anim_set_repeat_count(&a, LV_ANIM_REPEAT_INFINITE);
lv_obj_set_style_anim(chat_message_label_, &a, LV_PART_MAIN);
lv_obj_set_style_anim_duration(chat_message_label_, lv_anim_speed_clamped(60, 300, 60000), LV_PART_MAIN);
lv_obj_set_style_anim_duration(chat_message_label_, lv_anim_speed_clamped(60, 300, 60000),
LV_PART_MAIN);
low_battery_popup_ = lv_obj_create(screen);
lv_obj_set_scrollbar_mode(low_battery_popup_, LV_SCROLLBAR_MODE_OFF);
@@ -325,7 +331,7 @@ void OledDisplay::SetupUI_128x32() {
emotion_label_ = lv_label_create(content_);
lv_obj_set_style_text_font(emotion_label_, large_icon_font, 0);
lv_label_set_text(emotion_label_, FONT_AWESOME_MICROCHIP_AI);
lv_label_set_text(emotion_label_, MATERIAL_SYMBOLS_ROBOT_2);
lv_obj_center(emotion_label_);
/* Right side */
@@ -381,19 +387,28 @@ void OledDisplay::SetupUI_128x32() {
lv_anim_set_delay(&a, 1000);
lv_anim_set_repeat_count(&a, LV_ANIM_REPEAT_INFINITE);
lv_obj_set_style_anim(chat_message_label_, &a, LV_PART_MAIN);
lv_obj_set_style_anim_duration(chat_message_label_, lv_anim_speed_clamped(60, 300, 60000), LV_PART_MAIN);
lv_obj_set_style_anim_duration(chat_message_label_, lv_anim_speed_clamped(60, 300, 60000),
LV_PART_MAIN);
}
void OledDisplay::SetEmotion(const char* emotion) {
const char* utf8 = font_awesome_get_utf8(emotion);
auto lvgl_theme = static_cast<LvglTheme*>(current_theme_);
const char* utf8 = noto_emoji_get_utf8(emotion);
const lv_font_t* emotion_font = lvgl_theme->emoji_font()->font();
if (utf8 == nullptr) {
utf8 = material_symbols_get_utf8(emotion);
emotion_font = lvgl_theme->large_icon_font()->font();
}
DisplayLockGuard lock(this);
if (emotion_label_ == nullptr) {
return;
}
if (utf8 != nullptr) {
lv_obj_set_style_text_font(emotion_label_, emotion_font, 0);
lv_label_set_text(emotion_label_, utf8);
} else {
lv_label_set_text(emotion_label_, FONT_AWESOME_NEUTRAL);
lv_obj_set_style_text_font(emotion_label_, lvgl_theme->emoji_font()->font(), 0);
lv_label_set_text(emotion_label_, NOTO_EMOJI_NEUTRAL);
}
}
main/display/text_glyph.cc
+3
View File
@@ -0,0 +1,3 @@
#include "text_glyph.h"
bool TextGlyphStorageUsesPsram() { return heap_caps_get_total_size(MALLOC_CAP_SPIRAM) > 0; }
main/display/text_glyph.h
+67
View File
@@ -0,0 +1,67 @@
#pragma once
#include <esp_heap_caps.h>
#include <cstddef>
#include <cstdint>
#include <cstdlib>
#include <limits>
#include <type_traits>
#include <vector>
bool TextGlyphStorageUsesPsram();
template <typename T>
class TextGlyphAllocator {
public:
using value_type = T;
using is_always_equal = std::true_type;
TextGlyphAllocator() noexcept = default;
template <typename U>
TextGlyphAllocator(const TextGlyphAllocator<U>&) noexcept {}
T* allocate(size_t count) {
if (count == 0) {
return nullptr;
}
if (count > std::numeric_limits<size_t>::max() / sizeof(T)) {
std::abort();
}
uint32_t caps = MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT;
if (TextGlyphStorageUsesPsram()) {
caps = MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT;
}
auto ptr = static_cast<T*>(heap_caps_malloc(count * sizeof(T), caps));
if (ptr == nullptr) {
std::abort();
}
return ptr;
}
void deallocate(T* ptr, size_t) noexcept { heap_caps_free(ptr); }
};
template <typename T, typename U>
bool operator==(const TextGlyphAllocator<T>&, const TextGlyphAllocator<U>&) {
return true;
}
template <typename T, typename U>
bool operator!=(const TextGlyphAllocator<T>&, const TextGlyphAllocator<U>&) {
return false;
}
template <typename T>
using TextGlyphVector = std::vector<T, TextGlyphAllocator<T>>;
struct TextGlyph {
uint32_t codepoint = 0;
uint32_t adv_w = 0;
uint16_t box_w = 0;
uint16_t box_h = 0;
int16_t ofs_x = 0;
int16_t ofs_y = 0;
TextGlyphVector<uint8_t> bitmap;
};
main/idf_component.yml
+1 -1
View File
@@ -30,7 +30,7 @@ dependencies:
version: ==0.6.0
rules:
- if: target not in [esp32]
78/xiaozhi-fonts: ~1.6.0
78/xiaozhi-fonts: ~2.0.0
espressif/led_strip: ~3.0.2
espressif/esp_codec_dev: ~1.5.6
espressif/esp-sr: ~2.4.6
main/protocols/mqtt_protocol.cc
+52 -39
View File
@@ -1,11 +1,11 @@
#include "mqtt_protocol.h"
#include "board.h"
#include "application.h"
#include "board.h"
#include "settings.h"
#include <esp_log.h>
#include <cstring>
#include <arpa/inet.h>
#include <cstring>
#include "assets/lang_config.h"
#define TAG "MQTT"
@@ -15,19 +15,20 @@ MqttProtocol::MqttProtocol() {
// Initialize reconnect timer
esp_timer_create_args_t reconnect_timer_args = {
.callback = [](void* arg) {
MqttProtocol* protocol = (MqttProtocol*)arg;
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateIdle) {
ESP_LOGI(TAG, "Reconnecting to MQTT server");
auto alive = protocol->alive_; // Capture alive flag
app.Schedule([protocol, alive]() {
if (*alive) {
protocol->StartMqttClient(false);
}
});
}
},
.callback =
[](void* arg) {
MqttProtocol* protocol = (MqttProtocol*)arg;
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateIdle) {
ESP_LOGI(TAG, "Reconnecting to MQTT server");
auto alive = protocol->alive_; // Capture alive flag
app.Schedule([protocol, alive]() {
if (*alive) {
protocol->StartMqttClient(false);
}
});
}
},
.arg = this,
};
esp_timer_create(&reconnect_timer_args, &reconnect_timer_);
@@ -35,19 +36,21 @@ MqttProtocol::MqttProtocol() {
MqttProtocol::~MqttProtocol() {
ESP_LOGI(TAG, "MqttProtocol deinit");
// Mark as dead first to prevent any pending scheduled tasks from executing
*alive_ = false;
if (reconnect_timer_ != nullptr) {
esp_timer_stop(reconnect_timer_);
esp_timer_delete(reconnect_timer_);
}
std::unique_ptr<Udp> udp;
{
std::lock_guard<std::mutex> lock(channel_mutex_);
udp_.reset();
udp = std::move(udp_);
}
udp.reset();
mqtt_.reset();
{
@@ -57,15 +60,13 @@ MqttProtocol::~MqttProtocol() {
aes_key_id_ = PSA_KEY_ID_NULL;
}
}
if (event_group_handle_ != nullptr) {
vEventGroupDelete(event_group_handle_);
}
}
bool MqttProtocol::Start() {
return StartMqttClient(false);
}
bool MqttProtocol::Start() { return StartMqttClient(false); }
bool MqttProtocol::StartMqttClient(bool report_error) {
if (mqtt_ != nullptr) {
@@ -97,7 +98,8 @@ bool MqttProtocol::StartMqttClient(bool report_error) {
if (on_disconnected_ != nullptr) {
on_disconnected_();
}
ESP_LOGI(TAG, "MQTT disconnected, schedule reconnect in %d seconds", MQTT_RECONNECT_INTERVAL_MS / 1000);
ESP_LOGI(TAG, "MQTT disconnected, schedule reconnect in %d seconds",
MQTT_RECONNECT_INTERVAL_MS / 1000);
esp_timer_start_once(reconnect_timer_, MQTT_RECONNECT_INTERVAL_MS * 1000);
});
@@ -125,7 +127,8 @@ bool MqttProtocol::StartMqttClient(bool report_error) {
ParseServerHello(root);
} else if (strcmp(type->valuestring, "goodbye") == 0) {
auto session_id = cJSON_GetObjectItem(root, "session_id");
ESP_LOGI(TAG, "Received goodbye message, session_id: %s", cJSON_IsString(session_id) ? session_id->valuestring : "null");
ESP_LOGI(TAG, "Received goodbye message, session_id: %s",
cJSON_IsString(session_id) ? session_id->valuestring : "null");
if (cJSON_IsString(session_id) && session_id_ == session_id->valuestring) {
auto alive = alive_; // Capture alive flag
Application::GetInstance().Schedule([this, alive]() {
@@ -198,8 +201,8 @@ bool MqttProtocol::SendAudio(std::unique_ptr<AudioStreamPacket> packet) {
encrypted.resize(aes_nonce_.size() + packet->payload.size());
memcpy(encrypted.data(), nonce.data(), nonce.size());
if (!CryptAesCtr(reinterpret_cast<const uint8_t*>(packet->payload.data()), packet->payload.size(),
reinterpret_cast<const uint8_t*>(nonce.data()),
if (!CryptAesCtr(reinterpret_cast<const uint8_t*>(packet->payload.data()),
packet->payload.size(), reinterpret_cast<const uint8_t*>(nonce.data()),
reinterpret_cast<uint8_t*>(&encrypted[nonce.size()]))) {
ESP_LOGE(TAG, "Failed to encrypt audio data");
return false;
@@ -209,10 +212,12 @@ bool MqttProtocol::SendAudio(std::unique_ptr<AudioStreamPacket> packet) {
}
void MqttProtocol::CloseAudioChannel(bool send_goodbye) {
std::unique_ptr<Udp> udp;
{
std::lock_guard<std::mutex> lock(channel_mutex_);
udp_.reset();
udp = std::move(udp_);
}
udp.reset();
ESP_LOGI(TAG, "Closing audio channel, send_goodbye: %d", send_goodbye);
@@ -249,7 +254,8 @@ bool MqttProtocol::OpenAudioChannel() {
}
// 等待服务器响应
EventBits_t bits = xEventGroupWaitBits(event_group_handle_, MQTT_PROTOCOL_SERVER_HELLO_EVENT, pdTRUE, pdFALSE, pdMS_TO_TICKS(10000));
EventBits_t bits = xEventGroupWaitBits(event_group_handle_, MQTT_PROTOCOL_SERVER_HELLO_EVENT,
pdTRUE, pdFALSE, pdMS_TO_TICKS(10000));
if (!(bits & MQTT_PROTOCOL_SERVER_HELLO_EVENT)) {
ESP_LOGE(TAG, "Failed to receive server hello");
SetError(Lang::Strings::SERVER_TIMEOUT);
@@ -284,8 +290,7 @@ bool MqttProtocol::OpenAudioChannel() {
sequence = ntohl(sequence);
if (data.size() != kAudioHeaderSize + payload_len) {
ESP_LOGE(TAG, "Audio payload length mismatch: header=%u, datagram=%zu",
static_cast<unsigned>(payload_len),
data.size() - kAudioHeaderSize);
static_cast<unsigned>(payload_len), data.size() - kAudioHeaderSize);
return;
}
@@ -310,7 +315,8 @@ bool MqttProtocol::OpenAudioChannel() {
packet->frame_duration = server_frame_duration_;
packet->timestamp = timestamp;
packet->payload.resize(decrypted_size);
if (!CryptAesCtr(encrypted, decrypted_size, nonce, reinterpret_cast<uint8_t*>(packet->payload.data()))) {
if (!CryptAesCtr(encrypted, decrypted_size, nonce,
reinterpret_cast<uint8_t*>(packet->payload.data()))) {
ESP_LOGE(TAG, "Failed to decrypt audio data");
return;
}
@@ -354,6 +360,7 @@ std::string MqttProtocol::GetHelloMessage() {
#endif
cJSON_AddBoolToObject(features, "mcp", true);
cJSON_AddItemToObject(root, "features", features);
AddTextFontCapabilities(root);
cJSON* audio_params = cJSON_CreateObject();
cJSON_AddStringToObject(audio_params, "format", "opus");
cJSON_AddNumberToObject(audio_params, "sample_rate", 16000);
@@ -411,12 +418,13 @@ void MqttProtocol::ParseServerHello(const cJSON* root) {
const int udp_port = port->valueint;
// auto encryption = cJSON_GetObjectItem(udp, "encryption")->valuestring;
// ESP_LOGI(TAG, "UDP server: %s, port: %d, encryption: %s", udp_server_.c_str(), udp_port_, encryption);
// ESP_LOGI(TAG, "UDP server: %s, port: %d, encryption: %s", udp_server_.c_str(), udp_port_,
// encryption);
std::string aes_nonce;
std::string aes_key;
if (!DecodeHexString(nonce_item->valuestring, aes_nonce) ||
!DecodeHexString(key_item->valuestring, aes_key) ||
aes_nonce.size() != 16 || aes_key.size() != 16) {
!DecodeHexString(key_item->valuestring, aes_key) || aes_nonce.size() != 16 ||
aes_key.size() != 16) {
ESP_LOGE(TAG, "Invalid AES key or nonce length");
return;
}
@@ -458,9 +466,11 @@ void MqttProtocol::ParseServerHello(const cJSON* root) {
xEventGroupSetBits(event_group_handle_, MQTT_PROTOCOL_SERVER_HELLO_EVENT);
}
bool MqttProtocol::CryptAesCtr(const uint8_t* input, size_t input_size, const uint8_t* nonce, uint8_t* output) {
bool MqttProtocol::CryptAesCtr(const uint8_t* input, size_t input_size, const uint8_t* nonce,
uint8_t* output) {
std::lock_guard<std::mutex> lock(crypto_mutex_);
if (aes_key_id_ == PSA_KEY_ID_NULL || input == nullptr || nonce == nullptr || output == nullptr) {
if (aes_key_id_ == PSA_KEY_ID_NULL || input == nullptr || nonce == nullptr ||
output == nullptr) {
return false;
}
@@ -490,9 +500,12 @@ bool MqttProtocol::CryptAesCtr(const uint8_t* input, size_t input_size, const ui
}
static inline int CharToHex(char c) {
if (c >= '0' && c <= '9') return c - '0';
if (c >= 'A' && c <= 'F') return c - 'A' + 10;
if (c >= 'a' && c <= 'f') return c - 'a' + 10;
if (c >= '0' && c <= '9')
return c - '0';
if (c >= 'A' && c <= 'F')
return c - 'A' + 10;
if (c >= 'a' && c <= 'f')
return c - 'a' + 10;
return -1;
}
main/protocols/protocol.cc
+30 -11
View File
@@ -1,14 +1,34 @@
#include "protocol.h"
#include "assets.h"
#include <esp_log.h>
#define TAG "Protocol"
void Protocol::AddTextFontCapabilities(cJSON* root) {
auto capability = Assets::GetInstance().text_font_capability();
cJSON* features = cJSON_GetObjectItem(root, "features");
if (cJSON_IsObject(features)) {
cJSON_AddBoolToObject(features, "glyph_push", capability.glyph_push);
}
if (!capability.glyph_push) {
return;
}
cJSON* font = cJSON_CreateObject();
cJSON_AddStringToObject(font, "bundle", capability.bundle.c_str());
cJSON_AddStringToObject(font, "charset", capability.charset.c_str());
cJSON_AddNumberToObject(font, "size", capability.size);
cJSON_AddNumberToObject(font, "bpp", capability.bpp);
cJSON_AddItemToObject(root, "text_font", font);
}
void Protocol::OnIncomingJson(std::function<void(const cJSON* root)> callback) {
on_incoming_json_ = callback;
}
void Protocol::OnIncomingAudio(std::function<void(std::unique_ptr<AudioStreamPacket> packet)> callback) {
void Protocol::OnIncomingAudio(
std::function<void(std::unique_ptr<AudioStreamPacket> packet)> callback) {
on_incoming_audio_ = callback;
}
@@ -24,13 +44,9 @@ void Protocol::OnNetworkError(std::function<void(const std::string& message)> ca
on_network_error_ = callback;
}
void Protocol::OnConnected(std::function<void()> callback) {
on_connected_ = callback;
}
void Protocol::OnConnected(std::function<void()> callback) { on_connected_ = callback; }
void Protocol::OnDisconnected(std::function<void()> callback) {
on_disconnected_ = callback;
}
void Protocol::OnDisconnected(std::function<void()> callback) { on_disconnected_ = callback; }
void Protocol::SetError(const std::string& message) {
error_occurred_ = true;
@@ -49,8 +65,9 @@ void Protocol::SendAbortSpeaking(AbortReason reason) {
}
void Protocol::SendWakeWordDetected(const std::string& wake_word) {
std::string json = "{\"session_id\":\"" + session_id_ +
"\",\"type\":\"listen\",\"state\":\"detect\",\"text\":\"" + wake_word + "\"}";
std::string json = "{\"session_id\":\"" + session_id_ +
"\",\"type\":\"listen\",\"state\":\"detect\",\"text\":\"" + wake_word +
"\"}";
SendText(json);
}
@@ -69,12 +86,14 @@ void Protocol::SendStartListening(ListeningMode mode) {
}
void Protocol::SendStopListening() {
std::string message = "{\"session_id\":\"" + session_id_ + "\",\"type\":\"listen\",\"state\":\"stop\"}";
std::string message =
"{\"session_id\":\"" + session_id_ + "\",\"type\":\"listen\",\"state\":\"stop\"}";
SendText(message);
}
void Protocol::SendMcpMessage(const std::string& payload) {
std::string message = "{\"session_id\":\"" + session_id_ + "\",\"type\":\"mcp\",\"payload\":" + payload + "}";
std::string message =
"{\"session_id\":\"" + session_id_ + "\",\"type\":\"mcp\",\"payload\":" + payload + "}";
SendText(message);
}
main/protocols/protocol.h
+9 -18
View File
@@ -2,9 +2,9 @@
#define PROTOCOL_H
#include <cJSON.h>
#include <string>
#include <functional>
#include <chrono>
#include <functional>
#include <string>
#include <vector>
struct AudioStreamPacket {
@@ -30,30 +30,21 @@ struct BinaryProtocol3 {
uint8_t payload[];
} __attribute__((packed));
enum AbortReason {
kAbortReasonNone,
kAbortReasonWakeWordDetected
};
enum AbortReason { kAbortReasonNone, kAbortReasonWakeWordDetected };
enum ListeningMode {
kListeningModeAutoStop,
kListeningModeManualStop,
kListeningModeRealtime // 需要 AEC 支持
kListeningModeRealtime // 需要 AEC 支持
};
class Protocol {
public:
virtual ~Protocol() = default;
inline int server_sample_rate() const {
return server_sample_rate_;
}
inline int server_frame_duration() const {
return server_frame_duration_;
}
inline const std::string& session_id() const {
return session_id_;
}
inline int server_sample_rate() const { return server_sample_rate_; }
inline int server_frame_duration() const { return server_frame_duration_; }
inline const std::string& session_id() const { return session_id_; }
void OnIncomingAudio(std::function<void(std::unique_ptr<AudioStreamPacket> packet)> callback);
void OnIncomingJson(std::function<void(const cJSON* root)> callback);
@@ -92,7 +83,7 @@ protected:
virtual bool SendText(const std::string& text) = 0;
virtual void SetError(const std::string& message);
virtual bool IsTimeout() const;
static void AddTextFontCapabilities(cJSON* root);
};
#endif // PROTOCOL_H
#endif // PROTOCOL_H
main/protocols/text_glyph_payload.cc
+102
View File
@@ -0,0 +1,102 @@
#include "text_glyph_payload.h"
#include "assets.h"
#include <esp_log.h>
#include <mbedtls/base64.h>
#include <cstring>
#include <utility>
#define TAG "TextGlyphPayload"
namespace TextGlyphPayload {
bool Parse(const cJSON* root, std::vector<TextGlyph>& result, uint8_t& bpp) {
auto payload = cJSON_GetObjectItem(root, "glyph_push");
if (!cJSON_IsObject(payload)) {
return true;
}
auto version = cJSON_GetObjectItem(payload, "v");
auto bundle = cJSON_GetObjectItem(payload, "bundle");
auto size = cJSON_GetObjectItem(payload, "size");
auto wire_bpp = cJSON_GetObjectItem(payload, "bpp");
auto glyphs = cJSON_GetObjectItem(payload, "glyphs");
auto capability = Assets::GetInstance().text_font_capability();
if (!capability.glyph_push || !cJSON_IsNumber(version) || version->valueint != 1 ||
!cJSON_IsString(bundle) || capability.bundle != bundle->valuestring ||
!cJSON_IsNumber(size) || size->valuedouble != capability.size ||
!cJSON_IsNumber(wire_bpp) || wire_bpp->valuedouble != capability.bpp ||
!cJSON_IsArray(glyphs) || cJSON_GetArraySize(glyphs) > 64) {
ESP_LOGW(TAG, "Rejected incompatible glyph payload");
return false;
}
const uint8_t parsed_bpp = static_cast<uint8_t>(wire_bpp->valueint);
std::vector<TextGlyph> parsed;
parsed.reserve(cJSON_GetArraySize(glyphs));
size_t total_bitmap_bytes = 0;
cJSON* item = nullptr;
cJSON_ArrayForEach (item, glyphs) {
auto codepoint = cJSON_GetObjectItem(item, "codepoint");
auto adv_w = cJSON_GetObjectItem(item, "adv_w");
auto box_w = cJSON_GetObjectItem(item, "box_w");
auto box_h = cJSON_GetObjectItem(item, "box_h");
auto ofs_x = cJSON_GetObjectItem(item, "ofs_x");
auto ofs_y = cJSON_GetObjectItem(item, "ofs_y");
auto bitmap = cJSON_GetObjectItem(item, "bitmap");
if (!cJSON_IsNumber(codepoint) || codepoint->valuedouble <= 0 ||
codepoint->valuedouble > 0x10FFFF || !cJSON_IsNumber(adv_w) || adv_w->valuedouble < 0 ||
adv_w->valuedouble > UINT32_MAX || !cJSON_IsNumber(box_w) || box_w->valueint < 0 ||
box_w->valueint > 64 || !cJSON_IsNumber(box_h) || box_h->valueint < 0 ||
box_h->valueint > 64 || !cJSON_IsNumber(ofs_x) || ofs_x->valueint < INT16_MIN ||
ofs_x->valueint > INT16_MAX || !cJSON_IsNumber(ofs_y) || ofs_y->valueint < INT16_MIN ||
ofs_y->valueint > INT16_MAX || !cJSON_IsString(bitmap)) {
ESP_LOGW(TAG, "Rejected malformed glyph");
return false;
}
const size_t expected =
(static_cast<size_t>(box_w->valueint) * box_h->valueint * parsed_bpp + 7) / 8;
total_bitmap_bytes += expected;
if (total_bitmap_bytes > 64 * 1024) {
ESP_LOGW(TAG, "Rejected oversized glyph payload");
return false;
}
TextGlyph glyph;
glyph.codepoint = static_cast<uint32_t>(codepoint->valuedouble);
glyph.adv_w = static_cast<uint32_t>(adv_w->valuedouble);
glyph.box_w = static_cast<uint16_t>(box_w->valueint);
glyph.box_h = static_cast<uint16_t>(box_h->valueint);
glyph.ofs_x = static_cast<int16_t>(ofs_x->valueint);
glyph.ofs_y = static_cast<int16_t>(ofs_y->valueint);
glyph.bitmap.resize(expected);
const size_t encoded_length = strlen(bitmap->valuestring);
if (encoded_length > ((expected + 2) / 3) * 4 + 4) {
ESP_LOGW(TAG, "Rejected oversized base64 bitmap");
return false;
}
if (expected == 0) {
if (encoded_length != 0) {
return false;
}
parsed.push_back(std::move(glyph));
continue;
}
size_t decoded = 0;
int rc = mbedtls_base64_decode(glyph.bitmap.data(), glyph.bitmap.size(), &decoded,
reinterpret_cast<const unsigned char*>(bitmap->valuestring),
encoded_length);
if (rc != 0 || decoded != expected) {
ESP_LOGW(TAG, "Rejected invalid bitmap for U+%04lX",
static_cast<unsigned long>(glyph.codepoint));
return false;
}
parsed.push_back(std::move(glyph));
}
result = std::move(parsed);
bpp = parsed_bpp;
return true;
}
} // namespace TextGlyphPayload
main/protocols/text_glyph_payload.h
+14
View File
@@ -0,0 +1,14 @@
#pragma once
#include "display/text_glyph.h"
#include <cJSON.h>
#include <cstdint>
#include <vector>
namespace TextGlyphPayload {
bool Parse(const cJSON* root, std::vector<TextGlyph>& result, uint8_t& bpp);
} // namespace TextGlyphPayload
main/protocols/websocket_protocol.cc
+13 -17
View File
@@ -1,24 +1,20 @@
#include "websocket_protocol.h"
#include "board.h"
#include "system_info.h"
#include "application.h"
#include "board.h"
#include "settings.h"
#include "system_info.h"
#include <cstring>
#include <cJSON.h>
#include <esp_log.h>
#include <arpa/inet.h>
#include <cJSON.h>
#include <cstring>
#include "assets/lang_config.h"
#define TAG "WS"
WebsocketProtocol::WebsocketProtocol() {
event_group_handle_ = xEventGroupCreate();
}
WebsocketProtocol::WebsocketProtocol() { event_group_handle_ = xEventGroupCreate(); }
WebsocketProtocol::~WebsocketProtocol() {
vEventGroupDelete(event_group_handle_);
}
WebsocketProtocol::~WebsocketProtocol() { vEventGroupDelete(event_group_handle_); }
bool WebsocketProtocol::Start() {
// Only connect to server when audio channel is needed
@@ -123,8 +119,7 @@ bool WebsocketProtocol::OpenAudioChannel() {
.sample_rate = server_sample_rate_,
.frame_duration = server_frame_duration_,
.timestamp = bp2->timestamp,
.payload = std::vector<uint8_t>(payload, payload + bp2->payload_size)
}));
.payload = std::vector<uint8_t>(payload, payload + bp2->payload_size)}));
} else if (version_ == 3) {
BinaryProtocol3* bp3 = (BinaryProtocol3*)data;
bp3->type = bp3->type;
@@ -134,15 +129,13 @@ bool WebsocketProtocol::OpenAudioChannel() {
.sample_rate = server_sample_rate_,
.frame_duration = server_frame_duration_,
.timestamp = 0,
.payload = std::vector<uint8_t>(payload, payload + bp3->payload_size)
}));
.payload = std::vector<uint8_t>(payload, payload + bp3->payload_size)}));
} else {
on_incoming_audio_(std::make_unique<AudioStreamPacket>(AudioStreamPacket{
.sample_rate = server_sample_rate_,
.frame_duration = server_frame_duration_,
.timestamp = 0,
.payload = std::vector<uint8_t>((uint8_t*)data, (uint8_t*)data + len)
}));
.payload = std::vector<uint8_t>((uint8_t*)data, (uint8_t*)data + len)}));
}
}
} else {
@@ -186,7 +179,9 @@ bool WebsocketProtocol::OpenAudioChannel() {
}
// Wait for server hello
EventBits_t bits = xEventGroupWaitBits(event_group_handle_, WEBSOCKET_PROTOCOL_SERVER_HELLO_EVENT, pdTRUE, pdFALSE, pdMS_TO_TICKS(10000));
EventBits_t bits =
xEventGroupWaitBits(event_group_handle_, WEBSOCKET_PROTOCOL_SERVER_HELLO_EVENT, pdTRUE,
pdFALSE, pdMS_TO_TICKS(10000));
if (!(bits & WEBSOCKET_PROTOCOL_SERVER_HELLO_EVENT)) {
ESP_LOGE(TAG, "Failed to receive server hello");
SetError(Lang::Strings::SERVER_TIMEOUT);
@@ -211,6 +206,7 @@ std::string WebsocketProtocol::GetHelloMessage() {
#endif
cJSON_AddBoolToObject(features, "mcp", true);
cJSON_AddItemToObject(root, "features", features);
AddTextFontCapabilities(root);
cJSON_AddStringToObject(root, "transport", "websocket");
cJSON* audio_params = cJSON_CreateObject();
cJSON_AddStringToObject(audio_params, "format", "opus");
scripts/build_default_assets.py
+50 -27
View File
@@ -13,6 +13,7 @@ Usage:
import argparse
import io
import os
import re
import shutil
import sys
import json
@@ -273,7 +274,8 @@ def process_extra_files(extra_files_dir, assets_dir):
return extra_files_list
def generate_index_json(assets_dir, srmodels, text_font, emoji_collection, extra_files=None, multinet_model_info=None):
def generate_index_json(assets_dir, srmodels, text_font, emoji_collection, extra_files=None,
multinet_model_info=None, font_bundle_id=None):
"""Generate index.json file"""
index_data = {
"version": 1
@@ -283,7 +285,22 @@ def generate_index_json(assets_dir, srmodels, text_font, emoji_collection, extra
index_data["srmodels"] = srmodels
if text_font:
if not font_bundle_id:
raise ValueError("font_bundle_id is required when packaging a text font")
index_data["text_font"] = text_font
stem = os.path.splitext(text_font)[0]
marker = '_common_'
if marker in stem:
match = re.fullmatch(r"(\d+)_(\d+)", stem.split(marker, 1)[1])
if not match:
raise ValueError(f"invalid Noto text font name: {text_font}")
text_font_meta = {
"charset": "common",
"size": int(match.group(1)),
"bpp": int(match.group(2)),
"bundle": font_bundle_id,
}
index_data["text_font_meta"] = text_font_meta
if emoji_collection:
index_data["emoji_collection"] = emoji_collection
@@ -663,7 +680,7 @@ def get_multinet_model_paths(model_names, esp_sr_model_path):
return valid_paths
def get_text_font_path(builtin_text_font, xiaozhi_fonts_path):
def get_text_font_path(builtin_text_font, noto_fonts_path):
"""
Get the text font path if needed
Returns the font file path or None if no font is needed
@@ -671,13 +688,9 @@ def get_text_font_path(builtin_text_font, xiaozhi_fonts_path):
if not builtin_text_font or 'basic' not in builtin_text_font:
return None
# Convert from basic to common font name
# e.g., font_puhui_basic_16_4 -> font_puhui_common_16_4.bin
if builtin_text_font.startswith('font_noto_'):
font_name = builtin_text_font.replace('basic', 'qwen') + '.bin'
else:
font_name = builtin_text_font.replace('basic', 'common') + '.bin'
font_path = os.path.join(xiaozhi_fonts_path, 'cbin', font_name)
# basic is linked into firmware; common is loaded from the assets partition.
font_name = builtin_text_font.replace('_basic_', '_common_') + '.bin'
font_path = os.path.join(noto_fonts_path, 'cbin', font_name)
if os.path.exists(font_path):
return font_path
@@ -686,14 +699,13 @@ def get_text_font_path(builtin_text_font, xiaozhi_fonts_path):
return None
def get_emoji_collection_path(default_emoji_collection, xiaozhi_fonts_path, project_root=None):
def get_emoji_collection_path(default_emoji_collection, noto_fonts_path, project_root=None):
"""
Get the emoji collection path if needed
Returns the emoji directory path or None if no emoji collection is needed
Supports:
- PNG emoji collections from xiaozhi-fonts (e.g., emojis_32, twemoji_64)
- GIF emoji collections from xiaozhi-fonts (e.g., noto-emoji_128, noto-emoji_64)
- PNG emoji collections from noto-fonts (e.g., noto-color-emoji_32)
- Otto GIF emoji collection (otto-gif)
"""
if not default_emoji_collection:
@@ -713,13 +725,13 @@ def get_emoji_collection_path(default_emoji_collection, xiaozhi_fonts_path, proj
print("Warning: project_root not provided, cannot locate otto-gif collection")
return None
# Try PNG emoji collections first (e.g., emojis_32, twemoji_64)
emoji_path = os.path.join(xiaozhi_fonts_path, 'png', default_emoji_collection)
# Try PNG emoji collections first.
emoji_path = os.path.join(noto_fonts_path, 'png', default_emoji_collection)
if os.path.exists(emoji_path):
return emoji_path
# Try GIF emoji collections (e.g., noto-emoji_128, noto-emoji_64, noto-emoji_32)
emoji_path = os.path.join(xiaozhi_fonts_path, 'gif', default_emoji_collection)
emoji_path = os.path.join(noto_fonts_path, 'gif', default_emoji_collection)
if os.path.exists(emoji_path):
return emoji_path
@@ -727,7 +739,9 @@ def get_emoji_collection_path(default_emoji_collection, xiaozhi_fonts_path, proj
return None
def build_assets_integrated(wakenet_model_paths, multinet_model_paths, text_font_path, emoji_collection_path, extra_files_path, output_path, multinet_model_info=None):
def build_assets_integrated(wakenet_model_paths, multinet_model_paths, text_font_path,
emoji_collection_path, extra_files_path, output_path,
multinet_model_info=None, font_bundle_id=None):
"""
Build assets using integrated functions (no external dependencies)
"""
@@ -751,7 +765,8 @@ def build_assets_integrated(wakenet_model_paths, multinet_model_paths, text_font
extra_files = process_extra_files(extra_files_path, assets_dir) if extra_files_path else None
# Generate index.json
generate_index_json(assets_dir, srmodels, text_font, emoji_collection, extra_files, multinet_model_info)
generate_index_json(assets_dir, srmodels, text_font, emoji_collection, extra_files,
multinet_model_info, font_bundle_id)
# Generate config.json for packing
config_path = generate_config_json(temp_build_dir, assets_dir)
@@ -791,17 +806,17 @@ def build_assets_integrated(wakenet_model_paths, multinet_model_paths, text_font
def main():
parser = argparse.ArgumentParser(description='Build default assets based on configuration')
parser.add_argument('--sdkconfig', required=True, help='Path to sdkconfig file')
parser.add_argument('--builtin_text_font', help='Builtin text font name (e.g., font_puhui_basic_16_4)')
parser.add_argument('--emoji_collection', help='Default emoji collection name (e.g., emojis_32)')
parser.add_argument('--builtin_text_font', help='Builtin text font name (e.g., font_noto_sans_basic_16_4)')
parser.add_argument('--emoji_collection', help='Default emoji collection name (e.g., noto-color-emoji_32)')
parser.add_argument('--output', required=True, help='Output path for assets.bin')
parser.add_argument('--esp_sr_model_path', help='Path to ESP-SR model directory')
parser.add_argument('--xiaozhi_fonts_path', help='Path to xiaozhi-fonts component directory')
parser.add_argument('--noto_fonts_path', help='Path to noto-fonts component directory')
parser.add_argument('--extra_files', help='Path to extra files directory to be included in assets')
args = parser.parse_args()
# Set default paths if not provided
if not args.esp_sr_model_path or not args.xiaozhi_fonts_path:
if not args.esp_sr_model_path or not args.noto_fonts_path:
# Calculate project root from script location
script_dir = os.path.dirname(os.path.abspath(__file__))
project_root = os.path.dirname(script_dir)
@@ -809,8 +824,8 @@ def main():
if not args.esp_sr_model_path:
args.esp_sr_model_path = os.path.join(project_root, "managed_components", "espressif__esp-sr", "model")
if not args.xiaozhi_fonts_path:
args.xiaozhi_fonts_path = os.path.join(project_root, "components", "xiaozhi-fonts")
if not args.noto_fonts_path:
args.noto_fonts_path = os.path.join(project_root, "components", "noto-fonts")
print("Building default assets...")
print(f" sdkconfig: {args.sdkconfig}")
@@ -854,13 +869,20 @@ def main():
print(f" multinet models: {', '.join(multinet_model_names)} (will be packaged)")
# Get text font path if needed
text_font_path = get_text_font_path(args.builtin_text_font, args.xiaozhi_fonts_path)
text_font_path = get_text_font_path(args.builtin_text_font, args.noto_fonts_path)
font_bundle_id = None
if text_font_path:
manifest_path = os.path.join(args.noto_fonts_path, "manifest.json")
with open(manifest_path, 'r', encoding='utf-8') as f:
font_bundle_id = json.load(f).get("bundle_id")
if not isinstance(font_bundle_id, str) or not font_bundle_id:
raise ValueError("noto-fonts manifest.json must define bundle_id")
# Get emoji collection path if needed
# Calculate project root from script location for otto-gif support
script_dir = os.path.dirname(os.path.abspath(__file__))
project_root = os.path.dirname(script_dir)
emoji_collection_path = get_emoji_collection_path(args.emoji_collection, args.xiaozhi_fonts_path, project_root)
emoji_collection_path = get_emoji_collection_path(args.emoji_collection, args.noto_fonts_path, project_root)
# Get extra files path if provided
extra_files_path = args.extra_files
@@ -901,8 +923,9 @@ def main():
return
# Build the assets
success = build_assets_integrated(wakenet_model_paths, multinet_model_paths, text_font_path, emoji_collection_path,
extra_files_path, args.output, multinet_model_info)
success = build_assets_integrated(
wakenet_model_paths, multinet_model_paths, text_font_path, emoji_collection_path,
extra_files_path, args.output, multinet_model_info, font_bundle_id)
if not success:
sys.exit(1)
scripts/spiffs_assets/README.md
+4 -4
View File
@@ -39,14 +39,14 @@
# 完整参数示例
./build.py \
--wakenet_model ../../managed_components/espressif__esp-sr/model/wakenet_model/wn9_nihaoxiaozhi_tts \
--text_font ../../components/xiaozhi-fonts/build/font_puhui_common_20_4.bin \
--emoji_collection ../../components/xiaozhi-fonts/build/emojis_64/
--text_font ../../components/noto-fonts/cbin/font_noto_sans_common_20_4.bin \
--emoji_collection ../../components/noto-fonts/png/noto-color-emoji_64/
# 仅处理字体文件
./build.py --text_font ../../components/xiaozhi-fonts/build/font_puhui_common_20_4.bin
./build.py --text_font ../../components/noto-fonts/cbin/font_noto_sans_common_20_4.bin
# 仅处理表情符号
./build.py --emoji_collection ../../components/xiaozhi-fonts/build/emojis_64/
./build.py --emoji_collection ../../components/noto-fonts/png/noto-color-emoji_64/
```
## 工作流程
scripts/spiffs_assets/build.py
+3 -3
View File
@@ -9,8 +9,8 @@ Usage:
Example:
./build.py --wakenet_model ../../managed_components/espressif__esp-sr/model/wakenet_model/wn9_nihaoxiaozhi_tts \
--text_font ../../components/xiaozhi-fonts/build/font_puhui_common_20_4.bin \
--emoji_collection ../../components/xiaozhi-fonts/build/emojis_64/
--text_font ../../components/noto-fonts/cbin/font_noto_sans_common_20_4.bin \
--emoji_collection ../../components/noto-fonts/png/noto-color-emoji_64/
"""
import os
@@ -397,4 +397,4 @@ def main():
if __name__ == "__main__":
main()
main()
scripts/spiffs_assets/build_all.py
+9 -10
View File
@@ -8,7 +8,7 @@ This script calls build.py with different combinations of:
- emoji_collections
And generates assets.bin files with names like:
wn9_nihaoxiaozhi_tts-font_puhui_common_20_4-emojis_32.bin
wn9_nihaoxiaozhi_tts-font_noto_sans_common_20_4-noto-color-emoji_32.bin
"""
import os
@@ -40,11 +40,11 @@ def build_assets(wakenet_model, text_font, emoji_collection, build_dir, final_di
cmd.extend(["--wakenet_model", wakenet_path])
if text_font != "none":
text_font_path = os.path.join("../../components/78__xiaozhi-fonts/cbin", f"{text_font}.bin")
text_font_path = os.path.join("../../components/noto-fonts/cbin", f"{text_font}.bin")
cmd.extend(["--text_font", text_font_path])
if emoji_collection != "none":
emoji_path = os.path.join("../../components/xiaozhi-fonts/build", emoji_collection)
emoji_path = os.path.join("../../components/noto-fonts/png", emoji_collection)
cmd.extend(["--emoji_collection", emoji_path])
print(f"\n正在构建: {wakenet_model}-{text_font}-{emoji_collection}")
@@ -87,16 +87,16 @@ def main():
text_fonts = [
"none",
"font_puhui_common_14_1",
"font_puhui_common_16_4",
"font_puhui_common_20_4",
"font_puhui_common_30_4",
"font_noto_sans_common_14_1",
"font_noto_sans_common_16_4",
"font_noto_sans_common_20_4",
"font_noto_sans_common_30_4",
]
emoji_collections = [
"none",
"emojis_32",
"emojis_64",
"noto-color-emoji_32",
"noto-color-emoji_64",
]
# Get script directory
@@ -145,4 +145,3 @@ def main():
if __name__ == "__main__":
main()
scripts/tests/test_build_default_assets.py
+42
View File
@@ -0,0 +1,42 @@
import importlib.util
import json
import tempfile
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
SPEC = importlib.util.spec_from_file_location(
"build_default_assets", ROOT / "scripts" / "build_default_assets.py"
)
BUILD = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(BUILD)
class BuildDefaultAssetsTest(unittest.TestCase):
def test_text_font_metadata_uses_bundle_charset_size_and_bpp(self):
with tempfile.TemporaryDirectory() as directory:
assets = Path(directory)
BUILD.generate_index_json(
str(assets),
None,
"font_noto_sans_common_20_4.bin",
None,
font_bundle_id="noto-v1",
)
index = json.loads((assets / "index.json").read_text(encoding="utf-8"))
self.assertEqual(
index["text_font_meta"],
{"charset": "common", "size": 20, "bpp": 4, "bundle": "noto-v1"},
)
def test_text_font_requires_bundle(self):
with tempfile.TemporaryDirectory() as directory:
with self.assertRaises(ValueError):
BUILD.generate_index_json(
directory, None, "font_noto_sans_common_20_4.bin", None
)
if __name__ == "__main__":
unittest.main()