mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-06-10 01:15:58 +00:00
feat: MiniMax provider for image/video/podcast skills + new music-generation skill (#3437)
* docs(spec): MiniMax integration for generation skills + new music skill Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(plan): MiniMax generation providers implementation plan Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(skills): add importlib loader + FakeResp for skill tests * test(skills): register loaded module in sys.modules; raise requests.HTTPError in FakeResp * feat(image-generation): add MiniMax provider with env auto-detect Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(image-generation): guard unknown provider, derive ref MIME, strengthen tests Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(video-generation): add MiniMax provider with async poll/download Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(video-generation): surface base_resp errors while polling; add timeout test * feat(podcast-generation): add MiniMax t2a_v2 provider with env auto-detect Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(podcast-generation): restore TTS credential guard; add volcengine + voice tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(music-generation): new MiniMax music skill via skill-creator Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(music-generation): treat empty lyrics as absent; test no-audio-data path * refactor(skills): add request timeouts to MiniMax network calls Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Potential fix for pull request finding 'Explicit returns mixed with implicit (fall through) returns' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * fix(models): strip inconsistent user-message names for MiniMax chat DeerFlow middlewares tag user messages with provenance names (user-input, summary, loop_warning); langchain serializes them into the OpenAI-compatible payload and MiniMax rejects mismatched user-message names with "user name must be consistent (2013)". PatchedChatMiniMax now drops the per-message name from user-role messages. Point the config.example MiniMax models at PatchedChatMiniMax so they also get reasoning_content mapping. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(image-generation): MiniMax sends JSON prompt field, guard 1500-char limit MiniMax image-01 takes one text string capped at 1500 chars, but the skill was sending the whole structured JSON. The MiniMax provider now extracts the JSON `prompt` field (relying on prompt_optimizer to expand it) and fails fast with a clear error before calling the API when that field exceeds 1500 chars. Authoring stays provider-agnostic; Gemini still receives the full JSON. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(podcast-generation): per-provider TTS concurrency and retry/backoff Each TTS provider owns its concurrency internally — MiniMax runs single-threaded to reduce rate-limit failures, Volcengine keeps 4 workers — with automatic retry and backoff on transient HTTP and base_resp errors. No caller-facing concurrency knob. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(skills): address Copilot review comments on generation skills - video: add raise_for_status + timeout to the Gemini download/POST/poll calls so non-2xx responses surface as clear HTTP errors instead of JSON/KeyError or hangs - video: check the task Fail status before the generic base_resp check so the failure keeps its task_id context - video/image: create the output file parent directory before writing (matching music-generation) so nested output paths do not raise FileNotFoundError - music: require a non-empty prompt and fail fast with ValueError instead of sending an empty prompt to the API Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(scripts): reclaim dev ports across worktrees in make stop/dev All deer-flow worktrees (main checkout + linked worktrees) hardcode the same dev ports (8001/3000/2026), so a service started from any worktree must be reclaimable from another. stop_all now resolves the set of worktree roots (DEERFLOW_ROOTS) and treats a process as deer-flow-owned when its open files live under any of them. It also force-kills survivors on 2026 alongside 8001/3000, fixing `make dev` aborting on the nginx port preflight when a prior nginx lingered on 2026. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(view-image): hide the injected image-context message from the UI ViewImageMiddleware injects a HumanMessage (text + base64 images) so the vision model can see viewed images, but it was the only internal injector that set neither hide_from_ui nor a hidden name, so it leaked into the chat UI (and IM channels) as a user bubble reading "Here are the images you've viewed:". Mark it with additional_kwargs={"hide_from_ui": True}, matching todo/dynamic_context injections, which the frontend isHiddenFromUIMessage and the channel sender already honor. The model still receives the full content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(minimax): mark M2.7 models as text-only (no vision) MiniMax M2.7 / M2.7-highspeed do not support vision; only M3 does. The provider config asserted vision support for M2.7 in four places. - config.example.yaml: 4 M2.7 entries -> supports_vision: false - backend/docs/CONFIGURATION.md: M2.7 + highspeed -> supports_vision: false - wizard: add LLMProvider.model_vision_overrides + extra_config_for() so selecting an M2.7 model writes supports_vision: false while M3 (default) keeps vision; wire it through setup_wizard.py - tests: M2.7-highspeed fixture -> supports_vision=False; add test_minimax_vision_is_per_model Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
This commit is contained in:
DanielWalnut
+66
-10
@@ -62,9 +62,56 @@ done
|
||||
|
||||
# ── Stop helper ──────────────────────────────────────────────────────────────
|
||||
|
||||
_is_repo_pid() {
|
||||
local pid=$1
|
||||
lsof -p "$pid" 2>/dev/null | grep -F "$REPO_ROOT" >/dev/null
|
||||
# Every deer-flow worktree (the main checkout + each linked worktree) hardcodes
|
||||
# the same dev ports (8001/3000/2026), so a service started from ANY of them
|
||||
# must be reclaimable from here — otherwise `make stop`/`make dev` in this
|
||||
# worktree can neither kill nor take over a port held by a sibling worktree.
|
||||
# DEERFLOW_ROOTS is that set of roots; processes living outside all of them
|
||||
# (e.g. an unrelated project on port 3000) are still never touched.
|
||||
# Sorted most-specific-first (longest path first): a linked worktree lives
|
||||
# under the main checkout, so both roots are substrings of its files — checking
|
||||
# the deeper root first attributes a reclaimed port to the right worktree.
|
||||
DEERFLOW_ROOTS="$(
|
||||
{
|
||||
printf '%s\n' "$REPO_ROOT"
|
||||
git -C "$REPO_ROOT" worktree list --porcelain 2>/dev/null |
|
||||
awk '/^worktree /{print $2}'
|
||||
} | awk 'NF && !seen[$0]++ {print length($0)"\t"$0}' | sort -rn | sed 's/^[0-9]*\t//'
|
||||
)"
|
||||
|
||||
# True if PID has an open file/cwd under any deer-flow worktree root. The
|
||||
# trailing slash keeps a sibling dir like ".../deer-flow-notes" from matching
|
||||
# the ".../deer-flow" root.
|
||||
_is_deerflow_pid() {
|
||||
local pid=$1 files root
|
||||
files=$(lsof -p "$pid" 2>/dev/null) || return 1
|
||||
while IFS= read -r root; do
|
||||
[ -n "$root" ] || continue
|
||||
case "$files" in
|
||||
*"$root"/*) return 0 ;;
|
||||
esac
|
||||
done <<< "$DEERFLOW_ROOTS"
|
||||
return 1
|
||||
}
|
||||
|
||||
# Report ports about to be reclaimed from a *different* worktree, so stopping
|
||||
# (or starting, which stops first) isn't silently killing someone else's run.
|
||||
_report_reclaimed_ports() {
|
||||
local port pid files root owner
|
||||
for port in 8001 3000 2026; do
|
||||
for pid in $(lsof -nP -iTCP:"$port" -sTCP:LISTEN -t 2>/dev/null); do
|
||||
_is_deerflow_pid "$pid" || continue
|
||||
files=$(lsof -p "$pid" 2>/dev/null)
|
||||
case "$files" in *"$REPO_ROOT"/*) continue ;; esac # this worktree — normal
|
||||
owner=""
|
||||
while IFS= read -r root; do
|
||||
[ -n "$root" ] || continue
|
||||
case "$files" in *"$root"/*) owner="$root"; break ;; esac
|
||||
done <<< "$DEERFLOW_ROOTS"
|
||||
echo " ↻ Reclaiming port $port from another worktree: ${owner:-?}"
|
||||
break
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
_kill_repo_processes() {
|
||||
@@ -73,7 +120,7 @@ _kill_repo_processes() {
|
||||
local pids=""
|
||||
|
||||
while IFS= read -r pid; do
|
||||
if [ -n "$pid" ] && _is_repo_pid "$pid"; then
|
||||
if [ -n "$pid" ] && _is_deerflow_pid "$pid"; then
|
||||
case " $pids " in
|
||||
*" $pid "*) ;;
|
||||
*) pids="$pids $pid" ;;
|
||||
@@ -92,7 +139,7 @@ _kill_repo_port() {
|
||||
local pids=""
|
||||
|
||||
while IFS= read -r pid; do
|
||||
if [ -n "$pid" ] && _is_repo_pid "$pid"; then
|
||||
if [ -n "$pid" ] && _is_deerflow_pid "$pid"; then
|
||||
case " $pids " in
|
||||
*" $pid "*) ;;
|
||||
*) pids="$pids $pid" ;;
|
||||
@@ -141,11 +188,15 @@ _is_repo_nginx_pid() {
|
||||
esac
|
||||
|
||||
args=$(ps -p "$pid" -o args= 2>/dev/null) || return 1
|
||||
case "$args" in
|
||||
*"$REPO_ROOT/docker/nginx/nginx.local.conf"*|*"$REPO_ROOT"*) return 0 ;;
|
||||
esac
|
||||
local root
|
||||
while IFS= read -r root; do
|
||||
[ -n "$root" ] || continue
|
||||
case "$args" in
|
||||
*"$root"/docker/nginx/nginx.local.conf*|*"$root"/*) return 0 ;;
|
||||
esac
|
||||
done <<< "$DEERFLOW_ROOTS"
|
||||
|
||||
_is_repo_pid "$pid"
|
||||
_is_deerflow_pid "$pid"
|
||||
}
|
||||
|
||||
_kill_repo_nginx() {
|
||||
@@ -175,6 +226,7 @@ _kill_repo_nginx() {
|
||||
|
||||
stop_all() {
|
||||
echo "Stopping all services..."
|
||||
_report_reclaimed_ports
|
||||
_kill_repo_processes "uvicorn app.gateway.app:app"
|
||||
_kill_repo_processes "next dev"
|
||||
_kill_repo_processes "next start"
|
||||
@@ -182,9 +234,13 @@ stop_all() {
|
||||
nginx -c "$REPO_ROOT/docker/nginx/nginx.local.conf" -p "$REPO_ROOT" -s quit 2>/dev/null || true
|
||||
sleep 1
|
||||
_kill_repo_nginx
|
||||
# Force-kill any survivors still holding the service ports
|
||||
# Force-kill any survivors still holding the service ports. 2026 is included
|
||||
# so a lingering nginx (or any deer-flow process) that _kill_repo_nginx did
|
||||
# not match by name still gets reclaimed — otherwise `make dev` fails its
|
||||
# nginx port preflight.
|
||||
_kill_repo_port 8001
|
||||
_kill_repo_port 3000
|
||||
_kill_repo_port 2026
|
||||
./scripts/cleanup-containers.sh deer-flow-sandbox 2>/dev/null || true
|
||||
echo "✓ All services stopped"
|
||||
}
|
||||
|
||||
@@ -85,7 +85,7 @@ def main() -> int:
|
||||
display_name=f"{llm.provider.display_name} / {llm.model_name}",
|
||||
api_key_field=llm.provider.api_key_field,
|
||||
env_var=llm.provider.env_var,
|
||||
extra_model_config=llm.provider.extra_config or None,
|
||||
extra_model_config=llm.provider.extra_config_for(llm.model_name) or None,
|
||||
base_url=llm.base_url,
|
||||
search_use=search_provider.use if search_provider else None,
|
||||
search_tool_name=search_provider.tool_name if search_provider else "web_search",
|
||||
|
||||
@@ -19,10 +19,24 @@ class LLMProvider:
|
||||
api_key_field: str = "api_key"
|
||||
# Extra config fields beyond the common ones (merged into YAML)
|
||||
extra_config: dict = field(default_factory=dict)
|
||||
# Per-model supports_vision overrides for providers whose models differ in
|
||||
# capability (e.g. MiniMax M3 supports vision but M2.7 is text-only). The
|
||||
# provider-level extra_config holds the default (default_model) capability.
|
||||
model_vision_overrides: dict[str, bool] = field(default_factory=dict)
|
||||
auth_hint: str | None = None
|
||||
base_url_prompt: str | None = None
|
||||
model_prompt: str | None = None
|
||||
|
||||
def extra_config_for(self, model_name: str) -> dict:
|
||||
"""Return extra_config for a selected model, applying per-model overrides.
|
||||
|
||||
Does not mutate the shared provider-level ``extra_config``.
|
||||
"""
|
||||
config = dict(self.extra_config)
|
||||
if model_name in self.model_vision_overrides:
|
||||
config["supports_vision"] = self.model_vision_overrides[model_name]
|
||||
return config
|
||||
|
||||
|
||||
@dataclass
|
||||
class WebProvider:
|
||||
@@ -313,6 +327,10 @@ LLM_PROVIDERS: list[LLMProvider] = [
|
||||
"supports_vision": True,
|
||||
"supports_thinking": True,
|
||||
},
|
||||
model_vision_overrides={
|
||||
"MiniMax-M2.7": False,
|
||||
"MiniMax-M2.7-highspeed": False,
|
||||
},
|
||||
),
|
||||
LLMProvider(
|
||||
name="minimax_cn",
|
||||
@@ -332,6 +350,10 @@ LLM_PROVIDERS: list[LLMProvider] = [
|
||||
"supports_vision": True,
|
||||
"supports_thinking": True,
|
||||
},
|
||||
model_vision_overrides={
|
||||
"MiniMax-M2.7": False,
|
||||
"MiniMax-M2.7-highspeed": False,
|
||||
},
|
||||
),
|
||||
LLMProvider(
|
||||
name="openrouter",
|
||||
|
||||
Reference in New Issue
Block a user