mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-06-10 09:25:57 +00:00
feat: MiniMax provider for image/video/podcast skills + new music-generation skill (#3437)
* docs(spec): MiniMax integration for generation skills + new music skill Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(plan): MiniMax generation providers implementation plan Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(skills): add importlib loader + FakeResp for skill tests * test(skills): register loaded module in sys.modules; raise requests.HTTPError in FakeResp * feat(image-generation): add MiniMax provider with env auto-detect Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(image-generation): guard unknown provider, derive ref MIME, strengthen tests Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(video-generation): add MiniMax provider with async poll/download Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(video-generation): surface base_resp errors while polling; add timeout test * feat(podcast-generation): add MiniMax t2a_v2 provider with env auto-detect Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(podcast-generation): restore TTS credential guard; add volcengine + voice tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(music-generation): new MiniMax music skill via skill-creator Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(music-generation): treat empty lyrics as absent; test no-audio-data path * refactor(skills): add request timeouts to MiniMax network calls Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Potential fix for pull request finding 'Explicit returns mixed with implicit (fall through) returns' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * fix(models): strip inconsistent user-message names for MiniMax chat DeerFlow middlewares tag user messages with provenance names (user-input, summary, loop_warning); langchain serializes them into the OpenAI-compatible payload and MiniMax rejects mismatched user-message names with "user name must be consistent (2013)". PatchedChatMiniMax now drops the per-message name from user-role messages. Point the config.example MiniMax models at PatchedChatMiniMax so they also get reasoning_content mapping. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(image-generation): MiniMax sends JSON prompt field, guard 1500-char limit MiniMax image-01 takes one text string capped at 1500 chars, but the skill was sending the whole structured JSON. The MiniMax provider now extracts the JSON `prompt` field (relying on prompt_optimizer to expand it) and fails fast with a clear error before calling the API when that field exceeds 1500 chars. Authoring stays provider-agnostic; Gemini still receives the full JSON. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(podcast-generation): per-provider TTS concurrency and retry/backoff Each TTS provider owns its concurrency internally — MiniMax runs single-threaded to reduce rate-limit failures, Volcengine keeps 4 workers — with automatic retry and backoff on transient HTTP and base_resp errors. No caller-facing concurrency knob. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(skills): address Copilot review comments on generation skills - video: add raise_for_status + timeout to the Gemini download/POST/poll calls so non-2xx responses surface as clear HTTP errors instead of JSON/KeyError or hangs - video: check the task Fail status before the generic base_resp check so the failure keeps its task_id context - video/image: create the output file parent directory before writing (matching music-generation) so nested output paths do not raise FileNotFoundError - music: require a non-empty prompt and fail fast with ValueError instead of sending an empty prompt to the API Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(scripts): reclaim dev ports across worktrees in make stop/dev All deer-flow worktrees (main checkout + linked worktrees) hardcode the same dev ports (8001/3000/2026), so a service started from any worktree must be reclaimable from another. stop_all now resolves the set of worktree roots (DEERFLOW_ROOTS) and treats a process as deer-flow-owned when its open files live under any of them. It also force-kills survivors on 2026 alongside 8001/3000, fixing `make dev` aborting on the nginx port preflight when a prior nginx lingered on 2026. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(view-image): hide the injected image-context message from the UI ViewImageMiddleware injects a HumanMessage (text + base64 images) so the vision model can see viewed images, but it was the only internal injector that set neither hide_from_ui nor a hidden name, so it leaked into the chat UI (and IM channels) as a user bubble reading "Here are the images you've viewed:". Mark it with additional_kwargs={"hide_from_ui": True}, matching todo/dynamic_context injections, which the frontend isHiddenFromUIMessage and the channel sender already honor. The model still receives the full content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(minimax): mark M2.7 models as text-only (no vision) MiniMax M2.7 / M2.7-highspeed do not support vision; only M3 does. The provider config asserted vision support for M2.7 in four places. - config.example.yaml: 4 M2.7 entries -> supports_vision: false - backend/docs/CONFIGURATION.md: M2.7 + highspeed -> supports_vision: false - wizard: add LLMProvider.model_vision_overrides + extra_config_for() so selecting an M2.7 model writes supports_vision: false while M3 (default) keeps vision; wire it through setup_wizard.py - tests: M2.7-highspeed fixture -> supports_vision=False; add test_minimax_vision_is_per_model Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
This commit is contained in:
DanielWalnut
@@ -0,0 +1,76 @@
|
||||
---
|
||||
name: music-generation
|
||||
description: Use this skill when the user requests to generate, create, compose, or produce music or songs — background music, theme songs, jingles, or instrumental tracks. Generates a song from a style/mood prompt and optional lyrics via the MiniMax music API.
|
||||
---
|
||||
|
||||
# Music Generation Skill
|
||||
|
||||
## Overview
|
||||
|
||||
This skill generates songs (vocal or instrumental) from a structured JSON spec using the
|
||||
MiniMax music generation API (`/v1/music_generation`). You describe the style/mood/scene in
|
||||
`prompt`, optionally provide `lyrics`, and the script returns an MP3.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1: Understand Requirements
|
||||
|
||||
Identify the desired style, mood, scene, language, and whether the user wants vocals or a
|
||||
pure instrumental track. Decide whether to supply lyrics or let the model write them.
|
||||
|
||||
### Step 2: Create the Spec JSON
|
||||
|
||||
Write a JSON file in `/mnt/user-data/workspace/` named `{descriptive-name}.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "Rainy Night Cafe",
|
||||
"prompt": "indie folk, melancholic, introspective, walking alone, cafe",
|
||||
"lyrics": "[verse]\nStreetlights glow the night wind sighs\n[chorus]\nPush the wooden door warm air inside"
|
||||
}
|
||||
```
|
||||
|
||||
Fields:
|
||||
- `title` (optional): a human-readable name.
|
||||
- `prompt` (required): style, mood, and scene. Drives the musical character.
|
||||
- `lyrics` (optional): song lyrics. Use `\n` between lines and structure tags such as
|
||||
`[Intro]`, `[Verse]`, `[Pre Chorus]`, `[Chorus]`, `[Bridge]`, `[Outro]`.
|
||||
- `is_instrumental` (optional, bool): set `true` for a pure instrumental track (no lyrics needed).
|
||||
|
||||
Behavior:
|
||||
- `lyrics` provided → those lyrics are sung.
|
||||
- `is_instrumental: true` → instrumental, no vocals.
|
||||
- neither → the model auto-writes lyrics from `prompt` (`lyrics_optimizer`).
|
||||
|
||||
### Step 3: Execute Generation
|
||||
|
||||
```bash
|
||||
python /mnt/skills/public/music-generation/scripts/generate.py \
|
||||
--prompt-file /mnt/user-data/workspace/rainy-night-cafe.json \
|
||||
--output-file /mnt/user-data/outputs/rainy-night-cafe.mp3
|
||||
```
|
||||
|
||||
Parameters:
|
||||
- `--prompt-file`: Absolute path to the JSON spec (required).
|
||||
- `--output-file`: Absolute path for the output MP3 (required).
|
||||
|
||||
[!NOTE]
|
||||
Do NOT read the python file, just call it with the parameters.
|
||||
|
||||
## Environment
|
||||
|
||||
- `MINIMAX_API_KEY` (required): your MiniMax interface key.
|
||||
- `MINIMAX_API_HOST` (optional): default `https://api.minimaxi.com`.
|
||||
- `MINIMAX_MUSIC_MODEL` (optional): default `music-2.6-free` (works for all API-key users);
|
||||
paid/Token-Plan users can set `music-2.6` for higher limits.
|
||||
|
||||
## Output Handling
|
||||
|
||||
- Music is saved as MP3 (typically in `/mnt/user-data/outputs/`).
|
||||
- Share the generated file with the user using the present_files tool.
|
||||
- Offer to iterate on style or lyrics if adjustments are needed.
|
||||
|
||||
## Notes
|
||||
|
||||
- Keep `prompt` focused on style/mood/scene; put the actual sung words in `lyrics`.
|
||||
- For non-English songs, write `lyrics` in the target language.
|
||||
@@ -0,0 +1,82 @@
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
|
||||
import requests
|
||||
|
||||
MINIMAX_DEFAULT_HOST = "https://api.minimaxi.com"
|
||||
|
||||
|
||||
def _check_base_resp(payload: dict) -> None:
|
||||
base = payload.get("base_resp") or {}
|
||||
if base.get("status_code", 0) != 0:
|
||||
raise Exception(f"MiniMax error {base.get('status_code')}: {base.get('status_msg')}")
|
||||
|
||||
|
||||
def generate_music(prompt_file: str, output_file: str) -> str:
|
||||
"""Generate a song from a JSON spec via MiniMax /v1/music_generation.
|
||||
|
||||
Spec JSON: {"title": str, "prompt": str, "lyrics"?: str, "is_instrumental"?: bool}
|
||||
- lyrics given -> use them (supports [Verse]/[Chorus] structure tags, \\n lines)
|
||||
- is_instrumental true -> pure music, no lyrics needed
|
||||
- otherwise -> lyrics_optimizer auto-writes lyrics from prompt
|
||||
"""
|
||||
with open(prompt_file, "r", encoding="utf-8") as f:
|
||||
spec = json.load(f)
|
||||
|
||||
api_key = os.getenv("MINIMAX_API_KEY")
|
||||
if not api_key:
|
||||
return "MINIMAX_API_KEY is not set"
|
||||
|
||||
prompt = (spec.get("prompt") or "").strip()
|
||||
if not prompt:
|
||||
raise ValueError("`prompt` is required in the music spec")
|
||||
lyrics = spec.get("lyrics") or None # treat empty string the same as absent
|
||||
is_instrumental = bool(spec.get("is_instrumental", False))
|
||||
|
||||
body = {
|
||||
"model": os.getenv("MINIMAX_MUSIC_MODEL", "music-2.6-free"),
|
||||
"prompt": prompt,
|
||||
"output_format": "hex",
|
||||
"audio_setting": {"sample_rate": 44100, "bitrate": 256000, "format": "mp3"},
|
||||
}
|
||||
if lyrics:
|
||||
body["lyrics"] = lyrics
|
||||
elif is_instrumental:
|
||||
body["is_instrumental"] = True
|
||||
else:
|
||||
body["lyrics_optimizer"] = True
|
||||
|
||||
host = os.getenv("MINIMAX_API_HOST", MINIMAX_DEFAULT_HOST).rstrip("/")
|
||||
response = requests.post(
|
||||
f"{host}/v1/music_generation",
|
||||
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
|
||||
json=body,
|
||||
timeout=300,
|
||||
)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
_check_base_resp(payload)
|
||||
audio_hex = (payload.get("data") or {}).get("audio")
|
||||
if not audio_hex:
|
||||
raise Exception("MiniMax returned no audio data")
|
||||
|
||||
output_dir = os.path.dirname(output_file)
|
||||
if output_dir:
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
with open(output_file, "wb") as f:
|
||||
f.write(bytes.fromhex(audio_hex))
|
||||
return f"Successfully generated music to {output_file}"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description="Generate music using MiniMax API")
|
||||
parser.add_argument("--prompt-file", required=True,
|
||||
help="Absolute path to JSON spec file {title, prompt, lyrics?, is_instrumental?}")
|
||||
parser.add_argument("--output-file", required=True, help="Output path for generated MP3")
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
print(generate_music(args.prompt_file, args.output_file))
|
||||
except Exception as e:
|
||||
print(f"Error while generating music: {e}")
|
||||
Reference in New Issue
Block a user