mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-06-10 17:35:57 +00:00
feat: MiniMax provider for image/video/podcast skills + new music-generation skill (#3437)
* docs(spec): MiniMax integration for generation skills + new music skill Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(plan): MiniMax generation providers implementation plan Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(skills): add importlib loader + FakeResp for skill tests * test(skills): register loaded module in sys.modules; raise requests.HTTPError in FakeResp * feat(image-generation): add MiniMax provider with env auto-detect Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(image-generation): guard unknown provider, derive ref MIME, strengthen tests Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(video-generation): add MiniMax provider with async poll/download Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(video-generation): surface base_resp errors while polling; add timeout test * feat(podcast-generation): add MiniMax t2a_v2 provider with env auto-detect Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(podcast-generation): restore TTS credential guard; add volcengine + voice tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(music-generation): new MiniMax music skill via skill-creator Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(music-generation): treat empty lyrics as absent; test no-audio-data path * refactor(skills): add request timeouts to MiniMax network calls Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Potential fix for pull request finding 'Explicit returns mixed with implicit (fall through) returns' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * fix(models): strip inconsistent user-message names for MiniMax chat DeerFlow middlewares tag user messages with provenance names (user-input, summary, loop_warning); langchain serializes them into the OpenAI-compatible payload and MiniMax rejects mismatched user-message names with "user name must be consistent (2013)". PatchedChatMiniMax now drops the per-message name from user-role messages. Point the config.example MiniMax models at PatchedChatMiniMax so they also get reasoning_content mapping. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(image-generation): MiniMax sends JSON prompt field, guard 1500-char limit MiniMax image-01 takes one text string capped at 1500 chars, but the skill was sending the whole structured JSON. The MiniMax provider now extracts the JSON `prompt` field (relying on prompt_optimizer to expand it) and fails fast with a clear error before calling the API when that field exceeds 1500 chars. Authoring stays provider-agnostic; Gemini still receives the full JSON. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(podcast-generation): per-provider TTS concurrency and retry/backoff Each TTS provider owns its concurrency internally — MiniMax runs single-threaded to reduce rate-limit failures, Volcengine keeps 4 workers — with automatic retry and backoff on transient HTTP and base_resp errors. No caller-facing concurrency knob. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(skills): address Copilot review comments on generation skills - video: add raise_for_status + timeout to the Gemini download/POST/poll calls so non-2xx responses surface as clear HTTP errors instead of JSON/KeyError or hangs - video: check the task Fail status before the generic base_resp check so the failure keeps its task_id context - video/image: create the output file parent directory before writing (matching music-generation) so nested output paths do not raise FileNotFoundError - music: require a non-empty prompt and fail fast with ValueError instead of sending an empty prompt to the API Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(scripts): reclaim dev ports across worktrees in make stop/dev All deer-flow worktrees (main checkout + linked worktrees) hardcode the same dev ports (8001/3000/2026), so a service started from any worktree must be reclaimable from another. stop_all now resolves the set of worktree roots (DEERFLOW_ROOTS) and treats a process as deer-flow-owned when its open files live under any of them. It also force-kills survivors on 2026 alongside 8001/3000, fixing `make dev` aborting on the nginx port preflight when a prior nginx lingered on 2026. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(view-image): hide the injected image-context message from the UI ViewImageMiddleware injects a HumanMessage (text + base64 images) so the vision model can see viewed images, but it was the only internal injector that set neither hide_from_ui nor a hidden name, so it leaked into the chat UI (and IM channels) as a user bubble reading "Here are the images you've viewed:". Mark it with additional_kwargs={"hide_from_ui": True}, matching todo/dynamic_context injections, which the frontend isHiddenFromUIMessage and the channel sender already honor. The model still receives the full content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(minimax): mark M2.7 models as text-only (no vision) MiniMax M2.7 / M2.7-highspeed do not support vision; only M3 does. The provider config asserted vision support for M2.7 in four places. - config.example.yaml: 4 M2.7 entries -> supports_vision: false - backend/docs/CONFIGURATION.md: M2.7 + highspeed -> supports_vision: false - wizard: add LLMProvider.model_vision_overrides + extra_config_for() so selecting an M2.7 model writes supports_vision: false while M3 (default) keeps vision; wire it through setup_wizard.py - tests: M2.7-highspeed fixture -> supports_vision=False; add test_minimax_vision_is_per_model Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
This commit is contained in:
DanielWalnut
@@ -1,32 +1,196 @@
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
|
||||
import requests
|
||||
from PIL import Image
|
||||
|
||||
MINIMAX_DEFAULT_HOST = "https://api.minimaxi.com"
|
||||
# MiniMax image-01 caps the prompt at 1500 characters and rejects longer requests
|
||||
# with a generic "invalid params" error, so validate before calling the API.
|
||||
MINIMAX_PROMPT_MAX_CHARS = 1500
|
||||
|
||||
|
||||
def validate_image(image_path: str) -> bool:
|
||||
"""
|
||||
Validate if an image file can be opened and is not corrupted.
|
||||
|
||||
Args:
|
||||
image_path: Path to the image file
|
||||
|
||||
Returns:
|
||||
True if the image is valid and can be opened, False otherwise
|
||||
"""
|
||||
"""Validate if an image file can be opened and is not corrupted."""
|
||||
from PIL import Image # lazy import: keeps module importable without Pillow
|
||||
|
||||
try:
|
||||
with Image.open(image_path) as img:
|
||||
img.verify() # Verify that it's a valid image
|
||||
# Re-open to check if it can be fully loaded (verify() may not catch all issues)
|
||||
with Image.open(image_path) as img:
|
||||
img.load() # Force load the image data
|
||||
with Image.open(image_path) as image:
|
||||
image.verify()
|
||||
with Image.open(image_path) as image:
|
||||
image.load()
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"Warning: Image '{image_path}' is invalid or corrupted: {e}")
|
||||
except Exception as exc:
|
||||
print(f"Warning: Image '{image_path}' is invalid or corrupted: {exc}")
|
||||
return False
|
||||
|
||||
|
||||
def _resolve_provider(override_env: str, existing_provider: str, has_existing_creds: bool) -> str:
|
||||
"""Pick the generation provider.
|
||||
|
||||
1. Explicit <SKILL>_PROVIDER override wins.
|
||||
2. Otherwise prefer the existing provider when its credentials are present.
|
||||
3. Otherwise fall back to MiniMax when MINIMAX_API_KEY is set.
|
||||
"""
|
||||
override = os.getenv(override_env)
|
||||
if override:
|
||||
return override.strip().lower()
|
||||
if has_existing_creds:
|
||||
return existing_provider
|
||||
if os.getenv("MINIMAX_API_KEY"):
|
||||
return "minimax"
|
||||
raise ValueError(
|
||||
f"No credentials found. Set GEMINI_API_KEY for {existing_provider}, "
|
||||
f"or MINIMAX_API_KEY for minimax (optionally force with {override_env})."
|
||||
)
|
||||
|
||||
|
||||
def _minimax_host() -> str:
|
||||
return os.getenv("MINIMAX_API_HOST", MINIMAX_DEFAULT_HOST).rstrip("/")
|
||||
|
||||
|
||||
def _check_base_resp(payload: dict) -> None:
|
||||
base = payload.get("base_resp") or {}
|
||||
if base.get("status_code", 0) != 0:
|
||||
raise Exception(
|
||||
f"MiniMax error {base.get('status_code')}: {base.get('status_msg')}"
|
||||
)
|
||||
|
||||
|
||||
def _guess_mime(image_path: str) -> str:
|
||||
ext = os.path.splitext(image_path)[1].lower()
|
||||
return {
|
||||
".png": "image/png",
|
||||
".webp": "image/webp",
|
||||
".gif": "image/gif",
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
}.get(ext, "image/jpeg")
|
||||
|
||||
|
||||
def _to_data_url(image_path: str) -> str:
|
||||
with open(image_path, "rb") as f:
|
||||
b64 = base64.b64encode(f.read()).decode("utf-8")
|
||||
return f"data:{_guess_mime(image_path)};base64,{b64}"
|
||||
|
||||
|
||||
def _ensure_output_dir(output_file: str) -> None:
|
||||
"""Create the output file's parent directory so nested paths don't fail."""
|
||||
output_dir = os.path.dirname(output_file)
|
||||
if output_dir:
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
|
||||
|
||||
def _minimax_prompt(raw: str) -> str:
|
||||
"""Extract the single text prompt MiniMax image-01 expects.
|
||||
|
||||
The shared prompt file is structured JSON (a consolidated ``prompt`` plus
|
||||
Gemini-oriented fields like ``style`` / ``composition`` / ``negative_prompt``),
|
||||
but MiniMax consumes one string and expands it via ``prompt_optimizer``. The
|
||||
provider adapts the input itself — the caller never needs to know MiniMax is
|
||||
active. Use the JSON ``prompt`` field; fall back to the raw text for plain-text
|
||||
prompt files or JSON without a ``prompt`` field.
|
||||
"""
|
||||
text = raw.strip()
|
||||
try:
|
||||
data = json.loads(text)
|
||||
except (ValueError, json.JSONDecodeError):
|
||||
return text
|
||||
if isinstance(data, dict):
|
||||
core = data.get("prompt")
|
||||
if isinstance(core, str) and core.strip():
|
||||
return core.strip()
|
||||
return text
|
||||
|
||||
|
||||
def _generate_image_minimax(
|
||||
prompt: str, reference_images: list[str], output_file: str, aspect_ratio: str
|
||||
) -> str:
|
||||
api_key = os.getenv("MINIMAX_API_KEY")
|
||||
if not api_key:
|
||||
return "MINIMAX_API_KEY is not set"
|
||||
prompt = _minimax_prompt(prompt)
|
||||
if len(prompt) > MINIMAX_PROMPT_MAX_CHARS:
|
||||
return (
|
||||
f"Prompt is {len(prompt)} characters but MiniMax image-01 accepts at most "
|
||||
f"{MINIMAX_PROMPT_MAX_CHARS}. Shorten the prompt to stay within the limit; "
|
||||
f"reference images plus a tighter description usually recover the detail."
|
||||
)
|
||||
body = {
|
||||
"model": os.getenv("MINIMAX_IMAGE_MODEL", "image-01"),
|
||||
"prompt": prompt,
|
||||
"aspect_ratio": aspect_ratio,
|
||||
"response_format": "base64",
|
||||
"n": 1,
|
||||
"prompt_optimizer": True,
|
||||
}
|
||||
if reference_images:
|
||||
# Reference images are passed as character subjects as-is; unlike the Gemini
|
||||
# path we do not pre-validate them — invalid files surface as a MiniMax API error.
|
||||
body["subject_reference"] = [
|
||||
{"type": "character", "image_file": _to_data_url(p)} for p in reference_images
|
||||
]
|
||||
response = requests.post(
|
||||
f"{_minimax_host()}/v1/image_generation",
|
||||
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
|
||||
json=body,
|
||||
timeout=60,
|
||||
)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
_check_base_resp(payload)
|
||||
images = (payload.get("data") or {}).get("image_base64") or []
|
||||
if not images:
|
||||
raise Exception("MiniMax returned no image data")
|
||||
_ensure_output_dir(output_file)
|
||||
with open(output_file, "wb") as f:
|
||||
f.write(base64.b64decode(images[0]))
|
||||
return f"Successfully generated image to {output_file}"
|
||||
|
||||
|
||||
def _generate_image_gemini(
|
||||
prompt: str, reference_images: list[str], output_file: str, aspect_ratio: str
|
||||
) -> str:
|
||||
parts = []
|
||||
valid_reference_images = []
|
||||
for ref_img in reference_images:
|
||||
if validate_image(ref_img):
|
||||
valid_reference_images.append(ref_img)
|
||||
else:
|
||||
print(f"Skipping invalid reference image: {ref_img}")
|
||||
if len(valid_reference_images) < len(reference_images):
|
||||
skipped = len(reference_images) - len(valid_reference_images)
|
||||
print(f"Note: {skipped} reference image(s) were skipped due to validation failure.")
|
||||
|
||||
for reference_image in valid_reference_images:
|
||||
with open(reference_image, "rb") as f:
|
||||
image_b64 = base64.b64encode(f.read()).decode("utf-8")
|
||||
parts.append({"inlineData": {"mimeType": "image/jpeg", "data": image_b64}})
|
||||
|
||||
api_key = os.getenv("GEMINI_API_KEY")
|
||||
if not api_key:
|
||||
return "GEMINI_API_KEY is not set"
|
||||
response = requests.post(
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/gemini-3-pro-image-preview:generateContent",
|
||||
headers={"x-goog-api-key": api_key, "Content-Type": "application/json"},
|
||||
json={
|
||||
"generationConfig": {"imageConfig": {"aspectRatio": aspect_ratio}},
|
||||
"contents": [{"parts": [*parts, {"text": prompt}]}],
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
response_parts: list[dict] = data["candidates"][0]["content"]["parts"]
|
||||
image_parts = [part for part in response_parts if part.get("inlineData", False)]
|
||||
if len(image_parts) == 1:
|
||||
base64_image = image_parts[0]["inlineData"]["data"]
|
||||
_ensure_output_dir(output_file)
|
||||
with open(output_file, "wb") as f:
|
||||
f.write(base64.b64decode(base64_image))
|
||||
return f"Successfully generated image to {output_file}"
|
||||
raise Exception("Failed to generate image")
|
||||
|
||||
|
||||
def generate_image(
|
||||
prompt_file: str,
|
||||
reference_images: list[str],
|
||||
@@ -35,98 +199,30 @@ def generate_image(
|
||||
) -> str:
|
||||
with open(prompt_file, "r", encoding="utf-8") as f:
|
||||
prompt = f.read()
|
||||
parts = []
|
||||
i = 0
|
||||
|
||||
# Filter out invalid reference images
|
||||
valid_reference_images = []
|
||||
for ref_img in reference_images:
|
||||
if validate_image(ref_img):
|
||||
valid_reference_images.append(ref_img)
|
||||
else:
|
||||
print(f"Skipping invalid reference image: {ref_img}")
|
||||
|
||||
if len(valid_reference_images) < len(reference_images):
|
||||
print(f"Note: {len(reference_images) - len(valid_reference_images)} reference image(s) were skipped due to validation failure.")
|
||||
|
||||
for reference_image in valid_reference_images:
|
||||
i += 1
|
||||
with open(reference_image, "rb") as f:
|
||||
image_b64 = base64.b64encode(f.read()).decode("utf-8")
|
||||
parts.append(
|
||||
{
|
||||
"inlineData": {
|
||||
"mimeType": "image/jpeg",
|
||||
"data": image_b64,
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
api_key = os.getenv("GEMINI_API_KEY")
|
||||
if not api_key:
|
||||
return "GEMINI_API_KEY is not set"
|
||||
response = requests.post(
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/gemini-3-pro-image-preview:generateContent",
|
||||
headers={
|
||||
"x-goog-api-key": api_key,
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
json={
|
||||
"generationConfig": {"imageConfig": {"aspectRatio": aspect_ratio}},
|
||||
"contents": [{"parts": [*parts, {"text": prompt}]}],
|
||||
},
|
||||
provider = _resolve_provider(
|
||||
"IMAGE_GENERATION_PROVIDER", "gemini", bool(os.getenv("GEMINI_API_KEY"))
|
||||
)
|
||||
response.raise_for_status()
|
||||
json = response.json()
|
||||
parts: list[dict] = json["candidates"][0]["content"]["parts"]
|
||||
image_parts = [part for part in parts if part.get("inlineData", False)]
|
||||
if len(image_parts) == 1:
|
||||
base64_image = image_parts[0]["inlineData"]["data"]
|
||||
# Save the image to a file
|
||||
with open(output_file, "wb") as f:
|
||||
f.write(base64.b64decode(base64_image))
|
||||
return f"Successfully generated image to {output_file}"
|
||||
else:
|
||||
raise Exception("Failed to generate image")
|
||||
if provider == "minimax":
|
||||
return _generate_image_minimax(prompt, reference_images, output_file, aspect_ratio)
|
||||
if provider in ("gemini", "google"):
|
||||
return _generate_image_gemini(prompt, reference_images, output_file, aspect_ratio)
|
||||
raise ValueError(f"Unknown image provider: {provider!r} (use 'gemini' or 'minimax')")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(description="Generate images using Gemini API")
|
||||
parser.add_argument(
|
||||
"--prompt-file",
|
||||
required=True,
|
||||
help="Absolute path to JSON prompt file",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--reference-images",
|
||||
nargs="*",
|
||||
default=[],
|
||||
help="Absolute paths to reference images (space-separated)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-file",
|
||||
required=True,
|
||||
help="Output path for generated image",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--aspect-ratio",
|
||||
required=False,
|
||||
default="16:9",
|
||||
help="Aspect ratio of the generated image",
|
||||
)
|
||||
|
||||
parser = argparse.ArgumentParser(description="Generate images using Gemini or MiniMax API")
|
||||
parser.add_argument("--prompt-file", required=True, help="Absolute path to JSON prompt file")
|
||||
parser.add_argument("--reference-images", nargs="*", default=[],
|
||||
help="Absolute paths to reference images (space-separated)")
|
||||
parser.add_argument("--output-file", required=True, help="Output path for generated image")
|
||||
parser.add_argument("--aspect-ratio", required=False, default="16:9",
|
||||
help="Aspect ratio of the generated image")
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
print(
|
||||
generate_image(
|
||||
args.prompt_file,
|
||||
args.reference_images,
|
||||
args.output_file,
|
||||
args.aspect_ratio,
|
||||
)
|
||||
)
|
||||
print(generate_image(args.prompt_file, args.reference_images,
|
||||
args.output_file, args.aspect_ratio))
|
||||
except Exception as e:
|
||||
print(f"Error while generating image: {e}")
|
||||
|
||||
Reference in New Issue
Block a user