Files
deer-flow/backend/packages/harness/deerflow/models/patched_minimax.py
T
DanielWalnut cd5bedaa74 feat: MiniMax provider for image/video/podcast skills + new music-generation skill (#3437)
* docs(spec): MiniMax integration for generation skills + new music skill

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(plan): MiniMax generation providers implementation plan

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(skills): add importlib loader + FakeResp for skill tests

* test(skills): register loaded module in sys.modules; raise requests.HTTPError in FakeResp

* feat(image-generation): add MiniMax provider with env auto-detect

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(image-generation): guard unknown provider, derive ref MIME, strengthen tests

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(video-generation): add MiniMax provider with async poll/download

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(video-generation): surface base_resp errors while polling; add timeout test

* feat(podcast-generation): add MiniMax t2a_v2 provider with env auto-detect

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(podcast-generation): restore TTS credential guard; add volcengine + voice tests

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(music-generation): new MiniMax music skill via skill-creator

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(music-generation): treat empty lyrics as absent; test no-audio-data path

* refactor(skills): add request timeouts to MiniMax network calls

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Potential fix for pull request finding 'Explicit returns mixed with implicit (fall through) returns'

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

* fix(models): strip inconsistent user-message names for MiniMax chat

DeerFlow middlewares tag user messages with provenance names (user-input, summary, loop_warning); langchain serializes them into the OpenAI-compatible payload and MiniMax rejects mismatched user-message names with "user name must be consistent (2013)". PatchedChatMiniMax now drops the per-message name from user-role messages. Point the config.example MiniMax models at PatchedChatMiniMax so they also get reasoning_content mapping.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(image-generation): MiniMax sends JSON prompt field, guard 1500-char limit

MiniMax image-01 takes one text string capped at 1500 chars, but the skill was sending the whole structured JSON. The MiniMax provider now extracts the JSON `prompt` field (relying on prompt_optimizer to expand it) and fails fast with a clear error before calling the API when that field exceeds 1500 chars. Authoring stays provider-agnostic; Gemini still receives the full JSON.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(podcast-generation): per-provider TTS concurrency and retry/backoff

Each TTS provider owns its concurrency internally — MiniMax runs single-threaded to reduce rate-limit failures, Volcengine keeps 4 workers — with automatic retry and backoff on transient HTTP and base_resp errors. No caller-facing concurrency knob.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(skills): address Copilot review comments on generation skills

- video: add raise_for_status + timeout to the Gemini download/POST/poll calls so non-2xx responses surface as clear HTTP errors instead of JSON/KeyError or hangs
- video: check the task Fail status before the generic base_resp check so the failure keeps its task_id context
- video/image: create the output file parent directory before writing (matching music-generation) so nested output paths do not raise FileNotFoundError
- music: require a non-empty prompt and fail fast with ValueError instead of sending an empty prompt to the API

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(scripts): reclaim dev ports across worktrees in make stop/dev

All deer-flow worktrees (main checkout + linked worktrees) hardcode the same dev ports (8001/3000/2026), so a service started from any worktree must be reclaimable from another. stop_all now resolves the set of worktree roots (DEERFLOW_ROOTS) and treats a process as deer-flow-owned when its open files live under any of them. It also force-kills survivors on 2026 alongside 8001/3000, fixing `make dev` aborting on the nginx port preflight when a prior nginx lingered on 2026.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(view-image): hide the injected image-context message from the UI

ViewImageMiddleware injects a HumanMessage (text + base64 images) so the vision model can see viewed images, but it was the only internal injector that set neither hide_from_ui nor a hidden name, so it leaked into the chat UI (and IM channels) as a user bubble reading "Here are the images you've viewed:". Mark it with additional_kwargs={"hide_from_ui": True}, matching todo/dynamic_context injections, which the frontend isHiddenFromUIMessage and the channel sender already honor. The model still receives the full content.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minimax): mark M2.7 models as text-only (no vision)

MiniMax M2.7 / M2.7-highspeed do not support vision; only M3 does. The
provider config asserted vision support for M2.7 in four places.

- config.example.yaml: 4 M2.7 entries -> supports_vision: false
- backend/docs/CONFIGURATION.md: M2.7 + highspeed -> supports_vision: false
- wizard: add LLMProvider.model_vision_overrides + extra_config_for() so
  selecting an M2.7 model writes supports_vision: false while M3 (default)
  keeps vision; wire it through setup_wizard.py
- tests: M2.7-highspeed fixture -> supports_vision=False; add
  test_minimax_vision_is_per_model

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-06-08 22:04:38 +08:00

240 lines
8.9 KiB
Python

"""Patched ChatOpenAI adapter for MiniMax reasoning output.
MiniMax's OpenAI-compatible chat completions API can return structured
``reasoning_details`` when ``extra_body.reasoning_split=true`` is enabled.
``langchain_openai.ChatOpenAI`` currently ignores that field, so DeerFlow's
frontend never receives reasoning content in the shape it expects.
This adapter preserves ``reasoning_split`` in the request payload and maps the
provider-specific reasoning field into ``additional_kwargs.reasoning_content``,
which DeerFlow already understands.
"""
from __future__ import annotations
import re
from collections.abc import Mapping
from typing import Any
from langchain_core.language_models import LanguageModelInput
from langchain_core.messages import AIMessage, AIMessageChunk
from langchain_core.outputs import ChatGeneration, ChatGenerationChunk, ChatResult
from langchain_openai import ChatOpenAI
from langchain_openai.chat_models.base import (
_convert_delta_to_message_chunk,
_create_usage_metadata,
)
_THINK_TAG_RE = re.compile(r"<think>\s*(.*?)\s*</think>", re.DOTALL)
def _extract_reasoning_text(
reasoning_details: Any,
*,
strip_parts: bool = True,
) -> str | None:
if not isinstance(reasoning_details, list):
return None
parts: list[str] = []
for item in reasoning_details:
if not isinstance(item, Mapping):
continue
text = item.get("text")
if isinstance(text, str):
normalized = text.strip() if strip_parts else text
if normalized.strip():
parts.append(normalized)
return "\n\n".join(parts) if parts else None
def _strip_inline_think_tags(content: str) -> tuple[str, str | None]:
reasoning_parts: list[str] = []
def _replace(match: re.Match[str]) -> str:
reasoning = match.group(1).strip()
if reasoning:
reasoning_parts.append(reasoning)
return ""
cleaned = _THINK_TAG_RE.sub(_replace, content).strip()
reasoning = "\n\n".join(reasoning_parts) if reasoning_parts else None
return cleaned, reasoning
def _merge_reasoning(*values: str | None) -> str | None:
merged: list[str] = []
for value in values:
if not value:
continue
normalized = value.strip()
if normalized and normalized not in merged:
merged.append(normalized)
return "\n\n".join(merged) if merged else None
def _with_reasoning_content(
message: AIMessage | AIMessageChunk,
reasoning: str | None,
*,
preserve_whitespace: bool = False,
):
if not reasoning:
return message
additional_kwargs = dict(message.additional_kwargs)
if preserve_whitespace:
existing = additional_kwargs.get("reasoning_content")
additional_kwargs["reasoning_content"] = f"{existing}{reasoning}" if isinstance(existing, str) else reasoning
else:
additional_kwargs["reasoning_content"] = _merge_reasoning(
additional_kwargs.get("reasoning_content"),
reasoning,
)
return message.model_copy(update={"additional_kwargs": additional_kwargs})
class PatchedChatMiniMax(ChatOpenAI):
"""ChatOpenAI adapter that preserves MiniMax reasoning output."""
def _get_request_payload(
self,
input_: LanguageModelInput,
*,
stop: list[str] | None = None,
**kwargs: Any,
) -> dict:
payload = super()._get_request_payload(input_, stop=stop, **kwargs)
extra_body = payload.get("extra_body")
if isinstance(extra_body, dict):
payload["extra_body"] = {
**extra_body,
"reasoning_split": True,
}
else:
payload["extra_body"] = {"reasoning_split": True}
self._strip_user_message_names(payload)
return payload
@staticmethod
def _strip_user_message_names(payload: dict) -> None:
"""Drop the per-message ``name`` field from user-role messages.
DeerFlow middlewares tag user messages with internal provenance names
(``user-input``, ``summary``, ``loop_warning``, ...). ``langchain_openai``
serializes those into the OpenAI-compatible request, but MiniMax requires
every user-role ``name`` to be identical and otherwise rejects the request
with ``invalid params, user name must be consistent (2013)``. MiniMax does
not use the per-message author name, so strip it.
"""
messages = payload.get("messages")
if not isinstance(messages, list):
return
for message in messages:
if isinstance(message, dict) and message.get("role") == "user":
message.pop("name", None)
def _convert_chunk_to_generation_chunk(
self,
chunk: dict,
default_chunk_class: type,
base_generation_info: dict | None,
) -> ChatGenerationChunk | None:
if chunk.get("type") == "content.delta":
return None
token_usage = chunk.get("usage")
choices = chunk.get("choices", []) or chunk.get("chunk", {}).get("choices", [])
usage_metadata = _create_usage_metadata(token_usage, chunk.get("service_tier")) if token_usage else None
if len(choices) == 0:
generation_chunk = ChatGenerationChunk(
message=default_chunk_class(content="", usage_metadata=usage_metadata),
generation_info=base_generation_info,
)
if self.output_version == "v1":
generation_chunk.message.content = []
generation_chunk.message.response_metadata["output_version"] = "v1"
return generation_chunk
choice = choices[0]
delta = choice.get("delta")
if delta is None:
return None
message_chunk = _convert_delta_to_message_chunk(delta, default_chunk_class)
generation_info = {**base_generation_info} if base_generation_info else {}
if finish_reason := choice.get("finish_reason"):
generation_info["finish_reason"] = finish_reason
if model_name := chunk.get("model"):
generation_info["model_name"] = model_name
if system_fingerprint := chunk.get("system_fingerprint"):
generation_info["system_fingerprint"] = system_fingerprint
if service_tier := chunk.get("service_tier"):
generation_info["service_tier"] = service_tier
logprobs = choice.get("logprobs")
if logprobs:
generation_info["logprobs"] = logprobs
reasoning = _extract_reasoning_text(
delta.get("reasoning_details"),
strip_parts=False,
)
if isinstance(message_chunk, AIMessageChunk):
if usage_metadata:
message_chunk.usage_metadata = usage_metadata
if reasoning:
message_chunk = _with_reasoning_content(
message_chunk,
reasoning,
preserve_whitespace=True,
)
message_chunk.response_metadata["model_provider"] = "openai"
return ChatGenerationChunk(
message=message_chunk,
generation_info=generation_info or None,
)
def _create_chat_result(
self,
response: dict | Any,
generation_info: dict | None = None,
) -> ChatResult:
result = super()._create_chat_result(response, generation_info)
response_dict = response if isinstance(response, dict) else response.model_dump()
choices = response_dict.get("choices", [])
generations: list[ChatGeneration] = []
for index, generation in enumerate(result.generations):
choice = choices[index] if index < len(choices) else {}
message = generation.message
if isinstance(message, AIMessage):
content = message.content if isinstance(message.content, str) else None
cleaned_content = content
inline_reasoning = None
if isinstance(content, str):
cleaned_content, inline_reasoning = _strip_inline_think_tags(content)
choice_message = choice.get("message", {}) if isinstance(choice, Mapping) else {}
split_reasoning = _extract_reasoning_text(choice_message.get("reasoning_details"))
merged_reasoning = _merge_reasoning(split_reasoning, inline_reasoning)
updated_message = message
if cleaned_content is not None and cleaned_content != message.content:
updated_message = updated_message.model_copy(update={"content": cleaned_content})
if merged_reasoning:
updated_message = _with_reasoning_content(updated_message, merged_reasoning)
generation = ChatGeneration(
message=updated_message,
generation_info=generation.generation_info,
)
generations.append(generation)
return ChatResult(generations=generations, llm_output=result.llm_output)