fix(runtime): suppress tool execution when provider safety-terminates with tool_calls (#3035)

* fix(runtime): suppress tool execution when provider safety-terminates with tool_calls

When a provider stops generation for safety reasons (OpenAI/Moonshot
finish_reason=content_filter, Anthropic stop_reason=refusal, Gemini
finish_reason=SAFETY/BLOCKLIST/PROHIBITED_CONTENT/SPII/RECITATION/
IMAGE_SAFETY/...), the response may still carry truncated tool_calls.
LangChain's tool router treats any non-empty tool_calls as executable,
so partial arguments (e.g. write_file with a half-finished markdown)
get dispatched and the agent loops on retry.

Add SafetyFinishReasonMiddleware at after_model: detect safety
termination via a pluggable detector registry, clear both structured
tool_calls and raw additional_kwargs.tool_calls / function_call,
preserve response_metadata.finish_reason for downstream observers,
stamp additional_kwargs.safety_termination for traces, append a
user-facing explanation to message content (list-aware for thinking
blocks), and emit a safety_termination custom stream event so SSE
consumers can reconcile any "tool starting..." UI.

Default detectors cover OpenAI-compatible content_filter, Anthropic
refusal, and Gemini safety enums (text + image). Custom providers are
added via reflection (same pattern as guardrails). Wired into both
lead-agent and subagent runtimes.

Closes #3028

* fix(runtime): persist safety_termination as a middleware audit event

Address review on #3035: the SSE custom event is great for live
consumers but invisible to post-run audit. RunEventStore should carry
its own row so operators can answer "which runs were safety-suppressed
today?" from a single SQL query without joining the message body.

Worker now exposes the run-scoped RunJournal via
runtime.context["__run_journal"] (sentinel key, internal channel).
SafetyFinishReasonMiddleware calls the previously-unused
RunJournal.record_middleware, which emits

  event_type = "middleware:safety_termination"
  category   = "middleware"
  content    = {name, hook, action, changes={
                  detector, reason_field, reason_value,
                  suppressed_tool_call_count,
                  suppressed_tool_call_names,
                  suppressed_tool_call_ids,
                  message_id, extras}}

Tool *arguments* are deliberately excluded — those are the very content
the provider filtered and persisting them would defeat the purpose of
the safety filter (per review note in #3035).

Graceful skips when journal is absent (subagent runtime, unit tests,
no-event-store local dev). Journal exceptions never propagate into the
agent loop.

Refs #3028

* fix(runtime): satisfy ruff format + address Copilot review

- ruff format on safety_finish_reason_config.py and e2e demo (CI lint
  failed on ruff format --check; backend Makefile lint target runs
  ruff check AND ruff format --check).
- Docstring on SafetyFinishReasonConfig now says resolve_variable to
  match the actual loader used in from_config (the wording was
  resolve_class previously; behavior is unchanged — resolve_variable
  mirrors how guardrails.provider is loaded).
- Switch the AIMessage type check in SafetyFinishReasonMiddleware._apply
  from getattr(last, "type") == "ai" to isinstance(last, AIMessage),
  matching TokenUsageMiddleware / TodoMiddleware / ViewImageMiddleware
  / SummarizationMiddleware which are the dominant pattern.

Refs #3028
This commit is contained in:
Xinmin Zeng
2026-05-22 21:20:28 +08:00
committed by GitHub
parent 253542ea0d
commit be0eae9825
14 changed files with 1936 additions and 5 deletions
+36 -1
View File
@@ -15,7 +15,7 @@
# ============================================================================
# Bump this number when the config schema changes.
# Run `make config-upgrade` to merge new fields into your local config.yaml.
config_version: 9
config_version: 10
# ============================================================================
# Logging
@@ -535,6 +535,41 @@ loop_detection:
# warn: 150
# hard_limit: 300
# ============================================================================
# Provider Safety Termination Configuration
# ============================================================================
# Intercept AIMessages where the provider stopped generation for safety reasons
# (e.g. OpenAI finish_reason='content_filter', Anthropic stop_reason='refusal',
# Gemini finish_reason='SAFETY') while still returning tool_calls. The
# tool_calls in such responses are typically truncated/unreliable and must
# not be executed. See issue #3028 for the full failure mode.
#
# Detectors are loaded by class path via reflection (same pattern as
# guardrails / models / tools). The built-in set covers OpenAI-compatible
# content_filter, Anthropic refusal, and Gemini SAFETY/BLOCKLIST/
# PROHIBITED_CONTENT/SPII/RECITATION.
safety_finish_reason:
enabled: true
# Leave `detectors` unset to use the built-in detector set. Set to a
# non-empty list to fully override (use `enabled: false` to disable instead
# of providing an empty list).
#
# Example — extend the OpenAI-compatible detector for a Chinese provider
# whose gateway uses a non-standard finish_reason token:
# detectors:
# - use: deerflow.agents.middlewares.safety_termination_detectors:OpenAICompatibleContentFilterDetector
# config:
# finish_reasons: ["content_filter", "sensitive", "risk_control"]
# - use: deerflow.agents.middlewares.safety_termination_detectors:AnthropicRefusalDetector
# - use: deerflow.agents.middlewares.safety_termination_detectors:GeminiSafetyDetector
#
# Example — add a custom detector for an in-house provider:
# detectors:
# - use: my_company.deerflow_ext:WenxinSafetyDetector
# config:
# error_codes: [336003, 17, 18]
# ============================================================================
# Sandbox Configuration
# ============================================================================