mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-05-22 16:06:50 +00:00
df95154282
* fix(tracing): propagate session_id and user_id into Langfuse traces
Adds Langfuse v4 reserved trace attributes (langfuse_session_id,
langfuse_user_id, langfuse_trace_name, langfuse_tags) to
RunnableConfig.metadata inside the run worker, so the langchain
CallbackHandler can lift them onto the root trace.
- New deerflow.tracing.metadata.build_langfuse_trace_metadata() returns
the reserved keys when Langfuse is in the enabled providers, else {}.
- worker.run_agent merges them with setdefault so caller-supplied keys
win, allowing per-request overrides from upstream metadata.
- session_id mirrors the LangGraph thread_id; user_id reads
get_effective_user_id() (falls back to "default" in no-auth mode).
- trace_name defaults to "lead-agent"; tags carry env and model name
when DEER_FLOW_ENV (or ENVIRONMENT) and a model name are present.
Closes #2930
* fix(tracing): attach Langfuse callback at graph root so metadata propagates
The first commit injected ``langfuse_session_id`` / ``langfuse_user_id`` /
``langfuse_trace_name`` / ``langfuse_tags`` into ``RunnableConfig.metadata``,
but on ``main`` the Langfuse callback is attached at *model* level
(``models/factory.py``). LangChain still threads ``parent_run_id`` through
the contextvar, so the handler sees the model as a nested observation and
``__on_llm_action`` strips the ``langfuse_*`` keys
(``keep_langfuse_trace_attributes=False``). The trace's top-level
``sessionId`` / ``userId`` therefore stayed empty in deer-flow's LangGraph
runtime — confirmed live against a real Langfuse instance.
This commit moves the callback to the **graph invocation root** so the
handler fires ``on_chain_start(parent_run_id=None)`` and runs the
``propagate_attributes`` path that actually lifts ``session_id`` /
``user_id`` onto the trace:
- ``models/factory.py``: add ``attach_tracing`` keyword (default ``True``)
so standalone callers (``MemoryUpdater``, etc.) keep their direct
model-level tracing.
- ``agents/lead_agent/agent.py``: call ``build_tracing_callbacks()`` once
inside ``_make_lead_agent`` and append the result to
``config["callbacks"]``; the four in-graph ``create_chat_model`` sites
(bootstrap, default agent, sync + async summarization) pass
``attach_tracing=False`` to avoid duplicate spans.
- ``agents/middlewares/title_middleware.py``: same ``attach_tracing=False``
for the title-generation model, since it inherits the graph's
RunnableConfig via ``_get_runnable_config``.
Test updates:
- ``tests/test_lead_agent_model_resolution.py`` and
``tests/test_title_middleware_core_logic.py``: extend the fake
``create_chat_model`` signatures / mock assertions to accept the new
``attach_tracing`` kwarg.
- ``tests/test_worker_langfuse_metadata.py``: switch the no-user fallback
test from direct ContextVar mutation to ``monkeypatch.setattr`` on
``get_effective_user_id`` to avoid pollution across the langfuse OTel
global tracer provider.
- ``tests/conftest.py``: add an autouse fixture that resets
``deerflow.config.title_config._title_config`` to its pristine default
after every test. Any test that loads the real ``config.yaml`` (via
``get_app_config()``) calls ``load_title_config_from_dict`` and mutates
the module-level singleton, which previously poisoned the
title-middleware suite when run after, e.g., the new
``test_worker_langfuse_metadata.py`` cases. The fixture is independent
of this PR's main change but unblocks the cross-file test run.
Live verification (same Langfuse instance as before):
- Drove ``worker.run_agent`` against the real ``make_lead_agent`` +
``gpt-4o-mini`` for three distinct ``user_context`` identities
(``fancy-engineer``, ``alice-pm``, ``bob-designer``).
- Each run produced one ``lead-agent`` trace whose top-level
``sessionId`` / ``userId`` / ``tags`` carry the expected values, e.g.
``session=e2e-2930-8f347c-alice-pm user=alice-pm name='lead-agent'
tags=['model:gpt-4o-mini']``.
Refs #2930.
* fix(tracing): extend root-callback + metadata injection to the embedded client
Addresses Copilot review on PR #2944.
Commit 2 disabled model-level tracing for ``TitleMiddleware`` and
``_create_summarization_middleware`` because ``_make_lead_agent`` now
attaches the tracing callbacks at the graph invocation root. But the
embedded ``DeerFlowClient`` does not call ``_make_lead_agent`` — it
calls ``_build_middlewares`` directly and never appends the tracing
handlers to its ``RunnableConfig``. So under the embedded path,
title-generation and summarization LLM calls were left untraced —
a regression introduced by this PR.
This commit mirrors the gateway worker's injection in
``DeerFlowClient.stream``:
- Append ``build_tracing_callbacks()`` to ``config["callbacks"]`` so
the Langfuse handler sees ``on_chain_start(parent_run_id=None)`` at
the graph root and runs the ``propagate_attributes`` path.
- Merge ``build_langfuse_trace_metadata(...)`` into
``config["metadata"]`` with ``setdefault`` so caller-supplied keys
still win.
- ``_ensure_agent`` now creates its main model with
``attach_tracing=False`` to avoid duplicate spans now that the
callback lives at the graph root.
Docs:
- ``backend/CLAUDE.md`` Tracing section rewritten to describe the
graph-root attachment model (replacing the inaccurate
"at model-creation time" wording).
- ``README.md`` Langfuse section now lists both injection points
(worker + client) instead of only the worker path.
Tests:
- ``tests/test_client_langfuse_metadata.py`` (new, 3 cases):
callbacks + metadata are injected when Langfuse is enabled,
caller-supplied metadata overrides win via ``setdefault``, and the
injection is inert when Langfuse is disabled.
Live verification on the real Langfuse instance:
=== user=fancy-client ===
id=cbd22847.. session=client-2930-6b9491-fancy-client user=fancy-client name='lead-agent'
=== user=alice-client ===
id=b4f6f576.. session=client-2930-6b9491-alice-client user=alice-client name='lead-agent'
Refs #2930.
* refactor(tracing): address maintainer review on PR #2944
Addresses @WillemJiang's 5 comments.
1. Duplicated metadata-injection code between worker.py and client.py
New ``deerflow.tracing.inject_langfuse_metadata(config, ...)`` helper
takes the 10-line build + merge + setdefault logic that was duplicated
in ``runtime/runs/worker.py`` and ``client.py``. Both callers now share
a single source of truth, so the two paths cannot drift.
2. Direct private-attribute mutation in conftest.py and tests
Added public ``reset_tracing_config()`` / ``reset_title_config()``
functions. ``tests/conftest.py`` and every test that previously did
``tracing_module._tracing_config = None`` or
``title_module._title_config = TitleConfig()`` now goes through the
public API. A future internal rename will surface as an ImportError
instead of a silent no-op.
3. client.py reading os.environ directly
``DeerFlowClient.__init__`` grows an optional ``environment`` parameter
so programmatic callers can pass the deployment label explicitly.
``stream()`` consults ``self._environment`` first and only falls back
to ``DEER_FLOW_ENV`` / ``ENVIRONMENT`` env vars when nothing was
passed in. Backwards compatible — env-var behaviour preserved for
callers that opt to keep using it.
4. build_tracing_callbacks() cached on hot path
Not implemented. Inspected the langfuse v4 ``langchain.CallbackHandler``
constructor: it only resolves the module-level singleton client via
``get_client()`` and initialises a few dicts (no I/O, no env parsing
at construction time). The build is essentially free. Caching would
trade a non-measurable speedup for two real risks: handler instances
carry per-run state internally (``_run_states``, ``_root_run_states``,
``last_trace_id``), and tracing config can be reloaded by env-var
changes between runs. Will revisit if profiling ever shows it as
a hot spot.
5. attach_tracing=False easy to forget at new in-graph call sites
- Module docstring at the top of ``lead_agent/agent.py`` documents
the invariant ("every in-graph ``create_chat_model`` MUST pass
``attach_tracing=False``") and enumerates the current sites.
- New regression test
``test_make_lead_agent_attaches_tracing_callbacks_at_graph_root`` in
``tests/test_lead_agent_model_resolution.py`` locks both halves of
the invariant: ``config["callbacks"]`` carries the tracing handler
after ``_make_lead_agent``, AND every ``create_chat_model`` call
captured by the test passes ``attach_tracing=False``. A future
in-graph site that forgets the flag will fail this test.
Lint clean. Full touched-suite bundle: 246 passed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
162 lines
5.3 KiB
Python
162 lines
5.3 KiB
Python
import os
|
|
import threading
|
|
|
|
from pydantic import BaseModel, Field
|
|
|
|
_config_lock = threading.Lock()
|
|
|
|
|
|
class LangSmithTracingConfig(BaseModel):
|
|
"""Configuration for LangSmith tracing."""
|
|
|
|
enabled: bool = Field(...)
|
|
api_key: str | None = Field(...)
|
|
project: str = Field(...)
|
|
endpoint: str = Field(...)
|
|
|
|
@property
|
|
def is_configured(self) -> bool:
|
|
return self.enabled and bool(self.api_key)
|
|
|
|
def validate(self) -> None:
|
|
if self.enabled and not self.api_key:
|
|
raise ValueError("LangSmith tracing is enabled but LANGSMITH_API_KEY (or LANGCHAIN_API_KEY) is not set.")
|
|
|
|
|
|
class LangfuseTracingConfig(BaseModel):
|
|
"""Configuration for Langfuse tracing."""
|
|
|
|
enabled: bool = Field(...)
|
|
public_key: str | None = Field(...)
|
|
secret_key: str | None = Field(...)
|
|
host: str = Field(...)
|
|
|
|
@property
|
|
def is_configured(self) -> bool:
|
|
return self.enabled and bool(self.public_key) and bool(self.secret_key)
|
|
|
|
def validate(self) -> None:
|
|
if not self.enabled:
|
|
return
|
|
missing: list[str] = []
|
|
if not self.public_key:
|
|
missing.append("LANGFUSE_PUBLIC_KEY")
|
|
if not self.secret_key:
|
|
missing.append("LANGFUSE_SECRET_KEY")
|
|
if missing:
|
|
raise ValueError(f"Langfuse tracing is enabled but required settings are missing: {', '.join(missing)}")
|
|
|
|
|
|
class TracingConfig(BaseModel):
|
|
"""Tracing configuration for supported providers."""
|
|
|
|
langsmith: LangSmithTracingConfig = Field(...)
|
|
langfuse: LangfuseTracingConfig = Field(...)
|
|
|
|
@property
|
|
def is_configured(self) -> bool:
|
|
return bool(self.enabled_providers)
|
|
|
|
@property
|
|
def explicitly_enabled_providers(self) -> list[str]:
|
|
enabled: list[str] = []
|
|
if self.langsmith.enabled:
|
|
enabled.append("langsmith")
|
|
if self.langfuse.enabled:
|
|
enabled.append("langfuse")
|
|
return enabled
|
|
|
|
@property
|
|
def enabled_providers(self) -> list[str]:
|
|
enabled: list[str] = []
|
|
if self.langsmith.is_configured:
|
|
enabled.append("langsmith")
|
|
if self.langfuse.is_configured:
|
|
enabled.append("langfuse")
|
|
return enabled
|
|
|
|
def validate_enabled(self) -> None:
|
|
self.langsmith.validate()
|
|
self.langfuse.validate()
|
|
|
|
|
|
_tracing_config: TracingConfig | None = None
|
|
|
|
|
|
_TRUTHY_VALUES = {"1", "true", "yes", "on"}
|
|
|
|
|
|
def _env_flag_preferred(*names: str) -> bool:
|
|
"""Return the boolean value of the first env var that is present and non-empty."""
|
|
for name in names:
|
|
value = os.environ.get(name)
|
|
if value is not None and value.strip():
|
|
return value.strip().lower() in _TRUTHY_VALUES
|
|
return False
|
|
|
|
|
|
def _first_env_value(*names: str) -> str | None:
|
|
"""Return the first non-empty environment value from candidate names."""
|
|
for name in names:
|
|
value = os.environ.get(name)
|
|
if value and value.strip():
|
|
return value.strip()
|
|
return None
|
|
|
|
|
|
def get_tracing_config() -> TracingConfig:
|
|
"""Get the current tracing configuration from environment variables."""
|
|
global _tracing_config
|
|
if _tracing_config is not None:
|
|
return _tracing_config
|
|
with _config_lock:
|
|
if _tracing_config is not None:
|
|
return _tracing_config
|
|
_tracing_config = TracingConfig(
|
|
langsmith=LangSmithTracingConfig(
|
|
enabled=_env_flag_preferred("LANGSMITH_TRACING", "LANGCHAIN_TRACING_V2", "LANGCHAIN_TRACING"),
|
|
api_key=_first_env_value("LANGSMITH_API_KEY", "LANGCHAIN_API_KEY"),
|
|
project=_first_env_value("LANGSMITH_PROJECT", "LANGCHAIN_PROJECT") or "deer-flow",
|
|
endpoint=_first_env_value("LANGSMITH_ENDPOINT", "LANGCHAIN_ENDPOINT") or "https://api.smith.langchain.com",
|
|
),
|
|
langfuse=LangfuseTracingConfig(
|
|
enabled=_env_flag_preferred("LANGFUSE_TRACING"),
|
|
public_key=_first_env_value("LANGFUSE_PUBLIC_KEY"),
|
|
secret_key=_first_env_value("LANGFUSE_SECRET_KEY"),
|
|
host=_first_env_value("LANGFUSE_BASE_URL") or "https://cloud.langfuse.com",
|
|
),
|
|
)
|
|
return _tracing_config
|
|
|
|
|
|
def get_enabled_tracing_providers() -> list[str]:
|
|
"""Return the configured tracing providers that are enabled and complete."""
|
|
return get_tracing_config().enabled_providers
|
|
|
|
|
|
def get_explicitly_enabled_tracing_providers() -> list[str]:
|
|
"""Return tracing providers explicitly enabled by config, even if incomplete."""
|
|
return get_tracing_config().explicitly_enabled_providers
|
|
|
|
|
|
def validate_enabled_tracing_providers() -> None:
|
|
"""Validate that any explicitly enabled providers are fully configured."""
|
|
get_tracing_config().validate_enabled()
|
|
|
|
|
|
def is_tracing_enabled() -> bool:
|
|
"""Check if any tracing provider is enabled and fully configured."""
|
|
return get_tracing_config().is_configured
|
|
|
|
|
|
def reset_tracing_config() -> None:
|
|
"""Discard the cached :class:`TracingConfig` so the next call rebuilds it.
|
|
|
|
Public API so that tests do not have to reach into the private
|
|
``_tracing_config`` module attribute. A future internal rename would
|
|
silently break callers that mutate the attribute directly.
|
|
"""
|
|
global _tracing_config
|
|
with _config_lock:
|
|
_tracing_config = None
|