Files
deer-flow/backend/packages/harness/deerflow/config/tracing_config.py
T
4af6178358 feat(trace): add agent observability with Monocle (#4024)
* Add Monocle tracing

Enable Monocle (OpenTelemetry tracing for LLM apps) with one setup call plus the monocle_apptrace dependency. setup_monocle_telemetry auto-instruments the frameworks already in use and writes traces to .monocle/. Additive; no changes to application logic.

* Config-gate Monocle telemetry in the Gateway lifespan

Addresses review on #4024: moves setup_monocle_telemetry out of agents/__init__ import time into the Gateway lifespan, gated by MonocleTracingConfig (MONOCLE_TRACING env, default off). Warns on the Langfuse/global-OTel-provider conflict and relies on monocle_apptrace's own duplicate-setup guard and existing-provider attach. Pins monocle_apptrace>=0.8.8 (+ uv.lock), adds .monocle/ to .gitignore, adds tests (default-off / toggle-on / no import-time setup), and documents exporters, Okahu, and the VS Code viewer in README, config.example.yaml, and backend/AGENTS.md.

* Clarify Monocle/Langfuse single-provider guidance

Make the docstring, warning, and AGENTS.md consistent with the README: only one library can own the global OpenTelemetry provider; Monocle initializes at startup before Langfuse's per-run handler, so enabling both drops Langfuse's spans — enable one OTel tracer (LangSmith, a callback, coexists fine).

* Address review: optional extra, exporter validation, off-box warning, tests

Responds to the second review round.

- Make monocle_apptrace an optional extra (deerflow-harness[monocle], re-exposed
  as deer-flow[monocle]) following the boxlite/tui precedent, so a default
  install no longer pulls the OpenTelemetry stack. It stays pinned in the dev
  group for the tracing tests, and enabling MONOCLE_TRACING without the extra
  raises a clear install error.
- Warn loudly at startup whenever any exporter other than `file` is configured,
  since those move prompts, tool inputs/outputs, and completions beyond the
  local .monocle/ directory.
- Validate MONOCLE_EXPORTERS against the known exporter names and require
  OKAHU_API_KEY when okahu is selected, mirroring the Langfuse pattern.
  Validation runs from Monocle's own init (not validate_enabled) so a config
  typo can never fail agent runs; errors surface at Gateway startup instead.
- Grow the tests from 5 to 13: caplog coverage for the Langfuse-conflict and
  off-box warnings, exporter validation cases, a stronger import-time
  regression that asserts the global TracerProvider is not replaced, and a
  subprocess double-invoke test exercising the real check_duplicate_setup.
- Docs: config.example.yaml block retitled to a dedicated tracing header;
  README documents the [monocle] install and scopes tracing to Gateway runs.

* docs: align Monocle README section with the other tracing providers

Lead with what Monocle is and captures, drop the install step (the dev
group already ships monocle_apptrace via uv sync; unusual installs get
the RuntimeError), and point the missing-package error at the repo-native
command (uv sync --extra monocle / deerflow-harness[monocle]).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address review: verified Langfuse coexistence, lifespan test, scope docs

Responds to the third review round.

The Langfuse conflict claim was wrong, verified empirically against langfuse
4.5.1 in both init orders: whichever library initializes second reuses the
existing global TracerProvider and attaches its own span processor, so neither
side loses spans. Dropped the warning and its tests, corrected the README,
AGENTS.md, and config.example.yaml statements, and pinned the verified behavior
with test_coexists_with_langfuse (real monocle + real langfuse in a subprocess,
no mocks). One honest caveat documented: both processors see all spans, so
Monocle's exporters also capture Langfuse's spans when both are enabled.

Also from the review:
- Document the Gateway-only scope in AGENTS.md: the lifespan is the sole call
  site, so the embedded DeerFlowClient and TUI are not instrumented; embedded
  users call setup_monocle_tracing_if_enabled() themselves.
- Add test_gateway_lifespan_initializes_monocle pinning the lifespan wiring.
- Comment why MonocleTracingConfig.is_configured is intentionally coarser than
  LangSmith/Langfuse (composite validation lives in validate() at startup).
- Note that monocle_exporters_list takes the comma-separated string as-is.
- Module-level importorskip("monocle_apptrace") so minimal installs collect
  the test module cleanly.

* docs: reword Monocle intro sentence

* fix(tests): run the import-time regression in a subprocess

test_no_import_time_setup deleted deerflow.agents* from sys.modules and
re-imported to force __init__ to re-execute. The re-import creates new module
objects, and restoring the old sys.modules entries afterwards leaves the parent
package's attribute bindings pointing at the new ones, so any later test that
resolves a deerflow.agents.* dotted path (monkeypatch.setattr in
test_summarization_middleware, test_thread_data_middleware, and others) failed
with "module 'deerflow.agents' has no attribute ...".

Run the check in a subprocess instead: the import is genuinely fresh, the
assertion is stronger (the provider must still be the SDK-less proxy, proving
nothing was installed at any point), and no module identity leaks into the
rest of the suite.

* Address review: console warning scope, embedded hint, honest naming, doc alignment

Responds to the post-approval review round:

- Scope the off-box exporter warning to the remote exporters (okahu, s3,
  blob, gcs): console writes to local stdout and no longer trips it.
  config.example.yaml's data-handling note now distinguishes file /
  console / remote likewise.
- Rename MonocleTracingConfig.is_configured to is_enabled so the boolean
  reads as what it checks; the exporter-dependent credential check stays
  in validate(), run at Gateway startup.
- Hint on the embedded path: build_tracing_callbacks() logs a debug line
  when MONOCLE_TRACING is set but setup never ran in this process, so
  embedded DeerFlowClient/TUI users are not left with silent no-op
  tracing. Backed by a process-global setup flag.
- Re-export setup_monocle_tracing_if_enabled from deerflow.tracing,
  matching the package convention.
- Note the deliberate fail-open-at-startup contrast with
  LangSmith/Langfuse in the lifespan, and the OTel SDK-internals
  dependency in the coexistence test.
- Test hygiene: clear MONOCLE_* env in the tracing config/factory
  fixtures; reset the setup flag in the monocle test fixture; reword the
  README Langfuse-spans claim as the shared-provider inference it is.
- Document that .monocle/ trace files are never rotated or cleaned up.

* fix(tests): pin the factory logger level in the embedded-hint tests

configure_logging() from earlier tests in the full suite pins an explicit
INFO level on the logger hierarchy, so a root-level caplog.at_level(DEBUG)
never sees the factory's debug hint. Scope caplog to
deerflow.tracing.factory so the test is independent of suite ordering.

* Address review: co-export disclosure, lifespan failure test, exporter parse dedup

- Off-box warning now notes that Langfuse's spans are exported too when
  both providers are enabled and share the global OTel provider; pinned
  both ways by tests.
- Pin the lifespan fail-open contract: a raising Monocle setup is logged
  and the Gateway keeps serving (pragma dropped now that the path is
  exercised). README notes a config error is reported at startup and
  tracing stays off until restart.
- Hoist exporter parsing into MonocleTracingConfig.exporter_list so
  validate() and the off-box warning cannot diverge, and note the
  upstream coupling on the exporter allow-list.
- Reduce config.example.yaml's Monocle block to a pointer; the capture,
  retention, and data-handling detail lives in README's Monocle section.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-14 08:58:06 +08:00

213 lines
7.3 KiB
Python

import os
import threading
from pydantic import BaseModel, Field
_config_lock = threading.Lock()
class LangSmithTracingConfig(BaseModel):
"""Configuration for LangSmith tracing."""
enabled: bool = Field(...)
api_key: str | None = Field(...)
project: str = Field(...)
endpoint: str = Field(...)
@property
def is_configured(self) -> bool:
return self.enabled and bool(self.api_key)
def validate(self) -> None:
if self.enabled and not self.api_key:
raise ValueError("LangSmith tracing is enabled but LANGSMITH_API_KEY (or LANGCHAIN_API_KEY) is not set.")
class LangfuseTracingConfig(BaseModel):
"""Configuration for Langfuse tracing."""
enabled: bool = Field(...)
public_key: str | None = Field(...)
secret_key: str | None = Field(...)
host: str = Field(...)
@property
def is_configured(self) -> bool:
return self.enabled and bool(self.public_key) and bool(self.secret_key)
def validate(self) -> None:
if not self.enabled:
return
missing: list[str] = []
if not self.public_key:
missing.append("LANGFUSE_PUBLIC_KEY")
if not self.secret_key:
missing.append("LANGFUSE_SECRET_KEY")
if missing:
raise ValueError(f"Langfuse tracing is enabled but required settings are missing: {', '.join(missing)}")
# Manual mirror of monocle_apptrace's supported exporters, kept local so a typo
# fails at startup with a clear message instead of an opaque upstream error.
# Update this tuple when a monocle_apptrace bump adds or renames an exporter.
_MONOCLE_EXPORTERS = ("file", "console", "okahu", "s3", "blob", "gcs")
class MonocleTracingConfig(BaseModel):
"""Configuration for Monocle telemetry."""
enabled: bool = Field(...)
exporters: str = Field(...)
okahu_api_key: str | None = Field(...)
@property
def is_enabled(self) -> bool:
# Unlike the siblings' is_configured, no credential check here: that is
# exporter-dependent and lives in validate(), run at Gateway startup.
return self.enabled
@property
def exporter_list(self) -> list[str]:
"""The configured exporters, parsed once so validation and setup agree."""
return [e.strip() for e in self.exporters.split(",") if e.strip()]
def validate(self) -> None:
if not self.enabled:
return
selected = self.exporter_list
unknown = [e for e in selected if e not in _MONOCLE_EXPORTERS]
if unknown:
raise ValueError(f"MONOCLE_EXPORTERS has unknown exporter(s): {', '.join(unknown)}. Allowed: {', '.join(_MONOCLE_EXPORTERS)}.")
if "okahu" in selected and not self.okahu_api_key:
raise ValueError("Monocle 'okahu' exporter is selected but OKAHU_API_KEY is not set.")
class TracingConfig(BaseModel):
"""Tracing configuration for supported providers."""
langsmith: LangSmithTracingConfig = Field(...)
langfuse: LangfuseTracingConfig = Field(...)
monocle: MonocleTracingConfig = Field(...)
@property
def is_configured(self) -> bool:
return bool(self.enabled_providers)
@property
def explicitly_enabled_providers(self) -> list[str]:
enabled: list[str] = []
if self.langsmith.enabled:
enabled.append("langsmith")
if self.langfuse.enabled:
enabled.append("langfuse")
return enabled
@property
def enabled_providers(self) -> list[str]:
enabled: list[str] = []
if self.langsmith.is_configured:
enabled.append("langsmith")
if self.langfuse.is_configured:
enabled.append("langfuse")
return enabled
def validate_enabled(self) -> None:
self.langsmith.validate()
self.langfuse.validate()
_tracing_config: TracingConfig | None = None
_TRUTHY_VALUES = {"1", "true", "yes", "on"}
def _env_flag_preferred(*names: str) -> bool:
"""Return the boolean value of the first env var that is present and non-empty."""
for name in names:
value = os.environ.get(name)
if value is not None and value.strip():
return value.strip().lower() in _TRUTHY_VALUES
return False
def _first_env_value(*names: str) -> str | None:
"""Return the first non-empty environment value from candidate names."""
for name in names:
value = os.environ.get(name)
if value and value.strip():
return value.strip()
return None
def get_tracing_config() -> TracingConfig:
"""Get the current tracing configuration from environment variables."""
global _tracing_config
if _tracing_config is not None:
return _tracing_config
with _config_lock:
if _tracing_config is not None:
return _tracing_config
_tracing_config = TracingConfig(
langsmith=LangSmithTracingConfig(
enabled=_env_flag_preferred("LANGSMITH_TRACING", "LANGCHAIN_TRACING_V2", "LANGCHAIN_TRACING"),
api_key=_first_env_value("LANGSMITH_API_KEY", "LANGCHAIN_API_KEY"),
project=_first_env_value("LANGSMITH_PROJECT", "LANGCHAIN_PROJECT") or "deer-flow",
endpoint=_first_env_value("LANGSMITH_ENDPOINT", "LANGCHAIN_ENDPOINT") or "https://api.smith.langchain.com",
),
langfuse=LangfuseTracingConfig(
enabled=_env_flag_preferred("LANGFUSE_TRACING"),
public_key=_first_env_value("LANGFUSE_PUBLIC_KEY"),
secret_key=_first_env_value("LANGFUSE_SECRET_KEY"),
host=_first_env_value("LANGFUSE_BASE_URL") or "https://cloud.langfuse.com",
),
monocle=MonocleTracingConfig(
enabled=_env_flag_preferred("MONOCLE_TRACING"),
exporters=_first_env_value("MONOCLE_EXPORTERS") or "file",
okahu_api_key=_first_env_value("OKAHU_API_KEY"),
),
)
return _tracing_config
def get_enabled_tracing_providers() -> list[str]:
"""Return the configured tracing providers that are enabled and complete."""
return get_tracing_config().enabled_providers
def get_explicitly_enabled_tracing_providers() -> list[str]:
"""Return tracing providers explicitly enabled by config, even if incomplete."""
return get_tracing_config().explicitly_enabled_providers
def validate_enabled_tracing_providers() -> None:
"""Validate that any explicitly enabled providers are fully configured."""
get_tracing_config().validate_enabled()
def is_tracing_enabled() -> bool:
"""Check if any tracing provider is enabled and fully configured."""
return get_tracing_config().is_configured
def is_monocle_tracing_enabled() -> bool:
"""Whether Monocle OTel observability is enabled (via ``MONOCLE_TRACING``).
Kept separate from :func:`get_enabled_tracing_providers` because Monocle is a
process-global instrumentor activated at startup, not a per-run LangChain
callback.
"""
return get_tracing_config().monocle.is_enabled
def reset_tracing_config() -> None:
"""Discard the cached :class:`TracingConfig` so the next call rebuilds it.
Public API so that tests do not have to reach into the private
``_tracing_config`` module attribute. A future internal rename would
silently break callers that mutate the attribute directly.
"""
global _tracing_config
with _config_lock:
_tracing_config = None