2 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
3 people	867235389d	Add trace-based behavioral tests with Monocle Test Tools (#4025) ... * Add Monocle behavioral test suite Trace-based tests under tests/monocle/ asserting against the agent's Monocle execution traces: 4 offline tests load a recorded trace by file (with_trace_source), one per curated question, plus 1 live end-to-end test. Fluent structural asserts (agent, tools, input/output, token/duration budget); additive only, no app-code changes. * Address review: restructure suite, move under backend/tests/monocle/ Responds to the review on this PR. Behavioural coverage is now the live tests, not frozen fixtures. Keep one offline test as a worked example of the fluent assertion API (loads a recorded trace by file, no keys), and add two live tests that drive the agent end-to-end through DeerFlowClient and assert on the trace the real run emits (web-research and sandbox paths). The live tests skip without OPENAI_API_KEY or the app. Other review fixes: - Move the suite under backend/tests/monocle/ so backend pytest collects it. - Split helpers into _helpers.py; conftest is fixtures-only (run_agent), with Monocle setup owned by the validator and .env load scoped to the live path. - Resolve the model from config.yaml instead of hardcoding gpt-4o; skip the live tests when config.yaml is absent. - Keep one trace with a stable name (web_research_ev_battery.json); drop the other three (removes ~2,200 lines of fixture blobs). - Drop the flaky wall-clock duration bound on live runs. - Keep monocle_test_tools in a standalone requirements.txt rather than the backend dev group: it hard-depends on the ML eval stack (torch, transformers, sentence-transformers, ~48 packages, +950 lines in uv.lock), so isolating it keeps the app's locked deps clean. importorskip skips the suite when absent. * docs(monocle tests): explain the golden-trace workflow Add a "How this is meant to be used" section: capture a run you are happy with as a golden, labelled trace, turn it into assertions (the offline example), then point the same assertions at the live agent so every later run has to reproduce that behaviour. * Address review: fix docstring pytest paths, state the suite is not run in CI The docstring commands now use the backend/tests/monocle/ form (matching the README, which also gains the backend-dir uv variant), and the README states explicitly that the suite is skipped in CI and run on demand. * Address review: document the committed trace, pin assertions context - README: new section on the committed trace. It is a full, unmodified real-run recording (system prompt of the recording date + fetched web content, no credentials), committed whole so the offline example parses a genuine trace. The offline assertions are pinned to this trace and the monocle_apptrace 0.8.8 span shapes; re-record when prompt, tools, or model change. - README: note that the monocle_trace_asserter fixture comes from monocle_test_tools' auto-registered pytest plugin (pytest11 entry point). - test_deerflow.py: comment why web_fetch asserts min_count=2 rather than the recorded exact count of 5 (fetch counts vary run to run; keep it a floor). - requirements.txt: loose pin python-dotenv>=1.0. * Address review: make live tests explicit opt-in, drop OpenAI-only gate Two execution-gating fixes from review: - Live tests are now opt-in via MONOCLE_LIVE_TESTS=1 (default off). Previously the documented offline command collected the live tests too, and on a configured checkout (.env + config.yaml present) they would run for real, spending model tokens and hitting the network. Now the plain `pytest backend/tests/monocle/` run cannot go live regardless of what credentials are present; test_live_gate_defaults_off pins the gate. - The run_agent fixture no longer requires OPENAI_API_KEY. config.yaml resolves the model, which may be any provider (Anthropic, Gemini, Volcengine, ...), so a hard-coded OpenAI gate skipped valid configurations and passed invalid ones. Credentials are validated by the configured model itself. README and docstrings updated to match: offline command is offline by construction, live is MONOCLE_LIVE_TESTS=1, credentials described as the configured model's rather than OpenAI's. Verified both modes: default run is 2 passed 2 skipped with no network; opted in, all 4 pass with real end-to-end runs. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-19 18:26:26 +08:00
4 people	4af6178358	feat(trace): add agent observability with Monocle (#4024) ... * Add Monocle tracing Enable Monocle (OpenTelemetry tracing for LLM apps) with one setup call plus the monocle_apptrace dependency. setup_monocle_telemetry auto-instruments the frameworks already in use and writes traces to .monocle/. Additive; no changes to application logic. * Config-gate Monocle telemetry in the Gateway lifespan Addresses review on #4024: moves setup_monocle_telemetry out of agents/__init__ import time into the Gateway lifespan, gated by MonocleTracingConfig (MONOCLE_TRACING env, default off). Warns on the Langfuse/global-OTel-provider conflict and relies on monocle_apptrace's own duplicate-setup guard and existing-provider attach. Pins monocle_apptrace>=0.8.8 (+ uv.lock), adds .monocle/ to .gitignore, adds tests (default-off / toggle-on / no import-time setup), and documents exporters, Okahu, and the VS Code viewer in README, config.example.yaml, and backend/AGENTS.md. * Clarify Monocle/Langfuse single-provider guidance Make the docstring, warning, and AGENTS.md consistent with the README: only one library can own the global OpenTelemetry provider; Monocle initializes at startup before Langfuse's per-run handler, so enabling both drops Langfuse's spans — enable one OTel tracer (LangSmith, a callback, coexists fine). * Address review: optional extra, exporter validation, off-box warning, tests Responds to the second review round. - Make monocle_apptrace an optional extra (deerflow-harness[monocle], re-exposed as deer-flow[monocle]) following the boxlite/tui precedent, so a default install no longer pulls the OpenTelemetry stack. It stays pinned in the dev group for the tracing tests, and enabling MONOCLE_TRACING without the extra raises a clear install error. - Warn loudly at startup whenever any exporter other than `file` is configured, since those move prompts, tool inputs/outputs, and completions beyond the local .monocle/ directory. - Validate MONOCLE_EXPORTERS against the known exporter names and require OKAHU_API_KEY when okahu is selected, mirroring the Langfuse pattern. Validation runs from Monocle's own init (not validate_enabled) so a config typo can never fail agent runs; errors surface at Gateway startup instead. - Grow the tests from 5 to 13: caplog coverage for the Langfuse-conflict and off-box warnings, exporter validation cases, a stronger import-time regression that asserts the global TracerProvider is not replaced, and a subprocess double-invoke test exercising the real check_duplicate_setup. - Docs: config.example.yaml block retitled to a dedicated tracing header; README documents the [monocle] install and scopes tracing to Gateway runs. * docs: align Monocle README section with the other tracing providers Lead with what Monocle is and captures, drop the install step (the dev group already ships monocle_apptrace via uv sync; unusual installs get the RuntimeError), and point the missing-package error at the repo-native command (uv sync --extra monocle / deerflow-harness[monocle]). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review: verified Langfuse coexistence, lifespan test, scope docs Responds to the third review round. The Langfuse conflict claim was wrong, verified empirically against langfuse 4.5.1 in both init orders: whichever library initializes second reuses the existing global TracerProvider and attaches its own span processor, so neither side loses spans. Dropped the warning and its tests, corrected the README, AGENTS.md, and config.example.yaml statements, and pinned the verified behavior with test_coexists_with_langfuse (real monocle + real langfuse in a subprocess, no mocks). One honest caveat documented: both processors see all spans, so Monocle's exporters also capture Langfuse's spans when both are enabled. Also from the review: - Document the Gateway-only scope in AGENTS.md: the lifespan is the sole call site, so the embedded DeerFlowClient and TUI are not instrumented; embedded users call setup_monocle_tracing_if_enabled() themselves. - Add test_gateway_lifespan_initializes_monocle pinning the lifespan wiring. - Comment why MonocleTracingConfig.is_configured is intentionally coarser than LangSmith/Langfuse (composite validation lives in validate() at startup). - Note that monocle_exporters_list takes the comma-separated string as-is. - Module-level importorskip("monocle_apptrace") so minimal installs collect the test module cleanly. * docs: reword Monocle intro sentence * fix(tests): run the import-time regression in a subprocess test_no_import_time_setup deleted deerflow.agents* from sys.modules and re-imported to force __init__ to re-execute. The re-import creates new module objects, and restoring the old sys.modules entries afterwards leaves the parent package's attribute bindings pointing at the new ones, so any later test that resolves a deerflow.agents.* dotted path (monkeypatch.setattr in test_summarization_middleware, test_thread_data_middleware, and others) failed with "module 'deerflow.agents' has no attribute ...". Run the check in a subprocess instead: the import is genuinely fresh, the assertion is stronger (the provider must still be the SDK-less proxy, proving nothing was installed at any point), and no module identity leaks into the rest of the suite. * Address review: console warning scope, embedded hint, honest naming, doc alignment Responds to the post-approval review round: - Scope the off-box exporter warning to the remote exporters (okahu, s3, blob, gcs): console writes to local stdout and no longer trips it. config.example.yaml's data-handling note now distinguishes file / console / remote likewise. - Rename MonocleTracingConfig.is_configured to is_enabled so the boolean reads as what it checks; the exporter-dependent credential check stays in validate(), run at Gateway startup. - Hint on the embedded path: build_tracing_callbacks() logs a debug line when MONOCLE_TRACING is set but setup never ran in this process, so embedded DeerFlowClient/TUI users are not left with silent no-op tracing. Backed by a process-global setup flag. - Re-export setup_monocle_tracing_if_enabled from deerflow.tracing, matching the package convention. - Note the deliberate fail-open-at-startup contrast with LangSmith/Langfuse in the lifespan, and the OTel SDK-internals dependency in the coexistence test. - Test hygiene: clear MONOCLE_* env in the tracing config/factory fixtures; reset the setup flag in the monocle test fixture; reword the README Langfuse-spans claim as the shared-provider inference it is. - Document that .monocle/ trace files are never rotated or cleaned up. * fix(tests): pin the factory logger level in the embedded-hint tests configure_logging() from earlier tests in the full suite pins an explicit INFO level on the logger hierarchy, so a root-level caplog.at_level(DEBUG) never sees the factory's debug hint. Scope caplog to deerflow.tracing.factory so the test is independent of suite ordering. * Address review: co-export disclosure, lifespan failure test, exporter parse dedup - Off-box warning now notes that Langfuse's spans are exported too when both providers are enabled and share the global OTel provider; pinned both ways by tests. - Pin the lifespan fail-open contract: a raising Monocle setup is logged and the Gateway keeps serving (pragma dropped now that the path is exercised). README notes a config error is reported at startup and tracing stays off until restart. - Hoist exporter parsing into MonocleTracingConfig.exporter_list so validate() and the off-box warning cannot diverge, and note the upstream coupling on the exporter allow-list. - Reduce config.example.yaml's Monocle block to a pointer; the capture, retention, and data-handling detail lives in README's Monocle section. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-14 08:58:06 +08:00