Add trace-based behavioral tests with Monocle Test Tools (#4025)

* Add Monocle behavioral test suite

Trace-based tests under tests/monocle/ asserting against the agent's Monocle execution traces: 4 offline tests load a recorded trace by file (with_trace_source), one per curated question, plus 1 live end-to-end test. Fluent structural asserts (agent, tools, input/output, token/duration budget); additive only, no app-code changes.

* Address review: restructure suite, move under backend/tests/monocle/

Responds to the review on this PR.

Behavioural coverage is now the live tests, not frozen fixtures. Keep one
offline test as a worked example of the fluent assertion API (loads a recorded
trace by file, no keys), and add two live tests that drive the agent end-to-end
through DeerFlowClient and assert on the trace the real run emits (web-research
and sandbox paths). The live tests skip without OPENAI_API_KEY or the app.

Other review fixes:
- Move the suite under backend/tests/monocle/ so backend pytest collects it.
- Split helpers into _helpers.py; conftest is fixtures-only (run_agent), with
  Monocle setup owned by the validator and .env load scoped to the live path.
- Resolve the model from config.yaml instead of hardcoding gpt-4o; skip the
  live tests when config.yaml is absent.
- Keep one trace with a stable name (web_research_ev_battery.json); drop the
  other three (removes ~2,200 lines of fixture blobs).
- Drop the flaky wall-clock duration bound on live runs.
- Keep monocle_test_tools in a standalone requirements.txt rather than the
  backend dev group: it hard-depends on the ML eval stack (torch, transformers,
  sentence-transformers, ~48 packages, +950 lines in uv.lock), so isolating it
  keeps the app's locked deps clean. importorskip skips the suite when absent.

* docs(monocle tests): explain the golden-trace workflow

Add a "How this is meant to be used" section: capture a run you are happy with
as a golden, labelled trace, turn it into assertions (the offline example), then
point the same assertions at the live agent so every later run has to reproduce
that behaviour.

* Address review: fix docstring pytest paths, state the suite is not run in CI

The docstring commands now use the backend/tests/monocle/ form (matching
the README, which also gains the backend-dir uv variant), and the README
states explicitly that the suite is skipped in CI and run on demand.

* Address review: document the committed trace, pin assertions context

- README: new section on the committed trace. It is a full, unmodified
  real-run recording (system prompt of the recording date + fetched web
  content, no credentials), committed whole so the offline example parses a
  genuine trace. The offline assertions are pinned to this trace and the
  monocle_apptrace 0.8.8 span shapes; re-record when prompt, tools, or model
  change.
- README: note that the monocle_trace_asserter fixture comes from
  monocle_test_tools' auto-registered pytest plugin (pytest11 entry point).
- test_deerflow.py: comment why web_fetch asserts min_count=2 rather than the
  recorded exact count of 5 (fetch counts vary run to run; keep it a floor).
- requirements.txt: loose pin python-dotenv>=1.0.

* Address review: make live tests explicit opt-in, drop OpenAI-only gate

Two execution-gating fixes from review:

- Live tests are now opt-in via MONOCLE_LIVE_TESTS=1 (default off). Previously
  the documented offline command collected the live tests too, and on a
  configured checkout (.env + config.yaml present) they would run for real,
  spending model tokens and hitting the network. Now the plain
  `pytest backend/tests/monocle/` run cannot go live regardless of what
  credentials are present; test_live_gate_defaults_off pins the gate.
- The run_agent fixture no longer requires OPENAI_API_KEY. config.yaml resolves
  the model, which may be any provider (Anthropic, Gemini, Volcengine, ...), so
  a hard-coded OpenAI gate skipped valid configurations and passed invalid
  ones. Credentials are validated by the configured model itself.

README and docstrings updated to match: offline command is offline by
construction, live is MONOCLE_LIVE_TESTS=1, credentials described as the
configured model's rather than OpenAI's.

Verified both modes: default run is 2 passed 2 skipped with no network; opted
in, all 4 pass with real end-to-end runs.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

backend/tests/monocle/README.md

@@ -0,0 +1,126 @@
+# DeerFlow behavioural tests (Monocle Test Tools)
+
+Trace-based tests for DeerFlow. Monocle records each run as a structured trace
+(the agent invocation, every tool call, token usage, timings), and these tests
+assert against that trace with [Monocle Test Tools](https://github.com/monocle2ai/monocle).
+
+## How this is meant to be used
+
+Instrument the agent with Monocle and run it against a question. Once it answers
+the way you expect and makes the agent and tool calls you expect, capture that
+run as a trace. That trace is a golden, labelled reference for the question: a
+record of correct behaviour, not just sample data. You turn it into assertions
+(the offline example shows how), and then you point those same assertions at the
+live agent for the same question, so every later run has to reproduce that
+behaviour. The offline test is where you pin down what good looks like; the live
+test is what enforces it against a real run.
+
+## Layers
+
+The suite has two:
+
+- **One offline example** (`test_assertion_api_example`) loads a recorded trace
+  from file and shows the full fluent vocabulary in one place. It needs no keys
+  and no network. Because it asserts against frozen JSON, it guards the trace
+  format and the asserter wiring, not DeerFlow's behaviour. Treat it as the
+  worked example for writing your own assertions.
+- **Two live tests** drive the agent end-to-end and assert on the trace the real
+  run emits. These are the behavioural guards: a change that alters routing, tool
+  selection, or token cost is caught here. They are explicit opt-in via
+  `MONOCLE_LIVE_TESTS=1` and skip by default, so a plain run never spends model
+  tokens or hits the network, even on a fully configured checkout.
+
+## Layout
+
+- `test_deerflow.py` — the offline example + two live tests
+- `conftest.py` — the `run_agent` fixture (live path only)
+- `_helpers.py` — paths and `run_deerflow()`
+- `traces/` — the recorded trace the offline example loads
+- `requirements.txt` — standalone dependencies
+
+## The committed trace
+
+`traces/web_research_ev_battery.json` is a full, unmodified recording of a real
+run, committed whole so the offline example parses a genuine trace. That means
+it embeds the DeerFlow system prompt as of the recording date and the content
+the run fetched from the web, alongside the span structure the assertions read.
+It contains no credentials.
+
+The offline assertions are pinned to this exact trace and to the
+`monocle_apptrace` 0.8.8 span shapes: the `LangGraph` agent span name, the tool
+names, and the input phrasing. A rename of any of those breaks the offline test
+even when behaviour is unchanged; re-record the trace when the prompt, tools, or
+model change.
+
+## Run
+
+`monocle_test_tools` hard-depends on the ML eval stack (torch, transformers,
+sentence-transformers), so it is a standalone `requirements.txt` install rather
+than a backend dependency. When it is absent (e.g. a plain backend venv) the
+whole suite skips cleanly via `pytest.importorskip`.
+
+Because that dependency is deliberately absent from the backend deps, **none of
+these tests run in CI** — `make test` collects and skips the whole module,
+including the offline example. This is an on-demand suite: install the
+requirements and run it locally (or wire a dedicated CI job with the
+requirements installed) when changing agent behaviour, tools, or routing.
+
+```bash
+# from the repo root
+pip install -r backend/tests/monocle/requirements.txt
+
+# offline — no network, no keys; the live tests skip unless opted in
+pytest backend/tests/monocle/
+
+# opt in to the live behavioural tests (real model calls + web requests)
+MONOCLE_LIVE_TESTS=1 pytest backend/tests/monocle/
+```
+
+Or, following the backend convention (from `backend/`, with uv):
+
+```bash
+uv pip install -r tests/monocle/requirements.txt
+uv run pytest tests/monocle/                          # offline
+MONOCLE_LIVE_TESTS=1 uv run pytest tests/monocle/     # + live
+```
+
+The live tests are opt-in by design: without `MONOCLE_LIVE_TESTS=1` they skip
+even on a checkout where credentials and `config.yaml` are present, so the
+default command can never spend tokens or write to a sandbox. When opted in,
+they still skip if the DeerFlow app is not importable or `config.yaml` is
+missing. Model credentials are validated by the configured model itself —
+`config.yaml` may select any provider (OpenAI, Anthropic, Gemini, and so on),
+so there is no hard-coded key requirement. DeerFlow's `web_search` is
+DuckDuckGo and needs no key of its own.
+
+The `monocle_trace_asserter` fixture is provided by `monocle_test_tools`' own
+pytest plugin, which registers automatically on install (a `pytest11` entry
+point); no `pytest_plugins` configuration is needed.
+
+## Add your own test
+
+1. Run DeerFlow under Monocle and capture a trace of a run you are happy with
+   (Monocle writes trace JSON to `.monocle/` by default).
+2. For an offline example, move it into `traces/` and load it with
+   `monocle_trace_asserter.with_trace_source("file", trace_path=path)`.
+3. For a behavioural test, drive the agent live via the `run_agent` fixture and
+   `monocle_trace_asserter.validator.test_workflow(run_agent, {"test_input": (...)})`.
+4. Assert with the fluent API: `called_agent(...)`, `called_tool(...)`,
+   `contains_input` / `contains_any_output(...)`, `under_token_limit(...)`,
+   `under_duration(..., span_type="workflow")`.
+
+## Evaluations (note)
+
+Structural assertions are the coverage here. Content/quality evaluations are
+**not** wired in this suite because, on the current `monocle_test_tools`, local
+evals do **not** compose with file-loaded traces:
+
+- Declarative `test_spans[].eval` (`comparer:"metric"`) is silently ignored by
+  `validator.validate()` — `_evaluate_span` has no call sites, so an assertion
+  that should fail (e.g. a required keyword that is absent) passes vacuously.
+- The fluent `check_eval()` path is wired for the Okahu eval-service signature
+  (`filtered_spans=`), which the local evaluators (`keyword_presence`, etc.) do
+  not accept — it raises `TypeError`.
+
+So local evals are omitted rather than added as vacuous no-ops. The Okahu eval
+layer (needs `OKAHU_API_KEY`) remains an option for content grading.

backend/tests/monocle/_helpers.py

@@ -0,0 +1,42 @@
+"""Helpers for the DeerFlow Monocle behavioural tests.
+
+Kept out of ``conftest.py`` so nothing imports ``conftest`` as a module.
+Monocle instrumentation is owned by the Test Tools validator (installed by the
+``monocle_trace_asserter`` fixture), so ``run_deerflow`` only drives the agent;
+the already-installed instrumentation captures the run's spans.
+"""
+
+from __future__ import annotations
+
+import os
+import uuid
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+TRACES = HERE / "traces"
+REPO_ROOT = HERE.parents[2]  # backend/tests/monocle -> backend/tests -> backend -> repo root
+CONFIG_PATH = REPO_ROOT / "config.yaml"
+
+_TRUTHY = {"1", "true", "yes", "on"}
+
+
+def live_tests_enabled() -> bool:
+    """Whether the live tests are explicitly opted into via ``MONOCLE_LIVE_TESTS``.
+
+    Off by default so the plain ``pytest backend/tests/monocle/`` run can never
+    spend model tokens, hit the network, or write to a sandbox — even on a fully
+    configured checkout where credentials and ``config.yaml`` are present.
+    """
+    return os.getenv("MONOCLE_LIVE_TESTS", "").strip().lower() in _TRUTHY
+
+
+def run_deerflow(message: str) -> str:
+    """Run the DeerFlow agent once and return its response text.
+
+    The model is resolved from ``config.yaml`` (no hardcoded override) so the
+    live test exercises DeerFlow's own model-resolution path.
+    """
+    from deerflow.client import DeerFlowClient
+
+    client = DeerFlowClient(config_path=str(CONFIG_PATH))
+    return client.chat(message, thread_id=f"monocle-test-{uuid.uuid4().hex[:8]}")

backend/tests/monocle/conftest.py

@@ -0,0 +1,42 @@
+"""Fixtures for the DeerFlow Monocle behavioural tests.
+
+Only fixtures live here. Paths and ``run_deerflow`` are in ``_helpers.py`` so
+nothing imports ``conftest`` as a module. The ``sys.path`` insert (mirroring the
+backend root ``conftest.py``) makes ``_helpers`` importable under any pytest
+import mode. The ``.env`` load is scoped to the live fixture, so collecting or
+running the offline test never reads secrets.
+"""
+
+from __future__ import annotations
+
+import sys
+from collections.abc import Callable
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+
+@pytest.fixture
+def run_agent() -> Callable[[str], str]:
+    """Live agent runner. Explicit opt-in, so a default run can never go live.
+
+    Skips unless ``MONOCLE_LIVE_TESTS=1`` is set, when the DeerFlow app is not
+    importable (e.g. a test-tools-only venv), or when ``config.yaml`` is absent.
+    Provider credentials are validated by the configured model itself —
+    ``config.yaml`` may select any provider, not just OpenAI, so there is no
+    hard-coded key check here.
+    """
+    from _helpers import CONFIG_PATH, REPO_ROOT, live_tests_enabled, run_deerflow
+
+    if not live_tests_enabled():
+        pytest.skip("live tests are opt-in: set MONOCLE_LIVE_TESTS=1")
+    pytest.importorskip("deerflow", reason="DeerFlow app not importable in this venv")
+
+    from dotenv import load_dotenv
+
+    load_dotenv(REPO_ROOT / ".env")
+    if not CONFIG_PATH.exists():
+        pytest.skip(f"config.yaml not found at {CONFIG_PATH}")
+    return run_deerflow

backend/tests/monocle/requirements.txt

@@ -0,0 +1,13 @@
+# Standalone deps for the trace-based behavioural suite (tests/monocle/).
+#
+# Kept out of backend/pyproject.toml on purpose: monocle_test_tools hard-depends
+# on the ML eval stack (bert-score, sentence-transformers, transformers -> torch
+# + the CUDA wheels, ~48 packages, +950 lines in uv.lock). Isolating it here
+# keeps that weight out of the app's locked deps. The suite skips cleanly
+# (pytest.importorskip) when this is not installed.
+#
+# Pinned to 0.8.8: the file trace source the offline example loads
+# (with_trace_source("file", trace_path=...)) does not exist in 0.7.x.
+monocle_test_tools==0.8.8
+# Auto-loads the repo .env for the live tests (optional).
+python-dotenv>=1.0

backend/tests/monocle/test_deerflow.py

@@ -0,0 +1,113 @@
+"""Trace-based behavioural tests for DeerFlow, using Monocle Test Tools.
+
+Two layers:
+
+* One **offline example** (``test_assertion_api_example``) loads a recorded
+  trace from file and shows the full fluent vocabulary in one place. It needs no
+  keys and no network, but because it asserts against frozen JSON it guards the
+  trace format and the asserter wiring, not DeerFlow's behaviour. Treat it as the
+  worked example for writing your own assertions.
+* Two **live tests** drive the agent end-to-end through ``run_agent`` and assert
+  on the trace the real run emits. These are the behavioural guards: a change
+  that alters routing, tool selection, or token cost is caught here. They are
+  **explicit opt-in** via ``MONOCLE_LIVE_TESTS=1`` (default off, so a plain run
+  never spends tokens or hits the network) and need the DeerFlow app plus the
+  configured model's credentials.
+
+The whole module is skipped when ``monocle_test_tools`` is not installed (see the
+``importorskip`` below), so a plain backend venv collects it without error.
+
+    pytest backend/tests/monocle/                                  # offline only
+    MONOCLE_LIVE_TESTS=1 pytest backend/tests/monocle/             # + live tests
+
+See ``README.md`` for how to add your own.
+"""
+
+from pathlib import Path
+
+import pytest
+
+# monocle_test_tools hard-depends on the ML eval stack (torch, transformers,
+# sentence-transformers), so it is a standalone requirements.txt install rather
+# than a backend dependency. Skip the whole module when it is not present (e.g.
+# a plain backend CI venv) instead of erroring at collection.
+pytest.importorskip("monocle_test_tools", reason="pip install -r tests/monocle/requirements.txt")
+
+from _helpers import live_tests_enabled  # noqa: E402
+from monocle_test_tools import TraceAssertion  # noqa: E402
+
+TRACES = Path(__file__).resolve().parent / "traces"
+EXAMPLE_TRACE = str(TRACES / "web_research_ev_battery.json")
+
+
+def test_live_gate_defaults_off(monkeypatch):
+    """The live tests must be opt-in: gate closed by default, open only on the flag.
+
+    This is what keeps the plain ``pytest backend/tests/monocle/`` run incapable
+    of model calls, web requests, or sandbox writes, even on a checkout where
+    credentials and ``config.yaml`` are present.
+    """
+    monkeypatch.delenv("MONOCLE_LIVE_TESTS", raising=False)
+    assert live_tests_enabled() is False
+    monkeypatch.setenv("MONOCLE_LIVE_TESTS", "1")
+    assert live_tests_enabled() is True
+    monkeypatch.setenv("MONOCLE_LIVE_TESTS", "0")
+    assert live_tests_enabled() is False
+
+
+# --- Offline example: the full assertion vocabulary against a recorded trace ---
+
+
+def test_assertion_api_example(monocle_trace_asserter: TraceAssertion):
+    """Worked example: every fluent assertion this suite uses, in one place.
+
+    Loads a recorded web-research run (solid-state EV battery briefing) and
+    asserts which agent ran, what it was asked and produced, which tools it
+    called (and did not), and its token/duration budget. Copy this shape when
+    writing a behavioural test — then point it at a live run (see the live tests
+    below) so it actually guards behaviour.
+    """
+    monocle_trace_asserter.with_trace_source("file", trace_path=EXAMPLE_TRACE)
+
+    monocle_trace_asserter.called_agent("LangGraph").contains_input("solid-state EV batteries")
+    monocle_trace_asserter.contains_any_output("solid-state", "battery", "batteries", "EV")
+    monocle_trace_asserter.called_tool("web_search", "LangGraph")
+    # The recorded run made 5 web_fetch calls, but the intent is "researched by
+    # fetching at least a couple of sources". Fetch counts genuinely vary run to
+    # run, so keep this a floor rather than tightening it to the exact count.
+    monocle_trace_asserter.called_tool("web_fetch", "LangGraph", min_count=2)
+    monocle_trace_asserter.does_not_call_tool("image_search", "LangGraph")
+    monocle_trace_asserter.under_token_limit(100_000)
+    monocle_trace_asserter.under_duration(60, span_type="workflow")
+
+
+# --- Live: drive the agent and assert on the trace the real run emits ----------
+# Output text varies run to run, so these assert structure + a lenient token
+# budget only. Duration is omitted: a live run doing LLM calls and network I/O
+# is inherently variable and would flake a wall-clock bound.
+
+
+def test_web_research_live(monocle_trace_asserter: TraceAssertion, run_agent):
+    """Live web-research path: the agent researches and uses ``web_search``."""
+    monocle_trace_asserter.validator.test_workflow(
+        run_agent,
+        {"test_input": ("Research the current state of solid-state EV batteries in 2025 and write a 1-page markdown briefing with sources.",)},
+    )
+
+    monocle_trace_asserter.called_agent("LangGraph").contains_input("solid-state EV batteries")
+    monocle_trace_asserter.contains_any_output("solid-state", "battery", "batteries", "EV")
+    monocle_trace_asserter.called_tool("web_search", "LangGraph")
+    monocle_trace_asserter.under_token_limit(200_000)
+
+
+def test_sandbox_write_file_live(monocle_trace_asserter: TraceAssertion, run_agent):
+    """Live sandbox path: the agent authors a file with ``write_file`` and stays off the web."""
+    monocle_trace_asserter.validator.test_workflow(
+        run_agent,
+        {"test_input": ("Write a Python script that prints the first 10 Fibonacci numbers and save it to a file named fib.py in the sandbox.",)},
+    )
+
+    monocle_trace_asserter.called_agent("LangGraph").contains_input("Fibonacci")
+    monocle_trace_asserter.called_tool("write_file")
+    monocle_trace_asserter.does_not_call_tool("web_search", "LangGraph")
+    monocle_trace_asserter.under_token_limit(100_000)