feat(tests): add Blockbuster runtime gate for event-loop blocking IO (#3229)

* feat(tests): add Blockbuster runtime gate for event-loop blocking IO

Adds a strict runtime gate that fails CI when sync blocking IO calls run
on the asyncio event loop thread through DeerFlow business code.

Components:
- backend/tests/support/detectors/blocking_io_runtime.py — Blockbuster
  context scoped to `app.*` and `deerflow.*` so test infrastructure,
  pytest internals, and third-party libraries stay silent.
- backend/tests/blocking_io/conftest.py — pytest_runtest_protocol
  hookwrapper that wraps every item (setup + call + teardown) with the
  strict context. Respects `@pytest.mark.allow_blocking_io` opt-out.
- backend/tests/blocking_io/test_skills_load.py — regression anchor for
  the #1917 fix (asyncio.to_thread offload around
  LocalSkillStorage.load_skills).
- backend/tests/blocking_io/test_sqlite_lifespan.py — regression anchor
  for the #1912 fix (asyncio.to_thread offload around
  ensure_sqlite_parent_dir).
- backend/tests/blocking_io/test_gate_smoke.py — meta-test asserting the
  gate actually catches unoffloaded blocking IO and that the
  `@pytest.mark.allow_blocking_io` opt-out works.
- backend/Makefile — `make test-blocking-io` target.
- .github/workflows/backend-blocking-io-tests.yml — hard-fail PR gate on
  ubuntu-latest. Windows matrix deferred to follow-up.

Dependencies:
- blockbuster>=1.5.26,<1.6 added to dev group.

Coverage boundary (called out in PR body): the gate only catches blocking
IO on code paths the test suite actually exercises. Static AST inventory
(separate, informational) is the complementary coverage tool. Three blind
spot categories — untested paths, mocked-away paths, env-mismatched paths
— are documented in the PR description.

Findings surfaced while authoring this PR:
- resolve_sqlite_conn_str in runtime/store/_sqlite_utils.py:19 does sync
  Path.resolve() -> os.path.abspath on the lifespan loop thread, ahead of
  the #1912 fix. Not addressed here; tracked as follow-up.

Tests: 4 passed locally (`make test-blocking-io`).
Lint/format: clean (`ruff check` and `ruff format --check`).

* fix(tests): scope Blockbuster gate to blocking-io suite

* fix(tests): harden Blockbuster runtime gate

* test(blocking-io): add project rule extension point

* test(blocking-io): address review cleanup
This commit is contained in:
AochenShen99
2026-05-26 23:03:49 +08:00
committed by GitHub
parent f68bcb771c
commit e344be8d94
13 changed files with 431 additions and 16 deletions
@@ -0,0 +1,46 @@
name: Backend Blocking IO
on:
push:
branches: ["main"]
paths:
- "backend/**"
- ".github/workflows/backend-blocking-io-tests.yml"
pull_request:
types: [opened, synchronize, reopened, ready_for_review]
paths:
- "backend/**"
- ".github/workflows/backend-blocking-io-tests.yml"
concurrency:
group: blocking-io-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
backend-blocking-io:
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install uv
uses: astral-sh/setup-uv@v3
- name: Install backend dependencies
working-directory: backend
run: uv sync --group dev
- name: Run blocking IO regression tests
working-directory: backend
run: make test-blocking-io
+26 -6
View File
@@ -88,18 +88,38 @@ make stop # Stop all services
**Backend directory** (for backend development only):
```bash
make install # Install backend dependencies
make dev # Run Gateway API with reload (port 8001)
make gateway # Run Gateway API only (port 8001)
make test # Run all backend tests
make lint # Lint with ruff
make format # Format code with ruff
make install # Install backend dependencies
make dev # Run Gateway API with reload (port 8001)
make gateway # Run Gateway API only (port 8001)
make test # Run all backend tests
make test-blocking-io # Run strict Blockbuster runtime gate on tests/blocking_io/
make lint # Lint with ruff
make format # Format code with ruff
```
Regression tests related to Docker/provisioner behavior:
- `tests/test_docker_sandbox_mode_detection.py` (mode detection from `config.yaml`)
- `tests/test_provisioner_kubeconfig.py` (kubeconfig file/directory handling)
Blocking-IO runtime gate (`tests/blocking_io/`):
- Wraps every item under `tests/blocking_io/` with a strict Blockbuster
context scoped to `app.*` and `deerflow.*` (see
`tests/support/detectors/blocking_io_runtime.py`). Any sync blocking IO
call whose stack passes through DeerFlow business code while running on
the asyncio event loop raises `BlockingError` and fails the test.
- Two regression anchors live there: `test_skills_load.py` (locks the
`asyncio.to_thread` offload around `LocalSkillStorage.load_skills`, fix
for #1917) and `test_sqlite_lifespan.py` (locks the offload around
SQLite path resolution plus `ensure_sqlite_parent_dir`, fix for #1912).
- `test_gate_smoke.py` is a meta-test asserting the gate actually catches
unoffloaded blocking IO and that the `@pytest.mark.allow_blocking_io`
opt-out works.
- Coverage boundary: the gate only sees code that test execution actually
touches. Static AST coverage is a separate concern (out of scope for
this PR).
- CI: runs on every PR via `.github/workflows/backend-blocking-io-tests.yml`,
hard-fail.
Boundary check (harness → app import firewall):
- `tests/test_harness_boundary.py` — ensures `packages/harness/deerflow/` never imports from `app.*`
+3
View File
@@ -10,6 +10,9 @@ gateway:
test:
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest tests/ -v
test-blocking-io:
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest tests/blocking_io -q --tb=short
lint:
uvx ruff check .
uvx ruff format --check .
@@ -34,6 +34,19 @@ from deerflow.runtime.store._sqlite_utils import ensure_sqlite_parent_dir, resol
logger = logging.getLogger(__name__)
def _prepare_sqlite_checkpointer_path(raw: str) -> str:
conn_str = resolve_sqlite_conn_str(raw)
ensure_sqlite_parent_dir(conn_str)
return conn_str
def _prepare_database_sqlite_checkpointer_path(db_config) -> str:
conn_str = db_config.checkpointer_sqlite_path
ensure_sqlite_parent_dir(conn_str)
return conn_str
# ---------------------------------------------------------------------------
# Async factory
# ---------------------------------------------------------------------------
@@ -54,8 +67,7 @@ async def _async_checkpointer(config) -> AsyncIterator[Checkpointer]:
except ImportError as exc:
raise ImportError(SQLITE_INSTALL) from exc
conn_str = resolve_sqlite_conn_str(config.connection_string or "store.db")
await asyncio.to_thread(ensure_sqlite_parent_dir, conn_str)
conn_str = await asyncio.to_thread(_prepare_sqlite_checkpointer_path, config.connection_string or "store.db")
async with AsyncSqliteSaver.from_conn_string(conn_str) as saver:
await saver.setup()
yield saver
@@ -98,8 +110,7 @@ async def _async_checkpointer_from_database(db_config) -> AsyncIterator[Checkpoi
except ImportError as exc:
raise ImportError(SQLITE_INSTALL) from exc
conn_str = db_config.checkpointer_sqlite_path
ensure_sqlite_parent_dir(conn_str)
conn_str = await asyncio.to_thread(_prepare_database_sqlite_checkpointer_path, db_config)
async with AsyncSqliteSaver.from_conn_string(conn_str) as saver:
await saver.setup()
yield saver
+2
View File
@@ -29,6 +29,7 @@ discord = ["discord.py>=2.7.0"]
[dependency-groups]
dev = [
"blockbuster>=1.5.26,<1.6",
"prompt-toolkit>=3.0.0",
"pytest>=9.0.3",
"pytest-asyncio>=1.3.0",
@@ -38,6 +39,7 @@ dev = [
[tool.pytest.ini_options]
markers = [
"no_auto_user: disable the conftest autouse contextvar fixture for this test",
"allow_blocking_io: opt out of the strict Blockbuster gate in tests/blocking_io/",
]
[tool.uv]
+37
View File
@@ -0,0 +1,37 @@
"""Pytest conftest for the strict Blockbuster runtime gate.
Activates `detect_blocking_io_strict()` around the entire pytest item
protocol (setup + call + teardown) so blocking IO in async fixtures and
lifespan code is also caught, not just blocking IO inside the test body.
Scope: only applies to items whose path is under `backend/tests/blocking_io/`.
Pytest registers conftest hookwrappers globally once the file is loaded,
so an explicit path filter is required to keep the strict gate from
firing on unrelated tests when the full suite is collected.
Opt-out: mark a test with `@pytest.mark.allow_blocking_io` to skip the gate.
"""
from __future__ import annotations
from collections.abc import Generator
from pathlib import Path
import pytest
from support.detectors.blocking_io_runtime import detect_blocking_io_strict
_BLOCKING_IO_TEST_ROOT = Path(__file__).resolve().parent
@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_protocol(item: pytest.Item, nextitem: pytest.Item | None) -> Generator[None, None, None]:
if not _is_blocking_io_item(item) or item.get_closest_marker("allow_blocking_io") is not None:
yield
return
with detect_blocking_io_strict():
yield
def _is_blocking_io_item(item: pytest.Item) -> bool:
return Path(item.path).resolve().is_relative_to(_BLOCKING_IO_TEST_ROOT)
@@ -0,0 +1,55 @@
"""Smoke test: the strict Blockbuster gate is wired up and actively catching.
Independent of any specific production code path, asserts that calling a
known blocking IO function directly from an `async def` (without an
`asyncio.to_thread` wrapper) raises `BlockingError`. If this test ever
stops raising, the gate machinery itself is broken — typical causes are
`scanned_modules` misconfiguration, accidental removal of the Blockbuster
dev dependency, or the conftest hookwrapper no longer firing.
This is the meta-test that protects every other test in this directory
from silent regressions (a green gate that no longer catches anything is
worse than no gate at all).
"""
from __future__ import annotations
import os
from pathlib import Path
import pytest
from blockbuster import BlockingError
from support.detectors.blocking_io_runtime import detect_blocking_io_strict
pytestmark = pytest.mark.asyncio
async def test_gate_catches_unoffloaded_blocking_io_in_deerflow_module(tmp_path: Path) -> None:
from deerflow.runtime.store._sqlite_utils import ensure_sqlite_parent_dir
db_file = tmp_path / "subdir" / "store.db"
with pytest.raises(BlockingError):
ensure_sqlite_parent_dir(str(db_file))
async def test_gate_restores_blockbuster_patches_after_exceptions() -> None:
original_stat = os.stat
with pytest.raises(RuntimeError, match="boom"):
with detect_blocking_io_strict():
raise RuntimeError("boom")
assert os.stat is original_stat
@pytest.mark.allow_blocking_io
async def test_allow_blocking_io_marker_opts_out_of_gate(tmp_path: Path) -> None:
"""Verify the @pytest.mark.allow_blocking_io opt-out actually disables the gate."""
from deerflow.runtime.store._sqlite_utils import ensure_sqlite_parent_dir
db_file = tmp_path / "subdir" / "store.db"
ensure_sqlite_parent_dir(str(db_file))
assert db_file.parent.exists()
@@ -0,0 +1,102 @@
"""Regression test: skill loading must remain releasable to a worker thread.
Anchors the production offload from `subagents/executor.py:_load_skills`,
where both `get_or_new_skill_storage` and the sync `storage.load_skills(...)`
method are dispatched via `asyncio.to_thread`. That fix addressed #1917,
where `os.walk` inside `load_skills` blocked the LangGraph async event loop.
This test invokes the production `_load_skills()` call path under the strict
Blockbuster context against a real `LocalSkillStorage` instance pointed at
a tmp directory. If the production `asyncio.to_thread` offload is removed,
Blockbuster raises `BlockingError` and this test fails.
"""
from __future__ import annotations
import importlib
import sys
from collections.abc import Iterator
from contextlib import contextmanager
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import MagicMock
import pytest
pytestmark = pytest.mark.asyncio
_MISSING = object()
_EXECUTOR_IMPORT_MOCKS = (
"deerflow.agents",
"deerflow.agents.thread_state",
"deerflow.models",
)
def _seed_skill(skills_root: Path) -> None:
skill = skills_root / "public" / "demo"
skill.mkdir(parents=True, exist_ok=True)
(skill / "SKILL.md").write_text(
"---\nname: demo\ndescription: regression-test skill\n---\n# demo\n",
encoding="utf-8",
)
@contextmanager
def _real_subagent_executor() -> Iterator[type]:
"""Import the real executor despite the suite-level circular-import mock."""
original_modules = {name: sys.modules.get(name, _MISSING) for name in _EXECUTOR_IMPORT_MOCKS}
original_executor = sys.modules.get("deerflow.subagents.executor", _MISSING)
parent_module = sys.modules.get("deerflow.subagents")
original_parent_executor = getattr(parent_module, "executor", _MISSING) if parent_module is not None else _MISSING
sys.modules.pop("deerflow.subagents.executor", None)
for name in _EXECUTOR_IMPORT_MOCKS:
sys.modules[name] = MagicMock()
try:
executor_module = importlib.import_module("deerflow.subagents.executor")
yield executor_module.SubagentExecutor
finally:
if original_executor is _MISSING:
sys.modules.pop("deerflow.subagents.executor", None)
else:
sys.modules["deerflow.subagents.executor"] = original_executor
if parent_module is not None:
if original_parent_executor is _MISSING:
try:
delattr(parent_module, "executor")
except AttributeError:
pass
else:
parent_module.executor = original_parent_executor
for name, module in original_modules.items():
if module is _MISSING:
sys.modules.pop(name, None)
else:
sys.modules[name] = module
async def test_load_skills_via_to_thread_does_not_block_event_loop(tmp_path: Path) -> None:
from deerflow.config.skills_config import SkillsConfig
from deerflow.subagents.config import SubagentConfig
_seed_skill(tmp_path)
with _real_subagent_executor() as SubagentExecutor:
executor = SubagentExecutor(
config=SubagentConfig(
name="demo",
description="Loads skills through the production async path.",
),
tools=[],
app_config=SimpleNamespace(skills=SkillsConfig(path=str(tmp_path))),
parent_model="test-model",
)
skills = await executor._load_skills()
assert isinstance(skills, list)
assert any(s.name == "demo" for s in skills)
@@ -0,0 +1,52 @@
"""Regression test: sqlite path setup must run off the event loop.
Anchors the production offload from
`runtime/checkpointer/async_provider.py:_async_checkpointer`, where SQLite
path resolution and `ensure_sqlite_parent_dir` are dispatched via
`await asyncio.to_thread(...)`.
That fix addressed #1912, where the sync `Path.mkdir` / `os.mkdir` inside
`ensure_sqlite_parent_dir` ran on the FastAPI lifespan event loop thread
and blocked startup.
This test invokes the production `_async_checkpointer()` path under the
strict Blockbuster context. The target path's parent does not yet exist, so
the underlying path resolution and `os.mkdir` both execute. If either step is
regressed to run directly on the event loop, Blockbuster raises
`BlockingError` and this test fails.
"""
from __future__ import annotations
import sys
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
pytestmark = pytest.mark.asyncio
async def test_async_checkpointer_sqlite_setup_does_not_block_event_loop(tmp_path: Path) -> None:
from deerflow.config.checkpointer_config import CheckpointerConfig
from deerflow.runtime.checkpointer.async_provider import _async_checkpointer
db_file = tmp_path / "subdir" / "store.db"
mock_saver = AsyncMock()
mock_context_manager = AsyncMock()
mock_context_manager.__aenter__.return_value = mock_saver
mock_context_manager.__aexit__.return_value = False
mock_saver_cls = MagicMock()
mock_saver_cls.from_conn_string.return_value = mock_context_manager
mock_module = MagicMock()
mock_module.AsyncSqliteSaver = mock_saver_cls
with patch.dict(sys.modules, {"langgraph.checkpoint.sqlite.aio": mock_module}):
async with _async_checkpointer(CheckpointerConfig(type="sqlite", connection_string=str(db_file))) as saver:
assert saver is mock_saver
assert db_file.parent.exists()
mock_saver_cls.from_conn_string.assert_called_once_with(str(db_file.resolve()))
mock_saver.setup.assert_awaited_once()
@@ -0,0 +1,44 @@
"""Strict Blockbuster runtime context scoped to DeerFlow business code.
Creates a `BlockBuster` instance with `scanned_modules=("app", "deerflow")`
so that test infrastructure (pytest, langchain, importlib, third-party libs)
is out of scope and does not produce false positives. Only loop-blocking
sync IO whose caller stack passes through `app.*` or `deerflow.*` raises
`BlockingError`.
Used by `backend/tests/blocking_io/conftest.py` to gate the regression suite.
"""
from __future__ import annotations
from collections.abc import Iterator
from contextlib import contextmanager
from blockbuster import BlockBuster, BlockBusterFunction, BlockingError
_SCANNED_MODULES: tuple[str, ...] = ("app", "deerflow")
# Add DeerFlow-local rules here only when Blockbuster's default rule set misses
# a generic blocking primitive used by production code. If a path is invisible
# because no test exercises it, add a production-path runtime anchor instead.
_PROJECT_BLOCKING_RULES: tuple[tuple[str, BlockBusterFunction], ...] = ()
def _install_project_rules(bb: BlockBuster) -> None:
for name, rule in _PROJECT_BLOCKING_RULES:
bb.functions[name] = rule
@contextmanager
def detect_blocking_io_strict() -> Iterator[BlockBuster]:
"""Activate Blockbuster scoped to app.* and deerflow.* callers only."""
bb = BlockBuster(scanned_modules=list(_SCANNED_MODULES))
_install_project_rules(bb)
try:
bb.activate()
yield bb
finally:
bb.deactivate()
__all__ = ["BlockingError", "detect_blocking_io_strict"]
+47 -6
View File
@@ -291,7 +291,7 @@ class TestAsyncCheckpointer:
@pytest.mark.anyio
async def test_sqlite_creates_parent_dir_via_to_thread(self):
"""Async SQLite setup should move mkdir off the event loop."""
from deerflow.runtime.checkpointer.async_provider import make_checkpointer
from deerflow.runtime.checkpointer.async_provider import _prepare_sqlite_checkpointer_path, make_checkpointer
mock_config = MagicMock()
mock_config.checkpointer = CheckpointerConfig(type="sqlite", connection_string="relative/test.db")
@@ -310,22 +310,63 @@ class TestAsyncCheckpointer:
with (
patch("deerflow.runtime.checkpointer.async_provider.get_app_config", return_value=mock_config),
patch.dict(sys.modules, {"langgraph.checkpoint.sqlite.aio": mock_module}),
patch("deerflow.runtime.checkpointer.async_provider.asyncio.to_thread", new_callable=AsyncMock) as mock_to_thread,
patch(
"deerflow.runtime.checkpointer.async_provider.resolve_sqlite_conn_str",
"deerflow.runtime.checkpointer.async_provider.asyncio.to_thread",
new_callable=AsyncMock,
return_value="/tmp/resolved/test.db",
),
) as mock_to_thread,
):
async with make_checkpointer() as saver:
assert saver is mock_saver
mock_to_thread.assert_awaited_once()
called_fn, called_path = mock_to_thread.await_args.args
assert called_fn.__name__ == "ensure_sqlite_parent_dir"
assert called_path == "/tmp/resolved/test.db"
assert called_fn is _prepare_sqlite_checkpointer_path
assert called_path == "relative/test.db"
mock_saver_cls.from_conn_string.assert_called_once_with("/tmp/resolved/test.db")
mock_saver.setup.assert_awaited_once()
@pytest.mark.anyio
async def test_database_sqlite_creates_parent_dir_via_to_thread(self):
"""Unified database SQLite setup should also move path IO off the event loop."""
from deerflow.config.database_config import DatabaseConfig
from deerflow.runtime.checkpointer.async_provider import _prepare_database_sqlite_checkpointer_path, make_checkpointer
db_config = DatabaseConfig(backend="sqlite", sqlite_dir="relative-data")
mock_config = MagicMock()
mock_config.checkpointer = None
mock_config.database = db_config
mock_saver = AsyncMock()
mock_cm = AsyncMock()
mock_cm.__aenter__.return_value = mock_saver
mock_cm.__aexit__.return_value = False
mock_saver_cls = MagicMock()
mock_saver_cls.from_conn_string.return_value = mock_cm
mock_module = MagicMock()
mock_module.AsyncSqliteSaver = mock_saver_cls
with (
patch("deerflow.runtime.checkpointer.async_provider.get_app_config", return_value=mock_config),
patch.dict(sys.modules, {"langgraph.checkpoint.sqlite.aio": mock_module}),
patch(
"deerflow.runtime.checkpointer.async_provider.asyncio.to_thread",
new_callable=AsyncMock,
return_value="/tmp/data/deerflow.db",
) as mock_to_thread,
):
async with make_checkpointer() as saver:
assert saver is mock_saver
mock_to_thread.assert_awaited_once()
called_fn, called_db_config = mock_to_thread.await_args.args
assert called_fn is _prepare_database_sqlite_checkpointer_path
assert called_db_config is db_config
mock_saver_cls.from_conn_string.assert_called_once_with("/tmp/data/deerflow.db")
mock_saver.setup.assert_awaited_once()
# ---------------------------------------------------------------------------
# app_config.py integration
+2
View File
@@ -772,6 +772,7 @@ postgres = [
[package.dev-dependencies]
dev = [
{ name = "blockbuster" },
{ name = "prompt-toolkit" },
{ name = "pytest" },
{ name = "pytest-asyncio" },
@@ -803,6 +804,7 @@ provides-extras = ["postgres", "discord"]
[package.metadata.requires-dev]
dev = [
{ name = "blockbuster", specifier = ">=1.5.26,<1.6" },
{ name = "prompt-toolkit", specifier = ">=3.0.0" },
{ name = "pytest", specifier = ">=9.0.3" },
{ name = "pytest-asyncio", specifier = ">=1.3.0" },