mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-06-13 19:06:01 +00:00
aa015462a7
* Add user-owned IM channel connections * Fix dev startup and channel connect popup * Use async channel connect flow * Harden dev service daemon startup * Support local IM channel connections * Align IM connections with local channels * Fix safe user id digest algorithm * Address Copilot IM channel feedback * Address IM channel review comments * Support all integrated IM channel connections * Format additional channel connection tests * Keep unavailable channel connect buttons clickable * Fix IM channel provider icons * Add runtime setup for enabled IM channels * Guard global shortcut key handling * Keep configured IM channels editable * Avoid password autofill for channel secrets * Make channel threads visible to connection owners * Persist IM runtime config locally * Allow disconnecting runtime IM channels * Route no-auth channel sessions to local user * Use default user for auth-disabled local mode * Show IM channel source on threads * Prefill IM channel runtime config * Reflect IM channel runtime health * Ignore Feishu message read events * Ignore Feishu non-content message events * Let setup wizard enable IM channels * Fix frontend formatting after merge * Stabilize backend tests without local config * Isolate channel runtime config tests * Address channel connection review comments * Use sha256 user buckets with legacy migration * Ensure runtime IM channels are ready after restart * Persist disconnected IM channel state * Address channel connection review comments * Address channel connection review findings Frontend connect flow: - Open the runtime-config dialog only when a provider still needs credentials; configured providers go straight to the connect flow, so the binding-code/deep-link path is reachable from the UI again. - After saving credentials, continue into the connect flow when a user binding is still required (multi-user mode) instead of stopping at a "Connected" toast. - Extract shared provider-state helpers to core/channels/provider-state and add unit + e2e coverage for the direct-connect and configure-then-connect paths. Provider status semantics: - Report connection_status from the user's newest connection row; with no binding it is not_connected, except in auth-disabled local mode where a configured running channel is effectively connected. Concurrency and event-loop correctness: - Offload ChannelRuntimeConfigStore construction and writes, channel service construction, and Slack connection replies to threads; add a tests/blocking_io/ anchor for the runtime-config handlers. - Consume binding codes with a conditional UPDATE so a code can only be used once under concurrent workers; retry upsert_connection as an update when a concurrent insert wins the unique constraint. - Serialize ensure_channel_ready per channel so concurrent provider polls cannot double-start a channel worker. Config and migration hardening: - Stop mutating the get_app_config()-cached Telegram provider config; the runtime store now owns the UI-entered bot username. - Register channel_connections in STARTUP_ONLY_FIELDS with the standardized startup-only Field description. - Match the legacy unsafe-id bucket by recomputing its exact SHA-1 name so another user's same-prefix bucket can never be migrated. - Remove the unused Telegram process_webhook_update path and document src/core/channels in the frontend docs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address PR review comments on authz scoping and channel runtime Security (review feedback from ShenAC-SAC): - Scope internal-token callers to the connection owner carried in X-DeerFlow-Owner-User-Id instead of bypassing owner checks outright, in both require_permission(owner_check=True) and the stateless run endpoints. Internal callers keep access to their own and shared/legacy threads, and may claim a default-owned channel thread for its real owner, but a leaked internal token no longer grants cross-user thread access. - Require admin privileges for POST/DELETE /api/channels/{provider}/ runtime-config: runtime credentials and channel workers are instance-wide shared state (same model as the MCP config API). Read-only provider listing stays available to all users. Performance (review feedback from willem-bd): - Skip the redundant thread channel-metadata PATCH after the first successful backfill per thread. - Reuse the per-connection Slack WebClient until its token changes instead of constructing one per outbound message. - Reconcile channel readiness for all providers concurrently in GET /api/channels/providers. Also resolve the code-quality unused-import flag in the blocking-io anchor by pre-importing the channel service via importlib. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix prettier formatting in provider-state test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Reconcile UI runtime channel config with config reload on restart Main now reloads a channel's config.yaml entry on restart_channel() (#3514, issue #3497). Adapt the user-owned connection flow to coexist: - configure_channel() restarts with reload_config=False — the caller just supplied the authoritative config (browser-entered credentials that are never written to config.yaml), so a file reload must not clobber it with the stale on-disk entry. - _load_channel_config() re-applies the UI runtime-store overlay used at startup, so an operator-triggered restart keeps browser-entered credentials for channels without a config.yaml entry and does not resurrect a channel disconnected from the UI. - Offload the reload's disk IO (config.yaml + runtime store) with asyncio.to_thread, matching the blocking-IO policy on this branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
244 lines
10 KiB
Python
244 lines
10 KiB
Python
"""SQLAlchemy-backed thread metadata repository."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
from datetime import UTC, datetime
|
|
from typing import Any
|
|
|
|
from sqlalchemy import select, update
|
|
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
|
|
|
from deerflow.persistence.json_compat import json_match
|
|
from deerflow.persistence.thread_meta.base import InvalidMetadataFilterError, ThreadMetaStore
|
|
from deerflow.persistence.thread_meta.model import ThreadMetaRow
|
|
from deerflow.runtime.user_context import AUTO, _AutoSentinel, resolve_user_id
|
|
from deerflow.utils.time import coerce_iso
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
class ThreadMetaRepository(ThreadMetaStore):
|
|
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
|
self._sf = session_factory
|
|
|
|
@staticmethod
|
|
def _row_to_dict(row: ThreadMetaRow) -> dict[str, Any]:
|
|
d = row.to_dict()
|
|
d["metadata"] = d.pop("metadata_json", None) or {}
|
|
for key in ("created_at", "updated_at"):
|
|
val = d.get(key)
|
|
if isinstance(val, datetime):
|
|
# SQLite drops tzinfo despite ``DateTime(timezone=True)``;
|
|
# ``coerce_iso`` normalizes naive values as UTC so the wire format always carries tz.
|
|
d[key] = coerce_iso(val)
|
|
return d
|
|
|
|
async def create(
|
|
self,
|
|
thread_id: str,
|
|
*,
|
|
assistant_id: str | None = None,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
display_name: str | None = None,
|
|
metadata: dict | None = None,
|
|
) -> dict:
|
|
# Auto-resolve user_id from contextvar when AUTO; explicit None
|
|
# creates an orphan row (used by migration scripts).
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.create")
|
|
now = datetime.now(UTC)
|
|
row = ThreadMetaRow(
|
|
thread_id=thread_id,
|
|
assistant_id=assistant_id,
|
|
user_id=resolved_user_id,
|
|
display_name=display_name,
|
|
metadata_json=metadata or {},
|
|
created_at=now,
|
|
updated_at=now,
|
|
)
|
|
async with self._sf() as session:
|
|
session.add(row)
|
|
await session.commit()
|
|
await session.refresh(row)
|
|
return self._row_to_dict(row)
|
|
|
|
async def get(
|
|
self,
|
|
thread_id: str,
|
|
*,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
) -> dict | None:
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.get")
|
|
async with self._sf() as session:
|
|
row = await session.get(ThreadMetaRow, thread_id)
|
|
if row is None:
|
|
return None
|
|
# Enforce owner filter unless explicitly bypassed (user_id=None).
|
|
if resolved_user_id is not None and row.user_id != resolved_user_id:
|
|
return None
|
|
return self._row_to_dict(row)
|
|
|
|
async def check_access(self, thread_id: str, user_id: str, *, require_existing: bool = False) -> bool:
|
|
"""Check if ``user_id`` has access to ``thread_id``.
|
|
|
|
Two modes — one row, two distinct semantics depending on what
|
|
the caller is about to do:
|
|
|
|
- ``require_existing=False`` (default, permissive):
|
|
Returns True for: row missing (untracked legacy thread),
|
|
``row.user_id`` is None (shared / pre-auth data),
|
|
or ``row.user_id == user_id``. Use for **read-style**
|
|
decorators where treating an untracked thread as accessible
|
|
preserves backward-compat.
|
|
|
|
- ``require_existing=True`` (strict):
|
|
Returns True **only** when the row exists AND
|
|
(``row.user_id == user_id`` OR ``row.user_id is None``).
|
|
Use for **destructive / mutating** decorators (DELETE, PATCH,
|
|
state-update) so a thread that has *already been deleted*
|
|
cannot be re-targeted by any caller — closing the
|
|
delete-idempotence cross-user gap where the row vanishing
|
|
made every other user appear to "own" it.
|
|
"""
|
|
async with self._sf() as session:
|
|
row = await session.get(ThreadMetaRow, thread_id)
|
|
if row is None:
|
|
return not require_existing
|
|
if row.user_id is None:
|
|
return True
|
|
return row.user_id == user_id
|
|
|
|
async def search(
|
|
self,
|
|
*,
|
|
metadata: dict[str, Any] | None = None,
|
|
status: str | None = None,
|
|
limit: int = 100,
|
|
offset: int = 0,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
) -> list[dict[str, Any]]:
|
|
"""Search threads with optional metadata and status filters.
|
|
|
|
Owner filter is enforced by default: caller must be in a user
|
|
context. Pass ``user_id=None`` to bypass (migration/CLI).
|
|
"""
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.search")
|
|
stmt = select(ThreadMetaRow).order_by(ThreadMetaRow.updated_at.desc(), ThreadMetaRow.thread_id.desc())
|
|
if resolved_user_id is not None:
|
|
stmt = stmt.where(ThreadMetaRow.user_id == resolved_user_id)
|
|
if status:
|
|
stmt = stmt.where(ThreadMetaRow.status == status)
|
|
|
|
if metadata:
|
|
applied = 0
|
|
for key, value in metadata.items():
|
|
try:
|
|
stmt = stmt.where(json_match(ThreadMetaRow.metadata_json, key, value))
|
|
applied += 1
|
|
except (ValueError, TypeError) as exc:
|
|
logger.warning("Skipping metadata filter key %s: %s", ascii(key), exc)
|
|
if applied == 0:
|
|
# Comma-separated plain string (no list repr / nested
|
|
# quoting) so the 400 detail surfaced by the Gateway is
|
|
# easy for clients to read. Sorted for determinism.
|
|
rejected_keys = ", ".join(sorted(str(k) for k in metadata))
|
|
raise InvalidMetadataFilterError(f"All metadata filter keys were rejected as unsafe: {rejected_keys}")
|
|
|
|
stmt = stmt.limit(limit).offset(offset)
|
|
async with self._sf() as session:
|
|
result = await session.execute(stmt)
|
|
return [self._row_to_dict(r) for r in result.scalars()]
|
|
|
|
async def _check_ownership(self, session: AsyncSession, thread_id: str, resolved_user_id: str | None) -> bool:
|
|
"""Return True if the row exists and is owned (or filter bypassed)."""
|
|
if resolved_user_id is None:
|
|
return True # explicit bypass
|
|
row = await session.get(ThreadMetaRow, thread_id)
|
|
return row is not None and row.user_id == resolved_user_id
|
|
|
|
async def update_display_name(
|
|
self,
|
|
thread_id: str,
|
|
display_name: str,
|
|
*,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
) -> None:
|
|
"""Update the display_name (title) for a thread."""
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.update_display_name")
|
|
async with self._sf() as session:
|
|
if not await self._check_ownership(session, thread_id, resolved_user_id):
|
|
return
|
|
await session.execute(update(ThreadMetaRow).where(ThreadMetaRow.thread_id == thread_id).values(display_name=display_name, updated_at=datetime.now(UTC)))
|
|
await session.commit()
|
|
|
|
async def update_status(
|
|
self,
|
|
thread_id: str,
|
|
status: str,
|
|
*,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
) -> None:
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.update_status")
|
|
async with self._sf() as session:
|
|
if not await self._check_ownership(session, thread_id, resolved_user_id):
|
|
return
|
|
await session.execute(update(ThreadMetaRow).where(ThreadMetaRow.thread_id == thread_id).values(status=status, updated_at=datetime.now(UTC)))
|
|
await session.commit()
|
|
|
|
async def update_metadata(
|
|
self,
|
|
thread_id: str,
|
|
metadata: dict,
|
|
*,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
) -> None:
|
|
"""Merge ``metadata`` into ``metadata_json``.
|
|
|
|
Read-modify-write inside a single session/transaction so concurrent
|
|
callers see consistent state. No-op if the row does not exist or
|
|
the user_id check fails.
|
|
"""
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.update_metadata")
|
|
async with self._sf() as session:
|
|
row = await session.get(ThreadMetaRow, thread_id)
|
|
if row is None:
|
|
return
|
|
if resolved_user_id is not None and row.user_id != resolved_user_id:
|
|
return
|
|
merged = dict(row.metadata_json or {})
|
|
merged.update(metadata)
|
|
row.metadata_json = merged
|
|
row.updated_at = datetime.now(UTC)
|
|
await session.commit()
|
|
|
|
async def update_owner(
|
|
self,
|
|
thread_id: str,
|
|
owner_user_id: str,
|
|
*,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
) -> None:
|
|
"""Move a thread metadata row to ``owner_user_id``."""
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.update_owner")
|
|
async with self._sf() as session:
|
|
if not await self._check_ownership(session, thread_id, resolved_user_id):
|
|
return
|
|
await session.execute(update(ThreadMetaRow).where(ThreadMetaRow.thread_id == thread_id).values(user_id=owner_user_id, updated_at=datetime.now(UTC)))
|
|
await session.commit()
|
|
|
|
async def delete(
|
|
self,
|
|
thread_id: str,
|
|
*,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
) -> None:
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.delete")
|
|
async with self._sf() as session:
|
|
row = await session.get(ThreadMetaRow, thread_id)
|
|
if row is None:
|
|
return
|
|
if resolved_user_id is not None and row.user_id != resolved_user_id:
|
|
return
|
|
await session.delete(row)
|
|
await session.commit()
|