e9deb6c2f2
* perf(harness): push thread metadata filters into SQL Replace Python-side metadata filtering (5x overfetch + in-memory match) with database-side json_extract predicates so LIMIT/OFFSET pagination is exact regardless of match density. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com> * fix(harness): add dialect-aware JsonMatch compiler for type-safe metadata SQL filters Replace SQLAlchemy JSON index/comparator APIs with a custom JsonMatch ColumnElement that compiles to json_type/json_extract on SQLite and jsonb_typeof/->>/-> on PostgreSQL. Tighten key validation regex to single-segment identifiers, handle None/bool/numeric value types with json_type-based discrimination, and strengthen test coverage for edge cases and discriminability. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com> * fix(harness): address Copilot review comments on JSON metadata filters - Use json_typeof instead of jsonb_typeof in PostgreSQL compiler; the metadata_json column is JSON not JSONB so jsonb_typeof would error at runtime on any PostgreSQL backend - Align _is_safe_json_key with json_match's _KEY_CHARSET_RE so keys containing hyphens or leading digits are not silently skipped - Add thread_id as secondary ORDER BY in search() to make pagination deterministic when updated_at values collide; remove asyncio.sleep from the pagination regression test Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> * fix(harness): address remaining review comments on metadata SQL filters - Remove _is_safe_json_key() and reuse json_match ValueError to avoid validator drift (Copilot #3217603895, #3217411616) - Raise ValueError when all metadata keys are rejected so callers never get silent unfiltered results (WillemJiang) - Fix integer precision: split int/float branches, bind int as Integer() with INTEGER/BIGINT CAST instead of float() coercion (Copilot #3217603972) - Fix jsonb_typeof -> json_typeof on JSON column (Copilot #3217411579) - Replace manual _cleanup() calls with async yield fixture so teardown always runs (Copilot #3217604019) - Remove asyncio.sleep(0.01) pagination ordering; use thread_id secondary sort instead (Copilot #3217411636) - Add type annotations to _bind/_build_clause/_compile_* and remove EOL comments from _Dialect fields (coding.mdc) - Expand test coverage: boolean/null/mixed-type/large-int precision, partial unsafe-key skip with caplog assertion Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(harness): address third-round Copilot review comments on JsonMatch - Reject unsupported value types (list, dict, ...) in JsonMatch.__init__ with TypeError so inherit_cache=True never receives an unhashable value and callers get an explicit error instead of silent str() coercion (Copilot #3217933201) - Upgrade int bindparam from Integer() to BigInteger() to align with BIGINT CAST and avoid overflow on large integers (Copilot #3217933252) - Catch TypeError alongside ValueError in search() so non-string metadata keys are warned and skipped rather than raising unexpectedly (Copilot #3217933300) - Add three tests: json_match rejects unsupported value types, search() warns and raises on non-string key, search() warns and raises on unsupported value type Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(harness): address fourth-round Copilot review comments on JsonMatch - Add CASE WHEN guard for PostgreSQL integer matching: json_typeof returns 'number' for both ints and floats; wrap CAST in CASE with regex guard '^-?[0-9]+$' so float rows never trigger CAST error (Copilot #3218413860) - Validate isinstance(key, str) before regex match in JsonMatch.__init__ so non-string keys raise ValueError consistently instead of TypeError from re.match (Copilot #3218413900) - Include exception message in metadata filter skip warning so callers can distinguish invalid key from unsupported value type (Copilot #3218413924) - Update tests: assert CASE WHEN guard in PG int compilation, cover non-string key ValueError in test_json_match_rejects_unsafe_key Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(harness): align ThreadMetaStore.search() signature with sql.py implementation Use `dict[str, Any]` for `metadata` and `list[dict[str, Any]]` as return type in base class and MemoryThreadMetaStore to resolve an LSP signature mismatch; also correct a test docstring that cited the wrong exception type. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(harness): surface InvalidMetadataFilterError as HTTP 400 in search endpoint Replace bare ValueError with a domain-specific InvalidMetadataFilterError (subclass of ValueError) so the Gateway handler can catch it and return HTTP 400 instead of letting it bubble up as a 500. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com> * fix(harness): sanitize metadata keys in log output to prevent log injection Use ascii() instead of %r to escape control characters in client-supplied metadata keys before logging, preventing multiline/forged log entries. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(harness): validate metadata filters at API boundary and dedupe key/value rules - Add Pydantic ``field_validator`` on ``ThreadSearchRequest.metadata`` so unsafe keys / unsupported value types are rejected with HTTP 422 from both SQL and memory backends (closes Copilot review 3218830849). - Export ``validate_metadata_filter_key`` / ``validate_metadata_filter_value`` (and ``ALLOWED_FILTER_VALUE_TYPES``) from ``json_compat`` and have ``JsonMatch.__init__`` reuse them — the Gateway-side validator and the SQL-side ``JsonMatch`` constructor now share one admission rule and cannot drift. - Format ``InvalidMetadataFilterError`` rejected-keys list as a comma-separated plain string instead of a Python list repr so the surfaced HTTP 400 detail is readable (closes Copilot review 3218830899). - Update router tests to cover both 422 boundary paths plus the 400 defense-in-depth path when a backend still raises the error. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(harness): harden JsonMatch compile-time key validation against __init__ bypass Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> * fix: address review feedback on metadata filter SQL push-down - Add signed 64-bit range check to validate_metadata_filter_value; give out-of-range ints a distinct TypeError message. - Replace assert guards in _compile_sqlite/_compile_pg with explicit if/raise so they survive python -O optimisation. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4 <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
226 lines
9.1 KiB
Python
226 lines
9.1 KiB
Python
"""SQLAlchemy-backed thread metadata repository."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
from datetime import UTC, datetime
|
|
from typing import Any
|
|
|
|
from sqlalchemy import select, update
|
|
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
|
|
|
from deerflow.persistence.json_compat import json_match
|
|
from deerflow.persistence.thread_meta.base import InvalidMetadataFilterError, ThreadMetaStore
|
|
from deerflow.persistence.thread_meta.model import ThreadMetaRow
|
|
from deerflow.runtime.user_context import AUTO, _AutoSentinel, resolve_user_id
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
class ThreadMetaRepository(ThreadMetaStore):
|
|
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
|
self._sf = session_factory
|
|
|
|
@staticmethod
|
|
def _row_to_dict(row: ThreadMetaRow) -> dict[str, Any]:
|
|
d = row.to_dict()
|
|
d["metadata"] = d.pop("metadata_json", None) or {}
|
|
for key in ("created_at", "updated_at"):
|
|
val = d.get(key)
|
|
if isinstance(val, datetime):
|
|
d[key] = val.isoformat()
|
|
return d
|
|
|
|
async def create(
|
|
self,
|
|
thread_id: str,
|
|
*,
|
|
assistant_id: str | None = None,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
display_name: str | None = None,
|
|
metadata: dict | None = None,
|
|
) -> dict:
|
|
# Auto-resolve user_id from contextvar when AUTO; explicit None
|
|
# creates an orphan row (used by migration scripts).
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.create")
|
|
now = datetime.now(UTC)
|
|
row = ThreadMetaRow(
|
|
thread_id=thread_id,
|
|
assistant_id=assistant_id,
|
|
user_id=resolved_user_id,
|
|
display_name=display_name,
|
|
metadata_json=metadata or {},
|
|
created_at=now,
|
|
updated_at=now,
|
|
)
|
|
async with self._sf() as session:
|
|
session.add(row)
|
|
await session.commit()
|
|
await session.refresh(row)
|
|
return self._row_to_dict(row)
|
|
|
|
async def get(
|
|
self,
|
|
thread_id: str,
|
|
*,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
) -> dict | None:
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.get")
|
|
async with self._sf() as session:
|
|
row = await session.get(ThreadMetaRow, thread_id)
|
|
if row is None:
|
|
return None
|
|
# Enforce owner filter unless explicitly bypassed (user_id=None).
|
|
if resolved_user_id is not None and row.user_id != resolved_user_id:
|
|
return None
|
|
return self._row_to_dict(row)
|
|
|
|
async def check_access(self, thread_id: str, user_id: str, *, require_existing: bool = False) -> bool:
|
|
"""Check if ``user_id`` has access to ``thread_id``.
|
|
|
|
Two modes — one row, two distinct semantics depending on what
|
|
the caller is about to do:
|
|
|
|
- ``require_existing=False`` (default, permissive):
|
|
Returns True for: row missing (untracked legacy thread),
|
|
``row.user_id`` is None (shared / pre-auth data),
|
|
or ``row.user_id == user_id``. Use for **read-style**
|
|
decorators where treating an untracked thread as accessible
|
|
preserves backward-compat.
|
|
|
|
- ``require_existing=True`` (strict):
|
|
Returns True **only** when the row exists AND
|
|
(``row.user_id == user_id`` OR ``row.user_id is None``).
|
|
Use for **destructive / mutating** decorators (DELETE, PATCH,
|
|
state-update) so a thread that has *already been deleted*
|
|
cannot be re-targeted by any caller — closing the
|
|
delete-idempotence cross-user gap where the row vanishing
|
|
made every other user appear to "own" it.
|
|
"""
|
|
async with self._sf() as session:
|
|
row = await session.get(ThreadMetaRow, thread_id)
|
|
if row is None:
|
|
return not require_existing
|
|
if row.user_id is None:
|
|
return True
|
|
return row.user_id == user_id
|
|
|
|
async def search(
|
|
self,
|
|
*,
|
|
metadata: dict[str, Any] | None = None,
|
|
status: str | None = None,
|
|
limit: int = 100,
|
|
offset: int = 0,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
) -> list[dict[str, Any]]:
|
|
"""Search threads with optional metadata and status filters.
|
|
|
|
Owner filter is enforced by default: caller must be in a user
|
|
context. Pass ``user_id=None`` to bypass (migration/CLI).
|
|
"""
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.search")
|
|
stmt = select(ThreadMetaRow).order_by(ThreadMetaRow.updated_at.desc(), ThreadMetaRow.thread_id.desc())
|
|
if resolved_user_id is not None:
|
|
stmt = stmt.where(ThreadMetaRow.user_id == resolved_user_id)
|
|
if status:
|
|
stmt = stmt.where(ThreadMetaRow.status == status)
|
|
|
|
if metadata:
|
|
applied = 0
|
|
for key, value in metadata.items():
|
|
try:
|
|
stmt = stmt.where(json_match(ThreadMetaRow.metadata_json, key, value))
|
|
applied += 1
|
|
except (ValueError, TypeError) as exc:
|
|
logger.warning("Skipping metadata filter key %s: %s", ascii(key), exc)
|
|
if applied == 0:
|
|
# Comma-separated plain string (no list repr / nested
|
|
# quoting) so the 400 detail surfaced by the Gateway is
|
|
# easy for clients to read. Sorted for determinism.
|
|
rejected_keys = ", ".join(sorted(str(k) for k in metadata))
|
|
raise InvalidMetadataFilterError(f"All metadata filter keys were rejected as unsafe: {rejected_keys}")
|
|
|
|
stmt = stmt.limit(limit).offset(offset)
|
|
async with self._sf() as session:
|
|
result = await session.execute(stmt)
|
|
return [self._row_to_dict(r) for r in result.scalars()]
|
|
|
|
async def _check_ownership(self, session: AsyncSession, thread_id: str, resolved_user_id: str | None) -> bool:
|
|
"""Return True if the row exists and is owned (or filter bypassed)."""
|
|
if resolved_user_id is None:
|
|
return True # explicit bypass
|
|
row = await session.get(ThreadMetaRow, thread_id)
|
|
return row is not None and row.user_id == resolved_user_id
|
|
|
|
async def update_display_name(
|
|
self,
|
|
thread_id: str,
|
|
display_name: str,
|
|
*,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
) -> None:
|
|
"""Update the display_name (title) for a thread."""
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.update_display_name")
|
|
async with self._sf() as session:
|
|
if not await self._check_ownership(session, thread_id, resolved_user_id):
|
|
return
|
|
await session.execute(update(ThreadMetaRow).where(ThreadMetaRow.thread_id == thread_id).values(display_name=display_name, updated_at=datetime.now(UTC)))
|
|
await session.commit()
|
|
|
|
async def update_status(
|
|
self,
|
|
thread_id: str,
|
|
status: str,
|
|
*,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
) -> None:
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.update_status")
|
|
async with self._sf() as session:
|
|
if not await self._check_ownership(session, thread_id, resolved_user_id):
|
|
return
|
|
await session.execute(update(ThreadMetaRow).where(ThreadMetaRow.thread_id == thread_id).values(status=status, updated_at=datetime.now(UTC)))
|
|
await session.commit()
|
|
|
|
async def update_metadata(
|
|
self,
|
|
thread_id: str,
|
|
metadata: dict,
|
|
*,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
) -> None:
|
|
"""Merge ``metadata`` into ``metadata_json``.
|
|
|
|
Read-modify-write inside a single session/transaction so concurrent
|
|
callers see consistent state. No-op if the row does not exist or
|
|
the user_id check fails.
|
|
"""
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.update_metadata")
|
|
async with self._sf() as session:
|
|
row = await session.get(ThreadMetaRow, thread_id)
|
|
if row is None:
|
|
return
|
|
if resolved_user_id is not None and row.user_id != resolved_user_id:
|
|
return
|
|
merged = dict(row.metadata_json or {})
|
|
merged.update(metadata)
|
|
row.metadata_json = merged
|
|
row.updated_at = datetime.now(UTC)
|
|
await session.commit()
|
|
|
|
async def delete(
|
|
self,
|
|
thread_id: str,
|
|
*,
|
|
user_id: str | None | _AutoSentinel = AUTO,
|
|
) -> None:
|
|
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.delete")
|
|
async with self._sf() as session:
|
|
row = await session.get(ThreadMetaRow, thread_id)
|
|
if row is None:
|
|
return
|
|
if resolved_user_id is not None and row.user_id != resolved_user_id:
|
|
return
|
|
await session.delete(row)
|
|
await session.commit()
|