open/agent_delta
1
0
Fork 0
mirror of https://github.com/furyhawk/agent_delta.git synced 2026-07-20 09:45:36 +00:00
Code Issues Packages Projects Releases Wiki Activity

init

Browse Source
This commit is contained in:
furyhawk
2026-06-15 14:22:07 +08:00
commit 521ab9b7c9
543 changed files with 64516 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/commands/add-endpoint.md
+40
View File
@@ -0,0 +1,40 @@
---
description: Scaffold a new API endpoint with full layering
---
Create a new API endpoint: $ARGUMENTS
Follow the project's layered architecture. Create files in this order:
1. **Schema** (`backend/app/schemas/<entity>.py`):
- Inherit `BaseSchema` (and `TimestampSchema` for Read)
- Create `*Create`, `*Update`, `*Read`, `*List` models
- Use `Field()` with constraints, `EmailStr` where applicable
2. **DB Model** (`backend/app/db/models/<entity>.py`):
- Inherit `Base, TimestampMixin`
- Use `Mapped[type]` + `mapped_column()`
- Add `__repr__`, relationships with `cascade="all, delete-orphan"`
3. **Repository** (`backend/app/repositories/<entity>_repo.py`):
- Stateless async functions: `get_by_id`, `get_multi`, `create`, `update`, `delete`
- Use `db.flush()` + `db.refresh()`, keyword-only args after `db`
4. **Service** (`backend/app/services/<entity>.py`):
- Class with `__init__(self, db: AsyncSession)`
- Raise `NotFoundError`, `AlreadyExistsError` as appropriate
5. **DI** (`backend/app/api/deps.py`):
- Add factory function and `Annotated` alias: `EntitySvc = Annotated[EntityService, Depends(get_entity_service)]`
6. **Route** (`backend/app/api/routes/v1/<entity>.py`):
- CRUD: GET list, GET by id, POST (201), PATCH, DELETE (204)
- Use DI aliases, `response_model`, `-> Any` return type
7. **Register** router in `backend/app/api/routes/v1/__init__.py`
8. **Migration**: `cd backend && uv run alembic revision --autogenerate -m "Add <entity> table"`
9. **Test** (`backend/tests/`): mirror source structure
10. Lint: `cd backend && uv run ruff check . --fix && uv run ruff format .`
.claude/commands/fix-issue.md
+16
View File
@@ -0,0 +1,16 @@
---
description: Investigate and fix an issue
---
Fix the issue: $ARGUMENTS
1. **Understand** — search the codebase for relevant code, read the files, understand current behavior
2. **Reproduce** — if possible, identify a test case or request that triggers the issue
3. **Root cause** — trace through Routes → Services → Repositories to find where the bug originates
4. **Fix** — implement the fix following project conventions:
- Domain exceptions in services (not HTTP errors)
- `db.flush()` in repositories (not `commit`)
- Type hints on all changed signatures
5. **Test** — run `cd backend && uv run pytest` to verify no regressions
6. **Lint** — run `cd backend && uv run ruff check . --fix && uv run ruff format .`
7. **Summary** — explain what was changed and why
.claude/commands/review.md
+31
View File
@@ -0,0 +1,31 @@
---
description: Review code changes against project conventions
---
Review all staged and unstaged changes in the current branch.
For each changed file, verify:
**Architecture:**
- Routes only call services, never repositories
- Services raise domain exceptions (NotFoundError, AlreadyExistsError, etc.), not HTTP exceptions
- Repositories use `db.flush()` + `db.refresh()`, never `db.commit()`
- DI uses Annotated aliases from `deps.py` (CurrentUser, *Svc), not raw `Depends()` in signatures
**Schemas & Types:**
- Separate Create/Update/Read/List Pydantic models
- Type hints on all function signatures (params + return)
- Modern syntax: `str | None` not `Optional[str]`
- Route return type is `-> Any`
**Code Quality:**
- No debug code (print, commented-out code, TODO without issue reference)
- No security issues (SQL injection, exposed secrets, missing auth)
- Consistent naming (snake_case functions, PascalCase classes)
- Imports ordered: stdlib → third-party → local
**Validation:**
1. Run `cd backend && uv run ruff check .`
2. Run `cd backend && uv run pytest` (if test files changed)
Provide findings with specific file:line references and suggest fixes.
.claude/rules/api-conventions.md
+90
View File
@@ -0,0 +1,90 @@
---
description: API design, REST conventions, auth, pagination, response format
globs: ["backend/app/api/**/*.py"]
---
# API Conventions
## Route Structure
- All routes under `/api/v1/` prefix
- One file per domain entity in `api/routes/v1/`
- Use `APIRouter()` with tags
## HTTP Methods & Status Codes
```python
# GET — read
@router.get("/{id}", response_model=EntityRead)
# GET list — paginated
@router.get("", response_model=EntityList)
# POST — create (201)
@router.post("", response_model=EntityRead, status_code=status.HTTP_201_CREATED)
# PATCH — partial update
@router.patch("/{id}", response_model=EntityRead)
# DELETE — no content (204)
@router.delete("/{id}", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
```
## Pagination
```python
@router.get("", response_model=ConversationList)
async def list_items(
service: ConversationSvc,
user: CurrentUser,
skip: int = Query(0, ge=0, description="Items to skip"),
limit: int = Query(50, ge=1, le=100, description="Max items to return"),
) -> Any:
items, total = await service.list(user_id=user.id, skip=skip, limit=limit)
return ConversationList(items=items, total=total)
```
## Authentication
- `CurrentUser` — JWT Bearer token (any authenticated user)
- `CurrentAdmin` — JWT + admin role check via `RoleChecker`
- `ValidAPIKey` — API key from header (service-to-service)
```python
# Protected endpoint
async def get_profile(user: CurrentUser) -> Any: ...
# Admin-only endpoint
async def delete_user(user: CurrentAdmin) -> Any: ...
# API key endpoint
async def webhook_callback(api_key: ValidAPIKey) -> Any: ...
```
## Response Format
All route handlers return `-> Any`. The `response_model` parameter handles serialization.
Error responses follow this JSON structure:
```json
{
"error": {
"code": "NOT_FOUND",
"message": "User not found",
"details": {"user_id": "..."}
}
}
```
## File Upload
```python
@router.post("/me/avatar", response_model=UserRead)
async def upload_avatar(
file: UploadFile = File(...),
user: CurrentUser,
service: UserSvc,
) -> Any:
data = await file.read()
return await service.update_avatar(user.id, data, file.filename or "avatar.jpg")
```
.claude/rules/architecture.md
+138
View File
@@ -0,0 +1,138 @@
---
description: Layered architecture patterns — Routes, Services, Repositories, DI
globs: ["backend/app/**/*.py"]
---
# Architecture
## Layered Architecture: Routes → Services → Repositories
Routes NEVER import or call repositories directly. Always go through a service.
## Repositories (`app/repositories/`)
Pure data access — stateless functions (not classes):
```python
async def get_by_id(db: AsyncSession, entity_id: UUID) -> Entity | None:
result = await db.execute(select(Entity).where(Entity.id == entity_id))
return result.scalar_one_or_none()
async def create(db: AsyncSession, *, field1: str, field2: str) -> Entity:
entity = Entity(field1=field1, field2=field2)
db.add(entity)
await db.flush()
await db.refresh(entity)
return entity
async def update(db: AsyncSession, *, db_entity: Entity, update_data: dict[str, Any]) -> Entity:
for field, value in update_data.items():
setattr(db_entity, field, value)
await db.flush()
await db.refresh(db_entity)
return db_entity
async def delete(db: AsyncSession, entity_id: UUID) -> Entity | None:
entity = await get_by_id(db, entity_id)
if entity:
await db.delete(entity)
await db.flush()
return entity
```
Rules:
- ALWAYS `db.flush()` + `db.refresh()`, NEVER `db.commit()` — session auto-commits in `get_db_session`
- Use keyword-only args after `db`: `create(db, *, email: str, name: str)`
- Return the entity (or None for get/delete), never return IDs or dicts
- Functions are async for PostgreSQL/MongoDB, sync for SQLite
## Services (`app/services/`)
Business logic — class-based with DB session:
```python
class UserService:
def __init__(self, db: AsyncSession):
self.db = db
async def get_by_id(self, user_id: UUID) -> User:
user = await user_repo.get_by_id(self.db, user_id)
if not user:
raise NotFoundError(message="User not found", details={"user_id": user_id})
return user
async def create(self, data: UserCreate) -> User:
existing = await user_repo.get_by_email(self.db, data.email)
if existing:
raise AlreadyExistsError(message="Email already registered", details={"email": data.email})
hashed_password = get_password_hash(data.password)
return await user_repo.create(self.db, email=data.email, hashed_password=hashed_password)
```
Rules:
- Raise domain exceptions, NEVER return error codes or None for "not found"
- Services call repo functions, never build raw queries
- One service per domain entity
## Thin vs. thick domains
Services come in two shapes — choose based on whether the domain owns infrastructure (clients, adapters, pipelines, parsers, templates):
**Thin domain → flat module (`app/services/<domain>.py`).** Default. Just a class with `db`, repo calls, and domain exceptions. Examples: `user.py`, `conversation.py`, `invitation.py`.
**Thick domain → subpackage (`app/services/<domain>/`).** When the domain has its own infra. The subpackage contains both the service classes AND the infra (clients, adapters, pipeline modules, domain-specific exceptions). External callers only import from the package root — sub-modules are package-internal.
```
app/services/billing/
├── __init__.py # re-exports BillingService (the public facade)
├── facade.py # BillingService — the only thing routes see
├── checkout_service.py # internal sub-service
├── credit_service.py
├── subscription_service.py
├── webhook_handler.py
├── stripe_client.py # external API client (infra)
├── pricing.py # pure data
├── exceptions.py # domain-specific, inherits from core/exceptions
└── handlers/ # event handler modules (infra)
```
Other thick domains using the same shape: `services/rag/` (ingestion + vectorstore + embeddings + connectors), `services/channels/` (Slack + Telegram adapters + router), `services/email/` (providers + templates).
Rules for thick subpackages:
- Public API: only the top-level facade exported from `__init__.py`. Routes/workers never import sub-modules directly.
- Domain-specific exceptions live in the subpackage and inherit from `core/exceptions.py` base classes.
- Top-level `app/` is reserved for framework concerns (`api/`, `core/`, `db/`, `repositories/`, `schemas/`, `services/`, `worker/`, `agents/`, `commands/`, `clients/`). No new top-level domain packages.
## Dependency Injection (`app/api/deps.py`)
Use `Annotated` type aliases — never raw `Depends()` in route signatures:
```python
DBSession = Annotated[AsyncSession, Depends(get_db_session)]
UserSvc = Annotated[UserService, Depends(get_user_service)]
CurrentUser = Annotated[User, Depends(get_current_user)]
CurrentAdmin = Annotated[User, Depends(RoleChecker(UserRole.ADMIN))]
```
Service factories take `DBSession` and return service instances:
```python
def get_user_service(db: DBSession) -> UserService:
return UserService(db)
```
## Routes (`app/api/routes/v1/`)
HTTP layer only — validate, delegate, return:
```python
@router.get("/{user_id}", response_model=UserRead)
async def get_user(user_id: UUID, service: UserSvc, user: CurrentUser) -> Any:
return await service.get_by_id(user_id)
```
Rules:
- Return type is always `-> Any` (response_model handles serialization)
- Use `status_code=status.HTTP_201_CREATED` for POST, `HTTP_204_NO_CONTENT` for DELETE
- DELETE endpoints: `response_model=None`
- Pagination: `skip: int = Query(0, ge=0)`, `limit: int = Query(50, ge=1, le=100)`
.claude/rules/code-style.md
+67
View File
@@ -0,0 +1,67 @@
---
description: Code style, formatting, naming, imports, and type hints
globs: ["backend/**/*.py", "*.py"]
---
# Code Style
## Formatting
- Use `ruff` for linting and formatting: `ruff check . --fix && ruff format .`
- Line length: 120 characters
## Type Hints
- Type hints on ALL function signatures — parameters and return types
- Use modern syntax: `str | None` not `Optional[str]`, `list[User]` not `List[User]`
- Use `Annotated[Type, Depends(...)]` for DI (defined as aliases in `deps.py`)
- Use `dict[str, Any]` for generic dicts
- Use `Literal["value1", "value2"]` for string enums in schemas
- Use `TYPE_CHECKING` block for circular import resolution:
```python
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from app.db.models.session import Session
```
## Naming
| Element | Convention | Example |
|---------|-----------|---------|
| Files | snake_case | `user_repo.py`, `conversation_service.py` |
| Classes | PascalCase | `UserService`, `ConversationRead` |
| Functions/variables | snake_case | `get_by_id`, `user_service` |
| Constants | UPPER_CASE | `DEFAULT_SYSTEM_PROMPT` |
| Private | _leading_underscore | `_create_agent` |
| DB tables | snake_case plural | `users`, `conversations` |
| API URLs | kebab-case | `/api/v1/conversations` |
## Imports — strictly ordered, separated by blank lines
```python
# 1. Standard library
import logging
from collections.abc import AsyncGenerator
from datetime import UTC, datetime
from typing import Annotated, Any
from uuid import UUID
# 2. Third-party
from fastapi import APIRouter, Depends, Query, status
from pydantic import BaseModel, ConfigDict, Field
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
# 3. Local application
from app.api.deps import CurrentUser, UserSvc
from app.core.exceptions import NotFoundError
from app.schemas.user import UserCreate, UserRead
```
## Other Conventions
- `datetime.now(UTC)` not `datetime.utcnow()`
- `secrets.compare_digest()` for constant-time comparisons
- `__repr__` on all DB models
- Async for PostgreSQL/MongoDB I/O, sync for SQLite
- Keyword-only args in repo functions after `db` parameter
.claude/rules/exceptions-security.md
+54
View File
@@ -0,0 +1,54 @@
---
description: Exception handling patterns and security conventions
globs: ["backend/app/core/**/*.py", "backend/app/services/**/*.py"]
---
# Exceptions & Security
## Domain Exceptions (`app/core/exceptions.py`)
All extend `AppException`. Always pass `message` and `details`:
```python
raise NotFoundError(message="User not found", details={"user_id": str(user_id)})
raise AlreadyExistsError(message="Email already registered", details={"email": email})
raise AuthenticationError(message="Invalid or expired token")
raise AuthorizationError(message="Role 'admin' required for this action")
```
Exception handlers in `api/exception_handlers.py` automatically:
- Map to HTTP status codes
- Log with structured context (path, method, error code)
- Return consistent JSON error format
- Add `WWW-Authenticate: Bearer` header on 401
## Security Patterns
JWT auth (`core/security.py`):
- `create_access_token(subject)` / `create_refresh_token(subject)` — encode with `jwt.encode()`
- `verify_token(token)` → `dict | None` — decode with `jwt.decode()`
- Token payload: `{"exp": ..., "sub": user_id, "type": "access"|"refresh"}`
Password hashing:
- `get_password_hash(password)` — bcrypt
- `verify_password(plain, hashed)` — bcrypt `checkpw`
- NEVER store plain passwords
API keys:
- `secrets.compare_digest()` for constant-time comparison
- `APIKeyHeader(name=settings.API_KEY_HEADER, auto_error=False)`
## Role-Based Access Control
```python
class RoleChecker:
def __init__(self, required_role: UserRole) -> None:
self.required_role = required_role
async def __call__(self, user: Annotated[User, Depends(get_current_user)]) -> User:
if not user.has_role(self.required_role):
raise AuthorizationError(message=f"Role '{self.required_role.value}' required")
return user
CurrentAdmin = Annotated[User, Depends(RoleChecker(UserRole.ADMIN))]
```
.claude/rules/frontend.md
+27
View File
@@ -0,0 +1,27 @@
---
description: Frontend conventions for Next.js
globs: ["frontend/**/*.ts", "frontend/**/*.tsx", "frontend/**/*.css"]
---
# Frontend Conventions
## Stack
- Next.js 15 with App Router
- TypeScript strict mode
- Tailwind CSS for styling
- i18n support built-in
## Structure
- Pages in `frontend/src/app/` following Next.js App Router conventions
- Reusable components in `frontend/src/components/`
- API client functions in `frontend/src/lib/`
- Types in `frontend/src/types/`
## Conventions
- Use `"use client"` directive only when component needs client-side interactivity
- Prefer Server Components by default
- Use `fetch` with proper error handling for API calls
- Keep components small and focused — extract when a component exceeds ~100 lines
.claude/rules/schemas-models.md
+90
View File
@@ -0,0 +1,90 @@
---
description: Pydantic schema patterns and SQLAlchemy model conventions
globs: ["backend/app/schemas/**/*.py", "backend/app/db/models/**/*.py", "backend/app/db/base.py"]
---
# Schemas & Models
## Pydantic Schemas (`app/schemas/`)
Base schema with shared config:
```python
class BaseSchema(BaseModel):
model_config = ConfigDict(
from_attributes=True,
populate_by_name=True,
str_strip_whitespace=True,
)
```
Separate models per operation:
```python
class UserCreate(BaseSchema):
email: EmailStr = Field(max_length=255)
password: str = Field(min_length=8, max_length=128)
full_name: str | None = Field(default=None, max_length=255)
class UserUpdate(BaseSchema):
email: EmailStr | None = Field(default=None, max_length=255)
password: str | None = Field(default=None, min_length=8, max_length=128)
full_name: str | None = Field(default=None, max_length=255)
is_active: bool | None = None
class UserRead(BaseSchema, TimestampSchema):
id: UUID
email: EmailStr
full_name: str | None = None
role: UserRole = UserRole.USER
avatar_url: str | None = None
class UserList(BaseSchema):
items: list[UserRead]
total: int
```
Rules:
- `*Create` — required fields for creation, with `Field()` constraints
- `*Update` — all fields optional (`type | None = None`)
- `*Read` — includes `id` and timestamps, inherits `TimestampSchema`
- `*List` — `items` list + `total` count
- Use `@field_validator` for complex deserialization (e.g., JSON string → dict)
## SQLAlchemy Models (`app/db/models/`)
```python
class User(Base, TimestampMixin):
__tablename__ = "users"
id: Mapped[UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
email: Mapped[str] = mapped_column(String(255), unique=True, index=True, nullable=False)
hashed_password: Mapped[str | None] = mapped_column(String(255), nullable=True)
is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
conversations: Mapped[list["Conversation"]] = relationship(
"Conversation", back_populates="user", cascade="all, delete-orphan"
)
def __repr__(self) -> str:
return f"<User(id={self.id}, email={self.email})>"
```
Rules:
- Always inherit `Base` and `TimestampMixin` (provides `created_at`, `updated_at`)
- Use `Mapped[type]` with `mapped_column()` for all columns
- ForeignKey with `ondelete="CASCADE"` for parent references
- Always define `__repr__`
- Naming convention in `Base.metadata`: `{table}_{col}_key`, `{table}_{col}_fkey`, etc.
## TimestampMixin
```python
class TimestampMixin:
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), nullable=False
)
updated_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), onupdate=func.now(), nullable=True
)
```
.claude/rules/testing.md
+85
View File
@@ -0,0 +1,85 @@
---
description: Testing standards, fixtures, async test patterns
globs: ["backend/tests/**/*.py", "tests/**/*.py", "**/test_*.py", "**/conftest.py"]
---
# Testing
## Running Tests
```bash
cd backend
uv run pytest # all tests
uv run pytest tests/test_file.py -v # single file
uv run pytest -k "test_name" -v # by name
uv run pytest --cov=app # with coverage
```
## Structure
- Mirror source layout: `app/services/user.py` → `tests/services/test_user.py`
- Shared fixtures in `tests/conftest.py`
## Naming
```python
# test_<action>_<scenario>_<expected_result>
def test_create_user_with_duplicate_email_raises_already_exists_error
def test_get_conversation_not_found_raises_not_found_error
def test_list_conversations_returns_only_user_owned
```
## Fixtures
```python
@pytest.fixture
def user_service(db: AsyncSession) -> UserService:
return UserService(db)
@pytest.fixture
async def test_user(db: AsyncSession) -> User:
return await user_repo.create(db, email="test@example.com", hashed_password="hashed")
```
## Async Tests
```python
import pytest
@pytest.mark.asyncio
async def test_get_user_by_id(user_service: UserService, test_user: User):
result = await user_service.get_by_id(test_user.id)
assert result.email == test_user.email
```
## API Tests
Use `httpx.AsyncClient`, not `TestClient`:
```python
@pytest.mark.asyncio
async def test_create_user(client: AsyncClient, auth_headers: dict):
response = await client.post(
"/api/v1/users",
json={"email": "new@example.com", "password": "securepass123"},
headers=auth_headers,
)
assert response.status_code == 201
assert response.json()["email"] == "new@example.com"
```
## Exception Testing
```python
@pytest.mark.asyncio
async def test_get_user_not_found(user_service: UserService):
with pytest.raises(NotFoundError):
await user_service.get_by_id(UUID("00000000-0000-0000-0000-000000000000"))
```
## Rules
- Each test is independent — no shared mutable state
- Use plain `assert` (pytest rewrites for detailed output)
- One logical assertion per test (multiple asserts are fine if testing one behavior)
- Use factory fixtures for test data, not raw dicts
.claude/settings.json
+17
View File
@@ -0,0 +1,17 @@
{
"permissions": {
"allow": [
"Read",
"Glob",
"Grep",
"Bash(git status*)",
"Bash(git diff*)",
"Bash(git log*)",
"Bash(git branch*)",
"Bash(cd backend && uv run pytest*)",
"Bash(cd backend && uv run ruff*)",
"Bash(cd backend && uv run ty*)",
"Bash(cd backend && uv run alembic*)"
]
}
}
.github/workflows/ci.yml
+177
View File
@@ -0,0 +1,177 @@
name: CI
on:
push:
branches: [main, master]
pull_request:
branches: [main, master]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
version: "latest"
- name: Set up Python
run: uv python install 3.13
- name: Install dependencies
run: uv sync --directory backend --dev
- name: Run ruff check
run: uv run --directory backend ruff check app tests cli
- name: Run ruff format check
run: uv run --directory backend ruff format app tests cli --check
- name: Run ty
run: uv run --directory backend ty check
test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: test_db
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
redis:
image: redis:7-alpine
ports:
- 6379:6379
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
version: "latest"
- name: Set up Python
run: uv python install 3.13
- name: Install dependencies
run: uv sync --directory backend --dev
- name: Run tests
run: uv run --directory backend pytest tests/ -v --cov=app --cov-report=xml
env:
POSTGRES_HOST: localhost
POSTGRES_PORT: 5432
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: test_db
REDIS_HOST: localhost
REDIS_PORT: 6379
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
with:
files: ./backend/coverage.xml
fail_ci_if_error: false
test-frontend:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Bun
uses: oven-sh/setup-bun@v2
- name: Install dependencies
run: bun install --frozen-lockfile
working-directory: frontend
- name: Run lint
run: bun run lint
working-directory: frontend
- name: Run type check
run: bun run type-check
working-directory: frontend
- name: Run unit tests with coverage
run: bun run test:coverage
working-directory: frontend
- name: Install Playwright browsers
run: bunx playwright install --with-deps
working-directory: frontend
- name: Run E2E tests
run: bun run test:e2e
working-directory: frontend
env:
CI: true
security:
name: Security Scan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
version: "latest"
- name: Set up Python
run: uv python install 3.13
- name: Install dependencies
run: uv sync --directory backend --dev
- name: Install pip-audit
run: uv pip install pip-audit
- name: Run pip-audit
run: uv run pip-audit --require-hashes=false --progress-spinner=off
working-directory: backend
docker:
runs-on: ubuntu-latest
needs: [lint, test]
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build Docker image
uses: docker/build-push-action@v6
with:
context: ./backend
push: false
load: true
tags: agent_delta:latest
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@v0.36.0
with:
image-ref: agent_delta:latest
format: table
exit-code: 0
severity: CRITICAL,HIGH
.gitignore
+103
View File
@@ -0,0 +1,103 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
# Python lib directories (but not frontend src/lib)
/lib/
/lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
# PyInstaller
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
# Translations
*.mo
*.pot
# Environments
.env
.env.local
.env.*.local
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# IDE
.idea/
.vscode/
*.swp
*.swo
*~
# ty
.ty_cache/
# ruff
.ruff_cache/
# Jupyter Notebook
.ipynb_checkpoints
# pytype
.pytype/
# Cython debug symbols
cython_debug/
# Local development
*.db
*.sqlite
*.sqlite3
# Logs
*.log
logs/
# Docker
.docker/
# OS
.DS_Store
Thumbs.db
# Project specific
AGENTS.md
+66
View File
@@ -0,0 +1,66 @@
# AGENTS.md
This file provides guidance for AI coding agents (Codex, Copilot, Cursor, Zed, OpenCode).
## Project Overview
**agent_delta** - FastAPI application generated with [Full-Stack AI Agent Template](https://github.com/vstorm-co/full-stack-ai-agent-template).
**Stack:** FastAPI + Pydantic v2, PostgreSQL
, JWT + API Key auth, Redis
, pydantic_deep (openai), RAG (qdrant), Next.js 15 (i18n)
## Commands
```bash
# Run server
cd backend && uv run uvicorn app.main:app --reload
# Tests & lint
pytest
ruff check . --fix && ruff format .
# Migrations
uv run alembic upgrade head
uv run alembic revision --autogenerate -m "Description"
# RAG
uv run agent_delta rag-ingest /path/to/file.pdf --collection docs
uv run agent_delta rag-search "query" --collection docs
# Sync Sources
uv run agent_delta cmd rag-sources
uv run agent_delta cmd rag-source-add
uv run agent_delta cmd rag-source-sync
```
## Project Structure
```
backend/app/
├── api/routes/v1/ # Endpoints
├── services/ # Business logic
├── repositories/ # Data access
├── schemas/ # Pydantic models
├── db/models/ # DB models
├── agents/ # AI agents
├── rag/ # RAG (embeddings, vector store, ingestion)
│ └── connectors/ # Sync source connectors
└── commands/ # CLI commands
```
## Key Conventions
- `db.flush()` in repositories, not `commit()`
- Services raise `NotFoundError`, `AlreadyExistsError`
- Separate `Create`, `Update`, `Response` schemas
- Commands auto-discovered from `app/commands/`
- Document ingestion via CLI and API upload
- Sync sources: configurable connectors with scheduled sync
## More Info
- `docs/architecture.md` - Architecture details
- `docs/adding_features.md` - How to add features
- `docs/testing.md` - Testing guide
- `docs/patterns.md` - Code patterns
CLAUDE.md
+153
View File
@@ -0,0 +1,153 @@
# CLAUDE.md
## Project Overview
**agent_delta** - FastAPI application generated with [Full-Stack AI Agent Template](https://github.com/vstorm-co/full-stack-ai-agent-template).
**Stack:** FastAPI + Pydantic v2, PostgreSQL (async via asyncpg)
, JWT + API Key auth, Redis, RAG (qdrant), Taskiq, Next.js 15 (i18n)
## Commands
```bash
# Backend
cd backend
uv run uvicorn app.main:app --reload --port 8000
uv run pytest
uv run pytest tests/test_file.py::test_name -v
uv run ruff check . --fix && uv run ruff format .
uv run ty check
# Database migrations
uv run alembic upgrade head
uv run alembic revision --autogenerate -m "Description"
# Frontend
cd frontend
bun dev
bun test
bun run lint
# Docker
docker compose up -d
# RAG
uv run agent_delta rag-collections
uv run agent_delta rag-ingest /path/to/file.pdf --collection docs
uv run agent_delta rag-search "query" --collection docs
# Sync Sources
uv run agent_delta cmd rag-sources
uv run agent_delta cmd rag-source-add
uv run agent_delta cmd rag-source-sync
```
## Project Structure
```
backend/app/
├── main.py # FastAPI app with lifespan (startup/shutdown)
├── api/
│ ├── deps.py # Annotated DI aliases (DBSession, CurrentUser, *Svc)
│ ├── exception_handlers.py
│ └── routes/v1/ # HTTP endpoints — call services, never repos
├── core/
│ ├── config.py # pydantic-settings Settings class
│ ├── security.py # JWT (PyJWT), bcrypt password hashing, API key verification
│ ├── exceptions.py # Domain exceptions (AppException → NotFoundError, etc.)
│ └── middleware.py # RequestID, SecurityHeaders, CORS
├── db/
│ ├── base.py # DeclarativeBase, TimestampMixin, naming convention
│ ├── session.py # Engine, async_session_maker, get_db_session (auto-commit)
│ └── models/ # SQLAlchemy models (Mapped[] type hints)
├── schemas/ # Pydantic v2 models: *Create, *Update, *Read, *List
├── repositories/ # Data access functions — db.flush(), never commit
├── services/ # Business logic — flat *.py for thin domains, subpackage for thick
│ ├── user.py # thin: just a class with db + repo calls
│ ├── rag/ # thick: ingestion + vectorstore + embeddings + connectors
├── agents/ # AI agent wrappers + tools
├── worker/ # Background tasks (Celery/Taskiq/ARQ + in-process)
└── commands/ # CLI commands (auto-discovered)
```
## Architecture: Routes → Services → Repositories
**Routes** (`api/routes/v1/`) — HTTP layer only: validate input via Pydantic, call service, return response. Never import repositories.
**Services** (`services/`) — Business logic: class with `__init__(self, db)`, orchestrate repos, raise domain exceptions (`NotFoundError`, `AlreadyExistsError`, etc.).
**Repositories** (`repositories/`) — Pure data access functions. Always use `db.flush()` + `db.refresh()`, NEVER `db.commit()`. Session auto-commits via `get_db_session`.
## Dependency Injection Pattern
All DI uses `Annotated` type aliases defined in `api/deps.py`:
```python
# deps.py
DBSession = Annotated[AsyncSession, Depends(get_db_session)]
UserSvc = Annotated[UserService, Depends(get_user_service)]
CurrentUser = Annotated[User, Depends(get_current_user)]
CurrentAdmin = Annotated[User, Depends(RoleChecker(UserRole.ADMIN))]
# Route usage — no raw Depends() in function signatures
@router.get("/{id}", response_model=ConversationRead)
async def get_conversation(
id: UUID, service: ConversationSvc, user: CurrentUser
) -> Any:
return await service.get(id, user_id=user.id)
```
## Schema Conventions (Pydantic v2)
- Base: `BaseSchema` with `ConfigDict(from_attributes=True, str_strip_whitespace=True)`
- Separate models per operation: `*Create`, `*Update`, `*Read`
- List responses: `*List` with `items: list[*Read]` and `total: int`
- Update schemas: all fields `Optional` (`str | None = None`)
- Use `Field(max_length=255)`, `Field(min_length=8)`, `EmailStr`
- `@field_validator` for deserialization (e.g., JSON string → dict for SQLite)
- IDs are `UUID` type
## Exception Handling
Domain exceptions in `core/exceptions.py` — all extend `AppException`:
| Exception | HTTP | Code |
|-----------|------|------|
| `NotFoundError` | 404 | `NOT_FOUND` |
| `AlreadyExistsError` | 409 | `ALREADY_EXISTS` |
| `ValidationError` | 422 | `VALIDATION_ERROR` |
| `AuthenticationError` | 401 | `AUTHENTICATION_ERROR` |
| `AuthorizationError` | 403 | `AUTHORIZATION_ERROR` |
| `BadRequestError` | 400 | `BAD_REQUEST` |
| `ExternalServiceError` | 503 | `EXTERNAL_SERVICE_ERROR` |
Always pass `message` and `details` dict: `raise NotFoundError(message="User not found", details={"user_id": id})`
## Response Format
```python
# Single item — use response_model
@router.get("/{id}", response_model=ConversationRead)
# List — return *List schema
@router.get("", response_model=ConversationList)
# Create — 201
@router.post("", response_model=UserRead, status_code=status.HTTP_201_CREATED)
# Delete — 204, no body
@router.delete("/{id}", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
# All route return types are -> Any (avoids double Pydantic validation)
```
## Key Conventions
- Return type `-> Any` on route handlers (response_model handles serialization)
- Use `Query(default, ge=0, le=100, description="...")` for query params
- Keyword-only args in repo functions: `create(db, *, email: str, name: str)`
- `__repr__` on all DB models
- `datetime.now(UTC)` not `datetime.utcnow()`
- `secrets.compare_digest()` for API key comparison
- `TypedDict` for lifespan state
- Imports: stdlib → third-party → local, with `TYPE_CHECKING` block for circular refs
CONTRIBUTING.md
+64
View File
@@ -0,0 +1,64 @@
# Contributing to agent_delta
## Development setup
```bash
# Backend (uv-based)
cd backend
uv sync # install all deps including dev extras
cp .env.example .env # then fill in required vars (see ENV_VARS.md)
uv run uvicorn app.main:app --reload --port 8000
uv run alembic upgrade head # apply migrations
# Frontend (bun-based)
cd ../frontend
bun install
bun dev # http://localhost:3000
# Or everything in Docker
docker compose up
```
## Code style
- **Python:** ruff (`uv run ruff check . --fix && uv run ruff format .`). Line length 120.
- **Type hints:** modern syntax (`str | None`, `list[X]`, `dict[str, Any]`). Use `Annotated[T, Depends(...)]` for DI in route signatures.
- **TypeScript:** strict mode, no `any` unless typed external API. ESLint + Prettier (run `bun run lint`).
- **Imports:** stdlib → third-party → local, separated by blank lines. Use `TYPE_CHECKING` block to break circular refs.
- **Datetime:** `datetime.now(UTC)` not `datetime.utcnow()`.
- **Comparisons:** `secrets.compare_digest()` for tokens/keys (constant-time).
## Testing
```bash
cd backend
uv run pytest # all backend tests
uv run pytest tests/test_file.py::test -v # single test
uv run pytest -k "name_substring" -v # by name pattern
uv run pytest --cov=app # with coverage
```
```bash
cd frontend
bun test # vitest
bunx tsc --noEmit # type-check without emit
```
## Architecture rules
- **Routes** never import repositories directly. Always go through a service.
- **Services** raise domain exceptions (`NotFoundError`, `AlreadyExistsError`) — never return `None` for "not found".
- **Repositories** use `db.flush()` + `db.refresh()`, NEVER `db.commit()` (session auto-commits in `get_db_session`).
- **Pydantic schemas:** separate `*Create`, `*Update`, `*Read`, `*List` per operation.
- **Migrations:** one Alembic revision per logical change; never edit a merged migration.
See `docs/architecture.md` for the full layered architecture rules.
## Pull-request checklist
- [ ] `uv run ruff check . && uv run ruff format --check .` clean
- [ ] `cd frontend && bunx tsc --noEmit` clean
- [ ] Tests added for new code paths; `uv run pytest` green
- [ ] If schema changed: alembic migration committed (`uv run alembic revision --autogenerate -m "..."`)
- [ ] Updated `ENV_VARS.md` if new env vars added
- [ ] Updated `CHANGELOG.md` (if applicable)
ENV_VARS.md
+66
View File
@@ -0,0 +1,66 @@
# Environment variables
Reference for `agent_delta` runtime configuration. The
authoritative source is `backend/.env.example` — this doc explains what each
group is for and which are required vs optional.
> Quick start: copy `backend/.env.example` to `backend/.env` and fill in the
> blanks marked **Required**. Defaults are sensible for local development.
## Project
| Variable | Required | Default | Description |
|---|---|---|---|
| `PROJECT_NAME` | optional | `agent_delta` | Used in logs, OpenAPI title, email templates |
| `DEBUG` | optional | `true` | When `true`, FastAPI returns full tracebacks |
| `ENVIRONMENT` | optional | `local` | Free-form tag: `local` / `staging` / `production` |
| `TIMEZONE` | optional | `UTC` | IANA TZ name (e.g. `Europe/Warsaw`) |
| `BACKEND_URL` | optional | `http://localhost:8000` | Used by frontend BFF + email link generation |
| `FRONTEND_URL` | optional | `http://localhost:3000` | Used by password-reset / magic-link emails |
## Auth & secrets
| Variable | Required | Default | Description |
|---|---|---|---|
| `SECRET_KEY` | **required in prod** | (generated) | JWT signing key. Rotating invalidates all tokens |
| `API_KEY` | **required in prod** | (generated) | Static admin/service-to-service key for `X-API-Key` header |
| `ACCESS_TOKEN_EXPIRE_MINUTES` | optional | `30` | JWT access token lifetime |
| `REFRESH_TOKEN_EXPIRE_MINUTES` | optional | `10080` | JWT refresh token lifetime (7 days) |
## Database
| Variable | Required | Default | Description |
|---|---|---|---|
| `DATABASE_URL` | **required** | `postgresql+asyncpg://...` | Full async connection string |
| `DB_POOL_SIZE` | optional | `5` | Number of long-lived connections |
| `DB_MAX_OVERFLOW` | optional | `10` | Burst capacity above pool size |
## LLM / AI
| Variable | Required | Default | Description |
|---|---|---|---|
| `OPENAI_API_KEY` | **required** | — | From platform.openai.com |
| `AI_MODEL` | optional | `gpt-5.5` | Default model used by agent (provider-specific) |
| `LOGFIRE_TOKEN` | optional | — | When set, ships traces to Logfire (logfire.pydantic.dev) |
## RAG (qdrant)
| Variable | Required | Default | Description |
|---|---|---|---|
| `QDRANT_URL` | **required** | `http://localhost:6333` | Qdrant REST endpoint |
| `QDRANT_API_KEY` | optional | — | Auth (cloud Qdrant) |
## Redis
| Variable | Required | Default | Description |
|---|---|---|---|
| `REDIS_URL` | **required** | `redis://localhost:6379/0` | Used by cache, rate-limiter, session store |
## Validation
```bash
# Confirm settings load without errors:
cd backend && uv run python -c "from app.core.config import settings; print(settings.model_dump_json(indent=2))"
```
If any **Required** var is missing, FastAPI raises `pydantic_settings.SettingsError` on startup — check the message for which field.
MANUAL_STEPS.md
+72
View File
@@ -0,0 +1,72 @@
# Manual setup steps for agent_delta
The generator created the code. These are the **one-time external setup steps**
that can't be automated — accounts to create, keys to copy, services to provision.
> Skip ahead to "After every deploy" at the bottom for things you'll re-do
> regularly. Items above are one-time per environment.
---
## Secrets
```bash
cp backend/.env.example backend/.env
```
Then in `backend/.env`:
- [ ] **`SECRET_KEY`** — replace with a fresh value: `openssl rand -hex 32`
- [ ] **`API_KEY`** — replace with a fresh value: `openssl rand -hex 32`
These are used to sign JWTs and authenticate service-to-service calls. Rotate at every environment promotion (dev → staging → prod each get their own).
## PostgreSQL
- [ ] Provision a PostgreSQL ≥ 14 instance (local: `docker compose up -d db`; managed: Neon / Supabase / RDS / Cloud SQL).
- [ ] Set `DATABASE_URL` in `.env` to the **async** connection string: `postgresql+asyncpg://user:pass@host:5432/dbname`.
- [ ] Run migrations: `cd backend && uv run alembic upgrade head`.
## OpenAI
- [ ] Create API key at https://platform.openai.com/api-keys.
- [ ] Set `OPENAI_API_KEY` in `.env`.
- [ ] (Optional) Set spending limit on OpenAI dashboard to avoid surprise bills.
## RAG (qdrant)
- [ ] Local: `docker compose up -d qdrant`.
- [ ] Cloud: provision Qdrant Cloud, set `QDRANT_URL` + `QDRANT_API_KEY`.
- [ ] (Optional) Ingest seed documents: `uv run agent_delta rag-ingest /path/to/file.pdf --collection docs`.
## Redis
- [ ] Local: `docker compose up -d redis` (already in compose file).
- [ ] Managed: Upstash / Redis Cloud / ElastiCache. Set `REDIS_URL` in `.env`.
## Logfire (Pydantic observability)
- [ ] Create account at https://logfire.pydantic.dev.
- [ ] Run `uv run logfire auth` once locally to bootstrap.
- [ ] Get write token → set `LOGFIRE_TOKEN` in `.env` for non-local environments.
---
## After every deploy
- [ ] Run database migrations: `alembic upgrade head` (CI step or post-deploy job).
- [ ] Smoke test `/api/v1/health` returns `{"status": "ok"}`.
- [ ] Frontend loads, login → dashboard flow works.
- [ ] Logs flowing to your aggregator.
---
## Where to find more
- `ENV_VARS.md` — exhaustive env var reference
- `docs/deploy.md` — platform-specific deployment recipes
- `SECURITY.md` — security model + production hardening checklist
- `CONTRIBUTING.md` — dev environment setup
- `docs/architecture.md` — codebase layered architecture rules
Makefile
+377
View File
@@ -0,0 +1,377 @@
.PHONY: install format lint test run clean help db-init dev dev-down dev-logs dev-rebuild dev-frontend docker-clean stage stage-down prod prod-down
# === Environments ===========================================================
# `make dev` — local development (docker-compose.dev.yml + bind-mounted source)
# `make stage` — staging (docker-compose.yml — built images, no live reload)
# `make prod` — production (docker-compose.prod.yml — needs backend/.env + nginx)
# Each env has matching -down / -logs / -rebuild siblings.
# Wait for postgres to accept connections. Polls pg_isready instead of a
# fixed sleep — handles slow startups and cold-start image pulls.
define _wait_for_db
@echo "Waiting for PostgreSQL ($(1))..."
@for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do \
if docker compose -f $(1) exec -T db pg_isready -U postgres >/dev/null 2>&1; then \
echo " ✅ DB ready"; exit 0; \
fi; \
printf '.'; sleep 2; \
done; \
echo " ❌ DB not ready after 30s — check 'make dev-logs'"; exit 1
endef
# Fold the opt-in "antv" profile into `make dev` so the diagram sidecar starts
# automatically alongside db/app. The agent connects only when ENABLE_ANTV_CHARTS=true.
COMPOSE_DEV_PROFILES := --profile antv
# === Local dev: build → up → migrate ===
# Idempotent — re-run anytime. Migrations are no-ops when already at head;
# admin seeding is a separate target (`make seed`) so re-running `make dev`
# doesn't keep retrying user creation.
dev:
@echo "▶ Building backend image…"
docker compose -f docker-compose.dev.yml build app
@echo "▶ Starting services…"
@if ! docker compose -f docker-compose.dev.yml $(COMPOSE_DEV_PROFILES) up -d; then \
echo ""; \
echo "⚠ First start failed. Tearing down stale containers and retrying once…"; \
echo " (volumes preserved — DB data is safe; use 'make clean' for a full wipe)"; \
docker compose -f docker-compose.dev.yml down --remove-orphans; \
docker compose -f docker-compose.dev.yml $(COMPOSE_DEV_PROFILES) up -d; \
fi
$(call _wait_for_db,docker-compose.dev.yml)
@echo "▶ Applying migrations…"
docker compose -f docker-compose.dev.yml exec -T app agent_delta db upgrade
@echo ""
@echo "🚀 Dev stack ready:"
@echo " API: http://localhost:8000"
@echo " Docs: http://localhost:8000/docs"
@echo " Admin: http://localhost:8000/admin"
@echo " Frontend: http://localhost:3000 (run 'make dev-frontend' or 'cd frontend && bun dev')"
@echo ""
@echo "First time? Run 'make seed' to create the default admin user."
# === First-time setup: seed default admin user (one-shot) ===
# Skipped when admin@example.com already exists. Safe to run again — exits
# clean either way. Replace email/password before deploying anywhere real.
seed:
@echo "▶ Seeding admin user (admin@example.com / admin123)…"
@if docker compose -f docker-compose.dev.yml exec -T app \
agent_delta user list 2>/dev/null \
| grep -q "admin@example.com"; then \
echo " (admin@example.com already exists — nothing to do)"; \
else \
docker compose -f docker-compose.dev.yml exec -T app \
agent_delta user create \
--email admin@example.com --password admin123 --superuser \
&& echo " ✅ Admin created. Login at http://localhost:8000/admin"; \
fi
# Convenience: bootstrap a fresh checkout end-to-end.
bootstrap: dev seed
dev-down:
docker compose -f docker-compose.dev.yml $(COMPOSE_DEV_PROFILES) down
# Full wipe — containers, networks, AND volumes. Use after a corrupted state
# (e.g. detached networks, port conflicts that left orphans). DESTROYS DB data.
docker-clean:
@echo "▶ Removing containers, networks, AND volumes for the dev stack…"
@echo " ⚠️ This deletes all local DB data and uploaded files."
docker compose -f docker-compose.dev.yml $(COMPOSE_DEV_PROFILES) down -v --remove-orphans
@echo "✅ Cleaned. Run 'make dev' to start fresh."
dev-logs:
docker compose -f docker-compose.dev.yml $(COMPOSE_DEV_PROFILES) logs -f
dev-rebuild:
docker compose -f docker-compose.dev.yml build --no-cache app
docker compose -f docker-compose.dev.yml up -d --force-recreate app
dev-frontend:
docker compose -f docker-compose.frontend.yml up -d
@echo ""
@echo "✅ Frontend at http://localhost:3000 (backend must be up — 'make dev')"
# === Staging: built images, no bind mounts (production-like, local DB) ===
stage:
docker compose -f docker-compose.yml up -d --build
$(call _wait_for_db,docker-compose.yml)
docker compose -f docker-compose.yml exec -T app agent_delta db upgrade
@echo "✅ Staging stack at http://localhost:8000"
stage-down:
docker compose -f docker-compose.yml down
# === Production: external Nginx, real secrets in backend/.env ===
prod:
@test -f backend/.env || (echo "❌ backend/.env missing — run 'cp backend/.env.example backend/.env' and fill in real secrets" && exit 1)
docker compose --env-file backend/.env -f docker-compose.prod.yml up -d --build
@echo "▶ Waiting for DB then running migrations…"
@sleep 5
docker compose --env-file backend/.env -f docker-compose.prod.yml exec -T app agent_delta db upgrade
@echo "✅ Production stack up. Configure your nginx host with nginx/nginx.conf"
prod-down:
docker compose --env-file backend/.env -f docker-compose.prod.yml down
prod-logs:
docker compose --env-file backend/.env -f docker-compose.prod.yml logs -f
# Legacy alias
quickstart: dev
# === Setup ===
install:
uv sync --directory backend --dev
@echo ""
@echo "✅ Installation complete!"
@echo ""
@echo "Next steps:"
@echo " • make docker-db # Start PostgreSQL"
@echo " • make db-upgrade # Apply migrations"
@echo " • make run # Start development server"
@echo ""
@echo "Note: backend/.env is pre-configured for development"
# === Code Quality ===
format:
uv run --directory backend ruff format app tests cli
uv run --directory backend ruff check app tests cli --fix
lint:
uv run --directory backend ruff check app tests cli
uv run --directory backend ruff format app tests cli --check
uv run --directory backend ty check
# === Testing ===
test:
uv run --directory backend pytest tests/ -v
test-cov:
uv run --directory backend pytest tests/ -v --cov=app --cov-report=html --cov-report=term-missing
# === Database ===
db-init: docker-db
@echo "Waiting for PostgreSQL to be ready..."
@sleep 8
cd backend && uv run agent_delta db migrate -m "initial" || true
cd backend && uv run agent_delta db upgrade
@echo ""
@echo "✅ Database initialized!"
db-migrate:
@read -p "Migration message: " msg; \
uv run --directory backend agent_delta db migrate -m "$$msg"
db-upgrade:
uv run --directory backend agent_delta db upgrade
db-downgrade:
uv run --directory backend agent_delta db downgrade
db-current:
uv run --directory backend agent_delta db current
db-history:
uv run --directory backend agent_delta db history
# === Server ===
run:
uv run --directory backend agent_delta server run --reload
run-prod:
uv run --directory backend agent_delta server run --host 0.0.0.0 --port 8000
routes:
uv run --directory backend agent_delta server routes
# === Users ===
create-admin:
@echo "Creating admin user..."
uv run --directory backend agent_delta user create-admin
user-create:
uv run --directory backend agent_delta user create
user-list:
uv run --directory backend agent_delta user list
# === Taskiq ===
taskiq-worker:
uv run --directory backend agent_delta taskiq worker
taskiq-scheduler:
uv run --directory backend agent_delta taskiq scheduler
# === Docker: Backend (Development) ===
docker-up:
docker-compose build app
docker-compose up -d
@echo ""
@echo "✅ Backend services started!"
@echo " API: http://localhost:8000"
@echo " Docs: http://localhost:8000/docs"
@echo " PostgreSQL: localhost:5432"
@echo " Redis: localhost:6379"
docker-down:
docker-compose down
docker-compose -f docker-compose.frontend.yml down 2>/dev/null || true
docker-logs:
docker-compose logs -f
docker-build:
docker-compose build
docker-shell:
docker-compose exec app /bin/bash
# === Docker: Frontend (Development) ===
docker-frontend:
docker-compose -f docker-compose.frontend.yml up -d
@echo ""
@echo "✅ Frontend started!"
@echo " URL: http://localhost:3000"
@echo ""
@echo "Note: Backend must be running (make docker-up)"
docker-frontend-down:
docker-compose -f docker-compose.frontend.yml down
docker-frontend-logs:
docker-compose -f docker-compose.frontend.yml logs -f
docker-frontend-build:
docker-compose -f docker-compose.frontend.yml build
# === Docker: Production (with Traefik) ===
docker-prod:
docker-compose -f docker-compose.prod.yml up -d
@echo ""
@echo "✅ Production services started with Traefik!"
@echo ""
@echo "Endpoints (replace DOMAIN with your domain):"
@echo " Frontend: https://$$DOMAIN"
@echo " API: https://api.$$DOMAIN"
@echo " Traefik: https://traefik.$$DOMAIN"
docker-prod-down:
docker-compose -f docker-compose.prod.yml down
docker-prod-logs:
docker-compose -f docker-compose.prod.yml logs -f
docker-prod-build:
docker-compose -f docker-compose.prod.yml build
# === Docker: Individual Services ===
docker-db:
docker-compose up -d db
@echo ""
@echo "✅ PostgreSQL started on port 5432"
@echo " Connection: postgresql://postgres:postgres@localhost:5432/agent_delta"
docker-db-stop:
docker-compose stop db
docker-redis:
docker-compose up -d redis
@echo ""
@echo "✅ Redis started on port 6379"
docker-redis-stop:
docker-compose stop redis
# === Vercel (Frontend Deployment) ===
vercel-deploy:
cd frontend && npx vercel --prod
@echo ""
@echo "✅ Frontend deployed to Vercel!"
@echo " Set environment variables in Vercel dashboard:"
@echo " BACKEND_URL=https://api.your-domain.com"
@echo " BACKEND_WS_URL=wss://api.your-domain.com"
@echo " NEXT_PUBLIC_AUTH_ENABLED=true"
@echo " NEXT_PUBLIC_RAG_ENABLED=true"
# === Cleanup ===
clean:
find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
find . -type d -name .pytest_cache -exec rm -rf {} + 2>/dev/null || true
find . -type d -name .ruff_cache -exec rm -rf {} + 2>/dev/null || true
find . -type d -name .ty_cache -exec rm -rf {} + 2>/dev/null || true
rm -rf htmlcov/ .coverage coverage.xml
# === Help ===
help:
@echo ""
@echo "agent_delta - Available Commands"
@echo "======================================"
@echo ""
@echo "🚀 Bootstrap (first-time setup):"
@echo " make bootstrap 'make dev' + 'make seed' — full setup from a fresh clone"
@echo ""
@echo "Day-to-day dev:"
@echo " make dev Build + start dev stack + apply migrations (idempotent)"
@echo " make seed One-shot admin seed (admin@example.com / admin123)"
@echo " make dev-down Stop dev stack"
@echo " make dev-logs Tail dev container logs"
@echo " make dev-rebuild Force-rebuild backend image"
@echo " make docker-clean Wipe containers + networks + volumes (DESTROYS data)"
@echo " make dev-frontend Start frontend container (after 'make dev')"
@echo ""
@echo "📦 Other environments:"
@echo " make stage Production-like stack on localhost (no bind mounts)"
@echo " make prod Production stack (requires backend/.env + nginx)"
@echo ""
@echo "Setup (without Docker):"
@echo " make install Install Python deps + pre-commit hooks"
@echo ""
@echo "Development:"
@echo " make run Start dev server (with hot reload)"
@echo " make test Run tests"
@echo " make lint Check code quality"
@echo " make format Auto-format code"
@echo ""
@echo "Database:"
@echo " make db-init Initialize database (start + migrate)"
@echo " make db-migrate Create new migration"
@echo " make db-upgrade Apply migrations"
@echo " make db-downgrade Rollback last migration"
@echo " make db-current Show current migration"
@echo ""
@echo "Users:"
@echo " make create-admin Create admin user (for SQLAdmin access)"
@echo " make user-create Create new user (interactive)"
@echo " make user-list List all users"
@echo ""
@echo "RAG:"
@echo " uv run agent_delta rag-ingest <path> -c <collection> Ingest files"
@echo " uv run agent_delta rag-search <query> -c <collection> Search"
@echo " uv run agent_delta rag-collections List collections"
@echo " uv run agent_delta rag-sources List sync sources"
@echo " uv run agent_delta rag-source-add Add sync source"
@echo " uv run agent_delta rag-source-sync <id> Trigger sync"
@echo ""
@echo "Taskiq:"
@echo " make taskiq-worker Start Taskiq worker"
@echo " make taskiq-scheduler Start Taskiq scheduler"
@echo ""
@echo "Docker (Development):"
@echo " make docker-up Start backend services"
@echo " make docker-down Stop all services"
@echo " make docker-logs View backend logs"
@echo " make docker-build Build backend images"
@echo " make docker-frontend Start frontend (separate)"
@echo " make docker-frontend-down Stop frontend"
@echo " make docker-db Start only PostgreSQL"
@echo " make docker-redis Start only Redis"
@echo ""
@echo "Docker (Production with Traefik):"
@echo " make docker-prod Start production stack"
@echo " make docker-prod-down Stop production stack"
@echo " make docker-prod-logs View production logs"
@echo ""
@echo "Other:"
@echo " make routes Show all API routes"
@echo " make clean Clean cache files"
@echo ""
README.md
+277
View File
@@ -0,0 +1,277 @@
# agent_delta
agent delta project
> Generated with [Full-Stack AI Agent Template](https://github.com/vstorm-co/full-stack-ai-agent-template).
---
## Stack
| Component | Technology |
|-----------|-----------|
| **Backend** | FastAPI + Pydantic v2 |
| **Database** | PostgreSQL (async via asyncpg) |
| **Auth** | JWT + refresh tokens + API keys |
| **Cache** | Redis |
| **AI Framework** | pydantic_deep (openai) |
| **RAG** | qdrant vector store |
| **Tasks** | taskiq |
| **Frontend** | Next.js 15 + React 19 + Tailwind v4 |
---
## Prerequisites
| Tool | Version | Install |
|---|---|---|
| **Docker** | Desktop / Engine 24+ | <https://docs.docker.com/get-docker/> |
| **Make** | GNU Make 3.81+ (preinstalled on macOS/Linux) | Windows: install via [chocolatey](https://chocolatey.org/) `choco install make` or use WSL2 |
| **uv** | latest | `curl -LsSf https://astral.sh/uv/install.sh \| sh` |
| **bun** | 1.x | `curl -fsSL https://bun.sh/install \| bash` (or use `npm` / `pnpm` if you prefer) |
> **Windows users:** the Makefile and shell helpers assume bash. Use **WSL2** or **Git Bash** for the smoothest experience. The Docker workflow below works identically on macOS, Linux, and WSL2.
---
## Quick Start (Local Dev)
### First time
```bash
make bootstrap # = make dev + make seed
```
That's the only command you need on a fresh clone. After this, day-to-day is just `make dev`.
### Subsequent runs
```bash
make dev
```
`make dev` is **idempotent** — re-run it any time. It will:
1. Build the backend Docker image (cached after first run)
2. Start services via `docker-compose.dev.yml` (with hot-reload bind mounts)
3. Poll Postgres until it accepts connections (`pg_isready` — no fixed sleeps)
4. Apply pending Alembic migrations (no-op if already at head)
It does **not** re-seed the admin user — that lives in `make seed` and is run once. This way `make dev` stays cheap to re-run after every code/config change.
**Then access:**
- API: <http://localhost:8000>
- Docs: <http://localhost:8000/docs>
- Admin: <http://localhost:8000/admin> — `admin@example.com` / `admin123` after `make seed`
- Frontend: <http://localhost:3000> — start with `make dev-frontend` (Docker) or `cd frontend && bun install && bun dev` (local)
### Day-to-day commands
```bash
make dev # bootstrap or restart (idempotent, no admin re-seed)
make seed # one-shot admin creation (no-op if admin already exists)
make dev-down # stop everything
make dev-logs # tail logs (Ctrl-C to exit)
make dev-rebuild # force-rebuild backend image (after pyproject.toml change)
make dev-frontend # start the Next.js container
```
If you prefer running the backend on the host (not in Docker) — useful for breakpoints / IDE debugging:
```bash
make install # uv sync + pre-commit install
docker compose -f docker-compose.dev.yml up -d db redis milvus etcd minio
make db-upgrade # apply migrations
make run # run uvicorn locally with --reload
```
---
## Environments
| `make` target | Compose file | Use case |
|---|---|---|
| `make dev` | `docker-compose.dev.yml` | Local development with hot-reload + bind-mounted source. |
| `make stage` | `docker-compose.yml` | Production-like build, no bind mounts, runs on localhost. Good for sanity-checking before deploy. |
| `make prod` | `docker-compose.prod.yml` | Production. Requires `backend/.env` (copy from `backend/.env.example`, fill real secrets) and an external Nginx using `nginx/nginx.conf`. |
Each env has matching `-down`, `-logs`, `-rebuild` siblings (e.g. `make stage-down`).
---
## Project Structure
```
backend/app/
├── main.py # FastAPI app + lifespan
├── api/
│ ├── deps.py # Annotated DI aliases (DBSession, CurrentUser, *Svc)
│ ├── exception_handlers.py
│ └── routes/v1/ # HTTP endpoints — call services, never repos
├── core/
│ ├── config.py # pydantic-settings (reads .env)
│ ├── security.py # JWT, bcrypt, API key verification
│ ├── exceptions.py # AppException → NotFound / Auth / etc.
│ └── middleware.py
├── db/
│ ├── base.py # DeclarativeBase + TimestampMixin
│ └── models/ # SQLAlchemy models (Mapped[] type hints)
├── schemas/ # Pydantic v2: *Create / *Update / *Read / *List
├── repositories/ # Data access — db.flush() never commit
├── services/ # Business logic — raises domain exceptions
├── agents/ # AI agent wrappers + tools
├── rag/ # RAG: vectorstore + embeddings + ingestion + sources
│ └── connectors/ # Pluggable sync sources (Google Drive, S3, …)
├── worker/
│ ├── background/ # FastAPI BackgroundTasks fallback (in-process)
│ └── tasks/ # Distributed tasks (taskiq)
└── commands/ # Click CLI commands (auto-discovered by `agent_delta cmd …`)
frontend/src/
├── app/
│ ├── [locale]/ # next-intl routes (en/pl)
│ │ └── (dashboard)/ # Authenticated app
│ └── api/ # Server-side API proxies (forward auth cookies)
├── components/ # React components (chat, marketing, ui primitives)
├── hooks/ # useAuth, useChat, useConversations, …
├── stores/ # Zustand stores
└── lib/ # api-client, server-api, utils
```
---
## CLI
The generated project ships a Click CLI exposed as `agent_delta` (after `make install`):
```bash
agent_delta server run --reload # dev server
agent_delta db upgrade # apply migrations
agent_delta db migrate -m "message" # create new migration
agent_delta user create-admin # interactive admin creation
agent_delta rag-ingest <path> -c docs # ingest local files
agent_delta rag-search "query" -c docs # semantic search
agent_delta rag-collections # list collections
```
Run `make help` for a categorized list, or `agent_delta --help` for full CLI docs.
---
## Configuration
All backend config lives in `backend/.env` (committed for dev defaults). Key variables:
```bash
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=agent_delta
# OpenAI — required for chat + embeddings
OPENAI_API_KEY=sk-…
```
See `backend/.env.example` for the full list with comments.
For production, **never** commit secrets — `backend/.env` is gitignored. Fill it with real values on the server (or inject them via your platform's secret manager: Doppler, AWS Secrets Manager, GitHub Actions secrets, etc.). The same `backend/.env` is used for dev and prod — there is no separate `.env.prod`.
---
## Development
| Command | What it does |
|---|---|
| `make test` | Run pytest |
| `make lint` | Run ruff check + format check + ty |
| `make format` | Auto-format with ruff |
| `make db-migrate` | Generate a new migration from model changes (interactive) |
| `make db-upgrade` | Apply pending migrations |
| `make db-downgrade` | Roll back one migration |
| `make db-current` | Show current head |
| `make create-admin` | Interactive admin creation |
| `make user-list` | List all users |
---
## RAG (Knowledge Base)
Using **qdrant** as the vector store with **openai** embeddings.
```bash
# Ingest local files (recursive)
agent_delta rag-ingest /path/to/docs/ --collection documents --recursive
# Semantic search
agent_delta rag-search "your query" --collection documents
```
PDF parsing uses **pymupdf**. See `docs/howto/add-rag-source.md` to add a new source connector.
---
## Frontend
```bash
cd frontend
bun install
bun dev # http://localhost:3000
bun run lint
bun run build
```
The frontend talks to the backend through Next.js API route handlers in `src/app/api/*` (server-side proxy that forwards auth cookies to the FastAPI backend). Direct calls to `localhost:8000` from the browser are deliberately avoided.
i18n (PL + EN) ships out of the box via `next-intl`. Add a new locale by extending `messages/<lang>.json` and `src/i18n.ts`.
---
## Deployment
### Frontend → Vercel
```bash
cd frontend && npx vercel --prod
```
Set in the Vercel dashboard:
- `BACKEND_URL` = `https://api.your-domain.com`
- `BACKEND_WS_URL` = `wss://api.your-domain.com`
- `NEXT_PUBLIC_AUTH_ENABLED` = `true`
- `NEXT_PUBLIC_RAG_ENABLED` = `true`
### Backend → your server
```bash
# 1. SSH to the box, clone the repo
# 2. cp backend/.env.example backend/.env, fill in real secrets
# 3. Configure nginx using nginx/nginx.conf as reference
# 4. Bring up the stack:
make prod
# Day-to-day:
make prod-logs
make prod-down
```
Migrations run automatically on `make prod`. For a fresh deploy on a new host, the same `make prod` is the bootstrap command.
---
## Guides
| Guide | What |
|-------|-------|
| `docs/howto/add-api-endpoint.md` | Add a new REST endpoint |
| `docs/howto/add-agent-tool.md` | Create an agent tool |
| `docs/howto/customize-agent-prompt.md` | Tune system prompts |
| `docs/howto/add-background-task.md` | Add a background task |
| `docs/howto/add-rag-source.md` | Add a RAG document source |
| `docs/howto/add-sync-connector.md` | Build a custom sync connector |
---
*Generated with [Full-Stack AI Agent Template](https://github.com/vstorm-co/full-stack-ai-agent-template) v0.2.11.*
SECURITY.md
+54
View File
@@ -0,0 +1,54 @@
# Security
## Reporting a vulnerability
Email: **your@email.com** (or open a private security advisory on the repo). Please include:
- Affected version / commit
- Steps to reproduce
- Impact assessment (data exposure / privilege escalation / DoS / …)
We aim to acknowledge within 48h and ship a fix within 7 days for high-severity issues.
---
## Security model
### Authentication
- **JWT (`HS256`)** signed with `SECRET_KEY`. Access token TTL = `ACCESS_TOKEN_EXPIRE_MINUTES` (default 30 min). Refresh token TTL = `REFRESH_TOKEN_EXPIRE_MINUTES` (default 7 days).
- **Password hashing:** bcrypt via `passlib`. Plain passwords never persisted.
- **Stateless JWT** — no DB session table. Logout is client-side (drop tokens). For server-side revocation, regenerate with `--session-management`.
- **Admin API key** — static `settings.API_KEY` matched via `X-API-Key` header for service-to-service calls. Constant-time compared with `secrets.compare_digest()`.
### Authorization
- **Role-based** via `RoleChecker` dep (`UserRole.USER` / `UserRole.ADMIN`).
### Transport / network
- **CORS** — origin list from `settings.CORS_ORIGINS`. Restrict to your domains in production.
- **HTTPS** — enforce via reverse proxy (Nginx / Traefik / ALB). Strict-Transport-Security header set in middleware when `ENVIRONMENT=production`.
- **CSP** — frontend sets `frame-ancestors 'none'` by default to prevent click-jacking. See `frontend/next.config.ts` headers block.
### Data
- **Secrets** — read from environment via `pydantic-settings`. Never committed. See `.env.example` + `ENV_VARS.md`.
- **Audit log** — admin-mutating actions (user updates, deletes, impersonations, role changes) recorded in `app_admin_audit_log` table with actor + IP + payload snapshot.
- **RAG documents** — file uploads scoped per-org. No public read endpoint; all retrieval happens server-side during chat.
### Hardening checklist for production
- [ ] Rotate `SECRET_KEY` and `API_KEY` from generated defaults.
- [ ] Set `DEBUG=false` and `ENVIRONMENT=production`.
- [ ] Restrict `CORS_ORIGINS` to your domain(s).
- [ ] Tune `RATE_LIMIT_REQUESTS` / `RATE_LIMIT_PERIOD` in `.env`.
- [ ] Enforce HTTPS at the proxy layer.
- [ ] Run `pip-audit` / `bun audit` in CI for dependency vulnerabilities.
- [ ] Configure database backups + restore test schedule.
## Known limitations
- **No 2FA / MFA** out of the box. Plan to add TOTP via `pyotp` — see `notes/thingstofix.md` §A.13.
- **No SAML / OIDC** beyond Google OAuth. Enterprise SSO needs custom IdP integration.
- **No automatic PII redaction** in logs — be careful what you log.
- **No server-side session revocation** — JWTs valid until expiry. Compromised tokens require `SECRET_KEY` rotation (invalidates ALL sessions). Enable `--session-management` for selective revocation.
backend/.dockerignore
+57
View File
@@ -0,0 +1,57 @@
# Git
.git
.gitignore
# Python
__pycache__
*.py[cod]
*$py.class
*.so
.Python
.venv
venv/
ENV/
# IDE
.idea
.vscode
*.swp
*.swo
# Testing
.pytest_cache
.coverage
htmlcov/
.tox
.nox
# Documentation
docs/_build/
*.md
!README.md
# Build artifacts
dist/
build/
*.egg-info/
# Development files
.env
.env.local
*.db
*.sqlite
# Docker
Dockerfile*
docker-compose*.yml
.docker
# CI/CD
.github/
.gitlab-ci.yml
# Misc
.DS_Store
Thumbs.db
*.log
backend/.env.example
+114
View File
@@ -0,0 +1,114 @@
# agent_delta Environment Variables
# === Project ===
PROJECT_NAME=agent_delta
DEBUG=true
ENVIRONMENT=local
TIMEZONE=UTC # IANA timezone: UTC, Europe/Warsaw, America/New_York
MODELS_CACHE_DIR=./models_cache
# === Logfire ===
# Get your token at https://logfire.pydantic.dev
LOGFIRE_TOKEN=
LOGFIRE_SERVICE_NAME=agent_delta
LOGFIRE_ENVIRONMENT=development
# === PostgreSQL ===
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=agent_delta
# === JWT Auth ===
# Generate with: openssl rand -hex 32
SECRET_KEY=change-me-in-production-use-openssl-rand-hex-32
ACCESS_TOKEN_EXPIRE_MINUTES=10080
ALGORITHM=HS256
# === API Key Auth ===
API_KEY=change-me-in-production
API_KEY_HEADER=X-API-Key
# === Redis ===
REDIS_HOST=localhost
REDIS_PORT=6379
# REDIS_PASSWORD=
REDIS_DB=0
# === RAG Configuration ===
RAG_DEFAULT_COLLECTION=documents
RAG_TOP_K=10
RAG_CHUNK_SIZE=512
RAG_CHUNK_OVERLAP=50
RAG_CHUNKING_STRATEGY=recursive # recursive, markdown, or fixed
RAG_HYBRID_SEARCH=false # Enable BM25 + vector hybrid search
RAG_ENABLE_OCR=false # OCR fallback for scanned PDFs (requires tesseract-ocr installed)
# Vector Database (Qdrant)
QDRANT_HOST=localhost
QDRANT_PORT=6333
QDRANT_API_KEY=
# Reranker
HF_TOKEN=
CROSS_ENCODER_MODEL=cross-encoder/ms-marco-MiniLM-L6-v2
# === Taskiq ===
TASKIQ_BROKER_URL=redis://localhost:6379/1
TASKIQ_RESULT_BACKEND=redis://localhost:6379/1
# === AI Agent (pydantic_deep, openai) ===
OPENAI_API_KEY=
AI_MODEL=gpt-5.5
AI_TEMPERATURE=0.7
# === Web Search (Tavily) ===
# Get your API key at https://tavily.com
TAVILY_API_KEY=
# === AntV charts (advanced diagrams via mcp-server-chart sidecar) ===
# Opt-in. The create_map tool works without this; only the AntV diagram tools
# need the sidecar, which `make dev` starts automatically.
ENABLE_ANTV_CHARTS=false
# MCP endpoint of the antvis-chart sidecar (default matches docker-compose).
ANTV_MCP_URL=http://antvis-chart:1122/mcp
# Optional self-hosted GPT-Vis render backend (empty = AntV's public service).
ANTV_VIS_REQUEST_SERVER=
# Comma-separated AntV tools to disable. Leave empty to use the sidecar's
# default (drops basic charts that overlap create_chart, and China-only maps).
ANTV_DISABLED_TOOLS=
# Embeddings
# OpenAI Embeddings
OPENAI_API_KEY=
EMBEDDING_MODEL=text-embedding-3-small
# Chunking
RAG_CHUNK_SIZE=512
RAG_CHUNK_OVERLAP=50
# Retrieval
RAG_DEFAULT_COLLECTION=documents
RAG_TOP_K=10
# Reranker
# PDF Parser
# Google Drive (optional, for document ingestion)
# === CORS ===
# JSON list of allowed origins (default: localhost:3000, localhost:8080)
# Note: "*" is blocked in production - specify explicit origins
CORS_ORIGINS=["http://localhost:3000","http://localhost:8080"]
# === Docker Production (Traefik) ===
# Domain for production deployment
DOMAIN=example.com
# Let's Encrypt email for SSL certificates
ACME_EMAIL=admin@example.com
# Traefik dashboard auth (generate with: htpasswd -nb admin password)
# TRAEFIK_DASHBOARD_AUTH=admin:$$apr1$$...
# Redis password for production
REDIS_PASSWORD=change-me-in-production
backend/.pre-commit-config.yaml
+2
View File
@@ -0,0 +1,2 @@
# Pre-commit is disabled for this project
backend/Dockerfile
+54
View File
@@ -0,0 +1,54 @@
# Build stage
FROM python:3.13-slim AS builder
ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1
WORKDIR /app
# Install uv
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv
ENV UV_COMPILE_BYTECODE=1
ENV UV_LINK_MODE=copy
# Install dependencies
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=bind,source=uv.lock,target=uv.lock \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
uv sync --frozen --no-install-project --no-dev
# Copy application
COPY . /app
# Install project
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev
# Runtime stage
FROM python:3.13-slim
ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1
WORKDIR /app
# Copy virtual environment from builder
COPY --from=builder /app/.venv /app/.venv
COPY --from=builder /app /app
# Add venv to path
ENV PATH="/app/.venv/bin:$PATH"
# Create non-root user and writable directories
RUN adduser --disabled-password --gecos "" appuser && \
mkdir -p /app/media /app/data /app/chroma_data /app/models_cache && \
chown -R appuser:appuser /app
USER appuser
EXPOSE 8000
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD python -c "import httpx; httpx.get('http://localhost:8000/api/v1/health')" || exit 1
CMD ["python", "-m", "cli.commands", "server", "run", "--host", "0.0.0.0", "--port", "8000"]
backend/alembic.ini
+45
View File
@@ -0,0 +1,45 @@
# Alembic Configuration
[alembic]
script_location = alembic
prepend_sys_path = .
version_path_separator = os
# Human-readable migration file names: 2024-01-15_add_users_table.py
file_template = %%(year)d-%%(month).2d-%%(day).2d_%%(slug)s
[post_write_hooks]
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
qualname =
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
backend/alembic/env.py
+75
View File
@@ -0,0 +1,75 @@
"""Alembic migration environment."""
# ruff: noqa: I001 - Imports structured for Jinja2 template conditionals
from logging.config import fileConfig
from alembic import context
from sqlalchemy import engine_from_config, pool
from sqlmodel import SQLModel
from app.core.config import settings
# Import all models here to ensure they are registered with metadata
from app.db.models.user import User # noqa: F401
from app.db.models.conversation import Conversation, Message, ToolCall # noqa: F401
from app.db.models.message_rating import MessageRating # noqa: F401
from app.db.models.webhook import Webhook, WebhookDelivery # noqa: F401
from app.db.models.chat_file import ChatFile # noqa: F401
from app.db.models.rag_document import RAGDocument # noqa: F401
from app.db.models.sync_log import SyncLog # noqa: F401
from app.db.models.sync_source import SyncSource # noqa: F401
config = context.config
if config.config_file_name is not None:
fileConfig(config.config_file_name)
target_metadata = SQLModel.metadata
# Ensure SQLite data directory exists before connecting
def get_url() -> str:
"""Get database URL from settings."""
return settings.DATABASE_URL_SYNC
def run_migrations_offline() -> None:
"""Run migrations in 'offline' mode."""
url = get_url()
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online() -> None:
"""Run migrations in 'online' mode."""
configuration = config.get_section(config.config_ini_section) or {}
configuration["sqlalchemy.url"] = get_url()
connectable = engine_from_config(
configuration,
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
)
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
backend/alembic/script.py.mako
+28
View File
@@ -0,0 +1,28 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
import sqlmodel
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision: str = ${repr(up_revision)}
down_revision: Union[str, None] = ${repr(down_revision)}
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
def upgrade() -> None:
${upgrades if upgrades else "pass"}
def downgrade() -> None:
${downgrades if downgrades else "pass"}
backend/alembic/versions/.gitkeep
View File
backend/alembic/versions/0000_create_users.py
+53
View File
@@ -0,0 +1,53 @@
"""create users table
Revision ID: 0000_users
Revises:
Create Date: 2026-06-15T06:18:30.904341+00:00
Base table required by every later migration. Mirrors the current User model
including is_app_admin (later flagged in 0003 — included here so the table
is usable immediately when enable_teams=false) and onboarding_completed_at
(0016 mirror — same reason). OAuth columns are present only when an OAuth
provider was selected, keeping the schema minimal for password-only setups.
"""
import sqlalchemy as sa
from sqlalchemy.dialects.postgresql import UUID as PG_UUID
from alembic import op
revision = "0000_users"
down_revision = None
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"users",
sa.Column(
"id",
PG_UUID(as_uuid=True),
primary_key=True,
server_default=sa.text("gen_random_uuid()"),
),
sa.Column("email", sa.String(255), nullable=False, unique=True, index=True),
sa.Column("hashed_password", sa.String(255), nullable=True),
sa.Column("full_name", sa.String(255), nullable=True),
sa.Column("is_active", sa.Boolean(), nullable=False, server_default=sa.true()),
sa.Column("role", sa.String(50), nullable=False, server_default="user"),
sa.Column("is_app_admin", sa.Boolean(), nullable=False, server_default=sa.false()),
sa.Column("avatar_url", sa.String(500), nullable=True),
sa.Column("onboarding_completed_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
)
def downgrade() -> None:
op.drop_table("users")
backend/alembic/versions/0001_create_organization_tables.py
+18
View File
@@ -0,0 +1,18 @@
"""create organization tables — skipped (enable_teams=false or no SQL DB)
Revision ID: 0001_org
"""
# This migration is a no-op when enable_teams is false.
revision = "0001_org"
down_revision = "0000_users"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0002_backfill_personal_orgs.py
+17
View File
@@ -0,0 +1,17 @@
"""backfill personal orgs — skipped (enable_teams=false or no SQL DB)
Revision ID: 0002_backfill_orgs
"""
revision = "0002_backfill_orgs"
down_revision = "0001_org"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0003_add_is_app_admin.py
+17
View File
@@ -0,0 +1,17 @@
"""add is_app_admin — skipped (enable_teams=false or no SQL DB)
Revision ID: 0003_is_app_admin
"""
revision = "0003_is_app_admin"
down_revision = "0002_backfill_orgs"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0004_5_create_core_tables.py
+246
View File
@@ -0,0 +1,246 @@
"""create core tables (conversations, messages, chat_files, sessions, rag, channels)
Revision ID: 0004_5_core_tables
Revises: 0004_audit_log
Create Date: 2026-06-15T06:18:30.904341+00:00
Creates the core conversational, RAG, channel, and session tables that
later migrations modify (0005, 0006, 0007, 0009). Splits feature-flagged
groups so the schema only contains what was selected at generation time.
"""
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
from sqlalchemy.dialects.postgresql import UUID as PG_UUID
from alembic import op
revision = "0004_5_core_tables"
down_revision = "0004_audit_log"
branch_labels = None
depends_on = None
def _id_col() -> sa.Column:
return sa.Column(
"id",
PG_UUID(as_uuid=True),
primary_key=True,
server_default=sa.text("gen_random_uuid()"),
)
def _user_fk(*, nullable: bool, ondelete: str = "CASCADE") -> sa.Column:
return sa.Column(
"user_id",
PG_UUID(as_uuid=True),
sa.ForeignKey("users.id", ondelete=ondelete),
nullable=nullable,
)
_UUID = PG_UUID(as_uuid=True)
_JSONB = postgresql.JSONB(astext_type=sa.Text())
def upgrade() -> None:
op.create_table(
"conversations",
_id_col(),
_user_fk(nullable=True),
sa.Column("title", sa.String(255), nullable=True),
sa.Column("is_archived", sa.Boolean(), nullable=False, server_default=sa.false()),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_conversations_user_id", "conversations", ["user_id"])
op.create_table(
"messages",
_id_col(),
sa.Column(
"conversation_id",
_UUID,
sa.ForeignKey("conversations.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("role", sa.String(20), nullable=False),
sa.Column("content", sa.Text(), nullable=False),
sa.Column("model_name", sa.String(100), nullable=True),
sa.Column("tokens_used", sa.Integer(), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_messages_conversation_id", "messages", ["conversation_id"])
op.create_table(
"tool_calls",
_id_col(),
sa.Column(
"message_id", _UUID, sa.ForeignKey("messages.id", ondelete="CASCADE"), nullable=False
),
sa.Column("tool_call_id", sa.String(100), nullable=False),
sa.Column("tool_name", sa.String(100), nullable=False),
sa.Column("args", _JSONB, nullable=False),
sa.Column("result", sa.Text(), nullable=True),
sa.Column("status", sa.String(20), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("duration_ms", sa.Integer(), nullable=True),
)
op.create_index("ix_tool_calls_message_id", "tool_calls", ["message_id"])
op.create_table(
"chat_files",
_id_col(),
_user_fk(nullable=False),
sa.Column(
"message_id", _UUID, sa.ForeignKey("messages.id", ondelete="CASCADE"), nullable=True
),
sa.Column("filename", sa.String(255), nullable=False),
sa.Column("mime_type", sa.String(100), nullable=False),
sa.Column("size", sa.Integer(), nullable=False),
sa.Column("storage_path", sa.String(500), nullable=False),
sa.Column("file_type", sa.String(20), nullable=False),
sa.Column("parsed_content", sa.Text(), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_chat_files_user_id", "chat_files", ["user_id"])
op.create_table(
"conversation_shares",
_id_col(),
sa.Column(
"conversation_id",
_UUID,
sa.ForeignKey("conversations.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"shared_by", _UUID, sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False
),
sa.Column(
"shared_with", _UUID, sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=True
),
sa.Column("share_token", sa.String(64), nullable=True, unique=True),
sa.Column("permission", sa.String(10), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.UniqueConstraint("conversation_id", "shared_with", name="uq_share_conv_user"),
)
op.create_index(
"ix_conversation_shares_conversation_id", "conversation_shares", ["conversation_id"]
)
op.create_index("ix_conversation_shares_shared_with", "conversation_shares", ["shared_with"])
op.create_table(
"message_ratings",
_id_col(),
sa.Column(
"message_id", _UUID, sa.ForeignKey("messages.id", ondelete="CASCADE"), nullable=False
),
_user_fk(nullable=False),
sa.Column("rating", sa.Integer(), nullable=False),
sa.Column("comment", sa.Text(), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
sa.CheckConstraint("rating IN (1, -1)", name="message_ratings_ck_rating_value_check"),
sa.UniqueConstraint("message_id", "user_id", name="uq_message_user_rating"),
)
op.create_index("ix_message_ratings_message_id", "message_ratings", ["message_id"])
op.create_index("ix_message_ratings_user_id", "message_ratings", ["user_id"])
op.create_table(
"rag_documents",
_id_col(),
sa.Column("collection_name", sa.String(255), nullable=False),
sa.Column("filename", sa.String(255), nullable=False),
sa.Column("filesize", sa.Integer(), nullable=False),
sa.Column("filetype", sa.String(20), nullable=False),
sa.Column("storage_path", sa.String(500), nullable=True),
sa.Column("status", sa.String(20), nullable=False),
sa.Column("error_message", sa.Text(), nullable=True),
sa.Column("vector_document_id", sa.String(255), nullable=True),
sa.Column("chunk_count", sa.Integer(), nullable=False, server_default="0"),
sa.Column(
"started_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_rag_documents_collection_name", "rag_documents", ["collection_name"])
op.create_table(
"sync_sources",
_id_col(),
sa.Column("name", sa.String(255), nullable=False),
sa.Column("connector_type", sa.String(20), nullable=False),
sa.Column("collection_name", sa.String(255), nullable=False),
sa.Column("config", _JSONB, server_default="{}", nullable=False),
sa.Column("sync_mode", sa.String(20), nullable=False),
sa.Column("schedule_minutes", sa.Integer(), nullable=True),
sa.Column("is_active", sa.Boolean(), nullable=False, server_default=sa.true()),
sa.Column("last_sync_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("last_sync_status", sa.String(20), nullable=True),
sa.Column("last_error", sa.Text(), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_sync_sources_collection_name", "sync_sources", ["collection_name"])
op.create_table(
"sync_logs",
_id_col(),
sa.Column("source", sa.String(20), nullable=False),
sa.Column("collection_name", sa.String(255), nullable=False),
sa.Column(
"sync_source_id",
_UUID,
sa.ForeignKey("sync_sources.id", ondelete="SET NULL"),
nullable=True,
),
sa.Column("status", sa.String(20), nullable=False),
sa.Column("mode", sa.String(20), nullable=False),
sa.Column("total_files", sa.Integer(), nullable=False, server_default="0"),
sa.Column("ingested", sa.Integer(), nullable=False, server_default="0"),
sa.Column("updated", sa.Integer(), nullable=False, server_default="0"),
sa.Column("skipped", sa.Integer(), nullable=False, server_default="0"),
sa.Column("failed", sa.Integer(), nullable=False, server_default="0"),
sa.Column("error_message", sa.Text(), nullable=True),
sa.Column(
"started_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_sync_logs_collection_name", "sync_logs", ["collection_name"])
def downgrade() -> None:
op.drop_table("sync_logs")
op.drop_table("sync_sources")
op.drop_table("rag_documents")
op.drop_table("message_ratings")
op.drop_table("conversation_shares")
op.drop_table("chat_files")
op.drop_table("tool_calls")
op.drop_table("messages")
op.drop_table("conversations")
backend/alembic/versions/0004_create_audit_log.py
+17
View File
@@ -0,0 +1,17 @@
"""create audit log — skipped (enable_teams=false or no SQL DB)
Revision ID: 0004_audit_log
"""
revision = "0004_audit_log"
down_revision = "0003_is_app_admin"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0005_add_org_id_to_conversations_and_rag.py
+17
View File
@@ -0,0 +1,17 @@
"""add organization_id to conversations — skipped (enable_teams=false or no SQL DB)
Revision ID: 0005_org_tenant_isolation
"""
revision = "0005_org_tenant_isolation"
down_revision = "0004_5_core_tables"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0006_backfill_org_id_on_conversations.py
+17
View File
@@ -0,0 +1,17 @@
"""backfill organization_id on conversations — skipped (enable_teams=false or no JWT)
Revision ID: 0006_backfill_conv_org
"""
revision = "0006_backfill_conv_org"
down_revision = "0005_org_tenant_isolation"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0007_create_knowledge_bases.py
+17
View File
@@ -0,0 +1,17 @@
"""create knowledge_bases — skipped (enable_teams/enable_rag/use_jwt=false or no SQL DB)
Revision ID: 0007_knowledge_bases
"""
revision = "0007_knowledge_bases"
down_revision = "0006_backfill_conv_org"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0008_backfill_default_kbs.py
+17
View File
@@ -0,0 +1,17 @@
"""backfill default KBs — skipped (enable_teams/enable_rag/use_jwt=false or no SQL DB)
Revision ID: 0008_backfill_default_kbs
"""
revision = "0008_backfill_default_kbs"
down_revision = "0007_knowledge_bases"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0009_conv_active_kb_ids.py
+17
View File
@@ -0,0 +1,17 @@
"""add active_knowledge_base_ids to conversations — skipped (enable_teams/enable_rag/use_jwt=false or no SQL DB)
Revision ID: 0009_conv_active_kb_ids
"""
revision = "0009_conv_active_kb_ids"
down_revision = "0008_backfill_default_kbs"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0010_org_billing_seats.py
+17
View File
@@ -0,0 +1,17 @@
"""add billing seats to organizations — skipped (enable_billing/enable_teams=false or no SQL DB)
Revision ID: 0010_org_billing_seats
"""
revision = "0010_org_billing_seats"
down_revision = "0009_conv_active_kb_ids"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0011_create_plan_price_tables.py
+17
View File
@@ -0,0 +1,17 @@
"""create plan and price tables — skipped (enable_billing=false or no SQL DB)
Revision ID: 0011_create_plan_price_tables
"""
revision = "0011_create_plan_price_tables"
down_revision = "0010_org_billing_seats"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0012_create_subscription_table.py
+17
View File
@@ -0,0 +1,17 @@
"""create subscription table — skipped (enable_billing=false or no SQL DB)
Revision ID: 0012_create_subscription_table
"""
revision = "0012_create_subscription_table"
down_revision = "0011_create_plan_price_tables"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0013_create_stripe_event_table.py
+17
View File
@@ -0,0 +1,17 @@
"""create stripe_event table — skipped (enable_billing=false or no SQL DB)
Revision ID: 0013_create_stripe_event_table
"""
revision = "0013_create_stripe_event_table"
down_revision = "0012_create_subscription_table"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0014_credits_usage_events.py
+17
View File
@@ -0,0 +1,17 @@
"""create credit_transaction and usage_event — skipped (enable_billing/enable_credits_system=false or no SQL DB)
Revision ID: 0014_credits_usage_events
"""
revision = "0014_credits_usage_events"
down_revision = "0013_create_stripe_event_table"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0015_create_mv_usage_daily.py
+18
View File
@@ -0,0 +1,18 @@
"""create mv_usage_daily — skipped (PostgreSQL + billing + credits not all enabled)
Revision ID: 0015_create_mv_usage_daily
Revises: 0014_credits_usage_events
"""
revision = "0015_create_mv_usage_daily"
down_revision = "0014_credits_usage_events"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0016_user_onboarding_at.py
+44
View File
@@ -0,0 +1,44 @@
"""add onboarding_completed_at to users
Revision ID: 0016_user_onboarding_at
Revises: 0015_create_mv_usage_daily
Create Date: 2026-05-08T00:00:00+00:00
Adds:
- users.onboarding_completed_at — nullable timestamptz; null means the user
hasn't completed onboarding yet, set when they finish the wizard.
"""
import sqlalchemy as sa
from alembic import op
revision = "0016_user_onboarding_at"
down_revision = "0015_create_mv_usage_daily"
branch_labels = None
depends_on = None
def upgrade() -> None:
# Idempotent — `0000_users` already includes this column when running on a
# fresh DB. Older deployments that stamped past 0000 still need it added.
bind = op.get_bind()
inspector = sa.inspect(bind)
cols = {c["name"] for c in inspector.get_columns("users")}
if "onboarding_completed_at" not in cols:
op.add_column(
"users",
sa.Column(
"onboarding_completed_at",
sa.DateTime(timezone=True),
nullable=True,
),
)
def downgrade() -> None:
bind = op.get_bind()
inspector = sa.inspect(bind)
cols = {c["name"] for c in inspector.get_columns("users")}
if "onboarding_completed_at" in cols:
op.drop_column("users", "onboarding_completed_at")
backend/alembic/versions/0018_user_slash_commands.py
+65
View File
@@ -0,0 +1,65 @@
"""create user_slash_commands table
Revision ID: 0018_user_slash_commands
Revises: 0017_create_api_keys_table
Create Date: 2026-05-10T00:00:00+00:00
Stores per-user slash command settings for the chat palette:
- Custom commands (``prompt`` is set) — quick prompt shortcuts.
- Built-in overrides (``prompt`` is NULL) — record only ``is_enabled`` for
one of the frontend's BUILTIN_COMMANDS, so users can hide ones they don't
use.
The unique ``(user_id, name)`` constraint prevents both classes of row from
co-existing for the same name on a single user.
"""
import sqlalchemy as sa
from sqlalchemy.dialects.postgresql import UUID as PG_UUID
from alembic import op
revision = "0018_user_slash_commands"
down_revision = "0016_user_onboarding_at"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"user_slash_commands",
sa.Column(
"id",
PG_UUID(as_uuid=True),
primary_key=True,
server_default=sa.text("gen_random_uuid()"),
),
sa.Column(
"user_id",
PG_UUID(as_uuid=True),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("name", sa.String(64), nullable=False),
sa.Column("prompt", sa.Text(), nullable=True),
sa.Column(
"is_enabled",
sa.Boolean(),
nullable=False,
server_default=sa.true(),
),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
sa.UniqueConstraint("user_id", "name", name="uq_user_slash_commands_user_name"),
)
op.create_index("ix_user_slash_commands_user_id", "user_slash_commands", ["user_id"])
def downgrade() -> None:
op.drop_index("ix_user_slash_commands_user_id", table_name="user_slash_commands")
op.drop_table("user_slash_commands")
backend/app/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""agent delta project"""
__version__ = "0.1.0"
backend/app/agents/__init__.py
+10
View File
@@ -0,0 +1,10 @@
"""AI Agents module using PydanticDeep.
This module contains a deep agentic coding assistant built with pydantic-deep.
PydanticDeep is built on PydanticAI and provides filesystem operations,
task management, subagent delegation, skills, memory, and Docker sandbox support.
"""
from app.agents.pydantic_deep_assistant import PydanticDeepAssistant, PydanticDeepContext
__all__ = ["PydanticDeepAssistant", "PydanticDeepContext"]
backend/app/agents/prompts.py
+96
View File
@@ -0,0 +1,96 @@
"""System prompts for AI agents.
Centralized location for all agent prompts to make them easy to find and modify.
The default prompt follows an outcome-first style: it defines who the assistant
is, how it should behave, and how to format answers — then trusts the model to
choose a good path. Avoid re-introducing long process checklists or absolute
"ALWAYS / NEVER / EXCLUSIVELY" rules for judgment calls; they make the assistant
mechanical and, in the RAG case, cause it to wrongly refuse general questions.
"""
DEFAULT_SYSTEM_PROMPT = """You are a knowledgeable, capable AI assistant. Help the user accomplish their task or answer their question as well as you can.
# Personality
Be approachable, steady, and direct. Assume the user is competent and acting in good faith. Prefer making progress over stopping for clarification when the request is clear enough to attempt — use reasonable assumptions and state them briefly. Ask a narrow clarifying question only when the missing information would materially change the answer.
Stay concise without being curt: give enough context for the user to understand and trust the answer, then stop. Use examples or simple analogies when they make a point land. When correcting the user or disagreeing, be candid but constructive; if you are wrong, acknowledge it plainly and fix it. Match the user's tone within professional bounds, and avoid emojis and profanity unless the user clearly invites that style.
# Answering
Answer from your own broad knowledge by default. You are a general-purpose assistant, not a document-lookup bot — questions about the world, concepts, code, math, science, history, culture, writing, and everyday advice should be answered directly and helpfully.
Say you don't know only when the answer genuinely depends on private, user-specific, or very recent information you cannot access. Never refuse or hedge on a general-knowledge question just because the topic isn't in a connected data source. If a request is ambiguous, answer the most likely intent and note the assumption rather than stalling.
# Output
Let formatting serve comprehension. Default to clear plain paragraphs for explanations and discussion. Reach for headers, bullets, or numbered lists only when they genuinely make the answer easier to scan — steps, comparisons, or rankings — or when the user asks for them. Honor explicit formatting and length preferences from the user. Lead with the conclusion, then the supporting detail, then any caveats."""
DEFAULT_SYSTEM_PROMPT += """
# Maps
You can render an interactive map with the `create_map` tool. Use it whenever the
user wants to see places located geographically (cities, offices, routes, points
of interest). Supply latitude/longitude for each marker from your own knowledge
(e.g. Warsaw ≈ 52.23, 21.01; New York ≈ 40.71, -74.01). Give each marker a short
label, and an optional description. Don't repeat the JSON — briefly describe the
map you created. Each map is rendered to the user the moment you call
`create_map`; a map from an earlier turn is already on screen, so never re-create
it — only call `create_map` for the user's current request."""
# AntV diagram tools attach only at runtime (when ENABLE_ANTV_CHARTS is set), so
# their guidance is gated the same way. `create_map` above is always available.
ANTV_CHART_GUIDANCE = """
# Advanced diagrams
Beyond `create_chart`, you have AntV `generate_*` tools for diagram types the
basic chart tool can't express — flowcharts, mind maps, org charts, sankey,
fishbone, network/graph, treemap, word clouds, radar, funnel, histogram, and
more. Use them when the user asks for that specific diagram, or when the
relationship is structural (process, hierarchy, flow) rather than a plain
numeric series. Prefer `create_chart` for ordinary line/bar/pie/area/scatter.
Keep every node, label, and description short — a few words at most. Many of
these diagrams render nodes in a fixed-width box and truncate longer text with
an ellipsis ("…"), so write "Verify email", not "Send the verification email and
wait for confirmation". Put any detail in your reply, not in the node.
After the tool returns an image, briefly describe it — don't paste the URL. The
image is shown to the user immediately; a diagram from an earlier turn is already
on screen, so never regenerate it — only call these tools for the current request."""
def _antv_guidance() -> str:
"""AntV diagram guidance — included only when the MCP sidecar is enabled."""
from app.core.config import settings
return ANTV_CHART_GUIDANCE if settings.ENABLE_ANTV_CHARTS else ""
DEFAULT_SYSTEM_PROMPT += _antv_guidance()
def get_system_prompt_with_rag() -> str:
"""Get the default prompt plus knowledge-base (RAG) usage guidance.
Returns:
System prompt that treats `search_documents` as a tool to use when the
question is about the user's own documents/data — while still answering
general questions directly from the model's own knowledge.
"""
return f"""{DEFAULT_SYSTEM_PROMPT}
# Knowledge base
You have a `search_documents` tool that searches documents and data the user has added to this workspace.
When to search:
- The question is about the user's own documents, files, policies, projects, or other workspace/organization-specific information.
- The user explicitly refers to "the docs", an uploaded file, or internal information.
- A factual claim in your answer should be backed by their source material.
When NOT to search: general knowledge, common concepts, code, math, definitions, or anything you can already answer well. Do not search just to check whether something happens to be in the knowledge base, and never tell the user a topic "isn't in the knowledge base" when it is a question you can simply answer yourself.
Retrieval budget: start with one focused search using short, distinctive keywords. Search again only if the results miss the core question, a needed fact/figure/owner/date/source is missing, or the user asked for comprehensive coverage or a comparison. Don't search again merely to rephrase or pad the answer.
Citations: when you use retrieved documents, attach numbered references like [1], [2] to the specific claims they support, and list those sources at the end (filename, plus page if available). Cite only sources that appear in the search results — never fabricate citations, filenames, or page numbers.
Missing evidence is not automatically a "no". If the documents don't cover the question, say briefly what you couldn't find, then still help: answer from general knowledge where that's appropriate (and note that you're doing so), or ask for the specific document or detail you'd need."""
backend/app/agents/pydantic_deep_assistant.py
+413
View File
@@ -0,0 +1,413 @@
"""PydanticDeep assistant — deep agentic coding assistant.
PydanticDeep is built on PydanticAI and provides:
- Filesystem tools: ls, read_file, write_file, edit_file, glob, grep
- Task management: write_todos, planning subagent
- Subagent delegation
- Skills system (SKILL.md files)
- Memory persistence (MEMORY.md across sessions)
- Context file discovery (AGENTS.md, SOUL.md from workspace root)
- Built-in web search and web fetch
Backend types (set via PYDANTIC_DEEP_BACKEND_TYPE):
"state" — In-memory (default, no persistence)
"daytona" — Daytona cloud workspace (isolated, cloud-native)
Configuration via settings:
PYDANTIC_DEEP_BACKEND_TYPE : "state" | "daytona" (default: "state")
PYDANTIC_DEEP_INCLUDE_SUBAGENTS : enable subagent delegation (default: True)
PYDANTIC_DEEP_INCLUDE_SKILLS : enable skills system (default: True)
PYDANTIC_DEEP_INCLUDE_PLAN : enable planner subagent (default: True)
PYDANTIC_DEEP_INCLUDE_MEMORY : enable persistent MEMORY.md (default: True)
PYDANTIC_DEEP_INCLUDE_EXECUTE : enable shell execution (default: False)
PYDANTIC_DEEP_WEB_SEARCH : enable built-in web search (default: True)
"""
import inspect
import logging
from typing import Any, TypedDict
from pydantic_ai import Agent
from pydantic_ai_backends import BackendProtocol, StateBackend
from pydantic_deep import DeepAgentDeps, create_deep_agent
from app.agents.prompts import get_system_prompt_with_rag
from app.agents.tools.antv_chart import get_antv_toolset
from app.agents.tools.map_tool import MapMarker, create_map
from app.agents.tools.rag_tool import search_knowledge_base
from app.core.config import settings
logger = logging.getLogger(__name__)
# Map LLM_PROVIDER settings value → pydantic-ai provider prefix
_PROVIDER_PREFIXES: dict[str, str] = {
"openai": "openai",
"anthropic": "anthropic",
"google": "google-gla", # Google AI (Gemini) via GOOGLE_API_KEY
"openrouter": "openrouter",
}
class PydanticDeepContext(TypedDict, total=False):
"""Runtime context passed through the agent run."""
user_id: str | None
user_name: str | None
metadata: dict[str, Any]
async def _rag_search(query: str, top_k: int = 5) -> str:
"""Search the knowledge base for relevant documents.
Use this tool to find information from uploaded documents before answering
user queries. Searches across all available collections automatically.
Cite sources by referring to the document filename from the search results.
Args:
query: The search query string.
top_k: Number of top results to retrieve (default: 5).
Returns:
Formatted string with search results including content and scores.
"""
return await search_knowledge_base(query=query, top_k=top_k)
def create_map_tool(
title: str,
markers: list[MapMarker],
center: list[float] | None = None,
zoom: int | None = None,
) -> str:
"""Create an interactive map to show places geographically for the user.
Use whenever the user asks to show, map, or locate places. Provide
latitude/longitude for each marker from your own knowledge (e.g. Warsaw ≈
52.23, 21.01). Do not repeat the returned JSON — just briefly describe the
map you created.
Args:
title: Short map title.
markers: One entry per place, each with lat, lng and a short label
(plus optional description and color). Must not be empty.
center: Optional [lat, lng] center (auto-fit to markers if omitted).
zoom: Optional zoom level 1-18 (mainly useful for a single marker).
"""
return create_map(
title=title,
markers=[m.model_dump() for m in markers],
center=center,
zoom=zoom,
)
class PydanticDeepAssistant:
"""Deep agentic assistant powered by pydantic-deep.
Wraps create_deep_agent() with web-app friendly defaults:
- StateBackend by default (in-memory, ephemeral)
- Optional DaytonaSandbox for cloud-native workspaces
- Conversation scoped by conversation_id (history_messages_path)
- Non-interactive mode (no human-in-the-loop approval prompts)
"""
def __init__(
self,
model_name: str | None = None,
thinking_effort: str | None = None,
conversation_id: str = "default",
user_id: str | None = None,
user_name: str | None = None,
backend_override: Any = None,
history_messages_path: str | None = None,
):
self.model_name = model_name or settings.AI_MODEL
self.thinking_effort = thinking_effort
self.conversation_id = conversation_id
self.user_id = user_id
self.user_name = user_name
# Allow project-scoped WS endpoint to inject a pre-built backend/history path
self._backend_override = backend_override
self._history_messages_path = history_messages_path
self._agent: Agent | None = None
self._deps: DeepAgentDeps | None = None
def _get_model_string(self) -> str:
"""Build pydantic-ai model string with provider prefix.
If the model name already contains ':', returns it unchanged.
Otherwise prepends the provider prefix from LLM_PROVIDER setting.
Examples:
"gpt-5.5" → "openai:gpt-5.5"
"claude-opus-4-7" → "anthropic:claude-opus-4-7"
"gemini-2.5-flash" → "google-gla:gemini-2.5-flash"
"openai:gpt-5.5" → "openai:gpt-5.5" (unchanged)
"""
if ":" in self.model_name:
return self.model_name
prefix = _PROVIDER_PREFIXES.get(settings.LLM_PROVIDER, settings.LLM_PROVIDER)
return f"{prefix}:{self.model_name}"
def _create_backend(self) -> BackendProtocol:
"""Create the file-storage backend based on PYDANTIC_DEEP_BACKEND_TYPE."""
backend_type = settings.PYDANTIC_DEEP_BACKEND_TYPE
if backend_type == "daytona":
try:
from pydantic_ai_backends import DaytonaSandbox
return DaytonaSandbox(
workspace_id=f"pd-{self.conversation_id}",
)
except ImportError:
logger.warning(
"Daytona backend not available — "
"install 'pydantic-ai-backend[daytona]'. Falling back to StateBackend."
)
return StateBackend()
# Default: in-memory, no cross-connection persistence
return StateBackend()
def _get_system_prompt(self) -> str:
"""Return the base system prompt."""
return get_system_prompt_with_rag()
def _build_agent_and_deps(self) -> tuple[Agent[DeepAgentDeps, str], DeepAgentDeps]:
"""Instantiate the pydantic-deep agent and its dependencies."""
backend = (
self._backend_override if self._backend_override is not None else self._create_backend()
)
model_str = self._get_model_string()
history_path = (
self._history_messages_path
if self._history_messages_path is not None
else f".pydantic-deep/sessions/{self.conversation_id}/messages.json"
)
logger.info(
"Creating PydanticDeep agent — model=%s backend=%s conversation=%s",
model_str,
type(backend).__name__,
self.conversation_id,
)
# Extra tools exposed as standalone pydantic-ai Tools.
# pydantic-deep merges them into the agent automatically.
from pydantic_ai import Tool as PAITool
extra_tools: list[Any] = []
extra_tools.append(PAITool(_rag_search))
extra_tools.append(PAITool(create_map_tool))
# Attach the AntV toolset only if this create_deep_agent build accepts a
# `toolsets` kwarg — guards against versions that don't expose it.
antv_toolset = get_antv_toolset()
antv_kwargs: dict[str, Any] = {}
if (
antv_toolset is not None
and "toolsets" in inspect.signature(create_deep_agent).parameters
):
antv_kwargs["toolsets"] = [antv_toolset]
agent = create_deep_agent(
model=model_str,
backend=backend,
instructions=self._get_system_prompt(),
# Per-conversation history persistence
history_messages_path=history_path,
# Built-in tool groups
include_filesystem=True,
include_execute=settings.PYDANTIC_DEEP_INCLUDE_EXECUTE,
include_todo=True,
include_subagents=settings.PYDANTIC_DEEP_INCLUDE_SUBAGENTS,
include_builtin_subagents=settings.PYDANTIC_DEEP_INCLUDE_SUBAGENTS,
include_skills=settings.PYDANTIC_DEEP_INCLUDE_SKILLS,
include_plan=settings.PYDANTIC_DEEP_INCLUDE_PLAN,
# Memory: shared MEMORY.md across all conversations in the workspace
include_memory=settings.PYDANTIC_DEEP_INCLUDE_MEMORY,
memory_dir=".pydantic-deep",
# Context discovery: reads AGENTS.md, SOUL.md from workspace root
context_discovery=True,
# Web tools (built-in pydantic-ai WebSearch / WebFetch)
web_search=settings.PYDANTIC_DEEP_WEB_SEARCH,
web_fetch=True,
# Context management: auto-compress history at token budget
context_manager=True,
# Cost tracking
cost_tracking=True,
# Thinking effort — "auto" lets pydantic-ai resolve per-provider
thinking=self.thinking_effort or "auto",
# Extra tools (e.g. RAG search)
**({"tools": extra_tools} if extra_tools else {}),
**antv_kwargs,
)
deps = DeepAgentDeps(backend=backend)
return agent, deps
# Workspace file upload (Docker / Daytona only)
async def write_file_to_workspace(self, rel_path: str, content: bytes | str) -> bool:
"""Write a file into the sandbox workspace (Daytona).
For StateBackend (in-memory) this is a no-op — callers should fall back
to including file content inline in the user message.
Args:
rel_path: Workspace-relative path, e.g. ``uploads/report.pdf``.
content: File content (bytes or str).
Returns:
True if the file was written to the sandbox filesystem.
"""
backend = self.deps.backend
data = content if isinstance(content, bytes) else content.encode("utf-8")
# -- Generic: backend exposes upload_bytes / write_file method ------
upload_fn = getattr(backend, "upload_bytes", None) or getattr(backend, "write_file", None)
if upload_fn is not None:
try:
await upload_fn(rel_path, data)
return True
except Exception as exc:
logger.warning("Failed to write %s via backend API: %s", rel_path, exc)
return False
# -- StateBackend: no real filesystem --------------------------------
return False
# Properties: lazy creation of agent + deps on first access
@property
def agent(self) -> Agent:
"""Get or create the underlying pydantic-ai Agent."""
if self._agent is None:
self._agent, self._deps = self._build_agent_and_deps()
return self._agent
@property
def deps(self) -> DeepAgentDeps:
"""Get or create the DeepAgentDeps for this assistant."""
if self._deps is None:
self._agent, self._deps = self._build_agent_and_deps()
return self._deps
# Run / stream
async def run(
self,
user_input: str,
history: list[dict[str, str]] | None = None,
context: PydanticDeepContext | None = None,
) -> tuple[str, list[Any], PydanticDeepContext]:
"""Run the agent and return the full response.
Note: pydantic-deep manages conversation history internally via the
backend (history_messages_path). The ``history`` parameter is accepted
for API parity with other agent wrappers but is not used.
Args:
user_input: User's message.
history: Ignored — pydantic-deep persists history internally.
context: Optional runtime context (user_id, user_name, metadata).
Returns:
Tuple of (output_text, tool_events, context).
"""
agent_context: PydanticDeepContext = context if context is not None else {}
logger.info("Running PydanticDeep agent: %s…", user_input[:100])
result = await self.agent.run(user_input, deps=self.deps)
tool_events: list[Any] = []
for message in result.all_messages():
if hasattr(message, "parts"):
for part in message.parts:
if hasattr(part, "tool_name"):
tool_events.append(part)
logger.info("PydanticDeep run complete. Output: %d chars", len(result.output))
return result.output, tool_events, agent_context
async def stream(
self,
user_input: str,
history: list[dict[str, str]] | None = None,
context: PydanticDeepContext | None = None,
):
"""Stream agent execution token by token.
Note: pydantic-deep manages conversation history internally via the
backend (history_messages_path). The ``history`` parameter is accepted
for API parity with other agent wrappers but is not used.
Args:
user_input: User's message.
history: Ignored — pydantic-deep persists history internally.
context: Optional runtime context.
Yields:
Tuples of ("messages", (chunk, metadata)) for LLM tokens.
"""
logger.info("Streaming PydanticDeep agent: %s…", user_input[:100])
async with self.agent.run_stream(user_input, deps=self.deps) as response:
async for text in response.stream_text(delta=True):
yield "messages", (text, {})
# Factory helpers
def get_agent(
model_name: str | None = None,
thinking_effort: str | None = None,
conversation_id: str = "default",
user_id: str | None = None,
user_name: str | None = None,
backend_override: Any = None,
history_messages_path: str | None = None,
) -> PydanticDeepAssistant:
"""Create a PydanticDeepAssistant instance.
Args:
model_name: Override AI_MODEL from settings.
thinking_effort: Extended thinking effort level (e.g. "low", "medium", "high").
conversation_id: Scope history to this conversation (default: "default").
user_id: Optional user identifier for context.
user_name: Optional user display name for context.
backend_override: Pre-built backend (e.g. DaytonaSandbox for a project).
When provided, bypasses _create_backend() entirely.
history_messages_path: Override the history file path inside the backend.
Useful for project-scoped chats where each chat has its own path.
Returns:
Configured PydanticDeepAssistant.
"""
return PydanticDeepAssistant(
model_name=model_name,
thinking_effort=thinking_effort,
conversation_id=conversation_id,
user_id=user_id,
user_name=user_name,
backend_override=backend_override,
history_messages_path=history_messages_path,
)
async def run_agent(
user_input: str,
conversation_id: str = "default",
context: PydanticDeepContext | None = None,
) -> tuple[str, list[Any], PydanticDeepContext]:
"""One-shot convenience wrapper for running the agent.
Args:
user_input: User's message.
conversation_id: Conversation scope for history.
context: Optional runtime context.
Returns:
Tuple of (output_text, tool_events, context).
"""
assistant = get_agent(conversation_id=conversation_id)
return await assistant.run(user_input, context)
backend/app/agents/tools/__init__.py
+15
View File
@@ -0,0 +1,15 @@
"""Agent tools module.
This module contains utility functions that can be used as agent tools.
Tools are registered in the agent definition using @agent.tool decorator.
"""
from app.agents.tools.datetime_tool import get_current_datetime
from app.agents.tools.map_tool import create_map
from app.agents.tools.rag_tool import search_knowledge_base, search_knowledge_base_sync
from app.agents.tools.web_search import parse_web_search, web_search, web_search_sync
__all__ = ["get_current_datetime"]
__all__ += ["parse_web_search", "web_search", "web_search_sync"]
__all__ += ["search_knowledge_base", "search_knowledge_base_sync"]
__all__ += ["create_map"]
backend/app/agents/tools/antv_chart.py
+35
View File
@@ -0,0 +1,35 @@
"""AntV chart MCP server integration.
The AntV ``mcp-server-chart`` sidecar exposes advanced diagram tools (flowchart,
mind-map, org-chart, sankey, fishbone, network, treemap, radar, funnel, ...)
over MCP (streamable HTTP). Each AI framework attaches MCP differently, so this
module exposes one loader per framework family.
All loaders no-op (return ``None`` / ``[]``) when ``ENABLE_ANTV_CHARTS`` is
false, and degrade gracefully (log a warning + return empty) if the sidecar or
adapter is unavailable — so the agent always starts, with or without AntV.
"""
import logging
from typing import Any
from app.core.config import settings
logger = logging.getLogger(__name__)
def get_antv_toolset() -> Any | None:
"""Return a PydanticAI MCP toolset for the AntV server, or None if disabled.
PydanticAI connects lazily when the agent runs, so nothing async happens
here — the toolset is simply handed to ``Agent(toolsets=[...])``.
"""
if not settings.ENABLE_ANTV_CHARTS:
return None
try:
from pydantic_ai.mcp import MCPServerStreamableHTTP
return MCPServerStreamableHTTP(settings.ANTV_MCP_URL)
except Exception as exc:
logger.warning("AntV MCP toolset unavailable, continuing without it: %s", exc)
return None
backend/app/agents/tools/datetime_tool.py
+13
View File
@@ -0,0 +1,13 @@
"""Date and time utilities for agents."""
from datetime import UTC, datetime
def get_current_datetime() -> dict[str, str]:
"""Get the current date and time (UTC)."""
now = datetime.now(UTC)
return {
"date": now.strftime("%Y-%m-%d"),
"time": now.strftime("%H:%M:%S"),
"datetime": now.strftime("%Y-%m-%d %H:%M:%S"),
}
backend/app/agents/tools/map_tool.py
+100
View File
@@ -0,0 +1,100 @@
"""Map-generation tool for agents.
Mirrors ``chart_tool``: the agent calls ``create_map`` with marker coordinates
and the tool returns a structured ``MapSpec`` as a JSON string. The web frontend
parses it and renders an interactive map with Leaflet + OpenStreetMap tiles
(global coverage, no API key). The model supplies latitude/longitude for known
places directly from its own knowledge.
"""
from typing import Any
from pydantic import BaseModel, Field, ValidationError, field_validator
MAX_MARKERS = 100
class MapMarker(BaseModel):
"""A single point on the map."""
lat: float = Field(ge=-90, le=90, description="Latitude in decimal degrees.")
lng: float = Field(ge=-180, le=180, description="Longitude in decimal degrees.")
label: str = Field(max_length=120, description="Short marker label.")
description: str | None = Field(
default=None, max_length=300, description="Optional detail shown on click."
)
color: str | None = Field(default=None, description="Hex color override, e.g. '#ef4444'.")
class MapSpec(BaseModel):
"""Canonical map payload produced by the tool and rendered by the frontend."""
kind: str = Field(default="map")
title: str = Field(max_length=200)
markers: list[MapMarker]
center: list[float] | None = Field(
default=None, description="Optional [lat, lng] center; auto-fit to markers if omitted."
)
zoom: int | None = Field(default=None, ge=1, le=18, description="Optional zoom (1-18).")
@field_validator("markers")
@classmethod
def _validate_markers(cls, v: list[MapMarker]) -> list[MapMarker]:
"""Require at least one marker and cap the total."""
if not v:
raise ValueError("markers must contain at least one point")
if len(v) > MAX_MARKERS:
raise ValueError(f"too many markers (max {MAX_MARKERS})")
return v
@field_validator("center")
@classmethod
def _validate_center(cls, v: list[float] | None) -> list[float] | None:
"""Require center, when given, to be an in-range [lat, lng] pair."""
if v is None:
return v
if len(v) != 2:
raise ValueError("center must be [lat, lng]")
lat, lng = v
if not (-90 <= lat <= 90) or not (-180 <= lng <= 180):
raise ValueError("center out of range (lat must be -90..90, lng -180..180)")
return v
def create_map(
title: str,
markers: list[dict[str, Any]],
center: list[float] | None = None,
zoom: int | None = None,
) -> str:
"""Create an interactive map for the user.
Use this whenever the user asks to show, map, or locate places
geographically. Provide latitude/longitude for each marker from your own
knowledge (e.g. Warsaw ≈ 52.23, 21.01). The map renders interactively in the
web chat with OpenStreetMap tiles.
Args:
title: Short title shown above the map.
markers: List of {"lat", "lng", "label", "description"?, "color"?}.
center: Optional [lat, lng] center. If omitted, the map auto-fits to the
markers.
zoom: Optional zoom level (1-18). Mainly useful with a single marker.
Returns:
A JSON string with the map specification. Do not repeat this JSON back to
the user — just briefly describe the map you created.
"""
try:
spec = MapSpec(
title=title,
markers=[MapMarker(**m) for m in markers],
center=center,
zoom=zoom,
)
except ValidationError as e:
return f"Could not build map — invalid arguments: {e.errors()}"
except (TypeError, ValueError) as e:
return f"Could not build map: {e}"
return spec.model_dump_json()
backend/app/agents/tools/rag_tool.py
+145
View File
@@ -0,0 +1,145 @@
"""RAG tool for agent knowledge base search."""
import asyncio
import logging
from concurrent.futures import ThreadPoolExecutor
from typing import TYPE_CHECKING
logger = logging.getLogger(__name__)
if TYPE_CHECKING:
from app.services.rag.retrieval import BaseRetrievalService
_retrieval_service: "BaseRetrievalService | None" = None
def _get_retrieval_service() -> "BaseRetrievalService":
"""Get or create retrieval service singleton."""
global _retrieval_service
if _retrieval_service is not None:
return _retrieval_service
from app.core.config import settings
from app.services.rag.embeddings import EmbeddingService
from app.services.rag.retrieval import RetrievalService
from app.services.rag.vectorstore import QdrantVectorStore
rag_settings = settings.rag
embedding_service = EmbeddingService(rag_settings)
vector_store = QdrantVectorStore(rag_settings, embedding_service)
_retrieval_service = RetrievalService(vector_store, rag_settings)
return _retrieval_service
def get_retrieval_service() -> "BaseRetrievalService":
"""Get the RetrievalService singleton."""
return _get_retrieval_service()
def _format_results(results: list) -> str:
if not results:
return "No relevant documents found in the knowledge base."
formatted = []
for i, result in enumerate(results, start=1):
source = result.metadata.get("filename", "unknown")
page = result.metadata.get("page_num", "")
chunk = result.metadata.get("chunk_num", "")
col = result.metadata.get("collection", "")
page_info = f", page {page}" if page else ""
chunk_info = f", chunk {chunk}" if chunk else ""
col_info = f" [{col}]" if col else ""
formatted.append(
f"[{i}] Source: {source}{page_info}{chunk_info}{col_info} (score: {result.score:.3f})\n"
f"{result.content}"
)
return "Search results (cite sources using [1], [2], etc. in your response):\n\n" + "\n\n".join(
formatted
)
async def search_knowledge_base(
query: str,
collection: str | None = None,
collections: list[str] | None = None,
top_k: int = 5,
) -> str:
"""Search the knowledge base and return formatted results.
Args:
query: The search query string.
collection: Name of a single collection. If None, uses RAG_DEFAULT_COLLECTION env var.
collections: List of collection names for cross-collection search (overrides collection).
top_k: Number of top results to retrieve (default: 5).
"""
import os
from typing import Any
service: Any = get_retrieval_service()
default_collection = os.environ.get("RAG_DEFAULT_COLLECTION", "all")
target_collection = collection or default_collection
if collections and len(collections) > 1:
results = await service.retrieve_multi(
query=query,
collection_names=collections,
limit=top_k,
)
elif target_collection == "all":
try:
all_collections = await service.store.list_collections()
if not all_collections:
return "No collections found in the knowledge base."
if len(all_collections) == 1:
results = await service.retrieve(
query=query, collection_name=all_collections[0], limit=top_k
)
else:
results = await service.retrieve_multi(
query=query, collection_names=all_collections, limit=top_k
)
except Exception as e:
logger.error(f"Failed to list collections: {e}")
return f"Error accessing knowledge base: {e}"
else:
results = await service.retrieve(
query=query,
collection_name=target_collection,
limit=top_k,
)
return _format_results(results)
def _run_async_search(query: str, collection: str | None, top_k: int) -> str:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
return loop.run_until_complete(search_knowledge_base(query, collection, top_k=top_k))
finally:
loop.close()
def search_knowledge_base_sync(
query: str,
collection: str | None = None,
top_k: int = 5,
) -> str:
"""Synchronous wrapper for search_knowledge_base. Use in CrewAI agents."""
logger.debug(
"search_knowledge_base_sync called: query=%s, collection=%s, top_k=%s",
query,
collection,
top_k,
)
try:
with ThreadPoolExecutor(max_workers=1) as executor:
future = executor.submit(_run_async_search, query, collection, top_k)
result = future.result()
logger.debug("search_knowledge_base_sync completed successfully")
return result
except Exception as e:
logger.error("search_knowledge_base_sync failed: %s", str(e), exc_info=True)
raise
__all__ = ["search_knowledge_base", "search_knowledge_base_sync"]
backend/app/agents/tools/web_search.py
+114
View File
@@ -0,0 +1,114 @@
"""Web search tool using Tavily API.
Provides AI agents with the ability to search the web for current information.
Requires TAVILY_API_KEY environment variable.
Get your API key at: https://tavily.com
The tool returns a JSON ``WebSearchResults`` payload (not prose) so the web
frontend can render results richly — clickable titles, domains, snippets —
the same way the chart tool returns a structured spec. Modern LLMs read JSON
tool results fine and can still cite by title/URL.
"""
import json
from typing import Literal
from pydantic import BaseModel, Field, ValidationError
from app.core.config import settings
class WebSearchResult(BaseModel):
"""A single web search hit."""
title: str
url: str
content: str
score: float | None = None
class WebSearchResults(BaseModel):
"""Structured web search payload returned by the ``web_search`` tool."""
kind: Literal["web_search"] = "web_search"
query: str
results: list[WebSearchResult] = Field(default_factory=list)
async def web_search(
query: str,
max_results: int = 5,
search_depth: str = "basic",
) -> str:
"""Search the web for current information using Tavily.
Args:
query: The search query string.
max_results: Maximum number of results to return (1-10, default: 5).
search_depth: Search depth - "basic" (fast) or "advanced" (thorough).
Returns:
A JSON string with the structured results (a ``WebSearchResults``
payload). Use the titles/URLs to cite sources in your answer; do not
repeat the raw JSON back to the user.
"""
try:
from tavily import AsyncTavilyClient
except ImportError:
return "Web search unavailable: the 'tavily' package is not installed."
try:
client = AsyncTavilyClient(api_key=settings.TAVILY_API_KEY)
response = await client.search(
query=query,
max_results=min(max_results, 10),
search_depth=search_depth,
)
except Exception as e:
# Surface a usable message to the agent instead of crashing the turn.
return f"Web search failed: {e}"
results = [
WebSearchResult(
title=r.get("title") or "Untitled",
url=r.get("url") or "",
content=(r.get("content") or "")[:1000],
score=r.get("score"),
)
for r in response.get("results", [])
]
return WebSearchResults(query=query, results=results).model_dump_json()
def web_search_sync(
query: str,
max_results: int = 5,
search_depth: str = "basic",
) -> str:
"""Synchronous wrapper for web_search (for CrewAI/LangChain sync tools)."""
import asyncio
loop = asyncio.new_event_loop()
try:
return loop.run_until_complete(web_search(query, max_results, search_depth))
finally:
loop.close()
def parse_web_search(result: str) -> WebSearchResults | None:
"""Parse a ``web_search`` tool result back into a model.
Returns None when the result is an error/plain string rather than a
structured payload (the frontend/channel layers fall back to plain text).
"""
try:
payload = json.loads(result)
except (json.JSONDecodeError, TypeError):
return None
if not isinstance(payload, dict) or payload.get("kind") != "web_search":
return None
try:
return WebSearchResults.model_validate(payload)
except ValidationError:
return None
backend/app/api/deps.py
+451
View File
@@ -0,0 +1,451 @@
"""API dependencies.
Dependency injection factories for services, repositories, and authentication.
"""
# ruff: noqa: I001 - Imports structured for Jinja2 template conditionals
from typing import Annotated
from fastapi import Depends
from fastapi.security import OAuth2PasswordBearer
from app.core.config import settings
from app.db.session import get_db_session
from sqlalchemy.ext.asyncio import AsyncSession
DBSession = Annotated[AsyncSession, Depends(get_db_session)]
from fastapi import Request
from app.clients.redis import RedisClient
async def get_redis(request: Request) -> RedisClient:
"""Get Redis client from lifespan state."""
return request.state.redis # type: ignore[no-any-return]
Redis = Annotated[RedisClient, Depends(get_redis)]
# === Service Dependencies ===
from app.services.user import UserService
from app.services.webhook import WebhookService
from app.services.conversation import ConversationService
def get_user_service(db: DBSession) -> UserService:
"""Create UserService instance with database session."""
return UserService(db)
UserSvc = Annotated[UserService, Depends(get_user_service)]
def get_webhook_service(db: DBSession) -> WebhookService:
"""Create WebhookService instance with database session."""
return WebhookService(db)
WebhookSvc = Annotated[WebhookService, Depends(get_webhook_service)]
def get_conversation_service(db: DBSession) -> ConversationService:
"""Create ConversationService instance with database session."""
return ConversationService(db)
ConversationSvc = Annotated[ConversationService, Depends(get_conversation_service)]
from app.services.conversation_share import ConversationShareService
def get_conversation_share_service(db: DBSession) -> ConversationShareService:
"""Create ConversationShareService instance with database session."""
return ConversationShareService(db)
ConversationShareSvc = Annotated[ConversationShareService, Depends(get_conversation_share_service)]
from app.services.project import ProjectService
def get_project_service(db: DBSession) -> ProjectService:
"""Create ProjectService instance with database session."""
return ProjectService(db)
ProjectSvc = Annotated[ProjectService, Depends(get_project_service)]
# Message rating service
from app.services.message_rating import MessageRatingService
def get_rating_service(db: DBSession) -> MessageRatingService:
"""Create MessageRatingService instance with database session."""
return MessageRatingService(db)
MessageRatingSvc = Annotated[MessageRatingService, Depends(get_rating_service)]
from app.services.rag_document import RAGDocumentService
from app.services.rag_sync import RAGSyncService
from app.services.sync_source import SyncSourceService
def get_rag_document_service(db: DBSession) -> RAGDocumentService:
"""Create RAGDocumentService instance with database session."""
return RAGDocumentService(db)
def get_rag_sync_service(db: DBSession) -> RAGSyncService:
"""Create RAGSyncService instance with database session."""
return RAGSyncService(db)
def get_sync_source_service(db: DBSession) -> SyncSourceService:
"""Create SyncSourceService instance with database session."""
return SyncSourceService(db)
RAGDocumentSvc = Annotated[RAGDocumentService, Depends(get_rag_document_service)]
RAGSyncSvc = Annotated[RAGSyncService, Depends(get_rag_sync_service)]
SyncSourceSvc = Annotated[SyncSourceService, Depends(get_sync_source_service)]
from app.services.rag_status import RAGStatusService
def get_rag_status_service() -> RAGStatusService:
"""Create RAGStatusService instance (no DB)."""
return RAGStatusService()
RAGStatusSvc = Annotated[RAGStatusService, Depends(get_rag_status_service)]
from app.services.file_upload import FileUploadService
def get_file_upload_service(db: DBSession) -> FileUploadService:
"""Create FileUploadService instance with database session."""
return FileUploadService(db)
FileUploadSvc = Annotated[FileUploadService, Depends(get_file_upload_service)]
# === Authentication Dependencies ===
from app.core.exceptions import AuthenticationError, AuthorizationError
from app.db.models.user import User, UserRole
oauth2_scheme = OAuth2PasswordBearer(tokenUrl=f"{settings.API_V1_STR}/auth/login")
async def get_current_user(
token: Annotated[str, Depends(oauth2_scheme)],
user_service: UserSvc,
) -> User:
"""Get current authenticated user from JWT token.
Returns the full User object including role information.
Raises:
AuthenticationError: If token is invalid or user not found.
"""
from uuid import UUID
from app.core.security import verify_token
payload = verify_token(token)
if payload is None:
raise AuthenticationError(message="Invalid or expired token")
# Ensure this is an access token, not a refresh token
if payload.get("type") != "access":
raise AuthenticationError(message="Invalid token type")
user_id = payload.get("sub")
if user_id is None:
raise AuthenticationError(message="Invalid token payload")
user = await user_service.get_by_id(UUID(user_id))
if not user.is_active:
raise AuthenticationError(message="User account is disabled")
return user
class RoleChecker:
"""Dependency class for role-based access control.
Usage:
# Require admin role
@router.get("/admin-only")
async def admin_endpoint(
user: Annotated[User, Depends(RoleChecker(UserRole.ADMIN))]
):
...
# Require any authenticated user
@router.get("/users")
async def users_endpoint(
user: Annotated[User, Depends(get_current_user)]
):
...
"""
def __init__(self, required_role: UserRole) -> None:
self.required_role = required_role
async def __call__(
self,
user: Annotated[User, Depends(get_current_user)],
) -> User:
"""Check if user has the required role.
Raises:
AuthorizationError: If user doesn't have the required role.
"""
if not user.has_role(self.required_role):
raise AuthorizationError(
message=f"Role '{self.required_role.value}' required for this action"
)
return user
async def get_current_active_superuser(
current_user: Annotated[User, Depends(get_current_user)],
) -> User:
"""Get current user and verify they are a superuser.
Raises:
AuthorizationError: If user is not a superuser.
"""
if not current_user.has_role(UserRole.ADMIN):
raise AuthorizationError(message="Admin privileges required")
return current_user
# Type aliases for dependency injection
CurrentUser = Annotated[User, Depends(get_current_user)]
CurrentSuperuser = Annotated[User, Depends(get_current_active_superuser)]
CurrentAdmin = Annotated[User, Depends(RoleChecker(UserRole.ADMIN))]
# is_app_admin is a global flag on the User model — independent of team
# membership. Routes guarded by this dep (e.g. /admin/users) stay reachable
# even when teams are disabled, so the dep itself must not be gated.
async def _require_app_admin(user: CurrentUser) -> "User": # type: ignore[name-defined]
"""Raises 403 unless the user has the is_app_admin flag set."""
if not getattr(user, "is_app_admin", False):
raise AuthorizationError(message="App admin privileges required")
return user
CurrentAppAdmin = Annotated["User", Depends(_require_app_admin)] # type: ignore[valid-type]
# WebSocket authentication dependency
from fastapi import WebSocket, Cookie
_WS_TOKEN_PROTOCOL_PREFIX = "access_token."
def _extract_ws_auth(websocket: WebSocket) -> tuple[str | None, str | None]:
"""Parse Sec-WebSocket-Protocol header for an auth token + app subprotocol.
Clients pass the token as a subprotocol of the form
``access_token.<JWT>`` alongside an optional application subprotocol
(e.g. ``chat``). Returns (token, app_subprotocol) — either may be None.
"""
raw = websocket.headers.get("sec-websocket-protocol") or ""
token: str | None = None
app_subprotocol: str | None = None
for proto in (p.strip() for p in raw.split(",") if p.strip()):
if proto.startswith(_WS_TOKEN_PROTOCOL_PREFIX):
token = proto[len(_WS_TOKEN_PROTOCOL_PREFIX) :]
elif app_subprotocol is None:
app_subprotocol = proto
return token, app_subprotocol
async def get_current_user_ws(
websocket: WebSocket,
access_token: str | None = Cookie(None),
) -> User:
"""Authenticate a WebSocket connection.
Token sources, checked in order:
1. ``Sec-WebSocket-Protocol`` header, in the form ``access_token.<JWT>``.
The chosen application subprotocol (e.g. ``chat``) is echoed back on
``accept()`` via ``websocket.state.accept_subprotocol``.
2. Same-origin ``access_token`` cookie (fallback for same-origin clients).
Tokens in query strings are NOT accepted — they leak into logs and
Referer headers.
Raises:
AuthenticationError: If token is invalid or user not found.
"""
from uuid import UUID
from app.core.security import verify_token
subprotocol_token, app_subprotocol = _extract_ws_auth(websocket)
websocket.state.accept_subprotocol = app_subprotocol
auth_token = subprotocol_token or access_token
if not auth_token:
await websocket.close(code=4001, reason="Missing authentication token")
raise AuthenticationError(message="Missing authentication token")
payload = verify_token(auth_token)
if payload is None:
await websocket.close(code=4001, reason="Invalid or expired token")
raise AuthenticationError(message="Invalid or expired token")
if payload.get("type") != "access":
await websocket.close(code=4001, reason="Invalid token type")
raise AuthenticationError(message="Invalid token type")
user_id = payload.get("sub")
if user_id is None:
await websocket.close(code=4001, reason="Invalid token payload")
raise AuthenticationError(message="Invalid token payload")
from app.db.session import get_db_context
async with get_db_context() as db:
user_service = UserService(db)
user = await user_service.get_by_id(UUID(user_id))
if not user.is_active:
await websocket.close(code=4001, reason="User account is disabled")
raise AuthenticationError(message="User account is disabled")
# Eagerly load all columns, then detach from session to avoid
# "instance not bound to a Session" errors after the context manager exits
await db.refresh(user)
db.expunge(user)
return user
import secrets
from fastapi.security import APIKeyHeader
api_key_header = APIKeyHeader(name=settings.API_KEY_HEADER, auto_error=False)
async def verify_api_key(
api_key: Annotated[str | None, Depends(api_key_header)],
) -> str:
"""Verify API key from header.
Uses constant-time comparison to prevent timing attacks.
Raises:
AuthenticationError: If API key is missing.
AuthorizationError: If API key is invalid.
"""
if api_key is None:
raise AuthenticationError(message="API Key header missing")
if not secrets.compare_digest(api_key, settings.API_KEY):
raise AuthorizationError(message="Invalid API Key")
return api_key
ValidAPIKey = Annotated[str, Depends(verify_api_key)]
# === RAG Service Dependencies ===
from app.services.rag.embeddings import EmbeddingService
from app.services.rag.ingestion import IngestionService
from app.services.rag.documents import DocumentProcessor
from fastapi import Request
from app.core.config import settings
from app.services.rag.retrieval import RetrievalService
from app.services.rag.vectorstore import QdrantVectorStore
def get_embedding_service(request: Request) -> EmbeddingService:
"""Get embedding service from lifespan state or create new if not available."""
if request and hasattr(request.state, "embedding_service"):
return request.state.embedding_service # type: ignore[no-any-return]
return EmbeddingService(settings=settings.rag)
# Type Alias for the Embedder
EmbeddingSvc = Annotated[EmbeddingService, Depends(get_embedding_service)]
from app.services.rag.vectorstore import BaseVectorStore
def get_vectorstore(request: Request, embedder: EmbeddingSvc) -> BaseVectorStore:
"""Get vector store client from lifespan state or create new."""
if request and hasattr(request.state, "vector_store"):
return request.state.vector_store # type: ignore[no-any-return]
return QdrantVectorStore(settings=settings.rag, embedding_service=embedder)
VectorStoreSvc = Annotated[BaseVectorStore, Depends(get_vectorstore)]
def get_retrieval_service(vector_store: VectorStoreSvc) -> RetrievalService:
"""Create RetrievalService instance."""
from app.services.rag.reranker import RerankService
rerank_service = RerankService(settings=settings.rag)
return RetrievalService(
vector_store=vector_store,
settings=settings.rag,
rerank_service=rerank_service,
)
RetrievalSvc = Annotated[RetrievalService, Depends(get_retrieval_service)]
def get_document_processor() -> DocumentProcessor:
"""Create DocumentProcessor instance."""
return DocumentProcessor(settings=settings.rag)
DocumentProcessorSvc = Annotated[DocumentProcessor, Depends(get_document_processor)]
def get_ingestion_service(
processor: DocumentProcessorSvc,
vector_store: VectorStoreSvc,
request: Request,
) -> IngestionService:
"""Create IngestionService instance."""
# Wire webhook dispatch for RAG events
async def on_rag_event(event: str, data: dict):
from app.services.webhook import WebhookService
db = request.state.db if hasattr(request.state, "db") else None
if db:
webhook_service = WebhookService(db)
await webhook_service.dispatch_event(event, data)
return IngestionService(processor=processor, vector_store=vector_store, on_event=on_rag_event)
IngestionSvc = Annotated[IngestionService, Depends(get_ingestion_service)]
from app.services.user_slash_command import UserSlashCommandService
def get_user_slash_command_service(db: DBSession) -> UserSlashCommandService:
return UserSlashCommandService(db)
UserSlashCommandSvc = Annotated[UserSlashCommandService, Depends(get_user_slash_command_service)]
from app.services.admin import AdminService
def get_admin_service(db: DBSession) -> AdminService:
"""Create AdminService instance — used by admin REST routes (always
available, independent of the optional SQLAdmin UI)."""
return AdminService(db)
AdminSvc = Annotated[AdminService, Depends(get_admin_service)]
backend/app/api/exception_handlers.py
+108
View File
@@ -0,0 +1,108 @@
"""Exception handlers for FastAPI application.
These handlers convert domain exceptions to proper HTTP responses.
WebSocket connections that raise an ``AppException`` before ``accept()`` are
handled too — Starlette closes the socket with 403 and we just log the
incident; we cannot return an HTTP body for a non-HTTP scope.
"""
import logging
from typing import Any
from fastapi import FastAPI
from fastapi.responses import JSONResponse
from starlette.requests import HTTPConnection
from app.core.exceptions import AppException
logger = logging.getLogger(__name__)
def _connection_meta(conn: HTTPConnection) -> dict[str, Any]:
"""Common log fields shared by HTTP requests and WebSocket connections.
``method`` exists only on HTTP ``Request`` — for WebSockets we surface the
scope type so log filters can still distinguish the two.
"""
return {
"path": conn.url.path,
"method": getattr(conn, "method", None) or conn.scope.get("type", "unknown"),
}
def _is_websocket(conn: HTTPConnection) -> bool:
return conn.scope.get("type") == "websocket"
async def app_exception_handler(request: HTTPConnection, exc: AppException) -> JSONResponse | None:
"""Handle application exceptions for both HTTP and WebSocket scopes.
Logs 5xx errors as errors and 4xx as warnings. Returns a JSON response
for HTTP scopes; returns ``None`` for WebSocket scopes (Starlette will
close the socket on its own).
"""
log_extra = {
"error_code": exc.code,
"status_code": exc.status_code,
"details": exc.details,
**_connection_meta(request),
}
if exc.status_code >= 500:
logger.error(f"{exc.code}: {exc.message}", extra=log_extra)
else:
logger.warning(f"{exc.code}: {exc.message}", extra=log_extra)
if _is_websocket(request):
return None
headers: dict[str, str] = {}
if exc.status_code == 401:
headers["WWW-Authenticate"] = "Bearer"
return JSONResponse(
status_code=exc.status_code,
content={
"error": {
"code": exc.code,
"message": exc.message,
"details": exc.details or None,
}
},
headers=headers,
)
async def unhandled_exception_handler(
request: HTTPConnection, exc: Exception
) -> JSONResponse | None:
"""Handle unexpected exceptions.
Logs the full exception but returns a generic error to the client
to avoid leaking sensitive information.
"""
logger.exception("Unhandled exception", extra=_connection_meta(request))
if _is_websocket(request):
return None
return JSONResponse(
status_code=500,
content={
"error": {
"code": "INTERNAL_ERROR",
"message": "An unexpected error occurred",
"details": None,
}
},
)
def register_exception_handlers(app: FastAPI) -> None:
"""Register all exception handlers on the FastAPI app.
Call this after creating the FastAPI application instance.
"""
app.add_exception_handler(AppException, app_exception_handler)
# Uncomment to catch all unhandled exceptions:
# app.add_exception_handler(Exception, unhandled_exception_handler)
backend/app/api/router.py
+10
View File
@@ -0,0 +1,10 @@
"""API router aggregation."""
from fastapi import APIRouter
from app.api.routes.v1 import v1_router
api_router = APIRouter()
# API v1 routes (prefix is set in main.py via settings.API_V1_STR)
api_router.include_router(v1_router)
backend/app/api/routes/__init__.py
+9
View File
@@ -0,0 +1,9 @@
"""API routes.
This package contains versioned API routes.
Add new versions by creating new folders (e.g., v2/) and updating router.py.
"""
from app.api.routes.v1 import v1_router
__all__ = ["v1_router"]
backend/app/api/routes/v1/__init__.py
+61
View File
@@ -0,0 +1,61 @@
"""API v1 router aggregation."""
# ruff: noqa: I001 - Imports structured for Jinja2 template conditionals
from fastapi import APIRouter
from app.api.routes.v1 import health
from app.api.routes.v1 import admin_users, auth, users
from app.api.routes.v1 import admin_ratings
from app.api.routes.v1 import conversations
from app.api.routes.v1 import admin_conversations
from app.api.routes.v1 import projects
from app.api.routes.v1 import webhooks
from app.api.routes.v1 import agent
from app.api.routes.v1 import rag
from app.api.routes.v1 import files
from app.api.routes.v1 import me_slash_commands
from app.api.routes.v1 import admin_stats
v1_router = APIRouter()
# Health check routes (no auth required)
v1_router.include_router(health.router, tags=["health"])
# Authentication routes
v1_router.include_router(auth.router, prefix="/auth", tags=["auth"])
# User routes
v1_router.include_router(users.router, prefix="/users", tags=["users"])
# Admin: message-rating analytics
v1_router.include_router(admin_ratings.router, prefix="/admin/ratings", tags=["admin:ratings"])
# Conversation routes (AI chat persistence)
v1_router.include_router(conversations.router, prefix="/conversations", tags=["conversations"])
# Project management routes (DeepAgents)
v1_router.include_router(projects.router, prefix="/projects", tags=["projects"])
# Webhook routes
v1_router.include_router(webhooks.router, prefix="/webhooks", tags=["webhooks"])
# AI Agent routes
v1_router.include_router(agent.router, tags=["agent"])
# RAG routes
v1_router.include_router(rag.router, prefix="/rag", tags=["rag"])
# File upload/download routes
v1_router.include_router(files.router, tags=["files"])
# Admin: conversation browser
v1_router.include_router(
admin_conversations.router, prefix="/admin/conversations", tags=["admin-conversations"]
)
# Admin: user management + impersonation
v1_router.include_router(admin_users.router, prefix="/admin/users", tags=["admin:users"])
v1_router.include_router(
me_slash_commands.router, prefix="/me/slash-commands", tags=["me:slash-commands"]
)
v1_router.include_router(admin_stats.router, prefix="/admin", tags=["admin:stats"])
backend/app/api/routes/v1/admin_conversations.py
+78
View File
@@ -0,0 +1,78 @@
"""Admin conversation and user browsing routes.
All endpoints require admin role.
Endpoints:
GET /admin/conversations — List all conversations (paginated, filterable)
GET /admin/conversations/{id} — Get any conversation with messages (read-only)
GET /admin/users — List all users with conversation counts
GET /admin/users/{user_id}/conversations — List conversations for a specific user
"""
from typing import Any, Literal
from uuid import UUID
from fastapi import APIRouter, Query
from app.api.deps import ConversationSvc, CurrentAdmin, UserSvc
from app.schemas.conversation import ConversationReadWithMessages
from app.schemas.conversation_share import AdminConversationList, AdminUserList
router = APIRouter()
@router.get("", response_model=AdminConversationList)
async def admin_list_conversations(
service: ConversationSvc,
_: CurrentAdmin,
skip: int = Query(0, ge=0, description="Items to skip"),
limit: int = Query(50, ge=1, le=100, description="Max items to return"),
search: str | None = Query(default=None, description="Search by title"),
user_id: UUID | None = Query(default=None, description="Filter by user ID"),
status: Literal["active", "archived", "all"] = Query(
"active", description="Filter by archival status"
),
sort_by: Literal["title", "owner", "messages", "created_at", "updated_at"] = Query(
"updated_at", description="Sort column"
),
sort_dir: Literal["asc", "desc"] = Query("desc", description="Sort direction"),
) -> Any:
"""List all conversations across all users (admin only)."""
return await service.admin_list_with_users(
skip=skip,
limit=limit,
search=search,
user_id=user_id,
include_archived=status == "all",
archived_only=status == "archived",
sort_by=sort_by,
sort_dir=sort_dir,
)
@router.get("/users", response_model=AdminUserList)
async def admin_list_users(
user_service: UserSvc,
_: CurrentAdmin,
skip: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=500),
search: str | None = Query(default=None, description="Search by email or name"),
sort_by: Literal["email", "full_name", "conversations", "created_at"] = Query(
"created_at", description="Sort column"
),
sort_dir: Literal["asc", "desc"] = Query("desc", description="Sort direction"),
) -> Any:
"""List all users with conversation counts (admin only)."""
return await user_service.admin_list_with_counts(
skip=skip, limit=limit, search=search, sort_by=sort_by, sort_dir=sort_dir
)
@router.get("/{conversation_id}", response_model=ConversationReadWithMessages)
async def admin_get_conversation(
conversation_id: UUID,
service: ConversationSvc,
_: CurrentAdmin,
) -> Any:
"""Get any conversation with messages (admin read-only access)."""
return await service.get_conversation_with_messages(conversation_id)
backend/app/api/routes/v1/admin_ratings.py
+95
View File
@@ -0,0 +1,95 @@
"""Admin endpoints for message ratings.
Provides endpoints for administrators to view and analyze user ratings
on AI assistant messages.
The endpoints are:
- GET /admin/ratings - List all ratings with filtering
- GET /admin/ratings/summary - Get aggregated rating statistics
- GET /admin/ratings/export - Export ratings as JSON or CSV
"""
from typing import Any
from fastapi import APIRouter, Query
from fastapi.responses import JSONResponse, StreamingResponse
from app.api.deps import CurrentAdmin, MessageRatingSvc
from app.schemas.message_rating import MessageRatingList, RatingSummary
router = APIRouter()
@router.get("", response_model=MessageRatingList)
async def list_ratings_admin(
rating_service: MessageRatingSvc,
_: CurrentAdmin,
skip: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=100),
rating_filter: int | None = Query(None, ge=-1, le=1, description="Filter by rating value"),
with_comments_only: bool = Query(False, description="Only show ratings with comments"),
) -> Any:
"""List all ratings with filtering (admin only).
Returns paginated list of ratings with optional filters:
- rating_filter: Filter by rating value (1 for likes, -1 for dislikes)
- with_comments_only: Only return ratings that have comments
Results are ordered by creation date (newest first).
"""
items, total = await rating_service.list_ratings(
skip=skip,
limit=limit,
rating_filter=rating_filter,
with_comments_only=with_comments_only,
)
return MessageRatingList(items=items, total=total)
@router.get("/summary", response_model=RatingSummary)
async def get_rating_summary(
rating_service: MessageRatingSvc,
_: CurrentAdmin,
days: int = Query(30, ge=1, le=365, description="Number of days to include"),
) -> Any:
"""Get aggregated rating statistics (admin only).
Returns summary statistics including:
- Total ratings count
- Like/dislike counts
- Average rating (-1.0 to 1.0)
- Count of ratings with comments
- Daily breakdown of ratings
The `days` parameter controls the time window (default: 30 days).
"""
return await rating_service.get_summary(days=days)
@router.get("/export")
async def export_ratings(
rating_service: MessageRatingSvc,
_: CurrentAdmin,
export_format: str = Query("json", description="Export format: 'json' or 'csv'"),
rating_filter: int | None = Query(None, ge=-1, le=1, description="Filter by rating value"),
with_comments_only: bool = Query(False, description="Only show ratings with comments"),
) -> Any:
"""Export all ratings as JSON or CSV (admin only).
CSV is streamed row-by-row; JSON collects into a single document.
"""
result = await rating_service.export_ratings(
export_format=export_format,
rating_filter=rating_filter,
with_comments_only=with_comments_only,
)
if result.media_type == "text/csv":
return StreamingResponse(
result.payload,
media_type="text/csv",
headers={"Content-Disposition": result.content_disposition},
)
return JSONResponse(
content=result.payload,
headers={"Content-Disposition": result.content_disposition},
)
backend/app/api/routes/v1/admin_stats.py
+37
View File
@@ -0,0 +1,37 @@
"""Admin observability — workspace stats + Stripe event log."""
from typing import Any
from fastapi import APIRouter, Query
from app.api.deps import AdminSvc, CurrentAdmin
from app.schemas.admin import AdminStats, StripeEventList, StripeEventRead
router = APIRouter()
@router.get("/stats", response_model=AdminStats)
async def get_admin_stats(
service: AdminSvc,
_user: CurrentAdmin,
) -> Any:
"""Aggregate workspace metrics. Billing fields are 0 when billing is
disabled in the deployment."""
return await service.workspace_stats()
@router.get("/stripe-events", response_model=StripeEventList)
async def list_stripe_events(
service: AdminSvc,
_user: CurrentAdmin,
skip: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=200),
) -> Any:
"""List recent Stripe webhook events from the idempotency log.
Best-effort: returns empty list when the StripeEvent table doesn't exist
(billing disabled in this deployment).
"""
rows, total = await service.list_stripe_events(skip=skip, limit=limit)
items = [StripeEventRead.model_validate(row) for row in rows]
return StripeEventList(items=items, total=total)
backend/app/api/routes/v1/admin_users.py
+93
View File
@@ -0,0 +1,93 @@
"""Admin user management routes.
All endpoints require the admin role (CurrentAdmin).
Endpoints:
GET /admin/users — List all users (paginated + search)
GET /admin/users/{user_id} — Get a single user
PATCH /admin/users/{user_id} — Update user (role, is_active, is_app_admin)
DELETE /admin/users/{user_id} — Hard-delete a user
POST /admin/users/{user_id}/impersonate — Issue short-lived token to act as user
"""
from datetime import timedelta
from typing import Any
from uuid import UUID
from fastapi import APIRouter, Query, Request, status
from app.api.deps import CurrentAdmin, DBSession, UserSvc
from app.core.security import create_access_token
from app.schemas.conversation_share import AdminUserList
from app.schemas.user import UserRead, UserUpdate
router = APIRouter()
@router.get("", response_model=AdminUserList)
async def list_users(
_: CurrentAdmin,
service: UserSvc,
skip: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=500),
search: str | None = Query(None),
) -> Any:
result = await service.admin_list_with_counts(skip=skip, limit=limit, search=search)
return result
@router.get("/{user_id}", response_model=UserRead)
async def get_user(
user_id: UUID,
_: CurrentAdmin,
service: UserSvc,
) -> Any:
return await service.get_by_id(user_id)
@router.patch("/{user_id}", response_model=UserRead)
async def update_user(
request: Request,
user_id: UUID,
user_in: UserUpdate,
admin: CurrentAdmin,
db: DBSession,
service: UserSvc,
) -> Any:
user = await service.update(user_id, user_in)
return user
@router.delete("/{user_id}", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
async def delete_user(
request: Request,
user_id: UUID,
admin: CurrentAdmin,
db: DBSession,
service: UserSvc,
) -> None:
await service.get_by_id(user_id) # raises 404 if not found
await service.delete(user_id)
@router.post("/{user_id}/impersonate", response_model=dict)
async def impersonate_user(
request: Request,
user_id: UUID,
admin: CurrentAdmin,
db: DBSession,
service: UserSvc,
) -> Any:
"""Issue a short-lived (1h) access token to act as the target user."""
target = await service.get_by_id(user_id)
token = create_access_token(
subject=str(target.id),
expires_delta=timedelta(hours=1),
)
return {
"access_token": token,
"token_type": "bearer",
"impersonated_user_id": str(target.id),
"impersonated_by": str(admin.id),
"expires_in": 3600,
}
backend/app/api/routes/v1/agent.py
+242
View File
@@ -0,0 +1,242 @@
"""AI Agent WebSocket route.
The route is just lifecycle plumbing — auth, accept, dispatch loop, disconnect.
Per-turn orchestration lives in :class:`app.services.agent_session.AgentSession`.
"""
import logging
from typing import Any
from uuid import UUID
from fastapi import APIRouter, Depends, WebSocket, WebSocketDisconnect
from pydantic_ai import (
FinalResultEvent,
FunctionToolCallEvent,
FunctionToolResultEvent,
PartDeltaEvent,
PartStartEvent,
TextPartDelta,
)
from pydantic_ai.messages import TextPart
from pydantic_ai_backends import StateBackend
from app.agents.pydantic_deep_assistant import PydanticDeepContext, get_agent
from app.api.deps import get_conversation_service, get_current_user_ws, get_project_service
from app.core.config import settings
from app.db.models.user import User
from app.db.session import get_db_context
from app.schemas.conversation import ConversationCreate, MessageCreate
from app.services.agent import AgentConnectionManager, send_event
from app.services.agent_session import AgentSession
logger = logging.getLogger(__name__)
router = APIRouter()
manager = AgentConnectionManager()
@router.get("/agent/models")
async def list_models() -> dict[str, Any]:
"""Return available LLM models and the current default."""
return {
"default": settings.AI_MODEL,
"models": settings.AI_AVAILABLE_MODELS,
}
@router.websocket("/ws/agent")
async def agent_websocket(
websocket: WebSocket,
user: User = Depends(get_current_user_ws),
) -> None:
"""WebSocket endpoint for the AI agent.
Streams agent events to the client. Each incoming JSON message is forwarded to
:class:`AgentSession.process_message`.
Expected input format::
{
"message": "user message here",
"file_ids": ["..."],
"conversation_id": "optional-uuid",
"model": "optional-model-override",
"thinking_effort": "optional"
}
Authentication: handled by ``get_current_user_ws`` (JWT).
"""
if user is None:
return
await manager.connect(websocket)
session = AgentSession(
websocket,
user,
)
try:
while True:
try:
data = await websocket.receive_json()
except WebSocketDisconnect:
break
try:
await session.process_message(data)
except WebSocketDisconnect:
logger.info("Client disconnected during agent processing")
break
finally:
manager.disconnect(websocket)
@router.websocket("/ws/projects/{project_id}/chats/{conversation_id}")
async def project_chat_websocket(
project_id: UUID,
conversation_id: UUID,
websocket: WebSocket,
user: User = Depends(get_current_user_ws),
) -> None:
"""WebSocket endpoint for project-scoped PydanticDeep chat.
One Docker container per project is shared across all chats.
Chat history is stored per-chat inside the project volume at:
.pydantic-deep/sessions/{conversation_id}/messages.json
"""
await manager.connect(websocket)
context: PydanticDeepContext = {}
context["user_id"] = str(user.id) if user else None
context["user_name"] = user.email if user else None
try:
# Verify project access
async with get_db_context() as db:
project_service = get_project_service(db)
try:
await project_service.get(project_id, user_id=user.id)
except Exception as exc:
await websocket.close(code=4003, reason=str(exc))
return
backend: Any = StateBackend()
assistant = get_agent(
conversation_id=str(conversation_id),
backend_override=backend,
history_messages_path=f".pydantic-deep/sessions/{conversation_id}/messages.json",
)
# Ensure the conversation record exists and is linked to the project
async with get_db_context() as db:
conv_service = get_conversation_service(db)
try:
await conv_service.get_conversation(conversation_id, user_id=user.id)
except Exception:
conv = await conv_service.create_conversation(
ConversationCreate(
user_id=user.id,
project_id=project_id,
)
)
await send_event(
websocket,
"conversation_created",
{"conversation_id": str(conv.id), "project_id": str(project_id)},
)
while True:
data = await websocket.receive_json()
user_message = data.get("message", "")
if not user_message:
await send_event(websocket, "error", {"message": "Empty message"})
continue
await send_event(websocket, "user_prompt", {"content": user_message})
async with get_db_context() as db:
conv_service = get_conversation_service(db)
try:
await conv_service.add_message(
conversation_id,
MessageCreate(role="user", content=user_message),
)
except Exception as exc:
logger.warning("Failed to persist user message: %s", exc)
try:
await send_event(websocket, "model_request_start", {})
async with assistant.agent.run_stream(user_message, deps=assistant.deps) as stream:
async for event in stream.stream_events():
if isinstance(event, PartDeltaEvent) and isinstance(
event.delta, TextPartDelta
):
await send_event(
websocket, "text_delta", {"delta": event.delta.content_delta}
)
elif isinstance(event, PartStartEvent) and isinstance(event.part, TextPart):
pass
elif isinstance(event, FunctionToolCallEvent):
await send_event(
websocket,
"tool_call",
{
"tool_name": event.part.tool_name,
"args": str(event.part.args),
},
)
elif isinstance(event, FunctionToolResultEvent):
await send_event(
websocket,
"tool_result",
{
"tool_name": event.result.tool_name,
"content": str(event.result.content),
},
)
elif isinstance(event, FinalResultEvent):
await send_event(
websocket, "final_result", {"content": str(event.output)}
)
result = stream.result()
async with get_db_context() as db:
conv_service = get_conversation_service(db)
try:
await conv_service.add_message(
conversation_id,
MessageCreate(
role="assistant",
content=getattr(result, "output", ""),
model_name=getattr(assistant, "model_name", None),
),
)
except Exception as exc:
logger.warning("Failed to persist assistant response: %s", exc)
await send_event(
websocket,
"complete",
{
"conversation_id": str(conversation_id),
"project_id": str(project_id),
},
)
except WebSocketDisconnect:
logger.info("Client disconnected during project chat")
break
except Exception as exc:
logger.exception("Error in project chat: %s", exc)
await send_event(websocket, "error", {"message": str(exc)})
except WebSocketDisconnect:
pass
finally:
manager.disconnect(websocket)
backend/app/api/routes/v1/auth.py
+92
View File
@@ -0,0 +1,92 @@
"""Authentication routes."""
import logging
from typing import Annotated, Any
from fastapi import APIRouter, Depends, Request, status
from fastapi.security import OAuth2PasswordRequestForm
from app.api.deps import CurrentUser, UserSvc
from app.core.exceptions import AuthenticationError
from app.core.security import (
create_access_token,
create_refresh_token,
verify_token,
)
from app.schemas.token import RefreshTokenRequest, Token
from app.schemas.user import UserCreate, UserRead
logger = logging.getLogger(__name__)
router = APIRouter()
@router.post("/login", response_model=Token)
async def login(
request: Request,
form_data: Annotated[OAuth2PasswordRequestForm, Depends()],
user_service: UserSvc,
) -> Any:
"""OAuth2 compatible token login.
Returns access token and refresh token.
Raises domain exceptions handled by exception handlers.
"""
user = await user_service.authenticate(form_data.username, form_data.password)
access_token = create_access_token(subject=str(user.id))
refresh_token = create_refresh_token(subject=str(user.id))
return Token(access_token=access_token, refresh_token=refresh_token)
@router.post("/register", response_model=UserRead, status_code=status.HTTP_201_CREATED)
async def register(
user_in: UserCreate,
user_service: UserSvc,
) -> Any:
"""Register a new user.
Raises AlreadyExistsError if email is already registered.
"""
user = await user_service.register(user_in)
return user
@router.post("/refresh", response_model=Token)
async def refresh_token(
request: Request,
body: RefreshTokenRequest,
user_service: UserSvc,
) -> Any:
"""Get new access token using refresh token.
Raises AuthenticationError if refresh token is invalid or expired.
"""
# No DB-backed sessions — validate the refresh JWT directly.
payload = verify_token(body.refresh_token)
if not payload or payload.get("type") != "refresh":
raise AuthenticationError(message="Invalid or expired refresh token")
user_id = payload.get("sub")
if not user_id:
raise AuthenticationError(message="Invalid refresh token")
user = await user_service.get_by_id(user_id)
if not user.is_active:
raise AuthenticationError(message="User account is disabled")
access_token = create_access_token(subject=str(user.id))
new_refresh_token = create_refresh_token(subject=str(user.id))
return Token(access_token=access_token, refresh_token=new_refresh_token)
@router.post("/logout", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
async def logout(
body: RefreshTokenRequest,
) -> None:
"""No-op without session tracking. Clients drop their JWTs locally."""
return None
@router.get("/me", response_model=UserRead)
async def get_current_user_info(current_user: CurrentUser) -> Any:
"""Get current authenticated user information."""
return current_user
backend/app/api/routes/v1/conversations.py
+363
View File
@@ -0,0 +1,363 @@
"""Conversation API routes for AI chat persistence.
Provides CRUD operations for conversations and messages.
The endpoints are:
- GET /conversations - List user's conversations
- POST /conversations - Create a new conversation
- GET /conversations/{id} - Get a conversation with messages
- PATCH /conversations/{id} - Update conversation title/archived status
- DELETE /conversations/{id} - Delete a conversation
- POST /conversations/{id}/messages - Add a message to conversation
- GET /conversations/{id}/messages - List messages in conversation
"""
from typing import Any
from uuid import UUID
from fastapi import APIRouter, Query, Response, status
from fastapi.responses import JSONResponse
from app.api.deps import (
ConversationShareSvc,
ConversationSvc,
CurrentAdmin,
CurrentUser,
MessageRatingSvc,
)
from app.schemas.conversation import (
ConversationAdminList,
ConversationCreate,
ConversationList,
ConversationRead,
ConversationReadWithMessages,
ConversationUpdate,
MessageCreate,
MessageList,
MessageRead,
)
from app.schemas.conversation_share import (
ConversationShareCreate,
ConversationShareList,
ConversationShareRead,
)
from app.schemas.message_rating import (
MessageRatingCreate,
MessageRatingRead,
)
router = APIRouter()
@router.get("/export")
async def export_conversations(
conversation_service: ConversationSvc,
_: CurrentAdmin,
) -> Any:
"""Export all conversations with messages and tool calls (admin only)."""
export_data = await conversation_service.export_all()
return JSONResponse(
content={"conversations": export_data, "total": len(export_data)},
headers={"Content-Disposition": 'attachment; filename="conversations_export.json"'},
)
@router.get("/admin-list", response_model=ConversationAdminList)
async def list_conversations_admin(
conversation_service: ConversationSvc,
_: CurrentAdmin,
skip: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=100),
include_archived: bool = Query(True, description="Include archived conversations"),
search: str | None = Query(None, max_length=100, description="Search by title or ID prefix"),
) -> Any:
"""List all conversations with message counts (admin only).
Returns paginated conversations without message content.
"""
items, total = await conversation_service.list_conversations_admin(
skip=skip,
limit=limit,
include_archived=include_archived,
search=search,
)
return ConversationAdminList(items=items, total=total)
@router.get("", response_model=ConversationList)
async def list_conversations(
conversation_service: ConversationSvc,
current_user: CurrentUser,
skip: int = Query(0, ge=0, description="Number of conversations to skip"),
limit: int = Query(50, ge=1, le=100, description="Maximum conversations to return"),
include_archived: bool = Query(False, description="Include archived conversations"),
) -> Any:
"""List conversations for the current user.
Returns conversations ordered by most recently updated.
"""
items, total = await conversation_service.list_conversations(
user_id=current_user.id,
skip=skip,
limit=limit,
include_archived=include_archived,
)
return ConversationList(items=items, total=total) # type: ignore[arg-type]
@router.post("", response_model=ConversationRead, status_code=status.HTTP_201_CREATED)
async def create_conversation(
conversation_service: ConversationSvc,
current_user: CurrentUser,
data: ConversationCreate | None = None,
) -> Any:
"""Create a new conversation.
The title is optional and can be set later.
"""
if data is None:
data = ConversationCreate()
data = data.model_copy(update={"user_id": current_user.id})
return await conversation_service.create_conversation(data)
@router.get("/{conversation_id}", response_model=ConversationReadWithMessages)
async def get_conversation(
conversation_id: UUID,
conversation_service: ConversationSvc,
current_user: CurrentUser,
) -> Any:
"""Get a conversation with all its messages.
Raises 404 if the conversation does not exist.
"""
uid = None if current_user.role == "admin" else current_user.id
return await conversation_service.get_conversation(
conversation_id,
include_messages=True,
user_id=uid,
)
@router.patch("/{conversation_id}", response_model=ConversationRead)
async def update_conversation(
conversation_id: UUID,
data: ConversationUpdate,
conversation_service: ConversationSvc,
current_user: CurrentUser,
) -> Any:
"""Update a conversation's title or archived status.
Raises 404 if the conversation does not exist.
"""
return await conversation_service.update_conversation(
conversation_id,
data,
user_id=current_user.id,
)
@router.delete("/{conversation_id}", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
async def delete_conversation(
conversation_id: UUID,
conversation_service: ConversationSvc,
current_user: CurrentUser,
) -> None:
"""Delete a conversation and all its messages.
Raises 404 if the conversation does not exist.
"""
await conversation_service.delete_conversation(
conversation_id,
user_id=current_user.id,
)
@router.post(
"/{conversation_id}/archive",
response_model=ConversationRead,
)
async def archive_conversation(
conversation_id: UUID,
conversation_service: ConversationSvc,
current_user: CurrentUser,
) -> Any:
"""Archive a conversation.
Archived conversations are hidden from the default list view.
"""
return await conversation_service.archive_conversation(
conversation_id,
user_id=current_user.id,
)
@router.get("/{conversation_id}/messages", response_model=MessageList)
async def list_messages(
conversation_id: UUID,
conversation_service: ConversationSvc,
current_user: CurrentUser,
skip: int = Query(0, ge=0),
limit: int = Query(100, ge=1, le=500),
) -> Any:
"""List messages in a conversation.
Returns messages ordered by creation time (oldest first).
"""
uid = None if current_user.role == "admin" else current_user.id
items, total = await conversation_service.list_messages(
conversation_id,
skip=skip,
limit=limit,
include_tool_calls=True,
user_id=uid,
)
return MessageList(items=items, total=total) # type: ignore[arg-type]
@router.post(
"/{conversation_id}/messages",
response_model=MessageRead,
status_code=status.HTTP_201_CREATED,
)
async def add_message(
conversation_id: UUID,
data: MessageCreate,
conversation_service: ConversationSvc,
current_user: CurrentUser,
) -> Any:
"""Add a message to a conversation.
Raises 404 if the conversation does not exist.
"""
return await conversation_service.add_message(conversation_id, data)
@router.post(
"/{conversation_id}/messages/{message_id}/rate",
response_model=MessageRatingRead,
status_code=status.HTTP_200_OK,
)
async def rate_message(
conversation_id: UUID,
message_id: UUID,
data: MessageRatingCreate,
rating_service: MessageRatingSvc,
current_user: CurrentUser,
response: Response,
) -> Any:
"""Rate an assistant message.
Creates a new rating or updates an existing one.
Only assistant messages can be rated.
Args:
conversation_id: The conversation containing the message
message_id: The message to rate
data: Rating value (1 for like, -1 for dislike) and optional comment
Returns:
201 Created for new rating, 200 OK when updating existing rating.
"""
rating, is_new = await rating_service.rate_message(
conversation_id=conversation_id,
message_id=message_id,
user_id=current_user.id,
data=data,
)
if is_new:
response.status_code = status.HTTP_201_CREATED
return rating
@router.delete(
"/{conversation_id}/messages/{message_id}/rate",
status_code=status.HTTP_204_NO_CONTENT,
response_model=None,
)
async def remove_rating(
conversation_id: UUID,
message_id: UUID,
rating_service: MessageRatingSvc,
current_user: CurrentUser,
) -> None:
"""Remove your rating from a message.
Args:
conversation_id: The conversation containing the message
message_id: The message to remove rating from
"""
await rating_service.remove_rating(
conversation_id=conversation_id,
message_id=message_id,
user_id=current_user.id,
)
@router.get("/shared-with-me", response_model=ConversationList)
async def list_shared_with_me(
share_service: ConversationShareSvc,
current_user: CurrentUser,
skip: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=100),
) -> Any:
"""List conversations shared with the current user."""
items, total = await share_service.list_shared_with_me(current_user.id, skip=skip, limit=limit)
return ConversationList(items=items, total=total)
@router.post(
"/{conversation_id}/shares",
response_model=ConversationShareRead,
status_code=status.HTTP_201_CREATED,
)
async def share_conversation(
conversation_id: UUID,
data: ConversationShareCreate,
share_service: ConversationShareSvc,
current_user: CurrentUser,
) -> Any:
"""Share a conversation with another user or generate a public link."""
result = await share_service.share_conversation(
conversation_id,
shared_by=current_user.id,
shared_with=data.shared_with,
generate_link=data.generate_link,
permission=data.permission,
)
return result["share"]
@router.get("/{conversation_id}/shares", response_model=ConversationShareList)
async def list_shares(
conversation_id: UUID,
share_service: ConversationShareSvc,
current_user: CurrentUser,
) -> Any:
"""List all shares for a conversation (owner only)."""
shares = await share_service.list_shares(conversation_id, current_user.id)
return ConversationShareList(items=shares, total=len(shares))
@router.delete(
"/{conversation_id}/shares/{share_id}",
status_code=status.HTTP_204_NO_CONTENT,
response_model=None,
)
async def revoke_share(
conversation_id: UUID,
share_id: UUID,
share_service: ConversationShareSvc,
current_user: CurrentUser,
) -> None:
"""Revoke a conversation share."""
await share_service.revoke_share(share_id, current_user.id)
@router.get("/shared/{token}")
async def get_shared_conversation(
token: str,
share_service: ConversationShareSvc,
) -> Any:
"""Access a shared conversation via public token (no auth required)."""
return await share_service.get_by_token(token)
backend/app/api/routes/v1/files.py
+121
View File
@@ -0,0 +1,121 @@
"""File upload and download endpoints for chat attachments."""
import logging
from typing import Any
from uuid import UUID
from fastapi import APIRouter, File, HTTPException, UploadFile, status
from fastapi.responses import FileResponse
from app.api.deps import CurrentUser, FileUploadSvc
from app.core.exceptions import NotFoundError
from app.schemas.file import FileInfo, FileUploadResponse
from app.services.file_storage import get_file_storage
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/files", tags=["files"])
@router.post("/upload", response_model=FileUploadResponse, status_code=status.HTTP_201_CREATED)
async def upload_file(
file_upload_svc: FileUploadSvc,
current_user: CurrentUser,
file: UploadFile = File(...),
) -> Any:
"""Upload a file for use in chat."""
data = await file.read()
is_valid, error = file_upload_svc.validate_upload(file.content_type, len(data))
if not is_valid:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=error)
file_type = file_upload_svc.classify_file(file.content_type or "", file.filename or "unknown")
parsed_content = await file_upload_svc.parse_content(data, file_type, file.content_type or "")
storage = get_file_storage()
storage_path = await storage.save(str(current_user.id), file.filename or "unknown", data)
chat_file = await file_upload_svc.create_chat_file(
user_id=current_user.id,
filename=file.filename or "unknown",
mime_type=file.content_type or "application/octet-stream",
size=len(data),
storage_path=storage_path,
file_type=file_type,
parsed_content=parsed_content,
)
return FileUploadResponse(
id=chat_file.id,
filename=chat_file.filename,
mime_type=chat_file.mime_type,
size=chat_file.size,
file_type=chat_file.file_type,
)
@router.get("/{file_id}")
async def download_file(
file_id: UUID,
file_upload_svc: FileUploadSvc,
current_user: CurrentUser,
disposition: str = "inline",
) -> Any:
"""Serve a file. Only the owner can access their files.
By default the response is ``Content-Disposition: inline`` so PDFs, images
and audio/video render directly inside an ``<iframe>`` / media tag (used
by the chat file-preview panel). Pass ``?disposition=attachment`` to force
the browser's download dialog (used by the explicit "Download" button).
"""
try:
chat_file = await file_upload_svc.get_user_file(file_id, current_user.id)
except NotFoundError:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="File not found"
) from None
storage = get_file_storage()
file_path = storage.get_full_path(chat_file.storage_path)
if not file_path:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="File not found on disk")
# FastAPI's ``FileResponse(filename=...)`` always uses ``attachment`` —
# build the header manually so we can switch to ``inline`` for previews.
mode = "attachment" if disposition == "attachment" else "inline"
safe_name = chat_file.filename.replace('"', "")
# The chat file-preview panel embeds this URL in an iframe (PDFs, HTML,
# etc). Default ``X-Frame-Options: DENY`` from SecurityHeadersMiddleware
# would break that, so opt this endpoint down to SAMEORIGIN. The CSP
# ``frame-ancestors 'self'`` is the modern equivalent — browsers honor
# whichever they recognize.
headers = {
"Content-Disposition": f'{mode}; filename="{safe_name}"',
"X-Frame-Options": "SAMEORIGIN",
"Content-Security-Policy": "frame-ancestors 'self'",
}
return FileResponse(path=file_path, media_type=chat_file.mime_type, headers=headers)
@router.get("/{file_id}/info", response_model=FileInfo)
async def get_file_info(
file_id: UUID,
file_upload_svc: FileUploadSvc,
current_user: CurrentUser,
) -> Any:
"""Get file metadata. Only the owner can access."""
try:
chat_file = await file_upload_svc.get_user_file(file_id, current_user.id)
except NotFoundError:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="File not found"
) from None
return FileInfo(
id=chat_file.id,
filename=chat_file.filename,
mime_type=chat_file.mime_type,
size=chat_file.size,
file_type=chat_file.file_type,
created_at=chat_file.created_at,
user_id=chat_file.user_id,
)
backend/app/api/routes/v1/health.py
+186
View File
@@ -0,0 +1,186 @@
"""Health check endpoints.
Provides Kubernetes-compatible health check endpoints:
- /health - Simple liveness check
- /health/live - Detailed liveness probe
- /health/ready - Readiness probe with dependency checks
"""
# ruff: noqa: I001 - Imports structured for Jinja2 template conditionals
from datetime import UTC, datetime
from typing import Any
from fastapi import APIRouter
from fastapi.responses import JSONResponse
from sqlalchemy import text
from app.api.deps import DBSession, Redis
from app.core.config import settings
from app.services.health import build_health_response
router = APIRouter()
@router.get("/health")
async def health_check() -> dict[str, Any]:
"""Simple liveness probe - check if application is running.
This is a lightweight check that should always succeed if the
application is running. Use this for basic connectivity tests.
Returns:
{"status": "healthy"}
"""
return {
"status": "healthy",
"max_upload_size_mb": settings.MAX_UPLOAD_SIZE_MB,
}
@router.get("/health/live")
async def liveness_probe() -> dict[str, Any]:
"""Detailed liveness probe for Kubernetes.
This endpoint is designed for Kubernetes liveness probes.
It checks if the application process is alive and responding.
Failure indicates the container should be restarted.
Returns:
Structured response with timestamp and service info.
"""
return build_health_response(
status="alive",
details={
"version": getattr(settings, "VERSION", "1.0.0"),
"environment": settings.ENVIRONMENT,
},
)
@router.get("/health/ready", response_model=None)
async def readiness_probe(
db: DBSession,
redis: Redis,
) -> dict[str, Any] | JSONResponse:
"""Readiness probe for Kubernetes.
This endpoint checks if all dependencies are ready to handle traffic.
It verifies database connections, Redis, and other critical services.
Failure indicates traffic should be temporarily diverted.
Checks performed:
- Database connectivity
- Redis connectivity
Returns:
Structured response with individual check results.
Returns 503 if any critical check fails.
"""
checks: dict[str, dict[str, Any]] = {}
# Database check
try:
start = datetime.now(UTC)
await db.execute(text("SELECT 1"))
latency_ms = (datetime.now(UTC) - start).total_seconds() * 1000
checks["database"] = {
"status": "healthy",
"latency_ms": round(latency_ms, 2),
"type": "postgresql",
}
except Exception as e:
checks["database"] = {
"status": "unhealthy",
"error": str(e),
"type": "postgresql",
}
# Redis check
try:
start = datetime.now(UTC)
is_healthy = await redis.ping()
latency_ms = (datetime.now(UTC) - start).total_seconds() * 1000
if is_healthy:
checks["redis"] = {
"status": "healthy",
"latency_ms": round(latency_ms, 2),
}
else:
checks["redis"] = {
"status": "unhealthy",
"error": "Ping failed",
}
except Exception as e:
checks["redis"] = {
"status": "unhealthy",
"error": str(e),
}
# Vector store — Qdrant connectivity probe (TCP).
try:
import socket
start = datetime.now(UTC)
with socket.create_connection((settings.QDRANT_HOST, settings.QDRANT_PORT), timeout=2):
pass
latency_ms = (datetime.now(UTC) - start).total_seconds() * 1000
checks["vector_store"] = {
"status": "healthy",
"latency_ms": round(latency_ms, 2),
"type": "qdrant",
}
except Exception as e:
checks["vector_store"] = {
"status": "unhealthy",
"error": str(e),
"type": "qdrant",
}
# LLM provider — config-only check (avoid spending money on a probe call).
llm_provider = (getattr(settings, "LLM_PROVIDER", None) or "").lower()
if llm_provider:
key_field = {
"openai": "OPENAI_API_KEY",
"anthropic": "ANTHROPIC_API_KEY",
"google": "GOOGLE_API_KEY",
"openrouter": "OPENROUTER_API_KEY",
}.get(llm_provider)
api_key = getattr(settings, key_field, None) if key_field else None
checks["llm"] = {
"status": "healthy" if api_key else "unhealthy",
"provider": llm_provider,
"detail": "API key configured" if api_key else "API key missing",
}
else:
checks["llm"] = {"status": "unknown", "detail": "not configured"}
# Determine overall health — only db + redis are critical for readiness.
critical = {k: v for k, v in checks.items() if k in ("database", "redis")}
all_healthy = (
all(check.get("status") == "healthy" for check in critical.values()) if critical else True
)
# The admin /system page reads each service from the top level, so flatten
# the checks alongside the structured `checks` field for K8s probes.
response_data = build_health_response(
status="ready" if all_healthy else "not_ready",
checks=checks,
)
response_data.update(checks)
if not all_healthy:
return JSONResponse(status_code=503, content=response_data)
return response_data
# Backward compatibility - keep /ready endpoint
@router.get("/ready", response_model=None)
async def readiness_check(
db: DBSession,
redis: Redis,
) -> dict[str, Any] | JSONResponse:
"""Readiness check (alias for /health/ready).
Deprecated: Use /health/ready instead.
"""
return await readiness_probe(
db=db,
redis=redis,
)
backend/app/api/routes/v1/me_slash_commands.py
+85
View File
@@ -0,0 +1,85 @@
"""User-scoped slash command settings.
Routes are nested under ``/me/slash-commands`` because they're always
operating on the current user — there's no cross-user view of these.
"""
from typing import Any
from uuid import UUID
from fastapi import APIRouter, status
from app.api.deps import CurrentUser, UserSlashCommandSvc
from app.schemas.user_slash_command import (
BuiltinOverrideUpsert,
UserSlashCommandCustomCreate,
UserSlashCommandList,
UserSlashCommandRead,
UserSlashCommandUpdate,
)
router = APIRouter()
@router.get("", response_model=UserSlashCommandList)
async def list_slash_commands(service: UserSlashCommandSvc, user: CurrentUser) -> Any:
"""List the current user's custom commands and built-in overrides."""
items, total = await service.list_for_user(user_id=user.id)
return UserSlashCommandList(
items=[UserSlashCommandRead.model_validate(c) for c in items],
total=total,
)
@router.post(
"/custom",
response_model=UserSlashCommandRead,
status_code=status.HTTP_201_CREATED,
)
async def create_custom_command(
data: UserSlashCommandCustomCreate,
service: UserSlashCommandSvc,
user: CurrentUser,
) -> Any:
"""Create a user-defined command with a stored prompt body."""
db_cmd = await service.create_custom(user_id=user.id, data=data)
return UserSlashCommandRead.model_validate(db_cmd)
@router.put("/builtin", response_model=UserSlashCommandRead)
async def upsert_builtin_override(
data: BuiltinOverrideUpsert,
service: UserSlashCommandSvc,
user: CurrentUser,
) -> Any:
"""Toggle a built-in command on or off for the current user."""
db_cmd = await service.upsert_builtin_override(
user_id=user.id, name=data.name, is_enabled=data.is_enabled
)
return UserSlashCommandRead.model_validate(db_cmd)
@router.patch(
"/{command_id}",
response_model=UserSlashCommandRead,
)
async def update_slash_command(
command_id: UUID,
data: UserSlashCommandUpdate,
service: UserSlashCommandSvc,
user: CurrentUser,
) -> Any:
"""Patch a custom command. Built-in overrides accept only ``is_enabled``."""
db_cmd = await service.update(user_id=user.id, command_id=command_id, data=data)
return UserSlashCommandRead.model_validate(db_cmd)
@router.delete("/{command_id}", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
async def delete_slash_command(
command_id: UUID,
service: UserSlashCommandSvc,
user: CurrentUser,
) -> Any:
"""Delete a custom command, or remove a built-in override (re-enables it)."""
await service.delete(user_id=user.id, command_id=command_id)
return None
backend/app/api/routes/v1/projects.py
+154
View File
@@ -0,0 +1,154 @@
"""Project management routes — CRUD and member management for DeepAgents projects."""
from typing import Any
from uuid import UUID
from fastapi import APIRouter, Query, status
from app.api.deps import CurrentUser, ProjectSvc
from app.schemas.project import (
ProjectCreate,
ProjectList,
ProjectMemberCreate,
ProjectMemberList,
ProjectMemberRead,
ProjectMemberUpdate,
ProjectRead,
ProjectUpdate,
)
router = APIRouter()
# Project CRUD
@router.get("", response_model=ProjectList)
async def list_projects(
service: ProjectSvc,
user: CurrentUser,
skip: int = Query(0, ge=0, description="Items to skip"),
limit: int = Query(50, ge=1, le=100, description="Max items to return"),
include_archived: bool = Query(False, description="Include archived projects"),
) -> Any:
"""List all projects accessible to the current user (owned + member)."""
items, total = await service.list_for_user(
user.id,
skip=skip,
limit=limit,
include_archived=include_archived,
)
return ProjectList(items=items, total=total)
@router.post("", response_model=ProjectRead, status_code=status.HTTP_201_CREATED)
async def create_project(
data: ProjectCreate,
service: ProjectSvc,
user: CurrentUser,
) -> Any:
"""Create a new project. A Docker volume is provisioned on creation."""
return await service.create(data, owner_id=user.id)
@router.get("/{project_id}", response_model=ProjectRead)
async def get_project(
project_id: UUID,
service: ProjectSvc,
user: CurrentUser,
) -> Any:
"""Get a project by ID. Requires at least viewer access."""
return await service.get(project_id, user_id=user.id)
@router.patch("/{project_id}", response_model=ProjectRead)
async def update_project(
project_id: UUID,
data: ProjectUpdate,
service: ProjectSvc,
user: CurrentUser,
) -> Any:
"""Update a project. Requires admin role or ownership."""
return await service.update(project_id, data, user_id=user.id)
@router.delete("/{project_id}", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
async def delete_project(
project_id: UUID,
service: ProjectSvc,
user: CurrentUser,
) -> None:
"""Delete a project permanently (removes Docker container and volume).
Only the project owner can delete it.
"""
await service.delete(project_id, user_id=user.id)
@router.post("/{project_id}/archive", response_model=ProjectRead)
async def archive_project(
project_id: UUID,
service: ProjectSvc,
user: CurrentUser,
) -> Any:
"""Archive a project (soft-delete). Only the owner can archive."""
return await service.archive(project_id, user_id=user.id)
# Member Management
@router.get("/{project_id}/members", response_model=ProjectMemberList)
async def list_members(
project_id: UUID,
service: ProjectSvc,
user: CurrentUser,
) -> Any:
"""List all members of a project. Requires viewer access."""
members = await service.list_members(project_id, user_id=user.id)
return ProjectMemberList(items=members, total=len(members))
@router.post(
"/{project_id}/members",
response_model=ProjectMemberRead,
status_code=status.HTTP_201_CREATED,
)
async def add_member(
project_id: UUID,
data: ProjectMemberCreate,
service: ProjectSvc,
user: CurrentUser,
) -> Any:
"""Add a user to a project. Requires admin role or ownership."""
return await service.add_member(project_id, data, inviter_id=user.id)
@router.patch("/{project_id}/members/{user_id}", response_model=ProjectMemberRead)
async def update_member(
project_id: UUID,
user_id: UUID,
data: ProjectMemberUpdate,
service: ProjectSvc,
user: CurrentUser,
) -> Any:
"""Update a member's role. Requires admin role or ownership."""
return await service.update_member(project_id, user_id, data, requester_id=user.id)
@router.delete(
"/{project_id}/members/{user_id}",
status_code=status.HTTP_204_NO_CONTENT,
response_model=None,
)
async def remove_member(
project_id: UUID,
user_id: UUID,
service: ProjectSvc,
user: CurrentUser,
) -> None:
"""Remove a member from a project.
Members can remove themselves. Admin/owner can remove anyone.
"""
await service.remove_member(project_id, user_id, requester_id=user.id)
backend/app/api/routes/v1/rag.py
+370
View File
@@ -0,0 +1,370 @@
"""RAG API routes — collection management, search, document upload, sync, status stream.
Routes are HTTP plumbing only. Business logic, file I/O, task dispatch, and Redis
pub/sub all live in their respective services. Domain exceptions raised by services are
mapped to HTTP responses by the global exception handlers in
``app.api.exception_handlers``; routes do not catch and re-wrap them.
"""
from collections.abc import AsyncIterable
from typing import Any
from fastapi import APIRouter, File, Query, UploadFile, status
from fastapi.responses import FileResponse
from fastapi.sse import EventSourceResponse, ServerSentEvent
from app.api.deps import (
CurrentAdmin,
CurrentUser,
IngestionSvc,
RAGDocumentSvc,
RAGStatusSvc,
RAGSyncSvc,
RetrievalSvc,
SyncSourceSvc,
VectorStoreSvc,
)
from app.core.config import settings
from app.core.exceptions import NotFoundError
from app.schemas.rag import (
RAGCollectionInfo,
RAGCollectionList,
RAGDocumentList,
RAGIngestResponse,
RAGMessageResponse,
RAGRetryResponse,
RAGSearchRequest,
RAGSearchResponse,
RAGSearchResult,
RAGSyncLogList,
RAGSyncRequest,
RAGSyncResponse,
RAGTrackedDocumentList,
)
from app.schemas.sync_source import (
ConnectorList,
SyncSourceCreate,
SyncSourceList,
SyncSourceRead,
SyncSourceUpdate,
)
from app.services.rag.config import get_supported_formats
router = APIRouter()
@router.get("/supported-formats")
async def get_supported_formats_endpoint() -> Any:
"""Return file formats supported by the current PDF parser configuration."""
parser_name = getattr(settings, "PDF_PARSER", "pymupdf")
return {"parser": parser_name, "formats": sorted(get_supported_formats(parser_name))}
@router.get("/collections", response_model=RAGCollectionList)
async def list_collections(
vector_store: VectorStoreSvc,
_: CurrentAdmin,
) -> Any:
"""List all available collections in the vector store."""
names = await vector_store.list_collections()
return RAGCollectionList(items=names)
@router.post(
"/collections/{name}",
status_code=status.HTTP_201_CREATED,
response_model=RAGMessageResponse,
)
async def create_collection(
name: str,
vector_store: VectorStoreSvc,
_: CurrentAdmin,
) -> Any:
"""Create and initialize a new collection."""
await vector_store.create_collection(name)
return RAGMessageResponse(message=f"Collection '{name}' created successfully.")
@router.delete(
"/collections/{name}",
status_code=status.HTTP_204_NO_CONTENT,
response_model=None,
)
async def drop_collection(
name: str,
vector_store: VectorStoreSvc,
rag_doc_svc: RAGDocumentSvc,
_: CurrentAdmin,
) -> None:
"""Drop an entire collection — vectors and all SQL document records."""
await vector_store.delete_collection(name)
await rag_doc_svc.delete_by_collection(name)
@router.get("/collections/{name}/info", response_model=RAGCollectionInfo)
async def get_collection_info(
name: str,
vector_store: VectorStoreSvc,
_: CurrentAdmin,
) -> Any:
"""Retrieve stats for a specific collection."""
return await vector_store.get_collection_info(name)
@router.get("/collections/{name}/documents", response_model=RAGDocumentList)
async def list_documents(
name: str,
vector_store: VectorStoreSvc,
_: CurrentAdmin,
) -> Any:
"""List all documents in a specific collection."""
return await vector_store.get_document_list(name)
@router.post("/search", response_model=RAGSearchResponse)
async def search_documents(
request: RAGSearchRequest,
retrieval_service: RetrievalSvc,
_: CurrentUser,
use_reranker: bool = Query(False, description="Whether to use reranking (if configured)"),
) -> Any:
"""Search for relevant document chunks. Supports multi-collection search."""
if request.collection_names and len(request.collection_names) > 1:
results = await retrieval_service.retrieve_multi(
query=request.query,
collection_names=request.collection_names,
limit=request.limit,
min_score=request.min_score,
use_reranker=use_reranker,
)
else:
collection = (
request.collection_names[0] if request.collection_names else request.collection_name
)
results = await retrieval_service.retrieve(
query=request.query,
collection_name=collection,
limit=request.limit,
min_score=request.min_score,
filter=request.filter or "",
use_reranker=use_reranker,
)
api_results = [RAGSearchResult(**hit.model_dump()) for hit in results]
return RAGSearchResponse(results=api_results)
@router.delete(
"/collections/{name}/documents/{document_id}",
status_code=status.HTTP_204_NO_CONTENT,
response_model=None,
)
async def delete_document(
name: str,
document_id: str,
ingestion_service: IngestionSvc,
_: CurrentAdmin,
) -> None:
"""Delete a specific document by its ID from a collection."""
success = await ingestion_service.remove_document(name, document_id)
if not success:
raise NotFoundError(
message="Document not found",
details={"collection": name, "document_id": document_id},
)
@router.post(
"/collections/{name}/ingest",
response_model=RAGIngestResponse,
response_model_exclude_none=True,
status_code=status.HTTP_202_ACCEPTED,
)
async def ingest_file(
name: str,
rag_doc_svc: RAGDocumentSvc,
vector_store: VectorStoreSvc,
_: CurrentAdmin,
file: UploadFile = File(...),
replace: bool = Query(False),
) -> Any:
"""Upload and queue a file for ingestion into a collection."""
data = await file.read()
return await rag_doc_svc.dispatch_upload(
collection_name=name,
file_data=data,
filename=file.filename or "unknown",
replace=replace,
vector_store=vector_store,
)
@router.get("/documents", response_model=RAGTrackedDocumentList)
async def list_rag_documents(
rag_doc_svc: RAGDocumentSvc,
_: CurrentAdmin,
collection_name: str | None = Query(None),
) -> Any:
"""List tracked RAG documents."""
return await rag_doc_svc.list_documents(collection_name)
@router.get("/documents/{doc_id}/download")
async def download_rag_document(
doc_id: str,
rag_doc_svc: RAGDocumentSvc,
_: CurrentAdmin,
) -> Any:
"""Download the original file."""
file_path, filename, mime_type = await rag_doc_svc.get_download_info(doc_id)
return FileResponse(path=file_path, filename=filename, media_type=mime_type)
@router.delete(
"/documents/{doc_id}",
status_code=status.HTTP_204_NO_CONTENT,
response_model=None,
)
async def delete_rag_document(
doc_id: str,
rag_doc_svc: RAGDocumentSvc,
ingestion_service: IngestionSvc,
_: CurrentAdmin,
) -> None:
"""Delete a document from SQL, vector store, and file storage."""
await rag_doc_svc.delete_document(doc_id, ingestion_service)
@router.post("/documents/{doc_id}/retry", response_model=RAGRetryResponse)
async def retry_ingestion(
doc_id: str,
rag_doc_svc: RAGDocumentSvc,
_: CurrentAdmin,
) -> Any:
"""Retry a failed document ingestion."""
doc = await rag_doc_svc.retry_ingestion(doc_id)
return RAGRetryResponse(id=str(doc.id), status="processing", message="Retry queued")
@router.get("/sync/logs", response_model=RAGSyncLogList)
async def list_sync_logs(
rag_sync_svc: RAGSyncSvc,
_: CurrentAdmin,
collection_name: str | None = Query(None),
limit: int = Query(20, ge=1, le=100),
) -> Any:
"""List sync operation logs."""
return await rag_sync_svc.list_sync_logs(collection_name=collection_name, limit=limit)
@router.post("/sync/local", response_model=RAGSyncResponse)
async def trigger_local_sync(
request: RAGSyncRequest,
rag_sync_svc: RAGSyncSvc,
_: CurrentAdmin,
) -> Any:
"""Trigger a local directory sync via background task."""
sync_log = await rag_sync_svc.start_local_sync(
collection_name=request.collection_name,
mode=request.mode,
path=request.path,
)
return RAGSyncResponse(
id=str(sync_log.id),
status="running",
message=f"Sync started for '{request.collection_name}' (mode={request.mode})",
)
@router.delete("/sync/{sync_id}", response_model=RAGMessageResponse)
async def cancel_sync(
sync_id: str,
rag_sync_svc: RAGSyncSvc,
_: CurrentAdmin,
) -> Any:
"""Cancel a running sync operation."""
await rag_sync_svc.cancel_sync(sync_id)
return RAGMessageResponse(message="Sync cancelled")
@router.get("/sync/sources", response_model=SyncSourceList)
async def list_sync_sources(
sync_source_svc: SyncSourceSvc,
_: CurrentAdmin,
) -> Any:
"""List all configured sync sources."""
return await sync_source_svc.list_sources()
@router.post(
"/sync/sources",
response_model=SyncSourceRead,
status_code=status.HTTP_201_CREATED,
)
async def create_sync_source(
data: SyncSourceCreate,
sync_source_svc: SyncSourceSvc,
_: CurrentAdmin,
) -> Any:
"""Create a new sync source configuration."""
return await sync_source_svc.create_source(data)
@router.patch("/sync/sources/{source_id}", response_model=SyncSourceRead)
async def update_sync_source(
source_id: str,
data: SyncSourceUpdate,
sync_source_svc: SyncSourceSvc,
_: CurrentAdmin,
) -> Any:
"""Update an existing sync source configuration."""
return await sync_source_svc.update_source(source_id, data)
@router.delete(
"/sync/sources/{source_id}",
status_code=status.HTTP_204_NO_CONTENT,
response_model=None,
)
async def delete_sync_source(
source_id: str,
sync_source_svc: SyncSourceSvc,
_: CurrentAdmin,
) -> None:
"""Delete a sync source configuration."""
await sync_source_svc.delete_source(source_id)
@router.post("/sync/sources/{source_id}/trigger", response_model=RAGSyncResponse)
async def trigger_sync_source(
source_id: str,
sync_source_svc: SyncSourceSvc,
_: CurrentAdmin,
) -> Any:
"""Trigger a manual sync for a configured source."""
sync_log = await sync_source_svc.trigger_sync(source_id)
return RAGSyncResponse(
id=str(sync_log.id),
status="running",
message=f"Sync triggered for source '{source_id}'",
)
@router.get("/sync/connectors", response_model=ConnectorList)
async def list_connectors(
sync_source_svc: SyncSourceSvc,
_: CurrentAdmin,
) -> Any:
"""List available sync connector types with their config schemas."""
return sync_source_svc.list_connectors()
@router.get("/status/stream", response_class=EventSourceResponse)
async def rag_status_stream(
rag_status_svc: RAGStatusSvc,
) -> AsyncIterable[ServerSentEvent]:
"""SSE endpoint for real-time RAG ingestion status updates.
Subscribes to the ``rag_status`` Redis pub/sub channel; the browser auto-reconnects
via the EventSource API.
"""
return rag_status_svc.stream_events()
backend/app/api/routes/v1/users.py
+133
View File
@@ -0,0 +1,133 @@
# ruff: noqa: I001 - Imports structured for Jinja2 template conditionals
"""User management routes."""
from typing import Any
from uuid import UUID
from fastapi import APIRouter, File, HTTPException, UploadFile, status
from fastapi.responses import FileResponse
from fastapi_pagination import Page
from app.api.deps import (
CurrentAdmin,
CurrentUser,
UserSvc,
)
from app.db.models.user import UserRole
from app.schemas.user import UserRead, UserUpdate
from app.services.file_storage import get_file_storage
router = APIRouter()
@router.get("/me", response_model=UserRead)
async def read_current_user(
current_user: CurrentUser,
) -> Any:
"""Get current user.
Returns the authenticated user's profile including their role.
"""
return current_user
@router.patch("/me", response_model=UserRead)
async def update_current_user(
user_in: UserUpdate,
current_user: CurrentUser,
user_service: UserSvc,
) -> Any:
"""Update current user.
Users can update their own profile (email, full_name).
Role changes require admin privileges.
"""
# Prevent non-admin users from changing their own role
if user_in.role is not None and not current_user.has_role(UserRole.ADMIN):
user_in.role = None
user = await user_service.update(current_user.id, user_in)
return user
@router.post("/me/avatar", response_model=UserRead)
async def upload_avatar(
user_service: UserSvc,
current_user: CurrentUser,
file: UploadFile = File(...),
) -> Any:
"""Upload or replace avatar image for the current user."""
data = await file.read()
try:
user = await user_service.update_avatar(
current_user.id, data, file.filename or "avatar.jpg", file.content_type or ""
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e)) from None
return user
@router.get("/avatar/{user_id}")
async def get_avatar(user_id: UUID, user_service: UserSvc) -> Any:
"""Get user avatar image."""
user = await user_service.get_by_id(user_id)
if not user.avatar_url:
raise HTTPException(status_code=404, detail="No avatar set")
storage = get_file_storage()
file_path = storage.get_full_path(user.avatar_url)
if not file_path:
raise HTTPException(status_code=404, detail="Avatar file not found")
return FileResponse(path=file_path, media_type="image/jpeg")
@router.get("", response_model=Page[UserRead])
async def read_users(
user_service: UserSvc,
_: CurrentAdmin,
) -> Any:
"""Get all users (admin only)."""
return await user_service.list_paginated()
@router.get("/{user_id}", response_model=UserRead)
async def read_user(
user_id: UUID,
user_service: UserSvc,
_: CurrentAdmin,
) -> Any:
"""Get user by ID (admin only).
Raises NotFoundError if user does not exist.
"""
user = await user_service.get_by_id(user_id)
return user
@router.patch("/{user_id}", response_model=UserRead)
async def update_user_by_id(
user_id: UUID,
user_in: UserUpdate,
user_service: UserSvc,
_: CurrentAdmin,
) -> Any:
"""Update user by ID (admin only).
Admins can update any user including their role.
Raises NotFoundError if user does not exist.
"""
user = await user_service.update(user_id, user_in)
return user
@router.delete("/{user_id}", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
async def delete_user_by_id(
user_id: UUID,
user_service: UserSvc,
_: CurrentAdmin,
) -> None:
"""Delete user by ID (admin only).
Raises NotFoundError if user does not exist.
"""
await user_service.delete(user_id)
backend/app/api/routes/v1/webhooks.py
+176
View File
@@ -0,0 +1,176 @@
"""Webhook management routes."""
from typing import Any
from uuid import UUID
from fastapi import APIRouter, Query, status
from app.api.deps import CurrentAdmin, WebhookSvc
from app.schemas.webhook import (
WebhookCreate,
WebhookDeliveryListResponse,
WebhookDeliveryRead,
WebhookListResponse,
WebhookRead,
WebhookSecretResponse,
WebhookTestResponse,
WebhookUpdate,
)
router = APIRouter()
@router.post("", response_model=WebhookRead, status_code=status.HTTP_201_CREATED)
async def create_webhook(
data: WebhookCreate,
webhook_service: WebhookSvc,
current_user: CurrentAdmin,
) -> Any:
"""Create a new webhook subscription."""
webhook = await webhook_service.create_webhook(
data,
user_id=current_user.id,
)
return WebhookRead(
id=webhook.id,
name=webhook.name,
url=webhook.url,
events=webhook.events,
is_active=webhook.is_active,
description=webhook.description,
created_at=webhook.created_at,
updated_at=webhook.updated_at,
)
@router.get("", response_model=WebhookListResponse)
async def list_webhooks(
webhook_service: WebhookSvc,
current_user: CurrentAdmin,
skip: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=100),
) -> Any:
"""List all webhooks."""
webhooks, total = await webhook_service.list_webhooks(
user_id=current_user.id,
skip=skip,
limit=limit,
)
return WebhookListResponse(
items=[
WebhookRead(
id=w.id,
name=w.name,
url=w.url,
events=w.events,
is_active=w.is_active,
description=w.description,
created_at=w.created_at,
updated_at=w.updated_at,
)
for w in webhooks
],
total=total,
)
@router.get("/{webhook_id}", response_model=WebhookRead)
async def get_webhook(
webhook_id: UUID,
webhook_service: WebhookSvc,
_: CurrentAdmin,
) -> Any:
"""Get a webhook by ID."""
webhook = await webhook_service.get_webhook(webhook_id)
return WebhookRead(
id=webhook.id,
name=webhook.name,
url=webhook.url,
events=webhook.events,
is_active=webhook.is_active,
description=webhook.description,
created_at=webhook.created_at,
updated_at=webhook.updated_at,
)
@router.patch("/{webhook_id}", response_model=WebhookRead)
async def update_webhook(
webhook_id: UUID,
data: WebhookUpdate,
webhook_service: WebhookSvc,
_: CurrentAdmin,
) -> Any:
"""Update a webhook."""
webhook = await webhook_service.update_webhook(webhook_id, data)
return WebhookRead(
id=webhook.id,
name=webhook.name,
url=webhook.url,
events=webhook.events,
is_active=webhook.is_active,
description=webhook.description,
created_at=webhook.created_at,
updated_at=webhook.updated_at,
)
@router.delete("/{webhook_id}", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
async def delete_webhook(
webhook_id: UUID,
webhook_service: WebhookSvc,
_: CurrentAdmin,
) -> None:
"""Delete a webhook."""
await webhook_service.delete_webhook(webhook_id)
@router.post("/{webhook_id}/test", response_model=WebhookTestResponse)
async def test_webhook(
webhook_id: UUID,
webhook_service: WebhookSvc,
_: CurrentAdmin,
) -> Any:
"""Send a test event to the webhook."""
result = await webhook_service.test_webhook(webhook_id)
return WebhookTestResponse(**result)
@router.post("/{webhook_id}/regenerate-secret", response_model=WebhookSecretResponse)
async def regenerate_webhook_secret(
webhook_id: UUID,
webhook_service: WebhookSvc,
_: CurrentAdmin,
) -> Any:
"""Regenerate the webhook secret."""
new_secret = await webhook_service.regenerate_secret(webhook_id)
return WebhookSecretResponse(secret=new_secret)
@router.get("/{webhook_id}/deliveries", response_model=WebhookDeliveryListResponse)
async def list_webhook_deliveries(
webhook_id: UUID,
webhook_service: WebhookSvc,
_: CurrentAdmin,
skip: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=100),
) -> Any:
"""Get delivery history for a webhook."""
deliveries, total = await webhook_service.get_deliveries(webhook_id, skip=skip, limit=limit)
return WebhookDeliveryListResponse(
items=[
WebhookDeliveryRead(
id=d.id,
webhook_id=d.webhook_id,
event_type=d.event_type,
response_status=d.response_status,
error_message=d.error_message,
attempt_count=d.attempt_count,
success=d.success,
created_at=d.created_at,
delivered_at=d.delivered_at,
)
for d in deliveries
],
total=total,
)
backend/app/api/versioning.py
+219
View File
@@ -0,0 +1,219 @@
"""API versioning utilities and deprecation handling.
This module provides tools for managing API version deprecation:
- Deprecation middleware for entire API versions
- Deprecation decorator for individual endpoints
- RFC 8594 compliant deprecation headers
"""
import logging
from collections.abc import Callable
from datetime import datetime
from functools import wraps
from typing import Any
from fastapi import Request, Response
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from starlette.types import ASGIApp
logger = logging.getLogger(__name__)
class VersionDeprecationMiddleware(BaseHTTPMiddleware):
"""Middleware to add deprecation headers for deprecated API versions.
Adds RFC 8594 compliant headers:
- Deprecation: Indicates the version is deprecated
- Sunset: Indicates when the version will be removed
- Link: Points to migration documentation
Usage in main.py:
app.add_middleware(
VersionDeprecationMiddleware,
deprecated_versions={"v1": {"sunset": "2025-06-01", "link": "/docs/migration/v2"}},
)
"""
def __init__(
self,
app: ASGIApp,
deprecated_versions: dict[str, dict[str, str]] | None = None,
) -> None:
"""Initialize the middleware.
Args:
app: The ASGI application.
deprecated_versions: Dict mapping version prefixes to deprecation info.
Each entry should have:
- sunset: ISO date string when version will be removed (optional)
- link: URL to migration documentation (optional)
- message: Custom deprecation message (optional)
Example:
{
"v1": {
"sunset": "2025-06-01",
"link": "https://api.example.com/docs/migration/v2",
"message": "Please migrate to API v2",
}
}
"""
super().__init__(app)
self.deprecated_versions = deprecated_versions or {}
async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
"""Process the request and add deprecation headers if needed."""
response = await call_next(request)
# Check if request path matches a deprecated version
path = request.url.path
for version, info in self.deprecated_versions.items():
if f"/api/{version}/" in path or path.endswith(f"/api/{version}"):
self._add_deprecation_headers(response, version, info)
self._log_deprecated_usage(request, version)
break
return response
def _add_deprecation_headers(
self, response: Response, version: str, info: dict[str, str]
) -> None:
"""Add RFC 8594 deprecation headers to the response."""
# Deprecation header - indicates the API is deprecated
response.headers["Deprecation"] = "true"
# Sunset header - when the API will be removed
if sunset := info.get("sunset"):
# Convert to HTTP date format
sunset_date = datetime.fromisoformat(sunset)
response.headers["Sunset"] = sunset_date.strftime("%a, %d %b %Y %H:%M:%S GMT")
# Link header - documentation for migration
if link := info.get("link"):
response.headers["Link"] = f'<{link}>; rel="deprecation"'
# Custom warning header
message = info.get("message", f"API {version} is deprecated")
response.headers["X-API-Deprecation-Warning"] = message
def _log_deprecated_usage(self, request: Request, version: str) -> None:
"""Log usage of deprecated API version for monitoring."""
logger.warning(
"Deprecated API version accessed: %s %s %s",
version,
request.method,
request.url.path,
)
def deprecated(
sunset: str | None = None,
message: str | None = None,
link: str | None = None,
) -> Callable[..., Any]:
"""Decorator to mark an endpoint as deprecated.
Adds deprecation headers to responses from the decorated endpoint.
Use this for deprecating individual endpoints within an active API version.
Args:
sunset: ISO date string when endpoint will be removed.
message: Custom deprecation message.
link: URL to migration documentation.
Usage:
@router.get("/old-endpoint")
@deprecated(
sunset="2025-06-01",
message="Use /new-endpoint instead",
link="/docs/migration",
)
async def old_endpoint():
...
"""
def decorator(func: Callable[..., Any]) -> Callable[..., Any]:
@wraps(func)
async def wrapper(*args: Any, **kwargs: Any) -> Any:
# Get the response from the endpoint
result = await func(*args, **kwargs)
# Find Response object in args (FastAPI injects it)
response = None
for arg in args:
if isinstance(arg, Response):
response = arg
break
for value in kwargs.values():
if isinstance(value, Response):
response = value
break
# If we have a Response object, add headers
if response:
response.headers["Deprecation"] = "true"
if sunset:
sunset_date = datetime.fromisoformat(sunset)
response.headers["Sunset"] = sunset_date.strftime("%a, %d %b %Y %H:%M:%S GMT")
if link:
response.headers["Link"] = f'<{link}>; rel="deprecation"'
if message:
response.headers["X-API-Deprecation-Warning"] = message
return result
# Add deprecation info to OpenAPI schema
wrapper.__doc__ = (
f"{func.__doc__ or ''}\n\n"
f"**DEPRECATED**"
f"{f': {message}' if message else ''}"
f"{f' (Sunset: {sunset})' if sunset else ''}"
)
return wrapper
return decorator
# Example usage documentation
"""
## Adding a New API Version
1. Create a new version folder:
```
app/api/routes/v2/
├── __init__.py
├── health.py
├── auth.py
└── ...
```
2. Create the v2 router in `v2/__init__.py`:
```python
from fastapi import APIRouter
v2_router = APIRouter()
# Include routes...
```
3. Add the v2 router in `app/api/router.py`:
```python
from app.api.routes.v2 import v2_router
api_router.include_router(v1_router, prefix="/v1")
api_router.include_router(v2_router, prefix="/v2")
```
4. Mark v1 as deprecated in `main.py`:
```python
app.add_middleware(
VersionDeprecationMiddleware,
deprecated_versions={
"v1": {
"sunset": "2025-12-31",
"link": "/docs/migration/v2",
"message": "Please migrate to API v2",
}
},
)
```
"""
backend/app/clients/__init__.py
+10
View File
@@ -0,0 +1,10 @@
"""External service clients.
This module contains thin wrappers around external services like Redis.
"""
from app.clients.redis import RedisClient
__all__ = [
"RedisClient",
]
backend/app/clients/redis.py
+84
View File
@@ -0,0 +1,84 @@
"""Redis client wrapper.
Provides a class-based Redis client for connection management and operations.
"""
from redis import asyncio as aioredis
from app.core.config import settings
class RedisClient:
"""Redis client wrapper for connection lifecycle management.
Usage in FastAPI lifespan:
async with contextmanager():
redis = RedisClient(settings.REDIS_URL)
await redis.connect()
yield {"redis": redis}
await redis.close()
"""
def __init__(self, url: str | None = None):
self.url = url or settings.REDIS_URL
self.client: aioredis.Redis | None = None
async def connect(self) -> None:
"""Connect to Redis server."""
self.client = aioredis.from_url( # type: ignore[no-untyped-call]
self.url,
encoding="utf-8",
decode_responses=True,
)
async def close(self) -> None:
"""Close Redis connection."""
if self.client:
await self.client.close()
self.client = None
async def get(self, key: str) -> str | None:
"""Get a value by key."""
if not self.client:
raise RuntimeError("Redis client not connected")
return await self.client.get(key) # type: ignore[no-any-return]
async def set(
self,
key: str,
value: str,
ttl: int | None = None,
) -> None:
"""Set a value with optional TTL (in seconds)."""
if not self.client:
raise RuntimeError("Redis client not connected")
await self.client.set(key, value, ex=ttl)
async def delete(self, key: str) -> int:
"""Delete a key. Returns number of keys deleted."""
if not self.client:
raise RuntimeError("Redis client not connected")
return await self.client.delete(key) # type: ignore[no-any-return]
async def exists(self, key: str) -> bool:
"""Check if key exists."""
if not self.client:
raise RuntimeError("Redis client not connected")
return bool(await self.client.exists(key))
async def ping(self) -> bool:
"""Ping Redis server. Returns True if connected."""
if not self.client:
return False
try:
await self.client.ping()
return True
except Exception:
return False
@property
def raw(self) -> aioredis.Redis:
"""Access the underlying aioredis client for advanced operations."""
if not self.client:
raise RuntimeError("Redis client not connected")
return self.client
backend/app/commands/__init__.py
+120
View File
@@ -0,0 +1,120 @@
"""
Custom commands system with auto-discovery.
This module provides a Django-like custom commands system for FastAPI + Click.
Commands are auto-discovered from this package and registered to the CLI.
Usage:
# In app/commands/my_command.py
from app.commands import command
import click
@command("my-command", help="Description of my command")
@click.option("--option", "-o", help="Some option")
def my_command(option: str):
click.echo(f"Running with {option}")
# Then use it:
# project cmd my-command --option value
"""
import importlib
import pkgutil
from collections.abc import Callable
from pathlib import Path
from typing import Any
import click
# Registry for custom commands
_commands: list[click.Command] = []
_discovered = False
def command(name: str | None = None, **kwargs: Any) -> Callable[..., Any]:
"""
Decorator to register a custom command.
Args:
name: Command name (defaults to function name with underscores replaced by hyphens)
**kwargs: Additional arguments passed to click.command()
Example:
@command("seed", help="Seed database with initial data")
@click.option("--count", "-c", default=10)
def seed_data(count: int):
click.echo(f"Seeding {count} records...")
"""
def decorator(func: Callable[..., Any]) -> click.Command:
cmd_name = name or func.__name__.replace("_", "-")
cmd: click.Command = click.command(cmd_name, **kwargs)(func) # type: ignore[no-untyped-call]
_commands.append(cmd)
return cmd
return decorator
def discover_commands() -> list[click.Command]:
"""
Auto-discover all commands in this package.
Imports all modules in the app.commands package (except those starting with _)
which triggers the @command decorator to register them.
Returns:
List of discovered click.Command objects
"""
global _discovered
if _discovered:
return _commands
package_dir = Path(__file__).parent
for _, module_name, _ in pkgutil.iter_modules([str(package_dir)]):
if module_name.startswith("_"):
continue
try:
importlib.import_module(f"app.commands.{module_name}")
except ImportError as e:
click.secho(
f"Warning: Failed to import command module '{module_name}': {e}", fg="yellow"
)
_discovered = True
return _commands
def register_commands(cli: click.Group) -> None:
"""
Register all discovered commands to a CLI group.
Args:
cli: The click.Group to add commands to
"""
commands = discover_commands()
for cmd in commands:
cli.add_command(cmd)
def success(message: str) -> None:
"""Print success message in green."""
click.secho(message, fg="green")
def error(message: str) -> None:
"""Print error message in red."""
click.secho(message, fg="red")
def warning(message: str) -> None:
"""Print warning message in yellow."""
click.secho(message, fg="yellow")
def info(message: str) -> None:
"""Print info message."""
click.echo(message)
backend/app/commands/cleanup.py
+45
View File
@@ -0,0 +1,45 @@
"""
Cleanup old or stale data from the database.
This command is useful for maintenance tasks.
"""
import asyncio
from datetime import UTC, datetime, timedelta
import click
from app.commands import command, info, success, warning
@command("cleanup", help="Clean up old data from the database")
@click.option("--days", "-d", default=90, type=int, help="Delete records older than N days")
@click.option("--dry-run", is_flag=True, help="Show what would be deleted without making changes")
@click.option("--force", "-f", is_flag=True, help="Skip confirmation prompt")
def cleanup(days: int, dry_run: bool, force: bool) -> None:
"""
Remove old records from the database.
Example:
project cmd cleanup --days 90
project cmd cleanup --days 30 --dry-run
project cmd cleanup --days 7 --force
"""
cutoff_date = datetime.now(UTC) - timedelta(days=days)
if dry_run:
info(f"[DRY RUN] Would delete records older than {cutoff_date}")
return
if not force and not click.confirm(
f"Delete all records older than {days} days ({cutoff_date})?"
):
warning("Aborted.")
return
async def _cleanup() -> None:
info(f"Cleaning up records older than {cutoff_date}...")
total_deleted = 0
success(f"Done. Total deleted: {total_deleted} rows.")
asyncio.run(_cleanup())
backend/app/commands/example.py
+28
View File
@@ -0,0 +1,28 @@
"""
Example custom command.
This is a template showing how to create custom CLI commands.
Copy this file and modify it to create your own commands.
"""
import click
from app.commands import command, info, success
@command("hello", help="Example command that greets the user")
@click.option("--name", "-n", default="World", help="Name to greet")
@click.option("--count", "-c", default=1, type=int, help="Number of greetings")
def hello(name: str, count: int) -> None:
"""
Greet someone multiple times.
Example:
project cmd hello --name Alice --count 3
"""
info(f"Greeting {name} {count} time(s)...")
for i in range(count):
click.echo(f" [{i + 1}] Hello, {name}!")
success("Done!")
backend/app/commands/rag.py
+638
View File
@@ -0,0 +1,638 @@
"""
RAG CLI commands for document management and retrieval.
Commands:
rag-collections - List collections with stats
rag-ingest - Ingest file/directory
rag-search - Search knowledge base
rag-drop - Drop collection
rag-stats - Overall RAG system statistics
rag-sources - List configured sync sources
rag-source-add - Add a new sync source
rag-source-remove - Remove a sync source
rag-source-sync - Trigger sync for a source (or all)
"""
import asyncio
from pathlib import Path
import click
from app.commands import command, error, info, success, warning
from app.services.rag.config import DocumentExtensions, RAGSettings
from app.services.rag.documents import DocumentProcessor
from app.services.rag.embeddings import EmbeddingService
from app.services.rag.ingestion import IngestionService
from app.services.rag.retrieval import RetrievalService
from app.services.rag.vectorstore import BaseVectorStore, QdrantVectorStore
def get_rag_services() -> tuple[
RAGSettings, BaseVectorStore, DocumentProcessor, RetrievalService, IngestionService
]:
"""Initialize RAG services for CLI usage.
Creates and returns all necessary RAG service components:
- Settings (RAG configuration)
- Vector store (Milvus)
- Document processor
- Retrieval service
- Ingestion service
Returns:
Tuple of (settings, vector_store, processor, retrieval, ingestion) services.
"""
settings = RAGSettings()
embedder = EmbeddingService(settings=settings)
vector_store = QdrantVectorStore(settings=settings, embedding_service=embedder)
processor = DocumentProcessor(settings=settings)
retrieval = RetrievalService(vector_store=vector_store, settings=settings)
ingestion = IngestionService(processor=processor, vector_store=vector_store)
return settings, vector_store, processor, retrieval, ingestion
async def list_collections_async(vector_store: BaseVectorStore) -> None:
"""List all collections with their stats.
Args:
vector_store: The Milvus vector store to query.
"""
collection_names = await vector_store.list_collections()
if not collection_names:
info("No collections found.")
return
click.echo(f"\nFound {len(collection_names)} collection(s):\n")
for name in collection_names:
try:
info_obj = await vector_store.get_collection_info(name)
click.echo(f" {name}")
click.echo(f" Vectors: {info_obj.total_vectors:,}")
click.echo(f" Dimension: {info_obj.dim}")
click.echo(f" Status: {info_obj.indexing_status}")
click.echo()
except Exception as e:
warning(f"Could not get info for '{name}': {e}")
@command("rag-collections", help="List collections with stats")
def rag_collections() -> None:
"""List all available collections in the vector store with their statistics."""
_, vector_store, _, _, _ = get_rag_services()
asyncio.run(list_collections_async(vector_store))
async def ingest_path_async(
path: str,
collection: str,
recursive: bool,
vector_store: BaseVectorStore,
processor: DocumentProcessor,
ingestion: IngestionService,
replace: bool = True,
sync_mode: str = "full",
) -> None:
"""Ingest files from a path (file or directory).
Args:
path: Path to a file or directory to ingest.
collection: Target collection name.
recursive: Whether to recursively process directories.
vector_store: The Milvus vector store.
processor: Document processor for parsing files.
ingestion: Ingestion service for storing documents.
"""
target_path = Path(path).resolve()
if not target_path.exists():
error(f"Path does not exist: {target_path}")
return
# Collect files to process
if target_path.is_file():
files = [target_path]
elif target_path.is_dir():
if recursive:
files = list(target_path.rglob("*"))
files = [f for f in files if f.is_file() and not f.name.startswith(".")]
else:
files = list(target_path.iterdir())
files = [f for f in files if f.is_file() and not f.name.startswith(".")]
else:
error(f"Invalid path: {target_path}")
return
if not files:
warning("No files found to ingest.")
return
# Filter by allowed extensions
allowed_extensions = {ext.value for ext in DocumentExtensions}
files = [f for f in files if f.suffix.lower() in allowed_extensions]
if not files:
warning(f"No supported files found. Allowed: {', '.join(allowed_extensions)}")
return
import hashlib
from tqdm import tqdm
from app.db.session import get_db_context
from app.services.rag_document import RAGDocumentService
from app.services.rag_sync import RAGSyncService
info(f"Syncing {len(files)} file(s) into '{collection}' (mode={sync_mode})...")
success_count = 0
error_count = 0
replaced_count = 0
skipped_count = 0
# Create SyncLog
async with get_db_context() as db:
sync_log = await RAGSyncService(db).create_sync_log(
source="local", collection_name=collection, mode=sync_mode
)
sync_log_id = str(sync_log.id)
with tqdm(files, unit="file", desc="Syncing", ncols=80) as pbar:
for filepath in pbar:
pbar.set_postfix_str(filepath.name[:30], refresh=True)
# Sync mode checks
source_path = str(filepath.resolve())
if sync_mode in ("new_only", "update_only"):
existing_id: str | None = await ingestion.find_existing(collection, source_path)
if sync_mode == "new_only":
if existing_id:
# File exists — check if content changed via hash
file_hash: str = hashlib.sha256(filepath.read_bytes()).hexdigest()
existing_hash: str | None = await ingestion.get_existing_hash(
collection, source_path
)
if existing_hash and file_hash == existing_hash:
skipped_count += 1
continue
# Hash changed — will re-ingest below
elif sync_mode == "update_only":
if not existing_id:
# Not in collection — skip (update_only ignores new files)
skipped_count += 1
continue
file_hash = hashlib.sha256(filepath.read_bytes()).hexdigest()
existing_hash = await ingestion.get_existing_hash(collection, source_path)
if existing_hash and file_hash == existing_hash:
skipped_count += 1
continue
# Create RAGDocument record in SQL
async with get_db_context() as db:
rag_doc = await RAGDocumentService(db).create_document(
collection_name=collection,
filename=filepath.name,
filesize=filepath.stat().st_size,
filetype=filepath.suffix.lstrip(".").lower(),
)
doc_id = str(rag_doc.id)
try:
result = await ingestion.ingest_file(
filepath=filepath, collection_name=collection, replace=replace
)
if result.status.value == "done":
success_count += 1
if result.message and "replaced" in result.message:
replaced_count += 1
async with get_db_context() as db:
await RAGDocumentService(db).complete_ingestion(
doc_id, vector_document_id=result.document_id
)
else:
error_count += 1
tqdm.write(f" ✗ {filepath.name}: {result.error_message}")
async with get_db_context() as db:
await RAGDocumentService(db).fail_ingestion(
doc_id, error_message=result.error_message or "Unknown error"
)
except Exception as e:
error_count += 1
tqdm.write(f" ✗ {filepath.name}: {e!s}")
async with get_db_context() as db:
await RAGDocumentService(db).fail_ingestion(doc_id, error_message=str(e))
# Update SyncLog
async with get_db_context() as db:
await RAGSyncService(db).complete_sync(
sync_log_id,
status="done" if error_count == 0 else "error",
total_files=len(files),
ingested=success_count - replaced_count,
updated=replaced_count,
skipped=skipped_count,
failed=error_count,
)
click.echo()
msg = f"Done: {success_count} ingested"
if replaced_count > 0:
msg += f" ({replaced_count} updated)"
if skipped_count > 0:
msg += f", {skipped_count} skipped"
success(msg)
if error_count > 0:
error(f"Failed: {error_count} files")
@command("rag-ingest", help="Ingest file/directory into knowledge base")
@click.argument("path", type=click.Path(exists=True))
@click.option(
"--collection",
"-c",
default="documents",
help="Collection name (default: documents)",
)
@click.option(
"--recursive/--no-recursive",
"-r",
default=False,
help="Recursively process directories (default: False)",
)
@click.option(
"--replace/--no-replace",
default=True,
help="Replace existing documents with same source path (default: True)",
)
@click.option(
"--sync-mode",
type=click.Choice(["full", "new_only", "update_only"]),
default="full",
help="Sync mode: full (replace all), new_only (add new + update changed), update_only (only update changed, skip new)",
)
def rag_ingest(path: str, collection: str, recursive: bool, replace: bool, sync_mode: str) -> None:
"""
Ingest a file or directory into the knowledge base.
PATH: Path to a file or directory to ingest.
Example:
project cmd rag-ingest ./docs
project cmd rag-ingest ./docs --sync-mode new_only
project cmd rag-ingest ./docs --sync-mode update_only
"""
_, vector_store, processor, _, ingestion = get_rag_services()
asyncio.run(
ingest_path_async(
path, collection, recursive, vector_store, processor, ingestion, replace, sync_mode
)
)
async def search_async(
query: str,
collection: str,
top_k: int,
retrieval: RetrievalService,
) -> None:
"""Search the knowledge base.
Args:
query: The search query.
collection: Target collection name.
top_k: Number of results to return.
retrieval: Retrieval service for searching.
"""
info(f"Searching collection '{collection}' for: \"{query}\"")
click.echo()
results = await retrieval.retrieve(
query=query,
collection_name=collection,
limit=top_k,
)
if not results:
warning("No results found.")
return
for i, result in enumerate(results, 1):
click.echo(f"--- Result {i} (score: {result.score:.4f}) ---")
# Show source info if available
if result.metadata:
filename = result.metadata.get("filename", "Unknown")
page_num = result.metadata.get("page_num", "?")
click.echo(f"Source: {filename} (page {page_num})")
# Show content (truncated)
content = result.content[:500]
if len(result.content) > 500:
content += "..."
click.echo(content)
click.echo()
@command("rag-search", help="Search knowledge base")
@click.argument("query")
@click.option(
"--collection",
"-c",
default="documents",
help="Collection name (default: documents)",
)
@click.option(
"--top-k",
"-k",
default=4,
type=int,
help="Number of results to return (default: 4)",
)
def rag_search(query: str, collection: str, top_k: int) -> None:
"""
Search the knowledge base for relevant content.
QUERY: The search query.
Example:
project cmd rag-search "what is fastapi"
project cmd rag-search "deployment guide" --collection docs --top-k 10
"""
_, _, _, retrieval, _ = get_rag_services()
asyncio.run(search_async(query, collection, top_k, retrieval))
async def drop_collection_async(collection: str, yes: bool, vector_store: BaseVectorStore) -> None:
"""Drop a collection.
Args:
collection: Name of the collection to drop.
yes: Whether to skip confirmation prompt.
vector_store: The Milvus vector store.
"""
if not yes:
click.confirm(
f"Are you sure you want to drop collection '{collection}'? This cannot be undone.",
abort=True,
)
try:
await vector_store.delete_collection(collection)
success(f"Collection '{collection}' dropped successfully.")
except Exception as e:
error(f"Failed to drop collection: {e}")
@command("rag-drop", help="Drop a collection")
@click.argument("collection")
@click.option(
"--yes",
"-y",
is_flag=True,
help="Skip confirmation prompt",
)
def rag_drop(collection: str, yes: bool) -> None:
"""
Drop a collection and all its data.
COLLECTION: Name of the collection to drop.
Example:
project cmd rag-drop my_collection
project cmd rag-drop my_collection --yes
"""
_, vector_store, _, _, _ = get_rag_services()
asyncio.run(drop_collection_async(collection, yes, vector_store))
@command("rag-stats", help="Show overall RAG system statistics")
def rag_stats() -> None:
"""Display overall RAG system statistics."""
settings, vector_store, _, _, _ = get_rag_services()
asyncio.run(stats_async(settings, vector_store))
async def stats_async(settings: RAGSettings, vector_store: BaseVectorStore) -> None:
"""Show RAG system statistics.
Args:
settings: RAG configuration settings.
vector_store: The Milvus vector store.
"""
click.echo("RAG System Statistics")
click.echo("=" * 40)
# Collection info
try:
collection_names = await vector_store.list_collections()
click.echo(f"\nCollections: {len(collection_names)}")
except Exception as e:
warning(f"Could not list collections: {e}")
collection_names = []
# Configuration
click.echo("\nConfiguration:")
click.echo(f" Embedding model: {settings.embeddings_config.model}")
click.echo(f" Embedding dimension: {settings.embeddings_config.dim}")
click.echo(f" Chunk size: {settings.chunk_size}")
click.echo(f" Chunk overlap: {settings.chunk_overlap}")
click.echo(f" Parser method: {settings.pdf_parser.method}")
# Per-collection stats
if collection_names:
click.echo("\nCollection Details:")
total_vectors = 0
for name in collection_names:
try:
info_obj = await vector_store.get_collection_info(name)
click.echo(f" {name}:")
click.echo(f" Vectors: {info_obj.total_vectors:,}")
total_vectors += info_obj.total_vectors
except Exception:
click.echo(f" {name}: Error getting info")
click.echo(f"\nTotal vectors: {total_vectors:,}")
click.echo()
@command("rag-sources", help="List configured sync sources")
def rag_sources() -> None:
"""List all configured sync sources with their status."""
from app.db.session import get_db_context
async def _list() -> None:
async with get_db_context() as db:
from app.services.sync_source import SyncSourceService
svc = SyncSourceService(db)
sources = await svc.list_sources()
if not sources:
info("No sync sources configured.")
return
click.echo(f"\nFound {len(sources)} sync source(s):\n")
for s in sources:
status_str = s.last_sync_status or "never"
active_str = "active" if s.is_active else "inactive"
click.echo(f" [{active_str}] {s.name} (id={s.id})")
click.echo(f" Type: {s.connector_type}")
click.echo(f" Collection: {s.collection_name}")
click.echo(f" Sync mode: {s.sync_mode}")
if s.schedule_minutes:
click.echo(f" Schedule: every {s.schedule_minutes} min")
else:
click.echo(" Schedule: manual")
click.echo(f" Last sync: {status_str}")
if s.last_error:
click.echo(f" Last error: {s.last_error}")
click.echo()
asyncio.run(_list())
@command("rag-source-add", help="Add a new sync source")
@click.option("--name", required=True, help="Source name")
@click.option("--type", "connector_type", required=True, help="Connector type (e.g. gdrive, s3)")
@click.option("--collection", required=True, help="Target collection name")
@click.option("--config", "config_json", required=True, help="Config JSON string")
@click.option(
"--sync-mode",
default="new_only",
type=click.Choice(["full", "new_only", "update_only"]),
help="Sync mode",
)
@click.option(
"--schedule",
"schedule_minutes",
type=int,
default=0,
help="Schedule interval in minutes (0=manual)",
)
def rag_source_add(
name: str,
connector_type: str,
collection: str,
config_json: str,
sync_mode: str,
schedule_minutes: int,
) -> None:
"""
Add a new sync source configuration.
Example:
project cmd rag-source-add --name "My Drive" --type gdrive --collection docs \\
--config '{"folder_id": "abc123"}' --sync-mode new_only
"""
import json as _json
try:
config_dict = _json.loads(config_json)
except _json.JSONDecodeError as e:
error(f"Invalid JSON config: {e}")
return
from app.schemas.sync_source import SyncSourceCreate
data = SyncSourceCreate(
name=name,
connector_type=connector_type,
collection_name=collection,
config=config_dict,
sync_mode=sync_mode,
schedule_minutes=schedule_minutes if schedule_minutes > 0 else None,
)
from app.db.session import get_db_context
async def _create() -> None:
async with get_db_context() as db:
from app.services.sync_source import SyncSourceService
svc = SyncSourceService(db)
try:
source = await svc.create_source(data)
success(f"Sync source created: {source.name} (id={source.id})")
except ValueError as e:
error(f"Failed to create source: {e}")
asyncio.run(_create())
@command("rag-source-remove", help="Remove a sync source")
@click.argument("source_id")
@click.option("--yes", "-y", is_flag=True, help="Skip confirmation prompt")
def rag_source_remove(source_id: str, yes: bool) -> None:
"""
Remove a sync source configuration.
SOURCE_ID: The ID of the sync source to remove.
Example:
project cmd rag-source-remove abc-123-def
"""
if not yes:
click.confirm(f"Are you sure you want to remove sync source '{source_id}'?", abort=True)
from app.db.session import get_db_context
async def _remove() -> None:
async with get_db_context() as db:
from app.services.sync_source import SyncSourceService
svc = SyncSourceService(db)
try:
await svc.delete_source(source_id)
success(f"Sync source '{source_id}' removed.")
except Exception as e:
error(f"Failed to remove source: {e}")
asyncio.run(_remove())
@command("rag-source-sync", help="Trigger sync for a source")
@click.argument("source_id", required=False)
@click.option("--all", "sync_all", is_flag=True, help="Sync all active sources")
def rag_source_sync(source_id: str | None, sync_all: bool) -> None:
"""
Trigger sync for a configured source (or all active sources).
SOURCE_ID: The ID of the sync source to sync (optional if --all).
Example:
project cmd rag-source-sync abc-123-def
project cmd rag-source-sync --all
"""
if not source_id and not sync_all:
error("Provide a SOURCE_ID or use --all to sync all active sources.")
return
from app.db.session import get_db_context
async def _sync() -> None:
async with get_db_context() as db:
from app.services.sync_source import SyncSourceService
svc = SyncSourceService(db)
if sync_all:
sources = await svc.list_sources(is_active=True)
if not sources:
warning("No active sync sources found.")
return
info(f"Triggering sync for {len(sources)} active source(s)...")
for s in sources:
try:
log = await svc.trigger_sync(str(s.id))
success(f" {s.name}: sync started (log_id={log.id})")
except Exception as e:
error(f" {s.name}: failed - {e}")
else:
try:
assert source_id is not None
log = await svc.trigger_sync(source_id)
success(f"Sync triggered (log_id={log.id})")
except Exception as e:
error(f"Failed to trigger sync: {e}")
asyncio.run(_sync())
backend/app/commands/seed.py
+113
View File
@@ -0,0 +1,113 @@
# ruff: noqa: I001 - Imports structured for Jinja2 template conditionals
"""
Seed database with sample data.
This command is useful for development and testing.
Uses random data generation - install faker for better data:
uv add faker --group dev
"""
import asyncio
import random
import string
import click
from app.commands import command, info, success, warning
# Try to import Faker for better data generation
try:
from faker import Faker
fake = Faker()
HAS_FAKER = True
except ImportError:
HAS_FAKER = False
fake = None
def random_email() -> str:
"""Generate a random email address."""
if HAS_FAKER:
return str(fake.email())
random_str = "".join(random.choices(string.ascii_lowercase, k=8))
return f"{random_str}@example.com"
def random_name() -> str:
"""Generate a random full name."""
if HAS_FAKER:
return str(fake.name())
first_names = ["John", "Jane", "Bob", "Alice", "Charlie", "Diana", "Eve", "Frank"]
last_names = ["Smith", "Johnson", "Williams", "Brown", "Jones", "Garcia", "Miller", "Davis"]
return f"{random.choice(first_names)} {random.choice(last_names)}"
@command("seed", help="Seed database with sample data")
@click.option("--count", "-c", default=10, type=int, help="Number of records to create")
@click.option("--clear", is_flag=True, help="Clear existing data before seeding")
@click.option("--dry-run", is_flag=True, help="Show what would be created without making changes")
@click.option("--users/--no-users", default=True, help="Seed users (default: True)")
def seed(
count: int,
clear: bool,
dry_run: bool,
users: bool,
) -> None:
"""
Seed the database with sample data for development.
Example:
project cmd seed --count 50
project cmd seed --clear --count 100
project cmd seed --dry-run
project cmd seed --no-users # Skip user seeding
"""
if not HAS_FAKER:
warning(
"Faker not installed. Using basic random data. For better data: uv add faker --group dev"
)
if dry_run:
info(f"[DRY RUN] Would create {count} sample records per entity")
if clear:
info("[DRY RUN] Would clear existing data first")
if users:
info("[DRY RUN] Would create users")
return
from app.db.session import get_db_context
from app.services.user import UserService
from app.schemas.user import UserCreate
async def _seed() -> None:
async with get_db_context() as db:
user_svc = UserService(db)
created_counts = {}
if users:
if clear:
info("Clearing existing users (except admins)...")
await user_svc.delete_non_admins()
if await user_svc.has_any() and not clear:
info("Users already exist. Use --clear to replace them.")
else:
info(f"Creating {count} sample users...")
for _ in range(count):
await user_svc.register(
UserCreate(
email=random_email(),
password="password123",
full_name=random_name(),
)
)
created_counts["users"] = count
if created_counts:
summary = ", ".join(f"{v} {k}" for k, v in created_counts.items())
success(f"Created: {summary}")
else:
info("No records created.")
asyncio.run(_seed())
backend/app/core/__init__.py
+5
View File
@@ -0,0 +1,5 @@
"""Core application configuration and utilities."""
from .config import settings
__all__ = ["settings"]
backend/app/core/cache.py
+14
View File
@@ -0,0 +1,14 @@
"""Caching configuration using fastapi-cache2."""
from fastapi_cache import FastAPICache
from fastapi_cache.backends.redis import RedisBackend
from app.clients.redis import RedisClient
def setup_cache(redis: RedisClient) -> None:
"""Initialize FastAPI cache with Redis backend.
Uses the shared Redis client from lifespan state.
"""
FastAPICache.init(RedisBackend(redis.raw), prefix="agent_delta:cache:")
backend/app/core/config.py
+267
View File
@@ -0,0 +1,267 @@
"""Application configuration using Pydantic BaseSettings."""
# ruff: noqa: I001 - Imports structured for Jinja2 template conditionals
from pathlib import Path
from typing import Literal
from pydantic import computed_field, field_validator, ValidationInfo
from pydantic_settings import BaseSettings, SettingsConfigDict
def find_env_file() -> Path | None:
"""Find .env file in current or parent directories."""
current = Path.cwd()
for path in [current, current.parent]:
env_file = path / ".env"
if env_file.exists():
return env_file
return None
class Settings(BaseSettings):
"""Application settings."""
model_config = SettingsConfigDict(
env_file=find_env_file(),
env_ignore_empty=True,
extra="ignore",
)
# === Project ===
PROJECT_NAME: str = "agent_delta"
API_V1_STR: str = "/api/v1"
DEBUG: bool = False
ENVIRONMENT: Literal["development", "local", "staging", "production"] = "local"
TIMEZONE: str = "UTC" # IANA timezone (e.g. "UTC", "Europe/Warsaw", "America/New_York")
MODELS_CACHE_DIR: Path = Path("./models_cache")
MEDIA_DIR: Path = Path("./media")
MAX_UPLOAD_SIZE_MB: int = 50 # Max file upload size in MB
# Soft per-org storage cap surfaced on /billing — not enforced yet (5 GB).
STORAGE_SOFT_LIMIT_BYTES: int = 5 * 1024 * 1024 * 1024
# === Logfire ===
LOGFIRE_TOKEN: str | None = None
LOGFIRE_SERVICE_NAME: str = "agent_delta"
LOGFIRE_ENVIRONMENT: str = "development"
# === Database (PostgreSQL async) ===
POSTGRES_HOST: str = "localhost"
POSTGRES_PORT: int = 5432
POSTGRES_USER: str = "postgres"
POSTGRES_PASSWORD: str = ""
POSTGRES_DB: str = "agent_delta"
@computed_field # type: ignore[prop-decorator]
@property
def DATABASE_URL(self) -> str:
"""Build async PostgreSQL connection URL."""
return (
f"postgresql+asyncpg://{self.POSTGRES_USER}:{self.POSTGRES_PASSWORD}"
f"@{self.POSTGRES_HOST}:{self.POSTGRES_PORT}/{self.POSTGRES_DB}"
)
@computed_field # type: ignore[prop-decorator]
@property
def DATABASE_URL_SYNC(self) -> str:
"""Build sync PostgreSQL connection URL (for Alembic)."""
return (
f"postgresql://{self.POSTGRES_USER}:{self.POSTGRES_PASSWORD}"
f"@{self.POSTGRES_HOST}:{self.POSTGRES_PORT}/{self.POSTGRES_DB}"
)
# Pool configuration
DB_POOL_SIZE: int = 5
DB_MAX_OVERFLOW: int = 10
DB_POOL_TIMEOUT: int = 30
# === Auth (SECRET_KEY for JWT/Session/Admin) ===
SECRET_KEY: str = "change-me-in-production-use-openssl-rand-hex-32"
@field_validator("SECRET_KEY")
@classmethod
def validate_secret_key(cls, v: str, info: ValidationInfo) -> str:
"""Validate SECRET_KEY is secure in production."""
if len(v) < 32:
raise ValueError("SECRET_KEY must be at least 32 characters long")
# Get environment from values if available
env = info.data.get("ENVIRONMENT", "local") if info.data else "local"
if v == "change-me-in-production-use-openssl-rand-hex-32" and env == "production":
raise ValueError(
"SECRET_KEY must be changed in production! "
"Generate a secure key with: openssl rand -hex 32"
)
return v
# === JWT Settings ===
ACCESS_TOKEN_EXPIRE_MINUTES: int = 30 # 30 minutes
REFRESH_TOKEN_EXPIRE_MINUTES: int = 60 * 24 * 7 # 7 days
ALGORITHM: str = "HS256"
# Public URL of the frontend; used to build OAuth redirect targets and
# Stripe checkout/portal return URLs. Always declared (not gated) because
# the billing model_validator references it unconditionally.
FRONTEND_URL: str = "http://localhost:3000"
# === Auth (API Key) ===
API_KEY: str = "change-me-in-production"
API_KEY_HEADER: str = "X-API-Key"
@field_validator("API_KEY")
@classmethod
def validate_api_key(cls, v: str, info: ValidationInfo) -> str:
"""Validate API_KEY is set in production."""
env = info.data.get("ENVIRONMENT", "local") if info.data else "local"
if v == "change-me-in-production" and env == "production":
raise ValueError(
"API_KEY must be changed in production! "
"Generate a secure key with: openssl rand -hex 32"
)
return v
# === Redis ===
REDIS_HOST: str = "localhost"
REDIS_PORT: int = 6379
REDIS_PASSWORD: str | None = None
REDIS_DB: int = 0
@computed_field # type: ignore[prop-decorator]
@property
def REDIS_URL(self) -> str:
"""Build Redis connection URL."""
if self.REDIS_PASSWORD:
return f"redis://:{self.REDIS_PASSWORD}@{self.REDIS_HOST}:{self.REDIS_PORT}/{self.REDIS_DB}"
return f"redis://{self.REDIS_HOST}:{self.REDIS_PORT}/{self.REDIS_DB}"
# === Rate Limiting ===
RATE_LIMIT_REQUESTS: int = 100
RATE_LIMIT_PERIOD: int = 60 # seconds
# === Taskiq ===
TASKIQ_BROKER_URL: str = "redis://localhost:6379/1"
TASKIQ_RESULT_BACKEND: str = "redis://localhost:6379/1"
# === AI Agent (pydantic_deep, openai) ===
OPENAI_API_KEY: str = ""
AI_MODEL: str = "gpt-5.5"
AI_TEMPERATURE: float = 0.7
AI_THINKING_ENABLED: bool = False
AI_THINKING_EFFORT: str = "medium" # "low", "medium", "high"
AI_AVAILABLE_MODELS: list[str] = [
"gpt-5.5",
"gpt-5.5-pro",
"gpt-5.4",
"gpt-5.4-pro",
"gpt-5.4-mini",
"gpt-5.4-nano",
"gpt-5-mini",
"gpt-5-nano",
"gpt-5",
"gpt-4.1",
]
AI_FRAMEWORK: str = "pydantic_deep"
LLM_PROVIDER: str = "openai"
# === Web Search (Tavily) ===
TAVILY_API_KEY: str = ""
# === AntV charts (advanced diagrams via mcp-server-chart sidecar) ===
# Opt-in at runtime: the code ships, but stays off until you flip this on
# and start the antvis-chart sidecar (docker compose --profile antv).
ENABLE_ANTV_CHARTS: bool = False
# MCP endpoint of the antvis-chart sidecar (streamable HTTP)
ANTV_MCP_URL: str = "http://antvis-chart:1122/mcp"
# Optional self-hosted GPT-Vis render backend (empty = AntV public service)
ANTV_VIS_REQUEST_SERVER: str = ""
# Comma-separated AntV tools to disable — defaults (set on the sidecar) drop
# the basic charts that overlap create_chart and the China-only maps.
ANTV_DISABLED_TOOLS: str = ""
# === PydanticDeep Configuration ===
# Backend type: "state" (in-memory) or "daytona" (Daytona cloud workspace)
PYDANTIC_DEEP_BACKEND_TYPE: str = "state"
# Feature flags
PYDANTIC_DEEP_INCLUDE_SUBAGENTS: bool = True # subagent delegation
PYDANTIC_DEEP_INCLUDE_SKILLS: bool = True # SKILL.md discovery
PYDANTIC_DEEP_INCLUDE_PLAN: bool = True # planner subagent
PYDANTIC_DEEP_INCLUDE_MEMORY: bool = True # MEMORY.md persistence
PYDANTIC_DEEP_INCLUDE_EXECUTE: bool = False # shell execution (security risk — off by default)
PYDANTIC_DEEP_WEB_SEARCH: bool = True # built-in pydantic-ai web search
# === RAG (Retrieval Augmented Generation) ===
# Vector Database (Qdrant)
QDRANT_HOST: str = "localhost"
QDRANT_PORT: int = 6333
QDRANT_API_KEY: str = ""
# Embeddings
EMBEDDING_MODEL: str = "text-embedding-3-small"
# Chunking
RAG_CHUNK_SIZE: int = 512
RAG_CHUNK_OVERLAP: int = 50
# Retrieval
RAG_DEFAULT_COLLECTION: str = "documents"
RAG_TOP_K: int = 10
RAG_CHUNKING_STRATEGY: str = "recursive" # recursive, markdown, or fixed
RAG_HYBRID_SEARCH: bool = False # Enable BM25 + vector hybrid search
RAG_ENABLE_OCR: bool = False # OCR fallback for scanned PDFs (requires tesseract)
# Reranker
HF_TOKEN: str = ""
CROSS_ENCODER_MODEL: str = "cross-encoder/ms-marco-MiniLM-L6-v2"
# Document Parser
# Google Drive (optional, for document ingestion via service account)
# S3 (optional, for document ingestion from S3/MinIO)
# === CORS ===
CORS_ORIGINS: list[str] = [
"http://localhost:3000",
"http://localhost:8080",
]
CORS_ALLOW_CREDENTIALS: bool = True
CORS_ALLOW_METHODS: list[str] = ["*"]
CORS_ALLOW_HEADERS: list[str] = ["*"]
@field_validator("CORS_ORIGINS")
@classmethod
def validate_cors_origins(cls, v: list[str], info: ValidationInfo) -> list[str]:
"""Warn if CORS_ORIGINS is too permissive in production."""
env = info.data.get("ENVIRONMENT", "local") if info.data else "local"
if "*" in v and env == "production":
raise ValueError(
"CORS_ORIGINS cannot contain '*' in production! Specify explicit allowed origins."
)
return v
@computed_field # type: ignore[prop-decorator]
@property
def rag(self) -> "RAGSettings":
"""Build RAG-specific settings."""
from app.services.rag.config import RAGSettings, DocumentParser, PdfParser, EmbeddingsConfig
pdf_parser = PdfParser()
return RAGSettings(
collection_name=self.RAG_DEFAULT_COLLECTION,
chunk_size=self.RAG_CHUNK_SIZE,
chunk_overlap=self.RAG_CHUNK_OVERLAP,
chunking_strategy=self.RAG_CHUNKING_STRATEGY,
enable_hybrid_search=self.RAG_HYBRID_SEARCH,
enable_ocr=self.RAG_ENABLE_OCR,
embeddings_config=EmbeddingsConfig(model=self.EMBEDDING_MODEL),
document_parser=DocumentParser(),
pdf_parser=pdf_parser,
)
# Rebuild Settings to resolve RAGSettings forward reference
from app.services.rag.config import RAGSettings
Settings.model_rebuild()
settings = Settings()
backend/app/core/csrf.py
+148
View File
@@ -0,0 +1,148 @@
"""CSRF protection middleware for FastAPI.
This module provides CSRF (Cross-Site Request Forgery) protection for
state-changing HTTP methods (POST, PUT, PATCH, DELETE).
The protection works by:
1. Setting a CSRF token in a cookie on initial request
2. Requiring the token to be sent in a header for state-changing requests
3. Comparing the cookie token with the header token
Usage:
Add to your main.py:
from app.core.csrf import CSRFMiddleware
app.add_middleware(CSRFMiddleware)
For endpoints that should be exempt (e.g., login):
@router.post("/login", tags=["csrf-exempt"])
async def login(...):
...
"""
import secrets
from typing import Any, ClassVar
from fastapi import Request, Response
from fastapi.responses import JSONResponse
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from starlette.types import ASGIApp
from app.core.config import settings
class CSRFMiddleware(BaseHTTPMiddleware):
"""CSRF protection middleware.
Protects against Cross-Site Request Forgery attacks by requiring
a token to be present in both a cookie and a header for state-changing requests.
"""
# Methods that require CSRF protection
PROTECTED_METHODS: ClassVar[set[str]] = {"POST", "PUT", "PATCH", "DELETE"}
# Cookie settings
COOKIE_NAME: ClassVar[str] = "csrf_token"
HEADER_NAME: ClassVar[str] = "X-CSRF-Token"
# Paths to exclude from CSRF protection
EXEMPT_PATHS: ClassVar[set[str]] = {
"/api/v1/auth/login",
"/api/v1/auth/register",
"/api/v1/auth/refresh",
"/api/v1/health",
"/api/v1/ready",
"/docs",
"/openapi.json",
"/redoc",
}
def __init__(self, app: ASGIApp, **kwargs: Any) -> None:
super().__init__(app)
self.exempt_paths: set[str] = set(kwargs.get("exempt_paths", self.EXEMPT_PATHS))
self.cookie_name: str = kwargs.get("cookie_name", self.COOKIE_NAME)
self.header_name: str = kwargs.get("header_name", self.HEADER_NAME)
async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
"""Handle the request and apply CSRF protection."""
# Skip for exempt paths
if self._is_exempt(request):
return await call_next(request)
# Get or generate CSRF token
csrf_token = request.cookies.get(self.cookie_name)
if not csrf_token:
csrf_token = self._generate_token()
# Check CSRF for protected methods
if request.method in self.PROTECTED_METHODS:
header_token = request.headers.get(self.header_name)
if not header_token:
return JSONResponse(
status_code=403,
content={
"detail": "CSRF token missing",
"message": f"Include the '{self.header_name}' header with the CSRF token",
},
)
if not secrets.compare_digest(csrf_token, header_token):
return JSONResponse(
status_code=403,
content={
"detail": "CSRF token invalid",
"message": "The CSRF token does not match",
},
)
# Process the request
response = await call_next(request)
# Set CSRF token cookie if not present
if not request.cookies.get(self.cookie_name):
response.set_cookie(
key=self.cookie_name,
value=csrf_token,
httponly=False, # JavaScript needs to read this
secure=not settings.DEBUG,
samesite="lax",
max_age=3600 * 24, # 24 hours
)
return response
def _is_exempt(self, request: Request) -> bool:
"""Check if the request path is exempt from CSRF protection."""
path = request.url.path
# Check exact path matches
if path in self.exempt_paths:
return True
# Check path prefixes
for exempt in self.exempt_paths:
if path.startswith(exempt):
return True
# Check if endpoint has "csrf-exempt" tag
route = request.scope.get("route")
return bool(route and hasattr(route, "tags") and "csrf-exempt" in route.tags)
@staticmethod
def _generate_token() -> str:
"""Generate a secure CSRF token."""
return secrets.token_urlsafe(32)
def get_csrf_token(request: Request) -> str:
"""Get the current CSRF token from cookies or generate a new one.
Use this in templates or API responses to provide the token to clients.
"""
token = request.cookies.get(CSRFMiddleware.COOKIE_NAME)
if not token:
token = secrets.token_urlsafe(32)
return token
backend/app/core/exceptions.py
+130
View File
@@ -0,0 +1,130 @@
"""Application exceptions.
Domain exceptions with HTTP status codes for the hybrid approach.
These exceptions are caught by exception handlers and converted to proper HTTP responses.
"""
from typing import Any
class AppException(Exception):
"""Base exception for all application errors.
Attributes:
message: Human-readable error message.
code: Machine-readable error code for clients.
status_code: HTTP status code to return.
details: Additional error details (e.g., field names, IDs).
"""
message: str = "An error occurred"
code: str = "APP_ERROR"
status_code: int = 500
def __init__(
self,
message: str | None = None,
code: str | None = None,
details: dict[str, Any] | None = None,
):
self.message = message or self.__class__.message
self.code = code or self.__class__.code
self.details = details or {}
super().__init__(self.message)
def __repr__(self) -> str:
return f"{self.__class__.__name__}(message={self.message!r}, code={self.code!r})"
# === 4xx Client Errors ===
class NotFoundError(AppException):
"""Resource not found (404)."""
message = "Resource not found"
code = "NOT_FOUND"
status_code = 404
class AlreadyExistsError(AppException):
"""Resource already exists (409)."""
message = "Resource already exists"
code = "ALREADY_EXISTS"
status_code = 409
class ValidationError(AppException):
"""Validation error (422)."""
message = "Validation error"
code = "VALIDATION_ERROR"
status_code = 422
class AuthenticationError(AppException):
"""Authentication failed (401)."""
message = "Authentication failed"
code = "AUTHENTICATION_ERROR"
status_code = 401
class AuthorizationError(AppException):
"""Authorization failed - insufficient permissions (403)."""
message = "Insufficient permissions"
code = "AUTHORIZATION_ERROR"
status_code = 403
class RateLimitError(AppException):
"""Rate limit exceeded (429)."""
message = "Rate limit exceeded"
code = "RATE_LIMIT_EXCEEDED"
status_code = 429
class BadRequestError(AppException):
"""Bad request (400)."""
message = "Bad request"
code = "BAD_REQUEST"
status_code = 400
class PaymentRequiredError(AppException):
"""Payment required — seat or usage limit reached (402)."""
message = "Payment required"
code = "PAYMENT_REQUIRED"
status_code = 402
# === 5xx Server Errors ===
class ExternalServiceError(AppException):
"""External service unavailable (503)."""
message = "External service unavailable"
code = "EXTERNAL_SERVICE_ERROR"
status_code = 503
class DatabaseError(AppException):
"""Database error (500)."""
message = "Database error"
code = "DATABASE_ERROR"
status_code = 500
class InternalError(AppException):
"""Internal server error (500)."""
message = "Internal server error"
code = "INTERNAL_ERROR"
status_code = 500
backend/app/core/logfire_setup.py
+32
View File
@@ -0,0 +1,32 @@
"""Logfire observability configuration."""
from typing import Any
import logfire
from app.core.config import settings
def setup_logfire() -> None:
"""Configure Logfire instrumentation."""
logfire.configure(
token=settings.LOGFIRE_TOKEN,
service_name=settings.LOGFIRE_SERVICE_NAME,
environment=settings.LOGFIRE_ENVIRONMENT,
send_to_logfire="if-token-present",
)
def instrument_app(app: Any) -> None:
"""Instrument FastAPI app with Logfire."""
logfire.instrument_fastapi(app)
def instrument_asyncpg() -> None:
"""Instrument asyncpg for PostgreSQL."""
logfire.instrument_asyncpg()
def instrument_redis() -> None:
"""Instrument Redis."""
logfire.instrument_redis()
backend/app/core/logging.py
+78
View File
@@ -0,0 +1,78 @@
"""Logging utilities — PII redaction filter for GDPR/compliance safety."""
import logging
import re
from typing import ClassVar
class PiiRedactionFilter(logging.Filter):
"""Logging filter that redacts personally identifiable information.
Automatically scrubs email addresses, JWT tokens, API keys, bearer tokens,
and password-like values from log messages to prevent PII leaks to
log aggregators (Datadog, CloudWatch, Logfire, etc.).
Usage:
logging.getLogger().addFilter(PiiRedactionFilter())
"""
PATTERNS: ClassVar[list[tuple[re.Pattern[str], str]]] = [
# Email addresses
(re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"), "[EMAIL_REDACTED]"),
# JWT tokens (header.payload.signature)
(
re.compile(r"eyJ[A-Za-z0-9_-]{10,}\.eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]+"),
"[JWT_REDACTED]",
),
# OpenAI API keys
(re.compile(r"sk-[a-zA-Z0-9]{20,}"), "[API_KEY_REDACTED]"),
# Anthropic API keys
(re.compile(r"sk-ant-[a-zA-Z0-9_-]{20,}"), "[API_KEY_REDACTED]"),
# Generic long hex/base64 secrets (40+ chars, likely tokens)
(
re.compile(
r"(?:token|key|secret|password|authorization)[=: ]+['\"]?([A-Za-z0-9_/+=.-]{40,})",
re.IGNORECASE,
),
"[SECRET_REDACTED]",
),
# Bearer tokens in headers
(re.compile(r"Bearer\s+[A-Za-z0-9._~+/=-]{10,}"), "Bearer [TOKEN_REDACTED]"),
# Password/secret in key=value or key: value patterns
(
re.compile(
r"(password|passwd|pwd|secret_key|api_key|apikey|auth_token|access_token|refresh_token)"
r"[\s]*[=:]\s*['\"]?\S+['\"]?",
re.IGNORECASE,
),
r"\1=[REDACTED]",
),
]
def filter(self, record: logging.LogRecord) -> bool:
"""Redact PII from log record message and args."""
if isinstance(record.msg, str):
record.msg = self._redact(record.msg)
if record.args:
if isinstance(record.args, dict):
record.args = {
k: self._redact(v) if isinstance(v, str) else v for k, v in record.args.items()
}
elif isinstance(record.args, tuple):
record.args = tuple(
self._redact(a) if isinstance(a, str) else a for a in record.args
)
return True
def _redact(self, value: str) -> str:
for pattern, replacement in self.PATTERNS:
value = pattern.sub(replacement, value)
return value
def setup_logging() -> None:
"""Configure root logger with PII redaction filter."""
root_logger = logging.getLogger()
# Avoid adding duplicate filters
if not any(isinstance(f, PiiRedactionFilter) for f in root_logger.filters):
root_logger.addFilter(PiiRedactionFilter())
backend/app/core/middleware.py
+102
View File
@@ -0,0 +1,102 @@
"""Application middleware."""
from typing import ClassVar
from uuid import uuid4
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from starlette.requests import Request
from starlette.responses import Response
from starlette.types import ASGIApp
class RequestIDMiddleware(BaseHTTPMiddleware):
"""Middleware that adds a unique request ID to each request.
The request ID is taken from the X-Request-ID header if present,
otherwise a new UUID is generated. The ID is added to the response
headers and is available in request.state.request_id.
"""
async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
"""Add request ID to request state and response headers."""
request_id = request.headers.get("X-Request-ID", str(uuid4()))
request.state.request_id = request_id
response = await call_next(request)
response.headers["X-Request-ID"] = request_id
return response
class SecurityHeadersMiddleware(BaseHTTPMiddleware):
"""Middleware that adds security headers to all responses.
This includes:
- Content-Security-Policy (CSP)
- X-Content-Type-Options
- X-Frame-Options
- X-XSS-Protection
- Referrer-Policy
- Permissions-Policy
Usage:
app.add_middleware(SecurityHeadersMiddleware)
# Or with custom CSP:
app.add_middleware(
SecurityHeadersMiddleware,
csp_directives={
"default-src": "'self'",
"script-src": "'self' 'unsafe-inline'",
}
)
"""
DEFAULT_CSP_DIRECTIVES: ClassVar[dict[str, str]] = {
"default-src": "'self'",
"script-src": "'self'",
"style-src": "'self' 'unsafe-inline'", # Allow inline styles for some UI libs
"img-src": "'self' data: https:",
"font-src": "'self' data:",
"connect-src": "'self'",
"frame-ancestors": "'none'",
"base-uri": "'self'",
"form-action": "'self'",
}
def __init__(
self,
app: ASGIApp,
csp_directives: dict[str, str] | None = None,
exclude_paths: set[str] | None = None,
) -> None:
super().__init__(app)
self.csp_directives = csp_directives or self.DEFAULT_CSP_DIRECTIVES
self.exclude_paths = exclude_paths or {"/docs", "/redoc", "/openapi.json"}
async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
"""Add security headers to the response."""
response = await call_next(request)
# Skip for docs/openapi endpoints which need different CSP
if request.url.path in self.exclude_paths:
return response
# Build CSP header
csp_value = "; ".join(
f"{directive} {value}" for directive, value in self.csp_directives.items()
)
# Add security headers — respect any already set by the route, so an
# endpoint can opt into less restrictive framing (e.g. user-content
# files served inline for the chat preview panel).
response.headers.setdefault("Content-Security-Policy", csp_value)
response.headers.setdefault("X-Content-Type-Options", "nosniff")
response.headers.setdefault("X-Frame-Options", "DENY")
response.headers.setdefault("X-XSS-Protection", "1; mode=block")
response.headers.setdefault("Referrer-Policy", "strict-origin-when-cross-origin")
response.headers["Permissions-Policy"] = (
"accelerometer=(), camera=(), geolocation=(), gyroscope=(), "
"magnetometer=(), microphone=(), payment=(), usb=()"
)
return response
backend/app/core/rate_limit.py
+55
View File
@@ -0,0 +1,55 @@
"""Rate limiting configuration using slowapi.
Default rate limit: 100 requests per 60 seconds.
Override with RATE_LIMIT_REQUESTS and RATE_LIMIT_PERIOD environment variables.
"""
from slowapi import Limiter
from slowapi.util import get_remote_address
from app.core.config import settings
def get_default_rate_limit() -> str:
"""Get default rate limit string from settings.
Returns a rate limit string like "100/minute" or "60/second".
"""
requests = settings.RATE_LIMIT_REQUESTS
period = settings.RATE_LIMIT_PERIOD
# Convert period to a human-readable format
period_map = {
60: "minute",
3600: "hour",
86400: "day",
}
if period in period_map:
return f"{requests}/{period_map[period]}"
# For custom periods, use "per X seconds"
return f"{requests}/{period} seconds"
# Rate limiter instance with configurable default
limiter = Limiter(
key_func=get_remote_address,
default_limits=[get_default_rate_limit()],
)
# Common rate limit decorators for convenience
# Usage: @rate_limit_low, @rate_limit_medium, @rate_limit_high
def rate_limit_low(limit: str = "10/minute"):
"""Low rate limit for expensive operations."""
return limiter.limit(limit)
def rate_limit_medium(limit: str = "30/minute"):
"""Medium rate limit for standard operations."""
return limiter.limit(limit)
def rate_limit_high(limit: str = "100/minute"):
"""High rate limit for lightweight operations."""
return limiter.limit(limit)
backend/app/core/sanitize.py
+433
View File
@@ -0,0 +1,433 @@
"""Input sanitization utilities.
This module provides security-focused input sanitization functions:
- HTML sanitization to prevent XSS attacks
- Path traversal prevention for file operations
- Webhook URL validation to prevent SSRF attacks
- Common input cleaning utilities
Note: SQL injection is prevented by using SQLAlchemy ORM with parameterized queries.
"""
import html
import ipaddress
import os
import re
import socket
import unicodedata
from pathlib import Path
from typing import TypeVar
from urllib.parse import urlparse
# Default allowed HTML tags for rich text content
DEFAULT_ALLOWED_TAGS = frozenset(
{
"a",
"abbr",
"acronym",
"b",
"blockquote",
"br",
"code",
"em",
"i",
"li",
"ol",
"p",
"pre",
"strong",
"ul",
}
)
# Default allowed HTML attributes
DEFAULT_ALLOWED_ATTRIBUTES = {
"a": frozenset({"href", "title", "rel"}),
"abbr": frozenset({"title"}),
"acronym": frozenset({"title"}),
}
# Allowed URL schemes for webhook URLs
WEBHOOK_ALLOWED_SCHEMES = frozenset({"http", "https"})
# Shared Address Space (RFC 6598) — CGNAT range.
# Python 3.11+ no longer classifies 100.64.0.0/10 as private or reserved,
# so we block it explicitly. Covers cloud metadata endpoints like
# Alibaba Cloud's 100.100.100.200.
_CGNAT_NETWORK = ipaddress.ip_network("100.64.0.0/10")
class SSRFBlockedError(ValueError):
"""Raised when a URL is blocked by SSRF protection.
Dedicated exception type to avoid fragile string matching when
distinguishing SSRF blocks from other ValueErrors.
"""
def sanitize_html(
content: str,
allowed_tags: frozenset[str] | None = None,
strip: bool = True,
) -> str:
"""Sanitize HTML content to prevent XSS attacks.
This is a simple implementation that escapes all HTML.
For rich text support, consider using the `bleach` library.
Args:
content: The HTML content to sanitize.
allowed_tags: Not used in simple mode (for bleach compatibility).
strip: Not used in simple mode (for bleach compatibility).
Returns:
Escaped HTML-safe string.
Example:
>>> sanitize_html("<script>alert('xss')</script>")
"&lt;script&gt;alert('xss')&lt;/script&gt;"
"""
if not content:
return ""
return html.escape(content)
def sanitize_filename(filename: str, allow_unicode: bool = False) -> str:
"""Sanitize a filename to prevent path traversal and unsafe characters.
Args:
filename: The filename to sanitize.
allow_unicode: Whether to allow unicode characters.
Returns:
A safe filename string.
Example:
>>> sanitize_filename("../../../etc/passwd")
"etc_passwd"
>>> sanitize_filename("hello world.txt")
"hello_world.txt"
"""
if not filename:
return ""
# Normalize unicode
if allow_unicode:
filename = unicodedata.normalize("NFKC", filename)
else:
filename = unicodedata.normalize("NFKD", filename).encode("ascii", "ignore").decode("ascii")
# Get just the filename (remove any path components)
filename = os.path.basename(filename)
# Remove null bytes
filename = filename.replace("\x00", "")
# Replace path separators and special characters
filename = re.sub(r"[/\\:*?\"<>|]", "_", filename)
# Replace multiple underscores/spaces with single underscore
filename = re.sub(r"[\s_]+", "_", filename)
# Remove leading/trailing underscores and dots
filename = filename.strip("._")
# Ensure we have a valid filename
if not filename:
return "unnamed"
return filename
def validate_safe_path(
base_dir: Path | str,
user_path: str,
) -> Path:
"""Validate that a user-provided path is within the allowed base directory.
Prevents path traversal attacks by ensuring the resolved path
is within the expected directory.
Args:
base_dir: The base directory that all paths must be within.
user_path: The user-provided path to validate.
Returns:
The resolved, safe path.
Raises:
ValueError: If the path would escape the base directory.
Example:
>>> validate_safe_path("/uploads", "../../../etc/passwd")
Raises ValueError
>>> validate_safe_path("/uploads", "images/photo.jpg")
Path("/uploads/images/photo.jpg")
"""
base_path = Path(base_dir).resolve()
user_path_sanitized = sanitize_filename(user_path.lstrip("/\\"))
# Resolve the full path
full_path = (base_path / user_path_sanitized).resolve()
# Check if the resolved path is within the base directory
try:
full_path.relative_to(base_path)
except ValueError as err:
raise ValueError(
f"Path traversal detected: {user_path!r} would escape {base_dir!r}"
) from err
return full_path
def _is_ip_blocked(ip_str: str) -> bool:
"""Check if an IP address is private, reserved, loopback, or link-local.
Args:
ip_str: The IP address string to check.
Returns:
True if the address should be blocked, False if it's safe.
"""
try:
addr = ipaddress.ip_address(ip_str)
except ValueError:
# If we can't parse it, block it to be safe
return True
return (
addr.is_private
or addr.is_reserved
or addr.is_loopback
or addr.is_link_local
or addr.is_multicast
or addr.is_unspecified
or addr in _CGNAT_NETWORK
)
def validate_webhook_url(
url: str,
allowed_schemes: frozenset[str] | None = None,
) -> str:
"""Validate a webhook URL to prevent SSRF attacks.
Checks that the URL:
- Uses an allowed scheme (http/https only by default)
- Does not contain userinfo (credentials in the URL)
- Does not point to private, reserved, loopback, or link-local IP addresses
- Resolves via DNS to a public IP (prevents DNS rebinding attacks)
Args:
url: The webhook URL to validate.
allowed_schemes: Allowed URL schemes. Defaults to {"http", "https"}.
Returns:
The validated URL string.
Raises:
SSRFBlockedError: If the URL is blocked by SSRF protection.
ValueError: If the URL is malformed.
Example:
>>> validate_webhook_url("https://example.com/webhook")
"https://example.com/webhook"
>>> validate_webhook_url("http://169.254.169.254/latest/meta-data/")
Raises SSRFBlockedError
"""
if allowed_schemes is None:
allowed_schemes = WEBHOOK_ALLOWED_SCHEMES
# Parse the URL
try:
parsed = urlparse(url)
except Exception as err:
raise ValueError(f"Invalid webhook URL: {url!r}") from err
# Validate scheme
if parsed.scheme not in allowed_schemes:
raise SSRFBlockedError(
f"URL scheme {parsed.scheme!r} is not allowed. "
f"Allowed schemes: {', '.join(sorted(allowed_schemes))}"
)
# Extract hostname
hostname = parsed.hostname
if not hostname:
raise ValueError(f"Invalid webhook URL: no hostname found in {url!r}")
# Reject URLs with userinfo (credentials) to prevent URL parsing ambiguities
# e.g. http://user:pass@host/ or http://foo@169.254.169.254%00@public.com/
if parsed.username is not None or parsed.password is not None:
raise SSRFBlockedError(
"Webhook URL must not contain credentials (userinfo). "
"Remove the user:password@ portion from the URL."
)
# Try to parse hostname directly as an IP address
try:
addr = ipaddress.ip_address(hostname)
if _is_ip_blocked(str(addr)):
raise SSRFBlockedError(
f"Webhook URL blocked: {hostname!r} resolves to a private/internal "
f"address. SSRF protection does not allow requests to internal networks."
)
return url
except SSRFBlockedError:
raise
except ValueError:
# Not an IP literal — continue to DNS resolution below
pass
# Determine the correct default port based on the scheme
default_port = 443 if parsed.scheme == "https" else 80
port = parsed.port or default_port
# Resolve hostname via DNS and check all returned addresses
# TODO: socket.getaddrinfo() is blocking I/O — in async code paths
# (PostgreSQL, MongoDB) consider using loop.getaddrinfo() or run_in_executor.
try:
addr_infos = socket.getaddrinfo(hostname, port, proto=socket.IPPROTO_TCP)
except socket.gaierror as err:
raise SSRFBlockedError(
f"Webhook URL blocked: unable to resolve hostname {hostname!r}"
) from err
if not addr_infos:
raise SSRFBlockedError(
f"Webhook URL blocked: hostname {hostname!r} did not resolve to any address"
)
for _family, _type, _proto, _canonname, sockaddr in addr_infos:
ip_str = str(sockaddr[0])
if _is_ip_blocked(ip_str):
raise SSRFBlockedError(
f"Webhook URL blocked: {hostname!r} resolves to private/internal "
f"address {ip_str!r}. SSRF protection does not allow requests to "
f"internal networks."
)
return url
def sanitize_string(
value: str,
max_length: int | None = None,
allow_newlines: bool = True,
strip_whitespace: bool = True,
) -> str:
"""Sanitize a string input with various options.
Args:
value: The string to sanitize.
max_length: Maximum allowed length (truncates if exceeded).
allow_newlines: Whether to preserve newlines.
strip_whitespace: Whether to strip leading/trailing whitespace.
Returns:
Sanitized string.
"""
if not value:
return ""
# Strip null bytes and other control characters (except newlines if allowed)
if allow_newlines:
value = re.sub(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]", "", value)
else:
value = re.sub(r"[\x00-\x1f\x7f]", "", value)
# Strip whitespace if requested
if strip_whitespace:
value = value.strip()
# Truncate if needed
if max_length is not None and len(value) > max_length:
value = value[:max_length]
return value
def sanitize_email(email: str) -> str:
"""Basic email sanitization.
Note: For proper email validation, use Pydantic's EmailStr type.
This function only performs basic cleaning.
Args:
email: The email address to sanitize.
Returns:
Lowercased, stripped email.
"""
if not email:
return ""
return email.strip().lower()
T = TypeVar("T", int, float)
def sanitize_numeric(
value: str | int | float,
value_type: type[T],
min_value: T | None = None,
max_value: T | None = None,
default: T | None = None,
) -> T | None:
"""Sanitize and validate a numeric value.
Args:
value: The value to sanitize (can be string or numeric).
value_type: The expected type (int or float).
min_value: Minimum allowed value.
max_value: Maximum allowed value.
default: Default value if conversion fails.
Returns:
The sanitized numeric value, or default if invalid.
Example:
>>> sanitize_numeric("100", int, min_value=0, max_value=1000)
100
>>> sanitize_numeric("abc", int, default=0)
0
"""
try:
result = value_type(value)
if min_value is not None and result < min_value:
result = min_value
if max_value is not None and result > max_value:
result = max_value
return result
except (ValueError, TypeError):
return default
def escape_sql_like(pattern: str, escape_char: str = "\\") -> str:
"""Escape special characters in a LIKE pattern.
Use this when building LIKE queries with user input.
Args:
pattern: The pattern to escape.
escape_char: The escape character to use.
Returns:
Escaped pattern safe for use in LIKE queries.
Example:
>>> escape_sql_like("100%")
"100\\%"
>>> escape_sql_like("under_score")
"under\\_score"
"""
# Escape the escape character first, then special chars
pattern = pattern.replace(escape_char, escape_char + escape_char)
pattern = pattern.replace("%", escape_char + "%")
pattern = pattern.replace("_", escape_char + "_")
return pattern
backend/app/core/security.py
+103
View File
@@ -0,0 +1,103 @@
"""Security utilities for JWT authentication."""
from datetime import UTC, datetime, timedelta
from typing import Any
import bcrypt
import jwt
from app.core.config import settings
def create_access_token(
subject: str | Any,
expires_delta: timedelta | None = None,
) -> str:
"""Create a JWT access token."""
if expires_delta:
expire = datetime.now(UTC) + expires_delta
else:
expire = datetime.now(UTC) + timedelta(minutes=settings.ACCESS_TOKEN_EXPIRE_MINUTES)
to_encode = {"exp": expire, "sub": str(subject), "type": "access"}
return jwt.encode(to_encode, settings.SECRET_KEY, algorithm=settings.ALGORITHM)
def create_refresh_token(
subject: str | Any,
expires_delta: timedelta | None = None,
) -> str:
"""Create a JWT refresh token."""
if expires_delta:
expire = datetime.now(UTC) + expires_delta
else:
expire = datetime.now(UTC) + timedelta(minutes=settings.REFRESH_TOKEN_EXPIRE_MINUTES)
to_encode = {"exp": expire, "sub": str(subject), "type": "refresh"}
return jwt.encode(to_encode, settings.SECRET_KEY, algorithm=settings.ALGORITHM)
def verify_token(token: str) -> dict[str, Any] | None:
"""Verify a JWT token and return payload."""
try:
payload = jwt.decode(
token,
settings.SECRET_KEY,
algorithms=[settings.ALGORITHM],
)
return payload
except jwt.PyJWTError:
return None
def create_password_reset_token(
subject: str | Any,
expires_delta: timedelta | None = None,
) -> str:
"""Single-use JWT for password reset.
Short-lived (1h default). The `type` claim distinguishes it from access /
refresh / magic-link tokens — a stolen reset token can't be used as an
access token.
"""
expire = datetime.now(UTC) + (expires_delta or timedelta(hours=1))
to_encode = {"exp": expire, "sub": str(subject), "type": "password_reset"}
return jwt.encode(to_encode, settings.SECRET_KEY, algorithm=settings.ALGORITHM)
def create_magic_link_token(
subject: str | Any,
expires_delta: timedelta | None = None,
) -> str:
"""Sign-in-by-email JWT. Short-lived (15 min default)."""
expire = datetime.now(UTC) + (expires_delta or timedelta(minutes=15))
to_encode = {"exp": expire, "sub": str(subject), "type": "magic_link"}
return jwt.encode(to_encode, settings.SECRET_KEY, algorithm=settings.ALGORITHM)
def verify_special_token(token: str, expected_type: str) -> dict[str, Any] | None:
"""Verify a non-access JWT (password_reset, magic_link) and require a
specific `type` claim. Returns payload on success, None otherwise.
"""
payload = verify_token(token)
if payload is None:
return None
if payload.get("type") != expected_type:
return None
return payload
def verify_password(plain_password: str, hashed_password: str) -> bool:
"""Verify a password against a hash."""
return bcrypt.checkpw(
plain_password.encode("utf-8"),
hashed_password.encode("utf-8"),
)
def get_password_hash(password: str) -> str:
"""Hash a password."""
return bcrypt.hashpw(
password.encode("utf-8"),
bcrypt.gensalt(),
).decode("utf-8")
backend/app/db/__init__.py
+6
View File
@@ -0,0 +1,6 @@
"""Database module."""
# SQLModel uses SQLModel class directly as base, no separate Base class needed
from app.db.base import TimestampMixin
__all__ = ["TimestampMixin"]
backend/app/db/base.py
+33
View File
@@ -0,0 +1,33 @@
"""SQLModel base model."""
from datetime import datetime
from sqlalchemy import DateTime, func
from sqlmodel import Field, SQLModel
# Naming convention for database constraints and indexes
# This ensures consistent naming across all migrations
NAMING_CONVENTION = {
"ix": "%(column_0_label)s_idx",
"uq": "%(table_name)s_%(column_0_name)s_key",
"ck": "%(table_name)s_%(constraint_name)s_check",
"fk": "%(table_name)s_%(column_0_name)s_fkey",
"pk": "%(table_name)s_pkey",
}
# Apply naming convention to SQLModel metadata
SQLModel.metadata.naming_convention = NAMING_CONVENTION
class TimestampMixin(SQLModel):
"""Mixin for created_at and updated_at timestamps."""
created_at: datetime = Field(
sa_type=DateTime(timezone=True),
sa_column_kwargs={"server_default": func.now(), "nullable": False},
)
updated_at: datetime | None = Field(
default=None,
sa_type=DateTime(timezone=True),
sa_column_kwargs={"onupdate": func.now(), "nullable": True},
)
backend/app/db/models/__init__.py
+32
View File
@@ -0,0 +1,32 @@
"""Database models."""
# ruff: noqa: I001, RUF022 - Imports structured for Jinja2 template conditionals
from app.db.models.user import User
from app.db.models.conversation import Conversation, Message, ToolCall
from app.db.models.webhook import Webhook, WebhookDelivery
from app.db.models.chat_file import ChatFile
from app.db.models.message_rating import MessageRating
from app.db.models.rag_document import RAGDocument
from app.db.models.sync_log import SyncLog
from app.db.models.sync_source import SyncSource
from app.db.models.conversation_share import ConversationShare
from app.db.models.project import Project, ProjectMember
from app.db.models.user_slash_command import UserSlashCommand
__all__ = [
"User",
"Conversation",
"Message",
"ToolCall",
"Webhook",
"WebhookDelivery",
"ChatFile",
"MessageRating",
"RAGDocument",
"SyncLog",
"SyncSource",
"ConversationShare",
"Project",
"ProjectMember",
"UserSlashCommand",
]
backend/app/db/models/chat_file.py
+43
View File
@@ -0,0 +1,43 @@
"""ChatFile database model - stores metadata for files uploaded in chat."""
import uuid
from sqlalchemy import Column, ForeignKey, Integer, String, Text
from sqlalchemy.dialects.postgresql import UUID as PG_UUID
from sqlmodel import Field, SQLModel
from app.db.base import TimestampMixin
class ChatFile(TimestampMixin, SQLModel, table=True):
"""Tracks files uploaded by users in chat conversations."""
__tablename__ = "chat_files"
id: uuid.UUID = Field(
default_factory=uuid.uuid4,
sa_column=Column(PG_UUID(as_uuid=True), primary_key=True),
)
user_id: uuid.UUID = Field(
sa_column=Column(
PG_UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
)
message_id: uuid.UUID | None = Field(
default=None,
sa_column=Column(
PG_UUID(as_uuid=True), ForeignKey("messages.id", ondelete="CASCADE"), nullable=True
),
)
filename: str = Field(sa_column=Column(String(255), nullable=False))
mime_type: str = Field(sa_column=Column(String(100), nullable=False))
size: int = Field(sa_column=Column(Integer, nullable=False))
storage_path: str = Field(sa_column=Column(String(500), nullable=False))
file_type: str = Field(sa_column=Column(String(20), nullable=False))
parsed_content: str | None = Field(default=None, sa_column=Column(Text, nullable=True))
def __repr__(self) -> str:
return f"<ChatFile(id={self.id}, filename={self.filename}, type={self.file_type})>"

Some files were not shown because too many files have changed in this diff Show More