furyhawk/ai_agent
1
0
Fork 0
mirror of https://github.com/furyhawk/ai_agent.git synced 2026-07-20 09:45:44 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: add Zustand stores for conversation, file preview, sidebar, theme, and knowledge base selection

Browse Source 
- Implemented `conversation-store` for managing conversations and messages.
- Created `file-preview-store` to handle file preview state.
- Added `sidebar-store` for sidebar visibility management.
- Developed `theme-store` for theme persistence and management.
- Introduced `kb-selection-store` for managing active knowledge base selections with persistence.

chore: define API and chat types

- Added types for API responses, authentication, chat messages, conversations, and projects.
- Defined interfaces for various entities including users, sessions, and message ratings.

build: configure TypeScript and testing setup

- Set up `tsconfig.json` for TypeScript configuration.
- Created `vitest.config.ts` for testing configuration with Vitest.
- Added `vitest.setup.ts` for global test setup including mocks for Next.js router and media queries.
- Configured Vercel deployment settings in `vercel.json`.
This commit is contained in:
furyhawk
2026-06-11 16:54:43 +08:00
commit 8351e73d39
528 changed files with 63713 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/commands/add-endpoint.md
+40
View File
@@ -0,0 +1,40 @@
---
description: Scaffold a new API endpoint with full layering
---
Create a new API endpoint: $ARGUMENTS
Follow the project's layered architecture. Create files in this order:
1. **Schema** (`backend/app/schemas/<entity>.py`):
- Inherit `BaseSchema` (and `TimestampSchema` for Read)
- Create `*Create`, `*Update`, `*Read`, `*List` models
- Use `Field()` with constraints, `EmailStr` where applicable
2. **DB Model** (`backend/app/db/models/<entity>.py`):
- Inherit `Base, TimestampMixin`
- Use `Mapped[type]` + `mapped_column()`
- Add `__repr__`, relationships with `cascade="all, delete-orphan"`
3. **Repository** (`backend/app/repositories/<entity>_repo.py`):
- Stateless async functions: `get_by_id`, `get_multi`, `create`, `update`, `delete`
- Use `db.flush()` + `db.refresh()`, keyword-only args after `db`
4. **Service** (`backend/app/services/<entity>.py`):
- Class with `__init__(self, db: AsyncSession)`
- Raise `NotFoundError`, `AlreadyExistsError` as appropriate
5. **DI** (`backend/app/api/deps.py`):
- Add factory function and `Annotated` alias: `EntitySvc = Annotated[EntityService, Depends(get_entity_service)]`
6. **Route** (`backend/app/api/routes/v1/<entity>.py`):
- CRUD: GET list, GET by id, POST (201), PATCH, DELETE (204)
- Use DI aliases, `response_model`, `-> Any` return type
7. **Register** router in `backend/app/api/routes/v1/__init__.py`
8. **Migration**: `cd backend && uv run alembic revision --autogenerate -m "Add <entity> table"`
9. **Test** (`backend/tests/`): mirror source structure
10. Lint: `cd backend && uv run ruff check . --fix && uv run ruff format .`
.claude/commands/fix-issue.md
+16
View File
@@ -0,0 +1,16 @@
---
description: Investigate and fix an issue
---
Fix the issue: $ARGUMENTS
1. **Understand** — search the codebase for relevant code, read the files, understand current behavior
2. **Reproduce** — if possible, identify a test case or request that triggers the issue
3. **Root cause** — trace through Routes → Services → Repositories to find where the bug originates
4. **Fix** — implement the fix following project conventions:
- Domain exceptions in services (not HTTP errors)
- `db.flush()` in repositories (not `commit`)
- Type hints on all changed signatures
5. **Test** — run `cd backend && uv run pytest` to verify no regressions
6. **Lint** — run `cd backend && uv run ruff check . --fix && uv run ruff format .`
7. **Summary** — explain what was changed and why
.claude/commands/review.md
+31
View File
@@ -0,0 +1,31 @@
---
description: Review code changes against project conventions
---
Review all staged and unstaged changes in the current branch.
For each changed file, verify:
**Architecture:**
- Routes only call services, never repositories
- Services raise domain exceptions (NotFoundError, AlreadyExistsError, etc.), not HTTP exceptions
- Repositories use `db.flush()` + `db.refresh()`, never `db.commit()`
- DI uses Annotated aliases from `deps.py` (CurrentUser, *Svc), not raw `Depends()` in signatures
**Schemas & Types:**
- Separate Create/Update/Read/List Pydantic models
- Type hints on all function signatures (params + return)
- Modern syntax: `str | None` not `Optional[str]`
- Route return type is `-> Any`
**Code Quality:**
- No debug code (print, commented-out code, TODO without issue reference)
- No security issues (SQL injection, exposed secrets, missing auth)
- Consistent naming (snake_case functions, PascalCase classes)
- Imports ordered: stdlib → third-party → local
**Validation:**
1. Run `cd backend && uv run ruff check .`
2. Run `cd backend && uv run pytest` (if test files changed)
Provide findings with specific file:line references and suggest fixes.
.claude/rules/api-conventions.md
+90
View File
@@ -0,0 +1,90 @@
---
description: API design, REST conventions, auth, pagination, response format
globs: ["backend/app/api/**/*.py"]
---
# API Conventions
## Route Structure
- All routes under `/api/v1/` prefix
- One file per domain entity in `api/routes/v1/`
- Use `APIRouter()` with tags
## HTTP Methods & Status Codes
```python
# GET — read
@router.get("/{id}", response_model=EntityRead)
# GET list — paginated
@router.get("", response_model=EntityList)
# POST — create (201)
@router.post("", response_model=EntityRead, status_code=status.HTTP_201_CREATED)
# PATCH — partial update
@router.patch("/{id}", response_model=EntityRead)
# DELETE — no content (204)
@router.delete("/{id}", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
```
## Pagination
```python
@router.get("", response_model=ConversationList)
async def list_items(
service: ConversationSvc,
user: CurrentUser,
skip: int = Query(0, ge=0, description="Items to skip"),
limit: int = Query(50, ge=1, le=100, description="Max items to return"),
) -> Any:
items, total = await service.list(user_id=user.id, skip=skip, limit=limit)
return ConversationList(items=items, total=total)
```
## Authentication
- `CurrentUser` — JWT Bearer token (any authenticated user)
- `CurrentAdmin` — JWT + admin role check via `RoleChecker`
- `ValidAPIKey` — API key from header (service-to-service)
```python
# Protected endpoint
async def get_profile(user: CurrentUser) -> Any: ...
# Admin-only endpoint
async def delete_user(user: CurrentAdmin) -> Any: ...
# API key endpoint
async def webhook_callback(api_key: ValidAPIKey) -> Any: ...
```
## Response Format
All route handlers return `-> Any`. The `response_model` parameter handles serialization.
Error responses follow this JSON structure:
```json
{
"error": {
"code": "NOT_FOUND",
"message": "User not found",
"details": {"user_id": "..."}
}
}
```
## File Upload
```python
@router.post("/me/avatar", response_model=UserRead)
async def upload_avatar(
file: UploadFile = File(...),
user: CurrentUser,
service: UserSvc,
) -> Any:
data = await file.read()
return await service.update_avatar(user.id, data, file.filename or "avatar.jpg")
```
.claude/rules/architecture.md
+138
View File
@@ -0,0 +1,138 @@
---
description: Layered architecture patterns — Routes, Services, Repositories, DI
globs: ["backend/app/**/*.py"]
---
# Architecture
## Layered Architecture: Routes → Services → Repositories
Routes NEVER import or call repositories directly. Always go through a service.
## Repositories (`app/repositories/`)
Pure data access — stateless functions (not classes):
```python
async def get_by_id(db: AsyncSession, entity_id: UUID) -> Entity | None:
result = await db.execute(select(Entity).where(Entity.id == entity_id))
return result.scalar_one_or_none()
async def create(db: AsyncSession, *, field1: str, field2: str) -> Entity:
entity = Entity(field1=field1, field2=field2)
db.add(entity)
await db.flush()
await db.refresh(entity)
return entity
async def update(db: AsyncSession, *, db_entity: Entity, update_data: dict[str, Any]) -> Entity:
for field, value in update_data.items():
setattr(db_entity, field, value)
await db.flush()
await db.refresh(db_entity)
return db_entity
async def delete(db: AsyncSession, entity_id: UUID) -> Entity | None:
entity = await get_by_id(db, entity_id)
if entity:
await db.delete(entity)
await db.flush()
return entity
```
Rules:
- ALWAYS `db.flush()` + `db.refresh()`, NEVER `db.commit()` — session auto-commits in `get_db_session`
- Use keyword-only args after `db`: `create(db, *, email: str, name: str)`
- Return the entity (or None for get/delete), never return IDs or dicts
- Functions are async for PostgreSQL/MongoDB, sync for SQLite
## Services (`app/services/`)
Business logic — class-based with DB session:
```python
class UserService:
def __init__(self, db: AsyncSession):
self.db = db
async def get_by_id(self, user_id: UUID) -> User:
user = await user_repo.get_by_id(self.db, user_id)
if not user:
raise NotFoundError(message="User not found", details={"user_id": user_id})
return user
async def create(self, data: UserCreate) -> User:
existing = await user_repo.get_by_email(self.db, data.email)
if existing:
raise AlreadyExistsError(message="Email already registered", details={"email": data.email})
hashed_password = get_password_hash(data.password)
return await user_repo.create(self.db, email=data.email, hashed_password=hashed_password)
```
Rules:
- Raise domain exceptions, NEVER return error codes or None for "not found"
- Services call repo functions, never build raw queries
- One service per domain entity
## Thin vs. thick domains
Services come in two shapes — choose based on whether the domain owns infrastructure (clients, adapters, pipelines, parsers, templates):
**Thin domain → flat module (`app/services/<domain>.py`).** Default. Just a class with `db`, repo calls, and domain exceptions. Examples: `user.py`, `conversation.py`, `invitation.py`.
**Thick domain → subpackage (`app/services/<domain>/`).** When the domain has its own infra. The subpackage contains both the service classes AND the infra (clients, adapters, pipeline modules, domain-specific exceptions). External callers only import from the package root — sub-modules are package-internal.
```
app/services/billing/
├── __init__.py # re-exports BillingService (the public facade)
├── facade.py # BillingService — the only thing routes see
├── checkout_service.py # internal sub-service
├── credit_service.py
├── subscription_service.py
├── webhook_handler.py
├── stripe_client.py # external API client (infra)
├── pricing.py # pure data
├── exceptions.py # domain-specific, inherits from core/exceptions
└── handlers/ # event handler modules (infra)
```
Other thick domains using the same shape: `services/rag/` (ingestion + vectorstore + embeddings + connectors), `services/channels/` (Slack + Telegram adapters + router), `services/email/` (providers + templates).
Rules for thick subpackages:
- Public API: only the top-level facade exported from `__init__.py`. Routes/workers never import sub-modules directly.
- Domain-specific exceptions live in the subpackage and inherit from `core/exceptions.py` base classes.
- Top-level `app/` is reserved for framework concerns (`api/`, `core/`, `db/`, `repositories/`, `schemas/`, `services/`, `worker/`, `agents/`, `commands/`, `clients/`). No new top-level domain packages.
## Dependency Injection (`app/api/deps.py`)
Use `Annotated` type aliases — never raw `Depends()` in route signatures:
```python
DBSession = Annotated[AsyncSession, Depends(get_db_session)]
UserSvc = Annotated[UserService, Depends(get_user_service)]
CurrentUser = Annotated[User, Depends(get_current_user)]
CurrentAdmin = Annotated[User, Depends(RoleChecker(UserRole.ADMIN))]
```
Service factories take `DBSession` and return service instances:
```python
def get_user_service(db: DBSession) -> UserService:
return UserService(db)
```
## Routes (`app/api/routes/v1/`)
HTTP layer only — validate, delegate, return:
```python
@router.get("/{user_id}", response_model=UserRead)
async def get_user(user_id: UUID, service: UserSvc, user: CurrentUser) -> Any:
return await service.get_by_id(user_id)
```
Rules:
- Return type is always `-> Any` (response_model handles serialization)
- Use `status_code=status.HTTP_201_CREATED` for POST, `HTTP_204_NO_CONTENT` for DELETE
- DELETE endpoints: `response_model=None`
- Pagination: `skip: int = Query(0, ge=0)`, `limit: int = Query(50, ge=1, le=100)`
.claude/rules/code-style.md
+67
View File
@@ -0,0 +1,67 @@
---
description: Code style, formatting, naming, imports, and type hints
globs: ["backend/**/*.py", "*.py"]
---
# Code Style
## Formatting
- Use `ruff` for linting and formatting: `ruff check . --fix && ruff format .`
- Line length: 120 characters
## Type Hints
- Type hints on ALL function signatures — parameters and return types
- Use modern syntax: `str | None` not `Optional[str]`, `list[User]` not `List[User]`
- Use `Annotated[Type, Depends(...)]` for DI (defined as aliases in `deps.py`)
- Use `dict[str, Any]` for generic dicts
- Use `Literal["value1", "value2"]` for string enums in schemas
- Use `TYPE_CHECKING` block for circular import resolution:
```python
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from app.db.models.session import Session
```
## Naming
| Element | Convention | Example |
|---------|-----------|---------|
| Files | snake_case | `user_repo.py`, `conversation_service.py` |
| Classes | PascalCase | `UserService`, `ConversationRead` |
| Functions/variables | snake_case | `get_by_id`, `user_service` |
| Constants | UPPER_CASE | `DEFAULT_SYSTEM_PROMPT` |
| Private | _leading_underscore | `_create_agent` |
| DB tables | snake_case plural | `users`, `conversations` |
| API URLs | kebab-case | `/api/v1/conversations` |
## Imports — strictly ordered, separated by blank lines
```python
# 1. Standard library
import logging
from collections.abc import AsyncGenerator
from datetime import UTC, datetime
from typing import Annotated, Any
from uuid import UUID
# 2. Third-party
from fastapi import APIRouter, Depends, Query, status
from pydantic import BaseModel, ConfigDict, Field
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
# 3. Local application
from app.api.deps import CurrentUser, UserSvc
from app.core.exceptions import NotFoundError
from app.schemas.user import UserCreate, UserRead
```
## Other Conventions
- `datetime.now(UTC)` not `datetime.utcnow()`
- `secrets.compare_digest()` for constant-time comparisons
- `__repr__` on all DB models
- Async for PostgreSQL/MongoDB I/O, sync for SQLite
- Keyword-only args in repo functions after `db` parameter
.claude/rules/exceptions-security.md
+54
View File
@@ -0,0 +1,54 @@
---
description: Exception handling patterns and security conventions
globs: ["backend/app/core/**/*.py", "backend/app/services/**/*.py"]
---
# Exceptions & Security
## Domain Exceptions (`app/core/exceptions.py`)
All extend `AppException`. Always pass `message` and `details`:
```python
raise NotFoundError(message="User not found", details={"user_id": str(user_id)})
raise AlreadyExistsError(message="Email already registered", details={"email": email})
raise AuthenticationError(message="Invalid or expired token")
raise AuthorizationError(message="Role 'admin' required for this action")
```
Exception handlers in `api/exception_handlers.py` automatically:
- Map to HTTP status codes
- Log with structured context (path, method, error code)
- Return consistent JSON error format
- Add `WWW-Authenticate: Bearer` header on 401
## Security Patterns
JWT auth (`core/security.py`):
- `create_access_token(subject)` / `create_refresh_token(subject)` — encode with `jwt.encode()`
- `verify_token(token)` → `dict | None` — decode with `jwt.decode()`
- Token payload: `{"exp": ..., "sub": user_id, "type": "access"|"refresh"}`
Password hashing:
- `get_password_hash(password)` — bcrypt
- `verify_password(plain, hashed)` — bcrypt `checkpw`
- NEVER store plain passwords
API keys:
- `secrets.compare_digest()` for constant-time comparison
- `APIKeyHeader(name=settings.API_KEY_HEADER, auto_error=False)`
## Role-Based Access Control
```python
class RoleChecker:
def __init__(self, required_role: UserRole) -> None:
self.required_role = required_role
async def __call__(self, user: Annotated[User, Depends(get_current_user)]) -> User:
if not user.has_role(self.required_role):
raise AuthorizationError(message=f"Role '{self.required_role.value}' required")
return user
CurrentAdmin = Annotated[User, Depends(RoleChecker(UserRole.ADMIN))]
```
.claude/rules/frontend.md
+27
View File
@@ -0,0 +1,27 @@
---
description: Frontend conventions for Next.js
globs: ["frontend/**/*.ts", "frontend/**/*.tsx", "frontend/**/*.css"]
---
# Frontend Conventions
## Stack
- Next.js 15 with App Router
- TypeScript strict mode
- Tailwind CSS for styling
- i18n support built-in
## Structure
- Pages in `frontend/src/app/` following Next.js App Router conventions
- Reusable components in `frontend/src/components/`
- API client functions in `frontend/src/lib/`
- Types in `frontend/src/types/`
## Conventions
- Use `"use client"` directive only when component needs client-side interactivity
- Prefer Server Components by default
- Use `fetch` with proper error handling for API calls
- Keep components small and focused — extract when a component exceeds ~100 lines
.claude/rules/schemas-models.md
+90
View File
@@ -0,0 +1,90 @@
---
description: Pydantic schema patterns and SQLAlchemy model conventions
globs: ["backend/app/schemas/**/*.py", "backend/app/db/models/**/*.py", "backend/app/db/base.py"]
---
# Schemas & Models
## Pydantic Schemas (`app/schemas/`)
Base schema with shared config:
```python
class BaseSchema(BaseModel):
model_config = ConfigDict(
from_attributes=True,
populate_by_name=True,
str_strip_whitespace=True,
)
```
Separate models per operation:
```python
class UserCreate(BaseSchema):
email: EmailStr = Field(max_length=255)
password: str = Field(min_length=8, max_length=128)
full_name: str | None = Field(default=None, max_length=255)
class UserUpdate(BaseSchema):
email: EmailStr | None = Field(default=None, max_length=255)
password: str | None = Field(default=None, min_length=8, max_length=128)
full_name: str | None = Field(default=None, max_length=255)
is_active: bool | None = None
class UserRead(BaseSchema, TimestampSchema):
id: UUID
email: EmailStr
full_name: str | None = None
role: UserRole = UserRole.USER
avatar_url: str | None = None
class UserList(BaseSchema):
items: list[UserRead]
total: int
```
Rules:
- `*Create` — required fields for creation, with `Field()` constraints
- `*Update` — all fields optional (`type | None = None`)
- `*Read` — includes `id` and timestamps, inherits `TimestampSchema`
- `*List` — `items` list + `total` count
- Use `@field_validator` for complex deserialization (e.g., JSON string → dict)
## SQLAlchemy Models (`app/db/models/`)
```python
class User(Base, TimestampMixin):
__tablename__ = "users"
id: Mapped[UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
email: Mapped[str] = mapped_column(String(255), unique=True, index=True, nullable=False)
hashed_password: Mapped[str | None] = mapped_column(String(255), nullable=True)
is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
conversations: Mapped[list["Conversation"]] = relationship(
"Conversation", back_populates="user", cascade="all, delete-orphan"
)
def __repr__(self) -> str:
return f"<User(id={self.id}, email={self.email})>"
```
Rules:
- Always inherit `Base` and `TimestampMixin` (provides `created_at`, `updated_at`)
- Use `Mapped[type]` with `mapped_column()` for all columns
- ForeignKey with `ondelete="CASCADE"` for parent references
- Always define `__repr__`
- Naming convention in `Base.metadata`: `{table}_{col}_key`, `{table}_{col}_fkey`, etc.
## TimestampMixin
```python
class TimestampMixin:
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), nullable=False
)
updated_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), onupdate=func.now(), nullable=True
)
```
.claude/rules/testing.md
+85
View File
@@ -0,0 +1,85 @@
---
description: Testing standards, fixtures, async test patterns
globs: ["backend/tests/**/*.py", "tests/**/*.py", "**/test_*.py", "**/conftest.py"]
---
# Testing
## Running Tests
```bash
cd backend
uv run pytest # all tests
uv run pytest tests/test_file.py -v # single file
uv run pytest -k "test_name" -v # by name
uv run pytest --cov=app # with coverage
```
## Structure
- Mirror source layout: `app/services/user.py` → `tests/services/test_user.py`
- Shared fixtures in `tests/conftest.py`
## Naming
```python
# test_<action>_<scenario>_<expected_result>
def test_create_user_with_duplicate_email_raises_already_exists_error
def test_get_conversation_not_found_raises_not_found_error
def test_list_conversations_returns_only_user_owned
```
## Fixtures
```python
@pytest.fixture
def user_service(db: AsyncSession) -> UserService:
return UserService(db)
@pytest.fixture
async def test_user(db: AsyncSession) -> User:
return await user_repo.create(db, email="test@example.com", hashed_password="hashed")
```
## Async Tests
```python
import pytest
@pytest.mark.asyncio
async def test_get_user_by_id(user_service: UserService, test_user: User):
result = await user_service.get_by_id(test_user.id)
assert result.email == test_user.email
```
## API Tests
Use `httpx.AsyncClient`, not `TestClient`:
```python
@pytest.mark.asyncio
async def test_create_user(client: AsyncClient, auth_headers: dict):
response = await client.post(
"/api/v1/users",
json={"email": "new@example.com", "password": "securepass123"},
headers=auth_headers,
)
assert response.status_code == 201
assert response.json()["email"] == "new@example.com"
```
## Exception Testing
```python
@pytest.mark.asyncio
async def test_get_user_not_found(user_service: UserService):
with pytest.raises(NotFoundError):
await user_service.get_by_id(UUID("00000000-0000-0000-0000-000000000000"))
```
## Rules
- Each test is independent — no shared mutable state
- Use plain `assert` (pytest rewrites for detailed output)
- One logical assertion per test (multiple asserts are fine if testing one behavior)
- Use factory fixtures for test data, not raw dicts
.claude/settings.json
+17
View File
@@ -0,0 +1,17 @@
{
"permissions": {
"allow": [
"Read",
"Glob",
"Grep",
"Bash(git status*)",
"Bash(git diff*)",
"Bash(git log*)",
"Bash(git branch*)",
"Bash(cd backend && uv run pytest*)",
"Bash(cd backend && uv run ruff*)",
"Bash(cd backend && uv run ty*)",
"Bash(cd backend && uv run alembic*)"
]
}
}
.github/workflows/ci.yml
+177
View File
@@ -0,0 +1,177 @@
name: CI
on:
push:
branches: [main, master]
pull_request:
branches: [main, master]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
version: "latest"
- name: Set up Python
run: uv python install 3.13
- name: Install dependencies
run: uv sync --directory backend --dev
- name: Run ruff check
run: uv run --directory backend ruff check app tests cli
- name: Run ruff format check
run: uv run --directory backend ruff format app tests cli --check
- name: Run ty
run: uv run --directory backend ty check
test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: test_db
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
redis:
image: redis:7-alpine
ports:
- 6379:6379
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
version: "latest"
- name: Set up Python
run: uv python install 3.13
- name: Install dependencies
run: uv sync --directory backend --dev
- name: Run tests
run: uv run --directory backend pytest tests/ -v --cov=app --cov-report=xml
env:
POSTGRES_HOST: localhost
POSTGRES_PORT: 5432
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: test_db
REDIS_HOST: localhost
REDIS_PORT: 6379
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
with:
files: ./backend/coverage.xml
fail_ci_if_error: false
test-frontend:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Bun
uses: oven-sh/setup-bun@v2
- name: Install dependencies
run: bun install --frozen-lockfile
working-directory: frontend
- name: Run lint
run: bun run lint
working-directory: frontend
- name: Run type check
run: bun run type-check
working-directory: frontend
- name: Run unit tests with coverage
run: bun run test:coverage
working-directory: frontend
- name: Install Playwright browsers
run: bunx playwright install --with-deps
working-directory: frontend
- name: Run E2E tests
run: bun run test:e2e
working-directory: frontend
env:
CI: true
security:
name: Security Scan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
version: "latest"
- name: Set up Python
run: uv python install 3.13
- name: Install dependencies
run: uv sync --directory backend --dev
- name: Install pip-audit
run: uv pip install pip-audit
- name: Run pip-audit
run: uv run pip-audit --require-hashes=false --progress-spinner=off
working-directory: backend
docker:
runs-on: ubuntu-latest
needs: [lint, test]
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build Docker image
uses: docker/build-push-action@v6
with:
context: ./backend
push: false
load: true
tags: ai_agent:latest
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@v0.36.0
with:
image-ref: ai_agent:latest
format: table
exit-code: 0
severity: CRITICAL,HIGH
.gitignore
+103
View File
@@ -0,0 +1,103 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
# Python lib directories (but not frontend src/lib)
/lib/
/lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
# PyInstaller
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
# Translations
*.mo
*.pot
# Environments
.env
.env.local
.env.*.local
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# IDE
.idea/
.vscode/
*.swp
*.swo
*~
# ty
.ty_cache/
# ruff
.ruff_cache/
# Jupyter Notebook
.ipynb_checkpoints
# pytype
.pytype/
# Cython debug symbols
cython_debug/
# Local development
*.db
*.sqlite
*.sqlite3
# Logs
*.log
logs/
# Docker
.docker/
# OS
.DS_Store
Thumbs.db
# Project specific
AGENTS.md
+66
View File
@@ -0,0 +1,66 @@
# AGENTS.md
This file provides guidance for AI coding agents (Codex, Copilot, Cursor, Zed, OpenCode).
## Project Overview
**ai_agent** - FastAPI application generated with [Full-Stack AI Agent Template](https://github.com/vstorm-co/full-stack-ai-agent-template).
**Stack:** FastAPI + Pydantic v2, PostgreSQL
, JWT + API Key auth, Redis
, pydantic_ai (openai), RAG (milvus), Next.js 15 (i18n)
## Commands
```bash
# Run server
cd backend && uv run uvicorn app.main:app --reload
# Tests & lint
pytest
ruff check . --fix && ruff format .
# Migrations
uv run alembic upgrade head
uv run alembic revision --autogenerate -m "Description"
# RAG
uv run ai_agent rag-ingest /path/to/file.pdf --collection docs
uv run ai_agent rag-search "query" --collection docs
# Sync Sources
uv run ai_agent cmd rag-sources
uv run ai_agent cmd rag-source-add
uv run ai_agent cmd rag-source-sync
```
## Project Structure
```
backend/app/
├── api/routes/v1/ # Endpoints
├── services/ # Business logic
├── repositories/ # Data access
├── schemas/ # Pydantic models
├── db/models/ # DB models
├── agents/ # AI agents
├── rag/ # RAG (embeddings, vector store, ingestion)
│ └── connectors/ # Sync source connectors
└── commands/ # CLI commands
```
## Key Conventions
- `db.flush()` in repositories, not `commit()`
- Services raise `NotFoundError`, `AlreadyExistsError`
- Separate `Create`, `Update`, `Response` schemas
- Commands auto-discovered from `app/commands/`
- Document ingestion via CLI and API upload
- Sync sources: configurable connectors with scheduled sync
## More Info
- `docs/architecture.md` - Architecture details
- `docs/adding_features.md` - How to add features
- `docs/testing.md` - Testing guide
- `docs/patterns.md` - Code patterns
CLAUDE.md
+153
View File
@@ -0,0 +1,153 @@
# CLAUDE.md
## Project Overview
**ai_agent** - FastAPI application generated with [Full-Stack AI Agent Template](https://github.com/vstorm-co/full-stack-ai-agent-template).
**Stack:** FastAPI + Pydantic v2, PostgreSQL (async via asyncpg)
, JWT + API Key auth, Redis, PydanticAI, RAG (milvus), Taskiq, Next.js 15 (i18n)
## Commands
```bash
# Backend
cd backend
uv run uvicorn app.main:app --reload --port 8033
uv run pytest
uv run pytest tests/test_file.py::test_name -v
uv run ruff check . --fix && uv run ruff format .
uv run ty check
# Database migrations
uv run alembic upgrade head
uv run alembic revision --autogenerate -m "Description"
# Frontend
cd frontend
bun dev
bun test
bun run lint
# Docker
docker compose up -d
# RAG
uv run ai_agent rag-collections
uv run ai_agent rag-ingest /path/to/file.pdf --collection docs
uv run ai_agent rag-search "query" --collection docs
# Sync Sources
uv run ai_agent cmd rag-sources
uv run ai_agent cmd rag-source-add
uv run ai_agent cmd rag-source-sync
```
## Project Structure
```
backend/app/
├── main.py # FastAPI app with lifespan (startup/shutdown)
├── api/
│ ├── deps.py # Annotated DI aliases (DBSession, CurrentUser, *Svc)
│ ├── exception_handlers.py
│ └── routes/v1/ # HTTP endpoints — call services, never repos
├── core/
│ ├── config.py # pydantic-settings Settings class
│ ├── security.py # JWT (PyJWT), bcrypt password hashing, API key verification
│ ├── exceptions.py # Domain exceptions (AppException → NotFoundError, etc.)
│ └── middleware.py # RequestID, SecurityHeaders, CORS
├── db/
│ ├── base.py # DeclarativeBase, TimestampMixin, naming convention
│ ├── session.py # Engine, async_session_maker, get_db_session (auto-commit)
│ └── models/ # SQLAlchemy models (Mapped[] type hints)
├── schemas/ # Pydantic v2 models: *Create, *Update, *Read, *List
├── repositories/ # Data access functions — db.flush(), never commit
├── services/ # Business logic — flat *.py for thin domains, subpackage for thick
│ ├── user.py # thin: just a class with db + repo calls
│ ├── rag/ # thick: ingestion + vectorstore + embeddings + connectors
├── agents/ # AI agent wrappers + tools
├── worker/ # Background tasks (Celery/Taskiq/ARQ + in-process)
└── commands/ # CLI commands (auto-discovered)
```
## Architecture: Routes → Services → Repositories
**Routes** (`api/routes/v1/`) — HTTP layer only: validate input via Pydantic, call service, return response. Never import repositories.
**Services** (`services/`) — Business logic: class with `__init__(self, db)`, orchestrate repos, raise domain exceptions (`NotFoundError`, `AlreadyExistsError`, etc.).
**Repositories** (`repositories/`) — Pure data access functions. Always use `db.flush()` + `db.refresh()`, NEVER `db.commit()`. Session auto-commits via `get_db_session`.
## Dependency Injection Pattern
All DI uses `Annotated` type aliases defined in `api/deps.py`:
```python
# deps.py
DBSession = Annotated[AsyncSession, Depends(get_db_session)]
UserSvc = Annotated[UserService, Depends(get_user_service)]
CurrentUser = Annotated[User, Depends(get_current_user)]
CurrentAdmin = Annotated[User, Depends(RoleChecker(UserRole.ADMIN))]
# Route usage — no raw Depends() in function signatures
@router.get("/{id}", response_model=ConversationRead)
async def get_conversation(
id: UUID, service: ConversationSvc, user: CurrentUser
) -> Any:
return await service.get(id, user_id=user.id)
```
## Schema Conventions (Pydantic v2)
- Base: `BaseSchema` with `ConfigDict(from_attributes=True, str_strip_whitespace=True)`
- Separate models per operation: `*Create`, `*Update`, `*Read`
- List responses: `*List` with `items: list[*Read]` and `total: int`
- Update schemas: all fields `Optional` (`str | None = None`)
- Use `Field(max_length=255)`, `Field(min_length=8)`, `EmailStr`
- `@field_validator` for deserialization (e.g., JSON string → dict for SQLite)
- IDs are `UUID` type
## Exception Handling
Domain exceptions in `core/exceptions.py` — all extend `AppException`:
| Exception | HTTP | Code |
|-----------|------|------|
| `NotFoundError` | 404 | `NOT_FOUND` |
| `AlreadyExistsError` | 409 | `ALREADY_EXISTS` |
| `ValidationError` | 422 | `VALIDATION_ERROR` |
| `AuthenticationError` | 401 | `AUTHENTICATION_ERROR` |
| `AuthorizationError` | 403 | `AUTHORIZATION_ERROR` |
| `BadRequestError` | 400 | `BAD_REQUEST` |
| `ExternalServiceError` | 503 | `EXTERNAL_SERVICE_ERROR` |
Always pass `message` and `details` dict: `raise NotFoundError(message="User not found", details={"user_id": id})`
## Response Format
```python
# Single item — use response_model
@router.get("/{id}", response_model=ConversationRead)
# List — return *List schema
@router.get("", response_model=ConversationList)
# Create — 201
@router.post("", response_model=UserRead, status_code=status.HTTP_201_CREATED)
# Delete — 204, no body
@router.delete("/{id}", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
# All route return types are -> Any (avoids double Pydantic validation)
```
## Key Conventions
- Return type `-> Any` on route handlers (response_model handles serialization)
- Use `Query(default, ge=0, le=100, description="...")` for query params
- Keyword-only args in repo functions: `create(db, *, email: str, name: str)`
- `__repr__` on all DB models
- `datetime.now(UTC)` not `datetime.utcnow()`
- `secrets.compare_digest()` for API key comparison
- `TypedDict` for lifespan state
- Imports: stdlib → third-party → local, with `TYPE_CHECKING` block for circular refs
CONTRIBUTING.md
+74
View File
@@ -0,0 +1,74 @@
# Contributing to ai_agent
## Development setup
```bash
# Backend (uv-based)
cd backend
uv sync # install all deps including dev extras
cp .env.example .env # then fill in required vars (see ENV_VARS.md)
uv run uvicorn app.main:app --reload --port 8033
uv run alembic upgrade head # apply migrations
# Frontend (bun-based)
cd ../frontend
bun install
bun dev # http://localhost:3033
# Or everything in Docker
docker compose up
```
## Code style
- **Python:** ruff (`uv run ruff check . --fix && uv run ruff format .`). Line length 120.
- **Type hints:** modern syntax (`str | None`, `list[X]`, `dict[str, Any]`). Use `Annotated[T, Depends(...)]` for DI in route signatures.
- **TypeScript:** strict mode, no `any` unless typed external API. ESLint + Prettier (run `bun run lint`).
- **Imports:** stdlib → third-party → local, separated by blank lines. Use `TYPE_CHECKING` block to break circular refs.
- **Datetime:** `datetime.now(UTC)` not `datetime.utcnow()`.
- **Comparisons:** `secrets.compare_digest()` for tokens/keys (constant-time).
## Testing
```bash
cd backend
uv run pytest # all backend tests
uv run pytest tests/test_file.py::test -v # single test
uv run pytest -k "name_substring" -v # by name pattern
uv run pytest --cov=app # with coverage
```
```bash
cd frontend
bun test # vitest
bunx tsc --noEmit # type-check without emit
```
## Architecture rules
- **Routes** never import repositories directly. Always go through a service.
- **Services** raise domain exceptions (`NotFoundError`, `AlreadyExistsError`) — never return `None` for "not found".
- **Repositories** use `db.flush()` + `db.refresh()`, NEVER `db.commit()` (session auto-commits in `get_db_session`).
- **Pydantic schemas:** separate `*Create`, `*Update`, `*Read`, `*List` per operation.
- **Migrations:** one Alembic revision per logical change; never edit a merged migration.
See `docs/architecture.md` for the full layered architecture rules.
## Pre-commit
Configured via `.pre-commit-config.yaml`. Install once:
```bash
uv run pre-commit install
```
Will run ruff + (frontend lint if present) on every commit. Bypass with `--no-verify` only when fixing a hook bug.
## Pull-request checklist
- [ ] `uv run ruff check . && uv run ruff format --check .` clean
- [ ] `cd frontend && bunx tsc --noEmit` clean
- [ ] Tests added for new code paths; `uv run pytest` green
- [ ] If schema changed: alembic migration committed (`uv run alembic revision --autogenerate -m "..."`)
- [ ] Updated `ENV_VARS.md` if new env vars added
- [ ] Updated `CHANGELOG.md` (if applicable)
ENV_VARS.md
+68
View File
@@ -0,0 +1,68 @@
# Environment variables
Reference for `ai_agent` runtime configuration. The
authoritative source is `backend/.env.example` — this doc explains what each
group is for and which are required vs optional.
> Quick start: copy `backend/.env.example` to `backend/.env` and fill in the
> blanks marked **Required**. Defaults are sensible for local development.
## Project
| Variable | Required | Default | Description |
|---|---|---|---|
| `PROJECT_NAME` | optional | `ai_agent` | Used in logs, OpenAPI title, email templates |
| `DEBUG` | optional | `true` | When `true`, FastAPI returns full tracebacks |
| `ENVIRONMENT` | optional | `local` | Free-form tag: `local` / `staging` / `production` |
| `TIMEZONE` | optional | `UTC` | IANA TZ name (e.g. `Europe/Warsaw`) |
| `BACKEND_URL` | optional | `http://localhost:8033` | Used by frontend BFF + email link generation |
| `FRONTEND_URL` | optional | `http://localhost:3033` | Used by password-reset / magic-link emails |
## Auth & secrets
| Variable | Required | Default | Description |
|---|---|---|---|
| `SECRET_KEY` | **required in prod** | (generated) | JWT signing key. Rotating invalidates all tokens |
| `API_KEY` | **required in prod** | (generated) | Static admin/service-to-service key for `X-API-Key` header |
| `ACCESS_TOKEN_EXPIRE_MINUTES` | optional | `30` | JWT access token lifetime |
| `REFRESH_TOKEN_EXPIRE_MINUTES` | optional | `10080` | JWT refresh token lifetime (7 days) |
| `GOOGLE_OAUTH_CLIENT_ID` | required | — | From Google Cloud Console → OAuth credentials |
| `GOOGLE_OAUTH_CLIENT_SECRET` | required | — | jw |
## Database
| Variable | Required | Default | Description |
|---|---|---|---|
| `DATABASE_URL` | **required** | `postgresql+asyncpg://...` | Full async connection string |
| `DB_POOL_SIZE` | optional | `5` | Number of long-lived connections |
| `DB_MAX_OVERFLOW` | optional | `10` | Burst capacity above pool size |
## LLM / AI
| Variable | Required | Default | Description |
|---|---|---|---|
| `OPENAI_API_KEY` | **required** | — | From platform.openai.com |
| `AI_MODEL` | optional | `gpt-5.5` | Default model used by agent (provider-specific) |
| `LOGFIRE_TOKEN` | optional | — | When set, ships traces to Logfire (logfire.pydantic.dev) |
## RAG (milvus)
| Variable | Required | Default | Description |
|---|---|---|---|
| `MILVUS_URI` | **required** | `http://localhost:19530` | Milvus gRPC endpoint |
| `MILVUS_TOKEN` | optional | — | Auth token (cloud Milvus) |
## Redis
| Variable | Required | Default | Description |
|---|---|---|---|
| `REDIS_URL` | **required** | `redis://localhost:6379/0` | Used by session store |
## Validation
```bash
# Confirm settings load without errors:
cd backend && uv run python -c "from app.core.config import settings; print(settings.model_dump_json(indent=2))"
```
If any **Required** var is missing, FastAPI raises `pydantic_settings.SettingsError` on startup — check the message for which field.
MANUAL_STEPS.md
+79
View File
@@ -0,0 +1,79 @@
# Manual setup steps for ai_agent
The generator created the code. These are the **one-time external setup steps**
that can't be automated — accounts to create, keys to copy, services to provision.
> Skip ahead to "After every deploy" at the bottom for things you'll re-do
> regularly. Items above are one-time per environment.
---
## Secrets
```bash
cp backend/.env.example backend/.env
```
Then in `backend/.env`:
- [ ] **`SECRET_KEY`** — replace with a fresh value: `openssl rand -hex 32`
- [ ] **`API_KEY`** — replace with a fresh value: `openssl rand -hex 32`
These are used to sign JWTs and authenticate service-to-service calls. Rotate at every environment promotion (dev → staging → prod each get their own).
## PostgreSQL
- [ ] Provision a PostgreSQL ≥ 14 instance (local: `docker compose up -d db`; managed: Neon / Supabase / RDS / Cloud SQL).
- [ ] Set `DATABASE_URL` in `.env` to the **async** connection string: `postgresql+asyncpg://user:pass@host:5432/dbname`.
- [ ] Run migrations: `cd backend && uv run alembic upgrade head`.
## OpenAI
- [ ] Create API key at https://platform.openai.com/api-keys.
- [ ] Set `OPENAI_API_KEY` in `.env`.
- [ ] (Optional) Set spending limit on OpenAI dashboard to avoid surprise bills.
## Google OAuth
- [ ] Go to https://console.cloud.google.com/ → APIs & Services → Credentials → Create OAuth client ID.
- [ ] Application type: **Web application**.
- [ ] Authorized redirect URIs: `http://localhost:3033/auth/callback`. Add prod URL when deploying.
- [ ] Copy **Client ID** + **Client secret** → set `GOOGLE_OAUTH_CLIENT_ID` + `GOOGLE_OAUTH_CLIENT_SECRET` in `.env`.
## RAG (milvus)
- [ ] Local: `docker compose up -d milvus etcd minio` (already in `docker-compose.yml`).
- [ ] Cloud: provision via Zilliz Cloud, set `MILVUS_URI` + `MILVUS_TOKEN`.
- [ ] (Optional) Ingest seed documents: `uv run ai_agent rag-ingest /path/to/file.pdf --collection docs`.
## Redis
- [ ] Local: `docker compose up -d redis` (already in compose file).
- [ ] Managed: Upstash / Redis Cloud / ElastiCache. Set `REDIS_URL` in `.env`.
## Logfire (Pydantic observability)
- [ ] Create account at https://logfire.pydantic.dev.
- [ ] Run `uv run logfire auth` once locally to bootstrap.
- [ ] Get write token → set `LOGFIRE_TOKEN` in `.env` for non-local environments.
---
## After every deploy
- [ ] Run database migrations: `alembic upgrade head` (CI step or post-deploy job).
- [ ] Smoke test `/api/v1/health` returns `{"status": "ok"}`.
- [ ] Frontend loads, login → dashboard flow works.
- [ ] Logs flowing to your aggregator.
---
## Where to find more
- `ENV_VARS.md` — exhaustive env var reference
- `docs/deploy.md` — platform-specific deployment recipes
- `SECURITY.md` — security model + production hardening checklist
- `CONTRIBUTING.md` — dev environment setup
- `docs/architecture.md` — codebase layered architecture rules
Makefile
+389
View File
@@ -0,0 +1,389 @@
.PHONY: install format lint test run clean help db-init dev dev-down dev-logs dev-rebuild dev-frontend docker-clean stage stage-down prod prod-down
# === Container Runtime Detection ============================================
# Auto-detect podman or docker. Override at invocation:
# make DOCKER=podman DOCKER_COMPOSE="podman compose" dev
DOCKER := $(shell command -v podman 2>/dev/null || command -v docker 2>/dev/null || echo docker)
ifneq ($(findstring podman,$(DOCKER)),)
DOCKER_COMPOSE := podman compose
else
DOCKER_COMPOSE := docker compose
endif
# === Environments ===========================================================
# `make dev` — local development ($(DOCKER_COMPOSE) .dev.yml + bind-mounted source)
# `make stage` — staging ($(DOCKER_COMPOSE) .yml — built images, no live reload)
# `make prod` — production ($(DOCKER_COMPOSE) .prod.yml — needs backend/.env + nginx)
# Each env has matching -down / -logs / -rebuild siblings.
# Wait for postgres to accept connections. Polls pg_isready instead of a
# fixed sleep — handles slow startups and cold-start image pulls.
define _wait_for_db
@echo "Waiting for PostgreSQL ($(1))..."
@for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do \
if $(DOCKER_COMPOSE) -f $(1) exec -T db pg_isready -U postgres >/dev/null 2>&1; then \
echo " ✅ DB ready"; exit 0; \
fi; \
printf '.'; sleep 2; \
done; \
echo " ❌ DB not ready after 30s — check 'make dev-logs'"; exit 1
endef
# === Local dev: build → up → migrate ===
# Idempotent — re-run anytime. Migrations are no-ops when already at head;
# admin seeding is a separate target (`make seed`) so re-running `make dev`
# doesn't keep retrying user creation.
dev:
@echo "▶ Building backend image…"
$(DOCKER_COMPOSE) -f docker-compose.dev.yml build app
@echo "▶ Starting services…"
@if ! $(DOCKER_COMPOSE) -f docker-compose.dev.yml up -d; then \
echo ""; \
echo "⚠ First start failed. Tearing down stale containers and retrying once…"; \
echo " (volumes preserved — DB data is safe; use 'make clean' for a full wipe)"; \
$(DOCKER_COMPOSE) -f docker-compose.dev.yml down --remove-orphans; \
$(DOCKER_COMPOSE) -f docker-compose.dev.yml up -d; \
fi
$(call _wait_for_db,docker-compose.dev.yml)
@echo "▶ Applying migrations…"
$(DOCKER_COMPOSE) -f docker-compose.dev.yml exec -T app ai_agent db upgrade
@echo ""
@echo "🚀 Dev stack ready:"
@echo " API: http://localhost:8033"
@echo " Docs: http://localhost:8033/docs"
@echo " Admin: http://localhost:8033/admin"
@echo " Frontend: http://localhost:3033 (run 'make dev-frontend' or 'cd frontend && bun dev')"
@echo ""
@echo "First time? Run 'make seed' to create the default admin user."
# === First-time setup: seed default admin user (one-shot) ===
# Skipped when admin@example.com already exists. Safe to run again — exits
# clean either way. Replace email/password before deploying anywhere real.
seed:
@echo "▶ Seeding admin user (admin@example.com / admin123)…"
@if $(DOCKER_COMPOSE) -f docker-compose.dev.yml exec -T app \
ai_agent user list 2>/dev/null \
| grep -q "admin@example.com"; then \
echo " (admin@example.com already exists — nothing to do)"; \
else \
$(DOCKER_COMPOSE) -f docker-compose.dev.yml exec -T app \
ai_agent user create \
--email admin@example.com --password admin123 --superuser \
&& echo " ✅ Admin created. Login at http://localhost:8033/admin"; \
fi
# Convenience: bootstrap a fresh checkout end-to-end.
bootstrap: dev seed
dev-down:
$(DOCKER_COMPOSE) -f docker-compose.dev.yml down
# Full wipe — containers, networks, AND volumes. Use after a corrupted state
# (e.g. detached networks, port conflicts that left orphans). DESTROYS DB data.
docker-clean:
@echo "▶ Removing containers, networks, AND volumes for the dev stack…"
@echo " ⚠️ This deletes all local DB data and uploaded files."
$(DOCKER_COMPOSE) -f docker-compose.dev.yml down -v --remove-orphans
@echo "✅ Cleaned. Run 'make dev' to start fresh."
dev-logs:
$(DOCKER_COMPOSE) -f docker-compose.dev.yml logs -f
dev-rebuild:
$(DOCKER_COMPOSE) -f docker-compose.dev.yml build --no-cache app
$(DOCKER_COMPOSE) -f docker-compose.dev.yml up -d --force-recreate app
dev-frontend:
$(DOCKER_COMPOSE) -f docker-compose.frontend.yml up -d
@echo ""
@echo "✅ Frontend at http://localhost:3033 (backend must be up — 'make dev')"
# === Staging: built images, no bind mounts (production-like, local DB) ===
stage:
$(DOCKER_COMPOSE) -f docker-compose.yml up -d --build
$(call _wait_for_db,docker-compose.yml)
$(DOCKER_COMPOSE) -f docker-compose.yml exec -T app ai_agent db upgrade
@echo "✅ Staging stack at http://localhost:8033"
stage-down:
$(DOCKER_COMPOSE) -f docker-compose.yml down
# === Production: external Nginx, real secrets in backend/.env ===
prod:
@test -f backend/.env || (echo "❌ backend/.env missing — run 'cp backend/.env.example backend/.env' and fill in real secrets" && exit 1)
$(DOCKER_COMPOSE) --env-file backend/.env -f docker-compose.prod.yml up -d --build
@echo "▶ Waiting for DB then running migrations…"
@sleep 5
$(DOCKER_COMPOSE) --env-file backend/.env -f docker-compose.prod.yml exec -T app ai_agent db upgrade
@echo "✅ Production stack up. Configure your nginx host with nginx/nginx.conf"
prod-down:
$(DOCKER_COMPOSE) --env-file backend/.env -f docker-compose.prod.yml down
prod-logs:
$(DOCKER_COMPOSE) --env-file backend/.env -f docker-compose.prod.yml logs -f
# Legacy alias
quickstart: dev
# === Setup ===
install:
uv sync --directory backend --dev
@if git rev-parse --git-dir > /dev/null 2>&1; then \
uv run --directory backend pre-commit install; \
else \
echo "⚠️ Not a git repository - skipping pre-commit install"; \
echo " Run 'git init && make install' to set up pre-commit hooks"; \
fi
@echo ""
@echo "✅ Installation complete!"
@echo ""
@echo "Next steps:"
@echo " • make docker-db # Start PostgreSQL"
@echo " • make db-upgrade # Apply migrations"
@echo " • make run # Start development server"
@echo ""
@echo "Note: backend/.env is pre-configured for development"
# === Code Quality ===
format:
uv run --directory backend ruff format app tests cli
uv run --directory backend ruff check app tests cli --fix
lint:
uv run --directory backend ruff check app tests cli
uv run --directory backend ruff format app tests cli --check
uv run --directory backend ty check
# === Testing ===
test:
uv run --directory backend pytest tests/ -v
test-cov:
uv run --directory backend pytest tests/ -v --cov=app --cov-report=html --cov-report=term-missing
# === Database ===
db-init: docker-db
@echo "Waiting for PostgreSQL to be ready..."
@sleep 8
cd backend && uv run ai_agent db migrate -m "initial" || true
cd backend && uv run ai_agent db upgrade
@echo ""
@echo "✅ Database initialized!"
db-migrate:
@read -p "Migration message: " msg; \
uv run --directory backend ai_agent db migrate -m "$$msg"
db-upgrade:
uv run --directory backend ai_agent db upgrade
db-downgrade:
uv run --directory backend ai_agent db downgrade
db-current:
uv run --directory backend ai_agent db current
db-history:
uv run --directory backend ai_agent db history
# === Server ===
run:
uv run --directory backend ai_agent server run --reload
run-prod:
uv run --directory backend ai_agent server run --host 0.0.0.0 --port 8000
routes:
uv run --directory backend ai_agent server routes
# === Users ===
create-admin:
@echo "Creating admin user..."
uv run --directory backend ai_agent user create-admin
user-create:
uv run --directory backend ai_agent user create
user-list:
uv run --directory backend ai_agent user list
# === Taskiq ===
taskiq-worker:
uv run --directory backend ai_agent taskiq worker
taskiq-scheduler:
uv run --directory backend ai_agent taskiq scheduler
# === Docker: Backend (Development) ===
docker-up:
$(DOCKER_COMPOSE) build app
$(DOCKER_COMPOSE) up -d
@echo ""
@echo "✅ Backend services started!"
@echo " API: http://localhost:8033"
@echo " Docs: http://localhost:8033/docs"
@echo " PostgreSQL: localhost:5432"
@echo " Redis: localhost:6379"
docker-down:
$(DOCKER_COMPOSE) down
$(DOCKER_COMPOSE) -f docker-compose.frontend.yml down 2>/dev/null || true
docker-logs:
$(DOCKER_COMPOSE) logs -f
docker-build:
$(DOCKER_COMPOSE) build
docker-shell:
$(DOCKER_COMPOSE) exec app /bin/bash
# === Docker: Frontend (Development) ===
docker-frontend:
$(DOCKER_COMPOSE) -f docker-compose.frontend.yml up -d
@echo ""
@echo "✅ Frontend started!"
@echo " URL: http://localhost:3033"
@echo ""
@echo "Note: Backend must be running (make docker-up)"
docker-frontend-down:
$(DOCKER_COMPOSE) -f docker-compose.frontend.yml down
docker-frontend-logs:
$(DOCKER_COMPOSE) -f docker-compose.frontend.yml logs -f
docker-frontend-build:
$(DOCKER_COMPOSE) -f docker-compose.frontend.yml build
# === Docker: Production (with Traefik) ===
docker-prod:
$(DOCKER_COMPOSE) -f docker-compose.prod.yml up -d
@echo ""
@echo "✅ Production services started with Traefik!"
@echo ""
@echo "Endpoints (replace DOMAIN with your domain):"
@echo " Frontend: https://$$DOMAIN"
@echo " API: https://api.$$DOMAIN"
@echo " Traefik: https://traefik.$$DOMAIN"
docker-prod-down:
$(DOCKER_COMPOSE) -f docker-compose.prod.yml down
docker-prod-logs:
$(DOCKER_COMPOSE) -f docker-compose.prod.yml logs -f
docker-prod-build:
$(DOCKER_COMPOSE) -f docker-compose.prod.yml build
# === Docker: Individual Services ===
docker-db:
$(DOCKER_COMPOSE) up -d db
@echo ""
@echo "✅ PostgreSQL started on port 5432"
@echo " Connection: postgresql://postgres:postgres@localhost:5432/ai_agent"
docker-db-stop:
$(DOCKER_COMPOSE) stop db
docker-redis:
$(DOCKER_COMPOSE) up -d redis
@echo ""
@echo "✅ Redis started on port 6379"
docker-redis-stop:
$(DOCKER_COMPOSE) stop redis
# === Vercel (Frontend Deployment) ===
vercel-deploy:
cd frontend && npx vercel --prod
@echo ""
@echo "✅ Frontend deployed to Vercel!"
@echo " Set environment variables in Vercel dashboard:"
@echo " BACKEND_URL=https://api.your-domain.com"
@echo " BACKEND_WS_URL=wss://api.your-domain.com"
@echo " NEXT_PUBLIC_AUTH_ENABLED=true"
@echo " NEXT_PUBLIC_RAG_ENABLED=true"
# === Cleanup ===
clean:
find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
find . -type d -name .pytest_cache -exec rm -rf {} + 2>/dev/null || true
find . -type d -name .ruff_cache -exec rm -rf {} + 2>/dev/null || true
find . -type d -name .ty_cache -exec rm -rf {} + 2>/dev/null || true
rm -rf htmlcov/ .coverage coverage.xml
# === Help ===
help:
@echo ""
@echo "ai_agent - Available Commands"
@echo "======================================"
@echo ""
@echo "🚀 Bootstrap (first-time setup):"
@echo " make bootstrap 'make dev' + 'make seed' — full setup from a fresh clone"
@echo ""
@echo "Day-to-day dev:"
@echo " make dev Build + start dev stack + apply migrations (idempotent)"
@echo " make seed One-shot admin seed (admin@example.com / admin123)"
@echo " make dev-down Stop dev stack"
@echo " make dev-logs Tail dev container logs"
@echo " make dev-rebuild Force-rebuild backend image"
@echo " make docker-clean Wipe containers + networks + volumes (DESTROYS data)"
@echo " make dev-frontend Start frontend container (after 'make dev')"
@echo ""
@echo "📦 Other environments:"
@echo " make stage Production-like stack on localhost (no bind mounts)"
@echo " make prod Production stack (requires backend/.env + nginx)"
@echo ""
@echo "Setup (without Docker):"
@echo " make install Install Python deps + pre-commit hooks"
@echo ""
@echo "Development:"
@echo " make run Start dev server (with hot reload)"
@echo " make test Run tests"
@echo " make lint Check code quality"
@echo " make format Auto-format code"
@echo ""
@echo "Database:"
@echo " make db-init Initialize database (start + migrate)"
@echo " make db-migrate Create new migration"
@echo " make db-upgrade Apply migrations"
@echo " make db-downgrade Rollback last migration"
@echo " make db-current Show current migration"
@echo ""
@echo "Users:"
@echo " make create-admin Create admin user (for SQLAdmin access)"
@echo " make user-create Create new user (interactive)"
@echo " make user-list List all users"
@echo ""
@echo "RAG:"
@echo " uv run ai_agent rag-ingest <path> -c <collection> Ingest files"
@echo " uv run ai_agent rag-search <query> -c <collection> Search"
@echo " uv run ai_agent rag-collections List collections"
@echo " uv run ai_agent rag-sources List sync sources"
@echo " uv run ai_agent rag-source-add Add sync source"
@echo " uv run ai_agent rag-source-sync <id> Trigger sync"
@echo ""
@echo "Taskiq:"
@echo " make taskiq-worker Start Taskiq worker"
@echo " make taskiq-scheduler Start Taskiq scheduler"
@echo ""
@echo "Docker (Development):"
@echo " make docker-up Start backend services"
@echo " make docker-down Stop all services"
@echo " make docker-logs View backend logs"
@echo " make docker-build Build backend images"
@echo " make docker-frontend Start frontend (separate)"
@echo " make docker-frontend-down Stop frontend"
@echo " make docker-db Start only PostgreSQL"
@echo " make docker-redis Start only Redis"
@echo ""
@echo "Docker (Production with Traefik):"
@echo " make docker-prod Start production stack"
@echo " make docker-prod-down Stop production stack"
@echo " make docker-prod-logs View production logs"
@echo ""
@echo "Other:"
@echo " make routes Show all API routes"
@echo " make clean Clean cache files"
@echo ""
README.md
+281
View File
@@ -0,0 +1,281 @@
# ai_agent
My FastAPI project
> Generated with [Full-Stack AI Agent Template](https://github.com/vstorm-co/full-stack-ai-agent-template).
---
## Stack
| Component | Technology |
|-----------|-----------|
| **Backend** | FastAPI + Pydantic v2 |
| **Database** | PostgreSQL (async via asyncpg) |
| **Auth** | JWT + refresh tokens + API keys + OAuth |
| **Cache** | Redis |
| **AI Framework** | pydantic_ai (openai) |
| **RAG** | milvus vector store |
| **Tasks** | taskiq |
| **Frontend** | Next.js 15 + React 19 + Tailwind v4 |
---
## Prerequisites
| Tool | Version | Install |
|---|---|---|
| **Docker** | Desktop / Engine 24+ | <https://docs.docker.com/get-docker/> |
| **Make** | GNU Make 3.81+ (preinstalled on macOS/Linux) | Windows: install via [chocolatey](https://chocolatey.org/) `choco install make` or use WSL2 |
| **uv** | latest | `curl -LsSf https://astral.sh/uv/install.sh \| sh` |
| **bun** | 1.x | `curl -fsSL https://bun.sh/install \| bash` (or use `npm` / `pnpm` if you prefer) |
> **Windows users:** the Makefile and shell helpers assume bash. Use **WSL2** or **Git Bash** for the smoothest experience. The Docker workflow below works identically on macOS, Linux, and WSL2.
---
## Quick Start (Local Dev)
### First time
```bash
make bootstrap # = make dev + make seed
```
That's the only command you need on a fresh clone. After this, day-to-day is just `make dev`.
### Subsequent runs
```bash
make dev
```
`make dev` is **idempotent** — re-run it any time. It will:
1. Build the backend Docker image (cached after first run)
2. Start services via `docker-compose.dev.yml` (with hot-reload bind mounts)
3. Poll Postgres until it accepts connections (`pg_isready` — no fixed sleeps)
4. Apply pending Alembic migrations (no-op if already at head)
It does **not** re-seed the admin user — that lives in `make seed` and is run once. This way `make dev` stays cheap to re-run after every code/config change.
**Then access:**
- API: <http://localhost:8033>
- Docs: <http://localhost:8033/docs>
- Admin: <http://localhost:8033/admin> — `admin@example.com` / `admin123` after `make seed`
- Frontend: <http://localhost:3033> — start with `make dev-frontend` (Docker) or `cd frontend && bun install && bun dev` (local)
### Day-to-day commands
```bash
make dev # bootstrap or restart (idempotent, no admin re-seed)
make seed # one-shot admin creation (no-op if admin already exists)
make dev-down # stop everything
make dev-logs # tail logs (Ctrl-C to exit)
make dev-rebuild # force-rebuild backend image (after pyproject.toml change)
make dev-frontend # start the Next.js container
```
If you prefer running the backend on the host (not in Docker) — useful for breakpoints / IDE debugging:
```bash
make install # uv sync + pre-commit install
docker compose -f docker-compose.dev.yml up -d db redis milvus etcd minio
make db-upgrade # apply migrations
make run # run uvicorn locally with --reload
```
---
## Environments
| `make` target | Compose file | Use case |
|---|---|---|
| `make dev` | `docker-compose.dev.yml` | Local development with hot-reload + bind-mounted source. |
| `make stage` | `docker-compose.yml` | Production-like build, no bind mounts, runs on localhost. Good for sanity-checking before deploy. |
| `make prod` | `docker-compose.prod.yml` | Production. Requires `backend/.env` (copy from `backend/.env.example`, fill real secrets) and an external Nginx using `nginx/nginx.conf`. |
Each env has matching `-down`, `-logs`, `-rebuild` siblings (e.g. `make stage-down`).
---
## Project Structure
```
backend/app/
├── main.py # FastAPI app + lifespan
├── api/
│ ├── deps.py # Annotated DI aliases (DBSession, CurrentUser, *Svc)
│ ├── exception_handlers.py
│ └── routes/v1/ # HTTP endpoints — call services, never repos
├── core/
│ ├── config.py # pydantic-settings (reads .env)
│ ├── security.py # JWT, bcrypt, API key verification
│ ├── exceptions.py # AppException → NotFound / Auth / etc.
│ └── middleware.py
├── db/
│ ├── base.py # DeclarativeBase + TimestampMixin
│ └── models/ # SQLAlchemy models (Mapped[] type hints)
├── schemas/ # Pydantic v2: *Create / *Update / *Read / *List
├── repositories/ # Data access — db.flush() never commit
├── services/ # Business logic — raises domain exceptions
├── agents/ # AI agent wrappers + tools
├── rag/ # RAG: vectorstore + embeddings + ingestion + sources
│ └── connectors/ # Pluggable sync sources (Google Drive, S3, …)
├── worker/
│ ├── background/ # FastAPI BackgroundTasks fallback (in-process)
│ └── tasks/ # Distributed tasks (taskiq)
└── commands/ # Click CLI commands (auto-discovered by `ai_agent cmd …`)
frontend/src/
├── app/
│ ├── [locale]/ # next-intl routes (en/pl)
│ │ └── (dashboard)/ # Authenticated app
│ └── api/ # Server-side API proxies (forward auth cookies)
├── components/ # React components (chat, marketing, ui primitives)
├── hooks/ # useAuth, useChat, useConversations, …
├── stores/ # Zustand stores
└── lib/ # api-client, server-api, utils
```
---
## CLI
The generated project ships a Click CLI exposed as `ai_agent` (after `make install`):
```bash
ai_agent server run --reload # dev server
ai_agent db upgrade # apply migrations
ai_agent db migrate -m "message" # create new migration
ai_agent user create-admin # interactive admin creation
ai_agent rag-ingest <path> -c docs # ingest local files
ai_agent rag-search "query" -c docs # semantic search
ai_agent rag-collections # list collections
```
Run `make help` for a categorized list, or `ai_agent --help` for full CLI docs.
---
## Configuration
All backend config lives in `backend/.env` (committed for dev defaults). Key variables:
```bash
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=ai_agent
# OpenAI — required for chat + embeddings
OPENAI_API_KEY=sk-…
# Google OAuth (Sign in with Google)
GOOGLE_CLIENT_ID=…
GOOGLE_CLIENT_SECRET=…
```
See `backend/.env.example` for the full list with comments.
For production, **never** commit secrets — `backend/.env` is gitignored. Fill it with real values on the server (or inject them via your platform's secret manager: Doppler, AWS Secrets Manager, GitHub Actions secrets, etc.). The same `backend/.env` is used for dev and prod — there is no separate `.env.prod`.
---
## Development
| Command | What it does |
|---|---|
| `make test` | Run pytest |
| `make lint` | Run ruff check + format check + ty |
| `make format` | Auto-format with ruff |
| `make db-migrate` | Generate a new migration from model changes (interactive) |
| `make db-upgrade` | Apply pending migrations |
| `make db-downgrade` | Roll back one migration |
| `make db-current` | Show current head |
| `make create-admin` | Interactive admin creation |
| `make user-list` | List all users |
---
## RAG (Knowledge Base)
Using **milvus** as the vector store with **openai** embeddings.
```bash
# Ingest local files (recursive)
ai_agent rag-ingest /path/to/docs/ --collection documents --recursive
# Semantic search
ai_agent rag-search "your query" --collection documents
```
PDF parsing uses **pymupdf**. See `docs/howto/add-rag-source.md` to add a new source connector.
---
## Frontend
```bash
cd frontend
bun install
bun dev # http://localhost:3033
bun run lint
bun run build
```
The frontend talks to the backend through Next.js API route handlers in `src/app/api/*` (server-side proxy that forwards auth cookies to the FastAPI backend). Direct calls to `localhost:8033` from the browser are deliberately avoided.
i18n (PL + EN) ships out of the box via `next-intl`. Add a new locale by extending `messages/<lang>.json` and `src/i18n.ts`.
---
## Deployment
### Frontend → Vercel
```bash
cd frontend && npx vercel --prod
```
Set in the Vercel dashboard:
- `BACKEND_URL` = `https://api.your-domain.com`
- `BACKEND_WS_URL` = `wss://api.your-domain.com`
- `NEXT_PUBLIC_AUTH_ENABLED` = `true`
- `NEXT_PUBLIC_RAG_ENABLED` = `true`
### Backend → your server
```bash
# 1. SSH to the box, clone the repo
# 2. cp backend/.env.example backend/.env, fill in real secrets
# 3. Configure nginx using nginx/nginx.conf as reference
# 4. Bring up the stack:
make prod
# Day-to-day:
make prod-logs
make prod-down
```
Migrations run automatically on `make prod`. For a fresh deploy on a new host, the same `make prod` is the bootstrap command.
---
## Guides
| Guide | What |
|-------|-------|
| `docs/howto/add-api-endpoint.md` | Add a new REST endpoint |
| `docs/howto/add-agent-tool.md` | Create an agent tool |
| `docs/howto/customize-agent-prompt.md` | Tune system prompts |
| `docs/howto/add-background-task.md` | Add a background task |
| `docs/howto/add-rag-source.md` | Add a RAG document source |
| `docs/howto/add-sync-connector.md` | Build a custom sync connector |
---
*Generated with [Full-Stack AI Agent Template](https://github.com/vstorm-co/full-stack-ai-agent-template) v0.2.10.*
SECURITY.md
+55
View File
@@ -0,0 +1,55 @@
# Security
## Reporting a vulnerability
Email: **your@email.com** (or open a private security advisory on the repo). Please include:
- Affected version / commit
- Steps to reproduce
- Impact assessment (data exposure / privilege escalation / DoS / …)
We aim to acknowledge within 48h and ship a fix within 7 days for high-severity issues.
---
## Security model
### Authentication
- **JWT (`HS256`)** signed with `SECRET_KEY`. Access token TTL = `ACCESS_TOKEN_EXPIRE_MINUTES` (default 30 min). Refresh token TTL = `REFRESH_TOKEN_EXPIRE_MINUTES` (default 7 days).
- **Password hashing:** bcrypt via `passlib`. Plain passwords never persisted.
- **OAuth 2.0 (Google)** — auth-code flow. Token validated server-side, internal user record looked up/created by email.
- **Stateless JWT** — no DB session table. Logout is client-side (drop tokens). For server-side revocation, regenerate with `--session-management`.
- **Admin API key** — static `settings.API_KEY` matched via `X-API-Key` header for service-to-service calls. Constant-time compared with `secrets.compare_digest()`.
### Authorization
- **Role-based** via `RoleChecker` dep (`UserRole.USER` / `UserRole.ADMIN`).
- **Admin pages** require `role=admin`. Sensitive ops (impersonate user, system-health) gated separately.
### Transport / network
- **CORS** — origin list from `settings.CORS_ORIGINS`. Restrict to your domains in production.
- **HTTPS** — enforce via reverse proxy (Nginx / Traefik / ALB). Strict-Transport-Security header set in middleware when `ENVIRONMENT=production`.
- **CSP** — frontend sets `frame-ancestors 'none'` by default to prevent click-jacking. See `frontend/next.config.ts` headers block.
### Data
- **Secrets** — read from environment via `pydantic-settings`. Never committed. See `.env.example` + `ENV_VARS.md`.
- **Audit log** — admin-mutating actions (user updates, deletes, impersonations, role changes) recorded in `app_admin_audit_log` table with actor + IP + payload snapshot.
- **RAG documents** — file uploads scoped per-org. No public read endpoint; all retrieval happens server-side during chat.
### Hardening checklist for production
- [ ] Rotate `SECRET_KEY` and `API_KEY` from generated defaults.
- [ ] Set `DEBUG=false` and `ENVIRONMENT=production`.
- [ ] Restrict `CORS_ORIGINS` to your domain(s).
- [ ] Enforce HTTPS at the proxy layer.
- [ ] Run `pip-audit` / `bun audit` in CI for dependency vulnerabilities.
- [ ] Configure database backups + restore test schedule.
## Known limitations
- **No 2FA / MFA** out of the box. Plan to add TOTP via `pyotp` — see `notes/thingstofix.md` §A.13.
- **No SAML / OIDC** beyond Google OAuth. Enterprise SSO needs custom IdP integration.
- **No automatic PII redaction** in logs — be careful what you log.
- **No server-side session revocation** — JWTs valid until expiry. Compromised tokens require `SECRET_KEY` rotation (invalidates ALL sessions). Enable `--session-management` for selective revocation.
backend/.dockerignore
+57
View File
@@ -0,0 +1,57 @@
# Git
.git
.gitignore
# Python
__pycache__
*.py[cod]
*$py.class
*.so
.Python
.venv
venv/
ENV/
# IDE
.idea
.vscode
*.swp
*.swo
# Testing
.pytest_cache
.coverage
htmlcov/
.tox
.nox
# Documentation
docs/_build/
*.md
!README.md
# Build artifacts
dist/
build/
*.egg-info/
# Development files
.env
.env.local
*.db
*.sqlite
# Docker
Dockerfile*
docker-compose*.yml
.docker
# CI/CD
.github/
.gitlab-ci.yml
# Misc
.DS_Store
Thumbs.db
*.log
backend/.env.example
+103
View File
@@ -0,0 +1,103 @@
# ai_agent Environment Variables
# === Project ===
PROJECT_NAME=ai_agent
DEBUG=true
ENVIRONMENT=local
TIMEZONE=UTC # IANA timezone: UTC, Europe/Warsaw, America/New_York
MODELS_CACHE_DIR=./models_cache
# === Logfire ===
# Get your token at https://logfire.pydantic.dev
LOGFIRE_TOKEN=
LOGFIRE_SERVICE_NAME=ai_agent
LOGFIRE_ENVIRONMENT=development
# === PostgreSQL ===
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=ai_agent
# === JWT Auth ===
# Generate with: openssl rand -hex 32
SECRET_KEY=change-me-in-production-use-openssl-rand-hex-32
ACCESS_TOKEN_EXPIRE_MINUTES=10080
ALGORITHM=HS256
# === API Key Auth ===
API_KEY=change-me-in-production
API_KEY_HEADER=X-API-Key
# === Redis ===
REDIS_HOST=localhost
REDIS_PORT=6379
# REDIS_PASSWORD=
REDIS_DB=0
# === RAG Configuration ===
RAG_DEFAULT_COLLECTION=documents
RAG_TOP_K=10
RAG_CHUNK_SIZE=512
RAG_CHUNK_OVERLAP=50
RAG_CHUNKING_STRATEGY=recursive # recursive, markdown, or fixed
RAG_HYBRID_SEARCH=false # Enable BM25 + vector hybrid search
RAG_ENABLE_OCR=false # OCR fallback for scanned PDFs (requires tesseract-ocr installed)
# Vector Database (Milvus)
MILVUS_HOST=localhost
MILVUS_PORT=19530
MILVUS_DATABASE=default
MILVUS_TOKEN=root:Milvus
# Reranker
HF_TOKEN=
CROSS_ENCODER_MODEL=cross-encoder/ms-marco-MiniLM-L6-v2
# Image Description (LLM vision)
RAG_IMAGE_DESCRIPTION_MODEL= # empty = use AI_MODEL
# === Taskiq ===
TASKIQ_BROKER_URL=redis://localhost:6379/1
TASKIQ_RESULT_BACKEND=redis://localhost:6379/1
# === AI Agent (pydantic_ai, openai) ===
OPENAI_API_KEY=
AI_MODEL=gpt-5.5
AI_TEMPERATURE=0.7
AI_THINKING_ENABLED=false
AI_THINKING_EFFORT=medium # low, medium, high
# Embeddings
# OpenAI Embeddings
OPENAI_API_KEY=
EMBEDDING_MODEL=text-embedding-3-small
# Chunking
RAG_CHUNK_SIZE=512
RAG_CHUNK_OVERLAP=50
# Retrieval
RAG_DEFAULT_COLLECTION=documents
RAG_TOP_K=10
# Reranker
# PDF Parser
# Google Drive (optional, for document ingestion)
# === CORS ===
# JSON list of allowed origins (default: localhost:3000, localhost:8080)
# Note: "*" is blocked in production - specify explicit origins
CORS_ORIGINS=["http://localhost:3000","http://localhost:8080"]
# === Docker Production (Traefik) ===
# Domain for production deployment
DOMAIN=example.com
# Let's Encrypt email for SSL certificates
ACME_EMAIL=admin@example.com
# Traefik dashboard auth (generate with: htpasswd -nb admin password)
# TRAEFIK_DASHBOARD_AUTH=admin:$$apr1$$...
# Redis password for production
REDIS_PASSWORD=change-me-in-production
backend/.pre-commit-config.yaml
+25
View File
@@ -0,0 +1,25 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-toml
- id: check-added-large-files
args: ['--maxkb=1000']
- id: check-merge-conflict
- id: detect-private-key
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.0
hooks:
- id: ruff
args: [--fix]
- id: ruff-format
- repo: https://github.com/astral-sh/ty-pre-commit
rev: v0.0.29
hooks:
- id: ty
backend/Dockerfile
+54
View File
@@ -0,0 +1,54 @@
# Build stage
FROM python:3.13-slim AS builder
ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1
WORKDIR /app
# Install uv
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv
ENV UV_COMPILE_BYTECODE=1
ENV UV_LINK_MODE=copy
# Install dependencies
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=bind,source=uv.lock,target=uv.lock \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
uv sync --frozen --no-install-project --no-dev
# Copy application
COPY . /app
# Install project
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev
# Runtime stage
FROM python:3.13-slim
ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1
WORKDIR /app
# Copy virtual environment from builder
COPY --from=builder /app/.venv /app/.venv
COPY --from=builder /app /app
# Add venv to path
ENV PATH="/app/.venv/bin:$PATH"
# Create non-root user and writable directories
RUN adduser --disabled-password --gecos "" appuser && \
mkdir -p /app/media /app/data /app/chroma_data /app/models_cache && \
chown -R appuser:appuser /app
USER appuser
EXPOSE 8033
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD python -c "import httpx; httpx.get('http://localhost:8033/api/v1/health')" || exit 1
CMD ["python", "-m", "cli.commands", "server", "run", "--host", "0.0.0.0", "--port", "8033"]
backend/alembic.ini
+45
View File
@@ -0,0 +1,45 @@
# Alembic Configuration
[alembic]
script_location = alembic
prepend_sys_path = .
version_path_separator = os
# Human-readable migration file names: 2024-01-15_add_users_table.py
file_template = %%(year)d-%%(month).2d-%%(day).2d_%%(slug)s
[post_write_hooks]
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
qualname =
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
backend/alembic/env.py
+74
View File
@@ -0,0 +1,74 @@
"""Alembic migration environment."""
# ruff: noqa: I001 - Imports structured for Jinja2 template conditionals
from logging.config import fileConfig
from alembic import context
from sqlalchemy import engine_from_config, pool
from app.core.config import settings
from app.db.base import Base
# Import all models here to ensure they are registered with metadata
from app.db.models.user import User # noqa: F401
from app.db.models.conversation import Conversation, Message, ToolCall # noqa: F401
from app.db.models.message_rating import MessageRating # noqa: F401
from app.db.models.chat_file import ChatFile # noqa: F401
from app.db.models.rag_document import RAGDocument # noqa: F401
from app.db.models.sync_log import SyncLog # noqa: F401
from app.db.models.sync_source import SyncSource # noqa: F401
config = context.config
if config.config_file_name is not None:
fileConfig(config.config_file_name)
target_metadata = Base.metadata
# Ensure SQLite data directory exists before connecting
def get_url() -> str:
"""Get database URL from settings."""
return settings.DATABASE_URL_SYNC
def run_migrations_offline() -> None:
"""Run migrations in 'offline' mode."""
url = get_url()
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online() -> None:
"""Run migrations in 'online' mode."""
configuration = config.get_section(config.config_ini_section) or {}
configuration["sqlalchemy.url"] = get_url()
connectable = engine_from_config(
configuration,
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
)
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
backend/alembic/script.py.mako
+27
View File
@@ -0,0 +1,27 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision: str = ${repr(up_revision)}
down_revision: Union[str, None] = ${repr(down_revision)}
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
def upgrade() -> None:
${upgrades if upgrades else "pass"}
def downgrade() -> None:
${downgrades if downgrades else "pass"}
backend/alembic/versions/.gitkeep
View File
backend/alembic/versions/0000_create_users.py
+55
View File
@@ -0,0 +1,55 @@
"""create users table
Revision ID: 0000_users
Revises:
Create Date: 2026-06-11T07:19:21.193991+00:00
Base table required by every later migration. Mirrors the current User model
including is_app_admin (later flagged in 0003 — included here so the table
is usable immediately when enable_teams=false) and onboarding_completed_at
(0016 mirror — same reason). OAuth columns are present only when an OAuth
provider was selected, keeping the schema minimal for password-only setups.
"""
import sqlalchemy as sa
from sqlalchemy.dialects.postgresql import UUID as PG_UUID
from alembic import op
revision = "0000_users"
down_revision = None
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"users",
sa.Column(
"id",
PG_UUID(as_uuid=True),
primary_key=True,
server_default=sa.text("gen_random_uuid()"),
),
sa.Column("email", sa.String(255), nullable=False, unique=True, index=True),
sa.Column("hashed_password", sa.String(255), nullable=True),
sa.Column("full_name", sa.String(255), nullable=True),
sa.Column("is_active", sa.Boolean(), nullable=False, server_default=sa.true()),
sa.Column("role", sa.String(50), nullable=False, server_default="user"),
sa.Column("is_app_admin", sa.Boolean(), nullable=False, server_default=sa.false()),
sa.Column("avatar_url", sa.String(500), nullable=True),
sa.Column("onboarding_completed_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("oauth_provider", sa.String(32), nullable=True, index=True),
sa.Column("oauth_id", sa.String(255), nullable=True, index=True),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
)
def downgrade() -> None:
op.drop_table("users")
backend/alembic/versions/0001_create_organization_tables.py
+18
View File
@@ -0,0 +1,18 @@
"""create organization tables — skipped (enable_teams=false or no SQL DB)
Revision ID: 0001_org
"""
# This migration is a no-op when enable_teams is false.
revision = "0001_org"
down_revision = "0000_users"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0002_backfill_personal_orgs.py
+17
View File
@@ -0,0 +1,17 @@
"""backfill personal orgs — skipped (enable_teams=false or no SQL DB)
Revision ID: 0002_backfill_orgs
"""
revision = "0002_backfill_orgs"
down_revision = "0001_org"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0003_add_is_app_admin.py
+17
View File
@@ -0,0 +1,17 @@
"""add is_app_admin — skipped (enable_teams=false or no SQL DB)
Revision ID: 0003_is_app_admin
"""
revision = "0003_is_app_admin"
down_revision = "0002_backfill_orgs"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0004_5_create_core_tables.py
+246
View File
@@ -0,0 +1,246 @@
"""create core tables (conversations, messages, chat_files, sessions, rag, channels)
Revision ID: 0004_5_core_tables
Revises: 0004_audit_log
Create Date: 2026-06-11T07:19:21.193991+00:00
Creates the core conversational, RAG, channel, and session tables that
later migrations modify (0005, 0006, 0007, 0009). Splits feature-flagged
groups so the schema only contains what was selected at generation time.
"""
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
from sqlalchemy.dialects.postgresql import UUID as PG_UUID
from alembic import op
revision = "0004_5_core_tables"
down_revision = "0004_audit_log"
branch_labels = None
depends_on = None
def _id_col() -> sa.Column:
return sa.Column(
"id",
PG_UUID(as_uuid=True),
primary_key=True,
server_default=sa.text("gen_random_uuid()"),
)
def _user_fk(*, nullable: bool, ondelete: str = "CASCADE") -> sa.Column:
return sa.Column(
"user_id",
PG_UUID(as_uuid=True),
sa.ForeignKey("users.id", ondelete=ondelete),
nullable=nullable,
)
_UUID = PG_UUID(as_uuid=True)
_JSONB = postgresql.JSONB(astext_type=sa.Text())
def upgrade() -> None:
op.create_table(
"conversations",
_id_col(),
_user_fk(nullable=True),
sa.Column("title", sa.String(255), nullable=True),
sa.Column("is_archived", sa.Boolean(), nullable=False, server_default=sa.false()),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_conversations_user_id", "conversations", ["user_id"])
op.create_table(
"messages",
_id_col(),
sa.Column(
"conversation_id",
_UUID,
sa.ForeignKey("conversations.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("role", sa.String(20), nullable=False),
sa.Column("content", sa.Text(), nullable=False),
sa.Column("model_name", sa.String(100), nullable=True),
sa.Column("tokens_used", sa.Integer(), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_messages_conversation_id", "messages", ["conversation_id"])
op.create_table(
"tool_calls",
_id_col(),
sa.Column(
"message_id", _UUID, sa.ForeignKey("messages.id", ondelete="CASCADE"), nullable=False
),
sa.Column("tool_call_id", sa.String(100), nullable=False),
sa.Column("tool_name", sa.String(100), nullable=False),
sa.Column("args", _JSONB, nullable=False),
sa.Column("result", sa.Text(), nullable=True),
sa.Column("status", sa.String(20), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("duration_ms", sa.Integer(), nullable=True),
)
op.create_index("ix_tool_calls_message_id", "tool_calls", ["message_id"])
op.create_table(
"chat_files",
_id_col(),
_user_fk(nullable=False),
sa.Column(
"message_id", _UUID, sa.ForeignKey("messages.id", ondelete="CASCADE"), nullable=True
),
sa.Column("filename", sa.String(255), nullable=False),
sa.Column("mime_type", sa.String(100), nullable=False),
sa.Column("size", sa.Integer(), nullable=False),
sa.Column("storage_path", sa.String(500), nullable=False),
sa.Column("file_type", sa.String(20), nullable=False),
sa.Column("parsed_content", sa.Text(), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_chat_files_user_id", "chat_files", ["user_id"])
op.create_table(
"conversation_shares",
_id_col(),
sa.Column(
"conversation_id",
_UUID,
sa.ForeignKey("conversations.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"shared_by", _UUID, sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False
),
sa.Column(
"shared_with", _UUID, sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=True
),
sa.Column("share_token", sa.String(64), nullable=True, unique=True),
sa.Column("permission", sa.String(10), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.UniqueConstraint("conversation_id", "shared_with", name="uq_share_conv_user"),
)
op.create_index(
"ix_conversation_shares_conversation_id", "conversation_shares", ["conversation_id"]
)
op.create_index("ix_conversation_shares_shared_with", "conversation_shares", ["shared_with"])
op.create_table(
"message_ratings",
_id_col(),
sa.Column(
"message_id", _UUID, sa.ForeignKey("messages.id", ondelete="CASCADE"), nullable=False
),
_user_fk(nullable=False),
sa.Column("rating", sa.Integer(), nullable=False),
sa.Column("comment", sa.Text(), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
sa.CheckConstraint("rating IN (1, -1)", name="message_ratings_ck_rating_value_check"),
sa.UniqueConstraint("message_id", "user_id", name="uq_message_user_rating"),
)
op.create_index("ix_message_ratings_message_id", "message_ratings", ["message_id"])
op.create_index("ix_message_ratings_user_id", "message_ratings", ["user_id"])
op.create_table(
"rag_documents",
_id_col(),
sa.Column("collection_name", sa.String(255), nullable=False),
sa.Column("filename", sa.String(255), nullable=False),
sa.Column("filesize", sa.Integer(), nullable=False),
sa.Column("filetype", sa.String(20), nullable=False),
sa.Column("storage_path", sa.String(500), nullable=True),
sa.Column("status", sa.String(20), nullable=False),
sa.Column("error_message", sa.Text(), nullable=True),
sa.Column("vector_document_id", sa.String(255), nullable=True),
sa.Column("chunk_count", sa.Integer(), nullable=False, server_default="0"),
sa.Column(
"started_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_rag_documents_collection_name", "rag_documents", ["collection_name"])
op.create_table(
"sync_sources",
_id_col(),
sa.Column("name", sa.String(255), nullable=False),
sa.Column("connector_type", sa.String(20), nullable=False),
sa.Column("collection_name", sa.String(255), nullable=False),
sa.Column("config", _JSONB, server_default="{}", nullable=False),
sa.Column("sync_mode", sa.String(20), nullable=False),
sa.Column("schedule_minutes", sa.Integer(), nullable=True),
sa.Column("is_active", sa.Boolean(), nullable=False, server_default=sa.true()),
sa.Column("last_sync_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("last_sync_status", sa.String(20), nullable=True),
sa.Column("last_error", sa.Text(), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_sync_sources_collection_name", "sync_sources", ["collection_name"])
op.create_table(
"sync_logs",
_id_col(),
sa.Column("source", sa.String(20), nullable=False),
sa.Column("collection_name", sa.String(255), nullable=False),
sa.Column(
"sync_source_id",
_UUID,
sa.ForeignKey("sync_sources.id", ondelete="SET NULL"),
nullable=True,
),
sa.Column("status", sa.String(20), nullable=False),
sa.Column("mode", sa.String(20), nullable=False),
sa.Column("total_files", sa.Integer(), nullable=False, server_default="0"),
sa.Column("ingested", sa.Integer(), nullable=False, server_default="0"),
sa.Column("updated", sa.Integer(), nullable=False, server_default="0"),
sa.Column("skipped", sa.Integer(), nullable=False, server_default="0"),
sa.Column("failed", sa.Integer(), nullable=False, server_default="0"),
sa.Column("error_message", sa.Text(), nullable=True),
sa.Column(
"started_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_sync_logs_collection_name", "sync_logs", ["collection_name"])
def downgrade() -> None:
op.drop_table("sync_logs")
op.drop_table("sync_sources")
op.drop_table("rag_documents")
op.drop_table("message_ratings")
op.drop_table("conversation_shares")
op.drop_table("chat_files")
op.drop_table("tool_calls")
op.drop_table("messages")
op.drop_table("conversations")
backend/alembic/versions/0004_create_audit_log.py
+17
View File
@@ -0,0 +1,17 @@
"""create audit log — skipped (enable_teams=false or no SQL DB)
Revision ID: 0004_audit_log
"""
revision = "0004_audit_log"
down_revision = "0003_is_app_admin"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0005_add_org_id_to_conversations_and_rag.py
+17
View File
@@ -0,0 +1,17 @@
"""add organization_id to conversations — skipped (enable_teams=false or no SQL DB)
Revision ID: 0005_org_tenant_isolation
"""
revision = "0005_org_tenant_isolation"
down_revision = "0004_5_core_tables"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0006_backfill_org_id_on_conversations.py
+17
View File
@@ -0,0 +1,17 @@
"""backfill organization_id on conversations — skipped (enable_teams=false or no JWT)
Revision ID: 0006_backfill_conv_org
"""
revision = "0006_backfill_conv_org"
down_revision = "0005_org_tenant_isolation"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0007_create_knowledge_bases.py
+17
View File
@@ -0,0 +1,17 @@
"""create knowledge_bases — skipped (enable_teams/enable_rag/use_jwt=false or no SQL DB)
Revision ID: 0007_knowledge_bases
"""
revision = "0007_knowledge_bases"
down_revision = "0006_backfill_conv_org"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0008_backfill_default_kbs.py
+17
View File
@@ -0,0 +1,17 @@
"""backfill default KBs — skipped (enable_teams/enable_rag/use_jwt=false or no SQL DB)
Revision ID: 0008_backfill_default_kbs
"""
revision = "0008_backfill_default_kbs"
down_revision = "0007_knowledge_bases"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0009_conv_active_kb_ids.py
+17
View File
@@ -0,0 +1,17 @@
"""add active_knowledge_base_ids to conversations — skipped (enable_teams/enable_rag/use_jwt=false or no SQL DB)
Revision ID: 0009_conv_active_kb_ids
"""
revision = "0009_conv_active_kb_ids"
down_revision = "0008_backfill_default_kbs"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0010_org_billing_seats.py
+17
View File
@@ -0,0 +1,17 @@
"""add billing seats to organizations — skipped (enable_billing/enable_teams=false or no SQL DB)
Revision ID: 0010_org_billing_seats
"""
revision = "0010_org_billing_seats"
down_revision = "0009_conv_active_kb_ids"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0011_create_plan_price_tables.py
+17
View File
@@ -0,0 +1,17 @@
"""create plan and price tables — skipped (enable_billing=false or no SQL DB)
Revision ID: 0011_create_plan_price_tables
"""
revision = "0011_create_plan_price_tables"
down_revision = "0010_org_billing_seats"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0012_create_subscription_table.py
+17
View File
@@ -0,0 +1,17 @@
"""create subscription table — skipped (enable_billing=false or no SQL DB)
Revision ID: 0012_create_subscription_table
"""
revision = "0012_create_subscription_table"
down_revision = "0011_create_plan_price_tables"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0013_create_stripe_event_table.py
+17
View File
@@ -0,0 +1,17 @@
"""create stripe_event table — skipped (enable_billing=false or no SQL DB)
Revision ID: 0013_create_stripe_event_table
"""
revision = "0013_create_stripe_event_table"
down_revision = "0012_create_subscription_table"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0014_credits_usage_events.py
+17
View File
@@ -0,0 +1,17 @@
"""create credit_transaction and usage_event — skipped (enable_billing/enable_credits_system=false or no SQL DB)
Revision ID: 0014_credits_usage_events
"""
revision = "0014_credits_usage_events"
down_revision = "0013_create_stripe_event_table"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0015_create_mv_usage_daily.py
+18
View File
@@ -0,0 +1,18 @@
"""create mv_usage_daily — skipped (PostgreSQL + billing + credits not all enabled)
Revision ID: 0015_create_mv_usage_daily
Revises: 0014_credits_usage_events
"""
revision = "0015_create_mv_usage_daily"
down_revision = "0014_credits_usage_events"
branch_labels = None
depends_on = None
def upgrade() -> None:
pass
def downgrade() -> None:
pass
backend/alembic/versions/0016_user_onboarding_at.py
+44
View File
@@ -0,0 +1,44 @@
"""add onboarding_completed_at to users
Revision ID: 0016_user_onboarding_at
Revises: 0015_create_mv_usage_daily
Create Date: 2026-05-08T00:00:00+00:00
Adds:
- users.onboarding_completed_at — nullable timestamptz; null means the user
hasn't completed onboarding yet, set when they finish the wizard.
"""
import sqlalchemy as sa
from alembic import op
revision = "0016_user_onboarding_at"
down_revision = "0015_create_mv_usage_daily"
branch_labels = None
depends_on = None
def upgrade() -> None:
# Idempotent — `0000_users` already includes this column when running on a
# fresh DB. Older deployments that stamped past 0000 still need it added.
bind = op.get_bind()
inspector = sa.inspect(bind)
cols = {c["name"] for c in inspector.get_columns("users")}
if "onboarding_completed_at" not in cols:
op.add_column(
"users",
sa.Column(
"onboarding_completed_at",
sa.DateTime(timezone=True),
nullable=True,
),
)
def downgrade() -> None:
bind = op.get_bind()
inspector = sa.inspect(bind)
cols = {c["name"] for c in inspector.get_columns("users")}
if "onboarding_completed_at" in cols:
op.drop_column("users", "onboarding_completed_at")
backend/alembic/versions/0018_user_slash_commands.py
+65
View File
@@ -0,0 +1,65 @@
"""create user_slash_commands table
Revision ID: 0018_user_slash_commands
Revises: 0017_create_api_keys_table
Create Date: 2026-05-10T00:00:00+00:00
Stores per-user slash command settings for the chat palette:
- Custom commands (``prompt`` is set) — quick prompt shortcuts.
- Built-in overrides (``prompt`` is NULL) — record only ``is_enabled`` for
one of the frontend's BUILTIN_COMMANDS, so users can hide ones they don't
use.
The unique ``(user_id, name)`` constraint prevents both classes of row from
co-existing for the same name on a single user.
"""
import sqlalchemy as sa
from sqlalchemy.dialects.postgresql import UUID as PG_UUID
from alembic import op
revision = "0018_user_slash_commands"
down_revision = "0016_user_onboarding_at"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"user_slash_commands",
sa.Column(
"id",
PG_UUID(as_uuid=True),
primary_key=True,
server_default=sa.text("gen_random_uuid()"),
),
sa.Column(
"user_id",
PG_UUID(as_uuid=True),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("name", sa.String(64), nullable=False),
sa.Column("prompt", sa.Text(), nullable=True),
sa.Column(
"is_enabled",
sa.Boolean(),
nullable=False,
server_default=sa.true(),
),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
sa.UniqueConstraint("user_id", "name", name="uq_user_slash_commands_user_name"),
)
op.create_index("ix_user_slash_commands_user_id", "user_slash_commands", ["user_id"])
def downgrade() -> None:
op.drop_index("ix_user_slash_commands_user_id", table_name="user_slash_commands")
op.drop_table("user_slash_commands")
backend/app/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""My FastAPI project"""
__version__ = "0.1.0"
backend/app/admin.py
+398
View File
@@ -0,0 +1,398 @@
"""SQLAdmin configuration with automatic model discovery."""
from typing import Any, ClassVar
from fastapi import FastAPI
from sqladmin import Admin, ModelView
from sqladmin.authentication import AuthenticationBackend
from sqlalchemy import String, inspect
from sqlalchemy.engine import Engine
from sqlalchemy.orm import DeclarativeBase
from starlette.requests import Request
from app.core.config import settings
from app.core.security import verify_password
from app.db.base import Base
from app.db.models.conversation import ToolCall
from app.db.models.user import User, UserRole
# Columns that should be excluded from forms (sensitive data)
SENSITIVE_COLUMN_PATTERNS: list[str] = [
"password",
"hashed_password",
"secret",
"token",
"api_key",
"refresh_token",
]
# Columns that should be searchable by default (string columns)
SEARCHABLE_COLUMN_TYPES: tuple[type, ...] = (String,)
# Columns that are auto-generated and should be excluded from create/edit forms
AUTO_GENERATED_COLUMNS: list[str] = [
"created_at",
"updated_at",
]
# Model icons mapping (model name -> Font Awesome icon)
MODEL_ICONS: dict[str, str] = {
"User": "fa-solid fa-user",
"Session": "fa-solid fa-key",
"Conversation": "fa-solid fa-comments",
"Message": "fa-solid fa-message",
"ToolCall": "fa-solid fa-wrench",
"Webhook": "fa-solid fa-link",
"WebhookDelivery": "fa-solid fa-paper-plane",
}
def discover_models(base: type[DeclarativeBase]) -> list[type]:
"""Discover all SQLAlchemy models registered with the given Base.
Args:
base: The SQLAlchemy DeclarativeBase class.
Returns:
List of model classes that inherit from the Base.
"""
return [mapper.class_ for mapper in base.registry.mappers]
def get_model_columns(model: type) -> list[str]:
"""Get all column names from a SQLAlchemy model.
Args:
model: The SQLAlchemy model class.
Returns:
List of column names.
"""
mapper: Any = inspect(model)
return [column.key for column in mapper.columns]
def get_searchable_columns(model: type) -> list[str]:
"""Get columns suitable for searching (String type columns).
Args:
model: The SQLAlchemy model class.
Returns:
List of searchable column names.
"""
mapper: Any = inspect(model)
searchable = []
for column in mapper.columns:
# Include String columns that are not sensitive
is_searchable_type = isinstance(column.type, SEARCHABLE_COLUMN_TYPES)
is_sensitive = any(pattern in column.key.lower() for pattern in SENSITIVE_COLUMN_PATTERNS)
if is_searchable_type and not is_sensitive:
searchable.append(column.key)
return searchable
def get_sortable_columns(model: type) -> list[str]:
"""Get columns suitable for sorting.
Args:
model: The SQLAlchemy model class.
Returns:
List of sortable column names.
"""
mapper: Any = inspect(model)
return [column.key for column in mapper.columns]
def get_form_excluded_columns(model: type) -> list[str]:
"""Get columns that should be excluded from create/edit forms.
Excludes sensitive columns and auto-generated columns.
Args:
model: The SQLAlchemy model class.
Returns:
List of column names to exclude from forms.
"""
excluded = []
for column_name in get_model_columns(model):
# Exclude sensitive columns
if (
any(pattern in column_name.lower() for pattern in SENSITIVE_COLUMN_PATTERNS)
or column_name in AUTO_GENERATED_COLUMNS
):
excluded.append(column_name)
return excluded
def pluralize(name: str) -> str:
"""Simple pluralization for model names.
Args:
name: Singular name.
Returns:
Pluralized name.
"""
if name.endswith("y"):
return name[:-1] + "ies"
elif name.endswith("s") or name.endswith("x") or name.endswith("ch") or name.endswith("sh"):
return name + "es"
return name + "s"
def create_model_admin(
model: type,
*,
name: str | None = None,
name_plural: str | None = None,
icon: str | None = None,
column_list: list[Any] | None = None,
column_searchable_list: list[Any] | None = None,
column_sortable_list: list[Any] | None = None,
form_excluded_columns: list[Any] | None = None,
can_create: bool = True,
can_edit: bool = True,
can_delete: bool = True,
can_view_details: bool = True,
) -> type[ModelView]:
"""Dynamically create a ModelView class for a SQLAlchemy model.
Args:
model: The SQLAlchemy model class.
name: Display name (defaults to model class name).
name_plural: Plural display name (defaults to auto-pluralized name).
icon: Font Awesome icon class.
column_list: Columns to display in list view.
column_searchable_list: Columns to enable search on.
column_sortable_list: Columns to enable sorting on.
form_excluded_columns: Columns to exclude from forms.
can_create: Allow creating new records.
can_edit: Allow editing records.
can_delete: Allow deleting records.
can_view_details: Allow viewing record details.
Returns:
A dynamically created ModelView subclass.
"""
import types
model_name = model.__name__
# Use provided values or generate defaults
_name = name or model_name
_name_plural = name_plural or pluralize(_name)
_icon = icon or MODEL_ICONS.get(model_name, "fa-solid fa-database")
# Get column attributes from the model
_column_list = column_list
if _column_list is None:
columns = get_model_columns(model)
_column_list = [getattr(model, col) for col in columns if hasattr(model, col)]
_column_searchable_list = column_searchable_list
if _column_searchable_list is None:
searchable = get_searchable_columns(model)
_column_searchable_list = [getattr(model, col) for col in searchable if hasattr(model, col)]
_column_sortable_list = column_sortable_list
if _column_sortable_list is None:
sortable = get_sortable_columns(model)
_column_sortable_list = [getattr(model, col) for col in sortable if hasattr(model, col)]
_form_excluded_columns = form_excluded_columns
if _form_excluded_columns is None:
excluded = get_form_excluded_columns(model)
_form_excluded_columns = [getattr(model, col) for col in excluded if hasattr(model, col)]
# Create class attributes in the exec_body callback
def exec_body(ns: dict[str, Any]) -> None:
ns["name"] = _name
ns["name_plural"] = _name_plural
ns["icon"] = _icon
ns["column_list"] = _column_list
ns["column_searchable_list"] = _column_searchable_list
ns["column_sortable_list"] = _column_sortable_list
ns["form_excluded_columns"] = _form_excluded_columns
ns["can_create"] = can_create
ns["can_edit"] = can_edit
ns["can_delete"] = can_delete
ns["can_view_details"] = can_view_details
# Add ClassVar type hints for sqladmin compatibility
ns["__annotations__"] = {
"column_list": ClassVar,
"column_searchable_list": ClassVar,
"column_sortable_list": ClassVar,
"form_excluded_columns": ClassVar,
"can_create": ClassVar,
"can_edit": ClassVar,
"can_delete": ClassVar,
"can_view_details": ClassVar,
}
# Create the class using types.new_class to properly pass model kwarg to metaclass
class_name = f"{model_name}Admin"
admin_class = types.new_class(
class_name,
(ModelView,),
{"model": model}, # Pass model to metaclass
exec_body,
)
return admin_class
def register_models_auto(
admin: Admin,
base: type[DeclarativeBase],
*,
exclude_models: list[type] | None = None,
custom_configs: dict[type, dict[str, Any]] | None = None,
) -> list[type[ModelView]]:
"""Auto-discover and register all models with the admin panel.
Args:
admin: The SQLAdmin instance.
base: The SQLAlchemy DeclarativeBase class.
exclude_models: Models to exclude from auto-registration.
custom_configs: Custom configuration overrides per model.
Returns:
List of registered ModelView classes.
"""
exclude_models = exclude_models or []
custom_configs = custom_configs or {}
registered_views: list[type[ModelView]] = []
models = discover_models(base)
for model in models:
if model in exclude_models:
continue
# Get custom config for this model if provided
config = custom_configs.get(model, {})
# Create and register the admin view
admin_class = create_model_admin(model, **config)
admin.add_view(admin_class)
registered_views.append(admin_class)
return registered_views
# SQLAdmin requires a synchronous engine
_sync_engine: Engine | None = None
def get_sync_engine() -> Engine:
"""Get or create the synchronous engine for SQLAdmin."""
global _sync_engine
if _sync_engine is None:
from sqlalchemy import create_engine
_sync_engine = create_engine(settings.DATABASE_URL_SYNC, echo=settings.DEBUG)
return _sync_engine
class AdminAuth(AuthenticationBackend):
"""Admin panel authentication backend.
Requires superuser credentials to access the admin panel.
"""
async def login(self, request: Request) -> bool:
"""Validate admin login credentials."""
form = await request.form()
email = form.get("username")
password = form.get("password")
if not email or not password:
return False
assert isinstance(email, str)
assert isinstance(password, str)
# Get user from database
from sqlalchemy.orm import Session as DBSession
with DBSession(get_sync_engine()) as session:
user = session.query(User).filter(User.email == email).first()
if (
user
and user.hashed_password
and verify_password(password, user.hashed_password)
and user.has_role(UserRole.ADMIN)
):
# Store user info in session
request.session["admin_user_id"] = str(user.id)
request.session["admin_email"] = user.email
return True
return False
async def logout(self, request: Request) -> bool:
"""Clear admin session."""
request.session.clear()
return True
async def authenticate(self, request: Request) -> bool:
"""Check if user is authenticated."""
admin_user_id = request.session.get("admin_user_id")
if not admin_user_id:
return False
# Verify user still exists and is superuser
from sqlalchemy.orm import Session as DBSession
with DBSession(get_sync_engine()) as session:
user = session.query(User).filter(User.id == admin_user_id).first()
if user and user.has_role(UserRole.ADMIN) and user.is_active:
return True
# User no longer valid, clear session
request.session.clear()
return False
CUSTOM_MODEL_CONFIGS: dict[type, dict[str, Any]] = {
User: {
"icon": "fa-solid fa-user",
"form_excluded_columns": [User.hashed_password, User.created_at, User.updated_at],
},
ToolCall: {
"icon": "fa-solid fa-wrench",
"can_create": False, # Tool calls are created by the agent
},
}
def setup_admin(app: FastAPI) -> Admin:
"""Setup SQLAdmin for the FastAPI app with automatic model discovery.
Automatically discovers all SQLAlchemy models from the Base registry
and creates admin views for them with sensible defaults.
Custom configurations can be provided in CUSTOM_MODEL_CONFIGS to override
default behavior for specific models.
"""
sync_engine = get_sync_engine()
authentication_backend = AdminAuth(secret_key=settings.SECRET_KEY)
admin = Admin(
app,
sync_engine,
title="ai_agent Admin",
authentication_backend=authentication_backend,
)
# Auto-register all models from Base with custom configs
register_models_auto(
admin,
Base,
custom_configs=CUSTOM_MODEL_CONFIGS,
)
return admin
backend/app/agents/__init__.py
+9
View File
@@ -0,0 +1,9 @@
"""AI Agents module using PydanticAI.
This module contains agents that handle AI-powered interactions.
Tools are defined in the tools/ subdirectory.
"""
from app.agents.assistant import AssistantAgent, Deps
__all__ = ["AssistantAgent", "Deps"]
backend/app/agents/assistant.py
+302
View File
@@ -0,0 +1,302 @@
"""Assistant agent with PydanticAI.
The main conversational agent that can be extended with custom tools.
"""
import logging
from dataclasses import dataclass, field
from typing import Any
from pydantic_ai import Agent, ModelRetry, RunContext
from pydantic_ai.capabilities import (
ReinjectSystemPrompt,
Thinking,
)
from pydantic_ai.messages import (
ModelRequest,
ModelResponse,
SystemPromptPart,
TextPart,
UserPromptPart,
)
from pydantic_ai.models.openai import OpenAIResponsesModel
from pydantic_ai.providers.openai import OpenAIProvider
from pydantic_ai.settings import ModelSettings
from app.agents.prompts import get_system_prompt_with_rag
from app.agents.tools import get_current_datetime
from app.agents.tools.chart_tool import create_chart
from app.agents.tools.rag_tool import search_knowledge_base
from app.core.config import settings
logger = logging.getLogger(__name__)
def _build_model(model_name: str):
"""OpenAI-only deployment."""
return OpenAIResponsesModel(
model_name or settings.AI_MODEL,
provider=OpenAIProvider(api_key=settings.OPENAI_API_KEY),
)
@dataclass
class Deps:
"""Dependencies for the assistant agent.
These are passed to tools via RunContext.
"""
user_id: str | None = None
user_name: str | None = None
metadata: dict[str, Any] = field(default_factory=dict)
class AssistantAgent:
"""Assistant agent wrapper for conversational AI.
Encapsulates agent creation and execution with tool support.
"""
def __init__(
self,
model_name: str | None = None,
temperature: float | None = None,
system_prompt: str | None = None,
thinking_effort: str | None = None,
):
self.model_name = model_name or settings.AI_MODEL
# ``temperature`` stays ``None`` when caller didn't set it — don't fall
# back to settings.AI_TEMPERATURE here. Reasoning/o-series models
# (gpt-5.5, o1, …) reject the parameter entirely, so we only forward
# it to the model when explicitly requested.
self.temperature = temperature
self.thinking_effort = (
thinking_effort
if thinking_effort is not None
else (settings.AI_THINKING_EFFORT if settings.AI_THINKING_ENABLED else None)
)
self.system_prompt = system_prompt or get_system_prompt_with_rag()
self._agent: Agent[Deps, str] | None = None
def _create_agent(self) -> Agent[Deps, str]:
"""Create and configure the PydanticAI agent."""
model = _build_model(self.model_name)
capabilities = [ReinjectSystemPrompt()]
if self.thinking_effort:
capabilities.append(Thinking(effort=self.thinking_effort))
# The unified ``Thinking()`` capability enables reasoning, but for the
# OpenAI Responses API it sets only the effort — not the *summary*
# field that controls whether the model streams reasoning summaries
# back to the client. Without ``openai_reasoning_summary`` set, the
# model reasons internally and we never see ThinkingPart events.
# ``openai_*``-prefixed fields on TypedDict settings are silently
# ignored by other providers, so this is safe to apply unconditionally.
model_settings: ModelSettings = ModelSettings()
if self.temperature is not None:
model_settings["temperature"] = self.temperature
if self.thinking_effort:
model_settings["openai_reasoning_summary"] = "auto" # type: ignore[typeddict-unknown-key] # ty: ignore[invalid-key]
agent = Agent[Deps, str](
model=model,
model_settings=model_settings,
system_prompt=self.system_prompt,
capabilities=capabilities,
)
self._register_tools(agent)
return agent
def _register_tools(self, agent: Agent[Deps, str]) -> None:
"""Register all tools on the agent."""
@agent.tool_plain
def current_datetime() -> dict[str, str]:
"""Get the current date and time.
Use this tool when you need to know the current date or time.
"""
return get_current_datetime()
@agent.tool
async def search_documents(ctx: RunContext[Deps], query: str, top_k: int = 5) -> str:
"""Search the knowledge base for relevant documents.
Use this tool to find information from uploaded documents before answering user queries.
Cite sources by referring to the document filename from the search results.
Args:
query: The search query string.
top_k: Number of top results to retrieve (default: 5).
Returns:
Formatted string with search results including content and scores.
"""
try:
return await search_knowledge_base(query=query, top_k=top_k)
except Exception as e:
raise ModelRetry("Knowledge base temporarily unavailable, please try again.") from e
@agent.tool_plain
def create_chart_tool(
chart_type: str,
title: str,
data: list[dict[str, Any]],
series: list[dict[str, Any]] | None = None,
x_key: str = "x",
style: dict[str, Any] | None = None,
) -> str:
"""Create a chart (line/bar/pie/area/scatter) to visualize data for the user.
Use whenever the user asks to plot, chart, graph, or visualize numbers,
trends, comparisons, or distributions. Do not repeat the returned JSON
back to the user — just briefly describe the chart you created.
Args:
chart_type: One of "line", "bar", "pie", "area", "scatter".
title: Short chart title.
data: Row dicts, e.g. [{"x": "Jan", "revenue": 120}]. For pie:
[{"x": "Chrome", "value": 64}, ...].
series: Optional [{"key", "label"?, "color"?}] selecting fields to plot.
x_key: Row field for the x-axis / pie label (default "x").
style: Optional {"palette", "grid", "legend", "x_label", "y_label", "stacked"}.
"""
return create_chart(
chart_type=chart_type, # type: ignore[arg-type]
title=title,
data=data,
series=series,
x_key=x_key,
style=style,
)
@staticmethod
def _build_model_history(
history: list[dict[str, str]] | None,
) -> list[ModelRequest | ModelResponse]:
model_history: list[ModelRequest | ModelResponse] = []
for msg in history or []:
if msg["role"] == "user":
model_history.append(ModelRequest(parts=[UserPromptPart(content=msg["content"])]))
elif msg["role"] == "assistant":
model_history.append(ModelResponse(parts=[TextPart(content=msg["content"])]))
elif msg["role"] == "system":
model_history.append(ModelRequest(parts=[SystemPromptPart(content=msg["content"])]))
return model_history
@property
def agent(self) -> Agent[Deps, str]:
"""Get or create the agent instance."""
if self._agent is None:
self._agent = self._create_agent()
return self._agent
async def run(
self,
user_input: str,
history: list[dict[str, str]] | None = None,
deps: Deps | None = None,
) -> tuple[str, list[Any], Deps]:
"""Run agent and return the output along with tool call events.
Args:
user_input: User's message.
history: Conversation history as list of {"role": "...", "content": "..."}.
deps: Optional dependencies. If not provided, a new Deps will be created.
Returns:
Tuple of (output_text, tool_events, deps).
"""
agent_deps = deps if deps is not None else Deps()
logger.info(f"Running agent with user input: {user_input[:100]}...")
result = await self.agent.run(
user_input,
deps=agent_deps,
message_history=self._build_model_history(history),
)
tool_events: list[Any] = []
for message in result.all_messages():
if hasattr(message, "parts"):
for part in message.parts:
if hasattr(part, "tool_name"):
tool_events.append(part)
logger.info(f"Agent run complete. Output length: {len(result.output)} chars")
return result.output, tool_events, agent_deps
async def iter(
self,
user_input: str,
history: list[dict[str, str]] | None = None,
deps: Deps | None = None,
) -> Any:
"""Stream agent execution with full event access.
Args:
user_input: User's message.
history: Conversation history.
deps: Optional dependencies.
Yields:
Agent events for streaming responses.
"""
agent_deps = deps if deps is not None else Deps()
async with self.agent.iter(
user_input,
deps=agent_deps,
message_history=self._build_model_history(history),
) as run:
async for event in run:
yield event
def get_agent(
model_name: str | None = None,
thinking_effort: str | None = None,
temperature: float | None = None,
) -> AssistantAgent:
"""Factory function to create an AssistantAgent.
Args:
model_name: Override the default AI model.
thinking_effort: Override thinking effort ("low", "medium", "high", or None to disable).
temperature: Sampling temperature (typically 0.0-2.0). ``None`` falls back to
``settings.AI_TEMPERATURE``.
Returns:
Configured AssistantAgent instance.
"""
return AssistantAgent(
model_name=model_name,
thinking_effort=thinking_effort,
temperature=temperature,
)
async def run_agent(
user_input: str,
history: list[dict[str, str]],
deps: Deps | None = None,
) -> tuple[str, list[Any], Deps]:
"""Run agent and return the output along with tool call events.
This is a convenience function for backwards compatibility.
Args:
user_input: User's message.
history: Conversation history.
deps: Optional dependencies.
Returns:
Tuple of (output_text, tool_events, deps).
"""
agent = get_agent()
return await agent.run(user_input, history, deps)
backend/app/agents/prompts.py
+67
View File
@@ -0,0 +1,67 @@
"""System prompts for AI agents.
Centralized location for all agent prompts to make them easy to find and modify.
The default prompt follows an outcome-first style: it defines who the assistant
is, how it should behave, and how to format answers — then trusts the model to
choose a good path. Avoid re-introducing long process checklists or absolute
"ALWAYS / NEVER / EXCLUSIVELY" rules for judgment calls; they make the assistant
mechanical and, in the RAG case, cause it to wrongly refuse general questions.
"""
DEFAULT_SYSTEM_PROMPT = """You are a knowledgeable, capable AI assistant. Help the user accomplish their task or answer their question as well as you can.
# Personality
Be approachable, steady, and direct. Assume the user is competent and acting in good faith. Prefer making progress over stopping for clarification when the request is clear enough to attempt — use reasonable assumptions and state them briefly. Ask a narrow clarifying question only when the missing information would materially change the answer.
Stay concise without being curt: give enough context for the user to understand and trust the answer, then stop. Use examples or simple analogies when they make a point land. When correcting the user or disagreeing, be candid but constructive; if you are wrong, acknowledge it plainly and fix it. Match the user's tone within professional bounds, and avoid emojis and profanity unless the user clearly invites that style.
# Answering
Answer from your own broad knowledge by default. You are a general-purpose assistant, not a document-lookup bot — questions about the world, concepts, code, math, science, history, culture, writing, and everyday advice should be answered directly and helpfully.
Say you don't know only when the answer genuinely depends on private, user-specific, or very recent information you cannot access. Never refuse or hedge on a general-knowledge question just because the topic isn't in a connected data source. If a request is ambiguous, answer the most likely intent and note the assumption rather than stalling.
# Output
Let formatting serve comprehension. Default to clear plain paragraphs for explanations and discussion. Reach for headers, bullets, or numbered lists only when they genuinely make the answer easier to scan — steps, comparisons, or rankings — or when the user asks for them. Honor explicit formatting and length preferences from the user. Lead with the conclusion, then the supporting detail, then any caveats."""
DEFAULT_SYSTEM_PROMPT += """
# Charts
You can render charts with the `create_chart` tool (line, bar, pie, area, scatter).
- Call it whenever the user asks to plot, chart, graph, compare, or visualize
numbers, trends, or distributions — or when a visual makes the answer clearer.
- Pick the chart_type that fits: trends over time -> line/area, category
comparison -> bar, parts of a whole -> pie, correlation -> scatter.
- Pass tidy rows in `data` (e.g. [{"x": "Jan", "revenue": 120, "cost": 80}]).
For pie charts use [{"x": "Chrome", "value": 64}, ...].
- You may override styling via `style` (palette, grid, legend, axis labels,
stacked) when the user requests a specific look.
- After the tool returns, do not repeat the JSON. Briefly describe the chart
and its key takeaway in plain language."""
def get_system_prompt_with_rag() -> str:
"""Get the default prompt plus knowledge-base (RAG) usage guidance.
Returns:
System prompt that treats `search_documents` as a tool to use when the
question is about the user's own documents/data — while still answering
general questions directly from the model's own knowledge.
"""
return f"""{DEFAULT_SYSTEM_PROMPT}
# Knowledge base
You have a `search_documents` tool that searches documents and data the user has added to this workspace.
When to search:
- The question is about the user's own documents, files, policies, projects, or other workspace/organization-specific information.
- The user explicitly refers to "the docs", an uploaded file, or internal information.
- A factual claim in your answer should be backed by their source material.
When NOT to search: general knowledge, common concepts, code, math, definitions, or anything you can already answer well. Do not search just to check whether something happens to be in the knowledge base, and never tell the user a topic "isn't in the knowledge base" when it is a question you can simply answer yourself.
Retrieval budget: start with one focused search using short, distinctive keywords. Search again only if the results miss the core question, a needed fact/figure/owner/date/source is missing, or the user asked for comprehensive coverage or a comparison. Don't search again merely to rephrase or pad the answer.
Citations: when you use retrieved documents, attach numbered references like [1], [2] to the specific claims they support, and list those sources at the end (filename, plus page if available). Cite only sources that appear in the search results — never fabricate citations, filenames, or page numbers.
Missing evidence is not automatically a "no". If the documents don't cover the question, say briefly what you couldn't find, then still help: answer from general knowledge where that's appropriate (and note that you're doing so), or ask for the specific document or detail you'd need."""
backend/app/agents/tools/__init__.py
+13
View File
@@ -0,0 +1,13 @@
"""Agent tools module.
This module contains utility functions that can be used as agent tools.
Tools are registered in the agent definition using @agent.tool decorator.
"""
from app.agents.tools.chart_tool import create_chart, parse_chart_spec
from app.agents.tools.datetime_tool import get_current_datetime
from app.agents.tools.rag_tool import search_knowledge_base, search_knowledge_base_sync
__all__ = ["get_current_datetime"]
__all__ += ["search_knowledge_base", "search_knowledge_base_sync"]
__all__ += ["create_chart", "parse_chart_spec"]
backend/app/agents/tools/chart_tool.py
+162
View File
@@ -0,0 +1,162 @@
"""Chart-generation tool for agents.
The agent calls ``create_chart`` to produce a structured chart specification.
The tool returns the spec as a JSON string — this single representation is:
- captured uniformly by every agent framework as the tool result,
- persisted verbatim in ``tool_calls.result`` (no DB migration needed),
- parsed by the web frontend and rendered interactively with Recharts,
- (Slack/Telegram render as a markdown table fallback.)
The agent may override the default styling (palette, grid, legend, axis
labels, stacking) via the ``style`` argument.
"""
import json
from typing import Any, Literal
from pydantic import BaseModel, Field, ValidationError, field_validator
ChartType = Literal["line", "bar", "pie", "area", "scatter"]
# Cap payload size so a runaway model can't emit a multi-megabyte tool result.
MAX_DATA_POINTS = 500
MAX_SERIES = 12
class ChartSeries(BaseModel):
"""One plotted series — maps a key in each data row to a labelled line/bar."""
key: str = Field(description="Field name in each data row to plot.")
label: str | None = Field(default=None, description="Legend label (defaults to key).")
color: str | None = Field(default=None, description="Hex color override, e.g. '#6366f1'.")
class ChartStyle(BaseModel):
"""Agent-controlled styling overrides on top of the frontend defaults."""
palette: list[str] | None = Field(
default=None, description="Custom color palette (hex), applied series-by-series."
)
grid: bool = Field(default=True, description="Show background grid.")
legend: bool = Field(default=True, description="Show the legend.")
x_label: str | None = Field(default=None, description="X-axis title.")
y_label: str | None = Field(default=None, description="Y-axis title.")
stacked: bool = Field(default=False, description="Stack bar/area series.")
class ChartSpec(BaseModel):
"""Canonical chart payload produced by the tool and consumed by every surface."""
kind: Literal["chart"] = "chart"
chart_type: ChartType
title: str = Field(max_length=200)
data: list[dict[str, Any]] = Field(description="Rows, e.g. [{'x': 'Q1', 'revenue': 120}].")
x_key: str = Field(default="x", description="Row field used for the x-axis / pie label.")
series: list[ChartSeries] = Field(default_factory=list)
style: ChartStyle = Field(default_factory=ChartStyle)
@field_validator("data")
@classmethod
def _validate_data(cls, v: list[dict[str, Any]]) -> list[dict[str, Any]]:
if not v:
raise ValueError("data must contain at least one row")
if len(v) > MAX_DATA_POINTS:
raise ValueError(f"data has too many rows (max {MAX_DATA_POINTS})")
return v
@field_validator("series")
@classmethod
def _validate_series(cls, v: list[ChartSeries]) -> list[ChartSeries]:
if len(v) > MAX_SERIES:
raise ValueError(f"too many series (max {MAX_SERIES})")
return v
def _infer_series(data: list[dict[str, Any]], x_key: str) -> list[ChartSeries]:
"""Derive series from the first row: every numeric field except the x-axis key."""
if not data:
return []
first = data[0]
inferred: list[ChartSeries] = []
for key, value in first.items():
if key == x_key:
continue
if isinstance(value, bool):
continue
if isinstance(value, int | float):
inferred.append(ChartSeries(key=key))
return inferred
def create_chart(
chart_type: ChartType,
title: str,
data: list[dict[str, Any]],
series: list[dict[str, Any]] | None = None,
x_key: str = "x",
style: dict[str, Any] | None = None,
) -> str:
"""Create a chart for the user.
Use this whenever the user asks to visualize numbers, trends, comparisons,
or distributions. The chart renders interactively in the web chat and as an
image on messaging channels.
Args:
chart_type: One of "line", "bar", "pie", "area", "scatter".
title: Short chart title shown above the plot.
data: List of row dicts, e.g. [{"x": "Jan", "revenue": 120, "cost": 80}].
For pie charts use [{"x": "Chrome", "value": 64}, ...].
series: Optional list of {"key", "label"?, "color"?} selecting which row
fields to plot. If omitted, every numeric field (except x_key) is plotted.
x_key: The row field used for the x-axis (or pie slice label). Default "x".
style: Optional overrides — {"palette": ["#6366f1", ...], "grid": true,
"legend": true, "x_label": "...", "y_label": "...", "stacked": false}.
Returns:
A JSON string with the chart specification. Do not repeat this JSON back
to the user — just briefly describe the chart you created.
"""
try:
resolved_series = (
[ChartSeries(**s) for s in series] if series else _infer_series(data, x_key)
)
spec = ChartSpec(
chart_type=chart_type,
title=title,
data=data,
x_key=x_key,
series=resolved_series,
style=ChartStyle(**style) if style else ChartStyle(),
)
except ValidationError as e:
return f"Could not build chart — invalid arguments: {e.errors()}"
except (TypeError, ValueError) as e:
return f"Could not build chart: {e}"
if not spec.series:
return (
"Could not build chart: no numeric series found. Provide a `series` "
"list or ensure data rows contain numeric fields besides x_key."
)
return spec.model_dump_json()
def parse_chart_spec(result: str) -> ChartSpec | None:
"""Parse a ``create_chart`` tool result back into a ChartSpec.
Returns None if the result is an error string rather than a valid spec
(used by the channel/web delivery layers).
"""
try:
payload = json.loads(result)
except (json.JSONDecodeError, TypeError):
return None
if not isinstance(payload, dict) or payload.get("kind") != "chart":
return None
try:
return ChartSpec.model_validate(payload)
except ValidationError:
return None
backend/app/agents/tools/datetime_tool.py
+13
View File
@@ -0,0 +1,13 @@
"""Date and time utilities for agents."""
from datetime import UTC, datetime
def get_current_datetime() -> dict[str, str]:
"""Get the current date and time (UTC)."""
now = datetime.now(UTC)
return {
"date": now.strftime("%Y-%m-%d"),
"time": now.strftime("%H:%M:%S"),
"datetime": now.strftime("%Y-%m-%d %H:%M:%S"),
}
backend/app/agents/tools/rag_tool.py
+145
View File
@@ -0,0 +1,145 @@
"""RAG tool for agent knowledge base search."""
import asyncio
import logging
from concurrent.futures import ThreadPoolExecutor
from typing import TYPE_CHECKING
logger = logging.getLogger(__name__)
if TYPE_CHECKING:
from app.services.rag.retrieval import BaseRetrievalService
_retrieval_service: "BaseRetrievalService | None" = None
def _get_retrieval_service() -> "BaseRetrievalService":
"""Get or create retrieval service singleton."""
global _retrieval_service
if _retrieval_service is not None:
return _retrieval_service
from app.core.config import settings
from app.services.rag.embeddings import EmbeddingService
from app.services.rag.retrieval import RetrievalService
from app.services.rag.vectorstore import MilvusVectorStore
rag_settings = settings.rag
embedding_service = EmbeddingService(rag_settings)
vector_store = MilvusVectorStore(rag_settings, embedding_service)
_retrieval_service = RetrievalService(vector_store, rag_settings)
return _retrieval_service
def get_retrieval_service() -> "BaseRetrievalService":
"""Get the RetrievalService singleton."""
return _get_retrieval_service()
def _format_results(results: list) -> str:
if not results:
return "No relevant documents found in the knowledge base."
formatted = []
for i, result in enumerate(results, start=1):
source = result.metadata.get("filename", "unknown")
page = result.metadata.get("page_num", "")
chunk = result.metadata.get("chunk_num", "")
col = result.metadata.get("collection", "")
page_info = f", page {page}" if page else ""
chunk_info = f", chunk {chunk}" if chunk else ""
col_info = f" [{col}]" if col else ""
formatted.append(
f"[{i}] Source: {source}{page_info}{chunk_info}{col_info} (score: {result.score:.3f})\n"
f"{result.content}"
)
return "Search results (cite sources using [1], [2], etc. in your response):\n\n" + "\n\n".join(
formatted
)
async def search_knowledge_base(
query: str,
collection: str | None = None,
collections: list[str] | None = None,
top_k: int = 5,
) -> str:
"""Search the knowledge base and return formatted results.
Args:
query: The search query string.
collection: Name of a single collection. If None, uses RAG_DEFAULT_COLLECTION env var.
collections: List of collection names for cross-collection search (overrides collection).
top_k: Number of top results to retrieve (default: 5).
"""
import os
from typing import Any
service: Any = get_retrieval_service()
default_collection = os.environ.get("RAG_DEFAULT_COLLECTION", "all")
target_collection = collection or default_collection
if collections and len(collections) > 1:
results = await service.retrieve_multi(
query=query,
collection_names=collections,
limit=top_k,
)
elif target_collection == "all":
try:
all_collections = await service.store.list_collections()
if not all_collections:
return "No collections found in the knowledge base."
if len(all_collections) == 1:
results = await service.retrieve(
query=query, collection_name=all_collections[0], limit=top_k
)
else:
results = await service.retrieve_multi(
query=query, collection_names=all_collections, limit=top_k
)
except Exception as e:
logger.error(f"Failed to list collections: {e}")
return f"Error accessing knowledge base: {e}"
else:
results = await service.retrieve(
query=query,
collection_name=target_collection,
limit=top_k,
)
return _format_results(results)
def _run_async_search(query: str, collection: str | None, top_k: int) -> str:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
return loop.run_until_complete(search_knowledge_base(query, collection, top_k=top_k))
finally:
loop.close()
def search_knowledge_base_sync(
query: str,
collection: str | None = None,
top_k: int = 5,
) -> str:
"""Synchronous wrapper for search_knowledge_base. Use in CrewAI agents."""
logger.debug(
"search_knowledge_base_sync called: query=%s, collection=%s, top_k=%s",
query,
collection,
top_k,
)
try:
with ThreadPoolExecutor(max_workers=1) as executor:
future = executor.submit(_run_async_search, query, collection, top_k)
result = future.result()
logger.debug("search_knowledge_base_sync completed successfully")
return result
except Exception as e:
logger.error("search_knowledge_base_sync failed: %s", str(e), exc_info=True)
raise
__all__ = ["search_knowledge_base", "search_knowledge_base_sync"]
backend/app/api/deps.py
+422
View File
@@ -0,0 +1,422 @@
"""API dependencies.
Dependency injection factories for services, repositories, and authentication.
"""
# ruff: noqa: I001 - Imports structured for Jinja2 template conditionals
from typing import Annotated
from fastapi import Depends
from fastapi.security import OAuth2PasswordBearer
from app.core.config import settings
from app.db.session import get_db_session
from sqlalchemy.ext.asyncio import AsyncSession
DBSession = Annotated[AsyncSession, Depends(get_db_session)]
from fastapi import Request
from app.clients.redis import RedisClient
async def get_redis(request: Request) -> RedisClient:
"""Get Redis client from lifespan state."""
return request.state.redis # type: ignore[no-any-return]
Redis = Annotated[RedisClient, Depends(get_redis)]
# === Service Dependencies ===
from app.services.user import UserService
from app.services.conversation import ConversationService
def get_user_service(db: DBSession) -> UserService:
"""Create UserService instance with database session."""
return UserService(db)
UserSvc = Annotated[UserService, Depends(get_user_service)]
def get_conversation_service(db: DBSession) -> ConversationService:
"""Create ConversationService instance with database session."""
return ConversationService(db)
ConversationSvc = Annotated[ConversationService, Depends(get_conversation_service)]
from app.services.conversation_share import ConversationShareService
def get_conversation_share_service(db: DBSession) -> ConversationShareService:
"""Create ConversationShareService instance with database session."""
return ConversationShareService(db)
ConversationShareSvc = Annotated[ConversationShareService, Depends(get_conversation_share_service)]
# Message rating service
from app.services.message_rating import MessageRatingService
def get_rating_service(db: DBSession) -> MessageRatingService:
"""Create MessageRatingService instance with database session."""
return MessageRatingService(db)
MessageRatingSvc = Annotated[MessageRatingService, Depends(get_rating_service)]
from app.services.rag_document import RAGDocumentService
from app.services.rag_sync import RAGSyncService
from app.services.sync_source import SyncSourceService
def get_rag_document_service(db: DBSession) -> RAGDocumentService:
"""Create RAGDocumentService instance with database session."""
return RAGDocumentService(db)
def get_rag_sync_service(db: DBSession) -> RAGSyncService:
"""Create RAGSyncService instance with database session."""
return RAGSyncService(db)
def get_sync_source_service(db: DBSession) -> SyncSourceService:
"""Create SyncSourceService instance with database session."""
return SyncSourceService(db)
RAGDocumentSvc = Annotated[RAGDocumentService, Depends(get_rag_document_service)]
RAGSyncSvc = Annotated[RAGSyncService, Depends(get_rag_sync_service)]
SyncSourceSvc = Annotated[SyncSourceService, Depends(get_sync_source_service)]
from app.services.rag_status import RAGStatusService
def get_rag_status_service() -> RAGStatusService:
"""Create RAGStatusService instance (no DB)."""
return RAGStatusService()
RAGStatusSvc = Annotated[RAGStatusService, Depends(get_rag_status_service)]
from app.services.file_upload import FileUploadService
def get_file_upload_service(db: DBSession) -> FileUploadService:
"""Create FileUploadService instance with database session."""
return FileUploadService(db)
FileUploadSvc = Annotated[FileUploadService, Depends(get_file_upload_service)]
# === Authentication Dependencies ===
from app.core.exceptions import AuthenticationError, AuthorizationError
from app.db.models.user import User, UserRole
oauth2_scheme = OAuth2PasswordBearer(tokenUrl=f"{settings.API_V1_STR}/auth/login")
async def get_current_user(
token: Annotated[str, Depends(oauth2_scheme)],
user_service: UserSvc,
) -> User:
"""Get current authenticated user from JWT token.
Returns the full User object including role information.
Raises:
AuthenticationError: If token is invalid or user not found.
"""
from uuid import UUID
from app.core.security import verify_token
payload = verify_token(token)
if payload is None:
raise AuthenticationError(message="Invalid or expired token")
# Ensure this is an access token, not a refresh token
if payload.get("type") != "access":
raise AuthenticationError(message="Invalid token type")
user_id = payload.get("sub")
if user_id is None:
raise AuthenticationError(message="Invalid token payload")
user = await user_service.get_by_id(UUID(user_id))
if not user.is_active:
raise AuthenticationError(message="User account is disabled")
return user
class RoleChecker:
"""Dependency class for role-based access control.
Usage:
# Require admin role
@router.get("/admin-only")
async def admin_endpoint(
user: Annotated[User, Depends(RoleChecker(UserRole.ADMIN))]
):
...
# Require any authenticated user
@router.get("/users")
async def users_endpoint(
user: Annotated[User, Depends(get_current_user)]
):
...
"""
def __init__(self, required_role: UserRole) -> None:
self.required_role = required_role
async def __call__(
self,
user: Annotated[User, Depends(get_current_user)],
) -> User:
"""Check if user has the required role.
Raises:
AuthorizationError: If user doesn't have the required role.
"""
if not user.has_role(self.required_role):
raise AuthorizationError(
message=f"Role '{self.required_role.value}' required for this action"
)
return user
async def get_current_active_superuser(
current_user: Annotated[User, Depends(get_current_user)],
) -> User:
"""Get current user and verify they are a superuser.
Raises:
AuthorizationError: If user is not a superuser.
"""
if not current_user.has_role(UserRole.ADMIN):
raise AuthorizationError(message="Admin privileges required")
return current_user
# Type aliases for dependency injection
CurrentUser = Annotated[User, Depends(get_current_user)]
CurrentSuperuser = Annotated[User, Depends(get_current_active_superuser)]
CurrentAdmin = Annotated[User, Depends(RoleChecker(UserRole.ADMIN))]
# is_app_admin is a global flag on the User model — independent of team
# membership. Routes guarded by this dep (e.g. /admin/users) stay reachable
# even when teams are disabled, so the dep itself must not be gated.
async def _require_app_admin(user: CurrentUser) -> "User": # type: ignore[name-defined]
"""Raises 403 unless the user has the is_app_admin flag set."""
if not getattr(user, "is_app_admin", False):
raise AuthorizationError(message="App admin privileges required")
return user
CurrentAppAdmin = Annotated["User", Depends(_require_app_admin)] # type: ignore[valid-type]
# WebSocket authentication dependency
from fastapi import WebSocket, Cookie
_WS_TOKEN_PROTOCOL_PREFIX = "access_token."
def _extract_ws_auth(websocket: WebSocket) -> tuple[str | None, str | None]:
"""Parse Sec-WebSocket-Protocol header for an auth token + app subprotocol.
Clients pass the token as a subprotocol of the form
``access_token.<JWT>`` alongside an optional application subprotocol
(e.g. ``chat``). Returns (token, app_subprotocol) — either may be None.
"""
raw = websocket.headers.get("sec-websocket-protocol") or ""
token: str | None = None
app_subprotocol: str | None = None
for proto in (p.strip() for p in raw.split(",") if p.strip()):
if proto.startswith(_WS_TOKEN_PROTOCOL_PREFIX):
token = proto[len(_WS_TOKEN_PROTOCOL_PREFIX) :]
elif app_subprotocol is None:
app_subprotocol = proto
return token, app_subprotocol
async def get_current_user_ws(
websocket: WebSocket,
access_token: str | None = Cookie(None),
) -> User:
"""Authenticate a WebSocket connection.
Token sources, checked in order:
1. ``Sec-WebSocket-Protocol`` header, in the form ``access_token.<JWT>``.
The chosen application subprotocol (e.g. ``chat``) is echoed back on
``accept()`` via ``websocket.state.accept_subprotocol``.
2. Same-origin ``access_token`` cookie (fallback for same-origin clients).
Tokens in query strings are NOT accepted — they leak into logs and
Referer headers.
Raises:
AuthenticationError: If token is invalid or user not found.
"""
from uuid import UUID
from app.core.security import verify_token
subprotocol_token, app_subprotocol = _extract_ws_auth(websocket)
websocket.state.accept_subprotocol = app_subprotocol
auth_token = subprotocol_token or access_token
if not auth_token:
await websocket.close(code=4001, reason="Missing authentication token")
raise AuthenticationError(message="Missing authentication token")
payload = verify_token(auth_token)
if payload is None:
await websocket.close(code=4001, reason="Invalid or expired token")
raise AuthenticationError(message="Invalid or expired token")
if payload.get("type") != "access":
await websocket.close(code=4001, reason="Invalid token type")
raise AuthenticationError(message="Invalid token type")
user_id = payload.get("sub")
if user_id is None:
await websocket.close(code=4001, reason="Invalid token payload")
raise AuthenticationError(message="Invalid token payload")
from app.db.session import get_db_context
async with get_db_context() as db:
user_service = UserService(db)
user = await user_service.get_by_id(UUID(user_id))
if not user.is_active:
await websocket.close(code=4001, reason="User account is disabled")
raise AuthenticationError(message="User account is disabled")
# Eagerly load all columns, then detach from session to avoid
# "instance not bound to a Session" errors after the context manager exits
await db.refresh(user)
db.expunge(user)
return user
import secrets
from fastapi.security import APIKeyHeader
api_key_header = APIKeyHeader(name=settings.API_KEY_HEADER, auto_error=False)
async def verify_api_key(
api_key: Annotated[str | None, Depends(api_key_header)],
) -> str:
"""Verify API key from header.
Uses constant-time comparison to prevent timing attacks.
Raises:
AuthenticationError: If API key is missing.
AuthorizationError: If API key is invalid.
"""
if api_key is None:
raise AuthenticationError(message="API Key header missing")
if not secrets.compare_digest(api_key, settings.API_KEY):
raise AuthorizationError(message="Invalid API Key")
return api_key
ValidAPIKey = Annotated[str, Depends(verify_api_key)]
# === RAG Service Dependencies ===
from app.services.rag.embeddings import EmbeddingService
from app.services.rag.ingestion import IngestionService
from app.services.rag.documents import DocumentProcessor
from fastapi import Request
from app.core.config import settings
from app.services.rag.retrieval import RetrievalService
from app.services.rag.vectorstore import MilvusVectorStore
def get_embedding_service(request: Request) -> EmbeddingService:
"""Get embedding service from lifespan state or create new if not available."""
if request and hasattr(request.state, "embedding_service"):
return request.state.embedding_service # type: ignore[no-any-return]
return EmbeddingService(settings=settings.rag)
# Type Alias for the Embedder
EmbeddingSvc = Annotated[EmbeddingService, Depends(get_embedding_service)]
from app.services.rag.vectorstore import BaseVectorStore
def get_vectorstore(request: Request, embedder: EmbeddingSvc) -> BaseVectorStore:
"""Get vector store client from lifespan state or create new."""
if request and hasattr(request.state, "vector_store"):
return request.state.vector_store # type: ignore[no-any-return]
return MilvusVectorStore(settings=settings.rag, embedding_service=embedder)
VectorStoreSvc = Annotated[BaseVectorStore, Depends(get_vectorstore)]
def get_retrieval_service(vector_store: VectorStoreSvc) -> RetrievalService:
"""Create RetrievalService instance."""
from app.services.rag.reranker import RerankService
rerank_service = RerankService(settings=settings.rag)
return RetrievalService(
vector_store=vector_store,
settings=settings.rag,
rerank_service=rerank_service,
)
RetrievalSvc = Annotated[RetrievalService, Depends(get_retrieval_service)]
def get_document_processor() -> DocumentProcessor:
"""Create DocumentProcessor instance."""
return DocumentProcessor(settings=settings.rag)
DocumentProcessorSvc = Annotated[DocumentProcessor, Depends(get_document_processor)]
def get_ingestion_service(
processor: DocumentProcessorSvc,
vector_store: VectorStoreSvc,
) -> IngestionService:
"""Create IngestionService instance."""
return IngestionService(processor=processor, vector_store=vector_store)
IngestionSvc = Annotated[IngestionService, Depends(get_ingestion_service)]
from app.services.user_slash_command import UserSlashCommandService
def get_user_slash_command_service(db: DBSession) -> UserSlashCommandService:
return UserSlashCommandService(db)
UserSlashCommandSvc = Annotated[UserSlashCommandService, Depends(get_user_slash_command_service)]
from app.services.admin import AdminService
def get_admin_service(db: DBSession) -> AdminService:
"""Create AdminService instance — used by admin REST routes (always
available, independent of the optional SQLAdmin UI)."""
return AdminService(db)
AdminSvc = Annotated[AdminService, Depends(get_admin_service)]
backend/app/api/exception_handlers.py
+108
View File
@@ -0,0 +1,108 @@
"""Exception handlers for FastAPI application.
These handlers convert domain exceptions to proper HTTP responses.
WebSocket connections that raise an ``AppException`` before ``accept()`` are
handled too — Starlette closes the socket with 403 and we just log the
incident; we cannot return an HTTP body for a non-HTTP scope.
"""
import logging
from typing import Any
from fastapi import FastAPI
from fastapi.responses import JSONResponse
from starlette.requests import HTTPConnection
from app.core.exceptions import AppException
logger = logging.getLogger(__name__)
def _connection_meta(conn: HTTPConnection) -> dict[str, Any]:
"""Common log fields shared by HTTP requests and WebSocket connections.
``method`` exists only on HTTP ``Request`` — for WebSockets we surface the
scope type so log filters can still distinguish the two.
"""
return {
"path": conn.url.path,
"method": getattr(conn, "method", None) or conn.scope.get("type", "unknown"),
}
def _is_websocket(conn: HTTPConnection) -> bool:
return conn.scope.get("type") == "websocket"
async def app_exception_handler(request: HTTPConnection, exc: AppException) -> JSONResponse | None:
"""Handle application exceptions for both HTTP and WebSocket scopes.
Logs 5xx errors as errors and 4xx as warnings. Returns a JSON response
for HTTP scopes; returns ``None`` for WebSocket scopes (Starlette will
close the socket on its own).
"""
log_extra = {
"error_code": exc.code,
"status_code": exc.status_code,
"details": exc.details,
**_connection_meta(request),
}
if exc.status_code >= 500:
logger.error(f"{exc.code}: {exc.message}", extra=log_extra)
else:
logger.warning(f"{exc.code}: {exc.message}", extra=log_extra)
if _is_websocket(request):
return None
headers: dict[str, str] = {}
if exc.status_code == 401:
headers["WWW-Authenticate"] = "Bearer"
return JSONResponse(
status_code=exc.status_code,
content={
"error": {
"code": exc.code,
"message": exc.message,
"details": exc.details or None,
}
},
headers=headers,
)
async def unhandled_exception_handler(
request: HTTPConnection, exc: Exception
) -> JSONResponse | None:
"""Handle unexpected exceptions.
Logs the full exception but returns a generic error to the client
to avoid leaking sensitive information.
"""
logger.exception("Unhandled exception", extra=_connection_meta(request))
if _is_websocket(request):
return None
return JSONResponse(
status_code=500,
content={
"error": {
"code": "INTERNAL_ERROR",
"message": "An unexpected error occurred",
"details": None,
}
},
)
def register_exception_handlers(app: FastAPI) -> None:
"""Register all exception handlers on the FastAPI app.
Call this after creating the FastAPI application instance.
"""
app.add_exception_handler(AppException, app_exception_handler)
# Uncomment to catch all unhandled exceptions:
# app.add_exception_handler(Exception, unhandled_exception_handler)
backend/app/api/router.py
+10
View File
@@ -0,0 +1,10 @@
"""API router aggregation."""
from fastapi import APIRouter
from app.api.routes.v1 import v1_router
api_router = APIRouter()
# API v1 routes (prefix is set in main.py via settings.API_V1_STR)
api_router.include_router(v1_router)
backend/app/api/routes/__init__.py
+9
View File
@@ -0,0 +1,9 @@
"""API routes.
This package contains versioned API routes.
Add new versions by creating new folders (e.g., v2/) and updating router.py.
"""
from app.api.routes.v1 import v1_router
__all__ = ["v1_router"]
backend/app/api/routes/v1/__init__.py
+57
View File
@@ -0,0 +1,57 @@
"""API v1 router aggregation."""
# ruff: noqa: I001 - Imports structured for Jinja2 template conditionals
from fastapi import APIRouter
from app.api.routes.v1 import health
from app.api.routes.v1 import admin_users, auth, users
from app.api.routes.v1 import admin_ratings
from app.api.routes.v1 import oauth
from app.api.routes.v1 import conversations
from app.api.routes.v1 import admin_conversations
from app.api.routes.v1 import agent
from app.api.routes.v1 import rag
from app.api.routes.v1 import files
from app.api.routes.v1 import me_slash_commands
from app.api.routes.v1 import admin_stats
v1_router = APIRouter()
# Health check routes (no auth required)
v1_router.include_router(health.router, tags=["health"])
# Authentication routes
v1_router.include_router(auth.router, prefix="/auth", tags=["auth"])
# User routes
v1_router.include_router(users.router, prefix="/users", tags=["users"])
# Admin: message-rating analytics
v1_router.include_router(admin_ratings.router, prefix="/admin/ratings", tags=["admin:ratings"])
# OAuth2 routes
v1_router.include_router(oauth.router, prefix="/oauth", tags=["oauth"])
# Conversation routes (AI chat persistence)
v1_router.include_router(conversations.router, prefix="/conversations", tags=["conversations"])
# AI Agent routes
v1_router.include_router(agent.router, tags=["agent"])
# RAG routes
v1_router.include_router(rag.router, prefix="/rag", tags=["rag"])
# File upload/download routes
v1_router.include_router(files.router, tags=["files"])
# Admin: conversation browser
v1_router.include_router(
admin_conversations.router, prefix="/admin/conversations", tags=["admin-conversations"]
)
# Admin: user management + impersonation
v1_router.include_router(admin_users.router, prefix="/admin/users", tags=["admin:users"])
v1_router.include_router(
me_slash_commands.router, prefix="/me/slash-commands", tags=["me:slash-commands"]
)
v1_router.include_router(admin_stats.router, prefix="/admin", tags=["admin:stats"])
backend/app/api/routes/v1/admin_conversations.py
+78
View File
@@ -0,0 +1,78 @@
"""Admin conversation and user browsing routes.
All endpoints require admin role.
Endpoints:
GET /admin/conversations — List all conversations (paginated, filterable)
GET /admin/conversations/{id} — Get any conversation with messages (read-only)
GET /admin/users — List all users with conversation counts
GET /admin/users/{user_id}/conversations — List conversations for a specific user
"""
from typing import Any, Literal
from uuid import UUID
from fastapi import APIRouter, Query
from app.api.deps import ConversationSvc, CurrentAdmin, UserSvc
from app.schemas.conversation import ConversationReadWithMessages
from app.schemas.conversation_share import AdminConversationList, AdminUserList
router = APIRouter()
@router.get("", response_model=AdminConversationList)
async def admin_list_conversations(
service: ConversationSvc,
_: CurrentAdmin,
skip: int = Query(0, ge=0, description="Items to skip"),
limit: int = Query(50, ge=1, le=100, description="Max items to return"),
search: str | None = Query(default=None, description="Search by title"),
user_id: UUID | None = Query(default=None, description="Filter by user ID"),
status: Literal["active", "archived", "all"] = Query(
"active", description="Filter by archival status"
),
sort_by: Literal["title", "owner", "messages", "created_at", "updated_at"] = Query(
"updated_at", description="Sort column"
),
sort_dir: Literal["asc", "desc"] = Query("desc", description="Sort direction"),
) -> Any:
"""List all conversations across all users (admin only)."""
return await service.admin_list_with_users(
skip=skip,
limit=limit,
search=search,
user_id=user_id,
include_archived=status == "all",
archived_only=status == "archived",
sort_by=sort_by,
sort_dir=sort_dir,
)
@router.get("/users", response_model=AdminUserList)
async def admin_list_users(
user_service: UserSvc,
_: CurrentAdmin,
skip: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=500),
search: str | None = Query(default=None, description="Search by email or name"),
sort_by: Literal["email", "full_name", "conversations", "created_at"] = Query(
"created_at", description="Sort column"
),
sort_dir: Literal["asc", "desc"] = Query("desc", description="Sort direction"),
) -> Any:
"""List all users with conversation counts (admin only)."""
return await user_service.admin_list_with_counts(
skip=skip, limit=limit, search=search, sort_by=sort_by, sort_dir=sort_dir
)
@router.get("/{conversation_id}", response_model=ConversationReadWithMessages)
async def admin_get_conversation(
conversation_id: UUID,
service: ConversationSvc,
_: CurrentAdmin,
) -> Any:
"""Get any conversation with messages (admin read-only access)."""
return await service.get_conversation_with_messages(conversation_id)
backend/app/api/routes/v1/admin_ratings.py
+95
View File
@@ -0,0 +1,95 @@
"""Admin endpoints for message ratings.
Provides endpoints for administrators to view and analyze user ratings
on AI assistant messages.
The endpoints are:
- GET /admin/ratings - List all ratings with filtering
- GET /admin/ratings/summary - Get aggregated rating statistics
- GET /admin/ratings/export - Export ratings as JSON or CSV
"""
from typing import Any
from fastapi import APIRouter, Query
from fastapi.responses import JSONResponse, StreamingResponse
from app.api.deps import CurrentAdmin, MessageRatingSvc
from app.schemas.message_rating import MessageRatingList, RatingSummary
router = APIRouter()
@router.get("", response_model=MessageRatingList)
async def list_ratings_admin(
rating_service: MessageRatingSvc,
_: CurrentAdmin,
skip: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=100),
rating_filter: int | None = Query(None, ge=-1, le=1, description="Filter by rating value"),
with_comments_only: bool = Query(False, description="Only show ratings with comments"),
) -> Any:
"""List all ratings with filtering (admin only).
Returns paginated list of ratings with optional filters:
- rating_filter: Filter by rating value (1 for likes, -1 for dislikes)
- with_comments_only: Only return ratings that have comments
Results are ordered by creation date (newest first).
"""
items, total = await rating_service.list_ratings(
skip=skip,
limit=limit,
rating_filter=rating_filter,
with_comments_only=with_comments_only,
)
return MessageRatingList(items=items, total=total)
@router.get("/summary", response_model=RatingSummary)
async def get_rating_summary(
rating_service: MessageRatingSvc,
_: CurrentAdmin,
days: int = Query(30, ge=1, le=365, description="Number of days to include"),
) -> Any:
"""Get aggregated rating statistics (admin only).
Returns summary statistics including:
- Total ratings count
- Like/dislike counts
- Average rating (-1.0 to 1.0)
- Count of ratings with comments
- Daily breakdown of ratings
The `days` parameter controls the time window (default: 30 days).
"""
return await rating_service.get_summary(days=days)
@router.get("/export")
async def export_ratings(
rating_service: MessageRatingSvc,
_: CurrentAdmin,
export_format: str = Query("json", description="Export format: 'json' or 'csv'"),
rating_filter: int | None = Query(None, ge=-1, le=1, description="Filter by rating value"),
with_comments_only: bool = Query(False, description="Only show ratings with comments"),
) -> Any:
"""Export all ratings as JSON or CSV (admin only).
CSV is streamed row-by-row; JSON collects into a single document.
"""
result = await rating_service.export_ratings(
export_format=export_format,
rating_filter=rating_filter,
with_comments_only=with_comments_only,
)
if result.media_type == "text/csv":
return StreamingResponse(
result.payload,
media_type="text/csv",
headers={"Content-Disposition": result.content_disposition},
)
return JSONResponse(
content=result.payload,
headers={"Content-Disposition": result.content_disposition},
)
backend/app/api/routes/v1/admin_stats.py
+37
View File
@@ -0,0 +1,37 @@
"""Admin observability — workspace stats + Stripe event log."""
from typing import Any
from fastapi import APIRouter, Query
from app.api.deps import AdminSvc, CurrentAdmin
from app.schemas.admin import AdminStats, StripeEventList, StripeEventRead
router = APIRouter()
@router.get("/stats", response_model=AdminStats)
async def get_admin_stats(
service: AdminSvc,
_user: CurrentAdmin,
) -> Any:
"""Aggregate workspace metrics. Billing fields are 0 when billing is
disabled in the deployment."""
return await service.workspace_stats()
@router.get("/stripe-events", response_model=StripeEventList)
async def list_stripe_events(
service: AdminSvc,
_user: CurrentAdmin,
skip: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=200),
) -> Any:
"""List recent Stripe webhook events from the idempotency log.
Best-effort: returns empty list when the StripeEvent table doesn't exist
(billing disabled in this deployment).
"""
rows, total = await service.list_stripe_events(skip=skip, limit=limit)
items = [StripeEventRead.model_validate(row) for row in rows]
return StripeEventList(items=items, total=total)
backend/app/api/routes/v1/admin_users.py
+93
View File
@@ -0,0 +1,93 @@
"""Admin user management routes.
All endpoints require the admin role (CurrentAdmin).
Endpoints:
GET /admin/users — List all users (paginated + search)
GET /admin/users/{user_id} — Get a single user
PATCH /admin/users/{user_id} — Update user (role, is_active, is_app_admin)
DELETE /admin/users/{user_id} — Hard-delete a user
POST /admin/users/{user_id}/impersonate — Issue short-lived token to act as user
"""
from datetime import timedelta
from typing import Any
from uuid import UUID
from fastapi import APIRouter, Query, Request, status
from app.api.deps import CurrentAdmin, DBSession, UserSvc
from app.core.security import create_access_token
from app.schemas.conversation_share import AdminUserList
from app.schemas.user import UserRead, UserUpdate
router = APIRouter()
@router.get("", response_model=AdminUserList)
async def list_users(
_: CurrentAdmin,
service: UserSvc,
skip: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=500),
search: str | None = Query(None),
) -> Any:
result = await service.admin_list_with_counts(skip=skip, limit=limit, search=search)
return result
@router.get("/{user_id}", response_model=UserRead)
async def get_user(
user_id: UUID,
_: CurrentAdmin,
service: UserSvc,
) -> Any:
return await service.get_by_id(user_id)
@router.patch("/{user_id}", response_model=UserRead)
async def update_user(
request: Request,
user_id: UUID,
user_in: UserUpdate,
admin: CurrentAdmin,
db: DBSession,
service: UserSvc,
) -> Any:
user = await service.update(user_id, user_in)
return user
@router.delete("/{user_id}", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
async def delete_user(
request: Request,
user_id: UUID,
admin: CurrentAdmin,
db: DBSession,
service: UserSvc,
) -> None:
await service.get_by_id(user_id) # raises 404 if not found
await service.delete(user_id)
@router.post("/{user_id}/impersonate", response_model=dict)
async def impersonate_user(
request: Request,
user_id: UUID,
admin: CurrentAdmin,
db: DBSession,
service: UserSvc,
) -> Any:
"""Issue a short-lived (1h) access token to act as the target user."""
target = await service.get_by_id(user_id)
token = create_access_token(
subject=str(target.id),
expires_delta=timedelta(hours=1),
)
return {
"access_token": token,
"token_type": "bearer",
"impersonated_user_id": str(target.id),
"impersonated_by": str(admin.id),
"expires_in": 3600,
}
backend/app/api/routes/v1/agent.py
+78
View File
@@ -0,0 +1,78 @@
"""AI Agent WebSocket route.
The route is just lifecycle plumbing — auth, accept, dispatch loop, disconnect.
Per-turn orchestration lives in :class:`app.services.agent_session.AgentSession`.
"""
import logging
from typing import Any
from fastapi import APIRouter, Depends, WebSocket, WebSocketDisconnect
from app.api.deps import get_current_user_ws
from app.core.config import settings
from app.db.models.user import User
from app.services.agent import AgentConnectionManager
from app.services.agent_session import AgentSession
logger = logging.getLogger(__name__)
router = APIRouter()
manager = AgentConnectionManager()
@router.get("/agent/models")
async def list_models() -> dict[str, Any]:
"""Return available LLM models and the current default."""
return {
"default": settings.AI_MODEL,
"models": settings.AI_AVAILABLE_MODELS,
}
@router.websocket("/ws/agent")
async def agent_websocket(
websocket: WebSocket,
user: User = Depends(get_current_user_ws),
) -> None:
"""WebSocket endpoint for the AI agent.
Streams agent events to the client. Each incoming JSON message is forwarded to
:class:`AgentSession.process_message`.
Expected input format::
{
"message": "user message here",
"file_ids": ["..."],
"conversation_id": "optional-uuid",
"model": "optional-model-override",
"thinking_effort": "optional"
}
Authentication: handled by ``get_current_user_ws`` (JWT).
"""
if user is None:
return
await manager.connect(websocket)
session = AgentSession(
websocket,
user,
)
try:
while True:
try:
data = await websocket.receive_json()
except WebSocketDisconnect:
break
try:
await session.process_message(data)
except WebSocketDisconnect:
logger.info("Client disconnected during agent processing")
break
finally:
manager.disconnect(websocket)
backend/app/api/routes/v1/auth.py
+92
View File
@@ -0,0 +1,92 @@
"""Authentication routes."""
import logging
from typing import Annotated, Any
from fastapi import APIRouter, Depends, Request, status
from fastapi.security import OAuth2PasswordRequestForm
from app.api.deps import CurrentUser, UserSvc
from app.core.exceptions import AuthenticationError
from app.core.security import (
create_access_token,
create_refresh_token,
verify_token,
)
from app.schemas.token import RefreshTokenRequest, Token
from app.schemas.user import UserCreate, UserRead
logger = logging.getLogger(__name__)
router = APIRouter()
@router.post("/login", response_model=Token)
async def login(
request: Request,
form_data: Annotated[OAuth2PasswordRequestForm, Depends()],
user_service: UserSvc,
) -> Any:
"""OAuth2 compatible token login.
Returns access token and refresh token.
Raises domain exceptions handled by exception handlers.
"""
user = await user_service.authenticate(form_data.username, form_data.password)
access_token = create_access_token(subject=str(user.id))
refresh_token = create_refresh_token(subject=str(user.id))
return Token(access_token=access_token, refresh_token=refresh_token)
@router.post("/register", response_model=UserRead, status_code=status.HTTP_201_CREATED)
async def register(
user_in: UserCreate,
user_service: UserSvc,
) -> Any:
"""Register a new user.
Raises AlreadyExistsError if email is already registered.
"""
user = await user_service.register(user_in)
return user
@router.post("/refresh", response_model=Token)
async def refresh_token(
request: Request,
body: RefreshTokenRequest,
user_service: UserSvc,
) -> Any:
"""Get new access token using refresh token.
Raises AuthenticationError if refresh token is invalid or expired.
"""
# No DB-backed sessions — validate the refresh JWT directly.
payload = verify_token(body.refresh_token)
if not payload or payload.get("type") != "refresh":
raise AuthenticationError(message="Invalid or expired refresh token")
user_id = payload.get("sub")
if not user_id:
raise AuthenticationError(message="Invalid refresh token")
user = await user_service.get_by_id(user_id)
if not user.is_active:
raise AuthenticationError(message="User account is disabled")
access_token = create_access_token(subject=str(user.id))
new_refresh_token = create_refresh_token(subject=str(user.id))
return Token(access_token=access_token, refresh_token=new_refresh_token)
@router.post("/logout", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
async def logout(
body: RefreshTokenRequest,
) -> None:
"""No-op without session tracking. Clients drop their JWTs locally."""
return None
@router.get("/me", response_model=UserRead)
async def get_current_user_info(current_user: CurrentUser) -> Any:
"""Get current authenticated user information."""
return current_user
backend/app/api/routes/v1/conversations.py
+363
View File
@@ -0,0 +1,363 @@
"""Conversation API routes for AI chat persistence.
Provides CRUD operations for conversations and messages.
The endpoints are:
- GET /conversations - List user's conversations
- POST /conversations - Create a new conversation
- GET /conversations/{id} - Get a conversation with messages
- PATCH /conversations/{id} - Update conversation title/archived status
- DELETE /conversations/{id} - Delete a conversation
- POST /conversations/{id}/messages - Add a message to conversation
- GET /conversations/{id}/messages - List messages in conversation
"""
from typing import Any
from uuid import UUID
from fastapi import APIRouter, Query, Response, status
from fastapi.responses import JSONResponse
from app.api.deps import (
ConversationShareSvc,
ConversationSvc,
CurrentAdmin,
CurrentUser,
MessageRatingSvc,
)
from app.schemas.conversation import (
ConversationAdminList,
ConversationCreate,
ConversationList,
ConversationRead,
ConversationReadWithMessages,
ConversationUpdate,
MessageCreate,
MessageList,
MessageRead,
)
from app.schemas.conversation_share import (
ConversationShareCreate,
ConversationShareList,
ConversationShareRead,
)
from app.schemas.message_rating import (
MessageRatingCreate,
MessageRatingRead,
)
router = APIRouter()
@router.get("/export")
async def export_conversations(
conversation_service: ConversationSvc,
_: CurrentAdmin,
) -> Any:
"""Export all conversations with messages and tool calls (admin only)."""
export_data = await conversation_service.export_all()
return JSONResponse(
content={"conversations": export_data, "total": len(export_data)},
headers={"Content-Disposition": 'attachment; filename="conversations_export.json"'},
)
@router.get("/admin-list", response_model=ConversationAdminList)
async def list_conversations_admin(
conversation_service: ConversationSvc,
_: CurrentAdmin,
skip: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=100),
include_archived: bool = Query(True, description="Include archived conversations"),
search: str | None = Query(None, max_length=100, description="Search by title or ID prefix"),
) -> Any:
"""List all conversations with message counts (admin only).
Returns paginated conversations without message content.
"""
items, total = await conversation_service.list_conversations_admin(
skip=skip,
limit=limit,
include_archived=include_archived,
search=search,
)
return ConversationAdminList(items=items, total=total)
@router.get("", response_model=ConversationList)
async def list_conversations(
conversation_service: ConversationSvc,
current_user: CurrentUser,
skip: int = Query(0, ge=0, description="Number of conversations to skip"),
limit: int = Query(50, ge=1, le=100, description="Maximum conversations to return"),
include_archived: bool = Query(False, description="Include archived conversations"),
) -> Any:
"""List conversations for the current user.
Returns conversations ordered by most recently updated.
"""
items, total = await conversation_service.list_conversations(
user_id=current_user.id,
skip=skip,
limit=limit,
include_archived=include_archived,
)
return ConversationList(items=items, total=total) # type: ignore[arg-type]
@router.post("", response_model=ConversationRead, status_code=status.HTTP_201_CREATED)
async def create_conversation(
conversation_service: ConversationSvc,
current_user: CurrentUser,
data: ConversationCreate | None = None,
) -> Any:
"""Create a new conversation.
The title is optional and can be set later.
"""
if data is None:
data = ConversationCreate()
data = data.model_copy(update={"user_id": current_user.id})
return await conversation_service.create_conversation(data)
@router.get("/{conversation_id}", response_model=ConversationReadWithMessages)
async def get_conversation(
conversation_id: UUID,
conversation_service: ConversationSvc,
current_user: CurrentUser,
) -> Any:
"""Get a conversation with all its messages.
Raises 404 if the conversation does not exist.
"""
uid = None if current_user.role == "admin" else current_user.id
return await conversation_service.get_conversation(
conversation_id,
include_messages=True,
user_id=uid,
)
@router.patch("/{conversation_id}", response_model=ConversationRead)
async def update_conversation(
conversation_id: UUID,
data: ConversationUpdate,
conversation_service: ConversationSvc,
current_user: CurrentUser,
) -> Any:
"""Update a conversation's title or archived status.
Raises 404 if the conversation does not exist.
"""
return await conversation_service.update_conversation(
conversation_id,
data,
user_id=current_user.id,
)
@router.delete("/{conversation_id}", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
async def delete_conversation(
conversation_id: UUID,
conversation_service: ConversationSvc,
current_user: CurrentUser,
) -> None:
"""Delete a conversation and all its messages.
Raises 404 if the conversation does not exist.
"""
await conversation_service.delete_conversation(
conversation_id,
user_id=current_user.id,
)
@router.post(
"/{conversation_id}/archive",
response_model=ConversationRead,
)
async def archive_conversation(
conversation_id: UUID,
conversation_service: ConversationSvc,
current_user: CurrentUser,
) -> Any:
"""Archive a conversation.
Archived conversations are hidden from the default list view.
"""
return await conversation_service.archive_conversation(
conversation_id,
user_id=current_user.id,
)
@router.get("/{conversation_id}/messages", response_model=MessageList)
async def list_messages(
conversation_id: UUID,
conversation_service: ConversationSvc,
current_user: CurrentUser,
skip: int = Query(0, ge=0),
limit: int = Query(100, ge=1, le=500),
) -> Any:
"""List messages in a conversation.
Returns messages ordered by creation time (oldest first).
"""
uid = None if current_user.role == "admin" else current_user.id
items, total = await conversation_service.list_messages(
conversation_id,
skip=skip,
limit=limit,
include_tool_calls=True,
user_id=uid,
)
return MessageList(items=items, total=total) # type: ignore[arg-type]
@router.post(
"/{conversation_id}/messages",
response_model=MessageRead,
status_code=status.HTTP_201_CREATED,
)
async def add_message(
conversation_id: UUID,
data: MessageCreate,
conversation_service: ConversationSvc,
current_user: CurrentUser,
) -> Any:
"""Add a message to a conversation.
Raises 404 if the conversation does not exist.
"""
return await conversation_service.add_message(conversation_id, data)
@router.post(
"/{conversation_id}/messages/{message_id}/rate",
response_model=MessageRatingRead,
status_code=status.HTTP_200_OK,
)
async def rate_message(
conversation_id: UUID,
message_id: UUID,
data: MessageRatingCreate,
rating_service: MessageRatingSvc,
current_user: CurrentUser,
response: Response,
) -> Any:
"""Rate an assistant message.
Creates a new rating or updates an existing one.
Only assistant messages can be rated.
Args:
conversation_id: The conversation containing the message
message_id: The message to rate
data: Rating value (1 for like, -1 for dislike) and optional comment
Returns:
201 Created for new rating, 200 OK when updating existing rating.
"""
rating, is_new = await rating_service.rate_message(
conversation_id=conversation_id,
message_id=message_id,
user_id=current_user.id,
data=data,
)
if is_new:
response.status_code = status.HTTP_201_CREATED
return rating
@router.delete(
"/{conversation_id}/messages/{message_id}/rate",
status_code=status.HTTP_204_NO_CONTENT,
response_model=None,
)
async def remove_rating(
conversation_id: UUID,
message_id: UUID,
rating_service: MessageRatingSvc,
current_user: CurrentUser,
) -> None:
"""Remove your rating from a message.
Args:
conversation_id: The conversation containing the message
message_id: The message to remove rating from
"""
await rating_service.remove_rating(
conversation_id=conversation_id,
message_id=message_id,
user_id=current_user.id,
)
@router.get("/shared-with-me", response_model=ConversationList)
async def list_shared_with_me(
share_service: ConversationShareSvc,
current_user: CurrentUser,
skip: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=100),
) -> Any:
"""List conversations shared with the current user."""
items, total = await share_service.list_shared_with_me(current_user.id, skip=skip, limit=limit)
return ConversationList(items=items, total=total)
@router.post(
"/{conversation_id}/shares",
response_model=ConversationShareRead,
status_code=status.HTTP_201_CREATED,
)
async def share_conversation(
conversation_id: UUID,
data: ConversationShareCreate,
share_service: ConversationShareSvc,
current_user: CurrentUser,
) -> Any:
"""Share a conversation with another user or generate a public link."""
result = await share_service.share_conversation(
conversation_id,
shared_by=current_user.id,
shared_with=data.shared_with,
generate_link=data.generate_link,
permission=data.permission,
)
return result["share"]
@router.get("/{conversation_id}/shares", response_model=ConversationShareList)
async def list_shares(
conversation_id: UUID,
share_service: ConversationShareSvc,
current_user: CurrentUser,
) -> Any:
"""List all shares for a conversation (owner only)."""
shares = await share_service.list_shares(conversation_id, current_user.id)
return ConversationShareList(items=shares, total=len(shares))
@router.delete(
"/{conversation_id}/shares/{share_id}",
status_code=status.HTTP_204_NO_CONTENT,
response_model=None,
)
async def revoke_share(
conversation_id: UUID,
share_id: UUID,
share_service: ConversationShareSvc,
current_user: CurrentUser,
) -> None:
"""Revoke a conversation share."""
await share_service.revoke_share(share_id, current_user.id)
@router.get("/shared/{token}")
async def get_shared_conversation(
token: str,
share_service: ConversationShareSvc,
) -> Any:
"""Access a shared conversation via public token (no auth required)."""
return await share_service.get_by_token(token)
backend/app/api/routes/v1/files.py
+121
View File
@@ -0,0 +1,121 @@
"""File upload and download endpoints for chat attachments."""
import logging
from typing import Any
from uuid import UUID
from fastapi import APIRouter, File, HTTPException, UploadFile, status
from fastapi.responses import FileResponse
from app.api.deps import CurrentUser, FileUploadSvc
from app.core.exceptions import NotFoundError
from app.schemas.file import FileInfo, FileUploadResponse
from app.services.file_storage import get_file_storage
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/files", tags=["files"])
@router.post("/upload", response_model=FileUploadResponse, status_code=status.HTTP_201_CREATED)
async def upload_file(
file_upload_svc: FileUploadSvc,
current_user: CurrentUser,
file: UploadFile = File(...),
) -> Any:
"""Upload a file for use in chat."""
data = await file.read()
is_valid, error = file_upload_svc.validate_upload(file.content_type, len(data))
if not is_valid:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=error)
file_type = file_upload_svc.classify_file(file.content_type or "", file.filename or "unknown")
parsed_content = await file_upload_svc.parse_content(data, file_type, file.content_type or "")
storage = get_file_storage()
storage_path = await storage.save(str(current_user.id), file.filename or "unknown", data)
chat_file = await file_upload_svc.create_chat_file(
user_id=current_user.id,
filename=file.filename or "unknown",
mime_type=file.content_type or "application/octet-stream",
size=len(data),
storage_path=storage_path,
file_type=file_type,
parsed_content=parsed_content,
)
return FileUploadResponse(
id=chat_file.id,
filename=chat_file.filename,
mime_type=chat_file.mime_type,
size=chat_file.size,
file_type=chat_file.file_type,
)
@router.get("/{file_id}")
async def download_file(
file_id: UUID,
file_upload_svc: FileUploadSvc,
current_user: CurrentUser,
disposition: str = "inline",
) -> Any:
"""Serve a file. Only the owner can access their files.
By default the response is ``Content-Disposition: inline`` so PDFs, images
and audio/video render directly inside an ``<iframe>`` / media tag (used
by the chat file-preview panel). Pass ``?disposition=attachment`` to force
the browser's download dialog (used by the explicit "Download" button).
"""
try:
chat_file = await file_upload_svc.get_user_file(file_id, current_user.id)
except NotFoundError:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="File not found"
) from None
storage = get_file_storage()
file_path = storage.get_full_path(chat_file.storage_path)
if not file_path:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="File not found on disk")
# FastAPI's ``FileResponse(filename=...)`` always uses ``attachment`` —
# build the header manually so we can switch to ``inline`` for previews.
mode = "attachment" if disposition == "attachment" else "inline"
safe_name = chat_file.filename.replace('"', "")
# The chat file-preview panel embeds this URL in an iframe (PDFs, HTML,
# etc). Default ``X-Frame-Options: DENY`` from SecurityHeadersMiddleware
# would break that, so opt this endpoint down to SAMEORIGIN. The CSP
# ``frame-ancestors 'self'`` is the modern equivalent — browsers honor
# whichever they recognize.
headers = {
"Content-Disposition": f'{mode}; filename="{safe_name}"',
"X-Frame-Options": "SAMEORIGIN",
"Content-Security-Policy": "frame-ancestors 'self'",
}
return FileResponse(path=file_path, media_type=chat_file.mime_type, headers=headers)
@router.get("/{file_id}/info", response_model=FileInfo)
async def get_file_info(
file_id: UUID,
file_upload_svc: FileUploadSvc,
current_user: CurrentUser,
) -> Any:
"""Get file metadata. Only the owner can access."""
try:
chat_file = await file_upload_svc.get_user_file(file_id, current_user.id)
except NotFoundError:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="File not found"
) from None
return FileInfo(
id=chat_file.id,
filename=chat_file.filename,
mime_type=chat_file.mime_type,
size=chat_file.size,
file_type=chat_file.file_type,
created_at=chat_file.created_at,
user_id=chat_file.user_id,
)
backend/app/api/routes/v1/health.py
+186
View File
@@ -0,0 +1,186 @@
"""Health check endpoints.
Provides Kubernetes-compatible health check endpoints:
- /health - Simple liveness check
- /health/live - Detailed liveness probe
- /health/ready - Readiness probe with dependency checks
"""
# ruff: noqa: I001 - Imports structured for Jinja2 template conditionals
from datetime import UTC, datetime
from typing import Any
from fastapi import APIRouter
from fastapi.responses import JSONResponse
from sqlalchemy import text
from app.api.deps import DBSession, Redis
from app.core.config import settings
from app.services.health import build_health_response
router = APIRouter()
@router.get("/health")
async def health_check() -> dict[str, Any]:
"""Simple liveness probe - check if application is running.
This is a lightweight check that should always succeed if the
application is running. Use this for basic connectivity tests.
Returns:
{"status": "healthy"}
"""
return {
"status": "healthy",
"max_upload_size_mb": settings.MAX_UPLOAD_SIZE_MB,
}
@router.get("/health/live")
async def liveness_probe() -> dict[str, Any]:
"""Detailed liveness probe for Kubernetes.
This endpoint is designed for Kubernetes liveness probes.
It checks if the application process is alive and responding.
Failure indicates the container should be restarted.
Returns:
Structured response with timestamp and service info.
"""
return build_health_response(
status="alive",
details={
"version": getattr(settings, "VERSION", "1.0.0"),
"environment": settings.ENVIRONMENT,
},
)
@router.get("/health/ready", response_model=None)
async def readiness_probe(
db: DBSession,
redis: Redis,
) -> dict[str, Any] | JSONResponse:
"""Readiness probe for Kubernetes.
This endpoint checks if all dependencies are ready to handle traffic.
It verifies database connections, Redis, and other critical services.
Failure indicates traffic should be temporarily diverted.
Checks performed:
- Database connectivity
- Redis connectivity
Returns:
Structured response with individual check results.
Returns 503 if any critical check fails.
"""
checks: dict[str, dict[str, Any]] = {}
# Database check
try:
start = datetime.now(UTC)
await db.execute(text("SELECT 1"))
latency_ms = (datetime.now(UTC) - start).total_seconds() * 1000
checks["database"] = {
"status": "healthy",
"latency_ms": round(latency_ms, 2),
"type": "postgresql",
}
except Exception as e:
checks["database"] = {
"status": "unhealthy",
"error": str(e),
"type": "postgresql",
}
# Redis check
try:
start = datetime.now(UTC)
is_healthy = await redis.ping()
latency_ms = (datetime.now(UTC) - start).total_seconds() * 1000
if is_healthy:
checks["redis"] = {
"status": "healthy",
"latency_ms": round(latency_ms, 2),
}
else:
checks["redis"] = {
"status": "unhealthy",
"error": "Ping failed",
}
except Exception as e:
checks["redis"] = {
"status": "unhealthy",
"error": str(e),
}
# Vector store — Milvus connectivity probe (TCP).
try:
import socket
start = datetime.now(UTC)
with socket.create_connection((settings.MILVUS_HOST, settings.MILVUS_PORT), timeout=2):
pass
latency_ms = (datetime.now(UTC) - start).total_seconds() * 1000
checks["vector_store"] = {
"status": "healthy",
"latency_ms": round(latency_ms, 2),
"type": "milvus",
}
except Exception as e:
checks["vector_store"] = {
"status": "unhealthy",
"error": str(e),
"type": "milvus",
}
# LLM provider — config-only check (avoid spending money on a probe call).
llm_provider = (getattr(settings, "LLM_PROVIDER", None) or "").lower()
if llm_provider:
key_field = {
"openai": "OPENAI_API_KEY",
"anthropic": "ANTHROPIC_API_KEY",
"google": "GOOGLE_API_KEY",
"openrouter": "OPENROUTER_API_KEY",
}.get(llm_provider)
api_key = getattr(settings, key_field, None) if key_field else None
checks["llm"] = {
"status": "healthy" if api_key else "unhealthy",
"provider": llm_provider,
"detail": "API key configured" if api_key else "API key missing",
}
else:
checks["llm"] = {"status": "unknown", "detail": "not configured"}
# Determine overall health — only db + redis are critical for readiness.
critical = {k: v for k, v in checks.items() if k in ("database", "redis")}
all_healthy = (
all(check.get("status") == "healthy" for check in critical.values()) if critical else True
)
# The admin /system page reads each service from the top level, so flatten
# the checks alongside the structured `checks` field for K8s probes.
response_data = build_health_response(
status="ready" if all_healthy else "not_ready",
checks=checks,
)
response_data.update(checks)
if not all_healthy:
return JSONResponse(status_code=503, content=response_data)
return response_data
# Backward compatibility - keep /ready endpoint
@router.get("/ready", response_model=None)
async def readiness_check(
db: DBSession,
redis: Redis,
) -> dict[str, Any] | JSONResponse:
"""Readiness check (alias for /health/ready).
Deprecated: Use /health/ready instead.
"""
return await readiness_probe(
db=db,
redis=redis,
)
backend/app/api/routes/v1/me_slash_commands.py
+85
View File
@@ -0,0 +1,85 @@
"""User-scoped slash command settings.
Routes are nested under ``/me/slash-commands`` because they're always
operating on the current user — there's no cross-user view of these.
"""
from typing import Any
from uuid import UUID
from fastapi import APIRouter, status
from app.api.deps import CurrentUser, UserSlashCommandSvc
from app.schemas.user_slash_command import (
BuiltinOverrideUpsert,
UserSlashCommandCustomCreate,
UserSlashCommandList,
UserSlashCommandRead,
UserSlashCommandUpdate,
)
router = APIRouter()
@router.get("", response_model=UserSlashCommandList)
async def list_slash_commands(service: UserSlashCommandSvc, user: CurrentUser) -> Any:
"""List the current user's custom commands and built-in overrides."""
items, total = await service.list_for_user(user_id=user.id)
return UserSlashCommandList(
items=[UserSlashCommandRead.model_validate(c) for c in items],
total=total,
)
@router.post(
"/custom",
response_model=UserSlashCommandRead,
status_code=status.HTTP_201_CREATED,
)
async def create_custom_command(
data: UserSlashCommandCustomCreate,
service: UserSlashCommandSvc,
user: CurrentUser,
) -> Any:
"""Create a user-defined command with a stored prompt body."""
db_cmd = await service.create_custom(user_id=user.id, data=data)
return UserSlashCommandRead.model_validate(db_cmd)
@router.put("/builtin", response_model=UserSlashCommandRead)
async def upsert_builtin_override(
data: BuiltinOverrideUpsert,
service: UserSlashCommandSvc,
user: CurrentUser,
) -> Any:
"""Toggle a built-in command on or off for the current user."""
db_cmd = await service.upsert_builtin_override(
user_id=user.id, name=data.name, is_enabled=data.is_enabled
)
return UserSlashCommandRead.model_validate(db_cmd)
@router.patch(
"/{command_id}",
response_model=UserSlashCommandRead,
)
async def update_slash_command(
command_id: UUID,
data: UserSlashCommandUpdate,
service: UserSlashCommandSvc,
user: CurrentUser,
) -> Any:
"""Patch a custom command. Built-in overrides accept only ``is_enabled``."""
db_cmd = await service.update(user_id=user.id, command_id=command_id, data=data)
return UserSlashCommandRead.model_validate(db_cmd)
@router.delete("/{command_id}", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
async def delete_slash_command(
command_id: UUID,
service: UserSlashCommandSvc,
user: CurrentUser,
) -> Any:
"""Delete a custom command, or remove a built-in override (re-enables it)."""
await service.delete(user_id=user.id, command_id=command_id)
return None
backend/app/api/routes/v1/oauth.py
+58
View File
@@ -0,0 +1,58 @@
"""OAuth2 authentication routes."""
import logging
from urllib.parse import urlencode
from fastapi import APIRouter, Request
from fastapi.responses import RedirectResponse
from app.api.deps import UserSvc
from app.core.config import settings
from app.core.oauth import oauth
from app.core.security import create_access_token, create_refresh_token
logger = logging.getLogger(__name__)
router = APIRouter()
@router.get("/google/login")
async def google_login(request: Request):
"""Redirect to Google OAuth2 login page."""
return await oauth.google.authorize_redirect(request, settings.GOOGLE_REDIRECT_URI)
@router.get("/google/callback")
async def google_callback(request: Request, user_service: UserSvc):
"""Handle Google OAuth2 callback."""
frontend = settings.FRONTEND_URL.rstrip("/")
try:
token = await oauth.google.authorize_access_token(request)
user_info = token.get("userinfo")
if not user_info:
params = urlencode({"error": "Failed to get user info from Google"})
return RedirectResponse(url=f"{frontend}/login?{params}")
user = await user_service.get_or_create_oauth_user(
provider="google",
provider_id=user_info.get("sub"),
email=user_info.get("email"),
full_name=user_info.get("name"),
)
access_token = create_access_token(subject=str(user.id))
refresh_token = create_refresh_token(subject=str(user.id))
params = urlencode(
{
"access_token": access_token,
"refresh_token": refresh_token,
}
)
return RedirectResponse(url=f"{frontend}/auth/callback?{params}")
except Exception:
logger.exception("google_oauth_callback_failed")
params = urlencode({"error": "Sign-in failed. Please try again."})
return RedirectResponse(url=f"{frontend}/login?{params}")
backend/app/api/routes/v1/rag.py
+370
View File
@@ -0,0 +1,370 @@
"""RAG API routes — collection management, search, document upload, sync, status stream.
Routes are HTTP plumbing only. Business logic, file I/O, task dispatch, and Redis
pub/sub all live in their respective services. Domain exceptions raised by services are
mapped to HTTP responses by the global exception handlers in
``app.api.exception_handlers``; routes do not catch and re-wrap them.
"""
from collections.abc import AsyncIterable
from typing import Any
from fastapi import APIRouter, File, Query, UploadFile, status
from fastapi.responses import FileResponse
from fastapi.sse import EventSourceResponse, ServerSentEvent
from app.api.deps import (
CurrentAdmin,
CurrentUser,
IngestionSvc,
RAGDocumentSvc,
RAGStatusSvc,
RAGSyncSvc,
RetrievalSvc,
SyncSourceSvc,
VectorStoreSvc,
)
from app.core.config import settings
from app.core.exceptions import NotFoundError
from app.schemas.rag import (
RAGCollectionInfo,
RAGCollectionList,
RAGDocumentList,
RAGIngestResponse,
RAGMessageResponse,
RAGRetryResponse,
RAGSearchRequest,
RAGSearchResponse,
RAGSearchResult,
RAGSyncLogList,
RAGSyncRequest,
RAGSyncResponse,
RAGTrackedDocumentList,
)
from app.schemas.sync_source import (
ConnectorList,
SyncSourceCreate,
SyncSourceList,
SyncSourceRead,
SyncSourceUpdate,
)
from app.services.rag.config import get_supported_formats
router = APIRouter()
@router.get("/supported-formats")
async def get_supported_formats_endpoint() -> Any:
"""Return file formats supported by the current PDF parser configuration."""
parser_name = getattr(settings, "PDF_PARSER", "pymupdf")
return {"parser": parser_name, "formats": sorted(get_supported_formats(parser_name))}
@router.get("/collections", response_model=RAGCollectionList)
async def list_collections(
vector_store: VectorStoreSvc,
_: CurrentAdmin,
) -> Any:
"""List all available collections in the vector store."""
names = await vector_store.list_collections()
return RAGCollectionList(items=names)
@router.post(
"/collections/{name}",
status_code=status.HTTP_201_CREATED,
response_model=RAGMessageResponse,
)
async def create_collection(
name: str,
vector_store: VectorStoreSvc,
_: CurrentAdmin,
) -> Any:
"""Create and initialize a new collection."""
await vector_store.create_collection(name)
return RAGMessageResponse(message=f"Collection '{name}' created successfully.")
@router.delete(
"/collections/{name}",
status_code=status.HTTP_204_NO_CONTENT,
response_model=None,
)
async def drop_collection(
name: str,
vector_store: VectorStoreSvc,
rag_doc_svc: RAGDocumentSvc,
_: CurrentAdmin,
) -> None:
"""Drop an entire collection — vectors and all SQL document records."""
await vector_store.delete_collection(name)
await rag_doc_svc.delete_by_collection(name)
@router.get("/collections/{name}/info", response_model=RAGCollectionInfo)
async def get_collection_info(
name: str,
vector_store: VectorStoreSvc,
_: CurrentAdmin,
) -> Any:
"""Retrieve stats for a specific collection."""
return await vector_store.get_collection_info(name)
@router.get("/collections/{name}/documents", response_model=RAGDocumentList)
async def list_documents(
name: str,
vector_store: VectorStoreSvc,
_: CurrentAdmin,
) -> Any:
"""List all documents in a specific collection."""
return await vector_store.get_document_list(name)
@router.post("/search", response_model=RAGSearchResponse)
async def search_documents(
request: RAGSearchRequest,
retrieval_service: RetrievalSvc,
_: CurrentUser,
use_reranker: bool = Query(False, description="Whether to use reranking (if configured)"),
) -> Any:
"""Search for relevant document chunks. Supports multi-collection search."""
if request.collection_names and len(request.collection_names) > 1:
results = await retrieval_service.retrieve_multi(
query=request.query,
collection_names=request.collection_names,
limit=request.limit,
min_score=request.min_score,
use_reranker=use_reranker,
)
else:
collection = (
request.collection_names[0] if request.collection_names else request.collection_name
)
results = await retrieval_service.retrieve(
query=request.query,
collection_name=collection,
limit=request.limit,
min_score=request.min_score,
filter=request.filter or "",
use_reranker=use_reranker,
)
api_results = [RAGSearchResult(**hit.model_dump()) for hit in results]
return RAGSearchResponse(results=api_results)
@router.delete(
"/collections/{name}/documents/{document_id}",
status_code=status.HTTP_204_NO_CONTENT,
response_model=None,
)
async def delete_document(
name: str,
document_id: str,
ingestion_service: IngestionSvc,
_: CurrentAdmin,
) -> None:
"""Delete a specific document by its ID from a collection."""
success = await ingestion_service.remove_document(name, document_id)
if not success:
raise NotFoundError(
message="Document not found",
details={"collection": name, "document_id": document_id},
)
@router.post(
"/collections/{name}/ingest",
response_model=RAGIngestResponse,
response_model_exclude_none=True,
status_code=status.HTTP_202_ACCEPTED,
)
async def ingest_file(
name: str,
rag_doc_svc: RAGDocumentSvc,
vector_store: VectorStoreSvc,
_: CurrentAdmin,
file: UploadFile = File(...),
replace: bool = Query(False),
) -> Any:
"""Upload and queue a file for ingestion into a collection."""
data = await file.read()
return await rag_doc_svc.dispatch_upload(
collection_name=name,
file_data=data,
filename=file.filename or "unknown",
replace=replace,
vector_store=vector_store,
)
@router.get("/documents", response_model=RAGTrackedDocumentList)
async def list_rag_documents(
rag_doc_svc: RAGDocumentSvc,
_: CurrentAdmin,
collection_name: str | None = Query(None),
) -> Any:
"""List tracked RAG documents."""
return await rag_doc_svc.list_documents(collection_name)
@router.get("/documents/{doc_id}/download")
async def download_rag_document(
doc_id: str,
rag_doc_svc: RAGDocumentSvc,
_: CurrentAdmin,
) -> Any:
"""Download the original file."""
file_path, filename, mime_type = await rag_doc_svc.get_download_info(doc_id)
return FileResponse(path=file_path, filename=filename, media_type=mime_type)
@router.delete(
"/documents/{doc_id}",
status_code=status.HTTP_204_NO_CONTENT,
response_model=None,
)
async def delete_rag_document(
doc_id: str,
rag_doc_svc: RAGDocumentSvc,
ingestion_service: IngestionSvc,
_: CurrentAdmin,
) -> None:
"""Delete a document from SQL, vector store, and file storage."""
await rag_doc_svc.delete_document(doc_id, ingestion_service)
@router.post("/documents/{doc_id}/retry", response_model=RAGRetryResponse)
async def retry_ingestion(
doc_id: str,
rag_doc_svc: RAGDocumentSvc,
_: CurrentAdmin,
) -> Any:
"""Retry a failed document ingestion."""
doc = await rag_doc_svc.retry_ingestion(doc_id)
return RAGRetryResponse(id=str(doc.id), status="processing", message="Retry queued")
@router.get("/sync/logs", response_model=RAGSyncLogList)
async def list_sync_logs(
rag_sync_svc: RAGSyncSvc,
_: CurrentAdmin,
collection_name: str | None = Query(None),
limit: int = Query(20, ge=1, le=100),
) -> Any:
"""List sync operation logs."""
return await rag_sync_svc.list_sync_logs(collection_name=collection_name, limit=limit)
@router.post("/sync/local", response_model=RAGSyncResponse)
async def trigger_local_sync(
request: RAGSyncRequest,
rag_sync_svc: RAGSyncSvc,
_: CurrentAdmin,
) -> Any:
"""Trigger a local directory sync via background task."""
sync_log = await rag_sync_svc.start_local_sync(
collection_name=request.collection_name,
mode=request.mode,
path=request.path,
)
return RAGSyncResponse(
id=str(sync_log.id),
status="running",
message=f"Sync started for '{request.collection_name}' (mode={request.mode})",
)
@router.delete("/sync/{sync_id}", response_model=RAGMessageResponse)
async def cancel_sync(
sync_id: str,
rag_sync_svc: RAGSyncSvc,
_: CurrentAdmin,
) -> Any:
"""Cancel a running sync operation."""
await rag_sync_svc.cancel_sync(sync_id)
return RAGMessageResponse(message="Sync cancelled")
@router.get("/sync/sources", response_model=SyncSourceList)
async def list_sync_sources(
sync_source_svc: SyncSourceSvc,
_: CurrentAdmin,
) -> Any:
"""List all configured sync sources."""
return await sync_source_svc.list_sources()
@router.post(
"/sync/sources",
response_model=SyncSourceRead,
status_code=status.HTTP_201_CREATED,
)
async def create_sync_source(
data: SyncSourceCreate,
sync_source_svc: SyncSourceSvc,
_: CurrentAdmin,
) -> Any:
"""Create a new sync source configuration."""
return await sync_source_svc.create_source(data)
@router.patch("/sync/sources/{source_id}", response_model=SyncSourceRead)
async def update_sync_source(
source_id: str,
data: SyncSourceUpdate,
sync_source_svc: SyncSourceSvc,
_: CurrentAdmin,
) -> Any:
"""Update an existing sync source configuration."""
return await sync_source_svc.update_source(source_id, data)
@router.delete(
"/sync/sources/{source_id}",
status_code=status.HTTP_204_NO_CONTENT,
response_model=None,
)
async def delete_sync_source(
source_id: str,
sync_source_svc: SyncSourceSvc,
_: CurrentAdmin,
) -> None:
"""Delete a sync source configuration."""
await sync_source_svc.delete_source(source_id)
@router.post("/sync/sources/{source_id}/trigger", response_model=RAGSyncResponse)
async def trigger_sync_source(
source_id: str,
sync_source_svc: SyncSourceSvc,
_: CurrentAdmin,
) -> Any:
"""Trigger a manual sync for a configured source."""
sync_log = await sync_source_svc.trigger_sync(source_id)
return RAGSyncResponse(
id=str(sync_log.id),
status="running",
message=f"Sync triggered for source '{source_id}'",
)
@router.get("/sync/connectors", response_model=ConnectorList)
async def list_connectors(
sync_source_svc: SyncSourceSvc,
_: CurrentAdmin,
) -> Any:
"""List available sync connector types with their config schemas."""
return sync_source_svc.list_connectors()
@router.get("/status/stream", response_class=EventSourceResponse)
async def rag_status_stream(
rag_status_svc: RAGStatusSvc,
) -> AsyncIterable[ServerSentEvent]:
"""SSE endpoint for real-time RAG ingestion status updates.
Subscribes to the ``rag_status`` Redis pub/sub channel; the browser auto-reconnects
via the EventSource API.
"""
return rag_status_svc.stream_events()
backend/app/api/routes/v1/users.py
+133
View File
@@ -0,0 +1,133 @@
# ruff: noqa: I001 - Imports structured for Jinja2 template conditionals
"""User management routes."""
from typing import Any
from uuid import UUID
from fastapi import APIRouter, File, HTTPException, UploadFile, status
from fastapi.responses import FileResponse
from fastapi_pagination import Page
from app.api.deps import (
CurrentAdmin,
CurrentUser,
UserSvc,
)
from app.db.models.user import UserRole
from app.schemas.user import UserRead, UserUpdate
from app.services.file_storage import get_file_storage
router = APIRouter()
@router.get("/me", response_model=UserRead)
async def read_current_user(
current_user: CurrentUser,
) -> Any:
"""Get current user.
Returns the authenticated user's profile including their role.
"""
return current_user
@router.patch("/me", response_model=UserRead)
async def update_current_user(
user_in: UserUpdate,
current_user: CurrentUser,
user_service: UserSvc,
) -> Any:
"""Update current user.
Users can update their own profile (email, full_name).
Role changes require admin privileges.
"""
# Prevent non-admin users from changing their own role
if user_in.role is not None and not current_user.has_role(UserRole.ADMIN):
user_in.role = None
user = await user_service.update(current_user.id, user_in)
return user
@router.post("/me/avatar", response_model=UserRead)
async def upload_avatar(
user_service: UserSvc,
current_user: CurrentUser,
file: UploadFile = File(...),
) -> Any:
"""Upload or replace avatar image for the current user."""
data = await file.read()
try:
user = await user_service.update_avatar(
current_user.id, data, file.filename or "avatar.jpg", file.content_type or ""
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e)) from None
return user
@router.get("/avatar/{user_id}")
async def get_avatar(user_id: UUID, user_service: UserSvc) -> Any:
"""Get user avatar image."""
user = await user_service.get_by_id(user_id)
if not user.avatar_url:
raise HTTPException(status_code=404, detail="No avatar set")
storage = get_file_storage()
file_path = storage.get_full_path(user.avatar_url)
if not file_path:
raise HTTPException(status_code=404, detail="Avatar file not found")
return FileResponse(path=file_path, media_type="image/jpeg")
@router.get("", response_model=Page[UserRead])
async def read_users(
user_service: UserSvc,
_: CurrentAdmin,
) -> Any:
"""Get all users (admin only)."""
return await user_service.list_paginated()
@router.get("/{user_id}", response_model=UserRead)
async def read_user(
user_id: UUID,
user_service: UserSvc,
_: CurrentAdmin,
) -> Any:
"""Get user by ID (admin only).
Raises NotFoundError if user does not exist.
"""
user = await user_service.get_by_id(user_id)
return user
@router.patch("/{user_id}", response_model=UserRead)
async def update_user_by_id(
user_id: UUID,
user_in: UserUpdate,
user_service: UserSvc,
_: CurrentAdmin,
) -> Any:
"""Update user by ID (admin only).
Admins can update any user including their role.
Raises NotFoundError if user does not exist.
"""
user = await user_service.update(user_id, user_in)
return user
@router.delete("/{user_id}", status_code=status.HTTP_204_NO_CONTENT, response_model=None)
async def delete_user_by_id(
user_id: UUID,
user_service: UserSvc,
_: CurrentAdmin,
) -> None:
"""Delete user by ID (admin only).
Raises NotFoundError if user does not exist.
"""
await user_service.delete(user_id)
backend/app/api/versioning.py
+219
View File
@@ -0,0 +1,219 @@
"""API versioning utilities and deprecation handling.
This module provides tools for managing API version deprecation:
- Deprecation middleware for entire API versions
- Deprecation decorator for individual endpoints
- RFC 8594 compliant deprecation headers
"""
import logging
from collections.abc import Callable
from datetime import datetime
from functools import wraps
from typing import Any
from fastapi import Request, Response
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from starlette.types import ASGIApp
logger = logging.getLogger(__name__)
class VersionDeprecationMiddleware(BaseHTTPMiddleware):
"""Middleware to add deprecation headers for deprecated API versions.
Adds RFC 8594 compliant headers:
- Deprecation: Indicates the version is deprecated
- Sunset: Indicates when the version will be removed
- Link: Points to migration documentation
Usage in main.py:
app.add_middleware(
VersionDeprecationMiddleware,
deprecated_versions={"v1": {"sunset": "2025-06-01", "link": "/docs/migration/v2"}},
)
"""
def __init__(
self,
app: ASGIApp,
deprecated_versions: dict[str, dict[str, str]] | None = None,
) -> None:
"""Initialize the middleware.
Args:
app: The ASGI application.
deprecated_versions: Dict mapping version prefixes to deprecation info.
Each entry should have:
- sunset: ISO date string when version will be removed (optional)
- link: URL to migration documentation (optional)
- message: Custom deprecation message (optional)
Example:
{
"v1": {
"sunset": "2025-06-01",
"link": "https://api.example.com/docs/migration/v2",
"message": "Please migrate to API v2",
}
}
"""
super().__init__(app)
self.deprecated_versions = deprecated_versions or {}
async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
"""Process the request and add deprecation headers if needed."""
response = await call_next(request)
# Check if request path matches a deprecated version
path = request.url.path
for version, info in self.deprecated_versions.items():
if f"/api/{version}/" in path or path.endswith(f"/api/{version}"):
self._add_deprecation_headers(response, version, info)
self._log_deprecated_usage(request, version)
break
return response
def _add_deprecation_headers(
self, response: Response, version: str, info: dict[str, str]
) -> None:
"""Add RFC 8594 deprecation headers to the response."""
# Deprecation header - indicates the API is deprecated
response.headers["Deprecation"] = "true"
# Sunset header - when the API will be removed
if sunset := info.get("sunset"):
# Convert to HTTP date format
sunset_date = datetime.fromisoformat(sunset)
response.headers["Sunset"] = sunset_date.strftime("%a, %d %b %Y %H:%M:%S GMT")
# Link header - documentation for migration
if link := info.get("link"):
response.headers["Link"] = f'<{link}>; rel="deprecation"'
# Custom warning header
message = info.get("message", f"API {version} is deprecated")
response.headers["X-API-Deprecation-Warning"] = message
def _log_deprecated_usage(self, request: Request, version: str) -> None:
"""Log usage of deprecated API version for monitoring."""
logger.warning(
"Deprecated API version accessed: %s %s %s",
version,
request.method,
request.url.path,
)
def deprecated(
sunset: str | None = None,
message: str | None = None,
link: str | None = None,
) -> Callable[..., Any]:
"""Decorator to mark an endpoint as deprecated.
Adds deprecation headers to responses from the decorated endpoint.
Use this for deprecating individual endpoints within an active API version.
Args:
sunset: ISO date string when endpoint will be removed.
message: Custom deprecation message.
link: URL to migration documentation.
Usage:
@router.get("/old-endpoint")
@deprecated(
sunset="2025-06-01",
message="Use /new-endpoint instead",
link="/docs/migration",
)
async def old_endpoint():
...
"""
def decorator(func: Callable[..., Any]) -> Callable[..., Any]:
@wraps(func)
async def wrapper(*args: Any, **kwargs: Any) -> Any:
# Get the response from the endpoint
result = await func(*args, **kwargs)
# Find Response object in args (FastAPI injects it)
response = None
for arg in args:
if isinstance(arg, Response):
response = arg
break
for value in kwargs.values():
if isinstance(value, Response):
response = value
break
# If we have a Response object, add headers
if response:
response.headers["Deprecation"] = "true"
if sunset:
sunset_date = datetime.fromisoformat(sunset)
response.headers["Sunset"] = sunset_date.strftime("%a, %d %b %Y %H:%M:%S GMT")
if link:
response.headers["Link"] = f'<{link}>; rel="deprecation"'
if message:
response.headers["X-API-Deprecation-Warning"] = message
return result
# Add deprecation info to OpenAPI schema
wrapper.__doc__ = (
f"{func.__doc__ or ''}\n\n"
f"**DEPRECATED**"
f"{f': {message}' if message else ''}"
f"{f' (Sunset: {sunset})' if sunset else ''}"
)
return wrapper
return decorator
# Example usage documentation
"""
## Adding a New API Version
1. Create a new version folder:
```
app/api/routes/v2/
├── __init__.py
├── health.py
├── auth.py
└── ...
```
2. Create the v2 router in `v2/__init__.py`:
```python
from fastapi import APIRouter
v2_router = APIRouter()
# Include routes...
```
3. Add the v2 router in `app/api/router.py`:
```python
from app.api.routes.v2 import v2_router
api_router.include_router(v1_router, prefix="/v1")
api_router.include_router(v2_router, prefix="/v2")
```
4. Mark v1 as deprecated in `main.py`:
```python
app.add_middleware(
VersionDeprecationMiddleware,
deprecated_versions={
"v1": {
"sunset": "2025-12-31",
"link": "/docs/migration/v2",
"message": "Please migrate to API v2",
}
},
)
```
"""
backend/app/clients/__init__.py
+10
View File
@@ -0,0 +1,10 @@
"""External service clients.
This module contains thin wrappers around external services like Redis.
"""
from app.clients.redis import RedisClient
__all__ = [
"RedisClient",
]
backend/app/clients/redis.py
+84
View File
@@ -0,0 +1,84 @@
"""Redis client wrapper.
Provides a class-based Redis client for connection management and operations.
"""
from redis import asyncio as aioredis
from app.core.config import settings
class RedisClient:
"""Redis client wrapper for connection lifecycle management.
Usage in FastAPI lifespan:
async with contextmanager():
redis = RedisClient(settings.REDIS_URL)
await redis.connect()
yield {"redis": redis}
await redis.close()
"""
def __init__(self, url: str | None = None):
self.url = url or settings.REDIS_URL
self.client: aioredis.Redis | None = None
async def connect(self) -> None:
"""Connect to Redis server."""
self.client = aioredis.from_url( # type: ignore[no-untyped-call]
self.url,
encoding="utf-8",
decode_responses=True,
)
async def close(self) -> None:
"""Close Redis connection."""
if self.client:
await self.client.close()
self.client = None
async def get(self, key: str) -> str | None:
"""Get a value by key."""
if not self.client:
raise RuntimeError("Redis client not connected")
return await self.client.get(key) # type: ignore[no-any-return]
async def set(
self,
key: str,
value: str,
ttl: int | None = None,
) -> None:
"""Set a value with optional TTL (in seconds)."""
if not self.client:
raise RuntimeError("Redis client not connected")
await self.client.set(key, value, ex=ttl)
async def delete(self, key: str) -> int:
"""Delete a key. Returns number of keys deleted."""
if not self.client:
raise RuntimeError("Redis client not connected")
return await self.client.delete(key) # type: ignore[no-any-return]
async def exists(self, key: str) -> bool:
"""Check if key exists."""
if not self.client:
raise RuntimeError("Redis client not connected")
return bool(await self.client.exists(key))
async def ping(self) -> bool:
"""Ping Redis server. Returns True if connected."""
if not self.client:
return False
try:
await self.client.ping()
return True
except Exception:
return False
@property
def raw(self) -> aioredis.Redis:
"""Access the underlying aioredis client for advanced operations."""
if not self.client:
raise RuntimeError("Redis client not connected")
return self.client
backend/app/commands/__init__.py
+120
View File
@@ -0,0 +1,120 @@
"""
Custom commands system with auto-discovery.
This module provides a Django-like custom commands system for FastAPI + Click.
Commands are auto-discovered from this package and registered to the CLI.
Usage:
# In app/commands/my_command.py
from app.commands import command
import click
@command("my-command", help="Description of my command")
@click.option("--option", "-o", help="Some option")
def my_command(option: str):
click.echo(f"Running with {option}")
# Then use it:
# project cmd my-command --option value
"""
import importlib
import pkgutil
from collections.abc import Callable
from pathlib import Path
from typing import Any
import click
# Registry for custom commands
_commands: list[click.Command] = []
_discovered = False
def command(name: str | None = None, **kwargs: Any) -> Callable[..., Any]:
"""
Decorator to register a custom command.
Args:
name: Command name (defaults to function name with underscores replaced by hyphens)
**kwargs: Additional arguments passed to click.command()
Example:
@command("seed", help="Seed database with initial data")
@click.option("--count", "-c", default=10)
def seed_data(count: int):
click.echo(f"Seeding {count} records...")
"""
def decorator(func: Callable[..., Any]) -> click.Command:
cmd_name = name or func.__name__.replace("_", "-")
cmd: click.Command = click.command(cmd_name, **kwargs)(func) # type: ignore[no-untyped-call]
_commands.append(cmd)
return cmd
return decorator
def discover_commands() -> list[click.Command]:
"""
Auto-discover all commands in this package.
Imports all modules in the app.commands package (except those starting with _)
which triggers the @command decorator to register them.
Returns:
List of discovered click.Command objects
"""
global _discovered
if _discovered:
return _commands
package_dir = Path(__file__).parent
for _, module_name, _ in pkgutil.iter_modules([str(package_dir)]):
if module_name.startswith("_"):
continue
try:
importlib.import_module(f"app.commands.{module_name}")
except ImportError as e:
click.secho(
f"Warning: Failed to import command module '{module_name}': {e}", fg="yellow"
)
_discovered = True
return _commands
def register_commands(cli: click.Group) -> None:
"""
Register all discovered commands to a CLI group.
Args:
cli: The click.Group to add commands to
"""
commands = discover_commands()
for cmd in commands:
cli.add_command(cmd)
def success(message: str) -> None:
"""Print success message in green."""
click.secho(message, fg="green")
def error(message: str) -> None:
"""Print error message in red."""
click.secho(message, fg="red")
def warning(message: str) -> None:
"""Print warning message in yellow."""
click.secho(message, fg="yellow")
def info(message: str) -> None:
"""Print info message."""
click.echo(message)
backend/app/commands/cleanup.py
+45
View File
@@ -0,0 +1,45 @@
"""
Cleanup old or stale data from the database.
This command is useful for maintenance tasks.
"""
import asyncio
from datetime import UTC, datetime, timedelta
import click
from app.commands import command, info, success, warning
@command("cleanup", help="Clean up old data from the database")
@click.option("--days", "-d", default=90, type=int, help="Delete records older than N days")
@click.option("--dry-run", is_flag=True, help="Show what would be deleted without making changes")
@click.option("--force", "-f", is_flag=True, help="Skip confirmation prompt")
def cleanup(days: int, dry_run: bool, force: bool) -> None:
"""
Remove old records from the database.
Example:
project cmd cleanup --days 90
project cmd cleanup --days 30 --dry-run
project cmd cleanup --days 7 --force
"""
cutoff_date = datetime.now(UTC) - timedelta(days=days)
if dry_run:
info(f"[DRY RUN] Would delete records older than {cutoff_date}")
return
if not force and not click.confirm(
f"Delete all records older than {days} days ({cutoff_date})?"
):
warning("Aborted.")
return
async def _cleanup() -> None:
info(f"Cleaning up records older than {cutoff_date}...")
total_deleted = 0
success(f"Done. Total deleted: {total_deleted} rows.")
asyncio.run(_cleanup())
backend/app/commands/example.py
+28
View File
@@ -0,0 +1,28 @@
"""
Example custom command.
This is a template showing how to create custom CLI commands.
Copy this file and modify it to create your own commands.
"""
import click
from app.commands import command, info, success
@command("hello", help="Example command that greets the user")
@click.option("--name", "-n", default="World", help="Name to greet")
@click.option("--count", "-c", default=1, type=int, help="Number of greetings")
def hello(name: str, count: int) -> None:
"""
Greet someone multiple times.
Example:
project cmd hello --name Alice --count 3
"""
info(f"Greeting {name} {count} time(s)...")
for i in range(count):
click.echo(f" [{i + 1}] Hello, {name}!")
success("Done!")
backend/app/commands/rag.py
+638
View File
@@ -0,0 +1,638 @@
"""
RAG CLI commands for document management and retrieval.
Commands:
rag-collections - List collections with stats
rag-ingest - Ingest file/directory
rag-search - Search knowledge base
rag-drop - Drop collection
rag-stats - Overall RAG system statistics
rag-sources - List configured sync sources
rag-source-add - Add a new sync source
rag-source-remove - Remove a sync source
rag-source-sync - Trigger sync for a source (or all)
"""
import asyncio
from pathlib import Path
import click
from app.commands import command, error, info, success, warning
from app.services.rag.config import DocumentExtensions, RAGSettings
from app.services.rag.documents import DocumentProcessor
from app.services.rag.embeddings import EmbeddingService
from app.services.rag.ingestion import IngestionService
from app.services.rag.retrieval import RetrievalService
from app.services.rag.vectorstore import BaseVectorStore, MilvusVectorStore
def get_rag_services() -> tuple[
RAGSettings, BaseVectorStore, DocumentProcessor, RetrievalService, IngestionService
]:
"""Initialize RAG services for CLI usage.
Creates and returns all necessary RAG service components:
- Settings (RAG configuration)
- Vector store (Milvus)
- Document processor
- Retrieval service
- Ingestion service
Returns:
Tuple of (settings, vector_store, processor, retrieval, ingestion) services.
"""
settings = RAGSettings()
embedder = EmbeddingService(settings=settings)
vector_store = MilvusVectorStore(settings=settings, embedding_service=embedder)
processor = DocumentProcessor(settings=settings)
retrieval = RetrievalService(vector_store=vector_store, settings=settings)
ingestion = IngestionService(processor=processor, vector_store=vector_store)
return settings, vector_store, processor, retrieval, ingestion
async def list_collections_async(vector_store: BaseVectorStore) -> None:
"""List all collections with their stats.
Args:
vector_store: The Milvus vector store to query.
"""
collection_names = await vector_store.list_collections()
if not collection_names:
info("No collections found.")
return
click.echo(f"\nFound {len(collection_names)} collection(s):\n")
for name in collection_names:
try:
info_obj = await vector_store.get_collection_info(name)
click.echo(f" {name}")
click.echo(f" Vectors: {info_obj.total_vectors:,}")
click.echo(f" Dimension: {info_obj.dim}")
click.echo(f" Status: {info_obj.indexing_status}")
click.echo()
except Exception as e:
warning(f"Could not get info for '{name}': {e}")
@command("rag-collections", help="List collections with stats")
def rag_collections() -> None:
"""List all available collections in the vector store with their statistics."""
_, vector_store, _, _, _ = get_rag_services()
asyncio.run(list_collections_async(vector_store))
async def ingest_path_async(
path: str,
collection: str,
recursive: bool,
vector_store: BaseVectorStore,
processor: DocumentProcessor,
ingestion: IngestionService,
replace: bool = True,
sync_mode: str = "full",
) -> None:
"""Ingest files from a path (file or directory).
Args:
path: Path to a file or directory to ingest.
collection: Target collection name.
recursive: Whether to recursively process directories.
vector_store: The Milvus vector store.
processor: Document processor for parsing files.
ingestion: Ingestion service for storing documents.
"""
target_path = Path(path).resolve()
if not target_path.exists():
error(f"Path does not exist: {target_path}")
return
# Collect files to process
if target_path.is_file():
files = [target_path]
elif target_path.is_dir():
if recursive:
files = list(target_path.rglob("*"))
files = [f for f in files if f.is_file() and not f.name.startswith(".")]
else:
files = list(target_path.iterdir())
files = [f for f in files if f.is_file() and not f.name.startswith(".")]
else:
error(f"Invalid path: {target_path}")
return
if not files:
warning("No files found to ingest.")
return
# Filter by allowed extensions
allowed_extensions = {ext.value for ext in DocumentExtensions}
files = [f for f in files if f.suffix.lower() in allowed_extensions]
if not files:
warning(f"No supported files found. Allowed: {', '.join(allowed_extensions)}")
return
import hashlib
from tqdm import tqdm
from app.db.session import get_db_context
from app.services.rag_document import RAGDocumentService
from app.services.rag_sync import RAGSyncService
info(f"Syncing {len(files)} file(s) into '{collection}' (mode={sync_mode})...")
success_count = 0
error_count = 0
replaced_count = 0
skipped_count = 0
# Create SyncLog
async with get_db_context() as db:
sync_log = await RAGSyncService(db).create_sync_log(
source="local", collection_name=collection, mode=sync_mode
)
sync_log_id = str(sync_log.id)
with tqdm(files, unit="file", desc="Syncing", ncols=80) as pbar:
for filepath in pbar:
pbar.set_postfix_str(filepath.name[:30], refresh=True)
# Sync mode checks
source_path = str(filepath.resolve())
if sync_mode in ("new_only", "update_only"):
existing_id: str | None = await ingestion.find_existing(collection, source_path)
if sync_mode == "new_only":
if existing_id:
# File exists — check if content changed via hash
file_hash: str = hashlib.sha256(filepath.read_bytes()).hexdigest()
existing_hash: str | None = await ingestion.get_existing_hash(
collection, source_path
)
if existing_hash and file_hash == existing_hash:
skipped_count += 1
continue
# Hash changed — will re-ingest below
elif sync_mode == "update_only":
if not existing_id:
# Not in collection — skip (update_only ignores new files)
skipped_count += 1
continue
file_hash = hashlib.sha256(filepath.read_bytes()).hexdigest()
existing_hash = await ingestion.get_existing_hash(collection, source_path)
if existing_hash and file_hash == existing_hash:
skipped_count += 1
continue
# Create RAGDocument record in SQL
async with get_db_context() as db:
rag_doc = await RAGDocumentService(db).create_document(
collection_name=collection,
filename=filepath.name,
filesize=filepath.stat().st_size,
filetype=filepath.suffix.lstrip(".").lower(),
)
doc_id = str(rag_doc.id)
try:
result = await ingestion.ingest_file(
filepath=filepath, collection_name=collection, replace=replace
)
if result.status.value == "done":
success_count += 1
if result.message and "replaced" in result.message:
replaced_count += 1
async with get_db_context() as db:
await RAGDocumentService(db).complete_ingestion(
doc_id, vector_document_id=result.document_id
)
else:
error_count += 1
tqdm.write(f" ✗ {filepath.name}: {result.error_message}")
async with get_db_context() as db:
await RAGDocumentService(db).fail_ingestion(
doc_id, error_message=result.error_message or "Unknown error"
)
except Exception as e:
error_count += 1
tqdm.write(f" ✗ {filepath.name}: {e!s}")
async with get_db_context() as db:
await RAGDocumentService(db).fail_ingestion(doc_id, error_message=str(e))
# Update SyncLog
async with get_db_context() as db:
await RAGSyncService(db).complete_sync(
sync_log_id,
status="done" if error_count == 0 else "error",
total_files=len(files),
ingested=success_count - replaced_count,
updated=replaced_count,
skipped=skipped_count,
failed=error_count,
)
click.echo()
msg = f"Done: {success_count} ingested"
if replaced_count > 0:
msg += f" ({replaced_count} updated)"
if skipped_count > 0:
msg += f", {skipped_count} skipped"
success(msg)
if error_count > 0:
error(f"Failed: {error_count} files")
@command("rag-ingest", help="Ingest file/directory into knowledge base")
@click.argument("path", type=click.Path(exists=True))
@click.option(
"--collection",
"-c",
default="documents",
help="Collection name (default: documents)",
)
@click.option(
"--recursive/--no-recursive",
"-r",
default=False,
help="Recursively process directories (default: False)",
)
@click.option(
"--replace/--no-replace",
default=True,
help="Replace existing documents with same source path (default: True)",
)
@click.option(
"--sync-mode",
type=click.Choice(["full", "new_only", "update_only"]),
default="full",
help="Sync mode: full (replace all), new_only (add new + update changed), update_only (only update changed, skip new)",
)
def rag_ingest(path: str, collection: str, recursive: bool, replace: bool, sync_mode: str) -> None:
"""
Ingest a file or directory into the knowledge base.
PATH: Path to a file or directory to ingest.
Example:
project cmd rag-ingest ./docs
project cmd rag-ingest ./docs --sync-mode new_only
project cmd rag-ingest ./docs --sync-mode update_only
"""
_, vector_store, processor, _, ingestion = get_rag_services()
asyncio.run(
ingest_path_async(
path, collection, recursive, vector_store, processor, ingestion, replace, sync_mode
)
)
async def search_async(
query: str,
collection: str,
top_k: int,
retrieval: RetrievalService,
) -> None:
"""Search the knowledge base.
Args:
query: The search query.
collection: Target collection name.
top_k: Number of results to return.
retrieval: Retrieval service for searching.
"""
info(f"Searching collection '{collection}' for: \"{query}\"")
click.echo()
results = await retrieval.retrieve(
query=query,
collection_name=collection,
limit=top_k,
)
if not results:
warning("No results found.")
return
for i, result in enumerate(results, 1):
click.echo(f"--- Result {i} (score: {result.score:.4f}) ---")
# Show source info if available
if result.metadata:
filename = result.metadata.get("filename", "Unknown")
page_num = result.metadata.get("page_num", "?")
click.echo(f"Source: {filename} (page {page_num})")
# Show content (truncated)
content = result.content[:500]
if len(result.content) > 500:
content += "..."
click.echo(content)
click.echo()
@command("rag-search", help="Search knowledge base")
@click.argument("query")
@click.option(
"--collection",
"-c",
default="documents",
help="Collection name (default: documents)",
)
@click.option(
"--top-k",
"-k",
default=4,
type=int,
help="Number of results to return (default: 4)",
)
def rag_search(query: str, collection: str, top_k: int) -> None:
"""
Search the knowledge base for relevant content.
QUERY: The search query.
Example:
project cmd rag-search "what is fastapi"
project cmd rag-search "deployment guide" --collection docs --top-k 10
"""
_, _, _, retrieval, _ = get_rag_services()
asyncio.run(search_async(query, collection, top_k, retrieval))
async def drop_collection_async(collection: str, yes: bool, vector_store: BaseVectorStore) -> None:
"""Drop a collection.
Args:
collection: Name of the collection to drop.
yes: Whether to skip confirmation prompt.
vector_store: The Milvus vector store.
"""
if not yes:
click.confirm(
f"Are you sure you want to drop collection '{collection}'? This cannot be undone.",
abort=True,
)
try:
await vector_store.delete_collection(collection)
success(f"Collection '{collection}' dropped successfully.")
except Exception as e:
error(f"Failed to drop collection: {e}")
@command("rag-drop", help="Drop a collection")
@click.argument("collection")
@click.option(
"--yes",
"-y",
is_flag=True,
help="Skip confirmation prompt",
)
def rag_drop(collection: str, yes: bool) -> None:
"""
Drop a collection and all its data.
COLLECTION: Name of the collection to drop.
Example:
project cmd rag-drop my_collection
project cmd rag-drop my_collection --yes
"""
_, vector_store, _, _, _ = get_rag_services()
asyncio.run(drop_collection_async(collection, yes, vector_store))
@command("rag-stats", help="Show overall RAG system statistics")
def rag_stats() -> None:
"""Display overall RAG system statistics."""
settings, vector_store, _, _, _ = get_rag_services()
asyncio.run(stats_async(settings, vector_store))
async def stats_async(settings: RAGSettings, vector_store: BaseVectorStore) -> None:
"""Show RAG system statistics.
Args:
settings: RAG configuration settings.
vector_store: The Milvus vector store.
"""
click.echo("RAG System Statistics")
click.echo("=" * 40)
# Collection info
try:
collection_names = await vector_store.list_collections()
click.echo(f"\nCollections: {len(collection_names)}")
except Exception as e:
warning(f"Could not list collections: {e}")
collection_names = []
# Configuration
click.echo("\nConfiguration:")
click.echo(f" Embedding model: {settings.embeddings_config.model}")
click.echo(f" Embedding dimension: {settings.embeddings_config.dim}")
click.echo(f" Chunk size: {settings.chunk_size}")
click.echo(f" Chunk overlap: {settings.chunk_overlap}")
click.echo(f" Parser method: {settings.pdf_parser.method}")
# Per-collection stats
if collection_names:
click.echo("\nCollection Details:")
total_vectors = 0
for name in collection_names:
try:
info_obj = await vector_store.get_collection_info(name)
click.echo(f" {name}:")
click.echo(f" Vectors: {info_obj.total_vectors:,}")
total_vectors += info_obj.total_vectors
except Exception:
click.echo(f" {name}: Error getting info")
click.echo(f"\nTotal vectors: {total_vectors:,}")
click.echo()
@command("rag-sources", help="List configured sync sources")
def rag_sources() -> None:
"""List all configured sync sources with their status."""
from app.db.session import get_db_context
async def _list() -> None:
async with get_db_context() as db:
from app.services.sync_source import SyncSourceService
svc = SyncSourceService(db)
sources = await svc.list_sources()
if not sources:
info("No sync sources configured.")
return
click.echo(f"\nFound {len(sources)} sync source(s):\n")
for s in sources:
status_str = s.last_sync_status or "never"
active_str = "active" if s.is_active else "inactive"
click.echo(f" [{active_str}] {s.name} (id={s.id})")
click.echo(f" Type: {s.connector_type}")
click.echo(f" Collection: {s.collection_name}")
click.echo(f" Sync mode: {s.sync_mode}")
if s.schedule_minutes:
click.echo(f" Schedule: every {s.schedule_minutes} min")
else:
click.echo(" Schedule: manual")
click.echo(f" Last sync: {status_str}")
if s.last_error:
click.echo(f" Last error: {s.last_error}")
click.echo()
asyncio.run(_list())
@command("rag-source-add", help="Add a new sync source")
@click.option("--name", required=True, help="Source name")
@click.option("--type", "connector_type", required=True, help="Connector type (e.g. gdrive, s3)")
@click.option("--collection", required=True, help="Target collection name")
@click.option("--config", "config_json", required=True, help="Config JSON string")
@click.option(
"--sync-mode",
default="new_only",
type=click.Choice(["full", "new_only", "update_only"]),
help="Sync mode",
)
@click.option(
"--schedule",
"schedule_minutes",
type=int,
default=0,
help="Schedule interval in minutes (0=manual)",
)
def rag_source_add(
name: str,
connector_type: str,
collection: str,
config_json: str,
sync_mode: str,
schedule_minutes: int,
) -> None:
"""
Add a new sync source configuration.
Example:
project cmd rag-source-add --name "My Drive" --type gdrive --collection docs \\
--config '{"folder_id": "abc123"}' --sync-mode new_only
"""
import json as _json
try:
config_dict = _json.loads(config_json)
except _json.JSONDecodeError as e:
error(f"Invalid JSON config: {e}")
return
from app.schemas.sync_source import SyncSourceCreate
data = SyncSourceCreate(
name=name,
connector_type=connector_type,
collection_name=collection,
config=config_dict,
sync_mode=sync_mode,
schedule_minutes=schedule_minutes if schedule_minutes > 0 else None,
)
from app.db.session import get_db_context
async def _create() -> None:
async with get_db_context() as db:
from app.services.sync_source import SyncSourceService
svc = SyncSourceService(db)
try:
source = await svc.create_source(data)
success(f"Sync source created: {source.name} (id={source.id})")
except ValueError as e:
error(f"Failed to create source: {e}")
asyncio.run(_create())
@command("rag-source-remove", help="Remove a sync source")
@click.argument("source_id")
@click.option("--yes", "-y", is_flag=True, help="Skip confirmation prompt")
def rag_source_remove(source_id: str, yes: bool) -> None:
"""
Remove a sync source configuration.
SOURCE_ID: The ID of the sync source to remove.
Example:
project cmd rag-source-remove abc-123-def
"""
if not yes:
click.confirm(f"Are you sure you want to remove sync source '{source_id}'?", abort=True)
from app.db.session import get_db_context
async def _remove() -> None:
async with get_db_context() as db:
from app.services.sync_source import SyncSourceService
svc = SyncSourceService(db)
try:
await svc.delete_source(source_id)
success(f"Sync source '{source_id}' removed.")
except Exception as e:
error(f"Failed to remove source: {e}")
asyncio.run(_remove())
@command("rag-source-sync", help="Trigger sync for a source")
@click.argument("source_id", required=False)
@click.option("--all", "sync_all", is_flag=True, help="Sync all active sources")
def rag_source_sync(source_id: str | None, sync_all: bool) -> None:
"""
Trigger sync for a configured source (or all active sources).
SOURCE_ID: The ID of the sync source to sync (optional if --all).
Example:
project cmd rag-source-sync abc-123-def
project cmd rag-source-sync --all
"""
if not source_id and not sync_all:
error("Provide a SOURCE_ID or use --all to sync all active sources.")
return
from app.db.session import get_db_context
async def _sync() -> None:
async with get_db_context() as db:
from app.services.sync_source import SyncSourceService
svc = SyncSourceService(db)
if sync_all:
sources = await svc.list_sources(is_active=True)
if not sources:
warning("No active sync sources found.")
return
info(f"Triggering sync for {len(sources)} active source(s)...")
for s in sources:
try:
log = await svc.trigger_sync(str(s.id))
success(f" {s.name}: sync started (log_id={log.id})")
except Exception as e:
error(f" {s.name}: failed - {e}")
else:
try:
assert source_id is not None
log = await svc.trigger_sync(source_id)
success(f"Sync triggered (log_id={log.id})")
except Exception as e:
error(f"Failed to trigger sync: {e}")
asyncio.run(_sync())
backend/app/commands/seed.py
+113
View File
@@ -0,0 +1,113 @@
# ruff: noqa: I001 - Imports structured for Jinja2 template conditionals
"""
Seed database with sample data.
This command is useful for development and testing.
Uses random data generation - install faker for better data:
uv add faker --group dev
"""
import asyncio
import random
import string
import click
from app.commands import command, info, success, warning
# Try to import Faker for better data generation
try:
from faker import Faker
fake = Faker()
HAS_FAKER = True
except ImportError:
HAS_FAKER = False
fake = None
def random_email() -> str:
"""Generate a random email address."""
if HAS_FAKER:
return str(fake.email())
random_str = "".join(random.choices(string.ascii_lowercase, k=8))
return f"{random_str}@example.com"
def random_name() -> str:
"""Generate a random full name."""
if HAS_FAKER:
return str(fake.name())
first_names = ["John", "Jane", "Bob", "Alice", "Charlie", "Diana", "Eve", "Frank"]
last_names = ["Smith", "Johnson", "Williams", "Brown", "Jones", "Garcia", "Miller", "Davis"]
return f"{random.choice(first_names)} {random.choice(last_names)}"
@command("seed", help="Seed database with sample data")
@click.option("--count", "-c", default=10, type=int, help="Number of records to create")
@click.option("--clear", is_flag=True, help="Clear existing data before seeding")
@click.option("--dry-run", is_flag=True, help="Show what would be created without making changes")
@click.option("--users/--no-users", default=True, help="Seed users (default: True)")
def seed(
count: int,
clear: bool,
dry_run: bool,
users: bool,
) -> None:
"""
Seed the database with sample data for development.
Example:
project cmd seed --count 50
project cmd seed --clear --count 100
project cmd seed --dry-run
project cmd seed --no-users # Skip user seeding
"""
if not HAS_FAKER:
warning(
"Faker not installed. Using basic random data. For better data: uv add faker --group dev"
)
if dry_run:
info(f"[DRY RUN] Would create {count} sample records per entity")
if clear:
info("[DRY RUN] Would clear existing data first")
if users:
info("[DRY RUN] Would create users")
return
from app.db.session import get_db_context
from app.services.user import UserService
from app.schemas.user import UserCreate
async def _seed() -> None:
async with get_db_context() as db:
user_svc = UserService(db)
created_counts = {}
if users:
if clear:
info("Clearing existing users (except admins)...")
await user_svc.delete_non_admins()
if await user_svc.has_any() and not clear:
info("Users already exist. Use --clear to replace them.")
else:
info(f"Creating {count} sample users...")
for _ in range(count):
await user_svc.register(
UserCreate(
email=random_email(),
password="password123",
full_name=random_name(),
)
)
created_counts["users"] = count
if created_counts:
summary = ", ".join(f"{v} {k}" for k, v in created_counts.items())
success(f"Created: {summary}")
else:
info("No records created.")
asyncio.run(_seed())
backend/app/core/__init__.py
+5
View File
@@ -0,0 +1,5 @@
"""Core application configuration and utilities."""
from .config import settings
__all__ = ["settings"]
backend/app/core/config.py
+254
View File
@@ -0,0 +1,254 @@
"""Application configuration using Pydantic BaseSettings."""
# ruff: noqa: I001 - Imports structured for Jinja2 template conditionals
from pathlib import Path
from typing import Literal
from pydantic import computed_field, field_validator, ValidationInfo
from pydantic_settings import BaseSettings, SettingsConfigDict
def find_env_file() -> Path | None:
"""Find .env file in current or parent directories."""
current = Path.cwd()
for path in [current, current.parent]:
env_file = path / ".env"
if env_file.exists():
return env_file
return None
class Settings(BaseSettings):
"""Application settings."""
model_config = SettingsConfigDict(
env_file=find_env_file(),
env_ignore_empty=True,
extra="ignore",
)
# === Project ===
PROJECT_NAME: str = "ai_agent"
API_V1_STR: str = "/api/v1"
DEBUG: bool = False
ENVIRONMENT: Literal["development", "local", "staging", "production"] = "local"
TIMEZONE: str = "UTC" # IANA timezone (e.g. "UTC", "Europe/Warsaw", "America/New_York")
MODELS_CACHE_DIR: Path = Path("./models_cache")
MEDIA_DIR: Path = Path("./media")
MAX_UPLOAD_SIZE_MB: int = 50 # Max file upload size in MB
# Soft per-org storage cap surfaced on /billing — not enforced yet (5 GB).
STORAGE_SOFT_LIMIT_BYTES: int = 5 * 1024 * 1024 * 1024
# === Logfire ===
LOGFIRE_TOKEN: str | None = None
LOGFIRE_SERVICE_NAME: str = "ai_agent"
LOGFIRE_ENVIRONMENT: str = "development"
# === Database (PostgreSQL async) ===
POSTGRES_HOST: str = "localhost"
POSTGRES_PORT: int = 5432
POSTGRES_USER: str = "postgres"
POSTGRES_PASSWORD: str = ""
POSTGRES_DB: str = "ai_agent"
@computed_field # type: ignore[prop-decorator]
@property
def DATABASE_URL(self) -> str:
"""Build async PostgreSQL connection URL."""
return (
f"postgresql+asyncpg://{self.POSTGRES_USER}:{self.POSTGRES_PASSWORD}"
f"@{self.POSTGRES_HOST}:{self.POSTGRES_PORT}/{self.POSTGRES_DB}"
)
@computed_field # type: ignore[prop-decorator]
@property
def DATABASE_URL_SYNC(self) -> str:
"""Build sync PostgreSQL connection URL (for Alembic)."""
return (
f"postgresql://{self.POSTGRES_USER}:{self.POSTGRES_PASSWORD}"
f"@{self.POSTGRES_HOST}:{self.POSTGRES_PORT}/{self.POSTGRES_DB}"
)
# Pool configuration
DB_POOL_SIZE: int = 5
DB_MAX_OVERFLOW: int = 10
DB_POOL_TIMEOUT: int = 30
# === Auth (SECRET_KEY for JWT/Session/Admin) ===
SECRET_KEY: str = "change-me-in-production-use-openssl-rand-hex-32"
@field_validator("SECRET_KEY")
@classmethod
def validate_secret_key(cls, v: str, info: ValidationInfo) -> str:
"""Validate SECRET_KEY is secure in production."""
if len(v) < 32:
raise ValueError("SECRET_KEY must be at least 32 characters long")
# Get environment from values if available
env = info.data.get("ENVIRONMENT", "local") if info.data else "local"
if v == "change-me-in-production-use-openssl-rand-hex-32" and env == "production":
raise ValueError(
"SECRET_KEY must be changed in production! "
"Generate a secure key with: openssl rand -hex 32"
)
return v
# === JWT Settings ===
ACCESS_TOKEN_EXPIRE_MINUTES: int = 30 # 30 minutes
REFRESH_TOKEN_EXPIRE_MINUTES: int = 60 * 24 * 7 # 7 days
ALGORITHM: str = "HS256"
# Public URL of the frontend; used to build OAuth redirect targets and
# Stripe checkout/portal return URLs. Always declared (not gated) because
# the billing model_validator references it unconditionally.
FRONTEND_URL: str = "http://localhost:3033"
# === OAuth2 (Google) ===
GOOGLE_CLIENT_ID: str = ""
GOOGLE_CLIENT_SECRET: str = ""
GOOGLE_REDIRECT_URI: str = "http://localhost:8033/api/v1/oauth/google/callback"
# === Auth (API Key) ===
API_KEY: str = "change-me-in-production"
API_KEY_HEADER: str = "X-API-Key"
@field_validator("API_KEY")
@classmethod
def validate_api_key(cls, v: str, info: ValidationInfo) -> str:
"""Validate API_KEY is set in production."""
env = info.data.get("ENVIRONMENT", "local") if info.data else "local"
if v == "change-me-in-production" and env == "production":
raise ValueError(
"API_KEY must be changed in production! "
"Generate a secure key with: openssl rand -hex 32"
)
return v
# === Redis ===
REDIS_HOST: str = "localhost"
REDIS_PORT: int = 6379
REDIS_PASSWORD: str | None = None
REDIS_DB: int = 0
@computed_field # type: ignore[prop-decorator]
@property
def REDIS_URL(self) -> str:
"""Build Redis connection URL."""
if self.REDIS_PASSWORD:
return f"redis://:{self.REDIS_PASSWORD}@{self.REDIS_HOST}:{self.REDIS_PORT}/{self.REDIS_DB}"
return f"redis://{self.REDIS_HOST}:{self.REDIS_PORT}/{self.REDIS_DB}"
# === Taskiq ===
TASKIQ_BROKER_URL: str = "redis://localhost:6379/1"
TASKIQ_RESULT_BACKEND: str = "redis://localhost:6379/1"
# === AI Agent (pydantic_ai, openai) ===
OPENAI_API_KEY: str = ""
AI_MODEL: str = "gpt-5.5"
AI_TEMPERATURE: float = 0.7
AI_THINKING_ENABLED: bool = False
AI_THINKING_EFFORT: str = "medium" # "low", "medium", "high"
AI_AVAILABLE_MODELS: list[str] = [
"gpt-5.5",
"gpt-5.5-pro",
"gpt-5.4",
"gpt-5.4-pro",
"gpt-5.4-mini",
"gpt-5.4-nano",
"gpt-5-mini",
"gpt-5-nano",
"gpt-5",
"gpt-4.1",
]
AI_FRAMEWORK: str = "pydantic_ai"
LLM_PROVIDER: str = "openai"
# === RAG (Retrieval Augmented Generation) ===
# Vector Database (Milvus)
MILVUS_HOST: str = "localhost"
MILVUS_PORT: int = 19530
MILVUS_DATABASE: str = "default"
MILVUS_TOKEN: str = "root:Milvus"
@computed_field # type: ignore[prop-decorator]
@property
def MILVUS_URI(self) -> str:
"""Build Milvus connection URI."""
return f"http://{self.MILVUS_HOST}:{self.MILVUS_PORT}"
# Embeddings
EMBEDDING_MODEL: str = "text-embedding-3-small"
# Chunking
RAG_CHUNK_SIZE: int = 512
RAG_CHUNK_OVERLAP: int = 50
# Retrieval
RAG_DEFAULT_COLLECTION: str = "documents"
RAG_TOP_K: int = 10
RAG_CHUNKING_STRATEGY: str = "recursive" # recursive, markdown, or fixed
RAG_HYBRID_SEARCH: bool = False # Enable BM25 + vector hybrid search
RAG_ENABLE_OCR: bool = False # OCR fallback for scanned PDFs (requires tesseract)
# Reranker
HF_TOKEN: str = ""
CROSS_ENCODER_MODEL: str = "cross-encoder/ms-marco-MiniLM-L6-v2"
# Document Parser
# Image Description (LLM vision)
RAG_ENABLE_IMAGE_DESCRIPTION: bool = True # set to false to disable LLM image description
RAG_IMAGE_DESCRIPTION_MODEL: str = "" # empty = use AI_MODEL
# Google Drive (optional, for document ingestion via service account)
# S3 (optional, for document ingestion from S3/MinIO)
# === CORS ===
CORS_ORIGINS: list[str] = [
"http://localhost:3033",
"http://localhost:8033",
]
CORS_ALLOW_CREDENTIALS: bool = True
CORS_ALLOW_METHODS: list[str] = ["*"]
CORS_ALLOW_HEADERS: list[str] = ["*"]
@field_validator("CORS_ORIGINS")
@classmethod
def validate_cors_origins(cls, v: list[str], info: ValidationInfo) -> list[str]:
"""Warn if CORS_ORIGINS is too permissive in production."""
env = info.data.get("ENVIRONMENT", "local") if info.data else "local"
if "*" in v and env == "production":
raise ValueError(
"CORS_ORIGINS cannot contain '*' in production! Specify explicit allowed origins."
)
return v
@computed_field # type: ignore[prop-decorator]
@property
def rag(self) -> "RAGSettings":
"""Build RAG-specific settings."""
from app.services.rag.config import RAGSettings, DocumentParser, PdfParser, EmbeddingsConfig
pdf_parser = PdfParser()
return RAGSettings(
collection_name=self.RAG_DEFAULT_COLLECTION,
chunk_size=self.RAG_CHUNK_SIZE,
chunk_overlap=self.RAG_CHUNK_OVERLAP,
chunking_strategy=self.RAG_CHUNKING_STRATEGY,
enable_hybrid_search=self.RAG_HYBRID_SEARCH,
enable_ocr=self.RAG_ENABLE_OCR,
embeddings_config=EmbeddingsConfig(model=self.EMBEDDING_MODEL),
document_parser=DocumentParser(),
pdf_parser=pdf_parser,
enable_image_description=self.RAG_ENABLE_IMAGE_DESCRIPTION,
image_description_model=self.RAG_IMAGE_DESCRIPTION_MODEL,
)
# Rebuild Settings to resolve RAGSettings forward reference
from app.services.rag.config import RAGSettings
Settings.model_rebuild()
settings = Settings()
backend/app/core/csrf.py
+148
View File
@@ -0,0 +1,148 @@
"""CSRF protection middleware for FastAPI.
This module provides CSRF (Cross-Site Request Forgery) protection for
state-changing HTTP methods (POST, PUT, PATCH, DELETE).
The protection works by:
1. Setting a CSRF token in a cookie on initial request
2. Requiring the token to be sent in a header for state-changing requests
3. Comparing the cookie token with the header token
Usage:
Add to your main.py:
from app.core.csrf import CSRFMiddleware
app.add_middleware(CSRFMiddleware)
For endpoints that should be exempt (e.g., login):
@router.post("/login", tags=["csrf-exempt"])
async def login(...):
...
"""
import secrets
from typing import Any, ClassVar
from fastapi import Request, Response
from fastapi.responses import JSONResponse
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from starlette.types import ASGIApp
from app.core.config import settings
class CSRFMiddleware(BaseHTTPMiddleware):
"""CSRF protection middleware.
Protects against Cross-Site Request Forgery attacks by requiring
a token to be present in both a cookie and a header for state-changing requests.
"""
# Methods that require CSRF protection
PROTECTED_METHODS: ClassVar[set[str]] = {"POST", "PUT", "PATCH", "DELETE"}
# Cookie settings
COOKIE_NAME: ClassVar[str] = "csrf_token"
HEADER_NAME: ClassVar[str] = "X-CSRF-Token"
# Paths to exclude from CSRF protection
EXEMPT_PATHS: ClassVar[set[str]] = {
"/api/v1/auth/login",
"/api/v1/auth/register",
"/api/v1/auth/refresh",
"/api/v1/health",
"/api/v1/ready",
"/docs",
"/openapi.json",
"/redoc",
}
def __init__(self, app: ASGIApp, **kwargs: Any) -> None:
super().__init__(app)
self.exempt_paths: set[str] = set(kwargs.get("exempt_paths", self.EXEMPT_PATHS))
self.cookie_name: str = kwargs.get("cookie_name", self.COOKIE_NAME)
self.header_name: str = kwargs.get("header_name", self.HEADER_NAME)
async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
"""Handle the request and apply CSRF protection."""
# Skip for exempt paths
if self._is_exempt(request):
return await call_next(request)
# Get or generate CSRF token
csrf_token = request.cookies.get(self.cookie_name)
if not csrf_token:
csrf_token = self._generate_token()
# Check CSRF for protected methods
if request.method in self.PROTECTED_METHODS:
header_token = request.headers.get(self.header_name)
if not header_token:
return JSONResponse(
status_code=403,
content={
"detail": "CSRF token missing",
"message": f"Include the '{self.header_name}' header with the CSRF token",
},
)
if not secrets.compare_digest(csrf_token, header_token):
return JSONResponse(
status_code=403,
content={
"detail": "CSRF token invalid",
"message": "The CSRF token does not match",
},
)
# Process the request
response = await call_next(request)
# Set CSRF token cookie if not present
if not request.cookies.get(self.cookie_name):
response.set_cookie(
key=self.cookie_name,
value=csrf_token,
httponly=False, # JavaScript needs to read this
secure=not settings.DEBUG,
samesite="lax",
max_age=3600 * 24, # 24 hours
)
return response
def _is_exempt(self, request: Request) -> bool:
"""Check if the request path is exempt from CSRF protection."""
path = request.url.path
# Check exact path matches
if path in self.exempt_paths:
return True
# Check path prefixes
for exempt in self.exempt_paths:
if path.startswith(exempt):
return True
# Check if endpoint has "csrf-exempt" tag
route = request.scope.get("route")
return bool(route and hasattr(route, "tags") and "csrf-exempt" in route.tags)
@staticmethod
def _generate_token() -> str:
"""Generate a secure CSRF token."""
return secrets.token_urlsafe(32)
def get_csrf_token(request: Request) -> str:
"""Get the current CSRF token from cookies or generate a new one.
Use this in templates or API responses to provide the token to clients.
"""
token = request.cookies.get(CSRFMiddleware.COOKIE_NAME)
if not token:
token = secrets.token_urlsafe(32)
return token
backend/app/core/exceptions.py
+130
View File
@@ -0,0 +1,130 @@
"""Application exceptions.
Domain exceptions with HTTP status codes for the hybrid approach.
These exceptions are caught by exception handlers and converted to proper HTTP responses.
"""
from typing import Any
class AppException(Exception):
"""Base exception for all application errors.
Attributes:
message: Human-readable error message.
code: Machine-readable error code for clients.
status_code: HTTP status code to return.
details: Additional error details (e.g., field names, IDs).
"""
message: str = "An error occurred"
code: str = "APP_ERROR"
status_code: int = 500
def __init__(
self,
message: str | None = None,
code: str | None = None,
details: dict[str, Any] | None = None,
):
self.message = message or self.__class__.message
self.code = code or self.__class__.code
self.details = details or {}
super().__init__(self.message)
def __repr__(self) -> str:
return f"{self.__class__.__name__}(message={self.message!r}, code={self.code!r})"
# === 4xx Client Errors ===
class NotFoundError(AppException):
"""Resource not found (404)."""
message = "Resource not found"
code = "NOT_FOUND"
status_code = 404
class AlreadyExistsError(AppException):
"""Resource already exists (409)."""
message = "Resource already exists"
code = "ALREADY_EXISTS"
status_code = 409
class ValidationError(AppException):
"""Validation error (422)."""
message = "Validation error"
code = "VALIDATION_ERROR"
status_code = 422
class AuthenticationError(AppException):
"""Authentication failed (401)."""
message = "Authentication failed"
code = "AUTHENTICATION_ERROR"
status_code = 401
class AuthorizationError(AppException):
"""Authorization failed - insufficient permissions (403)."""
message = "Insufficient permissions"
code = "AUTHORIZATION_ERROR"
status_code = 403
class RateLimitError(AppException):
"""Rate limit exceeded (429)."""
message = "Rate limit exceeded"
code = "RATE_LIMIT_EXCEEDED"
status_code = 429
class BadRequestError(AppException):
"""Bad request (400)."""
message = "Bad request"
code = "BAD_REQUEST"
status_code = 400
class PaymentRequiredError(AppException):
"""Payment required — seat or usage limit reached (402)."""
message = "Payment required"
code = "PAYMENT_REQUIRED"
status_code = 402
# === 5xx Server Errors ===
class ExternalServiceError(AppException):
"""External service unavailable (503)."""
message = "External service unavailable"
code = "EXTERNAL_SERVICE_ERROR"
status_code = 503
class DatabaseError(AppException):
"""Database error (500)."""
message = "Database error"
code = "DATABASE_ERROR"
status_code = 500
class InternalError(AppException):
"""Internal server error (500)."""
message = "Internal server error"
code = "INTERNAL_ERROR"
status_code = 500
backend/app/core/logfire_setup.py
+32
View File
@@ -0,0 +1,32 @@
"""Logfire observability configuration."""
from typing import Any
import logfire
from app.core.config import settings
def setup_logfire() -> None:
"""Configure Logfire instrumentation."""
logfire.configure(
token=settings.LOGFIRE_TOKEN,
service_name=settings.LOGFIRE_SERVICE_NAME,
environment=settings.LOGFIRE_ENVIRONMENT,
send_to_logfire="if-token-present",
)
def instrument_app(app: Any) -> None:
"""Instrument FastAPI app with Logfire."""
logfire.instrument_fastapi(app)
def instrument_asyncpg() -> None:
"""Instrument asyncpg for PostgreSQL."""
logfire.instrument_asyncpg()
def instrument_pydantic_ai() -> None:
"""Instrument PydanticAI for AI agent observability."""
logfire.instrument_pydantic_ai()
backend/app/core/logging.py
+78
View File
@@ -0,0 +1,78 @@
"""Logging utilities — PII redaction filter for GDPR/compliance safety."""
import logging
import re
from typing import ClassVar
class PiiRedactionFilter(logging.Filter):
"""Logging filter that redacts personally identifiable information.
Automatically scrubs email addresses, JWT tokens, API keys, bearer tokens,
and password-like values from log messages to prevent PII leaks to
log aggregators (Datadog, CloudWatch, Logfire, etc.).
Usage:
logging.getLogger().addFilter(PiiRedactionFilter())
"""
PATTERNS: ClassVar[list[tuple[re.Pattern[str], str]]] = [
# Email addresses
(re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"), "[EMAIL_REDACTED]"),
# JWT tokens (header.payload.signature)
(
re.compile(r"eyJ[A-Za-z0-9_-]{10,}\.eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]+"),
"[JWT_REDACTED]",
),
# OpenAI API keys
(re.compile(r"sk-[a-zA-Z0-9]{20,}"), "[API_KEY_REDACTED]"),
# Anthropic API keys
(re.compile(r"sk-ant-[a-zA-Z0-9_-]{20,}"), "[API_KEY_REDACTED]"),
# Generic long hex/base64 secrets (40+ chars, likely tokens)
(
re.compile(
r"(?:token|key|secret|password|authorization)[=: ]+['\"]?([A-Za-z0-9_/+=.-]{40,})",
re.IGNORECASE,
),
"[SECRET_REDACTED]",
),
# Bearer tokens in headers
(re.compile(r"Bearer\s+[A-Za-z0-9._~+/=-]{10,}"), "Bearer [TOKEN_REDACTED]"),
# Password/secret in key=value or key: value patterns
(
re.compile(
r"(password|passwd|pwd|secret_key|api_key|apikey|auth_token|access_token|refresh_token)"
r"[\s]*[=:]\s*['\"]?\S+['\"]?",
re.IGNORECASE,
),
r"\1=[REDACTED]",
),
]
def filter(self, record: logging.LogRecord) -> bool:
"""Redact PII from log record message and args."""
if isinstance(record.msg, str):
record.msg = self._redact(record.msg)
if record.args:
if isinstance(record.args, dict):
record.args = {
k: self._redact(v) if isinstance(v, str) else v for k, v in record.args.items()
}
elif isinstance(record.args, tuple):
record.args = tuple(
self._redact(a) if isinstance(a, str) else a for a in record.args
)
return True
def _redact(self, value: str) -> str:
for pattern, replacement in self.PATTERNS:
value = pattern.sub(replacement, value)
return value
def setup_logging() -> None:
"""Configure root logger with PII redaction filter."""
root_logger = logging.getLogger()
# Avoid adding duplicate filters
if not any(isinstance(f, PiiRedactionFilter) for f in root_logger.filters):
root_logger.addFilter(PiiRedactionFilter())
backend/app/core/middleware.py
+102
View File
@@ -0,0 +1,102 @@
"""Application middleware."""
from typing import ClassVar
from uuid import uuid4
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from starlette.requests import Request
from starlette.responses import Response
from starlette.types import ASGIApp
class RequestIDMiddleware(BaseHTTPMiddleware):
"""Middleware that adds a unique request ID to each request.
The request ID is taken from the X-Request-ID header if present,
otherwise a new UUID is generated. The ID is added to the response
headers and is available in request.state.request_id.
"""
async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
"""Add request ID to request state and response headers."""
request_id = request.headers.get("X-Request-ID", str(uuid4()))
request.state.request_id = request_id
response = await call_next(request)
response.headers["X-Request-ID"] = request_id
return response
class SecurityHeadersMiddleware(BaseHTTPMiddleware):
"""Middleware that adds security headers to all responses.
This includes:
- Content-Security-Policy (CSP)
- X-Content-Type-Options
- X-Frame-Options
- X-XSS-Protection
- Referrer-Policy
- Permissions-Policy
Usage:
app.add_middleware(SecurityHeadersMiddleware)
# Or with custom CSP:
app.add_middleware(
SecurityHeadersMiddleware,
csp_directives={
"default-src": "'self'",
"script-src": "'self' 'unsafe-inline'",
}
)
"""
DEFAULT_CSP_DIRECTIVES: ClassVar[dict[str, str]] = {
"default-src": "'self'",
"script-src": "'self'",
"style-src": "'self' 'unsafe-inline'", # Allow inline styles for some UI libs
"img-src": "'self' data: https:",
"font-src": "'self' data:",
"connect-src": "'self'",
"frame-ancestors": "'none'",
"base-uri": "'self'",
"form-action": "'self'",
}
def __init__(
self,
app: ASGIApp,
csp_directives: dict[str, str] | None = None,
exclude_paths: set[str] | None = None,
) -> None:
super().__init__(app)
self.csp_directives = csp_directives or self.DEFAULT_CSP_DIRECTIVES
self.exclude_paths = exclude_paths or {"/docs", "/redoc", "/openapi.json"}
async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
"""Add security headers to the response."""
response = await call_next(request)
# Skip for docs/openapi endpoints which need different CSP
if request.url.path in self.exclude_paths:
return response
# Build CSP header
csp_value = "; ".join(
f"{directive} {value}" for directive, value in self.csp_directives.items()
)
# Add security headers — respect any already set by the route, so an
# endpoint can opt into less restrictive framing (e.g. user-content
# files served inline for the chat preview panel).
response.headers.setdefault("Content-Security-Policy", csp_value)
response.headers.setdefault("X-Content-Type-Options", "nosniff")
response.headers.setdefault("X-Frame-Options", "DENY")
response.headers.setdefault("X-XSS-Protection", "1; mode=block")
response.headers.setdefault("Referrer-Policy", "strict-origin-when-cross-origin")
response.headers["Permissions-Policy"] = (
"accelerometer=(), camera=(), geolocation=(), gyroscope=(), "
"magnetometer=(), microphone=(), payment=(), usb=()"
)
return response
backend/app/core/oauth.py
+16
View File
@@ -0,0 +1,16 @@
"""OAuth2 client configuration."""
from authlib.integrations.starlette_client import OAuth
from app.core.config import settings
oauth = OAuth()
# Configure Google OAuth2
oauth.register(
name="google",
client_id=settings.GOOGLE_CLIENT_ID,
client_secret=settings.GOOGLE_CLIENT_SECRET,
server_metadata_url="https://accounts.google.com/.well-known/openid-configuration",
client_kwargs={"scope": "openid email profile"},
)
backend/app/core/sanitize.py
+433
View File
@@ -0,0 +1,433 @@
"""Input sanitization utilities.
This module provides security-focused input sanitization functions:
- HTML sanitization to prevent XSS attacks
- Path traversal prevention for file operations
- Webhook URL validation to prevent SSRF attacks
- Common input cleaning utilities
Note: SQL injection is prevented by using SQLAlchemy ORM with parameterized queries.
"""
import html
import ipaddress
import os
import re
import socket
import unicodedata
from pathlib import Path
from typing import TypeVar
from urllib.parse import urlparse
# Default allowed HTML tags for rich text content
DEFAULT_ALLOWED_TAGS = frozenset(
{
"a",
"abbr",
"acronym",
"b",
"blockquote",
"br",
"code",
"em",
"i",
"li",
"ol",
"p",
"pre",
"strong",
"ul",
}
)
# Default allowed HTML attributes
DEFAULT_ALLOWED_ATTRIBUTES = {
"a": frozenset({"href", "title", "rel"}),
"abbr": frozenset({"title"}),
"acronym": frozenset({"title"}),
}
# Allowed URL schemes for webhook URLs
WEBHOOK_ALLOWED_SCHEMES = frozenset({"http", "https"})
# Shared Address Space (RFC 6598) — CGNAT range.
# Python 3.11+ no longer classifies 100.64.0.0/10 as private or reserved,
# so we block it explicitly. Covers cloud metadata endpoints like
# Alibaba Cloud's 100.100.100.200.
_CGNAT_NETWORK = ipaddress.ip_network("100.64.0.0/10")
class SSRFBlockedError(ValueError):
"""Raised when a URL is blocked by SSRF protection.
Dedicated exception type to avoid fragile string matching when
distinguishing SSRF blocks from other ValueErrors.
"""
def sanitize_html(
content: str,
allowed_tags: frozenset[str] | None = None,
strip: bool = True,
) -> str:
"""Sanitize HTML content to prevent XSS attacks.
This is a simple implementation that escapes all HTML.
For rich text support, consider using the `bleach` library.
Args:
content: The HTML content to sanitize.
allowed_tags: Not used in simple mode (for bleach compatibility).
strip: Not used in simple mode (for bleach compatibility).
Returns:
Escaped HTML-safe string.
Example:
>>> sanitize_html("<script>alert('xss')</script>")
"&lt;script&gt;alert('xss')&lt;/script&gt;"
"""
if not content:
return ""
return html.escape(content)
def sanitize_filename(filename: str, allow_unicode: bool = False) -> str:
"""Sanitize a filename to prevent path traversal and unsafe characters.
Args:
filename: The filename to sanitize.
allow_unicode: Whether to allow unicode characters.
Returns:
A safe filename string.
Example:
>>> sanitize_filename("../../../etc/passwd")
"etc_passwd"
>>> sanitize_filename("hello world.txt")
"hello_world.txt"
"""
if not filename:
return ""
# Normalize unicode
if allow_unicode:
filename = unicodedata.normalize("NFKC", filename)
else:
filename = unicodedata.normalize("NFKD", filename).encode("ascii", "ignore").decode("ascii")
# Get just the filename (remove any path components)
filename = os.path.basename(filename)
# Remove null bytes
filename = filename.replace("\x00", "")
# Replace path separators and special characters
filename = re.sub(r"[/\\:*?\"<>|]", "_", filename)
# Replace multiple underscores/spaces with single underscore
filename = re.sub(r"[\s_]+", "_", filename)
# Remove leading/trailing underscores and dots
filename = filename.strip("._")
# Ensure we have a valid filename
if not filename:
return "unnamed"
return filename
def validate_safe_path(
base_dir: Path | str,
user_path: str,
) -> Path:
"""Validate that a user-provided path is within the allowed base directory.
Prevents path traversal attacks by ensuring the resolved path
is within the expected directory.
Args:
base_dir: The base directory that all paths must be within.
user_path: The user-provided path to validate.
Returns:
The resolved, safe path.
Raises:
ValueError: If the path would escape the base directory.
Example:
>>> validate_safe_path("/uploads", "../../../etc/passwd")
Raises ValueError
>>> validate_safe_path("/uploads", "images/photo.jpg")
Path("/uploads/images/photo.jpg")
"""
base_path = Path(base_dir).resolve()
user_path_sanitized = sanitize_filename(user_path.lstrip("/\\"))
# Resolve the full path
full_path = (base_path / user_path_sanitized).resolve()
# Check if the resolved path is within the base directory
try:
full_path.relative_to(base_path)
except ValueError as err:
raise ValueError(
f"Path traversal detected: {user_path!r} would escape {base_dir!r}"
) from err
return full_path
def _is_ip_blocked(ip_str: str) -> bool:
"""Check if an IP address is private, reserved, loopback, or link-local.
Args:
ip_str: The IP address string to check.
Returns:
True if the address should be blocked, False if it's safe.
"""
try:
addr = ipaddress.ip_address(ip_str)
except ValueError:
# If we can't parse it, block it to be safe
return True
return (
addr.is_private
or addr.is_reserved
or addr.is_loopback
or addr.is_link_local
or addr.is_multicast
or addr.is_unspecified
or addr in _CGNAT_NETWORK
)
def validate_webhook_url(
url: str,
allowed_schemes: frozenset[str] | None = None,
) -> str:
"""Validate a webhook URL to prevent SSRF attacks.
Checks that the URL:
- Uses an allowed scheme (http/https only by default)
- Does not contain userinfo (credentials in the URL)
- Does not point to private, reserved, loopback, or link-local IP addresses
- Resolves via DNS to a public IP (prevents DNS rebinding attacks)
Args:
url: The webhook URL to validate.
allowed_schemes: Allowed URL schemes. Defaults to {"http", "https"}.
Returns:
The validated URL string.
Raises:
SSRFBlockedError: If the URL is blocked by SSRF protection.
ValueError: If the URL is malformed.
Example:
>>> validate_webhook_url("https://example.com/webhook")
"https://example.com/webhook"
>>> validate_webhook_url("http://169.254.169.254/latest/meta-data/")
Raises SSRFBlockedError
"""
if allowed_schemes is None:
allowed_schemes = WEBHOOK_ALLOWED_SCHEMES
# Parse the URL
try:
parsed = urlparse(url)
except Exception as err:
raise ValueError(f"Invalid webhook URL: {url!r}") from err
# Validate scheme
if parsed.scheme not in allowed_schemes:
raise SSRFBlockedError(
f"URL scheme {parsed.scheme!r} is not allowed. "
f"Allowed schemes: {', '.join(sorted(allowed_schemes))}"
)
# Extract hostname
hostname = parsed.hostname
if not hostname:
raise ValueError(f"Invalid webhook URL: no hostname found in {url!r}")
# Reject URLs with userinfo (credentials) to prevent URL parsing ambiguities
# e.g. http://user:pass@host/ or http://foo@169.254.169.254%00@public.com/
if parsed.username is not None or parsed.password is not None:
raise SSRFBlockedError(
"Webhook URL must not contain credentials (userinfo). "
"Remove the user:password@ portion from the URL."
)
# Try to parse hostname directly as an IP address
try:
addr = ipaddress.ip_address(hostname)
if _is_ip_blocked(str(addr)):
raise SSRFBlockedError(
f"Webhook URL blocked: {hostname!r} resolves to a private/internal "
f"address. SSRF protection does not allow requests to internal networks."
)
return url
except SSRFBlockedError:
raise
except ValueError:
# Not an IP literal — continue to DNS resolution below
pass
# Determine the correct default port based on the scheme
default_port = 443 if parsed.scheme == "https" else 80
port = parsed.port or default_port
# Resolve hostname via DNS and check all returned addresses
# TODO: socket.getaddrinfo() is blocking I/O — in async code paths
# (PostgreSQL, MongoDB) consider using loop.getaddrinfo() or run_in_executor.
try:
addr_infos = socket.getaddrinfo(hostname, port, proto=socket.IPPROTO_TCP)
except socket.gaierror as err:
raise SSRFBlockedError(
f"Webhook URL blocked: unable to resolve hostname {hostname!r}"
) from err
if not addr_infos:
raise SSRFBlockedError(
f"Webhook URL blocked: hostname {hostname!r} did not resolve to any address"
)
for _family, _type, _proto, _canonname, sockaddr in addr_infos:
ip_str = str(sockaddr[0])
if _is_ip_blocked(ip_str):
raise SSRFBlockedError(
f"Webhook URL blocked: {hostname!r} resolves to private/internal "
f"address {ip_str!r}. SSRF protection does not allow requests to "
f"internal networks."
)
return url
def sanitize_string(
value: str,
max_length: int | None = None,
allow_newlines: bool = True,
strip_whitespace: bool = True,
) -> str:
"""Sanitize a string input with various options.
Args:
value: The string to sanitize.
max_length: Maximum allowed length (truncates if exceeded).
allow_newlines: Whether to preserve newlines.
strip_whitespace: Whether to strip leading/trailing whitespace.
Returns:
Sanitized string.
"""
if not value:
return ""
# Strip null bytes and other control characters (except newlines if allowed)
if allow_newlines:
value = re.sub(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]", "", value)
else:
value = re.sub(r"[\x00-\x1f\x7f]", "", value)
# Strip whitespace if requested
if strip_whitespace:
value = value.strip()
# Truncate if needed
if max_length is not None and len(value) > max_length:
value = value[:max_length]
return value
def sanitize_email(email: str) -> str:
"""Basic email sanitization.
Note: For proper email validation, use Pydantic's EmailStr type.
This function only performs basic cleaning.
Args:
email: The email address to sanitize.
Returns:
Lowercased, stripped email.
"""
if not email:
return ""
return email.strip().lower()
T = TypeVar("T", int, float)
def sanitize_numeric(
value: str | int | float,
value_type: type[T],
min_value: T | None = None,
max_value: T | None = None,
default: T | None = None,
) -> T | None:
"""Sanitize and validate a numeric value.
Args:
value: The value to sanitize (can be string or numeric).
value_type: The expected type (int or float).
min_value: Minimum allowed value.
max_value: Maximum allowed value.
default: Default value if conversion fails.
Returns:
The sanitized numeric value, or default if invalid.
Example:
>>> sanitize_numeric("100", int, min_value=0, max_value=1000)
100
>>> sanitize_numeric("abc", int, default=0)
0
"""
try:
result = value_type(value)
if min_value is not None and result < min_value:
result = min_value
if max_value is not None and result > max_value:
result = max_value
return result
except (ValueError, TypeError):
return default
def escape_sql_like(pattern: str, escape_char: str = "\\") -> str:
"""Escape special characters in a LIKE pattern.
Use this when building LIKE queries with user input.
Args:
pattern: The pattern to escape.
escape_char: The escape character to use.
Returns:
Escaped pattern safe for use in LIKE queries.
Example:
>>> escape_sql_like("100%")
"100\\%"
>>> escape_sql_like("under_score")
"under\\_score"
"""
# Escape the escape character first, then special chars
pattern = pattern.replace(escape_char, escape_char + escape_char)
pattern = pattern.replace("%", escape_char + "%")
pattern = pattern.replace("_", escape_char + "_")
return pattern
backend/app/core/security.py
+103
View File
@@ -0,0 +1,103 @@
"""Security utilities for JWT authentication."""
from datetime import UTC, datetime, timedelta
from typing import Any
import bcrypt
import jwt
from app.core.config import settings
def create_access_token(
subject: str | Any,
expires_delta: timedelta | None = None,
) -> str:
"""Create a JWT access token."""
if expires_delta:
expire = datetime.now(UTC) + expires_delta
else:
expire = datetime.now(UTC) + timedelta(minutes=settings.ACCESS_TOKEN_EXPIRE_MINUTES)
to_encode = {"exp": expire, "sub": str(subject), "type": "access"}
return jwt.encode(to_encode, settings.SECRET_KEY, algorithm=settings.ALGORITHM)
def create_refresh_token(
subject: str | Any,
expires_delta: timedelta | None = None,
) -> str:
"""Create a JWT refresh token."""
if expires_delta:
expire = datetime.now(UTC) + expires_delta
else:
expire = datetime.now(UTC) + timedelta(minutes=settings.REFRESH_TOKEN_EXPIRE_MINUTES)
to_encode = {"exp": expire, "sub": str(subject), "type": "refresh"}
return jwt.encode(to_encode, settings.SECRET_KEY, algorithm=settings.ALGORITHM)
def verify_token(token: str) -> dict[str, Any] | None:
"""Verify a JWT token and return payload."""
try:
payload = jwt.decode(
token,
settings.SECRET_KEY,
algorithms=[settings.ALGORITHM],
)
return payload
except jwt.PyJWTError:
return None
def create_password_reset_token(
subject: str | Any,
expires_delta: timedelta | None = None,
) -> str:
"""Single-use JWT for password reset.
Short-lived (1h default). The `type` claim distinguishes it from access /
refresh / magic-link tokens — a stolen reset token can't be used as an
access token.
"""
expire = datetime.now(UTC) + (expires_delta or timedelta(hours=1))
to_encode = {"exp": expire, "sub": str(subject), "type": "password_reset"}
return jwt.encode(to_encode, settings.SECRET_KEY, algorithm=settings.ALGORITHM)
def create_magic_link_token(
subject: str | Any,
expires_delta: timedelta | None = None,
) -> str:
"""Sign-in-by-email JWT. Short-lived (15 min default)."""
expire = datetime.now(UTC) + (expires_delta or timedelta(minutes=15))
to_encode = {"exp": expire, "sub": str(subject), "type": "magic_link"}
return jwt.encode(to_encode, settings.SECRET_KEY, algorithm=settings.ALGORITHM)
def verify_special_token(token: str, expected_type: str) -> dict[str, Any] | None:
"""Verify a non-access JWT (password_reset, magic_link) and require a
specific `type` claim. Returns payload on success, None otherwise.
"""
payload = verify_token(token)
if payload is None:
return None
if payload.get("type") != expected_type:
return None
return payload
def verify_password(plain_password: str, hashed_password: str) -> bool:
"""Verify a password against a hash."""
return bcrypt.checkpw(
plain_password.encode("utf-8"),
hashed_password.encode("utf-8"),
)
def get_password_hash(password: str) -> str:
"""Hash a password."""
return bcrypt.hashpw(
password.encode("utf-8"),
bcrypt.gensalt(),
).decode("utf-8")
backend/app/db/__init__.py
+5
View File
@@ -0,0 +1,5 @@
"""Database module."""
from app.db.base import Base
__all__ = ["Base"]
backend/app/db/base.py
+37
View File
@@ -0,0 +1,37 @@
"""SQLAlchemy base model."""
from datetime import datetime
from sqlalchemy import DateTime, MetaData, func
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
# Naming convention for database constraints and indexes
# This ensures consistent naming across all migrations
NAMING_CONVENTION = {
"ix": "%(column_0_label)s_idx",
"uq": "%(table_name)s_%(column_0_name)s_key",
"ck": "%(table_name)s_%(constraint_name)s_check",
"fk": "%(table_name)s_%(column_0_name)s_fkey",
"pk": "%(table_name)s_pkey",
}
class Base(DeclarativeBase):
"""Base class for all SQLAlchemy models."""
metadata = MetaData(naming_convention=NAMING_CONVENTION)
class TimestampMixin:
"""Mixin for created_at and updated_at timestamps."""
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
server_default=func.now(),
nullable=False,
)
updated_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True),
onupdate=func.now(),
nullable=True,
)
backend/app/db/models/__init__.py
+26
View File
@@ -0,0 +1,26 @@
"""Database models."""
# ruff: noqa: I001, RUF022 - Imports structured for Jinja2 template conditionals
from app.db.models.user import User
from app.db.models.conversation import Conversation, Message, ToolCall
from app.db.models.chat_file import ChatFile
from app.db.models.message_rating import MessageRating
from app.db.models.rag_document import RAGDocument
from app.db.models.sync_log import SyncLog
from app.db.models.sync_source import SyncSource
from app.db.models.conversation_share import ConversationShare
from app.db.models.user_slash_command import UserSlashCommand
__all__ = [
"User",
"Conversation",
"Message",
"ToolCall",
"ChatFile",
"MessageRating",
"RAGDocument",
"SyncLog",
"SyncSource",
"ConversationShare",
"UserSlashCommand",
]
backend/app/db/models/chat_file.py
+37
View File
@@ -0,0 +1,37 @@
"""ChatFile database model - stores metadata for files uploaded in chat."""
import uuid
from sqlalchemy import ForeignKey, Integer, String, Text
from sqlalchemy.dialects.postgresql import UUID as PG_UUID
from sqlalchemy.orm import Mapped, mapped_column
from app.db.base import Base, TimestampMixin
class ChatFile(Base, TimestampMixin):
"""Tracks files uploaded by users in chat conversations."""
__tablename__ = "chat_files"
id: Mapped[uuid.UUID] = mapped_column(
PG_UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
)
user_id: Mapped[uuid.UUID] = mapped_column(
PG_UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
index=True,
)
message_id: Mapped[uuid.UUID | None] = mapped_column(
PG_UUID(as_uuid=True), ForeignKey("messages.id", ondelete="CASCADE"), nullable=True
)
filename: Mapped[str] = mapped_column(String(255), nullable=False)
mime_type: Mapped[str] = mapped_column(String(100), nullable=False)
size: Mapped[int] = mapped_column(Integer, nullable=False)
storage_path: Mapped[str] = mapped_column(String(500), nullable=False)
file_type: Mapped[str] = mapped_column(String(20), nullable=False)
parsed_content: Mapped[str | None] = mapped_column(Text, nullable=True)
def __repr__(self) -> str:
return f"<ChatFile(id={self.id}, filename={self.filename}, type={self.file_type})>"
backend/app/db/models/conversation.py
+138
View File
@@ -0,0 +1,138 @@
"""Conversation and message models for AI chat persistence."""
import uuid
from datetime import datetime
from typing import TYPE_CHECKING
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, String, Text
from sqlalchemy.dialects.postgresql import JSONB, UUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.db.base import Base, TimestampMixin
if TYPE_CHECKING:
from app.db.models.chat_file import ChatFile
class Conversation(Base, TimestampMixin):
"""Conversation model - groups messages in a chat session.
Attributes:
id: Unique conversation identifier
user_id: Optional user who owns this conversation (if auth enabled)
project_id: Optional project this conversation belongs to (if pydantic_deep)
title: Auto-generated or user-defined title
is_archived: Whether the conversation is archived
messages: List of messages in this conversation
"""
__tablename__ = "conversations"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
user_id: Mapped[uuid.UUID | None] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=True,
index=True,
)
title: Mapped[str | None] = mapped_column(String(255), nullable=True)
is_archived: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
# Relationships
messages: Mapped[list["Message"]] = relationship(
"Message",
back_populates="conversation",
cascade="all, delete-orphan",
order_by="Message.created_at",
)
def __repr__(self) -> str:
return f"<Conversation(id={self.id}, title={self.title})>"
class Message(Base, TimestampMixin):
"""Message model - individual message in a conversation.
Attributes:
id: Unique message identifier
conversation_id: The conversation this message belongs to
role: Message role (user, assistant, system)
content: Message text content
model_name: AI model used (for assistant messages)
tokens_used: Token count for this message
tool_calls: List of tool calls made in this message
"""
__tablename__ = "messages"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
conversation_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("conversations.id", ondelete="CASCADE"),
nullable=False,
index=True,
)
role: Mapped[str] = mapped_column(String(20), nullable=False) # user, assistant, system
content: Mapped[str] = mapped_column(Text, nullable=False)
model_name: Mapped[str | None] = mapped_column(String(100), nullable=True)
tokens_used: Mapped[int | None] = mapped_column(Integer, nullable=True)
# Relationships
conversation: Mapped["Conversation"] = relationship("Conversation", back_populates="messages")
tool_calls: Mapped[list["ToolCall"]] = relationship(
"ToolCall",
back_populates="message",
cascade="all, delete-orphan",
order_by="ToolCall.started_at",
)
files: Mapped[list["ChatFile"]] = relationship(
"ChatFile",
foreign_keys="ChatFile.message_id",
cascade="all, delete-orphan",
)
def __repr__(self) -> str:
return f"<Message(id={self.id}, role={self.role})>"
class ToolCall(Base):
"""ToolCall model - record of a tool invocation.
Attributes:
id: Unique tool call identifier
message_id: The assistant message that triggered this call
tool_call_id: External ID from PydanticAI
tool_name: Name of the tool that was called
args: JSON arguments passed to the tool
result: Result returned by the tool
status: Current status (pending, running, completed, failed)
started_at: When the tool call started
completed_at: When the tool call completed
duration_ms: Execution time in milliseconds
"""
__tablename__ = "tool_calls"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
message_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("messages.id", ondelete="CASCADE"),
nullable=False,
index=True,
)
tool_call_id: Mapped[str] = mapped_column(String(100), nullable=False)
tool_name: Mapped[str] = mapped_column(String(100), nullable=False)
args: Mapped[dict[str, object]] = mapped_column(JSONB, nullable=False, default=dict)
result: Mapped[str | None] = mapped_column(Text, nullable=True)
status: Mapped[str] = mapped_column(
String(20), nullable=False, default="pending"
) # pending, running, completed, failed
started_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
completed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
duration_ms: Mapped[int | None] = mapped_column(Integer, nullable=True)
# Relationships
message: Mapped["Message"] = relationship("Message", back_populates="tool_calls")
def __repr__(self) -> str:
return f"<ToolCall(id={self.id}, tool_name={self.tool_name}, status={self.status})>"
backend/app/db/models/conversation_share.py
+49
View File
@@ -0,0 +1,49 @@
"""ConversationShare model — sharing conversations between users (PostgreSQL async)."""
import uuid
from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, String, UniqueConstraint
from sqlalchemy.dialects.postgresql import UUID
from sqlalchemy.orm import Mapped, mapped_column
from sqlalchemy.sql import func
from app.db.base import Base
class ConversationShare(Base):
"""Share record granting a user access to another user's conversation."""
__tablename__ = "conversation_shares"
__table_args__ = (
UniqueConstraint("conversation_id", "shared_with", name="uq_share_conv_user"),
)
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
conversation_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("conversations.id", ondelete="CASCADE"),
nullable=False,
index=True,
)
shared_by: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
)
shared_with: Mapped[uuid.UUID | None] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=True,
index=True,
)
share_token: Mapped[str | None] = mapped_column(String(64), nullable=True, unique=True)
permission: Mapped[str] = mapped_column(
String(10), nullable=False, default="view"
) # view | edit
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), nullable=False
)
def __repr__(self) -> str:
return f"<ConversationShare(id={self.id}, conv={self.conversation_id}, with={self.shared_with})>"
backend/app/db/models/message_rating.py
+56
View File
@@ -0,0 +1,56 @@
"""Message rating model for user feedback on AI responses.
This module is only imported when JWT auth is enabled (see
`app/db/models/__init__.py` and `alembic/env.py`).
"""
import uuid
from typing import TYPE_CHECKING
from sqlalchemy import CheckConstraint, ForeignKey, Integer, Text, UniqueConstraint
from sqlalchemy.dialects.postgresql import UUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.db.base import Base, TimestampMixin
if TYPE_CHECKING:
from app.db.models.conversation import Message
from app.db.models.user import User
class MessageRating(Base, TimestampMixin):
"""User rating for AI assistant messages."""
__tablename__ = "message_ratings"
__table_args__ = (
UniqueConstraint("message_id", "user_id", name="uq_message_user_rating"),
CheckConstraint("rating IN (1, -1)", name="ck_rating_value"),
)
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
message_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("messages.id", ondelete="CASCADE"),
nullable=False,
index=True,
)
user_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
index=True,
)
rating: Mapped[int] = mapped_column(Integer, nullable=False) # 1 or -1
comment: Mapped[str | None] = mapped_column(Text, nullable=True)
message: Mapped["Message"] = relationship(
"Message",
foreign_keys="MessageRating.message_id",
)
user: Mapped["User"] = relationship(
"User",
foreign_keys="MessageRating.user_id",
)
def __repr__(self) -> str:
return f"<MessageRating(id={self.id}, rating={self.rating})>"

Some files were not shown because too many files have changed in this diff Show More