feat: add documentation for agent implementation and development commands

CLAUDE.md

@@ -0,0 +1,58 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Development Commands
+
+### Backend (Python / FastAPI)
+- **Install dependencies**: `make install` (runs `uv sync`)
+- **Run backend (dev)**: `make run` (runs `uv run python -m backend.main` with reload)
+- **Setup environment**: `make setup` (copies `.env.example` to `.env` and runs the app)
+
+### Frontend (React / Vite / Bun)
+- **Install dependencies**: `make frontend-install` (runs `bun install`)
+- **Run dev server**: `make frontend-dev` (runs `bun run dev`)
+- **Build production**: `make frontend-build` (runs `bun run build`)
+
+### Orchestration & Containers
+- **Start all services**: `make compose-up` (Docker/Podman compose)
+- **Follow logs**: `make compose-logs`
+- **Stop all services**: `make compose-down`
+- **Force rebuild images**: `make compose-rebuild`
+
+### Maintenance
+- **Clean artifacts**: `make clean`
+
+## Architecture & Structure
+
+### Big Picture Overview
+Agent Alpha is a full-stack agentic AI application consisting of a FastAPI backend, a React frontend, and a RAG (Retrieval-Augmented Generation) pipeline. It utilizes **pydantic-ai** for agent logic and **uv** for Python dependency management.
+
+### Backend Architecture (`/backend`)
+The backend follows a layered architecture:
+- **Core (`/backend/core`)**: The heart of the application. Contains configuration (`config.py`), database engine initialization (`database.py`), ORM models (`models.py`), and the primary agent lifecycle/inference logic (`agent.py`).
+- **Database & Repositories**: 
+    - **Models**: SQL Alchemy ORM models are split between general domain models (in `core`) and RAG-specific models (in `db/models` like `ChatFile`, `RagDocument`).
+    - **Repositories**: Abstracted CRUD operations for all DB models are located in `/backend/repositories`.
+- **RAG Pipeline (`/backend/rag`)**: Handles the document ingestion lifecycle. 
+    - `connectors.py` manages sync sources.
+    - `ingestion.py` manages the Parse → Chunk → Embed → Store pipeline.
+    - `retrieval.py` and `reranker.py` handle multi-stage vector search and scoring.
+    - `vectorstore.py` interfaces with Milvus.
+- **Services**: Domain-specific logic for file storage, RAG tracking, status streaming (via Redis/SSE), and synchronization.
+- **Routes & Schemas**: FastAPI endpoints (`/routes`) are paired with Pydantic models (`/schemas`) for request validation and response serialization.
+- **Worker**: Handles asynchronous tasks (like heavy RAG ingestion) via an in-process dispatcher, designed to eventually move to a Redis-backed ARQ setup.
+
+### Frontend Architecture (`/frontend`)
+A modern React SPA built with Vite and Tailwind CSS.
+- **Core Components**: `App.tsx` handles routing between the main chat UI, the Admin dashboard, and the RAG management dashboard.
+- **API Client**: Centralized API interaction logic in `api.ts`.
+- **State & Proxy**: The dev server proxies `/api/*` requests to the FastAPI backend.
+
+### Skills (`/skills`)
+Modular agent capabilities are defined as "Skills". Each skill contains its own instructions and logic, allowing the main agent to dynamically expand its toolkit (e.g., financial analysis, brand guidelines).
+
+## Key Patterns
+- **Async First**: Most backend operations use `async/await` for database access and external API calls.
+- **Dependency Injection**: Used extensively in FastAPI routes via the `dependencies.py` module.
+- **RAG Flow**: Document Upload $\rightarrow$ Validation $\rightarrow$ Persistence $\rightarrow$ Parsing $\rightarrow$ Chunking $\rightarrow$ Embedding $\rightarrow$ Milvus Storage.

agents.md

@@ -0,0 +1,34 @@
+# Agent Implementation & Architecture
+
+## Overview
+This document outlines the architecture and implementation requirements for AI agents within the Agent Alpha platform. All agent logic must strictly adhere to the **Repository + Service** pattern defined in `/docs/architecture.md`.
+
+## Agent Structure
+Agents are built using the `pydantic-ai` framework. Every agent must be modularized as follows:
+
+### 1. Core Agent (`backend/core/`)
+The core module should only handle high-level agent lifecycle and basic configuration. It is a "Thin Service" that orchestrates the interaction between the user request, the dependencies, and the LLM.
+
+### 2. Factory Layer (`backend/services/factory/`)
+Complex logic for initializing agents—including model provider setup, logfire integration, tool registration, and environment variable validation—must reside in specialized factories (e.g., `AgentFactory`).
+
+### 3. Service Layer (`backend/services/`)
+Business logic related to agent capabilities must be moved to dedicated services:
+- **RAG Service**: Handles the construction of retrieval tools and interaction with vector stores.
+- **Skill Service**: Manages the discovery, registration, and execution of modular "Skills".
+- **Memory Service**: Orchestrates persistence of conversation history and state.
+
+### 4. Repository Layer (`backend/repositories/`)
+All direct interactions with data sources (SQLAlchemy models, Milvus vector store, local filesystems) must be abstracted into repositories.
+- Repositories are responsible for CRUD operations and state persistence.
+- Repositories use `db.flush()` to ensure data integrity without premature commits in transactional blocks.
+
+## Tooling & Skills
+- **Tools**: Must be decorated with `@agent.tool`. Descriptions must be highly descriptive for LLM comprehension.
+- **Skills**: Modular capabilities stored in `/skills/`. These are registered by the `SkillService` and injected into agents as tools.
+- **Dependencies**: Use FastAPI's Dependency Injection system to provide Repositories and Services to Agent logic via `RunContext`.
+
+## Patterns
+- **Async First**: All agent operations must be asynchronous.
+- **Thin Routes**: FastAPI endpoints must only validate input schemas and call the appropriate Service.
+- **Dependency Injection**: Never instantiate repositories or services directly inside an agent tool; always use the provided context dependencies.

plans/binary-mapping-cook.md

@@ -0,0 +1,45 @@
+---
+name: agent-refactor-plan
+description: Plan to refactor backend/core/agent.py to follow Repository+Service pattern and agents.md guidelines.
+metadata:
+  type: project
+---
+
+# Context
+The current implementation of `/backend/core/agent.py` violates the "Repository + Service" architecture described in `docs/architecture.md`. It contains heavy configuration logic, mixed business logic (RAG tool building), and direct filesystem persistence logic. The goal is to refactor this into a clean, layered architecture as specified in the new `agents.md`.
+
+# Proposed Approach
+I will refactor the agent system by extracting concerns into their respective layers:
+1. **Factory Layer**: Extract initialization logic (Logfire, model providers) from `AgentService` into a new `AgentFactory` in `backend/services/`.
+2. **Repository Layer**: Create a `MemoryRepository` to handle local filesystem persistence for the agent's memory.
+3. **RAG Service**: Move RAG tool construction out of the core agent logic and into a dedicated `RAGService`.
+4. **Thin Core**: Refactor `AgentService` in `/backend/core/agent.py` to act as a thin orchestrator that receives pre-configured agents and dependencies via injection.
+
+# Implementation Steps
+
+### Phase 1: Infrastructure Setup
+- Create `backend/repositories/memory_repository.py` to encapsulate `LocalBackend` logic.
+- Create `backend/services/agent_factory.py` to house the complex initialization logic currently in `AgentService.initialize`.
+- Create `backend/services/rag_service.py` to handle RAG tool construction.
+
+### Phase 2: Core Refactoring
+- Modify `backend/core/agent.py`:
+    - Remove `LocalBackend` and path resolution (Move to `MemoryRepository`).
+    - Remove initialization logic (Move to `AgentFactory`).
+    - Remove `_build_rag_tools` (Move to `RAGService`).
+    - Update `AgentService` to accept a pre-constructed agent and dependencies.
+
+### Phase 3: Dependency Injection & Integration
+- Update FastAPI routes or the main entry point to use the new `AgentFactory` and `RAGService`.
+- Ensure all tools in the `Skill` system are correctly registered via the new services.
+
+# Critical Files
+- `backend/core/agent.py`
+- `backend/services/agent_factory.py`
+- `backend/repositories/memory_repository.py`
+- `backend/services/rag_service.py`
+
+# Verification
+- Run `make run` to ensure the backend starts.
+- Test the `ask` endpoint to verify that the agent still responds with correct RAG results and memory persistence.
+- Verify that the new `AgentFactory` correctly initializes all subagents and skills as before.