From 96a973194703f56fdbbd0484a548ad7fa61e9d1b Mon Sep 17 00:00:00 2001 From: Kit Langton Date: Fri, 10 Jul 2026 13:26:25 -0400 Subject: [PATCH] feat(core): replace instruction checkpoints with value-delta sync (#36254) --- AGENTS.md | 4 +- CONTEXT.md | 62 +-- .../client/src/promise/generated/client.ts | 2 +- .../client/src/promise/generated/types.ts | 28 +- packages/core/schema.json | 332 +++++++++---- .../core/src/control-plane/move-session.ts | 10 +- packages/core/src/database/migration.gen.ts | 1 + .../20260710025429_instruction_sync.ts | 86 ++++ packages/core/src/database/schema.gen.ts | 23 +- packages/core/src/event.ts | 23 +- packages/core/src/instruction-discovery.ts | 22 +- packages/core/src/instructions/builtins.ts | 35 +- packages/core/src/instructions/index.ts | 419 ++++++++-------- packages/core/src/mcp/guidance.ts | 24 +- packages/core/src/reference/guidance.ts | 14 +- packages/core/src/session.ts | 4 +- packages/core/src/session/compaction.ts | 2 +- packages/core/src/session/history.ts | 58 +-- .../src/session/instruction-checkpoint.ts | 130 ----- .../core/src/session/instruction-entry.ts | 80 ++- .../core/src/session/instruction-state.ts | 338 +++++++++++++ packages/core/src/session/message-updater.ts | 11 +- packages/core/src/session/projector.ts | 46 +- packages/core/src/session/runner/llm.ts | 21 +- packages/core/src/session/sql.ts | 20 +- packages/core/src/skill/guidance.ts | 15 +- packages/core/src/tool/read.ts | 2 +- packages/core/test/database-migration.test.ts | 107 +++- packages/core/test/event.test.ts | 34 +- .../core/test/instruction-discovery.test.ts | 50 +- .../core/test/instructions/builtins.test.ts | 19 +- packages/core/test/instructions/index.test.ts | 455 +++++++----------- packages/core/test/lib/instructions.ts | 43 ++ packages/core/test/move-session.test.ts | 20 +- packages/core/test/reference-guidance.test.ts | 28 +- .../core/test/session-instructions.test.ts | 4 +- packages/core/test/session-projector.test.ts | 16 +- .../core/test/session-runner-recorded.test.ts | 1 + packages/core/test/session-runner.test.ts | 282 +++++++---- packages/core/test/skill/guidance.test.ts | 71 +-- packages/docs/build/plugins.mdx | 88 ++-- packages/docs/compaction.mdx | 24 +- packages/docs/instructions.mdx | 32 +- packages/protocol/src/groups/session.ts | 2 +- packages/schema/src/durable-event-manifest.ts | 4 +- packages/schema/src/instruction-entry.ts | 12 + packages/schema/src/instruction.ts | 21 + packages/schema/src/session-event.ts | 14 +- packages/schema/test/event-manifest.test.ts | 4 +- packages/sdk/js/src/v2/gen/types.gen.ts | 48 +- packages/tui/src/context/data.tsx | 6 +- packages/tui/src/routes/session/index.tsx | 25 +- packages/tui/test/cli/tui/data.test.tsx | 6 +- specs/v2/README.md | 1 + specs/v2/instruction-sync-proposal.md | 154 ++++++ specs/v2/schema-changelog.md | 14 + specs/v2/session.md | 10 +- 57 files changed, 2091 insertions(+), 1316 deletions(-) create mode 100644 packages/core/src/database/migration/20260710025429_instruction_sync.ts delete mode 100644 packages/core/src/session/instruction-checkpoint.ts create mode 100644 packages/core/src/session/instruction-state.ts create mode 100644 packages/core/test/lib/instructions.ts create mode 100644 packages/schema/src/instruction.ts create mode 100644 specs/v2/instruction-sync-proposal.md diff --git a/AGENTS.md b/AGENTS.md index 2cd225fba0..f6f3c970e8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -164,5 +164,5 @@ const table = sqliteTable("session", { - Keep delivery vocabulary explicit. Prompts steer by default and promote at the next safe step boundary while the current drain requires continuation. An explicit `queue` input remains pending until the Session would otherwise become idle; promote one queued input at that boundary, then reevaluate continuation before promoting another. Promoting any new user input resets the selected agent's step allowance; a batch of steers resets it once. - One step is one logical LLM call; its durable record covers only the model-visible span. Do not write "provider turn", and do not use bare "turn" for a single call: "turn" is reserved for the future assistant-turn unit containing all steps from prompt promotion until the session would go idle. - Keep EventV2 replay owner claims separate from clustered Session execution ownership. -- Keep the Instructions algebra and built-ins in `src/instructions`; keep instruction producers with their observed domains, and keep Session History selection plus `InstructionCheckpoint` and `InstructionEntry` persistence Session-owned. `InstructionDiscovery` observes ambient global and upward-project instructions. The runner composes built-ins, discovery, guidance, and entries explicitly in `loadInstructions`; there is no instruction registry. -- The durable `Instructions.Applied` record is what the model was last told, per instruction source. Reconciliation narrates drift through `session.instructions.updated` and never rewrites the baseline; only completed compaction rebaselines, while Session movement or committed revert resets the `InstructionCheckpoint`. Unavailable sources keep the model's prior belief, blocking only a Session's first instruction baseline. +- Keep the Instructions algebra and built-ins in `src/instructions`; keep instruction producers with their observed domains, and keep Session History selection plus `InstructionState` and `InstructionEntry` persistence Session-owned. `InstructionDiscovery` observes ambient global and upward-project instructions. The runner composes built-ins, discovery, guidance, and entries explicitly in `loadInstructions`; there is no instruction registry. +- `session.instructions.updated` stores only changed source keys and content hashes. Blob values live once in `instruction_blob`; `instruction_state` is a rebuildable fold cache, never primary state. Render initial instructions and chronological updates from values during request assembly. Completed compaction moves the instruction epoch; Session movement and committed revert clear it. Unavailable sources retain the last value and block only the initial complete delta. diff --git a/CONTEXT.md b/CONTEXT.md index 71ae93ac43..79611f30f5 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -13,11 +13,11 @@ The opaque algebra of independently refreshable typed instruction sources that r _Avoid_: Model Context, System Context **Session History**: -The projected chronological conversation selected for a **Step** after applying the active compaction and **InstructionCheckpoint** baseline cutoffs. +The projected chronological conversation selected for a **Step** after applying the active compaction boundary and interleaving derived **Instruction Updates** from the current **Instruction Epoch**. _Avoid_: Session Context **Instruction Source**: -One independently observed typed value within **Instructions**, represented by a stable namespaced key, JSON codec, loader, pure baseline/update renderers, and an optional removal renderer. +One independently read typed value within **Instructions**, represented by a stable namespaced key, canonical JSON codec, pure first/changed renderers, and an optional removal renderer. _Avoid_: Prompt fragment **InstructionEntry**: @@ -26,22 +26,25 @@ One API-managed, durable, per-Session instruction value. Its slash-free client k **InstructionDiscovery**: The Location-scoped service that observes ambient global and upward-project `AGENTS.md` files as one ordered aggregate **Instruction Source**. -**InstructionCheckpoint**: -The Session-owned durable instruction baseline, baseline sequence, and `Instructions.Applied` record used to prepare later Steps. +**Instruction State**: +The Session-owned projection cache of one instruction log fold: epoch start, values at that start, current values, and the last folded sequence. It is rebuilt from durable events and never authors model-visible facts. **Instruction Update**: -A durable chronological System message published as `session.instructions.updated` that tells the model the newly effective state of one or more changed **Instruction Sources**. -_Avoid_: System notification, raw text diff +A durable `session.instructions.updated` value delta admitted at a **Safe Step Boundary**. Its model-visible System text is rendered from stored values at request assembly and is never persisted verbatim. +_Avoid_: Correction, stored prose, raw text diff -**Instruction Baseline**: -The exact joined instruction text stored by **InstructionCheckpoint** and sent as immutable provider-cache prefix state until completed compaction rebaselines it or Session movement or committed revert resets it. +**Initial Instructions**: +The deterministic instruction text rendered from values at the current **Instruction Epoch** start and sent as provider-cache prefix state until completed compaction moves the epoch or Session movement or committed revert resets it. _Avoid_: Live system prompt -**Applied Instructions**: -The overwriteable model-hidden `Instructions.Applied` record in **InstructionCheckpoint**, containing what the model was last told per **Instruction Source**. +**Instruction Epoch**: +The span between completed compactions. Its start is the last `session.compaction.ended` sequence, or the initial complete instruction delta when no prior epoch exists. + +**Instruction Values**: +The key-to-hash map produced by folding instruction deltas in durable sequence order. Hash bodies live once in the content-addressed instruction blob store. **Unavailable Instruction Source**: -An expected temporary inability to observe an **Instruction Source** value; the runtime retains its prior effective state and emits no update, while an unavailable source blocks creation of the first complete **Instruction Baseline**. +An expected temporary inability to read an **Instruction Source** value; the runtime retains its prior effective value and emits no update, while an unavailable source blocks the initial complete delta. **Safe Step Boundary**: The point during Step preparation, after prior tool settlement and before durable input promotion, where instruction changes may be admitted chronologically. @@ -53,7 +56,7 @@ A durable user input accepted into the Session inbox but not yet included in **S The durable transition that removes an **Admitted Prompt** from pending input and appends its user message to **Session History**. **Step**: -One logical LLM call spanning pre-flight instruction checkpoint preparation, input promotion, request build, and compaction check; the provider stream; and tool settlement. +One logical LLM call spanning pre-flight instruction synchronization, input promotion, request build, and compaction check; the provider stream; and tool settlement. _Avoid_: provider turn, turn (unqualified) **Physical Attempt**: @@ -108,18 +111,16 @@ _Avoid_: Response envelope ## Relationships - **Instructions** is an opaque carrier composed from zero or more **Instruction Sources**. -- **Model Context** is broader than **Instructions**. For each **Step**, the runner assembles the selected agent or provider system text, the **Instruction Baseline**, **Session History**, available tools, and step-local additions into one model request. -- **Session History** contains projected conversational messages and admitted **Instruction Updates**; the active **Instruction Baseline** remains separate provider-request state. +- **Model Context** is broader than **Instructions**. For each **Step**, the runner assembles the selected agent or provider system text, **Initial Instructions**, **Session History**, available tools, and step-local additions into one model request. +- **Session History** persists conversational messages. The runner derives model-facing **Instruction Update** messages from value deltas and interleaves them by durable sequence; **Initial Instructions** remain separate provider-request state. - The runner explicitly loads and combines instruction built-ins, **InstructionDiscovery**, selected-agent skill guidance, reference guidance, MCP guidance, and **InstructionEntry** values. There is no instruction registry. - `Instructions.combine(...)` preserves caller order and rejects duplicate stable namespaced source keys. The runner loads its producers concurrently, then combines them in its fixed declared order. -- Each **Instruction Source** loader returns one coherent typed value or explicitly reports unavailability. `Instructions.make(...)` hides the value type so differently typed sources compose uniformly; its codec compares and stores the value, while pure renderers produce baseline, update, and optional removal text. -- `Instructions.initialize(...)` observes composed **Instructions** once and produces a complete **Instruction Baseline** with **Applied Instructions**. -- `Instructions.reconcile(...)` observes composed **Instructions** once and returns either unchanged or one combined chronological update. It never rewrites the baseline. -- `Instructions.rebaseline(...)` renders a fresh baseline after completed compaction, recalling previously applied values for sources that are temporarily unavailable. -- A changed **Instruction Source** may contribute text to one **Instruction Update** containing the newly effective state. -- An **Instruction Update** persists the exact combined rendered text sent to the model through `session.instructions.updated`. -- **Applied Instructions** advances atomically with the corresponding durable **Instruction Update**. -- **Applied Instructions** stores one codec-encoded JSON value and, for removable sources, a pre-rendered removal message per stable **Instruction Source** key. +- Each **Instruction Source** read returns one coherent typed value, explicit removal, or temporary unavailability. `Instructions.make(...)` hides the value type so differently typed sources compose uniformly; its canonical codec defines storage and hash equivalence, while pure renderers produce first, changed, and optional removal text. +- `Instructions.read(...)` reads every composed source concurrently and exactly once at the boundary. `Instructions.diff(...)` compares encoded-value hashes with current **Instruction Values** and returns one delta plus new blob bodies. +- `Instructions.renderInitial(...)` renders values at the **Instruction Epoch** start. `Instructions.renderUpdate(...)` renders one hydrated delta against the values immediately before it. +- A changed **Instruction Source** contributes its hash to one **Instruction Update**; explicit removal contributes the `"removed"` sentinel. +- An **Instruction Update** persists only its value delta. Rendered text is derived during request assembly and excluded from compaction summaries. +- The instruction blob insert, durable delta, and **Instruction State** advance commit atomically. - Changes from multiple **Instruction Sources** admitted at one safe boundary combine into one **Instruction Update**. - Instruction changes are sampled and admitted lazily at a **Safe Step Boundary**, never pushed asynchronously when their source changes. - At a **Safe Step Boundary**, prior tool results are already settled; instruction preparation completes before newly admitted user input promotes. @@ -130,28 +131,27 @@ _Avoid_: Response envelope - A **Session Drain** is process-local coordination rather than a durable domain entity. Durable recovery must reason from prompts, projected history, physical attempts, and tool state rather than inventing an enclosing execution identity. - An **Execution** contains one or more **Session Drains**; a **Session Drain** contains one reserved assistant-turn span at a time; that span contains **Steps**; and each **Step** contains one or more **Physical Attempts** plus any tool calls it requires. - A **Step** record covers only the model-visible span from first assistant output through tool settlement; pre-flight leaves no record, and one Step settles at most one record. -- The first **Step** renders the latest complete **Instruction Baseline** and creates its **InstructionCheckpoint** without emitting a redundant **Instruction Update**; an unavailable initial source blocks the Step instead of persisting an incomplete baseline. +- The first **Step** admits one complete delta and renders **Initial Instructions** without narrating that delta in history; an unavailable initial source blocks the Step instead of persisting incomplete values. - Instruction preparation precedes durable input promotion on every Step so an unavailable first baseline leaves pending input untouched and later updates enter history before newly promoted input. -- Completed compaction rebaselines the **InstructionCheckpoint** from current **Instructions** and removes earlier **Instruction Updates** from active projected model history while preserving durable audit history. -- A newly composed **Instruction Source** absent from **Applied Instructions** emits its baseline rendering once at the next **Safe Step Boundary**. +- Completed compaction moves the **Instruction Epoch** to the exact `session.compaction.ended` sequence and copies current hashes to the epoch's initial values. Earlier updates leave active model history while durable deltas remain. +- A newly composed **Instruction Source** absent from current **Instruction Values** emits its first rendering once at the next **Safe Step Boundary**. - **Unavailable Instruction Source** uses stale-while-revalidate semantics and is distinct from a successfully loaded absence, which may emit removal text. - **InstructionDiscovery** observes ambient instructions as one ordered aggregate **Instruction Source**. - Ambient discovery reads global and upward-project `AGENTS.md` files and honors `OPENCODE_DISABLE_PROJECT_CONFIG` for project files. - After a successful internal file or directory read, nearby `AGENTS.md` files toward the Location root are injected once per Session as durable synthetic instruction messages. - **InstructionEntry** stores API-managed per-Session JSON values. Each entry contributes one `api/` **Instruction Source**, so adding, replacing, or removing an entry is reconciled at the next **Safe Step Boundary**. - Location-scoped instruction producers naturally re-resolve when a moved Session next runs in its destination Location. -- Moving a Session resets its **InstructionCheckpoint**, so the destination must initialize a complete baseline before another prompt can promote. Committed revert also resets the checkpoint. +- Moving a Session clears its **Instruction State**, so the destination must admit a complete delta before another prompt can promote. Committed revert does the same; replay derives both resets from their durable events. - Selected-agent available-skill guidance is an **Instruction Source** composed explicitly by the runner. It lists only names and descriptions permitted for that agent; skill bodies and locations are exposed only through the permission-checked `skill` tool. - The selected agent and model are sampled when a **Step** starts. Changes admitted after that boundary apply to the next Step and do not restart the current Step. - An agent switch that changes selected-agent guidance produces an **Instruction Update** while preserving the current baseline. - Local tool authorization and pending permission requests retain the effective agent of the **Step** that issued the call; a later agent switch cannot change that call's policy. - Instruction source changes never wake idle Sessions; the next naturally scheduled **Safe Step Boundary** loads and compares current values lazily. - Once admitted, an **Instruction Update** remains durable even if the following **Physical Attempt** fails and is replayed unchanged on retry. -- **Instruction Updates** remain durable Session-message history; normal user-facing transcript surfaces may hide them. +- **Instruction Updates** remain durable value history but are not `session_message` rows. Clients display changed keys rather than model-facing prose. - The date **Instruction Source** initially preserves host-local calendar-date behavior; a configured user timezone may replace that default later. -- An **Instruction Baseline** is stored durably and reused verbatim across process restarts until rebaseline or reset. -- An **Instruction Baseline** durably preserves the exact joined text used for its part of the active provider-cache prefix. -- A model/provider switch preserves the current **InstructionCheckpoint** and chronological conversation history; the new selection applies to the next **Step**. +- **Initial Instructions** are recomputed deterministically from durable values for every request; rendered bytes are not stored. +- A model/provider switch preserves current **Instruction Values**, the **Instruction Epoch**, and chronological conversation history; the new selection applies to the next **Step**. - **Native Continuation Metadata** remains in durable history. Step projection includes it only for a successful exact originating provider/model match; failed Steps and incompatible models omit opaque metadata, while non-empty visible reasoning lowers to ordinary assistant text after a model switch. This conservative relation may widen only when recorded provider tests establish compatibility. - **Model Request Options** remain provider-semantic through Catalog resolution. The Session runner maps them into the LLM package's provider-option namespace; the selected protocol adapter alone owns provider wire encoding. - **Generation Controls**, protocol-semantic **Model Request Options**, and compatibility request body fields are separate Catalog domains. A shared ingestion adapter partitions legacy and models.dev AI-SDK-shaped options before routing. @@ -195,7 +195,7 @@ _Avoid_: Response envelope - `sessions.message({ sessionID, messageID })` is a required resource lookup. An unknown Session fails with `SessionNotFoundError`; a known Session with an absent or differently owned message fails with `MessageNotFoundError` without disclosing cross-Session ownership. Absence is not represented as `undefined` across the public HTTP boundary. - `sessions.interrupt({ sessionID })` first verifies that the durable Session exists, failing with `SessionNotFoundError` otherwise. For a known Session, interruption is idempotent: idle, already-settled, or locally unowned execution is a no-op. - `sessions.active()` snapshots the current process's foreground Session drain registry as a record of Session IDs to `{ type: "running" }`. Missing IDs are inactive; background subagents and tasks do not make their parent Session active, and process restart clears the registry. -- `sessions.context({ sessionID })` preserves the existing message-only operation. It returns projected **Session History**; it does not include or represent the complete **Model Context**, whose system text, **Instruction Baseline**, tools, and step-local additions remain separate. +- `sessions.context({ sessionID })` preserves the existing message-only operation. It returns projected **Session History**; it does not include or represent the complete **Model Context**, whose agent system text, **Initial Instructions**, tools, and step-local additions remain separate. - **Open question**: Should a future, separately named operation expose complete **Model Context**, including the instruction baseline, applied instruction metadata, tools, and step-local additions? - `sessions.prompt(...)` exposes `resume?: boolean`. Omitting it preserves durable admission followed by an advisory execution wake; `resume: false` requests durable admit-only behavior. - The public operation remains `sessions.prompt(...)`; `SessionPending.admit` is the internal primitive, while the public `Admission` result and `resume` option express its durable admission semantics. diff --git a/packages/client/src/promise/generated/client.ts b/packages/client/src/promise/generated/client.ts index 12c8aefd32..28c223e6b5 100644 --- a/packages/client/src/promise/generated/client.ts +++ b/packages/client/src/promise/generated/client.ts @@ -701,7 +701,7 @@ export function make(options: ClientOptions) { path: `/api/session/${encodeURIComponent(input.sessionID)}/instructions/entries/${encodeURIComponent(input.key)}`, body: { value: input["value"] }, successStatus: 204, - declaredStatuses: [404, 400, 401], + declaredStatuses: [404, 413, 400, 401], empty: true, }, requestOptions, diff --git a/packages/client/src/promise/generated/types.ts b/packages/client/src/promise/generated/types.ts index 17d21221a7..c6a703c5fa 100644 --- a/packages/client/src/promise/generated/types.ts +++ b/packages/client/src/promise/generated/types.ts @@ -90,6 +90,18 @@ export type UnknownError = { export const isUnknownError = (value: unknown): value is UnknownError => typeof value === "object" && value !== null && "_tag" in value && value["_tag"] === "UnknownError" +export type InstructionEntryValueTooLargeError = { + readonly _tag: "InstructionEntryValueTooLargeError" + readonly actualBytes: number + readonly maxBytes: number + readonly message: string +} +export const isInstructionEntryValueTooLargeError = (value: unknown): value is InstructionEntryValueTooLargeError => + typeof value === "object" && + value !== null && + "_tag" in value && + value["_tag"] === "InstructionEntryValueTooLargeError" + export type ProviderNotFoundError = { readonly _tag: "ProviderNotFoundError" readonly providerID: string @@ -1250,9 +1262,9 @@ export type SessionLogOutput = created: number metadata?: { [x: string]: unknown } type: "session.forked" - durable: { aggregateID: string; seq: number; version: 1 } + durable: { aggregateID: string; seq: number; version: 2 } location?: { directory: string; workspaceID?: string } - data: { sessionID: string; parentID: string; from?: string } + data: { sessionID: string; parentID: string; parentSeq: number; from?: string } } | { id: string @@ -1339,9 +1351,9 @@ export type SessionLogOutput = created: number metadata?: { [x: string]: unknown } type: "session.instructions.updated" - durable: { aggregateID: string; seq: number; version: 1 } + durable: { aggregateID: string; seq: number; version: 2 } location?: { directory: string; workspaceID?: string } - data: { sessionID: string; text: string } + data: { sessionID: string; delta: { [x: string]: string | "removed" } } } | { id: string @@ -4734,9 +4746,9 @@ export type EventSubscribeOutput = created: number metadata?: { [x: string]: unknown } type: "session.forked" - durable: { aggregateID: string; seq: number; version: 1 } + durable: { aggregateID: string; seq: number; version: 2 } location?: { directory: string; workspaceID?: string } - data: { sessionID: string; parentID: string; from?: string } + data: { sessionID: string; parentID: string; parentSeq: number; from?: string } } | { id: string @@ -4823,9 +4835,9 @@ export type EventSubscribeOutput = created: number metadata?: { [x: string]: unknown } type: "session.instructions.updated" - durable: { aggregateID: string; seq: number; version: 1 } + durable: { aggregateID: string; seq: number; version: 2 } location?: { directory: string; workspaceID?: string } - data: { sessionID: string; text: string } + data: { sessionID: string; delta: { [x: string]: string | "removed" } } } | { id: string diff --git a/packages/core/schema.json b/packages/core/schema.json index 5d216eefc9..82a395b4e4 100644 --- a/packages/core/schema.json +++ b/packages/core/schema.json @@ -1,8 +1,10 @@ { "version": "7", "dialect": "sqlite", - "id": "2c759c08-79c8-4179-9e15-d2fea45c9dec", - "prevIds": ["01451b27-1e51-4657-b2d0-4b457dffa3ec"], + "id": "5f0a1db8-d4bf-42c3-becb-96b46fe66bed", + "prevIds": [ + "666138ef-82cb-4a9a-a765-e6669a436ff3" + ], "ddl": [ { "name": "workspace", @@ -49,13 +51,17 @@ "entityType": "tables" }, { - "name": "instruction_checkpoint", + "name": "instruction_blob", "entityType": "tables" }, { "name": "instruction_entry", "entityType": "tables" }, + { + "name": "instruction_state", + "entityType": "tables" + }, { "name": "message", "entityType": "tables" @@ -786,39 +792,19 @@ "autoincrement": false, "default": null, "generated": null, - "name": "session_id", + "name": "hash", "entityType": "columns", - "table": "instruction_checkpoint" + "table": "instruction_blob" }, { "type": "text", - "notNull": true, + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "baseline", + "name": "value", "entityType": "columns", - "table": "instruction_checkpoint" - }, - { - "type": "text", - "notNull": true, - "autoincrement": false, - "default": null, - "generated": null, - "name": "snapshot", - "entityType": "columns", - "table": "instruction_checkpoint" - }, - { - "type": "integer", - "notNull": true, - "autoincrement": false, - "default": null, - "generated": null, - "name": "baseline_seq", - "entityType": "columns", - "table": "instruction_checkpoint" + "table": "instruction_blob" }, { "type": "text", @@ -842,7 +828,7 @@ }, { "type": "text", - "notNull": true, + "notNull": false, "autoincrement": false, "default": null, "generated": null, @@ -850,6 +836,16 @@ "entityType": "columns", "table": "instruction_entry" }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": "false", + "generated": null, + "name": "removed", + "entityType": "columns", + "table": "instruction_entry" + }, { "type": "integer", "notNull": true, @@ -870,6 +866,56 @@ "entityType": "columns", "table": "instruction_entry" }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "session_id", + "entityType": "columns", + "table": "instruction_state" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "epoch_start", + "entityType": "columns", + "table": "instruction_state" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "through_seq", + "entityType": "columns", + "table": "instruction_state" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "initial_values", + "entityType": "columns", + "table": "instruction_state" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "current_values", + "entityType": "columns", + "table": "instruction_state" + }, { "type": "text", "notNull": false, @@ -1180,6 +1226,16 @@ "entityType": "columns", "table": "session" }, + { + "type": "integer", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "fork_seq", + "entityType": "columns", + "table": "session" + }, { "type": "text", "notNull": true, @@ -1501,9 +1557,13 @@ "table": "session_share" }, { - "columns": ["project_id"], + "columns": [ + "project_id" + ], "tableTo": "project", - "columnsTo": ["id"], + "columnsTo": [ + "id" + ], "onUpdate": "NO ACTION", "onDelete": "CASCADE", "nameExplicit": false, @@ -1512,9 +1572,13 @@ "table": "workspace" }, { - "columns": ["active_account_id"], + "columns": [ + "active_account_id" + ], "tableTo": "account", - "columnsTo": ["id"], + "columnsTo": [ + "id" + ], "onUpdate": "NO ACTION", "onDelete": "SET NULL", "nameExplicit": false, @@ -1523,9 +1587,13 @@ "table": "account_state" }, { - "columns": ["aggregate_id"], + "columns": [ + "aggregate_id" + ], "tableTo": "event_sequence", - "columnsTo": ["aggregate_id"], + "columnsTo": [ + "aggregate_id" + ], "onUpdate": "NO ACTION", "onDelete": "CASCADE", "nameExplicit": false, @@ -1534,9 +1602,13 @@ "table": "event" }, { - "columns": ["project_id"], + "columns": [ + "project_id" + ], "tableTo": "project", - "columnsTo": ["id"], + "columnsTo": [ + "id" + ], "onUpdate": "NO ACTION", "onDelete": "CASCADE", "nameExplicit": false, @@ -1545,9 +1617,13 @@ "table": "permission" }, { - "columns": ["project_id"], + "columns": [ + "project_id" + ], "tableTo": "project", - "columnsTo": ["id"], + "columnsTo": [ + "id" + ], "onUpdate": "NO ACTION", "onDelete": "CASCADE", "nameExplicit": false, @@ -1556,20 +1632,13 @@ "table": "project_directory" }, { - "columns": ["session_id"], + "columns": [ + "session_id" + ], "tableTo": "session", - "columnsTo": ["id"], - "onUpdate": "NO ACTION", - "onDelete": "CASCADE", - "nameExplicit": false, - "name": "fk_instruction_checkpoint_session_id_session_id_fk", - "entityType": "fks", - "table": "instruction_checkpoint" - }, - { - "columns": ["session_id"], - "tableTo": "session", - "columnsTo": ["id"], + "columnsTo": [ + "id" + ], "onUpdate": "NO ACTION", "onDelete": "CASCADE", "nameExplicit": false, @@ -1578,9 +1647,28 @@ "table": "instruction_entry" }, { - "columns": ["session_id"], + "columns": [ + "session_id" + ], "tableTo": "session", - "columnsTo": ["id"], + "columnsTo": [ + "id" + ], + "onUpdate": "NO ACTION", + "onDelete": "CASCADE", + "nameExplicit": false, + "name": "fk_instruction_state_session_id_session_id_fk", + "entityType": "fks", + "table": "instruction_state" + }, + { + "columns": [ + "session_id" + ], + "tableTo": "session", + "columnsTo": [ + "id" + ], "onUpdate": "NO ACTION", "onDelete": "CASCADE", "nameExplicit": false, @@ -1589,9 +1677,13 @@ "table": "message" }, { - "columns": ["message_id"], + "columns": [ + "message_id" + ], "tableTo": "message", - "columnsTo": ["id"], + "columnsTo": [ + "id" + ], "onUpdate": "NO ACTION", "onDelete": "CASCADE", "nameExplicit": false, @@ -1600,9 +1692,13 @@ "table": "part" }, { - "columns": ["session_id"], + "columns": [ + "session_id" + ], "tableTo": "session", - "columnsTo": ["id"], + "columnsTo": [ + "id" + ], "onUpdate": "NO ACTION", "onDelete": "CASCADE", "nameExplicit": false, @@ -1611,9 +1707,13 @@ "table": "session_message" }, { - "columns": ["session_id"], + "columns": [ + "session_id" + ], "tableTo": "session", - "columnsTo": ["id"], + "columnsTo": [ + "id" + ], "onUpdate": "NO ACTION", "onDelete": "CASCADE", "nameExplicit": false, @@ -1622,9 +1722,13 @@ "table": "session_pending" }, { - "columns": ["project_id"], + "columns": [ + "project_id" + ], "tableTo": "project", - "columnsTo": ["id"], + "columnsTo": [ + "id" + ], "onUpdate": "NO ACTION", "onDelete": "CASCADE", "nameExplicit": false, @@ -1633,9 +1737,13 @@ "table": "session" }, { - "columns": ["session_id"], + "columns": [ + "session_id" + ], "tableTo": "session", - "columnsTo": ["id"], + "columnsTo": [ + "id" + ], "onUpdate": "NO ACTION", "onDelete": "CASCADE", "nameExplicit": false, @@ -1644,133 +1752,183 @@ "table": "session_share" }, { - "columns": ["email", "url"], + "columns": [ + "email", + "url" + ], "nameExplicit": false, "name": "control_account_pk", "entityType": "pks", "table": "control_account" }, { - "columns": ["project_id", "directory"], + "columns": [ + "project_id", + "directory" + ], "nameExplicit": false, "name": "project_directory_pk", "entityType": "pks", "table": "project_directory" }, { - "columns": ["session_id", "key"], + "columns": [ + "session_id", + "key" + ], "nameExplicit": false, "name": "instruction_entry_pk", "entityType": "pks", "table": "instruction_entry" }, { - "columns": ["id"], + "columns": [ + "id" + ], "nameExplicit": false, "name": "workspace_pk", "table": "workspace", "entityType": "pks" }, { - "columns": ["name"], + "columns": [ + "name" + ], "nameExplicit": false, "name": "data_migration_pk", "table": "data_migration", "entityType": "pks" }, { - "columns": ["id"], + "columns": [ + "id" + ], "nameExplicit": false, "name": "account_state_pk", "table": "account_state", "entityType": "pks" }, { - "columns": ["id"], + "columns": [ + "id" + ], "nameExplicit": false, "name": "account_pk", "table": "account", "entityType": "pks" }, { - "columns": ["id"], + "columns": [ + "id" + ], "nameExplicit": false, "name": "credential_pk", "table": "credential", "entityType": "pks" }, { - "columns": ["aggregate_id"], + "columns": [ + "aggregate_id" + ], "nameExplicit": false, "name": "event_sequence_pk", "table": "event_sequence", "entityType": "pks" }, { - "columns": ["id"], + "columns": [ + "id" + ], "nameExplicit": false, "name": "event_pk", "table": "event", "entityType": "pks" }, { - "columns": ["id"], + "columns": [ + "id" + ], "nameExplicit": false, "name": "permission_pk", "table": "permission", "entityType": "pks" }, { - "columns": ["id"], + "columns": [ + "id" + ], "nameExplicit": false, "name": "project_pk", "table": "project", "entityType": "pks" }, { - "columns": ["session_id"], + "columns": [ + "hash" + ], "nameExplicit": false, - "name": "instruction_checkpoint_pk", - "table": "instruction_checkpoint", + "name": "instruction_blob_pk", + "table": "instruction_blob", "entityType": "pks" }, { - "columns": ["id"], + "columns": [ + "session_id" + ], + "nameExplicit": false, + "name": "instruction_state_pk", + "table": "instruction_state", + "entityType": "pks" + }, + { + "columns": [ + "id" + ], "nameExplicit": false, "name": "message_pk", "table": "message", "entityType": "pks" }, { - "columns": ["id"], + "columns": [ + "id" + ], "nameExplicit": false, "name": "part_pk", "table": "part", "entityType": "pks" }, { - "columns": ["id"], + "columns": [ + "id" + ], "nameExplicit": false, "name": "session_message_pk", "table": "session_message", "entityType": "pks" }, { - "columns": ["id"], + "columns": [ + "id" + ], "nameExplicit": false, "name": "session_input_pk", "table": "session_pending", "entityType": "pks" }, { - "columns": ["id"], + "columns": [ + "id" + ], "nameExplicit": false, "name": "session_pk", "table": "session", "entityType": "pks" }, { - "columns": ["session_id"], + "columns": [ + "session_id" + ], "nameExplicit": false, "name": "session_share_pk", "table": "session_share", @@ -2079,5 +2237,5 @@ "table": "session" } ], - "renames": ["session_input->session_pending"] -} + "renames": [] +} \ No newline at end of file diff --git a/packages/core/src/control-plane/move-session.ts b/packages/core/src/control-plane/move-session.ts index 6f0c6e3081..84d135d3f9 100644 --- a/packages/core/src/control-plane/move-session.ts +++ b/packages/core/src/control-plane/move-session.ts @@ -8,6 +8,7 @@ import { Location } from "../location" import { ProjectV2 } from "../project" import { SessionV2 } from "../session" import { SessionEvent } from "../session/event" +import { SessionExecution } from "../session/execution" import { SessionSchema } from "../session/schema" import { SessionStore } from "../session/store" import { AbsolutePath, RelativePath } from "../schema" @@ -73,6 +74,7 @@ const layer = Layer.effect( const events = yield* EventV2.Service const project = yield* ProjectV2.Service const sessions = yield* SessionStore.Service + const execution = yield* SessionExecution.Service const moveSession = Effect.fn("MoveSession.moveSession")(function* (input: Input) { const current = yield* sessions.get(input.sessionID) @@ -86,6 +88,12 @@ const layer = Layer.effect( return yield* new DestinationProjectMismatchError({ expected: current.projectID, actual: destination.id }) } + // A move must not race active execution: a mid-drain relocation would let + // the source Location dispatch a request assembled under stale instructions + // and history. Serialize like removal does — stop the drain, then move. + yield* execution.interrupt(input.sessionID) + yield* execution.awaitIdle(input.sessionID) + const moveChanges = input.moveChanges && source.directory !== destination.directory const sourceRepository = moveChanges ? yield* git.repo.discover(current.location.directory) : undefined if (moveChanges && !sourceRepository) @@ -143,5 +151,5 @@ const layer = Layer.effect( export const node = makeGlobalNode({ service: Service, layer, - deps: [Git.node, EventV2.node, ProjectV2.node, SessionStore.node], + deps: [Git.node, EventV2.node, ProjectV2.node, SessionStore.node, SessionExecution.node], }) diff --git a/packages/core/src/database/migration.gen.ts b/packages/core/src/database/migration.gen.ts index 1df6328e7a..2039393030 100644 --- a/packages/core/src/database/migration.gen.ts +++ b/packages/core/src/database/migration.gen.ts @@ -53,5 +53,6 @@ export const migrations = ( import("./migration/20260709025533_drop-todo"), import("./migration/20260709163752_time_suspended"), import("./migration/20260709190621_session_pending_table"), + import("./migration/20260710025429_instruction_sync"), ]) ).map((module) => module.default) satisfies DatabaseMigration.Migration[] diff --git a/packages/core/src/database/migration/20260710025429_instruction_sync.ts b/packages/core/src/database/migration/20260710025429_instruction_sync.ts new file mode 100644 index 0000000000..6239fd9bce --- /dev/null +++ b/packages/core/src/database/migration/20260710025429_instruction_sync.ts @@ -0,0 +1,86 @@ +import { Effect } from "effect" +import type { DatabaseMigration } from "../migration" + +export default { + id: "20260710025429_instruction_sync", + up(tx) { + return Effect.gen(function* () { + yield* tx.run(`ALTER TABLE \`session\` ADD \`fork_seq\` integer;`) + yield* tx.run(`PRAGMA foreign_keys=OFF;`) + yield* tx.run(` + CREATE TABLE \`__new_instruction_entry\` ( + \`session_id\` text NOT NULL, + \`key\` text NOT NULL, + \`value\` text, + \`removed\` integer DEFAULT false NOT NULL, + \`time_created\` integer NOT NULL, + \`time_updated\` integer NOT NULL, + CONSTRAINT \`instruction_entry_pk\` PRIMARY KEY(\`session_id\`, \`key\`), + CONSTRAINT \`fk_instruction_entry_session_id_session_id_fk\` FOREIGN KEY (\`session_id\`) REFERENCES \`session\`(\`id\`) ON DELETE CASCADE + ); + `) + yield* tx.run(` + INSERT INTO \`__new_instruction_entry\`( + \`session_id\`, \`key\`, \`value\`, \`removed\`, \`time_created\`, \`time_updated\` + ) + SELECT \`session_id\`, \`key\`, \`value\`, false, \`time_created\`, \`time_updated\` + FROM \`instruction_entry\`; + `) + yield* tx.run(`DROP TABLE \`instruction_entry\`;`) + yield* tx.run(`ALTER TABLE \`__new_instruction_entry\` RENAME TO \`instruction_entry\`;`) + yield* tx.run(`PRAGMA foreign_keys=ON;`) + yield* tx.run(` + CREATE TABLE \`instruction_blob\` ( + \`hash\` text PRIMARY KEY, + \`value\` text + ); + `) + yield* tx.run(` + CREATE TABLE \`instruction_state\` ( + \`session_id\` text PRIMARY KEY, + \`epoch_start\` integer NOT NULL, + \`through_seq\` integer NOT NULL, + \`initial_values\` text NOT NULL, + \`current_values\` text NOT NULL, + CONSTRAINT \`fk_instruction_state_session_id_session_id_fk\` FOREIGN KEY (\`session_id\`) REFERENCES \`session\`(\`id\`) ON DELETE CASCADE + ); + `) + // Persisted System rows were exclusively pre-beta instruction prose, + // including fork copies whose message IDs no longer match the source event. + yield* tx.run(`DELETE FROM \`session_message\` WHERE \`type\` = 'system';`) + yield* tx.run(` + UPDATE \`session\` + SET \`fork_seq\` = COALESCE( + ( + SELECT MIN(\`seq\`) - 1 + FROM \`event\` + WHERE \`aggregate_id\` = \`session\`.\`id\` AND \`seq\` > 0 + ), + ( + SELECT \`seq\` + FROM \`event_sequence\` + WHERE \`aggregate_id\` = \`session\`.\`id\` + ), + 0 + ) + WHERE \`fork_session_id\` IS NOT NULL; + `) + yield* tx.run(` + UPDATE \`event\` + SET + \`type\` = 'session.forked.2', + \`data\` = json_set( + \`data\`, + '$.parentSeq', + COALESCE( + (SELECT \`fork_seq\` FROM \`session\` WHERE \`id\` = \`event\`.\`aggregate_id\`), + 0 + ) + ) + WHERE \`type\` = 'session.forked.1'; + `) + yield* tx.run(`DELETE FROM \`event\` WHERE \`type\` = 'session.instructions.updated.1';`) + yield* tx.run(`DROP TABLE \`instruction_checkpoint\`;`) + }) + }, +} satisfies DatabaseMigration.Migration diff --git a/packages/core/src/database/schema.gen.ts b/packages/core/src/database/schema.gen.ts index f9b27c0e1a..7d9fc7937b 100644 --- a/packages/core/src/database/schema.gen.ts +++ b/packages/core/src/database/schema.gen.ts @@ -126,25 +126,33 @@ export default { ); `) yield* tx.run(` - CREATE TABLE \`instruction_checkpoint\` ( - \`session_id\` text PRIMARY KEY, - \`baseline\` text NOT NULL, - \`snapshot\` text NOT NULL, - \`baseline_seq\` integer NOT NULL, - CONSTRAINT \`fk_instruction_checkpoint_session_id_session_id_fk\` FOREIGN KEY (\`session_id\`) REFERENCES \`session\`(\`id\`) ON DELETE CASCADE + CREATE TABLE \`instruction_blob\` ( + \`hash\` text PRIMARY KEY, + \`value\` text ); `) yield* tx.run(` CREATE TABLE \`instruction_entry\` ( \`session_id\` text NOT NULL, \`key\` text NOT NULL, - \`value\` text NOT NULL, + \`value\` text, + \`removed\` integer DEFAULT false NOT NULL, \`time_created\` integer NOT NULL, \`time_updated\` integer NOT NULL, CONSTRAINT \`instruction_entry_pk\` PRIMARY KEY(\`session_id\`, \`key\`), CONSTRAINT \`fk_instruction_entry_session_id_session_id_fk\` FOREIGN KEY (\`session_id\`) REFERENCES \`session\`(\`id\`) ON DELETE CASCADE ); `) + yield* tx.run(` + CREATE TABLE \`instruction_state\` ( + \`session_id\` text PRIMARY KEY, + \`epoch_start\` integer NOT NULL, + \`through_seq\` integer NOT NULL, + \`initial_values\` text NOT NULL, + \`current_values\` text NOT NULL, + CONSTRAINT \`fk_instruction_state_session_id_session_id_fk\` FOREIGN KEY (\`session_id\`) REFERENCES \`session\`(\`id\`) ON DELETE CASCADE + ); + `) yield* tx.run(` CREATE TABLE \`message\` ( \`id\` text PRIMARY KEY, @@ -198,6 +206,7 @@ export default { \`parent_id\` text, \`fork_session_id\` text, \`fork_message_id\` text, + \`fork_seq\` integer, \`slug\` text NOT NULL, \`directory\` text NOT NULL, \`path\` text, diff --git a/packages/core/src/event.ts b/packages/core/src/event.ts index 8641c92859..886d76a0f1 100644 --- a/packages/core/src/event.ts +++ b/packages/core/src/event.ts @@ -137,7 +137,8 @@ export interface Interface { ) => Effect.Effect> readonly subscribe: Subscribe /** - * Durable, ordered, gap-free per-aggregate log read. `follow: false` + * Durable, ordered per-aggregate log read. Forked aggregates may reserve an + * inherited prefix before their first child-authored event. `follow: false` * completes at the end of the log; `follow: true` replays then transitions * to live. Both modes emit one `Synced` marker at the captured replay * watermark. @@ -203,7 +204,6 @@ export const layerWith = (options?: LayerOptions) => typed: new Map>(), } const projectors = new Map() - // TODO: Bind durable projectors to exact type+version before supporting incompatible historical payloads. const listeners = new Array() const { db } = yield* Database.Service const logReadPageSize = options?.logReadPageSize ?? 512 @@ -260,7 +260,7 @@ export const layerWith = (options?: LayerOptions) => }), ) } - const list = projectors.get(event.type) ?? [] + const list = projectors.get(versionedType(definition.type, durable.version)) ?? [] return yield* Effect.uninterruptible( Effect.gen(function* () { const committed = yield* db @@ -515,18 +515,6 @@ export const layerWith = (options?: LayerOptions) => }), ) } - const start = events[0]?.seq ?? 0 - for (const [index, event] of events.entries()) { - const seq = start + index - if (event.seq !== seq) { - yield* Effect.die( - new InvalidDurableEventError({ - type: event.type, - message: `Replay sequence mismatch at index ${index}: expected ${seq}, got ${event.seq}`, - }), - ) - } - } for (const event of events) { yield* replay(event, options) } @@ -727,9 +715,10 @@ export const layerWith = (options?: LayerOptions) => const project = (definition: D, projector: Subscriber): Effect.Effect => Effect.sync(() => { - const list = projectors.get(definition.type) ?? [] + const key = definition.durable ? versionedType(definition.type, definition.durable.version) : definition.type + const list = projectors.get(key) ?? [] list.push((event) => projector(event as Payload)) - projectors.set(definition.type, list) + projectors.set(key, list) }) return Service.of({ diff --git a/packages/core/src/instruction-discovery.ts b/packages/core/src/instruction-discovery.ts index 0ac7b8a12d..f838083b40 100644 --- a/packages/core/src/instruction-discovery.ts +++ b/packages/core/src/instruction-discovery.ts @@ -31,15 +31,17 @@ const layer = Layer.effect( const global = yield* Global.Service const location = yield* Location.Service - const source = (value: ReadonlyArray | Instructions.Unavailable) => - Instructions.make({ + const source = (value: ReadonlyArray | Instructions.Unavailable | Instructions.Removed) => + Instructions.make>({ key, codec: Schema.toCodecJson(Files), - load: Effect.succeed(value), - baseline: render, - update: (_previous, current) => - `These instructions replace all previously loaded ambient instructions.\n\n${render(current)}`, - removed: () => "Previously loaded instructions no longer apply.", + read: Effect.succeed(value), + render: { + initial: render, + changed: (_previous, current) => + `These instructions replace all previously loaded ambient instructions.\n\n${render(current)}`, + removed: () => "Previously loaded instructions no longer apply.", + }, }) const observe = Effect.fn("InstructionDiscovery.observe")(function* () { @@ -82,11 +84,7 @@ const layer = Layer.effect( load: () => observe().pipe( Effect.map((files) => - files === Instructions.unavailable - ? source(files) - : files.length === 0 - ? Instructions.empty - : source(files), + Array.isArray(files) && files.length === 0 ? source(Instructions.removed) : source(files), ), Effect.catch(() => Effect.succeed(source(Instructions.unavailable))), Effect.catchDefect(() => Effect.succeed(source(Instructions.unavailable))), diff --git a/packages/core/src/instructions/builtins.ts b/packages/core/src/instructions/builtins.ts index 0414cc98c1..b116971834 100644 --- a/packages/core/src/instructions/builtins.ts +++ b/packages/core/src/instructions/builtins.ts @@ -15,29 +15,34 @@ const layer = Layer.effect( Service, Effect.gen(function* () { const location = yield* Location.Service - const environment = [ - "", - ` Working directory: ${location.directory}`, - ` Workspace root folder: ${location.project.directory}`, - ` Is directory a git repo: ${location.vcs?.type === "git" ? "yes" : "no"}`, - ` Platform: ${process.platform}`, - "", - ].join("\n") const instructions = Instructions.combine([ Instructions.make({ key: Instructions.Key.make("core/environment"), codec: Schema.toCodecJson(Schema.String), - load: Effect.succeed(environment), - baseline: (environment) => - ["Here is some useful information about the environment you are running in:", environment].join("\n"), - update: (_previous, environment) => ["The environment you are running in is now:", environment].join("\n"), + read: Effect.sync(() => + [ + "", + ` Working directory: ${location.directory}`, + ` Workspace root folder: ${location.project.directory}`, + ` Is directory a git repo: ${location.vcs?.type === "git" ? "yes" : "no"}`, + ` Platform: ${process.platform}`, + "", + ].join("\n"), + ), + render: { + initial: (environment) => + ["Here is some useful information about the environment you are running in:", environment].join("\n"), + changed: (_previous, environment) => ["The environment you are running in is now:", environment].join("\n"), + }, }), Instructions.make({ key: Instructions.Key.make("core/date"), codec: Schema.toCodecJson(Schema.String), - load: DateTime.nowAsDate.pipe(Effect.map((date) => date.toDateString())), - baseline: (date) => `Today's date: ${date}`, - update: (_previous, date) => `Today's date is now: ${date}`, + read: DateTime.nowAsDate.pipe(Effect.map((date) => date.toDateString())), + render: { + initial: (date) => `Today's date: ${date}`, + changed: (_previous, date) => `Today's date is now: ${date}`, + }, }), ]) diff --git a/packages/core/src/instructions/index.ts b/packages/core/src/instructions/index.ts index 7fe86cfa16..d7af2a78a3 100644 --- a/packages/core/src/instructions/index.ts +++ b/packages/core/src/instructions/index.ts @@ -1,80 +1,70 @@ export * as Instructions from "./index" -import { Effect, Option, Schema } from "effect" +import { createHash } from "crypto" +import { Instruction } from "@opencode-ai/schema/instruction" +import { Data, Effect, Option, Schema } from "effect" -/** - * Models privileged instructions as independently refreshable typed sources. - * - * `Source` describes how to observe, compare, and render one value. `make` - * closes over `A`, producing opaque `Instructions` that compose uniformly with - * instructions built from other value types. - * - * The durable `Applied` record tracks what the model was last told, per source: - * it is the model's current belief. Interpreters uphold one invariant — - * `reconcile` never rewrites the baseline; it only narrates drift as update - * text. Only `rebaseline` (compaction) and `initialize` (first step) produce - * baseline text. - * - * Returning `unavailable` means observation failed temporarily. It differs from - * removing a source from the instructions: the model's prior belief stands. - * `reconcile` retains the applied value silently, and `rebaseline` restates the - * belief by rendering the last-applied value instead of a live observation. - * - * @module - */ +export const Key = Instruction.Key +export type Key = Instruction.Key +export const Hash = Instruction.Hash +export type Hash = Instruction.Hash +export const Values = Instruction.Values +export type Values = Instruction.Values +export const Delta = Instruction.Delta +export type Delta = Instruction.Delta -/** Stable namespaced identity for one independently refreshable instruction source. */ -export const Key = Schema.String.check(Schema.isPattern(/^[a-z0-9][a-z0-9._-]*\/[a-z0-9][a-z0-9._/-]*$/)).pipe( - Schema.brand("Instructions.Key"), -) -export type Key = typeof Key.Type +type NonValue = Data.TaggedEnum<{ Unavailable: {}; Removed: {} }> +const NonValue = Data.taggedEnum() -/** Indicates that a source could not be observed without treating it as removed. */ -export const unavailable = Symbol.for("@opencode/Instructions.Unavailable") +/** The read failed temporarily; the stored value stands. */ +export const unavailable = NonValue.Unavailable() export type Unavailable = typeof unavailable -/** Defines one typed source before its value type is hidden by `make`. */ -export interface Source { +/** An observed absence: the source exists but its value is gone. */ +export const removed = NonValue.Removed() +export type Removed = typeof removed + +/** + * One composable instruction source over canonical JSON — the same + * representation that is hashed, stored, and replayed. `make` builds one from + * a typed definition; renderers returning `undefined` skip (undecodable or + * unrenderable historical values). + */ +export interface Source { readonly key: Key - readonly codec: Schema.Codec - readonly load: Effect.Effect - readonly baseline: (current: A) => string - readonly update: (previous: A, current: A) => string - readonly removed?: (previous: A) => string + readonly read: Effect.Effect + readonly initial: (value: Schema.Json) => string | undefined + readonly changed: (previous: Schema.Json, current: Schema.Json) => string | undefined + readonly removed: (previous: Schema.Json) => string | undefined } -const InstructionsTypeId: unique symbol = Symbol.for("@opencode/Instructions") - -/** Opaque carrier for composable instruction sources. */ -export interface Instructions { - readonly [InstructionsTypeId]: ReadonlyArray +export declare namespace Source { + /** The typed definition supplied when constructing a source. */ + export interface Definition { + readonly key: Key + readonly codec: Schema.Codec + readonly read: Effect.Effect + readonly render: { + readonly initial: (current: A) => string + readonly changed: (previous: A, current: A) => string + readonly removed?: (previous: A) => string + } + } } -/** The value last applied to the model for one admitted source. */ -export const AppliedSource = Schema.Struct({ - value: Schema.Json, - removed: Schema.optional(Schema.NonEmptyString), -}) -export type AppliedSource = typeof AppliedSource.Type +/** Ordered sources; identical values render identical bytes. */ +export type Instructions = ReadonlyArray -/** Durable record of what the model currently believes, per source. */ -export const Applied = Schema.Record(Key, AppliedSource) -export type Applied = Readonly> +export type ReadResult = ReadonlyArray<{ + readonly key: Key + readonly value: Schema.Json | Unavailable | Removed +}> -/** A rendered baseline together with the applied values it was rendered from. */ -export interface Baseline { - readonly text: string - readonly applied: Applied +export interface Admission { + readonly delta: Delta + readonly blobs: Readonly> } -export interface Updated { - readonly _tag: "Updated" - readonly text: string - readonly applied: Applied -} - -export type ReconcileResult = { readonly _tag: "Unchanged" } | Updated - export class InitializationBlocked extends Schema.TaggedErrorClass()( "Instructions.InitializationBlocked", { keys: Schema.Array(Key) }, @@ -92,71 +82,140 @@ export class DuplicateKeyError extends Schema.TaggedErrorClass - /** Restates the model's belief from a last-applied value when the source cannot be observed. */ - readonly recall: (stored: AppliedSource) => string | undefined -} +export const empty: Instructions = [] -interface Observed { - readonly applied: AppliedSource - readonly baseline: () => string - /** `undefined` means unchanged. An undecodable previous value re-renders the baseline (treat-as-new). */ - readonly update: (previous: AppliedSource) => string | undefined -} - -interface Entry { - readonly key: Key - readonly recall: PackedSource["recall"] - readonly observed: Observed | Unavailable -} - -/** The identity instruction set. */ -export const empty = instructions([]) - -/** Closes a typed source into instructions that compose with differently typed sources. */ -export function make(source: Source): Instructions { +/** Closes a typed definition into one `Source`, so differently typed sources compose. */ +export function make(source: Source.Definition): Instructions { const decode = Schema.decodeUnknownOption(source.codec) const encode = Schema.encodeSync(source.codec) - const equivalent = Schema.toEquivalence(source.codec) - const baseline = (value: A) => requireText(source.key, "baseline", source.baseline(value)) - return instructions([ + const initial = (value: A) => requireText(source.key, "initial", source.render.initial(value)) + const decodeValue = (value: Schema.Json) => Option.getOrUndefined(decode(value)) + return [ { key: source.key, - recall: (stored) => - Option.match(decode(stored.value), { - onNone: () => undefined, - onSome: baseline, - }), - load: source.load.pipe( + read: source.read.pipe( Effect.map((value) => { - if (isUnavailable(value)) return value - return { - applied: { - value: encode(value), - ...(source.removed ? { removed: requireText(source.key, "removal", source.removed(value)) } : {}), - }, - baseline: () => baseline(value), - update: (previous) => - Option.match(decode(previous.value), { - onNone: () => baseline(value), - onSome: (decoded) => - equivalent(decoded, value) - ? undefined - : requireText(source.key, "update", source.update(decoded, value)), - }), - } satisfies Observed + if (isUnavailable(value)) return unavailable + if (isRemoved(value)) return removed + return encode(value) }), ), + initial: (value) => { + const decoded = decodeValue(value) + return decoded === undefined ? undefined : initial(decoded) + }, + changed: (previous, current) => { + const before = decodeValue(previous) + const after = decodeValue(current) + if (after === undefined) return undefined + if (before === undefined) return initial(after) + return requireText(source.key, "changed", source.render.changed(before, after)) + }, + removed: (previous) => { + const decoded = decodeValue(previous) + return decoded === undefined || source.render.removed === undefined + ? undefined + : requireText(source.key, "removed", source.render.removed(decoded)) + }, }, - ]) + ] +} + +export function combine(values: ReadonlyArray): Instructions { + const sources = values.flat() + const keys = new Set() + for (const source of sources) { + if (keys.has(source.key)) throw new DuplicateKeyError({ key: source.key }) + keys.add(source.key) + } + return sources +} + +export function read(value: Instructions): Effect.Effect { + return Effect.forEach( + value, + (source) => source.read.pipe(Effect.map((observed) => ({ key: source.key, value: observed }))), + { concurrency: "unbounded" }, + ) +} + +export function diff(observed: ReadResult, previous?: Values): Effect.Effect { + const blocked = previous ? [] : observed.flatMap((entry) => (isUnavailable(entry.value) ? [entry.key] : [])) + if (blocked.length > 0) return Effect.fail(new InitializationBlocked({ keys: blocked })) + const delta: Record = {} + const blobs: Record = {} + for (const entry of observed) { + if (isUnavailable(entry.value)) continue + if (isRemoved(entry.value)) { + if (previous && Object.hasOwn(previous, entry.key)) delta[entry.key] = Instruction.removed + continue + } + const next = hash(entry.value) + if (previous?.[entry.key] === next) continue + delta[entry.key] = next + blobs[next] = entry.value + } + return Effect.succeed({ delta, blobs }) +} + +export function renderInitial(value: Instructions, values: Readonly>) { + return render( + value.flatMap((source) => { + if (!Object.hasOwn(values, source.key)) return [] + const text = source.initial(values[source.key]) + return text === undefined ? [] : [text] + }), + ) +} + +export function renderUpdate( + value: Instructions, + previous: Readonly>, + delta: Readonly>>, +) { + return render( + value.flatMap((source) => { + if (!Object.hasOwn(delta, source.key)) return [] + const current = delta[source.key] + if (Option.isNone(current)) { + if (!Object.hasOwn(previous, source.key)) return [] + const text = source.removed(previous[source.key]) + return text === undefined ? [] : [text] + } + const next = current.value + const text = Object.hasOwn(previous, source.key) + ? source.changed(previous[source.key], next) + : source.initial(next) + return text === undefined ? [] : [text] + }), + ) +} + +export function hash(value: Schema.Json) { + return Hash.make(createHash("sha256").update(canonical(value)).digest("hex")) +} + +export function applyDelta( + values: Readonly>, + delta: Readonly>>, +): Readonly> { + const result: Record = { ...values } + for (const [key, value] of Object.entries(delta)) { + if (Option.isNone(value)) delete result[key] + else result[key] = value.value + } + return result +} + +export function applyHashDelta(values: Values, delta: Delta): Values { + const result: Record = { ...values } + for (const [key, value] of Object.entries(delta)) { + if (value === Instruction.removed) delete result[key] + else result[key] = value + } + return result } -/** - * Keyed three-way diff for list-shaped sources rendering delta updates. - * `changed` compares two values sharing a key; entries equal under it are dropped. - */ export function diffByKey( previous: ReadonlyArray, current: ReadonlyArray, @@ -179,129 +238,31 @@ export function diffByKey( } } -/** Combines instructions in order and rejects duplicate source keys immediately. */ -export function combine(values: ReadonlyArray): Instructions { - const sources = values.flatMap((value) => value[InstructionsTypeId]) - assertUniqueKeys(sources) - return instructions(sources) -} - -const observe = (value: Instructions) => - Effect.forEach( - value[InstructionsTypeId], - (source) => - source.load.pipe(Effect.map((observed): Entry => ({ key: source.key, recall: source.recall, observed }))), - { concurrency: "unbounded" }, - ) - -/** Creates the first baseline. Blocks rather than admit a baseline missing an unobservable source. */ -export function initialize(value: Instructions): Effect.Effect { - return observe(value).pipe( - Effect.flatMap((entries) => { - const blocked = entries.flatMap((entry) => (entry.observed === unavailable ? [entry.key] : [])) - if (blocked.length > 0) return new InitializationBlocked({ keys: blocked }) - const parts: string[] = [] - const applied: Record = {} - for (const entry of entries) { - if (entry.observed === unavailable) continue - parts.push(entry.observed.baseline()) - applied[entry.key] = entry.observed.applied - } - return Effect.succeed({ text: render(parts), applied }) - }), - ) -} - -/** Narrates drift between current source values and the model's beliefs. Never rewrites the baseline. */ -export function reconcile(value: Instructions, previous: Applied): Effect.Effect { - return observe(value).pipe( - Effect.map((entries): ReconcileResult => { - const updates: string[] = [] - const applied: Record = {} - for (const entry of entries) { - const stored = get(previous, entry.key) - if (entry.observed === unavailable) { - // The prior belief stands while the source cannot be observed. - if (stored) applied[entry.key] = stored - continue - } - if (!stored) { - updates.push(entry.observed.baseline()) - applied[entry.key] = entry.observed.applied - continue - } - const text = entry.observed.update(stored) - if (text === undefined) { - applied[entry.key] = stored - continue - } - updates.push(text) - applied[entry.key] = entry.observed.applied - } - const keys = new Set(entries.map((entry) => entry.key)) - for (const key of Object.keys(previous).sort()) { - if (keys.has(key)) continue - const removed = previous[key].removed - // An unannounced removal retains the belief; it clears at the next rebaseline. - if (removed === undefined) applied[key] = previous[key] - else updates.push(removed) - } - if (updates.length === 0) return { _tag: "Unchanged" } - return { _tag: "Updated", text: render(updates), applied } - }), - ) -} - -/** Rebuilds the baseline, restating unobservable sources from the model's last-applied beliefs. */ -export function rebaseline(value: Instructions, previous: Applied): Effect.Effect { - return observe(value).pipe( - Effect.map((entries): Baseline => { - const parts: string[] = [] - const applied: Record = {} - for (const entry of entries) { - if (entry.observed !== unavailable) { - parts.push(entry.observed.baseline()) - applied[entry.key] = entry.observed.applied - continue - } - const stored = get(previous, entry.key) - if (!stored) continue - const text = entry.recall(stored) - // An undecodable belief cannot be restated; the source re-announces when observable again. - if (text === undefined) continue - parts.push(text) - applied[entry.key] = stored - } - return { text: render(parts), applied } - }), - ) -} - -function instructions(sources: ReadonlyArray): Instructions { - return { [InstructionsTypeId]: sources } -} - function render(parts: ReadonlyArray) { return parts.join("\n\n") } -function get(applied: Applied, key: Key) { - return Object.hasOwn(applied, key) ? applied[key] : undefined -} - +// Reference-equality guards: `A` in a typed source may itself be JSON shaped +// like these singletons, so identity, never structure, discriminates. function isUnavailable(value: unknown): value is Unavailable { return value === unavailable } +function isRemoved(value: unknown): value is Removed { + return value === removed +} + +function canonical(value: Schema.Json): string { + if (Array.isArray(value)) return `[${value.map(canonical).join(",")}]` + if (value !== null && typeof value === "object") + return `{${Object.entries(value) + .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)) + .map(([key, entry]) => `${JSON.stringify(key)}:${canonical(entry)}`) + .join(",")}}` + return JSON.stringify(value) +} + function requireText(key: Key, kind: string, text: string) { if (text.length === 0) throw new Error(`Instruction source ${key} rendered an empty ${kind}`) return text } - -function assertUniqueKeys(sources: ReadonlyArray) { - const keys = new Set() - for (const source of sources) { - if (keys.has(source.key)) throw new DuplicateKeyError({ key: source.key }) - keys.add(source.key) - } -} diff --git a/packages/core/src/mcp/guidance.ts b/packages/core/src/mcp/guidance.ts index 4f2c13b72c..4be0d412f5 100644 --- a/packages/core/src/mcp/guidance.ts +++ b/packages/core/src/mcp/guidance.ts @@ -70,8 +70,19 @@ export const layer = Layer.effect( load: Effect.fn("McpGuidance.load")(function* (selection) { const agent = selection.info if (!agent) return Instructions.empty + const source = (value: ReadonlyArray | Instructions.Removed) => + Instructions.make>({ + key: Instructions.Key.make("core/mcp-guidance"), + codec: Schema.toCodecJson(Schema.Array(Summary)), + read: Effect.succeed(value), + render: { + initial: render, + changed: update, + removed: () => "MCP server instructions are no longer available.", + }, + }) if (Flag.CODEMODE_ENABLED && PermissionV2.evaluate("execute", "*", agent.permissions).effect === "deny") - return Instructions.empty + return source(Instructions.removed) const [instructions, tools] = yield* Effect.all([mcp.instructions(), mcp.tools()], { concurrency: "unbounded", }) @@ -88,15 +99,8 @@ export const layer = Layer.effect( ) }) .map((item) => ({ server: item.server, instructions: item.instructions })) - if (visible.length === 0) return Instructions.empty - return Instructions.make({ - key: Instructions.Key.make("core/mcp-guidance"), - codec: Schema.toCodecJson(Schema.Array(Summary)), - load: Effect.succeed(visible), - baseline: render, - update, - removed: () => "MCP server instructions are no longer available.", - }) + .toSorted((a, b) => a.server.localeCompare(b.server)) + return source(visible.length === 0 ? Instructions.removed : visible) }), }) }), diff --git a/packages/core/src/reference/guidance.ts b/packages/core/src/reference/guidance.ts index 1ddc1a3a8b..a9fd9eb980 100644 --- a/packages/core/src/reference/guidance.ts +++ b/packages/core/src/reference/guidance.ts @@ -74,14 +74,16 @@ const layer = Layer.effect( description: reference.description, })) .toSorted((a, b) => a.name.localeCompare(b.name)) - if (available.length === 0) return Instructions.empty - return Instructions.make({ + return Instructions.make>({ key: Instructions.Key.make("core/reference-guidance"), codec: Schema.toCodecJson(Schema.Array(Summary)), - load: Effect.succeed(available), - baseline: render, - update, - removed: () => "Project reference guidance is no longer available. Do not use previously listed references.", + read: Effect.succeed(available.length === 0 ? Instructions.removed : available), + render: { + initial: render, + changed: update, + removed: () => + "Project reference guidance is no longer available. Do not use previously listed references.", + }, }) }), }) diff --git a/packages/core/src/session.ts b/packages/core/src/session.ts index 468c477dab..a888498c15 100644 --- a/packages/core/src/session.ts +++ b/packages/core/src/session.ts @@ -194,7 +194,7 @@ export interface Interface { */ readonly pending: (sessionID: SessionSchema.ID) => Effect.Effect /** - * Durable, ordered, gap-free session log read. Replays public durable + * Durable, ordered session log read. Replays public durable * session events after the exclusive `after` cursor, emits a `Synced` * marker at the captured replay watermark, then continues live when `follow` * is set. @@ -384,9 +384,11 @@ const layer = Layer.effect( if (input.messageID && !boundary) return yield* new MessageNotFoundError({ sessionID: input.sessionID, messageID: input.messageID }) const sessionID = SessionSchema.ID.create() + const parentSeq = boundary ? boundary.seq - 1 : yield* EventV2.latestSequence(db, parent.id) yield* events.publish(SessionEvent.Forked, { sessionID, parentID: parent.id, + parentSeq, from: input.messageID, }) return yield* result.get(sessionID).pipe(Effect.orDie) diff --git a/packages/core/src/session/compaction.ts b/packages/core/src/session/compaction.ts index a84abe5ce6..d9db191b57 100644 --- a/packages/core/src/session/compaction.ts +++ b/packages/core/src/session/compaction.ts @@ -159,7 +159,7 @@ const select = ( tokens: number, ): { readonly head: string; readonly recent: string } | undefined => { const conversation = messages - .filter((message) => message.type !== "compaction") + .filter((message) => message.type !== "compaction" && message.type !== "system") .flatMap((message) => { const text = serialize(message) return text ? [{ message, text }] : [] diff --git a/packages/core/src/session/history.ts b/packages/core/src/session/history.ts index f2e3122c96..c41671271f 100644 --- a/packages/core/src/session/history.ts +++ b/packages/core/src/session/history.ts @@ -1,10 +1,12 @@ -import { and, asc, desc, eq, gt, gte, ne, or, sql } from "drizzle-orm" +import { and, asc, desc, eq, gte, sql } from "drizzle-orm" import { Effect, Schema } from "effect" import { Database } from "../database/database" import { MessageDecodeError } from "./error" import { SessionMessage } from "./message" import { SessionSchema } from "./schema" -import { InstructionCheckpointTable, SessionMessageTable } from "./sql" +import { Instructions } from "../instructions/index" +import { InstructionState } from "./instruction-state" +import { SessionMessageTable } from "./sql" type DatabaseService = Database.Interface["db"] @@ -31,7 +33,6 @@ const messageRows = Effect.fnUntraced(function* ( db: DatabaseService, sessionID: SessionSchema.ID, compaction: { readonly seq: number } | undefined, - baselineSeq?: number, ) { const rows = yield* db .select() @@ -39,20 +40,7 @@ const messageRows = Effect.fnUntraced(function* ( .where( and( eq(SessionMessageTable.session_id, sessionID), - // Keep system updates visible in the gap between a completed compaction - // and the next prepared step's rebaseline, when their content is not yet - // folded into a new baseline. - compaction - ? or( - gte(SessionMessageTable.seq, compaction.seq), - baselineSeq === undefined - ? undefined - : and(eq(SessionMessageTable.type, "system"), gt(SessionMessageTable.seq, baselineSeq)), - ) - : undefined, - baselineSeq === undefined - ? undefined - : or(ne(SessionMessageTable.type, "system"), gt(SessionMessageTable.seq, baselineSeq)), + compaction ? gte(SessionMessageTable.seq, compaction.seq) : undefined, ), ) .orderBy(asc(SessionMessageTable.seq)) @@ -73,30 +61,32 @@ const decodeMessageRow = (row: typeof SessionMessageTable.$inferSelect) => ) export const load = Effect.fn("SessionHistory.load")(function* (db: DatabaseService, sessionID: SessionSchema.ID) { - const [epoch, compaction] = yield* Effect.all( - [ - db - .select({ baselineSeq: InstructionCheckpointTable.baseline_seq }) - .from(InstructionCheckpointTable) - .where(eq(InstructionCheckpointTable.session_id, sessionID)) - .get() - .pipe(Effect.orDie), - latestCompaction(db, sessionID), - ], - { concurrency: "unbounded" }, + return yield* Effect.forEach( + yield* messageRows(db, sessionID, yield* latestCompaction(db, sessionID)), + decodeMessageRow, ) - return yield* Effect.forEach(yield* messageRows(db, sessionID, compaction, epoch?.baselineSeq), decodeMessageRow) }) export const entriesForRunner = Effect.fn("SessionHistory.entriesForRunner")(function* ( db: DatabaseService, sessionID: SessionSchema.ID, - baselineSeq: number, + instructions: Instructions.Instructions, ) { - const rows = yield* messageRows(db, sessionID, yield* latestCompaction(db, sessionID), baselineSeq) - return yield* Effect.forEach(rows, (row) => - decodeMessageRow(row).pipe(Effect.map((message) => ({ seq: row.seq, message }))), - ) + return yield* db + .transaction(() => + Effect.gen(function* () { + const rows = yield* messageRows(db, sessionID, yield* latestCompaction(db, sessionID)) + const messages = yield* Effect.forEach(rows, (row) => + decodeMessageRow(row).pipe(Effect.map((message) => ({ seq: row.seq, message }))), + ) + const assembled = yield* InstructionState.assemble(db, sessionID, instructions) + return { + initial: assembled.initial, + entries: [...messages, ...assembled.updates].toSorted((a, b) => a.seq - b.seq), + } + }), + ) + .pipe(Effect.orDie) }) /** Returns the session's sole user message, or `undefined` once a second one exists. */ diff --git a/packages/core/src/session/instruction-checkpoint.ts b/packages/core/src/session/instruction-checkpoint.ts deleted file mode 100644 index 47cd006d52..0000000000 --- a/packages/core/src/session/instruction-checkpoint.ts +++ /dev/null @@ -1,130 +0,0 @@ -export * as InstructionCheckpoint from "./instruction-checkpoint" - -import { eq } from "drizzle-orm" -import { Effect, Option, Schema } from "effect" -import type { Database } from "../database/database" -import { EventV2 } from "../event" -import { Instructions } from "../instructions/index" -import { SessionEvent } from "./event" -import { SessionHistory } from "./history" -import { SessionSchema } from "./schema" -import { InstructionCheckpointTable } from "./sql" - -type DatabaseService = Database.Interface["db"] - -const decodeApplied = Schema.decodeUnknownOption(Instructions.Applied) - -/** - * Loads or creates the session's durable instruction checkpoint, narrating any - * drift since the model was last told as a chronological update. Completed - * compaction rebaselines; nothing else rewrites the baseline. Runs before - * input promotion so a blocked first step leaves pending inputs untouched. - */ -export const prepare = Effect.fn("InstructionCheckpoint.prepare")(function* ( - db: DatabaseService, - events: EventV2.Interface, - instructions: Effect.Effect, - sessionID: SessionSchema.ID, -) { - const [value, stored, compaction] = yield* Effect.all( - [instructions, find(db, sessionID), SessionHistory.latestCompaction(db, sessionID)], - { concurrency: "unbounded" }, - ) - if (!stored) { - const baseline = yield* Instructions.initialize(value) - const baselineSeq = yield* insert(db, sessionID, baseline) - return { baseline: baseline.text, baselineSeq } - } - - // The applied record is comparison state only; an undecodable one heals by - // treating every source as new, re-announcing baselines as updates. - const applied = Option.getOrElse(decodeApplied(stored.snapshot), () => ({})) - if (compaction !== undefined && compaction.seq > stored.baseline_seq) { - const baseline = yield* Instructions.rebaseline(value, applied) - yield* rewrite(db, sessionID, compaction.seq, baseline) - return { baseline: baseline.text, baselineSeq: compaction.seq } - } - const result = yield* Instructions.reconcile(value, applied) - if (result._tag === "Unchanged") return { baseline: stored.baseline, baselineSeq: stored.baseline_seq } - - yield* events.publish( - SessionEvent.InstructionsUpdated, - { sessionID, text: result.text }, - { commit: () => advance(db, sessionID, result.applied).pipe(Effect.orDie) }, - ) - return { baseline: stored.baseline, baselineSeq: stored.baseline_seq } -}) - -export const reset = Effect.fn("InstructionCheckpoint.reset")(function* ( - db: DatabaseService, - sessionID: SessionSchema.ID, -) { - yield* db - .delete(InstructionCheckpointTable) - .where(eq(InstructionCheckpointTable.session_id, sessionID)) - .run() - .pipe(Effect.orDie) -}) - -const find = Effect.fnUntraced(function* (db: DatabaseService, sessionID: SessionSchema.ID) { - return yield* db - .select() - .from(InstructionCheckpointTable) - .where(eq(InstructionCheckpointTable.session_id, sessionID)) - .get() - .pipe(Effect.orDie) -}) - -const insert = Effect.fnUntraced(function* ( - db: DatabaseService, - sessionID: SessionSchema.ID, - baseline: Instructions.Baseline, -) { - const baselineSeq = yield* EventV2.latestSequence(db, sessionID) - yield* db - .insert(InstructionCheckpointTable) - .values({ - session_id: sessionID, - baseline: baseline.text, - snapshot: baseline.applied, - baseline_seq: baselineSeq, - }) - .run() - .pipe(Effect.orDie) - return baselineSeq -}) - -const rewrite = Effect.fnUntraced(function* ( - db: DatabaseService, - sessionID: SessionSchema.ID, - baselineSeq: number, - baseline: Instructions.Baseline, -) { - const updated = yield* db - .update(InstructionCheckpointTable) - .set({ - baseline: baseline.text, - snapshot: baseline.applied, - baseline_seq: baselineSeq, - }) - .where(eq(InstructionCheckpointTable.session_id, sessionID)) - .returning({ sessionID: InstructionCheckpointTable.session_id }) - .get() - .pipe(Effect.orDie) - if (!updated) return yield* Effect.die(new Error("Instruction checkpoint not found")) -}) - -const advance = Effect.fnUntraced(function* ( - db: DatabaseService, - sessionID: SessionSchema.ID, - applied: Instructions.Applied, -) { - const updated = yield* db - .update(InstructionCheckpointTable) - .set({ snapshot: applied }) - .where(eq(InstructionCheckpointTable.session_id, sessionID)) - .returning({ sessionID: InstructionCheckpointTable.session_id }) - .get() - .pipe(Effect.orDie) - if (!updated) return yield* Effect.die(new Error("Instruction checkpoint not found")) -}) diff --git a/packages/core/src/session/instruction-entry.ts b/packages/core/src/session/instruction-entry.ts index a74a174d12..43ccb99b24 100644 --- a/packages/core/src/session/instruction-entry.ts +++ b/packages/core/src/session/instruction-entry.ts @@ -1,6 +1,6 @@ export * as InstructionEntry from "./instruction-entry" -import { and, asc, eq } from "drizzle-orm" +import { and, asc, eq, isNotNull, isNull, ne, or } from "drizzle-orm" import { Context, Effect, Layer, Schema } from "effect" import { InstructionEntry } from "@opencode-ai/schema/instruction-entry" import { Database } from "../database/database" @@ -13,6 +13,8 @@ export const Key = InstructionEntry.Key export type Key = typeof Key.Type export const Info = InstructionEntry.Info export type Info = typeof Info.Type +export const MaxValueBytes = InstructionEntry.MaxValueBytes +export const ValueTooLargeError = InstructionEntry.ValueTooLargeError export interface Interface { readonly list: (sessionID: SessionSchema.ID) => Effect.Effect> @@ -20,7 +22,7 @@ export interface Interface { readonly sessionID: SessionSchema.ID readonly key: Key readonly value: Schema.Json - }) => Effect.Effect + }) => Effect.Effect readonly remove: (input: { readonly sessionID: SessionSchema.ID; readonly key: Key }) => Effect.Effect /** Produces one Instructions source per stored entry, keyed `api/`. */ readonly load: (sessionID: SessionSchema.ID) => Effect.Effect @@ -35,18 +37,20 @@ const renderBlock = (key: Key, value: Schema.Json) => // Rendering stays mechanism-neutral: the model sees session context, not how // it was attached. Only chronological updates and removals carry narration. -const source = (entry: Info) => - Instructions.make({ +const source = (entry: Info & { readonly removed: boolean }) => + Instructions.make({ key: Instructions.Key.make(`api/${entry.key}`), codec: Schema.toCodecJson(Schema.Json), - load: Effect.succeed(entry.value), - baseline: (value) => renderBlock(entry.key, value), - update: (_previous, value) => - [ - `The context under "${entry.key}" changed and supersedes the previous value:`, - renderBlock(entry.key, value), - ].join("\n"), - removed: () => `The context under "${entry.key}" no longer applies. Disregard it.`, + read: Effect.succeed(entry.removed ? Instructions.removed : entry.value), + render: { + initial: (value) => renderBlock(entry.key, value), + changed: (_previous, value) => + [ + `The context under "${entry.key}" changed and supersedes the previous value:`, + renderBlock(entry.key, value), + ].join("\n"), + removed: () => `The context under "${entry.key}" no longer applies. Disregard it.`, + }, }) const layer = Layer.effect( @@ -54,15 +58,27 @@ const layer = Layer.effect( Effect.gen(function* () { const { db } = yield* Database.Service - const list = Effect.fn("InstructionEntry.list")(function* (sessionID: SessionSchema.ID) { - const rows = yield* db - .select() + const rows = Effect.fnUntraced(function* (sessionID: SessionSchema.ID, includeRemoved: boolean) { + return yield* db + .select({ + key: InstructionEntryTable.key, + value: InstructionEntryTable.value, + removed: InstructionEntryTable.removed, + }) .from(InstructionEntryTable) - .where(eq(InstructionEntryTable.session_id, sessionID)) + .where( + and( + eq(InstructionEntryTable.session_id, sessionID), + includeRemoved ? undefined : eq(InstructionEntryTable.removed, false), + ), + ) .orderBy(asc(InstructionEntryTable.key)) .all() .pipe(Effect.orDie) - return rows.map((row) => ({ key: row.key, value: row.value })) + }) + + const list = Effect.fn("InstructionEntry.list")(function* (sessionID: SessionSchema.ID) { + return (yield* rows(sessionID, false)).map((row) => ({ key: row.key, value: row.value })) }) const put = Effect.fn("InstructionEntry.put")(function* (input: { @@ -70,12 +86,24 @@ const layer = Layer.effect( readonly key: Key readonly value: Schema.Json }) { + const actualBytes = Buffer.byteLength(JSON.stringify(input.value), "utf8") + if (actualBytes > MaxValueBytes) + yield* new ValueTooLargeError({ + actualBytes, + maxBytes: MaxValueBytes, + message: `Instruction entry value is ${actualBytes} bytes; the limit is ${MaxValueBytes} bytes`, + }) + const changed = + input.value === null + ? isNotNull(InstructionEntryTable.value) + : or(isNull(InstructionEntryTable.value), ne(InstructionEntryTable.value, input.value)) yield* db .insert(InstructionEntryTable) - .values({ session_id: input.sessionID, key: input.key, value: input.value }) + .values({ session_id: input.sessionID, key: input.key, value: input.value, removed: false }) .onConflictDoUpdate({ target: [InstructionEntryTable.session_id, InstructionEntryTable.key], - set: { value: input.value, time_updated: Date.now() }, + set: { value: input.value, removed: false, time_updated: Date.now() }, + setWhere: or(eq(InstructionEntryTable.removed, true), changed), }) .run() .pipe(Effect.orDie) @@ -86,15 +114,21 @@ const layer = Layer.effect( readonly key: Key }) { yield* db - .delete(InstructionEntryTable) - .where(and(eq(InstructionEntryTable.session_id, input.sessionID), eq(InstructionEntryTable.key, input.key))) + .update(InstructionEntryTable) + .set({ value: null, removed: true, time_updated: Date.now() }) + .where( + and( + eq(InstructionEntryTable.session_id, input.sessionID), + eq(InstructionEntryTable.key, input.key), + eq(InstructionEntryTable.removed, false), + ), + ) .run() .pipe(Effect.orDie) }) const load = Effect.fn("InstructionEntry.load")(function* (sessionID: SessionSchema.ID) { - const entries = yield* list(sessionID) - return Instructions.combine(entries.map(source)) + return Instructions.combine((yield* rows(sessionID, true)).map(source)) }) return Service.of({ list, put, remove, load }) diff --git a/packages/core/src/session/instruction-state.ts b/packages/core/src/session/instruction-state.ts new file mode 100644 index 0000000000..d20ad87b5b --- /dev/null +++ b/packages/core/src/session/instruction-state.ts @@ -0,0 +1,338 @@ +export * as InstructionState from "./instruction-state" + +import { and, asc, desc, eq, gt, inArray, lte, sql } from "drizzle-orm" +import { DateTime, Effect, Option, Schema } from "effect" +import type { Database } from "../database/database" +import { EventV2 } from "../event" +import { EventTable } from "../event/sql" +import { Instructions } from "../instructions/index" +import { SessionEvent } from "./event" +import { SessionMessage } from "./message" +import { SessionSchema } from "./schema" +import { InstructionBlobTable, InstructionStateTable, SessionTable } from "./sql" + +type DatabaseService = Database.Interface["db"] + +const decodeInstructionsUpdated = Schema.decodeUnknownSync(SessionEvent.InstructionsUpdated.data) + +export const prepare = Effect.fn("InstructionState.prepare")(function* ( + db: DatabaseService, + events: EventV2.Interface, + instructions: Instructions.Instructions, + sessionID: SessionSchema.ID, +) { + const [observed, stored] = yield* Effect.all([Instructions.read(instructions), ensure(db, sessionID)], { + concurrency: "unbounded", + }) + const admission = yield* Instructions.diff(observed, stored?.current_values) + if (!stored || Object.keys(admission.delta).length > 0) { + yield* events.publish( + SessionEvent.InstructionsUpdated, + { sessionID, delta: admission.delta }, + { + commit: () => insertBlobs(db, admission.blobs), + }, + ) + } +}) + +export const apply = Effect.fn("InstructionState.apply")(function* ( + db: DatabaseService, + sessionID: SessionSchema.ID, + seq: number, + delta: Instructions.Delta, +) { + const stored = yield* find(db, sessionID) + const current = Instructions.applyHashDelta(stored?.current_values ?? {}, delta) + if (!stored) { + yield* db + .insert(InstructionStateTable) + .values({ + session_id: sessionID, + epoch_start: seq, + through_seq: seq, + initial_values: current, + current_values: current, + }) + .run() + .pipe(Effect.orDie) + return + } + yield* db + .update(InstructionStateTable) + .set({ through_seq: seq, current_values: current }) + .where(eq(InstructionStateTable.session_id, sessionID)) + .run() + .pipe(Effect.orDie) +}) + +export const advanceEpoch = Effect.fn("InstructionState.advanceEpoch")(function* ( + db: DatabaseService, + sessionID: SessionSchema.ID, + epochStart: number, +) { + yield* db + .update(InstructionStateTable) + .set({ + epoch_start: epochStart, + through_seq: epochStart, + initial_values: sql`${InstructionStateTable.current_values}`, + }) + .where(eq(InstructionStateTable.session_id, sessionID)) + .run() + .pipe(Effect.orDie) +}) + +export const reset = Effect.fn("InstructionState.reset")(function* (db: DatabaseService, sessionID: SessionSchema.ID) { + yield* db + .delete(InstructionStateTable) + .where(eq(InstructionStateTable.session_id, sessionID)) + .run() + .pipe(Effect.orDie) +}) + +export const rebuild = Effect.fn("InstructionState.rebuild")(function* ( + db: DatabaseService, + sessionID: SessionSchema.ID, +) { + const folded = fold(yield* instructionEvents(db, sessionID)) + if (!folded) { + yield* reset(db, sessionID) + return undefined + } + const state = { + session_id: sessionID, + epoch_start: folded.epochStart, + through_seq: folded.throughSeq, + initial_values: folded.initial, + current_values: folded.current, + } + yield* db + .insert(InstructionStateTable) + .values(state) + .onConflictDoUpdate({ + target: InstructionStateTable.session_id, + set: { + epoch_start: folded.epochStart, + through_seq: folded.throughSeq, + initial_values: folded.initial, + current_values: folded.current, + }, + }) + .run() + .pipe(Effect.orDie) + return state +}) + +export const assemble = Effect.fn("InstructionState.assemble")(function* ( + db: DatabaseService, + sessionID: SessionSchema.ID, + instructions: Instructions.Instructions, +) { + const state = yield* find(db, sessionID) + if (!state) return yield* Effect.die(new Error(`Instruction state not found during assembly: ${sessionID}`)) + const rows = yield* instructionUpdatesAfter(db, sessionID, state.epoch_start) + const updates = rows.map((row) => ({ + row, + delta: decodeInstructionsUpdated(row.data).delta, + })) + const blobs = yield* loadBlobs(db, [ + ...Object.values(state.initial_values), + ...updates.flatMap((update) => + Object.values(update.delta).filter((hash): hash is Instructions.Hash => hash !== "removed"), + ), + ]) + const valuesAtStart = dereference(state.initial_values, blobs) + let values = valuesAtStart + const result: Array<{ readonly seq: number; readonly message: SessionMessage.System }> = [] + for (const update of updates) { + const delta = dereferenceDelta(update.delta, blobs) + const text = Instructions.renderUpdate(instructions, values, delta) + if (text.length > 0) + result.push({ + seq: update.row.seq, + message: SessionMessage.System.make({ + id: SessionMessage.ID.fromEvent(EventV2.ID.make(update.row.id)), + type: "system", + text, + time: { created: DateTime.makeUnsafe(update.row.created) }, + }), + }) + values = Instructions.applyDelta(values, delta) + } + return { initial: Instructions.renderInitial(instructions, valuesAtStart), updates: result } +}) + +const find = Effect.fnUntraced(function* (db: DatabaseService, sessionID: SessionSchema.ID) { + return yield* db + .select() + .from(InstructionStateTable) + .where(eq(InstructionStateTable.session_id, sessionID)) + .get() + .pipe(Effect.orDie) +}) + +const ensure = Effect.fnUntraced(function* (db: DatabaseService, sessionID: SessionSchema.ID) { + const stored = yield* find(db, sessionID) + if (!stored) return yield* rebuild(db, sessionID) + const latest = yield* db + .select({ seq: EventTable.seq }) + .from(EventTable) + .where(and(eq(EventTable.aggregate_id, sessionID), inArray(EventTable.type, relevantEventTypes))) + .orderBy(desc(EventTable.seq)) + .limit(1) + .get() + .pipe(Effect.orDie) + if (!latest || latest.seq <= stored.through_seq) return stored + return yield* rebuild(db, sessionID) +}) + +const insertBlobs = Effect.fnUntraced(function* (db: DatabaseService, blobs: Readonly>) { + const rows = Object.entries(blobs).map(([hash, value]) => ({ hash: Instructions.Hash.make(hash), value })) + if (rows.length === 0) return + yield* db.insert(InstructionBlobTable).values(rows).onConflictDoNothing().run().pipe(Effect.orDie) +}) + +const loadBlobs = Effect.fnUntraced(function* (db: DatabaseService, values: ReadonlyArray) { + const hashes = [...new Set(values)] + const batches = Array.from({ length: Math.ceil(hashes.length / 500) }, (_, index) => + hashes.slice(index * 500, (index + 1) * 500), + ) + const rows = (yield* Effect.forEach( + batches, + (batch) => + db.select().from(InstructionBlobTable).where(inArray(InstructionBlobTable.hash, batch)).all().pipe(Effect.orDie), + { concurrency: 4 }, + )).flat() + const blobs = new Map(rows.map((row) => [row.hash, row.value])) + for (const hash of hashes) { + if (!blobs.has(hash)) return yield* Effect.die(new Error(`Instruction blob not found: ${hash}`)) + } + return blobs +}) + +function dereference(values: Instructions.Values, blobs: ReadonlyMap) { + return Object.fromEntries(Object.entries(values).map(([key, hash]) => [key, requireBlob(blobs, hash)])) as Readonly< + Record + > +} + +function dereferenceDelta(delta: Instructions.Delta, blobs: ReadonlyMap) { + return Object.fromEntries( + Object.entries(delta).map(([key, hash]) => [ + key, + hash === "removed" ? Option.none() : Option.some(requireBlob(blobs, hash)), + ]), + ) as Readonly>> +} + +function requireBlob(blobs: ReadonlyMap, hash: Instructions.Hash) { + const value = blobs.get(hash) + if (value === undefined) throw new Error(`Instruction blob not found: ${hash}`) + return value +} + +const instructionEventType = EventV2.versionedType( + SessionEvent.InstructionsUpdated.type, + SessionEvent.InstructionsUpdated.durable.version, +) +const compactionEventType = EventV2.versionedType( + SessionEvent.Compaction.Ended.type, + SessionEvent.Compaction.Ended.durable.version, +) +const movedEventType = EventV2.versionedType(SessionEvent.Moved.type, SessionEvent.Moved.durable.version) +const revertedEventType = EventV2.versionedType( + SessionEvent.RevertEvent.Committed.type, + SessionEvent.RevertEvent.Committed.durable.version, +) +const relevantEventTypes = [instructionEventType, compactionEventType, movedEventType, revertedEventType] + +type InstructionEventRow = typeof EventTable.$inferSelect + +const instructionEvents = Effect.fnUntraced(function* ( + db: DatabaseService, + sessionID: SessionSchema.ID, +): Effect.fn.Return> { + return yield* eventRows(db, sessionID, relevantEventTypes) +}) + +const instructionUpdatesAfter = Effect.fnUntraced(function* ( + db: DatabaseService, + sessionID: SessionSchema.ID, + after: number, +) { + return yield* eventRows(db, sessionID, [instructionEventType], after) +}) + +const eventRows = Effect.fnUntraced(function* ( + db: DatabaseService, + sessionID: SessionSchema.ID, + types: ReadonlyArray, + after?: number, +): Effect.fn.Return> { + const segments = (yield* lineage(db, sessionID)).filter( + (segment) => after === undefined || segment.through === undefined || segment.through > after, + ) + return (yield* Effect.forEach(segments, (segment) => + db + .select() + .from(EventTable) + .where( + and( + eq(EventTable.aggregate_id, segment.sessionID), + inArray(EventTable.type, types), + segment.through === undefined ? undefined : lte(EventTable.seq, segment.through), + after === undefined ? undefined : gt(EventTable.seq, after), + ), + ) + .orderBy(asc(EventTable.seq)) + .all() + .pipe(Effect.orDie), + )).flat() +}) + +const lineage = Effect.fnUntraced(function* ( + db: DatabaseService, + sessionID: SessionSchema.ID, + through?: number, +): Effect.fn.Return> { + const session = yield* db + .select({ parentID: SessionTable.fork_session_id, forkSeq: SessionTable.fork_seq }) + .from(SessionTable) + .where(eq(SessionTable.id, sessionID)) + .get() + .pipe(Effect.orDie) + const inherited = + session?.parentID && session.forkSeq !== null + ? yield* lineage( + db, + session.parentID, + through === undefined ? session.forkSeq : Math.min(session.forkSeq, through), + ) + : [] + return [...inherited, { sessionID, ...(through === undefined ? {} : { through }) }] +}) + +function fold(rows: ReadonlyArray) { + return rows.reduce< + | { + readonly epochStart: number + readonly throughSeq: number + readonly initial: Instructions.Values + readonly current: Instructions.Values + } + | undefined + >((state, row) => { + if (row.type === movedEventType || row.type === revertedEventType) return undefined + if (row.type === compactionEventType) + return state + ? { epochStart: row.seq, throughSeq: row.seq, initial: state.current, current: state.current } + : undefined + if (row.type !== instructionEventType) return state + const delta = decodeInstructionsUpdated(row.data).delta + const current = Instructions.applyHashDelta(state?.current ?? {}, delta) + return state + ? { ...state, throughSeq: row.seq, current } + : { epochStart: row.seq, throughSeq: row.seq, initial: current, current } + }, undefined) +} diff --git a/packages/core/src/session/message-updater.ts b/packages/core/src/session/message-updater.ts index 5d5ac25668..2f7f1d3d12 100644 --- a/packages/core/src/session/message-updater.ts +++ b/packages/core/src/session/message-updater.ts @@ -178,16 +178,7 @@ export function update(adapter: Adapter, event: SessionEvent.Event) { "session.execution.succeeded": () => clearCurrentRetry, "session.execution.failed": () => clearCurrentRetry, "session.execution.interrupted": () => clearCurrentRetry, - "session.instructions.updated": (event) => - adapter.appendMessage( - SessionMessage.System.make({ - id: SessionMessage.ID.fromEvent(event.id), - type: "system", - text: event.data.text, - metadata: event.metadata, - time: { created: event.created }, - }), - ), + "session.instructions.updated": () => Effect.void, "session.synthetic": (event) => { return adapter.appendMessage( SessionMessage.Synthetic.make({ diff --git a/packages/core/src/session/projector.ts b/packages/core/src/session/projector.ts index 4d302108b5..70362cc89b 100644 --- a/packages/core/src/session/projector.ts +++ b/packages/core/src/session/projector.ts @@ -13,22 +13,18 @@ import { SessionMessage } from "./message" import { SessionMessageUpdater } from "./message-updater" import { SessionPending } from "./pending" import { WorkspaceV2 } from "../workspace" -import { InstructionCheckpoint } from "./instruction-checkpoint" -import { - MessageTable, - PartTable, - InstructionCheckpointTable, - SessionPendingTable, - SessionMessageTable, - SessionTable, -} from "./sql" +import { InstructionState } from "./instruction-state" +import { MessageTable, PartTable, SessionPendingTable, SessionMessageTable, SessionTable } from "./sql" import type { DeepMutable } from "../schema" import { Slug } from "../util/slug" import { Money } from "@opencode-ai/schema/money" type DatabaseService = Database.Interface["db"] type CurrentDurableEvent = Extract -type MessageEvent = Exclude +type MessageEvent = Exclude< + CurrentDurableEvent, + typeof SessionEvent.Forked.Type | typeof SessionEvent.Deleted.Type | typeof SessionEvent.InstructionsUpdated.Type +> const decodeMessage = Schema.decodeUnknownSync(SessionMessage.Info) const encodeMessage = Schema.encodeSync(SessionMessage.Info) @@ -212,6 +208,7 @@ const projectFork = Effect.fn("SessionProjector.projectFork")(function* ( parent_id: null, fork_session_id: event.data.parentID, fork_message_id: event.data.from, + fork_seq: event.data.parentSeq, project_id: parent.project_id, workspace_id: parent.workspace_id, slug: Slug.create(), @@ -236,23 +233,6 @@ const projectFork = Effect.fn("SessionProjector.projectFork")(function* ( .pipe(Effect.orDie) if (!stored) return yield* Effect.die(new SessionAlreadyProjected()) - // The fork inherits the parent's transcript, so it inherits the context - // checkpoint that transcript was built against: copied message seqs keep - // folding at the same baseline horizon. - const checkpoint = yield* db - .select() - .from(InstructionCheckpointTable) - .where(eq(InstructionCheckpointTable.session_id, event.data.parentID)) - .get() - .pipe(Effect.orDie) - if (checkpoint) { - yield* db - .insert(InstructionCheckpointTable) - .values({ ...checkpoint, session_id: event.data.sessionID }) - .run() - .pipe(Effect.orDie) - } - let cursor = -1 while (copiedSeq !== undefined) { const rows = yield* db @@ -334,7 +314,8 @@ const projectFork = Effect.fn("SessionProjector.projectFork")(function* ( cursor = rows.at(-1)!.seq } - if (copiedSeq !== undefined) yield* EventV2.reserveSequence(db, event.data.sessionID, copiedSeq) + yield* EventV2.reserveSequence(db, event.data.sessionID, event.data.parentSeq) + yield* InstructionState.rebuild(db, event.data.sessionID) }) function run(db: DatabaseService, event: MessageEvent) { @@ -522,7 +503,7 @@ const layer = Layer.effectDiscard( .where(eq(SessionTable.id, event.data.sessionID)) .run() .pipe(Effect.orDie) - yield* InstructionCheckpoint.reset(db, event.data.sessionID) + yield* InstructionState.reset(db, event.data.sessionID) }), ) yield* events.project(SessionV1.Event.Deleted, (event) => @@ -688,7 +669,9 @@ const layer = Layer.effectDiscard( yield* events.project(SessionEvent.Execution.Succeeded, (event) => run(db, event)) yield* events.project(SessionEvent.Execution.Failed, (event) => run(db, event)) yield* events.project(SessionEvent.Execution.Interrupted, (event) => run(db, event)) - yield* events.project(SessionEvent.InstructionsUpdated, (event) => run(db, event)) + yield* events.project(SessionEvent.InstructionsUpdated, (event) => + InstructionState.apply(db, event.data.sessionID, event.durable.seq, event.data.delta), + ) yield* events.project(SessionEvent.Synthetic, (event) => run(db, event)) yield* events.project(SessionEvent.Skill.Activated, (event) => run(db, event)) yield* events.project(SessionEvent.Shell.Started, (event) => run(db, event)) @@ -722,6 +705,7 @@ const layer = Layer.effectDiscard( yield* events.project(SessionEvent.Compaction.Ended, (event) => Effect.gen(function* () { yield* run(db, event) + yield* InstructionState.advanceEpoch(db, event.data.sessionID, event.durable.seq) if (event.durable === undefined) return yield* Effect.die(new Error("Durable Session event is missing aggregate sequence")) if (event.data.reason === "manual") @@ -793,7 +777,7 @@ const layer = Layer.effectDiscard( .where(eq(SessionTable.id, event.data.sessionID)) .run() .pipe(Effect.orDie) - yield* InstructionCheckpoint.reset(db, event.data.sessionID) + yield* InstructionState.reset(db, event.data.sessionID) }), ) yield* events.subscribe([SessionEvent.Step.Ended, SessionEvent.Step.Failed]).pipe( diff --git a/packages/core/src/session/runner/llm.ts b/packages/core/src/session/runner/llm.ts index 0d72d20080..932470d869 100644 --- a/packages/core/src/session/runner/llm.ts +++ b/packages/core/src/session/runner/llm.ts @@ -29,7 +29,7 @@ import { InstructionEntry } from "../instruction-entry" import { QuestionTool } from "../../tool/question" import { ToolRegistry } from "../../tool/registry" import { ToolOutputStore } from "../../tool-output-store" -import { InstructionCheckpoint } from "../instruction-checkpoint" +import { InstructionState } from "../instruction-state" import { SessionCompaction } from "../compaction" import { SessionEvent } from "../event" import { SessionHistory } from "../history" @@ -112,6 +112,8 @@ const layer = Layer.effect( if (!session) return yield* Effect.die(new Error(`Session not found: ${sessionID}`)) return session }) + const isCurrentLocation = (session: SessionSchema.Info) => + session.location.directory === location.directory && session.location.workspaceID === location.workspaceID const failInterruptedTools = Effect.fn("SessionRunner.failInterruptedTools")(function* ( sessionID: SessionSchema.ID, @@ -160,20 +162,15 @@ const layer = Layer.effect( assistantMessageID?: SessionMessage.ID, ) { const session = yield* getSession(sessionID) - if (session.location.directory !== location.directory || session.location.workspaceID !== location.workspaceID) - return yield* Effect.interrupt + if (!isCurrentLocation(session)) return yield* Effect.interrupt yield* plugins.flush const agent = yield* agents.select(session.agent) const agentInfo = agent.info if (!agentInfo) return yield* new AgentNotFoundError({ sessionID: session.id, agent: session.agent ?? agent.id }) // Establish what the model knows before admitting what the user said, so // a blocked first step leaves pending inputs untouched. - const checkpoint = yield* InstructionCheckpoint.prepare( - db, - events, - loadInstructions(agent, session.id), - session.id, - ) + const instructions = yield* loadInstructions(agent, session.id) + yield* InstructionState.prepare(db, events, instructions, session.id) let currentStep = step if (promotion) { let promoted = 0 @@ -187,8 +184,8 @@ const layer = Layer.effect( const resolved = yield* models.resolve(session) const model = resolved.model const providerMetadataKey = model.route.providerMetadataKey ?? model.provider - const entries = yield* SessionHistory.entriesForRunner(db, session.id, checkpoint.baselineSeq) - const context = entries.map((entry) => entry.message) + const history = yield* SessionHistory.entriesForRunner(db, session.id, instructions) + const context = history.entries.map((entry) => entry.message) const compactionInput = { sessionID: session.id, messages: context, model } if (compaction.required(compactionInput) && !(yield* SessionPending.compaction(db, session.id))) { const compacted = yield* compaction.compact(compactionInput) @@ -201,7 +198,7 @@ const layer = Layer.effect( const request = LLM.request({ model, providerOptions: { openai: { promptCacheKey } }, - system: [agentInfo.system ? agentInfo.system : SessionRunnerSystemPrompt.provider(model), checkpoint.baseline] + system: [agentInfo.system ? agentInfo.system : SessionRunnerSystemPrompt.provider(model), history.initial] .filter((part): part is string => part !== undefined && part.length > 0) .map(SystemPart.make), messages: [ diff --git a/packages/core/src/session/sql.ts b/packages/core/src/session/sql.ts index 505ecec383..21d0ab5f26 100644 --- a/packages/core/src/session/sql.ts +++ b/packages/core/src/session/sql.ts @@ -11,7 +11,7 @@ import type { SessionSchema } from "./schema" import type { MessageID, PartID, SessionV1 } from "../v1/session" import { WorkspaceV2 } from "../workspace" import { Timestamps } from "../database/schema.sql" -import type { Instructions } from "../instructions/index" +import type { Instruction } from "@opencode-ai/schema/instruction" import type { Session } from "@opencode-ai/schema/session" import type { SyntheticData, UserData } from "@opencode-ai/schema/session-pending" import type { RevertV1 } from "@opencode-ai/schema/session-revert" @@ -33,6 +33,7 @@ export const SessionTable = sqliteTable( parent_id: text().$type(), fork_session_id: text().$type(), fork_message_id: text().$type(), + fork_seq: integer(), slug: text().notNull(), directory: directoryColumn().notNull(), path: pathColumn(), @@ -159,18 +160,25 @@ export const InstructionEntryTable = sqliteTable( .notNull() .references(() => SessionTable.id, { onDelete: "cascade" }), key: text().notNull(), - value: text({ mode: "json" }).notNull().$type(), + value: text({ mode: "json" }).$type(), + removed: integer({ mode: "boolean" }).notNull().default(false), ...Timestamps, }, (table) => [primaryKey({ columns: [table.session_id, table.key] })], ) -export const InstructionCheckpointTable = sqliteTable("instruction_checkpoint", { +export const InstructionBlobTable = sqliteTable("instruction_blob", { + hash: text().$type().primaryKey(), + value: text({ mode: "json" }).$type(), +}) + +export const InstructionStateTable = sqliteTable("instruction_state", { session_id: text() .$type() .primaryKey() .references(() => SessionTable.id, { onDelete: "cascade" }), - baseline: text().notNull(), - snapshot: text({ mode: "json" }).notNull().$type(), - baseline_seq: integer().notNull(), + epoch_start: integer().notNull(), + through_seq: integer().notNull(), + initial_values: text({ mode: "json" }).notNull().$type(), + current_values: text({ mode: "json" }).notNull().$type(), }) diff --git a/packages/core/src/skill/guidance.ts b/packages/core/src/skill/guidance.ts index 5eae66065b..692d92a3a5 100644 --- a/packages/core/src/skill/guidance.ts +++ b/packages/core/src/skill/guidance.ts @@ -3,7 +3,6 @@ export * as SkillGuidance from "./guidance" import { makeLocationNode } from "../effect/app-node" import { Context, Effect, Layer, Schema } from "effect" import { AgentV2 } from "../agent" -import { PermissionV2 } from "../permission" import { SkillV2 } from "../skill" import { Instructions } from "../instructions/index" @@ -73,8 +72,6 @@ const layer = Layer.effect( const agent = selection.info if (!agent) return Instructions.empty const permitted = SkillV2.available(yield* skills.list(), agent) - if (permitted.length === 0 && PermissionV2.evaluate("skill", "*", agent.permissions).effect === "deny") - return Instructions.empty const available = permitted .flatMap((skill) => skill.description === undefined || skill.autoinvoke === false @@ -82,13 +79,15 @@ const layer = Layer.effect( : [{ id: skill.id, name: skill.name, description: skill.description }], ) .toSorted((a, b) => a.id.localeCompare(b.id)) - return Instructions.make({ + return Instructions.make>({ key: Instructions.Key.make("core/skill-guidance"), codec: Schema.toCodecJson(Schema.Array(Summary)), - load: Effect.succeed(available), - baseline: render, - update, - removed: () => "Skill guidance is no longer available. Do not use any previously listed skill.", + read: Effect.succeed(available.length === 0 ? Instructions.removed : available), + render: { + initial: render, + changed: update, + removed: () => "Skill guidance is no longer available. Do not use any previously listed skill.", + }, }) }), }) diff --git a/packages/core/src/tool/read.ts b/packages/core/src/tool/read.ts index 001a29ba55..b71680142b 100644 --- a/packages/core/src/tool/read.ts +++ b/packages/core/src/tool/read.ts @@ -102,7 +102,7 @@ export const Plugin = { const resolved = yield* fs.resolve(target.canonical) const root = yield* fs.resolve(location.directory) // up() searches its stop directory, so the Location-root AGENTS.md (already - // supplied by the core/instructions baseline) is dropped by the dirname filter. + // supplied by core initial instructions) is dropped by the dirname filter. const discovered = yield* fs.up({ targets: [FILENAME], start: type === "directory" ? resolved : dirname(resolved), diff --git a/packages/core/test/database-migration.test.ts b/packages/core/test/database-migration.test.ts index 436f680cc3..326855ab51 100644 --- a/packages/core/test/database-migration.test.ts +++ b/packages/core/test/database-migration.test.ts @@ -23,6 +23,7 @@ import sessionPendingTableMigration from "@opencode-ai/core/database/migration/2 import renameInstructionsMigration from "@opencode-ai/core/database/migration/20260705180000_rename_instructions" import addSessionForkMigration from "@opencode-ai/core/database/migration/20260706223930_add-session-fork" import timeSuspendedMigration from "@opencode-ai/core/database/migration/20260709163752_time_suspended" +import instructionSyncMigration from "@opencode-ai/core/database/migration/20260710025429_instruction_sync" import { AppNodeBuilder } from "@opencode-ai/core/effect/app-node-builder" import { LayerNode } from "@opencode-ai/core/effect/layer-node" import { EventV2 } from "@opencode-ai/core/event" @@ -359,12 +360,12 @@ describe("DatabaseMigration", () => { ).toEqual({ name: "session_pending" }) expect( yield* db.get(sql`SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'instruction_checkpoint'`), - ).toEqual({ name: "instruction_checkpoint" }) - expect( - yield* db.get( - sql`SELECT name FROM pragma_table_info('instruction_checkpoint') WHERE name IN ('agent', 'replacement_seq', 'revision')`, - ), ).toBeUndefined() + expect( + yield* db.all( + sql`SELECT name FROM sqlite_master WHERE type = 'table' AND name IN ('instruction_blob', 'instruction_state') ORDER BY name`, + ), + ).toEqual([{ name: "instruction_blob" }, { name: "instruction_state" }]) expect(yield* db.get(sql`SELECT count(*) as count FROM migration`)).toEqual({ count: migrations.length }) expect( yield* db.all( @@ -507,6 +508,91 @@ describe("DatabaseMigration", () => { ) }) + test("deletes pre-beta instruction events and projected System messages", async () => { + await run( + Effect.gen(function* () { + const db = yield* makeDb + yield* db.run(sql`CREATE TABLE session (id text PRIMARY KEY, fork_session_id text)`) + yield* db.run( + sql`CREATE TABLE instruction_entry (session_id text NOT NULL, key text NOT NULL, value text NOT NULL, time_created integer NOT NULL, time_updated integer NOT NULL, PRIMARY KEY(session_id, key))`, + ) + yield* db.run(sql`CREATE TABLE instruction_checkpoint (session_id text PRIMARY KEY)`) + yield* db.run(sql`CREATE TABLE event_sequence (aggregate_id text PRIMARY KEY, seq integer NOT NULL)`) + yield* db.run( + sql`CREATE TABLE event (id text PRIMARY KEY, aggregate_id text NOT NULL, seq integer NOT NULL, type text NOT NULL, data text NOT NULL)`, + ) + yield* db.run(sql`CREATE TABLE session_message (id text PRIMARY KEY, type text NOT NULL)`) + yield* db.run(sql`INSERT INTO session VALUES ('ses_test', NULL)`) + yield* db.run( + sql`INSERT INTO event VALUES ('evt_instruction', 'ses_test', 0, 'session.instructions.updated.1', '{"sessionID":"ses_test","text":"changed"}')`, + ) + yield* db.run( + sql`INSERT INTO event VALUES ('evt_other', 'ses_test', 1, 'session.synthetic.1', '{"sessionID":"ses_test","text":"keep"}')`, + ) + yield* db.run( + sql`INSERT INTO session_message VALUES ('msg_instruction', 'system'), ('msg_other', 'system'), ('msg_user', 'user')`, + ) + yield* db.run(sql`INSERT INTO instruction_entry VALUES ('ses_test', 'plan', '"ready"', 1, 2)`) + + yield* DatabaseMigration.applyOnly(db, [instructionSyncMigration]) + + expect(yield* db.all(sql`SELECT id, type FROM event`)).toEqual([ + { id: "evt_other", type: "session.synthetic.1" }, + ]) + expect(yield* db.all(sql`SELECT id, type FROM session_message ORDER BY id`)).toEqual([ + { id: "msg_user", type: "user" }, + ]) + expect(yield* db.get(sql`SELECT * FROM instruction_entry`)).toEqual({ + session_id: "ses_test", + key: "plan", + value: '"ready"', + removed: 0, + time_created: 1, + time_updated: 2, + }) + expect( + yield* db.get(sql`SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'instruction_checkpoint'`), + ).toBeUndefined() + }), + ) + }) + + test("records the authoritative parent sequence on existing forks", async () => { + await run( + Effect.gen(function* () { + const db = yield* makeDb + yield* db.run(sql`CREATE TABLE session (id text PRIMARY KEY, fork_session_id text)`) + yield* db.run( + sql`CREATE TABLE instruction_entry (session_id text NOT NULL, key text NOT NULL, value text NOT NULL, time_created integer NOT NULL, time_updated integer NOT NULL, PRIMARY KEY(session_id, key))`, + ) + yield* db.run(sql`CREATE TABLE instruction_checkpoint (session_id text PRIMARY KEY)`) + yield* db.run(sql`CREATE TABLE session_message (id text PRIMARY KEY, type text NOT NULL)`) + yield* db.run(sql`CREATE TABLE event_sequence (aggregate_id text PRIMARY KEY, seq integer NOT NULL)`) + yield* db.run( + sql`CREATE TABLE event (id text PRIMARY KEY, aggregate_id text NOT NULL, seq integer NOT NULL, type text NOT NULL, data text NOT NULL)`, + ) + yield* db.run(sql`INSERT INTO session VALUES ('ses_child', 'ses_parent')`) + yield* db.run(sql`INSERT INTO event_sequence VALUES ('ses_child', 8)`) + yield* db.run( + sql`INSERT INTO event VALUES ('evt_fork', 'ses_child', 0, 'session.forked.1', '{"sessionID":"ses_child","parentID":"ses_parent"}')`, + ) + yield* db.run( + sql`INSERT INTO event VALUES ('evt_instruction', 'ses_child', 5, 'session.instructions.updated.1', '{"sessionID":"ses_child","text":"changed"}')`, + ) + yield* db.run(sql`INSERT INTO event VALUES ('evt_input', 'ses_child', 6, 'session.input.admitted.1', '{}')`) + + yield* DatabaseMigration.applyOnly(db, [instructionSyncMigration]) + + expect(yield* db.get(sql`SELECT fork_seq FROM session`)).toEqual({ fork_seq: 4 }) + expect(yield* db.get(sql`SELECT type, data FROM event WHERE seq = 0`)).toEqual({ + type: "session.forked.2", + data: '{"sessionID":"ses_child","parentID":"ses_parent","parentSeq":4}', + }) + expect(yield* db.get(sql`SELECT id FROM event WHERE id = 'evt_instruction'`)).toBeUndefined() + }), + ) + }) + test("keeps legacy credential fields nullable", async () => { await run( Effect.gen(function* () { @@ -639,17 +725,14 @@ describe("DatabaseMigration", () => { yield* db.run( sql`INSERT INTO session_message (id, session_id, type, seq, time_created, time_updated, data) VALUES ('projected', 'session', 'user', 9, 1, 1, '{}')`, ) - yield* db.run( - sql`INSERT INTO instruction_checkpoint (session_id, baseline, snapshot, baseline_seq) VALUES ('session', 'baseline', '{}', 9)`, - ) - yield* db.run(sql`ALTER TABLE instruction_checkpoint RENAME TO session_context_epoch`) + yield* db.run(sql`CREATE TABLE session_context_epoch (session_id text PRIMARY KEY)`) // The partial compaction index embeds the qualified table name, so it // must drop before the historical rename dance and recreate after. yield* db.run(sql`DROP INDEX session_pending_session_compaction_idx`) yield* db.run(sql`ALTER TABLE session_pending RENAME TO session_input`) yield* db.run(sql`DELETE FROM migration WHERE id = ${simplifySessionPendingMigration.id}`) yield* DatabaseMigration.applyOnly(db, [simplifySessionPendingMigration]) - yield* db.run(sql`ALTER TABLE session_context_epoch RENAME TO instruction_checkpoint`) + yield* db.run(sql`DROP TABLE session_context_epoch`) yield* db.run(sql`ALTER TABLE session_input RENAME TO session_pending`) yield* db.run( sql`CREATE UNIQUE INDEX session_pending_session_compaction_idx ON session_pending (session_id) WHERE "session_pending"."type" = 'compaction'`, @@ -685,7 +768,7 @@ describe("DatabaseMigration", () => { (SELECT COUNT(*) FROM workspace) AS workspaces, (SELECT COUNT(*) FROM session_pending) AS sessionInputs, (SELECT COUNT(*) FROM session_message) AS sessionMessages, - (SELECT COUNT(*) FROM instruction_checkpoint) AS instructionCheckpoints, + (SELECT COUNT(*) FROM instruction_state) AS instructionStates, (SELECT seq FROM event_sequence WHERE aggregate_id = 'session') AS seq, (SELECT type FROM event WHERE aggregate_id = 'session') AS eventType `), @@ -697,7 +780,7 @@ describe("DatabaseMigration", () => { workspaces: 0, sessionInputs: 0, sessionMessages: 0, - instructionCheckpoints: 0, + instructionStates: 0, seq: 0, eventType: "session.updated.1", }) diff --git a/packages/core/test/event.test.ts b/packages/core/test/event.test.ts index f531ea7990..25030f6e53 100644 --- a/packages/core/test/event.test.ts +++ b/packages/core/test/event.test.ts @@ -55,6 +55,17 @@ const SyncSent = EventV2.durable({ }, }) +const VersionedMessageV1 = EventV2.durable({ + type: "test.versioned", + durable: { version: 1, aggregate: "id" }, + schema: { id: Schema.String }, +}) +const VersionedMessageV2 = EventV2.durable({ + type: "test.versioned", + durable: { version: 2, aggregate: "id" }, + schema: { id: Schema.String }, +}) + const GlobalMessage = EventV2.ephemeral({ type: "test.global", schema: { @@ -722,16 +733,35 @@ describe("EventV2", () => { yield* events.replay({ id: EventV2.ID.create(), created: DateTime.makeUnsafe(0), - type: EventV2.versionedType(SessionEvent.InstructionsUpdated.type, 1), + type: EventV2.versionedType(SessionEvent.InstructionsUpdated.type, 2), seq: 0, aggregateID, - data: { sessionID: aggregateID, text: "context" }, + data: { sessionID: aggregateID, delta: { "core/context": "0".repeat(64) } }, }) expect(received[0]?.created).toEqual(DateTime.makeUnsafe(0)) }), ) + it.effect("dispatches durable projectors by exact event version", () => + Effect.gen(function* () { + const events = yield* EventV2.Service + const aggregateID = Session.ID.create() + const received = new Array() + yield* events.project(VersionedMessageV2, (event) => + Effect.sync(() => { + received.push(event) + }), + ) + + yield* events.publish(VersionedMessageV1, { id: aggregateID }) + yield* events.publish(VersionedMessageV2, { id: aggregateID }) + + expect(received).toHaveLength(1) + expect(received[0]?.durable.version).toBe(EventV2.Version.make(2)) + }), + ) + it.effect("replay defects on unknown event type", () => Effect.gen(function* () { const events = yield* EventV2.Service diff --git a/packages/core/test/instruction-discovery.test.ts b/packages/core/test/instruction-discovery.test.ts index 2477d05124..bc8724a837 100644 --- a/packages/core/test/instruction-discovery.test.ts +++ b/packages/core/test/instruction-discovery.test.ts @@ -9,10 +9,10 @@ import { Global } from "@opencode-ai/core/global" import { InstructionDiscovery } from "@opencode-ai/core/instruction-discovery" import { Location } from "@opencode-ai/core/location" import { AbsolutePath } from "@opencode-ai/core/schema" -import { Instructions } from "@opencode-ai/core/instructions" import { location } from "./fixture/location" import { tmpdir } from "./fixture/tmpdir" import { testEffect } from "./lib/effect" +import { readInitial, readUpdate, state } from "./lib/instructions" const it = testEffect(Layer.empty) @@ -69,7 +69,7 @@ describe("InstructionDiscovery", () => { ), ) - const initialized = yield* Instructions.initialize(yield* load) + const initialized = yield* readInitial(yield* load) expect(initialized.text).toBe( [ `Instructions from: ${globalFile}\nglobal`, @@ -80,29 +80,24 @@ describe("InstructionDiscovery", () => { expect(initialized.text).not.toContain("outside") yield* Effect.promise(() => fs.writeFile(packageFile, "changed")) - expect(yield* Instructions.reconcile(yield* load, initialized.applied)).toMatchObject({ - _tag: "Updated", - text: expect.stringContaining(`Instructions from: ${packageFile}\nchanged`), - }) + expect((yield* readUpdate(yield* load, initialized)).text).toContain( + `Instructions from: ${packageFile}\nchanged`, + ) yield* Effect.promise(() => fs.rm(packageFile)) - const partial = yield* Instructions.reconcile(yield* load, initialized.applied) - expect(partial).toEqual({ - _tag: "Updated", - text: [ + const partial = yield* readUpdate(yield* load, initialized) + expect(partial.text).toBe( + [ "These instructions replace all previously loaded ambient instructions.", `Instructions from: ${globalFile}\nglobal`, `Instructions from: ${projectFile}\nproject`, ].join("\n\n"), - applied: expect.any(Object), - }) + ) yield* Effect.promise(() => Promise.all([fs.rm(globalFile), fs.rm(projectFile)])) - expect(yield* Instructions.reconcile(yield* load, initialized.applied)).toEqual({ - _tag: "Updated", - text: "Previously loaded instructions no longer apply.", - applied: {}, - }) + expect((yield* readUpdate(yield* load, initialized)).text).toBe( + "Previously loaded instructions no longer apply.", + ) }), ), ), @@ -130,7 +125,7 @@ describe("InstructionDiscovery", () => { ), ) - expect((yield* Instructions.initialize(context)).text).toBe(`Instructions from: ${file}\n`) + expect((yield* readInitial(context)).text).toBe(`Instructions from: ${file}\n`) }), ), ), @@ -161,13 +156,9 @@ describe("InstructionDiscovery", () => { ) expect( - yield* Instructions.reconcile(context, { - "core/instructions": { - value: [{ path: "/repo/AGENTS.md", content: "old" }], - removed: "Previously loaded instructions no longer apply.", - }, - }), - ).toEqual({ _tag: "Unchanged" }) + (yield* readUpdate(context, state({ "core/instructions": [{ path: "/repo/AGENTS.md", content: "old" }] }))) + .changed, + ).toBe(false) }), ) @@ -201,13 +192,8 @@ describe("InstructionDiscovery", () => { ) expect( - yield* Instructions.reconcile(context, { - "core/instructions": { - value: [{ path: file, content: "old" }], - removed: "Previously loaded instructions no longer apply.", - }, - }), - ).toEqual({ _tag: "Unchanged" }) + (yield* readUpdate(context, state({ "core/instructions": [{ path: file, content: "old" }] }))).changed, + ).toBe(false) }), ) diff --git a/packages/core/test/instructions/builtins.test.ts b/packages/core/test/instructions/builtins.test.ts index f66ebb1b81..43f2c92aee 100644 --- a/packages/core/test/instructions/builtins.test.ts +++ b/packages/core/test/instructions/builtins.test.ts @@ -6,10 +6,10 @@ import { Location } from "@opencode-ai/core/location" import { FSUtil } from "@opencode-ai/core/fs-util" import { Global } from "@opencode-ai/core/global" import { AbsolutePath } from "@opencode-ai/core/schema" -import { Instructions } from "@opencode-ai/core/instructions" import { InstructionBuiltIns } from "@opencode-ai/core/instructions/builtins" import { location } from "../fixture/location" import { testEffect } from "../lib/effect" +import { readInitial, readUpdate } from "../lib/instructions" const directory = AbsolutePath.make(FSUtil.resolve("/repo/packages/core")) const projectDirectory = AbsolutePath.make(FSUtil.resolve("/repo")) @@ -36,7 +36,7 @@ describe("InstructionBuiltIns", () => { Effect.gen(function* () { yield* TestClock.setTime(timestamp) const context = yield* InstructionBuiltIns.Service - const initialized = yield* Instructions.initialize(yield* context.load()) + const initialized = yield* readInitial(yield* context.load()) expect(initialized.text).toBe( [ @@ -54,19 +54,16 @@ describe("InstructionBuiltIns", () => { }), ) - it.effect("reconciles the date without repeating unchanged environment instructions", () => + it.effect("updates the date without repeating unchanged environment instructions", () => Effect.gen(function* () { yield* TestClock.setTime(timestamp) const context = yield* InstructionBuiltIns.Service - const initialized = yield* Instructions.initialize(yield* context.load()) + const initialized = yield* readInitial(yield* context.load()) yield* TestClock.setTime(timestamp + 24 * 60 * 60 * 1000) - const refreshed = yield* Instructions.reconcile(yield* context.load(), initialized.applied) + const refreshed = yield* readUpdate(yield* context.load(), initialized) - expect(refreshed).toMatchObject({ - _tag: "Updated", - text: `Today's date is now: ${localDate(timestamp + 24 * 60 * 60 * 1000)}`, - }) + expect(refreshed.text).toBe(`Today's date is now: ${localDate(timestamp + 24 * 60 * 60 * 1000)}`) }), ) @@ -74,10 +71,10 @@ describe("InstructionBuiltIns", () => { Effect.gen(function* () { yield* TestClock.setTime(timestamp) const context = yield* InstructionBuiltIns.Service - const initialized = yield* Instructions.initialize(yield* context.load()) + const initialized = yield* readInitial(yield* context.load()) yield* TestClock.setTime(timestamp + 60 * 60 * 1000) - expect(yield* Instructions.reconcile(yield* context.load(), initialized.applied)).toEqual({ _tag: "Unchanged" }) + expect((yield* readUpdate(yield* context.load(), initialized)).changed).toBe(false) }), ) }) diff --git a/packages/core/test/instructions/index.test.ts b/packages/core/test/instructions/index.test.ts index 2d0a3b801a..1b74bd130c 100644 --- a/packages/core/test/instructions/index.test.ts +++ b/packages/core/test/instructions/index.test.ts @@ -1,137 +1,142 @@ import { describe, expect } from "bun:test" -import { Cause, Effect, Exit, Schema } from "effect" +import { Cause, Effect, Exit, Option, Schema } from "effect" import { Instructions } from "@opencode-ai/core/instructions" import { it } from "../lib/effect" -const key = Instructions.Key.make -const stringContext = (input: { +const key = (value: string) => Instructions.Key.make(value) +const source = (input: { key: string - value: string | Instructions.Unavailable - baseline?: (value: string) => string - update?: (previous: string, current: string) => string + value: string | Instructions.Unavailable | Instructions.Removed + initial?: (value: string) => string + changed?: (previous: string, current: string) => string removed?: (value: string) => string }) => Instructions.make({ key: key(input.key), codec: Schema.toCodecJson(Schema.String), - load: Effect.succeed(input.value), - baseline: input.baseline ?? String, - update: input.update ?? ((_previous, current) => current), - removed: input.removed, + read: Effect.succeed(input.value), + render: { + initial: input.initial ?? String, + changed: input.changed ?? ((_previous, current) => current), + removed: input.removed, + }, }) describe("Instructions", () => { - it.effect("stores the canonical JSON encoding of the loaded value", () => + it.effect("reads each source once and derives the initial delta and text", () => Effect.gen(function* () { - const context = Instructions.make({ + let reads = 0 + const instructions = Instructions.make({ key: key("core/date"), - codec: Schema.toCodecJson(Schema.DateFromString), - load: Effect.succeed(new Date("2026-06-03T12:00:00.000Z")), - baseline: (date) => date.toISOString(), - update: (_previous, date) => date.toISOString(), - removed: () => "Date removed", - }) - - expect((yield* Instructions.initialize(context)).applied["core/date"].value).toBe("2026-06-03T12:00:00.000Z") - }), - ) - - it.effect("loads once and initializes a baseline with the applied values", () => - Effect.gen(function* () { - let loads = 0 - const context = Instructions.combine([ - Instructions.make({ - key: key("core/date"), - codec: Schema.toCodecJson(Schema.String), - load: Effect.sync(() => { - loads++ - return "2026-06-03" - }), - baseline: (date) => `Today's date is ${date}.`, - update: (previous, current) => `The date changed from ${previous} to ${current}.`, - removed: () => "The date was removed.", + codec: Schema.toCodecJson(Schema.String), + read: Effect.sync(() => { + reads++ + return "2026-07-09" }), - stringContext({ key: "core/location", value: "/repo", baseline: (value) => `Directory: ${value}` }), - ]) - - expect(yield* Instructions.initialize(context)).toEqual({ - text: "Today's date is 2026-06-03.\n\nDirectory: /repo", - applied: { - "core/date": { value: "2026-06-03", removed: "The date was removed." }, - "core/location": { value: "/repo" }, - }, - }) - expect(loads).toBe(1) - }), - ) - - it.effect("renders updates only after a structured value changes", () => - Effect.gen(function* () { - const previous = { - "core/date": { value: "2026-06-03", removed: "The date was removed." }, - "core/location": { value: "/repo", removed: "Removed: /repo" }, - } - const changed = Instructions.combine([ - stringContext({ - key: "core/date", - value: "2026-06-04", - update: (before, current) => `The date changed from ${before} to ${current}.`, - removed: () => "The date was removed.", - }), - stringContext({ key: "core/location", value: "/repo" }), - ]) - - expect(yield* Instructions.reconcile(changed, previous)).toEqual({ - _tag: "Updated", - text: "The date changed from 2026-06-03 to 2026-06-04.", - applied: { - "core/date": { value: "2026-06-04", removed: "The date was removed." }, - "core/location": { value: "/repo", removed: "Removed: /repo" }, + render: { + initial: (date) => `Today's date: ${date}`, + changed: (previous, current) => `The date changed from ${previous} to ${current}`, }, }) + const admitted = yield* Instructions.read(instructions).pipe(Effect.flatMap(Instructions.diff)) + const hash = Instructions.hash("2026-07-09") + + expect(reads).toBe(1) + expect(admitted).toEqual({ + delta: { "core/date": hash }, + blobs: { [hash]: "2026-07-09" }, + }) + expect(Instructions.renderInitial(instructions, { "core/date": "2026-07-09" })).toBe("Today's date: 2026-07-09") + }), + ) + + it.effect("derives no delta when the encoded value is unchanged", () => + Effect.gen(function* () { + const instructions = source({ key: "core/date", value: "2026-07-09" }) + const admitted = yield* Instructions.read(instructions).pipe( + Effect.flatMap((observed) => Instructions.diff(observed, { "core/date": Instructions.hash("2026-07-09") })), + ) + + expect(admitted).toEqual({ delta: {}, blobs: {} }) + }), + ) + + it.effect("renders a changed value from stored values", () => + Effect.gen(function* () { + const instructions = source({ + key: "core/date", + value: "2026-07-10", + changed: (previous, current) => `The date changed from ${previous} to ${current}`, + }) + const admitted = yield* Instructions.read(instructions).pipe( + Effect.flatMap((observed) => Instructions.diff(observed, { "core/date": Instructions.hash("2026-07-09") })), + ) + + expect(admitted.delta).toEqual({ "core/date": Instructions.hash("2026-07-10") }) expect( - yield* Instructions.reconcile( - Instructions.combine([ - stringContext({ key: "core/date", value: "2026-06-03", removed: () => "The date was removed." }), - stringContext({ key: "core/location", value: "/repo" }), - ]), - previous, + Instructions.renderUpdate( + instructions, + { "core/date": "2026-07-09" }, + { "core/date": Option.some("2026-07-10") }, ), - ).toEqual({ _tag: "Unchanged" }) + ).toBe("The date changed from 2026-07-09 to 2026-07-10") }), ) - it.effect("uses the baseline for a newly added source", () => + it.effect("admits and renders an observed removal", () => Effect.gen(function* () { - const context = stringContext({ - key: "core/skills", - value: "effect", - baseline: (skill) => `Available skill: ${skill}`, + const instructions = source({ + key: "core/remote", + value: Instructions.removed, + removed: (previous) => `Stop applying ${previous}`, }) + const admitted = yield* Instructions.read(instructions).pipe( + Effect.flatMap((observed) => Instructions.diff(observed, { "core/remote": Instructions.hash("instructions") })), + ) - expect(yield* Instructions.reconcile(context, {})).toEqual({ - _tag: "Updated", - text: "Available skill: effect", - applied: { "core/skills": { value: "effect" } }, + expect(admitted).toEqual({ delta: { "core/remote": "removed" }, blobs: {} }) + expect( + Instructions.renderUpdate(instructions, { "core/remote": "instructions" }, { "core/remote": Option.none() }), + ).toBe("Stop applying instructions") + }), + ) + + it.effect("treats JSON null as a value rather than a removal", () => + Effect.gen(function* () { + const instructions = Instructions.make({ + key: key("api/value"), + codec: Schema.toCodecJson(Schema.Json), + read: Effect.succeed(null), + render: { + initial: String, + changed: (_previous, current) => String(current), + removed: () => "removed", + }, + }) + const admitted = yield* Instructions.read(instructions).pipe( + Effect.flatMap((observed) => Instructions.diff(observed, { "api/value": Instructions.hash("previous") })), + ) + + expect(admitted).toEqual({ + delta: { "api/value": Instructions.hash(null) }, + blobs: { [Instructions.hash(null)]: null }, + }) + expect( + Instructions.renderUpdate(instructions, { "api/value": "previous" }, { "api/value": Option.some(null) }), + ).toBe("null") + expect(Instructions.applyDelta({ "api/value": "previous" }, { "api/value": Option.some(null) })).toEqual({ + "api/value": null, }) }), ) - it.effect("retains the belief while a source is temporarily unavailable", () => + it.effect("blocks the initial delta while any source is unavailable", () => Effect.gen(function* () { - const previous = { "core/remote": { value: "instructions", removed: "Instructions removed" } } - const context = stringContext({ key: "core/remote", value: Instructions.unavailable }) - - expect(yield* Instructions.reconcile(context, previous)).toEqual({ _tag: "Unchanged" }) - }), - ) - - it.effect("blocks initialization while a source is unavailable", () => - Effect.gen(function* () { - const exit = yield* Instructions.initialize( - stringContext({ key: "core/remote", value: Instructions.unavailable }), - ).pipe(Effect.exit) + const exit = yield* Instructions.read(source({ key: "core/remote", value: Instructions.unavailable })).pipe( + Effect.flatMap(Instructions.diff), + Effect.exit, + ) expect(Exit.isFailure(exit)).toBe(true) if (Exit.isFailure(exit)) @@ -139,176 +144,89 @@ describe("Instructions", () => { }), ) - it.effect("emits the previously stored removal message", () => + it.effect("keeps the stored value while a source is unavailable mid-session", () => Effect.gen(function* () { - expect( - yield* Instructions.reconcile(Instructions.empty, { - "core/instructions": { value: "contents", removed: "Instructions removed; stop applying them." }, - }), - ).toEqual({ - _tag: "Updated", - text: "Instructions removed; stop applying them.", - applied: {}, - }) + const admitted = yield* Instructions.read(source({ key: "core/remote", value: Instructions.unavailable })).pipe( + Effect.flatMap((observed) => Instructions.diff(observed, { "core/remote": Instructions.hash("instructions") })), + ) + + expect(admitted).toEqual({ delta: {}, blobs: {} }) }), ) - it.effect("retains an unannounced removal silently", () => + it.effect("does not infer removal when a source is absent from the current version", () => Effect.gen(function* () { - expect(yield* Instructions.reconcile(Instructions.empty, { "core/date": { value: "2026-06-04" } })).toEqual({ - _tag: "Unchanged", - }) + const admitted = yield* Instructions.read(Instructions.empty).pipe( + Effect.flatMap((observed) => + Instructions.diff(observed, { "core/retired": Instructions.hash("old instructions") }), + ), + ) - // The retained belief survives alongside other updates. - expect( - yield* Instructions.reconcile(stringContext({ key: "core/skills", value: "effect" }), { - "core/date": { value: "2026-06-04" }, - }), - ).toEqual({ - _tag: "Updated", - text: "effect", - applied: { - "core/skills": { value: "effect" }, - "core/date": { value: "2026-06-04" }, - }, - }) + expect(admitted).toEqual({ delta: {}, blobs: {} }) }), ) - it.effect("renders multiple removals in stable key order", () => + it.effect("renders a newly added source with its initial renderer", () => Effect.gen(function* () { - expect( - yield* Instructions.reconcile(Instructions.empty, { - "core/z": { value: "z", removed: "Removed z" }, - "core/a": { value: "a", removed: "Removed a" }, - }), - ).toMatchObject({ _tag: "Updated", text: "Removed a\n\nRemoved z" }) + const instructions = source({ + key: "core/skills", + value: "effect", + initial: (skill) => `Available skill: ${skill}`, + }) + + expect(Instructions.renderUpdate(instructions, {}, { "core/skills": Option.some("effect") })).toBe( + "Available skill: effect", + ) + }), + ) + + it.effect("hashes objects independently of key order", () => + Effect.sync(() => { + expect(Instructions.hash({ a: 1, b: { x: true, y: false } })).toBe( + Instructions.hash({ b: { y: false, x: true }, a: 1 }), + ) + }), + ) + + it.effect("renders sources in composition order", () => + Effect.sync(() => { + const instructions = Instructions.combine([ + source({ key: "core/date", value: "date" }), + source({ key: "core/location", value: "location" }), + ]) + + expect(Instructions.renderInitial(instructions, { "core/date": "date", "core/location": "location" })).toBe( + "date\n\nlocation", + ) + }), + ) + + it.effect("rejects duplicate source keys", () => + Effect.sync(() => { + expect(() => + Instructions.combine([source({ key: "core/date", value: "one" }), source({ key: "core/date", value: "two" })]), + ).toThrow(new Instructions.DuplicateKeyError({ key: key("core/date") })) }), ) it.effect("rejects empty model-visible renderings", () => - Effect.gen(function* () { - const exit = yield* Instructions.initialize( - stringContext({ key: "core/empty", value: "value", baseline: () => "" }), - ).pipe(Effect.exit) + Effect.sync(() => { + const instructions = source({ key: "core/empty", value: "value", initial: () => "" }) - expect(Exit.isFailure(exit)).toBe(true) - if (Exit.isFailure(exit)) expect(Cause.pretty(exit.cause)).toContain("rendered an empty baseline") + expect(() => Instructions.renderInitial(instructions, { "core/empty": "value" })).toThrow( + "Instruction source core/empty rendered an empty initial", + ) }), ) - it.effect("re-announces the baseline when a stored value no longer decodes", () => - Effect.gen(function* () { - expect( - yield* Instructions.reconcile(stringContext({ key: "core/date", value: "2026-06-04" }), { - "core/date": { value: 42, removed: "Date removed" }, - }), - ).toEqual({ - _tag: "Updated", - text: "2026-06-04", - applied: { "core/date": { value: "2026-06-04" } }, - }) - }), - ) - - it.effect("renders undecodable re-announcements alongside other updates", () => - Effect.gen(function* () { - const context = Instructions.combine([ - stringContext({ - key: "core/date", - value: "2026-06-04", - update: (before, current) => `${before} -> ${current}`, - }), - stringContext({ key: "core/location", value: "/repo" }), - ]) - - expect( - yield* Instructions.reconcile(context, { - "core/date": { value: "2026-06-03" }, - "core/location": { value: 42 }, - }), - ).toEqual({ - _tag: "Updated", - text: "2026-06-03 -> 2026-06-04\n\n/repo", - applied: { - "core/date": { value: "2026-06-04" }, - "core/location": { value: "/repo" }, - }, - }) - }), - ) - - it.effect("rebaselines from one coherent source observation", () => - Effect.gen(function* () { - let loads = 0 - const context = Instructions.make({ - key: key("core/date"), - codec: Schema.toCodecJson(Schema.String), - load: Effect.sync(() => { - loads++ - return "2026-06-04" - }), - baseline: String, - update: (_previous, current) => current, - }) - - expect(yield* Instructions.rebaseline(context, { "core/date": { value: "2026-06-03" } })).toEqual({ - text: "2026-06-04", - applied: { "core/date": { value: "2026-06-04" } }, - }) - expect(loads).toBe(1) - }), - ) - - it.effect("rebaselines an unavailable source from the last-applied belief", () => - Effect.gen(function* () { - const context = Instructions.combine([ - stringContext({ key: "core/date", value: "2026-06-04" }), - stringContext({ - key: "core/remote", - value: Instructions.unavailable, - baseline: (value) => `Instructions: ${value}`, - }), - ]) - - expect( - yield* Instructions.rebaseline(context, { - "core/remote": { value: "contents", removed: "Instructions removed" }, - }), - ).toEqual({ - text: "2026-06-04\n\nInstructions: contents", - applied: { - "core/date": { value: "2026-06-04" }, - "core/remote": { value: "contents", removed: "Instructions removed" }, - }, - }) - }), - ) - - it.effect("drops undecodable beliefs and removed sources at rebaseline", () => - Effect.gen(function* () { - const context = stringContext({ key: "core/remote", value: Instructions.unavailable }) - - // Undecodable belief cannot be restated; removed source entries self-clean. - expect( - yield* Instructions.rebaseline(context, { - "core/remote": { value: 42 }, - "core/gone": { value: "gone" }, - }), - ).toEqual({ text: "", applied: {} }) - }), - ) - - it.effect("diffs list values by key with a changed comparator", () => + it.effect("diffs list values by key", () => Effect.sync(() => { const previous = [ { name: "effect", description: "Build with Effect" }, - { name: "debugging", description: "Diagnose bugs" }, { name: "retired", description: "Old" }, ] const current = [ { name: "effect", description: "Build with Effect v4" }, - { name: "debugging", description: "Diagnose bugs" }, { name: "writing", description: "Write prose" }, ] @@ -331,47 +249,4 @@ describe("Instructions", () => { }) }), ) - - it.effect("rejects duplicate source keys", () => - Effect.sync(() => { - expect(() => - Instructions.combine([ - stringContext({ key: "core/date", value: "one" }), - stringContext({ key: "core/date", value: "two" }), - ]), - ).toThrow(new Instructions.DuplicateKeyError({ key: key("core/date") })) - }), - ) - - it.effect("combines instructions in order", () => - Effect.gen(function* () { - expect( - (yield* Instructions.initialize( - Instructions.combine([ - stringContext({ key: "core/date", value: "date" }), - stringContext({ key: "core/location", value: "location" }), - ]), - )).text, - ).toBe("date\n\nlocation") - }), - ) - - it.effect("requires namespaced source keys", () => - Effect.sync(() => { - const decodeKey = Schema.decodeUnknownSync(Instructions.Key) - - expect(decodeKey("core/date")).toBe(key("core/date")) - expect(() => decodeKey("date")).toThrow() - }), - ) - - it.effect("requires namespaced applied keys", () => - Effect.sync(() => { - const decodeApplied = Schema.decodeUnknownSync(Instructions.Applied) - - expect(Object.keys(decodeApplied({ "core/date": { value: "date" } }))).toEqual(["core/date"]) - expect(() => decodeApplied({ date: { value: "date" } })).toThrow() - expect(() => decodeApplied({ "core/date": { value: "date", removed: "" } })).toThrow() - }), - ) }) diff --git a/packages/core/test/lib/instructions.ts b/packages/core/test/lib/instructions.ts new file mode 100644 index 0000000000..ca4efa22ca --- /dev/null +++ b/packages/core/test/lib/instructions.ts @@ -0,0 +1,43 @@ +import { Effect, Option, Schema } from "effect" +import { Instructions } from "@opencode-ai/core/instructions" + +export interface State { + readonly values: Readonly> +} + +export const state = (values: Readonly>): State => ({ values }) + +const hashes = (values: Readonly>): Instructions.Values => + Object.fromEntries(Object.entries(values).map(([key, value]) => [key, Instructions.hash(value)])) + +export const readInitial = (instructions: Instructions.Instructions) => + Effect.gen(function* () { + const admission = yield* Instructions.read(instructions).pipe(Effect.flatMap(Instructions.diff)) + const current = state( + Object.fromEntries( + Object.entries(admission.delta).flatMap(([key, hash]) => + hash === "removed" ? [] : [[key, admission.blobs[hash]]], + ), + ), + ) + return { ...current, text: Instructions.renderInitial(instructions, current.values) } + }) + +export const readUpdate = (instructions: Instructions.Instructions, previous: State) => + Effect.gen(function* () { + const admission = yield* Instructions.read(instructions).pipe( + Effect.flatMap((observed) => Instructions.diff(observed, hashes(previous.values))), + ) + const delta = Object.fromEntries( + Object.entries(admission.delta).map(([key, hash]) => [ + key, + hash === "removed" ? Option.none() : Option.some(admission.blobs[hash]), + ]), + ) as Readonly>> + const values = Instructions.applyDelta(previous.values, delta) + return { + values, + text: Instructions.renderUpdate(instructions, previous.values, delta), + changed: Object.keys(admission.delta).length > 0, + } + }) diff --git a/packages/core/test/move-session.test.ts b/packages/core/test/move-session.test.ts index 4460c7599a..c550de1d3b 100644 --- a/packages/core/test/move-session.test.ts +++ b/packages/core/test/move-session.test.ts @@ -3,7 +3,7 @@ import { $ } from "bun" import fs from "fs/promises" import path from "path" import { eq } from "drizzle-orm" -import { Effect } from "effect" +import { Effect, Layer } from "effect" import { MoveSession } from "@opencode-ai/core/control-plane/move-session" import { Database } from "@opencode-ai/core/database/database" import { AppNodeBuilder } from "@opencode-ai/core/effect/app-node-builder" @@ -15,12 +15,26 @@ import { ProjectTable } from "@opencode-ai/core/project/sql" import { ProjectDirectories } from "@opencode-ai/core/project/directories" import { AbsolutePath } from "@opencode-ai/core/schema" import { SessionV2 } from "@opencode-ai/core/session" +import { SessionExecution } from "@opencode-ai/core/session/execution" import { SessionProjector } from "@opencode-ai/core/session/projector" import { SessionTable } from "@opencode-ai/core/session/sql" import { SessionStore } from "@opencode-ai/core/session/store" import { tmpdir } from "./fixture/tmpdir" import { testEffect } from "./lib/effect" +// Records the execution serialization a move must perform before relocating. +const executionCalls: string[] = [] +const recordingExecution = Layer.succeed( + SessionExecution.Service, + SessionExecution.Service.of({ + active: Effect.succeed(new Set()), + resume: () => Effect.void, + wake: () => Effect.void, + interrupt: (sessionID) => Effect.sync(() => void executionCalls.push(`interrupt:${sessionID}`)), + awaitIdle: (sessionID) => Effect.sync(() => void executionCalls.push(`awaitIdle:${sessionID}`)), + }), +) + const it = testEffect( AppNodeBuilder.build( LayerNode.group([ @@ -32,6 +46,7 @@ const it = testEffect( SessionProjector.node, SessionStore.node, ]), + [[SessionExecution.node, recordingExecution]], ), ) @@ -92,10 +107,13 @@ describe("MoveSession", () => { .run() .pipe(Effect.orDie) + executionCalls.length = 0 yield* MoveSession.Service.use((service) => service.moveSession({ sessionID, destination: { directory: moved }, moveChanges: true }), ) + // The move stops active execution before any relocation side effect. + expect(executionCalls).toEqual([`interrupt:${sessionID}`, `awaitIdle:${sessionID}`]) expect(yield* Effect.promise(() => fs.readFile(path.join(moved, "tracked.txt"), "utf8"))).toBe("changed\n") expect(yield* Effect.promise(() => fs.readFile(path.join(moved, "untracked.txt"), "utf8"))).toBe("new\n") expect(yield* Effect.promise(() => fs.readFile(path.join(source, "tracked.txt"), "utf8"))).toBe("initial\n") diff --git a/packages/core/test/reference-guidance.test.ts b/packages/core/test/reference-guidance.test.ts index fe34873f96..17d412f902 100644 --- a/packages/core/test/reference-guidance.test.ts +++ b/packages/core/test/reference-guidance.test.ts @@ -4,8 +4,8 @@ import { AppNodeBuilder } from "@opencode-ai/core/effect/app-node-builder" import { AbsolutePath } from "@opencode-ai/core/schema" import { Reference } from "@opencode-ai/core/reference" import { ReferenceGuidance } from "@opencode-ai/core/reference/guidance" -import { Instructions } from "@opencode-ai/core/instructions/index" import { it } from "./lib/effect" +import { readInitial, readUpdate } from "./lib/instructions" const guidanceLayer = (referenceLayer: Layer.Layer) => AppNodeBuilder.build(ReferenceGuidance.node, [[Reference.node, referenceLayer]]) @@ -14,7 +14,7 @@ describe("ReferenceGuidance", () => { it.effect("lists available references in the instructions", () => Effect.gen(function* () { const guidance = yield* ReferenceGuidance.Service - const generation = yield* Instructions.initialize(yield* guidance.load()) + const generation = yield* readInitial(yield* guidance.load()) expect(generation.text).toContain("") expect(generation.text).toContain("docs") @@ -46,7 +46,7 @@ describe("ReferenceGuidance", () => { it.effect("omits guidance when no references are available", () => Effect.gen(function* () { const guidance = yield* ReferenceGuidance.Service - const generation = yield* Instructions.initialize(yield* guidance.load()) + const generation = yield* readInitial(yield* guidance.load()) expect(generation.text).toBe("") }).pipe(Effect.provide(guidanceLayer(Layer.mock(Reference.Service, { list: () => Effect.succeed([]) })))), ) @@ -54,7 +54,7 @@ describe("ReferenceGuidance", () => { it.effect("omits references without descriptions", () => Effect.gen(function* () { const guidance = yield* ReferenceGuidance.Service - const generation = yield* Instructions.initialize(yield* guidance.load()) + const generation = yield* readInitial(yield* guidance.load()) expect(generation.text).toBe("") }).pipe( Effect.provide( @@ -85,13 +85,12 @@ describe("ReferenceGuidance", () => { let references = [reference("docs", "Use for product documentation")] return Effect.gen(function* () { const guidance = yield* ReferenceGuidance.Service - const initialized = yield* Instructions.initialize(yield* guidance.load()) + const initialized = yield* readInitial(yield* guidance.load()) references = [reference("docs", "Use for product documentation"), reference("examples", "Use for examples")] - const added = yield* Instructions.reconcile(yield* guidance.load(), initialized.applied) - expect(added).toMatchObject({ - _tag: "Updated", - text: [ + const added = yield* readUpdate(yield* guidance.load(), initialized) + expect(added.text).toBe( + [ "New project references are available in addition to those previously listed:", " ", " examples", @@ -99,15 +98,12 @@ describe("ReferenceGuidance", () => { " Use for examples", " ", ].join("\n"), - }) + ) references = [reference("examples", "Use for examples")] - expect( - yield* Instructions.reconcile(yield* guidance.load(), added._tag === "Updated" ? added.applied : {}), - ).toMatchObject({ - _tag: "Updated", - text: "The following project references are no longer available and must not be used: docs.", - }) + expect((yield* readUpdate(yield* guidance.load(), added)).text).toBe( + "The following project references are no longer available and must not be used: docs.", + ) }).pipe(Effect.provide(guidanceLayer(Layer.mock(Reference.Service, { list: () => Effect.succeed(references) })))) }) }) diff --git a/packages/core/test/session-instructions.test.ts b/packages/core/test/session-instructions.test.ts index 6faa0ce52c..a682c1450e 100644 --- a/packages/core/test/session-instructions.test.ts +++ b/packages/core/test/session-instructions.test.ts @@ -162,7 +162,7 @@ describe("SessionInstructions", () => { const sessionID = (yield* session.create({ location: Location.Ref.make({ directory: dir }) })).id // A read deep under sub/ discovers deep and sub AGENTS.md, walking up to but - // excluding the Location root (already supplied by the core/instructions baseline). + // excluding the Location root (already supplied by core initial instructions). yield* settleTool(registry, readCall(sessionID, "call-deep", "sub/deep/file.txt")) const firstInjected = yield* synthetics(sessionID) @@ -235,7 +235,7 @@ describe("SessionInstructions", () => { const sessionID = (yield* session.create({ location: Location.Ref.make({ directory: dir }) })).id // Listing packages/foo/ discovers its own AGENTS.md, walking up to but excluding - // the Location root (already supplied by the core/instructions baseline). + // the Location root (already supplied by core initial instructions). yield* settleTool(registry, readCall(sessionID, "call-list", "packages/foo")) const firstInjected = yield* synthetics(sessionID) diff --git a/packages/core/test/session-projector.test.ts b/packages/core/test/session-projector.test.ts index e03fd3faff..cddafaf52b 100644 --- a/packages/core/test/session-projector.test.ts +++ b/packages/core/test/session-projector.test.ts @@ -23,7 +23,7 @@ import { fromRow } from "@opencode-ai/core/session/info" import { SessionPending } from "@opencode-ai/core/session/pending" import { Shell } from "@opencode-ai/schema/shell" import { - InstructionCheckpointTable, + InstructionStateTable, SessionPendingTable, SessionMessageTable, SessionTable, @@ -204,8 +204,14 @@ describe("SessionProjector", () => { ]) .run() yield* db - .insert(InstructionCheckpointTable) - .values({ session_id: sessionID, baseline: "baseline", snapshot: {}, baseline_seq: 0 }) + .insert(InstructionStateTable) + .values({ + session_id: sessionID, + epoch_start: 0, + through_seq: 0, + initial_values: {}, + current_values: {}, + }) .run() const events = yield* EventV2.Service yield* events.publish(SessionEvent.RevertEvent.Staged, { @@ -238,8 +244,8 @@ describe("SessionProjector", () => { tokens_cache_read: 3, tokens_cache_write: 1, }) - // A committed revert resets the context checkpoint so the next turn re-initializes. - expect(yield* db.select().from(InstructionCheckpointTable).get().pipe(Effect.orDie)).toBeUndefined() + // A committed revert resets the fold cache so the next boundary establishes a new epoch. + expect(yield* db.select().from(InstructionStateTable).get().pipe(Effect.orDie)).toBeUndefined() }), ) diff --git a/packages/core/test/session-runner-recorded.test.ts b/packages/core/test/session-runner-recorded.test.ts index 7fa33da84a..d4fb21fbe3 100644 --- a/packages/core/test/session-runner-recorded.test.ts +++ b/packages/core/test/session-runner-recorded.test.ts @@ -200,6 +200,7 @@ describe("SessionRunnerLLM recorded", () => { .all()).map((event) => event.type), ).toEqual([ "session.input.admitted.1", + "session.instructions.updated.2", "session.input.promoted.1", "session.step.started.1", "session.text.started.1", diff --git a/packages/core/test/session-runner.test.ts b/packages/core/test/session-runner.test.ts index 1ab51f4f51..30499aca1b 100644 --- a/packages/core/test/session-runner.test.ts +++ b/packages/core/test/session-runner.test.ts @@ -46,7 +46,7 @@ import { Config } from "@opencode-ai/core/config" import { ConfigCompaction } from "@opencode-ai/core/config/compaction" import { Tool } from "@opencode-ai/core/tool/tool" import { - InstructionCheckpointTable, + InstructionStateTable, SessionPendingTable, SessionMessageTable, SessionTable, @@ -285,22 +285,22 @@ const skillBaselines = new Map() const systemContext = Layer.mock(InstructionBuiltIns.Service, { load: () => Effect.sync(() => - Instructions.combine( - systemRemoved - ? [] - : [ - Instructions.make({ - key: systemContextKey, - codec: Schema.toCodecJson(Schema.String), - load: systemLoadHook.pipe( - Effect.andThen(Effect.sync(() => (systemUnavailable ? Instructions.unavailable : systemBaseline))), - ), - baseline: String, - update: (_previous, current) => current, - removed: () => "System context source removed: test/context", - }), - ], - ), + Instructions.make({ + key: systemContextKey, + codec: Schema.toCodecJson(Schema.String), + read: systemLoadHook.pipe( + Effect.andThen( + Effect.sync(() => + systemUnavailable ? Instructions.unavailable : systemRemoved ? Instructions.removed : systemBaseline, + ), + ), + ), + render: { + initial: String, + changed: (_previous, current) => current, + removed: () => "System context source removed: test/context", + }, + }), ), }) const instructionContext = Layer.mock(InstructionDiscovery.Service, { load: () => Effect.succeed(Instructions.empty) }) @@ -311,10 +311,12 @@ const skillGuidance = Layer.mock(SkillGuidance.Service, { ? Instructions.make({ key: Instructions.Key.make("test/skill-guidance"), codec: Schema.toCodecJson(Schema.String), - load: Effect.succeed(skillBaselines.get(agent.id)!), - baseline: String, - update: (_previous, current) => current, - removed: () => "Skill guidance removed", + read: Effect.succeed(skillBaselines.get(agent.id)!), + render: { + initial: String, + changed: (_previous, current) => current, + removed: () => "Skill guidance removed", + }, }) : Instructions.empty, ), @@ -577,6 +579,7 @@ const replaySessionProjection = (id: SessionV2.ID) => .pipe(Effect.orDie) yield* events.remove(id) + yield* db.delete(InstructionStateTable).where(eq(InstructionStateTable.session_id, id)).run().pipe(Effect.orDie) yield* db.delete(SessionPendingTable).where(eq(SessionPendingTable.session_id, id)).run().pipe(Effect.orDie) yield* db.delete(SessionMessageTable).where(eq(SessionMessageTable.session_id, id)).run().pipe(Effect.orDie) yield* events.replayAll( @@ -942,11 +945,7 @@ describe("SessionRunnerLLM", () => { expect(requests).toHaveLength(0) expect(yield* SessionPending.has(db, sessionID, "steer")).toBe(true) expect( - yield* db - .select() - .from(InstructionCheckpointTable) - .where(eq(InstructionCheckpointTable.session_id, sessionID)) - .get(), + yield* db.select().from(InstructionStateTable).where(eq(InstructionStateTable.session_id, sessionID)).get(), ).toBeUndefined() systemUnavailable = false @@ -971,11 +970,7 @@ describe("SessionRunnerLLM", () => { location: Location.Ref.make({ directory: AbsolutePath.make("/moved") }), }) expect( - yield* db - .select() - .from(InstructionCheckpointTable) - .where(eq(InstructionCheckpointTable.session_id, sessionID)) - .get(), + yield* db.select().from(InstructionStateTable).where(eq(InstructionStateTable.session_id, sessionID)).get(), ).toBeUndefined() yield* admit(session, "Second") @@ -987,66 +982,121 @@ describe("SessionRunnerLLM", () => { }), ) - it.effect("copies the context checkpoint to a fork", () => + it.effect("forks instruction values at the selected message instead of the parent's latest state", () => Effect.gen(function* () { const session = yield* setup - const { db } = yield* Database.Service - yield* admit(session, "First") + const first = yield* admit(session, "First") + yield* session.resume(sessionID) + systemBaseline = "Changed context" + const second = yield* admit(session, "Second") + yield* session.resume(sessionID) + systemBaseline = "Latest context" + yield* admit(session, "Third") yield* session.resume(sessionID) - const forked = yield* session.fork({ sessionID }) - - const parent = yield* db - .select() - .from(InstructionCheckpointTable) - .where(eq(InstructionCheckpointTable.session_id, sessionID)) - .get() - .pipe(Effect.orDie) - expect(parent).toBeDefined() + const forked = yield* session.fork({ sessionID, messageID: second.id }) expect( - yield* db + yield* (yield* Database.Service).db .select() - .from(InstructionCheckpointTable) - .where(eq(InstructionCheckpointTable.session_id, forked.id)) - .get() - .pipe(Effect.orDie), - ).toEqual({ ...parent!, session_id: forked.id }) + .from(InstructionStateTable) + .where(eq(InstructionStateTable.session_id, forked.id)) + .get(), + ).toMatchObject({ + initial_values: { "test/context": Instructions.hash("Initial context") }, + current_values: { "test/context": Instructions.hash("Changed context") }, + }) + yield* session.prompt({ sessionID: forked.id, text: "Forked", resume: false }) + yield* session.resume(forked.id) + + expect(requests.at(-1)?.system.map((part) => part.text)).toEqual([defaultSystem, "Initial context"]) + expect(systemTexts(requests.at(-1)!)).toContain("Changed context") + expect(systemTexts(requests.at(-1)!)).toContain("Latest context") + + const { db } = yield* Database.Service + const events = yield* EventV2.Service + const recorded = yield* db + .select() + .from(EventTable) + .where(eq(EventTable.aggregate_id, forked.id)) + .orderBy(asc(EventTable.seq)) + .all() + yield* events.remove(forked.id) + yield* db.delete(SessionTable).where(eq(SessionTable.id, forked.id)).run() + yield* events.replayAll( + recorded.map((event) => ({ + id: event.id, + created: DateTime.makeUnsafe(event.created), + aggregateID: event.aggregate_id, + seq: event.seq, + type: event.type, + data: event.data, + })), + ) + expect( + yield* db.select().from(InstructionStateTable).where(eq(InstructionStateTable.session_id, forked.id)).get(), + ).toMatchObject({ current_values: { "test/context": Instructions.hash("Latest context") } }) }), ) - it.effect("heals an undecodable stored applied record by re-announcing context", () => + it.effect("caps nested fork instruction ancestry at the selected message", () => + Effect.gen(function* () { + const session = yield* setup + yield* admit(session, "First") + yield* session.resume(sessionID) + systemBaseline = "Changed context" + const second = yield* admit(session, "Second") + yield* session.resume(sessionID) + + const child = yield* session.fork({ sessionID, messageID: second.id }) + const inheritedFirst = (yield* session.messages({ sessionID: child.id })).find( + (message) => message.type === "user" && message.text === "First", + ) + if (!inheritedFirst) return yield* Effect.die(new Error("Nested fork boundary message not found")) + const grandchild = yield* session.fork({ sessionID: child.id, messageID: inheritedFirst.id }) + + expect( + yield* (yield* Database.Service).db + .select() + .from(InstructionStateTable) + .where(eq(InstructionStateTable.session_id, grandchild.id)) + .get(), + ).toMatchObject({ + initial_values: { "test/context": Instructions.hash("Initial context") }, + current_values: { "test/context": Instructions.hash("Initial context") }, + }) + }), + ) + + it.effect("rebuilds a missing instruction cache without admitting another delta", () => Effect.gen(function* () { const session = yield* setup const { db } = yield* Database.Service yield* admit(session, "First") yield* session.resume(sessionID) - yield* db - .update(InstructionCheckpointTable) - .set({ snapshot: { invalid: { value: "bad" } } }) - .where(eq(InstructionCheckpointTable.session_id, sessionID)) - .run() - .pipe(Effect.orDie) + yield* db.delete(InstructionStateTable).where(eq(InstructionStateTable.session_id, sessionID)).run() yield* admit(session, "Second") requests.length = 0 yield* session.resume(sessionID) - // Comparison state was lost, so every source re-announces as new. expect(requests).toHaveLength(1) expect(requests[0]?.system.map((part) => part.text)).toEqual([defaultSystem, "Initial context"]) - expect(requests[0]?.messages.map((message) => message.role)).toEqual(["user", "system", "user"]) - expect(requests[0]?.messages.at(1)?.content).toEqual([{ type: "text", text: "Initial context" }]) - const healed = yield* db - .select({ snapshot: InstructionCheckpointTable.snapshot }) - .from(InstructionCheckpointTable) - .where(eq(InstructionCheckpointTable.session_id, sessionID)) - .get() - .pipe(Effect.orDie) - expect(healed?.snapshot).toEqual({ "test/context": { value: "Initial context", removed: expect.any(String) } }) + expect(requests[0]?.messages.map((message) => message.role)).toEqual(["user", "user"]) + expect( + yield* db + .select({ id: EventTable.id }) + .from(EventTable) + .where(eq(EventTable.type, "session.instructions.updated.2")) + .all(), + ).toHaveLength(1) + expect(yield* db.select().from(InstructionStateTable).get()).toMatchObject({ + initial_values: { "test/context": Instructions.hash("Initial context") }, + current_values: { "test/context": Instructions.hash("Initial context") }, + }) }), ) - it.effect("reuses one durable baseline after the context producer changes", () => + it.effect("keeps the initial instructions stable and derives a chronological update from values", () => Effect.gen(function* () { const session = yield* setup yield* admit(session, "First") @@ -1062,18 +1112,26 @@ describe("SessionRunnerLLM", () => { ]) expect(requests[1]?.messages.map((message) => message.role)).toEqual(["user", "system", "user"]) expect(requests[1]?.messages.at(1)?.content).toEqual([{ type: "text", text: "Changed context" }]) - expect(yield* session.messages({ sessionID })).toHaveLength(3) + expect(yield* session.messages({ sessionID })).toHaveLength(2) const { db } = yield* Database.Service - expect( - yield* db - .select({ id: EventTable.id }) - .from(EventTable) - .where(eq(EventTable.type, "session.instructions.updated.1")) - .all() - .pipe(Effect.orDie), - ).toHaveLength(1) + const updates = yield* db + .select({ data: EventTable.data }) + .from(EventTable) + .where(eq(EventTable.type, "session.instructions.updated.2")) + .orderBy(asc(EventTable.seq)) + .all() + .pipe(Effect.orDie) + expect(updates).toHaveLength(2) + expect(updates[0]?.data).toEqual({ + sessionID, + delta: { "test/context": Instructions.hash("Initial context") }, + }) + expect(updates[1]?.data).toEqual({ + sessionID, + delta: { "test/context": Instructions.hash("Changed context") }, + }) yield* replaySessionProjection(sessionID) - expect(yield* session.messages({ sessionID })).toHaveLength(3) + expect(yield* session.messages({ sessionID })).toHaveLength(2) }), ) @@ -1160,7 +1218,7 @@ describe("SessionRunnerLLM", () => { }), ) - it.effect("uses only the agent prompt and durable baseline as system parts", () => + it.effect("uses only the agent prompt and initial instructions as system parts", () => Effect.gen(function* () { const session = yield* setup const agent = yield* AgentV2.Service @@ -1350,11 +1408,11 @@ describe("SessionRunnerLLM", () => { expect(requests[1]?.messages.at(1)?.content).toEqual([ { type: "text", text: "System context source removed: test/context" }, ]) - expect(yield* session.messages({ sessionID })).toHaveLength(3) + expect(yield* session.messages({ sessionID })).toHaveLength(2) }), ) - it.effect("renders API context entries through the belief lifecycle", () => + it.effect("renders API context entries through add, change, and removal", () => Effect.gen(function* () { const session = yield* setup const contextEntries = yield* InstructionEntry.Service @@ -1363,7 +1421,7 @@ describe("SessionRunnerLLM", () => { yield* session.resume(sessionID) - // String values render verbatim inside the tagged block at baseline. + // String values render verbatim inside the initial tagged block. expect(requests[0]?.system.map((part) => part.text)).toEqual([ defaultSystem, ["Initial context", "", '', "production", ""].join("\n"), @@ -1403,7 +1461,49 @@ describe("SessionRunnerLLM", () => { }), ) - it.effect("keeps the baseline and chronological System updates after a model switch", () => + it.effect("retains JSON null API entries as values", () => + Effect.gen(function* () { + const session = yield* setup + const entries = yield* InstructionEntry.Service + yield* entries.put({ sessionID, key: "nullable", value: "present" }) + yield* admit(session, "First") + yield* session.resume(sessionID) + + yield* entries.put({ sessionID, key: "nullable", value: null }) + yield* admit(session, "Second") + yield* session.resume(sessionID) + + expect(requests[1]?.messages.at(1)?.content).toEqual([ + { + type: "text", + text: [ + 'The context under "nullable" changed and supersedes the previous value:', + '', + "null", + "", + ].join("\n"), + }, + ]) + expect(yield* entries.list(sessionID)).toEqual([{ key: "nullable", value: null }]) + }), + ) + + it.effect("rejects API instruction entries larger than 8KB", () => + Effect.gen(function* () { + yield* setup + const entries = yield* InstructionEntry.Service + + const exit = yield* entries + .put({ sessionID, key: "oversized", value: "x".repeat(InstructionEntry.MaxValueBytes) }) + .pipe(Effect.exit) + + expect(Exit.isFailure(exit)).toBe(true) + if (Exit.isFailure(exit)) expect(Cause.squash(exit.cause)).toBeInstanceOf(InstructionEntry.ValueTooLargeError) + expect(yield* entries.list(sessionID)).toEqual([]) + }), + ) + + it.effect("keeps initial instructions and chronological updates after a model switch", () => Effect.gen(function* () { const session = yield* setup const events = yield* EventV2.Service @@ -1430,20 +1530,18 @@ describe("SessionRunnerLLM", () => { expect(requests[2]?.messages.filter((message) => message.role === "system")).toHaveLength(2) expect((yield* session.context(sessionID)).map((message) => message.type)).toEqual([ "user", - "system", "user", "model-switched", - "system", "user", ]) yield* replaySessionProjection(sessionID) - expect(yield* session.messages({ sessionID })).toHaveLength(6) + expect(yield* session.messages({ sessionID })).toHaveLength(4) yield* admit(session, "Fourth") yield* session.resume(sessionID) }), ) - it.effect("preserves the baseline while context is temporarily unavailable", () => + it.effect("preserves instruction values while a source is temporarily unavailable", () => Effect.gen(function* () { const session = yield* setup const events = yield* EventV2.Service @@ -1470,7 +1568,7 @@ describe("SessionRunnerLLM", () => { }), ) - it.effect("rebuilds the baseline directly after completed compaction", () => + it.effect("moves the epoch at compaction and narrates later changes", () => Effect.gen(function* () { const session = yield* setup const events = yield* EventV2.Service @@ -1494,8 +1592,10 @@ describe("SessionRunnerLLM", () => { expect(requests.map((request) => request.system.map((part) => part.text))).toEqual([ [defaultSystem, "Initial context"], - [defaultSystem, "Replacement context"], + [defaultSystem, "Initial context"], ]) + expect(requests[1]?.messages.map((message) => message.role)).toEqual(["user", "system", "user"]) + expect(requests[1]?.messages.at(1)?.content).toEqual([{ type: "text", text: "Replacement context" }]) yield* replaySessionProjection(sessionID) yield* admit(session, "Third") yield* session.resume(sessionID) @@ -1953,7 +2053,7 @@ describe("SessionRunnerLLM", () => { }), ) - it.effect("rebaselines after compaction from the last-applied belief while unobservable", () => + it.effect("uses epoch values after compaction while a source is unavailable", () => Effect.gen(function* () { const session = yield* setup const events = yield* EventV2.Service @@ -1978,7 +2078,7 @@ describe("SessionRunnerLLM", () => { yield* admit(session, "Third") yield* session.resume(sessionID) - // The rebaseline proceeds while the source is unobservable, restating the model's belief. + // Compaction already moved current values into the new epoch before the unavailable read. expect(requests.at(-1)?.system.map((part) => part.text)).toEqual([defaultSystem, "Changed context"]) expect(systemTexts(requests.at(-1)!)).not.toContain("Changed context") }), @@ -2861,7 +2961,7 @@ describe("SessionRunnerLLM", () => { }) yield* (yield* SessionExecution.Service).wake(sessionID) - yield* Effect.yieldNow + while (requests.length === 0) yield* Effect.yieldNow expect(requests).toHaveLength(1) expect(userTexts(requests[0]!)).toEqual(["Wait in queue"]) diff --git a/packages/core/test/skill/guidance.test.ts b/packages/core/test/skill/guidance.test.ts index 1c5b5b2839..c2816661d9 100644 --- a/packages/core/test/skill/guidance.test.ts +++ b/packages/core/test/skill/guidance.test.ts @@ -5,9 +5,9 @@ import { AgentV2 } from "@opencode-ai/core/agent" import { AppNodeBuilder } from "@opencode-ai/core/effect/app-node-builder" import { AbsolutePath } from "@opencode-ai/core/schema" import { SkillV2 } from "@opencode-ai/core/skill" -import { Instructions } from "@opencode-ai/core/instructions" import { SkillGuidance } from "@opencode-ai/core/skill/guidance" import { it } from "../lib/effect" +import { readInitial, readUpdate } from "../lib/instructions" const build = AgentV2.ID.make("build") const effect = SkillV2.Info.make({ @@ -45,7 +45,7 @@ const layer = (list: () => SkillV2.Info[]) => ]) describe("SkillGuidance", () => { - it.effect("renders described agent skills and reconciles the complete available list", () => { + it.effect("renders described agent skills and updates the complete available list", () => { const agent = AgentV2.Info.make({ ...AgentV2.Info.empty(build), permissions: [{ action: "skill", resource: "denied", effect: "deny" }], @@ -53,9 +53,7 @@ describe("SkillGuidance", () => { let skills = [hidden, denied, manual, effect] return Effect.gen(function* () { const guidance = yield* SkillGuidance.Service - const initialized = yield* guidance - .load({ id: agent.id, info: agent }) - .pipe(Effect.flatMap(Instructions.initialize)) + const initialized = yield* guidance.load({ id: agent.id, info: agent }).pipe(Effect.flatMap(readInitial)) expect(initialized.text).toBe( [ @@ -76,11 +74,8 @@ describe("SkillGuidance", () => { expect( yield* guidance .load({ id: agent.id, info: agent }) - .pipe(Effect.flatMap((context) => Instructions.reconcile(context, initialized.applied))), - ).toMatchObject({ - _tag: "Updated", - text: "The following skill IDs are no longer available and must not be used: effect.", - }) + .pipe(Effect.flatMap((context) => readUpdate(context, initialized))), + ).toMatchObject({ text: "Skill guidance is no longer available. Do not use any previously listed skill." }) }).pipe(Effect.provide(layer(() => skills))) }) @@ -96,17 +91,14 @@ describe("SkillGuidance", () => { let skills = [effect] return Effect.gen(function* () { const guidance = yield* SkillGuidance.Service - const initialized = yield* guidance - .load({ id: agent.id, info: agent }) - .pipe(Effect.flatMap(Instructions.initialize)) + const initialized = yield* guidance.load({ id: agent.id, info: agent }).pipe(Effect.flatMap(readInitial)) skills = [effect, debugging] const added = yield* guidance .load({ id: agent.id, info: agent }) - .pipe(Effect.flatMap((context) => Instructions.reconcile(context, initialized.applied))) - expect(added).toMatchObject({ - _tag: "Updated", - text: [ + .pipe(Effect.flatMap((context) => readUpdate(context, initialized))) + expect(added.text).toBe( + [ "New skills are available in addition to those previously listed:", " ", " debugging", @@ -114,18 +106,13 @@ describe("SkillGuidance", () => { " Diagnose hard bugs", " ", ].join("\n"), - }) + ) skills = [debugging] const removed = yield* guidance .load({ id: agent.id, info: agent }) - .pipe( - Effect.flatMap((context) => Instructions.reconcile(context, added._tag === "Updated" ? added.applied : {})), - ) - expect(removed).toMatchObject({ - _tag: "Updated", - text: "The following skill IDs are no longer available and must not be used: effect.", - }) + .pipe(Effect.flatMap((context) => readUpdate(context, added))) + expect(removed.text).toBe("The following skill IDs are no longer available and must not be used: effect.") }).pipe(Effect.provide(layer(() => skills))) }) @@ -134,17 +121,14 @@ describe("SkillGuidance", () => { let skills = [effect] return Effect.gen(function* () { const guidance = yield* SkillGuidance.Service - const initialized = yield* guidance - .load({ id: agent.id, info: agent }) - .pipe(Effect.flatMap(Instructions.initialize)) + const initialized = yield* guidance.load({ id: agent.id, info: agent }).pipe(Effect.flatMap(readInitial)) skills = [SkillV2.Info.make({ ...effect, description: "Build applications with Effect v4" })] expect( yield* guidance .load({ id: agent.id, info: agent }) - .pipe(Effect.flatMap((context) => Instructions.reconcile(context, initialized.applied))), + .pipe(Effect.flatMap((context) => readUpdate(context, initialized))), ).toMatchObject({ - _tag: "Updated", text: expect.stringContaining( "The available skills have changed. This list supersedes the previous available skills list.", ), @@ -159,12 +143,7 @@ describe("SkillGuidance", () => { }) return Effect.gen(function* () { const guidance = yield* SkillGuidance.Service - expect(yield* guidance.load({ id: agent.id, info: agent }).pipe(Effect.flatMap(Instructions.initialize))).toEqual( - { - text: "", - applied: {}, - }, - ) + expect((yield* guidance.load({ id: agent.id, info: agent }).pipe(Effect.flatMap(readInitial))).text).toBe("") }).pipe(Effect.provide(layer(() => [effect]))) }) @@ -178,12 +157,7 @@ describe("SkillGuidance", () => { }) return Effect.gen(function* () { const guidance = yield* SkillGuidance.Service - expect(yield* guidance.load({ id: agent.id, info: agent }).pipe(Effect.flatMap(Instructions.initialize))).toEqual( - { - text: "", - applied: {}, - }, - ) + expect((yield* guidance.load({ id: agent.id, info: agent }).pipe(Effect.flatMap(readInitial))).text).toBe("") }).pipe(Effect.provide(layer(() => [effect]))) }) @@ -197,9 +171,9 @@ describe("SkillGuidance", () => { }) return Effect.gen(function* () { const guidance = yield* SkillGuidance.Service - expect( - (yield* guidance.load({ id: agent.id, info: agent }).pipe(Effect.flatMap(Instructions.initialize))).text, - ).toContain("Effect") + expect((yield* guidance.load({ id: agent.id, info: agent }).pipe(Effect.flatMap(readInitial))).text).toContain( + "Effect", + ) }).pipe(Effect.provide(layer(() => [effect]))) }) @@ -214,12 +188,7 @@ describe("SkillGuidance", () => { }) return Effect.gen(function* () { const guidance = yield* SkillGuidance.Service - expect(yield* guidance.load({ id: agent.id, info: agent }).pipe(Effect.flatMap(Instructions.initialize))).toEqual( - { - text: "", - applied: {}, - }, - ) + expect((yield* guidance.load({ id: agent.id, info: agent }).pipe(Effect.flatMap(readInitial))).text).toBe("") }).pipe(Effect.provide(layer(() => [effect]))) }) }) diff --git a/packages/docs/build/plugins.mdx b/packages/docs/build/plugins.mdx index 0b609db5d2..08c8ce4dec 100644 --- a/packages/docs/build/plugins.mdx +++ b/packages/docs/build/plugins.mdx @@ -8,9 +8,8 @@ integrations, references, skills, and tools; intercept model requests and tool execution; and call a subset of the V2 client. - The V2 plugin API is beta. Entrypoints, hooks, draft shapes, and configuration - may change before the stable release. Use the `/v2` exports described on this - page. + The V2 plugin API is beta. Entrypoints, hooks, draft shapes, and configuration may change before the stable release. + Use the `/v2` exports described on this page. ## Load plugins @@ -34,10 +33,10 @@ Add ordered entries to the `plugins` field in `opencode.json(c)`: "package": "./plugins/reviewer.ts", "options": { "agent": "reviewer", - "strict": true - } - } - ] + "strict": true, + }, + }, + ], } ``` @@ -80,12 +79,7 @@ applied in order: ```jsonc title="opencode.jsonc" { - "plugins": [ - "./plugins/reviewer.ts", - "-acme.reviewer", - "-opencode.provider.*", - "opencode.provider.openai" - ] + "plugins": ["./plugins/reviewer.ts", "-acme.reviewer", "-opencode.provider.*", "opencode.provider.openai"], } ``` @@ -133,9 +127,7 @@ export default Plugin.define({ id: "acme.reviewer", setup: async (ctx) => { const description = - typeof ctx.options.description === "string" - ? ctx.options.description - : "Reviews code for regressions" + typeof ctx.options.description === "string" ? ctx.options.description : "Reviews code for regressions" await ctx.agent.transform((agents) => { agents.update("reviewer", (agent) => { @@ -175,22 +167,22 @@ Its read and action methods use the same inputs and responses as the client. It adds plugin-only methods for transforms, runtime hooks, reloads, registrations, and plugin options. -| Capability | Available operations | -| --- | --- | -| `ctx.agent` | `list`, `transform`, `reload` | -| `ctx.catalog.provider` | `list`, `get` | -| `ctx.catalog.model` | `list`, `default` | -| `ctx.catalog` | `transform`, `reload` | -| `ctx.command` | `list`, `transform`, `reload` | -| `ctx.integration` | `list`, `get`, `connect`, `attempt`, `transform`, `reload`, and connection lookup/resolution | -| `ctx.plugin` | `list` currently active plugin IDs | -| `ctx.reference` | `list`, `transform`, `reload` | -| `ctx.session` | `create`, `get`, `prompt`, `command`, `interrupt`, and `hook` | -| `ctx.skill` | `list`, `transform`, `reload` | -| `ctx.tool` | `transform` and `hook` | -| `ctx.aisdk` | `hook` | -| `ctx.event` | `subscribe` to the current public server event stream | -| `ctx.options` | Readonly options from the matching config object | +| Capability | Available operations | +| ---------------------- | -------------------------------------------------------------------------------------------- | +| `ctx.agent` | `list`, `transform`, `reload` | +| `ctx.catalog.provider` | `list`, `get` | +| `ctx.catalog.model` | `list`, `default` | +| `ctx.catalog` | `transform`, `reload` | +| `ctx.command` | `list`, `transform`, `reload` | +| `ctx.integration` | `list`, `get`, `connect`, `attempt`, `transform`, `reload`, and connection lookup/resolution | +| `ctx.plugin` | `list` currently active plugin IDs | +| `ctx.reference` | `list`, `transform`, `reload` | +| `ctx.session` | `create`, `get`, `prompt`, `command`, `interrupt`, and `hook` | +| `ctx.skill` | `list`, `transform`, `reload` | +| `ctx.tool` | `transform` and `hook` | +| `ctx.aisdk` | `hook` | +| `ctx.event` | `subscribe` to the current public server event stream | +| `ctx.options` | Readonly options from the matching config object | ### Transform hooks @@ -198,15 +190,15 @@ Transform hooks let a plugin modify how OpenCode is configured. Use them to add or remove definitions, override settings, choose defaults, and provide tools or other sources. -| Transform | Draft operations | -| --- | --- | -| `agent.transform` | `list`, `get`, `default`, `update`, `remove` | -| `catalog.transform` | Provider `list`, `get`, `update`, `remove`; model `get`, `update`, `remove`; default model `get`, `set` | -| `command.transform` | `list`, `get`, `update`, `remove` | -| `integration.transform` | Integration `list`, `get`, `update`, `remove`; method `list`, `update`, `remove` | -| `reference.transform` | `add`, `remove`, `list` | -| `skill.transform` | `source`, `list` | -| `tool.transform` | `add` | +| Transform | Draft operations | +| ----------------------- | ------------------------------------------------------------------------------------------------------- | +| `agent.transform` | `list`, `get`, `default`, `update`, `remove` | +| `catalog.transform` | Provider `list`, `get`, `update`, `remove`; model `get`, `update`, `remove`; default model `get`, `set` | +| `command.transform` | `list`, `get`, `update`, `remove` | +| `integration.transform` | Integration `list`, `get`, `update`, `remove`; method `list`, `update`, `remove` | +| `reference.transform` | `add`, `remove`, `list` | +| `skill.transform` | `source`, `list` | +| `tool.transform` | `add` | Here's an example that keeps models synced from a remote source: @@ -250,13 +242,13 @@ without restarting OpenCode. Runtime hooks intercept live operations. Their event objects expose specific mutable fields: -| Hook | Mutable fields | -| --- | --- | -| `ctx.aisdk.hook("sdk", callback)` | `sdk`, after inspecting `model`, `package`, and `options` | -| `ctx.aisdk.hook("language", callback)` | `language`, after inspecting `model`, `sdk`, and `options` | -| `ctx.session.hook("request", callback)` | `system`, `messages`, and the `tools` record immediately before model dispatch | -| `ctx.tool.hook("execute.before", callback)` | `input`, before the selected tool executes | -| `ctx.tool.hook("execute.after", callback)` | `result`, `output`, and `outputPaths`, after execution settles | +| Hook | Mutable fields | +| ------------------------------------------- | ------------------------------------------------------------------------------ | +| `ctx.aisdk.hook("sdk", callback)` | `sdk`, after inspecting `model`, `package`, and `options` | +| `ctx.aisdk.hook("language", callback)` | `language`, after inspecting `model`, `sdk`, and `options` | +| `ctx.session.hook("request", callback)` | `system`, `messages`, and the `tools` record immediately before model dispatch | +| `ctx.tool.hook("execute.before", callback)` | `input`, before the selected tool executes | +| `ctx.tool.hook("execute.after", callback)` | `result`, `output`, and `outputPaths`, after execution settles | For example, remove a tool from selected model requests and normalize another tool's input: diff --git a/packages/docs/compaction.mdx b/packages/docs/compaction.mdx index 1a43bbbcd9..dd338e6929 100644 --- a/packages/docs/compaction.mdx +++ b/packages/docs/compaction.mdx @@ -114,22 +114,24 @@ and file or media attachments become textual descriptors rather than embedded data. On later compactions, V2 updates the previous summary and carries forward its retained recent context before selecting a new tail. -The completed checkpoint is presented to the model as historical conversation +The completed compaction is presented to the model as historical conversation context, explicitly not as new instructions. Running and failed compactions are not included in model context. -## Instructions and checkpoints +## Compaction advances the instruction epoch -Conversation compaction and the durable instruction checkpoint are separate. -Before each model step, V2 compares live instruction sources with what that -session's model was last told. Ordinary changes are durable chronological -system updates and do not rewrite the established instruction baseline. +Conversation compaction and instruction synchronization are separate. Before +promoting pending input, V2 compares live instruction sources with the latest +admitted values. Ordinary changes become durable value deltas; their +model-facing System messages are derived during request assembly rather than +persisted. -After a completed compaction, the next model step creates a fresh instruction -baseline from current sources. If a source is temporarily unavailable, V2 -restates the last-applied value instead of treating it as removed. Session -movement and a committed revert reset the instruction checkpoint as well. See -[Instructions](/instructions) for source ordering and update behavior. +Completed compaction advances the instruction epoch at the exact ended-event +sequence and makes the currently admitted values initial. It does not reread +sources or publish an instruction event. Session movement and committed revert +clear the instruction fold so the next safe boundary requires one complete +source read. See [Instructions](/instructions) for source ordering and update +behavior. ## Current limitations diff --git a/packages/docs/instructions.mdx b/packages/docs/instructions.mdx index b928a835d9..e7bebb42cc 100644 --- a/packages/docs/instructions.mdx +++ b/packages/docs/instructions.mdx @@ -5,8 +5,9 @@ description: "" Instructions are privileged context that guide an agent throughout a session. V2 combines built-in context, discovered `AGENTS.md` files, and dynamic sources -such as skill, reference, MCP, and session context into a durable instruction -baseline. +such as skill, reference, MCP, and session context. It stores source values as +durable deltas, then renders initial instructions and chronological updates when +assembling each model request. ## AGENTS.md @@ -92,7 +93,7 @@ See [Config](/config) for config locations and general precedence. ## Ordering The selected agent or provider system prompt is sent first. OpenCode then sends -the session's instruction baseline, composed in this order: +the session's initial instructions, composed in this order: 1. Built-in environment and date context. 2. Ambient `AGENTS.md` discovery. @@ -101,24 +102,27 @@ the session's instruction baseline, composed in this order: These sources are combined; ordering is not an override mechanism. Nested `AGENTS.md` files discovered by reads are chronological session entries rather -than part of the baseline. +than part of the initial instructions. ## Changes -Before each model step, V2 compares live instruction sources with what that -session's model was last told: +Before promoting pending input, V2 compares live instruction sources with the +latest admitted source values: - A new or changed ambient `AGENTS.md` aggregate is announced as a system update that replaces the previous ambient aggregate. - Removing all ambient files announces that the previous ambient instructions no longer apply. - A temporary read or discovery failure preserves the session's last known - instructions instead of treating them as deleted. If no baseline exists yet, - the first model step waits until required sources are available. -- Completed conversation compaction creates a fresh baseline from the current - sources. Moving a session or committing a revert also resets its instruction - checkpoint so the next step establishes a new baseline. + instructions instead of treating them as deleted. If no instruction epoch + exists yet, pending input waits until every source is available. +- Completed conversation compaction advances the instruction epoch, making the + currently admitted values initial without rereading sources or authoring an + instruction event. +- Moving a session or committing a revert clears the instruction fold. The next + safe boundary requires one complete source read before promoting input. -Updates are durable session history. OpenCode does not rewrite the original -baseline on every change; it records the change so subsequent model steps see -both the established baseline and the chronological update. +The durable event stores changed source keys and value hashes, not rendered +prose. During request assembly, OpenCode renders the epoch's initial values and +interleaves later changes as chronological System messages. Clients see changed +keys but never the privileged value bodies. diff --git a/packages/protocol/src/groups/session.ts b/packages/protocol/src/groups/session.ts index 19499b7709..2e6f2ddb8c 100644 --- a/packages/protocol/src/groups/session.ts +++ b/packages/protocol/src/groups/session.ts @@ -518,7 +518,7 @@ export const makeSessionGroup = (sessionLo params: { sessionID: Session.ID, key: InstructionEntry.Key }, payload: Schema.Struct({ value: Schema.Json }), success: HttpApiSchema.NoContent, - error: SessionNotFoundError, + error: [SessionNotFoundError, InstructionEntry.ValueTooLargeError], }) .middleware(sessionLocationMiddleware) .annotateMerge( diff --git a/packages/schema/src/durable-event-manifest.ts b/packages/schema/src/durable-event-manifest.ts index 8f8870f581..73c8505062 100644 --- a/packages/schema/src/durable-event-manifest.ts +++ b/packages/schema/src/durable-event-manifest.ts @@ -5,8 +5,8 @@ import { SessionEvent } from "./session-event.js" import { SessionV1 } from "./session-v1.js" export const SessionDurable = { - definitions: Event.durableMap(SessionEvent.Definitions), + definitions: Event.durableMap(SessionEvent.DurableDefinitions), schema: SessionEvent.Durable, } as const -export const Durable = Event.durableMap([...SessionV1.Event.Definitions, ...SessionEvent.Definitions]) +export const Durable = Event.durableMap([...SessionV1.Event.Definitions, ...SessionEvent.DurableDefinitions]) diff --git a/packages/schema/src/instruction-entry.ts b/packages/schema/src/instruction-entry.ts index 09c7e39de1..6e25e9cbaf 100644 --- a/packages/schema/src/instruction-entry.ts +++ b/packages/schema/src/instruction-entry.ts @@ -18,3 +18,15 @@ export const Info = Schema.Struct({ value: Schema.Json.annotate({ description: "JSON value attached to the session's instructions" }), }).annotate({ identifier: "InstructionEntry.Info" }) export interface Info extends Schema.Schema.Type {} + +export const MaxValueBytes = 8 * 1024 + +export class ValueTooLargeError extends Schema.TaggedErrorClass()( + "InstructionEntryValueTooLargeError", + { + actualBytes: Schema.Int, + maxBytes: Schema.Int, + message: Schema.String, + }, + { httpApiStatus: 413 }, +) {} diff --git a/packages/schema/src/instruction.ts b/packages/schema/src/instruction.ts new file mode 100644 index 0000000000..a2c1e93a1c --- /dev/null +++ b/packages/schema/src/instruction.ts @@ -0,0 +1,21 @@ +export * as Instruction from "./instruction.js" + +import { Schema } from "effect" + +export const Key = Schema.String.check(Schema.isPattern(/^[a-z0-9][a-z0-9._-]*\/[a-z0-9][a-z0-9._/-]*$/)).pipe( + Schema.brand("Instruction.Key"), +) +export type Key = typeof Key.Type + +export const Hash = Schema.String.check(Schema.isPattern(/^[a-f0-9]{64}$/)).pipe(Schema.brand("Instruction.Hash")) +export type Hash = typeof Hash.Type + +export const Values = Schema.Record(Key, Hash) +export type Values = Readonly> + +export const Removed = Schema.Literal("removed") +export type Removed = typeof Removed.Type +export const removed = Removed.make("removed") + +export const Delta = Schema.Record(Schema.String, Schema.Union([Hash, Removed])) +export type Delta = typeof Delta.Type diff --git a/packages/schema/src/session-event.ts b/packages/schema/src/session-event.ts index b2b6ec08ef..61341168ce 100644 --- a/packages/schema/src/session-event.ts +++ b/packages/schema/src/session-event.ts @@ -14,6 +14,7 @@ import { SessionMessage } from "./session-message.js" import { Revert } from "./session-revert.js" import { Shell as ShellSchema } from "./shell.js" import { SessionError } from "./session-error.js" +import { Instruction } from "./instruction.js" import { Agent } from "./agent.js" import { Skill as SkillSchema } from "./skill.js" import { Money } from "./money.js" @@ -105,10 +106,14 @@ export type Deleted = typeof Deleted.Type export const Forked = Event.durable({ type: "session.forked", - ...options, + durable: { + aggregate: "sessionID", + version: 2, + }, schema: { ...Base, parentID: SessionID, + parentSeq: Schema.Int.check(Schema.isGreaterThanOrEqualTo(-1)), from: SessionMessage.ID.pipe(optional), }, }) @@ -159,10 +164,13 @@ export namespace Execution { export const InstructionsUpdated = Event.durable({ type: "session.instructions.updated", - ...options, + durable: { + aggregate: "sessionID", + version: 2, + }, schema: { ...Base, - text: Schema.String, + delta: Instruction.Delta, }, }) export type InstructionsUpdated = typeof InstructionsUpdated.Type diff --git a/packages/schema/test/event-manifest.test.ts b/packages/schema/test/event-manifest.test.ts index 80d179ae25..0df33ec8c3 100644 --- a/packages/schema/test/event-manifest.test.ts +++ b/packages/schema/test/event-manifest.test.ts @@ -104,14 +104,14 @@ describe("public event manifest", () => { "session.model.selected.1", "session.moved.1", "session.renamed.1", - "session.forked.1", + "session.forked.2", "session.input.promoted.1", "session.input.admitted.1", "session.execution.started.1", "session.execution.succeeded.1", "session.execution.failed.1", "session.execution.interrupted.1", - "session.instructions.updated.1", + "session.instructions.updated.2", "session.synthetic.1", "session.skill.activated.1", "session.shell.started.1", diff --git a/packages/sdk/js/src/v2/gen/types.gen.ts b/packages/sdk/js/src/v2/gen/types.gen.ts index 56a9bff6b6..cd391aab58 100644 --- a/packages/sdk/js/src/v2/gen/types.gen.ts +++ b/packages/sdk/js/src/v2/gen/types.gen.ts @@ -829,6 +829,7 @@ export type GlobalEvent = { properties: { sessionID: string parentID: string + parentSeq: number from?: string } } @@ -884,7 +885,9 @@ export type GlobalEvent = { type: "session.instructions.updated" properties: { sessionID: string - text: string + delta: { + [key: string]: string | "removed" + } } } | { @@ -2880,6 +2883,13 @@ export type UnknownError1 = { ref?: string } +export type InstructionEntryValueTooLargeError = { + _tag: "InstructionEntryValueTooLargeError" + actualBytes: number + maxBytes: number + message: string +} + export type Shell1 = { id: string status: "running" | "exited" | "timeout" | "killed" @@ -3787,13 +3797,14 @@ export type SyncEventSessionForked = { type: "sync" id: string syncEvent: { - type: "session.forked.1" + type: "session.forked.2" id: string seq: number aggregateID: string data: { sessionID: string parentID: string + parentSeq: number from?: string } } @@ -3892,13 +3903,15 @@ export type SyncEventSessionInstructionsUpdated = { type: "sync" id: string syncEvent: { - type: "session.instructions.updated.1" + type: "session.instructions.updated.2" id: string seq: number aggregateID: string data: { sessionID: string - text: string + delta: { + [key: string]: string | "removed" + } } } } @@ -4866,12 +4879,13 @@ export type SessionForked = { durable: { aggregateID: string seq: number - version: 1 + version: 2 } location?: LocationRef data: { sessionID: string parentID: string + parentSeq: number from?: string } } @@ -4999,12 +5013,14 @@ export type SessionInstructionsUpdated = { durable: { aggregateID: string seq: number - version: 1 + version: 2 } location?: LocationRef data: { sessionID: string - text: string + delta: { + [key: string]: string | "removed" + } } } @@ -7171,6 +7187,7 @@ export type EventSessionForked = { properties: { sessionID: string parentID: string + parentSeq: number from?: string } } @@ -7233,7 +7250,9 @@ export type EventSessionInstructionsUpdated = { type: "session.instructions.updated" properties: { sessionID: string - text: string + delta: { + [key: string]: string | "removed" + } } } @@ -9096,12 +9115,13 @@ export type SessionForkedV2 = { durable: { aggregateID: string seq: number - version: 1 + version: 2 } location?: LocationRefV2 data: { sessionID: string parentID: string + parentSeq: number from?: string } } @@ -9258,12 +9278,14 @@ export type SessionInstructionsUpdatedV2 = { durable: { aggregateID: string seq: number - version: 1 + version: 2 } location?: LocationRefV2 data: { sessionID: string - text: string + delta: { + [key: string]: string | "removed" + } } } @@ -16253,6 +16275,10 @@ export type V2SessionInstructionsEntryPutErrors = { * SessionNotFoundError */ 404: SessionNotFoundError + /** + * InstructionEntryValueTooLargeError + */ + 413: InstructionEntryValueTooLargeError } export type V2SessionInstructionsEntryPutError = diff --git a/packages/tui/src/context/data.tsx b/packages/tui/src/context/data.tsx index b3a4c60eca..59f15d74d6 100644 --- a/packages/tui/src/context/data.tsx +++ b/packages/tui/src/context/data.tsx @@ -358,7 +358,7 @@ export const { use: useData, provider: DataProvider } = createSimpleContext({ message.append(draft, index, { id: messageIDFromEvent(event.id), type: "system", - text: event.data.text, + text: `Instructions updated: ${Object.keys(event.data.delta).join(", ")}`, metadata: event.metadata, time: { created: event.created }, }) @@ -736,9 +736,7 @@ export const { use: useData, provider: DataProvider } = createSimpleContext({ if (store.session.form[event.data.form.sessionID]?.some((form) => form.id === event.data.form.id)) break setStore("session", "form", event.data.form.sessionID, [ ...(store.session.form[event.data.form.sessionID] ?? []), - event.data.form.sessionID === "global" - ? { ...event.data.form, location: event.location } - : event.data.form, + event.data.form.sessionID === "global" ? { ...event.data.form, location: event.location } : event.data.form, ]) break case "form.replied": diff --git a/packages/tui/src/routes/session/index.tsx b/packages/tui/src/routes/session/index.tsx index 9f71e3e943..13c200101a 100644 --- a/packages/tui/src/routes/session/index.tsx +++ b/packages/tui/src/routes/session/index.tsx @@ -268,15 +268,13 @@ export function Session() { navigate({ type: "home" }) return } - void data.session.form - .refresh("global", info.location) - .catch((error) => - toast.show({ - message: `Failed to refresh global forms: ${errorMessage(error)}`, - variant: "error", - duration: 5000, - }), - ) + void data.session.form.refresh("global", info.location).catch((error) => + toast.show({ + message: `Failed to refresh global forms: ${errorMessage(error)}`, + variant: "error", + duration: 5000, + }), + ) project.workspace.set(info.location.workspaceID) editor.reconnect(info.location.directory) if (route.sessionID === sessionID && scroll) scroll.scrollBy(100_000) @@ -761,12 +759,7 @@ export function Session() { const sessionData = session() if (!sessionData) return - const options = await DialogExportOptions.show( - dialog, - showThinking(), - showDetails(), - showAssistantMetadata(), - ) + const options = await DialogExportOptions.show(dialog, showThinking(), showDetails(), showAssistantMetadata()) if (options === null) return @@ -1312,7 +1305,7 @@ function SessionSwitchMessageV2(props: { message: SessionMessageInfo }) { function SessionNoticeMessageV2(props: { message: SessionMessageInfo }) { const { theme } = useTheme() const text = () => { - if (props.message.type === "system") return "Instructions updated" + if (props.message.type === "system") return props.message.text if (props.message.type === "synthetic") return props.message.description ?? "" return "" } diff --git a/packages/tui/test/cli/tui/data.test.tsx b/packages/tui/test/cli/tui/data.test.tsx index f2f996bf5e..fade50e722 100644 --- a/packages/tui/test/cli/tui/data.test.tsx +++ b/packages/tui/test/cli/tui/data.test.tsx @@ -2129,10 +2129,10 @@ test("projects live instruction updates with their message ID", async () => { id: "evt_instructions_1", created: 0, type: "session.instructions.updated", - durable: durable("session-1"), + durable: durable("session-1", 0, 2), data: { sessionID: "session-1", - text: "Updated instructions", + delta: { "core/date": "0".repeat(64) }, }, }) @@ -2140,7 +2140,7 @@ test("projects live instruction updates with their message ID", async () => { expect(sync.session.message.list("session-1")?.[0]).toMatchObject({ id: SessionMessage.ID.fromEvent(EventV2.ID.make("evt_instructions_1")), type: "system", - text: "Updated instructions", + text: "Instructions updated: core/date", time: { created: 0 }, }) } finally { diff --git a/specs/v2/README.md b/specs/v2/README.md index 34d4b6cf1e..b46edec8a8 100644 --- a/specs/v2/README.md +++ b/specs/v2/README.md @@ -30,6 +30,7 @@ Generated clients follow the assembled public `HttpApi`. GitHub issues own activ | Document | Status | Job | | ----------------------------------------------------------------- | -------------------------- | ---------------------------------------------------------------------------- | | [Managed restart continuation](./session-restart-continuation.md) | Accepted and implemented | Record why graceful managed-service restart uses private Session suspension. | +| [Instruction sync](./instruction-sync-proposal.md) | Accepted and implemented | Record why instruction state is value deltas plus derived rendering. | | [Provider policy](./provider-policy.md) | Proposed and unimplemented | Explore provider authorization independently from provider configuration. | ## Historical Context diff --git a/specs/v2/instruction-sync-proposal.md b/specs/v2/instruction-sync-proposal.md new file mode 100644 index 0000000000..29164a4583 --- /dev/null +++ b/specs/v2/instruction-sync-proposal.md @@ -0,0 +1,154 @@ +# Instruction Sync: V2 Architecture + +Status: implemented on `instruction-sync-v2` (2026-07-10). + +## Principle + +The model is a replica that OpenCode can write but cannot read or edit. The transcript is the one-way channel. Instruction sync keeps mutable privileged context (`AGENTS.md`, guidance, API entries, date, and environment) current over that channel without rewriting text that was already sent. + +**The durable log stores only irreducible facts: which source values changed, and when. Everything else is a function of the log and current renderer code.** + +## Durable Fact + +```typescript +"session.instructions.updated.2" { + sessionID: Session.ID + delta: Record +} +``` + +A hash overwrites one source value. The literal `"removed"` removes it (chosen over JSON `null` because record-value nullability does not survive every client generator; it cannot collide with a 64-hex hash). The event stores no rendered text, mode, baseline, or snapshot. + +Each hash body is canonical JSON stored once in the machine-local `instruction_blob` table. Hashes are local pointers, not cross-machine promises. + +## Epochs And Folds + +An instruction epoch is the span between completed compactions. `epochStart` is the sequence of the last `session.compaction.ended`, or the initial complete v2 delta when no epoch exists. + +Folding deltas in durable sequence order derives: + +```text +values through epochStart -> renderInitial -> initial instructions +each delta after epochStart -> renderUpdate -> chronological System message +final values -> next boundary comparison state +``` + +Completed compaction moves the epoch by copying current hashes to initial hashes at the exact ended-event sequence. It does not read sources or publish an instruction event. + +Session movement and committed revert clear the fold. The next boundary must establish one complete delta before input promotion. + +## Projection Cache + +```text +instruction_state + session_id + epoch_start + through_seq + initial_values + current_values +``` + +This row is derived state. The boundary compares `through_seq` with the latest relevant durable sequence. A missing or stale row folds the log and rewrites the cache without publishing an event. + +The relevant reducer inputs are: + +- `session.instructions.updated.2`: apply the delta; the first one establishes an epoch, including an empty complete delta. +- `session.compaction.ended.1`: make current values initial and move `epochStart`. +- `session.moved.1`: clear values. +- `session.revert.committed.1`: clear values. +- `session.forked.2`: derive from parent ancestry through its frozen `parentSeq`. + +## Sources + +```typescript +interface Source { + readonly key: Key + readonly read: Effect + readonly initial: (value: Json) => string | undefined + readonly changed: (previous: Json, current: Json) => string | undefined + readonly removed: (previous: Json) => string | undefined +} + +namespace Source { + interface Definition { + readonly key: Key + readonly codec: Schema.Codec + readonly read: Effect + readonly render: { + readonly initial: (value: A) => string + readonly changed: (previous: A, current: A) => string + readonly removed?: (previous: A) => string + } + } +} + +type Instructions = ReadonlyArray + +declare function make(definition: Source.Definition): Instructions +``` + +Producers author a typed `Source.Definition`. `make` captures its codec and renderers in one JSON-level `Source`, the representation used for heterogeneous composition, durable values, and historical rendering. `Instructions` is an ordered collection of those sources; combining collections preserves order and rejects duplicate keys. + +`read` runs once per source at the safe boundary, never at layer construction or request assembly. Codecs must be canonical: object keys are canonicalized by the hash function, while source-owned collections must have deterministic order and values must not contain observation timestamps. + +`Unavailable` means the read failed temporarily. The initial complete delta blocks while any source is unavailable; later boundaries retain its prior hash silently. + +`Removed` is an observed absence. If the key currently has a value, the next delta stores `"removed"` and assembly calls the source's removal renderer. A source that disappears from a software upgrade does not imply removal; its retained value becomes invisible while its renderer is absent. + +## Safe Boundary + +Once per physical attempt, before input promotion: + +1. Load the selected agent and compose built-ins, discovery, skill guidance, reference guidance, MCP guidance, and API entries in fixed order. +2. Read every source concurrently exactly once. +3. Encode and hash values; compare with `instruction_state.current_values`. +4. At the initial v2 boundary, require a complete read and admit one complete delta, including `{}` for a truly empty set. +5. For later boundaries, insert new blobs and admit one delta only when a hash or explicit removal changed. +6. Promote pending input. +7. Read projected messages, epoch values, blobs, and post-epoch deltas in one database transaction. +8. Render initial instructions and interleave derived update messages by durable sequence. + +`MoveSession` interrupts any active drain and awaits idle before publishing `session.moved`, matching the best-effort ordering used by Session removal. + +The blob inserts, durable event, and fold-cache advance share the event transaction. + +## Forks + +`session.forked.2` carries `parentSeq`, the authoritative parent event cutoff. For a fork before message N, the cutoff is `message.seq - 1`, so instruction changes admitted immediately before that message are inherited while later parent state is not. + +The child stores the cutoff as `session.fork_seq`. Its virtual instruction log is the parent's ancestry through that cutoff followed by child events. Child event sequence reservation begins after the cutoff, preserving chronological interleaving with copied message rows. Replay accepts the intentional fork gap because the fork projector reserves the inherited prefix before later child events replay. + +## API Entries + +Each visible entry is one `api/` source. DELETE marks the row as a hidden tombstone rather than physically removing it, preserving the renderer needed to admit and narrate the removal; list responses hide tombstones. The nullable value column preserves JSON `null`, while the separate tombstone flag distinguishes removal. A later PUT revives the same source. + +PUT measures encoded JSON in UTF-8 and rejects values larger than 8KB with `InstructionEntryValueTooLargeError` (HTTP 413). Values are never truncated. + +## Content-Addressed Storage + +The blob store grows by one row per distinct encoded value. No GC ships initially. This is an at-rest deduplication policy; deleting a Session does not remove values only that Session referenced, so clients must not put secrets in API entries. + +If retention becomes necessary, add mark-and-sweep: walk live v2 deltas for referenced hashes and delete the rest. No schema change or eager reference counter is required. + +The blob store is machine/tenant scoped and must never deduplicate across tenants. + +**Storage format is not wire format.** Any future V2 sync, export, share, or workspace-transfer boundary must hydrate referenced values, verify each body against its hash on ingestion, and insert blobs before replaying the event. Current V2 has no cross-machine durable replay surface; hashes are sufficient for local event logs and key-only clients. + +## Client Projection + +Instruction deltas do not project `session_message` rows. The TUI derives a non-model-facing notice from event keys, for example `Instructions updated: core/date, api/plan`. Model-facing update prose exists only during runner assembly and is excluded from compaction summaries. + +## Migration + +Migration deletes pre-beta `session.instructions.updated.1` events and their event-derived System rows, then drops `instruction_checkpoint`. It leaves unrelated events and System messages intact. The next safe boundary establishes one complete v2 delta. + +Existing `session.forked.1` rows migrate to v2 with the event prefix reserved by their original projection as `parentSeq`. + +## Accepted Costs + +- Renderer changes can change request bytes for identical stored values, causing one provider-cache miss. They do not create an instruction delta. +- Rendered text is not retained verbatim. +- Source additions or software removals are silent unless a source explicitly reads `Removed`. +- Clients display changed keys, not privileged prose. +- Blob GC is deferred. +- Pre-beta instruction events are deleted during migration; logs with resulting sequence gaps are not guaranteed to replay into a blank database. diff --git a/specs/v2/schema-changelog.md b/specs/v2/schema-changelog.md index 005b6c9822..4aae4391bd 100644 --- a/specs/v2/schema-changelog.md +++ b/specs/v2/schema-changelog.md @@ -2,6 +2,20 @@ Status: **Historical pre-release compatibility ledger.** Older entries retain the names and behavior that were accurate when written; current contracts live in Protocol, Schema, Core, and the indexed specifications. +## 2026-07-10: Replace Instruction Checkpoints With Value Deltas + +- Replace rendered `session.instructions.updated.1` prose with `session.instructions.updated.2 { delta }`, where values are SHA-256 hashes and the literal `"removed"` means removal. +- Add the global `instruction_blob` content-addressed value store and rebuildable per-Session `instruction_state` fold cache. Drop `instruction_checkpoint` and its stored baseline, snapshot, and baseline sequence. +- Add an authoritative parent event sequence to `session.forked.2` so forks derive instruction values from the selected parent prefix rather than copying the parent's latest state. +- Keep removed API entries as hidden source tombstones so the next safe boundary can admit and render a revocation. Reject API entry JSON larger than 8KB with a typed HTTP 413 error. +- Render initial instructions and chronological updates from stored values during request assembly. Instruction update prose is no longer a `session_message` row or compaction input; clients display changed keys from the durable delta. + +Compatibility: + +- Delete pre-beta `session.instructions.updated.1` events and their event-derived System rows instead of carrying a legacy event schema. The next safe boundary establishes one complete v2 delta. +- Existing forks are assigned the sequence reserved by their original projection and their durable event is rewritten to v2 with that cutoff. +- Blob GC is intentionally deferred. A future sync/export boundary must hydrate referenced blobs on the wire and re-hash them on ingestion. + ## 2026-07-09: Make Session Input Storage Pending-Only And Rename It To Session Pending - Rename the `SessionInput` schema namespace to `SessionPending` and the `session_input` table to `session_pending`. diff --git a/specs/v2/session.md b/specs/v2/session.md index d13cf027bb..763d580719 100644 --- a/specs/v2/session.md +++ b/specs/v2/session.md @@ -59,19 +59,19 @@ Each retry attempt is a distinct Step, consumes the selected agent's allowance, A normalized content-filter finish fails the Step. Any partial streamed content remains visible. -## Instructions Are Session-Owned +## Instructions Are Value Deltas -`InstructionCheckpoint` stores the exact instruction baseline last shown to the model and an `Instructions.Applied` record for each source. The runner explicitly combines built-ins, ambient discovery, selected-agent skill guidance, references, MCP guidance, and API-managed instruction entries. There is no instruction registry. +Instruction sync persists values, never rendered privileged prose. The only durable fact is `session.instructions.updated { delta }`, mapping each changed source key to a SHA-256 content hash, with the literal `"removed"` for observed absence. Canonical JSON bodies live once in the machine-local `instruction_blob` store; `instruction_state` is a rebuildable fold cache, never primary state. The runner explicitly combines built-ins, ambient discovery, selected-agent skill guidance, references, MCP guidance, and API-managed instruction entries. There is no instruction registry. -The first complete observation establishes the baseline before input promotion. Later changes publish chronological `session.instructions.updated` messages and advance applied state atomically. An unavailable source preserves the model's previous belief; it blocks only a Session that has never established a complete baseline. +At each Safe Step Boundary the runner reads every source concurrently exactly once, hashes encoded values, and admits one delta atomically with its new blobs before input promotion. The initial delta must be complete; an unavailable source blocks only that initial delta and otherwise silently retains the stored value. Initial instructions and chronological update messages are rendered from stored values during request assembly and are never persisted; clients display changed keys. -Completed compaction rebaselines instructions. Session movement and committed revert reset the checkpoint. Model selection affects request assembly but is not itself an instruction source. +An instruction epoch spans completed compactions. `session.compaction.ended` moves the epoch start to its exact sequence, making current values initial, without reading sources or authoring an instruction event. Session movement and committed revert clear the fold. Forks record an authoritative parent sequence and derive values from the parent's ancestry through that cutoff. Model selection affects request assembly but is not itself an instruction source. See the [instruction sync design](./instruction-sync-proposal.md). ## Compaction Rebuilds Active History Before each Step, the runner estimates the complete model-visible request against the selected model's context window and reserved output headroom. When compaction is enabled, model limits are known, and enough older Session History is available, the runner may store a structured rolling summary plus bounded recent context instead of sending an over-budget request. -The full transcript remains durable. Active model history after the checkpoint contains the summary and retained recent context; provider-native continuation state does not cross the compaction boundary. +The full transcript remains durable. Active model history after the compaction boundary contains the summary and retained recent context; provider-native continuation state does not cross that boundary. If the provider reports context overflow before durable assistant output or tool execution, the runner may perform one overflow-triggered compaction and rebuild the same logical Step. A second overflow or any overflow after durable output is terminal.