diff --git a/docs/cli/doctor.md b/docs/cli/doctor.md index b5c36ca0987..d4d82a06161 100644 --- a/docs/cli/doctor.md +++ b/docs/cli/doctor.md @@ -203,7 +203,13 @@ the container normally. the canonical shared state database at `/state/openclaw.sqlite`. It does not accept an arbitrary database path, is never invoked by normal Gateway operation, and is not part of -`openclaw doctor --fix`. +`openclaw doctor --fix`. The command acquires the same state ownership lock as +Gateway startup and holds it through validation, checkpointing, `VACUUM`, and +the final integrity checks. It refuses to run while a Gateway or another +SQLite maintenance command owns that lock. The state lock remains active when +`OPENCLAW_ALLOW_MULTI_GATEWAY=1` skips the per-config Gateway singleton, so an +operator shell does not need to inherit the Gateway service's environment for +maintenance to detect it. Stop the Gateway and create a verified backup first: @@ -293,8 +299,16 @@ transcripts, run `openclaw doctor --session-sqlite compact --session-sqlite-all- to checkpoint WAL files, run `VACUUM`, and report before/after database and WAL sizes. Compaction requires a regular file with the current agent schema, the selected agent's durable owner metadata, and no open handle in the doctor -process. This is explicit offline maintenance: stop the Gateway first so normal -writes cannot race the checkpoint or `VACUUM`. +process. The destructive `import`, `compact`, `recover`, and `restore` modes +hold the same state ownership lock as Gateway startup for their full operation; +`inspect`, `dry-run`, and `validate` remain read-only and do not take it. Stop +the Gateway first. Destructive modes fail instead of racing live writes or +racing another maintenance command. A destructive `--session-sqlite-store` +target must be inside the active state directory; set `OPENCLAW_STATE_DIR` to +the store's owning state directory before maintaining another installation. +Existing hard-linked targets are rejected because another path can share the +same database inode outside the locked state directory. The same ownership +checks cover SQLite WAL, shared-memory, and rollback-journal sidecars. Each import writes a manifest under `~/.openclaw/session-sqlite-migration-runs/` before moving transcript artifacts diff --git a/docs/gateway/gateway-lock.md b/docs/gateway/gateway-lock.md index e6586d71df6..40b7dedfcc5 100644 --- a/docs/gateway/gateway-lock.md +++ b/docs/gateway/gateway-lock.md @@ -8,23 +8,26 @@ title: "Gateway lock" ## Why -- Only one gateway process should own a given config + port on a host; run additional gateways with isolated profiles and unique ports. +- Only one gateway process should own a state directory; run additional gateways with isolated profiles, state directories, configs, and ports. - Survive crashes/SIGKILL without leaving stale lock files behind. - Fail fast with a clear error when another gateway already owns the port. -## Two layers +## Three layers -Startup enforces single-instance ownership in two independent steps, in order: +Startup enforces ownership in three steps, in order: -1. **File lock** acquires a per-config lock file under the state lock directory. As part of acquiring it, startup probes the configured port for a live listener to detect a stale (crashed) lock owner. -2. **Socket bind** binds the HTTP/WebSocket listener (default `ws://127.0.0.1:18789`) as an exclusive TCP listener. +1. **State ownership lock** acquires a lock keyed by the canonical state directory. Every Gateway participates, including Gateways started with `OPENCLAW_ALLOW_MULTI_GATEWAY=1`, so destructive SQLite maintenance cannot race a live owner. +2. **Config lock** acquires the historical per-config lock and records the runtime port. Multi-Gateway mode skips this config singleton but retains the state ownership lock. +3. **Socket bind** binds the HTTP/WebSocket listener (default `ws://127.0.0.1:18789`) as an exclusive TCP listener. Each layer can fail independently and throws its own `GatewayLockError`. -### File lock +### State and config locks -- If the lock file is missing, the recorded owner process is gone, or the owner's port probe shows no live listener, startup reclaims the lock and continues. -- If the lock is actively held and none of the above apply, startup retries for up to 5 seconds (default) before giving up: +- Lock liveness comes from the recorded PID, platform process start identity when available, and Gateway process identity. A verified owner remains authoritative during startup before its port begins listening. +- A dedicated SQLite coordinator serializes metadata inspection, stale-owner reclamation, and lock replacement. Its exclusive transaction is released automatically if the owning process crashes. +- If a lock file is missing or the recorded owner process is gone, startup reclaims the lock and continues. +- If either lock is actively held, startup retries for up to 5 seconds (default) before giving up: ```text GatewayLockError("gateway already running (pid ); lock timeout after ms") @@ -45,11 +48,13 @@ Each layer can fail independently and throws its own `GatewayLockError`. GatewayLockError("failed to bind gateway socket on ws://127.0.0.1:: ") ``` -On shutdown, the gateway closes the HTTP/WebSocket server and removes the lock file. +On shutdown, the gateway closes the HTTP/WebSocket server and removes its state +and config lock files. ## Operational notes - If the port is occupied by a different, non-gateway process, the error is the same; free the port or choose another with `openclaw gateway --port `. +- `OPENCLAW_ALLOW_MULTI_GATEWAY=1` permits multiple config/runtime instances, not shared mutable state. Each instance still needs a unique `OPENCLAW_STATE_DIR`. - Under a service supervisor, a new gateway process that hits either error above first probes `/healthz` on the existing process. If that process is healthy, the new process leaves it in control instead of failing. On systemd, it exits with code `78`; the unit's `RestartPreventExitStatus=78` stops `Restart=always` from looping on a lock or `EADDRINUSE` conflict. If the existing process never becomes healthy, the health-probe retry is time-bounded and startup then fails with the lock error above instead of looping forever. - The macOS app keeps its own lightweight PID guard before spawning the gateway; the file lock and socket bind above are the actual runtime enforcement. diff --git a/docs/gateway/multiple-gateways.md b/docs/gateway/multiple-gateways.md index a63229692b3..274b259d685 100644 --- a/docs/gateway/multiple-gateways.md +++ b/docs/gateway/multiple-gateways.md @@ -91,7 +91,9 @@ Keep these unique per Gateway instance: | `gateway.port` (or `--port`) | Unique per instance | | Derived browser/CDP ports | See below | -Sharing any of these causes config races and port conflicts. +Sharing any of these causes config, state, or port conflicts. Gateway startup +enforces unique state-directory ownership even when +`OPENCLAW_ALLOW_MULTI_GATEWAY=1` skips the per-config singleton. ## Port mapping (derived) diff --git a/docs/install/fly.md b/docs/install/fly.md index b53c1261634..667305a0ca6 100644 --- a/docs/install/fly.md +++ b/docs/install/fly.md @@ -275,7 +275,13 @@ fly machine update --vm-memory 2048 -y Gateway refuses to start with "already running" errors after a container restart. -The single-instance lock file lives at `/openclaw-/gateway..lock` (Linux: `/tmp/openclaw-/gateway..lock`), not on the persistent `/data` volume, so a full container restart normally clears it along with the rest of the container filesystem. If the lock survives (for example a `fly machine restart` that preserves the container filesystem) and blocks startup, remove it manually: +The runtime lock files live at `/openclaw-/gateway..lock` +and `gateway.state..lock` (Linux: +`/tmp/openclaw-/gateway.*.lock`), not on the persistent `/data` volume, so +a full container restart normally clears them along with the rest of the +container filesystem. If a lock survives (for example a `fly machine restart` +that preserves the container filesystem) and blocks startup, remove it +manually: ```bash fly ssh console --command "rm -f /tmp/openclaw-*/gateway.*.lock" diff --git a/src/cli/command-catalog.ts b/src/cli/command-catalog.ts index ff145dd1047..5455cc7c011 100644 --- a/src/cli/command-catalog.ts +++ b/src/cli/command-catalog.ts @@ -52,6 +52,18 @@ export type CliCommandCatalogEntry = { }; }; +function hasCliOption(argv: readonly string[], name: string): boolean { + for (const arg of argv.slice(2)) { + if (arg === "--") { + return false; + } + if (arg === name || arg.startsWith(`${name}=`)) { + return true; + } + } + return false; +} + /** Command path registry used before Commander registration has loaded all plugins. */ export const cliCommandCatalog: readonly CliCommandCatalogEntry[] = [ { @@ -285,6 +297,9 @@ export const cliCommandCatalog: readonly CliCommandCatalogEntry[] = [ policy: { bypassConfigGuard: true, loadPlugins: "never", + // Shared-state maintenance must acquire exclusive ownership before any + // config-health observation can open the canonical SQLite database. + networkProxy: ({ argv }) => (hasCliOption(argv, "--state-sqlite") ? "bypass" : "default"), }, }, { commandPath: ["exec-approvals"], policy: { networkProxy: "bypass" } }, diff --git a/src/cli/command-path-policy.test.ts b/src/cli/command-path-policy.test.ts index d7e3aa93caf..0d35d7b38db 100644 --- a/src/cli/command-path-policy.test.ts +++ b/src/cli/command-path-policy.test.ts @@ -206,10 +206,24 @@ describe("command-path-policy", () => { loadPlugins: "never", networkProxy: "bypass", }); - expectResolvedPolicy(["doctor"], { + const doctorPolicy = resolveCliCommandPathPolicy(["doctor"]); + expectNetworkProxyResolver(doctorPolicy); + expect(doctorPolicy).toMatchObject({ bypassConfigGuard: true, loadPlugins: "never", }); + expect( + doctorPolicy.networkProxy({ + argv: ["node", "openclaw", "doctor"], + commandPath: ["doctor"], + }), + ).toBe("default"); + expect( + doctorPolicy.networkProxy({ + argv: ["node", "openclaw", "doctor", "--state-sqlite=compact"], + commandPath: ["doctor"], + }), + ).toBe("bypass"); expectResolvedPolicy(["config", "validate"], { bypassConfigGuard: true, loadPlugins: "never", diff --git a/src/cli/run-main.test.ts b/src/cli/run-main.test.ts index 99ac9a0d29c..702d7d3641d 100644 --- a/src/cli/run-main.test.ts +++ b/src/cli/run-main.test.ts @@ -229,6 +229,16 @@ describe("shouldStartProxyForCli", () => { expect(shouldStartProxyForCli(["node", "openclaw", "devices"])).toBe(false); expect(shouldStartProxyForCli(["node", "openclaw", "mcp"])).toBe(false); }); + + it("skips managed proxy routing before shared-state SQLite maintenance", () => { + expect( + shouldStartProxyForCli(["node", "openclaw", "doctor", "--state-sqlite", "compact", "--json"]), + ).toBe(false); + expect( + shouldStartProxyForCli(["node", "openclaw", "doctor", "--state-sqlite=compact", "--json"]), + ).toBe(false); + expect(shouldStartProxyForCli(["node", "openclaw", "doctor", "--lint"])).toBe(true); + }); }); describe("shouldUseRootHelpFastPath", () => { diff --git a/src/commands/doctor-session-transcripts.test.ts b/src/commands/doctor-session-transcripts.test.ts index 59d4ce92324..babedf6697a 100644 --- a/src/commands/doctor-session-transcripts.test.ts +++ b/src/commands/doctor-session-transcripts.test.ts @@ -8,6 +8,7 @@ import { SessionManager } from "../agents/sessions/session-manager.js"; const note = vi.hoisted(() => vi.fn()); const runDoctorSessionSqlite = vi.hoisted(() => vi.fn()); +const withDoctorSqliteMaintenanceLock = vi.hoisted(() => vi.fn()); vi.mock("../../packages/terminal-core/src/note.js", () => ({ note, @@ -17,6 +18,10 @@ vi.mock("./doctor-session-sqlite.js", () => ({ runDoctorSessionSqlite, })); +vi.mock("./doctor-sqlite-maintenance-lock.js", () => ({ + withDoctorSqliteMaintenanceLock, +})); + import { detectSessionTranscriptHealthIssues, noteSessionTranscriptHealth, @@ -49,6 +54,9 @@ describe("doctor session transcript repair", () => { beforeEach(async () => { note.mockClear(); runDoctorSessionSqlite.mockReset(); + withDoctorSqliteMaintenanceLock + .mockReset() + .mockImplementation(async (params: { run: () => unknown }) => await params.run()); root = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-doctor-transcripts-")); }); @@ -227,6 +235,11 @@ describe("doctor session transcript repair", () => { env, mode: "import", }); + expect(withDoctorSqliteMaintenanceLock).toHaveBeenCalledWith({ + env, + operation: "session SQLite import", + run: expect.any(Function), + }); expect(note).toHaveBeenCalledWith( expect.stringContaining("Legacy entries: 1"), "Session SQLite", @@ -237,6 +250,38 @@ describe("doctor session transcript repair", () => { ); }); + it("keeps session SQLite dry-run read-only without taking maintenance ownership", async () => { + const sessionsDir = path.join(root, "agents", "main", "sessions"); + await fs.mkdir(sessionsDir, { recursive: true }); + runDoctorSessionSqlite.mockResolvedValueOnce({ + totals: { + archivedTranscriptFiles: 0, + archivedUnreferencedJsonlFiles: 0, + importedTranscriptEvents: 0, + issues: 0, + legacyEntries: 0, + sqliteEntries: 0, + unreferencedJsonlFiles: 0, + validatedTranscriptEvents: 0, + }, + }); + const env = { ...process.env, OPENCLAW_STATE_DIR: root }; + + await noteSessionTranscriptHealth({ + env, + sessionDirs: [sessionsDir], + sessionSqlite: true, + shouldRepair: false, + }); + + expect(runDoctorSessionSqlite).toHaveBeenCalledWith({ + allAgents: true, + env, + mode: "dry-run", + }); + expect(withDoctorSqliteMaintenanceLock).not.toHaveBeenCalled(); + }); + it("repairs supported current-version linear transcripts", async () => { const filePath = await writeTranscript([ { type: "session", version: 3, id: "session-linear", timestamp: "2026-06-15T00:00:00Z" }, diff --git a/src/commands/doctor-session-transcripts.ts b/src/commands/doctor-session-transcripts.ts index 440a2b75b13..13dd9634839 100644 --- a/src/commands/doctor-session-transcripts.ts +++ b/src/commands/doctor-session-transcripts.ts @@ -18,6 +18,7 @@ import { } from "../config/sessions/transcript-tree.js"; import type { HealthFinding, HealthRepairEffect } from "../flows/health-checks.js"; import { shortenHomePath } from "../utils.js"; +import { withDoctorSqliteMaintenanceLock } from "./doctor-sqlite-maintenance-lock.js"; const SESSION_TRANSCRIPTS_CHECK_ID = "core/doctor/session-transcripts"; @@ -496,11 +497,19 @@ async function noteSessionSqliteMigrationHealth(params: { // Public doctor owns the operator-facing SQLite import; the targeted // --session-sqlite subcommand remains the diagnostic/proof surface. const { runDoctorSessionSqlite } = await import("./doctor-session-sqlite.js"); - const report = await runDoctorSessionSqlite({ - allAgents: true, - env: params.env, - mode: params.shouldRepair ? "import" : "dry-run", - }); + const runSessionSqlite = async () => + await runDoctorSessionSqlite({ + allAgents: true, + env: params.env, + mode: params.shouldRepair ? "import" : "dry-run", + }); + const report = params.shouldRepair + ? await withDoctorSqliteMaintenanceLock({ + env: params.env, + operation: "session SQLite import", + run: runSessionSqlite, + }) + : await runSessionSqlite(); if ( report.totals.legacyEntries === 0 && report.totals.unreferencedJsonlFiles === 0 && diff --git a/src/commands/doctor-sqlite-maintenance-lock.test.ts b/src/commands/doctor-sqlite-maintenance-lock.test.ts new file mode 100644 index 00000000000..a64cdbb5fd6 --- /dev/null +++ b/src/commands/doctor-sqlite-maintenance-lock.test.ts @@ -0,0 +1,360 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { useAutoCleanupTempDirTracker } from "../../test/helpers/temp-dir.js"; +import { acquireGatewayLock, GatewayLockError } from "../infra/gateway-lock.js"; +import { + isDestructiveDoctorSessionSqliteMode, + withDoctorSqliteMaintenanceLock, +} from "./doctor-sqlite-maintenance-lock.js"; + +const tempDirs = useAutoCleanupTempDirTracker((cleanup) => { + afterEach(cleanup); +}); + +async function createLockFixture() { + const root = tempDirs.make("openclaw-doctor-sqlite-lock-"); + const stateDir = path.join(root, "state"); + const configPath = path.join(stateDir, "openclaw.json"); + const lockDir = path.join(root, "locks"); + await fs.mkdir(stateDir, { recursive: true }); + await fs.writeFile(configPath, "{}\n", "utf8"); + const env = { + ...process.env, + OPENCLAW_CONFIG_PATH: configPath, + OPENCLAW_STATE_DIR: stateDir, + VITEST: "1", + }; + return { + env, + lockDir, + lockOptions: { + lockDir, + platform: "darwin" as const, + pollIntervalMs: 2, + readProcessCmdline: () => ["openclaw-gateway"], + timeoutMs: 15, + }, + }; +} + +afterEach(() => { + vi.restoreAllMocks(); +}); + +describe("doctor SQLite maintenance lock", () => { + it.each([ + ["inspect", false], + ["dry-run", false], + ["validate", false], + ["import", true], + ["compact", true], + ["restore", true], + ["recover", true], + ] as const)("classifies %s mode as destructive=%s", (mode, expected) => { + expect(isDestructiveDoctorSessionSqliteMode(mode)).toBe(expected); + }); + + it("refuses maintenance while a verified Gateway owns the state lock", async () => { + const fixture = await createLockFixture(); + const gatewayLock = await acquireGatewayLock({ + allowInTests: true, + env: fixture.env, + lockDir: fixture.lockDir, + platform: "darwin", + port: 18789, + }); + if (!gatewayLock) { + throw new Error("expected Gateway lock"); + } + const run = vi.fn(); + + try { + await expect( + withDoctorSqliteMaintenanceLock( + { + env: fixture.env, + operation: "state SQLite compaction", + run, + }, + { lockOptions: fixture.lockOptions }, + ), + ).rejects.toThrow(/Gateway or another SQLite maintenance command owns/); + expect(run).not.toHaveBeenCalled(); + } finally { + await gatewayLock.release(); + } + }); + + it("prevents Gateway startup until maintenance releases ownership", async () => { + const fixture = await createLockFixture(); + let allowMaintenanceToFinish: (() => void) | undefined; + const maintenanceMayFinish = new Promise((resolve) => { + allowMaintenanceToFinish = resolve; + }); + let markMaintenanceStarted: (() => void) | undefined; + const maintenanceStarted = new Promise((resolve) => { + markMaintenanceStarted = resolve; + }); + const maintenance = withDoctorSqliteMaintenanceLock( + { + env: fixture.env, + operation: "session SQLite compaction", + run: async () => { + markMaintenanceStarted?.(); + await maintenanceMayFinish; + return "done"; + }, + }, + { lockOptions: fixture.lockOptions }, + ); + await maintenanceStarted; + + await expect( + acquireGatewayLock({ + allowInTests: true, + env: fixture.env, + lockDir: fixture.lockDir, + platform: "darwin", + port: 18789, + pollIntervalMs: 2, + readProcessCmdline: () => ["openclaw", "doctor", "--session-sqlite", "compact"], + timeoutMs: 15, + }), + ).rejects.toBeInstanceOf(GatewayLockError); + + allowMaintenanceToFinish?.(); + await expect(maintenance).resolves.toBe("done"); + + const gatewayLock = await acquireGatewayLock({ + allowInTests: true, + env: fixture.env, + lockDir: fixture.lockDir, + platform: "darwin", + port: 18789, + timeoutMs: 15, + }); + if (!gatewayLock) { + throw new Error("expected Gateway lock after maintenance release"); + } + await gatewayLock.release(); + }); + + it("releases ownership after maintenance fails", async () => { + const fixture = await createLockFixture(); + + await expect( + withDoctorSqliteMaintenanceLock( + { + env: fixture.env, + operation: "session SQLite restore", + run: () => { + throw new Error("restore failed"); + }, + }, + { lockOptions: fixture.lockOptions }, + ), + ).rejects.toThrow("restore failed"); + + const gatewayLock = await acquireGatewayLock({ + allowInTests: true, + env: fixture.env, + lockDir: fixture.lockDir, + platform: "darwin", + port: 18789, + timeoutMs: 15, + }); + if (!gatewayLock) { + throw new Error("expected Gateway lock after failed maintenance"); + } + await gatewayLock.release(); + }); + + it("blocks maintenance when the Gateway used the multi-Gateway override", async () => { + const fixture = await createLockFixture(); + const run = vi.fn(); + const gatewayLock = await acquireGatewayLock({ + allowInTests: true, + env: { ...fixture.env, OPENCLAW_ALLOW_MULTI_GATEWAY: "1" }, + lockDir: fixture.lockDir, + platform: "darwin", + port: 18789, + timeoutMs: 15, + }); + if (!gatewayLock) { + throw new Error("expected state ownership lock"); + } + + try { + await expect( + withDoctorSqliteMaintenanceLock( + { + env: fixture.env, + operation: "state SQLite compaction", + run, + }, + { lockOptions: fixture.lockOptions }, + ), + ).rejects.toThrow(/Gateway or another SQLite maintenance command owns/); + expect(run).not.toHaveBeenCalled(); + } finally { + await gatewayLock.release(); + } + }); + + it("uses the state ownership lock when maintenance inherits the multi-Gateway override", async () => { + const fixture = await createLockFixture(); + + await expect( + withDoctorSqliteMaintenanceLock( + { + env: { ...fixture.env, OPENCLAW_ALLOW_MULTI_GATEWAY: "1" }, + operation: "state SQLite compaction", + run: () => "done", + }, + { lockOptions: fixture.lockOptions }, + ), + ).resolves.toBe("done"); + }); + + it("refuses explicit destructive targets outside the locked state directory", async () => { + const fixture = await createLockFixture(); + const externalPath = path.join( + tempDirs.make("openclaw-external-session-store-"), + "sessions.json", + ); + const run = vi.fn(); + + await expect( + withDoctorSqliteMaintenanceLock( + { + env: fixture.env, + operation: "session SQLite compaction", + protectedPaths: [externalPath], + run, + }, + { lockOptions: fixture.lockOptions }, + ), + ).rejects.toThrow(/outside the active OpenClaw state directory/); + expect(run).not.toHaveBeenCalled(); + + const gatewayLock = await acquireGatewayLock({ + allowInTests: true, + env: fixture.env, + lockDir: fixture.lockDir, + platform: "darwin", + port: 18789, + timeoutMs: 15, + }); + if (!gatewayLock) { + throw new Error("expected Gateway lock after ownership validation failure"); + } + await gatewayLock.release(); + }); + + it("refuses dangling in-state symlinks that resolve outside ownership", async () => { + if (process.platform === "win32") { + return; + } + const fixture = await createLockFixture(); + const sessionsDir = path.join(fixture.env.OPENCLAW_STATE_DIR, "agents", "main", "sessions"); + const storePath = path.join(sessionsDir, "sessions.json"); + const outsideTarget = path.join( + tempDirs.make("openclaw-dangling-session-target-"), + "missing.json", + ); + await fs.mkdir(sessionsDir, { recursive: true }); + await fs.symlink(outsideTarget, storePath); + const run = vi.fn(); + + await expect( + withDoctorSqliteMaintenanceLock( + { + env: fixture.env, + operation: "session SQLite import", + protectedPaths: [storePath], + run, + }, + { lockOptions: fixture.lockOptions }, + ), + ).rejects.toThrow(/outside the active OpenClaw state directory/); + expect(run).not.toHaveBeenCalled(); + }); + + it("refuses outside path aliases that resolve into the locked state directory", async () => { + if (process.platform === "win32") { + return; + } + const fixture = await createLockFixture(); + const externalAlias = path.join( + path.dirname(fixture.env.OPENCLAW_STATE_DIR), + "external-state-alias", + ); + await fs.symlink(fixture.env.OPENCLAW_STATE_DIR, externalAlias, "dir"); + const storePath = path.join(externalAlias, "agents", "main", "sessions", "sessions.json"); + const run = vi.fn(); + + await expect( + withDoctorSqliteMaintenanceLock( + { + env: fixture.env, + operation: "session SQLite import", + protectedPaths: [storePath], + run, + }, + { lockOptions: fixture.lockOptions }, + ), + ).rejects.toThrow(/outside the active OpenClaw state directory/); + expect(run).not.toHaveBeenCalled(); + }); + + it("refuses in-state hard links that can alias storage outside ownership", async () => { + const fixture = await createLockFixture(); + const sessionsDir = path.join(fixture.env.OPENCLAW_STATE_DIR, "agents", "main", "sessions"); + const storePath = path.join(sessionsDir, "sessions.json"); + const externalDir = path.join(path.dirname(fixture.env.OPENCLAW_STATE_DIR), "external-state"); + const externalPath = path.join(externalDir, "sessions.json"); + await fs.mkdir(sessionsDir, { recursive: true }); + await fs.mkdir(externalDir, { recursive: true }); + await fs.writeFile(externalPath, "{}\n", "utf8"); + await fs.link(externalPath, storePath); + const run = vi.fn(); + + await expect( + withDoctorSqliteMaintenanceLock( + { + env: fixture.env, + operation: "session SQLite compaction", + protectedPaths: [storePath], + run, + }, + { lockOptions: fixture.lockOptions }, + ), + ).rejects.toThrow(/hard-linked path/); + expect(run).not.toHaveBeenCalled(); + await expect(fs.readFile(externalPath, "utf8")).resolves.toBe("{}\n"); + }); + + it("allows explicit destructive targets owned by the locked state directory", async () => { + const fixture = await createLockFixture(); + const storePath = path.join( + fixture.env.OPENCLAW_STATE_DIR, + "agents", + "main", + "sessions", + "sessions.json", + ); + + await expect( + withDoctorSqliteMaintenanceLock( + { + env: fixture.env, + operation: "session SQLite compaction", + protectedPaths: [storePath], + run: () => "done", + }, + { lockOptions: fixture.lockOptions }, + ), + ).resolves.toBe("done"); + }); +}); diff --git a/src/commands/doctor-sqlite-maintenance-lock.ts b/src/commands/doctor-sqlite-maintenance-lock.ts new file mode 100644 index 00000000000..db87715016f --- /dev/null +++ b/src/commands/doctor-sqlite-maintenance-lock.ts @@ -0,0 +1,122 @@ +/** Serializes offline SQLite maintenance against the Gateway state owner. */ +import fs from "node:fs"; +import path from "node:path"; +import { resolveStateDir } from "../config/paths.js"; +import { resolvePathViaExistingAncestorSync, resolveRootPathSync } from "../infra/boundary-path.js"; +import { + acquireGatewayLock, + GatewayLockError, + type GatewayLockOptions, +} from "../infra/gateway-lock.js"; +import { isPathInside } from "../infra/path-guards.js"; +import type { DoctorSessionSqliteMode } from "./doctor-session-sqlite-types.js"; + +const MAINTENANCE_LOCK_TIMEOUT_MS = 250; +const MAINTENANCE_LOCK_POLL_INTERVAL_MS = 25; + +type MaintenanceLockOptions = Pick< + GatewayLockOptions, + | "lockDir" + | "now" + | "platform" + | "pollIntervalMs" + | "readProcessCmdline" + | "readProcessStartTime" + | "sleep" + | "staleMs" + | "timeoutMs" +>; + +type DoctorSqliteMaintenanceLockDeps = { + acquireLock?: typeof acquireGatewayLock; + lockOptions?: MaintenanceLockOptions; +}; + +function assertMaintenancePathsOwnedByStateDir( + env: NodeJS.ProcessEnv, + operation: string, + protectedPaths: readonly string[], +): void { + if (protectedPaths.length === 0) { + return; + } + const stateDir = path.resolve(resolveStateDir(env)); + const stateCanonicalDir = resolvePathViaExistingAncestorSync(stateDir); + for (const protectedPath of protectedPaths) { + const absolutePath = path.resolve(protectedPath); + let resolvedPath: ReturnType; + try { + if (!isPathInside(stateDir, absolutePath) && !isPathInside(stateCanonicalDir, absolutePath)) { + throw new Error("path is not lexically owned by the active state directory"); + } + resolvedPath = resolveRootPathSync({ + absolutePath, + boundaryLabel: "OpenClaw state directory", + rootCanonicalPath: stateCanonicalDir, + rootPath: stateDir, + }); + } catch (error) { + throw new Error( + `Cannot run ${operation} for a path outside the active OpenClaw state directory: ${protectedPath}. Set OPENCLAW_STATE_DIR to the owning state directory and retry.`, + { cause: error }, + ); + } + if ( + resolvedPath.exists && + resolvedPath.kind === "file" && + fs.statSync(resolvedPath.canonicalPath).nlink > 1 + ) { + throw new Error( + `Cannot run ${operation} for a hard-linked path: ${protectedPath}. Remove the additional hard link and retry.`, + ); + } + } +} + +export function isDestructiveDoctorSessionSqliteMode(mode: DoctorSessionSqliteMode): boolean { + return mode === "import" || mode === "compact" || mode === "restore" || mode === "recover"; +} + +/** Run one destructive doctor operation while excluding Gateway startup and peer maintenance. */ +export async function withDoctorSqliteMaintenanceLock( + params: { + env?: NodeJS.ProcessEnv; + operation: string; + protectedPaths?: readonly string[]; + run: () => Promise | T; + }, + deps: DoctorSqliteMaintenanceLockDeps = {}, +): Promise { + const env = params.env ?? process.env; + const acquireLock = deps.acquireLock ?? acquireGatewayLock; + const lockOptions = deps.lockOptions; + let lock: Awaited>; + try { + lock = await acquireLock({ + ...lockOptions, + allowInTests: true, + env, + pollIntervalMs: lockOptions?.pollIntervalMs ?? MAINTENANCE_LOCK_POLL_INTERVAL_MS, + role: "sqlite-maintenance", + timeoutMs: lockOptions?.timeoutMs ?? MAINTENANCE_LOCK_TIMEOUT_MS, + }); + } catch (error) { + if (error instanceof GatewayLockError) { + throw new Error( + `Cannot run ${params.operation} while the Gateway or another SQLite maintenance command owns this OpenClaw state directory. Stop the Gateway and retry.`, + { cause: error }, + ); + } + throw error; + } + if (!lock) { + throw new Error(`Cannot run ${params.operation} without exclusive OpenClaw state ownership.`); + } + + try { + assertMaintenancePathsOwnedByStateDir(env, params.operation, params.protectedPaths ?? []); + return await params.run(); + } finally { + await lock.release(); + } +} diff --git a/src/commands/doctor-state-sqlite-compact.test.ts b/src/commands/doctor-state-sqlite-compact.test.ts index 1560e92b4cc..74c596b083a 100644 --- a/src/commands/doctor-state-sqlite-compact.test.ts +++ b/src/commands/doctor-state-sqlite-compact.test.ts @@ -138,10 +138,10 @@ function expectOwnerOnlySqlitePermissions(sqlitePath: string): void { } describe("runDoctorStateSqliteCompact", () => { - it("reports a missing canonical database as skipped", () => { + it("reports a missing canonical database as skipped", async () => { const env = createStateEnv(); - expect(runDoctorStateSqliteCompact({ env })).toEqual({ + await expect(runDoctorStateSqliteCompact({ env })).resolves.toEqual({ mode: "compact", path: resolveOpenClawStateSqlitePath(env), reason: "missing", @@ -149,11 +149,11 @@ describe("runDoctorStateSqliteCompact", () => { }); }); - it("compacts the canonical database and reports verified before/after state", () => { + it("compacts the canonical database and reports verified before/after state", async () => { const env = createStateEnv(); const sqlitePath = seedStateDatabase({ env, withBloat: true }); - const report = runDoctorStateSqliteCompact({ env }); + const report = await runDoctorStateSqliteCompact({ env }); expectCompletedReport(report); expect(report.path).toBe(sqlitePath); @@ -169,20 +169,20 @@ describe("runDoctorStateSqliteCompact", () => { expect(report.integrityCheck).toBe("ok"); }); - it.skipIf(process.platform === "win32")("reapplies owner-only SQLite permissions", () => { + it.skipIf(process.platform === "win32")("reapplies owner-only SQLite permissions", async () => { const env = createStateEnv(); const sqlitePath = seedStateDatabase({ env, withBloat: true }); - runDoctorStateSqliteCompact({ env }); + await runDoctorStateSqliteCompact({ env }); expectOwnerOnlySqlitePermissions(sqlitePath); }); - it("rejects non-global schema metadata before mutation", () => { + it("rejects non-global schema metadata before mutation", async () => { const env = createStateEnv(); const sqlitePath = seedStateDatabase({ env, role: "agent", withBloat: true }); - expect(() => runDoctorStateSqliteCompact({ env })).toThrow(/schema role agent.*global/); + await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(/schema role agent.*global/); const sqlite = requireNodeSqlite(); const database = new sqlite.DatabaseSync(sqlitePath, { readOnly: true }); @@ -199,11 +199,11 @@ describe("runDoctorStateSqliteCompact", () => { ["future", OPENCLAW_STATE_SCHEMA_VERSION + 1, /uses newer schema version/], ] as const)( "rejects a %s shared-state schema before mutation", - (_label, schemaVersion, message) => { + async (_label, schemaVersion, message) => { const env = createStateEnv(); const sqlitePath = seedStateDatabase({ env, schemaVersion }); - expect(() => runDoctorStateSqliteCompact({ env })).toThrow(message); + await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(message); const sqlite = requireNodeSqlite(); const database = new sqlite.DatabaseSync(sqlitePath, { readOnly: true }); @@ -217,7 +217,7 @@ describe("runDoctorStateSqliteCompact", () => { it.skipIf(process.platform === "win32")( "refuses a symlink at the canonical database path", - () => { + async () => { const env = createStateEnv(); const canonicalPath = resolveOpenClawStateSqlitePath(env); const externalEnv = createStateEnv(); @@ -225,18 +225,45 @@ describe("runDoctorStateSqliteCompact", () => { fs.mkdirSync(path.dirname(canonicalPath), { recursive: true }); fs.symlinkSync(externalPath, canonicalPath); - expect(() => runDoctorStateSqliteCompact({ env })).toThrow(/not a regular file/); + await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(/not a regular file/); }, ); - it("refuses compaction while this process owns an open shared-state handle", () => { + it("refuses compaction while this process owns an open shared-state handle", async () => { const env = createStateEnv(); openOpenClawStateDatabase({ env }); - expect(() => runDoctorStateSqliteCompact({ env })).toThrow(/already open in this process/); + await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow( + /already open in this process/, + ); }); - it("treats a busy truncating checkpoint as failure", () => { + it("leaves the database untouched when exclusive state ownership is unavailable", async () => { + const env = createStateEnv(); + const sqlitePath = seedStateDatabase({ env, withBloat: true }); + + await expect( + runDoctorStateSqliteCompact( + { env }, + { + withMaintenanceLock: async () => { + throw new Error("Gateway owns this OpenClaw state directory"); + }, + }, + ), + ).rejects.toThrow(/Gateway owns/); + + const sqlite = requireNodeSqlite(); + const database = new sqlite.DatabaseSync(sqlitePath, { readOnly: true }); + try { + expect(readPragma(database, "auto_vacuum")).toBe(0); + expect(readPragma(database, "freelist_count")).toBeGreaterThan(0); + } finally { + database.close(); + } + }); + + it("treats a busy truncating checkpoint as failure", async () => { const env = createStateEnv(); const sqlitePath = seedStateDatabase({ env }); const sqlite = requireNodeSqlite(); @@ -247,7 +274,7 @@ describe("runDoctorStateSqliteCompact", () => { writer.exec("INSERT INTO compact_payload (payload) VALUES ('newer wal frame');"); // Exercise the real busy checkpoint result without waiting the production lock timeout. - expect(() => runDoctorStateSqliteCompact({ env }, { busyTimeoutMs: 0 })).toThrow( + await expect(runDoctorStateSqliteCompact({ env }, { busyTimeoutMs: 0 })).rejects.toThrow( /checkpoint remained busy/, ); expect(readPragma(writer, "auto_vacuum")).toBe(0); @@ -258,7 +285,7 @@ describe("runDoctorStateSqliteCompact", () => { } }); - it("rejects stale secondary indexes before mutating the database", () => { + it("rejects stale secondary indexes before mutating the database", async () => { const env = createStateEnv(); const sqlitePath = seedStateDatabase({ env, withBloat: true }); createUnsafeIndexDrift(sqlitePath); @@ -277,7 +304,7 @@ describe("runDoctorStateSqliteCompact", () => { before.close(); } - expect(() => runDoctorStateSqliteCompact({ env })).toThrow( + await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow( /integrity_check failed.*missing from index unsafe_index_records_value/iu, ); @@ -297,7 +324,7 @@ describe("runDoctorStateSqliteCompact", () => { } }); - it("rejects foreign-key violations before mutating the database", () => { + it("rejects foreign-key violations before mutating the database", async () => { const env = createStateEnv(); const sqlitePath = seedStateDatabase({ env, withBloat: true }); const sqlite = requireNodeSqlite(); @@ -326,7 +353,7 @@ describe("runDoctorStateSqliteCompact", () => { corrupted.close(); } - expect(() => runDoctorStateSqliteCompact({ env })).toThrow( + await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow( /foreign_key_check failed.*compact_children row 1 references compact_parents \(foreign key 0\)/iu, ); diff --git a/src/commands/doctor-state-sqlite-compact.ts b/src/commands/doctor-state-sqlite-compact.ts index b4cc86ed4f1..555ca4ba8c2 100644 --- a/src/commands/doctor-state-sqlite-compact.ts +++ b/src/commands/doctor-state-sqlite-compact.ts @@ -10,6 +10,7 @@ import { compactDoctorSqliteFile, type DoctorSqliteCompactSnapshot, } from "./doctor-sqlite-compact.js"; +import { withDoctorSqliteMaintenanceLock } from "./doctor-sqlite-maintenance-lock.js"; export type DoctorStateSqliteCompactReport = | { @@ -35,13 +36,14 @@ type DoctorStateSqliteCompactOptions = { type DoctorStateSqliteCompactDeps = { busyTimeoutMs?: number; + withMaintenanceLock?: typeof withDoctorSqliteMaintenanceLock; }; /** Compact only the canonical shared state database resolved for this invocation. */ -export function runDoctorStateSqliteCompact( +export async function runDoctorStateSqliteCompact( options: DoctorStateSqliteCompactOptions = {}, deps: DoctorStateSqliteCompactDeps = {}, -): DoctorStateSqliteCompactReport { +): Promise { const env = options.env ?? process.env; const sqlitePath = resolveOpenClawStateSqlitePath(env); const stat = readCanonicalStateDatabaseStat(sqlitePath); @@ -56,25 +58,32 @@ export function runDoctorStateSqliteCompact( if (!stat.isFile()) { throw new Error(`Canonical OpenClaw state database is not a regular file: ${sqlitePath}`); } - if (isOpenClawStateDatabaseOpen()) { - throw new Error( - "The shared OpenClaw state database is already open in this process. Stop OpenClaw and retry.", - ); - } + const withMaintenanceLock = deps.withMaintenanceLock ?? withDoctorSqliteMaintenanceLock; + return await withMaintenanceLock({ + env, + operation: "state SQLite compaction", + run: () => { + if (isOpenClawStateDatabaseOpen()) { + throw new Error( + "The shared OpenClaw state database is already open in this process. Stop OpenClaw and retry.", + ); + } - const compact = compactDoctorSqliteFile({ - afterMutation: () => ensureOpenClawStatePermissions(sqlitePath, env), - ...(deps.busyTimeoutMs !== undefined ? { busyTimeoutMs: deps.busyTimeoutMs } : {}), - sqlitePath, - validateBeforeMutation: (database) => - assertOpenClawStateDatabaseForMaintenance(database, { pathname: sqlitePath }), + const compact = compactDoctorSqliteFile({ + afterMutation: () => ensureOpenClawStatePermissions(sqlitePath, env), + ...(deps.busyTimeoutMs !== undefined ? { busyTimeoutMs: deps.busyTimeoutMs } : {}), + sqlitePath, + validateBeforeMutation: (database) => + assertOpenClawStateDatabaseForMaintenance(database, { pathname: sqlitePath }), + }); + return { + ...compact, + mode: "compact", + path: sqlitePath, + skipped: false, + }; + }, }); - return { - ...compact, - mode: "compact", - path: sqlitePath, - skipped: false, - }; } function readCanonicalStateDatabaseStat(sqlitePath: string): fs.Stats | undefined { diff --git a/src/commands/doctor.test.ts b/src/commands/doctor.test.ts index bd1ba70dcc5..bfb02067004 100644 --- a/src/commands/doctor.test.ts +++ b/src/commands/doctor.test.ts @@ -1,4 +1,5 @@ // Doctor command tests cover probe orchestration, fix mode, and runtime command output. +import path from "node:path"; import { beforeEach, describe, expect, it, vi } from "vitest"; const mocks = vi.hoisted(() => ({ @@ -6,6 +7,7 @@ const mocks = vi.hoisted(() => ({ runPostUpgradeProbes: vi.fn(), runDoctorStateSqliteCompact: vi.fn(), runDoctorSessionSqlite: vi.fn(), + withDoctorSqliteMaintenanceLock: vi.fn(), resolveInstalledPluginIndexStorePath: vi.fn(() => "/tmp/openclaw-installed-plugins.json"), })); @@ -21,6 +23,12 @@ vi.mock("./doctor-state-sqlite-compact.js", () => ({ runDoctorStateSqliteCompact: mocks.runDoctorStateSqliteCompact, })); +vi.mock("./doctor-sqlite-maintenance-lock.js", () => ({ + isDestructiveDoctorSessionSqliteMode: (mode: string) => + mode === "import" || mode === "compact" || mode === "restore" || mode === "recover", + withDoctorSqliteMaintenanceLock: mocks.withDoctorSqliteMaintenanceLock, +})); + vi.mock("./doctor-session-sqlite-github-issue.js", () => ({ createSessionSqliteGithubIssue: mocks.createSessionSqliteGithubIssue, })); @@ -34,6 +42,9 @@ const { doctorCommand } = await import("./doctor.js"); describe("doctorCommand", () => { beforeEach(() => { vi.clearAllMocks(); + mocks.withDoctorSqliteMaintenanceLock.mockImplementation( + async (params: { run: () => unknown }) => await params.run(), + ); }); it("writes post-upgrade JSON through the runtime before exiting with findings", async () => { @@ -108,10 +119,124 @@ describe("doctorCommand", () => { agent: "main", mode: "inspect", }); + expect(mocks.withDoctorSqliteMaintenanceLock).not.toHaveBeenCalled(); expect(runtime.writeJson).toHaveBeenCalledWith(report, 2); expect(runtime.exit).toHaveBeenCalledWith(0); }); + it("holds exclusive state ownership for destructive session sqlite modes", async () => { + const report = { + mode: "restore", + targets: [], + totals: { + archivedTranscriptFiles: 0, + archivedUnreferencedJsonlFiles: 0, + importedEntries: 0, + importedTranscriptEvents: 0, + issues: 0, + legacyEntries: 0, + sqliteEntries: 0, + targets: 0, + unreferencedJsonlFiles: 0, + validatedEntries: 0, + validatedTranscriptEvents: 0, + }, + }; + mocks.runDoctorSessionSqlite.mockResolvedValueOnce(report); + const runtime = { + log: vi.fn(), + error: vi.fn(), + writeStdout: vi.fn(), + writeJson: vi.fn(), + exit: vi.fn((code: number) => { + throw new Error(`exit:${code}`); + }), + }; + + await expect( + doctorCommand(runtime, { + sessionSqlite: "restore", + sessionSqliteAllAgents: true, + }), + ).rejects.toThrow("exit:0"); + + expect(mocks.withDoctorSqliteMaintenanceLock).toHaveBeenCalledWith({ + env: process.env, + operation: "session SQLite restore", + run: expect.any(Function), + }); + expect(mocks.runDoctorSessionSqlite).toHaveBeenCalledWith({ + allAgents: true, + mode: "restore", + }); + }); + + it("binds explicit destructive session stores to the maintenance lock", async () => { + const report = { + mode: "compact", + targets: [], + totals: { + archivedTranscriptFiles: 0, + archivedUnreferencedJsonlFiles: 0, + importedEntries: 0, + importedTranscriptEvents: 0, + issues: 0, + legacyEntries: 0, + sqliteEntries: 0, + targets: 0, + unreferencedJsonlFiles: 0, + validatedEntries: 0, + validatedTranscriptEvents: 0, + }, + }; + mocks.runDoctorSessionSqlite.mockResolvedValueOnce(report); + const runtime = { + log: vi.fn(), + error: vi.fn(), + writeStdout: vi.fn(), + writeJson: vi.fn(), + exit: vi.fn((code: number) => { + throw new Error(`exit:${code}`); + }), + }; + const stateDir = path.resolve(process.env.OPENCLAW_STATE_DIR ?? ".openclaw"); + const storePath = path.join(stateDir, "agents", "main", "sessions", "sessions.json"); + const sqlitePath = path.join(stateDir, "agents", "main", "agent", "openclaw-agent.sqlite"); + + await expect( + doctorCommand(runtime, { + sessionSqlite: "compact", + sessionSqliteStore: storePath, + }), + ).rejects.toThrow("exit:0"); + + expect(mocks.withDoctorSqliteMaintenanceLock).toHaveBeenCalledWith({ + env: process.env, + operation: "session SQLite compact", + protectedPaths: [ + storePath, + sqlitePath, + `${sqlitePath}-wal`, + `${sqlitePath}-shm`, + `${sqlitePath}-journal`, + ], + run: expect.any(Function), + }); + }); + + it("rejects conflicting explicit-store selectors before taking maintenance ownership", async () => { + await expect( + doctorCommand(undefined, { + sessionSqlite: "compact", + sessionSqliteAgent: "ops", + sessionSqliteStore: path.resolve("stores", "{agentId}", "sessions.json"), + }), + ).rejects.toThrow("--store cannot be combined with --agent or --all-agents"); + + expect(mocks.withDoctorSqliteMaintenanceLock).not.toHaveBeenCalled(); + expect(mocks.runDoctorSessionSqlite).not.toHaveBeenCalled(); + }); + it("writes shared-state sqlite compaction JSON through the runtime", async () => { const report = { after: { @@ -135,7 +260,7 @@ describe("doctorCommand", () => { reclaimedBytes: 12_288, skipped: false, }; - mocks.runDoctorStateSqliteCompact.mockReturnValueOnce(report); + mocks.runDoctorStateSqliteCompact.mockResolvedValueOnce(report); const runtime = { log: vi.fn(), error: vi.fn(), diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index 42144e1ff2a..4b3be1cc88b 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -1,15 +1,52 @@ /** Top-level doctor command wrapper, including post-upgrade probe mode. */ +import { resolveSqliteTargetFromSessionStorePath } from "../config/sessions/session-sqlite-target.js"; +import { resolveSessionStoreTargets } from "../config/sessions/targets.js"; +import { resolveSqliteDatabaseFilePaths } from "../infra/sqlite-files.js"; import { defaultRuntime, type RuntimeEnv, writeRuntimeJson } from "../runtime.js"; import { runPostUpgradeProbes } from "./doctor-post-upgrade.js"; import type { DoctorOptions } from "./doctor-prompter.js"; import type { DoctorSessionSqliteReport } from "./doctor-session-sqlite.js"; +import { + isDestructiveDoctorSessionSqliteMode, + withDoctorSqliteMaintenanceLock, +} from "./doctor-sqlite-maintenance-lock.js"; + +function resolveExplicitSessionSqliteMaintenancePaths(options: DoctorOptions): string[] { + if (!options.sessionSqliteStore) { + return []; + } + // Explicit path mode intentionally bypasses runtime config. Resolve through + // the same selector as the migration so ownership checks cover exact targets. + const targets = resolveSessionStoreTargets( + {}, + { + store: options.sessionSqliteStore, + ...(options.sessionSqliteAgent ? { agent: options.sessionSqliteAgent } : {}), + ...(options.sessionSqliteAllAgents ? { allAgents: true } : {}), + }, + { env: process.env }, + ); + const protectedPaths = new Set(); + for (const target of targets) { + protectedPaths.add(target.storePath); + const sqlitePath = resolveSqliteTargetFromSessionStorePath(target.storePath, { + agentId: target.agentId, + }).path; + if (sqlitePath) { + for (const databasePath of resolveSqliteDatabaseFilePaths(sqlitePath)) { + protectedPaths.add(databasePath); + } + } + } + return [...protectedPaths]; +} /** Runs doctor or the post-upgrade probe submode using the provided runtime. */ export async function doctorCommand(runtime?: RuntimeEnv, options?: DoctorOptions): Promise { if (options?.stateSqlite) { const outputRuntime = runtime ?? defaultRuntime; const { runDoctorStateSqliteCompact } = await import("./doctor-state-sqlite-compact.js"); - const report = runDoctorStateSqliteCompact(); + const report = await runDoctorStateSqliteCompact(); if (options.json) { writeRuntimeJson(outputRuntime, report); } else if (report.skipped) { @@ -30,14 +67,26 @@ export async function doctorCommand(runtime?: RuntimeEnv, options?: DoctorOption } if (options?.sessionSqlite) { const outputRuntime = runtime ?? defaultRuntime; + const sessionSqliteMode = options.sessionSqlite; const { runDoctorSessionSqlite } = await import("./doctor-session-sqlite.js"); - const report = await runDoctorSessionSqlite({ - mode: options.sessionSqlite, - ...(options.sessionSqliteStore ? { store: options.sessionSqliteStore } : {}), - ...(options.sessionSqliteAgent ? { agent: options.sessionSqliteAgent } : {}), - ...(options.sessionSqliteAllAgents ? { allAgents: true } : {}), - }); - if (options.sessionSqlite === "recover" && options.sessionSqliteGithubIssue === true) { + const runSessionSqlite = async () => + await runDoctorSessionSqlite({ + mode: sessionSqliteMode, + ...(options.sessionSqliteStore ? { store: options.sessionSqliteStore } : {}), + ...(options.sessionSqliteAgent ? { agent: options.sessionSqliteAgent } : {}), + ...(options.sessionSqliteAllAgents ? { allAgents: true } : {}), + }); + const report = isDestructiveDoctorSessionSqliteMode(sessionSqliteMode) + ? await withDoctorSqliteMaintenanceLock({ + env: process.env, + operation: `session SQLite ${sessionSqliteMode}`, + ...(options.sessionSqliteStore + ? { protectedPaths: resolveExplicitSessionSqliteMaintenancePaths(options) } + : {}), + run: runSessionSqlite, + }) + : await runSessionSqlite(); + if (sessionSqliteMode === "recover" && options.sessionSqliteGithubIssue === true) { await maybeCreateSessionSqliteGithubIssue(outputRuntime, report, options); } if (options.json) { diff --git a/src/flows/doctor-health-contributions.ts b/src/flows/doctor-health-contributions.ts index a3f74b362e1..5e18a588d7a 100644 --- a/src/flows/doctor-health-contributions.ts +++ b/src/flows/doctor-health-contributions.ts @@ -673,7 +673,10 @@ async function runSessionLocksHealth(ctx: DoctorHealthFlowContext): Promise { const { noteSessionTranscriptHealth } = await import("../commands/doctor-session-transcripts.js"); - await noteSessionTranscriptHealth({ shouldRepair: ctx.prompter.shouldRepair }); + await noteSessionTranscriptHealth({ + env: ctx.env ?? process.env, + shouldRepair: ctx.prompter.shouldRepair, + }); } async function runSessionSnapshotsHealth(ctx: DoctorHealthFlowContext): Promise { diff --git a/src/infra/gateway-lock.test.ts b/src/infra/gateway-lock.test.ts index 6a9a51b76e9..11f0e48728d 100644 --- a/src/infra/gateway-lock.test.ts +++ b/src/infra/gateway-lock.test.ts @@ -66,37 +66,15 @@ function expectGatewayLock(lock: Awaited>) function resolveLockPath(env: NodeJS.ProcessEnv) { const stateDir = resolveStateDir(env); const configPath = resolveConfigPath(env, stateDir); - const hash = createHash("sha256").update(configPath).digest("hex").slice(0, 8); + const configHash = createHash("sha256").update(configPath).digest("hex").slice(0, 8); + const canonicalStateDir = fsSync.realpathSync.native(path.resolve(stateDir)); + const stateHash = createHash("sha256").update(canonicalStateDir).digest("hex").slice(0, 8); const lockDir = resolveTestLockDir(); - return { lockPath: path.join(lockDir, `gateway.${hash}.lock`), configPath }; -} - -function makeProcStat(pid: number, startTime: number) { - const fields = [ - "R", - "1", - "1", - "1", - "1", - "1", - "1", - "1", - "1", - "1", - "1", - "1", - "1", - "1", - "1", - "1", - "1", - "1", - "1", - String(startTime), - "1", - "1", - ]; - return `${pid} (node) ${fields.join(" ")}`; + return { + lockPath: path.join(lockDir, `gateway.${configHash}.lock`), + configPath, + stateLockPath: path.join(lockDir, `gateway.state.${stateHash}.lock`), + }; } function createLockPayload(params: { @@ -104,12 +82,14 @@ function createLockPayload(params: { startTime: number; createdAt?: string; port?: number; + role?: "gateway" | "sqlite-maintenance"; }) { return { pid: process.pid, createdAt: params.createdAt ?? new Date().toISOString(), configPath: params.configPath, ...(params.port ? { port: params.port } : {}), + ...(params.role ? { role: params.role } : {}), startTime: params.startTime, }; } @@ -210,6 +190,78 @@ describe("gateway lock", () => { await expectGatewayLock(lock2).release(); }); + it("serializes different config paths that resolve to the same state directory", async () => { + const stateDir = await fixtureRootTracker.make("shared-state"); + const configA = path.join(stateDir, "gateway-a.json"); + const configB = path.join(stateDir, "gateway-b.json"); + await fs.writeFile(configA, "{}", "utf8"); + await fs.writeFile(configB, "{}", "utf8"); + const envA = { + ...process.env, + OPENCLAW_CONFIG_PATH: configA, + OPENCLAW_STATE_DIR: stateDir, + }; + const envB = { + ...process.env, + OPENCLAW_CONFIG_PATH: configB, + OPENCLAW_STATE_DIR: stateDir, + }; + const lock = expectGatewayLock( + await acquireForTest(envA, { + platform: "darwin", + }), + ); + + try { + await expect( + acquireForTest(envB, { + platform: "darwin", + readProcessCmdline: () => ["openclaw-gateway"], + timeoutMs: 15, + }), + ).rejects.toBeInstanceOf(GatewayLockError); + } finally { + await lock.release(); + } + }); + + it.skipIf(process.platform === "win32")( + "canonicalizes state-directory aliases before choosing the ownership lock", + async () => { + const stateDir = await fixtureRootTracker.make("canonical-state"); + const aliasRoot = await fixtureRootTracker.make("canonical-alias"); + const stateAlias = path.join(aliasRoot, "state-link"); + const configA = path.join(stateDir, "gateway-a.json"); + const configB = path.join(aliasRoot, "gateway-b.json"); + await fs.writeFile(configA, "{}", "utf8"); + await fs.writeFile(configB, "{}", "utf8"); + await fs.symlink(stateDir, stateAlias); + const envA = { + ...process.env, + OPENCLAW_CONFIG_PATH: configA, + OPENCLAW_STATE_DIR: stateDir, + }; + const envB = { + ...process.env, + OPENCLAW_CONFIG_PATH: configB, + OPENCLAW_STATE_DIR: stateAlias, + }; + const lock = expectGatewayLock(await acquireForTest(envA, { platform: "darwin" })); + + try { + await expect( + acquireForTest(envB, { + platform: "darwin", + readProcessCmdline: () => ["openclaw-gateway"], + timeoutMs: 15, + }), + ).rejects.toBeInstanceOf(GatewayLockError); + } finally { + await lock.release(); + } + }, + ); + it("records and reads the active runtime port from a verified gateway lock", async () => { const env = await makeEnv(); const lock = expectGatewayLock( @@ -234,10 +286,69 @@ describe("gateway lock", () => { } }); + it("reads the active runtime port from state ownership without a config lock", async () => { + const env = { + ...(await makeEnv()), + OPENCLAW_ALLOW_MULTI_GATEWAY: "1", + VITEST: "", + }; + const lock = expectGatewayLock( + await acquireForTest(env, { + platform: "darwin", + port: 48789, + readProcessCmdline: () => ["openclaw-gateway"], + }), + ); + + try { + const { lockPath } = resolveLockPath(env); + await expect(fs.access(lockPath)).rejects.toMatchObject({ code: "ENOENT" }); + await expect( + readActiveGatewayLockPort({ + env, + lockDir: resolveTestLockDir(), + platform: "darwin", + readProcessCmdline: () => ["openclaw-gateway"], + }), + ).resolves.toBe(48789); + } finally { + await lock.release(); + } + }); + + it("reads the active runtime port across configs that share a state directory", async () => { + const envA = await makeEnv(); + const configB = path.join(resolveStateDir(envA), "gateway-b.json"); + await fs.writeFile(configB, "{}", "utf8"); + const envB = { ...envA, OPENCLAW_CONFIG_PATH: configB }; + const lock = expectGatewayLock( + await acquireForTest(envA, { + platform: "darwin", + port: 48789, + readProcessCmdline: () => ["openclaw-gateway"], + }), + ); + + try { + const { lockPath } = resolveLockPath(envB); + await expect(fs.access(lockPath)).rejects.toMatchObject({ code: "ENOENT" }); + await expect( + readActiveGatewayLockPort({ + env: envB, + lockDir: resolveTestLockDir(), + platform: "darwin", + readProcessCmdline: () => ["openclaw-gateway"], + }), + ).resolves.toBe(48789); + } finally { + await lock.release(); + } + }); + it("keeps a retitled gateway lock owned during concurrent acquisition", async () => { const env = await makeEnv(); const lock = expectGatewayLock(await acquireForTest(env, { platform: "darwin", port: 48789 })); - const connectSpy = createPortProbeConnectionSpy("connect"); + const connectSpy = createPortProbeConnectionSpy("refused"); try { await expect( @@ -248,12 +359,64 @@ describe("gateway lock", () => { readProcessCmdline: () => ["openclaw-gateway"], }), ).rejects.toBeInstanceOf(GatewayLockError); - expect(connectSpy).toHaveBeenCalled(); + expect(connectSpy).not.toHaveBeenCalled(); } finally { await lock.release(); } }); + it("keeps a verified owner when a second gateway requests a different unbound port", async () => { + const env = await makeEnv(); + const lock = expectGatewayLock( + await acquireForTest(env, { + platform: "darwin", + port: 18789, + }), + ); + const connectSpy = createPortProbeConnectionSpy("refused"); + + try { + await expect( + acquireForTest(env, { + platform: "darwin", + port: 28789, + timeoutMs: 15, + readProcessCmdline: () => ["openclaw-gateway"], + }), + ).rejects.toBeInstanceOf(GatewayLockError); + expect(connectSpy).not.toHaveBeenCalled(); + } finally { + connectSpy.mockRestore(); + await lock.release(); + } + }); + + it("keeps a live SQLite maintenance owner when the requested gateway port is free", async () => { + const env = await makeEnv(); + const lock = expectGatewayLock( + await acquireForTest(env, { + ...({ role: "sqlite-maintenance" } as GatewayLockOptions), + platform: "darwin", + }), + ); + const connectSpy = createPortProbeConnectionSpy("refused"); + + try { + await expect( + acquireForTest(env, { + platform: "darwin", + port: 18789, + timeoutMs: 15, + readProcessCmdline: () => ["openclaw", "doctor", "--state-sqlite", "compact"], + }), + ).rejects.toBeInstanceOf(GatewayLockError); + expect(connectSpy).not.toHaveBeenCalled(); + } finally { + connectSpy.mockRestore(); + await lock.release(); + } + }); + it("ignores active-port metadata when the lock owner cannot be verified", async () => { const env = await makeEnv(); const { lockPath, configPath } = resolveLockPath(env); @@ -276,25 +439,66 @@ describe("gateway lock", () => { const payload = createLockPayload({ configPath, startTime: 111 }); await fs.writeFile(lockPath, JSON.stringify(payload), "utf8"); - const statValue = makeProcStat(process.pid, 222); - const spy = mockProcStatRead({ - onProcRead: () => statValue, - }); - const lock = await acquireForTest(env, { timeoutMs: 80, pollIntervalMs: 5, platform: "linux", + readProcessStartTime: () => 222, }); const acquiredLock = expectGatewayLock(lock); await acquiredLock.release(); - spy.mockRestore(); + }); + + it("serializes concurrent stale-lock reclamation", async () => { + vi.useRealTimers(); + const env = await makeEnv(); + const { configPath, stateLockPath } = resolveLockPath(env); + await fs.mkdir(path.dirname(stateLockPath), { recursive: true }); + await fs.writeFile( + stateLockPath, + JSON.stringify(createLockPayload({ configPath, startTime: 111 })), + "utf8", + ); + + const attempts = await Promise.allSettled([ + acquireForTest(env, { + platform: "linux", + readProcessStartTime: () => 222, + timeoutMs: 80, + }), + acquireForTest(env, { + platform: "linux", + readProcessStartTime: () => 222, + timeoutMs: 25, + }), + ]); + const acquired = attempts.filter( + (result): result is PromiseFulfilledResult => + result.status === "fulfilled" && result.value !== null, + ); + const rejected = attempts.filter( + (result): result is PromiseRejectedResult => result.status === "rejected", + ); + + expect(acquired).toHaveLength(1); + expect(rejected).toHaveLength(1); + expect(rejected[0]?.reason).toBeInstanceOf(GatewayLockError); + await expect(fs.access(stateLockPath)).resolves.toBeUndefined(); + + const acquiredResult = acquired[0]; + if (!acquiredResult) { + throw new Error("Expected one successful stale-lock contender"); + } + await acquiredResult.value.release(); + const nextLock = expectGatewayLock(await acquireForTest(env)); + await nextLock.release(); }); it("keeps lock on linux when proc access fails unless stale", async () => { vi.useRealTimers(); const env = await makeEnv(); + const { stateLockPath } = resolveLockPath(env); await writeLockFile(env); const spy = createEaccesProcStatSpy(); @@ -302,12 +506,196 @@ describe("gateway lock", () => { timeoutMs: 15, staleMs: 10_000, platform: "linux", + readProcessCmdline: () => null, }); await expect(pending).rejects.toBeInstanceOf(GatewayLockError); + await expect(fs.access(stateLockPath)).rejects.toMatchObject({ code: "ENOENT" }); spy.mockRestore(); }); + it("keeps a verified maintenance owner when process start identity is unavailable", async () => { + vi.useRealTimers(); + const env = await makeEnv(); + const { lockPath, configPath } = resolveLockPath(env); + await fs.writeFile( + lockPath, + JSON.stringify( + createLockPayload({ + configPath, + createdAt: "2000-01-01T00:00:00.000Z", + role: "sqlite-maintenance", + startTime: 111, + }), + ), + "utf8", + ); + const spy = createEaccesProcStatSpy(); + + try { + await expect( + acquireForTest(env, { + timeoutMs: 15, + staleMs: 0, + platform: "linux", + readProcessCmdline: () => [ + "node", + "/srv/openclaw/openclaw.mjs", + "doctor", + "--state-sqlite", + "compact", + ], + }), + ).rejects.toBeInstanceOf(GatewayLockError); + } finally { + spy.mockRestore(); + } + }); + + it("reclaims a maintenance lock when its live pid belongs to another process", async () => { + vi.useRealTimers(); + const env = await makeEnv(); + const { lockPath, configPath } = resolveLockPath(env); + await fs.writeFile( + lockPath, + JSON.stringify( + createLockPayload({ + configPath, + role: "sqlite-maintenance", + startTime: 111, + }), + ), + "utf8", + ); + const spy = createEaccesProcStatSpy(); + + try { + const lock = await acquireForTest(env, { + platform: "linux", + readProcessCmdline: () => ["node", "worker.js"], + timeoutMs: 80, + }); + await expectGatewayLock(lock).release(); + } finally { + spy.mockRestore(); + } + }); + + it("reclaims a Windows maintenance lock when the pid creation time changed", async () => { + vi.useRealTimers(); + const env = await makeEnv(); + const { lockPath, configPath } = resolveLockPath(env); + await fs.writeFile( + lockPath, + JSON.stringify( + createLockPayload({ + configPath, + role: "sqlite-maintenance", + startTime: 111, + }), + ), + "utf8", + ); + + const lock = await acquireForTest(env, { + platform: "win32", + readProcessCmdline: () => ["openclaw", "doctor", "--state-sqlite", "compact"], + readProcessStartTime: () => 222, + timeoutMs: 80, + }); + await expectGatewayLock(lock).release(); + }); + + it("keeps a Windows maintenance lock when the pid creation time matches", async () => { + vi.useRealTimers(); + const env = await makeEnv(); + const { lockPath, configPath } = resolveLockPath(env); + await fs.writeFile( + lockPath, + JSON.stringify( + createLockPayload({ + configPath, + createdAt: "2000-01-01T00:00:00.000Z", + role: "sqlite-maintenance", + startTime: 111, + }), + ), + "utf8", + ); + + await expect( + acquireForTest(env, { + platform: "win32", + readProcessCmdline: () => ["node", "worker.js"], + readProcessStartTime: () => 111, + staleMs: 0, + timeoutMs: 15, + }), + ).rejects.toBeInstanceOf(GatewayLockError); + }); + + it("fails closed for a recent maintenance owner with unreadable process identity", async () => { + vi.useRealTimers(); + const env = await makeEnv(); + const { lockPath, configPath } = resolveLockPath(env); + await fs.writeFile( + lockPath, + JSON.stringify( + createLockPayload({ + configPath, + role: "sqlite-maintenance", + startTime: 111, + }), + ), + "utf8", + ); + const spy = createEaccesProcStatSpy(); + + try { + await expect( + acquireForTest(env, { + platform: "linux", + readProcessCmdline: () => null, + staleMs: 10_000, + timeoutMs: 15, + }), + ).rejects.toBeInstanceOf(GatewayLockError); + } finally { + spy.mockRestore(); + } + }); + + it("ages out an old maintenance owner with unreadable process identity", async () => { + vi.useRealTimers(); + const env = await makeEnv(); + const { lockPath, configPath } = resolveLockPath(env); + await fs.writeFile( + lockPath, + JSON.stringify( + createLockPayload({ + configPath, + createdAt: "2000-01-01T00:00:00.000Z", + role: "sqlite-maintenance", + startTime: 111, + }), + ), + "utf8", + ); + const spy = createEaccesProcStatSpy(); + + try { + const lock = await acquireForTest(env, { + platform: "linux", + readProcessCmdline: () => null, + staleMs: 0, + timeoutMs: 80, + }); + await expectGatewayLock(lock).release(); + } finally { + spy.mockRestore(); + } + }); + it("keeps lock when fs.stat fails until payload is stale", async () => { vi.useRealTimers(); const env = await makeEnv(); @@ -321,6 +709,7 @@ describe("gateway lock", () => { timeoutMs: 20, staleMs: 10_000, platform: "linux", + readProcessCmdline: () => null, }); await expect(pending).rejects.toBeInstanceOf(GatewayLockError); @@ -328,11 +717,10 @@ describe("gateway lock", () => { statSpy.mockRestore(); }); - it("treats lock as stale when owner pid is alive but configured port is free", async () => { + it("reclaims a lock when its live pid belongs to a non-gateway process", async () => { vi.useRealTimers(); const env = await makeEnv(); await writeRecentLockFile(env); - const connectSpy = createPortProbeConnectionSpy("refused"); const lock = await acquireForTest(env, { timeoutMs: 80, @@ -340,9 +728,9 @@ describe("gateway lock", () => { staleMs: 10_000, platform: "darwin", port: 18789, + readProcessCmdline: () => ["node", "worker.js"], }); await expectGatewayLock(lock).release(); - connectSpy.mockRestore(); }); it("keeps lock when configured port is busy and owner pid is alive", async () => { @@ -392,13 +780,34 @@ describe("gateway lock", () => { expect(sleepDelays).toEqual([5]); }); - it("returns null when multi-gateway override is enabled", async () => { + it("keeps state ownership when the config singleton override is enabled", async () => { const env = await makeEnv(); - const lock = await acquireGatewayLock({ - env: { ...env, OPENCLAW_ALLOW_MULTI_GATEWAY: "1", VITEST: "" }, - lockDir: resolveTestLockDir(), - }); - expect(lock).toBeNull(); + const { lockPath, stateLockPath } = resolveLockPath(env); + const lock = expectGatewayLock( + await acquireGatewayLock({ + allowInTests: true, + env: { ...env, OPENCLAW_ALLOW_MULTI_GATEWAY: "1", VITEST: "" }, + lockDir: resolveTestLockDir(), + }), + ); + + try { + expect(lock.lockPath).toBe(stateLockPath); + await expect(fs.access(stateLockPath)).resolves.toBeUndefined(); + await expect(fs.access(lockPath)).rejects.toMatchObject({ code: "ENOENT" }); + await expect( + acquireGatewayLock({ + allowInTests: true, + env, + lockDir: resolveTestLockDir(), + platform: "darwin", + readProcessCmdline: () => ["openclaw-gateway"], + timeoutMs: 15, + }), + ).rejects.toBeInstanceOf(GatewayLockError); + } finally { + await lock.release(); + } }); it("returns null in test env unless allowInTests is set", async () => { @@ -451,7 +860,7 @@ describe("gateway lock", () => { it("closes handle and removes lock file when writeFile fails after open succeeds", async () => { vi.useRealTimers(); const env = await makeEnv(); - const { lockPath } = resolveLockPath(env); + const { stateLockPath } = resolveLockPath(env); const writeError = Object.assign(new Error("ENOSPC: no space left on device"), { code: "ENOSPC", @@ -459,7 +868,7 @@ describe("gateway lock", () => { const close = vi.fn<() => Promise>().mockResolvedValue(undefined); const mockHandle = { writeFile: vi.fn().mockImplementation(async () => { - await fs.writeFile(lockPath, "partial", "utf8"); + await fs.writeFile(stateLockPath, "partial", "utf8"); throw writeError; }), close, @@ -473,7 +882,7 @@ describe("gateway lock", () => { }); expect(close).toHaveBeenCalledTimes(1); - await expect(fs.access(lockPath)).rejects.toMatchObject({ code: "ENOENT" }); + await expect(fs.access(stateLockPath)).rejects.toMatchObject({ code: "ENOENT" }); openSpy.mockRestore(); }); @@ -492,6 +901,7 @@ describe("gateway lock", () => { platform: "win32", port: 18789, readProcessCmdline: () => ["chrome.exe", "--no-sandbox"], + readProcessStartTime: () => null, }); await expectGatewayLock(lock).release(); @@ -516,6 +926,7 @@ describe("gateway lock", () => { "gateway", "run", ], + readProcessStartTime: () => null, }); await expect(pending).rejects.toBeInstanceOf(GatewayLockError); @@ -536,6 +947,7 @@ describe("gateway lock", () => { platform: "win32", port: 18789, readProcessCmdline: () => null, + readProcessStartTime: () => null, }); await expect(pending).rejects.toBeInstanceOf(GatewayLockError); diff --git a/src/infra/gateway-lock.ts b/src/infra/gateway-lock.ts index 7905224e08f..de66bc616fa 100644 --- a/src/infra/gateway-lock.ts +++ b/src/infra/gateway-lock.ts @@ -2,8 +2,8 @@ import { execFileSync } from "node:child_process"; import fsSync from "node:fs"; import fs from "node:fs/promises"; -import net from "node:net"; import path from "node:path"; +import type { DatabaseSync } from "node:sqlite"; import { resolvePositiveTimerTimeoutMs, resolveTimerTimeoutMs, @@ -11,22 +11,28 @@ import { } from "@openclaw/normalization-core/number-coercion"; import { z } from "zod"; import { resolveConfigPath, resolveGatewayLockDir, resolveStateDir } from "../config/paths.js"; -import { isPidAlive } from "../shared/pid-alive.js"; +import { getFileLockProcessStartTime, isPidAlive } from "../shared/pid-alive.js"; import { safeParseJsonWithSchema } from "../utils/zod-parse.js"; import { sha256HexPrefix } from "./crypto-digest.js"; -import { isGatewayArgv, parseProcCmdline } from "./gateway-process-argv.js"; -import { readWindowsProcessArgsSync } from "./windows-port-pids.js"; +import { isGatewayArgv, isOpenClawCommandArgv, parseProcCmdline } from "./gateway-process-argv.js"; +import { requireNodeSqlite } from "./node-sqlite.js"; +import { isSqliteLockError } from "./sqlite-transaction.js"; +import { + readWindowsProcessArgsSync, + readWindowsProcessStartTimeSync, +} from "./windows-port-pids.js"; const DEFAULT_TIMEOUT_MS = 5000; const DEFAULT_POLL_INTERVAL_MS = 100; const DEFAULT_STALE_MS = 30_000; -const DEFAULT_PORT_PROBE_TIMEOUT_MS = 1000; type LockPayload = { pid: number; createdAt: string; configPath: string; port?: number; + role?: GatewayLockRole; + stateDir?: string; startTime?: number; }; @@ -35,15 +41,20 @@ const LockPayloadSchema = z.object({ createdAt: z.string(), configPath: z.string(), port: z.number().int().min(1).max(65_535).optional(), + role: z.enum(["gateway", "sqlite-maintenance"]).optional(), + stateDir: z.string().optional(), startTime: z.number().optional(), }) as z.ZodType; type GatewayLockHandle = { lockPath: string; + stateLockPath: string; configPath: string; release: () => Promise; }; +export type GatewayLockRole = "gateway" | "sqlite-maintenance"; + export type GatewayLockOptions = { env?: NodeJS.ProcessEnv; timeoutMs?: number; @@ -55,8 +66,11 @@ export type GatewayLockOptions = { now?: () => number; sleep?: (ms: number) => Promise; lockDir?: string; + role?: GatewayLockRole; /** Override process command-line reader (testing seam). */ readProcessCmdline?: (pid: number) => string[] | null; + /** Override process start-identity reader (testing seam). */ + readProcessStartTime?: (pid: number) => number | null; }; export class GatewayLockError extends Error { @@ -71,6 +85,45 @@ export class GatewayLockError extends Error { type LockOwnerStatus = "alive" | "dead" | "unknown"; +type GatewayLockCoordinator = { + release: () => void; +}; + +function tryAcquireGatewayLockCoordinator(lockPath: string): GatewayLockCoordinator | null { + const { DatabaseSync } = requireNodeSqlite(); + const coordinatorDb: DatabaseSync = new DatabaseSync(`${lockPath}.sqlite`); + try { + coordinatorDb.exec("PRAGMA busy_timeout = 0; BEGIN EXCLUSIVE;"); + } catch (error) { + try { + coordinatorDb.close(); + } catch {} + if (isSqliteLockError(error)) { + return null; + } + throw error; + } + + return { + release: () => { + let releaseError: unknown; + try { + coordinatorDb.exec("ROLLBACK"); + } catch (error) { + releaseError = error; + } + try { + coordinatorDb.close(); + } catch (error) { + releaseError ??= error; + } + if (releaseError) { + throw new GatewayLockError("failed to release gateway lock coordinator", releaseError); + } + }, + }; +} + function readLinuxCmdline(pid: number): string[] | null { try { const raw = fsSync.readFileSync(`/proc/${pid}/cmdline`, "utf8"); @@ -111,48 +164,13 @@ function readDarwinCmdline(pid: number): string[] | null { } } -function readLinuxStartTime(pid: number): number | null { - try { - const raw = fsSync.readFileSync(`/proc/${pid}/stat`, "utf8").trim(); - const closeParen = raw.lastIndexOf(")"); - if (closeParen < 0) { - return null; - } - const rest = raw.slice(closeParen + 1).trim(); - const fields = rest.split(/\s+/); - const startTime = Number.parseInt(fields[19] ?? "", 10); - return Number.isFinite(startTime) ? startTime : null; - } catch { +function readProcessStartTime(pid: number, platform: NodeJS.Platform): number | null { + if (platform !== process.platform) { return null; } -} - -async function checkPortFree(port: number, host = "127.0.0.1"): Promise { - return await new Promise((resolve) => { - const socket = net.createConnection({ port, host }); - let settled = false; - const finish = (result: boolean) => { - if (settled) { - return; - } - settled = true; - clearTimeout(timer); - socket.removeAllListeners(); - socket.destroy(); - resolve(result); - }; - const timer = setTimeout(() => { - // Conservative for liveness checks: timeout usually means no responsive - // local listener, so treat the lock owner as stale. - finish(true); - }, DEFAULT_PORT_PROBE_TIMEOUT_MS); - socket.once("connect", () => { - finish(false); - }); - socket.once("error", () => { - finish(true); - }); - }); + return platform === "win32" + ? readWindowsProcessStartTimeSync(pid, CMDLINE_EXEC_TIMEOUT_MS) + : getFileLockProcessStartTime(pid); } function defaultReadProcessCmdline(pid: number, platform: NodeJS.Platform): string[] | null { @@ -172,35 +190,36 @@ async function resolveGatewayOwnerStatus( pid: number, payload: LockPayload | null, platform: NodeJS.Platform, - port: number | undefined, readCmdline?: (pid: number) => string[] | null, + readStartTime?: (pid: number) => number | null, opts: { trustUnknownCmdlineOwner?: boolean } = {}, ): Promise { - if (port != null) { - const portFree = await checkPortFree(port); - if (portFree) { - return "dead"; - } - } - + const role = payload?.role ?? "gateway"; if (!isPidAlive(pid)) { return "dead"; } - // On Linux, an extra start-time comparison catches PID recycling even when - // the replacement process also looks like a gateway (same argv shape). - if (platform === "linux") { - const payloadStartTime = payload?.startTime; - if (Number.isFinite(payloadStartTime)) { - const currentStartTime = readLinuxStartTime(pid); - if (currentStartTime == null) { - return "unknown"; - } + // Process start identity catches PID recycling even when the replacement + // process has the same argv shape. + const payloadStartTime = payload?.startTime; + if (Number.isFinite(payloadStartTime)) { + const currentStartTime = ( + readStartTime ?? ((ownerPid) => readProcessStartTime(ownerPid, platform)) + )(pid); + if (currentStartTime != null) { return currentStartTime === payloadStartTime ? "alive" : "dead"; } } const readFn = readCmdline ?? ((p: number) => defaultReadProcessCmdline(p, platform)); + if (role === "sqlite-maintenance") { + const args = readFn(pid); + if (!args) { + return "unknown"; + } + return isOpenClawCommandArgv(args, "doctor") ? "alive" : "dead"; + } + const args = readFn(pid); if (!args) { // Cmdline reader unavailable or failed. On Linux legacy locks (no @@ -223,29 +242,69 @@ async function readLockPayload(lockPath: string): Promise { } } -function resolveGatewayLockPath(env: NodeJS.ProcessEnv, lockDir = resolveGatewayLockDir()) { - const stateDir = resolveStateDir(env); - const configPath = resolveConfigPath(env, stateDir); - const hash = sha256HexPrefix(configPath, 8); - const lockPath = path.join(lockDir, `gateway.${hash}.lock`); - return { lockPath, configPath }; +function canonicalizeStateDir(stateDir: string): string { + const resolved = path.resolve(stateDir); + try { + return fsSync.realpathSync.native(resolved); + } catch { + const missingSegments: string[] = []; + let current = resolved; + while (true) { + const parent = path.dirname(current); + if (parent === current) { + return resolved; + } + missingSegments.push(path.basename(current)); + current = parent; + try { + return path.join(fsSync.realpathSync.native(current), ...missingSegments.toReversed()); + } catch { + // Keep walking so aliases in an existing ancestor still share one lock. + } + } + } +} + +function resolveGatewayLockPaths(env: NodeJS.ProcessEnv, lockDir = resolveGatewayLockDir()) { + const resolvedStateDir = resolveStateDir(env); + const stateDir = canonicalizeStateDir(resolvedStateDir); + const configPath = resolveConfigPath(env, resolvedStateDir); + const configHash = sha256HexPrefix(configPath, 8); + const stateHash = sha256HexPrefix(stateDir, 8); + return { + configLockPath: path.join(lockDir, `gateway.${configHash}.lock`), + configPath, + stateDir, + stateLockPath: path.join(lockDir, `gateway.state.${stateHash}.lock`), + }; } export async function readActiveGatewayLockPort( - opts: Pick = {}, + opts: Pick< + GatewayLockOptions, + "env" | "lockDir" | "platform" | "readProcessCmdline" | "readProcessStartTime" + > = {}, ): Promise { const env = opts.env ?? process.env; - const { lockPath } = resolveGatewayLockPath(env, opts.lockDir); + const { configLockPath, stateLockPath } = resolveGatewayLockPaths(env, opts.lockDir); + const configPort = await readVerifiedGatewayLockPort(configLockPath, opts); + return configPort ?? (await readVerifiedGatewayLockPort(stateLockPath, opts)); +} + +async function readVerifiedGatewayLockPort( + lockPath: string, + opts: Pick, +): Promise { const payload = await readLockPayload(lockPath); - if (!payload?.port) { + if (!payload?.port || payload.role === "sqlite-maintenance") { return undefined; } const ownerStatus = await resolveGatewayOwnerStatus( payload.pid, payload, opts.platform ?? process.platform, - undefined, opts.readProcessCmdline, + opts.readProcessStartTime, { trustUnknownCmdlineOwner: false }, ); return ownerStatus === "alive" ? payload.port : undefined; @@ -256,13 +315,77 @@ export async function acquireGatewayLock( ): Promise { const env = opts.env ?? process.env; const allowInTests = opts.allowInTests === true; - if ( - env.OPENCLAW_ALLOW_MULTI_GATEWAY === "1" || - (!allowInTests && (env.VITEST || env.NODE_ENV === "test")) - ) { + if (!allowInTests && (env.VITEST || env.NODE_ENV === "test")) { return null; } + const role = opts.role ?? "gateway"; + const paths = resolveGatewayLockPaths(env, opts.lockDir); + const stateLock = await acquireLockFile({ + ...opts, + configPath: paths.configPath, + env, + lockPath: paths.stateLockPath, + role, + stateDir: paths.stateDir, + }); + const shouldAcquireConfigLock = + role === "sqlite-maintenance" || env.OPENCLAW_ALLOW_MULTI_GATEWAY !== "1"; + if (!shouldAcquireConfigLock) { + return { + ...stateLock, + stateLockPath: stateLock.lockPath, + }; + } + try { + const configLock = await acquireLockFile({ + ...opts, + configPath: paths.configPath, + env, + lockPath: paths.configLockPath, + role, + stateDir: paths.stateDir, + }); + return { + ...configLock, + stateLockPath: stateLock.lockPath, + release: async () => { + let releaseError: Error | undefined; + try { + await configLock.release(); + } catch (error) { + releaseError = + error instanceof Error + ? error + : new GatewayLockError("failed to release config lock", error); + } + try { + await stateLock.release(); + } catch (error) { + releaseError ??= + error instanceof Error + ? error + : new GatewayLockError("failed to release state lock", error); + } + if (releaseError) { + throw releaseError; + } + }, + }; + } catch (error) { + await stateLock.release().catch(() => undefined); + throw error; + } +} + +async function acquireLockFile( + opts: GatewayLockOptions & { + configPath: string; + lockPath: string; + role: GatewayLockRole; + stateDir: string; + }, +): Promise> { const timeoutMs = resolveTimerTimeoutMs(opts.timeoutMs, DEFAULT_TIMEOUT_MS, 0); const pollIntervalMs = resolvePositiveTimerTimeoutMs( opts.pollIntervalMs, @@ -271,6 +394,7 @@ export async function acquireGatewayLock( const staleMs = resolveTimerTimeoutMs(opts.staleMs, DEFAULT_STALE_MS, 0); const platform = opts.platform ?? process.platform; const port = opts.port; + const role = opts.role; const now = opts.now ?? Date.now; const sleep = opts.sleep ?? @@ -278,95 +402,161 @@ export async function acquireGatewayLock( await new Promise((resolve) => { setTimeout(resolve, ms); })); - const { lockPath, configPath } = resolveGatewayLockPath(env, opts.lockDir); + const { configPath, lockPath, stateDir } = opts; await fs.mkdir(path.dirname(lockPath), { recursive: true }); const startedAt = now(); let lastPayload: LockPayload | null = null; while (now() - startedAt < timeoutMs) { + let coordinator: GatewayLockCoordinator | null; try { - const handle = await fs.open(lockPath, "wx"); - try { - const startTime = platform === "linux" ? readLinuxStartTime(process.pid) : null; - const payload: LockPayload = { - pid: process.pid, - createdAt: resolveTimestampMsToIsoString(now()), - configPath, - }; - if (typeof port === "number" && Number.isInteger(port) && port > 0 && port <= 65_535) { - payload.port = port; - } - if (typeof startTime === "number" && Number.isFinite(startTime)) { - payload.startTime = startTime; - } - await handle.writeFile(JSON.stringify(payload), "utf8"); - } catch (error) { - // Acquisition owns both resources until the release callback exists. - // Unwind them if payload preparation fails before ownership transfers. - await handle.close().catch(() => undefined); - await fs.rm(lockPath, { force: true }).catch(() => undefined); - throw error; - } - return { - lockPath, - configPath, - release: async () => { - await handle.close().catch(() => undefined); - await fs.rm(lockPath, { force: true }); - }, - }; - } catch (err) { - const code = (err as { code?: unknown }).code; - if (code !== "EEXIST") { - throw new GatewayLockError(`failed to acquire gateway lock at ${lockPath}`, err); - } + coordinator = tryAcquireGatewayLockCoordinator(lockPath); + } catch (error) { + throw new GatewayLockError(`failed to acquire gateway lock at ${lockPath}`, error); + } + if (!coordinator) { lastPayload = await readLockPayload(lockPath); - const ownerPid = lastPayload?.pid; - const ownerStatus = ownerPid - ? await resolveGatewayOwnerStatus( - ownerPid, - lastPayload, - platform, - port, - opts.readProcessCmdline, - ) - : "unknown"; - if (ownerStatus === "dead" && ownerPid) { - await fs.rm(lockPath, { force: true }); - continue; - } - if (ownerStatus !== "alive") { - let stale = false; - if (lastPayload?.createdAt) { - const createdAt = Date.parse(lastPayload.createdAt); - stale = Number.isFinite(createdAt) ? now() - createdAt > staleMs : false; - } - if (!stale) { + } else { + let handle: Awaited> | undefined; + let acquisitionError: unknown; + let waitForOwner = false; + try { + while (!handle && !waitForOwner) { + let candidateHandle: Awaited>; try { - const st = await fs.stat(lockPath); - stale = now() - st.mtimeMs > staleMs; - } catch { - // On Windows or locked filesystems we may be unable to stat the - // lock file even though the existing gateway is still healthy. - // Treat the lock as non-stale so we keep waiting instead of - // forcefully removing another gateway's lock. - stale = false; + candidateHandle = await fs.open(lockPath, "wx"); + } catch (error) { + const code = (error as { code?: unknown }).code; + if (code !== "EEXIST") { + throw error; + } + + lastPayload = await readLockPayload(lockPath); + const ownerPid = lastPayload?.pid; + const ownerStatus = ownerPid + ? await resolveGatewayOwnerStatus( + ownerPid, + lastPayload, + platform, + opts.readProcessCmdline, + opts.readProcessStartTime, + ) + : "unknown"; + if (ownerStatus === "dead" && ownerPid) { + await fs.rm(lockPath, { force: true }); + continue; + } + if (ownerStatus !== "alive") { + let stale = false; + if (lastPayload?.createdAt) { + const createdAt = Date.parse(lastPayload.createdAt); + stale = Number.isFinite(createdAt) ? now() - createdAt > staleMs : false; + } + if (!stale) { + try { + const st = await fs.stat(lockPath); + stale = now() - st.mtimeMs > staleMs; + } catch { + // On Windows or locked filesystems we may be unable to stat the + // lock file even though the existing gateway is still healthy. + // Treat the lock as non-stale so we keep waiting instead of + // forcefully removing another gateway's lock. + stale = false; + } + } + if (stale) { + await fs.rm(lockPath, { force: true }); + continue; + } + } + waitForOwner = true; + continue; + } + + try { + const startTime = ( + opts.readProcessStartTime ?? ((pid) => readProcessStartTime(pid, platform)) + )(process.pid); + const payload: LockPayload = { + pid: process.pid, + createdAt: resolveTimestampMsToIsoString(now()), + configPath, + stateDir, + }; + if (typeof port === "number" && Number.isInteger(port) && port > 0 && port <= 65_535) { + payload.port = port; + } + if (role !== "gateway") { + payload.role = role; + } + if (typeof startTime === "number" && Number.isFinite(startTime)) { + payload.startTime = startTime; + } + await candidateHandle.writeFile(JSON.stringify(payload), "utf8"); + handle = candidateHandle; + } catch (error) { + // Acquisition owns both resources until the release callback exists. + // Unwind them if payload preparation fails before ownership transfers. + await candidateHandle.close().catch(() => undefined); + await fs.rm(lockPath, { force: true }).catch(() => undefined); + throw error; } } - if (stale) { - await fs.rm(lockPath, { force: true }); - continue; - } + } catch (error) { + acquisitionError = error; } - const remainingMs = timeoutMs - (now() - startedAt); - if (remainingMs <= 0) { - break; + if (handle) { + return { + lockPath, + configPath, + release: async () => { + let releaseError: unknown; + try { + await handle.close(); + } catch (error) { + releaseError = error; + } + try { + await fs.rm(lockPath, { force: true }); + } catch (error) { + releaseError ??= error; + } + try { + coordinator.release(); + } catch (error) { + releaseError ??= error; + } + if (releaseError) { + throw new GatewayLockError( + `failed to release gateway lock at ${lockPath}`, + releaseError, + ); + } + }, + }; + } + + try { + coordinator.release(); + } catch (error) { + acquisitionError ??= error; + } + if (acquisitionError) { + throw new GatewayLockError( + `failed to acquire gateway lock at ${lockPath}`, + acquisitionError, + ); } - await sleep(Math.min(pollIntervalMs, remainingMs)); } + + const remainingMs = timeoutMs - (now() - startedAt); + if (remainingMs <= 0) { + break; + } + await sleep(Math.min(pollIntervalMs, remainingMs)); } const owner = lastPayload?.pid ? ` (pid ${lastPayload.pid})` : ""; diff --git a/src/infra/gateway-process-argv.test.ts b/src/infra/gateway-process-argv.test.ts index b3f9d6585b2..e4170a19c5b 100644 --- a/src/infra/gateway-process-argv.test.ts +++ b/src/infra/gateway-process-argv.test.ts @@ -1,6 +1,6 @@ // Tests gateway process argv parsing for diagnostics. import { describe, expect, it } from "vitest"; -import { isGatewayArgv, parseProcCmdline } from "./gateway-process-argv.js"; +import { isGatewayArgv, isOpenClawCommandArgv, parseProcCmdline } from "./gateway-process-argv.js"; describe("parseProcCmdline", () => { it("splits null-delimited argv and trims empty entries", () => { @@ -53,3 +53,22 @@ describe("isGatewayArgv", () => { expect(isGatewayArgv(["python", "gateway", "script.py"])).toBe(false); }); }); + +describe("isOpenClawCommandArgv", () => { + it("matches doctor across source, built, and installed entrypoints", () => { + expect(isOpenClawCommandArgv(["node", "/srv/openclaw/openclaw.mjs", "doctor"], "doctor")).toBe( + true, + ); + expect( + isOpenClawCommandArgv(["NODE", "C:\\OpenClaw\\DIST\\ENTRY.JS", "DOCTOR"], "doctor"), + ).toBe(true); + expect(isOpenClawCommandArgv(["C:\\bin\\openclaw.cmd", "doctor", "--fix"], "doctor")).toBe( + true, + ); + }); + + it("rejects other OpenClaw commands and unrelated doctor processes", () => { + expect(isOpenClawCommandArgv(["openclaw", "gateway"], "doctor")).toBe(false); + expect(isOpenClawCommandArgv(["python", "doctor", "worker.py"], "doctor")).toBe(false); + }); +}); diff --git a/src/infra/gateway-process-argv.ts b/src/infra/gateway-process-argv.ts index ee985cc4fd2..d155f8c842d 100644 --- a/src/infra/gateway-process-argv.ts +++ b/src/infra/gateway-process-argv.ts @@ -6,33 +6,37 @@ function normalizeProcArg(arg: string): string { return normalizeLowercaseStringOrEmpty(arg.replaceAll("\\", "/")); } +const ENTRY_CANDIDATES = [ + "dist/index.js", + "dist/entry.js", + "openclaw.mjs", + "scripts/run-node.mjs", + "src/entry.ts", + "src/index.ts", +] as const; + export function parseProcCmdline(raw: string): string[] { return normalizeStringEntries(raw.split("\0")); } +export function isOpenClawCommandArgv(args: string[], command: string): boolean { + const normalized = args.map(normalizeProcArg); + const exe = (normalized[0] ?? "").replace(/\.(bat|cmd|exe)$/i, ""); + if (!normalized.includes(normalizeProcArg(command))) { + return false; + } + if (normalized.some((arg) => ENTRY_CANDIDATES.some((entry) => arg.endsWith(entry)))) { + return true; + } + return exe.endsWith("/openclaw") || exe === "openclaw"; +} + export function isGatewayArgv(args: string[], opts?: { allowGatewayBinary?: boolean }): boolean { const normalized = args.map(normalizeProcArg); const exe = (normalized[0] ?? "").replace(/\.(bat|cmd|exe)$/i, ""); const isGatewayBinary = exe.endsWith("/openclaw-gateway") || exe === "openclaw-gateway"; - if (!normalized.includes("gateway")) { + if (!isOpenClawCommandArgv(args, "gateway")) { return opts?.allowGatewayBinary === true && isGatewayBinary; } - - const entryCandidates = [ - "dist/index.js", - "dist/entry.js", - "openclaw.mjs", - "scripts/run-node.mjs", - "src/entry.ts", - "src/index.ts", - ]; - if (normalized.some((arg) => entryCandidates.some((entry) => arg.endsWith(entry)))) { - return true; - } - - return ( - exe.endsWith("/openclaw") || - exe === "openclaw" || - (opts?.allowGatewayBinary === true && isGatewayBinary) - ); + return true; } diff --git a/src/infra/windows-port-pids.test.ts b/src/infra/windows-port-pids.test.ts new file mode 100644 index 00000000000..2b27c1258ec --- /dev/null +++ b/src/infra/windows-port-pids.test.ts @@ -0,0 +1,45 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { getWindowsPowerShellExePath, getWindowsWmicExePath } from "./windows-install-roots.js"; +import { readWindowsProcessStartTimeSync } from "./windows-port-pids.js"; + +const spawnSyncMock = vi.hoisted(() => vi.fn()); + +vi.mock("node:child_process", async (importOriginal) => ({ + ...(await importOriginal()), + spawnSync: spawnSyncMock, +})); + +describe("readWindowsProcessStartTimeSync", () => { + beforeEach(() => { + spawnSyncMock.mockReset(); + }); + + it("reads an ISO creation time through PowerShell", () => { + spawnSyncMock.mockReturnValueOnce({ + status: 0, + stdout: "2026-07-13T07:20:49.1234567Z", + } as never); + + expect(readWindowsProcessStartTimeSync(123, 1000)).toBe(Date.parse("2026-07-13T07:20:49.123Z")); + expect(spawnSyncMock.mock.calls[0]?.[0]).toBe(getWindowsPowerShellExePath()); + }); + + it("falls back to WMIC DMTF creation time output", () => { + spawnSyncMock.mockReturnValueOnce({ status: 1, stdout: "" } as never).mockReturnValueOnce({ + status: 0, + stdout: Buffer.from("CreationDate=20260713092049.123456+120\r\n"), + } as never); + + expect(readWindowsProcessStartTimeSync(456, 1000)).toBe(Date.parse("2026-07-13T07:20:49.123Z")); + expect(spawnSyncMock.mock.calls[1]?.[0]).toBe(getWindowsWmicExePath()); + }); + + it("returns null when process creation time is unavailable", () => { + spawnSyncMock + .mockReturnValueOnce({ status: 1, stdout: "" } as never) + .mockReturnValueOnce({ status: 1, stdout: Buffer.alloc(0) } as never); + + expect(readWindowsProcessStartTimeSync(789, 1000)).toBeNull(); + expect(readWindowsProcessStartTimeSync(0, 1000)).toBeNull(); + }); +}); diff --git a/src/infra/windows-port-pids.ts b/src/infra/windows-port-pids.ts index 66bbd907aef..0723a783c11 100644 --- a/src/infra/windows-port-pids.ts +++ b/src/infra/windows-port-pids.ts @@ -1,4 +1,4 @@ -// Resolves Windows process ids that own listening ports. +// Resolves Windows process identity and listening-port ownership. import { spawnSync } from "node:child_process"; import { normalizeLowercaseStringOrEmpty } from "@openclaw/normalization-core/string-coerce"; import { normalizeStringEntries } from "@openclaw/normalization-core/string-normalization"; @@ -81,7 +81,7 @@ export function readWindowsListeningPidsResultSync( } // --------------------------------------------------------------------------- -// Windows process-args reading (PowerShell → WMIC fallback) +// Windows process identity reading (PowerShell → WMIC fallback) // --------------------------------------------------------------------------- function decodeWindowsProcessOutput(output: Buffer | string): string { @@ -105,6 +105,77 @@ function extractWindowsCommandLine(raw: Buffer | string): string | null { return lines.find((line) => normalizeLowercaseStringOrEmpty(line) !== "commandline") ?? null; } +function parseWindowsProcessStartTime(raw: Buffer | string): number | null { + const lines = normalizeStringEntries(decodeWindowsProcessOutput(raw).split(/\r?\n/)); + const value = + lines + .find((line) => normalizeLowercaseStringOrEmpty(line).startsWith("creationdate=")) + ?.slice("creationdate=".length) + .trim() ?? + lines.find((line) => normalizeLowercaseStringOrEmpty(line) !== "creationdate") ?? + ""; + const parsedIso = Date.parse(value); + if (Number.isFinite(parsedIso)) { + return parsedIso; + } + const dmtf = value.match(/^(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})\.(\d{6})([+-])(\d{3})$/); + if (!dmtf) { + return null; + } + const [, year, month, day, hour, minute, second, microseconds, offsetSign, offset] = dmtf; + const localTimeMs = Date.UTC( + Number(year), + Number(month) - 1, + Number(day), + Number(hour), + Number(minute), + Number(second), + Math.floor(Number(microseconds) / 1000), + ); + const offsetMs = Number(offset) * 60_000 * (offsetSign === "+" ? 1 : -1); + return localTimeMs - offsetMs; +} + +/** Read a stable Windows process creation time for lock-owner identity checks. */ +export function readWindowsProcessStartTimeSync( + pid: number, + timeoutMs = DEFAULT_TIMEOUT_MS, +): number | null { + if (!Number.isInteger(pid) || pid <= 0) { + return null; + } + const powershell = spawnSync( + getWindowsPowerShellExePath(), + [ + "-NoProfile", + "-NonInteractive", + "-Command", + `$process = Get-CimInstance Win32_Process -Filter "ProcessId = ${pid}" -ErrorAction Stop; [Console]::Out.Write($process.CreationDate.ToUniversalTime().ToString("o"))`, + ], + { + encoding: "utf8", + timeout: timeoutMs, + windowsHide: true, + }, + ); + if (!powershell.error && powershell.status === 0) { + const startTime = parseWindowsProcessStartTime(powershell.stdout); + if (startTime !== null) { + return startTime; + } + } + const wmic = spawnSync( + getWindowsWmicExePath(), + ["process", "where", `ProcessId=${pid}`, "get", "CreationDate", "/value"], + { + timeout: timeoutMs, + windowsHide: true, + stdio: ["ignore", "pipe", "ignore"], + }, + ); + return !wmic.error && wmic.status === 0 ? parseWindowsProcessStartTime(wmic.stdout) : null; +} + export function readWindowsProcessArgsSync( pid: number, timeoutMs = DEFAULT_TIMEOUT_MS, diff --git a/test/cli-state-sqlite.e2e.test.ts b/test/cli-state-sqlite.e2e.test.ts new file mode 100644 index 00000000000..a7b79b4c2f3 --- /dev/null +++ b/test/cli-state-sqlite.e2e.test.ts @@ -0,0 +1,199 @@ +// SQLite CLI E2E tests cover startup and target ownership before offline maintenance. +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import path from "node:path"; +import { withTempHome } from "openclaw/plugin-sdk/test-env"; +import { describe, expect, it } from "vitest"; +import { + closeOpenClawStateDatabase, + openOpenClawStateDatabase, +} from "../src/state/openclaw-state-db.js"; + +describe("SQLite CLI maintenance ownership", () => { + it("compacts after full CLI startup without retaining a config-health database handle", async () => { + await withTempHome( + async (tempHome) => { + const stateDir = path.join(tempHome, ".openclaw"); + const env: NodeJS.ProcessEnv = { + ...process.env, + HOME: tempHome, + USERPROFILE: tempHome, + OPENCLAW_DISABLE_BUNDLED_PLUGINS: "1", + OPENCLAW_STATE_DIR: stateDir, + OPENCLAW_TEST_FAST: "1", + }; + delete env.OPENCLAW_CONFIG_PATH; + delete env.OPENCLAW_HOME; + delete env.VITEST; + + try { + const database = openOpenClawStateDatabase({ env }); + database.db.exec(` + CREATE TABLE compact_cli_payload ( + id INTEGER PRIMARY KEY, + payload TEXT NOT NULL + ); + BEGIN IMMEDIATE; + `); + const insert = database.db.prepare( + "INSERT INTO compact_cli_payload (payload) VALUES (?)", + ); + for (let index = 0; index < 256; index += 1) { + insert.run(`${index}:${"x".repeat(8_192)}`); + } + database.db.exec(` + COMMIT; + DELETE FROM compact_cli_payload; + PRAGMA wal_checkpoint(TRUNCATE); + `); + } finally { + closeOpenClawStateDatabase(); + } + + const entry = path.resolve(process.cwd(), "src/entry.ts"); + const result = spawnSync( + process.execPath, + ["--import", "tsx", entry, "doctor", "--state-sqlite", "compact", "--json"], + { + cwd: process.cwd(), + env, + encoding: "utf8", + timeout: 60_000, + }, + ); + + expect(result.status, result.stderr || result.stdout).toBe(0); + const report = JSON.parse(result.stdout.trim()) as { + after: { autoVacuum: number; freelistPages: number }; + before: { freelistPages: number }; + integrityCheck: string; + quickCheck: string; + skipped: boolean; + }; + expect(report).toMatchObject({ + after: { + autoVacuum: 2, + freelistPages: 0, + }, + integrityCheck: "ok", + quickCheck: "ok", + skipped: false, + }); + expect(report.before.freelistPages).toBeGreaterThan(0); + expect(fs.existsSync(path.join(stateDir, "state", "openclaw.sqlite"))).toBe(true); + }, + { prefix: "openclaw-state-sqlite-cli-" }, + ); + }, 90_000); + + it("rejects destructive explicit session stores outside the active state owner", async () => { + await withTempHome( + async (tempHome) => { + const stateDir = path.join(tempHome, ".openclaw"); + const externalStorePath = path.join( + tempHome, + "external-state", + "agents", + "main", + "sessions", + "sessions.json", + ); + const env: NodeJS.ProcessEnv = { + ...process.env, + HOME: tempHome, + USERPROFILE: tempHome, + OPENCLAW_DISABLE_BUNDLED_PLUGINS: "1", + OPENCLAW_STATE_DIR: stateDir, + OPENCLAW_TEST_FAST: "1", + }; + delete env.OPENCLAW_CONFIG_PATH; + delete env.OPENCLAW_HOME; + delete env.VITEST; + + const entry = path.resolve(process.cwd(), "src/entry.ts"); + const result = spawnSync( + process.execPath, + [ + "--import", + "tsx", + entry, + "doctor", + "--session-sqlite", + "compact", + "--session-sqlite-store", + externalStorePath, + "--json", + ], + { + cwd: process.cwd(), + env, + encoding: "utf8", + timeout: 60_000, + }, + ); + + expect(result.status).not.toBe(0); + expect(`${result.stderr}\n${result.stdout}`).toContain( + "outside the active OpenClaw state directory", + ); + expect(fs.existsSync(externalStorePath)).toBe(false); + }, + { prefix: "openclaw-session-sqlite-cli-" }, + ); + }, 90_000); + + it("rejects hard-linked SQLite sidecars before destructive maintenance", async () => { + await withTempHome( + async (tempHome) => { + const stateDir = path.join(tempHome, ".openclaw"); + const storePath = path.join(stateDir, "agents", "main", "sessions", "sessions.json"); + const sqlitePath = path.join(stateDir, "agents", "main", "agent", "openclaw-agent.sqlite"); + const externalWalPath = path.join(tempHome, "external-state", "openclaw-agent.sqlite-wal"); + fs.mkdirSync(path.dirname(storePath), { recursive: true }); + fs.mkdirSync(path.dirname(sqlitePath), { recursive: true }); + fs.mkdirSync(path.dirname(externalWalPath), { recursive: true }); + fs.writeFileSync(storePath, "{}\n", "utf8"); + fs.writeFileSync(externalWalPath, "external wal\n", "utf8"); + fs.linkSync(externalWalPath, `${sqlitePath}-wal`); + const env: NodeJS.ProcessEnv = { + ...process.env, + HOME: tempHome, + USERPROFILE: tempHome, + OPENCLAW_DISABLE_BUNDLED_PLUGINS: "1", + OPENCLAW_STATE_DIR: stateDir, + OPENCLAW_TEST_FAST: "1", + }; + delete env.OPENCLAW_CONFIG_PATH; + delete env.OPENCLAW_HOME; + delete env.VITEST; + + const entry = path.resolve(process.cwd(), "src/entry.ts"); + const result = spawnSync( + process.execPath, + [ + "--import", + "tsx", + entry, + "doctor", + "--session-sqlite", + "compact", + "--session-sqlite-store", + storePath, + "--json", + ], + { + cwd: process.cwd(), + env, + encoding: "utf8", + timeout: 60_000, + }, + ); + + expect(result.status).not.toBe(0); + expect(`${result.stderr}\n${result.stdout}`).toContain("hard-linked path"); + expect(fs.readFileSync(externalWalPath, "utf8")).toBe("external wal\n"); + }, + { prefix: "openclaw-session-sqlite-sidecar-cli-" }, + ); + }, 90_000); +});