fix(sqlite): serialize offline maintenance with gateway (#106210)

This commit is contained in:
Vincent Koc
2026-07-13 18:15:32 +08:00
committed by GitHub
parent b395dfca85
commit e474ba38e8
23 changed files with 2045 additions and 290 deletions
+17 -3
View File
@@ -203,7 +203,13 @@ the container normally.
the canonical shared state database at
`<state-dir>/state/openclaw.sqlite`. It does not accept an arbitrary database
path, is never invoked by normal Gateway operation, and is not part of
`openclaw doctor --fix`.
`openclaw doctor --fix`. The command acquires the same state ownership lock as
Gateway startup and holds it through validation, checkpointing, `VACUUM`, and
the final integrity checks. It refuses to run while a Gateway or another
SQLite maintenance command owns that lock. The state lock remains active when
`OPENCLAW_ALLOW_MULTI_GATEWAY=1` skips the per-config Gateway singleton, so an
operator shell does not need to inherit the Gateway service's environment for
maintenance to detect it.
Stop the Gateway and create a verified backup first:
@@ -293,8 +299,16 @@ transcripts, run `openclaw doctor --session-sqlite compact --session-sqlite-all-
to checkpoint WAL files, run `VACUUM`, and report before/after database and WAL
sizes. Compaction requires a regular file with the current agent schema, the
selected agent's durable owner metadata, and no open handle in the doctor
process. This is explicit offline maintenance: stop the Gateway first so normal
writes cannot race the checkpoint or `VACUUM`.
process. The destructive `import`, `compact`, `recover`, and `restore` modes
hold the same state ownership lock as Gateway startup for their full operation;
`inspect`, `dry-run`, and `validate` remain read-only and do not take it. Stop
the Gateway first. Destructive modes fail instead of racing live writes or
racing another maintenance command. A destructive `--session-sqlite-store`
target must be inside the active state directory; set `OPENCLAW_STATE_DIR` to
the store's owning state directory before maintaining another installation.
Existing hard-linked targets are rejected because another path can share the
same database inode outside the locked state directory. The same ownership
checks cover SQLite WAL, shared-memory, and rollback-journal sidecars.
Each import writes a manifest under
`~/.openclaw/session-sqlite-migration-runs/` before moving transcript artifacts
+14 -9
View File
@@ -8,23 +8,26 @@ title: "Gateway lock"
## Why
- Only one gateway process should own a given config + port on a host; run additional gateways with isolated profiles and unique ports.
- Only one gateway process should own a state directory; run additional gateways with isolated profiles, state directories, configs, and ports.
- Survive crashes/SIGKILL without leaving stale lock files behind.
- Fail fast with a clear error when another gateway already owns the port.
## Two layers
## Three layers
Startup enforces single-instance ownership in two independent steps, in order:
Startup enforces ownership in three steps, in order:
1. **File lock** acquires a per-config lock file under the state lock directory. As part of acquiring it, startup probes the configured port for a live listener to detect a stale (crashed) lock owner.
2. **Socket bind** binds the HTTP/WebSocket listener (default `ws://127.0.0.1:18789`) as an exclusive TCP listener.
1. **State ownership lock** acquires a lock keyed by the canonical state directory. Every Gateway participates, including Gateways started with `OPENCLAW_ALLOW_MULTI_GATEWAY=1`, so destructive SQLite maintenance cannot race a live owner.
2. **Config lock** acquires the historical per-config lock and records the runtime port. Multi-Gateway mode skips this config singleton but retains the state ownership lock.
3. **Socket bind** binds the HTTP/WebSocket listener (default `ws://127.0.0.1:18789`) as an exclusive TCP listener.
Each layer can fail independently and throws its own `GatewayLockError`.
### File lock
### State and config locks
- If the lock file is missing, the recorded owner process is gone, or the owner's port probe shows no live listener, startup reclaims the lock and continues.
- If the lock is actively held and none of the above apply, startup retries for up to 5 seconds (default) before giving up:
- Lock liveness comes from the recorded PID, platform process start identity when available, and Gateway process identity. A verified owner remains authoritative during startup before its port begins listening.
- A dedicated SQLite coordinator serializes metadata inspection, stale-owner reclamation, and lock replacement. Its exclusive transaction is released automatically if the owning process crashes.
- If a lock file is missing or the recorded owner process is gone, startup reclaims the lock and continues.
- If either lock is actively held, startup retries for up to 5 seconds (default) before giving up:
```text
GatewayLockError("gateway already running (pid <pid>); lock timeout after <ms>ms")
@@ -45,11 +48,13 @@ Each layer can fail independently and throws its own `GatewayLockError`.
GatewayLockError("failed to bind gateway socket on ws://127.0.0.1:<port>: <cause>")
```
On shutdown, the gateway closes the HTTP/WebSocket server and removes the lock file.
On shutdown, the gateway closes the HTTP/WebSocket server and removes its state
and config lock files.
## Operational notes
- If the port is occupied by a different, non-gateway process, the error is the same; free the port or choose another with `openclaw gateway --port <port>`.
- `OPENCLAW_ALLOW_MULTI_GATEWAY=1` permits multiple config/runtime instances, not shared mutable state. Each instance still needs a unique `OPENCLAW_STATE_DIR`.
- Under a service supervisor, a new gateway process that hits either error above first probes `/healthz` on the existing process. If that process is healthy, the new process leaves it in control instead of failing. On systemd, it exits with code `78`; the unit's `RestartPreventExitStatus=78` stops `Restart=always` from looping on a lock or `EADDRINUSE` conflict. If the existing process never becomes healthy, the health-probe retry is time-bounded and startup then fails with the lock error above instead of looping forever.
- The macOS app keeps its own lightweight PID guard before spawning the gateway; the file lock and socket bind above are the actual runtime enforcement.
+3 -1
View File
@@ -91,7 +91,9 @@ Keep these unique per Gateway instance:
| `gateway.port` (or `--port`) | Unique per instance |
| Derived browser/CDP ports | See below |
Sharing any of these causes config races and port conflicts.
Sharing any of these causes config, state, or port conflicts. Gateway startup
enforces unique state-directory ownership even when
`OPENCLAW_ALLOW_MULTI_GATEWAY=1` skips the per-config singleton.
## Port mapping (derived)
+7 -1
View File
@@ -275,7 +275,13 @@ fly machine update <machine-id> --vm-memory 2048 -y
Gateway refuses to start with "already running" errors after a container restart.
The single-instance lock file lives at `<tmpdir>/openclaw-<uid>/gateway.<hash>.lock` (Linux: `/tmp/openclaw-<uid>/gateway.<hash>.lock`), not on the persistent `/data` volume, so a full container restart normally clears it along with the rest of the container filesystem. If the lock survives (for example a `fly machine restart` that preserves the container filesystem) and blocks startup, remove it manually:
The runtime lock files live at `<tmpdir>/openclaw-<uid>/gateway.<hash>.lock`
and `gateway.state.<hash>.lock` (Linux:
`/tmp/openclaw-<uid>/gateway.*.lock`), not on the persistent `/data` volume, so
a full container restart normally clears them along with the rest of the
container filesystem. If a lock survives (for example a `fly machine restart`
that preserves the container filesystem) and blocks startup, remove it
manually:
```bash
fly ssh console --command "rm -f /tmp/openclaw-*/gateway.*.lock"
+15
View File
@@ -52,6 +52,18 @@ export type CliCommandCatalogEntry = {
};
};
function hasCliOption(argv: readonly string[], name: string): boolean {
for (const arg of argv.slice(2)) {
if (arg === "--") {
return false;
}
if (arg === name || arg.startsWith(`${name}=`)) {
return true;
}
}
return false;
}
/** Command path registry used before Commander registration has loaded all plugins. */
export const cliCommandCatalog: readonly CliCommandCatalogEntry[] = [
{
@@ -285,6 +297,9 @@ export const cliCommandCatalog: readonly CliCommandCatalogEntry[] = [
policy: {
bypassConfigGuard: true,
loadPlugins: "never",
// Shared-state maintenance must acquire exclusive ownership before any
// config-health observation can open the canonical SQLite database.
networkProxy: ({ argv }) => (hasCliOption(argv, "--state-sqlite") ? "bypass" : "default"),
},
},
{ commandPath: ["exec-approvals"], policy: { networkProxy: "bypass" } },
+15 -1
View File
@@ -206,10 +206,24 @@ describe("command-path-policy", () => {
loadPlugins: "never",
networkProxy: "bypass",
});
expectResolvedPolicy(["doctor"], {
const doctorPolicy = resolveCliCommandPathPolicy(["doctor"]);
expectNetworkProxyResolver(doctorPolicy);
expect(doctorPolicy).toMatchObject({
bypassConfigGuard: true,
loadPlugins: "never",
});
expect(
doctorPolicy.networkProxy({
argv: ["node", "openclaw", "doctor"],
commandPath: ["doctor"],
}),
).toBe("default");
expect(
doctorPolicy.networkProxy({
argv: ["node", "openclaw", "doctor", "--state-sqlite=compact"],
commandPath: ["doctor"],
}),
).toBe("bypass");
expectResolvedPolicy(["config", "validate"], {
bypassConfigGuard: true,
loadPlugins: "never",
+10
View File
@@ -229,6 +229,16 @@ describe("shouldStartProxyForCli", () => {
expect(shouldStartProxyForCli(["node", "openclaw", "devices"])).toBe(false);
expect(shouldStartProxyForCli(["node", "openclaw", "mcp"])).toBe(false);
});
it("skips managed proxy routing before shared-state SQLite maintenance", () => {
expect(
shouldStartProxyForCli(["node", "openclaw", "doctor", "--state-sqlite", "compact", "--json"]),
).toBe(false);
expect(
shouldStartProxyForCli(["node", "openclaw", "doctor", "--state-sqlite=compact", "--json"]),
).toBe(false);
expect(shouldStartProxyForCli(["node", "openclaw", "doctor", "--lint"])).toBe(true);
});
});
describe("shouldUseRootHelpFastPath", () => {
@@ -8,6 +8,7 @@ import { SessionManager } from "../agents/sessions/session-manager.js";
const note = vi.hoisted(() => vi.fn());
const runDoctorSessionSqlite = vi.hoisted(() => vi.fn());
const withDoctorSqliteMaintenanceLock = vi.hoisted(() => vi.fn());
vi.mock("../../packages/terminal-core/src/note.js", () => ({
note,
@@ -17,6 +18,10 @@ vi.mock("./doctor-session-sqlite.js", () => ({
runDoctorSessionSqlite,
}));
vi.mock("./doctor-sqlite-maintenance-lock.js", () => ({
withDoctorSqliteMaintenanceLock,
}));
import {
detectSessionTranscriptHealthIssues,
noteSessionTranscriptHealth,
@@ -49,6 +54,9 @@ describe("doctor session transcript repair", () => {
beforeEach(async () => {
note.mockClear();
runDoctorSessionSqlite.mockReset();
withDoctorSqliteMaintenanceLock
.mockReset()
.mockImplementation(async (params: { run: () => unknown }) => await params.run());
root = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-doctor-transcripts-"));
});
@@ -227,6 +235,11 @@ describe("doctor session transcript repair", () => {
env,
mode: "import",
});
expect(withDoctorSqliteMaintenanceLock).toHaveBeenCalledWith({
env,
operation: "session SQLite import",
run: expect.any(Function),
});
expect(note).toHaveBeenCalledWith(
expect.stringContaining("Legacy entries: 1"),
"Session SQLite",
@@ -237,6 +250,38 @@ describe("doctor session transcript repair", () => {
);
});
it("keeps session SQLite dry-run read-only without taking maintenance ownership", async () => {
const sessionsDir = path.join(root, "agents", "main", "sessions");
await fs.mkdir(sessionsDir, { recursive: true });
runDoctorSessionSqlite.mockResolvedValueOnce({
totals: {
archivedTranscriptFiles: 0,
archivedUnreferencedJsonlFiles: 0,
importedTranscriptEvents: 0,
issues: 0,
legacyEntries: 0,
sqliteEntries: 0,
unreferencedJsonlFiles: 0,
validatedTranscriptEvents: 0,
},
});
const env = { ...process.env, OPENCLAW_STATE_DIR: root };
await noteSessionTranscriptHealth({
env,
sessionDirs: [sessionsDir],
sessionSqlite: true,
shouldRepair: false,
});
expect(runDoctorSessionSqlite).toHaveBeenCalledWith({
allAgents: true,
env,
mode: "dry-run",
});
expect(withDoctorSqliteMaintenanceLock).not.toHaveBeenCalled();
});
it("repairs supported current-version linear transcripts", async () => {
const filePath = await writeTranscript([
{ type: "session", version: 3, id: "session-linear", timestamp: "2026-06-15T00:00:00Z" },
+14 -5
View File
@@ -18,6 +18,7 @@ import {
} from "../config/sessions/transcript-tree.js";
import type { HealthFinding, HealthRepairEffect } from "../flows/health-checks.js";
import { shortenHomePath } from "../utils.js";
import { withDoctorSqliteMaintenanceLock } from "./doctor-sqlite-maintenance-lock.js";
const SESSION_TRANSCRIPTS_CHECK_ID = "core/doctor/session-transcripts";
@@ -496,11 +497,19 @@ async function noteSessionSqliteMigrationHealth(params: {
// Public doctor owns the operator-facing SQLite import; the targeted
// --session-sqlite subcommand remains the diagnostic/proof surface.
const { runDoctorSessionSqlite } = await import("./doctor-session-sqlite.js");
const report = await runDoctorSessionSqlite({
allAgents: true,
env: params.env,
mode: params.shouldRepair ? "import" : "dry-run",
});
const runSessionSqlite = async () =>
await runDoctorSessionSqlite({
allAgents: true,
env: params.env,
mode: params.shouldRepair ? "import" : "dry-run",
});
const report = params.shouldRepair
? await withDoctorSqliteMaintenanceLock({
env: params.env,
operation: "session SQLite import",
run: runSessionSqlite,
})
: await runSessionSqlite();
if (
report.totals.legacyEntries === 0 &&
report.totals.unreferencedJsonlFiles === 0 &&
@@ -0,0 +1,360 @@
import fs from "node:fs/promises";
import path from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import { useAutoCleanupTempDirTracker } from "../../test/helpers/temp-dir.js";
import { acquireGatewayLock, GatewayLockError } from "../infra/gateway-lock.js";
import {
isDestructiveDoctorSessionSqliteMode,
withDoctorSqliteMaintenanceLock,
} from "./doctor-sqlite-maintenance-lock.js";
const tempDirs = useAutoCleanupTempDirTracker((cleanup) => {
afterEach(cleanup);
});
async function createLockFixture() {
const root = tempDirs.make("openclaw-doctor-sqlite-lock-");
const stateDir = path.join(root, "state");
const configPath = path.join(stateDir, "openclaw.json");
const lockDir = path.join(root, "locks");
await fs.mkdir(stateDir, { recursive: true });
await fs.writeFile(configPath, "{}\n", "utf8");
const env = {
...process.env,
OPENCLAW_CONFIG_PATH: configPath,
OPENCLAW_STATE_DIR: stateDir,
VITEST: "1",
};
return {
env,
lockDir,
lockOptions: {
lockDir,
platform: "darwin" as const,
pollIntervalMs: 2,
readProcessCmdline: () => ["openclaw-gateway"],
timeoutMs: 15,
},
};
}
afterEach(() => {
vi.restoreAllMocks();
});
describe("doctor SQLite maintenance lock", () => {
it.each([
["inspect", false],
["dry-run", false],
["validate", false],
["import", true],
["compact", true],
["restore", true],
["recover", true],
] as const)("classifies %s mode as destructive=%s", (mode, expected) => {
expect(isDestructiveDoctorSessionSqliteMode(mode)).toBe(expected);
});
it("refuses maintenance while a verified Gateway owns the state lock", async () => {
const fixture = await createLockFixture();
const gatewayLock = await acquireGatewayLock({
allowInTests: true,
env: fixture.env,
lockDir: fixture.lockDir,
platform: "darwin",
port: 18789,
});
if (!gatewayLock) {
throw new Error("expected Gateway lock");
}
const run = vi.fn();
try {
await expect(
withDoctorSqliteMaintenanceLock(
{
env: fixture.env,
operation: "state SQLite compaction",
run,
},
{ lockOptions: fixture.lockOptions },
),
).rejects.toThrow(/Gateway or another SQLite maintenance command owns/);
expect(run).not.toHaveBeenCalled();
} finally {
await gatewayLock.release();
}
});
it("prevents Gateway startup until maintenance releases ownership", async () => {
const fixture = await createLockFixture();
let allowMaintenanceToFinish: (() => void) | undefined;
const maintenanceMayFinish = new Promise<void>((resolve) => {
allowMaintenanceToFinish = resolve;
});
let markMaintenanceStarted: (() => void) | undefined;
const maintenanceStarted = new Promise<void>((resolve) => {
markMaintenanceStarted = resolve;
});
const maintenance = withDoctorSqliteMaintenanceLock(
{
env: fixture.env,
operation: "session SQLite compaction",
run: async () => {
markMaintenanceStarted?.();
await maintenanceMayFinish;
return "done";
},
},
{ lockOptions: fixture.lockOptions },
);
await maintenanceStarted;
await expect(
acquireGatewayLock({
allowInTests: true,
env: fixture.env,
lockDir: fixture.lockDir,
platform: "darwin",
port: 18789,
pollIntervalMs: 2,
readProcessCmdline: () => ["openclaw", "doctor", "--session-sqlite", "compact"],
timeoutMs: 15,
}),
).rejects.toBeInstanceOf(GatewayLockError);
allowMaintenanceToFinish?.();
await expect(maintenance).resolves.toBe("done");
const gatewayLock = await acquireGatewayLock({
allowInTests: true,
env: fixture.env,
lockDir: fixture.lockDir,
platform: "darwin",
port: 18789,
timeoutMs: 15,
});
if (!gatewayLock) {
throw new Error("expected Gateway lock after maintenance release");
}
await gatewayLock.release();
});
it("releases ownership after maintenance fails", async () => {
const fixture = await createLockFixture();
await expect(
withDoctorSqliteMaintenanceLock(
{
env: fixture.env,
operation: "session SQLite restore",
run: () => {
throw new Error("restore failed");
},
},
{ lockOptions: fixture.lockOptions },
),
).rejects.toThrow("restore failed");
const gatewayLock = await acquireGatewayLock({
allowInTests: true,
env: fixture.env,
lockDir: fixture.lockDir,
platform: "darwin",
port: 18789,
timeoutMs: 15,
});
if (!gatewayLock) {
throw new Error("expected Gateway lock after failed maintenance");
}
await gatewayLock.release();
});
it("blocks maintenance when the Gateway used the multi-Gateway override", async () => {
const fixture = await createLockFixture();
const run = vi.fn();
const gatewayLock = await acquireGatewayLock({
allowInTests: true,
env: { ...fixture.env, OPENCLAW_ALLOW_MULTI_GATEWAY: "1" },
lockDir: fixture.lockDir,
platform: "darwin",
port: 18789,
timeoutMs: 15,
});
if (!gatewayLock) {
throw new Error("expected state ownership lock");
}
try {
await expect(
withDoctorSqliteMaintenanceLock(
{
env: fixture.env,
operation: "state SQLite compaction",
run,
},
{ lockOptions: fixture.lockOptions },
),
).rejects.toThrow(/Gateway or another SQLite maintenance command owns/);
expect(run).not.toHaveBeenCalled();
} finally {
await gatewayLock.release();
}
});
it("uses the state ownership lock when maintenance inherits the multi-Gateway override", async () => {
const fixture = await createLockFixture();
await expect(
withDoctorSqliteMaintenanceLock(
{
env: { ...fixture.env, OPENCLAW_ALLOW_MULTI_GATEWAY: "1" },
operation: "state SQLite compaction",
run: () => "done",
},
{ lockOptions: fixture.lockOptions },
),
).resolves.toBe("done");
});
it("refuses explicit destructive targets outside the locked state directory", async () => {
const fixture = await createLockFixture();
const externalPath = path.join(
tempDirs.make("openclaw-external-session-store-"),
"sessions.json",
);
const run = vi.fn();
await expect(
withDoctorSqliteMaintenanceLock(
{
env: fixture.env,
operation: "session SQLite compaction",
protectedPaths: [externalPath],
run,
},
{ lockOptions: fixture.lockOptions },
),
).rejects.toThrow(/outside the active OpenClaw state directory/);
expect(run).not.toHaveBeenCalled();
const gatewayLock = await acquireGatewayLock({
allowInTests: true,
env: fixture.env,
lockDir: fixture.lockDir,
platform: "darwin",
port: 18789,
timeoutMs: 15,
});
if (!gatewayLock) {
throw new Error("expected Gateway lock after ownership validation failure");
}
await gatewayLock.release();
});
it("refuses dangling in-state symlinks that resolve outside ownership", async () => {
if (process.platform === "win32") {
return;
}
const fixture = await createLockFixture();
const sessionsDir = path.join(fixture.env.OPENCLAW_STATE_DIR, "agents", "main", "sessions");
const storePath = path.join(sessionsDir, "sessions.json");
const outsideTarget = path.join(
tempDirs.make("openclaw-dangling-session-target-"),
"missing.json",
);
await fs.mkdir(sessionsDir, { recursive: true });
await fs.symlink(outsideTarget, storePath);
const run = vi.fn();
await expect(
withDoctorSqliteMaintenanceLock(
{
env: fixture.env,
operation: "session SQLite import",
protectedPaths: [storePath],
run,
},
{ lockOptions: fixture.lockOptions },
),
).rejects.toThrow(/outside the active OpenClaw state directory/);
expect(run).not.toHaveBeenCalled();
});
it("refuses outside path aliases that resolve into the locked state directory", async () => {
if (process.platform === "win32") {
return;
}
const fixture = await createLockFixture();
const externalAlias = path.join(
path.dirname(fixture.env.OPENCLAW_STATE_DIR),
"external-state-alias",
);
await fs.symlink(fixture.env.OPENCLAW_STATE_DIR, externalAlias, "dir");
const storePath = path.join(externalAlias, "agents", "main", "sessions", "sessions.json");
const run = vi.fn();
await expect(
withDoctorSqliteMaintenanceLock(
{
env: fixture.env,
operation: "session SQLite import",
protectedPaths: [storePath],
run,
},
{ lockOptions: fixture.lockOptions },
),
).rejects.toThrow(/outside the active OpenClaw state directory/);
expect(run).not.toHaveBeenCalled();
});
it("refuses in-state hard links that can alias storage outside ownership", async () => {
const fixture = await createLockFixture();
const sessionsDir = path.join(fixture.env.OPENCLAW_STATE_DIR, "agents", "main", "sessions");
const storePath = path.join(sessionsDir, "sessions.json");
const externalDir = path.join(path.dirname(fixture.env.OPENCLAW_STATE_DIR), "external-state");
const externalPath = path.join(externalDir, "sessions.json");
await fs.mkdir(sessionsDir, { recursive: true });
await fs.mkdir(externalDir, { recursive: true });
await fs.writeFile(externalPath, "{}\n", "utf8");
await fs.link(externalPath, storePath);
const run = vi.fn();
await expect(
withDoctorSqliteMaintenanceLock(
{
env: fixture.env,
operation: "session SQLite compaction",
protectedPaths: [storePath],
run,
},
{ lockOptions: fixture.lockOptions },
),
).rejects.toThrow(/hard-linked path/);
expect(run).not.toHaveBeenCalled();
await expect(fs.readFile(externalPath, "utf8")).resolves.toBe("{}\n");
});
it("allows explicit destructive targets owned by the locked state directory", async () => {
const fixture = await createLockFixture();
const storePath = path.join(
fixture.env.OPENCLAW_STATE_DIR,
"agents",
"main",
"sessions",
"sessions.json",
);
await expect(
withDoctorSqliteMaintenanceLock(
{
env: fixture.env,
operation: "session SQLite compaction",
protectedPaths: [storePath],
run: () => "done",
},
{ lockOptions: fixture.lockOptions },
),
).resolves.toBe("done");
});
});
@@ -0,0 +1,122 @@
/** Serializes offline SQLite maintenance against the Gateway state owner. */
import fs from "node:fs";
import path from "node:path";
import { resolveStateDir } from "../config/paths.js";
import { resolvePathViaExistingAncestorSync, resolveRootPathSync } from "../infra/boundary-path.js";
import {
acquireGatewayLock,
GatewayLockError,
type GatewayLockOptions,
} from "../infra/gateway-lock.js";
import { isPathInside } from "../infra/path-guards.js";
import type { DoctorSessionSqliteMode } from "./doctor-session-sqlite-types.js";
const MAINTENANCE_LOCK_TIMEOUT_MS = 250;
const MAINTENANCE_LOCK_POLL_INTERVAL_MS = 25;
type MaintenanceLockOptions = Pick<
GatewayLockOptions,
| "lockDir"
| "now"
| "platform"
| "pollIntervalMs"
| "readProcessCmdline"
| "readProcessStartTime"
| "sleep"
| "staleMs"
| "timeoutMs"
>;
type DoctorSqliteMaintenanceLockDeps = {
acquireLock?: typeof acquireGatewayLock;
lockOptions?: MaintenanceLockOptions;
};
function assertMaintenancePathsOwnedByStateDir(
env: NodeJS.ProcessEnv,
operation: string,
protectedPaths: readonly string[],
): void {
if (protectedPaths.length === 0) {
return;
}
const stateDir = path.resolve(resolveStateDir(env));
const stateCanonicalDir = resolvePathViaExistingAncestorSync(stateDir);
for (const protectedPath of protectedPaths) {
const absolutePath = path.resolve(protectedPath);
let resolvedPath: ReturnType<typeof resolveRootPathSync>;
try {
if (!isPathInside(stateDir, absolutePath) && !isPathInside(stateCanonicalDir, absolutePath)) {
throw new Error("path is not lexically owned by the active state directory");
}
resolvedPath = resolveRootPathSync({
absolutePath,
boundaryLabel: "OpenClaw state directory",
rootCanonicalPath: stateCanonicalDir,
rootPath: stateDir,
});
} catch (error) {
throw new Error(
`Cannot run ${operation} for a path outside the active OpenClaw state directory: ${protectedPath}. Set OPENCLAW_STATE_DIR to the owning state directory and retry.`,
{ cause: error },
);
}
if (
resolvedPath.exists &&
resolvedPath.kind === "file" &&
fs.statSync(resolvedPath.canonicalPath).nlink > 1
) {
throw new Error(
`Cannot run ${operation} for a hard-linked path: ${protectedPath}. Remove the additional hard link and retry.`,
);
}
}
}
export function isDestructiveDoctorSessionSqliteMode(mode: DoctorSessionSqliteMode): boolean {
return mode === "import" || mode === "compact" || mode === "restore" || mode === "recover";
}
/** Run one destructive doctor operation while excluding Gateway startup and peer maintenance. */
export async function withDoctorSqliteMaintenanceLock<T>(
params: {
env?: NodeJS.ProcessEnv;
operation: string;
protectedPaths?: readonly string[];
run: () => Promise<T> | T;
},
deps: DoctorSqliteMaintenanceLockDeps = {},
): Promise<T> {
const env = params.env ?? process.env;
const acquireLock = deps.acquireLock ?? acquireGatewayLock;
const lockOptions = deps.lockOptions;
let lock: Awaited<ReturnType<typeof acquireGatewayLock>>;
try {
lock = await acquireLock({
...lockOptions,
allowInTests: true,
env,
pollIntervalMs: lockOptions?.pollIntervalMs ?? MAINTENANCE_LOCK_POLL_INTERVAL_MS,
role: "sqlite-maintenance",
timeoutMs: lockOptions?.timeoutMs ?? MAINTENANCE_LOCK_TIMEOUT_MS,
});
} catch (error) {
if (error instanceof GatewayLockError) {
throw new Error(
`Cannot run ${params.operation} while the Gateway or another SQLite maintenance command owns this OpenClaw state directory. Stop the Gateway and retry.`,
{ cause: error },
);
}
throw error;
}
if (!lock) {
throw new Error(`Cannot run ${params.operation} without exclusive OpenClaw state ownership.`);
}
try {
assertMaintenancePathsOwnedByStateDir(env, params.operation, params.protectedPaths ?? []);
return await params.run();
} finally {
await lock.release();
}
}
@@ -138,10 +138,10 @@ function expectOwnerOnlySqlitePermissions(sqlitePath: string): void {
}
describe("runDoctorStateSqliteCompact", () => {
it("reports a missing canonical database as skipped", () => {
it("reports a missing canonical database as skipped", async () => {
const env = createStateEnv();
expect(runDoctorStateSqliteCompact({ env })).toEqual({
await expect(runDoctorStateSqliteCompact({ env })).resolves.toEqual({
mode: "compact",
path: resolveOpenClawStateSqlitePath(env),
reason: "missing",
@@ -149,11 +149,11 @@ describe("runDoctorStateSqliteCompact", () => {
});
});
it("compacts the canonical database and reports verified before/after state", () => {
it("compacts the canonical database and reports verified before/after state", async () => {
const env = createStateEnv();
const sqlitePath = seedStateDatabase({ env, withBloat: true });
const report = runDoctorStateSqliteCompact({ env });
const report = await runDoctorStateSqliteCompact({ env });
expectCompletedReport(report);
expect(report.path).toBe(sqlitePath);
@@ -169,20 +169,20 @@ describe("runDoctorStateSqliteCompact", () => {
expect(report.integrityCheck).toBe("ok");
});
it.skipIf(process.platform === "win32")("reapplies owner-only SQLite permissions", () => {
it.skipIf(process.platform === "win32")("reapplies owner-only SQLite permissions", async () => {
const env = createStateEnv();
const sqlitePath = seedStateDatabase({ env, withBloat: true });
runDoctorStateSqliteCompact({ env });
await runDoctorStateSqliteCompact({ env });
expectOwnerOnlySqlitePermissions(sqlitePath);
});
it("rejects non-global schema metadata before mutation", () => {
it("rejects non-global schema metadata before mutation", async () => {
const env = createStateEnv();
const sqlitePath = seedStateDatabase({ env, role: "agent", withBloat: true });
expect(() => runDoctorStateSqliteCompact({ env })).toThrow(/schema role agent.*global/);
await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(/schema role agent.*global/);
const sqlite = requireNodeSqlite();
const database = new sqlite.DatabaseSync(sqlitePath, { readOnly: true });
@@ -199,11 +199,11 @@ describe("runDoctorStateSqliteCompact", () => {
["future", OPENCLAW_STATE_SCHEMA_VERSION + 1, /uses newer schema version/],
] as const)(
"rejects a %s shared-state schema before mutation",
(_label, schemaVersion, message) => {
async (_label, schemaVersion, message) => {
const env = createStateEnv();
const sqlitePath = seedStateDatabase({ env, schemaVersion });
expect(() => runDoctorStateSqliteCompact({ env })).toThrow(message);
await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(message);
const sqlite = requireNodeSqlite();
const database = new sqlite.DatabaseSync(sqlitePath, { readOnly: true });
@@ -217,7 +217,7 @@ describe("runDoctorStateSqliteCompact", () => {
it.skipIf(process.platform === "win32")(
"refuses a symlink at the canonical database path",
() => {
async () => {
const env = createStateEnv();
const canonicalPath = resolveOpenClawStateSqlitePath(env);
const externalEnv = createStateEnv();
@@ -225,18 +225,45 @@ describe("runDoctorStateSqliteCompact", () => {
fs.mkdirSync(path.dirname(canonicalPath), { recursive: true });
fs.symlinkSync(externalPath, canonicalPath);
expect(() => runDoctorStateSqliteCompact({ env })).toThrow(/not a regular file/);
await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(/not a regular file/);
},
);
it("refuses compaction while this process owns an open shared-state handle", () => {
it("refuses compaction while this process owns an open shared-state handle", async () => {
const env = createStateEnv();
openOpenClawStateDatabase({ env });
expect(() => runDoctorStateSqliteCompact({ env })).toThrow(/already open in this process/);
await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(
/already open in this process/,
);
});
it("treats a busy truncating checkpoint as failure", () => {
it("leaves the database untouched when exclusive state ownership is unavailable", async () => {
const env = createStateEnv();
const sqlitePath = seedStateDatabase({ env, withBloat: true });
await expect(
runDoctorStateSqliteCompact(
{ env },
{
withMaintenanceLock: async () => {
throw new Error("Gateway owns this OpenClaw state directory");
},
},
),
).rejects.toThrow(/Gateway owns/);
const sqlite = requireNodeSqlite();
const database = new sqlite.DatabaseSync(sqlitePath, { readOnly: true });
try {
expect(readPragma(database, "auto_vacuum")).toBe(0);
expect(readPragma(database, "freelist_count")).toBeGreaterThan(0);
} finally {
database.close();
}
});
it("treats a busy truncating checkpoint as failure", async () => {
const env = createStateEnv();
const sqlitePath = seedStateDatabase({ env });
const sqlite = requireNodeSqlite();
@@ -247,7 +274,7 @@ describe("runDoctorStateSqliteCompact", () => {
writer.exec("INSERT INTO compact_payload (payload) VALUES ('newer wal frame');");
// Exercise the real busy checkpoint result without waiting the production lock timeout.
expect(() => runDoctorStateSqliteCompact({ env }, { busyTimeoutMs: 0 })).toThrow(
await expect(runDoctorStateSqliteCompact({ env }, { busyTimeoutMs: 0 })).rejects.toThrow(
/checkpoint remained busy/,
);
expect(readPragma(writer, "auto_vacuum")).toBe(0);
@@ -258,7 +285,7 @@ describe("runDoctorStateSqliteCompact", () => {
}
});
it("rejects stale secondary indexes before mutating the database", () => {
it("rejects stale secondary indexes before mutating the database", async () => {
const env = createStateEnv();
const sqlitePath = seedStateDatabase({ env, withBloat: true });
createUnsafeIndexDrift(sqlitePath);
@@ -277,7 +304,7 @@ describe("runDoctorStateSqliteCompact", () => {
before.close();
}
expect(() => runDoctorStateSqliteCompact({ env })).toThrow(
await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(
/integrity_check failed.*missing from index unsafe_index_records_value/iu,
);
@@ -297,7 +324,7 @@ describe("runDoctorStateSqliteCompact", () => {
}
});
it("rejects foreign-key violations before mutating the database", () => {
it("rejects foreign-key violations before mutating the database", async () => {
const env = createStateEnv();
const sqlitePath = seedStateDatabase({ env, withBloat: true });
const sqlite = requireNodeSqlite();
@@ -326,7 +353,7 @@ describe("runDoctorStateSqliteCompact", () => {
corrupted.close();
}
expect(() => runDoctorStateSqliteCompact({ env })).toThrow(
await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(
/foreign_key_check failed.*compact_children row 1 references compact_parents \(foreign key 0\)/iu,
);
+28 -19
View File
@@ -10,6 +10,7 @@ import {
compactDoctorSqliteFile,
type DoctorSqliteCompactSnapshot,
} from "./doctor-sqlite-compact.js";
import { withDoctorSqliteMaintenanceLock } from "./doctor-sqlite-maintenance-lock.js";
export type DoctorStateSqliteCompactReport =
| {
@@ -35,13 +36,14 @@ type DoctorStateSqliteCompactOptions = {
type DoctorStateSqliteCompactDeps = {
busyTimeoutMs?: number;
withMaintenanceLock?: typeof withDoctorSqliteMaintenanceLock;
};
/** Compact only the canonical shared state database resolved for this invocation. */
export function runDoctorStateSqliteCompact(
export async function runDoctorStateSqliteCompact(
options: DoctorStateSqliteCompactOptions = {},
deps: DoctorStateSqliteCompactDeps = {},
): DoctorStateSqliteCompactReport {
): Promise<DoctorStateSqliteCompactReport> {
const env = options.env ?? process.env;
const sqlitePath = resolveOpenClawStateSqlitePath(env);
const stat = readCanonicalStateDatabaseStat(sqlitePath);
@@ -56,25 +58,32 @@ export function runDoctorStateSqliteCompact(
if (!stat.isFile()) {
throw new Error(`Canonical OpenClaw state database is not a regular file: ${sqlitePath}`);
}
if (isOpenClawStateDatabaseOpen()) {
throw new Error(
"The shared OpenClaw state database is already open in this process. Stop OpenClaw and retry.",
);
}
const withMaintenanceLock = deps.withMaintenanceLock ?? withDoctorSqliteMaintenanceLock;
return await withMaintenanceLock({
env,
operation: "state SQLite compaction",
run: () => {
if (isOpenClawStateDatabaseOpen()) {
throw new Error(
"The shared OpenClaw state database is already open in this process. Stop OpenClaw and retry.",
);
}
const compact = compactDoctorSqliteFile({
afterMutation: () => ensureOpenClawStatePermissions(sqlitePath, env),
...(deps.busyTimeoutMs !== undefined ? { busyTimeoutMs: deps.busyTimeoutMs } : {}),
sqlitePath,
validateBeforeMutation: (database) =>
assertOpenClawStateDatabaseForMaintenance(database, { pathname: sqlitePath }),
const compact = compactDoctorSqliteFile({
afterMutation: () => ensureOpenClawStatePermissions(sqlitePath, env),
...(deps.busyTimeoutMs !== undefined ? { busyTimeoutMs: deps.busyTimeoutMs } : {}),
sqlitePath,
validateBeforeMutation: (database) =>
assertOpenClawStateDatabaseForMaintenance(database, { pathname: sqlitePath }),
});
return {
...compact,
mode: "compact",
path: sqlitePath,
skipped: false,
};
},
});
return {
...compact,
mode: "compact",
path: sqlitePath,
skipped: false,
};
}
function readCanonicalStateDatabaseStat(sqlitePath: string): fs.Stats | undefined {
+126 -1
View File
@@ -1,4 +1,5 @@
// Doctor command tests cover probe orchestration, fix mode, and runtime command output.
import path from "node:path";
import { beforeEach, describe, expect, it, vi } from "vitest";
const mocks = vi.hoisted(() => ({
@@ -6,6 +7,7 @@ const mocks = vi.hoisted(() => ({
runPostUpgradeProbes: vi.fn(),
runDoctorStateSqliteCompact: vi.fn(),
runDoctorSessionSqlite: vi.fn(),
withDoctorSqliteMaintenanceLock: vi.fn(),
resolveInstalledPluginIndexStorePath: vi.fn(() => "/tmp/openclaw-installed-plugins.json"),
}));
@@ -21,6 +23,12 @@ vi.mock("./doctor-state-sqlite-compact.js", () => ({
runDoctorStateSqliteCompact: mocks.runDoctorStateSqliteCompact,
}));
vi.mock("./doctor-sqlite-maintenance-lock.js", () => ({
isDestructiveDoctorSessionSqliteMode: (mode: string) =>
mode === "import" || mode === "compact" || mode === "restore" || mode === "recover",
withDoctorSqliteMaintenanceLock: mocks.withDoctorSqliteMaintenanceLock,
}));
vi.mock("./doctor-session-sqlite-github-issue.js", () => ({
createSessionSqliteGithubIssue: mocks.createSessionSqliteGithubIssue,
}));
@@ -34,6 +42,9 @@ const { doctorCommand } = await import("./doctor.js");
describe("doctorCommand", () => {
beforeEach(() => {
vi.clearAllMocks();
mocks.withDoctorSqliteMaintenanceLock.mockImplementation(
async (params: { run: () => unknown }) => await params.run(),
);
});
it("writes post-upgrade JSON through the runtime before exiting with findings", async () => {
@@ -108,10 +119,124 @@ describe("doctorCommand", () => {
agent: "main",
mode: "inspect",
});
expect(mocks.withDoctorSqliteMaintenanceLock).not.toHaveBeenCalled();
expect(runtime.writeJson).toHaveBeenCalledWith(report, 2);
expect(runtime.exit).toHaveBeenCalledWith(0);
});
it("holds exclusive state ownership for destructive session sqlite modes", async () => {
const report = {
mode: "restore",
targets: [],
totals: {
archivedTranscriptFiles: 0,
archivedUnreferencedJsonlFiles: 0,
importedEntries: 0,
importedTranscriptEvents: 0,
issues: 0,
legacyEntries: 0,
sqliteEntries: 0,
targets: 0,
unreferencedJsonlFiles: 0,
validatedEntries: 0,
validatedTranscriptEvents: 0,
},
};
mocks.runDoctorSessionSqlite.mockResolvedValueOnce(report);
const runtime = {
log: vi.fn(),
error: vi.fn(),
writeStdout: vi.fn(),
writeJson: vi.fn(),
exit: vi.fn((code: number) => {
throw new Error(`exit:${code}`);
}),
};
await expect(
doctorCommand(runtime, {
sessionSqlite: "restore",
sessionSqliteAllAgents: true,
}),
).rejects.toThrow("exit:0");
expect(mocks.withDoctorSqliteMaintenanceLock).toHaveBeenCalledWith({
env: process.env,
operation: "session SQLite restore",
run: expect.any(Function),
});
expect(mocks.runDoctorSessionSqlite).toHaveBeenCalledWith({
allAgents: true,
mode: "restore",
});
});
it("binds explicit destructive session stores to the maintenance lock", async () => {
const report = {
mode: "compact",
targets: [],
totals: {
archivedTranscriptFiles: 0,
archivedUnreferencedJsonlFiles: 0,
importedEntries: 0,
importedTranscriptEvents: 0,
issues: 0,
legacyEntries: 0,
sqliteEntries: 0,
targets: 0,
unreferencedJsonlFiles: 0,
validatedEntries: 0,
validatedTranscriptEvents: 0,
},
};
mocks.runDoctorSessionSqlite.mockResolvedValueOnce(report);
const runtime = {
log: vi.fn(),
error: vi.fn(),
writeStdout: vi.fn(),
writeJson: vi.fn(),
exit: vi.fn((code: number) => {
throw new Error(`exit:${code}`);
}),
};
const stateDir = path.resolve(process.env.OPENCLAW_STATE_DIR ?? ".openclaw");
const storePath = path.join(stateDir, "agents", "main", "sessions", "sessions.json");
const sqlitePath = path.join(stateDir, "agents", "main", "agent", "openclaw-agent.sqlite");
await expect(
doctorCommand(runtime, {
sessionSqlite: "compact",
sessionSqliteStore: storePath,
}),
).rejects.toThrow("exit:0");
expect(mocks.withDoctorSqliteMaintenanceLock).toHaveBeenCalledWith({
env: process.env,
operation: "session SQLite compact",
protectedPaths: [
storePath,
sqlitePath,
`${sqlitePath}-wal`,
`${sqlitePath}-shm`,
`${sqlitePath}-journal`,
],
run: expect.any(Function),
});
});
it("rejects conflicting explicit-store selectors before taking maintenance ownership", async () => {
await expect(
doctorCommand(undefined, {
sessionSqlite: "compact",
sessionSqliteAgent: "ops",
sessionSqliteStore: path.resolve("stores", "{agentId}", "sessions.json"),
}),
).rejects.toThrow("--store cannot be combined with --agent or --all-agents");
expect(mocks.withDoctorSqliteMaintenanceLock).not.toHaveBeenCalled();
expect(mocks.runDoctorSessionSqlite).not.toHaveBeenCalled();
});
it("writes shared-state sqlite compaction JSON through the runtime", async () => {
const report = {
after: {
@@ -135,7 +260,7 @@ describe("doctorCommand", () => {
reclaimedBytes: 12_288,
skipped: false,
};
mocks.runDoctorStateSqliteCompact.mockReturnValueOnce(report);
mocks.runDoctorStateSqliteCompact.mockResolvedValueOnce(report);
const runtime = {
log: vi.fn(),
error: vi.fn(),
+57 -8
View File
@@ -1,15 +1,52 @@
/** Top-level doctor command wrapper, including post-upgrade probe mode. */
import { resolveSqliteTargetFromSessionStorePath } from "../config/sessions/session-sqlite-target.js";
import { resolveSessionStoreTargets } from "../config/sessions/targets.js";
import { resolveSqliteDatabaseFilePaths } from "../infra/sqlite-files.js";
import { defaultRuntime, type RuntimeEnv, writeRuntimeJson } from "../runtime.js";
import { runPostUpgradeProbes } from "./doctor-post-upgrade.js";
import type { DoctorOptions } from "./doctor-prompter.js";
import type { DoctorSessionSqliteReport } from "./doctor-session-sqlite.js";
import {
isDestructiveDoctorSessionSqliteMode,
withDoctorSqliteMaintenanceLock,
} from "./doctor-sqlite-maintenance-lock.js";
function resolveExplicitSessionSqliteMaintenancePaths(options: DoctorOptions): string[] {
if (!options.sessionSqliteStore) {
return [];
}
// Explicit path mode intentionally bypasses runtime config. Resolve through
// the same selector as the migration so ownership checks cover exact targets.
const targets = resolveSessionStoreTargets(
{},
{
store: options.sessionSqliteStore,
...(options.sessionSqliteAgent ? { agent: options.sessionSqliteAgent } : {}),
...(options.sessionSqliteAllAgents ? { allAgents: true } : {}),
},
{ env: process.env },
);
const protectedPaths = new Set<string>();
for (const target of targets) {
protectedPaths.add(target.storePath);
const sqlitePath = resolveSqliteTargetFromSessionStorePath(target.storePath, {
agentId: target.agentId,
}).path;
if (sqlitePath) {
for (const databasePath of resolveSqliteDatabaseFilePaths(sqlitePath)) {
protectedPaths.add(databasePath);
}
}
}
return [...protectedPaths];
}
/** Runs doctor or the post-upgrade probe submode using the provided runtime. */
export async function doctorCommand(runtime?: RuntimeEnv, options?: DoctorOptions): Promise<void> {
if (options?.stateSqlite) {
const outputRuntime = runtime ?? defaultRuntime;
const { runDoctorStateSqliteCompact } = await import("./doctor-state-sqlite-compact.js");
const report = runDoctorStateSqliteCompact();
const report = await runDoctorStateSqliteCompact();
if (options.json) {
writeRuntimeJson(outputRuntime, report);
} else if (report.skipped) {
@@ -30,14 +67,26 @@ export async function doctorCommand(runtime?: RuntimeEnv, options?: DoctorOption
}
if (options?.sessionSqlite) {
const outputRuntime = runtime ?? defaultRuntime;
const sessionSqliteMode = options.sessionSqlite;
const { runDoctorSessionSqlite } = await import("./doctor-session-sqlite.js");
const report = await runDoctorSessionSqlite({
mode: options.sessionSqlite,
...(options.sessionSqliteStore ? { store: options.sessionSqliteStore } : {}),
...(options.sessionSqliteAgent ? { agent: options.sessionSqliteAgent } : {}),
...(options.sessionSqliteAllAgents ? { allAgents: true } : {}),
});
if (options.sessionSqlite === "recover" && options.sessionSqliteGithubIssue === true) {
const runSessionSqlite = async () =>
await runDoctorSessionSqlite({
mode: sessionSqliteMode,
...(options.sessionSqliteStore ? { store: options.sessionSqliteStore } : {}),
...(options.sessionSqliteAgent ? { agent: options.sessionSqliteAgent } : {}),
...(options.sessionSqliteAllAgents ? { allAgents: true } : {}),
});
const report = isDestructiveDoctorSessionSqliteMode(sessionSqliteMode)
? await withDoctorSqliteMaintenanceLock({
env: process.env,
operation: `session SQLite ${sessionSqliteMode}`,
...(options.sessionSqliteStore
? { protectedPaths: resolveExplicitSessionSqliteMaintenancePaths(options) }
: {}),
run: runSessionSqlite,
})
: await runSessionSqlite();
if (sessionSqliteMode === "recover" && options.sessionSqliteGithubIssue === true) {
await maybeCreateSessionSqliteGithubIssue(outputRuntime, report, options);
}
if (options.json) {
+4 -1
View File
@@ -673,7 +673,10 @@ async function runSessionLocksHealth(ctx: DoctorHealthFlowContext): Promise<void
async function runSessionTranscriptsHealth(ctx: DoctorHealthFlowContext): Promise<void> {
const { noteSessionTranscriptHealth } = await import("../commands/doctor-session-transcripts.js");
await noteSessionTranscriptHealth({ shouldRepair: ctx.prompter.shouldRepair });
await noteSessionTranscriptHealth({
env: ctx.env ?? process.env,
shouldRepair: ctx.prompter.shouldRepair,
});
}
async function runSessionSnapshotsHealth(ctx: DoctorHealthFlowContext): Promise<void> {
+462 -50
View File
@@ -66,37 +66,15 @@ function expectGatewayLock(lock: Awaited<ReturnType<typeof acquireGatewayLock>>)
function resolveLockPath(env: NodeJS.ProcessEnv) {
const stateDir = resolveStateDir(env);
const configPath = resolveConfigPath(env, stateDir);
const hash = createHash("sha256").update(configPath).digest("hex").slice(0, 8);
const configHash = createHash("sha256").update(configPath).digest("hex").slice(0, 8);
const canonicalStateDir = fsSync.realpathSync.native(path.resolve(stateDir));
const stateHash = createHash("sha256").update(canonicalStateDir).digest("hex").slice(0, 8);
const lockDir = resolveTestLockDir();
return { lockPath: path.join(lockDir, `gateway.${hash}.lock`), configPath };
}
function makeProcStat(pid: number, startTime: number) {
const fields = [
"R",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
String(startTime),
"1",
"1",
];
return `${pid} (node) ${fields.join(" ")}`;
return {
lockPath: path.join(lockDir, `gateway.${configHash}.lock`),
configPath,
stateLockPath: path.join(lockDir, `gateway.state.${stateHash}.lock`),
};
}
function createLockPayload(params: {
@@ -104,12 +82,14 @@ function createLockPayload(params: {
startTime: number;
createdAt?: string;
port?: number;
role?: "gateway" | "sqlite-maintenance";
}) {
return {
pid: process.pid,
createdAt: params.createdAt ?? new Date().toISOString(),
configPath: params.configPath,
...(params.port ? { port: params.port } : {}),
...(params.role ? { role: params.role } : {}),
startTime: params.startTime,
};
}
@@ -210,6 +190,78 @@ describe("gateway lock", () => {
await expectGatewayLock(lock2).release();
});
it("serializes different config paths that resolve to the same state directory", async () => {
const stateDir = await fixtureRootTracker.make("shared-state");
const configA = path.join(stateDir, "gateway-a.json");
const configB = path.join(stateDir, "gateway-b.json");
await fs.writeFile(configA, "{}", "utf8");
await fs.writeFile(configB, "{}", "utf8");
const envA = {
...process.env,
OPENCLAW_CONFIG_PATH: configA,
OPENCLAW_STATE_DIR: stateDir,
};
const envB = {
...process.env,
OPENCLAW_CONFIG_PATH: configB,
OPENCLAW_STATE_DIR: stateDir,
};
const lock = expectGatewayLock(
await acquireForTest(envA, {
platform: "darwin",
}),
);
try {
await expect(
acquireForTest(envB, {
platform: "darwin",
readProcessCmdline: () => ["openclaw-gateway"],
timeoutMs: 15,
}),
).rejects.toBeInstanceOf(GatewayLockError);
} finally {
await lock.release();
}
});
it.skipIf(process.platform === "win32")(
"canonicalizes state-directory aliases before choosing the ownership lock",
async () => {
const stateDir = await fixtureRootTracker.make("canonical-state");
const aliasRoot = await fixtureRootTracker.make("canonical-alias");
const stateAlias = path.join(aliasRoot, "state-link");
const configA = path.join(stateDir, "gateway-a.json");
const configB = path.join(aliasRoot, "gateway-b.json");
await fs.writeFile(configA, "{}", "utf8");
await fs.writeFile(configB, "{}", "utf8");
await fs.symlink(stateDir, stateAlias);
const envA = {
...process.env,
OPENCLAW_CONFIG_PATH: configA,
OPENCLAW_STATE_DIR: stateDir,
};
const envB = {
...process.env,
OPENCLAW_CONFIG_PATH: configB,
OPENCLAW_STATE_DIR: stateAlias,
};
const lock = expectGatewayLock(await acquireForTest(envA, { platform: "darwin" }));
try {
await expect(
acquireForTest(envB, {
platform: "darwin",
readProcessCmdline: () => ["openclaw-gateway"],
timeoutMs: 15,
}),
).rejects.toBeInstanceOf(GatewayLockError);
} finally {
await lock.release();
}
},
);
it("records and reads the active runtime port from a verified gateway lock", async () => {
const env = await makeEnv();
const lock = expectGatewayLock(
@@ -234,10 +286,69 @@ describe("gateway lock", () => {
}
});
it("reads the active runtime port from state ownership without a config lock", async () => {
const env = {
...(await makeEnv()),
OPENCLAW_ALLOW_MULTI_GATEWAY: "1",
VITEST: "",
};
const lock = expectGatewayLock(
await acquireForTest(env, {
platform: "darwin",
port: 48789,
readProcessCmdline: () => ["openclaw-gateway"],
}),
);
try {
const { lockPath } = resolveLockPath(env);
await expect(fs.access(lockPath)).rejects.toMatchObject({ code: "ENOENT" });
await expect(
readActiveGatewayLockPort({
env,
lockDir: resolveTestLockDir(),
platform: "darwin",
readProcessCmdline: () => ["openclaw-gateway"],
}),
).resolves.toBe(48789);
} finally {
await lock.release();
}
});
it("reads the active runtime port across configs that share a state directory", async () => {
const envA = await makeEnv();
const configB = path.join(resolveStateDir(envA), "gateway-b.json");
await fs.writeFile(configB, "{}", "utf8");
const envB = { ...envA, OPENCLAW_CONFIG_PATH: configB };
const lock = expectGatewayLock(
await acquireForTest(envA, {
platform: "darwin",
port: 48789,
readProcessCmdline: () => ["openclaw-gateway"],
}),
);
try {
const { lockPath } = resolveLockPath(envB);
await expect(fs.access(lockPath)).rejects.toMatchObject({ code: "ENOENT" });
await expect(
readActiveGatewayLockPort({
env: envB,
lockDir: resolveTestLockDir(),
platform: "darwin",
readProcessCmdline: () => ["openclaw-gateway"],
}),
).resolves.toBe(48789);
} finally {
await lock.release();
}
});
it("keeps a retitled gateway lock owned during concurrent acquisition", async () => {
const env = await makeEnv();
const lock = expectGatewayLock(await acquireForTest(env, { platform: "darwin", port: 48789 }));
const connectSpy = createPortProbeConnectionSpy("connect");
const connectSpy = createPortProbeConnectionSpy("refused");
try {
await expect(
@@ -248,12 +359,64 @@ describe("gateway lock", () => {
readProcessCmdline: () => ["openclaw-gateway"],
}),
).rejects.toBeInstanceOf(GatewayLockError);
expect(connectSpy).toHaveBeenCalled();
expect(connectSpy).not.toHaveBeenCalled();
} finally {
await lock.release();
}
});
it("keeps a verified owner when a second gateway requests a different unbound port", async () => {
const env = await makeEnv();
const lock = expectGatewayLock(
await acquireForTest(env, {
platform: "darwin",
port: 18789,
}),
);
const connectSpy = createPortProbeConnectionSpy("refused");
try {
await expect(
acquireForTest(env, {
platform: "darwin",
port: 28789,
timeoutMs: 15,
readProcessCmdline: () => ["openclaw-gateway"],
}),
).rejects.toBeInstanceOf(GatewayLockError);
expect(connectSpy).not.toHaveBeenCalled();
} finally {
connectSpy.mockRestore();
await lock.release();
}
});
it("keeps a live SQLite maintenance owner when the requested gateway port is free", async () => {
const env = await makeEnv();
const lock = expectGatewayLock(
await acquireForTest(env, {
...({ role: "sqlite-maintenance" } as GatewayLockOptions),
platform: "darwin",
}),
);
const connectSpy = createPortProbeConnectionSpy("refused");
try {
await expect(
acquireForTest(env, {
platform: "darwin",
port: 18789,
timeoutMs: 15,
readProcessCmdline: () => ["openclaw", "doctor", "--state-sqlite", "compact"],
}),
).rejects.toBeInstanceOf(GatewayLockError);
expect(connectSpy).not.toHaveBeenCalled();
} finally {
connectSpy.mockRestore();
await lock.release();
}
});
it("ignores active-port metadata when the lock owner cannot be verified", async () => {
const env = await makeEnv();
const { lockPath, configPath } = resolveLockPath(env);
@@ -276,25 +439,66 @@ describe("gateway lock", () => {
const payload = createLockPayload({ configPath, startTime: 111 });
await fs.writeFile(lockPath, JSON.stringify(payload), "utf8");
const statValue = makeProcStat(process.pid, 222);
const spy = mockProcStatRead({
onProcRead: () => statValue,
});
const lock = await acquireForTest(env, {
timeoutMs: 80,
pollIntervalMs: 5,
platform: "linux",
readProcessStartTime: () => 222,
});
const acquiredLock = expectGatewayLock(lock);
await acquiredLock.release();
spy.mockRestore();
});
it("serializes concurrent stale-lock reclamation", async () => {
vi.useRealTimers();
const env = await makeEnv();
const { configPath, stateLockPath } = resolveLockPath(env);
await fs.mkdir(path.dirname(stateLockPath), { recursive: true });
await fs.writeFile(
stateLockPath,
JSON.stringify(createLockPayload({ configPath, startTime: 111 })),
"utf8",
);
const attempts = await Promise.allSettled([
acquireForTest(env, {
platform: "linux",
readProcessStartTime: () => 222,
timeoutMs: 80,
}),
acquireForTest(env, {
platform: "linux",
readProcessStartTime: () => 222,
timeoutMs: 25,
}),
]);
const acquired = attempts.filter(
(result): result is PromiseFulfilledResult<GatewayLock> =>
result.status === "fulfilled" && result.value !== null,
);
const rejected = attempts.filter(
(result): result is PromiseRejectedResult => result.status === "rejected",
);
expect(acquired).toHaveLength(1);
expect(rejected).toHaveLength(1);
expect(rejected[0]?.reason).toBeInstanceOf(GatewayLockError);
await expect(fs.access(stateLockPath)).resolves.toBeUndefined();
const acquiredResult = acquired[0];
if (!acquiredResult) {
throw new Error("Expected one successful stale-lock contender");
}
await acquiredResult.value.release();
const nextLock = expectGatewayLock(await acquireForTest(env));
await nextLock.release();
});
it("keeps lock on linux when proc access fails unless stale", async () => {
vi.useRealTimers();
const env = await makeEnv();
const { stateLockPath } = resolveLockPath(env);
await writeLockFile(env);
const spy = createEaccesProcStatSpy();
@@ -302,12 +506,196 @@ describe("gateway lock", () => {
timeoutMs: 15,
staleMs: 10_000,
platform: "linux",
readProcessCmdline: () => null,
});
await expect(pending).rejects.toBeInstanceOf(GatewayLockError);
await expect(fs.access(stateLockPath)).rejects.toMatchObject({ code: "ENOENT" });
spy.mockRestore();
});
it("keeps a verified maintenance owner when process start identity is unavailable", async () => {
vi.useRealTimers();
const env = await makeEnv();
const { lockPath, configPath } = resolveLockPath(env);
await fs.writeFile(
lockPath,
JSON.stringify(
createLockPayload({
configPath,
createdAt: "2000-01-01T00:00:00.000Z",
role: "sqlite-maintenance",
startTime: 111,
}),
),
"utf8",
);
const spy = createEaccesProcStatSpy();
try {
await expect(
acquireForTest(env, {
timeoutMs: 15,
staleMs: 0,
platform: "linux",
readProcessCmdline: () => [
"node",
"/srv/openclaw/openclaw.mjs",
"doctor",
"--state-sqlite",
"compact",
],
}),
).rejects.toBeInstanceOf(GatewayLockError);
} finally {
spy.mockRestore();
}
});
it("reclaims a maintenance lock when its live pid belongs to another process", async () => {
vi.useRealTimers();
const env = await makeEnv();
const { lockPath, configPath } = resolveLockPath(env);
await fs.writeFile(
lockPath,
JSON.stringify(
createLockPayload({
configPath,
role: "sqlite-maintenance",
startTime: 111,
}),
),
"utf8",
);
const spy = createEaccesProcStatSpy();
try {
const lock = await acquireForTest(env, {
platform: "linux",
readProcessCmdline: () => ["node", "worker.js"],
timeoutMs: 80,
});
await expectGatewayLock(lock).release();
} finally {
spy.mockRestore();
}
});
it("reclaims a Windows maintenance lock when the pid creation time changed", async () => {
vi.useRealTimers();
const env = await makeEnv();
const { lockPath, configPath } = resolveLockPath(env);
await fs.writeFile(
lockPath,
JSON.stringify(
createLockPayload({
configPath,
role: "sqlite-maintenance",
startTime: 111,
}),
),
"utf8",
);
const lock = await acquireForTest(env, {
platform: "win32",
readProcessCmdline: () => ["openclaw", "doctor", "--state-sqlite", "compact"],
readProcessStartTime: () => 222,
timeoutMs: 80,
});
await expectGatewayLock(lock).release();
});
it("keeps a Windows maintenance lock when the pid creation time matches", async () => {
vi.useRealTimers();
const env = await makeEnv();
const { lockPath, configPath } = resolveLockPath(env);
await fs.writeFile(
lockPath,
JSON.stringify(
createLockPayload({
configPath,
createdAt: "2000-01-01T00:00:00.000Z",
role: "sqlite-maintenance",
startTime: 111,
}),
),
"utf8",
);
await expect(
acquireForTest(env, {
platform: "win32",
readProcessCmdline: () => ["node", "worker.js"],
readProcessStartTime: () => 111,
staleMs: 0,
timeoutMs: 15,
}),
).rejects.toBeInstanceOf(GatewayLockError);
});
it("fails closed for a recent maintenance owner with unreadable process identity", async () => {
vi.useRealTimers();
const env = await makeEnv();
const { lockPath, configPath } = resolveLockPath(env);
await fs.writeFile(
lockPath,
JSON.stringify(
createLockPayload({
configPath,
role: "sqlite-maintenance",
startTime: 111,
}),
),
"utf8",
);
const spy = createEaccesProcStatSpy();
try {
await expect(
acquireForTest(env, {
platform: "linux",
readProcessCmdline: () => null,
staleMs: 10_000,
timeoutMs: 15,
}),
).rejects.toBeInstanceOf(GatewayLockError);
} finally {
spy.mockRestore();
}
});
it("ages out an old maintenance owner with unreadable process identity", async () => {
vi.useRealTimers();
const env = await makeEnv();
const { lockPath, configPath } = resolveLockPath(env);
await fs.writeFile(
lockPath,
JSON.stringify(
createLockPayload({
configPath,
createdAt: "2000-01-01T00:00:00.000Z",
role: "sqlite-maintenance",
startTime: 111,
}),
),
"utf8",
);
const spy = createEaccesProcStatSpy();
try {
const lock = await acquireForTest(env, {
platform: "linux",
readProcessCmdline: () => null,
staleMs: 0,
timeoutMs: 80,
});
await expectGatewayLock(lock).release();
} finally {
spy.mockRestore();
}
});
it("keeps lock when fs.stat fails until payload is stale", async () => {
vi.useRealTimers();
const env = await makeEnv();
@@ -321,6 +709,7 @@ describe("gateway lock", () => {
timeoutMs: 20,
staleMs: 10_000,
platform: "linux",
readProcessCmdline: () => null,
});
await expect(pending).rejects.toBeInstanceOf(GatewayLockError);
@@ -328,11 +717,10 @@ describe("gateway lock", () => {
statSpy.mockRestore();
});
it("treats lock as stale when owner pid is alive but configured port is free", async () => {
it("reclaims a lock when its live pid belongs to a non-gateway process", async () => {
vi.useRealTimers();
const env = await makeEnv();
await writeRecentLockFile(env);
const connectSpy = createPortProbeConnectionSpy("refused");
const lock = await acquireForTest(env, {
timeoutMs: 80,
@@ -340,9 +728,9 @@ describe("gateway lock", () => {
staleMs: 10_000,
platform: "darwin",
port: 18789,
readProcessCmdline: () => ["node", "worker.js"],
});
await expectGatewayLock(lock).release();
connectSpy.mockRestore();
});
it("keeps lock when configured port is busy and owner pid is alive", async () => {
@@ -392,13 +780,34 @@ describe("gateway lock", () => {
expect(sleepDelays).toEqual([5]);
});
it("returns null when multi-gateway override is enabled", async () => {
it("keeps state ownership when the config singleton override is enabled", async () => {
const env = await makeEnv();
const lock = await acquireGatewayLock({
env: { ...env, OPENCLAW_ALLOW_MULTI_GATEWAY: "1", VITEST: "" },
lockDir: resolveTestLockDir(),
});
expect(lock).toBeNull();
const { lockPath, stateLockPath } = resolveLockPath(env);
const lock = expectGatewayLock(
await acquireGatewayLock({
allowInTests: true,
env: { ...env, OPENCLAW_ALLOW_MULTI_GATEWAY: "1", VITEST: "" },
lockDir: resolveTestLockDir(),
}),
);
try {
expect(lock.lockPath).toBe(stateLockPath);
await expect(fs.access(stateLockPath)).resolves.toBeUndefined();
await expect(fs.access(lockPath)).rejects.toMatchObject({ code: "ENOENT" });
await expect(
acquireGatewayLock({
allowInTests: true,
env,
lockDir: resolveTestLockDir(),
platform: "darwin",
readProcessCmdline: () => ["openclaw-gateway"],
timeoutMs: 15,
}),
).rejects.toBeInstanceOf(GatewayLockError);
} finally {
await lock.release();
}
});
it("returns null in test env unless allowInTests is set", async () => {
@@ -451,7 +860,7 @@ describe("gateway lock", () => {
it("closes handle and removes lock file when writeFile fails after open succeeds", async () => {
vi.useRealTimers();
const env = await makeEnv();
const { lockPath } = resolveLockPath(env);
const { stateLockPath } = resolveLockPath(env);
const writeError = Object.assign(new Error("ENOSPC: no space left on device"), {
code: "ENOSPC",
@@ -459,7 +868,7 @@ describe("gateway lock", () => {
const close = vi.fn<() => Promise<void>>().mockResolvedValue(undefined);
const mockHandle = {
writeFile: vi.fn().mockImplementation(async () => {
await fs.writeFile(lockPath, "partial", "utf8");
await fs.writeFile(stateLockPath, "partial", "utf8");
throw writeError;
}),
close,
@@ -473,7 +882,7 @@ describe("gateway lock", () => {
});
expect(close).toHaveBeenCalledTimes(1);
await expect(fs.access(lockPath)).rejects.toMatchObject({ code: "ENOENT" });
await expect(fs.access(stateLockPath)).rejects.toMatchObject({ code: "ENOENT" });
openSpy.mockRestore();
});
@@ -492,6 +901,7 @@ describe("gateway lock", () => {
platform: "win32",
port: 18789,
readProcessCmdline: () => ["chrome.exe", "--no-sandbox"],
readProcessStartTime: () => null,
});
await expectGatewayLock(lock).release();
@@ -516,6 +926,7 @@ describe("gateway lock", () => {
"gateway",
"run",
],
readProcessStartTime: () => null,
});
await expect(pending).rejects.toBeInstanceOf(GatewayLockError);
@@ -536,6 +947,7 @@ describe("gateway lock", () => {
platform: "win32",
port: 18789,
readProcessCmdline: () => null,
readProcessStartTime: () => null,
});
await expect(pending).rejects.toBeInstanceOf(GatewayLockError);
+339 -149
View File
@@ -2,8 +2,8 @@
import { execFileSync } from "node:child_process";
import fsSync from "node:fs";
import fs from "node:fs/promises";
import net from "node:net";
import path from "node:path";
import type { DatabaseSync } from "node:sqlite";
import {
resolvePositiveTimerTimeoutMs,
resolveTimerTimeoutMs,
@@ -11,22 +11,28 @@ import {
} from "@openclaw/normalization-core/number-coercion";
import { z } from "zod";
import { resolveConfigPath, resolveGatewayLockDir, resolveStateDir } from "../config/paths.js";
import { isPidAlive } from "../shared/pid-alive.js";
import { getFileLockProcessStartTime, isPidAlive } from "../shared/pid-alive.js";
import { safeParseJsonWithSchema } from "../utils/zod-parse.js";
import { sha256HexPrefix } from "./crypto-digest.js";
import { isGatewayArgv, parseProcCmdline } from "./gateway-process-argv.js";
import { readWindowsProcessArgsSync } from "./windows-port-pids.js";
import { isGatewayArgv, isOpenClawCommandArgv, parseProcCmdline } from "./gateway-process-argv.js";
import { requireNodeSqlite } from "./node-sqlite.js";
import { isSqliteLockError } from "./sqlite-transaction.js";
import {
readWindowsProcessArgsSync,
readWindowsProcessStartTimeSync,
} from "./windows-port-pids.js";
const DEFAULT_TIMEOUT_MS = 5000;
const DEFAULT_POLL_INTERVAL_MS = 100;
const DEFAULT_STALE_MS = 30_000;
const DEFAULT_PORT_PROBE_TIMEOUT_MS = 1000;
type LockPayload = {
pid: number;
createdAt: string;
configPath: string;
port?: number;
role?: GatewayLockRole;
stateDir?: string;
startTime?: number;
};
@@ -35,15 +41,20 @@ const LockPayloadSchema = z.object({
createdAt: z.string(),
configPath: z.string(),
port: z.number().int().min(1).max(65_535).optional(),
role: z.enum(["gateway", "sqlite-maintenance"]).optional(),
stateDir: z.string().optional(),
startTime: z.number().optional(),
}) as z.ZodType<LockPayload>;
type GatewayLockHandle = {
lockPath: string;
stateLockPath: string;
configPath: string;
release: () => Promise<void>;
};
export type GatewayLockRole = "gateway" | "sqlite-maintenance";
export type GatewayLockOptions = {
env?: NodeJS.ProcessEnv;
timeoutMs?: number;
@@ -55,8 +66,11 @@ export type GatewayLockOptions = {
now?: () => number;
sleep?: (ms: number) => Promise<void>;
lockDir?: string;
role?: GatewayLockRole;
/** Override process command-line reader (testing seam). */
readProcessCmdline?: (pid: number) => string[] | null;
/** Override process start-identity reader (testing seam). */
readProcessStartTime?: (pid: number) => number | null;
};
export class GatewayLockError extends Error {
@@ -71,6 +85,45 @@ export class GatewayLockError extends Error {
type LockOwnerStatus = "alive" | "dead" | "unknown";
type GatewayLockCoordinator = {
release: () => void;
};
function tryAcquireGatewayLockCoordinator(lockPath: string): GatewayLockCoordinator | null {
const { DatabaseSync } = requireNodeSqlite();
const coordinatorDb: DatabaseSync = new DatabaseSync(`${lockPath}.sqlite`);
try {
coordinatorDb.exec("PRAGMA busy_timeout = 0; BEGIN EXCLUSIVE;");
} catch (error) {
try {
coordinatorDb.close();
} catch {}
if (isSqliteLockError(error)) {
return null;
}
throw error;
}
return {
release: () => {
let releaseError: unknown;
try {
coordinatorDb.exec("ROLLBACK");
} catch (error) {
releaseError = error;
}
try {
coordinatorDb.close();
} catch (error) {
releaseError ??= error;
}
if (releaseError) {
throw new GatewayLockError("failed to release gateway lock coordinator", releaseError);
}
},
};
}
function readLinuxCmdline(pid: number): string[] | null {
try {
const raw = fsSync.readFileSync(`/proc/${pid}/cmdline`, "utf8");
@@ -111,48 +164,13 @@ function readDarwinCmdline(pid: number): string[] | null {
}
}
function readLinuxStartTime(pid: number): number | null {
try {
const raw = fsSync.readFileSync(`/proc/${pid}/stat`, "utf8").trim();
const closeParen = raw.lastIndexOf(")");
if (closeParen < 0) {
return null;
}
const rest = raw.slice(closeParen + 1).trim();
const fields = rest.split(/\s+/);
const startTime = Number.parseInt(fields[19] ?? "", 10);
return Number.isFinite(startTime) ? startTime : null;
} catch {
function readProcessStartTime(pid: number, platform: NodeJS.Platform): number | null {
if (platform !== process.platform) {
return null;
}
}
async function checkPortFree(port: number, host = "127.0.0.1"): Promise<boolean> {
return await new Promise<boolean>((resolve) => {
const socket = net.createConnection({ port, host });
let settled = false;
const finish = (result: boolean) => {
if (settled) {
return;
}
settled = true;
clearTimeout(timer);
socket.removeAllListeners();
socket.destroy();
resolve(result);
};
const timer = setTimeout(() => {
// Conservative for liveness checks: timeout usually means no responsive
// local listener, so treat the lock owner as stale.
finish(true);
}, DEFAULT_PORT_PROBE_TIMEOUT_MS);
socket.once("connect", () => {
finish(false);
});
socket.once("error", () => {
finish(true);
});
});
return platform === "win32"
? readWindowsProcessStartTimeSync(pid, CMDLINE_EXEC_TIMEOUT_MS)
: getFileLockProcessStartTime(pid);
}
function defaultReadProcessCmdline(pid: number, platform: NodeJS.Platform): string[] | null {
@@ -172,35 +190,36 @@ async function resolveGatewayOwnerStatus(
pid: number,
payload: LockPayload | null,
platform: NodeJS.Platform,
port: number | undefined,
readCmdline?: (pid: number) => string[] | null,
readStartTime?: (pid: number) => number | null,
opts: { trustUnknownCmdlineOwner?: boolean } = {},
): Promise<LockOwnerStatus> {
if (port != null) {
const portFree = await checkPortFree(port);
if (portFree) {
return "dead";
}
}
const role = payload?.role ?? "gateway";
if (!isPidAlive(pid)) {
return "dead";
}
// On Linux, an extra start-time comparison catches PID recycling even when
// the replacement process also looks like a gateway (same argv shape).
if (platform === "linux") {
const payloadStartTime = payload?.startTime;
if (Number.isFinite(payloadStartTime)) {
const currentStartTime = readLinuxStartTime(pid);
if (currentStartTime == null) {
return "unknown";
}
// Process start identity catches PID recycling even when the replacement
// process has the same argv shape.
const payloadStartTime = payload?.startTime;
if (Number.isFinite(payloadStartTime)) {
const currentStartTime = (
readStartTime ?? ((ownerPid) => readProcessStartTime(ownerPid, platform))
)(pid);
if (currentStartTime != null) {
return currentStartTime === payloadStartTime ? "alive" : "dead";
}
}
const readFn = readCmdline ?? ((p: number) => defaultReadProcessCmdline(p, platform));
if (role === "sqlite-maintenance") {
const args = readFn(pid);
if (!args) {
return "unknown";
}
return isOpenClawCommandArgv(args, "doctor") ? "alive" : "dead";
}
const args = readFn(pid);
if (!args) {
// Cmdline reader unavailable or failed. On Linux legacy locks (no
@@ -223,29 +242,69 @@ async function readLockPayload(lockPath: string): Promise<LockPayload | null> {
}
}
function resolveGatewayLockPath(env: NodeJS.ProcessEnv, lockDir = resolveGatewayLockDir()) {
const stateDir = resolveStateDir(env);
const configPath = resolveConfigPath(env, stateDir);
const hash = sha256HexPrefix(configPath, 8);
const lockPath = path.join(lockDir, `gateway.${hash}.lock`);
return { lockPath, configPath };
function canonicalizeStateDir(stateDir: string): string {
const resolved = path.resolve(stateDir);
try {
return fsSync.realpathSync.native(resolved);
} catch {
const missingSegments: string[] = [];
let current = resolved;
while (true) {
const parent = path.dirname(current);
if (parent === current) {
return resolved;
}
missingSegments.push(path.basename(current));
current = parent;
try {
return path.join(fsSync.realpathSync.native(current), ...missingSegments.toReversed());
} catch {
// Keep walking so aliases in an existing ancestor still share one lock.
}
}
}
}
function resolveGatewayLockPaths(env: NodeJS.ProcessEnv, lockDir = resolveGatewayLockDir()) {
const resolvedStateDir = resolveStateDir(env);
const stateDir = canonicalizeStateDir(resolvedStateDir);
const configPath = resolveConfigPath(env, resolvedStateDir);
const configHash = sha256HexPrefix(configPath, 8);
const stateHash = sha256HexPrefix(stateDir, 8);
return {
configLockPath: path.join(lockDir, `gateway.${configHash}.lock`),
configPath,
stateDir,
stateLockPath: path.join(lockDir, `gateway.state.${stateHash}.lock`),
};
}
export async function readActiveGatewayLockPort(
opts: Pick<GatewayLockOptions, "env" | "lockDir" | "platform" | "readProcessCmdline"> = {},
opts: Pick<
GatewayLockOptions,
"env" | "lockDir" | "platform" | "readProcessCmdline" | "readProcessStartTime"
> = {},
): Promise<number | undefined> {
const env = opts.env ?? process.env;
const { lockPath } = resolveGatewayLockPath(env, opts.lockDir);
const { configLockPath, stateLockPath } = resolveGatewayLockPaths(env, opts.lockDir);
const configPort = await readVerifiedGatewayLockPort(configLockPath, opts);
return configPort ?? (await readVerifiedGatewayLockPort(stateLockPath, opts));
}
async function readVerifiedGatewayLockPort(
lockPath: string,
opts: Pick<GatewayLockOptions, "platform" | "readProcessCmdline" | "readProcessStartTime">,
): Promise<number | undefined> {
const payload = await readLockPayload(lockPath);
if (!payload?.port) {
if (!payload?.port || payload.role === "sqlite-maintenance") {
return undefined;
}
const ownerStatus = await resolveGatewayOwnerStatus(
payload.pid,
payload,
opts.platform ?? process.platform,
undefined,
opts.readProcessCmdline,
opts.readProcessStartTime,
{ trustUnknownCmdlineOwner: false },
);
return ownerStatus === "alive" ? payload.port : undefined;
@@ -256,13 +315,77 @@ export async function acquireGatewayLock(
): Promise<GatewayLockHandle | null> {
const env = opts.env ?? process.env;
const allowInTests = opts.allowInTests === true;
if (
env.OPENCLAW_ALLOW_MULTI_GATEWAY === "1" ||
(!allowInTests && (env.VITEST || env.NODE_ENV === "test"))
) {
if (!allowInTests && (env.VITEST || env.NODE_ENV === "test")) {
return null;
}
const role = opts.role ?? "gateway";
const paths = resolveGatewayLockPaths(env, opts.lockDir);
const stateLock = await acquireLockFile({
...opts,
configPath: paths.configPath,
env,
lockPath: paths.stateLockPath,
role,
stateDir: paths.stateDir,
});
const shouldAcquireConfigLock =
role === "sqlite-maintenance" || env.OPENCLAW_ALLOW_MULTI_GATEWAY !== "1";
if (!shouldAcquireConfigLock) {
return {
...stateLock,
stateLockPath: stateLock.lockPath,
};
}
try {
const configLock = await acquireLockFile({
...opts,
configPath: paths.configPath,
env,
lockPath: paths.configLockPath,
role,
stateDir: paths.stateDir,
});
return {
...configLock,
stateLockPath: stateLock.lockPath,
release: async () => {
let releaseError: Error | undefined;
try {
await configLock.release();
} catch (error) {
releaseError =
error instanceof Error
? error
: new GatewayLockError("failed to release config lock", error);
}
try {
await stateLock.release();
} catch (error) {
releaseError ??=
error instanceof Error
? error
: new GatewayLockError("failed to release state lock", error);
}
if (releaseError) {
throw releaseError;
}
},
};
} catch (error) {
await stateLock.release().catch(() => undefined);
throw error;
}
}
async function acquireLockFile(
opts: GatewayLockOptions & {
configPath: string;
lockPath: string;
role: GatewayLockRole;
stateDir: string;
},
): Promise<Omit<GatewayLockHandle, "stateLockPath">> {
const timeoutMs = resolveTimerTimeoutMs(opts.timeoutMs, DEFAULT_TIMEOUT_MS, 0);
const pollIntervalMs = resolvePositiveTimerTimeoutMs(
opts.pollIntervalMs,
@@ -271,6 +394,7 @@ export async function acquireGatewayLock(
const staleMs = resolveTimerTimeoutMs(opts.staleMs, DEFAULT_STALE_MS, 0);
const platform = opts.platform ?? process.platform;
const port = opts.port;
const role = opts.role;
const now = opts.now ?? Date.now;
const sleep =
opts.sleep ??
@@ -278,95 +402,161 @@ export async function acquireGatewayLock(
await new Promise((resolve) => {
setTimeout(resolve, ms);
}));
const { lockPath, configPath } = resolveGatewayLockPath(env, opts.lockDir);
const { configPath, lockPath, stateDir } = opts;
await fs.mkdir(path.dirname(lockPath), { recursive: true });
const startedAt = now();
let lastPayload: LockPayload | null = null;
while (now() - startedAt < timeoutMs) {
let coordinator: GatewayLockCoordinator | null;
try {
const handle = await fs.open(lockPath, "wx");
try {
const startTime = platform === "linux" ? readLinuxStartTime(process.pid) : null;
const payload: LockPayload = {
pid: process.pid,
createdAt: resolveTimestampMsToIsoString(now()),
configPath,
};
if (typeof port === "number" && Number.isInteger(port) && port > 0 && port <= 65_535) {
payload.port = port;
}
if (typeof startTime === "number" && Number.isFinite(startTime)) {
payload.startTime = startTime;
}
await handle.writeFile(JSON.stringify(payload), "utf8");
} catch (error) {
// Acquisition owns both resources until the release callback exists.
// Unwind them if payload preparation fails before ownership transfers.
await handle.close().catch(() => undefined);
await fs.rm(lockPath, { force: true }).catch(() => undefined);
throw error;
}
return {
lockPath,
configPath,
release: async () => {
await handle.close().catch(() => undefined);
await fs.rm(lockPath, { force: true });
},
};
} catch (err) {
const code = (err as { code?: unknown }).code;
if (code !== "EEXIST") {
throw new GatewayLockError(`failed to acquire gateway lock at ${lockPath}`, err);
}
coordinator = tryAcquireGatewayLockCoordinator(lockPath);
} catch (error) {
throw new GatewayLockError(`failed to acquire gateway lock at ${lockPath}`, error);
}
if (!coordinator) {
lastPayload = await readLockPayload(lockPath);
const ownerPid = lastPayload?.pid;
const ownerStatus = ownerPid
? await resolveGatewayOwnerStatus(
ownerPid,
lastPayload,
platform,
port,
opts.readProcessCmdline,
)
: "unknown";
if (ownerStatus === "dead" && ownerPid) {
await fs.rm(lockPath, { force: true });
continue;
}
if (ownerStatus !== "alive") {
let stale = false;
if (lastPayload?.createdAt) {
const createdAt = Date.parse(lastPayload.createdAt);
stale = Number.isFinite(createdAt) ? now() - createdAt > staleMs : false;
}
if (!stale) {
} else {
let handle: Awaited<ReturnType<typeof fs.open>> | undefined;
let acquisitionError: unknown;
let waitForOwner = false;
try {
while (!handle && !waitForOwner) {
let candidateHandle: Awaited<ReturnType<typeof fs.open>>;
try {
const st = await fs.stat(lockPath);
stale = now() - st.mtimeMs > staleMs;
} catch {
// On Windows or locked filesystems we may be unable to stat the
// lock file even though the existing gateway is still healthy.
// Treat the lock as non-stale so we keep waiting instead of
// forcefully removing another gateway's lock.
stale = false;
candidateHandle = await fs.open(lockPath, "wx");
} catch (error) {
const code = (error as { code?: unknown }).code;
if (code !== "EEXIST") {
throw error;
}
lastPayload = await readLockPayload(lockPath);
const ownerPid = lastPayload?.pid;
const ownerStatus = ownerPid
? await resolveGatewayOwnerStatus(
ownerPid,
lastPayload,
platform,
opts.readProcessCmdline,
opts.readProcessStartTime,
)
: "unknown";
if (ownerStatus === "dead" && ownerPid) {
await fs.rm(lockPath, { force: true });
continue;
}
if (ownerStatus !== "alive") {
let stale = false;
if (lastPayload?.createdAt) {
const createdAt = Date.parse(lastPayload.createdAt);
stale = Number.isFinite(createdAt) ? now() - createdAt > staleMs : false;
}
if (!stale) {
try {
const st = await fs.stat(lockPath);
stale = now() - st.mtimeMs > staleMs;
} catch {
// On Windows or locked filesystems we may be unable to stat the
// lock file even though the existing gateway is still healthy.
// Treat the lock as non-stale so we keep waiting instead of
// forcefully removing another gateway's lock.
stale = false;
}
}
if (stale) {
await fs.rm(lockPath, { force: true });
continue;
}
}
waitForOwner = true;
continue;
}
try {
const startTime = (
opts.readProcessStartTime ?? ((pid) => readProcessStartTime(pid, platform))
)(process.pid);
const payload: LockPayload = {
pid: process.pid,
createdAt: resolveTimestampMsToIsoString(now()),
configPath,
stateDir,
};
if (typeof port === "number" && Number.isInteger(port) && port > 0 && port <= 65_535) {
payload.port = port;
}
if (role !== "gateway") {
payload.role = role;
}
if (typeof startTime === "number" && Number.isFinite(startTime)) {
payload.startTime = startTime;
}
await candidateHandle.writeFile(JSON.stringify(payload), "utf8");
handle = candidateHandle;
} catch (error) {
// Acquisition owns both resources until the release callback exists.
// Unwind them if payload preparation fails before ownership transfers.
await candidateHandle.close().catch(() => undefined);
await fs.rm(lockPath, { force: true }).catch(() => undefined);
throw error;
}
}
if (stale) {
await fs.rm(lockPath, { force: true });
continue;
}
} catch (error) {
acquisitionError = error;
}
const remainingMs = timeoutMs - (now() - startedAt);
if (remainingMs <= 0) {
break;
if (handle) {
return {
lockPath,
configPath,
release: async () => {
let releaseError: unknown;
try {
await handle.close();
} catch (error) {
releaseError = error;
}
try {
await fs.rm(lockPath, { force: true });
} catch (error) {
releaseError ??= error;
}
try {
coordinator.release();
} catch (error) {
releaseError ??= error;
}
if (releaseError) {
throw new GatewayLockError(
`failed to release gateway lock at ${lockPath}`,
releaseError,
);
}
},
};
}
try {
coordinator.release();
} catch (error) {
acquisitionError ??= error;
}
if (acquisitionError) {
throw new GatewayLockError(
`failed to acquire gateway lock at ${lockPath}`,
acquisitionError,
);
}
await sleep(Math.min(pollIntervalMs, remainingMs));
}
const remainingMs = timeoutMs - (now() - startedAt);
if (remainingMs <= 0) {
break;
}
await sleep(Math.min(pollIntervalMs, remainingMs));
}
const owner = lastPayload?.pid ? ` (pid ${lastPayload.pid})` : "";
+20 -1
View File
@@ -1,6 +1,6 @@
// Tests gateway process argv parsing for diagnostics.
import { describe, expect, it } from "vitest";
import { isGatewayArgv, parseProcCmdline } from "./gateway-process-argv.js";
import { isGatewayArgv, isOpenClawCommandArgv, parseProcCmdline } from "./gateway-process-argv.js";
describe("parseProcCmdline", () => {
it("splits null-delimited argv and trims empty entries", () => {
@@ -53,3 +53,22 @@ describe("isGatewayArgv", () => {
expect(isGatewayArgv(["python", "gateway", "script.py"])).toBe(false);
});
});
describe("isOpenClawCommandArgv", () => {
it("matches doctor across source, built, and installed entrypoints", () => {
expect(isOpenClawCommandArgv(["node", "/srv/openclaw/openclaw.mjs", "doctor"], "doctor")).toBe(
true,
);
expect(
isOpenClawCommandArgv(["NODE", "C:\\OpenClaw\\DIST\\ENTRY.JS", "DOCTOR"], "doctor"),
).toBe(true);
expect(isOpenClawCommandArgv(["C:\\bin\\openclaw.cmd", "doctor", "--fix"], "doctor")).toBe(
true,
);
});
it("rejects other OpenClaw commands and unrelated doctor processes", () => {
expect(isOpenClawCommandArgv(["openclaw", "gateway"], "doctor")).toBe(false);
expect(isOpenClawCommandArgv(["python", "doctor", "worker.py"], "doctor")).toBe(false);
});
});
+23 -19
View File
@@ -6,33 +6,37 @@ function normalizeProcArg(arg: string): string {
return normalizeLowercaseStringOrEmpty(arg.replaceAll("\\", "/"));
}
const ENTRY_CANDIDATES = [
"dist/index.js",
"dist/entry.js",
"openclaw.mjs",
"scripts/run-node.mjs",
"src/entry.ts",
"src/index.ts",
] as const;
export function parseProcCmdline(raw: string): string[] {
return normalizeStringEntries(raw.split("\0"));
}
export function isOpenClawCommandArgv(args: string[], command: string): boolean {
const normalized = args.map(normalizeProcArg);
const exe = (normalized[0] ?? "").replace(/\.(bat|cmd|exe)$/i, "");
if (!normalized.includes(normalizeProcArg(command))) {
return false;
}
if (normalized.some((arg) => ENTRY_CANDIDATES.some((entry) => arg.endsWith(entry)))) {
return true;
}
return exe.endsWith("/openclaw") || exe === "openclaw";
}
export function isGatewayArgv(args: string[], opts?: { allowGatewayBinary?: boolean }): boolean {
const normalized = args.map(normalizeProcArg);
const exe = (normalized[0] ?? "").replace(/\.(bat|cmd|exe)$/i, "");
const isGatewayBinary = exe.endsWith("/openclaw-gateway") || exe === "openclaw-gateway";
if (!normalized.includes("gateway")) {
if (!isOpenClawCommandArgv(args, "gateway")) {
return opts?.allowGatewayBinary === true && isGatewayBinary;
}
const entryCandidates = [
"dist/index.js",
"dist/entry.js",
"openclaw.mjs",
"scripts/run-node.mjs",
"src/entry.ts",
"src/index.ts",
];
if (normalized.some((arg) => entryCandidates.some((entry) => arg.endsWith(entry)))) {
return true;
}
return (
exe.endsWith("/openclaw") ||
exe === "openclaw" ||
(opts?.allowGatewayBinary === true && isGatewayBinary)
);
return true;
}
+45
View File
@@ -0,0 +1,45 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { getWindowsPowerShellExePath, getWindowsWmicExePath } from "./windows-install-roots.js";
import { readWindowsProcessStartTimeSync } from "./windows-port-pids.js";
const spawnSyncMock = vi.hoisted(() => vi.fn());
vi.mock("node:child_process", async (importOriginal) => ({
...(await importOriginal<typeof import("node:child_process")>()),
spawnSync: spawnSyncMock,
}));
describe("readWindowsProcessStartTimeSync", () => {
beforeEach(() => {
spawnSyncMock.mockReset();
});
it("reads an ISO creation time through PowerShell", () => {
spawnSyncMock.mockReturnValueOnce({
status: 0,
stdout: "2026-07-13T07:20:49.1234567Z",
} as never);
expect(readWindowsProcessStartTimeSync(123, 1000)).toBe(Date.parse("2026-07-13T07:20:49.123Z"));
expect(spawnSyncMock.mock.calls[0]?.[0]).toBe(getWindowsPowerShellExePath());
});
it("falls back to WMIC DMTF creation time output", () => {
spawnSyncMock.mockReturnValueOnce({ status: 1, stdout: "" } as never).mockReturnValueOnce({
status: 0,
stdout: Buffer.from("CreationDate=20260713092049.123456+120\r\n"),
} as never);
expect(readWindowsProcessStartTimeSync(456, 1000)).toBe(Date.parse("2026-07-13T07:20:49.123Z"));
expect(spawnSyncMock.mock.calls[1]?.[0]).toBe(getWindowsWmicExePath());
});
it("returns null when process creation time is unavailable", () => {
spawnSyncMock
.mockReturnValueOnce({ status: 1, stdout: "" } as never)
.mockReturnValueOnce({ status: 1, stdout: Buffer.alloc(0) } as never);
expect(readWindowsProcessStartTimeSync(789, 1000)).toBeNull();
expect(readWindowsProcessStartTimeSync(0, 1000)).toBeNull();
});
});
+73 -2
View File
@@ -1,4 +1,4 @@
// Resolves Windows process ids that own listening ports.
// Resolves Windows process identity and listening-port ownership.
import { spawnSync } from "node:child_process";
import { normalizeLowercaseStringOrEmpty } from "@openclaw/normalization-core/string-coerce";
import { normalizeStringEntries } from "@openclaw/normalization-core/string-normalization";
@@ -81,7 +81,7 @@ export function readWindowsListeningPidsResultSync(
}
// ---------------------------------------------------------------------------
// Windows process-args reading (PowerShell → WMIC fallback)
// Windows process identity reading (PowerShell → WMIC fallback)
// ---------------------------------------------------------------------------
function decodeWindowsProcessOutput(output: Buffer | string): string {
@@ -105,6 +105,77 @@ function extractWindowsCommandLine(raw: Buffer | string): string | null {
return lines.find((line) => normalizeLowercaseStringOrEmpty(line) !== "commandline") ?? null;
}
function parseWindowsProcessStartTime(raw: Buffer | string): number | null {
const lines = normalizeStringEntries(decodeWindowsProcessOutput(raw).split(/\r?\n/));
const value =
lines
.find((line) => normalizeLowercaseStringOrEmpty(line).startsWith("creationdate="))
?.slice("creationdate=".length)
.trim() ??
lines.find((line) => normalizeLowercaseStringOrEmpty(line) !== "creationdate") ??
"";
const parsedIso = Date.parse(value);
if (Number.isFinite(parsedIso)) {
return parsedIso;
}
const dmtf = value.match(/^(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})\.(\d{6})([+-])(\d{3})$/);
if (!dmtf) {
return null;
}
const [, year, month, day, hour, minute, second, microseconds, offsetSign, offset] = dmtf;
const localTimeMs = Date.UTC(
Number(year),
Number(month) - 1,
Number(day),
Number(hour),
Number(minute),
Number(second),
Math.floor(Number(microseconds) / 1000),
);
const offsetMs = Number(offset) * 60_000 * (offsetSign === "+" ? 1 : -1);
return localTimeMs - offsetMs;
}
/** Read a stable Windows process creation time for lock-owner identity checks. */
export function readWindowsProcessStartTimeSync(
pid: number,
timeoutMs = DEFAULT_TIMEOUT_MS,
): number | null {
if (!Number.isInteger(pid) || pid <= 0) {
return null;
}
const powershell = spawnSync(
getWindowsPowerShellExePath(),
[
"-NoProfile",
"-NonInteractive",
"-Command",
`$process = Get-CimInstance Win32_Process -Filter "ProcessId = ${pid}" -ErrorAction Stop; [Console]::Out.Write($process.CreationDate.ToUniversalTime().ToString("o"))`,
],
{
encoding: "utf8",
timeout: timeoutMs,
windowsHide: true,
},
);
if (!powershell.error && powershell.status === 0) {
const startTime = parseWindowsProcessStartTime(powershell.stdout);
if (startTime !== null) {
return startTime;
}
}
const wmic = spawnSync(
getWindowsWmicExePath(),
["process", "where", `ProcessId=${pid}`, "get", "CreationDate", "/value"],
{
timeout: timeoutMs,
windowsHide: true,
stdio: ["ignore", "pipe", "ignore"],
},
);
return !wmic.error && wmic.status === 0 ? parseWindowsProcessStartTime(wmic.stdout) : null;
}
export function readWindowsProcessArgsSync(
pid: number,
timeoutMs = DEFAULT_TIMEOUT_MS,
+199
View File
@@ -0,0 +1,199 @@
// SQLite CLI E2E tests cover startup and target ownership before offline maintenance.
import { spawnSync } from "node:child_process";
import fs from "node:fs";
import path from "node:path";
import { withTempHome } from "openclaw/plugin-sdk/test-env";
import { describe, expect, it } from "vitest";
import {
closeOpenClawStateDatabase,
openOpenClawStateDatabase,
} from "../src/state/openclaw-state-db.js";
describe("SQLite CLI maintenance ownership", () => {
it("compacts after full CLI startup without retaining a config-health database handle", async () => {
await withTempHome(
async (tempHome) => {
const stateDir = path.join(tempHome, ".openclaw");
const env: NodeJS.ProcessEnv = {
...process.env,
HOME: tempHome,
USERPROFILE: tempHome,
OPENCLAW_DISABLE_BUNDLED_PLUGINS: "1",
OPENCLAW_STATE_DIR: stateDir,
OPENCLAW_TEST_FAST: "1",
};
delete env.OPENCLAW_CONFIG_PATH;
delete env.OPENCLAW_HOME;
delete env.VITEST;
try {
const database = openOpenClawStateDatabase({ env });
database.db.exec(`
CREATE TABLE compact_cli_payload (
id INTEGER PRIMARY KEY,
payload TEXT NOT NULL
);
BEGIN IMMEDIATE;
`);
const insert = database.db.prepare(
"INSERT INTO compact_cli_payload (payload) VALUES (?)",
);
for (let index = 0; index < 256; index += 1) {
insert.run(`${index}:${"x".repeat(8_192)}`);
}
database.db.exec(`
COMMIT;
DELETE FROM compact_cli_payload;
PRAGMA wal_checkpoint(TRUNCATE);
`);
} finally {
closeOpenClawStateDatabase();
}
const entry = path.resolve(process.cwd(), "src/entry.ts");
const result = spawnSync(
process.execPath,
["--import", "tsx", entry, "doctor", "--state-sqlite", "compact", "--json"],
{
cwd: process.cwd(),
env,
encoding: "utf8",
timeout: 60_000,
},
);
expect(result.status, result.stderr || result.stdout).toBe(0);
const report = JSON.parse(result.stdout.trim()) as {
after: { autoVacuum: number; freelistPages: number };
before: { freelistPages: number };
integrityCheck: string;
quickCheck: string;
skipped: boolean;
};
expect(report).toMatchObject({
after: {
autoVacuum: 2,
freelistPages: 0,
},
integrityCheck: "ok",
quickCheck: "ok",
skipped: false,
});
expect(report.before.freelistPages).toBeGreaterThan(0);
expect(fs.existsSync(path.join(stateDir, "state", "openclaw.sqlite"))).toBe(true);
},
{ prefix: "openclaw-state-sqlite-cli-" },
);
}, 90_000);
it("rejects destructive explicit session stores outside the active state owner", async () => {
await withTempHome(
async (tempHome) => {
const stateDir = path.join(tempHome, ".openclaw");
const externalStorePath = path.join(
tempHome,
"external-state",
"agents",
"main",
"sessions",
"sessions.json",
);
const env: NodeJS.ProcessEnv = {
...process.env,
HOME: tempHome,
USERPROFILE: tempHome,
OPENCLAW_DISABLE_BUNDLED_PLUGINS: "1",
OPENCLAW_STATE_DIR: stateDir,
OPENCLAW_TEST_FAST: "1",
};
delete env.OPENCLAW_CONFIG_PATH;
delete env.OPENCLAW_HOME;
delete env.VITEST;
const entry = path.resolve(process.cwd(), "src/entry.ts");
const result = spawnSync(
process.execPath,
[
"--import",
"tsx",
entry,
"doctor",
"--session-sqlite",
"compact",
"--session-sqlite-store",
externalStorePath,
"--json",
],
{
cwd: process.cwd(),
env,
encoding: "utf8",
timeout: 60_000,
},
);
expect(result.status).not.toBe(0);
expect(`${result.stderr}\n${result.stdout}`).toContain(
"outside the active OpenClaw state directory",
);
expect(fs.existsSync(externalStorePath)).toBe(false);
},
{ prefix: "openclaw-session-sqlite-cli-" },
);
}, 90_000);
it("rejects hard-linked SQLite sidecars before destructive maintenance", async () => {
await withTempHome(
async (tempHome) => {
const stateDir = path.join(tempHome, ".openclaw");
const storePath = path.join(stateDir, "agents", "main", "sessions", "sessions.json");
const sqlitePath = path.join(stateDir, "agents", "main", "agent", "openclaw-agent.sqlite");
const externalWalPath = path.join(tempHome, "external-state", "openclaw-agent.sqlite-wal");
fs.mkdirSync(path.dirname(storePath), { recursive: true });
fs.mkdirSync(path.dirname(sqlitePath), { recursive: true });
fs.mkdirSync(path.dirname(externalWalPath), { recursive: true });
fs.writeFileSync(storePath, "{}\n", "utf8");
fs.writeFileSync(externalWalPath, "external wal\n", "utf8");
fs.linkSync(externalWalPath, `${sqlitePath}-wal`);
const env: NodeJS.ProcessEnv = {
...process.env,
HOME: tempHome,
USERPROFILE: tempHome,
OPENCLAW_DISABLE_BUNDLED_PLUGINS: "1",
OPENCLAW_STATE_DIR: stateDir,
OPENCLAW_TEST_FAST: "1",
};
delete env.OPENCLAW_CONFIG_PATH;
delete env.OPENCLAW_HOME;
delete env.VITEST;
const entry = path.resolve(process.cwd(), "src/entry.ts");
const result = spawnSync(
process.execPath,
[
"--import",
"tsx",
entry,
"doctor",
"--session-sqlite",
"compact",
"--session-sqlite-store",
storePath,
"--json",
],
{
cwd: process.cwd(),
env,
encoding: "utf8",
timeout: 60_000,
},
);
expect(result.status).not.toBe(0);
expect(`${result.stderr}\n${result.stdout}`).toContain("hard-linked path");
expect(fs.readFileSync(externalWalPath, "utf8")).toBe("external wal\n");
},
{ prefix: "openclaw-session-sqlite-sidecar-cli-" },
);
}, 90_000);
});