mirror of
https://github.com/openclaw/openclaw.git
synced 2026-07-21 02:06:43 +00:00
fix(sqlite): serialize offline maintenance with gateway (#106210)
This commit is contained in:
+17
-3
@@ -203,7 +203,13 @@ the container normally.
|
||||
the canonical shared state database at
|
||||
`<state-dir>/state/openclaw.sqlite`. It does not accept an arbitrary database
|
||||
path, is never invoked by normal Gateway operation, and is not part of
|
||||
`openclaw doctor --fix`.
|
||||
`openclaw doctor --fix`. The command acquires the same state ownership lock as
|
||||
Gateway startup and holds it through validation, checkpointing, `VACUUM`, and
|
||||
the final integrity checks. It refuses to run while a Gateway or another
|
||||
SQLite maintenance command owns that lock. The state lock remains active when
|
||||
`OPENCLAW_ALLOW_MULTI_GATEWAY=1` skips the per-config Gateway singleton, so an
|
||||
operator shell does not need to inherit the Gateway service's environment for
|
||||
maintenance to detect it.
|
||||
|
||||
Stop the Gateway and create a verified backup first:
|
||||
|
||||
@@ -293,8 +299,16 @@ transcripts, run `openclaw doctor --session-sqlite compact --session-sqlite-all-
|
||||
to checkpoint WAL files, run `VACUUM`, and report before/after database and WAL
|
||||
sizes. Compaction requires a regular file with the current agent schema, the
|
||||
selected agent's durable owner metadata, and no open handle in the doctor
|
||||
process. This is explicit offline maintenance: stop the Gateway first so normal
|
||||
writes cannot race the checkpoint or `VACUUM`.
|
||||
process. The destructive `import`, `compact`, `recover`, and `restore` modes
|
||||
hold the same state ownership lock as Gateway startup for their full operation;
|
||||
`inspect`, `dry-run`, and `validate` remain read-only and do not take it. Stop
|
||||
the Gateway first. Destructive modes fail instead of racing live writes or
|
||||
racing another maintenance command. A destructive `--session-sqlite-store`
|
||||
target must be inside the active state directory; set `OPENCLAW_STATE_DIR` to
|
||||
the store's owning state directory before maintaining another installation.
|
||||
Existing hard-linked targets are rejected because another path can share the
|
||||
same database inode outside the locked state directory. The same ownership
|
||||
checks cover SQLite WAL, shared-memory, and rollback-journal sidecars.
|
||||
|
||||
Each import writes a manifest under
|
||||
`~/.openclaw/session-sqlite-migration-runs/` before moving transcript artifacts
|
||||
|
||||
@@ -8,23 +8,26 @@ title: "Gateway lock"
|
||||
|
||||
## Why
|
||||
|
||||
- Only one gateway process should own a given config + port on a host; run additional gateways with isolated profiles and unique ports.
|
||||
- Only one gateway process should own a state directory; run additional gateways with isolated profiles, state directories, configs, and ports.
|
||||
- Survive crashes/SIGKILL without leaving stale lock files behind.
|
||||
- Fail fast with a clear error when another gateway already owns the port.
|
||||
|
||||
## Two layers
|
||||
## Three layers
|
||||
|
||||
Startup enforces single-instance ownership in two independent steps, in order:
|
||||
Startup enforces ownership in three steps, in order:
|
||||
|
||||
1. **File lock** acquires a per-config lock file under the state lock directory. As part of acquiring it, startup probes the configured port for a live listener to detect a stale (crashed) lock owner.
|
||||
2. **Socket bind** binds the HTTP/WebSocket listener (default `ws://127.0.0.1:18789`) as an exclusive TCP listener.
|
||||
1. **State ownership lock** acquires a lock keyed by the canonical state directory. Every Gateway participates, including Gateways started with `OPENCLAW_ALLOW_MULTI_GATEWAY=1`, so destructive SQLite maintenance cannot race a live owner.
|
||||
2. **Config lock** acquires the historical per-config lock and records the runtime port. Multi-Gateway mode skips this config singleton but retains the state ownership lock.
|
||||
3. **Socket bind** binds the HTTP/WebSocket listener (default `ws://127.0.0.1:18789`) as an exclusive TCP listener.
|
||||
|
||||
Each layer can fail independently and throws its own `GatewayLockError`.
|
||||
|
||||
### File lock
|
||||
### State and config locks
|
||||
|
||||
- If the lock file is missing, the recorded owner process is gone, or the owner's port probe shows no live listener, startup reclaims the lock and continues.
|
||||
- If the lock is actively held and none of the above apply, startup retries for up to 5 seconds (default) before giving up:
|
||||
- Lock liveness comes from the recorded PID, platform process start identity when available, and Gateway process identity. A verified owner remains authoritative during startup before its port begins listening.
|
||||
- A dedicated SQLite coordinator serializes metadata inspection, stale-owner reclamation, and lock replacement. Its exclusive transaction is released automatically if the owning process crashes.
|
||||
- If a lock file is missing or the recorded owner process is gone, startup reclaims the lock and continues.
|
||||
- If either lock is actively held, startup retries for up to 5 seconds (default) before giving up:
|
||||
|
||||
```text
|
||||
GatewayLockError("gateway already running (pid <pid>); lock timeout after <ms>ms")
|
||||
@@ -45,11 +48,13 @@ Each layer can fail independently and throws its own `GatewayLockError`.
|
||||
GatewayLockError("failed to bind gateway socket on ws://127.0.0.1:<port>: <cause>")
|
||||
```
|
||||
|
||||
On shutdown, the gateway closes the HTTP/WebSocket server and removes the lock file.
|
||||
On shutdown, the gateway closes the HTTP/WebSocket server and removes its state
|
||||
and config lock files.
|
||||
|
||||
## Operational notes
|
||||
|
||||
- If the port is occupied by a different, non-gateway process, the error is the same; free the port or choose another with `openclaw gateway --port <port>`.
|
||||
- `OPENCLAW_ALLOW_MULTI_GATEWAY=1` permits multiple config/runtime instances, not shared mutable state. Each instance still needs a unique `OPENCLAW_STATE_DIR`.
|
||||
- Under a service supervisor, a new gateway process that hits either error above first probes `/healthz` on the existing process. If that process is healthy, the new process leaves it in control instead of failing. On systemd, it exits with code `78`; the unit's `RestartPreventExitStatus=78` stops `Restart=always` from looping on a lock or `EADDRINUSE` conflict. If the existing process never becomes healthy, the health-probe retry is time-bounded and startup then fails with the lock error above instead of looping forever.
|
||||
- The macOS app keeps its own lightweight PID guard before spawning the gateway; the file lock and socket bind above are the actual runtime enforcement.
|
||||
|
||||
|
||||
@@ -91,7 +91,9 @@ Keep these unique per Gateway instance:
|
||||
| `gateway.port` (or `--port`) | Unique per instance |
|
||||
| Derived browser/CDP ports | See below |
|
||||
|
||||
Sharing any of these causes config races and port conflicts.
|
||||
Sharing any of these causes config, state, or port conflicts. Gateway startup
|
||||
enforces unique state-directory ownership even when
|
||||
`OPENCLAW_ALLOW_MULTI_GATEWAY=1` skips the per-config singleton.
|
||||
|
||||
## Port mapping (derived)
|
||||
|
||||
|
||||
+7
-1
@@ -275,7 +275,13 @@ fly machine update <machine-id> --vm-memory 2048 -y
|
||||
|
||||
Gateway refuses to start with "already running" errors after a container restart.
|
||||
|
||||
The single-instance lock file lives at `<tmpdir>/openclaw-<uid>/gateway.<hash>.lock` (Linux: `/tmp/openclaw-<uid>/gateway.<hash>.lock`), not on the persistent `/data` volume, so a full container restart normally clears it along with the rest of the container filesystem. If the lock survives (for example a `fly machine restart` that preserves the container filesystem) and blocks startup, remove it manually:
|
||||
The runtime lock files live at `<tmpdir>/openclaw-<uid>/gateway.<hash>.lock`
|
||||
and `gateway.state.<hash>.lock` (Linux:
|
||||
`/tmp/openclaw-<uid>/gateway.*.lock`), not on the persistent `/data` volume, so
|
||||
a full container restart normally clears them along with the rest of the
|
||||
container filesystem. If a lock survives (for example a `fly machine restart`
|
||||
that preserves the container filesystem) and blocks startup, remove it
|
||||
manually:
|
||||
|
||||
```bash
|
||||
fly ssh console --command "rm -f /tmp/openclaw-*/gateway.*.lock"
|
||||
|
||||
@@ -52,6 +52,18 @@ export type CliCommandCatalogEntry = {
|
||||
};
|
||||
};
|
||||
|
||||
function hasCliOption(argv: readonly string[], name: string): boolean {
|
||||
for (const arg of argv.slice(2)) {
|
||||
if (arg === "--") {
|
||||
return false;
|
||||
}
|
||||
if (arg === name || arg.startsWith(`${name}=`)) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Command path registry used before Commander registration has loaded all plugins. */
|
||||
export const cliCommandCatalog: readonly CliCommandCatalogEntry[] = [
|
||||
{
|
||||
@@ -285,6 +297,9 @@ export const cliCommandCatalog: readonly CliCommandCatalogEntry[] = [
|
||||
policy: {
|
||||
bypassConfigGuard: true,
|
||||
loadPlugins: "never",
|
||||
// Shared-state maintenance must acquire exclusive ownership before any
|
||||
// config-health observation can open the canonical SQLite database.
|
||||
networkProxy: ({ argv }) => (hasCliOption(argv, "--state-sqlite") ? "bypass" : "default"),
|
||||
},
|
||||
},
|
||||
{ commandPath: ["exec-approvals"], policy: { networkProxy: "bypass" } },
|
||||
|
||||
@@ -206,10 +206,24 @@ describe("command-path-policy", () => {
|
||||
loadPlugins: "never",
|
||||
networkProxy: "bypass",
|
||||
});
|
||||
expectResolvedPolicy(["doctor"], {
|
||||
const doctorPolicy = resolveCliCommandPathPolicy(["doctor"]);
|
||||
expectNetworkProxyResolver(doctorPolicy);
|
||||
expect(doctorPolicy).toMatchObject({
|
||||
bypassConfigGuard: true,
|
||||
loadPlugins: "never",
|
||||
});
|
||||
expect(
|
||||
doctorPolicy.networkProxy({
|
||||
argv: ["node", "openclaw", "doctor"],
|
||||
commandPath: ["doctor"],
|
||||
}),
|
||||
).toBe("default");
|
||||
expect(
|
||||
doctorPolicy.networkProxy({
|
||||
argv: ["node", "openclaw", "doctor", "--state-sqlite=compact"],
|
||||
commandPath: ["doctor"],
|
||||
}),
|
||||
).toBe("bypass");
|
||||
expectResolvedPolicy(["config", "validate"], {
|
||||
bypassConfigGuard: true,
|
||||
loadPlugins: "never",
|
||||
|
||||
@@ -229,6 +229,16 @@ describe("shouldStartProxyForCli", () => {
|
||||
expect(shouldStartProxyForCli(["node", "openclaw", "devices"])).toBe(false);
|
||||
expect(shouldStartProxyForCli(["node", "openclaw", "mcp"])).toBe(false);
|
||||
});
|
||||
|
||||
it("skips managed proxy routing before shared-state SQLite maintenance", () => {
|
||||
expect(
|
||||
shouldStartProxyForCli(["node", "openclaw", "doctor", "--state-sqlite", "compact", "--json"]),
|
||||
).toBe(false);
|
||||
expect(
|
||||
shouldStartProxyForCli(["node", "openclaw", "doctor", "--state-sqlite=compact", "--json"]),
|
||||
).toBe(false);
|
||||
expect(shouldStartProxyForCli(["node", "openclaw", "doctor", "--lint"])).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("shouldUseRootHelpFastPath", () => {
|
||||
|
||||
@@ -8,6 +8,7 @@ import { SessionManager } from "../agents/sessions/session-manager.js";
|
||||
|
||||
const note = vi.hoisted(() => vi.fn());
|
||||
const runDoctorSessionSqlite = vi.hoisted(() => vi.fn());
|
||||
const withDoctorSqliteMaintenanceLock = vi.hoisted(() => vi.fn());
|
||||
|
||||
vi.mock("../../packages/terminal-core/src/note.js", () => ({
|
||||
note,
|
||||
@@ -17,6 +18,10 @@ vi.mock("./doctor-session-sqlite.js", () => ({
|
||||
runDoctorSessionSqlite,
|
||||
}));
|
||||
|
||||
vi.mock("./doctor-sqlite-maintenance-lock.js", () => ({
|
||||
withDoctorSqliteMaintenanceLock,
|
||||
}));
|
||||
|
||||
import {
|
||||
detectSessionTranscriptHealthIssues,
|
||||
noteSessionTranscriptHealth,
|
||||
@@ -49,6 +54,9 @@ describe("doctor session transcript repair", () => {
|
||||
beforeEach(async () => {
|
||||
note.mockClear();
|
||||
runDoctorSessionSqlite.mockReset();
|
||||
withDoctorSqliteMaintenanceLock
|
||||
.mockReset()
|
||||
.mockImplementation(async (params: { run: () => unknown }) => await params.run());
|
||||
root = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-doctor-transcripts-"));
|
||||
});
|
||||
|
||||
@@ -227,6 +235,11 @@ describe("doctor session transcript repair", () => {
|
||||
env,
|
||||
mode: "import",
|
||||
});
|
||||
expect(withDoctorSqliteMaintenanceLock).toHaveBeenCalledWith({
|
||||
env,
|
||||
operation: "session SQLite import",
|
||||
run: expect.any(Function),
|
||||
});
|
||||
expect(note).toHaveBeenCalledWith(
|
||||
expect.stringContaining("Legacy entries: 1"),
|
||||
"Session SQLite",
|
||||
@@ -237,6 +250,38 @@ describe("doctor session transcript repair", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps session SQLite dry-run read-only without taking maintenance ownership", async () => {
|
||||
const sessionsDir = path.join(root, "agents", "main", "sessions");
|
||||
await fs.mkdir(sessionsDir, { recursive: true });
|
||||
runDoctorSessionSqlite.mockResolvedValueOnce({
|
||||
totals: {
|
||||
archivedTranscriptFiles: 0,
|
||||
archivedUnreferencedJsonlFiles: 0,
|
||||
importedTranscriptEvents: 0,
|
||||
issues: 0,
|
||||
legacyEntries: 0,
|
||||
sqliteEntries: 0,
|
||||
unreferencedJsonlFiles: 0,
|
||||
validatedTranscriptEvents: 0,
|
||||
},
|
||||
});
|
||||
const env = { ...process.env, OPENCLAW_STATE_DIR: root };
|
||||
|
||||
await noteSessionTranscriptHealth({
|
||||
env,
|
||||
sessionDirs: [sessionsDir],
|
||||
sessionSqlite: true,
|
||||
shouldRepair: false,
|
||||
});
|
||||
|
||||
expect(runDoctorSessionSqlite).toHaveBeenCalledWith({
|
||||
allAgents: true,
|
||||
env,
|
||||
mode: "dry-run",
|
||||
});
|
||||
expect(withDoctorSqliteMaintenanceLock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("repairs supported current-version linear transcripts", async () => {
|
||||
const filePath = await writeTranscript([
|
||||
{ type: "session", version: 3, id: "session-linear", timestamp: "2026-06-15T00:00:00Z" },
|
||||
|
||||
@@ -18,6 +18,7 @@ import {
|
||||
} from "../config/sessions/transcript-tree.js";
|
||||
import type { HealthFinding, HealthRepairEffect } from "../flows/health-checks.js";
|
||||
import { shortenHomePath } from "../utils.js";
|
||||
import { withDoctorSqliteMaintenanceLock } from "./doctor-sqlite-maintenance-lock.js";
|
||||
|
||||
const SESSION_TRANSCRIPTS_CHECK_ID = "core/doctor/session-transcripts";
|
||||
|
||||
@@ -496,11 +497,19 @@ async function noteSessionSqliteMigrationHealth(params: {
|
||||
// Public doctor owns the operator-facing SQLite import; the targeted
|
||||
// --session-sqlite subcommand remains the diagnostic/proof surface.
|
||||
const { runDoctorSessionSqlite } = await import("./doctor-session-sqlite.js");
|
||||
const report = await runDoctorSessionSqlite({
|
||||
allAgents: true,
|
||||
env: params.env,
|
||||
mode: params.shouldRepair ? "import" : "dry-run",
|
||||
});
|
||||
const runSessionSqlite = async () =>
|
||||
await runDoctorSessionSqlite({
|
||||
allAgents: true,
|
||||
env: params.env,
|
||||
mode: params.shouldRepair ? "import" : "dry-run",
|
||||
});
|
||||
const report = params.shouldRepair
|
||||
? await withDoctorSqliteMaintenanceLock({
|
||||
env: params.env,
|
||||
operation: "session SQLite import",
|
||||
run: runSessionSqlite,
|
||||
})
|
||||
: await runSessionSqlite();
|
||||
if (
|
||||
report.totals.legacyEntries === 0 &&
|
||||
report.totals.unreferencedJsonlFiles === 0 &&
|
||||
|
||||
@@ -0,0 +1,360 @@
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { useAutoCleanupTempDirTracker } from "../../test/helpers/temp-dir.js";
|
||||
import { acquireGatewayLock, GatewayLockError } from "../infra/gateway-lock.js";
|
||||
import {
|
||||
isDestructiveDoctorSessionSqliteMode,
|
||||
withDoctorSqliteMaintenanceLock,
|
||||
} from "./doctor-sqlite-maintenance-lock.js";
|
||||
|
||||
const tempDirs = useAutoCleanupTempDirTracker((cleanup) => {
|
||||
afterEach(cleanup);
|
||||
});
|
||||
|
||||
async function createLockFixture() {
|
||||
const root = tempDirs.make("openclaw-doctor-sqlite-lock-");
|
||||
const stateDir = path.join(root, "state");
|
||||
const configPath = path.join(stateDir, "openclaw.json");
|
||||
const lockDir = path.join(root, "locks");
|
||||
await fs.mkdir(stateDir, { recursive: true });
|
||||
await fs.writeFile(configPath, "{}\n", "utf8");
|
||||
const env = {
|
||||
...process.env,
|
||||
OPENCLAW_CONFIG_PATH: configPath,
|
||||
OPENCLAW_STATE_DIR: stateDir,
|
||||
VITEST: "1",
|
||||
};
|
||||
return {
|
||||
env,
|
||||
lockDir,
|
||||
lockOptions: {
|
||||
lockDir,
|
||||
platform: "darwin" as const,
|
||||
pollIntervalMs: 2,
|
||||
readProcessCmdline: () => ["openclaw-gateway"],
|
||||
timeoutMs: 15,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
describe("doctor SQLite maintenance lock", () => {
|
||||
it.each([
|
||||
["inspect", false],
|
||||
["dry-run", false],
|
||||
["validate", false],
|
||||
["import", true],
|
||||
["compact", true],
|
||||
["restore", true],
|
||||
["recover", true],
|
||||
] as const)("classifies %s mode as destructive=%s", (mode, expected) => {
|
||||
expect(isDestructiveDoctorSessionSqliteMode(mode)).toBe(expected);
|
||||
});
|
||||
|
||||
it("refuses maintenance while a verified Gateway owns the state lock", async () => {
|
||||
const fixture = await createLockFixture();
|
||||
const gatewayLock = await acquireGatewayLock({
|
||||
allowInTests: true,
|
||||
env: fixture.env,
|
||||
lockDir: fixture.lockDir,
|
||||
platform: "darwin",
|
||||
port: 18789,
|
||||
});
|
||||
if (!gatewayLock) {
|
||||
throw new Error("expected Gateway lock");
|
||||
}
|
||||
const run = vi.fn();
|
||||
|
||||
try {
|
||||
await expect(
|
||||
withDoctorSqliteMaintenanceLock(
|
||||
{
|
||||
env: fixture.env,
|
||||
operation: "state SQLite compaction",
|
||||
run,
|
||||
},
|
||||
{ lockOptions: fixture.lockOptions },
|
||||
),
|
||||
).rejects.toThrow(/Gateway or another SQLite maintenance command owns/);
|
||||
expect(run).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
await gatewayLock.release();
|
||||
}
|
||||
});
|
||||
|
||||
it("prevents Gateway startup until maintenance releases ownership", async () => {
|
||||
const fixture = await createLockFixture();
|
||||
let allowMaintenanceToFinish: (() => void) | undefined;
|
||||
const maintenanceMayFinish = new Promise<void>((resolve) => {
|
||||
allowMaintenanceToFinish = resolve;
|
||||
});
|
||||
let markMaintenanceStarted: (() => void) | undefined;
|
||||
const maintenanceStarted = new Promise<void>((resolve) => {
|
||||
markMaintenanceStarted = resolve;
|
||||
});
|
||||
const maintenance = withDoctorSqliteMaintenanceLock(
|
||||
{
|
||||
env: fixture.env,
|
||||
operation: "session SQLite compaction",
|
||||
run: async () => {
|
||||
markMaintenanceStarted?.();
|
||||
await maintenanceMayFinish;
|
||||
return "done";
|
||||
},
|
||||
},
|
||||
{ lockOptions: fixture.lockOptions },
|
||||
);
|
||||
await maintenanceStarted;
|
||||
|
||||
await expect(
|
||||
acquireGatewayLock({
|
||||
allowInTests: true,
|
||||
env: fixture.env,
|
||||
lockDir: fixture.lockDir,
|
||||
platform: "darwin",
|
||||
port: 18789,
|
||||
pollIntervalMs: 2,
|
||||
readProcessCmdline: () => ["openclaw", "doctor", "--session-sqlite", "compact"],
|
||||
timeoutMs: 15,
|
||||
}),
|
||||
).rejects.toBeInstanceOf(GatewayLockError);
|
||||
|
||||
allowMaintenanceToFinish?.();
|
||||
await expect(maintenance).resolves.toBe("done");
|
||||
|
||||
const gatewayLock = await acquireGatewayLock({
|
||||
allowInTests: true,
|
||||
env: fixture.env,
|
||||
lockDir: fixture.lockDir,
|
||||
platform: "darwin",
|
||||
port: 18789,
|
||||
timeoutMs: 15,
|
||||
});
|
||||
if (!gatewayLock) {
|
||||
throw new Error("expected Gateway lock after maintenance release");
|
||||
}
|
||||
await gatewayLock.release();
|
||||
});
|
||||
|
||||
it("releases ownership after maintenance fails", async () => {
|
||||
const fixture = await createLockFixture();
|
||||
|
||||
await expect(
|
||||
withDoctorSqliteMaintenanceLock(
|
||||
{
|
||||
env: fixture.env,
|
||||
operation: "session SQLite restore",
|
||||
run: () => {
|
||||
throw new Error("restore failed");
|
||||
},
|
||||
},
|
||||
{ lockOptions: fixture.lockOptions },
|
||||
),
|
||||
).rejects.toThrow("restore failed");
|
||||
|
||||
const gatewayLock = await acquireGatewayLock({
|
||||
allowInTests: true,
|
||||
env: fixture.env,
|
||||
lockDir: fixture.lockDir,
|
||||
platform: "darwin",
|
||||
port: 18789,
|
||||
timeoutMs: 15,
|
||||
});
|
||||
if (!gatewayLock) {
|
||||
throw new Error("expected Gateway lock after failed maintenance");
|
||||
}
|
||||
await gatewayLock.release();
|
||||
});
|
||||
|
||||
it("blocks maintenance when the Gateway used the multi-Gateway override", async () => {
|
||||
const fixture = await createLockFixture();
|
||||
const run = vi.fn();
|
||||
const gatewayLock = await acquireGatewayLock({
|
||||
allowInTests: true,
|
||||
env: { ...fixture.env, OPENCLAW_ALLOW_MULTI_GATEWAY: "1" },
|
||||
lockDir: fixture.lockDir,
|
||||
platform: "darwin",
|
||||
port: 18789,
|
||||
timeoutMs: 15,
|
||||
});
|
||||
if (!gatewayLock) {
|
||||
throw new Error("expected state ownership lock");
|
||||
}
|
||||
|
||||
try {
|
||||
await expect(
|
||||
withDoctorSqliteMaintenanceLock(
|
||||
{
|
||||
env: fixture.env,
|
||||
operation: "state SQLite compaction",
|
||||
run,
|
||||
},
|
||||
{ lockOptions: fixture.lockOptions },
|
||||
),
|
||||
).rejects.toThrow(/Gateway or another SQLite maintenance command owns/);
|
||||
expect(run).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
await gatewayLock.release();
|
||||
}
|
||||
});
|
||||
|
||||
it("uses the state ownership lock when maintenance inherits the multi-Gateway override", async () => {
|
||||
const fixture = await createLockFixture();
|
||||
|
||||
await expect(
|
||||
withDoctorSqliteMaintenanceLock(
|
||||
{
|
||||
env: { ...fixture.env, OPENCLAW_ALLOW_MULTI_GATEWAY: "1" },
|
||||
operation: "state SQLite compaction",
|
||||
run: () => "done",
|
||||
},
|
||||
{ lockOptions: fixture.lockOptions },
|
||||
),
|
||||
).resolves.toBe("done");
|
||||
});
|
||||
|
||||
it("refuses explicit destructive targets outside the locked state directory", async () => {
|
||||
const fixture = await createLockFixture();
|
||||
const externalPath = path.join(
|
||||
tempDirs.make("openclaw-external-session-store-"),
|
||||
"sessions.json",
|
||||
);
|
||||
const run = vi.fn();
|
||||
|
||||
await expect(
|
||||
withDoctorSqliteMaintenanceLock(
|
||||
{
|
||||
env: fixture.env,
|
||||
operation: "session SQLite compaction",
|
||||
protectedPaths: [externalPath],
|
||||
run,
|
||||
},
|
||||
{ lockOptions: fixture.lockOptions },
|
||||
),
|
||||
).rejects.toThrow(/outside the active OpenClaw state directory/);
|
||||
expect(run).not.toHaveBeenCalled();
|
||||
|
||||
const gatewayLock = await acquireGatewayLock({
|
||||
allowInTests: true,
|
||||
env: fixture.env,
|
||||
lockDir: fixture.lockDir,
|
||||
platform: "darwin",
|
||||
port: 18789,
|
||||
timeoutMs: 15,
|
||||
});
|
||||
if (!gatewayLock) {
|
||||
throw new Error("expected Gateway lock after ownership validation failure");
|
||||
}
|
||||
await gatewayLock.release();
|
||||
});
|
||||
|
||||
it("refuses dangling in-state symlinks that resolve outside ownership", async () => {
|
||||
if (process.platform === "win32") {
|
||||
return;
|
||||
}
|
||||
const fixture = await createLockFixture();
|
||||
const sessionsDir = path.join(fixture.env.OPENCLAW_STATE_DIR, "agents", "main", "sessions");
|
||||
const storePath = path.join(sessionsDir, "sessions.json");
|
||||
const outsideTarget = path.join(
|
||||
tempDirs.make("openclaw-dangling-session-target-"),
|
||||
"missing.json",
|
||||
);
|
||||
await fs.mkdir(sessionsDir, { recursive: true });
|
||||
await fs.symlink(outsideTarget, storePath);
|
||||
const run = vi.fn();
|
||||
|
||||
await expect(
|
||||
withDoctorSqliteMaintenanceLock(
|
||||
{
|
||||
env: fixture.env,
|
||||
operation: "session SQLite import",
|
||||
protectedPaths: [storePath],
|
||||
run,
|
||||
},
|
||||
{ lockOptions: fixture.lockOptions },
|
||||
),
|
||||
).rejects.toThrow(/outside the active OpenClaw state directory/);
|
||||
expect(run).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("refuses outside path aliases that resolve into the locked state directory", async () => {
|
||||
if (process.platform === "win32") {
|
||||
return;
|
||||
}
|
||||
const fixture = await createLockFixture();
|
||||
const externalAlias = path.join(
|
||||
path.dirname(fixture.env.OPENCLAW_STATE_DIR),
|
||||
"external-state-alias",
|
||||
);
|
||||
await fs.symlink(fixture.env.OPENCLAW_STATE_DIR, externalAlias, "dir");
|
||||
const storePath = path.join(externalAlias, "agents", "main", "sessions", "sessions.json");
|
||||
const run = vi.fn();
|
||||
|
||||
await expect(
|
||||
withDoctorSqliteMaintenanceLock(
|
||||
{
|
||||
env: fixture.env,
|
||||
operation: "session SQLite import",
|
||||
protectedPaths: [storePath],
|
||||
run,
|
||||
},
|
||||
{ lockOptions: fixture.lockOptions },
|
||||
),
|
||||
).rejects.toThrow(/outside the active OpenClaw state directory/);
|
||||
expect(run).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("refuses in-state hard links that can alias storage outside ownership", async () => {
|
||||
const fixture = await createLockFixture();
|
||||
const sessionsDir = path.join(fixture.env.OPENCLAW_STATE_DIR, "agents", "main", "sessions");
|
||||
const storePath = path.join(sessionsDir, "sessions.json");
|
||||
const externalDir = path.join(path.dirname(fixture.env.OPENCLAW_STATE_DIR), "external-state");
|
||||
const externalPath = path.join(externalDir, "sessions.json");
|
||||
await fs.mkdir(sessionsDir, { recursive: true });
|
||||
await fs.mkdir(externalDir, { recursive: true });
|
||||
await fs.writeFile(externalPath, "{}\n", "utf8");
|
||||
await fs.link(externalPath, storePath);
|
||||
const run = vi.fn();
|
||||
|
||||
await expect(
|
||||
withDoctorSqliteMaintenanceLock(
|
||||
{
|
||||
env: fixture.env,
|
||||
operation: "session SQLite compaction",
|
||||
protectedPaths: [storePath],
|
||||
run,
|
||||
},
|
||||
{ lockOptions: fixture.lockOptions },
|
||||
),
|
||||
).rejects.toThrow(/hard-linked path/);
|
||||
expect(run).not.toHaveBeenCalled();
|
||||
await expect(fs.readFile(externalPath, "utf8")).resolves.toBe("{}\n");
|
||||
});
|
||||
|
||||
it("allows explicit destructive targets owned by the locked state directory", async () => {
|
||||
const fixture = await createLockFixture();
|
||||
const storePath = path.join(
|
||||
fixture.env.OPENCLAW_STATE_DIR,
|
||||
"agents",
|
||||
"main",
|
||||
"sessions",
|
||||
"sessions.json",
|
||||
);
|
||||
|
||||
await expect(
|
||||
withDoctorSqliteMaintenanceLock(
|
||||
{
|
||||
env: fixture.env,
|
||||
operation: "session SQLite compaction",
|
||||
protectedPaths: [storePath],
|
||||
run: () => "done",
|
||||
},
|
||||
{ lockOptions: fixture.lockOptions },
|
||||
),
|
||||
).resolves.toBe("done");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,122 @@
|
||||
/** Serializes offline SQLite maintenance against the Gateway state owner. */
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { resolveStateDir } from "../config/paths.js";
|
||||
import { resolvePathViaExistingAncestorSync, resolveRootPathSync } from "../infra/boundary-path.js";
|
||||
import {
|
||||
acquireGatewayLock,
|
||||
GatewayLockError,
|
||||
type GatewayLockOptions,
|
||||
} from "../infra/gateway-lock.js";
|
||||
import { isPathInside } from "../infra/path-guards.js";
|
||||
import type { DoctorSessionSqliteMode } from "./doctor-session-sqlite-types.js";
|
||||
|
||||
const MAINTENANCE_LOCK_TIMEOUT_MS = 250;
|
||||
const MAINTENANCE_LOCK_POLL_INTERVAL_MS = 25;
|
||||
|
||||
type MaintenanceLockOptions = Pick<
|
||||
GatewayLockOptions,
|
||||
| "lockDir"
|
||||
| "now"
|
||||
| "platform"
|
||||
| "pollIntervalMs"
|
||||
| "readProcessCmdline"
|
||||
| "readProcessStartTime"
|
||||
| "sleep"
|
||||
| "staleMs"
|
||||
| "timeoutMs"
|
||||
>;
|
||||
|
||||
type DoctorSqliteMaintenanceLockDeps = {
|
||||
acquireLock?: typeof acquireGatewayLock;
|
||||
lockOptions?: MaintenanceLockOptions;
|
||||
};
|
||||
|
||||
function assertMaintenancePathsOwnedByStateDir(
|
||||
env: NodeJS.ProcessEnv,
|
||||
operation: string,
|
||||
protectedPaths: readonly string[],
|
||||
): void {
|
||||
if (protectedPaths.length === 0) {
|
||||
return;
|
||||
}
|
||||
const stateDir = path.resolve(resolveStateDir(env));
|
||||
const stateCanonicalDir = resolvePathViaExistingAncestorSync(stateDir);
|
||||
for (const protectedPath of protectedPaths) {
|
||||
const absolutePath = path.resolve(protectedPath);
|
||||
let resolvedPath: ReturnType<typeof resolveRootPathSync>;
|
||||
try {
|
||||
if (!isPathInside(stateDir, absolutePath) && !isPathInside(stateCanonicalDir, absolutePath)) {
|
||||
throw new Error("path is not lexically owned by the active state directory");
|
||||
}
|
||||
resolvedPath = resolveRootPathSync({
|
||||
absolutePath,
|
||||
boundaryLabel: "OpenClaw state directory",
|
||||
rootCanonicalPath: stateCanonicalDir,
|
||||
rootPath: stateDir,
|
||||
});
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`Cannot run ${operation} for a path outside the active OpenClaw state directory: ${protectedPath}. Set OPENCLAW_STATE_DIR to the owning state directory and retry.`,
|
||||
{ cause: error },
|
||||
);
|
||||
}
|
||||
if (
|
||||
resolvedPath.exists &&
|
||||
resolvedPath.kind === "file" &&
|
||||
fs.statSync(resolvedPath.canonicalPath).nlink > 1
|
||||
) {
|
||||
throw new Error(
|
||||
`Cannot run ${operation} for a hard-linked path: ${protectedPath}. Remove the additional hard link and retry.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function isDestructiveDoctorSessionSqliteMode(mode: DoctorSessionSqliteMode): boolean {
|
||||
return mode === "import" || mode === "compact" || mode === "restore" || mode === "recover";
|
||||
}
|
||||
|
||||
/** Run one destructive doctor operation while excluding Gateway startup and peer maintenance. */
|
||||
export async function withDoctorSqliteMaintenanceLock<T>(
|
||||
params: {
|
||||
env?: NodeJS.ProcessEnv;
|
||||
operation: string;
|
||||
protectedPaths?: readonly string[];
|
||||
run: () => Promise<T> | T;
|
||||
},
|
||||
deps: DoctorSqliteMaintenanceLockDeps = {},
|
||||
): Promise<T> {
|
||||
const env = params.env ?? process.env;
|
||||
const acquireLock = deps.acquireLock ?? acquireGatewayLock;
|
||||
const lockOptions = deps.lockOptions;
|
||||
let lock: Awaited<ReturnType<typeof acquireGatewayLock>>;
|
||||
try {
|
||||
lock = await acquireLock({
|
||||
...lockOptions,
|
||||
allowInTests: true,
|
||||
env,
|
||||
pollIntervalMs: lockOptions?.pollIntervalMs ?? MAINTENANCE_LOCK_POLL_INTERVAL_MS,
|
||||
role: "sqlite-maintenance",
|
||||
timeoutMs: lockOptions?.timeoutMs ?? MAINTENANCE_LOCK_TIMEOUT_MS,
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof GatewayLockError) {
|
||||
throw new Error(
|
||||
`Cannot run ${params.operation} while the Gateway or another SQLite maintenance command owns this OpenClaw state directory. Stop the Gateway and retry.`,
|
||||
{ cause: error },
|
||||
);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
if (!lock) {
|
||||
throw new Error(`Cannot run ${params.operation} without exclusive OpenClaw state ownership.`);
|
||||
}
|
||||
|
||||
try {
|
||||
assertMaintenancePathsOwnedByStateDir(env, params.operation, params.protectedPaths ?? []);
|
||||
return await params.run();
|
||||
} finally {
|
||||
await lock.release();
|
||||
}
|
||||
}
|
||||
@@ -138,10 +138,10 @@ function expectOwnerOnlySqlitePermissions(sqlitePath: string): void {
|
||||
}
|
||||
|
||||
describe("runDoctorStateSqliteCompact", () => {
|
||||
it("reports a missing canonical database as skipped", () => {
|
||||
it("reports a missing canonical database as skipped", async () => {
|
||||
const env = createStateEnv();
|
||||
|
||||
expect(runDoctorStateSqliteCompact({ env })).toEqual({
|
||||
await expect(runDoctorStateSqliteCompact({ env })).resolves.toEqual({
|
||||
mode: "compact",
|
||||
path: resolveOpenClawStateSqlitePath(env),
|
||||
reason: "missing",
|
||||
@@ -149,11 +149,11 @@ describe("runDoctorStateSqliteCompact", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("compacts the canonical database and reports verified before/after state", () => {
|
||||
it("compacts the canonical database and reports verified before/after state", async () => {
|
||||
const env = createStateEnv();
|
||||
const sqlitePath = seedStateDatabase({ env, withBloat: true });
|
||||
|
||||
const report = runDoctorStateSqliteCompact({ env });
|
||||
const report = await runDoctorStateSqliteCompact({ env });
|
||||
|
||||
expectCompletedReport(report);
|
||||
expect(report.path).toBe(sqlitePath);
|
||||
@@ -169,20 +169,20 @@ describe("runDoctorStateSqliteCompact", () => {
|
||||
expect(report.integrityCheck).toBe("ok");
|
||||
});
|
||||
|
||||
it.skipIf(process.platform === "win32")("reapplies owner-only SQLite permissions", () => {
|
||||
it.skipIf(process.platform === "win32")("reapplies owner-only SQLite permissions", async () => {
|
||||
const env = createStateEnv();
|
||||
const sqlitePath = seedStateDatabase({ env, withBloat: true });
|
||||
|
||||
runDoctorStateSqliteCompact({ env });
|
||||
await runDoctorStateSqliteCompact({ env });
|
||||
|
||||
expectOwnerOnlySqlitePermissions(sqlitePath);
|
||||
});
|
||||
|
||||
it("rejects non-global schema metadata before mutation", () => {
|
||||
it("rejects non-global schema metadata before mutation", async () => {
|
||||
const env = createStateEnv();
|
||||
const sqlitePath = seedStateDatabase({ env, role: "agent", withBloat: true });
|
||||
|
||||
expect(() => runDoctorStateSqliteCompact({ env })).toThrow(/schema role agent.*global/);
|
||||
await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(/schema role agent.*global/);
|
||||
|
||||
const sqlite = requireNodeSqlite();
|
||||
const database = new sqlite.DatabaseSync(sqlitePath, { readOnly: true });
|
||||
@@ -199,11 +199,11 @@ describe("runDoctorStateSqliteCompact", () => {
|
||||
["future", OPENCLAW_STATE_SCHEMA_VERSION + 1, /uses newer schema version/],
|
||||
] as const)(
|
||||
"rejects a %s shared-state schema before mutation",
|
||||
(_label, schemaVersion, message) => {
|
||||
async (_label, schemaVersion, message) => {
|
||||
const env = createStateEnv();
|
||||
const sqlitePath = seedStateDatabase({ env, schemaVersion });
|
||||
|
||||
expect(() => runDoctorStateSqliteCompact({ env })).toThrow(message);
|
||||
await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(message);
|
||||
|
||||
const sqlite = requireNodeSqlite();
|
||||
const database = new sqlite.DatabaseSync(sqlitePath, { readOnly: true });
|
||||
@@ -217,7 +217,7 @@ describe("runDoctorStateSqliteCompact", () => {
|
||||
|
||||
it.skipIf(process.platform === "win32")(
|
||||
"refuses a symlink at the canonical database path",
|
||||
() => {
|
||||
async () => {
|
||||
const env = createStateEnv();
|
||||
const canonicalPath = resolveOpenClawStateSqlitePath(env);
|
||||
const externalEnv = createStateEnv();
|
||||
@@ -225,18 +225,45 @@ describe("runDoctorStateSqliteCompact", () => {
|
||||
fs.mkdirSync(path.dirname(canonicalPath), { recursive: true });
|
||||
fs.symlinkSync(externalPath, canonicalPath);
|
||||
|
||||
expect(() => runDoctorStateSqliteCompact({ env })).toThrow(/not a regular file/);
|
||||
await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(/not a regular file/);
|
||||
},
|
||||
);
|
||||
|
||||
it("refuses compaction while this process owns an open shared-state handle", () => {
|
||||
it("refuses compaction while this process owns an open shared-state handle", async () => {
|
||||
const env = createStateEnv();
|
||||
openOpenClawStateDatabase({ env });
|
||||
|
||||
expect(() => runDoctorStateSqliteCompact({ env })).toThrow(/already open in this process/);
|
||||
await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(
|
||||
/already open in this process/,
|
||||
);
|
||||
});
|
||||
|
||||
it("treats a busy truncating checkpoint as failure", () => {
|
||||
it("leaves the database untouched when exclusive state ownership is unavailable", async () => {
|
||||
const env = createStateEnv();
|
||||
const sqlitePath = seedStateDatabase({ env, withBloat: true });
|
||||
|
||||
await expect(
|
||||
runDoctorStateSqliteCompact(
|
||||
{ env },
|
||||
{
|
||||
withMaintenanceLock: async () => {
|
||||
throw new Error("Gateway owns this OpenClaw state directory");
|
||||
},
|
||||
},
|
||||
),
|
||||
).rejects.toThrow(/Gateway owns/);
|
||||
|
||||
const sqlite = requireNodeSqlite();
|
||||
const database = new sqlite.DatabaseSync(sqlitePath, { readOnly: true });
|
||||
try {
|
||||
expect(readPragma(database, "auto_vacuum")).toBe(0);
|
||||
expect(readPragma(database, "freelist_count")).toBeGreaterThan(0);
|
||||
} finally {
|
||||
database.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("treats a busy truncating checkpoint as failure", async () => {
|
||||
const env = createStateEnv();
|
||||
const sqlitePath = seedStateDatabase({ env });
|
||||
const sqlite = requireNodeSqlite();
|
||||
@@ -247,7 +274,7 @@ describe("runDoctorStateSqliteCompact", () => {
|
||||
writer.exec("INSERT INTO compact_payload (payload) VALUES ('newer wal frame');");
|
||||
|
||||
// Exercise the real busy checkpoint result without waiting the production lock timeout.
|
||||
expect(() => runDoctorStateSqliteCompact({ env }, { busyTimeoutMs: 0 })).toThrow(
|
||||
await expect(runDoctorStateSqliteCompact({ env }, { busyTimeoutMs: 0 })).rejects.toThrow(
|
||||
/checkpoint remained busy/,
|
||||
);
|
||||
expect(readPragma(writer, "auto_vacuum")).toBe(0);
|
||||
@@ -258,7 +285,7 @@ describe("runDoctorStateSqliteCompact", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects stale secondary indexes before mutating the database", () => {
|
||||
it("rejects stale secondary indexes before mutating the database", async () => {
|
||||
const env = createStateEnv();
|
||||
const sqlitePath = seedStateDatabase({ env, withBloat: true });
|
||||
createUnsafeIndexDrift(sqlitePath);
|
||||
@@ -277,7 +304,7 @@ describe("runDoctorStateSqliteCompact", () => {
|
||||
before.close();
|
||||
}
|
||||
|
||||
expect(() => runDoctorStateSqliteCompact({ env })).toThrow(
|
||||
await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(
|
||||
/integrity_check failed.*missing from index unsafe_index_records_value/iu,
|
||||
);
|
||||
|
||||
@@ -297,7 +324,7 @@ describe("runDoctorStateSqliteCompact", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects foreign-key violations before mutating the database", () => {
|
||||
it("rejects foreign-key violations before mutating the database", async () => {
|
||||
const env = createStateEnv();
|
||||
const sqlitePath = seedStateDatabase({ env, withBloat: true });
|
||||
const sqlite = requireNodeSqlite();
|
||||
@@ -326,7 +353,7 @@ describe("runDoctorStateSqliteCompact", () => {
|
||||
corrupted.close();
|
||||
}
|
||||
|
||||
expect(() => runDoctorStateSqliteCompact({ env })).toThrow(
|
||||
await expect(runDoctorStateSqliteCompact({ env })).rejects.toThrow(
|
||||
/foreign_key_check failed.*compact_children row 1 references compact_parents \(foreign key 0\)/iu,
|
||||
);
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@ import {
|
||||
compactDoctorSqliteFile,
|
||||
type DoctorSqliteCompactSnapshot,
|
||||
} from "./doctor-sqlite-compact.js";
|
||||
import { withDoctorSqliteMaintenanceLock } from "./doctor-sqlite-maintenance-lock.js";
|
||||
|
||||
export type DoctorStateSqliteCompactReport =
|
||||
| {
|
||||
@@ -35,13 +36,14 @@ type DoctorStateSqliteCompactOptions = {
|
||||
|
||||
type DoctorStateSqliteCompactDeps = {
|
||||
busyTimeoutMs?: number;
|
||||
withMaintenanceLock?: typeof withDoctorSqliteMaintenanceLock;
|
||||
};
|
||||
|
||||
/** Compact only the canonical shared state database resolved for this invocation. */
|
||||
export function runDoctorStateSqliteCompact(
|
||||
export async function runDoctorStateSqliteCompact(
|
||||
options: DoctorStateSqliteCompactOptions = {},
|
||||
deps: DoctorStateSqliteCompactDeps = {},
|
||||
): DoctorStateSqliteCompactReport {
|
||||
): Promise<DoctorStateSqliteCompactReport> {
|
||||
const env = options.env ?? process.env;
|
||||
const sqlitePath = resolveOpenClawStateSqlitePath(env);
|
||||
const stat = readCanonicalStateDatabaseStat(sqlitePath);
|
||||
@@ -56,25 +58,32 @@ export function runDoctorStateSqliteCompact(
|
||||
if (!stat.isFile()) {
|
||||
throw new Error(`Canonical OpenClaw state database is not a regular file: ${sqlitePath}`);
|
||||
}
|
||||
if (isOpenClawStateDatabaseOpen()) {
|
||||
throw new Error(
|
||||
"The shared OpenClaw state database is already open in this process. Stop OpenClaw and retry.",
|
||||
);
|
||||
}
|
||||
const withMaintenanceLock = deps.withMaintenanceLock ?? withDoctorSqliteMaintenanceLock;
|
||||
return await withMaintenanceLock({
|
||||
env,
|
||||
operation: "state SQLite compaction",
|
||||
run: () => {
|
||||
if (isOpenClawStateDatabaseOpen()) {
|
||||
throw new Error(
|
||||
"The shared OpenClaw state database is already open in this process. Stop OpenClaw and retry.",
|
||||
);
|
||||
}
|
||||
|
||||
const compact = compactDoctorSqliteFile({
|
||||
afterMutation: () => ensureOpenClawStatePermissions(sqlitePath, env),
|
||||
...(deps.busyTimeoutMs !== undefined ? { busyTimeoutMs: deps.busyTimeoutMs } : {}),
|
||||
sqlitePath,
|
||||
validateBeforeMutation: (database) =>
|
||||
assertOpenClawStateDatabaseForMaintenance(database, { pathname: sqlitePath }),
|
||||
const compact = compactDoctorSqliteFile({
|
||||
afterMutation: () => ensureOpenClawStatePermissions(sqlitePath, env),
|
||||
...(deps.busyTimeoutMs !== undefined ? { busyTimeoutMs: deps.busyTimeoutMs } : {}),
|
||||
sqlitePath,
|
||||
validateBeforeMutation: (database) =>
|
||||
assertOpenClawStateDatabaseForMaintenance(database, { pathname: sqlitePath }),
|
||||
});
|
||||
return {
|
||||
...compact,
|
||||
mode: "compact",
|
||||
path: sqlitePath,
|
||||
skipped: false,
|
||||
};
|
||||
},
|
||||
});
|
||||
return {
|
||||
...compact,
|
||||
mode: "compact",
|
||||
path: sqlitePath,
|
||||
skipped: false,
|
||||
};
|
||||
}
|
||||
|
||||
function readCanonicalStateDatabaseStat(sqlitePath: string): fs.Stats | undefined {
|
||||
|
||||
+126
-1
@@ -1,4 +1,5 @@
|
||||
// Doctor command tests cover probe orchestration, fix mode, and runtime command output.
|
||||
import path from "node:path";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const mocks = vi.hoisted(() => ({
|
||||
@@ -6,6 +7,7 @@ const mocks = vi.hoisted(() => ({
|
||||
runPostUpgradeProbes: vi.fn(),
|
||||
runDoctorStateSqliteCompact: vi.fn(),
|
||||
runDoctorSessionSqlite: vi.fn(),
|
||||
withDoctorSqliteMaintenanceLock: vi.fn(),
|
||||
resolveInstalledPluginIndexStorePath: vi.fn(() => "/tmp/openclaw-installed-plugins.json"),
|
||||
}));
|
||||
|
||||
@@ -21,6 +23,12 @@ vi.mock("./doctor-state-sqlite-compact.js", () => ({
|
||||
runDoctorStateSqliteCompact: mocks.runDoctorStateSqliteCompact,
|
||||
}));
|
||||
|
||||
vi.mock("./doctor-sqlite-maintenance-lock.js", () => ({
|
||||
isDestructiveDoctorSessionSqliteMode: (mode: string) =>
|
||||
mode === "import" || mode === "compact" || mode === "restore" || mode === "recover",
|
||||
withDoctorSqliteMaintenanceLock: mocks.withDoctorSqliteMaintenanceLock,
|
||||
}));
|
||||
|
||||
vi.mock("./doctor-session-sqlite-github-issue.js", () => ({
|
||||
createSessionSqliteGithubIssue: mocks.createSessionSqliteGithubIssue,
|
||||
}));
|
||||
@@ -34,6 +42,9 @@ const { doctorCommand } = await import("./doctor.js");
|
||||
describe("doctorCommand", () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
mocks.withDoctorSqliteMaintenanceLock.mockImplementation(
|
||||
async (params: { run: () => unknown }) => await params.run(),
|
||||
);
|
||||
});
|
||||
|
||||
it("writes post-upgrade JSON through the runtime before exiting with findings", async () => {
|
||||
@@ -108,10 +119,124 @@ describe("doctorCommand", () => {
|
||||
agent: "main",
|
||||
mode: "inspect",
|
||||
});
|
||||
expect(mocks.withDoctorSqliteMaintenanceLock).not.toHaveBeenCalled();
|
||||
expect(runtime.writeJson).toHaveBeenCalledWith(report, 2);
|
||||
expect(runtime.exit).toHaveBeenCalledWith(0);
|
||||
});
|
||||
|
||||
it("holds exclusive state ownership for destructive session sqlite modes", async () => {
|
||||
const report = {
|
||||
mode: "restore",
|
||||
targets: [],
|
||||
totals: {
|
||||
archivedTranscriptFiles: 0,
|
||||
archivedUnreferencedJsonlFiles: 0,
|
||||
importedEntries: 0,
|
||||
importedTranscriptEvents: 0,
|
||||
issues: 0,
|
||||
legacyEntries: 0,
|
||||
sqliteEntries: 0,
|
||||
targets: 0,
|
||||
unreferencedJsonlFiles: 0,
|
||||
validatedEntries: 0,
|
||||
validatedTranscriptEvents: 0,
|
||||
},
|
||||
};
|
||||
mocks.runDoctorSessionSqlite.mockResolvedValueOnce(report);
|
||||
const runtime = {
|
||||
log: vi.fn(),
|
||||
error: vi.fn(),
|
||||
writeStdout: vi.fn(),
|
||||
writeJson: vi.fn(),
|
||||
exit: vi.fn((code: number) => {
|
||||
throw new Error(`exit:${code}`);
|
||||
}),
|
||||
};
|
||||
|
||||
await expect(
|
||||
doctorCommand(runtime, {
|
||||
sessionSqlite: "restore",
|
||||
sessionSqliteAllAgents: true,
|
||||
}),
|
||||
).rejects.toThrow("exit:0");
|
||||
|
||||
expect(mocks.withDoctorSqliteMaintenanceLock).toHaveBeenCalledWith({
|
||||
env: process.env,
|
||||
operation: "session SQLite restore",
|
||||
run: expect.any(Function),
|
||||
});
|
||||
expect(mocks.runDoctorSessionSqlite).toHaveBeenCalledWith({
|
||||
allAgents: true,
|
||||
mode: "restore",
|
||||
});
|
||||
});
|
||||
|
||||
it("binds explicit destructive session stores to the maintenance lock", async () => {
|
||||
const report = {
|
||||
mode: "compact",
|
||||
targets: [],
|
||||
totals: {
|
||||
archivedTranscriptFiles: 0,
|
||||
archivedUnreferencedJsonlFiles: 0,
|
||||
importedEntries: 0,
|
||||
importedTranscriptEvents: 0,
|
||||
issues: 0,
|
||||
legacyEntries: 0,
|
||||
sqliteEntries: 0,
|
||||
targets: 0,
|
||||
unreferencedJsonlFiles: 0,
|
||||
validatedEntries: 0,
|
||||
validatedTranscriptEvents: 0,
|
||||
},
|
||||
};
|
||||
mocks.runDoctorSessionSqlite.mockResolvedValueOnce(report);
|
||||
const runtime = {
|
||||
log: vi.fn(),
|
||||
error: vi.fn(),
|
||||
writeStdout: vi.fn(),
|
||||
writeJson: vi.fn(),
|
||||
exit: vi.fn((code: number) => {
|
||||
throw new Error(`exit:${code}`);
|
||||
}),
|
||||
};
|
||||
const stateDir = path.resolve(process.env.OPENCLAW_STATE_DIR ?? ".openclaw");
|
||||
const storePath = path.join(stateDir, "agents", "main", "sessions", "sessions.json");
|
||||
const sqlitePath = path.join(stateDir, "agents", "main", "agent", "openclaw-agent.sqlite");
|
||||
|
||||
await expect(
|
||||
doctorCommand(runtime, {
|
||||
sessionSqlite: "compact",
|
||||
sessionSqliteStore: storePath,
|
||||
}),
|
||||
).rejects.toThrow("exit:0");
|
||||
|
||||
expect(mocks.withDoctorSqliteMaintenanceLock).toHaveBeenCalledWith({
|
||||
env: process.env,
|
||||
operation: "session SQLite compact",
|
||||
protectedPaths: [
|
||||
storePath,
|
||||
sqlitePath,
|
||||
`${sqlitePath}-wal`,
|
||||
`${sqlitePath}-shm`,
|
||||
`${sqlitePath}-journal`,
|
||||
],
|
||||
run: expect.any(Function),
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects conflicting explicit-store selectors before taking maintenance ownership", async () => {
|
||||
await expect(
|
||||
doctorCommand(undefined, {
|
||||
sessionSqlite: "compact",
|
||||
sessionSqliteAgent: "ops",
|
||||
sessionSqliteStore: path.resolve("stores", "{agentId}", "sessions.json"),
|
||||
}),
|
||||
).rejects.toThrow("--store cannot be combined with --agent or --all-agents");
|
||||
|
||||
expect(mocks.withDoctorSqliteMaintenanceLock).not.toHaveBeenCalled();
|
||||
expect(mocks.runDoctorSessionSqlite).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("writes shared-state sqlite compaction JSON through the runtime", async () => {
|
||||
const report = {
|
||||
after: {
|
||||
@@ -135,7 +260,7 @@ describe("doctorCommand", () => {
|
||||
reclaimedBytes: 12_288,
|
||||
skipped: false,
|
||||
};
|
||||
mocks.runDoctorStateSqliteCompact.mockReturnValueOnce(report);
|
||||
mocks.runDoctorStateSqliteCompact.mockResolvedValueOnce(report);
|
||||
const runtime = {
|
||||
log: vi.fn(),
|
||||
error: vi.fn(),
|
||||
|
||||
+57
-8
@@ -1,15 +1,52 @@
|
||||
/** Top-level doctor command wrapper, including post-upgrade probe mode. */
|
||||
import { resolveSqliteTargetFromSessionStorePath } from "../config/sessions/session-sqlite-target.js";
|
||||
import { resolveSessionStoreTargets } from "../config/sessions/targets.js";
|
||||
import { resolveSqliteDatabaseFilePaths } from "../infra/sqlite-files.js";
|
||||
import { defaultRuntime, type RuntimeEnv, writeRuntimeJson } from "../runtime.js";
|
||||
import { runPostUpgradeProbes } from "./doctor-post-upgrade.js";
|
||||
import type { DoctorOptions } from "./doctor-prompter.js";
|
||||
import type { DoctorSessionSqliteReport } from "./doctor-session-sqlite.js";
|
||||
import {
|
||||
isDestructiveDoctorSessionSqliteMode,
|
||||
withDoctorSqliteMaintenanceLock,
|
||||
} from "./doctor-sqlite-maintenance-lock.js";
|
||||
|
||||
function resolveExplicitSessionSqliteMaintenancePaths(options: DoctorOptions): string[] {
|
||||
if (!options.sessionSqliteStore) {
|
||||
return [];
|
||||
}
|
||||
// Explicit path mode intentionally bypasses runtime config. Resolve through
|
||||
// the same selector as the migration so ownership checks cover exact targets.
|
||||
const targets = resolveSessionStoreTargets(
|
||||
{},
|
||||
{
|
||||
store: options.sessionSqliteStore,
|
||||
...(options.sessionSqliteAgent ? { agent: options.sessionSqliteAgent } : {}),
|
||||
...(options.sessionSqliteAllAgents ? { allAgents: true } : {}),
|
||||
},
|
||||
{ env: process.env },
|
||||
);
|
||||
const protectedPaths = new Set<string>();
|
||||
for (const target of targets) {
|
||||
protectedPaths.add(target.storePath);
|
||||
const sqlitePath = resolveSqliteTargetFromSessionStorePath(target.storePath, {
|
||||
agentId: target.agentId,
|
||||
}).path;
|
||||
if (sqlitePath) {
|
||||
for (const databasePath of resolveSqliteDatabaseFilePaths(sqlitePath)) {
|
||||
protectedPaths.add(databasePath);
|
||||
}
|
||||
}
|
||||
}
|
||||
return [...protectedPaths];
|
||||
}
|
||||
|
||||
/** Runs doctor or the post-upgrade probe submode using the provided runtime. */
|
||||
export async function doctorCommand(runtime?: RuntimeEnv, options?: DoctorOptions): Promise<void> {
|
||||
if (options?.stateSqlite) {
|
||||
const outputRuntime = runtime ?? defaultRuntime;
|
||||
const { runDoctorStateSqliteCompact } = await import("./doctor-state-sqlite-compact.js");
|
||||
const report = runDoctorStateSqliteCompact();
|
||||
const report = await runDoctorStateSqliteCompact();
|
||||
if (options.json) {
|
||||
writeRuntimeJson(outputRuntime, report);
|
||||
} else if (report.skipped) {
|
||||
@@ -30,14 +67,26 @@ export async function doctorCommand(runtime?: RuntimeEnv, options?: DoctorOption
|
||||
}
|
||||
if (options?.sessionSqlite) {
|
||||
const outputRuntime = runtime ?? defaultRuntime;
|
||||
const sessionSqliteMode = options.sessionSqlite;
|
||||
const { runDoctorSessionSqlite } = await import("./doctor-session-sqlite.js");
|
||||
const report = await runDoctorSessionSqlite({
|
||||
mode: options.sessionSqlite,
|
||||
...(options.sessionSqliteStore ? { store: options.sessionSqliteStore } : {}),
|
||||
...(options.sessionSqliteAgent ? { agent: options.sessionSqliteAgent } : {}),
|
||||
...(options.sessionSqliteAllAgents ? { allAgents: true } : {}),
|
||||
});
|
||||
if (options.sessionSqlite === "recover" && options.sessionSqliteGithubIssue === true) {
|
||||
const runSessionSqlite = async () =>
|
||||
await runDoctorSessionSqlite({
|
||||
mode: sessionSqliteMode,
|
||||
...(options.sessionSqliteStore ? { store: options.sessionSqliteStore } : {}),
|
||||
...(options.sessionSqliteAgent ? { agent: options.sessionSqliteAgent } : {}),
|
||||
...(options.sessionSqliteAllAgents ? { allAgents: true } : {}),
|
||||
});
|
||||
const report = isDestructiveDoctorSessionSqliteMode(sessionSqliteMode)
|
||||
? await withDoctorSqliteMaintenanceLock({
|
||||
env: process.env,
|
||||
operation: `session SQLite ${sessionSqliteMode}`,
|
||||
...(options.sessionSqliteStore
|
||||
? { protectedPaths: resolveExplicitSessionSqliteMaintenancePaths(options) }
|
||||
: {}),
|
||||
run: runSessionSqlite,
|
||||
})
|
||||
: await runSessionSqlite();
|
||||
if (sessionSqliteMode === "recover" && options.sessionSqliteGithubIssue === true) {
|
||||
await maybeCreateSessionSqliteGithubIssue(outputRuntime, report, options);
|
||||
}
|
||||
if (options.json) {
|
||||
|
||||
@@ -673,7 +673,10 @@ async function runSessionLocksHealth(ctx: DoctorHealthFlowContext): Promise<void
|
||||
|
||||
async function runSessionTranscriptsHealth(ctx: DoctorHealthFlowContext): Promise<void> {
|
||||
const { noteSessionTranscriptHealth } = await import("../commands/doctor-session-transcripts.js");
|
||||
await noteSessionTranscriptHealth({ shouldRepair: ctx.prompter.shouldRepair });
|
||||
await noteSessionTranscriptHealth({
|
||||
env: ctx.env ?? process.env,
|
||||
shouldRepair: ctx.prompter.shouldRepair,
|
||||
});
|
||||
}
|
||||
|
||||
async function runSessionSnapshotsHealth(ctx: DoctorHealthFlowContext): Promise<void> {
|
||||
|
||||
+462
-50
@@ -66,37 +66,15 @@ function expectGatewayLock(lock: Awaited<ReturnType<typeof acquireGatewayLock>>)
|
||||
function resolveLockPath(env: NodeJS.ProcessEnv) {
|
||||
const stateDir = resolveStateDir(env);
|
||||
const configPath = resolveConfigPath(env, stateDir);
|
||||
const hash = createHash("sha256").update(configPath).digest("hex").slice(0, 8);
|
||||
const configHash = createHash("sha256").update(configPath).digest("hex").slice(0, 8);
|
||||
const canonicalStateDir = fsSync.realpathSync.native(path.resolve(stateDir));
|
||||
const stateHash = createHash("sha256").update(canonicalStateDir).digest("hex").slice(0, 8);
|
||||
const lockDir = resolveTestLockDir();
|
||||
return { lockPath: path.join(lockDir, `gateway.${hash}.lock`), configPath };
|
||||
}
|
||||
|
||||
function makeProcStat(pid: number, startTime: number) {
|
||||
const fields = [
|
||||
"R",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
"1",
|
||||
String(startTime),
|
||||
"1",
|
||||
"1",
|
||||
];
|
||||
return `${pid} (node) ${fields.join(" ")}`;
|
||||
return {
|
||||
lockPath: path.join(lockDir, `gateway.${configHash}.lock`),
|
||||
configPath,
|
||||
stateLockPath: path.join(lockDir, `gateway.state.${stateHash}.lock`),
|
||||
};
|
||||
}
|
||||
|
||||
function createLockPayload(params: {
|
||||
@@ -104,12 +82,14 @@ function createLockPayload(params: {
|
||||
startTime: number;
|
||||
createdAt?: string;
|
||||
port?: number;
|
||||
role?: "gateway" | "sqlite-maintenance";
|
||||
}) {
|
||||
return {
|
||||
pid: process.pid,
|
||||
createdAt: params.createdAt ?? new Date().toISOString(),
|
||||
configPath: params.configPath,
|
||||
...(params.port ? { port: params.port } : {}),
|
||||
...(params.role ? { role: params.role } : {}),
|
||||
startTime: params.startTime,
|
||||
};
|
||||
}
|
||||
@@ -210,6 +190,78 @@ describe("gateway lock", () => {
|
||||
await expectGatewayLock(lock2).release();
|
||||
});
|
||||
|
||||
it("serializes different config paths that resolve to the same state directory", async () => {
|
||||
const stateDir = await fixtureRootTracker.make("shared-state");
|
||||
const configA = path.join(stateDir, "gateway-a.json");
|
||||
const configB = path.join(stateDir, "gateway-b.json");
|
||||
await fs.writeFile(configA, "{}", "utf8");
|
||||
await fs.writeFile(configB, "{}", "utf8");
|
||||
const envA = {
|
||||
...process.env,
|
||||
OPENCLAW_CONFIG_PATH: configA,
|
||||
OPENCLAW_STATE_DIR: stateDir,
|
||||
};
|
||||
const envB = {
|
||||
...process.env,
|
||||
OPENCLAW_CONFIG_PATH: configB,
|
||||
OPENCLAW_STATE_DIR: stateDir,
|
||||
};
|
||||
const lock = expectGatewayLock(
|
||||
await acquireForTest(envA, {
|
||||
platform: "darwin",
|
||||
}),
|
||||
);
|
||||
|
||||
try {
|
||||
await expect(
|
||||
acquireForTest(envB, {
|
||||
platform: "darwin",
|
||||
readProcessCmdline: () => ["openclaw-gateway"],
|
||||
timeoutMs: 15,
|
||||
}),
|
||||
).rejects.toBeInstanceOf(GatewayLockError);
|
||||
} finally {
|
||||
await lock.release();
|
||||
}
|
||||
});
|
||||
|
||||
it.skipIf(process.platform === "win32")(
|
||||
"canonicalizes state-directory aliases before choosing the ownership lock",
|
||||
async () => {
|
||||
const stateDir = await fixtureRootTracker.make("canonical-state");
|
||||
const aliasRoot = await fixtureRootTracker.make("canonical-alias");
|
||||
const stateAlias = path.join(aliasRoot, "state-link");
|
||||
const configA = path.join(stateDir, "gateway-a.json");
|
||||
const configB = path.join(aliasRoot, "gateway-b.json");
|
||||
await fs.writeFile(configA, "{}", "utf8");
|
||||
await fs.writeFile(configB, "{}", "utf8");
|
||||
await fs.symlink(stateDir, stateAlias);
|
||||
const envA = {
|
||||
...process.env,
|
||||
OPENCLAW_CONFIG_PATH: configA,
|
||||
OPENCLAW_STATE_DIR: stateDir,
|
||||
};
|
||||
const envB = {
|
||||
...process.env,
|
||||
OPENCLAW_CONFIG_PATH: configB,
|
||||
OPENCLAW_STATE_DIR: stateAlias,
|
||||
};
|
||||
const lock = expectGatewayLock(await acquireForTest(envA, { platform: "darwin" }));
|
||||
|
||||
try {
|
||||
await expect(
|
||||
acquireForTest(envB, {
|
||||
platform: "darwin",
|
||||
readProcessCmdline: () => ["openclaw-gateway"],
|
||||
timeoutMs: 15,
|
||||
}),
|
||||
).rejects.toBeInstanceOf(GatewayLockError);
|
||||
} finally {
|
||||
await lock.release();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("records and reads the active runtime port from a verified gateway lock", async () => {
|
||||
const env = await makeEnv();
|
||||
const lock = expectGatewayLock(
|
||||
@@ -234,10 +286,69 @@ describe("gateway lock", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("reads the active runtime port from state ownership without a config lock", async () => {
|
||||
const env = {
|
||||
...(await makeEnv()),
|
||||
OPENCLAW_ALLOW_MULTI_GATEWAY: "1",
|
||||
VITEST: "",
|
||||
};
|
||||
const lock = expectGatewayLock(
|
||||
await acquireForTest(env, {
|
||||
platform: "darwin",
|
||||
port: 48789,
|
||||
readProcessCmdline: () => ["openclaw-gateway"],
|
||||
}),
|
||||
);
|
||||
|
||||
try {
|
||||
const { lockPath } = resolveLockPath(env);
|
||||
await expect(fs.access(lockPath)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(
|
||||
readActiveGatewayLockPort({
|
||||
env,
|
||||
lockDir: resolveTestLockDir(),
|
||||
platform: "darwin",
|
||||
readProcessCmdline: () => ["openclaw-gateway"],
|
||||
}),
|
||||
).resolves.toBe(48789);
|
||||
} finally {
|
||||
await lock.release();
|
||||
}
|
||||
});
|
||||
|
||||
it("reads the active runtime port across configs that share a state directory", async () => {
|
||||
const envA = await makeEnv();
|
||||
const configB = path.join(resolveStateDir(envA), "gateway-b.json");
|
||||
await fs.writeFile(configB, "{}", "utf8");
|
||||
const envB = { ...envA, OPENCLAW_CONFIG_PATH: configB };
|
||||
const lock = expectGatewayLock(
|
||||
await acquireForTest(envA, {
|
||||
platform: "darwin",
|
||||
port: 48789,
|
||||
readProcessCmdline: () => ["openclaw-gateway"],
|
||||
}),
|
||||
);
|
||||
|
||||
try {
|
||||
const { lockPath } = resolveLockPath(envB);
|
||||
await expect(fs.access(lockPath)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(
|
||||
readActiveGatewayLockPort({
|
||||
env: envB,
|
||||
lockDir: resolveTestLockDir(),
|
||||
platform: "darwin",
|
||||
readProcessCmdline: () => ["openclaw-gateway"],
|
||||
}),
|
||||
).resolves.toBe(48789);
|
||||
} finally {
|
||||
await lock.release();
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps a retitled gateway lock owned during concurrent acquisition", async () => {
|
||||
const env = await makeEnv();
|
||||
const lock = expectGatewayLock(await acquireForTest(env, { platform: "darwin", port: 48789 }));
|
||||
const connectSpy = createPortProbeConnectionSpy("connect");
|
||||
const connectSpy = createPortProbeConnectionSpy("refused");
|
||||
|
||||
try {
|
||||
await expect(
|
||||
@@ -248,12 +359,64 @@ describe("gateway lock", () => {
|
||||
readProcessCmdline: () => ["openclaw-gateway"],
|
||||
}),
|
||||
).rejects.toBeInstanceOf(GatewayLockError);
|
||||
expect(connectSpy).toHaveBeenCalled();
|
||||
expect(connectSpy).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
await lock.release();
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps a verified owner when a second gateway requests a different unbound port", async () => {
|
||||
const env = await makeEnv();
|
||||
const lock = expectGatewayLock(
|
||||
await acquireForTest(env, {
|
||||
platform: "darwin",
|
||||
port: 18789,
|
||||
}),
|
||||
);
|
||||
const connectSpy = createPortProbeConnectionSpy("refused");
|
||||
|
||||
try {
|
||||
await expect(
|
||||
acquireForTest(env, {
|
||||
platform: "darwin",
|
||||
port: 28789,
|
||||
timeoutMs: 15,
|
||||
readProcessCmdline: () => ["openclaw-gateway"],
|
||||
}),
|
||||
).rejects.toBeInstanceOf(GatewayLockError);
|
||||
expect(connectSpy).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
connectSpy.mockRestore();
|
||||
await lock.release();
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps a live SQLite maintenance owner when the requested gateway port is free", async () => {
|
||||
const env = await makeEnv();
|
||||
const lock = expectGatewayLock(
|
||||
await acquireForTest(env, {
|
||||
...({ role: "sqlite-maintenance" } as GatewayLockOptions),
|
||||
platform: "darwin",
|
||||
}),
|
||||
);
|
||||
const connectSpy = createPortProbeConnectionSpy("refused");
|
||||
|
||||
try {
|
||||
await expect(
|
||||
acquireForTest(env, {
|
||||
platform: "darwin",
|
||||
port: 18789,
|
||||
timeoutMs: 15,
|
||||
readProcessCmdline: () => ["openclaw", "doctor", "--state-sqlite", "compact"],
|
||||
}),
|
||||
).rejects.toBeInstanceOf(GatewayLockError);
|
||||
expect(connectSpy).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
connectSpy.mockRestore();
|
||||
await lock.release();
|
||||
}
|
||||
});
|
||||
|
||||
it("ignores active-port metadata when the lock owner cannot be verified", async () => {
|
||||
const env = await makeEnv();
|
||||
const { lockPath, configPath } = resolveLockPath(env);
|
||||
@@ -276,25 +439,66 @@ describe("gateway lock", () => {
|
||||
const payload = createLockPayload({ configPath, startTime: 111 });
|
||||
await fs.writeFile(lockPath, JSON.stringify(payload), "utf8");
|
||||
|
||||
const statValue = makeProcStat(process.pid, 222);
|
||||
const spy = mockProcStatRead({
|
||||
onProcRead: () => statValue,
|
||||
});
|
||||
|
||||
const lock = await acquireForTest(env, {
|
||||
timeoutMs: 80,
|
||||
pollIntervalMs: 5,
|
||||
platform: "linux",
|
||||
readProcessStartTime: () => 222,
|
||||
});
|
||||
const acquiredLock = expectGatewayLock(lock);
|
||||
|
||||
await acquiredLock.release();
|
||||
spy.mockRestore();
|
||||
});
|
||||
|
||||
it("serializes concurrent stale-lock reclamation", async () => {
|
||||
vi.useRealTimers();
|
||||
const env = await makeEnv();
|
||||
const { configPath, stateLockPath } = resolveLockPath(env);
|
||||
await fs.mkdir(path.dirname(stateLockPath), { recursive: true });
|
||||
await fs.writeFile(
|
||||
stateLockPath,
|
||||
JSON.stringify(createLockPayload({ configPath, startTime: 111 })),
|
||||
"utf8",
|
||||
);
|
||||
|
||||
const attempts = await Promise.allSettled([
|
||||
acquireForTest(env, {
|
||||
platform: "linux",
|
||||
readProcessStartTime: () => 222,
|
||||
timeoutMs: 80,
|
||||
}),
|
||||
acquireForTest(env, {
|
||||
platform: "linux",
|
||||
readProcessStartTime: () => 222,
|
||||
timeoutMs: 25,
|
||||
}),
|
||||
]);
|
||||
const acquired = attempts.filter(
|
||||
(result): result is PromiseFulfilledResult<GatewayLock> =>
|
||||
result.status === "fulfilled" && result.value !== null,
|
||||
);
|
||||
const rejected = attempts.filter(
|
||||
(result): result is PromiseRejectedResult => result.status === "rejected",
|
||||
);
|
||||
|
||||
expect(acquired).toHaveLength(1);
|
||||
expect(rejected).toHaveLength(1);
|
||||
expect(rejected[0]?.reason).toBeInstanceOf(GatewayLockError);
|
||||
await expect(fs.access(stateLockPath)).resolves.toBeUndefined();
|
||||
|
||||
const acquiredResult = acquired[0];
|
||||
if (!acquiredResult) {
|
||||
throw new Error("Expected one successful stale-lock contender");
|
||||
}
|
||||
await acquiredResult.value.release();
|
||||
const nextLock = expectGatewayLock(await acquireForTest(env));
|
||||
await nextLock.release();
|
||||
});
|
||||
|
||||
it("keeps lock on linux when proc access fails unless stale", async () => {
|
||||
vi.useRealTimers();
|
||||
const env = await makeEnv();
|
||||
const { stateLockPath } = resolveLockPath(env);
|
||||
await writeLockFile(env);
|
||||
const spy = createEaccesProcStatSpy();
|
||||
|
||||
@@ -302,12 +506,196 @@ describe("gateway lock", () => {
|
||||
timeoutMs: 15,
|
||||
staleMs: 10_000,
|
||||
platform: "linux",
|
||||
readProcessCmdline: () => null,
|
||||
});
|
||||
await expect(pending).rejects.toBeInstanceOf(GatewayLockError);
|
||||
await expect(fs.access(stateLockPath)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
|
||||
spy.mockRestore();
|
||||
});
|
||||
|
||||
it("keeps a verified maintenance owner when process start identity is unavailable", async () => {
|
||||
vi.useRealTimers();
|
||||
const env = await makeEnv();
|
||||
const { lockPath, configPath } = resolveLockPath(env);
|
||||
await fs.writeFile(
|
||||
lockPath,
|
||||
JSON.stringify(
|
||||
createLockPayload({
|
||||
configPath,
|
||||
createdAt: "2000-01-01T00:00:00.000Z",
|
||||
role: "sqlite-maintenance",
|
||||
startTime: 111,
|
||||
}),
|
||||
),
|
||||
"utf8",
|
||||
);
|
||||
const spy = createEaccesProcStatSpy();
|
||||
|
||||
try {
|
||||
await expect(
|
||||
acquireForTest(env, {
|
||||
timeoutMs: 15,
|
||||
staleMs: 0,
|
||||
platform: "linux",
|
||||
readProcessCmdline: () => [
|
||||
"node",
|
||||
"/srv/openclaw/openclaw.mjs",
|
||||
"doctor",
|
||||
"--state-sqlite",
|
||||
"compact",
|
||||
],
|
||||
}),
|
||||
).rejects.toBeInstanceOf(GatewayLockError);
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("reclaims a maintenance lock when its live pid belongs to another process", async () => {
|
||||
vi.useRealTimers();
|
||||
const env = await makeEnv();
|
||||
const { lockPath, configPath } = resolveLockPath(env);
|
||||
await fs.writeFile(
|
||||
lockPath,
|
||||
JSON.stringify(
|
||||
createLockPayload({
|
||||
configPath,
|
||||
role: "sqlite-maintenance",
|
||||
startTime: 111,
|
||||
}),
|
||||
),
|
||||
"utf8",
|
||||
);
|
||||
const spy = createEaccesProcStatSpy();
|
||||
|
||||
try {
|
||||
const lock = await acquireForTest(env, {
|
||||
platform: "linux",
|
||||
readProcessCmdline: () => ["node", "worker.js"],
|
||||
timeoutMs: 80,
|
||||
});
|
||||
await expectGatewayLock(lock).release();
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("reclaims a Windows maintenance lock when the pid creation time changed", async () => {
|
||||
vi.useRealTimers();
|
||||
const env = await makeEnv();
|
||||
const { lockPath, configPath } = resolveLockPath(env);
|
||||
await fs.writeFile(
|
||||
lockPath,
|
||||
JSON.stringify(
|
||||
createLockPayload({
|
||||
configPath,
|
||||
role: "sqlite-maintenance",
|
||||
startTime: 111,
|
||||
}),
|
||||
),
|
||||
"utf8",
|
||||
);
|
||||
|
||||
const lock = await acquireForTest(env, {
|
||||
platform: "win32",
|
||||
readProcessCmdline: () => ["openclaw", "doctor", "--state-sqlite", "compact"],
|
||||
readProcessStartTime: () => 222,
|
||||
timeoutMs: 80,
|
||||
});
|
||||
await expectGatewayLock(lock).release();
|
||||
});
|
||||
|
||||
it("keeps a Windows maintenance lock when the pid creation time matches", async () => {
|
||||
vi.useRealTimers();
|
||||
const env = await makeEnv();
|
||||
const { lockPath, configPath } = resolveLockPath(env);
|
||||
await fs.writeFile(
|
||||
lockPath,
|
||||
JSON.stringify(
|
||||
createLockPayload({
|
||||
configPath,
|
||||
createdAt: "2000-01-01T00:00:00.000Z",
|
||||
role: "sqlite-maintenance",
|
||||
startTime: 111,
|
||||
}),
|
||||
),
|
||||
"utf8",
|
||||
);
|
||||
|
||||
await expect(
|
||||
acquireForTest(env, {
|
||||
platform: "win32",
|
||||
readProcessCmdline: () => ["node", "worker.js"],
|
||||
readProcessStartTime: () => 111,
|
||||
staleMs: 0,
|
||||
timeoutMs: 15,
|
||||
}),
|
||||
).rejects.toBeInstanceOf(GatewayLockError);
|
||||
});
|
||||
|
||||
it("fails closed for a recent maintenance owner with unreadable process identity", async () => {
|
||||
vi.useRealTimers();
|
||||
const env = await makeEnv();
|
||||
const { lockPath, configPath } = resolveLockPath(env);
|
||||
await fs.writeFile(
|
||||
lockPath,
|
||||
JSON.stringify(
|
||||
createLockPayload({
|
||||
configPath,
|
||||
role: "sqlite-maintenance",
|
||||
startTime: 111,
|
||||
}),
|
||||
),
|
||||
"utf8",
|
||||
);
|
||||
const spy = createEaccesProcStatSpy();
|
||||
|
||||
try {
|
||||
await expect(
|
||||
acquireForTest(env, {
|
||||
platform: "linux",
|
||||
readProcessCmdline: () => null,
|
||||
staleMs: 10_000,
|
||||
timeoutMs: 15,
|
||||
}),
|
||||
).rejects.toBeInstanceOf(GatewayLockError);
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("ages out an old maintenance owner with unreadable process identity", async () => {
|
||||
vi.useRealTimers();
|
||||
const env = await makeEnv();
|
||||
const { lockPath, configPath } = resolveLockPath(env);
|
||||
await fs.writeFile(
|
||||
lockPath,
|
||||
JSON.stringify(
|
||||
createLockPayload({
|
||||
configPath,
|
||||
createdAt: "2000-01-01T00:00:00.000Z",
|
||||
role: "sqlite-maintenance",
|
||||
startTime: 111,
|
||||
}),
|
||||
),
|
||||
"utf8",
|
||||
);
|
||||
const spy = createEaccesProcStatSpy();
|
||||
|
||||
try {
|
||||
const lock = await acquireForTest(env, {
|
||||
platform: "linux",
|
||||
readProcessCmdline: () => null,
|
||||
staleMs: 0,
|
||||
timeoutMs: 80,
|
||||
});
|
||||
await expectGatewayLock(lock).release();
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps lock when fs.stat fails until payload is stale", async () => {
|
||||
vi.useRealTimers();
|
||||
const env = await makeEnv();
|
||||
@@ -321,6 +709,7 @@ describe("gateway lock", () => {
|
||||
timeoutMs: 20,
|
||||
staleMs: 10_000,
|
||||
platform: "linux",
|
||||
readProcessCmdline: () => null,
|
||||
});
|
||||
await expect(pending).rejects.toBeInstanceOf(GatewayLockError);
|
||||
|
||||
@@ -328,11 +717,10 @@ describe("gateway lock", () => {
|
||||
statSpy.mockRestore();
|
||||
});
|
||||
|
||||
it("treats lock as stale when owner pid is alive but configured port is free", async () => {
|
||||
it("reclaims a lock when its live pid belongs to a non-gateway process", async () => {
|
||||
vi.useRealTimers();
|
||||
const env = await makeEnv();
|
||||
await writeRecentLockFile(env);
|
||||
const connectSpy = createPortProbeConnectionSpy("refused");
|
||||
|
||||
const lock = await acquireForTest(env, {
|
||||
timeoutMs: 80,
|
||||
@@ -340,9 +728,9 @@ describe("gateway lock", () => {
|
||||
staleMs: 10_000,
|
||||
platform: "darwin",
|
||||
port: 18789,
|
||||
readProcessCmdline: () => ["node", "worker.js"],
|
||||
});
|
||||
await expectGatewayLock(lock).release();
|
||||
connectSpy.mockRestore();
|
||||
});
|
||||
|
||||
it("keeps lock when configured port is busy and owner pid is alive", async () => {
|
||||
@@ -392,13 +780,34 @@ describe("gateway lock", () => {
|
||||
expect(sleepDelays).toEqual([5]);
|
||||
});
|
||||
|
||||
it("returns null when multi-gateway override is enabled", async () => {
|
||||
it("keeps state ownership when the config singleton override is enabled", async () => {
|
||||
const env = await makeEnv();
|
||||
const lock = await acquireGatewayLock({
|
||||
env: { ...env, OPENCLAW_ALLOW_MULTI_GATEWAY: "1", VITEST: "" },
|
||||
lockDir: resolveTestLockDir(),
|
||||
});
|
||||
expect(lock).toBeNull();
|
||||
const { lockPath, stateLockPath } = resolveLockPath(env);
|
||||
const lock = expectGatewayLock(
|
||||
await acquireGatewayLock({
|
||||
allowInTests: true,
|
||||
env: { ...env, OPENCLAW_ALLOW_MULTI_GATEWAY: "1", VITEST: "" },
|
||||
lockDir: resolveTestLockDir(),
|
||||
}),
|
||||
);
|
||||
|
||||
try {
|
||||
expect(lock.lockPath).toBe(stateLockPath);
|
||||
await expect(fs.access(stateLockPath)).resolves.toBeUndefined();
|
||||
await expect(fs.access(lockPath)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(
|
||||
acquireGatewayLock({
|
||||
allowInTests: true,
|
||||
env,
|
||||
lockDir: resolveTestLockDir(),
|
||||
platform: "darwin",
|
||||
readProcessCmdline: () => ["openclaw-gateway"],
|
||||
timeoutMs: 15,
|
||||
}),
|
||||
).rejects.toBeInstanceOf(GatewayLockError);
|
||||
} finally {
|
||||
await lock.release();
|
||||
}
|
||||
});
|
||||
|
||||
it("returns null in test env unless allowInTests is set", async () => {
|
||||
@@ -451,7 +860,7 @@ describe("gateway lock", () => {
|
||||
it("closes handle and removes lock file when writeFile fails after open succeeds", async () => {
|
||||
vi.useRealTimers();
|
||||
const env = await makeEnv();
|
||||
const { lockPath } = resolveLockPath(env);
|
||||
const { stateLockPath } = resolveLockPath(env);
|
||||
|
||||
const writeError = Object.assign(new Error("ENOSPC: no space left on device"), {
|
||||
code: "ENOSPC",
|
||||
@@ -459,7 +868,7 @@ describe("gateway lock", () => {
|
||||
const close = vi.fn<() => Promise<void>>().mockResolvedValue(undefined);
|
||||
const mockHandle = {
|
||||
writeFile: vi.fn().mockImplementation(async () => {
|
||||
await fs.writeFile(lockPath, "partial", "utf8");
|
||||
await fs.writeFile(stateLockPath, "partial", "utf8");
|
||||
throw writeError;
|
||||
}),
|
||||
close,
|
||||
@@ -473,7 +882,7 @@ describe("gateway lock", () => {
|
||||
});
|
||||
|
||||
expect(close).toHaveBeenCalledTimes(1);
|
||||
await expect(fs.access(lockPath)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(fs.access(stateLockPath)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
|
||||
openSpy.mockRestore();
|
||||
});
|
||||
@@ -492,6 +901,7 @@ describe("gateway lock", () => {
|
||||
platform: "win32",
|
||||
port: 18789,
|
||||
readProcessCmdline: () => ["chrome.exe", "--no-sandbox"],
|
||||
readProcessStartTime: () => null,
|
||||
});
|
||||
await expectGatewayLock(lock).release();
|
||||
|
||||
@@ -516,6 +926,7 @@ describe("gateway lock", () => {
|
||||
"gateway",
|
||||
"run",
|
||||
],
|
||||
readProcessStartTime: () => null,
|
||||
});
|
||||
await expect(pending).rejects.toBeInstanceOf(GatewayLockError);
|
||||
|
||||
@@ -536,6 +947,7 @@ describe("gateway lock", () => {
|
||||
platform: "win32",
|
||||
port: 18789,
|
||||
readProcessCmdline: () => null,
|
||||
readProcessStartTime: () => null,
|
||||
});
|
||||
await expect(pending).rejects.toBeInstanceOf(GatewayLockError);
|
||||
|
||||
|
||||
+339
-149
@@ -2,8 +2,8 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import fsSync from "node:fs";
|
||||
import fs from "node:fs/promises";
|
||||
import net from "node:net";
|
||||
import path from "node:path";
|
||||
import type { DatabaseSync } from "node:sqlite";
|
||||
import {
|
||||
resolvePositiveTimerTimeoutMs,
|
||||
resolveTimerTimeoutMs,
|
||||
@@ -11,22 +11,28 @@ import {
|
||||
} from "@openclaw/normalization-core/number-coercion";
|
||||
import { z } from "zod";
|
||||
import { resolveConfigPath, resolveGatewayLockDir, resolveStateDir } from "../config/paths.js";
|
||||
import { isPidAlive } from "../shared/pid-alive.js";
|
||||
import { getFileLockProcessStartTime, isPidAlive } from "../shared/pid-alive.js";
|
||||
import { safeParseJsonWithSchema } from "../utils/zod-parse.js";
|
||||
import { sha256HexPrefix } from "./crypto-digest.js";
|
||||
import { isGatewayArgv, parseProcCmdline } from "./gateway-process-argv.js";
|
||||
import { readWindowsProcessArgsSync } from "./windows-port-pids.js";
|
||||
import { isGatewayArgv, isOpenClawCommandArgv, parseProcCmdline } from "./gateway-process-argv.js";
|
||||
import { requireNodeSqlite } from "./node-sqlite.js";
|
||||
import { isSqliteLockError } from "./sqlite-transaction.js";
|
||||
import {
|
||||
readWindowsProcessArgsSync,
|
||||
readWindowsProcessStartTimeSync,
|
||||
} from "./windows-port-pids.js";
|
||||
|
||||
const DEFAULT_TIMEOUT_MS = 5000;
|
||||
const DEFAULT_POLL_INTERVAL_MS = 100;
|
||||
const DEFAULT_STALE_MS = 30_000;
|
||||
const DEFAULT_PORT_PROBE_TIMEOUT_MS = 1000;
|
||||
|
||||
type LockPayload = {
|
||||
pid: number;
|
||||
createdAt: string;
|
||||
configPath: string;
|
||||
port?: number;
|
||||
role?: GatewayLockRole;
|
||||
stateDir?: string;
|
||||
startTime?: number;
|
||||
};
|
||||
|
||||
@@ -35,15 +41,20 @@ const LockPayloadSchema = z.object({
|
||||
createdAt: z.string(),
|
||||
configPath: z.string(),
|
||||
port: z.number().int().min(1).max(65_535).optional(),
|
||||
role: z.enum(["gateway", "sqlite-maintenance"]).optional(),
|
||||
stateDir: z.string().optional(),
|
||||
startTime: z.number().optional(),
|
||||
}) as z.ZodType<LockPayload>;
|
||||
|
||||
type GatewayLockHandle = {
|
||||
lockPath: string;
|
||||
stateLockPath: string;
|
||||
configPath: string;
|
||||
release: () => Promise<void>;
|
||||
};
|
||||
|
||||
export type GatewayLockRole = "gateway" | "sqlite-maintenance";
|
||||
|
||||
export type GatewayLockOptions = {
|
||||
env?: NodeJS.ProcessEnv;
|
||||
timeoutMs?: number;
|
||||
@@ -55,8 +66,11 @@ export type GatewayLockOptions = {
|
||||
now?: () => number;
|
||||
sleep?: (ms: number) => Promise<void>;
|
||||
lockDir?: string;
|
||||
role?: GatewayLockRole;
|
||||
/** Override process command-line reader (testing seam). */
|
||||
readProcessCmdline?: (pid: number) => string[] | null;
|
||||
/** Override process start-identity reader (testing seam). */
|
||||
readProcessStartTime?: (pid: number) => number | null;
|
||||
};
|
||||
|
||||
export class GatewayLockError extends Error {
|
||||
@@ -71,6 +85,45 @@ export class GatewayLockError extends Error {
|
||||
|
||||
type LockOwnerStatus = "alive" | "dead" | "unknown";
|
||||
|
||||
type GatewayLockCoordinator = {
|
||||
release: () => void;
|
||||
};
|
||||
|
||||
function tryAcquireGatewayLockCoordinator(lockPath: string): GatewayLockCoordinator | null {
|
||||
const { DatabaseSync } = requireNodeSqlite();
|
||||
const coordinatorDb: DatabaseSync = new DatabaseSync(`${lockPath}.sqlite`);
|
||||
try {
|
||||
coordinatorDb.exec("PRAGMA busy_timeout = 0; BEGIN EXCLUSIVE;");
|
||||
} catch (error) {
|
||||
try {
|
||||
coordinatorDb.close();
|
||||
} catch {}
|
||||
if (isSqliteLockError(error)) {
|
||||
return null;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
|
||||
return {
|
||||
release: () => {
|
||||
let releaseError: unknown;
|
||||
try {
|
||||
coordinatorDb.exec("ROLLBACK");
|
||||
} catch (error) {
|
||||
releaseError = error;
|
||||
}
|
||||
try {
|
||||
coordinatorDb.close();
|
||||
} catch (error) {
|
||||
releaseError ??= error;
|
||||
}
|
||||
if (releaseError) {
|
||||
throw new GatewayLockError("failed to release gateway lock coordinator", releaseError);
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function readLinuxCmdline(pid: number): string[] | null {
|
||||
try {
|
||||
const raw = fsSync.readFileSync(`/proc/${pid}/cmdline`, "utf8");
|
||||
@@ -111,48 +164,13 @@ function readDarwinCmdline(pid: number): string[] | null {
|
||||
}
|
||||
}
|
||||
|
||||
function readLinuxStartTime(pid: number): number | null {
|
||||
try {
|
||||
const raw = fsSync.readFileSync(`/proc/${pid}/stat`, "utf8").trim();
|
||||
const closeParen = raw.lastIndexOf(")");
|
||||
if (closeParen < 0) {
|
||||
return null;
|
||||
}
|
||||
const rest = raw.slice(closeParen + 1).trim();
|
||||
const fields = rest.split(/\s+/);
|
||||
const startTime = Number.parseInt(fields[19] ?? "", 10);
|
||||
return Number.isFinite(startTime) ? startTime : null;
|
||||
} catch {
|
||||
function readProcessStartTime(pid: number, platform: NodeJS.Platform): number | null {
|
||||
if (platform !== process.platform) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async function checkPortFree(port: number, host = "127.0.0.1"): Promise<boolean> {
|
||||
return await new Promise<boolean>((resolve) => {
|
||||
const socket = net.createConnection({ port, host });
|
||||
let settled = false;
|
||||
const finish = (result: boolean) => {
|
||||
if (settled) {
|
||||
return;
|
||||
}
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
socket.removeAllListeners();
|
||||
socket.destroy();
|
||||
resolve(result);
|
||||
};
|
||||
const timer = setTimeout(() => {
|
||||
// Conservative for liveness checks: timeout usually means no responsive
|
||||
// local listener, so treat the lock owner as stale.
|
||||
finish(true);
|
||||
}, DEFAULT_PORT_PROBE_TIMEOUT_MS);
|
||||
socket.once("connect", () => {
|
||||
finish(false);
|
||||
});
|
||||
socket.once("error", () => {
|
||||
finish(true);
|
||||
});
|
||||
});
|
||||
return platform === "win32"
|
||||
? readWindowsProcessStartTimeSync(pid, CMDLINE_EXEC_TIMEOUT_MS)
|
||||
: getFileLockProcessStartTime(pid);
|
||||
}
|
||||
|
||||
function defaultReadProcessCmdline(pid: number, platform: NodeJS.Platform): string[] | null {
|
||||
@@ -172,35 +190,36 @@ async function resolveGatewayOwnerStatus(
|
||||
pid: number,
|
||||
payload: LockPayload | null,
|
||||
platform: NodeJS.Platform,
|
||||
port: number | undefined,
|
||||
readCmdline?: (pid: number) => string[] | null,
|
||||
readStartTime?: (pid: number) => number | null,
|
||||
opts: { trustUnknownCmdlineOwner?: boolean } = {},
|
||||
): Promise<LockOwnerStatus> {
|
||||
if (port != null) {
|
||||
const portFree = await checkPortFree(port);
|
||||
if (portFree) {
|
||||
return "dead";
|
||||
}
|
||||
}
|
||||
|
||||
const role = payload?.role ?? "gateway";
|
||||
if (!isPidAlive(pid)) {
|
||||
return "dead";
|
||||
}
|
||||
|
||||
// On Linux, an extra start-time comparison catches PID recycling even when
|
||||
// the replacement process also looks like a gateway (same argv shape).
|
||||
if (platform === "linux") {
|
||||
const payloadStartTime = payload?.startTime;
|
||||
if (Number.isFinite(payloadStartTime)) {
|
||||
const currentStartTime = readLinuxStartTime(pid);
|
||||
if (currentStartTime == null) {
|
||||
return "unknown";
|
||||
}
|
||||
// Process start identity catches PID recycling even when the replacement
|
||||
// process has the same argv shape.
|
||||
const payloadStartTime = payload?.startTime;
|
||||
if (Number.isFinite(payloadStartTime)) {
|
||||
const currentStartTime = (
|
||||
readStartTime ?? ((ownerPid) => readProcessStartTime(ownerPid, platform))
|
||||
)(pid);
|
||||
if (currentStartTime != null) {
|
||||
return currentStartTime === payloadStartTime ? "alive" : "dead";
|
||||
}
|
||||
}
|
||||
|
||||
const readFn = readCmdline ?? ((p: number) => defaultReadProcessCmdline(p, platform));
|
||||
if (role === "sqlite-maintenance") {
|
||||
const args = readFn(pid);
|
||||
if (!args) {
|
||||
return "unknown";
|
||||
}
|
||||
return isOpenClawCommandArgv(args, "doctor") ? "alive" : "dead";
|
||||
}
|
||||
|
||||
const args = readFn(pid);
|
||||
if (!args) {
|
||||
// Cmdline reader unavailable or failed. On Linux legacy locks (no
|
||||
@@ -223,29 +242,69 @@ async function readLockPayload(lockPath: string): Promise<LockPayload | null> {
|
||||
}
|
||||
}
|
||||
|
||||
function resolveGatewayLockPath(env: NodeJS.ProcessEnv, lockDir = resolveGatewayLockDir()) {
|
||||
const stateDir = resolveStateDir(env);
|
||||
const configPath = resolveConfigPath(env, stateDir);
|
||||
const hash = sha256HexPrefix(configPath, 8);
|
||||
const lockPath = path.join(lockDir, `gateway.${hash}.lock`);
|
||||
return { lockPath, configPath };
|
||||
function canonicalizeStateDir(stateDir: string): string {
|
||||
const resolved = path.resolve(stateDir);
|
||||
try {
|
||||
return fsSync.realpathSync.native(resolved);
|
||||
} catch {
|
||||
const missingSegments: string[] = [];
|
||||
let current = resolved;
|
||||
while (true) {
|
||||
const parent = path.dirname(current);
|
||||
if (parent === current) {
|
||||
return resolved;
|
||||
}
|
||||
missingSegments.push(path.basename(current));
|
||||
current = parent;
|
||||
try {
|
||||
return path.join(fsSync.realpathSync.native(current), ...missingSegments.toReversed());
|
||||
} catch {
|
||||
// Keep walking so aliases in an existing ancestor still share one lock.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function resolveGatewayLockPaths(env: NodeJS.ProcessEnv, lockDir = resolveGatewayLockDir()) {
|
||||
const resolvedStateDir = resolveStateDir(env);
|
||||
const stateDir = canonicalizeStateDir(resolvedStateDir);
|
||||
const configPath = resolveConfigPath(env, resolvedStateDir);
|
||||
const configHash = sha256HexPrefix(configPath, 8);
|
||||
const stateHash = sha256HexPrefix(stateDir, 8);
|
||||
return {
|
||||
configLockPath: path.join(lockDir, `gateway.${configHash}.lock`),
|
||||
configPath,
|
||||
stateDir,
|
||||
stateLockPath: path.join(lockDir, `gateway.state.${stateHash}.lock`),
|
||||
};
|
||||
}
|
||||
|
||||
export async function readActiveGatewayLockPort(
|
||||
opts: Pick<GatewayLockOptions, "env" | "lockDir" | "platform" | "readProcessCmdline"> = {},
|
||||
opts: Pick<
|
||||
GatewayLockOptions,
|
||||
"env" | "lockDir" | "platform" | "readProcessCmdline" | "readProcessStartTime"
|
||||
> = {},
|
||||
): Promise<number | undefined> {
|
||||
const env = opts.env ?? process.env;
|
||||
const { lockPath } = resolveGatewayLockPath(env, opts.lockDir);
|
||||
const { configLockPath, stateLockPath } = resolveGatewayLockPaths(env, opts.lockDir);
|
||||
const configPort = await readVerifiedGatewayLockPort(configLockPath, opts);
|
||||
return configPort ?? (await readVerifiedGatewayLockPort(stateLockPath, opts));
|
||||
}
|
||||
|
||||
async function readVerifiedGatewayLockPort(
|
||||
lockPath: string,
|
||||
opts: Pick<GatewayLockOptions, "platform" | "readProcessCmdline" | "readProcessStartTime">,
|
||||
): Promise<number | undefined> {
|
||||
const payload = await readLockPayload(lockPath);
|
||||
if (!payload?.port) {
|
||||
if (!payload?.port || payload.role === "sqlite-maintenance") {
|
||||
return undefined;
|
||||
}
|
||||
const ownerStatus = await resolveGatewayOwnerStatus(
|
||||
payload.pid,
|
||||
payload,
|
||||
opts.platform ?? process.platform,
|
||||
undefined,
|
||||
opts.readProcessCmdline,
|
||||
opts.readProcessStartTime,
|
||||
{ trustUnknownCmdlineOwner: false },
|
||||
);
|
||||
return ownerStatus === "alive" ? payload.port : undefined;
|
||||
@@ -256,13 +315,77 @@ export async function acquireGatewayLock(
|
||||
): Promise<GatewayLockHandle | null> {
|
||||
const env = opts.env ?? process.env;
|
||||
const allowInTests = opts.allowInTests === true;
|
||||
if (
|
||||
env.OPENCLAW_ALLOW_MULTI_GATEWAY === "1" ||
|
||||
(!allowInTests && (env.VITEST || env.NODE_ENV === "test"))
|
||||
) {
|
||||
if (!allowInTests && (env.VITEST || env.NODE_ENV === "test")) {
|
||||
return null;
|
||||
}
|
||||
const role = opts.role ?? "gateway";
|
||||
const paths = resolveGatewayLockPaths(env, opts.lockDir);
|
||||
const stateLock = await acquireLockFile({
|
||||
...opts,
|
||||
configPath: paths.configPath,
|
||||
env,
|
||||
lockPath: paths.stateLockPath,
|
||||
role,
|
||||
stateDir: paths.stateDir,
|
||||
});
|
||||
const shouldAcquireConfigLock =
|
||||
role === "sqlite-maintenance" || env.OPENCLAW_ALLOW_MULTI_GATEWAY !== "1";
|
||||
if (!shouldAcquireConfigLock) {
|
||||
return {
|
||||
...stateLock,
|
||||
stateLockPath: stateLock.lockPath,
|
||||
};
|
||||
}
|
||||
|
||||
try {
|
||||
const configLock = await acquireLockFile({
|
||||
...opts,
|
||||
configPath: paths.configPath,
|
||||
env,
|
||||
lockPath: paths.configLockPath,
|
||||
role,
|
||||
stateDir: paths.stateDir,
|
||||
});
|
||||
return {
|
||||
...configLock,
|
||||
stateLockPath: stateLock.lockPath,
|
||||
release: async () => {
|
||||
let releaseError: Error | undefined;
|
||||
try {
|
||||
await configLock.release();
|
||||
} catch (error) {
|
||||
releaseError =
|
||||
error instanceof Error
|
||||
? error
|
||||
: new GatewayLockError("failed to release config lock", error);
|
||||
}
|
||||
try {
|
||||
await stateLock.release();
|
||||
} catch (error) {
|
||||
releaseError ??=
|
||||
error instanceof Error
|
||||
? error
|
||||
: new GatewayLockError("failed to release state lock", error);
|
||||
}
|
||||
if (releaseError) {
|
||||
throw releaseError;
|
||||
}
|
||||
},
|
||||
};
|
||||
} catch (error) {
|
||||
await stateLock.release().catch(() => undefined);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async function acquireLockFile(
|
||||
opts: GatewayLockOptions & {
|
||||
configPath: string;
|
||||
lockPath: string;
|
||||
role: GatewayLockRole;
|
||||
stateDir: string;
|
||||
},
|
||||
): Promise<Omit<GatewayLockHandle, "stateLockPath">> {
|
||||
const timeoutMs = resolveTimerTimeoutMs(opts.timeoutMs, DEFAULT_TIMEOUT_MS, 0);
|
||||
const pollIntervalMs = resolvePositiveTimerTimeoutMs(
|
||||
opts.pollIntervalMs,
|
||||
@@ -271,6 +394,7 @@ export async function acquireGatewayLock(
|
||||
const staleMs = resolveTimerTimeoutMs(opts.staleMs, DEFAULT_STALE_MS, 0);
|
||||
const platform = opts.platform ?? process.platform;
|
||||
const port = opts.port;
|
||||
const role = opts.role;
|
||||
const now = opts.now ?? Date.now;
|
||||
const sleep =
|
||||
opts.sleep ??
|
||||
@@ -278,95 +402,161 @@ export async function acquireGatewayLock(
|
||||
await new Promise((resolve) => {
|
||||
setTimeout(resolve, ms);
|
||||
}));
|
||||
const { lockPath, configPath } = resolveGatewayLockPath(env, opts.lockDir);
|
||||
const { configPath, lockPath, stateDir } = opts;
|
||||
await fs.mkdir(path.dirname(lockPath), { recursive: true });
|
||||
|
||||
const startedAt = now();
|
||||
let lastPayload: LockPayload | null = null;
|
||||
|
||||
while (now() - startedAt < timeoutMs) {
|
||||
let coordinator: GatewayLockCoordinator | null;
|
||||
try {
|
||||
const handle = await fs.open(lockPath, "wx");
|
||||
try {
|
||||
const startTime = platform === "linux" ? readLinuxStartTime(process.pid) : null;
|
||||
const payload: LockPayload = {
|
||||
pid: process.pid,
|
||||
createdAt: resolveTimestampMsToIsoString(now()),
|
||||
configPath,
|
||||
};
|
||||
if (typeof port === "number" && Number.isInteger(port) && port > 0 && port <= 65_535) {
|
||||
payload.port = port;
|
||||
}
|
||||
if (typeof startTime === "number" && Number.isFinite(startTime)) {
|
||||
payload.startTime = startTime;
|
||||
}
|
||||
await handle.writeFile(JSON.stringify(payload), "utf8");
|
||||
} catch (error) {
|
||||
// Acquisition owns both resources until the release callback exists.
|
||||
// Unwind them if payload preparation fails before ownership transfers.
|
||||
await handle.close().catch(() => undefined);
|
||||
await fs.rm(lockPath, { force: true }).catch(() => undefined);
|
||||
throw error;
|
||||
}
|
||||
return {
|
||||
lockPath,
|
||||
configPath,
|
||||
release: async () => {
|
||||
await handle.close().catch(() => undefined);
|
||||
await fs.rm(lockPath, { force: true });
|
||||
},
|
||||
};
|
||||
} catch (err) {
|
||||
const code = (err as { code?: unknown }).code;
|
||||
if (code !== "EEXIST") {
|
||||
throw new GatewayLockError(`failed to acquire gateway lock at ${lockPath}`, err);
|
||||
}
|
||||
coordinator = tryAcquireGatewayLockCoordinator(lockPath);
|
||||
} catch (error) {
|
||||
throw new GatewayLockError(`failed to acquire gateway lock at ${lockPath}`, error);
|
||||
}
|
||||
|
||||
if (!coordinator) {
|
||||
lastPayload = await readLockPayload(lockPath);
|
||||
const ownerPid = lastPayload?.pid;
|
||||
const ownerStatus = ownerPid
|
||||
? await resolveGatewayOwnerStatus(
|
||||
ownerPid,
|
||||
lastPayload,
|
||||
platform,
|
||||
port,
|
||||
opts.readProcessCmdline,
|
||||
)
|
||||
: "unknown";
|
||||
if (ownerStatus === "dead" && ownerPid) {
|
||||
await fs.rm(lockPath, { force: true });
|
||||
continue;
|
||||
}
|
||||
if (ownerStatus !== "alive") {
|
||||
let stale = false;
|
||||
if (lastPayload?.createdAt) {
|
||||
const createdAt = Date.parse(lastPayload.createdAt);
|
||||
stale = Number.isFinite(createdAt) ? now() - createdAt > staleMs : false;
|
||||
}
|
||||
if (!stale) {
|
||||
} else {
|
||||
let handle: Awaited<ReturnType<typeof fs.open>> | undefined;
|
||||
let acquisitionError: unknown;
|
||||
let waitForOwner = false;
|
||||
try {
|
||||
while (!handle && !waitForOwner) {
|
||||
let candidateHandle: Awaited<ReturnType<typeof fs.open>>;
|
||||
try {
|
||||
const st = await fs.stat(lockPath);
|
||||
stale = now() - st.mtimeMs > staleMs;
|
||||
} catch {
|
||||
// On Windows or locked filesystems we may be unable to stat the
|
||||
// lock file even though the existing gateway is still healthy.
|
||||
// Treat the lock as non-stale so we keep waiting instead of
|
||||
// forcefully removing another gateway's lock.
|
||||
stale = false;
|
||||
candidateHandle = await fs.open(lockPath, "wx");
|
||||
} catch (error) {
|
||||
const code = (error as { code?: unknown }).code;
|
||||
if (code !== "EEXIST") {
|
||||
throw error;
|
||||
}
|
||||
|
||||
lastPayload = await readLockPayload(lockPath);
|
||||
const ownerPid = lastPayload?.pid;
|
||||
const ownerStatus = ownerPid
|
||||
? await resolveGatewayOwnerStatus(
|
||||
ownerPid,
|
||||
lastPayload,
|
||||
platform,
|
||||
opts.readProcessCmdline,
|
||||
opts.readProcessStartTime,
|
||||
)
|
||||
: "unknown";
|
||||
if (ownerStatus === "dead" && ownerPid) {
|
||||
await fs.rm(lockPath, { force: true });
|
||||
continue;
|
||||
}
|
||||
if (ownerStatus !== "alive") {
|
||||
let stale = false;
|
||||
if (lastPayload?.createdAt) {
|
||||
const createdAt = Date.parse(lastPayload.createdAt);
|
||||
stale = Number.isFinite(createdAt) ? now() - createdAt > staleMs : false;
|
||||
}
|
||||
if (!stale) {
|
||||
try {
|
||||
const st = await fs.stat(lockPath);
|
||||
stale = now() - st.mtimeMs > staleMs;
|
||||
} catch {
|
||||
// On Windows or locked filesystems we may be unable to stat the
|
||||
// lock file even though the existing gateway is still healthy.
|
||||
// Treat the lock as non-stale so we keep waiting instead of
|
||||
// forcefully removing another gateway's lock.
|
||||
stale = false;
|
||||
}
|
||||
}
|
||||
if (stale) {
|
||||
await fs.rm(lockPath, { force: true });
|
||||
continue;
|
||||
}
|
||||
}
|
||||
waitForOwner = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
try {
|
||||
const startTime = (
|
||||
opts.readProcessStartTime ?? ((pid) => readProcessStartTime(pid, platform))
|
||||
)(process.pid);
|
||||
const payload: LockPayload = {
|
||||
pid: process.pid,
|
||||
createdAt: resolveTimestampMsToIsoString(now()),
|
||||
configPath,
|
||||
stateDir,
|
||||
};
|
||||
if (typeof port === "number" && Number.isInteger(port) && port > 0 && port <= 65_535) {
|
||||
payload.port = port;
|
||||
}
|
||||
if (role !== "gateway") {
|
||||
payload.role = role;
|
||||
}
|
||||
if (typeof startTime === "number" && Number.isFinite(startTime)) {
|
||||
payload.startTime = startTime;
|
||||
}
|
||||
await candidateHandle.writeFile(JSON.stringify(payload), "utf8");
|
||||
handle = candidateHandle;
|
||||
} catch (error) {
|
||||
// Acquisition owns both resources until the release callback exists.
|
||||
// Unwind them if payload preparation fails before ownership transfers.
|
||||
await candidateHandle.close().catch(() => undefined);
|
||||
await fs.rm(lockPath, { force: true }).catch(() => undefined);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
if (stale) {
|
||||
await fs.rm(lockPath, { force: true });
|
||||
continue;
|
||||
}
|
||||
} catch (error) {
|
||||
acquisitionError = error;
|
||||
}
|
||||
|
||||
const remainingMs = timeoutMs - (now() - startedAt);
|
||||
if (remainingMs <= 0) {
|
||||
break;
|
||||
if (handle) {
|
||||
return {
|
||||
lockPath,
|
||||
configPath,
|
||||
release: async () => {
|
||||
let releaseError: unknown;
|
||||
try {
|
||||
await handle.close();
|
||||
} catch (error) {
|
||||
releaseError = error;
|
||||
}
|
||||
try {
|
||||
await fs.rm(lockPath, { force: true });
|
||||
} catch (error) {
|
||||
releaseError ??= error;
|
||||
}
|
||||
try {
|
||||
coordinator.release();
|
||||
} catch (error) {
|
||||
releaseError ??= error;
|
||||
}
|
||||
if (releaseError) {
|
||||
throw new GatewayLockError(
|
||||
`failed to release gateway lock at ${lockPath}`,
|
||||
releaseError,
|
||||
);
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
try {
|
||||
coordinator.release();
|
||||
} catch (error) {
|
||||
acquisitionError ??= error;
|
||||
}
|
||||
if (acquisitionError) {
|
||||
throw new GatewayLockError(
|
||||
`failed to acquire gateway lock at ${lockPath}`,
|
||||
acquisitionError,
|
||||
);
|
||||
}
|
||||
await sleep(Math.min(pollIntervalMs, remainingMs));
|
||||
}
|
||||
|
||||
const remainingMs = timeoutMs - (now() - startedAt);
|
||||
if (remainingMs <= 0) {
|
||||
break;
|
||||
}
|
||||
await sleep(Math.min(pollIntervalMs, remainingMs));
|
||||
}
|
||||
|
||||
const owner = lastPayload?.pid ? ` (pid ${lastPayload.pid})` : "";
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// Tests gateway process argv parsing for diagnostics.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { isGatewayArgv, parseProcCmdline } from "./gateway-process-argv.js";
|
||||
import { isGatewayArgv, isOpenClawCommandArgv, parseProcCmdline } from "./gateway-process-argv.js";
|
||||
|
||||
describe("parseProcCmdline", () => {
|
||||
it("splits null-delimited argv and trims empty entries", () => {
|
||||
@@ -53,3 +53,22 @@ describe("isGatewayArgv", () => {
|
||||
expect(isGatewayArgv(["python", "gateway", "script.py"])).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("isOpenClawCommandArgv", () => {
|
||||
it("matches doctor across source, built, and installed entrypoints", () => {
|
||||
expect(isOpenClawCommandArgv(["node", "/srv/openclaw/openclaw.mjs", "doctor"], "doctor")).toBe(
|
||||
true,
|
||||
);
|
||||
expect(
|
||||
isOpenClawCommandArgv(["NODE", "C:\\OpenClaw\\DIST\\ENTRY.JS", "DOCTOR"], "doctor"),
|
||||
).toBe(true);
|
||||
expect(isOpenClawCommandArgv(["C:\\bin\\openclaw.cmd", "doctor", "--fix"], "doctor")).toBe(
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects other OpenClaw commands and unrelated doctor processes", () => {
|
||||
expect(isOpenClawCommandArgv(["openclaw", "gateway"], "doctor")).toBe(false);
|
||||
expect(isOpenClawCommandArgv(["python", "doctor", "worker.py"], "doctor")).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -6,33 +6,37 @@ function normalizeProcArg(arg: string): string {
|
||||
return normalizeLowercaseStringOrEmpty(arg.replaceAll("\\", "/"));
|
||||
}
|
||||
|
||||
const ENTRY_CANDIDATES = [
|
||||
"dist/index.js",
|
||||
"dist/entry.js",
|
||||
"openclaw.mjs",
|
||||
"scripts/run-node.mjs",
|
||||
"src/entry.ts",
|
||||
"src/index.ts",
|
||||
] as const;
|
||||
|
||||
export function parseProcCmdline(raw: string): string[] {
|
||||
return normalizeStringEntries(raw.split("\0"));
|
||||
}
|
||||
|
||||
export function isOpenClawCommandArgv(args: string[], command: string): boolean {
|
||||
const normalized = args.map(normalizeProcArg);
|
||||
const exe = (normalized[0] ?? "").replace(/\.(bat|cmd|exe)$/i, "");
|
||||
if (!normalized.includes(normalizeProcArg(command))) {
|
||||
return false;
|
||||
}
|
||||
if (normalized.some((arg) => ENTRY_CANDIDATES.some((entry) => arg.endsWith(entry)))) {
|
||||
return true;
|
||||
}
|
||||
return exe.endsWith("/openclaw") || exe === "openclaw";
|
||||
}
|
||||
|
||||
export function isGatewayArgv(args: string[], opts?: { allowGatewayBinary?: boolean }): boolean {
|
||||
const normalized = args.map(normalizeProcArg);
|
||||
const exe = (normalized[0] ?? "").replace(/\.(bat|cmd|exe)$/i, "");
|
||||
const isGatewayBinary = exe.endsWith("/openclaw-gateway") || exe === "openclaw-gateway";
|
||||
if (!normalized.includes("gateway")) {
|
||||
if (!isOpenClawCommandArgv(args, "gateway")) {
|
||||
return opts?.allowGatewayBinary === true && isGatewayBinary;
|
||||
}
|
||||
|
||||
const entryCandidates = [
|
||||
"dist/index.js",
|
||||
"dist/entry.js",
|
||||
"openclaw.mjs",
|
||||
"scripts/run-node.mjs",
|
||||
"src/entry.ts",
|
||||
"src/index.ts",
|
||||
];
|
||||
if (normalized.some((arg) => entryCandidates.some((entry) => arg.endsWith(entry)))) {
|
||||
return true;
|
||||
}
|
||||
|
||||
return (
|
||||
exe.endsWith("/openclaw") ||
|
||||
exe === "openclaw" ||
|
||||
(opts?.allowGatewayBinary === true && isGatewayBinary)
|
||||
);
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { getWindowsPowerShellExePath, getWindowsWmicExePath } from "./windows-install-roots.js";
|
||||
import { readWindowsProcessStartTimeSync } from "./windows-port-pids.js";
|
||||
|
||||
const spawnSyncMock = vi.hoisted(() => vi.fn());
|
||||
|
||||
vi.mock("node:child_process", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("node:child_process")>()),
|
||||
spawnSync: spawnSyncMock,
|
||||
}));
|
||||
|
||||
describe("readWindowsProcessStartTimeSync", () => {
|
||||
beforeEach(() => {
|
||||
spawnSyncMock.mockReset();
|
||||
});
|
||||
|
||||
it("reads an ISO creation time through PowerShell", () => {
|
||||
spawnSyncMock.mockReturnValueOnce({
|
||||
status: 0,
|
||||
stdout: "2026-07-13T07:20:49.1234567Z",
|
||||
} as never);
|
||||
|
||||
expect(readWindowsProcessStartTimeSync(123, 1000)).toBe(Date.parse("2026-07-13T07:20:49.123Z"));
|
||||
expect(spawnSyncMock.mock.calls[0]?.[0]).toBe(getWindowsPowerShellExePath());
|
||||
});
|
||||
|
||||
it("falls back to WMIC DMTF creation time output", () => {
|
||||
spawnSyncMock.mockReturnValueOnce({ status: 1, stdout: "" } as never).mockReturnValueOnce({
|
||||
status: 0,
|
||||
stdout: Buffer.from("CreationDate=20260713092049.123456+120\r\n"),
|
||||
} as never);
|
||||
|
||||
expect(readWindowsProcessStartTimeSync(456, 1000)).toBe(Date.parse("2026-07-13T07:20:49.123Z"));
|
||||
expect(spawnSyncMock.mock.calls[1]?.[0]).toBe(getWindowsWmicExePath());
|
||||
});
|
||||
|
||||
it("returns null when process creation time is unavailable", () => {
|
||||
spawnSyncMock
|
||||
.mockReturnValueOnce({ status: 1, stdout: "" } as never)
|
||||
.mockReturnValueOnce({ status: 1, stdout: Buffer.alloc(0) } as never);
|
||||
|
||||
expect(readWindowsProcessStartTimeSync(789, 1000)).toBeNull();
|
||||
expect(readWindowsProcessStartTimeSync(0, 1000)).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -1,4 +1,4 @@
|
||||
// Resolves Windows process ids that own listening ports.
|
||||
// Resolves Windows process identity and listening-port ownership.
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { normalizeLowercaseStringOrEmpty } from "@openclaw/normalization-core/string-coerce";
|
||||
import { normalizeStringEntries } from "@openclaw/normalization-core/string-normalization";
|
||||
@@ -81,7 +81,7 @@ export function readWindowsListeningPidsResultSync(
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Windows process-args reading (PowerShell → WMIC fallback)
|
||||
// Windows process identity reading (PowerShell → WMIC fallback)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function decodeWindowsProcessOutput(output: Buffer | string): string {
|
||||
@@ -105,6 +105,77 @@ function extractWindowsCommandLine(raw: Buffer | string): string | null {
|
||||
return lines.find((line) => normalizeLowercaseStringOrEmpty(line) !== "commandline") ?? null;
|
||||
}
|
||||
|
||||
function parseWindowsProcessStartTime(raw: Buffer | string): number | null {
|
||||
const lines = normalizeStringEntries(decodeWindowsProcessOutput(raw).split(/\r?\n/));
|
||||
const value =
|
||||
lines
|
||||
.find((line) => normalizeLowercaseStringOrEmpty(line).startsWith("creationdate="))
|
||||
?.slice("creationdate=".length)
|
||||
.trim() ??
|
||||
lines.find((line) => normalizeLowercaseStringOrEmpty(line) !== "creationdate") ??
|
||||
"";
|
||||
const parsedIso = Date.parse(value);
|
||||
if (Number.isFinite(parsedIso)) {
|
||||
return parsedIso;
|
||||
}
|
||||
const dmtf = value.match(/^(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})\.(\d{6})([+-])(\d{3})$/);
|
||||
if (!dmtf) {
|
||||
return null;
|
||||
}
|
||||
const [, year, month, day, hour, minute, second, microseconds, offsetSign, offset] = dmtf;
|
||||
const localTimeMs = Date.UTC(
|
||||
Number(year),
|
||||
Number(month) - 1,
|
||||
Number(day),
|
||||
Number(hour),
|
||||
Number(minute),
|
||||
Number(second),
|
||||
Math.floor(Number(microseconds) / 1000),
|
||||
);
|
||||
const offsetMs = Number(offset) * 60_000 * (offsetSign === "+" ? 1 : -1);
|
||||
return localTimeMs - offsetMs;
|
||||
}
|
||||
|
||||
/** Read a stable Windows process creation time for lock-owner identity checks. */
|
||||
export function readWindowsProcessStartTimeSync(
|
||||
pid: number,
|
||||
timeoutMs = DEFAULT_TIMEOUT_MS,
|
||||
): number | null {
|
||||
if (!Number.isInteger(pid) || pid <= 0) {
|
||||
return null;
|
||||
}
|
||||
const powershell = spawnSync(
|
||||
getWindowsPowerShellExePath(),
|
||||
[
|
||||
"-NoProfile",
|
||||
"-NonInteractive",
|
||||
"-Command",
|
||||
`$process = Get-CimInstance Win32_Process -Filter "ProcessId = ${pid}" -ErrorAction Stop; [Console]::Out.Write($process.CreationDate.ToUniversalTime().ToString("o"))`,
|
||||
],
|
||||
{
|
||||
encoding: "utf8",
|
||||
timeout: timeoutMs,
|
||||
windowsHide: true,
|
||||
},
|
||||
);
|
||||
if (!powershell.error && powershell.status === 0) {
|
||||
const startTime = parseWindowsProcessStartTime(powershell.stdout);
|
||||
if (startTime !== null) {
|
||||
return startTime;
|
||||
}
|
||||
}
|
||||
const wmic = spawnSync(
|
||||
getWindowsWmicExePath(),
|
||||
["process", "where", `ProcessId=${pid}`, "get", "CreationDate", "/value"],
|
||||
{
|
||||
timeout: timeoutMs,
|
||||
windowsHide: true,
|
||||
stdio: ["ignore", "pipe", "ignore"],
|
||||
},
|
||||
);
|
||||
return !wmic.error && wmic.status === 0 ? parseWindowsProcessStartTime(wmic.stdout) : null;
|
||||
}
|
||||
|
||||
export function readWindowsProcessArgsSync(
|
||||
pid: number,
|
||||
timeoutMs = DEFAULT_TIMEOUT_MS,
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
// SQLite CLI E2E tests cover startup and target ownership before offline maintenance.
|
||||
import { spawnSync } from "node:child_process";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { withTempHome } from "openclaw/plugin-sdk/test-env";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
closeOpenClawStateDatabase,
|
||||
openOpenClawStateDatabase,
|
||||
} from "../src/state/openclaw-state-db.js";
|
||||
|
||||
describe("SQLite CLI maintenance ownership", () => {
|
||||
it("compacts after full CLI startup without retaining a config-health database handle", async () => {
|
||||
await withTempHome(
|
||||
async (tempHome) => {
|
||||
const stateDir = path.join(tempHome, ".openclaw");
|
||||
const env: NodeJS.ProcessEnv = {
|
||||
...process.env,
|
||||
HOME: tempHome,
|
||||
USERPROFILE: tempHome,
|
||||
OPENCLAW_DISABLE_BUNDLED_PLUGINS: "1",
|
||||
OPENCLAW_STATE_DIR: stateDir,
|
||||
OPENCLAW_TEST_FAST: "1",
|
||||
};
|
||||
delete env.OPENCLAW_CONFIG_PATH;
|
||||
delete env.OPENCLAW_HOME;
|
||||
delete env.VITEST;
|
||||
|
||||
try {
|
||||
const database = openOpenClawStateDatabase({ env });
|
||||
database.db.exec(`
|
||||
CREATE TABLE compact_cli_payload (
|
||||
id INTEGER PRIMARY KEY,
|
||||
payload TEXT NOT NULL
|
||||
);
|
||||
BEGIN IMMEDIATE;
|
||||
`);
|
||||
const insert = database.db.prepare(
|
||||
"INSERT INTO compact_cli_payload (payload) VALUES (?)",
|
||||
);
|
||||
for (let index = 0; index < 256; index += 1) {
|
||||
insert.run(`${index}:${"x".repeat(8_192)}`);
|
||||
}
|
||||
database.db.exec(`
|
||||
COMMIT;
|
||||
DELETE FROM compact_cli_payload;
|
||||
PRAGMA wal_checkpoint(TRUNCATE);
|
||||
`);
|
||||
} finally {
|
||||
closeOpenClawStateDatabase();
|
||||
}
|
||||
|
||||
const entry = path.resolve(process.cwd(), "src/entry.ts");
|
||||
const result = spawnSync(
|
||||
process.execPath,
|
||||
["--import", "tsx", entry, "doctor", "--state-sqlite", "compact", "--json"],
|
||||
{
|
||||
cwd: process.cwd(),
|
||||
env,
|
||||
encoding: "utf8",
|
||||
timeout: 60_000,
|
||||
},
|
||||
);
|
||||
|
||||
expect(result.status, result.stderr || result.stdout).toBe(0);
|
||||
const report = JSON.parse(result.stdout.trim()) as {
|
||||
after: { autoVacuum: number; freelistPages: number };
|
||||
before: { freelistPages: number };
|
||||
integrityCheck: string;
|
||||
quickCheck: string;
|
||||
skipped: boolean;
|
||||
};
|
||||
expect(report).toMatchObject({
|
||||
after: {
|
||||
autoVacuum: 2,
|
||||
freelistPages: 0,
|
||||
},
|
||||
integrityCheck: "ok",
|
||||
quickCheck: "ok",
|
||||
skipped: false,
|
||||
});
|
||||
expect(report.before.freelistPages).toBeGreaterThan(0);
|
||||
expect(fs.existsSync(path.join(stateDir, "state", "openclaw.sqlite"))).toBe(true);
|
||||
},
|
||||
{ prefix: "openclaw-state-sqlite-cli-" },
|
||||
);
|
||||
}, 90_000);
|
||||
|
||||
it("rejects destructive explicit session stores outside the active state owner", async () => {
|
||||
await withTempHome(
|
||||
async (tempHome) => {
|
||||
const stateDir = path.join(tempHome, ".openclaw");
|
||||
const externalStorePath = path.join(
|
||||
tempHome,
|
||||
"external-state",
|
||||
"agents",
|
||||
"main",
|
||||
"sessions",
|
||||
"sessions.json",
|
||||
);
|
||||
const env: NodeJS.ProcessEnv = {
|
||||
...process.env,
|
||||
HOME: tempHome,
|
||||
USERPROFILE: tempHome,
|
||||
OPENCLAW_DISABLE_BUNDLED_PLUGINS: "1",
|
||||
OPENCLAW_STATE_DIR: stateDir,
|
||||
OPENCLAW_TEST_FAST: "1",
|
||||
};
|
||||
delete env.OPENCLAW_CONFIG_PATH;
|
||||
delete env.OPENCLAW_HOME;
|
||||
delete env.VITEST;
|
||||
|
||||
const entry = path.resolve(process.cwd(), "src/entry.ts");
|
||||
const result = spawnSync(
|
||||
process.execPath,
|
||||
[
|
||||
"--import",
|
||||
"tsx",
|
||||
entry,
|
||||
"doctor",
|
||||
"--session-sqlite",
|
||||
"compact",
|
||||
"--session-sqlite-store",
|
||||
externalStorePath,
|
||||
"--json",
|
||||
],
|
||||
{
|
||||
cwd: process.cwd(),
|
||||
env,
|
||||
encoding: "utf8",
|
||||
timeout: 60_000,
|
||||
},
|
||||
);
|
||||
|
||||
expect(result.status).not.toBe(0);
|
||||
expect(`${result.stderr}\n${result.stdout}`).toContain(
|
||||
"outside the active OpenClaw state directory",
|
||||
);
|
||||
expect(fs.existsSync(externalStorePath)).toBe(false);
|
||||
},
|
||||
{ prefix: "openclaw-session-sqlite-cli-" },
|
||||
);
|
||||
}, 90_000);
|
||||
|
||||
it("rejects hard-linked SQLite sidecars before destructive maintenance", async () => {
|
||||
await withTempHome(
|
||||
async (tempHome) => {
|
||||
const stateDir = path.join(tempHome, ".openclaw");
|
||||
const storePath = path.join(stateDir, "agents", "main", "sessions", "sessions.json");
|
||||
const sqlitePath = path.join(stateDir, "agents", "main", "agent", "openclaw-agent.sqlite");
|
||||
const externalWalPath = path.join(tempHome, "external-state", "openclaw-agent.sqlite-wal");
|
||||
fs.mkdirSync(path.dirname(storePath), { recursive: true });
|
||||
fs.mkdirSync(path.dirname(sqlitePath), { recursive: true });
|
||||
fs.mkdirSync(path.dirname(externalWalPath), { recursive: true });
|
||||
fs.writeFileSync(storePath, "{}\n", "utf8");
|
||||
fs.writeFileSync(externalWalPath, "external wal\n", "utf8");
|
||||
fs.linkSync(externalWalPath, `${sqlitePath}-wal`);
|
||||
const env: NodeJS.ProcessEnv = {
|
||||
...process.env,
|
||||
HOME: tempHome,
|
||||
USERPROFILE: tempHome,
|
||||
OPENCLAW_DISABLE_BUNDLED_PLUGINS: "1",
|
||||
OPENCLAW_STATE_DIR: stateDir,
|
||||
OPENCLAW_TEST_FAST: "1",
|
||||
};
|
||||
delete env.OPENCLAW_CONFIG_PATH;
|
||||
delete env.OPENCLAW_HOME;
|
||||
delete env.VITEST;
|
||||
|
||||
const entry = path.resolve(process.cwd(), "src/entry.ts");
|
||||
const result = spawnSync(
|
||||
process.execPath,
|
||||
[
|
||||
"--import",
|
||||
"tsx",
|
||||
entry,
|
||||
"doctor",
|
||||
"--session-sqlite",
|
||||
"compact",
|
||||
"--session-sqlite-store",
|
||||
storePath,
|
||||
"--json",
|
||||
],
|
||||
{
|
||||
cwd: process.cwd(),
|
||||
env,
|
||||
encoding: "utf8",
|
||||
timeout: 60_000,
|
||||
},
|
||||
);
|
||||
|
||||
expect(result.status).not.toBe(0);
|
||||
expect(`${result.stderr}\n${result.stdout}`).toContain("hard-linked path");
|
||||
expect(fs.readFileSync(externalWalPath, "utf8")).toBe("external wal\n");
|
||||
},
|
||||
{ prefix: "openclaw-session-sqlite-sidecar-cli-" },
|
||||
);
|
||||
}, 90_000);
|
||||
});
|
||||
Reference in New Issue
Block a user