feat(backup): add verified SQLite snapshots (#105718)

* feat(backup): expose verified SQLite snapshots

Co-authored-by: Gio Della-Libera <giodl73@gmail.com>

* feat(sqlite): add verified snapshot repository (#105525)

Co-authored-by: Gio Della-Libera <giodl73@gmail.com>

* test(sqlite): use canonical snapshot schemas

* fix(backup): restrict SQLite snapshot roles

* fix(sqlite): harden snapshot staging paths

Co-authored-by: Gio Della-Libera <giodl73@gmail.com>

* fix(sqlite): pin snapshot repository paths

Co-authored-by: Gio Della-Libera <giodl73@gmail.com>

* fix(sqlite): secure Windows snapshot staging

Co-authored-by: Gio Della-Libera <giodl73@gmail.com>

* fix(sqlite): validate macOS ACL captures

* test(sqlite): preserve stat uid type

* style(sqlite): satisfy snapshot lint

---------

Co-authored-by: Gio Della-Libera <giodl73@gmail.com>
This commit is contained in:
Vincent Koc
2026-07-13 06:27:14 +08:00
committed by GitHub
co-authored by Gio Della-Libera
parent 95f142bf0c
commit 2f25e9cba3
14 changed files with 4508 additions and 8 deletions
+1
View File
@@ -6,6 +6,7 @@ Docs: https://docs.openclaw.ai
### Changes
- **SQLite snapshots:** add `openclaw backup sqlite create|list|verify|restore` for compact, verified global and per-agent database artifacts with fresh-target-only restore. (#94805) Thanks @giodl73-repo.
- **GPT-5.6 Ultra and runtime switching:** support Sol, Terra, and Luna across OpenClaw and Codex engines; keep model, runtime, and thinking selection atomic through `/model` and fallback; and add live matrix coverage for both harnesses. (#98021) Thanks @anyech.
- **OpenAI GPT-5.6 defaults:** use `openai/gpt-5.6` (Sol alias) for fresh API-key setup and exact `openai/gpt-5.6-sol` for fresh Codex/OAuth setup, while preserving existing primaries, fallbacks, aliases, and explicit GPT-5.5 selections. (#103234)
- **Meta provider:** add bundled `muse-spark-1.1` model support with Responses API streaming, tool calls, encrypted reasoning replay, onboarding, and standalone npm/ClawHub distribution. (#102873) Thanks @HamidShojanazeri.
+47 -1
View File
@@ -1,7 +1,8 @@
---
summary: "CLI reference for `openclaw backup` (create local backup archives)"
summary: "CLI reference for `openclaw backup` (archives and SQLite snapshots)"
read_when:
- You want a first-class backup archive for local OpenClaw state
- You need a compact, verified snapshot of one OpenClaw SQLite database
- You want to preview which paths would be included before reset or uninstall
title: "Backup"
---
@@ -18,6 +19,12 @@ openclaw backup create --verify
openclaw backup create --no-include-workspace
openclaw backup create --only-config
openclaw backup verify ./2026-03-09T08-00-00.000+08-00-openclaw-backup.tar.gz
openclaw backup sqlite create --global --repository ~/Backups/openclaw-sqlite
openclaw backup sqlite create --agent main --repository ~/Backups/openclaw-sqlite
openclaw backup sqlite list --repository ~/Backups/openclaw-sqlite
openclaw backup sqlite verify ~/Backups/openclaw-sqlite/<snapshot-id>
openclaw backup sqlite verify ~/Backups/openclaw-sqlite/<snapshot-id> --scratch ~/Private/openclaw-scratch
openclaw backup sqlite restore ~/Backups/openclaw-sqlite/<snapshot-id> --target ./restored/openclaw.sqlite
```
## Notes
@@ -28,6 +35,45 @@ openclaw backup verify ./2026-03-09T08-00-00.000+08-00-openclaw-backup.tar.gz
- `openclaw backup verify <archive>` checks that the archive contains exactly one root manifest, rejects traversal-style archive paths and SQLite sidecars, confirms every manifest-declared payload exists, validates every SQLite snapshot's file shape, and runs full integrity and role checks on canonical OpenClaw databases. Dedicated plugin schemas remain opaque because they may require owner-defined SQLite capabilities. `openclaw backup create --verify` runs that validation immediately after writing the archive.
- `openclaw backup create --only-config` backs up just the active JSON config file.
## SQLite snapshots
Use `openclaw backup sqlite` when you need a portable artifact for one OpenClaw-owned SQLite database instead of a broad state archive.
Snapshot creation accepts exactly one named source:
| Command | Database |
| --------------------------------------------------------------- | ---------------------- |
| `openclaw backup sqlite create --global --repository <dir>` | Shared OpenClaw state |
| `openclaw backup sqlite create --agent <id> --repository <dir>` | One per-agent database |
The repository contains one directory per committed snapshot. Each snapshot directory contains exactly:
- `manifest.json`
- `database.sqlite`
Snapshot creation verifies the live database before reading it, uses SQLite `VACUUM INTO` to capture committed WAL state into a compact database, verifies the generated database again, and publishes the completed directory without overwriting existing paths. Global snapshots remove transient delivery queue rows and compact again so deleted queue payloads are not retained in free pages.
Do not copy live `.sqlite`, `-wal`, `-shm`, or `-journal` files as a portability artifact. Copy only completed snapshot directories.
SQLite snapshots can contain auth profiles, session state, plugin state, and other sensitive records. Protect repositories with the same permissions, encryption, retention policy, and destination restrictions as the live OpenClaw state directory.
### Verify and restore
```bash
openclaw backup sqlite verify <snapshot-directory>
openclaw backup sqlite restore <snapshot-directory> --target <new-database-path>
```
Verification checks the strict manifest shape, artifact size and SHA-256, SQLite integrity, foreign keys, schema version, database role and owner, and OpenClaw-owned index definitions.
Verification validates a private content-pinned copy so pathname races cannot swap the bytes SQLite inspects. By default, that temporary copy is created beside the snapshot repository and removed before the command returns. The staging root and its ancestor chain must prevent other users from replacing it. POSIX roots must be current-user-owned and not group/world writable; sticky ancestors such as `/tmp` are accepted for user-owned children. macOS ACL grants that expose or make staging replaceable are rejected. Windows roots and ancestors must be owned by the current user or a trusted OS principal, with ACLs that deny untrusted staging access. For a read-only mount or network share, pass `--scratch <existing-private-directory>` on storage with equivalent encryption and destination controls.
Snapshot creation applies the same owner, ACL, ancestor, and path-identity checks to the repository before staging or publishing database bytes.
Restore repeats verification and writes only to a fresh target. It refuses an existing target, `-wal`, `-shm`, or `-journal` sidecar and never performs an in-place replacement of a live OpenClaw database. The target parent has the same path-security requirements as verification scratch. Activating a restored database remains an explicit offline operator step.
Snapshot repositories are local directories. Scheduling, upload, retention, incremental WAL bundles, failover, and restore-on-boot behavior are intentionally outside this command.
## What gets backed up
`openclaw backup create` plans sources from your local OpenClaw install:
+1 -1
View File
@@ -34,7 +34,7 @@ const coreCliCommandCatalog = defineCommandDescriptorCatalog([
},
{
name: "backup",
description: "Create and verify local backup archives for OpenClaw state",
description: "Create and verify backup archives and SQLite snapshots",
hasSubcommands: true,
},
{
+19
View File
@@ -719,6 +719,25 @@ describe("registerPreActionHooks", () => {
expect(ensurePluginRegistryLoadedMock).not.toHaveBeenCalled();
});
it("bypasses config guard for SQLite snapshot recovery commands", async () => {
await runPreAction({
parseArgv: ["backup", "sqlite", "restore"],
processArgv: [
"node",
"openclaw",
"backup",
"sqlite",
"restore",
"/tmp/snapshot",
"--target",
"/tmp/restore.sqlite",
],
});
expect(ensureConfigReadyMock).not.toHaveBeenCalled();
expect(ensurePluginRegistryLoadedMock).not.toHaveBeenCalled();
});
it("routes logs to stderr during plugin loading in --json mode and restores after", async () => {
let stderrDuringPluginLoad = false;
ensurePluginRegistryLoadedMock.mockImplementation(() => {
+106
View File
@@ -5,6 +5,10 @@ import { registerBackupCommand } from "./register.backup.js";
const mocks = vi.hoisted(() => ({
backupCreateCommand: vi.fn(),
backupSqliteCreateCommand: vi.fn(),
backupSqliteListCommand: vi.fn(),
backupSqliteRestoreCommand: vi.fn(),
backupSqliteVerifyCommand: vi.fn(),
backupVerifyCommand: vi.fn(),
runtime: {
log: vi.fn(),
@@ -14,6 +18,10 @@ const mocks = vi.hoisted(() => ({
}));
const backupCreateCommand = mocks.backupCreateCommand;
const backupSqliteCreateCommand = mocks.backupSqliteCreateCommand;
const backupSqliteListCommand = mocks.backupSqliteListCommand;
const backupSqliteRestoreCommand = mocks.backupSqliteRestoreCommand;
const backupSqliteVerifyCommand = mocks.backupSqliteVerifyCommand;
const backupVerifyCommand = mocks.backupVerifyCommand;
const runtime = mocks.runtime;
@@ -25,6 +33,13 @@ vi.mock("../../commands/backup-verify.js", () => ({
backupVerifyCommand: mocks.backupVerifyCommand,
}));
vi.mock("../../commands/backup-sqlite.js", () => ({
backupSqliteCreateCommand: mocks.backupSqliteCreateCommand,
backupSqliteListCommand: mocks.backupSqliteListCommand,
backupSqliteRestoreCommand: mocks.backupSqliteRestoreCommand,
backupSqliteVerifyCommand: mocks.backupSqliteVerifyCommand,
}));
vi.mock("../../runtime.js", () => ({
defaultRuntime: mocks.runtime,
}));
@@ -39,6 +54,10 @@ describe("registerBackupCommand", () => {
beforeEach(() => {
vi.clearAllMocks();
backupCreateCommand.mockResolvedValue(undefined);
backupSqliteCreateCommand.mockResolvedValue(undefined);
backupSqliteListCommand.mockResolvedValue(undefined);
backupSqliteRestoreCommand.mockResolvedValue(undefined);
backupSqliteVerifyCommand.mockResolvedValue(undefined);
backupVerifyCommand.mockResolvedValue(undefined);
});
@@ -93,4 +112,91 @@ describe("registerBackupCommand", () => {
expect(options.archive).toBe("/tmp/openclaw-backup.tar.gz");
expect(options.json).toBe(true);
});
it("registers the SQLite snapshot command group", () => {
const program = new Command();
registerBackupCommand(program);
const backup = program.commands.find((command) => command.name() === "backup");
const sqlite = backup?.commands.find((command) => command.name() === "sqlite");
expect(sqlite?.commands.map((command) => command.name()).toSorted()).toEqual([
"create",
"list",
"restore",
"verify",
]);
});
it("runs SQLite snapshot create for named OpenClaw databases", async () => {
await runCli([
"backup",
"sqlite",
"create",
"--global",
"--repository",
"/tmp/snapshots",
"--json",
]);
expect(backupSqliteCreateCommand).toHaveBeenCalledWith(runtime, {
global: true,
agent: undefined,
repository: "/tmp/snapshots",
json: true,
});
await runCli([
"backup",
"sqlite",
"create",
"--agent",
"main",
"--repository",
"/tmp/snapshots",
]);
expect(backupSqliteCreateCommand).toHaveBeenLastCalledWith(runtime, {
global: false,
agent: "main",
repository: "/tmp/snapshots",
json: false,
});
});
it("runs SQLite snapshot list, verify, and restore", async () => {
await runCli(["backup", "sqlite", "list", "--repository", "/tmp/snapshots", "--json"]);
expect(backupSqliteListCommand).toHaveBeenCalledWith(runtime, {
repository: "/tmp/snapshots",
json: true,
});
await runCli([
"backup",
"sqlite",
"verify",
"/tmp/snapshots/one",
"--scratch",
"/tmp/private-scratch",
"--json",
]);
expect(backupSqliteVerifyCommand).toHaveBeenCalledWith(runtime, "/tmp/snapshots/one", {
scratch: "/tmp/private-scratch",
json: true,
});
await runCli([
"backup",
"sqlite",
"restore",
"/tmp/snapshots/one",
"--target",
"/tmp/restored.sqlite",
"--json",
]);
expect(backupSqliteRestoreCommand).toHaveBeenCalledWith(runtime, "/tmp/snapshots/one", {
target: "/tmp/restored.sqlite",
json: true,
});
});
});
+93 -1
View File
@@ -2,6 +2,12 @@
import type { Command } from "commander";
import { formatDocsLink } from "../../../packages/terminal-core/src/links.js";
import { theme } from "../../../packages/terminal-core/src/theme.js";
import {
backupSqliteCreateCommand,
backupSqliteListCommand,
backupSqliteRestoreCommand,
backupSqliteVerifyCommand,
} from "../../commands/backup-sqlite.js";
import { backupVerifyCommand } from "../../commands/backup-verify.js";
import { backupCreateCommand } from "../../commands/backup.js";
import { defaultRuntime } from "../../runtime.js";
@@ -12,7 +18,7 @@ import { formatHelpExamples } from "../help-format.js";
export function registerBackupCommand(program: Command) {
const backup = program
.command("backup")
.description("Create and verify local backup archives for OpenClaw state")
.description("Create and verify backup archives and SQLite snapshots")
.addHelpText(
"after",
() =>
@@ -91,4 +97,90 @@ export function registerBackupCommand(program: Command) {
});
});
});
registerBackupSqliteCommands(backup);
}
function registerBackupSqliteCommands(backup: Command): void {
const sqlite = backup
.command("sqlite")
.description("Create, list, verify, and restore SQLite snapshots")
.action(() => {
sqlite.outputHelp();
process.exitCode = 1;
});
sqlite
.command("create")
.description("Create a compact, verified snapshot of an OpenClaw SQLite database")
.option("--global", "Snapshot the shared OpenClaw state database", false)
.option("--agent <id>", "Snapshot one per-agent OpenClaw database")
.requiredOption("--repository <path>", "Snapshot repository directory")
.option("--json", "Output JSON", false)
.addHelpText(
"after",
() =>
`\n${theme.heading("Examples:")}\n${formatHelpExamples([
[
"openclaw backup sqlite create --global --repository ~/Backups/openclaw-sqlite",
"Snapshot the shared state database.",
],
[
"openclaw backup sqlite create --agent main --repository ~/Backups/openclaw-sqlite",
"Snapshot the main agent database.",
],
])}`,
)
.action(async (opts) => {
await runCommandWithRuntime(defaultRuntime, async () => {
await backupSqliteCreateCommand(defaultRuntime, {
global: Boolean(opts.global),
agent: opts.agent as string | undefined,
repository: opts.repository as string,
json: Boolean(opts.json),
});
});
});
sqlite
.command("list")
.description("List committed snapshots in a repository")
.requiredOption("--repository <path>", "Snapshot repository directory")
.option("--json", "Output JSON", false)
.action(async (opts) => {
await runCommandWithRuntime(defaultRuntime, async () => {
await backupSqliteListCommand(defaultRuntime, {
repository: opts.repository as string,
json: Boolean(opts.json),
});
});
});
sqlite
.command("verify <snapshot>")
.description("Verify a snapshot manifest, artifact hash, SQLite integrity, and database owner")
.option("--scratch <path>", "Existing private directory for verification copies")
.option("--json", "Output JSON", false)
.action(async (snapshot, opts) => {
await runCommandWithRuntime(defaultRuntime, async () => {
await backupSqliteVerifyCommand(defaultRuntime, snapshot as string, {
scratch: opts.scratch as string | undefined,
json: Boolean(opts.json),
});
});
});
sqlite
.command("restore <snapshot>")
.description("Restore a verified snapshot to a new SQLite database path")
.requiredOption("--target <path>", "Fresh target path; existing files and sidecars are refused")
.option("--json", "Output JSON", false)
.action(async (snapshot, opts) => {
await runCommandWithRuntime(defaultRuntime, async () => {
await backupSqliteRestoreCommand(defaultRuntime, snapshot as string, {
target: opts.target as string,
json: Boolean(opts.json),
});
});
});
}
+290
View File
@@ -0,0 +1,290 @@
import fs from "node:fs/promises";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { useAutoCleanupTempDirTracker } from "../../test/helpers/temp-dir.js";
import { requireNodeSqlite } from "../infra/node-sqlite.js";
import type { RuntimeEnv } from "../runtime.js";
import { createLocalSqliteSnapshotProvider } from "../snapshot/local-repository.js";
import { OPENCLAW_AGENT_SCHEMA_VERSION } from "../state/openclaw-agent-db.js";
import { resolveOpenClawAgentSqlitePath } from "../state/openclaw-agent-db.paths.js";
import { OPENCLAW_AGENT_SCHEMA_SQL } from "../state/openclaw-agent-schema.generated.js";
import { OPENCLAW_STATE_SCHEMA_VERSION } from "../state/openclaw-state-db.js";
import { resolveOpenClawStateSqlitePath } from "../state/openclaw-state-db.paths.js";
import { OPENCLAW_STATE_SCHEMA_SQL } from "../state/openclaw-state-schema.generated.js";
import {
backupSqliteCreateCommand,
backupSqliteListCommand,
backupSqliteRestoreCommand,
backupSqliteVerifyCommand,
} from "./backup-sqlite.js";
const tempDirs = useAutoCleanupTempDirTracker(afterEach);
let previousStateDir: string | undefined;
beforeEach(() => {
previousStateDir = process.env.OPENCLAW_STATE_DIR;
});
afterEach(() => {
if (previousStateDir === undefined) {
delete process.env.OPENCLAW_STATE_DIR;
} else {
process.env.OPENCLAW_STATE_DIR = previousStateDir;
}
});
function createGlobalDatabase(databasePath: string): void {
const sqlite = requireNodeSqlite();
const database = new sqlite.DatabaseSync(databasePath);
try {
database.exec(`
PRAGMA journal_mode = WAL;
PRAGMA wal_autocheckpoint = 0;
${OPENCLAW_STATE_SCHEMA_SQL}
PRAGMA user_version = ${OPENCLAW_STATE_SCHEMA_VERSION};
CREATE TABLE durable_entries (
id INTEGER PRIMARY KEY,
value TEXT NOT NULL
);
`);
database
.prepare(
`
INSERT INTO schema_meta (
meta_key,
role,
schema_version,
agent_id,
app_version,
created_at,
updated_at
) VALUES ('primary', 'global', ?, NULL, NULL, 1, 1)
`,
)
.run(OPENCLAW_STATE_SCHEMA_VERSION);
database
.prepare(
`
INSERT INTO delivery_queue_entries (
queue_name,
id,
status,
entry_json,
enqueued_at,
updated_at
) VALUES ('delivery', 'queued', 'pending', ?, 1, 1)
`,
)
.run('{"payload":"do-not-restore"}');
database.prepare("INSERT INTO durable_entries (value) VALUES (?)").run("checkpointed");
database.exec("PRAGMA wal_checkpoint(TRUNCATE);");
database.prepare("INSERT INTO durable_entries (value) VALUES (?)").run("committed-in-wal");
} finally {
database.close();
}
}
function createAgentDatabase(databasePath: string, agentId: string): void {
const sqlite = requireNodeSqlite();
const database = new sqlite.DatabaseSync(databasePath);
try {
database.exec(`
${OPENCLAW_AGENT_SCHEMA_SQL}
PRAGMA user_version = ${OPENCLAW_AGENT_SCHEMA_VERSION};
CREATE TABLE durable_entries (
id INTEGER PRIMARY KEY,
value TEXT NOT NULL
);
`);
database
.prepare(
`
INSERT INTO schema_meta (
meta_key,
role,
schema_version,
agent_id,
app_version,
created_at,
updated_at
) VALUES ('primary', 'agent', ?, ?, NULL, 1, 1)
`,
)
.run(OPENCLAW_AGENT_SCHEMA_VERSION, agentId);
database.prepare("INSERT INTO durable_entries (value) VALUES (?)").run("agent-state");
} finally {
database.close();
}
}
describe("SQLite backup commands", () => {
it("creates, lists, verifies, and fresh-restores the global database", async () => {
const tempDir = tempDirs.make("openclaw-backup-sqlite-");
const stateDir = path.join(tempDir, "state");
const repositoryPath = path.join(tempDir, "snapshots");
const scratchPath = path.join(tempDir, "scratch");
const restorePath = path.join(tempDir, "restore", "openclaw.sqlite");
process.env.OPENCLAW_STATE_DIR = stateDir;
const databasePath = resolveOpenClawStateSqlitePath();
await fs.mkdir(path.dirname(databasePath), { recursive: true });
await fs.mkdir(scratchPath, { mode: 0o700 });
await fs.chmod(scratchPath, 0o700);
createGlobalDatabase(databasePath);
const runtime = createRuntimeCapture();
const created = await backupSqliteCreateCommand(runtime, {
global: true,
repository: repositoryPath,
json: true,
});
expect(created.manifest.database).toMatchObject({
role: "global",
basename: "openclaw.sqlite",
userVersion: OPENCLAW_STATE_SCHEMA_VERSION,
});
expect(JSON.parse(runtime.logs.shift() ?? "{}")).toEqual(created);
const listed = await backupSqliteListCommand(runtime, {
repository: repositoryPath,
json: true,
});
expect(listed.snapshots).toHaveLength(1);
expect(listed.snapshots[0]?.manifest.snapshotId).toBe(created.manifest.snapshotId);
const verified = await backupSqliteVerifyCommand(runtime, created.snapshotPath, {
scratch: scratchPath,
json: true,
});
expect(verified.manifest).toEqual(created.manifest);
await expect(fs.readdir(scratchPath)).resolves.toEqual([]);
const restored = await backupSqliteRestoreCommand(runtime, created.snapshotPath, {
target: restorePath,
json: true,
});
expect(restored).toMatchObject({
ok: true,
snapshotPath: created.snapshotPath,
targetPath: restorePath,
});
expect(runtime.errors).toEqual([]);
const sqlite = requireNodeSqlite();
const restoredDatabase = new sqlite.DatabaseSync(restorePath, { readOnly: true });
try {
expect(
restoredDatabase.prepare("SELECT value FROM durable_entries ORDER BY id").all(),
).toEqual([{ value: "checkpointed" }, { value: "committed-in-wal" }]);
expect(
restoredDatabase.prepare("SELECT COUNT(*) AS count FROM delivery_queue_entries").get(),
).toEqual({ count: 0 });
} finally {
restoredDatabase.close();
}
});
it("creates a snapshot for a normalized per-agent database", async () => {
const tempDir = tempDirs.make("openclaw-backup-sqlite-");
const stateDir = path.join(tempDir, "state");
const repositoryPath = path.join(tempDir, "snapshots");
process.env.OPENCLAW_STATE_DIR = stateDir;
const databasePath = resolveOpenClawAgentSqlitePath({ agentId: "ops-team" });
await fs.mkdir(path.dirname(databasePath), { recursive: true });
createAgentDatabase(databasePath, "ops-team");
const runtime = createRuntimeCapture();
const created = await backupSqliteCreateCommand(runtime, {
agent: "Ops Team",
repository: repositoryPath,
});
expect(created.manifest.database).toEqual({
role: "agent",
agentId: "ops-team",
basename: "openclaw-agent.sqlite",
userVersion: OPENCLAW_AGENT_SCHEMA_VERSION,
});
expect(runtime.logs).toEqual([expect.stringContaining("Database: agent:ops-team")]);
expect(runtime.errors).toEqual([]);
});
it("requires exactly one named OpenClaw database source", async () => {
const runtime = createRuntimeCapture();
await expect(
backupSqliteCreateCommand(runtime, { repository: "/tmp/snapshots" }),
).rejects.toThrow("Choose a SQLite snapshot source");
await expect(
backupSqliteCreateCommand(runtime, {
global: true,
agent: "main",
repository: "/tmp/snapshots",
}),
).rejects.toThrow("Choose exactly one SQLite snapshot source");
});
it("requires repository, snapshot, and restore target paths", async () => {
const runtime = createRuntimeCapture();
await expect(backupSqliteCreateCommand(runtime, { global: true })).rejects.toThrow(
"Missing required --repository value",
);
await expect(backupSqliteVerifyCommand(runtime, " ", {})).rejects.toThrow(
"Missing required <snapshot> value",
);
await expect(backupSqliteRestoreCommand(runtime, "/tmp/snapshot", {})).rejects.toThrow(
"Missing required --target value",
);
});
it("rejects generic provider artifacts before verify or restore", async () => {
const tempDir = tempDirs.make("openclaw-backup-sqlite-");
const databasePath = path.join(tempDir, "generic.sqlite");
const repositoryPath = path.join(tempDir, "snapshots");
const restorePath = path.join(tempDir, "restore", "generic.sqlite");
const sqlite = requireNodeSqlite();
const database = new sqlite.DatabaseSync(databasePath);
try {
database.exec("CREATE TABLE entries (id INTEGER PRIMARY KEY);");
} finally {
database.close();
}
const snapshot = await createLocalSqliteSnapshotProvider({ repositoryPath }).create({
path: databasePath,
identity: { role: "generic", id: "generic-test" },
});
const runtime = createRuntimeCapture();
await expect(backupSqliteListCommand(runtime, { repository: repositoryPath })).rejects.toThrow(
/database role generic is not allowed/u,
);
await expect(backupSqliteVerifyCommand(runtime, snapshot.ref.path, {})).rejects.toThrow(
/database role generic is not allowed/u,
);
await expect(
backupSqliteRestoreCommand(runtime, snapshot.ref.path, { target: restorePath }),
).rejects.toThrow(/database role generic is not allowed/u);
await expect(fs.access(restorePath)).rejects.toMatchObject({ code: "ENOENT" });
});
});
function createRuntimeCapture(): RuntimeEnv & {
logs: string[];
errors: string[];
} {
const logs: string[] = [];
const errors: string[] = [];
return {
logs,
errors,
log(value) {
logs.push(String(value));
},
error(value) {
errors.push(String(value));
},
exit(code) {
throw new Error(`exit ${code}`);
},
};
}
+269
View File
@@ -0,0 +1,269 @@
import fs from "node:fs/promises";
import path from "node:path";
import { normalizeAgentId } from "../routing/session-key.js";
import { type RuntimeEnv, writeRuntimeJson } from "../runtime.js";
import { createLocalSqliteSnapshotProvider } from "../snapshot/local-repository.js";
import type {
SnapshotDatabaseManifest,
SnapshotManifest,
SnapshotRef,
SnapshotSummary,
} from "../snapshot/snapshot-provider.js";
import { resolveOpenClawAgentSqlitePath } from "../state/openclaw-agent-db.paths.js";
import { resolveOpenClawStateSqlitePath } from "../state/openclaw-state-db.paths.js";
import { resolveUserPath, shortenHomePath } from "../utils.js";
export type BackupSqliteCreateOptions = {
global?: boolean;
agent?: string;
repository?: string;
json?: boolean;
};
export type BackupSqliteRepositoryOptions = {
repository?: string;
json?: boolean;
};
export type BackupSqliteJsonOptions = {
json?: boolean;
};
export type BackupSqliteVerifyOptions = BackupSqliteJsonOptions & {
scratch?: string;
};
export type BackupSqliteRestoreOptions = BackupSqliteJsonOptions & {
target?: string;
};
type BackupSqliteCreateResult = {
ok: true;
snapshotPath: string;
manifest: SnapshotManifest;
};
type BackupSqliteListResult = {
ok: true;
repositoryPath: string;
snapshots: SnapshotSummary[];
};
type BackupSqliteVerifyResult = {
ok: true;
snapshotPath: string;
manifest: SnapshotManifest;
};
type BackupSqliteRestoreResult = BackupSqliteVerifyResult & {
targetPath: string;
};
type ResolvedSnapshotDatabase = {
path: string;
identity: { role: "global" } | { role: "agent"; agentId: string };
};
const OPENCLAW_SNAPSHOT_READ_OPTIONS = {
allowedDatabaseRoles: ["global", "agent"],
} as const;
export async function backupSqliteCreateCommand(
runtime: RuntimeEnv,
options: BackupSqliteCreateOptions,
): Promise<BackupSqliteCreateResult> {
const repositoryPath = resolveRequiredPath(options.repository, "--repository");
const database = await resolveSnapshotDatabase(options);
const result = await createLocalSqliteSnapshotProvider({ repositoryPath }).create(database);
const report: BackupSqliteCreateResult = {
ok: true,
snapshotPath: result.ref.path,
manifest: result.manifest,
};
writeCreateResult(runtime, options, report);
return report;
}
export async function backupSqliteListCommand(
runtime: RuntimeEnv,
options: BackupSqliteRepositoryOptions,
): Promise<BackupSqliteListResult> {
const repositoryPath = resolveRequiredPath(options.repository, "--repository");
const snapshots = await createLocalSqliteSnapshotProvider({
repositoryPath,
...OPENCLAW_SNAPSHOT_READ_OPTIONS,
}).list();
const report: BackupSqliteListResult = {
ok: true,
repositoryPath,
snapshots,
};
writeListResult(runtime, options, report);
return report;
}
export async function backupSqliteVerifyCommand(
runtime: RuntimeEnv,
snapshot: string,
options: BackupSqliteVerifyOptions,
): Promise<BackupSqliteVerifyResult> {
const resolved = resolveSnapshot(snapshot, options.scratch);
const verified = await resolved.provider.verify(resolved.ref);
const report: BackupSqliteVerifyResult = {
ok: true,
snapshotPath: resolved.ref.path,
manifest: verified.manifest,
};
writeVerifyResult(runtime, options, report);
return report;
}
export async function backupSqliteRestoreCommand(
runtime: RuntimeEnv,
snapshot: string,
options: BackupSqliteRestoreOptions,
): Promise<BackupSqliteRestoreResult> {
const resolved = resolveSnapshot(snapshot);
const targetPath = resolveRequiredPath(options.target, "--target");
const restored = await resolved.provider.restoreFresh(resolved.ref, targetPath);
const report: BackupSqliteRestoreResult = {
ok: true,
snapshotPath: resolved.ref.path,
targetPath,
manifest: restored.manifest,
};
writeRestoreResult(runtime, options, report);
return report;
}
async function resolveSnapshotDatabase(
options: BackupSqliteCreateOptions,
): Promise<ResolvedSnapshotDatabase> {
const rawAgentId = options.agent?.trim();
if (options.global === true && rawAgentId) {
throw new Error("Choose exactly one SQLite snapshot source: --global or --agent <id>.");
}
if (options.global !== true && !rawAgentId) {
throw new Error("Choose a SQLite snapshot source: --global or --agent <id>.");
}
if (options.global === true) {
return {
path: await fs.realpath(resolveOpenClawStateSqlitePath()),
identity: { role: "global" },
};
}
const agentId = normalizeAgentId(rawAgentId);
return {
path: await fs.realpath(resolveOpenClawAgentSqlitePath({ agentId })),
identity: { role: "agent", agentId },
};
}
function resolveSnapshot(
snapshot: string,
scratch?: string,
): {
provider: ReturnType<typeof createLocalSqliteSnapshotProvider>;
ref: SnapshotRef;
} {
const snapshotPath = resolveRequiredPath(snapshot, "<snapshot>");
const repositoryPath = path.dirname(snapshotPath);
const validationRootPath = scratch
? resolveRequiredPath(scratch, "--scratch")
: path.dirname(repositoryPath);
return {
provider: createLocalSqliteSnapshotProvider({
repositoryPath,
validationRootPath,
...OPENCLAW_SNAPSHOT_READ_OPTIONS,
}),
ref: { path: snapshotPath },
};
}
function resolveRequiredPath(value: string | undefined, label: string): string {
const trimmed = value?.trim();
if (!trimmed) {
throw new Error(`Missing required ${label} value.`);
}
return path.resolve(resolveUserPath(trimmed));
}
function formatDatabaseIdentity(database: SnapshotDatabaseManifest): string {
if (database.role === "global") {
return "global";
}
if (database.role === "agent") {
return `agent:${database.agentId}`;
}
return database.id;
}
function writeCreateResult(
runtime: RuntimeEnv,
options: BackupSqliteJsonOptions,
report: BackupSqliteCreateResult,
): void {
if (options.json) {
writeRuntimeJson(runtime, report);
return;
}
runtime.log(
[
`SQLite snapshot created: ${shortenHomePath(report.snapshotPath)}`,
`Database: ${formatDatabaseIdentity(report.manifest.database)}`,
`Size: ${report.manifest.artifact.sizeBytes} bytes`,
].join("\n"),
);
}
function writeListResult(
runtime: RuntimeEnv,
options: BackupSqliteJsonOptions,
report: BackupSqliteListResult,
): void {
if (options.json) {
writeRuntimeJson(runtime, report);
return;
}
if (report.snapshots.length === 0) {
runtime.log(`No SQLite snapshots in ${shortenHomePath(report.repositoryPath)}.`);
return;
}
runtime.log(
report.snapshots
.map(
(snapshot) =>
`${snapshot.manifest.createdAt} ${formatDatabaseIdentity(snapshot.manifest.database)} ${snapshot.manifest.artifact.sizeBytes} bytes ${shortenHomePath(snapshot.ref.path)}`,
)
.join("\n"),
);
}
function writeVerifyResult(
runtime: RuntimeEnv,
options: BackupSqliteJsonOptions,
report: BackupSqliteVerifyResult,
): void {
if (options.json) {
writeRuntimeJson(runtime, report);
return;
}
runtime.log(
`SQLite snapshot verified: ${shortenHomePath(report.snapshotPath)} (${formatDatabaseIdentity(report.manifest.database)})`,
);
}
function writeRestoreResult(
runtime: RuntimeEnv,
options: BackupSqliteJsonOptions,
report: BackupSqliteRestoreResult,
): void {
if (options.json) {
writeRuntimeJson(runtime, report);
return;
}
runtime.log(
`SQLite snapshot restored: ${shortenHomePath(report.targetPath)} (${formatDatabaseIdentity(report.manifest.database)})`,
);
}
+24 -1
View File
@@ -4,13 +4,18 @@ import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import { requireNodeSqlite } from "./node-sqlite.js";
import { createVerifiedSqliteSnapshot } from "./sqlite-snapshot.js";
import { createPrivateSqliteDirectory, createVerifiedSqliteSnapshot } from "./sqlite-snapshot.js";
const tempDirs: string[] = [];
async function createTempDir(): Promise<string> {
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-sqlite-snapshot-"));
tempDirs.push(tempDir);
if (process.platform === "win32") {
const privateTempDir = path.join(tempDir, "private");
await createPrivateSqliteDirectory(privateTempDir);
return privateTempDir;
}
return tempDir;
}
@@ -49,6 +54,24 @@ function createUnsafeIndexDrift(sqlitePath: string): void {
}
describe("createVerifiedSqliteSnapshot", () => {
it.runIf(process.platform === "win32")(
"creates private staging directories exclusively under races",
async () => {
const tempDir = await createTempDir();
const directoryPath = path.join(tempDir, "private");
const results = await Promise.allSettled([
createPrivateSqliteDirectory(directoryPath),
createPrivateSqliteDirectory(directoryPath),
]);
expect(results.filter((result) => result.status === "fulfilled")).toHaveLength(1);
const rejected = results.find((result) => result.status === "rejected");
expect(rejected).toBeDefined();
expect((rejected as PromiseRejectedResult).reason).toMatchObject({ code: "EEXIST" });
await expect(fs.lstat(directoryPath)).resolves.toMatchObject({});
},
);
it("captures committed WAL state and removes deleted page contents", async () => {
const tempDir = await createTempDir();
const sourcePath = path.join(tempDir, "source.sqlite");
+137 -4
View File
@@ -6,12 +6,78 @@ import type { FileHandle } from "node:fs/promises";
import path from "node:path";
import type { DatabaseSync } from "node:sqlite";
import { loadSqliteVecExtension } from "../../packages/memory-host-sdk/src/engine-storage.js";
import { runExec } from "../process/exec.js";
import { formatErrorMessage } from "./errors.js";
import { sameFileIdentity } from "./fs-safe-advanced.js";
import { requireNodeSqlite } from "./node-sqlite.js";
import { resolveSystemBin } from "./resolve-system-bin.js";
import { assertSqliteIntegrity } from "./sqlite-integrity.js";
import { readSqliteUserVersion } from "./sqlite-user-version.js";
const SQLITE_DIRECTORY_MODE = 0o700;
const WINDOWS_DIRECTORY_EXISTS_MARKER = "OPENCLAW_SQLITE_DIRECTORY_EXISTS";
// Managed directory creation accepts existing paths. CreateDirectoryW applies the
// protected DACL atomically while preserving fail-if-exists semantics.
const WINDOWS_PRIVATE_DIRECTORY_NATIVE_SOURCE = `
using System;
using System.Runtime.InteropServices;
public static class OpenClawPrivateDirectory
{
[StructLayout(LayoutKind.Sequential)]
private struct SecurityAttributes
{
public int Length;
public IntPtr SecurityDescriptor;
public int InheritHandle;
}
[DllImport("advapi32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
private static extern bool ConvertStringSecurityDescriptorToSecurityDescriptorW(
string securityDescriptor,
uint revision,
out IntPtr convertedSecurityDescriptor,
out uint convertedSecurityDescriptorSize);
[DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
private static extern bool CreateDirectoryW(
string path,
ref SecurityAttributes securityAttributes);
[DllImport("kernel32.dll")]
private static extern IntPtr LocalFree(IntPtr memory);
public static int Create(string path, string securityDescriptor)
{
IntPtr descriptor;
uint descriptorSize;
if (!ConvertStringSecurityDescriptorToSecurityDescriptorW(
securityDescriptor,
1,
out descriptor,
out descriptorSize))
{
return Marshal.GetLastWin32Error();
}
try
{
var attributes = new SecurityAttributes
{
Length = Marshal.SizeOf(typeof(SecurityAttributes)),
SecurityDescriptor = descriptor,
InheritHandle = 0,
};
return CreateDirectoryW(path, ref attributes) ? 0 : Marshal.GetLastWin32Error();
}
finally
{
LocalFree(descriptor);
}
}
}
`;
export type SqliteSnapshotValidator = (database: DatabaseSync, databaseLabel: string) => void;
export type CreateVerifiedSqliteSnapshotOptions = {
@@ -51,6 +117,71 @@ export type VerifiedSqliteSnapshot = {
userVersion: number;
};
export async function createPrivateSqliteDirectory(directoryPath: string): Promise<void> {
if (process.platform !== "win32") {
await fs.mkdir(directoryPath, { mode: SQLITE_DIRECTORY_MODE });
return;
}
const encodedPath = Buffer.from(directoryPath, "utf8").toString("base64");
const encodedNativeSource = Buffer.from(WINDOWS_PRIVATE_DIRECTORY_NATIVE_SOURCE, "utf8").toString(
"base64",
);
const command = [
"$ErrorActionPreference = 'Stop'",
`$path = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String('${encodedPath}'))`,
`$nativeSource = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String('${encodedNativeSource}'))`,
"Add-Type -TypeDefinition $nativeSource -Language CSharp",
"$current = [System.Security.Principal.WindowsIdentity]::GetCurrent().User",
"$security = New-Object System.Security.AccessControl.DirectorySecurity",
"$security.SetAccessRuleProtection($true, $false)",
"$security.SetOwner($current)",
"$inheritance = [System.Security.AccessControl.InheritanceFlags]::ContainerInherit -bor [System.Security.AccessControl.InheritanceFlags]::ObjectInherit",
"$propagation = [System.Security.AccessControl.PropagationFlags]::None",
"foreach ($sidValue in @($current.Value, 'S-1-5-18', 'S-1-5-32-544')) { $sid = New-Object System.Security.Principal.SecurityIdentifier($sidValue); $rule = New-Object System.Security.AccessControl.FileSystemAccessRule($sid, [System.Security.AccessControl.FileSystemRights]::FullControl, $inheritance, $propagation, [System.Security.AccessControl.AccessControlType]::Allow); [void]$security.AddAccessRule($rule) }",
"$sections = [System.Security.AccessControl.AccessControlSections]::Owner -bor [System.Security.AccessControl.AccessControlSections]::Access",
"$sddl = $security.GetSecurityDescriptorSddlForm($sections)",
"$errorCode = [OpenClawPrivateDirectory]::Create($path, $sddl)",
`if ($errorCode -eq 80 -or $errorCode -eq 183) { throw '${WINDOWS_DIRECTORY_EXISTS_MARKER}' }`,
"if ($errorCode -ne 0) { $exception = New-Object System.ComponentModel.Win32Exception($errorCode); throw $exception }",
].join("; ");
const powershell = resolveSystemBin("powershell");
if (!powershell) {
throw new Error("Unable to resolve PowerShell for private Windows SQLite staging.");
}
const encodedCommand = Buffer.from(command, "utf16le").toString("base64");
try {
await runExec(
powershell,
["-NoLogo", "-NoProfile", "-NonInteractive", "-EncodedCommand", encodedCommand],
{
timeoutMs: 10_000,
maxBuffer: 64 * 1024,
},
);
} catch (error) {
if (String(error).includes(WINDOWS_DIRECTORY_EXISTS_MARKER)) {
const existsError = new Error(`Private SQLite directory already exists: ${directoryPath}`);
(existsError as NodeJS.ErrnoException).code = "EEXIST";
throw existsError;
}
throw new Error(`Unable to create private Windows SQLite directory: ${directoryPath}`, {
cause: error,
});
}
}
export async function createPrivateSqliteTempDirectory(
rootPath: string,
prefix: string,
): Promise<string> {
if (process.platform !== "win32") {
return await fs.mkdtemp(path.join(rootPath, prefix));
}
const directoryPath = path.join(rootPath, `${prefix}${randomUUID()}`);
await createPrivateSqliteDirectory(directoryPath);
return directoryPath;
}
async function assertRegularSourceFile(sourcePath: string): Promise<void> {
const stat = await fs.lstat(sourcePath);
if (!stat.isFile()) {
@@ -419,8 +550,9 @@ export async function publishVerifiedSqliteFile(
): Promise<void> {
await assertTargetAbsent(options.targetPath);
const targetDirectory = path.dirname(options.targetPath);
const stagingDir = await fs.mkdtemp(
path.join(targetDirectory, `.sqlite-publish-${randomUUID()}-`),
const stagingDir = await createPrivateSqliteTempDirectory(
targetDirectory,
`.sqlite-publish-${randomUUID()}-`,
);
const stagedPath = path.join(stagingDir, "database.sqlite");
let stagingIdentity: Stats | undefined;
@@ -646,8 +778,9 @@ export async function createVerifiedSqliteSnapshot(
await assertRegularSourceFile(options.sourcePath);
await assertTargetAbsent(options.targetPath);
const stagingDir = await fs.mkdtemp(
path.join(path.dirname(options.targetPath), ".sqlite-snapshot-"),
const stagingDir = await createPrivateSqliteTempDirectory(
path.dirname(options.targetPath),
".sqlite-snapshot-",
);
await fs.chmod(stagingDir, 0o700);
const stagedPath = path.join(stagingDir, "database.sqlite");
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+316
View File
@@ -0,0 +1,316 @@
import { createHash } from "node:crypto";
import type { BigIntStats, Stats } from "node:fs";
import fs from "node:fs/promises";
import path from "node:path";
import { sameFileIdentity } from "../infra/fs-safe-advanced.js";
import { root } from "../infra/fs-safe.js";
import { isValidAgentId, normalizeAgentId } from "../routing/session-key.js";
import {
SNAPSHOT_MANIFEST_FILENAME,
SNAPSHOT_SQLITE_FILENAME,
type SnapshotDatabaseManifest,
type SnapshotManifest,
} from "./snapshot-provider.js";
const MAX_MANIFEST_BYTES = 1024 * 1024;
const MAX_SQLITE_USER_VERSION = 2_147_483_647;
const MIN_SQLITE_USER_VERSION = -2_147_483_648;
const SNAPSHOT_ID_PATTERN = /^[a-zA-Z0-9][a-zA-Z0-9._-]{0,254}$/;
const SHA256_PATTERN = /^[a-f0-9]{64}$/;
export type SnapshotArtifactDigest = {
sha256: string;
sizeBytes: number;
stat: Stats;
};
type OpenFileHandle = Awaited<ReturnType<typeof fs.open>>;
export function containsAsciiControlCharacter(value: string): boolean {
for (let index = 0; index < value.length; index += 1) {
const code = value.charCodeAt(index);
if (code <= 0x1f || code === 0x7f) {
return true;
}
}
return false;
}
export async function hashSnapshotArtifact(snapshotDir: string): Promise<SnapshotArtifactDigest> {
const snapshotRoot = await root(snapshotDir);
const opened = await snapshotRoot.open(SNAPSHOT_SQLITE_FILENAME, {
hardlinks: "reject",
symlinks: "reject",
});
try {
return { ...(await hashFileHandle(opened.handle)), stat: opened.stat };
} finally {
await opened.handle.close();
}
}
export async function copySnapshotArtifact(
snapshotDir: string,
targetPath: string,
): Promise<SnapshotArtifactDigest> {
const snapshotRoot = await root(snapshotDir);
const source = await snapshotRoot.open(SNAPSHOT_SQLITE_FILENAME, {
hardlinks: "reject",
symlinks: "reject",
});
let target: OpenFileHandle | undefined;
let targetIdentity: Stats | undefined;
try {
target = await fs.open(targetPath, "wx+", 0o600);
targetIdentity = await target.stat();
const digest = await hashFileHandle(source.handle, target);
await target.sync();
const finalIdentity = await target.stat();
const currentIdentity = await fs.lstat(targetPath);
if (
!sameFileIdentity(targetIdentity, finalIdentity) ||
!sameFileIdentity(targetIdentity, currentIdentity)
) {
throw new Error(`Snapshot restore staging file changed during copy: ${targetPath}`);
}
return { ...digest, stat: finalIdentity };
} catch (error) {
await target?.close().catch(() => undefined);
target = undefined;
if (targetIdentity) {
const currentIdentity = await fs.lstat(targetPath).catch(() => undefined);
if (currentIdentity && sameFileIdentity(targetIdentity, currentIdentity)) {
await fs.unlink(targetPath).catch(() => undefined);
}
}
throw error;
} finally {
await target?.close().catch(() => undefined);
await source.handle.close().catch(() => undefined);
}
}
async function hashFileHandle(
source: OpenFileHandle,
target?: OpenFileHandle,
): Promise<Omit<SnapshotArtifactDigest, "stat">> {
const initialStat = await source.stat({ bigint: true });
const hash = createHash("sha256");
const buffer = Buffer.allocUnsafe(1024 * 1024);
let sizeBytes = 0;
while (true) {
const { bytesRead } = await source.read(buffer, 0, buffer.length, sizeBytes);
if (bytesRead === 0) {
break;
}
hash.update(buffer.subarray(0, bytesRead));
let bytesWritten = 0;
if (target) {
while (bytesWritten < bytesRead) {
const result = await target.write(
buffer,
bytesWritten,
bytesRead - bytesWritten,
sizeBytes + bytesWritten,
);
if (result.bytesWritten === 0) {
throw new Error("Snapshot restore staging copy made no progress.");
}
bytesWritten += result.bytesWritten;
}
}
sizeBytes += bytesRead;
}
const finalStat = await source.stat({ bigint: true });
if (!sameMutationFingerprint(initialStat, finalStat)) {
throw new Error("Snapshot artifact changed while being read.");
}
return { sha256: hash.digest("hex"), sizeBytes };
}
function sameMutationFingerprint(left: BigIntStats, right: BigIntStats): boolean {
return (
left.birthtimeNs === right.birthtimeNs &&
left.ctimeNs === right.ctimeNs &&
left.dev === right.dev &&
left.ino === right.ino &&
left.mtimeNs === right.mtimeNs &&
left.size === right.size
);
}
export async function writeSnapshotManifest(
snapshotDir: string,
manifest: SnapshotManifest,
): Promise<void> {
const manifestPath = path.join(snapshotDir, SNAPSHOT_MANIFEST_FILENAME);
const handle = await fs.open(manifestPath, "wx+", 0o600);
try {
await handle.writeFile(`${JSON.stringify(manifest, null, 2)}\n`, "utf8");
await handle.sync();
} finally {
await handle.close();
}
}
export async function readSnapshotManifest(
snapshotDir: string,
expectedSnapshotId = path.basename(snapshotDir),
): Promise<SnapshotManifest> {
const snapshotRoot = await root(snapshotDir);
const manifestPath = path.join(snapshotDir, SNAPSHOT_MANIFEST_FILENAME);
const result = await snapshotRoot.read(SNAPSHOT_MANIFEST_FILENAME, {
hardlinks: "reject",
maxBytes: MAX_MANIFEST_BYTES,
symlinks: "reject",
});
let parsed: unknown;
try {
parsed = JSON.parse(result.buffer.toString("utf8")) as unknown;
} catch (error) {
throw new Error(`Snapshot manifest is not valid JSON: ${manifestPath}`, { cause: error });
}
return parseSnapshotManifest(parsed, manifestPath, expectedSnapshotId);
}
export function parseSnapshotManifest(
value: unknown,
manifestPath: string,
expectedSnapshotId: string,
): SnapshotManifest {
const record = requireRecord(value, "manifest", manifestPath);
requireExactKeys(record, ["schemaVersion", "snapshotId", "createdAt", "database", "artifact"]);
if (record.schemaVersion !== 1) {
throw new Error(
`Unsupported snapshot manifest schemaVersion ${String(record.schemaVersion)}: ${manifestPath}`,
);
}
const snapshotId = requireSnapshotId(record.snapshotId, manifestPath);
if (snapshotId !== expectedSnapshotId) {
throw new Error(
`Snapshot manifest id ${snapshotId} does not match directory ${expectedSnapshotId}: ${manifestPath}`,
);
}
const createdAt = requireCanonicalTimestamp(record.createdAt, manifestPath);
const database = parseSnapshotDatabase(record.database, manifestPath);
const artifactRecord = requireRecord(record.artifact, "artifact", manifestPath);
requireExactKeys(artifactRecord, ["path", "sha256", "sizeBytes"]);
if (artifactRecord.path !== SNAPSHOT_SQLITE_FILENAME) {
throw new Error(
`Snapshot manifest artifact.path must be ${SNAPSHOT_SQLITE_FILENAME}: ${manifestPath}`,
);
}
if (typeof artifactRecord.sha256 !== "string" || !SHA256_PATTERN.test(artifactRecord.sha256)) {
throw new Error(`Snapshot manifest artifact.sha256 is invalid: ${manifestPath}`);
}
if (!Number.isSafeInteger(artifactRecord.sizeBytes) || Number(artifactRecord.sizeBytes) <= 0) {
throw new Error(`Snapshot manifest artifact.sizeBytes is invalid: ${manifestPath}`);
}
return {
schemaVersion: 1,
snapshotId,
createdAt,
database,
artifact: {
path: SNAPSHOT_SQLITE_FILENAME,
sha256: artifactRecord.sha256,
sizeBytes: Number(artifactRecord.sizeBytes),
},
};
}
function parseSnapshotDatabase(value: unknown, manifestPath: string): SnapshotDatabaseManifest {
const database = requireRecord(value, "database", manifestPath);
const role = database.role;
const basename = requireSafeText(database.basename, "database.basename", manifestPath, 255);
if (path.basename(basename) !== basename || basename === "." || basename === "..") {
throw new Error(`Snapshot manifest database.basename is invalid: ${manifestPath}`);
}
const userVersion = requireSqliteUserVersion(database.userVersion, manifestPath);
if (role === "global") {
requireExactKeys(database, ["role", "basename", "userVersion"]);
return { role, basename, userVersion };
}
if (role === "agent") {
requireExactKeys(database, ["role", "agentId", "basename", "userVersion"]);
const agentId = requireSafeText(database.agentId, "database.agentId", manifestPath, 64);
if (!isValidAgentId(agentId) || normalizeAgentId(agentId) !== agentId) {
throw new Error(`Snapshot manifest database.agentId is invalid: ${manifestPath}`);
}
return { role, agentId, basename, userVersion };
}
if (role === "generic") {
requireExactKeys(database, ["role", "id", "basename", "userVersion"]);
const id = requireSafeText(database.id, "database.id", manifestPath, 256);
return { role, id, basename, userVersion };
}
throw new Error(`Snapshot manifest database.role is invalid: ${manifestPath}`);
}
function requireRecord(
value: unknown,
field: string,
manifestPath: string,
): Record<string, unknown> {
if (!value || typeof value !== "object" || Array.isArray(value)) {
throw new Error(`Snapshot manifest ${field} must be an object: ${manifestPath}`);
}
return value as Record<string, unknown>;
}
function requireExactKeys(record: Record<string, unknown>, expectedKeys: readonly string[]): void {
const actual = Object.keys(record).toSorted();
const expected = [...expectedKeys].toSorted();
if (actual.length !== expected.length || actual.some((key, index) => key !== expected[index])) {
throw new Error(
`Snapshot manifest fields must be exactly ${expectedKeys.join(", ")}; got ${actual.join(", ")}`,
);
}
}
function requireSnapshotId(value: unknown, manifestPath: string): string {
if (typeof value !== "string" || !SNAPSHOT_ID_PATTERN.test(value)) {
throw new Error(`Snapshot manifest snapshotId is invalid: ${manifestPath}`);
}
return value;
}
function requireCanonicalTimestamp(value: unknown, manifestPath: string): string {
if (typeof value !== "string") {
throw new Error(`Snapshot manifest createdAt is invalid: ${manifestPath}`);
}
const parsed = new Date(value);
if (!Number.isFinite(parsed.getTime()) || parsed.toISOString() !== value) {
throw new Error(`Snapshot manifest createdAt is not canonical ISO 8601: ${manifestPath}`);
}
return value;
}
function requireSafeText(
value: unknown,
field: string,
manifestPath: string,
maxLength: number,
): string {
if (
typeof value !== "string" ||
value.length === 0 ||
value.length > maxLength ||
value.trim() !== value ||
containsAsciiControlCharacter(value)
) {
throw new Error(`Snapshot manifest ${field} is invalid: ${manifestPath}`);
}
return value;
}
function requireSqliteUserVersion(value: unknown, manifestPath: string): number {
if (
!Number.isSafeInteger(value) ||
Number(value) < MIN_SQLITE_USER_VERSION ||
Number(value) > MAX_SQLITE_USER_VERSION
) {
throw new Error(`Snapshot manifest database.userVersion is invalid: ${manifestPath}`);
}
return Number(value);
}
+66
View File
@@ -0,0 +1,66 @@
export const SNAPSHOT_MANIFEST_FILENAME = "manifest.json";
export const SNAPSHOT_SQLITE_FILENAME = "database.sqlite";
export type SnapshotDatabaseIdentity =
| { readonly role: "global" }
| { readonly role: "agent"; readonly agentId: string }
| { readonly role: "generic"; readonly id: string };
export type SnapshotDatabaseRef = {
readonly path: string;
readonly identity: SnapshotDatabaseIdentity;
};
export type SnapshotDatabaseManifest =
| {
readonly role: "global";
readonly basename: string;
readonly userVersion: number;
}
| {
readonly role: "agent";
readonly agentId: string;
readonly basename: string;
readonly userVersion: number;
}
| {
readonly role: "generic";
readonly id: string;
readonly basename: string;
readonly userVersion: number;
};
export type SnapshotManifest = {
readonly schemaVersion: 1;
readonly snapshotId: string;
readonly createdAt: string;
readonly database: SnapshotDatabaseManifest;
readonly artifact: {
readonly path: typeof SNAPSHOT_SQLITE_FILENAME;
readonly sha256: string;
readonly sizeBytes: number;
};
};
export type SnapshotRef = {
readonly path: string;
};
export type SnapshotResult = {
readonly ref: SnapshotRef;
readonly manifest: SnapshotManifest;
};
export type SnapshotVerificationResult = {
readonly ok: true;
readonly manifest: SnapshotManifest;
};
export type SnapshotSummary = SnapshotResult;
export type SqliteSnapshotProvider = {
create(database: SnapshotDatabaseRef): Promise<SnapshotResult>;
list(): Promise<SnapshotSummary[]>;
restoreFresh(snapshot: SnapshotRef, targetPath: string): Promise<SnapshotVerificationResult>;
verify(snapshot: SnapshotRef): Promise<SnapshotVerificationResult>;
};