docs(gateway): document restart and crash recovery behavior (#103985)

* docs(gateway): document restart and crash recovery behavior

* chore(docs): allowlist intentional command:nwe hook example in spellcheck

docs/docs.json

@@ -1622,6 +1622,7 @@
                     "pages": [
                       "gateway/gateway-lock",
                       "gateway/background-process",
+                      "gateway/restart-recovery",
                       "gateway/multiple-gateways"
                     ]
                   },

docs/docs_map.md

@@ -3672,6 +3672,20 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
   - H4: Troubleshooting
   - H2: Related
 
+## gateway/restart-recovery.md
+
+- Route: /gateway/restart-recovery
+- Headings:
+  - H2: What survives a restart
+  - H2: Graceful restarts drain first
+  - H2: How interrupted work is detected
+  - H2: Automatic resume
+  - H3: Subagents
+  - H3: Background tasks
+  - H3: Agent-requested restarts
+  - H2: Safety valves and observability
+  - H2: What is not resumed
+
 ## gateway/sandbox-vs-tool-policy-vs-elevated.md
 
 - Route: /gateway/sandbox-vs-tool-policy-vs-elevated

docs/gateway/restart-recovery.md

@@ -0,0 +1,118 @@
+---
+summary: "What survives a gateway restart or crash: interrupted agent turns resume automatically, subagents and background tasks recover, queued deliveries drain"
+read_when:
+  - You want to know whether restarting the gateway loses in-progress agent work
+  - An agent run was interrupted by a restart, crash, or config reload
+  - You are debugging automatic session recovery after the gateway comes back up
+title: "Restart recovery"
+---
+
+Restarting the gateway does not lose agent state. Conversations, transcripts,
+scheduled jobs, background task records, and queued outbound messages all live
+on disk, and work that was interrupted mid-turn is detected and resumed
+automatically after the gateway comes back up. No manual intervention is
+required, and there is nothing to configure: recovery is always on.
+
+This page describes what survives a restart, how interrupted work is detected,
+and what the automatic resume looks like.
+
+## What survives a restart
+
+| State                         | Storage                                             | Behavior across restart                                                 |
+| ----------------------------- | --------------------------------------------------- | ----------------------------------------------------------------------- |
+| Conversation history          | JSONL transcripts + per-agent session store on disk | Untouched; sessions continue from the stored transcript                 |
+| Interrupted main-session turn | Recovery markers in the session store               | Automatically resumed a few seconds after startup                       |
+| Subagent runs                 | SQLite (shared state database)                      | Registry restored on boot; interrupted runs resumed                     |
+| Background tasks              | SQLite (shared state database)                      | Reconciled on boot; orphaned runs recovered or marked lost              |
+| Queued outbound deliveries    | SQLite delivery queue                               | Drained after restart; undelivered replies are retried                  |
+| Scheduled (cron) jobs         | SQLite cron store                                   | Schedules persist; the scheduler re-arms on boot                        |
+| Restart continuation          | SQLite restart sentinel                             | One-shot follow-up dispatched to the session that asked for the restart |
+
+## Graceful restarts drain first
+
+A requested restart (`openclaw gateway restart`, a config change that requires
+a restart, or a gateway update) does not kill in-flight work immediately. The
+gateway stops accepting new work, then waits for active agent turns and
+background tasks to finish, up to a drain budget (5 minutes by default). Most
+restarts therefore interrupt nothing at all.
+
+Only work that cannot finish inside the drain budget (or any run interrupted
+by a forced restart or a crash) is aborted — and before that happens, each
+affected session is marked for recovery.
+
+## How interrupted work is detected
+
+Two complementary mechanisms mark sessions whose turn did not finish:
+
+- **At shutdown:** during the restart drain, every session with an active run
+  is stamped with a recovery marker in the session store before the run is
+  aborted.
+- **At startup:** the gateway scans session stores for sessions that still
+  claim to be running but have no live owner in the new process. This catches
+  hard crashes and kills where no shutdown code ran. Stale transcript lock
+  files are cleaned up at the same time.
+
+## Automatic resume
+
+A few seconds after startup, the gateway re-dispatches each marked session
+with a synthetic system message telling the agent its previous turn was
+interrupted by a restart and to continue from the existing transcript. If a
+final reply had already been produced but not delivered, its text is included
+so the agent can deliver it instead of redoing the work. Recovery retries up
+to 3 times with exponential backoff.
+
+Before resuming, the gateway checks that the transcript tail is safe to
+continue from. If it is not (for example, the turn ended on a stale pending
+approval), the session is not blindly re-run; the agent instead posts a short
+notice asking the user to resend the last request.
+
+### Subagents
+
+Subagent runs are persisted in the shared SQLite state database, so the
+subagent registry survives the process. On boot the registry is restored and
+interrupted subagent sessions are resumed with their original task context.
+Two safety valves apply:
+
+- Runs interrupted more than 2 hours ago are finalized instead of resumed, so
+  a gateway that was down overnight does not resurrect stale work.
+- A session that repeatedly fails to recover is tombstoned as wedged so
+  recovery cannot loop forever.
+
+### Background tasks
+
+The [background task registry](/automation/tasks) is SQLite-backed and
+reconciled on boot and on a periodic interval: durable outcomes recorded by
+finished runs are recovered, and runs whose owning process disappeared are
+marked lost after a grace period instead of hanging forever.
+
+### Agent-requested restarts
+
+When the agent itself triggers a restart (applying a config change, updating
+the gateway, or an explicit restart request), a restart sentinel is written to
+SQLite before the process exits. After boot the gateway posts the outcome back
+to the originating chat and dispatches a one-shot continuation turn so the
+agent picks up exactly where it left off, on the same channel and thread.
+
+## Safety valves and observability
+
+- **Crash-loop breaker:** 3 unclean boots within 5 minutes trip a breaker that
+  suppresses auto-start side services on the next boot, so a crashing gateway
+  does not amplify itself. It recovers once the unclean-boot window drains.
+- **Metrics:** recovery activity is exported via
+  [Prometheus](/gateway/prometheus) as `openclaw_session_recovery_total` and
+  `openclaw_session_recovery_age_seconds`.
+- **Logs:** recovery decisions are logged under the
+  `main-session-restart-recovery` and `subagent-interrupted-resume`
+  subsystems.
+
+## What is not resumed
+
+- Sessions excluded from main-session recovery because another owner already
+  handles them: subagent sessions (subagent recovery), cron sessions (the
+  scheduler re-runs on schedule), and ACP-managed sessions (the connected IDE
+  or client owns the resume).
+- Sessions whose transcript tail cannot be safely continued; these get the
+  resend notice described above instead of a silent re-run.
+- Work that was never admitted: messages arriving during the drain window are
+  rejected with an explicit restart error rather than silently queued into a
+  dying process.

scripts/codespell-ignore.txt

@@ -14,3 +14,4 @@ Brining
 wit
 allowIn
 planText
+nwe