* ci: add hourly PR CI sweeper for dropped pull_request runs
Fresh PRs race GitHub's merge-ref computation: the open-event CI run can
drop entirely or be created as an un-rerunnable startup_failure (~10-16
runs daily). The sweeper lists recently updated open PRs hourly, finds
heads whose only pull_request-event CI runs are startup failures (or
missing), and re-fires the event by close/reopen with the Barnacle app
token (GITHUB_TOKEN events would not trigger workflows).
Safety: 10-minute quiet window, 24h lookback, skips drafts, merge
conflicts, pending mergeability, and auto-merge PRs (close cancels
auto-merge); revalidates state, head, and CI attachment immediately
before mutating; per-PR budget of two sweeper closes and a per-sweep cap
of ten; reopen-on-unknown ownership so a stranded close (silent) always
loses to a spurious reopen (visible); manual dispatch supports dry_run.
Accepted tradeoffs are documented inline: shared-SHA PR topologies can
mask a dropped run (skip-only miss; run.pull_requests matching would
break fork PRs), and app-auth failover at worst doubles the close budget.
* test(ci): exercise pr-ci-sweeper runner with a faked client
* fix(test): lint-clean pr-ci-sweeper runner fakes
* fix(ci): bound all openclaw-npm-release git fetch operations with timeout
Wrap all six git fetch commands with a 120-second deadline
and 10-second termination grace period, matching the pattern
established in #108940.
Fixes#109176
* test(ci): cover npm release fetch deadlines
Co-authored-by: wanyongstar <wan.yong@xydigit.com>
* ci: raise Control UI startup budget
Unblock exact-head builds after the compressed startup bundle reached the previous 370 KiB ratchet.
* ci: drop superseded Control UI budget bump
Current main now carries the smaller 340 KiB startup ratchet from #110528, so remove the temporary 372 KiB adjustment while keeping the PR head conflict-free.
---------
Co-authored-by: Peter Steinberger <steipete@gmail.com>
## Summary
- Problem: The `git fetch` in `macos-release.yml` had no timeout bound, risking indefinite CI hangs on network stalls. Same pattern already fixed for Mantis Crabbox workflows in #108940.
- Solution: Add `timeout --signal=TERM --kill-after=10s 120s` before the `git fetch` command.
- What changed: Wrapped the `git fetch --no-tags origin` command with timeout.
- What did NOT change: All release logic remains unchanged.
## Real behavior proof
- **Behavior addressed:** CI macos-release could hang during git fetch.
- **Evidence:** Same timeout pattern as merged Crabbox fix (#108940 / 16c2bbb).
- **Change:** `timeout --signal=TERM --kill-after=10s 120s git fetch --no-tags origin` replaces `git fetch --no-tags origin`.
## Root Cause
- The git fetch was added before the timeout-bounding pattern was standardized across CI workflows.
## Summary
- Problem: The `git clone` of `openclaw/docs.git` in `docs-sync-publish.yml` had no timeout bound, risking indefinite CI hangs on network stalls. Same pattern already fixed for Mantis Crabbox workflows in #108940.
- Solution: Add `timeout --signal=TERM --kill-after=10s 120s` before the `git clone` command.
- What changed: Wrapped the `git clone` command with timeout, matching the bounded pattern from #108940.
- What did NOT change: All retry logic, publish sync steps, and non-network operations remain unchanged.
## Real behavior proof
- **Behavior addressed:** CI docs-sync-publish workflow could hang indefinitely during git clone of external docs repo.
- **Real environment tested:** Pattern analysis and cross-reference with already-merged Crabbox fix (#108940 / 16c2bbb).
- **Exact change:** `if timeout --signal=TERM --kill-after=10s 120s git clone \` replaces `if git clone \`.
- **Evidence after fix:** Same `timeout --signal=TERM --kill-after=10s 120s` wrapper applied to Crabbox source retrieval.
- **What was not tested:** Live CI run.
## Root Cause
- The git clone was added before the timeout-bounding pattern was standardized across CI workflows. Missing automated check for unbounded external git operations.
* perf(ci): cut the pre-fan-out critical path on canonical runs
Two changes to the run head and matrix shape:
1. runner-admission was a hosted 90s sleep every run queued behind
(observed 1.7min hosted-queue latency before the sleep started). The
debounce now lives at the tail of preflight: heavy jobs all need
preflight, so a superseding main push still cancels the run before
fan-out while only one 4 vCPU runner has been spent, and preflight's
own work usually exceeds the window so the residual sleep is zero.
security-fast (hosted, dependency-free) starts immediately.
2. Canonical main pushes now use the compact bin plan like PRs: the
82-job named matrix drained the runner pool for ~4.5min (job starts
trickled from minute 5.2 to 9.7 in run 29592647843) with no
branch-protection consumer for per-shard names on main. Coverage is
identical; dispatch/release-gate runs keep the full named matrix.
* perf(test): boot TUI PTY suite fixtures concurrently
tsx+TUI startup dominated the harness file's wall time and the three
suite PTYs booted serially. Boot them concurrently in beforeAll
(allSettled so a failed boot still assigns survivors for afterAll
cleanup); the env-specific fixtures never receive input, so their tests
only await their own readiness output. The slow-startup test now proves
frame ordering on the append-only output, which the old sequential
waits did not. File wall 10.2s -> ~4.9s, 3/3 repeat runs green.
* docs(ci): align gate and debounce descriptions with the removed admission job
* test(tooling): wait for readiness file content, not existence
writeFileSync creates the file before its bytes land, so the existence
poll raced the child's write on loaded runners and read an empty ready
file (observed in compact-small-4, run 29615028678). Poll for non-empty
content at both readiness sites.
Success-only gating starved the deps/vitest snapshots: under merge
traffic every main push cancels the prior CI run, so the deadline
writer skipped for hours (15:34->18:45 observed) while PRs paid full
reinstalls against stale fingerprints. The warm job checks out main
tip regardless of the triggering run, so cancelled completions are
safe refresh signals; failed runs still skip.
The installation hit Blacksmith's backing-disk cap because sticky keys
embedded PR numbers and content hashes, minting a new disk per PR and
per input change until every mount 429-failed fleet-wide. node-deps
(#109752) and ext-boundary (#109804) were already re-keyed; this
converts the last two minters and audits the third suspect.
Keys changed:
- Vitest fs transform cache: `vitest-fs-v2-<pr-N|protected>-...` ->
the one existing `vitest-fs-v2-protected-<os>-<arch>-node-<ver>`
disk. Entries are content-hash keyed (vitest sha1 over
id+content+NODE_ENV+version+env config), so cross-PR sharing is safe
by construction: PRs now mount the protected snapshot directly
(read-only, commit false) and the separate PR-only seed mount is
deleted. Writers (non-PR shard writer + scheduled warm run) keep
explicit commit true. Disk count: 1 + O(open PRs) -> 1.
- Gradle: `gradle-v1-<task>-<pr-N|protected>-<hashFiles(deps)>` ->
`gradle-v2-<task>`. Task scope stays (light ktlint must not seed
heavy build lanes); PR number and dependency hash leave the key. The
dependency hash moved to an in-job fingerprint marker that makes the
non-PR writer rebuild its snapshot cold when inputs change, bounding
disk growth; PR mounts are read-only and safely reuse stale
snapshots because Gradle caches are content-addressed. Disk count:
O(tasks x PRs x dependency bumps) -> O(tasks) (7 today).
- Docker builder (useblacksmith/setup-docker-builder): audited, out of
scope. The action owns its key internally and always uses the repo
name only (src/setup_builder.ts getStickyDisk), so it is already one
disk per installation repo and cannot mint per-PR disks.
PR warm-path behavior changes:
- Vitest: PRs lose only PR-local warm entries for files the PR itself
changed (previously persisted on the pr-N disk between pushes);
changed files re-transform once per push, unchanged files still hit
the protected snapshot. PRs touching transform inputs
(lockfile/tsconfig/package.json) run cold per push since the
generation wipe is now local to the discarded clone.
- Gradle: dependency-bump PRs get warmer (stale-but-valid protected
snapshot instead of a cold fresh key); other PRs are unchanged.
Deleted `.github/workflows/pr-cache-cleanup.yml`: it existed for the
per-PR cache layer (#109425) and only deleted GitHub actions/cache
archives, which GitHub's own LRU/TTL eviction already handles for the
remaining fork/Windows per-PR archive paths.
Guards: pinned the O(1) vitest key + non-PR commit gate, added a
Gradle per-task key/writer/fingerprint guard, added a repo-wide scan
asserting no useblacksmith/stickydisk key ever contains
github.event.pull_request.number or hashFiles(), and replaced the
cleanup-workflow assertions with a stays-deleted check.
check-additional-extension-package-boundary is the light-run critical-path
pole. Measured on runs 29564442266/29564411446/29564259862 (8 vCPU, sticky
mounts 429-failing): Setup Node ~40s cold install, shard step 142-161s =
~101s cold plugin-sdk dts prep (nine parallel tsgo builds) + ~58s compiling
122 plugins at concurrency 6 + ~1s canary; warm actions/cache path on a main
push (29551077288) runs the shard in 67s.
Levers chosen from that data:
- Re-key the ext-boundary sticky disk from per-PR/per-config keys (v1) to one
O(1) protected key (v2), following the node_modules snapshot fix in
#109752. Blacksmith's installation-wide 1000-disk cap is saturated and
every mount 429s; per-PR keys are a direct contributor. The config/scripts/
lockfile hash moves from the key into the in-job .source-trees marker next
to the existing tree-OID + node/pnpm gate. Single semantic writer: the
boundary lane commits explicitly (true, not if-missing) on protected pushes
only; PR mounts and the check-lint consumer stay read-only, and the seed
step is now writer-only (saves ~11s per PR cold run).
- Bump the lane 8 -> 32 vCPU and raise OPENCLAW_EXTENSION_BOUNDARY_CONCURRENCY
6 -> 16: both the parallel dts prep and the 122 plugin compiles scale with
cores at similar billed core-minutes (same precedent as check-lint/deps).
Expected pole time: warm PR runs (protected snapshot tracks every main push)
drop to roughly checkout + install-skip + restore + incremental shard, well
under 2 minutes; cold runs that touch boundary trees shrink via the core
bump. Guarded in ci-workflow-guards so the O(1) key, single-writer commit,
and fingerprint composition cannot silently regress.
* perf(sdk): load Claude CLI identity from a narrow plugin artifact
src/plugin-sdk/anthropic-cli.ts snapshots CLAUDE_CLI_BACKEND_ID at module
scope through the sync facade loader, which jiti-evaluates the full
anthropic api.js barrel on source checkouts: 130.86s of self time per
cold worker on CI (Testbox import profile), silently stalling every job
whose graph reaches cli-runner/prepare.ts. A narrow cli-api.js artifact
carries the two static facts; Testbox proof: the reliability+helpers
agents-core pair drops from 157.6s to 14.3s.
* perf(ci): stripe unit-fast and tooling node tests on the full plan
core-fast ran the import-bound unit-fast graph as one job (247s vitest,
181s module evaluation) and core-tooling as one serial job (241s); both
now stripe on the full plan like the compact plan, so the compact-only
expansion is deleted and the docker helper config rides with the
isolated shard on both plans. Group hints refreshed from main run
29551077288; stale per-file commands hints dropped (the consolidation
landed); cache-writer selection and the warm workflow track the stripe
names.
* fix(state): tolerate vanished sqlite sidecars in agent-db permission sweep
existsSync+chmodSync raced SQLite's own WAL/SHM cleanup: a checkpoint or
close between the two calls throws ENOENT (observed from the transcript
reconcile worker in server-startup-web-fetch-bind on CI). chmod directly
and swallow only ENOENT, which removes the TOCTOU window.
