mirror of
https://github.com/openclaw/openclaw.git
synced 2026-07-21 02:06:43 +00:00
improve: simplify PR context and evidence (#94676)
* improve: simplify PR context and evidence * improve: decouple PR context from proof labels * fix: satisfy PR context lint
This commit is contained in:
Hannes Rudolph
@@ -107,16 +107,9 @@ Reject:
|
||||
|
||||
## PR Body Proof
|
||||
|
||||
Use the repo PR template. Include these exact labels:
|
||||
|
||||
```text
|
||||
Behavior addressed:
|
||||
Real environment tested:
|
||||
Exact steps or command run after this patch:
|
||||
Evidence after fix:
|
||||
Observed result after fix:
|
||||
What was not tested:
|
||||
```
|
||||
Use the repo PR template. Include authored `## What Problem This Solves` and
|
||||
`## Evidence` sections. Keep the body focused on intent and the most useful
|
||||
validation evidence; inspect the code, tests, and CI before judging correctness.
|
||||
|
||||
## Existing PR Rules
|
||||
|
||||
|
||||
@@ -1,118 +1,57 @@
|
||||
## Summary
|
||||
<!--
|
||||
Optional linked context:
|
||||
Add a visible `Closes #<issue-number>` or `Related: #<issue-number>` line
|
||||
below this comment.
|
||||
|
||||
What problem does this PR solve?
|
||||
Required PR title:
|
||||
type: user-facing description
|
||||
Use a parenthesized scope only when it adds clarity:
|
||||
fix(auth): login redirect loops when session cookie is expired
|
||||
|
||||
Why does this matter now?
|
||||
Types: feat, fix, improve, refactor, docs, chore.
|
||||
For fixes, describe the user-visible symptom and trigger:
|
||||
fix: task list fails to load when user has no environments
|
||||
Avoid implementation details such as:
|
||||
fix: add null check to task query
|
||||
-->
|
||||
|
||||
What is the intended outcome?
|
||||
## What Problem This Solves
|
||||
|
||||
What is intentionally out of scope?
|
||||
<!--
|
||||
Describe the concrete user, product, or operational problem.
|
||||
For fixes, begin with:
|
||||
"Fixes an issue where users <do X> would <experience Y> when <condition>."
|
||||
or:
|
||||
"Resolves a problem where..."
|
||||
|
||||
What does success look like?
|
||||
Name the affected UI surface or workflow. Do not describe the code-level cause here.
|
||||
-->
|
||||
|
||||
What should reviewers focus on?
|
||||
## Why This Change Was Made
|
||||
|
||||
<details>
|
||||
<summary>Summary guidance</summary>
|
||||
<!--
|
||||
In one or two sentences, explain the complete shipped solution, key design
|
||||
decisions, and relevant boundaries or non-goals. Include implementation detail
|
||||
only when it helps reviewers understand user-visible behavior or risk.
|
||||
Avoid file-by-file narration.
|
||||
-->
|
||||
|
||||
This PR description is the contributor's durable explanation of the change. Write it for human maintainers first; ClawSweeper and Barnacle use the same text to understand intent, proof, risk, and current review state.
|
||||
## User Impact
|
||||
|
||||
Describe the intent and outcome in 2-5 bullets. Avoid restating the diff; reviewers and bots can read the changed files.
|
||||
<!--
|
||||
State what users, operators, or developers can now do or expect. Lead with the
|
||||
concrete benefit and use user-facing language. If there is no user-visible
|
||||
impact, say so plainly.
|
||||
-->
|
||||
|
||||
If this PR fixes a plugin beta-release blocker, title it `fix(<plugin-id>): beta blocker - <summary>` and link the matching `Beta blocker: <plugin-name> - <summary>` issue labeled `beta-blocker`. Contributors cannot label PRs, so the title is the PR-side signal for maintainers and automation.
|
||||
## Evidence
|
||||
|
||||
</details>
|
||||
<!--
|
||||
Show the most useful proof that this change works. Screenshots, screencasts,
|
||||
terminal output, focused tests, CI results, live observations, redacted logs,
|
||||
and artifact links are all useful. Include before/after evidence for visual
|
||||
changes when it clarifies the result.
|
||||
|
||||
## Linked context
|
||||
|
||||
Which issue does this close?
|
||||
|
||||
Closes #
|
||||
|
||||
Which issues, PRs, or discussions are related?
|
||||
|
||||
Related #
|
||||
|
||||
Was this requested by a maintainer or owner?
|
||||
|
||||
<details>
|
||||
<summary>Linked context guidance</summary>
|
||||
|
||||
Link the issue, PR, discussion, maintainer request, or owner request that explains why this PR should exist. Maintainer context helps reviewers and automation distinguish intended work from drive-by churn.
|
||||
|
||||
</details>
|
||||
|
||||
## Real behavior proof (required for external PRs)
|
||||
|
||||
- Behavior or issue addressed:
|
||||
- Real environment tested:
|
||||
- Exact steps or command run after this patch:
|
||||
- Evidence after fix (screenshot, recording, terminal capture, console output, redacted runtime log, linked artifact, or copied live output):
|
||||
- Observed result after fix:
|
||||
- What was not tested:
|
||||
- Proof limitations or environment constraints:
|
||||
- Before evidence (optional but encouraged):
|
||||
|
||||
<details>
|
||||
<summary>Real behavior proof guidance</summary>
|
||||
|
||||
External contributors must show after-fix evidence from a real OpenClaw setup. Unit tests, mocks, lint, typechecks, snapshots, and CI are supplemental only.
|
||||
|
||||
Screenshots are encouraged even for CLI, console, text, or log changes. Terminal screenshots, copied live output, redacted runtime logs, recordings, and linked artifacts count.
|
||||
|
||||
If your environment cannot produce the ideal proof, explain that under `Proof limitations or environment constraints` so reviewers and ClawSweeper can direct the next step properly.
|
||||
|
||||
Be mindful of private information like IP addresses, API keys, phone numbers, non-public endpoints, or other private details when providing evidence.
|
||||
|
||||
</details>
|
||||
|
||||
## Tests and validation
|
||||
|
||||
Which commands did you run?
|
||||
|
||||
What regression coverage was added or updated?
|
||||
|
||||
What failed before this fix, if known?
|
||||
|
||||
If no test was added, why not?
|
||||
|
||||
<details>
|
||||
<summary>Testing guidance</summary>
|
||||
|
||||
List focused commands, not every incidental check. CI is useful support, but external PRs still need real behavior proof above when behavior changes.
|
||||
|
||||
</details>
|
||||
|
||||
## Risk checklist
|
||||
|
||||
Did user-visible behavior change? (`Yes/No`)
|
||||
|
||||
Did config, environment, or migration behavior change? (`Yes/No`)
|
||||
|
||||
Did security, auth, secrets, network, or tool execution behavior change? (`Yes/No`)
|
||||
|
||||
What is the highest-risk area?
|
||||
|
||||
How is that risk mitigated?
|
||||
|
||||
<details>
|
||||
<summary>Risk guidance</summary>
|
||||
|
||||
Use this for author judgment that is not obvious from the diff. ClawSweeper can see touched files, but it cannot know which behavior you think is risky, why the risk is acceptable, or what mitigation reviewers should verify.
|
||||
|
||||
</details>
|
||||
|
||||
## Current review state
|
||||
|
||||
What is the next action?
|
||||
|
||||
What is still waiting on author, maintainer, CI, or external proof?
|
||||
|
||||
Which bot or reviewer comments were addressed?
|
||||
|
||||
<details>
|
||||
<summary>Review state guidance</summary>
|
||||
|
||||
Keep this as the durable state for review progress. If useful information appears in comments, fold the current next action or blocker back here so maintainers and ClawSweeper do not need to reconstruct state from comment history.
|
||||
|
||||
</details>
|
||||
Reviewers will inspect the code, tests, and CI. Use this section to make the
|
||||
validation easy to understand, not to restate the diff.
|
||||
-->
|
||||
|
||||
@@ -35,7 +35,7 @@ Skills own workflows; root owns hard policy and routing.
|
||||
- One-sided fixes need sibling-surface proof, an explanation for why siblings are unaffected, or explicit follow-up work.
|
||||
- Changelog findings: see Docs / Changelog.
|
||||
- Public ClawSweeper comments prefer `https://docs.openclaw.ai/...` when a public docs page exists; structured evidence still cites repo files, lines, SHAs.
|
||||
- Findings need current source, shipped/current behavior, tests/CI evidence, and dependency contract proof when dependency-backed behavior is involved. Validation is judged against touched and sibling surfaces plus this file's commands; real behavior proof matters for user-visible changes, with Telegram/Desktop proof for Telegram-visible behavior when feasible.
|
||||
- Findings need current source, shipped/current behavior, tests/CI evidence, and dependency contract proof when dependency-backed behavior is involved. Validation is judged against touched and sibling surfaces plus this file's commands; clear evidence matters for user-visible changes, with Telegram/Desktop proof for Telegram-visible behavior when feasible.
|
||||
- Prefer findings for concrete behavior regressions, missing changed-surface proof, owner-boundary violations, security/API contract issues, or docs/config mismatches.
|
||||
- Do not file findings for repo policy preference when changed code follows the relevant scoped guide and no user-visible, runtime, security, or maintainer-risk impact is shown.
|
||||
|
||||
@@ -165,13 +165,12 @@ Skills own workflows; root owns hard policy and routing.
|
||||
- Representing user: if user already has a comment/thread for the point, update/reply there when possible; avoid duplicate PR/issue comments.
|
||||
- No surprise GH writes: chat must mention every posted/updated public comment with URL.
|
||||
- GH comments with backticks, `$`, or shell snippets: use heredoc/body file, not inline double-quoted `--body`.
|
||||
- PR create: real body required. Include Summary + Verification; mention refs, behavior, and proof.
|
||||
- PR create: real body required. Use the current template: `What Problem This Solves`, `Why This Change Was Made`, `User Impact`, and `Evidence`; include visible refs, behavior, and validation.
|
||||
- PR create/refresh: keep PR branches takeover-ready. Use a branch maintainers can push to, or for fork PRs ensure `maintainer_can_modify` / GitHub's `Allow edits by maintainers` is enabled unless explicitly told otherwise or GitHub's Actions/secrets warning makes that unsafe.
|
||||
- GitHub issue/PR create: read `$agent-transcript`; ask about sanitized transcript logs when available.
|
||||
- Contributor PRs: parsed `Real behavior proof` uses exact `field: value` labels: `Behavior addressed`, `Real environment tested`, `Exact steps or command run after this patch`, `Evidence after fix`, `Observed result after fix`, `What was not tested`.
|
||||
- Contributor PRs: parsed context requires authored `What Problem This Solves` and `Evidence` sections. Do not require field-level proof forms; reviewers inspect code, tests, and CI for correctness.
|
||||
- PR artifacts/screenshots: attach to PR/comment/external artifact store. Never push screenshots, videos, proof images, or proof assets to OpenClaw or any product repo branch, including temp artifact branches. Use Crabbox artifact publishing plus the manifest URL. Do not commit `.github/pr-assets`.
|
||||
- CI polling: exact SHA, relevant checks only, minimal fields. Skip routine noise (`Auto response`, `Labeler`, docs agents, performance/stale). Logs only after failure/completion or concrete need.
|
||||
- OpenClaw write-access maintainers may skip `Real behavior proof` when local tests or Crabbox verified behavior; record proof in PR verification.
|
||||
- Agent PR landing to `main`: use only the repo-native `scripts/pr` wrapper: run `scripts/pr review-init <PR>`, follow its emitted checkout/guard guidance, initialize and complete review artifacts with `scripts/pr review-artifacts-init <PR>`, validate them with `scripts/pr review-validate-artifacts <PR>`, then run `scripts/pr prepare-run <PR>` and `scripts/pr merge-run <PR>`; do not idle on `auto-response` or `check-docs`.
|
||||
|
||||
## Code
|
||||
|
||||
+2
-2
@@ -106,7 +106,7 @@ For coordinated change sets that genuinely need more than 20 PRs, join the **#cl
|
||||
## Before You PR
|
||||
|
||||
- Test locally with your OpenClaw instance
|
||||
- External PRs must include a filled **Real behavior proof** section in the PR body. Show the real setup you tested, the exact command or steps you ran after the patch, after-fix evidence, the observed result, and anything you did not test. Screenshots, recordings, terminal screenshots, console output, copied live output, linked artifacts, and redacted runtime logs all count. Unit tests, mocks, snapshots, lint, typechecks, and CI are useful but do not satisfy this requirement by themselves. Maintainers may apply `proof: override` only when the proof gate should not apply.
|
||||
- External PRs must describe the user, product, or operational problem in **What Problem This Solves** and include useful validation in **Evidence**. Focused tests, CI results, screenshots, recordings, terminal output, live observations, redacted logs, and artifact links all count. Reviewers will inspect the code, tests, and CI; use the PR body to explain intent and make validation easy to understand.
|
||||
- Keep PRs takeover-ready: open them from a branch maintainers can push to. For fork PRs, leave GitHub's **Allow edits by maintainers** option enabled so maintainers can finish urgent fixes, changelog entries, or merge prep when needed. If GitHub shows **Allow edits and access to secrets by maintainers**, enable it only when that workflow/secrets access is acceptable and say so in the PR.
|
||||
- Do not edit `CHANGELOG.md` in contributor PRs. Maintainers or ClawSweeper add the changelog entry when landing user-facing changes.
|
||||
- Run tests: `pnpm build && pnpm check && pnpm test`
|
||||
@@ -169,7 +169,7 @@ Built with Codex, Claude, or other AI tools? **Awesome - just mark it!**
|
||||
Please include in your PR:
|
||||
|
||||
- [ ] Mark as AI-assisted in the PR title or description
|
||||
- [ ] Include human-run real behavior proof from your own setup. AI-generated tests, mocks, lint, typechecks, and CI output are supplemental only; they do not prove the fix works for users.
|
||||
- [ ] Include a concise **Evidence** section with the most useful validation. Reviewers will inspect the code, tests, and CI rather than relying on the PR body alone.
|
||||
- [ ] Include prompts or session logs if possible (super helpful!)
|
||||
- [ ] Confirm you understand what the code does
|
||||
- [ ] If you have access to Codex, run `codex review --base origin/main` locally and address the findings before asking for review
|
||||
|
||||
+7
-19
@@ -47,33 +47,21 @@ Use `pnpm ci:timings`, `pnpm ci:timings:recent`, or `node scripts/ci-run-timings
|
||||
|
||||
For pull request runs, the terminal timing-summary job runs the helper from the trusted base revision before passing `GH_TOKEN` to `gh run view`. That keeps the tokened query out of branch-controlled code while still summarizing the pull request's current CI run.
|
||||
|
||||
## Real behavior proof
|
||||
## PR context and evidence
|
||||
|
||||
External contributor PRs run a `Real behavior proof` gate from
|
||||
External contributor PRs run a PR context and evidence gate from
|
||||
`.github/workflows/real-behavior-proof.yml`. The workflow checks out the trusted
|
||||
base commit and evaluates the PR body only; it does not execute code from the
|
||||
contributor branch.
|
||||
|
||||
The gate applies to PR authors who are not repository owners, members,
|
||||
collaborators, or bots. It passes when the PR body contains a
|
||||
`Real behavior proof` section with filled values for:
|
||||
|
||||
- `Behavior or issue addressed`
|
||||
- `Real environment tested`
|
||||
- `Exact steps or command run after this patch`
|
||||
- `Evidence after fix`
|
||||
- `Observed result after fix`
|
||||
- `What was not tested`
|
||||
|
||||
The evidence must show the changed behavior after the patch in a real OpenClaw
|
||||
setup. Screenshots, recordings, terminal captures, console output, copied live
|
||||
output, redacted runtime logs, and linked artifacts all count. Unit tests, mocks,
|
||||
snapshots, lint, typechecks, and CI results are useful supporting verification,
|
||||
but they do not satisfy this gate by themselves.
|
||||
collaborators, or bots. It passes when the PR body contains authored
|
||||
`What Problem This Solves` and `Evidence` sections. Evidence can be a focused
|
||||
test, CI result, screenshot, recording, terminal output, live observation,
|
||||
redacted log, or artifact link. The body provides intent and useful validation;
|
||||
reviewers inspect the code, tests, and CI to assess correctness.
|
||||
|
||||
When the check fails, update the PR body instead of pushing another code commit.
|
||||
Maintainers can apply `proof: override` only when the proof gate should not
|
||||
apply to that PR.
|
||||
|
||||
## Scope and routing
|
||||
|
||||
|
||||
@@ -1,14 +1,11 @@
|
||||
// Barnacle owns deterministic GitHub triage and auto-response behavior.
|
||||
|
||||
import {
|
||||
MOCK_ONLY_PROOF_LABEL,
|
||||
NEEDS_REAL_BEHAVIOR_PROOF_LABEL,
|
||||
PROOF_OVERRIDE_LABEL,
|
||||
NEEDS_PR_CONTEXT_LABEL,
|
||||
PROOF_SUFFICIENT_LABEL,
|
||||
PROOF_SUPPLIED_LABEL,
|
||||
evaluateRealBehaviorProof,
|
||||
hasClawSweeperExactHeadProof,
|
||||
labelsForRealBehaviorProof,
|
||||
evaluatePullRequestContext,
|
||||
hasAuthoredPullRequestSection,
|
||||
labelsForPullRequestContext,
|
||||
} from "./real-behavior-proof-policy.mjs";
|
||||
|
||||
const activePrLimit = 20;
|
||||
@@ -156,25 +153,9 @@ export const managedLabelSpecs = {
|
||||
color: "C5DEF5",
|
||||
description: "Candidate: PR template appears mostly untouched.",
|
||||
},
|
||||
[NEEDS_REAL_BEHAVIOR_PROOF_LABEL]: {
|
||||
[NEEDS_PR_CONTEXT_LABEL]: {
|
||||
color: "C5DEF5",
|
||||
description: "Candidate: external PR needs after-fix proof from a real setup.",
|
||||
},
|
||||
[MOCK_ONLY_PROOF_LABEL]: {
|
||||
color: "C5DEF5",
|
||||
description: "Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI.",
|
||||
},
|
||||
[PROOF_SUPPLIED_LABEL]: {
|
||||
color: "C2E0C6",
|
||||
description: "External PR includes structured after-fix real behavior proof.",
|
||||
},
|
||||
[PROOF_SUFFICIENT_LABEL]: {
|
||||
color: "0E8A16",
|
||||
description: "ClawSweeper judged the real behavior proof convincing.",
|
||||
},
|
||||
[PROOF_OVERRIDE_LABEL]: {
|
||||
color: "C2E0C6",
|
||||
description: "Maintainer override for the external PR real behavior proof gate.",
|
||||
description: "Candidate: external PR body lacks required problem context or evidence.",
|
||||
},
|
||||
"triage: dirty-candidate": {
|
||||
color: "C5DEF5",
|
||||
@@ -196,8 +177,7 @@ export const candidateLabels = {
|
||||
docsDiscoverability: "triage: docs-discoverability",
|
||||
testOnlyNoBug: "triage: test-only-no-bug",
|
||||
refactorOnly: "triage: refactor-only",
|
||||
needsRealBehaviorProof: NEEDS_REAL_BEHAVIOR_PROOF_LABEL,
|
||||
mockOnlyProof: MOCK_ONLY_PROOF_LABEL,
|
||||
needsPrContext: NEEDS_PR_CONTEXT_LABEL,
|
||||
dirtyCandidate: "triage: dirty-candidate",
|
||||
riskyInfra: "triage: risky-infra",
|
||||
externalPluginCandidate: "triage: external-plugin-candidate",
|
||||
@@ -240,26 +220,16 @@ const maintainerAuthorLabel = "maintainer";
|
||||
const privilegedAuthorAssociations = new Set(["OWNER", "MEMBER", "COLLABORATOR"]);
|
||||
const privilegedRepositoryRoles = new Set(["admin", "maintain", "write"]);
|
||||
const candidateLabelValues = Object.values(candidateLabels);
|
||||
const structuralProofLabelValues = [
|
||||
NEEDS_REAL_BEHAVIOR_PROOF_LABEL,
|
||||
MOCK_ONLY_PROOF_LABEL,
|
||||
PROOF_SUPPLIED_LABEL,
|
||||
];
|
||||
const structuralContextLabelValues = [NEEDS_PR_CONTEXT_LABEL];
|
||||
const noisyPrMessage =
|
||||
"Closing this PR because it looks dirty (too many unrelated or unexpected changes). This usually happens when a branch picks up unrelated commits or a merge went sideways. Please recreate the PR from a clean branch.";
|
||||
|
||||
const candidateActionRules = [
|
||||
{
|
||||
label: candidateLabels.needsRealBehaviorProof,
|
||||
label: candidateLabels.needsPrContext,
|
||||
close: true,
|
||||
message:
|
||||
"Closing this PR because it does not include real behavior proof. Please reopen or resubmit with after-fix evidence from a real OpenClaw setup; terminal screenshots, console output, redacted logs, recordings, linked artifacts, and copied live output count. Unit tests, mocks, snapshots, lint, typechecks, and CI are supplemental only.",
|
||||
},
|
||||
{
|
||||
label: candidateLabels.mockOnlyProof,
|
||||
close: true,
|
||||
message:
|
||||
"Closing this PR because the proof only shows tests, mocks, snapshots, lint, typechecks, or CI. Please reopen or resubmit with after-fix evidence from a real OpenClaw setup; terminal screenshots, console output, redacted logs, recordings, linked artifacts, and copied live output count.",
|
||||
"Closing this PR because its body lacks a clear problem statement or evidence. Please reopen or resubmit with the user, product, or operational problem and the most useful validation evidence, such as a focused test, CI result, screenshot, recording, terminal output, log, or artifact.",
|
||||
},
|
||||
{
|
||||
label: candidateLabels.dirtyCandidate,
|
||||
@@ -356,7 +326,7 @@ function hasMostlyBlankTemplate(body) {
|
||||
if (!body) {
|
||||
return true;
|
||||
}
|
||||
const emptyFields = [
|
||||
const legacyEmptyFields = [
|
||||
"Problem",
|
||||
"Why it matters",
|
||||
"What changed",
|
||||
@@ -368,13 +338,22 @@ function hasMostlyBlankTemplate(body) {
|
||||
const regex = new RegExp(`^\\s*-\\s*${escapedField}(?: \\([^)]*\\))?:\\s*$`, "im");
|
||||
return regex.test(body);
|
||||
}).length;
|
||||
const hasTemplateIntro = body.includes("Describe the problem and fix in 2–5 bullets");
|
||||
const hasLegacyTemplateIntro = body.includes("Describe the problem and fix in 2–5 bullets");
|
||||
const emptyClosingRef = /^\s*-\s*(?:Closes|Related)\s+#\s*$/im.test(body);
|
||||
return hasTemplateIntro && emptyFields >= 3 && emptyClosingRef;
|
||||
const hasNewTemplateIntro = body.includes(
|
||||
"Describe the concrete user, product, or operational problem.",
|
||||
);
|
||||
return (
|
||||
(hasLegacyTemplateIntro && legacyEmptyFields >= 3 && emptyClosingRef) ||
|
||||
(hasNewTemplateIntro &&
|
||||
!hasAuthoredPullRequestSection("What Problem This Solves", body) &&
|
||||
!hasAuthoredPullRequestSection("Evidence", body))
|
||||
);
|
||||
}
|
||||
|
||||
function stripPullRequestTemplateBoilerplate(text) {
|
||||
return text
|
||||
.replace(/<!--[\s\S]*?-->/g, "")
|
||||
.replace(/^#{2,3}\s+.*$/gm, "")
|
||||
.replace(/^-\s*\[[ xX]\]\s+.*$/gm, "")
|
||||
.replace(/^-\s*(?:Closes|Related)\s+#\s*$/gim, "")
|
||||
@@ -397,6 +376,9 @@ function hasConcreteBehaviorContext(body, text) {
|
||||
if (hasLinkedReference(text)) {
|
||||
return true;
|
||||
}
|
||||
if (hasAuthoredPullRequestSection("What Problem This Solves", body)) {
|
||||
return true;
|
||||
}
|
||||
if (
|
||||
hasFilledTemplateLine(body, "Problem") &&
|
||||
hasFilledTemplateLine(body, "Why it matters") &&
|
||||
@@ -486,6 +468,7 @@ export function classifyPullRequestCandidateLabels(pullRequest, files) {
|
||||
const filenames = files.map((file) => file.filename);
|
||||
const body = pullRequest.body ?? "";
|
||||
const text = `${pullRequest.title ?? ""}\n${body}`;
|
||||
const contentText = stripPullRequestTemplateBoilerplate(text);
|
||||
const lowerText = text.toLowerCase();
|
||||
const linkedReference = hasLinkedReference(text);
|
||||
const blankTemplate = hasMostlyBlankTemplate(body);
|
||||
@@ -500,8 +483,8 @@ export function classifyPullRequestCandidateLabels(pullRequest, files) {
|
||||
}
|
||||
|
||||
labelsToAdd.push(
|
||||
...labelsForRealBehaviorProof(
|
||||
evaluateRealBehaviorProof({
|
||||
...labelsForPullRequestContext(
|
||||
evaluatePullRequestContext({
|
||||
pullRequest,
|
||||
}),
|
||||
),
|
||||
@@ -544,7 +527,7 @@ export function classifyPullRequestCandidateLabels(pullRequest, files) {
|
||||
if (
|
||||
!linkedReference &&
|
||||
!concreteBehaviorContext &&
|
||||
/\b(refactor|cleanup|clean up|rename|formatting|style-only|style only)\b/i.test(text)
|
||||
/\b(refactor|cleanup|clean up|rename|formatting|style-only|style only)\b/i.test(contentText)
|
||||
) {
|
||||
labelsToAdd.push(candidateLabels.refactorOnly);
|
||||
}
|
||||
@@ -768,15 +751,6 @@ async function listPullRequestFiles(github, context, pullRequest) {
|
||||
});
|
||||
}
|
||||
|
||||
async function listIssueComments(github, context, issueNumber) {
|
||||
return github.paginate(github.rest.issues.listComments, {
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: issueNumber,
|
||||
per_page: 100,
|
||||
});
|
||||
}
|
||||
|
||||
async function addMissingLabels(github, context, core, issueNumber, labels, labelSet) {
|
||||
const missingLabels = labels.filter((label) => !labelSet.has(label));
|
||||
if (missingLabels.length === 0) {
|
||||
@@ -798,90 +772,19 @@ function isClawSweeperOwnedLabel(label) {
|
||||
return label === "clawsweeper" || label.startsWith("clawsweeper:");
|
||||
}
|
||||
|
||||
function isActiveClawSweeperWork(pullRequest, labelSet) {
|
||||
const authorLogin = pullRequest.user?.login ?? "";
|
||||
const headRef = pullRequest.head?.ref ?? "";
|
||||
return (
|
||||
/clawsweeper/i.test(authorLogin) ||
|
||||
headRef.startsWith("clawsweeper/") ||
|
||||
[...labelSet].some(isClawSweeperOwnedLabel)
|
||||
);
|
||||
}
|
||||
|
||||
function shouldRemoveProofSufficientLabel(
|
||||
context,
|
||||
pullRequest,
|
||||
labelSet,
|
||||
proofEvaluation,
|
||||
hasExactHeadClawSweeperProof,
|
||||
) {
|
||||
if (hasExactHeadClawSweeperProof) {
|
||||
return false;
|
||||
}
|
||||
if (proofEvaluation.status === "override") {
|
||||
return false;
|
||||
}
|
||||
if (isActiveClawSweeperWork(pullRequest, labelSet)) {
|
||||
return false;
|
||||
}
|
||||
if (!["edited", "synchronize"].includes(context.payload.action)) {
|
||||
return false;
|
||||
}
|
||||
if (proofEvaluation.status !== "passed") {
|
||||
return true;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
const negativeProofLabels = new Set([NEEDS_REAL_BEHAVIOR_PROOF_LABEL, MOCK_ONLY_PROOF_LABEL]);
|
||||
|
||||
function shouldPreserveClawSweeperProofJudgment(context, labelSet) {
|
||||
return (
|
||||
labelSet.has(PROOF_SUFFICIENT_LABEL) &&
|
||||
!["edited", "synchronize"].includes(context.payload.action)
|
||||
);
|
||||
}
|
||||
|
||||
async function applyPullRequestCandidateLabels(github, context, core, pullRequest, labelSet) {
|
||||
const files = await listPullRequestFiles(github, context, pullRequest);
|
||||
const hasExactHeadClawSweeperProof =
|
||||
labelSet.has(PROOF_SUFFICIENT_LABEL) &&
|
||||
hasClawSweeperExactHeadProof({
|
||||
pullRequest,
|
||||
comments: await listIssueComments(github, context, pullRequest.number),
|
||||
});
|
||||
const proofEvaluation = evaluateRealBehaviorProof({
|
||||
pullRequest: {
|
||||
...pullRequest,
|
||||
labels: [...labelSet].map((name) => ({ name })),
|
||||
},
|
||||
});
|
||||
const classifiedLabels = classifyPullRequestCandidateLabels(
|
||||
const candidateLabelsToApply = classifyPullRequestCandidateLabels(
|
||||
{
|
||||
...pullRequest,
|
||||
labels: [...labelSet].map((name) => ({ name })),
|
||||
},
|
||||
files,
|
||||
);
|
||||
const candidateLabelsToApply = shouldPreserveClawSweeperProofJudgment(context, labelSet)
|
||||
? classifiedLabels.filter((label) => !negativeProofLabels.has(label))
|
||||
: classifiedLabels;
|
||||
const staleProofLabels = structuralProofLabelValues.filter(
|
||||
const staleContextLabels = structuralContextLabelValues.filter(
|
||||
(label) => labelSet.has(label) && !candidateLabelsToApply.includes(label),
|
||||
);
|
||||
if (
|
||||
labelSet.has(PROOF_SUFFICIENT_LABEL) &&
|
||||
shouldRemoveProofSufficientLabel(
|
||||
context,
|
||||
pullRequest,
|
||||
labelSet,
|
||||
proofEvaluation,
|
||||
hasExactHeadClawSweeperProof,
|
||||
)
|
||||
) {
|
||||
staleProofLabels.push(PROOF_SUFFICIENT_LABEL);
|
||||
}
|
||||
await removeLabels(github, context, pullRequest.number, staleProofLabels, labelSet);
|
||||
await removeLabels(github, context, pullRequest.number, staleContextLabels, labelSet);
|
||||
await addMissingLabels(
|
||||
github,
|
||||
context,
|
||||
|
||||
@@ -1,19 +1,12 @@
|
||||
#!/usr/bin/env node
|
||||
// Checks PR real-behavior proof labels/comments and writes GitHub Action outputs.
|
||||
// Checks external PR body context and evidence.
|
||||
import { readFileSync } from "node:fs";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import {
|
||||
DEFAULT_GITHUB_API_TIMEOUT_MS,
|
||||
evaluateClawSweeperExactHeadProof,
|
||||
evaluateRealBehaviorProof,
|
||||
evaluatePullRequestContext,
|
||||
isMaintainerTeamMember,
|
||||
readBoundedGitHubApiJson,
|
||||
withGitHubApiTimeout,
|
||||
} from "./real-behavior-proof-policy.mjs";
|
||||
|
||||
const PROOF_COMMENTS_PER_PAGE = 100;
|
||||
const MAX_PROOF_COMMENT_PAGES = 10;
|
||||
|
||||
function escapeCommandValue(value) {
|
||||
return String(value)
|
||||
.replace(/%/g, "%25")
|
||||
@@ -22,139 +15,6 @@ function escapeCommandValue(value) {
|
||||
.replace(/:/g, "%3A");
|
||||
}
|
||||
|
||||
function isTooLargeBodyError(error) {
|
||||
return error?.code === "ETOOBIG";
|
||||
}
|
||||
|
||||
async function fetchProofCommentPage({
|
||||
owner,
|
||||
repo,
|
||||
issueNumber,
|
||||
token,
|
||||
fetchImpl,
|
||||
timeoutMs,
|
||||
page,
|
||||
perPage,
|
||||
}) {
|
||||
const url = new URL(
|
||||
`https://api.github.com/repos/${owner}/${repo}/issues/${issueNumber}/comments`,
|
||||
);
|
||||
url.searchParams.set("per_page", String(perPage));
|
||||
url.searchParams.set("page", String(page));
|
||||
const response = await withGitHubApiTimeout(
|
||||
`proof comment lookup page ${page}`,
|
||||
timeoutMs,
|
||||
(signal) =>
|
||||
fetchImpl(url, {
|
||||
headers: {
|
||||
Accept: "application/vnd.github+json",
|
||||
Authorization: `Bearer ${token}`,
|
||||
"X-GitHub-Api-Version": "2022-11-28",
|
||||
},
|
||||
signal,
|
||||
}),
|
||||
);
|
||||
if (!response.ok) {
|
||||
throw new Error(`comments API returned ${response.status}`);
|
||||
}
|
||||
return await withGitHubApiTimeout(`proof comment response page ${page}`, timeoutMs, (signal) =>
|
||||
readBoundedGitHubApiJson(response, `proof comment response page ${page}`, undefined, {
|
||||
signal,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
async function fetchOversizedProofCommentPageIndividually({
|
||||
owner,
|
||||
repo,
|
||||
issueNumber,
|
||||
token,
|
||||
fetchImpl,
|
||||
timeoutMs,
|
||||
page,
|
||||
perPage,
|
||||
}) {
|
||||
const comments = [];
|
||||
const firstSinglePage = (page - 1) * perPage + 1;
|
||||
for (let offset = 0; offset < perPage; offset += 1) {
|
||||
try {
|
||||
const pageComments = await fetchProofCommentPage({
|
||||
owner,
|
||||
repo,
|
||||
issueNumber,
|
||||
token,
|
||||
fetchImpl,
|
||||
timeoutMs,
|
||||
page: firstSinglePage + offset,
|
||||
perPage: 1,
|
||||
});
|
||||
comments.push(...pageComments);
|
||||
if (pageComments.length === 0) {
|
||||
return { comments, exhausted: true };
|
||||
}
|
||||
} catch (error) {
|
||||
if (!isTooLargeBodyError(error)) {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
return { comments, exhausted: false };
|
||||
}
|
||||
|
||||
async function fetchProofCommentPageWithFallback(params) {
|
||||
try {
|
||||
const comments = await fetchProofCommentPage(params);
|
||||
return {
|
||||
comments,
|
||||
exhausted: comments.length < params.perPage,
|
||||
};
|
||||
} catch (error) {
|
||||
if (!isTooLargeBodyError(error) || params.perPage === 1) {
|
||||
throw error;
|
||||
}
|
||||
return await fetchOversizedProofCommentPageIndividually(params);
|
||||
}
|
||||
}
|
||||
|
||||
export async function fetchProofComments({
|
||||
owner,
|
||||
repo,
|
||||
issueNumber,
|
||||
tokens,
|
||||
fetchImpl = fetch,
|
||||
timeoutMs = DEFAULT_GITHUB_API_TIMEOUT_MS,
|
||||
}) {
|
||||
let lastError;
|
||||
for (const token of tokens.filter(Boolean)) {
|
||||
const comments = [];
|
||||
try {
|
||||
for (let page = 1; page <= MAX_PROOF_COMMENT_PAGES; page += 1) {
|
||||
const result = await fetchProofCommentPageWithFallback({
|
||||
owner,
|
||||
repo,
|
||||
issueNumber,
|
||||
token,
|
||||
fetchImpl,
|
||||
timeoutMs,
|
||||
page,
|
||||
perPage: PROOF_COMMENTS_PER_PAGE,
|
||||
});
|
||||
comments.push(...result.comments);
|
||||
if (result.exhausted) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
return comments;
|
||||
} catch (error) {
|
||||
lastError = error;
|
||||
}
|
||||
}
|
||||
throw toLintErrorObject(
|
||||
lastError ?? new Error("No GitHub token available for proof comment lookup."),
|
||||
"Non-Error thrown",
|
||||
);
|
||||
}
|
||||
|
||||
function isMainModule() {
|
||||
return Boolean(process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href);
|
||||
}
|
||||
@@ -162,14 +22,14 @@ function isMainModule() {
|
||||
async function main(env = process.env) {
|
||||
const eventPath = env.GITHUB_EVENT_PATH;
|
||||
if (!eventPath) {
|
||||
console.error("::error title=Real behavior proof failed::GITHUB_EVENT_PATH is not set.");
|
||||
console.error("::error title=PR context check failed::GITHUB_EVENT_PATH is not set.");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const event = JSON.parse(readFileSync(eventPath, "utf8"));
|
||||
const pullRequest = event.pull_request;
|
||||
if (!pullRequest) {
|
||||
console.log("No pull_request payload found; skipping real behavior proof gate.");
|
||||
console.log("No pull_request payload found; skipping PR context check.");
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
@@ -180,7 +40,7 @@ async function main(env = process.env) {
|
||||
try {
|
||||
if (await isMaintainerTeamMember({ token: appToken, org, login: authorLogin })) {
|
||||
console.log(
|
||||
`PR author @${authorLogin} is an active member of the ${org}/maintainer team; skipping real behavior proof gate.`,
|
||||
`PR author @${authorLogin} is an active member of the ${org}/maintainer team; skipping PR context check.`,
|
||||
);
|
||||
process.exit(0);
|
||||
}
|
||||
@@ -191,61 +51,17 @@ async function main(env = process.env) {
|
||||
}
|
||||
}
|
||||
|
||||
const evaluation = evaluateRealBehaviorProof({ pullRequest });
|
||||
const evaluation = evaluatePullRequestContext({ pullRequest });
|
||||
if (evaluation.passed) {
|
||||
console.log(evaluation.reason);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const repository = env.GITHUB_REPOSITORY;
|
||||
if ((appToken || env.GITHUB_TOKEN) && repository && pullRequest.number) {
|
||||
const [owner, repo] = repository.split("/");
|
||||
try {
|
||||
const comments = await fetchProofComments({
|
||||
owner,
|
||||
repo,
|
||||
issueNumber: pullRequest.number,
|
||||
tokens: [appToken, env.GITHUB_TOKEN],
|
||||
});
|
||||
|
||||
const clawSweeperEvaluation = evaluateClawSweeperExactHeadProof({
|
||||
pullRequest,
|
||||
comments,
|
||||
});
|
||||
if (clawSweeperEvaluation.passed) {
|
||||
console.log(clawSweeperEvaluation.reason);
|
||||
process.exit(0);
|
||||
}
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
`::warning title=Proof verdict comment lookup failed::${escapeCommandValue(error?.message ?? String(error))}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const message = `${evaluation.reason} Add after-fix evidence from a real OpenClaw setup in the PR body. Screenshots, recordings, terminal screenshots, console output, redacted runtime logs, linked artifacts, or copied live output count. Unit tests, mocks, snapshots, lint, typechecks, and CI are supplemental only. A maintainer can apply proof: override when appropriate.`;
|
||||
console.error(`::error title=Real behavior proof required::${escapeCommandValue(message)}`);
|
||||
const message = `${evaluation.reason} Add a concise problem statement and the most useful validation evidence to the PR body. Focused tests, CI results, screenshots, recordings, terminal output, live observations, redacted logs, and artifact links all count.`;
|
||||
console.error(`::error title=PR context required::${escapeCommandValue(message)}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
export const testing = {
|
||||
fetchProofComments,
|
||||
};
|
||||
|
||||
if (isMainModule()) {
|
||||
await main();
|
||||
}
|
||||
|
||||
function toLintErrorObject(value, fallbackMessage) {
|
||||
if (value instanceof Error) {
|
||||
return value;
|
||||
}
|
||||
if (typeof value === "string") {
|
||||
return new Error(value);
|
||||
}
|
||||
const error = new Error(fallbackMessage, { cause: value });
|
||||
if ((typeof value === "object" && value !== null) || typeof value === "function") {
|
||||
Object.assign(error, value);
|
||||
}
|
||||
return error;
|
||||
}
|
||||
|
||||
@@ -1,12 +1,10 @@
|
||||
// Shared real-behavior proof policy for GitHub PR checks and label decisions.
|
||||
// Shared PR context and evidence policy for GitHub checks and label decisions.
|
||||
import { readBoundedResponseText } from "../lib/bounded-response.mjs";
|
||||
|
||||
/** Label that lets maintainers override real-behavior proof requirements. */
|
||||
/** ClawSweeper-owned labels that OpenClaw preserves but does not mutate. */
|
||||
export const PROOF_OVERRIDE_LABEL = "proof: override";
|
||||
export const PROOF_SUPPLIED_LABEL = "proof: supplied";
|
||||
export const PROOF_SUFFICIENT_LABEL = "proof: sufficient";
|
||||
export const NEEDS_REAL_BEHAVIOR_PROOF_LABEL = "triage: needs-real-behavior-proof";
|
||||
export const MOCK_ONLY_PROOF_LABEL = "triage: mock-only-proof";
|
||||
export const NEEDS_PR_CONTEXT_LABEL = "triage: needs-pr-context";
|
||||
export const MAINTAINER_TEAM_SLUG = "maintainer";
|
||||
export const DEFAULT_GITHUB_API_TIMEOUT_MS = 30_000;
|
||||
export const GITHUB_API_RESPONSE_BODY_MAX_BYTES = 1024 * 1024;
|
||||
@@ -16,27 +14,10 @@ const CLAWSWEEPER_BOT_LOGINS = new Set(["clawsweeper[bot]", "openclaw-clawsweepe
|
||||
|
||||
const privilegedAuthorAssociations = new Set(["OWNER", "MEMBER", "COLLABORATOR"]);
|
||||
|
||||
const requiredProofFields = [
|
||||
{
|
||||
key: "behavior",
|
||||
names: ["Behavior or issue addressed", "Issue addressed", "Behavior addressed"],
|
||||
},
|
||||
{
|
||||
key: "environment",
|
||||
names: ["Real environment tested", "Environment tested", "Real setup tested"],
|
||||
},
|
||||
{
|
||||
key: "steps",
|
||||
names: [
|
||||
"Exact steps or command run after this patch",
|
||||
"Exact steps or command run after the patch",
|
||||
"Exact steps or command run after fix",
|
||||
"Steps run after the patch",
|
||||
"Command run after the patch",
|
||||
],
|
||||
},
|
||||
{
|
||||
key: "evidence",
|
||||
// Existing open PRs still use the previous structured section. Remove these
|
||||
// fallbacks once those PRs no longer need body revalidation.
|
||||
const legacyProofFields = {
|
||||
evidence: {
|
||||
names: [
|
||||
"Evidence after fix",
|
||||
"After-fix evidence",
|
||||
@@ -44,44 +25,33 @@ const requiredProofFields = [
|
||||
"Evidence",
|
||||
],
|
||||
},
|
||||
{
|
||||
key: "observedResult",
|
||||
names: ["Observed result after fix", "Observed result after the fix", "Observed result"],
|
||||
},
|
||||
{
|
||||
key: "notTested",
|
||||
names: ["What was not tested", "Not tested"],
|
||||
allowNone: true,
|
||||
problem: {
|
||||
names: ["Behavior or issue addressed", "Issue addressed", "Behavior addressed"],
|
||||
},
|
||||
};
|
||||
|
||||
const legacyProofFieldNames = [
|
||||
...legacyProofFields.problem.names,
|
||||
"Real environment tested",
|
||||
"Environment tested",
|
||||
"Real setup tested",
|
||||
"Exact steps or command run after this patch",
|
||||
"Exact steps or command run after the patch",
|
||||
"Exact steps or command run after fix",
|
||||
"Steps run after the patch",
|
||||
"Command run after the patch",
|
||||
...legacyProofFields.evidence.names,
|
||||
"Observed result after fix",
|
||||
"Observed result after the fix",
|
||||
"Observed result",
|
||||
"What was not tested",
|
||||
"Not tested",
|
||||
"Before evidence",
|
||||
"Before evidence optional",
|
||||
];
|
||||
|
||||
const allProofFieldNames = requiredProofFields
|
||||
.flatMap((field) => field.names)
|
||||
.concat(["Before evidence", "Before evidence optional"]);
|
||||
|
||||
const missingValueRegex =
|
||||
/^(?:n\/?a|not applicable|tbd|todo|unknown|unsure|none provided|no evidence|not tested|untested|did not test|didn't test|could not test|couldn't test|-|\[[^\]]*\])\.?$/i;
|
||||
|
||||
const standaloneMissingProofRegex =
|
||||
/^\s*(?:[-*]\s*)?(?:n\/?a|not applicable|not tested|untested|no evidence|did not test|didn't test|could not test|couldn't test)\s*\.?\s*$/im;
|
||||
|
||||
const mockOnlyEvidenceRegex =
|
||||
/\b(?:pnpm|npm|yarn|bun)\s+(?:run\s+)?(?:test|vitest|lint|typecheck|tsgo|build|check)\b|\b(?:vitest|unit tests?|mock(?:ed|s)?|snapshots?|lint|typechecks?|tsgo|ci(?:\s+passes?)?)\b/i;
|
||||
|
||||
const artifactEvidenceRegex =
|
||||
/!\[[^\]]*\]\([^)]+\)|github\.com\/user-attachments\/assets\/|github\.com\/[^/\s]+\/[^/\s]+\/actions\/runs\/\d+\/artifacts\/\d+|https?:\/\/\S+\.(?:png|jpe?g|gif|webp|mp4|mov|webm)\b/i;
|
||||
|
||||
const evidenceDescriptorRegex =
|
||||
/\b(?:screenshot|screen\s*recording|recording|terminal\s+(?:capture|screenshot|transcript|output)|console\s+(?:output|log)|runtime\s+logs?|redacted\s+logs?|live\s+output|actual\s+output|observed\s+output|stdout|stderr|stack trace|trace excerpt|log excerpt|linked\s+artifacts?|artifact\s+links?)\b|```[\s\S]*\n[\s\S]*\n```/i;
|
||||
|
||||
const liveCommandRegex =
|
||||
/\b(?:openclaw|node|docker|curl|gh|ssh|adb|xcrun|xcodebuild|open|npm\s+run|pnpm\s+openclaw)\b/i;
|
||||
|
||||
const mockOnlyEvidenceStripRegex =
|
||||
/\b(?:pnpm|npm|yarn|bun)\s+(?:run\s+)?(?:test|vitest|lint|typecheck|tsgo|build|check)\b|\b(?:vitest|unit tests?|mock(?:ed|s)?|snapshots?|lint|typechecks?|tsgo|ci(?:\s+passes?)?|tests?|passed|passes|green|success|succeeded|with|and|the|branch|only|output|transcript|capture|fenced)\b/gi;
|
||||
|
||||
const evidenceDescriptorStripRegex =
|
||||
/\b(?:screenshot|screen\s*recording|recording|terminal\s+(?:capture|screenshot|transcript|output)|console\s+(?:output|log)|runtime\s+logs?|redacted\s+logs?|live\s+output|actual\s+output|observed\s+output|stdout|stderr|stack trace|trace excerpt|log excerpt|linked\s+artifacts?|artifact\s+links?)\b/gi;
|
||||
/^(?:n\/?a|none|not applicable|tbd|todo|unknown|unsure|none provided|no evidence|not tested|untested|did not test|didn't test|could not test|couldn't test|-|(?:-{3,}|\*{3,}|_{3,})|\[[^\]]*\])\.?$/i;
|
||||
|
||||
function escapeRegex(text) {
|
||||
return text.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
@@ -142,12 +112,58 @@ function normalizeLineEndings(text = "") {
|
||||
return text.replace(/\r\n?/g, "\n");
|
||||
}
|
||||
|
||||
function labelNames(labels) {
|
||||
return new Set(
|
||||
(labels ?? [])
|
||||
.map((label) => (typeof label === "string" ? label : label?.name))
|
||||
.filter((label) => typeof label === "string"),
|
||||
);
|
||||
function maskHtmlComments(text) {
|
||||
let commentOpen = false;
|
||||
let fenceMarker = "";
|
||||
return text
|
||||
.split("\n")
|
||||
.map((line) => {
|
||||
if (fenceMarker) {
|
||||
fenceMarker = nextFenceMarker(line, fenceMarker);
|
||||
return line;
|
||||
}
|
||||
|
||||
let maskedLine = line;
|
||||
if (commentOpen) {
|
||||
const end = maskedLine.indexOf("-->");
|
||||
if (end < 0) {
|
||||
return maskedLine.replace(/[^\n]/g, " ");
|
||||
}
|
||||
maskedLine = `${maskedLine.slice(0, end + 3).replace(/[^\n]/g, " ")}${maskedLine.slice(end + 3)}`;
|
||||
commentOpen = false;
|
||||
}
|
||||
|
||||
if (nextFenceMarker(maskedLine, "")) {
|
||||
fenceMarker = nextFenceMarker(maskedLine, "");
|
||||
return maskedLine;
|
||||
}
|
||||
|
||||
let offset = 0;
|
||||
while (offset < maskedLine.length) {
|
||||
const start = maskedLine.indexOf("<!--", offset);
|
||||
if (start < 0) {
|
||||
break;
|
||||
}
|
||||
const end = maskedLine.indexOf("-->", start + 4);
|
||||
if (end < 0) {
|
||||
maskedLine = `${maskedLine.slice(0, start)}${maskedLine
|
||||
.slice(start)
|
||||
.replace(/[^\n]/g, " ")}`;
|
||||
commentOpen = true;
|
||||
break;
|
||||
}
|
||||
maskedLine = `${maskedLine.slice(0, start)}${maskedLine
|
||||
.slice(start, end + 3)
|
||||
.replace(/[^\n]/g, " ")}${maskedLine.slice(end + 3)}`;
|
||||
offset = end + 3;
|
||||
}
|
||||
return maskedLine;
|
||||
})
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
function stripHtmlComments(text) {
|
||||
return maskHtmlComments(text);
|
||||
}
|
||||
|
||||
function isAutomationUser(user = {}, fallbackLogin = "") {
|
||||
@@ -168,10 +184,6 @@ export function isExternalPullRequest(pullRequest) {
|
||||
return !privilegedAuthorAssociations.has(authorAssociation);
|
||||
}
|
||||
|
||||
export function hasProofOverride(labels) {
|
||||
return labelNames(labels).has(PROOF_OVERRIDE_LABEL);
|
||||
}
|
||||
|
||||
export async function isMaintainerTeamMember({
|
||||
token,
|
||||
org,
|
||||
@@ -241,27 +253,32 @@ function nextFenceMarker(line, fenceMarker) {
|
||||
return fenceMarker;
|
||||
}
|
||||
|
||||
function isMarkdownHeadingLine(line) {
|
||||
return /^#{1,6}\s+\S/.test(line);
|
||||
function markdownHeadingLevel(line) {
|
||||
return line.match(/^(#{1,6})\s+\S/)?.[1].length ?? 0;
|
||||
}
|
||||
|
||||
function extractMarkdownSections(headingRegex, body = "") {
|
||||
// Normalize CRLF → LF so regexes and section slicing see GitHub web-editor PR
|
||||
// bodies the same way as locally-authored Markdown.
|
||||
const normalizedBody = normalizeLineEndings(body);
|
||||
const headingBody = maskHtmlComments(normalizedBody);
|
||||
const sections = [];
|
||||
const matcher = new RegExp(headingRegex.source, headingRegex.flags.replaceAll("g", ""));
|
||||
let fenceMarker = "";
|
||||
let sectionHeadingLevel = 0;
|
||||
let sectionStart = -1;
|
||||
let lineStart = 0;
|
||||
for (const line of normalizedBody.split("\n")) {
|
||||
for (const line of headingBody.split("\n")) {
|
||||
const match = !fenceMarker ? line.match(matcher) : null;
|
||||
if (sectionStart >= 0 && !fenceMarker && isMarkdownHeadingLine(line)) {
|
||||
const headingLevel = !fenceMarker ? markdownHeadingLevel(line) : 0;
|
||||
if (sectionStart >= 0 && headingLevel > 0 && headingLevel <= sectionHeadingLevel) {
|
||||
sections.push(normalizedBody.slice(sectionStart, lineStart === 0 ? 0 : lineStart - 1).trim());
|
||||
sectionStart = -1;
|
||||
sectionHeadingLevel = 0;
|
||||
}
|
||||
if (match) {
|
||||
sectionStart = lineStart + (match.index ?? 0) + match[0].length;
|
||||
sectionHeadingLevel = headingLevel;
|
||||
}
|
||||
fenceMarker = nextFenceMarker(line, fenceMarker);
|
||||
lineStart += line.length + 1;
|
||||
@@ -272,18 +289,19 @@ function extractMarkdownSections(headingRegex, body = "") {
|
||||
return sections;
|
||||
}
|
||||
|
||||
function extractMarkdownSection(headingRegex, body = "") {
|
||||
return extractMarkdownSections(headingRegex, body)[0] ?? "";
|
||||
export function extractEvidenceSections(body = "") {
|
||||
return extractMarkdownSections(/^#{2,6}\s+evidence\b[^\n]*$/im, body);
|
||||
}
|
||||
|
||||
export function extractRealBehaviorProofSections(body = "") {
|
||||
export function hasAuthoredPullRequestSection(heading, body = "") {
|
||||
const headingPattern = new RegExp(`^#{2,6}\\s+${escapeRegex(heading)}\\b[^\\n]*$`, "im");
|
||||
return !isMissingValue(extractMarkdownSections(headingPattern, body).at(-1) ?? "");
|
||||
}
|
||||
|
||||
function extractLegacyProofSections(body = "") {
|
||||
return extractMarkdownSections(/^#{2,6}\s+real behavior proof\b[^\n]*$/im, body);
|
||||
}
|
||||
|
||||
function extractOutOfScopeFollowUpsSection(body = "") {
|
||||
return extractMarkdownSection(/^#{2,6}\s+out-of-scope follow-ups\b[^\n]*$/im, body);
|
||||
}
|
||||
|
||||
function fieldLineRegex(name) {
|
||||
return new RegExp(
|
||||
`^\\s*(?:[-*]\\s*)?(?:\\*\\*)?${escapeRegex(name)}(?:\\s*\\([^)]*\\))?(?:\\*\\*)?\\s*:\\s*(.*)$`,
|
||||
@@ -291,18 +309,18 @@ function fieldLineRegex(name) {
|
||||
);
|
||||
}
|
||||
|
||||
function proofFieldLineValue(line) {
|
||||
const matchingName = allProofFieldNames.find((name) => fieldLineRegex(name).test(line));
|
||||
function legacyProofFieldLineValue(line) {
|
||||
const matchingName = legacyProofFieldNames.find((name) => fieldLineRegex(name).test(line));
|
||||
const match = matchingName ? line.match(fieldLineRegex(matchingName)) : null;
|
||||
return match?.[1] ?? null;
|
||||
}
|
||||
|
||||
function isAnyProofFieldLine(line) {
|
||||
return proofFieldLineValue(line) !== null;
|
||||
function isAnyLegacyProofFieldLine(line) {
|
||||
return legacyProofFieldLineValue(line) !== null;
|
||||
}
|
||||
|
||||
function extractFieldValue(section, field) {
|
||||
const lines = normalizeLineEndings(section).split("\n");
|
||||
const lines = maskHtmlComments(normalizeLineEndings(section)).split("\n");
|
||||
let fenceMarker = "";
|
||||
for (let index = 0; index < lines.length; index += 1) {
|
||||
const line = lines[index];
|
||||
@@ -310,7 +328,7 @@ function extractFieldValue(section, field) {
|
||||
? field.names.find((name) => fieldLineRegex(name).test(line))
|
||||
: null;
|
||||
if (!matchingName) {
|
||||
const fenceLine = !fenceMarker ? (proofFieldLineValue(line) ?? line) : line;
|
||||
const fenceLine = !fenceMarker ? (legacyProofFieldLineValue(line) ?? line) : line;
|
||||
fenceMarker = nextFenceMarker(fenceLine, fenceMarker);
|
||||
continue;
|
||||
}
|
||||
@@ -320,7 +338,10 @@ function extractFieldValue(section, field) {
|
||||
fenceMarker = nextFenceMarker(valueLines[0], "");
|
||||
for (let next = index + 1; next < lines.length; next += 1) {
|
||||
const lineLocal = lines[next];
|
||||
if (!fenceMarker && (isMarkdownHeadingLine(lineLocal) || isAnyProofFieldLine(lineLocal))) {
|
||||
if (
|
||||
!fenceMarker &&
|
||||
(markdownHeadingLevel(lineLocal) > 0 || isAnyLegacyProofFieldLine(lineLocal))
|
||||
) {
|
||||
break;
|
||||
}
|
||||
valueLines.push(lineLocal);
|
||||
@@ -331,62 +352,28 @@ function extractFieldValue(section, field) {
|
||||
return "";
|
||||
}
|
||||
|
||||
function proofContentOutsideFences(section) {
|
||||
let fenceMarker = "";
|
||||
const contentLines = [];
|
||||
for (const line of normalizeLineEndings(section).split("\n")) {
|
||||
if (fenceMarker) {
|
||||
fenceMarker = nextFenceMarker(line, fenceMarker);
|
||||
continue;
|
||||
}
|
||||
const contentLine = proofFieldLineValue(line) ?? line;
|
||||
const nextMarker = nextFenceMarker(contentLine, fenceMarker);
|
||||
const isFenceBoundary = nextMarker !== fenceMarker;
|
||||
if (!isFenceBoundary) {
|
||||
contentLines.push(contentLine);
|
||||
}
|
||||
fenceMarker = nextMarker;
|
||||
}
|
||||
return contentLines.join("\n");
|
||||
}
|
||||
|
||||
function stripMarkdownFenceMarkers(value) {
|
||||
return normalizeLineEndings(value)
|
||||
return stripHtmlComments(normalizeLineEndings(value))
|
||||
.split("\n")
|
||||
.filter((line) => !/^ {0,3}(?:`{3,}|~{3,})(?:.*)?$/.test(line))
|
||||
.join("\n")
|
||||
.trim();
|
||||
}
|
||||
|
||||
function isMissingValue(value, field) {
|
||||
function isMissingValue(value) {
|
||||
const trimmed = stripMarkdownFenceMarkers(value).replace(/^\s*[-*]\s+/, "");
|
||||
if (!trimmed) {
|
||||
return true;
|
||||
}
|
||||
if (
|
||||
field.allowNone &&
|
||||
/^(?:none|nothing else|no known gaps|no additional gaps)$/i.test(trimmed)
|
||||
) {
|
||||
return false;
|
||||
}
|
||||
return missingValueRegex.test(trimmed);
|
||||
}
|
||||
|
||||
function hasNonMockEvidencePayload(value) {
|
||||
const payload = value
|
||||
.replace(evidenceDescriptorStripRegex, "")
|
||||
.replace(mockOnlyEvidenceStripRegex, "")
|
||||
.replace(/```(?:\w+)?|```/g, "")
|
||||
.replace(/[`$>:\-_.()[\]\s]+/g, "");
|
||||
return Boolean(payload);
|
||||
}
|
||||
|
||||
function result(status, reason, details = {}) {
|
||||
return {
|
||||
status,
|
||||
reason,
|
||||
applies: ["passed", "missing", "mock_only", "insufficient", "override"].includes(status),
|
||||
passed: ["passed", "skipped", "override", CLAWSWEEPER_PROOF_VERDICT_STATUS].includes(status),
|
||||
applies: ["passed", "missing", "insufficient"].includes(status),
|
||||
passed: ["passed", "skipped", CLAWSWEEPER_PROOF_VERDICT_STATUS].includes(status),
|
||||
...details,
|
||||
};
|
||||
}
|
||||
@@ -438,106 +425,47 @@ export function evaluateClawSweeperExactHeadProof({ pullRequest, comments = [] }
|
||||
if (hasClawSweeperExactHeadProof({ pullRequest, comments })) {
|
||||
return result(
|
||||
CLAWSWEEPER_PROOF_VERDICT_STATUS,
|
||||
"ClawSweeper accepted real behavior proof for the exact PR head.",
|
||||
"ClawSweeper accepted the PR evidence for the exact PR head.",
|
||||
);
|
||||
}
|
||||
return result("insufficient", "No exact-head ClawSweeper proof verdict was found.");
|
||||
}
|
||||
|
||||
function evaluateRealBehaviorProofSection(section, body) {
|
||||
const fields = Object.fromEntries(
|
||||
requiredProofFields.map((field) => [field.key, extractFieldValue(section, field)]),
|
||||
);
|
||||
if (!fields.notTested) {
|
||||
fields.notTested = extractOutOfScopeFollowUpsSection(body);
|
||||
}
|
||||
const missingFields = requiredProofFields
|
||||
.filter((field) => isMissingValue(fields[field.key] ?? "", field))
|
||||
.map((field) => field.key);
|
||||
if (missingFields.length > 0) {
|
||||
return result(
|
||||
"missing",
|
||||
`Real behavior proof is missing required field content: ${missingFields.join(", ")}.`,
|
||||
{ fields, missingFields },
|
||||
);
|
||||
}
|
||||
|
||||
const proofContent = proofContentOutsideFences(section);
|
||||
if (standaloneMissingProofRegex.test(proofContent)) {
|
||||
return result("insufficient", "Real behavior proof says the changed behavior was not tested.", {
|
||||
fields,
|
||||
});
|
||||
}
|
||||
|
||||
const evidenceContent = [fields.evidence, fields.observedResult].join("\n");
|
||||
const proofContentForMockDetection = [fields.evidence, fields.observedResult, fields.steps].join(
|
||||
"\n",
|
||||
);
|
||||
const hasArtifactEvidence = artifactEvidenceRegex.test(evidenceContent);
|
||||
const hasNonMockPayload = hasNonMockEvidencePayload(evidenceContent);
|
||||
const hasMockEvidenceSignal = mockOnlyEvidenceRegex.test(proofContentForMockDetection);
|
||||
if (hasMockEvidenceSignal && !hasArtifactEvidence && !hasNonMockPayload) {
|
||||
return result(
|
||||
"mock_only",
|
||||
"Unit tests, mocks, snapshots, lint, typechecks, and CI are supplemental and do not count as real behavior proof.",
|
||||
{ fields },
|
||||
);
|
||||
}
|
||||
|
||||
const hasRealEvidence =
|
||||
hasArtifactEvidence ||
|
||||
(evidenceDescriptorRegex.test(evidenceContent) && hasNonMockPayload) ||
|
||||
liveCommandRegex.test(evidenceContent);
|
||||
if (hasMockEvidenceSignal && !hasRealEvidence) {
|
||||
return result(
|
||||
"mock_only",
|
||||
"Unit tests, mocks, snapshots, lint, typechecks, and CI are supplemental and do not count as real behavior proof.",
|
||||
{ fields },
|
||||
);
|
||||
}
|
||||
|
||||
if (!hasRealEvidence) {
|
||||
return result(
|
||||
"insufficient",
|
||||
"Real behavior proof must include an after-fix screenshot, recording, terminal capture, console output, redacted runtime log, linked artifact, or copied live output.",
|
||||
{ fields },
|
||||
);
|
||||
}
|
||||
|
||||
return result("passed", "External PR includes after-fix real behavior proof.", { fields });
|
||||
}
|
||||
|
||||
export function evaluateRealBehaviorProof({ pullRequest, labels } = {}) {
|
||||
const currentLabels = labels ?? pullRequest?.labels ?? [];
|
||||
if (hasProofOverride(currentLabels)) {
|
||||
return result("override", `Maintainer override label ${PROOF_OVERRIDE_LABEL} is present.`);
|
||||
}
|
||||
export function evaluatePullRequestContext({ pullRequest } = {}) {
|
||||
if (!isExternalPullRequest(pullRequest)) {
|
||||
return result("skipped", "Maintainer, collaborator, or bot PRs do not require this gate.");
|
||||
}
|
||||
|
||||
const body = pullRequest?.body ?? "";
|
||||
const sections = extractRealBehaviorProofSections(body);
|
||||
if (sections.length === 0) {
|
||||
const latestLegacyProof = extractLegacyProofSections(body).at(-1) ?? "";
|
||||
const hasAuthoredProblem = hasAuthoredPullRequestSection("What Problem This Solves", body);
|
||||
const hasLegacyProblem = !isMissingValue(
|
||||
extractFieldValue(latestLegacyProof, legacyProofFields.problem),
|
||||
);
|
||||
const hasAuthoredEvidence = hasAuthoredPullRequestSection("Evidence", body);
|
||||
const hasLegacyEvidence = !isMissingValue(
|
||||
extractFieldValue(latestLegacyProof, legacyProofFields.evidence),
|
||||
);
|
||||
const missingSections = [];
|
||||
if (!hasAuthoredProblem && !hasLegacyProblem) {
|
||||
missingSections.push("What Problem This Solves");
|
||||
}
|
||||
if (!hasAuthoredEvidence && !hasLegacyEvidence) {
|
||||
missingSections.push("Evidence");
|
||||
}
|
||||
if (missingSections.length > 0) {
|
||||
return result(
|
||||
"missing",
|
||||
"External PRs must include a Real behavior proof section with after-fix evidence from a real setup.",
|
||||
`External PRs must include authored ${missingSections.join(" and ")} sections.`,
|
||||
{ missingSections },
|
||||
);
|
||||
}
|
||||
|
||||
const latestSection = sections.at(-1) ?? "";
|
||||
return evaluateRealBehaviorProofSection(latestSection, body);
|
||||
return result("passed", "External PR includes problem context and evidence.");
|
||||
}
|
||||
|
||||
export function labelsForRealBehaviorProof(evaluation) {
|
||||
if (evaluation.status === "passed") {
|
||||
return [PROOF_SUPPLIED_LABEL];
|
||||
}
|
||||
if (evaluation.status === "mock_only") {
|
||||
return [MOCK_ONLY_PROOF_LABEL];
|
||||
}
|
||||
export function labelsForPullRequestContext(evaluation) {
|
||||
if (evaluation.status === "missing" || evaluation.status === "insufficient") {
|
||||
return [NEEDS_REAL_BEHAVIOR_PROOF_LABEL];
|
||||
return [NEEDS_PR_CONTEXT_LABEL];
|
||||
}
|
||||
return [];
|
||||
}
|
||||
|
||||
@@ -152,7 +152,7 @@ Run relevant tests.
|
||||
Commit with conventional message.
|
||||
Push to PUSH_REMOTE.
|
||||
Open PR against SOURCE_REPO BASE_BRANCH.
|
||||
PR body: Summary + Verification + Fixes SOURCE_REPO#<n>.
|
||||
PR body: What Problem This Solves + Why This Change Was Made + User Impact + Evidence + visible Fixes SOURCE_REPO#<n>.
|
||||
Report PR URL or failure reason.
|
||||
Send completion/failure with openclaw message send if route provided.
|
||||
```
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
// Barnacle Auto Response tests cover barnacle auto response script behavior.
|
||||
import { readFileSync } from "node:fs";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
candidateLabels,
|
||||
@@ -9,32 +10,13 @@ import {
|
||||
import {
|
||||
PROOF_OVERRIDE_LABEL,
|
||||
PROOF_SUFFICIENT_LABEL,
|
||||
PROOF_SUPPLIED_LABEL,
|
||||
} from "../../scripts/github/real-behavior-proof-policy.mjs";
|
||||
|
||||
const blankTemplateBody = [
|
||||
"## Summary",
|
||||
"",
|
||||
"Describe the problem and fix in 2–5 bullets:",
|
||||
"",
|
||||
"- Problem:",
|
||||
"- Why it matters:",
|
||||
"- What changed:",
|
||||
"- What did NOT change (scope boundary):",
|
||||
"",
|
||||
"## Linked Issue/PR",
|
||||
"",
|
||||
"- Closes #",
|
||||
"- Related #",
|
||||
"",
|
||||
"## Root Cause (if applicable)",
|
||||
"",
|
||||
"- Root cause:",
|
||||
"",
|
||||
"## Regression Test Plan (if applicable)",
|
||||
"",
|
||||
"- Target test or file:",
|
||||
].join("\n");
|
||||
const clawSweeperProofSuppliedLabel = "proof: supplied";
|
||||
const blankTemplateBody = readFileSync(
|
||||
new URL("../../.github/pull_request_template.md", import.meta.url),
|
||||
"utf8",
|
||||
);
|
||||
|
||||
function pr(title: string, body = blankTemplateBody) {
|
||||
return {
|
||||
@@ -43,25 +25,20 @@ function pr(title: string, body = blankTemplateBody) {
|
||||
};
|
||||
}
|
||||
|
||||
function realBehaviorProofBody(evidence: string, overrides: Record<string, string> = {}) {
|
||||
function prContextBody(evidence: string, overrides: Record<string, string> = {}) {
|
||||
const fields = {
|
||||
behavior: "Gateway status now reports the Discord channel as ready.",
|
||||
environment: "macOS 15.4, Node 24, local OpenClaw gateway, redacted Discord token.",
|
||||
steps: "pnpm openclaw gateway restart and pnpm openclaw gateway status",
|
||||
problem: "Gateway status did not report the Discord channel as ready.",
|
||||
evidence,
|
||||
observedResult: "The gateway stayed connected and Discord reported ready.",
|
||||
notTested: "No known gaps.",
|
||||
...overrides,
|
||||
};
|
||||
return [
|
||||
"## Real behavior proof",
|
||||
"## What Problem This Solves",
|
||||
"",
|
||||
`- Behavior or issue addressed: ${fields.behavior}`,
|
||||
`- Real environment tested: ${fields.environment}`,
|
||||
`- Exact steps or command run after this patch: ${fields.steps}`,
|
||||
`- Evidence after fix: ${fields.evidence}`,
|
||||
`- Observed result after fix: ${fields.observedResult}`,
|
||||
`- What was not tested: ${fields.notTested}`,
|
||||
fields.problem,
|
||||
"",
|
||||
"## Evidence",
|
||||
"",
|
||||
fields.evidence,
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
@@ -269,9 +246,6 @@ describe("barnacle-auto-response", () => {
|
||||
expect(managedLabelSpecs["r: third-party-extension"].description).toContain("ClawHub");
|
||||
expect(managedLabelSpecs["r: bluebubbles"].description).toContain("deprecated");
|
||||
expect(managedLabelSpecs["r: too-many-prs"].description).toContain("twenty active PRs");
|
||||
expect(managedLabelSpecs[PROOF_SUPPLIED_LABEL].color).toBe("C2E0C6");
|
||||
expect(managedLabelSpecs[PROOF_SUFFICIENT_LABEL].color).toBe("0E8A16");
|
||||
|
||||
for (const label of Object.values(candidateLabels)) {
|
||||
expect(managedLabelSpecs).toHaveProperty(label);
|
||||
expect(managedLabelSpecs[label].description).toMatch(/^Candidate:/);
|
||||
@@ -285,7 +259,7 @@ describe("barnacle-auto-response", () => {
|
||||
|
||||
expect(labels).toEqual([
|
||||
candidateLabels.blankTemplate,
|
||||
candidateLabels.needsRealBehaviorProof,
|
||||
candidateLabels.needsPrContext,
|
||||
candidateLabels.lowSignalDocs,
|
||||
candidateLabels.docsDiscoverability,
|
||||
]);
|
||||
@@ -298,64 +272,76 @@ describe("barnacle-auto-response", () => {
|
||||
|
||||
expect(labels).toEqual([
|
||||
candidateLabels.blankTemplate,
|
||||
candidateLabels.needsRealBehaviorProof,
|
||||
candidateLabels.needsPrContext,
|
||||
candidateLabels.testOnlyNoBug,
|
||||
]);
|
||||
});
|
||||
|
||||
it("labels external PRs that are missing real behavior proof", () => {
|
||||
it("uses the latest case-insensitive context sections after template boilerplate", () => {
|
||||
const body = [
|
||||
blankTemplateBody,
|
||||
"## What problem this solves",
|
||||
"",
|
||||
"Gateway status did not report the Discord channel as ready.",
|
||||
"",
|
||||
"## evidence",
|
||||
"",
|
||||
"pnpm test passed.",
|
||||
].join("\n\n");
|
||||
const labels = classifyPullRequestCandidateLabels(pr("Fix gateway status", body), [
|
||||
file("src/gateway/server.ts"),
|
||||
]);
|
||||
|
||||
expect(labels).not.toContain(clawSweeperProofSuppliedLabel);
|
||||
expect(labels).not.toContain(candidateLabels.blankTemplate);
|
||||
expect(labels).not.toContain(candidateLabels.needsPrContext);
|
||||
});
|
||||
|
||||
it("labels external PRs that are missing context or evidence", () => {
|
||||
const labels = classifyPullRequestCandidateLabels(pr("Fix gateway startup"), [
|
||||
file("src/gateway/server.ts"),
|
||||
]);
|
||||
|
||||
expect(labels).toContain(candidateLabels.needsRealBehaviorProof);
|
||||
expect(labels).not.toContain(candidateLabels.mockOnlyProof);
|
||||
expect(labels).toContain(candidateLabels.needsPrContext);
|
||||
});
|
||||
|
||||
it("labels external PRs whose proof is only tests or mocks", () => {
|
||||
it("accepts focused test evidence without assigning a proof label", () => {
|
||||
const labels = classifyPullRequestCandidateLabels(
|
||||
pr("Fix gateway startup", prContextBody("pnpm test passed with Vitest mocks.")),
|
||||
[file("src/gateway/server.ts")],
|
||||
);
|
||||
|
||||
expect(labels).not.toContain(clawSweeperProofSuppliedLabel);
|
||||
expect(labels).not.toContain(candidateLabels.needsPrContext);
|
||||
});
|
||||
|
||||
it("accepts external PR evidence without assigning a proof label", () => {
|
||||
const labels = classifyPullRequestCandidateLabels(
|
||||
pr(
|
||||
"Fix gateway startup",
|
||||
realBehaviorProofBody("pnpm test passed with Vitest mocks.", {
|
||||
steps: "pnpm test",
|
||||
observedResult: "CI passes.",
|
||||
}),
|
||||
prContextBody(""),
|
||||
),
|
||||
[file("src/gateway/server.ts")],
|
||||
);
|
||||
|
||||
expect(labels).toContain(candidateLabels.mockOnlyProof);
|
||||
expect(labels).not.toContain(candidateLabels.needsRealBehaviorProof);
|
||||
expect(labels).not.toContain(clawSweeperProofSuppliedLabel);
|
||||
expect(labels).not.toContain(candidateLabels.needsPrContext);
|
||||
});
|
||||
|
||||
it("labels external PRs that include real behavior proof as supplied", () => {
|
||||
it("accepts CRLF-formatted screenshot evidence without assigning a proof label", () => {
|
||||
const labels = classifyPullRequestCandidateLabels(
|
||||
pr(
|
||||
"Fix gateway startup",
|
||||
realBehaviorProofBody(""),
|
||||
prContextBody("").replace(
|
||||
/\n/g,
|
||||
"\r\n",
|
||||
),
|
||||
),
|
||||
[file("src/gateway/server.ts")],
|
||||
);
|
||||
|
||||
expect(labels).toContain(PROOF_SUPPLIED_LABEL);
|
||||
expect(labels).not.toContain(candidateLabels.needsRealBehaviorProof);
|
||||
expect(labels).not.toContain(candidateLabels.mockOnlyProof);
|
||||
});
|
||||
|
||||
it("labels CRLF-formatted external PRs with screenshot proof as supplied", () => {
|
||||
const labels = classifyPullRequestCandidateLabels(
|
||||
pr(
|
||||
"Fix gateway startup",
|
||||
realBehaviorProofBody(
|
||||
"",
|
||||
).replace(/\n/g, "\r\n"),
|
||||
),
|
||||
[file("src/gateway/server.ts")],
|
||||
);
|
||||
|
||||
expect(labels).toContain(PROOF_SUPPLIED_LABEL);
|
||||
expect(labels).not.toContain(candidateLabels.needsRealBehaviorProof);
|
||||
expect(labels).not.toContain(candidateLabels.mockOnlyProof);
|
||||
expect(labels).not.toContain(clawSweeperProofSuppliedLabel);
|
||||
expect(labels).not.toContain(candidateLabels.needsPrContext);
|
||||
});
|
||||
|
||||
it("uses linked issues as context and suppresses low-signal docs labels", () => {
|
||||
@@ -687,7 +673,7 @@ describe("barnacle-auto-response", () => {
|
||||
expect(calls.addLabels).toStrictEqual([
|
||||
expectedAddLabels(123, [
|
||||
candidateLabels.blankTemplate,
|
||||
candidateLabels.needsRealBehaviorProof,
|
||||
candidateLabels.needsPrContext,
|
||||
candidateLabels.refactorOnly,
|
||||
candidateLabels.dirtyCandidate,
|
||||
]),
|
||||
@@ -710,7 +696,7 @@ describe("barnacle-auto-response", () => {
|
||||
expect(calls.addLabels).toStrictEqual([
|
||||
expectedAddLabels(123, [
|
||||
candidateLabels.blankTemplate,
|
||||
candidateLabels.needsRealBehaviorProof,
|
||||
candidateLabels.needsPrContext,
|
||||
candidateLabels.refactorOnly,
|
||||
]),
|
||||
]);
|
||||
@@ -718,27 +704,30 @@ describe("barnacle-auto-response", () => {
|
||||
expect(calls.update).toStrictEqual([]);
|
||||
});
|
||||
|
||||
it("removes stale structural proof labels but preserves sufficient proof when override is present", async () => {
|
||||
it("removes stale context labels without changing ClawSweeper proof labels", async () => {
|
||||
const { calls, github } = barnacleGithub([file("src/gateway/server.ts")]);
|
||||
|
||||
await runBarnacleAutoResponse({
|
||||
github,
|
||||
context: barnacleContext({}, [
|
||||
candidateLabels.needsRealBehaviorProof,
|
||||
candidateLabels.mockOnlyProof,
|
||||
PROOF_SUPPLIED_LABEL,
|
||||
PROOF_SUFFICIENT_LABEL,
|
||||
PROOF_OVERRIDE_LABEL,
|
||||
]),
|
||||
context: barnacleContext(
|
||||
{
|
||||
body: prContextBody("pnpm test passed."),
|
||||
},
|
||||
[
|
||||
candidateLabels.needsPrContext,
|
||||
"triage: mock-only-proof",
|
||||
clawSweeperProofSuppliedLabel,
|
||||
PROOF_SUFFICIENT_LABEL,
|
||||
PROOF_OVERRIDE_LABEL,
|
||||
],
|
||||
),
|
||||
core: {
|
||||
info: () => undefined,
|
||||
},
|
||||
});
|
||||
|
||||
expect(calls.removeLabel).toStrictEqual([
|
||||
expectedRemoveLabel(123, candidateLabels.needsRealBehaviorProof),
|
||||
expectedRemoveLabel(123, candidateLabels.mockOnlyProof),
|
||||
expectedRemoveLabel(123, PROOF_SUPPLIED_LABEL),
|
||||
expectedRemoveLabel(123, candidateLabels.needsPrContext),
|
||||
]);
|
||||
expect(calls.update).toStrictEqual([]);
|
||||
});
|
||||
@@ -750,9 +739,7 @@ describe("barnacle-auto-response", () => {
|
||||
github,
|
||||
context: barnacleContext(
|
||||
{
|
||||
body: realBehaviorProofBody(
|
||||
"",
|
||||
),
|
||||
body: prContextBody(""),
|
||||
},
|
||||
[PROOF_OVERRIDE_LABEL, PROOF_SUFFICIENT_LABEL],
|
||||
{
|
||||
@@ -771,18 +758,16 @@ describe("barnacle-auto-response", () => {
|
||||
expect(calls.update).toEqual([]);
|
||||
});
|
||||
|
||||
it("removes stale negative proof labels and adds supplied when proof is present", async () => {
|
||||
it("removes stale context labels without adding proof labels", async () => {
|
||||
const { calls, github } = barnacleGithub([file("src/gateway/server.ts")]);
|
||||
|
||||
await runBarnacleAutoResponse({
|
||||
github,
|
||||
context: barnacleContext(
|
||||
{
|
||||
body: realBehaviorProofBody(
|
||||
"",
|
||||
),
|
||||
body: prContextBody(""),
|
||||
},
|
||||
[candidateLabels.needsRealBehaviorProof, candidateLabels.mockOnlyProof],
|
||||
[candidateLabels.needsPrContext, "triage: mock-only-proof"],
|
||||
),
|
||||
core: {
|
||||
info: () => undefined,
|
||||
@@ -790,14 +775,13 @@ describe("barnacle-auto-response", () => {
|
||||
});
|
||||
|
||||
expect(calls.removeLabel).toStrictEqual([
|
||||
expectedRemoveLabel(123, candidateLabels.needsRealBehaviorProof),
|
||||
expectedRemoveLabel(123, candidateLabels.mockOnlyProof),
|
||||
expectedRemoveLabel(123, candidateLabels.needsPrContext),
|
||||
]);
|
||||
expect(calls.addLabels).toStrictEqual([expectedAddLabels(123, [PROOF_SUPPLIED_LABEL])]);
|
||||
expect(calls.addLabels).toStrictEqual([]);
|
||||
});
|
||||
|
||||
it.each(["edited", "synchronize"])(
|
||||
"removes stale sufficient proof label after PR %s events",
|
||||
"preserves ClawSweeper sufficient proof labels after PR %s events",
|
||||
async (action) => {
|
||||
const { calls, github } = barnacleGithub([file("src/gateway/server.ts")]);
|
||||
|
||||
@@ -805,11 +789,11 @@ describe("barnacle-auto-response", () => {
|
||||
github,
|
||||
context: barnacleContext(
|
||||
{
|
||||
body: realBehaviorProofBody(
|
||||
body: prContextBody(
|
||||
"",
|
||||
),
|
||||
},
|
||||
[PROOF_SUPPLIED_LABEL, PROOF_SUFFICIENT_LABEL],
|
||||
[clawSweeperProofSuppliedLabel, PROOF_SUFFICIENT_LABEL],
|
||||
{ action },
|
||||
),
|
||||
core: {
|
||||
@@ -817,7 +801,7 @@ describe("barnacle-auto-response", () => {
|
||||
},
|
||||
});
|
||||
|
||||
expect(calls.removeLabel).toEqual([expectedRemoveLabel(123, PROOF_SUFFICIENT_LABEL)]);
|
||||
expect(calls.removeLabel).toEqual([]);
|
||||
},
|
||||
);
|
||||
|
||||
@@ -858,7 +842,7 @@ describe("barnacle-auto-response", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("removes sufficient proof on synchronize when the matching marker is forged", async () => {
|
||||
it("preserves sufficient proof on synchronize when the matching marker is forged", async () => {
|
||||
const headSha = "06ee95df6608d29a395c52ba8ab53fdd93a9dc4f";
|
||||
const { calls, github } = barnacleGithub([file("src/gateway/server.ts")], {
|
||||
comments: [
|
||||
@@ -887,7 +871,7 @@ describe("barnacle-auto-response", () => {
|
||||
},
|
||||
});
|
||||
|
||||
expect(calls.removeLabel).toEqual([expectedRemoveLabel(123, PROOF_SUFFICIENT_LABEL)]);
|
||||
expect(calls.removeLabel).toEqual([]);
|
||||
});
|
||||
|
||||
it("preserves stale sufficient proof while ClawSweeper automerge owns the PR", async () => {
|
||||
@@ -901,11 +885,9 @@ describe("barnacle-auto-response", () => {
|
||||
ref: "fix/memory-search-event-loop-yield-81172",
|
||||
sha: "0ede3d716805e7d2ced8df37c6666af510dc9e19",
|
||||
},
|
||||
body: realBehaviorProofBody(
|
||||
"",
|
||||
),
|
||||
body: prContextBody(""),
|
||||
},
|
||||
[PROOF_SUPPLIED_LABEL, PROOF_SUFFICIENT_LABEL, "clawsweeper:automerge"],
|
||||
[clawSweeperProofSuppliedLabel, PROOF_SUFFICIENT_LABEL, "clawsweeper:automerge"],
|
||||
{ action: "synchronize" },
|
||||
),
|
||||
core: {
|
||||
@@ -927,11 +909,9 @@ describe("barnacle-auto-response", () => {
|
||||
ref: "clawsweeper/repair-pr-83758",
|
||||
sha: "0ede3d716805e7d2ced8df37c6666af510dc9e19",
|
||||
},
|
||||
body: realBehaviorProofBody(
|
||||
"",
|
||||
),
|
||||
body: prContextBody(""),
|
||||
},
|
||||
[PROOF_SUPPLIED_LABEL, PROOF_SUFFICIENT_LABEL],
|
||||
[clawSweeperProofSuppliedLabel, PROOF_SUFFICIENT_LABEL],
|
||||
{ action: "synchronize" },
|
||||
),
|
||||
core: {
|
||||
@@ -949,15 +929,13 @@ describe("barnacle-auto-response", () => {
|
||||
github,
|
||||
context: barnacleContext(
|
||||
{
|
||||
body: realBehaviorProofBody(
|
||||
"",
|
||||
),
|
||||
body: prContextBody(""),
|
||||
user: {
|
||||
login: "clawsweeper[bot]",
|
||||
type: "Bot",
|
||||
},
|
||||
},
|
||||
[PROOF_SUPPLIED_LABEL, PROOF_SUFFICIENT_LABEL],
|
||||
[clawSweeperProofSuppliedLabel, PROOF_SUFFICIENT_LABEL],
|
||||
{ action: "synchronize" },
|
||||
),
|
||||
core: {
|
||||
@@ -977,11 +955,9 @@ describe("barnacle-auto-response", () => {
|
||||
github,
|
||||
context: barnacleContext(
|
||||
{
|
||||
body: realBehaviorProofBody(
|
||||
"",
|
||||
),
|
||||
body: prContextBody(""),
|
||||
},
|
||||
[PROOF_SUPPLIED_LABEL, PROOF_SUFFICIENT_LABEL],
|
||||
[clawSweeperProofSuppliedLabel, PROOF_SUFFICIENT_LABEL],
|
||||
{
|
||||
action: "labeled",
|
||||
label: { name: PROOF_SUFFICIENT_LABEL },
|
||||
@@ -996,7 +972,7 @@ describe("barnacle-auto-response", () => {
|
||||
expect(calls.removeLabel).toEqual([]);
|
||||
});
|
||||
|
||||
it("preserves sufficient proof on unrelated label events even without body proof", async () => {
|
||||
it("adds missing context labels even when proof is sufficient", async () => {
|
||||
const { calls, github } = barnacleGithub([file("src/gateway/server.ts")]);
|
||||
|
||||
await runBarnacleAutoResponse({
|
||||
@@ -1012,19 +988,19 @@ describe("barnacle-auto-response", () => {
|
||||
});
|
||||
|
||||
expect(calls.removeLabel).toEqual([]);
|
||||
expect(calls.addLabels.flatMap((call) => call.labels)).not.toContain(
|
||||
candidateLabels.needsRealBehaviorProof,
|
||||
expect(calls.addLabels.flatMap((call) => call.labels)).toContain(
|
||||
candidateLabels.needsPrContext,
|
||||
);
|
||||
});
|
||||
|
||||
it("does not re-add negative proof labels while sufficient proof is present", async () => {
|
||||
it("re-adds missing context labels while sufficient proof is present", async () => {
|
||||
const { calls, github } = barnacleGithub([file("src/gateway/server.ts")]);
|
||||
|
||||
await runBarnacleAutoResponse({
|
||||
github,
|
||||
context: barnacleContext({}, [PROOF_SUFFICIENT_LABEL], {
|
||||
action: "unlabeled",
|
||||
label: { name: candidateLabels.needsRealBehaviorProof },
|
||||
label: { name: candidateLabels.needsPrContext },
|
||||
sender: { login: "maintainer", type: "User" },
|
||||
}),
|
||||
core: {
|
||||
@@ -1033,35 +1009,29 @@ describe("barnacle-auto-response", () => {
|
||||
});
|
||||
|
||||
expect(calls.removeLabel).toEqual([]);
|
||||
expect(calls.addLabels.flatMap((call) => call.labels)).not.toContain(
|
||||
candidateLabels.needsRealBehaviorProof,
|
||||
expect(calls.addLabels.flatMap((call) => call.labels)).toContain(
|
||||
candidateLabels.needsPrContext,
|
||||
);
|
||||
});
|
||||
|
||||
it("removes negative proof labels when sufficient proof is already present", async () => {
|
||||
it("keeps context labels when sufficient proof is already present", async () => {
|
||||
const { calls, github } = barnacleGithub([file("src/gateway/server.ts")]);
|
||||
|
||||
await runBarnacleAutoResponse({
|
||||
github,
|
||||
context: barnacleContext(
|
||||
{},
|
||||
[PROOF_SUFFICIENT_LABEL, candidateLabels.needsRealBehaviorProof],
|
||||
{
|
||||
action: "labeled",
|
||||
label: { name: "status: ready for maintainer look" },
|
||||
sender: { login: "openclaw-clawsweeper[bot]", type: "Bot" },
|
||||
},
|
||||
),
|
||||
context: barnacleContext({}, [PROOF_SUFFICIENT_LABEL, candidateLabels.needsPrContext], {
|
||||
action: "labeled",
|
||||
label: { name: "status: ready for maintainer look" },
|
||||
sender: { login: "openclaw-clawsweeper[bot]", type: "Bot" },
|
||||
}),
|
||||
core: {
|
||||
info: () => undefined,
|
||||
},
|
||||
});
|
||||
|
||||
expect(calls.removeLabel).toEqual([
|
||||
expectedRemoveLabel(123, candidateLabels.needsRealBehaviorProof),
|
||||
]);
|
||||
expect(calls.removeLabel).toEqual([]);
|
||||
expect(calls.addLabels.flatMap((call) => call.labels)).not.toContain(
|
||||
candidateLabels.needsRealBehaviorProof,
|
||||
candidateLabels.needsPrContext,
|
||||
);
|
||||
});
|
||||
|
||||
@@ -1164,7 +1134,7 @@ describe("barnacle-auto-response", () => {
|
||||
expect(calls.removeLabel).toStrictEqual([expectedRemoveLabel(123, "trigger-response")]);
|
||||
expect(calls.createComment).toHaveLength(1);
|
||||
expect(calls.createComment[0]?.issue_number).toBe(123);
|
||||
expect(calls.createComment[0]?.body).toContain("does not include real behavior proof");
|
||||
expect(calls.createComment[0]?.body).toContain("lacks a clear problem statement or evidence");
|
||||
expect(calls.update).toStrictEqual([expectedIssueUpdate(123, "closed")]);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,196 +0,0 @@
|
||||
// Real Behavior Proof Check tests cover real behavior proof check script behavior.
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { fetchProofComments } from "../../scripts/github/real-behavior-proof-check.mjs";
|
||||
|
||||
function stalledResponse() {
|
||||
let keepAlive: ReturnType<typeof setTimeout> | undefined;
|
||||
const reader = {
|
||||
read: () =>
|
||||
new Promise<ReadableStreamReadResult<Uint8Array>>(() => {
|
||||
keepAlive = setTimeout(() => {}, 10_000);
|
||||
}),
|
||||
cancel: vi.fn(() => {
|
||||
if (keepAlive) {
|
||||
clearTimeout(keepAlive);
|
||||
}
|
||||
return Promise.resolve();
|
||||
}),
|
||||
releaseLock: vi.fn(),
|
||||
};
|
||||
return {
|
||||
ok: true,
|
||||
status: 200,
|
||||
headers: new Headers(),
|
||||
body: {
|
||||
getReader: () => reader,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function contentLengthResponse(contentLength: number) {
|
||||
const cancel = vi.fn(() => Promise.resolve());
|
||||
return {
|
||||
ok: true,
|
||||
status: 200,
|
||||
headers: new Headers({ "content-length": String(contentLength) }),
|
||||
body: { cancel },
|
||||
cancel,
|
||||
};
|
||||
}
|
||||
|
||||
function chunkedResponse(chunks: Uint8Array[]) {
|
||||
const cancel = vi.fn(() => Promise.resolve());
|
||||
const read = vi.fn();
|
||||
for (const chunk of chunks) {
|
||||
read.mockResolvedValueOnce({ done: false, value: chunk });
|
||||
}
|
||||
read.mockResolvedValueOnce({ done: true, value: undefined });
|
||||
return {
|
||||
ok: true,
|
||||
status: 200,
|
||||
headers: new Headers(),
|
||||
body: {
|
||||
getReader: () => ({
|
||||
read,
|
||||
cancel,
|
||||
releaseLock: vi.fn(),
|
||||
}),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe("real-behavior-proof-check GitHub lookups", () => {
|
||||
it("aborts stalled proof comment fetches", async () => {
|
||||
const fetch = vi.fn((_url: URL, init: RequestInit) => {
|
||||
return new Promise((_resolve, reject) => {
|
||||
init.signal?.addEventListener("abort", () =>
|
||||
reject(toLintErrorObject(init.signal?.reason, "Non-Error rejection")),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
await expect(
|
||||
fetchProofComments({
|
||||
fetchImpl: fetch as typeof globalThis.fetch,
|
||||
issueNumber: 123,
|
||||
owner: "openclaw",
|
||||
repo: "openclaw",
|
||||
timeoutMs: 5,
|
||||
tokens: ["tok"],
|
||||
}),
|
||||
).rejects.toThrow(/proof comment lookup page 1 timed out after 5ms/);
|
||||
});
|
||||
|
||||
it("times out stalled proof comment response bodies", async () => {
|
||||
const fetch = vi.fn().mockResolvedValue(stalledResponse());
|
||||
|
||||
await expect(
|
||||
fetchProofComments({
|
||||
fetchImpl: fetch as typeof globalThis.fetch,
|
||||
issueNumber: 123,
|
||||
owner: "openclaw",
|
||||
repo: "openclaw",
|
||||
timeoutMs: 5,
|
||||
tokens: ["tok"],
|
||||
}),
|
||||
).rejects.toThrow(/proof comment response page 1 timed out after 5ms/);
|
||||
});
|
||||
|
||||
it("skips oversized proof comment bodies by content length after narrow fallback", async () => {
|
||||
const response = contentLengthResponse(1024 * 1024 + 1);
|
||||
const fetch = vi.fn((url: URL) => {
|
||||
const perPage = url.searchParams.get("per_page");
|
||||
const page = url.searchParams.get("page");
|
||||
if ((perPage === "100" && page === "1") || (perPage === "1" && page === "1")) {
|
||||
return Promise.resolve(response);
|
||||
}
|
||||
return Promise.resolve(new Response("[]", { status: 200 }));
|
||||
});
|
||||
|
||||
await expect(
|
||||
fetchProofComments({
|
||||
fetchImpl: fetch as typeof globalThis.fetch,
|
||||
issueNumber: 123,
|
||||
owner: "openclaw",
|
||||
repo: "openclaw",
|
||||
tokens: ["tok"],
|
||||
}),
|
||||
).resolves.toEqual([]);
|
||||
expect(response.cancel).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("skips oversized streamed proof comment bodies after narrow fallback", async () => {
|
||||
const encoder = new TextEncoder();
|
||||
const oversizedResponse = () =>
|
||||
chunkedResponse([
|
||||
encoder.encode("["),
|
||||
encoder.encode(" ".repeat(1024 * 1024)),
|
||||
encoder.encode("]"),
|
||||
]);
|
||||
const fetch = vi.fn((url: URL) => {
|
||||
const perPage = url.searchParams.get("per_page");
|
||||
const page = url.searchParams.get("page");
|
||||
if ((perPage === "100" && page === "1") || (perPage === "1" && page === "1")) {
|
||||
return Promise.resolve(oversizedResponse());
|
||||
}
|
||||
return Promise.resolve(new Response("[]", { status: 200 }));
|
||||
});
|
||||
|
||||
await expect(
|
||||
fetchProofComments({
|
||||
fetchImpl: fetch as typeof globalThis.fetch,
|
||||
issueNumber: 123,
|
||||
owner: "openclaw",
|
||||
repo: "openclaw",
|
||||
tokens: ["tok"],
|
||||
}),
|
||||
).resolves.toEqual([]);
|
||||
});
|
||||
|
||||
it("falls back to one-comment pages when a bulk comment page is oversized", async () => {
|
||||
const fetch = vi.fn((url: URL) => {
|
||||
const perPage = url.searchParams.get("per_page");
|
||||
const page = url.searchParams.get("page");
|
||||
if (perPage === "100" && page === "1") {
|
||||
return Promise.resolve(contentLengthResponse(1024 * 1024 + 1));
|
||||
}
|
||||
if (perPage === "1" && page === "1") {
|
||||
return Promise.resolve(contentLengthResponse(1024 * 1024 + 1));
|
||||
}
|
||||
if (perPage === "1" && page === "2") {
|
||||
return Promise.resolve(new Response('[{"id":2,"body":"trusted proof"}]', { status: 200 }));
|
||||
}
|
||||
return Promise.resolve(new Response("[]", { status: 200 }));
|
||||
});
|
||||
|
||||
await expect(
|
||||
fetchProofComments({
|
||||
fetchImpl: fetch as typeof globalThis.fetch,
|
||||
issueNumber: 123,
|
||||
owner: "openclaw",
|
||||
repo: "openclaw",
|
||||
tokens: ["tok"],
|
||||
}),
|
||||
).resolves.toEqual([{ id: 2, body: "trusted proof" }]);
|
||||
|
||||
expect(
|
||||
fetch.mock.calls.map(
|
||||
([url]) => `${url.searchParams.get("per_page")}:${url.searchParams.get("page")}`,
|
||||
),
|
||||
).toEqual(["100:1", "1:1", "1:2", "1:3"]);
|
||||
});
|
||||
});
|
||||
|
||||
function toLintErrorObject(value: unknown, fallbackMessage: string): Error {
|
||||
if (value instanceof Error) {
|
||||
return value;
|
||||
}
|
||||
if (typeof value === "string") {
|
||||
return new Error(value);
|
||||
}
|
||||
const error = new Error(fallbackMessage, { cause: value });
|
||||
if ((typeof value === "object" && value !== null) || typeof value === "function") {
|
||||
Object.assign(error, value);
|
||||
}
|
||||
return error;
|
||||
}
|
||||
@@ -1,18 +1,22 @@
|
||||
// Real Behavior Proof Policy tests cover real behavior proof policy script behavior.
|
||||
// PR Context And Evidence Policy tests cover GitHub PR-body policy behavior.
|
||||
import { readFileSync } from "node:fs";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
MOCK_ONLY_PROOF_LABEL,
|
||||
NEEDS_REAL_BEHAVIOR_PROOF_LABEL,
|
||||
NEEDS_PR_CONTEXT_LABEL,
|
||||
PROOF_OVERRIDE_LABEL,
|
||||
PROOF_SUPPLIED_LABEL,
|
||||
evaluateClawSweeperExactHeadProof,
|
||||
evaluateRealBehaviorProof,
|
||||
evaluatePullRequestContext,
|
||||
hasClawSweeperExactHeadProof,
|
||||
isMaintainerTeamMember,
|
||||
labelsForRealBehaviorProof,
|
||||
labelsForPullRequestContext,
|
||||
readBoundedGitHubApiJson,
|
||||
} from "../../scripts/github/real-behavior-proof-policy.mjs";
|
||||
|
||||
const blankTemplateBody = readFileSync(
|
||||
new URL("../../.github/pull_request_template.md", import.meta.url),
|
||||
"utf8",
|
||||
);
|
||||
|
||||
function externalPr(body: string, overrides: Record<string, unknown> = {}) {
|
||||
return {
|
||||
body,
|
||||
@@ -28,23 +32,18 @@ function externalPr(body: string, overrides: Record<string, unknown> = {}) {
|
||||
|
||||
function proofBody(evidence: string, overrides: Record<string, string> = {}) {
|
||||
const fields = {
|
||||
behavior: "Gateway startup no longer drops the configured Discord channel.",
|
||||
environment: "macOS 15.4, Node 24, local OpenClaw gateway with a redacted Discord token.",
|
||||
steps: "pnpm openclaw gateway restart, then pnpm openclaw gateway status",
|
||||
problem: "The gateway dropped the configured Discord channel during startup.",
|
||||
evidence,
|
||||
observedResult: "The gateway stayed connected and the Discord channel showed ready.",
|
||||
notTested: "No known gaps.",
|
||||
...overrides,
|
||||
};
|
||||
return [
|
||||
"## Real behavior proof",
|
||||
"## What Problem This Solves",
|
||||
"",
|
||||
`- Behavior or issue addressed: ${fields.behavior}`,
|
||||
`- Real environment tested: ${fields.environment}`,
|
||||
`- Exact steps or command run after this patch: ${fields.steps}`,
|
||||
`- Evidence after fix: ${fields.evidence}`,
|
||||
`- Observed result after fix: ${fields.observedResult}`,
|
||||
`- What was not tested: ${fields.notTested}`,
|
||||
fields.problem,
|
||||
"",
|
||||
"## Evidence",
|
||||
"",
|
||||
fields.evidence,
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
@@ -109,17 +108,17 @@ describe("real-behavior-proof-policy", () => {
|
||||
["Terminal transcript:", "```text", "$ openclaw gateway status", "discord ready", "```"].join(
|
||||
"\n",
|
||||
),
|
||||
])("passes external PRs with real after-fix evidence: %s", (evidence) => {
|
||||
const evaluation = evaluateRealBehaviorProof({
|
||||
])("passes external PRs with evidence: %s", (evidence) => {
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(proofBody(evidence)),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("passed");
|
||||
expect(labelsForRealBehaviorProof(evaluation)).toEqual([PROOF_SUPPLIED_LABEL]);
|
||||
expect(labelsForPullRequestContext(evaluation)).toEqual([]);
|
||||
});
|
||||
|
||||
it("passes CRLF-formatted external PRs with screenshot proof", () => {
|
||||
const evaluation = evaluateRealBehaviorProof({
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(
|
||||
proofBody("").replace(
|
||||
/\n/g,
|
||||
@@ -129,18 +128,117 @@ describe("real-behavior-proof-policy", () => {
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("passed");
|
||||
expect(evaluation.fields).toStrictEqual({
|
||||
behavior: "Gateway startup no longer drops the configured Discord channel.",
|
||||
evidence: "",
|
||||
environment: "macOS 15.4, Node 24, local OpenClaw gateway with a redacted Discord token.",
|
||||
notTested: "No known gaps.",
|
||||
observedResult: "The gateway stayed connected and the Discord channel showed ready.",
|
||||
steps: "pnpm openclaw gateway restart, then pnpm openclaw gateway status",
|
||||
});
|
||||
expect(labelsForRealBehaviorProof(evaluation)).toEqual([PROOF_SUPPLIED_LABEL]);
|
||||
expect(labelsForPullRequestContext(evaluation)).toEqual([]);
|
||||
});
|
||||
|
||||
it("keeps proof fields after Markdown headings copied inside fenced output", () => {
|
||||
it("requires authored problem content instead of template comments", () => {
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(
|
||||
proofBody("").replace(
|
||||
"The gateway dropped the configured Discord channel during startup.",
|
||||
"<!-- Describe the concrete user, product, or operational problem. -->",
|
||||
),
|
||||
),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("missing");
|
||||
expect(evaluation.missingSections).toEqual(["What Problem This Solves"]);
|
||||
});
|
||||
|
||||
it("does not accept the untouched current template", () => {
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(blankTemplateBody),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("missing");
|
||||
expect(evaluation.missingSections).toEqual(["What Problem This Solves", "Evidence"]);
|
||||
});
|
||||
|
||||
it("does not accept sections hidden by an unclosed HTML comment", () => {
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(`<!--\n${proofBody("pnpm test passed.")}`),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("missing");
|
||||
expect(evaluation.missingSections).toEqual(["What Problem This Solves", "Evidence"]);
|
||||
});
|
||||
|
||||
it("accepts literal HTML comments inside fenced evidence", () => {
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(
|
||||
proofBody(["```html", "<!-- captured fragment", "<p>ready</p>", "```"].join("\n")),
|
||||
),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("passed");
|
||||
});
|
||||
|
||||
it("accepts nested Markdown headings inside Evidence", () => {
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(
|
||||
proofBody(["### Focused tests", "", "`pnpm test` passed."].join("\n")),
|
||||
),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("passed");
|
||||
});
|
||||
|
||||
it("rejects None as evidence", () => {
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(proofBody("None")),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("missing");
|
||||
expect(evaluation.missingSections).toEqual(["Evidence"]);
|
||||
});
|
||||
|
||||
it("rejects Markdown separators as context and evidence", () => {
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(proofBody("---", { problem: "***" })),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("missing");
|
||||
expect(evaluation.missingSections).toEqual(["What Problem This Solves", "Evidence"]);
|
||||
});
|
||||
|
||||
it("does not accept legacy fields hidden by HTML comments", () => {
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(
|
||||
[
|
||||
"## Real behavior proof",
|
||||
"",
|
||||
"<!--",
|
||||
"Behavior addressed: The gateway dropped the configured Discord channel during startup.",
|
||||
"Evidence after fix: pnpm test passed.",
|
||||
"-->",
|
||||
].join("\n"),
|
||||
),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("missing");
|
||||
expect(evaluation.missingSections).toEqual(["What Problem This Solves", "Evidence"]);
|
||||
});
|
||||
|
||||
it("accepts legacy behavior fields while open PRs still use the old template", () => {
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(
|
||||
[
|
||||
"## Real behavior proof",
|
||||
"",
|
||||
"- Behavior addressed: The gateway dropped the configured Discord channel during startup.",
|
||||
"- Real environment tested: macOS 15.4, Node 24, local OpenClaw gateway.",
|
||||
"- Exact steps or command run after this patch: pnpm openclaw gateway restart",
|
||||
"- Evidence after fix: ",
|
||||
"- Observed result after fix: The gateway stayed connected and Discord reported ready.",
|
||||
"- What was not tested: No known gaps.",
|
||||
].join("\n"),
|
||||
),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("passed");
|
||||
});
|
||||
|
||||
it("accepts Markdown headings copied inside fenced evidence", () => {
|
||||
const body = proofBody(
|
||||
[
|
||||
"Terminal transcript:",
|
||||
@@ -159,25 +257,16 @@ describe("real-behavior-proof-policy", () => {
|
||||
"discord ready",
|
||||
"```",
|
||||
].join("\n"),
|
||||
{
|
||||
observedResult: "Plugin rules appear after the plugin boundary.",
|
||||
notTested: "Live model attribution smoke.",
|
||||
},
|
||||
);
|
||||
const evaluation = evaluateRealBehaviorProof({
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(body),
|
||||
});
|
||||
|
||||
expect(evaluation.fields?.evidence).toContain("openclaw gateway status");
|
||||
expect(evaluation.status).toBe("passed");
|
||||
expect(evaluation.fields?.observedResult).toBe(
|
||||
"Plugin rules appear after the plugin boundary.",
|
||||
);
|
||||
expect(evaluation.fields?.notTested).toBe("Live model attribution smoke.");
|
||||
expect(labelsForRealBehaviorProof(evaluation)).toEqual([PROOF_SUPPLIED_LABEL]);
|
||||
expect(labelsForPullRequestContext(evaluation)).toEqual([]);
|
||||
});
|
||||
|
||||
it("uses the latest real behavior proof section when duplicates exist", () => {
|
||||
it("uses the latest Evidence section when duplicates exist", () => {
|
||||
const validProof = proofBody(
|
||||
[
|
||||
"Terminal transcript:",
|
||||
@@ -188,36 +277,35 @@ describe("real-behavior-proof-policy", () => {
|
||||
"```",
|
||||
].join("\n"),
|
||||
);
|
||||
const mockOnlyProof = proofBody("Focused tests passed: 2 files, 36 tests.", {
|
||||
steps: "pnpm test",
|
||||
observedResult: "CI passes.",
|
||||
});
|
||||
const testEvidence = proofBody("Focused tests passed: 2 files, 36 tests.");
|
||||
|
||||
const laterValid = evaluateRealBehaviorProof({
|
||||
const laterValid = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(
|
||||
[mockOnlyProof, "## Summary", "- Keep the detailed proof below.", validProof].join("\n\n"),
|
||||
[testEvidence, "## Summary", "- Keep the detailed proof below.", validProof].join("\n\n"),
|
||||
),
|
||||
});
|
||||
const laterInvalid = evaluateRealBehaviorProof({
|
||||
const laterInvalid = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(
|
||||
[validProof, "## Summary", "- Latest edit replaced proof with tests.", mockOnlyProof].join(
|
||||
[validProof, "## Evidence", "<!-- Add the most useful validation evidence. -->"].join(
|
||||
"\n\n",
|
||||
),
|
||||
),
|
||||
});
|
||||
|
||||
expect(laterValid.status).toBe("passed");
|
||||
expect(laterValid.fields?.evidence).toContain("openclaw doctor --non-interactive");
|
||||
expect(labelsForRealBehaviorProof(laterValid)).toEqual([PROOF_SUPPLIED_LABEL]);
|
||||
expect(laterInvalid.status).toBe("mock_only");
|
||||
expect(labelsForRealBehaviorProof(laterInvalid)).toEqual([MOCK_ONLY_PROOF_LABEL]);
|
||||
expect(labelsForPullRequestContext(laterValid)).toEqual([]);
|
||||
expect(laterInvalid.status).toBe("missing");
|
||||
expect(laterInvalid.missingSections).toEqual(["Evidence"]);
|
||||
});
|
||||
|
||||
it("accepts out-of-scope follow-ups as not-tested proof detail", () => {
|
||||
const body = [
|
||||
"## Real behavior proof",
|
||||
"## What Problem This Solves",
|
||||
"",
|
||||
"Cron validation should retain the configured low thinking level.",
|
||||
"",
|
||||
"## Evidence",
|
||||
"",
|
||||
"- Behavior addressed: Cron validation keeps Google Gemini 3 low thinking.",
|
||||
"- Real environment tested: Local macOS source checkout, Node 24.",
|
||||
"- Exact steps or command run after this patch:",
|
||||
" 1. Built the local checkout with `node scripts/build-all.mjs`.",
|
||||
@@ -233,22 +321,22 @@ describe("real-behavior-proof-policy", () => {
|
||||
"- No live systemd cron schedule was tested.",
|
||||
"- No real Google provider request was sent.",
|
||||
].join("\n");
|
||||
const evaluation = evaluateRealBehaviorProof({
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(body),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("passed");
|
||||
expect(evaluation.fields?.notTested).toBe(
|
||||
"- No live systemd cron schedule was tested.\n- No real Google provider request was sent.",
|
||||
);
|
||||
expect(labelsForRealBehaviorProof(evaluation)).toEqual([PROOF_SUPPLIED_LABEL]);
|
||||
expect(labelsForPullRequestContext(evaluation)).toEqual([]);
|
||||
});
|
||||
|
||||
it("accepts source PR proof when explicit gaps live in out-of-scope follow-ups", () => {
|
||||
const body = [
|
||||
"## Real behavior proof",
|
||||
"## What Problem This Solves",
|
||||
"",
|
||||
"Cron validation downgraded Google Gemini 3 low thinking to off.",
|
||||
"",
|
||||
"## Evidence",
|
||||
"",
|
||||
'- Behavior addressed: Cron/provider thinking validation no longer downgrades `google/gemini-3-flash-preview` `thinkingDefault: "low"` to `"off"` when cached catalog metadata says `reasoning:false` but the Google provider policy says Gemini 3 supports low thinking.',
|
||||
"- Real environment tested: Local macOS source checkout, Node v24.8.0, OpenClaw 2026.5.21 (c8a35c4), local `openclaw` shim pointed at the freshly built checkout. No channel credentials or provider API keys were used.",
|
||||
"- Exact steps or command run after this patch:",
|
||||
" 1. Built the local checkout with `node scripts/build-all.mjs`.",
|
||||
@@ -267,127 +355,49 @@ describe("real-behavior-proof-policy", () => {
|
||||
"- No catalog refresh or provider model-list behavior is changed in this PR.",
|
||||
"- No channel, gateway allowlist, credential, or auth-profile behavior is changed in this PR.",
|
||||
].join("\n");
|
||||
const evaluation = evaluateRealBehaviorProof({
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(body),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("passed");
|
||||
expect(evaluation.fields?.notTested).toBe(
|
||||
[
|
||||
"- No live systemd cron schedule is added in this PR.",
|
||||
"- No real Google provider request is sent in this PR.",
|
||||
"- No catalog refresh or provider model-list behavior is changed in this PR.",
|
||||
"- No channel, gateway allowlist, credential, or auth-profile behavior is changed in this PR.",
|
||||
].join("\n"),
|
||||
);
|
||||
expect(labelsForRealBehaviorProof(evaluation)).toEqual([PROOF_SUPPLIED_LABEL]);
|
||||
expect(labelsForPullRequestContext(evaluation)).toEqual([]);
|
||||
});
|
||||
|
||||
it("fails external PRs without a real behavior proof section", () => {
|
||||
const evaluation = evaluateRealBehaviorProof({
|
||||
it("fails external PRs without required context and evidence", () => {
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr("## Summary\n\n- Fixed startup."),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("missing");
|
||||
expect(labelsForRealBehaviorProof(evaluation)).toEqual([NEEDS_REAL_BEHAVIOR_PROOF_LABEL]);
|
||||
expect(labelsForPullRequestContext(evaluation)).toEqual([NEEDS_PR_CONTEXT_LABEL]);
|
||||
});
|
||||
|
||||
it("fails external PRs that say the changed behavior was not tested", () => {
|
||||
const evaluation = evaluateRealBehaviorProof({
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(proofBody("not tested")),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("missing");
|
||||
expect(labelsForRealBehaviorProof(evaluation)).toEqual([NEEDS_REAL_BEHAVIOR_PROOF_LABEL]);
|
||||
expect(labelsForPullRequestContext(evaluation)).toEqual([NEEDS_PR_CONTEXT_LABEL]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
"not tested",
|
||||
"not tested.",
|
||||
"- not tested",
|
||||
"- not tested.",
|
||||
"did not test",
|
||||
"did not test.",
|
||||
"could not test",
|
||||
"could not test.",
|
||||
])("fails external PRs with fenced missing-proof field values: %s", (missingProof) => {
|
||||
const evidence = [
|
||||
"Terminal transcript:",
|
||||
"```text",
|
||||
"$ openclaw gateway status",
|
||||
"discord ready",
|
||||
"```",
|
||||
].join("\n");
|
||||
const notTested = ["```text", missingProof, "```"].join("\n");
|
||||
|
||||
const evaluation = evaluateRealBehaviorProof({
|
||||
pullRequest: externalPr(proofBody(evidence, { notTested })),
|
||||
it("accepts focused test and CI evidence", () => {
|
||||
const evaluation = evaluatePullRequestContext({
|
||||
pullRequest: externalPr(proofBody("pnpm test passed and CI is green.")),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("missing");
|
||||
expect(evaluation.missingFields).toEqual(["notTested"]);
|
||||
expect(labelsForRealBehaviorProof(evaluation)).toEqual([NEEDS_REAL_BEHAVIOR_PROOF_LABEL]);
|
||||
expect(evaluation.status).toBe("passed");
|
||||
expect(labelsForPullRequestContext(evaluation)).toEqual([]);
|
||||
});
|
||||
|
||||
it("fails external PRs whose proof is only tests, mocks, snapshots, lint, typecheck, or CI", () => {
|
||||
const evaluation = evaluateRealBehaviorProof({
|
||||
pullRequest: externalPr(
|
||||
proofBody("pnpm test passed and Vitest mocks cover the branch.", {
|
||||
steps: "pnpm test",
|
||||
observedResult: "CI passes.",
|
||||
}),
|
||||
),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("mock_only");
|
||||
expect(labelsForRealBehaviorProof(evaluation)).toEqual([MOCK_ONLY_PROOF_LABEL]);
|
||||
});
|
||||
|
||||
it("fails external PRs whose only copied output is a fenced test or CI transcript", () => {
|
||||
const evaluation = evaluateRealBehaviorProof({
|
||||
pullRequest: externalPr(
|
||||
proofBody(["```text", "$ pnpm test", "CI passed with Vitest mocks", "```"].join("\n"), {
|
||||
steps: "pnpm test",
|
||||
observedResult: "CI passes.",
|
||||
}),
|
||||
),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("mock_only");
|
||||
expect(labelsForRealBehaviorProof(evaluation)).toEqual([MOCK_ONLY_PROOF_LABEL]);
|
||||
});
|
||||
|
||||
it("fails external PRs whose terminal label only contains test or CI output", () => {
|
||||
const evaluation = evaluateRealBehaviorProof({
|
||||
pullRequest: externalPr(
|
||||
proofBody(
|
||||
[
|
||||
"Terminal transcript:",
|
||||
"```text",
|
||||
"$ pnpm test",
|
||||
"CI passed with Vitest mocks",
|
||||
"```",
|
||||
].join("\n"),
|
||||
{
|
||||
steps: "pnpm test",
|
||||
observedResult: "CI passes.",
|
||||
},
|
||||
),
|
||||
),
|
||||
});
|
||||
|
||||
expect(evaluation.status).toBe("mock_only");
|
||||
expect(labelsForRealBehaviorProof(evaluation)).toEqual([MOCK_ONLY_PROOF_LABEL]);
|
||||
});
|
||||
|
||||
it("passes maintainer, bot, and override cases", () => {
|
||||
it("skips maintainer and bot PRs but requires context from external PRs", () => {
|
||||
expect(
|
||||
evaluateRealBehaviorProof({
|
||||
evaluatePullRequestContext({
|
||||
pullRequest: externalPr("", { author_association: "MEMBER" }),
|
||||
}).status,
|
||||
).toBe("skipped");
|
||||
expect(
|
||||
evaluateRealBehaviorProof({
|
||||
evaluatePullRequestContext({
|
||||
pullRequest: externalPr("", {
|
||||
user: {
|
||||
login: "renovate[bot]",
|
||||
@@ -397,10 +407,10 @@ describe("real-behavior-proof-policy", () => {
|
||||
}).status,
|
||||
).toBe("skipped");
|
||||
expect(
|
||||
evaluateRealBehaviorProof({
|
||||
evaluatePullRequestContext({
|
||||
pullRequest: externalPr("", { labels: [{ name: PROOF_OVERRIDE_LABEL }] }),
|
||||
}).status,
|
||||
).toBe("override");
|
||||
).toBe("missing");
|
||||
});
|
||||
|
||||
it("accepts ClawSweeper pass verdict comments only for the exact PR head", () => {
|
||||
|
||||
@@ -302,7 +302,7 @@ describe("scripts/test-projects changed-target routing", () => {
|
||||
});
|
||||
expect(resolveChangedTestTargetPlan(["scripts/github/real-behavior-proof-check.mjs"])).toEqual({
|
||||
mode: "targets",
|
||||
targets: ["test/scripts/real-behavior-proof-check.test.ts"],
|
||||
targets: ["test/vitest/vitest.tooling.config.ts"],
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
Reference in New Issue
Block a user