mirror of
https://github.com/openclaw/openclaw.git
synced 2026-07-21 10:16:44 +00:00
feat(qa): crabline channel driver (#91502)
* feat(qa): add crabline channel driver seam * feat: run crabline channel driver smoke * chore: keep crabline qa dependency dev-only * refactor(qa): keep crabline driver details opaque * chore(qa): pin crabline to merged driver API * feat(qa): drive channel driver from profiles * fix(qa): declare crabline runtime peer * feat(qa): resolve crabline channel from scenarios * feat(qa): treat unsupported profile channels as coverage gaps * Revert "feat(qa): treat unsupported profile channels as coverage gaps" This reverts commit 65a97016558705514a1c1ec74870de9b0a6e7886. * fix(qa): adapt crabline driver to chat sdk cli * refactor(qa): pass channel driver metadata directly * chore(qa): update crabline provider pin * chore(qa): default channel scenarios to driver * chore: repair qa dependency lockfile * chore: allow native qa dependency builds * fix(qa): satisfy crabline driver lint * fix(qa): satisfy crabline ci gates * Use crabline transport for smoke QA profile * fix(qa): keep crabline driver opt-in * fix(qa): reuse crabline telegram driver token * fix(qa): route smoke profile through crabline * fix(qa): run full smoke profile lane * fix(qa): remove smoke scenario workflow filter * fix: stabilize crabline smoke qa profile * fix: pin crabline qa dependency * test: keep crabline smoke credential-free * fix: skip visible reasoning lane for crabline smoke * fix: unblock crabline qa ci * Update crabline dependency * Pin crabline to merged main * Use Crabline fake provider servers
This commit is contained in:
Dallin Romney
@@ -73,9 +73,10 @@ pnpm openclaw qa run \
|
||||
--output-dir .artifacts/qa-e2e/smoke-ci-profile-dispatch
|
||||
```
|
||||
|
||||
Use `smoke-ci` for deterministic no-live-service proof and `release` for the
|
||||
Stable/LTS proof lane. When a command also needs an OpenClaw root profile, put
|
||||
the root profile before the QA command:
|
||||
Use `smoke-ci` for deterministic profile proof with mock model providers and
|
||||
Crabline fake provider servers. Use `release` for Stable/LTS proof against live
|
||||
channels. When a command also needs an OpenClaw root profile, put the root
|
||||
profile before the QA command:
|
||||
|
||||
```bash
|
||||
pnpm openclaw --profile work qa run --qa-profile smoke-ci
|
||||
@@ -197,7 +198,10 @@ witness video when `MANTIS_DISCORD_VIEWER_CHROME_PROFILE_DIR` or
|
||||
environment. That viewer profile is only for visual capture; the pass/fail
|
||||
decision still comes from the Discord REST oracle.
|
||||
|
||||
CI uses the same command surface in `.github/workflows/qa-live-transports-convex.yml`. Scheduled and default manual runs execute the fast Matrix profile with live frontier credentials, `--fast`, and `OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS=3000`. Manual `matrix_profile=all` fans out into the five profile shards so the exhaustive catalog can run in parallel while keeping one artifact directory per shard.
|
||||
CI uses the same command surface in `.github/workflows/qa-live-transports-convex.yml`.
|
||||
Scheduled and default manual runs execute the fast Matrix profile with live
|
||||
frontier credentials, `--fast`, and `OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS=3000`.
|
||||
Manual `matrix_profile=all` fans out into the five profile shards.
|
||||
|
||||
For transport-real Telegram, Discord, Slack, and WhatsApp smoke lanes:
|
||||
|
||||
@@ -857,7 +861,10 @@ provider names.
|
||||
|
||||
## Transport adapters
|
||||
|
||||
`qa-lab` owns a generic transport seam for YAML QA scenarios. `qa-channel` is the first adapter on that seam, but the design target is wider: future real or synthetic channels should plug into the same suite runner instead of adding a transport-specific QA runner.
|
||||
`qa-lab` owns a generic transport seam for YAML QA scenarios. `qa-channel` is
|
||||
the synthetic default. `crabline` starts local provider-shaped servers and runs
|
||||
OpenClaw's normal channel plugins against them. `live` is reserved for real
|
||||
provider credentials and external channels.
|
||||
|
||||
At the architecture level, the split is:
|
||||
|
||||
@@ -867,10 +874,10 @@ At the architecture level, the split is:
|
||||
|
||||
### Adding a channel
|
||||
|
||||
Adding a channel to the YAML QA system requires exactly two things:
|
||||
|
||||
1. A transport adapter for the channel.
|
||||
2. A scenario pack that exercises the channel contract.
|
||||
Adding a channel to the YAML QA system requires the channel implementation plus
|
||||
a scenario pack that exercises the channel contract. For smoke CI coverage, add
|
||||
the matching Crabline fake provider server and expose it through the `crabline`
|
||||
driver.
|
||||
|
||||
Do not add a new top-level QA command root when the shared `qa-lab` host can own the flow.
|
||||
|
||||
|
||||
@@ -16,6 +16,7 @@
|
||||
"@openclaw/plugin-sdk": "workspace:*",
|
||||
"@openclaw/slack": "workspace:*",
|
||||
"@openclaw/whatsapp": "workspace:*",
|
||||
"crabline": "github:openclaw/crabline#b3513f66053788c6a7bd2bc76fbfc7201f647d29",
|
||||
"openclaw": "2026.5.28"
|
||||
},
|
||||
"peerDependencies": {
|
||||
|
||||
@@ -340,6 +340,8 @@ describe("qa cli runtime", () => {
|
||||
repoRoot: process.cwd(),
|
||||
outputDir: path.join(process.cwd(), ".artifacts", "qa-e2e", "scenario-test"),
|
||||
transportId: "qa-channel",
|
||||
channelDriver: undefined,
|
||||
channelDriverSelection: undefined,
|
||||
primaryModel: "mock-openai/gpt-5.5",
|
||||
alternateModel: undefined,
|
||||
fastMode: undefined,
|
||||
@@ -431,6 +433,7 @@ describe("qa cli runtime", () => {
|
||||
profile: "smoke-ci",
|
||||
surface: "agent-runtime-and-provider-execution",
|
||||
category: "agent-runtime-and-provider-execution.agent-turn-execution",
|
||||
scenarioIds: ["dm-chat-baseline"],
|
||||
transportId: "qa-channel",
|
||||
fastMode: true,
|
||||
concurrency: 2,
|
||||
@@ -442,12 +445,16 @@ describe("qa cli runtime", () => {
|
||||
repoRoot: path.resolve("/tmp/openclaw-repo"),
|
||||
outputDir: path.resolve("/tmp/openclaw-repo", ".artifacts/qa-e2e/smoke-ci"),
|
||||
transportId: "qa-channel",
|
||||
channelDriver: "crabline",
|
||||
providerMode: "mock-openai",
|
||||
fastMode: true,
|
||||
concurrency: 2,
|
||||
});
|
||||
expect(suiteArgs.scenarioIds).toEqual(expect.arrayContaining(["dm-chat-baseline"]));
|
||||
expect(suiteArgs.scenarioIds).not.toContain("thinking-slash-model-remap");
|
||||
expect(suiteArgs.channelDriverSelection).toMatchObject({
|
||||
channel: "telegram",
|
||||
channelDriver: "crabline",
|
||||
});
|
||||
expect(suiteArgs.scenarioIds).toEqual(["dm-chat-baseline"]);
|
||||
expect(process.env.OPENCLAW_QA_PROFILE).toBe("release");
|
||||
const evidence = JSON.parse(await fs.readFile(suiteEvidencePath, "utf8")) as {
|
||||
evidenceMode?: unknown;
|
||||
@@ -493,6 +500,20 @@ describe("qa cli runtime", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("passes non-Crabline profile channel drivers as declarative suite metadata", async () => {
|
||||
await runQaProfileCommand({
|
||||
repoRoot: "/tmp/openclaw-repo",
|
||||
profile: "release",
|
||||
surface: "agent-runtime-and-provider-execution",
|
||||
category: "agent-runtime-and-provider-execution.agent-turn-execution",
|
||||
providerMode: "mock-openai",
|
||||
});
|
||||
|
||||
const suiteArgs = mockFirstObjectArg(runQaSuite);
|
||||
expect(suiteArgs.channelDriver).toBe("live");
|
||||
expect(suiteArgs.channelDriverSelection).toBeUndefined();
|
||||
});
|
||||
|
||||
it("rejects qa profile runs that do not match taxonomy categories", async () => {
|
||||
await expect(
|
||||
runQaProfileCommand({
|
||||
@@ -506,6 +527,20 @@ describe("qa cli runtime", () => {
|
||||
expect(runQaSuite).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("rejects qa profile scenario filters outside the selected taxonomy categories", async () => {
|
||||
await expect(
|
||||
runQaProfileCommand({
|
||||
repoRoot: "/tmp/openclaw-repo",
|
||||
profile: "smoke-ci",
|
||||
category: "agent-runtime-and-provider-execution.agent-turn-execution",
|
||||
scenarioIds: ["not-a-real-scenario"],
|
||||
}),
|
||||
).rejects.toThrow(
|
||||
"qa run did not find taxonomy scenarios for --qa-profile smoke-ci --category agent-runtime-and-provider-execution.agent-turn-execution --scenario not-a-real-scenario.",
|
||||
);
|
||||
expect(runQaSuite).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("rejects qa profile runs whose profile is not declared in taxonomy.yaml", async () => {
|
||||
await expect(
|
||||
runQaProfileCommand({
|
||||
@@ -532,6 +567,8 @@ describe("qa cli runtime", () => {
|
||||
repoRoot: path.resolve("/tmp/openclaw-repo"),
|
||||
outputDir: path.resolve("/tmp/openclaw-repo", ".artifacts/qa/frontier"),
|
||||
transportId: "qa-channel",
|
||||
channelDriver: undefined,
|
||||
channelDriverSelection: undefined,
|
||||
providerMode: "live-frontier",
|
||||
primaryModel: "openai/gpt-5.5",
|
||||
alternateModel: "anthropic/claude-sonnet-4-6",
|
||||
@@ -541,6 +578,48 @@ describe("qa cli runtime", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("uses the Crabline default channel when selected scenarios do not request one", async () => {
|
||||
await runQaSuiteCommand({
|
||||
repoRoot: "/tmp/openclaw-repo",
|
||||
outputDir: ".artifacts/qa/multipass-telegram",
|
||||
providerMode: "mock-openai",
|
||||
channelDriver: "crabline",
|
||||
scenarioIds: ["channel-chat-baseline"],
|
||||
});
|
||||
|
||||
expect(runQaSuite).toHaveBeenCalledWith({
|
||||
repoRoot: path.resolve("/tmp/openclaw-repo"),
|
||||
outputDir: path.resolve("/tmp/openclaw-repo", ".artifacts/qa/multipass-telegram"),
|
||||
transportId: "qa-channel",
|
||||
channelDriver: "crabline",
|
||||
channelDriverSelection: {
|
||||
capabilityMatrixPath: "crabline-fake-provider-capabilities.json",
|
||||
channel: "telegram",
|
||||
channelDriver: "crabline",
|
||||
smokeArtifactPath: "crabline-fake-provider-smoke.json",
|
||||
},
|
||||
providerMode: "mock-openai",
|
||||
primaryModel: undefined,
|
||||
alternateModel: undefined,
|
||||
fastMode: undefined,
|
||||
scenarioIds: ["channel-chat-baseline"],
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps Crabline channel-driver independent from the VM runner", async () => {
|
||||
await expect(
|
||||
runQaSuiteCommand({
|
||||
repoRoot: "/tmp/openclaw-repo",
|
||||
providerMode: "mock-openai",
|
||||
channelDriver: "crabline",
|
||||
channel: "telegram",
|
||||
runner: "multipass",
|
||||
}),
|
||||
).rejects.toThrow("--channel-driver crabline requires --runner host.");
|
||||
expect(runQaSuite).not.toHaveBeenCalled();
|
||||
expect(runQaMultipass).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("passes explicit suite plugin enablements into the host gateway run", async () => {
|
||||
await runQaSuiteCommand({
|
||||
repoRoot: "/tmp/openclaw-repo",
|
||||
@@ -553,6 +632,8 @@ describe("qa cli runtime", () => {
|
||||
repoRoot: path.resolve("/tmp/openclaw-repo"),
|
||||
outputDir: undefined,
|
||||
transportId: "qa-channel",
|
||||
channelDriver: undefined,
|
||||
channelDriverSelection: undefined,
|
||||
providerMode: "mock-openai",
|
||||
primaryModel: undefined,
|
||||
alternateModel: undefined,
|
||||
@@ -574,6 +655,8 @@ describe("qa cli runtime", () => {
|
||||
repoRoot: path.resolve("/tmp/openclaw-repo"),
|
||||
outputDir: undefined,
|
||||
transportId: "qa-channel",
|
||||
channelDriver: undefined,
|
||||
channelDriverSelection: undefined,
|
||||
providerMode: "mock-openai",
|
||||
primaryModel: undefined,
|
||||
alternateModel: undefined,
|
||||
@@ -624,6 +707,8 @@ describe("qa cli runtime", () => {
|
||||
repoRoot: path.resolve("/tmp/openclaw-repo"),
|
||||
outputDir: undefined,
|
||||
transportId: "qa-channel",
|
||||
channelDriver: undefined,
|
||||
channelDriverSelection: undefined,
|
||||
providerMode: "mock-openai",
|
||||
primaryModel: undefined,
|
||||
alternateModel: undefined,
|
||||
@@ -2020,6 +2105,8 @@ describe("qa cli runtime", () => {
|
||||
repoRoot: path.resolve("/tmp/openclaw-repo"),
|
||||
outputDir: undefined,
|
||||
transportId: "qa-channel",
|
||||
channelDriver: undefined,
|
||||
channelDriverSelection: undefined,
|
||||
providerMode: "mock-openai",
|
||||
primaryModel: "openai/gpt-5.5",
|
||||
alternateModel: "anthropic/claude-opus-4-8",
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
// Qa Lab plugin module implements cli behavior.
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import {
|
||||
OPENCLAW_CRABLINE_DEFAULT_CHANNEL,
|
||||
resolveOpenClawCrablineChannelDriverSelection,
|
||||
} from "crabline";
|
||||
import { formatErrorMessage } from "openclaw/plugin-sdk/error-runtime";
|
||||
import { parseStrictPositiveInteger } from "openclaw/plugin-sdk/number-runtime";
|
||||
import { uniqueStrings } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
@@ -71,13 +75,15 @@ import {
|
||||
import { resolveQaScenarioPackScenarioIds } from "./scenario-packs.js";
|
||||
import { attachQaProfileScorecardEvidenceToFile } from "./scorecard-evidence.js";
|
||||
import {
|
||||
qaScorecardChannelDriverSchema,
|
||||
readQaScorecardTaxonomyReport,
|
||||
type QaScorecardCategoryCoverageReport,
|
||||
type QaScorecardChannelDriver,
|
||||
type QaScorecardEvidenceMode,
|
||||
} from "./scorecard-taxonomy.js";
|
||||
import { isQaSelfCheckSuccessful } from "./self-check.js";
|
||||
import { runQaFlowSuiteFromRuntime, runQaSuite } from "./suite-launch.runtime.js";
|
||||
import { scenarioMatchesQaProviderLane } from "./suite-planning.js";
|
||||
import { resolveQaSuiteScenarioChannel, scenarioMatchesQaProviderLane } from "./suite-planning.js";
|
||||
import { readQaSuiteFailedOrSkippedScenarioCountFromFile } from "./suite-summary.js";
|
||||
import {
|
||||
buildTokenEfficiencyReport,
|
||||
@@ -131,9 +137,12 @@ export type QaProfileCommandOptions = QaScenarioRunCommandOptions & {
|
||||
profile: string;
|
||||
surface?: string;
|
||||
category?: string;
|
||||
scenarioIds?: string[];
|
||||
};
|
||||
|
||||
export type QaSuiteCommandOptions = QaScenarioRunCommandOptions & {
|
||||
channelDriver?: string;
|
||||
channel?: string;
|
||||
runner?: string;
|
||||
thinking?: string;
|
||||
cliAuthMode?: string;
|
||||
@@ -150,6 +159,20 @@ export type QaSuiteCommandOptions = QaScenarioRunCommandOptions & {
|
||||
runtimeParityTier?: string[];
|
||||
};
|
||||
|
||||
function normalizeQaSuiteChannelDriver(
|
||||
input?: string | null,
|
||||
): QaScorecardChannelDriver | undefined {
|
||||
const normalized = input?.trim().toLowerCase();
|
||||
if (!normalized) {
|
||||
return undefined;
|
||||
}
|
||||
const parsed = qaScorecardChannelDriverSchema.safeParse(normalized);
|
||||
if (parsed.success) {
|
||||
return parsed.data;
|
||||
}
|
||||
throw new Error(`--channel-driver must be one of qa-channel, crabline, or live, got "${input}".`);
|
||||
}
|
||||
|
||||
function resolveQaManualLaneModels(opts: {
|
||||
providerMode: QaProviderMode;
|
||||
primaryModel?: string;
|
||||
@@ -659,6 +682,10 @@ export async function runQaProfileCommand(opts: QaProfileCommandOptions) {
|
||||
opts.profile,
|
||||
scorecardReport.profiles.map((entry) => entry.id),
|
||||
);
|
||||
const profileReport = scorecardReport.profiles.find((entry) => entry.id === profile);
|
||||
if (!profileReport) {
|
||||
throw new Error(`taxonomy.yaml does not define QA run profile ${profile}.`);
|
||||
}
|
||||
const categories = scorecardReport.categories.filter((category) =>
|
||||
qaScorecardCategoryMatchesRunProfile(category, {
|
||||
profile,
|
||||
@@ -673,9 +700,34 @@ export async function runQaProfileCommand(opts: QaProfileCommandOptions) {
|
||||
const scenarioBySourcePath = new Map(
|
||||
scenarioPack.scenarios.map((scenario) => [scenario.sourcePath, scenario] as const),
|
||||
);
|
||||
const taxonomyScenarios = uniqueStrings(categories.flatMap((category) => category.scenarioRefs))
|
||||
const taxonomyScenariosForCategories = uniqueStrings(
|
||||
categories.flatMap((category) => category.scenarioRefs),
|
||||
)
|
||||
.map((scenarioRef) => scenarioBySourcePath.get(scenarioRef))
|
||||
.filter((scenario): scenario is NonNullable<typeof scenario> => scenario !== undefined);
|
||||
const requestedScenarioIds = uniqueStrings(
|
||||
(opts.scenarioIds ?? []).map((scenarioId) => scenarioId.trim()).filter(Boolean),
|
||||
);
|
||||
const taxonomyScenarios =
|
||||
requestedScenarioIds.length === 0
|
||||
? taxonomyScenariosForCategories
|
||||
: taxonomyScenariosForCategories.filter((scenario) =>
|
||||
requestedScenarioIds.includes(scenario.id),
|
||||
);
|
||||
if (requestedScenarioIds.length > 0 && taxonomyScenarios.length === 0) {
|
||||
throw new Error(
|
||||
`qa run did not find taxonomy scenarios for ${formatQaRunProfileFilterList(opts)} --scenario ${requestedScenarioIds.join(",")}.`,
|
||||
);
|
||||
}
|
||||
const matchedScenarioIds = new Set(taxonomyScenarios.map((scenario) => scenario.id));
|
||||
const missingScenarioIds = requestedScenarioIds.filter(
|
||||
(scenarioId) => !matchedScenarioIds.has(scenarioId),
|
||||
);
|
||||
if (missingScenarioIds.length > 0) {
|
||||
throw new Error(
|
||||
`qa run did not find taxonomy scenarios for ${formatQaRunProfileFilterList(opts)} --scenario ${missingScenarioIds.join(",")}.`,
|
||||
);
|
||||
}
|
||||
const providerMode = opts.providerMode ?? defaultQaRunProfileProviderMode(profile);
|
||||
const normalizedProviderMode = normalizeQaProviderMode(providerMode);
|
||||
const primaryModel = opts.primaryModel?.trim() || defaultQaModelForMode(normalizedProviderMode);
|
||||
@@ -709,6 +761,7 @@ export async function runQaProfileCommand(opts: QaProfileCommandOptions) {
|
||||
scenarioIds: scenarios.map((scenario) => scenario.id),
|
||||
concurrency: opts.concurrency,
|
||||
allowFailures: opts.allowFailures,
|
||||
channelDriver: profileReport.channelDriver,
|
||||
});
|
||||
evidencePath =
|
||||
suiteResult && "evidencePath" in suiteResult ? suiteResult.evidencePath : undefined;
|
||||
@@ -729,6 +782,30 @@ export async function runQaProfileCommand(opts: QaProfileCommandOptions) {
|
||||
process.stdout.write(`QA profile scorecard: ${evidencePath}\n`);
|
||||
}
|
||||
|
||||
function selectQaScenarioDefinitionsForChannelResolution(params: {
|
||||
scenarioIds: string[];
|
||||
providerMode: QaProviderMode;
|
||||
primaryModel: string;
|
||||
claudeCliAuthMode?: QaCliBackendAuthMode;
|
||||
}) {
|
||||
const scenarios = readQaScenarioPack().scenarios;
|
||||
if (params.scenarioIds.length > 0) {
|
||||
const scenarioById = new Map(scenarios.map((scenario) => [scenario.id, scenario]));
|
||||
return params.scenarioIds.flatMap((scenarioId) => {
|
||||
const scenario = scenarioById.get(scenarioId);
|
||||
return scenario ? [scenario] : [];
|
||||
});
|
||||
}
|
||||
return scenarios.filter((scenario) =>
|
||||
scenarioMatchesQaProviderLane({
|
||||
scenario,
|
||||
providerMode: params.providerMode,
|
||||
primaryModel: params.primaryModel,
|
||||
claudeCliAuthMode: params.claudeCliAuthMode,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
function normalizeQaRunProfile(value: string, profileIds: readonly string[]) {
|
||||
if (profileIds.length === 0) {
|
||||
throw new Error("taxonomy.yaml does not define QA run profiles.");
|
||||
@@ -765,13 +842,19 @@ function qaScorecardCategoryMatchesRunProfile(
|
||||
|
||||
function formatQaRunProfileNoMatchMessage(
|
||||
opts: Pick<QaProfileCommandOptions, "profile" | "surface" | "category">,
|
||||
) {
|
||||
return `qa run did not find taxonomy categories for ${formatQaRunProfileFilterList(opts)}.`;
|
||||
}
|
||||
|
||||
function formatQaRunProfileFilterList(
|
||||
opts: Pick<QaProfileCommandOptions, "profile" | "surface" | "category">,
|
||||
) {
|
||||
const filters = [
|
||||
`--qa-profile ${opts.profile}`,
|
||||
opts.surface?.trim() ? `--surface ${opts.surface.trim()}` : null,
|
||||
opts.category?.trim() ? `--category ${opts.category.trim()}` : null,
|
||||
].filter((filter): filter is string => filter !== null);
|
||||
return `qa run did not find taxonomy categories for ${filters.join(" ")}.`;
|
||||
return filters.join(" ");
|
||||
}
|
||||
|
||||
async function withTemporaryQaProfileEnv<T>(profile: string, run: () => Promise<T>): Promise<T> {
|
||||
@@ -810,12 +893,36 @@ export async function runQaSuiteCommand(opts: QaSuiteCommandOptions) {
|
||||
const claudeCliAuthMode = parseQaCliBackendAuthMode(opts.cliAuthMode);
|
||||
const primaryModel = normalizeQaOptionalModelRef(opts.primaryModel);
|
||||
const alternateModel = normalizeQaOptionalModelRef(opts.alternateModel);
|
||||
const channelDriver = normalizeQaSuiteChannelDriver(opts.channelDriver);
|
||||
if (opts.channel?.trim() && channelDriver !== "crabline") {
|
||||
throw new Error(
|
||||
"--channel override is currently only supported with --channel-driver crabline.",
|
||||
);
|
||||
}
|
||||
if (runner !== "host" && runner !== "multipass") {
|
||||
throw new Error(`--runner must be one of host or multipass, got "${opts.runner}".`);
|
||||
}
|
||||
if (opts.preflight === true && runner !== "host") {
|
||||
throw new Error("--preflight requires --runner host.");
|
||||
}
|
||||
if (channelDriver === "crabline" && runner !== "host") {
|
||||
throw new Error("--channel-driver crabline requires --runner host.");
|
||||
}
|
||||
const channelDriverSelection =
|
||||
channelDriver === "crabline"
|
||||
? resolveOpenClawCrablineChannelDriverSelection({
|
||||
channel: resolveQaSuiteScenarioChannel({
|
||||
defaultChannel: OPENCLAW_CRABLINE_DEFAULT_CHANNEL,
|
||||
explicitChannel: opts.channel,
|
||||
scenarios: selectQaScenarioDefinitionsForChannelResolution({
|
||||
scenarioIds,
|
||||
providerMode,
|
||||
primaryModel: primaryModel ?? defaultQaModelForMode(providerMode),
|
||||
claudeCliAuthMode,
|
||||
}),
|
||||
}),
|
||||
})
|
||||
: undefined;
|
||||
if (
|
||||
runner === "host" &&
|
||||
(opts.image !== undefined ||
|
||||
@@ -884,6 +991,8 @@ export async function runQaSuiteCommand(opts: QaSuiteCommandOptions) {
|
||||
outputDir: resolveRepoRelativeOutputDir(repoRoot, opts.outputDir),
|
||||
evidenceMode: opts.evidenceMode,
|
||||
transportId,
|
||||
channelDriver,
|
||||
channelDriverSelection,
|
||||
...(opts.providerMode !== undefined ? { providerMode } : {}),
|
||||
primaryModel,
|
||||
alternateModel,
|
||||
|
||||
@@ -217,6 +217,8 @@ describe("qa cli registration", () => {
|
||||
"agent-runtime-and-provider-execution",
|
||||
"--category",
|
||||
"agent-runtime-and-provider-execution.agent-turn-execution",
|
||||
"--scenario",
|
||||
"dm-chat-baseline",
|
||||
"--evidence-mode",
|
||||
"slim",
|
||||
"--transport",
|
||||
@@ -239,6 +241,7 @@ describe("qa cli registration", () => {
|
||||
profile: "smoke-ci",
|
||||
surface: "agent-runtime-and-provider-execution",
|
||||
category: "agent-runtime-and-provider-execution.agent-turn-execution",
|
||||
scenarioIds: ["dm-chat-baseline"],
|
||||
evidenceMode: "slim",
|
||||
transportId: "qa-channel",
|
||||
providerMode: "mock-openai",
|
||||
@@ -255,6 +258,7 @@ describe("qa cli registration", () => {
|
||||
["--output-dir", [".artifacts/qa-e2e/smoke-ci"]],
|
||||
["--surface", ["agent-runtime-and-provider-execution"]],
|
||||
["--category", ["agent-runtime-and-provider-execution.agent-turn-execution"]],
|
||||
["--scenario", ["dm-chat-baseline"]],
|
||||
["--evidence-mode", ["slim"]],
|
||||
["--exclude-test-execution-evidence", []],
|
||||
["--transport", ["qa-channel"]],
|
||||
|
||||
@@ -40,6 +40,7 @@ type QaRunCliOptions = QaLabSelfCheckCommandOptions &
|
||||
qaProfile?: QaProfileCommandOptions["profile"];
|
||||
surface?: QaProfileCommandOptions["surface"];
|
||||
category?: QaProfileCommandOptions["category"];
|
||||
scenario?: QaProfileCommandOptions["scenarioIds"];
|
||||
evidenceMode?: QaProfileCommandOptions["evidenceMode"];
|
||||
excludeTestExecutionEvidence?: boolean;
|
||||
};
|
||||
@@ -48,6 +49,7 @@ const QA_RUN_PROFILE_ONLY_OPTIONS = [
|
||||
{ optionName: "outputDir", flag: "--output-dir" },
|
||||
{ optionName: "surface", flag: "--surface" },
|
||||
{ optionName: "category", flag: "--category" },
|
||||
{ optionName: "scenario", flag: "--scenario" },
|
||||
{ optionName: "evidenceMode", flag: "--evidence-mode" },
|
||||
{ optionName: "excludeTestExecutionEvidence", flag: "--exclude-test-execution-evidence" },
|
||||
{ optionName: "transport", flag: "--transport" },
|
||||
@@ -62,6 +64,8 @@ const QA_RUN_PROFILE_ONLY_OPTIONS = [
|
||||
const QA_RUN_SELF_CHECK_ONLY_OPTIONS = [{ optionName: "output", flag: "--output" }] as const;
|
||||
|
||||
type QaSuiteCliOptions = QaScenarioRunCliOptions & {
|
||||
channelDriver?: QaSuiteCommandOptions["channelDriver"];
|
||||
channel?: QaSuiteCommandOptions["channel"];
|
||||
runner?: QaSuiteCommandOptions["runner"];
|
||||
thinking?: QaSuiteCommandOptions["thinking"];
|
||||
cliAuthMode?: QaSuiteCommandOptions["cliAuthMode"];
|
||||
@@ -396,6 +400,12 @@ export function registerQaLabCli(program: Command) {
|
||||
.option("--qa-profile <id>", "Run the QA profile from taxonomy.yaml")
|
||||
.option("--surface <id>", "Limit --qa-profile to a taxonomy surface id")
|
||||
.option("--category <id>", "Limit --qa-profile to a taxonomy category id")
|
||||
.option(
|
||||
"--scenario <id>",
|
||||
"Limit --qa-profile to a scenario id (repeatable)",
|
||||
collectString,
|
||||
[],
|
||||
)
|
||||
.option(
|
||||
"--evidence-mode <mode>",
|
||||
"Set profile qa-evidence.json mode: full or slim",
|
||||
@@ -430,6 +440,7 @@ export function registerQaLabCli(program: Command) {
|
||||
profile: opts.qaProfile,
|
||||
surface: opts.surface,
|
||||
category: opts.category,
|
||||
scenarioIds: opts.scenario,
|
||||
evidenceMode: resolveQaEvidenceModeOptions(opts),
|
||||
transportId: opts.transport,
|
||||
providerMode: opts.providerMode,
|
||||
@@ -453,6 +464,11 @@ export function registerQaLabCli(program: Command) {
|
||||
.option("--output-dir <path>", "Suite artifact directory")
|
||||
.option("--runner <kind>", "Execution runner: host or multipass", "host")
|
||||
.option("--transport <id>", "QA transport id", "qa-channel")
|
||||
.option("--channel-driver <id>", "QA channel driver: qa-channel, crabline, or live")
|
||||
.option(
|
||||
"--channel <id>",
|
||||
"Internal host QA channel override for --channel-driver; defaults to scenario/default",
|
||||
)
|
||||
.option("--provider-mode <mode>", formatQaProviderModeHelp())
|
||||
.option("--model <ref>", "Primary provider/model ref")
|
||||
.option("--alt-model <ref>", "Alternate provider/model ref")
|
||||
@@ -504,6 +520,8 @@ export function registerQaLabCli(program: Command) {
|
||||
repoRoot: opts.repoRoot,
|
||||
outputDir: opts.outputDir,
|
||||
transportId: opts.transport,
|
||||
channelDriver: opts.channelDriver,
|
||||
channel: opts.channel,
|
||||
runner: opts.runner,
|
||||
providerMode: opts.providerMode,
|
||||
primaryModel: opts.model,
|
||||
|
||||
@@ -35,12 +35,14 @@ function testMaturityTaxonomy(params?: {
|
||||
id: "smoke-ci",
|
||||
description: "Test smoke profile.",
|
||||
includeAllCategories: false,
|
||||
categoryIds: [],
|
||||
channelDriver: "crabline" as const,
|
||||
categoryIds: [categoryId],
|
||||
},
|
||||
{
|
||||
id: "release",
|
||||
description: "Test release profile.",
|
||||
includeAllCategories: params?.includeAllCategories ?? false,
|
||||
channelDriver: "qa-channel" as const,
|
||||
categoryIds: [
|
||||
...(params?.includeAllCategories ? [] : (params?.profileCategoryIds ?? [categoryId])),
|
||||
],
|
||||
@@ -128,6 +130,17 @@ describe("qa coverage report", () => {
|
||||
"whatsapp",
|
||||
]);
|
||||
expect(inventory.scorecardTaxonomy.profileCount).toBe(2);
|
||||
expect(
|
||||
inventory.scorecardTaxonomy.profiles.find((profile) => profile.id === "smoke-ci"),
|
||||
).toMatchObject({
|
||||
channelDriver: "crabline",
|
||||
evidenceMode: "slim",
|
||||
});
|
||||
expect(
|
||||
inventory.scorecardTaxonomy.profiles.find((profile) => profile.id === "release"),
|
||||
).toMatchObject({
|
||||
channelDriver: "live",
|
||||
});
|
||||
expect(inventory.scorecardTaxonomy.categoryCount).toBeGreaterThan(200);
|
||||
expect(inventory.scorecardTaxonomy.requiredCategoryCount).toBeGreaterThan(0);
|
||||
expect(inventory.scorecardTaxonomy.requiredCategoryCount).toBeLessThanOrEqual(
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
// Qa Lab tests cover Crabline fake-provider transport integration behavior.
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { OPENCLAW_CRABLINE_MANIFEST_PATH } from "crabline";
|
||||
import { fetchWithSsrFGuard } from "openclaw/plugin-sdk/ssrf-runtime";
|
||||
import { withTempDir } from "openclaw/plugin-sdk/test-env";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { createQaBusState } from "./bus-state.js";
|
||||
import { createQaCrablineTransportAdapter } from "./crabline-transport.js";
|
||||
|
||||
function createSelection() {
|
||||
return {
|
||||
capabilityMatrixPath: "crabline-fake-provider-capabilities.json",
|
||||
channel: "telegram",
|
||||
channelDriver: "crabline",
|
||||
smokeArtifactPath: "crabline-fake-provider-smoke.json",
|
||||
} as const;
|
||||
}
|
||||
|
||||
describe("crabline transport", () => {
|
||||
it("configures OpenClaw's Telegram plugin against a Crabline fake provider server", async () => {
|
||||
await withTempDir("qa-crabline-transport-", async (outputDir) => {
|
||||
const transport = await createQaCrablineTransportAdapter({
|
||||
outputDir,
|
||||
selection: createSelection(),
|
||||
state: createQaBusState(),
|
||||
});
|
||||
|
||||
try {
|
||||
expect(transport.id).toBe("crabline");
|
||||
expect(transport.requiredPluginIds).toEqual(["telegram"]);
|
||||
expect(transport.createGatewayConfig({ baseUrl: "http://127.0.0.1:1" })).toMatchObject({
|
||||
channels: {
|
||||
telegram: {
|
||||
apiRoot: expect.stringMatching(/^http:\/\/127\.0\.0\.1:\d+$/u),
|
||||
botToken: "424242:crabline-telegram-token",
|
||||
dmPolicy: "open",
|
||||
enabled: true,
|
||||
groupPolicy: "open",
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(transport.buildAgentDelivery({ target: "dm:alice" })).toEqual({
|
||||
channel: "telegram",
|
||||
to: "100001",
|
||||
replyChannel: "telegram",
|
||||
replyTo: "100001",
|
||||
});
|
||||
|
||||
const manifest = JSON.parse(
|
||||
await fs.readFile(path.join(outputDir, OPENCLAW_CRABLINE_MANIFEST_PATH), "utf8"),
|
||||
) as {
|
||||
provider?: string;
|
||||
};
|
||||
expect(manifest.provider).toBe("telegram");
|
||||
} finally {
|
||||
await transport.cleanup?.();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
it("injects inbound messages through Crabline and mirrors Telegram sends into normalized state", async () => {
|
||||
await withTempDir("qa-crabline-transport-", async (outputDir) => {
|
||||
const transport = await createQaCrablineTransportAdapter({
|
||||
outputDir,
|
||||
selection: createSelection(),
|
||||
state: createQaBusState(),
|
||||
});
|
||||
|
||||
try {
|
||||
await transport.state.addInboundMessage({
|
||||
conversation: {
|
||||
id: "alice",
|
||||
kind: "direct",
|
||||
},
|
||||
senderId: "alice",
|
||||
senderName: "Alice",
|
||||
text: "DM baseline marker check.",
|
||||
});
|
||||
|
||||
const config = transport.createGatewayConfig({ baseUrl: "http://127.0.0.1:1" });
|
||||
const telegram = config.channels?.telegram as
|
||||
| { apiRoot?: string; botToken?: string }
|
||||
| undefined;
|
||||
expect(telegram?.apiRoot).toBeTruthy();
|
||||
expect(telegram?.botToken).toBeTruthy();
|
||||
const { response, release } = await fetchWithSsrFGuard({
|
||||
url: `${telegram?.apiRoot}/bot${telegram?.botToken}/sendMessage`,
|
||||
init: {
|
||||
body: JSON.stringify({
|
||||
chat_id: "100001",
|
||||
text: "assistant via fake telegram",
|
||||
}),
|
||||
headers: { "content-type": "application/json" },
|
||||
method: "POST",
|
||||
},
|
||||
policy: { allowPrivateNetwork: true },
|
||||
auditContext: "qa-lab-crabline-transport-test",
|
||||
});
|
||||
await release();
|
||||
expect(response.ok).toBe(true);
|
||||
|
||||
await expect(
|
||||
transport.state.waitFor({
|
||||
direction: "outbound",
|
||||
kind: "message-text",
|
||||
textIncludes: "assistant via fake telegram",
|
||||
timeoutMs: 1_000,
|
||||
}),
|
||||
).resolves.toMatchObject({
|
||||
conversation: {
|
||||
id: "alice",
|
||||
kind: "direct",
|
||||
},
|
||||
direction: "outbound",
|
||||
text: "assistant via fake telegram",
|
||||
});
|
||||
} finally {
|
||||
await transport.cleanup?.();
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,300 @@
|
||||
// Qa Lab plugin module implements Crabline fake-provider transport behavior.
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { setTimeout as sleep } from "node:timers/promises";
|
||||
import {
|
||||
OPENCLAW_CRABLINE_MANIFEST_PATH,
|
||||
startOpenClawCrablineAdapter,
|
||||
type OpenClawCrablineChannelDriverSelection,
|
||||
type StartedOpenClawCrablineAdapter,
|
||||
} from "crabline";
|
||||
import type { OpenClawConfig } from "openclaw/plugin-sdk/config-contracts";
|
||||
import { formatErrorMessage } from "openclaw/plugin-sdk/error-runtime";
|
||||
import { fetchWithSsrFGuard } from "openclaw/plugin-sdk/ssrf-runtime";
|
||||
import { createQaBusState, type QaBusState } from "./bus-state.js";
|
||||
import { QaSuiteInfraError } from "./errors.js";
|
||||
import { QaStateBackedTransportAdapter } from "./qa-transport.js";
|
||||
import type {
|
||||
QaTransportActionName,
|
||||
QaTransportGatewayClient,
|
||||
QaTransportGatewayConfig,
|
||||
QaTransportReportParams,
|
||||
QaTransportState,
|
||||
} from "./qa-transport.js";
|
||||
import type {
|
||||
QaBusInboundMessageInput,
|
||||
QaBusOutboundMessageInput,
|
||||
QaBusSearchMessagesInput,
|
||||
QaBusWaitForInput,
|
||||
} from "./runtime-api.js";
|
||||
|
||||
const CRABLINE_TRANSPORT_ID = "crabline";
|
||||
const RECORDER_SYNC_INTERVAL_MS = 50;
|
||||
|
||||
type QaCrablineTransportState = QaTransportState & {
|
||||
cleanup: () => Promise<void>;
|
||||
};
|
||||
|
||||
async function waitForCrablineReady(params: {
|
||||
accountId: string;
|
||||
channel: string;
|
||||
gateway: QaTransportGatewayClient;
|
||||
timeoutMs?: number;
|
||||
pollIntervalMs?: number;
|
||||
}) {
|
||||
const timeoutMs = params.timeoutMs ?? 45_000;
|
||||
const pollIntervalMs = params.pollIntervalMs ?? 500;
|
||||
const startedAt = Date.now();
|
||||
let lastAccountStatus = `no ${params.channel} accounts reported`;
|
||||
let lastProbeError: string | null = null;
|
||||
|
||||
while (Date.now() - startedAt < timeoutMs) {
|
||||
try {
|
||||
const payload = (await params.gateway.call(
|
||||
"channels.status",
|
||||
{ probe: false, timeoutMs: 2_000 },
|
||||
{ timeoutMs: 5_000 },
|
||||
)) as {
|
||||
channelAccounts?: Record<
|
||||
string,
|
||||
Array<{
|
||||
accountId?: string;
|
||||
running?: boolean;
|
||||
restartPending?: boolean;
|
||||
}>
|
||||
>;
|
||||
};
|
||||
const accounts = payload.channelAccounts?.[params.channel] ?? [];
|
||||
const account = accounts.find((entry) => entry.accountId === params.accountId) ?? accounts[0];
|
||||
lastProbeError = null;
|
||||
lastAccountStatus = account
|
||||
? JSON.stringify({
|
||||
accountId: account.accountId ?? null,
|
||||
running: account.running ?? null,
|
||||
restartPending: account.restartPending ?? null,
|
||||
})
|
||||
: `no ${params.channel} accounts reported`;
|
||||
if (account?.running && account.restartPending !== true) {
|
||||
return;
|
||||
}
|
||||
} catch (error) {
|
||||
lastProbeError = formatErrorMessage(error);
|
||||
}
|
||||
await sleep(pollIntervalMs);
|
||||
}
|
||||
|
||||
throw new QaSuiteInfraError(
|
||||
"transport_ready_timeout",
|
||||
[
|
||||
`timed out after ${timeoutMs}ms waiting for ${params.channel} ready`,
|
||||
`last status: ${lastAccountStatus}`,
|
||||
...(lastProbeError ? [`last probe error: ${lastProbeError}`] : []),
|
||||
].join("; "),
|
||||
);
|
||||
}
|
||||
|
||||
async function postCrablineInbound(params: {
|
||||
adapter: StartedOpenClawCrablineAdapter;
|
||||
providerBody: Record<string, unknown>;
|
||||
}) {
|
||||
const { response, release } = await fetchWithSsrFGuard({
|
||||
url: params.adapter.manifest.endpoints.adminInboundUrl,
|
||||
init: {
|
||||
body: JSON.stringify(params.providerBody),
|
||||
headers: { "content-type": "application/json" },
|
||||
method: "POST",
|
||||
},
|
||||
policy: { allowPrivateNetwork: true },
|
||||
auditContext: `qa-lab-crabline-${params.adapter.channel}-inbound`,
|
||||
});
|
||||
try {
|
||||
if (!response.ok) {
|
||||
throw new Error(
|
||||
`Crabline ${params.adapter.channel} inbound injection failed with HTTP ${response.status}.`,
|
||||
);
|
||||
}
|
||||
} finally {
|
||||
await release();
|
||||
}
|
||||
}
|
||||
|
||||
function createCrablineState(params: {
|
||||
adapter: StartedOpenClawCrablineAdapter;
|
||||
state: QaBusState;
|
||||
}): QaCrablineTransportState {
|
||||
const baseState = params.state;
|
||||
const targetByProviderTarget = new Map<string, string>();
|
||||
let recorderLineCursor = 0;
|
||||
let syncPromise: Promise<void> | null = null;
|
||||
|
||||
const syncRecorder = async () => {
|
||||
if (syncPromise) {
|
||||
return await syncPromise;
|
||||
}
|
||||
syncPromise = (async () => {
|
||||
const text = await fs
|
||||
.readFile(params.adapter.manifest.recorderPath, "utf8")
|
||||
.catch((error: unknown) => {
|
||||
if ((error as NodeJS.ErrnoException).code === "ENOENT") {
|
||||
return "";
|
||||
}
|
||||
throw error;
|
||||
});
|
||||
const lines = text.split(/\r?\n/u).filter((line) => line.trim().length > 0);
|
||||
for (const line of lines.slice(recorderLineCursor)) {
|
||||
const parsed = JSON.parse(line) as unknown;
|
||||
const outbound = params.adapter.createOutboundFromRecorderEvent({
|
||||
event: parsed,
|
||||
targetByProviderTarget,
|
||||
}) as QaBusOutboundMessageInput | null;
|
||||
if (outbound) {
|
||||
baseState.addOutboundMessage(outbound);
|
||||
}
|
||||
}
|
||||
recorderLineCursor = lines.length;
|
||||
})();
|
||||
try {
|
||||
await syncPromise;
|
||||
} finally {
|
||||
syncPromise = null;
|
||||
}
|
||||
};
|
||||
|
||||
const interval = setInterval(() => {
|
||||
void syncRecorder().catch(() => undefined);
|
||||
}, RECORDER_SYNC_INTERVAL_MS);
|
||||
interval.unref?.();
|
||||
|
||||
return {
|
||||
async reset() {
|
||||
await syncRecorder();
|
||||
baseState.reset();
|
||||
targetByProviderTarget.clear();
|
||||
recorderLineCursor = await fs
|
||||
.readFile(params.adapter.manifest.recorderPath, "utf8")
|
||||
.then((text) => text.split(/\r?\n/u).filter((line) => line.trim().length > 0).length)
|
||||
.catch(() => 0);
|
||||
},
|
||||
getSnapshot: baseState.getSnapshot.bind(baseState),
|
||||
async addInboundMessage(input: QaBusInboundMessageInput) {
|
||||
const providerInbound = params.adapter.createInbound({ input });
|
||||
targetByProviderTarget.set(providerInbound.providerTargetKey, providerInbound.qaTarget);
|
||||
const message = baseState.addInboundMessage({
|
||||
...input,
|
||||
conversation: providerInbound.stateConversation,
|
||||
...(providerInbound.threadId ? { threadId: providerInbound.threadId } : {}),
|
||||
});
|
||||
await postCrablineInbound({
|
||||
adapter: params.adapter,
|
||||
providerBody: providerInbound.providerBody,
|
||||
});
|
||||
return message;
|
||||
},
|
||||
addOutboundMessage: baseState.addOutboundMessage.bind(baseState),
|
||||
readMessage: baseState.readMessage.bind(baseState),
|
||||
async searchMessages(input: QaBusSearchMessagesInput) {
|
||||
await syncRecorder();
|
||||
return baseState.searchMessages(input);
|
||||
},
|
||||
async waitFor(input: QaBusWaitForInput) {
|
||||
await syncRecorder();
|
||||
return await baseState.waitFor(input);
|
||||
},
|
||||
async cleanup() {
|
||||
clearInterval(interval);
|
||||
await syncRecorder();
|
||||
await params.adapter.close();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
class QaCrablineTransport extends QaStateBackedTransportAdapter {
|
||||
readonly #adapter: StartedOpenClawCrablineAdapter;
|
||||
readonly #selection: OpenClawCrablineChannelDriverSelection;
|
||||
readonly #state: QaCrablineTransportState;
|
||||
|
||||
constructor(params: {
|
||||
adapter: StartedOpenClawCrablineAdapter;
|
||||
selection: OpenClawCrablineChannelDriverSelection;
|
||||
state: QaCrablineTransportState;
|
||||
}) {
|
||||
super({
|
||||
id: CRABLINE_TRANSPORT_ID,
|
||||
label: `crabline fake ${params.selection.channel}`,
|
||||
accountId: params.adapter.accountId,
|
||||
requiredPluginIds: params.adapter.requiredPluginIds,
|
||||
state: params.state,
|
||||
});
|
||||
this.#adapter = params.adapter;
|
||||
this.#selection = params.selection;
|
||||
this.#state = params.state;
|
||||
}
|
||||
|
||||
createGatewayConfig = (params: { baseUrl: string }): QaTransportGatewayConfig =>
|
||||
this.#adapter.createGatewayConfig(params) as QaTransportGatewayConfig;
|
||||
|
||||
waitReady = (params: {
|
||||
gateway: QaTransportGatewayClient;
|
||||
timeoutMs?: number;
|
||||
pollIntervalMs?: number;
|
||||
}) =>
|
||||
waitForCrablineReady({
|
||||
...params,
|
||||
accountId: this.#adapter.accountId,
|
||||
channel: this.#adapter.channel,
|
||||
});
|
||||
|
||||
buildAgentDelivery = ({ target }: { target: string }) =>
|
||||
this.#adapter.createAgentDelivery({ target });
|
||||
|
||||
handleAction = async (_params: {
|
||||
action: QaTransportActionName;
|
||||
args: Record<string, unknown>;
|
||||
cfg: OpenClawConfig;
|
||||
accountId?: string | null;
|
||||
}) => {
|
||||
throw new Error(`Crabline fake-provider transport does not support ${_params.action} yet.`);
|
||||
};
|
||||
|
||||
createReportNotes = (_params: QaTransportReportParams) => [
|
||||
`Runs OpenClaw's ${this.#selection.channel} channel plugin against a Crabline fake provider server.`,
|
||||
"No live channel service or external credential lease is required.",
|
||||
];
|
||||
|
||||
async cleanup() {
|
||||
await this.#state.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
export async function createQaCrablineTransportAdapter(params: {
|
||||
outputDir: string;
|
||||
selection: OpenClawCrablineChannelDriverSelection;
|
||||
state?: QaBusState;
|
||||
}) {
|
||||
const recorderPath = path.join(
|
||||
params.outputDir,
|
||||
"artifacts",
|
||||
"crabline",
|
||||
`${params.selection.channel}-fake-provider.jsonl`,
|
||||
);
|
||||
await fs.mkdir(path.dirname(recorderPath), { recursive: true });
|
||||
const adapter = await startOpenClawCrablineAdapter({
|
||||
channel: params.selection.channel,
|
||||
openclawConfig: {},
|
||||
recorderPath,
|
||||
});
|
||||
await fs.writeFile(
|
||||
path.join(params.outputDir, OPENCLAW_CRABLINE_MANIFEST_PATH),
|
||||
`${JSON.stringify(adapter.manifest, null, 2)}\n`,
|
||||
"utf8",
|
||||
);
|
||||
|
||||
return new QaCrablineTransport({
|
||||
adapter,
|
||||
selection: params.selection,
|
||||
state: createCrablineState({
|
||||
adapter,
|
||||
state: params.state ?? createQaBusState(),
|
||||
}),
|
||||
});
|
||||
}
|
||||
@@ -122,7 +122,7 @@ export async function runQaManualLane(params: QaManualLaneParams) {
|
||||
message: params.message,
|
||||
deliver: true,
|
||||
channel: delivery.channel,
|
||||
to: "dm:qa-operator",
|
||||
to: delivery.to ?? "dm:qa-operator",
|
||||
replyChannel: delivery.replyChannel,
|
||||
replyTo: delivery.replyTo,
|
||||
},
|
||||
|
||||
@@ -1351,7 +1351,10 @@ function buildAssistantText(
|
||||
return "Protocol note: I do not have enough context to say what you usually want for QA movie night.";
|
||||
}
|
||||
if (/qa private final reply warning check/i.test(prompt)) {
|
||||
return "QA-STRANDED-85714 is present in this private final reply and I am not calling the message tool. This second sentence makes the omitted delivery substantive enough for the warning check.";
|
||||
return [
|
||||
"QA-STRANDED-85714 confirms this is a substantive private final reply that intentionally stays outside the message tool path for the warning check.",
|
||||
"The response is long enough to exercise message_tool_only private-final detection while remaining private to the agent transcript.",
|
||||
].join(" ");
|
||||
}
|
||||
if (/tool continuity check/i.test(prompt) && toolOutput) {
|
||||
return `Protocol note: model switch handoff confirmed on ${model || "the requested model"}. QA mission from QA_KICKOFF_TASK.md still applies: understand this OpenClaw repo from source + docs before acting.`;
|
||||
|
||||
@@ -172,6 +172,7 @@ export type QaTransportAdapter = {
|
||||
}) => Promise<void>;
|
||||
buildAgentDelivery: (params: { target: string }) => {
|
||||
channel: string;
|
||||
to?: string;
|
||||
replyChannel: string;
|
||||
replyTo: string;
|
||||
};
|
||||
@@ -182,6 +183,7 @@ export type QaTransportAdapter = {
|
||||
accountId?: string | null;
|
||||
}) => Promise<unknown>;
|
||||
createReportNotes: (params: QaTransportReportParams) => string[];
|
||||
cleanup?: () => Promise<void>;
|
||||
};
|
||||
|
||||
export abstract class QaStateBackedTransportAdapter implements QaTransportAdapter {
|
||||
@@ -230,6 +232,7 @@ export abstract class QaStateBackedTransportAdapter implements QaTransportAdapte
|
||||
}) => Promise<void>;
|
||||
abstract buildAgentDelivery: (params: { target: string }) => {
|
||||
channel: string;
|
||||
to?: string;
|
||||
replyChannel: string;
|
||||
replyTo: string;
|
||||
};
|
||||
|
||||
@@ -59,14 +59,23 @@ const qaScenarioRepoRefSchema = z
|
||||
message: "repo refs must not be absolute or contain parent-directory segments",
|
||||
});
|
||||
|
||||
const qaScenarioChannelSchema = z
|
||||
.string()
|
||||
.trim()
|
||||
.regex(/^[a-z0-9]+(?:[.-][a-z0-9]+)*$/, {
|
||||
message: "scenario execution channel ids must use lowercase dotted or dashed tokens",
|
||||
});
|
||||
|
||||
const qaFlowScenarioExecutionSchema = z.object({
|
||||
kind: z.literal("flow").default("flow"),
|
||||
summary: z.string().trim().min(1).optional(),
|
||||
channel: qaScenarioChannelSchema.optional(),
|
||||
config: qaScenarioConfigSchema.optional(),
|
||||
});
|
||||
|
||||
const qaTestFileScenarioExecutionBaseSchema = z.object({
|
||||
summary: z.string().trim().min(1).optional(),
|
||||
channel: qaScenarioChannelSchema.optional(),
|
||||
path: qaScenarioRepoRefSchema,
|
||||
config: qaScenarioConfigSchema.optional(),
|
||||
});
|
||||
|
||||
@@ -28,12 +28,14 @@ function isRepoRootRelativeRef(value: string) {
|
||||
|
||||
const qaCoverageEvidenceRoleSchema = z.enum(["primary", "secondary"]);
|
||||
export const qaScorecardEvidenceModeSchema = z.enum(["full", "slim"]);
|
||||
export const qaScorecardChannelDriverSchema = z.enum(["qa-channel", "crabline", "live"]);
|
||||
|
||||
const qaScorecardProfileSchema = z.object({
|
||||
id: qaScorecardIdSchema,
|
||||
description: z.string().trim().min(1),
|
||||
evidenceMode: qaScorecardEvidenceModeSchema.optional(),
|
||||
includeAllCategories: z.boolean().default(false),
|
||||
channelDriver: qaScorecardChannelDriverSchema.default("qa-channel"),
|
||||
categoryIds: z.array(qaScorecardIdSchema).default([]),
|
||||
});
|
||||
|
||||
@@ -83,6 +85,20 @@ const qaMaturityTaxonomySchema = z
|
||||
message: `profile ${profile.id} cannot set categoryIds when includeAllCategories is true`,
|
||||
});
|
||||
}
|
||||
if (profile.channelDriver === "crabline" && profile.includeAllCategories) {
|
||||
ctx.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
path: ["profiles", profileIndex, "includeAllCategories"],
|
||||
message: `profile ${profile.id} cannot set includeAllCategories when channelDriver is crabline`,
|
||||
});
|
||||
}
|
||||
if (profile.channelDriver === "crabline" && !profile.categoryIds.length) {
|
||||
ctx.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
path: ["profiles", profileIndex, "categoryIds"],
|
||||
message: `profile ${profile.id} requires categoryIds when channelDriver is crabline`,
|
||||
});
|
||||
}
|
||||
|
||||
const seenProfileCategoryIds = new Set<string>();
|
||||
for (const [categoryIndex, categoryId] of profile.categoryIds.entries()) {
|
||||
@@ -101,6 +117,7 @@ const qaMaturityTaxonomySchema = z
|
||||
export type QaNativeCoverageEvidenceKind = "script" | "vitest" | "playwright";
|
||||
export type QaScorecardEvidenceKind = QaNativeCoverageEvidenceKind | "qa-scenario";
|
||||
export type QaScorecardEvidenceMode = z.infer<typeof qaScorecardEvidenceModeSchema>;
|
||||
export type QaScorecardChannelDriver = z.infer<typeof qaScorecardChannelDriverSchema>;
|
||||
type QaCoverageEvidenceRole = z.infer<typeof qaCoverageEvidenceRoleSchema>;
|
||||
type QaMaturityTaxonomy = z.infer<typeof qaMaturityTaxonomySchema>;
|
||||
|
||||
@@ -146,6 +163,7 @@ export type QaScorecardCategoryCoverageReport = {
|
||||
export type QaScorecardProfileReport = {
|
||||
id: string;
|
||||
evidenceMode: QaScorecardEvidenceMode;
|
||||
channelDriver: QaScorecardChannelDriver;
|
||||
categoryIds: string[];
|
||||
};
|
||||
|
||||
@@ -333,12 +351,14 @@ export function readQaScorecardFeatureCoverageByCategory(repoRoot?: string) {
|
||||
export function readQaScorecardProfileOptions(profileId: string | undefined, repoRoot?: string) {
|
||||
const profile = profileId?.trim();
|
||||
if (!profile) {
|
||||
return { evidenceMode: "full" as const };
|
||||
return { evidenceMode: "full" as const, channelDriver: "qa-channel" as const };
|
||||
}
|
||||
const profileOptions = readQaMaturityTaxonomy(repoRoot)?.profiles.find(
|
||||
(entry) => entry.id === profile,
|
||||
);
|
||||
return {
|
||||
evidenceMode:
|
||||
readQaMaturityTaxonomy(repoRoot)?.profiles.find((entry) => entry.id === profile)
|
||||
?.evidenceMode ?? "full",
|
||||
evidenceMode: profileOptions?.evidenceMode ?? "full",
|
||||
channelDriver: profileOptions?.channelDriver ?? "qa-channel",
|
||||
};
|
||||
}
|
||||
|
||||
@@ -478,6 +498,7 @@ export function buildQaScorecardTaxonomyReport(params: {
|
||||
return {
|
||||
id: profile.id,
|
||||
evidenceMode: profile.evidenceMode ?? "full",
|
||||
channelDriver: profile.channelDriver,
|
||||
categoryIds: validCategoryIds,
|
||||
};
|
||||
}) ?? [];
|
||||
|
||||
@@ -10,6 +10,7 @@ import {
|
||||
collectQaSuitePluginIds,
|
||||
mapQaSuiteWithConcurrency,
|
||||
normalizeQaSuiteConcurrency,
|
||||
resolveQaSuiteScenarioChannel,
|
||||
resolveQaSuiteWorkerStartStaggerMs,
|
||||
resolveQaSuiteOutputDir,
|
||||
scenarioRequiresControlUi,
|
||||
@@ -241,6 +242,47 @@ describe("qa suite planning helpers", () => {
|
||||
).toEqual(["third", "first"]);
|
||||
});
|
||||
|
||||
it("resolves driver channels from scenario execution with explicit and default fallbacks", () => {
|
||||
expect(
|
||||
resolveQaSuiteScenarioChannel({
|
||||
defaultChannel: "telegram",
|
||||
scenarios: [makeQaSuiteTestScenario("plain")],
|
||||
}),
|
||||
).toBe("telegram");
|
||||
expect(
|
||||
resolveQaSuiteScenarioChannel({
|
||||
defaultChannel: "telegram",
|
||||
scenarios: [
|
||||
makeQaSuiteTestScenario("plain"),
|
||||
makeQaSuiteTestScenario("slack-flow", { channel: "slack" }),
|
||||
],
|
||||
}),
|
||||
).toBe("slack");
|
||||
expect(
|
||||
resolveQaSuiteScenarioChannel({
|
||||
defaultChannel: "telegram",
|
||||
explicitChannel: "slack",
|
||||
scenarios: [makeQaSuiteTestScenario("slack-flow", { channel: "slack" })],
|
||||
}),
|
||||
).toBe("slack");
|
||||
expect(() =>
|
||||
resolveQaSuiteScenarioChannel({
|
||||
defaultChannel: "telegram",
|
||||
explicitChannel: "telegram",
|
||||
scenarios: [makeQaSuiteTestScenario("slack-flow", { channel: "slack" })],
|
||||
}),
|
||||
).toThrow("--channel telegram conflicts with selected scenario execution.channel slack.");
|
||||
expect(() =>
|
||||
resolveQaSuiteScenarioChannel({
|
||||
defaultChannel: "telegram",
|
||||
scenarios: [
|
||||
makeQaSuiteTestScenario("slack-flow", { channel: "slack" }),
|
||||
makeQaSuiteTestScenario("telegram-flow", { channel: "telegram" }),
|
||||
],
|
||||
}),
|
||||
).toThrow("Selected QA scenarios require multiple channels");
|
||||
});
|
||||
|
||||
it("collects unique scenario-declared bundled plugins in encounter order", () => {
|
||||
const scenarios = [
|
||||
makeQaSuiteTestScenario("generic", { plugins: ["active-memory", "memory-wiki"] }),
|
||||
|
||||
@@ -97,6 +97,45 @@ function selectQaFlowSuiteScenarios(params: {
|
||||
);
|
||||
}
|
||||
|
||||
function listQaSuiteScenarioChannels(
|
||||
scenarios: ReturnType<typeof readQaBootstrapScenarioCatalog>["scenarios"],
|
||||
) {
|
||||
return [
|
||||
...new Set(
|
||||
scenarios
|
||||
.map((scenario) => scenario.execution.channel?.trim().toLowerCase())
|
||||
.filter((channel): channel is string => Boolean(channel)),
|
||||
),
|
||||
];
|
||||
}
|
||||
|
||||
function resolveQaSuiteScenarioChannel(params: {
|
||||
defaultChannel: string;
|
||||
explicitChannel?: string | null;
|
||||
scenarios: ReturnType<typeof readQaBootstrapScenarioCatalog>["scenarios"];
|
||||
}) {
|
||||
const scenarioChannels = listQaSuiteScenarioChannels(params.scenarios);
|
||||
const explicitChannel = params.explicitChannel?.trim().toLowerCase();
|
||||
if (explicitChannel) {
|
||||
const conflictingChannels = scenarioChannels.filter((channel) => channel !== explicitChannel);
|
||||
if (conflictingChannels.length > 0) {
|
||||
throw new Error(
|
||||
`--channel ${explicitChannel} conflicts with selected scenario execution.channel ${conflictingChannels.join(", ")}.`,
|
||||
);
|
||||
}
|
||||
return explicitChannel;
|
||||
}
|
||||
if (scenarioChannels.length === 0) {
|
||||
return params.defaultChannel;
|
||||
}
|
||||
if (scenarioChannels.length === 1) {
|
||||
return scenarioChannels[0];
|
||||
}
|
||||
throw new Error(
|
||||
`Selected QA scenarios require multiple channels (${scenarioChannels.join(", ")}); split the run by channel.`,
|
||||
);
|
||||
}
|
||||
|
||||
function collectQaSuitePluginIds(
|
||||
scenarios: ReturnType<typeof readQaBootstrapScenarioCatalog>["scenarios"],
|
||||
) {
|
||||
@@ -277,6 +316,7 @@ export {
|
||||
collectQaSuitePluginIds,
|
||||
mapQaSuiteWithConcurrency,
|
||||
normalizeQaSuiteConcurrency,
|
||||
resolveQaSuiteScenarioChannel,
|
||||
resolveQaSuiteWorkerStartStaggerMs,
|
||||
resolveQaSuiteOutputDir,
|
||||
scenarioRequiresControlUi,
|
||||
|
||||
@@ -595,6 +595,7 @@ describe("qa suite runtime agent process helpers", () => {
|
||||
transport: {
|
||||
buildAgentDelivery: vi.fn(() => ({
|
||||
channel: "qa-channel",
|
||||
to: "transport-target",
|
||||
replyChannel: "reply-channel",
|
||||
replyTo: "reply-target",
|
||||
})),
|
||||
@@ -616,11 +617,13 @@ describe("qa suite runtime agent process helpers", () => {
|
||||
replyChannel?: string;
|
||||
replyTo?: string;
|
||||
sessionKey?: string;
|
||||
to?: string;
|
||||
}
|
||||
| undefined;
|
||||
expect(agentPayload?.sessionKey).toBe("session-1");
|
||||
expect(agentPayload?.message).toBe("hello");
|
||||
expect(agentPayload?.channel).toBe("qa-channel");
|
||||
expect(agentPayload?.to).toBe("transport-target");
|
||||
expect(agentPayload?.replyChannel).toBe("reply-channel");
|
||||
expect(agentPayload?.replyTo).toBe("reply-target");
|
||||
expect(gatewayArgs?.[2]).toBeTypeOf("object");
|
||||
|
||||
@@ -324,7 +324,7 @@ async function startAgentRun(
|
||||
message: params.message,
|
||||
deliver: true,
|
||||
channel: delivery.channel,
|
||||
to: target,
|
||||
to: delivery.to ?? target,
|
||||
replyChannel: delivery.replyChannel,
|
||||
replyTo: delivery.replyTo,
|
||||
...(params.threadId ? { threadId: params.threadId } : {}),
|
||||
|
||||
@@ -5,6 +5,7 @@ import { QaSuiteArtifactError } from "./errors.js";
|
||||
import type { QaEvidenceSummaryJson } from "./evidence-summary.js";
|
||||
import type { QaProviderMode } from "./model-selection.js";
|
||||
import type { RuntimeId, RuntimeParityResult } from "./runtime-parity.js";
|
||||
import type { QaScorecardChannelDriver } from "./scorecard-taxonomy.js";
|
||||
|
||||
type QaSuiteSummaryScenario = {
|
||||
name: string;
|
||||
@@ -55,6 +56,10 @@ export type QaSuiteSummaryJson = {
|
||||
alternateModelName: string | null;
|
||||
fastMode: boolean;
|
||||
concurrency: number;
|
||||
channelDriver: QaScorecardChannelDriver | null;
|
||||
channel: string | null;
|
||||
channelCapabilityMatrixPath: string | null;
|
||||
channelDriverSmokePath: string | null;
|
||||
scenarioIds: string[] | null;
|
||||
runtimePair?: [RuntimeId, RuntimeId] | null;
|
||||
};
|
||||
|
||||
@@ -6,6 +6,7 @@ type QaSuiteTestScenario = ReturnType<typeof readQaBootstrapScenarioCatalog>["sc
|
||||
export function makeQaSuiteTestScenario(
|
||||
id: string,
|
||||
params: {
|
||||
channel?: string;
|
||||
config?: Record<string, unknown>;
|
||||
plugins?: string[];
|
||||
gatewayConfigPatch?: Record<string, unknown>;
|
||||
@@ -27,6 +28,7 @@ export function makeQaSuiteTestScenario(
|
||||
sourcePath: `qa/scenarios/${id}.yaml`,
|
||||
execution: {
|
||||
kind: "flow",
|
||||
...(params.channel ? { channel: params.channel } : {}),
|
||||
...(params.config ? { config: params.config } : {}),
|
||||
flow: { steps: [{ name: "noop", actions: [{ assert: "true" }] }] },
|
||||
},
|
||||
|
||||
@@ -34,9 +34,42 @@ describe("buildQaSuiteSummaryJson", () => {
|
||||
expect(json.run.alternateModelName).toBe("gpt-5.5-alt");
|
||||
expect(json.run.fastMode).toBe(true);
|
||||
expect(json.run.concurrency).toBe(2);
|
||||
expect(json.run.channelDriver).toBeNull();
|
||||
expect(json.run.channel).toBeNull();
|
||||
expect(json.run.channelCapabilityMatrixPath).toBeNull();
|
||||
expect(json.run.channelDriverSmokePath).toBeNull();
|
||||
expect(json.run.scenarioIds).toBeNull();
|
||||
});
|
||||
|
||||
it("records Crabline channel-driver metadata when selected", () => {
|
||||
const json = buildQaSuiteSummaryJson({
|
||||
...baseParams,
|
||||
channelDriverSelection: {
|
||||
capabilityMatrixPath: "crabline-fake-provider-capabilities.json",
|
||||
channel: "telegram",
|
||||
channelDriver: "crabline",
|
||||
smokeArtifactPath: "crabline-fake-provider-smoke.json",
|
||||
},
|
||||
});
|
||||
|
||||
expect(json.run.channelDriver).toBe("crabline");
|
||||
expect(json.run.channel).toBe("telegram");
|
||||
expect(json.run.channelCapabilityMatrixPath).toBe("crabline-fake-provider-capabilities.json");
|
||||
expect(json.run.channelDriverSmokePath).toBe("crabline-fake-provider-smoke.json");
|
||||
});
|
||||
|
||||
it("records declarative non-Crabline channel-driver metadata", () => {
|
||||
const json = buildQaSuiteSummaryJson({
|
||||
...baseParams,
|
||||
channelDriver: "live",
|
||||
});
|
||||
|
||||
expect(json.run.channelDriver).toBe("live");
|
||||
expect(json.run.channel).toBeNull();
|
||||
expect(json.run.channelCapabilityMatrixPath).toBeNull();
|
||||
expect(json.run.channelDriverSmokePath).toBeNull();
|
||||
});
|
||||
|
||||
it("includes scenarioIds in run metadata when provided", () => {
|
||||
const scenarioIds = ["approval-turn-tool-followthrough", "subagent-handoff", "memory-recall"];
|
||||
const json = buildQaSuiteSummaryJson({
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
// Qa Lab tests cover suite plugin behavior.
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { QA_EVIDENCE_FILENAME, QA_EVIDENCE_SUMMARY_KIND } from "./evidence-summary.js";
|
||||
@@ -8,16 +7,19 @@ import type { QaLabServerHandle } from "./lab-server.types.js";
|
||||
import type { QaTransportAdapter } from "./qa-transport.js";
|
||||
import { makeQaSuiteTestScenario } from "./suite-test-helpers.js";
|
||||
import { qaSuiteProgressTesting, runQaFlowSuite } from "./suite.js";
|
||||
import { createTempDirHarness } from "./temp-dir.test-helper.js";
|
||||
|
||||
const fetchWithSsrFGuardMock = vi.hoisted(() => vi.fn());
|
||||
const tempDirs = createTempDirHarness();
|
||||
|
||||
vi.mock("openclaw/plugin-sdk/ssrf-runtime", () => ({
|
||||
fetchWithSsrFGuard: fetchWithSsrFGuardMock,
|
||||
}));
|
||||
|
||||
afterEach(() => {
|
||||
afterEach(async () => {
|
||||
fetchWithSsrFGuardMock.mockReset();
|
||||
vi.useRealTimers();
|
||||
await tempDirs.cleanup();
|
||||
});
|
||||
|
||||
function makeQaSuiteTestLabHandle(): QaLabServerHandle {
|
||||
@@ -228,7 +230,7 @@ describe("qa suite", () => {
|
||||
});
|
||||
|
||||
it("writes standalone evidence while keeping suite summary evidence-free", async () => {
|
||||
const outputDir = await fs.mkdtemp(path.join(os.tmpdir(), "qa-suite-artifacts-"));
|
||||
const outputDir = await tempDirs.makeTempDir("qa-suite-artifacts-");
|
||||
try {
|
||||
const artifacts = await qaSuiteProgressTesting.writeQaSuiteArtifacts({
|
||||
outputDir,
|
||||
@@ -272,6 +274,80 @@ describe("qa suite", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("writes Crabline channel-driver smoke artifacts when selected", async () => {
|
||||
const outputDir = await tempDirs.makeTempDir("qa-suite-crabline-");
|
||||
try {
|
||||
fetchWithSsrFGuardMock.mockResolvedValue({
|
||||
response: {
|
||||
ok: true,
|
||||
json: vi.fn(async () => ({
|
||||
ok: true,
|
||||
result: {
|
||||
is_bot: true,
|
||||
username: "crabline_bot",
|
||||
},
|
||||
})),
|
||||
},
|
||||
release: vi.fn(async () => {}),
|
||||
});
|
||||
|
||||
const artifacts = await qaSuiteProgressTesting.writeQaSuiteArtifacts({
|
||||
outputDir,
|
||||
startedAt: new Date("2026-04-11T00:00:00.000Z"),
|
||||
finishedAt: new Date("2026-04-11T00:01:00.000Z"),
|
||||
scenarios: [{ name: "Telegram DM", status: "pass", steps: [] }],
|
||||
scenarioDefinitions: [
|
||||
{
|
||||
...makeQaSuiteTestScenario("telegram-dm", {
|
||||
surface: "channel",
|
||||
}),
|
||||
coverage: {
|
||||
primary: ["channels.dm"],
|
||||
},
|
||||
},
|
||||
],
|
||||
transport: {
|
||||
id: "qa-channel",
|
||||
createReportNotes: () => [],
|
||||
} as unknown as QaTransportAdapter,
|
||||
providerMode: "mock-openai",
|
||||
primaryModel: "mock-openai/gpt-5.5",
|
||||
alternateModel: "mock-openai/gpt-5.5-alt",
|
||||
fastMode: true,
|
||||
concurrency: 1,
|
||||
channelDriverSelection: {
|
||||
capabilityMatrixPath: "crabline-fake-provider-capabilities.json",
|
||||
channel: "telegram",
|
||||
channelDriver: "crabline",
|
||||
smokeArtifactPath: "crabline-fake-provider-smoke.json",
|
||||
},
|
||||
});
|
||||
|
||||
const matrix = JSON.parse(
|
||||
await fs.readFile(path.join(outputDir, "crabline-fake-provider-capabilities.json"), "utf8"),
|
||||
) as {
|
||||
report?: { result?: { selectedChannel?: string; supportedChannels?: string[] } };
|
||||
};
|
||||
expect(matrix.report?.result).toMatchObject({
|
||||
selectedChannel: "telegram",
|
||||
supportedChannels: ["telegram"],
|
||||
});
|
||||
const smoke = JSON.parse(
|
||||
await fs.readFile(path.join(outputDir, "crabline-fake-provider-smoke.json"), "utf8"),
|
||||
) as { smoke?: { result?: { ok?: boolean; provider?: string } } };
|
||||
expect(smoke.smoke?.result).toMatchObject({ ok: true, provider: "telegram" });
|
||||
const evidence = JSON.parse(await fs.readFile(artifacts.evidencePath, "utf8")) as {
|
||||
entries?: Array<{ execution?: { channel?: { driver?: string; id?: string } } }>;
|
||||
};
|
||||
expect(evidence.entries?.[0]?.execution?.channel).toMatchObject({
|
||||
driver: "crabline",
|
||||
id: "telegram",
|
||||
});
|
||||
} finally {
|
||||
await fs.rm(outputDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("arms gateway heap checkpoint env only when requested", () => {
|
||||
expect(
|
||||
qaSuiteProgressTesting.buildQaGatewayHeapCheckpointRuntimeEnvPatch({
|
||||
|
||||
+133
-10
@@ -2,6 +2,11 @@
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { setTimeout as sleep } from "node:timers/promises";
|
||||
import {
|
||||
createOpenClawCrablineChannelReportNotes,
|
||||
runOpenClawCrablineChannelDriverSmoke,
|
||||
type OpenClawCrablineChannelDriverSelection,
|
||||
} from "crabline";
|
||||
import { disposeRegisteredAgentHarnesses } from "openclaw/plugin-sdk/agent-harness";
|
||||
import type { OpenClawConfig } from "openclaw/plugin-sdk/config-contracts";
|
||||
import { formatErrorMessage } from "openclaw/plugin-sdk/error-runtime";
|
||||
@@ -55,7 +60,7 @@ import {
|
||||
type QaSeedScenarioWithSource,
|
||||
} from "./scenario-catalog.js";
|
||||
import { runScenarioFlow } from "./scenario-flow-runner.js";
|
||||
import type { QaScorecardEvidenceMode } from "./scorecard-taxonomy.js";
|
||||
import type { QaScorecardChannelDriver, QaScorecardEvidenceMode } from "./scorecard-taxonomy.js";
|
||||
import {
|
||||
applyQaMergePatch,
|
||||
collectQaSuiteGatewayConfigPatch,
|
||||
@@ -105,12 +110,34 @@ type QaSuiteEnvironment = {
|
||||
|
||||
export type QaSuiteStartLabFn = (params?: QaLabServerStartParams) => Promise<QaLabServerHandle>;
|
||||
|
||||
async function createQaSuiteTransportAdapter(params: {
|
||||
channelDriverSelection?: OpenClawCrablineChannelDriverSelection | null;
|
||||
outputDir: string;
|
||||
state: QaLabServerHandle["state"];
|
||||
transportId: QaTransportId;
|
||||
}) {
|
||||
if (params.channelDriverSelection) {
|
||||
const { createQaCrablineTransportAdapter } = await import("./crabline-transport.js");
|
||||
return await createQaCrablineTransportAdapter({
|
||||
outputDir: params.outputDir,
|
||||
selection: params.channelDriverSelection,
|
||||
state: params.state,
|
||||
});
|
||||
}
|
||||
return createQaTransportAdapter({
|
||||
id: params.transportId,
|
||||
state: params.state,
|
||||
});
|
||||
}
|
||||
|
||||
export type QaSuiteRunParams = {
|
||||
evidenceMode?: QaScorecardEvidenceMode;
|
||||
repoRoot?: string;
|
||||
outputDir?: string;
|
||||
providerMode?: QaProviderMode;
|
||||
transportId?: QaTransportId;
|
||||
channelDriver?: QaScorecardChannelDriver;
|
||||
channelDriverSelection?: OpenClawCrablineChannelDriverSelection | null;
|
||||
primaryModel?: string;
|
||||
alternateModel?: string;
|
||||
fastMode?: boolean;
|
||||
@@ -394,6 +421,7 @@ function buildRuntimeParityScenarioResult(params: {
|
||||
|
||||
function createQaSuiteReportNotes(params: {
|
||||
transport: QaTransportAdapter;
|
||||
channelDriverSelection?: OpenClawCrablineChannelDriverSelection | null;
|
||||
providerMode: QaProviderMode;
|
||||
primaryModel: string;
|
||||
alternateModel: string;
|
||||
@@ -401,7 +429,10 @@ function createQaSuiteReportNotes(params: {
|
||||
concurrency: number;
|
||||
isolatedWorkers?: boolean;
|
||||
}) {
|
||||
return params.transport.createReportNotes(params);
|
||||
return [
|
||||
...params.transport.createReportNotes(params),
|
||||
...createOpenClawCrablineChannelReportNotes(params.channelDriverSelection),
|
||||
];
|
||||
}
|
||||
|
||||
function buildQaIsolatedScenarioWorkerParams(params: {
|
||||
@@ -409,6 +440,8 @@ function buildQaIsolatedScenarioWorkerParams(params: {
|
||||
outputDir: string;
|
||||
providerMode: QaProviderMode;
|
||||
transportId: QaTransportId;
|
||||
channelDriver?: QaScorecardChannelDriver;
|
||||
channelDriverSelection?: OpenClawCrablineChannelDriverSelection | null;
|
||||
primaryModel: string;
|
||||
alternateModel: string;
|
||||
fastMode: boolean;
|
||||
@@ -421,6 +454,8 @@ function buildQaIsolatedScenarioWorkerParams(params: {
|
||||
outputDir: params.outputDir,
|
||||
providerMode: params.providerMode,
|
||||
transportId: params.transportId,
|
||||
channelDriver: params.channelDriver,
|
||||
channelDriverSelection: params.channelDriverSelection,
|
||||
primaryModel: params.primaryModel,
|
||||
alternateModel: params.alternateModel,
|
||||
fastMode: params.fastMode,
|
||||
@@ -526,6 +561,8 @@ export type QaSuiteSummaryJsonParams = {
|
||||
alternateModel: string;
|
||||
fastMode: boolean;
|
||||
concurrency: number;
|
||||
channelDriver?: QaScorecardChannelDriver | null;
|
||||
channelDriverSelection?: OpenClawCrablineChannelDriverSelection | null;
|
||||
scenarioIds?: readonly string[];
|
||||
runtimePair?: [RuntimeId, RuntimeId];
|
||||
};
|
||||
@@ -585,6 +622,10 @@ export function buildQaSuiteSummaryJson(params: QaSuiteSummaryJsonParams): QaSui
|
||||
alternateModelName: alternateSplit?.model ?? null,
|
||||
fastMode: params.fastMode,
|
||||
concurrency: params.concurrency,
|
||||
channelDriver: params.channelDriver ?? params.channelDriverSelection?.channelDriver ?? null,
|
||||
channel: params.channelDriverSelection?.channel ?? null,
|
||||
channelCapabilityMatrixPath: params.channelDriverSelection?.capabilityMatrixPath ?? null,
|
||||
channelDriverSmokePath: params.channelDriverSelection?.smokeArtifactPath ?? null,
|
||||
scenarioIds:
|
||||
params.scenarioIds && params.scenarioIds.length > 0 ? [...params.scenarioIds] : null,
|
||||
runtimePair: params.runtimePair ?? null,
|
||||
@@ -605,6 +646,8 @@ async function runQaRuntimeParitySuite(params: {
|
||||
thinkingDefault?: QaThinkingLevel;
|
||||
claudeCliAuthMode?: QaCliBackendAuthMode;
|
||||
enabledPluginIds?: string[];
|
||||
channelDriver?: QaScorecardChannelDriver | null;
|
||||
channelDriverSelection?: OpenClawCrablineChannelDriverSelection | null;
|
||||
concurrency: number;
|
||||
selectedScenarios: ReturnType<typeof readQaBootstrapScenarioCatalog>["scenarios"];
|
||||
startLab?: QaSuiteStartLabFn;
|
||||
@@ -623,9 +666,11 @@ async function runQaRuntimeParitySuite(params: {
|
||||
port: 0,
|
||||
embeddedGateway: "disabled",
|
||||
}));
|
||||
const transport = createQaTransportAdapter({
|
||||
id: params.transportId,
|
||||
const transport = await createQaSuiteTransportAdapter({
|
||||
channelDriverSelection: params.channelDriverSelection,
|
||||
outputDir: params.outputDir,
|
||||
state: lab.state,
|
||||
transportId: params.transportId,
|
||||
});
|
||||
const liveScenarioOutcomes: QaLabScenarioOutcome[] = params.selectedScenarios.map((scenario) => ({
|
||||
id: scenario.id,
|
||||
@@ -677,6 +722,8 @@ async function runQaRuntimeParitySuite(params: {
|
||||
outputDir: cellOutputDir,
|
||||
providerMode: params.providerMode,
|
||||
transportId: params.transportId,
|
||||
channelDriver: params.channelDriver ?? undefined,
|
||||
channelDriverSelection: params.channelDriverSelection,
|
||||
primaryModel: remapModelRefForForcedRuntime({
|
||||
modelRef: params.primaryModel,
|
||||
providerMode: params.providerMode,
|
||||
@@ -778,6 +825,8 @@ async function runQaRuntimeParitySuite(params: {
|
||||
alternateModel: params.alternateModel,
|
||||
fastMode: params.fastMode,
|
||||
concurrency: params.concurrency,
|
||||
channelDriver: params.channelDriver,
|
||||
channelDriverSelection: params.channelDriverSelection,
|
||||
scenarioIds:
|
||||
params.scenarioIds && params.scenarioIds.length > 0
|
||||
? params.selectedScenarios.map((scenario) => scenario.id)
|
||||
@@ -806,6 +855,7 @@ async function runQaRuntimeParitySuite(params: {
|
||||
watchUrl: lab.baseUrl,
|
||||
} satisfies QaSuiteResult;
|
||||
} finally {
|
||||
await transport.cleanup?.();
|
||||
if (ownsLab) {
|
||||
await lab.stop();
|
||||
}
|
||||
@@ -830,6 +880,8 @@ async function writeQaSuiteArtifacts(params: {
|
||||
alternateModel: string;
|
||||
fastMode: boolean;
|
||||
concurrency: number;
|
||||
channelDriver?: QaScorecardChannelDriver | null;
|
||||
channelDriverSelection?: OpenClawCrablineChannelDriverSelection | null;
|
||||
isolatedWorkers?: boolean;
|
||||
scenarioIds?: readonly string[];
|
||||
runtimePair?: [RuntimeId, RuntimeId];
|
||||
@@ -837,6 +889,25 @@ async function writeQaSuiteArtifacts(params: {
|
||||
const reportPath = path.join(params.outputDir, "qa-suite-report.md");
|
||||
const summaryPath = path.join(params.outputDir, "qa-suite-summary.json");
|
||||
const evidencePath = path.join(params.outputDir, QA_EVIDENCE_FILENAME);
|
||||
const crablineChannelDriverSelection = params.channelDriverSelection;
|
||||
const crablineChannelDriverSmoke = crablineChannelDriverSelection
|
||||
? await runOpenClawCrablineChannelDriverSmoke({
|
||||
outputDir: params.outputDir,
|
||||
selection: crablineChannelDriverSelection,
|
||||
})
|
||||
: undefined;
|
||||
const crablineChannelDriverArtifactPaths = crablineChannelDriverSelection
|
||||
? [
|
||||
{
|
||||
kind: "channel-capability-matrix",
|
||||
path: crablineChannelDriverSelection.capabilityMatrixPath,
|
||||
},
|
||||
{
|
||||
kind: "channel-driver-smoke",
|
||||
path: crablineChannelDriverSelection.smokeArtifactPath,
|
||||
},
|
||||
]
|
||||
: [];
|
||||
const report = renderQaMarkdownReport({
|
||||
title: "OpenClaw QA Scenario Suite",
|
||||
startedAt: params.startedAt,
|
||||
@@ -856,9 +927,11 @@ async function writeQaSuiteArtifacts(params: {
|
||||
artifactPaths: [
|
||||
{ kind: "summary", path: path.basename(summaryPath) },
|
||||
{ kind: "report", path: path.basename(reportPath) },
|
||||
...crablineChannelDriverArtifactPaths,
|
||||
],
|
||||
evidenceMode: params.evidenceMode,
|
||||
channelId: params.transport.id,
|
||||
channelId: params.channelDriverSelection?.channel ?? params.transport.id,
|
||||
channelDriver: params.channelDriver ?? params.channelDriverSelection?.channelDriver,
|
||||
env: process.env,
|
||||
generatedAt: params.finishedAt.toISOString(),
|
||||
primaryModel: params.primaryModel,
|
||||
@@ -867,6 +940,40 @@ async function writeQaSuiteArtifacts(params: {
|
||||
scenarioResults: params.scenarios,
|
||||
})
|
||||
: undefined;
|
||||
if (crablineChannelDriverSelection && crablineChannelDriverSmoke) {
|
||||
await fs.writeFile(
|
||||
path.join(params.outputDir, crablineChannelDriverSelection.capabilityMatrixPath),
|
||||
`${JSON.stringify(
|
||||
{
|
||||
version: 1,
|
||||
source: "openclaw/crabline",
|
||||
channelDriver: crablineChannelDriverSelection.channelDriver,
|
||||
selectedChannel: crablineChannelDriverSelection.channel,
|
||||
manifestPath: crablineChannelDriverSmoke.manifestPath,
|
||||
report: crablineChannelDriverSmoke.capabilityReport,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
"utf8",
|
||||
);
|
||||
await fs.writeFile(
|
||||
path.join(params.outputDir, crablineChannelDriverSelection.smokeArtifactPath),
|
||||
`${JSON.stringify(
|
||||
{
|
||||
version: 1,
|
||||
source: "openclaw/crabline",
|
||||
channelDriver: crablineChannelDriverSelection.channelDriver,
|
||||
selectedChannel: crablineChannelDriverSelection.channel,
|
||||
manifestPath: crablineChannelDriverSmoke.manifestPath,
|
||||
smoke: crablineChannelDriverSmoke.smoke,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
"utf8",
|
||||
);
|
||||
}
|
||||
await fs.writeFile(reportPath, report, "utf8");
|
||||
if (evidence) {
|
||||
await fs.writeFile(evidencePath, `${JSON.stringify(evidence, null, 2)}\n`, "utf8");
|
||||
@@ -1055,7 +1162,7 @@ export async function runQaFlowSuite(params?: QaSuiteRunParams): Promise<QaSuite
|
||||
const concurrency = normalizeQaSuiteConcurrency(
|
||||
params?.concurrency,
|
||||
selectedScenarios.length,
|
||||
defaultQaSuiteConcurrencyForTransport(transportId),
|
||||
params?.channelDriverSelection ? 1 : defaultQaSuiteConcurrencyForTransport(transportId),
|
||||
);
|
||||
const progressEnabled = shouldLogQaSuiteProgress();
|
||||
const gatewayHeapCheckpointsEnabled = shouldCaptureGatewayHeapCheckpoints();
|
||||
@@ -1078,6 +1185,8 @@ export async function runQaFlowSuite(params?: QaSuiteRunParams): Promise<QaSuite
|
||||
startedAt,
|
||||
providerMode,
|
||||
transportId,
|
||||
channelDriverSelection: params?.channelDriverSelection,
|
||||
channelDriver: params?.channelDriver,
|
||||
primaryModel,
|
||||
alternateModel,
|
||||
fastMode,
|
||||
@@ -1105,9 +1214,11 @@ export async function runQaFlowSuite(params?: QaSuiteRunParams): Promise<QaSuite
|
||||
port: 0,
|
||||
embeddedGateway: "disabled",
|
||||
}));
|
||||
const transport = createQaTransportAdapter({
|
||||
id: transportId,
|
||||
const transport = await createQaSuiteTransportAdapter({
|
||||
channelDriverSelection: params?.channelDriverSelection,
|
||||
outputDir,
|
||||
state: lab.state,
|
||||
transportId,
|
||||
});
|
||||
const liveScenarioOutcomes: QaLabScenarioOutcome[] = selectedScenarios.map((scenario) => ({
|
||||
id: scenario.id,
|
||||
@@ -1151,6 +1262,8 @@ export async function runQaFlowSuite(params?: QaSuiteRunParams): Promise<QaSuite
|
||||
alternateModel,
|
||||
fastMode,
|
||||
concurrency,
|
||||
channelDriver: params?.channelDriver,
|
||||
channelDriverSelection: params?.channelDriverSelection,
|
||||
isolatedWorkers: true,
|
||||
scenarioIds:
|
||||
params?.scenarioIds && params.scenarioIds.length > 0
|
||||
@@ -1199,6 +1312,8 @@ export async function runQaFlowSuite(params?: QaSuiteRunParams): Promise<QaSuite
|
||||
outputDir: scenarioOutputDir,
|
||||
providerMode,
|
||||
transportId,
|
||||
channelDriver: params?.channelDriver,
|
||||
channelDriverSelection: params?.channelDriverSelection,
|
||||
primaryModel,
|
||||
alternateModel,
|
||||
fastMode,
|
||||
@@ -1296,6 +1411,8 @@ export async function runQaFlowSuite(params?: QaSuiteRunParams): Promise<QaSuite
|
||||
alternateModel,
|
||||
fastMode,
|
||||
concurrency,
|
||||
channelDriver: params?.channelDriver,
|
||||
channelDriverSelection: params?.channelDriverSelection,
|
||||
isolatedWorkers: true,
|
||||
// When the caller supplied an explicit non-empty --scenario filter,
|
||||
// record the executed (post-selectQaFlowSuiteScenarios-normalized) ids
|
||||
@@ -1327,6 +1444,7 @@ export async function runQaFlowSuite(params?: QaSuiteRunParams): Promise<QaSuite
|
||||
watchUrl: lab.baseUrl,
|
||||
} satisfies QaSuiteResult;
|
||||
} finally {
|
||||
await transport.cleanup?.();
|
||||
await disposeRegisteredAgentHarnesses();
|
||||
if (ownsLab) {
|
||||
await lab.stop();
|
||||
@@ -1347,9 +1465,11 @@ export async function runQaFlowSuite(params?: QaSuiteRunParams): Promise<QaSuite
|
||||
}));
|
||||
writeQaSuiteProgress(progressEnabled, `lab ready: ${sanitizeQaSuiteProgressValue(lab.baseUrl)}`);
|
||||
await waitForQaLabReadyOrStopOwned({ lab, ownsLab });
|
||||
const transport = createQaTransportAdapter({
|
||||
id: transportId,
|
||||
const transport = await createQaSuiteTransportAdapter({
|
||||
channelDriverSelection: params?.channelDriverSelection,
|
||||
outputDir,
|
||||
state: lab.state,
|
||||
transportId,
|
||||
});
|
||||
writeQaSuiteProgress(progressEnabled, `provider start: ${providerMode}`);
|
||||
const mock = await startQaProviderServer(providerMode);
|
||||
@@ -1561,6 +1681,8 @@ export async function runQaFlowSuite(params?: QaSuiteRunParams): Promise<QaSuite
|
||||
alternateModel,
|
||||
fastMode,
|
||||
concurrency,
|
||||
channelDriver: params?.channelDriver,
|
||||
channelDriverSelection: params?.channelDriverSelection,
|
||||
isolatedWorkers: false,
|
||||
// Same "filtered → executed list, unfiltered → null" convention as
|
||||
// the concurrent-path writeQaSuiteArtifacts call above.
|
||||
@@ -1600,6 +1722,7 @@ export async function runQaFlowSuite(params?: QaSuiteRunParams): Promise<QaSuite
|
||||
keepTemp,
|
||||
preserveToDir: keepTemp ? undefined : preserveGatewayRuntimeDir,
|
||||
});
|
||||
await transport.cleanup?.();
|
||||
await disposeRegisteredAgentHarnesses();
|
||||
await mock?.stop();
|
||||
if (ownsLab) {
|
||||
|
||||
Generated
+16
@@ -1357,6 +1357,9 @@ importers:
|
||||
'@openclaw/whatsapp':
|
||||
specifier: workspace:*
|
||||
version: link:../whatsapp
|
||||
crabline:
|
||||
specifier: github:openclaw/crabline#b3513f66053788c6a7bd2bc76fbfc7201f647d29
|
||||
version: https://codeload.github.com/openclaw/crabline/tar.gz/b3513f66053788c6a7bd2bc76fbfc7201f647d29
|
||||
openclaw:
|
||||
specifier: 2026.5.28
|
||||
version: 2026.5.28
|
||||
@@ -4989,6 +4992,12 @@ packages:
|
||||
resolution: {integrity: sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw==}
|
||||
engines: {node: '>= 0.10'}
|
||||
|
||||
crabline@https://codeload.github.com/openclaw/crabline/tar.gz/b3513f66053788c6a7bd2bc76fbfc7201f647d29:
|
||||
resolution: {gitHosted: true, tarball: https://codeload.github.com/openclaw/crabline/tar.gz/b3513f66053788c6a7bd2bc76fbfc7201f647d29}
|
||||
version: 0.1.0
|
||||
engines: {node: '>=22'}
|
||||
hasBin: true
|
||||
|
||||
croner@10.0.1:
|
||||
resolution: {integrity: sha512-ixNtAJndqh173VQ4KodSdJEI6nuioBWI0V1ITNKhZZsO0pEMoDxz539T4FTTbSZ/xIOSuDnzxLVRqBVSvPNE2g==}
|
||||
engines: {node: '>=18.0'}
|
||||
@@ -10979,6 +10988,13 @@ snapshots:
|
||||
object-assign: 4.1.1
|
||||
vary: 1.1.2
|
||||
|
||||
crabline@https://codeload.github.com/openclaw/crabline/tar.gz/b3513f66053788c6a7bd2bc76fbfc7201f647d29:
|
||||
dependencies:
|
||||
commander: 14.0.3
|
||||
picocolors: 1.1.1
|
||||
yaml: 2.9.0
|
||||
zod: 4.4.3
|
||||
|
||||
croner@10.0.1: {}
|
||||
|
||||
cross-spawn@7.0.6:
|
||||
|
||||
@@ -140,6 +140,7 @@ allowBuilds:
|
||||
openclaw: true
|
||||
"@openclaw/proxyline": true
|
||||
clawpdf: true
|
||||
crabline: true
|
||||
rastermill: true
|
||||
|
||||
packageExtensions:
|
||||
|
||||
@@ -208,7 +208,7 @@ flow:
|
||||
message:
|
||||
expr: "`expected medium OpenAI reasoning payload, got ${JSON.stringify(maxRequest?.body?.reasoning ?? null)}`"
|
||||
- if:
|
||||
expr: "env.transport.id !== 'qa-channel'"
|
||||
expr: "env.transport.id !== 'qa-channel' && env.transport.id !== 'crabline'"
|
||||
then:
|
||||
- call: waitForCondition
|
||||
saveAs: maxReasoning
|
||||
@@ -220,4 +220,4 @@ flow:
|
||||
expr: "maxReasoning.text.trimStart().startsWith('Reasoning:')"
|
||||
message:
|
||||
expr: "`missing max reasoning message near answer: ${recentOutboundSummary(state, 6)}`"
|
||||
detailsExpr: "env.transport.id === 'qa-channel' ? `answer=${maxAnswer.text}; medium reasoning=${env.mock ? String(maxRequest?.body?.reasoning?.effort ?? '') : 'live'}; qa-channel suppresses reasoning delivery` : `answer=${maxAnswer.text}; reasoning=${maxReasoning.text}`"
|
||||
detailsExpr: "env.transport.id === 'qa-channel' || env.transport.id === 'crabline' ? `answer=${maxAnswer.text}; medium reasoning=${env.mock ? String(maxRequest?.body?.reasoning?.effort ?? '') : 'live'}; ${env.transport.id} suppresses reasoning delivery` : `answer=${maxAnswer.text}; reasoning=${maxReasoning.text}`"
|
||||
|
||||
@@ -25,6 +25,7 @@ const OPTIONAL_UNDECLARED_RUNTIME_IMPORTS = new Map<string, Set<string>>([
|
||||
// @discordjs/voice still probes the native addon in its dependency report path.
|
||||
new Set(["@discordjs/opus"]),
|
||||
],
|
||||
["extensions/qa-lab", new Set(["crabline"])],
|
||||
]);
|
||||
const INDIRECT_RUNTIME_DEPENDENCIES = new Map<string, Set<string>>([
|
||||
[
|
||||
|
||||
+4
-2
@@ -7,9 +7,10 @@ snapshot:
|
||||
source_ref: origin/main@41eef4a7965
|
||||
profiles:
|
||||
- id: smoke-ci
|
||||
description: Deterministic PR and merge proof with mock model providers, synthetic qa-channel,
|
||||
and local SDK-backed channel upstreams; no live external service required.
|
||||
description: Deterministic PR and merge proof with mock model providers and the
|
||||
Crabline-backed channel driver; no live external service required.
|
||||
evidenceMode: slim
|
||||
channelDriver: crabline
|
||||
categoryIds:
|
||||
- agent-runtime-and-provider-execution.agent-turn-execution
|
||||
- agent-runtime-and-provider-execution.model-and-runtime-selection
|
||||
@@ -49,6 +50,7 @@ profiles:
|
||||
description: Stable/LTS proof selector for live providers, live channels, package artifacts,
|
||||
upgrade paths, and platform proof where the claim depends on real upstreams or release
|
||||
artifacts.
|
||||
channelDriver: live
|
||||
categoryIds:
|
||||
- gateway-runtime.approvals-and-remote-execution
|
||||
- gateway-runtime.http-apis
|
||||
|
||||
Reference in New Issue
Block a user