diff --git a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md index d58b8690..db0b8ab2 100644 --- a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md +++ b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md @@ -11,7 +11,22 @@ 最新增量证据(2026-08-19): -- D-366/ADR-0459(已接受;产品视觉入口待完成):在既有 Run management plane 增加 `run.cancellation.summary`,由强认证 User 以 +- D-367/ADR-0460(已接受;有界 blocked drill-down 与 Console 可选接入待完成):在现有 `ql3 run`/`ql3-run-client` 增加一次性 + `status --config=... --assertion=... --project=... [--format=text|json]` 产品入口。它在内存生成固定 `run.cancellation.summary` 命令,仍经同一 + exact codec、TLS 1.3、Run 专用 mTLS/OIDC、固定 management route 和响应交叉不变量校验;原 `--command` 私有文件模式保持兼容。默认 text 是无 ANSI + 的确定性 Project 状态卡,JSON 使用 `qinglong/run-cancellation-status@v1`;两者只含 D-366 低敏计数与结论。告警映射固定为 + `clear→ok/0`、`converging→warning/10`、`attention_required→critical/20`,查询失败仍为 1、用法错误仍为 64,使外部 supervisor 无需解析自然语言。 + 命令一次请求后退出,不轮询、不重试、不缓存、不保持 socket,也不改变 Cluster readiness。实现只在已有 `cluster-admin/run-management` 与 + `management-support` 内增加真实职责,不新增 package、binary、依赖、服务、端口、timer、queue、cache、数据库权限、migration 或 Kubernetes 对象; + Edge/Standalone 闭包不变化。未知 blocked Run 的发现仍须后续独立的有界 cursor 契约,不能把 status 扩成无界列表或把第二组管理 authority 默认塞进 + Copilot Console。聚焦门 `12/12`、CLI/product 真实进程门 `15/15`;Cluster Admin 全量为 `407 total / 404 pass / 3 conditional skip / 0 fail`, + backend 为 `1,489 total / 1,487 pass / 2 conditional skip / 0 fail`,18-package clean build/test 退出 0。四项架构审计与 `14/14` Local artifact audit + 全部 compatible;workspace 保持 18 packages、无 single/shallow package,Cluster Admin 为 123 个 source、122 个 nested source、仅 1 个受审 binary root + entry。基础 Edge/Standalone 保持 `2,589,998 / 2,590,076` bytes,Application+AI 保持 `4,493,151 / 4,493,283` bytes,MCP 保持 + `7,315,930 / 7,316,038` bytes。PostgreSQL 18.6 arm64 HA `145/145`、timeline `1→2`,报告 SHA-256 为 + `59a568d0511cde671946ebf6df09f88868a3d591c5021c90bc27d4715411091e`,独立 evidence audit compatible 且零 finding。 + +- D-366/ADR-0459(已接受;一次性产品 CLI 由 D-367 完成,Console 可选接入待完成):在既有 Run management plane 增加 `run.cancellation.summary`,由强认证 User 以 `run.read` 按需读取 Project 级 PostgreSQL 快照。响应只有五态 dispatch 计数、due/expired-lease 信号、四种 blocking-result 计数、最早 blocked 时间和 `clear|converging|attention_required`/`none|wait|inspect` 固定结论,不返回 Run/Attempt/Worker identity 或 lease capability。blocked 触发 `attention_required`,但不错误撤回整个 Cluster readiness;due/expired 只作为 caller-driven 收敛信号。查询与 allowed audit 位于同一 5 秒 @@ -22,7 +37,7 @@ `145/145` 执行 blocked summary→inspect→rearm→production delivery→WAL→promotion,timeline `1→2`,报告 SHA-256 为 `d763157b3a781e305add3c6f0c5080820b1d65b5feefe60be6fa7006c0050107`。 -- D-365/ADR-0458(已接受;聚合出口由 D-366 完成,产品视觉入口待完成):在既有隔离 `cluster-admin` Run management plane 上增加 +- D-365/ADR-0458(已接受;聚合出口由 D-366、一次性产品 CLI 由 D-367 完成):在既有隔离 `cluster-admin` Run management plane 上增加 `run.cancellation.inspect` 与 `run.cancellation.rearm`,不新建 package、服务、端口、timer、连接池或 Kubernetes 对象。inspect 要求强认证 User 与 `run.read`,viewer 可读取 Run/dispatch 的固定低敏投影,但永不返回 lease owner、raw token 或 token digest;rearm 要求 `run.stop`,只接受 `blocked + expected dispatch version + expected blocking result` 的精确 CAS,retry delay 固定为 1 秒至 24 小时并由 PostgreSQL diff --git a/docs/adr/ADR-0005-durable-cancellation-dispatch.md b/docs/adr/ADR-0005-durable-cancellation-dispatch.md index 678d0898..3546836f 100644 --- a/docs/adr/ADR-0005-durable-cancellation-dispatch.md +++ b/docs/adr/ADR-0005-durable-cancellation-dispatch.md @@ -155,9 +155,11 @@ PID 可复用,可能终止无关进程,禁止。 ADR-0459 进一步增加 Project-scoped、caller-driven summary,以固定计数和三态 assessment 提供 blocked/availability 告警出口,不改变全局 readiness,也不新增采集器。 +ADR-0460 已在现有 `ql3 run` 增加 one-shot status 产品入口:同一强认证 summary 被投影为低敏 text/JSON 状态卡,并以 `0/10/20` 区分 clear、converging 与 attention-required;没有新增轮询器、临时 command 文件或常驻 authority。 + HTTP worker 已通过默认关闭的 manual-only manifest bootstrap 接入 Local Supervisor:只有 accepted 且全部 gate 通过时才启动,失败或 shutdown 时有界停止。以下工作仍未完成,因此它仍只允许显式 canary,不得扩大到默认生产流量: -- 产品控制台中的 Project 状态卡、告警路由和有界 blocked drill-down;Project 聚合出口与私有 operator 处置协议已经实现。 +- Copilot Console 的可选 Project 状态卡与有界 blocked drill-down;Project 聚合出口、一次性产品状态卡、告警退出码和私有 operator 处置协议已经实现。 - 固定 edge 设备的数据库写放大、RSS、时延和磁盘基准。 - 固定 Project allowlist 的外部时序指标适配;数据库事实驱动的按需 availability/blocked 汇总已经实现。 - 首次真实目标实例完整激活/回滚仪式与共享 config 多写者 authority。 @@ -180,3 +182,4 @@ HTTP worker 已通过默认关闭的 manual-only manifest bootstrap 接入 Local 14. 强认证 viewer 只能读取无 lease capability 的低敏诊断;只有 `run.stop` authority 可按 blocked/version/result 精确 rearm。 15. rearm、RunEvent 与 allowed audit 原子提交,数据库时间决定 retry due;stale fence、授权漂移与 mutation drift 均失败关闭。 16. Project summary 的五态与 blocking-result 计数必须交叉守恒,blocked 只产生 `attention_required` 告警而不撤回全局 readiness,且响应不包含 Run/Attempt/lease identity。 +17. `ql3 run status` 必须只发送一次固定 summary,text/JSON 事实一致,`0/10/20` 与三态严格映射;原 command-file 模式和 Edge/Standalone 闭包不得变化。 diff --git a/docs/adr/ADR-0458-least-privilege-cancellation-diagnostics-and-rearm.md b/docs/adr/ADR-0458-least-privilege-cancellation-diagnostics-and-rearm.md index ce144d97..d0b9ccb0 100644 --- a/docs/adr/ADR-0458-least-privilege-cancellation-diagnostics-and-rearm.md +++ b/docs/adr/ADR-0458-least-privilege-cancellation-diagnostics-and-rearm.md @@ -62,4 +62,4 @@ QingLong 同时面向低配路由器与集群。该能力只属于 Cluster opera ## 后续 -ADR-0459 已用既有管理面完成数据库事实驱动的 Project 级 blocked/availability 汇总和固定告警结论;产品视觉入口、告警路由和有界 drill-down 仍待实现。CloudNativePG live failover、多副本容量压力、固定 Linux x64/arm64 与物理 Edge 资源证据继续作为发布最终化门,不由单机 Docker HA 结果替代。 +ADR-0459 已用既有管理面完成数据库事实驱动的 Project 级 blocked/availability 汇总;ADR-0460 已增加一次性产品状态卡、稳定 JSON 与告警退出码。有界 blocked drill-down 和 Console 可选接入仍待实现。CloudNativePG live failover、多副本容量压力、固定 Linux x64/arm64 与物理 Edge 资源证据继续作为发布最终化门,不由单机 Docker HA 结果替代。 diff --git a/docs/adr/ADR-0459-project-scoped-cancellation-availability-summary.md b/docs/adr/ADR-0459-project-scoped-cancellation-availability-summary.md index 9a2fb772..80c9c025 100644 --- a/docs/adr/ADR-0459-project-scoped-cancellation-availability-summary.md +++ b/docs/adr/ADR-0459-project-scoped-cancellation-availability-summary.md @@ -58,4 +58,4 @@ QingLong 同时面对小型路由设备和多副本 Cluster。Local/Edge 不应 ## 后续 -产品控制台可把该 summary 渲染为 Project 级状态卡和告警入口,并通过单 Run inspect/rearm 完成 drill-down。CloudNativePG live failover、多副本容量压力、固定 Linux x64/arm64 和物理 Edge 资源证据仍是独立发布门;本 ADR 不把 Docker HA 或按需汇总冒充这些现场证据。 +ADR-0460 已用现有 `ql3 run status` 完成一次性 Project 状态卡、稳定 JSON 和 `0/10/20` 告警退出码,不增加常驻组件或混合 Copilot Console authority。未知 blocked Run 的有界发现与 Console 可选接入仍需独立设计。CloudNativePG live failover、多副本容量压力、固定 Linux x64/arm64 和物理 Edge 资源证据仍是独立发布门;本 ADR 不把 Docker HA 或按需汇总冒充这些现场证据。 diff --git a/docs/adr/ADR-0460-one-shot-cancellation-status-product-entry.md b/docs/adr/ADR-0460-one-shot-cancellation-status-product-entry.md new file mode 100644 index 00000000..ac058d82 --- /dev/null +++ b/docs/adr/ADR-0460-one-shot-cancellation-status-product-entry.md @@ -0,0 +1,62 @@ +# ADR-0460:一次性 Cancellation 可用性状态卡与告警退出码 + +- 状态:Accepted +- 日期:2026-08-19 +- 关联 RFC:QL-RFC-0001 D-367、PR-5、PR-7 +- 关联 ADR:ADR-0005、ADR-0458、ADR-0459 +- Amends:ADR-0459 的产品入口与告警路由边界 + +## 上下文 + +ADR-0459 已让 Run management plane 返回数据库事实驱动的 Project 级 cancellation summary,但现有 `ql3 run` 仍要求 operator 手工准备完整 command JSON,再自行解释三态 assessment。该协议可以被脚本调用,却还不是可直接使用的产品入口,也没有稳定的告警退出语义。 + +QingLong 的部署跨度很大。Cluster operator 需要可读状态卡和可供 systemd、CronJob、CI 或外部告警器路由的机器结果;路由器级 Edge/Standalone 则不能为 Cluster 可观测性引入 daemon、轮询、PostgreSQL 客户端或新的制品闭包。Copilot Console 又只有普通 Project API Bearer authority,直接把 Run management mTLS/OIDC 凭据并入该常驻本机 BFF 会混合两种权限边界。 + +## 决策 + +1. 在现有 `ql3 run`/`ql3-run-client` 增加一次性 `status` 模式:`ql3 run status --config=... --assertion=... --project=... [--format=text|json]`。它继续由统一产品 CLI 无 shell 委派同一 binary,不新增 workspace package、binary、服务或端口。 +2. `status` 只在一次调用内生成 request/audit UUID,并构造固定 `run.cancellation.summary` 命令。Project、配置文件和短生命周期 assertion 仍显式提供;operator context 只可补充稳定的 Run management config,不保存 assertion。调用方不能提供 Run/Attempt identity、状态过滤、时间窗口、计数或服务端时间。 +3. 通用认证管理客户端允许包内调用方传入内存命令,但仍执行同一 exact-shape normalizer、TLS 1.3、Run 专用 mTLS、OIDC Bearer、固定 management path、128 KiB 响应上限、单连接和响应校验。既有 `--command` 私有文件模式保持字节读取、权限检查、错误语义和输出格式不变。 +4. 默认 `text` 输出是无 ANSI 控制字符、确定性、适合终端与日志的状态卡;`json` 输出使用固定 `qinglong/run-cancellation-status@v1` schema。两者只包含 request/Project、数据库观察时间、assessment/action、固定 dispatch/signal/blocking 计数和可选最早 blocked 时间,不增加 Run/Attempt/Worker、lease、PID、命令、环境、Secret、日志或错误原文。 +5. 产品告警映射固定为 `clear → ok/0`、`converging → warning/10`、`attention_required → critical/20`。配置/网络/协议失败仍为 `1`,CLI 用法错误仍为 `64`;因此外部 supervisor 可区分“等待 caller-driven 收敛”“需要 inspect”与“查询本身失败”,而无需解析自然语言。 +6. 命令严格 one-shot:一次 summary、一次 allowed/denied audit,然后退出;不重试、不轮询、不缓存、不保持 socket,也不改变 `/readyz`。blocked 仍不撤回整个 Cluster readiness。 +7. 该入口只进入 Cluster Admin 制品。Edge/Standalone 依赖图和制品不得包含 `cluster-admin`、`cluster-postgres`、`pg`、状态 CLI 或 TLS/OIDC 管理凭据。 +8. D-367 不返回 blocked Run 列表。用户从已知 Run 继续使用既有 inspect/rearm;从聚合发现未知 blocked Run 的能力必须以后用独立、有界、稳定 cursor、低敏 identity 和索引证明的 drill-down 契约完成,不能让 status 偷偷退化为无界扫描。 + +## 被拒绝的替代方案 + +### 在 Copilot Console 直接复用普通 Project API 凭据 + +拒绝。summary 位于独立 Run management authority,普通 Bearer 不应绕过 mTLS、OIDC purpose 和专用 Policy/audit。把第二组高权限凭据强制塞进 Console 也会扩大常驻 BFF 的秘密与故障面;Console 接入应在未来以显式可选 authority 单独设计。 + +### 新建 exporter、告警 daemon 或后台轮询器 + +拒绝。它会增加常驻进程、连接和 cadence,并让小规模 Cluster 为无人查看的状态持续付费。一次性命令可由现有部署侧调度器按自身策略调用。 + +### 客户端从列表或本地缓存推导 assessment + +拒绝。列表可能分页且跨页漂移,本地缓存也会在 failover 后失真。状态卡只投影经过服务器和客户端双重交叉不变量校验的单事务 summary。 + +### 所有非 clear 状态返回同一个退出码 + +拒绝。converging 需要等待,attention_required 需要 inspect;合并后外部告警器只能解析 JSON 或把正常收敛当成阻塞故障。 + +## 资源、安全与部署影响 + +- Cluster 每次调用只建立一个短生命周期 mTLS 连接,使用既有 Run manager pool 完成一个最长 5 秒的 SERIALIZABLE summary/audit 事务;客户端不创建 Agent keep-alive、缓存、timer 或临时 command 文件。 +- 文本卡和 JSON 使用同一已校验 projection;Project/时间/计数均有既有协议上限,不存在动态指标 label 或任意服务端路径。 +- 旧命令文件模式继续支持 retry、stop、summary、inspect 和 rearm,升级不要求重写现有自动化。 +- 新源码保留在已有 `cluster-admin/run-management` 内;通用内存命令入口位于已有 `management-support`,没有单文件 package 或 `src` 根平铺。 + +## 验证 + +- `runCancellationStatus` 与 management-client 聚焦门 `12/12`,CLI/product 真实进程门 `15/15`;覆盖内存 summary 命令、三态 severity/退出码、低敏文本卡、错误 operation 拒绝和原 command-file/mTLS route 兼容。 +- 真实本机 TLS 1.3/mTLS CLI 集成测试验证只向 `/api/v3/runs/management` POST 一次 `run.cancellation.summary`,不带 Run ID,并以 `attention_required` 返回 JSON 与退出码 20;help 与非法 Project 在 I/O 前失败关闭。 +- Cluster Admin 全量 `407 total / 404 pass / 3 conditional skip / 0 fail`;backend 全量 `1,489 total / 1,487 pass / 2 conditional skip / 0 fail`;18-package clean build/test 退出 0。 +- package boundary、Cluster dependency、Edge import、Cluster deployment 四项审计均 compatible;workspace 保持 18 packages、无 single/shallow package,Cluster Admin 为 123 个 source、122 个 nested source、仅 1 个受审 binary root entry。 +- `14/14` Local artifact audit 均 compatible;基础 Edge/Standalone 为 `2,589,998 / 2,590,076` bytes,Application+AI 为 `4,493,151 / 4,493,283` bytes,MCP 为 `7,315,930 / 7,316,038` bytes,证明 Cluster-only 入口未进入低配设备闭包。 +- PostgreSQL 18.6 arm64 HA `145/145`,timeline `1→2`,报告 SHA-256 `59a568d0511cde671946ebf6df09f88868a3d591c5021c90bc27d4715411091e`;独立 evidence audit 为 `compatible=true`、零 finding。 + +## 后续 + +D-368 可设计 Project-scoped blocked drill-down:固定小页、稳定数据库 cursor、只返回继续 inspect 所需的最低 identity,并证明索引、事务一致性、Policy/audit 和多副本 HA。Copilot Console 状态卡应复用本 ADR 的 projection 和显式 Run management authority,但不得默认持有该 authority 或建立轮询。 diff --git a/packages/ql3-cluster-admin/src/management-support/pluginPackageManagementClient.ts b/packages/ql3-cluster-admin/src/management-support/pluginPackageManagementClient.ts index c122f7fc..b54cc3b9 100644 --- a/packages/ql3-cluster-admin/src/management-support/pluginPackageManagementClient.ts +++ b/packages/ql3-cluster-admin/src/management-support/pluginPackageManagementClient.ts @@ -49,6 +49,12 @@ export interface ClusterPluginPackageManagementClientPaths { readonly assertionFile: string; } +export interface ClusterAuthenticatedManagementCommandExecution { + readonly configFile: string; + readonly assertionFile: string; + readonly command: Command; +} + export interface ClusterPluginPackageManagementClientResult { readonly schemaVersion: 1; readonly requestId: string; @@ -1119,11 +1125,23 @@ export async function executeClusterAuthenticatedManagementClient< Command, Result, >( - paths: ClusterPluginPackageManagementClientPaths, + execution: + | ClusterPluginPackageManagementClientPaths + | ClusterAuthenticatedManagementCommandExecution, protocol: ClusterAuthenticatedManagementClientProtocol, connectionOptions?: ClusterPluginPackageManagementClientConnectionOptions, ): Promise>> { - exactObject(paths, ['configFile', 'commandFile', 'assertionFile']); + const inlineCommand = + execution !== null && + typeof execution === 'object' && + !Array.isArray(execution) && + Object.hasOwn(execution, 'command'); + exactObject( + execution, + inlineCommand + ? ['configFile', 'command', 'assertionFile'] + : ['configFile', 'commandFile', 'assertionFile'], + ); if ( !protocol || typeof protocol !== 'object' || @@ -1160,17 +1178,19 @@ export async function executeClusterAuthenticatedManagementClient< | undefined; try { prepared = prepareClusterAuthenticatedManagementClientConfiguration( - paths.configFile, + execution.configFile, protocol.managementPath, protocol.clientCertificate, ); - commandBytes = readCanonicalFile( - paths.commandFile, - MAX_COMMAND_BYTES, - 'private', - ); + if (!inlineCommand) { + commandBytes = readCanonicalFile( + (execution as ClusterPluginPackageManagementClientPaths).commandFile, + MAX_COMMAND_BYTES, + 'private', + ); + } assertionBytes = readCanonicalFile( - paths.assertionFile, + execution.assertionFile, MAX_ASSERTION_BYTES, 'private', ); @@ -1183,7 +1203,12 @@ export async function executeClusterAuthenticatedManagementClient< clientCertificateBytes, clientPrivateKeyBytes, } = prepared; - const command = protocol.normalizeCommand(parseJson(commandBytes)); + const command = protocol.normalizeCommand( + inlineCommand + ? (execution as ClusterAuthenticatedManagementCommandExecution) + .command + : parseJson(commandBytes!), + ); const assertion = assertionBytes.toString('ascii'); if ( assertionBytes.some((byte) => byte > 0x7f) || diff --git a/packages/ql3-cluster-admin/src/product-cli/productCommand.ts b/packages/ql3-cluster-admin/src/product-cli/productCommand.ts index 6f96f4f2..4e0573de 100644 --- a/packages/ql3-cluster-admin/src/product-cli/productCommand.ts +++ b/packages/ql3-cluster-admin/src/product-cli/productCommand.ts @@ -87,7 +87,7 @@ export const QINGLONG3_CLUSTER_PRODUCT_COMMANDS: readonly QingLong3ClusterProduc name: 'run', binary: 'ql3-run-client', target: 'run-management/runManagementClientCli.js', - description: 'retry or stop Runs under strong authentication', + description: 'observe, retry or stop Runs under strong authentication', }), Object.freeze({ name: 'automation', diff --git a/packages/ql3-cluster-admin/src/run-management/runCancellationStatus.ts b/packages/ql3-cluster-admin/src/run-management/runCancellationStatus.ts new file mode 100644 index 00000000..fd830e87 --- /dev/null +++ b/packages/ql3-cluster-admin/src/run-management/runCancellationStatus.ts @@ -0,0 +1,127 @@ +import { randomUUID } from 'node:crypto'; + +import type { ClusterRunManagementClientResult } from './runManagementClient'; +import { + RUN_CANCELLATION_DISPATCH_SUMMARY_REQUEST_SCHEMA, + normalizeClusterRunManagementCommand, + type ClusterRunManagementCancellationSummaryCommand, + type ClusterRunManagementCancellationSummaryTransportResult, +} from './runManagementTransport'; + +export const RUN_CANCELLATION_STATUS_SCHEMA = + 'qinglong/run-cancellation-status@v1' as const; + +export type RunCancellationStatusExitCode = 0 | 10 | 20; +export type RunCancellationStatusSeverity = 'ok' | 'warning' | 'critical'; + +type CancellationSummary = + ClusterRunManagementCancellationSummaryTransportResult['summary']; + +export interface RunCancellationStatusObservation { + readonly schemaVersion: 1; + readonly schema: typeof RUN_CANCELLATION_STATUS_SCHEMA; + readonly component: 'qinglong3-run-management-client'; + readonly event: 'cancellation_status_observed'; + readonly requestId: string; + readonly projectId: string; + readonly observedAtMs: number; + readonly assessment: CancellationSummary['assessment']; + readonly operatorAction: CancellationSummary['operatorAction']; + readonly severity: RunCancellationStatusSeverity; + readonly exitCode: RunCancellationStatusExitCode; + readonly dispatches: CancellationSummary['dispatches']; + readonly signals: CancellationSummary['signals']; + readonly blockingResults: CancellationSummary['blockingResults']; + readonly oldestBlockedAtMs?: number; +} + +export function createRunCancellationStatusCommand( + projectId: string, + createUuid: () => string = randomUUID, +): Readonly { + const requestId = createUuid(); + const auditEventId = createUuid(); + let failureAuditEventId = createUuid(); + for ( + let attempts = 0; + failureAuditEventId === auditEventId && attempts < 3; + attempts += 1 + ) { + failureAuditEventId = createUuid(); + } + return normalizeClusterRunManagementCommand({ + schemaVersion: 1, + operation: 'run.cancellation.summary', + request: { + projectId, + requestId, + auditEventId, + failureAuditEventId, + body: { schema: RUN_CANCELLATION_DISPATCH_SUMMARY_REQUEST_SCHEMA }, + }, + }) as Readonly; +} + +export function projectRunCancellationStatus( + result: Readonly, +): Readonly { + if (result.result.operation !== 'run.cancellation.summary') { + throw new TypeError('Run cancellation status requires a summary result'); + } + const summary = result.result.summary; + const severity: RunCancellationStatusSeverity = + summary.assessment === 'clear' + ? 'ok' + : summary.assessment === 'converging' + ? 'warning' + : 'critical'; + const exitCode: RunCancellationStatusExitCode = + severity === 'ok' ? 0 : severity === 'warning' ? 10 : 20; + return Object.freeze({ + schemaVersion: 1, + schema: RUN_CANCELLATION_STATUS_SCHEMA, + component: 'qinglong3-run-management-client', + event: 'cancellation_status_observed', + requestId: result.requestId, + projectId: summary.projectId, + observedAtMs: summary.observedAtMs, + assessment: summary.assessment, + operatorAction: summary.operatorAction, + severity, + exitCode, + dispatches: summary.dispatches, + signals: summary.signals, + blockingResults: summary.blockingResults, + ...(summary.oldestBlockedAtMs === undefined + ? {} + : { oldestBlockedAtMs: summary.oldestBlockedAtMs }), + }); +} + +function label(value: string): string { + return value.replaceAll('_', ' ').toUpperCase(); +} + +export function formatRunCancellationStatusCard( + status: Readonly, +): string { + const dispatch = status.dispatches; + const blocking = status.blockingResults; + return [ + 'QingLong 3.0 / Cancellation Availability', + `PROJECT ${status.projectId}`, + `ASSESSMENT ${label(status.assessment)}`, + `ACTION ${label(status.operatorAction)}`, + `ALERT ${status.severity.toUpperCase()} (exit ${status.exitCode})`, + `OBSERVED ${new Date(status.observedAtMs).toISOString()}`, + `DISPATCHES total=${dispatch.total} pending=${dispatch.pending} leased=${dispatch.leased} retry_wait=${dispatch.retryWait} dispatched=${dispatch.dispatched} blocked=${dispatch.blocked}`, + `SIGNALS due=${status.signals.due} expired_lease=${status.signals.expiredLease}`, + `BLOCKING identity_mismatch=${blocking.identityMismatch} pid_mismatch=${blocking.pidMismatch} unsupported=${blocking.unsupported} invalid=${blocking.invalid}`, + `OLDEST_BLOCK ${ + status.oldestBlockedAtMs === undefined + ? '-' + : new Date(status.oldestBlockedAtMs).toISOString() + }`, + `REQUEST ${status.requestId}`, + ].join('\n'); +} diff --git a/packages/ql3-cluster-admin/src/run-management/runManagementClient.ts b/packages/ql3-cluster-admin/src/run-management/runManagementClient.ts index 6870b4f6..2db2490b 100644 --- a/packages/ql3-cluster-admin/src/run-management/runManagementClient.ts +++ b/packages/ql3-cluster-admin/src/run-management/runManagementClient.ts @@ -14,6 +14,7 @@ import { import { ClusterPluginPackageManagementClientRequestError, executeClusterAuthenticatedManagementClient, + type ClusterAuthenticatedManagementCommandExecution, type ClusterAuthenticatedManagementClientResult, type ClusterPluginPackageManagementClientConnectionOptions, type ClusterPluginPackageManagementClientPaths, @@ -35,6 +36,8 @@ export type ClusterRunManagementClientConnectionOptions = ClusterPluginPackageManagementClientConnectionOptions; export type ClusterRunManagementClientResult = ClusterAuthenticatedManagementClientResult; +export type ClusterRunManagementCommandExecution = + ClusterAuthenticatedManagementCommandExecution; function invalid(): never { throw new ClusterPluginPackageManagementClientRequestError(); @@ -427,3 +430,14 @@ export function executeClusterRunManagementClient( connectionOptions, ); } + +export function executeClusterRunManagementCommand( + execution: ClusterRunManagementCommandExecution, + connectionOptions?: ClusterRunManagementClientConnectionOptions, +): Promise> { + return executeClusterAuthenticatedManagementClient( + execution, + PROTOCOL, + connectionOptions, + ); +} diff --git a/packages/ql3-cluster-admin/src/run-management/runManagementClientCli.ts b/packages/ql3-cluster-admin/src/run-management/runManagementClientCli.ts index c12a7525..ae507547 100644 --- a/packages/ql3-cluster-admin/src/run-management/runManagementClientCli.ts +++ b/packages/ql3-cluster-admin/src/run-management/runManagementClientCli.ts @@ -1,16 +1,73 @@ #!/usr/bin/env node import { ClusterPluginPackageManagementClientRemoteError } from '../management-support/pluginPackageManagementClient'; -import { executeClusterRunManagementClient } from './runManagementClient'; +import { + executeClusterRunManagementClient, + executeClusterRunManagementCommand, +} from './runManagementClient'; +import { + createRunCancellationStatusCommand, + formatRunCancellationStatusCard, + projectRunCancellationStatus, +} from './runCancellationStatus'; -const USAGE = - 'Usage: ql3-run-client --config=/absolute/client.json --command=/absolute/command.json --assertion=/absolute/assertion.jwt'; +const USAGE = [ + 'Usage: ql3-run-client --config=/absolute/client.json --command=/absolute/command.json --assertion=/absolute/assertion.jwt', + ' ql3-run-client status --config=/absolute/client.json --assertion=/absolute/assertion.jwt --project=PROJECT [--format=text|json]', + '', + 'Status exit codes: 0=clear, 10=converging, 20=attention_required.', +].join('\n'); +const PROJECT_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/; -function argumentsFrom(argv: readonly string[]): Readonly<{ - configFile: string; - commandFile: string; - assertionFile: string; -}> | null { +type RunManagementClientArguments = + | Readonly<{ + kind: 'command'; + configFile: string; + commandFile: string; + assertionFile: string; + }> + | Readonly<{ + kind: 'status'; + configFile: string; + assertionFile: string; + projectId: string; + format: 'text' | 'json'; + }>; + +function argumentsFrom( + argv: readonly string[], +): Readonly | null { + const statusCount = argv.filter((argument) => argument === 'status').length; + if (statusCount > 0) { + if (statusCount !== 1 || argv.length < 4 || argv.length > 5) return null; + const values = new Map(); + for (const argument of argv) { + if (argument === 'status') continue; + const match = /^--(config|assertion|project|format)=(.+)$/.exec(argument); + if (!match || values.has(match[1]!)) return null; + values.set(match[1]!, match[2]!); + } + if ( + !values.has('config') || + !values.get('config')!.startsWith('/') || + !values.has('assertion') || + !values.get('assertion')!.startsWith('/') || + !values.has('project') || + !PROJECT_ID.test(values.get('project')!) || + (values.has('format') && + values.get('format') !== 'text' && + values.get('format') !== 'json') + ) { + return null; + } + return Object.freeze({ + kind: 'status', + configFile: values.get('config')!, + assertionFile: values.get('assertion')!, + projectId: values.get('project')!, + format: (values.get('format') ?? 'text') as 'text' | 'json', + }); + } if (argv.length !== 3) return null; const values = new Map(); for (const argument of argv) { @@ -18,8 +75,14 @@ function argumentsFrom(argv: readonly string[]): Readonly<{ if (!match || values.has(match[1]!)) return null; values.set(match[1]!, match[2]!); } - if (!values.has('config') || !values.has('command') || !values.has('assertion')) return null; + if ( + !values.has('config') || + !values.has('command') || + !values.has('assertion') + ) + return null; return Object.freeze({ + kind: 'command', configFile: values.get('config')!, commandFile: values.get('command')!, assertionFile: values.get('assertion')!, @@ -32,13 +95,18 @@ function failureFact(error: unknown): Readonly> { schemaVersion: 1, component: 'qinglong3-run-management-client', event: 'command_failed', - code: typeof candidate?.code === 'string' ? candidate.code : 'QL3_RUN_MANAGEMENT_CLIENT_FAILED', + code: + typeof candidate?.code === 'string' + ? candidate.code + : 'QL3_RUN_MANAGEMENT_CLIENT_FAILED', ...(error instanceof ClusterPluginPackageManagementClientRemoteError ? { statusCode: error.statusCode, responseCode: error.responseCode, requestId: error.requestId, - ...(error.retryAfterSeconds === null ? {} : { retryAfterSeconds: error.retryAfterSeconds }), + ...(error.retryAfterSeconds === null + ? {} + : { retryAfterSeconds: error.retryAfterSeconds }), } : {}), }); @@ -51,13 +119,43 @@ async function run(argv: readonly string[]): Promise { } const paths = argumentsFrom(argv); if (!paths) { - process.stderr.write(`${JSON.stringify({ schemaVersion: 1, component: 'qinglong3-run-management-client', event: 'usage_invalid', code: 'QL3_RUN_MANAGEMENT_CLIENT_USAGE_INVALID' })}\n`); + process.stderr.write( + `${JSON.stringify({ + schemaVersion: 1, + component: 'qinglong3-run-management-client', + event: 'usage_invalid', + code: 'QL3_RUN_MANAGEMENT_CLIENT_USAGE_INVALID', + })}\n`, + ); process.exitCode = 64; return; } try { + if (paths.kind === 'status') { + const result = await executeClusterRunManagementCommand({ + configFile: paths.configFile, + assertionFile: paths.assertionFile, + command: createRunCancellationStatusCommand(paths.projectId), + }); + const status = projectRunCancellationStatus(result); + process.stdout.write( + paths.format === 'json' + ? `${JSON.stringify(status)}\n` + : `${formatRunCancellationStatusCard(status)}\n`, + ); + process.exitCode = status.exitCode; + return; + } const result = await executeClusterRunManagementClient(paths); - process.stdout.write(`${JSON.stringify({ schemaVersion: 1, component: 'qinglong3-run-management-client', event: 'command_completed', requestId: result.requestId, result: result.result })}\n`); + process.stdout.write( + `${JSON.stringify({ + schemaVersion: 1, + component: 'qinglong3-run-management-client', + event: 'command_completed', + requestId: result.requestId, + result: result.result, + })}\n`, + ); } catch (error) { process.stderr.write(`${JSON.stringify(failureFact(error))}\n`); process.exitCode = 1; diff --git a/packages/ql3-cluster-admin/test/productCli.test.cjs b/packages/ql3-cluster-admin/test/productCli.test.cjs index a23e805c..1f5a4c09 100644 --- a/packages/ql3-cluster-admin/test/productCli.test.cjs +++ b/packages/ql3-cluster-admin/test/productCli.test.cjs @@ -375,7 +375,7 @@ test('catalog exposes only reviewed product entrypoints from the same package', test('help and version are bounded installation-derived product facts', () => { const help = qingLong3ClusterProductHelp(); assert.match(help, /^Usage: ql3-cluster-admin \[arguments\]/); - assert.match(help, /\n run\s+retry or stop Runs/); + assert.match(help, /\n run\s+observe, retry or stop Runs/); assert.match(help, /\n copilot\s+diagnose, inspect, read or cancel Runs/); assert.match(help, /\n copilot-mcp\s+serve the bounded Cluster Copilot MCP/); assert.match(help, /\n copilot-console\s+open the loopback-only read-only/); diff --git a/packages/ql3-cluster-admin/test/runCancellationStatus.test.cjs b/packages/ql3-cluster-admin/test/runCancellationStatus.test.cjs new file mode 100644 index 00000000..90a70fbf --- /dev/null +++ b/packages/ql3-cluster-admin/test/runCancellationStatus.test.cjs @@ -0,0 +1,121 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const test = require('node:test'); + +const { + createRunCancellationStatusCommand, + formatRunCancellationStatusCard, + projectRunCancellationStatus, +} = require('../dist/run-management/runCancellationStatus.js'); + +const uuids = [ + '019f9500-0000-4000-8000-000000000001', + '019f9500-0000-4000-8000-000000000002', + '019f9500-0000-4000-8000-000000000003', +]; + +function result(assessment) { + const blocked = assessment === 'attention_required' ? 1 : 0; + const pending = assessment === 'converging' ? 1 : 0; + return { + schemaVersion: 1, + requestId: 'request-summary-1', + result: { + schemaVersion: 1, + operation: 'run.cancellation.summary', + summary: { + schema: 'qinglong/run-cancellation-dispatch-summary@v1', + projectId: 'project-1', + observedAtMs: 1_700_000_000_000, + assessment, + operatorAction: + assessment === 'clear' + ? 'none' + : assessment === 'converging' + ? 'wait' + : 'inspect', + dispatches: { + total: blocked + pending, + pending, + leased: 0, + retryWait: 0, + dispatched: 0, + blocked, + }, + signals: { due: pending, expiredLease: 0 }, + blockingResults: { + identityMismatch: blocked, + pidMismatch: 0, + unsupported: 0, + invalid: 0, + }, + ...(blocked === 0 ? {} : { oldestBlockedAtMs: 1_699_999_999_000 }), + }, + }, + }; +} + +test('builds one exact Project summary command without a command file', () => { + let index = 0; + assert.deepEqual( + createRunCancellationStatusCommand('project-1', () => uuids[index++]), + { + schemaVersion: 1, + operation: 'run.cancellation.summary', + request: { + projectId: 'project-1', + requestId: uuids[0], + auditEventId: uuids[1], + failureAuditEventId: uuids[2], + body: { + schema: 'qinglong/run-cancellation-dispatch-summary-request@v1', + }, + }, + }, + ); +}); + +test('maps clear, converging and attention assessments to stable alert exits', () => { + const cases = [ + ['clear', 'ok', 0], + ['converging', 'warning', 10], + ['attention_required', 'critical', 20], + ]; + for (const [assessment, severity, exitCode] of cases) { + const status = projectRunCancellationStatus(result(assessment)); + assert.equal(status.schema, 'qinglong/run-cancellation-status@v1'); + assert.equal(status.assessment, assessment); + assert.equal(status.severity, severity); + assert.equal(status.exitCode, exitCode); + assert.equal(status.projectId, 'project-1'); + assert.equal(Object.hasOwn(status, 'oldestBlockedAtMs'), exitCode === 20); + } +}); + +test('renders a deterministic low-sensitive operator card', () => { + const card = formatRunCancellationStatusCard( + projectRunCancellationStatus(result('attention_required')), + ); + assert.match(card, /^QingLong 3\.0 \/ Cancellation Availability\n/); + assert.match(card, /ASSESSMENT ATTENTION REQUIRED/); + assert.match(card, /ALERT CRITICAL \(exit 20\)/); + assert.match(card, /DISPATCHES total=1 .* blocked=1/); + assert.match(card, /BLOCKING identity_mismatch=1/); + assert.doesNotMatch( + card, + /runId|attemptId|leaseOwner|leaseToken|command|environment|secret/i, + ); +}); + +test('refuses to project a non-summary management result', () => { + assert.throws( + () => + projectRunCancellationStatus({ + schemaVersion: 1, + requestId: 'request-1', + result: { schemaVersion: 1, operation: 'run.stop', stop: {} }, + }), + /requires a summary result/, + ); +}); diff --git a/packages/ql3-cluster-admin/test/runManagementClient.test.cjs b/packages/ql3-cluster-admin/test/runManagementClient.test.cjs index 3d85bc22..3478fc93 100644 --- a/packages/ql3-cluster-admin/test/runManagementClient.test.cjs +++ b/packages/ql3-cluster-admin/test/runManagementClient.test.cjs @@ -15,6 +15,7 @@ const { afterEach, test } = require('node:test'); const { executeClusterRunManagementClient, + executeClusterRunManagementCommand, validateClusterRunManagementClientResult, } = require('@qinglong/cluster-admin/run-management-client'); const { @@ -217,6 +218,32 @@ test('accepts only the exact Run route before opening one mTLS connection', asyn assert.equal(connects, 1); }); +test('accepts a normalized in-memory summary command without a command file', async () => { + const paths = clientFiles(); + let connects = 0; + await assert.rejects( + executeClusterRunManagementCommand( + { + configFile: paths.configFile, + assertionFile: paths.assertionFile, + command: summaryCommand, + }, + { + async connect(target) { + connects += 1; + assert.deepEqual(target, { + hostname: 'run.example.test', + port: 8448, + }); + throw new Error('expected-connect-stop'); + }, + }, + ), + { code: 'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_REQUEST_FAILED' }, + ); + assert.equal(connects, 1); +}); + test('rejects response target, execution placement and shape drift', () => { for (const candidate of [ response({ projectId: 'project-2' }), diff --git a/packages/ql3-cluster-admin/test/runManagementClientCli.test.cjs b/packages/ql3-cluster-admin/test/runManagementClientCli.test.cjs new file mode 100644 index 00000000..68975432 --- /dev/null +++ b/packages/ql3-cluster-admin/test/runManagementClientCli.test.cjs @@ -0,0 +1,217 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const { spawn } = require('node:child_process'); +const fs = require('node:fs'); +const { createServer } = require('node:https'); +const os = require('node:os'); +const path = require('node:path'); +const test = require('node:test'); + +const packageRoot = path.resolve(__dirname, '..'); +const cliPath = path.join( + packageRoot, + 'dist', + 'run-management', + 'runManagementClientCli.js', +); +const tlsFixture = path.resolve( + packageRoot, + '../ql3-cluster-control/test/fixtures/mtls', +); + +function privateFile(directory, name, contents) { + const filePath = path.join(directory, name); + fs.writeFileSync(filePath, contents, { mode: 0o600 }); + return fs.realpathSync(filePath); +} + +function runCli(args) { + return new Promise((resolve, reject) => { + const child = spawn(process.execPath, [cliPath, ...args], { + cwd: packageRoot, + stdio: ['ignore', 'pipe', 'pipe'], + }); + const stdout = []; + const stderr = []; + child.stdout.on('data', (chunk) => stdout.push(chunk)); + child.stderr.on('data', (chunk) => stderr.push(chunk)); + child.once('error', reject); + child.once('close', (status, signal) => { + resolve({ + status, + signal, + stdout: Buffer.concat(stdout).toString('utf8'), + stderr: Buffer.concat(stderr).toString('utf8'), + }); + }); + }); +} + +test('status mode calls only the summary operation and emits an alert exit', async (t) => { + const directory = fs.realpathSync( + fs.mkdtempSync(path.join(os.tmpdir(), 'ql3-run-status-cli-')), + ); + t.after(() => fs.rmSync(directory, { recursive: true, force: true })); + const caFile = privateFile( + directory, + 'ca.pem', + fs.readFileSync(path.join(tlsFixture, 'ca-cert.pem')), + ); + const clientCertificateFile = privateFile( + directory, + 'client.crt', + fs.readFileSync(path.join(tlsFixture, 'client-cert.pem')), + ); + const clientPrivateKeyFile = privateFile( + directory, + 'client.key', + fs.readFileSync(path.join(tlsFixture, 'client-key.pem')), + ); + const requests = []; + const server = createServer( + { + key: fs.readFileSync(path.join(tlsFixture, 'server-key.pem')), + cert: fs.readFileSync(path.join(tlsFixture, 'server-cert.pem')), + ca: fs.readFileSync(path.join(tlsFixture, 'ca-cert.pem')), + requestCert: true, + rejectUnauthorized: true, + minVersion: 'TLSv1.3', + maxVersion: 'TLSv1.3', + }, + (request, response) => { + const chunks = []; + request.on('data', (chunk) => chunks.push(chunk)); + request.on('end', () => { + const command = JSON.parse(Buffer.concat(chunks).toString('utf8')); + requests.push({ + method: request.method, + path: request.url, + authorization: request.headers.authorization, + authorized: request.socket.authorized, + command, + }); + const bytes = Buffer.from( + JSON.stringify({ + schemaVersion: 1, + requestId: command.request.requestId, + result: { + schemaVersion: 1, + operation: 'run.cancellation.summary', + summary: { + schema: 'qinglong/run-cancellation-dispatch-summary@v1', + projectId: 'project-1', + observedAtMs: 1_700_000_000_000, + assessment: 'attention_required', + operatorAction: 'inspect', + dispatches: { + total: 1, + pending: 0, + leased: 0, + retryWait: 0, + dispatched: 0, + blocked: 1, + }, + signals: { due: 0, expiredLease: 0 }, + blockingResults: { + identityMismatch: 1, + pidMismatch: 0, + unsupported: 0, + invalid: 0, + }, + oldestBlockedAtMs: 1_699_999_999_000, + }, + }, + }), + 'utf8', + ); + response.writeHead(200, { + 'content-type': 'application/json; charset=utf-8', + 'content-length': String(bytes.byteLength), + }); + response.end(bytes); + }); + }, + ); + await new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(0, '127.0.0.1', resolve); + }); + t.after( + () => + new Promise((resolve) => { + server.close(() => resolve()); + }), + ); + const address = server.address(); + assert.notEqual(address, null); + assert.notEqual(typeof address, 'string'); + const configFile = privateFile( + directory, + 'client.json', + JSON.stringify({ + schemaVersion: 1, + endpoint: `https://localhost:${address.port}/api/v3/runs/management`, + servername: 'localhost', + caFile, + clientCertificateFile, + clientPrivateKeyFile, + requestTimeoutMs: 2_000, + }), + ); + const assertionFile = privateFile( + directory, + 'assertion.jwt', + 'eyJhbGciOiJFZERTQSJ9.eyJzdWIiOiJvcGVyYXRvci0xIn0.c2lnbmF0dXJl', + ); + + const result = await runCli([ + `--config=${configFile}`, + 'status', + `--assertion=${assertionFile}`, + '--project=project-1', + '--format=json', + ]); + + assert.equal(result.status, 20, result.stderr); + assert.equal(result.signal, null); + assert.equal(result.stderr, ''); + const output = JSON.parse(result.stdout); + assert.equal(output.schema, 'qinglong/run-cancellation-status@v1'); + assert.equal(output.assessment, 'attention_required'); + assert.equal(output.severity, 'critical'); + assert.equal(output.exitCode, 20); + assert.equal(output.dispatches.blocked, 1); + assert.equal(requests.length, 1); + assert.equal(requests[0].method, 'POST'); + assert.equal(requests[0].path, '/api/v3/runs/management'); + assert.equal(requests[0].authorized, true); + assert.match(requests[0].authorization, /^Bearer [A-Za-z0-9_-]+\./); + assert.equal(requests[0].command.operation, 'run.cancellation.summary'); + assert.deepEqual(requests[0].command.request.body, { + schema: 'qinglong/run-cancellation-dispatch-summary-request@v1', + }); + assert.equal(Object.hasOwn(requests[0].command.request, 'runId'), false); +}); + +test('help documents status routing and invalid projects fail before I/O', async () => { + const help = await runCli(['--help']); + assert.equal(help.status, 0); + assert.match(help.stdout, /ql3-run-client status/); + assert.match(help.stdout, /0=clear, 10=converging, 20=attention_required/); + + const rejected = await runCli([ + 'status', + '--config=/private/client.json', + '--assertion=/private/assertion.jwt', + '--project=../escape', + ]); + assert.equal(rejected.status, 64); + assert.equal(rejected.stdout, ''); + assert.deepEqual(JSON.parse(rejected.stderr), { + schemaVersion: 1, + component: 'qinglong3-run-management-client', + event: 'usage_invalid', + code: 'QL3_RUN_MANAGEMENT_CLIENT_USAGE_INVALID', + }); +}); diff --git a/test/back/ql3PackageBoundaryAudit.test.cjs b/test/back/ql3PackageBoundaryAudit.test.cjs index b8050e46..5bdcb5d8 100644 --- a/test/back/ql3PackageBoundaryAudit.test.cjs +++ b/test/back/ql3PackageBoundaryAudit.test.cjs @@ -340,10 +340,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: clusterAdmin.rootSourceFileRoles, }, { - sourceFiles: 122, + sourceFiles: 123, rootSourceFiles: 1, rootSourceLines: 61, - nestedSourceFiles: 121, + nestedSourceFiles: 122, rootSourceFileRoles: { 'modelInvocationMigrationCli.ts': 'binary_entry', },