From d85cd88c2d755558426c577b42699d77dd997ca1 Mon Sep 17 00:00:00 2001 From: whyour Date: Wed, 12 Aug 2026 18:47:36 +0800 Subject: [PATCH] test(ql3): accept cluster run management live gate --- docs/QINGLONG_3_0_ARCHITECTURE_RFC.md | 17 ++-- ...run-management-kubernetes-live-evidence.md | 13 +-- docs/adr/README.md | 2 +- .../pluginPackageManagementClient.ts | 9 +- .../test/runManagementClient.test.cjs | 90 ++++++++++++++++++- ...l3-management-kubernetes-live-platform.cjs | 12 ++- ...un-management-kubernetes-live-contract.cjs | 37 +++++--- ...nManagementKubernetesLiveContract.test.cjs | 36 ++++++++ 8 files changed, 185 insertions(+), 31 deletions(-) diff --git a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md index e1ec5d1b..50d39fcf 100644 --- a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md +++ b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md @@ -11,19 +11,22 @@ 最新增量证据（2026-08-12）： -- D-298/ADR-0386（实现完成；固定本机 live 证据待锁定输入可达后补录） +- D-298/ADR-0386（已接受） Cluster `run.retry | run.stop` 已建立人工触发的真实三节点 Kubernetes 组合门：1 control-plane + 2 worker K3s/Flannel、3 实例 CloudNativePG 1.30.0/PostgreSQL 18.4、2 个跨节点 Run manager Pod、TLS 1.3 mTLS、purpose-bound OIDC strong User、identity generation overlap/revoke/rollback、client CRL rotation、CNPG primary promotion、数据库断连 readiness/liveness fence、CNI 与 RBAC least privilege，并以最终 PostgreSQL facts 验证 retry/stop 首写和 exact replay 无重复。报告 schema 固定 migration 57/control-core capability 56，只允许 `0600` content-free evidence，敏感材料与隐藏 limitation 均失败。实现没有新增 package、生产依赖、migration、表、 - 角色或默认 workload；live 代码按通用管理 Kubernetes helper、Run scenario、离线 audit 分层，现有 Approval 回归通过。当前静态/负向合同 - 20/20；本机首次 live 取锁定 CNPG manifest 时 GitHub 443 连接 75 秒无数据超时且无缓存，因此没有降低 checksum 或把静态结果冒充 live - 证据。完整 18-package build/test 退出 0，backend 1,174 pass/2 conditional skip/0 fail，14 个 Profile artifact 全部 compatible；基础 - Edge 为 2,467,343 bytes/295 files/53 modules，RSS delta 10,928,128 bytes，低于既有低配门限。package/dependency/local-image boundary + 角色或默认 workload；live 代码按通用管理 Kubernetes helper、Run scenario、离线 audit 分层，现有 Approval 回归通过。锁定 CNPG + manifest SHA-256 `f8bede43fe4ee0d478c2355b204a36876b2ae4faac60f2a9452280b293da3b88` 后，本机 arm64 真实门完整通过： + K3s `v1.34.3+k3s1` 三节点、CNPG/PostgreSQL 三实例、双跨节点 manager、identity generation/CRL rotation、primary promotion、数据库 + fail-closed recovery、CNI/RBAC/DB least privilege 全绿，重复 mutation 为 0；报告权限 `0600`、离线审计零 finding，SHA-256 + `4be50ad1ebbe0b9fea76ecac33133ea709461d44acbff730c60a37fe3fd2921a`，运行后临时 Docker 资源零残留。真实组合门同时修正共享 client + 缺少精确 Run route、本地镜像 placeholder 替换误伤 annotation，以及 live 重放 JTI 漂移三处问题；生产强认证/审计围栏与 60 秒限流参数均未 + 放宽。完整 18-package clean build/test 退出 0，backend 1,177 tests、1,175 pass/2 conditional skip/0 fail，14 个 Profile artifact 全部 + compatible；基础 Edge 为 2,467,343 bytes/295 files/53 modules，RSS delta 10,960,896 bytes，低于既有低配门限。package/dependency/local-image boundary 全绿，仍为 18 package、无 single-source/shallow package。PostgreSQL 18.4 arm64 HA 通过 123 gates、timeline `1→2`，报告 SHA-256 - `6adb8c9de8929ff54b522e9a251e3081d9dd004c1a91f72f83c33288ddce63a9` 且 Docker 零残留。固定 K3s/CNPG live report 是 ADR 从 - Proposed 转为 Accepted 的唯一剩余证据。 + `6adb8c9de8929ff54b522e9a251e3081d9dd004c1a91f72f83c33288ddce63a9` 且 Docker 零残留。 - D-297/ADR-0385（已接受） Local Edge/Standalone 已补齐强认证 `run.stop` 产品入口，并与既有 `run.retry` 统一为同一个 caller-driven `ql3-run retry|stop` binary，不新增 package、migration、表、索引、进程、listener、timer、watcher、连接、cache 或 sidecar。stop 只接受 POSIX 私有命令文件 diff --git a/docs/adr/ADR-0386-cluster-run-management-kubernetes-live-evidence.md b/docs/adr/ADR-0386-cluster-run-management-kubernetes-live-evidence.md index 6f4d767a..3f2b4ce4 100644 --- a/docs/adr/ADR-0386-cluster-run-management-kubernetes-live-evidence.md +++ b/docs/adr/ADR-0386-cluster-run-management-kubernetes-live-evidence.md @@ -1,6 +1,6 @@ # ADR-0386：Cluster Run Management Kubernetes 多节点实证 -- 状态：Proposed（实现与静态回归完成；固定本机 live 证据待锁定输入可达后补录） +- 状态：Accepted - 日期：2026-08-12 - 关联 RFC：QL-RFC-0001 D-298 - 前置决策：ADR-0359、ADR-0364、ADR-0366、ADR-0382、ADR-0383、ADR-0384 @@ -38,13 +38,16 @@ Cluster 已有强认证 `run.retry | run.stop`、专用 `ql3_run_manager`、两 ## 验收状态 -- Run/Approval live 静态合同、离线审计与既有 deployment 回归：20/20；Run report 审计含 topology/schema/deployment/client/rotation/availability/isolation/durability/secret 反向 fixture。 +- 锁定的 CNPG v1.30.0 manifest SHA-256 为 `f8bede43fe4ee0d478c2355b204a36876b2ae4faac60f2a9452280b293da3b88`；本机 arm64 真实三节点 K3s/CNPG Run gate 已通过，报告时间 `2026-08-12T08:25:36.761Z`，报告 SHA-256 为 `4be50ad1ebbe0b9fea76ecac33133ea709461d44acbff730c60a37fe3fd2921a`，权限 `0600`，离线审计零 finding。 +- 实证为 K3s `v1.34.3+k3s1`、3 个 CNI Ready 节点、CloudNativePG 1.30.0/PostgreSQL 18.4 三实例、2 个跨节点 Ready manager；migration 57/control-core capability 56。retry 状态为 `accepted → existing → existing`，stop 为 `accepted → already_requested → already_requested`，最终 retry Run/Attempt/Event 为 `1/1/2`、cancel Event 为 1、重复 mutation 为 0。 +- identity generation overlap/revoke/rollback、mTLS CRL 全 Pod replacement、CNPG primary promotion、数据库断连 readiness withdrawal/liveness preservation、恢复后强制 Pod replacement、CNI ingress/egress 与 RBAC/数据库角色 least privilege 全部通过；运行后 `ql3-run-live-*` 容器、网络和临时镜像均无残留。 +- 真实门首先暴露并修正三处组合缺口：共享产品 client 未允许精确 Run route；本地镜像替换误伤非镜像 zero-digest annotation；live 重放重新签发 JTI 导致认证/audit fence 漂移。最终重放在签名 key rotation 中固定认证会话 JTI，保留生产端强认证围栏；现场只把测试渲染的 HTTP rate window 收窄为 1 秒，生产清单仍为 peer 30/global 300 次每 60 秒。 +- Run/Approval live 静态合同、离线审计与既有 deployment 回归通过；Run report 审计含 topology/schema/deployment/client/rotation/availability/isolation/durability/secret 反向 fixture。 - runner、scenario、platform、identity、PKI 与 audit 均通过 Node syntax check；现有 Run deployment 合同通过。 -- 本机真实门尝试下载校验和锁定的 CNPG manifest 时，`github.com:443` 在 75 秒无数据后超时；本机无缓存。未降低 checksum、未改用未审计输入，也未把静态结果记录为 live 通过。 -- 完整 18-package build/test 退出 0；backend 1,174 pass/2 conditional skip/0 fail；14 个 Edge/Standalone Profile artifact 全部 compatible。基础 Edge 为 2,467,343 bytes/295 files/53 loaded modules，RSS delta 10,928,128 bytes，仍低于 4 MiB/512 files/16 MiB 门限。 +- 完整 18-package clean build/test 退出 0；backend 1,177 tests、1,175 pass/2 conditional skip/0 fail；14 个 Edge/Standalone Profile artifact 全部 compatible。基础 Edge 为 2,467,343 bytes/295 files/53 loaded modules，RSS delta 10,960,896 bytes，仍低于 4 MiB/512 files/16 MiB 门限。 - package/dependency/local-image boundary 全部 compatible，18 个 workspace package 无 single-source/shallow package；本切片没有改变任一 Local artifact 的生产闭包。 - PostgreSQL 18.4 arm64 HA 通过 123 gates、primary timeline `1→2`，报告 SHA-256 `6adb8c9de8929ff54b522e9a251e3081d9dd004c1a91f72f83c33288ddce63a9`；运行后 `ql3-ha-*` container、volume、network 均为空。 -- 固定 K3s/CloudNativePG live report 仍是 ADR 从 Proposed 转为 Accepted 的唯一未满足证据。 +- 固定 K3s/CloudNativePG live report 已满足，本 ADR 转为 Accepted。 ## 后果 diff --git a/docs/adr/README.md b/docs/adr/README.md index 58e768d0..f1c9ed78 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -389,7 +389,7 @@ | [ADR-0383](./ADR-0383-strong-cluster-run-management-plane.md) | 强认证的 Cluster Run Management Plane 与专用数据库角色 | Accepted | | [ADR-0384](./ADR-0384-strong-cluster-run-stop-management.md) | 强认证的 Cluster Run Stop Management 与列级数据库权限 | Accepted | | [ADR-0385](./ADR-0385-strong-local-run-stop-product-entry.md) | 强认证的 Local Run Stop 产品入口与原子审计 | Accepted | -| [ADR-0386](./ADR-0386-cluster-run-management-kubernetes-live-evidence.md) | Cluster Run Management Kubernetes 多节点实证 | Proposed（实现完成，固定 live 证据待补录） | +| [ADR-0386](./ADR-0386-cluster-run-management-kubernetes-live-evidence.md) | Cluster Run Management Kubernetes 多节点实证 | Accepted | ## 规则 diff --git a/packages/ql3-cluster-admin/src/management-support/pluginPackageManagementClient.ts b/packages/ql3-cluster-admin/src/management-support/pluginPackageManagementClient.ts index 22fe64c4..52bc3c6d 100644 --- a/packages/ql3-cluster-admin/src/management-support/pluginPackageManagementClient.ts +++ b/packages/ql3-cluster-admin/src/management-support/pluginPackageManagementClient.ts @@ -29,6 +29,7 @@ const ALLOWED_MANAGEMENT_PATHS = new Set([ '/api/v3/automations/management', '/api/v3/approvals/management', '/api/v3/provider-credentials/management', + '/api/v3/runs/management', ]); const MAX_CONFIG_BYTES = 16 * 1024; const MAX_ASSERTION_BYTES = 16 * 1024; @@ -777,7 +778,10 @@ function rawHeaderCount(rawHeaders: readonly string[], name: string): number { return count; } -export async function executeClusterAuthenticatedManagementClient( +export async function executeClusterAuthenticatedManagementClient< + Command, + Result, +>( paths: ClusterPluginPackageManagementClientPaths, protocol: ClusterAuthenticatedManagementClientProtocol, connectionOptions?: ClusterPluginPackageManagementClientConnectionOptions, @@ -919,7 +923,8 @@ export async function executeClusterAuthenticatedManagementClient { + for (const directory of temporaryDirectories.splice(0)) { + rmSync(directory, { recursive: true, force: true }); + } +}); + const command = normalizeClusterRunManagementCommand({ schemaVersion: 1, operation: 'run.retry', @@ -81,6 +151,24 @@ test('validates one low-sensitive retry response against the request fence', () ); }); +test('accepts only the exact Run route before opening one mTLS connection', async () => { + let connects = 0; + await assert.rejects( + executeClusterRunManagementClient(clientFiles(), { + async connect(target) { + connects += 1; + assert.deepEqual(target, { + hostname: 'run.example.test', + port: 8448, + }); + throw new Error('expected-connect-stop'); + }, + }), + { code: 'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_REQUEST_FAILED' }, + ); + assert.equal(connects, 1); +}); + test('rejects response target, execution placement and shape drift', () => { for (const candidate of [ response({ projectId: 'project-2' }), diff --git a/scripts/lib/ql3-management-kubernetes-live-platform.cjs b/scripts/lib/ql3-management-kubernetes-live-platform.cjs index a89d927a..72c47d7e 100644 --- a/scripts/lib/ql3-management-kubernetes-live-platform.cjs +++ b/scripts/lib/ql3-management-kubernetes-live-platform.cjs @@ -10,10 +10,16 @@ function imageIdDigest(image) { } function localManifest(rendered, imageName, localImage) { - const occurrences = rendered.split(imageName).length - 1; - assert.ok(occurrences >= 1, 'reviewed image reference is missing'); + const zeroDigest = 'sha256:' + '0'.repeat(64); + const placeholder = imageName + '@' + zeroDigest; + const occurrences = rendered.split(placeholder).length - 1; + assert.equal( + occurrences, + 1, + 'rendered manifest must contain one reviewed image placeholder', + ); return rendered - .replaceAll(imageName, localImage) + .replace(placeholder, localImage) .replaceAll('imagePullPolicy: IfNotPresent', 'imagePullPolicy: Never'); } diff --git a/scripts/ql3-run-management-kubernetes-live-contract.cjs b/scripts/ql3-run-management-kubernetes-live-contract.cjs index 80efebee..aeb9d391 100644 --- a/scripts/ql3-run-management-kubernetes-live-contract.cjs +++ b/scripts/ql3-run-management-kubernetes-live-contract.cjs @@ -403,9 +403,16 @@ async function main(argv = process.argv.slice(2)) { adminImage, ); assert.equal(managerManifest.split(ZERO_DIGEST).length - 1, 2); + const productionRateWindowPattern = + /(- name: QL3_RUN_MANAGEMENT_RATE_WINDOW_MS\n\s+value: )"60000"/g; + assert.equal( + managerManifest.match(productionRateWindowPattern)?.length ?? 0, + 1, + ); managerManifest = managerManifest .replace(ZERO_DIGEST, sha256(pkiMaterial.ca)) - .replace(ZERO_DIGEST, sha256(pkiMaterial.clientCrl)); + .replace(ZERO_DIGEST, sha256(pkiMaterial.clientCrl)) + .replace(productionRateWindowPattern, '$1"1000"'); fixture.kubectl(['apply', '-f', '-'], { input: managerManifest + '\n' }); waitManagementRollout(managerOptions(fixture)); let managerPods = await readyManagementPods(managerOptions(fixture)); @@ -452,7 +459,10 @@ async function main(argv = process.argv.slice(2)) { networkPolicyLabel: 'qinglong.io/run-management-client', clientCliPath: '/opt/qinglong/node_modules/@qinglong/cluster-admin/dist/run-management/runManagementClientCli.js', - retryableClientCodes: ['QL3_RUN_MANAGEMENT_CLIENT_FAILED'], + retryableClientCodes: [ + 'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_REQUEST_FAILED', + 'QL3_RUN_MANAGEMENT_CLIENT_FAILED', + ], description: 'Run management', }); const retryMutationId = crypto.randomUUID(); @@ -463,12 +473,14 @@ async function main(argv = process.argv.slice(2)) { retryMutationId, 1, ); + const retryAuthenticationJti = 'run-live-retry-session-' + values.suffix; + const retryBearer = identity.assertion(oldKey, retryAuthenticationJti); const retryAccepted = await executeClient( { name: 'ql3-run-retry-accepted', target: managerPods[0], command: retry, - bearer: identity.assertion(oldKey), + bearer: retryBearer, clientCertificate: pkiMaterial.oldClientCertificate, clientKey: pkiMaterial.oldClientKey, }, @@ -480,7 +492,7 @@ async function main(argv = process.argv.slice(2)) { name: 'ql3-run-retry-replay', target: managerPods[1], command: retry, - bearer: identity.assertion(oldKey), + bearer: retryBearer, clientCertificate: pkiMaterial.newClientCertificate, clientKey: pkiMaterial.newClientKey, }, @@ -540,12 +552,13 @@ async function main(argv = process.argv.slice(2)) { stopMutationId, 4, ); + const stopAuthenticationJti = 'run-live-stop-session-' + values.suffix; const overlapOld = await executeClient( { name: 'ql3-run-stop-overlap-old', target: managerPods[0], command: stop, - bearer: identity.assertion(oldKey), + bearer: identity.assertion(oldKey, stopAuthenticationJti), clientCertificate: pkiMaterial.newClientCertificate, clientKey: pkiMaterial.newClientKey, }, @@ -556,7 +569,7 @@ async function main(argv = process.argv.slice(2)) { name: 'ql3-run-stop-overlap-new', target: managerPods[1], command: stop, - bearer: identity.assertion(newKey), + bearer: identity.assertion(newKey, stopAuthenticationJti), clientCertificate: pkiMaterial.newClientCertificate, clientKey: pkiMaterial.newClientKey, }, @@ -582,7 +595,7 @@ async function main(argv = process.argv.slice(2)) { name: 'ql3-run-stop-revoked-key', target: managerPods[0], command: stop, - bearer: identity.assertion(oldKey), + bearer: identity.assertion(oldKey, stopAuthenticationJti), clientCertificate: pkiMaterial.newClientCertificate, clientKey: pkiMaterial.newClientKey, }, @@ -593,7 +606,7 @@ async function main(argv = process.argv.slice(2)) { name: 'ql3-run-stop-active-key', target: managerPods[1], command: stop, - bearer: identity.assertion(newKey), + bearer: identity.assertion(newKey, stopAuthenticationJti), clientCertificate: pkiMaterial.newClientCertificate, clientKey: pkiMaterial.newClientKey, }, @@ -687,7 +700,7 @@ async function main(argv = process.argv.slice(2)) { name: 'ql3-run-retry-revoked-cert', target: managerPods[0], command: retry, - bearer: identity.assertion(newKey), + bearer: identity.assertion(newKey, retryAuthenticationJti), clientCertificate: pkiMaterial.oldClientCertificate, clientKey: pkiMaterial.oldClientKey, }, @@ -698,7 +711,7 @@ async function main(argv = process.argv.slice(2)) { name: 'ql3-run-retry-active-cert', target: managerPods[1], command: retry, - bearer: identity.assertion(newKey), + bearer: identity.assertion(newKey, retryAuthenticationJti), clientCertificate: pkiMaterial.newClientCertificate, clientKey: pkiMaterial.newClientKey, }, @@ -782,7 +795,7 @@ async function main(argv = process.argv.slice(2)) { name: 'ql3-run-database-unavailable-' + String(index + 1), target: pod, command: stop, - bearer: identity.assertion(newKey), + bearer: identity.assertion(newKey, stopAuthenticationJti), clientCertificate: pkiMaterial.newClientCertificate, clientKey: pkiMaterial.newClientKey, }, @@ -862,7 +875,7 @@ async function main(argv = process.argv.slice(2)) { name: 'ql3-run-database-recovered-' + String(index + 1), target: pod, command: stop, - bearer: identity.assertion(newKey), + bearer: identity.assertion(newKey, stopAuthenticationJti), clientCertificate: pkiMaterial.newClientCertificate, clientKey: pkiMaterial.newClientKey, }, diff --git a/test/back/ql3RunManagementKubernetesLiveContract.test.cjs b/test/back/ql3RunManagementKubernetesLiveContract.test.cjs index fba62dc8..800e81c6 100644 --- a/test/back/ql3RunManagementKubernetesLiveContract.test.cjs +++ b/test/back/ql3RunManagementKubernetesLiveContract.test.cjs @@ -9,6 +9,28 @@ const { retryCommand, stopCommand, } = require('../../scripts/ql3-run-management-kubernetes-live-contract.cjs'); +const { + localManifest, +} = require('../../scripts/lib/ql3-management-kubernetes-live-platform.cjs'); + +test('Run live local manifest replaces only the exact image placeholder', () => { + const zeroDigest = 'sha256:' + '0'.repeat(64); + const image = 'registry.example.com/qinglong/control'; + const rendered = [ + `image: ${image}@${zeroDigest}`, + 'imagePullPolicy: IfNotPresent', + `qinglong.io/client-ca-sha256: ${zeroDigest}`, + ].join('\n'); + const local = localManifest(rendered, image, 'ql3-control-live:test'); + assert.match(local, /image: ql3-control-live:test/); + assert.match(local, /imagePullPolicy: Never/); + assert.match(local, new RegExp(`client-ca-sha256: ${zeroDigest}`)); + assert.doesNotMatch(local, new RegExp(`${image}@${zeroDigest}`)); + assert.throws( + () => localManifest(rendered.replace('@' + zeroDigest, ''), image, 'x:y'), + /one reviewed image placeholder/, + ); +}); test('Run management live report path is mandatory before mutation begins', () => { const script = path.resolve( @@ -90,6 +112,20 @@ test('Run live runner remains opt-in, layered and observation backed', () => { assert.match(source, /reviewedOperatorManifest\(operatorManifestFile\)/); assert.match(source, /validateRunManagementKubernetesLiveReport/); assert.match(source, /createManagementClientExecutor/); + assert.match(source, /QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_REQUEST_FAILED/); + assert.match(source, /QL3_RUN_MANAGEMENT_CLIENT_FAILED/); + assert.match(source, /const productionRateWindowPattern =/); + assert.match(source, /const retryAuthenticationJti =/); + assert.match(source, /const stopAuthenticationJti =/); + assert.match( + source, + /const retryBearer = identity\.assertion\(oldKey, retryAuthenticationJti\)/, + ); + assert.equal( + source.split('bearer: retryBearer').length - 1, + 2, + 'an exact mutation replay must retain its original authentication fence', + ); assert.match(source, /durableRunManagementFacts/); assert.match(source, /Run identity ledger rollback surge failure/); assert.match(source, /CloudNativePG primary promotion/);