test(ql3): accept cluster run management live gate

docs/QINGLONG_3_0_ARCHITECTURE_RFC.md

@@ -11,19 +11,22 @@
 
 最新增量证据（2026-08-12）：
 
-- D-298/ADR-0386（实现完成；固定本机 live 证据待锁定输入可达后补录）
+- D-298/ADR-0386（已接受）
   Cluster `run.retry | run.stop` 已建立人工触发的真实三节点 Kubernetes 组合门：1 control-plane + 2 worker K3s/Flannel、3 实例
   CloudNativePG 1.30.0/PostgreSQL 18.4、2 个跨节点 Run manager Pod、TLS 1.3 mTLS、purpose-bound OIDC strong User、identity
   generation overlap/revoke/rollback、client CRL rotation、CNPG primary promotion、数据库断连 readiness/liveness fence、CNI 与 RBAC
   least privilege，并以最终 PostgreSQL facts 验证 retry/stop 首写和 exact replay 无重复。报告 schema 固定 migration 57/control-core
   capability 56，只允许 `0600` content-free evidence，敏感材料与隐藏 limitation 均失败。实现没有新增 package、生产依赖、migration、表、
-  角色或默认 workload；live 代码按通用管理 Kubernetes helper、Run scenario、离线 audit 分层，现有 Approval 回归通过。当前静态/负向合同
+  角色或默认 workload；live 代码按通用管理 Kubernetes helper、Run scenario、离线 audit 分层，现有 Approval 回归通过。锁定 CNPG
-  20/20；本机首次 live 取锁定 CNPG manifest 时 GitHub 443 连接 75 秒无数据超时且无缓存，因此没有降低 checksum 或把静态结果冒充 live
+  manifest SHA-256 `f8bede43fe4ee0d478c2355b204a36876b2ae4faac60f2a9452280b293da3b88` 后，本机 arm64 真实门完整通过：
-  证据。完整 18-package build/test 退出 0，backend 1,174 pass/2 conditional skip/0 fail，14 个 Profile artifact 全部 compatible；基础
+  K3s `v1.34.3+k3s1` 三节点、CNPG/PostgreSQL 三实例、双跨节点 manager、identity generation/CRL rotation、primary promotion、数据库
-  Edge 为 2,467,343 bytes/295 files/53 modules，RSS delta 10,928,128 bytes，低于既有低配门限。package/dependency/local-image boundary
+  fail-closed recovery、CNI/RBAC/DB least privilege 全绿，重复 mutation 为 0；报告权限 `0600`、离线审计零 finding，SHA-256
+  `4be50ad1ebbe0b9fea76ecac33133ea709461d44acbff730c60a37fe3fd2921a`，运行后临时 Docker 资源零残留。真实组合门同时修正共享 client
+  缺少精确 Run route、本地镜像 placeholder 替换误伤 annotation，以及 live 重放 JTI 漂移三处问题；生产强认证/审计围栏与 60 秒限流参数均未
+  放宽。完整 18-package clean build/test 退出 0，backend 1,177 tests、1,175 pass/2 conditional skip/0 fail，14 个 Profile artifact 全部
+  compatible；基础 Edge 为 2,467,343 bytes/295 files/53 modules，RSS delta 10,960,896 bytes，低于既有低配门限。package/dependency/local-image boundary
   全绿，仍为 18 package、无 single-source/shallow package。PostgreSQL 18.4 arm64 HA 通过 123 gates、timeline `1→2`，报告 SHA-256
-  `6adb8c9de8929ff54b522e9a251e3081d9dd004c1a91f72f83c33288ddce63a9` 且 Docker 零残留。固定 K3s/CNPG live report 是 ADR 从
+  `6adb8c9de8929ff54b522e9a251e3081d9dd004c1a91f72f83c33288ddce63a9` 且 Docker 零残留。
-  Proposed 转为 Accepted 的唯一剩余证据。
 - D-297/ADR-0385（已接受）
   Local Edge/Standalone 已补齐强认证 `run.stop` 产品入口，并与既有 `run.retry` 统一为同一个 caller-driven `ql3-run retry|stop`
   binary，不新增 package、migration、表、索引、进程、listener、timer、watcher、连接、cache 或 sidecar。stop 只接受 POSIX 私有命令文件

docs/adr/ADR-0386-cluster-run-management-kubernetes-live-evidence.md

@@ -1,6 +1,6 @@
 # ADR-0386：Cluster Run Management Kubernetes 多节点实证
 
-- 状态：Proposed（实现与静态回归完成；固定本机 live 证据待锁定输入可达后补录）
+- 状态：Accepted
 - 日期：2026-08-12
 - 关联 RFC：QL-RFC-0001 D-298
 - 前置决策：ADR-0359、ADR-0364、ADR-0366、ADR-0382、ADR-0383、ADR-0384
@@ -38,13 +38,16 @@ Cluster 已有强认证 `run.retry | run.stop`、专用 `ql3_run_manager`、两
 
 ## 验收状态
 
-- Run/Approval live 静态合同、离线审计与既有 deployment 回归：20/20；Run report 审计含 topology/schema/deployment/client/rotation/availability/isolation/durability/secret 反向 fixture。
+- 锁定的 CNPG v1.30.0 manifest SHA-256 为 `f8bede43fe4ee0d478c2355b204a36876b2ae4faac60f2a9452280b293da3b88`；本机 arm64 真实三节点 K3s/CNPG Run gate 已通过，报告时间 `2026-08-12T08:25:36.761Z`，报告 SHA-256 为 `4be50ad1ebbe0b9fea76ecac33133ea709461d44acbff730c60a37fe3fd2921a`，权限 `0600`，离线审计零 finding。
+- 实证为 K3s `v1.34.3+k3s1`、3 个 CNI Ready 节点、CloudNativePG 1.30.0/PostgreSQL 18.4 三实例、2 个跨节点 Ready manager；migration 57/control-core capability 56。retry 状态为 `accepted → existing → existing`，stop 为 `accepted → already_requested → already_requested`，最终 retry Run/Attempt/Event 为 `1/1/2`、cancel Event 为 1、重复 mutation 为 0。
+- identity generation overlap/revoke/rollback、mTLS CRL 全 Pod replacement、CNPG primary promotion、数据库断连 readiness withdrawal/liveness preservation、恢复后强制 Pod replacement、CNI ingress/egress 与 RBAC/数据库角色 least privilege 全部通过；运行后 `ql3-run-live-*` 容器、网络和临时镜像均无残留。
+- 真实门首先暴露并修正三处组合缺口：共享产品 client 未允许精确 Run route；本地镜像替换误伤非镜像 zero-digest annotation；live 重放重新签发 JTI 导致认证/audit fence 漂移。最终重放在签名 key rotation 中固定认证会话 JTI，保留生产端强认证围栏；现场只把测试渲染的 HTTP rate window 收窄为 1 秒，生产清单仍为 peer 30/global 300 次每 60 秒。
+- Run/Approval live 静态合同、离线审计与既有 deployment 回归通过；Run report 审计含 topology/schema/deployment/client/rotation/availability/isolation/durability/secret 反向 fixture。
 - runner、scenario、platform、identity、PKI 与 audit 均通过 Node syntax check；现有 Run deployment 合同通过。
-- 本机真实门尝试下载校验和锁定的 CNPG manifest 时，`github.com:443` 在 75 秒无数据后超时；本机无缓存。未降低 checksum、未改用未审计输入，也未把静态结果记录为 live 通过。
+- 完整 18-package clean build/test 退出 0；backend 1,177 tests、1,175 pass/2 conditional skip/0 fail；14 个 Edge/Standalone Profile artifact 全部 compatible。基础 Edge 为 2,467,343 bytes/295 files/53 loaded modules，RSS delta 10,960,896 bytes，仍低于 4 MiB/512 files/16 MiB 门限。
-- 完整 18-package build/test 退出 0；backend 1,174 pass/2 conditional skip/0 fail；14 个 Edge/Standalone Profile artifact 全部 compatible。基础 Edge 为 2,467,343 bytes/295 files/53 loaded modules，RSS delta 10,928,128 bytes，仍低于 4 MiB/512 files/16 MiB 门限。
 - package/dependency/local-image boundary 全部 compatible，18 个 workspace package 无 single-source/shallow package；本切片没有改变任一 Local artifact 的生产闭包。
 - PostgreSQL 18.4 arm64 HA 通过 123 gates、primary timeline `1→2`，报告 SHA-256 `6adb8c9de8929ff54b522e9a251e3081d9dd004c1a91f72f83c33288ddce63a9`；运行后 `ql3-ha-*` container、volume、network 均为空。
-- 固定 K3s/CloudNativePG live report 仍是 ADR 从 Proposed 转为 Accepted 的唯一未满足证据。
+- 固定 K3s/CloudNativePG live report 已满足，本 ADR 转为 Accepted。
 
 ## 后果
 

docs/adr/README.md

@@ -389,7 +389,7 @@
 | [ADR-0383](./ADR-0383-strong-cluster-run-management-plane.md)                                                 | 强认证的 Cluster Run Management Plane 与专用数据库角色                                                                           | Accepted                                |
 | [ADR-0384](./ADR-0384-strong-cluster-run-stop-management.md)                                                  | 强认证的 Cluster Run Stop Management 与列级数据库权限                                                                            | Accepted                                |
 | [ADR-0385](./ADR-0385-strong-local-run-stop-product-entry.md)                                                 | 强认证的 Local Run Stop 产品入口与原子审计                                                                                        | Accepted                                |
-| [ADR-0386](./ADR-0386-cluster-run-management-kubernetes-live-evidence.md)                                     | Cluster Run Management Kubernetes 多节点实证                                                                                       | Proposed（实现完成，固定 live 证据待补录） |
+| [ADR-0386](./ADR-0386-cluster-run-management-kubernetes-live-evidence.md)                                     | Cluster Run Management Kubernetes 多节点实证                                                                                       | Accepted |
 
 ## 规则
 

packages/ql3-cluster-admin/src/management-support/pluginPackageManagementClient.ts

@@ -29,6 +29,7 @@ const ALLOWED_MANAGEMENT_PATHS = new Set([
   '/api/v3/automations/management',
   '/api/v3/approvals/management',
   '/api/v3/provider-credentials/management',
+  '/api/v3/runs/management',
 ]);
 const MAX_CONFIG_BYTES = 16 * 1024;
 const MAX_ASSERTION_BYTES = 16 * 1024;
@@ -777,7 +778,10 @@ function rawHeaderCount(rawHeaders: readonly string[], name: string): number {
   return count;
 }
 
-export async function executeClusterAuthenticatedManagementClient<Command, Result>(
+export async function executeClusterAuthenticatedManagementClient<
+  Command,
+  Result,
+>(
   paths: ClusterPluginPackageManagementClientPaths,
   protocol: ClusterAuthenticatedManagementClientProtocol<Command, Result>,
   connectionOptions?: ClusterPluginPackageManagementClientConnectionOptions,
@@ -919,7 +923,8 @@ export async function executeClusterAuthenticatedManagementClient<Command, Resul
         }
       } catch (error) {
         if (
-          error instanceof ClusterPluginPackageManagementClientConfigurationError
+          error instanceof
+          ClusterPluginPackageManagementClientConfigurationError
         ) {
           throw error;
         }

packages/ql3-cluster-admin/test/runManagementClient.test.cjs

@@ -1,9 +1,20 @@
 'use strict';
 
 const assert = require('node:assert/strict');
-const { test } = require('node:test');
+const {
+  chmodSync,
+  mkdtempSync,
+  readFileSync,
+  realpathSync,
+  rmSync,
+  writeFileSync,
+} = require('node:fs');
+const { tmpdir } = require('node:os');
+const { join, resolve } = require('node:path');
+const { afterEach, test } = require('node:test');
 
 const {
+  executeClusterRunManagementClient,
   validateClusterRunManagementClientResult,
 } = require('@qinglong/cluster-admin/run-management-client');
 const {
@@ -13,6 +24,65 @@ const {
   ClusterPluginPackageManagementClientRequestError,
 } = require('@qinglong/cluster-admin/plugin-package-management-client');
 
+const FIXTURES = resolve(
+  __dirname,
+  '../../ql3-cluster-control/test/fixtures/mtls',
+);
+const temporaryDirectories = [];
+
+function privateWrite(filePath, value) {
+  writeFileSync(filePath, value, { mode: 0o600 });
+  chmodSync(filePath, 0o600);
+}
+
+function clientFiles() {
+  const directory = realpathSync(
+    mkdtempSync(join(tmpdir(), 'ql3-run-management-client-')),
+  );
+  temporaryDirectories.push(directory);
+  const paths = {
+    configFile: join(directory, 'client.json'),
+    commandFile: join(directory, 'command.json'),
+    assertionFile: join(directory, 'assertion.jwt'),
+  };
+  const caFile = join(directory, 'ca.crt');
+  const clientCertificateFile = join(directory, 'client.crt');
+  const clientPrivateKeyFile = join(directory, 'client.key');
+  privateWrite(caFile, readFileSync(join(FIXTURES, 'ca-cert.pem')));
+  privateWrite(
+    clientCertificateFile,
+    readFileSync(join(FIXTURES, 'client-cert.pem')),
+  );
+  privateWrite(
+    clientPrivateKeyFile,
+    readFileSync(join(FIXTURES, 'client-key.pem')),
+  );
+  privateWrite(
+    paths.configFile,
+    `${JSON.stringify({
+      schemaVersion: 1,
+      endpoint: 'https://run.example.test:8448/api/v3/runs/management',
+      servername: 'run.example.test',
+      caFile,
+      clientCertificateFile,
+      clientPrivateKeyFile,
+      requestTimeoutMs: 1_000,
+    })}\n`,
+  );
+  privateWrite(paths.commandFile, `${JSON.stringify(command)}\n`);
+  privateWrite(
+    paths.assertionFile,
+    'eyJhbGciOiJFZERTQSJ9.eyJzdWIiOiJ1In0.c2lnbmF0dXJl',
+  );
+  return paths;
+}
+
+afterEach(() => {
+  for (const directory of temporaryDirectories.splice(0)) {
+    rmSync(directory, { recursive: true, force: true });
+  }
+});
+
 const command = normalizeClusterRunManagementCommand({
   schemaVersion: 1,
   operation: 'run.retry',
@@ -81,6 +151,24 @@ test('validates one low-sensitive retry response against the request fence', ()
   );
 });
 
+test('accepts only the exact Run route before opening one mTLS connection', async () => {
+  let connects = 0;
+  await assert.rejects(
+    executeClusterRunManagementClient(clientFiles(), {
+      async connect(target) {
+        connects += 1;
+        assert.deepEqual(target, {
+          hostname: 'run.example.test',
+          port: 8448,
+        });
+        throw new Error('expected-connect-stop');
+      },
+    }),
+    { code: 'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_REQUEST_FAILED' },
+  );
+  assert.equal(connects, 1);
+});
+
 test('rejects response target, execution placement and shape drift', () => {
   for (const candidate of [
     response({ projectId: 'project-2' }),

scripts/lib/ql3-management-kubernetes-live-platform.cjs

@@ -10,10 +10,16 @@ function imageIdDigest(image) {
 }
 
 function localManifest(rendered, imageName, localImage) {
-  const occurrences = rendered.split(imageName).length - 1;
+  const zeroDigest = 'sha256:' + '0'.repeat(64);
-  assert.ok(occurrences >= 1, 'reviewed image reference is missing');
+  const placeholder = imageName + '@' + zeroDigest;
+  const occurrences = rendered.split(placeholder).length - 1;
+  assert.equal(
+    occurrences,
+    1,
+    'rendered manifest must contain one reviewed image placeholder',
+  );
   return rendered
-    .replaceAll(imageName, localImage)
+    .replace(placeholder, localImage)
     .replaceAll('imagePullPolicy: IfNotPresent', 'imagePullPolicy: Never');
 }
 

scripts/ql3-run-management-kubernetes-live-contract.cjs

@@ -403,9 +403,16 @@ async function main(argv = process.argv.slice(2)) {
       adminImage,
     );
     assert.equal(managerManifest.split(ZERO_DIGEST).length - 1, 2);
+    const productionRateWindowPattern =
+      /(- name: QL3_RUN_MANAGEMENT_RATE_WINDOW_MS\n\s+value: )"60000"/g;
+    assert.equal(
+      managerManifest.match(productionRateWindowPattern)?.length ?? 0,
+      1,
+    );
     managerManifest = managerManifest
       .replace(ZERO_DIGEST, sha256(pkiMaterial.ca))
-      .replace(ZERO_DIGEST, sha256(pkiMaterial.clientCrl));
+      .replace(ZERO_DIGEST, sha256(pkiMaterial.clientCrl))
+      .replace(productionRateWindowPattern, '$1"1000"');
     fixture.kubectl(['apply', '-f', '-'], { input: managerManifest + '\n' });
     waitManagementRollout(managerOptions(fixture));
     let managerPods = await readyManagementPods(managerOptions(fixture));
@@ -452,7 +459,10 @@ async function main(argv = process.argv.slice(2)) {
       networkPolicyLabel: 'qinglong.io/run-management-client',
       clientCliPath:
         '/opt/qinglong/node_modules/@qinglong/cluster-admin/dist/run-management/runManagementClientCli.js',
-      retryableClientCodes: ['QL3_RUN_MANAGEMENT_CLIENT_FAILED'],
+      retryableClientCodes: [
+        'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_REQUEST_FAILED',
+        'QL3_RUN_MANAGEMENT_CLIENT_FAILED',
+      ],
       description: 'Run management',
     });
     const retryMutationId = crypto.randomUUID();
@@ -463,12 +473,14 @@ async function main(argv = process.argv.slice(2)) {
       retryMutationId,
       1,
     );
+    const retryAuthenticationJti = 'run-live-retry-session-' + values.suffix;
+    const retryBearer = identity.assertion(oldKey, retryAuthenticationJti);
     const retryAccepted = await executeClient(
       {
         name: 'ql3-run-retry-accepted',
         target: managerPods[0],
         command: retry,
-        bearer: identity.assertion(oldKey),
+        bearer: retryBearer,
         clientCertificate: pkiMaterial.oldClientCertificate,
         clientKey: pkiMaterial.oldClientKey,
       },
@@ -480,7 +492,7 @@ async function main(argv = process.argv.slice(2)) {
         name: 'ql3-run-retry-replay',
         target: managerPods[1],
         command: retry,
-        bearer: identity.assertion(oldKey),
+        bearer: retryBearer,
         clientCertificate: pkiMaterial.newClientCertificate,
         clientKey: pkiMaterial.newClientKey,
       },
@@ -540,12 +552,13 @@ async function main(argv = process.argv.slice(2)) {
       stopMutationId,
       4,
     );
+    const stopAuthenticationJti = 'run-live-stop-session-' + values.suffix;
     const overlapOld = await executeClient(
       {
         name: 'ql3-run-stop-overlap-old',
         target: managerPods[0],
         command: stop,
-        bearer: identity.assertion(oldKey),
+        bearer: identity.assertion(oldKey, stopAuthenticationJti),
         clientCertificate: pkiMaterial.newClientCertificate,
         clientKey: pkiMaterial.newClientKey,
       },
@@ -556,7 +569,7 @@ async function main(argv = process.argv.slice(2)) {
         name: 'ql3-run-stop-overlap-new',
         target: managerPods[1],
         command: stop,
-        bearer: identity.assertion(newKey),
+        bearer: identity.assertion(newKey, stopAuthenticationJti),
         clientCertificate: pkiMaterial.newClientCertificate,
         clientKey: pkiMaterial.newClientKey,
       },
@@ -582,7 +595,7 @@ async function main(argv = process.argv.slice(2)) {
         name: 'ql3-run-stop-revoked-key',
         target: managerPods[0],
         command: stop,
-        bearer: identity.assertion(oldKey),
+        bearer: identity.assertion(oldKey, stopAuthenticationJti),
         clientCertificate: pkiMaterial.newClientCertificate,
         clientKey: pkiMaterial.newClientKey,
       },
@@ -593,7 +606,7 @@ async function main(argv = process.argv.slice(2)) {
         name: 'ql3-run-stop-active-key',
         target: managerPods[1],
         command: stop,
-        bearer: identity.assertion(newKey),
+        bearer: identity.assertion(newKey, stopAuthenticationJti),
         clientCertificate: pkiMaterial.newClientCertificate,
         clientKey: pkiMaterial.newClientKey,
       },
@@ -687,7 +700,7 @@ async function main(argv = process.argv.slice(2)) {
         name: 'ql3-run-retry-revoked-cert',
         target: managerPods[0],
         command: retry,
-        bearer: identity.assertion(newKey),
+        bearer: identity.assertion(newKey, retryAuthenticationJti),
         clientCertificate: pkiMaterial.oldClientCertificate,
         clientKey: pkiMaterial.oldClientKey,
       },
@@ -698,7 +711,7 @@ async function main(argv = process.argv.slice(2)) {
         name: 'ql3-run-retry-active-cert',
         target: managerPods[1],
         command: retry,
-        bearer: identity.assertion(newKey),
+        bearer: identity.assertion(newKey, retryAuthenticationJti),
         clientCertificate: pkiMaterial.newClientCertificate,
         clientKey: pkiMaterial.newClientKey,
       },
@@ -782,7 +795,7 @@ async function main(argv = process.argv.slice(2)) {
             name: 'ql3-run-database-unavailable-' + String(index + 1),
             target: pod,
             command: stop,
-            bearer: identity.assertion(newKey),
+            bearer: identity.assertion(newKey, stopAuthenticationJti),
             clientCertificate: pkiMaterial.newClientCertificate,
             clientKey: pkiMaterial.newClientKey,
           },
@@ -862,7 +875,7 @@ async function main(argv = process.argv.slice(2)) {
             name: 'ql3-run-database-recovered-' + String(index + 1),
             target: pod,
             command: stop,
-            bearer: identity.assertion(newKey),
+            bearer: identity.assertion(newKey, stopAuthenticationJti),
             clientCertificate: pkiMaterial.newClientCertificate,
             clientKey: pkiMaterial.newClientKey,
           },

test/back/ql3RunManagementKubernetesLiveContract.test.cjs

@@ -9,6 +9,28 @@ const {
   retryCommand,
   stopCommand,
 } = require('../../scripts/ql3-run-management-kubernetes-live-contract.cjs');
+const {
+  localManifest,
+} = require('../../scripts/lib/ql3-management-kubernetes-live-platform.cjs');
+
+test('Run live local manifest replaces only the exact image placeholder', () => {
+  const zeroDigest = 'sha256:' + '0'.repeat(64);
+  const image = 'registry.example.com/qinglong/control';
+  const rendered = [
+    `image: ${image}@${zeroDigest}`,
+    'imagePullPolicy: IfNotPresent',
+    `qinglong.io/client-ca-sha256: ${zeroDigest}`,
+  ].join('\n');
+  const local = localManifest(rendered, image, 'ql3-control-live:test');
+  assert.match(local, /image: ql3-control-live:test/);
+  assert.match(local, /imagePullPolicy: Never/);
+  assert.match(local, new RegExp(`client-ca-sha256: ${zeroDigest}`));
+  assert.doesNotMatch(local, new RegExp(`${image}@${zeroDigest}`));
+  assert.throws(
+    () => localManifest(rendered.replace('@' + zeroDigest, ''), image, 'x:y'),
+    /one reviewed image placeholder/,
+  );
+});
 
 test('Run management live report path is mandatory before mutation begins', () => {
   const script = path.resolve(
@@ -90,6 +112,20 @@ test('Run live runner remains opt-in, layered and observation backed', () => {
   assert.match(source, /reviewedOperatorManifest\(operatorManifestFile\)/);
   assert.match(source, /validateRunManagementKubernetesLiveReport/);
   assert.match(source, /createManagementClientExecutor/);
+  assert.match(source, /QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_REQUEST_FAILED/);
+  assert.match(source, /QL3_RUN_MANAGEMENT_CLIENT_FAILED/);
+  assert.match(source, /const productionRateWindowPattern =/);
+  assert.match(source, /const retryAuthenticationJti =/);
+  assert.match(source, /const stopAuthenticationJti =/);
+  assert.match(
+    source,
+    /const retryBearer = identity\.assertion\(oldKey, retryAuthenticationJti\)/,
+  );
+  assert.equal(
+    source.split('bearer: retryBearer').length - 1,
+    2,
+    'an exact mutation replay must retain its original authentication fence',
+  );
   assert.match(source, /durableRunManagementFacts/);
   assert.match(source, /Run identity ledger rollback surge failure/);
   assert.match(source, /CloudNativePG primary promotion/);