diff --git a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md index eca2b602..0c826302 100644 --- a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md +++ b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md @@ -11,6 +11,19 @@ 最新增量证据(2026-08-13): +- D-303/ADR-0391(已接受) + Cluster operator context 在离线 `validate` 之后增加显式、只读的 `ql3-cluster-admin context probe`。probe 先完整预检全部 + context entry,任何晚出现的配置错误都会在首个网络请求前失败关闭;随后才按固定 catalog 顺序,以 production TLS 1.3、CA、 + servername、mTLS client certificate 或 Kubernetes PortForward 配置逐项 `GET /readyz`。探针不读取 command/assertion,不发送 + Authorization、body 或业务 management POST,不重试、不切换 Pod,响应限 1 KiB,只接受精确 `200/ready` 或 `503/not_ready`。 + 输出仅含 command、transport、ready 状态及 `mutation:false`;配置错误退出 78,不可达/协议错误或任一 not-ready 退出 69。 + TLS preparation 与六类 management route/client-certificate policy 收敛为一个 package-private 真源,既有 mutation client 的公开 + subpath 与请求语义不变。能力仍只存在于短生命周期 Cluster Admin,不新增 package、依赖、binary、listener、timer、controller、 + workload 或 sidecar;Local/Edge、Cluster Control 与 Worker 零导入。Cluster Admin 完整 302 pass/2 条件 skip,18-package clean + build/test 退出 0,backend 1,188 pass/2 skip;五项边界审计零 finding。真实 arm64 Admin image 为 330,487,296 bytes,并在 + `10001:10001`、network none、read-only root、drop ALL、128 MiB/32 PIDs 下完成本地 TLS readiness 契约。14 个 Local Profile + artifact 与 D-302 对应字节数一致;PostgreSQL 18.4 arm64 HA 123 项 gate 全绿、timeline `1→2`,证据 SHA-256 为 + `e7c1743e932f2d7c35dc9153cdf5bc4a03356a38d93fce5507354652aa207a05`,独立审计与 Docker 清理通过。完整验证证据见 ADR-0391。 - D-302/ADR-0390(已接受) Cluster operator context 增加无网络、无 mutation 的内建 `ql3-cluster-admin context validate` 预检。它先复用 owner-private context reader,再让每个 entry 经过与真实请求相同的 production HTTPS/Kubernetes configuration preparation,验证精确 route、hostname、CA、 diff --git a/docs/adr/ADR-0391-read-only-cluster-operator-context-readiness.md b/docs/adr/ADR-0391-read-only-cluster-operator-context-readiness.md new file mode 100644 index 00000000..35fa4f3a --- /dev/null +++ b/docs/adr/ADR-0391-read-only-cluster-operator-context-readiness.md @@ -0,0 +1,53 @@ +# ADR-0391:Read-only Cluster Operator Context Readiness + +- 状态:Accepted +- 日期:2026-08-13 +- 关联 RFC:QL-RFC-0001 D-303 +- 前置决策:ADR-0247、ADR-0250、ADR-0388、ADR-0389、ADR-0390 + +## 上下文 + +ADR-0390 的 `context validate` 能在完全离线状态证明 context、TLS、mTLS 与 Kubernetes 配置可解析,但不能证明发布窗口中的目标进程已就绪、证书链仍被线上端点接受或 Kubernetes tunnel 可建立。用任一 management POST 试探会读取短生命周期 assertion 和 command,并可能进入认证、审计、quota 或 mutation 路径;在工作站另写 curl/kubectl 流程则会绕过生产 client 的 hostname、TLS 1.3、mTLS 与 PortForward 约束。 + +## 决策 + +1. 在既有 Cluster Admin facade 内增加 `context probe --context=/absolute/operator-context.json`。它不是第八个远程业务命令,不新增 binary、package、依赖、常驻进程或部署 authority。 +2. probe 必须先调用完整离线 `context validate`。所有 entry 均通过后才能产生第一个网络请求,避免前项已访问而后项配置错误的部分探测。 +3. 每项只使用与 production client 相同的 config preparation、CA、servername、TLS 1.3 和 mTLS client certificate。Kubernetes 项复用受审 kubeconfig preparation、单个 ready Pod 选择与 PortForward tunnel;不得读取 ambient kubeconfig。 +4. 唯一请求为 `GET /readyz`,无 Authorization、Content-Type 或 body。它不读取 command/assertion,不调用 management route,不查询数据库,不产生 mutation,不重试,也不在连接失败后切换 Pod。 +5. 响应硬限 1 KiB、identity encoding、精确 JSON content type/schema;只接受 `200 {schemaVersion:1,status:"ready"}` 或 `503 {schemaVersion:1,status:"not_ready"}`,拒绝 redirect、额外字段、重复/错误长度、压缩、畸形 UTF-8/JSON 和其他状态。 +6. 输出只包含固定 command 名、`https|kubernetes-port-forward`、`ready|not_ready`、固定 method/path 与 `mutation:false`。不得输出 endpoint、port、namespace、Pod、路径、证书主体或错误对象。全部 ready 退出 0;任一 not-ready、连接或协议失败退出 69;离线配置错误仍退出 78;语法错误退出 64。 +7. management route 与 client-certificate class 只有一个 package-private policy 真源。配置 preparation 和 readiness 实现不从 package manifest 导出;既有公开 client subpath、POST 语义、错误码与调用方保持不变。 +8. 能力只存在于短生命周期 Cluster Admin image。Local/Edge、Cluster Control、Worker 的依赖闭包、文件、RSS、listener、timer、数据库与制品不得变化。 + +## 不采用方案 + +- **直接执行一条 inspect 命令**:仍需 assertion/command,会留下认证或审计事实,且把 readiness 与业务授权混为一体。 +- **通用 URL/方法探针**:扩大 SSRF 与 authority 面,无法约束到受审 endpoint 和固定路径。 +- **自动重试或多 Pod failover**:隐藏单次现场状态、增加窗口时间,并把短生命周期诊断工具变成控制器。 +- **把 probe 放进 Cluster Control 或 sidecar**:会给常驻运行时增加网络、定时器和依赖成本,低配 Local/Edge 也没有该需求。 +- **为 readiness 新建 package**:只有 Cluster Admin 一个制品消费者,拆包会复制 manifest、发布和供应链表面。 + +## 操作顺序 + +1. `ql3-cluster-admin context validate --context=/absolute/operator-context.json` +2. `ql3-cluster-admin context probe --context=/absolute/operator-context.json` +3. 只有前两步符合维护计划,操作者才另行提供精确 command 与短生命周期 assertion 执行业务命令。 + +probe 不是持续监控、负载均衡健康检查或 mutation 成功保证;退出 0 只表示这一时刻全部受审 readiness 端点返回 ready。 + +## 验收门 + +- 无证书 TLS 与 mTLS 真实握手、ready/not-ready、不可达、redirect、畸形与超限响应; +- Kubernetes 单 Pod tunnel、连接丢失时无重试/切换,以及低敏错误映射; +- context 全量离线先验、固定顺序、无 command/assertion/Authorization/body 与输出 secret scan; +- 真实 Admin image 在 non-root、read-only root、network none 容器内启动本地 TLS fixture,证明固定 GET 契约及资源上限; +- Cluster Admin、18-package clean build/test、backend、package/dependency/deployment/image/Local image、14 Local Profile artifact 与 PostgreSQL HA 全量不回归。 + +## 当前证据 + +- readiness/TLS/mTLS/Kubernetes tunnel、context product 与 release 定向契约 93/93。Cluster Admin 完整 304 tests、302 pass/2 条外部服务条件 skip/0 fail;18-package clean build/test 退出 0;backend 1,190 tests、1,188 pass/2 skip/0 fail。 +- workspace 保持 18 package;Cluster Admin 99 source 中 98 nested/1 个既有 root binary,无 single-source/shallow package且未增加依赖或公开 subpath。package/dependency/deployment/image release/Local image 五项边界审计均 compatible、零 finding。 +- 当前源码真实 arm64 Admin image 为 330,487,296 bytes,较 D-302 增加 23,768 bytes;在 `10001:10001`、read-only root、network none、drop ALL、no-new-privileges、0.25 CPU、128 MiB/32 PIDs 下启动同容器 loopback TLS fixture,报告 `contextReadiness=true`,并证明精确 `GET /readyz`、无 Authorization、0-byte body。 +- 14 个 Local Profile artifact 全部 compatible 且与 D-302 字节数一致:最小 Edge 为 2,467,343 bytes/295 files/53 modules,最大 Standalone MCP 为 7,168,978 bytes/778 files/213 modules,证明 Router/Edge 闭包零增量。 +- PostgreSQL 18.4 arm64 HA 123 项 gate 全绿,timeline `1→2`;私有报告 SHA-256 为 `e7c1743e932f2d7c35dc9153cdf5bc4a03356a38d93fce5507354652aa207a05`,独立 evidence audit `compatible=true`、零 finding,测试容器、网络与卷均零残留。 diff --git a/docs/adr/README.md b/docs/adr/README.md index 024800f1..115a2600 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -394,6 +394,7 @@ | [ADR-0388](./ADR-0388-bounded-unified-cluster-product-cli.md) | 有界统一 Cluster Operator 产品 CLI | Accepted | | [ADR-0389](./ADR-0389-owner-private-cluster-operator-context.md) | Owner-private Cluster Operator Context | Accepted | | [ADR-0390](./ADR-0390-offline-cluster-operator-context-preflight.md) | Offline Cluster Operator Context Preflight | Accepted | +| [ADR-0391](./ADR-0391-read-only-cluster-operator-context-readiness.md) | Read-only Cluster Operator Context Readiness | Accepted | ## 规则 diff --git a/docs/operations/ql3-cluster-operator-context.md b/docs/operations/ql3-cluster-operator-context.md new file mode 100644 index 00000000..cb16e6ba --- /dev/null +++ b/docs/operations/ql3-cluster-operator-context.md @@ -0,0 +1,21 @@ +# QingLong 3.0 Cluster Operator Context + +Cluster operator context 只为短生命周期 `ql3-cluster-admin` 复用受审 client 与 Kubernetes 配置路径。它不是 credential store,不得包含 assertion、command、token、private key、endpoint 内容或默认业务操作;context 及引用的私有文件必须是 absolute、canonical、当前 UID 所有、非 symlink 的 `0600` regular file。 + +维护窗口按固定顺序执行: + +```sh +ql3-cluster-admin context validate \ + --context=/secure/qinglong3/operator-context.json + +ql3-cluster-admin context probe \ + --context=/secure/qinglong3/operator-context.json +``` + +`validate` 完全离线,验证所有 context entry、TLS/mTLS 与 Kubernetes 配置,不建立连接。`probe` 会再次先完成全量离线验证,然后按固定 catalog 顺序使用 production TLS 1.3 或 Kubernetes PortForward 发送无 Authorization、无 body 的 `GET /readyz`;它不会执行 management POST、读取 assertion/command、重试或切换 Pod。 + +- `validate` 成功退出 0;配置错误输出低敏 `QL3_CLUSTER_PRODUCT_CONTEXT_INVALID` 并退出 78。 +- `probe` 全部 ready 退出 0;明确 not-ready、不可达或响应协议错误退出 69。not-ready 会输出各 command 的低敏状态;连接/协议失败只输出 `QL3_CLUSTER_PRODUCT_CONTEXT_PROBE_FAILED`。 +- 退出 0 只证明探测时刻 readiness,不代表后续 mutation 一定成功。业务命令仍需单独、显式提供精确 command 与短生命周期 assertion。 + +不要把 probe 放进 cron、sidecar、router/Edge Profile 或 Cluster Control。持续监控应使用独立、最小权限的可观测性面;该命令保留为人工发布/维护窗口的有界诊断工具。 diff --git a/packages/ql3-cluster-admin/src/management-support/managementClientConfiguration.ts b/packages/ql3-cluster-admin/src/management-support/managementClientConfiguration.ts new file mode 100644 index 00000000..1836d0c3 --- /dev/null +++ b/packages/ql3-cluster-admin/src/management-support/managementClientConfiguration.ts @@ -0,0 +1,426 @@ +/** Package-private configuration preparation shared by management clients. */ +import { createPrivateKey, X509Certificate } from 'node:crypto'; +import { + closeSync, + constants, + fstatSync, + lstatSync, + openSync, + readSync, + realpathSync, +} from 'node:fs'; +import { isIP } from 'node:net'; +import { isAbsolute } from 'node:path'; +import { TextDecoder } from 'node:util'; + +export type ClusterAuthenticatedManagementClientKind = + | 'package' + | 'worker-credential' + | 'automation' + | 'approval' + | 'model-credential' + | 'run'; + +const MANAGEMENT_CLIENT_POLICIES: Readonly< + Record< + ClusterAuthenticatedManagementClientKind, + Readonly<{ + managementPath: string; + clientCertificate: 'forbidden' | 'required'; + }> + > +> = Object.freeze({ + package: Object.freeze({ + managementPath: '/api/v3/plugin-packages/management', + clientCertificate: 'forbidden', + }), + 'worker-credential': Object.freeze({ + managementPath: '/api/v3/worker-credentials/management', + clientCertificate: 'required', + }), + automation: Object.freeze({ + managementPath: '/api/v3/automations/management', + clientCertificate: 'required', + }), + approval: Object.freeze({ + managementPath: '/api/v3/approvals/management', + clientCertificate: 'required', + }), + 'model-credential': Object.freeze({ + managementPath: '/api/v3/provider-credentials/management', + clientCertificate: 'required', + }), + run: Object.freeze({ + managementPath: '/api/v3/runs/management', + clientCertificate: 'required', + }), +}); + +const MAXIMUM_CONFIG_BYTES = 16 * 1024; +const MAXIMUM_CA_BYTES = 256 * 1024; +const MAXIMUM_CLIENT_CERTIFICATE_BYTES = 256 * 1024; +const MAXIMUM_CLIENT_PRIVATE_KEY_BYTES = 256 * 1024; +const CONTROL_PATTERN = /[\u0000-\u001f\u007f]/; +const DNS_NAME_PATTERN = + /^(?=.{1,253}$)[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?(?:\.[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?)*$/; + +type JsonObject = Record; + +export interface ClusterAuthenticatedManagementClientConfigurationSummary { + readonly schemaVersion: 1; + readonly managementPath: string; + readonly transport: 'https'; + readonly clientCertificate: 'forbidden' | 'required'; +} + +export interface PreparedClusterAuthenticatedManagementClientConfiguration { + readonly endpoint: URL; + readonly servername: string; + readonly port: number; + readonly requestTimeoutMs: number; + readonly caBytes: Buffer; + readonly clientCertificateBytes?: Buffer; + readonly clientPrivateKeyBytes?: Buffer; + dispose(): void; +} + +export class ClusterPluginPackageManagementClientConfigurationError extends TypeError { + readonly code = 'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_CONFIG_INVALID'; + + constructor() { + super('Plugin Package management client configuration is invalid'); + this.name = 'ClusterPluginPackageManagementClientConfigurationError'; + } +} + +function configurationFailure(): ClusterPluginPackageManagementClientConfigurationError { + return new ClusterPluginPackageManagementClientConfigurationError(); +} + +function exactObject( + value: unknown, + expectedKeys: readonly string[], +): asserts value is JsonObject { + if (!value || typeof value !== 'object' || Array.isArray(value)) { + throw configurationFailure(); + } + const actual = Object.keys(value).sort(); + const expected = [...expectedKeys].sort(); + if ( + actual.length !== expected.length || + actual.some((key, index) => key !== expected[index]) + ) { + throw configurationFailure(); + } +} + +function currentUid(): number { + if (typeof process.getuid !== 'function') throw configurationFailure(); + const uid = process.getuid(); + if (!Number.isSafeInteger(uid) || uid < 0) throw configurationFailure(); + return uid; +} + +export function readCanonicalFile( + filePath: string, + maximumBytes: number, + mode: 'private' | 'public-integrity', +): Buffer { + if ( + typeof filePath !== 'string' || + !isAbsolute(filePath) || + filePath.length > 4_096 || + CONTROL_PATTERN.test(filePath) + ) { + throw configurationFailure(); + } + let before; + try { + before = lstatSync(filePath); + if ( + !before.isFile() || + before.isSymbolicLink() || + before.size < 1 || + before.size > maximumBytes || + realpathSync(filePath) !== filePath + ) { + throw configurationFailure(); + } + } catch (error) { + if (error instanceof ClusterPluginPackageManagementClientConfigurationError) { + throw error; + } + throw configurationFailure(); + } + const uid = currentUid(); + const permissions = before.mode & 0o777; + if ( + (mode === 'private' && (before.uid !== uid || permissions !== 0o600)) || + (mode === 'public-integrity' && before.uid !== uid && before.uid !== 0) || + (mode === 'public-integrity' && (permissions & 0o022) !== 0) + ) { + throw configurationFailure(); + } + + let descriptor = -1; + let bytes: Buffer | undefined; + try { + descriptor = openSync( + filePath, + constants.O_RDONLY | + ((constants as unknown as Readonly>).O_CLOEXEC ?? + 0) | + (constants.O_NOFOLLOW ?? 0), + ); + const opened = fstatSync(descriptor); + if ( + !opened.isFile() || + opened.dev !== before.dev || + opened.ino !== before.ino || + opened.uid !== before.uid || + opened.mode !== before.mode || + opened.size !== before.size + ) { + throw configurationFailure(); + } + bytes = Buffer.alloc(opened.size); + let offset = 0; + while (offset < bytes.length) { + const count = readSync( + descriptor, + bytes, + offset, + bytes.length - offset, + offset, + ); + if (count < 1) throw configurationFailure(); + offset += count; + } + const after = fstatSync(descriptor); + if ( + after.dev !== opened.dev || + after.ino !== opened.ino || + after.uid !== opened.uid || + after.mode !== opened.mode || + after.size !== opened.size + ) { + throw configurationFailure(); + } + return bytes; + } catch (error) { + bytes?.fill(0); + if (error instanceof ClusterPluginPackageManagementClientConfigurationError) { + throw error; + } + throw configurationFailure(); + } finally { + if (descriptor >= 0) closeSync(descriptor); + } +} + +function parseJson(bytes: Buffer): unknown { + try { + return JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(bytes)); + } catch { + throw configurationFailure(); + } +} + +export function isReviewedClusterAuthenticatedManagementClientProtocol( + managementPath: string, + clientCertificate: 'forbidden' | 'required', +): boolean { + return Object.values(MANAGEMENT_CLIENT_POLICIES).some( + (policy) => + policy.managementPath === managementPath && + policy.clientCertificate === clientCertificate, + ); +} + +export function prepareClusterAuthenticatedManagementClientConfiguration( + configFile: string, + managementPath: string, + clientCertificate: 'forbidden' | 'required', +): PreparedClusterAuthenticatedManagementClientConfiguration { + if ( + !isReviewedClusterAuthenticatedManagementClientProtocol( + managementPath, + clientCertificate, + ) + ) { + throw configurationFailure(); + } + let configBytes: Buffer | undefined; + let caBytes: Buffer | undefined; + let clientCertificateBytes: Buffer | undefined; + let clientPrivateKeyBytes: Buffer | undefined; + try { + configBytes = readCanonicalFile( + configFile, + MAXIMUM_CONFIG_BYTES, + 'private', + ); + const config = parseJson(configBytes); + exactObject( + config, + clientCertificate === 'required' + ? [ + 'schemaVersion', + 'endpoint', + 'servername', + 'caFile', + 'clientCertificateFile', + 'clientPrivateKeyFile', + 'requestTimeoutMs', + ] + : [ + 'schemaVersion', + 'endpoint', + 'servername', + 'caFile', + 'requestTimeoutMs', + ], + ); + if ( + config.schemaVersion !== 1 || + typeof config.endpoint !== 'string' || + typeof config.servername !== 'string' || + !DNS_NAME_PATTERN.test(config.servername) || + isIP(config.servername) !== 0 || + typeof config.caFile !== 'string' || + (clientCertificate === 'required' && + (typeof config.clientCertificateFile !== 'string' || + typeof config.clientPrivateKeyFile !== 'string')) || + !Number.isSafeInteger(config.requestTimeoutMs) || + (config.requestTimeoutMs as number) < 1_000 || + (config.requestTimeoutMs as number) > 30_000 + ) { + throw configurationFailure(); + } + const servername = config.servername; + const requestTimeoutMs = config.requestTimeoutMs as number; + let endpoint: URL; + try { + endpoint = new URL(config.endpoint); + } catch { + throw configurationFailure(); + } + if ( + endpoint.protocol !== 'https:' || + endpoint.username !== '' || + endpoint.password !== '' || + endpoint.search !== '' || + endpoint.hash !== '' || + endpoint.pathname !== managementPath || + endpoint.hostname !== servername || + isIP(endpoint.hostname) !== 0 + ) { + throw configurationFailure(); + } + const port = endpoint.port === '' ? 443 : Number(endpoint.port); + if (!Number.isSafeInteger(port) || port < 1 || port > 65_535) { + throw configurationFailure(); + } + caBytes = readCanonicalFile( + config.caFile as string, + MAXIMUM_CA_BYTES, + 'public-integrity', + ); + try { + new X509Certificate(caBytes); + } catch { + throw configurationFailure(); + } + if (clientCertificate === 'required') { + clientCertificateBytes = readCanonicalFile( + config.clientCertificateFile as string, + MAXIMUM_CLIENT_CERTIFICATE_BYTES, + 'public-integrity', + ); + clientPrivateKeyBytes = readCanonicalFile( + config.clientPrivateKeyFile as string, + MAXIMUM_CLIENT_PRIVATE_KEY_BYTES, + 'private', + ); + try { + const certificate = new X509Certificate(clientCertificateBytes); + const privateKey = createPrivateKey(clientPrivateKeyBytes); + if (!certificate.checkPrivateKey(privateKey)) { + throw configurationFailure(); + } + } catch (error) { + if (error instanceof ClusterPluginPackageManagementClientConfigurationError) { + throw error; + } + throw configurationFailure(); + } + } + let disposed = false; + return Object.freeze({ + endpoint, + servername, + port, + requestTimeoutMs, + caBytes, + ...(clientCertificateBytes === undefined + ? {} + : { + clientCertificateBytes, + clientPrivateKeyBytes: clientPrivateKeyBytes!, + }), + dispose() { + if (disposed) return; + disposed = true; + caBytes?.fill(0); + clientCertificateBytes?.fill(0); + clientPrivateKeyBytes?.fill(0); + }, + }); + } catch (error) { + caBytes?.fill(0); + clientCertificateBytes?.fill(0); + clientPrivateKeyBytes?.fill(0); + if (error instanceof ClusterPluginPackageManagementClientConfigurationError) { + throw error; + } + throw configurationFailure(); + } finally { + configBytes?.fill(0); + } +} + +export function prepareClusterAuthenticatedManagementClientKindConfiguration( + configFile: string, + kind: ClusterAuthenticatedManagementClientKind, +): PreparedClusterAuthenticatedManagementClientConfiguration { + const policy = MANAGEMENT_CLIENT_POLICIES[kind]; + if (policy === undefined) throw configurationFailure(); + return prepareClusterAuthenticatedManagementClientConfiguration( + configFile, + policy.managementPath, + policy.clientCertificate, + ); +} + +export function validateClusterAuthenticatedManagementClientConfiguration( + configFile: string, + kind: ClusterAuthenticatedManagementClientKind, +): Readonly { + const policy = MANAGEMENT_CLIENT_POLICIES[kind]; + if (policy === undefined) throw configurationFailure(); + const prepared = + prepareClusterAuthenticatedManagementClientKindConfiguration( + configFile, + kind, + ); + try { + return Object.freeze({ + schemaVersion: 1, + managementPath: policy.managementPath, + transport: 'https', + clientCertificate: policy.clientCertificate, + }); + } finally { + prepared.dispose(); + } +} diff --git a/packages/ql3-cluster-admin/src/management-support/managementReadinessProbe.ts b/packages/ql3-cluster-admin/src/management-support/managementReadinessProbe.ts new file mode 100644 index 00000000..5ffe459d --- /dev/null +++ b/packages/ql3-cluster-admin/src/management-support/managementReadinessProbe.ts @@ -0,0 +1,300 @@ +/** Bounded read-only readiness probe for reviewed cluster management clients. */ +import { Agent as HttpsAgent, request as httpsRequest } from 'node:https'; +import { Duplex } from 'node:stream'; +import { connect as tlsConnect } from 'node:tls'; +import { TextDecoder } from 'node:util'; + +import { + ClusterPluginPackageManagementClientRequestError, + type ClusterAuthenticatedManagementClientKind, + type ClusterPluginPackageManagementClientConnectionOptions, + type ClusterPluginPackageManagementClientRawConnection, +} from './pluginPackageManagementClient'; +import { + ClusterPluginPackageManagementClientConfigurationError, + prepareClusterAuthenticatedManagementClientKindConfiguration, + type PreparedClusterAuthenticatedManagementClientConfiguration, +} from './managementClientConfiguration'; + +const MAXIMUM_RESPONSE_BYTES = 1_024; + +export interface ClusterAuthenticatedManagementClientReadiness { + readonly schemaVersion: 1; + readonly transport: 'https'; + readonly ready: boolean; +} + +function configurationFailure(): ClusterPluginPackageManagementClientConfigurationError { + return new ClusterPluginPackageManagementClientConfigurationError(); +} + +function rawHeaderCount(rawHeaders: readonly string[], name: string): number { + let count = 0; + for (let index = 0; index < rawHeaders.length; index += 2) { + if (rawHeaders[index]?.toLowerCase() === name) count += 1; + } + return count; +} + +function readinessEnvelope( + value: unknown, +): Readonly<{ schemaVersion: unknown; status: unknown }> { + if (!value || typeof value !== 'object' || Array.isArray(value)) { + throw new ClusterPluginPackageManagementClientRequestError(); + } + const keys = Object.keys(value).sort(); + if ( + keys.length !== 2 || + keys[0] !== 'schemaVersion' || + keys[1] !== 'status' + ) { + throw new ClusterPluginPackageManagementClientRequestError(); + } + return value as Readonly<{ schemaVersion: unknown; status: unknown }>; +} + +function connectionOptionsValid( + value: ClusterPluginPackageManagementClientConnectionOptions | undefined, +): boolean { + return ( + value === undefined || + (!!value && + typeof value === 'object' && + !Array.isArray(value) && + Object.keys(value).length === 1 && + typeof value.connect === 'function') + ); +} + +export async function probeClusterAuthenticatedManagementClientReadiness( + configFile: string, + kind: ClusterAuthenticatedManagementClientKind, + connectionOptions?: ClusterPluginPackageManagementClientConnectionOptions, +): Promise> { + if (!connectionOptionsValid(connectionOptions)) throw configurationFailure(); + let prepared: + | PreparedClusterAuthenticatedManagementClientConfiguration + | undefined; + let rawConnection: + | ClusterPluginPackageManagementClientRawConnection + | undefined; + let connectionAgent: HttpsAgent | undefined; + try { + prepared = prepareClusterAuthenticatedManagementClientKindConfiguration( + configFile, + kind, + ); + const { + endpoint, + servername, + port, + requestTimeoutMs, + caBytes, + clientCertificateBytes, + clientPrivateKeyBytes, + } = prepared; + if (connectionOptions) { + rawConnection = await connectionOptions.connect( + Object.freeze({ hostname: endpoint.hostname, port }), + ); + if ( + !rawConnection || + typeof rawConnection !== 'object' || + !(rawConnection.stream instanceof Duplex) || + typeof rawConnection.close !== 'function' + ) { + throw new ClusterPluginPackageManagementClientRequestError(); + } + const establishedConnection = rawConnection; + connectionAgent = new HttpsAgent({ + keepAlive: false, + maxSockets: 1, + maxFreeSockets: 0, + }); + connectionAgent.createConnection = (_options, callback) => { + const socket = tlsConnect({ + socket: establishedConnection.stream, + ca: caBytes, + ...(clientCertificateBytes === undefined + ? {} + : { + cert: clientCertificateBytes, + key: clientPrivateKeyBytes, + }), + servername, + minVersion: 'TLSv1.3', + maxVersion: 'TLSv1.3', + rejectUnauthorized: true, + }); + if (callback) { + let reported = false; + socket.once('secureConnect', () => { + if (reported) return; + reported = true; + callback(null, socket); + }); + socket.once('error', (error) => { + if (reported) return; + reported = true; + callback(error, socket); + }); + } + return socket; + }; + } + return await new Promise((resolve, reject) => { + let settled = false; + const chunks: Buffer[] = []; + let length = 0; + const clearChunks = () => { + for (const chunk of chunks) chunk.fill(0); + }; + const finish = ( + error: unknown, + result?: Readonly, + ) => { + if (settled) return; + settled = true; + if (error) reject(error); + else resolve(result!); + }; + const request = httpsRequest( + { + protocol: 'https:', + hostname: endpoint.hostname, + port, + path: '/readyz', + method: 'GET', + servername, + ca: caBytes, + ...(clientCertificateBytes === undefined + ? {} + : { + cert: clientCertificateBytes, + key: clientPrivateKeyBytes, + }), + minVersion: 'TLSv1.3', + maxVersion: 'TLSv1.3', + rejectUnauthorized: true, + agent: connectionAgent ?? false, + headers: { + accept: 'application/json', + 'accept-encoding': 'identity', + connection: 'close', + }, + }, + (response) => { + response.once('aborted', () => { + clearChunks(); + finish(new ClusterPluginPackageManagementClientRequestError()); + }); + response.once('error', (error) => { + clearChunks(); + finish( + new ClusterPluginPackageManagementClientRequestError(error), + ); + }); + response.on('data', (chunk: Buffer | string) => { + const bytes = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk); + length += bytes.length; + if (length > MAXIMUM_RESPONSE_BYTES) { + const error = + new ClusterPluginPackageManagementClientRequestError(); + bytes.fill(0); + clearChunks(); + response.destroy(); + request.destroy(error); + finish(error); + return; + } + chunks.push(bytes); + }); + response.once('end', () => { + try { + if ( + rawHeaderCount(response.rawHeaders, 'content-type') !== 1 || + response.headers['content-type'] !== + 'application/json; charset=utf-8' || + response.headers['content-encoding'] !== undefined || + rawHeaderCount(response.rawHeaders, 'content-length') > 1 || + (response.headers['content-length'] !== undefined && + (!/^(?:0|[1-9][0-9]*)$/.test( + response.headers['content-length'], + ) || + Number(response.headers['content-length']) !== length)) + ) { + throw new ClusterPluginPackageManagementClientRequestError(); + } + const bytes = Buffer.concat(chunks, length); + let parsed: unknown; + try { + parsed = JSON.parse( + new TextDecoder('utf-8', { fatal: true }).decode(bytes), + ); + } finally { + bytes.fill(0); + clearChunks(); + } + const record = readinessEnvelope(parsed); + const ready = + response.statusCode === 200 && record.status === 'ready'; + const notReady = + response.statusCode === 503 && + record.status === 'not_ready'; + if (record.schemaVersion !== 1 || (!ready && !notReady)) { + throw new ClusterPluginPackageManagementClientRequestError(); + } + finish( + undefined, + Object.freeze({ + schemaVersion: 1, + transport: 'https', + ready, + }), + ); + } catch (error) { + finish( + error instanceof + ClusterPluginPackageManagementClientRequestError + ? error + : new ClusterPluginPackageManagementClientRequestError( + error, + ), + ); + } + }); + }, + ); + request.setTimeout(requestTimeoutMs, () => { + request.destroy( + new ClusterPluginPackageManagementClientRequestError(), + ); + }); + request.once('error', (error) => { + clearChunks(); + finish( + error instanceof ClusterPluginPackageManagementClientRequestError + ? error + : new ClusterPluginPackageManagementClientRequestError(error), + ); + }); + request.end(); + }); + } catch (error) { + if ( + error instanceof ClusterPluginPackageManagementClientConfigurationError || + error instanceof ClusterPluginPackageManagementClientRequestError + ) { + throw error; + } + throw new ClusterPluginPackageManagementClientRequestError(error); + } finally { + connectionAgent?.destroy(); + try { + await rawConnection?.close(); + } catch { + // Probe outcome remains authoritative after bounded resource cleanup. + } + prepared?.dispose(); + } +} diff --git a/packages/ql3-cluster-admin/src/management-support/pluginPackageManagementClient.ts b/packages/ql3-cluster-admin/src/management-support/pluginPackageManagementClient.ts index def78f09..745fb6dd 100644 --- a/packages/ql3-cluster-admin/src/management-support/pluginPackageManagementClient.ts +++ b/packages/ql3-cluster-admin/src/management-support/pluginPackageManagementClient.ts @@ -1,20 +1,28 @@ /** Shared one-shot authenticated client boundary for cluster management planes. */ -import { - closeSync, - constants, - fstatSync, - lstatSync, - openSync, - readSync, - realpathSync, -} from 'node:fs'; import { Agent as HttpsAgent, request as httpsRequest } from 'node:https'; -import { isIP } from 'node:net'; -import { isAbsolute } from 'node:path'; import { Duplex } from 'node:stream'; import { connect as tlsConnect } from 'node:tls'; import { TextDecoder } from 'node:util'; -import { createPrivateKey, X509Certificate } from 'node:crypto'; +import { + ClusterPluginPackageManagementClientConfigurationError, + isReviewedClusterAuthenticatedManagementClientProtocol, + prepareClusterAuthenticatedManagementClientConfiguration, + readCanonicalFile, + validateClusterAuthenticatedManagementClientConfiguration, + type ClusterAuthenticatedManagementClientConfigurationSummary, + type ClusterAuthenticatedManagementClientKind, + type PreparedClusterAuthenticatedManagementClientConfiguration, +} from './managementClientConfiguration'; + +export { + ClusterPluginPackageManagementClientConfigurationError, + readCanonicalFile, + validateClusterAuthenticatedManagementClientConfiguration, +}; +export type { + ClusterAuthenticatedManagementClientConfigurationSummary, + ClusterAuthenticatedManagementClientKind, +}; import { normalizeClusterPluginPackageManagementCommand, @@ -23,59 +31,11 @@ import { } from '../plugin-package/management/pluginPackageManagementTransport'; const MANAGEMENT_PATH = '/api/v3/plugin-packages/management'; -export type ClusterAuthenticatedManagementClientKind = - | 'package' - | 'worker-credential' - | 'automation' - | 'approval' - | 'model-credential' - | 'run'; - -const MANAGEMENT_CLIENT_POLICIES: Readonly< - Record< - ClusterAuthenticatedManagementClientKind, - Readonly<{ - managementPath: string; - clientCertificate: 'forbidden' | 'required'; - }> - > -> = Object.freeze({ - package: Object.freeze({ - managementPath: MANAGEMENT_PATH, - clientCertificate: 'forbidden', - }), - 'worker-credential': Object.freeze({ - managementPath: '/api/v3/worker-credentials/management', - clientCertificate: 'required', - }), - automation: Object.freeze({ - managementPath: '/api/v3/automations/management', - clientCertificate: 'required', - }), - approval: Object.freeze({ - managementPath: '/api/v3/approvals/management', - clientCertificate: 'required', - }), - 'model-credential': Object.freeze({ - managementPath: '/api/v3/provider-credentials/management', - clientCertificate: 'required', - }), - run: Object.freeze({ - managementPath: '/api/v3/runs/management', - clientCertificate: 'required', - }), -}); -const MAX_CONFIG_BYTES = 16 * 1024; const MAX_ASSERTION_BYTES = 16 * 1024; const MAX_COMMAND_BYTES = 256 * 1024; -const MAX_CA_BYTES = 256 * 1024; -const MAX_CLIENT_CERTIFICATE_BYTES = 256 * 1024; -const MAX_CLIENT_PRIVATE_KEY_BYTES = 256 * 1024; const MAX_RESPONSE_BYTES = 128 * 1024; const CONTROL_PATTERN = /[\u0000-\u001f\u007f]/; const PACKAGE_NAME_PATTERN = /^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$/; -const DNS_NAME_PATTERN = - /^(?=.{1,253}$)[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?(?:\.[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?)*$/; const ASSERTION_PATTERN = /^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/; const TOKEN_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$/; const DIGEST_PATTERN = /^[0-9a-f]{64}$/; @@ -123,15 +83,6 @@ export interface ClusterPluginPackageManagementClientConnectionOptions { ): Promise; } -export class ClusterPluginPackageManagementClientConfigurationError extends TypeError { - readonly code = 'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_CONFIG_INVALID'; - - constructor() { - super('Plugin Package management client configuration is invalid'); - this.name = 'ClusterPluginPackageManagementClientConfigurationError'; - } -} - export class ClusterPluginPackageManagementClientRequestError extends Error { readonly code = 'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_REQUEST_FAILED'; @@ -176,114 +127,6 @@ function exactObject( } } -function currentUid(): number { - if (typeof process.getuid !== 'function') throw configurationFailure(); - const uid = process.getuid(); - if (!Number.isSafeInteger(uid) || uid < 0) throw configurationFailure(); - return uid; -} - -export function readCanonicalFile( - filePath: string, - maximumBytes: number, - mode: 'private' | 'public-integrity', -): Buffer { - if ( - typeof filePath !== 'string' || - !isAbsolute(filePath) || - filePath.length > 4_096 || - CONTROL_PATTERN.test(filePath) - ) { - throw configurationFailure(); - } - let before; - try { - before = lstatSync(filePath); - if ( - !before.isFile() || - before.isSymbolicLink() || - before.size < 1 || - before.size > maximumBytes || - realpathSync(filePath) !== filePath - ) { - throw configurationFailure(); - } - } catch (error) { - if ( - error instanceof ClusterPluginPackageManagementClientConfigurationError - ) { - throw error; - } - throw configurationFailure(); - } - const uid = currentUid(); - const permissions = before.mode & 0o777; - if ( - (mode === 'private' && (before.uid !== uid || permissions !== 0o600)) || - (mode === 'public-integrity' && before.uid !== uid && before.uid !== 0) || - (mode === 'public-integrity' && (permissions & 0o022) !== 0) - ) { - throw configurationFailure(); - } - - let descriptor = -1; - let bytes: Buffer | undefined; - try { - descriptor = openSync( - filePath, - constants.O_RDONLY | - ((constants as unknown as Readonly>).O_CLOEXEC ?? - 0) | - (constants.O_NOFOLLOW ?? 0), - ); - const opened = fstatSync(descriptor); - if ( - !opened.isFile() || - opened.dev !== before.dev || - opened.ino !== before.ino || - opened.uid !== before.uid || - opened.mode !== before.mode || - opened.size !== before.size - ) { - throw configurationFailure(); - } - bytes = Buffer.alloc(opened.size); - let offset = 0; - while (offset < bytes.length) { - const count = readSync( - descriptor, - bytes, - offset, - bytes.length - offset, - offset, - ); - if (count < 1) throw configurationFailure(); - offset += count; - } - const after = fstatSync(descriptor); - if ( - after.dev !== opened.dev || - after.ino !== opened.ino || - after.uid !== opened.uid || - after.mode !== opened.mode || - after.size !== opened.size - ) { - throw configurationFailure(); - } - return bytes; - } catch (error) { - bytes?.fill(0); - if ( - error instanceof ClusterPluginPackageManagementClientConfigurationError - ) { - throw error; - } - throw configurationFailure(); - } finally { - if (descriptor >= 0) closeSync(descriptor); - } -} - function decodeUtf8(bytes: Buffer): string { try { return new TextDecoder('utf-8', { fatal: true }).decode(bytes); @@ -812,201 +655,6 @@ function rawHeaderCount(rawHeaders: readonly string[], name: string): number { return count; } -export interface ClusterAuthenticatedManagementClientConfigurationSummary { - readonly schemaVersion: 1; - readonly managementPath: string; - readonly transport: 'https'; - readonly clientCertificate: 'forbidden' | 'required'; -} - -interface PreparedClusterAuthenticatedManagementClientConfiguration { - readonly endpoint: URL; - readonly servername: string; - readonly port: number; - readonly requestTimeoutMs: number; - readonly caBytes: Buffer; - readonly clientCertificateBytes?: Buffer; - readonly clientPrivateKeyBytes?: Buffer; - dispose(): void; -} - -function prepareClusterAuthenticatedManagementClientConfiguration( - configFile: string, - managementPath: string, - clientCertificate: 'forbidden' | 'required', -): PreparedClusterAuthenticatedManagementClientConfiguration { - if ( - !Object.values(MANAGEMENT_CLIENT_POLICIES).some( - (policy) => - policy.managementPath === managementPath && - policy.clientCertificate === clientCertificate, - ) - ) { - throw configurationFailure(); - } - let configBytes: Buffer | undefined; - let caBytes: Buffer | undefined; - let clientCertificateBytes: Buffer | undefined; - let clientPrivateKeyBytes: Buffer | undefined; - try { - configBytes = readCanonicalFile(configFile, MAX_CONFIG_BYTES, 'private'); - const config = parseJson(configBytes); - exactObject( - config, - clientCertificate === 'required' - ? [ - 'schemaVersion', - 'endpoint', - 'servername', - 'caFile', - 'clientCertificateFile', - 'clientPrivateKeyFile', - 'requestTimeoutMs', - ] - : [ - 'schemaVersion', - 'endpoint', - 'servername', - 'caFile', - 'requestTimeoutMs', - ], - ); - if ( - config.schemaVersion !== 1 || - typeof config.endpoint !== 'string' || - typeof config.servername !== 'string' || - !DNS_NAME_PATTERN.test(config.servername) || - isIP(config.servername) !== 0 || - typeof config.caFile !== 'string' || - (clientCertificate === 'required' && - (typeof config.clientCertificateFile !== 'string' || - typeof config.clientPrivateKeyFile !== 'string')) || - !Number.isSafeInteger(config.requestTimeoutMs) || - (config.requestTimeoutMs as number) < 1_000 || - (config.requestTimeoutMs as number) > 30_000 - ) { - throw configurationFailure(); - } - const servername = config.servername; - const requestTimeoutMs = config.requestTimeoutMs as number; - let endpoint: URL; - try { - endpoint = new URL(config.endpoint); - } catch { - throw configurationFailure(); - } - if ( - endpoint.protocol !== 'https:' || - endpoint.username !== '' || - endpoint.password !== '' || - endpoint.search !== '' || - endpoint.hash !== '' || - endpoint.pathname !== managementPath || - endpoint.hostname !== servername || - isIP(endpoint.hostname) !== 0 - ) { - throw configurationFailure(); - } - const port = endpoint.port === '' ? 443 : Number(endpoint.port); - if (!Number.isSafeInteger(port) || port < 1 || port > 65_535) { - throw configurationFailure(); - } - caBytes = readCanonicalFile( - config.caFile as string, - MAX_CA_BYTES, - 'public-integrity', - ); - try { - new X509Certificate(caBytes); - } catch { - throw configurationFailure(); - } - if (clientCertificate === 'required') { - clientCertificateBytes = readCanonicalFile( - config.clientCertificateFile as string, - MAX_CLIENT_CERTIFICATE_BYTES, - 'public-integrity', - ); - clientPrivateKeyBytes = readCanonicalFile( - config.clientPrivateKeyFile as string, - MAX_CLIENT_PRIVATE_KEY_BYTES, - 'private', - ); - try { - const certificate = new X509Certificate(clientCertificateBytes); - const privateKey = createPrivateKey(clientPrivateKeyBytes); - if (!certificate.checkPrivateKey(privateKey)) { - throw configurationFailure(); - } - } catch (error) { - if ( - error instanceof - ClusterPluginPackageManagementClientConfigurationError - ) { - throw error; - } - throw configurationFailure(); - } - } - let disposed = false; - return Object.freeze({ - endpoint, - servername, - port, - requestTimeoutMs, - caBytes, - ...(clientCertificateBytes === undefined - ? {} - : { - clientCertificateBytes, - clientPrivateKeyBytes: clientPrivateKeyBytes!, - }), - dispose() { - if (disposed) return; - disposed = true; - caBytes?.fill(0); - clientCertificateBytes?.fill(0); - clientPrivateKeyBytes?.fill(0); - }, - }); - } catch (error) { - caBytes?.fill(0); - clientCertificateBytes?.fill(0); - clientPrivateKeyBytes?.fill(0); - if ( - error instanceof ClusterPluginPackageManagementClientConfigurationError - ) { - throw error; - } - throw configurationFailure(); - } finally { - configBytes?.fill(0); - } -} - -export function validateClusterAuthenticatedManagementClientConfiguration( - configFile: string, - kind: ClusterAuthenticatedManagementClientKind, -): Readonly { - const policy = MANAGEMENT_CLIENT_POLICIES[kind]; - if (policy === undefined) throw configurationFailure(); - const prepared = prepareClusterAuthenticatedManagementClientConfiguration( - configFile, - policy.managementPath, - policy.clientCertificate, - ); - try { - return Object.freeze({ - schemaVersion: 1, - managementPath: policy.managementPath, - transport: 'https', - clientCertificate: policy.clientCertificate, - }); - } finally { - prepared.dispose(); - } -} - export async function executeClusterAuthenticatedManagementClient< Command, Result, @@ -1030,10 +678,9 @@ export async function executeClusterAuthenticatedManagementClient< 'validateResult', ].includes(key), ) || - !Object.values(MANAGEMENT_CLIENT_POLICIES).some( - (policy) => - policy.managementPath === protocol.managementPath && - policy.clientCertificate === protocol.clientCertificate, + !isReviewedClusterAuthenticatedManagementClientProtocol( + protocol.managementPath, + protocol.clientCertificate, ) || typeof protocol.normalizeCommand !== 'function' || typeof protocol.validateResult !== 'function' || diff --git a/packages/ql3-cluster-admin/src/plugin-package/management/pluginPackageManagementKubernetesClient.ts b/packages/ql3-cluster-admin/src/plugin-package/management/pluginPackageManagementKubernetesClient.ts index 3501e776..dc212838 100644 --- a/packages/ql3-cluster-admin/src/plugin-package/management/pluginPackageManagementKubernetesClient.ts +++ b/packages/ql3-cluster-admin/src/plugin-package/management/pluginPackageManagementKubernetesClient.ts @@ -16,6 +16,10 @@ import { type ClusterPluginPackageManagementClientRawConnection, type ClusterPluginPackageManagementClientResult, } from '../../management-support/pluginPackageManagementClient'; +import { + probeClusterAuthenticatedManagementClientReadiness, + type ClusterAuthenticatedManagementClientReadiness, +} from '../../management-support/managementReadinessProbe'; const MAX_KUBERNETES_CONFIG_BYTES = 16 * 1024; const MAX_KUBECONFIG_BYTES = 256 * 1024; @@ -874,3 +878,94 @@ export async function executeClusterPluginPackageManagementKubernetesClient( prepared?.dispose(); } } + +export async function probeClusterPluginPackageManagementKubernetesReadiness( + configFile: string, + kubernetesFile: string, + options: ClusterPluginPackageManagementKubernetesClientOptions = {}, +): Promise> { + if ( + !options || + typeof options !== 'object' || + Array.isArray(options) || + Object.keys(options).some((key) => key !== 'createRuntime') || + (options.createRuntime !== undefined && + typeof options.createRuntime !== 'function') + ) { + throw configurationFailure(); + } + let prepared: PreparedKubernetesClientConfiguration | undefined; + try { + prepared = await prepareKubernetesClientConfiguration(kubernetesFile); + const { config, kubeConfig, kubernetes } = prepared; + const runtime = (options.createRuntime ?? productionRuntime)( + kubeConfig, + kubernetes, + ); + if ( + !runtime || + typeof runtime !== 'object' || + typeof runtime.pods?.listNamespacedPod !== 'function' || + typeof runtime.openPortForward !== 'function' + ) { + throw configurationFailure(); + } + const expectedHostname = `${MANAGEMENT_NAME}.${config.namespace}.svc`; + return await probeClusterAuthenticatedManagementClientReadiness( + configFile, + 'package', + { + async connect(target) { + if ( + target.hostname !== expectedHostname || + target.port !== MANAGEMENT_PORT + ) { + throw configurationFailure(); + } + const list = await deadline( + runtime.pods.listNamespacedPod({ + namespace: config.namespace, + labelSelector: MANAGEMENT_LABEL_SELECTOR, + limit: 3, + timeoutSeconds: Math.ceil(config.apiTimeoutMs / 1_000), + watch: false, + }), + config.apiTimeoutMs, + ); + const podName = selectManagementPod(list, config.namespace); + return await deadline( + runtime.openPortForward({ + namespace: config.namespace, + podName, + port: MANAGEMENT_PORT, + }), + config.apiTimeoutMs, + async (connection) => { + await connection.close(); + }, + ); + }, + }, + ); + } catch (error) { + if ( + error instanceof ClusterPluginPackageManagementClientRequestError && + error.cause instanceof + ClusterPluginPackageManagementKubernetesClientTunnelError + ) { + throw error.cause; + } + if ( + error instanceof + ClusterPluginPackageManagementKubernetesClientConfigurationError || + error instanceof ClusterPluginPackageManagementKubernetesClientTunnelError || + error instanceof ClusterPluginPackageManagementClientConfigurationError || + error instanceof ClusterPluginPackageManagementClientRequestError + ) { + throw error; + } + throw new ClusterPluginPackageManagementKubernetesClientTunnelError(error); + } finally { + prepared?.dispose(); + } +} diff --git a/packages/ql3-cluster-admin/src/product-cli/cli.ts b/packages/ql3-cluster-admin/src/product-cli/cli.ts index ae55a299..d964d3c3 100644 --- a/packages/ql3-cluster-admin/src/product-cli/cli.ts +++ b/packages/ql3-cluster-admin/src/product-cli/cli.ts @@ -4,8 +4,12 @@ import { spawn, type ChildProcess } from 'node:child_process'; import { constants } from 'node:os'; import { resolveQingLong3ClusterProductCommand } from './productCommand'; -import { QingLong3ClusterProductContextError } from './productContext'; -import { validateQingLong3ClusterProductContext } from './productContext'; +import { + probeQingLong3ClusterProductContext, + QingLong3ClusterProductContextError, + QingLong3ClusterProductContextProbeError, + validateQingLong3ClusterProductContext, +} from './productContext'; const FORWARDED_SIGNALS = Object.freeze([ 'SIGINT', @@ -132,6 +136,14 @@ async function main(argv: readonly string[]): Promise { process.stdout.write(`${JSON.stringify(result)}\n`); return; } + if (resolution.kind === 'context-probe') { + const result = await probeQingLong3ClusterProductContext( + resolution.contextFile, + ); + process.stdout.write(`${JSON.stringify(result)}\n`); + if (!result.allReady) process.exitCode = 69; + return; + } invoke(resolution.targetFilePath, resolution.argv); } catch (error) { if ( @@ -152,6 +164,24 @@ async function main(argv: readonly string[]): Promise { process.exitCode = 78; return; } + if ( + error instanceof QingLong3ClusterProductContextProbeError || + (error !== null && + typeof error === 'object' && + 'code' in error && + error.code === 'QL3_CLUSTER_PRODUCT_CONTEXT_PROBE_FAILED') + ) { + process.stderr.write( + `${JSON.stringify( + lowSensitivityFailure( + 'QL3_CLUSTER_PRODUCT_CONTEXT_PROBE_FAILED', + 'QingLong 3.0 Cluster operator context probe failed', + ), + )}\n`, + ); + process.exitCode = 69; + return; + } process.stderr.write( `${JSON.stringify( lowSensitivityFailure( diff --git a/packages/ql3-cluster-admin/src/product-cli/productCommand.ts b/packages/ql3-cluster-admin/src/product-cli/productCommand.ts index a86d6054..dbfffc62 100644 --- a/packages/ql3-cluster-admin/src/product-cli/productCommand.ts +++ b/packages/ql3-cluster-admin/src/product-cli/productCommand.ts @@ -14,6 +14,7 @@ export type QingLong3ClusterProductCommandResolution = | Readonly<{ kind: 'help'; output: string }> | Readonly<{ kind: 'version'; output: string }> | Readonly<{ kind: 'context-validation'; contextFile: string }> + | Readonly<{ kind: 'context-probe'; contextFile: string }> | Readonly<{ kind: 'invoke'; command: QingLong3ClusterProductCommandDefinition; @@ -175,6 +176,7 @@ export function qingLong3ClusterProductHelp(): string { '', 'Local operator commands:', ' context validate --context=/absolute/operator-context.json', + ' context probe --context=/absolute/operator-context.json', '', 'Use `ql3-cluster-admin --help` for command-specific usage.', 'Use `--context=/absolute/operator-context.json` to inject only stable client paths.', @@ -200,7 +202,7 @@ export function resolveQingLong3ClusterProductCommand( if (argv[0] === 'context') { if ( argv.length !== 3 || - argv[1] !== 'validate' || + (argv[1] !== 'validate' && argv[1] !== 'probe') || !argv[2]!.startsWith('--context=') || argv[2] === '--context=' ) { @@ -215,7 +217,7 @@ export function resolveQingLong3ClusterProductCommand( resolveInstalledTarget(distRoot, definition); } return Object.freeze({ - kind: 'context-validation', + kind: argv[1] === 'validate' ? 'context-validation' : 'context-probe', contextFile: argv[2]!.slice('--context='.length), }); } diff --git a/packages/ql3-cluster-admin/src/product-cli/productContext.ts b/packages/ql3-cluster-admin/src/product-cli/productContext.ts index 3e902e85..8faebaf9 100644 --- a/packages/ql3-cluster-admin/src/product-cli/productContext.ts +++ b/packages/ql3-cluster-admin/src/product-cli/productContext.ts @@ -10,9 +10,15 @@ import { import { isAbsolute } from 'node:path'; import { TextDecoder } from 'node:util'; -import { validateClusterAuthenticatedManagementClientConfiguration } from '../management-support/pluginPackageManagementClient'; +import { + validateClusterAuthenticatedManagementClientConfiguration, +} from '../management-support/pluginPackageManagementClient'; import type { ClusterAuthenticatedManagementClientKind } from '../management-support/pluginPackageManagementClient'; -import { validateClusterPluginPackageManagementKubernetesConfiguration } from '../plugin-package/management/pluginPackageManagementKubernetesClient'; +import { probeClusterAuthenticatedManagementClientReadiness } from '../management-support/managementReadinessProbe'; +import { + probeClusterPluginPackageManagementKubernetesReadiness, + validateClusterPluginPackageManagementKubernetesConfiguration, +} from '../plugin-package/management/pluginPackageManagementKubernetesClient'; const MAXIMUM_CONTEXT_BYTES = 64 * 1024; const MAXIMUM_PATH_BYTES = 4_096; @@ -57,6 +63,22 @@ export interface QingLong3ClusterProductContextValidation { readonly mutation: false; } +export interface QingLong3ClusterProductContextProbe { + readonly schemaVersion: 1; + readonly component: 'qinglong3-cluster-product-cli'; + readonly event: 'context_probed'; + readonly commandCount: number; + readonly commands: readonly Readonly<{ + name: ContextCommandName; + transport: 'https' | 'kubernetes-port-forward'; + status: 'ready' | 'not_ready'; + }>[]; + readonly allReady: boolean; + readonly requestMethod: 'GET'; + readonly requestPath: '/readyz'; + readonly mutation: false; +} + const CONTEXT_COMMAND_CLIENT_KINDS: Readonly< Record > = Object.freeze({ @@ -78,6 +100,15 @@ export class QingLong3ClusterProductContextError extends TypeError { } } +export class QingLong3ClusterProductContextProbeError extends Error { + readonly code = 'QL3_CLUSTER_PRODUCT_CONTEXT_PROBE_FAILED'; + + constructor() { + super('QingLong 3.0 Cluster operator context probe failed'); + this.name = 'QingLong3ClusterProductContextProbeError'; + } +} + function invalid(): never { throw new QingLong3ClusterProductContextError(); } @@ -328,3 +359,62 @@ export async function validateQingLong3ClusterProductContext( throw new QingLong3ClusterProductContextError(); } } + +export async function probeQingLong3ClusterProductContext( + contextFile: string, +): Promise> { + try { + await validateQingLong3ClusterProductContext(contextFile); + const context = loadQingLong3ClusterProductContext(contextFile); + const commands: Array< + QingLong3ClusterProductContextProbe['commands'][number] + > = []; + for (const name of CONTEXT_COMMANDS) { + const command = context.commands[name]; + if (command === undefined) continue; + const result = + name === 'package-kubernetes' + ? await probeClusterPluginPackageManagementKubernetesReadiness( + command.configFile, + command.kubernetesFile!, + ) + : await probeClusterAuthenticatedManagementClientReadiness( + command.configFile, + CONTEXT_COMMAND_CLIENT_KINDS[name], + ); + commands.push( + Object.freeze({ + name, + transport: + name === 'package-kubernetes' + ? 'kubernetes-port-forward' + : result.transport, + status: result.ready ? 'ready' : 'not_ready', + }), + ); + } + return Object.freeze({ + schemaVersion: 1, + component: 'qinglong3-cluster-product-cli', + event: 'context_probed', + commandCount: commands.length, + commands: Object.freeze(commands), + allReady: commands.every(({ status }) => status === 'ready'), + requestMethod: 'GET', + requestPath: '/readyz', + mutation: false, + }); + } catch (error) { + if (error instanceof QingLong3ClusterProductContextError) throw error; + if ( + error !== null && + typeof error === 'object' && + 'code' in error && + typeof error.code === 'string' && + error.code.endsWith('CONFIG_INVALID') + ) { + throw new QingLong3ClusterProductContextError(); + } + throw new QingLong3ClusterProductContextProbeError(); + } +} diff --git a/packages/ql3-cluster-admin/test/pluginPackageManagementClient.test.cjs b/packages/ql3-cluster-admin/test/pluginPackageManagementClient.test.cjs index d7f6237e..681a1ef5 100644 --- a/packages/ql3-cluster-admin/test/pluginPackageManagementClient.test.cjs +++ b/packages/ql3-cluster-admin/test/pluginPackageManagementClient.test.cjs @@ -20,6 +20,10 @@ const { ClusterPluginPackageManagementClientRequestError, executeClusterPluginPackageManagementClient, } = require('@qinglong/cluster-admin/plugin-package-management-client'); +const publicClientModule = require('@qinglong/cluster-admin/plugin-package-management-client'); +const { + probeClusterAuthenticatedManagementClientReadiness, +} = require('../dist/management-support/managementReadinessProbe.js'); const CA_CERT = resolve( __dirname, @@ -33,12 +37,35 @@ const SERVER_CERT = resolve( __dirname, '../../ql3-cluster-control/test/fixtures/mtls/server-cert.pem', ); +const CLIENT_KEY = resolve( + __dirname, + '../../ql3-cluster-control/test/fixtures/mtls/client-key.pem', +); +const CLIENT_CERT = resolve( + __dirname, + '../../ql3-cluster-control/test/fixtures/mtls/client-cert.pem', +); const CLIENT_CLI = resolve( __dirname, '../dist/plugin-package/management/pluginPackageManagementClientCli.js', ); const ASSERTION = 'eyJhbGciOiJFUzI1NiJ9.eyJzdWIiOiJvcGVyYXRvciJ9.c2ln'; +test('keeps owned TLS preparation out of the public client subpath', () => { + assert.equal( + publicClientModule.prepareClusterAuthenticatedManagementClientConfiguration, + undefined, + ); + assert.equal( + publicClientModule.prepareClusterAuthenticatedManagementClientKindConfiguration, + undefined, + ); + assert.equal( + publicClientModule.probeClusterAuthenticatedManagementClientReadiness, + undefined, + ); +}); + function inspectCommand(operation = 'plugin-package.inspect') { return { schemaVersion: 1, @@ -392,13 +419,14 @@ function createClientFiles(port, command = inspectCommand()) { }; } -async function startServer(handler) { +async function startServer(handler, options = {}) { const server = createServer( { key: readFileSync(SERVER_KEY), cert: readFileSync(SERVER_CERT), minVersion: 'TLSv1.3', maxVersion: 'TLSv1.3', + ...options, }, handler, ); @@ -475,6 +503,142 @@ test('sends one TLS 1.3 management command and validates the low-sensitive resul } }); +test('probes only the fixed TLS readiness endpoint without management authority', async () => { + const received = []; + let ready = true; + const fixture = await startServer((request, response) => { + const chunks = []; + request.on('data', (chunk) => chunks.push(chunk)); + request.once('end', () => { + received.push({ + method: request.method, + path: request.url, + authorization: request.headers.authorization, + contentType: request.headers['content-type'], + bodyBytes: Buffer.concat(chunks).length, + protocol: request.socket.getProtocol(), + }); + sendJson(response, ready ? 200 : 503, { + schemaVersion: 1, + status: ready ? 'ready' : 'not_ready', + }); + }); + }); + const files = createClientFiles(fixture.port); + try { + assert.deepEqual( + await probeClusterAuthenticatedManagementClientReadiness( + files.paths.configFile, + 'package', + ), + { schemaVersion: 1, transport: 'https', ready: true }, + ); + ready = false; + assert.deepEqual( + await probeClusterAuthenticatedManagementClientReadiness( + files.paths.configFile, + 'package', + ), + { schemaVersion: 1, transport: 'https', ready: false }, + ); + assert.deepEqual(received, [ + { + method: 'GET', + path: '/readyz', + authorization: undefined, + contentType: undefined, + bodyBytes: 0, + protocol: 'TLSv1.3', + }, + { + method: 'GET', + path: '/readyz', + authorization: undefined, + contentType: undefined, + bodyBytes: 0, + protocol: 'TLSv1.3', + }, + ]); + } finally { + await fixture.close(); + rmSync(files.directory, { recursive: true, force: true }); + } +}); + +test('presents the reviewed client certificate for mTLS readiness', async () => { + let authorized = false; + const fixture = await startServer( + (request, response) => { + authorized = request.socket.authorized; + sendJson(response, 200, { schemaVersion: 1, status: 'ready' }); + }, + { + ca: readFileSync(CA_CERT), + requestCert: true, + rejectUnauthorized: true, + }, + ); + const files = createClientFiles(fixture.port); + const clientKeyFile = join(files.directory, 'client-key.pem'); + privateWrite(clientKeyFile, readFileSync(CLIENT_KEY, 'utf8')); + privateWrite(files.paths.configFile, { + schemaVersion: 1, + endpoint: `https://localhost:${fixture.port}/api/v3/runs/management`, + servername: 'localhost', + caFile: CA_CERT, + clientCertificateFile: CLIENT_CERT, + clientPrivateKeyFile: clientKeyFile, + requestTimeoutMs: 1_000, + }); + try { + assert.deepEqual( + await probeClusterAuthenticatedManagementClientReadiness( + files.paths.configFile, + 'run', + ), + { schemaVersion: 1, transport: 'https', ready: true }, + ); + assert.equal(authorized, true); + } finally { + await fixture.close(); + rmSync(files.directory, { recursive: true, force: true }); + } +}); + +test('readiness probe rejects unreviewed status and bounded response drift', async () => { + let behavior = 'wrong-status'; + const fixture = await startServer((_request, response) => { + if (behavior === 'wrong-status') { + sendJson(response, 200, { schemaVersion: 1, status: 'live' }); + return; + } + if (behavior === 'redirect') { + sendJson(response, 302, { schemaVersion: 1, status: 'ready' }); + return; + } + response.writeHead(200, { + 'content-type': 'application/json; charset=utf-8', + }); + response.end(Buffer.alloc(1_025, 0x61)); + }); + const files = createClientFiles(fixture.port); + try { + for (const next of ['wrong-status', 'redirect', 'oversized']) { + behavior = next; + await assert.rejects( + probeClusterAuthenticatedManagementClientReadiness( + files.paths.configFile, + 'package', + ), + ClusterPluginPackageManagementClientRequestError, + ); + } + } finally { + await fixture.close(); + rmSync(files.directory, { recursive: true, force: true }); + } +}); + test('permits and validates exactly the fourteen public management operations', async () => { const received = []; const fixture = await startServer((request, response) => { diff --git a/packages/ql3-cluster-admin/test/pluginPackageManagementKubernetesClient.test.cjs b/packages/ql3-cluster-admin/test/pluginPackageManagementKubernetesClient.test.cjs index c03ef61e..5ed65003 100644 --- a/packages/ql3-cluster-admin/test/pluginPackageManagementKubernetesClient.test.cjs +++ b/packages/ql3-cluster-admin/test/pluginPackageManagementKubernetesClient.test.cjs @@ -20,6 +20,7 @@ const { ClusterPluginPackageManagementKubernetesClientTunnelError, executeClusterPluginPackageManagementKubernetesClient, openClusterPluginPackageManagementPortForward, + probeClusterPluginPackageManagementKubernetesReadiness, } = require( '@qinglong/cluster-admin/plugin-package-management-kubernetes-client' ); @@ -302,6 +303,85 @@ test('uses one ready Pod tunnel and preserves end-to-end TLS 1.3 hostname verifi } }); +test('probes readiness through one reviewed Pod tunnel without an assertion or command', async () => { + const requests = []; + const server = await startServer((request, response) => { + const chunks = []; + request.on('data', (chunk) => chunks.push(chunk)); + request.once('end', () => { + requests.push({ + method: request.method, + path: request.url, + authorization: request.headers.authorization, + bodyBytes: Buffer.concat(chunks).length, + protocol: request.socket.getProtocol(), + servername: request.socket.servername, + }); + sendJson(response, { schemaVersion: 1, status: 'ready' }); + }); + }); + const files = createClientFiles(); + const calls = { lists: 0, tunnels: 0, closes: 0 }; + try { + const result = + await probeClusterPluginPackageManagementKubernetesReadiness( + files.paths.configFile, + files.paths.kubernetesFile, + { + createRuntime() { + return { + pods: { + async listNamespacedPod() { + calls.lists += 1; + return { + items: [ + readyPod( + 'ql3-plugin-package-management-aaaaa-11111', + ), + ], + }; + }, + }, + async openPortForward() { + calls.tunnels += 1; + const stream = connectTcp({ + host: '127.0.0.1', + port: server.port, + }); + return { + stream, + close() { + calls.closes += 1; + stream.end(); + }, + }; + }, + }; + }, + }, + ); + assert.deepEqual(result, { + schemaVersion: 1, + transport: 'https', + ready: true, + }); + assert.deepEqual(calls, { lists: 1, tunnels: 1, closes: 1 }); + assert.deepEqual(requests, [ + { + method: 'GET', + path: '/readyz', + authorization: undefined, + bodyBytes: 0, + protocol: 'TLSv1.3', + servername: SERVICE_HOST, + }, + ]); + } finally { + await server.close(); + rmSync(files.directory, { recursive: true, force: true }); + } +}); + test('rejects ambient, executable, proxied, insecure, and file-backed kubeconfig authority', async () => { const cases = [ kubeconfig({ cluster: { 'insecure-skip-tls-verify': true } }), diff --git a/packages/ql3-cluster-admin/test/productCli.test.cjs b/packages/ql3-cluster-admin/test/productCli.test.cjs index 88298b17..fd61d57c 100644 --- a/packages/ql3-cluster-admin/test/productCli.test.cjs +++ b/packages/ql3-cluster-admin/test/productCli.test.cjs @@ -1,10 +1,11 @@ const assert = require('node:assert/strict'); -const { spawnSync } = require('node:child_process'); +const { spawn, spawnSync } = require('node:child_process'); const { EventEmitter } = require('node:events'); const fs = require('node:fs'); const os = require('node:os'); const path = require('node:path'); const { test } = require('node:test'); +const { createServer } = require('node:https'); const packageRoot = path.resolve(__dirname, '..'); const moduleDirectory = path.join(packageRoot, 'dist', 'product-cli'); @@ -21,6 +22,18 @@ const privateKeyFixture = path.join( 'fixtures', 'management-service-key.pem', ); +const localhostCertificateFixture = path.resolve( + packageRoot, + '../ql3-cluster-control/test/fixtures/mtls/server-cert.pem', +); +const localhostCaFixture = path.resolve( + packageRoot, + '../ql3-cluster-control/test/fixtures/mtls/ca-cert.pem', +); +const localhostPrivateKeyFixture = path.resolve( + packageRoot, + '../ql3-cluster-control/test/fixtures/mtls/server-key.pem', +); const manifest = JSON.parse( fs.readFileSync(path.join(packageRoot, 'package.json'), 'utf8'), ); @@ -49,6 +62,65 @@ function runCli(args) { }); } +function runCliAsync(args) { + return new Promise((resolvePromise, reject) => { + const child = spawn(process.execPath, [cliPath, ...args], { + cwd: packageRoot, + stdio: ['ignore', 'pipe', 'pipe'], + }); + const stdout = []; + const stderr = []; + child.stdout.on('data', (chunk) => stdout.push(chunk)); + child.stderr.on('data', (chunk) => stderr.push(chunk)); + child.once('error', reject); + child.once('close', (status, signal) => { + resolvePromise({ + status, + signal, + stdout: Buffer.concat(stdout).toString('utf8'), + stderr: Buffer.concat(stderr).toString('utf8'), + }); + }); + }); +} + +async function startReadinessServer(status) { + const server = createServer( + { + key: fs.readFileSync(localhostPrivateKeyFixture), + cert: fs.readFileSync(localhostCertificateFixture), + minVersion: 'TLSv1.3', + maxVersion: 'TLSv1.3', + }, + (request, response) => { + assert.equal(request.method, 'GET'); + assert.equal(request.url, '/readyz'); + assert.equal(request.headers.authorization, undefined); + const body = Buffer.from( + JSON.stringify({ schemaVersion: 1, status: status.value }), + ); + response.writeHead(status.value === 'ready' ? 200 : 503, { + 'content-type': 'application/json; charset=utf-8', + 'content-length': String(body.length), + }); + response.end(body); + }, + ); + await new Promise((resolvePromise, reject) => { + server.once('error', reject); + server.listen(0, '127.0.0.1', resolvePromise); + }); + return { + port: server.address().port, + close: () => + new Promise((resolvePromise, reject) => { + server.close((error) => + error ? reject(error) : resolvePromise(), + ); + }), + }; +} + function privateFile(directory, name, contents) { const filePath = path.join(directory, name); fs.writeFileSync(filePath, contents, { mode: 0o600 }); @@ -521,6 +593,84 @@ test('validates the complete operator context offline without operational author assert.equal(validated.stdout.includes('bounded-token'), false); }); +test('probes a context with fixed read-only readiness semantics and exit status', async (t) => { + const status = { value: 'ready' }; + const server = await startReadinessServer(status); + t.after(() => server.close()); + const directory = fs.realpathSync( + fs.mkdtempSync(path.join(os.tmpdir(), 'ql3-cluster-probe-context-')), + ); + t.after(() => fs.rmSync(directory, { recursive: true, force: true })); + const caFile = privateFile( + directory, + 'ca.pem', + fs.readFileSync(localhostCaFixture), + ); + const configFile = privateFile( + directory, + 'package.json', + JSON.stringify({ + schemaVersion: 1, + endpoint: `https://localhost:${server.port}/api/v3/plugin-packages/management`, + servername: 'localhost', + caFile, + requestTimeoutMs: 1_000, + }), + ); + const contextFile = privateFile( + directory, + 'operator-context.json', + JSON.stringify({ + schemaVersion: 1, + commands: { package: { configFile } }, + }), + ); + assert.deepEqual( + resolveQingLong3ClusterProductCommand( + ['context', 'probe', `--context=${contextFile}`], + moduleDirectory, + ), + { kind: 'context-probe', contextFile }, + ); + + const ready = await runCliAsync([ + 'context', + 'probe', + `--context=${contextFile}`, + ]); + assert.equal( + ready.status, + 0, + JSON.stringify({ stdout: ready.stdout, stderr: ready.stderr }), + ); + assert.equal(ready.stderr, ''); + assert.deepEqual(JSON.parse(ready.stdout), { + schemaVersion: 1, + component: 'qinglong3-cluster-product-cli', + event: 'context_probed', + commandCount: 1, + commands: [{ name: 'package', transport: 'https', status: 'ready' }], + allReady: true, + requestMethod: 'GET', + requestPath: '/readyz', + mutation: false, + }); + assert.equal(ready.stdout.includes(directory), false); + assert.equal(ready.stdout.includes('localhost'), false); + + status.value = 'not_ready'; + const notReady = await runCliAsync([ + 'context', + 'probe', + `--context=${contextFile}`, + ]); + assert.equal(notReady.status, 69); + assert.equal(notReady.stderr, ''); + const fact = JSON.parse(notReady.stdout); + assert.equal(fact.allReady, false); + assert.equal(fact.commands[0].status, 'not_ready'); +}); + test('context validation fails closed for invalid client configuration and syntax', (t) => { const fixture = validContextFixture(t); fs.writeFileSync(fixture.commands.run.configFile, '{}', { mode: 0o600 }); @@ -544,6 +694,8 @@ test('context validation fails closed for invalid client configuration and synta ['context', 'validate'], ['context', 'validate', '--context'], ['context', 'validate', '--context='], + ['context', 'probe'], + ['context', 'probe', '--context='], ['context', 'inspect', `--context=${fixture.contextFile}`], ]) { const rejected = resolveQingLong3ClusterProductCommand( diff --git a/scripts/ql3-cluster-admin-product-live-contract.cjs b/scripts/ql3-cluster-admin-product-live-contract.cjs index 244da915..e806cf66 100644 --- a/scripts/ql3-cluster-admin-product-live-contract.cjs +++ b/scripts/ql3-cluster-admin-product-live-contract.cjs @@ -1,6 +1,7 @@ 'use strict'; const { execFileSync } = require('node:child_process'); +const { resolve } = require('node:path'); const IMAGE_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._/:@-]{0,255}$/u; const ENTRYPOINT = [ @@ -77,9 +78,29 @@ function runImage(image, args) { } function runOperatorContextContract(image) { + const fixtureRoot = resolve( + __dirname, + '../packages/ql3-cluster-control/test/fixtures/mtls', + ); + const readinessServerSource = String.raw` +const { readFileSync, writeFileSync } = require('node:fs'); +const { createServer } = require('node:https'); +const server = createServer({ key: readFileSync('/evidence/server-key.pem'), cert: readFileSync('/evidence/server-cert.pem'), minVersion: 'TLSv1.3', maxVersion: 'TLSv1.3' }, (request, response) => { + const chunks = []; + request.on('data', (chunk) => chunks.push(chunk)); + request.once('end', () => { + writeFileSync('/tmp/readiness-observation.json', JSON.stringify({ method: request.method, path: request.url, authorization: request.headers.authorization ?? null, bodyBytes: Buffer.concat(chunks).length }), { mode: 0o600 }); + const body = Buffer.from(JSON.stringify({ schemaVersion: 1, status: 'ready' })); + response.writeHead(200, { 'content-type': 'application/json; charset=utf-8', 'content-length': String(body.length) }); + response.end(body); + }); +}); +server.listen(0, '127.0.0.1', () => writeFileSync('/tmp/readiness-port', String(server.address().port), { mode: 0o600 })); +process.once('SIGTERM', () => server.close(() => process.exit(0))); +`; const source = String.raw` -const { spawnSync } = require('node:child_process'); -const { writeFileSync } = require('node:fs'); +const { spawn, spawnSync } = require('node:child_process'); +const { existsSync, readFileSync, writeFileSync } = require('node:fs'); const { rootCertificates } = require('node:tls'); const facade = '/opt/qinglong/node_modules/@qinglong/cluster-admin/dist/product-cli/cli.js'; const config = '/tmp/run-client.json'; @@ -104,12 +125,27 @@ const validated = spawnSync(process.execPath, [facade, 'context', 'validate', '- let validationFact; try { validationFact = JSON.parse(validated.stdout); } catch { process.exit(25); } if (validated.status !== 0 || validated.stderr !== '' || validationFact.event !== 'context_valid' || validationFact.commandCount !== 1 || validationFact.networkAccess !== false || validationFact.mutation !== false || JSON.stringify(validationFact.commands) !== JSON.stringify([{ name: 'package', transport: 'https', clientCertificate: 'forbidden' }]) || validated.stdout.includes('/tmp/') || validated.stdout.includes('manager.example.test')) process.exit(26); +const readinessServer = spawn(process.execPath, ['-e', ${JSON.stringify(readinessServerSource)}], { stdio: 'ignore' }); +const waitArray = new Int32Array(new SharedArrayBuffer(4)); +for (let attempt = 0; attempt < 200 && !existsSync('/tmp/readiness-port'); attempt += 1) Atomics.wait(waitArray, 0, 0, 10); +if (!existsSync('/tmp/readiness-port')) process.exit(27); +const readinessPort = Number(readFileSync('/tmp/readiness-port', 'utf8')); +writeFileSync(ca, readFileSync('/evidence/ca-cert.pem'), { mode: 0o600 }); +writeFileSync(config, JSON.stringify({ schemaVersion: 1, endpoint: 'https://localhost:' + readinessPort + '/api/v3/plugin-packages/management', servername: 'localhost', caFile: ca, requestTimeoutMs: 1000 }), { mode: 0o600 }); +const probed = spawnSync(process.execPath, [facade, 'context', 'probe', '--context=' + context], { encoding: 'utf8' }); +readinessServer.kill('SIGTERM'); +let probeFact; +try { probeFact = JSON.parse(probed.stdout); } catch { process.exit(28); } +if (probed.status !== 0 || probed.stderr !== '' || probeFact.event !== 'context_probed' || probeFact.commandCount !== 1 || probeFact.allReady !== true || probeFact.requestMethod !== 'GET' || probeFact.requestPath !== '/readyz' || probeFact.mutation !== false || JSON.stringify(probeFact.commands) !== JSON.stringify([{ name: 'package', transport: 'https', status: 'ready' }]) || probed.stdout.includes('/tmp/') || probed.stdout.includes('localhost')) process.exit(29); +let readinessObservation; +try { readinessObservation = JSON.parse(readFileSync('/tmp/readiness-observation.json', 'utf8')); } catch { process.exit(30); } +if (JSON.stringify(readinessObservation) !== JSON.stringify({ method: 'GET', path: '/readyz', authorization: null, bodyBytes: 0 })) process.exit(31); writeFileSync(context, JSON.stringify({ schemaVersion: 1, commands: { run: { configFile: config, assertionFile: assertion } } }), { mode: 0o600 }); const rejected = spawnSync(process.execPath, [facade, 'run', '--context=' + context, '--command=' + command, '--assertion=' + assertion], { encoding: 'utf8' }); let rejectedFailure; try { rejectedFailure = JSON.parse(rejected.stderr); } catch { process.exit(23); } if (rejected.status !== 78 || rejectedFailure.code !== 'QL3_CLUSTER_PRODUCT_CONTEXT_INVALID' || rejected.stdout !== '' || rejected.stderr.includes('/tmp/') || rejected.stderr.includes('assertion.jwt')) process.exit(24); -process.stdout.write(JSON.stringify({ schemaVersion: 1, injected: true, contextPreflight: true, secretFieldsRejected: true })); +process.stdout.write(JSON.stringify({ schemaVersion: 1, injected: true, contextPreflight: true, contextReadiness: true, secretFieldsRejected: true })); `; const output = docker([ 'run', @@ -131,6 +167,12 @@ process.stdout.write(JSON.stringify({ schemaVersion: 1, injected: true, contextP '0.25', '--tmpfs', '/tmp:rw,noexec,nosuid,nodev,size=8m,mode=700,uid=10001,gid=10001', + '--volume', + `${resolve(fixtureRoot, 'ca-cert.pem')}:/evidence/ca-cert.pem:ro`, + '--volume', + `${resolve(fixtureRoot, 'server-cert.pem')}:/evidence/server-cert.pem:ro`, + '--volume', + `${resolve(fixtureRoot, 'server-key.pem')}:/evidence/server-key.pem:ro`, '--entrypoint', 'node', image, @@ -147,6 +189,7 @@ process.stdout.write(JSON.stringify({ schemaVersion: 1, injected: true, contextP result?.schemaVersion !== 1 || result?.injected !== true || result?.contextPreflight !== true || + result?.contextReadiness !== true || result?.secretFieldsRejected !== true ) { fail('operator context contract drifted'); @@ -203,6 +246,7 @@ function main() { commandCount: COMMANDS.length, operatorContext: true, contextPreflight: true, + contextReadiness: true, isolation: Object.freeze({ readOnlyRoot: true, network: 'none', diff --git a/scripts/ql3-cluster-image-release-audit.cjs b/scripts/ql3-cluster-image-release-audit.cjs index 14c687fe..16fd6d57 100644 --- a/scripts/ql3-cluster-image-release-audit.cjs +++ b/scripts/ql3-cluster-image-release-audit.cjs @@ -289,8 +289,8 @@ function auditClusterImageCiWorkflow( ); requirePattern( adminProductLiveContract, - /runOperatorContextContract\(image\);[\s\S]*operatorContext: true,[\s\S]*contextPreflight: true/, - 'native admin image contract must verify owner-private operator context injection and offline preflight', + /runOperatorContextContract\(image\);[\s\S]*operatorContext: true,[\s\S]*contextPreflight: true,[\s\S]*contextReadiness: true/, + 'native admin image contract must verify owner-private operator context injection, offline preflight and read-only readiness', ); requirePattern( source, @@ -335,6 +335,7 @@ function auditClusterImageCiWorkflow( clusterAdminProductFacade: true, clusterAdminOperatorContext: true, clusterAdminContextPreflight: true, + clusterAdminContextReadiness: true, ociAttestations: true, osVulnerabilityScan: { scanner: 'trivy@0.70.0', diff --git a/test/back/ql3ClusterImageReleaseAudit.test.cjs b/test/back/ql3ClusterImageReleaseAudit.test.cjs index d259095e..1abf209b 100644 --- a/test/back/ql3ClusterImageReleaseAudit.test.cjs +++ b/test/back/ql3ClusterImageReleaseAudit.test.cjs @@ -29,6 +29,7 @@ test('accepts the reviewed native CI and digest release contracts', () => { clusterAdminProductFacade: true, clusterAdminOperatorContext: true, clusterAdminContextPreflight: true, + clusterAdminContextReadiness: true, ociAttestations: true, osVulnerabilityScan: { scanner: 'trivy@0.70.0', @@ -145,6 +146,24 @@ test('rejects a Cluster Admin live gate that omits offline context preflight', ( ); }); +test('rejects a Cluster Admin live gate that omits read-only context readiness', () => { + const contract = fs.readFileSync( + path.join(root, 'scripts/ql3-cluster-admin-product-live-contract.cjs'), + 'utf8', + ); + assert.throws( + () => + auditClusterImageCiWorkflow( + ciSource, + contract.replace( + 'contextPreflight: true,\n contextReadiness: true', + 'contextPreflight: true,\n contextReadiness: false', + ), + ), + /read-only readiness/, + ); +}); + test('rejects removal of the native cluster-admin image gate', () => { const mutated = ciSource.replace( 'image_arch: arm64\n image: admin', diff --git a/test/back/ql3PackageBoundaryAudit.test.cjs b/test/back/ql3PackageBoundaryAudit.test.cjs index 7a15544c..be394ab7 100644 --- a/test/back/ql3PackageBoundaryAudit.test.cjs +++ b/test/back/ql3PackageBoundaryAudit.test.cjs @@ -340,10 +340,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: clusterAdmin.rootSourceFileRoles, }, { - sourceFiles: 97, + sourceFiles: 99, rootSourceFiles: 1, rootSourceLines: 61, - nestedSourceFiles: 96, + nestedSourceFiles: 98, rootSourceFileRoles: { 'modelInvocationMigrationCli.ts': 'binary_entry', },