feat(ql3): add read-only cluster context readiness

This commit is contained in:
whyour
2026-08-13 02:14:17 +08:00
parent a83b4c5e8d
commit a6ad636251
18 changed files with 1529 additions and 391 deletions
+13
View File
@@ -11,6 +11,19 @@
最新增量证据(2026-08-13):
- D-303/ADR-0391(已接受)
Cluster operator context 在离线 `validate` 之后增加显式、只读的 `ql3-cluster-admin context probe`。probe 先完整预检全部
context entry,任何晚出现的配置错误都会在首个网络请求前失败关闭;随后才按固定 catalog 顺序,以 production TLS 1.3、CA、
servername、mTLS client certificate 或 Kubernetes PortForward 配置逐项 `GET /readyz`。探针不读取 command/assertion,不发送
Authorization、body 或业务 management POST,不重试、不切换 Pod,响应限 1 KiB,只接受精确 `200/ready``503/not_ready`
输出仅含 command、transport、ready 状态及 `mutation:false`;配置错误退出 78,不可达/协议错误或任一 not-ready 退出 69。
TLS preparation 与六类 management route/client-certificate policy 收敛为一个 package-private 真源,既有 mutation client 的公开
subpath 与请求语义不变。能力仍只存在于短生命周期 Cluster Admin,不新增 package、依赖、binary、listener、timer、controller、
workload 或 sidecarLocal/Edge、Cluster Control 与 Worker 零导入。Cluster Admin 完整 302 pass/2 条件 skip18-package clean
build/test 退出 0backend 1,188 pass/2 skip;五项边界审计零 finding。真实 arm64 Admin image 为 330,487,296 bytes,并在
`10001:10001`、network none、read-only root、drop ALL、128 MiB/32 PIDs 下完成本地 TLS readiness 契约。14 个 Local Profile
artifact 与 D-302 对应字节数一致;PostgreSQL 18.4 arm64 HA 123 项 gate 全绿、timeline `1→2`,证据 SHA-256 为
`e7c1743e932f2d7c35dc9153cdf5bc4a03356a38d93fce5507354652aa207a05`,独立审计与 Docker 清理通过。完整验证证据见 ADR-0391。
- D-302/ADR-0390(已接受)
Cluster operator context 增加无网络、无 mutation 的内建 `ql3-cluster-admin context validate` 预检。它先复用 owner-private context
reader,再让每个 entry 经过与真实请求相同的 production HTTPS/Kubernetes configuration preparation,验证精确 route、hostname、CA、
@@ -0,0 +1,53 @@
# ADR-0391Read-only Cluster Operator Context Readiness
- 状态:Accepted
- 日期:2026-08-13
- 关联 RFCQL-RFC-0001 D-303
- 前置决策:ADR-0247、ADR-0250、ADR-0388、ADR-0389、ADR-0390
## 上下文
ADR-0390 的 `context validate` 能在完全离线状态证明 context、TLS、mTLS 与 Kubernetes 配置可解析,但不能证明发布窗口中的目标进程已就绪、证书链仍被线上端点接受或 Kubernetes tunnel 可建立。用任一 management POST 试探会读取短生命周期 assertion 和 command,并可能进入认证、审计、quota 或 mutation 路径;在工作站另写 curl/kubectl 流程则会绕过生产 client 的 hostname、TLS 1.3、mTLS 与 PortForward 约束。
## 决策
1. 在既有 Cluster Admin facade 内增加 `context probe --context=/absolute/operator-context.json`。它不是第八个远程业务命令,不新增 binary、package、依赖、常驻进程或部署 authority。
2. probe 必须先调用完整离线 `context validate`。所有 entry 均通过后才能产生第一个网络请求,避免前项已访问而后项配置错误的部分探测。
3. 每项只使用与 production client 相同的 config preparation、CA、servername、TLS 1.3 和 mTLS client certificate。Kubernetes 项复用受审 kubeconfig preparation、单个 ready Pod 选择与 PortForward tunnel;不得读取 ambient kubeconfig。
4. 唯一请求为 `GET /readyz`,无 Authorization、Content-Type 或 body。它不读取 command/assertion,不调用 management route,不查询数据库,不产生 mutation,不重试,也不在连接失败后切换 Pod。
5. 响应硬限 1 KiB、identity encoding、精确 JSON content type/schema;只接受 `200 {schemaVersion:1,status:"ready"}``503 {schemaVersion:1,status:"not_ready"}`,拒绝 redirect、额外字段、重复/错误长度、压缩、畸形 UTF-8/JSON 和其他状态。
6. 输出只包含固定 command 名、`https|kubernetes-port-forward``ready|not_ready`、固定 method/path 与 `mutation:false`。不得输出 endpoint、port、namespace、Pod、路径、证书主体或错误对象。全部 ready 退出 0;任一 not-ready、连接或协议失败退出 69;离线配置错误仍退出 78;语法错误退出 64。
7. management route 与 client-certificate class 只有一个 package-private policy 真源。配置 preparation 和 readiness 实现不从 package manifest 导出;既有公开 client subpath、POST 语义、错误码与调用方保持不变。
8. 能力只存在于短生命周期 Cluster Admin image。Local/Edge、Cluster Control、Worker 的依赖闭包、文件、RSS、listener、timer、数据库与制品不得变化。
## 不采用方案
- **直接执行一条 inspect 命令**:仍需 assertion/command,会留下认证或审计事实,且把 readiness 与业务授权混为一体。
- **通用 URL/方法探针**:扩大 SSRF 与 authority 面,无法约束到受审 endpoint 和固定路径。
- **自动重试或多 Pod failover**:隐藏单次现场状态、增加窗口时间,并把短生命周期诊断工具变成控制器。
- **把 probe 放进 Cluster Control 或 sidecar**:会给常驻运行时增加网络、定时器和依赖成本,低配 Local/Edge 也没有该需求。
- **为 readiness 新建 package**:只有 Cluster Admin 一个制品消费者,拆包会复制 manifest、发布和供应链表面。
## 操作顺序
1. `ql3-cluster-admin context validate --context=/absolute/operator-context.json`
2. `ql3-cluster-admin context probe --context=/absolute/operator-context.json`
3. 只有前两步符合维护计划,操作者才另行提供精确 command 与短生命周期 assertion 执行业务命令。
probe 不是持续监控、负载均衡健康检查或 mutation 成功保证;退出 0 只表示这一时刻全部受审 readiness 端点返回 ready。
## 验收门
- 无证书 TLS 与 mTLS 真实握手、ready/not-ready、不可达、redirect、畸形与超限响应;
- Kubernetes 单 Pod tunnel、连接丢失时无重试/切换,以及低敏错误映射;
- context 全量离线先验、固定顺序、无 command/assertion/Authorization/body 与输出 secret scan
- 真实 Admin image 在 non-root、read-only root、network none 容器内启动本地 TLS fixture,证明固定 GET 契约及资源上限;
- Cluster Admin、18-package clean build/test、backend、package/dependency/deployment/image/Local image、14 Local Profile artifact 与 PostgreSQL HA 全量不回归。
## 当前证据
- readiness/TLS/mTLS/Kubernetes tunnel、context product 与 release 定向契约 93/93。Cluster Admin 完整 304 tests、302 pass/2 条外部服务条件 skip/0 fail18-package clean build/test 退出 0backend 1,190 tests、1,188 pass/2 skip/0 fail。
- workspace 保持 18 packageCluster Admin 99 source 中 98 nested/1 个既有 root binary,无 single-source/shallow package且未增加依赖或公开 subpath。package/dependency/deployment/image release/Local image 五项边界审计均 compatible、零 finding。
- 当前源码真实 arm64 Admin image 为 330,487,296 bytes,较 D-302 增加 23,768 bytes;在 `10001:10001`、read-only root、network none、drop ALL、no-new-privileges、0.25 CPU、128 MiB/32 PIDs 下启动同容器 loopback TLS fixture,报告 `contextReadiness=true`,并证明精确 `GET /readyz`、无 Authorization、0-byte body。
- 14 个 Local Profile artifact 全部 compatible 且与 D-302 字节数一致:最小 Edge 为 2,467,343 bytes/295 files/53 modules,最大 Standalone MCP 为 7,168,978 bytes/778 files/213 modules,证明 Router/Edge 闭包零增量。
- PostgreSQL 18.4 arm64 HA 123 项 gate 全绿,timeline `1→2`;私有报告 SHA-256 为 `e7c1743e932f2d7c35dc9153cdf5bc4a03356a38d93fce5507354652aa207a05`,独立 evidence audit `compatible=true`、零 finding,测试容器、网络与卷均零残留。
+1
View File
@@ -394,6 +394,7 @@
| [ADR-0388](./ADR-0388-bounded-unified-cluster-product-cli.md) | 有界统一 Cluster Operator 产品 CLI | Accepted |
| [ADR-0389](./ADR-0389-owner-private-cluster-operator-context.md) | Owner-private Cluster Operator Context | Accepted |
| [ADR-0390](./ADR-0390-offline-cluster-operator-context-preflight.md) | Offline Cluster Operator Context Preflight | Accepted |
| [ADR-0391](./ADR-0391-read-only-cluster-operator-context-readiness.md) | Read-only Cluster Operator Context Readiness | Accepted |
## 规则
@@ -0,0 +1,21 @@
# QingLong 3.0 Cluster Operator Context
Cluster operator context 只为短生命周期 `ql3-cluster-admin` 复用受审 client 与 Kubernetes 配置路径。它不是 credential store,不得包含 assertion、command、token、private key、endpoint 内容或默认业务操作;context 及引用的私有文件必须是 absolute、canonical、当前 UID 所有、非 symlink 的 `0600` regular file。
维护窗口按固定顺序执行:
```sh
ql3-cluster-admin context validate \
--context=/secure/qinglong3/operator-context.json
ql3-cluster-admin context probe \
--context=/secure/qinglong3/operator-context.json
```
`validate` 完全离线,验证所有 context entry、TLS/mTLS 与 Kubernetes 配置,不建立连接。`probe` 会再次先完成全量离线验证,然后按固定 catalog 顺序使用 production TLS 1.3 或 Kubernetes PortForward 发送无 Authorization、无 body 的 `GET /readyz`;它不会执行 management POST、读取 assertion/command、重试或切换 Pod。
- `validate` 成功退出 0;配置错误输出低敏 `QL3_CLUSTER_PRODUCT_CONTEXT_INVALID` 并退出 78。
- `probe` 全部 ready 退出 0;明确 not-ready、不可达或响应协议错误退出 69。not-ready 会输出各 command 的低敏状态;连接/协议失败只输出 `QL3_CLUSTER_PRODUCT_CONTEXT_PROBE_FAILED`
- 退出 0 只证明探测时刻 readiness,不代表后续 mutation 一定成功。业务命令仍需单独、显式提供精确 command 与短生命周期 assertion。
不要把 probe 放进 cron、sidecar、router/Edge Profile 或 Cluster Control。持续监控应使用独立、最小权限的可观测性面;该命令保留为人工发布/维护窗口的有界诊断工具。
@@ -0,0 +1,426 @@
/** Package-private configuration preparation shared by management clients. */
import { createPrivateKey, X509Certificate } from 'node:crypto';
import {
closeSync,
constants,
fstatSync,
lstatSync,
openSync,
readSync,
realpathSync,
} from 'node:fs';
import { isIP } from 'node:net';
import { isAbsolute } from 'node:path';
import { TextDecoder } from 'node:util';
export type ClusterAuthenticatedManagementClientKind =
| 'package'
| 'worker-credential'
| 'automation'
| 'approval'
| 'model-credential'
| 'run';
const MANAGEMENT_CLIENT_POLICIES: Readonly<
Record<
ClusterAuthenticatedManagementClientKind,
Readonly<{
managementPath: string;
clientCertificate: 'forbidden' | 'required';
}>
>
> = Object.freeze({
package: Object.freeze({
managementPath: '/api/v3/plugin-packages/management',
clientCertificate: 'forbidden',
}),
'worker-credential': Object.freeze({
managementPath: '/api/v3/worker-credentials/management',
clientCertificate: 'required',
}),
automation: Object.freeze({
managementPath: '/api/v3/automations/management',
clientCertificate: 'required',
}),
approval: Object.freeze({
managementPath: '/api/v3/approvals/management',
clientCertificate: 'required',
}),
'model-credential': Object.freeze({
managementPath: '/api/v3/provider-credentials/management',
clientCertificate: 'required',
}),
run: Object.freeze({
managementPath: '/api/v3/runs/management',
clientCertificate: 'required',
}),
});
const MAXIMUM_CONFIG_BYTES = 16 * 1024;
const MAXIMUM_CA_BYTES = 256 * 1024;
const MAXIMUM_CLIENT_CERTIFICATE_BYTES = 256 * 1024;
const MAXIMUM_CLIENT_PRIVATE_KEY_BYTES = 256 * 1024;
const CONTROL_PATTERN = /[\u0000-\u001f\u007f]/;
const DNS_NAME_PATTERN =
/^(?=.{1,253}$)[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?(?:\.[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?)*$/;
type JsonObject = Record<string, unknown>;
export interface ClusterAuthenticatedManagementClientConfigurationSummary {
readonly schemaVersion: 1;
readonly managementPath: string;
readonly transport: 'https';
readonly clientCertificate: 'forbidden' | 'required';
}
export interface PreparedClusterAuthenticatedManagementClientConfiguration {
readonly endpoint: URL;
readonly servername: string;
readonly port: number;
readonly requestTimeoutMs: number;
readonly caBytes: Buffer;
readonly clientCertificateBytes?: Buffer;
readonly clientPrivateKeyBytes?: Buffer;
dispose(): void;
}
export class ClusterPluginPackageManagementClientConfigurationError extends TypeError {
readonly code = 'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_CONFIG_INVALID';
constructor() {
super('Plugin Package management client configuration is invalid');
this.name = 'ClusterPluginPackageManagementClientConfigurationError';
}
}
function configurationFailure(): ClusterPluginPackageManagementClientConfigurationError {
return new ClusterPluginPackageManagementClientConfigurationError();
}
function exactObject(
value: unknown,
expectedKeys: readonly string[],
): asserts value is JsonObject {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw configurationFailure();
}
const actual = Object.keys(value).sort();
const expected = [...expectedKeys].sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
throw configurationFailure();
}
}
function currentUid(): number {
if (typeof process.getuid !== 'function') throw configurationFailure();
const uid = process.getuid();
if (!Number.isSafeInteger(uid) || uid < 0) throw configurationFailure();
return uid;
}
export function readCanonicalFile(
filePath: string,
maximumBytes: number,
mode: 'private' | 'public-integrity',
): Buffer {
if (
typeof filePath !== 'string' ||
!isAbsolute(filePath) ||
filePath.length > 4_096 ||
CONTROL_PATTERN.test(filePath)
) {
throw configurationFailure();
}
let before;
try {
before = lstatSync(filePath);
if (
!before.isFile() ||
before.isSymbolicLink() ||
before.size < 1 ||
before.size > maximumBytes ||
realpathSync(filePath) !== filePath
) {
throw configurationFailure();
}
} catch (error) {
if (error instanceof ClusterPluginPackageManagementClientConfigurationError) {
throw error;
}
throw configurationFailure();
}
const uid = currentUid();
const permissions = before.mode & 0o777;
if (
(mode === 'private' && (before.uid !== uid || permissions !== 0o600)) ||
(mode === 'public-integrity' && before.uid !== uid && before.uid !== 0) ||
(mode === 'public-integrity' && (permissions & 0o022) !== 0)
) {
throw configurationFailure();
}
let descriptor = -1;
let bytes: Buffer | undefined;
try {
descriptor = openSync(
filePath,
constants.O_RDONLY |
((constants as unknown as Readonly<Record<string, number>>).O_CLOEXEC ??
0) |
(constants.O_NOFOLLOW ?? 0),
);
const opened = fstatSync(descriptor);
if (
!opened.isFile() ||
opened.dev !== before.dev ||
opened.ino !== before.ino ||
opened.uid !== before.uid ||
opened.mode !== before.mode ||
opened.size !== before.size
) {
throw configurationFailure();
}
bytes = Buffer.alloc(opened.size);
let offset = 0;
while (offset < bytes.length) {
const count = readSync(
descriptor,
bytes,
offset,
bytes.length - offset,
offset,
);
if (count < 1) throw configurationFailure();
offset += count;
}
const after = fstatSync(descriptor);
if (
after.dev !== opened.dev ||
after.ino !== opened.ino ||
after.uid !== opened.uid ||
after.mode !== opened.mode ||
after.size !== opened.size
) {
throw configurationFailure();
}
return bytes;
} catch (error) {
bytes?.fill(0);
if (error instanceof ClusterPluginPackageManagementClientConfigurationError) {
throw error;
}
throw configurationFailure();
} finally {
if (descriptor >= 0) closeSync(descriptor);
}
}
function parseJson(bytes: Buffer): unknown {
try {
return JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(bytes));
} catch {
throw configurationFailure();
}
}
export function isReviewedClusterAuthenticatedManagementClientProtocol(
managementPath: string,
clientCertificate: 'forbidden' | 'required',
): boolean {
return Object.values(MANAGEMENT_CLIENT_POLICIES).some(
(policy) =>
policy.managementPath === managementPath &&
policy.clientCertificate === clientCertificate,
);
}
export function prepareClusterAuthenticatedManagementClientConfiguration(
configFile: string,
managementPath: string,
clientCertificate: 'forbidden' | 'required',
): PreparedClusterAuthenticatedManagementClientConfiguration {
if (
!isReviewedClusterAuthenticatedManagementClientProtocol(
managementPath,
clientCertificate,
)
) {
throw configurationFailure();
}
let configBytes: Buffer | undefined;
let caBytes: Buffer | undefined;
let clientCertificateBytes: Buffer | undefined;
let clientPrivateKeyBytes: Buffer | undefined;
try {
configBytes = readCanonicalFile(
configFile,
MAXIMUM_CONFIG_BYTES,
'private',
);
const config = parseJson(configBytes);
exactObject(
config,
clientCertificate === 'required'
? [
'schemaVersion',
'endpoint',
'servername',
'caFile',
'clientCertificateFile',
'clientPrivateKeyFile',
'requestTimeoutMs',
]
: [
'schemaVersion',
'endpoint',
'servername',
'caFile',
'requestTimeoutMs',
],
);
if (
config.schemaVersion !== 1 ||
typeof config.endpoint !== 'string' ||
typeof config.servername !== 'string' ||
!DNS_NAME_PATTERN.test(config.servername) ||
isIP(config.servername) !== 0 ||
typeof config.caFile !== 'string' ||
(clientCertificate === 'required' &&
(typeof config.clientCertificateFile !== 'string' ||
typeof config.clientPrivateKeyFile !== 'string')) ||
!Number.isSafeInteger(config.requestTimeoutMs) ||
(config.requestTimeoutMs as number) < 1_000 ||
(config.requestTimeoutMs as number) > 30_000
) {
throw configurationFailure();
}
const servername = config.servername;
const requestTimeoutMs = config.requestTimeoutMs as number;
let endpoint: URL;
try {
endpoint = new URL(config.endpoint);
} catch {
throw configurationFailure();
}
if (
endpoint.protocol !== 'https:' ||
endpoint.username !== '' ||
endpoint.password !== '' ||
endpoint.search !== '' ||
endpoint.hash !== '' ||
endpoint.pathname !== managementPath ||
endpoint.hostname !== servername ||
isIP(endpoint.hostname) !== 0
) {
throw configurationFailure();
}
const port = endpoint.port === '' ? 443 : Number(endpoint.port);
if (!Number.isSafeInteger(port) || port < 1 || port > 65_535) {
throw configurationFailure();
}
caBytes = readCanonicalFile(
config.caFile as string,
MAXIMUM_CA_BYTES,
'public-integrity',
);
try {
new X509Certificate(caBytes);
} catch {
throw configurationFailure();
}
if (clientCertificate === 'required') {
clientCertificateBytes = readCanonicalFile(
config.clientCertificateFile as string,
MAXIMUM_CLIENT_CERTIFICATE_BYTES,
'public-integrity',
);
clientPrivateKeyBytes = readCanonicalFile(
config.clientPrivateKeyFile as string,
MAXIMUM_CLIENT_PRIVATE_KEY_BYTES,
'private',
);
try {
const certificate = new X509Certificate(clientCertificateBytes);
const privateKey = createPrivateKey(clientPrivateKeyBytes);
if (!certificate.checkPrivateKey(privateKey)) {
throw configurationFailure();
}
} catch (error) {
if (error instanceof ClusterPluginPackageManagementClientConfigurationError) {
throw error;
}
throw configurationFailure();
}
}
let disposed = false;
return Object.freeze({
endpoint,
servername,
port,
requestTimeoutMs,
caBytes,
...(clientCertificateBytes === undefined
? {}
: {
clientCertificateBytes,
clientPrivateKeyBytes: clientPrivateKeyBytes!,
}),
dispose() {
if (disposed) return;
disposed = true;
caBytes?.fill(0);
clientCertificateBytes?.fill(0);
clientPrivateKeyBytes?.fill(0);
},
});
} catch (error) {
caBytes?.fill(0);
clientCertificateBytes?.fill(0);
clientPrivateKeyBytes?.fill(0);
if (error instanceof ClusterPluginPackageManagementClientConfigurationError) {
throw error;
}
throw configurationFailure();
} finally {
configBytes?.fill(0);
}
}
export function prepareClusterAuthenticatedManagementClientKindConfiguration(
configFile: string,
kind: ClusterAuthenticatedManagementClientKind,
): PreparedClusterAuthenticatedManagementClientConfiguration {
const policy = MANAGEMENT_CLIENT_POLICIES[kind];
if (policy === undefined) throw configurationFailure();
return prepareClusterAuthenticatedManagementClientConfiguration(
configFile,
policy.managementPath,
policy.clientCertificate,
);
}
export function validateClusterAuthenticatedManagementClientConfiguration(
configFile: string,
kind: ClusterAuthenticatedManagementClientKind,
): Readonly<ClusterAuthenticatedManagementClientConfigurationSummary> {
const policy = MANAGEMENT_CLIENT_POLICIES[kind];
if (policy === undefined) throw configurationFailure();
const prepared =
prepareClusterAuthenticatedManagementClientKindConfiguration(
configFile,
kind,
);
try {
return Object.freeze({
schemaVersion: 1,
managementPath: policy.managementPath,
transport: 'https',
clientCertificate: policy.clientCertificate,
});
} finally {
prepared.dispose();
}
}
@@ -0,0 +1,300 @@
/** Bounded read-only readiness probe for reviewed cluster management clients. */
import { Agent as HttpsAgent, request as httpsRequest } from 'node:https';
import { Duplex } from 'node:stream';
import { connect as tlsConnect } from 'node:tls';
import { TextDecoder } from 'node:util';
import {
ClusterPluginPackageManagementClientRequestError,
type ClusterAuthenticatedManagementClientKind,
type ClusterPluginPackageManagementClientConnectionOptions,
type ClusterPluginPackageManagementClientRawConnection,
} from './pluginPackageManagementClient';
import {
ClusterPluginPackageManagementClientConfigurationError,
prepareClusterAuthenticatedManagementClientKindConfiguration,
type PreparedClusterAuthenticatedManagementClientConfiguration,
} from './managementClientConfiguration';
const MAXIMUM_RESPONSE_BYTES = 1_024;
export interface ClusterAuthenticatedManagementClientReadiness {
readonly schemaVersion: 1;
readonly transport: 'https';
readonly ready: boolean;
}
function configurationFailure(): ClusterPluginPackageManagementClientConfigurationError {
return new ClusterPluginPackageManagementClientConfigurationError();
}
function rawHeaderCount(rawHeaders: readonly string[], name: string): number {
let count = 0;
for (let index = 0; index < rawHeaders.length; index += 2) {
if (rawHeaders[index]?.toLowerCase() === name) count += 1;
}
return count;
}
function readinessEnvelope(
value: unknown,
): Readonly<{ schemaVersion: unknown; status: unknown }> {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw new ClusterPluginPackageManagementClientRequestError();
}
const keys = Object.keys(value).sort();
if (
keys.length !== 2 ||
keys[0] !== 'schemaVersion' ||
keys[1] !== 'status'
) {
throw new ClusterPluginPackageManagementClientRequestError();
}
return value as Readonly<{ schemaVersion: unknown; status: unknown }>;
}
function connectionOptionsValid(
value: ClusterPluginPackageManagementClientConnectionOptions | undefined,
): boolean {
return (
value === undefined ||
(!!value &&
typeof value === 'object' &&
!Array.isArray(value) &&
Object.keys(value).length === 1 &&
typeof value.connect === 'function')
);
}
export async function probeClusterAuthenticatedManagementClientReadiness(
configFile: string,
kind: ClusterAuthenticatedManagementClientKind,
connectionOptions?: ClusterPluginPackageManagementClientConnectionOptions,
): Promise<Readonly<ClusterAuthenticatedManagementClientReadiness>> {
if (!connectionOptionsValid(connectionOptions)) throw configurationFailure();
let prepared:
| PreparedClusterAuthenticatedManagementClientConfiguration
| undefined;
let rawConnection:
| ClusterPluginPackageManagementClientRawConnection
| undefined;
let connectionAgent: HttpsAgent | undefined;
try {
prepared = prepareClusterAuthenticatedManagementClientKindConfiguration(
configFile,
kind,
);
const {
endpoint,
servername,
port,
requestTimeoutMs,
caBytes,
clientCertificateBytes,
clientPrivateKeyBytes,
} = prepared;
if (connectionOptions) {
rawConnection = await connectionOptions.connect(
Object.freeze({ hostname: endpoint.hostname, port }),
);
if (
!rawConnection ||
typeof rawConnection !== 'object' ||
!(rawConnection.stream instanceof Duplex) ||
typeof rawConnection.close !== 'function'
) {
throw new ClusterPluginPackageManagementClientRequestError();
}
const establishedConnection = rawConnection;
connectionAgent = new HttpsAgent({
keepAlive: false,
maxSockets: 1,
maxFreeSockets: 0,
});
connectionAgent.createConnection = (_options, callback) => {
const socket = tlsConnect({
socket: establishedConnection.stream,
ca: caBytes,
...(clientCertificateBytes === undefined
? {}
: {
cert: clientCertificateBytes,
key: clientPrivateKeyBytes,
}),
servername,
minVersion: 'TLSv1.3',
maxVersion: 'TLSv1.3',
rejectUnauthorized: true,
});
if (callback) {
let reported = false;
socket.once('secureConnect', () => {
if (reported) return;
reported = true;
callback(null, socket);
});
socket.once('error', (error) => {
if (reported) return;
reported = true;
callback(error, socket);
});
}
return socket;
};
}
return await new Promise((resolve, reject) => {
let settled = false;
const chunks: Buffer[] = [];
let length = 0;
const clearChunks = () => {
for (const chunk of chunks) chunk.fill(0);
};
const finish = (
error: unknown,
result?: Readonly<ClusterAuthenticatedManagementClientReadiness>,
) => {
if (settled) return;
settled = true;
if (error) reject(error);
else resolve(result!);
};
const request = httpsRequest(
{
protocol: 'https:',
hostname: endpoint.hostname,
port,
path: '/readyz',
method: 'GET',
servername,
ca: caBytes,
...(clientCertificateBytes === undefined
? {}
: {
cert: clientCertificateBytes,
key: clientPrivateKeyBytes,
}),
minVersion: 'TLSv1.3',
maxVersion: 'TLSv1.3',
rejectUnauthorized: true,
agent: connectionAgent ?? false,
headers: {
accept: 'application/json',
'accept-encoding': 'identity',
connection: 'close',
},
},
(response) => {
response.once('aborted', () => {
clearChunks();
finish(new ClusterPluginPackageManagementClientRequestError());
});
response.once('error', (error) => {
clearChunks();
finish(
new ClusterPluginPackageManagementClientRequestError(error),
);
});
response.on('data', (chunk: Buffer | string) => {
const bytes = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
length += bytes.length;
if (length > MAXIMUM_RESPONSE_BYTES) {
const error =
new ClusterPluginPackageManagementClientRequestError();
bytes.fill(0);
clearChunks();
response.destroy();
request.destroy(error);
finish(error);
return;
}
chunks.push(bytes);
});
response.once('end', () => {
try {
if (
rawHeaderCount(response.rawHeaders, 'content-type') !== 1 ||
response.headers['content-type'] !==
'application/json; charset=utf-8' ||
response.headers['content-encoding'] !== undefined ||
rawHeaderCount(response.rawHeaders, 'content-length') > 1 ||
(response.headers['content-length'] !== undefined &&
(!/^(?:0|[1-9][0-9]*)$/.test(
response.headers['content-length'],
) ||
Number(response.headers['content-length']) !== length))
) {
throw new ClusterPluginPackageManagementClientRequestError();
}
const bytes = Buffer.concat(chunks, length);
let parsed: unknown;
try {
parsed = JSON.parse(
new TextDecoder('utf-8', { fatal: true }).decode(bytes),
);
} finally {
bytes.fill(0);
clearChunks();
}
const record = readinessEnvelope(parsed);
const ready =
response.statusCode === 200 && record.status === 'ready';
const notReady =
response.statusCode === 503 &&
record.status === 'not_ready';
if (record.schemaVersion !== 1 || (!ready && !notReady)) {
throw new ClusterPluginPackageManagementClientRequestError();
}
finish(
undefined,
Object.freeze({
schemaVersion: 1,
transport: 'https',
ready,
}),
);
} catch (error) {
finish(
error instanceof
ClusterPluginPackageManagementClientRequestError
? error
: new ClusterPluginPackageManagementClientRequestError(
error,
),
);
}
});
},
);
request.setTimeout(requestTimeoutMs, () => {
request.destroy(
new ClusterPluginPackageManagementClientRequestError(),
);
});
request.once('error', (error) => {
clearChunks();
finish(
error instanceof ClusterPluginPackageManagementClientRequestError
? error
: new ClusterPluginPackageManagementClientRequestError(error),
);
});
request.end();
});
} catch (error) {
if (
error instanceof ClusterPluginPackageManagementClientConfigurationError ||
error instanceof ClusterPluginPackageManagementClientRequestError
) {
throw error;
}
throw new ClusterPluginPackageManagementClientRequestError(error);
} finally {
connectionAgent?.destroy();
try {
await rawConnection?.close();
} catch {
// Probe outcome remains authoritative after bounded resource cleanup.
}
prepared?.dispose();
}
}
@@ -1,20 +1,28 @@
/** Shared one-shot authenticated client boundary for cluster management planes. */
import {
closeSync,
constants,
fstatSync,
lstatSync,
openSync,
readSync,
realpathSync,
} from 'node:fs';
import { Agent as HttpsAgent, request as httpsRequest } from 'node:https';
import { isIP } from 'node:net';
import { isAbsolute } from 'node:path';
import { Duplex } from 'node:stream';
import { connect as tlsConnect } from 'node:tls';
import { TextDecoder } from 'node:util';
import { createPrivateKey, X509Certificate } from 'node:crypto';
import {
ClusterPluginPackageManagementClientConfigurationError,
isReviewedClusterAuthenticatedManagementClientProtocol,
prepareClusterAuthenticatedManagementClientConfiguration,
readCanonicalFile,
validateClusterAuthenticatedManagementClientConfiguration,
type ClusterAuthenticatedManagementClientConfigurationSummary,
type ClusterAuthenticatedManagementClientKind,
type PreparedClusterAuthenticatedManagementClientConfiguration,
} from './managementClientConfiguration';
export {
ClusterPluginPackageManagementClientConfigurationError,
readCanonicalFile,
validateClusterAuthenticatedManagementClientConfiguration,
};
export type {
ClusterAuthenticatedManagementClientConfigurationSummary,
ClusterAuthenticatedManagementClientKind,
};
import {
normalizeClusterPluginPackageManagementCommand,
@@ -23,59 +31,11 @@ import {
} from '../plugin-package/management/pluginPackageManagementTransport';
const MANAGEMENT_PATH = '/api/v3/plugin-packages/management';
export type ClusterAuthenticatedManagementClientKind =
| 'package'
| 'worker-credential'
| 'automation'
| 'approval'
| 'model-credential'
| 'run';
const MANAGEMENT_CLIENT_POLICIES: Readonly<
Record<
ClusterAuthenticatedManagementClientKind,
Readonly<{
managementPath: string;
clientCertificate: 'forbidden' | 'required';
}>
>
> = Object.freeze({
package: Object.freeze({
managementPath: MANAGEMENT_PATH,
clientCertificate: 'forbidden',
}),
'worker-credential': Object.freeze({
managementPath: '/api/v3/worker-credentials/management',
clientCertificate: 'required',
}),
automation: Object.freeze({
managementPath: '/api/v3/automations/management',
clientCertificate: 'required',
}),
approval: Object.freeze({
managementPath: '/api/v3/approvals/management',
clientCertificate: 'required',
}),
'model-credential': Object.freeze({
managementPath: '/api/v3/provider-credentials/management',
clientCertificate: 'required',
}),
run: Object.freeze({
managementPath: '/api/v3/runs/management',
clientCertificate: 'required',
}),
});
const MAX_CONFIG_BYTES = 16 * 1024;
const MAX_ASSERTION_BYTES = 16 * 1024;
const MAX_COMMAND_BYTES = 256 * 1024;
const MAX_CA_BYTES = 256 * 1024;
const MAX_CLIENT_CERTIFICATE_BYTES = 256 * 1024;
const MAX_CLIENT_PRIVATE_KEY_BYTES = 256 * 1024;
const MAX_RESPONSE_BYTES = 128 * 1024;
const CONTROL_PATTERN = /[\u0000-\u001f\u007f]/;
const PACKAGE_NAME_PATTERN = /^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$/;
const DNS_NAME_PATTERN =
/^(?=.{1,253}$)[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?(?:\.[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?)*$/;
const ASSERTION_PATTERN = /^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/;
const TOKEN_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$/;
const DIGEST_PATTERN = /^[0-9a-f]{64}$/;
@@ -123,15 +83,6 @@ export interface ClusterPluginPackageManagementClientConnectionOptions {
): Promise<ClusterPluginPackageManagementClientRawConnection>;
}
export class ClusterPluginPackageManagementClientConfigurationError extends TypeError {
readonly code = 'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_CONFIG_INVALID';
constructor() {
super('Plugin Package management client configuration is invalid');
this.name = 'ClusterPluginPackageManagementClientConfigurationError';
}
}
export class ClusterPluginPackageManagementClientRequestError extends Error {
readonly code = 'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_REQUEST_FAILED';
@@ -176,114 +127,6 @@ function exactObject(
}
}
function currentUid(): number {
if (typeof process.getuid !== 'function') throw configurationFailure();
const uid = process.getuid();
if (!Number.isSafeInteger(uid) || uid < 0) throw configurationFailure();
return uid;
}
export function readCanonicalFile(
filePath: string,
maximumBytes: number,
mode: 'private' | 'public-integrity',
): Buffer {
if (
typeof filePath !== 'string' ||
!isAbsolute(filePath) ||
filePath.length > 4_096 ||
CONTROL_PATTERN.test(filePath)
) {
throw configurationFailure();
}
let before;
try {
before = lstatSync(filePath);
if (
!before.isFile() ||
before.isSymbolicLink() ||
before.size < 1 ||
before.size > maximumBytes ||
realpathSync(filePath) !== filePath
) {
throw configurationFailure();
}
} catch (error) {
if (
error instanceof ClusterPluginPackageManagementClientConfigurationError
) {
throw error;
}
throw configurationFailure();
}
const uid = currentUid();
const permissions = before.mode & 0o777;
if (
(mode === 'private' && (before.uid !== uid || permissions !== 0o600)) ||
(mode === 'public-integrity' && before.uid !== uid && before.uid !== 0) ||
(mode === 'public-integrity' && (permissions & 0o022) !== 0)
) {
throw configurationFailure();
}
let descriptor = -1;
let bytes: Buffer | undefined;
try {
descriptor = openSync(
filePath,
constants.O_RDONLY |
((constants as unknown as Readonly<Record<string, number>>).O_CLOEXEC ??
0) |
(constants.O_NOFOLLOW ?? 0),
);
const opened = fstatSync(descriptor);
if (
!opened.isFile() ||
opened.dev !== before.dev ||
opened.ino !== before.ino ||
opened.uid !== before.uid ||
opened.mode !== before.mode ||
opened.size !== before.size
) {
throw configurationFailure();
}
bytes = Buffer.alloc(opened.size);
let offset = 0;
while (offset < bytes.length) {
const count = readSync(
descriptor,
bytes,
offset,
bytes.length - offset,
offset,
);
if (count < 1) throw configurationFailure();
offset += count;
}
const after = fstatSync(descriptor);
if (
after.dev !== opened.dev ||
after.ino !== opened.ino ||
after.uid !== opened.uid ||
after.mode !== opened.mode ||
after.size !== opened.size
) {
throw configurationFailure();
}
return bytes;
} catch (error) {
bytes?.fill(0);
if (
error instanceof ClusterPluginPackageManagementClientConfigurationError
) {
throw error;
}
throw configurationFailure();
} finally {
if (descriptor >= 0) closeSync(descriptor);
}
}
function decodeUtf8(bytes: Buffer): string {
try {
return new TextDecoder('utf-8', { fatal: true }).decode(bytes);
@@ -812,201 +655,6 @@ function rawHeaderCount(rawHeaders: readonly string[], name: string): number {
return count;
}
export interface ClusterAuthenticatedManagementClientConfigurationSummary {
readonly schemaVersion: 1;
readonly managementPath: string;
readonly transport: 'https';
readonly clientCertificate: 'forbidden' | 'required';
}
interface PreparedClusterAuthenticatedManagementClientConfiguration {
readonly endpoint: URL;
readonly servername: string;
readonly port: number;
readonly requestTimeoutMs: number;
readonly caBytes: Buffer;
readonly clientCertificateBytes?: Buffer;
readonly clientPrivateKeyBytes?: Buffer;
dispose(): void;
}
function prepareClusterAuthenticatedManagementClientConfiguration(
configFile: string,
managementPath: string,
clientCertificate: 'forbidden' | 'required',
): PreparedClusterAuthenticatedManagementClientConfiguration {
if (
!Object.values(MANAGEMENT_CLIENT_POLICIES).some(
(policy) =>
policy.managementPath === managementPath &&
policy.clientCertificate === clientCertificate,
)
) {
throw configurationFailure();
}
let configBytes: Buffer | undefined;
let caBytes: Buffer | undefined;
let clientCertificateBytes: Buffer | undefined;
let clientPrivateKeyBytes: Buffer | undefined;
try {
configBytes = readCanonicalFile(configFile, MAX_CONFIG_BYTES, 'private');
const config = parseJson(configBytes);
exactObject(
config,
clientCertificate === 'required'
? [
'schemaVersion',
'endpoint',
'servername',
'caFile',
'clientCertificateFile',
'clientPrivateKeyFile',
'requestTimeoutMs',
]
: [
'schemaVersion',
'endpoint',
'servername',
'caFile',
'requestTimeoutMs',
],
);
if (
config.schemaVersion !== 1 ||
typeof config.endpoint !== 'string' ||
typeof config.servername !== 'string' ||
!DNS_NAME_PATTERN.test(config.servername) ||
isIP(config.servername) !== 0 ||
typeof config.caFile !== 'string' ||
(clientCertificate === 'required' &&
(typeof config.clientCertificateFile !== 'string' ||
typeof config.clientPrivateKeyFile !== 'string')) ||
!Number.isSafeInteger(config.requestTimeoutMs) ||
(config.requestTimeoutMs as number) < 1_000 ||
(config.requestTimeoutMs as number) > 30_000
) {
throw configurationFailure();
}
const servername = config.servername;
const requestTimeoutMs = config.requestTimeoutMs as number;
let endpoint: URL;
try {
endpoint = new URL(config.endpoint);
} catch {
throw configurationFailure();
}
if (
endpoint.protocol !== 'https:' ||
endpoint.username !== '' ||
endpoint.password !== '' ||
endpoint.search !== '' ||
endpoint.hash !== '' ||
endpoint.pathname !== managementPath ||
endpoint.hostname !== servername ||
isIP(endpoint.hostname) !== 0
) {
throw configurationFailure();
}
const port = endpoint.port === '' ? 443 : Number(endpoint.port);
if (!Number.isSafeInteger(port) || port < 1 || port > 65_535) {
throw configurationFailure();
}
caBytes = readCanonicalFile(
config.caFile as string,
MAX_CA_BYTES,
'public-integrity',
);
try {
new X509Certificate(caBytes);
} catch {
throw configurationFailure();
}
if (clientCertificate === 'required') {
clientCertificateBytes = readCanonicalFile(
config.clientCertificateFile as string,
MAX_CLIENT_CERTIFICATE_BYTES,
'public-integrity',
);
clientPrivateKeyBytes = readCanonicalFile(
config.clientPrivateKeyFile as string,
MAX_CLIENT_PRIVATE_KEY_BYTES,
'private',
);
try {
const certificate = new X509Certificate(clientCertificateBytes);
const privateKey = createPrivateKey(clientPrivateKeyBytes);
if (!certificate.checkPrivateKey(privateKey)) {
throw configurationFailure();
}
} catch (error) {
if (
error instanceof
ClusterPluginPackageManagementClientConfigurationError
) {
throw error;
}
throw configurationFailure();
}
}
let disposed = false;
return Object.freeze({
endpoint,
servername,
port,
requestTimeoutMs,
caBytes,
...(clientCertificateBytes === undefined
? {}
: {
clientCertificateBytes,
clientPrivateKeyBytes: clientPrivateKeyBytes!,
}),
dispose() {
if (disposed) return;
disposed = true;
caBytes?.fill(0);
clientCertificateBytes?.fill(0);
clientPrivateKeyBytes?.fill(0);
},
});
} catch (error) {
caBytes?.fill(0);
clientCertificateBytes?.fill(0);
clientPrivateKeyBytes?.fill(0);
if (
error instanceof ClusterPluginPackageManagementClientConfigurationError
) {
throw error;
}
throw configurationFailure();
} finally {
configBytes?.fill(0);
}
}
export function validateClusterAuthenticatedManagementClientConfiguration(
configFile: string,
kind: ClusterAuthenticatedManagementClientKind,
): Readonly<ClusterAuthenticatedManagementClientConfigurationSummary> {
const policy = MANAGEMENT_CLIENT_POLICIES[kind];
if (policy === undefined) throw configurationFailure();
const prepared = prepareClusterAuthenticatedManagementClientConfiguration(
configFile,
policy.managementPath,
policy.clientCertificate,
);
try {
return Object.freeze({
schemaVersion: 1,
managementPath: policy.managementPath,
transport: 'https',
clientCertificate: policy.clientCertificate,
});
} finally {
prepared.dispose();
}
}
export async function executeClusterAuthenticatedManagementClient<
Command,
Result,
@@ -1030,10 +678,9 @@ export async function executeClusterAuthenticatedManagementClient<
'validateResult',
].includes(key),
) ||
!Object.values(MANAGEMENT_CLIENT_POLICIES).some(
(policy) =>
policy.managementPath === protocol.managementPath &&
policy.clientCertificate === protocol.clientCertificate,
!isReviewedClusterAuthenticatedManagementClientProtocol(
protocol.managementPath,
protocol.clientCertificate,
) ||
typeof protocol.normalizeCommand !== 'function' ||
typeof protocol.validateResult !== 'function' ||
@@ -16,6 +16,10 @@ import {
type ClusterPluginPackageManagementClientRawConnection,
type ClusterPluginPackageManagementClientResult,
} from '../../management-support/pluginPackageManagementClient';
import {
probeClusterAuthenticatedManagementClientReadiness,
type ClusterAuthenticatedManagementClientReadiness,
} from '../../management-support/managementReadinessProbe';
const MAX_KUBERNETES_CONFIG_BYTES = 16 * 1024;
const MAX_KUBECONFIG_BYTES = 256 * 1024;
@@ -874,3 +878,94 @@ export async function executeClusterPluginPackageManagementKubernetesClient(
prepared?.dispose();
}
}
export async function probeClusterPluginPackageManagementKubernetesReadiness(
configFile: string,
kubernetesFile: string,
options: ClusterPluginPackageManagementKubernetesClientOptions = {},
): Promise<Readonly<ClusterAuthenticatedManagementClientReadiness>> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some((key) => key !== 'createRuntime') ||
(options.createRuntime !== undefined &&
typeof options.createRuntime !== 'function')
) {
throw configurationFailure();
}
let prepared: PreparedKubernetesClientConfiguration | undefined;
try {
prepared = await prepareKubernetesClientConfiguration(kubernetesFile);
const { config, kubeConfig, kubernetes } = prepared;
const runtime = (options.createRuntime ?? productionRuntime)(
kubeConfig,
kubernetes,
);
if (
!runtime ||
typeof runtime !== 'object' ||
typeof runtime.pods?.listNamespacedPod !== 'function' ||
typeof runtime.openPortForward !== 'function'
) {
throw configurationFailure();
}
const expectedHostname = `${MANAGEMENT_NAME}.${config.namespace}.svc`;
return await probeClusterAuthenticatedManagementClientReadiness(
configFile,
'package',
{
async connect(target) {
if (
target.hostname !== expectedHostname ||
target.port !== MANAGEMENT_PORT
) {
throw configurationFailure();
}
const list = await deadline(
runtime.pods.listNamespacedPod({
namespace: config.namespace,
labelSelector: MANAGEMENT_LABEL_SELECTOR,
limit: 3,
timeoutSeconds: Math.ceil(config.apiTimeoutMs / 1_000),
watch: false,
}),
config.apiTimeoutMs,
);
const podName = selectManagementPod(list, config.namespace);
return await deadline(
runtime.openPortForward({
namespace: config.namespace,
podName,
port: MANAGEMENT_PORT,
}),
config.apiTimeoutMs,
async (connection) => {
await connection.close();
},
);
},
},
);
} catch (error) {
if (
error instanceof ClusterPluginPackageManagementClientRequestError &&
error.cause instanceof
ClusterPluginPackageManagementKubernetesClientTunnelError
) {
throw error.cause;
}
if (
error instanceof
ClusterPluginPackageManagementKubernetesClientConfigurationError ||
error instanceof ClusterPluginPackageManagementKubernetesClientTunnelError ||
error instanceof ClusterPluginPackageManagementClientConfigurationError ||
error instanceof ClusterPluginPackageManagementClientRequestError
) {
throw error;
}
throw new ClusterPluginPackageManagementKubernetesClientTunnelError(error);
} finally {
prepared?.dispose();
}
}
@@ -4,8 +4,12 @@ import { spawn, type ChildProcess } from 'node:child_process';
import { constants } from 'node:os';
import { resolveQingLong3ClusterProductCommand } from './productCommand';
import { QingLong3ClusterProductContextError } from './productContext';
import { validateQingLong3ClusterProductContext } from './productContext';
import {
probeQingLong3ClusterProductContext,
QingLong3ClusterProductContextError,
QingLong3ClusterProductContextProbeError,
validateQingLong3ClusterProductContext,
} from './productContext';
const FORWARDED_SIGNALS = Object.freeze([
'SIGINT',
@@ -132,6 +136,14 @@ async function main(argv: readonly string[]): Promise<void> {
process.stdout.write(`${JSON.stringify(result)}\n`);
return;
}
if (resolution.kind === 'context-probe') {
const result = await probeQingLong3ClusterProductContext(
resolution.contextFile,
);
process.stdout.write(`${JSON.stringify(result)}\n`);
if (!result.allReady) process.exitCode = 69;
return;
}
invoke(resolution.targetFilePath, resolution.argv);
} catch (error) {
if (
@@ -152,6 +164,24 @@ async function main(argv: readonly string[]): Promise<void> {
process.exitCode = 78;
return;
}
if (
error instanceof QingLong3ClusterProductContextProbeError ||
(error !== null &&
typeof error === 'object' &&
'code' in error &&
error.code === 'QL3_CLUSTER_PRODUCT_CONTEXT_PROBE_FAILED')
) {
process.stderr.write(
`${JSON.stringify(
lowSensitivityFailure(
'QL3_CLUSTER_PRODUCT_CONTEXT_PROBE_FAILED',
'QingLong 3.0 Cluster operator context probe failed',
),
)}\n`,
);
process.exitCode = 69;
return;
}
process.stderr.write(
`${JSON.stringify(
lowSensitivityFailure(
@@ -14,6 +14,7 @@ export type QingLong3ClusterProductCommandResolution =
| Readonly<{ kind: 'help'; output: string }>
| Readonly<{ kind: 'version'; output: string }>
| Readonly<{ kind: 'context-validation'; contextFile: string }>
| Readonly<{ kind: 'context-probe'; contextFile: string }>
| Readonly<{
kind: 'invoke';
command: QingLong3ClusterProductCommandDefinition;
@@ -175,6 +176,7 @@ export function qingLong3ClusterProductHelp(): string {
'',
'Local operator commands:',
' context validate --context=/absolute/operator-context.json',
' context probe --context=/absolute/operator-context.json',
'',
'Use `ql3-cluster-admin <command> --help` for command-specific usage.',
'Use `--context=/absolute/operator-context.json` to inject only stable client paths.',
@@ -200,7 +202,7 @@ export function resolveQingLong3ClusterProductCommand(
if (argv[0] === 'context') {
if (
argv.length !== 3 ||
argv[1] !== 'validate' ||
(argv[1] !== 'validate' && argv[1] !== 'probe') ||
!argv[2]!.startsWith('--context=') ||
argv[2] === '--context='
) {
@@ -215,7 +217,7 @@ export function resolveQingLong3ClusterProductCommand(
resolveInstalledTarget(distRoot, definition);
}
return Object.freeze({
kind: 'context-validation',
kind: argv[1] === 'validate' ? 'context-validation' : 'context-probe',
contextFile: argv[2]!.slice('--context='.length),
});
}
@@ -10,9 +10,15 @@ import {
import { isAbsolute } from 'node:path';
import { TextDecoder } from 'node:util';
import { validateClusterAuthenticatedManagementClientConfiguration } from '../management-support/pluginPackageManagementClient';
import {
validateClusterAuthenticatedManagementClientConfiguration,
} from '../management-support/pluginPackageManagementClient';
import type { ClusterAuthenticatedManagementClientKind } from '../management-support/pluginPackageManagementClient';
import { validateClusterPluginPackageManagementKubernetesConfiguration } from '../plugin-package/management/pluginPackageManagementKubernetesClient';
import { probeClusterAuthenticatedManagementClientReadiness } from '../management-support/managementReadinessProbe';
import {
probeClusterPluginPackageManagementKubernetesReadiness,
validateClusterPluginPackageManagementKubernetesConfiguration,
} from '../plugin-package/management/pluginPackageManagementKubernetesClient';
const MAXIMUM_CONTEXT_BYTES = 64 * 1024;
const MAXIMUM_PATH_BYTES = 4_096;
@@ -57,6 +63,22 @@ export interface QingLong3ClusterProductContextValidation {
readonly mutation: false;
}
export interface QingLong3ClusterProductContextProbe {
readonly schemaVersion: 1;
readonly component: 'qinglong3-cluster-product-cli';
readonly event: 'context_probed';
readonly commandCount: number;
readonly commands: readonly Readonly<{
name: ContextCommandName;
transport: 'https' | 'kubernetes-port-forward';
status: 'ready' | 'not_ready';
}>[];
readonly allReady: boolean;
readonly requestMethod: 'GET';
readonly requestPath: '/readyz';
readonly mutation: false;
}
const CONTEXT_COMMAND_CLIENT_KINDS: Readonly<
Record<ContextCommandName, ClusterAuthenticatedManagementClientKind>
> = Object.freeze({
@@ -78,6 +100,15 @@ export class QingLong3ClusterProductContextError extends TypeError {
}
}
export class QingLong3ClusterProductContextProbeError extends Error {
readonly code = 'QL3_CLUSTER_PRODUCT_CONTEXT_PROBE_FAILED';
constructor() {
super('QingLong 3.0 Cluster operator context probe failed');
this.name = 'QingLong3ClusterProductContextProbeError';
}
}
function invalid(): never {
throw new QingLong3ClusterProductContextError();
}
@@ -328,3 +359,62 @@ export async function validateQingLong3ClusterProductContext(
throw new QingLong3ClusterProductContextError();
}
}
export async function probeQingLong3ClusterProductContext(
contextFile: string,
): Promise<Readonly<QingLong3ClusterProductContextProbe>> {
try {
await validateQingLong3ClusterProductContext(contextFile);
const context = loadQingLong3ClusterProductContext(contextFile);
const commands: Array<
QingLong3ClusterProductContextProbe['commands'][number]
> = [];
for (const name of CONTEXT_COMMANDS) {
const command = context.commands[name];
if (command === undefined) continue;
const result =
name === 'package-kubernetes'
? await probeClusterPluginPackageManagementKubernetesReadiness(
command.configFile,
command.kubernetesFile!,
)
: await probeClusterAuthenticatedManagementClientReadiness(
command.configFile,
CONTEXT_COMMAND_CLIENT_KINDS[name],
);
commands.push(
Object.freeze({
name,
transport:
name === 'package-kubernetes'
? 'kubernetes-port-forward'
: result.transport,
status: result.ready ? 'ready' : 'not_ready',
}),
);
}
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-cluster-product-cli',
event: 'context_probed',
commandCount: commands.length,
commands: Object.freeze(commands),
allReady: commands.every(({ status }) => status === 'ready'),
requestMethod: 'GET',
requestPath: '/readyz',
mutation: false,
});
} catch (error) {
if (error instanceof QingLong3ClusterProductContextError) throw error;
if (
error !== null &&
typeof error === 'object' &&
'code' in error &&
typeof error.code === 'string' &&
error.code.endsWith('CONFIG_INVALID')
) {
throw new QingLong3ClusterProductContextError();
}
throw new QingLong3ClusterProductContextProbeError();
}
}
@@ -20,6 +20,10 @@ const {
ClusterPluginPackageManagementClientRequestError,
executeClusterPluginPackageManagementClient,
} = require('@qinglong/cluster-admin/plugin-package-management-client');
const publicClientModule = require('@qinglong/cluster-admin/plugin-package-management-client');
const {
probeClusterAuthenticatedManagementClientReadiness,
} = require('../dist/management-support/managementReadinessProbe.js');
const CA_CERT = resolve(
__dirname,
@@ -33,12 +37,35 @@ const SERVER_CERT = resolve(
__dirname,
'../../ql3-cluster-control/test/fixtures/mtls/server-cert.pem',
);
const CLIENT_KEY = resolve(
__dirname,
'../../ql3-cluster-control/test/fixtures/mtls/client-key.pem',
);
const CLIENT_CERT = resolve(
__dirname,
'../../ql3-cluster-control/test/fixtures/mtls/client-cert.pem',
);
const CLIENT_CLI = resolve(
__dirname,
'../dist/plugin-package/management/pluginPackageManagementClientCli.js',
);
const ASSERTION = 'eyJhbGciOiJFUzI1NiJ9.eyJzdWIiOiJvcGVyYXRvciJ9.c2ln';
test('keeps owned TLS preparation out of the public client subpath', () => {
assert.equal(
publicClientModule.prepareClusterAuthenticatedManagementClientConfiguration,
undefined,
);
assert.equal(
publicClientModule.prepareClusterAuthenticatedManagementClientKindConfiguration,
undefined,
);
assert.equal(
publicClientModule.probeClusterAuthenticatedManagementClientReadiness,
undefined,
);
});
function inspectCommand(operation = 'plugin-package.inspect') {
return {
schemaVersion: 1,
@@ -392,13 +419,14 @@ function createClientFiles(port, command = inspectCommand()) {
};
}
async function startServer(handler) {
async function startServer(handler, options = {}) {
const server = createServer(
{
key: readFileSync(SERVER_KEY),
cert: readFileSync(SERVER_CERT),
minVersion: 'TLSv1.3',
maxVersion: 'TLSv1.3',
...options,
},
handler,
);
@@ -475,6 +503,142 @@ test('sends one TLS 1.3 management command and validates the low-sensitive resul
}
});
test('probes only the fixed TLS readiness endpoint without management authority', async () => {
const received = [];
let ready = true;
const fixture = await startServer((request, response) => {
const chunks = [];
request.on('data', (chunk) => chunks.push(chunk));
request.once('end', () => {
received.push({
method: request.method,
path: request.url,
authorization: request.headers.authorization,
contentType: request.headers['content-type'],
bodyBytes: Buffer.concat(chunks).length,
protocol: request.socket.getProtocol(),
});
sendJson(response, ready ? 200 : 503, {
schemaVersion: 1,
status: ready ? 'ready' : 'not_ready',
});
});
});
const files = createClientFiles(fixture.port);
try {
assert.deepEqual(
await probeClusterAuthenticatedManagementClientReadiness(
files.paths.configFile,
'package',
),
{ schemaVersion: 1, transport: 'https', ready: true },
);
ready = false;
assert.deepEqual(
await probeClusterAuthenticatedManagementClientReadiness(
files.paths.configFile,
'package',
),
{ schemaVersion: 1, transport: 'https', ready: false },
);
assert.deepEqual(received, [
{
method: 'GET',
path: '/readyz',
authorization: undefined,
contentType: undefined,
bodyBytes: 0,
protocol: 'TLSv1.3',
},
{
method: 'GET',
path: '/readyz',
authorization: undefined,
contentType: undefined,
bodyBytes: 0,
protocol: 'TLSv1.3',
},
]);
} finally {
await fixture.close();
rmSync(files.directory, { recursive: true, force: true });
}
});
test('presents the reviewed client certificate for mTLS readiness', async () => {
let authorized = false;
const fixture = await startServer(
(request, response) => {
authorized = request.socket.authorized;
sendJson(response, 200, { schemaVersion: 1, status: 'ready' });
},
{
ca: readFileSync(CA_CERT),
requestCert: true,
rejectUnauthorized: true,
},
);
const files = createClientFiles(fixture.port);
const clientKeyFile = join(files.directory, 'client-key.pem');
privateWrite(clientKeyFile, readFileSync(CLIENT_KEY, 'utf8'));
privateWrite(files.paths.configFile, {
schemaVersion: 1,
endpoint: `https://localhost:${fixture.port}/api/v3/runs/management`,
servername: 'localhost',
caFile: CA_CERT,
clientCertificateFile: CLIENT_CERT,
clientPrivateKeyFile: clientKeyFile,
requestTimeoutMs: 1_000,
});
try {
assert.deepEqual(
await probeClusterAuthenticatedManagementClientReadiness(
files.paths.configFile,
'run',
),
{ schemaVersion: 1, transport: 'https', ready: true },
);
assert.equal(authorized, true);
} finally {
await fixture.close();
rmSync(files.directory, { recursive: true, force: true });
}
});
test('readiness probe rejects unreviewed status and bounded response drift', async () => {
let behavior = 'wrong-status';
const fixture = await startServer((_request, response) => {
if (behavior === 'wrong-status') {
sendJson(response, 200, { schemaVersion: 1, status: 'live' });
return;
}
if (behavior === 'redirect') {
sendJson(response, 302, { schemaVersion: 1, status: 'ready' });
return;
}
response.writeHead(200, {
'content-type': 'application/json; charset=utf-8',
});
response.end(Buffer.alloc(1_025, 0x61));
});
const files = createClientFiles(fixture.port);
try {
for (const next of ['wrong-status', 'redirect', 'oversized']) {
behavior = next;
await assert.rejects(
probeClusterAuthenticatedManagementClientReadiness(
files.paths.configFile,
'package',
),
ClusterPluginPackageManagementClientRequestError,
);
}
} finally {
await fixture.close();
rmSync(files.directory, { recursive: true, force: true });
}
});
test('permits and validates exactly the fourteen public management operations', async () => {
const received = [];
const fixture = await startServer((request, response) => {
@@ -20,6 +20,7 @@ const {
ClusterPluginPackageManagementKubernetesClientTunnelError,
executeClusterPluginPackageManagementKubernetesClient,
openClusterPluginPackageManagementPortForward,
probeClusterPluginPackageManagementKubernetesReadiness,
} = require(
'@qinglong/cluster-admin/plugin-package-management-kubernetes-client'
);
@@ -302,6 +303,85 @@ test('uses one ready Pod tunnel and preserves end-to-end TLS 1.3 hostname verifi
}
});
test('probes readiness through one reviewed Pod tunnel without an assertion or command', async () => {
const requests = [];
const server = await startServer((request, response) => {
const chunks = [];
request.on('data', (chunk) => chunks.push(chunk));
request.once('end', () => {
requests.push({
method: request.method,
path: request.url,
authorization: request.headers.authorization,
bodyBytes: Buffer.concat(chunks).length,
protocol: request.socket.getProtocol(),
servername: request.socket.servername,
});
sendJson(response, { schemaVersion: 1, status: 'ready' });
});
});
const files = createClientFiles();
const calls = { lists: 0, tunnels: 0, closes: 0 };
try {
const result =
await probeClusterPluginPackageManagementKubernetesReadiness(
files.paths.configFile,
files.paths.kubernetesFile,
{
createRuntime() {
return {
pods: {
async listNamespacedPod() {
calls.lists += 1;
return {
items: [
readyPod(
'ql3-plugin-package-management-aaaaa-11111',
),
],
};
},
},
async openPortForward() {
calls.tunnels += 1;
const stream = connectTcp({
host: '127.0.0.1',
port: server.port,
});
return {
stream,
close() {
calls.closes += 1;
stream.end();
},
};
},
};
},
},
);
assert.deepEqual(result, {
schemaVersion: 1,
transport: 'https',
ready: true,
});
assert.deepEqual(calls, { lists: 1, tunnels: 1, closes: 1 });
assert.deepEqual(requests, [
{
method: 'GET',
path: '/readyz',
authorization: undefined,
bodyBytes: 0,
protocol: 'TLSv1.3',
servername: SERVICE_HOST,
},
]);
} finally {
await server.close();
rmSync(files.directory, { recursive: true, force: true });
}
});
test('rejects ambient, executable, proxied, insecure, and file-backed kubeconfig authority', async () => {
const cases = [
kubeconfig({ cluster: { 'insecure-skip-tls-verify': true } }),
@@ -1,10 +1,11 @@
const assert = require('node:assert/strict');
const { spawnSync } = require('node:child_process');
const { spawn, spawnSync } = require('node:child_process');
const { EventEmitter } = require('node:events');
const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');
const { test } = require('node:test');
const { createServer } = require('node:https');
const packageRoot = path.resolve(__dirname, '..');
const moduleDirectory = path.join(packageRoot, 'dist', 'product-cli');
@@ -21,6 +22,18 @@ const privateKeyFixture = path.join(
'fixtures',
'management-service-key.pem',
);
const localhostCertificateFixture = path.resolve(
packageRoot,
'../ql3-cluster-control/test/fixtures/mtls/server-cert.pem',
);
const localhostCaFixture = path.resolve(
packageRoot,
'../ql3-cluster-control/test/fixtures/mtls/ca-cert.pem',
);
const localhostPrivateKeyFixture = path.resolve(
packageRoot,
'../ql3-cluster-control/test/fixtures/mtls/server-key.pem',
);
const manifest = JSON.parse(
fs.readFileSync(path.join(packageRoot, 'package.json'), 'utf8'),
);
@@ -49,6 +62,65 @@ function runCli(args) {
});
}
function runCliAsync(args) {
return new Promise((resolvePromise, reject) => {
const child = spawn(process.execPath, [cliPath, ...args], {
cwd: packageRoot,
stdio: ['ignore', 'pipe', 'pipe'],
});
const stdout = [];
const stderr = [];
child.stdout.on('data', (chunk) => stdout.push(chunk));
child.stderr.on('data', (chunk) => stderr.push(chunk));
child.once('error', reject);
child.once('close', (status, signal) => {
resolvePromise({
status,
signal,
stdout: Buffer.concat(stdout).toString('utf8'),
stderr: Buffer.concat(stderr).toString('utf8'),
});
});
});
}
async function startReadinessServer(status) {
const server = createServer(
{
key: fs.readFileSync(localhostPrivateKeyFixture),
cert: fs.readFileSync(localhostCertificateFixture),
minVersion: 'TLSv1.3',
maxVersion: 'TLSv1.3',
},
(request, response) => {
assert.equal(request.method, 'GET');
assert.equal(request.url, '/readyz');
assert.equal(request.headers.authorization, undefined);
const body = Buffer.from(
JSON.stringify({ schemaVersion: 1, status: status.value }),
);
response.writeHead(status.value === 'ready' ? 200 : 503, {
'content-type': 'application/json; charset=utf-8',
'content-length': String(body.length),
});
response.end(body);
},
);
await new Promise((resolvePromise, reject) => {
server.once('error', reject);
server.listen(0, '127.0.0.1', resolvePromise);
});
return {
port: server.address().port,
close: () =>
new Promise((resolvePromise, reject) => {
server.close((error) =>
error ? reject(error) : resolvePromise(),
);
}),
};
}
function privateFile(directory, name, contents) {
const filePath = path.join(directory, name);
fs.writeFileSync(filePath, contents, { mode: 0o600 });
@@ -521,6 +593,84 @@ test('validates the complete operator context offline without operational author
assert.equal(validated.stdout.includes('bounded-token'), false);
});
test('probes a context with fixed read-only readiness semantics and exit status', async (t) => {
const status = { value: 'ready' };
const server = await startReadinessServer(status);
t.after(() => server.close());
const directory = fs.realpathSync(
fs.mkdtempSync(path.join(os.tmpdir(), 'ql3-cluster-probe-context-')),
);
t.after(() => fs.rmSync(directory, { recursive: true, force: true }));
const caFile = privateFile(
directory,
'ca.pem',
fs.readFileSync(localhostCaFixture),
);
const configFile = privateFile(
directory,
'package.json',
JSON.stringify({
schemaVersion: 1,
endpoint: `https://localhost:${server.port}/api/v3/plugin-packages/management`,
servername: 'localhost',
caFile,
requestTimeoutMs: 1_000,
}),
);
const contextFile = privateFile(
directory,
'operator-context.json',
JSON.stringify({
schemaVersion: 1,
commands: { package: { configFile } },
}),
);
assert.deepEqual(
resolveQingLong3ClusterProductCommand(
['context', 'probe', `--context=${contextFile}`],
moduleDirectory,
),
{ kind: 'context-probe', contextFile },
);
const ready = await runCliAsync([
'context',
'probe',
`--context=${contextFile}`,
]);
assert.equal(
ready.status,
0,
JSON.stringify({ stdout: ready.stdout, stderr: ready.stderr }),
);
assert.equal(ready.stderr, '');
assert.deepEqual(JSON.parse(ready.stdout), {
schemaVersion: 1,
component: 'qinglong3-cluster-product-cli',
event: 'context_probed',
commandCount: 1,
commands: [{ name: 'package', transport: 'https', status: 'ready' }],
allReady: true,
requestMethod: 'GET',
requestPath: '/readyz',
mutation: false,
});
assert.equal(ready.stdout.includes(directory), false);
assert.equal(ready.stdout.includes('localhost'), false);
status.value = 'not_ready';
const notReady = await runCliAsync([
'context',
'probe',
`--context=${contextFile}`,
]);
assert.equal(notReady.status, 69);
assert.equal(notReady.stderr, '');
const fact = JSON.parse(notReady.stdout);
assert.equal(fact.allReady, false);
assert.equal(fact.commands[0].status, 'not_ready');
});
test('context validation fails closed for invalid client configuration and syntax', (t) => {
const fixture = validContextFixture(t);
fs.writeFileSync(fixture.commands.run.configFile, '{}', { mode: 0o600 });
@@ -544,6 +694,8 @@ test('context validation fails closed for invalid client configuration and synta
['context', 'validate'],
['context', 'validate', '--context'],
['context', 'validate', '--context='],
['context', 'probe'],
['context', 'probe', '--context='],
['context', 'inspect', `--context=${fixture.contextFile}`],
]) {
const rejected = resolveQingLong3ClusterProductCommand(
@@ -1,6 +1,7 @@
'use strict';
const { execFileSync } = require('node:child_process');
const { resolve } = require('node:path');
const IMAGE_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._/:@-]{0,255}$/u;
const ENTRYPOINT = [
@@ -77,9 +78,29 @@ function runImage(image, args) {
}
function runOperatorContextContract(image) {
const fixtureRoot = resolve(
__dirname,
'../packages/ql3-cluster-control/test/fixtures/mtls',
);
const readinessServerSource = String.raw`
const { readFileSync, writeFileSync } = require('node:fs');
const { createServer } = require('node:https');
const server = createServer({ key: readFileSync('/evidence/server-key.pem'), cert: readFileSync('/evidence/server-cert.pem'), minVersion: 'TLSv1.3', maxVersion: 'TLSv1.3' }, (request, response) => {
const chunks = [];
request.on('data', (chunk) => chunks.push(chunk));
request.once('end', () => {
writeFileSync('/tmp/readiness-observation.json', JSON.stringify({ method: request.method, path: request.url, authorization: request.headers.authorization ?? null, bodyBytes: Buffer.concat(chunks).length }), { mode: 0o600 });
const body = Buffer.from(JSON.stringify({ schemaVersion: 1, status: 'ready' }));
response.writeHead(200, { 'content-type': 'application/json; charset=utf-8', 'content-length': String(body.length) });
response.end(body);
});
});
server.listen(0, '127.0.0.1', () => writeFileSync('/tmp/readiness-port', String(server.address().port), { mode: 0o600 }));
process.once('SIGTERM', () => server.close(() => process.exit(0)));
`;
const source = String.raw`
const { spawnSync } = require('node:child_process');
const { writeFileSync } = require('node:fs');
const { spawn, spawnSync } = require('node:child_process');
const { existsSync, readFileSync, writeFileSync } = require('node:fs');
const { rootCertificates } = require('node:tls');
const facade = '/opt/qinglong/node_modules/@qinglong/cluster-admin/dist/product-cli/cli.js';
const config = '/tmp/run-client.json';
@@ -104,12 +125,27 @@ const validated = spawnSync(process.execPath, [facade, 'context', 'validate', '-
let validationFact;
try { validationFact = JSON.parse(validated.stdout); } catch { process.exit(25); }
if (validated.status !== 0 || validated.stderr !== '' || validationFact.event !== 'context_valid' || validationFact.commandCount !== 1 || validationFact.networkAccess !== false || validationFact.mutation !== false || JSON.stringify(validationFact.commands) !== JSON.stringify([{ name: 'package', transport: 'https', clientCertificate: 'forbidden' }]) || validated.stdout.includes('/tmp/') || validated.stdout.includes('manager.example.test')) process.exit(26);
const readinessServer = spawn(process.execPath, ['-e', ${JSON.stringify(readinessServerSource)}], { stdio: 'ignore' });
const waitArray = new Int32Array(new SharedArrayBuffer(4));
for (let attempt = 0; attempt < 200 && !existsSync('/tmp/readiness-port'); attempt += 1) Atomics.wait(waitArray, 0, 0, 10);
if (!existsSync('/tmp/readiness-port')) process.exit(27);
const readinessPort = Number(readFileSync('/tmp/readiness-port', 'utf8'));
writeFileSync(ca, readFileSync('/evidence/ca-cert.pem'), { mode: 0o600 });
writeFileSync(config, JSON.stringify({ schemaVersion: 1, endpoint: 'https://localhost:' + readinessPort + '/api/v3/plugin-packages/management', servername: 'localhost', caFile: ca, requestTimeoutMs: 1000 }), { mode: 0o600 });
const probed = spawnSync(process.execPath, [facade, 'context', 'probe', '--context=' + context], { encoding: 'utf8' });
readinessServer.kill('SIGTERM');
let probeFact;
try { probeFact = JSON.parse(probed.stdout); } catch { process.exit(28); }
if (probed.status !== 0 || probed.stderr !== '' || probeFact.event !== 'context_probed' || probeFact.commandCount !== 1 || probeFact.allReady !== true || probeFact.requestMethod !== 'GET' || probeFact.requestPath !== '/readyz' || probeFact.mutation !== false || JSON.stringify(probeFact.commands) !== JSON.stringify([{ name: 'package', transport: 'https', status: 'ready' }]) || probed.stdout.includes('/tmp/') || probed.stdout.includes('localhost')) process.exit(29);
let readinessObservation;
try { readinessObservation = JSON.parse(readFileSync('/tmp/readiness-observation.json', 'utf8')); } catch { process.exit(30); }
if (JSON.stringify(readinessObservation) !== JSON.stringify({ method: 'GET', path: '/readyz', authorization: null, bodyBytes: 0 })) process.exit(31);
writeFileSync(context, JSON.stringify({ schemaVersion: 1, commands: { run: { configFile: config, assertionFile: assertion } } }), { mode: 0o600 });
const rejected = spawnSync(process.execPath, [facade, 'run', '--context=' + context, '--command=' + command, '--assertion=' + assertion], { encoding: 'utf8' });
let rejectedFailure;
try { rejectedFailure = JSON.parse(rejected.stderr); } catch { process.exit(23); }
if (rejected.status !== 78 || rejectedFailure.code !== 'QL3_CLUSTER_PRODUCT_CONTEXT_INVALID' || rejected.stdout !== '' || rejected.stderr.includes('/tmp/') || rejected.stderr.includes('assertion.jwt')) process.exit(24);
process.stdout.write(JSON.stringify({ schemaVersion: 1, injected: true, contextPreflight: true, secretFieldsRejected: true }));
process.stdout.write(JSON.stringify({ schemaVersion: 1, injected: true, contextPreflight: true, contextReadiness: true, secretFieldsRejected: true }));
`;
const output = docker([
'run',
@@ -131,6 +167,12 @@ process.stdout.write(JSON.stringify({ schemaVersion: 1, injected: true, contextP
'0.25',
'--tmpfs',
'/tmp:rw,noexec,nosuid,nodev,size=8m,mode=700,uid=10001,gid=10001',
'--volume',
`${resolve(fixtureRoot, 'ca-cert.pem')}:/evidence/ca-cert.pem:ro`,
'--volume',
`${resolve(fixtureRoot, 'server-cert.pem')}:/evidence/server-cert.pem:ro`,
'--volume',
`${resolve(fixtureRoot, 'server-key.pem')}:/evidence/server-key.pem:ro`,
'--entrypoint',
'node',
image,
@@ -147,6 +189,7 @@ process.stdout.write(JSON.stringify({ schemaVersion: 1, injected: true, contextP
result?.schemaVersion !== 1 ||
result?.injected !== true ||
result?.contextPreflight !== true ||
result?.contextReadiness !== true ||
result?.secretFieldsRejected !== true
) {
fail('operator context contract drifted');
@@ -203,6 +246,7 @@ function main() {
commandCount: COMMANDS.length,
operatorContext: true,
contextPreflight: true,
contextReadiness: true,
isolation: Object.freeze({
readOnlyRoot: true,
network: 'none',
+3 -2
View File
@@ -289,8 +289,8 @@ function auditClusterImageCiWorkflow(
);
requirePattern(
adminProductLiveContract,
/runOperatorContextContract\(image\);[\s\S]*operatorContext: true,[\s\S]*contextPreflight: true/,
'native admin image contract must verify owner-private operator context injection and offline preflight',
/runOperatorContextContract\(image\);[\s\S]*operatorContext: true,[\s\S]*contextPreflight: true,[\s\S]*contextReadiness: true/,
'native admin image contract must verify owner-private operator context injection, offline preflight and read-only readiness',
);
requirePattern(
source,
@@ -335,6 +335,7 @@ function auditClusterImageCiWorkflow(
clusterAdminProductFacade: true,
clusterAdminOperatorContext: true,
clusterAdminContextPreflight: true,
clusterAdminContextReadiness: true,
ociAttestations: true,
osVulnerabilityScan: {
scanner: 'trivy@0.70.0',
@@ -29,6 +29,7 @@ test('accepts the reviewed native CI and digest release contracts', () => {
clusterAdminProductFacade: true,
clusterAdminOperatorContext: true,
clusterAdminContextPreflight: true,
clusterAdminContextReadiness: true,
ociAttestations: true,
osVulnerabilityScan: {
scanner: 'trivy@0.70.0',
@@ -145,6 +146,24 @@ test('rejects a Cluster Admin live gate that omits offline context preflight', (
);
});
test('rejects a Cluster Admin live gate that omits read-only context readiness', () => {
const contract = fs.readFileSync(
path.join(root, 'scripts/ql3-cluster-admin-product-live-contract.cjs'),
'utf8',
);
assert.throws(
() =>
auditClusterImageCiWorkflow(
ciSource,
contract.replace(
'contextPreflight: true,\n contextReadiness: true',
'contextPreflight: true,\n contextReadiness: false',
),
),
/read-only readiness/,
);
});
test('rejects removal of the native cluster-admin image gate', () => {
const mutated = ciSource.replace(
'image_arch: arm64\n image: admin',
+2 -2
View File
@@ -340,10 +340,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
rootSourceFileRoles: clusterAdmin.rootSourceFileRoles,
},
{
sourceFiles: 97,
sourceFiles: 99,
rootSourceFiles: 1,
rootSourceLines: 61,
nestedSourceFiles: 96,
nestedSourceFiles: 98,
rootSourceFileRoles: {
'modelInvocationMigrationCli.ts': 'binary_entry',
},