diff --git a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md index 9758bc8c..883d0768 100644 --- a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md +++ b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md @@ -11,6 +11,27 @@ 最新增量证据（2026-08-16）： +- D-341/ADR-0433（已接受；真实公开 catalog 运行待实际 release tag）：Cluster 最后一跳不再交给裸 + `kubectl apply -f locked.yaml`。新增工作站级 `cluster.deployment.preflight|apply|receipt.audit`，从 owner-private canonical + command 出发独立复验 Kubernetes v2 lock/report、locked manifest、release-set/catalog/workflow/image/annotation 闭包，并固定 + kubectl executable digest、kubeconfig digest/context、`kube-system` Namespace UID 与 field manager + `qinglong3-catalog-lock`。kubeconfig `exec`/`auth-provider`、ambient namespace/HOME、`--force-conflicts`、symlink/replaceable input、 + lock/target 漂移均失败关闭。preflight 只执行 pinned kubectl 的 server-side strict dry-run；apply 在再次 dry-run 后执行显式 + server-side apply，再以同一 executable 的 managed-fields convergence read 复验 live UID/resourceVersion、期望字段、image/catalog + authority 与 Apply ownership；不用会隐式启动 ambient `diff` 的 `kubectl diff`。成功才发布绑定 command/preflight/lock/cluster/ + executable/step transcript digest 的 no-replace receipt；多资源非原子事实显式为 false，响应丢失使用同 command 幂等重放后收敛， + 不猜测自动删除或回滚。每个 kubectl 调用使用一次性私有 HOME/XDG/TMP，结束清理。实现没有新增 workspace package、生产依赖、 + controller、webhook、CRD、RBAC、ServiceAccount、Pod、listener、timer、数据库或 Pool；Local/Edge/Standalone 零导入、零制品增量。 + 定向契约 10/10；完整 backend 1,327 项为 1,325 pass/2 条件 skip/0 fail；18-package clean build/test 退出 0。10 项架构/部署 + 审计与 14 档 Local artifact 全部 compatible，package boundary 保持 18 packages、`singleSourcePackages=[]`、 + `shallowSourcePackages=[]`；最小 Edge 仍为 2,589,890 bytes/315 files，14 档字节数均与 D-340 一致。Cluster Admin exact + dry-run pack 保持 250 files、271,238-byte tarball、1,690,196-byte unpacked。隔离三节点 K3s `v1.34.3+k3s1`/arm64 真实运行 + 6 个资源、4 个零副本 Deployment，server-side dry-run/apply/convergence/offline receipt audit 全部通过；最终 preflight/receipt + digest 为 `sha256:7c6db6236a704cd7791f94268b3883d6385a75a993a14d1611791f885fb65404`/ + `sha256:58817a667bccf28c068fff40619cf13d2d0a1be84153de66a137b005feb536ba`，容器、网络和仓库 `.kube` cache 零残留。经允许 + 重跑 PostgreSQL 18.6 arm64 physical HA，142/142、timeline `1→2`，报告 SHA-256 + `f5f4398229d0122e86f159bed005111b7f5ecacee9ce38ab6d532b0986b540cf`，离线审计通过且容器、卷、网络零残留。live 使用的 + synthetic lock 只验证目标 apply 语义，不冒充尚未产生的公开 GHCR catalog。 - D-340/ADR-0432（已接受；真实公开 catalog 运行待实际 release tag）：Local/Compose 最后一跳不再接受裸 `image`。现有 `@qinglong/local-owner-cli` 的 prepare/upgrade 只接收 owner-private `releaseSelection.path + expectedSelectionDigest`，以 `O_NOFOLLOW` stable descriptor 有界读取不超过 64 KiB 的 canonical v2 selection，并重新验证 self-digest、3.x release identity、release-set/catalog manifest/catalog report digest 闭包、exact workflow identity、immutable catalog reference、唯一 GHCR Local application digest 与 explicit root policy。Compose revision/active selection 升为 `qinglong/local-compose-image-selection@v2`，持久保存完整 catalog/release authority；rollback 精确复制目标 revision authority，Preflight、Apply、Restore、Evidence 与 Status 共用同一 fail-closed parser。prepare/upgrade 仍只发布 revision，不联网、不执行 rollout、不修改数据库，也不新增 package、生产依赖、常驻进程或 Cluster 对象；低配设备每次显式命令只增加一次最多 64 KiB 的私有文件读取、canonical JSON 校验和 SHA-256，Cluster 路径不变。旧 v1 裸 image 在尚未正式发布的 3.0 中失败关闭，孵化环境须从原 catalog-bound selection 重新 prepare。Local 定向 30/30、物理 Edge Compose storage 6/6；Local Owner 全量 171 项为 166 pass/5 条件 skip/0 fail，backend 共 1,317 项为 1,315 pass/2 条件 skip/0 fail，18-package clean build/test 退出 0。10 项架构/部署审计与 14 档 Local artifact 全部 compatible，package boundary 保持 18 packages、`singleSourcePackages=[]`、`shallowSourcePackages=[]`；Edge/Standalone 默认制品为 2,589,890/2,589,968 bytes，application 为 3,632,769/3,632,889 bytes，application-api 为 3,800,322/3,800,466 bytes，AI 为 3,069,143/3,069,233 bytes，application+AI 为 4,493,043/4,493,175 bytes，MCP 为 7,315,930/7,316,038 bytes。Cluster Admin exact dry-run pack 为 250 files、271,238-byte tarball、1,690,196-byte unpacked。经允许重新运行的 PostgreSQL 18.6 arm64 physical HA 通过 142/142、timeline `1→2`，报告 SHA-256 为 `07c914551ec700da26b42cd42760ccb3b28ad31266a8bae5f62dee38eb97e6a9`，离线审计通过且无 `ql3-ha-*` Docker 资源残留。测试中的 synthetic selection 只验证本地 Compose 兼容性，不冒充公开 catalog ceremony；公开 GHCR catalog 尚未实际产生，因此不宣称真实线上验签成功。 - D-339/ADR-0431（已接受；真实公开 catalog 运行待实际 release tag）：D337 的 deployment-lock CLI 不再接受一份无法证明来源的松散 `--release-set`；Local/Kubernetes create/audit 必须同时接收 exact source repository 和 D338 生成的 owner-private three-file `--consumption-bundle`，先完整离线重建 release-set、raw OCI manifest、catalog plan/receipt、六步 argv/transcript digest 与 self-digest report，再把同一次 audit 读取的 release-set 对象交给 materializer，避免验真后重新按裸路径读取。Local selection 与 Kubernetes lock schema 升为 v2，显式绑定 consumption schema、source repository、exact workflow identity、catalog immutable reference、manifest digest、consumption report digest、release-set digest 和 `discoveryTagAuthority=none`；Cluster 被改写资源与 Pod template 也新增 catalog manifest/report digest annotations。旧 `--release-set`、bundle symlink/open shape、identity/scope/owner/image-count/digest 漂移均在创建任何输出前失败关闭。offline audit 诚实保持 `externalToolResultsReplayed=false`；本 Gate 不联网、不访问 Kubernetes API、不执行 Compose rollout/`kubectl apply`、不修改数据库，也不新增 package、生产依赖或运行期组件。供应链工作仍留在可信工作站，低配设备只接收 catalog-bound Local v2 selection 与一个 immutable image reference；Cluster 复用既有 post-render 流程。完整定向发布链 123/123；backend 共 1,317 项，1,315 pass/2 条件 skip/0 fail；18-package clean build/test 退出 0，package boundary 保持 18 packages、`singleSourcePackages=[]`、`shallowSourcePackages=[]`。10 项架构/部署审计与 14 档 Local artifact 全部 compatible，默认 Edge/Standalone 为 2,589,890/2,589,968 bytes，application+AI 为 4,493,043/4,493,175 bytes，MCP 为 7,315,930/7,316,038 bytes；Cluster Admin pack 保持 250 files、271,238-byte tarball、1,690,196-byte unpacked。本 Gate 不改变数据库或 HA 拓扑，复用紧邻发布 Gate 的 PostgreSQL 18.6 arm64 physical HA 基线而不把它声明为本阶段新证据。公开 GHCR catalog 未实际产生，因此不宣称真实线上验签成功。 - D-338/ADR-0430（已接受；真实公开 catalog 运行待实际 release tag）：发布端的 durable catalog 不再由部署者通过松散 shell 重定向手工消费。可信工作站上的 `ql3-release-catalog-consumption-ceremony.cjs` 从 exact source version/revision/tag、closed `local|cluster|all` scope 与 owner/source repository 推导唯一 discovery ref，前后两次解析必须得到同一 digest，后续只使用 catalog `@sha256:` immutable reference。ceremony 以绝对路径、dev/inode/size/SHA-256 固定 `regctl|cosign|gh`，owner-private token 只进入单个 GitHub provenance verifier；环境、cache/config/tmp 与最终写入均有封闭边界。下载的 canonical release set 经过 standalone identity/family/self-digest inspection，raw OCI manifest 同时按 digest、media type、empty config、单 layer、basename、size/content digest 与四项 annotation 重建 publication plan/receipt。成功后才以 `0700` no-replace 目录和三项 `0600` 文件发布 release set、raw manifest 与 self-digest report；offline audit 要求 exact-three-file，并完全重建结构/manifest/report，同时诚实声明网络签名结果未离线 replay。该 ceremony 无 registry/GitHub mutation、deployment action authority、Compose/Kubernetes apply 或数据库访问；D337 继续只消费审计后的 release set。Local/低配设备不安装任何工作站工具，Cluster 也不新增 controller/CRD/RBAC。独立 ceremony 20/20、完整定向发布链 121/121 已通过；backend 共 1,315 项，1,313 pass/2 条件 skip/0 fail；18-package clean build/test 退出 0，package boundary 保持 18 packages、`singleSourcePackages=[]`、`shallowSourcePackages=[]`。10 项架构/部署审计与 14 档 Local artifact 全部 compatible，默认 Edge/Standalone 为 2,589,890/2,589,968 bytes，application+AI 为 4,493,043/4,493,175 bytes，MCP 为 7,315,930/7,316,038 bytes；Cluster Admin pack 保持 250 files、271,238-byte tarball、1,690,196-byte unpacked。本 Gate 不改变数据库或 HA 拓扑，复用紧邻发布 Gate 的 PostgreSQL 18.6 arm64 physical HA 基线而不把它声明为本阶段新证据。公开 GHCR catalog 仍未实际产生，因此本门不宣称已取得真实 Cosign/GitHub/registry 成功证据。 diff --git a/docs/adr/ADR-0433-catalog-bound-kubernetes-target-deployment-ceremony.md b/docs/adr/ADR-0433-catalog-bound-kubernetes-target-deployment-ceremony.md new file mode 100644 index 00000000..cd780c9d --- /dev/null +++ b/docs/adr/ADR-0433-catalog-bound-kubernetes-target-deployment-ceremony.md @@ -0,0 +1,86 @@ +# ADR-0433：Catalog-bound Kubernetes 目标部署 Ceremony + +- 状态：Accepted +- 日期：2026-08-16 +- 关联 RFC：QL-RFC-0001 D-03、D-14、D-337、D-339、D-341 +- 关联 ADR：ADR-0429、ADR-0431、ADR-0432 + +## 上下文 + +ADR-0431 已把 verified release catalog 物化为 Kubernetes v2 locked manifest/report，但最后一步仍由运维者直接执行 +`kubectl apply -f locked.yaml`。该命令没有再次绑定 report self-digest、目标 cluster、kubeconfig、kubectl executable 或 +field manager；检查过的文件、实际输入和实际 API target 之间仍存在复制、context 漂移与 TOCTOU 窗口。成功输出也没有 durable +receipt，响应丢失后只能凭人工判断是否重放。 + +Cluster 需要解决这个缺口，但不能把 Kubernetes、YAML 或 registry 工具带到 Local/Edge，也不能为一次发布新增常驻 controller、 +webhook 或长期 ServiceAccount authority。 + +## 决策 + +1. 在现有维护工作站脚本边界增加 `cluster.deployment.preflight`、`cluster.deployment.apply` 与 + `cluster.deployment.receipt.audit` 三种 canonical command；只提供一个 + `pnpm cluster-deployment:ql3 -- --command-file=/absolute/private/command.json` 入口，不新增 workspace package 或生产依赖。 +2. command、locked manifest、lock report、kubeconfig、preflight 与 receipt 必须位于 current-UID 的 canonical `0700` 目录，文件为 + canonical、current-UID、单链接 `0600` regular file。所有输入使用 `O_NOFOLLOW|O_CLOEXEC` stable descriptor 有界读取，调用前后 + 复验 dev/inode/size/mtime/ctime 与 SHA-256。kubectl 必须是 absolute canonical、current-UID 或 root owner、不可 group/other write + 的单链接 executable，并由 command 固定其 SHA-256。 +3. 目标 consumer 独立验证 `qinglong/kubernetes-deployment-lock@v2` exact shape/self-digest、3.x release identity、catalog workflow/ + immutable reference/release-set 闭包、required role 顺序、全部 GHCR digest reference、manifest byte digest、资源/authority 数量、 + 五项 release annotation，以及未知或畸形 QingLong image authority 为零。不能只相信 materializer 的 success stdout。 +4. kubeconfig 必须由 command 固定 SHA-256 和 explicit context；禁止 `exec` 与 legacy `auth-provider`，避免稳定文件读取后再隐式执行 + ambient credential plugin。每次网络动作前显式读取 `kube-system` Namespace UID，并与人工审核的 `expectedClusterUid` 精确匹配。 +5. preflight 只执行固定 manager `qinglong3-catalog-lock` 的 `kubectl apply --server-side --dry-run=server --validate=strict -f=-`，通过 + stdin 发送已验证的内存字节，不让 kubectl 按路径二次读取 manifest。所有承载 QingLong image authority 的资源必须显式携带 + namespace，禁止依赖 context 的 ambient default namespace。成功后才 no-replace 发布 self-digest preflight report；它明确 + `networkAccess=true`、`kubernetesMutation=false`。目标 namespace 必须预先存在，否则包含 namespaced resources 的 dry-run 自然失败关闭。 +6. apply 必须消费 exact preflight digest，并重新执行全部离线检查、cluster UID 与 server-side dry-run；不使用 + `--force-conflicts`。实际 apply 后以 `kubectl get -f=- -o=json --show-managed-fields=true` 读取同一资源集合：每个对象必须有 + UID/resourceVersion，全部期望字段递归匹配；承载 QingLong image authority 的资源必须由固定 field manager 以 `Apply` 持有，所有 + image/catalog annotation 再次精确验证。结束时再次检查 cluster UID 和全部稳定文件。 +7. 只有所有步骤成功后才以 `0600` no-replace 发布 self-digest receipt。receipt 绑定 mutation、command、preflight、lock/manifest/ + catalog digest、cluster UID、kubeconfig/kubectl digest、各 argv/stdout/stderr digest 与 byte count。已有 exact receipt 的相同 command + 离线返回；apply 成功但 receipt 丢失时，使用同一 command/field manager 重放 server-side apply，再以 live convergence 收敛。 +8. Kubernetes 多资源 apply 不是事务，receipt 必须固定 `crossResourceAtomicity=false`。失败时不自动删除或回滚资源；先审计 live state， + 再用上一份 catalog-bound lock 执行新的显式 roll-forward。offline receipt audit 只重建 canonical/self/command binding，明确 + `externalResultsReplayed=false`，不伪称离线重放 API Server 结果。 + +## 部署与资源影响 + +- Local/Edge/Standalone 零导入、零制品增量、零常驻 CPU/RSS/网络/写放大；低配路由器继续只消费 Local v2 selection。 +- Cluster 不新增 controller、webhook、CRD、RBAC、ServiceAccount、Pod、listener、timer、watcher、数据库、migration、SQL 或 Pool。 + ceremony 使用部署者原有 kubeconfig 权限并在命令结束后退出。每个 kubectl 子进程使用独立 `0700` 临时 HOME/XDG cache/TMPDIR， + 结束即清理，不读取 ambient HOME，也不在仓库或 operator home 留下 discovery cache。 +- 实现内聚在 `scripts/lib/ql3-kubernetes-deployment-ceremony.cjs` 与薄 CLI；没有为单一工作站流程拆出浅 workspace package。 +- 每次 preflight 为一次 cluster identity read 和一次 server-side dry-run；apply 为 identity read、dry-run、apply、convergence read、末次 + identity read。manifest/report/kubeconfig 各有明确 byte ceiling，process stdout/stderr 各最多 4 MiB。 + +## 被拒绝的替代方案 + +### 继续文档化裸 `kubectl apply` + +拒绝。它无法证明 apply 的文件、report、context、cluster 和 executable 与人工审核对象相同，也没有 response-loss recovery receipt。 + +### 使用 `kubectl diff` 证明收敛 + +拒绝。真实 K3s 门证明 kubectl 会隐式从 `$PATH` 启动外部 `diff`，扩大未固定的 executable authority。受审 convergence read 只调用 +同一 pinned kubectl，并直接验证 live object identity、managed field ownership 和完整期望字段。 + +### 自动 `--force-conflicts` 或失败后删除资源 + +拒绝。强夺其他 field manager 或跨资源猜测回滚会扩大故障；冲突必须失败关闭，多资源非原子事实必须进入 receipt 与人工恢复流程。 + +### 在集群内新增持续部署 controller + +拒绝。当前缺口是显式 release rollout 的最后一跳，不值得新增常驻 availability、credential、certificate 与升级故障域。 + +## 验证 + +- 定向契约 10/10，覆盖 lock/report/manifest/annotation、cluster UID、文件权限、kubeconfig executable auth、server dry-run、apply、live + convergence、field manager、receipt 重签、response replay、closed CLI 与低敏失败； +- 隔离三节点 K3s `v1.34.3+k3s1`/Linux arm64 真实运行 6 个资源、4 个零副本 Deployment；preflight、server-side apply、live + convergence 与 offline receipt audit 全部通过，固定 manager 为 `qinglong3-catalog-lock`，临时 Docker container/network 零残留； +- 最终 live cluster UID 为 `7b2a5391-41a3-4905-90cc-3b831bef0058`，preflight digest 为 + `sha256:7c6db6236a704cd7791f94268b3883d6385a75a993a14d1611791f885fb65404`，receipt digest 为 + `sha256:58817a667bccf28c068fff40619cf13d2d0a1be84153de66a137b005feb536ba`。该 synthetic lock 只验证目标 apply 语义，不冒充公开 + GHCR catalog ceremony； +- 完整 backend、18-package、边界审计与制品结果记录在 QL-RFC-0001 D-341。 diff --git a/docs/adr/README.md b/docs/adr/README.md index c5e5878e..b74fe220 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -436,6 +436,7 @@ | [ADR-0430](./ADR-0430-auditable-release-catalog-consumption-ceremony.md) | 可审计的 Release Catalog 消费工作站 Ceremony | Accepted（真实公开 catalog 运行待实际 release tag） | | [ADR-0431](./ADR-0431-catalog-bound-deployment-lock-chain.md) | Catalog-bound Deployment Lock 证据链 | Accepted（真实公开 catalog 运行待实际 release tag） | | [ADR-0432](./ADR-0432-target-side-catalog-bound-local-compose-revisions.md) | 目标侧 Catalog-bound Local Compose 修订 | Accepted | +| [ADR-0433](./ADR-0433-catalog-bound-kubernetes-target-deployment-ceremony.md) | Catalog-bound Kubernetes 目标部署 Ceremony | Accepted | ## 规则 diff --git a/docs/operations/ql3-release-set-deployment.md b/docs/operations/ql3-release-set-deployment.md index c4c7edc1..6b95ab37 100644 --- a/docs/operations/ql3-release-set-deployment.md +++ b/docs/operations/ql3-release-set-deployment.md @@ -176,8 +176,111 @@ materializer 只改写 Pod、Deployment、StatefulSet、DaemonSet、ReplicaSet manifest、consumption report digest、source revision 与 version annotation；未知位置的完整 QingLong role image authority、畸形 已知 container image、缺少 required role、YAML alias/cycle/非 mapping、超限输入或已有输出文件都会失败关闭。 -审计成功并完成人工差异检查后，才由有权限的独立步骤执行 `kubectl apply -f "${locked}"`。不要直接 apply -`${rendered}`，也不要使用 `kubectl apply -k` 绕过 deployment lock。 +审计成功并完成人工差异检查后，不要再直接执行裸 `kubectl apply -f "${locked}"`，也不要 apply `${rendered}` 或使用 +`kubectl apply -k` 绕过 deployment lock。目标侧必须使用下面的独立 preflight/apply ceremony。 + +### Kubernetes 目标 preflight 与 apply + +先确认所有承载 QingLong image authority 的资源都显式填写 `metadata.namespace`，且目标 Namespace 已存在；禁止依赖 context 的 +ambient default namespace。server-side dry-run 不会持久化同一 multi-document 输入中排在前面的 Namespace，因此不能把“创建 +Namespace”和“在该 Namespace 内验证首批对象”混成一次隐式动作。取得并人工核对目标 cluster 的 +`kube-system` Namespace UID： + +```sh +kubectl --kubeconfig="${kubeconfig}" --context="${context}" \ + get namespace kube-system -o=jsonpath='{.metadata.uid}' +``` + +ceremony 目录必须为当前 UID 的 canonical `0700` 目录；command、locked manifest、lock report、kubeconfig、preflight 与 receipt +都必须是单链接 `0600` 文件。kubectl 使用 realpath 后的 absolute executable，并记录其 SHA-256。kubeconfig 禁止 `exec` 与 +`auth-provider`。每个 kubectl 调用使用独立临时 HOME/XDG cache/TMPDIR，结束即清理，不读取 ambient HOME 或在当前目录创建 +`.kube/cache`。以下是 preflight command 的逻辑结构；实际文件必须用 `JSON.stringify(value) + "\n"` 写成单行 canonical JSON， +并以 `0600` no-replace 创建： + +```json +{ + "schemaVersion": 1, + "schema": "qinglong/kubernetes-deployment-command@v1", + "operation": "cluster.deployment.preflight", + "request": { + "preflightId": "", + "lockedManifest": { + "path": "", + "expectedDigest": "" + }, + "lockReport": { + "path": "", + "expectedDigest": "" + }, + "kubectl": { + "path": "", + "expectedDigest": "" + }, + "kubeconfig": { + "path": "", + "expectedDigest": "" + }, + "context": "", + "expectedClusterUid": "", + "output": "" + } +} +``` + +运行： + +```sh +pnpm cluster-deployment:ql3 -- --command-file="${preflight_command}" +``` + +成功后，人工核对返回的 `preflightDigest`、lock/catalog digest、cluster UID 和 +`kubernetesMutation:false`。apply 必须使用新的 mutation UUID，并精确复用所有 target/input authority： + +```json +{ + "schemaVersion": 1, + "schema": "qinglong/kubernetes-deployment-command@v1", + "operation": "cluster.deployment.apply", + "request": { + "mutationId": "", + "preflight": { + "path": "", + "expectedDigest": "" + }, + "lockedManifest": { + "path": "", + "expectedDigest": "" + }, + "lockReport": { + "path": "", + "expectedDigest": "" + }, + "kubectl": { + "path": "", + "expectedDigest": "" + }, + "kubeconfig": { + "path": "", + "expectedDigest": "" + }, + "context": "", + "expectedClusterUid": "", + "output": "" + } +} +``` + +同样通过 `cluster-deployment:ql3` 执行。apply 会重新完成 lock 检查、cluster identity 和 server-side dry-run，然后以固定 +`qinglong3-catalog-lock` field manager 执行 server-side apply，不使用 `--force-conflicts`；最后读取 live objects，验证 +UID/resourceVersion、完整期望字段、四类 immutable image/catalog annotations 和受影响资源的 managed-field ownership，再次确认 +cluster UID 后才发布 receipt。 + +离线审计使用 `cluster.deployment.receipt.audit` command，其中 `applyCommand.expectedDigest` 是 apply command 文件完整字节的 +SHA-256，`receipt.expectedDigest` 是 receipt 内的 `receiptDigest`。审计不会访问 Kubernetes API，结果必须保持 +`externalResultsReplayed:false`、`kubernetesMutation:false`。 + +多资源 apply 不是事务，也不提供自动删除式 rollback。失败或 receipt 响应丢失时保留原文件：同一 command/field manager 可安全 +重放并重新做 live convergence；需要回退时，以上一份 catalog-bound lock 创建新的显式 preflight/apply，而不是修改旧 receipt。 ## 准入检查 @@ -189,8 +292,9 @@ manifest、consumption report digest、source revision 与 version annotation； 4. Kubernetes 必须先渲染 overlay，再用离线 post-render materializer 生成和复验 v2 locked manifest；嵌套 overlay 的 `newName`/digest 不是最终 authority。Local 必须生成并审计 v2 service selection。两族输出都必须绑定同一 catalog manifest、 consumption report 与 release-set digest，并且只能消费 release set 中的 `@sha256:` reference。 -5. rollout 前再次确认 catalog receipt/immutable reference 与已检查文件一致。version/source/catalog tag 都只能用于 - 发现；部署始终以 release set 中的镜像 digest 为准。 +5. rollout 前再次确认 catalog receipt/immutable reference 与已检查文件一致。Kubernetes 必须把 locked manifest/report、pinned + kubectl/kubeconfig 和目标 cluster UID 绑定进 preflight/apply receipt；version/source/catalog tag 都只能用于发现，部署始终以 + release set 中的镜像 digest 为准。 ## 低资源设备 diff --git a/package.json b/package.json index 249645d1..e497a0de 100644 --- a/package.json +++ b/package.json @@ -115,6 +115,8 @@ "audit:image-release:ql3": "node scripts/ql3-cluster-image-release-audit.cjs", "release-catalog-consumption:ql3": "node scripts/ql3-release-catalog-consumption-ceremony.cjs", "deployment-lock:ql3": "node scripts/ql3-deployment-lock-contract.cjs", + "cluster-deployment:ql3": "node scripts/ql3-kubernetes-deployment-ceremony.cjs", + "test:cluster-deployment-live:ql3": "node scripts/ql3-kubernetes-deployment-live-contract.cjs", "audit:deployment-lock-surfaces:ql3": "node scripts/ql3-deployment-lock-contract.cjs --mode=surfaces-audit", "audit:image-os-vulnerability-policy:ql3": "node scripts/ql3-image-os-vulnerability-policy.cjs", "audit:cluster-image-release:ql3": "node scripts/ql3-cluster-image-release-audit.cjs", diff --git a/scripts/lib/ql3-kubernetes-deployment-ceremony.cjs b/scripts/lib/ql3-kubernetes-deployment-ceremony.cjs new file mode 100644 index 00000000..2d8ec785 --- /dev/null +++ b/scripts/lib/ql3-kubernetes-deployment-ceremony.cjs @@ -0,0 +1,1600 @@ +'use strict'; + +const crypto = require('node:crypto'); +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const { spawnSync } = require('node:child_process'); +const yaml = require('js-yaml'); + +const COMMAND_SCHEMA = 'qinglong/kubernetes-deployment-command@v1'; +const LOCK_SCHEMA = 'qinglong/kubernetes-deployment-lock@v2'; +const CATALOG_SCHEMA = 'qinglong/release-catalog-consumption-ceremony@v1'; +const PREFLIGHT_SCHEMA = 'qinglong/kubernetes-deployment-preflight@v1'; +const RECEIPT_SCHEMA = 'qinglong/kubernetes-deployment-receipt@v1'; +const FIELD_MANAGER = 'qinglong3-catalog-lock'; +const MAX_COMMAND_BYTES = 64 * 1024; +const MAX_MANIFEST_BYTES = 8 * 1024 * 1024; +const MAX_REPORT_BYTES = 1024 * 1024; +const MAX_KUBECONFIG_BYTES = 1024 * 1024; +const MAX_EXECUTABLE_BYTES = 256 * 1024 * 1024; +const MAX_PROCESS_OUTPUT_BYTES = 4 * 1024 * 1024; +const MAX_RESOURCE_COUNT = 2048; +const MAX_STRUCTURE_DEPTH = 64; +const PROCESS_TIMEOUT_MS = 60_000; +const DIGEST_PATTERN = /^sha256:[a-f0-9]{64}$/u; +const UUID_PATTERN = + /^[a-f0-9]{8}-[a-f0-9]{4}-[1-8][a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12}$/u; +const VERSION_PATTERN = + /^3\.(?:0|[1-9]\d*)\.(?:0|[1-9]\d*)(?:-[0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*)?(?:\+[0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*)?$/u; +const CONTEXT_PATTERN = /^[A-Za-z0-9](?:[A-Za-z0-9._-]{0,252})$/u; +const REPOSITORY_PATTERN = + /^[a-z0-9](?:[a-z0-9-]{0,37}[a-z0-9])?\/[a-z0-9._-]{1,100}$/u; +const ROLE_ORDER = Object.freeze(['control', 'control-ai', 'admin', 'worker']); +const IMAGE_NAMES = Object.freeze({ + control: 'qinglong3-cluster-control', + 'control-ai': 'qinglong3-cluster-control-ai', + admin: 'qinglong3-cluster-admin', + worker: 'qinglong3-worker', +}); +const ANNOTATION_KEYS = Object.freeze({ + releaseSet: 'qinglong.io/release-set-digest', + catalogManifest: 'qinglong.io/release-catalog-manifest-digest', + catalogReport: 'qinglong.io/release-catalog-report-digest', + sourceRevision: 'qinglong.io/release-source-revision', + version: 'qinglong.io/release-version', +}); + +class QingLong3KubernetesDeploymentCeremonyError extends Error { + constructor(message) { + super(`QingLong 3 Kubernetes deployment ceremony failed: ${message}`); + this.name = 'QingLong3KubernetesDeploymentCeremonyError'; + this.code = 'QL3_KUBERNETES_DEPLOYMENT_CEREMONY_FAILED'; + } +} + +function fail(message) { + throw new QingLong3KubernetesDeploymentCeremonyError(message); +} + +function sha256(value) { + return `sha256:${crypto.createHash('sha256').update(value).digest('hex')}`; +} + +function canonicalJson(value) { + return `${JSON.stringify(value)}\n`; +} + +function isPlainObject(value) { + if (value === null || typeof value !== 'object' || Array.isArray(value)) { + return false; + } + const prototype = Object.getPrototypeOf(value); + return prototype === Object.prototype || prototype === null; +} + +function exactKeys(value, expected) { + return ( + isPlainObject(value) && + JSON.stringify(Object.keys(value).sort()) === + JSON.stringify([...expected].sort()) + ); +} + +function resolveCanonicalAbsolute(input, label) { + if (typeof input !== 'string' || !path.isAbsolute(input)) { + fail(`${label} path must be absolute`); + } + const resolved = path.resolve(input); + if (resolved !== input) fail(`${label} path must be normalized`); + return resolved; +} + +function validatePrivateParent(filePath, label, uid = process.getuid()) { + const parent = path.dirname(filePath); + let status; + try { + status = fs.lstatSync(parent); + } catch { + fail(`${label} parent is unavailable`); + } + if ( + !status.isDirectory() || + status.isSymbolicLink() || + status.uid !== uid || + (status.mode & 0o777) !== 0o700 || + fs.realpathSync(parent) !== parent + ) { + fail(`${label} parent must be one current-owner canonical 0700 directory`); + } + return parent; +} + +function descriptorIdentity(status) { + return Object.freeze({ + dev: status.dev, + ino: status.ino, + size: status.size, + mtimeMs: status.mtimeMs, + ctimeMs: status.ctimeMs, + }); +} + +function sameIdentity(left, right) { + return ( + left.dev === right.dev && + left.ino === right.ino && + left.size === right.size && + left.mtimeMs === right.mtimeMs && + left.ctimeMs === right.ctimeMs + ); +} + +function readStableFile( + input, + label, + maximumBytes, + { privateFile = true, executable = false } = {}, +) { + const target = resolveCanonicalAbsolute(input, label); + if (privateFile) validatePrivateParent(target, label); + let lexical; + try { + lexical = fs.lstatSync(target); + } catch { + fail(`${label} is unavailable`); + } + const uid = process.getuid(); + const allowedOwner = privateFile + ? lexical.uid === uid + : lexical.uid === uid || lexical.uid === 0; + const validMode = privateFile + ? (lexical.mode & 0o777) === 0o600 + : (lexical.mode & 0o022) === 0 && (lexical.mode & 0o111) !== 0; + if ( + !lexical.isFile() || + lexical.isSymbolicLink() || + lexical.nlink !== 1 || + !allowedOwner || + !validMode || + lexical.size < 1 || + lexical.size > maximumBytes || + fs.realpathSync(target) !== target || + (executable && (lexical.mode & 0o111) === 0) + ) { + fail(`${label} file authority is invalid`); + } + const flags = + fs.constants.O_RDONLY | + (fs.constants.O_NOFOLLOW ?? 0) | + (fs.constants.O_CLOEXEC ?? 0); + let descriptor; + try { + descriptor = fs.openSync(target, flags); + } catch { + fail(`${label} cannot be opened safely`); + } + try { + const before = fs.fstatSync(descriptor); + if ( + !before.isFile() || + before.dev !== lexical.dev || + before.ino !== lexical.ino || + before.size !== lexical.size + ) { + fail(`${label} descriptor identity is invalid`); + } + const chunks = []; + const hash = crypto.createHash('sha256'); + let total = 0; + while (true) { + const chunk = Buffer.allocUnsafe(Math.min(64 * 1024, maximumBytes + 1)); + const read = fs.readSync(descriptor, chunk, 0, chunk.length, null); + if (read === 0) break; + total += read; + if (total > maximumBytes) fail(`${label} exceeds its byte limit`); + const actual = chunk.subarray(0, read); + hash.update(actual); + if (!executable) chunks.push(actual); + } + const after = fs.fstatSync(descriptor); + if (!sameIdentity(descriptorIdentity(before), descriptorIdentity(after))) { + fail(`${label} changed while it was read`); + } + const contents = executable ? undefined : Buffer.concat(chunks); + return Object.freeze({ + path: target, + identity: descriptorIdentity(after), + bytes: total, + digest: `sha256:${hash.digest('hex')}`, + ...(contents === undefined ? {} : { contents }), + }); + } finally { + fs.closeSync(descriptor); + } +} + +function utf8(file, label) { + const contents = file.contents.toString('utf8'); + if (!Buffer.from(contents, 'utf8').equals(file.contents)) { + fail(`${label} must contain valid UTF-8`); + } + return contents; +} + +function parseCanonicalJson(contents, label) { + let parsed; + try { + parsed = JSON.parse(contents); + } catch { + fail(`${label} must contain valid JSON`); + } + if (canonicalJson(parsed) !== contents) { + fail(`${label} must use canonical JSON encoding`); + } + return parsed; +} + +function readCanonicalJson(input, label, maximumBytes) { + const file = readStableFile(input, label, maximumBytes); + return Object.freeze({ + ...file, + value: parseCanonicalJson(utf8(file, label), label), + }); +} + +function artifact(value, label) { + if ( + !exactKeys(value, ['path', 'expectedDigest']) || + typeof value.path !== 'string' || + !DIGEST_PATTERN.test(value.expectedDigest || '') + ) { + fail(`${label} authority is invalid`); + } + return Object.freeze({ + path: resolveCanonicalAbsolute(value.path, label), + expectedDigest: value.expectedDigest, + }); +} + +function verifyExpected(file, expectedDigest, label) { + if (file.digest !== expectedDigest) fail(`${label} digest changed`); + return file; +} + +function outputTarget(input, label) { + const target = resolveCanonicalAbsolute(input, label); + validatePrivateParent(target, label); + if (fs.existsSync(target)) fail(`${label} must be unused`); + return target; +} + +function writeNoReplace(target, value) { + fs.writeFileSync(target, canonicalJson(value), { + encoding: 'utf8', + mode: 0o600, + flag: 'wx', + }); +} + +function parseCommand(value) { + if ( + !exactKeys(value, ['schemaVersion', 'schema', 'operation', 'request']) || + value.schemaVersion !== 1 || + value.schema !== COMMAND_SCHEMA + ) { + fail('command shape is invalid'); + } + const request = value.request; + if (value.operation === 'cluster.deployment.preflight') { + if ( + !exactKeys(request, [ + 'preflightId', + 'lockedManifest', + 'lockReport', + 'kubectl', + 'kubeconfig', + 'context', + 'expectedClusterUid', + 'output', + ]) || + !UUID_PATTERN.test(request.preflightId || '') || + !CONTEXT_PATTERN.test(request.context || '') || + !UUID_PATTERN.test(request.expectedClusterUid || '') || + typeof request.output !== 'string' + ) { + fail('preflight command is invalid'); + } + return Object.freeze({ + schemaVersion: 1, + schema: COMMAND_SCHEMA, + operation: value.operation, + request: Object.freeze({ + preflightId: request.preflightId, + lockedManifest: artifact(request.lockedManifest, 'locked manifest'), + lockReport: artifact(request.lockReport, 'lock report'), + kubectl: artifact(request.kubectl, 'kubectl'), + kubeconfig: artifact(request.kubeconfig, 'kubeconfig'), + context: request.context, + expectedClusterUid: request.expectedClusterUid, + output: resolveCanonicalAbsolute(request.output, 'preflight output'), + }), + }); + } + if (value.operation === 'cluster.deployment.apply') { + if ( + !exactKeys(request, [ + 'mutationId', + 'preflight', + 'lockedManifest', + 'lockReport', + 'kubectl', + 'kubeconfig', + 'context', + 'expectedClusterUid', + 'output', + ]) || + !UUID_PATTERN.test(request.mutationId || '') || + !CONTEXT_PATTERN.test(request.context || '') || + !UUID_PATTERN.test(request.expectedClusterUid || '') || + typeof request.output !== 'string' + ) { + fail('apply command is invalid'); + } + return Object.freeze({ + schemaVersion: 1, + schema: COMMAND_SCHEMA, + operation: value.operation, + request: Object.freeze({ + mutationId: request.mutationId, + preflight: artifact(request.preflight, 'preflight report'), + lockedManifest: artifact(request.lockedManifest, 'locked manifest'), + lockReport: artifact(request.lockReport, 'lock report'), + kubectl: artifact(request.kubectl, 'kubectl'), + kubeconfig: artifact(request.kubeconfig, 'kubeconfig'), + context: request.context, + expectedClusterUid: request.expectedClusterUid, + output: resolveCanonicalAbsolute( + request.output, + 'apply receipt output', + ), + }), + }); + } + if (value.operation === 'cluster.deployment.receipt.audit') { + if (!exactKeys(request, ['applyCommand', 'receipt'])) { + fail('receipt audit command is invalid'); + } + return Object.freeze({ + schemaVersion: 1, + schema: COMMAND_SCHEMA, + operation: value.operation, + request: Object.freeze({ + applyCommand: artifact(request.applyCommand, 'apply command'), + receipt: artifact(request.receipt, 'deployment receipt'), + }), + }); + } + fail('command operation is invalid'); +} + +function readCommand(input) { + const commandFile = readCanonicalJson( + input, + 'deployment command', + MAX_COMMAND_BYTES, + ); + return Object.freeze({ + file: commandFile, + command: parseCommand(commandFile.value), + commandDigest: commandFile.digest, + }); +} + +function validateRelease(release) { + if ( + !exactKeys(release, ['version', 'sourceRevision', 'sourceRef', 'scope']) || + !VERSION_PATTERN.test(release.version || '') || + !/^[a-f0-9]{40}$/u.test(release.sourceRevision || '') || + release.sourceRef !== `refs/tags/v${release.version}` || + !['cluster', 'all'].includes(release.scope) + ) { + fail('deployment lock release identity is invalid'); + } +} + +function validateCatalog(catalog, release, releaseSetDigest) { + if ( + !exactKeys(catalog, [ + 'schema', + 'sourceRepository', + 'workflowIdentity', + 'immutableReference', + 'manifestDigest', + 'consumptionReportDigest', + 'releaseSetDigest', + 'discoveryTagAuthority', + ]) || + catalog.schema !== CATALOG_SCHEMA || + !REPOSITORY_PATTERN.test(catalog.sourceRepository || '') || + catalog.workflowIdentity !== + `https://github.com/${catalog.sourceRepository}/.github/workflows/ql3-image-release.yml@${release.sourceRef}` || + !DIGEST_PATTERN.test(catalog.manifestDigest || '') || + !DIGEST_PATTERN.test(catalog.consumptionReportDigest || '') || + catalog.releaseSetDigest !== releaseSetDigest || + catalog.discoveryTagAuthority !== 'none' + ) { + fail('deployment lock catalog authority is invalid'); + } + const match = new RegExp( + `^ghcr\\.io/([a-z0-9](?:[a-z0-9-]{0,37}[a-z0-9])?)/qinglong3-release-catalog@${catalog.manifestDigest}$`, + 'u', + ).exec(catalog.immutableReference || ''); + if (!match) fail('deployment lock catalog reference is invalid'); + return match[1]; +} + +function validateLockReport(report) { + if ( + !exactKeys(report, [ + 'schemaVersion', + 'schema', + 'release', + 'releaseSetDigest', + 'catalog', + 'deploymentFamily', + 'requiredImages', + 'imageOccurrences', + 'manifest', + 'verification', + 'lockDigest', + ]) || + report.schemaVersion !== 1 || + report.schema !== LOCK_SCHEMA || + report.deploymentFamily !== 'cluster' || + !DIGEST_PATTERN.test(report.releaseSetDigest || '') || + !DIGEST_PATTERN.test(report.lockDigest || '') + ) { + fail('deployment lock report shape is invalid'); + } + validateRelease(report.release); + const repositoryOwner = validateCatalog( + report.catalog, + report.release, + report.releaseSetDigest, + ); + if ( + !Array.isArray(report.requiredImages) || + report.requiredImages.length < 1 || + JSON.stringify(report.requiredImages) !== + JSON.stringify( + ROLE_ORDER.filter((role) => report.requiredImages.includes(role)), + ) || + !Array.isArray(report.imageOccurrences) || + report.imageOccurrences.length !== ROLE_ORDER.length + ) { + fail('deployment lock role closure is invalid'); + } + const references = {}; + const counts = {}; + for (let index = 0; index < ROLE_ORDER.length; index += 1) { + const role = ROLE_ORDER[index]; + const occurrence = report.imageOccurrences[index]; + const repository = IMAGE_NAMES[role]; + if ( + !exactKeys(occurrence, ['name', 'reference', 'count']) || + occurrence.name !== role || + !new RegExp( + `^ghcr\\.io/${repositoryOwner}/${repository}@sha256:[a-f0-9]{64}$`, + 'u', + ).test(occurrence.reference || '') || + !Number.isSafeInteger(occurrence.count) || + occurrence.count < 0 || + occurrence.count > MAX_RESOURCE_COUNT * 16 || + (report.requiredImages.includes(role) && occurrence.count < 1) + ) { + fail(`deployment lock image occurrence is invalid: ${role}`); + } + references[role] = occurrence.reference; + counts[role] = occurrence.count; + } + if ( + !exactKeys(report.manifest, [ + 'inputDigest', + 'outputDigest', + 'resources', + 'changedResources', + 'admissionAuthorityCount', + ]) || + !DIGEST_PATTERN.test(report.manifest.inputDigest || '') || + !DIGEST_PATTERN.test(report.manifest.outputDigest || '') || + !Number.isSafeInteger(report.manifest.resources) || + report.manifest.resources < 1 || + report.manifest.resources > MAX_RESOURCE_COUNT || + !Number.isSafeInteger(report.manifest.changedResources) || + report.manifest.changedResources < 1 || + report.manifest.changedResources > report.manifest.resources || + !Number.isSafeInteger(report.manifest.admissionAuthorityCount) || + report.manifest.admissionAuthorityCount < 0 || + report.manifest.admissionAuthorityCount > report.manifest.resources || + !exactKeys(report.verification, [ + 'releaseSet', + 'sourceRecordsReplayed', + 'catalogConsumption', + 'externalToolResultsReplayed', + 'unknownImageAuthorities', + 'mutableQingLongImages', + 'networkAccess', + 'kubernetesMutation', + ]) || + JSON.stringify(report.verification) !== + JSON.stringify({ + releaseSet: 'standalone_structure_identity_and_self_digest', + sourceRecordsReplayed: false, + catalogConsumption: 'offline_reconstructed', + externalToolResultsReplayed: false, + unknownImageAuthorities: 0, + mutableQingLongImages: 0, + networkAccess: false, + kubernetesMutation: false, + }) + ) { + fail('deployment lock verification is invalid'); + } + const { lockDigest, ...unsigned } = report; + if (lockDigest !== sha256(JSON.stringify(unsigned))) { + fail('deployment lock self digest is invalid'); + } + return Object.freeze({ + report, + repositoryOwner, + references: Object.freeze(references), + counts: Object.freeze(counts), + }); +} + +function validateStructure(value, state, depth = 0) { + if (depth > MAX_STRUCTURE_DEPTH) fail('locked manifest is too deep'); + if (value === null || typeof value !== 'object') return; + if (state.seen.has(value)) fail('locked manifest aliases are not allowed'); + state.seen.add(value); + if (Array.isArray(value)) { + for (const entry of value) validateStructure(entry, state, depth + 1); + return; + } + if (!isPlainObject(value)) fail('locked manifest value is invalid'); + for (const [key, entry] of Object.entries(value)) { + if (!key || /[\u0000-\u001f\u007f]/u.test(key)) { + fail('locked manifest key is invalid'); + } + validateStructure(entry, state, depth + 1); + } +} + +function parseManifest(contents) { + const resources = []; + try { + yaml.loadAll( + contents, + (resource) => { + if (resource !== undefined && resource !== null) + resources.push(resource); + }, + { json: false }, + ); + } catch { + fail('locked manifest must be duplicate-free YAML'); + } + if (resources.length < 1 || resources.length > MAX_RESOURCE_COUNT) { + fail('locked manifest resource count is invalid'); + } + const state = { seen: new WeakSet() }; + for (const resource of resources) { + if (!isPlainObject(resource)) fail('locked manifest resource is invalid'); + validateStructure(resource, state); + if ( + typeof resource.apiVersion !== 'string' || + typeof resource.kind !== 'string' || + !isPlainObject(resource.metadata) || + typeof resource.metadata.name !== 'string' + ) { + fail('locked manifest resource identity is invalid'); + } + } + return resources; +} + +function roleFromImage(value) { + if (typeof value !== 'string') return null; + for (const role of ROLE_ORDER) { + if ( + new RegExp( + `^(?:[A-Za-z0-9][A-Za-z0-9._-]*(?::[0-9]+)?/)*${IMAGE_NAMES[role]}(?::[^@\\s]+|@sha256:[a-f0-9]{64})$`, + 'u', + ).test(value) + ) { + return role; + } + } + return null; +} + +function mentionsRoleImageName(value) { + return ( + typeof value === 'string' && + ROLE_ORDER.some((role) => value.includes(IMAGE_NAMES[role])) + ); +} + +function podTemplateFor(resource) { + if ( + ['Deployment', 'StatefulSet', 'DaemonSet', 'ReplicaSet'].includes( + resource.kind, + ) + ) { + return resource.spec?.template; + } + if (resource.kind === 'Job') return resource.spec?.template; + if (resource.kind === 'CronJob') { + return resource.spec?.jobTemplate?.spec?.template; + } + return undefined; +} + +function podSpecFor(resource) { + if (resource.kind === 'Pod') return resource.spec; + return podTemplateFor(resource)?.spec; +} + +function expectedAnnotations(report) { + return Object.freeze({ + [ANNOTATION_KEYS.releaseSet]: report.releaseSetDigest, + [ANNOTATION_KEYS.catalogManifest]: report.catalog.manifestDigest, + [ANNOTATION_KEYS.catalogReport]: report.catalog.consumptionReportDigest, + [ANNOTATION_KEYS.sourceRevision]: report.release.sourceRevision, + [ANNOTATION_KEYS.version]: report.release.version, + }); +} + +function assertAnnotations(target, expected, label) { + const annotations = target?.metadata?.annotations; + if (!isPlainObject(annotations)) fail(`${label} annotations are invalid`); + for (const [key, value] of Object.entries(expected)) { + if (annotations[key] !== value) fail(`${label} annotations drifted`); + } +} + +function inspectContainerList(list, authority, handled, counters) { + if (list === undefined) return 0; + if (!Array.isArray(list)) fail('locked manifest container list is invalid'); + let matched = 0; + for (const container of list) { + if (!isPlainObject(container)) fail('locked manifest container is invalid'); + const role = roleFromImage(container.image); + if (role === null) { + if (mentionsRoleImageName(container.image)) { + fail('locked manifest container image is malformed'); + } + continue; + } + if (container.image !== authority.references[role]) { + fail(`locked manifest image authority drifted: ${role}`); + } + handled.add(container); + counters[role] += 1; + matched += 1; + } + return matched; +} + +function scanUnhandled(value, authority, handled, parent, parentKey) { + if (typeof value === 'string') { + const role = roleFromImage(value); + if ( + role !== null && + (value !== authority.references[role] || + parentKey !== 'image' || + !handled.has(parent)) + ) { + fail('locked manifest contains an unhandled QingLong image authority'); + } + if (role === null && mentionsRoleImageName(value)) { + fail('locked manifest contains a malformed QingLong image authority'); + } + return; + } + if (value === null || typeof value !== 'object') return; + if (Array.isArray(value)) { + for (let index = 0; index < value.length; index += 1) { + scanUnhandled(value[index], authority, handled, value, String(index)); + } + return; + } + for (const [key, entry] of Object.entries(value)) { + scanUnhandled(entry, authority, handled, value, key); + } +} + +function inspectResourceAuthorities(resources, authority, label) { + const report = authority.report; + const counters = Object.fromEntries(ROLE_ORDER.map((role) => [role, 0])); + const handled = new WeakSet(); + const annotations = expectedAnnotations(report); + let changedResources = 0; + let admissionAuthorityCount = 0; + for (const resource of resources) { + const podSpec = podSpecFor(resource); + let matched = 0; + if (podSpec !== undefined) { + if (!isPlainObject(podSpec)) fail(`${label} Pod spec is invalid`); + for (const key of [ + 'initContainers', + 'containers', + 'ephemeralContainers', + ]) { + matched += inspectContainerList( + podSpec[key], + authority, + handled, + counters, + ); + } + } + if ( + resource.kind === 'ConfigMap' && + resource.metadata.name === 'ql3-plugin-package-secret-action-admission' + ) { + const role = roleFromImage(resource.data?.image); + if ( + role !== 'admin' || + resource.data.image !== authority.references.admin + ) { + fail('locked manifest admission authority is invalid'); + } + handled.add(resource.data); + counters.admin += 1; + matched += 1; + admissionAuthorityCount += 1; + } + if (matched > 0) { + if ( + typeof resource.metadata.namespace !== 'string' || + !CONTEXT_PATTERN.test(resource.metadata.namespace) + ) { + fail(`${label} resource namespace must be explicit`); + } + assertAnnotations(resource, annotations, `${label} resource`); + const template = podTemplateFor(resource); + if (template !== undefined) { + assertAnnotations(template, annotations, `${label} Pod template`); + } + changedResources += 1; + } + } + for (const resource of resources) { + scanUnhandled(resource, authority, handled, undefined, undefined); + } + if ( + JSON.stringify(counters) !== JSON.stringify(authority.counts) || + changedResources !== report.manifest.changedResources || + admissionAuthorityCount !== report.manifest.admissionAuthorityCount + ) { + fail('locked manifest authority counts changed'); + } + return Object.freeze({ + resourceCount: resources.length, + changedResources, + admissionAuthorityCount, + counts: Object.freeze({ ...counters }), + }); +} + +function inspectLockedManifest(contents, authority) { + const report = authority.report; + if (sha256(Buffer.from(contents, 'utf8')) !== report.manifest.outputDigest) { + fail('locked manifest digest changed'); + } + const resources = parseManifest(contents); + if (resources.length !== report.manifest.resources) { + fail('locked manifest resource count changed'); + } + inspectResourceAuthorities(resources, authority, 'locked manifest'); + return Object.freeze({ resources: Object.freeze(resources) }); +} + +function resourceIdentity(resource) { + const namespace = resource.metadata?.namespace ?? ''; + return `${resource.apiVersion}\u0000${resource.kind}\u0000${namespace}\u0000${ + resource.metadata?.name ?? '' + }`; +} + +function containsQingLongAuthority(value, references) { + if (typeof value === 'string') { + const role = roleFromImage(value); + return role !== null && value === references[role]; + } + if (value === null || typeof value !== 'object') return false; + if (Array.isArray(value)) { + return value.some((entry) => containsQingLongAuthority(entry, references)); + } + return Object.values(value).some((entry) => + containsQingLongAuthority(entry, references), + ); +} + +function assertDesiredSubset(desired, actual, depth = 0) { + if (depth > MAX_STRUCTURE_DEPTH) fail('live Kubernetes object is too deep'); + if (desired === null || typeof desired !== 'object') { + if (desired !== actual) fail('live Kubernetes object drifted'); + return; + } + if (Array.isArray(desired)) { + if (!Array.isArray(actual) || actual.length !== desired.length) { + fail('live Kubernetes array drifted'); + } + for (let index = 0; index < desired.length; index += 1) { + assertDesiredSubset(desired[index], actual[index], depth + 1); + } + return; + } + if (!isPlainObject(actual)) fail('live Kubernetes object is invalid'); + for (const [key, value] of Object.entries(desired)) { + if (!Object.hasOwn(actual, key)) fail('live Kubernetes field is missing'); + assertDesiredSubset(value, actual[key], depth + 1); + } +} + +function inspectConvergenceOutput(contents, inputs) { + let value; + try { + value = JSON.parse(contents); + } catch { + fail('Kubernetes convergence response is invalid'); + } + const items = value?.kind === 'List' ? value.items : undefined; + if ( + !Array.isArray(items) || + items.length !== inputs.authority.report.manifest.resources || + items.some((item) => !isPlainObject(item)) + ) { + fail('Kubernetes convergence resource count changed'); + } + const liveByIdentity = new Map(); + for (const item of items) { + const identity = resourceIdentity(item); + if (liveByIdentity.has(identity)) { + fail('Kubernetes convergence identity is duplicated'); + } + liveByIdentity.set(identity, item); + } + const desiredResources = inspectLockedManifest( + inputs.manifestContents, + inputs.authority, + ).resources; + for (const desired of desiredResources) { + const live = liveByIdentity.get(resourceIdentity(desired)); + if ( + live === undefined || + typeof live.metadata?.uid !== 'string' || + typeof live.metadata?.resourceVersion !== 'string' + ) { + fail('Kubernetes convergence identity is unavailable'); + } + if ( + containsQingLongAuthority(desired, inputs.authority.references) && + (!Array.isArray(live.metadata.managedFields) || + !live.metadata.managedFields.some( + (entry) => + entry?.manager === FIELD_MANAGER && entry?.operation === 'Apply', + )) + ) { + fail('Kubernetes convergence field authority is unavailable'); + } + assertDesiredSubset(desired, live); + } + inspectResourceAuthorities(items, inputs.authority, 'live Kubernetes'); +} + +function validateKubeconfig(contents, context) { + let config; + try { + config = yaml.load(contents, { json: false }); + } catch { + fail('kubeconfig is invalid'); + } + if ( + !isPlainObject(config) || + !Array.isArray(config.contexts) || + !config.contexts.some( + (entry) => + isPlainObject(entry) && + entry.name === context && + isPlainObject(entry.context) && + typeof entry.context.cluster === 'string', + ) || + !Array.isArray(config.users) + ) { + fail('kubeconfig context is unavailable'); + } + for (const entry of config.users) { + const user = entry?.user; + if ( + !isPlainObject(entry) || + !isPlainObject(user) || + Object.hasOwn(user, 'exec') || + Object.hasOwn(user, 'auth-provider') + ) { + fail('kubeconfig executable authentication is forbidden'); + } + } +} + +function inspectInputs(request) { + const manifest = verifyExpected( + readStableFile( + request.lockedManifest.path, + 'locked manifest', + MAX_MANIFEST_BYTES, + ), + request.lockedManifest.expectedDigest, + 'locked manifest', + ); + const reportFile = readCanonicalJson( + request.lockReport.path, + 'lock report', + MAX_REPORT_BYTES, + ); + const authority = validateLockReport(reportFile.value); + if (authority.report.lockDigest !== request.lockReport.expectedDigest) { + fail('lock report digest changed'); + } + const manifestContents = utf8(manifest, 'locked manifest'); + inspectLockedManifest(manifestContents, authority); + if (manifest.digest !== authority.report.manifest.outputDigest) { + fail('locked manifest and report digest differ'); + } + const executable = verifyExpected( + readStableFile(request.kubectl.path, 'kubectl', MAX_EXECUTABLE_BYTES, { + privateFile: false, + executable: true, + }), + request.kubectl.expectedDigest, + 'kubectl', + ); + const kubeconfig = verifyExpected( + readStableFile(request.kubeconfig.path, 'kubeconfig', MAX_KUBECONFIG_BYTES), + request.kubeconfig.expectedDigest, + 'kubeconfig', + ); + validateKubeconfig(utf8(kubeconfig, 'kubeconfig'), request.context); + return Object.freeze({ + manifest, + manifestContents, + reportFile, + authority, + executable, + kubeconfig, + }); +} + +function defaultRunProcess(executable, args, input) { + const privateHome = fs.mkdtempSync( + path.join(os.tmpdir(), 'ql3-kubectl-home-'), + ); + fs.chmodSync(privateHome, 0o700); + try { + return spawnSync(executable, args, { + input, + encoding: 'utf8', + timeout: PROCESS_TIMEOUT_MS, + maxBuffer: MAX_PROCESS_OUTPUT_BYTES, + shell: false, + windowsHide: true, + env: Object.freeze({ + HOME: privateHome, + XDG_CACHE_HOME: privateHome, + TMPDIR: privateHome, + LANG: 'C', + LC_ALL: 'C', + NO_COLOR: '1', + }), + }); + } finally { + fs.rmSync(privateHome, { recursive: true, force: true }); + } +} + +function runStep(inputs, request, name, argumentsAfterTarget, runner) { + const args = [ + `--kubeconfig=${inputs.kubeconfig.path}`, + `--context=${request.context}`, + '--request-timeout=60s', + ...argumentsAfterTarget, + ]; + const result = runner(inputs.executable.path, args, inputs.manifestContents); + const stdout = typeof result.stdout === 'string' ? result.stdout : ''; + const stderr = typeof result.stderr === 'string' ? result.stderr : ''; + if ( + result.error || + !Number.isInteger(result.status) || + Buffer.byteLength(stdout, 'utf8') > MAX_PROCESS_OUTPUT_BYTES || + Buffer.byteLength(stderr, 'utf8') > MAX_PROCESS_OUTPUT_BYTES + ) { + fail(`kubectl ${name} could not complete safely`); + } + return Object.freeze({ + name, + status: result.status, + argvDigest: sha256(Buffer.from(JSON.stringify(args), 'utf8')), + stdoutDigest: sha256(Buffer.from(stdout, 'utf8')), + stdoutBytes: Buffer.byteLength(stdout, 'utf8'), + stderrDigest: sha256(Buffer.from(stderr, 'utf8')), + stderrBytes: Buffer.byteLength(stderr, 'utf8'), + stdout, + }); +} + +function identityStep(inputs, request, runner, name) { + const step = runStep( + inputs, + request, + name, + ['get', 'namespace', 'kube-system', '-o=jsonpath={.metadata.uid}'], + runner, + ); + if (step.status !== 0 || step.stdout.trim() !== request.expectedClusterUid) { + fail('Kubernetes cluster identity changed'); + } + const { stdout, ...record } = step; + return Object.freeze(record); +} + +function dryRunStep(inputs, request, runner, name = 'server_side_dry_run') { + const step = runStep( + inputs, + request, + name, + [ + 'apply', + '--server-side', + '--dry-run=server', + `--field-manager=${FIELD_MANAGER}`, + '--validate=strict', + '-f=-', + '-o=name', + ], + runner, + ); + if (step.status !== 0) fail('Kubernetes server-side dry-run was rejected'); + const { stdout, ...record } = step; + return Object.freeze(record); +} + +function convergenceReadStep(inputs, request, runner) { + const step = runStep( + inputs, + request, + 'server_side_convergence_read', + ['get', '-f=-', '-o=json', '--show-managed-fields=true'], + runner, + ); + if (step.status !== 0) fail('Kubernetes convergence read failed'); + inspectConvergenceOutput(step.stdout, inputs); + const { stdout, ...record } = step; + return Object.freeze(record); +} + +function revalidateStableInputs(inputs, request) { + for (const [label, previous, maximumBytes, options] of [ + ['locked manifest', inputs.manifest, MAX_MANIFEST_BYTES, {}], + ['lock report', inputs.reportFile, MAX_REPORT_BYTES, {}], + [ + 'kubectl', + inputs.executable, + MAX_EXECUTABLE_BYTES, + { privateFile: false, executable: true }, + ], + ['kubeconfig', inputs.kubeconfig, MAX_KUBECONFIG_BYTES, {}], + ]) { + const current = readStableFile(previous.path, label, maximumBytes, options); + if ( + current.digest !== previous.digest || + !sameIdentity(current.identity, previous.identity) + ) { + fail(`${label} changed during the ceremony`); + } + } + if ( + inputs.manifest.digest !== request.lockedManifest.expectedDigest || + inputs.authority.report.lockDigest !== request.lockReport.expectedDigest || + inputs.executable.digest !== request.kubectl.expectedDigest || + inputs.kubeconfig.digest !== request.kubeconfig.expectedDigest + ) { + fail('deployment input authority changed'); + } +} + +function lockSummary(inputs) { + const report = inputs.authority.report; + return Object.freeze({ + lockDigest: report.lockDigest, + manifestDigest: inputs.manifest.digest, + lockReportDigest: inputs.reportFile.digest, + releaseSetDigest: report.releaseSetDigest, + catalogManifestDigest: report.catalog.manifestDigest, + catalogReportDigest: report.catalog.consumptionReportDigest, + catalogImmutableReference: report.catalog.immutableReference, + release: Object.freeze({ ...report.release }), + requiredImages: Object.freeze([...report.requiredImages]), + }); +} + +function targetSummary(inputs, request) { + return Object.freeze({ + context: request.context, + clusterUid: request.expectedClusterUid, + kubeconfigDigest: inputs.kubeconfig.digest, + fieldManager: FIELD_MANAGER, + }); +} + +function toolSummary(inputs) { + return Object.freeze({ + name: 'kubectl', + executableDigest: inputs.executable.digest, + }); +} + +function validatePreflight(value) { + if ( + !exactKeys(value, [ + 'schemaVersion', + 'schema', + 'preflightId', + 'commandDigest', + 'lock', + 'target', + 'tool', + 'steps', + 'verification', + 'preflightDigest', + ]) || + value.schemaVersion !== 1 || + value.schema !== PREFLIGHT_SCHEMA || + !UUID_PATTERN.test(value.preflightId || '') || + !DIGEST_PATTERN.test(value.commandDigest || '') || + !DIGEST_PATTERN.test(value.preflightDigest || '') || + !Array.isArray(value.steps) || + value.steps.length !== 2 || + value.steps[0]?.name !== 'cluster_identity_before' || + value.steps[1]?.name !== 'server_side_dry_run' || + !exactKeys(value.verification, [ + 'catalogBoundLock', + 'clusterIdentityBound', + 'serverSideDryRun', + 'networkAccess', + 'kubernetesMutation', + 'externalResultsReplayed', + 'ambientHome', + 'ephemeralCache', + ]) || + JSON.stringify(value.verification) !== + JSON.stringify({ + catalogBoundLock: true, + clusterIdentityBound: true, + serverSideDryRun: true, + networkAccess: true, + kubernetesMutation: false, + externalResultsReplayed: false, + ambientHome: false, + ephemeralCache: true, + }) + ) { + fail('deployment preflight report is invalid'); + } + validateLockSummary(value.lock); + validateTargetSummary(value.target); + validateToolSummary(value.tool); + for (const step of value.steps) validateStepRecord(step); + if (value.steps.some((step) => step.status !== 0)) { + fail('deployment preflight contains a failed step'); + } + const { preflightDigest, ...unsigned } = value; + if ( + preflightDigest !== sha256(Buffer.from(JSON.stringify(unsigned), 'utf8')) + ) { + fail('deployment preflight self digest is invalid'); + } + return value; +} + +function validateLockSummary(value) { + if ( + !exactKeys(value, [ + 'lockDigest', + 'manifestDigest', + 'lockReportDigest', + 'releaseSetDigest', + 'catalogManifestDigest', + 'catalogReportDigest', + 'catalogImmutableReference', + 'release', + 'requiredImages', + ]) || + !DIGEST_PATTERN.test(value.lockDigest || '') || + !DIGEST_PATTERN.test(value.manifestDigest || '') || + !DIGEST_PATTERN.test(value.lockReportDigest || '') || + !DIGEST_PATTERN.test(value.releaseSetDigest || '') || + !DIGEST_PATTERN.test(value.catalogManifestDigest || '') || + !DIGEST_PATTERN.test(value.catalogReportDigest || '') || + !Array.isArray(value.requiredImages) || + value.requiredImages.length < 1 || + JSON.stringify(value.requiredImages) !== + JSON.stringify( + ROLE_ORDER.filter((role) => value.requiredImages.includes(role)), + ) + ) { + fail('deployment lock summary is invalid'); + } + validateRelease(value.release); + if ( + !new RegExp( + `^ghcr\\.io/[a-z0-9](?:[a-z0-9-]{0,37}[a-z0-9])?/qinglong3-release-catalog@${value.catalogManifestDigest}$`, + 'u', + ).test(value.catalogImmutableReference || '') + ) { + fail('deployment lock summary catalog reference is invalid'); + } +} + +function validateTargetSummary(value) { + if ( + !exactKeys(value, [ + 'context', + 'clusterUid', + 'kubeconfigDigest', + 'fieldManager', + ]) || + value.fieldManager !== FIELD_MANAGER || + !CONTEXT_PATTERN.test(value.context || '') || + !UUID_PATTERN.test(value.clusterUid || '') || + !DIGEST_PATTERN.test(value.kubeconfigDigest || '') + ) { + fail('deployment target summary is invalid'); + } +} + +function validateToolSummary(value) { + if ( + !exactKeys(value, ['name', 'executableDigest']) || + value.name !== 'kubectl' || + !DIGEST_PATTERN.test(value.executableDigest || '') + ) { + fail('deployment tool summary is invalid'); + } +} + +function validateStepRecord(step) { + if ( + !exactKeys(step, [ + 'name', + 'status', + 'argvDigest', + 'stdoutDigest', + 'stdoutBytes', + 'stderrDigest', + 'stderrBytes', + ]) || + typeof step.name !== 'string' || + !Number.isInteger(step.status) || + !DIGEST_PATTERN.test(step.argvDigest || '') || + !DIGEST_PATTERN.test(step.stdoutDigest || '') || + !Number.isSafeInteger(step.stdoutBytes) || + step.stdoutBytes < 0 || + step.stdoutBytes > MAX_PROCESS_OUTPUT_BYTES || + !DIGEST_PATTERN.test(step.stderrDigest || '') || + !Number.isSafeInteger(step.stderrBytes) || + step.stderrBytes < 0 || + step.stderrBytes > MAX_PROCESS_OUTPUT_BYTES + ) { + fail('deployment command step is invalid'); + } +} + +function createPreflight(commandState, dependencies = {}) { + const request = commandState.command.request; + const output = outputTarget(request.output, 'preflight output'); + const inputs = inspectInputs(request); + const runner = dependencies.runProcess ?? defaultRunProcess; + const steps = [ + identityStep(inputs, request, runner, 'cluster_identity_before'), + dryRunStep(inputs, request, runner), + ]; + revalidateStableInputs(inputs, request); + const unsigned = { + schemaVersion: 1, + schema: PREFLIGHT_SCHEMA, + preflightId: request.preflightId, + commandDigest: commandState.commandDigest, + lock: lockSummary(inputs), + target: targetSummary(inputs, request), + tool: toolSummary(inputs), + steps, + verification: { + catalogBoundLock: true, + clusterIdentityBound: true, + serverSideDryRun: true, + networkAccess: true, + kubernetesMutation: false, + externalResultsReplayed: false, + ambientHome: false, + ephemeralCache: true, + }, + }; + const result = Object.freeze({ + ...unsigned, + preflightDigest: sha256(Buffer.from(JSON.stringify(unsigned), 'utf8')), + }); + writeNoReplace(output, result); + return result; +} + +function readAndValidatePreflight(request, inputs) { + const file = readCanonicalJson( + request.preflight.path, + 'preflight report', + MAX_REPORT_BYTES, + ); + const report = validatePreflight(file.value); + if ( + report.preflightDigest !== request.preflight.expectedDigest || + JSON.stringify(report.lock) !== JSON.stringify(lockSummary(inputs)) || + JSON.stringify(report.target) !== + JSON.stringify(targetSummary(inputs, request)) || + JSON.stringify(report.tool) !== JSON.stringify(toolSummary(inputs)) + ) { + fail('deployment preflight binding changed'); + } + return Object.freeze({ file, report }); +} + +function validateReceipt(value, expectedCommandDigest) { + if ( + !exactKeys(value, [ + 'schemaVersion', + 'schema', + 'mutationId', + 'commandDigest', + 'preflightDigest', + 'lock', + 'target', + 'tool', + 'steps', + 'verification', + 'receiptDigest', + ]) || + value.schemaVersion !== 1 || + value.schema !== RECEIPT_SCHEMA || + !UUID_PATTERN.test(value.mutationId || '') || + value.commandDigest !== expectedCommandDigest || + !DIGEST_PATTERN.test(value.preflightDigest || '') || + !DIGEST_PATTERN.test(value.receiptDigest || '') || + !Array.isArray(value.steps) || + JSON.stringify(value.steps.map((step) => step.name)) !== + JSON.stringify([ + 'cluster_identity_before', + 'server_side_dry_run', + 'server_side_apply', + 'server_side_convergence_read', + 'cluster_identity_after', + ]) || + !exactKeys(value.verification, [ + 'catalogBoundLock', + 'clusterIdentityBound', + 'serverSideDryRun', + 'serverSideApply', + 'convergenceRead', + 'networkAccess', + 'kubernetesMutation', + 'crossResourceAtomicity', + 'externalResultsReplayed', + 'recovery', + 'ambientHome', + 'ephemeralCache', + ]) || + JSON.stringify(value.verification) !== + JSON.stringify({ + catalogBoundLock: true, + clusterIdentityBound: true, + serverSideDryRun: true, + serverSideApply: true, + convergenceRead: true, + networkAccess: true, + kubernetesMutation: true, + crossResourceAtomicity: false, + externalResultsReplayed: false, + recovery: 'reapply_exact_lock_with_same_field_manager', + ambientHome: false, + ephemeralCache: true, + }) + ) { + fail('deployment receipt is invalid'); + } + validateLockSummary(value.lock); + validateTargetSummary(value.target); + validateToolSummary(value.tool); + for (const step of value.steps) validateStepRecord(step); + if (value.steps.some((step) => step.status !== 0)) { + fail('deployment receipt contains a failed step'); + } + const { receiptDigest, ...unsigned } = value; + if (receiptDigest !== sha256(Buffer.from(JSON.stringify(unsigned), 'utf8'))) { + fail('deployment receipt self digest is invalid'); + } + return value; +} + +function existingReceipt(request, commandDigest) { + if (!fs.existsSync(request.output)) return null; + const file = readCanonicalJson( + request.output, + 'existing deployment receipt', + MAX_REPORT_BYTES, + ); + const receipt = validateReceipt(file.value, commandDigest); + if ( + receipt.mutationId !== request.mutationId || + receipt.preflightDigest !== request.preflight.expectedDigest + ) { + fail('existing deployment receipt binding changed'); + } + return receipt; +} + +function applyDeployment(commandState, dependencies = {}) { + const request = commandState.command.request; + const resumed = existingReceipt(request, commandState.commandDigest); + if (resumed !== null) { + const inputs = inspectInputs(request); + const preflight = readAndValidatePreflight(request, inputs); + if ( + resumed.preflightDigest !== preflight.report.preflightDigest || + JSON.stringify(resumed.lock) !== JSON.stringify(lockSummary(inputs)) || + JSON.stringify(resumed.target) !== + JSON.stringify(targetSummary(inputs, request)) || + JSON.stringify(resumed.tool) !== JSON.stringify(toolSummary(inputs)) + ) { + fail('existing deployment receipt authority changed'); + } + revalidateStableInputs(inputs, request); + return resumed; + } + const output = outputTarget(request.output, 'apply receipt output'); + const inputs = inspectInputs(request); + const preflight = readAndValidatePreflight(request, inputs); + const runner = dependencies.runProcess ?? defaultRunProcess; + const steps = [ + identityStep(inputs, request, runner, 'cluster_identity_before'), + dryRunStep(inputs, request, runner), + ]; + const applied = runStep( + inputs, + request, + 'server_side_apply', + [ + 'apply', + '--server-side', + `--field-manager=${FIELD_MANAGER}`, + '--validate=strict', + '-f=-', + '-o=name', + ], + runner, + ); + if (applied.status !== 0) fail('Kubernetes server-side apply failed'); + const { stdout: _appliedOutput, ...appliedRecord } = applied; + steps.push(Object.freeze(appliedRecord)); + steps.push(convergenceReadStep(inputs, request, runner)); + steps.push(identityStep(inputs, request, runner, 'cluster_identity_after')); + revalidateStableInputs(inputs, request); + const unsigned = { + schemaVersion: 1, + schema: RECEIPT_SCHEMA, + mutationId: request.mutationId, + commandDigest: commandState.commandDigest, + preflightDigest: preflight.report.preflightDigest, + lock: lockSummary(inputs), + target: targetSummary(inputs, request), + tool: toolSummary(inputs), + steps, + verification: { + catalogBoundLock: true, + clusterIdentityBound: true, + serverSideDryRun: true, + serverSideApply: true, + convergenceRead: true, + networkAccess: true, + kubernetesMutation: true, + crossResourceAtomicity: false, + externalResultsReplayed: false, + recovery: 'reapply_exact_lock_with_same_field_manager', + ambientHome: false, + ephemeralCache: true, + }, + }; + const receipt = Object.freeze({ + ...unsigned, + receiptDigest: sha256(Buffer.from(JSON.stringify(unsigned), 'utf8')), + }); + writeNoReplace(output, receipt); + return receipt; +} + +function auditReceipt(command) { + const applyFile = verifyExpected( + readCanonicalJson( + command.request.applyCommand.path, + 'apply command', + MAX_COMMAND_BYTES, + ), + command.request.applyCommand.expectedDigest, + 'apply command', + ); + const applyCommand = parseCommand(applyFile.value); + if (applyCommand.operation !== 'cluster.deployment.apply') { + fail('receipt audit requires an apply command'); + } + const receiptFile = readCanonicalJson( + command.request.receipt.path, + 'deployment receipt', + MAX_REPORT_BYTES, + ); + const receipt = validateReceipt(receiptFile.value, applyFile.digest); + if ( + receipt.receiptDigest !== command.request.receipt.expectedDigest || + receipt.mutationId !== applyCommand.request.mutationId || + receipt.preflightDigest !== applyCommand.request.preflight.expectedDigest || + applyCommand.request.output !== command.request.receipt.path + ) { + fail('deployment receipt command binding changed'); + } + return Object.freeze({ + compatible: true, + deploymentFamily: 'cluster', + mutationId: receipt.mutationId, + receiptDigest: receipt.receiptDigest, + preflightDigest: receipt.preflightDigest, + lockDigest: receipt.lock.lockDigest, + manifestDigest: receipt.lock.manifestDigest, + clusterUid: receipt.target.clusterUid, + externalResultsReplayed: false, + kubernetesMutation: false, + }); +} + +function executeCommand(commandFile, dependencies = {}) { + const commandState = readCommand(commandFile); + if (commandState.command.operation === 'cluster.deployment.preflight') { + return createPreflight(commandState, dependencies); + } + if (commandState.command.operation === 'cluster.deployment.apply') { + return applyDeployment(commandState, dependencies); + } + return auditReceipt(commandState.command); +} + +module.exports = Object.freeze({ + COMMAND_SCHEMA, + FIELD_MANAGER, + LOCK_SCHEMA, + PREFLIGHT_SCHEMA, + RECEIPT_SCHEMA, + QingLong3KubernetesDeploymentCeremonyError, + canonicalJson, + executeCommand, + parseCommand, + validateLockReport, + validatePreflight, + validateReceipt, +}); diff --git a/scripts/ql3-kubernetes-deployment-ceremony.cjs b/scripts/ql3-kubernetes-deployment-ceremony.cjs new file mode 100644 index 00000000..aca10951 --- /dev/null +++ b/scripts/ql3-kubernetes-deployment-ceremony.cjs @@ -0,0 +1,49 @@ +#!/usr/bin/env node + +'use strict'; + +const { + QingLong3KubernetesDeploymentCeremonyError, + canonicalJson, + executeCommand, +} = require('./lib/ql3-kubernetes-deployment-ceremony.cjs'); + +function commandFile(argv) { + if (argv.length !== 1) throw new Error('arguments are invalid'); + const match = /^--command-file=(.+)$/u.exec(argv[0]); + if (!match) throw new Error('arguments are invalid'); + return match[1]; +} + +function lowSensitivityFailure() { + return Object.freeze({ + schemaVersion: 1, + component: 'qinglong3-kubernetes-deployment-ceremony', + code: 'QL3_KUBERNETES_DEPLOYMENT_CEREMONY_FAILED', + message: 'QingLong 3 Kubernetes deployment ceremony failed', + }); +} + +function runCli(argv, output = process.stdout, errorOutput = process.stderr) { + try { + const result = executeCommand(commandFile(argv)); + output.write(canonicalJson(result)); + return result; + } catch (error) { + errorOutput.write(canonicalJson(lowSensitivityFailure())); + if ( + !(error instanceof QingLong3KubernetesDeploymentCeremonyError) && + process.env.QL3_DEBUG_DEPLOYMENT_CEREMONY === 'true' + ) { + errorOutput.write( + `${error instanceof Error ? error.message : 'error'}\n`, + ); + } + process.exitCode = 1; + return undefined; + } +} + +if (require.main === module) runCli(process.argv.slice(2)); + +module.exports = Object.freeze({ commandFile, runCli }); diff --git a/scripts/ql3-kubernetes-deployment-live-contract.cjs b/scripts/ql3-kubernetes-deployment-live-contract.cjs new file mode 100644 index 00000000..d3f6ffbd --- /dev/null +++ b/scripts/ql3-kubernetes-deployment-live-contract.cjs @@ -0,0 +1,377 @@ +#!/usr/bin/env node + +'use strict'; + +const crypto = require('node:crypto'); +const fs = require('node:fs'); +const path = require('node:path'); +const { + COMMAND_SCHEMA, + FIELD_MANAGER, + canonicalJson, + executeCommand, +} = require('./lib/ql3-kubernetes-deployment-ceremony.cjs'); +const { K3sDockerLiveFixture } = require('./lib/ql3-k3s-docker-live.cjs'); +const { readReleaseIdentity } = require('./lib/ql3-release-identity.cjs'); + +const ROOT = path.resolve(__dirname, '..'); +const VERSION = readReleaseIdentity(ROOT).version; +const SOURCE_REVISION = 'd'.repeat(40); +const NAMESPACE = 'ql3-deployment-live'; +const CONTEXT = 'default'; +const OWNER = 'qinglong-release'; +const ROLE_ORDER = Object.freeze(['control', 'control-ai', 'admin', 'worker']); +const IMAGE_NAMES = Object.freeze({ + control: 'qinglong3-cluster-control', + 'control-ai': 'qinglong3-cluster-control-ai', + admin: 'qinglong3-cluster-admin', + worker: 'qinglong3-worker', +}); + +function fail(message) { + throw new Error( + `QingLong Kubernetes deployment live contract failed: ${message}`, + ); +} + +function digest(value) { + return `sha256:${crypto.createHash('sha256').update(value).digest('hex')}`; +} + +function privateFile(directory, name, contents) { + const target = path.join(directory, name); + fs.writeFileSync(target, contents, { mode: 0o600, flag: 'wx' }); + return target; +} + +function executablePath(input) { + const candidates = [ + input, + '/Applications/Docker.app/Contents/Resources/bin/kubectl', + ...(process.env.PATH ?? '') + .split(path.delimiter) + .filter(Boolean) + .map((directory) => path.join(directory, input)), + ]; + for (const candidate of candidates) { + try { + const resolved = fs.realpathSync(candidate); + if (fs.lstatSync(resolved).isFile()) return resolved; + } catch {} + } + fail('kubectl executable is unavailable'); +} + +function references() { + return Object.fromEntries( + ROLE_ORDER.map((role, index) => [ + role, + `ghcr.io/${OWNER}/${IMAGE_NAMES[role]}@sha256:${String(index + 1).repeat( + 64, + )}`, + ]), + ); +} + +function lockedArtifacts() { + const releaseSetDigest = digest('d341-live-release-set'); + const catalogManifestDigest = digest('d341-live-catalog-manifest'); + const catalogReportDigest = digest('d341-live-catalog-report'); + const imageReferences = references(); + const annotations = { + 'qinglong.io/release-set-digest': releaseSetDigest, + 'qinglong.io/release-catalog-manifest-digest': catalogManifestDigest, + 'qinglong.io/release-catalog-report-digest': catalogReportDigest, + 'qinglong.io/release-source-revision': SOURCE_REVISION, + 'qinglong.io/release-version': VERSION, + }; + const deployment = (role) => ({ + apiVersion: 'apps/v1', + kind: 'Deployment', + metadata: { + name: `ql3-${role.replace('control-ai', 'control-ai')}`, + namespace: NAMESPACE, + annotations, + }, + spec: { + replicas: 0, + selector: { matchLabels: { 'app.kubernetes.io/name': `ql3-${role}` } }, + template: { + metadata: { + labels: { 'app.kubernetes.io/name': `ql3-${role}` }, + annotations, + }, + spec: { + containers: [ + { + name: role, + image: imageReferences[role], + command: ['/bin/false'], + }, + ], + }, + }, + }, + }); + const resources = [ + { + apiVersion: 'v1', + kind: 'Namespace', + metadata: { name: NAMESPACE }, + }, + ...ROLE_ORDER.map(deployment), + { + apiVersion: 'v1', + kind: 'ConfigMap', + metadata: { + name: 'ql3-plugin-package-secret-action-admission', + namespace: NAMESPACE, + annotations, + }, + data: { image: imageReferences.admin }, + }, + ]; + const manifest = `${resources + .map((resource) => JSON.stringify(resource)) + .join('\n---\n')}\n`; + const unsigned = { + schemaVersion: 1, + schema: 'qinglong/kubernetes-deployment-lock@v2', + release: { + version: VERSION, + sourceRevision: SOURCE_REVISION, + sourceRef: `refs/tags/v${VERSION}`, + scope: 'cluster', + }, + releaseSetDigest, + catalog: { + schema: 'qinglong/release-catalog-consumption-ceremony@v1', + sourceRepository: `${OWNER}/qinglong`, + workflowIdentity: `https://github.com/${OWNER}/qinglong/.github/workflows/ql3-image-release.yml@refs/tags/v${VERSION}`, + immutableReference: `ghcr.io/${OWNER}/qinglong3-release-catalog@${catalogManifestDigest}`, + manifestDigest: catalogManifestDigest, + consumptionReportDigest: catalogReportDigest, + releaseSetDigest, + discoveryTagAuthority: 'none', + }, + deploymentFamily: 'cluster', + requiredImages: [...ROLE_ORDER], + imageOccurrences: ROLE_ORDER.map((role) => ({ + name: role, + reference: imageReferences[role], + count: role === 'admin' ? 2 : 1, + })), + manifest: { + inputDigest: digest('d341-live-source-render'), + outputDigest: digest(manifest), + resources: resources.length, + changedResources: 5, + admissionAuthorityCount: 1, + }, + verification: { + releaseSet: 'standalone_structure_identity_and_self_digest', + sourceRecordsReplayed: false, + catalogConsumption: 'offline_reconstructed', + externalToolResultsReplayed: false, + unknownImageAuthorities: 0, + mutableQingLongImages: 0, + networkAccess: false, + kubernetesMutation: false, + }, + }; + return Object.freeze({ + manifest, + report: Object.freeze({ + ...unsigned, + lockDigest: digest(JSON.stringify(unsigned)), + }), + imageReferences: Object.freeze(imageReferences), + }); +} + +function writeCommand(directory, name, operation, request) { + return privateFile( + directory, + name, + canonicalJson({ + schemaVersion: 1, + schema: COMMAND_SCHEMA, + operation, + request, + }), + ); +} + +async function main() { + const fixture = new K3sDockerLiveFixture({ + prefix: 'ql3-deploy-live', + kubectl: process.env.QL3_KUBECTL_BIN, + }); + let evidence; + let cleanupComplete = false; + try { + const nodes = await fixture.start(); + fs.chmodSync(fixture.temporary, 0o700); + const ceremonyDirectory = fs.realpathSync(fixture.temporary); + const kubeconfig = fs.realpathSync(fixture.kubeconfig); + const kubectl = executablePath(fixture.kubectlBinary); + const clusterUid = fixture + .kubectl( + ['get', 'namespace', 'kube-system', '-o=jsonpath={.metadata.uid}'], + { capture: true, quiet: true }, + ) + .stdout.trim(); + fixture.apply({ + apiVersion: 'v1', + kind: 'Namespace', + metadata: { name: NAMESPACE }, + }); + const artifacts = lockedArtifacts(); + const manifestPath = privateFile( + ceremonyDirectory, + 'locked.yaml', + artifacts.manifest, + ); + const reportPath = privateFile( + ceremonyDirectory, + 'lock.json', + canonicalJson(artifacts.report), + ); + const common = { + lockedManifest: { + path: manifestPath, + expectedDigest: artifacts.report.manifest.outputDigest, + }, + lockReport: { + path: reportPath, + expectedDigest: artifacts.report.lockDigest, + }, + kubectl: { + path: kubectl, + expectedDigest: digest(fs.readFileSync(kubectl)), + }, + kubeconfig: { + path: kubeconfig, + expectedDigest: digest(fs.readFileSync(kubeconfig)), + }, + context: CONTEXT, + expectedClusterUid: clusterUid, + }; + const preflightPath = path.join(ceremonyDirectory, 'preflight.json'); + const preflightCommand = writeCommand( + ceremonyDirectory, + 'preflight-command.json', + 'cluster.deployment.preflight', + { + preflightId: crypto.randomUUID(), + ...common, + output: preflightPath, + }, + ); + const preflight = executeCommand(preflightCommand); + const receiptPath = path.join(ceremonyDirectory, 'receipt.json'); + const applyCommand = writeCommand( + ceremonyDirectory, + 'apply-command.json', + 'cluster.deployment.apply', + { + mutationId: crypto.randomUUID(), + preflight: { + path: preflightPath, + expectedDigest: preflight.preflightDigest, + }, + ...common, + output: receiptPath, + }, + ); + const receipt = executeCommand(applyCommand); + const auditCommand = writeCommand( + ceremonyDirectory, + 'audit-command.json', + 'cluster.deployment.receipt.audit', + { + applyCommand: { + path: applyCommand, + expectedDigest: digest(fs.readFileSync(applyCommand)), + }, + receipt: { path: receiptPath, expectedDigest: receipt.receiptDigest }, + }, + ); + const audit = executeCommand(auditCommand); + const deployments = fixture.kubectlJson([ + 'get', + 'deployments', + '-n', + NAMESPACE, + '--show-managed-fields=true', + ]).items; + if (deployments.length !== ROLE_ORDER.length) { + fail('applied deployment count is invalid'); + } + for (const deployment of deployments) { + const container = deployment.spec?.template?.spec?.containers?.[0]; + if ( + !Object.values(artifacts.imageReferences).includes(container?.image) + ) { + fail('applied immutable image authority is invalid'); + } + if ( + !deployment.metadata?.managedFields?.some( + (entry) => entry.manager === FIELD_MANAGER, + ) + ) { + fail('server-side apply field manager is unavailable'); + } + } + const version = JSON.parse( + fixture.kubectl(['version', '-o=json'], { + capture: true, + quiet: true, + }).stdout, + ); + evidence = Object.freeze({ + schemaVersion: 1, + schema: 'qinglong/kubernetes-deployment-live-evidence@v1', + kubernetes: { + serverVersion: version.serverVersion.gitVersion, + architecture: version.serverVersion.platform, + nodeCount: nodes.length, + clusterUid, + }, + deployment: { + namespace: NAMESPACE, + resourceCount: artifacts.report.manifest.resources, + deploymentCount: deployments.length, + replicas: 0, + fieldManager: FIELD_MANAGER, + immutableImages: true, + }, + preflightDigest: preflight.preflightDigest, + receiptDigest: receipt.receiptDigest, + receiptAuditCompatible: audit.compatible, + serverSideDryRun: preflight.verification.serverSideDryRun, + serverSideApply: receipt.verification.serverSideApply, + convergenceRead: receipt.verification.convergenceRead, + crossResourceAtomicity: receipt.verification.crossResourceAtomicity, + }); + } finally { + await fixture.cleanup().catch(() => undefined); + cleanupComplete = true; + } + process.stdout.write(canonicalJson({ ...evidence, cleanupComplete })); +} + +if (require.main === module) { + main().catch((error) => { + process.stderr.write( + `${ + error instanceof Error + ? error.message + : 'deployment live contract failed' + }\n`, + ); + process.exitCode = 1; + }); +} + +module.exports = Object.freeze({ lockedArtifacts, main }); diff --git a/test/back/ql3KubernetesDeploymentCeremony.test.cjs b/test/back/ql3KubernetesDeploymentCeremony.test.cjs new file mode 100644 index 00000000..4ef469c8 --- /dev/null +++ b/test/back/ql3KubernetesDeploymentCeremony.test.cjs @@ -0,0 +1,786 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const crypto = require('node:crypto'); +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const { spawnSync } = require('node:child_process'); +const test = require('node:test'); +const yaml = require('js-yaml'); +const { + COMMAND_SCHEMA, + FIELD_MANAGER, + PREFLIGHT_SCHEMA, + RECEIPT_SCHEMA, + canonicalJson, + executeCommand, + parseCommand, + validateLockReport, +} = require('../../scripts/lib/ql3-kubernetes-deployment-ceremony.cjs'); +const { + commandFile, +} = require('../../scripts/ql3-kubernetes-deployment-ceremony.cjs'); + +const CLUSTER_UID = '123e4567-e89b-42d3-a456-426614174000'; +const RELEASE_SET_DIGEST = digest('release-set'); +const CATALOG_MANIFEST_DIGEST = digest('catalog-manifest'); +const CATALOG_REPORT_DIGEST = digest('catalog-report'); +const SOURCE_REVISION = 'd'.repeat(40); +const VERSION = '3.0.0-alpha.0'; +const CONTEXT = 'qinglong-production'; +const REFERENCES = Object.freeze({ + control: image('qinglong3-cluster-control', '1'), + 'control-ai': image('qinglong3-cluster-control-ai', '2'), + admin: image('qinglong3-cluster-admin', '3'), + worker: image('qinglong3-worker', '4'), +}); + +function digest(value) { + return `sha256:${crypto.createHash('sha256').update(value).digest('hex')}`; +} + +function image(repository, digit) { + return `ghcr.io/qinglong-release/${repository}@sha256:${digit.repeat(64)}`; +} + +function annotations() { + return { + 'qinglong.io/release-set-digest': RELEASE_SET_DIGEST, + 'qinglong.io/release-catalog-manifest-digest': CATALOG_MANIFEST_DIGEST, + 'qinglong.io/release-catalog-report-digest': CATALOG_REPORT_DIGEST, + 'qinglong.io/release-source-revision': SOURCE_REVISION, + 'qinglong.io/release-version': VERSION, + }; +} + +function manifest() { + const metadata = (name) => ({ + name, + namespace: 'qinglong-system', + annotations: annotations(), + }); + const deployment = (name, containerName, reference) => ({ + apiVersion: 'apps/v1', + kind: 'Deployment', + metadata: metadata(name), + spec: { + selector: { matchLabels: { app: name } }, + template: { + metadata: { labels: { app: name }, annotations: annotations() }, + spec: { containers: [{ name: containerName, image: reference }] }, + }, + }, + }); + const resources = [ + deployment('ql3-control', 'control', REFERENCES.control), + deployment('ql3-control-ai', 'control-ai', REFERENCES['control-ai']), + { + apiVersion: 'batch/v1', + kind: 'Job', + metadata: metadata('ql3-admin'), + spec: { + template: { + metadata: { annotations: annotations() }, + spec: { + restartPolicy: 'Never', + containers: [{ name: 'admin', image: REFERENCES.admin }], + }, + }, + }, + }, + deployment('ql3-worker', 'worker', REFERENCES.worker), + { + apiVersion: 'v1', + kind: 'ConfigMap', + metadata: metadata('ql3-plugin-package-secret-action-admission'), + data: { image: REFERENCES.admin }, + }, + ]; + return `${resources + .map((resource) => JSON.stringify(resource)) + .join('\n---\n')}\n`; +} + +function lockReport(manifestContents = manifest()) { + const unsigned = { + schemaVersion: 1, + schema: 'qinglong/kubernetes-deployment-lock@v2', + release: { + version: VERSION, + sourceRevision: SOURCE_REVISION, + sourceRef: `refs/tags/v${VERSION}`, + scope: 'cluster', + }, + releaseSetDigest: RELEASE_SET_DIGEST, + catalog: { + schema: 'qinglong/release-catalog-consumption-ceremony@v1', + sourceRepository: 'qinglong-release/qinglong', + workflowIdentity: `https://github.com/qinglong-release/qinglong/.github/workflows/ql3-image-release.yml@refs/tags/v${VERSION}`, + immutableReference: `ghcr.io/qinglong-release/qinglong3-release-catalog@${CATALOG_MANIFEST_DIGEST}`, + manifestDigest: CATALOG_MANIFEST_DIGEST, + consumptionReportDigest: CATALOG_REPORT_DIGEST, + releaseSetDigest: RELEASE_SET_DIGEST, + discoveryTagAuthority: 'none', + }, + deploymentFamily: 'cluster', + requiredImages: ['control', 'control-ai', 'admin', 'worker'], + imageOccurrences: [ + { name: 'control', reference: REFERENCES.control, count: 1 }, + { + name: 'control-ai', + reference: REFERENCES['control-ai'], + count: 1, + }, + { name: 'admin', reference: REFERENCES.admin, count: 2 }, + { name: 'worker', reference: REFERENCES.worker, count: 1 }, + ], + manifest: { + inputDigest: digest('source-render'), + outputDigest: digest(manifestContents), + resources: 5, + changedResources: 5, + admissionAuthorityCount: 1, + }, + verification: { + releaseSet: 'standalone_structure_identity_and_self_digest', + sourceRecordsReplayed: false, + catalogConsumption: 'offline_reconstructed', + externalToolResultsReplayed: false, + unknownImageAuthorities: 0, + mutableQingLongImages: 0, + networkAccess: false, + kubernetesMutation: false, + }, + }; + return { + ...unsigned, + lockDigest: digest(JSON.stringify(unsigned)), + }; +} + +function fixture(t) { + const directory = fs.realpathSync( + fs.mkdtempSync(path.join(os.tmpdir(), 'ql3-kubernetes-deployment-')), + ); + fs.chmodSync(directory, 0o700); + t.after(() => fs.rmSync(directory, { recursive: true, force: true })); + const lockedManifest = privateFile(directory, 'locked.yaml', manifest()); + const report = lockReport(); + const lockReportPath = privateFile( + directory, + 'lock.json', + canonicalJson(report), + ); + const kubectl = privateFile( + directory, + 'kubectl', + '#!/bin/sh\nexit 97\n', + 0o700, + ); + const kubeconfig = privateFile( + directory, + 'kubeconfig.yaml', + `apiVersion: v1 +kind: Config +current-context: ${CONTEXT} +clusters: + - name: ql3 + cluster: + server: https://cluster.example.test + certificate-authority-data: Y2E= +contexts: + - name: ${CONTEXT} + context: + cluster: ql3 + user: operator +users: + - name: operator + user: + token: bounded-test-token +`, + ); + return { + directory, + lockedManifest, + lockReportPath, + kubectl, + kubeconfig, + report, + }; +} + +function privateFile(directory, name, contents, mode = 0o600) { + const target = path.join(directory, name); + fs.writeFileSync(target, contents, { mode }); + return target; +} + +function fileDigest(filePath) { + return digest(fs.readFileSync(filePath)); +} + +function writeCommand(directory, name, operation, request) { + const value = { + schemaVersion: 1, + schema: COMMAND_SCHEMA, + operation, + request, + }; + return privateFile(directory, name, canonicalJson(value)); +} + +function commonRequest(fixtureValue) { + return { + lockedManifest: { + path: fixtureValue.lockedManifest, + expectedDigest: fixtureValue.report.manifest.outputDigest, + }, + lockReport: { + path: fixtureValue.lockReportPath, + expectedDigest: fixtureValue.report.lockDigest, + }, + kubectl: { + path: fixtureValue.kubectl, + expectedDigest: fileDigest(fixtureValue.kubectl), + }, + kubeconfig: { + path: fixtureValue.kubeconfig, + expectedDigest: fileDigest(fixtureValue.kubeconfig), + }, + context: CONTEXT, + expectedClusterUid: CLUSTER_UID, + }; +} + +function successfulRunner(calls) { + return (_executable, args, input) => { + calls.push({ args: [...args], input }); + if (args.includes('get') && args.includes('-f=-')) { + return { status: 0, stdout: convergenceList(), stderr: '' }; + } + if (args.includes('get')) { + return { status: 0, stdout: CLUSTER_UID, stderr: '' }; + } + if (args.includes('--dry-run=server')) { + return { status: 0, stdout: 'deployment.apps/ql3-control\n', stderr: '' }; + } + return { status: 0, stdout: 'deployment.apps/ql3-control\n', stderr: '' }; + }; +} + +function convergenceList() { + const resources = []; + yaml.loadAll(manifest(), (resource) => resources.push(resource)); + return JSON.stringify({ + apiVersion: 'v1', + kind: 'List', + items: resources.map((resource, index) => ({ + ...resource, + metadata: { + ...resource.metadata, + uid: `123e4567-e89b-42d3-a456-4266141741${String(index).padStart( + 2, + '0', + )}`, + resourceVersion: String(index + 1), + managedFields: [ + { + manager: FIELD_MANAGER, + operation: 'Apply', + apiVersion: resource.apiVersion, + }, + ], + }, + })), + }); +} + +function prepare(t) { + const value = fixture(t); + const output = path.join(value.directory, 'preflight.json'); + const command = writeCommand( + value.directory, + 'preflight-command.json', + 'cluster.deployment.preflight', + { + preflightId: '123e4567-e89b-42d3-a456-426614174001', + ...commonRequest(value), + output, + }, + ); + const calls = []; + const report = executeCommand(command, { + runProcess: successfulRunner(calls), + }); + return { + ...value, + preflightCommand: command, + preflightPath: output, + preflight: report, + calls, + }; +} + +test('preflight binds one catalog lock to one Kubernetes target without mutation', (t) => { + const value = prepare(t); + assert.equal(value.preflight.schema, PREFLIGHT_SCHEMA); + assert.equal(value.preflight.lock.lockDigest, value.report.lockDigest); + assert.equal( + value.preflight.lock.manifestDigest, + value.report.manifest.outputDigest, + ); + assert.equal(value.preflight.target.clusterUid, CLUSTER_UID); + assert.equal(value.preflight.target.fieldManager, FIELD_MANAGER); + assert.equal(value.preflight.verification.serverSideDryRun, true); + assert.equal(value.preflight.verification.kubernetesMutation, false); + assert.deepEqual( + value.preflight.steps.map(({ name }) => name), + ['cluster_identity_before', 'server_side_dry_run'], + ); + assert.equal(fs.statSync(value.preflightPath).mode & 0o777, 0o600); + assert.equal(value.calls.length, 2); + assert.equal(value.calls[0].args.includes('get'), true); + assert.equal(value.calls[1].args.includes('--dry-run=server'), true); + assert.equal( + value.calls[1].args.includes(`--field-manager=${FIELD_MANAGER}`), + true, + ); + assert.equal(value.calls[1].input, manifest()); +}); + +test('preflight rejects lock, manifest and annotation drift before network access', (t) => { + const cases = [ + (target) => fs.appendFileSync(target.lockedManifest, '\n'), + (target) => { + const drifted = { ...target.report, lockDigest: digest('forged') }; + fs.writeFileSync(target.lockReportPath, canonicalJson(drifted)); + }, + (target) => { + const driftedManifest = manifest().replace( + CATALOG_REPORT_DIGEST, + digest('different-catalog-report'), + ); + fs.writeFileSync(target.lockedManifest, driftedManifest); + const report = lockReport(driftedManifest); + fs.writeFileSync(target.lockReportPath, canonicalJson(report)); + target.report = report; + }, + (target) => { + const implicitNamespaceManifest = manifest().replace( + '"namespace":"qinglong-system",', + '', + ); + fs.writeFileSync(target.lockedManifest, implicitNamespaceManifest); + const report = lockReport(implicitNamespaceManifest); + fs.writeFileSync(target.lockReportPath, canonicalJson(report)); + target.report = report; + }, + ]; + for (const [index, mutate] of cases.entries()) { + const fresh = fixture(t); + mutate(fresh); + const output = path.join(fresh.directory, `rejected-${index}.json`); + const request = commonRequest(fresh); + request.lockedManifest.expectedDigest = fresh.report.manifest.outputDigest; + request.lockReport.expectedDigest = fresh.report.lockDigest; + const command = writeCommand( + fresh.directory, + `rejected-command-${index}.json`, + 'cluster.deployment.preflight', + { + preflightId: `123e4567-e89b-42d3-a456-42661417400${index + 2}`, + ...request, + output, + }, + ); + let calls = 0; + assert.throws( + () => + executeCommand(command, { + runProcess() { + calls += 1; + return { status: 0, stdout: CLUSTER_UID, stderr: '' }; + }, + }), + /deployment ceremony failed/, + ); + assert.equal(calls, 0); + assert.equal(fs.existsSync(output), false); + } +}); + +test('preflight rejects the wrong cluster and rejected server dry-run', (t) => { + for (const failure of ['identity', 'dry-run']) { + const value = fixture(t); + const output = path.join(value.directory, `${failure}.json`); + const command = writeCommand( + value.directory, + `${failure}-command.json`, + 'cluster.deployment.preflight', + { + preflightId: + failure === 'identity' + ? '123e4567-e89b-42d3-a456-426614174010' + : '123e4567-e89b-42d3-a456-426614174011', + ...commonRequest(value), + output, + }, + ); + assert.throws(() => + executeCommand(command, { + runProcess(_executable, args) { + if (args.includes('get')) { + return { + status: 0, + stdout: + failure === 'identity' ? crypto.randomUUID() : CLUSTER_UID, + stderr: '', + }; + } + return { status: 1, stdout: '', stderr: 'redacted admission error' }; + }, + }), + ); + assert.equal(fs.existsSync(output), false); + } +}); + +test('kubeconfig executable authentication and weak private files fail closed', (t) => { + for (const mode of ['exec', 'public']) { + const value = fixture(t); + if (mode === 'exec') { + fs.writeFileSync( + value.kubeconfig, + fs + .readFileSync(value.kubeconfig, 'utf8') + .replace( + 'token: bounded-test-token', + 'exec:\n command: owned', + ), + ); + } else { + fs.chmodSync(value.kubeconfig, 0o644); + } + const output = path.join(value.directory, `${mode}.json`); + const command = writeCommand( + value.directory, + `${mode}-command.json`, + 'cluster.deployment.preflight', + { + preflightId: + mode === 'exec' + ? '123e4567-e89b-42d3-a456-426614174020' + : '123e4567-e89b-42d3-a456-426614174021', + ...commonRequest(value), + output, + }, + ); + let calls = 0; + assert.throws(() => + executeCommand(command, { + runProcess() { + calls += 1; + return { status: 0, stdout: CLUSTER_UID, stderr: '' }; + }, + }), + ); + assert.equal(calls, 0); + assert.equal(fs.existsSync(output), false); + } +}); + +test('apply revalidates preflight, mutates explicitly and proves convergence', (t) => { + const value = prepare(t); + const receiptPath = path.join(value.directory, 'receipt.json'); + const applyCommand = writeCommand( + value.directory, + 'apply-command.json', + 'cluster.deployment.apply', + { + mutationId: '123e4567-e89b-42d3-a456-426614174030', + preflight: { + path: value.preflightPath, + expectedDigest: value.preflight.preflightDigest, + }, + ...commonRequest(value), + output: receiptPath, + }, + ); + const calls = []; + const receipt = executeCommand(applyCommand, { + runProcess: successfulRunner(calls), + }); + assert.equal(receipt.schema, RECEIPT_SCHEMA); + assert.equal(receipt.preflightDigest, value.preflight.preflightDigest); + assert.equal(receipt.verification.kubernetesMutation, true); + assert.equal(receipt.verification.crossResourceAtomicity, false); + assert.equal( + receipt.verification.recovery, + 'reapply_exact_lock_with_same_field_manager', + ); + assert.deepEqual( + receipt.steps.map(({ name }) => name), + [ + 'cluster_identity_before', + 'server_side_dry_run', + 'server_side_apply', + 'server_side_convergence_read', + 'cluster_identity_after', + ], + ); + assert.equal(calls.length, 5); + assert.equal(fs.statSync(receiptPath).mode & 0o777, 0o600); + + const auditCommand = writeCommand( + value.directory, + 'audit-command.json', + 'cluster.deployment.receipt.audit', + { + applyCommand: { + path: applyCommand, + expectedDigest: fileDigest(applyCommand), + }, + receipt: { path: receiptPath, expectedDigest: receipt.receiptDigest }, + }, + ); + assert.deepEqual(executeCommand(auditCommand), { + compatible: true, + deploymentFamily: 'cluster', + mutationId: receipt.mutationId, + receiptDigest: receipt.receiptDigest, + preflightDigest: receipt.preflightDigest, + lockDigest: value.report.lockDigest, + manifestDigest: value.report.manifest.outputDigest, + clusterUid: CLUSTER_UID, + externalResultsReplayed: false, + kubernetesMutation: false, + }); + + let replayCalls = 0; + assert.equal( + executeCommand(applyCommand, { + runProcess() { + replayCalls += 1; + throw new Error('must not run'); + }, + }).receiptDigest, + receipt.receiptDigest, + ); + assert.equal(replayCalls, 0); +}); + +test('apply failure or post-apply drift never publishes a success receipt', (t) => { + for (const failure of ['apply', 'convergence', 'identity-after']) { + const value = prepare(t); + const receiptPath = path.join(value.directory, `${failure}-receipt.json`); + const applyCommand = writeCommand( + value.directory, + `${failure}-apply-command.json`, + 'cluster.deployment.apply', + { + mutationId: + failure === 'apply' + ? '123e4567-e89b-42d3-a456-426614174040' + : failure === 'convergence' + ? '123e4567-e89b-42d3-a456-426614174041' + : '123e4567-e89b-42d3-a456-426614174042', + preflight: { + path: value.preflightPath, + expectedDigest: value.preflight.preflightDigest, + }, + ...commonRequest(value), + output: receiptPath, + }, + ); + let identityCount = 0; + assert.throws(() => + executeCommand(applyCommand, { + runProcess(_executable, args) { + if (args.includes('get') && args.includes('-f=-')) { + return { + status: failure === 'convergence' ? 1 : 0, + stdout: failure === 'convergence' ? '' : convergenceList(), + stderr: + failure === 'convergence' ? 'redacted convergence error' : '', + }; + } + if (args.includes('get')) { + identityCount += 1; + return { + status: 0, + stdout: + failure === 'identity-after' && identityCount === 2 + ? crypto.randomUUID() + : CLUSTER_UID, + stderr: '', + }; + } + if (args.includes('--dry-run=server')) { + return { status: 0, stdout: 'dry-run', stderr: '' }; + } + return { + status: failure === 'apply' ? 1 : 0, + stdout: '', + stderr: failure === 'apply' ? 'redacted apply error' : '', + }; + }, + }), + ); + assert.equal(fs.existsSync(receiptPath), false); + } +}); + +test('receipt audit rejects a different command or recomputed receipt', (t) => { + const value = prepare(t); + const receiptPath = path.join(value.directory, 'receipt.json'); + const applyCommand = writeCommand( + value.directory, + 'apply-command.json', + 'cluster.deployment.apply', + { + mutationId: '123e4567-e89b-42d3-a456-426614174050', + preflight: { + path: value.preflightPath, + expectedDigest: value.preflight.preflightDigest, + }, + ...commonRequest(value), + output: receiptPath, + }, + ); + const receipt = executeCommand(applyCommand, { + runProcess: successfulRunner([]), + }); + const forged = { ...receipt, mutationId: crypto.randomUUID() }; + const { receiptDigest: ignored, ...unsigned } = forged; + forged.receiptDigest = digest(JSON.stringify(unsigned)); + fs.writeFileSync(receiptPath, canonicalJson(forged)); + const auditCommand = writeCommand( + value.directory, + 'audit-command.json', + 'cluster.deployment.receipt.audit', + { + applyCommand: { + path: applyCommand, + expectedDigest: fileDigest(applyCommand), + }, + receipt: { path: receiptPath, expectedDigest: receipt.receiptDigest }, + }, + ); + assert.throws(() => executeCommand(auditCommand)); +}); + +test('command surface is closed and lock reports require exact canonical identity', () => { + assert.equal( + commandFile(['--command-file=/private/command.json']), + '/private/command.json', + ); + assert.throws(() => commandFile([])); + assert.throws(() => commandFile(['--command-file=a', '--extra=b'])); + assert.throws(() => + parseCommand({ + schemaVersion: 1, + schema: COMMAND_SCHEMA, + operation: 'cluster.deployment.apply-now', + request: {}, + }), + ); + const report = lockReport(); + assert.equal(validateLockReport(report).report.lockDigest, report.lockDigest); + const drifted = JSON.parse(JSON.stringify(report)); + drifted.catalog.discoveryTagAuthority = 'fallback'; + const { lockDigest: ignored, ...unsigned } = drifted; + drifted.lockDigest = digest(JSON.stringify(unsigned)); + assert.throws(() => validateLockReport(drifted)); +}); + +test('repository exposes the reviewed ceremony and removes the bare apply handoff', () => { + const root = path.resolve(__dirname, '../..'); + const packageManifest = JSON.parse( + fs.readFileSync(path.join(root, 'package.json'), 'utf8'), + ); + assert.equal( + packageManifest.scripts['cluster-deployment:ql3'], + 'node scripts/ql3-kubernetes-deployment-ceremony.cjs', + ); + assert.equal( + packageManifest.scripts['test:cluster-deployment-live:ql3'], + 'node scripts/ql3-kubernetes-deployment-live-contract.cjs', + ); + const operations = fs.readFileSync( + path.join(root, 'docs/operations/ql3-release-set-deployment.md'), + 'utf8', + ); + assert.match(operations, /cluster\.deployment\.preflight/); + assert.match(operations, /cluster\.deployment\.apply/); + assert.match(operations, /cluster\.deployment\.receipt\.audit/); + assert.match(operations, /qinglong3-catalog-lock/); + assert.equal( + operations.includes('才由有权限的独立步骤执行 `kubectl apply'), + false, + ); +}); + +test('thin CLI uses the pinned executable and keeps failures low-sensitive', (t) => { + const value = fixture(t); + fs.writeFileSync( + value.kubectl, + `#!${process.execPath} +'use strict'; +const args = process.argv.slice(2); +if (args.includes('get') && args.includes('-f=-')) process.stdout.write(${JSON.stringify( + convergenceList(), + )}); +else if (args.includes('get')) process.stdout.write(${JSON.stringify( + CLUSTER_UID, + )}); +else if (args.includes('--dry-run=server')) process.stdout.write('deployment.apps/ql3-control\\n'); +else process.exitCode = 91; +`, + { mode: 0o700 }, + ); + const output = path.join(value.directory, 'cli-preflight.json'); + const command = writeCommand( + value.directory, + 'cli-command.json', + 'cluster.deployment.preflight', + { + preflightId: '123e4567-e89b-42d3-a456-426614174060', + ...commonRequest(value), + output, + }, + ); + const cli = path.resolve( + __dirname, + '../../scripts/ql3-kubernetes-deployment-ceremony.cjs', + ); + const accepted = spawnSync( + process.execPath, + [cli, `--command-file=${command}`], + { encoding: 'utf8' }, + ); + assert.equal( + accepted.status, + 0, + JSON.stringify({ stdout: accepted.stdout, stderr: accepted.stderr }), + ); + assert.equal(accepted.stderr, ''); + assert.equal(JSON.parse(accepted.stdout).schema, PREFLIGHT_SCHEMA); + + fs.chmodSync(value.kubeconfig, 0o644); + const rejected = spawnSync( + process.execPath, + [cli, `--command-file=${command}`], + { encoding: 'utf8' }, + ); + assert.equal(rejected.status, 1); + assert.equal(rejected.stdout, ''); + assert.deepEqual(JSON.parse(rejected.stderr), { + schemaVersion: 1, + component: 'qinglong3-kubernetes-deployment-ceremony', + code: 'QL3_KUBERNETES_DEPLOYMENT_CEREMONY_FAILED', + message: 'QingLong 3 Kubernetes deployment ceremony failed', + }); + assert.equal(rejected.stderr.includes(value.directory), false); + assert.equal(rejected.stderr.includes(CONTEXT), false); +});