From 2e848ef7c0dd127139fbcb960e6409573a0c7a5a Mon Sep 17 00:00:00 2001 From: whyour Date: Mon, 17 Aug 2026 01:41:10 +0800 Subject: [PATCH] feat(ql3): fence cluster deployment transitions --- docs/QINGLONG_3_0_ARCHITECTURE_RFC.md | 23 + ...ntory-closed-kubernetes-deployment-head.md | 80 ++ docs/adr/README.md | 1 + docs/operations/ql3-release-set-deployment.md | 47 +- .../src/deployment/foundation/contract.ts | 12 +- .../src/deployment/foundation/error.ts | 8 + .../service-manager/serviceBridge.ts | 2 +- .../service-manager/serviceBridgeContract.ts | 2 +- .../service-manager/serviceBridgeFiles.ts | 2 +- .../service-manager/serviceCutoverConsumer.ts | 6 +- .../service-manager/serviceCutoverJournal.ts | 2 +- .../service-manager/serviceManagerIntent.ts | 6 +- .../service-manager/serviceOutcomeContract.ts | 2 +- .../ql3-kubernetes-deployment-ceremony.cjs | 790 ++++++++++++++++-- ...l3-kubernetes-deployment-live-contract.cjs | 43 +- scripts/ql3-service-bridge-import-audit.cjs | 2 + .../ql3KubernetesDeploymentCeremony.test.cjs | 434 +++++++++- test/back/ql3PackageBoundaryAudit.test.cjs | 4 +- 18 files changed, 1337 insertions(+), 129 deletions(-) create mode 100644 docs/adr/ADR-0434-resource-inventory-closed-kubernetes-deployment-head.md create mode 100644 packages/ql3-local-owner-cli/src/deployment/foundation/error.ts diff --git a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md index 883d0768..87c30f7a 100644 --- a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md +++ b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md @@ -11,6 +11,29 @@ 最新增量证据(2026-08-16): +- D-342/ADR-0434(已接受;资源退休 ceremony 与真实公开 catalog 运行仍待后续 Gate):Cluster deployment command、preflight 与 + receipt 在尚未发布前直接升为 v2,并以目标 namespace 中唯一 `qinglong3-deployment-head` ConfigMap 建立部署顺序 authority。每次 + `install|upgrade|rollback` 都必须绑定 exact expected Head generation/deployment/lock/state digest;preflight 读取当前 Head,apply + 再用 API Server 返回的 opaque `resourceVersion` 执行 create/replace CAS,先取得 `applying`,完成五步 server-side apply/convergence + 后再提交 `committed`。相同意图可从 applying 或已提交但丢失本地 receipt 的窗口确定性恢复,陈旧 preflight、并发不同意图、Head + 漂移与本地 receipt 对已前进 Head 的重放全部失败关闭。locked manifest 必须完整覆盖 `control|control-ai|admin|worker`,自动推导并 + 排序唯一 resource inventory;upgrade 不能省略现存对象,rollback 必须与上一 deployment 使用相同 inventory,避免把普通 apply 不会 + 删除的残留对象冒充成功。删除仍保留给后续绑定 UID/resourceVersion 的独立 retirement ceremony,不采用 alpha prune。Head 只增加一个 + 小 ConfigMap 和 preflight 一次 GET、apply 两次 CAS;没有 controller、CRD、webhook、Pod、listener、timer、数据库、migration、Pool、 + 新 package 或第三方依赖,Local/Edge/Standalone 零导入、零制品增量。Service Bridge 审计同时发现其仅为错误类型而加载完整 foundation + contract;已把同一 class identity 下沉到既有 `local-owner-cli/deployment/foundation/error` 内部模块,各 importer 改用叶子入口,未拆出 + 浅 package。定向 deployment ceremony 14/14、Local Owner 171 项为 166 pass/5 条件 skip、完整 backend 1,331 项为 1,329 pass/2 条件 + skip/0 fail、18-package clean build/test 退出 0。10 项架构/部署审计与 14 档 Local artifact 全部 compatible,package boundary 仍为 + 18 packages、`singleSourcePackages=[]`、`shallowSourcePackages=[]`;Local Owner 的 107 个源码中 106 个位于领域目录。默认 + Edge/Standalone 保持 2,589,890/2,589,968 bytes,MCP 保持 7,315,930/7,316,038 bytes;Cluster Admin dry-run pack 保持 250 files、 + 271,238-byte tarball、1,690,196-byte unpacked。隔离三节点 K3s `v1.34.3+k3s1`/arm64 真实运行 6 个 release resource、4 个零副本 + Deployment 和 generation 1 committed Head,deployment/preflight/receipt digest 分别为 + `sha256:f911cda19195735d00fcc6317db2c22794855e5eb5795696b19a4239fb68a263`、 + `sha256:574f80dde330c5b4fe41c681906a67ffd6e4b608dedc6d955574771b2bd5c711`、 + `sha256:0f2edd2037a4df7ae6eb28616aa38a280498660443d0f091e1fac5c1ff532258`,资源清理完成。经允许重跑 PostgreSQL 18.6 + arm64 physical HA,142/142、timeline `1→2`,报告 SHA-256 + `8ba0b130100b48e19c602546b63844903e567d6997630f8c950b51cf41ee70ea`,独立证据审计通过且容器、卷、网络零残留。live 使用 synthetic + lock,只证明部署状态机和目标 API CAS,不冒充尚未产生的公开 GHCR catalog。 - D-341/ADR-0433(已接受;真实公开 catalog 运行待实际 release tag):Cluster 最后一跳不再交给裸 `kubectl apply -f locked.yaml`。新增工作站级 `cluster.deployment.preflight|apply|receipt.audit`,从 owner-private canonical command 出发独立复验 Kubernetes v2 lock/report、locked manifest、release-set/catalog/workflow/image/annotation 闭包,并固定 diff --git a/docs/adr/ADR-0434-resource-inventory-closed-kubernetes-deployment-head.md b/docs/adr/ADR-0434-resource-inventory-closed-kubernetes-deployment-head.md new file mode 100644 index 00000000..a296fe37 --- /dev/null +++ b/docs/adr/ADR-0434-resource-inventory-closed-kubernetes-deployment-head.md @@ -0,0 +1,80 @@ +# ADR-0434:Resource-inventory-closed Kubernetes Deployment Head + +- 状态:Accepted +- 日期:2026-08-16 +- 关联 RFC:QL-RFC-0001 D-03、D-14、D-341、D-342 +- 关联 ADR:ADR-0431、ADR-0433 +- 修正:ADR-0433 第 7、8 项的无目标状态重放与人工 roll-forward 边界 + +## 上下文 + +ADR-0433 把 catalog-bound lock、目标 cluster 与 pinned kubectl 绑定到最后一跳,但固定 field manager 的 server-side apply 只解决字段 +ownership,不提供部署意图的先后顺序 CAS。两个运维者从同一旧状态取得 preflight 后,后执行者仍可能覆盖先执行者;manifest 中移除的 +对象也不会被普通 apply 删除,若只检查新 manifest,旧对象可残留而 receipt 仍显示成功。 + +Kubernetes API 的 server-side apply 文档明确说明,它不适合依赖 current value 的条件更新;这种 lost-update 防护应使用包含当前 +`resourceVersion` 的 update/replace。`kubectl apply --prune` 的 allowlist 仍标记为 alpha,也不能替代显式资源退休协议。因此部署 +ceremony 需要一个很小的目标侧 durable head,而不是把并发或删除语义藏进 field ownership。 + +## 决策 + +1. 尚未发布的 deployment command/preflight/receipt 直接升级为 v2,不保留可绕过 Head 的 v1 双轨。每个 preflight/apply 必须显式提供 + `transitionKind=install|upgrade|rollback` 和 exact `expectedHead={generation,deploymentDigest,lockDigest,stateDigest}`。 +2. Cluster 生产转换只接受 `control,control-ai,admin,worker` 完整 role surface。locked manifest 推导唯一 control namespace,并生成按 + `apiVersion/kind/namespace/name` 排序、去重的完整 resource inventory;调用方不能另选 Head namespace 或提交手写 inventory。 +3. 目标 namespace 中固定使用一个 `qinglong3-deployment-head` ConfigMap,唯一数据键为 `head.json`。它保存 schema、phase、generation、 + exact transition、当前/前一 lock 与 inventory 摘要、target/tool authority、五项 workload step transcript digest、deployment digest 和 + self digest;不保存 manifest 正文、token、credential、kubeconfig 或命令输出正文。 +4. preflight 在 identity read 后读取 Head,要求它为空或 `committed`,并比较 exact expected Head。第一次 `install` 只接受空 Head; + `upgrade` 只接受严格递增 SemVer;`rollback` 只接受 Head 中精确的上一 deployment。 +5. apply 重读 Head 后,以 ConfigMap create 或带 GET 所得 opaque `resourceVersion` 的 replace 原子取得 `phase=applying`。相同 + command/mutation/preflight 可以恢复同一 applying intent;任何其他意图、陈旧 preflight、resourceVersion conflict 或 Head 漂移都在 + workload mutation 前失败关闭。 +6. server-side apply、convergence read 与末次 cluster identity 全部成功后,ceremony 再以 applying ConfigMap 的 resourceVersion replace + 为 `phase=committed`,然后才 no-replace 发布 receipt。若 committed response 已返回而本地 receipt 丢失,相同 command 可从 Head + 确定性重建;已有本地 receipt 的重放也必须联网确认 Head 尚未前进。 +7. `upgrade` 要求 active inventory 是 target inventory 的子集;省略现存对象立即失败,不能隐式 prune。`rollback` 额外要求 current、 + target 与 previous inventory 完全相同。需要删除对象时等待独立的 UID/resourceVersion delete-precondition retirement ceremony;本 + ADR 不把“对象仍残留”冒充成功回退。 +8. receipt 继续声明 Kubernetes 多资源 mutation 非事务,但新增 Head generation/deployment/state digest、完整 inventory、 + `deploymentHeadCas=true`、`resourceInventoryClosed=true` 与 + `recovery=resume_exact_transition_from_target_head`。offline receipt audit 不访问 API Server,也不宣称 Head 仍是当前状态。 + +## 部署与资源影响 + +- Local/Edge/Standalone 路径零导入、零制品增量;路由设备不创建 Head、不加载 Kubernetes/YAML/semver 或 Cluster role 代码。 +- Cluster 每个目标 namespace 只增加一个小型 ConfigMap;无 controller、CRD、webhook、ServiceAccount、Pod、listener、timer、watcher、 + 数据库、migration、SQL 或 Pool。preflight 比 ADR-0433 多一次 Head GET;首次 apply 多一次 create,后续 apply 为 acquire replace 和 + commit replace。 +- 逻辑继续内聚在根级工作站 ceremony,不新增浅 workspace package。Head 是发布排序证据,不是通用 desired-state controller。 +- namespace 必须先由受控 bootstrap 创建;否则 Head create 与 namespaced workload dry-run 都自然失败关闭。 + +## 被拒绝的替代方案 + +### 仅依赖相同 server-side apply field manager + +拒绝。field manager 管理字段 ownership,不比较调用者观察到的上一部署 generation,陈旧 writer 仍可成为最后写入者。 + +### 使用 `kubectl apply --prune` + +拒绝。当前 prune allowlist/ApplySet 仍有 alpha 边界,而且删除必须绑定已观察对象 UID/resourceVersion 与独立 receipt;隐式集合删除无法满足 +这一证明要求。 + +### 在 PostgreSQL 或新 controller 中保存部署锁 + +拒绝。最后一跳的并发事实应与目标 API Server 同故障域;额外数据库或常驻 reconciler 会扩大低资源、凭据、升级与可用性边界。 + +## 验证 + +- 定向 Node 契约 14/14,通过空 Head install、create/replace CAS、committed receipt、同意图 applying 恢复、陈旧 preflight 阻断、 + resource omission fail-closed、错误/漂移不发布 receipt、closed command 与低敏 CLI; +- 隔离三节点 K3s `v1.34.3+k3s1`/Linux arm64 真实运行 6 个 release resource、4 个零副本 Deployment 和一个固定 Head ConfigMap; + Head 从 absent 经 applying 到 generation 1 committed,deployment digest 为 + `sha256:f911cda19195735d00fcc6317db2c22794855e5eb5795696b19a4239fb68a263`,resource inventory 为 6,receipt audit 通过,临时 + container/network 清理完成; +- 完整 backend、18-package、边界、制品与 PostgreSQL HA 结果记录在 QL-RFC-0001 D-342。 + +## 规范依据 + +- [Kubernetes API concepts:resourceVersion 与 conditional update](https://kubernetes.io/docs/reference/using-api/api-concepts/) +- [Kubernetes declarative object management:prune/ApplySet 状态](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/) diff --git a/docs/adr/README.md b/docs/adr/README.md index b74fe220..8e7447c0 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -437,6 +437,7 @@ | [ADR-0431](./ADR-0431-catalog-bound-deployment-lock-chain.md) | Catalog-bound Deployment Lock 证据链 | Accepted(真实公开 catalog 运行待实际 release tag) | | [ADR-0432](./ADR-0432-target-side-catalog-bound-local-compose-revisions.md) | 目标侧 Catalog-bound Local Compose 修订 | Accepted | | [ADR-0433](./ADR-0433-catalog-bound-kubernetes-target-deployment-ceremony.md) | Catalog-bound Kubernetes 目标部署 Ceremony | Accepted | +| [ADR-0434](./ADR-0434-resource-inventory-closed-kubernetes-deployment-head.md) | Resource-inventory-closed Kubernetes Deployment Head | Accepted | ## 规则 diff --git a/docs/operations/ql3-release-set-deployment.md b/docs/operations/ql3-release-set-deployment.md index 6b95ab37..8deb1ee6 100644 --- a/docs/operations/ql3-release-set-deployment.md +++ b/docs/operations/ql3-release-set-deployment.md @@ -194,13 +194,16 @@ kubectl --kubeconfig="${kubeconfig}" --context="${context}" \ ceremony 目录必须为当前 UID 的 canonical `0700` 目录;command、locked manifest、lock report、kubeconfig、preflight 与 receipt 都必须是单链接 `0600` 文件。kubectl 使用 realpath 后的 absolute executable,并记录其 SHA-256。kubeconfig 禁止 `exec` 与 `auth-provider`。每个 kubectl 调用使用独立临时 HOME/XDG cache/TMPDIR,结束即清理,不读取 ambient HOME 或在当前目录创建 -`.kube/cache`。以下是 preflight command 的逻辑结构;实际文件必须用 `JSON.stringify(value) + "\n"` 写成单行 canonical JSON, -并以 `0600` no-replace 创建: +`.kube/cache`。目标 Namespace 还承载固定名称 `qinglong3-deployment-head` 的小型 ConfigMap。它不属于应用 manifest,也不由 +server-side apply 接管;ceremony 只通过 API Server 返回的 opaque `resourceVersion` 执行 create/replace CAS。第一次安装使用 +`install` 和空 Head;后续 `upgrade`/`rollback` 必须从上一份 receipt 的 `deploymentHead` 复制 generation、deployment/lock/state +digest,不能自行推测。以下是第一次安装的 preflight command 逻辑结构;实际文件必须用 `JSON.stringify(value) + "\n"` 写成单行 +canonical JSON,并以 `0600` no-replace 创建: ```json { "schemaVersion": 1, - "schema": "qinglong/kubernetes-deployment-command@v1", + "schema": "qinglong/kubernetes-deployment-command@v2", "operation": "cluster.deployment.preflight", "request": { "preflightId": "", @@ -222,6 +225,13 @@ ceremony 目录必须为当前 UID 的 canonical `0700` 目录;command、locke }, "context": "", "expectedClusterUid": "", + "transitionKind": "install", + "expectedHead": { + "generation": 0, + "deploymentDigest": null, + "lockDigest": null, + "stateDigest": null + }, "output": "" } } @@ -233,13 +243,14 @@ ceremony 目录必须为当前 UID 的 canonical `0700` 目录;command、locke pnpm cluster-deployment:ql3 -- --command-file="${preflight_command}" ``` -成功后,人工核对返回的 `preflightDigest`、lock/catalog digest、cluster UID 和 -`kubernetesMutation:false`。apply 必须使用新的 mutation UUID,并精确复用所有 target/input authority: +成功后,人工核对返回的 `preflightDigest`、lock/catalog digest、cluster UID、完整 resource inventory、`deploymentHead` 和 +`kubernetesMutation:false`。apply 必须使用新的 mutation UUID,并精确复用 transition、expected Head 与所有 target/input +authority: ```json { "schemaVersion": 1, - "schema": "qinglong/kubernetes-deployment-command@v1", + "schema": "qinglong/kubernetes-deployment-command@v2", "operation": "cluster.deployment.apply", "request": { "mutationId": "", @@ -265,22 +276,36 @@ pnpm cluster-deployment:ql3 -- --command-file="${preflight_command}" }, "context": "", "expectedClusterUid": "", + "transitionKind": "install", + "expectedHead": { + "generation": 0, + "deploymentDigest": null, + "lockDigest": null, + "stateDigest": null + }, "output": "" } } ``` -同样通过 `cluster-deployment:ql3` 执行。apply 会重新完成 lock 检查、cluster identity 和 server-side dry-run,然后以固定 -`qinglong3-catalog-lock` field manager 执行 server-side apply,不使用 `--force-conflicts`;最后读取 live objects,验证 +同样通过 `cluster-deployment:ql3` 执行。apply 先重读 Head 并以 create/resourceVersion replace 把唯一意图置为 `applying`,再重新 +完成 lock 检查、cluster identity 和 server-side dry-run,然后以固定 `qinglong3-catalog-lock` field manager 执行 server-side +apply,不使用 `--force-conflicts`;最后读取 live objects,验证 UID/resourceVersion、完整期望字段、四类 immutable image/catalog annotations 和受影响资源的 managed-field ownership,再次确认 -cluster UID 后才发布 receipt。 +cluster UID,最后以同一 ConfigMap 的新 resourceVersion 提交 `committed` Head 后才发布 receipt。Head 保存当前与前一 deployment +摘要、完整有序 resource inventory、五个 workload step digest 和 self digest;它不保存 credential、token 或 manifest 正文。 + +`upgrade` 只接受严格递增 SemVer,且 active inventory 必须是目标 inventory 的子集;遗漏对象不会借助隐式 prune 删除,而是在 +mutation 前失败关闭。`rollback` 只接受 Head 中精确的上一部署 lock,且当前/目标 inventory 必须完全相同;涉及资源退休时须等待 +独立的 UID/resourceVersion-precondition retirement ceremony,不能把遗留对象伪装成回滚成功。 离线审计使用 `cluster.deployment.receipt.audit` command,其中 `applyCommand.expectedDigest` 是 apply command 文件完整字节的 SHA-256,`receipt.expectedDigest` 是 receipt 内的 `receiptDigest`。审计不会访问 Kubernetes API,结果必须保持 `externalResultsReplayed:false`、`kubernetesMutation:false`。 -多资源 apply 不是事务,也不提供自动删除式 rollback。失败或 receipt 响应丢失时保留原文件:同一 command/field manager 可安全 -重放并重新做 live convergence;需要回退时,以上一份 catalog-bound lock 创建新的显式 preflight/apply,而不是修改旧 receipt。 +多资源 apply 不是事务,也不提供自动删除式 rollback。失败或 receipt 响应丢失时保留原文件:只有相同 command/mutation/preflight +意图能从 `applying` Head 重做 live convergence;不同意图和陈旧 preflight 都失败关闭。若 Head 已 `committed` 但本地 receipt 丢失, +同一 command 可从目标 Head 确定性重建 receipt。已有本地 receipt 的重放也会联网确认 Head 尚未前进,不能离线冒充当前部署。 ## 准入检查 diff --git a/packages/ql3-local-owner-cli/src/deployment/foundation/contract.ts b/packages/ql3-local-owner-cli/src/deployment/foundation/contract.ts index 0038f7fa..fb19ff42 100644 --- a/packages/ql3-local-owner-cli/src/deployment/foundation/contract.ts +++ b/packages/ql3-local-owner-cli/src/deployment/foundation/contract.ts @@ -8,6 +8,9 @@ import { type LocalComposeReleaseSelectionInput, type ResolvedLocalComposeReleaseSelection, } from '../compose/releaseSelection'; +import { LocalDeploymentConfigurationError } from './error'; + +export { LocalDeploymentConfigurationError } from './error'; const MAX_PATH_BYTES = 4_096; const SAFE_PATH_PATTERN = /^\/[A-Za-z0-9._/@-]+$/; @@ -364,15 +367,6 @@ export interface LocalDeploymentComposeEvidenceCollectionResult { }>; } -export class LocalDeploymentConfigurationError extends TypeError { - readonly code = 'QL3_LOCAL_DEPLOYMENT_CONFIGURATION_INVALID'; - - constructor(message: string, options?: ErrorOptions) { - super(`Local deployment configuration is invalid: ${message}`, options); - this.name = 'LocalDeploymentConfigurationError'; - } -} - function object(value: unknown, label: string): Record { if ( !value || diff --git a/packages/ql3-local-owner-cli/src/deployment/foundation/error.ts b/packages/ql3-local-owner-cli/src/deployment/foundation/error.ts new file mode 100644 index 00000000..d00719b5 --- /dev/null +++ b/packages/ql3-local-owner-cli/src/deployment/foundation/error.ts @@ -0,0 +1,8 @@ +export class LocalDeploymentConfigurationError extends TypeError { + readonly code = 'QL3_LOCAL_DEPLOYMENT_CONFIGURATION_INVALID'; + + constructor(message: string, options?: ErrorOptions) { + super(`Local deployment configuration is invalid: ${message}`, options); + this.name = 'LocalDeploymentConfigurationError'; + } +} diff --git a/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceBridge.ts b/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceBridge.ts index 7a346149..5b79925e 100644 --- a/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceBridge.ts +++ b/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceBridge.ts @@ -5,7 +5,7 @@ import { spawnSync } from 'node:child_process'; import { readPrivateLocalCommandFile } from '@qinglong/local-command-file'; -import { LocalDeploymentConfigurationError } from '../foundation/contract'; +import { LocalDeploymentConfigurationError } from '../foundation/error'; import { normalizeLocalServiceBridgeCommand, normalizeLocalServiceManagerIntent, diff --git a/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceBridgeContract.ts b/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceBridgeContract.ts index a43e2ed5..64c40fb4 100644 --- a/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceBridgeContract.ts +++ b/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceBridgeContract.ts @@ -1,7 +1,7 @@ import crypto from 'node:crypto'; import path from 'node:path'; -import { LocalDeploymentConfigurationError } from '../foundation/contract'; +import { LocalDeploymentConfigurationError } from '../foundation/error'; const MAX_PATH_BYTES = 4_096; const SAFE_PATH_PATTERN = /^\/[A-Za-z0-9._/@-]+$/; diff --git a/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceBridgeFiles.ts b/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceBridgeFiles.ts index 082d4f5b..e37e645c 100644 --- a/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceBridgeFiles.ts +++ b/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceBridgeFiles.ts @@ -1,7 +1,7 @@ import fs from 'node:fs'; import path from 'node:path'; -import { LocalDeploymentConfigurationError } from '../foundation/contract'; +import { LocalDeploymentConfigurationError } from '../foundation/error'; const MAX_FILE_BYTES = 1024 * 1024; diff --git a/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceCutoverConsumer.ts b/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceCutoverConsumer.ts index 04cf741b..19feccca 100644 --- a/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceCutoverConsumer.ts +++ b/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceCutoverConsumer.ts @@ -8,10 +8,8 @@ import { readPrivateLocalJsonFile, } from '@qinglong/local-command-file'; -import { - currentIdentity, - LocalDeploymentConfigurationError, -} from '../foundation/contract'; +import { currentIdentity } from '../foundation/contract'; +import { LocalDeploymentConfigurationError } from '../foundation/error'; import { advanceLocalCutoverInstanceHead, readLocalCutoverInstanceHead, diff --git a/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceCutoverJournal.ts b/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceCutoverJournal.ts index 7926d615..1147085e 100644 --- a/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceCutoverJournal.ts +++ b/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceCutoverJournal.ts @@ -3,7 +3,7 @@ import path from 'node:path'; import { readPrivateLocalCommandFile } from '@qinglong/local-command-file'; -import { LocalDeploymentConfigurationError } from '../foundation/contract'; +import { LocalDeploymentConfigurationError } from '../foundation/error'; import { preflightPublishedFile, publishExactFile } from '../foundation/files'; import { cutoverDigest } from '../cutover/targetEvidence'; import type { diff --git a/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceManagerIntent.ts b/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceManagerIntent.ts index 3bffd483..e1fc5203 100644 --- a/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceManagerIntent.ts +++ b/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceManagerIntent.ts @@ -7,10 +7,8 @@ import { readPrivateLocalJsonFile, } from '@qinglong/local-command-file'; -import { - currentIdentity, - LocalDeploymentConfigurationError, -} from '../foundation/contract'; +import { currentIdentity } from '../foundation/contract'; +import { LocalDeploymentConfigurationError } from '../foundation/error'; import { assertLocalCutoverTargetHead, localCutoverInstanceHeadPath, diff --git a/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceOutcomeContract.ts b/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceOutcomeContract.ts index 8887af77..139ada28 100644 --- a/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceOutcomeContract.ts +++ b/packages/ql3-local-owner-cli/src/deployment/service-manager/serviceOutcomeContract.ts @@ -1,6 +1,6 @@ import crypto from 'node:crypto'; -import { LocalDeploymentConfigurationError } from '../foundation/contract'; +import { LocalDeploymentConfigurationError } from '../foundation/error'; import type { LocalServiceManagerAction, LocalServiceManagerKind, diff --git a/scripts/lib/ql3-kubernetes-deployment-ceremony.cjs b/scripts/lib/ql3-kubernetes-deployment-ceremony.cjs index 2d8ec785..e0bb5d71 100644 --- a/scripts/lib/ql3-kubernetes-deployment-ceremony.cjs +++ b/scripts/lib/ql3-kubernetes-deployment-ceremony.cjs @@ -6,12 +6,16 @@ const os = require('node:os'); const path = require('node:path'); const { spawnSync } = require('node:child_process'); const yaml = require('js-yaml'); +const semver = require('semver'); -const COMMAND_SCHEMA = 'qinglong/kubernetes-deployment-command@v1'; +const COMMAND_SCHEMA = 'qinglong/kubernetes-deployment-command@v2'; const LOCK_SCHEMA = 'qinglong/kubernetes-deployment-lock@v2'; const CATALOG_SCHEMA = 'qinglong/release-catalog-consumption-ceremony@v1'; -const PREFLIGHT_SCHEMA = 'qinglong/kubernetes-deployment-preflight@v1'; -const RECEIPT_SCHEMA = 'qinglong/kubernetes-deployment-receipt@v1'; +const PREFLIGHT_SCHEMA = 'qinglong/kubernetes-deployment-preflight@v2'; +const RECEIPT_SCHEMA = 'qinglong/kubernetes-deployment-receipt@v2'; +const HEAD_SCHEMA = 'qinglong/kubernetes-deployment-head@v1'; +const HEAD_NAME = 'qinglong3-deployment-head'; +const HEAD_DATA_KEY = 'head.json'; const FIELD_MANAGER = 'qinglong3-catalog-lock'; const MAX_COMMAND_BYTES = 64 * 1024; const MAX_MANIFEST_BYTES = 8 * 1024 * 1024; @@ -31,6 +35,7 @@ const CONTEXT_PATTERN = /^[A-Za-z0-9](?:[A-Za-z0-9._-]{0,252})$/u; const REPOSITORY_PATTERN = /^[a-z0-9](?:[a-z0-9-]{0,37}[a-z0-9])?\/[a-z0-9._-]{1,100}$/u; const ROLE_ORDER = Object.freeze(['control', 'control-ai', 'admin', 'worker']); +const TRANSITION_KINDS = Object.freeze(['install', 'upgrade', 'rollback']); const IMAGE_NAMES = Object.freeze({ control: 'qinglong3-cluster-control', 'control-ai': 'qinglong3-cluster-control-ai', @@ -277,6 +282,44 @@ function writeNoReplace(target, value) { }); } +function expectedHead(value) { + if ( + !exactKeys(value, [ + 'generation', + 'deploymentDigest', + 'lockDigest', + 'stateDigest', + ]) || + !Number.isSafeInteger(value.generation) || + value.generation < 0 || + (value.generation === 0 && + (value.deploymentDigest !== null || + value.lockDigest !== null || + value.stateDigest !== null)) || + (value.generation > 0 && + (!DIGEST_PATTERN.test(value.deploymentDigest || '') || + !DIGEST_PATTERN.test(value.lockDigest || '') || + !DIGEST_PATTERN.test(value.stateDigest || ''))) + ) { + fail('expected deployment head is invalid'); + } + return Object.freeze({ ...value }); +} + +function transitionRequest(request) { + if (!TRANSITION_KINDS.includes(request.transitionKind)) { + fail('deployment transition kind is invalid'); + } + const head = expectedHead(request.expectedHead); + if ( + (request.transitionKind === 'install' && head.generation !== 0) || + (request.transitionKind !== 'install' && head.generation === 0) + ) { + fail('deployment transition head is inconsistent'); + } + return Object.freeze({ transitionKind: request.transitionKind, head }); +} + function parseCommand(value) { if ( !exactKeys(value, ['schemaVersion', 'schema', 'operation', 'request']) || @@ -296,6 +339,8 @@ function parseCommand(value) { 'kubeconfig', 'context', 'expectedClusterUid', + 'transitionKind', + 'expectedHead', 'output', ]) || !UUID_PATTERN.test(request.preflightId || '') || @@ -305,6 +350,7 @@ function parseCommand(value) { ) { fail('preflight command is invalid'); } + const transition = transitionRequest(request); return Object.freeze({ schemaVersion: 1, schema: COMMAND_SCHEMA, @@ -317,6 +363,8 @@ function parseCommand(value) { kubeconfig: artifact(request.kubeconfig, 'kubeconfig'), context: request.context, expectedClusterUid: request.expectedClusterUid, + transitionKind: transition.transitionKind, + expectedHead: transition.head, output: resolveCanonicalAbsolute(request.output, 'preflight output'), }), }); @@ -332,6 +380,8 @@ function parseCommand(value) { 'kubeconfig', 'context', 'expectedClusterUid', + 'transitionKind', + 'expectedHead', 'output', ]) || !UUID_PATTERN.test(request.mutationId || '') || @@ -341,6 +391,7 @@ function parseCommand(value) { ) { fail('apply command is invalid'); } + const transition = transitionRequest(request); return Object.freeze({ schemaVersion: 1, schema: COMMAND_SCHEMA, @@ -354,6 +405,8 @@ function parseCommand(value) { kubeconfig: artifact(request.kubeconfig, 'kubeconfig'), context: request.context, expectedClusterUid: request.expectedClusterUid, + transitionKind: transition.transitionKind, + expectedHead: transition.head, output: resolveCanonicalAbsolute( request.output, 'apply receipt output', @@ -809,6 +862,70 @@ function resourceIdentity(resource) { }`; } +function resourceInventory(resources) { + const inventory = resources + .map((resource) => { + const namespace = resource.metadata?.namespace ?? null; + const item = { + apiVersion: resource.apiVersion, + kind: resource.kind, + namespace, + name: resource.metadata?.name, + }; + if ( + Object.values(item).some( + (entry) => entry !== null && (typeof entry !== 'string' || !entry), + ) + ) { + fail('deployment resource inventory is invalid'); + } + return Object.freeze(item); + }) + .sort((left, right) => { + const leftIdentity = JSON.stringify(left); + const rightIdentity = JSON.stringify(right); + return leftIdentity < rightIdentity + ? -1 + : leftIdentity > rightIdentity + ? 1 + : 0; + }); + const identities = inventory.map((item) => JSON.stringify(item)); + if (new Set(identities).size !== identities.length) { + fail('deployment resource inventory is duplicated'); + } + return Object.freeze(inventory); +} + +function containsExactString(value, expected, depth = 0) { + if (depth > MAX_STRUCTURE_DEPTH) fail('locked manifest is too deep'); + if (typeof value === 'string') return value === expected; + if (value === null || typeof value !== 'object') return false; + if (Array.isArray(value)) { + return value.some((entry) => + containsExactString(entry, expected, depth + 1), + ); + } + return Object.values(value).some((entry) => + containsExactString(entry, expected, depth + 1), + ); +} + +function deploymentHeadNamespace(resources, authority) { + const namespaces = new Set( + resources + .filter((resource) => + containsExactString(resource, authority.references.control), + ) + .map((resource) => resource.metadata?.namespace) + .filter((namespace) => typeof namespace === 'string'), + ); + if (namespaces.size !== 1) { + fail('deployment control namespace authority is invalid'); + } + return [...namespaces][0]; +} + function containsQingLongAuthority(value, references) { if (typeof value === 'string') { const role = roleFromImage(value); @@ -950,10 +1067,31 @@ function inspectInputs(request) { fail('lock report digest changed'); } const manifestContents = utf8(manifest, 'locked manifest'); - inspectLockedManifest(manifestContents, authority); + const inspection = inspectLockedManifest(manifestContents, authority); if (manifest.digest !== authority.report.manifest.outputDigest) { fail('locked manifest and report digest differ'); } + if ( + JSON.stringify(authority.report.requiredImages) !== + JSON.stringify(ROLE_ORDER) + ) { + fail('cluster deployment requires the complete role surface'); + } + const headNamespace = deploymentHeadNamespace( + inspection.resources, + authority, + ); + if ( + inspection.resources.some( + (resource) => + resource.apiVersion === 'v1' && + resource.kind === 'ConfigMap' && + resource.metadata?.namespace === headNamespace && + resource.metadata?.name === HEAD_NAME, + ) + ) { + fail('locked manifest cannot own the deployment head'); + } const executable = verifyExpected( readStableFile(request.kubectl.path, 'kubectl', MAX_EXECUTABLE_BYTES, { privateFile: false, @@ -973,6 +1111,9 @@ function inspectInputs(request) { manifestContents, reportFile, authority, + resources: inspection.resources, + resourceInventory: resourceInventory(inspection.resources), + headNamespace, executable, kubeconfig, }); @@ -1005,14 +1146,21 @@ function defaultRunProcess(executable, args, input) { } } -function runStep(inputs, request, name, argumentsAfterTarget, runner) { +function runStep( + inputs, + request, + name, + argumentsAfterTarget, + runner, + input = inputs.manifestContents, +) { const args = [ `--kubeconfig=${inputs.kubeconfig.path}`, `--context=${request.context}`, '--request-timeout=60s', ...argumentsAfterTarget, ]; - const result = runner(inputs.executable.path, args, inputs.manifestContents); + const result = runner(inputs.executable.path, args, input); const stdout = typeof result.stdout === 'string' ? result.stdout : ''; const stderr = typeof result.stderr === 'string' ? result.stderr : ''; if ( @@ -1146,6 +1294,420 @@ function toolSummary(inputs) { }); } +function validateResourceInventory(value) { + if ( + !Array.isArray(value) || + value.length < 1 || + value.length > MAX_RESOURCE_COUNT + ) { + fail('deployment resource inventory is invalid'); + } + const serialized = []; + for (const item of value) { + if ( + !exactKeys(item, ['apiVersion', 'kind', 'namespace', 'name']) || + typeof item.apiVersion !== 'string' || + !item.apiVersion || + typeof item.kind !== 'string' || + !item.kind || + (item.namespace !== null && + (typeof item.namespace !== 'string' || !item.namespace)) || + typeof item.name !== 'string' || + !item.name + ) { + fail('deployment resource inventory is invalid'); + } + serialized.push(JSON.stringify(item)); + } + if ( + new Set(serialized).size !== serialized.length || + JSON.stringify([...serialized].sort()) !== JSON.stringify(serialized) + ) { + fail('deployment resource inventory ordering is invalid'); + } +} + +function deploymentPlan(inputs) { + return Object.freeze({ + lock: lockSummary(inputs), + resources: inputs.resourceInventory, + }); +} + +function validateDeploymentPlan(value) { + if (!exactKeys(value, ['lock', 'resources'])) { + fail('deployment plan is invalid'); + } + validateLockSummary(value.lock); + validateResourceInventory(value.resources); +} + +function validateDeploymentSummary(value) { + if (!exactKeys(value, ['deploymentDigest', 'lock', 'resources'])) { + fail('deployment summary is invalid'); + } + if (!DIGEST_PATTERN.test(value.deploymentDigest || '')) { + fail('deployment summary digest is invalid'); + } + validateLockSummary(value.lock); + validateResourceInventory(value.resources); +} + +function headSnapshot(namespace, state) { + if (state === null) { + return Object.freeze({ + namespace, + name: HEAD_NAME, + generation: 0, + deploymentDigest: null, + lockDigest: null, + stateDigest: null, + }); + } + if (state.phase !== 'committed') { + fail('deployment head is not committed'); + } + return Object.freeze({ + namespace, + name: HEAD_NAME, + generation: state.generation, + deploymentDigest: state.deployment.deploymentDigest, + lockDigest: state.deployment.lock.lockDigest, + stateDigest: state.stateDigest, + }); +} + +function validateHeadSnapshot(value) { + if ( + !exactKeys(value, [ + 'namespace', + 'name', + 'generation', + 'deploymentDigest', + 'lockDigest', + 'stateDigest', + ]) || + !CONTEXT_PATTERN.test(value.namespace || '') || + value.name !== HEAD_NAME + ) { + fail('deployment head snapshot is invalid'); + } + expectedHead({ + generation: value.generation, + deploymentDigest: value.deploymentDigest, + lockDigest: value.lockDigest, + stateDigest: value.stateDigest, + }); +} + +function expectedSnapshot(request, namespace) { + return Object.freeze({ + namespace, + name: HEAD_NAME, + ...request.expectedHead, + }); +} + +function validateTransition(current, request, planned) { + const expected = expectedSnapshot(request, current.namespace); + if (JSON.stringify(expected) !== JSON.stringify(current.snapshot)) { + fail('deployment head compare-and-swap authority changed'); + } + if (request.transitionKind === 'install') { + if (current.state !== null) + fail('deployment install requires an empty head'); + return; + } + if (current.state === null || current.state.phase !== 'committed') { + fail('deployment transition requires a committed head'); + } + const active = current.state.deployment; + const activeVersion = active.lock.release.version; + const plannedVersion = planned.lock.release.version; + if (request.transitionKind === 'upgrade') { + if (!semver.gt(plannedVersion, activeVersion)) { + fail('deployment upgrade version must increase'); + } + const plannedResources = new Set( + planned.resources.map((item) => JSON.stringify(item)), + ); + if ( + active.resources.some( + (item) => !plannedResources.has(JSON.stringify(item)), + ) + ) { + fail('deployment upgrade cannot retire resources implicitly'); + } + return; + } + if ( + current.state.previous === null || + !semver.lt(plannedVersion, activeVersion) || + JSON.stringify(planned.lock) !== + JSON.stringify(current.state.previous.lock) || + JSON.stringify(planned.resources) !== + JSON.stringify(current.state.previous.resources) || + JSON.stringify(planned.resources) !== JSON.stringify(active.resources) + ) { + fail('deployment rollback must restore the exact previous resource set'); + } +} + +function validateHeadTransition(value) { + if ( + !exactKeys(value, [ + 'kind', + 'mutationId', + 'commandDigest', + 'preflightDigest', + 'expectedHead', + ]) || + !TRANSITION_KINDS.includes(value.kind) || + !UUID_PATTERN.test(value.mutationId || '') || + !DIGEST_PATTERN.test(value.commandDigest || '') || + !DIGEST_PATTERN.test(value.preflightDigest || '') + ) { + fail('deployment head transition is invalid'); + } + validateHeadSnapshot(value.expectedHead); +} + +function deploymentEvidence(state) { + return Object.freeze({ + schemaVersion: 1, + schema: 'qinglong/kubernetes-deployment-evidence@v1', + generation: state.generation, + transition: state.transition, + previous: state.previous, + planned: state.planned, + target: state.target, + tool: state.tool, + steps: state.steps, + }); +} + +function validateHeadState(value) { + if ( + !exactKeys(value, [ + 'schemaVersion', + 'schema', + 'phase', + 'generation', + 'transition', + 'previous', + 'planned', + 'target', + 'tool', + 'steps', + 'deployment', + 'stateDigest', + ]) || + value.schemaVersion !== 1 || + value.schema !== HEAD_SCHEMA || + !['applying', 'committed'].includes(value.phase) || + !Number.isSafeInteger(value.generation) || + value.generation < 1 || + !DIGEST_PATTERN.test(value.stateDigest || '') || + !Array.isArray(value.steps) + ) { + fail('deployment head state is invalid'); + } + validateHeadTransition(value.transition); + if (value.previous !== null) validateDeploymentSummary(value.previous); + validateDeploymentPlan(value.planned); + validateTargetSummary(value.target); + validateToolSummary(value.tool); + for (const step of value.steps) validateStepRecord(step); + if (value.steps.some((step) => step.status !== 0)) { + fail('deployment head contains a failed step'); + } + if (value.phase === 'applying') { + if (value.steps.length !== 0 || value.deployment !== null) { + fail('applying deployment head is invalid'); + } + } else { + if ( + value.steps.length !== 5 || + JSON.stringify(value.steps.map((step) => step.name)) !== + JSON.stringify([ + 'cluster_identity_before', + 'server_side_dry_run', + 'server_side_apply', + 'server_side_convergence_read', + 'cluster_identity_after', + ]) || + value.deployment === null || + JSON.stringify(value.deployment.lock) !== + JSON.stringify(value.planned.lock) || + JSON.stringify(value.deployment.resources) !== + JSON.stringify(value.planned.resources) + ) { + fail('committed deployment head is invalid'); + } + validateDeploymentSummary(value.deployment); + const expectedDigest = sha256( + Buffer.from(JSON.stringify(deploymentEvidence(value)), 'utf8'), + ); + if (value.deployment.deploymentDigest !== expectedDigest) { + fail('deployment evidence digest is invalid'); + } + } + const { stateDigest, ...unsigned } = value; + if (stateDigest !== sha256(Buffer.from(JSON.stringify(unsigned), 'utf8'))) { + fail('deployment head self digest is invalid'); + } + return value; +} + +function headReadStep(inputs, request, runner) { + const step = runStep( + inputs, + request, + 'deployment_head_read', + [ + 'get', + 'configmap', + HEAD_NAME, + `--namespace=${inputs.headNamespace}`, + '--ignore-not-found=true', + '-o=json', + ], + runner, + '', + ); + if (step.status !== 0) fail('deployment head read failed'); + const { stdout: _stdout, ...record } = step; + if (!step.stdout.trim()) { + return Object.freeze({ + state: null, + resourceVersion: null, + namespace: inputs.headNamespace, + snapshot: headSnapshot(inputs.headNamespace, null), + record: Object.freeze(record), + }); + } + let configMap; + let state; + try { + configMap = JSON.parse(step.stdout); + state = JSON.parse(configMap?.data?.[HEAD_DATA_KEY]); + } catch { + fail('deployment head response is invalid'); + } + if ( + configMap?.apiVersion !== 'v1' || + configMap?.kind !== 'ConfigMap' || + configMap?.metadata?.name !== HEAD_NAME || + configMap?.metadata?.namespace !== inputs.headNamespace || + typeof configMap?.metadata?.resourceVersion !== 'string' || + !configMap.metadata.resourceVersion + ) { + fail('deployment head identity is invalid'); + } + validateHeadState(state); + return Object.freeze({ + state, + resourceVersion: configMap.metadata.resourceVersion, + namespace: inputs.headNamespace, + snapshot: + state.phase === 'committed' + ? headSnapshot(inputs.headNamespace, state) + : null, + record: Object.freeze(record), + }); +} + +function headConfigMap(inputs, state, resourceVersion) { + return JSON.stringify({ + apiVersion: 'v1', + kind: 'ConfigMap', + metadata: { + name: HEAD_NAME, + namespace: inputs.headNamespace, + labels: { 'app.kubernetes.io/managed-by': FIELD_MANAGER }, + ...(resourceVersion === null ? {} : { resourceVersion }), + }, + data: { [HEAD_DATA_KEY]: JSON.stringify(state) }, + }); +} + +function writeHead(inputs, request, runner, state, resourceVersion, name) { + const operation = resourceVersion === null ? 'create' : 'replace'; + const step = runStep( + inputs, + request, + name, + [operation, '--validate=strict', '-f=-', '-o=json'], + runner, + headConfigMap(inputs, state, resourceVersion), + ); + if (step.status !== 0) fail(`deployment head ${operation} was rejected`); + let written; + try { + written = JSON.parse(step.stdout); + } catch { + fail('deployment head write response is invalid'); + } + if ( + typeof written?.metadata?.resourceVersion !== 'string' || + !written.metadata.resourceVersion || + written?.data?.[HEAD_DATA_KEY] !== JSON.stringify(state) + ) { + fail('deployment head write did not converge'); + } + const { stdout, ...record } = step; + return Object.freeze({ + record: Object.freeze(record), + resourceVersion: written.metadata.resourceVersion, + }); +} + +function applyingHead(commandState, preflight, inputs, current) { + const unsigned = { + schemaVersion: 1, + schema: HEAD_SCHEMA, + phase: 'applying', + generation: current.state === null ? 1 : current.state.generation + 1, + transition: { + kind: commandState.command.request.transitionKind, + mutationId: commandState.command.request.mutationId, + commandDigest: commandState.commandDigest, + preflightDigest: preflight.report.preflightDigest, + expectedHead: current.snapshot, + }, + previous: + current.state === null + ? null + : Object.freeze({ ...current.state.deployment }), + planned: deploymentPlan(inputs), + target: targetSummary(inputs, commandState.command.request), + tool: toolSummary(inputs), + steps: [], + deployment: null, + }; + return Object.freeze({ + ...unsigned, + stateDigest: sha256(Buffer.from(JSON.stringify(unsigned), 'utf8')), + }); +} + +function committedHead(applying, steps) { + const evidenceState = { ...applying, phase: 'committed', steps }; + delete evidenceState.stateDigest; + const deployment = Object.freeze({ + deploymentDigest: sha256( + Buffer.from(JSON.stringify(deploymentEvidence(evidenceState)), 'utf8'), + ), + lock: applying.planned.lock, + resources: applying.planned.resources, + }); + const unsigned = { ...evidenceState, deployment }; + return Object.freeze({ + ...unsigned, + stateDigest: sha256(Buffer.from(JSON.stringify(unsigned), 'utf8')), + }); +} + function validatePreflight(value) { if ( !exactKeys(value, [ @@ -1153,7 +1715,10 @@ function validatePreflight(value) { 'schema', 'preflightId', 'commandDigest', + 'transitionKind', 'lock', + 'resourceInventory', + 'deploymentHead', 'target', 'tool', 'steps', @@ -1165,14 +1730,18 @@ function validatePreflight(value) { !UUID_PATTERN.test(value.preflightId || '') || !DIGEST_PATTERN.test(value.commandDigest || '') || !DIGEST_PATTERN.test(value.preflightDigest || '') || + !TRANSITION_KINDS.includes(value.transitionKind) || !Array.isArray(value.steps) || - value.steps.length !== 2 || + value.steps.length !== 3 || value.steps[0]?.name !== 'cluster_identity_before' || - value.steps[1]?.name !== 'server_side_dry_run' || + value.steps[1]?.name !== 'deployment_head_read' || + value.steps[2]?.name !== 'server_side_dry_run' || !exactKeys(value.verification, [ 'catalogBoundLock', 'clusterIdentityBound', 'serverSideDryRun', + 'deploymentHeadCompared', + 'resourceInventoryClosed', 'networkAccess', 'kubernetesMutation', 'externalResultsReplayed', @@ -1184,6 +1753,8 @@ function validatePreflight(value) { catalogBoundLock: true, clusterIdentityBound: true, serverSideDryRun: true, + deploymentHeadCompared: true, + resourceInventoryClosed: true, networkAccess: true, kubernetesMutation: false, externalResultsReplayed: false, @@ -1194,6 +1765,8 @@ function validatePreflight(value) { fail('deployment preflight report is invalid'); } validateLockSummary(value.lock); + validateResourceInventory(value.resourceInventory); + validateHeadSnapshot(value.deploymentHead); validateTargetSummary(value.target); validateToolSummary(value.tool); for (const step of value.steps) validateStepRecord(step); @@ -1307,17 +1880,28 @@ function createPreflight(commandState, dependencies = {}) { const output = outputTarget(request.output, 'preflight output'); const inputs = inspectInputs(request); const runner = dependencies.runProcess ?? defaultRunProcess; - const steps = [ - identityStep(inputs, request, runner, 'cluster_identity_before'), - dryRunStep(inputs, request, runner), - ]; + const identity = identityStep( + inputs, + request, + runner, + 'cluster_identity_before', + ); + const current = headReadStep(inputs, request, runner); + if (current.state !== null && current.state.phase !== 'committed') { + fail('deployment head contains an unfinished transition'); + } + validateTransition(current, request, deploymentPlan(inputs)); + const steps = [identity, current.record, dryRunStep(inputs, request, runner)]; revalidateStableInputs(inputs, request); const unsigned = { schemaVersion: 1, schema: PREFLIGHT_SCHEMA, preflightId: request.preflightId, commandDigest: commandState.commandDigest, + transitionKind: request.transitionKind, lock: lockSummary(inputs), + resourceInventory: inputs.resourceInventory, + deploymentHead: current.snapshot, target: targetSummary(inputs, request), tool: toolSummary(inputs), steps, @@ -1325,6 +1909,8 @@ function createPreflight(commandState, dependencies = {}) { catalogBoundLock: true, clusterIdentityBound: true, serverSideDryRun: true, + deploymentHeadCompared: true, + resourceInventoryClosed: true, networkAccess: true, kubernetesMutation: false, externalResultsReplayed: false, @@ -1349,7 +1935,12 @@ function readAndValidatePreflight(request, inputs) { const report = validatePreflight(file.value); if ( report.preflightDigest !== request.preflight.expectedDigest || + report.transitionKind !== request.transitionKind || JSON.stringify(report.lock) !== JSON.stringify(lockSummary(inputs)) || + JSON.stringify(report.resourceInventory) !== + JSON.stringify(inputs.resourceInventory) || + JSON.stringify(report.deploymentHead) !== + JSON.stringify(expectedSnapshot(request, inputs.headNamespace)) || JSON.stringify(report.target) !== JSON.stringify(targetSummary(inputs, request)) || JSON.stringify(report.tool) !== JSON.stringify(toolSummary(inputs)) @@ -1359,6 +1950,46 @@ function readAndValidatePreflight(request, inputs) { return Object.freeze({ file, report }); } +function receiptFromCommitted(state, namespace) { + if (state.phase !== 'committed') { + fail('deployment receipt requires a committed head'); + } + const unsigned = { + schemaVersion: 1, + schema: RECEIPT_SCHEMA, + mutationId: state.transition.mutationId, + commandDigest: state.transition.commandDigest, + preflightDigest: state.transition.preflightDigest, + transitionKind: state.transition.kind, + lock: state.deployment.lock, + resourceInventory: state.deployment.resources, + deploymentHead: headSnapshot(namespace, state), + target: state.target, + tool: state.tool, + steps: state.steps, + verification: { + catalogBoundLock: true, + clusterIdentityBound: true, + serverSideDryRun: true, + serverSideApply: true, + convergenceRead: true, + deploymentHeadCas: true, + resourceInventoryClosed: true, + networkAccess: true, + kubernetesMutation: true, + crossResourceAtomicity: false, + externalResultsReplayed: false, + recovery: 'resume_exact_transition_from_target_head', + ambientHome: false, + ephemeralCache: true, + }, + }; + return Object.freeze({ + ...unsigned, + receiptDigest: sha256(Buffer.from(JSON.stringify(unsigned), 'utf8')), + }); +} + function validateReceipt(value, expectedCommandDigest) { if ( !exactKeys(value, [ @@ -1367,7 +1998,10 @@ function validateReceipt(value, expectedCommandDigest) { 'mutationId', 'commandDigest', 'preflightDigest', + 'transitionKind', 'lock', + 'resourceInventory', + 'deploymentHead', 'target', 'tool', 'steps', @@ -1380,6 +2014,7 @@ function validateReceipt(value, expectedCommandDigest) { value.commandDigest !== expectedCommandDigest || !DIGEST_PATTERN.test(value.preflightDigest || '') || !DIGEST_PATTERN.test(value.receiptDigest || '') || + !TRANSITION_KINDS.includes(value.transitionKind) || !Array.isArray(value.steps) || JSON.stringify(value.steps.map((step) => step.name)) !== JSON.stringify([ @@ -1395,6 +2030,8 @@ function validateReceipt(value, expectedCommandDigest) { 'serverSideDryRun', 'serverSideApply', 'convergenceRead', + 'deploymentHeadCas', + 'resourceInventoryClosed', 'networkAccess', 'kubernetesMutation', 'crossResourceAtomicity', @@ -1410,11 +2047,13 @@ function validateReceipt(value, expectedCommandDigest) { serverSideDryRun: true, serverSideApply: true, convergenceRead: true, + deploymentHeadCas: true, + resourceInventoryClosed: true, networkAccess: true, kubernetesMutation: true, crossResourceAtomicity: false, externalResultsReplayed: false, - recovery: 'reapply_exact_lock_with_same_field_manager', + recovery: 'resume_exact_transition_from_target_head', ambientHome: false, ephemeralCache: true, }) @@ -1422,6 +2061,8 @@ function validateReceipt(value, expectedCommandDigest) { fail('deployment receipt is invalid'); } validateLockSummary(value.lock); + validateResourceInventory(value.resourceInventory); + validateHeadSnapshot(value.deploymentHead); validateTargetSummary(value.target); validateToolSummary(value.tool); for (const step of value.steps) validateStepRecord(step); @@ -1452,28 +2093,91 @@ function existingReceipt(request, commandDigest) { return receipt; } +function assertTransitionBinding(state, commandState, preflight, inputs) { + const request = commandState.command.request; + if ( + state.transition.kind !== request.transitionKind || + state.transition.mutationId !== request.mutationId || + state.transition.commandDigest !== commandState.commandDigest || + state.transition.preflightDigest !== preflight.report.preflightDigest || + JSON.stringify(state.transition.expectedHead) !== + JSON.stringify(preflight.report.deploymentHead) || + JSON.stringify(state.planned) !== JSON.stringify(deploymentPlan(inputs)) || + JSON.stringify(state.target) !== + JSON.stringify(targetSummary(inputs, request)) || + JSON.stringify(state.tool) !== JSON.stringify(toolSummary(inputs)) + ) { + fail('deployment head transition binding changed'); + } +} + function applyDeployment(commandState, dependencies = {}) { const request = commandState.command.request; const resumed = existingReceipt(request, commandState.commandDigest); + const output = + resumed === null + ? outputTarget(request.output, 'apply receipt output') + : request.output; + const inputs = inspectInputs(request); + const preflight = readAndValidatePreflight(request, inputs); + const runner = dependencies.runProcess ?? defaultRunProcess; + const current = headReadStep(inputs, request, runner); if (resumed !== null) { - const inputs = inspectInputs(request); - const preflight = readAndValidatePreflight(request, inputs); if ( + current.state === null || + current.state.phase !== 'committed' || resumed.preflightDigest !== preflight.report.preflightDigest || JSON.stringify(resumed.lock) !== JSON.stringify(lockSummary(inputs)) || + JSON.stringify(resumed.resourceInventory) !== + JSON.stringify(inputs.resourceInventory) || + JSON.stringify(resumed.deploymentHead) !== + JSON.stringify(headSnapshot(inputs.headNamespace, current.state)) || JSON.stringify(resumed.target) !== JSON.stringify(targetSummary(inputs, request)) || JSON.stringify(resumed.tool) !== JSON.stringify(toolSummary(inputs)) ) { fail('existing deployment receipt authority changed'); } + assertTransitionBinding(current.state, commandState, preflight, inputs); revalidateStableInputs(inputs, request); return resumed; } - const output = outputTarget(request.output, 'apply receipt output'); - const inputs = inspectInputs(request); - const preflight = readAndValidatePreflight(request, inputs); - const runner = dependencies.runProcess ?? defaultRunProcess; + if ( + current.state !== null && + current.state.phase === 'committed' && + current.state.transition.commandDigest === commandState.commandDigest + ) { + assertTransitionBinding(current.state, commandState, preflight, inputs); + const recovered = receiptFromCommitted(current.state, inputs.headNamespace); + writeNoReplace(output, recovered); + revalidateStableInputs(inputs, request); + return recovered; + } + let applying; + let headResourceVersion; + if (current.state !== null && current.state.phase === 'applying') { + assertTransitionBinding(current.state, commandState, preflight, inputs); + applying = current.state; + headResourceVersion = current.resourceVersion; + } else { + if ( + JSON.stringify(current.snapshot) !== + JSON.stringify(preflight.report.deploymentHead) + ) { + fail('deployment head changed after preflight'); + } + validateTransition(current, request, deploymentPlan(inputs)); + applying = applyingHead(commandState, preflight, inputs, current); + const acquired = writeHead( + inputs, + request, + runner, + applying, + current.resourceVersion, + 'deployment_head_acquire', + ); + headResourceVersion = acquired.resourceVersion; + } const steps = [ identityStep(inputs, request, runner, 'cluster_identity_before'), dryRunStep(inputs, request, runner), @@ -1498,35 +2202,16 @@ function applyDeployment(commandState, dependencies = {}) { steps.push(convergenceReadStep(inputs, request, runner)); steps.push(identityStep(inputs, request, runner, 'cluster_identity_after')); revalidateStableInputs(inputs, request); - const unsigned = { - schemaVersion: 1, - schema: RECEIPT_SCHEMA, - mutationId: request.mutationId, - commandDigest: commandState.commandDigest, - preflightDigest: preflight.report.preflightDigest, - lock: lockSummary(inputs), - target: targetSummary(inputs, request), - tool: toolSummary(inputs), - steps, - verification: { - catalogBoundLock: true, - clusterIdentityBound: true, - serverSideDryRun: true, - serverSideApply: true, - convergenceRead: true, - networkAccess: true, - kubernetesMutation: true, - crossResourceAtomicity: false, - externalResultsReplayed: false, - recovery: 'reapply_exact_lock_with_same_field_manager', - ambientHome: false, - ephemeralCache: true, - }, - }; - const receipt = Object.freeze({ - ...unsigned, - receiptDigest: sha256(Buffer.from(JSON.stringify(unsigned), 'utf8')), - }); + const committed = committedHead(applying, Object.freeze(steps)); + writeHead( + inputs, + request, + runner, + committed, + headResourceVersion, + 'deployment_head_commit', + ); + const receipt = receiptFromCommitted(committed, inputs.headNamespace); writeNoReplace(output, receipt); return receipt; } @@ -1555,6 +2240,7 @@ function auditReceipt(command) { receipt.receiptDigest !== command.request.receipt.expectedDigest || receipt.mutationId !== applyCommand.request.mutationId || receipt.preflightDigest !== applyCommand.request.preflight.expectedDigest || + receipt.transitionKind !== applyCommand.request.transitionKind || applyCommand.request.output !== command.request.receipt.path ) { fail('deployment receipt command binding changed'); @@ -1565,8 +2251,13 @@ function auditReceipt(command) { mutationId: receipt.mutationId, receiptDigest: receipt.receiptDigest, preflightDigest: receipt.preflightDigest, + transitionKind: receipt.transitionKind, lockDigest: receipt.lock.lockDigest, manifestDigest: receipt.lock.manifestDigest, + deploymentGeneration: receipt.deploymentHead.generation, + deploymentDigest: receipt.deploymentHead.deploymentDigest, + deploymentHeadStateDigest: receipt.deploymentHead.stateDigest, + resourceCount: receipt.resourceInventory.length, clusterUid: receipt.target.clusterUid, externalResultsReplayed: false, kubernetesMutation: false, @@ -1587,6 +2278,9 @@ function executeCommand(commandFile, dependencies = {}) { module.exports = Object.freeze({ COMMAND_SCHEMA, FIELD_MANAGER, + HEAD_DATA_KEY, + HEAD_NAME, + HEAD_SCHEMA, LOCK_SCHEMA, PREFLIGHT_SCHEMA, RECEIPT_SCHEMA, diff --git a/scripts/ql3-kubernetes-deployment-live-contract.cjs b/scripts/ql3-kubernetes-deployment-live-contract.cjs index d3f6ffbd..fa9f3cc4 100644 --- a/scripts/ql3-kubernetes-deployment-live-contract.cjs +++ b/scripts/ql3-kubernetes-deployment-live-contract.cjs @@ -8,6 +8,8 @@ const path = require('node:path'); const { COMMAND_SCHEMA, FIELD_MANAGER, + HEAD_DATA_KEY, + HEAD_NAME, canonicalJson, executeCommand, } = require('./lib/ql3-kubernetes-deployment-ceremony.cjs'); @@ -74,9 +76,9 @@ function references() { } function lockedArtifacts() { - const releaseSetDigest = digest('d341-live-release-set'); - const catalogManifestDigest = digest('d341-live-catalog-manifest'); - const catalogReportDigest = digest('d341-live-catalog-report'); + const releaseSetDigest = digest('d342-live-release-set'); + const catalogManifestDigest = digest('d342-live-catalog-manifest'); + const catalogReportDigest = digest('d342-live-catalog-report'); const imageReferences = references(); const annotations = { 'qinglong.io/release-set-digest': releaseSetDigest, @@ -162,7 +164,7 @@ function lockedArtifacts() { count: role === 'admin' ? 2 : 1, })), manifest: { - inputDigest: digest('d341-live-source-render'), + inputDigest: digest('d342-live-source-render'), outputDigest: digest(manifest), resources: resources.length, changedResources: 5, @@ -256,6 +258,13 @@ async function main() { }, context: CONTEXT, expectedClusterUid: clusterUid, + transitionKind: 'install', + expectedHead: { + generation: 0, + deploymentDigest: null, + lockDigest: null, + stateDigest: null, + }, }; const preflightPath = path.join(ceremonyDirectory, 'preflight.json'); const preflightCommand = writeCommand( @@ -298,6 +307,25 @@ async function main() { }, ); const audit = executeCommand(auditCommand); + const deploymentHeadConfigMap = fixture.kubectlJson([ + 'get', + 'configmap', + HEAD_NAME, + '-n', + NAMESPACE, + ]); + const deploymentHead = JSON.parse( + deploymentHeadConfigMap.data?.[HEAD_DATA_KEY], + ); + if ( + deploymentHead.phase !== 'committed' || + deploymentHead.generation !== 1 || + deploymentHead.stateDigest !== receipt.deploymentHead.stateDigest || + deploymentHead.deployment?.deploymentDigest !== + receipt.deploymentHead.deploymentDigest + ) { + fail('committed deployment head is invalid'); + } const deployments = fixture.kubectlJson([ 'get', 'deployments', @@ -345,6 +373,11 @@ async function main() { replicas: 0, fieldManager: FIELD_MANAGER, immutableImages: true, + headName: HEAD_NAME, + headPhase: deploymentHead.phase, + headGeneration: deploymentHead.generation, + deploymentDigest: deploymentHead.deployment.deploymentDigest, + resourceInventoryCount: deploymentHead.deployment.resources.length, }, preflightDigest: preflight.preflightDigest, receiptDigest: receipt.receiptDigest, @@ -352,6 +385,8 @@ async function main() { serverSideDryRun: preflight.verification.serverSideDryRun, serverSideApply: receipt.verification.serverSideApply, convergenceRead: receipt.verification.convergenceRead, + deploymentHeadCas: receipt.verification.deploymentHeadCas, + resourceInventoryClosed: receipt.verification.resourceInventoryClosed, crossResourceAtomicity: receipt.verification.crossResourceAtomicity, }); } finally { diff --git a/scripts/ql3-service-bridge-import-audit.cjs b/scripts/ql3-service-bridge-import-audit.cjs index 165ef4ad..e8039aa0 100644 --- a/scripts/ql3-service-bridge-import-audit.cjs +++ b/scripts/ql3-service-bridge-import-audit.cjs @@ -49,6 +49,8 @@ function main() { file.startsWith( 'packages/ql3-local-owner-cli/dist/deployment/service-manager/', ) || + file === + 'packages/ql3-local-owner-cli/dist/deployment/foundation/error.js' || file === 'packages/ql3-local-owner-cli/dist/deployment/foundation/contract.js' || file.startsWith('packages/ql3-local-command-file/dist/'); diff --git a/test/back/ql3KubernetesDeploymentCeremony.test.cjs b/test/back/ql3KubernetesDeploymentCeremony.test.cjs index 4ef469c8..6e5b35d4 100644 --- a/test/back/ql3KubernetesDeploymentCeremony.test.cjs +++ b/test/back/ql3KubernetesDeploymentCeremony.test.cjs @@ -11,6 +11,8 @@ const yaml = require('js-yaml'); const { COMMAND_SCHEMA, FIELD_MANAGER, + HEAD_DATA_KEY, + HEAD_NAME, PREFLIGHT_SCHEMA, RECEIPT_SCHEMA, canonicalJson, @@ -44,21 +46,21 @@ function image(repository, digit) { return `ghcr.io/qinglong-release/${repository}@sha256:${digit.repeat(64)}`; } -function annotations() { +function annotations(version = VERSION) { return { 'qinglong.io/release-set-digest': RELEASE_SET_DIGEST, 'qinglong.io/release-catalog-manifest-digest': CATALOG_MANIFEST_DIGEST, 'qinglong.io/release-catalog-report-digest': CATALOG_REPORT_DIGEST, 'qinglong.io/release-source-revision': SOURCE_REVISION, - 'qinglong.io/release-version': VERSION, + 'qinglong.io/release-version': version, }; } -function manifest() { +function manifest({ version = VERSION, extraResource = false } = {}) { const metadata = (name) => ({ name, namespace: 'qinglong-system', - annotations: annotations(), + annotations: annotations(version), }); const deployment = (name, containerName, reference) => ({ apiVersion: 'apps/v1', @@ -67,7 +69,10 @@ function manifest() { spec: { selector: { matchLabels: { app: name } }, template: { - metadata: { labels: { app: name }, annotations: annotations() }, + metadata: { + labels: { app: name }, + annotations: annotations(version), + }, spec: { containers: [{ name: containerName, image: reference }] }, }, }, @@ -81,7 +86,7 @@ function manifest() { metadata: metadata('ql3-admin'), spec: { template: { - metadata: { annotations: annotations() }, + metadata: { annotations: annotations(version) }, spec: { restartPolicy: 'Never', containers: [{ name: 'admin', image: REFERENCES.admin }], @@ -97,26 +102,36 @@ function manifest() { data: { image: REFERENCES.admin }, }, ]; + if (extraResource) { + resources.push({ + apiVersion: 'v1', + kind: 'ConfigMap', + metadata: metadata('ql3-extra-release-resource'), + data: { purpose: 'resource-inventory-closure-test' }, + }); + } return `${resources .map((resource) => JSON.stringify(resource)) .join('\n---\n')}\n`; } -function lockReport(manifestContents = manifest()) { +function lockReport(manifestContents = manifest(), version = VERSION) { + const resources = []; + yaml.loadAll(manifestContents, (resource) => resources.push(resource)); const unsigned = { schemaVersion: 1, schema: 'qinglong/kubernetes-deployment-lock@v2', release: { - version: VERSION, + version, sourceRevision: SOURCE_REVISION, - sourceRef: `refs/tags/v${VERSION}`, + sourceRef: `refs/tags/v${version}`, scope: 'cluster', }, releaseSetDigest: RELEASE_SET_DIGEST, catalog: { schema: 'qinglong/release-catalog-consumption-ceremony@v1', sourceRepository: 'qinglong-release/qinglong', - workflowIdentity: `https://github.com/qinglong-release/qinglong/.github/workflows/ql3-image-release.yml@refs/tags/v${VERSION}`, + workflowIdentity: `https://github.com/qinglong-release/qinglong/.github/workflows/ql3-image-release.yml@refs/tags/v${version}`, immutableReference: `ghcr.io/qinglong-release/qinglong3-release-catalog@${CATALOG_MANIFEST_DIGEST}`, manifestDigest: CATALOG_MANIFEST_DIGEST, consumptionReportDigest: CATALOG_REPORT_DIGEST, @@ -138,7 +153,7 @@ function lockReport(manifestContents = manifest()) { manifest: { inputDigest: digest('source-render'), outputDigest: digest(manifestContents), - resources: 5, + resources: resources.length, changedResources: 5, admissionAuthorityCount: 1, }, @@ -159,14 +174,19 @@ function lockReport(manifestContents = manifest()) { }; } -function fixture(t) { +function fixture(t, options = {}) { const directory = fs.realpathSync( fs.mkdtempSync(path.join(os.tmpdir(), 'ql3-kubernetes-deployment-')), ); fs.chmodSync(directory, 0o700); t.after(() => fs.rmSync(directory, { recursive: true, force: true })); - const lockedManifest = privateFile(directory, 'locked.yaml', manifest()); - const report = lockReport(); + const manifestContents = manifest(options); + const lockedManifest = privateFile( + directory, + 'locked.yaml', + manifestContents, + ); + const report = lockReport(manifestContents, options.version ?? VERSION); const lockReportPath = privateFile( directory, 'lock.json', @@ -207,6 +227,7 @@ users: kubectl, kubeconfig, report, + manifestContents, }; } @@ -250,28 +271,72 @@ function commonRequest(fixtureValue) { }, context: CONTEXT, expectedClusterUid: CLUSTER_UID, + transitionKind: 'install', + expectedHead: { + generation: 0, + deploymentDigest: null, + lockDigest: null, + stateDigest: null, + }, }; } -function successfulRunner(calls) { - return (_executable, args, input) => { +function successfulRunner(calls, server = {}, manifestContents = manifest()) { + server.head ??= null; + server.resourceVersion ??= 0; + const runner = (_executable, args, input) => { calls.push({ args: [...args], input }); if (args.includes('get') && args.includes('-f=-')) { - return { status: 0, stdout: convergenceList(), stderr: '' }; + return { + status: 0, + stdout: convergenceList(manifestContents), + stderr: '', + }; } - if (args.includes('get')) { + if (args.includes('get') && args.includes(HEAD_NAME)) { + return { + status: 0, + stdout: server.head === null ? '' : JSON.stringify(server.head), + stderr: '', + }; + } + if (args.includes('get') && args.includes('kube-system')) { return { status: 0, stdout: CLUSTER_UID, stderr: '' }; } if (args.includes('--dry-run=server')) { return { status: 0, stdout: 'deployment.apps/ql3-control\n', stderr: '' }; } + if (args.includes('create') || args.includes('replace')) { + const configMap = JSON.parse(input); + if ( + (args.includes('create') && server.head !== null) || + (args.includes('replace') && + (server.head === null || + configMap.metadata.resourceVersion !== + server.head.metadata.resourceVersion)) + ) { + return { status: 1, stdout: '', stderr: 'conflict' }; + } + server.resourceVersion += 1; + server.head = { + ...configMap, + metadata: { + ...configMap.metadata, + resourceVersion: String(server.resourceVersion), + uid: '123e4567-e89b-42d3-a456-426614174099', + }, + }; + return { status: 0, stdout: JSON.stringify(server.head), stderr: '' }; + } return { status: 0, stdout: 'deployment.apps/ql3-control\n', stderr: '' }; }; + runner.server = server; + return runner; } -function convergenceList() { +function convergenceList(manifestContents = manifest()) { const resources = []; - yaml.loadAll(manifest(), (resource) => resources.push(resource)); + yaml.loadAll(manifestContents, (resource) => resources.push(resource)); return JSON.stringify({ apiVersion: 'v1', kind: 'List', @@ -310,8 +375,9 @@ function prepare(t) { }, ); const calls = []; + const runner = successfulRunner(calls, {}, value.manifestContents); const report = executeCommand(command, { - runProcess: successfulRunner(calls), + runProcess: runner, }); return { ...value, @@ -319,6 +385,63 @@ function prepare(t) { preflightPath: output, preflight: report, calls, + runner, + }; +} + +function expectedFromReceipt(receipt) { + return { + generation: receipt.deploymentHead.generation, + deploymentDigest: receipt.deploymentHead.deploymentDigest, + lockDigest: receipt.deploymentHead.lockDigest, + stateDigest: receipt.deploymentHead.stateDigest, + }; +} + +function performTransition( + value, + runner, + transitionKind, + expectedHead, + suffix, +) { + const preflightPath = path.join(value.directory, `${suffix}-preflight.json`); + const preflightCommand = writeCommand( + value.directory, + `${suffix}-preflight-command.json`, + 'cluster.deployment.preflight', + { + preflightId: crypto.randomUUID(), + ...commonRequest(value), + transitionKind, + expectedHead, + output: preflightPath, + }, + ); + const preflight = executeCommand(preflightCommand, { runProcess: runner }); + const receiptPath = path.join(value.directory, `${suffix}-receipt.json`); + const applyCommand = writeCommand( + value.directory, + `${suffix}-apply-command.json`, + 'cluster.deployment.apply', + { + mutationId: crypto.randomUUID(), + preflight: { + path: preflightPath, + expectedDigest: preflight.preflightDigest, + }, + ...commonRequest(value), + transitionKind, + expectedHead, + output: receiptPath, + }, + ); + return { + preflight, + preflightPath, + applyCommand, + receiptPath, + receipt: executeCommand(applyCommand, { runProcess: runner }), }; } @@ -336,17 +459,18 @@ test('preflight binds one catalog lock to one Kubernetes target without mutation assert.equal(value.preflight.verification.kubernetesMutation, false); assert.deepEqual( value.preflight.steps.map(({ name }) => name), - ['cluster_identity_before', 'server_side_dry_run'], + ['cluster_identity_before', 'deployment_head_read', 'server_side_dry_run'], ); assert.equal(fs.statSync(value.preflightPath).mode & 0o777, 0o600); - assert.equal(value.calls.length, 2); + assert.equal(value.calls.length, 3); assert.equal(value.calls[0].args.includes('get'), true); - assert.equal(value.calls[1].args.includes('--dry-run=server'), true); + assert.equal(value.calls[1].args.includes(HEAD_NAME), true); + assert.equal(value.calls[2].args.includes('--dry-run=server'), true); assert.equal( - value.calls[1].args.includes(`--field-manager=${FIELD_MANAGER}`), + value.calls[2].args.includes(`--field-manager=${FIELD_MANAGER}`), true, ); - assert.equal(value.calls[1].input, manifest()); + assert.equal(value.calls[2].input, manifest()); }); test('preflight rejects lock, manifest and annotation drift before network access', (t) => { @@ -427,10 +551,11 @@ test('preflight rejects the wrong cluster and rejected server dry-run', (t) => { output, }, ); + const base = successfulRunner([]); assert.throws(() => executeCommand(command, { - runProcess(_executable, args) { - if (args.includes('get')) { + runProcess(executable, args, input) { + if (args.includes('get') && args.includes('kube-system')) { return { status: 0, stdout: @@ -438,7 +563,14 @@ test('preflight rejects the wrong cluster and rejected server dry-run', (t) => { stderr: '', }; } - return { status: 1, stdout: '', stderr: 'redacted admission error' }; + if (failure === 'dry-run' && args.includes('--dry-run=server')) { + return { + status: 1, + stdout: '', + stderr: 'redacted admission error', + }; + } + return base(executable, args, input); }, }), ); @@ -508,8 +640,12 @@ test('apply revalidates preflight, mutates explicitly and proves convergence', ( }, ); const calls = []; + value.calls.length = 0; const receipt = executeCommand(applyCommand, { - runProcess: successfulRunner(calls), + runProcess(executable, args, input) { + calls.push({ args: [...args], input }); + return value.runner(executable, args, input); + }, }); assert.equal(receipt.schema, RECEIPT_SCHEMA); assert.equal(receipt.preflightDigest, value.preflight.preflightDigest); @@ -517,7 +653,7 @@ test('apply revalidates preflight, mutates explicitly and proves convergence', ( assert.equal(receipt.verification.crossResourceAtomicity, false); assert.equal( receipt.verification.recovery, - 'reapply_exact_lock_with_same_field_manager', + 'resume_exact_transition_from_target_head', ); assert.deepEqual( receipt.steps.map(({ name }) => name), @@ -529,7 +665,7 @@ test('apply revalidates preflight, mutates explicitly and proves convergence', ( 'cluster_identity_after', ], ); - assert.equal(calls.length, 5); + assert.equal(calls.length, 8); assert.equal(fs.statSync(receiptPath).mode & 0o777, 0o600); const auditCommand = writeCommand( @@ -550,8 +686,13 @@ test('apply revalidates preflight, mutates explicitly and proves convergence', ( mutationId: receipt.mutationId, receiptDigest: receipt.receiptDigest, preflightDigest: receipt.preflightDigest, + transitionKind: 'install', lockDigest: value.report.lockDigest, manifestDigest: value.report.manifest.outputDigest, + deploymentGeneration: 1, + deploymentDigest: receipt.deploymentHead.deploymentDigest, + deploymentHeadStateDigest: receipt.deploymentHead.stateDigest, + resourceCount: 5, clusterUid: CLUSTER_UID, externalResultsReplayed: false, kubernetesMutation: false, @@ -560,14 +701,14 @@ test('apply revalidates preflight, mutates explicitly and proves convergence', ( let replayCalls = 0; assert.equal( executeCommand(applyCommand, { - runProcess() { + runProcess(executable, args, input) { replayCalls += 1; - throw new Error('must not run'); + return value.runner(executable, args, input); }, }).receiptDigest, receipt.receiptDigest, ); - assert.equal(replayCalls, 0); + assert.equal(replayCalls, 1); }); test('apply failure or post-apply drift never publishes a success receipt', (t) => { @@ -594,9 +735,10 @@ test('apply failure or post-apply drift never publishes a success receipt', (t) }, ); let identityCount = 0; + const base = value.runner; assert.throws(() => executeCommand(applyCommand, { - runProcess(_executable, args) { + runProcess(executable, args, input) { if (args.includes('get') && args.includes('-f=-')) { return { status: failure === 'convergence' ? 1 : 0, @@ -605,7 +747,7 @@ test('apply failure or post-apply drift never publishes a success receipt', (t) failure === 'convergence' ? 'redacted convergence error' : '', }; } - if (args.includes('get')) { + if (args.includes('get') && args.includes('kube-system')) { identityCount += 1; return { status: 0, @@ -619,11 +761,14 @@ test('apply failure or post-apply drift never publishes a success receipt', (t) if (args.includes('--dry-run=server')) { return { status: 0, stdout: 'dry-run', stderr: '' }; } - return { - status: failure === 'apply' ? 1 : 0, - stdout: '', - stderr: failure === 'apply' ? 'redacted apply error' : '', - }; + if (failure === 'apply' && args.includes('apply')) { + return { + status: 1, + stdout: '', + stderr: 'redacted apply error', + }; + } + return base(executable, args, input); }, }), ); @@ -631,6 +776,208 @@ test('apply failure or post-apply drift never publishes a success receipt', (t) } }); +test('apply resumes the exact applying intent from the target head', (t) => { + const value = prepare(t); + const receiptPath = path.join(value.directory, 'resumed-receipt.json'); + const applyCommand = writeCommand( + value.directory, + 'resumed-apply-command.json', + 'cluster.deployment.apply', + { + mutationId: '123e4567-e89b-42d3-a456-426614174043', + preflight: { + path: value.preflightPath, + expectedDigest: value.preflight.preflightDigest, + }, + ...commonRequest(value), + output: receiptPath, + }, + ); + let rejected = false; + assert.throws(() => + executeCommand(applyCommand, { + runProcess(executable, args, input) { + if ( + !rejected && + args.includes('apply') && + !args.includes('--dry-run=server') + ) { + rejected = true; + return { status: 1, stdout: '', stderr: 'transient rejection' }; + } + return value.runner(executable, args, input); + }, + }), + ); + assert.equal(fs.existsSync(receiptPath), false); + assert.equal( + JSON.parse(value.runner.server.head.data[HEAD_DATA_KEY]).phase, + 'applying', + ); + const receipt = executeCommand(applyCommand, { + runProcess: value.runner, + }); + assert.equal(receipt.deploymentHead.generation, 1); + assert.equal( + JSON.parse(value.runner.server.head.data[HEAD_DATA_KEY]).phase, + 'committed', + ); +}); + +test('a stale preflight cannot overtake a committed deployment intent', (t) => { + const value = prepare(t); + const competingPreflightPath = path.join( + value.directory, + 'competing-preflight.json', + ); + const competingPreflightCommand = writeCommand( + value.directory, + 'competing-preflight-command.json', + 'cluster.deployment.preflight', + { + preflightId: '123e4567-e89b-42d3-a456-426614174044', + ...commonRequest(value), + output: competingPreflightPath, + }, + ); + const competingPreflight = executeCommand(competingPreflightCommand, { + runProcess: value.runner, + }); + const winner = performTransition( + value, + value.runner, + 'install', + commonRequest(value).expectedHead, + 'winner', + ); + assert.equal(winner.receipt.deploymentHead.generation, 1); + const staleReceipt = path.join(value.directory, 'stale-receipt.json'); + const staleApply = writeCommand( + value.directory, + 'stale-apply-command.json', + 'cluster.deployment.apply', + { + mutationId: '123e4567-e89b-42d3-a456-426614174045', + preflight: { + path: competingPreflightPath, + expectedDigest: competingPreflight.preflightDigest, + }, + ...commonRequest(value), + output: staleReceipt, + }, + ); + assert.throws(() => executeCommand(staleApply, { runProcess: value.runner })); + assert.equal(fs.existsSync(staleReceipt), false); +}); + +test('upgrade fails closed when its target omits an active resource', (t) => { + const server = {}; + const installValue = fixture(t, { version: '3.0.0-alpha.0' }); + const installRunner = successfulRunner( + [], + server, + installValue.manifestContents, + ); + const installed = performTransition( + installValue, + installRunner, + 'install', + commonRequest(installValue).expectedHead, + 'inventory-install', + ); + const expandedValue = fixture(t, { + version: '3.0.0-alpha.1', + extraResource: true, + }); + const expandedRunner = successfulRunner( + [], + server, + expandedValue.manifestContents, + ); + const expanded = performTransition( + expandedValue, + expandedRunner, + 'upgrade', + expectedFromReceipt(installed.receipt), + 'inventory-expand', + ); + assert.equal(expanded.receipt.resourceInventory.length, 6); + + const reducedValue = fixture(t, { version: '3.0.0-alpha.2' }); + const reducedRunner = successfulRunner( + [], + server, + reducedValue.manifestContents, + ); + const reducedPreflightPath = path.join( + reducedValue.directory, + 'inventory-reduce-preflight.json', + ); + const reducedPreflight = writeCommand( + reducedValue.directory, + 'inventory-reduce-preflight-command.json', + 'cluster.deployment.preflight', + { + preflightId: crypto.randomUUID(), + ...commonRequest(reducedValue), + transitionKind: 'upgrade', + expectedHead: expectedFromReceipt(expanded.receipt), + output: reducedPreflightPath, + }, + ); + assert.throws(() => + executeCommand(reducedPreflight, { runProcess: reducedRunner }), + ); + assert.equal(fs.existsSync(reducedPreflightPath), false); +}); + +test('rollback restores only the exact previous lock with unchanged inventory', (t) => { + const server = {}; + const installValue = fixture(t, { version: '3.0.0-alpha.0' }); + const installRunner = successfulRunner( + [], + server, + installValue.manifestContents, + ); + const installed = performTransition( + installValue, + installRunner, + 'install', + commonRequest(installValue).expectedHead, + 'rollback-install', + ); + const upgradeValue = fixture(t, { version: '3.0.0-alpha.1' }); + const upgradeRunner = successfulRunner( + [], + server, + upgradeValue.manifestContents, + ); + const upgraded = performTransition( + upgradeValue, + upgradeRunner, + 'upgrade', + expectedFromReceipt(installed.receipt), + 'rollback-upgrade', + ); + const rolledBack = performTransition( + installValue, + installRunner, + 'rollback', + expectedFromReceipt(upgraded.receipt), + 'rollback-restore', + ); + assert.equal(rolledBack.receipt.transitionKind, 'rollback'); + assert.equal(rolledBack.receipt.deploymentHead.generation, 3); + assert.equal( + rolledBack.receipt.lock.lockDigest, + installed.receipt.lock.lockDigest, + ); + assert.deepEqual( + rolledBack.receipt.resourceInventory, + installed.receipt.resourceInventory, + ); +}); + test('receipt audit rejects a different command or recomputed receipt', (t) => { const value = prepare(t); const receiptPath = path.join(value.directory, 'receipt.json'); @@ -731,6 +1078,9 @@ const args = process.argv.slice(2); if (args.includes('get') && args.includes('-f=-')) process.stdout.write(${JSON.stringify( convergenceList(), )}); +else if (args.includes('get') && args.includes(${JSON.stringify( + HEAD_NAME, + )})) process.stdout.write(''); else if (args.includes('get')) process.stdout.write(${JSON.stringify( CLUSTER_UID, )}); diff --git a/test/back/ql3PackageBoundaryAudit.test.cjs b/test/back/ql3PackageBoundaryAudit.test.cjs index 29176dab..e890eb15 100644 --- a/test/back/ql3PackageBoundaryAudit.test.cjs +++ b/test/back/ql3PackageBoundaryAudit.test.cjs @@ -207,10 +207,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: localOwnerCli.rootSourceFileRoles, }, { - sourceFiles: 106, + sourceFiles: 107, rootSourceFiles: 1, rootSourceLines: 50, - nestedSourceFiles: 105, + nestedSourceFiles: 106, rootSourceFileRoles: { 'cli.ts': 'binary_entry' }, }, );