diff --git a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md index e065eb9f..1357d08c 100644 --- a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md +++ b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md @@ -93,6 +93,33 @@ readiness、观察窗和 rollback retention 之后另行强认证。Cluster 必须使用 PostgreSQL SERIALIZABLE ledger、外部 KMS/Secret provider 与 HA evidence， 不复用 Local SQLite/POSIX authority，也不得把明文写入 PostgreSQL、ConfigMap、Pod env 或 Job command。 +- D-400/ADR-0495（已验收）：第十一个 Secret/Config 切片补上 Cluster Legacy Env migration 的 + **content-free plan ledger baseline**，但没有把计划冒充实际迁移。profile-neutral + `qinglong/cluster-legacy-env-migration-plan@v1` 只保存 reconciliation bundle、reviewed + decision、candidate set、Task/Trigger revision set 和 plan 的 SHA-256 摘要，外加守恒计数、 + 同 Project 且固定 version 的 canonical SecretRef、effective bytes 与数据库时间；不保存源 + Env name/value、row body、plaintext/ciphertext、key ID、Task/Trigger spec 或 credential。 + 上限固定为 100,000 source rows、100,000 Tasks、500,000 Triggers、64 KiB effective + Secret 和 8 KiB JSON。 + + PostgreSQL `pg-0070-cluster-legacy-env-migration-plans` 将 control contract 推进到 v69， + 以 Project FK、plan/mutation/digest 唯一性和六组 named constraints 创建 append-only 表； + `plan_json` 必须与列值重建的 canonical JSONB 精确相等，因此额外 Env 字段不能藏入 durable + JSON。只有 `ql3_automation_manager` 拥有 `SELECT, INSERT`，无 UPDATE/DELETE；runtime、 + admin 和其余角色均无权限。repository 只从显式 + `cluster-legacy-env-migration-plan` subpath 导出，以数据库时间和短 SERIALIZABLE transaction + 完成 exact mutation replay、active Project snapshot、三次有界 serialization retry 与可选同事务 + hook；普通 SERIALIZABLE read 避免 `FOR SHARE` 隐含的 Project UPDATE 权限扩张。 + + runtime contract 4/4、repository 5/5、migration/schema/readiness/entrypoint 92/92 通过，两包 + 完整测试零失败。真实 PostgreSQL 18.4 已成功应用 70 条 migration，并证明 create/replay/read、 + automation UPDATE `42501`、runtime/admin SELECT `42501`，以及 widened `envName` JSON 被精确 + constraint 以 `23514` 拒绝。该切片复用既有两个 package，未增加 dependency、daemon 或 + Edge/Standalone import。它只关闭 plan ledger baseline；Cluster source scan/signed decision + 装配、逐项 Task/Trigger current-head revalidation 与 revision mutation、Secret material custody、 + migration receipt、HA promotion replay 和固定低性能设备证据仍是后续门禁。ADR-0491 因此继续 + Proposed。 + - D-396/ADR-0490（已验收）：Run History 不再只有永久 `manual_external`，但也没有被错误实现为 Legacy 日志到 3.0 Run ledger 的回灌。 新的 Local adapter 以 ADR-0482 sealed capture bundle 作为 append-only 保全资产：Legacy history 必须逐事实选择 `retain_both`，Target history 必须选择 `retain_target`；receipt 只绑定 signed review、application、bundle fingerprint、领域 inventory 与有界 fact counts，不保存表名、Run ID、 diff --git a/docs/adr/ADR-0491-bounded-secret-config-reconciliation-and-task-binding.md b/docs/adr/ADR-0491-bounded-secret-config-reconciliation-and-task-binding.md index cd581dc7..59391be4 100644 --- a/docs/adr/ADR-0491-bounded-secret-config-reconciliation-and-task-binding.md +++ b/docs/adr/ADR-0491-bounded-secret-config-reconciliation-and-task-binding.md @@ -1,6 +1,6 @@ # ADR-0491：有界 Secret/Config Reconciliation 与任务环境绑定 -- 状态：Proposed（D-397 已实现 Legacy Env inspection、私有有界 row plan、durable plan publication、独立 signed decision、逐项 Automation adoption provenance、Local SQLite 原子 application publisher、Owner prepared/apply/rollback 编排、ADR-0492 completion v3，并由 ADR-0494 完成 Cluster mounted-files provider live 子门；真实 Edge 空间证据、Cluster migration ledger 与直接外部 custody gate 尚未完成） +- 状态：Proposed（D-397 已实现 Legacy Env inspection、私有有界 row plan、durable plan publication、独立 signed decision、逐项 Automation adoption provenance、Local SQLite 原子 application publisher、Owner prepared/apply/rollback 编排、ADR-0492 completion v3；ADR-0494 完成 Cluster mounted-files provider live 子门，ADR-0495 完成 content-free Cluster plan ledger baseline；真实 Edge 空间证据、Cluster Task/Trigger mutation/receipt、HA replay 与直接外部 custody gate 尚未完成） - 日期：2026-08-23 - 决策：D-397 - 关联：ADR-0073、ADR-0074、ADR-0092、ADR-0094、ADR-0480、ADR-0482、ADR-0483、ADR-0484、ADR-0485、ADR-0486、ADR-0487、ADR-0488、ADR-0490 @@ -147,4 +147,4 @@ D-397 当前八切片已经实现：absent、unsupported、Edge over-budget、2. ADR-0494 已完成 Cluster `mounted-files` provider live 子门：真实三节点 K3s 中两个 management replica、direct exact-key executor 和两个跨节点 provider observer 完成 PostgreSQL durable approval/binding、Kubernetes atomic projection rotation、无 Secret API 权限/ServiceAccount token、只读 `0440`、内容脱敏及删除后 fail-closed；v2 私有报告 24/24 gates 为 true，并保持 v1 verifier 兼容。该门不增加 Edge 闭包，也不等于直接 Vault/KMS/HSM custody。 -转为 Accepted 前仍必须完成：固定低性能 Edge 设备的真实空间/写放大/断电恢复证据，以及 Cluster Legacy Env migration 的专用 PostgreSQL SERIALIZABLE ledger、Task/Trigger revision mutation、外部 custody adapter 和 HA promotion 后 receipt replay。ADR-0492 已完成本机 completion schema 演进和 completed-head 后 rollback material 回收，ADR-0493 又让没有 Legacy 身份输入的 fresh v52 目标身份经 signed `retain_target` 正确形成 no-effect，并精确消除六张已知目标表的 `unknown` 误判。Legacy `Auths/Users` 或真正未知表仍保持 manual；本切片的 Local Owner 编排、ADR-0494 的 mounted-files gate 或 PostgreSQL HA 证据都不得冒充完整 Cluster migration 与外部密钥托管。 +转为 Accepted 前仍必须完成：固定低性能 Edge 设备的真实空间/写放大/断电恢复证据，以及 Cluster Legacy Env migration 的逐项 Task/Trigger current-head revalidation、revision mutation/receipt、外部 custody adapter 和 HA promotion 后 receipt replay。ADR-0495 已完成专用 PostgreSQL SERIALIZABLE plan ledger baseline，但它只保存摘要、计数和 pinned SecretRef，不执行 Task/Trigger DML，也不接触 Secret material。ADR-0492 已完成本机 completion schema 演进和 completed-head 后 rollback material 回收，ADR-0493 又让没有 Legacy 身份输入的 fresh v52 目标身份经 signed `retain_target` 正确形成 no-effect，并精确消除六张已知目标表的 `unknown` 误判。Legacy `Auths/Users` 或真正未知表仍保持 manual；本切片的 Local Owner 编排、ADR-0494 的 mounted-files gate、ADR-0495 的 plan ledger 或 PostgreSQL HA 证据都不得冒充完整 Cluster migration 与外部密钥托管。 diff --git a/docs/adr/ADR-0495-content-free-cluster-legacy-env-migration-plan-ledger.md b/docs/adr/ADR-0495-content-free-cluster-legacy-env-migration-plan-ledger.md new file mode 100644 index 00000000..719d64aa --- /dev/null +++ b/docs/adr/ADR-0495-content-free-cluster-legacy-env-migration-plan-ledger.md @@ -0,0 +1,140 @@ +# ADR-0495：无敏感内容的 Cluster Legacy Env 迁移计划账本 + +- 状态：Accepted +- 日期：2026-08-24 +- 决策：D-400 +- 关联：ADR-0104、ADR-0233、ADR-0259、ADR-0491、ADR-0494 + +## 背景 + +ADR-0491 已完成 Local SQLite 上的 Legacy Env 检查、人工裁决、Secret application、 +Task/Trigger revision mutation、回滚和 completion；ADR-0494 又证明了 Cluster +`mounted-files` provider 的在线轮换。Cluster 仍缺少一个独立于 Local SQLite/POSIX +authority 的持久计划边界，无法在真正写 Secret、Task 或 Trigger 之前冻结源证据、目标 +revision 集合和幂等 mutation 身份。 + +该边界不能把 2.x `Envs` 的名称、值、行内容搬进 PostgreSQL，也不能为了“先跑起来” +让 runtime、admin 或通用 root export 取得迁移写权限。小型路由设备不应安装或加载本 +Cluster 能力；集群侧也必须有明确的行数、Task、Trigger 和有效载荷预算。 + +## 决策 + +### 1. 账本只保存不可逆摘要、计数和目标引用 + +新增 profile-neutral `qinglong/cluster-legacy-env-migration-plan@v1`。计划只包含： + +- plan、mutation、Project 身份； +- reconciliation bundle、reviewed decision、candidate set 三个 SHA-256 摘要； +- source/active/disabled/effective binding 计数； +- 同 Project、精确固定 version 的 canonical `SecretRef`； +- Task 与 Trigger revision set 摘要和计数； +- effective Secret 总字节数、数据库计划时间和 plan digest。 + +它不得包含源 Env 名称/值、row body、目标 plaintext/ciphertext、key ID、credential、 +Task spec、Trigger spec 或迁移日志。`SecretRef` 是既有目标路由引用，不是源 Env 名称； +其 payload 仍受 canonical encoding、同 Project 和固定 version 校验。 + +硬上限为 100,000 个源行、100,000 个 Task、500,000 个 Trigger、64 KiB effective +Secret 和 8 KiB plan JSON。active/disabled/source 计数必须守恒，至少存在一个 active +行、effective binding 和 Task。 + +### 2. PostgreSQL 只提供 append-only v69 计划表 + +`pg-0070-cluster-legacy-env-migration-plans` 将 control contract 推进到 v69，并新增 +`cluster_legacy_env_migration_plans`： + +- `plan_id` 为主键，`mutation_id` 与 `plan_digest` 各自唯一； +- Project 外键使用 `ON DELETE/UPDATE RESTRICT`； +- identity、digest、source、target、time、JSON 六组 named constraints； +- `plan_json` 必须与列值构造出的完整 canonical JSONB **精确相等**，不是包含关系， + 因而任何额外 Env 字段也会被数据库拒绝； +- 不提供 UPDATE、DELETE、TRUNCATE 或后台 GC。 + +迁移能力值新增 `cluster_legacy_env_migration_plan: 1`。Drizzle schema、readiness +contract、固定 migration checksum 和实际 SQL stream 必须保持同锁步。 + +### 3. 只有 Automation Manager 可以读取和追加 + +新表从 PUBLIC 撤销全部权限，只向 `ql3_automation_manager` 授予 `SELECT, INSERT`。 +runtime、admin、Worker ingress、Approval/Run/Package/Worker Credential 等其他角色均为 +零权限；migration owner 只用于 schema 管理与集成验证。 + +具体 repository 只由 +`@qinglong/cluster-postgres/cluster-legacy-env-migration-plan` 显式子路径发布，不进入 +package root、runtime 或 admin entrypoint。runtime-core 也只通过 +`@qinglong/runtime-core/cluster-legacy-env-migration-plan` 发布数据契约。 + +### 4. 发布使用数据库时间、SERIALIZABLE 和精确重放 + +repository 每次使用一个短 `SERIALIZABLE` transaction，设置 statement、lock 和 idle +transaction timeout；先按 `mutation_id` 查 exact replay，再验证 Project 在该可序列化 +快照中为 active，使用 `transaction_timestamp()` 生成 `plannedAtMs`，最后追加计划。 + +相同 mutation 与相同 intent 返回 durable `existing`；mutation drift、plan identity +占用、inactive Project 和 named constraint 冲突失败关闭。`40001`、`40P01`、`55P03` +最多重试三次。可选 transaction hook 只用于后续在**同一 Automation Manager 事务**中 +追加 Task/Trigger revision mutation；本 ADR 不调用它来声称那些 mutation 已实现。 + +Project 检查使用普通 SERIALIZABLE read，而不是 `FOR SHARE`。后者在 PostgreSQL 中还 +要求 UPDATE privilege，会错误扩大 Automation Manager 权限；可序列化顺序与外键已足以 +表达“计划发生在并发归档之前”的合法历史。 + +### 5. 不扩大 Edge/Standalone 闭包 + +实现复用既有 `runtime-core` 与 `cluster-postgres` package，不新增 workspace package、 +生产依赖、daemon、timer、watcher、controller 或缓存全集。Edge/Standalone 不导入 +PostgreSQL repository 或 migration；计数上限是 Cluster 计划输入预算，不是把集群表或 +Task 集合加载进小型设备。 + +## 被拒绝的替代方案 + +### 在 PostgreSQL 保存 Env 名称、值、密文或 key ID + +拒绝。密文仍会扩大 custody、rotation、backup 和 HA 泄漏面；计划账本只需要摘要和目标 +引用即可证明后续 mutation 的输入身份。 + +### 在同一切片直接改写全部 Task/Trigger head + +拒绝。计划发布和实际 DML 是两个可独立审计的 authority 阶段。没有 current-head +revalidation、逐项 mutation receipt 与完整 response-loss 测试前，不得把 plan 冒充迁移。 + +### 复用 Local SQLite 或 Plugin Package Secret Binding 表 + +拒绝。Local authority 不属于 Cluster；Plugin Package binding 的 installation/generation +身份也不能表达 Legacy Env reconciliation bundle、disabled preservation 和 Task/Trigger +revision set。 + +### 为一个 contract 和一个 repository 新建 package + +拒绝。它们分别属于既有 runtime migration contract 与 Cluster reconciliation storage +能力，新增微包会再次制造用户已经指出的单文件 package 问题。 + +## 当前验证 + +2026-08-24 已完成： + +- runtime contract 定向测试 4/4； +- PostgreSQL repository 定向测试 5/5； +- `@qinglong/runtime-core` 与 `@qinglong/cluster-postgres` 完整 package 测试零失败； +- PostgreSQL migration/schema/readiness/entrypoint 定向门 92/92； +- 真实 PostgreSQL 18.4 应用 70 条 migration 成功； +- 实际 automation-manager create/replay/read 成功，UPDATE 被 `42501` 拒绝； +- 实际 runtime/admin SELECT 被 `42501` 拒绝； +- 带源 `envName` 的 widened JSON 被 + `ql3_cluster_legacy_env_plan_json_check` 以 `23514` 拒绝。 + +## 边界与后续门禁 + +本 ADR 关闭 ADR-0491 的 **Cluster plan ledger baseline**，但不把 ADR-0491 整体转为 +Accepted，也不声明： + +1. Legacy source scan、sealed bundle 或 signed decision 已在 Cluster 中生产装配； +2. Task/Trigger current head 已逐项复验或 revision mutation 已提交； +3. Secret material 已写入 Kubernetes/Vault/KMS/HSM； +4. migration Job、短期身份、receipt 与 promotion 后 exact replay 已完成； +5. PostgreSQL physical failover、CloudNativePG promotion 或多架构结果已由本切片重新证明； +6. 固定低性能路由设备的空间、写放大、断电和恢复证据已完成。 + +下一切片应在 Automation Manager 同一事务内绑定 current Task/Trigger revision set、逐项 +mutation 与 receipt，同时保持计划表 append-only；随后再接入外部 custody 和 HA promotion +后的 exact replay。任何一步都不能回写源 Env 明文或放宽其他数据库角色。 diff --git a/docs/adr/README.md b/docs/adr/README.md index cc65a977..8af13ade 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -498,6 +498,7 @@ | [ADR-0492](./ADR-0492-compatible-secret-config-completion-v3.md) | 兼容 Secret/Config 的 Reconciliation Completion v3 | Accepted | | [ADR-0493](./ADR-0493-target-native-reconciliation-domain-preservation.md) | 目标原生 Reconciliation 域分类与身份保留 | Accepted | | [ADR-0494](./ADR-0494-postgresql-secret-binding-and-mounted-provider-live-rotation.md) | PostgreSQL Secret Binding 与 Mounted Provider 在线轮换门 | Accepted | +| [ADR-0495](./ADR-0495-content-free-cluster-legacy-env-migration-plan-ledger.md) | 无敏感内容的 Cluster Legacy Env 迁移计划账本 | Accepted | ## 规则 diff --git a/packages/ql3-cluster-control/test/application.test.cjs b/packages/ql3-cluster-control/test/application.test.cjs index a953cbc9..c7f11599 100644 --- a/packages/ql3-cluster-control/test/application.test.cjs +++ b/packages/ql3-cluster-control/test/application.test.cjs @@ -217,6 +217,7 @@ function runtimePrivileges() { plugin_package_workflow_admissions: [true, true, false, false], plugin_package_workflow_admission_steps: [true, true, false, false], plugin_package_workflow_task_attempt_admissions: [true, true, false, false], + cluster_legacy_env_migration_plans: [false, false, false, false], worker_execution_attestations: [true, false, false, false], run_events: [true, true, false, false], run_cancellation_dispatches: [true, true, true, false], diff --git a/packages/ql3-cluster-control/test/bootstrap.test.cjs b/packages/ql3-cluster-control/test/bootstrap.test.cjs index aa98d8ce..ef2cc429 100644 --- a/packages/ql3-cluster-control/test/bootstrap.test.cjs +++ b/packages/ql3-cluster-control/test/bootstrap.test.cjs @@ -131,6 +131,7 @@ function runtimePrivileges() { plugin_package_workflow_admissions: [true, true, false, false], plugin_package_workflow_admission_steps: [true, true, false, false], plugin_package_workflow_task_attempt_admissions: [true, true, false, false], + cluster_legacy_env_migration_plans: [false, false, false, false], worker_execution_attestations: [true, false, false, false], run_events: [true, true, false, false], run_cancellation_dispatches: [true, true, true, false], diff --git a/packages/ql3-cluster-postgres/package.json b/packages/ql3-cluster-postgres/package.json index c4c06efe..5ba1296a 100644 --- a/packages/ql3-cluster-postgres/package.json +++ b/packages/ql3-cluster-postgres/package.json @@ -65,6 +65,11 @@ "require": "./dist/entrypoints/automationManager.js", "default": "./dist/entrypoints/automationManager.js" }, + "./cluster-legacy-env-migration-plan": { + "types": "./dist/reconciliation/clusterLegacyEnvMigrationPlanRepository.d.ts", + "require": "./dist/reconciliation/clusterLegacyEnvMigrationPlanRepository.js", + "default": "./dist/reconciliation/clusterLegacyEnvMigrationPlanRepository.js" + }, "./task-start": { "types": "./dist/task-start/taskStartRepository.d.ts", "require": "./dist/task-start/taskStartRepository.js", diff --git a/packages/ql3-cluster-postgres/src/migration/migrationManifest.ts b/packages/ql3-cluster-postgres/src/migration/migrationManifest.ts index 5810215b..0fefdff7 100644 --- a/packages/ql3-cluster-postgres/src/migration/migrationManifest.ts +++ b/packages/ql3-cluster-postgres/src/migration/migrationManifest.ts @@ -353,5 +353,10 @@ export const postgresqlMainMigrationManifest: MigrationStreamManifest = checksum: '1191255575589abc2686b391827607abddb4edb78007245dbaaf45dc1c4e5e8b', }), + Object.freeze({ + id: 'pg-0070-cluster-legacy-env-migration-plans', + checksum: + '7cd6d993f48e7bcebcd62c93571a738d5117c9bcde33b974c5ac8962e2a03fe4', + }), ]), }); diff --git a/packages/ql3-cluster-postgres/src/migrations/index.ts b/packages/ql3-cluster-postgres/src/migrations/index.ts index bf3888ea..6f26f566 100644 --- a/packages/ql3-cluster-postgres/src/migrations/index.ts +++ b/packages/ql3-cluster-postgres/src/migrations/index.ts @@ -72,6 +72,7 @@ import { pg0066CancellationDispatchMigration } from '../run/migrations/pg-0066-c import { pg0067CancellationDispatchManagementMigration } from '../run-management/pg-0067-cancellation-dispatch-management'; import { pg0068CancellationDispatchProjectKeysetMigration } from '../run-management/pg-0068-cancellation-dispatch-project-keyset'; import { pg0069WorkerSessionManagementObservationMigration } from '../remote-execution/pg-0069-worker-session-management-observation'; +import { pg0070ClusterLegacyEnvMigrationPlansMigration } from '../reconciliation/pg-0070-cluster-legacy-env-migration-plans'; export const postgresqlMainMigrationStream: MigrationStreamDefinition = Object.freeze({ @@ -149,5 +150,6 @@ export const postgresqlMainMigrationStream: MigrationStreamDefinition; +type Queryable = Pick | Pick; + +export interface PostgresClusterLegacyEnvMigrationPlanTransactionContext { + readonly intent: Readonly; + readonly replay: Readonly | null; + readonly plan: Readonly; +} + +export type PostgresClusterLegacyEnvMigrationPlanTransactionHook = ( + client: PostgresClient, + context: Readonly, +) => Promise; + +function unavailable(): ClusterLegacyEnvMigrationPlanUnavailableError { + return new ClusterLegacyEnvMigrationPlanUnavailableError(); +} + +function planFromRow(row: Row): Readonly { + try { + return normalizeClusterLegacyEnvMigrationPlan( + postgresRequiredJsonObject( + row.planJson, + unavailable, + ) as unknown as ClusterLegacyEnvMigrationPlan, + ); + } catch (error) { + if (error instanceof ClusterLegacyEnvMigrationPlanUnavailableError) { + throw error; + } + throw unavailable(); + } +} + +async function findByPlanId( + queryable: Queryable, + planId: string, +): Promise | null> { + const result = await queryable.query( + `SELECT plan_json AS "planJson" + FROM "ql3"."cluster_legacy_env_migration_plans" + WHERE plan_id = $1 + LIMIT 2`, + [planId], + ); + if (result.rows.length === 0) return null; + if (result.rows.length !== 1) throw unavailable(); + const plan = planFromRow(result.rows[0]!); + if (plan.planId !== planId) throw unavailable(); + return plan; +} + +async function findByMutationId( + queryable: Queryable, + mutationId: string, +): Promise | null> { + const result = await queryable.query( + `SELECT plan_json AS "planJson" + FROM "ql3"."cluster_legacy_env_migration_plans" + WHERE mutation_id = $1 + LIMIT 2`, + [mutationId], + ); + if (result.rows.length === 0) return null; + if (result.rows.length !== 1) throw unavailable(); + const plan = planFromRow(result.rows[0]!); + if (plan.mutationId !== mutationId) throw unavailable(); + return plan; +} + +function mappedError(error: unknown): Error { + if ( + error instanceof ClusterLegacyEnvMigrationPlanConflictError || + error instanceof ClusterLegacyEnvMigrationPlanUnavailableError + ) { + return error; + } + const state = postgresSqlState(error); + if (state === '23503' || state === '23505' || state === '23514') { + return new ClusterLegacyEnvMigrationPlanConflictError(); + } + return unavailable(); +} + +/** + * Automation-manager-only append authority for content-free Cluster Legacy Env + * migration plans. It does not materialize Secrets or mutate Task/Trigger heads. + */ +export class PostgresClusterLegacyEnvMigrationPlanRepository + implements ClusterLegacyEnvMigrationPlanRepository +{ + constructor(private readonly pool: PostgresPool) { + if ( + !pool || + typeof pool.query !== 'function' || + typeof pool.connect !== 'function' + ) { + throw new TypeError( + 'PostgreSQL Cluster Legacy Env migration pool is invalid', + ); + } + } + + async findByPlanId( + planIdValue: string, + ): Promise | null> { + const planId = assertClusterLegacyEnvMigrationPlanIdentifier( + planIdValue, + 'planId', + ); + try { + return await findByPlanId(this.pool, planId); + } catch (error) { + throw mappedError(error); + } + } + + async publish( + intentValue: Readonly, + transactionHook?: PostgresClusterLegacyEnvMigrationPlanTransactionHook, + ): Promise< + Readonly<{ + status: 'created' | 'existing'; + plan: Readonly; + }> + > { + if ( + transactionHook !== undefined && + typeof transactionHook !== 'function' + ) { + throw new TypeError( + 'Cluster Legacy Env migration transaction hook is invalid', + ); + } + const intent = normalizeClusterLegacyEnvMigrationPlanIntent(intentValue); + + for ( + let attempt = 0; + attempt < POSTGRES_DEFINITION_TRANSACTION_ATTEMPTS; + attempt += 1 + ) { + let client: PostgresClient; + try { + client = await this.pool.connect(); + } catch { + throw unavailable(); + } + let began = false; + let transactionHookError: unknown; + try { + await configurePostgresDefinitionTransaction(client); + began = true; + + const replay = await findByMutationId(client, intent.mutationId); + if (replay) { + if (!clusterLegacyEnvMigrationPlanMatchesIntent(replay, intent)) { + throw new ClusterLegacyEnvMigrationPlanConflictError(); + } + if (transactionHook) { + try { + const hookResult = await transactionHook( + client, + Object.freeze({ intent, replay, plan: replay }), + ); + if (hookResult !== undefined) { + throw new TypeError( + 'Cluster Legacy Env migration transaction hook must not return a value', + ); + } + } catch (error) { + transactionHookError = error; + throw error; + } + } + await client.query('COMMIT'); + began = false; + return Object.freeze({ status: 'existing', plan: replay }); + } + + const project = await client.query<{ status: unknown }>( + `SELECT status + FROM "ql3"."projects" + WHERE id = $1`, + [intent.projectId], + ); + if (project.rows.length !== 1 || project.rows[0]?.status !== 'active') { + throw new ClusterLegacyEnvMigrationPlanConflictError(); + } + + const occupied = await findByPlanId(client, intent.planId); + if (occupied) throw new ClusterLegacyEnvMigrationPlanConflictError(); + + const clock = await client.query<{ plannedAtMs: unknown }>( + `SELECT floor(extract(epoch FROM transaction_timestamp()) * 1000)::bigint AS "plannedAtMs"`, + ); + if (clock.rows.length !== 1) throw unavailable(); + const plan = createClusterLegacyEnvMigrationPlan( + intent, + postgresRequiredInteger(clock.rows[0]?.plannedAtMs, unavailable), + ); + + await client.query( + `INSERT INTO "ql3"."cluster_legacy_env_migration_plans" ( + plan_id, mutation_id, project_id, plan_digest, + reconciliation_bundle_digest, decision_digest, + candidate_set_digest, source_row_count, active_row_count, + disabled_row_count, effective_binding_count, secret_ref, + task_revision_set_digest, trigger_revision_set_digest, + task_count, trigger_count, total_effective_bytes, + planned_at_ms, plan_json + ) VALUES ( + $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, + $11, $12, $13, $14, $15, $16, $17, $18, $19::jsonb + )`, + [ + plan.planId, + plan.mutationId, + plan.projectId, + plan.planDigest, + plan.source.reconciliationBundleDigest, + plan.source.decisionDigest, + plan.source.candidateSetDigest, + plan.source.sourceRowCount, + plan.source.activeRowCount, + plan.source.disabledRowCount, + plan.source.effectiveBindingCount, + plan.target.secretRef, + plan.target.taskRevisionSetDigest, + plan.target.triggerRevisionSetDigest, + plan.target.taskCount, + plan.target.triggerCount, + plan.target.totalEffectiveBytes, + plan.plannedAtMs, + JSON.stringify(plan), + ], + ); + + if (transactionHook) { + try { + const hookResult = await transactionHook( + client, + Object.freeze({ intent, replay: null, plan }), + ); + if (hookResult !== undefined) { + throw new TypeError( + 'Cluster Legacy Env migration transaction hook must not return a value', + ); + } + } catch (error) { + transactionHookError = error; + throw error; + } + } + await client.query('COMMIT'); + began = false; + return Object.freeze({ status: 'created', plan }); + } catch (error) { + if (began) await rollbackPostgresDefinitionTransaction(client); + const state = postgresSqlState(error); + if ( + state && + POSTGRES_DEFINITION_RETRYABLE_SQL_STATES.has(state) && + attempt + 1 < POSTGRES_DEFINITION_TRANSACTION_ATTEMPTS + ) { + continue; + } + if (error === transactionHookError && error instanceof Error) { + throw error; + } + throw mappedError(error); + } finally { + client.release(); + } + } + throw unavailable(); + } +} diff --git a/packages/ql3-cluster-postgres/src/reconciliation/pg-0070-cluster-legacy-env-migration-plans.ts b/packages/ql3-cluster-postgres/src/reconciliation/pg-0070-cluster-legacy-env-migration-plans.ts new file mode 100644 index 00000000..3d1d7841 --- /dev/null +++ b/packages/ql3-cluster-postgres/src/reconciliation/pg-0070-cluster-legacy-env-migration-plans.ts @@ -0,0 +1,105 @@ +import { CAPABILITIES_V68 } from '../remote-execution/pg-0069-worker-session-management-observation'; +import { definePostgresSqlMigration } from '../migrations/sqlMigration'; + +export const CAPABILITIES_V69 = CAPABILITIES_V68.replace( + '"cluster_execution_revision":1,', + '"cluster_execution_revision":1,"cluster_legacy_env_migration_plan":1,', +); + +export const pg0070ClusterLegacyEnvMigrationPlansMigration = + definePostgresSqlMigration({ + id: 'pg-0070-cluster-legacy-env-migration-plans', + statements: [ + ` +CREATE TABLE "ql3"."cluster_legacy_env_migration_plans" ( + plan_id varchar(128) PRIMARY KEY, + mutation_id varchar(128) NOT NULL, + project_id varchar(128) NOT NULL, + plan_digest char(64) NOT NULL, + reconciliation_bundle_digest char(64) NOT NULL, + decision_digest char(64) NOT NULL, + candidate_set_digest char(64) NOT NULL, + source_row_count integer NOT NULL, + active_row_count integer NOT NULL, + disabled_row_count integer NOT NULL, + effective_binding_count integer NOT NULL, + secret_ref varchar(512) NOT NULL, + task_revision_set_digest char(64) NOT NULL, + trigger_revision_set_digest char(64) NOT NULL, + task_count integer NOT NULL, + trigger_count integer NOT NULL, + total_effective_bytes integer NOT NULL, + planned_at_ms bigint NOT NULL, + plan_json jsonb NOT NULL, + CONSTRAINT ql3_cluster_legacy_env_plan_project_fk + FOREIGN KEY (project_id) REFERENCES "ql3"."projects" (id) + ON DELETE RESTRICT ON UPDATE RESTRICT, + CONSTRAINT ql3_cluster_legacy_env_plan_identity_check CHECK ( + plan_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' AND + mutation_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' AND + project_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' + ), + CONSTRAINT ql3_cluster_legacy_env_plan_digest_check CHECK ( + plan_digest ~ '^[0-9a-f]{64}$' AND + reconciliation_bundle_digest ~ '^[0-9a-f]{64}$' AND + decision_digest ~ '^[0-9a-f]{64}$' AND + candidate_set_digest ~ '^[0-9a-f]{64}$' AND + task_revision_set_digest ~ '^[0-9a-f]{64}$' AND + trigger_revision_set_digest ~ '^[0-9a-f]{64}$' + ), + CONSTRAINT ql3_cluster_legacy_env_plan_source_check CHECK ( + source_row_count BETWEEN 1 AND 100000 AND + active_row_count BETWEEN 1 AND 100000 AND + disabled_row_count BETWEEN 0 AND 100000 AND + source_row_count = active_row_count + disabled_row_count AND + effective_binding_count BETWEEN 1 AND active_row_count + ), + CONSTRAINT ql3_cluster_legacy_env_plan_target_check CHECK ( + secret_ref ~ '^qlsecret:v1:[A-Za-z0-9_-]+$' AND + octet_length(secret_ref) BETWEEN 14 AND 512 AND + task_count BETWEEN 1 AND 100000 AND + trigger_count BETWEEN 0 AND 500000 AND + total_effective_bytes BETWEEN 1 AND 65536 + ), + CONSTRAINT ql3_cluster_legacy_env_plan_time_check CHECK ( + planned_at_ms >= 0 + ), + CONSTRAINT ql3_cluster_legacy_env_plan_json_check CHECK ( + jsonb_typeof(plan_json) = 'object' AND + octet_length(plan_json::text) BETWEEN 2 AND 8192 AND + plan_json = jsonb_build_object( + 'schema', 'qinglong/cluster-legacy-env-migration-plan@v1', + 'planId', plan_id, + 'mutationId', mutation_id, + 'projectId', project_id, + 'source', jsonb_build_object( + 'reconciliationBundleDigest', reconciliation_bundle_digest, + 'decisionDigest', decision_digest, + 'candidateSetDigest', candidate_set_digest, + 'sourceRowCount', source_row_count, + 'activeRowCount', active_row_count, + 'disabledRowCount', disabled_row_count, + 'effectiveBindingCount', effective_binding_count + ), + 'target', jsonb_build_object( + 'secretRef', secret_ref, + 'taskRevisionSetDigest', task_revision_set_digest, + 'triggerRevisionSetDigest', trigger_revision_set_digest, + 'taskCount', task_count, + 'triggerCount', trigger_count, + 'totalEffectiveBytes', total_effective_bytes + ), + 'plannedAtMs', planned_at_ms, + 'planDigest', plan_digest + ) + ) +) + `.trim(), + `CREATE UNIQUE INDEX ql3_cluster_legacy_env_plan_mutation_uidx ON "ql3"."cluster_legacy_env_migration_plans" (mutation_id)`, + `CREATE UNIQUE INDEX ql3_cluster_legacy_env_plan_digest_uidx ON "ql3"."cluster_legacy_env_migration_plans" (plan_digest)`, + `CREATE INDEX ql3_cluster_legacy_env_plan_project_idx ON "ql3"."cluster_legacy_env_migration_plans" (project_id, planned_at_ms, plan_id)`, + `REVOKE ALL ON "ql3"."cluster_legacy_env_migration_plans" FROM PUBLIC`, + `GRANT SELECT, INSERT ON "ql3"."cluster_legacy_env_migration_plans" TO ql3_automation_manager`, + `DO $ql3$ BEGIN UPDATE "ql3"."schema_capabilities" SET contract_version = 69, migration_id = 'pg-0070-cluster-legacy-env-migration-plans', capabilities = '${CAPABILITIES_V69}'::jsonb, updated_at_ms = floor(extract(epoch FROM transaction_timestamp()) * 1000)::bigint WHERE contract_name = 'control-core' AND contract_version = 68 AND migration_id = 'pg-0069-worker-session-management-observation' AND capabilities = '${CAPABILITIES_V68}'::jsonb; IF NOT FOUND THEN RAISE EXCEPTION 'control-core capability is not at version 68' USING ERRCODE = 'check_violation'; END IF; END $ql3$`, + ], + }); diff --git a/packages/ql3-cluster-postgres/src/schema/schema.ts b/packages/ql3-cluster-postgres/src/schema/schema.ts index 55df24f0..900747a6 100644 --- a/packages/ql3-cluster-postgres/src/schema/schema.ts +++ b/packages/ql3-cluster-postgres/src/schema/schema.ts @@ -109,6 +109,77 @@ export const projects = ql3Schema.table( ], ); +export const clusterLegacyEnvMigrationPlans = ql3Schema.table( + 'cluster_legacy_env_migration_plans', + { + planId: varchar('plan_id', { length: 128 }).primaryKey(), + mutationId: varchar('mutation_id', { length: 128 }).notNull(), + projectId: varchar('project_id', { length: 128 }).notNull(), + planDigest: char('plan_digest', { length: 64 }).notNull(), + reconciliationBundleDigest: char('reconciliation_bundle_digest', { + length: 64, + }).notNull(), + decisionDigest: char('decision_digest', { length: 64 }).notNull(), + candidateSetDigest: char('candidate_set_digest', { length: 64 }).notNull(), + sourceRowCount: integer('source_row_count').notNull(), + activeRowCount: integer('active_row_count').notNull(), + disabledRowCount: integer('disabled_row_count').notNull(), + effectiveBindingCount: integer('effective_binding_count').notNull(), + secretRef: varchar('secret_ref', { length: 512 }).notNull(), + taskRevisionSetDigest: char('task_revision_set_digest', { + length: 64, + }).notNull(), + triggerRevisionSetDigest: char('trigger_revision_set_digest', { + length: 64, + }).notNull(), + taskCount: integer('task_count').notNull(), + triggerCount: integer('trigger_count').notNull(), + totalEffectiveBytes: integer('total_effective_bytes').notNull(), + plannedAtMs: bigint('planned_at_ms', { mode: 'number' }).notNull(), + planJson: jsonb('plan_json').$type>().notNull(), + }, + (table) => [ + foreignKey({ + name: 'ql3_cluster_legacy_env_plan_project_fk', + columns: [table.projectId], + foreignColumns: [projects.id], + }).onDelete('restrict'), + check( + 'ql3_cluster_legacy_env_plan_identity_check', + sql`${table.planId} ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' and ${table.mutationId} ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' and ${table.projectId} ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$'`, + ), + check( + 'ql3_cluster_legacy_env_plan_digest_check', + sql`${table.planDigest} ~ '^[0-9a-f]{64}$' and ${table.reconciliationBundleDigest} ~ '^[0-9a-f]{64}$' and ${table.decisionDigest} ~ '^[0-9a-f]{64}$' and ${table.candidateSetDigest} ~ '^[0-9a-f]{64}$' and ${table.taskRevisionSetDigest} ~ '^[0-9a-f]{64}$' and ${table.triggerRevisionSetDigest} ~ '^[0-9a-f]{64}$'`, + ), + check( + 'ql3_cluster_legacy_env_plan_source_check', + sql`${table.sourceRowCount} between 1 and 100000 and ${table.activeRowCount} between 1 and 100000 and ${table.disabledRowCount} between 0 and 100000 and ${table.sourceRowCount} = ${table.activeRowCount} + ${table.disabledRowCount} and ${table.effectiveBindingCount} between 1 and ${table.activeRowCount}`, + ), + check( + 'ql3_cluster_legacy_env_plan_target_check', + sql`${table.secretRef} ~ '^qlsecret:v1:[A-Za-z0-9_-]+$' and octet_length(${table.secretRef}) between 14 and 512 and ${table.taskCount} between 1 and 100000 and ${table.triggerCount} between 0 and 500000 and ${table.totalEffectiveBytes} between 1 and 65536`, + ), + check( + 'ql3_cluster_legacy_env_plan_time_check', + sql`${table.plannedAtMs} >= 0`, + ), + check( + 'ql3_cluster_legacy_env_plan_json_check', + sql`jsonb_typeof(${table.planJson}) = 'object' and octet_length(${table.planJson}::text) between 2 and 8192 and ${table.planJson} = jsonb_build_object('schema', 'qinglong/cluster-legacy-env-migration-plan@v1', 'planId', ${table.planId}, 'mutationId', ${table.mutationId}, 'projectId', ${table.projectId}, 'source', jsonb_build_object('reconciliationBundleDigest', ${table.reconciliationBundleDigest}, 'decisionDigest', ${table.decisionDigest}, 'candidateSetDigest', ${table.candidateSetDigest}, 'sourceRowCount', ${table.sourceRowCount}, 'activeRowCount', ${table.activeRowCount}, 'disabledRowCount', ${table.disabledRowCount}, 'effectiveBindingCount', ${table.effectiveBindingCount}), 'target', jsonb_build_object('secretRef', ${table.secretRef}, 'taskRevisionSetDigest', ${table.taskRevisionSetDigest}, 'triggerRevisionSetDigest', ${table.triggerRevisionSetDigest}, 'taskCount', ${table.taskCount}, 'triggerCount', ${table.triggerCount}, 'totalEffectiveBytes', ${table.totalEffectiveBytes}), 'plannedAtMs', ${table.plannedAtMs}, 'planDigest', ${table.planDigest})`, + ), + uniqueIndex('ql3_cluster_legacy_env_plan_mutation_uidx').on( + table.mutationId, + ), + uniqueIndex('ql3_cluster_legacy_env_plan_digest_uidx').on(table.planDigest), + index('ql3_cluster_legacy_env_plan_project_idx').on( + table.projectId, + table.plannedAtMs, + table.planId, + ), + ], +); + export const pluginPackageInstalls = ql3Schema.table( 'plugin_package_installs', { @@ -520,7 +591,9 @@ export const pluginPackageSecretBindingTransitionApprovalPlans = 'plugin_package_secret_binding_transition_approval_plans', { actionRef: varchar('action_ref', { length: 255 }).primaryKey(), - approvalPlanDigest: char('approval_plan_digest', { length: 64 }).notNull(), + approvalPlanDigest: char('approval_plan_digest', { + length: 64, + }).notNull(), transitionDigest: char('transition_digest', { length: 64 }).notNull(), generationDigest: char('generation_digest', { length: 64 }).notNull(), projectId: varchar('project_id', { length: 128 }).notNull(), @@ -1045,7 +1118,9 @@ export const approvedActionManualRecoveryResolutions = ql3Schema.table( resolvedById: varchar('resolved_by_id', { length: 255 }).notNull(), authenticationId: varchar('authentication_id', { length: 128 }).notNull(), assurance: varchar('assurance', { length: 32 }).notNull(), - authenticatedAtMs: bigint('authenticated_at_ms', { mode: 'number' }).notNull(), + authenticatedAtMs: bigint('authenticated_at_ms', { + mode: 'number', + }).notNull(), projectVersion: integer('project_version').notNull(), bindingVersion: integer('binding_version').notNull(), auditEventId: uuid('audit_event_id').notNull(), @@ -6213,6 +6288,7 @@ export const ql3PostgresTables = [ schemaMigrations, schemaCapabilities, projects, + clusterLegacyEnvMigrationPlans, pluginPackageInstalls, pluginPackageInstallHeads, pluginPackageInstallMutations, diff --git a/packages/ql3-cluster-postgres/src/schema/schemaContract.ts b/packages/ql3-cluster-postgres/src/schema/schemaContract.ts index 576a307c..9ef3165c 100644 --- a/packages/ql3-cluster-postgres/src/schema/schemaContract.ts +++ b/packages/ql3-cluster-postgres/src/schema/schemaContract.ts @@ -21,8 +21,8 @@ export interface PostgresSchemaContractTrigger { export interface PostgresSchemaContract { readonly schema: 'ql3'; readonly contractName: 'control-core'; - readonly contractVersion: 68; - readonly migrationId: 'pg-0069-worker-session-management-observation'; + readonly contractVersion: 69; + readonly migrationId: 'pg-0070-cluster-legacy-env-migration-plans'; readonly minimumServerMajor: 16; readonly maximumServerMajor: 18; readonly capabilities: Readonly<{ @@ -50,6 +50,7 @@ export interface PostgresSchemaContract { cluster_scheduler_admission: 1; database_role_grants: 1; cluster_execution_revision: 1; + cluster_legacy_env_migration_plan: 1; identity_admin: 1; plugin_package_admission: 1; plugin_package_authority_split: 1; @@ -122,8 +123,8 @@ export const postgresqlControlSchemaContract: PostgresSchemaContract = Object.freeze({ schema: 'ql3', contractName: 'control-core', - contractVersion: 68, - migrationId: 'pg-0069-worker-session-management-observation', + contractVersion: 69, + migrationId: 'pg-0070-cluster-legacy-env-migration-plans', minimumServerMajor: 16, maximumServerMajor: 18, capabilities: Object.freeze({ @@ -137,6 +138,7 @@ export const postgresqlControlSchemaContract: PostgresSchemaContract = automation_management_boundary: 1, automation_management_identity_keyset_ledger: 1, cluster_execution_revision: 1, + cluster_legacy_env_migration_plan: 1, cluster_recovery: 1, cluster_recovery_claim: 1, cluster_scheduler_admission: 1, @@ -228,6 +230,27 @@ export const postgresqlControlSchemaContract: PostgresSchemaContract = 'created_at_ms', 'updated_at_ms', ]), + table('cluster_legacy_env_migration_plans', [ + 'plan_id', + 'mutation_id', + 'project_id', + 'plan_digest', + 'reconciliation_bundle_digest', + 'decision_digest', + 'candidate_set_digest', + 'source_row_count', + 'active_row_count', + 'disabled_row_count', + 'effective_binding_count', + 'secret_ref', + 'task_revision_set_digest', + 'trigger_revision_set_digest', + 'task_count', + 'trigger_count', + 'total_effective_bytes', + 'planned_at_ms', + 'plan_json', + ]), table('plugin_package_installs', [ 'installation_id', 'project_id', @@ -1539,6 +1562,10 @@ export const postgresqlControlSchemaContract: PostgresSchemaContract = 'schema_capabilities_pkey', 'projects_pkey', 'ql3_projects_slug_uidx', + 'cluster_legacy_env_migration_plans_pkey', + 'ql3_cluster_legacy_env_plan_mutation_uidx', + 'ql3_cluster_legacy_env_plan_digest_uidx', + 'ql3_cluster_legacy_env_plan_project_idx', 'plugin_package_installs_pkey', 'ql3_plugin_package_installs_quarantine_target_key', 'ql3_plugin_package_installs_recovery_idx', @@ -1863,6 +1890,12 @@ export const postgresqlControlSchemaContract: PostgresSchemaContract = 'ql3_projects_version_check', 'ql3_projects_created_at_check', 'ql3_projects_updated_at_check', + 'ql3_cluster_legacy_env_plan_identity_check', + 'ql3_cluster_legacy_env_plan_digest_check', + 'ql3_cluster_legacy_env_plan_source_check', + 'ql3_cluster_legacy_env_plan_target_check', + 'ql3_cluster_legacy_env_plan_time_check', + 'ql3_cluster_legacy_env_plan_json_check', 'ql3_plugin_package_installs_identity_check', 'ql3_plugin_package_installs_operation_check', 'ql3_plugin_package_installs_state_check', @@ -2342,6 +2375,7 @@ export const postgresqlControlSchemaContract: PostgresSchemaContract = ]), foreignKeys: Object.freeze([ 'ql3_schema_capabilities_migration_fk', + 'ql3_cluster_legacy_env_plan_project_fk', 'ql3_plugin_package_installs_project_fk', 'ql3_plugin_package_install_heads_project_fk', 'ql3_plugin_package_install_heads_install_fk', diff --git a/packages/ql3-cluster-postgres/src/schema/schemaReadiness.ts b/packages/ql3-cluster-postgres/src/schema/schemaReadiness.ts index 022390b6..42221528 100644 --- a/packages/ql3-cluster-postgres/src/schema/schemaReadiness.ts +++ b/packages/ql3-cluster-postgres/src/schema/schemaReadiness.ts @@ -150,6 +150,12 @@ const REQUIRED_RUNTIME_PRIVILEGES = Object.freeze({ update: true, delete: false, }), + cluster_legacy_env_migration_plans: Object.freeze({ + select: false, + insert: false, + update: false, + delete: false, + }), plugin_package_installs: Object.freeze({ select: false, insert: false, @@ -699,6 +705,12 @@ const REQUIRED_ADMIN_PRIVILEGES = Object.freeze({ update: false, delete: false, }), + cluster_legacy_env_migration_plans: Object.freeze({ + select: false, + insert: false, + update: false, + delete: false, + }), plugin_package_installs: Object.freeze({ select: false, insert: false, @@ -1417,6 +1429,7 @@ const REQUIRED_AUTOMATION_MANAGER_PRIVILEGES: RequiredPrivileges = update: true, } : name === 'security_audit_events' || + name === 'cluster_legacy_env_migration_plans' || name === 'task_definition_revisions' || name === 'task_execution_revisions' || name === 'trigger_revisions' diff --git a/packages/ql3-cluster-postgres/test/clusterLegacyEnvMigrationPlanRepository.test.cjs b/packages/ql3-cluster-postgres/test/clusterLegacyEnvMigrationPlanRepository.test.cjs new file mode 100644 index 00000000..265c4b81 --- /dev/null +++ b/packages/ql3-cluster-postgres/test/clusterLegacyEnvMigrationPlanRepository.test.cjs @@ -0,0 +1,286 @@ +const assert = require('node:assert/strict'); +const { test } = require('node:test'); + +const { + ClusterLegacyEnvMigrationPlanConflictError, + ClusterLegacyEnvMigrationPlanUnavailableError, +} = require('@qinglong/runtime-core/cluster-legacy-env-migration-plan'); +const { createSecretRef } = require('@qinglong/runtime-core/secret-reference'); +const { + PostgresClusterLegacyEnvMigrationPlanRepository, +} = require('@qinglong/cluster-postgres/cluster-legacy-env-migration-plan'); + +function intent(overrides = {}) { + const projectId = overrides.projectId ?? 'project-a'; + return { + planId: 'legacy-env-plan-a', + mutationId: 'legacy-env-mutation-a', + projectId, + source: { + reconciliationBundleDigest: '1'.repeat(64), + decisionDigest: '2'.repeat(64), + candidateSetDigest: '3'.repeat(64), + sourceRowCount: 3, + activeRowCount: 2, + disabledRowCount: 1, + effectiveBindingCount: 2, + }, + target: { + secretRef: createSecretRef({ + projectId, + name: 'legacy-env-bundle', + version: 7, + }), + taskRevisionSetDigest: '4'.repeat(64), + triggerRevisionSetDigest: '5'.repeat(64), + taskCount: 2, + triggerCount: 3, + totalEffectiveBytes: 1024, + }, + ...overrides, + }; +} + +function fixture(options = {}) { + const plansById = new Map(); + const plansByMutation = new Map(); + const queries = []; + let serializationFailures = options.serializationFailures ?? 0; + let connections = 0; + + const pool = { + async query(text, values) { + queries.push({ scope: 'pool', text, values }); + if (text.includes('WHERE plan_id')) { + const plan = plansById.get(values[0]); + return { + rows: plan ? [{ planJson: plan }] : [], + rowCount: plan ? 1 : 0, + }; + } + throw new Error('unexpected pool query'); + }, + async connect() { + connections += 1; + return { + async query(text, values) { + queries.push({ scope: 'client', text, values }); + if ( + text === 'BEGIN ISOLATION LEVEL SERIALIZABLE' || + text === 'COMMIT' || + text === 'ROLLBACK' || + text.includes("set_config('") + ) { + return { rows: [], rowCount: 0 }; + } + if (text.includes('WHERE mutation_id')) { + if (serializationFailures > 0) { + serializationFailures -= 1; + throw Object.assign(new Error('serialization retry'), { + code: '40001', + }); + } + const plan = plansByMutation.get(values[0]); + return { + rows: plan ? [{ planJson: plan }] : [], + rowCount: plan ? 1 : 0, + }; + } + if (text.includes('FROM "ql3"."projects"')) { + return options.projectStatus === 'archived' + ? { rows: [{ status: 'archived' }], rowCount: 1 } + : { rows: [{ status: 'active' }], rowCount: 1 }; + } + if (text.includes('WHERE plan_id')) { + const plan = plansById.get(values[0]); + return { + rows: plan ? [{ planJson: plan }] : [], + rowCount: plan ? 1 : 0, + }; + } + if (text.includes('transaction_timestamp')) { + return { rows: [{ plannedAtMs: '12345' }], rowCount: 1 }; + } + if (text.includes('INSERT INTO')) { + const plan = JSON.parse(values[18]); + plansById.set(plan.planId, plan); + plansByMutation.set(plan.mutationId, plan); + return { rows: [], rowCount: 1 }; + } + if (text === 'SELECT hook_boundary') { + return { rows: [], rowCount: 0 }; + } + throw new Error(`unexpected client query: ${text}`); + }, + release() { + queries.push({ scope: 'client', text: 'RELEASE' }); + }, + }; + }, + }; + return { + repository: new PostgresClusterLegacyEnvMigrationPlanRepository(pool), + plansById, + plansByMutation, + queries, + connectionCount: () => connections, + }; +} + +test('publishes and exactly replays one content-free plan in serializable transactions', async () => { + const state = fixture(); + const hookContexts = []; + const hook = async (client, context) => { + hookContexts.push(context); + await client.query('SELECT hook_boundary'); + }; + const created = await state.repository.publish(intent(), hook); + const replay = await state.repository.publish(intent(), hook); + + assert.equal(created.status, 'created'); + assert.equal(replay.status, 'existing'); + assert.deepEqual(replay.plan, created.plan); + assert.equal(state.plansById.size, 1); + assert.equal(hookContexts[0].replay, null); + assert.deepEqual(hookContexts[1].replay, created.plan); + assert.equal( + state.queries.filter( + ({ text }) => text === 'BEGIN ISOLATION LEVEL SERIALIZABLE', + ).length, + 2, + ); + assert.equal(state.queries.filter(({ text }) => text === 'COMMIT').length, 2); + + const insert = state.queries.find(({ text }) => text.includes('INSERT INTO')); + const encodedPlan = insert.values[18]; + assert.doesNotMatch(encodedPlan, /TOKEN|secretValue|ciphertext|keyId/i); + assert.equal( + JSON.parse(encodedPlan).target.secretRef, + intent().target.secretRef, + ); +}); + +test('rejects mutation replay drift and inactive Projects without writing', async () => { + const replayState = fixture(); + await replayState.repository.publish(intent()); + await assert.rejects( + replayState.repository.publish( + intent({ + target: { ...intent().target, taskCount: 3 }, + }), + ), + ClusterLegacyEnvMigrationPlanConflictError, + ); + assert.equal( + replayState.queries.filter(({ text }) => text.includes('INSERT INTO')) + .length, + 1, + ); + assert.equal( + replayState.queries.some(({ text }) => text === 'ROLLBACK'), + true, + ); + + const archivedState = fixture({ projectStatus: 'archived' }); + await assert.rejects( + archivedState.repository.publish(intent()), + ClusterLegacyEnvMigrationPlanConflictError, + ); + assert.equal( + archivedState.queries.some(({ text }) => text.includes('INSERT INTO')), + false, + ); +}); + +test('retries bounded serializable failures and preserves the transaction hook error', async () => { + const state = fixture({ serializationFailures: 1 }); + const created = await state.repository.publish(intent()); + assert.equal(created.status, 'created'); + assert.equal(state.connectionCount(), 2); + assert.equal( + state.queries.filter(({ text }) => text === 'ROLLBACK').length, + 1, + ); + + const hookError = new Error('caller hook failed'); + await assert.rejects( + fixture().repository.publish(intent(), async () => { + throw hookError; + }), + (error) => error === hookError, + ); +}); + +test('fails closed on malformed durable JSON and hides raw storage errors', async () => { + const malformedPool = { + async query() { + return { rows: [{ planJson: { schema: 'wrong' } }], rowCount: 1 }; + }, + async connect() { + throw new Error('unused'); + }, + }; + await assert.rejects( + new PostgresClusterLegacyEnvMigrationPlanRepository( + malformedPool, + ).findByPlanId('legacy-env-plan-a'), + ClusterLegacyEnvMigrationPlanUnavailableError, + ); + + const identityDrift = fixture(); + const created = await identityDrift.repository.publish(intent()); + identityDrift.plansById.set('legacy-env-plan-b', created.plan); + await assert.rejects( + identityDrift.repository.findByPlanId('legacy-env-plan-b'), + ClusterLegacyEnvMigrationPlanUnavailableError, + ); + identityDrift.plansByMutation.set('legacy-env-mutation-b', created.plan); + await assert.rejects( + identityDrift.repository.publish( + intent({ + planId: 'legacy-env-plan-b', + mutationId: 'legacy-env-mutation-b', + }), + ), + ClusterLegacyEnvMigrationPlanUnavailableError, + ); + + const failedPool = { + async query() { + throw new Error('password=do-not-leak'); + }, + async connect() { + throw new Error('unused'); + }, + }; + await assert.rejects( + new PostgresClusterLegacyEnvMigrationPlanRepository( + failedPool, + ).findByPlanId('legacy-env-plan-a'), + (error) => { + assert.ok(error instanceof ClusterLegacyEnvMigrationPlanUnavailableError); + assert.doesNotMatch(error.message, /password|do-not-leak/i); + return true; + }, + ); +}); + +test('keeps the append authority behind its explicit package subpath', () => { + const root = require('@qinglong/cluster-postgres'); + const runtime = require('@qinglong/cluster-postgres/runtime'); + const admin = require('@qinglong/cluster-postgres/admin'); + const authority = require('@qinglong/cluster-postgres/cluster-legacy-env-migration-plan'); + assert.equal(root.PostgresClusterLegacyEnvMigrationPlanRepository, undefined); + assert.equal( + runtime.PostgresClusterLegacyEnvMigrationPlanRepository, + undefined, + ); + assert.equal( + admin.PostgresClusterLegacyEnvMigrationPlanRepository, + undefined, + ); + assert.equal( + typeof authority.PostgresClusterLegacyEnvMigrationPlanRepository, + 'function', + ); +}); diff --git a/packages/ql3-cluster-postgres/test/postgres.integration.test.cjs b/packages/ql3-cluster-postgres/test/postgres.integration.test.cjs index d3c0001a..b3eb6ffa 100644 --- a/packages/ql3-cluster-postgres/test/postgres.integration.test.cjs +++ b/packages/ql3-cluster-postgres/test/postgres.integration.test.cjs @@ -71,6 +71,9 @@ const { const { resolveClusterScheduleDecision, } = require('@qinglong/runtime-core/cluster-scheduler'); +const { + PostgresClusterLegacyEnvMigrationPlanRepository, +} = require('../dist/reconciliation/clusterLegacyEnvMigrationPlanRepository'); function nextMinute(schedule, afterMs) { if (schedule.expression !== '* * * * *' || schedule.timezone !== 'UTC') { @@ -877,7 +880,10 @@ if (!migrationConnectionString) { assert.equal(competing?.status, 'leased'); assert.equal(claimed.dispatch.version, 1); assert.equal(claimed.dispatch.dispatchCount, 1); - assert.equal(claimed.dispatch.createdAtMs >= Number(before.rows[0].nowMs), true); + assert.equal( + claimed.dispatch.createdAtMs >= Number(before.rows[0].nowMs), + true, + ); const rawLeaseToken = claimed.leaseToken; const stored = await migrationDatabase.pool.query( `SELECT lease_token_digest AS "leaseTokenDigest", @@ -963,11 +969,13 @@ if (!migrationConnectionString) { assert.equal(retry.dispatch.status, 'retry_wait'); assert.equal(retry.event.type, 'run.cancel_dispatch_failed'); assert.equal( - (await firstRepository.claim({ - ...candidate, - owner: 'primary-a', - leaseToken: 'lease-a-retry', - })).status, + ( + await firstRepository.claim({ + ...candidate, + owner: 'primary-a', + leaseToken: 'lease-a-retry', + }) + ).status, 'not_due', ); await migrationDatabase.pool.query( @@ -2199,9 +2207,7 @@ if (!migrationConnectionString) { const migrationDatabase = await open('migration'); try { await runPostgresMigrations({ pool: migrationDatabase.pool }); - await migrationDatabase.pool.query( - 'TRUNCATE TABLE "ql3"."runs" CASCADE', - ); + await migrationDatabase.pool.query('TRUNCATE TABLE "ql3"."runs" CASCADE'); await observeContractPublisherTrust(migrationDatabase.pool); await migrationDatabase.pool.query( `INSERT INTO "ql3"."projects" ( @@ -3847,12 +3853,13 @@ if (!migrationConnectionString) { const executions = new PostgresApprovedActionExecutionRepository( executorDatabase.pool, ); - const pendingSecretActions = - await executions.listReconciliableExecutions({ + const pendingSecretActions = await executions.listReconciliableExecutions( + { nowMs: claimedAtMs, limit: 1, actionTypes: [consumed.dispatch.action.actionType], - }); + }, + ); assert.equal(pendingSecretActions.truncated, false); assert.equal(pendingSecretActions.executions.length, 1); assert.equal( @@ -6532,4 +6539,136 @@ if (!migrationConnectionString) { await database.close(); } }); + + test('persists one content-free Legacy Env migration plan with isolated authority', async () => { + const projectId = `legacy-env-project-${process.pid}`; + const planId = `legacy-env-plan-${process.pid}`; + const mutationId = `legacy-env-mutation-${process.pid}`; + const migrationDatabase = await open('migration'); + const automationDatabase = await open('automation-manager'); + const runtimeDatabase = await open('runtime'); + const adminDatabase = await open('admin'); + try { + await runPostgresMigrations({ pool: migrationDatabase.pool }); + await migrationDatabase.pool.query( + `INSERT INTO "ql3"."projects" + (id, name, slug, status, version, created_at_ms, updated_at_ms) + VALUES ($1, $1, $1, 'active', 1, 1, 1) + ON CONFLICT (id) DO NOTHING`, + [projectId], + ); + const intent = { + planId, + mutationId, + projectId, + source: { + reconciliationBundleDigest: '1'.repeat(64), + decisionDigest: '2'.repeat(64), + candidateSetDigest: '3'.repeat(64), + sourceRowCount: 3, + activeRowCount: 2, + disabledRowCount: 1, + effectiveBindingCount: 2, + }, + target: { + secretRef: createSecretRef({ + projectId, + name: 'legacy-env-bundle', + version: 1, + }), + taskRevisionSetDigest: '4'.repeat(64), + triggerRevisionSetDigest: '5'.repeat(64), + taskCount: 2, + triggerCount: 1, + totalEffectiveBytes: 1024, + }, + }; + const repository = new PostgresClusterLegacyEnvMigrationPlanRepository( + automationDatabase.pool, + ); + const created = await repository.publish(intent); + const replay = await repository.publish(intent); + assert.equal(created.status, 'created'); + assert.equal(replay.status, 'existing'); + assert.deepEqual(replay.plan, created.plan); + assert.deepEqual(await repository.findByPlanId(planId), created.plan); + + const stored = await automationDatabase.pool.query( + `SELECT plan_json AS "planJson" + FROM "ql3"."cluster_legacy_env_migration_plans" + WHERE plan_id = $1`, + [planId], + ); + assert.equal(stored.rowCount, 1); + assert.deepEqual(stored.rows[0].planJson, created.plan); + assert.doesNotMatch( + JSON.stringify(stored.rows[0].planJson), + /TOKEN|secretValue|ciphertext|keyId/i, + ); + + const invalidPlanId = `${planId}-widened`; + const invalidMutationId = `${mutationId}-widened`; + const invalidDigest = 'f'.repeat(64); + await assert.rejects( + migrationDatabase.pool.query( + `INSERT INTO "ql3"."cluster_legacy_env_migration_plans" ( + plan_id, mutation_id, project_id, plan_digest, + reconciliation_bundle_digest, decision_digest, + candidate_set_digest, source_row_count, active_row_count, + disabled_row_count, effective_binding_count, secret_ref, + task_revision_set_digest, trigger_revision_set_digest, + task_count, trigger_count, total_effective_bytes, + planned_at_ms, plan_json + ) + SELECT $2::varchar, $3::varchar, project_id, $4::varchar, + reconciliation_bundle_digest, decision_digest, + candidate_set_digest, source_row_count, active_row_count, + disabled_row_count, effective_binding_count, secret_ref, + task_revision_set_digest, trigger_revision_set_digest, + task_count, trigger_count, total_effective_bytes, + planned_at_ms, + plan_json || jsonb_build_object( + 'planId', $2::varchar, + 'mutationId', $3::varchar, + 'planDigest', $4::varchar, + 'envName', 'TOKEN' + ) + FROM "ql3"."cluster_legacy_env_migration_plans" + WHERE plan_id = $1`, + [planId, invalidPlanId, invalidMutationId, invalidDigest], + ), + (error) => + error?.code === '23514' && + error?.constraint === 'ql3_cluster_legacy_env_plan_json_check', + ); + + await assert.rejects( + automationDatabase.pool.query( + `UPDATE "ql3"."cluster_legacy_env_migration_plans" + SET planned_at_ms = planned_at_ms + WHERE plan_id = $1`, + [planId], + ), + (error) => error?.code === '42501', + ); + for (const database of [runtimeDatabase, adminDatabase]) { + await assert.rejects( + database.pool.query( + `SELECT plan_id + FROM "ql3"."cluster_legacy_env_migration_plans" + WHERE plan_id = $1`, + [planId], + ), + (error) => error?.code === '42501', + ); + } + } finally { + await Promise.all([ + adminDatabase.close(), + runtimeDatabase.close(), + automationDatabase.close(), + migrationDatabase.close(), + ]); + } + }); } diff --git a/packages/ql3-cluster-postgres/test/postgresqlMigrationDefinitions.test.cjs b/packages/ql3-cluster-postgres/test/postgresqlMigrationDefinitions.test.cjs index 015d6698..cba23cc6 100644 --- a/packages/ql3-cluster-postgres/test/postgresqlMigrationDefinitions.test.cjs +++ b/packages/ql3-cluster-postgres/test/postgresqlMigrationDefinitions.test.cjs @@ -120,6 +120,7 @@ test('defines the immutable PostgreSQL capability and Run core stream', async () 'pg-0067-cancellation-dispatch-management', 'pg-0068-cancellation-dispatch-project-keyset', 'pg-0069-worker-session-management-observation', + 'pg-0070-cluster-legacy-env-migration-plans', ], ); for (const migration of postgresqlMainMigrationStream.migrations) { @@ -603,6 +604,11 @@ test('freezes every published PostgreSQL migration checksum', () => { checksum: '1191255575589abc2686b391827607abddb4edb78007245dbaaf45dc1c4e5e8b', }, + { + id: 'pg-0070-cluster-legacy-env-migration-plans', + checksum: + '7cd6d993f48e7bcebcd62c93571a738d5117c9bcde33b974c5ac8962e2a03fe4', + }, ]; assert.deepEqual( postgresqlMainMigrationStream.migrations.map(({ id, checksum }) => ({ @@ -2241,11 +2247,11 @@ test('advances capability v63 with manager-only immutable Secret transition plan sql, /GRANT SELECT ON "ql3"\."plugin_package_secret_binding_transition_approval_plans" TO ql3_package_manager, ql3_package_executor/, ); - assert.match( + assert.match(sql, /GRANT EXECUTE ON FUNCTION [^;]+ TO ql3_package_manager/); + assert.doesNotMatch( sql, - /GRANT EXECUTE ON FUNCTION [^;]+ TO ql3_package_manager/, + /GRANT EXECUTE ON FUNCTION [^;]+ TO ql3_package_executor/, ); - assert.doesNotMatch(sql, /GRANT EXECUTE ON FUNCTION [^;]+ TO ql3_package_executor/); assert.match(sql, /contract_version = 63/); assert.match( sql, @@ -2321,10 +2327,7 @@ test('advances capability v65 with database-timed fenced cancellation dispatch', sql, /CREATE UNIQUE INDEX ql3_run_attempts_run_id_uidx ON "ql3"\."run_attempts" \(run_id, id\)/, ); - assert.match( - sql, - /CREATE TABLE "ql3"\."run_cancellation_dispatches"/, - ); + assert.match(sql, /CREATE TABLE "ql3"\."run_cancellation_dispatches"/); assert.match( sql, /FOREIGN KEY \(run_id, attempt_id\)[\s\S]+REFERENCES "ql3"\."run_attempts" \(run_id, id\)/, @@ -2342,16 +2345,11 @@ test('advances capability v65 with database-timed fenced cancellation dispatch', assert.match(sql, /contract_version = 65/); assert.match(sql, /"run_cancellation_dispatch":1/); assert.match(sql, /contract_version = 64/); - assert.match( - sql, - /migration_id = 'pg-0065-approved-action-manual-recovery'/, - ); + assert.match(sql, /migration_id = 'pg-0065-approved-action-manual-recovery'/); }); test('advances capability v66 with least-privilege cancellation diagnostics and rearm', async () => { - const migration = migrationById( - 'pg-0067-cancellation-dispatch-management', - ); + const migration = migrationById('pg-0067-cancellation-dispatch-management'); const statements = []; await migration.up({ async query(statement) { @@ -2398,10 +2396,7 @@ test('advances capability v67 with a Project-scoped blocked keyset', async () => }, }); const sql = statements.join('\n'); - assert.match( - sql, - /ADD COLUMN project_id varchar\(128\)/, - ); + assert.match(sql, /ADD COLUMN project_id varchar\(128\)/); assert.match( sql, /SET project_id = run\.project_id FROM "ql3"\."runs" AS run/, @@ -2460,3 +2455,37 @@ test('advances capability v68 with read-only Worker session observation', async /migration_id = 'pg-0068-cancellation-dispatch-project-keyset'/, ); }); + +test('advances capability v69 with a content-free Legacy Env plan ledger', async () => { + const migration = migrationById('pg-0070-cluster-legacy-env-migration-plans'); + const statements = []; + await migration.up({ + async query(statement) { + statements.push(statement); + return { rows: [] }; + }, + }); + const sql = statements.join('\n'); + assert.match(sql, /CREATE TABLE "ql3"\."cluster_legacy_env_migration_plans"/); + assert.match(sql, /source_row_count BETWEEN 1 AND 100000/); + assert.match(sql, /total_effective_bytes BETWEEN 1 AND 65536/); + assert.match( + sql, + /plan_json = jsonb_build_object\([\s\S]+qinglong\/cluster-legacy-env-migration-plan@v1/, + ); + assert.match( + sql, + /GRANT SELECT, INSERT ON "ql3"\."cluster_legacy_env_migration_plans" TO ql3_automation_manager/, + ); + assert.doesNotMatch( + sql, + /GRANT (?:UPDATE|DELETE|TRUNCATE)[^;]+cluster_legacy_env_migration_plans/, + ); + assert.match(sql, /contract_version = 69/); + assert.match(sql, /"cluster_legacy_env_migration_plan":1/); + assert.match(sql, /contract_version = 68/); + assert.match( + sql, + /migration_id = 'pg-0069-worker-session-management-observation'/, + ); +}); diff --git a/packages/ql3-cluster-postgres/test/postgresqlSchemaReadiness.test.cjs b/packages/ql3-cluster-postgres/test/postgresqlSchemaReadiness.test.cjs index 0900776b..3314491e 100644 --- a/packages/ql3-cluster-postgres/test/postgresqlSchemaReadiness.test.cjs +++ b/packages/ql3-cluster-postgres/test/postgresqlSchemaReadiness.test.cjs @@ -33,6 +33,7 @@ function validPrivileges() { schema_migrations: [true, false, false, false], schema_capabilities: [true, false, false, false], projects: [true, true, true, false], + cluster_legacy_env_migration_plans: [false, false, false, false], task_definitions: [true, false, false, false], task_definition_revisions: [true, false, false, false], task_execution_revisions: [true, false, false, false], @@ -167,6 +168,7 @@ function validAdminPrivileges() { schema_migrations: [true, false, false, false], schema_capabilities: [true, false, false, false], projects: [true, false, false, false], + cluster_legacy_env_migration_plans: [false, false, false, false], task_definitions: [false, false, false, false], task_definition_revisions: [false, false, false, false], task_execution_revisions: [false, false, false, false], @@ -454,6 +456,7 @@ function automationManagerPrivileges() { 'project_role_bindings', 'plugin_package_task_ownerships', 'plugin_package_identity_keyset_ledger', + 'cluster_legacy_env_migration_plans', 'security_audit_events', 'task_definitions', 'task_definition_revisions', @@ -471,6 +474,7 @@ function automationManagerPrivileges() { 'trigger_revisions', 'trigger_schedules', 'plugin_package_identity_keyset_ledger', + 'cluster_legacy_env_migration_plans', ]); return postgresqlControlSchemaContract.tables.map(({ name: tableName }) => ({ tableName, @@ -791,9 +795,7 @@ function queryable(overrides = {}) { : 'runs'; assert.match( text, - new RegExp( - `format\\('%I\\.%I', \\$1::text, '${tableName}'\\)`, - ), + new RegExp(`format\\('%I\\.%I', \\$1::text, '${tableName}'\\)`), ); const columns = contract.tables.find( ({ name }) => name === tableName, @@ -836,7 +838,7 @@ test('accepts the exact PostgreSQL control schema and least-privilege runtime ro serverMajor: 16, currentUser: 'ql3_runtime', contractName: 'control-core', - contractVersion: 68, + contractVersion: 69, migrationIds: [ 'pg-0001-schema-capability', 'pg-0002-run-core', @@ -907,6 +909,7 @@ test('accepts the exact PostgreSQL control schema and least-privilege runtime ro 'pg-0067-cancellation-dispatch-management', 'pg-0068-cancellation-dispatch-project-keyset', 'pg-0069-worker-session-management-observation', + 'pg-0070-cluster-legacy-env-migration-plans', ], }); }); @@ -937,10 +940,10 @@ test('accepts the exact schema and isolated least-privilege admin role', async ( }), ); assert.equal(report.currentUser, 'ql3_admin'); - assert.equal(report.contractVersion, 68); + assert.equal(report.contractVersion, 69); assert.equal( report.migrationIds.at(-1), - 'pg-0069-worker-session-management-observation', + 'pg-0070-cluster-legacy-env-migration-plans', ); }); @@ -953,10 +956,10 @@ test('accepts the isolated least-privilege automation manager role', async () => }), ); assert.equal(report.currentUser, 'ql3_automation_manager'); - assert.equal(report.contractVersion, 68); + assert.equal(report.contractVersion, 69); assert.equal( report.migrationIds.at(-1), - 'pg-0069-worker-session-management-observation', + 'pg-0070-cluster-legacy-env-migration-plans', ); const widened = automationManagerPrivileges(); @@ -985,10 +988,10 @@ test('accepts the isolated least-privilege human Approval manager role', async ( }), ); assert.equal(report.currentUser, 'ql3_approval_manager'); - assert.equal(report.contractVersion, 68); + assert.equal(report.contractVersion, 69); assert.equal( report.migrationIds.at(-1), - 'pg-0069-worker-session-management-observation', + 'pg-0070-cluster-legacy-env-migration-plans', ); const widened = approvalManagerPrivileges(); @@ -1019,10 +1022,10 @@ test('accepts the isolated least-privilege Run manager role', async () => { }), ); assert.equal(report.currentUser, 'ql3_run_manager'); - assert.equal(report.contractVersion, 68); + assert.equal(report.contractVersion, 69); assert.equal( report.migrationIds.at(-1), - 'pg-0069-worker-session-management-observation', + 'pg-0070-cluster-legacy-env-migration-plans', ); const widened = runManagerPrivileges(); @@ -1183,10 +1186,10 @@ test('accepts the exact schema and isolated Worker ingress role', async () => { }), ); assert.equal(report.currentUser, 'ql3_worker_ingress'); - assert.equal(report.contractVersion, 68); + assert.equal(report.contractVersion, 69); assert.equal( report.migrationIds.at(-1), - 'pg-0069-worker-session-management-observation', + 'pg-0070-cluster-legacy-env-migration-plans', ); }); diff --git a/packages/ql3-runtime-core/package.json b/packages/ql3-runtime-core/package.json index 5af679fb..03da3f39 100644 --- a/packages/ql3-runtime-core/package.json +++ b/packages/ql3-runtime-core/package.json @@ -260,6 +260,9 @@ "builtin-run-log-excerpt-projection": [ "dist/tool-execution/builtin-run-log-excerpt/builtInRunLogExcerptProjection.d.ts" ], + "cluster-legacy-env-migration-plan": [ + "dist/migration/clusterLegacyEnvMigrationPlan.d.ts" + ], "run": [ "dist/run/run.d.ts" ], @@ -727,6 +730,11 @@ "require": "./dist/secret/secretReference.js", "default": "./dist/secret/secretReference.js" }, + "./cluster-legacy-env-migration-plan": { + "types": "./dist/migration/clusterLegacyEnvMigrationPlan.d.ts", + "require": "./dist/migration/clusterLegacyEnvMigrationPlan.js", + "default": "./dist/migration/clusterLegacyEnvMigrationPlan.js" + }, "./cluster-execution-revision": { "types": "./dist/task-definition/clusterExecutionRevision.d.ts", "require": "./dist/task-definition/clusterExecutionRevision.js", diff --git a/packages/ql3-runtime-core/src/migration/clusterLegacyEnvMigrationPlan.ts b/packages/ql3-runtime-core/src/migration/clusterLegacyEnvMigrationPlan.ts new file mode 100644 index 00000000..ade7b86d --- /dev/null +++ b/packages/ql3-runtime-core/src/migration/clusterLegacyEnvMigrationPlan.ts @@ -0,0 +1,434 @@ +import { createHash } from 'node:crypto'; + +import { parseSecretRef } from '../secret/secretReference'; + +export const CLUSTER_LEGACY_ENV_MIGRATION_PLAN_SCHEMA = + 'qinglong/cluster-legacy-env-migration-plan@v1' as const; +export const MAX_CLUSTER_LEGACY_ENV_SOURCE_ROWS = 100_000; +export const MAX_CLUSTER_LEGACY_ENV_TASKS = 100_000; +export const MAX_CLUSTER_LEGACY_ENV_TRIGGERS = 500_000; +export const MAX_CLUSTER_LEGACY_ENV_EFFECTIVE_BYTES = 64 * 1024; +export const MAX_CLUSTER_LEGACY_ENV_MIGRATION_PLAN_JSON_BYTES = 8 * 1024; + +export interface ClusterLegacyEnvMigrationSourceEvidence { + readonly reconciliationBundleDigest: string; + readonly decisionDigest: string; + readonly candidateSetDigest: string; + readonly sourceRowCount: number; + readonly activeRowCount: number; + readonly disabledRowCount: number; + readonly effectiveBindingCount: number; +} + +export interface ClusterLegacyEnvMigrationTarget { + readonly secretRef: string; + readonly taskRevisionSetDigest: string; + readonly triggerRevisionSetDigest: string; + readonly taskCount: number; + readonly triggerCount: number; + readonly totalEffectiveBytes: number; +} + +export interface ClusterLegacyEnvMigrationPlanIntent { + readonly planId: string; + readonly mutationId: string; + readonly projectId: string; + readonly source: Readonly; + readonly target: Readonly; +} + +export interface ClusterLegacyEnvMigrationPlan + extends ClusterLegacyEnvMigrationPlanIntent { + readonly schema: typeof CLUSTER_LEGACY_ENV_MIGRATION_PLAN_SCHEMA; + readonly plannedAtMs: number; + readonly planDigest: string; +} + +export interface ClusterLegacyEnvMigrationPlanRepository { + publish(intent: Readonly): Promise< + Readonly<{ + status: 'created' | 'existing'; + plan: Readonly; + }> + >; + findByPlanId( + planId: string, + ): Promise | null>; +} + +export class InvalidClusterLegacyEnvMigrationPlanError extends TypeError { + readonly code = 'CLUSTER_LEGACY_ENV_MIGRATION_PLAN_INVALID'; + + constructor(message: string) { + super(`Cluster Legacy Env migration plan is invalid: ${message}`); + this.name = 'InvalidClusterLegacyEnvMigrationPlanError'; + } +} + +export class ClusterLegacyEnvMigrationPlanConflictError extends Error { + readonly code = 'CLUSTER_LEGACY_ENV_MIGRATION_PLAN_CONFLICT'; + + constructor() { + super('Cluster Legacy Env migration plan conflicts with durable state'); + this.name = 'ClusterLegacyEnvMigrationPlanConflictError'; + } +} + +export class ClusterLegacyEnvMigrationPlanUnavailableError extends Error { + readonly code = 'CLUSTER_LEGACY_ENV_MIGRATION_PLAN_UNAVAILABLE'; + + constructor(options?: ErrorOptions) { + super('Cluster Legacy Env migration plan is unavailable', options); + this.name = 'ClusterLegacyEnvMigrationPlanUnavailableError'; + } +} + +const ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/; +const DIGEST_PATTERN = /^[0-9a-f]{64}$/; +const PLAN_DIGEST_DOMAIN = Buffer.from( + 'qinglong/cluster-legacy-env-migration-plan-digest@v1\0', + 'utf8', +); + +function invalid(message: string): never { + throw new InvalidClusterLegacyEnvMigrationPlanError(message); +} + +function record(value: unknown, label: string): Record { + if ( + !value || + typeof value !== 'object' || + Array.isArray(value) || + (Object.getPrototypeOf(value) !== Object.prototype && + Object.getPrototypeOf(value) !== null) + ) { + return invalid(`${label} must be an object`); + } + const descriptors = Object.getOwnPropertyDescriptors(value); + if ( + Object.values(descriptors).some( + (descriptor) => + descriptor.get !== undefined || + descriptor.set !== undefined || + descriptor.enumerable !== true, + ) + ) { + return invalid(`${label} must contain enumerable data properties`); + } + return value as Record; +} + +function exactKeys( + value: object, + expected: readonly string[], + label: string, +): void { + const keys = Reflect.ownKeys(value); + const canonical = [...expected].sort(); + if ( + keys.some((key) => typeof key !== 'string') || + keys.length !== canonical.length || + keys + .map(String) + .sort() + .some((key, index) => key !== canonical[index]) + ) { + invalid(`${label} shape is invalid`); + } +} + +export function assertClusterLegacyEnvMigrationPlanIdentifier( + value: unknown, + label: 'mutationId' | 'planId' | 'projectId', +): string { + if (typeof value !== 'string' || !ID_PATTERN.test(value)) { + return invalid(`${label} is invalid`); + } + return value; +} + +function digest(value: unknown, label: string): string { + if (typeof value !== 'string' || !DIGEST_PATTERN.test(value)) { + return invalid(`${label} is invalid`); + } + return value; +} + +function count(value: unknown, label: string, maximum: number): number { + if ( + !Number.isSafeInteger(value) || + (value as number) < 0 || + (value as number) > maximum + ) { + return invalid(`${label} is invalid`); + } + return value as number; +} + +function timestamp(value: unknown): number { + if (!Number.isSafeInteger(value) || (value as number) < 0) { + return invalid('plannedAtMs is invalid'); + } + return value as number; +} + +function sourceEvidence( + value: ClusterLegacyEnvMigrationSourceEvidence, +): Readonly { + const candidate = record(value, 'source'); + exactKeys( + candidate, + [ + 'activeRowCount', + 'candidateSetDigest', + 'decisionDigest', + 'disabledRowCount', + 'effectiveBindingCount', + 'reconciliationBundleDigest', + 'sourceRowCount', + ], + 'source', + ); + const sourceRowCount = count( + value.sourceRowCount, + 'sourceRowCount', + MAX_CLUSTER_LEGACY_ENV_SOURCE_ROWS, + ); + const activeRowCount = count( + value.activeRowCount, + 'activeRowCount', + MAX_CLUSTER_LEGACY_ENV_SOURCE_ROWS, + ); + const disabledRowCount = count( + value.disabledRowCount, + 'disabledRowCount', + MAX_CLUSTER_LEGACY_ENV_SOURCE_ROWS, + ); + const effectiveBindingCount = count( + value.effectiveBindingCount, + 'effectiveBindingCount', + MAX_CLUSTER_LEGACY_ENV_SOURCE_ROWS, + ); + if ( + sourceRowCount < 1 || + sourceRowCount !== activeRowCount + disabledRowCount || + activeRowCount < 1 || + effectiveBindingCount < 1 || + effectiveBindingCount > activeRowCount + ) { + return invalid('source row counts are inconsistent'); + } + return Object.freeze({ + reconciliationBundleDigest: digest( + value.reconciliationBundleDigest, + 'reconciliationBundleDigest', + ), + decisionDigest: digest(value.decisionDigest, 'decisionDigest'), + candidateSetDigest: digest(value.candidateSetDigest, 'candidateSetDigest'), + sourceRowCount, + activeRowCount, + disabledRowCount, + effectiveBindingCount, + }); +} + +function target( + value: ClusterLegacyEnvMigrationTarget, + projectId: string, +): Readonly { + const candidate = record(value, 'target'); + exactKeys( + candidate, + [ + 'secretRef', + 'taskCount', + 'taskRevisionSetDigest', + 'totalEffectiveBytes', + 'triggerCount', + 'triggerRevisionSetDigest', + ], + 'target', + ); + let parsedSecretRef; + try { + parsedSecretRef = parseSecretRef(value.secretRef); + } catch { + return invalid('secretRef is invalid'); + } + if ( + parsedSecretRef.projectId !== projectId || + parsedSecretRef.version === undefined + ) { + return invalid('secretRef must pin a version in the same Project'); + } + const taskCount = count( + value.taskCount, + 'taskCount', + MAX_CLUSTER_LEGACY_ENV_TASKS, + ); + if (taskCount < 1) return invalid('taskCount is invalid'); + const triggerCount = count( + value.triggerCount, + 'triggerCount', + MAX_CLUSTER_LEGACY_ENV_TRIGGERS, + ); + const totalEffectiveBytes = count( + value.totalEffectiveBytes, + 'totalEffectiveBytes', + MAX_CLUSTER_LEGACY_ENV_EFFECTIVE_BYTES, + ); + if (totalEffectiveBytes < 1) { + return invalid('totalEffectiveBytes is invalid'); + } + return Object.freeze({ + secretRef: value.secretRef, + taskRevisionSetDigest: digest( + value.taskRevisionSetDigest, + 'taskRevisionSetDigest', + ), + triggerRevisionSetDigest: digest( + value.triggerRevisionSetDigest, + 'triggerRevisionSetDigest', + ), + taskCount, + triggerCount, + totalEffectiveBytes, + }); +} + +export function normalizeClusterLegacyEnvMigrationPlanIntent( + value: ClusterLegacyEnvMigrationPlanIntent, +): Readonly { + const candidate = record(value, 'plan intent'); + exactKeys( + candidate, + ['mutationId', 'planId', 'projectId', 'source', 'target'], + 'plan intent', + ); + const projectId = assertClusterLegacyEnvMigrationPlanIdentifier( + value.projectId, + 'projectId', + ); + return Object.freeze({ + planId: assertClusterLegacyEnvMigrationPlanIdentifier( + value.planId, + 'planId', + ), + mutationId: assertClusterLegacyEnvMigrationPlanIdentifier( + value.mutationId, + 'mutationId', + ), + projectId, + source: sourceEvidence(value.source), + target: target(value.target, projectId), + }); +} + +function unsignedFields( + value: Omit, +): object { + return { + schema: value.schema, + planId: value.planId, + mutationId: value.mutationId, + projectId: value.projectId, + source: value.source, + target: value.target, + plannedAtMs: value.plannedAtMs, + }; +} + +export function clusterLegacyEnvMigrationPlanDigest( + value: Omit, +): string { + return createHash('sha256') + .update(PLAN_DIGEST_DOMAIN) + .update(JSON.stringify(unsignedFields(value)), 'utf8') + .digest('hex'); +} + +function bounded( + value: Readonly, +): Readonly { + if ( + Buffer.byteLength(JSON.stringify(value), 'utf8') > + MAX_CLUSTER_LEGACY_ENV_MIGRATION_PLAN_JSON_BYTES + ) { + return invalid('encoded plan exceeds the size limit'); + } + return value; +} + +export function createClusterLegacyEnvMigrationPlan( + intentValue: ClusterLegacyEnvMigrationPlanIntent, + plannedAtMsValue: number, +): Readonly { + const intent = normalizeClusterLegacyEnvMigrationPlanIntent(intentValue); + const unsigned = Object.freeze({ + schema: CLUSTER_LEGACY_ENV_MIGRATION_PLAN_SCHEMA, + ...intent, + plannedAtMs: timestamp(plannedAtMsValue), + }); + return bounded( + Object.freeze({ + ...unsigned, + planDigest: clusterLegacyEnvMigrationPlanDigest(unsigned), + }), + ); +} + +export function normalizeClusterLegacyEnvMigrationPlan( + value: ClusterLegacyEnvMigrationPlan, +): Readonly { + const candidate = record(value, 'plan'); + exactKeys( + candidate, + [ + 'mutationId', + 'planDigest', + 'planId', + 'plannedAtMs', + 'projectId', + 'schema', + 'source', + 'target', + ], + 'plan', + ); + if (value.schema !== CLUSTER_LEGACY_ENV_MIGRATION_PLAN_SCHEMA) { + return invalid('schema is invalid'); + } + const expected = createClusterLegacyEnvMigrationPlan( + { + planId: value.planId, + mutationId: value.mutationId, + projectId: value.projectId, + source: value.source, + target: value.target, + }, + value.plannedAtMs, + ); + if ( + typeof value.planDigest !== 'string' || + !DIGEST_PATTERN.test(value.planDigest) || + value.planDigest !== expected.planDigest + ) { + return invalid('planDigest does not match plan'); + } + return expected; +} + +export function clusterLegacyEnvMigrationPlanMatchesIntent( + planValue: ClusterLegacyEnvMigrationPlan, + intentValue: ClusterLegacyEnvMigrationPlanIntent, +): boolean { + const plan = normalizeClusterLegacyEnvMigrationPlan(planValue); + const intent = normalizeClusterLegacyEnvMigrationPlanIntent(intentValue); + return ( + plan.planId === intent.planId && + plan.mutationId === intent.mutationId && + plan.projectId === intent.projectId && + JSON.stringify(plan.source) === JSON.stringify(intent.source) && + JSON.stringify(plan.target) === JSON.stringify(intent.target) + ); +} diff --git a/packages/ql3-runtime-core/test/clusterLegacyEnvMigrationPlan.test.cjs b/packages/ql3-runtime-core/test/clusterLegacyEnvMigrationPlan.test.cjs new file mode 100644 index 00000000..bd88b29d --- /dev/null +++ b/packages/ql3-runtime-core/test/clusterLegacyEnvMigrationPlan.test.cjs @@ -0,0 +1,177 @@ +const assert = require('node:assert/strict'); +const { test } = require('node:test'); + +const { + CLUSTER_LEGACY_ENV_MIGRATION_PLAN_SCHEMA, + MAX_CLUSTER_LEGACY_ENV_EFFECTIVE_BYTES, + MAX_CLUSTER_LEGACY_ENV_SOURCE_ROWS, + MAX_CLUSTER_LEGACY_ENV_TASKS, + MAX_CLUSTER_LEGACY_ENV_TRIGGERS, + InvalidClusterLegacyEnvMigrationPlanError, + clusterLegacyEnvMigrationPlanMatchesIntent, + createClusterLegacyEnvMigrationPlan, + normalizeClusterLegacyEnvMigrationPlan, + normalizeClusterLegacyEnvMigrationPlanIntent, +} = require('@qinglong/runtime-core/cluster-legacy-env-migration-plan'); +const { createSecretRef } = require('@qinglong/runtime-core/secret-reference'); + +const DIGESTS = Object.freeze({ + reconciliationBundleDigest: '1'.repeat(64), + decisionDigest: '2'.repeat(64), + candidateSetDigest: '3'.repeat(64), + taskRevisionSetDigest: '4'.repeat(64), + triggerRevisionSetDigest: '5'.repeat(64), +}); + +function intent(overrides = {}) { + const projectId = overrides.projectId ?? 'project-a'; + return { + planId: 'legacy-env-plan-a', + mutationId: 'legacy-env-mutation-a', + projectId, + source: { + reconciliationBundleDigest: DIGESTS.reconciliationBundleDigest, + decisionDigest: DIGESTS.decisionDigest, + candidateSetDigest: DIGESTS.candidateSetDigest, + sourceRowCount: 4, + activeRowCount: 3, + disabledRowCount: 1, + effectiveBindingCount: 2, + }, + target: { + secretRef: createSecretRef({ + projectId, + name: 'legacy-env-bundle', + version: 7, + }), + taskRevisionSetDigest: DIGESTS.taskRevisionSetDigest, + triggerRevisionSetDigest: DIGESTS.triggerRevisionSetDigest, + taskCount: 2, + triggerCount: 3, + totalEffectiveBytes: 1024, + }, + ...overrides, + }; +} + +test('creates one frozen content-free migration plan with a stable digest', () => { + const value = createClusterLegacyEnvMigrationPlan(intent(), 12_345); + assert.equal(value.schema, CLUSTER_LEGACY_ENV_MIGRATION_PLAN_SCHEMA); + assert.match(value.planDigest, /^[0-9a-f]{64}$/); + assert.equal(Object.isFrozen(value), true); + assert.equal(Object.isFrozen(value.source), true); + assert.equal(Object.isFrozen(value.target), true); + assert.deepEqual(normalizeClusterLegacyEnvMigrationPlan(value), value); + assert.equal( + clusterLegacyEnvMigrationPlanMatchesIntent(value, intent()), + true, + ); + assert.deepEqual(Object.keys(value).sort(), [ + 'mutationId', + 'planDigest', + 'planId', + 'plannedAtMs', + 'projectId', + 'schema', + 'source', + 'target', + ]); + assert.equal(JSON.stringify(value).includes('legacy-env-bundle'), false); +}); + +test('requires a canonical version-pinned SecretRef in the same Project', () => { + const unversioned = intent(); + unversioned.target.secretRef = createSecretRef({ + projectId: unversioned.projectId, + name: 'legacy-env-bundle', + }); + assert.throws( + () => normalizeClusterLegacyEnvMigrationPlanIntent(unversioned), + InvalidClusterLegacyEnvMigrationPlanError, + ); + + const crossProject = intent(); + crossProject.target.secretRef = createSecretRef({ + projectId: 'project-b', + name: 'legacy-env-bundle', + version: 1, + }); + assert.throws( + () => normalizeClusterLegacyEnvMigrationPlanIntent(crossProject), + InvalidClusterLegacyEnvMigrationPlanError, + ); +}); + +test('enforces source consistency and router-safe bounded targets', () => { + const invalidValues = [ + { source: { ...intent().source, sourceRowCount: 5 } }, + { + source: { + ...intent().source, + sourceRowCount: MAX_CLUSTER_LEGACY_ENV_SOURCE_ROWS + 1, + }, + }, + { + target: { + ...intent().target, + taskCount: MAX_CLUSTER_LEGACY_ENV_TASKS + 1, + }, + }, + { + target: { + ...intent().target, + triggerCount: MAX_CLUSTER_LEGACY_ENV_TRIGGERS + 1, + }, + }, + { + target: { + ...intent().target, + totalEffectiveBytes: MAX_CLUSTER_LEGACY_ENV_EFFECTIVE_BYTES + 1, + }, + }, + ]; + for (const invalidValue of invalidValues) { + assert.throws( + () => normalizeClusterLegacyEnvMigrationPlanIntent(intent(invalidValue)), + InvalidClusterLegacyEnvMigrationPlanError, + ); + } +}); + +test('rejects widened shapes, accessors, symbols and tampered durable digests', () => { + const widened = intent(); + widened.source.envName = 'TOKEN'; + assert.throws( + () => normalizeClusterLegacyEnvMigrationPlanIntent(widened), + InvalidClusterLegacyEnvMigrationPlanError, + ); + + const symbol = intent(); + symbol.target[Symbol('secretValue')] = 'not-storable'; + assert.throws( + () => normalizeClusterLegacyEnvMigrationPlanIntent(symbol), + InvalidClusterLegacyEnvMigrationPlanError, + ); + + const accessor = intent(); + Object.defineProperty(accessor.source, 'sourceRowCount', { + enumerable: true, + get() { + return 4; + }, + }); + assert.throws( + () => normalizeClusterLegacyEnvMigrationPlanIntent(accessor), + InvalidClusterLegacyEnvMigrationPlanError, + ); + + const plan = createClusterLegacyEnvMigrationPlan(intent(), 12_345); + assert.throws( + () => + normalizeClusterLegacyEnvMigrationPlan({ + ...plan, + planDigest: 'f'.repeat(64), + }), + InvalidClusterLegacyEnvMigrationPlanError, + ); +}); diff --git a/test/back/ql3PackageBoundaryAudit.test.cjs b/test/back/ql3PackageBoundaryAudit.test.cjs index 22ff7aad..9e2c57e3 100644 --- a/test/back/ql3PackageBoundaryAudit.test.cjs +++ b/test/back/ql3PackageBoundaryAudit.test.cjs @@ -299,10 +299,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: runtimeCore.rootSourceFileRoles, }, { - sourceFiles: 171, + sourceFiles: 172, rootSourceFiles: 1, rootSourceLines: 160, - nestedSourceFiles: 170, + nestedSourceFiles: 171, rootSourceFileRoles: { 'index.ts': 'public_export' }, }, ); @@ -421,10 +421,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: clusterPostgres.rootSourceFileRoles, }, { - sourceFiles: 175, + sourceFiles: 177, rootSourceFiles: 1, rootSourceLines: 126, - nestedSourceFiles: 174, + nestedSourceFiles: 176, rootSourceFileRoles: { 'index.ts': 'public_export' }, }, );