diff --git a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md index 58f1fa0e..fee0e2dd 100644 --- a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md +++ b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md @@ -11,6 +11,26 @@ 最新增量证据（2026-08-21）： +- D-387/ADR-0480（已接受）：把 D-386 prepared model 推进为受认证、Project-scoped、可审计且可恢复的原子 application。 + 既有 `ql3-adoption` 增加 exact `local-data-directory.adoption.apply|apply.verify`；Owner credential 通过独立短生命周期 SQLite handle 建立 + `local_data_adoption` 强认证并要求目标 Project 的 `secret.manage`，publisher 在 `BEGIN IMMEDIATE` 内再次复验 credential、Project 与 + RoleBinding fence。SQLite contract v50 的 `0099/0100` 增加父 adoption ledger、逐 Secret binding、trigger、typed schema、readiness + relation 和 `legacy_data_directory_adoption=1` capability；所有 AES-256-GCM Secret envelope、逐项 `secret.create` audit、父 + `legacy-data.apply` audit、disabled model 与 canonical receipt 在一个事务提交，任一目标冲突整批回滚。DB COMMIT 是逻辑提交点；随后 + `.commit-incomplete → model rename 为 .reclaiming-model → Secret 文件覆盖/fsync/unlink → content-free commit.json`，exact apply replay + 可从 durable receipt 收敛 COMMIT-response-loss 和清理中断，`apply.verify` 只验证完整终态。最终明确 + `physicalErasureGuaranteed=false`，不把闪存/CoW overwrite 冒充介质擦除，也不激活配置、SSH、Task 或 service cutover。实现内聚在既有 + Local Owner/Admin/SQLite 的 data-directory 子目录，不新增 package、依赖、binary、daemon、listener、timer、watcher、网络或常驻资源； + Edge/Standalone 继续使用 128/512 Secret 上限。focused data-directory `13/13`、Local SQLite `236/236`、Local Owner + `208 total / 203 pass / 5 conditional skip / 0 fail` 与 backend `1,535 total / 1,533 pass / 2 conditional skip / 0 fail` 已通过，覆盖解密一致、 + stdout 脱敏、exact replay/verify、renamed-model 崩溃恢复、第二个 Secret 冲突的整批回滚和 viewer 拒绝零发布。18-package clean build/逐包测试、 + schema readiness v50/100 migrations/83 tables、八项架构审计、本地镜像审计和十四档 artifact audit 全 compatible；workspace 仍为 18 packages， + `singleSourcePackages=[]`、`shallowSourcePackages=[]`，Local Owner 为 `131 source / 130 nested / 1 root binary entry`。基础 Edge/Standalone + `2,611,978 / 2,612,056` bytes、319 files、58 modules，Adopted `2,831,713 / 2,831,836` bytes、339 files、59 modules，Application+AI + `4,515,571 / 4,515,703` bytes、514 files、142 modules，MCP `7,337,910 / 7,338,018` bytes、805 files、228 modules，均未放宽预算。 + GitNexus staged change audit 只允许本切片的预期存储/readiness/application 影响；`next` 相对 `develop` 的全分支 CRITICAL 差异另作为 3.0 + 孵化累计风险保留，不能归因于 D-387。本切片不改变 PostgreSQL schema、ACL、role、Pool、连接或 HA 拓扑，因此不重新占有 PostgreSQL HA + 证明。D-388 应把 committed receipt 接入 systemd/OpenRC/Compose deployment lineage。 - D-386/ADR-0479（已接受）：把 D-385 的 `config/db/ssh.d` 私有 snapshot 转成 no-replace、版本化、Project-bound 的 prepared model， 新增 exact `local-data-directory.adoption.transform|transform.verify`。`config.sh` 永不执行，只把简单非空 export 写入独立 `qinglong3-local-secret-value` 文件，非 export setting 退役，复杂/重复行转人工复核；Keyv 以 read-only defensive SQLite 和 reviewed diff --git a/docs/adr/ADR-0480-authenticated-atomic-legacy-data-application.md b/docs/adr/ADR-0480-authenticated-atomic-legacy-data-application.md new file mode 100644 index 00000000..1e5a4e5b --- /dev/null +++ b/docs/adr/ADR-0480-authenticated-atomic-legacy-data-application.md @@ -0,0 +1,150 @@ +# ADR-0480：受认证、原子且可恢复的 Legacy Data Application + +- 状态：Accepted +- 日期：2026-08-21 +- 关联：QL-RFC-0001、ADR-0207、ADR-0395、ADR-0476、ADR-0478、ADR-0479 + +## 上下文 + +ADR-0479 只生成 `activation=disabled` 的 prepared model。模型内含待导入 Secret 明文，因此不能长期作为运行态事实，也不能由转换器顺便 +写入目标数据库。应用阶段还必须同时解决四个问题：目标 Project 的强认证与授权、多个 Secret 和模型收据的原子提交、数据库提交后的 +明文回收，以及进程在数据库 COMMIT 与文件清理之间崩溃时的确定性恢复。 + +QingLong 3.0 同时面向低配路由设备和集群节点。本地流程不能为了迁移增加 daemon、队列或第二套数据库；Cluster 又不能因本地简化而 +失去 Project/RoleBinding fence、审计和幂等事实。SQLite 事务只能原子覆盖数据库，不能原子覆盖文件系统，因此协议必须明确承认并闭合 +这个边界，而不是宣称跨介质事务。 + +## 决策 + +### 1. 延续既有领域，不新增 workspace package + +既有 `ql3-adoption` 增加两个 exact、私有 command-file operation： + +- `local-data-directory.adoption.apply`； +- `local-data-directory.adoption.apply.verify`。 + +职责分别内聚在 `@qinglong/local-owner-cli/lifecycle/data-directory-adoption/application`、 +`@qinglong/local-admin/data-directory-adoption` 和 `@qinglong/local-sqlite/adoption/data-directory`。CLI 管文件 ceremony，Admin 管认证后的业务 +组合，SQLite adapter 管唯一写事务。它们不是独立部署单元，不按源码文件拆成微包。 + +### 2. 强认证和 Project Policy 是提交前提 + +命令只接受 deployment root 内的 Owner pepper keyring、credential presentation 和 Local Secret keyring 路径；敏感 credential 不进入 +command JSON 或 stdout。Owner credential 通过既有本地强认证建立 `local_data_adoption` authority，随后以目标 `projectId` 请求 +`secret.manage`。只有 `owner|admin` 的 active RoleBinding 可继续；publisher 在 `BEGIN IMMEDIATE` 内再次校验 credential、Project 和 +RoleBinding fence，消除预检到提交之间的权限漂移。 + +失败认证与拒绝授权写低敏 `legacy-data.apply` 审计，但不会读取 prepared Secret 或写入 adoption/Secret 表。 + +### 3. 一个 SQLite 事务提交完整批次 + +SQLite contract 从 v49 升到 v50，迁移 `0099/0100` 增加： + +- `QingLong3LegacyDataDirectoryAdoptions`：唯一 mutation、transformation、disabled model、publication digest、receipt 和提交时间； +- `QingLong3LegacyDataDirectoryAdoptionSecrets`：按 ordinal 绑定每个来源名称摘要、目标 Secret、value digest、Secret envelope 和 audit； +- trigger、FK、唯一索引、typed schema、readiness relation 和 `legacy_data_directory_adoption=1` capability。 + +publisher 在一个 `BEGIN IMMEDIATE` 中依次验证 exact replay、Project/RoleBinding fence、所有目标 Secret 当前不存在，再写 encrypted Secret +envelope、每个 `secret.create` audit、父 `legacy-data.apply` audit、disabled model、子项和 canonical receipt。任一目标冲突、外部 authority +复验失败或约束失败都会回滚整个批次，不允许“部分 Secret 已导入”。同一 mutation 的 exact replay 读取 durable receipt，不再次读取明文或 +创建新版本；同一 transformation 不能用另一 mutation 重复提交。 + +### 4. 只应用 `ready` 模型，永不顺带激活 + +apply 在数据库 mutation 前后重新验证 D-385 stage、SQLite activation 和 D-386 transformation binding。`manual_required`、目标漂移、源 +漂移、Project/profile/digest 不匹配都失败关闭。写入的 config、Keyv、SSH 和 manual-review 模型仍固定 `activation=disabled`；本 ADR +不授权执行旧 shell、恢复 `authInfo`、启用 SSH host、启动任务或切换服务。 + +Secret 明文只用于当前进程内的 AES-256-GCM envelope 生成。每个 Secret 使用批次 mutation 和目标 identity 推导的稳定 v4 mutation ID， +并绑定独立 `secret.create` audit。key material 使用后立即覆盖；JavaScript 字符串的运行时内存擦除不作不可验证的保证。 + +### 5. 数据库 receipt 是恢复权威 + +数据库 COMMIT 是逻辑提交点。提交后 CLI 在 transformation root 写 `.commit-incomplete`，把 `model` 原子 rename 为 +`.reclaiming-model`，逐个以 no-follow、owner/mode/inode 检查打开 Secret value 文件，覆盖、fsync、unlink，再删除其余 prepared model +文件。完成后写内容无关的 `commit.json`，最后删除 marker。成功根目录固定为： + +```text +transformationRoot/ + manifest.json + commit.json +``` + +如果进程在 COMMIT 后崩溃，同一 exact apply 先解析数据库 receipt，不依赖已被部分删除的 model；它只接受与 receipt 精确匹配的 marker 和 +`model|.reclaiming-model` 状态，然后继续清理并重建同一 `commit.json`。未知文件、双 model、marker 漂移或 commit 漂移全部失败关闭，保留 +现场供 operator 诊断。`apply.verify` 只验证完整终态，不隐式修复；恢复必须显式重放原 apply。 + +### 6. 明文回收不冒充物理擦除 + +协议保证 prepared model 中的明文文件被逻辑覆盖并 unlink，最终 transformation root 不再含 model 或 Secret value 文件。由于 SSD、闪存 +FTL、CoW、快照、备份和宿主文件系统可能保留旧块，receipt 固定声明 `physicalErasureGuaranteed=false`。需要物理销毁保证的部署者必须使用 +加密卷、销毁卷密钥或设备级擦除流程；不能把本地 overwrite 结果写成合规证明。 + +### 7. Profile 资源边界与部署边界 + +Edge/Standalone 继续使用 D-386 的 128/512 Secret 上限；单条 envelope 最高 16 KiB。事务只打开一个短生命周期 SQLite writer，认证使用 +独立短生命周期 handle 以避免嵌套 operation authority；没有网络、timer、watcher、后台 retry、缓存或常驻内存。Cluster 后续可以复用 +prepared-model/receipt 语义，但必须以 PostgreSQL、separation-of-duty 和 Cluster 专用 authority 实现，不能把本地 credential ceremony +直接搬进控制面。 + +## 被拒绝的替代方案 + +### 每个 Secret 独立调用现有 `secret.put` + +拒绝。第二个 Secret 冲突会留下第一个已提交，无法证明批次与 transformation receipt 一致。 + +### 先删除明文，再提交数据库 + +拒绝。数据库失败后会失去唯一待导入值，D-385 staging 也不能安全地代替已审核 transformation。 + +### 用文件锁或后台 daemon 模拟跨介质事务 + +拒绝。文件锁不能回滚 SQLite COMMIT；daemon 会给路由设备增加常驻资源和长期高权限。durable DB receipt 加调用方显式重放已经能闭合崩溃窗。 + +### 成功 apply 后直接启用配置、SSH 或服务 + +拒绝。数据提交、风险复核、运行时 activation 和 service cutover 是不同 authority；合并会让迁移 credential 获得过宽副作用。 + +### 声称 overwrite 等于安全物理擦除 + +拒绝。对现代闪存、CoW 和快照不可证明，会产生错误的安全与合规承诺。 + +## 影响 + +### 正面 + +- prepared model 首次具有强认证、Project-scoped、可审计的最终数据库提交路径； +- Secret、子审计、disabled model 和 receipt 同事务提交，目标冲突不会留下部分状态； +- COMMIT-response-loss 和清理中断可用同一命令收敛，不依赖残存明文； +- 成功后高敏 model 被回收，stdout、manifest、commit 和 receipt 只保留低敏摘要； +- workspace package 数量和低配常驻拓扑保持不变。 + +### 代价与限制 + +- SQLite contract 升到 v50，旧 v49 binary 不能写新库，部署镜像 label/readiness 必须同步升级； +- DB COMMIT 后、文件清理前的命令可能以失败退出，但 durable receipt 表明逻辑提交已发生，operator 必须重放同一 apply； +- 逻辑明文回收不保证介质级擦除； +- 本 ADR 不迁移 `scripts/upload` 到最终 runtime，不完成 service cutover，也不实现 Cluster 数据提交。 + +## 验证结果 + +- focused data-directory `13/13`，覆盖成功提交、Secret 解密一致、stdout 脱敏、exact apply/apply.verify replay、COMMIT 后 renamed-model 恢复、 + 第二个 Secret 冲突整批回滚和 viewer 拒绝零发布； +- Local SQLite `236/236`，覆盖 v50 migration manifest、typed schema、readiness、unknown table drift 和既有 repository 回归；schema readiness + 以 mode-0600 Edge 数据库实测为 contract v50、100 migrations、83 tables、SQLite 3.53.3、DELETE journal； +- Local Owner 为 `208 total / 203 pass / 5 conditional skip / 0 fail`；backend 为 + `1,535 total / 1,533 pass / 2 conditional skip / 0 fail`；`pnpm build:back` 和 18-package clean build/逐包测试退出 0； +- package boundary 保持 18 packages、`singleSourcePackages=[]`、`shallowSourcePackages=[]`；Local Admin、Local Owner、Local SQLite 新源码均进入 + 既有领域子目录，Local Owner 为 `131 source / 130 nested / 1 root binary entry`； +- 八项架构审计、本地镜像审计和十四档 artifact audit 全 compatible；基础 Edge/Standalone 为 319 files/58 modules，最小 artifact + 2,611,978 bytes；Application+AI 为 514 files/142 modules，最大 4,515,703 bytes；MCP 为 805 files/228 modules，最大 7,338,018 bytes， + 全部保留既有预算； +- GitNexus `compare develop` 如实返回整条 `next` 孵化分支的 CRITICAL 累计差异；提交前另以 staged diff 收窄并核验 D-387 的预期符号与流程影响， + 不用全分支风险替代切片审计； +- 本切片不修改 PostgreSQL schema、ACL、role、Pool、连接或 HA 拓扑，不重新占有 PostgreSQL HA 证明。 + +## 后续 + +- D-388：把 committed transformation receipt 接入 systemd/OpenRC/Compose deployment lineage，并保持 activation 与 rollback 独立授权； +- 在固定物理 Edge/NAS 上执行 RSS、I/O、磁盘峰值、ENOSPC、断电和加密卷销毁演练； +- 为 Cluster 定义 PostgreSQL/SoD 版本的 prepared-model application，而不是复用本地 Owner credential 文件。 diff --git a/docs/adr/README.md b/docs/adr/README.md index 80b9ae9b..3e1999ed 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -483,6 +483,7 @@ | [ADR-0477](./ADR-0477-bounded-legacy-data-directory-inventory.md) | 有界 Legacy Data Directory 盘点 | Accepted | | [ADR-0478](./ADR-0478-private-legacy-data-directory-staging.md) | 私有 Legacy Data Directory 暂存与稳定校验 | Accepted | | [ADR-0479](./ADR-0479-private-versioned-legacy-config-keyv-ssh-transformation.md) | 私有、版本化的 Legacy Config、Keyv 与 SSH 转换 | Accepted | +| [ADR-0480](./ADR-0480-authenticated-atomic-legacy-data-application.md) | 受认证、原子且可恢复的 Legacy Data Application | Accepted | ## 规则 diff --git a/docs/operations/ql3-local-data-directory-adoption.md b/docs/operations/ql3-local-data-directory-adoption.md index 344d507e..89510947 100644 --- a/docs/operations/ql3-local-data-directory-adoption.md +++ b/docs/operations/ql3-local-data-directory-adoption.md @@ -222,17 +222,90 @@ Secret 明文，必须按高敏迁移材料保护，不能写入日志、提交 成功返回 `status=verified`，evidence 应与 transform 完全相同。verify 会在目标校验前后重新验证 D-385 staging、当前 2.x source 和 SQLite activation，并检查目标 exact 文件集、私有 mode、Project/profile/path binding、Secret value digest 和完整模型树摘要。 -## 10. 崩溃残留与恢复 +## 10. 受认证提交 prepared model + +只有 transform 返回 `assessment=ready`、transform.verify 成功，且目标 Project 已存在 active Owner/Admin RoleBinding 时才能 apply。先准备： + +- deployment root 内 `0700` 的 Owner pepper keyring； +- deployment root 内 `0600` 的 Owner credential presentation； +- deployment root 内 `0600` 的 Local Secret keyring； +- 两个不同的 UUID v4：`mutationId` 与失败审计 `failureAuditEventId`。 + +credential token 和 Secret key material 不写入 command JSON。提交命令是在 transform.verify options 上增加以下 authority： + +```json +{ + "schemaVersion": 1, + "operation": "local-data-directory.adoption.apply", + "options": { + "deploymentRoot": "/opt/qinglong3/adoption", + "dataRoot": "/opt/qinglong/data", + "stagingRoot": "/opt/qinglong3/adoption/staging/reviewed-data", + "transformationRoot": "/opt/qinglong3/adoption/transformations/reviewed-data-v1", + "projectId": "", + "profile": "edge", + "expectedManifestDigest": "<64-hex-directory-manifest-digest>", + "expectedTransformationDigest": "<64-hex-transformation-digest>", + "ownerPepperKeyringDirectory": "/opt/qinglong3/adoption/owner-keys", + "credentialFilePath": "/opt/qinglong3/adoption/owner-credential.json", + "secretKeyringPath": "/opt/qinglong3/adoption/local-secret-keyring.json", + "mutationId": "", + "failureAuditEventId": "", + "requestId": "legacy-data-apply-20260821", + "sqlite": { + "sourcePath": "/opt/qinglong/data/db/database.sqlite", + "targetPath": "/opt/qinglong3/adoption/sqlite/qinglong3.sqlite", + "recoveryPath": "/opt/qinglong3/adoption/sqlite/database.pre-ql3.sqlite", + "manifestPath": "/opt/qinglong3/adoption/sqlite/adoption.json", + "activationPath": "/opt/qinglong3/adoption/sqlite/activation.json", + "expectedActivationDigest": "<64-hex-sqlite-activation-digest>" + } + } +} +``` + +apply 会以 Owner credential 建立强认证，要求目标 Project 的 `secret.manage`，并在一个 SQLite `BEGIN IMMEDIATE` 中提交所有加密 Secret、 +逐项 `secret.create` 审计、父 `legacy-data.apply` 审计、disabled model 和 canonical receipt。任何一个目标 Secret 已存在都会回滚整个批次。 +成功返回 `status=committed`、`databaseStatus=inserted` 和低敏 digest/count evidence，不返回 Secret 名称、Project ID 或明文。 + +成功后 transformation root 只剩 `manifest.json` 与 `commit.json`；`model/` 和其中的明文文件已逻辑覆盖并删除。结果固定声明 +`physicalErasureGuaranteed=false`：闪存 FTL、CoW、快照或备份可能保留旧块。需要介质级保证时应使用加密卷并销毁卷密钥或执行设备级擦除。 + +apply 只把模型持久化为 `activation=disabled`，不会启用 SSH、执行配置、启动任务或切换服务。 + +## 11. 校验 committed application + +保存 apply 返回的 `receiptDigest`。将原命令 operation 改为 `local-data-directory.adoption.apply.verify`，并在 options 增加： + +```json +{ + "expectedReceiptDigest": "<64-hex-receipt-digest>" +} +``` + +verify 使用相同强认证和 exact mutation，读取 durable receipt，验证 transformation manifest 与最终 `commit.json`，但不修复中间状态。成功返回 +`status=verified`、`databaseStatus=existing`。receipt、commit、Project/profile/transformation 或 root 文件集有任何漂移都会失败关闭。 + +## 12. 崩溃残留与恢复 stage 和 transform 都在创建目标根后立即写入 `.incomplete`。只有 payload/model 和 `manifest.json` 都持久化且静态校验通过后才删除。 -命令失败或进程崩溃后： +这些阶段失败或进程崩溃后： - 不要直接把残留目录当作恢复资产； - 不要在原路径重试，stage/transform 会 no-replace 拒绝； - 先保存现场用于诊断，再由 operator 显式移走残留目录，并使用一个新的空路径重试； - verify 遇到 `.incomplete`、额外文件或缺失文件一律失败关闭。 -## 11. 常见失败 +apply 的数据库 COMMIT 是逻辑提交点。COMMIT 后清理使用 `.commit-incomplete` 和 `.reclaiming-model`。若 apply 失败且 transformation root 出现 +这些条目： + +- 不要重新 transform，也不要手工删除 marker/model； +- 查询错误和目标数据库 readiness，保留完整诊断现场； +- 使用完全相同的 command、mutation ID 和 digest 重放 `local-data-directory.adoption.apply`； +- exact replay 从 durable receipt 继续回收 model 并重建同一 `commit.json`，不会再次创建 Secret； +- 未知文件、marker 漂移、双 model 或 receipt 不一致必须人工调查，不能用 apply.verify 或手改文件绕过。 + +## 13. 常见失败 - 根目录或条目 group/world 可写：修正 ownership/permission 后重新盘点； - symlink、硬链接或特殊文件：保留现场，确认来源和目标后人工处置；盘点不会跟随或读取； @@ -246,16 +319,22 @@ stage 和 transform 都在创建目标根后立即写入 `.incomplete`。只有 - Keyv schema、row 或字节超限：不要把数据库当 JSON dump 绕过，先确认版本和未知数据归属； - `manual_required`：从 D-385 staging 审核相应原始输入；不要执行旧 shell、复用 `authInfo` 或启用旧 SSH config； - transform verify drift：停止后续 apply，保留 D-385 staging 和 transformation root，重新建立新的版本化转换路径。 +- credential rejected：核对 credential presentation、pepper keyring、有效期和目标数据库绑定，不要把 token 填进 command JSON； +- `APPLICATION_FORBIDDEN`：当前主体不是目标 Project 的 active owner/admin；通过正式 Policy 管理流程修复 RoleBinding 后使用新失败审计 ID重试； +- `ADOPTION_CONFLICT`：目标 Secret、mutation 或 transformation 已存在不一致事实；批次已回滚，禁止删除现有 Secret 强行导入； +- apply 在数据库 COMMIT 后清理失败：重放完全相同的 apply；不要生成新 mutation，否则会与 transformation 唯一约束冲突； +- apply.verify 报终态不完整：它不会恢复；改为重放原 apply。 -## 12. 当前边界 +## 14. 当前边界 -本流程已经提供 inspect、私有 stage、版本化 transform 和两级稳定 verify。它仍不： +本流程已经提供 inspect、私有 stage、版本化 transform、受认证原子 apply、崩溃恢复和三级稳定 verify。它仍不： - 删除或修改任何源资产； -- 把 prepared Secret/settings/SSH binding 写入或激活到 3.0 目标数据库； +- 激活已提交的 settings/SSH binding、启动任务或切换服务； - 把历史日志/备份复制到默认目标，或复用跨架构 repo/dependency cache； - 授权 service-manager/Compose cutover 或 Legacy rollback； - 证明固定物理路由器/NAS 上的耗时、RSS、I/O、磁盘峰值和断电恢复。 -即使 transform 返回 `status=prepared` 且 transform.verify 返回 `status=verified`，仍不是 apply 或 cutover 授权。继续保留原始 2.x -data directory、SQLite recovery、D-385 staging 和两份 manifest，等待受认证 apply/commit 与部署 lineage 完成。 +apply 成功只证明目标数据库中的 encrypted Secret、disabled model、audit 和 receipt 原子一致，并且 prepared 明文文件已逻辑回收；它仍不是 +cutover 或 activation 授权。继续保留原始 2.x data directory、SQLite recovery、D-385 staging、manifest 和 `commit.json`，直到独立部署 +lineage 与回滚门完成。 diff --git a/docs/ql3-package-boundaries.json b/docs/ql3-package-boundaries.json index 3620c161..7a1eeadf 100644 --- a/docs/ql3-package-boundaries.json +++ b/docs/ql3-package-boundaries.json @@ -12,7 +12,7 @@ }, { "kind": "ordered_ledger", - "maxDirectSourceFiles": 99, + "maxDirectSourceFiles": 101, "path": "packages/ql3-local-sqlite/src/migrations", "rationale": "SQLite migrations are an append-only version ledger whose ordering and discoverability are safer in one reviewed directory." } @@ -55,12 +55,12 @@ "authorities": [ "Kubernetes mutation", "PostgreSQL management", - "optional mTLS/OIDC human Approval management", - "caller-driven Approval management client", - "loopback-only read-only Copilot Console", - "one-shot maintenance" + "optional mTLS/OIDC human Approval management", + "caller-driven Approval management client", + "loopback-only read-only Copilot Console", + "one-shot maintenance" ], - "rationale": "独立管理进程、一次性高权限命令和 operator-workstation Console 必须排除在常驻 cluster-control 之外；Approval 的 service/transport/process/client 与只读 Copilot Console 均属于同一 Cluster Admin 制品，分别使用内聚领域目录而不是新增微包。Console 的宿主启动器、配置模板与发布验签脚本随已签名的多架构 Admin OCI 分发，不形成第 19 个 workspace package 或第二套依赖树。" + "rationale": "独立管理进程、一次性高权限命令和 operator-workstation Console 必须排除在常驻 cluster-control 之外；Approval 的 service/transport/process/client 与只读 Copilot Console 均属于同一 Cluster Admin 制品，分别使用内聚领域目录而不是新增微包。Console 的宿主启动器、配置模板与发布验签脚本随已签名的多架构 Admin OCI 分发，不形成第 19 个 workspace package 或第二套依赖树。" }, { "path": "packages/ql3-cluster-control", diff --git a/packages/ql3-local-admin/package.json b/packages/ql3-local-admin/package.json index 88348bfd..9af79f7e 100644 --- a/packages/ql3-local-admin/package.json +++ b/packages/ql3-local-admin/package.json @@ -115,6 +115,11 @@ "require": "./dist/security-administration/secretAdministration.js", "default": "./dist/security-administration/secretAdministration.js" }, + "./data-directory-adoption": { + "types": "./dist/data-directory-adoption/dataDirectoryAdoption.d.ts", + "require": "./dist/data-directory-adoption/dataDirectoryAdoption.js", + "default": "./dist/data-directory-adoption/dataDirectoryAdoption.js" + }, "./task-definition-administration": { "types": "./dist/automation-administration/taskDefinitionAdministration.d.ts", "require": "./dist/automation-administration/taskDefinitionAdministration.js", diff --git a/packages/ql3-local-admin/src/data-directory-adoption/dataDirectoryAdoption.ts b/packages/ql3-local-admin/src/data-directory-adoption/dataDirectoryAdoption.ts new file mode 100644 index 00000000..25e4bdfc --- /dev/null +++ b/packages/ql3-local-admin/src/data-directory-adoption/dataDirectoryAdoption.ts @@ -0,0 +1,547 @@ +import { createHash } from 'node:crypto'; + +import { + createLocalDataDirectoryAdoptionReceipt, + createLocalDataDirectoryAdoptionSecretItem, + createLocalDataDirectorySourceNameDigest, + openLocalSqliteDataDirectoryAdoptionDatabase, + type LocalDataDirectoryAdoptionRecord, + type LocalDataDirectoryAdoptionSecretPublication, + type LocalDataDirectoryAppliedModel, +} from '@qinglong/local-sqlite/data-directory-adoption'; +import { + encryptLocalSecretEnvelope, + ownedLocalSecretKeyMaterial, +} from '@qinglong/local-secret'; +import { + LOCAL_SECRET_ALGORITHM, + assertLocalSecretName, + assertLocalSecretPlaintext, + assertLocalSecretProjectId, + type LocalSecretKeyProvider, +} from '@qinglong/runtime-core/local-secret'; +import { + ProjectPolicyEngine, + ProjectPolicyUnavailableError, +} from '@qinglong/runtime-core/project-policy'; +import { + normalizeSecurityPrincipal, + type SecurityPolicyDecision, + type SecurityPrincipal, +} from '@qinglong/runtime-core/security'; +import { + normalizeSecurityAuditRecord, + type SecurityAuditRecord, +} from '@qinglong/runtime-core/security-audit'; + +const DIGEST_PATTERN = /^[0-9a-f]{64}$/; +const UUID_V4_PATTERN = + /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/; +const REQUEST_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/; +const VALUE_FILE_PATTERN = /^secret-values\/[0-9a-f]{64}\.json$/; +const STRONG_USER_ASSURANCES = new Set([ + 'multi_factor', + 'hardware', + 'local_console', +]); + +export interface PreparedLocalDataDirectoryAdoptionSecret { + readonly kind: 'environment' | 'ssh_private_key'; + readonly sourceName: string; + readonly targetName: string; + readonly expectedCurrentVersion: 0; + readonly valueFile: string; + readonly valueDigest: string; + readonly plaintext: string; +} + +export interface PreparedLocalDataDirectoryAdoptionModel { + readonly model: Readonly; + readonly secrets: readonly Readonly[]; +} + +export interface ApplyReviewedLocalDataDirectoryAdoptionOptions { + readonly databasePath: string; + readonly profile: 'edge' | 'standalone'; + readonly projectId: string; + readonly mutationId: string; + readonly failureAuditEventId: string; + readonly requestId: string; + readonly sourceStageManifestDigest: string; + readonly transformationDigest: string; + readonly modelDigest: string; + readonly principal: Readonly; + readonly keyProvider: LocalSecretKeyProvider; + readonly observedAtMs: number; + readonly busyTimeoutMs?: number; + readonly loadPreparedModel: () => + | Readonly + | Promise>; + readonly confirmAuthenticationAuthority: () => void | Promise; + readonly confirmPreparedAuthority: () => void | Promise; +} + +export interface ApplyReviewedLocalDataDirectoryAdoptionResult { + readonly status: 'inserted' | 'existing'; + readonly adoption: Readonly; +} + +export class LocalDataDirectoryAdoptionApplicationConfigurationError extends TypeError { + readonly code = 'LOCAL_DATA_DIRECTORY_ADOPTION_APPLICATION_INVALID'; + + constructor(message: string, readonly cause?: unknown) { + super(`Local data directory adoption application is invalid: ${message}`); + this.name = 'LocalDataDirectoryAdoptionApplicationConfigurationError'; + } +} + +export class LocalDataDirectoryAdoptionApplicationAuthenticationError extends Error { + readonly code = + 'LOCAL_DATA_DIRECTORY_ADOPTION_APPLICATION_AUTHENTICATION_REQUIRED'; + + constructor() { + super( + 'Local data directory adoption application requires a strong principal', + ); + this.name = 'LocalDataDirectoryAdoptionApplicationAuthenticationError'; + } +} + +export class LocalDataDirectoryAdoptionApplicationAuthorizationError extends Error { + readonly code = 'LOCAL_DATA_DIRECTORY_ADOPTION_APPLICATION_FORBIDDEN'; + + constructor() { + super('Local data directory adoption application is not authorized'); + this.name = 'LocalDataDirectoryAdoptionApplicationAuthorizationError'; + } +} + +export class LocalDataDirectoryAdoptionApplicationUnavailableError extends Error { + readonly code = 'LOCAL_DATA_DIRECTORY_ADOPTION_APPLICATION_UNAVAILABLE'; + + constructor(readonly cause?: unknown) { + super('Local data directory adoption application is unavailable'); + this.name = 'LocalDataDirectoryAdoptionApplicationUnavailableError'; + } +} + +function exactKeys(value: object, expected: readonly string[]): boolean { + const actual = Object.keys(value).sort(); + const canonical = [...expected].sort(); + return ( + actual.length === canonical.length && + actual.every((key, index) => key === canonical[index]) + ); +} + +function assertOptions( + options: ApplyReviewedLocalDataDirectoryAdoptionOptions, +): void { + if ( + !options || + typeof options !== 'object' || + Array.isArray(options) || + !UUID_V4_PATTERN.test(options.mutationId) || + !UUID_V4_PATTERN.test(options.failureAuditEventId) || + options.failureAuditEventId === options.mutationId || + !REQUEST_ID_PATTERN.test(options.requestId) || + (options.profile !== 'edge' && options.profile !== 'standalone') || + ![ + options.sourceStageManifestDigest, + options.transformationDigest, + options.modelDigest, + ].every((digest) => DIGEST_PATTERN.test(digest)) || + !Number.isSafeInteger(options.observedAtMs) || + options.observedAtMs < 0 || + typeof options.loadPreparedModel !== 'function' || + typeof options.confirmAuthenticationAuthority !== 'function' || + typeof options.confirmPreparedAuthority !== 'function' || + !options.keyProvider || + typeof options.keyProvider.active !== 'function' || + typeof options.keyProvider.resolve !== 'function' || + (options.busyTimeoutMs !== undefined && + (!Number.isSafeInteger(options.busyTimeoutMs) || + options.busyTimeoutMs < 100 || + options.busyTimeoutMs > 30_000)) + ) { + throw new LocalDataDirectoryAdoptionApplicationConfigurationError( + 'options are invalid', + ); + } + try { + assertLocalSecretProjectId(options.projectId); + } catch (error) { + throw new LocalDataDirectoryAdoptionApplicationConfigurationError( + 'projectId is invalid', + error, + ); + } +} + +function strongPrincipal( + value: Readonly, + nowMs: number, +): Readonly { + let principal: Readonly; + try { + principal = normalizeSecurityPrincipal(value, nowMs); + } catch { + throw new LocalDataDirectoryAdoptionApplicationAuthenticationError(); + } + const human = + principal.subject.type === 'user' && + STRONG_USER_ASSURANCES.has(principal.assurance); + const system = + principal.subject.type === 'system' && principal.assurance === 'service'; + if (!human && !system) { + throw new LocalDataDirectoryAdoptionApplicationAuthenticationError(); + } + return principal; +} + +function auditRecord(options: { + readonly eventId: string; + readonly requestId: string; + readonly projectId: string; + readonly principal: Readonly | null; + readonly operationId: string; + readonly outcome: SecurityAuditRecord['outcome']; + readonly reasons: readonly string[]; + readonly decision: Readonly | null; + readonly occurredAtMs: number; +}): Readonly { + return normalizeSecurityAuditRecord({ + eventId: options.eventId, + requestId: options.requestId, + operationId: options.operationId, + projectId: options.projectId, + subject: options.principal?.subject ?? null, + authenticationId: options.principal?.authenticationId ?? null, + outcome: options.outcome, + reasons: options.reasons, + fence: options.decision?.fence ?? null, + occurredAtMs: options.occurredAtMs, + }); +} + +function deterministicMutationId( + batchMutationId: string, + identity: string, +): string { + const bytes = createHash('sha256') + .update('qinglong3.legacy-data-directory-secret-mutation.v1\0') + .update(batchMutationId) + .update('\0') + .update(identity) + .digest() + .subarray(0, 16); + bytes[6] = ((bytes[6] as number) & 0x0f) | 0x40; + bytes[8] = ((bytes[8] as number) & 0x3f) | 0x80; + const hex = bytes.toString('hex'); + return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice( + 12, + 16, + )}-${hex.slice(16, 20)}-${hex.slice(20)}`; +} + +function assertPreparedModel( + value: Readonly, + profile: 'edge' | 'standalone', +): void { + if ( + !value || + typeof value !== 'object' || + Array.isArray(value) || + !exactKeys(value, ['model', 'secrets']) || + !Array.isArray(value.secrets) || + value.secrets.length > (profile === 'edge' ? 128 : 512) + ) { + throw new LocalDataDirectoryAdoptionApplicationConfigurationError( + 'prepared model shape is invalid', + ); + } + const targets = new Set(); + for (const entry of value.secrets) { + if ( + !entry || + typeof entry !== 'object' || + Array.isArray(entry) || + !exactKeys(entry, [ + 'expectedCurrentVersion', + 'kind', + 'plaintext', + 'sourceName', + 'targetName', + 'valueDigest', + 'valueFile', + ]) || + (entry.kind !== 'environment' && entry.kind !== 'ssh_private_key') || + typeof entry.sourceName !== 'string' || + entry.sourceName.length < 1 || + entry.sourceName.includes('\0') || + entry.expectedCurrentVersion !== 0 || + !VALUE_FILE_PATTERN.test(entry.valueFile) || + !DIGEST_PATTERN.test(entry.valueDigest) || + createHash('sha256').update(entry.plaintext, 'utf8').digest('hex') !== + entry.valueDigest || + targets.has(entry.targetName) + ) { + throw new LocalDataDirectoryAdoptionApplicationConfigurationError( + 'prepared Secret entry is invalid', + ); + } + try { + assertLocalSecretName(entry.targetName); + assertLocalSecretPlaintext(entry.plaintext); + } catch (error) { + throw new LocalDataDirectoryAdoptionApplicationConfigurationError( + 'prepared Secret value is invalid', + error, + ); + } + targets.add(entry.targetName); + } +} + +function sameExpected( + record: Readonly, + options: Readonly, +): boolean { + return ( + record.mutationId === options.mutationId && + record.projectId === options.projectId && + record.profile === options.profile && + record.sourceStageManifestDigest === options.sourceStageManifestDigest && + record.transformationDigest === options.transformationDigest && + record.modelDigest === options.modelDigest + ); +} + +export async function applyReviewedLocalDataDirectoryAdoption( + options: Readonly, +): Promise> { + assertOptions(options); + const database = await openLocalSqliteDataDirectoryAdoptionDatabase({ + databasePath: options.databasePath, + profile: options.profile, + ...(options.busyTimeoutMs === undefined + ? {} + : { busyTimeoutMs: options.busyTimeoutMs }), + }); + try { + let principal: Readonly; + try { + principal = strongPrincipal(options.principal, options.observedAtMs); + await options.confirmAuthenticationAuthority(); + } catch (error) { + try { + await database.securityAudit.record( + auditRecord({ + eventId: options.failureAuditEventId, + requestId: options.requestId, + projectId: options.projectId, + principal: null, + operationId: 'legacy-data.apply', + outcome: 'authentication_rejected', + reasons: ['strong_authentication_required'], + decision: null, + occurredAtMs: options.observedAtMs, + }), + ); + } catch { + throw new LocalDataDirectoryAdoptionApplicationUnavailableError(); + } + throw error; + } + + const policy = new ProjectPolicyEngine(database.projectPolicy); + let decision: Readonly; + try { + decision = await policy.authorize( + principal, + options.projectId, + 'secret.manage', + ); + } catch (error) { + if (!(error instanceof ProjectPolicyUnavailableError)) { + throw new LocalDataDirectoryAdoptionApplicationUnavailableError(error); + } + try { + await database.securityAudit.record( + auditRecord({ + eventId: options.failureAuditEventId, + requestId: options.requestId, + projectId: options.projectId, + principal, + operationId: 'legacy-data.apply', + outcome: 'authorization_unavailable', + reasons: ['policy_unavailable'], + decision: null, + occurredAtMs: options.observedAtMs, + }), + ); + } catch { + throw new LocalDataDirectoryAdoptionApplicationUnavailableError(); + } + throw new LocalDataDirectoryAdoptionApplicationUnavailableError(); + } + if (decision.effect !== 'allow') { + try { + await database.securityAudit.record( + auditRecord({ + eventId: options.failureAuditEventId, + requestId: options.requestId, + projectId: options.projectId, + principal, + operationId: 'legacy-data.apply', + outcome: + decision.effect === 'require_approval' + ? 'approval_required' + : 'denied', + reasons: decision.reasons, + decision, + occurredAtMs: options.observedAtMs, + }), + ); + } catch { + throw new LocalDataDirectoryAdoptionApplicationUnavailableError(); + } + throw new LocalDataDirectoryAdoptionApplicationAuthorizationError(); + } + if (!decision.fence || decision.fence.bindingVersion === null) { + throw new LocalDataDirectoryAdoptionApplicationUnavailableError(); + } + + const existing = await database.publisher.resolve(options.mutationId); + if (existing) { + if (!sameExpected(existing, options)) { + throw new LocalDataDirectoryAdoptionApplicationConfigurationError( + 'mutation replay does not match the committed adoption', + ); + } + await options.confirmAuthenticationAuthority(); + return Object.freeze({ status: 'existing', adoption: existing }); + } + + const prepared = await options.loadPreparedModel(); + assertPreparedModel(prepared, options.profile); + const material = ownedLocalSecretKeyMaterial( + await options.keyProvider.active(), + ); + const publications: LocalDataDirectoryAdoptionSecretPublication[] = []; + try { + for (const [index, secret] of prepared.secrets.entries()) { + const ordinal = index + 1; + const secretMutationId = deterministicMutationId( + options.mutationId, + `${secret.kind}\0${secret.targetName}`, + ); + const envelope = encryptLocalSecretEnvelope( + { + projectId: options.projectId, + name: secret.targetName, + version: 1, + mutationId: secretMutationId, + keyId: material.keyId, + algorithm: LOCAL_SECRET_ALGORITHM, + createdAtMs: options.observedAtMs, + }, + secret.plaintext, + material.key, + ); + const item = createLocalDataDirectoryAdoptionSecretItem({ + projectId: options.projectId, + ordinal, + kind: secret.kind, + sourceNameDigest: createLocalDataDirectorySourceNameDigest( + secret.kind, + secret.sourceName, + ), + secretName: secret.targetName, + secretMutationId, + valueFile: secret.valueFile, + valueDigest: secret.valueDigest, + }); + publications.push( + Object.freeze({ + ordinal, + kind: secret.kind, + sourceNameDigest: item.sourceNameDigest, + valueFile: secret.valueFile, + valueDigest: secret.valueDigest, + envelope, + secretRef: item.secretRef, + itemDigest: item.itemDigest, + audit: auditRecord({ + eventId: secretMutationId, + requestId: options.requestId, + projectId: options.projectId, + principal, + operationId: 'secret.create', + outcome: 'allowed', + reasons: decision.reasons, + decision, + occurredAtMs: options.observedAtMs, + }), + }), + ); + } + } finally { + material.key.fill(0); + } + const receipt = createLocalDataDirectoryAdoptionReceipt({ + mutationId: options.mutationId, + projectId: options.projectId, + profile: options.profile, + sourceStageManifestDigest: options.sourceStageManifestDigest, + transformationDigest: options.transformationDigest, + modelDigest: options.modelDigest, + secrets: publications.map(({ envelope, ...item }) => + Object.freeze({ + ordinal: item.ordinal, + kind: item.kind, + sourceNameDigest: item.sourceNameDigest, + secretName: envelope.name, + secretVersion: 1 as const, + secretMutationId: envelope.mutationId, + valueFile: item.valueFile, + valueDigest: item.valueDigest, + secretRef: item.secretRef, + itemDigest: item.itemDigest, + }), + ), + committedAtMs: options.observedAtMs, + }); + const applied = await database.publisher.publish({ + mutationId: options.mutationId, + projectId: options.projectId, + profile: options.profile, + sourceStageManifestDigest: options.sourceStageManifestDigest, + transformationDigest: options.transformationDigest, + modelDigest: options.modelDigest, + model: prepared.model, + subject: principal.subject, + fence: decision.fence, + audit: auditRecord({ + eventId: options.mutationId, + requestId: options.requestId, + projectId: options.projectId, + principal, + operationId: 'legacy-data.apply', + outcome: 'allowed', + reasons: decision.reasons, + decision, + occurredAtMs: options.observedAtMs, + }), + secrets: Object.freeze(publications), + receipt, + async confirmExternalAuthority() { + await options.confirmPreparedAuthority(); + await options.confirmAuthenticationAuthority(); + }, + }); + return Object.freeze(applied); + } finally { + await database.close(); + } +} diff --git a/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/application/application.ts b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/application/application.ts new file mode 100644 index 00000000..09771671 --- /dev/null +++ b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/application/application.ts @@ -0,0 +1,272 @@ +import { + applyReviewedLocalDataDirectoryAdoption, + LocalDataDirectoryAdoptionApplicationConfigurationError, +} from '@qinglong/local-admin/data-directory-adoption'; +import { LocalSecretKeyringFileProvider } from '@qinglong/local-secret'; +import { openLocalSqliteBootstrapDatabase } from '@qinglong/local-sqlite/bootstrap'; +import { + establishAuthenticatedLocalCommand, + type AuthenticatedLocalCommand, +} from '@qinglong/local-owner-console/authenticated-command'; + +import { + LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_OPERATION, + LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_VERIFY_OPERATION, + LOCAL_DATA_DIRECTORY_ADOPTION_VERIFY_OPERATION, + LocalDataDirectoryAdoptionConfigurationError, + type ApplyLocalDataDirectoryAdoptionCommand, + type VerifyLocalDataDirectoryAdoptionApplicationCommand, + type VerifyLocalDataDirectoryAdoptionCommand, +} from '../contract'; +import { verifyLocalDataDirectoryAdoption } from '../staging'; +import { transformationAuthority } from '../transformation/files'; +import { + loadStaticTransformation, + verifyTransformationManifestBinding, +} from '../transformation/manifest'; +import { + reclaimCommittedTransformationModel, + verifyCommittedTransformationModel, +} from './cleanup'; + +export interface LocalDataDirectoryApplicationResult { + readonly schemaVersion: 1; + readonly operation: + | typeof LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_OPERATION + | typeof LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_VERIFY_OPERATION; + readonly status: 'committed' | 'verified'; + readonly evidence: Readonly<{ + profile: 'edge' | 'standalone'; + databaseStatus: 'inserted' | 'existing'; + sourceStageManifestDigest: string; + transformationDigest: string; + modelDigest: string; + publicationDigest: string; + receiptDigest: string; + commitDigest: string; + secretCount: number; + environmentSecretCount: number; + sshSecretCount: number; + committedAtMs: number; + modelReclaimed: true; + plaintextFilesRemoved: true; + physicalErasureGuaranteed: false; + }>; +} + +type ApplicationCommand = + | Readonly + | Readonly; + +function sourceVerificationCommand( + command: ApplicationCommand, +): Readonly { + return Object.freeze({ + schemaVersion: 1, + operation: LOCAL_DATA_DIRECTORY_ADOPTION_VERIFY_OPERATION, + options: Object.freeze({ + deploymentRoot: command.options.deploymentRoot, + dataRoot: command.options.dataRoot, + stagingRoot: command.options.stagingRoot, + profile: command.options.profile, + sqlite: command.options.sqlite, + expectedManifestDigest: command.options.expectedManifestDigest, + }), + }); +} + +async function loadPreparedModel( + command: ApplicationCommand, + authority: ReturnType, +) { + const before = await verifyLocalDataDirectoryAdoption( + sourceVerificationCommand(command), + ); + const loaded = loadStaticTransformation({ + authority, + profile: command.options.profile, + projectId: command.options.projectId, + sourceStageManifestDigest: command.options.expectedManifestDigest, + expectedTransformationDigest: command.options.expectedTransformationDigest, + }); + if ( + loaded.manifest.assessment !== 'ready' || + loaded.manifest.model.manualCategories !== 0 || + (loaded.prepared.model.manualReview as { readonly required?: unknown }) + .required !== false + ) { + throw new LocalDataDirectoryAdoptionApplicationConfigurationError( + 'manual review must be resolved in a new transformation before apply', + ); + } + const after = await verifyLocalDataDirectoryAdoption( + sourceVerificationCommand(command), + ); + if (JSON.stringify(before.evidence) !== JSON.stringify(after.evidence)) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'staged source changed while preparing the application transaction', + ); + } + return loaded.prepared; +} + +function result( + operation: LocalDataDirectoryApplicationResult['operation'], + status: LocalDataDirectoryApplicationResult['status'], + databaseStatus: 'inserted' | 'existing', + adoption: Awaited< + ReturnType + >['adoption'], + commit: ReturnType, +): Readonly { + return Object.freeze({ + schemaVersion: 1, + operation, + status, + evidence: Object.freeze({ + profile: adoption.profile, + databaseStatus, + sourceStageManifestDigest: adoption.sourceStageManifestDigest, + transformationDigest: adoption.transformationDigest, + modelDigest: adoption.modelDigest, + publicationDigest: adoption.publicationDigest, + receiptDigest: adoption.receiptDigest, + commitDigest: commit.commitDigest, + secretCount: adoption.receipt.secretCount, + environmentSecretCount: adoption.receipt.environmentSecretCount, + sshSecretCount: adoption.receipt.sshSecretCount, + committedAtMs: adoption.committedAtMs, + modelReclaimed: true, + plaintextFilesRemoved: true, + physicalErasureGuaranteed: false, + }), + }); +} + +async function authenticatedAuthority(command: ApplicationCommand): Promise< + Readonly<{ + authenticated: Readonly; + close(): Promise; + }> +> { + const database = await openLocalSqliteBootstrapDatabase({ + databasePath: command.options.sqlite.targetPath, + profile: command.options.profile, + ...(command.options.busyTimeoutMs === undefined + ? {} + : { busyTimeoutMs: command.options.busyTimeoutMs }), + }); + try { + const authenticated = await establishAuthenticatedLocalCommand(database, { + deploymentRoot: command.options.deploymentRoot, + databasePath: command.options.sqlite.targetPath, + ownerPepperKeyringDirectory: command.options.ownerPepperKeyringDirectory, + credentialFilePath: command.options.credentialFilePath, + authenticationNamespace: 'local_data_adoption', + }); + return Object.freeze({ + authenticated, + close: () => database.close(), + }); + } catch (error) { + await database.close(); + throw error; + } +} + +async function apply( + command: ApplicationCommand, + verifyOnly: boolean, +): Promise> { + const authority = transformationAuthority(command.options, false); + const manifest = verifyTransformationManifestBinding({ + authority, + profile: command.options.profile, + projectId: command.options.projectId, + sourceStageManifestDigest: command.options.expectedManifestDigest, + expectedTransformationDigest: command.options.expectedTransformationDigest, + }); + const authentication = await authenticatedAuthority(command); + try { + const applied = await applyReviewedLocalDataDirectoryAdoption({ + databasePath: command.options.sqlite.targetPath, + profile: command.options.profile, + projectId: command.options.projectId, + mutationId: command.options.mutationId, + failureAuditEventId: command.options.failureAuditEventId, + requestId: command.options.requestId, + sourceStageManifestDigest: command.options.expectedManifestDigest, + transformationDigest: command.options.expectedTransformationDigest, + modelDigest: manifest.model.digest, + principal: authentication.authenticated.principal, + keyProvider: new LocalSecretKeyringFileProvider( + command.options.secretKeyringPath, + ), + observedAtMs: Date.now(), + ...(command.options.busyTimeoutMs === undefined + ? {} + : { busyTimeoutMs: command.options.busyTimeoutMs }), + loadPreparedModel: verifyOnly + ? () => { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'application receipt does not exist', + ); + } + : () => loadPreparedModel(command, authority), + confirmAuthenticationAuthority: () => + authentication.authenticated.confirm(), + async confirmPreparedAuthority() { + await loadPreparedModel(command, authority); + }, + }); + if ( + applied.adoption.modelDigest !== manifest.model.digest || + applied.adoption.transformationDigest !== manifest.transformationDigest + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'database application does not match the transformation manifest', + ); + } + if (verifyOnly) { + const verifyCommand = + command as Readonly; + const commit = verifyCommittedTransformationModel({ + authority, + adoption: applied.adoption, + expectedReceiptDigest: verifyCommand.options.expectedReceiptDigest, + }); + return result( + LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_VERIFY_OPERATION, + 'verified', + applied.status, + applied.adoption, + commit, + ); + } + const commit = reclaimCommittedTransformationModel({ + authority, + adoption: applied.adoption, + }); + return result( + LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_OPERATION, + 'committed', + applied.status, + applied.adoption, + commit, + ); + } finally { + await authentication.close(); + } +} + +export function applyLocalDataDirectoryAdoption( + command: Readonly, +): Promise> { + return apply(command, false); +} + +export function verifyLocalDataDirectoryAdoptionApplication( + command: Readonly, +): Promise> { + return apply(command, true); +} diff --git a/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/application/cleanup.ts b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/application/cleanup.ts new file mode 100644 index 00000000..fccc4ba7 --- /dev/null +++ b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/application/cleanup.ts @@ -0,0 +1,468 @@ +import fs from 'node:fs'; +import path from 'node:path'; + +import { + createLocalDataDirectorySourceNameDigest, + type LocalDataDirectoryAdoptionRecord, +} from '@qinglong/local-sqlite/data-directory-adoption'; + +import { LocalDataDirectoryAdoptionConfigurationError } from '../contract'; +import { + assertPrivateDirectory, + sameStat, + sortedNames, + syncDirectory, +} from '../filesystem'; +import { sha256Text } from '../manifest'; +import { + readStablePrivateUtf8File, + writePrivateJson, + type TransformationAuthority, +} from '../transformation/files'; +import { verifyTransformationModel } from '../transformation/model'; +import { verifyTransformationManifestBinding } from '../transformation/manifest'; + +export const APPLICATION_COMMIT_NAME = 'commit.json'; +const COMMIT_INCOMPLETE_NAME = '.commit-incomplete'; +const RECLAIMING_MODEL_NAME = '.reclaiming-model'; +const MODEL_NAME = 'model'; +const MAX_JSON_BYTES = 1024 * 1024; +const ZERO_CHUNK = Buffer.alloc(64 * 1024); + +interface LocalDataDirectoryApplicationCommitPayload { + readonly schemaVersion: 1; + readonly kind: 'qinglong3-legacy-data-directory-application'; + readonly state: 'committed'; + readonly mutationId: string; + readonly profile: 'edge' | 'standalone'; + readonly projectIdDigest: string; + readonly sourceStageManifestDigest: string; + readonly transformationDigest: string; + readonly modelDigest: string; + readonly publicationDigest: string; + readonly receiptDigest: string; + readonly secretCount: number; + readonly environmentSecretCount: number; + readonly sshSecretCount: number; + readonly committedAtMs: number; + readonly reclamation: Readonly<{ + modelRemoved: true; + plaintextFilesRemoved: true; + physicalErasureGuaranteed: false; + }>; +} + +export interface LocalDataDirectoryApplicationCommit + extends LocalDataDirectoryApplicationCommitPayload { + readonly commitDigest: string; +} + +function exactKeys(value: object, expected: readonly string[]): boolean { + const actual = Object.keys(value).sort(); + const canonical = [...expected].sort(); + return ( + actual.length === canonical.length && + actual.every((key, index) => key === canonical[index]) + ); +} + +function expectedCommit( + adoption: Readonly, +): Readonly { + const payload: LocalDataDirectoryApplicationCommitPayload = { + schemaVersion: 1, + kind: 'qinglong3-legacy-data-directory-application', + state: 'committed', + mutationId: adoption.mutationId, + profile: adoption.profile, + projectIdDigest: sha256Text(adoption.projectId), + sourceStageManifestDigest: adoption.sourceStageManifestDigest, + transformationDigest: adoption.transformationDigest, + modelDigest: adoption.modelDigest, + publicationDigest: adoption.publicationDigest, + receiptDigest: adoption.receiptDigest, + secretCount: adoption.receipt.secretCount, + environmentSecretCount: adoption.receipt.environmentSecretCount, + sshSecretCount: adoption.receipt.sshSecretCount, + committedAtMs: adoption.committedAtMs, + reclamation: Object.freeze({ + modelRemoved: true, + plaintextFilesRemoved: true, + physicalErasureGuaranteed: false, + }), + }; + return Object.freeze({ + ...payload, + commitDigest: sha256Text(JSON.stringify(payload)), + }); +} + +function readJson(filePath: string, uid: number, label: string): unknown { + try { + return JSON.parse( + readStablePrivateUtf8File(filePath, uid, MAX_JSON_BYTES, label), + ); + } catch (error) { + if (error instanceof LocalDataDirectoryAdoptionConfigurationError) { + throw error; + } + throw new LocalDataDirectoryAdoptionConfigurationError( + `${label} JSON is invalid`, + error, + ); + } +} + +function sameJson(left: unknown, right: unknown): boolean { + return JSON.stringify(left) === JSON.stringify(right); +} + +function verifyMarker( + root: string, + uid: number, + adoption: Readonly, +): void { + const marker = readJson( + path.join(root, COMMIT_INCOMPLETE_NAME), + uid, + 'application recovery marker', + ); + if ( + !marker || + typeof marker !== 'object' || + Array.isArray(marker) || + !exactKeys(marker, [ + 'kind', + 'mutationId', + 'receiptDigest', + 'schemaVersion', + 'transformationDigest', + ]) + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'application recovery marker shape is invalid', + ); + } + const candidate = marker as Record; + if ( + candidate.schemaVersion !== 1 || + candidate.kind !== + 'qinglong3-legacy-data-directory-application-incomplete' || + candidate.mutationId !== adoption.mutationId || + candidate.transformationDigest !== adoption.transformationDigest || + candidate.receiptDigest !== adoption.receiptDigest + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'application recovery marker does not match the committed database', + ); + } +} + +function verifyCommit( + authority: Readonly, + adoption: Readonly, +): Readonly { + const actual = readJson( + path.join(authority.transformationRoot, APPLICATION_COMMIT_NAME), + authority.uid, + 'application commit', + ); + const expected = expectedCommit(adoption); + if (!sameJson(actual, expected)) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'application commit does not match the durable database receipt', + ); + } + return expected; +} + +function assertPreparedMatches( + modelRoot: string, + authority: Readonly, + adoption: Readonly, +): void { + const manifest = verifyTransformationManifestBinding({ + authority, + profile: adoption.profile, + projectId: adoption.projectId, + sourceStageManifestDigest: adoption.sourceStageManifestDigest, + expectedTransformationDigest: adoption.transformationDigest, + }); + const prepared = verifyTransformationModel({ + modelRoot, + uid: authority.uid, + projectId: adoption.projectId, + profile: adoption.profile, + expected: manifest.model, + }); + if ( + manifest.assessment !== 'ready' || + manifest.model.manualCategories !== 0 || + !sameJson(prepared.model, adoption.model) || + prepared.secrets.length !== adoption.secrets.length || + prepared.secrets.some((secret, index) => { + const stored = adoption.secrets[index]; + return ( + !stored || + stored.ordinal !== index + 1 || + stored.kind !== secret.kind || + stored.sourceNameDigest !== + createLocalDataDirectorySourceNameDigest( + secret.kind, + secret.sourceName, + ) || + stored.secretName !== secret.targetName || + stored.valueFile !== secret.valueFile || + stored.valueDigest !== secret.valueDigest + ); + }) + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'prepared model does not match the committed database receipt', + ); + } +} + +function privateEntry( + entryPath: string, + uid: number, + kind: 'file' | 'directory', +): fs.BigIntStats { + const stat = fs.lstatSync(entryPath, { bigint: true }); + if ( + stat.isSymbolicLink() || + stat.uid !== BigInt(uid) || + (stat.mode & 0o777n) !== (kind === 'file' ? 0o600n : 0o700n) || + (kind === 'file' + ? !stat.isFile() || stat.nlink !== 1n + : !stat.isDirectory()) + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'reclaiming model identity or mode is invalid', + ); + } + return stat; +} + +function overwriteAndUnlink(filePath: string, uid: number): void { + const expected = privateEntry(filePath, uid, 'file'); + if (expected.size < 0n || expected.size > BigInt(32 * 1024)) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'reclaiming Secret file size is invalid', + ); + } + const descriptor = fs.openSync( + filePath, + fs.constants.O_RDWR | (fs.constants.O_NOFOLLOW ?? 0), + ); + try { + const opened = fs.fstatSync(descriptor, { bigint: true }); + if (!sameStat(expected, opened)) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'reclaiming Secret identity changed before overwrite', + ); + } + let remaining = Number(opened.size); + let offset = 0; + while (remaining > 0) { + const count = Math.min(remaining, ZERO_CHUNK.length); + const written = fs.writeSync(descriptor, ZERO_CHUNK, 0, count, offset); + if (written < 1) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'reclaiming Secret overwrite made no progress', + ); + } + remaining -= written; + offset += written; + } + fs.fsyncSync(descriptor); + } finally { + fs.closeSync(descriptor); + } + fs.unlinkSync(filePath); +} + +function reclaimRenamedModel( + root: string, + uid: number, + adoption: Readonly, +): void { + const modelRoot = path.join(root, RECLAIMING_MODEL_NAME); + privateEntry(modelRoot, uid, 'directory'); + const allowedRootFiles = new Set([ + 'config.json', + 'keyv.json', + 'manual-review.json', + 'secret-imports.json', + 'ssh.json', + 'secret-values', + ]); + if (sortedNames(modelRoot).some((name) => !allowedRootFiles.has(name))) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'reclaiming model contains an unexpected entry', + ); + } + const secretRoot = path.join(modelRoot, 'secret-values'); + if (fs.existsSync(secretRoot)) { + privateEntry(secretRoot, uid, 'directory'); + const expectedFiles = new Set( + adoption.secrets.map(({ valueFile }) => path.basename(valueFile)), + ); + const actualFiles = sortedNames(secretRoot); + if (actualFiles.some((name) => !expectedFiles.has(name))) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'reclaiming Secret directory contains an unexpected entry', + ); + } + for (const name of actualFiles) { + overwriteAndUnlink(path.join(secretRoot, name), uid); + } + fs.rmdirSync(secretRoot); + syncDirectory(modelRoot); + } + for (const name of [ + 'config.json', + 'keyv.json', + 'ssh.json', + 'manual-review.json', + 'secret-imports.json', + ]) { + const filePath = path.join(modelRoot, name); + if (!fs.existsSync(filePath)) continue; + privateEntry(filePath, uid, 'file'); + fs.unlinkSync(filePath); + } + syncDirectory(modelRoot); + if (sortedNames(modelRoot).length !== 0) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'reclaiming model could not be emptied', + ); + } + fs.rmdirSync(modelRoot); + syncDirectory(root); +} + +export function reclaimCommittedTransformationModel(options: { + readonly authority: Readonly; + readonly adoption: Readonly; +}): Readonly { + const { authority, adoption } = options; + const root = authority.transformationRoot; + const before = assertPrivateDirectory( + root, + authority.uid, + 'transformationRoot', + ); + verifyTransformationManifestBinding({ + authority, + profile: adoption.profile, + projectId: adoption.projectId, + sourceStageManifestDigest: adoption.sourceStageManifestDigest, + expectedTransformationDigest: adoption.transformationDigest, + }); + const initialNames = sortedNames(root); + const allowedNames = new Set([ + 'manifest.json', + APPLICATION_COMMIT_NAME, + COMMIT_INCOMPLETE_NAME, + MODEL_NAME, + RECLAIMING_MODEL_NAME, + ]); + if ( + initialNames.some((name) => !allowedNames.has(name)) || + !initialNames.includes('manifest.json') || + (initialNames.includes(MODEL_NAME) && + initialNames.includes(RECLAIMING_MODEL_NAME)) + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'application recovery root shape is invalid', + ); + } + if (!initialNames.includes(COMMIT_INCOMPLETE_NAME)) { + if ( + initialNames.includes(APPLICATION_COMMIT_NAME) && + !initialNames.includes(MODEL_NAME) && + !initialNames.includes(RECLAIMING_MODEL_NAME) + ) { + return verifyCommit(authority, adoption); + } + writePrivateJson(path.join(root, COMMIT_INCOMPLETE_NAME), { + schemaVersion: 1, + kind: 'qinglong3-legacy-data-directory-application-incomplete', + mutationId: adoption.mutationId, + transformationDigest: adoption.transformationDigest, + receiptDigest: adoption.receiptDigest, + }); + syncDirectory(root); + } + verifyMarker(root, authority.uid, adoption); + + const modelRoot = path.join(root, MODEL_NAME); + const reclaimingRoot = path.join(root, RECLAIMING_MODEL_NAME); + if (fs.existsSync(modelRoot)) { + assertPreparedMatches(modelRoot, authority, adoption); + if (fs.existsSync(reclaimingRoot)) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'multiple application model recovery roots exist', + ); + } + fs.renameSync(modelRoot, reclaimingRoot); + syncDirectory(root); + } + if (fs.existsSync(reclaimingRoot)) { + reclaimRenamedModel(root, authority.uid, adoption); + } + + const commitPath = path.join(root, APPLICATION_COMMIT_NAME); + if (!fs.existsSync(commitPath)) { + writePrivateJson(commitPath, expectedCommit(adoption)); + syncDirectory(root); + } + const commit = verifyCommit(authority, adoption); + fs.unlinkSync(path.join(root, COMMIT_INCOMPLETE_NAME)); + syncDirectory(root); + if ( + JSON.stringify(sortedNames(root)) !== + JSON.stringify([APPLICATION_COMMIT_NAME, 'manifest.json'].sort()) + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'committed transformation root contains unexpected entries', + ); + } + const after = fs.lstatSync(root, { bigint: true }); + if ( + before.dev !== after.dev || + before.ino !== after.ino || + before.uid !== after.uid || + (after.mode & 0o777n) !== 0o700n + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'transformation root identity changed during application cleanup', + ); + } + return commit; +} + +export function verifyCommittedTransformationModel(options: { + readonly authority: Readonly; + readonly adoption: Readonly; + readonly expectedReceiptDigest: string; +}): Readonly { + if ( + options.adoption.receiptDigest !== options.expectedReceiptDigest || + JSON.stringify(sortedNames(options.authority.transformationRoot)) !== + JSON.stringify([APPLICATION_COMMIT_NAME, 'manifest.json'].sort()) + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'committed transformation evidence is incomplete', + ); + } + verifyTransformationManifestBinding({ + authority: options.authority, + profile: options.adoption.profile, + projectId: options.adoption.projectId, + sourceStageManifestDigest: options.adoption.sourceStageManifestDigest, + expectedTransformationDigest: options.adoption.transformationDigest, + }); + return verifyCommit(options.authority, options.adoption); +} diff --git a/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/command.ts b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/command.ts index 34a27dd6..01c763a7 100644 --- a/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/command.ts +++ b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/command.ts @@ -1,10 +1,17 @@ import { + LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_OPERATION, + LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_VERIFY_OPERATION, LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION, LOCAL_DATA_DIRECTORY_ADOPTION_STAGE_OPERATION, LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_OPERATION, LOCAL_DATA_DIRECTORY_ADOPTION_VERIFY_OPERATION, normalizeLocalDataDirectoryAdoptionCommand, } from './contract'; +import { + applyLocalDataDirectoryAdoption, + verifyLocalDataDirectoryAdoptionApplication, + type LocalDataDirectoryApplicationResult, +} from './application/application'; import { inspectLocalDataDirectoryAdoption, type LocalDataDirectoryAdoptionInspectResult, @@ -23,7 +30,8 @@ import { export type LocalDataDirectoryAdoptionProductCommandResult = | LocalDataDirectoryAdoptionInspectResult | LocalDataDirectoryAdoptionMutationResult - | LocalDataDirectoryTransformationResult; + | LocalDataDirectoryTransformationResult + | LocalDataDirectoryApplicationResult; export async function runLocalDataDirectoryAdoptionProductCommand( value: unknown, @@ -41,5 +49,13 @@ export async function runLocalDataDirectoryAdoptionProductCommand( if (command.operation === LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_OPERATION) { return transformLocalDataDirectoryAdoption(command); } + if (command.operation === LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_OPERATION) { + return applyLocalDataDirectoryAdoption(command); + } + if ( + command.operation === LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_VERIFY_OPERATION + ) { + return verifyLocalDataDirectoryAdoptionApplication(command); + } return verifyLocalDataDirectoryAdoptionTransformation(command); } diff --git a/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/contract.ts b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/contract.ts index 618b237d..158e0354 100644 --- a/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/contract.ts +++ b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/contract.ts @@ -3,6 +3,9 @@ import path from 'node:path'; const MAX_PATH_BYTES = 4_096; const DIGEST_PATTERN = /^[0-9a-f]{64}$/; const PROJECT_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/; +const UUID_V4_PATTERN = + /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/; +const REQUEST_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/; export const LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION = 'local-data-directory.adoption.inspect' as const; @@ -14,13 +17,19 @@ export const LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_OPERATION = 'local-data-directory.adoption.transform' as const; export const LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_VERIFY_OPERATION = 'local-data-directory.adoption.transform.verify' as const; +export const LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_OPERATION = + 'local-data-directory.adoption.apply' as const; +export const LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_VERIFY_OPERATION = + 'local-data-directory.adoption.apply.verify' as const; export type LocalDataDirectoryAdoptionOperation = | typeof LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION | typeof LOCAL_DATA_DIRECTORY_ADOPTION_STAGE_OPERATION | typeof LOCAL_DATA_DIRECTORY_ADOPTION_VERIFY_OPERATION | typeof LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_OPERATION - | typeof LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_VERIFY_OPERATION; + | typeof LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_VERIFY_OPERATION + | typeof LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_OPERATION + | typeof LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_VERIFY_OPERATION; export interface InspectLocalDataDirectoryAdoptionCommand { readonly schemaVersion: 1; @@ -85,12 +94,40 @@ export interface VerifyLocalDataDirectoryAdoptionTransformationCommand { }; } +interface LocalDataDirectoryAdoptionApplicationOptions + extends LocalDataDirectoryAdoptionTransformationOptions { + readonly expectedTransformationDigest: string; + readonly ownerPepperKeyringDirectory: string; + readonly credentialFilePath: string; + readonly secretKeyringPath: string; + readonly mutationId: string; + readonly failureAuditEventId: string; + readonly requestId: string; + readonly busyTimeoutMs?: number; +} + +export interface ApplyLocalDataDirectoryAdoptionCommand { + readonly schemaVersion: 1; + readonly operation: typeof LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_OPERATION; + readonly options: LocalDataDirectoryAdoptionApplicationOptions; +} + +export interface VerifyLocalDataDirectoryAdoptionApplicationCommand { + readonly schemaVersion: 1; + readonly operation: typeof LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_VERIFY_OPERATION; + readonly options: LocalDataDirectoryAdoptionApplicationOptions & { + readonly expectedReceiptDigest: string; + }; +} + export type LocalDataDirectoryAdoptionCommand = | InspectLocalDataDirectoryAdoptionCommand | StageLocalDataDirectoryAdoptionCommand | VerifyLocalDataDirectoryAdoptionCommand | TransformLocalDataDirectoryAdoptionCommand - | VerifyLocalDataDirectoryAdoptionTransformationCommand; + | VerifyLocalDataDirectoryAdoptionTransformationCommand + | ApplyLocalDataDirectoryAdoptionCommand + | VerifyLocalDataDirectoryAdoptionApplicationCommand; export class LocalDataDirectoryAdoptionConfigurationError extends TypeError { readonly code = 'LOCAL_DATA_DIRECTORY_ADOPTION_CONFIGURATION_INVALID'; @@ -129,7 +166,19 @@ export function isLocalDataDirectoryAdoptionOperation( value === LOCAL_DATA_DIRECTORY_ADOPTION_STAGE_OPERATION || value === LOCAL_DATA_DIRECTORY_ADOPTION_VERIFY_OPERATION || value === LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_OPERATION || - value === LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_VERIFY_OPERATION + value === LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_VERIFY_OPERATION || + value === LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_OPERATION || + value === LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_VERIFY_OPERATION + ); +} + +function descendant(root: string, candidate: string): boolean { + const relative = path.relative(root, candidate); + return ( + relative.length > 0 && + relative !== '..' && + !relative.startsWith(`..${path.sep}`) && + !path.isAbsolute(relative) ); } @@ -219,6 +268,10 @@ export function normalizeLocalDataDirectoryAdoptionCommand( ); } const options = candidate.options as Record; + const application = + candidate.operation === LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_OPERATION || + candidate.operation === + LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_VERIFY_OPERATION; let expectedKeys: readonly string[]; if (candidate.operation === LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION) { expectedKeys = ['dataRoot', 'profile']; @@ -245,9 +298,24 @@ export function normalizeLocalDataDirectoryAdoptionCommand( ? [] : ['projectId', 'transformationRoot']), ...(candidate.operation === - LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_VERIFY_OPERATION + LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_VERIFY_OPERATION || application ? ['expectedTransformationDigest'] : []), + ...(application + ? [ + 'credentialFilePath', + 'failureAuditEventId', + 'mutationId', + 'ownerPepperKeyringDirectory', + 'requestId', + 'secretKeyringPath', + ...(options.busyTimeoutMs === undefined ? [] : ['busyTimeoutMs']), + ] + : []), + ...(candidate.operation === + LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_VERIFY_OPERATION + ? ['expectedReceiptDigest'] + : []), ]; } if ( @@ -282,7 +350,8 @@ export function normalizeLocalDataDirectoryAdoptionCommand( candidate.operation === LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_OPERATION || candidate.operation === - LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_VERIFY_OPERATION + LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_VERIFY_OPERATION || + application ) { if ( !normalizedAbsolutePath(options.transformationRoot) || @@ -294,8 +363,9 @@ export function normalizeLocalDataDirectoryAdoptionCommand( ); } if ( - candidate.operation === - LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_VERIFY_OPERATION && + (candidate.operation === + LOCAL_DATA_DIRECTORY_ADOPTION_TRANSFORM_VERIFY_OPERATION || + application) && (typeof options.expectedTransformationDigest !== 'string' || !DIGEST_PATTERN.test(options.expectedTransformationDigest)) ) { @@ -303,6 +373,47 @@ export function normalizeLocalDataDirectoryAdoptionCommand( 'transformation digest is invalid', ); } + if (application) { + for (const key of [ + 'ownerPepperKeyringDirectory', + 'credentialFilePath', + 'secretKeyringPath', + ]) { + if ( + !normalizedAbsolutePath(options[key]) || + !descendant( + options.deploymentRoot as string, + options[key] as string, + ) + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'application authority path is invalid', + ); + } + } + if ( + options.credentialFilePath === options.secretKeyringPath || + typeof options.mutationId !== 'string' || + !UUID_V4_PATTERN.test(options.mutationId) || + typeof options.failureAuditEventId !== 'string' || + !UUID_V4_PATTERN.test(options.failureAuditEventId) || + options.failureAuditEventId === options.mutationId || + typeof options.requestId !== 'string' || + !REQUEST_ID_PATTERN.test(options.requestId) || + (options.busyTimeoutMs !== undefined && + (!Number.isSafeInteger(options.busyTimeoutMs) || + (options.busyTimeoutMs as number) < 100 || + (options.busyTimeoutMs as number) > 30_000)) || + (candidate.operation === + LOCAL_DATA_DIRECTORY_ADOPTION_APPLY_VERIFY_OPERATION && + (typeof options.expectedReceiptDigest !== 'string' || + !DIGEST_PATTERN.test(options.expectedReceiptDigest))) + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'application authority binding is invalid', + ); + } + } } } return Object.freeze(value as LocalDataDirectoryAdoptionCommand); diff --git a/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/transformation/manifest.ts b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/transformation/manifest.ts index 6d7aac04..229d1f25 100644 --- a/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/transformation/manifest.ts +++ b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/transformation/manifest.ts @@ -13,6 +13,7 @@ import { type LocalDataDirectoryTransformationManifest, type TransformationModelEvidence, type TransformationSourceEvidence, + type VerifiedTransformationModel, } from './model'; export const TRANSFORMATION_MANIFEST_NAME = 'manifest.json'; @@ -210,6 +211,54 @@ export function verifyStaticTransformation(options: { readonly sourceStageManifestDigest: string; readonly expectedTransformationDigest: string; }): Readonly { + return loadStaticTransformation(options).manifest; +} + +export function verifyTransformationManifestBinding(options: { + readonly authority: Readonly; + readonly profile: 'edge' | 'standalone'; + readonly projectId: string; + readonly sourceStageManifestDigest: string; + readonly expectedTransformationDigest: string; +}): Readonly { + const before = assertPrivateDirectory( + options.authority.transformationRoot, + options.authority.uid, + 'transformationRoot', + ); + const manifest = readManifest( + options.authority.transformationRoot, + options.authority.uid, + ); + if ( + manifest.transformationDigest !== options.expectedTransformationDigest || + manifest.profile !== options.profile || + manifest.projectIdDigest !== sha256Text(options.projectId) || + manifest.sourceStageManifestDigest !== options.sourceStageManifestDigest || + manifest.transformationRootPathDigest !== + sha256Text(options.authority.transformationRoot) || + !sameStat( + before, + fs.lstatSync(options.authority.transformationRoot, { bigint: true }), + ) + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'transformation manifest authority binding is invalid', + ); + } + return manifest; +} + +export function loadStaticTransformation(options: { + readonly authority: Readonly; + readonly profile: 'edge' | 'standalone'; + readonly projectId: string; + readonly sourceStageManifestDigest: string; + readonly expectedTransformationDigest: string; +}): Readonly<{ + manifest: Readonly; + prepared: Readonly; +}> { const before = assertPrivateDirectory( options.authority.transformationRoot, options.authority.uid, @@ -239,7 +288,7 @@ export function verifyStaticTransformation(options: { 'transformation manifest authority binding is invalid', ); } - verifyTransformationModel({ + const prepared = verifyTransformationModel({ modelRoot: path.join(options.authority.transformationRoot, 'model'), uid: options.authority.uid, projectId: options.projectId, @@ -256,5 +305,5 @@ export function verifyStaticTransformation(options: { 'transformation root changed during verification', ); } - return manifest; + return Object.freeze({ manifest, prepared }); } diff --git a/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/transformation/model.ts b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/transformation/model.ts index 23d7b9c5..d4dc8bab 100644 --- a/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/transformation/model.ts +++ b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/transformation/model.ts @@ -58,6 +58,22 @@ interface SecretImportEntry { readonly valueDigest: string; } +export interface VerifiedTransformationSecret extends SecretImportEntry { + readonly plaintext: string; +} + +export interface VerifiedTransformationModel { + readonly model: Readonly<{ + schema: 'qinglong/legacy-data-directory-applied-model@v1'; + activation: 'disabled'; + config: Readonly>; + keyv: Readonly>; + ssh: Readonly>; + manualReview: Readonly>; + }>; + readonly secrets: readonly Readonly[]; +} + function exactKeys(value: object, expected: readonly string[]): boolean { const actual = Object.keys(value).sort(); const canonical = [...expected].sort(); @@ -252,7 +268,7 @@ export function verifyTransformationModel(options: { readonly projectId: string; readonly profile: 'edge' | 'standalone'; readonly expected: Readonly; -}): void { +}): Readonly { if ( JSON.stringify(sortedNames(options.modelRoot)) !== JSON.stringify( @@ -270,17 +286,17 @@ export function verifyTransformationModel(options: { 'transformation model root contains unexpected entries', ); } - assertSchemaFile( + const config = assertSchemaFile( path.join(options.modelRoot, 'config.json'), options.uid, 'qinglong/legacy-config-transformation@v1', ); - assertSchemaFile( + const keyv = assertSchemaFile( path.join(options.modelRoot, 'keyv.json'), options.uid, 'qinglong/legacy-keyv-transformation@v1', ); - assertSchemaFile( + const ssh = assertSchemaFile( path.join(options.modelRoot, 'ssh.json'), options.uid, 'qinglong/legacy-ssh-transformation@v1', @@ -329,6 +345,7 @@ export function verifyTransformationModel(options: { } const expectedFiles: string[] = []; const targets = new Set(); + const secrets: VerifiedTransformationSecret[] = []; let environmentSecrets = 0; let sshSecrets = 0; for (const value of candidate.imports) { @@ -353,7 +370,13 @@ export function verifyTransformationModel(options: { if ( (entry.kind !== 'environment' && entry.kind !== 'ssh_private_key') || typeof entry.sourceName !== 'string' || + entry.sourceName.length < 1 || + Buffer.byteLength(entry.sourceName, 'utf8') > 255 || + /[\u0000-\u001f\u007f]/.test(entry.sourceName) || typeof entry.targetName !== 'string' || + entry.targetName.length < 1 || + Buffer.byteLength(entry.targetName, 'utf8') > 128 || + /[\u0000-\u001f\u007f]/.test(entry.targetName) || entry.expectedCurrentVersion !== 0 || typeof entry.valueFile !== 'string' || !SECRET_FILE_PATTERN.test(entry.valueFile) || @@ -396,6 +419,17 @@ export function verifyTransformationModel(options: { } if (entry.kind === 'environment') environmentSecrets += 1; else sshSecrets += 1; + secrets.push( + Object.freeze({ + kind: entry.kind, + sourceName: entry.sourceName, + targetName: entry.targetName, + expectedCurrentVersion: 0, + valueFile: entry.valueFile, + valueDigest: entry.valueDigest, + plaintext: secretValue.value, + }) as Readonly, + ); } expectedFiles.sort(); if ( @@ -427,4 +461,15 @@ export function verifyTransformationModel(options: { 'transformation model no longer matches the manifest', ); } + return Object.freeze({ + model: Object.freeze({ + schema: 'qinglong/legacy-data-directory-applied-model@v1' as const, + activation: 'disabled' as const, + config: Object.freeze(config), + keyv: Object.freeze(keyv), + ssh: Object.freeze(ssh), + manualReview: Object.freeze(manual), + }), + secrets: Object.freeze(secrets), + }); } diff --git a/packages/ql3-local-owner-cli/test/dataDirectoryAdoptionStageCli.test.cjs b/packages/ql3-local-owner-cli/test/dataDirectoryAdoptionStageCli.test.cjs index b74a3f9a..8999a125 100644 --- a/packages/ql3-local-owner-cli/test/dataDirectoryAdoptionStageCli.test.cjs +++ b/packages/ql3-local-owner-cli/test/dataDirectoryAdoptionStageCli.test.cjs @@ -6,6 +6,19 @@ const path = require('node:path'); const { DatabaseSync } = require('node:sqlite'); const { test } = require('node:test'); +const { + provisionLocalOwnerPepperKey, +} = require('@qinglong/local-owner-console'); +const { + LocalSecretKeyringFileProvider, + decryptLocalSecretEnvelopeToBuffer, + provisionLocalSecretKeyring, +} = require('@qinglong/local-secret'); +const { + apiCredentialSecretDigest, + formatApiCredentialToken, +} = require('@qinglong/runtime-core/api-credential-token'); + const BINARY = path.join(__dirname, '../dist/lifecycle/adoptionCli.js'); const DIRECTORY_INSPECT = 'local-data-directory.adoption.inspect'; const DIRECTORY_STAGE = 'local-data-directory.adoption.stage'; @@ -13,6 +26,18 @@ const DIRECTORY_VERIFY = 'local-data-directory.adoption.verify'; const DIRECTORY_TRANSFORM = 'local-data-directory.adoption.transform'; const DIRECTORY_TRANSFORM_VERIFY = 'local-data-directory.adoption.transform.verify'; +const DIRECTORY_APPLY = 'local-data-directory.adoption.apply'; +const DIRECTORY_APPLY_VERIFY = 'local-data-directory.adoption.apply.verify'; +const APPLICATION_CREDENTIAL_ID = 'data-adoption-owner'; +const APPLICATION_PEPPER_KEY_ID = 'data-adoption-owner-v1'; +const APPLICATION_PEPPER = Buffer.alloc(32, 121).toString('base64url'); +const APPLICATION_CREDENTIAL_SECRET = Buffer.alloc(32, 122).toString( + 'base64url', +); +const APPLICATION_TOKEN = formatApiCredentialToken( + APPLICATION_CREDENTIAL_ID, + APPLICATION_CREDENTIAL_SECRET, +); function privateDirectory(directoryPath) { fs.mkdirSync(directoryPath, { recursive: true, mode: 0o700 }); @@ -25,6 +50,15 @@ function privateFile(filePath, content) { fs.chmodSync(filePath, 0o600); } +function privatizeTree(root) { + fs.chmodSync(root, 0o700); + for (const entry of fs.readdirSync(root, { withFileTypes: true })) { + const entryPath = path.join(root, entry.name); + if (entry.isDirectory()) privatizeTree(entryPath); + else fs.chmodSync(entryPath, 0o600); + } +} + function createLegacyDatabase(sourcePath) { const source = new DatabaseSync(sourcePath); source.exec(` @@ -371,6 +405,184 @@ function transformationOptions(value, prepared, staged, projectId) { }; } +async function provisionApplicationAuthority(value, role = 'owner') { + const ownerPepperKeyringDirectory = path.join( + value.deploymentRoot, + 'owner-keys', + ); + const credentialFilePath = path.join( + value.deploymentRoot, + 'owner-credential.json', + ); + const secretKeyringPath = path.join( + value.deploymentRoot, + 'local-secret-keyring.json', + ); + privateDirectory(ownerPepperKeyringDirectory); + await provisionLocalSecretKeyring(secretKeyringPath); + const pepper = provisionLocalOwnerPepperKey({ + keyringDirectory: ownerPepperKeyringDirectory, + pepperKeyId: APPLICATION_PEPPER_KEY_ID, + randomBytes: () => Buffer.alloc(32, 121), + }); + const now = Date.now(); + const database = new DatabaseSync(value.targetPath); + try { + database + .prepare( + `INSERT INTO "QingLong3LocalOwnerPepperKeys" ( + "pepper_key_id", "material_digest", "backup_digest", "state", + "version", "register_mutation_id", "activate_mutation_id", + "registered_at_ms", "activated_at_ms" + ) VALUES (?, ?, ?, 'active', 2, ?, ?, ?, ?)`, + ) + .run( + APPLICATION_PEPPER_KEY_ID, + pepper.digest, + 'e'.repeat(64), + 'd3870000-0000-4000-8000-000000000001', + 'd3870000-0000-4000-8000-000000000002', + now - 2_000, + now - 1_500, + ); + database + .prepare( + `INSERT INTO "QingLong3LocalOwnerPepperActivations" ( + "generation", "mutation_id", "expected_generation", + "previous_pepper_key_id", "active_pepper_key_id", + "material_digest", "backup_digest", "activated_at_ms" + ) VALUES (1, ?, 0, NULL, ?, ?, ?, ?)`, + ) + .run( + 'd3870000-0000-4000-8000-000000000002', + APPLICATION_PEPPER_KEY_ID, + pepper.digest, + 'e'.repeat(64), + now - 1_500, + ); + database + .prepare( + `INSERT INTO "QingLong3IdentitySubjects" ( + "subject_type", "subject_id", "status", "version", + "created_at_ms", "updated_at_ms" + ) VALUES ('user', 'data-adoption-owner', 'active', 1, ?, ?)`, + ) + .run(now - 1_000, now - 1_000); + database + .prepare( + `INSERT INTO "QingLong3ApiCredentials" ( + "credential_id", "version", "state", "subject_type", + "subject_id", "secret_digest", "created_at_ms", + "not_before_at_ms", "expires_at_ms" + ) VALUES (?, 1, 'active', 'user', 'data-adoption-owner', ?, ?, ?, ?)`, + ) + .run( + APPLICATION_CREDENTIAL_ID, + apiCredentialSecretDigest( + APPLICATION_PEPPER, + APPLICATION_CREDENTIAL_ID, + APPLICATION_CREDENTIAL_SECRET, + ), + now - 1_000, + now - 1_000, + now + 10 * 60 * 1_000, + ); + database + .prepare( + `INSERT INTO "QingLong3ApiCredentialPepperBindings" ( + "credential_id", "credential_version", "pepper_key_id" + ) VALUES (?, 1, ?)`, + ) + .run(APPLICATION_CREDENTIAL_ID, APPLICATION_PEPPER_KEY_ID); + database + .prepare( + `INSERT INTO "QingLong3ProjectRoleBindings" ( + "project_id", "subject_type", "subject_id", "version", "state", + "role", "mutation_id", "changed_by_type", "changed_by_id", + "created_at_ms" + ) VALUES ( + 'default', 'user', 'data-adoption-owner', 1, 'active', ?, + 'd387-owner-binding', 'user', 'data-adoption-owner', ? + )`, + ) + .run(role, now - 500); + } finally { + database.close(); + } + fs.chmodSync(value.targetPath, 0o600); + privateFile( + credentialFilePath, + `${JSON.stringify({ + schemaVersion: 1, + kind: 'qinglong3-local-identity-credential-presentation', + token: APPLICATION_TOKEN, + })}\n`, + ); + return { + ownerPepperKeyringDirectory, + credentialFilePath, + secretKeyringPath, + }; +} + +function applicationOptions( + value, + prepared, + staged, + authority, + transformationDigest, +) { + return { + ...transformationOptions(value, prepared, staged, 'default'), + expectedTransformationDigest: transformationDigest, + ...authority, + mutationId: 'd3870000-0000-4000-8000-000000000010', + failureAuditEventId: 'd3870000-0000-4000-8000-000000000011', + requestId: 'd387-data-directory-apply', + }; +} + +function applicationDatabaseRows(databasePath) { + const database = new DatabaseSync(databasePath, { readOnly: true }); + try { + return { + adoptions: database + .prepare( + `SELECT mutation_id AS "mutationId", model_json AS "modelJson", + receipt_digest AS "receiptDigest" + FROM "QingLong3LegacyDataDirectoryAdoptions"`, + ) + .all(), + items: database + .prepare( + `SELECT secret_name AS "secretName", value_digest AS "valueDigest" + FROM "QingLong3LegacyDataDirectoryAdoptionSecrets" + ORDER BY ordinal`, + ) + .all(), + envelopes: database + .prepare( + `SELECT secret_name AS "secretName", version, mutation_id AS "mutationId", + key_id AS "keyId", algorithm, nonce, ciphertext, + auth_tag AS "authTag", created_at_ms AS "createdAtMs" + FROM "QingLong3LocalSecretEnvelopes" + ORDER BY secret_name`, + ) + .all(), + audits: database + .prepare( + `SELECT operation_id AS "operationId", outcome + FROM "QingLong3SecurityAuditEvents" + WHERE operation_id IN ('legacy-data.apply', 'secret.create') + ORDER BY operation_id, event_id`, + ) + .all(), + }; + } finally { + database.close(); + } +} + test('stages only reviewed payloads behind the real SQLite activation fence', (t) => { const value = fixture(t); const prepared = prepare(value); @@ -927,3 +1139,285 @@ test('widened transformation commands fail closed before source access', (t) => 'LOCAL_DATA_DIRECTORY_ADOPTION_CONFIGURATION_INVALID', ); }); + +test('atomically applies a ready transformation, reclaims plaintext and exactly replays', async (t) => { + const value = fixture(t); + const secrets = configureTransformationInput(value); + const { prepared, staged } = stageForTransformation(value); + const authority = await provisionApplicationAuthority(value); + const transformed = run( + value, + 'directory-transform-before-apply', + DIRECTORY_TRANSFORM, + transformationOptions(value, prepared, staged, 'default'), + ).result; + const options = applicationOptions( + value, + prepared, + staged, + authority, + transformed.evidence.transformationDigest, + ); + const recoveryModel = path.join(value.deploymentRoot, 'recovery-model'); + fs.cpSync(path.join(value.transformationRoot, 'model'), recoveryModel, { + recursive: true, + preserveTimestamps: true, + }); + privatizeTree(recoveryModel); + + const applied = run(value, 'directory-apply', DIRECTORY_APPLY, options); + assert.equal(applied.result.status, 'committed'); + assert.equal(applied.result.evidence.databaseStatus, 'inserted'); + assert.equal(applied.result.evidence.secretCount, 2); + assert.equal(applied.result.evidence.environmentSecretCount, 1); + assert.equal(applied.result.evidence.sshSecretCount, 1); + assert.equal(applied.result.evidence.modelReclaimed, true); + assert.equal(applied.result.evidence.plaintextFilesRemoved, true); + assert.equal(applied.result.evidence.physicalErasureGuaranteed, false); + for (const sensitive of [ + secrets.environmentValue, + secrets.sshValue, + secrets.authValue, + secrets.environmentName, + secrets.sshAlias, + ]) { + assert.equal(applied.child.stdout.includes(sensitive), false); + } + assert.deepEqual(fs.readdirSync(value.transformationRoot).sort(), [ + 'commit.json', + 'manifest.json', + ]); + const commit = JSON.parse( + fs.readFileSync(path.join(value.transformationRoot, 'commit.json'), 'utf8'), + ); + assert.equal(commit.reclamation.modelRemoved, true); + assert.equal(commit.reclamation.plaintextFilesRemoved, true); + assert.equal(commit.reclamation.physicalErasureGuaranteed, false); + assert.equal(commit.commitDigest, applied.result.evidence.commitDigest); + + const rows = applicationDatabaseRows(value.targetPath); + assert.equal(rows.adoptions.length, 1); + assert.equal(rows.items.length, 2); + assert.equal(rows.envelopes.length, 2); + assert.deepEqual( + rows.audits.map(({ operationId, outcome }) => [operationId, outcome]), + [ + ['legacy-data.apply', 'allowed'], + ['secret.create', 'allowed'], + ['secret.create', 'allowed'], + ], + ); + const durableModel = JSON.parse(rows.adoptions[0].modelJson); + assert.equal(durableModel.activation, 'disabled'); + assert.equal(durableModel.config.activation, 'disabled'); + assert.equal(durableModel.keyv.activation, 'disabled'); + assert.equal(durableModel.ssh.activation, 'disabled'); + + const provider = new LocalSecretKeyringFileProvider( + authority.secretKeyringPath, + ); + const plaintexts = []; + for (const envelope of rows.envelopes) { + const material = await provider.resolve(envelope.keyId); + assert.ok(material); + const plaintext = decryptLocalSecretEnvelopeToBuffer( + { + projectId: 'default', + name: envelope.secretName, + version: envelope.version, + mutationId: envelope.mutationId, + keyId: envelope.keyId, + algorithm: envelope.algorithm, + nonce: Buffer.from(envelope.nonce).toString('base64url'), + ciphertext: Buffer.from(envelope.ciphertext).toString('base64url'), + authTag: Buffer.from(envelope.authTag).toString('base64url'), + createdAtMs: envelope.createdAtMs, + }, + material.key, + ); + try { + plaintexts.push(plaintext.toString('utf8')); + } finally { + plaintext.fill(0); + material.key.fill(0); + } + } + assert.deepEqual( + plaintexts.sort(), + [secrets.environmentValue, secrets.sshValue].sort(), + ); + + const replayed = run( + value, + 'directory-apply-replay', + DIRECTORY_APPLY, + options, + ).result; + assert.equal(replayed.evidence.databaseStatus, 'existing'); + assert.equal( + replayed.evidence.receiptDigest, + applied.result.evidence.receiptDigest, + ); + assert.equal( + replayed.evidence.commitDigest, + applied.result.evidence.commitDigest, + ); + const verified = run( + value, + 'directory-apply-verify', + DIRECTORY_APPLY_VERIFY, + { + ...options, + expectedReceiptDigest: applied.result.evidence.receiptDigest, + }, + ).result; + assert.equal(verified.status, 'verified'); + assert.equal(verified.evidence.databaseStatus, 'existing'); + assert.equal( + verified.evidence.commitDigest, + applied.result.evidence.commitDigest, + ); + + fs.unlinkSync(path.join(value.transformationRoot, 'commit.json')); + fs.renameSync( + recoveryModel, + path.join(value.transformationRoot, '.reclaiming-model'), + ); + privateFile( + path.join(value.transformationRoot, '.commit-incomplete'), + `${JSON.stringify({ + schemaVersion: 1, + kind: 'qinglong3-legacy-data-directory-application-incomplete', + mutationId: options.mutationId, + transformationDigest: options.expectedTransformationDigest, + receiptDigest: applied.result.evidence.receiptDigest, + })}\n`, + ); + const recovered = run( + value, + 'directory-apply-recover-renamed-model', + DIRECTORY_APPLY, + options, + ).result; + assert.equal(recovered.evidence.databaseStatus, 'existing'); + assert.equal( + recovered.evidence.commitDigest, + applied.result.evidence.commitDigest, + ); + assert.deepEqual(fs.readdirSync(value.transformationRoot).sort(), [ + 'commit.json', + 'manifest.json', + ]); +}); + +test('rolls back every new Secret when one application target already exists', async (t) => { + const value = fixture(t); + configureTransformationInput(value); + const { prepared, staged } = stageForTransformation(value); + const authority = await provisionApplicationAuthority(value); + const transformed = run( + value, + 'directory-transform-before-conflict', + DIRECTORY_TRANSFORM, + transformationOptions(value, prepared, staged, 'default'), + ).result; + const imports = JSON.parse( + fs.readFileSync( + path.join(value.transformationRoot, 'model', 'secret-imports.json'), + 'utf8', + ), + ).imports; + assert.equal(imports.length, 2); + const conflictingName = imports[1].targetName; + const database = new DatabaseSync(value.targetPath); + try { + database + .prepare( + `INSERT INTO "QingLong3LocalSecretEnvelopes" ( + "project_id", "secret_name", "version", "mutation_id", "key_id", + "algorithm", "nonce", "ciphertext", "auth_tag", "created_at_ms" + ) VALUES ('default', ?, 1, ?, 'preexisting-v1', 'aes-256-gcm', ?, ?, ?, ?)`, + ) + .run( + conflictingName, + 'd3870000-0000-4000-8000-000000000020', + Buffer.alloc(12, 1), + Buffer.alloc(0), + Buffer.alloc(16, 2), + Date.now(), + ); + } finally { + database.close(); + } + + const conflict = runRaw( + value, + 'directory-apply-conflict', + DIRECTORY_APPLY, + applicationOptions( + value, + prepared, + staged, + authority, + transformed.evidence.transformationDigest, + ), + ); + assert.equal(conflict.status, 1); + assert.equal(conflict.stdout, ''); + assert.equal( + JSON.parse(conflict.stderr).code, + 'LOCAL_DATA_DIRECTORY_ADOPTION_CONFLICT', + ); + const rows = applicationDatabaseRows(value.targetPath); + assert.equal(rows.adoptions.length, 0); + assert.equal(rows.items.length, 0); + assert.equal(rows.envelopes.length, 1); + assert.equal(rows.envelopes[0].secretName, conflictingName); + assert.equal( + fs.existsSync(path.join(value.transformationRoot, 'model')), + true, + ); +}); + +test('rejects a non-admin application without publishing or reclaiming the model', async (t) => { + const value = fixture(t); + configureTransformationInput(value); + const { prepared, staged } = stageForTransformation(value); + const authority = await provisionApplicationAuthority(value, 'viewer'); + const transformed = run( + value, + 'directory-transform-before-denial', + DIRECTORY_TRANSFORM, + transformationOptions(value, prepared, staged, 'default'), + ).result; + const denied = runRaw( + value, + 'directory-apply-denied', + DIRECTORY_APPLY, + applicationOptions( + value, + prepared, + staged, + authority, + transformed.evidence.transformationDigest, + ), + ); + assert.equal(denied.status, 1); + assert.equal(denied.stdout, ''); + assert.equal( + JSON.parse(denied.stderr).code, + 'LOCAL_DATA_DIRECTORY_ADOPTION_APPLICATION_FORBIDDEN', + ); + const rows = applicationDatabaseRows(value.targetPath); + assert.equal(rows.adoptions.length, 0); + assert.equal(rows.items.length, 0); + assert.equal(rows.envelopes.length, 0); + assert.deepEqual( + rows.audits.map(({ operationId, outcome }) => [operationId, outcome]), + [['legacy-data.apply', 'denied']], + ); + assert.equal( + fs.existsSync(path.join(value.transformationRoot, 'model')), + true, + ); +}); diff --git a/packages/ql3-local-owner-cli/test/localDeployment.test.cjs b/packages/ql3-local-owner-cli/test/localDeployment.test.cjs index 3e9d1d1e..8a95cbcd 100644 --- a/packages/ql3-local-owner-cli/test/localDeployment.test.cjs +++ b/packages/ql3-local-owner-cli/test/localDeployment.test.cjs @@ -475,9 +475,9 @@ function composeDockerHarness( '/opt/qinglong/node_modules/@qinglong/local-application/dist/cli.js', ], Labels: { - 'io.qinglong.local.sqlite-contract-min': '49', - 'io.qinglong.local.sqlite-contract-max': '49', - 'io.qinglong.local.sqlite-write-contract': '49', + 'io.qinglong.local.sqlite-contract-min': '50', + 'io.qinglong.local.sqlite-contract-max': '50', + 'io.qinglong.local.sqlite-write-contract': '50', 'io.qinglong.local.application-config': '2', 'io.qinglong.local.compose-selection': '1', 'io.qinglong.ai': 'excluded', @@ -1172,9 +1172,9 @@ test('preflights exact local image, Compose merge and SQLite capability', async '/opt/qinglong/node_modules/@qinglong/local-application/dist/cli.js', ], Labels: { - 'io.qinglong.local.sqlite-contract-min': '49', - 'io.qinglong.local.sqlite-contract-max': '49', - 'io.qinglong.local.sqlite-write-contract': '49', + 'io.qinglong.local.sqlite-contract-min': '50', + 'io.qinglong.local.sqlite-contract-max': '50', + 'io.qinglong.local.sqlite-write-contract': '50', 'io.qinglong.local.application-config': '2', 'io.qinglong.local.compose-selection': '1', 'io.qinglong.ai': 'excluded', @@ -1226,7 +1226,7 @@ test('preflights exact local image, Compose merge and SQLite capability', async assert.equal(result.status, 'ready'); assert.equal(result.generation, 1); assert.equal(result.profile, 'edge'); - assert.equal(result.sqlite.contractVersion, 49); + assert.equal(result.sqlite.contractVersion, 50); assert.equal(result.image.architecture, 'arm64'); assert.equal(calls.length, 2); assert.deepEqual(calls[0].slice(0, 2), ['image', 'inspect']); @@ -1326,8 +1326,8 @@ test('applies one Compose generation and exactly replays its health receipt', as assert.equal(mode(receiptPath), 0o600); const receipt = JSON.parse(fs.readFileSync(receiptPath, 'utf8')); assert.deepEqual(receipt.sqlite, { - contractVersion: 49, - writeContractVersion: 49, + contractVersion: 50, + writeContractVersion: 50, writeObservation: 'unchanged', backup: null, }); @@ -1628,8 +1628,8 @@ test('rolls a failed Compose candidate forward to a healthy prior digest', async `${command.request.rolloutId}.sqlite`, ); assert.equal(mode(backupPath), 0o600); - assert.equal(receipt.sqlite.contractVersion, 49); - assert.equal(receipt.sqlite.writeContractVersion, 49); + assert.equal(receipt.sqlite.contractVersion, 50); + assert.equal(receipt.sqlite.writeContractVersion, 50); assert.equal(receipt.sqlite.writeObservation, 'changed'); assert.match(receipt.sqlite.backup.sha256, /^[0-9a-f]{64}$/); assert.equal(receipt.sqlite.backup.bytes > 0, true); diff --git a/packages/ql3-local-owner-cli/test/localReadiness.test.cjs b/packages/ql3-local-owner-cli/test/localReadiness.test.cjs index 6bb9bceb..9be3c3a0 100644 --- a/packages/ql3-local-owner-cli/test/localReadiness.test.cjs +++ b/packages/ql3-local-owner-cli/test/localReadiness.test.cjs @@ -34,8 +34,8 @@ test('inspects the exact fresh Profile schema without exposing its path', async assert.equal(result.status, 'ready'); assert.equal(result.profile, 'edge'); assert.equal(result.storage.contractName, 'local-control-core'); - assert.equal(result.storage.contractVersion, 49); - assert.equal(result.storage.migrationCount, 98); + assert.equal(result.storage.contractVersion, 50); + assert.equal(result.storage.migrationCount, 100); assert.equal(result.storage.journalMode, 'delete'); assert.equal(JSON.stringify(result).includes(state.directory), false); }); diff --git a/packages/ql3-local-sqlite/package.json b/packages/ql3-local-sqlite/package.json index 1c7271ea..8e32d133 100644 --- a/packages/ql3-local-sqlite/package.json +++ b/packages/ql3-local-sqlite/package.json @@ -75,6 +75,11 @@ "require": "./dist/adoption/legacyAdoptionDatabase.js", "default": "./dist/adoption/legacyAdoptionDatabase.js" }, + "./data-directory-adoption": { + "types": "./dist/adoption/data-directory/dataDirectoryAdoptionDatabase.d.ts", + "require": "./dist/adoption/data-directory/dataDirectoryAdoptionDatabase.js", + "default": "./dist/adoption/data-directory/dataDirectoryAdoptionDatabase.js" + }, "./plugin-package-install": { "types": "./dist/plugin-package/pluginPackageInstallRepository.d.ts", "require": "./dist/plugin-package/pluginPackageInstallRepository.js", diff --git a/packages/ql3-local-sqlite/src/adoption/data-directory/dataDirectoryAdoptionDatabase.ts b/packages/ql3-local-sqlite/src/adoption/data-directory/dataDirectoryAdoptionDatabase.ts new file mode 100644 index 00000000..002778ac --- /dev/null +++ b/packages/ql3-local-sqlite/src/adoption/data-directory/dataDirectoryAdoptionDatabase.ts @@ -0,0 +1,994 @@ +import { createHash } from 'node:crypto'; +import type { DatabaseSync } from 'node:sqlite'; + +import { + createLocalSecretRef, + normalizeLocalSecretEnvelope, + type LocalSecretEnvelope, +} from '@qinglong/runtime-core/local-secret'; +import { + normalizeProjectPolicySubject, + type ProjectPolicyRepository, +} from '@qinglong/runtime-core/project-policy'; +import type { + SecurityPolicyFence, + SecuritySubject, +} from '@qinglong/runtime-core/security'; +import { + normalizeSecurityAuditRecord, + type SecurityAuditRecord, + type SecurityAuditSink, +} from '@qinglong/runtime-core/security-audit'; + +import { LocalSqliteOperationAuthority } from '../../authority/operationAuthority'; +import { + auditLocalSqliteReadiness, + type LocalSqliteReadinessEvidence, +} from '../../readiness/readiness'; +import { LocalSqliteSecurityAuthorityStore } from '../../security/securityAuthorityStore'; +import { + assertLocalSqliteOptions, + assertLocalSqlitePathBoundary, + openLocalSqliteClient, + type LocalSqliteDatabaseOptions, + type LocalSqliteProfile, +} from '../../storage/config'; + +const DIGEST_PATTERN = /^[0-9a-f]{64}$/; +const UUID_V4_PATTERN = + /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/; +const VALUE_FILE_PATTERN = /^secret-values\/[0-9a-f]{64}\.json$/; + +export interface LocalDataDirectoryAppliedModel { + readonly schema: 'qinglong/legacy-data-directory-applied-model@v1'; + readonly activation: 'disabled'; + readonly config: Readonly>; + readonly keyv: Readonly>; + readonly ssh: Readonly>; + readonly manualReview: Readonly>; +} + +export interface LocalDataDirectoryAdoptionSecretPublication { + readonly ordinal: number; + readonly kind: 'environment' | 'ssh_private_key'; + readonly sourceNameDigest: string; + readonly valueFile: string; + readonly valueDigest: string; + readonly envelope: Readonly; + readonly secretRef: string; + readonly itemDigest: string; + readonly audit: Readonly; +} + +export interface LocalDataDirectoryAdoptionSecretRecord { + readonly ordinal: number; + readonly kind: 'environment' | 'ssh_private_key'; + readonly sourceNameDigest: string; + readonly secretName: string; + readonly secretVersion: 1; + readonly secretMutationId: string; + readonly valueFile: string; + readonly valueDigest: string; + readonly secretRef: string; + readonly itemDigest: string; +} + +export interface LocalDataDirectoryAdoptionReceiptPayload { + readonly schema: 'qinglong/legacy-data-directory-adoption-receipt@v1'; + readonly mutationId: string; + readonly projectId: string; + readonly profile: LocalSqliteProfile; + readonly sourceStageManifestDigest: string; + readonly transformationDigest: string; + readonly modelDigest: string; + readonly secretCount: number; + readonly environmentSecretCount: number; + readonly sshSecretCount: number; + readonly items: readonly Readonly<{ + ordinal: number; + kind: LocalDataDirectoryAdoptionSecretRecord['kind']; + sourceNameDigest: string; + secretRef: string; + valueFile: string; + valueDigest: string; + itemDigest: string; + }>[]; + readonly publicationDigest: string; + readonly auditEventId: string; + readonly committedAtMs: number; +} + +export interface LocalDataDirectoryAdoptionReceipt + extends LocalDataDirectoryAdoptionReceiptPayload { + readonly receiptDigest: string; +} + +export interface LocalDataDirectoryAdoptionRecord { + readonly mutationId: string; + readonly projectId: string; + readonly profile: LocalSqliteProfile; + readonly sourceStageManifestDigest: string; + readonly transformationDigest: string; + readonly modelDigest: string; + readonly model: Readonly; + readonly publicationDigest: string; + readonly auditEventId: string; + readonly committedAtMs: number; + readonly receiptDigest: string; + readonly receipt: Readonly; + readonly secrets: readonly Readonly[]; +} + +export interface PublishLocalDataDirectoryAdoptionCommand { + readonly mutationId: string; + readonly projectId: string; + readonly profile: LocalSqliteProfile; + readonly sourceStageManifestDigest: string; + readonly transformationDigest: string; + readonly modelDigest: string; + readonly model: Readonly; + readonly subject: Readonly; + readonly fence: Readonly; + readonly audit: Readonly; + readonly secrets: readonly Readonly[]; + readonly receipt: Readonly; + readonly confirmExternalAuthority: () => void | Promise; +} + +export interface PublishLocalDataDirectoryAdoptionResult { + readonly status: 'inserted' | 'existing'; + readonly adoption: Readonly; +} + +export class LocalDataDirectoryAdoptionConflictError extends Error { + readonly code = 'LOCAL_DATA_DIRECTORY_ADOPTION_CONFLICT'; + + constructor() { + super('Local data directory adoption conflicts with durable state'); + this.name = 'LocalDataDirectoryAdoptionConflictError'; + } +} + +export class LocalDataDirectoryAdoptionAuthorizationFenceConflictError extends Error { + readonly code = 'LOCAL_DATA_DIRECTORY_ADOPTION_AUTHORIZATION_FENCE_CONFLICT'; + + constructor() { + super('Local data directory adoption authorization fence changed'); + this.name = 'LocalDataDirectoryAdoptionAuthorizationFenceConflictError'; + } +} + +export class LocalDataDirectoryAdoptionUnavailableError extends Error { + readonly code = 'LOCAL_DATA_DIRECTORY_ADOPTION_UNAVAILABLE'; + + constructor(readonly cause?: unknown) { + super('Local data directory adoption storage is unavailable'); + this.name = 'LocalDataDirectoryAdoptionUnavailableError'; + } +} + +type Row = Record; + +function sha256(domain: string, value: string): string { + return createHash('sha256') + .update(domain, 'utf8') + .update(value, 'utf8') + .digest('hex'); +} + +function exactKeys(value: object, expected: readonly string[]): boolean { + const actual = Object.keys(value).sort(); + const canonical = [...expected].sort(); + return ( + actual.length === canonical.length && + actual.every((key, index) => key === canonical[index]) + ); +} + +function safeInteger(value: unknown, minimum = 0): value is number { + return Number.isSafeInteger(value) && (value as number) >= minimum; +} + +function text(row: Row, key: string): string { + const value = row[key]; + if (typeof value !== 'string') { + throw new LocalDataDirectoryAdoptionUnavailableError(); + } + return value; +} + +function integer(row: Row, key: string): number { + const value = row[key]; + if (!Number.isSafeInteger(value)) { + throw new LocalDataDirectoryAdoptionUnavailableError(); + } + return value as number; +} + +function json(row: Row, key: string): unknown { + try { + return JSON.parse(text(row, key)); + } catch (error) { + throw new LocalDataDirectoryAdoptionUnavailableError(error); + } +} + +function assertModel( + value: unknown, +): asserts value is LocalDataDirectoryAppliedModel { + if ( + !value || + typeof value !== 'object' || + Array.isArray(value) || + !exactKeys(value, [ + 'activation', + 'config', + 'keyv', + 'manualReview', + 'schema', + 'ssh', + ]) + ) { + throw new LocalDataDirectoryAdoptionConflictError(); + } + const model = value as Record; + const entries = [model.config, model.keyv, model.ssh, model.manualReview]; + if ( + model.schema !== 'qinglong/legacy-data-directory-applied-model@v1' || + model.activation !== 'disabled' || + entries.some( + (entry) => !entry || typeof entry !== 'object' || Array.isArray(entry), + ) || + (model.config as Row).schema !== + 'qinglong/legacy-config-transformation@v1' || + (model.config as Row).activation !== 'disabled' || + (model.keyv as Row).schema !== 'qinglong/legacy-keyv-transformation@v1' || + (model.keyv as Row).activation !== 'disabled' || + (model.ssh as Row).schema !== 'qinglong/legacy-ssh-transformation@v1' || + (model.ssh as Row).activation !== 'disabled' || + (model.manualReview as Row).schema !== + 'qinglong/legacy-data-directory-manual-review@v1' || + (model.manualReview as Row).required !== false || + (model.manualReview as Row).activation !== 'disabled' || + Buffer.byteLength(JSON.stringify(value), 'utf8') > 1024 * 1024 + ) { + throw new LocalDataDirectoryAdoptionConflictError(); + } +} + +function itemSemantic( + item: Omit, +): string { + return JSON.stringify(item); +} + +export function createLocalDataDirectorySourceNameDigest( + kind: LocalDataDirectoryAdoptionSecretRecord['kind'], + sourceName: string, +): string { + if ( + (kind !== 'environment' && kind !== 'ssh_private_key') || + typeof sourceName !== 'string' || + sourceName.length < 1 || + sourceName.includes('\0') + ) { + throw new LocalDataDirectoryAdoptionConflictError(); + } + return createHash('sha256') + .update('qinglong3.legacy-data-directory-source-name.v1\0') + .update(kind) + .update('\0') + .update(sourceName) + .digest('hex'); +} + +export function createLocalDataDirectoryAdoptionSecretItem(options: { + readonly projectId: string; + readonly ordinal: number; + readonly kind: LocalDataDirectoryAdoptionSecretRecord['kind']; + readonly sourceNameDigest: string; + readonly secretName: string; + readonly secretMutationId: string; + readonly valueFile: string; + readonly valueDigest: string; +}): Readonly { + return createSecretRecord(options); +} + +function createSecretRecord(options: { + readonly projectId: string; + readonly ordinal: number; + readonly kind: LocalDataDirectoryAdoptionSecretRecord['kind']; + readonly sourceNameDigest: string; + readonly secretName: string; + readonly secretMutationId: string; + readonly valueFile: string; + readonly valueDigest: string; +}): Readonly { + const secretRef = createLocalSecretRef({ + projectId: options.projectId, + name: options.secretName, + version: 1, + }); + const semantic = Object.freeze({ + ordinal: options.ordinal, + kind: options.kind, + sourceNameDigest: options.sourceNameDigest, + secretName: options.secretName, + secretVersion: 1 as const, + secretMutationId: options.secretMutationId, + valueFile: options.valueFile, + valueDigest: options.valueDigest, + secretRef, + }); + return Object.freeze({ + ...semantic, + itemDigest: sha256( + 'qinglong3.legacy-data-directory-adoption-secret-item.v1\0', + itemSemantic(semantic), + ), + }); +} + +function publicationDigest(options: { + readonly mutationId: string; + readonly projectId: string; + readonly sourceStageManifestDigest: string; + readonly transformationDigest: string; + readonly modelDigest: string; + readonly secrets: readonly Readonly[]; +}): string { + const hash = createHash('sha256') + .update('qinglong3.legacy-data-directory-adoption-publication.v1\0') + .update(options.mutationId) + .update('\0') + .update(options.projectId) + .update('\0') + .update(options.sourceStageManifestDigest) + .update('\0') + .update(options.transformationDigest) + .update('\0') + .update(options.modelDigest); + for (const item of options.secrets) hash.update('\0').update(item.itemDigest); + return hash.digest('hex'); +} + +export function createLocalDataDirectoryAdoptionReceipt(options: { + readonly mutationId: string; + readonly projectId: string; + readonly profile: LocalSqliteProfile; + readonly sourceStageManifestDigest: string; + readonly transformationDigest: string; + readonly modelDigest: string; + readonly secrets: readonly Readonly[]; + readonly committedAtMs: number; +}): Readonly { + const environmentSecretCount = options.secrets.filter( + ({ kind }) => kind === 'environment', + ).length; + const sshSecretCount = options.secrets.length - environmentSecretCount; + const publication = publicationDigest(options); + const payload: LocalDataDirectoryAdoptionReceiptPayload = { + schema: 'qinglong/legacy-data-directory-adoption-receipt@v1', + mutationId: options.mutationId, + projectId: options.projectId, + profile: options.profile, + sourceStageManifestDigest: options.sourceStageManifestDigest, + transformationDigest: options.transformationDigest, + modelDigest: options.modelDigest, + secretCount: options.secrets.length, + environmentSecretCount, + sshSecretCount, + items: Object.freeze( + options.secrets.map( + ({ + ordinal, + kind, + sourceNameDigest, + secretRef, + valueFile, + valueDigest, + itemDigest, + }) => + Object.freeze({ + ordinal, + kind, + sourceNameDigest, + secretRef, + valueFile, + valueDigest, + itemDigest, + }), + ), + ), + publicationDigest: publication, + auditEventId: options.mutationId, + committedAtMs: options.committedAtMs, + }; + return Object.freeze({ + ...payload, + receiptDigest: sha256( + 'qinglong3.legacy-data-directory-adoption-receipt.v1\0', + JSON.stringify(payload), + ), + }); +} + +function sameJson(left: unknown, right: unknown): boolean { + return JSON.stringify(left) === JSON.stringify(right); +} + +function parseRecord( + client: DatabaseSync, + row: Row, +): LocalDataDirectoryAdoptionRecord { + const profile = text(row, 'profile'); + if (profile !== 'edge' && profile !== 'standalone') { + throw new LocalDataDirectoryAdoptionUnavailableError(); + } + const mutationId = text(row, 'mutationId'); + const projectId = text(row, 'projectId'); + const model = json(row, 'modelJson'); + assertModel(model); + const itemRows = client + .prepare( + `SELECT item."ordinal" AS "ordinal", item."kind" AS "kind", + item."source_name_digest" AS "sourceNameDigest", + item."secret_name" AS "secretName", + item."secret_version" AS "secretVersion", + item."secret_mutation_id" AS "secretMutationId", + item."value_file" AS "valueFile", + item."value_digest" AS "valueDigest", + item."secret_ref" AS "secretRef", + item."item_digest" AS "itemDigest" + FROM "QingLong3LegacyDataDirectoryAdoptionSecrets" AS item + JOIN "QingLong3LocalSecretEnvelopes" AS secret + ON secret."project_id" = item."project_id" + AND secret."secret_name" = item."secret_name" + AND secret."version" = item."secret_version" + AND secret."mutation_id" = item."secret_mutation_id" + JOIN "QingLong3SecurityAuditEvents" AS audit + ON audit."event_id" = item."secret_mutation_id" + AND audit."project_id" = item."project_id" + AND audit."operation_id" = 'secret.create' + AND audit."outcome" = 'allowed' + WHERE item."adoption_mutation_id" = ? + ORDER BY item."ordinal"`, + ) + .all(mutationId) as Row[]; + const secrets = Object.freeze( + itemRows.map((item, index) => { + const kind = text(item, 'kind'); + const candidate = createSecretRecord({ + projectId, + ordinal: integer(item, 'ordinal'), + kind: + kind === 'environment' || kind === 'ssh_private_key' + ? kind + : (() => { + throw new LocalDataDirectoryAdoptionUnavailableError(); + })(), + sourceNameDigest: text(item, 'sourceNameDigest'), + secretName: text(item, 'secretName'), + secretMutationId: text(item, 'secretMutationId'), + valueFile: text(item, 'valueFile'), + valueDigest: text(item, 'valueDigest'), + }); + if ( + candidate.ordinal !== index + 1 || + candidate.secretVersion !== integer(item, 'secretVersion') || + candidate.secretRef !== text(item, 'secretRef') || + candidate.itemDigest !== text(item, 'itemDigest') + ) { + throw new LocalDataDirectoryAdoptionUnavailableError(); + } + return candidate; + }), + ); + const sourceStageManifestDigest = text(row, 'sourceStageManifestDigest'); + const transformationDigest = text(row, 'transformationDigest'); + const modelDigest = text(row, 'modelDigest'); + const committedAtMs = integer(row, 'committedAtMs'); + const receipt = createLocalDataDirectoryAdoptionReceipt({ + mutationId, + projectId, + profile, + sourceStageManifestDigest, + transformationDigest, + modelDigest, + secrets, + committedAtMs, + }); + const storedReceipt = json(row, 'receiptJson'); + if ( + secrets.length !== integer(row, 'secretCount') || + receipt.environmentSecretCount !== integer(row, 'environmentSecretCount') || + receipt.sshSecretCount !== integer(row, 'sshSecretCount') || + receipt.publicationDigest !== text(row, 'publicationDigest') || + receipt.auditEventId !== text(row, 'auditEventId') || + receipt.receiptDigest !== text(row, 'receiptDigest') || + !sameJson(receipt, storedReceipt) + ) { + throw new LocalDataDirectoryAdoptionUnavailableError(); + } + return Object.freeze({ + mutationId, + projectId, + profile, + sourceStageManifestDigest, + transformationDigest, + modelDigest, + model, + publicationDigest: receipt.publicationDigest, + auditEventId: receipt.auditEventId, + committedAtMs, + receiptDigest: receipt.receiptDigest, + receipt, + secrets, + }); +} + +const RECORD_SELECT = ` + adoption."mutation_id" AS "mutationId", + adoption."project_id" AS "projectId", + adoption."profile" AS "profile", + adoption."source_stage_manifest_digest" AS "sourceStageManifestDigest", + adoption."transformation_digest" AS "transformationDigest", + adoption."model_digest" AS "modelDigest", + adoption."secret_count" AS "secretCount", + adoption."environment_secret_count" AS "environmentSecretCount", + adoption."ssh_secret_count" AS "sshSecretCount", + adoption."model_json" AS "modelJson", + adoption."publication_digest" AS "publicationDigest", + adoption."audit_event_id" AS "auditEventId", + adoption."committed_at_ms" AS "committedAtMs", + adoption."receipt_digest" AS "receiptDigest", + adoption."receipt_json" AS "receiptJson"`; + +function findRecord( + client: DatabaseSync, + mutationId: string, + transformationDigest?: string, +): LocalDataDirectoryAdoptionRecord | null { + const rows = transformationDigest + ? (client + .prepare( + `SELECT ${RECORD_SELECT} + FROM "QingLong3LegacyDataDirectoryAdoptions" AS adoption + JOIN "QingLong3SecurityAuditEvents" AS audit + ON audit."event_id" = adoption."audit_event_id" + AND audit."project_id" = adoption."project_id" + AND audit."operation_id" = 'legacy-data.apply' + AND audit."outcome" = 'allowed' + WHERE adoption."mutation_id" = ? + OR adoption."transformation_digest" = ? + LIMIT 2`, + ) + .all(mutationId, transformationDigest) as Row[]) + : (client + .prepare( + `SELECT ${RECORD_SELECT} + FROM "QingLong3LegacyDataDirectoryAdoptions" AS adoption + JOIN "QingLong3SecurityAuditEvents" AS audit + ON audit."event_id" = adoption."audit_event_id" + AND audit."project_id" = adoption."project_id" + AND audit."operation_id" = 'legacy-data.apply' + AND audit."outcome" = 'allowed' + WHERE adoption."mutation_id" = ? + LIMIT 2`, + ) + .all(mutationId) as Row[]); + if (rows.length === 0) return null; + if (rows.length !== 1) throw new LocalDataDirectoryAdoptionConflictError(); + return parseRecord(client, rows[0]!); +} + +function insertAudit(client: DatabaseSync, audit: SecurityAuditRecord): void { + client + .prepare( + `INSERT INTO "QingLong3SecurityAuditEvents" ( + "event_id", "request_id", "operation_id", "project_id", + "subject_type", "subject_id", "authentication_id", "outcome", + "reasons_json", "fence_project_version", "fence_binding_version", + "occurred_at_ms" + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .run( + audit.eventId, + audit.requestId, + audit.operationId, + audit.projectId, + audit.subject?.type ?? null, + audit.subject?.id ?? null, + audit.authenticationId, + audit.outcome, + JSON.stringify(audit.reasons), + audit.fence?.projectVersion ?? null, + audit.fence?.bindingVersion ?? null, + audit.occurredAtMs, + ); +} + +function assertCommand(command: PublishLocalDataDirectoryAdoptionCommand): { + readonly subject: Readonly; + readonly audit: Readonly; + readonly secrets: readonly Readonly<{ + publication: LocalDataDirectoryAdoptionSecretPublication; + record: LocalDataDirectoryAdoptionSecretRecord; + envelope: LocalSecretEnvelope; + audit: SecurityAuditRecord; + }>[]; +} { + if ( + !command || + typeof command !== 'object' || + !UUID_V4_PATTERN.test(command.mutationId) || + (command.profile !== 'edge' && command.profile !== 'standalone') || + ![ + command.sourceStageManifestDigest, + command.transformationDigest, + command.modelDigest, + ].every((value) => DIGEST_PATTERN.test(value)) || + !Array.isArray(command.secrets) || + command.secrets.length > (command.profile === 'edge' ? 128 : 512) || + typeof command.confirmExternalAuthority !== 'function' + ) { + throw new LocalDataDirectoryAdoptionConflictError(); + } + assertModel(command.model); + const subject = normalizeProjectPolicySubject(command.subject); + const audit = normalizeSecurityAuditRecord(command.audit); + if ( + !command.fence || + !safeInteger(command.fence.projectVersion, 1) || + !safeInteger(command.fence.bindingVersion, 1) || + audit.eventId !== command.mutationId || + audit.operationId !== 'legacy-data.apply' || + audit.projectId !== command.projectId || + audit.subject?.type !== subject.type || + audit.subject.id !== subject.id || + audit.outcome !== 'allowed' || + audit.fence?.projectVersion !== command.fence.projectVersion || + audit.fence.bindingVersion !== command.fence.bindingVersion || + audit.occurredAtMs !== command.receipt.committedAtMs + ) { + throw new LocalDataDirectoryAdoptionConflictError(); + } + const names = new Set(); + const mutations = new Set([command.mutationId]); + const secrets = command.secrets.map((publication, index) => { + const envelope = normalizeLocalSecretEnvelope(publication.envelope); + const itemAudit = normalizeSecurityAuditRecord(publication.audit); + const record = createSecretRecord({ + projectId: command.projectId, + ordinal: publication.ordinal, + kind: publication.kind, + sourceNameDigest: publication.sourceNameDigest, + secretName: envelope.name, + secretMutationId: envelope.mutationId, + valueFile: publication.valueFile, + valueDigest: publication.valueDigest, + }); + if ( + publication.ordinal !== index + 1 || + envelope.projectId !== command.projectId || + envelope.version !== 1 || + !UUID_V4_PATTERN.test(envelope.mutationId) || + names.has(envelope.name) || + mutations.has(envelope.mutationId) || + !DIGEST_PATTERN.test(publication.sourceNameDigest) || + !DIGEST_PATTERN.test(publication.valueDigest) || + !VALUE_FILE_PATTERN.test(publication.valueFile) || + publication.secretRef !== record.secretRef || + publication.itemDigest !== record.itemDigest || + itemAudit.eventId !== envelope.mutationId || + itemAudit.operationId !== 'secret.create' || + itemAudit.projectId !== command.projectId || + itemAudit.subject?.type !== subject.type || + itemAudit.subject.id !== subject.id || + itemAudit.outcome !== 'allowed' || + itemAudit.fence?.projectVersion !== command.fence.projectVersion || + itemAudit.fence.bindingVersion !== command.fence.bindingVersion || + itemAudit.occurredAtMs !== command.receipt.committedAtMs + ) { + throw new LocalDataDirectoryAdoptionConflictError(); + } + names.add(envelope.name); + mutations.add(envelope.mutationId); + return Object.freeze({ publication, record, envelope, audit: itemAudit }); + }); + const expectedReceipt = createLocalDataDirectoryAdoptionReceipt({ + mutationId: command.mutationId, + projectId: command.projectId, + profile: command.profile, + sourceStageManifestDigest: command.sourceStageManifestDigest, + transformationDigest: command.transformationDigest, + modelDigest: command.modelDigest, + secrets: secrets.map(({ record }) => record), + committedAtMs: command.receipt.committedAtMs, + }); + if (!sameJson(expectedReceipt, command.receipt)) { + throw new LocalDataDirectoryAdoptionConflictError(); + } + return Object.freeze({ subject, audit, secrets: Object.freeze(secrets) }); +} + +function exactReplay( + existing: Readonly, + command: Readonly, +): boolean { + return ( + existing.mutationId === command.mutationId && + existing.projectId === command.projectId && + existing.profile === command.profile && + existing.sourceStageManifestDigest === command.sourceStageManifestDigest && + existing.transformationDigest === command.transformationDigest && + existing.modelDigest === command.modelDigest && + sameJson(existing.model, command.model) && + sameJson(existing.receipt, command.receipt) + ); +} + +export class LocalSqliteDataDirectoryAdoptionPublisher { + constructor(private readonly authority: LocalSqliteOperationAuthority) {} + + resolve( + mutationId: string, + ): Promise | null> { + if (!UUID_V4_PATTERN.test(mutationId)) { + return Promise.reject(new LocalDataDirectoryAdoptionConflictError()); + } + return this.authority.enqueue( + async () => findRecord(this.authority.client, mutationId), + () => new LocalDataDirectoryAdoptionUnavailableError(), + ); + } + + publish( + command: Readonly, + ): Promise { + const normalized = assertCommand(command); + return this.authority.enqueue( + async () => { + const client = this.authority.client; + let began = false; + try { + client.exec('BEGIN IMMEDIATE'); + began = true; + const replay = findRecord( + client, + command.mutationId, + command.transformationDigest, + ); + if (replay) { + if (!exactReplay(replay, command)) { + throw new LocalDataDirectoryAdoptionConflictError(); + } + await command.confirmExternalAuthority(); + client.exec('COMMIT'); + began = false; + return Object.freeze({ + status: 'existing' as const, + adoption: replay, + }); + } + + const project = client + .prepare( + `SELECT "version", "status" FROM "QingLong3Projects" + WHERE "id" = ? LIMIT 1`, + ) + .get(command.projectId) as Row | undefined; + if ( + !project || + integer(project, 'version') !== command.fence.projectVersion || + text(project, 'status') !== 'active' + ) { + throw new LocalDataDirectoryAdoptionAuthorizationFenceConflictError(); + } + const binding = client + .prepare( + `SELECT "version", "state", "role" + FROM "QingLong3ProjectRoleBindings" + WHERE "project_id" = ? AND "subject_type" = ? AND "subject_id" = ? + ORDER BY "version" DESC LIMIT 1`, + ) + .get( + command.projectId, + normalized.subject.type, + normalized.subject.id, + ) as Row | undefined; + if ( + !binding || + integer(binding, 'version') !== command.fence.bindingVersion || + text(binding, 'state') !== 'active' || + !['owner', 'admin'].includes(text(binding, 'role')) + ) { + throw new LocalDataDirectoryAdoptionAuthorizationFenceConflictError(); + } + + for (const { envelope, audit } of normalized.secrets) { + const current = client + .prepare( + `SELECT MAX("version") AS "version" + FROM "QingLong3LocalSecretEnvelopes" + WHERE "project_id" = ? AND "secret_name" = ?`, + ) + .get(envelope.projectId, envelope.name) as Row; + if (current.version !== null) { + throw new LocalDataDirectoryAdoptionConflictError(); + } + const nonce = Buffer.from(envelope.nonce, 'base64url'); + const ciphertext = Buffer.from(envelope.ciphertext, 'base64url'); + const authTag = Buffer.from(envelope.authTag, 'base64url'); + try { + client + .prepare( + `INSERT INTO "QingLong3LocalSecretEnvelopes" ( + "project_id", "secret_name", "version", "mutation_id", + "key_id", "algorithm", "nonce", "ciphertext", "auth_tag", + "created_at_ms" + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .run( + envelope.projectId, + envelope.name, + envelope.version, + envelope.mutationId, + envelope.keyId, + envelope.algorithm, + nonce, + ciphertext, + authTag, + envelope.createdAtMs, + ); + } finally { + nonce.fill(0); + ciphertext.fill(0); + authTag.fill(0); + } + insertAudit(client, audit); + } + insertAudit(client, normalized.audit); + client + .prepare( + `INSERT INTO "QingLong3LegacyDataDirectoryAdoptions" ( + "mutation_id", "project_id", "profile", + "source_stage_manifest_digest", "transformation_digest", + "model_digest", "secret_count", "environment_secret_count", + "ssh_secret_count", "model_json", "publication_digest", + "audit_event_id", "committed_at_ms", "receipt_digest", + "receipt_json" + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .run( + command.mutationId, + command.projectId, + command.profile, + command.sourceStageManifestDigest, + command.transformationDigest, + command.modelDigest, + command.receipt.secretCount, + command.receipt.environmentSecretCount, + command.receipt.sshSecretCount, + JSON.stringify(command.model), + command.receipt.publicationDigest, + command.mutationId, + command.receipt.committedAtMs, + command.receipt.receiptDigest, + JSON.stringify(command.receipt), + ); + for (const { record } of normalized.secrets) { + client + .prepare( + `INSERT INTO "QingLong3LegacyDataDirectoryAdoptionSecrets" ( + "adoption_mutation_id", "ordinal", "project_id", "kind", + "source_name_digest", "secret_name", "secret_version", + "secret_mutation_id", "value_file", "value_digest", + "secret_ref", "item_digest" + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .run( + command.mutationId, + record.ordinal, + command.projectId, + record.kind, + record.sourceNameDigest, + record.secretName, + record.secretVersion, + record.secretMutationId, + record.valueFile, + record.valueDigest, + record.secretRef, + record.itemDigest, + ); + } + const stored = findRecord(client, command.mutationId); + if (!stored || !exactReplay(stored, command)) { + throw new LocalDataDirectoryAdoptionUnavailableError(); + } + await command.confirmExternalAuthority(); + client.exec('COMMIT'); + began = false; + return Object.freeze({ + status: 'inserted' as const, + adoption: stored, + }); + } catch (error) { + if (began && client.isTransaction) { + try { + client.exec('ROLLBACK'); + } catch { + // Preserve the original failure. + } + } + if ( + error instanceof LocalDataDirectoryAdoptionConflictError || + error instanceof + LocalDataDirectoryAdoptionAuthorizationFenceConflictError || + error instanceof LocalDataDirectoryAdoptionUnavailableError + ) { + throw error; + } + if ( + error && + typeof error === 'object' && + 'code' in error && + typeof error.code === 'string' && + error.code.startsWith('SQLITE_CONSTRAINT') + ) { + throw new LocalDataDirectoryAdoptionConflictError(); + } + throw new LocalDataDirectoryAdoptionUnavailableError(error); + } + }, + () => new LocalDataDirectoryAdoptionUnavailableError(), + ); + } +} + +export interface LocalSqliteDataDirectoryAdoptionDatabase { + readonly profile: LocalSqliteProfile; + readonly readiness: LocalSqliteReadinessEvidence; + readonly projectPolicy: ProjectPolicyRepository; + readonly securityAudit: SecurityAuditSink; + readonly publisher: LocalSqliteDataDirectoryAdoptionPublisher; + close(): Promise; +} + +/** Short-lived data-directory adoption authority; runtime hosts must not import it. */ +export async function openLocalSqliteDataDirectoryAdoptionDatabase( + options: LocalSqliteDatabaseOptions, +): Promise { + assertLocalSqliteOptions(options); + assertLocalSqlitePathBoundary(options.databasePath, false); + const client = openLocalSqliteClient(options, false); + try { + const readiness = await auditLocalSqliteReadiness(client); + const authority = new LocalSqliteOperationAuthority(client); + const securityAuthority = new LocalSqliteSecurityAuthorityStore(authority); + const projectPolicy: ProjectPolicyRepository = Object.freeze({ + resolve: ( + ...[projectId, subject]: Parameters + ) => securityAuthority.resolve(projectId, subject), + append: (...[command]: Parameters) => + securityAuthority.append(command), + }); + let closePromise: Promise | undefined; + return Object.freeze({ + profile: options.profile, + readiness, + projectPolicy, + securityAudit: securityAuthority, + publisher: new LocalSqliteDataDirectoryAdoptionPublisher(authority), + close() { + if (closePromise) return closePromise; + closePromise = authority.close(); + return closePromise; + }, + }); + } catch (error) { + if (client.isOpen) client.close(); + throw error; + } +} diff --git a/packages/ql3-local-sqlite/src/adoption/data-directory/dataDirectoryAdoptionSchemaContract.ts b/packages/ql3-local-sqlite/src/adoption/data-directory/dataDirectoryAdoptionSchemaContract.ts new file mode 100644 index 00000000..9dd078c2 --- /dev/null +++ b/packages/ql3-local-sqlite/src/adoption/data-directory/dataDirectoryAdoptionSchemaContract.ts @@ -0,0 +1,17 @@ +export const LOCAL_DATA_DIRECTORY_ADOPTION_SECRET_GUARD_TRIGGER_SQL = ` +CREATE TRIGGER "ql3_legacy_data_directory_adoption_secret_guard" +BEFORE INSERT ON "QingLong3LegacyDataDirectoryAdoptionSecrets" +BEGIN + SELECT CASE WHEN NOT EXISTS ( + SELECT 1 + FROM "QingLong3LegacyDataDirectoryAdoptions" AS adoption + JOIN "QingLong3LocalSecretEnvelopes" AS secret + ON secret."project_id" = NEW."project_id" + AND secret."secret_name" = NEW."secret_name" + AND secret."version" = NEW."secret_version" + WHERE adoption."mutation_id" = NEW."adoption_mutation_id" + AND adoption."project_id" = NEW."project_id" + AND secret."mutation_id" = NEW."secret_mutation_id" + ) THEN RAISE(ABORT, 'legacy data directory adoption Secret authority mismatch') END; +END +`.trim(); diff --git a/packages/ql3-local-sqlite/src/migration/migration.ts b/packages/ql3-local-sqlite/src/migration/migration.ts index b782da3f..a4f9d108 100644 --- a/packages/ql3-local-sqlite/src/migration/migration.ts +++ b/packages/ql3-local-sqlite/src/migration/migration.ts @@ -108,6 +108,8 @@ import { local0095PluginPackageSecretBindingTargetGuardMigration } from '../migr import { local0096CapabilityV48Migration } from '../migrations/0096-capability-v48'; import { local0097PluginPackageSecretBindingTransitionReceiptsMigration } from '../migrations/0097-plugin-package-secret-binding-transition-receipts'; import { local0098CapabilityV49Migration } from '../migrations/0098-capability-v49'; +import { local0099LegacyDataDirectoryAdoptionsMigration } from '../migrations/0099-legacy-data-directory-adoptions'; +import { local0100CapabilityV50Migration } from '../migrations/0100-capability-v50'; import type { LocalSqliteMigrationContext } from '../migrations/sqlMigration'; import { LOCAL_SQLITE_MIGRATION_STREAM_ID, @@ -228,6 +230,8 @@ export const localSqliteMigrationDefinition: MigrationStreamDefinition= 0 + ) +) + `, + ` +CREATE UNIQUE INDEX "ql3_legacy_data_directory_adoption_transformation_uidx" +ON "QingLong3LegacyDataDirectoryAdoptions" ("transformation_digest") + `, + ` +CREATE UNIQUE INDEX "ql3_legacy_data_directory_adoption_receipt_uidx" +ON "QingLong3LegacyDataDirectoryAdoptions" ("receipt_digest") + `, + ` +CREATE INDEX "ql3_legacy_data_directory_adoption_project_time_idx" +ON "QingLong3LegacyDataDirectoryAdoptions" ( + "project_id", "committed_at_ms" DESC, "mutation_id" DESC +) + `, + ` +CREATE TABLE "QingLong3LegacyDataDirectoryAdoptionSecrets" ( + "adoption_mutation_id" TEXT NOT NULL, + "ordinal" INTEGER NOT NULL, + "project_id" TEXT NOT NULL, + "kind" TEXT NOT NULL, + "source_name_digest" TEXT NOT NULL, + "secret_name" TEXT NOT NULL, + "secret_version" INTEGER NOT NULL, + "secret_mutation_id" TEXT NOT NULL, + "value_file" TEXT NOT NULL, + "value_digest" TEXT NOT NULL, + "secret_ref" TEXT NOT NULL, + "item_digest" TEXT NOT NULL, + PRIMARY KEY ("adoption_mutation_id", "ordinal"), + CONSTRAINT ql3_legacy_data_directory_adoption_secret_parent_fk + FOREIGN KEY ("adoption_mutation_id") + REFERENCES "QingLong3LegacyDataDirectoryAdoptions" ("mutation_id") + ON DELETE RESTRICT ON UPDATE RESTRICT, + CONSTRAINT ql3_legacy_data_directory_adoption_secret_envelope_fk + FOREIGN KEY ("project_id", "secret_name", "secret_version") + REFERENCES "QingLong3LocalSecretEnvelopes" ("project_id", "secret_name", "version") + ON DELETE RESTRICT ON UPDATE RESTRICT, + CONSTRAINT ql3_legacy_data_directory_adoption_secret_audit_fk + FOREIGN KEY ("secret_mutation_id") + REFERENCES "QingLong3SecurityAuditEvents" ("event_id") + ON DELETE RESTRICT ON UPDATE RESTRICT, + CONSTRAINT ql3_legacy_data_directory_adoption_secret_identity_check CHECK ( + "ordinal" BETWEEN 1 AND 512 AND + length("project_id") BETWEEN 1 AND 128 AND + "kind" IN ('environment', 'ssh_private_key') AND + length("secret_name") BETWEEN 1 AND 128 AND + "secret_version" = 1 AND + length("secret_mutation_id") = 36 AND + substr("secret_mutation_id", 15, 1) = '4' AND + replace("secret_mutation_id", '-', '') NOT GLOB '*[^0-9a-f]*' AND + length("value_file") = 83 AND + "value_file" GLOB 'secret-values/[0-9a-f]*.json' AND + length("secret_ref") BETWEEN 1 AND 512 + ), + CONSTRAINT ql3_legacy_data_directory_adoption_secret_digest_check CHECK ( + length("source_name_digest") = 64 AND + "source_name_digest" NOT GLOB '*[^0-9a-f]*' AND + length("value_digest") = 64 AND + "value_digest" NOT GLOB '*[^0-9a-f]*' AND + length("item_digest") = 64 AND + "item_digest" NOT GLOB '*[^0-9a-f]*' + ) +) + `, + ` +CREATE UNIQUE INDEX "ql3_legacy_data_directory_adoption_secret_name_uidx" +ON "QingLong3LegacyDataDirectoryAdoptionSecrets" ( + "adoption_mutation_id", "secret_name" +) + `, + ` +CREATE UNIQUE INDEX "ql3_legacy_data_directory_adoption_secret_mutation_uidx" +ON "QingLong3LegacyDataDirectoryAdoptionSecrets" ("secret_mutation_id") + `, + ` +CREATE UNIQUE INDEX "ql3_legacy_data_directory_adoption_secret_item_uidx" +ON "QingLong3LegacyDataDirectoryAdoptionSecrets" ("item_digest") + `, + LOCAL_DATA_DIRECTORY_ADOPTION_SECRET_GUARD_TRIGGER_SQL, + ], + }); diff --git a/packages/ql3-local-sqlite/src/migrations/0100-capability-v50.ts b/packages/ql3-local-sqlite/src/migrations/0100-capability-v50.ts new file mode 100644 index 00000000..d50de1fb --- /dev/null +++ b/packages/ql3-local-sqlite/src/migrations/0100-capability-v50.ts @@ -0,0 +1,14 @@ +import { CAPABILITIES_V49 } from './0098-capability-v49'; +import { defineLocalSqliteMigration } from './sqlMigration'; + +export const CAPABILITIES_V50 = CAPABILITIES_V49.replace( + '"legacy_adoption_ledger":1,', + '"legacy_adoption_ledger":1,"legacy_data_directory_adoption":1,', +); + +export const local0100CapabilityV50Migration = defineLocalSqliteMigration({ + id: '0100-capability-v50', + statements: [ + `UPDATE "QingLong3SchemaCapabilities" SET contract_version = 50, migration_id = '0099-legacy-data-directory-adoptions', capabilities = '${CAPABILITIES_V50}', updated_at_ms = CAST(unixepoch('subsec') * 1000 AS INTEGER) WHERE contract_name = 'local-control-core' AND contract_version = 49 AND migration_id = '0097-plugin-package-secret-binding-transition-receipts' AND capabilities = '${CAPABILITIES_V49}'`, + ], +}); diff --git a/packages/ql3-local-sqlite/src/readiness/readiness.ts b/packages/ql3-local-sqlite/src/readiness/readiness.ts index 010df510..9d78e861 100644 --- a/packages/ql3-local-sqlite/src/readiness/readiness.ts +++ b/packages/ql3-local-sqlite/src/readiness/readiness.ts @@ -2,6 +2,7 @@ import { auditMigrationStreamHistory } from '@qinglong/runtime-core/migration-st import type { DatabaseSync } from 'node:sqlite'; import { localSqliteMigrationManifest } from '../migration/migrationManifest'; import { LocalSqliteMigrationStreamStore } from '../migration/migrationStreamStore'; +import { LOCAL_DATA_DIRECTORY_ADOPTION_SECRET_GUARD_TRIGGER_SQL } from '../adoption/data-directory/dataDirectoryAdoptionSchemaContract'; import { LOCAL_PLUGIN_PACKAGE_SECRET_MATERIALIZATION_TRIGGER_SQL } from '../plugin-package/pluginPackageSecretMaterializationSchemaContract'; import { LOCAL_PLUGIN_PACKAGE_SECRET_BINDING_TARGET_TRIGGER_SQL } from '../plugin-package/secret-binding/pluginPackageSecretBindingTargetSchemaContract'; import { LOCAL_PLUGIN_PACKAGE_SECRET_BINDING_TRANSITION_RECEIPT_TRIGGER_SQL } from '../plugin-package/secret-binding/transitionReceiptSchemaContract'; @@ -11,7 +12,15 @@ import { } from '../run/stepRunSchemaContract'; export const LOCAL_SQLITE_CONTRACT_NAME = 'local-control-core'; -export const LOCAL_SQLITE_CONTRACT_VERSION = 49; +export const LOCAL_SQLITE_CONTRACT_VERSION = 50; + +const LEGACY_DATA_DIRECTORY_ADOPTION_TRIGGERS = Object.freeze([ + Object.freeze({ + name: 'ql3_legacy_data_directory_adoption_secret_guard', + tableName: 'QingLong3LegacyDataDirectoryAdoptionSecrets', + sql: LOCAL_DATA_DIRECTORY_ADOPTION_SECRET_GUARD_TRIGGER_SQL, + }), +]); const PLUGIN_PACKAGE_SECRET_BINDING_TARGET_TRIGGERS = Object.freeze([ Object.freeze({ @@ -1389,6 +1398,51 @@ const REQUIRED_SCHEMA = Object.freeze({ 'ql3_legacy_adoptions_project_time_idx', ]), }), + QingLong3LegacyDataDirectoryAdoptions: Object.freeze({ + columns: Object.freeze([ + 'mutation_id', + 'project_id', + 'profile', + 'source_stage_manifest_digest', + 'transformation_digest', + 'model_digest', + 'secret_count', + 'environment_secret_count', + 'ssh_secret_count', + 'model_json', + 'publication_digest', + 'audit_event_id', + 'committed_at_ms', + 'receipt_digest', + 'receipt_json', + ]), + indexes: Object.freeze([ + 'ql3_legacy_data_directory_adoption_transformation_uidx', + 'ql3_legacy_data_directory_adoption_receipt_uidx', + 'ql3_legacy_data_directory_adoption_project_time_idx', + ]), + }), + QingLong3LegacyDataDirectoryAdoptionSecrets: Object.freeze({ + columns: Object.freeze([ + 'adoption_mutation_id', + 'ordinal', + 'project_id', + 'kind', + 'source_name_digest', + 'secret_name', + 'secret_version', + 'secret_mutation_id', + 'value_file', + 'value_digest', + 'secret_ref', + 'item_digest', + ]), + indexes: Object.freeze([ + 'ql3_legacy_data_directory_adoption_secret_name_uidx', + 'ql3_legacy_data_directory_adoption_secret_mutation_uidx', + 'ql3_legacy_data_directory_adoption_secret_item_uidx', + ]), + }), QingLong3IdentitySubjects: Object.freeze({ columns: Object.freeze([ 'subject_type', @@ -1793,6 +1847,7 @@ function assertRequiredSchema(client: DatabaseSync): number { ...PLUGIN_PACKAGE_AUTOMATION_DISPOSITION_TRIGGERS, ...PLUGIN_PACKAGE_SECRET_BINDING_TARGET_TRIGGERS, ...PLUGIN_PACKAGE_SECRET_MATERIALIZATION_TRIGGERS, + ...LEGACY_DATA_DIRECTORY_ADOPTION_TRIGGERS, ].sort((left, right) => left.name.localeCompare(right.name)); if ( triggerRows.length !== expectedTriggers.length || @@ -2603,11 +2658,10 @@ export async function auditLocalSqliteReadiness( !capability || capability.contract_name !== LOCAL_SQLITE_CONTRACT_NAME || capability.contract_version !== LOCAL_SQLITE_CONTRACT_VERSION || - capability.migration_id !== - '0097-plugin-package-secret-binding-transition-receipts' || + capability.migration_id !== '0099-legacy-data-directory-adoptions' || typeof capability.capabilities !== 'string' || capability.capabilities !== - '{"run_core":1,"run_retry_policy":1,"completion_receipt_journal":1,"local_dispatch_plan":1,"local_secret_envelope":1,"local_project_policy":1,"local_project_administration":1,"local_security_audit":1,"local_security_audit_compaction":1,"local_secret_authorized_mutation":1,"local_identity":1,"local_api_credential":1,"local_identity_provisioning":1,"local_identity_credential_administration":1,"local_owner_bootstrap":1,"local_owner_delivery_acknowledgement":1,"api_credential_pepper_binding":1,"local_owner_pepper_catalog":1,"local_owner_credential_recovery":1,"local_owner_pepper_reference_inspection":1,"local_owner_pepper_material_gc":1,"local_owner_delivery_acknowledgement_gc":1,"task_definition":1,"local_execution_revision_digest":1,"trigger_definition":1,"legacy_adoption_ledger":1,"local_scheduler_admission":1,"plugin_package_install":1,"approved_action":1,"plugin_package_admission":1,"approved_action_execution":1,"plugin_package_proposal":1,"plugin_package_materialized_revision":1,"plugin_package_secret_binding":1,"plugin_package_secret_binding_transition":1,"plugin_package_secret_binding_transition_receipt":1,"plugin_package_secret_materialization":1,"plugin_package_task_reconciliation":1,"project_tool_definition_snapshot":1,"step_run":1,"tool_execution_evidence":1,"tool_execution_start_barrier":1,"tool_invocation_artifact":1,"tool_execution_artifact_binding":1,"tool_execution_completion":1,"tool_execution_failure_completion":1,"tool_result_key_catalog":1,"tool_result_rekey":1,"plugin_package_quarantine":1,"plugin_package_lifecycle":1,"plugin_package_automation_publication":1,"plugin_package_automation_security_withdrawal":1,"plugin_package_workflow_admission":1,"plugin_package_workflow_run_list":1,"run_attempt_log_retention":1,"plugin_package_workflow_task_attempt_admission":1}' || + '{"run_core":1,"run_retry_policy":1,"completion_receipt_journal":1,"local_dispatch_plan":1,"local_secret_envelope":1,"local_project_policy":1,"local_project_administration":1,"local_security_audit":1,"local_security_audit_compaction":1,"local_secret_authorized_mutation":1,"local_identity":1,"local_api_credential":1,"local_identity_provisioning":1,"local_identity_credential_administration":1,"local_owner_bootstrap":1,"local_owner_delivery_acknowledgement":1,"api_credential_pepper_binding":1,"local_owner_pepper_catalog":1,"local_owner_credential_recovery":1,"local_owner_pepper_reference_inspection":1,"local_owner_pepper_material_gc":1,"local_owner_delivery_acknowledgement_gc":1,"task_definition":1,"local_execution_revision_digest":1,"trigger_definition":1,"legacy_adoption_ledger":1,"legacy_data_directory_adoption":1,"local_scheduler_admission":1,"plugin_package_install":1,"approved_action":1,"plugin_package_admission":1,"approved_action_execution":1,"plugin_package_proposal":1,"plugin_package_materialized_revision":1,"plugin_package_secret_binding":1,"plugin_package_secret_binding_transition":1,"plugin_package_secret_binding_transition_receipt":1,"plugin_package_secret_materialization":1,"plugin_package_task_reconciliation":1,"project_tool_definition_snapshot":1,"step_run":1,"tool_execution_evidence":1,"tool_execution_start_barrier":1,"tool_invocation_artifact":1,"tool_execution_artifact_binding":1,"tool_execution_completion":1,"tool_execution_failure_completion":1,"tool_result_key_catalog":1,"tool_result_rekey":1,"plugin_package_quarantine":1,"plugin_package_lifecycle":1,"plugin_package_automation_publication":1,"plugin_package_automation_security_withdrawal":1,"plugin_package_workflow_admission":1,"plugin_package_workflow_run_list":1,"run_attempt_log_retention":1,"plugin_package_workflow_task_attempt_admission":1}' || typeof capability.updated_at_ms !== 'number' || !Number.isSafeInteger(capability.updated_at_ms) || capability.updated_at_ms < 0 diff --git a/packages/ql3-local-sqlite/src/storage/schema.ts b/packages/ql3-local-sqlite/src/storage/schema.ts index 570f653a..813e870b 100644 --- a/packages/ql3-local-sqlite/src/storage/schema.ts +++ b/packages/ql3-local-sqlite/src/storage/schema.ts @@ -3328,6 +3328,144 @@ export const legacyAdoptions = sqliteTable( ], ); +export const legacyDataDirectoryAdoptions = sqliteTable( + 'QingLong3LegacyDataDirectoryAdoptions', + { + mutationId: text('mutation_id').primaryKey(), + projectId: text('project_id') + .notNull() + .references(() => localProjects.id, { + onDelete: 'restrict', + onUpdate: 'restrict', + }), + profile: text('profile').notNull(), + sourceStageManifestDigest: text('source_stage_manifest_digest').notNull(), + transformationDigest: text('transformation_digest').notNull(), + modelDigest: text('model_digest').notNull(), + secretCount: integer('secret_count').notNull(), + environmentSecretCount: integer('environment_secret_count').notNull(), + sshSecretCount: integer('ssh_secret_count').notNull(), + modelJson: text('model_json', { mode: 'json' }) + .$type>() + .notNull(), + publicationDigest: text('publication_digest').notNull(), + auditEventId: text('audit_event_id') + .notNull() + .references(() => localSecurityAuditEvents.eventId, { + onDelete: 'restrict', + onUpdate: 'restrict', + }), + committedAtMs: integer('committed_at_ms').notNull(), + receiptDigest: text('receipt_digest').notNull(), + receiptJson: text('receipt_json', { mode: 'json' }) + .$type>() + .notNull(), + }, + (table) => [ + check( + 'ql3_legacy_data_directory_adoption_identity_check', + sql`length(${table.mutationId}) = 36 and substr(${table.mutationId}, 15, 1) = '4' and replace(${table.mutationId}, '-', '') not glob '*[^0-9a-f]*' and ${table.auditEventId} = ${table.mutationId} and length(${table.projectId}) between 1 and 128`, + ), + check( + 'ql3_legacy_data_directory_adoption_profile_check', + sql`${table.profile} in ('edge', 'standalone')`, + ), + check( + 'ql3_legacy_data_directory_adoption_digest_check', + sql`length(${table.sourceStageManifestDigest}) = 64 and ${table.sourceStageManifestDigest} not glob '*[^0-9a-f]*' and length(${table.transformationDigest}) = 64 and ${table.transformationDigest} not glob '*[^0-9a-f]*' and length(${table.modelDigest}) = 64 and ${table.modelDigest} not glob '*[^0-9a-f]*' and length(${table.publicationDigest}) = 64 and ${table.publicationDigest} not glob '*[^0-9a-f]*' and length(${table.receiptDigest}) = 64 and ${table.receiptDigest} not glob '*[^0-9a-f]*'`, + ), + check( + 'ql3_legacy_data_directory_adoption_count_check', + sql`${table.secretCount} between 0 and case ${table.profile} when 'edge' then 128 else 512 end and ${table.environmentSecretCount} between 0 and ${table.secretCount} and ${table.sshSecretCount} between 0 and ${table.secretCount} and ${table.environmentSecretCount} + ${table.sshSecretCount} = ${table.secretCount}`, + ), + check( + 'ql3_legacy_data_directory_adoption_model_check', + sql`length(cast(${table.modelJson} as blob)) between 2 and 1048576 and json_valid(${table.modelJson}) and json_type(${table.modelJson}) = 'object' and json_extract(${table.modelJson}, '$.schema') = 'qinglong/legacy-data-directory-applied-model@v1' and json_extract(${table.modelJson}, '$.activation') = 'disabled' and json_extract(${table.modelJson}, '$.config.schema') = 'qinglong/legacy-config-transformation@v1' and json_extract(${table.modelJson}, '$.config.activation') = 'disabled' and json_extract(${table.modelJson}, '$.keyv.schema') = 'qinglong/legacy-keyv-transformation@v1' and json_extract(${table.modelJson}, '$.keyv.activation') = 'disabled' and json_extract(${table.modelJson}, '$.ssh.schema') = 'qinglong/legacy-ssh-transformation@v1' and json_extract(${table.modelJson}, '$.ssh.activation') = 'disabled' and json_extract(${table.modelJson}, '$.manualReview.schema') = 'qinglong/legacy-data-directory-manual-review@v1' and json_extract(${table.modelJson}, '$.manualReview.required') = 0 and json_extract(${table.modelJson}, '$.manualReview.activation') = 'disabled'`, + ), + check( + 'ql3_legacy_data_directory_adoption_receipt_check', + sql`length(cast(${table.receiptJson} as blob)) between 2 and 1048576 and json_valid(${table.receiptJson}) and json_type(${table.receiptJson}) = 'object' and json_extract(${table.receiptJson}, '$.schema') = 'qinglong/legacy-data-directory-adoption-receipt@v1' and json_extract(${table.receiptJson}, '$.mutationId') = ${table.mutationId} and json_extract(${table.receiptJson}, '$.projectId') = ${table.projectId} and json_extract(${table.receiptJson}, '$.profile') = ${table.profile} and json_extract(${table.receiptJson}, '$.sourceStageManifestDigest') = ${table.sourceStageManifestDigest} and json_extract(${table.receiptJson}, '$.transformationDigest') = ${table.transformationDigest} and json_extract(${table.receiptJson}, '$.modelDigest') = ${table.modelDigest} and json_extract(${table.receiptJson}, '$.secretCount') = ${table.secretCount} and json_extract(${table.receiptJson}, '$.environmentSecretCount') = ${table.environmentSecretCount} and json_extract(${table.receiptJson}, '$.sshSecretCount') = ${table.sshSecretCount} and json_extract(${table.receiptJson}, '$.publicationDigest') = ${table.publicationDigest} and json_extract(${table.receiptJson}, '$.auditEventId') = ${table.auditEventId} and json_extract(${table.receiptJson}, '$.committedAtMs') = ${table.committedAtMs} and json_extract(${table.receiptJson}, '$.receiptDigest') = ${table.receiptDigest}`, + ), + check( + 'ql3_legacy_data_directory_adoption_time_check', + sql`${table.committedAtMs} >= 0`, + ), + uniqueIndex('ql3_legacy_data_directory_adoption_transformation_uidx').on( + table.transformationDigest, + ), + uniqueIndex('ql3_legacy_data_directory_adoption_receipt_uidx').on( + table.receiptDigest, + ), + index('ql3_legacy_data_directory_adoption_project_time_idx').on( + table.projectId, + sql`${table.committedAtMs} desc`, + sql`${table.mutationId} desc`, + ), + ], +); + +export const legacyDataDirectoryAdoptionSecrets = sqliteTable( + 'QingLong3LegacyDataDirectoryAdoptionSecrets', + { + adoptionMutationId: text('adoption_mutation_id').notNull(), + ordinal: integer('ordinal').notNull(), + projectId: text('project_id').notNull(), + kind: text('kind').notNull(), + sourceNameDigest: text('source_name_digest').notNull(), + secretName: text('secret_name').notNull(), + secretVersion: integer('secret_version').notNull(), + secretMutationId: text('secret_mutation_id') + .notNull() + .references(() => localSecurityAuditEvents.eventId, { + onDelete: 'restrict', + onUpdate: 'restrict', + }), + valueFile: text('value_file').notNull(), + valueDigest: text('value_digest').notNull(), + secretRef: text('secret_ref').notNull(), + itemDigest: text('item_digest').notNull(), + }, + (table) => [ + primaryKey({ columns: [table.adoptionMutationId, table.ordinal] }), + foreignKey({ + columns: [table.adoptionMutationId], + foreignColumns: [legacyDataDirectoryAdoptions.mutationId], + name: 'ql3_legacy_data_directory_adoption_secret_parent_fk', + }) + .onDelete('restrict') + .onUpdate('restrict'), + foreignKey({ + columns: [table.projectId, table.secretName, table.secretVersion], + foreignColumns: [ + localSecretEnvelopes.projectId, + localSecretEnvelopes.name, + localSecretEnvelopes.version, + ], + name: 'ql3_legacy_data_directory_adoption_secret_envelope_fk', + }) + .onDelete('restrict') + .onUpdate('restrict'), + check( + 'ql3_legacy_data_directory_adoption_secret_identity_check', + sql`${table.ordinal} between 1 and 512 and length(${table.projectId}) between 1 and 128 and ${table.kind} in ('environment', 'ssh_private_key') and length(${table.secretName}) between 1 and 128 and ${table.secretVersion} = 1 and length(${table.secretMutationId}) = 36 and substr(${table.secretMutationId}, 15, 1) = '4' and replace(${table.secretMutationId}, '-', '') not glob '*[^0-9a-f]*' and length(${table.valueFile}) = 83 and ${table.valueFile} glob 'secret-values/[0-9a-f]*.json' and length(${table.secretRef}) between 1 and 512`, + ), + check( + 'ql3_legacy_data_directory_adoption_secret_digest_check', + sql`length(${table.sourceNameDigest}) = 64 and ${table.sourceNameDigest} not glob '*[^0-9a-f]*' and length(${table.valueDigest}) = 64 and ${table.valueDigest} not glob '*[^0-9a-f]*' and length(${table.itemDigest}) = 64 and ${table.itemDigest} not glob '*[^0-9a-f]*'`, + ), + uniqueIndex('ql3_legacy_data_directory_adoption_secret_name_uidx').on( + table.adoptionMutationId, + table.secretName, + ), + uniqueIndex('ql3_legacy_data_directory_adoption_secret_mutation_uidx').on( + table.secretMutationId, + ), + uniqueIndex('ql3_legacy_data_directory_adoption_secret_item_uidx').on( + table.itemDigest, + ), + ], +); + export const localIdentitySubjects = sqliteTable( 'QingLong3IdentitySubjects', { @@ -5042,6 +5180,8 @@ export const localSqliteSchema = Object.freeze({ toolExecutionResultRekeyHeads, toolResultKeyRetirementReceipts, legacyAdoptions, + legacyDataDirectoryAdoptions, + legacyDataDirectoryAdoptionSecrets, localIdentitySubjects, localApiCredentials, localApiCredentialPepperBindings, diff --git a/packages/ql3-local-sqlite/test/database.test.cjs b/packages/ql3-local-sqlite/test/database.test.cjs index fcc5c1dd..393b1724 100644 --- a/packages/ql3-local-sqlite/test/database.test.cjs +++ b/packages/ql3-local-sqlite/test/database.test.cjs @@ -148,9 +148,11 @@ test('creates a reviewed edge database and opens runtime only after readiness', '0096-capability-v48', '0097-plugin-package-secret-binding-transition-receipts', '0098-capability-v49', + '0099-legacy-data-directory-adoptions', + '0100-capability-v50', ]); assert.equal(migrated.readiness.contractName, 'local-control-core'); - assert.equal(migrated.readiness.contractVersion, 49); + assert.equal(migrated.readiness.contractVersion, 50); assert.equal(migrated.readiness.journalMode, 'delete'); assert.equal(fs.statSync(databasePath).mode & 0o777, 0o600); @@ -596,8 +598,8 @@ test('backfills v14 execution revisions with a verified independent digest', asy .get(), }, { - contract_version: 49, - migration_id: '0097-plugin-package-secret-binding-transition-receipts', + contract_version: 50, + migration_id: '0099-legacy-data-directory-adoptions', }, ); } finally { @@ -784,19 +786,19 @@ test('excludes reviewed optional feature tables while preserving unknown table d const options = { databasePath, profile: 'edge' }; await migrateLocalSqlitePath(options); const client = new DatabaseSync(databasePath); - assert.equal((await auditLocalSqlitePath(options)).tableCount, 81); + assert.equal((await auditLocalSqlitePath(options)).tableCount, 83); client.exec( 'CREATE TABLE "ModelInvocationFeatureHead" (feature_id TEXT PRIMARY KEY)', ); client.close(); - assert.equal((await auditLocalSqlitePath(options)).tableCount, 81); + assert.equal((await auditLocalSqlitePath(options)).tableCount, 83); const unknownClient = new DatabaseSync(databasePath); unknownClient.exec('CREATE TABLE "UserExtensionData" (id TEXT PRIMARY KEY)'); unknownClient.close(); - assert.equal((await auditLocalSqlitePath(options)).tableCount, 82); + assert.equal((await auditLocalSqlitePath(options)).tableCount, 84); const triggerClient = new DatabaseSync(databasePath); triggerClient.exec(` diff --git a/packages/ql3-local-sqlite/test/pluginPackageWorkflowAdmissionRepository.test.cjs b/packages/ql3-local-sqlite/test/pluginPackageWorkflowAdmissionRepository.test.cjs index 7adc309f..a5ed2b0e 100644 --- a/packages/ql3-local-sqlite/test/pluginPackageWorkflowAdmissionRepository.test.cjs +++ b/packages/ql3-local-sqlite/test/pluginPackageWorkflowAdmissionRepository.test.cjs @@ -156,7 +156,7 @@ test('atomically admits one generation-bound Workflow Run and exactly replays it }, { runs: 1, steps: 2, events: 3, mutations: 2, admissions: 1 }, ); - assert.equal((await auditLocalSqliteReadiness(client)).contractVersion, 49); + assert.equal((await auditLocalSqliteReadiness(client)).contractVersion, 50); }); test('runs an optional authorization guard inside new and replay transactions', async (t) => { @@ -288,7 +288,7 @@ test('exactly replays immutable admission after the Workflow StepRun advances', }, { status: 'running', version: 5, eventSequence: 5 }, ); - assert.equal((await auditLocalSqliteReadiness(client)).contractVersion, 49); + assert.equal((await auditLocalSqliteReadiness(client)).contractVersion, 50); }); test('fails closed before writing when the exact installation is not active', async (t) => { diff --git a/packages/ql3-local-sqlite/test/pluginPackageWorkflowTaskAttemptAdmissionRepository.test.cjs b/packages/ql3-local-sqlite/test/pluginPackageWorkflowTaskAttemptAdmissionRepository.test.cjs index 49c141bd..a3cd19e6 100644 --- a/packages/ql3-local-sqlite/test/pluginPackageWorkflowTaskAttemptAdmissionRepository.test.cjs +++ b/packages/ql3-local-sqlite/test/pluginPackageWorkflowTaskAttemptAdmissionRepository.test.cjs @@ -231,7 +231,7 @@ test('atomically admits the exact reconciled local Task revision and replays it' stepAttemptCount: 0, }, ); - assert.equal((await auditLocalSqliteReadiness(client)).contractVersion, 49); + assert.equal((await auditLocalSqliteReadiness(client)).contractVersion, 50); }); test('bounds candidate paging before SQL and fences cancellation', async (t) => { diff --git a/packages/ql3-local-sqlite/test/rolloutSafety.test.cjs b/packages/ql3-local-sqlite/test/rolloutSafety.test.cjs index ab354ba1..388c522f 100644 --- a/packages/ql3-local-sqlite/test/rolloutSafety.test.cjs +++ b/packages/ql3-local-sqlite/test/rolloutSafety.test.cjs @@ -40,9 +40,9 @@ test('creates and exactly replays a reviewed rollout backup', async (t) => { await migrateLocalSqlitePath(state); const prepared = await createLocalSqliteRolloutBackup(state); assert.equal(prepared.status, 'prepared'); - assert.equal(prepared.contractVersion, 49); - assert.equal(prepared.writeContractVersion, 49); - assert.equal(LOCAL_SQLITE_WRITE_CONTRACT_VERSION, 49); + assert.equal(prepared.contractVersion, 50); + assert.equal(prepared.writeContractVersion, 50); + assert.equal(LOCAL_SQLITE_WRITE_CONTRACT_VERSION, 50); assert.match(prepared.sha256, /^[0-9a-f]{64}$/); assert.equal(prepared.bytes > 0, true); assert.equal(prepared.pageCount > 0, true); diff --git a/scripts/ql3-cluster-dependency-audit.cjs b/scripts/ql3-cluster-dependency-audit.cjs index be81634f..2e6b7d04 100644 --- a/scripts/ql3-cluster-dependency-audit.cjs +++ b/scripts/ql3-cluster-dependency-audit.cjs @@ -1767,6 +1767,20 @@ function auditSourceImports(root, packagePath, findings) { 'src/lifecycle/data-directory-adoption/staging.ts' && specifier === '@qinglong/local-admin/runtime' ) && + !( + path.relative(packageDirectory, filePath) === + 'src/lifecycle/data-directory-adoption/application/application.ts' && + [ + '@qinglong/local-admin/data-directory-adoption', + '@qinglong/local-owner-console/authenticated-command', + '@qinglong/local-sqlite/bootstrap', + ].includes(specifier) + ) && + !( + path.relative(packageDirectory, filePath) === + 'src/lifecycle/data-directory-adoption/application/cleanup.ts' && + specifier === '@qinglong/local-sqlite/data-directory-adoption' + ) && !( path.relative(packageDirectory, filePath) === 'src/plugin-package/pluginPackageCommand.ts' && @@ -2093,6 +2107,9 @@ function auditSourceImports(root, packagePath, findings) { (path.relative(packageDirectory, filePath) === 'src/legacy-adoption/legacyCrontabPublisher.ts' && specifier === '@qinglong/local-sqlite/adoption') || + (path.relative(packageDirectory, filePath) === + 'src/data-directory-adoption/dataDirectoryAdoption.ts' && + specifier === '@qinglong/local-sqlite/data-directory-adoption') || (path.relative(packageDirectory, filePath) === 'src/adopted-profile/localAdoptedProfile.ts' && specifier === '@qinglong/local-sqlite/profile') || @@ -2229,6 +2246,14 @@ function auditSourceImports(root, packagePath, findings) { '@qinglong/runtime-core/security', '@qinglong/runtime-core/security-audit', ].includes(specifier)) || + (path.relative(packageDirectory, filePath) === + 'src/data-directory-adoption/dataDirectoryAdoption.ts' && + [ + '@qinglong/runtime-core/local-secret', + '@qinglong/runtime-core/project-policy', + '@qinglong/runtime-core/security', + '@qinglong/runtime-core/security-audit', + ].includes(specifier)) || (path.relative(packageDirectory, filePath) === 'src/security-administration/projectPolicyAdministration.ts' && [ @@ -2812,6 +2837,14 @@ function auditSourceImports(root, packagePath, findings) { ) { continue; } + if ( + packagePath === 'packages/ql3-local-owner-cli' && + path.relative(packageDirectory, filePath) === + 'src/lifecycle/data-directory-adoption/application/application.ts' && + specifier === '@qinglong/local-secret' + ) { + continue; + } if ( packagePath === 'packages/ql3-local-owner-cli' && path.relative(packageDirectory, filePath) === diff --git a/test/back/ql3ClusterDependencyAudit.test.cjs b/test/back/ql3ClusterDependencyAudit.test.cjs index 9a9b469e..f2df099c 100644 --- a/test/back/ql3ClusterDependencyAudit.test.cjs +++ b/test/back/ql3ClusterDependencyAudit.test.cjs @@ -1914,6 +1914,110 @@ test('confines adoption and Package command authorities to owner CLI subpaths', ); }); +test('confines data-directory application authority to exact reviewed owners', (t) => { + const root = fs.mkdtempSync( + path.join(os.tmpdir(), 'ql3-data-directory-application-boundary-'), + ); + const ownerApplicationDirectory = path.join( + root, + 'packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/application', + ); + const adminAdoptionDirectory = path.join( + root, + 'packages/ql3-local-admin/src/data-directory-adoption', + ); + fs.mkdirSync(ownerApplicationDirectory, { recursive: true }); + fs.mkdirSync(adminAdoptionDirectory, { recursive: true }); + fs.writeFileSync( + path.join(ownerApplicationDirectory, 'application.ts'), + [ + "import { application } from '@qinglong/local-admin/data-directory-adoption';", + "import { secrets } from '@qinglong/local-secret';", + "import { authenticated } from '@qinglong/local-owner-console/authenticated-command';", + "import { bootstrap } from '@qinglong/local-sqlite/bootstrap';", + ].join('\n'), + ); + fs.writeFileSync( + path.join(ownerApplicationDirectory, 'cleanup.ts'), + "import { cleanup } from '@qinglong/local-sqlite/data-directory-adoption';", + ); + fs.writeFileSync( + path.join(ownerApplicationDirectory, 'neighbor.ts'), + [ + "import { application } from '@qinglong/local-admin/data-directory-adoption';", + "import { secrets } from '@qinglong/local-secret';", + "import { cleanup } from '@qinglong/local-sqlite/data-directory-adoption';", + ].join('\n'), + ); + fs.writeFileSync( + path.join(adminAdoptionDirectory, 'dataDirectoryAdoption.ts'), + [ + "import { database } from '@qinglong/local-sqlite/data-directory-adoption';", + "import type { secret } from '@qinglong/runtime-core/local-secret';", + "import { policy } from '@qinglong/runtime-core/project-policy';", + "import type { decision } from '@qinglong/runtime-core/security';", + "import type { audit } from '@qinglong/runtime-core/security-audit';", + ].join('\n'), + ); + fs.writeFileSync( + path.join(adminAdoptionDirectory, 'neighbor.ts'), + [ + "import { database } from '@qinglong/local-sqlite/data-directory-adoption';", + "import type { secret } from '@qinglong/runtime-core/local-secret';", + ].join('\n'), + ); + t.after(() => fs.rmSync(root, { recursive: true, force: true })); + + const ownerFindings = []; + auditSourceImports(root, 'packages/ql3-local-owner-cli', ownerFindings); + assert.deepEqual( + ownerFindings.map(({ code, file, specifier }) => ({ + code, + file, + specifier, + })), + [ + { + code: 'FORBIDDEN_LOCAL_ADOPTION_CLI_AUTHORITY_IMPORT', + file: 'packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/application/neighbor.ts', + specifier: '@qinglong/local-admin/data-directory-adoption', + }, + { + code: 'FORBIDDEN_PACKAGE_SOURCE_IMPORT', + file: 'packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/application/neighbor.ts', + specifier: '@qinglong/local-secret', + }, + { + code: 'FORBIDDEN_LOCAL_ADOPTION_CLI_AUTHORITY_IMPORT', + file: 'packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/application/neighbor.ts', + specifier: '@qinglong/local-sqlite/data-directory-adoption', + }, + ], + ); + + const adminFindings = []; + auditSourceImports(root, 'packages/ql3-local-admin', adminFindings); + assert.deepEqual( + adminFindings.map(({ code, file, specifier }) => ({ + code, + file, + specifier, + })), + [ + { + code: 'FORBIDDEN_LOCAL_ADMIN_SQLITE_ENTRYPOINT', + file: 'packages/ql3-local-admin/src/data-directory-adoption/neighbor.ts', + specifier: '@qinglong/local-sqlite/data-directory-adoption', + }, + { + code: 'FORBIDDEN_LOCAL_ADMIN_RUNTIME_CORE_ENTRYPOINT', + file: 'packages/ql3-local-admin/src/data-directory-adoption/neighbor.ts', + specifier: '@qinglong/runtime-core/local-secret', + }, + ], + ); +}); + test('confines fresh setup authority to the reviewed owner CLI subpath', (t) => { const root = fs.mkdtempSync(path.join(os.tmpdir(), 'ql3-setup-boundary-')); const sourceDirectory = path.join(root, 'packages/ql3-local-owner-cli/src'); diff --git a/test/back/ql3PackageBoundaryAudit.test.cjs b/test/back/ql3PackageBoundaryAudit.test.cjs index 442a0784..5021f9d8 100644 --- a/test/back/ql3PackageBoundaryAudit.test.cjs +++ b/test/back/ql3PackageBoundaryAudit.test.cjs @@ -79,7 +79,7 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( }, { directory: 'packages/ql3-local-sqlite/src/migrations', - directSourceFiles: 99, + directSourceFiles: 101, reviewKind: 'ordered_ledger', }, ], @@ -97,10 +97,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: localAdmin.rootSourceFileRoles, }, { - sourceFiles: 44, + sourceFiles: 45, rootSourceFiles: 1, rootSourceLines: 9, - nestedSourceFiles: 43, + nestedSourceFiles: 44, rootSourceFileRoles: { 'runtime.ts': 'public_export' }, }, ); @@ -207,10 +207,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: localOwnerCli.rootSourceFileRoles, }, { - sourceFiles: 129, + sourceFiles: 131, rootSourceFiles: 1, rootSourceLines: 50, - nestedSourceFiles: 128, + nestedSourceFiles: 130, rootSourceFileRoles: { 'cli.ts': 'binary_entry' }, }, ); @@ -540,10 +540,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: localSqlite.rootSourceFileRoles, }, { - sourceFiles: 197, + sourceFiles: 201, rootSourceFiles: 1, rootSourceLines: 31, - nestedSourceFiles: 196, + nestedSourceFiles: 200, rootSourceFileRoles: { 'index.ts': 'public_export' }, }, );