diff --git a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md index 18a1fcbf..27ac8c7a 100644 --- a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md +++ b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md @@ -11,6 +11,21 @@ 最新增量证据（2026-08-21）： +- D-384/ADR-0477（已接受）：完整 2.x data directory 接管先落地为一次性、只读、有界 inventory，而不是直接复用 legacy shell `tar` + 或盲拷整个目录。既有 `ql3-adoption` 新增 exact 私有命令 `local-data-directory.adoption.inspect`，固定分类 + `config/db/ssh.d→transform`、`scripts/upload→copy_reviewed`、`log/syslog/bak→retain_external(root-only)`、 + `repo/raw/dep_cache/deps→regenerate(root-only)`；未知顶层条目只返回数量/摘要并转 `manual_review`。递归类别按 UTF-8 字节序、 + `lstat`/no-follow 和 stable descriptor 流式哈希；symlink、硬链接、特殊文件、错误 owner 与 group/world writable 条目不读取且计为 + unsafe，底层错误统一脱敏。Edge 限制 8192 项、512 MiB 总哈希、64 MiB 单文件、32 层，Standalone 为 65536 项、4 GiB、 + 512 MiB、64 层；目录用增量 `opendir` 在保存超预算名称前失败，文件只用 64 KiB 缓冲。该 operation 不复制、转换、归档或写源目录， + 也尚未绑定 D-383 SQLite activation；D-385 才设计双 digest fence 的 stage/verify。没有新增 package、dependency、binary、daemon、 + listener、timer、数据库连接或部署对象；Local Owner 新代码内聚在 `lifecycle/data-directory-adoption/`，workspace 保持 18 packages、 + `singleSourcePackages=[]`、`shallowSourcePackages=[]`、`118 source / 117 nested / 1 root binary entry`。D-384 focused `5/5`， + Local Owner `195 total / 190 pass / 5 conditional skip / 0 fail`；backend `1,535 total / 1,533 pass / 2 conditional skip / + 0 fail`，`pnpm build:back` 与 18-package clean build/逐包测试通过；八项架构审计与 14 档 artifact audit 全 compatible，基础 + Edge/Standalone、Adopted、Application+AI 和 MCP 制品体积/模块数均保持 D-383 基线，证明盘点 authority 未进入低配常驻闭包。 + 本切片不改变 PostgreSQL 语义，故不重跑且不重新占有 HA 证明。固定物理 Edge 的盘点/staging RSS、I/O、磁盘峰值、断电恢复， + 以及 config/Keyv/SSH 转换和 systemd/OpenRC/Compose lineage 仍待后续完成。 - D-383/ADR-0476（已接受）：把单个 2.x SQLite 主库接管从分散 API/合成 fixture 推进为产品级真实双态演练。既有一次性 `ql3-adoption` 新增 exact、私有 command-file 的 `inspect → stage → verify → activation`，从生产形态 Sequelize schema （Cron、Dependency、App、Auth、Env、Subscription、View、Stats、RunningInstance 与未知 Plugin-owned table）生成独立 recovery diff --git a/docs/adr/ADR-0477-bounded-legacy-data-directory-inventory.md b/docs/adr/ADR-0477-bounded-legacy-data-directory-inventory.md new file mode 100644 index 00000000..75278309 --- /dev/null +++ b/docs/adr/ADR-0477-bounded-legacy-data-directory-inventory.md @@ -0,0 +1,140 @@ +# ADR-0477：有界 Legacy Data Directory 盘点 + +- 状态：Accepted +- 日期：2026-08-21 +- 关联：QL-RFC-0001、ADR-0476 + +## 上下文 + +ADR-0476 已经证明单个生产形态 QingLong 2.x `database.sqlite` 可以经过 inspect、stage、verify、activation 和 clean/write-after +双态回滚分类接管到 3.0，但真实部署的 `data` 目录不只有主数据库。现行 2.x 配置和实际用户目录可包含： + +- `config/`、`scripts/`、`db/`、`upload/`、`ssh.d/`； +- `log/`、`syslog/`、`bak/`； +- `repo/`、`raw/`、`dep_cache/`、历史 `deps/`； +- 用户或插件创建的未知顶层条目。 + +直接复用 2.x `SystemService.exportData/importData` 不满足 3.0 接管要求。该路径以 shell 拼接 `tar`，默认只覆盖数据库和上传目录， +也没有固定资产分类、no-follow、硬链接拒绝、内容上限、确定性计划或敏感输出约束。直接打包整个 `data` 目录还会把日志、备份、 +跨架构依赖缓存、仓库 checkout、SSH 材料和未知插件资产混成一个不可审核恢复单元；对低内存路由设备尤其危险。 + +完整目录复制之前，需要一个短生命周期、只读、确定性、按 Profile 有界的产品入口先回答：哪些资产存在、哪些可进入后续复制、哪些必须 +转换、哪些只保留为外部恢复资产、哪些应在 3.0 重新生成，以及是否存在必须人工处理的不安全或未知条目。 + +## 决策 + +### 1. 复用现有一次性产品入口 + +在 `@qinglong/local-owner-cli` 的 `lifecycle/data-directory-adoption/` 内增加： + +- exact operation：`local-data-directory.adoption.inspect`； +- exact options：`dataRoot` 与 `profile`； +- 既有 `ql3-adoption run --command-file ...` 私有 command-file 入口。 + +不新增 workspace package、第三方依赖、binary、daemon、listener、watcher、timer、数据库连接或部署对象。实现不得进入 Edge、 +Standalone、Application、AI 或 MCP 常驻闭包。 + +### 2. 固定资产处置矩阵 + +| 类别 | 处置 | 盘点深度 | 原因 | +| --- | --- | --- | --- | +| `config` | `transform` | `recursive_content` | 需要迁移到 3.0 配置模型，不能盲拷旧配置 | +| `scripts` | `copy_reviewed` | `recursive_content` | 用户脚本是业务资产，但必须先审核安全和兼容性 | +| `db` | `transform` | `recursive_content` | 主库由 ADR-0476 迁移，Keyv/sidecar 需独立识别 | +| `upload` | `copy_reviewed` | `recursive_content` | 用户上传内容可保留，但必须受大小和文件类型边界约束 | +| `ssh.d` | `transform` | `recursive_content` | 属于敏感凭据材料，后续必须进入专用私有交付协议 | +| `log`、`syslog`、`bak` | `retain_external` | `root_only` | 作为历史/恢复资产保留，不进入默认 3.0 运行目录 | +| `repo`、`raw`、`dep_cache`、`deps` | `regenerate` | `root_only` | checkout、原始缓存和依赖缓存应按目标架构重建 | + +未知顶层条目只记录数量和名称摘要，不返回原始名称，计划状态固定为 `manual_review`。本阶段不允许 caller 覆盖处置矩阵或增加任意 +include/exclude glob。 + +### 3. No-follow、稳定身份与内容脱敏 + +盘点要求 `dataRoot` 为当前 UID 拥有、canonical、非符号链接、group/world 不可写的非根目录。递归类别按 UTF-8 字节序确定性遍历， +使用 `lstat` 且不跟随符号链接；只有当前 UID、group/world 不可写的普通单链接文件或目录可继续读取。符号链接、硬链接、多链接文件、 +特殊文件、错误 owner 和可被组/其他用户写入的条目只计为 unsafe，不读取其内容。 + +普通文件通过 `O_NOFOLLOW` descriptor 读取，打开前后的 device、inode、mode、link count、UID、size、mtime 和 ctime 必须一致。 +目录遍历前后也必须保持同一稳定身份。底层文件系统错误统一映射为固定错误码 +`LOCAL_DATA_DIRECTORY_ADOPTION_CONFIGURATION_INVALID`，CLI 不返回原始路径、任意文件名或内容。 + +结果只包含固定类别名、计数、逻辑/分配字节、宽读权限计数、unsafe 计数、SQLite 主库/Keyv/sidecar 识别计数、内容摘要、未知条目 +数量/摘要和完整 `planDigest`。宽读权限是审核信号；只有可写权限或身份/类型问题自动成为 unsafe。 + +### 4. Edge 与 Standalone 分离预算 + +| Profile | 最大条目 | 最大哈希字节 | 单文件上限 | 最大深度 | +| --- | ---: | ---: | ---: | ---: | +| Edge | 8,192 | 512 MiB | 64 MiB | 32 | +| Standalone | 65,536 | 4 GiB | 512 MiB | 64 | + +目录通过增量 `opendir` 枚举，并在保存超过剩余预算的名称前失败，不先用一次性 `readdir` 将任意数量的目录项装入内存。文件使用 +64 KiB 固定缓冲区流式哈希并在关闭前清零。`root_only` 类别不递归、不读取或哈希其子项，因此依赖缓存和历史日志规模不会进入 +盘点内存或 I/O 成本。 + +### 5. 本阶段只发布计划，不执行迁移 + +该 operation 只读源目录并将结果写到 stdout。它不创建目录副本、归档、manifest、recovery 或 activation，不修改 2.x 数据， +也不把目录计划自动绑定到 ADR-0476 的 SQLite plan/activation。后续 stage 必须重新验证稳定身份并显式绑定两个计划摘要；不能把本次 +inspect 输出直接当作复制授权。 + +## 被拒绝的替代方案 + +### 直接 tar 完整 data directory + +拒绝。它混合不同恢复语义、可能跟随或保存不安全链接、复制跨架构缓存，并让低配设备承担不可预测的空间和内存成本。 + +### 为目录接管再拆一个 workspace package + +拒绝。该能力只有一个短生命周期产品 owner，没有独立部署、依赖或版本生命周期；放入已有 Local Owner lifecycle 垂直目录更符合 +当前包边界规则，也避免恢复“一个文件一个包”的碎片化。 + +### 只统计文件大小，不读取内容摘要 + +拒绝。大小和时间不能把后续 stage 绑定到已审核内容；确定性流式摘要提供最小的漂移证明，同时不输出文件内容。 + +### 在 inspect 时复制或转换文件 + +拒绝。盘点和 mutation 混合会让未知/不安全条目在 operator 审核前产生目标副本，也无法建立清晰的 plan-digest fence。 + +## 影响 + +### 正面 + +- 完整 2.x data directory 首次获得固定、可审核的资产处置模型； +- Edge 和 Standalone 使用不同硬预算，低配设备不会继承集群节点规模假设； +- 未知资产、链接和权限漂移失败关闭，且不会泄露任意文件名或内容； +- 日志、备份、仓库和依赖缓存不会污染 3.0 默认运行目录； +- 没有增加 package 粒度、常驻资源或基础制品体积。 + +### 代价与限制 + +- 对递归类别执行全内容哈希，仍会产生与资产大小线性的磁盘读取； +- 正在写入的 2.x 目录可能因稳定身份检查失败，需要先停止 writer 后重试； +- `root_only` 只证明类别根的存在、权限和类型，不证明内部历史资产完整性； +- 当前没有 stage/verify/restore，也没有固定物理 Edge 的耗时、RSS、磁盘峰值与断电演练； +- 当前目录计划尚未与 SQLite activation、service-manager cutover 和 rollback lineage 形成统一证据链。 + +## 验证 + +- D-384 聚焦 data directory CLI：`5/5`； +- Local Owner：`195 total / 190 pass / 5 conditional skip / 0 fail`； +- backend：`1,535 total / 1,533 pass / 2 conditional skip / 0 fail`，`pnpm build:back` 通过； +- 18-package clean build 与逐包测试单次退出 0； +- package boundary、Cluster dependency、Edge import、Service Bridge import、Cluster/Worker deployment、Console 与 Console + distribution 八项审计全部 compatible/passed； +- workspace 保持 18 packages，`singleSourcePackages=[]`、`shallowSourcePackages=[]`；Local Owner 为 + `118 source / 117 nested / 1 root binary entry`； +- 14 档 Local artifact audit 全部 compatible，基础 Edge/Standalone、Adopted、Application+AI 与 MCP 的体积和 loaded-module + 基线未变化。 + +本阶段不修改 PostgreSQL schema、ACL、repository、role、Pool、连接或 failover 语义，因此不重跑且不重新占有 PostgreSQL HA +证明。 + +## 后续 + +- D-385：以 plan digest 和 ADR-0476 activation digest 为双 fence，设计 no-replace stage/verify manifest； +- 对 `config`、Keyv 与 `ssh.d` 定义显式转换/私有交付协议； +- 在固定物理 Edge/NAS 上测量完整目录盘点与 staging 的耗时、RSS、I/O、磁盘峰值和断电恢复； +- 把 staged data directory 证据接入 systemd/OpenRC/Compose cutover 与 rollback lineage。 diff --git a/docs/adr/README.md b/docs/adr/README.md index 75c18525..8eedbc9a 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -480,6 +480,7 @@ | [ADR-0474](./ADR-0474-bounded-legacy-core-readiness-proof.md) | 有界 Legacy Core Readiness Proof | Accepted（OpenRC live actor 待补） | | [ADR-0475](./ADR-0475-legacy-system-script-open-api-compatibility.md) | Legacy System、Script 与 Open API 兼容基线 | Accepted | | [ADR-0476](./ADR-0476-real-legacy-sqlite-upgrade-and-rollback-rehearsal.md) | 真实 Legacy SQLite 升级与回滚演练 | Accepted | +| [ADR-0477](./ADR-0477-bounded-legacy-data-directory-inventory.md) | 有界 Legacy Data Directory 盘点 | Accepted | ## 规则 diff --git a/docs/operations/ql3-local-data-directory-adoption.md b/docs/operations/ql3-local-data-directory-adoption.md new file mode 100644 index 00000000..fd4d930d --- /dev/null +++ b/docs/operations/ql3-local-data-directory-adoption.md @@ -0,0 +1,103 @@ +# QingLong 2.x Data Directory 盘点 + +本流程为完整 QingLong 2.x `data` 目录生成一个只读、确定性、按 Profile 有界的 3.0 接管计划。它不会复制、压缩、删除或修改 +任何文件，也不替代 [SQLite 接管流程](./ql3-local-sqlite-adoption.md)。 + +## 1. 前置条件 + +- 使用最终运行 QingLong 的同一个 POSIX 用户执行； +- `dataRoot` 必须是绝对、canonical、非根、非 symlink 的目录； +- `dataRoot` 必须由当前 UID 拥有，且 group/world 不可写； +- command file 继续遵守 `ql3-adoption` 的当前 UID、canonical、单链接、`0600` 私有文件要求； +- 生产盘点建议先停止 2.x writer。若文件或目录在盘点中变化，命令会失败关闭，不会给出部分成功计划。 + +## 2. 执行 inspect + +Edge/低配路由设备： + +```json +{ + "schemaVersion": 1, + "operation": "local-data-directory.adoption.inspect", + "options": { + "dataRoot": "/opt/qinglong/data", + "profile": "edge" + } +} +``` + +Standalone/NAS： + +```json +{ + "schemaVersion": 1, + "operation": "local-data-directory.adoption.inspect", + "options": { + "dataRoot": "/opt/qinglong/data", + "profile": "standalone" + } +} +``` + +```sh +chmod 0600 /secure/operator/ql3-data-directory-inspect.json +ql3-adoption run --command-file /secure/operator/ql3-data-directory-inspect.json +``` + +命令成功时返回 `status=inspected` 和 `qinglong3-legacy-data-directory-adoption-plan`。保存完整 JSON 和 `planDigest`，但不要把它 +当作已经授权复制的 manifest。 + +## 3. 审核处置矩阵 + +| 类别 | 默认处置 | 说明 | +| --- | --- | --- | +| `config` | `transform` | 迁移到 3.0 配置模型，不原样覆盖 | +| `scripts` | `copy_reviewed` | 审核兼容性和安全后复制 | +| `db` | `transform` | 主 SQLite 走独立流程；审核 Keyv 和 sidecar | +| `upload` | `copy_reviewed` | 审核文件类型与容量后复制 | +| `ssh.d` | `transform` | 进入后续私有凭据交付，不进入普通归档 | +| `log`、`syslog`、`bak` | `retain_external` | 作为历史/恢复资产另行保留 | +| `repo`、`raw`、`dep_cache`、`deps` | `regenerate` | 在目标架构重新 checkout/安装 | + +`recursive_content` 类别会稳定读取普通单链接文件并生成摘要；`root_only` 类别只检查类别根，不扫描内部内容。 + +## 4. 解释 assessment + +- `reviewable`：没有未知顶层条目，也没有 unsafe 条目；仍需 operator 审核分类、数量、字节和数据库识别计数； +- `manual_review`：出现未知顶层条目或 unsafe 条目；不得继续自动 staging； +- `broadReadableEntries > 0`：存在 group/world 可读条目，是敏感性审核信号，但不等同于可被外部修改； +- `activeSqliteSidecars > 0`：发现 `database.sqlite-*` 或 `keyv.sqlite-*` 活跃 sidecar，先停止 writer 并完成 SQLite + checkpoint/一致性处置后重新盘点； +- `primaryDatabaseFiles` 应按生产布局识别 `db/database.sqlite`；`legacyKeyValueDatabaseFiles` 识别 `db/keyv.sqlite`。 + +输出不会包含 `dataRoot` 原文、任意文件名或文件内容。未知名称只进入摘要。不要尝试从摘要反推或把摘要当作内容备份。 + +## 5. Profile 预算 + +| Profile | 最大条目 | 最大哈希字节 | 单文件上限 | 最大深度 | +| --- | ---: | ---: | ---: | ---: | +| Edge | 8,192 | 512 MiB | 64 MiB | 32 | +| Standalone | 65,536 | 4 GiB | 512 MiB | 64 | + +超过任一限制都会以 `LOCAL_DATA_DIRECTORY_ADOPTION_CONFIGURATION_INVALID` 失败。不要为了通过门禁临时改名、删除或排除资产；先保留 +现场并决定它应拆分为外部恢复资产、在目标重新生成，还是进入后续人工迁移协议。 + +## 6. 常见失败 + +- 根目录或条目 group/world 可写：修正 ownership/permission 后重新盘点； +- symlink、硬链接或特殊文件：保留现场，确认来源和目标后人工处置；盘点不会跟随或读取； +- 目录在盘点中变化：停止 2.x writer、同步器、下载器和仓库更新后重试； +- 单文件或总内容超过 Profile 预算：不要改用 `tar` 绕过；为该资产设计独立流式迁移/外部保留流程； +- 未知顶层条目：根据插件或用户资产来源登记明确处置，再进入后续 staging 设计。 + +## 7. 当前边界 + +本命令只生成只读计划。它尚不： + +- 复制、压缩、删除或转换任何资产； +- 创建 no-replace stage、verify manifest 或 recovery； +- 把目录计划绑定到 SQLite `planDigest`、`manifestDigest` 或 `activationDigest`； +- 授权 service-manager/Compose cutover 或 Legacy rollback； +- 证明固定物理路由器/NAS 上的耗时、RSS、I/O、磁盘峰值和断电恢复。 + +在后续 D-385 stage/verify 合同完成前，保留 inspect 输出和原始 2.x data directory，不要把目录计划当作自动迁移完成证明。 diff --git a/packages/ql3-local-owner-cli/src/lifecycle/adoptionCommand.ts b/packages/ql3-local-owner-cli/src/lifecycle/adoptionCommand.ts index 1df01ced..42ffa9bc 100644 --- a/packages/ql3-local-owner-cli/src/lifecycle/adoptionCommand.ts +++ b/packages/ql3-local-owner-cli/src/lifecycle/adoptionCommand.ts @@ -1,6 +1,8 @@ import { readPrivateLocalCommandFile } from '@qinglong/local-command-file'; import { runLegacyCrontabAdoptionCommandFile } from './adoption'; +import { isLocalDataDirectoryAdoptionOperation } from './data-directory-adoption/contract'; +import type { LocalDataDirectoryAdoptionInspectResult } from './data-directory-adoption/inventory'; import { isLocalSqliteAdoptionProductOperation, type LocalSqliteAdoptionProductOperation, @@ -9,7 +11,8 @@ import type { LocalSqliteAdoptionProductCommandResult } from './sqlite-adoption/ export type LocalAdoptionProductCommandResult = | Awaited> - | LocalSqliteAdoptionProductCommandResult; + | LocalSqliteAdoptionProductCommandResult + | LocalDataDirectoryAdoptionInspectResult; function operation(value: unknown): unknown { return value && typeof value === 'object' && !Array.isArray(value) @@ -28,6 +31,12 @@ export async function runLocalAdoptionProductCommandFile( return runLegacyCrontabAdoptionCommandFile(commandFilePath); } const selected = operation(candidate); + if (isLocalDataDirectoryAdoptionOperation(selected)) { + const { inspectLocalDataDirectoryAdoption } = await import( + './data-directory-adoption/inventory.js' + ); + return inspectLocalDataDirectoryAdoption(candidate); + } if (!isLocalSqliteAdoptionProductOperation(selected)) { return runLegacyCrontabAdoptionCommandFile(commandFilePath); } diff --git a/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/contract.ts b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/contract.ts new file mode 100644 index 00000000..77d1ae23 --- /dev/null +++ b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/contract.ts @@ -0,0 +1,88 @@ +import path from 'node:path'; + +const MAX_PATH_BYTES = 4_096; + +export const LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION = + 'local-data-directory.adoption.inspect' as const; + +export interface InspectLocalDataDirectoryAdoptionCommand { + readonly schemaVersion: 1; + readonly operation: typeof LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION; + readonly options: { + readonly dataRoot: string; + readonly profile: 'edge' | 'standalone'; + }; +} + +export class LocalDataDirectoryAdoptionConfigurationError extends TypeError { + readonly code = 'LOCAL_DATA_DIRECTORY_ADOPTION_CONFIGURATION_INVALID'; + + constructor(message: string, readonly cause?: unknown) { + super(`Local data directory adoption configuration is invalid: ${message}`); + this.name = 'LocalDataDirectoryAdoptionConfigurationError'; + } +} + +function exactKeys(value: object, expected: readonly string[]): boolean { + const actual = Object.keys(value).sort(); + const canonical = [...expected].sort(); + return ( + actual.length === canonical.length && + actual.every((key, index) => key === canonical[index]) + ); +} + +function normalizedAbsolutePath(value: unknown): value is string { + return ( + typeof value === 'string' && + path.isAbsolute(value) && + path.parse(value).root !== value && + path.normalize(value) === value && + !value.includes('\0') && + Buffer.byteLength(value, 'utf8') <= MAX_PATH_BYTES + ); +} + +export function isLocalDataDirectoryAdoptionOperation( + value: unknown, +): value is typeof LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION { + return value === LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION; +} + +export function normalizeInspectLocalDataDirectoryAdoptionCommand( + value: unknown, +): Readonly { + if ( + !value || + typeof value !== 'object' || + Array.isArray(value) || + !exactKeys(value, ['schemaVersion', 'operation', 'options']) + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'command shape is invalid', + ); + } + const candidate = value as Record; + if ( + candidate.schemaVersion !== 1 || + !isLocalDataDirectoryAdoptionOperation(candidate.operation) || + !candidate.options || + typeof candidate.options !== 'object' || + Array.isArray(candidate.options) || + !exactKeys(candidate.options, ['dataRoot', 'profile']) + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'command value is invalid', + ); + } + const options = candidate.options as Record; + if ( + !normalizedAbsolutePath(options.dataRoot) || + (options.profile !== 'edge' && options.profile !== 'standalone') + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'command options are invalid', + ); + } + return Object.freeze(value as InspectLocalDataDirectoryAdoptionCommand); +} diff --git a/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/inventory.ts b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/inventory.ts new file mode 100644 index 00000000..58b530fe --- /dev/null +++ b/packages/ql3-local-owner-cli/src/lifecycle/data-directory-adoption/inventory.ts @@ -0,0 +1,696 @@ +import crypto from 'node:crypto'; +import fs from 'node:fs'; +import path from 'node:path'; + +import { + LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION, + LocalDataDirectoryAdoptionConfigurationError, + normalizeInspectLocalDataDirectoryAdoptionCommand, + type InspectLocalDataDirectoryAdoptionCommand, +} from './contract'; + +const HASH_BUFFER_BYTES = 64 * 1024; +const MAX_RELATIVE_PATH_BYTES = 4_096; +const DIGEST_PATTERN = /^[0-9a-f]{64}$/; + +type DataDirectoryDisposition = + | 'copy_reviewed' + | 'transform' + | 'retain_external' + | 'regenerate'; + +type DataDirectoryInspection = 'recursive_content' | 'root_only'; + +interface CategoryPolicy { + readonly name: string; + readonly disposition: DataDirectoryDisposition; + readonly inspection: DataDirectoryInspection; +} + +interface InventoryBudget { + readonly maxEntries: number; + readonly maxHashedBytes: number; + readonly maxFileBytes: number; + readonly maxDepth: number; +} + +interface MutableCategorySummary { + entries: number; + directories: number; + regularFiles: number; + logicalBytes: number; + allocatedBytes: number; + broadReadableEntries: number; + unsafeEntries: number; + activeSqliteSidecars: number; + primaryDatabaseFiles: number; + legacyKeyValueDatabaseFiles: number; + hashedBytes: number; +} + +export interface LocalDataDirectoryCategoryEvidence { + readonly name: string; + readonly disposition: DataDirectoryDisposition; + readonly inspection: DataDirectoryInspection; + readonly present: boolean; + readonly entries: number; + readonly directories: number; + readonly regularFiles: number; + readonly logicalBytes: number | null; + readonly allocatedBytes: number | null; + readonly broadReadableEntries: number; + readonly unsafeEntries: number; + readonly activeSqliteSidecars: number; + readonly primaryDatabaseFiles: number; + readonly legacyKeyValueDatabaseFiles: number; + readonly contentDigest: string; +} + +export interface LocalDataDirectoryAdoptionEvidence { + readonly schemaVersion: 1; + readonly kind: 'qinglong3-legacy-data-directory-adoption-plan'; + readonly profile: 'edge' | 'standalone'; + readonly policyVersion: 1; + readonly dataRootPathDigest: string; + readonly budget: Readonly; + readonly assessment: 'reviewable' | 'manual_review'; + readonly categories: readonly LocalDataDirectoryCategoryEvidence[]; + readonly unknownTopLevelEntries: number; + readonly unknownTopLevelDigest: string; + readonly totalInspectedEntries: number; + readonly totalHashedBytes: number; + readonly totalUnsafeEntries: number; + readonly planDigest: string; +} + +export interface LocalDataDirectoryAdoptionInspectResult { + readonly schemaVersion: 1; + readonly operation: typeof LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION; + readonly status: 'inspected'; + readonly evidence: Readonly; +} + +const POLICIES: readonly CategoryPolicy[] = Object.freeze([ + Object.freeze({ + name: 'config', + disposition: 'transform', + inspection: 'recursive_content', + }), + Object.freeze({ + name: 'scripts', + disposition: 'copy_reviewed', + inspection: 'recursive_content', + }), + Object.freeze({ + name: 'db', + disposition: 'transform', + inspection: 'recursive_content', + }), + Object.freeze({ + name: 'upload', + disposition: 'copy_reviewed', + inspection: 'recursive_content', + }), + Object.freeze({ + name: 'ssh.d', + disposition: 'transform', + inspection: 'recursive_content', + }), + Object.freeze({ + name: 'log', + disposition: 'retain_external', + inspection: 'root_only', + }), + Object.freeze({ + name: 'syslog', + disposition: 'retain_external', + inspection: 'root_only', + }), + Object.freeze({ + name: 'bak', + disposition: 'retain_external', + inspection: 'root_only', + }), + Object.freeze({ + name: 'repo', + disposition: 'regenerate', + inspection: 'root_only', + }), + Object.freeze({ + name: 'raw', + disposition: 'regenerate', + inspection: 'root_only', + }), + Object.freeze({ + name: 'dep_cache', + disposition: 'regenerate', + inspection: 'root_only', + }), + Object.freeze({ + name: 'deps', + disposition: 'regenerate', + inspection: 'root_only', + }), +]); + +const EMPTY_DIGEST = crypto.createHash('sha256').digest('hex'); + +function digestText(value: string): string { + return crypto.createHash('sha256').update(value, 'utf8').digest('hex'); +} + +function budget(profile: 'edge' | 'standalone'): Readonly { + return Object.freeze( + profile === 'edge' + ? { + maxEntries: 8_192, + maxHashedBytes: 512 * 1024 * 1024, + maxFileBytes: 64 * 1024 * 1024, + maxDepth: 32, + } + : { + maxEntries: 65_536, + maxHashedBytes: 4 * 1024 * 1024 * 1024, + maxFileBytes: 512 * 1024 * 1024, + maxDepth: 64, + }, + ); +} + +function currentUid(): number { + if (typeof process.getuid !== 'function') { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'POSIX ownership is unavailable', + ); + } + return process.getuid(); +} + +function sameStat(left: fs.BigIntStats, right: fs.BigIntStats): boolean { + return ( + left.dev === right.dev && + left.ino === right.ino && + left.mode === right.mode && + left.nlink === right.nlink && + left.uid === right.uid && + left.size === right.size && + left.mtimeNs === right.mtimeNs && + left.ctimeNs === right.ctimeNs + ); +} + +function stableRoot(dataRoot: string, uid: number): fs.BigIntStats { + let stat: fs.BigIntStats; + try { + stat = fs.lstatSync(dataRoot, { bigint: true }); + } catch (error) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'dataRoot is unavailable', + error, + ); + } + if ( + !stat.isDirectory() || + stat.isSymbolicLink() || + stat.uid !== BigInt(uid) || + (stat.mode & 0o022n) !== 0n || + fs.realpathSync(dataRoot) !== dataRoot + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'dataRoot must be an owner-controlled canonical directory', + ); + } + return stat; +} + +function sortedDirectoryNames( + directoryPath: string, + maxNames: number, +): readonly string[] { + const directory = fs.opendirSync(directoryPath); + const names: string[] = []; + try { + for (;;) { + const entry = directory.readSync(); + if (entry === null) break; + names.push(entry.name); + if (names.length > maxNames) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data directory entry count exceeds the Profile budget', + ); + } + } + } finally { + directory.closeSync(); + } + return names.sort((left, right) => + Buffer.compare(Buffer.from(left, 'utf8'), Buffer.from(right, 'utf8')), + ); +} + +function relativePath(base: string, candidate: string): string { + const relative = path.relative(base, candidate); + if ( + relative.length < 1 || + path.isAbsolute(relative) || + relative === '..' || + relative.startsWith(`..${path.sep}`) || + Buffer.byteLength(relative, 'utf8') > MAX_RELATIVE_PATH_BYTES + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data directory entry path is invalid or too long', + ); + } + return relative.split(path.sep).join('/'); +} + +function updateMetadata( + hash: crypto.Hash, + relative: string, + stat: fs.BigIntStats, + kind: string, + contentDigest?: string, +): void { + hash.update( + `${JSON.stringify({ + relative, + kind, + mode: (stat.mode & 0o777n).toString(8), + uid: stat.uid.toString(), + links: stat.nlink.toString(), + bytes: stat.size.toString(), + modifiedAtNs: stat.mtimeNs.toString(), + changedAtNs: stat.ctimeNs.toString(), + ...(contentDigest === undefined ? {} : { contentDigest }), + })}\n`, + 'utf8', + ); +} + +function stableFileDigest(filePath: string, expected: fs.BigIntStats): string { + const descriptor = fs.openSync( + filePath, + fs.constants.O_RDONLY | (fs.constants.O_NOFOLLOW ?? 0), + ); + const buffer = Buffer.allocUnsafe(HASH_BUFFER_BYTES); + try { + const before = fs.fstatSync(descriptor, { bigint: true }); + if (!sameStat(expected, before) || !before.isFile()) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data directory file identity changed before inspection', + ); + } + const hash = crypto.createHash('sha256'); + for (;;) { + const count = fs.readSync(descriptor, buffer, 0, buffer.length, null); + if (count === 0) break; + hash.update(buffer.subarray(0, count)); + } + const after = fs.fstatSync(descriptor, { bigint: true }); + if (!sameStat(before, after)) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data directory file changed during inspection', + ); + } + return hash.digest('hex'); + } finally { + buffer.fill(0); + fs.closeSync(descriptor); + } +} + +function mutableSummary(): MutableCategorySummary { + return { + entries: 0, + directories: 0, + regularFiles: 0, + logicalBytes: 0, + allocatedBytes: 0, + broadReadableEntries: 0, + unsafeEntries: 0, + activeSqliteSidecars: 0, + primaryDatabaseFiles: 0, + legacyKeyValueDatabaseFiles: 0, + hashedBytes: 0, + }; +} + +function safeNumber(value: bigint, label: string): number { + if (value < 0n || value > BigInt(Number.MAX_SAFE_INTEGER)) { + throw new LocalDataDirectoryAdoptionConfigurationError( + `${label} exceeds the supported numeric range`, + ); + } + return Number(value); +} + +function addEntryBytes( + summary: MutableCategorySummary, + stat: fs.BigIntStats, +): void { + const logical = safeNumber(stat.size, 'entry bytes'); + const allocated = safeNumber(stat.blocks * 512n, 'allocated entry bytes'); + if ( + !Number.isSafeInteger(summary.logicalBytes + logical) || + !Number.isSafeInteger(summary.allocatedBytes + allocated) + ) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'category byte total exceeds the supported numeric range', + ); + } + summary.logicalBytes += logical; + summary.allocatedBytes += allocated; +} + +function inspectRecursiveCategory( + dataRoot: string, + categoryRoot: string, + rootStat: fs.BigIntStats, + uid: number, + limits: Readonly, + shared: { entries: number; hashedBytes: number }, +): Readonly<{ + summary: MutableCategorySummary; + contentDigest: string; +}> { + const summary = mutableSummary(); + const hash = crypto.createHash('sha256'); + + const visitDirectory = ( + directoryPath: string, + expected: fs.BigIntStats, + depth: number, + ): void => { + if (depth > limits.maxDepth) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data directory depth exceeds the Profile budget', + ); + } + const names = sortedDirectoryNames( + directoryPath, + limits.maxEntries - shared.entries, + ); + for (const name of names) { + const entryPath = path.join(directoryPath, name); + const relative = relativePath(dataRoot, entryPath); + const stat = fs.lstatSync(entryPath, { bigint: true }); + shared.entries += 1; + summary.entries += 1; + if (shared.entries > limits.maxEntries) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data directory entry count exceeds the Profile budget', + ); + } + addEntryBytes(summary, stat); + const mode = stat.mode & 0o777n; + if ((mode & 0o044n) !== 0n) summary.broadReadableEntries += 1; + const unsafeIdentity = + stat.uid !== BigInt(uid) || + (mode & 0o022n) !== 0n || + stat.isSymbolicLink() || + (!stat.isDirectory() && !stat.isFile()) || + (stat.isFile() && stat.nlink !== 1n); + if (unsafeIdentity) { + summary.unsafeEntries += 1; + updateMetadata(hash, relative, stat, 'unsafe'); + continue; + } + if (stat.isDirectory()) { + summary.directories += 1; + updateMetadata(hash, relative, stat, 'directory'); + visitDirectory(entryPath, stat, depth + 1); + const after = fs.lstatSync(entryPath, { bigint: true }); + if (!sameStat(stat, after)) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data directory changed during inspection', + ); + } + continue; + } + const size = safeNumber(stat.size, 'file bytes'); + if (size > limits.maxFileBytes) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data directory file exceeds the Profile budget', + ); + } + if (shared.hashedBytes + size > limits.maxHashedBytes) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data directory hashed bytes exceed the Profile budget', + ); + } + shared.hashedBytes += size; + summary.hashedBytes += size; + summary.regularFiles += 1; + const baseName = path.basename(entryPath); + if (relative === 'db/database.sqlite') summary.primaryDatabaseFiles += 1; + if (relative === 'db/keyv.sqlite') + summary.legacyKeyValueDatabaseFiles += 1; + if (/^(?:database|keyv)\.sqlite-(?:wal|shm|journal)$/.test(baseName)) { + summary.activeSqliteSidecars += 1; + } + updateMetadata( + hash, + relative, + stat, + 'file', + stableFileDigest(entryPath, stat), + ); + } + const after = fs.lstatSync(directoryPath, { bigint: true }); + if (!sameStat(expected, after)) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data directory changed during inspection', + ); + } + }; + + visitDirectory(categoryRoot, rootStat, 1); + return Object.freeze({ summary, contentDigest: hash.digest('hex') }); +} + +function inspectCategory( + policy: Readonly, + dataRoot: string, + uid: number, + limits: Readonly, + shared: { entries: number; hashedBytes: number }, +): Readonly { + const categoryRoot = path.join(dataRoot, policy.name); + let stat: fs.BigIntStats; + try { + stat = fs.lstatSync(categoryRoot, { bigint: true }); + } catch (error) { + if ( + error && + typeof error === 'object' && + 'code' in error && + error.code === 'ENOENT' + ) { + return Object.freeze({ + ...policy, + present: false, + entries: 0, + directories: 0, + regularFiles: 0, + logicalBytes: policy.inspection === 'root_only' ? null : 0, + allocatedBytes: policy.inspection === 'root_only' ? null : 0, + broadReadableEntries: 0, + unsafeEntries: 0, + activeSqliteSidecars: 0, + primaryDatabaseFiles: 0, + legacyKeyValueDatabaseFiles: 0, + contentDigest: EMPTY_DIGEST, + }); + } + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data category is unavailable', + error, + ); + } + const mode = stat.mode & 0o777n; + const safeDirectory = + stat.isDirectory() && + !stat.isSymbolicLink() && + stat.uid === BigInt(uid) && + (mode & 0o022n) === 0n; + if (!safeDirectory || policy.inspection === 'root_only') { + const evidence = Object.freeze({ + ...policy, + present: true, + entries: 0, + directories: 0, + regularFiles: 0, + logicalBytes: null, + allocatedBytes: null, + broadReadableEntries: (mode & 0o044n) !== 0n ? 1 : 0, + unsafeEntries: safeDirectory ? 0 : 1, + activeSqliteSidecars: 0, + primaryDatabaseFiles: 0, + legacyKeyValueDatabaseFiles: 0, + contentDigest: digestText( + JSON.stringify({ + category: policy.name, + mode: mode.toString(8), + uid: stat.uid.toString(), + kind: safeDirectory ? 'directory' : 'unsafe', + }), + ), + }); + const after = fs.lstatSync(categoryRoot, { bigint: true }); + if (!sameStat(stat, after)) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data category changed during inspection', + ); + } + return evidence; + } + const inspected = inspectRecursiveCategory( + dataRoot, + categoryRoot, + stat, + uid, + limits, + shared, + ); + return Object.freeze({ + ...policy, + present: true, + entries: inspected.summary.entries, + directories: inspected.summary.directories, + regularFiles: inspected.summary.regularFiles, + logicalBytes: inspected.summary.logicalBytes, + allocatedBytes: inspected.summary.allocatedBytes, + broadReadableEntries: inspected.summary.broadReadableEntries, + unsafeEntries: inspected.summary.unsafeEntries, + activeSqliteSidecars: inspected.summary.activeSqliteSidecars, + primaryDatabaseFiles: inspected.summary.primaryDatabaseFiles, + legacyKeyValueDatabaseFiles: inspected.summary.legacyKeyValueDatabaseFiles, + contentDigest: inspected.contentDigest, + }); +} + +function unknownTopLevelEvidence( + dataRoot: string, + known: ReadonlySet, + limits: Readonly, + shared: { entries: number; hashedBytes: number }, +): Readonly<{ count: number; digest: string }> { + const hash = crypto.createHash('sha256'); + let count = 0; + const names = sortedDirectoryNames( + dataRoot, + limits.maxEntries - shared.entries + known.size, + ); + for (const name of names) { + if (known.has(name)) continue; + count += 1; + shared.entries += 1; + if (shared.entries > limits.maxEntries) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data directory entry count exceeds the Profile budget', + ); + } + const stat = fs.lstatSync(path.join(dataRoot, name), { bigint: true }); + hash.update( + `${JSON.stringify({ + nameDigest: digestText(name), + mode: (stat.mode & 0o777n).toString(8), + uid: stat.uid.toString(), + kind: stat.isDirectory() + ? 'directory' + : stat.isFile() + ? 'file' + : stat.isSymbolicLink() + ? 'symlink' + : 'special', + })}\n`, + 'utf8', + ); + const after = fs.lstatSync(path.join(dataRoot, name), { bigint: true }); + if (!sameStat(stat, after)) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'unknown data directory entry changed during inspection', + ); + } + } + return Object.freeze({ count, digest: hash.digest('hex') }); +} + +function planDigest( + evidence: Omit, +): string { + return digestText(JSON.stringify(evidence)); +} + +export function inspectLocalDataDirectoryAdoption( + value: unknown, +): Readonly { + try { + const command: Readonly = + normalizeInspectLocalDataDirectoryAdoptionCommand(value); + const uid = currentUid(); + const rootBefore = stableRoot(command.options.dataRoot, uid); + const limits = budget(command.options.profile); + const shared = { entries: 0, hashedBytes: 0 }; + const categories = POLICIES.map((policy) => + inspectCategory(policy, command.options.dataRoot, uid, limits, shared), + ); + const unknown = unknownTopLevelEvidence( + command.options.dataRoot, + new Set(POLICIES.map((policy) => policy.name)), + limits, + shared, + ); + const rootAfter = stableRoot(command.options.dataRoot, uid); + if (!sameStat(rootBefore, rootAfter)) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'dataRoot changed during inspection', + ); + } + const totalUnsafeEntries = categories.reduce( + (total, category) => total + category.unsafeEntries, + 0, + ); + const payload = Object.freeze({ + schemaVersion: 1 as const, + kind: 'qinglong3-legacy-data-directory-adoption-plan' as const, + profile: command.options.profile, + policyVersion: 1 as const, + dataRootPathDigest: digestText(command.options.dataRoot), + budget: limits, + assessment: + totalUnsafeEntries === 0 && unknown.count === 0 + ? ('reviewable' as const) + : ('manual_review' as const), + categories: Object.freeze(categories), + unknownTopLevelEntries: unknown.count, + unknownTopLevelDigest: unknown.digest, + totalInspectedEntries: shared.entries, + totalHashedBytes: shared.hashedBytes, + totalUnsafeEntries, + }); + const evidence = Object.freeze({ + ...payload, + planDigest: planDigest(payload), + }); + if (!DIGEST_PATTERN.test(evidence.planDigest)) { + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data directory plan digest is invalid', + ); + } + return Object.freeze({ + schemaVersion: 1, + operation: LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION, + status: 'inspected', + evidence, + }); + } catch (error) { + if (error instanceof LocalDataDirectoryAdoptionConfigurationError) { + throw error; + } + throw new LocalDataDirectoryAdoptionConfigurationError( + 'data directory inspection failed', + error, + ); + } +} diff --git a/packages/ql3-local-owner-cli/test/dataDirectoryAdoptionCli.test.cjs b/packages/ql3-local-owner-cli/test/dataDirectoryAdoptionCli.test.cjs new file mode 100644 index 00000000..7471d2e4 --- /dev/null +++ b/packages/ql3-local-owner-cli/test/dataDirectoryAdoptionCli.test.cjs @@ -0,0 +1,274 @@ +const assert = require('node:assert/strict'); +const { spawnSync } = require('node:child_process'); +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const { test } = require('node:test'); + +const BINARY = path.join(__dirname, '../dist/lifecycle/adoptionCli.js'); +const OPERATION = 'local-data-directory.adoption.inspect'; + +const RECURSIVE_CATEGORIES = Object.freeze([ + ['config', 'transform'], + ['scripts', 'copy_reviewed'], + ['db', 'transform'], + ['upload', 'copy_reviewed'], + ['ssh.d', 'transform'], +]); +const ROOT_ONLY_CATEGORIES = Object.freeze([ + ['log', 'retain_external'], + ['syslog', 'retain_external'], + ['bak', 'retain_external'], + ['repo', 'regenerate'], + ['raw', 'regenerate'], + ['dep_cache', 'regenerate'], + ['deps', 'regenerate'], +]); + +function privateDirectory(directoryPath) { + fs.mkdirSync(directoryPath, { recursive: true, mode: 0o700 }); + fs.chmodSync(directoryPath, 0o700); +} + +function privateFile(filePath, content) { + privateDirectory(path.dirname(filePath)); + fs.writeFileSync(filePath, content, { mode: 0o600 }); + fs.chmodSync(filePath, 0o600); +} + +function fixture(t) { + const root = fs.realpathSync( + fs.mkdtempSync(path.join(os.tmpdir(), 'ql3-data-directory-adoption-')), + ); + fs.chmodSync(root, 0o700); + t.after(() => fs.rmSync(root, { recursive: true, force: true })); + const commandsDirectory = path.join(root, 'commands'); + const dataRoot = path.join(root, 'data'); + privateDirectory(commandsDirectory); + privateDirectory(dataRoot); + + privateFile(path.join(dataRoot, 'config', 'config.sh'), 'export A=1\n'); + privateFile( + path.join(dataRoot, 'scripts', 'jobs', 'example.sh'), + 'echo qinglong\n', + ); + privateFile(path.join(dataRoot, 'db', 'database.sqlite'), 'legacy-primary'); + privateFile(path.join(dataRoot, 'db', 'keyv.sqlite'), 'legacy-keyv'); + privateFile(path.join(dataRoot, 'upload', 'avatar.bin'), Buffer.from([1, 2])); + privateFile(path.join(dataRoot, 'ssh.d', 'repository-key'), 'private-key'); + + for (const [category] of ROOT_ONLY_CATEGORIES) { + privateFile( + path.join(dataRoot, category, 'nested', 'ignored-content'), + `ignored-${category}`, + ); + } + return { root, commandsDirectory, dataRoot }; +} + +function runRaw(value, name, command) { + const commandPath = path.join(value.commandsDirectory, `${name}.json`); + fs.writeFileSync(commandPath, `${JSON.stringify(command)}\n`, { + mode: 0o600, + }); + fs.chmodSync(commandPath, 0o600); + return spawnSync( + process.execPath, + [BINARY, 'run', '--command-file', commandPath], + { encoding: 'utf8' }, + ); +} + +function inspect(value, name = 'inspect', profile = 'edge') { + const child = runRaw(value, name, { + schemaVersion: 1, + operation: OPERATION, + options: { dataRoot: value.dataRoot, profile }, + }); + assert.equal(child.status, 0, child.stderr); + assert.equal(child.stderr, ''); + return { child, result: JSON.parse(child.stdout) }; +} + +function categoryMap(result) { + return new Map( + result.evidence.categories.map((category) => [category.name, category]), + ); +} + +test('data directory adoption emits a deterministic content-free migration plan', (t) => { + const value = fixture(t); + const first = inspect(value, 'first'); + const second = inspect(value, 'second'); + + assert.equal(first.result.schemaVersion, 1); + assert.equal(first.result.operation, OPERATION); + assert.equal(first.result.status, 'inspected'); + assert.equal( + first.result.evidence.kind, + 'qinglong3-legacy-data-directory-adoption-plan', + ); + assert.equal(first.result.evidence.profile, 'edge'); + assert.equal(first.result.evidence.policyVersion, 1); + assert.equal(first.result.evidence.assessment, 'reviewable'); + assert.equal(first.result.evidence.unknownTopLevelEntries, 0); + assert.equal(first.result.evidence.totalInspectedEntries, 7); + assert.equal(first.result.evidence.totalUnsafeEntries, 0); + assert.match(first.result.evidence.planDigest, /^[0-9a-f]{64}$/); + assert.deepEqual(second.result, first.result); + + const categories = categoryMap(first.result); + assert.equal(categories.size, 12); + for (const [name, disposition] of RECURSIVE_CATEGORIES) { + assert.deepEqual( + { + disposition: categories.get(name).disposition, + inspection: categories.get(name).inspection, + present: categories.get(name).present, + }, + { disposition, inspection: 'recursive_content', present: true }, + ); + } + for (const [name, disposition] of ROOT_ONLY_CATEGORIES) { + assert.deepEqual( + { + disposition: categories.get(name).disposition, + inspection: categories.get(name).inspection, + present: categories.get(name).present, + entries: categories.get(name).entries, + logicalBytes: categories.get(name).logicalBytes, + }, + { + disposition, + inspection: 'root_only', + present: true, + entries: 0, + logicalBytes: null, + }, + ); + } + assert.equal(categories.get('db').primaryDatabaseFiles, 1); + assert.equal(categories.get('db').legacyKeyValueDatabaseFiles, 1); + + assert.equal(first.child.stdout.includes(value.dataRoot), false); + assert.equal(first.child.stdout.includes('repository-key'), false); + assert.equal(first.child.stdout.includes('private-key'), false); + assert.equal(first.child.stdout.includes('example.sh'), false); +}); + +test('root-only cache and history contents do not enter the adoption plan', (t) => { + const value = fixture(t); + const before = inspect(value, 'before').result; + privateFile( + path.join(value.dataRoot, 'repo', 'nested', 'ignored-content'), + 'different-cross-architecture-cache', + ); + const afterRootOnlyChange = inspect(value, 'after-root-only').result; + assert.deepEqual(afterRootOnlyChange, before); + + privateFile( + path.join(value.dataRoot, 'scripts', 'jobs', 'example.sh'), + 'echo changed\n', + ); + const afterRelevantChange = inspect(value, 'after-relevant').result; + assert.notEqual( + afterRelevantChange.evidence.planDigest, + before.evidence.planDigest, + ); +}); + +test('links and unknown top-level entries fail closed without leaking names', (t) => { + const value = fixture(t); + const externalSecret = path.join(value.root, 'external-sensitive-value'); + privateFile(externalSecret, 'must-not-be-read'); + fs.linkSync( + externalSecret, + path.join(value.dataRoot, 'scripts', 'jobs', 'hard-linked-secret'), + ); + fs.symlinkSync( + externalSecret, + path.join(value.dataRoot, 'scripts', 'jobs', 'linked-secret'), + ); + const unknownName = 'customer-private-extension'; + privateFile( + path.join(value.dataRoot, unknownName), + 'unknown-sensitive-value', + ); + + const inspected = inspect(value, 'unsafe'); + assert.equal(inspected.result.evidence.assessment, 'manual_review'); + assert.equal(inspected.result.evidence.totalUnsafeEntries, 2); + assert.equal(inspected.result.evidence.unknownTopLevelEntries, 1); + assert.equal(categoryMap(inspected.result).get('scripts').unsafeEntries, 2); + for (const sensitive of [ + value.dataRoot, + externalSecret, + 'hard-linked-secret', + 'linked-secret', + unknownName, + 'must-not-be-read', + 'unknown-sensitive-value', + ]) { + assert.equal(inspected.child.stdout.includes(sensitive), false); + } +}); + +test('widened commands and unsafe roots are rejected with a stable public error', (t) => { + const value = fixture(t); + const widened = runRaw(value, 'widened', { + schemaVersion: 1, + operation: OPERATION, + options: { + dataRoot: value.dataRoot, + profile: 'edge', + extraAuthority: true, + }, + }); + assert.equal(widened.status, 1); + assert.equal(widened.stdout, ''); + assert.equal( + JSON.parse(widened.stderr).code, + 'LOCAL_DATA_DIRECTORY_ADOPTION_CONFIGURATION_INVALID', + ); + + fs.chmodSync(value.dataRoot, 0o777); + const unsafe = runRaw(value, 'unsafe-root', { + schemaVersion: 1, + operation: OPERATION, + options: { dataRoot: value.dataRoot, profile: 'edge' }, + }); + assert.equal(unsafe.status, 1); + assert.equal(unsafe.stdout, ''); + const error = JSON.parse(unsafe.stderr); + assert.equal( + error.code, + 'LOCAL_DATA_DIRECTORY_ADOPTION_CONFIGURATION_INVALID', + ); + assert.equal(unsafe.stderr.includes(value.dataRoot), false); +}); + +test('edge inspection enforces a per-file budget before reading content', (t) => { + const value = fixture(t); + const oversized = path.join( + value.dataRoot, + 'scripts', + 'oversized-private-file', + ); + privateFile(oversized, ''); + fs.truncateSync(oversized, 64 * 1024 * 1024 + 1); + + const child = runRaw(value, 'oversized', { + schemaVersion: 1, + operation: OPERATION, + options: { dataRoot: value.dataRoot, profile: 'edge' }, + }); + assert.equal(child.status, 1); + assert.equal(child.stdout, ''); + const error = JSON.parse(child.stderr); + assert.equal( + error.code, + 'LOCAL_DATA_DIRECTORY_ADOPTION_CONFIGURATION_INVALID', + ); + assert.match(error.message, /Profile budget/); + assert.equal(child.stderr.includes(oversized), false); +}); diff --git a/test/back/ql3PackageBoundaryAudit.test.cjs b/test/back/ql3PackageBoundaryAudit.test.cjs index c7448f48..bbc00ff5 100644 --- a/test/back/ql3PackageBoundaryAudit.test.cjs +++ b/test/back/ql3PackageBoundaryAudit.test.cjs @@ -207,10 +207,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: localOwnerCli.rootSourceFileRoles, }, { - sourceFiles: 116, + sourceFiles: 118, rootSourceFiles: 1, rootSourceLines: 50, - nestedSourceFiles: 115, + nestedSourceFiles: 117, rootSourceFileRoles: { 'cli.ts': 'binary_entry' }, }, );