mirror of
https://github.com/whyour/qinglong.git
synced 2026-09-21 01:32:44 +08:00
feat(ql3): inventory legacy data directory
This commit is contained in:
whyour
@@ -11,6 +11,21 @@
|
||||
|
||||
最新增量证据（2026-08-21）：
|
||||
|
||||
- D-384/ADR-0477（已接受）：完整 2.x data directory 接管先落地为一次性、只读、有界 inventory，而不是直接复用 legacy shell `tar`
|
||||
或盲拷整个目录。既有 `ql3-adoption` 新增 exact 私有命令 `local-data-directory.adoption.inspect`，固定分类
|
||||
`config/db/ssh.d→transform`、`scripts/upload→copy_reviewed`、`log/syslog/bak→retain_external(root-only)`、
|
||||
`repo/raw/dep_cache/deps→regenerate(root-only)`；未知顶层条目只返回数量/摘要并转 `manual_review`。递归类别按 UTF-8 字节序、
|
||||
`lstat`/no-follow 和 stable descriptor 流式哈希；symlink、硬链接、特殊文件、错误 owner 与 group/world writable 条目不读取且计为
|
||||
unsafe，底层错误统一脱敏。Edge 限制 8192 项、512 MiB 总哈希、64 MiB 单文件、32 层，Standalone 为 65536 项、4 GiB、
|
||||
512 MiB、64 层；目录用增量 `opendir` 在保存超预算名称前失败，文件只用 64 KiB 缓冲。该 operation 不复制、转换、归档或写源目录，
|
||||
也尚未绑定 D-383 SQLite activation；D-385 才设计双 digest fence 的 stage/verify。没有新增 package、dependency、binary、daemon、
|
||||
listener、timer、数据库连接或部署对象；Local Owner 新代码内聚在 `lifecycle/data-directory-adoption/`，workspace 保持 18 packages、
|
||||
`singleSourcePackages=[]`、`shallowSourcePackages=[]`、`118 source / 117 nested / 1 root binary entry`。D-384 focused `5/5`，
|
||||
Local Owner `195 total / 190 pass / 5 conditional skip / 0 fail`；backend `1,535 total / 1,533 pass / 2 conditional skip /
|
||||
0 fail`，`pnpm build:back` 与 18-package clean build/逐包测试通过；八项架构审计与 14 档 artifact audit 全 compatible，基础
|
||||
Edge/Standalone、Adopted、Application+AI 和 MCP 制品体积/模块数均保持 D-383 基线，证明盘点 authority 未进入低配常驻闭包。
|
||||
本切片不改变 PostgreSQL 语义，故不重跑且不重新占有 HA 证明。固定物理 Edge 的盘点/staging RSS、I/O、磁盘峰值、断电恢复，
|
||||
以及 config/Keyv/SSH 转换和 systemd/OpenRC/Compose lineage 仍待后续完成。
|
||||
- D-383/ADR-0476（已接受）：把单个 2.x SQLite 主库接管从分散 API/合成 fixture 推进为产品级真实双态演练。既有一次性
|
||||
`ql3-adoption` 新增 exact、私有 command-file 的 `inspect → stage → verify → activation`，从生产形态 Sequelize schema
|
||||
（Cron、Dependency、App、Auth、Env、Subscription、View、Stats、RunningInstance 与未知 Plugin-owned table）生成独立 recovery
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
# ADR-0477：有界 Legacy Data Directory 盘点
|
||||
|
||||
- 状态：Accepted
|
||||
- 日期：2026-08-21
|
||||
- 关联：QL-RFC-0001、ADR-0476
|
||||
|
||||
## 上下文
|
||||
|
||||
ADR-0476 已经证明单个生产形态 QingLong 2.x `database.sqlite` 可以经过 inspect、stage、verify、activation 和 clean/write-after
|
||||
双态回滚分类接管到 3.0，但真实部署的 `data` 目录不只有主数据库。现行 2.x 配置和实际用户目录可包含：
|
||||
|
||||
- `config/`、`scripts/`、`db/`、`upload/`、`ssh.d/`；
|
||||
- `log/`、`syslog/`、`bak/`；
|
||||
- `repo/`、`raw/`、`dep_cache/`、历史 `deps/`；
|
||||
- 用户或插件创建的未知顶层条目。
|
||||
|
||||
直接复用 2.x `SystemService.exportData/importData` 不满足 3.0 接管要求。该路径以 shell 拼接 `tar`，默认只覆盖数据库和上传目录，
|
||||
也没有固定资产分类、no-follow、硬链接拒绝、内容上限、确定性计划或敏感输出约束。直接打包整个 `data` 目录还会把日志、备份、
|
||||
跨架构依赖缓存、仓库 checkout、SSH 材料和未知插件资产混成一个不可审核恢复单元；对低内存路由设备尤其危险。
|
||||
|
||||
完整目录复制之前，需要一个短生命周期、只读、确定性、按 Profile 有界的产品入口先回答：哪些资产存在、哪些可进入后续复制、哪些必须
|
||||
转换、哪些只保留为外部恢复资产、哪些应在 3.0 重新生成，以及是否存在必须人工处理的不安全或未知条目。
|
||||
|
||||
## 决策
|
||||
|
||||
### 1. 复用现有一次性产品入口
|
||||
|
||||
在 `@qinglong/local-owner-cli` 的 `lifecycle/data-directory-adoption/` 内增加：
|
||||
|
||||
- exact operation：`local-data-directory.adoption.inspect`；
|
||||
- exact options：`dataRoot` 与 `profile`；
|
||||
- 既有 `ql3-adoption run --command-file ...` 私有 command-file 入口。
|
||||
|
||||
不新增 workspace package、第三方依赖、binary、daemon、listener、watcher、timer、数据库连接或部署对象。实现不得进入 Edge、
|
||||
Standalone、Application、AI 或 MCP 常驻闭包。
|
||||
|
||||
### 2. 固定资产处置矩阵
|
||||
|
||||
| 类别 | 处置 | 盘点深度 | 原因 |
|
||||
| --- | --- | --- | --- |
|
||||
| `config` | `transform` | `recursive_content` | 需要迁移到 3.0 配置模型，不能盲拷旧配置 |
|
||||
| `scripts` | `copy_reviewed` | `recursive_content` | 用户脚本是业务资产，但必须先审核安全和兼容性 |
|
||||
| `db` | `transform` | `recursive_content` | 主库由 ADR-0476 迁移，Keyv/sidecar 需独立识别 |
|
||||
| `upload` | `copy_reviewed` | `recursive_content` | 用户上传内容可保留，但必须受大小和文件类型边界约束 |
|
||||
| `ssh.d` | `transform` | `recursive_content` | 属于敏感凭据材料，后续必须进入专用私有交付协议 |
|
||||
| `log`、`syslog`、`bak` | `retain_external` | `root_only` | 作为历史/恢复资产保留，不进入默认 3.0 运行目录 |
|
||||
| `repo`、`raw`、`dep_cache`、`deps` | `regenerate` | `root_only` | checkout、原始缓存和依赖缓存应按目标架构重建 |
|
||||
|
||||
未知顶层条目只记录数量和名称摘要，不返回原始名称，计划状态固定为 `manual_review`。本阶段不允许 caller 覆盖处置矩阵或增加任意
|
||||
include/exclude glob。
|
||||
|
||||
### 3. No-follow、稳定身份与内容脱敏
|
||||
|
||||
盘点要求 `dataRoot` 为当前 UID 拥有、canonical、非符号链接、group/world 不可写的非根目录。递归类别按 UTF-8 字节序确定性遍历，
|
||||
使用 `lstat` 且不跟随符号链接；只有当前 UID、group/world 不可写的普通单链接文件或目录可继续读取。符号链接、硬链接、多链接文件、
|
||||
特殊文件、错误 owner 和可被组/其他用户写入的条目只计为 unsafe，不读取其内容。
|
||||
|
||||
普通文件通过 `O_NOFOLLOW` descriptor 读取，打开前后的 device、inode、mode、link count、UID、size、mtime 和 ctime 必须一致。
|
||||
目录遍历前后也必须保持同一稳定身份。底层文件系统错误统一映射为固定错误码
|
||||
`LOCAL_DATA_DIRECTORY_ADOPTION_CONFIGURATION_INVALID`，CLI 不返回原始路径、任意文件名或内容。
|
||||
|
||||
结果只包含固定类别名、计数、逻辑/分配字节、宽读权限计数、unsafe 计数、SQLite 主库/Keyv/sidecar 识别计数、内容摘要、未知条目
|
||||
数量/摘要和完整 `planDigest`。宽读权限是审核信号；只有可写权限或身份/类型问题自动成为 unsafe。
|
||||
|
||||
### 4. Edge 与 Standalone 分离预算
|
||||
|
||||
| Profile | 最大条目 | 最大哈希字节 | 单文件上限 | 最大深度 |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| Edge | 8,192 | 512 MiB | 64 MiB | 32 |
|
||||
| Standalone | 65,536 | 4 GiB | 512 MiB | 64 |
|
||||
|
||||
目录通过增量 `opendir` 枚举，并在保存超过剩余预算的名称前失败，不先用一次性 `readdir` 将任意数量的目录项装入内存。文件使用
|
||||
64 KiB 固定缓冲区流式哈希并在关闭前清零。`root_only` 类别不递归、不读取或哈希其子项，因此依赖缓存和历史日志规模不会进入
|
||||
盘点内存或 I/O 成本。
|
||||
|
||||
### 5. 本阶段只发布计划，不执行迁移
|
||||
|
||||
该 operation 只读源目录并将结果写到 stdout。它不创建目录副本、归档、manifest、recovery 或 activation，不修改 2.x 数据，
|
||||
也不把目录计划自动绑定到 ADR-0476 的 SQLite plan/activation。后续 stage 必须重新验证稳定身份并显式绑定两个计划摘要；不能把本次
|
||||
inspect 输出直接当作复制授权。
|
||||
|
||||
## 被拒绝的替代方案
|
||||
|
||||
### 直接 tar 完整 data directory
|
||||
|
||||
拒绝。它混合不同恢复语义、可能跟随或保存不安全链接、复制跨架构缓存，并让低配设备承担不可预测的空间和内存成本。
|
||||
|
||||
### 为目录接管再拆一个 workspace package
|
||||
|
||||
拒绝。该能力只有一个短生命周期产品 owner，没有独立部署、依赖或版本生命周期；放入已有 Local Owner lifecycle 垂直目录更符合
|
||||
当前包边界规则，也避免恢复“一个文件一个包”的碎片化。
|
||||
|
||||
### 只统计文件大小，不读取内容摘要
|
||||
|
||||
拒绝。大小和时间不能把后续 stage 绑定到已审核内容；确定性流式摘要提供最小的漂移证明，同时不输出文件内容。
|
||||
|
||||
### 在 inspect 时复制或转换文件
|
||||
|
||||
拒绝。盘点和 mutation 混合会让未知/不安全条目在 operator 审核前产生目标副本，也无法建立清晰的 plan-digest fence。
|
||||
|
||||
## 影响
|
||||
|
||||
### 正面
|
||||
|
||||
- 完整 2.x data directory 首次获得固定、可审核的资产处置模型；
|
||||
- Edge 和 Standalone 使用不同硬预算，低配设备不会继承集群节点规模假设；
|
||||
- 未知资产、链接和权限漂移失败关闭，且不会泄露任意文件名或内容；
|
||||
- 日志、备份、仓库和依赖缓存不会污染 3.0 默认运行目录；
|
||||
- 没有增加 package 粒度、常驻资源或基础制品体积。
|
||||
|
||||
### 代价与限制
|
||||
|
||||
- 对递归类别执行全内容哈希，仍会产生与资产大小线性的磁盘读取；
|
||||
- 正在写入的 2.x 目录可能因稳定身份检查失败，需要先停止 writer 后重试；
|
||||
- `root_only` 只证明类别根的存在、权限和类型，不证明内部历史资产完整性；
|
||||
- 当前没有 stage/verify/restore，也没有固定物理 Edge 的耗时、RSS、磁盘峰值与断电演练；
|
||||
- 当前目录计划尚未与 SQLite activation、service-manager cutover 和 rollback lineage 形成统一证据链。
|
||||
|
||||
## 验证
|
||||
|
||||
- D-384 聚焦 data directory CLI：`5/5`；
|
||||
- Local Owner：`195 total / 190 pass / 5 conditional skip / 0 fail`；
|
||||
- backend：`1,535 total / 1,533 pass / 2 conditional skip / 0 fail`，`pnpm build:back` 通过；
|
||||
- 18-package clean build 与逐包测试单次退出 0；
|
||||
- package boundary、Cluster dependency、Edge import、Service Bridge import、Cluster/Worker deployment、Console 与 Console
|
||||
distribution 八项审计全部 compatible/passed；
|
||||
- workspace 保持 18 packages，`singleSourcePackages=[]`、`shallowSourcePackages=[]`；Local Owner 为
|
||||
`118 source / 117 nested / 1 root binary entry`；
|
||||
- 14 档 Local artifact audit 全部 compatible，基础 Edge/Standalone、Adopted、Application+AI 与 MCP 的体积和 loaded-module
|
||||
基线未变化。
|
||||
|
||||
本阶段不修改 PostgreSQL schema、ACL、repository、role、Pool、连接或 failover 语义，因此不重跑且不重新占有 PostgreSQL HA
|
||||
证明。
|
||||
|
||||
## 后续
|
||||
|
||||
- D-385：以 plan digest 和 ADR-0476 activation digest 为双 fence，设计 no-replace stage/verify manifest；
|
||||
- 对 `config`、Keyv 与 `ssh.d` 定义显式转换/私有交付协议；
|
||||
- 在固定物理 Edge/NAS 上测量完整目录盘点与 staging 的耗时、RSS、I/O、磁盘峰值和断电恢复；
|
||||
- 把 staged data directory 证据接入 systemd/OpenRC/Compose cutover 与 rollback lineage。
|
||||
@@ -480,6 +480,7 @@
|
||||
| [ADR-0474](./ADR-0474-bounded-legacy-core-readiness-proof.md) | 有界 Legacy Core Readiness Proof | Accepted（OpenRC live actor 待补） |
|
||||
| [ADR-0475](./ADR-0475-legacy-system-script-open-api-compatibility.md) | Legacy System、Script 与 Open API 兼容基线 | Accepted |
|
||||
| [ADR-0476](./ADR-0476-real-legacy-sqlite-upgrade-and-rollback-rehearsal.md) | 真实 Legacy SQLite 升级与回滚演练 | Accepted |
|
||||
| [ADR-0477](./ADR-0477-bounded-legacy-data-directory-inventory.md) | 有界 Legacy Data Directory 盘点 | Accepted |
|
||||
|
||||
## 规则
|
||||
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
# QingLong 2.x Data Directory 盘点
|
||||
|
||||
本流程为完整 QingLong 2.x `data` 目录生成一个只读、确定性、按 Profile 有界的 3.0 接管计划。它不会复制、压缩、删除或修改
|
||||
任何文件，也不替代 [SQLite 接管流程](./ql3-local-sqlite-adoption.md)。
|
||||
|
||||
## 1. 前置条件
|
||||
|
||||
- 使用最终运行 QingLong 的同一个 POSIX 用户执行；
|
||||
- `dataRoot` 必须是绝对、canonical、非根、非 symlink 的目录；
|
||||
- `dataRoot` 必须由当前 UID 拥有，且 group/world 不可写；
|
||||
- command file 继续遵守 `ql3-adoption` 的当前 UID、canonical、单链接、`0600` 私有文件要求；
|
||||
- 生产盘点建议先停止 2.x writer。若文件或目录在盘点中变化，命令会失败关闭，不会给出部分成功计划。
|
||||
|
||||
## 2. 执行 inspect
|
||||
|
||||
Edge/低配路由设备：
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"operation": "local-data-directory.adoption.inspect",
|
||||
"options": {
|
||||
"dataRoot": "/opt/qinglong/data",
|
||||
"profile": "edge"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Standalone/NAS：
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"operation": "local-data-directory.adoption.inspect",
|
||||
"options": {
|
||||
"dataRoot": "/opt/qinglong/data",
|
||||
"profile": "standalone"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```sh
|
||||
chmod 0600 /secure/operator/ql3-data-directory-inspect.json
|
||||
ql3-adoption run --command-file /secure/operator/ql3-data-directory-inspect.json
|
||||
```
|
||||
|
||||
命令成功时返回 `status=inspected` 和 `qinglong3-legacy-data-directory-adoption-plan`。保存完整 JSON 和 `planDigest`，但不要把它
|
||||
当作已经授权复制的 manifest。
|
||||
|
||||
## 3. 审核处置矩阵
|
||||
|
||||
| 类别 | 默认处置 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| `config` | `transform` | 迁移到 3.0 配置模型，不原样覆盖 |
|
||||
| `scripts` | `copy_reviewed` | 审核兼容性和安全后复制 |
|
||||
| `db` | `transform` | 主 SQLite 走独立流程；审核 Keyv 和 sidecar |
|
||||
| `upload` | `copy_reviewed` | 审核文件类型与容量后复制 |
|
||||
| `ssh.d` | `transform` | 进入后续私有凭据交付，不进入普通归档 |
|
||||
| `log`、`syslog`、`bak` | `retain_external` | 作为历史/恢复资产另行保留 |
|
||||
| `repo`、`raw`、`dep_cache`、`deps` | `regenerate` | 在目标架构重新 checkout/安装 |
|
||||
|
||||
`recursive_content` 类别会稳定读取普通单链接文件并生成摘要；`root_only` 类别只检查类别根，不扫描内部内容。
|
||||
|
||||
## 4. 解释 assessment
|
||||
|
||||
- `reviewable`：没有未知顶层条目，也没有 unsafe 条目；仍需 operator 审核分类、数量、字节和数据库识别计数；
|
||||
- `manual_review`：出现未知顶层条目或 unsafe 条目；不得继续自动 staging；
|
||||
- `broadReadableEntries > 0`：存在 group/world 可读条目，是敏感性审核信号，但不等同于可被外部修改；
|
||||
- `activeSqliteSidecars > 0`：发现 `database.sqlite-*` 或 `keyv.sqlite-*` 活跃 sidecar，先停止 writer 并完成 SQLite
|
||||
checkpoint/一致性处置后重新盘点；
|
||||
- `primaryDatabaseFiles` 应按生产布局识别 `db/database.sqlite`；`legacyKeyValueDatabaseFiles` 识别 `db/keyv.sqlite`。
|
||||
|
||||
输出不会包含 `dataRoot` 原文、任意文件名或文件内容。未知名称只进入摘要。不要尝试从摘要反推或把摘要当作内容备份。
|
||||
|
||||
## 5. Profile 预算
|
||||
|
||||
| Profile | 最大条目 | 最大哈希字节 | 单文件上限 | 最大深度 |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| Edge | 8,192 | 512 MiB | 64 MiB | 32 |
|
||||
| Standalone | 65,536 | 4 GiB | 512 MiB | 64 |
|
||||
|
||||
超过任一限制都会以 `LOCAL_DATA_DIRECTORY_ADOPTION_CONFIGURATION_INVALID` 失败。不要为了通过门禁临时改名、删除或排除资产；先保留
|
||||
现场并决定它应拆分为外部恢复资产、在目标重新生成，还是进入后续人工迁移协议。
|
||||
|
||||
## 6. 常见失败
|
||||
|
||||
- 根目录或条目 group/world 可写：修正 ownership/permission 后重新盘点；
|
||||
- symlink、硬链接或特殊文件：保留现场，确认来源和目标后人工处置；盘点不会跟随或读取；
|
||||
- 目录在盘点中变化：停止 2.x writer、同步器、下载器和仓库更新后重试；
|
||||
- 单文件或总内容超过 Profile 预算：不要改用 `tar` 绕过；为该资产设计独立流式迁移/外部保留流程；
|
||||
- 未知顶层条目：根据插件或用户资产来源登记明确处置，再进入后续 staging 设计。
|
||||
|
||||
## 7. 当前边界
|
||||
|
||||
本命令只生成只读计划。它尚不：
|
||||
|
||||
- 复制、压缩、删除或转换任何资产；
|
||||
- 创建 no-replace stage、verify manifest 或 recovery；
|
||||
- 把目录计划绑定到 SQLite `planDigest`、`manifestDigest` 或 `activationDigest`；
|
||||
- 授权 service-manager/Compose cutover 或 Legacy rollback；
|
||||
- 证明固定物理路由器/NAS 上的耗时、RSS、I/O、磁盘峰值和断电恢复。
|
||||
|
||||
在后续 D-385 stage/verify 合同完成前，保留 inspect 输出和原始 2.x data directory，不要把目录计划当作自动迁移完成证明。
|
||||
@@ -1,6 +1,8 @@
|
||||
import { readPrivateLocalCommandFile } from '@qinglong/local-command-file';
|
||||
|
||||
import { runLegacyCrontabAdoptionCommandFile } from './adoption';
|
||||
import { isLocalDataDirectoryAdoptionOperation } from './data-directory-adoption/contract';
|
||||
import type { LocalDataDirectoryAdoptionInspectResult } from './data-directory-adoption/inventory';
|
||||
import {
|
||||
isLocalSqliteAdoptionProductOperation,
|
||||
type LocalSqliteAdoptionProductOperation,
|
||||
@@ -9,7 +11,8 @@ import type { LocalSqliteAdoptionProductCommandResult } from './sqlite-adoption/
|
||||
|
||||
export type LocalAdoptionProductCommandResult =
|
||||
| Awaited<ReturnType<typeof runLegacyCrontabAdoptionCommandFile>>
|
||||
| LocalSqliteAdoptionProductCommandResult;
|
||||
| LocalSqliteAdoptionProductCommandResult
|
||||
| LocalDataDirectoryAdoptionInspectResult;
|
||||
|
||||
function operation(value: unknown): unknown {
|
||||
return value && typeof value === 'object' && !Array.isArray(value)
|
||||
@@ -28,6 +31,12 @@ export async function runLocalAdoptionProductCommandFile(
|
||||
return runLegacyCrontabAdoptionCommandFile(commandFilePath);
|
||||
}
|
||||
const selected = operation(candidate);
|
||||
if (isLocalDataDirectoryAdoptionOperation(selected)) {
|
||||
const { inspectLocalDataDirectoryAdoption } = await import(
|
||||
'./data-directory-adoption/inventory.js'
|
||||
);
|
||||
return inspectLocalDataDirectoryAdoption(candidate);
|
||||
}
|
||||
if (!isLocalSqliteAdoptionProductOperation(selected)) {
|
||||
return runLegacyCrontabAdoptionCommandFile(commandFilePath);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
import path from 'node:path';
|
||||
|
||||
const MAX_PATH_BYTES = 4_096;
|
||||
|
||||
export const LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION =
|
||||
'local-data-directory.adoption.inspect' as const;
|
||||
|
||||
export interface InspectLocalDataDirectoryAdoptionCommand {
|
||||
readonly schemaVersion: 1;
|
||||
readonly operation: typeof LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION;
|
||||
readonly options: {
|
||||
readonly dataRoot: string;
|
||||
readonly profile: 'edge' | 'standalone';
|
||||
};
|
||||
}
|
||||
|
||||
export class LocalDataDirectoryAdoptionConfigurationError extends TypeError {
|
||||
readonly code = 'LOCAL_DATA_DIRECTORY_ADOPTION_CONFIGURATION_INVALID';
|
||||
|
||||
constructor(message: string, readonly cause?: unknown) {
|
||||
super(`Local data directory adoption configuration is invalid: ${message}`);
|
||||
this.name = 'LocalDataDirectoryAdoptionConfigurationError';
|
||||
}
|
||||
}
|
||||
|
||||
function exactKeys(value: object, expected: readonly string[]): boolean {
|
||||
const actual = Object.keys(value).sort();
|
||||
const canonical = [...expected].sort();
|
||||
return (
|
||||
actual.length === canonical.length &&
|
||||
actual.every((key, index) => key === canonical[index])
|
||||
);
|
||||
}
|
||||
|
||||
function normalizedAbsolutePath(value: unknown): value is string {
|
||||
return (
|
||||
typeof value === 'string' &&
|
||||
path.isAbsolute(value) &&
|
||||
path.parse(value).root !== value &&
|
||||
path.normalize(value) === value &&
|
||||
!value.includes('\0') &&
|
||||
Buffer.byteLength(value, 'utf8') <= MAX_PATH_BYTES
|
||||
);
|
||||
}
|
||||
|
||||
export function isLocalDataDirectoryAdoptionOperation(
|
||||
value: unknown,
|
||||
): value is typeof LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION {
|
||||
return value === LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION;
|
||||
}
|
||||
|
||||
export function normalizeInspectLocalDataDirectoryAdoptionCommand(
|
||||
value: unknown,
|
||||
): Readonly<InspectLocalDataDirectoryAdoptionCommand> {
|
||||
if (
|
||||
!value ||
|
||||
typeof value !== 'object' ||
|
||||
Array.isArray(value) ||
|
||||
!exactKeys(value, ['schemaVersion', 'operation', 'options'])
|
||||
) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'command shape is invalid',
|
||||
);
|
||||
}
|
||||
const candidate = value as Record<string, unknown>;
|
||||
if (
|
||||
candidate.schemaVersion !== 1 ||
|
||||
!isLocalDataDirectoryAdoptionOperation(candidate.operation) ||
|
||||
!candidate.options ||
|
||||
typeof candidate.options !== 'object' ||
|
||||
Array.isArray(candidate.options) ||
|
||||
!exactKeys(candidate.options, ['dataRoot', 'profile'])
|
||||
) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'command value is invalid',
|
||||
);
|
||||
}
|
||||
const options = candidate.options as Record<string, unknown>;
|
||||
if (
|
||||
!normalizedAbsolutePath(options.dataRoot) ||
|
||||
(options.profile !== 'edge' && options.profile !== 'standalone')
|
||||
) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'command options are invalid',
|
||||
);
|
||||
}
|
||||
return Object.freeze(value as InspectLocalDataDirectoryAdoptionCommand);
|
||||
}
|
||||
@@ -0,0 +1,696 @@
|
||||
import crypto from 'node:crypto';
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
|
||||
import {
|
||||
LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION,
|
||||
LocalDataDirectoryAdoptionConfigurationError,
|
||||
normalizeInspectLocalDataDirectoryAdoptionCommand,
|
||||
type InspectLocalDataDirectoryAdoptionCommand,
|
||||
} from './contract';
|
||||
|
||||
const HASH_BUFFER_BYTES = 64 * 1024;
|
||||
const MAX_RELATIVE_PATH_BYTES = 4_096;
|
||||
const DIGEST_PATTERN = /^[0-9a-f]{64}$/;
|
||||
|
||||
type DataDirectoryDisposition =
|
||||
| 'copy_reviewed'
|
||||
| 'transform'
|
||||
| 'retain_external'
|
||||
| 'regenerate';
|
||||
|
||||
type DataDirectoryInspection = 'recursive_content' | 'root_only';
|
||||
|
||||
interface CategoryPolicy {
|
||||
readonly name: string;
|
||||
readonly disposition: DataDirectoryDisposition;
|
||||
readonly inspection: DataDirectoryInspection;
|
||||
}
|
||||
|
||||
interface InventoryBudget {
|
||||
readonly maxEntries: number;
|
||||
readonly maxHashedBytes: number;
|
||||
readonly maxFileBytes: number;
|
||||
readonly maxDepth: number;
|
||||
}
|
||||
|
||||
interface MutableCategorySummary {
|
||||
entries: number;
|
||||
directories: number;
|
||||
regularFiles: number;
|
||||
logicalBytes: number;
|
||||
allocatedBytes: number;
|
||||
broadReadableEntries: number;
|
||||
unsafeEntries: number;
|
||||
activeSqliteSidecars: number;
|
||||
primaryDatabaseFiles: number;
|
||||
legacyKeyValueDatabaseFiles: number;
|
||||
hashedBytes: number;
|
||||
}
|
||||
|
||||
export interface LocalDataDirectoryCategoryEvidence {
|
||||
readonly name: string;
|
||||
readonly disposition: DataDirectoryDisposition;
|
||||
readonly inspection: DataDirectoryInspection;
|
||||
readonly present: boolean;
|
||||
readonly entries: number;
|
||||
readonly directories: number;
|
||||
readonly regularFiles: number;
|
||||
readonly logicalBytes: number | null;
|
||||
readonly allocatedBytes: number | null;
|
||||
readonly broadReadableEntries: number;
|
||||
readonly unsafeEntries: number;
|
||||
readonly activeSqliteSidecars: number;
|
||||
readonly primaryDatabaseFiles: number;
|
||||
readonly legacyKeyValueDatabaseFiles: number;
|
||||
readonly contentDigest: string;
|
||||
}
|
||||
|
||||
export interface LocalDataDirectoryAdoptionEvidence {
|
||||
readonly schemaVersion: 1;
|
||||
readonly kind: 'qinglong3-legacy-data-directory-adoption-plan';
|
||||
readonly profile: 'edge' | 'standalone';
|
||||
readonly policyVersion: 1;
|
||||
readonly dataRootPathDigest: string;
|
||||
readonly budget: Readonly<InventoryBudget>;
|
||||
readonly assessment: 'reviewable' | 'manual_review';
|
||||
readonly categories: readonly LocalDataDirectoryCategoryEvidence[];
|
||||
readonly unknownTopLevelEntries: number;
|
||||
readonly unknownTopLevelDigest: string;
|
||||
readonly totalInspectedEntries: number;
|
||||
readonly totalHashedBytes: number;
|
||||
readonly totalUnsafeEntries: number;
|
||||
readonly planDigest: string;
|
||||
}
|
||||
|
||||
export interface LocalDataDirectoryAdoptionInspectResult {
|
||||
readonly schemaVersion: 1;
|
||||
readonly operation: typeof LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION;
|
||||
readonly status: 'inspected';
|
||||
readonly evidence: Readonly<LocalDataDirectoryAdoptionEvidence>;
|
||||
}
|
||||
|
||||
const POLICIES: readonly CategoryPolicy[] = Object.freeze([
|
||||
Object.freeze({
|
||||
name: 'config',
|
||||
disposition: 'transform',
|
||||
inspection: 'recursive_content',
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'scripts',
|
||||
disposition: 'copy_reviewed',
|
||||
inspection: 'recursive_content',
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'db',
|
||||
disposition: 'transform',
|
||||
inspection: 'recursive_content',
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'upload',
|
||||
disposition: 'copy_reviewed',
|
||||
inspection: 'recursive_content',
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'ssh.d',
|
||||
disposition: 'transform',
|
||||
inspection: 'recursive_content',
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'log',
|
||||
disposition: 'retain_external',
|
||||
inspection: 'root_only',
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'syslog',
|
||||
disposition: 'retain_external',
|
||||
inspection: 'root_only',
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'bak',
|
||||
disposition: 'retain_external',
|
||||
inspection: 'root_only',
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'repo',
|
||||
disposition: 'regenerate',
|
||||
inspection: 'root_only',
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'raw',
|
||||
disposition: 'regenerate',
|
||||
inspection: 'root_only',
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'dep_cache',
|
||||
disposition: 'regenerate',
|
||||
inspection: 'root_only',
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'deps',
|
||||
disposition: 'regenerate',
|
||||
inspection: 'root_only',
|
||||
}),
|
||||
]);
|
||||
|
||||
const EMPTY_DIGEST = crypto.createHash('sha256').digest('hex');
|
||||
|
||||
function digestText(value: string): string {
|
||||
return crypto.createHash('sha256').update(value, 'utf8').digest('hex');
|
||||
}
|
||||
|
||||
function budget(profile: 'edge' | 'standalone'): Readonly<InventoryBudget> {
|
||||
return Object.freeze(
|
||||
profile === 'edge'
|
||||
? {
|
||||
maxEntries: 8_192,
|
||||
maxHashedBytes: 512 * 1024 * 1024,
|
||||
maxFileBytes: 64 * 1024 * 1024,
|
||||
maxDepth: 32,
|
||||
}
|
||||
: {
|
||||
maxEntries: 65_536,
|
||||
maxHashedBytes: 4 * 1024 * 1024 * 1024,
|
||||
maxFileBytes: 512 * 1024 * 1024,
|
||||
maxDepth: 64,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
function currentUid(): number {
|
||||
if (typeof process.getuid !== 'function') {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'POSIX ownership is unavailable',
|
||||
);
|
||||
}
|
||||
return process.getuid();
|
||||
}
|
||||
|
||||
function sameStat(left: fs.BigIntStats, right: fs.BigIntStats): boolean {
|
||||
return (
|
||||
left.dev === right.dev &&
|
||||
left.ino === right.ino &&
|
||||
left.mode === right.mode &&
|
||||
left.nlink === right.nlink &&
|
||||
left.uid === right.uid &&
|
||||
left.size === right.size &&
|
||||
left.mtimeNs === right.mtimeNs &&
|
||||
left.ctimeNs === right.ctimeNs
|
||||
);
|
||||
}
|
||||
|
||||
function stableRoot(dataRoot: string, uid: number): fs.BigIntStats {
|
||||
let stat: fs.BigIntStats;
|
||||
try {
|
||||
stat = fs.lstatSync(dataRoot, { bigint: true });
|
||||
} catch (error) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'dataRoot is unavailable',
|
||||
error,
|
||||
);
|
||||
}
|
||||
if (
|
||||
!stat.isDirectory() ||
|
||||
stat.isSymbolicLink() ||
|
||||
stat.uid !== BigInt(uid) ||
|
||||
(stat.mode & 0o022n) !== 0n ||
|
||||
fs.realpathSync(dataRoot) !== dataRoot
|
||||
) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'dataRoot must be an owner-controlled canonical directory',
|
||||
);
|
||||
}
|
||||
return stat;
|
||||
}
|
||||
|
||||
function sortedDirectoryNames(
|
||||
directoryPath: string,
|
||||
maxNames: number,
|
||||
): readonly string[] {
|
||||
const directory = fs.opendirSync(directoryPath);
|
||||
const names: string[] = [];
|
||||
try {
|
||||
for (;;) {
|
||||
const entry = directory.readSync();
|
||||
if (entry === null) break;
|
||||
names.push(entry.name);
|
||||
if (names.length > maxNames) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data directory entry count exceeds the Profile budget',
|
||||
);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
directory.closeSync();
|
||||
}
|
||||
return names.sort((left, right) =>
|
||||
Buffer.compare(Buffer.from(left, 'utf8'), Buffer.from(right, 'utf8')),
|
||||
);
|
||||
}
|
||||
|
||||
function relativePath(base: string, candidate: string): string {
|
||||
const relative = path.relative(base, candidate);
|
||||
if (
|
||||
relative.length < 1 ||
|
||||
path.isAbsolute(relative) ||
|
||||
relative === '..' ||
|
||||
relative.startsWith(`..${path.sep}`) ||
|
||||
Buffer.byteLength(relative, 'utf8') > MAX_RELATIVE_PATH_BYTES
|
||||
) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data directory entry path is invalid or too long',
|
||||
);
|
||||
}
|
||||
return relative.split(path.sep).join('/');
|
||||
}
|
||||
|
||||
function updateMetadata(
|
||||
hash: crypto.Hash,
|
||||
relative: string,
|
||||
stat: fs.BigIntStats,
|
||||
kind: string,
|
||||
contentDigest?: string,
|
||||
): void {
|
||||
hash.update(
|
||||
`${JSON.stringify({
|
||||
relative,
|
||||
kind,
|
||||
mode: (stat.mode & 0o777n).toString(8),
|
||||
uid: stat.uid.toString(),
|
||||
links: stat.nlink.toString(),
|
||||
bytes: stat.size.toString(),
|
||||
modifiedAtNs: stat.mtimeNs.toString(),
|
||||
changedAtNs: stat.ctimeNs.toString(),
|
||||
...(contentDigest === undefined ? {} : { contentDigest }),
|
||||
})}\n`,
|
||||
'utf8',
|
||||
);
|
||||
}
|
||||
|
||||
function stableFileDigest(filePath: string, expected: fs.BigIntStats): string {
|
||||
const descriptor = fs.openSync(
|
||||
filePath,
|
||||
fs.constants.O_RDONLY | (fs.constants.O_NOFOLLOW ?? 0),
|
||||
);
|
||||
const buffer = Buffer.allocUnsafe(HASH_BUFFER_BYTES);
|
||||
try {
|
||||
const before = fs.fstatSync(descriptor, { bigint: true });
|
||||
if (!sameStat(expected, before) || !before.isFile()) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data directory file identity changed before inspection',
|
||||
);
|
||||
}
|
||||
const hash = crypto.createHash('sha256');
|
||||
for (;;) {
|
||||
const count = fs.readSync(descriptor, buffer, 0, buffer.length, null);
|
||||
if (count === 0) break;
|
||||
hash.update(buffer.subarray(0, count));
|
||||
}
|
||||
const after = fs.fstatSync(descriptor, { bigint: true });
|
||||
if (!sameStat(before, after)) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data directory file changed during inspection',
|
||||
);
|
||||
}
|
||||
return hash.digest('hex');
|
||||
} finally {
|
||||
buffer.fill(0);
|
||||
fs.closeSync(descriptor);
|
||||
}
|
||||
}
|
||||
|
||||
function mutableSummary(): MutableCategorySummary {
|
||||
return {
|
||||
entries: 0,
|
||||
directories: 0,
|
||||
regularFiles: 0,
|
||||
logicalBytes: 0,
|
||||
allocatedBytes: 0,
|
||||
broadReadableEntries: 0,
|
||||
unsafeEntries: 0,
|
||||
activeSqliteSidecars: 0,
|
||||
primaryDatabaseFiles: 0,
|
||||
legacyKeyValueDatabaseFiles: 0,
|
||||
hashedBytes: 0,
|
||||
};
|
||||
}
|
||||
|
||||
function safeNumber(value: bigint, label: string): number {
|
||||
if (value < 0n || value > BigInt(Number.MAX_SAFE_INTEGER)) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
`${label} exceeds the supported numeric range`,
|
||||
);
|
||||
}
|
||||
return Number(value);
|
||||
}
|
||||
|
||||
function addEntryBytes(
|
||||
summary: MutableCategorySummary,
|
||||
stat: fs.BigIntStats,
|
||||
): void {
|
||||
const logical = safeNumber(stat.size, 'entry bytes');
|
||||
const allocated = safeNumber(stat.blocks * 512n, 'allocated entry bytes');
|
||||
if (
|
||||
!Number.isSafeInteger(summary.logicalBytes + logical) ||
|
||||
!Number.isSafeInteger(summary.allocatedBytes + allocated)
|
||||
) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'category byte total exceeds the supported numeric range',
|
||||
);
|
||||
}
|
||||
summary.logicalBytes += logical;
|
||||
summary.allocatedBytes += allocated;
|
||||
}
|
||||
|
||||
function inspectRecursiveCategory(
|
||||
dataRoot: string,
|
||||
categoryRoot: string,
|
||||
rootStat: fs.BigIntStats,
|
||||
uid: number,
|
||||
limits: Readonly<InventoryBudget>,
|
||||
shared: { entries: number; hashedBytes: number },
|
||||
): Readonly<{
|
||||
summary: MutableCategorySummary;
|
||||
contentDigest: string;
|
||||
}> {
|
||||
const summary = mutableSummary();
|
||||
const hash = crypto.createHash('sha256');
|
||||
|
||||
const visitDirectory = (
|
||||
directoryPath: string,
|
||||
expected: fs.BigIntStats,
|
||||
depth: number,
|
||||
): void => {
|
||||
if (depth > limits.maxDepth) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data directory depth exceeds the Profile budget',
|
||||
);
|
||||
}
|
||||
const names = sortedDirectoryNames(
|
||||
directoryPath,
|
||||
limits.maxEntries - shared.entries,
|
||||
);
|
||||
for (const name of names) {
|
||||
const entryPath = path.join(directoryPath, name);
|
||||
const relative = relativePath(dataRoot, entryPath);
|
||||
const stat = fs.lstatSync(entryPath, { bigint: true });
|
||||
shared.entries += 1;
|
||||
summary.entries += 1;
|
||||
if (shared.entries > limits.maxEntries) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data directory entry count exceeds the Profile budget',
|
||||
);
|
||||
}
|
||||
addEntryBytes(summary, stat);
|
||||
const mode = stat.mode & 0o777n;
|
||||
if ((mode & 0o044n) !== 0n) summary.broadReadableEntries += 1;
|
||||
const unsafeIdentity =
|
||||
stat.uid !== BigInt(uid) ||
|
||||
(mode & 0o022n) !== 0n ||
|
||||
stat.isSymbolicLink() ||
|
||||
(!stat.isDirectory() && !stat.isFile()) ||
|
||||
(stat.isFile() && stat.nlink !== 1n);
|
||||
if (unsafeIdentity) {
|
||||
summary.unsafeEntries += 1;
|
||||
updateMetadata(hash, relative, stat, 'unsafe');
|
||||
continue;
|
||||
}
|
||||
if (stat.isDirectory()) {
|
||||
summary.directories += 1;
|
||||
updateMetadata(hash, relative, stat, 'directory');
|
||||
visitDirectory(entryPath, stat, depth + 1);
|
||||
const after = fs.lstatSync(entryPath, { bigint: true });
|
||||
if (!sameStat(stat, after)) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data directory changed during inspection',
|
||||
);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const size = safeNumber(stat.size, 'file bytes');
|
||||
if (size > limits.maxFileBytes) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data directory file exceeds the Profile budget',
|
||||
);
|
||||
}
|
||||
if (shared.hashedBytes + size > limits.maxHashedBytes) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data directory hashed bytes exceed the Profile budget',
|
||||
);
|
||||
}
|
||||
shared.hashedBytes += size;
|
||||
summary.hashedBytes += size;
|
||||
summary.regularFiles += 1;
|
||||
const baseName = path.basename(entryPath);
|
||||
if (relative === 'db/database.sqlite') summary.primaryDatabaseFiles += 1;
|
||||
if (relative === 'db/keyv.sqlite')
|
||||
summary.legacyKeyValueDatabaseFiles += 1;
|
||||
if (/^(?:database|keyv)\.sqlite-(?:wal|shm|journal)$/.test(baseName)) {
|
||||
summary.activeSqliteSidecars += 1;
|
||||
}
|
||||
updateMetadata(
|
||||
hash,
|
||||
relative,
|
||||
stat,
|
||||
'file',
|
||||
stableFileDigest(entryPath, stat),
|
||||
);
|
||||
}
|
||||
const after = fs.lstatSync(directoryPath, { bigint: true });
|
||||
if (!sameStat(expected, after)) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data directory changed during inspection',
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
visitDirectory(categoryRoot, rootStat, 1);
|
||||
return Object.freeze({ summary, contentDigest: hash.digest('hex') });
|
||||
}
|
||||
|
||||
function inspectCategory(
|
||||
policy: Readonly<CategoryPolicy>,
|
||||
dataRoot: string,
|
||||
uid: number,
|
||||
limits: Readonly<InventoryBudget>,
|
||||
shared: { entries: number; hashedBytes: number },
|
||||
): Readonly<LocalDataDirectoryCategoryEvidence> {
|
||||
const categoryRoot = path.join(dataRoot, policy.name);
|
||||
let stat: fs.BigIntStats;
|
||||
try {
|
||||
stat = fs.lstatSync(categoryRoot, { bigint: true });
|
||||
} catch (error) {
|
||||
if (
|
||||
error &&
|
||||
typeof error === 'object' &&
|
||||
'code' in error &&
|
||||
error.code === 'ENOENT'
|
||||
) {
|
||||
return Object.freeze({
|
||||
...policy,
|
||||
present: false,
|
||||
entries: 0,
|
||||
directories: 0,
|
||||
regularFiles: 0,
|
||||
logicalBytes: policy.inspection === 'root_only' ? null : 0,
|
||||
allocatedBytes: policy.inspection === 'root_only' ? null : 0,
|
||||
broadReadableEntries: 0,
|
||||
unsafeEntries: 0,
|
||||
activeSqliteSidecars: 0,
|
||||
primaryDatabaseFiles: 0,
|
||||
legacyKeyValueDatabaseFiles: 0,
|
||||
contentDigest: EMPTY_DIGEST,
|
||||
});
|
||||
}
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data category is unavailable',
|
||||
error,
|
||||
);
|
||||
}
|
||||
const mode = stat.mode & 0o777n;
|
||||
const safeDirectory =
|
||||
stat.isDirectory() &&
|
||||
!stat.isSymbolicLink() &&
|
||||
stat.uid === BigInt(uid) &&
|
||||
(mode & 0o022n) === 0n;
|
||||
if (!safeDirectory || policy.inspection === 'root_only') {
|
||||
const evidence = Object.freeze({
|
||||
...policy,
|
||||
present: true,
|
||||
entries: 0,
|
||||
directories: 0,
|
||||
regularFiles: 0,
|
||||
logicalBytes: null,
|
||||
allocatedBytes: null,
|
||||
broadReadableEntries: (mode & 0o044n) !== 0n ? 1 : 0,
|
||||
unsafeEntries: safeDirectory ? 0 : 1,
|
||||
activeSqliteSidecars: 0,
|
||||
primaryDatabaseFiles: 0,
|
||||
legacyKeyValueDatabaseFiles: 0,
|
||||
contentDigest: digestText(
|
||||
JSON.stringify({
|
||||
category: policy.name,
|
||||
mode: mode.toString(8),
|
||||
uid: stat.uid.toString(),
|
||||
kind: safeDirectory ? 'directory' : 'unsafe',
|
||||
}),
|
||||
),
|
||||
});
|
||||
const after = fs.lstatSync(categoryRoot, { bigint: true });
|
||||
if (!sameStat(stat, after)) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data category changed during inspection',
|
||||
);
|
||||
}
|
||||
return evidence;
|
||||
}
|
||||
const inspected = inspectRecursiveCategory(
|
||||
dataRoot,
|
||||
categoryRoot,
|
||||
stat,
|
||||
uid,
|
||||
limits,
|
||||
shared,
|
||||
);
|
||||
return Object.freeze({
|
||||
...policy,
|
||||
present: true,
|
||||
entries: inspected.summary.entries,
|
||||
directories: inspected.summary.directories,
|
||||
regularFiles: inspected.summary.regularFiles,
|
||||
logicalBytes: inspected.summary.logicalBytes,
|
||||
allocatedBytes: inspected.summary.allocatedBytes,
|
||||
broadReadableEntries: inspected.summary.broadReadableEntries,
|
||||
unsafeEntries: inspected.summary.unsafeEntries,
|
||||
activeSqliteSidecars: inspected.summary.activeSqliteSidecars,
|
||||
primaryDatabaseFiles: inspected.summary.primaryDatabaseFiles,
|
||||
legacyKeyValueDatabaseFiles: inspected.summary.legacyKeyValueDatabaseFiles,
|
||||
contentDigest: inspected.contentDigest,
|
||||
});
|
||||
}
|
||||
|
||||
function unknownTopLevelEvidence(
|
||||
dataRoot: string,
|
||||
known: ReadonlySet<string>,
|
||||
limits: Readonly<InventoryBudget>,
|
||||
shared: { entries: number; hashedBytes: number },
|
||||
): Readonly<{ count: number; digest: string }> {
|
||||
const hash = crypto.createHash('sha256');
|
||||
let count = 0;
|
||||
const names = sortedDirectoryNames(
|
||||
dataRoot,
|
||||
limits.maxEntries - shared.entries + known.size,
|
||||
);
|
||||
for (const name of names) {
|
||||
if (known.has(name)) continue;
|
||||
count += 1;
|
||||
shared.entries += 1;
|
||||
if (shared.entries > limits.maxEntries) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data directory entry count exceeds the Profile budget',
|
||||
);
|
||||
}
|
||||
const stat = fs.lstatSync(path.join(dataRoot, name), { bigint: true });
|
||||
hash.update(
|
||||
`${JSON.stringify({
|
||||
nameDigest: digestText(name),
|
||||
mode: (stat.mode & 0o777n).toString(8),
|
||||
uid: stat.uid.toString(),
|
||||
kind: stat.isDirectory()
|
||||
? 'directory'
|
||||
: stat.isFile()
|
||||
? 'file'
|
||||
: stat.isSymbolicLink()
|
||||
? 'symlink'
|
||||
: 'special',
|
||||
})}\n`,
|
||||
'utf8',
|
||||
);
|
||||
const after = fs.lstatSync(path.join(dataRoot, name), { bigint: true });
|
||||
if (!sameStat(stat, after)) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'unknown data directory entry changed during inspection',
|
||||
);
|
||||
}
|
||||
}
|
||||
return Object.freeze({ count, digest: hash.digest('hex') });
|
||||
}
|
||||
|
||||
function planDigest(
|
||||
evidence: Omit<LocalDataDirectoryAdoptionEvidence, 'planDigest'>,
|
||||
): string {
|
||||
return digestText(JSON.stringify(evidence));
|
||||
}
|
||||
|
||||
export function inspectLocalDataDirectoryAdoption(
|
||||
value: unknown,
|
||||
): Readonly<LocalDataDirectoryAdoptionInspectResult> {
|
||||
try {
|
||||
const command: Readonly<InspectLocalDataDirectoryAdoptionCommand> =
|
||||
normalizeInspectLocalDataDirectoryAdoptionCommand(value);
|
||||
const uid = currentUid();
|
||||
const rootBefore = stableRoot(command.options.dataRoot, uid);
|
||||
const limits = budget(command.options.profile);
|
||||
const shared = { entries: 0, hashedBytes: 0 };
|
||||
const categories = POLICIES.map((policy) =>
|
||||
inspectCategory(policy, command.options.dataRoot, uid, limits, shared),
|
||||
);
|
||||
const unknown = unknownTopLevelEvidence(
|
||||
command.options.dataRoot,
|
||||
new Set(POLICIES.map((policy) => policy.name)),
|
||||
limits,
|
||||
shared,
|
||||
);
|
||||
const rootAfter = stableRoot(command.options.dataRoot, uid);
|
||||
if (!sameStat(rootBefore, rootAfter)) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'dataRoot changed during inspection',
|
||||
);
|
||||
}
|
||||
const totalUnsafeEntries = categories.reduce(
|
||||
(total, category) => total + category.unsafeEntries,
|
||||
0,
|
||||
);
|
||||
const payload = Object.freeze({
|
||||
schemaVersion: 1 as const,
|
||||
kind: 'qinglong3-legacy-data-directory-adoption-plan' as const,
|
||||
profile: command.options.profile,
|
||||
policyVersion: 1 as const,
|
||||
dataRootPathDigest: digestText(command.options.dataRoot),
|
||||
budget: limits,
|
||||
assessment:
|
||||
totalUnsafeEntries === 0 && unknown.count === 0
|
||||
? ('reviewable' as const)
|
||||
: ('manual_review' as const),
|
||||
categories: Object.freeze(categories),
|
||||
unknownTopLevelEntries: unknown.count,
|
||||
unknownTopLevelDigest: unknown.digest,
|
||||
totalInspectedEntries: shared.entries,
|
||||
totalHashedBytes: shared.hashedBytes,
|
||||
totalUnsafeEntries,
|
||||
});
|
||||
const evidence = Object.freeze({
|
||||
...payload,
|
||||
planDigest: planDigest(payload),
|
||||
});
|
||||
if (!DIGEST_PATTERN.test(evidence.planDigest)) {
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data directory plan digest is invalid',
|
||||
);
|
||||
}
|
||||
return Object.freeze({
|
||||
schemaVersion: 1,
|
||||
operation: LOCAL_DATA_DIRECTORY_ADOPTION_INSPECT_OPERATION,
|
||||
status: 'inspected',
|
||||
evidence,
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof LocalDataDirectoryAdoptionConfigurationError) {
|
||||
throw error;
|
||||
}
|
||||
throw new LocalDataDirectoryAdoptionConfigurationError(
|
||||
'data directory inspection failed',
|
||||
error,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,274 @@
|
||||
const assert = require('node:assert/strict');
|
||||
const { spawnSync } = require('node:child_process');
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const { test } = require('node:test');
|
||||
|
||||
const BINARY = path.join(__dirname, '../dist/lifecycle/adoptionCli.js');
|
||||
const OPERATION = 'local-data-directory.adoption.inspect';
|
||||
|
||||
const RECURSIVE_CATEGORIES = Object.freeze([
|
||||
['config', 'transform'],
|
||||
['scripts', 'copy_reviewed'],
|
||||
['db', 'transform'],
|
||||
['upload', 'copy_reviewed'],
|
||||
['ssh.d', 'transform'],
|
||||
]);
|
||||
const ROOT_ONLY_CATEGORIES = Object.freeze([
|
||||
['log', 'retain_external'],
|
||||
['syslog', 'retain_external'],
|
||||
['bak', 'retain_external'],
|
||||
['repo', 'regenerate'],
|
||||
['raw', 'regenerate'],
|
||||
['dep_cache', 'regenerate'],
|
||||
['deps', 'regenerate'],
|
||||
]);
|
||||
|
||||
function privateDirectory(directoryPath) {
|
||||
fs.mkdirSync(directoryPath, { recursive: true, mode: 0o700 });
|
||||
fs.chmodSync(directoryPath, 0o700);
|
||||
}
|
||||
|
||||
function privateFile(filePath, content) {
|
||||
privateDirectory(path.dirname(filePath));
|
||||
fs.writeFileSync(filePath, content, { mode: 0o600 });
|
||||
fs.chmodSync(filePath, 0o600);
|
||||
}
|
||||
|
||||
function fixture(t) {
|
||||
const root = fs.realpathSync(
|
||||
fs.mkdtempSync(path.join(os.tmpdir(), 'ql3-data-directory-adoption-')),
|
||||
);
|
||||
fs.chmodSync(root, 0o700);
|
||||
t.after(() => fs.rmSync(root, { recursive: true, force: true }));
|
||||
const commandsDirectory = path.join(root, 'commands');
|
||||
const dataRoot = path.join(root, 'data');
|
||||
privateDirectory(commandsDirectory);
|
||||
privateDirectory(dataRoot);
|
||||
|
||||
privateFile(path.join(dataRoot, 'config', 'config.sh'), 'export A=1\n');
|
||||
privateFile(
|
||||
path.join(dataRoot, 'scripts', 'jobs', 'example.sh'),
|
||||
'echo qinglong\n',
|
||||
);
|
||||
privateFile(path.join(dataRoot, 'db', 'database.sqlite'), 'legacy-primary');
|
||||
privateFile(path.join(dataRoot, 'db', 'keyv.sqlite'), 'legacy-keyv');
|
||||
privateFile(path.join(dataRoot, 'upload', 'avatar.bin'), Buffer.from([1, 2]));
|
||||
privateFile(path.join(dataRoot, 'ssh.d', 'repository-key'), 'private-key');
|
||||
|
||||
for (const [category] of ROOT_ONLY_CATEGORIES) {
|
||||
privateFile(
|
||||
path.join(dataRoot, category, 'nested', 'ignored-content'),
|
||||
`ignored-${category}`,
|
||||
);
|
||||
}
|
||||
return { root, commandsDirectory, dataRoot };
|
||||
}
|
||||
|
||||
function runRaw(value, name, command) {
|
||||
const commandPath = path.join(value.commandsDirectory, `${name}.json`);
|
||||
fs.writeFileSync(commandPath, `${JSON.stringify(command)}\n`, {
|
||||
mode: 0o600,
|
||||
});
|
||||
fs.chmodSync(commandPath, 0o600);
|
||||
return spawnSync(
|
||||
process.execPath,
|
||||
[BINARY, 'run', '--command-file', commandPath],
|
||||
{ encoding: 'utf8' },
|
||||
);
|
||||
}
|
||||
|
||||
function inspect(value, name = 'inspect', profile = 'edge') {
|
||||
const child = runRaw(value, name, {
|
||||
schemaVersion: 1,
|
||||
operation: OPERATION,
|
||||
options: { dataRoot: value.dataRoot, profile },
|
||||
});
|
||||
assert.equal(child.status, 0, child.stderr);
|
||||
assert.equal(child.stderr, '');
|
||||
return { child, result: JSON.parse(child.stdout) };
|
||||
}
|
||||
|
||||
function categoryMap(result) {
|
||||
return new Map(
|
||||
result.evidence.categories.map((category) => [category.name, category]),
|
||||
);
|
||||
}
|
||||
|
||||
test('data directory adoption emits a deterministic content-free migration plan', (t) => {
|
||||
const value = fixture(t);
|
||||
const first = inspect(value, 'first');
|
||||
const second = inspect(value, 'second');
|
||||
|
||||
assert.equal(first.result.schemaVersion, 1);
|
||||
assert.equal(first.result.operation, OPERATION);
|
||||
assert.equal(first.result.status, 'inspected');
|
||||
assert.equal(
|
||||
first.result.evidence.kind,
|
||||
'qinglong3-legacy-data-directory-adoption-plan',
|
||||
);
|
||||
assert.equal(first.result.evidence.profile, 'edge');
|
||||
assert.equal(first.result.evidence.policyVersion, 1);
|
||||
assert.equal(first.result.evidence.assessment, 'reviewable');
|
||||
assert.equal(first.result.evidence.unknownTopLevelEntries, 0);
|
||||
assert.equal(first.result.evidence.totalInspectedEntries, 7);
|
||||
assert.equal(first.result.evidence.totalUnsafeEntries, 0);
|
||||
assert.match(first.result.evidence.planDigest, /^[0-9a-f]{64}$/);
|
||||
assert.deepEqual(second.result, first.result);
|
||||
|
||||
const categories = categoryMap(first.result);
|
||||
assert.equal(categories.size, 12);
|
||||
for (const [name, disposition] of RECURSIVE_CATEGORIES) {
|
||||
assert.deepEqual(
|
||||
{
|
||||
disposition: categories.get(name).disposition,
|
||||
inspection: categories.get(name).inspection,
|
||||
present: categories.get(name).present,
|
||||
},
|
||||
{ disposition, inspection: 'recursive_content', present: true },
|
||||
);
|
||||
}
|
||||
for (const [name, disposition] of ROOT_ONLY_CATEGORIES) {
|
||||
assert.deepEqual(
|
||||
{
|
||||
disposition: categories.get(name).disposition,
|
||||
inspection: categories.get(name).inspection,
|
||||
present: categories.get(name).present,
|
||||
entries: categories.get(name).entries,
|
||||
logicalBytes: categories.get(name).logicalBytes,
|
||||
},
|
||||
{
|
||||
disposition,
|
||||
inspection: 'root_only',
|
||||
present: true,
|
||||
entries: 0,
|
||||
logicalBytes: null,
|
||||
},
|
||||
);
|
||||
}
|
||||
assert.equal(categories.get('db').primaryDatabaseFiles, 1);
|
||||
assert.equal(categories.get('db').legacyKeyValueDatabaseFiles, 1);
|
||||
|
||||
assert.equal(first.child.stdout.includes(value.dataRoot), false);
|
||||
assert.equal(first.child.stdout.includes('repository-key'), false);
|
||||
assert.equal(first.child.stdout.includes('private-key'), false);
|
||||
assert.equal(first.child.stdout.includes('example.sh'), false);
|
||||
});
|
||||
|
||||
test('root-only cache and history contents do not enter the adoption plan', (t) => {
|
||||
const value = fixture(t);
|
||||
const before = inspect(value, 'before').result;
|
||||
privateFile(
|
||||
path.join(value.dataRoot, 'repo', 'nested', 'ignored-content'),
|
||||
'different-cross-architecture-cache',
|
||||
);
|
||||
const afterRootOnlyChange = inspect(value, 'after-root-only').result;
|
||||
assert.deepEqual(afterRootOnlyChange, before);
|
||||
|
||||
privateFile(
|
||||
path.join(value.dataRoot, 'scripts', 'jobs', 'example.sh'),
|
||||
'echo changed\n',
|
||||
);
|
||||
const afterRelevantChange = inspect(value, 'after-relevant').result;
|
||||
assert.notEqual(
|
||||
afterRelevantChange.evidence.planDigest,
|
||||
before.evidence.planDigest,
|
||||
);
|
||||
});
|
||||
|
||||
test('links and unknown top-level entries fail closed without leaking names', (t) => {
|
||||
const value = fixture(t);
|
||||
const externalSecret = path.join(value.root, 'external-sensitive-value');
|
||||
privateFile(externalSecret, 'must-not-be-read');
|
||||
fs.linkSync(
|
||||
externalSecret,
|
||||
path.join(value.dataRoot, 'scripts', 'jobs', 'hard-linked-secret'),
|
||||
);
|
||||
fs.symlinkSync(
|
||||
externalSecret,
|
||||
path.join(value.dataRoot, 'scripts', 'jobs', 'linked-secret'),
|
||||
);
|
||||
const unknownName = 'customer-private-extension';
|
||||
privateFile(
|
||||
path.join(value.dataRoot, unknownName),
|
||||
'unknown-sensitive-value',
|
||||
);
|
||||
|
||||
const inspected = inspect(value, 'unsafe');
|
||||
assert.equal(inspected.result.evidence.assessment, 'manual_review');
|
||||
assert.equal(inspected.result.evidence.totalUnsafeEntries, 2);
|
||||
assert.equal(inspected.result.evidence.unknownTopLevelEntries, 1);
|
||||
assert.equal(categoryMap(inspected.result).get('scripts').unsafeEntries, 2);
|
||||
for (const sensitive of [
|
||||
value.dataRoot,
|
||||
externalSecret,
|
||||
'hard-linked-secret',
|
||||
'linked-secret',
|
||||
unknownName,
|
||||
'must-not-be-read',
|
||||
'unknown-sensitive-value',
|
||||
]) {
|
||||
assert.equal(inspected.child.stdout.includes(sensitive), false);
|
||||
}
|
||||
});
|
||||
|
||||
test('widened commands and unsafe roots are rejected with a stable public error', (t) => {
|
||||
const value = fixture(t);
|
||||
const widened = runRaw(value, 'widened', {
|
||||
schemaVersion: 1,
|
||||
operation: OPERATION,
|
||||
options: {
|
||||
dataRoot: value.dataRoot,
|
||||
profile: 'edge',
|
||||
extraAuthority: true,
|
||||
},
|
||||
});
|
||||
assert.equal(widened.status, 1);
|
||||
assert.equal(widened.stdout, '');
|
||||
assert.equal(
|
||||
JSON.parse(widened.stderr).code,
|
||||
'LOCAL_DATA_DIRECTORY_ADOPTION_CONFIGURATION_INVALID',
|
||||
);
|
||||
|
||||
fs.chmodSync(value.dataRoot, 0o777);
|
||||
const unsafe = runRaw(value, 'unsafe-root', {
|
||||
schemaVersion: 1,
|
||||
operation: OPERATION,
|
||||
options: { dataRoot: value.dataRoot, profile: 'edge' },
|
||||
});
|
||||
assert.equal(unsafe.status, 1);
|
||||
assert.equal(unsafe.stdout, '');
|
||||
const error = JSON.parse(unsafe.stderr);
|
||||
assert.equal(
|
||||
error.code,
|
||||
'LOCAL_DATA_DIRECTORY_ADOPTION_CONFIGURATION_INVALID',
|
||||
);
|
||||
assert.equal(unsafe.stderr.includes(value.dataRoot), false);
|
||||
});
|
||||
|
||||
test('edge inspection enforces a per-file budget before reading content', (t) => {
|
||||
const value = fixture(t);
|
||||
const oversized = path.join(
|
||||
value.dataRoot,
|
||||
'scripts',
|
||||
'oversized-private-file',
|
||||
);
|
||||
privateFile(oversized, '');
|
||||
fs.truncateSync(oversized, 64 * 1024 * 1024 + 1);
|
||||
|
||||
const child = runRaw(value, 'oversized', {
|
||||
schemaVersion: 1,
|
||||
operation: OPERATION,
|
||||
options: { dataRoot: value.dataRoot, profile: 'edge' },
|
||||
});
|
||||
assert.equal(child.status, 1);
|
||||
assert.equal(child.stdout, '');
|
||||
const error = JSON.parse(child.stderr);
|
||||
assert.equal(
|
||||
error.code,
|
||||
'LOCAL_DATA_DIRECTORY_ADOPTION_CONFIGURATION_INVALID',
|
||||
);
|
||||
assert.match(error.message, /Profile budget/);
|
||||
assert.equal(child.stderr.includes(oversized), false);
|
||||
});
|
||||
@@ -207,10 +207,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
|
||||
rootSourceFileRoles: localOwnerCli.rootSourceFileRoles,
|
||||
},
|
||||
{
|
||||
sourceFiles: 116,
|
||||
sourceFiles: 118,
|
||||
rootSourceFiles: 1,
|
||||
rootSourceLines: 50,
|
||||
nestedSourceFiles: 115,
|
||||
nestedSourceFiles: 117,
|
||||
rootSourceFileRoles: { 'cli.ts': 'binary_entry' },
|
||||
},
|
||||
);
|
||||
|
||||
Reference in New Issue
Block a user