mirror of
https://github.com/whyour/qinglong.git
synced 2026-09-20 08:05:22 +08:00
feat(ql3): prove shadow resource rollback
This commit is contained in:
whyour
@@ -143,7 +143,7 @@ jobs:
|
||||
pnpm build:back
|
||||
pnpm run build:packages:ql3
|
||||
- name: Test fail-closed Linux resource gate
|
||||
run: node --test test/back/ql3LinuxResourceGate.test.cjs test/back/ql3LinuxResourceWorkflow.test.cjs test/back/ql3PhysicalEdgeEvidence.test.cjs test/back/ql3PhysicalEdgeIdleSampler.test.cjs test/back/ql3PhysicalEdgeFaultProbe.test.cjs test/back/ql3PhysicalEdgeTaskScale.test.cjs
|
||||
run: node --test test/back/ql3LinuxResourceGate.test.cjs test/back/ql3LinuxResourceWorkflow.test.cjs test/back/ql3LegacyShadowResourceRollback.test.cjs test/back/ql3PhysicalEdgeEvidence.test.cjs test/back/ql3PhysicalEdgeIdleSampler.test.cjs test/back/ql3PhysicalEdgeFaultProbe.test.cjs test/back/ql3PhysicalEdgeTaskScale.test.cjs
|
||||
- name: Prepare architecture evidence directory
|
||||
run: mkdir -p "${RUNNER_TEMP}/ql3-linux-resource-evidence"
|
||||
- name: Stress router headroom without claiming a supported minimum
|
||||
|
||||
@@ -11,6 +11,20 @@
|
||||
|
||||
最新增量证据（2026-08-19）：
|
||||
|
||||
- D-359/ADR-0451、ADR-0452（已接受）：为 D-358 的闭合窗口审计增加真实 compiled-backend 资源与关闭回滚门。证据 schema 固定为
|
||||
`qinglong/legacy-shadow-resource-rollback-evidence@v1`；edge 为 8 candidates/1 page，standalone 为 128 candidates/4 pages，8 个样本分别精确执行
|
||||
16/64 条查询，SQLite database/WAL/SHM/journal 的 logical/allocated bytes 和 file count 前后必须一致。full 模式经过独立进程重启：enabled 与 off
|
||||
都执行真实 `ScheduleService.runTask` Legacy child 且 exit 0，Shadow Run 增量必须从 1 变为 0；off 还要求零 fact factory、默认 observer/Repository
|
||||
未加载。Linux arm64 router stress 保持 128 MiB/0.5 CPU/0 swap/64 PID，cgroup peak `80,416,768` bytes，edge audit p95 `3.914 ms`；Edge release
|
||||
保持 256 MiB/1 CPU/0 swap/128 PID，cgroup peak `139,079,680` bytes，edge full/standalone audit p95 分别为 `32.056/19.265 ms`，两档
|
||||
`memory.events max/oom/oom_kill` 增量均为 0。双 Node crash harness 留在 256 MiB 发布门，完整 Edge+Standalone Workflow/Prompt crash matrix 仍通过；路由门只跑
|
||||
单进程产品/恢复 workload，不提高内存或 timeout。只读 mount 验证改为解析路径实际生效的最具体 mount：`/workspace` 可继承只读 `/`，但任一可写 descendant
|
||||
仍失败。干净镜像同时暴露历史 `build:back` 被 stale artifact 掩盖的 TypeScript common-root 错位；新构建器在隔离 staging 编译，只原子发布
|
||||
`output/back` 到既有扁平 `static/build/*`，重写 source map，拒绝携带 `output/packages`，失败保留上一版。D-359 不新增 package、生产依赖、schema、
|
||||
migration、数据库 authority、timer/watcher、端口或部署对象。完整 backend 与 18-package clean build/test 退出 0；静态契约 `126/126`，四项可执行
|
||||
架构审计与 14/14 artifact budget 全部通过，制品字节保持 D-358 基线。D-359 不把 Docker arm64 冒充物理路由/断电/flash 证据。D-360 继续负责可信的
|
||||
Legacy→Shadow capture authority、具体 exporter 与正式 Primary gate。
|
||||
|
||||
- D-358/ADR-0450（已接受）：新增显式、只读、一次性的 Legacy Shadow 闭合窗口终态审计。调用方必须提供 origin 与
|
||||
`[windowStartMs, windowEndMs)`，cohort 固定为窗口内创建的 legacy-owned Shadow Run；只有窗口经过默认五分钟 settling、候选 keyset 与
|
||||
Legacy evidence 都完整且分母非零时，才输出 terminal agreement/full comparability permille。关联只接受 direct Run/Attempt reference、同 Cron
|
||||
@@ -9129,7 +9143,7 @@ flowchart LR
|
||||
| PR-1 Run Schema | Incubating | Run/RunAttempt/RunEvent schema、nullable cancel request 与 Attempt deadline 字段及恢复索引、CancellationDispatch 状态/version/lease/backoff schema、Repository port、临时 Sequelize adapter、统一事件大小/分页上限、跨 adapter RunRepository contract suite（原子事务、回滚、Run/Attempt/RetryPolicy CAS、唯一错误、分页与取消恢复）；ADR-0041 的 `pg-0003-run-retry-policy`、capability v2、driver-neutral PostgreSQL Run Repository 与真实 `pg.Pool` 上的共享 Repository/rollback/SQLSTATE contract；ADR-0063/0069/0071/0073/0074/0076 的独立 Node 24 local-sqlite typed schema、十二条 reviewed migration、capability v6、共享 operation authority、readiness/RunRepository/API credential repository/receipt journal/dispatch plan/encrypted Secret envelope/Project Policy/security audit/authorized mutation/stable Identity catalog、Drizzle↔真实 catalog table/column/index/CHECK/FK lockstep、base/adopted/application edge/standalone 产物门禁；ADR-0064 的 legacy baseline/plan digest、Online Backup recovery、side-by-side target migration、staged manifest、双库栅栏 activation、source 生命周期写栅栏、target stable identity 和重启语义；ADR-0065 的独立 cutover authority、外部副作用停机 evidence、append-only journal、start/restart/stop barrier 与 unknown→manual_required 收敛；ADR-0066 的 adopted storage→Run reconciliation→receipt maintenance→domain recovery→lifecycle→admission application gate、严格有界 recovery summary 与 admission-first reverse stop；ADR-0067 的 SQLite 事实驱动 Run 候选源、256 条硬上限、截断失败关闭和唯一 Repository authority；ADR-0068 的 receipt-first Reconciler、callback token/sequence fence、exact local-process identity、Attempt/Run/双 Event 原子终态推进和最终 verifier；ADR-0069 的 local-process 单向包边界、pre-spawn journal、受审 POSIX launcher、immutable receipt、exact identity 和 Profile-aware cleanup lifecycle；ADR-0070 的独立 local-execution、spawn 前后双 transaction CAS、callback digest、exact stop 补偿与 fail-closed starting 保留；ADR-0071 的独立 local-dispatch、不可变 revision/context、Secret-first materializer、Profile Artifact admission、4/64 MiB output hard quota 和窄 application facade；ADR-0073/0074 的 Project-bound SecretRef、AES-256-GCM、外置 keyring 生命周期、双 SQLite authority CAS、application preflight、强 Principal/Policy 和 envelope+audit 原子提交；ADR-0086 的本机 Owner provisioning/challenge/claim/delivery acknowledgement/credential recovery CLI；ADR-0377 的 Local/Cluster 同构、Profile-aware、Project-scoped Artifact range read | fresh database/pepper setup、credential rotation/GC 运维编排与 Secret/Project/Role/Approval 管理 CLI/API/UI、备份/rekey、2.x/target process controller、人工 recovery、target 写后 reconciliation 与完整 cutover/rollback 演练；retry 产品策略、Artifact retention/tombstone stack、具体本机 lifecycle 和 target executable；Linux x64/arm64、PID namespace、断电与固定路由设备门禁；PostgreSQL 16/18 双连接并发与 failover integration；Task revision/context 跨方言 contract/并发压力与引用感知 retention、Keyv 数据迁移 |
|
||||
| PR-2 Run 状态机 | Incubating | 纯转换表、终态/时间/错误/执行器元数据规则、Run version 与 event sequence CAS、事务性 RunCommandService、回滚测试 | 重复 Worker callback/fencing、并发数据库压力测试、Primary 执行链接入 |
|
||||
| PR-3 Executor 端口 | Incubating | ADR-0003、ExecutionSpec/Context/Handle/Result、Executor port、LocalProcessExecutor、进程组取消/超时升级、流式背压、Legacy Cron spec builder、真实进程 contract tests、可复现 edge 基准入口 | 固定 edge/多架构设备基线、Legacy builder 与 makeCommand 差异审计、Primary 生产流量接入 |
|
||||
| PR-4 Shadow Run | Incubating | origin 三态策略；默认关闭的 `QL3_SHADOW_ORIGINS`；manual、scheduled_node、boot、subscription、system 与 script 现有 ChildProcess 旁路观察；system crond 显式 origin marker、Shell execution ID、finish-only 准入、确定性 Run/Attempt 与 exact replay；`@once` 保持 manual、gRPC transport 不冒充 origin 的准入裁决；每个 worker 懒加载；Run/Attempt/Event 影子生命周期；稳定且不复制 caller 原文的 task identity/revision 与有界日志引用；同 worker 有界注册表和跨 worker 持久化候选关联；stop all/stop instance、Shell callback、乱序/迟到/歧义处理；监听前一次性、Profile-aware 的 keyset Startup Reconciler，终态证据补齐、lost/abandoned/pending 分流与 terminal Attempt response-loss 修复；origin-bounded 且逐级守恒的版本化 startup difference report、固定字段 metric batch 与一次性 collector；显式、只读、闭合窗口且 Profile-bounded 的 Shadow→Legacy 终态差异审计；失败开放和契约测试 | Legacy→Shadow capture authority、具体 exporter、资源压力、回滚演练和 Primary 门禁 |
|
||||
| PR-4 Shadow Run | Incubating | origin 三态策略；默认关闭的 `QL3_SHADOW_ORIGINS`；manual、scheduled_node、boot、subscription、system 与 script 现有 ChildProcess 旁路观察；system crond 显式 origin marker、Shell execution ID、finish-only 准入、确定性 Run/Attempt 与 exact replay；`@once` 保持 manual、gRPC transport 不冒充 origin 的准入裁决；每个 worker 懒加载；Run/Attempt/Event 影子生命周期；稳定且不复制 caller 原文的 task identity/revision 与有界日志引用；同 worker 有界注册表和跨 worker 持久化候选关联；stop all/stop instance、Shell callback、乱序/迟到/歧义处理；监听前一次性、Profile-aware 的 keyset Startup Reconciler，终态证据补齐、lost/abandoned/pending 分流与 terminal Attempt response-loss 修复；origin-bounded 且逐级守恒的版本化 startup difference report、固定字段 metric batch 与一次性 collector；显式、只读、闭合窗口且 Profile-bounded 的 Shadow→Legacy 终态差异审计；128/256 MiB Linux arm64 资源门、SQLite 零增长与 Shadow enabled→off 进程重启回滚；失败开放和契约测试 | Legacy→Shadow capture authority、具体 exporter 和 Primary 门禁 |
|
||||
| PR-5 Primary LocalExecutor | Incubating（默认不激活，仅 manifest-gated manual） | runtime-owned Run 创建器；持久化先于 spawn；Run/Attempt 完整成功、失败、取消、超时与 lost 闭环；Executor handle 身份校验；spawn 后激活写失败的 stop+lost 补偿；completion rejection 安全收敛；独立 Primary 幂等查询与唯一索引竞态裁决；durable `run.cancel_requested`、stop-before-signal、首次请求幂等、晚到完成裁决与待取消有界恢复查询；最多 64 条一页的 cross-worker cancellation source；独立 CancellationDispatch Repository 原子 claim/result、lease expiry 接管、owner/token/version fencing、指数退避与结果 RunEvent；最多 64 页的单周期 cancel supervisor；显式 start/stop、无重叠、错误隔离、停止等待有上限且 timer unref 的 lifecycle runner；Linux durable handle 的 PID/boot/start ticks/process-group 复验与 TERM/KILL controller；完整有界分页且 fail-closed 的 startup Reconcile supervisor；RunningInstance nullable `run_id/attempt_id` 关联；Primary 专用组合 Repository 在同一 SQLite 事务提交前投影 Crontab/RunningInstance，失败整体回滚；有界且防穿越的 legacy log output ref；manual owner seam、真实本机装配、单 spawn/fail-closed；严格 manual-only rollout manifest loader、短期审批/gate、配置哈希审计；HTTP worker 已接轻量 lazy bootstrap，accepted 后按 receipt-first reconcile→completion receipt lifecycle→timeout intent lifecycle→cancel dispatch lifecycle→router 顺序激活，失败撤销，监听失败和 shutdown 有界停止；Primary timeout 在 spawn 前持久化绝对 deadline，有界 source/requester/supervisor 只提交 timeout 意图并复用 CancellationDispatch；代码级 edge/standalone Profile 为各 lifecycle 提供不同 cadence 与页上限，cluster-control/worker 拒绝误装本机 SQLite Primary；统一 CompletionService 原子提交 Attempt/Run/双 Event，spawn 前保存 callback token hash、终态推进 sequence，实时回调与 receipt consumer 共享入口并覆盖两个清理 crash window；manual Primary 已接入受限 POSIX launcher、`0600` direct-file stdout/stderr、父进程退出后续写、不可覆盖 receipt 生产、回执环境清除、TERM 转发等待及 live transaction 后清理；Startup Reconciler receipt-first 双检查并在确定 exited 后执行 profile 化的单次 50/100 ms publish grace；`0007` 独立 CompletionReceiptJournal 在 spawn 前登记、为升级前 active Attempt 补登记并驱动周期扫描，使终态残留继续可发现；确定无效的已知 Attempt receipt 先持久化隔离状态，再进入确定性私有分片 quarantine；终态 missing 与 quarantine 按 edge/standalone retention 有界清理；非 Journal 文件具备只读优先、固定分片/条目上限、overflow fail-closed、显式同盘隔离的 Node 24 运维 CLI；扫描具备页上限、resume cursor、timer unref、无重叠、有界 stop 和低敏计数；ENOSPC 与 launcher receipt 存储失败有代码门禁 | 部署配置写入/审批入口与用户可见状态；PostgreSQL CancellationDispatch adapter；cluster-control 生产启动拓扑；固定 edge/Linux 多架构与真实磁盘压力基线、完整 2.x API 契约和回滚演练 |
|
||||
| PR-7 Worker Session、Run Lease 与启动协议基础 | Incubating（默认关闭，独立入口显式 opt-in） | ADR-0012/0013/0014/0021/0057–0061/0108–0121/0231–0239/0377；有界 capability/Placement/Dispatcher；SQLite 协议孵化与 PostgreSQL v9 Session/Run Lease/credential/attestation authority；immutable revision Placement、数据库时钟 keyset candidate、认证 Worker Pull、digest-only offer recovery；versioned capability-free ExecutionSpec response、stable claim 跨重启退避、单 owner 原子 inbox 准入与 TLS 1.3 mTLS/`ql3w` HTTPS client；同一 package journal 上 revision-fenced starting/spawn/started/running/completion 状态、callback digest、tagged no-spawn 与 ambiguous recovery；PostgreSQL starting/running/start-failure/completion 数据库权威事务、精确重放与 cancellation/timeout 优先终态；batch Secret delivery 在 Attempt advisory lock 下复验 Session/Lease/revision 完整围栏并复用单 Agent，Secret-before-Artifact materializer 将同一 log ID 交给 Executor/journal/running ACK；offer-scoped `wlog-*` 私有文件 spool、Edge/Node 容量策略、append/quota/path 防护、barrier 后 output ownership、受审 POSIX Executor、truncation fact、固定内存流式 source、认证 Artifact stream、共享 immutable store port、S3-compatible SSE/checksum/条件 promotion adapter、upload-before-completion 协调，以及 Local/Cluster 同构、Profile-aware、ETag-fenced range read；用户取消 run.stop mutation 以数据库时间写 intent/Event 并在事务内复验 Project/RoleBinding fence；非执行取消 convergence lifecycle、运行期 expiry 与安全 lost retry 已接入 cluster-control 单一全局 cadence；完整 generation/version/token/Attempt fencing；独立最小权限 Worker ingress、CA/CRL 与连接 generation 热重载；offer journal、spawn barrier、receipt-first recovery；独立 `@qinglong/worker-runtime` 的本地 P-256 CSR、key/chain/trust 验证、generation + active pointer 安装和持久退避；默认关闭的 production process 已装配具体 execution graph、完整 Session heartbeat/drain/offline、direct-file bootstrap、单 Agent/单 cadence、startup reconciliation、证书 maintenance、transport fail-close/recovery 与 Edge/Node 有界预算；真实 PostgreSQL 18 + Linux Node 合约已覆盖 Run completion、credential 和 CA 双轮换且保持同一 Session；真实 K3s 合约已覆盖 TLS/credential Secret 分权、双对象 CAS、Recreate 顺序、identity generation 与单节点 PVC recovery；所有能力默认不可达且受 edge/cluster import audit 约束 | 具体 cert-manager/Vault/SPIFFE/离线 CA adapter 与模板、ingress reload controller、生产 RBAC、证书到期告警和 `ql3w` credential recovery 产品面；具体 KMS/Vault Secret provider、对象存储 credential/temporary lifecycle 与 retention/tombstone；Worker 管理 API；真实 Kubernetes 多节点 CSI/node-loss/production 360 秒 drain 与固定 edge 文件系统 suspend/时钟/断电、x64/arm64 资源门禁 |
|
||||
| PR-8 Project/Policy/Approval Core | Incubating（默认拒绝、无生产业务执行入口） | ADR-0028；统一六类 ActorRef 与 exact-shape 校验；`0017` ownerless default Project 和 append-only versioned RoleBinding；owner/admin/operator/viewer 固定矩阵；Project 内 mutation 幂等、expected-version CAS、双 SQLite 连接竞争门禁；archived read-only、revocation、存储损坏 fail-closed；Agent 写/Secret/Tool `require_approval`；ADR-0047 把六类 subject、role/permission matrix 与 fence 抽到 runtime-core，`pg-0004-project-policy`/capability v3 建立 ownerless PostgreSQL baseline、严格 role/state CHECK、append-only runtime 权限、SERIALIZABLE Project lock、mutation replay、双连接单 winner 和 cluster admission authorizer；ADR-0049/`pg-0005` capability v4 建立 stable IdentitySubject、append-only digest-only API credential、真实 cluster bearer authenticator、write-only durable security audit 与最小权限 runtime role，且已验证 HTTP→credential→Policy→audit→handler 纵向链路；ADR-0051 建立 `/api/v3` 认证前 peer/global 双预算、transport-peer-only、无 timer 且有界内存的 overload shield；ADR-0027 Artifact authorizer adapter；ADR-0029 `AuthenticatedPrincipal` contract、`0018` digest-only versioned challenge、CSPRNG/TTL、同事务消费 challenge + 写首 owner、精确重放与双连接竞争/崩溃回滚门禁；ADR-0030 `0019` stable identity/binding、legacy HS384 + current-session membership、logout/platform/revoke/disable、single-factor 与损坏 fail-closed 门禁；ADR-0031 `0020` digest-bound ApprovalRequest、User-only decision、Project/Role version fence、精确 expiry/重放/并发裁决及同事务 immutable dispatch；ADR-0032 `0021` execution backfill、三表原子 consume、稳定 due keyset、claim/renew/start/result fencing、pre-start takeover/post-start recovery-required、attempt budget、handler inspect/digest barrier 和 bounded dispatcher；ADR-0033/`0022` control/resolution backfill、start/renew/completion 原子联动、稳定 recovery keyset、双 resolver claim/takeover、finding/result 精确重放、自动/人工终结、迟到 completion 单 winner 和 evidence-only bounded reconciler；ADR-0034/`0023` 首个 `run.create` canonical plan、Run/Attempt/Event/receipt 同事务、幂等 collision fail-closed、renew/终态 fence、真实 SQLite handler 与 automatic evidence provider；ADR-0035/`0024` 独立 `approval.recover` 矩阵、稳定 User + 五分钟强认证、Project/RoleBinding fence、human resolution + authorization fact 原子提交、撤权竞态与回滚门禁；ADR-0036 recovery-first 单 timer lifecycle、edge/standalone 独立 cadence/页预算、跨周期 cursor、非重叠与有界 stop；ADR-0074 以新的 Node 24 SQLite v5 ownerless Project/RoleBinding/audit authority 和独立 local-secret-admin 提供强 Principal、`secret.manage`、撤权 fence、envelope+allowed audit 原子提交及不回显语义；ADR-0086 以可信 POSIX console 和 staged delivery 完成本机首 Owner 产品 ceremony | fresh database/pepper setup 与安全迁移向导；`shareStore`/Express 到 authentication core 的 production migration；credential rotation/revocation API、mTLS/Worker enrollment、恢复码；Project/Role/Approval/Secret 管理 CLI/API/UI、audit retention/query/export/alert、preview Artifact/digest/immutable plan builder、真实 MFA/hardware adapter、人工 recovery API/UI/独立 rate limit 与审计事件、handler/provider registry、lifecycle startup/shutdown/指标/admission gate；PostgreSQL action/receipt/provider/recovery-authorization 与 OPA adapter、缓存 version 失效；Tool/Package/Secret/Shell 各自的 handler/evidence contract；Secret/Run/Tool/Workflow waiting_approval 全入口装配；完整回滚演练 |
|
||||
|
||||
@@ -239,7 +239,7 @@ Shadow Adapter 不得：
|
||||
|
||||
Shadow 转换仍必须遵守 ADR-0001。无法合法映射时追加 compat.transition_mismatch，不能强行覆盖终态。
|
||||
|
||||
当前 Alpha 切片对已审 Node worker origin 直接观察同一 ChildProcess 的 spawn、error 和 exit 事件，因此不依赖 Shell callback 才能形成基本终态。下述两级关联已补充 Shell callback、stop/cancel 和乱序/迟到回调；ADR-0448 又补充了监听前一次性启动恢复，ADR-0449 将其投影为 origin-bounded、版本化的差异报告与固定字段 metric batch。ADR-0450 再提供显式、只读、Profile-bounded 的闭合窗口终态审计；但 2.x RunningInstance 缺少可信 origin，因此它只证明已写 Shadow Run 到 Legacy evidence 的一致性，不能单独证明 Legacy→Shadow 捕获率或替代正式 Primary gate。
|
||||
当前 Alpha 切片对已审 Node worker origin 直接观察同一 ChildProcess 的 spawn、error 和 exit 事件，因此不依赖 Shell callback 才能形成基本终态。下述两级关联已补充 Shell callback、stop/cancel 和乱序/迟到回调；ADR-0448 又补充了监听前一次性启动恢复，ADR-0449 将其投影为 origin-bounded、版本化的差异报告与固定字段 metric batch。ADR-0450 再提供显式、只读、Profile-bounded 的闭合窗口终态审计；ADR-0451 已在 128 MiB router stress 与 256 MiB Edge release cgroup 中证明有界查询、SQLite 零增长和进程重启后的 Shadow-off 回滚。但 2.x RunningInstance 缺少可信 origin，因此这些证据只覆盖已写 Shadow Run 到 Legacy evidence 的一致性，不能单独证明 Legacy→Shadow 捕获率或替代正式 Primary gate。
|
||||
|
||||
### 9.4 `next` Alpha callback 与 stop 关联
|
||||
|
||||
@@ -253,7 +253,7 @@ Shadow 转换仍必须遵守 ADR-0001。无法合法映射时追加 compat.trans
|
||||
6. 取消事实在 Legacy kill 前投递；同 worker 的后续 exit 排在取消之后。跨 worker 使用持久化定位器尽力关联，任何查询或写入失败都不能阻断 kill 或改变 2.x API 响应。
|
||||
7. 乱序 finished 可以从 queued/claimed 补齐 dispatching、starting、running 和终态；重复终态 callback、取消后的迟到成功 callback 不覆盖终态，也不追加重复完成事件。
|
||||
|
||||
这仍不是完整的 Shadow→Primary 门禁：ADR-0448 已提供启动后有界批量扫描、终态证据补齐、lost/abandoned 收敛和两事务 response-loss 修复；ADR-0449 已增加 startup 差异报表、固定低基数 metric batch 与可注入单次 collector；ADR-0450 已完成不伪造反向捕获率的闭合窗口 Shadow→Legacy 终态差异查询。具体 exporter、Legacy→Shadow capture authority、资源/回滚演练和正式 Primary gate 尚未完成。后续能力不得让 edge 增加常驻 watcher 或无界内存队列。
|
||||
这仍不是完整的 Shadow→Primary 门禁：ADR-0448 已提供启动后有界批量扫描、终态证据补齐、lost/abandoned 收敛和两事务 response-loss 修复；ADR-0449 已增加 startup 差异报表、固定低基数 metric batch 与可注入单次 collector；ADR-0450 已完成不伪造反向捕获率的闭合窗口 Shadow→Legacy 终态差异查询；ADR-0451 已完成 Profile-aware 资源压力和 Shadow-off 进程重启回滚。具体 exporter、Legacy→Shadow capture authority 和正式 Primary gate 尚未完成。后续能力不得让 edge 增加常驻 watcher 或无界内存队列。
|
||||
|
||||
### 9.3 Shadow 写失败
|
||||
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
# ADR-0451：按 Profile 有界的 Legacy Shadow 资源与关闭回滚证据
|
||||
|
||||
- 状态：Accepted
|
||||
- 日期：2026-08-19
|
||||
- 关联 RFC：QL-RFC-0001 D-02、D-359、PR-4
|
||||
- 关联 ADR：ADR-0002、ADR-0088、ADR-0450
|
||||
- Amends：ADR-0450 的资源压力与 Shadow-off 回滚待办
|
||||
|
||||
## 上下文
|
||||
|
||||
ADR-0450 已冻结闭合窗口的 Shadow→Legacy 终态审计语义，但只证明查询结果，不证明它能在 128 MiB 路由档运行，也没有证明进程重启后把
|
||||
`QL3_SHADOW_ORIGINS` 从 enabled 改为 off 会立即停止新 Shadow 写入而继续执行 Legacy。单元测试或主机 RSS 不能替代受限 Linux cgroup；反过来，
|
||||
在 128 MiB 门中同时驻留 Node 测试父进程和预期被 SIGKILL 的子进程，也不是路由设备的单进程产品模型。
|
||||
|
||||
资源门此前还要求 `/workspace` 必须是独立只读 mount。只读容器根已经让该路径继承只读语义；额外复制约 1.5 GiB 工作区到匿名 volume 会把私有页缓存
|
||||
计入 128 MiB cgroup，制造与产品 RSS 无关的 `memory.events max`。门禁必须验证路径实际生效的最具体 mount，并拒绝 `/workspace/**` 的任何可写覆盖，
|
||||
而不是要求一种特定 mount 拓扑。
|
||||
|
||||
## 决策
|
||||
|
||||
1. 新增一次性证据入口 `benchmark:legacy-shadow`，报告 schema 固定为
|
||||
`qinglong/legacy-shadow-resource-rollback-evidence@v1`。报告只包含 Profile、预算、固定资源计数、回滚布尔结论和限制，不输出数据库路径、命令、Run、Attempt、
|
||||
Cron、PID、log 或 task identity。
|
||||
2. 审计 fixture 使用真实 Sequelize + SQLite、真实 migration、真实 Legacy Shadow writer 和 ADR-0450 auditor。edge 固定 8 个 candidate、`8 × 1 page`；
|
||||
standalone 固定 128 个 candidate、`32 × 4 pages`。每页必须精确执行一条 candidate 查询和一条 evidence 查询；8 个样本分别固定为 16 和 64 条查询。
|
||||
3. 审计连接为单连接、SQLite read-only。运行前后同时比较 database、WAL、SHM 与 journal 的 logical bytes、allocated bytes 和 file count；任一变化均失败。
|
||||
4. `full` 模式必须启动三个独立进程：audit、Shadow-enabled Legacy execution、重启后的 Shadow-off Legacy execution。enabled 进程必须执行真实
|
||||
`ScheduleService.runTask` 子进程、Legacy exit 0 且恰好新增一个 terminal Shadow Run；off 进程仍须 Legacy exit 0，但 configured origins 为空、fact factory
|
||||
调用为零、默认 Shadow observer/Repository 未加载且 Run 增量为零。
|
||||
5. Legacy `TaskLimit` 构造器的异步数据库初始化必须在调度和 teardown 前通过同一单例显式等待；证据进程不得用 sleep、吞掉 unhandled rejection 或提前关闭
|
||||
Sequelize 来伪造成功。
|
||||
6. `router-stress-ci` 保持 128 MiB、0 swap、0.5 CPU、64 PID、非 root、只读根、`NoNewPrivs` 和 seccomp，只运行 edge audit-only。该档位保留真实 Edge
|
||||
executor、产品 Workflow、SQLite lock 和失败升级恢复，但不运行需要同时驻留测试父/子 Node 的 crash matrix。
|
||||
7. `edge-release-ci` 保持 256 MiB、0 swap、1 CPU、128 PID，并运行 edge full rollback、standalone audit-only，以及 Edge+Standalone 的完整 Workflow/Prompt
|
||||
crash matrix。把双进程 crash harness 放到发布门不减少覆盖，也不把测试框架开销冒充路由产品最低内存。
|
||||
8. mount 验证按目标路径选择最具体 mount。`/workspace` 可继承只读 `/`，但其任一可写 descendant mount 都使门禁失败；`/tmp` 仍必须是独立、可写且有容量上限的
|
||||
tmpfs。门禁不得为满足断言而复制整个工作区。
|
||||
9. 资源报告继续对 `memory.events max/oom/oom_kill/oom_group_kill` 的任一增量失败，不以“没有 OOM”接受内存触顶；也不放宽既有 latency、RSS、写放大或
|
||||
SQLite integrity contract。
|
||||
10. 本证据只证明 CI 容器内的有界行为，不证明物理路由、flash wear、断电存活、生产任务内容、Legacy→Shadow capture rate、Primary eligibility、
|
||||
PostgreSQL 或 Cluster runtime。
|
||||
|
||||
## 被拒绝的替代方案
|
||||
|
||||
### 把路由内存上限提高到 192/256 MiB
|
||||
|
||||
拒绝。失败来自测试拓扑和 workspace volume 页缓存，不是产品 Edge workload 超出预算；扩大内存会掩盖错误建模。
|
||||
|
||||
### 增大 crash child 的 30 秒 timeout
|
||||
|
||||
拒绝。128 MiB 下父/子双 Node 的回收抖动不是业务时延。完整 crash matrix 已在 256 MiB 发布门保留，路由门不应靠等待更久制造假稳定。
|
||||
|
||||
### Shadow-off 只调用配置解析函数
|
||||
|
||||
拒绝。回滚证据必须经过进程重启、真实 Legacy child 和数据库计数，证明 off 路径在执行发生时不构造 fact、不加载默认写入组件。
|
||||
|
||||
## 验证
|
||||
|
||||
- 主机聚焦门 `23/23`；edge full 与 standalone audit-only 都使用 compiled backend 并通过数据库完整性检查。
|
||||
- Linux arm64 router gate：128 MiB/0.5 CPU/0 swap/64 PID，cgroup peak `80,416,768` bytes；所有 `memory.events` 增量为 0。edge audit 8 个样本、16/16 查询、
|
||||
p95 `3.914 ms`、RSS 增量 `131,072` bytes，SQLite storage 前后相同。
|
||||
- Linux arm64 edge release gate：256 MiB/1 CPU/0 swap/128 PID，cgroup peak `139,079,680` bytes；所有 `memory.events` 增量为 0。edge full audit p95
|
||||
`32.056 ms`、16/16 查询、RSS 增量 `3,407,872` bytes；enabled/off Legacy 均 exit 0，Run 增量分别为 1/0，进程峰值 RSS 分别为
|
||||
`94,224,384 / 92,577,792` bytes。standalone audit 为 32 pages、64/64 查询、p95 `19.265 ms`、RSS 增量 `1,572,864` bytes，SQLite storage
|
||||
前后相同。
|
||||
- 同一发布门完整通过 Workflow admission/control、Prompt model/outer transaction 等 Edge+Standalone crash matrix；Prompt live execution 与 exact replay
|
||||
固定发生 2 次 key resolution、1 次 key load，门禁拒绝计数漂移。
|
||||
- 完整 `test:back` 与 18-package clean build/test 均退出 0；五个 package/deployment/image/vulnerability 静态契约文件为 `126/126`，Edge import、
|
||||
Cluster dependency、package boundary 与 Cluster deployment 四个可执行审计均为 compatible 且零 finding。14 个 Local Profile artifact 全部通过预算；
|
||||
Edge/Standalone 字节依次为：base `2,589,998 / 2,590,076`、adopted `2,809,293 / 2,809,416`、application
|
||||
`3,632,877 / 3,632,997`、application-api `3,800,430 / 3,800,574`、AI `3,069,251 / 3,069,341`、application+AI
|
||||
`4,493,151 / 4,493,283`、MCP `7,315,930 / 7,316,038`。
|
||||
|
||||
## 后续
|
||||
|
||||
D-360 仍须建立可信的 origin-scoped Legacy→Shadow capture authority，组合 observer failure/capture evidence 与本 ADR/ADR-0450 的结果，再定义正式 Primary gate。
|
||||
固定物理 edge、真实 flash 写放大和断电演练必须作为独立现场证据，不能由本 ADR 的 Docker arm64 结果替代。
|
||||
@@ -0,0 +1,51 @@
|
||||
# ADR-0452：原子且扁平兼容的 Backend 构建发布
|
||||
|
||||
- 状态：Accepted
|
||||
- 日期：2026-08-19
|
||||
- 关联 RFC：QL-RFC-0001 D-359
|
||||
- 关联 ADR：ADR-0042、ADR-0106、ADR-0451
|
||||
|
||||
## 上下文
|
||||
|
||||
Backend TypeScript 开始引用 QL3 workspace source 后，TypeScript 的 common source root 扩展到仓库根。一次真正清空输出目录的构建会生成
|
||||
`static/build/back/app.js` 与 `static/build/packages/**`，但 2.x 生产入口和现有镜像仍要求 `static/build/app.js`、`token.js`、`runtime/**` 等稳定路径。
|
||||
开发机长期残留的旧 `static/build/*` 会掩盖该错位，使增量构建和测试通过，而干净镜像不可启动。
|
||||
|
||||
直接把所有生产入口改到 `static/build/back/**` 会扩大 2.x 部署、CLI、Docker 与 Shell 兼容面；把 workspace package 编译副本发布进 backend 目录又会形成第二套
|
||||
package authority。构建必须只发布 backend subtree，同时保持既有路径和可追踪 source map。
|
||||
|
||||
## 决策
|
||||
|
||||
1. `build:back` 唯一进入 `scripts/ql3-build-back.cjs`，不再直接把 `tsc` 输出到公开的 `static/build`。
|
||||
2. 每次构建在 `static/.back-build-<pid>-<nonce>/output` 隔离目录执行 `tsc -p back/tsconfig.json --outDir ...`。失败时保留当前公开构建，不允许半成品覆盖。
|
||||
3. 发布前必须验证 staged `back/app.js`、`back/token.js`、`back/runtime/adapters/local-process/localProcessExecutor.js` 等必要输出存在；
|
||||
`output/packages/**` 只作为编译闭包，不进入公开 backend。
|
||||
4. 只把 staged `output/back` 发布为扁平的 `static/build`，因此生产路径继续是 `static/build/app.js`、`runtime/**`，不要求修改 2.x 入口。
|
||||
5. 发布前重写 `.js.map` 的 relative sources，使其从最终扁平位置仍能解析到真实 `back/**` source；不得删除 map 或留下指向临时 staging 的路径。
|
||||
6. 发布采用同一父目录内 rename：现有 build 先改名为唯一 backup，staged backend 再 rename 到目标；只有新目录成功可见后才删除 backup。发布失败时恢复旧目录并清理精确
|
||||
staging/backup，不使用 shell、glob 或仓库级删除。
|
||||
7. 构建结束后 `static/build/back` 与 `static/build/packages` 必须不存在。测试和镜像门必须从干净输出执行，禁止依赖历史 artifact。
|
||||
|
||||
## 被拒绝的替代方案
|
||||
|
||||
### 修改所有生产入口指向 `static/build/back`
|
||||
|
||||
拒绝。它把 TypeScript 内部 common-root 变化泄露成部署契约变化，并扩大 2.x 回滚风险。
|
||||
|
||||
### 构建前清空 `static/build` 后直接运行 tsc
|
||||
|
||||
拒绝。它仍发布错误目录，并在编译失败时破坏最后一个可运行构建。
|
||||
|
||||
### 同时发布 `output/back` 与 `output/packages`
|
||||
|
||||
拒绝。QL3 package 已有独立 dist/artifact gate，backend 不应携带另一份 workspace package 实现。
|
||||
|
||||
## 验证
|
||||
|
||||
- 主机和干净 Linux arm64 镜像中的 `pnpm build:back` 均通过；最终目录只保留原有 root layout，不存在 `static/build/back` 或
|
||||
`static/build/packages`。
|
||||
- source map 从最终 `static/build/runtime/**` 位置可解析回仓库 `back/runtime/**`，不引用临时 staging。
|
||||
- Edge benchmark、D359 compiled-backend 资源/回滚门和完整 backend 回归均只读取新的干净发布结果。
|
||||
- 完整 backend 回归与 18-package clean build/test 均退出 0；五个静态契约文件 `126/126`，四个可执行架构审计和 14 个 Local Profile artifact
|
||||
预算全部通过，证明扁平 backend 发布没有改变 package closure 或 Profile 制品基线。
|
||||
- 本 ADR 不改变 package graph、生产依赖、schema、migration、数据库或部署 authority。
|
||||
@@ -454,6 +454,8 @@
|
||||
| [ADR-0448](./ADR-0448-bounded-legacy-shadow-startup-reconciliation.md) | 有界 Legacy Shadow 启动恢复 | Accepted |
|
||||
| [ADR-0449](./ADR-0449-versioned-legacy-shadow-startup-difference-report-and-metrics.md) | 版本化 Legacy Shadow 启动差异报告与指标批次 | Accepted |
|
||||
| [ADR-0450](./ADR-0450-closed-window-legacy-shadow-terminal-difference-audit.md) | 闭合窗口的 Legacy Shadow 终态差异审计 | Accepted |
|
||||
| [ADR-0451](./ADR-0451-profile-bounded-legacy-shadow-resource-and-off-rollback-evidence.md) | 按 Profile 有界的 Legacy Shadow 资源与关闭回滚证据 | Accepted |
|
||||
| [ADR-0452](./ADR-0452-atomic-flattened-backend-build-publication.md) | 原子且扁平兼容的 Backend 构建发布 | Accepted |
|
||||
|
||||
## 规则
|
||||
|
||||
|
||||
+2
-1
@@ -17,7 +17,7 @@
|
||||
"start:back": "nodemon ./back/app.ts",
|
||||
"start:front": "max dev",
|
||||
"build:front": "max build",
|
||||
"build:back": "tsc -p back/tsconfig.json",
|
||||
"build:back": "node scripts/ql3-build-back.cjs",
|
||||
"build:packages:ql3": "node scripts/ql3-clean-package-artifacts.cjs && pnpm -r --workspace-concurrency=1 --filter './packages/ql3-*' run build",
|
||||
"test:back": "node --test test/back/*.test.cjs",
|
||||
"test:packages:ql3": "pnpm run build:packages:ql3 && pnpm -r --workspace-concurrency=1 --filter './packages/ql3-*' exec sh -c 'node --test test/*.test.cjs'",
|
||||
@@ -54,6 +54,7 @@
|
||||
"test:plugin-package-recovery-e2e:ql3": "pnpm --filter @qinglong/cluster-admin check && pnpm --filter @qinglong/cluster-control check && node scripts/ql3-plugin-package-recovery-e2e-live-contract.cjs",
|
||||
"audit:plugin-package-recovery-e2e:ql3": "node scripts/ql3-plugin-package-recovery-e2e-live-audit.cjs",
|
||||
"benchmark:edge": "node scripts/ql3-edge-benchmark.cjs",
|
||||
"benchmark:legacy-shadow": "pnpm build:back && node scripts/ql3-legacy-shadow-resource-rollback.cjs --require-compiled",
|
||||
"benchmark:cluster-control": "node scripts/ql3-cluster-control-benchmark.cjs",
|
||||
"benchmark:worker": "node scripts/ql3-worker-resource-benchmark.cjs",
|
||||
"benchmark:ai-profile": "node scripts/ql3-ai-profile-benchmark.cjs",
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const { spawnSync } = require('node:child_process');
|
||||
|
||||
const ROOT = path.resolve(__dirname, '..');
|
||||
const STATIC_ROOT = path.join(ROOT, 'static');
|
||||
const TARGET = path.join(STATIC_ROOT, 'build');
|
||||
|
||||
class QingLong3BackendBuildError extends Error {
|
||||
constructor(message) {
|
||||
super(`QingLong backend build failed: ${message}`);
|
||||
this.name = 'QingLong3BackendBuildError';
|
||||
}
|
||||
}
|
||||
|
||||
function assertWithinStatic(target, label) {
|
||||
const relative = path.relative(STATIC_ROOT, target);
|
||||
if (!relative || relative.startsWith('..') || path.isAbsolute(relative)) {
|
||||
throw new QingLong3BackendBuildError(`${label} escaped static root`);
|
||||
}
|
||||
}
|
||||
|
||||
function listFiles(directory) {
|
||||
const files = [];
|
||||
const pending = [directory];
|
||||
while (pending.length > 0) {
|
||||
const current = pending.pop();
|
||||
for (const entry of fs.readdirSync(current, { withFileTypes: true })) {
|
||||
const entryPath = path.join(current, entry.name);
|
||||
if (entry.isDirectory()) pending.push(entryPath);
|
||||
else if (entry.isFile()) files.push(entryPath);
|
||||
else {
|
||||
throw new QingLong3BackendBuildError(
|
||||
'staged output contains a non-regular entry',
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
function sourceMapPlan(compiledBackend) {
|
||||
return listFiles(compiledBackend)
|
||||
.filter((filePath) => filePath.endsWith('.js.map'))
|
||||
.map((sourcePath) => {
|
||||
const relative = path.relative(compiledBackend, sourcePath);
|
||||
const destinationPath = path.join(TARGET, relative);
|
||||
const sourceMap = JSON.parse(fs.readFileSync(sourcePath, 'utf8'));
|
||||
if (!Array.isArray(sourceMap.sources)) {
|
||||
throw new QingLong3BackendBuildError(
|
||||
`source map ${relative} has no sources array`,
|
||||
);
|
||||
}
|
||||
return Object.freeze({
|
||||
sourcePath,
|
||||
destinationPath,
|
||||
sourceMap,
|
||||
absoluteSources: sourceMap.sources.map((source) => {
|
||||
if (typeof source !== 'string' || path.isAbsolute(source)) {
|
||||
throw new QingLong3BackendBuildError(
|
||||
`source map ${relative} contains an invalid source`,
|
||||
);
|
||||
}
|
||||
return path.resolve(path.dirname(sourcePath), source);
|
||||
}),
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function rewriteSourceMaps(plans) {
|
||||
for (const plan of plans) {
|
||||
plan.sourceMap.sources = plan.absoluteSources.map((absoluteSource) =>
|
||||
path
|
||||
.relative(path.dirname(plan.destinationPath), absoluteSource)
|
||||
.split(path.sep)
|
||||
.join('/'),
|
||||
);
|
||||
fs.writeFileSync(plan.sourcePath, JSON.stringify(plan.sourceMap));
|
||||
}
|
||||
}
|
||||
|
||||
function publishCompiledBackend(compiledBackend, stageRoot) {
|
||||
for (const required of [
|
||||
'app.js',
|
||||
'token.js',
|
||||
'runtime/adapters/local-process/localProcessExecutor.js',
|
||||
]) {
|
||||
if (!fs.statSync(path.join(compiledBackend, required)).isFile()) {
|
||||
throw new QingLong3BackendBuildError(
|
||||
`compiled backend is missing ${required}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
const maps = sourceMapPlan(compiledBackend);
|
||||
rewriteSourceMaps(maps);
|
||||
const backup = path.join(
|
||||
STATIC_ROOT,
|
||||
`.build-backup-${process.pid}-${Date.now()}`,
|
||||
);
|
||||
assertWithinStatic(TARGET, 'target');
|
||||
assertWithinStatic(backup, 'backup');
|
||||
let previousMoved = false;
|
||||
try {
|
||||
if (fs.existsSync(TARGET)) {
|
||||
fs.renameSync(TARGET, backup);
|
||||
previousMoved = true;
|
||||
}
|
||||
fs.renameSync(compiledBackend, TARGET);
|
||||
} catch (error) {
|
||||
if (!fs.existsSync(TARGET) && previousMoved && fs.existsSync(backup)) {
|
||||
fs.renameSync(backup, TARGET);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
if (previousMoved) fs.rmSync(backup, { recursive: true, force: true });
|
||||
fs.rmSync(stageRoot, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
function main() {
|
||||
fs.mkdirSync(STATIC_ROOT, { recursive: true });
|
||||
const stageRoot = fs.mkdtempSync(path.join(STATIC_ROOT, '.back-build-'));
|
||||
assertWithinStatic(stageRoot, 'stage');
|
||||
const output = path.join(stageRoot, 'output');
|
||||
try {
|
||||
const tsc = require.resolve('typescript/bin/tsc');
|
||||
const result = spawnSync(
|
||||
process.execPath,
|
||||
[tsc, '-p', 'back/tsconfig.json', '--outDir', output],
|
||||
{
|
||||
cwd: ROOT,
|
||||
encoding: 'utf8',
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
},
|
||||
);
|
||||
if (result.stdout) process.stdout.write(result.stdout);
|
||||
if (result.stderr) process.stderr.write(result.stderr);
|
||||
if (result.error) throw result.error;
|
||||
if (result.status !== 0) {
|
||||
throw new QingLong3BackendBuildError(
|
||||
`TypeScript compiler exited with ${result.status}`,
|
||||
);
|
||||
}
|
||||
const compiledBackend = path.join(output, 'back');
|
||||
if (!fs.existsSync(compiledBackend)) {
|
||||
throw new QingLong3BackendBuildError(
|
||||
'TypeScript output did not contain the backend subtree',
|
||||
);
|
||||
}
|
||||
publishCompiledBackend(compiledBackend, stageRoot);
|
||||
} catch (error) {
|
||||
fs.rmSync(stageRoot, { recursive: true, force: true });
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
QingLong3BackendBuildError,
|
||||
assertWithinStatic,
|
||||
listFiles,
|
||||
publishCompiledBackend,
|
||||
rewriteSourceMaps,
|
||||
sourceMapPlan,
|
||||
};
|
||||
|
||||
if (require.main === module) {
|
||||
try {
|
||||
main();
|
||||
} catch (error) {
|
||||
process.stderr.write(
|
||||
`${error instanceof Error ? error.message : String(error)}\n`,
|
||||
);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,776 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const { spawnSync } = require('node:child_process');
|
||||
const { performance } = require('node:perf_hooks');
|
||||
const sqlite3 = require('sqlite3');
|
||||
const { DataTypes, QueryTypes, Sequelize } = require('sequelize');
|
||||
const REPOSITORY_ROOT = path.resolve(__dirname, '..');
|
||||
process.env.QL_DIR ??= REPOSITORY_ROOT;
|
||||
const BUILD_ROOT = path.join(REPOSITORY_ROOT, 'static/build');
|
||||
const BUILD_AVAILABLE = fs.existsSync(
|
||||
path.join(
|
||||
BUILD_ROOT,
|
||||
'runtime/application/legacyShadowTerminalDifferenceAuditor.js',
|
||||
),
|
||||
);
|
||||
if (!BUILD_AVAILABLE) require('ts-node/register/transpile-only');
|
||||
const fromRuntime = (relativePath) =>
|
||||
require(path.join(
|
||||
BUILD_AVAILABLE ? BUILD_ROOT : path.join(REPOSITORY_ROOT, 'back'),
|
||||
relativePath,
|
||||
));
|
||||
|
||||
const MIB = 1024 * 1024;
|
||||
const BASE_TIME_MS = 1_750_200_000_000;
|
||||
const MAX_CHILD_OUTPUT_BYTES = 64 * 1024;
|
||||
const PROFILES = Object.freeze({
|
||||
edge: Object.freeze({ candidates: 8, pageSize: 8, maxPages: 1 }),
|
||||
standalone: Object.freeze({ candidates: 128, pageSize: 32, maxPages: 4 }),
|
||||
});
|
||||
const MEASURES = Object.freeze([
|
||||
'real_sequelize_sqlite_shadow_terminal_audit',
|
||||
'profile_maximum_closed_window_candidate_count',
|
||||
'bounded_candidate_and_evidence_query_count',
|
||||
'read_only_database_storage_stability',
|
||||
'process_restart_shadow_enabled_to_off',
|
||||
'real_legacy_child_execution_before_and_after_restart',
|
||||
'off_path_zero_fact_factory_and_shadow_write',
|
||||
]);
|
||||
const EXCLUSIONS = Object.freeze([
|
||||
'legacy_to_shadow_capture_rate',
|
||||
'physical_router_or_flash_wear',
|
||||
'power_loss_survival',
|
||||
'production_task_content_or_identity',
|
||||
'primary_execution_eligibility',
|
||||
'cluster_or_postgresql_runtime',
|
||||
]);
|
||||
|
||||
class QingLong3LegacyShadowResourceRollbackError extends Error {
|
||||
constructor(message) {
|
||||
super(
|
||||
`QingLong 3.0 Legacy Shadow resource/rollback gate failed: ${message}`,
|
||||
);
|
||||
this.name = 'QingLong3LegacyShadowResourceRollbackError';
|
||||
}
|
||||
}
|
||||
|
||||
function integerArgument(name, value, minimum, maximum) {
|
||||
if (!/^\d+$/.test(value)) {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
`${name} must be an integer`,
|
||||
);
|
||||
}
|
||||
const parsed = Number(value);
|
||||
if (!Number.isSafeInteger(parsed) || parsed < minimum || parsed > maximum) {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
`${name} must be between ${minimum} and ${maximum}`,
|
||||
);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function parseArguments(argv) {
|
||||
const options = {
|
||||
profile: 'edge',
|
||||
mode: 'full',
|
||||
samples: 8,
|
||||
maxAuditP95Ms: 2_000,
|
||||
maxRssDeltaBytes: 64 * MIB,
|
||||
requireCompiled: false,
|
||||
json: false,
|
||||
};
|
||||
for (const argument of argv) {
|
||||
if (argument === '--' || argument === '--json') {
|
||||
if (argument === '--json') options.json = true;
|
||||
} else if (argument === '--require-compiled') {
|
||||
options.requireCompiled = true;
|
||||
} else if (argument.startsWith('--profile=')) {
|
||||
const value = argument.slice('--profile='.length);
|
||||
if (!(value in PROFILES)) {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
'--profile must be edge or standalone',
|
||||
);
|
||||
}
|
||||
options.profile = value;
|
||||
} else if (argument.startsWith('--mode=')) {
|
||||
const value = argument.slice('--mode='.length);
|
||||
if (value !== 'audit-only' && value !== 'full') {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
'--mode must be audit-only or full',
|
||||
);
|
||||
}
|
||||
options.mode = value;
|
||||
} else if (argument.startsWith('--samples=')) {
|
||||
options.samples = integerArgument(
|
||||
'--samples',
|
||||
argument.slice('--samples='.length),
|
||||
1,
|
||||
32,
|
||||
);
|
||||
} else if (argument.startsWith('--max-audit-p95-ms=')) {
|
||||
options.maxAuditP95Ms = integerArgument(
|
||||
'--max-audit-p95-ms',
|
||||
argument.slice('--max-audit-p95-ms='.length),
|
||||
1,
|
||||
60_000,
|
||||
);
|
||||
} else if (argument.startsWith('--max-rss-delta-mb=')) {
|
||||
options.maxRssDeltaBytes =
|
||||
integerArgument(
|
||||
'--max-rss-delta-mb',
|
||||
argument.slice('--max-rss-delta-mb='.length),
|
||||
1,
|
||||
512,
|
||||
) * MIB;
|
||||
} else {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
`unsupported argument ${argument}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return Object.freeze(options);
|
||||
}
|
||||
|
||||
function nextIdentifier(sequence) {
|
||||
return `019f7500-0000-7000-8000-${String(sequence).padStart(12, '0')}`;
|
||||
}
|
||||
|
||||
async function createFixture(databasePath, profile) {
|
||||
const { defineSchemaMigrationModel } = fromRuntime('data/schemaMigration');
|
||||
const { runSchemaMigration } = fromRuntime('migrations/0002-run-schema');
|
||||
const { runningInstanceRunReferenceMigration } = fromRuntime(
|
||||
'migrations/0003-running-instance-run-reference',
|
||||
);
|
||||
const { runCancellationRequestMigration } = fromRuntime(
|
||||
'migrations/0004-run-cancellation-request',
|
||||
);
|
||||
const { runAttemptDeadlineMigration } = fromRuntime(
|
||||
'migrations/0006-run-attempt-deadline',
|
||||
);
|
||||
const { runMigrations } = fromRuntime('migrations/runner');
|
||||
const { createLegacyLogArtifactId } = fromRuntime(
|
||||
'runtime/compatibility/legacyTaskRevision',
|
||||
);
|
||||
const { LegacySequelizeRunRepository } = fromRuntime(
|
||||
'runtime/adapters/legacy-sequelize/runRepository',
|
||||
);
|
||||
const { LegacyShadowRunWriter } = fromRuntime(
|
||||
'runtime/application/legacyShadowRunWriter',
|
||||
);
|
||||
const database = new Sequelize({
|
||||
dialect: 'sqlite',
|
||||
storage: databasePath,
|
||||
logging: false,
|
||||
});
|
||||
try {
|
||||
await database.getQueryInterface().createTable('RunningInstances', {
|
||||
id: {
|
||||
type: DataTypes.INTEGER,
|
||||
primaryKey: true,
|
||||
autoIncrement: true,
|
||||
},
|
||||
cron_id: { type: DataTypes.INTEGER, allowNull: false },
|
||||
pid: { type: DataTypes.INTEGER, allowNull: true },
|
||||
log_path: { type: DataTypes.STRING, allowNull: true },
|
||||
started_at: { type: DataTypes.INTEGER, allowNull: false },
|
||||
finished_at: { type: DataTypes.INTEGER, allowNull: true },
|
||||
status: { type: DataTypes.INTEGER, allowNull: false },
|
||||
exit_code: { type: DataTypes.INTEGER, allowNull: true },
|
||||
});
|
||||
await runMigrations({
|
||||
database,
|
||||
migrationModel: defineSchemaMigrationModel(database),
|
||||
migrations: [
|
||||
runSchemaMigration,
|
||||
runningInstanceRunReferenceMigration,
|
||||
runCancellationRequestMigration,
|
||||
runAttemptDeadlineMigration,
|
||||
],
|
||||
logger: { info() {} },
|
||||
});
|
||||
let identifierSequence = 10_000;
|
||||
const repository = new LegacySequelizeRunRepository(database);
|
||||
const writer = new LegacyShadowRunWriter(repository, () => {
|
||||
identifierSequence += 1;
|
||||
return nextIdentifier(identifierSequence);
|
||||
});
|
||||
const rows = [];
|
||||
for (let index = 0; index < PROFILES[profile].candidates; index += 1) {
|
||||
const acceptedAtMs = BASE_TIME_MS + index * 2_000;
|
||||
const legacyCronId = index + 1;
|
||||
const pid = 20_000 + index;
|
||||
const logPath = `resource/${index}.log`;
|
||||
const reference = await writer.accept({
|
||||
origin: 'manual',
|
||||
projectId: 'default',
|
||||
taskId: `legacy-resource:${index}`,
|
||||
taskRevision: `sha256:${String(index).padStart(64, '0')}`,
|
||||
legacyCronId,
|
||||
triggerType: 'manual',
|
||||
acceptedAtMs,
|
||||
});
|
||||
await writer.spawned(reference, {
|
||||
atMs: acceptedAtMs + 100,
|
||||
pid,
|
||||
logArtifactId: createLegacyLogArtifactId(logPath),
|
||||
});
|
||||
await writer.running(reference, acceptedAtMs + 200);
|
||||
await writer.exited(reference, {
|
||||
atMs: acceptedAtMs + 500,
|
||||
exitCode: 0,
|
||||
});
|
||||
rows.push({
|
||||
cron_id: legacyCronId,
|
||||
run_id: reference.runId,
|
||||
attempt_id: reference.attemptId,
|
||||
pid,
|
||||
log_path: logPath,
|
||||
started_at: Math.floor((acceptedAtMs + 100) / 1_000),
|
||||
finished_at: Math.floor((acceptedAtMs + 500) / 1_000),
|
||||
status: 1,
|
||||
exit_code: 0,
|
||||
});
|
||||
}
|
||||
await database.getQueryInterface().bulkInsert('RunningInstances', rows);
|
||||
} finally {
|
||||
await database.close();
|
||||
}
|
||||
}
|
||||
|
||||
function fileStorage(databasePath) {
|
||||
const files = [
|
||||
databasePath,
|
||||
`${databasePath}-wal`,
|
||||
`${databasePath}-shm`,
|
||||
`${databasePath}-journal`,
|
||||
];
|
||||
let logicalBytes = 0;
|
||||
let allocatedBytes = 0;
|
||||
let fileCount = 0;
|
||||
for (const filePath of files) {
|
||||
try {
|
||||
const stat = fs.statSync(filePath);
|
||||
logicalBytes += stat.size;
|
||||
allocatedBytes += stat.blocks * 512;
|
||||
fileCount += 1;
|
||||
} catch (error) {
|
||||
if (error.code !== 'ENOENT') throw error;
|
||||
}
|
||||
}
|
||||
return Object.freeze({ logicalBytes, allocatedBytes, fileCount });
|
||||
}
|
||||
|
||||
function percentile(values, fraction) {
|
||||
const sorted = [...values].sort((left, right) => left - right);
|
||||
return sorted[Math.max(0, Math.ceil(sorted.length * fraction) - 1)];
|
||||
}
|
||||
|
||||
async function auditChild(options) {
|
||||
const { createLegacyLogArtifactId } = fromRuntime(
|
||||
'runtime/compatibility/legacyTaskRevision',
|
||||
);
|
||||
const { LegacySequelizeShadowTerminalDifferenceSource } = fromRuntime(
|
||||
'runtime/adapters/legacy-sequelize/legacyShadowTerminalDifferenceSource',
|
||||
);
|
||||
const { LegacyShadowTerminalDifferenceAuditor } = fromRuntime(
|
||||
'runtime/application/legacyShadowTerminalDifferenceAuditor',
|
||||
);
|
||||
const databasePath = process.env.QL3_SHADOW_DRILL_DATABASE;
|
||||
if (!path.isAbsolute(databasePath ?? '')) {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
'internal audit database is invalid',
|
||||
);
|
||||
}
|
||||
let queryCount = 0;
|
||||
const database = new Sequelize({
|
||||
dialect: 'sqlite',
|
||||
storage: databasePath,
|
||||
logging() {
|
||||
queryCount += 1;
|
||||
},
|
||||
dialectOptions: { mode: sqlite3.OPEN_READONLY },
|
||||
pool: { max: 1, min: 0, idle: 1_000, acquire: 5_000 },
|
||||
});
|
||||
const rssBeforeBytes = process.memoryUsage().rss;
|
||||
const durationsMs = [];
|
||||
let pages = 0;
|
||||
let scanned = 0;
|
||||
try {
|
||||
const source = new LegacySequelizeShadowTerminalDifferenceSource(
|
||||
database,
|
||||
createLegacyLogArtifactId,
|
||||
);
|
||||
const auditor = new LegacyShadowTerminalDifferenceAuditor(source);
|
||||
const profile = PROFILES[options.profile];
|
||||
const windowEndMs = BASE_TIME_MS + (profile.candidates - 1) * 2_000 + 1_000;
|
||||
for (let sample = 0; sample < options.samples; sample += 1) {
|
||||
const startedAt = performance.now();
|
||||
const report = await auditor.run({
|
||||
profile: options.profile,
|
||||
origins: ['manual'],
|
||||
windowStartMs: BASE_TIME_MS - 1,
|
||||
windowEndMs,
|
||||
observedAtMs: windowEndMs + 5 * 60_000,
|
||||
});
|
||||
durationsMs.push(performance.now() - startedAt);
|
||||
if (
|
||||
report.assessment !== 'matched' ||
|
||||
report.scanned !== profile.candidates ||
|
||||
report.remaining ||
|
||||
!report.evidenceComplete ||
|
||||
report.terminalAgreementPermille !== 1_000
|
||||
) {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
'audit child did not reproduce the closed matched cohort',
|
||||
);
|
||||
}
|
||||
pages += report.pages;
|
||||
scanned += report.scanned;
|
||||
}
|
||||
} finally {
|
||||
await database.close();
|
||||
}
|
||||
const rssAfterBytes = process.memoryUsage().rss;
|
||||
return Object.freeze({
|
||||
samples: options.samples,
|
||||
pages,
|
||||
scanned,
|
||||
queryCount,
|
||||
expectedQueryCount: pages * 2,
|
||||
durationP50Ms: Number(percentile(durationsMs, 0.5).toFixed(3)),
|
||||
durationP95Ms: Number(percentile(durationsMs, 0.95).toFixed(3)),
|
||||
durationMaxMs: Number(Math.max(...durationsMs).toFixed(3)),
|
||||
rssBeforeBytes,
|
||||
rssAfterBytes,
|
||||
rssDeltaBytes: Math.max(0, rssAfterBytes - rssBeforeBytes),
|
||||
peakProcessRssBytes: process.resourceUsage().maxRSS * 1024,
|
||||
});
|
||||
}
|
||||
|
||||
function logger() {
|
||||
return { info() {}, warn() {}, error() {} };
|
||||
}
|
||||
|
||||
async function runSummary(database) {
|
||||
const [row] = await database.query(
|
||||
`SELECT
|
||||
COUNT(*) AS count,
|
||||
SUM(CASE WHEN status = 'succeeded' THEN 1 ELSE 0 END) AS succeeded
|
||||
FROM Runs
|
||||
WHERE execution_owner = 'legacy'`,
|
||||
{ type: QueryTypes.SELECT },
|
||||
);
|
||||
return Object.freeze({
|
||||
count: Number(row.count),
|
||||
succeeded: Number(row.succeeded ?? 0),
|
||||
});
|
||||
}
|
||||
|
||||
async function rollbackChild(mode, expectedBefore) {
|
||||
const databasePath = process.env.QL3_SHADOW_DRILL_DATABASE;
|
||||
if (!path.isAbsolute(databasePath ?? '')) {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
'internal rollback database is invalid',
|
||||
);
|
||||
}
|
||||
const taskLimit = fromRuntime('shared/pLimit').default;
|
||||
await taskLimit.setCustomLimit();
|
||||
const ScheduleService = fromRuntime('services/schedule').default;
|
||||
const bridge = fromRuntime('runtime/compatibility/legacyExecutionBridge');
|
||||
const { sequelize } = fromRuntime('data');
|
||||
let shortCircuitFactCalls = 0;
|
||||
try {
|
||||
const configuredOrigins = bridge.configuredLegacyShadowOrigins();
|
||||
if (mode === 'off') {
|
||||
const observation = bridge.observeLegacyExecution('system', () => {
|
||||
shortCircuitFactCalls += 1;
|
||||
throw new Error('off path must not construct a Shadow fact');
|
||||
});
|
||||
if (observation !== undefined || shortCircuitFactCalls !== 0) {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
'off path did not short-circuit before Shadow fact construction',
|
||||
);
|
||||
}
|
||||
}
|
||||
const service = new ScheduleService(logger());
|
||||
const command = `${JSON.stringify(process.execPath)} -e "process.exit(0)"`;
|
||||
const result = await service.runTask(
|
||||
command,
|
||||
{},
|
||||
{
|
||||
id: 'opaque-resource-rollback',
|
||||
name: 'resource rollback fixture',
|
||||
schedule: '0 * * * *',
|
||||
runOrigin: 'system',
|
||||
},
|
||||
);
|
||||
const expectedAfter = expectedBefore + (mode === 'enabled' ? 1 : 0);
|
||||
const deadline = Date.now() + 10_000;
|
||||
let observed = await runSummary(sequelize);
|
||||
while (
|
||||
(observed.count !== expectedAfter ||
|
||||
observed.succeeded !== expectedAfter) &&
|
||||
Date.now() < deadline
|
||||
) {
|
||||
await new Promise((resolve) => setTimeout(resolve, 10));
|
||||
observed = await runSummary(sequelize);
|
||||
}
|
||||
const loadedModules = Object.keys(require.cache);
|
||||
const defaultObserverLoaded = loadedModules.some((filePath) =>
|
||||
/legacyShadowRunObserver\.(?:ts|js)$/.test(filePath),
|
||||
);
|
||||
const repositoryLoaded = loadedModules.some((filePath) =>
|
||||
/legacy-sequelize\/runRepository\.(?:ts|js)$/.test(filePath),
|
||||
);
|
||||
if (
|
||||
result?.code !== 0 ||
|
||||
observed.count !== expectedAfter ||
|
||||
observed.succeeded !== expectedAfter
|
||||
) {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
`${mode} restart did not preserve the expected Legacy result`,
|
||||
);
|
||||
}
|
||||
return Object.freeze({
|
||||
mode,
|
||||
configuredOrigins,
|
||||
legacyExitCode: result.code,
|
||||
runCountBefore: expectedBefore,
|
||||
runCountAfter: observed.count,
|
||||
succeededRunCountAfter: observed.succeeded,
|
||||
runDelta: observed.count - expectedBefore,
|
||||
shortCircuitFactCalls,
|
||||
defaultObserverLoaded,
|
||||
repositoryLoaded,
|
||||
peakProcessRssBytes: process.resourceUsage().maxRSS * 1024,
|
||||
});
|
||||
} finally {
|
||||
await sequelize.close();
|
||||
}
|
||||
}
|
||||
|
||||
function runInternalChild(args, env) {
|
||||
const result = spawnSync(process.execPath, [__filename, ...args], {
|
||||
cwd: path.resolve(__dirname, '..'),
|
||||
env: { ...process.env, ...env },
|
||||
encoding: 'utf8',
|
||||
timeout: 30_000,
|
||||
maxBuffer: MAX_CHILD_OUTPUT_BYTES,
|
||||
});
|
||||
if (result.error) throw result.error;
|
||||
if (result.status !== 0) {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
`internal child failed: ${(result.stderr || result.stdout).trim()}`,
|
||||
);
|
||||
}
|
||||
try {
|
||||
const lines = result.stdout
|
||||
.trim()
|
||||
.split('\n')
|
||||
.map((line) => line.trim())
|
||||
.filter(Boolean);
|
||||
for (const line of lines.reverse()) {
|
||||
try {
|
||||
return JSON.parse(line);
|
||||
} catch {
|
||||
// Some legacy dependencies emit bounded diagnostics on stdout.
|
||||
}
|
||||
}
|
||||
throw new Error('no JSON record was emitted');
|
||||
} catch (error) {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
`internal child returned invalid JSON: ${error.message}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function inspectDatabase(databasePath) {
|
||||
const database = new Sequelize({
|
||||
dialect: 'sqlite',
|
||||
storage: databasePath,
|
||||
logging: false,
|
||||
dialectOptions: { mode: sqlite3.OPEN_READONLY },
|
||||
pool: { max: 1, min: 0 },
|
||||
});
|
||||
try {
|
||||
const [integrity] = await database.query('PRAGMA integrity_check', {
|
||||
type: QueryTypes.SELECT,
|
||||
});
|
||||
return Object.freeze({
|
||||
integrity: integrity.integrity_check,
|
||||
runCount: (await runSummary(database)).count,
|
||||
});
|
||||
} finally {
|
||||
await database.close();
|
||||
}
|
||||
}
|
||||
|
||||
async function runEvidence(options) {
|
||||
const directory = fs.mkdtempSync(
|
||||
path.join(os.tmpdir(), 'ql3-legacy-shadow-resource-'),
|
||||
);
|
||||
const dataDirectory = path.join(directory, 'data');
|
||||
const databaseDirectory = path.join(dataDirectory, 'db');
|
||||
const databasePath = path.join(databaseDirectory, 'database.sqlite');
|
||||
const previousDataDirectory = process.env.QL_DATA_DIR;
|
||||
fs.mkdirSync(databaseDirectory, { recursive: true });
|
||||
process.env.QL_DATA_DIR = dataDirectory;
|
||||
try {
|
||||
await createFixture(databasePath, options.profile);
|
||||
const beforeAudit = fileStorage(databasePath);
|
||||
const childEnvironment = {
|
||||
QL_DATA_DIR: dataDirectory,
|
||||
QL3_SHADOW_DRILL_DATABASE: databasePath,
|
||||
};
|
||||
let audit;
|
||||
if (options.mode === 'audit-only') {
|
||||
const previousDatabase = process.env.QL3_SHADOW_DRILL_DATABASE;
|
||||
process.env.QL3_SHADOW_DRILL_DATABASE = databasePath;
|
||||
try {
|
||||
audit = await auditChild(options);
|
||||
} finally {
|
||||
if (previousDatabase === undefined) {
|
||||
delete process.env.QL3_SHADOW_DRILL_DATABASE;
|
||||
} else {
|
||||
process.env.QL3_SHADOW_DRILL_DATABASE = previousDatabase;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
audit = runInternalChild(
|
||||
[
|
||||
'--internal-child=audit',
|
||||
`--profile=${options.profile}`,
|
||||
`--samples=${options.samples}`,
|
||||
`--max-audit-p95-ms=${options.maxAuditP95Ms}`,
|
||||
`--max-rss-delta-mb=${options.maxRssDeltaBytes / MIB}`,
|
||||
],
|
||||
{ ...childEnvironment, QL3_SHADOW_ORIGINS: '' },
|
||||
);
|
||||
}
|
||||
const afterAudit = fileStorage(databasePath);
|
||||
const fixtureRunCount = PROFILES[options.profile].candidates;
|
||||
const enabled =
|
||||
options.mode === 'full'
|
||||
? runInternalChild(
|
||||
[
|
||||
'--internal-child=rollback-enabled',
|
||||
`--expected-before=${fixtureRunCount}`,
|
||||
],
|
||||
{ ...childEnvironment, QL3_SHADOW_ORIGINS: 'system' },
|
||||
)
|
||||
: undefined;
|
||||
const off =
|
||||
options.mode === 'full'
|
||||
? runInternalChild(
|
||||
[
|
||||
'--internal-child=rollback-off',
|
||||
`--expected-before=${fixtureRunCount + 1}`,
|
||||
],
|
||||
{ ...childEnvironment, QL3_SHADOW_ORIGINS: '' },
|
||||
)
|
||||
: undefined;
|
||||
const finalDatabase = await inspectDatabase(databasePath);
|
||||
const violations = [];
|
||||
if (audit.queryCount !== audit.expectedQueryCount) {
|
||||
violations.push('audit query count exceeded the two-query page contract');
|
||||
}
|
||||
if (audit.durationP95Ms > options.maxAuditP95Ms) {
|
||||
violations.push('audit p95 exceeded its configured resource budget');
|
||||
}
|
||||
if (audit.rssDeltaBytes > options.maxRssDeltaBytes) {
|
||||
violations.push(
|
||||
'audit RSS delta exceeded its configured resource budget',
|
||||
);
|
||||
}
|
||||
if (
|
||||
beforeAudit.logicalBytes !== afterAudit.logicalBytes ||
|
||||
beforeAudit.allocatedBytes !== afterAudit.allocatedBytes ||
|
||||
beforeAudit.fileCount !== afterAudit.fileCount
|
||||
) {
|
||||
violations.push('read-only audit changed SQLite storage');
|
||||
}
|
||||
if (
|
||||
options.mode === 'full' &&
|
||||
(enabled.runDelta !== 1 ||
|
||||
enabled.legacyExitCode !== 0 ||
|
||||
!enabled.defaultObserverLoaded ||
|
||||
!enabled.repositoryLoaded)
|
||||
) {
|
||||
violations.push(
|
||||
'enabled restart did not produce one Shadow terminal Run',
|
||||
);
|
||||
}
|
||||
if (
|
||||
options.mode === 'full' &&
|
||||
(off.runDelta !== 0 ||
|
||||
off.legacyExitCode !== 0 ||
|
||||
off.configuredOrigins.length !== 0 ||
|
||||
off.shortCircuitFactCalls !== 0 ||
|
||||
off.defaultObserverLoaded ||
|
||||
off.repositoryLoaded)
|
||||
) {
|
||||
violations.push('off restart retained Shadow initialization or writes');
|
||||
}
|
||||
if (
|
||||
finalDatabase.integrity !== 'ok' ||
|
||||
finalDatabase.runCount !==
|
||||
fixtureRunCount + (options.mode === 'full' ? 1 : 0)
|
||||
) {
|
||||
violations.push('rollback drill database did not converge safely');
|
||||
}
|
||||
return Object.freeze({
|
||||
schemaVersion: 1,
|
||||
fixture: 'qinglong/legacy-shadow-resource-rollback-evidence@v1',
|
||||
profile: options.profile,
|
||||
workload: Object.freeze({
|
||||
mode: options.mode,
|
||||
runtime: BUILD_AVAILABLE ? 'compiled_backend' : 'typescript_fallback',
|
||||
candidates: PROFILES[options.profile].candidates,
|
||||
pageSize: PROFILES[options.profile].pageSize,
|
||||
maxPages: PROFILES[options.profile].maxPages,
|
||||
samples: options.samples,
|
||||
}),
|
||||
audit: Object.freeze({
|
||||
...audit,
|
||||
databaseStorageBefore: beforeAudit,
|
||||
databaseStorageAfter: afterAudit,
|
||||
readOnlyStorageStable:
|
||||
beforeAudit.logicalBytes === afterAudit.logicalBytes &&
|
||||
beforeAudit.allocatedBytes === afterAudit.allocatedBytes &&
|
||||
beforeAudit.fileCount === afterAudit.fileCount,
|
||||
}),
|
||||
rollback:
|
||||
options.mode === 'full'
|
||||
? Object.freeze({
|
||||
performed: true,
|
||||
mechanism: 'process_restart_environment_disable',
|
||||
enabled: Object.freeze({
|
||||
configuredOrigins: enabled.configuredOrigins,
|
||||
legacyExitCode: enabled.legacyExitCode,
|
||||
runDelta: enabled.runDelta,
|
||||
defaultObserverLoaded: enabled.defaultObserverLoaded,
|
||||
repositoryLoaded: enabled.repositoryLoaded,
|
||||
peakProcessRssBytes: enabled.peakProcessRssBytes,
|
||||
}),
|
||||
off: Object.freeze({
|
||||
configuredOrigins: off.configuredOrigins,
|
||||
legacyExitCode: off.legacyExitCode,
|
||||
runDelta: off.runDelta,
|
||||
shortCircuitFactCalls: off.shortCircuitFactCalls,
|
||||
defaultObserverLoaded: off.defaultObserverLoaded,
|
||||
repositoryLoaded: off.repositoryLoaded,
|
||||
peakProcessRssBytes: off.peakProcessRssBytes,
|
||||
}),
|
||||
finalRunCount: finalDatabase.runCount,
|
||||
databaseIntegrity: finalDatabase.integrity,
|
||||
legacyContinued:
|
||||
enabled.legacyExitCode === 0 && off.legacyExitCode === 0,
|
||||
shadowWritesStopped: enabled.runDelta === 1 && off.runDelta === 0,
|
||||
physicalPowerLossProven: false,
|
||||
})
|
||||
: Object.freeze({
|
||||
performed: false,
|
||||
reason: 'separate_release_gate',
|
||||
finalRunCount: finalDatabase.runCount,
|
||||
databaseIntegrity: finalDatabase.integrity,
|
||||
}),
|
||||
qualification: Object.freeze({
|
||||
passed: violations.length === 0,
|
||||
measures: options.mode === 'full' ? MEASURES : MEASURES.slice(0, 4),
|
||||
doesNotProve:
|
||||
options.mode === 'full'
|
||||
? EXCLUSIONS
|
||||
: [...EXCLUSIONS, 'shadow_off_process_restart_rollback'],
|
||||
violations,
|
||||
}),
|
||||
});
|
||||
} finally {
|
||||
if (previousDataDirectory === undefined) {
|
||||
delete process.env.QL_DATA_DIR;
|
||||
} else {
|
||||
process.env.QL_DATA_DIR = previousDataDirectory;
|
||||
}
|
||||
fs.rmSync(directory, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const internalChild = process.argv.find((argument) =>
|
||||
argument.startsWith('--internal-child='),
|
||||
);
|
||||
if (internalChild) {
|
||||
const mode = internalChild.slice('--internal-child='.length);
|
||||
const profile =
|
||||
process.argv
|
||||
.find((argument) => argument.startsWith('--profile='))
|
||||
?.slice('--profile='.length) ?? 'edge';
|
||||
const samples = Number(
|
||||
process.argv
|
||||
.find((argument) => argument.startsWith('--samples='))
|
||||
?.slice('--samples='.length) ?? 1,
|
||||
);
|
||||
let report;
|
||||
if (mode === 'audit') {
|
||||
report = await auditChild({ profile, samples });
|
||||
} else if (mode === 'rollback-enabled' || mode === 'rollback-off') {
|
||||
const expectedBefore = Number(
|
||||
process.argv
|
||||
.find((argument) => argument.startsWith('--expected-before='))
|
||||
?.slice('--expected-before='.length),
|
||||
);
|
||||
if (!Number.isSafeInteger(expectedBefore) || expectedBefore < 0) {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
'internal expected Run count is invalid',
|
||||
);
|
||||
}
|
||||
report = await rollbackChild(
|
||||
mode === 'rollback-enabled' ? 'enabled' : 'off',
|
||||
expectedBefore,
|
||||
);
|
||||
} else {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
'internal child mode is invalid',
|
||||
);
|
||||
}
|
||||
process.stdout.write(`${JSON.stringify(report)}\n`);
|
||||
return;
|
||||
}
|
||||
const options = parseArguments(process.argv.slice(2));
|
||||
if (Number(process.versions.node.split('.')[0]) < 24) {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
'Node.js 24 or newer is required',
|
||||
);
|
||||
}
|
||||
if (options.requireCompiled && !BUILD_AVAILABLE) {
|
||||
throw new QingLong3LegacyShadowResourceRollbackError(
|
||||
'compiled backend is required for resource evidence',
|
||||
);
|
||||
}
|
||||
const report = await runEvidence(options);
|
||||
process.stdout.write(
|
||||
`${JSON.stringify(report, null, options.json ? 0 : 2)}\n`,
|
||||
);
|
||||
if (!report.qualification.passed) process.exitCode = 1;
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
EXCLUSIONS,
|
||||
MEASURES,
|
||||
PROFILES,
|
||||
QingLong3LegacyShadowResourceRollbackError,
|
||||
parseArguments,
|
||||
percentile,
|
||||
runEvidence,
|
||||
};
|
||||
|
||||
if (require.main === module) {
|
||||
main().catch((error) => {
|
||||
process.stderr.write(
|
||||
`${error instanceof Error ? error.message : String(error)}\n`,
|
||||
);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
}
|
||||
@@ -22,6 +22,9 @@ const RESOURCE_TIERS = Object.freeze({
|
||||
sqliteMaxTransactionP95Ms: 500,
|
||||
sqliteMaxBatchStallMs: 5_000,
|
||||
sqliteMaxRssDeltaMb: 32,
|
||||
shadowAuditSamples: 8,
|
||||
shadowAuditMaxP95Ms: 1_500,
|
||||
shadowAuditMaxRssDeltaMb: 32,
|
||||
workflowLockSamples: 16,
|
||||
workflowMaxLockP95Ms: 500,
|
||||
workflowMaxProcessRssMb: 96,
|
||||
@@ -41,6 +44,9 @@ const RESOURCE_TIERS = Object.freeze({
|
||||
sqliteMaxTransactionP95Ms: 250,
|
||||
sqliteMaxBatchStallMs: 2_500,
|
||||
sqliteMaxRssDeltaMb: 64,
|
||||
shadowAuditSamples: 8,
|
||||
shadowAuditMaxP95Ms: 1_000,
|
||||
shadowAuditMaxRssDeltaMb: 64,
|
||||
workflowLockSamples: 32,
|
||||
workflowMaxLockP95Ms: 250,
|
||||
workflowMaxProcessRssMb: 160,
|
||||
@@ -164,6 +170,33 @@ function parseMountOptions(raw) {
|
||||
return mounts;
|
||||
}
|
||||
|
||||
function mountOptionsForPath(mounts, targetPath) {
|
||||
let selectedPath;
|
||||
let selectedOptions;
|
||||
for (const [mountPath, options] of mounts) {
|
||||
const containsTarget =
|
||||
mountPath === targetPath ||
|
||||
(mountPath === '/' && targetPath.startsWith('/')) ||
|
||||
(mountPath !== '/' && targetPath.startsWith(`${mountPath}/`));
|
||||
if (
|
||||
containsTarget &&
|
||||
(selectedPath === undefined || mountPath.length > selectedPath.length)
|
||||
) {
|
||||
selectedPath = mountPath;
|
||||
selectedOptions = options;
|
||||
}
|
||||
}
|
||||
return selectedOptions;
|
||||
}
|
||||
|
||||
function isPathTreeReadOnly(mounts, targetPath) {
|
||||
if (!mountOptionsForPath(mounts, targetPath)?.includes('ro')) return false;
|
||||
return [...mounts].every(
|
||||
([mountPath, options]) =>
|
||||
!mountPath.startsWith(`${targetPath}/`) || !options.includes('rw'),
|
||||
);
|
||||
}
|
||||
|
||||
function readLinuxEnvelope(root = '/') {
|
||||
const read = (relativePath) =>
|
||||
fs.readFileSync(path.join(root, relativePath), 'utf8');
|
||||
@@ -219,12 +252,13 @@ function validateEnvelope(tierName, envelope, identity) {
|
||||
violations.push(`architecture ${identity.architecture} is unsupported`);
|
||||
}
|
||||
if (identity.uid === 0) violations.push('resource workload must be non-root');
|
||||
for (const mountPath of ['/', '/workspace']) {
|
||||
if (!envelope.mounts.get(mountPath)?.includes('ro')) {
|
||||
violations.push(`${mountPath} must be mounted read-only`);
|
||||
}
|
||||
if (!mountOptionsForPath(envelope.mounts, '/')?.includes('ro')) {
|
||||
violations.push('/ must be mounted read-only');
|
||||
}
|
||||
if (!envelope.mounts.get('/tmp')?.includes('rw')) {
|
||||
if (!isPathTreeReadOnly(envelope.mounts, '/workspace')) {
|
||||
violations.push('/workspace must be mounted read-only');
|
||||
}
|
||||
if (!mountOptionsForPath(envelope.mounts, '/tmp')?.includes('rw')) {
|
||||
violations.push('/tmp must be a writable bounded tmpfs');
|
||||
}
|
||||
return Object.freeze(violations);
|
||||
@@ -319,7 +353,7 @@ function createWorkloadPlans(root, tierName) {
|
||||
durableOutputBytes: PROMPT_RESOURCE_OUTPUT_BYTES,
|
||||
providerCalls: 2,
|
||||
keyLoads: 1,
|
||||
keyResolutions: 1,
|
||||
keyResolutions: 2,
|
||||
exactReplay: true,
|
||||
contentFree: true,
|
||||
maxLogicalWriteAmplificationPermille: 3_000,
|
||||
@@ -346,40 +380,44 @@ function createWorkloadPlans(root, tierName) {
|
||||
`--max-lock-p95-ms=${tier.workflowMaxLockP95Ms}`,
|
||||
]),
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'local-workflow-admission-crash-recovery',
|
||||
format: 'node_test',
|
||||
nodeArgs: Object.freeze([
|
||||
path.join(
|
||||
root,
|
||||
'packages/ql3-local-sqlite/test/pluginPackageWorkflowAdmissionCrashMatrix.test.cjs',
|
||||
),
|
||||
]),
|
||||
contract: Object.freeze({
|
||||
profiles: Object.freeze(['edge', 'standalone']),
|
||||
crashPointsPerProfile: 8,
|
||||
scenarios: 16,
|
||||
mechanism: 'process_sigkill_then_database_reopen',
|
||||
physicalPowerLossProven: false,
|
||||
}),
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'local-workflow-control-crash-recovery',
|
||||
format: 'node_test',
|
||||
nodeArgs: Object.freeze([
|
||||
path.join(
|
||||
root,
|
||||
'packages/ql3-local-sqlite/test/pluginPackageWorkflowTaskControlCrashMatrix.test.cjs',
|
||||
),
|
||||
]),
|
||||
contract: Object.freeze({
|
||||
profiles: Object.freeze(['edge', 'standalone']),
|
||||
crashPointsPerProfile: 8,
|
||||
scenarios: 16,
|
||||
conclusiveStopObserved: true,
|
||||
physicalPowerLossProven: false,
|
||||
}),
|
||||
}),
|
||||
...(tierName === 'edge-release-ci'
|
||||
? [
|
||||
Object.freeze({
|
||||
name: 'local-workflow-admission-crash-recovery',
|
||||
format: 'node_test',
|
||||
nodeArgs: Object.freeze([
|
||||
path.join(
|
||||
root,
|
||||
'packages/ql3-local-sqlite/test/pluginPackageWorkflowAdmissionCrashMatrix.test.cjs',
|
||||
),
|
||||
]),
|
||||
contract: Object.freeze({
|
||||
profiles: Object.freeze(['edge', 'standalone']),
|
||||
crashPointsPerProfile: 8,
|
||||
scenarios: 16,
|
||||
mechanism: 'process_sigkill_then_database_reopen',
|
||||
physicalPowerLossProven: false,
|
||||
}),
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'local-workflow-control-crash-recovery',
|
||||
format: 'node_test',
|
||||
nodeArgs: Object.freeze([
|
||||
path.join(
|
||||
root,
|
||||
'packages/ql3-local-sqlite/test/pluginPackageWorkflowTaskControlCrashMatrix.test.cjs',
|
||||
),
|
||||
]),
|
||||
contract: Object.freeze({
|
||||
profiles: Object.freeze(['edge', 'standalone']),
|
||||
crashPointsPerProfile: 8,
|
||||
scenarios: 16,
|
||||
conclusiveStopObserved: true,
|
||||
physicalPowerLossProven: false,
|
||||
}),
|
||||
}),
|
||||
]
|
||||
: []),
|
||||
...(tierName === 'edge-release-ci'
|
||||
? [
|
||||
Object.freeze({
|
||||
@@ -437,6 +475,42 @@ function createWorkloadPlans(root, tierName) {
|
||||
`--max-database-growth-bytes=${4 * MIB}`,
|
||||
]),
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'legacy-shadow-terminal-edge',
|
||||
script: path.join(
|
||||
root,
|
||||
'scripts/ql3-legacy-shadow-resource-rollback.cjs',
|
||||
),
|
||||
args: Object.freeze([
|
||||
'--json',
|
||||
'--require-compiled',
|
||||
'--profile=edge',
|
||||
`--mode=${tierName === 'edge-release-ci' ? 'full' : 'audit-only'}`,
|
||||
`--samples=${tier.shadowAuditSamples}`,
|
||||
`--max-audit-p95-ms=${tier.shadowAuditMaxP95Ms}`,
|
||||
`--max-rss-delta-mb=${tier.shadowAuditMaxRssDeltaMb}`,
|
||||
]),
|
||||
}),
|
||||
...(tierName === 'edge-release-ci'
|
||||
? [
|
||||
Object.freeze({
|
||||
name: 'legacy-shadow-terminal-standalone',
|
||||
script: path.join(
|
||||
root,
|
||||
'scripts/ql3-legacy-shadow-resource-rollback.cjs',
|
||||
),
|
||||
args: Object.freeze([
|
||||
'--json',
|
||||
'--require-compiled',
|
||||
'--profile=standalone',
|
||||
'--mode=audit-only',
|
||||
`--samples=${tier.shadowAuditSamples}`,
|
||||
`--max-audit-p95-ms=${tier.shadowAuditMaxP95Ms}`,
|
||||
`--max-rss-delta-mb=${tier.shadowAuditMaxRssDeltaMb}`,
|
||||
]),
|
||||
}),
|
||||
]
|
||||
: []),
|
||||
]);
|
||||
}
|
||||
return Object.freeze([
|
||||
@@ -656,10 +730,11 @@ function main() {
|
||||
pidsMax: after.pidsMax,
|
||||
noNewPrivileges: after.noNewPrivileges,
|
||||
seccompMode: after.seccompMode,
|
||||
rootReadOnly: after.mounts.get('/')?.includes('ro') ?? false,
|
||||
workspaceReadOnly:
|
||||
after.mounts.get('/workspace')?.includes('ro') ?? false,
|
||||
tmpWritable: after.mounts.get('/tmp')?.includes('rw') ?? false,
|
||||
rootReadOnly:
|
||||
mountOptionsForPath(after.mounts, '/')?.includes('ro') ?? false,
|
||||
workspaceReadOnly: isPathTreeReadOnly(after.mounts, '/workspace'),
|
||||
tmpWritable:
|
||||
mountOptionsForPath(after.mounts, '/tmp')?.includes('rw') ?? false,
|
||||
memoryEventsBefore: before.memoryEvents,
|
||||
memoryEventsAfter: after.memoryEvents,
|
||||
},
|
||||
@@ -679,6 +754,8 @@ module.exports = {
|
||||
RESOURCE_TIERS,
|
||||
QingLong3LinuxResourceGateError,
|
||||
createWorkloadPlans,
|
||||
isPathTreeReadOnly,
|
||||
mountOptionsForPath,
|
||||
parseArguments,
|
||||
parseCpuMax,
|
||||
parseKeyValueFile,
|
||||
|
||||
@@ -0,0 +1,207 @@
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const { spawnSync } = require('node:child_process');
|
||||
const { test } = require('node:test');
|
||||
const {
|
||||
EXCLUSIONS,
|
||||
MEASURES,
|
||||
PROFILES,
|
||||
parseArguments,
|
||||
percentile,
|
||||
} = require('../../scripts/ql3-legacy-shadow-resource-rollback.cjs');
|
||||
|
||||
const REPOSITORY_ROOT = path.resolve(__dirname, '../..');
|
||||
const SCRIPT = path.join(
|
||||
REPOSITORY_ROOT,
|
||||
'scripts/ql3-legacy-shadow-resource-rollback.cjs',
|
||||
);
|
||||
const NODE_MAJOR = Number(process.versions.node.split('.')[0]);
|
||||
|
||||
function execute(profile, mode, samples) {
|
||||
const result = spawnSync(
|
||||
process.execPath,
|
||||
[
|
||||
SCRIPT,
|
||||
`--profile=${profile}`,
|
||||
`--mode=${mode}`,
|
||||
`--samples=${samples}`,
|
||||
'--max-audit-p95-ms=10000',
|
||||
'--max-rss-delta-mb=512',
|
||||
'--json',
|
||||
],
|
||||
{
|
||||
cwd: REPOSITORY_ROOT,
|
||||
encoding: 'utf8',
|
||||
timeout: 30_000,
|
||||
maxBuffer: 128 * 1024,
|
||||
},
|
||||
);
|
||||
assert.equal(result.status, 0, result.stderr || result.stdout);
|
||||
return JSON.parse(result.stdout.trim().split('\n').at(-1));
|
||||
}
|
||||
|
||||
test('normalizes only bounded resource and rollback gate options', () => {
|
||||
const packageManifest = JSON.parse(
|
||||
fs.readFileSync(path.join(REPOSITORY_ROOT, 'package.json'), 'utf8'),
|
||||
);
|
||||
const buildSource = fs.readFileSync(
|
||||
path.join(REPOSITORY_ROOT, 'scripts/ql3-build-back.cjs'),
|
||||
'utf8',
|
||||
);
|
||||
assert.equal(
|
||||
packageManifest.scripts['build:back'],
|
||||
'node scripts/ql3-build-back.cjs',
|
||||
);
|
||||
assert.match(buildSource, /path\.join\(output, 'back'\)/);
|
||||
assert.match(buildSource, /fs\.renameSync\(compiledBackend, TARGET\)/);
|
||||
assert.match(buildSource, /rewriteSourceMaps\(maps\)/);
|
||||
assert.doesNotMatch(buildSource, /shell:\s*true/);
|
||||
assert.deepEqual(
|
||||
parseArguments([
|
||||
'--profile=standalone',
|
||||
'--mode=audit-only',
|
||||
'--samples=3',
|
||||
'--max-audit-p95-ms=2500',
|
||||
'--max-rss-delta-mb=48',
|
||||
'--require-compiled',
|
||||
'--json',
|
||||
]),
|
||||
{
|
||||
profile: 'standalone',
|
||||
mode: 'audit-only',
|
||||
samples: 3,
|
||||
maxAuditP95Ms: 2_500,
|
||||
maxRssDeltaBytes: 48 * 1024 * 1024,
|
||||
requireCompiled: true,
|
||||
json: true,
|
||||
},
|
||||
);
|
||||
assert.throws(
|
||||
() => parseArguments(['--profile=cluster']),
|
||||
/profile must be edge or standalone/,
|
||||
);
|
||||
assert.throws(
|
||||
() => parseArguments(['--mode=rollback-only']),
|
||||
/mode must be audit-only or full/,
|
||||
);
|
||||
assert.throws(
|
||||
() => parseArguments(['--samples=33']),
|
||||
/samples must be between 1 and 32/,
|
||||
);
|
||||
assert.throws(
|
||||
() => parseArguments(['--database=/private/data.sqlite']),
|
||||
/unsupported argument/,
|
||||
);
|
||||
assert.equal(PROFILES.edge.candidates, 8);
|
||||
assert.equal(PROFILES.standalone.candidates, 128);
|
||||
assert.deepEqual(MEASURES.slice(0, 4), [
|
||||
'real_sequelize_sqlite_shadow_terminal_audit',
|
||||
'profile_maximum_closed_window_candidate_count',
|
||||
'bounded_candidate_and_evidence_query_count',
|
||||
'read_only_database_storage_stability',
|
||||
]);
|
||||
assert.ok(EXCLUSIONS.includes('primary_execution_eligibility'));
|
||||
});
|
||||
|
||||
test('calculates deterministic nearest-rank latency percentiles', () => {
|
||||
assert.equal(percentile([9, 1, 7, 3], 0.5), 3);
|
||||
assert.equal(percentile([9, 1, 7, 3], 0.95), 9);
|
||||
});
|
||||
|
||||
test(
|
||||
'proves a real enabled-to-off process restart without stopping Legacy execution',
|
||||
{ skip: NODE_MAJOR < 24 ? 'resource evidence requires Node 24' : false },
|
||||
() => {
|
||||
const report = execute('edge', 'full', 2);
|
||||
|
||||
assert.equal(report.schemaVersion, 1);
|
||||
assert.equal(
|
||||
report.fixture,
|
||||
'qinglong/legacy-shadow-resource-rollback-evidence@v1',
|
||||
);
|
||||
assert.equal(report.profile, 'edge');
|
||||
assert.deepEqual(
|
||||
{
|
||||
mode: report.workload.mode,
|
||||
candidates: report.workload.candidates,
|
||||
pageSize: report.workload.pageSize,
|
||||
maxPages: report.workload.maxPages,
|
||||
samples: report.workload.samples,
|
||||
},
|
||||
{ mode: 'full', candidates: 8, pageSize: 8, maxPages: 1, samples: 2 },
|
||||
);
|
||||
assert.ok(
|
||||
['compiled_backend', 'typescript_fallback'].includes(
|
||||
report.workload.runtime,
|
||||
),
|
||||
);
|
||||
assert.equal(report.audit.scanned, 16);
|
||||
assert.equal(report.audit.queryCount, 4);
|
||||
assert.equal(report.audit.expectedQueryCount, 4);
|
||||
assert.equal(report.audit.readOnlyStorageStable, true);
|
||||
assert.equal(report.rollback.performed, true);
|
||||
assert.deepEqual(report.rollback.enabled.configuredOrigins, ['system']);
|
||||
assert.equal(report.rollback.enabled.legacyExitCode, 0);
|
||||
assert.equal(report.rollback.enabled.runDelta, 1);
|
||||
assert.equal(report.rollback.enabled.defaultObserverLoaded, true);
|
||||
assert.equal(report.rollback.enabled.repositoryLoaded, true);
|
||||
assert.deepEqual(report.rollback.off.configuredOrigins, []);
|
||||
assert.equal(report.rollback.off.legacyExitCode, 0);
|
||||
assert.equal(report.rollback.off.runDelta, 0);
|
||||
assert.equal(report.rollback.off.shortCircuitFactCalls, 0);
|
||||
assert.equal(report.rollback.off.defaultObserverLoaded, false);
|
||||
assert.equal(report.rollback.off.repositoryLoaded, false);
|
||||
assert.equal(report.rollback.legacyContinued, true);
|
||||
assert.equal(report.rollback.shadowWritesStopped, true);
|
||||
assert.equal(report.rollback.databaseIntegrity, 'ok');
|
||||
assert.equal(report.rollback.physicalPowerLossProven, false);
|
||||
assert.equal(report.qualification.passed, true);
|
||||
assert.deepEqual(report.qualification.violations, []);
|
||||
assert.deepEqual(report.qualification.measures, MEASURES);
|
||||
const serialized = JSON.stringify(report);
|
||||
for (const forbidden of [
|
||||
'database.sqlite',
|
||||
'opaque-resource-rollback',
|
||||
'process.exit',
|
||||
'legacy-resource:',
|
||||
'resource/0.log',
|
||||
]) {
|
||||
assert.equal(serialized.includes(forbidden), false);
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
test(
|
||||
'exercises the full Standalone page budget without claiming rollback twice',
|
||||
{ skip: NODE_MAJOR < 24 ? 'resource evidence requires Node 24' : false },
|
||||
() => {
|
||||
const report = execute('standalone', 'audit-only', 1);
|
||||
|
||||
assert.deepEqual(report.workload, {
|
||||
mode: 'audit-only',
|
||||
runtime: report.workload.runtime,
|
||||
candidates: 128,
|
||||
pageSize: 32,
|
||||
maxPages: 4,
|
||||
samples: 1,
|
||||
});
|
||||
assert.equal(report.audit.pages, 4);
|
||||
assert.equal(report.audit.scanned, 128);
|
||||
assert.equal(report.audit.queryCount, 8);
|
||||
assert.equal(report.audit.expectedQueryCount, 8);
|
||||
assert.equal(report.audit.readOnlyStorageStable, true);
|
||||
assert.deepEqual(report.rollback, {
|
||||
performed: false,
|
||||
reason: 'separate_release_gate',
|
||||
finalRunCount: 128,
|
||||
databaseIntegrity: 'ok',
|
||||
});
|
||||
assert.ok(
|
||||
report.qualification.doesNotProve.includes(
|
||||
'shadow_off_process_restart_rollback',
|
||||
),
|
||||
);
|
||||
assert.equal(report.qualification.passed, true);
|
||||
},
|
||||
);
|
||||
@@ -3,6 +3,8 @@ const { test } = require('node:test');
|
||||
const {
|
||||
RESOURCE_TIERS,
|
||||
createWorkloadPlans,
|
||||
isPathTreeReadOnly,
|
||||
mountOptionsForPath,
|
||||
parseArguments,
|
||||
parseCpuMax,
|
||||
parseKeyValueFile,
|
||||
@@ -69,6 +71,11 @@ test('parses bounded cgroup v2 and mount evidence', () => {
|
||||
assert.deepEqual(mounts.get('/'), ['ro', 'relatime']);
|
||||
assert.deepEqual(mounts.get('/tmp'), ['rw', 'nosuid', 'nodev']);
|
||||
assert.deepEqual(mounts.get('/workspace'), ['ro', 'nodev']);
|
||||
assert.deepEqual(mountOptionsForPath(mounts, '/workspace/cache'), [
|
||||
'ro',
|
||||
'nodev',
|
||||
]);
|
||||
assert.equal(isPathTreeReadOnly(mounts, '/workspace'), true);
|
||||
});
|
||||
|
||||
test('fails open hosts, root execution and a widened resource envelope', () => {
|
||||
@@ -81,6 +88,31 @@ test('fails open hosts, root execution and a widened resource envelope', () => {
|
||||
}),
|
||||
[],
|
||||
);
|
||||
const inheritedWorkspace = {
|
||||
...valid,
|
||||
mounts: new Map([
|
||||
['/', ['ro']],
|
||||
['/tmp', ['rw']],
|
||||
]),
|
||||
};
|
||||
assert.deepEqual(
|
||||
validateEnvelope('router-stress-ci', inheritedWorkspace, {
|
||||
platform: 'linux',
|
||||
architecture: 'arm64',
|
||||
uid: 65532,
|
||||
}),
|
||||
[],
|
||||
);
|
||||
assert.equal(
|
||||
isPathTreeReadOnly(
|
||||
new Map([
|
||||
['/', ['ro']],
|
||||
['/workspace/cache', ['rw']],
|
||||
]),
|
||||
'/workspace',
|
||||
),
|
||||
false,
|
||||
);
|
||||
const widened = {
|
||||
...valid,
|
||||
memoryMaxBytes: 256 * 1024 * 1024,
|
||||
@@ -115,23 +147,32 @@ test('builds tier-specific workload plans without shell commands', () => {
|
||||
'node-sqlite',
|
||||
'local-workflow-product',
|
||||
'local-workflow-sqlite-lock',
|
||||
'local-workflow-admission-crash-recovery',
|
||||
'local-workflow-control-crash-recovery',
|
||||
'plugin-package-failed-upgrade',
|
||||
'legacy-shadow-terminal-edge',
|
||||
],
|
||||
);
|
||||
assert.match(edge[0].script, /ql3-edge-benchmark\.cjs$/);
|
||||
assert.ok(edge[0].args.includes('--max-rss-delta-mb=64'));
|
||||
assert.equal(edge[2].format, 'node_test');
|
||||
const edgeShadow = edge.find(
|
||||
({ name }) => name === 'legacy-shadow-terminal-edge',
|
||||
);
|
||||
assert.match(edgeShadow.script, /ql3-legacy-shadow-resource-rollback\.cjs$/);
|
||||
assert.ok(edgeShadow.args.includes('--profile=edge'));
|
||||
assert.ok(edgeShadow.args.includes('--mode=audit-only'));
|
||||
assert.ok(edgeShadow.args.includes('--samples=8'));
|
||||
const edgeWorkflow = edge.find(
|
||||
({ name }) => name === 'local-workflow-product',
|
||||
);
|
||||
assert.equal(edgeWorkflow.format, 'node_test');
|
||||
assert.ok(
|
||||
edge[2].nodeArgs.some((argument) =>
|
||||
edgeWorkflow.nodeArgs.some((argument) =>
|
||||
/ql3-local-application\/test\/activation\.test\.cjs$/.test(argument),
|
||||
),
|
||||
);
|
||||
assert.equal(edge[2].maxProcessRssBytes, 96 * 1024 * 1024);
|
||||
assert.equal(edge[2].contract, undefined);
|
||||
assert.equal(edgeWorkflow.maxProcessRssBytes, 96 * 1024 * 1024);
|
||||
assert.equal(edgeWorkflow.contract, undefined);
|
||||
assert.equal(
|
||||
edge[2].nodeArgs.includes(
|
||||
edgeWorkflow.nodeArgs.includes(
|
||||
'--test-name-pattern=executes one admitted Workflow',
|
||||
),
|
||||
true,
|
||||
@@ -151,15 +192,20 @@ test('builds tier-specific workload plans without shell commands', () => {
|
||||
'local-ai-prompt-model-invocation-crash-recovery',
|
||||
'local-ai-prompt-outer-transaction-crash-recovery',
|
||||
'plugin-package-failed-upgrade',
|
||||
'legacy-shadow-terminal-edge',
|
||||
'legacy-shadow-terminal-standalone',
|
||||
],
|
||||
);
|
||||
const releaseWorkflow = edgeRelease.find(
|
||||
({ name }) => name === 'local-workflow-product',
|
||||
);
|
||||
assert.equal(
|
||||
edgeRelease[2].nodeArgs.includes(
|
||||
releaseWorkflow.nodeArgs.includes(
|
||||
'--test-name-pattern=executes one admitted Workflow|stops one running Workflow Task',
|
||||
),
|
||||
true,
|
||||
);
|
||||
assert.deepEqual(edgeRelease[2].contract, {
|
||||
assert.deepEqual(releaseWorkflow.contract, {
|
||||
kind: 'local_workflow_product_lifecycle',
|
||||
profile: 'edge',
|
||||
completedWorkflowSteps: 2,
|
||||
@@ -175,51 +221,99 @@ test('builds tier-specific workload plans without shell commands', () => {
|
||||
cancelAudits: 1,
|
||||
physicalPowerLossProven: false,
|
||||
});
|
||||
assert.equal(edgeRelease[3].maxProcessRssBytes, 192 * 1024 * 1024);
|
||||
assert.deepEqual(edgeRelease[3].env, {
|
||||
const releaseEdgeShadow = edgeRelease.find(
|
||||
({ name }) => name === 'legacy-shadow-terminal-edge',
|
||||
);
|
||||
const releaseStandaloneShadow = edgeRelease.find(
|
||||
({ name }) => name === 'legacy-shadow-terminal-standalone',
|
||||
);
|
||||
assert.ok(releaseEdgeShadow.args.includes('--mode=full'));
|
||||
assert.ok(releaseStandaloneShadow.args.includes('--profile=standalone'));
|
||||
assert.ok(releaseStandaloneShadow.args.includes('--mode=audit-only'));
|
||||
const edgePrompt = edgeRelease.find(
|
||||
({ name }) => name === 'local-ai-prompt-durable-output-edge',
|
||||
);
|
||||
const standalonePrompt = edgeRelease.find(
|
||||
({ name }) => name === 'local-ai-prompt-durable-output-standalone',
|
||||
);
|
||||
assert.equal(edgePrompt.maxProcessRssBytes, 192 * 1024 * 1024);
|
||||
assert.deepEqual(edgePrompt.env, {
|
||||
QL3_PROMPT_RESOURCE_PROFILE: 'edge',
|
||||
QL3_PROMPT_RESOURCE_OUTPUT_BYTES: String(512 * 1024),
|
||||
});
|
||||
assert.equal(edgeRelease[3].contract.providerCalls, 2);
|
||||
assert.equal(edgeRelease[3].contract.exactReplay, true);
|
||||
assert.equal(edgeRelease[3].contract.contentFree, true);
|
||||
assert.equal(edgeRelease[3].contract.durableOutputBytes, 512 * 1024);
|
||||
assert.equal(edgeRelease[3].contract.maxWalWriteAmplificationPermille, 0);
|
||||
assert.equal(edgeRelease[4].contract.profile, 'standalone');
|
||||
assert.equal(edgeRelease[4].contract.journalMode, 'wal');
|
||||
assert.equal(edgeRelease[4].contract.requireWalGrowth, true);
|
||||
assert.match(edge[3].script, /ql3-local-workflow-resource-benchmark\.cjs$/);
|
||||
assert.ok(edge[3].args.includes('--lock-samples=16'));
|
||||
assert.ok(edge[3].args.includes('--max-lock-p95-ms=500'));
|
||||
assert.equal(edge[4].contract.scenarios, 16);
|
||||
assert.equal(edge[4].contract.physicalPowerLossProven, false);
|
||||
assert.equal(edge[5].contract.scenarios, 16);
|
||||
assert.equal(edge[5].contract.conclusiveStopObserved, true);
|
||||
assert.equal(edge[5].contract.physicalPowerLossProven, false);
|
||||
assert.equal(edgeRelease[8].contract.scenarios, 14);
|
||||
assert.deepEqual(edgeRelease[8].contract.boundaries, [
|
||||
assert.equal(edgePrompt.contract.providerCalls, 2);
|
||||
assert.equal(edgePrompt.contract.keyLoads, 1);
|
||||
assert.equal(edgePrompt.contract.keyResolutions, 2);
|
||||
assert.equal(edgePrompt.contract.exactReplay, true);
|
||||
assert.equal(edgePrompt.contract.contentFree, true);
|
||||
assert.equal(edgePrompt.contract.durableOutputBytes, 512 * 1024);
|
||||
assert.equal(edgePrompt.contract.maxWalWriteAmplificationPermille, 0);
|
||||
assert.equal(standalonePrompt.contract.profile, 'standalone');
|
||||
assert.equal(standalonePrompt.contract.journalMode, 'wal');
|
||||
assert.equal(standalonePrompt.contract.requireWalGrowth, true);
|
||||
const workflowLock = edge.find(
|
||||
({ name }) => name === 'local-workflow-sqlite-lock',
|
||||
);
|
||||
assert.match(
|
||||
workflowLock.script,
|
||||
/ql3-local-workflow-resource-benchmark\.cjs$/,
|
||||
);
|
||||
assert.ok(workflowLock.args.includes('--lock-samples=16'));
|
||||
assert.ok(workflowLock.args.includes('--max-lock-p95-ms=500'));
|
||||
const releaseAdmissionCrash = edgeRelease.find(
|
||||
({ name }) => name === 'local-workflow-admission-crash-recovery',
|
||||
);
|
||||
const releaseControlCrash = edgeRelease.find(
|
||||
({ name }) => name === 'local-workflow-control-crash-recovery',
|
||||
);
|
||||
assert.equal(releaseAdmissionCrash.env, undefined);
|
||||
assert.deepEqual(releaseAdmissionCrash.contract.profiles, [
|
||||
'edge',
|
||||
'standalone',
|
||||
]);
|
||||
assert.equal(releaseAdmissionCrash.contract.scenarios, 16);
|
||||
assert.equal(releaseControlCrash.env, undefined);
|
||||
assert.deepEqual(releaseControlCrash.contract.profiles, [
|
||||
'edge',
|
||||
'standalone',
|
||||
]);
|
||||
assert.equal(releaseControlCrash.contract.scenarios, 16);
|
||||
const modelCrash = edgeRelease.find(
|
||||
({ name }) => name === 'local-ai-prompt-model-invocation-crash-recovery',
|
||||
);
|
||||
const transactionCrash = edgeRelease.find(
|
||||
({ name }) => name === 'local-ai-prompt-outer-transaction-crash-recovery',
|
||||
);
|
||||
assert.equal(modelCrash.contract.scenarios, 14);
|
||||
assert.deepEqual(modelCrash.contract.boundaries, [
|
||||
'model_start',
|
||||
'model_completion',
|
||||
]);
|
||||
assert.equal(edgeRelease[8].contract.physicalPowerLossProven, false);
|
||||
assert.equal(edgeRelease[9].contract.scenarios, 20);
|
||||
assert.deepEqual(edgeRelease[9].contract.operations, [
|
||||
assert.equal(modelCrash.contract.physicalPowerLossProven, false);
|
||||
assert.equal(transactionCrash.contract.scenarios, 20);
|
||||
assert.deepEqual(transactionCrash.contract.operations, [
|
||||
'admission',
|
||||
'finalization',
|
||||
]);
|
||||
assert.equal(edgeRelease[9].contract.exactReplay, true);
|
||||
assert.equal(edgeRelease[9].contract.contentFree, true);
|
||||
assert.equal(transactionCrash.contract.exactReplay, true);
|
||||
assert.equal(transactionCrash.contract.contentFree, true);
|
||||
assert.equal(
|
||||
edgeRelease[9].contract.promptAdmissionFinalizationCrashProven,
|
||||
transactionCrash.contract.promptAdmissionFinalizationCrashProven,
|
||||
true,
|
||||
);
|
||||
assert.equal(edgeRelease[9].contract.physicalPowerLossProven, false);
|
||||
assert.equal(transactionCrash.contract.physicalPowerLossProven, false);
|
||||
const edgeUpgrade = edge.find(
|
||||
({ name }) => name === 'plugin-package-failed-upgrade',
|
||||
);
|
||||
const releaseUpgrade = edgeRelease.find(
|
||||
({ name }) => name === 'plugin-package-failed-upgrade',
|
||||
);
|
||||
assert.match(
|
||||
edge.at(-1).script,
|
||||
edgeUpgrade.script,
|
||||
/ql3-plugin-package-recovery-edge-benchmark\.cjs$/,
|
||||
);
|
||||
assert.ok(edge.at(-1).args.includes('--max-rss-delta-mb=64'));
|
||||
assert.ok(edgeRelease.at(-1).args.includes('--max-rss-delta-mb=96'));
|
||||
assert.ok(edgeUpgrade.args.includes('--max-rss-delta-mb=64'));
|
||||
assert.ok(releaseUpgrade.args.includes('--max-rss-delta-mb=96'));
|
||||
const cluster = createWorkloadPlans('/workspace', 'cluster-control-ci');
|
||||
assert.deepEqual(
|
||||
cluster.map(({ name }) => name),
|
||||
@@ -256,7 +350,7 @@ test('fails closed when durable Prompt resource evidence drifts', () => {
|
||||
durableOutputBytes: 512 * 1024,
|
||||
providerCalls: 2,
|
||||
keyLoads: 1,
|
||||
keyResolutions: 1,
|
||||
keyResolutions: 2,
|
||||
liveOnlyKeyLoads: 0,
|
||||
exactReplay: true,
|
||||
contentFree: true,
|
||||
|
||||
Reference in New Issue
Block a user