diff --git a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md index 53163f4f..179728a5 100644 --- a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md +++ b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md @@ -11,7 +11,18 @@ 最新增量证据(2026-08-19): -- D-363/ADR-0456(已接受;Cluster 生产启动拓扑待接入):完成 profile-neutral CancellationDispatch canonical contract 与 PostgreSQL +- D-364/ADR-0457(已接受;运维可见性与实机容量门待完成):把 PostgreSQL CancellationDispatch 接入既有 caller-driven Remote Worker + lease-control 生产路径,没有新建扫描 timer 或第二调度 authority。只有已有 Session/RunDispatchLease/Attempt fence 产生 `stop_requested` 后才 claim; + `claimed` 必须先以 `termination_requested` 原子结算 dispatch 与 `run.cancel_dispatched`,验证 durable 结果后才向 Worker 释放停止响应;`dispatched` + 精确重放,foreign live lease、not-due 与 blocked 均失败关闭并撤回可用性。Workflow Task timeout 保持 `untracked`,不伪造父 Run cancellation。 + owner 复用 recovery replica ID,观察面只含固定状态/code/scope;包装层不拥有 timer、listener、queue、connection 或 lifecycle,shutdown 复用 Worker ingress + drain 后再关闭数据库的既有顺序。完整 backend `1,487 pass / 0 fail / 2 conditional skip`(总计 1,489);18-package clean/build 与顺序测试 + 退出 0;五项架构审计与 `14/14` Local artifact audit 通过,workspace 仍为 18 包且新增文件位于 `cluster-control/remote-execution` 子域;基础 + Edge/Standalone 保持 `2,589,998 / 2,590,076` bytes 且无 PostgreSQL 闭包。PostgreSQL 18.6 arm64 HA + `144/144` 保留双连接 claim/takeover/fence/retry/WAL/promotion 证据并新增 production settle-before-stop;timeline `1→2`,报告 SHA-256 为 + `4313b405c2ea56a3d44bc4907d5299b6e5c0d3062bb9c9f1522749d1021bd462`。 + +- D-363/ADR-0456(已接受;生产交付由 D-364 接入):完成 profile-neutral CancellationDispatch canonical contract 与 PostgreSQL `pg-0066-cancellation-dispatch`/capability v65 adapter。公共契约仅从显式子路径发布,调用方不提交当前时间、lease expiry 或绝对 retry timestamp; PostgreSQL 以 `transaction_timestamp()` 作为 lease/retry authority,按 Run→Attempt→dispatch 固定锁序执行 claim/result。raw lease token 只随成功 claim 返回,durable record、表、WAL 与事件只保存 domain-separated SHA-256 digest;结果事务原子完成 dispatch 更新、Run version CAS 与低敏 RunEvent,runtime @@ -9225,7 +9236,7 @@ flowchart LR | PR-2 Run 状态机 | Incubating | 纯转换表、终态/时间/错误/执行器元数据规则、Run version 与 event sequence CAS、事务性 RunCommandService、回滚测试 | 重复 Worker callback/fencing、并发数据库压力测试、Primary 执行链接入 | | PR-3 Executor 端口 | Incubating | ADR-0003、ExecutionSpec/Context/Handle/Result、Executor port、LocalProcessExecutor、进程组取消/超时升级、流式背压、Legacy Cron spec builder、真实进程 contract tests、可复现 edge 基准入口 | 固定 edge/多架构设备基线、Legacy builder 与 makeCommand 差异审计、Primary 生产流量接入 | | PR-4 Shadow Run | Incubating | origin 三态策略;默认关闭的 `QL3_SHADOW_ORIGINS`;manual、scheduled_node、boot、subscription、system 与 script 现有 ChildProcess 旁路观察;system crond 显式 origin marker、Shell execution ID、finish-only 准入、确定性 Run/Attempt 与 exact replay;`@once` 保持 manual、gRPC transport 不冒充 origin 的准入裁决;每个 worker 懒加载;Run/Attempt/Event 影子生命周期;稳定且不复制 caller 原文的 task identity/revision 与有界日志引用;同 worker 有界注册表和跨 worker 持久化候选关联;stop all/stop instance、Shell callback、乱序/迟到/歧义处理;监听前一次性、Profile-aware 的 keyset Startup Reconciler,终态证据补齐、lost/abandoned/pending 分流与 terminal Attempt response-loss 修复;origin-bounded 且逐级守恒的版本化 startup difference report、固定字段 metric batch 与一次性 collector;显式、只读、闭合窗口且 Profile-bounded 的 Shadow→Legacy 终态差异审计;128/256 MiB Linux arm64 资源门、SQLite 零增长与 Shadow enabled→off 进程重启回滚;process-epoch Legacy admission/capture/failure/pending 守恒;clean-shutdown `0600` no-replace capture+startup exporter;manual Edge 8/Standalone 32–128 canary;capture/terminal/resource 自包含 Primary bundle;rollout v2 loader 重算 source digest 与 eligibility;不可变 prepare/observe/resource/qualify 目标实例仪式、独立只读 audit;失败开放和契约测试 | 首次真实目标实例完整 canary 与 bootstrap activated 记录、其他 origin 独立 capture/Primary gate、固定物理 edge/flash/断电证据 | -| PR-5 Primary LocalExecutor | Incubating(默认不激活,仅 manifest-gated manual) | runtime-owned Run 创建器;持久化先于 spawn;Run/Attempt 完整成功、失败、取消、超时与 lost 闭环;Executor handle 身份校验;spawn 后激活写失败的 stop+lost 补偿;completion rejection 安全收敛;独立 Primary 幂等查询与唯一索引竞态裁决;durable `run.cancel_requested`、stop-before-signal、首次请求幂等、晚到完成裁决与待取消有界恢复查询;最多 64 条一页的 cross-worker cancellation source;独立 CancellationDispatch Repository 原子 claim/result、lease expiry 接管、owner/token/version fencing、指数退避与结果 RunEvent;PostgreSQL `pg-0066`/capability v65 adapter 以数据库时间、Run→Attempt→dispatch 锁序、digest-only durable token、最小 runtime 权限和原子 RunEvent/Run version CAS 提供多副本同构实现,真实双连接与 HA promotion 门已通过;最多 64 页的单周期 cancel supervisor;显式 start/stop、无重叠、错误隔离、停止等待有上限且 timer unref 的 lifecycle runner;Linux durable handle 的 PID/boot/start ticks/process-group 复验与 TERM/KILL controller;完整有界分页且 fail-closed 的 startup Reconcile supervisor;RunningInstance nullable `run_id/attempt_id` 关联;Primary 专用组合 Repository 在同一 SQLite 事务提交前投影 Crontab/RunningInstance,失败整体回滚;有界且防穿越的 legacy log output ref;manual owner seam、真实本机装配、单 spawn/fail-closed;严格 manual-only rollout manifest loader、短期审批/gate、配置哈希审计;HTTP worker 已接轻量 lazy bootstrap,accepted 后按 receipt-first reconcile→completion receipt lifecycle→timeout intent lifecycle→cancel dispatch lifecycle→router→durable active receipt 顺序激活,失败撤销,监听失败和 shutdown 以 stopping→有界清理→stopped/failed 失效;receipt 固定为单文件 observed-state projection,Linux 以 boot/PID/process-group/start ticks 复验,独立 auditor 支持 active 且 off/rolled-back 拒绝 live runtime;Primary timeout 在 spawn 前持久化绝对 deadline,有界 source/requester/supervisor 只提交 timeout 意图并复用 CancellationDispatch;代码级 edge/standalone Profile 为各 lifecycle 提供不同 cadence 与页上限,cluster-control/worker 拒绝误装本机 SQLite Primary;统一 CompletionService 原子提交 Attempt/Run/双 Event,spawn 前保存 callback token hash、终态推进 sequence,实时回调与 receipt consumer 共享入口并覆盖两个清理 crash window;manual Primary 已接入受限 POSIX launcher、`0600` direct-file stdout/stderr、父进程退出后续写、不可覆盖 receipt 生产、回执环境清除、TERM 转发等待及 live transaction 后清理;Startup Reconciler receipt-first 双检查并在确定 exited 后执行 profile 化的单次 50/100 ms publish grace;`0007` 独立 CompletionReceiptJournal 在 spawn 前登记、为升级前 active Attempt 补登记并驱动周期扫描,使终态残留继续可发现;确定无效的已知 Attempt receipt 先持久化隔离状态,再进入确定性私有分片 quarantine;终态 missing 与 quarantine 按 edge/standalone retention 有界清理;非 Journal 文件具备只读优先、固定分片/条目上限、overflow fail-closed、显式同盘隔离的 Node 24 运维 CLI;扫描具备页上限、resume cursor、timer unref、无重叠、有界 stop 和低敏计数;ENOSPC 与 launcher receipt 存储失败有代码门禁;显式最长 24 小时 approve 写入、`primary_selected` 只读状态、selection receipt、approval-expiry off 与 intent/completion crash-replay rollback | 首次真实目标实例完整激活/回滚仪式;cluster-control 生产启动拓扑;固定 edge/Linux 多架构与真实磁盘压力基线、完整 2.x API 契约和回滚演练;共享 config 多写者 authority | +| PR-5 Primary LocalExecutor | Incubating(默认不激活,仅 manifest-gated manual) | runtime-owned Run 创建器;持久化先于 spawn;Run/Attempt 完整成功、失败、取消、超时与 lost 闭环;Executor handle 身份校验;spawn 后激活写失败的 stop+lost 补偿;completion rejection 安全收敛;独立 Primary 幂等查询与唯一索引竞态裁决;durable `run.cancel_requested`、stop-before-signal、首次请求幂等、晚到完成裁决与待取消有界恢复查询;最多 64 条一页的 cross-worker cancellation source;独立 CancellationDispatch Repository 原子 claim/result、lease expiry 接管、owner/token/version fencing、指数退避与结果 RunEvent;PostgreSQL `pg-0066`/capability v65 adapter 以数据库时间、Run→Attempt→dispatch 锁序、digest-only durable token、最小 runtime 权限和原子 RunEvent/Run version CAS 提供多副本同构实现,真实双连接与 HA promotion 门已通过;caller-driven Worker lease-control 已以 settle-before-stop 接入 Cluster 生产组合,复用 ingress drain 且不新增 cadence;最多 64 页的单周期 cancel supervisor;显式 start/stop、无重叠、错误隔离、停止等待有上限且 timer unref 的 lifecycle runner;Linux durable handle 的 PID/boot/start ticks/process-group 复验与 TERM/KILL controller;完整有界分页且 fail-closed 的 startup Reconcile supervisor;RunningInstance nullable `run_id/attempt_id` 关联;Primary 专用组合 Repository 在同一 SQLite 事务提交前投影 Crontab/RunningInstance,失败整体回滚;有界且防穿越的 legacy log output ref;manual owner seam、真实本机装配、单 spawn/fail-closed;严格 manual-only rollout manifest loader、短期审批/gate、配置哈希审计;HTTP worker 已接轻量 lazy bootstrap,accepted 后按 receipt-first reconcile→completion receipt lifecycle→timeout intent lifecycle→cancel dispatch lifecycle→router→durable active receipt 顺序激活,失败撤销,监听失败和 shutdown 以 stopping→有界清理→stopped/failed 失效;receipt 固定为单文件 observed-state projection,Linux 以 boot/PID/process-group/start ticks 复验,独立 auditor 支持 active 且 off/rolled-back 拒绝 live runtime;Primary timeout 在 spawn 前持久化绝对 deadline,有界 source/requester/supervisor 只提交 timeout 意图并复用 CancellationDispatch;代码级 edge/standalone Profile 为各 lifecycle 提供不同 cadence 与页上限,cluster-control/worker 拒绝误装本机 SQLite Primary;统一 CompletionService 原子提交 Attempt/Run/双 Event,spawn 前保存 callback token hash、终态推进 sequence,实时回调与 receipt consumer 共享入口并覆盖两个清理 crash window;manual Primary 已接入受限 POSIX launcher、`0600` direct-file stdout/stderr、父进程退出后续写、不可覆盖 receipt 生产、回执环境清除、TERM 转发等待及 live transaction 后清理;Startup Reconciler receipt-first 双检查并在确定 exited 后执行 profile 化的单次 50/100 ms publish grace;`0007` 独立 CompletionReceiptJournal 在 spawn 前登记、为升级前 active Attempt 补登记并驱动周期扫描,使终态残留继续可发现;确定无效的已知 Attempt receipt 先持久化隔离状态,再进入确定性私有分片 quarantine;终态 missing 与 quarantine 按 edge/standalone retention 有界清理;非 Journal 文件具备只读优先、固定分片/条目上限、overflow fail-closed、显式同盘隔离的 Node 24 运维 CLI;扫描具备页上限、resume cursor、timer unref、无重叠、有界 stop 和低敏计数;ENOSPC 与 launcher receipt 存储失败有代码门禁;显式最长 24 小时 approve 写入、`primary_selected` 只读状态、selection receipt、approval-expiry off 与 intent/completion crash-replay rollback | 首次真实目标实例完整激活/回滚仪式;用户可见的 Cluster cancellation availability/blocked 处置面;固定 edge/Linux 多架构与真实磁盘压力基线、完整 2.x API 契约和回滚演练;共享 config 多写者 authority | | PR-7 Worker Session、Run Lease 与启动协议基础 | Incubating(默认关闭,独立入口显式 opt-in) | ADR-0012/0013/0014/0021/0057–0061/0108–0121/0231–0239/0377;有界 capability/Placement/Dispatcher;SQLite 协议孵化与 PostgreSQL v9 Session/Run Lease/credential/attestation authority;immutable revision Placement、数据库时钟 keyset candidate、认证 Worker Pull、digest-only offer recovery;versioned capability-free ExecutionSpec response、stable claim 跨重启退避、单 owner 原子 inbox 准入与 TLS 1.3 mTLS/`ql3w` HTTPS client;同一 package journal 上 revision-fenced starting/spawn/started/running/completion 状态、callback digest、tagged no-spawn 与 ambiguous recovery;PostgreSQL starting/running/start-failure/completion 数据库权威事务、精确重放与 cancellation/timeout 优先终态;batch Secret delivery 在 Attempt advisory lock 下复验 Session/Lease/revision 完整围栏并复用单 Agent,Secret-before-Artifact materializer 将同一 log ID 交给 Executor/journal/running ACK;offer-scoped `wlog-*` 私有文件 spool、Edge/Node 容量策略、append/quota/path 防护、barrier 后 output ownership、受审 POSIX Executor、truncation fact、固定内存流式 source、认证 Artifact stream、共享 immutable store port、S3-compatible SSE/checksum/条件 promotion adapter、upload-before-completion 协调,以及 Local/Cluster 同构、Profile-aware、ETag-fenced range read;用户取消 run.stop mutation 以数据库时间写 intent/Event 并在事务内复验 Project/RoleBinding fence;非执行取消 convergence lifecycle、运行期 expiry 与安全 lost retry 已接入 cluster-control 单一全局 cadence;完整 generation/version/token/Attempt fencing;独立最小权限 Worker ingress、CA/CRL 与连接 generation 热重载;offer journal、spawn barrier、receipt-first recovery;独立 `@qinglong/worker-runtime` 的本地 P-256 CSR、key/chain/trust 验证、generation + active pointer 安装和持久退避;默认关闭的 production process 已装配具体 execution graph、完整 Session heartbeat/drain/offline、direct-file bootstrap、单 Agent/单 cadence、startup reconciliation、证书 maintenance、transport fail-close/recovery 与 Edge/Node 有界预算;真实 PostgreSQL 18 + Linux Node 合约已覆盖 Run completion、credential 和 CA 双轮换且保持同一 Session;真实 K3s 合约已覆盖 TLS/credential Secret 分权、双对象 CAS、Recreate 顺序、identity generation 与单节点 PVC recovery;所有能力默认不可达且受 edge/cluster import audit 约束 | 具体 cert-manager/Vault/SPIFFE/离线 CA adapter 与模板、ingress reload controller、生产 RBAC、证书到期告警和 `ql3w` credential recovery 产品面;具体 KMS/Vault Secret provider、对象存储 credential/temporary lifecycle 与 retention/tombstone;Worker 管理 API;真实 Kubernetes 多节点 CSI/node-loss/production 360 秒 drain 与固定 edge 文件系统 suspend/时钟/断电、x64/arm64 资源门禁 | | PR-8 Project/Policy/Approval Core | Incubating(默认拒绝、无生产业务执行入口) | ADR-0028;统一六类 ActorRef 与 exact-shape 校验;`0017` ownerless default Project 和 append-only versioned RoleBinding;owner/admin/operator/viewer 固定矩阵;Project 内 mutation 幂等、expected-version CAS、双 SQLite 连接竞争门禁;archived read-only、revocation、存储损坏 fail-closed;Agent 写/Secret/Tool `require_approval`;ADR-0047 把六类 subject、role/permission matrix 与 fence 抽到 runtime-core,`pg-0004-project-policy`/capability v3 建立 ownerless PostgreSQL baseline、严格 role/state CHECK、append-only runtime 权限、SERIALIZABLE Project lock、mutation replay、双连接单 winner 和 cluster admission authorizer;ADR-0049/`pg-0005` capability v4 建立 stable IdentitySubject、append-only digest-only API credential、真实 cluster bearer authenticator、write-only durable security audit 与最小权限 runtime role,且已验证 HTTP→credential→Policy→audit→handler 纵向链路;ADR-0051 建立 `/api/v3` 认证前 peer/global 双预算、transport-peer-only、无 timer 且有界内存的 overload shield;ADR-0027 Artifact authorizer adapter;ADR-0029 `AuthenticatedPrincipal` contract、`0018` digest-only versioned challenge、CSPRNG/TTL、同事务消费 challenge + 写首 owner、精确重放与双连接竞争/崩溃回滚门禁;ADR-0030 `0019` stable identity/binding、legacy HS384 + current-session membership、logout/platform/revoke/disable、single-factor 与损坏 fail-closed 门禁;ADR-0031 `0020` digest-bound ApprovalRequest、User-only decision、Project/Role version fence、精确 expiry/重放/并发裁决及同事务 immutable dispatch;ADR-0032 `0021` execution backfill、三表原子 consume、稳定 due keyset、claim/renew/start/result fencing、pre-start takeover/post-start recovery-required、attempt budget、handler inspect/digest barrier 和 bounded dispatcher;ADR-0033/`0022` control/resolution backfill、start/renew/completion 原子联动、稳定 recovery keyset、双 resolver claim/takeover、finding/result 精确重放、自动/人工终结、迟到 completion 单 winner 和 evidence-only bounded reconciler;ADR-0034/`0023` 首个 `run.create` canonical plan、Run/Attempt/Event/receipt 同事务、幂等 collision fail-closed、renew/终态 fence、真实 SQLite handler 与 automatic evidence provider;ADR-0035/`0024` 独立 `approval.recover` 矩阵、稳定 User + 五分钟强认证、Project/RoleBinding fence、human resolution + authorization fact 原子提交、撤权竞态与回滚门禁;ADR-0036 recovery-first 单 timer lifecycle、edge/standalone 独立 cadence/页预算、跨周期 cursor、非重叠与有界 stop;ADR-0074 以新的 Node 24 SQLite v5 ownerless Project/RoleBinding/audit authority 和独立 local-secret-admin 提供强 Principal、`secret.manage`、撤权 fence、envelope+allowed audit 原子提交及不回显语义;ADR-0086 以可信 POSIX console 和 staged delivery 完成本机首 Owner 产品 ceremony | fresh database/pepper setup 与安全迁移向导;`shareStore`/Express 到 authentication core 的 production migration;credential rotation/revocation API、mTLS/Worker enrollment、恢复码;Project/Role/Approval/Secret 管理 CLI/API/UI、audit retention/query/export/alert、preview Artifact/digest/immutable plan builder、真实 MFA/hardware adapter、人工 recovery API/UI/独立 rate limit 与审计事件、handler/provider registry、lifecycle startup/shutdown/指标/admission gate;PostgreSQL action/receipt/provider/recovery-authorization 与 OPA adapter、缓存 version 失效;Tool/Package/Secret/Shell 各自的 handler/evidence contract;Secret/Run/Tool/Workflow waiting_approval 全入口装配;完整回滚演练 | diff --git a/docs/adr/ADR-0005-durable-cancellation-dispatch.md b/docs/adr/ADR-0005-durable-cancellation-dispatch.md index fffe0f4f..8e5f49f7 100644 --- a/docs/adr/ADR-0005-durable-cancellation-dispatch.md +++ b/docs/adr/ADR-0005-durable-cancellation-dispatch.md @@ -151,13 +151,13 @@ PID 可复用,可能终止无关进程,禁止。 ## 5. 当前孵化边界 -`next` 已实现 profile-neutral canonical contract、`0005-run-cancellation-dispatch`、legacy Sequelize/SQLite adapter、PostgreSQL `pg-0066-cancellation-dispatch`/capability v65 adapter、lease expiry 接管、fencing、退避、结果事件、Dispatcher、有界 Supervisor 和默认惰性的 lifecycle runner。PostgreSQL 结果事务按 Run→Attempt→dispatch 锁序完成 dispatch 更新、Run version CAS 与 RunEvent 追加;runtime 角色只取得新表的 SELECT/INSERT/UPDATE。 +`next` 已实现 profile-neutral canonical contract、`0005-run-cancellation-dispatch`、legacy Sequelize/SQLite adapter、PostgreSQL `pg-0066-cancellation-dispatch`/capability v65 adapter、lease expiry 接管、fencing、退避、结果事件、Dispatcher、有界 Supervisor 和默认惰性的 lifecycle runner。PostgreSQL 结果事务按 Run→Attempt→dispatch 锁序完成 dispatch 更新、Run version CAS 与 RunEvent 追加;runtime 角色只取得新表的 SELECT/INSERT/UPDATE。Cluster 生产交付已由 ADR-0457 接入既有 caller-driven Worker lease-control:Run-level stop 必须先结算 durable dispatch 再返回,Workflow Task timeout 则保持 `untracked`,不伪造父 Run 取消。 HTTP worker 已通过默认关闭的 manual-only manifest bootstrap 接入 Local Supervisor:只有 accepted 且全部 gate 通过时才启动,失败或 shutdown 时有界停止。以下工作仍未完成,因此它仍只允许显式 canary,不得扩大到默认生产流量: - 用户可见的运行指标、blocked 诊断和处置入口。 - 固定 edge 设备的数据库写放大、RSS、时延和磁盘基准。 -- cluster-control 对 PostgreSQL CancellationDispatch 的生产启动/停止拓扑与运维告警接线。 +- cluster-control 对 PostgreSQL CancellationDispatch 的用户可见 availability/blocked 指标、诊断与人工处置入口。 - 首次真实目标实例完整激活/回滚仪式与共享 config 多写者 authority。 ## 6. 验证门禁 @@ -174,3 +174,4 @@ HTTP worker 已通过默认关闭的 manual-only manifest bootstrap 接入 Local 10. Event 与日志不包含 handle、命令、环境和 Secret。 11. PostgreSQL 双连接只能产生一个 claim winner,raw token 不落库,数据库时间决定 lease/retry 到期。 12. v65 事实经 WAL 到达 standby,提升为新 Primary 后仍可读取;旧 owner/token/version 继续被 fencing。 +13. Cluster Worker lease-control 只在 durable dispatch 已结算或已重放时释放 Run-level stop;Workflow Task timeout 不写父 Run 取消事实。 diff --git a/docs/adr/ADR-0456-database-timed-postgresql-cancellation-dispatch.md b/docs/adr/ADR-0456-database-timed-postgresql-cancellation-dispatch.md index d801526c..5e6fc5b1 100644 --- a/docs/adr/ADR-0456-database-timed-postgresql-cancellation-dispatch.md +++ b/docs/adr/ADR-0456-database-timed-postgresql-cancellation-dispatch.md @@ -61,4 +61,4 @@ QingLong 3.0 还必须同时服务低配路由和集群节点。公共协议需 ## 后续 -下一阶段把 PostgreSQL CancellationDispatch Repository 接入 cluster-control 的明确生产 composition、单一 cadence、availability withdrawal、shutdown drain、指标与 blocked 处置面;随后补 CloudNativePG live failover、多副本压力、固定 x64/arm64 资源门。Local 侧如需消除 legacy raw-token 存量,另开兼容迁移 ADR,不与 Cluster rollout 混合。 +ADR-0457 已把 PostgreSQL CancellationDispatch Repository 接入既有 caller-driven Worker lease-control 生产 composition:它刻意不新增扫描 cadence,并复用 ingress drain 与数据库 shutdown 顺序。尚需用户可见的 availability/blocked 指标、诊断和人工处置面,以及 CloudNativePG live failover、多副本压力、固定 x64/arm64 资源门。Local 侧如需消除 legacy raw-token 存量,另开兼容迁移 ADR,不与 Cluster rollout 混合。 diff --git a/docs/adr/ADR-0457-worker-pull-cluster-cancellation-delivery.md b/docs/adr/ADR-0457-worker-pull-cluster-cancellation-delivery.md new file mode 100644 index 00000000..7602898f --- /dev/null +++ b/docs/adr/ADR-0457-worker-pull-cluster-cancellation-delivery.md @@ -0,0 +1,62 @@ +# ADR-0457:Worker 拉取路径上的 Cluster 取消交付 + +- 状态:Accepted +- 日期:2026-08-19 +- 关联 RFC:QL-RFC-0001 D-364、PR-5、PR-7 +- 关联 ADR:ADR-0005、ADR-0117、ADR-0238、ADR-0456 +- Amends:ADR-0005 的 Cluster 生产组合边界、ADR-0456 的后续拓扑描述 + +## 上下文 + +ADR-0456 已建立数据库计时、可接管且 capability 不落库的 PostgreSQL CancellationDispatch,但没有决定由哪个生产入口完成真实 Worker 停止交付。现有 Remote Worker 已通过认证 ingress 持有精确的 Session、RunDispatchLease、Attempt 和 lease fence,并以 caller-driven lease-control tick 获取续租或停止结果。另建 cluster-control 扫描 timer 会产生第二调度 authority;由 cluster-control 直接控制远端 PID 又会违反 ADR-0117 的 Worker 本机执行边界。 + +Run 取消与 Workflow Task timeout 还存在语义差异:前者具有 `Run.cancel_requested_at_ms`,应进入 durable CancellationDispatch;后者只终止当前 StepRun/Attempt,不得伪造父 Run 取消事实。 + +## 决策 + +1. `ClusterRemoteWorkerCancellationDispatchControl` 包装既有 `ClusterRemoteWorkerLeaseControlService`,而不新增扫描器。Worker 每次已认证的 lease-control 请求仍是唯一触发入口。 +2. `renewed`、`terminal` 等非停止结果完全绕过 CancellationDispatch。只有既有 lease-control 已产生精确 `stop_requested` 时,包装层才以同一 Run/Attempt 和停止时间 claim PostgreSQL CancellationDispatch。 +3. `claimed` 必须先以 `termination_requested` 原子结算 dispatch 与 `run.cancel_dispatched` 事件,验证 durable 结果后才向 Worker 释放原 `stop_requested`。Worker 随后在本机复验 durable handle 并执行停止;cluster-control 不接触 PID、进程组或本地 journal。 +4. `dispatched` 是可重放成功,直接释放相同停止结果;`leased` 与 `not_due` 表示另一副本仍拥有交付权,当前请求失败关闭并撤回可用性;`blocked` 同样失败关闭并进入低敏错误观察面。 +5. `not_eligible` 保留原停止结果,但仅记录 `untracked`。这是 Workflow Task timeout 或终态竞态的受审路径:不得为了统一表象写入不存在的 Run 取消意图。 +6. dispatch owner 复用 cluster-control recovery runtime 的稳定 replica owner ID。lease/event capability 默认由 CSPRNG 生成;生产回调只发布固定状态、错误 code 和 scope,不包含 Run、Worker、Attempt、token、错误原文或数据库细节。观察与诊断回调不是 authority,失败不得改变控制结果。 +7. 不新增 lifecycle。Worker ingress 已在 cluster-control shutdown 中先停止接收并 drain in-flight 请求,随后既有 application runtime 才停止 scheduler/recovery 并关闭数据库;包装层不拥有 timer、listener、queue、connection 或后台 Promise。 +8. 包能力只从 `@qinglong/cluster-control/cancellation-dispatch-control` 显式子路径发布;不扩大 runtime-core 根入口、不新增 workspace package 或生产依赖。 + +## 被拒绝的替代方案 + +### 新建 CancellationDispatch 扫描 cadence + +拒绝。它会与 Worker lease-control 形成双触发面,无法证明扫描 owner 与实际持有 RunDispatchLease 的 Worker 是同一执行权,并增加 timer、分页游标和 shutdown 协调成本。 + +### cluster-control 直接终止远端进程 + +拒绝。远端 durable handle、PID namespace 和本机进程身份只对 Worker 可验证。控制面只能交付停止意图,不能越过 Worker 执行边界。 + +### Workflow Task timeout 伪装成 Run cancellation + +拒绝。它会错误终止父 Workflow,并污染 Run 事件与取消审计。`not_eligible + untracked` 是有意的语义分支,不是漏记。 + +### 先返回停止,再异步补记 dispatch + +拒绝。进程或节点在响应后崩溃会留下不可证明的副作用;durable settlement 必须先于停止响应。 + +## 资源、安全与部署影响 + +- Edge/Standalone 闭包不变化,也不引入 `pg`;Cluster 复用现有连接池、HTTP/mTLS ingress 和 Worker cadence。 +- Cluster 每次 Run-level 停止最多增加既有 claim/result 短事务,无空闲扫描、常驻内存、端口、Kubernetes 对象或新连接。 +- 多副本 winner 仍由 PostgreSQL owner/token/version fence 决定;HTTP 重放只能得到 durable `dispatched`,不能生成第二个结果事件。 +- 数据库不可用、外国 live lease、retry 未到期或 blocked 时不释放停止响应,避免把未记账副作用表述为已交付。 + +## 验证 + +- 新增包装层契约 `8/8`,覆盖续租旁路、settle-before-stop、重放、foreign lease、blocked、Workflow timeout、结果失败与配置失败。 +- cluster-control 完整包 `269 pass / 0 fail / 2 conditional skip`;生产 process 回调只输出低敏枚举和稳定错误 code。 +- 完整 backend `1,487 pass / 0 fail / 2 conditional skip`(总计 1,489);18-package clean/build 与顺序测试单次退出 0。 +- package boundary、Edge import、cluster dependency、cluster deployment 与 service-manager bridge import 审计均通过;workspace package 仍为 18,`cluster-control` 新文件位于 `remote-execution` 子域,根目录计数不变。 +- `14/14` Local Profile artifact audit 通过;基础 Edge/Standalone 保持 `2,589,998 / 2,590,076` bytes,闭包不含 `pg` 或 Cluster package。 +- PostgreSQL 18.6 arm64 HA 门 `144/144`:原双连接单 claim、租约接管、stale fence、retry due、WAL/promotion 证据保留,并新增真实 cluster-control 包装层 `termination_requested` settle-before-stop;timeline `1→2`,报告 SHA-256 为 `4313b405c2ea56a3d44bc4907d5299b6e5c0d3062bb9c9f1522749d1021bd462`。 + +## 后续 + +仍需用户可见的 blocked/availability 指标、诊断与人工处置入口,以及 CloudNativePG live failover、多副本容量压力、固定 Linux x64/arm64 与物理 Edge 资源门。Local legacy raw-token 存量迁移继续保持独立议题。 diff --git a/packages/ql3-cluster-control/package.json b/packages/ql3-cluster-control/package.json index c953ca54..527476bd 100644 --- a/packages/ql3-cluster-control/package.json +++ b/packages/ql3-cluster-control/package.json @@ -175,6 +175,11 @@ "require": "./dist/remote-execution/remoteWorkerLeaseControlService.js", "default": "./dist/remote-execution/remoteWorkerLeaseControlService.js" }, + "./cancellation-dispatch-control": { + "types": "./dist/remote-execution/remoteWorkerCancellationDispatchControl.d.ts", + "require": "./dist/remote-execution/remoteWorkerCancellationDispatchControl.js", + "default": "./dist/remote-execution/remoteWorkerCancellationDispatchControl.js" + }, "./workflow-scheduler": { "types": "./dist/scheduling/workflowScheduler.d.ts", "require": "./dist/scheduling/workflowScheduler.js", diff --git a/packages/ql3-cluster-control/src/application-runtime/clusterControlRuntime.ts b/packages/ql3-cluster-control/src/application-runtime/clusterControlRuntime.ts index 1b81e316..8f72fd30 100644 --- a/packages/ql3-cluster-control/src/application-runtime/clusterControlRuntime.ts +++ b/packages/ql3-cluster-control/src/application-runtime/clusterControlRuntime.ts @@ -903,6 +903,9 @@ export async function bootstrapClusterControlRuntime( workerRuntime: createClusterWorkerRuntimePort( database.pool, options.workerRuntime, + { + cancellationDispatchOwnerId: recoveryRuntime.ownerId, + }, ), }), }); diff --git a/packages/ql3-cluster-control/src/application-runtime/productionApplication.ts b/packages/ql3-cluster-control/src/application-runtime/productionApplication.ts index 9085e203..58188280 100644 --- a/packages/ql3-cluster-control/src/application-runtime/productionApplication.ts +++ b/packages/ql3-cluster-control/src/application-runtime/productionApplication.ts @@ -75,6 +75,7 @@ import { createClusterControlCopilotFailureDiagnosisCancellationRoute, type ClusterCopilotFailureDiagnosisCancellationCapability, } from '../copilot/failure-diagnosis/failureDiagnosisCancellationRoute'; +import type { ClusterRemoteWorkerCancellationDispatchObservation } from '../remote-execution/remoteWorkerCancellationDispatchControl'; export const PRODUCTION_CLUSTER_CONTROL_ROUTE_OPERATIONS = Object.freeze([ 'task.get', @@ -150,6 +151,12 @@ export interface ProductionClusterWorkerIngressOptions { readonly artifactStore: ClusterRemoteWorkerArtifactStore; readonly secretProvider?: RemoteWorkerSecretValueProvider; readonly onDiagnostic?: (error: unknown) => void | Promise; + readonly onCancellationDispatch?: ( + observation: ClusterRemoteWorkerCancellationDispatchObservation, + ) => void | Promise; + readonly onCancellationDispatchDiagnostic?: ( + error: unknown, + ) => void | Promise; } export interface ProductionClusterControlApplicationOptions @@ -439,6 +446,28 @@ export function startProductionClusterControlApplication( ...(workerIngress.secretProvider === undefined ? {} : { secretProvider: workerIngress.secretProvider }), + ...( + workerIngress.onCancellationDispatch === undefined && + workerIngress.onCancellationDispatchDiagnostic === undefined + ? {} + : { + cancellationDispatch: { + ...(workerIngress.onCancellationDispatch === undefined + ? {} + : { + onObservation: + workerIngress.onCancellationDispatch, + }), + ...(workerIngress.onCancellationDispatchDiagnostic === + undefined + ? {} + : { + onDiagnostic: + workerIngress.onCancellationDispatchDiagnostic, + }), + }, + } + ), }, }), ...database, diff --git a/packages/ql3-cluster-control/src/production-process/processApplication.ts b/packages/ql3-cluster-control/src/production-process/processApplication.ts index fd372136..6cc2b58f 100644 --- a/packages/ql3-cluster-control/src/production-process/processApplication.ts +++ b/packages/ql3-cluster-control/src/production-process/processApplication.ts @@ -21,6 +21,7 @@ import { } from '../worker-ingress/workerIngressConfig'; import type { ClusterWorkerArtifactBinding } from '../artifact/workerArtifactBinding'; import type { RemoteWorkerSecretValueProvider } from '@qinglong/runtime-core/remote-secret-delivery'; +import type { ClusterRemoteWorkerCancellationDispatchObservation } from '../remote-execution/remoteWorkerCancellationDispatchControl'; export type ClusterControlProcessSignal = 'SIGINT' | 'SIGTERM'; @@ -34,10 +35,12 @@ export interface ClusterControlProcessEvent { readonly stopResult?: ClusterControlStopResult; readonly address?: Readonly<{ host: string; port: number }>; readonly activation?: ClusterControlActivationAudit; + readonly cancellationDispatch?: ClusterRemoteWorkerCancellationDispatchObservation; readonly diagnostic?: Readonly<{ scope: | 'scheduler' | 'cancellation-convergence' + | 'cancellation-dispatch' | 'log-retention' | 'database' | 'worker-ingress'; @@ -333,6 +336,32 @@ export async function runProductionClusterControlProcess( ), ).catch(() => undefined); }, + onCancellationDispatch(observation) { + void Promise.resolve( + options.emit( + event(replicaId, { + level: + observation.status === 'blocked' ? 'error' : 'info', + event: 'cancellation_dispatch', + cancellationDispatch: observation, + }), + ), + ).catch(() => undefined); + }, + onCancellationDispatchDiagnostic(error: unknown) { + void Promise.resolve( + options.emit( + event(replicaId, { + level: 'error', + event: 'runtime_diagnostic', + diagnostic: diagnosticFact( + 'cancellation-dispatch', + error, + ), + }), + ), + ).catch(() => undefined); + }, }, }), audit(record) { diff --git a/packages/ql3-cluster-control/src/remote-execution/remoteWorkerCancellationDispatchControl.ts b/packages/ql3-cluster-control/src/remote-execution/remoteWorkerCancellationDispatchControl.ts new file mode 100644 index 00000000..e1be8e04 --- /dev/null +++ b/packages/ql3-cluster-control/src/remote-execution/remoteWorkerCancellationDispatchControl.ts @@ -0,0 +1,238 @@ +// Remote execution owns cancellation delivery to the Worker that already holds +// the exact RunDispatchLease. This layer adds no timer, queue, or connection. +import { randomUUID } from 'node:crypto'; +import { + MAX_CANCELLATION_DISPATCH_LEASE_MS, + type CancellationDispatchRepository, +} from '@qinglong/runtime-core/cancellation-dispatch'; +import { + RemoteWorkerLeaseControlUnavailableError, + type RemoteWorkerLeaseControlCommand, + type RemoteWorkerLeaseControlResult, +} from '@qinglong/runtime-core/remote-worker-lease-control'; + +export type ClusterRemoteWorkerCancellationDispatchObservation = Readonly<{ + readonly status: + | 'dispatched' + | 'already_dispatched' + | 'untracked' + | 'deferred' + | 'blocked'; +}>; + +export interface ClusterRemoteWorkerCancellationDispatchControlOptions { + readonly ownerId: string; + readonly leaseDurationMs?: number; + readonly createLeaseToken?: () => string; + readonly createEventId?: () => string; + readonly onObservation?: ( + observation: ClusterRemoteWorkerCancellationDispatchObservation, + ) => void | Promise; + readonly onDiagnostic?: (error: unknown) => void | Promise; +} + +export class ClusterRemoteWorkerCancellationDispatchError extends Error { + readonly code = 'CLUSTER_REMOTE_CANCELLATION_DISPATCH_FAILED'; + + constructor( + readonly reason: + | 'invalid_configuration' + | 'claim_failed' + | 'result_failed' + | 'delivery_deferred' + | 'delivery_blocked', + options?: ErrorOptions, + ) { + super(`Cluster Remote Worker cancellation dispatch failed: ${reason}`, options); + this.name = 'ClusterRemoteWorkerCancellationDispatchError'; + } +} + +const OWNER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/u; +const OPTION_KEYS = new Set([ + 'createEventId', + 'createLeaseToken', + 'leaseDurationMs', + 'onDiagnostic', + 'onObservation', + 'ownerId', +]); + +function invalidConfiguration(): never { + throw new ClusterRemoteWorkerCancellationDispatchError( + 'invalid_configuration', + ); +} + +function capability(factory: () => string, name: string): string { + let value: unknown; + try { + value = factory(); + } catch (error) { + throw new ClusterRemoteWorkerCancellationDispatchError( + 'claim_failed', + { cause: error }, + ); + } + const maximum = name === 'eventId' ? 36 : 128; + if ( + typeof value !== 'string' || + value.length < 1 || + value.length > maximum || + /[\u0000-\u001f\u007f]/u.test(value) + ) { + throw new ClusterRemoteWorkerCancellationDispatchError('claim_failed'); + } + return value; +} + +/** + * Converts the existing caller-driven Worker lease-control tick into the only + * Cluster cancellation delivery path. A stop response is released only after + * its durable CancellationDispatch is settled, while Workflow-scoped timeout + * stops remain valid without forging a Run cancellation record. + */ +export class ClusterRemoteWorkerCancellationDispatchControl { + private readonly ownerId: string; + private readonly leaseDurationMs: number; + private readonly createLeaseToken: () => string; + private readonly createEventId: () => string; + private readonly onObservation?: ClusterRemoteWorkerCancellationDispatchControlOptions['onObservation']; + private readonly onDiagnostic?: ClusterRemoteWorkerCancellationDispatchControlOptions['onDiagnostic']; + + constructor( + private readonly leaseControl: Readonly<{ + control( + command: RemoteWorkerLeaseControlCommand, + ): Promise>; + }>, + private readonly dispatches: CancellationDispatchRepository, + options: ClusterRemoteWorkerCancellationDispatchControlOptions, + ) { + if ( + typeof leaseControl?.control !== 'function' || + typeof dispatches?.claim !== 'function' || + typeof dispatches?.recordResult !== 'function' || + !options || + typeof options !== 'object' || + Array.isArray(options) || + Object.keys(options).some((key) => !OPTION_KEYS.has(key)) || + !OWNER_PATTERN.test(options.ownerId ?? '') || + (options.leaseDurationMs !== undefined && + (!Number.isSafeInteger(options.leaseDurationMs) || + options.leaseDurationMs < 1 || + options.leaseDurationMs > MAX_CANCELLATION_DISPATCH_LEASE_MS)) || + (options.createLeaseToken !== undefined && + typeof options.createLeaseToken !== 'function') || + (options.createEventId !== undefined && + typeof options.createEventId !== 'function') || + (options.onObservation !== undefined && + typeof options.onObservation !== 'function') || + (options.onDiagnostic !== undefined && + typeof options.onDiagnostic !== 'function') + ) { + invalidConfiguration(); + } + this.ownerId = options.ownerId; + this.leaseDurationMs = options.leaseDurationMs ?? 30_000; + this.createLeaseToken = options.createLeaseToken ?? randomUUID; + this.createEventId = options.createEventId ?? randomUUID; + this.onObservation = options.onObservation; + this.onDiagnostic = options.onDiagnostic; + } + + async control( + command: RemoteWorkerLeaseControlCommand, + ): Promise> { + const result = await this.leaseControl.control(command); + if (result.status !== 'stop_requested') return result; + + let claim: Awaited>; + try { + claim = await this.dispatches.claim({ + runId: result.runId, + attemptId: result.attemptId, + requestedAtMs: result.stop!.requestedAtMs, + owner: this.ownerId, + leaseToken: capability(this.createLeaseToken, 'leaseToken'), + leaseDurationMs: this.leaseDurationMs, + }); + } catch (error) { + return this.unavailable('claim_failed', error); + } + + if (claim.status === 'not_eligible') { + // Workflow Task timeout is represented by its own event and does not set + // Run.cancel_requested_at_ms. The already-fenced Worker stop must remain + // deliverable without inventing a Run-level cancellation fact. + this.observe('untracked'); + return result; + } + if (claim.status === 'dispatched') { + this.observe('already_dispatched'); + return result; + } + if (claim.status === 'leased' || claim.status === 'not_due') { + this.observe('deferred'); + return this.unavailable('delivery_deferred'); + } + if (claim.status === 'blocked') { + this.observe('blocked'); + return this.unavailable('delivery_blocked'); + } + if (claim.status !== 'claimed') { + return this.unavailable('claim_failed'); + } + + try { + const settled = await this.dispatches.recordResult({ + runId: result.runId, + attemptId: result.attemptId, + owner: this.ownerId, + leaseToken: claim.leaseToken, + expectedVersion: claim.dispatch.version, + result: 'termination_requested', + eventId: capability(this.createEventId, 'eventId'), + }); + if ( + settled.dispatch.status !== 'dispatched' || + settled.dispatch.lastResult !== 'termination_requested' || + settled.event.type !== 'run.cancel_dispatched' + ) { + return this.unavailable('result_failed'); + } + } catch (error) { + return this.unavailable('result_failed', error); + } + this.observe('dispatched'); + return result; + } + + private unavailable( + reason: Exclude< + ClusterRemoteWorkerCancellationDispatchError['reason'], + 'invalid_configuration' + >, + cause?: unknown, + ): never { + const error = new ClusterRemoteWorkerCancellationDispatchError(reason, { + ...(cause === undefined ? {} : { cause }), + }); + this.diagnostic(error); + throw new RemoteWorkerLeaseControlUnavailableError({ cause: error }); + } + + private observe( + status: ClusterRemoteWorkerCancellationDispatchObservation['status'], + ): void { + if (!this.onObservation) return; + void Promise.resolve( + this.onObservation(Object.freeze({ status })), + ).catch(() => undefined); + } + + private diagnostic(error: unknown): void { + if (!this.onDiagnostic) return; + void Promise.resolve(this.onDiagnostic(error)).catch(() => undefined); + } +} diff --git a/packages/ql3-cluster-control/src/remote-execution/workerRuntimePort.ts b/packages/ql3-cluster-control/src/remote-execution/workerRuntimePort.ts index 1875d865..f810b574 100644 --- a/packages/ql3-cluster-control/src/remote-execution/workerRuntimePort.ts +++ b/packages/ql3-cluster-control/src/remote-execution/workerRuntimePort.ts @@ -4,6 +4,7 @@ import type { RemoteWorkerSecretValueProvider } from '@qinglong/runtime-core/rem import type { RunAttemptLogRangeReader } from '@qinglong/runtime-core/run-attempt-log-read'; import { PostgresClusterDispatchSource, + PostgresCancellationDispatchRepository, PostgresRemoteRunActivationRepository, PostgresRemoteWorkerCompletionRepository, PostgresRemoteWorkerLeaseControlRepository, @@ -21,11 +22,23 @@ import { type ClusterRemoteWorkerArtifactStore, } from './remoteWorkerCompletionService'; import { ClusterRemoteWorkerLeaseControlService } from './remoteWorkerLeaseControlService'; +import { + ClusterRemoteWorkerCancellationDispatchControl, + type ClusterRemoteWorkerCancellationDispatchControlOptions, +} from './remoteWorkerCancellationDispatchControl'; import type { WorkerIngressPipelineOptions } from '../worker-ingress/workerIngressPipeline'; export interface ClusterWorkerRuntimeDependencies { readonly artifactStore: ClusterRemoteWorkerArtifactStore; readonly secretProvider?: RemoteWorkerSecretValueProvider; + readonly cancellationDispatch?: Readonly<{ + readonly onObservation?: ClusterRemoteWorkerCancellationDispatchControlOptions['onObservation']; + readonly onDiagnostic?: ClusterRemoteWorkerCancellationDispatchControlOptions['onDiagnostic']; + }>; +} + +export interface ClusterWorkerRuntimePortOptions { + readonly cancellationDispatchOwnerId: string; } /** @@ -48,6 +61,7 @@ export interface ClusterWorkerRuntimePort { export function createClusterWorkerRuntimePort( pool: PostgresPool, dependencies: ClusterWorkerRuntimeDependencies, + options: ClusterWorkerRuntimePortOptions, ): Readonly { if (!pool || typeof pool.query !== 'function') { throw new TypeError('Cluster Worker runtime Pool is invalid'); @@ -59,6 +73,15 @@ export function createClusterWorkerRuntimePort( ) { throw new TypeError('Cluster Worker runtime dependencies are invalid'); } + if ( + !options || + typeof options !== 'object' || + Array.isArray(options) || + Object.keys(options).length !== 1 || + typeof options.cancellationDispatchOwnerId !== 'string' + ) { + throw new TypeError('Cluster Worker runtime options are invalid'); + } const workerSessions = new PostgresWorkerSessionRepository(pool); const completionRepository = new PostgresRemoteWorkerCompletionRepository( @@ -92,8 +115,25 @@ export function createClusterWorkerRuntimePort( completionRepository, dependencies.artifactStore, ), - leaseControl: new ClusterRemoteWorkerLeaseControlService( - new PostgresRemoteWorkerLeaseControlRepository(pool), + leaseControl: new ClusterRemoteWorkerCancellationDispatchControl( + new ClusterRemoteWorkerLeaseControlService( + new PostgresRemoteWorkerLeaseControlRepository(pool), + ), + new PostgresCancellationDispatchRepository(pool), + { + ownerId: options.cancellationDispatchOwnerId, + ...(dependencies.cancellationDispatch?.onObservation === undefined + ? {} + : { + onObservation: + dependencies.cancellationDispatch.onObservation, + }), + ...(dependencies.cancellationDispatch?.onDiagnostic === undefined + ? {} + : { + onDiagnostic: dependencies.cancellationDispatch.onDiagnostic, + }), + }, ), ...(readLogRange === undefined ? {} diff --git a/packages/ql3-cluster-control/test/processApplication.test.cjs b/packages/ql3-cluster-control/test/processApplication.test.cjs index ac070ad9..420b4044 100644 --- a/packages/ql3-cluster-control/test/processApplication.test.cjs +++ b/packages/ql3-cluster-control/test/processApplication.test.cjs @@ -206,6 +206,12 @@ test('starts the optional Worker listener and closes its lazy Artifact binding', code: 'S3Unavailable', }), ); + options.workerIngress.onCancellationDispatch({ status: 'dispatched' }); + options.workerIngress.onCancellationDispatchDiagnostic( + Object.assign(new Error('must-not-be-logged'), { + code: 'CANCEL_DISPATCH_UNAVAILABLE', + }), + ); return { status: 'active', address: { host: '0.0.0.0', port: 5800 }, @@ -248,6 +254,25 @@ test('starts the optional Worker listener and closes its lazy Artifact binding', ), true, ); + assert.equal( + facts.some( + (fact) => + fact.event === 'cancellation_dispatch' && + fact.level === 'info' && + fact.cancellationDispatch.status === 'dispatched', + ), + true, + ); + assert.equal( + facts.some( + (fact) => + fact.event === 'runtime_diagnostic' && + fact.diagnostic.scope === 'cancellation-dispatch' && + fact.diagnostic.code === 'CANCEL_DISPATCH_UNAVAILABLE' && + JSON.stringify(fact).includes('must-not-be-logged') === false, + ), + true, + ); }); test('creates the configured mounted Secret provider before Worker activation', async () => { diff --git a/packages/ql3-cluster-control/test/remoteWorkerCancellationDispatchControl.test.cjs b/packages/ql3-cluster-control/test/remoteWorkerCancellationDispatchControl.test.cjs new file mode 100644 index 00000000..be375cfa --- /dev/null +++ b/packages/ql3-cluster-control/test/remoteWorkerCancellationDispatchControl.test.cjs @@ -0,0 +1,293 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const { test } = require('node:test'); +const { + ClusterRemoteWorkerCancellationDispatchControl, + ClusterRemoteWorkerCancellationDispatchError, +} = require('@qinglong/cluster-control/cancellation-dispatch-control'); +const { + RemoteWorkerLeaseControlUnavailableError, +} = require('@qinglong/runtime-core/remote-worker-lease-control'); + +const COMMAND = Object.freeze({ + workerId: 'worker-1', + workerSessionId: '018f0000-0000-7000-8000-000000000001', + workerGeneration: 2, + projectId: 'project-1', + runId: 'run-1', + attemptId: 'attempt-1', + offerId: 'offer-1', + leaseGeneration: 3, + leaseToken: 'worker_generated_lease_capability_0000000000000001', + expectedLeaseVersion: 4, +}); +const STOP = Object.freeze({ + status: 'stop_requested', + projectId: 'project-1', + runId: 'run-1', + attemptId: 'attempt-1', + offerId: 'offer-1', + leaseGeneration: 3, + leaseVersion: 5, + renewedAtMs: 10_000, + expiresAtMs: 40_000, + stop: Object.freeze({ reason: 'user', requestedAtMs: 9_000 }), +}); + +function leasedDispatch() { + return Object.freeze({ + runId: 'run-1', + attemptId: 'attempt-1', + status: 'leased', + version: 1, + dispatchCount: 1, + leaseOwner: 'replica-1', + leaseTokenDigest: 'a'.repeat(64), + leaseExpiresAtMs: 40_000, + createdAtMs: 10_000, + updatedAtMs: 10_000, + }); +} + +function service(dispatches, overrides = {}) { + return new ClusterRemoteWorkerCancellationDispatchControl( + { + async control() { + return overrides.result ?? STOP; + }, + }, + dispatches, + { + ownerId: 'replica-1', + leaseDurationMs: 30_000, + createLeaseToken: () => 'cancel-token-1', + createEventId: () => '018f0000-0000-7000-8000-000000000011', + ...(overrides.onObservation === undefined + ? {} + : { onObservation: overrides.onObservation }), + ...(overrides.onDiagnostic === undefined + ? {} + : { onDiagnostic: overrides.onDiagnostic }), + }, + ); +} + +test('bypasses dispatch storage when lease control only renews', async () => { + let calls = 0; + const renewed = Object.freeze({ + ...STOP, + status: 'renewed', + stop: undefined, + }); + const control = service( + { + async claim() { + calls += 1; + throw new Error('must not claim'); + }, + async recordResult() { + calls += 1; + throw new Error('must not record'); + }, + }, + { result: renewed }, + ); + assert.equal(await control.control(COMMAND), renewed); + assert.equal(calls, 0); +}); + +test('settles one durable dispatch before releasing a Worker stop', async () => { + const observed = []; + let claimCommand; + let resultCommand; + const claimed = leasedDispatch(); + const control = service( + { + async claim(value) { + claimCommand = value; + return { status: 'claimed', dispatch: claimed, leaseToken: 'cancel-token-1' }; + }, + async recordResult(value) { + resultCommand = value; + return { + dispatch: { + ...claimed, + status: 'dispatched', + version: 2, + leaseOwner: undefined, + leaseTokenDigest: undefined, + leaseExpiresAtMs: undefined, + lastResult: 'termination_requested', + lastDispatchedAtMs: 10_001, + updatedAtMs: 10_001, + }, + event: { type: 'run.cancel_dispatched' }, + }; + }, + }, + { onObservation: (value) => observed.push(value) }, + ); + + assert.equal(await control.control(COMMAND), STOP); + assert.deepEqual(claimCommand, { + runId: 'run-1', + attemptId: 'attempt-1', + requestedAtMs: 9_000, + owner: 'replica-1', + leaseToken: 'cancel-token-1', + leaseDurationMs: 30_000, + }); + assert.deepEqual(resultCommand, { + runId: 'run-1', + attemptId: 'attempt-1', + owner: 'replica-1', + leaseToken: 'cancel-token-1', + expectedVersion: 1, + result: 'termination_requested', + eventId: '018f0000-0000-7000-8000-000000000011', + }); + assert.deepEqual(observed, [{ status: 'dispatched' }]); +}); + +test('releases an already-dispatched stop without a second result event', async () => { + let results = 0; + const observed = []; + const control = service( + { + async claim() { + return { + status: 'dispatched', + dispatch: { ...leasedDispatch(), status: 'dispatched' }, + }; + }, + async recordResult() { + results += 1; + throw new Error('must not record'); + }, + }, + { onObservation: (value) => observed.push(value) }, + ); + assert.equal(await control.control(COMMAND), STOP); + assert.equal(results, 0); + assert.deepEqual(observed, [{ status: 'already_dispatched' }]); +}); + +test('keeps a foreign live dispatch from releasing a duplicate stop', async () => { + const diagnostics = []; + const observed = []; + const control = service( + { + async claim() { + return { status: 'leased', dispatch: leasedDispatch() }; + }, + async recordResult() { + throw new Error('must not record'); + }, + }, + { + onObservation: (value) => observed.push(value), + onDiagnostic: (error) => diagnostics.push(error), + }, + ); + await assert.rejects( + control.control(COMMAND), + (error) => + error instanceof RemoteWorkerLeaseControlUnavailableError && + error.cause instanceof ClusterRemoteWorkerCancellationDispatchError && + error.cause.reason === 'delivery_deferred', + ); + assert.deepEqual(observed, [{ status: 'deferred' }]); + assert.equal(diagnostics[0].reason, 'delivery_deferred'); +}); + +test('fails closed and reports a durable blocked dispatch', async () => { + const diagnostics = []; + const observed = []; + const control = service( + { + async claim() { + return { status: 'blocked', dispatch: leasedDispatch() }; + }, + async recordResult() { + throw new Error('must not record'); + }, + }, + { + onObservation: (value) => observed.push(value), + onDiagnostic: (error) => diagnostics.push(error), + }, + ); + await assert.rejects( + control.control(COMMAND), + (error) => error.cause?.reason === 'delivery_blocked', + ); + assert.deepEqual(observed, [{ status: 'blocked' }]); + assert.equal(diagnostics[0].code, 'CLUSTER_REMOTE_CANCELLATION_DISPATCH_FAILED'); +}); + +test('preserves Workflow-scoped timeout stops without forging Run cancellation', async () => { + let results = 0; + const observed = []; + const control = service( + { + async claim() { + return { status: 'not_eligible' }; + }, + async recordResult() { + results += 1; + }, + }, + { onObservation: (value) => observed.push(value) }, + ); + assert.equal(await control.control(COMMAND), STOP); + assert.equal(results, 0); + assert.deepEqual(observed, [{ status: 'untracked' }]); +}); + +test('does not release a stop when durable result settlement fails', async () => { + const diagnostics = []; + const control = service( + { + async claim() { + return { + status: 'claimed', + dispatch: leasedDispatch(), + leaseToken: 'cancel-token-1', + }; + }, + async recordResult() { + throw new Error('database unavailable'); + }, + }, + { onDiagnostic: (error) => diagnostics.push(error) }, + ); + await assert.rejects( + control.control(COMMAND), + (error) => error.cause?.reason === 'result_failed', + ); + assert.equal(diagnostics[0].reason, 'result_failed'); +}); + +test('rejects widened or unbounded production configuration', () => { + const repository = { claim() {}, recordResult() {} }; + const leaseControl = { control() {} }; + assert.throws( + () => + new ClusterRemoteWorkerCancellationDispatchControl( + leaseControl, + repository, + { ownerId: '', extra: true }, + ), + /invalid_configuration/, + ); + assert.throws( + () => + new ClusterRemoteWorkerCancellationDispatchControl( + leaseControl, + repository, + { ownerId: 'replica-1', leaseDurationMs: 0 }, + ), + /invalid_configuration/, + ); +}); diff --git a/scripts/ql3-postgres-ha-cancellation-dispatch-fixture.cjs b/scripts/ql3-postgres-ha-cancellation-dispatch-fixture.cjs index 52cbe376..0f41f9b4 100644 --- a/scripts/ql3-postgres-ha-cancellation-dispatch-fixture.cjs +++ b/scripts/ql3-postgres-ha-cancellation-dispatch-fixture.cjs @@ -8,6 +8,9 @@ const { CancellationDispatchFenceRejectedError, digestCancellationDispatchLeaseToken, } = require('../packages/ql3-runtime-core/dist/run/cancellation-dispatch/cancellationDispatch.js'); +const { + ClusterRemoteWorkerCancellationDispatchControl, +} = require('../packages/ql3-cluster-control/dist/remote-execution/remoteWorkerCancellationDispatchControl.js'); const FIXTURE = Object.freeze({ runId: 'ha-cancel-run-d363', @@ -171,24 +174,35 @@ async function persistCancellationDispatchHaFixture(options) { SET next_attempt_at_ms = 0 WHERE run_id = $1`, [FIXTURE.runId], ); - const finalLease = await first.claim({ - ...candidate, - owner: 'ha-cancel-final', - leaseToken: 'ha-cancel-final-token', - }); - assert.equal(finalLease.status, 'claimed'); - assert.equal(finalLease.dispatch.dispatchCount, 3); - const terminal = await first.recordResult({ + const stopRequested = Object.freeze({ + status: 'stop_requested', + projectId: 'default', runId: FIXTURE.runId, attemptId: FIXTURE.attemptId, - owner: 'ha-cancel-final', - leaseToken: 'ha-cancel-final-token', - expectedVersion: finalLease.dispatch.version, - result: 'already_exited', - eventId: FIXTURE.terminalEventId, + offerId: 'ha-cancel-offer-d364', + leaseGeneration: 1, + leaseVersion: 2, + renewedAtMs: FIXTURE.requestedAtMs, + expiresAtMs: FIXTURE.requestedAtMs + 30_000, + stop: Object.freeze({ + reason: 'user', + requestedAtMs: FIXTURE.requestedAtMs, + }), }); - assert.equal(terminal.dispatch.status, 'dispatched'); - assert.equal(terminal.event.sequence, 2); + const deliveryObservations = []; + const delivery = new ClusterRemoteWorkerCancellationDispatchControl( + { async control() { return stopRequested; } }, + first, + { + ownerId: 'ha-cancel-final', + leaseDurationMs: 30_000, + createLeaseToken: () => 'ha-cancel-final-token', + createEventId: () => FIXTURE.terminalEventId, + onObservation: (observation) => deliveryObservations.push(observation), + }, + ); + assert.equal(await delivery.control({}), stopRequested); + assert.deepEqual(deliveryObservations, [{ status: 'dispatched' }]); await migrationPool.query( `WITH observed AS ( SELECT floor(extract(epoch FROM transaction_timestamp()) * 1000)::bigint @@ -225,7 +239,7 @@ async function persistCancellationDispatchHaFixture(options) { version: 5, dispatchCount: 3, leaseTokenDigest: null, - lastResult: 'already_exited', + lastResult: 'termination_requested', runVersion: 5, eventSequence: 3, eventCount: 3, @@ -239,6 +253,7 @@ async function persistCancellationDispatchHaFixture(options) { expiredLeaseTakenOver: true, staleLeaseFenced: true, retryDeferredUntilDue: true, + productionDeliverySettledBeforeStop: true, replicatedBeforePromotion: false, survivedPromotion: false, }; diff --git a/test/back/ql3PackageBoundaryAudit.test.cjs b/test/back/ql3PackageBoundaryAudit.test.cjs index ee542dcc..6feca1ea 100644 --- a/test/back/ql3PackageBoundaryAudit.test.cjs +++ b/test/back/ql3PackageBoundaryAudit.test.cjs @@ -385,10 +385,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: clusterControl.rootSourceFileRoles, }, { - sourceFiles: 64, + sourceFiles: 65, rootSourceFiles: 2, rootSourceLines: 195, - nestedSourceFiles: 62, + nestedSourceFiles: 63, rootSourceFileRoles: { 'aiCli.ts': 'binary_entry', 'cli.ts': 'binary_entry',