mirror of
https://github.com/whyour/qinglong.git
synced 2026-09-23 03:18:09 +08:00
feat(ql3): add fenced copilot diagnosis cancellation
This commit is contained in:
whyour
@@ -11,6 +11,7 @@
|
|||||||
|
|
||||||
最新增量证据（2026-08-16）：
|
最新增量证据（2026-08-16）：
|
||||||
|
|
||||||
|
- D-323/ADR-0415（已接受）：Cluster Copilot failure diagnosis 已补齐受围栏的 request-keyed 取消 mutation。`POST /api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses/{requestId}/cancellation` 复用 exact `qinglong/run-cancellation@v1` body、`run.stop` 当前 Policy、durable audit 与通用 PostgreSQL Run cancellation transaction；调用方只能提供 `mutationId`，diagnosis Run、Event、reason、Provider 与终态均由服务端从 admission plan/receipt 和 durable authority 派生。pre-Model 取消立即与 Model start 通过 Run/Step/version fence 竞争并原子 terminalize；Model start 获胜时只返回 `model_in_flight` durable intent，不伪造 Provider abort、终态、usage 或 cost，真实 completion/finalization 仍可成为最终 winner。相同或不同 mutation 都不能追加第二个 intent/Event 或覆盖首次 reason/time。能力只注入默认关闭的 Cluster AI profile，复用既有 AI Pool、repository、Policy pipeline 与进程，不新增 package、schema、Pool、连接、timer、watcher、queue、cache、端口、Pod、Service 或 Kubernetes 权限，Edge/Standalone 与普通 Cluster Control 均无该 route。目录审计曾拒绝 `failure-diagnosis` 达到 12 个直属源码文件；最终删除仅做 re-export 的单文件 façade，让公开 package subpath 直接指向 `cancellation/service`，而不是再拆 package 或放宽阈值。workspace 保持 18 package，`singleSourcePackages=[]`、`shallowSourcePackages=[]`，AI 194 个源码中 193 个、Cluster Control 64 个源码中 62 个位于嵌套职责目录。AI 254 pass/3 条件 skip、Cluster Control 261 pass/2 条件 skip、18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail；四项架构审计和 14 档 Local artifact 全部通过。默认 Edge/Standalone 为 2,589,890/2,589,968 bytes，Edge/Standalone AI application 为 4,493,043/4,493,175 bytes，证明 Cluster-only mutation 未进入低配闭包。PostgreSQL 18.6 arm64 physical HA 142/142、timeline `1→2`，覆盖 intent/Event/terminal receipt 同步复制、promotion 后 exact replay 与 in-flight 不伪造终态；报告 SHA-256 为 `5dbcffb74a3181aabee66a8f68ecfa7a65e0491a6f2ba24e2bc903c83da9d766`，离线审计零 finding。下一独立 Gate 可选择 CLI/UI/MCP 客户端或带 Provider abort acknowledgement 的取消语义，不得在没有 durable acknowledgement 时宣称外部调用已停止。
|
||||||
- D-322/ADR-0414（已接受）：Cluster Copilot failure diagnosis 已补齐 request-keyed 产品读模型。`GET /api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses/{requestId}` 只需 `run.read`，投影 running/terminal、取消/失败 stage/reason、authoritative admission/finalization time 与 durable usage/settled cost；同路径 `/output` 独立要求 `artifact.read`，只在 admission plan/receipt、Project/source Run、finalization、invocation 和 encrypted Artifact 全部 exact binding 且 current Project Policy 再授权后解析 historical key。调用者不能提交 Artifact、diagnosis Run、Model invocation、provider/model、价格、usage、outcome 或 key identity；deny/approval/absent/cross-target 统一 404，存储/Policy/key/decrypt/cipher 冲突统一 503。写 `capability` 与 `readCapability` 接口隔离，production 复用既有 AI PostgreSQL Pool、repository、Project Policy 与 projected output keyring，不新增 package、进程、端口、Pool、timer、watcher、queue、cache、Pod、Service 或 Kubernetes 权限；普通 Cluster Control、Edge/Standalone 均无这些 route。workspace 保持 18 package、`singleSourcePackages=[]`、`shallowSourcePackages=[]`，AI 193 个源码中 192 个、Cluster Control 62 个源码中 60 个位于嵌套领域目录。AI 249 pass/3 条件 skip、Cluster Control 256 pass/2 条件 skip、18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，四项架构审计和 14 档 Local artifact 全部通过；默认 Edge/Standalone 仍为 2,589,890/2,589,968 bytes，Edge/Standalone AI 为 3,069,143/3,069,233 bytes。PostgreSQL 18.6 arm64 physical HA 139/139、timeline `1→2`，成功密文输出在 standby 可读，提升后 request-keyed exact replay 且 provider 调用为零；报告 SHA-256 为 `22decb54cfb8735bf787fe0665c877c201fc7b44d3c3de16fdbfdab31b7ac2cd`，离线审计零 finding。下一独立 Gate 可评审取消 mutation、CLI/UI/MCP 客户端或真实 Provider，不得把它们混入读边界。
|
- D-322/ADR-0414（已接受）：Cluster Copilot failure diagnosis 已补齐 request-keyed 产品读模型。`GET /api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses/{requestId}` 只需 `run.read`，投影 running/terminal、取消/失败 stage/reason、authoritative admission/finalization time 与 durable usage/settled cost；同路径 `/output` 独立要求 `artifact.read`，只在 admission plan/receipt、Project/source Run、finalization、invocation 和 encrypted Artifact 全部 exact binding 且 current Project Policy 再授权后解析 historical key。调用者不能提交 Artifact、diagnosis Run、Model invocation、provider/model、价格、usage、outcome 或 key identity；deny/approval/absent/cross-target 统一 404，存储/Policy/key/decrypt/cipher 冲突统一 503。写 `capability` 与 `readCapability` 接口隔离，production 复用既有 AI PostgreSQL Pool、repository、Project Policy 与 projected output keyring，不新增 package、进程、端口、Pool、timer、watcher、queue、cache、Pod、Service 或 Kubernetes 权限；普通 Cluster Control、Edge/Standalone 均无这些 route。workspace 保持 18 package、`singleSourcePackages=[]`、`shallowSourcePackages=[]`，AI 193 个源码中 192 个、Cluster Control 62 个源码中 60 个位于嵌套领域目录。AI 249 pass/3 条件 skip、Cluster Control 256 pass/2 条件 skip、18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，四项架构审计和 14 档 Local artifact 全部通过；默认 Edge/Standalone 仍为 2,589,890/2,589,968 bytes，Edge/Standalone AI 为 3,069,143/3,069,233 bytes。PostgreSQL 18.6 arm64 physical HA 139/139、timeline `1→2`，成功密文输出在 standby 可读，提升后 request-keyed exact replay 且 provider 调用为零；报告 SHA-256 为 `22decb54cfb8735bf787fe0665c877c201fc7b44d3c3de16fdbfdab31b7ac2cd`，离线审计零 finding。下一独立 Gate 可评审取消 mutation、CLI/UI/MCP 客户端或真实 Provider，不得把它们混入读边界。
|
||||||
- D-321/ADR-0413（已接受）：Cluster Copilot failure diagnosis 已开放唯一、默认关闭的产品写入口 `POST /api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses`。该 route 只在显式 AI 进程且 `QL3_CLUSTER_AI_COPILOT_ENABLED=true` 时注入既有 Cluster Control route registry，完整复用同一认证器、`model.invoke` Project Policy、fail-closed 同步安全审计、HTTP/TLS body/response/concurrency 上限和生命周期；内部只读 Tool 仍独立复验其 exact `tool.call:*` Policy。耐久幂等身份直接绑定 `x-request-id`，body 只允许 schema 与 `traceId`，Project/source Run/principal 来自 canonical path 和认证，Attempt、日志范围、Tool、provider/model、预算、deadline、Policy fence、reason/outcome 均由服务端从数据库与只读配置派生。响应只投影 created/existing、source/diagnosis Run、终态 stage/reason/outcome 和加密输出 Artifact id/digest，不返回日志、Tool/Model plaintext、prompt、模型信息、密钥或内部异常。普通 Cluster Control、Edge、Standalone 和未启用 Copilot 的 AI 进程没有 route，也没有新增 package、进程、监听器、Pool/连接、timer、watcher、队列、cache、Pod、Service 或 Kubernetes 权限；workspace 仍为 18 package、无单文件/浅平 package，新增源码位于既有嵌套 Copilot 领域目录。Cluster Control 250 pass/2 条件 skip，18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，四项架构审计和 14 档 Local artifact 全部通过；默认 Edge/Standalone 保持 2,589,890/2,589,968 bytes。PostgreSQL 18.6 arm64 HA 137/137、timeline `1→2`，报告 SHA-256 为 `0a12b5c1102555823d43b5a93dd7868b98b194b491840a5242bab6fa2da26123`，离线审计零 finding。本 Gate 不改 migration/schema/role/SQL/HA 拓扑。下一 Gate 优先补同一 Policy 下的加密诊断输出读取与费用/取消可观测性；CLI/UI/MCP 只能复用该 API，不得直连 application capability。
|
- D-321/ADR-0413（已接受）：Cluster Copilot failure diagnosis 已开放唯一、默认关闭的产品写入口 `POST /api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses`。该 route 只在显式 AI 进程且 `QL3_CLUSTER_AI_COPILOT_ENABLED=true` 时注入既有 Cluster Control route registry，完整复用同一认证器、`model.invoke` Project Policy、fail-closed 同步安全审计、HTTP/TLS body/response/concurrency 上限和生命周期；内部只读 Tool 仍独立复验其 exact `tool.call:*` Policy。耐久幂等身份直接绑定 `x-request-id`，body 只允许 schema 与 `traceId`，Project/source Run/principal 来自 canonical path 和认证，Attempt、日志范围、Tool、provider/model、预算、deadline、Policy fence、reason/outcome 均由服务端从数据库与只读配置派生。响应只投影 created/existing、source/diagnosis Run、终态 stage/reason/outcome 和加密输出 Artifact id/digest，不返回日志、Tool/Model plaintext、prompt、模型信息、密钥或内部异常。普通 Cluster Control、Edge、Standalone 和未启用 Copilot 的 AI 进程没有 route，也没有新增 package、进程、监听器、Pool/连接、timer、watcher、队列、cache、Pod、Service 或 Kubernetes 权限；workspace 仍为 18 package、无单文件/浅平 package，新增源码位于既有嵌套 Copilot 领域目录。Cluster Control 250 pass/2 条件 skip，18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，四项架构审计和 14 档 Local artifact 全部通过；默认 Edge/Standalone 保持 2,589,890/2,589,968 bytes。PostgreSQL 18.6 arm64 HA 137/137、timeline `1→2`，报告 SHA-256 为 `0a12b5c1102555823d43b5a93dd7868b98b194b491840a5242bab6fa2da26123`，离线审计零 finding。本 Gate 不改 migration/schema/role/SQL/HA 拓扑。下一 Gate 优先补同一 Policy 下的加密诊断输出读取与费用/取消可观测性；CLI/UI/MCP 只能复用该 API，不得直连 application capability。
|
||||||
- D-320/ADR-0412（已接受）：Cluster Copilot failure diagnosis 已补齐产品入口前的非成功收敛边界。新增的 pre-Model terminalizer 只从 durable Tool failure、受审日志 projection 或数据库观察到的 deadline/cancellation 派生封闭 reason，在 Model start 不存在时以一个 SERIALIZABLE 事务原子提交 StepRun mutations、RunEvents、父 Run 终态和 `pg-9021` append-only content-free receipt；Tool `failed|timed_out`、日志 `not_found|pending|missing|retired`、五秒 Tool budget 不足、deadline 和 cancellation 均可 exact replay。Model 已开始后的 `outcome_unknown` 继续要求强 User 显式 `fail|cancel|retry` resolution；Copilot finalizer 现在对 `fail|cancel` 精确校验 resolution mutation 的 resolved Step digest，既不伪造失败也不自动重试 Provider。实现复用既有 AI package、Cluster AI 进程、Pool 与 ledger，不增加 package、进程、连接、timer、watcher、队列、cache 或产品 route；workspace 仍为 18 package、无单文件/浅平 package，AI 192 个源码中 191 个位于嵌套领域目录。AI 完整测试 244 pass/3 条件 skip，18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，四项架构审计和 14 档 Local artifact 全部通过。PostgreSQL 18.6 arm64 HA 137/137、timeline `1→2`，覆盖日志不可用 pre-Model terminalization、unknown completion→人工 fail resolution→Run finalization 及晋升后零外部副作用 exact replay；报告 SHA-256 为 `6eaeb20615a62d153c5a69687344f41f31351c6ecf111cfb9cbafad115538c83`，离线审计零 finding。下一 Gate 才增加认证、Policy、audit 与 source fence 保护的 Cluster API，并由 CLI/UI/MCP 复用，仍不得建立旁路执行器。
|
- D-320/ADR-0412（已接受）：Cluster Copilot failure diagnosis 已补齐产品入口前的非成功收敛边界。新增的 pre-Model terminalizer 只从 durable Tool failure、受审日志 projection 或数据库观察到的 deadline/cancellation 派生封闭 reason，在 Model start 不存在时以一个 SERIALIZABLE 事务原子提交 StepRun mutations、RunEvents、父 Run 终态和 `pg-9021` append-only content-free receipt；Tool `failed|timed_out`、日志 `not_found|pending|missing|retired`、五秒 Tool budget 不足、deadline 和 cancellation 均可 exact replay。Model 已开始后的 `outcome_unknown` 继续要求强 User 显式 `fail|cancel|retry` resolution；Copilot finalizer 现在对 `fail|cancel` 精确校验 resolution mutation 的 resolved Step digest，既不伪造失败也不自动重试 Provider。实现复用既有 AI package、Cluster AI 进程、Pool 与 ledger，不增加 package、进程、连接、timer、watcher、队列、cache 或产品 route；workspace 仍为 18 package、无单文件/浅平 package，AI 192 个源码中 191 个位于嵌套领域目录。AI 完整测试 244 pass/3 条件 skip，18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，四项架构审计和 14 档 Local artifact 全部通过。PostgreSQL 18.6 arm64 HA 137/137、timeline `1→2`，覆盖日志不可用 pre-Model terminalization、unknown completion→人工 fail resolution→Run finalization 及晋升后零外部副作用 exact replay；报告 SHA-256 为 `6eaeb20615a62d153c5a69687344f41f31351c6ecf111cfb9cbafad115538c83`，离线审计零 finding。下一 Gate 才增加认证、Policy、audit 与 source fence 保护的 Cluster API，并由 CLI/UI/MCP 复用，仍不得建立旁路执行器。
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# ADR-0415：受围栏的 Copilot 故障诊断取消入口
|
||||||
|
|
||||||
|
- 状态：Accepted
|
||||||
|
- 日期：2026-08-16
|
||||||
|
- 关联 RFC：QL-RFC-0001 D-323、Phase 2
|
||||||
|
|
||||||
|
## 背景
|
||||||
|
|
||||||
|
D-321 已提供受认证的故障诊断执行入口，D-322 已提供 request-keyed 状态与密文输出读取，但用户还不能按原始诊断 `requestId` 停止不再需要的诊断。直接要求调用者取消内部 diagnosis Run 会泄露实现 identity；直接复用普通 Run cancellation route 又无法验证 source Run、原始 request 与 diagnosis Run 的三方绑定。
|
||||||
|
|
||||||
|
现有 durable authority 已经足够：admission plan/receipt 唯一绑定 Project、source Run、request 与 diagnosis Run；通用 Run cancellation transaction 会重验当前 Project/RoleBinding Policy fence，并以数据库时间写唯一 cancellation intent/Event；pre-Model terminalizer 能在 Model start 不存在时原子终结 Tool/Model Step、父 Run 与 receipt。Model start 之后目前没有 Provider abort acknowledgement，因而不能把“已写停止意图”伪装成“Provider 已停止”。
|
||||||
|
|
||||||
|
## 决策
|
||||||
|
|
||||||
|
1. Cluster AI profile 新增：
|
||||||
|
`POST /api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses/{requestId}/cancellation`，固定 operation `copilot.failure_diagnosis.cancel`、permission `run.stop`，不接受 query。
|
||||||
|
2. 请求复用 profile-neutral exact body `qinglong/run-cancellation@v1`，仅含 `schema` 与调用方生成的 `mutationId`。调用方不能提交 diagnosis Run、Event ID、取消原因、Model invocation、Provider 或终态；Event ID 由服务端生成。
|
||||||
|
3. admission plan 与 receipt 必须同时存在，并 exact 绑定 path 中的 Project、source Run 和 request。服务端只使用 plan 中的 diagnosis Run ID 调用既有 PostgreSQL Run cancellation repository；不存在、跨 Project、跨 source Run 或漂移统一返回 404。
|
||||||
|
4. route admission 负责 Bearer authentication、`run.stop` 当前 Policy 判定与 durable audit；Run cancellation transaction 在写 intent/Event 的同一 SERIALIZABLE 事务中再次锁定并验证当前 Project/RoleBinding fence。撤权竞态返回 409，存储或 durable evidence 冲突返回 503。
|
||||||
|
5. Model start 尚不存在时，取消 intent 提交后立即调用既有 pre-Model terminalizer。terminalizer 与 Model start 通过 PostgreSQL Run/Step/version fence 竞争：terminalizer 获胜则 Run/未完成 Step 原子进入 `cancelled`；Model start 获胜则进入 `model_in_flight`。
|
||||||
|
6. Model 已开始时只承诺 durable cancellation intent，不伪造 Provider abort、usage、cost 或终态。真实 Provider completion 可以成为最终结果；`outcome_unknown` 仍必须走既有强 User resolution。后续若增加 Provider abort，必须另立包含 abort acknowledgement、late completion 与计费语义的 ADR。
|
||||||
|
7. 响应 schema 为 `qinglong/cluster-copilot-failure-diagnosis-cancellation-response@v1`，投影 `accepted|already_requested|already_terminal`、`terminal|model_in_flight`、exact target、diagnosis Run、当前/终态 outcome 与 cancellation fact。首次 intent 为 202，其余幂等重放为 200。
|
||||||
|
8. 相同 mutation 的响应丢失重放不得追加 Event 或再次递增版本；不同 mutation 也不得覆盖已有 reason/time。取消与 Model start/finalization 的任一竞态只能有一个合法 durable winner。
|
||||||
|
9. 实现留在既有 `@qinglong/ai` 和 `@qinglong/cluster-control` 的嵌套 Copilot 领域目录，复用同一 AI PostgreSQL Pool、repositories、Policy pipeline 与进程；不新增 package、schema、Pool、连接、timer、watcher、队列、cache、端口、Pod、Service 或 Kubernetes 权限。普通 Cluster Control、Edge 与 Standalone 不注册该 route。
|
||||||
|
|
||||||
|
## 不选择
|
||||||
|
|
||||||
|
- **让客户端直接调用 diagnosis Run 的通用 cancellation route**：暴露内部 Run identity，且无法证明 source/request binding。
|
||||||
|
- **Model start 后直接写 cancelled**：没有 Provider abort acknowledgement，会伪造外部副作用与计费事实，并允许迟到 completion 覆盖假终态。
|
||||||
|
- **为 Copilot cancellation 新建 package、表或 supervisor**：现有 admission、Run intent、terminalization 与 Model recovery ledger 已覆盖所需 authority；新资源只会复制状态机并增加低配设备成本。
|
||||||
|
- **在 MCP/UI/CLI 中同时实现写入口**：客户端需要独立的强认证、确认交互与错误展示门，不能与服务端 authority 混成一个不可审计改动。
|
||||||
|
|
||||||
|
## 验收
|
||||||
|
|
||||||
|
1. 单元/集成测试覆盖 invalid body、权限、跨 Project/source/request、Policy 撤权、首次/相同 mutation/不同 mutation、终态重放、pre-Model 原子 terminalization、Model-start winner、finalization race 与故障映射。
|
||||||
|
2. PostgreSQL HA 覆盖 intent/Event/terminal receipt 复制、promotion 后 exact replay、pre-Model winner 和 in-flight 不伪造终态。
|
||||||
|
3. `@qinglong/ai`、`@qinglong/cluster-control`、18-package clean build/test、backend、四项架构审计、14 档 Local artifact 与离线 HA 报告审计全部通过后才允许 D-323 阶段提交。
|
||||||
|
|
||||||
|
最终验收（2026-08-16）：AI 254 pass/3 条件 skip，Cluster Control 261 pass/2 条件 skip，18-package clean build/test 退出 0，backend 1,207 pass/2 条件 skip/0 fail；workspace 保持 18 package，`singleSourcePackages=[]`、`shallowSourcePackages=[]`。目录密度门拒绝新增的第 12 个 failure-diagnosis 直属文件后，删除了仅做 re-export 的单文件 façade，公开 subpath 直接指向内聚的 `cancellation/service`，未新增 package 或放宽目录阈值。四项架构审计、14 档 Edge/Standalone artifact 与 PostgreSQL HA 离线证据审计均通过；PostgreSQL 18.6 arm64 physical HA 为 142/142、timeline `1→2`，报告 SHA-256 为 `5dbcffb74a3181aabee66a8f68ecfa7a65e0491a6f2ba24e2bc903c83da9d766`。
|
||||||
@@ -416,6 +416,9 @@
|
|||||||
| [ADR-0410](./ADR-0410-cluster-copilot-failure-diagnosis-output-key-authority.md) | Cluster Copilot Failure Diagnosis Output Projected Key Authority | Accepted |
|
| [ADR-0410](./ADR-0410-cluster-copilot-failure-diagnosis-output-key-authority.md) | Cluster Copilot Failure Diagnosis Output Projected Key Authority | Accepted |
|
||||||
| [ADR-0411](./ADR-0411-default-off-cluster-copilot-composition.md) | 默认关闭的 Cluster Copilot Failure Diagnosis Composition | Accepted |
|
| [ADR-0411](./ADR-0411-default-off-cluster-copilot-composition.md) | 默认关闭的 Cluster Copilot Failure Diagnosis Composition | Accepted |
|
||||||
| [ADR-0412](./ADR-0412-durable-copilot-pre-model-terminalization-and-unknown-resolution.md) | 耐久 Copilot Pre-Model 终态化与 Model Unknown Resolution | Accepted |
|
| [ADR-0412](./ADR-0412-durable-copilot-pre-model-terminalization-and-unknown-resolution.md) | 耐久 Copilot Pre-Model 终态化与 Model Unknown Resolution | Accepted |
|
||||||
|
| [ADR-0413](./ADR-0413-default-off-cluster-copilot-failure-diagnosis-api.md) | 默认关闭的 Cluster Copilot Failure Diagnosis API | Accepted |
|
||||||
|
| [ADR-0414](./ADR-0414-copilot-failure-diagnosis-read-model.md) | Copilot 故障诊断请求键读模型 | Accepted |
|
||||||
|
| [ADR-0415](./ADR-0415-fenced-copilot-failure-diagnosis-cancellation.md) | 受围栏的 Copilot 故障诊断取消入口 | Accepted |
|
||||||
|
|
||||||
## 规则
|
## 规则
|
||||||
|
|
||||||
|
|||||||
@@ -65,6 +65,11 @@
|
|||||||
"require": "./dist/copilot/failure-diagnosis/read-model/service.js",
|
"require": "./dist/copilot/failure-diagnosis/read-model/service.js",
|
||||||
"default": "./dist/copilot/failure-diagnosis/read-model/service.js"
|
"default": "./dist/copilot/failure-diagnosis/read-model/service.js"
|
||||||
},
|
},
|
||||||
|
"./failure-diagnosis-cancellation": {
|
||||||
|
"types": "./dist/copilot/failure-diagnosis/cancellation/service.d.ts",
|
||||||
|
"require": "./dist/copilot/failure-diagnosis/cancellation/service.js",
|
||||||
|
"default": "./dist/copilot/failure-diagnosis/cancellation/service.js"
|
||||||
|
},
|
||||||
"./failure-diagnosis-pre-model-terminalization": {
|
"./failure-diagnosis-pre-model-terminalization": {
|
||||||
"types": "./dist/copilot/failure-diagnosis/preModelTerminalization.d.ts",
|
"types": "./dist/copilot/failure-diagnosis/preModelTerminalization.d.ts",
|
||||||
"require": "./dist/copilot/failure-diagnosis/preModelTerminalization.js",
|
"require": "./dist/copilot/failure-diagnosis/preModelTerminalization.js",
|
||||||
|
|||||||
@@ -0,0 +1,415 @@
|
|||||||
|
import { RUN_STATUSES, type RunStatus } from '@qinglong/runtime-core';
|
||||||
|
import type {
|
||||||
|
ClusterRunCancellationRepository,
|
||||||
|
ClusterRunCancellationResult,
|
||||||
|
} from '@qinglong/runtime-core/cluster-run-cancellation';
|
||||||
|
import type {
|
||||||
|
SecurityPolicyFence,
|
||||||
|
SecuritySubject,
|
||||||
|
} from '@qinglong/runtime-core/security';
|
||||||
|
|
||||||
|
import type {
|
||||||
|
CopilotFailureDiagnosisAdmissionRepository,
|
||||||
|
CopilotFailureDiagnosisAdmissionReceipt,
|
||||||
|
CopilotFailureDiagnosisExecutionPlan,
|
||||||
|
} from '../admission/contracts';
|
||||||
|
import { normalizeCopilotFailureDiagnosisAdmissionReceipt } from '../admission/durableEvidence';
|
||||||
|
import { normalizeCopilotFailureDiagnosisExecutionPlan } from '../admission/plan';
|
||||||
|
import {
|
||||||
|
terminalizeCopilotFailureDiagnosisBeforeModel,
|
||||||
|
type CopilotFailureDiagnosisPreModelTerminalizationDependencies,
|
||||||
|
} from '../terminalization/coordinator';
|
||||||
|
import {
|
||||||
|
CopilotFailureDiagnosisPreModelTerminalizationConflictError,
|
||||||
|
CopilotFailureDiagnosisPreModelTerminalizationNotReadyError,
|
||||||
|
CopilotFailureDiagnosisPreModelTerminalizationUnavailableError,
|
||||||
|
} from '../terminalization/contracts';
|
||||||
|
|
||||||
|
export const COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_RESULT_SCHEMA =
|
||||||
|
'qinglong/copilot-failure-diagnosis-cancellation-result@v1' as const;
|
||||||
|
|
||||||
|
export interface CopilotFailureDiagnosisCancellationCommand {
|
||||||
|
readonly projectId: string;
|
||||||
|
readonly sourceRunId: string;
|
||||||
|
readonly requestId: string;
|
||||||
|
readonly mutationId: string;
|
||||||
|
readonly eventId: string;
|
||||||
|
readonly subject: Readonly<SecuritySubject>;
|
||||||
|
readonly policyFence: Readonly<SecurityPolicyFence>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CopilotFailureDiagnosisCancellationResult {
|
||||||
|
readonly schema: typeof COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_RESULT_SCHEMA;
|
||||||
|
readonly status: ClusterRunCancellationResult['status'];
|
||||||
|
readonly convergence: 'terminal' | 'model_in_flight';
|
||||||
|
readonly projectId: string;
|
||||||
|
readonly sourceRunId: string;
|
||||||
|
readonly requestId: string;
|
||||||
|
readonly diagnosisRunId: string;
|
||||||
|
readonly runStatus: RunStatus;
|
||||||
|
readonly outcome: 'succeeded' | 'failed' | 'timed_out' | 'cancelled' | null;
|
||||||
|
readonly runVersion: number;
|
||||||
|
readonly eventSequence: number;
|
||||||
|
readonly cancelRequestedAtMs: number | null;
|
||||||
|
readonly cancelReason:
|
||||||
|
| 'user'
|
||||||
|
| 'policy'
|
||||||
|
| 'shutdown'
|
||||||
|
| 'reconcile'
|
||||||
|
| 'timeout'
|
||||||
|
| null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CopilotFailureDiagnosisCancellationDependencies {
|
||||||
|
readonly admissions: Pick<
|
||||||
|
CopilotFailureDiagnosisAdmissionRepository,
|
||||||
|
'findByRequestId' | 'findPlanByRequestId'
|
||||||
|
>;
|
||||||
|
readonly cancellations: ClusterRunCancellationRepository;
|
||||||
|
readonly terminalizations: CopilotFailureDiagnosisPreModelTerminalizationDependencies;
|
||||||
|
readonly terminalizeBeforeModel?: typeof terminalizeCopilotFailureDiagnosisBeforeModel;
|
||||||
|
}
|
||||||
|
|
||||||
|
export class InvalidCopilotFailureDiagnosisCancellationError extends TypeError {
|
||||||
|
readonly code = 'COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_INVALID';
|
||||||
|
|
||||||
|
constructor() {
|
||||||
|
super('Copilot failure diagnosis cancellation is invalid');
|
||||||
|
this.name = 'InvalidCopilotFailureDiagnosisCancellationError';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export class CopilotFailureDiagnosisCancellationNotFoundError extends Error {
|
||||||
|
readonly code = 'COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_NOT_FOUND';
|
||||||
|
|
||||||
|
constructor() {
|
||||||
|
super('Copilot failure diagnosis cancellation target does not exist');
|
||||||
|
this.name = 'CopilotFailureDiagnosisCancellationNotFoundError';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export class CopilotFailureDiagnosisCancellationUnavailableError extends Error {
|
||||||
|
readonly code = 'COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_UNAVAILABLE';
|
||||||
|
|
||||||
|
constructor(options?: ErrorOptions) {
|
||||||
|
super('Copilot failure diagnosis cancellation is unavailable', options);
|
||||||
|
this.name = 'CopilotFailureDiagnosisCancellationUnavailableError';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const IDENTITY = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
|
||||||
|
const RUN_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$/;
|
||||||
|
const TERMINAL = new Set<RunStatus>([
|
||||||
|
'succeeded',
|
||||||
|
'failed',
|
||||||
|
'cancelled',
|
||||||
|
'timed_out',
|
||||||
|
]);
|
||||||
|
|
||||||
|
function unavailable(cause?: unknown): never {
|
||||||
|
throw new CopilotFailureDiagnosisCancellationUnavailableError({
|
||||||
|
cause: cause instanceof Error ? cause : undefined,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function exactKeys(value: object, expected: readonly string[]): boolean {
|
||||||
|
const actual = Object.keys(value).sort();
|
||||||
|
const canonical = [...expected].sort();
|
||||||
|
return (
|
||||||
|
actual.length === canonical.length &&
|
||||||
|
actual.every((key, index) => key === canonical[index])
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function command(
|
||||||
|
value: CopilotFailureDiagnosisCancellationCommand,
|
||||||
|
): Readonly<CopilotFailureDiagnosisCancellationCommand> {
|
||||||
|
if (
|
||||||
|
!value ||
|
||||||
|
typeof value !== 'object' ||
|
||||||
|
Array.isArray(value) ||
|
||||||
|
!exactKeys(value, [
|
||||||
|
'eventId',
|
||||||
|
'mutationId',
|
||||||
|
'policyFence',
|
||||||
|
'projectId',
|
||||||
|
'requestId',
|
||||||
|
'sourceRunId',
|
||||||
|
'subject',
|
||||||
|
]) ||
|
||||||
|
!IDENTITY.test(value.projectId) ||
|
||||||
|
!RUN_ID.test(value.sourceRunId) ||
|
||||||
|
!IDENTITY.test(value.requestId) ||
|
||||||
|
!IDENTITY.test(value.mutationId) ||
|
||||||
|
!IDENTITY.test(value.eventId) ||
|
||||||
|
!value.subject ||
|
||||||
|
typeof value.subject !== 'object' ||
|
||||||
|
!value.policyFence ||
|
||||||
|
typeof value.policyFence !== 'object'
|
||||||
|
) {
|
||||||
|
throw new InvalidCopilotFailureDiagnosisCancellationError();
|
||||||
|
}
|
||||||
|
return Object.freeze({ ...value });
|
||||||
|
}
|
||||||
|
|
||||||
|
function evidenceMatches(
|
||||||
|
plan: Readonly<CopilotFailureDiagnosisExecutionPlan>,
|
||||||
|
receipt: Readonly<CopilotFailureDiagnosisAdmissionReceipt>,
|
||||||
|
): boolean {
|
||||||
|
return (
|
||||||
|
receipt.requestId === plan.requestId &&
|
||||||
|
receipt.planDigest === plan.planDigest &&
|
||||||
|
receipt.runId === plan.runId &&
|
||||||
|
receipt.sourceRunId === plan.source.runId &&
|
||||||
|
receipt.sourceRunVersion === plan.source.runVersion &&
|
||||||
|
receipt.sourceAttemptId === plan.source.attemptId &&
|
||||||
|
receipt.toolStepRunId === plan.toolStepRunId &&
|
||||||
|
receipt.modelStepRunId === plan.modelStepRunId
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function targetMatches(
|
||||||
|
plan: Readonly<CopilotFailureDiagnosisExecutionPlan>,
|
||||||
|
value: Readonly<CopilotFailureDiagnosisCancellationCommand>,
|
||||||
|
): boolean {
|
||||||
|
return (
|
||||||
|
plan.projectId === value.projectId &&
|
||||||
|
plan.source.runId === value.sourceRunId &&
|
||||||
|
plan.requestId === value.requestId
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function result(
|
||||||
|
target: Readonly<CopilotFailureDiagnosisCancellationCommand>,
|
||||||
|
cancellation: Readonly<ClusterRunCancellationResult>,
|
||||||
|
state: Readonly<{
|
||||||
|
convergence: CopilotFailureDiagnosisCancellationResult['convergence'];
|
||||||
|
runStatus: RunStatus;
|
||||||
|
runVersion: number;
|
||||||
|
eventSequence: number;
|
||||||
|
cancelRequestedAtMs?: number;
|
||||||
|
cancelReason?: NonNullable<
|
||||||
|
CopilotFailureDiagnosisCancellationResult['cancelReason']
|
||||||
|
>;
|
||||||
|
}>,
|
||||||
|
): Readonly<CopilotFailureDiagnosisCancellationResult> {
|
||||||
|
if (
|
||||||
|
cancellation.projectId !== target.projectId ||
|
||||||
|
!RUN_STATUSES.includes(state.runStatus) ||
|
||||||
|
!Number.isSafeInteger(state.runVersion) ||
|
||||||
|
state.runVersion < 0 ||
|
||||||
|
!Number.isSafeInteger(state.eventSequence) ||
|
||||||
|
state.eventSequence < 0 ||
|
||||||
|
(state.convergence === 'terminal') !== TERMINAL.has(state.runStatus) ||
|
||||||
|
(state.convergence === 'model_in_flight' && state.runStatus !== 'running')
|
||||||
|
) {
|
||||||
|
return unavailable();
|
||||||
|
}
|
||||||
|
return Object.freeze({
|
||||||
|
schema: COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_RESULT_SCHEMA,
|
||||||
|
status: cancellation.status,
|
||||||
|
convergence: state.convergence,
|
||||||
|
projectId: target.projectId,
|
||||||
|
sourceRunId: target.sourceRunId,
|
||||||
|
requestId: target.requestId,
|
||||||
|
diagnosisRunId: cancellation.runId,
|
||||||
|
runStatus: state.runStatus,
|
||||||
|
outcome: TERMINAL.has(state.runStatus)
|
||||||
|
? (state.runStatus as 'succeeded' | 'failed' | 'timed_out' | 'cancelled')
|
||||||
|
: null,
|
||||||
|
runVersion: state.runVersion,
|
||||||
|
eventSequence: state.eventSequence,
|
||||||
|
cancelRequestedAtMs: state.cancelRequestedAtMs ?? null,
|
||||||
|
cancelReason: state.cancelReason ?? null,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Resolves an external request key to its server-owned diagnosis Run, writes
|
||||||
|
* one ordinary Run cancellation intent, then converges only while no Model
|
||||||
|
* invocation start exists.
|
||||||
|
*/
|
||||||
|
export class CopilotFailureDiagnosisCancellationService {
|
||||||
|
readonly #dependencies: Readonly<CopilotFailureDiagnosisCancellationDependencies>;
|
||||||
|
readonly #terminalize: typeof terminalizeCopilotFailureDiagnosisBeforeModel;
|
||||||
|
|
||||||
|
constructor(dependencies: CopilotFailureDiagnosisCancellationDependencies) {
|
||||||
|
if (
|
||||||
|
!dependencies ||
|
||||||
|
typeof dependencies !== 'object' ||
|
||||||
|
Array.isArray(dependencies) ||
|
||||||
|
typeof dependencies.admissions?.findByRequestId !== 'function' ||
|
||||||
|
typeof dependencies.admissions?.findPlanByRequestId !== 'function' ||
|
||||||
|
typeof dependencies.cancellations?.requestUserCancellation !==
|
||||||
|
'function' ||
|
||||||
|
typeof dependencies.terminalizations?.repository?.findByRequestId !==
|
||||||
|
'function' ||
|
||||||
|
typeof dependencies.terminalizations?.repository?.readAuthority !==
|
||||||
|
'function' ||
|
||||||
|
typeof dependencies.terminalizations?.repository?.commit !== 'function' ||
|
||||||
|
(dependencies.terminalizeBeforeModel !== undefined &&
|
||||||
|
typeof dependencies.terminalizeBeforeModel !== 'function')
|
||||||
|
) {
|
||||||
|
throw new InvalidCopilotFailureDiagnosisCancellationError();
|
||||||
|
}
|
||||||
|
this.#dependencies = Object.freeze({ ...dependencies });
|
||||||
|
this.#terminalize =
|
||||||
|
dependencies.terminalizeBeforeModel ??
|
||||||
|
terminalizeCopilotFailureDiagnosisBeforeModel;
|
||||||
|
}
|
||||||
|
|
||||||
|
async cancel(
|
||||||
|
value: CopilotFailureDiagnosisCancellationCommand,
|
||||||
|
): Promise<Readonly<CopilotFailureDiagnosisCancellationResult>> {
|
||||||
|
const target = command(value);
|
||||||
|
let plan: Readonly<CopilotFailureDiagnosisExecutionPlan> | null;
|
||||||
|
let receipt: Readonly<CopilotFailureDiagnosisAdmissionReceipt> | null;
|
||||||
|
try {
|
||||||
|
const located = await Promise.all([
|
||||||
|
this.#dependencies.admissions.findPlanByRequestId(target.requestId),
|
||||||
|
this.#dependencies.admissions.findByRequestId(target.requestId),
|
||||||
|
]);
|
||||||
|
plan = located[0]
|
||||||
|
? normalizeCopilotFailureDiagnosisExecutionPlan(located[0])
|
||||||
|
: null;
|
||||||
|
receipt = located[1]
|
||||||
|
? normalizeCopilotFailureDiagnosisAdmissionReceipt(located[1])
|
||||||
|
: null;
|
||||||
|
} catch (cause) {
|
||||||
|
return unavailable(cause);
|
||||||
|
}
|
||||||
|
if (!plan || !receipt || !targetMatches(plan, target)) {
|
||||||
|
throw new CopilotFailureDiagnosisCancellationNotFoundError();
|
||||||
|
}
|
||||||
|
if (!evidenceMatches(plan, receipt)) return unavailable();
|
||||||
|
|
||||||
|
const cancellation =
|
||||||
|
await this.#dependencies.cancellations.requestUserCancellation({
|
||||||
|
projectId: target.projectId,
|
||||||
|
runId: plan.runId,
|
||||||
|
mutationId: target.mutationId,
|
||||||
|
eventId: target.eventId,
|
||||||
|
subject: target.subject,
|
||||||
|
policyFence: target.policyFence,
|
||||||
|
});
|
||||||
|
if (
|
||||||
|
cancellation.projectId !== target.projectId ||
|
||||||
|
cancellation.runId !== plan.runId
|
||||||
|
) {
|
||||||
|
return unavailable();
|
||||||
|
}
|
||||||
|
if (TERMINAL.has(cancellation.runStatus)) {
|
||||||
|
return result(target, cancellation, {
|
||||||
|
convergence: 'terminal',
|
||||||
|
runStatus: cancellation.runStatus,
|
||||||
|
runVersion: cancellation.runVersion,
|
||||||
|
eventSequence: cancellation.eventSequence,
|
||||||
|
...(cancellation.cancelRequestedAtMs === undefined
|
||||||
|
? {}
|
||||||
|
: { cancelRequestedAtMs: cancellation.cancelRequestedAtMs }),
|
||||||
|
...(cancellation.cancelReason === undefined
|
||||||
|
? {}
|
||||||
|
: { cancelReason: cancellation.cancelReason }),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
for (let attempt = 0; attempt < 2; attempt += 1) {
|
||||||
|
try {
|
||||||
|
const terminalized = await this.#terminalize(
|
||||||
|
target.requestId,
|
||||||
|
{ kind: 'boundary' },
|
||||||
|
this.#dependencies.terminalizations,
|
||||||
|
);
|
||||||
|
return result(target, cancellation, {
|
||||||
|
convergence: 'terminal',
|
||||||
|
runStatus: terminalized.receipt.outcome,
|
||||||
|
runVersion: terminalized.receipt.finalRunVersion,
|
||||||
|
eventSequence: terminalized.receipt.finalRunEventSequence,
|
||||||
|
...(cancellation.cancelRequestedAtMs === undefined
|
||||||
|
? {}
|
||||||
|
: { cancelRequestedAtMs: cancellation.cancelRequestedAtMs }),
|
||||||
|
...(cancellation.cancelReason === undefined
|
||||||
|
? {}
|
||||||
|
: { cancelReason: cancellation.cancelReason }),
|
||||||
|
});
|
||||||
|
} catch (cause) {
|
||||||
|
if (
|
||||||
|
!(
|
||||||
|
cause instanceof
|
||||||
|
CopilotFailureDiagnosisPreModelTerminalizationConflictError
|
||||||
|
) &&
|
||||||
|
!(
|
||||||
|
cause instanceof
|
||||||
|
CopilotFailureDiagnosisPreModelTerminalizationNotReadyError
|
||||||
|
)
|
||||||
|
) {
|
||||||
|
if (
|
||||||
|
cause instanceof
|
||||||
|
CopilotFailureDiagnosisPreModelTerminalizationUnavailableError
|
||||||
|
) {
|
||||||
|
return unavailable(cause);
|
||||||
|
}
|
||||||
|
return unavailable(cause);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const authority =
|
||||||
|
await this.#dependencies.terminalizations.repository.readAuthority(
|
||||||
|
target.requestId,
|
||||||
|
);
|
||||||
|
if (
|
||||||
|
!targetMatches(authority.plan, target) ||
|
||||||
|
authority.plan.planDigest !== plan.planDigest ||
|
||||||
|
authority.run.id !== plan.runId ||
|
||||||
|
authority.run.projectId !== target.projectId ||
|
||||||
|
authority.run.version !== authority.run.eventSequence
|
||||||
|
) {
|
||||||
|
return unavailable();
|
||||||
|
}
|
||||||
|
if (TERMINAL.has(authority.run.status)) {
|
||||||
|
return result(target, cancellation, {
|
||||||
|
convergence: 'terminal',
|
||||||
|
runStatus: authority.run.status,
|
||||||
|
runVersion: authority.run.version,
|
||||||
|
eventSequence: authority.run.eventSequence,
|
||||||
|
...(authority.run.cancelRequestedAtMs === undefined
|
||||||
|
? {}
|
||||||
|
: { cancelRequestedAtMs: authority.run.cancelRequestedAtMs }),
|
||||||
|
...(authority.run.cancelReason === undefined
|
||||||
|
? {}
|
||||||
|
: { cancelReason: authority.run.cancelReason }),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
authority.run.status === 'running' &&
|
||||||
|
authority.modelStartExists &&
|
||||||
|
authority.run.cancelRequestedAtMs !== undefined &&
|
||||||
|
authority.run.cancelReason !== undefined
|
||||||
|
) {
|
||||||
|
return result(target, cancellation, {
|
||||||
|
convergence: 'model_in_flight',
|
||||||
|
runStatus: 'running',
|
||||||
|
runVersion: authority.run.version,
|
||||||
|
eventSequence: authority.run.eventSequence,
|
||||||
|
cancelRequestedAtMs: authority.run.cancelRequestedAtMs,
|
||||||
|
cancelReason: authority.run.cancelReason,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
attempt === 0 &&
|
||||||
|
authority.run.status === 'running' &&
|
||||||
|
!authority.modelStartExists &&
|
||||||
|
authority.run.cancelRequestedAtMs !== undefined &&
|
||||||
|
authority.run.cancelReason !== undefined
|
||||||
|
) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
return unavailable();
|
||||||
|
} catch (cause) {
|
||||||
|
return unavailable(cause);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return unavailable();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,341 @@
|
|||||||
|
const assert = require('node:assert/strict');
|
||||||
|
const { test } = require('node:test');
|
||||||
|
|
||||||
|
const {
|
||||||
|
BUILTIN_RUN_LOG_EXCERPT_TOOL,
|
||||||
|
BUILTIN_RUN_LOG_EXCERPT_TOOL_DEFINITION,
|
||||||
|
createBuiltInRunLogExcerptToolHandlerBinding,
|
||||||
|
} = require('@qinglong/runtime-core/builtin-run-log-excerpt-tool');
|
||||||
|
const {
|
||||||
|
createPluginPackageResourceGenerationFromReferences,
|
||||||
|
} = require('@qinglong/runtime-core/plugin-package-resource-generation');
|
||||||
|
const {
|
||||||
|
createProjectToolDefinitionSnapshot,
|
||||||
|
projectToolDefinitionRegistry,
|
||||||
|
} = require('@qinglong/runtime-core/project-tool-definition-snapshot');
|
||||||
|
const {
|
||||||
|
prepareToolInvocation,
|
||||||
|
} = require('@qinglong/runtime-core/tool-registry');
|
||||||
|
const {
|
||||||
|
TrustedToolHandlerBindingRegistry,
|
||||||
|
createTrustedToolInvocationPlan,
|
||||||
|
} = require('@qinglong/runtime-core/trusted-tool-invocation');
|
||||||
|
const {
|
||||||
|
CopilotFailureDiagnosisCancellationNotFoundError,
|
||||||
|
CopilotFailureDiagnosisCancellationService,
|
||||||
|
CopilotFailureDiagnosisCancellationUnavailableError,
|
||||||
|
} = require('@qinglong/ai/failure-diagnosis-cancellation');
|
||||||
|
const {
|
||||||
|
createCopilotFailureDiagnosisAdmissionBundle,
|
||||||
|
prepareCopilotFailureDiagnosisExecution,
|
||||||
|
} = require('@qinglong/ai/failure-diagnosis-execution-admission');
|
||||||
|
const {
|
||||||
|
CopilotFailureDiagnosisPreModelTerminalizationConflictError,
|
||||||
|
} = require('@qinglong/ai/failure-diagnosis-pre-model-terminalization');
|
||||||
|
|
||||||
|
async function durablePlan() {
|
||||||
|
const generation = createPluginPackageResourceGenerationFromReferences({
|
||||||
|
installationId: 'install-cancel-test',
|
||||||
|
projectId: 'project-cancel',
|
||||||
|
packageName: 'qinglong',
|
||||||
|
lockDigest: 'a'.repeat(64),
|
||||||
|
generation: 1,
|
||||||
|
previousActiveLockDigest: null,
|
||||||
|
contentDigest: 'b'.repeat(64),
|
||||||
|
resources: [],
|
||||||
|
});
|
||||||
|
const snapshot = createProjectToolDefinitionSnapshot({
|
||||||
|
projectId: 'project-cancel',
|
||||||
|
contributions: [
|
||||||
|
{
|
||||||
|
generation,
|
||||||
|
revisionDigest: 'c'.repeat(64),
|
||||||
|
definitions: [BUILTIN_RUN_LOG_EXCERPT_TOOL_DEFINITION],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
});
|
||||||
|
const binding = createBuiltInRunLogExcerptToolHandlerBinding(snapshot, [
|
||||||
|
'cluster-control',
|
||||||
|
]);
|
||||||
|
const bindings = new TrustedToolHandlerBindingRegistry(snapshot, [binding]);
|
||||||
|
const principal = {
|
||||||
|
subject: { type: 'user', id: 'owner-cancel' },
|
||||||
|
authenticationId: 'auth-cancel',
|
||||||
|
authenticatedAtMs: 100,
|
||||||
|
expiresAtMs: 10_000,
|
||||||
|
assurance: 'multi_factor',
|
||||||
|
};
|
||||||
|
const invocation = await prepareToolInvocation(
|
||||||
|
projectToolDefinitionRegistry(snapshot),
|
||||||
|
{
|
||||||
|
projectId: 'project-cancel',
|
||||||
|
principal,
|
||||||
|
nowMs: 200,
|
||||||
|
tool: BUILTIN_RUN_LOG_EXCERPT_TOOL,
|
||||||
|
input: { runId: 'source-run-cancel', attemptId: 'source-attempt-cancel' },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
async authorize() {
|
||||||
|
return {
|
||||||
|
effect: 'allow',
|
||||||
|
reasons: ['role_grant'],
|
||||||
|
fence: { projectVersion: 2, bindingVersion: 3 },
|
||||||
|
};
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
|
const tool = createTrustedToolInvocationPlan(bindings, invocation, {
|
||||||
|
actionRef: 'cancel-log-tool',
|
||||||
|
inputArtifactId: 'cancel-input',
|
||||||
|
previewArtifactId: 'cancel-preview',
|
||||||
|
artifactKeyId: 'cancel-key',
|
||||||
|
artifactKey: Buffer.alloc(32, 0x11),
|
||||||
|
artifactNonce: Buffer.alloc(12, 0x22),
|
||||||
|
profile: 'cluster-control',
|
||||||
|
preview: {
|
||||||
|
title: 'Read failed Run log',
|
||||||
|
summary: 'Read bounded evidence',
|
||||||
|
fields: [
|
||||||
|
{ kind: 'identifier', label: 'Run', value: 'source-run-cancel' },
|
||||||
|
{
|
||||||
|
kind: 'identifier',
|
||||||
|
label: 'Attempt',
|
||||||
|
value: 'source-attempt-cancel',
|
||||||
|
},
|
||||||
|
],
|
||||||
|
warnings: ['potentially_sensitive_output'],
|
||||||
|
},
|
||||||
|
sealedAtMs: 300,
|
||||||
|
});
|
||||||
|
return prepareCopilotFailureDiagnosisExecution({
|
||||||
|
requestId: 'diagnosis-request-cancel',
|
||||||
|
traceId: 'diagnosis-trace-cancel',
|
||||||
|
source: {
|
||||||
|
runId: 'source-run-cancel',
|
||||||
|
runVersion: 4,
|
||||||
|
runStatus: 'failed',
|
||||||
|
attemptId: 'source-attempt-cancel',
|
||||||
|
attemptStatus: 'failed',
|
||||||
|
attemptFinishedAtMs: 250,
|
||||||
|
logArtifactId: 'source-log-cancel',
|
||||||
|
},
|
||||||
|
toolPlan: tool.plan,
|
||||||
|
bindings,
|
||||||
|
model: {
|
||||||
|
provider: 'provider-primary',
|
||||||
|
model: 'model-diagnosis',
|
||||||
|
modelBoundary: 'external',
|
||||||
|
responseLanguage: 'zh-CN',
|
||||||
|
maxOutputTokens: 256,
|
||||||
|
egressPolicy: {
|
||||||
|
schema: 'qinglong/copilot-model-egress-policy@v1',
|
||||||
|
revision: 'cancel-policy-v1',
|
||||||
|
potentiallySensitiveDataBoundaries: ['external'],
|
||||||
|
maxInputBytes: 64 * 1024,
|
||||||
|
maxOutputTokens: 512,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
plannedAtMs: 400,
|
||||||
|
deadlineAtMs: 8_000,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function command(overrides = {}) {
|
||||||
|
return {
|
||||||
|
projectId: 'project-cancel',
|
||||||
|
sourceRunId: 'source-run-cancel',
|
||||||
|
requestId: 'diagnosis-request-cancel',
|
||||||
|
mutationId: '11111111-1111-4111-8111-111111111111',
|
||||||
|
eventId: '22222222-2222-4222-8222-222222222222',
|
||||||
|
subject: { type: 'user', id: 'owner-cancel' },
|
||||||
|
policyFence: { projectVersion: 2, bindingVersion: 3 },
|
||||||
|
...overrides,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function fixture(plan, overrides = {}) {
|
||||||
|
const receipt = createCopilotFailureDiagnosisAdmissionBundle(plan).receipt;
|
||||||
|
const state = {
|
||||||
|
cancellationCalls: [],
|
||||||
|
terminalizeCalls: 0,
|
||||||
|
cancellation: {
|
||||||
|
status: 'accepted',
|
||||||
|
projectId: plan.projectId,
|
||||||
|
runId: plan.runId,
|
||||||
|
runStatus: 'running',
|
||||||
|
runVersion: 4,
|
||||||
|
eventSequence: 4,
|
||||||
|
cancelRequestedAtMs: 500,
|
||||||
|
cancelReason: 'user',
|
||||||
|
},
|
||||||
|
authority: {
|
||||||
|
plan,
|
||||||
|
run: {
|
||||||
|
id: plan.runId,
|
||||||
|
projectId: plan.projectId,
|
||||||
|
status: 'running',
|
||||||
|
version: 4,
|
||||||
|
eventSequence: 4,
|
||||||
|
cancelRequestedAtMs: 500,
|
||||||
|
cancelReason: 'user',
|
||||||
|
},
|
||||||
|
toolStep: {},
|
||||||
|
modelStep: {},
|
||||||
|
modelStartExists: false,
|
||||||
|
observedAtMs: 501,
|
||||||
|
},
|
||||||
|
...overrides,
|
||||||
|
};
|
||||||
|
const terminalizations = {
|
||||||
|
repository: {
|
||||||
|
async findByRequestId() {
|
||||||
|
return null;
|
||||||
|
},
|
||||||
|
async readAuthority() {
|
||||||
|
return state.authority;
|
||||||
|
},
|
||||||
|
async commit() {
|
||||||
|
throw new Error('not called by injected terminalizer');
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const service = new CopilotFailureDiagnosisCancellationService({
|
||||||
|
admissions: {
|
||||||
|
async findPlanByRequestId() {
|
||||||
|
return state.plan === undefined ? plan : state.plan;
|
||||||
|
},
|
||||||
|
async findByRequestId() {
|
||||||
|
return state.receipt === undefined ? receipt : state.receipt;
|
||||||
|
},
|
||||||
|
},
|
||||||
|
cancellations: {
|
||||||
|
async requestUserCancellation(value) {
|
||||||
|
state.cancellationCalls.push(value);
|
||||||
|
return state.cancellation;
|
||||||
|
},
|
||||||
|
},
|
||||||
|
terminalizations,
|
||||||
|
async terminalizeBeforeModel(requestId, trigger) {
|
||||||
|
state.terminalizeCalls += 1;
|
||||||
|
if (state.terminalizeError) throw state.terminalizeError;
|
||||||
|
assert.equal(requestId, plan.requestId);
|
||||||
|
assert.deepEqual(trigger, { kind: 'boundary' });
|
||||||
|
return {
|
||||||
|
status: 'created',
|
||||||
|
receipt: {
|
||||||
|
outcome: 'cancelled',
|
||||||
|
finalRunVersion: 7,
|
||||||
|
finalRunEventSequence: 7,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
},
|
||||||
|
});
|
||||||
|
return { service, state, receipt };
|
||||||
|
}
|
||||||
|
|
||||||
|
test('resolves the request key to a server-owned Run and terminalizes pre-Model', async () => {
|
||||||
|
const plan = await durablePlan();
|
||||||
|
const { service, state } = fixture(plan);
|
||||||
|
const result = await service.cancel(command());
|
||||||
|
assert.deepEqual(state.cancellationCalls, [
|
||||||
|
{
|
||||||
|
projectId: 'project-cancel',
|
||||||
|
runId: plan.runId,
|
||||||
|
mutationId: '11111111-1111-4111-8111-111111111111',
|
||||||
|
eventId: '22222222-2222-4222-8222-222222222222',
|
||||||
|
subject: { type: 'user', id: 'owner-cancel' },
|
||||||
|
policyFence: { projectVersion: 2, bindingVersion: 3 },
|
||||||
|
},
|
||||||
|
]);
|
||||||
|
assert.equal(state.terminalizeCalls, 1);
|
||||||
|
assert.equal(result.status, 'accepted');
|
||||||
|
assert.equal(result.convergence, 'terminal');
|
||||||
|
assert.equal(result.runStatus, 'cancelled');
|
||||||
|
assert.equal(result.outcome, 'cancelled');
|
||||||
|
assert.equal(result.diagnosisRunId, plan.runId);
|
||||||
|
assert.equal(result.cancelRequestedAtMs, 500);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('masks cross-target requests before writing a cancellation intent', async () => {
|
||||||
|
const plan = await durablePlan();
|
||||||
|
const { service, state } = fixture(plan);
|
||||||
|
await assert.rejects(
|
||||||
|
service.cancel(command({ sourceRunId: 'other-source-run' })),
|
||||||
|
CopilotFailureDiagnosisCancellationNotFoundError,
|
||||||
|
);
|
||||||
|
assert.equal(state.cancellationCalls.length, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('keeps a durable intent pending when Model start wins the race', async () => {
|
||||||
|
const plan = await durablePlan();
|
||||||
|
const conflict =
|
||||||
|
new CopilotFailureDiagnosisPreModelTerminalizationConflictError(
|
||||||
|
'Model started',
|
||||||
|
);
|
||||||
|
const { service, state } = fixture(plan, {
|
||||||
|
terminalizeError: conflict,
|
||||||
|
authority: {
|
||||||
|
plan,
|
||||||
|
run: {
|
||||||
|
id: plan.runId,
|
||||||
|
projectId: plan.projectId,
|
||||||
|
status: 'running',
|
||||||
|
version: 5,
|
||||||
|
eventSequence: 5,
|
||||||
|
cancelRequestedAtMs: 500,
|
||||||
|
cancelReason: 'user',
|
||||||
|
},
|
||||||
|
toolStep: {},
|
||||||
|
modelStep: {},
|
||||||
|
modelStartExists: true,
|
||||||
|
observedAtMs: 501,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const result = await service.cancel(command());
|
||||||
|
assert.equal(state.terminalizeCalls, 1);
|
||||||
|
assert.equal(result.convergence, 'model_in_flight');
|
||||||
|
assert.equal(result.runStatus, 'running');
|
||||||
|
assert.equal(result.outcome, null);
|
||||||
|
assert.equal(result.runVersion, 5);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('reports the real terminal winner instead of forging cancellation', async () => {
|
||||||
|
const plan = await durablePlan();
|
||||||
|
const { service } = fixture(plan, {
|
||||||
|
terminalizeError:
|
||||||
|
new CopilotFailureDiagnosisPreModelTerminalizationConflictError(),
|
||||||
|
authority: {
|
||||||
|
plan,
|
||||||
|
run: {
|
||||||
|
id: plan.runId,
|
||||||
|
projectId: plan.projectId,
|
||||||
|
status: 'succeeded',
|
||||||
|
version: 7,
|
||||||
|
eventSequence: 7,
|
||||||
|
cancelRequestedAtMs: 500,
|
||||||
|
cancelReason: 'user',
|
||||||
|
},
|
||||||
|
toolStep: {},
|
||||||
|
modelStep: {},
|
||||||
|
modelStartExists: true,
|
||||||
|
observedAtMs: 600,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const result = await service.cancel(command());
|
||||||
|
assert.equal(result.convergence, 'terminal');
|
||||||
|
assert.equal(result.runStatus, 'succeeded');
|
||||||
|
assert.equal(result.outcome, 'succeeded');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('fails closed when plan and admission receipt drift', async () => {
|
||||||
|
const plan = await durablePlan();
|
||||||
|
const base = fixture(plan);
|
||||||
|
const { service, state } = fixture(plan, {
|
||||||
|
receipt: { ...base.receipt, runId: 'diagnosis-run-drift' },
|
||||||
|
});
|
||||||
|
await assert.rejects(
|
||||||
|
service.cancel(command()),
|
||||||
|
CopilotFailureDiagnosisCancellationUnavailableError,
|
||||||
|
);
|
||||||
|
assert.equal(state.cancellationCalls.length, 0);
|
||||||
|
});
|
||||||
@@ -40,6 +40,11 @@
|
|||||||
"require": "./dist/application-runtime/copilot/failureDiagnosisComposition.js",
|
"require": "./dist/application-runtime/copilot/failureDiagnosisComposition.js",
|
||||||
"default": "./dist/application-runtime/copilot/failureDiagnosisComposition.js"
|
"default": "./dist/application-runtime/copilot/failureDiagnosisComposition.js"
|
||||||
},
|
},
|
||||||
|
"./copilot-cancellation-production": {
|
||||||
|
"types": "./dist/application-runtime/copilot/failureDiagnosisCancellationComposition.d.ts",
|
||||||
|
"require": "./dist/application-runtime/copilot/failureDiagnosisCancellationComposition.js",
|
||||||
|
"default": "./dist/application-runtime/copilot/failureDiagnosisCancellationComposition.js"
|
||||||
|
},
|
||||||
"./failure-diagnosis-output-keyring": {
|
"./failure-diagnosis-output-keyring": {
|
||||||
"types": "./dist/copilot/failure-diagnosis/outputProjectedKeyring.d.ts",
|
"types": "./dist/copilot/failure-diagnosis/outputProjectedKeyring.d.ts",
|
||||||
"require": "./dist/copilot/failure-diagnosis/outputProjectedKeyring.js",
|
"require": "./dist/copilot/failure-diagnosis/outputProjectedKeyring.js",
|
||||||
@@ -55,6 +60,11 @@
|
|||||||
"require": "./dist/copilot/failure-diagnosis/failureDiagnosisReadRoutes.js",
|
"require": "./dist/copilot/failure-diagnosis/failureDiagnosisReadRoutes.js",
|
||||||
"default": "./dist/copilot/failure-diagnosis/failureDiagnosisReadRoutes.js"
|
"default": "./dist/copilot/failure-diagnosis/failureDiagnosisReadRoutes.js"
|
||||||
},
|
},
|
||||||
|
"./copilot-cancellation-route": {
|
||||||
|
"types": "./dist/copilot/failure-diagnosis/failureDiagnosisCancellationRoute.d.ts",
|
||||||
|
"require": "./dist/copilot/failure-diagnosis/failureDiagnosisCancellationRoute.js",
|
||||||
|
"default": "./dist/copilot/failure-diagnosis/failureDiagnosisCancellationRoute.js"
|
||||||
|
},
|
||||||
"./http": {
|
"./http": {
|
||||||
"types": "./dist/transport/httpSurface.d.ts",
|
"types": "./dist/transport/httpSurface.d.ts",
|
||||||
"require": "./dist/transport/httpSurface.js",
|
"require": "./dist/transport/httpSurface.js",
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ import type { ModelGatewayProfileAudit } from '@qinglong/ai/profile';
|
|||||||
import type { DurableModelInvocationCoordinator } from '@qinglong/ai/durable-model-invocation';
|
import type { DurableModelInvocationCoordinator } from '@qinglong/ai/durable-model-invocation';
|
||||||
import { CopilotFailureDiagnosisModelCompletionCoordinator } from '@qinglong/ai/failure-diagnosis-model-execution';
|
import { CopilotFailureDiagnosisModelCompletionCoordinator } from '@qinglong/ai/failure-diagnosis-model-execution';
|
||||||
import type { CopilotFailureDiagnosisApplicationService } from '@qinglong/ai/failure-diagnosis-application';
|
import type { CopilotFailureDiagnosisApplicationService } from '@qinglong/ai/failure-diagnosis-application';
|
||||||
|
import type { CopilotFailureDiagnosisCancellationService } from '@qinglong/ai/failure-diagnosis-cancellation';
|
||||||
import { BoundModelProviderCredentialProvider } from '@qinglong/ai/provider-credential';
|
import { BoundModelProviderCredentialProvider } from '@qinglong/ai/provider-credential';
|
||||||
import { PostgresModelProviderCredentialReader } from '@qinglong/ai/postgres-model-provider-credential-storage';
|
import { PostgresModelProviderCredentialReader } from '@qinglong/ai/postgres-model-provider-credential-storage';
|
||||||
import { loadProjectedModelGatewayProviderAuthority } from '@qinglong/ai/projected-model-gateway-authority';
|
import { loadProjectedModelGatewayProviderAuthority } from '@qinglong/ai/projected-model-gateway-authority';
|
||||||
@@ -35,6 +36,10 @@ import {
|
|||||||
createProductionClusterCopilotFailureDiagnosisReadService,
|
createProductionClusterCopilotFailureDiagnosisReadService,
|
||||||
type CreateProductionClusterCopilotFailureDiagnosisReadServiceOptions,
|
type CreateProductionClusterCopilotFailureDiagnosisReadServiceOptions,
|
||||||
} from './copilot/failureDiagnosisReadComposition';
|
} from './copilot/failureDiagnosisReadComposition';
|
||||||
|
import {
|
||||||
|
createProductionClusterCopilotFailureDiagnosisCancellation,
|
||||||
|
type CreateProductionClusterCopilotFailureDiagnosisCancellationOptions,
|
||||||
|
} from './copilot/failureDiagnosisCancellationComposition';
|
||||||
|
|
||||||
export interface EnabledProductionClusterAiConfig {
|
export interface EnabledProductionClusterAiConfig {
|
||||||
readonly enabled: true;
|
readonly enabled: true;
|
||||||
@@ -63,6 +68,9 @@ export interface ProductionClusterAiControlApplicationOptions {
|
|||||||
) => ReturnType<
|
) => ReturnType<
|
||||||
typeof createProductionClusterCopilotFailureDiagnosisReadService
|
typeof createProductionClusterCopilotFailureDiagnosisReadService
|
||||||
>;
|
>;
|
||||||
|
readonly createCopilotCancellation?: (
|
||||||
|
options: CreateProductionClusterCopilotFailureDiagnosisCancellationOptions,
|
||||||
|
) => Readonly<CopilotFailureDiagnosisCancellationService>;
|
||||||
readonly openAiDatabase?: ReturnType<typeof createPostgresDatabaseOpener>;
|
readonly openAiDatabase?: ReturnType<typeof createPostgresDatabaseOpener>;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -263,11 +271,15 @@ export async function startProductionClusterAiControlApplication(
|
|||||||
const createCopilotRead =
|
const createCopilotRead =
|
||||||
options.createCopilotRead ??
|
options.createCopilotRead ??
|
||||||
createProductionClusterCopilotFailureDiagnosisReadService;
|
createProductionClusterCopilotFailureDiagnosisReadService;
|
||||||
|
const createCopilotCancellation =
|
||||||
|
options.createCopilotCancellation ??
|
||||||
|
createProductionClusterCopilotFailureDiagnosisCancellation;
|
||||||
if (
|
if (
|
||||||
typeof startControl !== 'function' ||
|
typeof startControl !== 'function' ||
|
||||||
typeof bootstrapPrompt !== 'function' ||
|
typeof bootstrapPrompt !== 'function' ||
|
||||||
typeof createCopilot !== 'function' ||
|
typeof createCopilot !== 'function' ||
|
||||||
typeof createCopilotRead !== 'function' ||
|
typeof createCopilotRead !== 'function' ||
|
||||||
|
typeof createCopilotCancellation !== 'function' ||
|
||||||
(options.openAiDatabase !== undefined &&
|
(options.openAiDatabase !== undefined &&
|
||||||
typeof options.openAiDatabase !== 'function')
|
typeof options.openAiDatabase !== 'function')
|
||||||
) {
|
) {
|
||||||
@@ -325,6 +337,9 @@ export async function startProductionClusterAiControlApplication(
|
|||||||
typeof createProductionClusterCopilotFailureDiagnosisReadService
|
typeof createProductionClusterCopilotFailureDiagnosisReadService
|
||||||
>
|
>
|
||||||
| undefined;
|
| undefined;
|
||||||
|
let copilotCancellationApplication:
|
||||||
|
| Readonly<CopilotFailureDiagnosisCancellationService>
|
||||||
|
| undefined;
|
||||||
let copilotSuccessfulCompletion:
|
let copilotSuccessfulCompletion:
|
||||||
| CopilotFailureDiagnosisModelCompletionCoordinator
|
| CopilotFailureDiagnosisModelCompletionCoordinator
|
||||||
| undefined;
|
| undefined;
|
||||||
@@ -455,6 +470,9 @@ export async function startProductionClusterAiControlApplication(
|
|||||||
pool: aiDatabase.pool,
|
pool: aiDatabase.pool,
|
||||||
prepared: preparedCopilot,
|
prepared: preparedCopilot,
|
||||||
});
|
});
|
||||||
|
copilotCancellationApplication = createCopilotCancellation({
|
||||||
|
pool: aiDatabase.pool,
|
||||||
|
});
|
||||||
}
|
}
|
||||||
controlApplication = await startControl({
|
controlApplication = await startControl({
|
||||||
...options.control,
|
...options.control,
|
||||||
@@ -482,12 +500,14 @@ export async function startProductionClusterAiControlApplication(
|
|||||||
},
|
},
|
||||||
}),
|
}),
|
||||||
...(copilotApplication === undefined ||
|
...(copilotApplication === undefined ||
|
||||||
copilotReadApplication === undefined
|
copilotReadApplication === undefined ||
|
||||||
|
copilotCancellationApplication === undefined
|
||||||
? {}
|
? {}
|
||||||
: {
|
: {
|
||||||
copilotFailureDiagnosis: {
|
copilotFailureDiagnosis: {
|
||||||
capability: copilotApplication,
|
capability: copilotApplication,
|
||||||
readCapability: copilotReadApplication,
|
readCapability: copilotReadApplication,
|
||||||
|
cancellationCapability: copilotCancellationApplication,
|
||||||
},
|
},
|
||||||
}),
|
}),
|
||||||
});
|
});
|
||||||
|
|||||||
+47
@@ -0,0 +1,47 @@
|
|||||||
|
import { CopilotFailureDiagnosisCancellationService } from '@qinglong/ai/failure-diagnosis-cancellation';
|
||||||
|
import { PostgresCopilotFailureDiagnosisAdmissionRepository } from '@qinglong/ai/postgres-failure-diagnosis-admission-storage';
|
||||||
|
import {
|
||||||
|
PostgresCopilotFailureDiagnosisPreModelTerminalizationRepository,
|
||||||
|
terminalizeCopilotFailureDiagnosisBeforeModel,
|
||||||
|
} from '@qinglong/ai/failure-diagnosis-pre-model-terminalization';
|
||||||
|
import {
|
||||||
|
PostgresClusterRunCancellationRepository,
|
||||||
|
type QingLongPostgresPool,
|
||||||
|
} from '@qinglong/cluster-postgres/runtime';
|
||||||
|
|
||||||
|
export interface CreateProductionClusterCopilotFailureDiagnosisCancellationOptions {
|
||||||
|
readonly pool: QingLongPostgresPool;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Reuses the AI Pool and existing Run/admission ledgers. The cancellation
|
||||||
|
* capability owns no connection, timer, listener or background lifecycle.
|
||||||
|
*/
|
||||||
|
export function createProductionClusterCopilotFailureDiagnosisCancellation(
|
||||||
|
options: CreateProductionClusterCopilotFailureDiagnosisCancellationOptions,
|
||||||
|
): Readonly<CopilotFailureDiagnosisCancellationService> {
|
||||||
|
if (
|
||||||
|
!options ||
|
||||||
|
typeof options !== 'object' ||
|
||||||
|
Array.isArray(options) ||
|
||||||
|
typeof options.pool?.query !== 'function' ||
|
||||||
|
typeof options.pool?.connect !== 'function'
|
||||||
|
) {
|
||||||
|
throw new TypeError(
|
||||||
|
'Production Cluster Copilot failure diagnosis cancellation dependencies are invalid',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const admissions = new PostgresCopilotFailureDiagnosisAdmissionRepository(
|
||||||
|
options.pool,
|
||||||
|
);
|
||||||
|
const terminalizations =
|
||||||
|
new PostgresCopilotFailureDiagnosisPreModelTerminalizationRepository(
|
||||||
|
options.pool,
|
||||||
|
);
|
||||||
|
return new CopilotFailureDiagnosisCancellationService({
|
||||||
|
admissions,
|
||||||
|
cancellations: new PostgresClusterRunCancellationRepository(options.pool),
|
||||||
|
terminalizations: Object.freeze({ repository: terminalizations }),
|
||||||
|
terminalizeBeforeModel: terminalizeCopilotFailureDiagnosisBeforeModel,
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -71,6 +71,10 @@ import {
|
|||||||
type ClusterCopilotFailureDiagnosisInspectionCapability,
|
type ClusterCopilotFailureDiagnosisInspectionCapability,
|
||||||
type ClusterCopilotFailureDiagnosisOutputReadCapability,
|
type ClusterCopilotFailureDiagnosisOutputReadCapability,
|
||||||
} from '../copilot/failure-diagnosis/failureDiagnosisReadRoutes';
|
} from '../copilot/failure-diagnosis/failureDiagnosisReadRoutes';
|
||||||
|
import {
|
||||||
|
createClusterControlCopilotFailureDiagnosisCancellationRoute,
|
||||||
|
type ClusterCopilotFailureDiagnosisCancellationCapability,
|
||||||
|
} from '../copilot/failure-diagnosis/failureDiagnosisCancellationRoute';
|
||||||
|
|
||||||
export const PRODUCTION_CLUSTER_CONTROL_ROUTE_OPERATIONS = Object.freeze([
|
export const PRODUCTION_CLUSTER_CONTROL_ROUTE_OPERATIONS = Object.freeze([
|
||||||
'task.get',
|
'task.get',
|
||||||
@@ -100,6 +104,7 @@ export const PRODUCTION_CLUSTER_CONTROL_OPTIONAL_ROUTE_OPERATIONS =
|
|||||||
'copilot.failure_diagnosis.execute',
|
'copilot.failure_diagnosis.execute',
|
||||||
'copilot.failure_diagnosis.read',
|
'copilot.failure_diagnosis.read',
|
||||||
'copilot.failure_diagnosis.output.read',
|
'copilot.failure_diagnosis.output.read',
|
||||||
|
'copilot.failure_diagnosis.cancel',
|
||||||
] as const);
|
] as const);
|
||||||
|
|
||||||
export interface ClusterCopilotFailureDiagnosisReadCapability
|
export interface ClusterCopilotFailureDiagnosisReadCapability
|
||||||
@@ -129,6 +134,7 @@ export interface ProductionClusterControlAssemblyOptions {
|
|||||||
readonly copilotFailureDiagnosis?: Readonly<{
|
readonly copilotFailureDiagnosis?: Readonly<{
|
||||||
readonly capability: ClusterCopilotFailureDiagnosisCapability;
|
readonly capability: ClusterCopilotFailureDiagnosisCapability;
|
||||||
readonly readCapability?: ClusterCopilotFailureDiagnosisReadCapability;
|
readonly readCapability?: ClusterCopilotFailureDiagnosisReadCapability;
|
||||||
|
readonly cancellationCapability?: ClusterCopilotFailureDiagnosisCancellationCapability;
|
||||||
}>;
|
}>;
|
||||||
readonly workerIngress?: Readonly<{
|
readonly workerIngress?: Readonly<{
|
||||||
readonly config: EnabledClusterWorkerIngressConfig;
|
readonly config: EnabledClusterWorkerIngressConfig;
|
||||||
@@ -178,6 +184,7 @@ export interface ProductionClusterControlApplicationOptions
|
|||||||
readonly copilotFailureDiagnosis?: Readonly<{
|
readonly copilotFailureDiagnosis?: Readonly<{
|
||||||
readonly capability: ClusterCopilotFailureDiagnosisCapability;
|
readonly capability: ClusterCopilotFailureDiagnosisCapability;
|
||||||
readonly readCapability?: ClusterCopilotFailureDiagnosisReadCapability;
|
readonly readCapability?: ClusterCopilotFailureDiagnosisReadCapability;
|
||||||
|
readonly cancellationCapability?: ClusterCopilotFailureDiagnosisCancellationCapability;
|
||||||
}>;
|
}>;
|
||||||
readonly workerIngress?: ProductionClusterWorkerIngressOptions;
|
readonly workerIngress?: ProductionClusterWorkerIngressOptions;
|
||||||
}
|
}
|
||||||
@@ -303,6 +310,14 @@ export function createProductionClusterControlApplicationStack(
|
|||||||
options.copilotFailureDiagnosis.readCapability,
|
options.copilotFailureDiagnosis.readCapability,
|
||||||
),
|
),
|
||||||
]),
|
]),
|
||||||
|
...(options.copilotFailureDiagnosis?.cancellationCapability === undefined
|
||||||
|
? []
|
||||||
|
: [
|
||||||
|
createClusterControlCopilotFailureDiagnosisCancellationRoute(
|
||||||
|
options.copilotFailureDiagnosis.cancellationCapability,
|
||||||
|
createEventId,
|
||||||
|
),
|
||||||
|
]),
|
||||||
];
|
];
|
||||||
const routes = createClusterControlRouteRegistry(routeDefinitions);
|
const routes = createClusterControlRouteRegistry(routeDefinitions);
|
||||||
const expectedRouteCount =
|
const expectedRouteCount =
|
||||||
@@ -313,7 +328,10 @@ export function createProductionClusterControlApplicationStack(
|
|||||||
(options.promptOutputRead === undefined ? 0 : 1) +
|
(options.promptOutputRead === undefined ? 0 : 1) +
|
||||||
(options.promptExecutionOutputRead === undefined ? 0 : 1) +
|
(options.promptExecutionOutputRead === undefined ? 0 : 1) +
|
||||||
(options.copilotFailureDiagnosis === undefined ? 0 : 1) +
|
(options.copilotFailureDiagnosis === undefined ? 0 : 1) +
|
||||||
(options.copilotFailureDiagnosis?.readCapability === undefined ? 0 : 2);
|
(options.copilotFailureDiagnosis?.readCapability === undefined ? 0 : 2) +
|
||||||
|
(options.copilotFailureDiagnosis?.cancellationCapability === undefined
|
||||||
|
? 0
|
||||||
|
: 1);
|
||||||
if (routes.size !== expectedRouteCount) {
|
if (routes.size !== expectedRouteCount) {
|
||||||
throw new Error('Production cluster-control route allowlist is incomplete');
|
throw new Error('Production cluster-control route allowlist is incomplete');
|
||||||
}
|
}
|
||||||
|
|||||||
+261
@@ -0,0 +1,261 @@
|
|||||||
|
// Cluster Copilot resolves an external request key before cancelling its Run.
|
||||||
|
import {
|
||||||
|
CopilotFailureDiagnosisCancellationNotFoundError,
|
||||||
|
CopilotFailureDiagnosisCancellationUnavailableError,
|
||||||
|
InvalidCopilotFailureDiagnosisCancellationError,
|
||||||
|
type CopilotFailureDiagnosisCancellationCommand,
|
||||||
|
} from '@qinglong/ai/failure-diagnosis-cancellation';
|
||||||
|
import {
|
||||||
|
CLUSTER_RUN_CANCELLATION_SCHEMA,
|
||||||
|
ClusterRunCancellationFenceRejectedError,
|
||||||
|
ClusterRunCancellationNotFoundError,
|
||||||
|
ClusterRunCancellationUnavailableError,
|
||||||
|
InvalidClusterRunCancellationError,
|
||||||
|
parseClusterRunCancellationRequestBody,
|
||||||
|
} from '@qinglong/runtime-core/cluster-run-cancellation';
|
||||||
|
|
||||||
|
import type { ClusterControlAdmissionResponse } from '../../transport/httpSurface';
|
||||||
|
import type {
|
||||||
|
ClusterControlAuthorizedOperationRequest,
|
||||||
|
ClusterControlRouteDefinition,
|
||||||
|
ClusterControlRouteParameters,
|
||||||
|
} from '../../transport/routeRegistry';
|
||||||
|
|
||||||
|
export const CLUSTER_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_RESPONSE_SCHEMA =
|
||||||
|
'qinglong/cluster-copilot-failure-diagnosis-cancellation-response@v1' as const;
|
||||||
|
|
||||||
|
export const CLUSTER_CONTROL_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_ROUTE =
|
||||||
|
Object.freeze({
|
||||||
|
method: 'POST' as const,
|
||||||
|
path: '/api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses/{requestId}/cancellation',
|
||||||
|
operationId: 'copilot.failure_diagnosis.cancel',
|
||||||
|
permission: 'run.stop',
|
||||||
|
projectParameter: 'projectId',
|
||||||
|
});
|
||||||
|
|
||||||
|
export interface ClusterCopilotFailureDiagnosisCancellationCapability {
|
||||||
|
cancel(
|
||||||
|
command: Readonly<CopilotFailureDiagnosisCancellationCommand>,
|
||||||
|
): Promise<unknown>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type ClusterCopilotFailureDiagnosisCancellationEventIdFactory =
|
||||||
|
() => string;
|
||||||
|
|
||||||
|
const IDENTITY = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
|
||||||
|
const RUN_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$/;
|
||||||
|
const OUTCOMES = new Set(['succeeded', 'failed', 'timed_out', 'cancelled']);
|
||||||
|
const STATUSES = new Set(['accepted', 'already_requested', 'already_terminal']);
|
||||||
|
const CANCEL_REASONS = new Set([
|
||||||
|
'user',
|
||||||
|
'policy',
|
||||||
|
'shutdown',
|
||||||
|
'reconcile',
|
||||||
|
'timeout',
|
||||||
|
]);
|
||||||
|
|
||||||
|
function response(
|
||||||
|
statusCode: number,
|
||||||
|
body: Readonly<Record<string, unknown>>,
|
||||||
|
): ClusterControlAdmissionResponse {
|
||||||
|
return Object.freeze({ statusCode, body: Object.freeze(body) });
|
||||||
|
}
|
||||||
|
|
||||||
|
function exactRecord(
|
||||||
|
value: unknown,
|
||||||
|
expected: readonly string[],
|
||||||
|
): Record<string, unknown> | null {
|
||||||
|
if (!value || typeof value !== 'object' || Array.isArray(value)) return null;
|
||||||
|
const record = value as Record<string, unknown>;
|
||||||
|
const actual = Object.keys(record).sort();
|
||||||
|
const canonical = [...expected].sort();
|
||||||
|
return actual.length === canonical.length &&
|
||||||
|
actual.every((key, index) => key === canonical[index])
|
||||||
|
? record
|
||||||
|
: null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function safeInteger(value: unknown): value is number {
|
||||||
|
return Number.isSafeInteger(value) && (value as number) >= 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
function projectResult(
|
||||||
|
value: unknown,
|
||||||
|
target: Readonly<{
|
||||||
|
projectId: string;
|
||||||
|
sourceRunId: string;
|
||||||
|
requestId: string;
|
||||||
|
}>,
|
||||||
|
): Readonly<Record<string, unknown>> | null {
|
||||||
|
const candidate = exactRecord(value, [
|
||||||
|
'cancelReason',
|
||||||
|
'cancelRequestedAtMs',
|
||||||
|
'convergence',
|
||||||
|
'diagnosisRunId',
|
||||||
|
'eventSequence',
|
||||||
|
'outcome',
|
||||||
|
'projectId',
|
||||||
|
'requestId',
|
||||||
|
'runStatus',
|
||||||
|
'runVersion',
|
||||||
|
'schema',
|
||||||
|
'sourceRunId',
|
||||||
|
'status',
|
||||||
|
]);
|
||||||
|
if (
|
||||||
|
!candidate ||
|
||||||
|
candidate.schema !==
|
||||||
|
'qinglong/copilot-failure-diagnosis-cancellation-result@v1' ||
|
||||||
|
candidate.projectId !== target.projectId ||
|
||||||
|
candidate.sourceRunId !== target.sourceRunId ||
|
||||||
|
candidate.requestId !== target.requestId ||
|
||||||
|
typeof candidate.diagnosisRunId !== 'string' ||
|
||||||
|
!RUN_ID.test(candidate.diagnosisRunId) ||
|
||||||
|
typeof candidate.status !== 'string' ||
|
||||||
|
!STATUSES.has(candidate.status) ||
|
||||||
|
!safeInteger(candidate.runVersion) ||
|
||||||
|
!safeInteger(candidate.eventSequence) ||
|
||||||
|
candidate.runVersion !== candidate.eventSequence ||
|
||||||
|
!(
|
||||||
|
(candidate.cancelRequestedAtMs === null &&
|
||||||
|
candidate.cancelReason === null) ||
|
||||||
|
(safeInteger(candidate.cancelRequestedAtMs) &&
|
||||||
|
typeof candidate.cancelReason === 'string' &&
|
||||||
|
CANCEL_REASONS.has(candidate.cancelReason))
|
||||||
|
)
|
||||||
|
) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
if (candidate.convergence === 'model_in_flight') {
|
||||||
|
if (
|
||||||
|
candidate.runStatus !== 'running' ||
|
||||||
|
candidate.outcome !== null ||
|
||||||
|
candidate.cancelRequestedAtMs === null
|
||||||
|
) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
} else if (candidate.convergence === 'terminal') {
|
||||||
|
if (
|
||||||
|
typeof candidate.runStatus !== 'string' ||
|
||||||
|
!OUTCOMES.has(candidate.runStatus) ||
|
||||||
|
candidate.outcome !== candidate.runStatus
|
||||||
|
) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
return Object.freeze({
|
||||||
|
schema: CLUSTER_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_RESPONSE_SCHEMA,
|
||||||
|
status: candidate.status,
|
||||||
|
convergence: candidate.convergence,
|
||||||
|
projectId: target.projectId,
|
||||||
|
sourceRunId: target.sourceRunId,
|
||||||
|
requestId: target.requestId,
|
||||||
|
diagnosisRunId: candidate.diagnosisRunId,
|
||||||
|
runStatus: candidate.runStatus,
|
||||||
|
outcome: candidate.outcome,
|
||||||
|
runVersion: candidate.runVersion,
|
||||||
|
eventSequence: candidate.eventSequence,
|
||||||
|
cancelRequestedAtMs: candidate.cancelRequestedAtMs,
|
||||||
|
cancelReason: candidate.cancelReason,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createClusterControlCopilotFailureDiagnosisCancellationRoute(
|
||||||
|
capability: ClusterCopilotFailureDiagnosisCancellationCapability,
|
||||||
|
createEventId: ClusterCopilotFailureDiagnosisCancellationEventIdFactory,
|
||||||
|
): Readonly<ClusterControlRouteDefinition> {
|
||||||
|
if (
|
||||||
|
!capability ||
|
||||||
|
typeof capability.cancel !== 'function' ||
|
||||||
|
typeof createEventId !== 'function'
|
||||||
|
) {
|
||||||
|
throw new TypeError(
|
||||||
|
'Cluster-control Copilot failure diagnosis cancellation route is invalid',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return Object.freeze({
|
||||||
|
...CLUSTER_CONTROL_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_ROUTE,
|
||||||
|
async handle(
|
||||||
|
authorized: ClusterControlAuthorizedOperationRequest,
|
||||||
|
parameters: ClusterControlRouteParameters,
|
||||||
|
) {
|
||||||
|
let body;
|
||||||
|
try {
|
||||||
|
body = parseClusterRunCancellationRequestBody(authorized.request.body);
|
||||||
|
} catch (error) {
|
||||||
|
return error instanceof InvalidClusterRunCancellationError
|
||||||
|
? response(400, {
|
||||||
|
code: 'invalid_copilot_failure_diagnosis_cancellation_request',
|
||||||
|
schema: CLUSTER_RUN_CANCELLATION_SCHEMA,
|
||||||
|
})
|
||||||
|
: response(503, {
|
||||||
|
code: 'copilot_failure_diagnosis_cancellation_unavailable',
|
||||||
|
});
|
||||||
|
}
|
||||||
|
const projectId = authorized.projectId;
|
||||||
|
const sourceRunId = parameters.runId;
|
||||||
|
const requestId = parameters.requestId;
|
||||||
|
if (
|
||||||
|
projectId === null ||
|
||||||
|
typeof sourceRunId !== 'string' ||
|
||||||
|
!RUN_ID.test(sourceRunId) ||
|
||||||
|
typeof requestId !== 'string' ||
|
||||||
|
!IDENTITY.test(requestId) ||
|
||||||
|
!authorized.policyFence ||
|
||||||
|
authorized.policyFence.bindingVersion === null
|
||||||
|
) {
|
||||||
|
return response(503, {
|
||||||
|
code: 'copilot_failure_diagnosis_cancellation_unavailable',
|
||||||
|
});
|
||||||
|
}
|
||||||
|
const target = Object.freeze({ projectId, sourceRunId, requestId });
|
||||||
|
try {
|
||||||
|
const result = await capability.cancel({
|
||||||
|
...target,
|
||||||
|
mutationId: body.mutationId,
|
||||||
|
eventId: createEventId(),
|
||||||
|
subject: authorized.principal.subject,
|
||||||
|
policyFence: authorized.policyFence,
|
||||||
|
});
|
||||||
|
const view = projectResult(result, target);
|
||||||
|
if (!view) {
|
||||||
|
return response(503, {
|
||||||
|
code: 'copilot_failure_diagnosis_cancellation_unavailable',
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return response(view.status === 'accepted' ? 202 : 200, view);
|
||||||
|
} catch (error) {
|
||||||
|
if (
|
||||||
|
error instanceof CopilotFailureDiagnosisCancellationNotFoundError ||
|
||||||
|
error instanceof ClusterRunCancellationNotFoundError
|
||||||
|
) {
|
||||||
|
return response(404, {
|
||||||
|
code: 'copilot_failure_diagnosis_not_found',
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if (error instanceof ClusterRunCancellationFenceRejectedError) {
|
||||||
|
return response(409, {
|
||||||
|
code: 'copilot_failure_diagnosis_cancellation_fence_rejected',
|
||||||
|
reason: error.reason,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
error instanceof InvalidCopilotFailureDiagnosisCancellationError ||
|
||||||
|
error instanceof
|
||||||
|
CopilotFailureDiagnosisCancellationUnavailableError ||
|
||||||
|
error instanceof InvalidClusterRunCancellationError ||
|
||||||
|
error instanceof ClusterRunCancellationUnavailableError
|
||||||
|
) {
|
||||||
|
return response(503, {
|
||||||
|
code: 'copilot_failure_diagnosis_cancellation_unavailable',
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return response(503, {
|
||||||
|
code: 'copilot_failure_diagnosis_cancellation_unavailable',
|
||||||
|
});
|
||||||
|
}
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -169,9 +169,11 @@ test('Copilot composition is explicit, shares the Prompt gateway and injects one
|
|||||||
const artifactStore = { put() {}, inspect() {}, readLogRange() {} };
|
const artifactStore = { put() {}, inspect() {}, readLogRange() {} };
|
||||||
const copilot = Object.freeze({ execute() {} });
|
const copilot = Object.freeze({ execute() {} });
|
||||||
const copilotRead = Object.freeze({ inspect() {}, readOutput() {} });
|
const copilotRead = Object.freeze({ inspect() {}, readOutput() {} });
|
||||||
|
const copilotCancellation = Object.freeze({ cancel() {} });
|
||||||
let registeredSink;
|
let registeredSink;
|
||||||
let created;
|
let created;
|
||||||
let createdRead;
|
let createdRead;
|
||||||
|
let createdCancellation;
|
||||||
let controlOptions;
|
let controlOptions;
|
||||||
try {
|
try {
|
||||||
await Promise.all([
|
await Promise.all([
|
||||||
@@ -233,6 +235,10 @@ test('Copilot composition is explicit, shares the Prompt gateway and injects one
|
|||||||
createdRead = options;
|
createdRead = options;
|
||||||
return copilotRead;
|
return copilotRead;
|
||||||
},
|
},
|
||||||
|
createCopilotCancellation(options) {
|
||||||
|
createdCancellation = options;
|
||||||
|
return copilotCancellation;
|
||||||
|
},
|
||||||
async startControl(options) {
|
async startControl(options) {
|
||||||
controlOptions = options;
|
controlOptions = options;
|
||||||
return {
|
return {
|
||||||
@@ -257,11 +263,16 @@ test('Copilot composition is explicit, shares the Prompt gateway and injects one
|
|||||||
assert.equal(created.artifactStore, artifactStore);
|
assert.equal(created.artifactStore, artifactStore);
|
||||||
assert.equal(createdRead.pool, fakePool);
|
assert.equal(createdRead.pool, fakePool);
|
||||||
assert.equal(typeof createdRead.prepared.outputKeys.resolve, 'function');
|
assert.equal(typeof createdRead.prepared.outputKeys.resolve, 'function');
|
||||||
|
assert.equal(createdCancellation.pool, fakePool);
|
||||||
assert.equal(controlOptions.copilotFailureDiagnosis.capability, copilot);
|
assert.equal(controlOptions.copilotFailureDiagnosis.capability, copilot);
|
||||||
assert.equal(
|
assert.equal(
|
||||||
controlOptions.copilotFailureDiagnosis.readCapability,
|
controlOptions.copilotFailureDiagnosis.readCapability,
|
||||||
copilotRead,
|
copilotRead,
|
||||||
);
|
);
|
||||||
|
assert.equal(
|
||||||
|
controlOptions.copilotFailureDiagnosis.cancellationCapability,
|
||||||
|
copilotCancellation,
|
||||||
|
);
|
||||||
assert.equal(await application.stop(), 'stopped');
|
assert.equal(await application.stop(), 'stopped');
|
||||||
} finally {
|
} finally {
|
||||||
config.fill(0);
|
config.fill(0);
|
||||||
|
|||||||
@@ -0,0 +1,232 @@
|
|||||||
|
const assert = require('node:assert/strict');
|
||||||
|
const { test } = require('node:test');
|
||||||
|
|
||||||
|
const {
|
||||||
|
CopilotFailureDiagnosisCancellationNotFoundError,
|
||||||
|
} = require('@qinglong/ai/failure-diagnosis-cancellation');
|
||||||
|
const {
|
||||||
|
CLUSTER_RUN_CANCELLATION_SCHEMA,
|
||||||
|
ClusterRunCancellationFenceRejectedError,
|
||||||
|
} = require('@qinglong/runtime-core/cluster-run-cancellation');
|
||||||
|
const {
|
||||||
|
CLUSTER_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_RESPONSE_SCHEMA,
|
||||||
|
CLUSTER_CONTROL_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_ROUTE,
|
||||||
|
createClusterControlCopilotFailureDiagnosisCancellationRoute,
|
||||||
|
} = require('@qinglong/cluster-control/copilot-cancellation-route');
|
||||||
|
|
||||||
|
function authorized(body, overrides = {}) {
|
||||||
|
return {
|
||||||
|
request: {
|
||||||
|
requestId: 'transport-request-1',
|
||||||
|
method: 'POST',
|
||||||
|
path: '/api/v3/projects/project-1/runs/source-run-1/copilot/failure-diagnoses/diagnosis-request-1/cancellation',
|
||||||
|
query: {},
|
||||||
|
headers: {},
|
||||||
|
signal: new AbortController().signal,
|
||||||
|
body,
|
||||||
|
},
|
||||||
|
principal: {
|
||||||
|
subject: { type: 'user', id: 'owner-1' },
|
||||||
|
authenticationId: 'credential-1',
|
||||||
|
authenticatedAtMs: 1,
|
||||||
|
expiresAtMs: 10_000,
|
||||||
|
assurance: 'multi_factor',
|
||||||
|
},
|
||||||
|
operationId: 'copilot.failure_diagnosis.cancel',
|
||||||
|
permission: 'run.stop',
|
||||||
|
projectId: 'project-1',
|
||||||
|
policyFence: { projectVersion: 3, bindingVersion: 7 },
|
||||||
|
...overrides,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const parameters = {
|
||||||
|
projectId: 'project-1',
|
||||||
|
runId: 'source-run-1',
|
||||||
|
requestId: 'diagnosis-request-1',
|
||||||
|
};
|
||||||
|
|
||||||
|
function body(overrides = {}) {
|
||||||
|
return {
|
||||||
|
schema: CLUSTER_RUN_CANCELLATION_SCHEMA,
|
||||||
|
mutationId: '11111111-1111-4111-8111-111111111111',
|
||||||
|
...overrides,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function result(overrides = {}) {
|
||||||
|
return {
|
||||||
|
schema: 'qinglong/copilot-failure-diagnosis-cancellation-result@v1',
|
||||||
|
status: 'accepted',
|
||||||
|
convergence: 'terminal',
|
||||||
|
projectId: 'project-1',
|
||||||
|
sourceRunId: 'source-run-1',
|
||||||
|
requestId: 'diagnosis-request-1',
|
||||||
|
diagnosisRunId: 'diagnosis-run-1',
|
||||||
|
runStatus: 'cancelled',
|
||||||
|
outcome: 'cancelled',
|
||||||
|
runVersion: 7,
|
||||||
|
eventSequence: 7,
|
||||||
|
cancelRequestedAtMs: 500,
|
||||||
|
cancelReason: 'user',
|
||||||
|
...overrides,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
test('defines an exact run.stop route and passes only fenced target facts', async () => {
|
||||||
|
let command;
|
||||||
|
const request = authorized(body());
|
||||||
|
const route = createClusterControlCopilotFailureDiagnosisCancellationRoute(
|
||||||
|
{
|
||||||
|
async cancel(value) {
|
||||||
|
command = value;
|
||||||
|
return result();
|
||||||
|
},
|
||||||
|
},
|
||||||
|
() => '22222222-2222-4222-8222-222222222222',
|
||||||
|
);
|
||||||
|
const response = await route.handle(request, parameters);
|
||||||
|
assert.deepEqual(
|
||||||
|
CLUSTER_CONTROL_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_ROUTE,
|
||||||
|
{
|
||||||
|
method: 'POST',
|
||||||
|
path: '/api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses/{requestId}/cancellation',
|
||||||
|
operationId: 'copilot.failure_diagnosis.cancel',
|
||||||
|
permission: 'run.stop',
|
||||||
|
projectParameter: 'projectId',
|
||||||
|
},
|
||||||
|
);
|
||||||
|
assert.deepEqual(command, {
|
||||||
|
projectId: 'project-1',
|
||||||
|
sourceRunId: 'source-run-1',
|
||||||
|
requestId: 'diagnosis-request-1',
|
||||||
|
mutationId: '11111111-1111-4111-8111-111111111111',
|
||||||
|
eventId: '22222222-2222-4222-8222-222222222222',
|
||||||
|
subject: request.principal.subject,
|
||||||
|
policyFence: request.policyFence,
|
||||||
|
});
|
||||||
|
assert.deepEqual(response, {
|
||||||
|
statusCode: 202,
|
||||||
|
body: {
|
||||||
|
schema: CLUSTER_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_RESPONSE_SCHEMA,
|
||||||
|
status: 'accepted',
|
||||||
|
convergence: 'terminal',
|
||||||
|
projectId: 'project-1',
|
||||||
|
sourceRunId: 'source-run-1',
|
||||||
|
requestId: 'diagnosis-request-1',
|
||||||
|
diagnosisRunId: 'diagnosis-run-1',
|
||||||
|
runStatus: 'cancelled',
|
||||||
|
outcome: 'cancelled',
|
||||||
|
runVersion: 7,
|
||||||
|
eventSequence: 7,
|
||||||
|
cancelRequestedAtMs: 500,
|
||||||
|
cancelReason: 'user',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('projects an in-flight durable intent without claiming Provider abort', async () => {
|
||||||
|
const route = createClusterControlCopilotFailureDiagnosisCancellationRoute(
|
||||||
|
{
|
||||||
|
async cancel() {
|
||||||
|
return result({
|
||||||
|
status: 'already_requested',
|
||||||
|
convergence: 'model_in_flight',
|
||||||
|
runStatus: 'running',
|
||||||
|
outcome: null,
|
||||||
|
runVersion: 6,
|
||||||
|
eventSequence: 6,
|
||||||
|
});
|
||||||
|
},
|
||||||
|
},
|
||||||
|
() => '22222222-2222-4222-8222-222222222222',
|
||||||
|
);
|
||||||
|
const response = await route.handle(authorized(body()), parameters);
|
||||||
|
assert.equal(response.statusCode, 200);
|
||||||
|
assert.equal(response.body.convergence, 'model_in_flight');
|
||||||
|
assert.equal(response.body.runStatus, 'running');
|
||||||
|
assert.equal(response.body.outcome, null);
|
||||||
|
assert.equal('providerAborted' in response.body, false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('rejects non-exact bodies before invoking the capability', async () => {
|
||||||
|
let calls = 0;
|
||||||
|
const route = createClusterControlCopilotFailureDiagnosisCancellationRoute(
|
||||||
|
{
|
||||||
|
async cancel() {
|
||||||
|
calls += 1;
|
||||||
|
return result();
|
||||||
|
},
|
||||||
|
},
|
||||||
|
() => '22222222-2222-4222-8222-222222222222',
|
||||||
|
);
|
||||||
|
for (const value of [
|
||||||
|
null,
|
||||||
|
{},
|
||||||
|
body({ runId: 'caller-selected' }),
|
||||||
|
body({ reason: 'timeout' }),
|
||||||
|
body({ mutationId: '' }),
|
||||||
|
]) {
|
||||||
|
const response = await route.handle(authorized(value), parameters);
|
||||||
|
assert.equal(response.statusCode, 400);
|
||||||
|
}
|
||||||
|
assert.equal(calls, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('fails closed on widened or identity-drifted capability results', async () => {
|
||||||
|
for (const value of [
|
||||||
|
result({ sourceRunId: 'other' }),
|
||||||
|
result({ privateProvider: 'must-not-cross' }),
|
||||||
|
result({ runVersion: 8 }),
|
||||||
|
result({ convergence: 'model_in_flight' }),
|
||||||
|
]) {
|
||||||
|
const route = createClusterControlCopilotFailureDiagnosisCancellationRoute(
|
||||||
|
{
|
||||||
|
async cancel() {
|
||||||
|
return value;
|
||||||
|
},
|
||||||
|
},
|
||||||
|
() => '22222222-2222-4222-8222-222222222222',
|
||||||
|
);
|
||||||
|
const response = await route.handle(authorized(body()), parameters);
|
||||||
|
assert.deepEqual(response, {
|
||||||
|
statusCode: 503,
|
||||||
|
body: {
|
||||||
|
code: 'copilot_failure_diagnosis_cancellation_unavailable',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('maps hidden targets, Policy races and storage failures to stable codes', async () => {
|
||||||
|
for (const [error, statusCode, code] of [
|
||||||
|
[
|
||||||
|
new CopilotFailureDiagnosisCancellationNotFoundError(),
|
||||||
|
404,
|
||||||
|
'copilot_failure_diagnosis_not_found',
|
||||||
|
],
|
||||||
|
[
|
||||||
|
new ClusterRunCancellationFenceRejectedError('authorization_changed'),
|
||||||
|
409,
|
||||||
|
'copilot_failure_diagnosis_cancellation_fence_rejected',
|
||||||
|
],
|
||||||
|
[
|
||||||
|
new Error('private storage detail'),
|
||||||
|
503,
|
||||||
|
'copilot_failure_diagnosis_cancellation_unavailable',
|
||||||
|
],
|
||||||
|
]) {
|
||||||
|
const route = createClusterControlCopilotFailureDiagnosisCancellationRoute(
|
||||||
|
{
|
||||||
|
async cancel() {
|
||||||
|
throw error;
|
||||||
|
},
|
||||||
|
},
|
||||||
|
() => '22222222-2222-4222-8222-222222222222',
|
||||||
|
);
|
||||||
|
const response = await route.handle(authorized(body()), parameters);
|
||||||
|
assert.equal(response.statusCode, statusCode);
|
||||||
|
assert.equal(response.body.code, code);
|
||||||
|
assert.equal(JSON.stringify(response).includes('private'), false);
|
||||||
|
}
|
||||||
|
});
|
||||||
@@ -732,6 +732,7 @@ test('optionally exposes Prompt execution behind shared admission and policy', a
|
|||||||
'copilot.failure_diagnosis.execute',
|
'copilot.failure_diagnosis.execute',
|
||||||
'copilot.failure_diagnosis.read',
|
'copilot.failure_diagnosis.read',
|
||||||
'copilot.failure_diagnosis.output.read',
|
'copilot.failure_diagnosis.output.read',
|
||||||
|
'copilot.failure_diagnosis.cancel',
|
||||||
]);
|
]);
|
||||||
const response = await invoke(
|
const response = await invoke(
|
||||||
stack,
|
stack,
|
||||||
@@ -762,6 +763,7 @@ test('optionally exposes Prompt execution behind shared admission and policy', a
|
|||||||
test('optionally exposes Copilot diagnosis behind shared authentication, Policy and audit', async () => {
|
test('optionally exposes Copilot diagnosis behind shared authentication, Policy and audit', async () => {
|
||||||
const { events, input } = fixture();
|
const { events, input } = fixture();
|
||||||
let command;
|
let command;
|
||||||
|
let cancellationCommand;
|
||||||
const capability = {
|
const capability = {
|
||||||
async execute(value) {
|
async execute(value) {
|
||||||
command = value;
|
command = value;
|
||||||
@@ -803,11 +805,30 @@ test('optionally exposes Copilot diagnosis behind shared authentication, Policy
|
|||||||
requestId: value.requestId,
|
requestId: value.requestId,
|
||||||
};
|
};
|
||||||
},
|
},
|
||||||
|
async cancel(value) {
|
||||||
|
cancellationCommand = value;
|
||||||
|
return {
|
||||||
|
schema: 'qinglong/copilot-failure-diagnosis-cancellation-result@v1',
|
||||||
|
status: 'accepted',
|
||||||
|
convergence: 'terminal',
|
||||||
|
projectId: value.projectId,
|
||||||
|
sourceRunId: value.sourceRunId,
|
||||||
|
requestId: value.requestId,
|
||||||
|
diagnosisRunId: 'diagnosis-run-1',
|
||||||
|
runStatus: 'cancelled',
|
||||||
|
outcome: 'cancelled',
|
||||||
|
runVersion: 7,
|
||||||
|
eventSequence: 7,
|
||||||
|
cancelRequestedAtMs: 2_000,
|
||||||
|
cancelReason: 'user',
|
||||||
|
};
|
||||||
|
},
|
||||||
};
|
};
|
||||||
const stack = createProductionClusterControlApplicationStack(input, {
|
const stack = createProductionClusterControlApplicationStack(input, {
|
||||||
copilotFailureDiagnosis: {
|
copilotFailureDiagnosis: {
|
||||||
capability,
|
capability,
|
||||||
readCapability: capability,
|
readCapability: capability,
|
||||||
|
cancellationCapability: capability,
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
const result = await invoke(
|
const result = await invoke(
|
||||||
@@ -849,6 +870,25 @@ test('optionally exposes Copilot diagnosis behind shared authentication, Policy
|
|||||||
);
|
);
|
||||||
assert.equal(inspection.statusCode, 404);
|
assert.equal(inspection.statusCode, 404);
|
||||||
assert.equal(output.statusCode, 404);
|
assert.equal(output.statusCode, 404);
|
||||||
|
const cancellation = await invoke(
|
||||||
|
stack,
|
||||||
|
metadata(
|
||||||
|
'/api/v3/projects/project-1/runs/run-1/copilot/failure-diagnoses/diagnosis-request-1/cancellation',
|
||||||
|
'POST',
|
||||||
|
{
|
||||||
|
schema: 'qinglong/run-cancellation@v1',
|
||||||
|
mutationId: '00000000-0000-4000-8000-000000000099',
|
||||||
|
},
|
||||||
|
),
|
||||||
|
);
|
||||||
|
assert.equal(cancellation.statusCode, 202);
|
||||||
|
assert.equal(cancellationCommand.projectId, 'project-1');
|
||||||
|
assert.equal(cancellationCommand.sourceRunId, 'run-1');
|
||||||
|
assert.equal(cancellationCommand.requestId, 'diagnosis-request-1');
|
||||||
|
assert.deepEqual(cancellationCommand.policyFence, {
|
||||||
|
projectVersion: 3,
|
||||||
|
bindingVersion: 7,
|
||||||
|
});
|
||||||
assert.equal(
|
assert.equal(
|
||||||
events.includes('audit:copilot.failure_diagnosis.read:allowed'),
|
events.includes('audit:copilot.failure_diagnosis.read:allowed'),
|
||||||
true,
|
true,
|
||||||
@@ -857,6 +897,10 @@ test('optionally exposes Copilot diagnosis behind shared authentication, Policy
|
|||||||
events.includes('audit:copilot.failure_diagnosis.output.read:allowed'),
|
events.includes('audit:copilot.failure_diagnosis.output.read:allowed'),
|
||||||
true,
|
true,
|
||||||
);
|
);
|
||||||
|
assert.equal(
|
||||||
|
events.includes('audit:copilot.failure_diagnosis.cancel:allowed'),
|
||||||
|
true,
|
||||||
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
test('keeps the Copilot route absent by default and never invokes it after Policy denial', async () => {
|
test('keeps the Copilot route absent by default and never invokes it after Policy denial', async () => {
|
||||||
@@ -879,6 +923,7 @@ test('keeps the Copilot route absent by default and never invokes it after Polic
|
|||||||
for (const path of [
|
for (const path of [
|
||||||
'/api/v3/projects/project-1/runs/run-1/copilot/failure-diagnoses/diagnosis-request-1',
|
'/api/v3/projects/project-1/runs/run-1/copilot/failure-diagnoses/diagnosis-request-1',
|
||||||
'/api/v3/projects/project-1/runs/run-1/copilot/failure-diagnoses/diagnosis-request-1/output',
|
'/api/v3/projects/project-1/runs/run-1/copilot/failure-diagnoses/diagnosis-request-1/output',
|
||||||
|
'/api/v3/projects/project-1/runs/run-1/copilot/failure-diagnoses/diagnosis-request-1/cancellation',
|
||||||
]) {
|
]) {
|
||||||
await assert.rejects(
|
await assert.rejects(
|
||||||
defaultStack.admission.prepare(metadata(path)),
|
defaultStack.admission.prepare(metadata(path)),
|
||||||
|
|||||||
@@ -1165,6 +1165,7 @@ function auditSourceImports(root, packagePath, findings) {
|
|||||||
'src/application-runtime/aiProductionApplication.ts': Object.freeze([
|
'src/application-runtime/aiProductionApplication.ts': Object.freeze([
|
||||||
'@qinglong/ai/durable-model-invocation',
|
'@qinglong/ai/durable-model-invocation',
|
||||||
'@qinglong/ai/failure-diagnosis-application',
|
'@qinglong/ai/failure-diagnosis-application',
|
||||||
|
'@qinglong/ai/failure-diagnosis-cancellation',
|
||||||
'@qinglong/ai/failure-diagnosis-model-execution',
|
'@qinglong/ai/failure-diagnosis-model-execution',
|
||||||
'@qinglong/ai/plugin-package-prompt-output-projected-keyring',
|
'@qinglong/ai/plugin-package-prompt-output-projected-keyring',
|
||||||
'@qinglong/ai/postgres-model-provider-credential-storage',
|
'@qinglong/ai/postgres-model-provider-credential-storage',
|
||||||
@@ -1193,6 +1194,14 @@ function auditSourceImports(root, packagePath, findings) {
|
|||||||
'@qinglong/ai/postgres-failure-diagnosis-admission-storage',
|
'@qinglong/ai/postgres-failure-diagnosis-admission-storage',
|
||||||
'@qinglong/ai/postgres-failure-diagnosis-model-execution-storage',
|
'@qinglong/ai/postgres-failure-diagnosis-model-execution-storage',
|
||||||
]),
|
]),
|
||||||
|
'src/application-runtime/copilot/failureDiagnosisCancellationComposition.ts':
|
||||||
|
Object.freeze([
|
||||||
|
'@qinglong/ai/failure-diagnosis-cancellation',
|
||||||
|
'@qinglong/ai/failure-diagnosis-pre-model-terminalization',
|
||||||
|
'@qinglong/ai/postgres-failure-diagnosis-admission-storage',
|
||||||
|
]),
|
||||||
|
'src/copilot/failure-diagnosis/failureDiagnosisCancellationRoute.ts':
|
||||||
|
Object.freeze(['@qinglong/ai/failure-diagnosis-cancellation']),
|
||||||
});
|
});
|
||||||
const sourceRelative = path.relative(packageDirectory, filePath);
|
const sourceRelative = path.relative(packageDirectory, filePath);
|
||||||
if (!clusterAiImports[sourceRelative]?.includes(specifier)) {
|
if (!clusterAiImports[sourceRelative]?.includes(specifier)) {
|
||||||
|
|||||||
@@ -88,6 +88,9 @@ const {
|
|||||||
const {
|
const {
|
||||||
CopilotFailureDiagnosisReadService,
|
CopilotFailureDiagnosisReadService,
|
||||||
} = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/read-model/service.js');
|
} = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/read-model/service.js');
|
||||||
|
const {
|
||||||
|
CopilotFailureDiagnosisCancellationService,
|
||||||
|
} = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/cancellation/service.js');
|
||||||
const {
|
const {
|
||||||
BUILTIN_RUN_LOG_EXCERPT_TOOL,
|
BUILTIN_RUN_LOG_EXCERPT_TOOL,
|
||||||
BUILTIN_RUN_LOG_EXCERPT_TOOL_DEFINITION,
|
BUILTIN_RUN_LOG_EXCERPT_TOOL_DEFINITION,
|
||||||
@@ -1807,6 +1810,34 @@ async function copilotFailureDiagnosisTerminalizationFacts(pool, requestId) {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function copilotFailureDiagnosisCancellationFacts(pool, requestId) {
|
||||||
|
const terminalization = await copilotFailureDiagnosisTerminalizationFacts(
|
||||||
|
pool,
|
||||||
|
requestId,
|
||||||
|
);
|
||||||
|
const result = await pool.query(
|
||||||
|
`SELECT run.cancel_requested_at_ms AS "cancelRequestedAtMs",
|
||||||
|
run.cancel_reason AS "cancelReason",
|
||||||
|
(SELECT count(*)::integer FROM "ql3"."run_events"
|
||||||
|
WHERE run_id = admission.run_id
|
||||||
|
AND type = 'run.cancel_requested') AS "cancelEventCount",
|
||||||
|
(SELECT count(*)::integer
|
||||||
|
FROM "ql3_ai"."model_invocation_starts"
|
||||||
|
WHERE invocation_id = admission.plan_json->>'modelInvocationId')
|
||||||
|
AS "modelStartCount",
|
||||||
|
(SELECT count(*)::integer
|
||||||
|
FROM "ql3_ai"."model_invocation_completions"
|
||||||
|
WHERE invocation_id = admission.plan_json->>'modelInvocationId')
|
||||||
|
AS "modelCompletionCount"
|
||||||
|
FROM "ql3_ai"."copilot_failure_diagnosis_admissions" AS admission
|
||||||
|
JOIN "ql3"."runs" AS run ON run.id = admission.run_id
|
||||||
|
WHERE admission.request_id = $1`,
|
||||||
|
[requestId],
|
||||||
|
);
|
||||||
|
assert.equal(result.rowCount, 1);
|
||||||
|
return { ...terminalization, ...result.rows[0] };
|
||||||
|
}
|
||||||
|
|
||||||
async function copilotFailureDiagnosisModelResolutionFacts(pool, requestId) {
|
async function copilotFailureDiagnosisModelResolutionFacts(pool, requestId) {
|
||||||
const result = await pool.query(
|
const result = await pool.query(
|
||||||
`SELECT admission.request_id AS "requestId",
|
`SELECT admission.request_id AS "requestId",
|
||||||
@@ -2646,6 +2677,134 @@ async function terminalizeCopilotFailureDiagnosisLogUnavailableHaEvidence(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function cancelCopilotFailureDiagnosisHaEvidence(options) {
|
||||||
|
const { port, report, fixture, expectedExisting = false } = options;
|
||||||
|
const runtimeDatabase = await databaseOpener(
|
||||||
|
'runtime',
|
||||||
|
databaseUrl(RUNTIME_USER, RUNTIME_PASSWORD, port),
|
||||||
|
expectedExisting
|
||||||
|
? 'ql3-ha-copilot-cancellation-promoted'
|
||||||
|
: 'ql3-ha-copilot-cancellation-primary',
|
||||||
|
)();
|
||||||
|
let promotedClient;
|
||||||
|
try {
|
||||||
|
const pool = expectedExisting
|
||||||
|
? await (async () => {
|
||||||
|
promotedClient = await runtimeDatabase.pool.connect();
|
||||||
|
await promotedClient.query(`SET synchronous_commit = 'local'`);
|
||||||
|
return {
|
||||||
|
query: promotedClient.query.bind(promotedClient),
|
||||||
|
async connect() {
|
||||||
|
return {
|
||||||
|
query: promotedClient.query.bind(promotedClient),
|
||||||
|
release() {},
|
||||||
|
};
|
||||||
|
},
|
||||||
|
};
|
||||||
|
})()
|
||||||
|
: runtimeDatabase.pool;
|
||||||
|
const admissions = new PostgresCopilotFailureDiagnosisAdmissionRepository(
|
||||||
|
pool,
|
||||||
|
);
|
||||||
|
const terminalizations =
|
||||||
|
new PostgresCopilotFailureDiagnosisPreModelTerminalizationRepository(
|
||||||
|
pool,
|
||||||
|
);
|
||||||
|
const service = new CopilotFailureDiagnosisCancellationService({
|
||||||
|
admissions,
|
||||||
|
cancellations: new PostgresClusterRunCancellationRepository(pool),
|
||||||
|
terminalizations: { repository: terminalizations },
|
||||||
|
terminalizeBeforeModel: terminalizeCopilotFailureDiagnosisBeforeModel,
|
||||||
|
});
|
||||||
|
const mutationId = report.cancellationMutationId ?? randomUUID();
|
||||||
|
const command = {
|
||||||
|
projectId: fixture.plan.projectId,
|
||||||
|
sourceRunId: fixture.plan.source.runId,
|
||||||
|
requestId: fixture.plan.requestId,
|
||||||
|
mutationId,
|
||||||
|
eventId: randomUUID(),
|
||||||
|
subject: fixture.principal.subject,
|
||||||
|
policyFence: fixture.plan.policyFence,
|
||||||
|
};
|
||||||
|
if (!expectedExisting) {
|
||||||
|
await assert.rejects(
|
||||||
|
service.cancel({ ...command, sourceRunId: 'other-source-run' }),
|
||||||
|
(error) =>
|
||||||
|
error?.code === 'COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_NOT_FOUND',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const first = await service.cancel(command);
|
||||||
|
const replay = await service.cancel({ ...command, eventId: randomUUID() });
|
||||||
|
assert.equal(
|
||||||
|
first.status,
|
||||||
|
expectedExisting ? 'already_terminal' : 'accepted',
|
||||||
|
);
|
||||||
|
assert.equal(first.convergence, 'terminal');
|
||||||
|
assert.equal(first.runStatus, 'cancelled');
|
||||||
|
assert.equal(first.outcome, 'cancelled');
|
||||||
|
assert.equal(replay.status, 'already_terminal');
|
||||||
|
assert.equal(replay.convergence, 'terminal');
|
||||||
|
assert.equal(replay.runStatus, 'cancelled');
|
||||||
|
const facts = await copilotFailureDiagnosisCancellationFacts(
|
||||||
|
runtimeDatabase.pool,
|
||||||
|
report.requestId,
|
||||||
|
);
|
||||||
|
assert.deepEqual(
|
||||||
|
{
|
||||||
|
reason: facts.reason,
|
||||||
|
outcome: facts.outcome,
|
||||||
|
runStatus: facts.runStatus,
|
||||||
|
runVersion: facts.runVersion,
|
||||||
|
runEventSequence: facts.runEventSequence,
|
||||||
|
eventCount: facts.eventCount,
|
||||||
|
mutationCount: facts.mutationCount,
|
||||||
|
terminalizationCount: facts.terminalizationCount,
|
||||||
|
toolStatus: facts.toolStatus,
|
||||||
|
modelStatus: facts.modelStatus,
|
||||||
|
cancelReason: facts.cancelReason,
|
||||||
|
cancelEventCount: facts.cancelEventCount,
|
||||||
|
modelStartCount: facts.modelStartCount,
|
||||||
|
modelCompletionCount: facts.modelCompletionCount,
|
||||||
|
receiptContainsLogContent: facts.receiptContainsLogContent,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
reason: 'cancellation_requested',
|
||||||
|
outcome: 'cancelled',
|
||||||
|
runStatus: 'cancelled',
|
||||||
|
runVersion: 7,
|
||||||
|
runEventSequence: 7,
|
||||||
|
eventCount: 7,
|
||||||
|
mutationCount: 4,
|
||||||
|
terminalizationCount: 1,
|
||||||
|
toolStatus: 'cancelled',
|
||||||
|
modelStatus: 'cancelled',
|
||||||
|
cancelReason: 'user',
|
||||||
|
cancelEventCount: 1,
|
||||||
|
modelStartCount: 0,
|
||||||
|
modelCompletionCount: 0,
|
||||||
|
receiptContainsLogContent: false,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
if (expectedExisting) {
|
||||||
|
report.afterPromotion = facts;
|
||||||
|
report.survivedPromotion = true;
|
||||||
|
report.promotedExactReplay = true;
|
||||||
|
report.promotedProviderCalls = 0;
|
||||||
|
} else {
|
||||||
|
report.cancellationMutationId = mutationId;
|
||||||
|
report.beforePromotion = facts;
|
||||||
|
report.cancellationExactReplay = true;
|
||||||
|
report.crossTargetHidden = true;
|
||||||
|
report.providerCalls = 0;
|
||||||
|
report.contentFree = true;
|
||||||
|
}
|
||||||
|
return facts;
|
||||||
|
} finally {
|
||||||
|
promotedClient?.release();
|
||||||
|
await runtimeDatabase.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function resolveCopilotFailureDiagnosisUnknownModelHaEvidence(options) {
|
async function resolveCopilotFailureDiagnosisUnknownModelHaEvidence(options) {
|
||||||
const {
|
const {
|
||||||
port,
|
port,
|
||||||
@@ -11233,6 +11392,8 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
let copilotFailureDiagnosisModelResolutionFixture;
|
let copilotFailureDiagnosisModelResolutionFixture;
|
||||||
let copilotFailureDiagnosisRead;
|
let copilotFailureDiagnosisRead;
|
||||||
let copilotFailureDiagnosisReadFixture;
|
let copilotFailureDiagnosisReadFixture;
|
||||||
|
let copilotFailureDiagnosisCancellation;
|
||||||
|
let copilotFailureDiagnosisCancellationFixture;
|
||||||
let modelProviderCredentialCatalog;
|
let modelProviderCredentialCatalog;
|
||||||
let modelProviderCredentialTestConnection;
|
let modelProviderCredentialTestConnection;
|
||||||
let runAttemptLogRetentionEvidence;
|
let runAttemptLogRetentionEvidence;
|
||||||
@@ -11449,6 +11610,13 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
});
|
});
|
||||||
copilotFailureDiagnosisRead = copilotRead.report;
|
copilotFailureDiagnosisRead = copilotRead.report;
|
||||||
copilotFailureDiagnosisReadFixture = copilotRead.fixture;
|
copilotFailureDiagnosisReadFixture = copilotRead.fixture;
|
||||||
|
const copilotCancellation =
|
||||||
|
await runCopilotFailureDiagnosisAdmissionHaEvidence({
|
||||||
|
primaryPort,
|
||||||
|
migrationPool: migrationDatabase.pool,
|
||||||
|
});
|
||||||
|
copilotFailureDiagnosisCancellation = copilotCancellation.report;
|
||||||
|
copilotFailureDiagnosisCancellationFixture = copilotCancellation.fixture;
|
||||||
} finally {
|
} finally {
|
||||||
await migrationDatabase.close();
|
await migrationDatabase.close();
|
||||||
}
|
}
|
||||||
@@ -11903,6 +12071,26 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
state: 'copilot_failure_diagnosis_pre_model_terminalized',
|
state: 'copilot_failure_diagnosis_pre_model_terminalized',
|
||||||
atMs: Number((performance.now() - startedAt).toFixed(3)),
|
atMs: Number((performance.now() - startedAt).toFixed(3)),
|
||||||
});
|
});
|
||||||
|
await cancelCopilotFailureDiagnosisHaEvidence({
|
||||||
|
port: primaryPort,
|
||||||
|
report: copilotFailureDiagnosisCancellation,
|
||||||
|
fixture: copilotFailureDiagnosisCancellationFixture,
|
||||||
|
});
|
||||||
|
await waitFor(async () => {
|
||||||
|
const replicated = await copilotFailureDiagnosisCancellationFacts(
|
||||||
|
standbyDatabase.pool,
|
||||||
|
copilotFailureDiagnosisCancellation.requestId,
|
||||||
|
);
|
||||||
|
return JSON.stringify(replicated) ===
|
||||||
|
JSON.stringify(copilotFailureDiagnosisCancellation.beforePromotion)
|
||||||
|
? replicated
|
||||||
|
: null;
|
||||||
|
}, 'Copilot failure diagnosis cancellation WAL replay');
|
||||||
|
copilotFailureDiagnosisCancellation.replicatedBeforePromotion = true;
|
||||||
|
timeline.push({
|
||||||
|
state: 'copilot_failure_diagnosis_cancellation_terminalized',
|
||||||
|
atMs: Number((performance.now() - startedAt).toFixed(3)),
|
||||||
|
});
|
||||||
await resolveCopilotFailureDiagnosisUnknownModelHaEvidence({
|
await resolveCopilotFailureDiagnosisUnknownModelHaEvidence({
|
||||||
port: primaryPort,
|
port: primaryPort,
|
||||||
report: copilotFailureDiagnosisModelResolution,
|
report: copilotFailureDiagnosisModelResolution,
|
||||||
@@ -12846,6 +13034,12 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
outputKeyByte: 0x74,
|
outputKeyByte: 0x74,
|
||||||
expectedExisting: true,
|
expectedExisting: true,
|
||||||
});
|
});
|
||||||
|
await cancelCopilotFailureDiagnosisHaEvidence({
|
||||||
|
port: standbyPort,
|
||||||
|
report: copilotFailureDiagnosisCancellation,
|
||||||
|
fixture: copilotFailureDiagnosisCancellationFixture,
|
||||||
|
expectedExisting: true,
|
||||||
|
});
|
||||||
await verifyModelProviderCredentialCatalogAfterPromotion({
|
await verifyModelProviderCredentialCatalogAfterPromotion({
|
||||||
promotedPort: standbyPort,
|
promotedPort: standbyPort,
|
||||||
promotedPool: promotedDatabase.pool,
|
promotedPool: promotedDatabase.pool,
|
||||||
@@ -13611,8 +13805,8 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
);
|
);
|
||||||
assert.deepEqual(sideEffects.rows, [
|
assert.deepEqual(sideEffects.rows, [
|
||||||
{
|
{
|
||||||
runs: 84,
|
runs: 86,
|
||||||
runEvents: 209,
|
runEvents: 216,
|
||||||
diagnosisReadRuns: 2,
|
diagnosisReadRuns: 2,
|
||||||
diagnosisReadRunEvents: 9,
|
diagnosisReadRunEvents: 9,
|
||||||
credentialDeliveries: 4,
|
credentialDeliveries: 4,
|
||||||
@@ -13734,6 +13928,7 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
copilotFailureDiagnosisTerminalization,
|
copilotFailureDiagnosisTerminalization,
|
||||||
copilotFailureDiagnosisModelResolution,
|
copilotFailureDiagnosisModelResolution,
|
||||||
copilotFailureDiagnosisRead,
|
copilotFailureDiagnosisRead,
|
||||||
|
copilotFailureDiagnosisCancellation,
|
||||||
modelProviderCredentialCatalog,
|
modelProviderCredentialCatalog,
|
||||||
modelProviderCredentialTestConnection,
|
modelProviderCredentialTestConnection,
|
||||||
runAttemptLogRetention,
|
runAttemptLogRetention,
|
||||||
@@ -13843,6 +14038,31 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
copilotFailureDiagnosisRead.promotedProviderCalls === 0 &&
|
copilotFailureDiagnosisRead.promotedProviderCalls === 0 &&
|
||||||
JSON.stringify(copilotFailureDiagnosisRead.afterPromotion) ===
|
JSON.stringify(copilotFailureDiagnosisRead.afterPromotion) ===
|
||||||
JSON.stringify(copilotFailureDiagnosisRead.beforePromotion),
|
JSON.stringify(copilotFailureDiagnosisRead.beforePromotion),
|
||||||
|
copilotFailureDiagnosisCancellationExactlyReplaysWithoutProviderCall:
|
||||||
|
copilotFailureDiagnosisCancellation.cancellationExactReplay &&
|
||||||
|
copilotFailureDiagnosisCancellation.crossTargetHidden &&
|
||||||
|
copilotFailureDiagnosisCancellation.providerCalls === 0 &&
|
||||||
|
copilotFailureDiagnosisCancellation.beforePromotion
|
||||||
|
.cancelEventCount === 1 &&
|
||||||
|
copilotFailureDiagnosisCancellation.beforePromotion
|
||||||
|
.terminalizationCount === 1 &&
|
||||||
|
copilotFailureDiagnosisCancellation.beforePromotion
|
||||||
|
.modelStartCount === 0 &&
|
||||||
|
copilotFailureDiagnosisCancellation.beforePromotion
|
||||||
|
.modelCompletionCount === 0 &&
|
||||||
|
copilotFailureDiagnosisCancellation.beforePromotion.runStatus ===
|
||||||
|
'cancelled',
|
||||||
|
copilotFailureDiagnosisCancellationIsContentFree:
|
||||||
|
copilotFailureDiagnosisCancellation.contentFree &&
|
||||||
|
copilotFailureDiagnosisCancellation.beforePromotion
|
||||||
|
.receiptContainsLogContent === false,
|
||||||
|
copilotFailureDiagnosisCancellationReplicatesAndSurvivesPromotion:
|
||||||
|
copilotFailureDiagnosisCancellation.replicatedBeforePromotion &&
|
||||||
|
copilotFailureDiagnosisCancellation.survivedPromotion &&
|
||||||
|
copilotFailureDiagnosisCancellation.promotedExactReplay &&
|
||||||
|
copilotFailureDiagnosisCancellation.promotedProviderCalls === 0 &&
|
||||||
|
JSON.stringify(copilotFailureDiagnosisCancellation.afterPromotion) ===
|
||||||
|
JSON.stringify(copilotFailureDiagnosisCancellation.beforePromotion),
|
||||||
modelProviderCredentialCatalogSurvivesPromotion:
|
modelProviderCredentialCatalogSurvivesPromotion:
|
||||||
modelProviderCredentialCatalog.survivedPromotion,
|
modelProviderCredentialCatalog.survivedPromotion,
|
||||||
modelProviderCredentialManagementIdentityLedgerSurvivesPromotion:
|
modelProviderCredentialManagementIdentityLedgerSurvivesPromotion:
|
||||||
|
|||||||
@@ -268,10 +268,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
|
|||||||
{
|
{
|
||||||
path: 'packages/ql3-ai',
|
path: 'packages/ql3-ai',
|
||||||
name: '@qinglong/ai',
|
name: '@qinglong/ai',
|
||||||
sourceFiles: 193,
|
sourceFiles: 194,
|
||||||
rootSourceFiles: 1,
|
rootSourceFiles: 1,
|
||||||
rootSourceLines: 16,
|
rootSourceLines: 16,
|
||||||
nestedSourceFiles: 192,
|
nestedSourceFiles: 193,
|
||||||
rootSourceFileHardCap: 1,
|
rootSourceFileHardCap: 1,
|
||||||
rootSourceLineHardCap: 16,
|
rootSourceLineHardCap: 16,
|
||||||
rootSourceFileRoles: {
|
rootSourceFileRoles: {
|
||||||
@@ -385,10 +385,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
|
|||||||
rootSourceFileRoles: clusterControl.rootSourceFileRoles,
|
rootSourceFileRoles: clusterControl.rootSourceFileRoles,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
sourceFiles: 62,
|
sourceFiles: 64,
|
||||||
rootSourceFiles: 2,
|
rootSourceFiles: 2,
|
||||||
rootSourceLines: 195,
|
rootSourceLines: 195,
|
||||||
nestedSourceFiles: 60,
|
nestedSourceFiles: 62,
|
||||||
rootSourceFileRoles: {
|
rootSourceFileRoles: {
|
||||||
'aiCli.ts': 'binary_entry',
|
'aiCli.ts': 'binary_entry',
|
||||||
'cli.ts': 'binary_entry',
|
'cli.ts': 'binary_entry',
|
||||||
|
|||||||
Reference in New Issue
Block a user