mirror of
https://github.com/whyour/qinglong.git
synced 2026-09-20 16:07:11 +08:00
feat(ql3): execute copilot diagnosis tools
This commit is contained in:
whyour
@@ -11,6 +11,7 @@
|
|||||||
|
|
||||||
最新增量证据（2026-08-15）：
|
最新增量证据（2026-08-15）：
|
||||||
|
|
||||||
|
- D-316/ADR-0408（已接受）：Cluster Copilot 现在能够从 ADR-0407 的 durable admission 恢复 exact `qinglong.run.log.excerpt@1.0.0` authority，复用通用 Trusted Tool start barrier、加密 success/failure completion、result catalog/rekey 和内建日志 adapter；确定性 start/completion identity 让 response-loss replay 直接打开既有证据，不重复读取日志或执行 adapter。Cluster invocation Artifact 使用独立 projected keyring，提供 active+historical material，但每次读取均重新执行 canonical path/symlink/mode/inode/realpath fence，且不取得 PostgreSQL Tool result generation authority。只有 exact `succeeded` completion 才能通过 `pg-9019` 的 SERIALIZABLE 事务把 Model Step 从 `pending` 原子推进到 `ready`，同时写 RunEvent、StepRunMutation 和 append-only unlock receipt；`failed|timed_out` 不解锁。本 Gate 不执行模型、不终态化 diagnosis Run，下一 Gate 是 ADR-0405 builder + Model Gateway + Copilot encrypted model completion/terminalization。实现仍为 18 个 package，无单文件/浅平 package；AI 175 个源码中 174 个、Cluster Control 56 个中 54 个位于嵌套目录，不新增依赖、进程、连接、timer/watcher/cache 或默认 Edge 成本。18-package clean build/test 全绿，AI 229 pass/3 条件 skip、backend 1,207 pass/2 条件 skip，四项架构审计零 finding；14 档 Local artifact 全通过，默认 Edge/Standalone 为 2,589,890/2,589,968 bytes。PostgreSQL 18.4 arm64 HA 130/130、timeline `1→2`，首次执行只读两次日志、密文不含敏感 fixture，提升后 exact replay 零日志读取；报告 SHA-256 为 `d525a303696e178d777b021b376729bd2c5382fb5eb7bc98466a2b79d3940517`，独立审计与 Docker 清理通过。
|
||||||
- D-315/ADR-0407（已接受）：Cluster Copilot 故障诊断不修改或重新打开终态源 Run，而是用源 Run/version、最新已结束 Attempt/status 与日志 Artifact 建立 exact fence，原子创建独立 `copilot_failure_diagnosis` Run、admission event、`ready` 的 `qinglong.run.log.excerpt@1.0.0` Tool Step、以其为父节点的 `pending` Model Step 和不可变 receipt。计划只接受 `cluster-control` 的 reviewed snapshot/binding/Policy/subject authority；`approval_required`、Tool contract/输入漂移或未显式允许的模型出口均在写库前失败关闭。`ql3_ai` 新增有界 admission ledger 与仅授予 runtime 的 `SECURITY DEFINER` source snapshot，在 SERIALIZABLE 事务内重验 active Project/binding 和源 Run/Attempt；response-loss replay 对 JSONB 做结构 exact 比较而不依赖无语义的对象键序。Copilot/Plugin 这类 StepRun 聚合从通用 Task orphan recovery 排除，后续由各自状态机恢复，不能伪造顶层 Attempt 或被错误终态化。本阶段只 admission，不执行 Tool/模型、不授予行动权；Tool encrypted completion、Model 解锁和 Copilot encrypted model completion 是下一 Gate。最终 18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，四项边界审计零 finding且 workspace 仍为 18 package、无单文件/浅平 package；14 档 Local Profile artifact 全通过，默认 Edge/Standalone 不引入 Cluster admission。PostgreSQL 18.4 arm64 HA 128/128、timeline `1→2`，报告 SHA-256 为 `a4ed1edec783e3f5b42507c0f8e11b94c59dbe44a57e691017d1445ec9d115e2`，证据审计与 Docker 清理通过。
|
- D-315/ADR-0407（已接受）：Cluster Copilot 故障诊断不修改或重新打开终态源 Run，而是用源 Run/version、最新已结束 Attempt/status 与日志 Artifact 建立 exact fence，原子创建独立 `copilot_failure_diagnosis` Run、admission event、`ready` 的 `qinglong.run.log.excerpt@1.0.0` Tool Step、以其为父节点的 `pending` Model Step 和不可变 receipt。计划只接受 `cluster-control` 的 reviewed snapshot/binding/Policy/subject authority；`approval_required`、Tool contract/输入漂移或未显式允许的模型出口均在写库前失败关闭。`ql3_ai` 新增有界 admission ledger 与仅授予 runtime 的 `SECURITY DEFINER` source snapshot，在 SERIALIZABLE 事务内重验 active Project/binding 和源 Run/Attempt；response-loss replay 对 JSONB 做结构 exact 比较而不依赖无语义的对象键序。Copilot/Plugin 这类 StepRun 聚合从通用 Task orphan recovery 排除，后续由各自状态机恢复，不能伪造顶层 Attempt 或被错误终态化。本阶段只 admission，不执行 Tool/模型、不授予行动权；Tool encrypted completion、Model 解锁和 Copilot encrypted model completion 是下一 Gate。最终 18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，四项边界审计零 finding且 workspace 仍为 18 package、无单文件/浅平 package；14 档 Local Profile artifact 全通过，默认 Edge/Standalone 不引入 Cluster admission。PostgreSQL 18.4 arm64 HA 128/128、timeline `1→2`，报告 SHA-256 为 `a4ed1edec783e3f5b42507c0f8e11b94c59dbe44a57e691017d1445ec9d115e2`，证据审计与 Docker 清理通过。
|
||||||
- D-314/ADR-0406（已接受）：Cluster Trusted Tool 的 encrypted completion 不再停留在 storage port。`@qinglong/cluster-control/trusted-tool-result-keyring` 新增只读 projected material authority：canonical v1 manifest 只含最多 16 个 canonical 32-byte key，不含 generation、active/state/retirement，provider 只有 `resolve(keyId)` 而没有 `active()`；因此 PostgreSQL `trusted-tool-results` catalog 仍是 active/decryptable 状态唯一 authority，completion 会以 catalog material proof 再次校验。runtime 每次调用重新执行 direct-root、in-root atomic symlink、single-link、64 KiB、只读/不可执行/other-inaccessible mode、dev/inode/size/mtime 与双 realpath fence，不持有 Kubernetes API、cache、watcher 或 timer。新增能力位于 Cluster Control 既有 `trusted-tool/key-management/`，并把 mounted Secret 与 keyring 的 projected-file/TOCTOU 逻辑收敛到 package-private `security/privateProjectedFile` 真源；公开 mounted Secret 行为不变，不新增 package、依赖、migration、连接、route 或默认 Profile importer。定向共享回归 7/7，Cluster Control 完整 234 pass/2 条条件 skip/0 fail；最终 18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，package/dependency/Edge import/Cluster deployment 四项审计零 finding。workspace 仍为 18 package、无单文件或浅平 package；Cluster Control 54 个源码中仅 2 个 binary entry 位于根层，52 个处于嵌套领域目录。14 档 Local Profile artifact 全部通过，默认 Edge/Standalone 保持 2,589,812/2,589,890 bytes，AI 保持 3,121,108/3,121,198 bytes，MCP 保持 7,315,930/7,316,038 bytes。PostgreSQL 18.4 arm64 HA 125/125、timeline `1→2`，报告 SHA-256 为 `26c817647ed984d8d4627a7cae1c95de06017a5d6d32dd3dfd01414ba029e542`，证据审计与 Docker 容器/网络/卷零残留。下一 Gate 是独立 diagnosis Run 的 Tool/Model Step admission 与 Copilot encrypted model completion，不能借用终态源 Run 或 Plugin Prompt plan。
|
- D-314/ADR-0406（已接受）：Cluster Trusted Tool 的 encrypted completion 不再停留在 storage port。`@qinglong/cluster-control/trusted-tool-result-keyring` 新增只读 projected material authority：canonical v1 manifest 只含最多 16 个 canonical 32-byte key，不含 generation、active/state/retirement，provider 只有 `resolve(keyId)` 而没有 `active()`；因此 PostgreSQL `trusted-tool-results` catalog 仍是 active/decryptable 状态唯一 authority，completion 会以 catalog material proof 再次校验。runtime 每次调用重新执行 direct-root、in-root atomic symlink、single-link、64 KiB、只读/不可执行/other-inaccessible mode、dev/inode/size/mtime 与双 realpath fence，不持有 Kubernetes API、cache、watcher 或 timer。新增能力位于 Cluster Control 既有 `trusted-tool/key-management/`，并把 mounted Secret 与 keyring 的 projected-file/TOCTOU 逻辑收敛到 package-private `security/privateProjectedFile` 真源；公开 mounted Secret 行为不变，不新增 package、依赖、migration、连接、route 或默认 Profile importer。定向共享回归 7/7，Cluster Control 完整 234 pass/2 条条件 skip/0 fail；最终 18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，package/dependency/Edge import/Cluster deployment 四项审计零 finding。workspace 仍为 18 package、无单文件或浅平 package；Cluster Control 54 个源码中仅 2 个 binary entry 位于根层，52 个处于嵌套领域目录。14 档 Local Profile artifact 全部通过，默认 Edge/Standalone 保持 2,589,812/2,589,890 bytes，AI 保持 3,121,108/3,121,198 bytes，MCP 保持 7,315,930/7,316,038 bytes。PostgreSQL 18.4 arm64 HA 125/125、timeline `1→2`，报告 SHA-256 为 `26c817647ed984d8d4627a7cae1c95de06017a5d6d32dd3dfd01414ba029e542`，证据审计与 Docker 容器/网络/卷零残留。下一 Gate 是独立 diagnosis Run 的 Tool/Model Step admission 与 Copilot encrypted model completion，不能借用终态源 Run 或 Plugin Prompt plan。
|
||||||
- D-313/ADR-0405（已接受）：新增 `@qinglong/ai/failure-diagnosis-prompt`，把 ADR-0403 的潜在敏感日志投影收敛为固定 system instruction + canonical JSON data envelope；日志只存在于 `log.content` JSON string value，不能通过引号、换行、伪造 role/schema 或 delimiter 拼接出新 message。builder 重新校验完整 trust/redaction/profile byte 契约，拒绝伪造 `safe`、行动权、未知字段与 byte/signal drift；envelope 不带 Run/Attempt、Artifact、path、cursor 或 content digest。部署者必须通过 `qinglong/copilot-model-egress-policy@v1` 显式允许 `potentially_sensitive` 数据进入 `on_device|external` 边界并提供输入/output token 双预算，空 allowlist 与 external 未授权均在 Model Gateway/Provider I/O 前失败关闭。输出只含 content-free egress evidence，并固定要求模型 completion 继承潜在敏感、仅加密持久化、禁止明文审计且无行动权；真正 Cluster Trusted Tool/model completion 仍需后续组合门。能力以 `ql3-ai/src/copilot/failure-diagnosis/` 三个内聚文件和精确 subpath 交付，不新增 package、依赖、迁移、连接或常驻组件。定向 12/12、AI 221 pass/3 条件 skip/0 fail；最终 18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，四项结构/部署审计零 finding，14 档 Local Profile artifact 全部通过。默认 Edge/Standalone 保持 2,589,812/2,589,890 bytes、315 files、56 modules，Edge/Standalone AI 保持 3,121,108/3,121,198 bytes、368 files、61 modules，MCP 保持 7,315,930/7,316,038 bytes、801 files、226 modules，证明未装配 subpath 被完全裁掉。PostgreSQL 18.4 arm64 HA 125/125、timeline `1→2`，报告 SHA-256 为 `2bbc8bdd0d90e6ec9ce82d2afcaec817679dddb82860c5d405a09d5e5458bece`，证据审计与 Docker 零残留。
|
- D-313/ADR-0405（已接受）：新增 `@qinglong/ai/failure-diagnosis-prompt`，把 ADR-0403 的潜在敏感日志投影收敛为固定 system instruction + canonical JSON data envelope；日志只存在于 `log.content` JSON string value，不能通过引号、换行、伪造 role/schema 或 delimiter 拼接出新 message。builder 重新校验完整 trust/redaction/profile byte 契约，拒绝伪造 `safe`、行动权、未知字段与 byte/signal drift；envelope 不带 Run/Attempt、Artifact、path、cursor 或 content digest。部署者必须通过 `qinglong/copilot-model-egress-policy@v1` 显式允许 `potentially_sensitive` 数据进入 `on_device|external` 边界并提供输入/output token 双预算，空 allowlist 与 external 未授权均在 Model Gateway/Provider I/O 前失败关闭。输出只含 content-free egress evidence，并固定要求模型 completion 继承潜在敏感、仅加密持久化、禁止明文审计且无行动权；真正 Cluster Trusted Tool/model completion 仍需后续组合门。能力以 `ql3-ai/src/copilot/failure-diagnosis/` 三个内聚文件和精确 subpath 交付，不新增 package、依赖、迁移、连接或常驻组件。定向 12/12、AI 221 pass/3 条件 skip/0 fail；最终 18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，四项结构/部署审计零 finding，14 档 Local Profile artifact 全部通过。默认 Edge/Standalone 保持 2,589,812/2,589,890 bytes、315 files、56 modules，Edge/Standalone AI 保持 3,121,108/3,121,198 bytes、368 files、61 modules，MCP 保持 7,315,930/7,316,038 bytes、801 files、226 modules，证明未装配 subpath 被完全裁掉。PostgreSQL 18.4 arm64 HA 125/125、timeline `1→2`，报告 SHA-256 为 `2bbc8bdd0d90e6ec9ce82d2afcaec817679dddb82860c5d405a09d5e5458bece`，证据审计与 Docker 零残留。
|
||||||
@@ -6942,9 +6943,12 @@ ADR-0407 已完成该 admission Gate：源失败/超时 Run 和最新已结束 A
|
|||||||
独立 diagnosis Run 在一个 SERIALIZABLE 事务内同时获得 admission event、ready Tool Step、依赖它的
|
独立 diagnosis Run 在一个 SERIALIZABLE 事务内同时获得 admission event、ready Tool Step、依赖它的
|
||||||
pending Model Step 与 receipt；Cluster runtime 的 source snapshot authority 会重验 active Project/binding
|
pending Model Step 与 receipt；Cluster runtime 的 source snapshot authority 会重验 active Project/binding
|
||||||
和日志 Artifact，response-loss replay 以结构语义比较 JSONB。该 Run 是 StepRun 编排聚合，不进入通用
|
和日志 Artifact，response-loss replay 以结构语义比较 JSONB。该 Run 是 StepRun 编排聚合，不进入通用
|
||||||
Task orphan recovery。当前阶段仍没有执行 Tool 或模型；下一 Gate 必须先形成可信 encrypted Tool
|
Task orphan recovery。ADR-0408 已进一步关闭 Tool execution Gate：coordinator 从 durable admission 恢复
|
||||||
completion，再原子解锁 Model Step，并以 ADR-0405 egress policy 和 Copilot 专用 encrypted model
|
exact Tool authority，复用通用 start barrier 与 encrypted completion；首次执行有界读取日志，response-loss
|
||||||
completion 收敛最终 Run。
|
或主库提升后的 exact replay 不再执行 adapter。只有成功 completion 才由 `pg-9019` 的原子 unlock ledger
|
||||||
|
把 Model Step 推进到 ready，失败或超时保持 pending。当前阶段仍没有执行模型或终态化 diagnosis Run；
|
||||||
|
下一 Gate 必须以 ADR-0405 builder 和 Model Gateway 形成 Copilot 专用 encrypted model completion，随后
|
||||||
|
原子收敛 Model Step、Run、取消、deadline 与恢复语义。
|
||||||
|
|
||||||
## 17. Tool Registry
|
## 17. Tool Registry
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,91 @@
|
|||||||
|
# ADR-0408：Cluster Copilot Failure Diagnosis Tool Execution
|
||||||
|
|
||||||
|
- 状态：Accepted
|
||||||
|
- 日期：2026-08-15
|
||||||
|
- 关联 RFC：QL-RFC-0001 D-316、Phase 2
|
||||||
|
- 关联 ADR：ADR-0163、ADR-0226、ADR-0403、ADR-0405、ADR-0406、ADR-0407
|
||||||
|
|
||||||
|
## 问题
|
||||||
|
|
||||||
|
ADR-0407 只把故障诊断原子接纳为独立 Run、一个 `ready` Tool Step 和一个依赖它的
|
||||||
|
`pending` Model Step。系统仍不能执行该 Tool，也不能在崩溃、响应丢失或 PostgreSQL
|
||||||
|
主库切换后证明 Tool 是否已经产生副作用、结果是否已经加密持久化，以及 Model Step
|
||||||
|
是否可以安全解锁。
|
||||||
|
|
||||||
|
若 Copilot 绕过共享 Trusted Tool start/completion 协议直接读取日志，重放可能重复执行
|
||||||
|
adapter；若先解锁 Model 再写 completion，模型可能读取缺失、失败或未验证的结果；若复用
|
||||||
|
Prompt output key 或让 projected file 决定 Tool result active generation，又会合并本应独立的
|
||||||
|
密钥域和 durable authority。
|
||||||
|
|
||||||
|
## 决策
|
||||||
|
|
||||||
|
1. 新增 Copilot failure-diagnosis Tool execution coordinator。它只接受 ADR-0407 已持久化的
|
||||||
|
`requestId`，重新读取并规范化 exact admission plan/receipt、当前 Project Tool snapshot 和
|
||||||
|
invocation input/preview Artifact；任何 plan、Definition、binding、Artifact 或 source fence
|
||||||
|
漂移均在日志读取前失败关闭。
|
||||||
|
2. coordinator 复用 Runtime Core 的通用 Trusted Tool execution evidence、start barrier、
|
||||||
|
encrypted success/failure completion、result catalog 和 rekey 协议，并只装配内建
|
||||||
|
`qinglong.run.log.excerpt@1.0.0` adapter。start/completion 使用从 plan 稳定派生的 identity；
|
||||||
|
exact replay 首先打开 durable barrier/completion，不再次读取日志或执行 adapter。
|
||||||
|
3. invocation Artifact 使用独立的 Cluster projected keyring。它包含一个 active key 和最多
|
||||||
|
16 个历史 key，提供 `active()` 与 `resolve(keyId)`，每次调用重新读取 canonical 32-byte
|
||||||
|
material，不使用 cache、watcher 或 timer，并复用既有 projected-file 的 canonical path、
|
||||||
|
symlink、权限、inode/size/mtime 和双 realpath fence。Tool result keyring 仍保持 resolve-only，
|
||||||
|
PostgreSQL result catalog 继续独占 active/decryptable generation authority；两个密钥域不得互换。
|
||||||
|
4. 只有规范化且与 admission exact 绑定的 `succeeded` Tool completion 才能解锁 Model Step。
|
||||||
|
`pg-9019-ai-copilot-failure-diagnosis-tool-unlocks` 新增 append-only unlock ledger；一个
|
||||||
|
SERIALIZABLE 事务同时把 Model Step 从 `pending` 推进到 `ready`、递增 Run version/event
|
||||||
|
sequence、写 RunEvent、StepRunMutation 和不可变 unlock receipt。Tool `failed|timed_out`
|
||||||
|
completion 不解锁 Model，也不伪造最终 diagnosis 结论。
|
||||||
|
5. unlock transaction 重新锁定并验证 admission、success completion、当前 Model Step 和 running
|
||||||
|
diagnosis Run；serialization/deadlock 只在事务提交前有界重试。response-loss replay 必须返回
|
||||||
|
结构完全一致的 receipt；缺失关联事实、旧 fence、不同 digest 或部分写入全部视为冲突/损坏。
|
||||||
|
6. `ql3_runtime` 对 unlock ledger 只有 `SELECT, INSERT`，没有 `UPDATE, DELETE`；数据库外调用者
|
||||||
|
不能直接把 Model Step 改为 ready。Tool execution 与 PostgreSQL storage 分别通过精确 AI
|
||||||
|
subpath 发布，纯 execution 入口不隐式 re-export PostgreSQL repository。
|
||||||
|
7. 本 Gate 到 Model Step `ready` 为止，不调用模型、不持久化模型明文、不终态化 diagnosis Run。
|
||||||
|
ADR-0405 prompt builder、Model Gateway、Copilot 专用 encrypted model completion、失败/取消/
|
||||||
|
deadline terminalization 和产品 API 属于后续 Gate。
|
||||||
|
8. 能力内聚在既有 `@qinglong/ai` 与 `@qinglong/cluster-control` 的领域子目录，不新增 workspace
|
||||||
|
package、依赖、进程、Pod、连接池、listener、daemon、timer、watcher 或 cache。默认 Edge/
|
||||||
|
Standalone 不导入 Cluster execution；显式本地 AI 制品也不会取得 PostgreSQL/S3 authority。
|
||||||
|
|
||||||
|
## 被否决方案
|
||||||
|
|
||||||
|
1. **Copilot 自建一套 Tool completion**：会分叉 start barrier、加密 Artifact、rekey 与恢复语义。
|
||||||
|
2. **admission 后直接调用日志 reader**：响应丢失时无法区分“未执行”和“已执行但未返回”。
|
||||||
|
3. **Tool 一启动就解锁 Model**：允许模型消费未完成、失败或未认证的结果。
|
||||||
|
4. **把失败 Tool completion 当成可诊断输入**：当前没有经过审定的失败 Prompt/terminalization
|
||||||
|
协议，会把基础设施错误伪装成业务诊断。
|
||||||
|
5. **共用 Prompt output/result/invocation keyring**：混淆密钥用途、active authority 与退役范围。
|
||||||
|
6. **为 keyring 或 coordinator 新建 package**：没有独立部署/ownership 边界，并会继续加剧包碎片化。
|
||||||
|
|
||||||
|
## 当前验证
|
||||||
|
|
||||||
|
1. admission/execution/unlock 与 migration/readiness 聚焦契约 21/21，projected invocation keyring
|
||||||
|
2/2；AI package 最终为 229 pass、3 条件 skip、0 fail，类型检查通过。
|
||||||
|
2. 18 个 QL3 package 的 clean build/test 全部退出 0；backend 为 1,207 pass、2 条件 skip、
|
||||||
|
0 fail。
|
||||||
|
3. package boundary、Cluster dependency、Edge import 和 Cluster deployment 四项审计均
|
||||||
|
compatible 且零 finding。workspace 仍为 18 个 package，`singleSourcePackages=[]`、
|
||||||
|
`shallowSourcePackages=[]`；AI 175 个源码中 174 个、Cluster Control 56 个中 54 个位于
|
||||||
|
嵌套领域目录。
|
||||||
|
4. 14 档 Local Profile artifact 全部通过。默认 Edge/Standalone 为 2,589,890 / 2,589,968
|
||||||
|
bytes；显式 Edge/Standalone AI 为 3,135,809 / 3,135,899 bytes；最大 Standalone MCP 为
|
||||||
|
7,316,038 bytes，均在各自门内。
|
||||||
|
5. PostgreSQL 18.4 arm64 physical HA 为 130/130 Gate、timeline `1→2`。门证明首次 Tool 执行只
|
||||||
|
读取两次日志、密文 JSON 不含敏感 fixture 文本、Model Step 原子进入 ready；主库提升后 exact
|
||||||
|
replay 的日志读取为零。报告 SHA-256 为
|
||||||
|
`d525a303696e178d777b021b376729bd2c5382fb5eb7bc98466a2b79d3940517`，独立审计零 finding，
|
||||||
|
临时 Docker 资源已清理。
|
||||||
|
|
||||||
|
## 后续门禁
|
||||||
|
|
||||||
|
1. 从 durable Tool completion 解密受信投影，经 ADR-0405 builder 生成 prompt，并通过 Model
|
||||||
|
Gateway 执行 provider credential、egress policy、quota、price 和 deadline fence。
|
||||||
|
2. 建立 Copilot 专用 encrypted model completion 和 response-loss recovery；模型输出不得进入
|
||||||
|
普通 completion、数据库 JSON、审计或日志明文。
|
||||||
|
3. 原子终态化 Model Step 与 diagnosis Run，并覆盖 Tool/Model 失败、取消、超时、未知结果和
|
||||||
|
主库切换后的恢复状态机。
|
||||||
|
4. 最后再开放默认关闭的 Cluster API/CLI/UI/MCP 产品入口，并补真实 S3、外部 Provider fault
|
||||||
|
injection、多副本并发和 plaintext negative evidence。
|
||||||
@@ -411,6 +411,7 @@
|
|||||||
| [ADR-0405](./ADR-0405-bounded-failure-diagnosis-prompt-and-model-egress-policy.md) | 有界故障诊断 Prompt 与显式模型出口策略 | Accepted |
|
| [ADR-0405](./ADR-0405-bounded-failure-diagnosis-prompt-and-model-egress-policy.md) | 有界故障诊断 Prompt 与显式模型出口策略 | Accepted |
|
||||||
| [ADR-0406](./ADR-0406-cluster-projected-tool-result-key-authority.md) | Cluster Projected Tool Result Key Authority | Accepted |
|
| [ADR-0406](./ADR-0406-cluster-projected-tool-result-key-authority.md) | Cluster Projected Tool Result Key Authority | Accepted |
|
||||||
| [ADR-0407](./ADR-0407-cluster-copilot-failure-diagnosis-run-admission.md) | Cluster Copilot Failure Diagnosis Run Admission | Accepted |
|
| [ADR-0407](./ADR-0407-cluster-copilot-failure-diagnosis-run-admission.md) | Cluster Copilot Failure Diagnosis Run Admission | Accepted |
|
||||||
|
| [ADR-0408](./ADR-0408-cluster-copilot-failure-diagnosis-tool-execution.md) | Cluster Copilot Failure Diagnosis Tool Execution 与原子 Model 解锁 | Accepted |
|
||||||
|
|
||||||
## 规则
|
## 规则
|
||||||
|
|
||||||
|
|||||||
@@ -40,6 +40,16 @@
|
|||||||
"require": "./dist/copilot/failure-diagnosis/postgresAdmissionRepository.js",
|
"require": "./dist/copilot/failure-diagnosis/postgresAdmissionRepository.js",
|
||||||
"default": "./dist/copilot/failure-diagnosis/postgresAdmissionRepository.js"
|
"default": "./dist/copilot/failure-diagnosis/postgresAdmissionRepository.js"
|
||||||
},
|
},
|
||||||
|
"./failure-diagnosis-tool-execution": {
|
||||||
|
"types": "./dist/copilot/failure-diagnosis/toolExecution.d.ts",
|
||||||
|
"require": "./dist/copilot/failure-diagnosis/toolExecution.js",
|
||||||
|
"default": "./dist/copilot/failure-diagnosis/toolExecution.js"
|
||||||
|
},
|
||||||
|
"./postgres-failure-diagnosis-tool-execution-storage": {
|
||||||
|
"types": "./dist/copilot/failure-diagnosis/postgresToolExecutionRepository.d.ts",
|
||||||
|
"require": "./dist/copilot/failure-diagnosis/postgresToolExecutionRepository.js",
|
||||||
|
"default": "./dist/copilot/failure-diagnosis/postgresToolExecutionRepository.js"
|
||||||
|
},
|
||||||
"./model-invocation": {
|
"./model-invocation": {
|
||||||
"types": "./dist/model-invocation/modelInvocation.d.ts",
|
"types": "./dist/model-invocation/modelInvocation.d.ts",
|
||||||
"require": "./dist/model-invocation/modelInvocation.js",
|
"require": "./dist/model-invocation/modelInvocation.js",
|
||||||
|
|||||||
@@ -175,7 +175,7 @@ export function createCopilotFailureDiagnosisAdmissionBundle(
|
|||||||
runId: plan.runId,
|
runId: plan.runId,
|
||||||
stepKey: 'collect-log',
|
stepKey: 'collect-log',
|
||||||
kind: 'tool',
|
kind: 'tool',
|
||||||
definitionRef: `trusted-tool-plan:${plan.tool.planDigest}`,
|
definitionRef: 'tool:qinglong.run.log.excerpt@1.0.0',
|
||||||
definitionDigest: plan.tool.definitionDigest,
|
definitionDigest: plan.tool.definitionDigest,
|
||||||
required: true,
|
required: true,
|
||||||
initialStatus: 'ready',
|
initialStatus: 'ready',
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
export * from './tool-execution/postgresUnlockRepository';
|
||||||
@@ -0,0 +1,131 @@
|
|||||||
|
import type { RunRecord } from '@qinglong/runtime-core';
|
||||||
|
import type { SecurityPrincipal } from '@qinglong/runtime-core/security';
|
||||||
|
import type { StepRunMutation } from '@qinglong/runtime-core/step-run';
|
||||||
|
import type {
|
||||||
|
ToolExecutionCompletionRecord,
|
||||||
|
ToolExecutionResultArtifactReference,
|
||||||
|
} from '@qinglong/runtime-core/tool-execution-completion';
|
||||||
|
import type { ToolPolicyAuthorizer } from '@qinglong/runtime-core/tool-registry';
|
||||||
|
|
||||||
|
import type {
|
||||||
|
CopilotFailureDiagnosisAdmissionRepository,
|
||||||
|
CopilotFailureDiagnosisExecutionPlan,
|
||||||
|
} from '../admission/contracts';
|
||||||
|
|
||||||
|
export const COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA =
|
||||||
|
'qinglong/copilot-failure-diagnosis-tool-unlock-receipt@v1' as const;
|
||||||
|
export const COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA =
|
||||||
|
'qinglong/copilot-failure-diagnosis-tool-unlock-command@v1' as const;
|
||||||
|
export const MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_BYTES =
|
||||||
|
16 * 1024;
|
||||||
|
export const MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_BYTES =
|
||||||
|
48 * 1024;
|
||||||
|
|
||||||
|
export interface CopilotFailureDiagnosisToolUnlockReceipt {
|
||||||
|
readonly schema: typeof COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA;
|
||||||
|
readonly requestId: string;
|
||||||
|
readonly planDigest: string;
|
||||||
|
readonly runId: string;
|
||||||
|
readonly startId: string;
|
||||||
|
readonly barrierDigest: string;
|
||||||
|
readonly toolStepRunId: string;
|
||||||
|
readonly toolCompletionDigest: string;
|
||||||
|
readonly resultArtifact: Readonly<ToolExecutionResultArtifactReference>;
|
||||||
|
readonly modelStepRunId: string;
|
||||||
|
readonly modelStepRunVersion: number;
|
||||||
|
readonly modelStepRunDigest: string;
|
||||||
|
readonly modelMutationId: string;
|
||||||
|
readonly modelMutationDigest: string;
|
||||||
|
readonly modelEventId: string;
|
||||||
|
readonly finalRunVersion: number;
|
||||||
|
readonly finalRunEventSequence: number;
|
||||||
|
readonly unlockedAtMs: number;
|
||||||
|
readonly receiptDigest: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CopilotFailureDiagnosisToolUnlockCommand {
|
||||||
|
readonly schema: typeof COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA;
|
||||||
|
readonly plan: Readonly<CopilotFailureDiagnosisExecutionPlan>;
|
||||||
|
readonly completion: Readonly<ToolExecutionCompletionRecord>;
|
||||||
|
readonly modelStepRunMutation: Readonly<StepRunMutation>;
|
||||||
|
readonly receipt: Readonly<CopilotFailureDiagnosisToolUnlockReceipt>;
|
||||||
|
readonly commandDigest: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CopilotFailureDiagnosisToolUnlockRepository {
|
||||||
|
findByRequestId(
|
||||||
|
requestId: string,
|
||||||
|
): Promise<Readonly<CopilotFailureDiagnosisToolUnlockReceipt> | null>;
|
||||||
|
commit(command: CopilotFailureDiagnosisToolUnlockCommand): Promise<
|
||||||
|
Readonly<{
|
||||||
|
status: 'created' | 'existing';
|
||||||
|
receipt: Readonly<CopilotFailureDiagnosisToolUnlockReceipt>;
|
||||||
|
}>
|
||||||
|
>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ExecuteCopilotFailureDiagnosisToolInput {
|
||||||
|
readonly requestId: string;
|
||||||
|
readonly principal: Readonly<SecurityPrincipal>;
|
||||||
|
readonly authorizer: ToolPolicyAuthorizer;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type CopilotFailureDiagnosisToolExecutionResult =
|
||||||
|
| Readonly<{
|
||||||
|
outcome: 'succeeded';
|
||||||
|
completionStatus: 'created' | 'existing';
|
||||||
|
unlockStatus: 'created' | 'existing';
|
||||||
|
completion: Readonly<ToolExecutionCompletionRecord>;
|
||||||
|
unlock: Readonly<CopilotFailureDiagnosisToolUnlockReceipt>;
|
||||||
|
}>
|
||||||
|
| Readonly<{
|
||||||
|
outcome: 'failed' | 'timed_out';
|
||||||
|
completionStatus: 'created' | 'existing';
|
||||||
|
unlockStatus: null;
|
||||||
|
}>;
|
||||||
|
|
||||||
|
export class InvalidCopilotFailureDiagnosisToolExecutionError extends TypeError {
|
||||||
|
readonly code = 'COPILOT_FAILURE_DIAGNOSIS_TOOL_EXECUTION_INVALID';
|
||||||
|
|
||||||
|
constructor(message: string) {
|
||||||
|
super(`Copilot failure diagnosis Tool execution is invalid: ${message}`);
|
||||||
|
this.name = 'InvalidCopilotFailureDiagnosisToolExecutionError';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export class CopilotFailureDiagnosisToolExecutionConflictError extends Error {
|
||||||
|
readonly code = 'COPILOT_FAILURE_DIAGNOSIS_TOOL_EXECUTION_CONFLICT';
|
||||||
|
|
||||||
|
constructor(message = 'durable Tool execution facts changed') {
|
||||||
|
super(`Copilot failure diagnosis Tool execution conflicts: ${message}`);
|
||||||
|
this.name = 'CopilotFailureDiagnosisToolExecutionConflictError';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export class CopilotFailureDiagnosisToolExecutionUnavailableError extends Error {
|
||||||
|
readonly code = 'COPILOT_FAILURE_DIAGNOSIS_TOOL_EXECUTION_UNAVAILABLE';
|
||||||
|
|
||||||
|
constructor(options?: ErrorOptions) {
|
||||||
|
super('Copilot failure diagnosis Tool execution is unavailable', options);
|
||||||
|
this.name = 'CopilotFailureDiagnosisToolExecutionUnavailableError';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export class CopilotFailureDiagnosisToolExecutionDeadlineExceededError extends Error {
|
||||||
|
readonly code = 'COPILOT_FAILURE_DIAGNOSIS_TOOL_EXECUTION_DEADLINE_EXCEEDED';
|
||||||
|
|
||||||
|
constructor() {
|
||||||
|
super('Copilot failure diagnosis Tool execution deadline was exceeded');
|
||||||
|
this.name = 'CopilotFailureDiagnosisToolExecutionDeadlineExceededError';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export type CopilotFailureDiagnosisToolExecutionAdmissionReader = Pick<
|
||||||
|
CopilotFailureDiagnosisAdmissionRepository,
|
||||||
|
'findByRequestId' | 'findPlanByRequestId'
|
||||||
|
>;
|
||||||
|
|
||||||
|
export type CopilotFailureDiagnosisRunAuthority = Pick<
|
||||||
|
RunRecord,
|
||||||
|
'id' | 'projectId' | 'status' | 'version' | 'eventSequence'
|
||||||
|
>;
|
||||||
@@ -0,0 +1,599 @@
|
|||||||
|
import { Buffer } from 'node:buffer';
|
||||||
|
import { createHash } from 'node:crypto';
|
||||||
|
|
||||||
|
import type { RunRepositoryReader } from '@qinglong/runtime-core/run-repository';
|
||||||
|
import {
|
||||||
|
transitionStepRunMutation,
|
||||||
|
type StepRunRepository,
|
||||||
|
} from '@qinglong/runtime-core/step-run';
|
||||||
|
import {
|
||||||
|
createToolExecutionEvidenceBundle,
|
||||||
|
toolExecutionAdmissionEvidence,
|
||||||
|
TOOL_EXECUTION_START_AUDIT_OPERATION,
|
||||||
|
} from '@qinglong/runtime-core/tool-execution-evidence';
|
||||||
|
import {
|
||||||
|
createToolExecutionStartCommand,
|
||||||
|
normalizeToolExecutionStartBarrierRecord,
|
||||||
|
type ToolExecutionStartBarrierRecord,
|
||||||
|
type ToolExecutionStartBarrierRepository,
|
||||||
|
} from '@qinglong/runtime-core/tool-execution-start-barrier';
|
||||||
|
import type { ToolExecutionCompletionRepository } from '@qinglong/runtime-core/tool-execution-completion';
|
||||||
|
import type { ToolExecutionFailureCompletionRepository } from '@qinglong/runtime-core/tool-execution-failure-completion';
|
||||||
|
import type { ToolExecutionResultRekeyReader } from '@qinglong/runtime-core/tool-result-rekey';
|
||||||
|
import type { ToolResultKeyCatalogReader } from '@qinglong/runtime-core/tool-result-key-catalog';
|
||||||
|
import type { ProjectToolDefinitionSnapshotRepository } from '@qinglong/runtime-core/project-tool-definition-snapshot';
|
||||||
|
import {
|
||||||
|
toolInvocationInputArtifactReference,
|
||||||
|
toolInvocationPreviewArtifactReference,
|
||||||
|
type ToolInvocationArtifactKeyProvider,
|
||||||
|
type ToolInvocationArtifactRepository,
|
||||||
|
} from '@qinglong/runtime-core/tool-invocation-artifact';
|
||||||
|
import {
|
||||||
|
admitTrustedToolExecution,
|
||||||
|
trustedToolContractIdentityDigest,
|
||||||
|
} from '@qinglong/runtime-core/trusted-tool-invocation';
|
||||||
|
import { TrustedToolExecutionAdapterRegistry } from '@qinglong/runtime-core/trusted-tool-execution';
|
||||||
|
import { executeAndCompleteTrustedTool } from '@qinglong/runtime-core/trusted-tool-completion';
|
||||||
|
import { BuiltInRunLogExcerptToolAdapter } from '@qinglong/runtime-core/builtin-run-log-excerpt-tool';
|
||||||
|
import type { RunAttemptLogReadPort } from '@qinglong/runtime-core/builtin-run-log-excerpt-projection';
|
||||||
|
|
||||||
|
import {
|
||||||
|
CopilotFailureDiagnosisToolExecutionConflictError,
|
||||||
|
CopilotFailureDiagnosisToolExecutionDeadlineExceededError,
|
||||||
|
CopilotFailureDiagnosisToolExecutionUnavailableError,
|
||||||
|
InvalidCopilotFailureDiagnosisToolExecutionError,
|
||||||
|
type CopilotFailureDiagnosisToolExecutionAdmissionReader,
|
||||||
|
type CopilotFailureDiagnosisToolExecutionResult,
|
||||||
|
type CopilotFailureDiagnosisToolUnlockRepository,
|
||||||
|
type ExecuteCopilotFailureDiagnosisToolInput,
|
||||||
|
} from './contracts';
|
||||||
|
import { restoreCopilotFailureDiagnosisTrustedToolAuthority } from './planAuthority';
|
||||||
|
import { createCopilotFailureDiagnosisToolUnlockCommand } from './unlockProtocol';
|
||||||
|
|
||||||
|
export interface CopilotFailureDiagnosisToolExecutionDependencies {
|
||||||
|
readonly admissions: CopilotFailureDiagnosisToolExecutionAdmissionReader;
|
||||||
|
readonly snapshots: Pick<
|
||||||
|
ProjectToolDefinitionSnapshotRepository,
|
||||||
|
'findCurrent'
|
||||||
|
>;
|
||||||
|
readonly artifacts: ToolInvocationArtifactRepository;
|
||||||
|
readonly invocationKeys: Pick<ToolInvocationArtifactKeyProvider, 'resolve'>;
|
||||||
|
readonly resultKeys: Pick<ToolInvocationArtifactKeyProvider, 'resolve'>;
|
||||||
|
readonly stepRuns: Pick<StepRunRepository, 'findById'>;
|
||||||
|
readonly runs: Pick<RunRepositoryReader, 'findRunById'>;
|
||||||
|
readonly barriers: ToolExecutionStartBarrierRepository;
|
||||||
|
readonly completions: ToolExecutionCompletionRepository;
|
||||||
|
readonly failureCompletions: ToolExecutionFailureCompletionRepository;
|
||||||
|
readonly resultKeyCatalog: ToolResultKeyCatalogReader;
|
||||||
|
readonly resultRekeys: ToolExecutionResultRekeyReader;
|
||||||
|
readonly logs: RunAttemptLogReadPort;
|
||||||
|
readonly unlocks: CopilotFailureDiagnosisToolUnlockRepository;
|
||||||
|
readonly now?: () => number;
|
||||||
|
readonly nonceFactory?: () => Uint8Array;
|
||||||
|
}
|
||||||
|
|
||||||
|
const IDENTITY_DOMAIN = Buffer.from(
|
||||||
|
'qinglong/copilot-failure-diagnosis-tool-execution-identity@v1\0',
|
||||||
|
'utf8',
|
||||||
|
);
|
||||||
|
|
||||||
|
function invalid(message: string): never {
|
||||||
|
throw new InvalidCopilotFailureDiagnosisToolExecutionError(message);
|
||||||
|
}
|
||||||
|
|
||||||
|
function unavailable(cause?: unknown): never {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionUnavailableError({
|
||||||
|
cause: cause instanceof Error ? cause : undefined,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function hash(value: unknown): string {
|
||||||
|
return createHash('sha256')
|
||||||
|
.update(IDENTITY_DOMAIN)
|
||||||
|
.update(JSON.stringify(value))
|
||||||
|
.digest('hex');
|
||||||
|
}
|
||||||
|
|
||||||
|
function identity(prefix: string, planDigest: string): string {
|
||||||
|
const maximumDigestLength = 35 - prefix.length;
|
||||||
|
return `${prefix}:${hash({ prefix, planDigest }).slice(
|
||||||
|
0,
|
||||||
|
maximumDigestLength,
|
||||||
|
)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function traceIdentity(planDigest: string): string {
|
||||||
|
return hash({ prefix: 'trace', planDigest }).slice(0, 32);
|
||||||
|
}
|
||||||
|
|
||||||
|
function spanIdentity(planDigest: string): string {
|
||||||
|
return hash({ prefix: 'span', planDigest }).slice(0, 16);
|
||||||
|
}
|
||||||
|
|
||||||
|
function auditEventIdentity(planDigest: string): string {
|
||||||
|
const value = hash({ prefix: 'audit', planDigest }).slice(0, 32).split('');
|
||||||
|
value[12] = '4';
|
||||||
|
value[16] = '8';
|
||||||
|
const hex = value.join('');
|
||||||
|
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(
|
||||||
|
12,
|
||||||
|
16,
|
||||||
|
)}-${hex.slice(16, 20)}-${hex.slice(20)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function clock(now: (() => number) | undefined): number {
|
||||||
|
let value: number;
|
||||||
|
try {
|
||||||
|
value = (now ?? Date.now)();
|
||||||
|
} catch (cause) {
|
||||||
|
return unavailable(cause);
|
||||||
|
}
|
||||||
|
if (!Number.isSafeInteger(value) || value < 0) return unavailable();
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
function sameValue(left: unknown, right: unknown): boolean {
|
||||||
|
return JSON.stringify(left) === JSON.stringify(right);
|
||||||
|
}
|
||||||
|
|
||||||
|
function assertDependencies(
|
||||||
|
dependencies: CopilotFailureDiagnosisToolExecutionDependencies,
|
||||||
|
): void {
|
||||||
|
if (
|
||||||
|
!dependencies ||
|
||||||
|
typeof dependencies !== 'object' ||
|
||||||
|
typeof dependencies.admissions?.findByRequestId !== 'function' ||
|
||||||
|
typeof dependencies.admissions?.findPlanByRequestId !== 'function' ||
|
||||||
|
typeof dependencies.snapshots?.findCurrent !== 'function' ||
|
||||||
|
typeof dependencies.artifacts?.findInput !== 'function' ||
|
||||||
|
typeof dependencies.artifacts?.findPreview !== 'function' ||
|
||||||
|
typeof dependencies.invocationKeys?.resolve !== 'function' ||
|
||||||
|
typeof dependencies.resultKeys?.resolve !== 'function' ||
|
||||||
|
typeof dependencies.stepRuns?.findById !== 'function' ||
|
||||||
|
typeof dependencies.runs?.findRunById !== 'function' ||
|
||||||
|
typeof dependencies.barriers?.findByStartId !== 'function' ||
|
||||||
|
typeof dependencies.barriers?.prepare !== 'function' ||
|
||||||
|
typeof dependencies.completions?.findByStartId !== 'function' ||
|
||||||
|
typeof dependencies.failureCompletions?.findByStartId !== 'function' ||
|
||||||
|
typeof dependencies.resultKeyCatalog?.findCurrent !== 'function' ||
|
||||||
|
typeof dependencies.resultRekeys?.findHeadByArtifactId !== 'function' ||
|
||||||
|
typeof dependencies.logs?.read !== 'function' ||
|
||||||
|
typeof dependencies.unlocks?.findByRequestId !== 'function' ||
|
||||||
|
typeof dependencies.unlocks?.commit !== 'function' ||
|
||||||
|
(dependencies.now !== undefined &&
|
||||||
|
typeof dependencies.now !== 'function') ||
|
||||||
|
(dependencies.nonceFactory !== undefined &&
|
||||||
|
typeof dependencies.nonceFactory !== 'function')
|
||||||
|
) {
|
||||||
|
return invalid('dependencies are invalid');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function durablePlan(
|
||||||
|
requestId: string,
|
||||||
|
dependencies: CopilotFailureDiagnosisToolExecutionDependencies,
|
||||||
|
) {
|
||||||
|
let plan;
|
||||||
|
let receipt;
|
||||||
|
try {
|
||||||
|
[plan, receipt] = await Promise.all([
|
||||||
|
dependencies.admissions.findPlanByRequestId(requestId),
|
||||||
|
dependencies.admissions.findByRequestId(requestId),
|
||||||
|
]);
|
||||||
|
} catch (cause) {
|
||||||
|
return unavailable(cause);
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
!plan ||
|
||||||
|
!receipt ||
|
||||||
|
plan.requestId !== requestId ||
|
||||||
|
receipt.requestId !== requestId ||
|
||||||
|
receipt.planDigest !== plan.planDigest ||
|
||||||
|
receipt.runId !== plan.runId ||
|
||||||
|
receipt.toolStepRunId !== plan.toolStepRunId ||
|
||||||
|
receipt.modelStepRunId !== plan.modelStepRunId
|
||||||
|
) {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||||
|
'diagnosis admission evidence is incomplete',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return plan;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function currentAuthority(
|
||||||
|
plan: Awaited<ReturnType<typeof durablePlan>>,
|
||||||
|
dependencies: CopilotFailureDiagnosisToolExecutionDependencies,
|
||||||
|
) {
|
||||||
|
let record;
|
||||||
|
try {
|
||||||
|
record = await dependencies.snapshots.findCurrent(plan.projectId);
|
||||||
|
} catch (cause) {
|
||||||
|
return unavailable(cause);
|
||||||
|
}
|
||||||
|
if (!record) {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||||
|
'the current Project Tool snapshot is absent',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return restoreCopilotFailureDiagnosisTrustedToolAuthority(
|
||||||
|
plan,
|
||||||
|
record.snapshot,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function assertArtifacts(
|
||||||
|
plan: Awaited<ReturnType<typeof durablePlan>>,
|
||||||
|
dependencies: CopilotFailureDiagnosisToolExecutionDependencies,
|
||||||
|
): Promise<void> {
|
||||||
|
let input;
|
||||||
|
let preview;
|
||||||
|
try {
|
||||||
|
[input, preview] = await Promise.all([
|
||||||
|
dependencies.artifacts.findInput(plan.tool.invocationArtifact.artifactId),
|
||||||
|
dependencies.artifacts.findPreview(plan.tool.previewArtifact.artifactId),
|
||||||
|
]);
|
||||||
|
} catch (cause) {
|
||||||
|
return unavailable(cause);
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
!input ||
|
||||||
|
!preview ||
|
||||||
|
!sameValue(
|
||||||
|
toolInvocationInputArtifactReference(input),
|
||||||
|
plan.tool.invocationArtifact,
|
||||||
|
) ||
|
||||||
|
!sameValue(
|
||||||
|
toolInvocationPreviewArtifactReference(preview),
|
||||||
|
plan.tool.previewArtifact,
|
||||||
|
) ||
|
||||||
|
input.projectId !== plan.projectId ||
|
||||||
|
input.actionRef !== plan.tool.actionRef ||
|
||||||
|
preview.projectId !== plan.projectId ||
|
||||||
|
preview.actionRef !== plan.tool.actionRef ||
|
||||||
|
input.sealedAtMs !== plan.tool.sealedAtMs ||
|
||||||
|
preview.sealedAtMs !== plan.tool.sealedAtMs
|
||||||
|
) {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||||
|
'the durable Tool invocation Artifacts changed',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function barrierMatches(
|
||||||
|
barrierValue: ToolExecutionStartBarrierRecord,
|
||||||
|
plan: Awaited<ReturnType<typeof durablePlan>>,
|
||||||
|
startId: string,
|
||||||
|
): Readonly<ToolExecutionStartBarrierRecord> {
|
||||||
|
const barrier = normalizeToolExecutionStartBarrierRecord(barrierValue);
|
||||||
|
if (
|
||||||
|
barrier.startId !== startId ||
|
||||||
|
barrier.projectId !== plan.projectId ||
|
||||||
|
barrier.runId !== plan.runId ||
|
||||||
|
barrier.stepRunId !== plan.toolStepRunId ||
|
||||||
|
barrier.actionRef !== plan.tool.actionRef ||
|
||||||
|
barrier.planDigest !== plan.tool.planDigest ||
|
||||||
|
barrier.actionDigest !== plan.tool.actionDigest ||
|
||||||
|
barrier.snapshotDigest !== plan.tool.snapshotDigest ||
|
||||||
|
barrier.definitionDigest !== plan.tool.definitionDigest ||
|
||||||
|
barrier.bindingDigest !== plan.tool.bindingDigest ||
|
||||||
|
!sameValue(barrier.invocationArtifact, plan.tool.invocationArtifact) ||
|
||||||
|
!sameValue(barrier.previewArtifact, plan.tool.previewArtifact) ||
|
||||||
|
!sameValue(barrier.requestedBy, plan.requestedBySubject) ||
|
||||||
|
barrier.profile !== 'cluster-control' ||
|
||||||
|
!sameValue(barrier.policyFence, plan.policyFence) ||
|
||||||
|
barrier.approvalRequestId !== null ||
|
||||||
|
barrier.approvalDispatchId !== null ||
|
||||||
|
barrier.approvalDispatchDigest !== null ||
|
||||||
|
barrier.previousStepRunVersion !== 1 ||
|
||||||
|
barrier.startedStepRunVersion !== 2
|
||||||
|
) {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||||
|
'the durable Tool start barrier changed',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return barrier;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function prepareStart(
|
||||||
|
plan: Awaited<ReturnType<typeof durablePlan>>,
|
||||||
|
input: ExecuteCopilotFailureDiagnosisToolInput,
|
||||||
|
authority: Awaited<ReturnType<typeof currentAuthority>>,
|
||||||
|
dependencies: CopilotFailureDiagnosisToolExecutionDependencies,
|
||||||
|
startId: string,
|
||||||
|
): Promise<Readonly<ToolExecutionStartBarrierRecord>> {
|
||||||
|
let existing;
|
||||||
|
try {
|
||||||
|
existing = await dependencies.barriers.findByStartId(startId);
|
||||||
|
} catch (cause) {
|
||||||
|
return unavailable(cause);
|
||||||
|
}
|
||||||
|
if (existing) return barrierMatches(existing, plan, startId);
|
||||||
|
|
||||||
|
await assertArtifacts(plan, dependencies);
|
||||||
|
let stepRun;
|
||||||
|
let run;
|
||||||
|
try {
|
||||||
|
[stepRun, run] = await Promise.all([
|
||||||
|
dependencies.stepRuns.findById(plan.toolStepRunId),
|
||||||
|
dependencies.runs.findRunById(plan.runId),
|
||||||
|
]);
|
||||||
|
} catch (cause) {
|
||||||
|
return unavailable(cause);
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
!stepRun ||
|
||||||
|
!run ||
|
||||||
|
stepRun.id !== plan.toolStepRunId ||
|
||||||
|
stepRun.runId !== plan.runId ||
|
||||||
|
stepRun.kind !== 'tool' ||
|
||||||
|
stepRun.status !== 'ready' ||
|
||||||
|
stepRun.version !== 1 ||
|
||||||
|
stepRun.definitionRef !== 'tool:qinglong.run.log.excerpt@1.0.0' ||
|
||||||
|
stepRun.definitionDigest !== plan.tool.definitionDigest ||
|
||||||
|
run.id !== plan.runId ||
|
||||||
|
run.projectId !== plan.projectId ||
|
||||||
|
run.status !== 'running' ||
|
||||||
|
run.version !== run.eventSequence
|
||||||
|
) {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||||
|
'the Tool StepRun is not startable',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const startedAtMs = clock(dependencies.now);
|
||||||
|
if (
|
||||||
|
startedAtMs < plan.plannedAtMs ||
|
||||||
|
startedAtMs + authority.plan.timeoutSeconds * 1_000 > plan.deadlineAtMs
|
||||||
|
) {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionDeadlineExceededError();
|
||||||
|
}
|
||||||
|
const evidence = createToolExecutionEvidenceBundle({
|
||||||
|
traceId: traceIdentity(plan.planDigest),
|
||||||
|
spanId: spanIdentity(plan.planDigest),
|
||||||
|
projectId: plan.projectId,
|
||||||
|
runId: plan.runId,
|
||||||
|
stepRunId: plan.toolStepRunId,
|
||||||
|
invocationPlanDigest: authority.plan.planDigest,
|
||||||
|
bindingDigest: authority.binding.bindingDigest,
|
||||||
|
adapterDigest: trustedToolContractIdentityDigest(authority.binding.adapter),
|
||||||
|
redactionContractDigest: trustedToolContractIdentityDigest(
|
||||||
|
authority.binding.redactionContract,
|
||||||
|
),
|
||||||
|
auditContractDigest: trustedToolContractIdentityDigest(
|
||||||
|
authority.binding.auditContract,
|
||||||
|
),
|
||||||
|
audit: {
|
||||||
|
eventId: auditEventIdentity(plan.planDigest),
|
||||||
|
requestId: plan.requestId,
|
||||||
|
operationId: TOOL_EXECUTION_START_AUDIT_OPERATION,
|
||||||
|
projectId: plan.projectId,
|
||||||
|
subject: plan.requestedBySubject,
|
||||||
|
authenticationId: input.principal.authenticationId,
|
||||||
|
outcome: 'allowed',
|
||||||
|
reasons: ['copilot_failure_diagnosis_tool_start'],
|
||||||
|
fence: plan.policyFence,
|
||||||
|
occurredAtMs: startedAtMs,
|
||||||
|
},
|
||||||
|
createdAtMs: startedAtMs,
|
||||||
|
});
|
||||||
|
const admission = await admitTrustedToolExecution(
|
||||||
|
authority.bindings,
|
||||||
|
authority.plan,
|
||||||
|
{
|
||||||
|
principal: input.principal,
|
||||||
|
profile: 'cluster-control',
|
||||||
|
nowMs: startedAtMs,
|
||||||
|
authorizer: input.authorizer,
|
||||||
|
evidence: {
|
||||||
|
stepRun: {
|
||||||
|
id: stepRun.id,
|
||||||
|
version: stepRun.version,
|
||||||
|
digest: stepRun.stepRunDigest,
|
||||||
|
},
|
||||||
|
...toolExecutionAdmissionEvidence(evidence),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
|
const mutationId = identity('cdstm', plan.planDigest);
|
||||||
|
const eventId = identity('cdste', plan.planDigest);
|
||||||
|
const mutation = transitionStepRunMutation(
|
||||||
|
stepRun,
|
||||||
|
{
|
||||||
|
expectedVersion: stepRun.version,
|
||||||
|
expectedDigest: stepRun.stepRunDigest,
|
||||||
|
mutationId,
|
||||||
|
to: 'running',
|
||||||
|
atMs: startedAtMs,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
expectedRunVersion: run.version,
|
||||||
|
expectedRunEventSequence: run.eventSequence,
|
||||||
|
eventId,
|
||||||
|
dedupeKey: eventId,
|
||||||
|
actor: plan.requestedBySubject,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
const command = createToolExecutionStartCommand({
|
||||||
|
startId,
|
||||||
|
admission,
|
||||||
|
evidence,
|
||||||
|
stepRunMutation: mutation,
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
const prepared = await dependencies.barriers.prepare(command);
|
||||||
|
if (!['created', 'existing'].includes(prepared.status)) {
|
||||||
|
return unavailable();
|
||||||
|
}
|
||||||
|
return barrierMatches(prepared.barrier, plan, startId);
|
||||||
|
} catch (cause) {
|
||||||
|
let recovered;
|
||||||
|
try {
|
||||||
|
recovered = await dependencies.barriers.findByStartId(startId);
|
||||||
|
} catch {
|
||||||
|
throw cause;
|
||||||
|
}
|
||||||
|
if (recovered) return barrierMatches(recovered, plan, startId);
|
||||||
|
throw cause;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function completionIdentities(planDigest: string) {
|
||||||
|
return Object.freeze({
|
||||||
|
success: Object.freeze({
|
||||||
|
create() {
|
||||||
|
return Object.freeze({
|
||||||
|
artifactId: identity('cdra', planDigest),
|
||||||
|
mutationId: identity('cdscm', planDigest),
|
||||||
|
eventId: identity('cdsce', planDigest),
|
||||||
|
});
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
failure: Object.freeze({
|
||||||
|
create() {
|
||||||
|
return Object.freeze({
|
||||||
|
mutationId: identity('cdfcm', planDigest),
|
||||||
|
eventId: identity('cdfce', planDigest),
|
||||||
|
});
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async function unlockModel(
|
||||||
|
plan: Awaited<ReturnType<typeof durablePlan>>,
|
||||||
|
completion: Parameters<
|
||||||
|
typeof createCopilotFailureDiagnosisToolUnlockCommand
|
||||||
|
>[0]['completion'],
|
||||||
|
dependencies: CopilotFailureDiagnosisToolExecutionDependencies,
|
||||||
|
) {
|
||||||
|
let existing;
|
||||||
|
try {
|
||||||
|
existing = await dependencies.unlocks.findByRequestId(plan.requestId);
|
||||||
|
} catch (cause) {
|
||||||
|
return unavailable(cause);
|
||||||
|
}
|
||||||
|
if (existing) {
|
||||||
|
if (
|
||||||
|
existing.planDigest !== plan.planDigest ||
|
||||||
|
existing.toolCompletionDigest !== completion.completionDigest
|
||||||
|
) {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||||
|
'the durable model unlock changed',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return Object.freeze({ status: 'existing' as const, receipt: existing });
|
||||||
|
}
|
||||||
|
let modelStepRun;
|
||||||
|
let run;
|
||||||
|
try {
|
||||||
|
[modelStepRun, run] = await Promise.all([
|
||||||
|
dependencies.stepRuns.findById(plan.modelStepRunId),
|
||||||
|
dependencies.runs.findRunById(plan.runId),
|
||||||
|
]);
|
||||||
|
} catch (cause) {
|
||||||
|
return unavailable(cause);
|
||||||
|
}
|
||||||
|
if (!modelStepRun || !run) return unavailable();
|
||||||
|
const command = createCopilotFailureDiagnosisToolUnlockCommand({
|
||||||
|
plan,
|
||||||
|
completion,
|
||||||
|
modelStepRun,
|
||||||
|
run: {
|
||||||
|
id: run.id,
|
||||||
|
projectId: run.projectId,
|
||||||
|
status: run.status,
|
||||||
|
version: run.version,
|
||||||
|
eventSequence: run.eventSequence,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
return await dependencies.unlocks.commit(command);
|
||||||
|
} catch (cause) {
|
||||||
|
let recovered;
|
||||||
|
try {
|
||||||
|
recovered = await dependencies.unlocks.findByRequestId(plan.requestId);
|
||||||
|
} catch {
|
||||||
|
throw cause;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
recovered &&
|
||||||
|
recovered.planDigest === plan.planDigest &&
|
||||||
|
recovered.toolCompletionDigest === completion.completionDigest
|
||||||
|
) {
|
||||||
|
return Object.freeze({
|
||||||
|
status: 'existing' as const,
|
||||||
|
receipt: recovered,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
throw cause;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function executeCopilotFailureDiagnosisTool(
|
||||||
|
input: ExecuteCopilotFailureDiagnosisToolInput,
|
||||||
|
dependencies: CopilotFailureDiagnosisToolExecutionDependencies,
|
||||||
|
): Promise<Readonly<CopilotFailureDiagnosisToolExecutionResult>> {
|
||||||
|
assertDependencies(dependencies);
|
||||||
|
if (
|
||||||
|
!input ||
|
||||||
|
typeof input !== 'object' ||
|
||||||
|
Array.isArray(input) ||
|
||||||
|
typeof input.requestId !== 'string' ||
|
||||||
|
typeof input.authorizer?.authorize !== 'function'
|
||||||
|
) {
|
||||||
|
return invalid('input is invalid');
|
||||||
|
}
|
||||||
|
const plan = await durablePlan(input.requestId, dependencies);
|
||||||
|
const authority = await currentAuthority(plan, dependencies);
|
||||||
|
const startId = identity('cds', plan.planDigest);
|
||||||
|
const barrier = await prepareStart(
|
||||||
|
plan,
|
||||||
|
input,
|
||||||
|
authority,
|
||||||
|
dependencies,
|
||||||
|
startId,
|
||||||
|
);
|
||||||
|
const definitions = authority.bindings.definitionRegistry();
|
||||||
|
const adapters = new TrustedToolExecutionAdapterRegistry(authority.bindings, [
|
||||||
|
new BuiltInRunLogExcerptToolAdapter(
|
||||||
|
authority.binding,
|
||||||
|
'cluster-control',
|
||||||
|
definitions,
|
||||||
|
dependencies.logs,
|
||||||
|
),
|
||||||
|
]);
|
||||||
|
const ids = completionIdentities(plan.planDigest);
|
||||||
|
const completed = await executeAndCompleteTrustedTool(barrier.startId, {
|
||||||
|
barriers: dependencies.barriers,
|
||||||
|
artifacts: dependencies.artifacts,
|
||||||
|
keys: dependencies.invocationKeys,
|
||||||
|
adapters,
|
||||||
|
completions: dependencies.completions,
|
||||||
|
failureCompletions: dependencies.failureCompletions,
|
||||||
|
stepRuns: dependencies.stepRuns,
|
||||||
|
runs: dependencies.runs,
|
||||||
|
resultKeyCatalog: dependencies.resultKeyCatalog,
|
||||||
|
resultRekeys: dependencies.resultRekeys,
|
||||||
|
resultKeys: dependencies.resultKeys,
|
||||||
|
identities: ids.success,
|
||||||
|
failureIdentities: ids.failure,
|
||||||
|
...(dependencies.now === undefined ? {} : { now: dependencies.now }),
|
||||||
|
...(dependencies.nonceFactory === undefined
|
||||||
|
? {}
|
||||||
|
: { nonceFactory: dependencies.nonceFactory }),
|
||||||
|
});
|
||||||
|
if (completed.outcome !== 'succeeded') {
|
||||||
|
return Object.freeze({
|
||||||
|
outcome: completed.outcome,
|
||||||
|
completionStatus: completed.status,
|
||||||
|
unlockStatus: null,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
const unlock = await unlockModel(plan, completed.completion, dependencies);
|
||||||
|
return Object.freeze({
|
||||||
|
outcome: 'succeeded' as const,
|
||||||
|
completionStatus: completed.status,
|
||||||
|
unlockStatus: unlock.status,
|
||||||
|
completion: completed.completion,
|
||||||
|
unlock: unlock.receipt,
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
import {
|
||||||
|
BUILTIN_RUN_LOG_EXCERPT_TIMEOUT_SECONDS,
|
||||||
|
BUILTIN_RUN_LOG_EXCERPT_TOOL,
|
||||||
|
createBuiltInRunLogExcerptToolHandlerBinding,
|
||||||
|
} from '@qinglong/runtime-core/builtin-run-log-excerpt-tool';
|
||||||
|
import {
|
||||||
|
normalizeProjectToolDefinitionSnapshot,
|
||||||
|
type ProjectToolDefinitionSnapshot,
|
||||||
|
} from '@qinglong/runtime-core/project-tool-definition-snapshot';
|
||||||
|
import {
|
||||||
|
TrustedToolHandlerBindingRegistry,
|
||||||
|
normalizeTrustedToolInvocationPlan,
|
||||||
|
type TrustedToolHandlerBinding,
|
||||||
|
type TrustedToolInvocationPlan,
|
||||||
|
} from '@qinglong/runtime-core/trusted-tool-invocation';
|
||||||
|
|
||||||
|
import type { CopilotFailureDiagnosisExecutionPlan } from '../admission/contracts';
|
||||||
|
import { normalizeCopilotFailureDiagnosisExecutionPlan } from '../admission/plan';
|
||||||
|
import {
|
||||||
|
CopilotFailureDiagnosisToolExecutionConflictError,
|
||||||
|
InvalidCopilotFailureDiagnosisToolExecutionError,
|
||||||
|
} from './contracts';
|
||||||
|
|
||||||
|
export interface CopilotFailureDiagnosisTrustedToolAuthority {
|
||||||
|
readonly plan: Readonly<TrustedToolInvocationPlan>;
|
||||||
|
readonly binding: Readonly<TrustedToolHandlerBinding>;
|
||||||
|
readonly bindings: TrustedToolHandlerBindingRegistry;
|
||||||
|
}
|
||||||
|
|
||||||
|
function invalid(message: string): never {
|
||||||
|
throw new InvalidCopilotFailureDiagnosisToolExecutionError(message);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function restoreCopilotFailureDiagnosisTrustedToolAuthority(
|
||||||
|
executionPlanValue: CopilotFailureDiagnosisExecutionPlan,
|
||||||
|
snapshotValue: ProjectToolDefinitionSnapshot,
|
||||||
|
): Readonly<CopilotFailureDiagnosisTrustedToolAuthority> {
|
||||||
|
const executionPlan =
|
||||||
|
normalizeCopilotFailureDiagnosisExecutionPlan(executionPlanValue);
|
||||||
|
const snapshot = normalizeProjectToolDefinitionSnapshot(snapshotValue);
|
||||||
|
if (
|
||||||
|
snapshot.projectId !== executionPlan.projectId ||
|
||||||
|
snapshot.snapshotDigest !== executionPlan.tool.snapshotDigest
|
||||||
|
) {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||||
|
'the current Project Tool snapshot changed',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const binding = createBuiltInRunLogExcerptToolHandlerBinding(snapshot, [
|
||||||
|
'cluster-control',
|
||||||
|
]);
|
||||||
|
if (
|
||||||
|
binding.bindingDigest !== executionPlan.tool.bindingDigest ||
|
||||||
|
binding.definitionDigest !== executionPlan.tool.definitionDigest
|
||||||
|
) {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||||
|
'the reviewed log Tool binding changed',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const bindings = new TrustedToolHandlerBindingRegistry(snapshot, [binding]);
|
||||||
|
let plan: Readonly<TrustedToolInvocationPlan>;
|
||||||
|
try {
|
||||||
|
plan = normalizeTrustedToolInvocationPlan(
|
||||||
|
{
|
||||||
|
schema: 'qinglong/trusted-tool-invocation-plan@v1',
|
||||||
|
status: 'ready',
|
||||||
|
actionType: 'tool.invoke',
|
||||||
|
actionRef: executionPlan.tool.actionRef,
|
||||||
|
projectId: executionPlan.projectId,
|
||||||
|
requestedBy: executionPlan.requestedBySubject,
|
||||||
|
tool: BUILTIN_RUN_LOG_EXCERPT_TOOL,
|
||||||
|
permission: 'tool.call:qinglong.run.log.excerpt',
|
||||||
|
requiredPermissions: ['artifact.read'],
|
||||||
|
effect: 'read',
|
||||||
|
risk: 'medium',
|
||||||
|
policyFence: executionPlan.policyFence,
|
||||||
|
profile: 'cluster-control',
|
||||||
|
snapshotDigest: executionPlan.tool.snapshotDigest,
|
||||||
|
definitionDigest: executionPlan.tool.definitionDigest,
|
||||||
|
binding,
|
||||||
|
timeoutSeconds: BUILTIN_RUN_LOG_EXCERPT_TIMEOUT_SECONDS,
|
||||||
|
invocationArtifact: executionPlan.tool.invocationArtifact,
|
||||||
|
invocationActionDigest: executionPlan.tool.invocationActionDigest,
|
||||||
|
previewArtifact: executionPlan.tool.previewArtifact,
|
||||||
|
actionDigest: executionPlan.tool.actionDigest,
|
||||||
|
sealedAtMs: executionPlan.tool.sealedAtMs,
|
||||||
|
planDigest: executionPlan.tool.planDigest,
|
||||||
|
},
|
||||||
|
bindings,
|
||||||
|
);
|
||||||
|
} catch (cause) {
|
||||||
|
if (cause instanceof CopilotFailureDiagnosisToolExecutionConflictError) {
|
||||||
|
throw cause;
|
||||||
|
}
|
||||||
|
return invalid('the admitted trusted Tool plan cannot be restored');
|
||||||
|
}
|
||||||
|
if (plan.planDigest !== executionPlan.tool.planDigest) {
|
||||||
|
return invalid('the restored trusted Tool plan digest changed');
|
||||||
|
}
|
||||||
|
return Object.freeze({ plan, binding, bindings });
|
||||||
|
}
|
||||||
+548
@@ -0,0 +1,548 @@
|
|||||||
|
import { isDeepStrictEqual } from 'node:util';
|
||||||
|
|
||||||
|
import type {
|
||||||
|
PostgresClient,
|
||||||
|
PostgresPool,
|
||||||
|
PostgresQueryable,
|
||||||
|
} from '@qinglong/runtime-core';
|
||||||
|
import {
|
||||||
|
normalizeToolExecutionCompletionRecord,
|
||||||
|
type ToolExecutionCompletionRecord,
|
||||||
|
} from '@qinglong/runtime-core/tool-execution-completion';
|
||||||
|
|
||||||
|
import { POSTGRES_MODEL_INVOCATION_SCHEMA } from '../../../migration/modelInvocationMigration';
|
||||||
|
import type { CopilotFailureDiagnosisExecutionPlan } from '../admission/contracts';
|
||||||
|
import { normalizeCopilotFailureDiagnosisExecutionPlan } from '../admission/plan';
|
||||||
|
import {
|
||||||
|
CopilotFailureDiagnosisToolExecutionConflictError,
|
||||||
|
CopilotFailureDiagnosisToolExecutionUnavailableError,
|
||||||
|
type CopilotFailureDiagnosisToolUnlockCommand,
|
||||||
|
type CopilotFailureDiagnosisToolUnlockReceipt,
|
||||||
|
type CopilotFailureDiagnosisToolUnlockRepository,
|
||||||
|
} from './contracts';
|
||||||
|
import {
|
||||||
|
normalizeCopilotFailureDiagnosisToolUnlockCommand,
|
||||||
|
normalizeCopilotFailureDiagnosisToolUnlockReceipt,
|
||||||
|
} from './unlockProtocol';
|
||||||
|
|
||||||
|
const UNLOCK_TABLE = 'copilot_failure_diagnosis_tool_unlocks';
|
||||||
|
const RETRYABLE_SQL_STATES = new Set(['40001', '40P01']);
|
||||||
|
const MAX_TRANSACTION_ATTEMPTS = 3;
|
||||||
|
|
||||||
|
type Row = Readonly<Record<string, unknown>>;
|
||||||
|
|
||||||
|
function unavailable(
|
||||||
|
cause?: unknown,
|
||||||
|
): CopilotFailureDiagnosisToolExecutionUnavailableError {
|
||||||
|
return new CopilotFailureDiagnosisToolExecutionUnavailableError({
|
||||||
|
cause: cause instanceof Error ? cause : undefined,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function sqlState(error: unknown): string | undefined {
|
||||||
|
if (!error || typeof error !== 'object') return undefined;
|
||||||
|
const value = (error as { code?: unknown }).code;
|
||||||
|
return typeof value === 'string' ? value : undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
function mapStorageError(error: unknown): Error {
|
||||||
|
if (
|
||||||
|
error instanceof CopilotFailureDiagnosisToolExecutionConflictError ||
|
||||||
|
error instanceof CopilotFailureDiagnosisToolExecutionUnavailableError
|
||||||
|
) {
|
||||||
|
return error;
|
||||||
|
}
|
||||||
|
if (['23503', '23505', '23514'].includes(sqlState(error) ?? '')) {
|
||||||
|
return new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||||
|
'a durable Tool unlock identity is already bound',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return unavailable(error);
|
||||||
|
}
|
||||||
|
|
||||||
|
function text(row: Row, key: string): string {
|
||||||
|
const value = row[key];
|
||||||
|
if (typeof value !== 'string' || value.length === 0) throw unavailable();
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
function integer(row: Row, key: string): number {
|
||||||
|
const value = row[key];
|
||||||
|
if (typeof value === 'number' && Number.isSafeInteger(value)) return value;
|
||||||
|
if (typeof value === 'string' && /^(0|[1-9]\d*)$/.test(value)) {
|
||||||
|
const parsed = Number(value);
|
||||||
|
if (Number.isSafeInteger(parsed)) return parsed;
|
||||||
|
}
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
|
||||||
|
function jsonObject(value: unknown): Record<string, unknown> {
|
||||||
|
let parsed = value;
|
||||||
|
if (typeof value === 'string') {
|
||||||
|
try {
|
||||||
|
parsed = JSON.parse(value);
|
||||||
|
} catch (cause) {
|
||||||
|
throw unavailable(cause);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
return parsed as Record<string, unknown>;
|
||||||
|
}
|
||||||
|
|
||||||
|
function same(left: unknown, right: unknown): boolean {
|
||||||
|
return isDeepStrictEqual(left, right);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function begin(client: PostgresClient): Promise<void> {
|
||||||
|
await client.query('BEGIN ISOLATION LEVEL SERIALIZABLE');
|
||||||
|
await client.query(`SELECT set_config('statement_timeout', $1, true)`, [
|
||||||
|
'5s',
|
||||||
|
]);
|
||||||
|
await client.query(`SELECT set_config('lock_timeout', $1, true)`, ['2s']);
|
||||||
|
await client.query(
|
||||||
|
`SELECT set_config('idle_in_transaction_session_timeout', $1, true)`,
|
||||||
|
['5s'],
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function rollback(client: PostgresClient): Promise<void> {
|
||||||
|
try {
|
||||||
|
await client.query('ROLLBACK');
|
||||||
|
} catch {
|
||||||
|
// Preserve the original transaction failure.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type StoredUnlock = Readonly<{
|
||||||
|
receipt: Readonly<CopilotFailureDiagnosisToolUnlockReceipt>;
|
||||||
|
commandDigest: string;
|
||||||
|
}>;
|
||||||
|
|
||||||
|
const UNLOCK_SELECT = `
|
||||||
|
unlock.request_id AS "requestId",
|
||||||
|
unlock.plan_digest AS "planDigest",
|
||||||
|
unlock.start_id AS "startId",
|
||||||
|
unlock.tool_completion_digest AS "toolCompletionDigest",
|
||||||
|
unlock.model_step_run_id AS "modelStepRunId",
|
||||||
|
unlock.model_step_run_version AS "modelStepRunVersion",
|
||||||
|
unlock.model_step_run_digest AS "modelStepRunDigest",
|
||||||
|
unlock.model_mutation_id AS "modelMutationId",
|
||||||
|
unlock.model_mutation_digest AS "modelMutationDigest",
|
||||||
|
unlock.model_event_id AS "modelEventId",
|
||||||
|
unlock.final_run_version AS "finalRunVersion",
|
||||||
|
unlock.final_run_event_sequence AS "finalRunEventSequence",
|
||||||
|
unlock.unlocked_at_ms AS "unlockedAtMs",
|
||||||
|
unlock.receipt_digest AS "receiptDigest",
|
||||||
|
unlock.command_digest AS "commandDigest",
|
||||||
|
unlock.receipt_json AS "receiptJson",
|
||||||
|
admission.plan_digest AS "joinedPlanDigest",
|
||||||
|
completion.completion_digest AS "joinedCompletionDigest",
|
||||||
|
mutation.mutation_digest AS "joinedMutationDigest",
|
||||||
|
mutation.step_run_digest AS "joinedModelStepRunDigest",
|
||||||
|
mutation.event_id AS "joinedEventId",
|
||||||
|
mutation.event_sequence AS "joinedEventSequence",
|
||||||
|
mutation.run_version AS "joinedRunVersion"`;
|
||||||
|
|
||||||
|
async function findRows(
|
||||||
|
queryable: PostgresQueryable,
|
||||||
|
where: string,
|
||||||
|
values: readonly unknown[],
|
||||||
|
): Promise<readonly Row[]> {
|
||||||
|
const result = await queryable.query<Row>(
|
||||||
|
`SELECT ${UNLOCK_SELECT}
|
||||||
|
FROM "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${UNLOCK_TABLE}" AS unlock
|
||||||
|
JOIN "${POSTGRES_MODEL_INVOCATION_SCHEMA}".
|
||||||
|
"copilot_failure_diagnosis_admissions" AS admission
|
||||||
|
ON admission.request_id = unlock.request_id
|
||||||
|
JOIN "ql3"."tool_execution_completions" AS completion
|
||||||
|
ON completion.start_id = unlock.start_id
|
||||||
|
JOIN "ql3"."step_run_mutations" AS mutation
|
||||||
|
ON mutation.mutation_id = unlock.model_mutation_id
|
||||||
|
WHERE ${where}
|
||||||
|
LIMIT 2`,
|
||||||
|
values,
|
||||||
|
);
|
||||||
|
return result.rows;
|
||||||
|
}
|
||||||
|
|
||||||
|
function storedUnlock(row: Row): StoredUnlock {
|
||||||
|
let receipt: Readonly<CopilotFailureDiagnosisToolUnlockReceipt>;
|
||||||
|
try {
|
||||||
|
receipt = normalizeCopilotFailureDiagnosisToolUnlockReceipt(
|
||||||
|
jsonObject(
|
||||||
|
row.receiptJson,
|
||||||
|
) as unknown as CopilotFailureDiagnosisToolUnlockReceipt,
|
||||||
|
);
|
||||||
|
} catch (cause) {
|
||||||
|
throw unavailable(cause);
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
text(row, 'requestId') !== receipt.requestId ||
|
||||||
|
text(row, 'planDigest') !== receipt.planDigest ||
|
||||||
|
text(row, 'startId') !== receipt.startId ||
|
||||||
|
text(row, 'toolCompletionDigest') !== receipt.toolCompletionDigest ||
|
||||||
|
text(row, 'modelStepRunId') !== receipt.modelStepRunId ||
|
||||||
|
integer(row, 'modelStepRunVersion') !== receipt.modelStepRunVersion ||
|
||||||
|
text(row, 'modelStepRunDigest') !== receipt.modelStepRunDigest ||
|
||||||
|
text(row, 'modelMutationId') !== receipt.modelMutationId ||
|
||||||
|
text(row, 'modelMutationDigest') !== receipt.modelMutationDigest ||
|
||||||
|
text(row, 'modelEventId') !== receipt.modelEventId ||
|
||||||
|
integer(row, 'finalRunVersion') !== receipt.finalRunVersion ||
|
||||||
|
integer(row, 'finalRunEventSequence') !== receipt.finalRunEventSequence ||
|
||||||
|
integer(row, 'unlockedAtMs') !== receipt.unlockedAtMs ||
|
||||||
|
text(row, 'receiptDigest') !== receipt.receiptDigest ||
|
||||||
|
text(row, 'joinedPlanDigest') !== receipt.planDigest ||
|
||||||
|
text(row, 'joinedCompletionDigest') !== receipt.toolCompletionDigest ||
|
||||||
|
text(row, 'joinedMutationDigest') !== receipt.modelMutationDigest ||
|
||||||
|
text(row, 'joinedModelStepRunDigest') !== receipt.modelStepRunDigest ||
|
||||||
|
text(row, 'joinedEventId') !== receipt.modelEventId ||
|
||||||
|
integer(row, 'joinedEventSequence') !== receipt.finalRunEventSequence ||
|
||||||
|
integer(row, 'joinedRunVersion') !== receipt.finalRunVersion
|
||||||
|
) {
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
return Object.freeze({
|
||||||
|
receipt,
|
||||||
|
commandDigest: text(row, 'commandDigest'),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async function updateModelStepRun(
|
||||||
|
client: PostgresClient,
|
||||||
|
command: Readonly<CopilotFailureDiagnosisToolUnlockCommand>,
|
||||||
|
): Promise<void> {
|
||||||
|
const mutation = command.modelStepRunMutation;
|
||||||
|
const step = mutation.stepRun;
|
||||||
|
const result = await client.query(
|
||||||
|
`UPDATE "ql3"."step_runs"
|
||||||
|
SET status = $1, version = $2, attempt_count = $3,
|
||||||
|
output_ref = $4, approval_request_id = $5, ready_at_ms = $6,
|
||||||
|
started_at_ms = $7, finished_at_ms = $8, result_code = $9,
|
||||||
|
error_summary = $10, updated_at_ms = $11,
|
||||||
|
last_mutation_id = $12, step_run_digest = $13,
|
||||||
|
step_run_json = $14::jsonb
|
||||||
|
WHERE id = $15 AND run_id = $16 AND version = $17
|
||||||
|
AND step_run_digest = $18 AND status = $19`,
|
||||||
|
[
|
||||||
|
step.status,
|
||||||
|
step.version,
|
||||||
|
step.attemptCount,
|
||||||
|
step.outputRef,
|
||||||
|
step.approvalRequestId,
|
||||||
|
step.readyAtMs,
|
||||||
|
step.startedAtMs,
|
||||||
|
step.finishedAtMs,
|
||||||
|
step.resultCode,
|
||||||
|
step.errorSummary,
|
||||||
|
step.updatedAtMs,
|
||||||
|
step.lastMutationId,
|
||||||
|
step.stepRunDigest,
|
||||||
|
JSON.stringify(step),
|
||||||
|
step.id,
|
||||||
|
step.runId,
|
||||||
|
mutation.expectedStepRunVersion,
|
||||||
|
mutation.expectedStepRunDigest,
|
||||||
|
mutation.previousStatus,
|
||||||
|
],
|
||||||
|
);
|
||||||
|
if (result.rowCount !== 1) {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||||
|
'the Model StepRun unlock fence changed',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function updateRun(
|
||||||
|
client: PostgresClient,
|
||||||
|
command: Readonly<CopilotFailureDiagnosisToolUnlockCommand>,
|
||||||
|
): Promise<void> {
|
||||||
|
const mutation = command.modelStepRunMutation;
|
||||||
|
const result = await client.query(
|
||||||
|
`UPDATE "ql3"."runs"
|
||||||
|
SET version = version + 1, event_sequence = event_sequence + 1
|
||||||
|
WHERE id = $1 AND status = 'running'
|
||||||
|
AND version = $2 AND event_sequence = $3`,
|
||||||
|
[
|
||||||
|
mutation.runId,
|
||||||
|
mutation.expectedRunVersion,
|
||||||
|
mutation.expectedRunEventSequence,
|
||||||
|
],
|
||||||
|
);
|
||||||
|
if (result.rowCount !== 1) {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||||
|
'the diagnosis Run unlock fence changed',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function insertEventAndMutation(
|
||||||
|
client: PostgresClient,
|
||||||
|
command: Readonly<CopilotFailureDiagnosisToolUnlockCommand>,
|
||||||
|
): Promise<void> {
|
||||||
|
const mutation = command.modelStepRunMutation;
|
||||||
|
const event = mutation.event;
|
||||||
|
await client.query(
|
||||||
|
`INSERT INTO "ql3"."run_events" (
|
||||||
|
id, run_id, sequence, type, dedupe_key, actor_type, actor_id,
|
||||||
|
attempt_id, step_run_id, payload, created_at_ms
|
||||||
|
) VALUES ($1, $2, $3, $4, $5, $6, $7, NULL, $8, $9::jsonb, $10)`,
|
||||||
|
[
|
||||||
|
event.id,
|
||||||
|
event.runId,
|
||||||
|
event.sequence,
|
||||||
|
event.type,
|
||||||
|
event.dedupeKey,
|
||||||
|
event.actorType,
|
||||||
|
event.actorId ?? null,
|
||||||
|
mutation.stepRun.id,
|
||||||
|
JSON.stringify(event.payload),
|
||||||
|
event.createdAtMs,
|
||||||
|
],
|
||||||
|
);
|
||||||
|
await client.query(
|
||||||
|
`INSERT INTO "ql3"."step_run_mutations" (
|
||||||
|
mutation_id, mutation_digest, run_id, step_run_id,
|
||||||
|
step_run_digest, event_id, event_sequence, run_version,
|
||||||
|
step_run_json, committed_at_ms
|
||||||
|
) VALUES (
|
||||||
|
$1, $2, $3, $4, $5, $6, $7, $8, $9::jsonb,
|
||||||
|
floor(extract(epoch FROM transaction_timestamp()) * 1000)::bigint
|
||||||
|
)`,
|
||||||
|
[
|
||||||
|
mutation.mutationId,
|
||||||
|
mutation.mutationDigest,
|
||||||
|
mutation.runId,
|
||||||
|
mutation.stepRun.id,
|
||||||
|
mutation.stepRun.stepRunDigest,
|
||||||
|
mutation.event.id,
|
||||||
|
mutation.event.sequence,
|
||||||
|
mutation.expectedRunVersion + 1,
|
||||||
|
JSON.stringify(mutation.stepRun),
|
||||||
|
],
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function insertUnlock(
|
||||||
|
client: PostgresClient,
|
||||||
|
command: Readonly<CopilotFailureDiagnosisToolUnlockCommand>,
|
||||||
|
): Promise<void> {
|
||||||
|
const receipt = command.receipt;
|
||||||
|
await client.query(
|
||||||
|
`INSERT INTO "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${UNLOCK_TABLE}" (
|
||||||
|
request_id, plan_digest, run_id, start_id, tool_step_run_id,
|
||||||
|
tool_completion_digest,
|
||||||
|
model_step_run_id, model_step_run_version, model_step_run_digest,
|
||||||
|
model_mutation_id, model_mutation_digest, model_event_id,
|
||||||
|
final_run_version, final_run_event_sequence, unlocked_at_ms,
|
||||||
|
receipt_digest, command_digest, receipt_json
|
||||||
|
) VALUES (
|
||||||
|
$1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13,
|
||||||
|
$14, $15, $16, $17, $18::jsonb
|
||||||
|
)`,
|
||||||
|
[
|
||||||
|
receipt.requestId,
|
||||||
|
receipt.planDigest,
|
||||||
|
receipt.runId,
|
||||||
|
receipt.startId,
|
||||||
|
receipt.toolStepRunId,
|
||||||
|
receipt.toolCompletionDigest,
|
||||||
|
receipt.modelStepRunId,
|
||||||
|
receipt.modelStepRunVersion,
|
||||||
|
receipt.modelStepRunDigest,
|
||||||
|
receipt.modelMutationId,
|
||||||
|
receipt.modelMutationDigest,
|
||||||
|
receipt.modelEventId,
|
||||||
|
receipt.finalRunVersion,
|
||||||
|
receipt.finalRunEventSequence,
|
||||||
|
receipt.unlockedAtMs,
|
||||||
|
receipt.receiptDigest,
|
||||||
|
command.commandDigest,
|
||||||
|
JSON.stringify(receipt),
|
||||||
|
],
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function currentEvidence(
|
||||||
|
client: PostgresClient,
|
||||||
|
command: Readonly<CopilotFailureDiagnosisToolUnlockCommand>,
|
||||||
|
): Promise<void> {
|
||||||
|
const mutation = command.modelStepRunMutation;
|
||||||
|
const result = await client.query<Row>(
|
||||||
|
`SELECT admission.plan_json AS "planJson",
|
||||||
|
completion.completion_json AS "completionJson",
|
||||||
|
model_step.kind AS "modelKind",
|
||||||
|
model_step.status AS "modelStatus",
|
||||||
|
model_step.version AS "modelVersion",
|
||||||
|
model_step.step_run_digest AS "modelDigest",
|
||||||
|
run.project_id AS "projectId", run.status AS "runStatus",
|
||||||
|
run.version AS "runVersion",
|
||||||
|
run.event_sequence AS "runEventSequence"
|
||||||
|
FROM "${POSTGRES_MODEL_INVOCATION_SCHEMA}".
|
||||||
|
"copilot_failure_diagnosis_admissions" AS admission
|
||||||
|
JOIN "ql3"."tool_execution_completions" AS completion
|
||||||
|
ON completion.start_id = $2
|
||||||
|
AND completion.run_id = admission.run_id
|
||||||
|
AND completion.step_run_id = admission.tool_step_run_id
|
||||||
|
JOIN "ql3"."step_runs" AS model_step
|
||||||
|
ON model_step.run_id = admission.run_id
|
||||||
|
AND model_step.id = admission.model_step_run_id
|
||||||
|
JOIN "ql3"."runs" AS run ON run.id = admission.run_id
|
||||||
|
WHERE admission.request_id = $1
|
||||||
|
LIMIT 2
|
||||||
|
FOR UPDATE OF model_step, run`,
|
||||||
|
[command.plan.requestId, command.completion.startId],
|
||||||
|
);
|
||||||
|
const row = result.rows[0];
|
||||||
|
let plan: Readonly<CopilotFailureDiagnosisExecutionPlan>;
|
||||||
|
let completion: Readonly<ToolExecutionCompletionRecord>;
|
||||||
|
try {
|
||||||
|
plan = row
|
||||||
|
? normalizeCopilotFailureDiagnosisExecutionPlan(
|
||||||
|
jsonObject(
|
||||||
|
row.planJson,
|
||||||
|
) as unknown as CopilotFailureDiagnosisExecutionPlan,
|
||||||
|
)
|
||||||
|
: command.plan;
|
||||||
|
completion = row
|
||||||
|
? normalizeToolExecutionCompletionRecord(
|
||||||
|
jsonObject(
|
||||||
|
row.completionJson,
|
||||||
|
) as unknown as ToolExecutionCompletionRecord,
|
||||||
|
)
|
||||||
|
: command.completion;
|
||||||
|
} catch (cause) {
|
||||||
|
throw unavailable(cause);
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
result.rows.length !== 1 ||
|
||||||
|
!row ||
|
||||||
|
!same(plan, command.plan) ||
|
||||||
|
!same(completion, command.completion) ||
|
||||||
|
text(row, 'modelKind') !== 'model' ||
|
||||||
|
text(row, 'modelStatus') !== mutation.previousStatus ||
|
||||||
|
integer(row, 'modelVersion') !== mutation.expectedStepRunVersion ||
|
||||||
|
text(row, 'modelDigest') !== mutation.expectedStepRunDigest ||
|
||||||
|
text(row, 'projectId') !== command.plan.projectId ||
|
||||||
|
text(row, 'runStatus') !== 'running' ||
|
||||||
|
integer(row, 'runVersion') !== mutation.expectedRunVersion ||
|
||||||
|
integer(row, 'runEventSequence') !== mutation.expectedRunEventSequence
|
||||||
|
) {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||||
|
'the admitted Tool completion or Model StepRun fence changed',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export class PostgresCopilotFailureDiagnosisToolUnlockRepository
|
||||||
|
implements CopilotFailureDiagnosisToolUnlockRepository
|
||||||
|
{
|
||||||
|
constructor(private readonly pool: PostgresPool) {
|
||||||
|
if (
|
||||||
|
!pool ||
|
||||||
|
typeof pool.query !== 'function' ||
|
||||||
|
typeof pool.connect !== 'function'
|
||||||
|
) {
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async findByRequestId(
|
||||||
|
requestId: string,
|
||||||
|
): Promise<Readonly<CopilotFailureDiagnosisToolUnlockReceipt> | null> {
|
||||||
|
if (
|
||||||
|
typeof requestId !== 'string' ||
|
||||||
|
!/^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/.test(requestId)
|
||||||
|
) {
|
||||||
|
throw new TypeError('Tool unlock request id is invalid');
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const rows = await findRows(this.pool, 'unlock.request_id = $1', [
|
||||||
|
requestId,
|
||||||
|
]);
|
||||||
|
if (rows.length > 1) throw unavailable();
|
||||||
|
return rows[0] ? storedUnlock(rows[0]).receipt : null;
|
||||||
|
} catch (error) {
|
||||||
|
throw mapStorageError(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async commit(commandValue: CopilotFailureDiagnosisToolUnlockCommand): Promise<
|
||||||
|
Readonly<{
|
||||||
|
status: 'created' | 'existing';
|
||||||
|
receipt: Readonly<CopilotFailureDiagnosisToolUnlockReceipt>;
|
||||||
|
}>
|
||||||
|
> {
|
||||||
|
const command =
|
||||||
|
normalizeCopilotFailureDiagnosisToolUnlockCommand(commandValue);
|
||||||
|
for (let attempt = 0; attempt < MAX_TRANSACTION_ATTEMPTS; attempt += 1) {
|
||||||
|
let client: PostgresClient;
|
||||||
|
try {
|
||||||
|
client = await this.pool.connect();
|
||||||
|
} catch (cause) {
|
||||||
|
throw unavailable(cause);
|
||||||
|
}
|
||||||
|
let began = false;
|
||||||
|
try {
|
||||||
|
await begin(client);
|
||||||
|
began = true;
|
||||||
|
const existingRows = await findRows(
|
||||||
|
client,
|
||||||
|
`unlock.request_id = $1
|
||||||
|
OR unlock.tool_completion_digest = $2
|
||||||
|
OR unlock.model_mutation_id = $3
|
||||||
|
OR unlock.model_event_id = $4`,
|
||||||
|
[
|
||||||
|
command.receipt.requestId,
|
||||||
|
command.receipt.toolCompletionDigest,
|
||||||
|
command.receipt.modelMutationId,
|
||||||
|
command.receipt.modelEventId,
|
||||||
|
],
|
||||||
|
);
|
||||||
|
if (existingRows.length > 1) {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionConflictError();
|
||||||
|
}
|
||||||
|
if (existingRows[0]) {
|
||||||
|
const stored = storedUnlock(existingRows[0]);
|
||||||
|
if (
|
||||||
|
stored.commandDigest !== command.commandDigest ||
|
||||||
|
!same(stored.receipt, command.receipt)
|
||||||
|
) {
|
||||||
|
throw new CopilotFailureDiagnosisToolExecutionConflictError();
|
||||||
|
}
|
||||||
|
await client.query('COMMIT');
|
||||||
|
began = false;
|
||||||
|
return Object.freeze({
|
||||||
|
status: 'existing' as const,
|
||||||
|
receipt: stored.receipt,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
await currentEvidence(client, command);
|
||||||
|
await updateModelStepRun(client, command);
|
||||||
|
await updateRun(client, command);
|
||||||
|
await insertEventAndMutation(client, command);
|
||||||
|
await insertUnlock(client, command);
|
||||||
|
await client.query('COMMIT');
|
||||||
|
began = false;
|
||||||
|
return Object.freeze({
|
||||||
|
status: 'created' as const,
|
||||||
|
receipt: command.receipt,
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
if (began) await rollback(client);
|
||||||
|
if (
|
||||||
|
RETRYABLE_SQL_STATES.has(sqlState(error) ?? '') &&
|
||||||
|
attempt + 1 < MAX_TRANSACTION_ATTEMPTS
|
||||||
|
) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
throw mapStorageError(error);
|
||||||
|
} finally {
|
||||||
|
client.release();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,424 @@
|
|||||||
|
import { Buffer } from 'node:buffer';
|
||||||
|
import { createHash } from 'node:crypto';
|
||||||
|
|
||||||
|
import {
|
||||||
|
normalizeStepRunMutation,
|
||||||
|
normalizeStepRunRecord,
|
||||||
|
transitionStepRunMutation,
|
||||||
|
type StepRunRecord,
|
||||||
|
} from '@qinglong/runtime-core/step-run';
|
||||||
|
import {
|
||||||
|
normalizeToolExecutionCompletionRecord,
|
||||||
|
type ToolExecutionCompletionRecord,
|
||||||
|
} from '@qinglong/runtime-core/tool-execution-completion';
|
||||||
|
|
||||||
|
import type { CopilotFailureDiagnosisExecutionPlan } from '../admission/contracts';
|
||||||
|
import { normalizeCopilotFailureDiagnosisExecutionPlan } from '../admission/plan';
|
||||||
|
import {
|
||||||
|
COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA,
|
||||||
|
COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA,
|
||||||
|
InvalidCopilotFailureDiagnosisToolExecutionError,
|
||||||
|
MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_BYTES,
|
||||||
|
MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_BYTES,
|
||||||
|
type CopilotFailureDiagnosisRunAuthority,
|
||||||
|
type CopilotFailureDiagnosisToolUnlockCommand,
|
||||||
|
type CopilotFailureDiagnosisToolUnlockReceipt,
|
||||||
|
} from './contracts';
|
||||||
|
|
||||||
|
const RECEIPT_DIGEST_DOMAIN = Buffer.from(
|
||||||
|
'qinglong/copilot-failure-diagnosis-tool-unlock-receipt-digest@v1\0',
|
||||||
|
'utf8',
|
||||||
|
);
|
||||||
|
const COMMAND_DIGEST_DOMAIN = Buffer.from(
|
||||||
|
'qinglong/copilot-failure-diagnosis-tool-unlock-command-digest@v1\0',
|
||||||
|
'utf8',
|
||||||
|
);
|
||||||
|
const IDENTITY_DOMAIN = Buffer.from(
|
||||||
|
'qinglong/copilot-failure-diagnosis-tool-unlock-identity@v1\0',
|
||||||
|
'utf8',
|
||||||
|
);
|
||||||
|
const DIGEST_PATTERN = /^[0-9a-f]{64}$/;
|
||||||
|
const IDENTITY_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
|
||||||
|
|
||||||
|
function invalid(message: string): never {
|
||||||
|
throw new InvalidCopilotFailureDiagnosisToolExecutionError(message);
|
||||||
|
}
|
||||||
|
|
||||||
|
function hash(domain: Uint8Array, value: unknown): string {
|
||||||
|
return createHash('sha256')
|
||||||
|
.update(domain)
|
||||||
|
.update(JSON.stringify(value))
|
||||||
|
.digest('hex');
|
||||||
|
}
|
||||||
|
|
||||||
|
function dataRecord(value: unknown, label: string): Record<string, unknown> {
|
||||||
|
if (
|
||||||
|
!value ||
|
||||||
|
typeof value !== 'object' ||
|
||||||
|
Array.isArray(value) ||
|
||||||
|
Object.getPrototypeOf(value) !== Object.prototype
|
||||||
|
) {
|
||||||
|
return invalid(`${label} must be a plain object`);
|
||||||
|
}
|
||||||
|
return value as Record<string, unknown>;
|
||||||
|
}
|
||||||
|
|
||||||
|
function exactKeys(
|
||||||
|
value: object,
|
||||||
|
keys: readonly string[],
|
||||||
|
label: string,
|
||||||
|
): void {
|
||||||
|
const actual = Object.keys(value).sort();
|
||||||
|
const expected = [...keys].sort();
|
||||||
|
if (
|
||||||
|
actual.length !== expected.length ||
|
||||||
|
actual.some((key, index) => key !== expected[index])
|
||||||
|
) {
|
||||||
|
invalid(`${label} shape is invalid`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function identity(value: unknown, label: string): string {
|
||||||
|
if (typeof value !== 'string' || !IDENTITY_PATTERN.test(value)) {
|
||||||
|
return invalid(`${label} is invalid`);
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
function digest(value: unknown, label: string): string {
|
||||||
|
if (typeof value !== 'string' || !DIGEST_PATTERN.test(value)) {
|
||||||
|
return invalid(`${label} is invalid`);
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
function integer(value: unknown, minimum: number, label: string): number {
|
||||||
|
if (!Number.isSafeInteger(value) || (value as number) < minimum) {
|
||||||
|
return invalid(`${label} is invalid`);
|
||||||
|
}
|
||||||
|
return value as number;
|
||||||
|
}
|
||||||
|
|
||||||
|
function evidenceIdentity(
|
||||||
|
prefix: 'cdum' | 'cdue',
|
||||||
|
planDigest: string,
|
||||||
|
completionDigest: string,
|
||||||
|
): string {
|
||||||
|
const maximumDigestLength = 35 - prefix.length;
|
||||||
|
return `${prefix}:${hash(IDENTITY_DOMAIN, {
|
||||||
|
prefix,
|
||||||
|
planDigest,
|
||||||
|
completionDigest,
|
||||||
|
}).slice(0, maximumDigestLength)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function copilotFailureDiagnosisToolUnlockReceiptDigest(
|
||||||
|
value: Omit<CopilotFailureDiagnosisToolUnlockReceipt, 'receiptDigest'>,
|
||||||
|
): string {
|
||||||
|
return hash(RECEIPT_DIGEST_DOMAIN, value);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function normalizeCopilotFailureDiagnosisToolUnlockReceipt(
|
||||||
|
value: CopilotFailureDiagnosisToolUnlockReceipt,
|
||||||
|
): Readonly<CopilotFailureDiagnosisToolUnlockReceipt> {
|
||||||
|
const candidate = dataRecord(value, 'Tool unlock receipt');
|
||||||
|
exactKeys(
|
||||||
|
candidate,
|
||||||
|
[
|
||||||
|
'barrierDigest',
|
||||||
|
'finalRunEventSequence',
|
||||||
|
'finalRunVersion',
|
||||||
|
'modelEventId',
|
||||||
|
'modelMutationDigest',
|
||||||
|
'modelMutationId',
|
||||||
|
'modelStepRunDigest',
|
||||||
|
'modelStepRunId',
|
||||||
|
'modelStepRunVersion',
|
||||||
|
'planDigest',
|
||||||
|
'receiptDigest',
|
||||||
|
'requestId',
|
||||||
|
'resultArtifact',
|
||||||
|
'runId',
|
||||||
|
'schema',
|
||||||
|
'startId',
|
||||||
|
'toolCompletionDigest',
|
||||||
|
'toolStepRunId',
|
||||||
|
'unlockedAtMs',
|
||||||
|
],
|
||||||
|
'Tool unlock receipt',
|
||||||
|
);
|
||||||
|
if (value.schema !== COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA) {
|
||||||
|
return invalid('Tool unlock receipt schema is unsupported');
|
||||||
|
}
|
||||||
|
const artifact = dataRecord(value.resultArtifact, 'result Artifact');
|
||||||
|
exactKeys(
|
||||||
|
artifact,
|
||||||
|
['artifactDigest', 'artifactId', 'executionResultDigest', 'outputDigest'],
|
||||||
|
'result Artifact',
|
||||||
|
);
|
||||||
|
const unsigned = Object.freeze({
|
||||||
|
schema: COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA,
|
||||||
|
requestId: identity(value.requestId, 'request id'),
|
||||||
|
planDigest: digest(value.planDigest, 'plan digest'),
|
||||||
|
runId: identity(value.runId, 'Run id'),
|
||||||
|
startId: identity(value.startId, 'start id'),
|
||||||
|
barrierDigest: digest(value.barrierDigest, 'barrier digest'),
|
||||||
|
toolStepRunId: identity(value.toolStepRunId, 'Tool StepRun id'),
|
||||||
|
toolCompletionDigest: digest(
|
||||||
|
value.toolCompletionDigest,
|
||||||
|
'Tool completion digest',
|
||||||
|
),
|
||||||
|
resultArtifact: Object.freeze({
|
||||||
|
artifactId: identity(value.resultArtifact.artifactId, 'Artifact id'),
|
||||||
|
artifactDigest: digest(
|
||||||
|
value.resultArtifact.artifactDigest,
|
||||||
|
'Artifact digest',
|
||||||
|
),
|
||||||
|
outputDigest: digest(value.resultArtifact.outputDigest, 'output digest'),
|
||||||
|
executionResultDigest: digest(
|
||||||
|
value.resultArtifact.executionResultDigest,
|
||||||
|
'execution result digest',
|
||||||
|
),
|
||||||
|
}),
|
||||||
|
modelStepRunId: identity(value.modelStepRunId, 'model StepRun id'),
|
||||||
|
modelStepRunVersion: integer(
|
||||||
|
value.modelStepRunVersion,
|
||||||
|
2,
|
||||||
|
'model StepRun version',
|
||||||
|
),
|
||||||
|
modelStepRunDigest: digest(
|
||||||
|
value.modelStepRunDigest,
|
||||||
|
'model StepRun digest',
|
||||||
|
),
|
||||||
|
modelMutationId: identity(value.modelMutationId, 'model mutation id'),
|
||||||
|
modelMutationDigest: digest(
|
||||||
|
value.modelMutationDigest,
|
||||||
|
'model mutation digest',
|
||||||
|
),
|
||||||
|
modelEventId: identity(value.modelEventId, 'model event id'),
|
||||||
|
finalRunVersion: integer(value.finalRunVersion, 1, 'final Run version'),
|
||||||
|
finalRunEventSequence: integer(
|
||||||
|
value.finalRunEventSequence,
|
||||||
|
1,
|
||||||
|
'final Run event sequence',
|
||||||
|
),
|
||||||
|
unlockedAtMs: integer(value.unlockedAtMs, 0, 'unlock time'),
|
||||||
|
} satisfies Omit<CopilotFailureDiagnosisToolUnlockReceipt, 'receiptDigest'>);
|
||||||
|
if (unsigned.finalRunVersion !== unsigned.finalRunEventSequence) {
|
||||||
|
return invalid('Tool unlock Run fence is invalid');
|
||||||
|
}
|
||||||
|
const receiptDigest = digest(value.receiptDigest, 'receipt digest');
|
||||||
|
if (
|
||||||
|
copilotFailureDiagnosisToolUnlockReceiptDigest(unsigned) !== receiptDigest
|
||||||
|
) {
|
||||||
|
return invalid('Tool unlock receipt digest does not match');
|
||||||
|
}
|
||||||
|
const normalized = Object.freeze({ ...unsigned, receiptDigest });
|
||||||
|
if (
|
||||||
|
Buffer.byteLength(JSON.stringify(normalized), 'utf8') >
|
||||||
|
MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_BYTES
|
||||||
|
) {
|
||||||
|
return invalid('Tool unlock receipt exceeds its byte budget');
|
||||||
|
}
|
||||||
|
return normalized;
|
||||||
|
}
|
||||||
|
|
||||||
|
function commandUnsigned(
|
||||||
|
value: Readonly<CopilotFailureDiagnosisToolUnlockCommand>,
|
||||||
|
): Omit<CopilotFailureDiagnosisToolUnlockCommand, 'commandDigest'> {
|
||||||
|
const { commandDigest: _commandDigest, ...unsigned } = value;
|
||||||
|
return unsigned;
|
||||||
|
}
|
||||||
|
|
||||||
|
function validateCommandBindings(
|
||||||
|
plan: Readonly<CopilotFailureDiagnosisExecutionPlan>,
|
||||||
|
completion: Readonly<ToolExecutionCompletionRecord>,
|
||||||
|
mutation: ReturnType<typeof normalizeStepRunMutation>,
|
||||||
|
receipt: Readonly<CopilotFailureDiagnosisToolUnlockReceipt>,
|
||||||
|
): void {
|
||||||
|
if (
|
||||||
|
completion.projectId !== plan.projectId ||
|
||||||
|
completion.runId !== plan.runId ||
|
||||||
|
completion.stepRunId !== plan.toolStepRunId ||
|
||||||
|
mutation.runId !== plan.runId ||
|
||||||
|
mutation.previousStatus !== 'pending' ||
|
||||||
|
mutation.stepRun.id !== plan.modelStepRunId ||
|
||||||
|
mutation.stepRun.runId !== plan.runId ||
|
||||||
|
mutation.stepRun.parentStepRunId !== plan.toolStepRunId ||
|
||||||
|
mutation.stepRun.kind !== 'model' ||
|
||||||
|
mutation.stepRun.status !== 'ready' ||
|
||||||
|
mutation.stepRun.inputRef !== `tool-result-step:${plan.toolStepRunId}` ||
|
||||||
|
mutation.stepRun.updatedAtMs !== completion.completedAtMs ||
|
||||||
|
receipt.requestId !== plan.requestId ||
|
||||||
|
receipt.planDigest !== plan.planDigest ||
|
||||||
|
receipt.runId !== plan.runId ||
|
||||||
|
receipt.startId !== completion.startId ||
|
||||||
|
receipt.barrierDigest !== completion.barrierDigest ||
|
||||||
|
receipt.toolStepRunId !== completion.stepRunId ||
|
||||||
|
receipt.toolCompletionDigest !== completion.completionDigest ||
|
||||||
|
JSON.stringify(receipt.resultArtifact) !==
|
||||||
|
JSON.stringify(completion.resultArtifact) ||
|
||||||
|
receipt.modelStepRunId !== mutation.stepRun.id ||
|
||||||
|
receipt.modelStepRunVersion !== mutation.stepRun.version ||
|
||||||
|
receipt.modelStepRunDigest !== mutation.stepRun.stepRunDigest ||
|
||||||
|
receipt.modelMutationId !== mutation.mutationId ||
|
||||||
|
receipt.modelMutationDigest !== mutation.mutationDigest ||
|
||||||
|
receipt.modelEventId !== mutation.event.id ||
|
||||||
|
receipt.finalRunVersion !== mutation.expectedRunVersion + 1 ||
|
||||||
|
receipt.finalRunEventSequence !== mutation.expectedRunEventSequence + 1 ||
|
||||||
|
receipt.unlockedAtMs !== mutation.stepRun.updatedAtMs
|
||||||
|
) {
|
||||||
|
return invalid('Tool unlock command bindings are inconsistent');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function normalizeCopilotFailureDiagnosisToolUnlockCommand(
|
||||||
|
value: CopilotFailureDiagnosisToolUnlockCommand,
|
||||||
|
): Readonly<CopilotFailureDiagnosisToolUnlockCommand> {
|
||||||
|
const candidate = dataRecord(value, 'Tool unlock command');
|
||||||
|
exactKeys(
|
||||||
|
candidate,
|
||||||
|
[
|
||||||
|
'commandDigest',
|
||||||
|
'completion',
|
||||||
|
'modelStepRunMutation',
|
||||||
|
'plan',
|
||||||
|
'receipt',
|
||||||
|
'schema',
|
||||||
|
],
|
||||||
|
'Tool unlock command',
|
||||||
|
);
|
||||||
|
if (value.schema !== COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA) {
|
||||||
|
return invalid('Tool unlock command schema is unsupported');
|
||||||
|
}
|
||||||
|
const plan = normalizeCopilotFailureDiagnosisExecutionPlan(value.plan);
|
||||||
|
const completion = normalizeToolExecutionCompletionRecord(value.completion);
|
||||||
|
const modelStepRunMutation = normalizeStepRunMutation(
|
||||||
|
value.modelStepRunMutation,
|
||||||
|
);
|
||||||
|
const receipt = normalizeCopilotFailureDiagnosisToolUnlockReceipt(
|
||||||
|
value.receipt,
|
||||||
|
);
|
||||||
|
validateCommandBindings(plan, completion, modelStepRunMutation, receipt);
|
||||||
|
const unsigned = Object.freeze({
|
||||||
|
schema: COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA,
|
||||||
|
plan,
|
||||||
|
completion,
|
||||||
|
modelStepRunMutation,
|
||||||
|
receipt,
|
||||||
|
} satisfies Omit<CopilotFailureDiagnosisToolUnlockCommand, 'commandDigest'>);
|
||||||
|
const commandDigest = digest(value.commandDigest, 'command digest');
|
||||||
|
if (hash(COMMAND_DIGEST_DOMAIN, unsigned) !== commandDigest) {
|
||||||
|
return invalid('Tool unlock command digest does not match');
|
||||||
|
}
|
||||||
|
const normalized = Object.freeze({ ...unsigned, commandDigest });
|
||||||
|
if (
|
||||||
|
Buffer.byteLength(JSON.stringify(normalized), 'utf8') >
|
||||||
|
MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_BYTES
|
||||||
|
) {
|
||||||
|
return invalid('Tool unlock command exceeds its byte budget');
|
||||||
|
}
|
||||||
|
return normalized;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createCopilotFailureDiagnosisToolUnlockCommand(input: {
|
||||||
|
readonly plan: CopilotFailureDiagnosisExecutionPlan;
|
||||||
|
readonly completion: ToolExecutionCompletionRecord;
|
||||||
|
readonly modelStepRun: StepRunRecord;
|
||||||
|
readonly run: CopilotFailureDiagnosisRunAuthority;
|
||||||
|
}): Readonly<CopilotFailureDiagnosisToolUnlockCommand> {
|
||||||
|
const plan = normalizeCopilotFailureDiagnosisExecutionPlan(input.plan);
|
||||||
|
const completion = normalizeToolExecutionCompletionRecord(input.completion);
|
||||||
|
const modelStepRun = normalizeStepRunRecord(input.modelStepRun);
|
||||||
|
const run = dataRecord(input.run, 'Run authority');
|
||||||
|
exactKeys(
|
||||||
|
run,
|
||||||
|
['eventSequence', 'id', 'projectId', 'status', 'version'],
|
||||||
|
'Run authority',
|
||||||
|
);
|
||||||
|
if (
|
||||||
|
input.run.id !== plan.runId ||
|
||||||
|
input.run.projectId !== plan.projectId ||
|
||||||
|
input.run.status !== 'running' ||
|
||||||
|
!Number.isSafeInteger(input.run.version) ||
|
||||||
|
input.run.version < 1 ||
|
||||||
|
!Number.isSafeInteger(input.run.eventSequence) ||
|
||||||
|
input.run.eventSequence !== input.run.version ||
|
||||||
|
completion.projectId !== plan.projectId ||
|
||||||
|
completion.runId !== plan.runId ||
|
||||||
|
completion.stepRunId !== plan.toolStepRunId ||
|
||||||
|
modelStepRun.id !== plan.modelStepRunId ||
|
||||||
|
modelStepRun.runId !== plan.runId ||
|
||||||
|
modelStepRun.parentStepRunId !== plan.toolStepRunId ||
|
||||||
|
modelStepRun.kind !== 'model' ||
|
||||||
|
modelStepRun.status !== 'pending'
|
||||||
|
) {
|
||||||
|
return invalid('Tool completion cannot unlock the model StepRun');
|
||||||
|
}
|
||||||
|
const mutationId = evidenceIdentity(
|
||||||
|
'cdum',
|
||||||
|
plan.planDigest,
|
||||||
|
completion.completionDigest,
|
||||||
|
);
|
||||||
|
const eventId = evidenceIdentity(
|
||||||
|
'cdue',
|
||||||
|
plan.planDigest,
|
||||||
|
completion.completionDigest,
|
||||||
|
);
|
||||||
|
const modelStepRunMutation = transitionStepRunMutation(
|
||||||
|
modelStepRun,
|
||||||
|
{
|
||||||
|
expectedVersion: modelStepRun.version,
|
||||||
|
expectedDigest: modelStepRun.stepRunDigest,
|
||||||
|
mutationId,
|
||||||
|
to: 'ready',
|
||||||
|
atMs: completion.completedAtMs,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
expectedRunVersion: input.run.version,
|
||||||
|
expectedRunEventSequence: input.run.eventSequence,
|
||||||
|
eventId,
|
||||||
|
dedupeKey: eventId,
|
||||||
|
actor: Object.freeze({
|
||||||
|
type: 'system' as const,
|
||||||
|
id: 'copilot-runtime',
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
);
|
||||||
|
const receiptUnsigned = Object.freeze({
|
||||||
|
schema: COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA,
|
||||||
|
requestId: plan.requestId,
|
||||||
|
planDigest: plan.planDigest,
|
||||||
|
runId: plan.runId,
|
||||||
|
startId: completion.startId,
|
||||||
|
barrierDigest: completion.barrierDigest,
|
||||||
|
toolStepRunId: completion.stepRunId,
|
||||||
|
toolCompletionDigest: completion.completionDigest,
|
||||||
|
resultArtifact: completion.resultArtifact,
|
||||||
|
modelStepRunId: modelStepRunMutation.stepRun.id,
|
||||||
|
modelStepRunVersion: modelStepRunMutation.stepRun.version,
|
||||||
|
modelStepRunDigest: modelStepRunMutation.stepRun.stepRunDigest,
|
||||||
|
modelMutationId: modelStepRunMutation.mutationId,
|
||||||
|
modelMutationDigest: modelStepRunMutation.mutationDigest,
|
||||||
|
modelEventId: modelStepRunMutation.event.id,
|
||||||
|
finalRunVersion: modelStepRunMutation.expectedRunVersion + 1,
|
||||||
|
finalRunEventSequence: modelStepRunMutation.expectedRunEventSequence + 1,
|
||||||
|
unlockedAtMs: completion.completedAtMs,
|
||||||
|
} satisfies Omit<CopilotFailureDiagnosisToolUnlockReceipt, 'receiptDigest'>);
|
||||||
|
const receipt = normalizeCopilotFailureDiagnosisToolUnlockReceipt({
|
||||||
|
...receiptUnsigned,
|
||||||
|
receiptDigest:
|
||||||
|
copilotFailureDiagnosisToolUnlockReceiptDigest(receiptUnsigned),
|
||||||
|
});
|
||||||
|
const unsigned = Object.freeze({
|
||||||
|
schema: COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA,
|
||||||
|
plan,
|
||||||
|
completion,
|
||||||
|
modelStepRunMutation,
|
||||||
|
receipt,
|
||||||
|
} satisfies Omit<CopilotFailureDiagnosisToolUnlockCommand, 'commandDigest'>);
|
||||||
|
return normalizeCopilotFailureDiagnosisToolUnlockCommand({
|
||||||
|
...unsigned,
|
||||||
|
commandDigest: hash(COMMAND_DIGEST_DOMAIN, unsigned),
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
export * from './tool-execution/contracts';
|
||||||
|
export * from './tool-execution/coordinator';
|
||||||
|
export * from './tool-execution/planAuthority';
|
||||||
|
export * from './tool-execution/unlockProtocol';
|
||||||
@@ -63,6 +63,8 @@ export const POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID =
|
|||||||
'pg-9017-ai-plugin-package-prompt-product-authorization';
|
'pg-9017-ai-plugin-package-prompt-product-authorization';
|
||||||
export const POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID =
|
export const POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID =
|
||||||
'pg-9018-ai-copilot-failure-diagnosis-admissions';
|
'pg-9018-ai-copilot-failure-diagnosis-admissions';
|
||||||
|
export const POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID =
|
||||||
|
'pg-9019-ai-copilot-failure-diagnosis-tool-unlocks';
|
||||||
export const POSTGRES_MODEL_INVOCATION_SCHEMA = 'ql3_ai';
|
export const POSTGRES_MODEL_INVOCATION_SCHEMA = 'ql3_ai';
|
||||||
export const LOCAL_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE =
|
export const LOCAL_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE =
|
||||||
'QingLong3AiSchemaMigrations';
|
'QingLong3AiSchemaMigrations';
|
||||||
|
|||||||
@@ -28,6 +28,7 @@ import {
|
|||||||
POSTGRES_PLUGIN_PACKAGE_PROMPT_OUTPUT_KEY_ROTATION_MIGRATION_ID,
|
POSTGRES_PLUGIN_PACKAGE_PROMPT_OUTPUT_KEY_ROTATION_MIGRATION_ID,
|
||||||
POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID,
|
POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID,
|
||||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
||||||
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||||
POSTGRES_MODEL_INVOCATION_SCHEMA,
|
POSTGRES_MODEL_INVOCATION_SCHEMA,
|
||||||
POSTGRES_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE,
|
POSTGRES_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE,
|
||||||
} from './identities';
|
} from './identities';
|
||||||
@@ -63,6 +64,7 @@ const POSTGRES_HISTORY_IDENTITY = Object.freeze({
|
|||||||
POSTGRES_MODEL_PROVIDER_CREDENTIAL_TEST_CONNECTION_MIGRATION_ID,
|
POSTGRES_MODEL_PROVIDER_CREDENTIAL_TEST_CONNECTION_MIGRATION_ID,
|
||||||
POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID,
|
POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID,
|
||||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
||||||
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||||
]),
|
]),
|
||||||
streamId: POSTGRES_MODEL_INVOCATION_MIGRATION_STREAM_ID,
|
streamId: POSTGRES_MODEL_INVOCATION_MIGRATION_STREAM_ID,
|
||||||
dialect: 'postgresql' as const,
|
dialect: 'postgresql' as const,
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ import type { PostgresQueryable } from '@qinglong/runtime-core';
|
|||||||
|
|
||||||
import {
|
import {
|
||||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
||||||
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||||
POSTGRES_MODEL_INVOCATION_SCHEMA,
|
POSTGRES_MODEL_INVOCATION_SCHEMA,
|
||||||
} from '../identities';
|
} from '../identities';
|
||||||
import { defineSqlMigration } from '../shared';
|
import { defineSqlMigration } from '../shared';
|
||||||
@@ -9,6 +10,7 @@ import { defineSqlMigration } from '../shared';
|
|||||||
const ADMISSION_TABLE = 'copilot_failure_diagnosis_admissions';
|
const ADMISSION_TABLE = 'copilot_failure_diagnosis_admissions';
|
||||||
const SOURCE_SNAPSHOT_FUNCTION =
|
const SOURCE_SNAPSHOT_FUNCTION =
|
||||||
'copilot_failure_diagnosis_admission_source_snapshot';
|
'copilot_failure_diagnosis_admission_source_snapshot';
|
||||||
|
const TOOL_UNLOCK_TABLE = 'copilot_failure_diagnosis_tool_unlocks';
|
||||||
|
|
||||||
const POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_TABLE_SQL = `
|
const POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_TABLE_SQL = `
|
||||||
CREATE TABLE "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${ADMISSION_TABLE}" (
|
CREATE TABLE "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${ADMISSION_TABLE}" (
|
||||||
@@ -215,6 +217,109 @@ const postgresCopilotFailureDiagnosisAdmissionMigration =
|
|||||||
(context, statement) => context.query(statement).then(() => undefined),
|
(context, statement) => context.query(statement).then(() => undefined),
|
||||||
);
|
);
|
||||||
|
|
||||||
|
const POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_TABLE_SQL = `
|
||||||
|
CREATE TABLE "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${TOOL_UNLOCK_TABLE}" (
|
||||||
|
request_id varchar(128) PRIMARY KEY,
|
||||||
|
plan_digest char(64) NOT NULL UNIQUE,
|
||||||
|
run_id varchar(36) NOT NULL,
|
||||||
|
start_id varchar(36) NOT NULL UNIQUE,
|
||||||
|
tool_step_run_id varchar(128) NOT NULL,
|
||||||
|
tool_completion_digest char(64) NOT NULL UNIQUE,
|
||||||
|
model_step_run_id varchar(128) NOT NULL UNIQUE,
|
||||||
|
model_step_run_version integer NOT NULL,
|
||||||
|
model_step_run_digest char(64) NOT NULL,
|
||||||
|
model_mutation_id varchar(36) NOT NULL UNIQUE,
|
||||||
|
model_mutation_digest char(64) NOT NULL UNIQUE,
|
||||||
|
model_event_id varchar(36) NOT NULL UNIQUE,
|
||||||
|
final_run_version integer NOT NULL,
|
||||||
|
final_run_event_sequence integer NOT NULL,
|
||||||
|
unlocked_at_ms bigint NOT NULL,
|
||||||
|
receipt_digest char(64) NOT NULL UNIQUE,
|
||||||
|
command_digest char(64) NOT NULL UNIQUE,
|
||||||
|
receipt_json jsonb NOT NULL,
|
||||||
|
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_admission_fk
|
||||||
|
FOREIGN KEY (request_id)
|
||||||
|
REFERENCES "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${ADMISSION_TABLE}"
|
||||||
|
(request_id) ON DELETE RESTRICT,
|
||||||
|
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_completion_fk
|
||||||
|
FOREIGN KEY (start_id)
|
||||||
|
REFERENCES "ql3"."tool_execution_completions" (start_id)
|
||||||
|
ON DELETE RESTRICT,
|
||||||
|
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_model_step_fk
|
||||||
|
FOREIGN KEY (run_id, model_step_run_id)
|
||||||
|
REFERENCES "ql3"."step_runs" (run_id, id) ON DELETE RESTRICT,
|
||||||
|
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_mutation_fk
|
||||||
|
FOREIGN KEY (model_mutation_id)
|
||||||
|
REFERENCES "ql3"."step_run_mutations" (mutation_id)
|
||||||
|
ON DELETE RESTRICT,
|
||||||
|
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_event_fk
|
||||||
|
FOREIGN KEY (model_event_id)
|
||||||
|
REFERENCES "ql3"."run_events" (id) ON DELETE RESTRICT,
|
||||||
|
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_identity_check CHECK (
|
||||||
|
request_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' AND
|
||||||
|
run_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$' AND
|
||||||
|
start_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$' AND
|
||||||
|
tool_step_run_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' AND
|
||||||
|
model_step_run_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' AND
|
||||||
|
model_mutation_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$' AND
|
||||||
|
model_event_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$'
|
||||||
|
),
|
||||||
|
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_version_check CHECK (
|
||||||
|
model_step_run_version >= 2 AND
|
||||||
|
final_run_version >= 1 AND
|
||||||
|
final_run_event_sequence = final_run_version AND
|
||||||
|
unlocked_at_ms >= 0
|
||||||
|
),
|
||||||
|
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_digest_check CHECK (
|
||||||
|
plan_digest ~ '^[0-9a-f]{64}$' AND
|
||||||
|
tool_completion_digest ~ '^[0-9a-f]{64}$' AND
|
||||||
|
model_step_run_digest ~ '^[0-9a-f]{64}$' AND
|
||||||
|
model_mutation_digest ~ '^[0-9a-f]{64}$' AND
|
||||||
|
receipt_digest ~ '^[0-9a-f]{64}$' AND
|
||||||
|
command_digest ~ '^[0-9a-f]{64}$'
|
||||||
|
),
|
||||||
|
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_json_check CHECK (
|
||||||
|
jsonb_typeof(receipt_json) = 'object' AND
|
||||||
|
octet_length(receipt_json::text) BETWEEN 2 AND 16384 AND
|
||||||
|
receipt_json @> jsonb_build_object(
|
||||||
|
'schema',
|
||||||
|
'qinglong/copilot-failure-diagnosis-tool-unlock-receipt@v1',
|
||||||
|
'requestId', request_id, 'planDigest', plan_digest,
|
||||||
|
'runId', run_id, 'startId', start_id,
|
||||||
|
'toolStepRunId', tool_step_run_id,
|
||||||
|
'toolCompletionDigest', tool_completion_digest,
|
||||||
|
'modelStepRunId', model_step_run_id,
|
||||||
|
'modelStepRunVersion', model_step_run_version,
|
||||||
|
'modelStepRunDigest', model_step_run_digest,
|
||||||
|
'modelMutationId', model_mutation_id,
|
||||||
|
'modelMutationDigest', model_mutation_digest,
|
||||||
|
'modelEventId', model_event_id,
|
||||||
|
'finalRunVersion', final_run_version,
|
||||||
|
'finalRunEventSequence', final_run_event_sequence,
|
||||||
|
'unlockedAtMs', unlocked_at_ms, 'receiptDigest', receipt_digest
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)`;
|
||||||
|
|
||||||
|
const postgresCopilotFailureDiagnosisToolUnlockMigration =
|
||||||
|
defineSqlMigration<PostgresQueryable>(
|
||||||
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||||
|
[
|
||||||
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_TABLE_SQL,
|
||||||
|
`CREATE UNIQUE INDEX ql3_ai_copilot_diagnosis_tool_unlock_fence_uidx
|
||||||
|
ON "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${TOOL_UNLOCK_TABLE}"
|
||||||
|
(run_id, tool_step_run_id, model_step_run_id)`,
|
||||||
|
`REVOKE ALL ON TABLE
|
||||||
|
"${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${TOOL_UNLOCK_TABLE}"
|
||||||
|
FROM PUBLIC`,
|
||||||
|
`GRANT SELECT, INSERT ON TABLE
|
||||||
|
"${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${TOOL_UNLOCK_TABLE}"
|
||||||
|
TO ql3_runtime`,
|
||||||
|
],
|
||||||
|
(context, statement) => context.query(statement).then(() => undefined),
|
||||||
|
);
|
||||||
|
|
||||||
export const postgresCopilotMigrations = Object.freeze([
|
export const postgresCopilotMigrations = Object.freeze([
|
||||||
postgresCopilotFailureDiagnosisAdmissionMigration,
|
postgresCopilotFailureDiagnosisAdmissionMigration,
|
||||||
|
postgresCopilotFailureDiagnosisToolUnlockMigration,
|
||||||
]);
|
]);
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
const assert = require('node:assert/strict');
|
const assert = require('node:assert/strict');
|
||||||
|
const { createHash } = require('node:crypto');
|
||||||
const { test } = require('node:test');
|
const { test } = require('node:test');
|
||||||
|
|
||||||
const {
|
const {
|
||||||
@@ -33,6 +34,12 @@ const {
|
|||||||
const {
|
const {
|
||||||
storedJsonEquals,
|
storedJsonEquals,
|
||||||
} = require('../dist/copilot/failure-diagnosis/admission/postgresRepository.js');
|
} = require('../dist/copilot/failure-diagnosis/admission/postgresRepository.js');
|
||||||
|
const {
|
||||||
|
COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA,
|
||||||
|
createCopilotFailureDiagnosisToolUnlockCommand,
|
||||||
|
normalizeCopilotFailureDiagnosisToolUnlockCommand,
|
||||||
|
restoreCopilotFailureDiagnosisTrustedToolAuthority,
|
||||||
|
} = require('../dist/copilot/failure-diagnosis/toolExecution.js');
|
||||||
|
|
||||||
const DIGEST_A = 'a'.repeat(64);
|
const DIGEST_A = 'a'.repeat(64);
|
||||||
const DIGEST_B = 'b'.repeat(64);
|
const DIGEST_B = 'b'.repeat(64);
|
||||||
@@ -189,6 +196,40 @@ async function plan(options = {}) {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function successfulToolCompletion(current) {
|
||||||
|
const unsigned = Object.freeze({
|
||||||
|
schema: 'qinglong/tool-execution-completion@v1',
|
||||||
|
startId: 'cds:diagnosis-tool-start',
|
||||||
|
projectId: current.projectId,
|
||||||
|
runId: current.runId,
|
||||||
|
stepRunId: current.toolStepRunId,
|
||||||
|
startedStepRunVersion: 2,
|
||||||
|
completedStepRunVersion: 3,
|
||||||
|
barrierDigest: DIGEST_A,
|
||||||
|
adapterDigest: DIGEST_B,
|
||||||
|
resultArtifact: Object.freeze({
|
||||||
|
artifactId: 'cdra:diagnosis-result',
|
||||||
|
artifactDigest: DIGEST_C,
|
||||||
|
outputDigest: DIGEST_A,
|
||||||
|
executionResultDigest: DIGEST_B,
|
||||||
|
}),
|
||||||
|
stepRunMutationId: 'cdscm:diagnosis-tool',
|
||||||
|
stepRunMutationDigest: DIGEST_C,
|
||||||
|
completedStepRunDigest: DIGEST_A,
|
||||||
|
runEventId: 'cdsce:diagnosis-tool',
|
||||||
|
completedAtMs: 3_000,
|
||||||
|
});
|
||||||
|
return Object.freeze({
|
||||||
|
...unsigned,
|
||||||
|
completionDigest: createHash('sha256')
|
||||||
|
.update(
|
||||||
|
Buffer.from('qinglong/tool-execution-completion-digest@v1\0', 'utf8'),
|
||||||
|
)
|
||||||
|
.update(JSON.stringify(unsigned))
|
||||||
|
.digest('hex'),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
test('creates one independent diagnosis Run with ready Tool and pending Model Steps', async () => {
|
test('creates one independent diagnosis Run with ready Tool and pending Model Steps', async () => {
|
||||||
const current = await plan();
|
const current = await plan();
|
||||||
assert.equal(current.schema, COPILOT_FAILURE_DIAGNOSIS_EXECUTION_PLAN_SCHEMA);
|
assert.equal(current.schema, COPILOT_FAILURE_DIAGNOSIS_EXECUTION_PLAN_SCHEMA);
|
||||||
@@ -224,6 +265,10 @@ test('creates one independent diagnosis Run with ready Tool and pending Model St
|
|||||||
},
|
},
|
||||||
{ kind: 'tool', status: 'ready', sequence: 2 },
|
{ kind: 'tool', status: 'ready', sequence: 2 },
|
||||||
);
|
);
|
||||||
|
assert.equal(
|
||||||
|
bundle.toolStepMutation.stepRun.definitionRef,
|
||||||
|
'tool:qinglong.run.log.excerpt@1.0.0',
|
||||||
|
);
|
||||||
assert.deepEqual(
|
assert.deepEqual(
|
||||||
{
|
{
|
||||||
kind: bundle.modelStepMutation.stepRun.kind,
|
kind: bundle.modelStepMutation.stepRun.kind,
|
||||||
@@ -257,6 +302,63 @@ test('creates one independent diagnosis Run with ready Tool and pending Model St
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('restores the exact admitted Tool and binds success to one Model unlock mutation', async () => {
|
||||||
|
const current = await plan();
|
||||||
|
const admission = createCopilotFailureDiagnosisAdmissionBundle(current);
|
||||||
|
const authority = restoreCopilotFailureDiagnosisTrustedToolAuthority(
|
||||||
|
current,
|
||||||
|
snapshot(),
|
||||||
|
);
|
||||||
|
assert.equal(authority.plan.planDigest, current.tool.planDigest);
|
||||||
|
assert.equal(authority.binding.bindingDigest, current.tool.bindingDigest);
|
||||||
|
|
||||||
|
const command = createCopilotFailureDiagnosisToolUnlockCommand({
|
||||||
|
plan: current,
|
||||||
|
completion: successfulToolCompletion(current),
|
||||||
|
modelStepRun: admission.modelStepMutation.stepRun,
|
||||||
|
run: {
|
||||||
|
id: current.runId,
|
||||||
|
projectId: current.projectId,
|
||||||
|
status: 'running',
|
||||||
|
version: 5,
|
||||||
|
eventSequence: 5,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
assert.equal(
|
||||||
|
command.receipt.schema,
|
||||||
|
COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA,
|
||||||
|
);
|
||||||
|
assert.equal(command.modelStepRunMutation.previousStatus, 'pending');
|
||||||
|
assert.equal(command.modelStepRunMutation.stepRun.status, 'ready');
|
||||||
|
assert.equal(command.receipt.finalRunVersion, 6);
|
||||||
|
assert.deepEqual(
|
||||||
|
normalizeCopilotFailureDiagnosisToolUnlockCommand(
|
||||||
|
JSON.parse(JSON.stringify(command)),
|
||||||
|
),
|
||||||
|
command,
|
||||||
|
);
|
||||||
|
assert.throws(
|
||||||
|
() =>
|
||||||
|
normalizeCopilotFailureDiagnosisToolUnlockCommand({
|
||||||
|
...command,
|
||||||
|
receipt: { ...command.receipt, finalRunVersion: 7 },
|
||||||
|
}),
|
||||||
|
TypeError,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('publishes Tool execution only through explicit AI subpaths', () => {
|
||||||
|
const root = require('../dist');
|
||||||
|
const execution = require('@qinglong/ai/failure-diagnosis-tool-execution');
|
||||||
|
const storage = require('@qinglong/ai/postgres-failure-diagnosis-tool-execution-storage');
|
||||||
|
assert.equal(root.executeCopilotFailureDiagnosisTool, undefined);
|
||||||
|
assert.equal(typeof execution.executeCopilotFailureDiagnosisTool, 'function');
|
||||||
|
assert.equal(
|
||||||
|
typeof storage.PostgresCopilotFailureDiagnosisToolUnlockRepository,
|
||||||
|
'function',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
test('rejects a Tool input detached from the source failure fence', async () => {
|
test('rejects a Tool input detached from the source failure fence', async () => {
|
||||||
await assert.rejects(
|
await assert.rejects(
|
||||||
() => plan({ inputAttemptId: 'attempt-unrelated' }),
|
() => plan({ inputAttemptId: 'attempt-unrelated' }),
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ const {
|
|||||||
LOCAL_MODEL_PRICE_CATALOG_AUTHORIZATION_MIGRATION_ID,
|
LOCAL_MODEL_PRICE_CATALOG_AUTHORIZATION_MIGRATION_ID,
|
||||||
LOCAL_MODEL_PRICE_CATALOG_MIGRATION_ID,
|
LOCAL_MODEL_PRICE_CATALOG_MIGRATION_ID,
|
||||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
||||||
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||||
POSTGRES_MODEL_INVOCATION_MIGRATION_ID,
|
POSTGRES_MODEL_INVOCATION_MIGRATION_ID,
|
||||||
POSTGRES_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE,
|
POSTGRES_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE,
|
||||||
POSTGRES_MODEL_INVOCATION_MIGRATION_STREAM_ID,
|
POSTGRES_MODEL_INVOCATION_MIGRATION_STREAM_ID,
|
||||||
@@ -426,6 +427,10 @@ test('PostgreSQL AI schema is an independent reviewed feature stream', async ()
|
|||||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
||||||
'pg-9018-ai-copilot-failure-diagnosis-admissions',
|
'pg-9018-ai-copilot-failure-diagnosis-admissions',
|
||||||
);
|
);
|
||||||
|
assert.equal(
|
||||||
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||||
|
'pg-9019-ai-copilot-failure-diagnosis-tool-unlocks',
|
||||||
|
);
|
||||||
assert.equal(
|
assert.equal(
|
||||||
POSTGRES_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE,
|
POSTGRES_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE,
|
||||||
'ai_schema_migrations',
|
'ai_schema_migrations',
|
||||||
@@ -507,9 +512,13 @@ test('PostgreSQL AI schema is an independent reviewed feature stream', async ()
|
|||||||
postgresModelInvocationMigrationDefinition.migrations[17].checksum,
|
postgresModelInvocationMigrationDefinition.migrations[17].checksum,
|
||||||
'cd0837c68ecc6c2bce58d048308d0239397b6347b4483f02372f966c05ae7ad6',
|
'cd0837c68ecc6c2bce58d048308d0239397b6347b4483f02372f966c05ae7ad6',
|
||||||
);
|
);
|
||||||
|
assert.equal(
|
||||||
|
postgresModelInvocationMigrationDefinition.migrations[18].checksum,
|
||||||
|
'341733fde691e81efafabc9749197514d83af6c851e91ea55d85074d2dec6b4f',
|
||||||
|
);
|
||||||
assert.equal(
|
assert.equal(
|
||||||
postgresModelInvocationMigrationDefinition.migrations.length,
|
postgresModelInvocationMigrationDefinition.migrations.length,
|
||||||
18,
|
19,
|
||||||
);
|
);
|
||||||
|
|
||||||
const diagnosisAdmissionStatements = [];
|
const diagnosisAdmissionStatements = [];
|
||||||
@@ -537,6 +546,25 @@ test('PostgreSQL AI schema is an independent reviewed feature stream', async ()
|
|||||||
assert.match(diagnosisAdmissionSql, /TO ql3_runtime/);
|
assert.match(diagnosisAdmissionSql, /TO ql3_runtime/);
|
||||||
assert.doesNotMatch(diagnosisAdmissionSql, /GRANT[^;]*(?:UPDATE|DELETE)/);
|
assert.doesNotMatch(diagnosisAdmissionSql, /GRANT[^;]*(?:UPDATE|DELETE)/);
|
||||||
|
|
||||||
|
const diagnosisToolUnlockStatements = [];
|
||||||
|
await postgresModelInvocationMigrationDefinition.migrations[18].up({
|
||||||
|
async query(statement) {
|
||||||
|
diagnosisToolUnlockStatements.push(statement);
|
||||||
|
return { rows: [] };
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const diagnosisToolUnlockSql = diagnosisToolUnlockStatements.join('\n');
|
||||||
|
assert.match(
|
||||||
|
diagnosisToolUnlockSql,
|
||||||
|
/CREATE TABLE "ql3_ai"\."copilot_failure_diagnosis_tool_unlocks"/,
|
||||||
|
);
|
||||||
|
assert.match(
|
||||||
|
diagnosisToolUnlockSql,
|
||||||
|
/FOREIGN KEY \(start_id\)[\s\S]*tool_execution_completions/,
|
||||||
|
);
|
||||||
|
assert.match(diagnosisToolUnlockSql, /TO ql3_runtime/);
|
||||||
|
assert.doesNotMatch(diagnosisToolUnlockSql, /GRANT[^;]*(?:UPDATE|DELETE)/);
|
||||||
|
|
||||||
const retirementStatements = [];
|
const retirementStatements = [];
|
||||||
await postgresModelInvocationMigrationDefinition.migrations[10].up({
|
await postgresModelInvocationMigrationDefinition.migrations[10].up({
|
||||||
async query(statement) {
|
async query(statement) {
|
||||||
|
|||||||
+1
-1
@@ -175,7 +175,7 @@ test('readiness binds exact migration history and least-privilege primary author
|
|||||||
assert.equal(report.ready, true);
|
assert.equal(report.ready, true);
|
||||||
assert.equal(
|
assert.equal(
|
||||||
report.migrationIds.at(-1),
|
report.migrationIds.at(-1),
|
||||||
'pg-9018-ai-copilot-failure-diagnosis-admissions',
|
'pg-9019-ai-copilot-failure-diagnosis-tool-unlocks',
|
||||||
);
|
);
|
||||||
assert.match(
|
assert.match(
|
||||||
queries[1],
|
queries[1],
|
||||||
|
|||||||
@@ -277,7 +277,7 @@ test('tester readiness freezes migration history and least privilege', async ()
|
|||||||
assert.equal(ready.ready, true);
|
assert.equal(ready.ready, true);
|
||||||
assert.equal(
|
assert.equal(
|
||||||
ready.migrationIds.at(-1),
|
ready.migrationIds.at(-1),
|
||||||
'pg-9018-ai-copilot-failure-diagnosis-admissions',
|
'pg-9019-ai-copilot-failure-diagnosis-tool-unlocks',
|
||||||
);
|
);
|
||||||
|
|
||||||
await assert.rejects(
|
await assert.rejects(
|
||||||
|
|||||||
@@ -125,6 +125,11 @@
|
|||||||
"require": "./dist/trusted-tool/key-management/toolResultProjectedKeyring.js",
|
"require": "./dist/trusted-tool/key-management/toolResultProjectedKeyring.js",
|
||||||
"default": "./dist/trusted-tool/key-management/toolResultProjectedKeyring.js"
|
"default": "./dist/trusted-tool/key-management/toolResultProjectedKeyring.js"
|
||||||
},
|
},
|
||||||
|
"./trusted-tool-invocation-keyring": {
|
||||||
|
"types": "./dist/trusted-tool/key-management/toolInvocationProjectedKeyring.d.ts",
|
||||||
|
"require": "./dist/trusted-tool/key-management/toolInvocationProjectedKeyring.js",
|
||||||
|
"default": "./dist/trusted-tool/key-management/toolInvocationProjectedKeyring.js"
|
||||||
|
},
|
||||||
"./remote-completion": {
|
"./remote-completion": {
|
||||||
"types": "./dist/remote-execution/remoteWorkerCompletionService.d.ts",
|
"types": "./dist/remote-execution/remoteWorkerCompletionService.d.ts",
|
||||||
"require": "./dist/remote-execution/remoteWorkerCompletionService.js",
|
"require": "./dist/remote-execution/remoteWorkerCompletionService.js",
|
||||||
|
|||||||
+178
@@ -0,0 +1,178 @@
|
|||||||
|
import { Buffer } from 'node:buffer';
|
||||||
|
import { createHash } from 'node:crypto';
|
||||||
|
|
||||||
|
export const CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA =
|
||||||
|
'qinglong/cluster-tool-invocation-projected-keyring@v1' as const;
|
||||||
|
export const MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES = 64 * 1024;
|
||||||
|
export const MAX_CLUSTER_TOOL_INVOCATION_PROJECTED_KEYS = 16;
|
||||||
|
|
||||||
|
const KEY_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/;
|
||||||
|
const BASE64URL_PATTERN = /^[A-Za-z0-9_-]+$/;
|
||||||
|
const PROJECTION_DIGEST_DOMAIN = Buffer.from(
|
||||||
|
'qinglong/cluster-tool-invocation-projected-keyring-digest@v1\0',
|
||||||
|
'utf8',
|
||||||
|
);
|
||||||
|
|
||||||
|
export interface ClusterToolInvocationKeyringManifest {
|
||||||
|
readonly schema: typeof CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA;
|
||||||
|
readonly activeKeyId: string;
|
||||||
|
readonly keys: Readonly<Record<string, string>>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ClusterToolInvocationKeyringSummary {
|
||||||
|
readonly schemaVersion: 1;
|
||||||
|
readonly activeKeyId: string;
|
||||||
|
readonly keyIds: readonly string[];
|
||||||
|
readonly projectionDigest: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export class InvalidClusterToolInvocationKeyringManifestError extends TypeError {
|
||||||
|
readonly code = 'QL3_CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_INVALID';
|
||||||
|
|
||||||
|
constructor() {
|
||||||
|
super('Cluster Tool invocation keyring manifest is invalid');
|
||||||
|
this.name = 'InvalidClusterToolInvocationKeyringManifestError';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function invalid(): never {
|
||||||
|
throw new InvalidClusterToolInvocationKeyringManifestError();
|
||||||
|
}
|
||||||
|
|
||||||
|
function dataRecord(value: unknown): Record<string, unknown> {
|
||||||
|
if (
|
||||||
|
!value ||
|
||||||
|
typeof value !== 'object' ||
|
||||||
|
Array.isArray(value) ||
|
||||||
|
Object.getPrototypeOf(value) !== Object.prototype
|
||||||
|
) {
|
||||||
|
return invalid();
|
||||||
|
}
|
||||||
|
return value as Record<string, unknown>;
|
||||||
|
}
|
||||||
|
|
||||||
|
function exactKeys(value: object, expected: readonly string[]): boolean {
|
||||||
|
const actual = Object.keys(value).sort();
|
||||||
|
const canonical = [...expected].sort();
|
||||||
|
return (
|
||||||
|
actual.length === canonical.length &&
|
||||||
|
actual.every((key, index) => key === canonical[index])
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function normalizeClusterToolInvocationKeyringManifest(
|
||||||
|
value: unknown,
|
||||||
|
): Readonly<ClusterToolInvocationKeyringManifest> {
|
||||||
|
const manifest = dataRecord(value);
|
||||||
|
if (
|
||||||
|
!exactKeys(manifest, ['activeKeyId', 'keys', 'schema']) ||
|
||||||
|
manifest.schema !== CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA ||
|
||||||
|
typeof manifest.activeKeyId !== 'string' ||
|
||||||
|
!KEY_ID_PATTERN.test(manifest.activeKeyId)
|
||||||
|
) {
|
||||||
|
return invalid();
|
||||||
|
}
|
||||||
|
const keys = dataRecord(manifest.keys);
|
||||||
|
const entries = Object.entries(keys).sort(([left], [right]) =>
|
||||||
|
left < right ? -1 : left > right ? 1 : 0,
|
||||||
|
);
|
||||||
|
if (
|
||||||
|
entries.length < 1 ||
|
||||||
|
entries.length > MAX_CLUSTER_TOOL_INVOCATION_PROJECTED_KEYS
|
||||||
|
) {
|
||||||
|
return invalid();
|
||||||
|
}
|
||||||
|
const normalized: (readonly [string, string])[] = [];
|
||||||
|
for (const [keyId, encoded] of entries) {
|
||||||
|
let material: Buffer | undefined;
|
||||||
|
try {
|
||||||
|
if (
|
||||||
|
!KEY_ID_PATTERN.test(keyId) ||
|
||||||
|
typeof encoded !== 'string' ||
|
||||||
|
!BASE64URL_PATTERN.test(encoded)
|
||||||
|
) {
|
||||||
|
return invalid();
|
||||||
|
}
|
||||||
|
material = Buffer.from(encoded, 'base64url');
|
||||||
|
if (
|
||||||
|
material.byteLength !== 32 ||
|
||||||
|
material.toString('base64url') !== encoded
|
||||||
|
) {
|
||||||
|
return invalid();
|
||||||
|
}
|
||||||
|
normalized.push(Object.freeze([keyId, encoded] as const));
|
||||||
|
} finally {
|
||||||
|
material?.fill(0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const normalizedKeys = Object.freeze(Object.fromEntries(normalized));
|
||||||
|
if (normalizedKeys[manifest.activeKeyId] === undefined) return invalid();
|
||||||
|
return Object.freeze({
|
||||||
|
schema: CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA,
|
||||||
|
activeKeyId: manifest.activeKeyId,
|
||||||
|
keys: normalizedKeys,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parseClusterToolInvocationKeyringManifest(
|
||||||
|
bytes: Buffer,
|
||||||
|
): Readonly<ClusterToolInvocationKeyringManifest> {
|
||||||
|
try {
|
||||||
|
if (
|
||||||
|
!Buffer.isBuffer(bytes) ||
|
||||||
|
bytes.byteLength < 1 ||
|
||||||
|
bytes.byteLength > MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES
|
||||||
|
) {
|
||||||
|
return invalid();
|
||||||
|
}
|
||||||
|
return normalizeClusterToolInvocationKeyringManifest(
|
||||||
|
JSON.parse(bytes.toString('utf8')),
|
||||||
|
);
|
||||||
|
} catch (error) {
|
||||||
|
if (error instanceof InvalidClusterToolInvocationKeyringManifestError) {
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
return invalid();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function canonicalClusterToolInvocationKeyringManifest(
|
||||||
|
value: ClusterToolInvocationKeyringManifest,
|
||||||
|
): Buffer {
|
||||||
|
const manifest = normalizeClusterToolInvocationKeyringManifest(value);
|
||||||
|
return Buffer.from(`${JSON.stringify(manifest)}\n`, 'utf8');
|
||||||
|
}
|
||||||
|
|
||||||
|
export function resolveClusterToolInvocationKeyringMaterial(
|
||||||
|
value: ClusterToolInvocationKeyringManifest,
|
||||||
|
keyId: string,
|
||||||
|
): Readonly<{ keyId: string; key: Uint8Array }> | null {
|
||||||
|
if (typeof keyId !== 'string' || !KEY_ID_PATTERN.test(keyId)) {
|
||||||
|
return invalid();
|
||||||
|
}
|
||||||
|
const manifest = normalizeClusterToolInvocationKeyringManifest(value);
|
||||||
|
const encoded = manifest.keys[keyId];
|
||||||
|
if (encoded === undefined) return null;
|
||||||
|
const key = Buffer.from(encoded, 'base64url');
|
||||||
|
if (key.byteLength !== 32 || key.toString('base64url') !== encoded) {
|
||||||
|
key.fill(0);
|
||||||
|
return invalid();
|
||||||
|
}
|
||||||
|
return Object.freeze({ keyId, key });
|
||||||
|
}
|
||||||
|
|
||||||
|
export function summarizeClusterToolInvocationKeyringManifest(
|
||||||
|
value: ClusterToolInvocationKeyringManifest,
|
||||||
|
): Readonly<ClusterToolInvocationKeyringSummary> {
|
||||||
|
const manifest = normalizeClusterToolInvocationKeyringManifest(value);
|
||||||
|
const keyIds = Object.freeze(Object.keys(manifest.keys).sort());
|
||||||
|
return Object.freeze({
|
||||||
|
schemaVersion: 1 as const,
|
||||||
|
activeKeyId: manifest.activeKeyId,
|
||||||
|
keyIds,
|
||||||
|
projectionDigest: createHash('sha256')
|
||||||
|
.update(PROJECTION_DIGEST_DOMAIN)
|
||||||
|
.update(JSON.stringify(manifest))
|
||||||
|
.digest('hex'),
|
||||||
|
});
|
||||||
|
}
|
||||||
+151
@@ -0,0 +1,151 @@
|
|||||||
|
import { Buffer } from 'node:buffer';
|
||||||
|
|
||||||
|
import type { ToolInvocationArtifactKeyProvider } from '@qinglong/runtime-core/tool-invocation-artifact';
|
||||||
|
|
||||||
|
import { PrivateProjectedFileReader } from '../../security/privateProjectedFile';
|
||||||
|
import {
|
||||||
|
MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES,
|
||||||
|
canonicalClusterToolInvocationKeyringManifest,
|
||||||
|
parseClusterToolInvocationKeyringManifest,
|
||||||
|
resolveClusterToolInvocationKeyringMaterial,
|
||||||
|
summarizeClusterToolInvocationKeyringManifest,
|
||||||
|
type ClusterToolInvocationKeyringManifest,
|
||||||
|
type ClusterToolInvocationKeyringSummary,
|
||||||
|
} from './toolInvocationKeyringManifest';
|
||||||
|
|
||||||
|
const DATA_FILE_NAME = /^[A-Za-z0-9][A-Za-z0-9._-]{0,252}$/;
|
||||||
|
|
||||||
|
export {
|
||||||
|
CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA,
|
||||||
|
MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES,
|
||||||
|
MAX_CLUSTER_TOOL_INVOCATION_PROJECTED_KEYS,
|
||||||
|
InvalidClusterToolInvocationKeyringManifestError,
|
||||||
|
canonicalClusterToolInvocationKeyringManifest,
|
||||||
|
normalizeClusterToolInvocationKeyringManifest,
|
||||||
|
parseClusterToolInvocationKeyringManifest,
|
||||||
|
resolveClusterToolInvocationKeyringMaterial,
|
||||||
|
summarizeClusterToolInvocationKeyringManifest,
|
||||||
|
type ClusterToolInvocationKeyringManifest,
|
||||||
|
type ClusterToolInvocationKeyringSummary,
|
||||||
|
} from './toolInvocationKeyringManifest';
|
||||||
|
|
||||||
|
export interface ClusterToolInvocationProjectedKeyringOptions {
|
||||||
|
readonly rootDirectory: string;
|
||||||
|
readonly dataFileName?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export class ClusterToolInvocationProjectedKeyringUnavailableError extends Error {
|
||||||
|
readonly code = 'QL3_CLUSTER_TOOL_INVOCATION_PROJECTED_KEYRING_UNAVAILABLE';
|
||||||
|
|
||||||
|
constructor(options?: ErrorOptions) {
|
||||||
|
super('Projected Cluster Tool invocation keyring is unavailable', options);
|
||||||
|
this.name = 'ClusterToolInvocationProjectedKeyringUnavailableError';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function unavailable(
|
||||||
|
cause?: unknown,
|
||||||
|
): ClusterToolInvocationProjectedKeyringUnavailableError {
|
||||||
|
return new ClusterToolInvocationProjectedKeyringUnavailableError({
|
||||||
|
cause: cause instanceof Error ? cause : undefined,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function dataFileName(value: unknown): string {
|
||||||
|
if (typeof value !== 'string' || !DATA_FILE_NAME.test(value)) {
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function readManifest(
|
||||||
|
reader: PrivateProjectedFileReader,
|
||||||
|
fileName: string,
|
||||||
|
): Promise<Readonly<ClusterToolInvocationKeyringManifest>> {
|
||||||
|
let bytes: Buffer | undefined;
|
||||||
|
let canonical: Buffer | undefined;
|
||||||
|
try {
|
||||||
|
bytes = await reader.read(fileName);
|
||||||
|
const manifest = parseClusterToolInvocationKeyringManifest(bytes);
|
||||||
|
canonical = canonicalClusterToolInvocationKeyringManifest(manifest);
|
||||||
|
if (!canonical.equals(bytes)) throw unavailable();
|
||||||
|
return manifest;
|
||||||
|
} catch (cause) {
|
||||||
|
throw cause instanceof ClusterToolInvocationProjectedKeyringUnavailableError
|
||||||
|
? cause
|
||||||
|
: unavailable(cause);
|
||||||
|
} finally {
|
||||||
|
bytes?.fill(0);
|
||||||
|
canonical?.fill(0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Read-only, no-cache invocation Artifact key authority for projections. */
|
||||||
|
export class ClusterToolInvocationProjectedKeyring
|
||||||
|
implements ToolInvocationArtifactKeyProvider
|
||||||
|
{
|
||||||
|
readonly #reader: PrivateProjectedFileReader;
|
||||||
|
readonly #dataFileName: string;
|
||||||
|
|
||||||
|
constructor(options: ClusterToolInvocationProjectedKeyringOptions) {
|
||||||
|
if (!options || typeof options !== 'object' || Array.isArray(options)) {
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
this.#reader = new PrivateProjectedFileReader({
|
||||||
|
rootDirectory: options.rootDirectory,
|
||||||
|
minimumBytes: 1,
|
||||||
|
maximumBytes: MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES,
|
||||||
|
access: 'read_only_keyring',
|
||||||
|
});
|
||||||
|
this.#dataFileName = dataFileName(options.dataFileName ?? 'keyring.json');
|
||||||
|
} catch (cause) {
|
||||||
|
throw unavailable(cause);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async verify(): Promise<Readonly<ClusterToolInvocationKeyringSummary>> {
|
||||||
|
return summarizeClusterToolInvocationKeyringManifest(
|
||||||
|
await readManifest(this.#reader, this.#dataFileName),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async active(): ReturnType<ToolInvocationArtifactKeyProvider['active']> {
|
||||||
|
try {
|
||||||
|
const manifest = await readManifest(this.#reader, this.#dataFileName);
|
||||||
|
return resolveClusterToolInvocationKeyringMaterial(
|
||||||
|
manifest,
|
||||||
|
manifest.activeKeyId,
|
||||||
|
)!;
|
||||||
|
} catch (cause) {
|
||||||
|
throw cause instanceof
|
||||||
|
ClusterToolInvocationProjectedKeyringUnavailableError
|
||||||
|
? cause
|
||||||
|
: unavailable(cause);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async resolve(
|
||||||
|
keyId: string,
|
||||||
|
): ReturnType<ToolInvocationArtifactKeyProvider['resolve']> {
|
||||||
|
try {
|
||||||
|
return resolveClusterToolInvocationKeyringMaterial(
|
||||||
|
await readManifest(this.#reader, this.#dataFileName),
|
||||||
|
keyId,
|
||||||
|
);
|
||||||
|
} catch (cause) {
|
||||||
|
throw cause instanceof
|
||||||
|
ClusterToolInvocationProjectedKeyringUnavailableError
|
||||||
|
? cause
|
||||||
|
: unavailable(cause);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function createClusterToolInvocationProjectedKeyring(
|
||||||
|
options: ClusterToolInvocationProjectedKeyringOptions,
|
||||||
|
): Promise<Readonly<ClusterToolInvocationProjectedKeyring>> {
|
||||||
|
const provider = new ClusterToolInvocationProjectedKeyring(options);
|
||||||
|
await provider.verify();
|
||||||
|
return provider;
|
||||||
|
}
|
||||||
@@ -0,0 +1,123 @@
|
|||||||
|
const assert = require('node:assert/strict');
|
||||||
|
const fs = require('node:fs/promises');
|
||||||
|
const os = require('node:os');
|
||||||
|
const path = require('node:path');
|
||||||
|
const { afterEach, test } = require('node:test');
|
||||||
|
|
||||||
|
const {
|
||||||
|
CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA,
|
||||||
|
ClusterToolInvocationProjectedKeyring,
|
||||||
|
ClusterToolInvocationProjectedKeyringUnavailableError,
|
||||||
|
canonicalClusterToolInvocationKeyringManifest,
|
||||||
|
createClusterToolInvocationProjectedKeyring,
|
||||||
|
} = require('@qinglong/cluster-control/trusted-tool-invocation-keyring');
|
||||||
|
|
||||||
|
const roots = [];
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
await Promise.all(
|
||||||
|
roots
|
||||||
|
.splice(0)
|
||||||
|
.map((root) => fs.rm(root, { recursive: true, force: true })),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
async function tempRoot() {
|
||||||
|
const root = await fs.mkdtemp(
|
||||||
|
path.join(os.tmpdir(), 'ql3-tool-invocation-keyring-'),
|
||||||
|
);
|
||||||
|
roots.push(root);
|
||||||
|
return root;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function publish(root, generationName, manifest, mode = 0o440) {
|
||||||
|
const generation = path.join(root, generationName);
|
||||||
|
await fs.mkdir(generation, { mode: 0o750 });
|
||||||
|
const target = path.join(generation, 'keyring.json');
|
||||||
|
await fs.writeFile(
|
||||||
|
target,
|
||||||
|
canonicalClusterToolInvocationKeyringManifest(manifest),
|
||||||
|
{ mode },
|
||||||
|
);
|
||||||
|
await fs.chmod(target, mode);
|
||||||
|
const next = path.join(root, '..data-next');
|
||||||
|
await fs.symlink(generationName, next);
|
||||||
|
await fs.rename(next, path.join(root, '..data'));
|
||||||
|
try {
|
||||||
|
await fs.symlink('..data/keyring.json', path.join(root, 'keyring.json'));
|
||||||
|
} catch (error) {
|
||||||
|
if (error.code !== 'EEXIST') throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function manifest(activeKeyId, keys) {
|
||||||
|
return Object.freeze({
|
||||||
|
schema: CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA,
|
||||||
|
activeKeyId,
|
||||||
|
keys: Object.freeze(keys),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
test('projected Tool invocation keyring rotates active material without caching', async () => {
|
||||||
|
const root = await tempRoot();
|
||||||
|
const keyOne = Buffer.alloc(32, 0x41);
|
||||||
|
const keyTwo = Buffer.alloc(32, 0x42);
|
||||||
|
await publish(
|
||||||
|
root,
|
||||||
|
'..2026_08_15_01',
|
||||||
|
manifest('invocation-key-one', {
|
||||||
|
'invocation-key-one': keyOne.toString('base64url'),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
const provider = await createClusterToolInvocationProjectedKeyring({
|
||||||
|
rootDirectory: root,
|
||||||
|
});
|
||||||
|
const first = await provider.active();
|
||||||
|
assert.equal(first.keyId, 'invocation-key-one');
|
||||||
|
assert.deepEqual(Buffer.from(first.key), keyOne);
|
||||||
|
first.key.fill(0);
|
||||||
|
|
||||||
|
await publish(
|
||||||
|
root,
|
||||||
|
'..2026_08_15_02',
|
||||||
|
manifest('invocation-key-two', {
|
||||||
|
'invocation-key-one': keyOne.toString('base64url'),
|
||||||
|
'invocation-key-two': keyTwo.toString('base64url'),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
const second = await provider.active();
|
||||||
|
assert.equal(second.keyId, 'invocation-key-two');
|
||||||
|
assert.deepEqual(Buffer.from(second.key), keyTwo);
|
||||||
|
second.key.fill(0);
|
||||||
|
const historical = await provider.resolve('invocation-key-one');
|
||||||
|
assert.ok(historical);
|
||||||
|
assert.deepEqual(Buffer.from(historical.key), keyOne);
|
||||||
|
historical.key.fill(0);
|
||||||
|
assert.equal(await provider.resolve('missing-key'), null);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('projected Tool invocation keyring rejects missing active and writable material', async () => {
|
||||||
|
const root = await tempRoot();
|
||||||
|
const key = Buffer.alloc(32, 0x51).toString('base64url');
|
||||||
|
assert.throws(
|
||||||
|
() =>
|
||||||
|
canonicalClusterToolInvocationKeyringManifest(
|
||||||
|
manifest('missing-key', { 'invocation-key-one': key }),
|
||||||
|
),
|
||||||
|
TypeError,
|
||||||
|
);
|
||||||
|
|
||||||
|
await publish(
|
||||||
|
root,
|
||||||
|
'..2026_08_15_01',
|
||||||
|
manifest('invocation-key-one', { 'invocation-key-one': key }),
|
||||||
|
0o640,
|
||||||
|
);
|
||||||
|
await assert.rejects(
|
||||||
|
() =>
|
||||||
|
new ClusterToolInvocationProjectedKeyring({
|
||||||
|
rootDirectory: root,
|
||||||
|
}).verify(),
|
||||||
|
ClusterToolInvocationProjectedKeyringUnavailableError,
|
||||||
|
);
|
||||||
|
});
|
||||||
@@ -26,7 +26,13 @@ const {
|
|||||||
PostgresRemoteWorkerCompletionRepository,
|
PostgresRemoteWorkerCompletionRepository,
|
||||||
PostgresRemoteWorkerLeaseControlRepository,
|
PostgresRemoteWorkerLeaseControlRepository,
|
||||||
PostgresRunAttemptLogRetentionClaimRepository,
|
PostgresRunAttemptLogRetentionClaimRepository,
|
||||||
|
PostgresRunRepository,
|
||||||
|
PostgresToolExecutionCompletionRepository,
|
||||||
|
PostgresToolExecutionFailureCompletionRepository,
|
||||||
|
PostgresToolExecutionStartBarrierRepository,
|
||||||
PostgresToolInvocationArtifactRepository,
|
PostgresToolInvocationArtifactRepository,
|
||||||
|
PostgresToolResultKeyCatalogReader,
|
||||||
|
PostgresToolResultRekeyReader,
|
||||||
PostgresWorkerSessionRepository,
|
PostgresWorkerSessionRepository,
|
||||||
} = require('../packages/ql3-cluster-postgres/dist/entrypoints/runtime.js');
|
} = require('../packages/ql3-cluster-postgres/dist/entrypoints/runtime.js');
|
||||||
const {
|
const {
|
||||||
@@ -34,6 +40,7 @@ const {
|
|||||||
} = require('../packages/ql3-cluster-postgres/dist/migration/migration.js');
|
} = require('../packages/ql3-cluster-postgres/dist/migration/migration.js');
|
||||||
const {
|
const {
|
||||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
||||||
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||||
POSTGRES_MODEL_INVOCATION_MIGRATION_ID,
|
POSTGRES_MODEL_INVOCATION_MIGRATION_ID,
|
||||||
POSTGRES_MODEL_INVOCATION_PRICING_MIGRATION_ID,
|
POSTGRES_MODEL_INVOCATION_PRICING_MIGRATION_ID,
|
||||||
POSTGRES_MODEL_INVOCATION_QUOTA_MIGRATION_ID,
|
POSTGRES_MODEL_INVOCATION_QUOTA_MIGRATION_ID,
|
||||||
@@ -59,6 +66,12 @@ const {
|
|||||||
const {
|
const {
|
||||||
PostgresCopilotFailureDiagnosisAdmissionRepository,
|
PostgresCopilotFailureDiagnosisAdmissionRepository,
|
||||||
} = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/postgresAdmissionRepository.js');
|
} = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/postgresAdmissionRepository.js');
|
||||||
|
const {
|
||||||
|
executeCopilotFailureDiagnosisTool,
|
||||||
|
} = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/toolExecution.js');
|
||||||
|
const {
|
||||||
|
PostgresCopilotFailureDiagnosisToolUnlockRepository,
|
||||||
|
} = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/postgresToolExecutionRepository.js');
|
||||||
const {
|
const {
|
||||||
BUILTIN_RUN_LOG_EXCERPT_TOOL,
|
BUILTIN_RUN_LOG_EXCERPT_TOOL,
|
||||||
BUILTIN_RUN_LOG_EXCERPT_TOOL_DEFINITION,
|
BUILTIN_RUN_LOG_EXCERPT_TOOL_DEFINITION,
|
||||||
@@ -1472,6 +1485,25 @@ async function modelInvocationFeatureFacts(pool) {
|
|||||||
'SELECT,INSERT,UPDATE,DELETE'
|
'SELECT,INSERT,UPDATE,DELETE'
|
||||||
)
|
)
|
||||||
) AS "copilotDiagnosisAdmissionRuntimeOnly",
|
) AS "copilotDiagnosisAdmissionRuntimeOnly",
|
||||||
|
has_table_privilege(
|
||||||
|
'ql3_runtime',
|
||||||
|
'ql3_ai.copilot_failure_diagnosis_tool_unlocks', 'SELECT,INSERT'
|
||||||
|
) AND NOT has_table_privilege(
|
||||||
|
'ql3_runtime',
|
||||||
|
'ql3_ai.copilot_failure_diagnosis_tool_unlocks', 'UPDATE,DELETE'
|
||||||
|
) AND NOT EXISTS (
|
||||||
|
SELECT 1
|
||||||
|
FROM unnest(ARRAY[
|
||||||
|
'ql3_admin', 'ql3_package_manager', 'ql3_package_executor',
|
||||||
|
'ql3_worker_ingress', 'ql3_ai_maintenance',
|
||||||
|
'ql3_ai_credential_manager', 'ql3_ai_credential_tester'
|
||||||
|
]::text[]) AS denied(role_name)
|
||||||
|
WHERE has_table_privilege(
|
||||||
|
denied.role_name,
|
||||||
|
'ql3_ai.copilot_failure_diagnosis_tool_unlocks',
|
||||||
|
'SELECT,INSERT,UPDATE,DELETE'
|
||||||
|
)
|
||||||
|
) AS "copilotDiagnosisToolUnlockRuntimeOnly",
|
||||||
has_function_privilege(
|
has_function_privilege(
|
||||||
'ql3_runtime',
|
'ql3_runtime',
|
||||||
'ql3_ai.copilot_failure_diagnosis_admission_source_snapshot(varchar,varchar,varchar,integer,integer,varchar,varchar)',
|
'ql3_ai.copilot_failure_diagnosis_admission_source_snapshot(varchar,varchar,varchar,integer,integer,varchar,varchar)',
|
||||||
@@ -1598,6 +1630,24 @@ async function copilotFailureDiagnosisAdmissionFacts(pool, requestId) {
|
|||||||
WHERE run_id = admission.run_id) AS "eventCount",
|
WHERE run_id = admission.run_id) AS "eventCount",
|
||||||
(SELECT count(*)::integer FROM "ql3"."step_run_mutations"
|
(SELECT count(*)::integer FROM "ql3"."step_run_mutations"
|
||||||
WHERE run_id = admission.run_id) AS "mutationCount",
|
WHERE run_id = admission.run_id) AS "mutationCount",
|
||||||
|
(SELECT count(*)::integer
|
||||||
|
FROM "ql3"."tool_execution_start_barriers"
|
||||||
|
WHERE run_id = admission.run_id
|
||||||
|
AND step_run_id = admission.tool_step_run_id)
|
||||||
|
AS "startCount",
|
||||||
|
(SELECT count(*)::integer
|
||||||
|
FROM "ql3"."tool_execution_completions"
|
||||||
|
WHERE run_id = admission.run_id
|
||||||
|
AND step_run_id = admission.tool_step_run_id)
|
||||||
|
AS "completionCount",
|
||||||
|
(SELECT count(*)::integer
|
||||||
|
FROM "ql3_ai"."copilot_failure_diagnosis_tool_unlocks"
|
||||||
|
WHERE request_id = admission.request_id) AS "unlockCount",
|
||||||
|
(SELECT completion.artifact_json::text
|
||||||
|
FROM "ql3"."tool_execution_completions" AS completion
|
||||||
|
WHERE completion.run_id = admission.run_id
|
||||||
|
AND completion.step_run_id = admission.tool_step_run_id
|
||||||
|
LIMIT 1) AS "resultArtifactJson",
|
||||||
tool.status AS "toolStatus", tool.kind AS "toolKind",
|
tool.status AS "toolStatus", tool.kind AS "toolKind",
|
||||||
model.status AS "modelStatus", model.kind AS "modelKind",
|
model.status AS "modelStatus", model.kind AS "modelKind",
|
||||||
model.parent_step_run_id AS "modelParentStepRunId",
|
model.parent_step_run_id AS "modelParentStepRunId",
|
||||||
@@ -1615,11 +1665,20 @@ async function copilotFailureDiagnosisAdmissionFacts(pool, requestId) {
|
|||||||
[requestId],
|
[requestId],
|
||||||
);
|
);
|
||||||
assert.equal(result.rowCount, 1);
|
assert.equal(result.rowCount, 1);
|
||||||
const { planJson, receiptJson, ...facts } = result.rows[0];
|
const { planJson, receiptJson, resultArtifactJson, ...facts } =
|
||||||
|
result.rows[0];
|
||||||
assert.equal(typeof planJson, 'string');
|
assert.equal(typeof planJson, 'string');
|
||||||
assert.equal(typeof receiptJson, 'string');
|
assert.equal(typeof receiptJson, 'string');
|
||||||
return {
|
return {
|
||||||
...facts,
|
...facts,
|
||||||
|
resultArtifactPresent: typeof resultArtifactJson === 'string',
|
||||||
|
resultArtifactJsonDigest:
|
||||||
|
typeof resultArtifactJson === 'string'
|
||||||
|
? createHash('sha256').update(resultArtifactJson).digest('hex')
|
||||||
|
: null,
|
||||||
|
resultArtifactContainsPlaintext:
|
||||||
|
typeof resultArtifactJson === 'string' &&
|
||||||
|
resultArtifactJson.includes('diagnosis-sensitive-line'),
|
||||||
planJsonDigest: createHash('sha256').update(planJson).digest('hex'),
|
planJsonDigest: createHash('sha256').update(planJson).digest('hex'),
|
||||||
receiptJsonDigest: createHash('sha256').update(receiptJson).digest('hex'),
|
receiptJsonDigest: createHash('sha256').update(receiptJson).digest('hex'),
|
||||||
};
|
};
|
||||||
@@ -1717,7 +1776,7 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) {
|
|||||||
subject: { type: 'user', id: subjectId },
|
subject: { type: 'user', id: subjectId },
|
||||||
authenticationId: `diagnosis-auth-${suffix}`,
|
authenticationId: `diagnosis-auth-${suffix}`,
|
||||||
authenticatedAtMs: baseTimeMs - 1_000,
|
authenticatedAtMs: baseTimeMs - 1_000,
|
||||||
expiresAtMs: baseTimeMs + 60_000,
|
expiresAtMs: baseTimeMs + 600_000,
|
||||||
assurance: 'multi_factor',
|
assurance: 'multi_factor',
|
||||||
});
|
});
|
||||||
const invocation = await prepareToolInvocation(
|
const invocation = await prepareToolInvocation(
|
||||||
@@ -1742,7 +1801,7 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) {
|
|||||||
const bindings = new TrustedToolHandlerBindingRegistry(snapshot, [
|
const bindings = new TrustedToolHandlerBindingRegistry(snapshot, [
|
||||||
createBuiltInRunLogExcerptToolHandlerBinding(snapshot, ['cluster-control']),
|
createBuiltInRunLogExcerptToolHandlerBinding(snapshot, ['cluster-control']),
|
||||||
]);
|
]);
|
||||||
const toolPlan = createTrustedToolInvocationPlan(bindings, invocation, {
|
const toolBundle = createTrustedToolInvocationPlan(bindings, invocation, {
|
||||||
actionRef: `diagnosis-log-${suffix}`,
|
actionRef: `diagnosis-log-${suffix}`,
|
||||||
inputArtifactId: `diagnosis-input-${suffix}`,
|
inputArtifactId: `diagnosis-input-${suffix}`,
|
||||||
previewArtifactId: `diagnosis-preview-${suffix}`,
|
previewArtifactId: `diagnosis-preview-${suffix}`,
|
||||||
@@ -1760,7 +1819,8 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) {
|
|||||||
warnings: ['potentially_sensitive_output'],
|
warnings: ['potentially_sensitive_output'],
|
||||||
},
|
},
|
||||||
sealedAtMs: baseTimeMs + 200,
|
sealedAtMs: baseTimeMs + 200,
|
||||||
}).plan;
|
});
|
||||||
|
const toolPlan = toolBundle.plan;
|
||||||
const plan = prepareCopilotFailureDiagnosisExecution({
|
const plan = prepareCopilotFailureDiagnosisExecution({
|
||||||
requestId,
|
requestId,
|
||||||
traceId: `diagnosis-trace-${suffix}`,
|
traceId: `diagnosis-trace-${suffix}`,
|
||||||
@@ -1793,6 +1853,12 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) {
|
|||||||
const repository = new PostgresCopilotFailureDiagnosisAdmissionRepository(
|
const repository = new PostgresCopilotFailureDiagnosisAdmissionRepository(
|
||||||
runtimeDatabase.pool,
|
runtimeDatabase.pool,
|
||||||
);
|
);
|
||||||
|
assert.deepEqual(
|
||||||
|
await new PostgresToolInvocationArtifactRepository(
|
||||||
|
runtimeDatabase.pool,
|
||||||
|
).put(toolBundle.inputArtifact, toolBundle.previewArtifact),
|
||||||
|
{ status: 'inserted' },
|
||||||
|
);
|
||||||
const first = await repository.admit(plan);
|
const first = await repository.admit(plan);
|
||||||
const replay = await repository.admit(plan);
|
const replay = await repository.admit(plan);
|
||||||
assert.equal(first.status, 'created');
|
assert.equal(first.status, 'created');
|
||||||
@@ -1836,20 +1902,221 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) {
|
|||||||
},
|
},
|
||||||
);
|
);
|
||||||
return {
|
return {
|
||||||
requestId,
|
report: {
|
||||||
sourceRunId,
|
requestId,
|
||||||
sourceAttemptId,
|
sourceRunId,
|
||||||
runId: plan.runId,
|
sourceAttemptId,
|
||||||
toolStepRunId: plan.toolStepRunId,
|
runId: plan.runId,
|
||||||
modelStepRunId: plan.modelStepRunId,
|
toolStepRunId: plan.toolStepRunId,
|
||||||
exactReplay: true,
|
modelStepRunId: plan.modelStepRunId,
|
||||||
beforePromotion: facts,
|
exactReplay: true,
|
||||||
|
toolExecutionExactReplay: false,
|
||||||
|
encryptedCompletion: false,
|
||||||
|
beforePromotion: facts,
|
||||||
|
},
|
||||||
|
fixture: {
|
||||||
|
baseTimeMs,
|
||||||
|
plan,
|
||||||
|
principal,
|
||||||
|
snapshot,
|
||||||
|
invocationKeyId: toolBundle.inputArtifact.keyId,
|
||||||
|
invocationKey: Buffer.alloc(32, 0x21),
|
||||||
|
},
|
||||||
};
|
};
|
||||||
} finally {
|
} finally {
|
||||||
await runtimeDatabase.close();
|
await runtimeDatabase.close();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function executeCopilotFailureDiagnosisToolHaEvidence(options) {
|
||||||
|
const {
|
||||||
|
port,
|
||||||
|
report,
|
||||||
|
fixture,
|
||||||
|
resultKeyFixture,
|
||||||
|
expectedExisting = false,
|
||||||
|
} = options;
|
||||||
|
const runtimeDatabase = await databaseOpener(
|
||||||
|
'runtime',
|
||||||
|
databaseUrl(RUNTIME_USER, RUNTIME_PASSWORD, port),
|
||||||
|
expectedExisting
|
||||||
|
? 'ql3-ha-copilot-diagnosis-tool-promoted'
|
||||||
|
: 'ql3-ha-copilot-diagnosis-tool-primary',
|
||||||
|
)();
|
||||||
|
let logReads = 0;
|
||||||
|
const logContent = Buffer.from(
|
||||||
|
'diagnosis-sensitive-line password=must-be-redacted',
|
||||||
|
'utf8',
|
||||||
|
);
|
||||||
|
const keys = {
|
||||||
|
async resolve(keyId) {
|
||||||
|
if (keyId !== fixture.invocationKeyId) return null;
|
||||||
|
return { keyId, key: Buffer.from(fixture.invocationKey) };
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const resultKeys = {
|
||||||
|
async resolve(keyId) {
|
||||||
|
if (keyId !== resultKeyFixture.resultKeyId) return null;
|
||||||
|
return {
|
||||||
|
keyId,
|
||||||
|
key: Buffer.alloc(32, resultKeyFixture.resultKeyByte),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const dependencies = {
|
||||||
|
admissions: new PostgresCopilotFailureDiagnosisAdmissionRepository(
|
||||||
|
runtimeDatabase.pool,
|
||||||
|
),
|
||||||
|
snapshots: {
|
||||||
|
async findCurrent(projectId) {
|
||||||
|
return projectId === fixture.plan.projectId
|
||||||
|
? { snapshot: fixture.snapshot }
|
||||||
|
: null;
|
||||||
|
},
|
||||||
|
},
|
||||||
|
artifacts: new PostgresToolInvocationArtifactRepository(
|
||||||
|
runtimeDatabase.pool,
|
||||||
|
),
|
||||||
|
invocationKeys: keys,
|
||||||
|
resultKeys,
|
||||||
|
stepRuns: new PostgresStepRunRepository(runtimeDatabase.pool),
|
||||||
|
runs: new PostgresRunRepository(runtimeDatabase.pool),
|
||||||
|
barriers: new PostgresToolExecutionStartBarrierRepository(
|
||||||
|
runtimeDatabase.pool,
|
||||||
|
),
|
||||||
|
completions: new PostgresToolExecutionCompletionRepository(
|
||||||
|
runtimeDatabase.pool,
|
||||||
|
),
|
||||||
|
failureCompletions: new PostgresToolExecutionFailureCompletionRepository(
|
||||||
|
runtimeDatabase.pool,
|
||||||
|
),
|
||||||
|
resultKeyCatalog: new PostgresToolResultKeyCatalogReader(
|
||||||
|
runtimeDatabase.pool,
|
||||||
|
),
|
||||||
|
resultRekeys: new PostgresToolResultRekeyReader(runtimeDatabase.pool),
|
||||||
|
logs: {
|
||||||
|
async read(request) {
|
||||||
|
logReads += 1;
|
||||||
|
assert.equal(request.projectId, fixture.plan.projectId);
|
||||||
|
assert.equal(request.runId, report.sourceRunId);
|
||||||
|
assert.equal(request.attemptId, report.sourceAttemptId);
|
||||||
|
const totalBytes = logContent.byteLength;
|
||||||
|
const start = Math.min(request.range.offset, totalBytes);
|
||||||
|
const endExclusive = Math.min(start + request.range.length, totalBytes);
|
||||||
|
return {
|
||||||
|
status: 'available',
|
||||||
|
projectId: request.projectId,
|
||||||
|
runId: request.runId,
|
||||||
|
attemptId: request.attemptId,
|
||||||
|
logArtifactId: fixture.plan.source.logArtifactId,
|
||||||
|
content: logContent.subarray(start, endExclusive),
|
||||||
|
start,
|
||||||
|
endExclusive,
|
||||||
|
totalBytes,
|
||||||
|
...(endExclusive < totalBytes ? { nextOffset: endExclusive } : {}),
|
||||||
|
truncation: { truncated: false, maximumBytes: 4_194_304 },
|
||||||
|
};
|
||||||
|
},
|
||||||
|
},
|
||||||
|
unlocks: new PostgresCopilotFailureDiagnosisToolUnlockRepository(
|
||||||
|
runtimeDatabase.pool,
|
||||||
|
),
|
||||||
|
now: () => fixture.baseTimeMs + 400,
|
||||||
|
nonceFactory: () => Buffer.alloc(12, 0x61),
|
||||||
|
};
|
||||||
|
const authorizer = {
|
||||||
|
async authorize() {
|
||||||
|
return {
|
||||||
|
effect: 'allow',
|
||||||
|
reasons: ['role_grant'],
|
||||||
|
fence: { projectVersion: 1, bindingVersion: 1 },
|
||||||
|
};
|
||||||
|
},
|
||||||
|
};
|
||||||
|
try {
|
||||||
|
const first = await executeCopilotFailureDiagnosisTool(
|
||||||
|
{
|
||||||
|
requestId: report.requestId,
|
||||||
|
principal: fixture.principal,
|
||||||
|
authorizer,
|
||||||
|
},
|
||||||
|
dependencies,
|
||||||
|
);
|
||||||
|
assert.equal(first.outcome, 'succeeded');
|
||||||
|
assert.equal(
|
||||||
|
first.completionStatus,
|
||||||
|
expectedExisting ? 'existing' : 'created',
|
||||||
|
);
|
||||||
|
assert.equal(first.unlockStatus, expectedExisting ? 'existing' : 'created');
|
||||||
|
if (!expectedExisting) {
|
||||||
|
const replay = await executeCopilotFailureDiagnosisTool(
|
||||||
|
{
|
||||||
|
requestId: report.requestId,
|
||||||
|
principal: fixture.principal,
|
||||||
|
authorizer,
|
||||||
|
},
|
||||||
|
dependencies,
|
||||||
|
);
|
||||||
|
assert.equal(replay.outcome, 'succeeded');
|
||||||
|
assert.equal(replay.completionStatus, 'existing');
|
||||||
|
assert.equal(replay.unlockStatus, 'existing');
|
||||||
|
assert.deepEqual(replay.completion, first.completion);
|
||||||
|
assert.deepEqual(replay.unlock, first.unlock);
|
||||||
|
assert.equal(logReads, 2);
|
||||||
|
} else {
|
||||||
|
assert.equal(logReads, 0);
|
||||||
|
}
|
||||||
|
const facts = await copilotFailureDiagnosisAdmissionFacts(
|
||||||
|
runtimeDatabase.pool,
|
||||||
|
report.requestId,
|
||||||
|
);
|
||||||
|
assert.deepEqual(
|
||||||
|
{
|
||||||
|
status: facts.status,
|
||||||
|
version: facts.version,
|
||||||
|
eventSequence: facts.eventSequence,
|
||||||
|
eventCount: facts.eventCount,
|
||||||
|
mutationCount: facts.mutationCount,
|
||||||
|
startCount: facts.startCount,
|
||||||
|
completionCount: facts.completionCount,
|
||||||
|
unlockCount: facts.unlockCount,
|
||||||
|
toolStatus: facts.toolStatus,
|
||||||
|
modelStatus: facts.modelStatus,
|
||||||
|
resultArtifactPresent: facts.resultArtifactPresent,
|
||||||
|
resultArtifactContainsPlaintext: facts.resultArtifactContainsPlaintext,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
status: 'running',
|
||||||
|
version: 6,
|
||||||
|
eventSequence: 6,
|
||||||
|
eventCount: 6,
|
||||||
|
mutationCount: 5,
|
||||||
|
startCount: 1,
|
||||||
|
completionCount: 1,
|
||||||
|
unlockCount: 1,
|
||||||
|
toolStatus: 'succeeded',
|
||||||
|
modelStatus: 'ready',
|
||||||
|
resultArtifactPresent: true,
|
||||||
|
resultArtifactContainsPlaintext: false,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
if (!expectedExisting) {
|
||||||
|
report.beforePromotion = facts;
|
||||||
|
report.toolExecutionExactReplay = true;
|
||||||
|
report.encryptedCompletion = true;
|
||||||
|
report.primaryToolAdapterReads = logReads;
|
||||||
|
} else {
|
||||||
|
report.promotedToolReplayReads = logReads;
|
||||||
|
report.toolExecutionSurvivedPromotion = true;
|
||||||
|
}
|
||||||
|
return facts;
|
||||||
|
} finally {
|
||||||
|
fixture.invocationKey.fill(0);
|
||||||
|
logContent.fill(0);
|
||||||
|
await runtimeDatabase.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function runModelProviderCredentialCatalogMatrix(options) {
|
async function runModelProviderCredentialCatalogMatrix(options) {
|
||||||
const { primaryPort, migrationPool } = options;
|
const { primaryPort, migrationPool } = options;
|
||||||
const suffix = `${process.pid}-${randomBytes(3).toString('hex')}`;
|
const suffix = `${process.pid}-${randomBytes(3).toString('hex')}`;
|
||||||
@@ -10300,6 +10567,7 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
let oldPrimaryRejoin;
|
let oldPrimaryRejoin;
|
||||||
let modelInvocationFeaturePromotion;
|
let modelInvocationFeaturePromotion;
|
||||||
let copilotFailureDiagnosisAdmission;
|
let copilotFailureDiagnosisAdmission;
|
||||||
|
let copilotFailureDiagnosisToolFixture;
|
||||||
let modelProviderCredentialCatalog;
|
let modelProviderCredentialCatalog;
|
||||||
let modelProviderCredentialTestConnection;
|
let modelProviderCredentialTestConnection;
|
||||||
let runAttemptLogRetentionEvidence;
|
let runAttemptLogRetentionEvidence;
|
||||||
@@ -10398,6 +10666,7 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
);
|
);
|
||||||
assert.deepEqual(beforePromotion.tables, [
|
assert.deepEqual(beforePromotion.tables, [
|
||||||
'copilot_failure_diagnosis_admissions',
|
'copilot_failure_diagnosis_admissions',
|
||||||
|
'copilot_failure_diagnosis_tool_unlocks',
|
||||||
'model_invocation_completions',
|
'model_invocation_completions',
|
||||||
'model_invocation_price_quotes',
|
'model_invocation_price_quotes',
|
||||||
'model_invocation_price_settlements',
|
'model_invocation_price_settlements',
|
||||||
@@ -10448,6 +10717,7 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
POSTGRES_PLUGIN_PACKAGE_PROMPT_OUTPUT_KEY_ROTATION_MIGRATION_ID,
|
POSTGRES_PLUGIN_PACKAGE_PROMPT_OUTPUT_KEY_ROTATION_MIGRATION_ID,
|
||||||
POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID,
|
POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID,
|
||||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
||||||
|
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||||
],
|
],
|
||||||
);
|
);
|
||||||
assert.deepEqual(beforePromotion.privileges, {
|
assert.deepEqual(beforePromotion.privileges, {
|
||||||
@@ -10472,16 +10742,19 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
promptOutputKeyRotationAuthoritySplit: true,
|
promptOutputKeyRotationAuthoritySplit: true,
|
||||||
promptSnapshotRuntimeOnly: true,
|
promptSnapshotRuntimeOnly: true,
|
||||||
copilotDiagnosisAdmissionRuntimeOnly: true,
|
copilotDiagnosisAdmissionRuntimeOnly: true,
|
||||||
|
copilotDiagnosisToolUnlockRuntimeOnly: true,
|
||||||
copilotDiagnosisSnapshotRuntimeOnly: true,
|
copilotDiagnosisSnapshotRuntimeOnly: true,
|
||||||
migrationHistoryRuntimeReadOnly: true,
|
migrationHistoryRuntimeReadOnly: true,
|
||||||
modelProviderCredentialManagementAuthoritySplit: true,
|
modelProviderCredentialManagementAuthoritySplit: true,
|
||||||
});
|
});
|
||||||
modelInvocationFeaturePromotion = { beforePromotion };
|
modelInvocationFeaturePromotion = { beforePromotion };
|
||||||
copilotFailureDiagnosisAdmission =
|
const copilotDiagnosis =
|
||||||
await runCopilotFailureDiagnosisAdmissionHaEvidence({
|
await runCopilotFailureDiagnosisAdmissionHaEvidence({
|
||||||
primaryPort,
|
primaryPort,
|
||||||
migrationPool: migrationDatabase.pool,
|
migrationPool: migrationDatabase.pool,
|
||||||
});
|
});
|
||||||
|
copilotFailureDiagnosisAdmission = copilotDiagnosis.report;
|
||||||
|
copilotFailureDiagnosisToolFixture = copilotDiagnosis.fixture;
|
||||||
} finally {
|
} finally {
|
||||||
await migrationDatabase.close();
|
await migrationDatabase.close();
|
||||||
}
|
}
|
||||||
@@ -10866,6 +11139,26 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
toolResultRekeyFaultPool.pool.end(),
|
toolResultRekeyFaultPool.pool.end(),
|
||||||
]);
|
]);
|
||||||
}
|
}
|
||||||
|
await executeCopilotFailureDiagnosisToolHaEvidence({
|
||||||
|
port: primaryPort,
|
||||||
|
report: copilotFailureDiagnosisAdmission,
|
||||||
|
fixture: copilotFailureDiagnosisToolFixture,
|
||||||
|
resultKeyFixture: toolResultKeyFixture,
|
||||||
|
});
|
||||||
|
await waitFor(async () => {
|
||||||
|
const replicated = await copilotFailureDiagnosisAdmissionFacts(
|
||||||
|
standbyDatabase.pool,
|
||||||
|
copilotFailureDiagnosisAdmission.requestId,
|
||||||
|
);
|
||||||
|
return JSON.stringify(replicated) ===
|
||||||
|
JSON.stringify(copilotFailureDiagnosisAdmission.beforePromotion)
|
||||||
|
? replicated
|
||||||
|
: null;
|
||||||
|
}, 'Copilot failure diagnosis Tool completion WAL replay');
|
||||||
|
timeline.push({
|
||||||
|
state: 'copilot_failure_diagnosis_tool_completed',
|
||||||
|
atMs: Number((performance.now() - startedAt).toFixed(3)),
|
||||||
|
});
|
||||||
await waitFor(async () => {
|
await waitFor(async () => {
|
||||||
const facts = await standbyDatabase.pool.query(
|
const facts = await standbyDatabase.pool.query(
|
||||||
`SELECT
|
`SELECT
|
||||||
@@ -11759,6 +12052,13 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
copilotFailureDiagnosisAdmission.afterPromotion =
|
copilotFailureDiagnosisAdmission.afterPromotion =
|
||||||
promotedCopilotFailureDiagnosisAdmission;
|
promotedCopilotFailureDiagnosisAdmission;
|
||||||
copilotFailureDiagnosisAdmission.survivedPromotion = true;
|
copilotFailureDiagnosisAdmission.survivedPromotion = true;
|
||||||
|
await executeCopilotFailureDiagnosisToolHaEvidence({
|
||||||
|
port: standbyPort,
|
||||||
|
report: copilotFailureDiagnosisAdmission,
|
||||||
|
fixture: copilotFailureDiagnosisToolFixture,
|
||||||
|
resultKeyFixture: toolResultKeyFixture,
|
||||||
|
expectedExisting: true,
|
||||||
|
});
|
||||||
await verifyModelProviderCredentialCatalogAfterPromotion({
|
await verifyModelProviderCredentialCatalogAfterPromotion({
|
||||||
promotedPort: standbyPort,
|
promotedPort: standbyPort,
|
||||||
promotedPool: promotedDatabase.pool,
|
promotedPool: promotedDatabase.pool,
|
||||||
@@ -12513,7 +12813,7 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
FROM "ql3"."worker_credential_deliveries") AS "credentialDeliveries"`,
|
FROM "ql3"."worker_credential_deliveries") AS "credentialDeliveries"`,
|
||||||
);
|
);
|
||||||
assert.deepEqual(sideEffects.rows, [
|
assert.deepEqual(sideEffects.rows, [
|
||||||
{ runs: 78, runEvents: 179, credentialDeliveries: 4 },
|
{ runs: 78, runEvents: 182, credentialDeliveries: 4 },
|
||||||
]);
|
]);
|
||||||
timeline.push({
|
timeline.push({
|
||||||
state: 'two_fresh_control_replicas_ready',
|
state: 'two_fresh_control_replicas_ready',
|
||||||
@@ -12644,14 +12944,29 @@ async function main(argv = process.argv.slice(2)) {
|
|||||||
optionalAiFeatureSchemaSurvivesPromotion: true,
|
optionalAiFeatureSchemaSurvivesPromotion: true,
|
||||||
copilotFailureDiagnosisAdmissionExactlyReplays:
|
copilotFailureDiagnosisAdmissionExactlyReplays:
|
||||||
copilotFailureDiagnosisAdmission.exactReplay &&
|
copilotFailureDiagnosisAdmission.exactReplay &&
|
||||||
copilotFailureDiagnosisAdmission.beforePromotion.eventCount === 3 &&
|
copilotFailureDiagnosisAdmission.beforePromotion.eventCount === 6 &&
|
||||||
copilotFailureDiagnosisAdmission.beforePromotion.mutationCount === 2,
|
copilotFailureDiagnosisAdmission.beforePromotion.mutationCount === 5,
|
||||||
|
copilotFailureDiagnosisToolCompletesEncryptedAndUnlocksModel:
|
||||||
|
copilotFailureDiagnosisAdmission.toolExecutionExactReplay &&
|
||||||
|
copilotFailureDiagnosisAdmission.encryptedCompletion &&
|
||||||
|
copilotFailureDiagnosisAdmission.beforePromotion.toolStatus ===
|
||||||
|
'succeeded' &&
|
||||||
|
copilotFailureDiagnosisAdmission.beforePromotion.modelStatus ===
|
||||||
|
'ready' &&
|
||||||
|
copilotFailureDiagnosisAdmission.beforePromotion.unlockCount === 1 &&
|
||||||
|
copilotFailureDiagnosisAdmission.beforePromotion
|
||||||
|
.resultArtifactContainsPlaintext === false,
|
||||||
|
copilotFailureDiagnosisToolReplaysAfterPromotionWithoutExecution:
|
||||||
|
copilotFailureDiagnosisAdmission.toolExecutionSurvivedPromotion &&
|
||||||
|
copilotFailureDiagnosisAdmission.promotedToolReplayReads === 0,
|
||||||
copilotFailureDiagnosisAdmissionReplicatesAndSurvivesPromotion:
|
copilotFailureDiagnosisAdmissionReplicatesAndSurvivesPromotion:
|
||||||
copilotFailureDiagnosisAdmission.replicatedBeforePromotion &&
|
copilotFailureDiagnosisAdmission.replicatedBeforePromotion &&
|
||||||
copilotFailureDiagnosisAdmission.survivedPromotion,
|
copilotFailureDiagnosisAdmission.survivedPromotion,
|
||||||
copilotFailureDiagnosisAdmissionUsesLeastPrivilegeRuntime:
|
copilotFailureDiagnosisAdmissionUsesLeastPrivilegeRuntime:
|
||||||
modelInvocationFeaturePromotion.beforePromotion.privileges
|
modelInvocationFeaturePromotion.beforePromotion.privileges
|
||||||
.copilotDiagnosisAdmissionRuntimeOnly &&
|
.copilotDiagnosisAdmissionRuntimeOnly &&
|
||||||
|
modelInvocationFeaturePromotion.beforePromotion.privileges
|
||||||
|
.copilotDiagnosisToolUnlockRuntimeOnly &&
|
||||||
modelInvocationFeaturePromotion.beforePromotion.privileges
|
modelInvocationFeaturePromotion.beforePromotion.privileges
|
||||||
.copilotDiagnosisSnapshotRuntimeOnly,
|
.copilotDiagnosisSnapshotRuntimeOnly,
|
||||||
modelProviderCredentialCatalogSurvivesPromotion:
|
modelProviderCredentialCatalogSurvivesPromotion:
|
||||||
|
|||||||
@@ -268,10 +268,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
|
|||||||
{
|
{
|
||||||
path: 'packages/ql3-ai',
|
path: 'packages/ql3-ai',
|
||||||
name: '@qinglong/ai',
|
name: '@qinglong/ai',
|
||||||
sourceFiles: 168,
|
sourceFiles: 175,
|
||||||
rootSourceFiles: 1,
|
rootSourceFiles: 1,
|
||||||
rootSourceLines: 16,
|
rootSourceLines: 16,
|
||||||
nestedSourceFiles: 167,
|
nestedSourceFiles: 174,
|
||||||
rootSourceFileHardCap: 1,
|
rootSourceFileHardCap: 1,
|
||||||
rootSourceLineHardCap: 16,
|
rootSourceLineHardCap: 16,
|
||||||
rootSourceFileRoles: {
|
rootSourceFileRoles: {
|
||||||
@@ -385,10 +385,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
|
|||||||
rootSourceFileRoles: clusterControl.rootSourceFileRoles,
|
rootSourceFileRoles: clusterControl.rootSourceFileRoles,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
sourceFiles: 54,
|
sourceFiles: 56,
|
||||||
rootSourceFiles: 2,
|
rootSourceFiles: 2,
|
||||||
rootSourceLines: 195,
|
rootSourceLines: 195,
|
||||||
nestedSourceFiles: 52,
|
nestedSourceFiles: 54,
|
||||||
rootSourceFileRoles: {
|
rootSourceFileRoles: {
|
||||||
'aiCli.ts': 'binary_entry',
|
'aiCli.ts': 'binary_entry',
|
||||||
'cli.ts': 'binary_entry',
|
'cli.ts': 'binary_entry',
|
||||||
|
|||||||
Reference in New Issue
Block a user