mirror of
https://github.com/whyour/qinglong.git
synced 2026-09-20 16:07:11 +08:00
feat(ql3): execute copilot diagnosis tools
This commit is contained in:
whyour
@@ -11,6 +11,7 @@
|
||||
|
||||
最新增量证据（2026-08-15）：
|
||||
|
||||
- D-316/ADR-0408（已接受）：Cluster Copilot 现在能够从 ADR-0407 的 durable admission 恢复 exact `qinglong.run.log.excerpt@1.0.0` authority，复用通用 Trusted Tool start barrier、加密 success/failure completion、result catalog/rekey 和内建日志 adapter；确定性 start/completion identity 让 response-loss replay 直接打开既有证据，不重复读取日志或执行 adapter。Cluster invocation Artifact 使用独立 projected keyring，提供 active+historical material，但每次读取均重新执行 canonical path/symlink/mode/inode/realpath fence，且不取得 PostgreSQL Tool result generation authority。只有 exact `succeeded` completion 才能通过 `pg-9019` 的 SERIALIZABLE 事务把 Model Step 从 `pending` 原子推进到 `ready`，同时写 RunEvent、StepRunMutation 和 append-only unlock receipt；`failed|timed_out` 不解锁。本 Gate 不执行模型、不终态化 diagnosis Run，下一 Gate 是 ADR-0405 builder + Model Gateway + Copilot encrypted model completion/terminalization。实现仍为 18 个 package，无单文件/浅平 package；AI 175 个源码中 174 个、Cluster Control 56 个中 54 个位于嵌套目录，不新增依赖、进程、连接、timer/watcher/cache 或默认 Edge 成本。18-package clean build/test 全绿，AI 229 pass/3 条件 skip、backend 1,207 pass/2 条件 skip，四项架构审计零 finding；14 档 Local artifact 全通过，默认 Edge/Standalone 为 2,589,890/2,589,968 bytes。PostgreSQL 18.4 arm64 HA 130/130、timeline `1→2`，首次执行只读两次日志、密文不含敏感 fixture，提升后 exact replay 零日志读取；报告 SHA-256 为 `d525a303696e178d777b021b376729bd2c5382fb5eb7bc98466a2b79d3940517`，独立审计与 Docker 清理通过。
|
||||
- D-315/ADR-0407（已接受）：Cluster Copilot 故障诊断不修改或重新打开终态源 Run，而是用源 Run/version、最新已结束 Attempt/status 与日志 Artifact 建立 exact fence，原子创建独立 `copilot_failure_diagnosis` Run、admission event、`ready` 的 `qinglong.run.log.excerpt@1.0.0` Tool Step、以其为父节点的 `pending` Model Step 和不可变 receipt。计划只接受 `cluster-control` 的 reviewed snapshot/binding/Policy/subject authority；`approval_required`、Tool contract/输入漂移或未显式允许的模型出口均在写库前失败关闭。`ql3_ai` 新增有界 admission ledger 与仅授予 runtime 的 `SECURITY DEFINER` source snapshot，在 SERIALIZABLE 事务内重验 active Project/binding 和源 Run/Attempt；response-loss replay 对 JSONB 做结构 exact 比较而不依赖无语义的对象键序。Copilot/Plugin 这类 StepRun 聚合从通用 Task orphan recovery 排除，后续由各自状态机恢复，不能伪造顶层 Attempt 或被错误终态化。本阶段只 admission，不执行 Tool/模型、不授予行动权；Tool encrypted completion、Model 解锁和 Copilot encrypted model completion 是下一 Gate。最终 18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，四项边界审计零 finding且 workspace 仍为 18 package、无单文件/浅平 package；14 档 Local Profile artifact 全通过，默认 Edge/Standalone 不引入 Cluster admission。PostgreSQL 18.4 arm64 HA 128/128、timeline `1→2`，报告 SHA-256 为 `a4ed1edec783e3f5b42507c0f8e11b94c59dbe44a57e691017d1445ec9d115e2`，证据审计与 Docker 清理通过。
|
||||
- D-314/ADR-0406（已接受）：Cluster Trusted Tool 的 encrypted completion 不再停留在 storage port。`@qinglong/cluster-control/trusted-tool-result-keyring` 新增只读 projected material authority：canonical v1 manifest 只含最多 16 个 canonical 32-byte key，不含 generation、active/state/retirement，provider 只有 `resolve(keyId)` 而没有 `active()`；因此 PostgreSQL `trusted-tool-results` catalog 仍是 active/decryptable 状态唯一 authority，completion 会以 catalog material proof 再次校验。runtime 每次调用重新执行 direct-root、in-root atomic symlink、single-link、64 KiB、只读/不可执行/other-inaccessible mode、dev/inode/size/mtime 与双 realpath fence，不持有 Kubernetes API、cache、watcher 或 timer。新增能力位于 Cluster Control 既有 `trusted-tool/key-management/`，并把 mounted Secret 与 keyring 的 projected-file/TOCTOU 逻辑收敛到 package-private `security/privateProjectedFile` 真源；公开 mounted Secret 行为不变，不新增 package、依赖、migration、连接、route 或默认 Profile importer。定向共享回归 7/7，Cluster Control 完整 234 pass/2 条条件 skip/0 fail；最终 18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，package/dependency/Edge import/Cluster deployment 四项审计零 finding。workspace 仍为 18 package、无单文件或浅平 package；Cluster Control 54 个源码中仅 2 个 binary entry 位于根层，52 个处于嵌套领域目录。14 档 Local Profile artifact 全部通过，默认 Edge/Standalone 保持 2,589,812/2,589,890 bytes，AI 保持 3,121,108/3,121,198 bytes，MCP 保持 7,315,930/7,316,038 bytes。PostgreSQL 18.4 arm64 HA 125/125、timeline `1→2`，报告 SHA-256 为 `26c817647ed984d8d4627a7cae1c95de06017a5d6d32dd3dfd01414ba029e542`，证据审计与 Docker 容器/网络/卷零残留。下一 Gate 是独立 diagnosis Run 的 Tool/Model Step admission 与 Copilot encrypted model completion，不能借用终态源 Run 或 Plugin Prompt plan。
|
||||
- D-313/ADR-0405（已接受）：新增 `@qinglong/ai/failure-diagnosis-prompt`，把 ADR-0403 的潜在敏感日志投影收敛为固定 system instruction + canonical JSON data envelope；日志只存在于 `log.content` JSON string value，不能通过引号、换行、伪造 role/schema 或 delimiter 拼接出新 message。builder 重新校验完整 trust/redaction/profile byte 契约，拒绝伪造 `safe`、行动权、未知字段与 byte/signal drift；envelope 不带 Run/Attempt、Artifact、path、cursor 或 content digest。部署者必须通过 `qinglong/copilot-model-egress-policy@v1` 显式允许 `potentially_sensitive` 数据进入 `on_device|external` 边界并提供输入/output token 双预算，空 allowlist 与 external 未授权均在 Model Gateway/Provider I/O 前失败关闭。输出只含 content-free egress evidence，并固定要求模型 completion 继承潜在敏感、仅加密持久化、禁止明文审计且无行动权；真正 Cluster Trusted Tool/model completion 仍需后续组合门。能力以 `ql3-ai/src/copilot/failure-diagnosis/` 三个内聚文件和精确 subpath 交付，不新增 package、依赖、迁移、连接或常驻组件。定向 12/12、AI 221 pass/3 条件 skip/0 fail；最终 18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，四项结构/部署审计零 finding，14 档 Local Profile artifact 全部通过。默认 Edge/Standalone 保持 2,589,812/2,589,890 bytes、315 files、56 modules，Edge/Standalone AI 保持 3,121,108/3,121,198 bytes、368 files、61 modules，MCP 保持 7,315,930/7,316,038 bytes、801 files、226 modules，证明未装配 subpath 被完全裁掉。PostgreSQL 18.4 arm64 HA 125/125、timeline `1→2`，报告 SHA-256 为 `2bbc8bdd0d90e6ec9ce82d2afcaec817679dddb82860c5d405a09d5e5458bece`，证据审计与 Docker 零残留。
|
||||
@@ -6942,9 +6943,12 @@ ADR-0407 已完成该 admission Gate：源失败/超时 Run 和最新已结束 A
|
||||
独立 diagnosis Run 在一个 SERIALIZABLE 事务内同时获得 admission event、ready Tool Step、依赖它的
|
||||
pending Model Step 与 receipt；Cluster runtime 的 source snapshot authority 会重验 active Project/binding
|
||||
和日志 Artifact，response-loss replay 以结构语义比较 JSONB。该 Run 是 StepRun 编排聚合，不进入通用
|
||||
Task orphan recovery。当前阶段仍没有执行 Tool 或模型；下一 Gate 必须先形成可信 encrypted Tool
|
||||
completion，再原子解锁 Model Step，并以 ADR-0405 egress policy 和 Copilot 专用 encrypted model
|
||||
completion 收敛最终 Run。
|
||||
Task orphan recovery。ADR-0408 已进一步关闭 Tool execution Gate：coordinator 从 durable admission 恢复
|
||||
exact Tool authority，复用通用 start barrier 与 encrypted completion；首次执行有界读取日志，response-loss
|
||||
或主库提升后的 exact replay 不再执行 adapter。只有成功 completion 才由 `pg-9019` 的原子 unlock ledger
|
||||
把 Model Step 推进到 ready，失败或超时保持 pending。当前阶段仍没有执行模型或终态化 diagnosis Run；
|
||||
下一 Gate 必须以 ADR-0405 builder 和 Model Gateway 形成 Copilot 专用 encrypted model completion，随后
|
||||
原子收敛 Model Step、Run、取消、deadline 与恢复语义。
|
||||
|
||||
## 17. Tool Registry
|
||||
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
# ADR-0408：Cluster Copilot Failure Diagnosis Tool Execution
|
||||
|
||||
- 状态：Accepted
|
||||
- 日期：2026-08-15
|
||||
- 关联 RFC：QL-RFC-0001 D-316、Phase 2
|
||||
- 关联 ADR：ADR-0163、ADR-0226、ADR-0403、ADR-0405、ADR-0406、ADR-0407
|
||||
|
||||
## 问题
|
||||
|
||||
ADR-0407 只把故障诊断原子接纳为独立 Run、一个 `ready` Tool Step 和一个依赖它的
|
||||
`pending` Model Step。系统仍不能执行该 Tool，也不能在崩溃、响应丢失或 PostgreSQL
|
||||
主库切换后证明 Tool 是否已经产生副作用、结果是否已经加密持久化，以及 Model Step
|
||||
是否可以安全解锁。
|
||||
|
||||
若 Copilot 绕过共享 Trusted Tool start/completion 协议直接读取日志，重放可能重复执行
|
||||
adapter；若先解锁 Model 再写 completion，模型可能读取缺失、失败或未验证的结果；若复用
|
||||
Prompt output key 或让 projected file 决定 Tool result active generation，又会合并本应独立的
|
||||
密钥域和 durable authority。
|
||||
|
||||
## 决策
|
||||
|
||||
1. 新增 Copilot failure-diagnosis Tool execution coordinator。它只接受 ADR-0407 已持久化的
|
||||
`requestId`，重新读取并规范化 exact admission plan/receipt、当前 Project Tool snapshot 和
|
||||
invocation input/preview Artifact；任何 plan、Definition、binding、Artifact 或 source fence
|
||||
漂移均在日志读取前失败关闭。
|
||||
2. coordinator 复用 Runtime Core 的通用 Trusted Tool execution evidence、start barrier、
|
||||
encrypted success/failure completion、result catalog 和 rekey 协议，并只装配内建
|
||||
`qinglong.run.log.excerpt@1.0.0` adapter。start/completion 使用从 plan 稳定派生的 identity；
|
||||
exact replay 首先打开 durable barrier/completion，不再次读取日志或执行 adapter。
|
||||
3. invocation Artifact 使用独立的 Cluster projected keyring。它包含一个 active key 和最多
|
||||
16 个历史 key，提供 `active()` 与 `resolve(keyId)`，每次调用重新读取 canonical 32-byte
|
||||
material，不使用 cache、watcher 或 timer，并复用既有 projected-file 的 canonical path、
|
||||
symlink、权限、inode/size/mtime 和双 realpath fence。Tool result keyring 仍保持 resolve-only，
|
||||
PostgreSQL result catalog 继续独占 active/decryptable generation authority；两个密钥域不得互换。
|
||||
4. 只有规范化且与 admission exact 绑定的 `succeeded` Tool completion 才能解锁 Model Step。
|
||||
`pg-9019-ai-copilot-failure-diagnosis-tool-unlocks` 新增 append-only unlock ledger；一个
|
||||
SERIALIZABLE 事务同时把 Model Step 从 `pending` 推进到 `ready`、递增 Run version/event
|
||||
sequence、写 RunEvent、StepRunMutation 和不可变 unlock receipt。Tool `failed|timed_out`
|
||||
completion 不解锁 Model，也不伪造最终 diagnosis 结论。
|
||||
5. unlock transaction 重新锁定并验证 admission、success completion、当前 Model Step 和 running
|
||||
diagnosis Run；serialization/deadlock 只在事务提交前有界重试。response-loss replay 必须返回
|
||||
结构完全一致的 receipt；缺失关联事实、旧 fence、不同 digest 或部分写入全部视为冲突/损坏。
|
||||
6. `ql3_runtime` 对 unlock ledger 只有 `SELECT, INSERT`，没有 `UPDATE, DELETE`；数据库外调用者
|
||||
不能直接把 Model Step 改为 ready。Tool execution 与 PostgreSQL storage 分别通过精确 AI
|
||||
subpath 发布，纯 execution 入口不隐式 re-export PostgreSQL repository。
|
||||
7. 本 Gate 到 Model Step `ready` 为止，不调用模型、不持久化模型明文、不终态化 diagnosis Run。
|
||||
ADR-0405 prompt builder、Model Gateway、Copilot 专用 encrypted model completion、失败/取消/
|
||||
deadline terminalization 和产品 API 属于后续 Gate。
|
||||
8. 能力内聚在既有 `@qinglong/ai` 与 `@qinglong/cluster-control` 的领域子目录，不新增 workspace
|
||||
package、依赖、进程、Pod、连接池、listener、daemon、timer、watcher 或 cache。默认 Edge/
|
||||
Standalone 不导入 Cluster execution；显式本地 AI 制品也不会取得 PostgreSQL/S3 authority。
|
||||
|
||||
## 被否决方案
|
||||
|
||||
1. **Copilot 自建一套 Tool completion**：会分叉 start barrier、加密 Artifact、rekey 与恢复语义。
|
||||
2. **admission 后直接调用日志 reader**：响应丢失时无法区分“未执行”和“已执行但未返回”。
|
||||
3. **Tool 一启动就解锁 Model**：允许模型消费未完成、失败或未认证的结果。
|
||||
4. **把失败 Tool completion 当成可诊断输入**：当前没有经过审定的失败 Prompt/terminalization
|
||||
协议，会把基础设施错误伪装成业务诊断。
|
||||
5. **共用 Prompt output/result/invocation keyring**：混淆密钥用途、active authority 与退役范围。
|
||||
6. **为 keyring 或 coordinator 新建 package**：没有独立部署/ownership 边界，并会继续加剧包碎片化。
|
||||
|
||||
## 当前验证
|
||||
|
||||
1. admission/execution/unlock 与 migration/readiness 聚焦契约 21/21，projected invocation keyring
|
||||
2/2；AI package 最终为 229 pass、3 条件 skip、0 fail，类型检查通过。
|
||||
2. 18 个 QL3 package 的 clean build/test 全部退出 0；backend 为 1,207 pass、2 条件 skip、
|
||||
0 fail。
|
||||
3. package boundary、Cluster dependency、Edge import 和 Cluster deployment 四项审计均
|
||||
compatible 且零 finding。workspace 仍为 18 个 package，`singleSourcePackages=[]`、
|
||||
`shallowSourcePackages=[]`；AI 175 个源码中 174 个、Cluster Control 56 个中 54 个位于
|
||||
嵌套领域目录。
|
||||
4. 14 档 Local Profile artifact 全部通过。默认 Edge/Standalone 为 2,589,890 / 2,589,968
|
||||
bytes；显式 Edge/Standalone AI 为 3,135,809 / 3,135,899 bytes；最大 Standalone MCP 为
|
||||
7,316,038 bytes，均在各自门内。
|
||||
5. PostgreSQL 18.4 arm64 physical HA 为 130/130 Gate、timeline `1→2`。门证明首次 Tool 执行只
|
||||
读取两次日志、密文 JSON 不含敏感 fixture 文本、Model Step 原子进入 ready；主库提升后 exact
|
||||
replay 的日志读取为零。报告 SHA-256 为
|
||||
`d525a303696e178d777b021b376729bd2c5382fb5eb7bc98466a2b79d3940517`，独立审计零 finding，
|
||||
临时 Docker 资源已清理。
|
||||
|
||||
## 后续门禁
|
||||
|
||||
1. 从 durable Tool completion 解密受信投影，经 ADR-0405 builder 生成 prompt，并通过 Model
|
||||
Gateway 执行 provider credential、egress policy、quota、price 和 deadline fence。
|
||||
2. 建立 Copilot 专用 encrypted model completion 和 response-loss recovery；模型输出不得进入
|
||||
普通 completion、数据库 JSON、审计或日志明文。
|
||||
3. 原子终态化 Model Step 与 diagnosis Run，并覆盖 Tool/Model 失败、取消、超时、未知结果和
|
||||
主库切换后的恢复状态机。
|
||||
4. 最后再开放默认关闭的 Cluster API/CLI/UI/MCP 产品入口，并补真实 S3、外部 Provider fault
|
||||
injection、多副本并发和 plaintext negative evidence。
|
||||
@@ -411,6 +411,7 @@
|
||||
| [ADR-0405](./ADR-0405-bounded-failure-diagnosis-prompt-and-model-egress-policy.md) | 有界故障诊断 Prompt 与显式模型出口策略 | Accepted |
|
||||
| [ADR-0406](./ADR-0406-cluster-projected-tool-result-key-authority.md) | Cluster Projected Tool Result Key Authority | Accepted |
|
||||
| [ADR-0407](./ADR-0407-cluster-copilot-failure-diagnosis-run-admission.md) | Cluster Copilot Failure Diagnosis Run Admission | Accepted |
|
||||
| [ADR-0408](./ADR-0408-cluster-copilot-failure-diagnosis-tool-execution.md) | Cluster Copilot Failure Diagnosis Tool Execution 与原子 Model 解锁 | Accepted |
|
||||
|
||||
## 规则
|
||||
|
||||
|
||||
@@ -40,6 +40,16 @@
|
||||
"require": "./dist/copilot/failure-diagnosis/postgresAdmissionRepository.js",
|
||||
"default": "./dist/copilot/failure-diagnosis/postgresAdmissionRepository.js"
|
||||
},
|
||||
"./failure-diagnosis-tool-execution": {
|
||||
"types": "./dist/copilot/failure-diagnosis/toolExecution.d.ts",
|
||||
"require": "./dist/copilot/failure-diagnosis/toolExecution.js",
|
||||
"default": "./dist/copilot/failure-diagnosis/toolExecution.js"
|
||||
},
|
||||
"./postgres-failure-diagnosis-tool-execution-storage": {
|
||||
"types": "./dist/copilot/failure-diagnosis/postgresToolExecutionRepository.d.ts",
|
||||
"require": "./dist/copilot/failure-diagnosis/postgresToolExecutionRepository.js",
|
||||
"default": "./dist/copilot/failure-diagnosis/postgresToolExecutionRepository.js"
|
||||
},
|
||||
"./model-invocation": {
|
||||
"types": "./dist/model-invocation/modelInvocation.d.ts",
|
||||
"require": "./dist/model-invocation/modelInvocation.js",
|
||||
|
||||
@@ -175,7 +175,7 @@ export function createCopilotFailureDiagnosisAdmissionBundle(
|
||||
runId: plan.runId,
|
||||
stepKey: 'collect-log',
|
||||
kind: 'tool',
|
||||
definitionRef: `trusted-tool-plan:${plan.tool.planDigest}`,
|
||||
definitionRef: 'tool:qinglong.run.log.excerpt@1.0.0',
|
||||
definitionDigest: plan.tool.definitionDigest,
|
||||
required: true,
|
||||
initialStatus: 'ready',
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
export * from './tool-execution/postgresUnlockRepository';
|
||||
@@ -0,0 +1,131 @@
|
||||
import type { RunRecord } from '@qinglong/runtime-core';
|
||||
import type { SecurityPrincipal } from '@qinglong/runtime-core/security';
|
||||
import type { StepRunMutation } from '@qinglong/runtime-core/step-run';
|
||||
import type {
|
||||
ToolExecutionCompletionRecord,
|
||||
ToolExecutionResultArtifactReference,
|
||||
} from '@qinglong/runtime-core/tool-execution-completion';
|
||||
import type { ToolPolicyAuthorizer } from '@qinglong/runtime-core/tool-registry';
|
||||
|
||||
import type {
|
||||
CopilotFailureDiagnosisAdmissionRepository,
|
||||
CopilotFailureDiagnosisExecutionPlan,
|
||||
} from '../admission/contracts';
|
||||
|
||||
export const COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA =
|
||||
'qinglong/copilot-failure-diagnosis-tool-unlock-receipt@v1' as const;
|
||||
export const COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA =
|
||||
'qinglong/copilot-failure-diagnosis-tool-unlock-command@v1' as const;
|
||||
export const MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_BYTES =
|
||||
16 * 1024;
|
||||
export const MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_BYTES =
|
||||
48 * 1024;
|
||||
|
||||
export interface CopilotFailureDiagnosisToolUnlockReceipt {
|
||||
readonly schema: typeof COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA;
|
||||
readonly requestId: string;
|
||||
readonly planDigest: string;
|
||||
readonly runId: string;
|
||||
readonly startId: string;
|
||||
readonly barrierDigest: string;
|
||||
readonly toolStepRunId: string;
|
||||
readonly toolCompletionDigest: string;
|
||||
readonly resultArtifact: Readonly<ToolExecutionResultArtifactReference>;
|
||||
readonly modelStepRunId: string;
|
||||
readonly modelStepRunVersion: number;
|
||||
readonly modelStepRunDigest: string;
|
||||
readonly modelMutationId: string;
|
||||
readonly modelMutationDigest: string;
|
||||
readonly modelEventId: string;
|
||||
readonly finalRunVersion: number;
|
||||
readonly finalRunEventSequence: number;
|
||||
readonly unlockedAtMs: number;
|
||||
readonly receiptDigest: string;
|
||||
}
|
||||
|
||||
export interface CopilotFailureDiagnosisToolUnlockCommand {
|
||||
readonly schema: typeof COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA;
|
||||
readonly plan: Readonly<CopilotFailureDiagnosisExecutionPlan>;
|
||||
readonly completion: Readonly<ToolExecutionCompletionRecord>;
|
||||
readonly modelStepRunMutation: Readonly<StepRunMutation>;
|
||||
readonly receipt: Readonly<CopilotFailureDiagnosisToolUnlockReceipt>;
|
||||
readonly commandDigest: string;
|
||||
}
|
||||
|
||||
export interface CopilotFailureDiagnosisToolUnlockRepository {
|
||||
findByRequestId(
|
||||
requestId: string,
|
||||
): Promise<Readonly<CopilotFailureDiagnosisToolUnlockReceipt> | null>;
|
||||
commit(command: CopilotFailureDiagnosisToolUnlockCommand): Promise<
|
||||
Readonly<{
|
||||
status: 'created' | 'existing';
|
||||
receipt: Readonly<CopilotFailureDiagnosisToolUnlockReceipt>;
|
||||
}>
|
||||
>;
|
||||
}
|
||||
|
||||
export interface ExecuteCopilotFailureDiagnosisToolInput {
|
||||
readonly requestId: string;
|
||||
readonly principal: Readonly<SecurityPrincipal>;
|
||||
readonly authorizer: ToolPolicyAuthorizer;
|
||||
}
|
||||
|
||||
export type CopilotFailureDiagnosisToolExecutionResult =
|
||||
| Readonly<{
|
||||
outcome: 'succeeded';
|
||||
completionStatus: 'created' | 'existing';
|
||||
unlockStatus: 'created' | 'existing';
|
||||
completion: Readonly<ToolExecutionCompletionRecord>;
|
||||
unlock: Readonly<CopilotFailureDiagnosisToolUnlockReceipt>;
|
||||
}>
|
||||
| Readonly<{
|
||||
outcome: 'failed' | 'timed_out';
|
||||
completionStatus: 'created' | 'existing';
|
||||
unlockStatus: null;
|
||||
}>;
|
||||
|
||||
export class InvalidCopilotFailureDiagnosisToolExecutionError extends TypeError {
|
||||
readonly code = 'COPILOT_FAILURE_DIAGNOSIS_TOOL_EXECUTION_INVALID';
|
||||
|
||||
constructor(message: string) {
|
||||
super(`Copilot failure diagnosis Tool execution is invalid: ${message}`);
|
||||
this.name = 'InvalidCopilotFailureDiagnosisToolExecutionError';
|
||||
}
|
||||
}
|
||||
|
||||
export class CopilotFailureDiagnosisToolExecutionConflictError extends Error {
|
||||
readonly code = 'COPILOT_FAILURE_DIAGNOSIS_TOOL_EXECUTION_CONFLICT';
|
||||
|
||||
constructor(message = 'durable Tool execution facts changed') {
|
||||
super(`Copilot failure diagnosis Tool execution conflicts: ${message}`);
|
||||
this.name = 'CopilotFailureDiagnosisToolExecutionConflictError';
|
||||
}
|
||||
}
|
||||
|
||||
export class CopilotFailureDiagnosisToolExecutionUnavailableError extends Error {
|
||||
readonly code = 'COPILOT_FAILURE_DIAGNOSIS_TOOL_EXECUTION_UNAVAILABLE';
|
||||
|
||||
constructor(options?: ErrorOptions) {
|
||||
super('Copilot failure diagnosis Tool execution is unavailable', options);
|
||||
this.name = 'CopilotFailureDiagnosisToolExecutionUnavailableError';
|
||||
}
|
||||
}
|
||||
|
||||
export class CopilotFailureDiagnosisToolExecutionDeadlineExceededError extends Error {
|
||||
readonly code = 'COPILOT_FAILURE_DIAGNOSIS_TOOL_EXECUTION_DEADLINE_EXCEEDED';
|
||||
|
||||
constructor() {
|
||||
super('Copilot failure diagnosis Tool execution deadline was exceeded');
|
||||
this.name = 'CopilotFailureDiagnosisToolExecutionDeadlineExceededError';
|
||||
}
|
||||
}
|
||||
|
||||
export type CopilotFailureDiagnosisToolExecutionAdmissionReader = Pick<
|
||||
CopilotFailureDiagnosisAdmissionRepository,
|
||||
'findByRequestId' | 'findPlanByRequestId'
|
||||
>;
|
||||
|
||||
export type CopilotFailureDiagnosisRunAuthority = Pick<
|
||||
RunRecord,
|
||||
'id' | 'projectId' | 'status' | 'version' | 'eventSequence'
|
||||
>;
|
||||
@@ -0,0 +1,599 @@
|
||||
import { Buffer } from 'node:buffer';
|
||||
import { createHash } from 'node:crypto';
|
||||
|
||||
import type { RunRepositoryReader } from '@qinglong/runtime-core/run-repository';
|
||||
import {
|
||||
transitionStepRunMutation,
|
||||
type StepRunRepository,
|
||||
} from '@qinglong/runtime-core/step-run';
|
||||
import {
|
||||
createToolExecutionEvidenceBundle,
|
||||
toolExecutionAdmissionEvidence,
|
||||
TOOL_EXECUTION_START_AUDIT_OPERATION,
|
||||
} from '@qinglong/runtime-core/tool-execution-evidence';
|
||||
import {
|
||||
createToolExecutionStartCommand,
|
||||
normalizeToolExecutionStartBarrierRecord,
|
||||
type ToolExecutionStartBarrierRecord,
|
||||
type ToolExecutionStartBarrierRepository,
|
||||
} from '@qinglong/runtime-core/tool-execution-start-barrier';
|
||||
import type { ToolExecutionCompletionRepository } from '@qinglong/runtime-core/tool-execution-completion';
|
||||
import type { ToolExecutionFailureCompletionRepository } from '@qinglong/runtime-core/tool-execution-failure-completion';
|
||||
import type { ToolExecutionResultRekeyReader } from '@qinglong/runtime-core/tool-result-rekey';
|
||||
import type { ToolResultKeyCatalogReader } from '@qinglong/runtime-core/tool-result-key-catalog';
|
||||
import type { ProjectToolDefinitionSnapshotRepository } from '@qinglong/runtime-core/project-tool-definition-snapshot';
|
||||
import {
|
||||
toolInvocationInputArtifactReference,
|
||||
toolInvocationPreviewArtifactReference,
|
||||
type ToolInvocationArtifactKeyProvider,
|
||||
type ToolInvocationArtifactRepository,
|
||||
} from '@qinglong/runtime-core/tool-invocation-artifact';
|
||||
import {
|
||||
admitTrustedToolExecution,
|
||||
trustedToolContractIdentityDigest,
|
||||
} from '@qinglong/runtime-core/trusted-tool-invocation';
|
||||
import { TrustedToolExecutionAdapterRegistry } from '@qinglong/runtime-core/trusted-tool-execution';
|
||||
import { executeAndCompleteTrustedTool } from '@qinglong/runtime-core/trusted-tool-completion';
|
||||
import { BuiltInRunLogExcerptToolAdapter } from '@qinglong/runtime-core/builtin-run-log-excerpt-tool';
|
||||
import type { RunAttemptLogReadPort } from '@qinglong/runtime-core/builtin-run-log-excerpt-projection';
|
||||
|
||||
import {
|
||||
CopilotFailureDiagnosisToolExecutionConflictError,
|
||||
CopilotFailureDiagnosisToolExecutionDeadlineExceededError,
|
||||
CopilotFailureDiagnosisToolExecutionUnavailableError,
|
||||
InvalidCopilotFailureDiagnosisToolExecutionError,
|
||||
type CopilotFailureDiagnosisToolExecutionAdmissionReader,
|
||||
type CopilotFailureDiagnosisToolExecutionResult,
|
||||
type CopilotFailureDiagnosisToolUnlockRepository,
|
||||
type ExecuteCopilotFailureDiagnosisToolInput,
|
||||
} from './contracts';
|
||||
import { restoreCopilotFailureDiagnosisTrustedToolAuthority } from './planAuthority';
|
||||
import { createCopilotFailureDiagnosisToolUnlockCommand } from './unlockProtocol';
|
||||
|
||||
export interface CopilotFailureDiagnosisToolExecutionDependencies {
|
||||
readonly admissions: CopilotFailureDiagnosisToolExecutionAdmissionReader;
|
||||
readonly snapshots: Pick<
|
||||
ProjectToolDefinitionSnapshotRepository,
|
||||
'findCurrent'
|
||||
>;
|
||||
readonly artifacts: ToolInvocationArtifactRepository;
|
||||
readonly invocationKeys: Pick<ToolInvocationArtifactKeyProvider, 'resolve'>;
|
||||
readonly resultKeys: Pick<ToolInvocationArtifactKeyProvider, 'resolve'>;
|
||||
readonly stepRuns: Pick<StepRunRepository, 'findById'>;
|
||||
readonly runs: Pick<RunRepositoryReader, 'findRunById'>;
|
||||
readonly barriers: ToolExecutionStartBarrierRepository;
|
||||
readonly completions: ToolExecutionCompletionRepository;
|
||||
readonly failureCompletions: ToolExecutionFailureCompletionRepository;
|
||||
readonly resultKeyCatalog: ToolResultKeyCatalogReader;
|
||||
readonly resultRekeys: ToolExecutionResultRekeyReader;
|
||||
readonly logs: RunAttemptLogReadPort;
|
||||
readonly unlocks: CopilotFailureDiagnosisToolUnlockRepository;
|
||||
readonly now?: () => number;
|
||||
readonly nonceFactory?: () => Uint8Array;
|
||||
}
|
||||
|
||||
const IDENTITY_DOMAIN = Buffer.from(
|
||||
'qinglong/copilot-failure-diagnosis-tool-execution-identity@v1\0',
|
||||
'utf8',
|
||||
);
|
||||
|
||||
function invalid(message: string): never {
|
||||
throw new InvalidCopilotFailureDiagnosisToolExecutionError(message);
|
||||
}
|
||||
|
||||
function unavailable(cause?: unknown): never {
|
||||
throw new CopilotFailureDiagnosisToolExecutionUnavailableError({
|
||||
cause: cause instanceof Error ? cause : undefined,
|
||||
});
|
||||
}
|
||||
|
||||
function hash(value: unknown): string {
|
||||
return createHash('sha256')
|
||||
.update(IDENTITY_DOMAIN)
|
||||
.update(JSON.stringify(value))
|
||||
.digest('hex');
|
||||
}
|
||||
|
||||
function identity(prefix: string, planDigest: string): string {
|
||||
const maximumDigestLength = 35 - prefix.length;
|
||||
return `${prefix}:${hash({ prefix, planDigest }).slice(
|
||||
0,
|
||||
maximumDigestLength,
|
||||
)}`;
|
||||
}
|
||||
|
||||
function traceIdentity(planDigest: string): string {
|
||||
return hash({ prefix: 'trace', planDigest }).slice(0, 32);
|
||||
}
|
||||
|
||||
function spanIdentity(planDigest: string): string {
|
||||
return hash({ prefix: 'span', planDigest }).slice(0, 16);
|
||||
}
|
||||
|
||||
function auditEventIdentity(planDigest: string): string {
|
||||
const value = hash({ prefix: 'audit', planDigest }).slice(0, 32).split('');
|
||||
value[12] = '4';
|
||||
value[16] = '8';
|
||||
const hex = value.join('');
|
||||
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(
|
||||
12,
|
||||
16,
|
||||
)}-${hex.slice(16, 20)}-${hex.slice(20)}`;
|
||||
}
|
||||
|
||||
function clock(now: (() => number) | undefined): number {
|
||||
let value: number;
|
||||
try {
|
||||
value = (now ?? Date.now)();
|
||||
} catch (cause) {
|
||||
return unavailable(cause);
|
||||
}
|
||||
if (!Number.isSafeInteger(value) || value < 0) return unavailable();
|
||||
return value;
|
||||
}
|
||||
|
||||
function sameValue(left: unknown, right: unknown): boolean {
|
||||
return JSON.stringify(left) === JSON.stringify(right);
|
||||
}
|
||||
|
||||
function assertDependencies(
|
||||
dependencies: CopilotFailureDiagnosisToolExecutionDependencies,
|
||||
): void {
|
||||
if (
|
||||
!dependencies ||
|
||||
typeof dependencies !== 'object' ||
|
||||
typeof dependencies.admissions?.findByRequestId !== 'function' ||
|
||||
typeof dependencies.admissions?.findPlanByRequestId !== 'function' ||
|
||||
typeof dependencies.snapshots?.findCurrent !== 'function' ||
|
||||
typeof dependencies.artifacts?.findInput !== 'function' ||
|
||||
typeof dependencies.artifacts?.findPreview !== 'function' ||
|
||||
typeof dependencies.invocationKeys?.resolve !== 'function' ||
|
||||
typeof dependencies.resultKeys?.resolve !== 'function' ||
|
||||
typeof dependencies.stepRuns?.findById !== 'function' ||
|
||||
typeof dependencies.runs?.findRunById !== 'function' ||
|
||||
typeof dependencies.barriers?.findByStartId !== 'function' ||
|
||||
typeof dependencies.barriers?.prepare !== 'function' ||
|
||||
typeof dependencies.completions?.findByStartId !== 'function' ||
|
||||
typeof dependencies.failureCompletions?.findByStartId !== 'function' ||
|
||||
typeof dependencies.resultKeyCatalog?.findCurrent !== 'function' ||
|
||||
typeof dependencies.resultRekeys?.findHeadByArtifactId !== 'function' ||
|
||||
typeof dependencies.logs?.read !== 'function' ||
|
||||
typeof dependencies.unlocks?.findByRequestId !== 'function' ||
|
||||
typeof dependencies.unlocks?.commit !== 'function' ||
|
||||
(dependencies.now !== undefined &&
|
||||
typeof dependencies.now !== 'function') ||
|
||||
(dependencies.nonceFactory !== undefined &&
|
||||
typeof dependencies.nonceFactory !== 'function')
|
||||
) {
|
||||
return invalid('dependencies are invalid');
|
||||
}
|
||||
}
|
||||
|
||||
async function durablePlan(
|
||||
requestId: string,
|
||||
dependencies: CopilotFailureDiagnosisToolExecutionDependencies,
|
||||
) {
|
||||
let plan;
|
||||
let receipt;
|
||||
try {
|
||||
[plan, receipt] = await Promise.all([
|
||||
dependencies.admissions.findPlanByRequestId(requestId),
|
||||
dependencies.admissions.findByRequestId(requestId),
|
||||
]);
|
||||
} catch (cause) {
|
||||
return unavailable(cause);
|
||||
}
|
||||
if (
|
||||
!plan ||
|
||||
!receipt ||
|
||||
plan.requestId !== requestId ||
|
||||
receipt.requestId !== requestId ||
|
||||
receipt.planDigest !== plan.planDigest ||
|
||||
receipt.runId !== plan.runId ||
|
||||
receipt.toolStepRunId !== plan.toolStepRunId ||
|
||||
receipt.modelStepRunId !== plan.modelStepRunId
|
||||
) {
|
||||
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||
'diagnosis admission evidence is incomplete',
|
||||
);
|
||||
}
|
||||
return plan;
|
||||
}
|
||||
|
||||
async function currentAuthority(
|
||||
plan: Awaited<ReturnType<typeof durablePlan>>,
|
||||
dependencies: CopilotFailureDiagnosisToolExecutionDependencies,
|
||||
) {
|
||||
let record;
|
||||
try {
|
||||
record = await dependencies.snapshots.findCurrent(plan.projectId);
|
||||
} catch (cause) {
|
||||
return unavailable(cause);
|
||||
}
|
||||
if (!record) {
|
||||
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||
'the current Project Tool snapshot is absent',
|
||||
);
|
||||
}
|
||||
return restoreCopilotFailureDiagnosisTrustedToolAuthority(
|
||||
plan,
|
||||
record.snapshot,
|
||||
);
|
||||
}
|
||||
|
||||
async function assertArtifacts(
|
||||
plan: Awaited<ReturnType<typeof durablePlan>>,
|
||||
dependencies: CopilotFailureDiagnosisToolExecutionDependencies,
|
||||
): Promise<void> {
|
||||
let input;
|
||||
let preview;
|
||||
try {
|
||||
[input, preview] = await Promise.all([
|
||||
dependencies.artifacts.findInput(plan.tool.invocationArtifact.artifactId),
|
||||
dependencies.artifacts.findPreview(plan.tool.previewArtifact.artifactId),
|
||||
]);
|
||||
} catch (cause) {
|
||||
return unavailable(cause);
|
||||
}
|
||||
if (
|
||||
!input ||
|
||||
!preview ||
|
||||
!sameValue(
|
||||
toolInvocationInputArtifactReference(input),
|
||||
plan.tool.invocationArtifact,
|
||||
) ||
|
||||
!sameValue(
|
||||
toolInvocationPreviewArtifactReference(preview),
|
||||
plan.tool.previewArtifact,
|
||||
) ||
|
||||
input.projectId !== plan.projectId ||
|
||||
input.actionRef !== plan.tool.actionRef ||
|
||||
preview.projectId !== plan.projectId ||
|
||||
preview.actionRef !== plan.tool.actionRef ||
|
||||
input.sealedAtMs !== plan.tool.sealedAtMs ||
|
||||
preview.sealedAtMs !== plan.tool.sealedAtMs
|
||||
) {
|
||||
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||
'the durable Tool invocation Artifacts changed',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function barrierMatches(
|
||||
barrierValue: ToolExecutionStartBarrierRecord,
|
||||
plan: Awaited<ReturnType<typeof durablePlan>>,
|
||||
startId: string,
|
||||
): Readonly<ToolExecutionStartBarrierRecord> {
|
||||
const barrier = normalizeToolExecutionStartBarrierRecord(barrierValue);
|
||||
if (
|
||||
barrier.startId !== startId ||
|
||||
barrier.projectId !== plan.projectId ||
|
||||
barrier.runId !== plan.runId ||
|
||||
barrier.stepRunId !== plan.toolStepRunId ||
|
||||
barrier.actionRef !== plan.tool.actionRef ||
|
||||
barrier.planDigest !== plan.tool.planDigest ||
|
||||
barrier.actionDigest !== plan.tool.actionDigest ||
|
||||
barrier.snapshotDigest !== plan.tool.snapshotDigest ||
|
||||
barrier.definitionDigest !== plan.tool.definitionDigest ||
|
||||
barrier.bindingDigest !== plan.tool.bindingDigest ||
|
||||
!sameValue(barrier.invocationArtifact, plan.tool.invocationArtifact) ||
|
||||
!sameValue(barrier.previewArtifact, plan.tool.previewArtifact) ||
|
||||
!sameValue(barrier.requestedBy, plan.requestedBySubject) ||
|
||||
barrier.profile !== 'cluster-control' ||
|
||||
!sameValue(barrier.policyFence, plan.policyFence) ||
|
||||
barrier.approvalRequestId !== null ||
|
||||
barrier.approvalDispatchId !== null ||
|
||||
barrier.approvalDispatchDigest !== null ||
|
||||
barrier.previousStepRunVersion !== 1 ||
|
||||
barrier.startedStepRunVersion !== 2
|
||||
) {
|
||||
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||
'the durable Tool start barrier changed',
|
||||
);
|
||||
}
|
||||
return barrier;
|
||||
}
|
||||
|
||||
async function prepareStart(
|
||||
plan: Awaited<ReturnType<typeof durablePlan>>,
|
||||
input: ExecuteCopilotFailureDiagnosisToolInput,
|
||||
authority: Awaited<ReturnType<typeof currentAuthority>>,
|
||||
dependencies: CopilotFailureDiagnosisToolExecutionDependencies,
|
||||
startId: string,
|
||||
): Promise<Readonly<ToolExecutionStartBarrierRecord>> {
|
||||
let existing;
|
||||
try {
|
||||
existing = await dependencies.barriers.findByStartId(startId);
|
||||
} catch (cause) {
|
||||
return unavailable(cause);
|
||||
}
|
||||
if (existing) return barrierMatches(existing, plan, startId);
|
||||
|
||||
await assertArtifacts(plan, dependencies);
|
||||
let stepRun;
|
||||
let run;
|
||||
try {
|
||||
[stepRun, run] = await Promise.all([
|
||||
dependencies.stepRuns.findById(plan.toolStepRunId),
|
||||
dependencies.runs.findRunById(plan.runId),
|
||||
]);
|
||||
} catch (cause) {
|
||||
return unavailable(cause);
|
||||
}
|
||||
if (
|
||||
!stepRun ||
|
||||
!run ||
|
||||
stepRun.id !== plan.toolStepRunId ||
|
||||
stepRun.runId !== plan.runId ||
|
||||
stepRun.kind !== 'tool' ||
|
||||
stepRun.status !== 'ready' ||
|
||||
stepRun.version !== 1 ||
|
||||
stepRun.definitionRef !== 'tool:qinglong.run.log.excerpt@1.0.0' ||
|
||||
stepRun.definitionDigest !== plan.tool.definitionDigest ||
|
||||
run.id !== plan.runId ||
|
||||
run.projectId !== plan.projectId ||
|
||||
run.status !== 'running' ||
|
||||
run.version !== run.eventSequence
|
||||
) {
|
||||
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||
'the Tool StepRun is not startable',
|
||||
);
|
||||
}
|
||||
const startedAtMs = clock(dependencies.now);
|
||||
if (
|
||||
startedAtMs < plan.plannedAtMs ||
|
||||
startedAtMs + authority.plan.timeoutSeconds * 1_000 > plan.deadlineAtMs
|
||||
) {
|
||||
throw new CopilotFailureDiagnosisToolExecutionDeadlineExceededError();
|
||||
}
|
||||
const evidence = createToolExecutionEvidenceBundle({
|
||||
traceId: traceIdentity(plan.planDigest),
|
||||
spanId: spanIdentity(plan.planDigest),
|
||||
projectId: plan.projectId,
|
||||
runId: plan.runId,
|
||||
stepRunId: plan.toolStepRunId,
|
||||
invocationPlanDigest: authority.plan.planDigest,
|
||||
bindingDigest: authority.binding.bindingDigest,
|
||||
adapterDigest: trustedToolContractIdentityDigest(authority.binding.adapter),
|
||||
redactionContractDigest: trustedToolContractIdentityDigest(
|
||||
authority.binding.redactionContract,
|
||||
),
|
||||
auditContractDigest: trustedToolContractIdentityDigest(
|
||||
authority.binding.auditContract,
|
||||
),
|
||||
audit: {
|
||||
eventId: auditEventIdentity(plan.planDigest),
|
||||
requestId: plan.requestId,
|
||||
operationId: TOOL_EXECUTION_START_AUDIT_OPERATION,
|
||||
projectId: plan.projectId,
|
||||
subject: plan.requestedBySubject,
|
||||
authenticationId: input.principal.authenticationId,
|
||||
outcome: 'allowed',
|
||||
reasons: ['copilot_failure_diagnosis_tool_start'],
|
||||
fence: plan.policyFence,
|
||||
occurredAtMs: startedAtMs,
|
||||
},
|
||||
createdAtMs: startedAtMs,
|
||||
});
|
||||
const admission = await admitTrustedToolExecution(
|
||||
authority.bindings,
|
||||
authority.plan,
|
||||
{
|
||||
principal: input.principal,
|
||||
profile: 'cluster-control',
|
||||
nowMs: startedAtMs,
|
||||
authorizer: input.authorizer,
|
||||
evidence: {
|
||||
stepRun: {
|
||||
id: stepRun.id,
|
||||
version: stepRun.version,
|
||||
digest: stepRun.stepRunDigest,
|
||||
},
|
||||
...toolExecutionAdmissionEvidence(evidence),
|
||||
},
|
||||
},
|
||||
);
|
||||
const mutationId = identity('cdstm', plan.planDigest);
|
||||
const eventId = identity('cdste', plan.planDigest);
|
||||
const mutation = transitionStepRunMutation(
|
||||
stepRun,
|
||||
{
|
||||
expectedVersion: stepRun.version,
|
||||
expectedDigest: stepRun.stepRunDigest,
|
||||
mutationId,
|
||||
to: 'running',
|
||||
atMs: startedAtMs,
|
||||
},
|
||||
{
|
||||
expectedRunVersion: run.version,
|
||||
expectedRunEventSequence: run.eventSequence,
|
||||
eventId,
|
||||
dedupeKey: eventId,
|
||||
actor: plan.requestedBySubject,
|
||||
},
|
||||
);
|
||||
const command = createToolExecutionStartCommand({
|
||||
startId,
|
||||
admission,
|
||||
evidence,
|
||||
stepRunMutation: mutation,
|
||||
});
|
||||
try {
|
||||
const prepared = await dependencies.barriers.prepare(command);
|
||||
if (!['created', 'existing'].includes(prepared.status)) {
|
||||
return unavailable();
|
||||
}
|
||||
return barrierMatches(prepared.barrier, plan, startId);
|
||||
} catch (cause) {
|
||||
let recovered;
|
||||
try {
|
||||
recovered = await dependencies.barriers.findByStartId(startId);
|
||||
} catch {
|
||||
throw cause;
|
||||
}
|
||||
if (recovered) return barrierMatches(recovered, plan, startId);
|
||||
throw cause;
|
||||
}
|
||||
}
|
||||
|
||||
function completionIdentities(planDigest: string) {
|
||||
return Object.freeze({
|
||||
success: Object.freeze({
|
||||
create() {
|
||||
return Object.freeze({
|
||||
artifactId: identity('cdra', planDigest),
|
||||
mutationId: identity('cdscm', planDigest),
|
||||
eventId: identity('cdsce', planDigest),
|
||||
});
|
||||
},
|
||||
}),
|
||||
failure: Object.freeze({
|
||||
create() {
|
||||
return Object.freeze({
|
||||
mutationId: identity('cdfcm', planDigest),
|
||||
eventId: identity('cdfce', planDigest),
|
||||
});
|
||||
},
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
async function unlockModel(
|
||||
plan: Awaited<ReturnType<typeof durablePlan>>,
|
||||
completion: Parameters<
|
||||
typeof createCopilotFailureDiagnosisToolUnlockCommand
|
||||
>[0]['completion'],
|
||||
dependencies: CopilotFailureDiagnosisToolExecutionDependencies,
|
||||
) {
|
||||
let existing;
|
||||
try {
|
||||
existing = await dependencies.unlocks.findByRequestId(plan.requestId);
|
||||
} catch (cause) {
|
||||
return unavailable(cause);
|
||||
}
|
||||
if (existing) {
|
||||
if (
|
||||
existing.planDigest !== plan.planDigest ||
|
||||
existing.toolCompletionDigest !== completion.completionDigest
|
||||
) {
|
||||
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||
'the durable model unlock changed',
|
||||
);
|
||||
}
|
||||
return Object.freeze({ status: 'existing' as const, receipt: existing });
|
||||
}
|
||||
let modelStepRun;
|
||||
let run;
|
||||
try {
|
||||
[modelStepRun, run] = await Promise.all([
|
||||
dependencies.stepRuns.findById(plan.modelStepRunId),
|
||||
dependencies.runs.findRunById(plan.runId),
|
||||
]);
|
||||
} catch (cause) {
|
||||
return unavailable(cause);
|
||||
}
|
||||
if (!modelStepRun || !run) return unavailable();
|
||||
const command = createCopilotFailureDiagnosisToolUnlockCommand({
|
||||
plan,
|
||||
completion,
|
||||
modelStepRun,
|
||||
run: {
|
||||
id: run.id,
|
||||
projectId: run.projectId,
|
||||
status: run.status,
|
||||
version: run.version,
|
||||
eventSequence: run.eventSequence,
|
||||
},
|
||||
});
|
||||
try {
|
||||
return await dependencies.unlocks.commit(command);
|
||||
} catch (cause) {
|
||||
let recovered;
|
||||
try {
|
||||
recovered = await dependencies.unlocks.findByRequestId(plan.requestId);
|
||||
} catch {
|
||||
throw cause;
|
||||
}
|
||||
if (
|
||||
recovered &&
|
||||
recovered.planDigest === plan.planDigest &&
|
||||
recovered.toolCompletionDigest === completion.completionDigest
|
||||
) {
|
||||
return Object.freeze({
|
||||
status: 'existing' as const,
|
||||
receipt: recovered,
|
||||
});
|
||||
}
|
||||
throw cause;
|
||||
}
|
||||
}
|
||||
|
||||
export async function executeCopilotFailureDiagnosisTool(
|
||||
input: ExecuteCopilotFailureDiagnosisToolInput,
|
||||
dependencies: CopilotFailureDiagnosisToolExecutionDependencies,
|
||||
): Promise<Readonly<CopilotFailureDiagnosisToolExecutionResult>> {
|
||||
assertDependencies(dependencies);
|
||||
if (
|
||||
!input ||
|
||||
typeof input !== 'object' ||
|
||||
Array.isArray(input) ||
|
||||
typeof input.requestId !== 'string' ||
|
||||
typeof input.authorizer?.authorize !== 'function'
|
||||
) {
|
||||
return invalid('input is invalid');
|
||||
}
|
||||
const plan = await durablePlan(input.requestId, dependencies);
|
||||
const authority = await currentAuthority(plan, dependencies);
|
||||
const startId = identity('cds', plan.planDigest);
|
||||
const barrier = await prepareStart(
|
||||
plan,
|
||||
input,
|
||||
authority,
|
||||
dependencies,
|
||||
startId,
|
||||
);
|
||||
const definitions = authority.bindings.definitionRegistry();
|
||||
const adapters = new TrustedToolExecutionAdapterRegistry(authority.bindings, [
|
||||
new BuiltInRunLogExcerptToolAdapter(
|
||||
authority.binding,
|
||||
'cluster-control',
|
||||
definitions,
|
||||
dependencies.logs,
|
||||
),
|
||||
]);
|
||||
const ids = completionIdentities(plan.planDigest);
|
||||
const completed = await executeAndCompleteTrustedTool(barrier.startId, {
|
||||
barriers: dependencies.barriers,
|
||||
artifacts: dependencies.artifacts,
|
||||
keys: dependencies.invocationKeys,
|
||||
adapters,
|
||||
completions: dependencies.completions,
|
||||
failureCompletions: dependencies.failureCompletions,
|
||||
stepRuns: dependencies.stepRuns,
|
||||
runs: dependencies.runs,
|
||||
resultKeyCatalog: dependencies.resultKeyCatalog,
|
||||
resultRekeys: dependencies.resultRekeys,
|
||||
resultKeys: dependencies.resultKeys,
|
||||
identities: ids.success,
|
||||
failureIdentities: ids.failure,
|
||||
...(dependencies.now === undefined ? {} : { now: dependencies.now }),
|
||||
...(dependencies.nonceFactory === undefined
|
||||
? {}
|
||||
: { nonceFactory: dependencies.nonceFactory }),
|
||||
});
|
||||
if (completed.outcome !== 'succeeded') {
|
||||
return Object.freeze({
|
||||
outcome: completed.outcome,
|
||||
completionStatus: completed.status,
|
||||
unlockStatus: null,
|
||||
});
|
||||
}
|
||||
const unlock = await unlockModel(plan, completed.completion, dependencies);
|
||||
return Object.freeze({
|
||||
outcome: 'succeeded' as const,
|
||||
completionStatus: completed.status,
|
||||
unlockStatus: unlock.status,
|
||||
completion: completed.completion,
|
||||
unlock: unlock.receipt,
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
import {
|
||||
BUILTIN_RUN_LOG_EXCERPT_TIMEOUT_SECONDS,
|
||||
BUILTIN_RUN_LOG_EXCERPT_TOOL,
|
||||
createBuiltInRunLogExcerptToolHandlerBinding,
|
||||
} from '@qinglong/runtime-core/builtin-run-log-excerpt-tool';
|
||||
import {
|
||||
normalizeProjectToolDefinitionSnapshot,
|
||||
type ProjectToolDefinitionSnapshot,
|
||||
} from '@qinglong/runtime-core/project-tool-definition-snapshot';
|
||||
import {
|
||||
TrustedToolHandlerBindingRegistry,
|
||||
normalizeTrustedToolInvocationPlan,
|
||||
type TrustedToolHandlerBinding,
|
||||
type TrustedToolInvocationPlan,
|
||||
} from '@qinglong/runtime-core/trusted-tool-invocation';
|
||||
|
||||
import type { CopilotFailureDiagnosisExecutionPlan } from '../admission/contracts';
|
||||
import { normalizeCopilotFailureDiagnosisExecutionPlan } from '../admission/plan';
|
||||
import {
|
||||
CopilotFailureDiagnosisToolExecutionConflictError,
|
||||
InvalidCopilotFailureDiagnosisToolExecutionError,
|
||||
} from './contracts';
|
||||
|
||||
export interface CopilotFailureDiagnosisTrustedToolAuthority {
|
||||
readonly plan: Readonly<TrustedToolInvocationPlan>;
|
||||
readonly binding: Readonly<TrustedToolHandlerBinding>;
|
||||
readonly bindings: TrustedToolHandlerBindingRegistry;
|
||||
}
|
||||
|
||||
function invalid(message: string): never {
|
||||
throw new InvalidCopilotFailureDiagnosisToolExecutionError(message);
|
||||
}
|
||||
|
||||
export function restoreCopilotFailureDiagnosisTrustedToolAuthority(
|
||||
executionPlanValue: CopilotFailureDiagnosisExecutionPlan,
|
||||
snapshotValue: ProjectToolDefinitionSnapshot,
|
||||
): Readonly<CopilotFailureDiagnosisTrustedToolAuthority> {
|
||||
const executionPlan =
|
||||
normalizeCopilotFailureDiagnosisExecutionPlan(executionPlanValue);
|
||||
const snapshot = normalizeProjectToolDefinitionSnapshot(snapshotValue);
|
||||
if (
|
||||
snapshot.projectId !== executionPlan.projectId ||
|
||||
snapshot.snapshotDigest !== executionPlan.tool.snapshotDigest
|
||||
) {
|
||||
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||
'the current Project Tool snapshot changed',
|
||||
);
|
||||
}
|
||||
const binding = createBuiltInRunLogExcerptToolHandlerBinding(snapshot, [
|
||||
'cluster-control',
|
||||
]);
|
||||
if (
|
||||
binding.bindingDigest !== executionPlan.tool.bindingDigest ||
|
||||
binding.definitionDigest !== executionPlan.tool.definitionDigest
|
||||
) {
|
||||
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||
'the reviewed log Tool binding changed',
|
||||
);
|
||||
}
|
||||
const bindings = new TrustedToolHandlerBindingRegistry(snapshot, [binding]);
|
||||
let plan: Readonly<TrustedToolInvocationPlan>;
|
||||
try {
|
||||
plan = normalizeTrustedToolInvocationPlan(
|
||||
{
|
||||
schema: 'qinglong/trusted-tool-invocation-plan@v1',
|
||||
status: 'ready',
|
||||
actionType: 'tool.invoke',
|
||||
actionRef: executionPlan.tool.actionRef,
|
||||
projectId: executionPlan.projectId,
|
||||
requestedBy: executionPlan.requestedBySubject,
|
||||
tool: BUILTIN_RUN_LOG_EXCERPT_TOOL,
|
||||
permission: 'tool.call:qinglong.run.log.excerpt',
|
||||
requiredPermissions: ['artifact.read'],
|
||||
effect: 'read',
|
||||
risk: 'medium',
|
||||
policyFence: executionPlan.policyFence,
|
||||
profile: 'cluster-control',
|
||||
snapshotDigest: executionPlan.tool.snapshotDigest,
|
||||
definitionDigest: executionPlan.tool.definitionDigest,
|
||||
binding,
|
||||
timeoutSeconds: BUILTIN_RUN_LOG_EXCERPT_TIMEOUT_SECONDS,
|
||||
invocationArtifact: executionPlan.tool.invocationArtifact,
|
||||
invocationActionDigest: executionPlan.tool.invocationActionDigest,
|
||||
previewArtifact: executionPlan.tool.previewArtifact,
|
||||
actionDigest: executionPlan.tool.actionDigest,
|
||||
sealedAtMs: executionPlan.tool.sealedAtMs,
|
||||
planDigest: executionPlan.tool.planDigest,
|
||||
},
|
||||
bindings,
|
||||
);
|
||||
} catch (cause) {
|
||||
if (cause instanceof CopilotFailureDiagnosisToolExecutionConflictError) {
|
||||
throw cause;
|
||||
}
|
||||
return invalid('the admitted trusted Tool plan cannot be restored');
|
||||
}
|
||||
if (plan.planDigest !== executionPlan.tool.planDigest) {
|
||||
return invalid('the restored trusted Tool plan digest changed');
|
||||
}
|
||||
return Object.freeze({ plan, binding, bindings });
|
||||
}
|
||||
+548
@@ -0,0 +1,548 @@
|
||||
import { isDeepStrictEqual } from 'node:util';
|
||||
|
||||
import type {
|
||||
PostgresClient,
|
||||
PostgresPool,
|
||||
PostgresQueryable,
|
||||
} from '@qinglong/runtime-core';
|
||||
import {
|
||||
normalizeToolExecutionCompletionRecord,
|
||||
type ToolExecutionCompletionRecord,
|
||||
} from '@qinglong/runtime-core/tool-execution-completion';
|
||||
|
||||
import { POSTGRES_MODEL_INVOCATION_SCHEMA } from '../../../migration/modelInvocationMigration';
|
||||
import type { CopilotFailureDiagnosisExecutionPlan } from '../admission/contracts';
|
||||
import { normalizeCopilotFailureDiagnosisExecutionPlan } from '../admission/plan';
|
||||
import {
|
||||
CopilotFailureDiagnosisToolExecutionConflictError,
|
||||
CopilotFailureDiagnosisToolExecutionUnavailableError,
|
||||
type CopilotFailureDiagnosisToolUnlockCommand,
|
||||
type CopilotFailureDiagnosisToolUnlockReceipt,
|
||||
type CopilotFailureDiagnosisToolUnlockRepository,
|
||||
} from './contracts';
|
||||
import {
|
||||
normalizeCopilotFailureDiagnosisToolUnlockCommand,
|
||||
normalizeCopilotFailureDiagnosisToolUnlockReceipt,
|
||||
} from './unlockProtocol';
|
||||
|
||||
const UNLOCK_TABLE = 'copilot_failure_diagnosis_tool_unlocks';
|
||||
const RETRYABLE_SQL_STATES = new Set(['40001', '40P01']);
|
||||
const MAX_TRANSACTION_ATTEMPTS = 3;
|
||||
|
||||
type Row = Readonly<Record<string, unknown>>;
|
||||
|
||||
function unavailable(
|
||||
cause?: unknown,
|
||||
): CopilotFailureDiagnosisToolExecutionUnavailableError {
|
||||
return new CopilotFailureDiagnosisToolExecutionUnavailableError({
|
||||
cause: cause instanceof Error ? cause : undefined,
|
||||
});
|
||||
}
|
||||
|
||||
function sqlState(error: unknown): string | undefined {
|
||||
if (!error || typeof error !== 'object') return undefined;
|
||||
const value = (error as { code?: unknown }).code;
|
||||
return typeof value === 'string' ? value : undefined;
|
||||
}
|
||||
|
||||
function mapStorageError(error: unknown): Error {
|
||||
if (
|
||||
error instanceof CopilotFailureDiagnosisToolExecutionConflictError ||
|
||||
error instanceof CopilotFailureDiagnosisToolExecutionUnavailableError
|
||||
) {
|
||||
return error;
|
||||
}
|
||||
if (['23503', '23505', '23514'].includes(sqlState(error) ?? '')) {
|
||||
return new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||
'a durable Tool unlock identity is already bound',
|
||||
);
|
||||
}
|
||||
return unavailable(error);
|
||||
}
|
||||
|
||||
function text(row: Row, key: string): string {
|
||||
const value = row[key];
|
||||
if (typeof value !== 'string' || value.length === 0) throw unavailable();
|
||||
return value;
|
||||
}
|
||||
|
||||
function integer(row: Row, key: string): number {
|
||||
const value = row[key];
|
||||
if (typeof value === 'number' && Number.isSafeInteger(value)) return value;
|
||||
if (typeof value === 'string' && /^(0|[1-9]\d*)$/.test(value)) {
|
||||
const parsed = Number(value);
|
||||
if (Number.isSafeInteger(parsed)) return parsed;
|
||||
}
|
||||
throw unavailable();
|
||||
}
|
||||
|
||||
function jsonObject(value: unknown): Record<string, unknown> {
|
||||
let parsed = value;
|
||||
if (typeof value === 'string') {
|
||||
try {
|
||||
parsed = JSON.parse(value);
|
||||
} catch (cause) {
|
||||
throw unavailable(cause);
|
||||
}
|
||||
}
|
||||
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
|
||||
throw unavailable();
|
||||
}
|
||||
return parsed as Record<string, unknown>;
|
||||
}
|
||||
|
||||
function same(left: unknown, right: unknown): boolean {
|
||||
return isDeepStrictEqual(left, right);
|
||||
}
|
||||
|
||||
async function begin(client: PostgresClient): Promise<void> {
|
||||
await client.query('BEGIN ISOLATION LEVEL SERIALIZABLE');
|
||||
await client.query(`SELECT set_config('statement_timeout', $1, true)`, [
|
||||
'5s',
|
||||
]);
|
||||
await client.query(`SELECT set_config('lock_timeout', $1, true)`, ['2s']);
|
||||
await client.query(
|
||||
`SELECT set_config('idle_in_transaction_session_timeout', $1, true)`,
|
||||
['5s'],
|
||||
);
|
||||
}
|
||||
|
||||
async function rollback(client: PostgresClient): Promise<void> {
|
||||
try {
|
||||
await client.query('ROLLBACK');
|
||||
} catch {
|
||||
// Preserve the original transaction failure.
|
||||
}
|
||||
}
|
||||
|
||||
type StoredUnlock = Readonly<{
|
||||
receipt: Readonly<CopilotFailureDiagnosisToolUnlockReceipt>;
|
||||
commandDigest: string;
|
||||
}>;
|
||||
|
||||
const UNLOCK_SELECT = `
|
||||
unlock.request_id AS "requestId",
|
||||
unlock.plan_digest AS "planDigest",
|
||||
unlock.start_id AS "startId",
|
||||
unlock.tool_completion_digest AS "toolCompletionDigest",
|
||||
unlock.model_step_run_id AS "modelStepRunId",
|
||||
unlock.model_step_run_version AS "modelStepRunVersion",
|
||||
unlock.model_step_run_digest AS "modelStepRunDigest",
|
||||
unlock.model_mutation_id AS "modelMutationId",
|
||||
unlock.model_mutation_digest AS "modelMutationDigest",
|
||||
unlock.model_event_id AS "modelEventId",
|
||||
unlock.final_run_version AS "finalRunVersion",
|
||||
unlock.final_run_event_sequence AS "finalRunEventSequence",
|
||||
unlock.unlocked_at_ms AS "unlockedAtMs",
|
||||
unlock.receipt_digest AS "receiptDigest",
|
||||
unlock.command_digest AS "commandDigest",
|
||||
unlock.receipt_json AS "receiptJson",
|
||||
admission.plan_digest AS "joinedPlanDigest",
|
||||
completion.completion_digest AS "joinedCompletionDigest",
|
||||
mutation.mutation_digest AS "joinedMutationDigest",
|
||||
mutation.step_run_digest AS "joinedModelStepRunDigest",
|
||||
mutation.event_id AS "joinedEventId",
|
||||
mutation.event_sequence AS "joinedEventSequence",
|
||||
mutation.run_version AS "joinedRunVersion"`;
|
||||
|
||||
async function findRows(
|
||||
queryable: PostgresQueryable,
|
||||
where: string,
|
||||
values: readonly unknown[],
|
||||
): Promise<readonly Row[]> {
|
||||
const result = await queryable.query<Row>(
|
||||
`SELECT ${UNLOCK_SELECT}
|
||||
FROM "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${UNLOCK_TABLE}" AS unlock
|
||||
JOIN "${POSTGRES_MODEL_INVOCATION_SCHEMA}".
|
||||
"copilot_failure_diagnosis_admissions" AS admission
|
||||
ON admission.request_id = unlock.request_id
|
||||
JOIN "ql3"."tool_execution_completions" AS completion
|
||||
ON completion.start_id = unlock.start_id
|
||||
JOIN "ql3"."step_run_mutations" AS mutation
|
||||
ON mutation.mutation_id = unlock.model_mutation_id
|
||||
WHERE ${where}
|
||||
LIMIT 2`,
|
||||
values,
|
||||
);
|
||||
return result.rows;
|
||||
}
|
||||
|
||||
function storedUnlock(row: Row): StoredUnlock {
|
||||
let receipt: Readonly<CopilotFailureDiagnosisToolUnlockReceipt>;
|
||||
try {
|
||||
receipt = normalizeCopilotFailureDiagnosisToolUnlockReceipt(
|
||||
jsonObject(
|
||||
row.receiptJson,
|
||||
) as unknown as CopilotFailureDiagnosisToolUnlockReceipt,
|
||||
);
|
||||
} catch (cause) {
|
||||
throw unavailable(cause);
|
||||
}
|
||||
if (
|
||||
text(row, 'requestId') !== receipt.requestId ||
|
||||
text(row, 'planDigest') !== receipt.planDigest ||
|
||||
text(row, 'startId') !== receipt.startId ||
|
||||
text(row, 'toolCompletionDigest') !== receipt.toolCompletionDigest ||
|
||||
text(row, 'modelStepRunId') !== receipt.modelStepRunId ||
|
||||
integer(row, 'modelStepRunVersion') !== receipt.modelStepRunVersion ||
|
||||
text(row, 'modelStepRunDigest') !== receipt.modelStepRunDigest ||
|
||||
text(row, 'modelMutationId') !== receipt.modelMutationId ||
|
||||
text(row, 'modelMutationDigest') !== receipt.modelMutationDigest ||
|
||||
text(row, 'modelEventId') !== receipt.modelEventId ||
|
||||
integer(row, 'finalRunVersion') !== receipt.finalRunVersion ||
|
||||
integer(row, 'finalRunEventSequence') !== receipt.finalRunEventSequence ||
|
||||
integer(row, 'unlockedAtMs') !== receipt.unlockedAtMs ||
|
||||
text(row, 'receiptDigest') !== receipt.receiptDigest ||
|
||||
text(row, 'joinedPlanDigest') !== receipt.planDigest ||
|
||||
text(row, 'joinedCompletionDigest') !== receipt.toolCompletionDigest ||
|
||||
text(row, 'joinedMutationDigest') !== receipt.modelMutationDigest ||
|
||||
text(row, 'joinedModelStepRunDigest') !== receipt.modelStepRunDigest ||
|
||||
text(row, 'joinedEventId') !== receipt.modelEventId ||
|
||||
integer(row, 'joinedEventSequence') !== receipt.finalRunEventSequence ||
|
||||
integer(row, 'joinedRunVersion') !== receipt.finalRunVersion
|
||||
) {
|
||||
throw unavailable();
|
||||
}
|
||||
return Object.freeze({
|
||||
receipt,
|
||||
commandDigest: text(row, 'commandDigest'),
|
||||
});
|
||||
}
|
||||
|
||||
async function updateModelStepRun(
|
||||
client: PostgresClient,
|
||||
command: Readonly<CopilotFailureDiagnosisToolUnlockCommand>,
|
||||
): Promise<void> {
|
||||
const mutation = command.modelStepRunMutation;
|
||||
const step = mutation.stepRun;
|
||||
const result = await client.query(
|
||||
`UPDATE "ql3"."step_runs"
|
||||
SET status = $1, version = $2, attempt_count = $3,
|
||||
output_ref = $4, approval_request_id = $5, ready_at_ms = $6,
|
||||
started_at_ms = $7, finished_at_ms = $8, result_code = $9,
|
||||
error_summary = $10, updated_at_ms = $11,
|
||||
last_mutation_id = $12, step_run_digest = $13,
|
||||
step_run_json = $14::jsonb
|
||||
WHERE id = $15 AND run_id = $16 AND version = $17
|
||||
AND step_run_digest = $18 AND status = $19`,
|
||||
[
|
||||
step.status,
|
||||
step.version,
|
||||
step.attemptCount,
|
||||
step.outputRef,
|
||||
step.approvalRequestId,
|
||||
step.readyAtMs,
|
||||
step.startedAtMs,
|
||||
step.finishedAtMs,
|
||||
step.resultCode,
|
||||
step.errorSummary,
|
||||
step.updatedAtMs,
|
||||
step.lastMutationId,
|
||||
step.stepRunDigest,
|
||||
JSON.stringify(step),
|
||||
step.id,
|
||||
step.runId,
|
||||
mutation.expectedStepRunVersion,
|
||||
mutation.expectedStepRunDigest,
|
||||
mutation.previousStatus,
|
||||
],
|
||||
);
|
||||
if (result.rowCount !== 1) {
|
||||
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||
'the Model StepRun unlock fence changed',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function updateRun(
|
||||
client: PostgresClient,
|
||||
command: Readonly<CopilotFailureDiagnosisToolUnlockCommand>,
|
||||
): Promise<void> {
|
||||
const mutation = command.modelStepRunMutation;
|
||||
const result = await client.query(
|
||||
`UPDATE "ql3"."runs"
|
||||
SET version = version + 1, event_sequence = event_sequence + 1
|
||||
WHERE id = $1 AND status = 'running'
|
||||
AND version = $2 AND event_sequence = $3`,
|
||||
[
|
||||
mutation.runId,
|
||||
mutation.expectedRunVersion,
|
||||
mutation.expectedRunEventSequence,
|
||||
],
|
||||
);
|
||||
if (result.rowCount !== 1) {
|
||||
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||
'the diagnosis Run unlock fence changed',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function insertEventAndMutation(
|
||||
client: PostgresClient,
|
||||
command: Readonly<CopilotFailureDiagnosisToolUnlockCommand>,
|
||||
): Promise<void> {
|
||||
const mutation = command.modelStepRunMutation;
|
||||
const event = mutation.event;
|
||||
await client.query(
|
||||
`INSERT INTO "ql3"."run_events" (
|
||||
id, run_id, sequence, type, dedupe_key, actor_type, actor_id,
|
||||
attempt_id, step_run_id, payload, created_at_ms
|
||||
) VALUES ($1, $2, $3, $4, $5, $6, $7, NULL, $8, $9::jsonb, $10)`,
|
||||
[
|
||||
event.id,
|
||||
event.runId,
|
||||
event.sequence,
|
||||
event.type,
|
||||
event.dedupeKey,
|
||||
event.actorType,
|
||||
event.actorId ?? null,
|
||||
mutation.stepRun.id,
|
||||
JSON.stringify(event.payload),
|
||||
event.createdAtMs,
|
||||
],
|
||||
);
|
||||
await client.query(
|
||||
`INSERT INTO "ql3"."step_run_mutations" (
|
||||
mutation_id, mutation_digest, run_id, step_run_id,
|
||||
step_run_digest, event_id, event_sequence, run_version,
|
||||
step_run_json, committed_at_ms
|
||||
) VALUES (
|
||||
$1, $2, $3, $4, $5, $6, $7, $8, $9::jsonb,
|
||||
floor(extract(epoch FROM transaction_timestamp()) * 1000)::bigint
|
||||
)`,
|
||||
[
|
||||
mutation.mutationId,
|
||||
mutation.mutationDigest,
|
||||
mutation.runId,
|
||||
mutation.stepRun.id,
|
||||
mutation.stepRun.stepRunDigest,
|
||||
mutation.event.id,
|
||||
mutation.event.sequence,
|
||||
mutation.expectedRunVersion + 1,
|
||||
JSON.stringify(mutation.stepRun),
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
async function insertUnlock(
|
||||
client: PostgresClient,
|
||||
command: Readonly<CopilotFailureDiagnosisToolUnlockCommand>,
|
||||
): Promise<void> {
|
||||
const receipt = command.receipt;
|
||||
await client.query(
|
||||
`INSERT INTO "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${UNLOCK_TABLE}" (
|
||||
request_id, plan_digest, run_id, start_id, tool_step_run_id,
|
||||
tool_completion_digest,
|
||||
model_step_run_id, model_step_run_version, model_step_run_digest,
|
||||
model_mutation_id, model_mutation_digest, model_event_id,
|
||||
final_run_version, final_run_event_sequence, unlocked_at_ms,
|
||||
receipt_digest, command_digest, receipt_json
|
||||
) VALUES (
|
||||
$1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13,
|
||||
$14, $15, $16, $17, $18::jsonb
|
||||
)`,
|
||||
[
|
||||
receipt.requestId,
|
||||
receipt.planDigest,
|
||||
receipt.runId,
|
||||
receipt.startId,
|
||||
receipt.toolStepRunId,
|
||||
receipt.toolCompletionDigest,
|
||||
receipt.modelStepRunId,
|
||||
receipt.modelStepRunVersion,
|
||||
receipt.modelStepRunDigest,
|
||||
receipt.modelMutationId,
|
||||
receipt.modelMutationDigest,
|
||||
receipt.modelEventId,
|
||||
receipt.finalRunVersion,
|
||||
receipt.finalRunEventSequence,
|
||||
receipt.unlockedAtMs,
|
||||
receipt.receiptDigest,
|
||||
command.commandDigest,
|
||||
JSON.stringify(receipt),
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
async function currentEvidence(
|
||||
client: PostgresClient,
|
||||
command: Readonly<CopilotFailureDiagnosisToolUnlockCommand>,
|
||||
): Promise<void> {
|
||||
const mutation = command.modelStepRunMutation;
|
||||
const result = await client.query<Row>(
|
||||
`SELECT admission.plan_json AS "planJson",
|
||||
completion.completion_json AS "completionJson",
|
||||
model_step.kind AS "modelKind",
|
||||
model_step.status AS "modelStatus",
|
||||
model_step.version AS "modelVersion",
|
||||
model_step.step_run_digest AS "modelDigest",
|
||||
run.project_id AS "projectId", run.status AS "runStatus",
|
||||
run.version AS "runVersion",
|
||||
run.event_sequence AS "runEventSequence"
|
||||
FROM "${POSTGRES_MODEL_INVOCATION_SCHEMA}".
|
||||
"copilot_failure_diagnosis_admissions" AS admission
|
||||
JOIN "ql3"."tool_execution_completions" AS completion
|
||||
ON completion.start_id = $2
|
||||
AND completion.run_id = admission.run_id
|
||||
AND completion.step_run_id = admission.tool_step_run_id
|
||||
JOIN "ql3"."step_runs" AS model_step
|
||||
ON model_step.run_id = admission.run_id
|
||||
AND model_step.id = admission.model_step_run_id
|
||||
JOIN "ql3"."runs" AS run ON run.id = admission.run_id
|
||||
WHERE admission.request_id = $1
|
||||
LIMIT 2
|
||||
FOR UPDATE OF model_step, run`,
|
||||
[command.plan.requestId, command.completion.startId],
|
||||
);
|
||||
const row = result.rows[0];
|
||||
let plan: Readonly<CopilotFailureDiagnosisExecutionPlan>;
|
||||
let completion: Readonly<ToolExecutionCompletionRecord>;
|
||||
try {
|
||||
plan = row
|
||||
? normalizeCopilotFailureDiagnosisExecutionPlan(
|
||||
jsonObject(
|
||||
row.planJson,
|
||||
) as unknown as CopilotFailureDiagnosisExecutionPlan,
|
||||
)
|
||||
: command.plan;
|
||||
completion = row
|
||||
? normalizeToolExecutionCompletionRecord(
|
||||
jsonObject(
|
||||
row.completionJson,
|
||||
) as unknown as ToolExecutionCompletionRecord,
|
||||
)
|
||||
: command.completion;
|
||||
} catch (cause) {
|
||||
throw unavailable(cause);
|
||||
}
|
||||
if (
|
||||
result.rows.length !== 1 ||
|
||||
!row ||
|
||||
!same(plan, command.plan) ||
|
||||
!same(completion, command.completion) ||
|
||||
text(row, 'modelKind') !== 'model' ||
|
||||
text(row, 'modelStatus') !== mutation.previousStatus ||
|
||||
integer(row, 'modelVersion') !== mutation.expectedStepRunVersion ||
|
||||
text(row, 'modelDigest') !== mutation.expectedStepRunDigest ||
|
||||
text(row, 'projectId') !== command.plan.projectId ||
|
||||
text(row, 'runStatus') !== 'running' ||
|
||||
integer(row, 'runVersion') !== mutation.expectedRunVersion ||
|
||||
integer(row, 'runEventSequence') !== mutation.expectedRunEventSequence
|
||||
) {
|
||||
throw new CopilotFailureDiagnosisToolExecutionConflictError(
|
||||
'the admitted Tool completion or Model StepRun fence changed',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
export class PostgresCopilotFailureDiagnosisToolUnlockRepository
|
||||
implements CopilotFailureDiagnosisToolUnlockRepository
|
||||
{
|
||||
constructor(private readonly pool: PostgresPool) {
|
||||
if (
|
||||
!pool ||
|
||||
typeof pool.query !== 'function' ||
|
||||
typeof pool.connect !== 'function'
|
||||
) {
|
||||
throw unavailable();
|
||||
}
|
||||
}
|
||||
|
||||
async findByRequestId(
|
||||
requestId: string,
|
||||
): Promise<Readonly<CopilotFailureDiagnosisToolUnlockReceipt> | null> {
|
||||
if (
|
||||
typeof requestId !== 'string' ||
|
||||
!/^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/.test(requestId)
|
||||
) {
|
||||
throw new TypeError('Tool unlock request id is invalid');
|
||||
}
|
||||
try {
|
||||
const rows = await findRows(this.pool, 'unlock.request_id = $1', [
|
||||
requestId,
|
||||
]);
|
||||
if (rows.length > 1) throw unavailable();
|
||||
return rows[0] ? storedUnlock(rows[0]).receipt : null;
|
||||
} catch (error) {
|
||||
throw mapStorageError(error);
|
||||
}
|
||||
}
|
||||
|
||||
async commit(commandValue: CopilotFailureDiagnosisToolUnlockCommand): Promise<
|
||||
Readonly<{
|
||||
status: 'created' | 'existing';
|
||||
receipt: Readonly<CopilotFailureDiagnosisToolUnlockReceipt>;
|
||||
}>
|
||||
> {
|
||||
const command =
|
||||
normalizeCopilotFailureDiagnosisToolUnlockCommand(commandValue);
|
||||
for (let attempt = 0; attempt < MAX_TRANSACTION_ATTEMPTS; attempt += 1) {
|
||||
let client: PostgresClient;
|
||||
try {
|
||||
client = await this.pool.connect();
|
||||
} catch (cause) {
|
||||
throw unavailable(cause);
|
||||
}
|
||||
let began = false;
|
||||
try {
|
||||
await begin(client);
|
||||
began = true;
|
||||
const existingRows = await findRows(
|
||||
client,
|
||||
`unlock.request_id = $1
|
||||
OR unlock.tool_completion_digest = $2
|
||||
OR unlock.model_mutation_id = $3
|
||||
OR unlock.model_event_id = $4`,
|
||||
[
|
||||
command.receipt.requestId,
|
||||
command.receipt.toolCompletionDigest,
|
||||
command.receipt.modelMutationId,
|
||||
command.receipt.modelEventId,
|
||||
],
|
||||
);
|
||||
if (existingRows.length > 1) {
|
||||
throw new CopilotFailureDiagnosisToolExecutionConflictError();
|
||||
}
|
||||
if (existingRows[0]) {
|
||||
const stored = storedUnlock(existingRows[0]);
|
||||
if (
|
||||
stored.commandDigest !== command.commandDigest ||
|
||||
!same(stored.receipt, command.receipt)
|
||||
) {
|
||||
throw new CopilotFailureDiagnosisToolExecutionConflictError();
|
||||
}
|
||||
await client.query('COMMIT');
|
||||
began = false;
|
||||
return Object.freeze({
|
||||
status: 'existing' as const,
|
||||
receipt: stored.receipt,
|
||||
});
|
||||
}
|
||||
|
||||
await currentEvidence(client, command);
|
||||
await updateModelStepRun(client, command);
|
||||
await updateRun(client, command);
|
||||
await insertEventAndMutation(client, command);
|
||||
await insertUnlock(client, command);
|
||||
await client.query('COMMIT');
|
||||
began = false;
|
||||
return Object.freeze({
|
||||
status: 'created' as const,
|
||||
receipt: command.receipt,
|
||||
});
|
||||
} catch (error) {
|
||||
if (began) await rollback(client);
|
||||
if (
|
||||
RETRYABLE_SQL_STATES.has(sqlState(error) ?? '') &&
|
||||
attempt + 1 < MAX_TRANSACTION_ATTEMPTS
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
throw mapStorageError(error);
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
}
|
||||
throw unavailable();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,424 @@
|
||||
import { Buffer } from 'node:buffer';
|
||||
import { createHash } from 'node:crypto';
|
||||
|
||||
import {
|
||||
normalizeStepRunMutation,
|
||||
normalizeStepRunRecord,
|
||||
transitionStepRunMutation,
|
||||
type StepRunRecord,
|
||||
} from '@qinglong/runtime-core/step-run';
|
||||
import {
|
||||
normalizeToolExecutionCompletionRecord,
|
||||
type ToolExecutionCompletionRecord,
|
||||
} from '@qinglong/runtime-core/tool-execution-completion';
|
||||
|
||||
import type { CopilotFailureDiagnosisExecutionPlan } from '../admission/contracts';
|
||||
import { normalizeCopilotFailureDiagnosisExecutionPlan } from '../admission/plan';
|
||||
import {
|
||||
COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA,
|
||||
COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA,
|
||||
InvalidCopilotFailureDiagnosisToolExecutionError,
|
||||
MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_BYTES,
|
||||
MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_BYTES,
|
||||
type CopilotFailureDiagnosisRunAuthority,
|
||||
type CopilotFailureDiagnosisToolUnlockCommand,
|
||||
type CopilotFailureDiagnosisToolUnlockReceipt,
|
||||
} from './contracts';
|
||||
|
||||
const RECEIPT_DIGEST_DOMAIN = Buffer.from(
|
||||
'qinglong/copilot-failure-diagnosis-tool-unlock-receipt-digest@v1\0',
|
||||
'utf8',
|
||||
);
|
||||
const COMMAND_DIGEST_DOMAIN = Buffer.from(
|
||||
'qinglong/copilot-failure-diagnosis-tool-unlock-command-digest@v1\0',
|
||||
'utf8',
|
||||
);
|
||||
const IDENTITY_DOMAIN = Buffer.from(
|
||||
'qinglong/copilot-failure-diagnosis-tool-unlock-identity@v1\0',
|
||||
'utf8',
|
||||
);
|
||||
const DIGEST_PATTERN = /^[0-9a-f]{64}$/;
|
||||
const IDENTITY_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
|
||||
|
||||
function invalid(message: string): never {
|
||||
throw new InvalidCopilotFailureDiagnosisToolExecutionError(message);
|
||||
}
|
||||
|
||||
function hash(domain: Uint8Array, value: unknown): string {
|
||||
return createHash('sha256')
|
||||
.update(domain)
|
||||
.update(JSON.stringify(value))
|
||||
.digest('hex');
|
||||
}
|
||||
|
||||
function dataRecord(value: unknown, label: string): Record<string, unknown> {
|
||||
if (
|
||||
!value ||
|
||||
typeof value !== 'object' ||
|
||||
Array.isArray(value) ||
|
||||
Object.getPrototypeOf(value) !== Object.prototype
|
||||
) {
|
||||
return invalid(`${label} must be a plain object`);
|
||||
}
|
||||
return value as Record<string, unknown>;
|
||||
}
|
||||
|
||||
function exactKeys(
|
||||
value: object,
|
||||
keys: readonly string[],
|
||||
label: string,
|
||||
): void {
|
||||
const actual = Object.keys(value).sort();
|
||||
const expected = [...keys].sort();
|
||||
if (
|
||||
actual.length !== expected.length ||
|
||||
actual.some((key, index) => key !== expected[index])
|
||||
) {
|
||||
invalid(`${label} shape is invalid`);
|
||||
}
|
||||
}
|
||||
|
||||
function identity(value: unknown, label: string): string {
|
||||
if (typeof value !== 'string' || !IDENTITY_PATTERN.test(value)) {
|
||||
return invalid(`${label} is invalid`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function digest(value: unknown, label: string): string {
|
||||
if (typeof value !== 'string' || !DIGEST_PATTERN.test(value)) {
|
||||
return invalid(`${label} is invalid`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function integer(value: unknown, minimum: number, label: string): number {
|
||||
if (!Number.isSafeInteger(value) || (value as number) < minimum) {
|
||||
return invalid(`${label} is invalid`);
|
||||
}
|
||||
return value as number;
|
||||
}
|
||||
|
||||
function evidenceIdentity(
|
||||
prefix: 'cdum' | 'cdue',
|
||||
planDigest: string,
|
||||
completionDigest: string,
|
||||
): string {
|
||||
const maximumDigestLength = 35 - prefix.length;
|
||||
return `${prefix}:${hash(IDENTITY_DOMAIN, {
|
||||
prefix,
|
||||
planDigest,
|
||||
completionDigest,
|
||||
}).slice(0, maximumDigestLength)}`;
|
||||
}
|
||||
|
||||
export function copilotFailureDiagnosisToolUnlockReceiptDigest(
|
||||
value: Omit<CopilotFailureDiagnosisToolUnlockReceipt, 'receiptDigest'>,
|
||||
): string {
|
||||
return hash(RECEIPT_DIGEST_DOMAIN, value);
|
||||
}
|
||||
|
||||
export function normalizeCopilotFailureDiagnosisToolUnlockReceipt(
|
||||
value: CopilotFailureDiagnosisToolUnlockReceipt,
|
||||
): Readonly<CopilotFailureDiagnosisToolUnlockReceipt> {
|
||||
const candidate = dataRecord(value, 'Tool unlock receipt');
|
||||
exactKeys(
|
||||
candidate,
|
||||
[
|
||||
'barrierDigest',
|
||||
'finalRunEventSequence',
|
||||
'finalRunVersion',
|
||||
'modelEventId',
|
||||
'modelMutationDigest',
|
||||
'modelMutationId',
|
||||
'modelStepRunDigest',
|
||||
'modelStepRunId',
|
||||
'modelStepRunVersion',
|
||||
'planDigest',
|
||||
'receiptDigest',
|
||||
'requestId',
|
||||
'resultArtifact',
|
||||
'runId',
|
||||
'schema',
|
||||
'startId',
|
||||
'toolCompletionDigest',
|
||||
'toolStepRunId',
|
||||
'unlockedAtMs',
|
||||
],
|
||||
'Tool unlock receipt',
|
||||
);
|
||||
if (value.schema !== COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA) {
|
||||
return invalid('Tool unlock receipt schema is unsupported');
|
||||
}
|
||||
const artifact = dataRecord(value.resultArtifact, 'result Artifact');
|
||||
exactKeys(
|
||||
artifact,
|
||||
['artifactDigest', 'artifactId', 'executionResultDigest', 'outputDigest'],
|
||||
'result Artifact',
|
||||
);
|
||||
const unsigned = Object.freeze({
|
||||
schema: COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA,
|
||||
requestId: identity(value.requestId, 'request id'),
|
||||
planDigest: digest(value.planDigest, 'plan digest'),
|
||||
runId: identity(value.runId, 'Run id'),
|
||||
startId: identity(value.startId, 'start id'),
|
||||
barrierDigest: digest(value.barrierDigest, 'barrier digest'),
|
||||
toolStepRunId: identity(value.toolStepRunId, 'Tool StepRun id'),
|
||||
toolCompletionDigest: digest(
|
||||
value.toolCompletionDigest,
|
||||
'Tool completion digest',
|
||||
),
|
||||
resultArtifact: Object.freeze({
|
||||
artifactId: identity(value.resultArtifact.artifactId, 'Artifact id'),
|
||||
artifactDigest: digest(
|
||||
value.resultArtifact.artifactDigest,
|
||||
'Artifact digest',
|
||||
),
|
||||
outputDigest: digest(value.resultArtifact.outputDigest, 'output digest'),
|
||||
executionResultDigest: digest(
|
||||
value.resultArtifact.executionResultDigest,
|
||||
'execution result digest',
|
||||
),
|
||||
}),
|
||||
modelStepRunId: identity(value.modelStepRunId, 'model StepRun id'),
|
||||
modelStepRunVersion: integer(
|
||||
value.modelStepRunVersion,
|
||||
2,
|
||||
'model StepRun version',
|
||||
),
|
||||
modelStepRunDigest: digest(
|
||||
value.modelStepRunDigest,
|
||||
'model StepRun digest',
|
||||
),
|
||||
modelMutationId: identity(value.modelMutationId, 'model mutation id'),
|
||||
modelMutationDigest: digest(
|
||||
value.modelMutationDigest,
|
||||
'model mutation digest',
|
||||
),
|
||||
modelEventId: identity(value.modelEventId, 'model event id'),
|
||||
finalRunVersion: integer(value.finalRunVersion, 1, 'final Run version'),
|
||||
finalRunEventSequence: integer(
|
||||
value.finalRunEventSequence,
|
||||
1,
|
||||
'final Run event sequence',
|
||||
),
|
||||
unlockedAtMs: integer(value.unlockedAtMs, 0, 'unlock time'),
|
||||
} satisfies Omit<CopilotFailureDiagnosisToolUnlockReceipt, 'receiptDigest'>);
|
||||
if (unsigned.finalRunVersion !== unsigned.finalRunEventSequence) {
|
||||
return invalid('Tool unlock Run fence is invalid');
|
||||
}
|
||||
const receiptDigest = digest(value.receiptDigest, 'receipt digest');
|
||||
if (
|
||||
copilotFailureDiagnosisToolUnlockReceiptDigest(unsigned) !== receiptDigest
|
||||
) {
|
||||
return invalid('Tool unlock receipt digest does not match');
|
||||
}
|
||||
const normalized = Object.freeze({ ...unsigned, receiptDigest });
|
||||
if (
|
||||
Buffer.byteLength(JSON.stringify(normalized), 'utf8') >
|
||||
MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_BYTES
|
||||
) {
|
||||
return invalid('Tool unlock receipt exceeds its byte budget');
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function commandUnsigned(
|
||||
value: Readonly<CopilotFailureDiagnosisToolUnlockCommand>,
|
||||
): Omit<CopilotFailureDiagnosisToolUnlockCommand, 'commandDigest'> {
|
||||
const { commandDigest: _commandDigest, ...unsigned } = value;
|
||||
return unsigned;
|
||||
}
|
||||
|
||||
function validateCommandBindings(
|
||||
plan: Readonly<CopilotFailureDiagnosisExecutionPlan>,
|
||||
completion: Readonly<ToolExecutionCompletionRecord>,
|
||||
mutation: ReturnType<typeof normalizeStepRunMutation>,
|
||||
receipt: Readonly<CopilotFailureDiagnosisToolUnlockReceipt>,
|
||||
): void {
|
||||
if (
|
||||
completion.projectId !== plan.projectId ||
|
||||
completion.runId !== plan.runId ||
|
||||
completion.stepRunId !== plan.toolStepRunId ||
|
||||
mutation.runId !== plan.runId ||
|
||||
mutation.previousStatus !== 'pending' ||
|
||||
mutation.stepRun.id !== plan.modelStepRunId ||
|
||||
mutation.stepRun.runId !== plan.runId ||
|
||||
mutation.stepRun.parentStepRunId !== plan.toolStepRunId ||
|
||||
mutation.stepRun.kind !== 'model' ||
|
||||
mutation.stepRun.status !== 'ready' ||
|
||||
mutation.stepRun.inputRef !== `tool-result-step:${plan.toolStepRunId}` ||
|
||||
mutation.stepRun.updatedAtMs !== completion.completedAtMs ||
|
||||
receipt.requestId !== plan.requestId ||
|
||||
receipt.planDigest !== plan.planDigest ||
|
||||
receipt.runId !== plan.runId ||
|
||||
receipt.startId !== completion.startId ||
|
||||
receipt.barrierDigest !== completion.barrierDigest ||
|
||||
receipt.toolStepRunId !== completion.stepRunId ||
|
||||
receipt.toolCompletionDigest !== completion.completionDigest ||
|
||||
JSON.stringify(receipt.resultArtifact) !==
|
||||
JSON.stringify(completion.resultArtifact) ||
|
||||
receipt.modelStepRunId !== mutation.stepRun.id ||
|
||||
receipt.modelStepRunVersion !== mutation.stepRun.version ||
|
||||
receipt.modelStepRunDigest !== mutation.stepRun.stepRunDigest ||
|
||||
receipt.modelMutationId !== mutation.mutationId ||
|
||||
receipt.modelMutationDigest !== mutation.mutationDigest ||
|
||||
receipt.modelEventId !== mutation.event.id ||
|
||||
receipt.finalRunVersion !== mutation.expectedRunVersion + 1 ||
|
||||
receipt.finalRunEventSequence !== mutation.expectedRunEventSequence + 1 ||
|
||||
receipt.unlockedAtMs !== mutation.stepRun.updatedAtMs
|
||||
) {
|
||||
return invalid('Tool unlock command bindings are inconsistent');
|
||||
}
|
||||
}
|
||||
|
||||
export function normalizeCopilotFailureDiagnosisToolUnlockCommand(
|
||||
value: CopilotFailureDiagnosisToolUnlockCommand,
|
||||
): Readonly<CopilotFailureDiagnosisToolUnlockCommand> {
|
||||
const candidate = dataRecord(value, 'Tool unlock command');
|
||||
exactKeys(
|
||||
candidate,
|
||||
[
|
||||
'commandDigest',
|
||||
'completion',
|
||||
'modelStepRunMutation',
|
||||
'plan',
|
||||
'receipt',
|
||||
'schema',
|
||||
],
|
||||
'Tool unlock command',
|
||||
);
|
||||
if (value.schema !== COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA) {
|
||||
return invalid('Tool unlock command schema is unsupported');
|
||||
}
|
||||
const plan = normalizeCopilotFailureDiagnosisExecutionPlan(value.plan);
|
||||
const completion = normalizeToolExecutionCompletionRecord(value.completion);
|
||||
const modelStepRunMutation = normalizeStepRunMutation(
|
||||
value.modelStepRunMutation,
|
||||
);
|
||||
const receipt = normalizeCopilotFailureDiagnosisToolUnlockReceipt(
|
||||
value.receipt,
|
||||
);
|
||||
validateCommandBindings(plan, completion, modelStepRunMutation, receipt);
|
||||
const unsigned = Object.freeze({
|
||||
schema: COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA,
|
||||
plan,
|
||||
completion,
|
||||
modelStepRunMutation,
|
||||
receipt,
|
||||
} satisfies Omit<CopilotFailureDiagnosisToolUnlockCommand, 'commandDigest'>);
|
||||
const commandDigest = digest(value.commandDigest, 'command digest');
|
||||
if (hash(COMMAND_DIGEST_DOMAIN, unsigned) !== commandDigest) {
|
||||
return invalid('Tool unlock command digest does not match');
|
||||
}
|
||||
const normalized = Object.freeze({ ...unsigned, commandDigest });
|
||||
if (
|
||||
Buffer.byteLength(JSON.stringify(normalized), 'utf8') >
|
||||
MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_BYTES
|
||||
) {
|
||||
return invalid('Tool unlock command exceeds its byte budget');
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
|
||||
export function createCopilotFailureDiagnosisToolUnlockCommand(input: {
|
||||
readonly plan: CopilotFailureDiagnosisExecutionPlan;
|
||||
readonly completion: ToolExecutionCompletionRecord;
|
||||
readonly modelStepRun: StepRunRecord;
|
||||
readonly run: CopilotFailureDiagnosisRunAuthority;
|
||||
}): Readonly<CopilotFailureDiagnosisToolUnlockCommand> {
|
||||
const plan = normalizeCopilotFailureDiagnosisExecutionPlan(input.plan);
|
||||
const completion = normalizeToolExecutionCompletionRecord(input.completion);
|
||||
const modelStepRun = normalizeStepRunRecord(input.modelStepRun);
|
||||
const run = dataRecord(input.run, 'Run authority');
|
||||
exactKeys(
|
||||
run,
|
||||
['eventSequence', 'id', 'projectId', 'status', 'version'],
|
||||
'Run authority',
|
||||
);
|
||||
if (
|
||||
input.run.id !== plan.runId ||
|
||||
input.run.projectId !== plan.projectId ||
|
||||
input.run.status !== 'running' ||
|
||||
!Number.isSafeInteger(input.run.version) ||
|
||||
input.run.version < 1 ||
|
||||
!Number.isSafeInteger(input.run.eventSequence) ||
|
||||
input.run.eventSequence !== input.run.version ||
|
||||
completion.projectId !== plan.projectId ||
|
||||
completion.runId !== plan.runId ||
|
||||
completion.stepRunId !== plan.toolStepRunId ||
|
||||
modelStepRun.id !== plan.modelStepRunId ||
|
||||
modelStepRun.runId !== plan.runId ||
|
||||
modelStepRun.parentStepRunId !== plan.toolStepRunId ||
|
||||
modelStepRun.kind !== 'model' ||
|
||||
modelStepRun.status !== 'pending'
|
||||
) {
|
||||
return invalid('Tool completion cannot unlock the model StepRun');
|
||||
}
|
||||
const mutationId = evidenceIdentity(
|
||||
'cdum',
|
||||
plan.planDigest,
|
||||
completion.completionDigest,
|
||||
);
|
||||
const eventId = evidenceIdentity(
|
||||
'cdue',
|
||||
plan.planDigest,
|
||||
completion.completionDigest,
|
||||
);
|
||||
const modelStepRunMutation = transitionStepRunMutation(
|
||||
modelStepRun,
|
||||
{
|
||||
expectedVersion: modelStepRun.version,
|
||||
expectedDigest: modelStepRun.stepRunDigest,
|
||||
mutationId,
|
||||
to: 'ready',
|
||||
atMs: completion.completedAtMs,
|
||||
},
|
||||
{
|
||||
expectedRunVersion: input.run.version,
|
||||
expectedRunEventSequence: input.run.eventSequence,
|
||||
eventId,
|
||||
dedupeKey: eventId,
|
||||
actor: Object.freeze({
|
||||
type: 'system' as const,
|
||||
id: 'copilot-runtime',
|
||||
}),
|
||||
},
|
||||
);
|
||||
const receiptUnsigned = Object.freeze({
|
||||
schema: COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA,
|
||||
requestId: plan.requestId,
|
||||
planDigest: plan.planDigest,
|
||||
runId: plan.runId,
|
||||
startId: completion.startId,
|
||||
barrierDigest: completion.barrierDigest,
|
||||
toolStepRunId: completion.stepRunId,
|
||||
toolCompletionDigest: completion.completionDigest,
|
||||
resultArtifact: completion.resultArtifact,
|
||||
modelStepRunId: modelStepRunMutation.stepRun.id,
|
||||
modelStepRunVersion: modelStepRunMutation.stepRun.version,
|
||||
modelStepRunDigest: modelStepRunMutation.stepRun.stepRunDigest,
|
||||
modelMutationId: modelStepRunMutation.mutationId,
|
||||
modelMutationDigest: modelStepRunMutation.mutationDigest,
|
||||
modelEventId: modelStepRunMutation.event.id,
|
||||
finalRunVersion: modelStepRunMutation.expectedRunVersion + 1,
|
||||
finalRunEventSequence: modelStepRunMutation.expectedRunEventSequence + 1,
|
||||
unlockedAtMs: completion.completedAtMs,
|
||||
} satisfies Omit<CopilotFailureDiagnosisToolUnlockReceipt, 'receiptDigest'>);
|
||||
const receipt = normalizeCopilotFailureDiagnosisToolUnlockReceipt({
|
||||
...receiptUnsigned,
|
||||
receiptDigest:
|
||||
copilotFailureDiagnosisToolUnlockReceiptDigest(receiptUnsigned),
|
||||
});
|
||||
const unsigned = Object.freeze({
|
||||
schema: COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA,
|
||||
plan,
|
||||
completion,
|
||||
modelStepRunMutation,
|
||||
receipt,
|
||||
} satisfies Omit<CopilotFailureDiagnosisToolUnlockCommand, 'commandDigest'>);
|
||||
return normalizeCopilotFailureDiagnosisToolUnlockCommand({
|
||||
...unsigned,
|
||||
commandDigest: hash(COMMAND_DIGEST_DOMAIN, unsigned),
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
export * from './tool-execution/contracts';
|
||||
export * from './tool-execution/coordinator';
|
||||
export * from './tool-execution/planAuthority';
|
||||
export * from './tool-execution/unlockProtocol';
|
||||
@@ -63,6 +63,8 @@ export const POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID =
|
||||
'pg-9017-ai-plugin-package-prompt-product-authorization';
|
||||
export const POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID =
|
||||
'pg-9018-ai-copilot-failure-diagnosis-admissions';
|
||||
export const POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID =
|
||||
'pg-9019-ai-copilot-failure-diagnosis-tool-unlocks';
|
||||
export const POSTGRES_MODEL_INVOCATION_SCHEMA = 'ql3_ai';
|
||||
export const LOCAL_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE =
|
||||
'QingLong3AiSchemaMigrations';
|
||||
|
||||
@@ -28,6 +28,7 @@ import {
|
||||
POSTGRES_PLUGIN_PACKAGE_PROMPT_OUTPUT_KEY_ROTATION_MIGRATION_ID,
|
||||
POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID,
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||
POSTGRES_MODEL_INVOCATION_SCHEMA,
|
||||
POSTGRES_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE,
|
||||
} from './identities';
|
||||
@@ -63,6 +64,7 @@ const POSTGRES_HISTORY_IDENTITY = Object.freeze({
|
||||
POSTGRES_MODEL_PROVIDER_CREDENTIAL_TEST_CONNECTION_MIGRATION_ID,
|
||||
POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID,
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||
]),
|
||||
streamId: POSTGRES_MODEL_INVOCATION_MIGRATION_STREAM_ID,
|
||||
dialect: 'postgresql' as const,
|
||||
|
||||
@@ -2,6 +2,7 @@ import type { PostgresQueryable } from '@qinglong/runtime-core';
|
||||
|
||||
import {
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||
POSTGRES_MODEL_INVOCATION_SCHEMA,
|
||||
} from '../identities';
|
||||
import { defineSqlMigration } from '../shared';
|
||||
@@ -9,6 +10,7 @@ import { defineSqlMigration } from '../shared';
|
||||
const ADMISSION_TABLE = 'copilot_failure_diagnosis_admissions';
|
||||
const SOURCE_SNAPSHOT_FUNCTION =
|
||||
'copilot_failure_diagnosis_admission_source_snapshot';
|
||||
const TOOL_UNLOCK_TABLE = 'copilot_failure_diagnosis_tool_unlocks';
|
||||
|
||||
const POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_TABLE_SQL = `
|
||||
CREATE TABLE "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${ADMISSION_TABLE}" (
|
||||
@@ -215,6 +217,109 @@ const postgresCopilotFailureDiagnosisAdmissionMigration =
|
||||
(context, statement) => context.query(statement).then(() => undefined),
|
||||
);
|
||||
|
||||
const POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_TABLE_SQL = `
|
||||
CREATE TABLE "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${TOOL_UNLOCK_TABLE}" (
|
||||
request_id varchar(128) PRIMARY KEY,
|
||||
plan_digest char(64) NOT NULL UNIQUE,
|
||||
run_id varchar(36) NOT NULL,
|
||||
start_id varchar(36) NOT NULL UNIQUE,
|
||||
tool_step_run_id varchar(128) NOT NULL,
|
||||
tool_completion_digest char(64) NOT NULL UNIQUE,
|
||||
model_step_run_id varchar(128) NOT NULL UNIQUE,
|
||||
model_step_run_version integer NOT NULL,
|
||||
model_step_run_digest char(64) NOT NULL,
|
||||
model_mutation_id varchar(36) NOT NULL UNIQUE,
|
||||
model_mutation_digest char(64) NOT NULL UNIQUE,
|
||||
model_event_id varchar(36) NOT NULL UNIQUE,
|
||||
final_run_version integer NOT NULL,
|
||||
final_run_event_sequence integer NOT NULL,
|
||||
unlocked_at_ms bigint NOT NULL,
|
||||
receipt_digest char(64) NOT NULL UNIQUE,
|
||||
command_digest char(64) NOT NULL UNIQUE,
|
||||
receipt_json jsonb NOT NULL,
|
||||
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_admission_fk
|
||||
FOREIGN KEY (request_id)
|
||||
REFERENCES "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${ADMISSION_TABLE}"
|
||||
(request_id) ON DELETE RESTRICT,
|
||||
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_completion_fk
|
||||
FOREIGN KEY (start_id)
|
||||
REFERENCES "ql3"."tool_execution_completions" (start_id)
|
||||
ON DELETE RESTRICT,
|
||||
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_model_step_fk
|
||||
FOREIGN KEY (run_id, model_step_run_id)
|
||||
REFERENCES "ql3"."step_runs" (run_id, id) ON DELETE RESTRICT,
|
||||
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_mutation_fk
|
||||
FOREIGN KEY (model_mutation_id)
|
||||
REFERENCES "ql3"."step_run_mutations" (mutation_id)
|
||||
ON DELETE RESTRICT,
|
||||
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_event_fk
|
||||
FOREIGN KEY (model_event_id)
|
||||
REFERENCES "ql3"."run_events" (id) ON DELETE RESTRICT,
|
||||
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_identity_check CHECK (
|
||||
request_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' AND
|
||||
run_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$' AND
|
||||
start_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$' AND
|
||||
tool_step_run_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' AND
|
||||
model_step_run_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' AND
|
||||
model_mutation_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$' AND
|
||||
model_event_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$'
|
||||
),
|
||||
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_version_check CHECK (
|
||||
model_step_run_version >= 2 AND
|
||||
final_run_version >= 1 AND
|
||||
final_run_event_sequence = final_run_version AND
|
||||
unlocked_at_ms >= 0
|
||||
),
|
||||
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_digest_check CHECK (
|
||||
plan_digest ~ '^[0-9a-f]{64}$' AND
|
||||
tool_completion_digest ~ '^[0-9a-f]{64}$' AND
|
||||
model_step_run_digest ~ '^[0-9a-f]{64}$' AND
|
||||
model_mutation_digest ~ '^[0-9a-f]{64}$' AND
|
||||
receipt_digest ~ '^[0-9a-f]{64}$' AND
|
||||
command_digest ~ '^[0-9a-f]{64}$'
|
||||
),
|
||||
CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_json_check CHECK (
|
||||
jsonb_typeof(receipt_json) = 'object' AND
|
||||
octet_length(receipt_json::text) BETWEEN 2 AND 16384 AND
|
||||
receipt_json @> jsonb_build_object(
|
||||
'schema',
|
||||
'qinglong/copilot-failure-diagnosis-tool-unlock-receipt@v1',
|
||||
'requestId', request_id, 'planDigest', plan_digest,
|
||||
'runId', run_id, 'startId', start_id,
|
||||
'toolStepRunId', tool_step_run_id,
|
||||
'toolCompletionDigest', tool_completion_digest,
|
||||
'modelStepRunId', model_step_run_id,
|
||||
'modelStepRunVersion', model_step_run_version,
|
||||
'modelStepRunDigest', model_step_run_digest,
|
||||
'modelMutationId', model_mutation_id,
|
||||
'modelMutationDigest', model_mutation_digest,
|
||||
'modelEventId', model_event_id,
|
||||
'finalRunVersion', final_run_version,
|
||||
'finalRunEventSequence', final_run_event_sequence,
|
||||
'unlockedAtMs', unlocked_at_ms, 'receiptDigest', receipt_digest
|
||||
)
|
||||
)
|
||||
)`;
|
||||
|
||||
const postgresCopilotFailureDiagnosisToolUnlockMigration =
|
||||
defineSqlMigration<PostgresQueryable>(
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||
[
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_TABLE_SQL,
|
||||
`CREATE UNIQUE INDEX ql3_ai_copilot_diagnosis_tool_unlock_fence_uidx
|
||||
ON "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${TOOL_UNLOCK_TABLE}"
|
||||
(run_id, tool_step_run_id, model_step_run_id)`,
|
||||
`REVOKE ALL ON TABLE
|
||||
"${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${TOOL_UNLOCK_TABLE}"
|
||||
FROM PUBLIC`,
|
||||
`GRANT SELECT, INSERT ON TABLE
|
||||
"${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${TOOL_UNLOCK_TABLE}"
|
||||
TO ql3_runtime`,
|
||||
],
|
||||
(context, statement) => context.query(statement).then(() => undefined),
|
||||
);
|
||||
|
||||
export const postgresCopilotMigrations = Object.freeze([
|
||||
postgresCopilotFailureDiagnosisAdmissionMigration,
|
||||
postgresCopilotFailureDiagnosisToolUnlockMigration,
|
||||
]);
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
const assert = require('node:assert/strict');
|
||||
const { createHash } = require('node:crypto');
|
||||
const { test } = require('node:test');
|
||||
|
||||
const {
|
||||
@@ -33,6 +34,12 @@ const {
|
||||
const {
|
||||
storedJsonEquals,
|
||||
} = require('../dist/copilot/failure-diagnosis/admission/postgresRepository.js');
|
||||
const {
|
||||
COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA,
|
||||
createCopilotFailureDiagnosisToolUnlockCommand,
|
||||
normalizeCopilotFailureDiagnosisToolUnlockCommand,
|
||||
restoreCopilotFailureDiagnosisTrustedToolAuthority,
|
||||
} = require('../dist/copilot/failure-diagnosis/toolExecution.js');
|
||||
|
||||
const DIGEST_A = 'a'.repeat(64);
|
||||
const DIGEST_B = 'b'.repeat(64);
|
||||
@@ -189,6 +196,40 @@ async function plan(options = {}) {
|
||||
});
|
||||
}
|
||||
|
||||
function successfulToolCompletion(current) {
|
||||
const unsigned = Object.freeze({
|
||||
schema: 'qinglong/tool-execution-completion@v1',
|
||||
startId: 'cds:diagnosis-tool-start',
|
||||
projectId: current.projectId,
|
||||
runId: current.runId,
|
||||
stepRunId: current.toolStepRunId,
|
||||
startedStepRunVersion: 2,
|
||||
completedStepRunVersion: 3,
|
||||
barrierDigest: DIGEST_A,
|
||||
adapterDigest: DIGEST_B,
|
||||
resultArtifact: Object.freeze({
|
||||
artifactId: 'cdra:diagnosis-result',
|
||||
artifactDigest: DIGEST_C,
|
||||
outputDigest: DIGEST_A,
|
||||
executionResultDigest: DIGEST_B,
|
||||
}),
|
||||
stepRunMutationId: 'cdscm:diagnosis-tool',
|
||||
stepRunMutationDigest: DIGEST_C,
|
||||
completedStepRunDigest: DIGEST_A,
|
||||
runEventId: 'cdsce:diagnosis-tool',
|
||||
completedAtMs: 3_000,
|
||||
});
|
||||
return Object.freeze({
|
||||
...unsigned,
|
||||
completionDigest: createHash('sha256')
|
||||
.update(
|
||||
Buffer.from('qinglong/tool-execution-completion-digest@v1\0', 'utf8'),
|
||||
)
|
||||
.update(JSON.stringify(unsigned))
|
||||
.digest('hex'),
|
||||
});
|
||||
}
|
||||
|
||||
test('creates one independent diagnosis Run with ready Tool and pending Model Steps', async () => {
|
||||
const current = await plan();
|
||||
assert.equal(current.schema, COPILOT_FAILURE_DIAGNOSIS_EXECUTION_PLAN_SCHEMA);
|
||||
@@ -224,6 +265,10 @@ test('creates one independent diagnosis Run with ready Tool and pending Model St
|
||||
},
|
||||
{ kind: 'tool', status: 'ready', sequence: 2 },
|
||||
);
|
||||
assert.equal(
|
||||
bundle.toolStepMutation.stepRun.definitionRef,
|
||||
'tool:qinglong.run.log.excerpt@1.0.0',
|
||||
);
|
||||
assert.deepEqual(
|
||||
{
|
||||
kind: bundle.modelStepMutation.stepRun.kind,
|
||||
@@ -257,6 +302,63 @@ test('creates one independent diagnosis Run with ready Tool and pending Model St
|
||||
);
|
||||
});
|
||||
|
||||
test('restores the exact admitted Tool and binds success to one Model unlock mutation', async () => {
|
||||
const current = await plan();
|
||||
const admission = createCopilotFailureDiagnosisAdmissionBundle(current);
|
||||
const authority = restoreCopilotFailureDiagnosisTrustedToolAuthority(
|
||||
current,
|
||||
snapshot(),
|
||||
);
|
||||
assert.equal(authority.plan.planDigest, current.tool.planDigest);
|
||||
assert.equal(authority.binding.bindingDigest, current.tool.bindingDigest);
|
||||
|
||||
const command = createCopilotFailureDiagnosisToolUnlockCommand({
|
||||
plan: current,
|
||||
completion: successfulToolCompletion(current),
|
||||
modelStepRun: admission.modelStepMutation.stepRun,
|
||||
run: {
|
||||
id: current.runId,
|
||||
projectId: current.projectId,
|
||||
status: 'running',
|
||||
version: 5,
|
||||
eventSequence: 5,
|
||||
},
|
||||
});
|
||||
assert.equal(
|
||||
command.receipt.schema,
|
||||
COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA,
|
||||
);
|
||||
assert.equal(command.modelStepRunMutation.previousStatus, 'pending');
|
||||
assert.equal(command.modelStepRunMutation.stepRun.status, 'ready');
|
||||
assert.equal(command.receipt.finalRunVersion, 6);
|
||||
assert.deepEqual(
|
||||
normalizeCopilotFailureDiagnosisToolUnlockCommand(
|
||||
JSON.parse(JSON.stringify(command)),
|
||||
),
|
||||
command,
|
||||
);
|
||||
assert.throws(
|
||||
() =>
|
||||
normalizeCopilotFailureDiagnosisToolUnlockCommand({
|
||||
...command,
|
||||
receipt: { ...command.receipt, finalRunVersion: 7 },
|
||||
}),
|
||||
TypeError,
|
||||
);
|
||||
});
|
||||
|
||||
test('publishes Tool execution only through explicit AI subpaths', () => {
|
||||
const root = require('../dist');
|
||||
const execution = require('@qinglong/ai/failure-diagnosis-tool-execution');
|
||||
const storage = require('@qinglong/ai/postgres-failure-diagnosis-tool-execution-storage');
|
||||
assert.equal(root.executeCopilotFailureDiagnosisTool, undefined);
|
||||
assert.equal(typeof execution.executeCopilotFailureDiagnosisTool, 'function');
|
||||
assert.equal(
|
||||
typeof storage.PostgresCopilotFailureDiagnosisToolUnlockRepository,
|
||||
'function',
|
||||
);
|
||||
});
|
||||
|
||||
test('rejects a Tool input detached from the source failure fence', async () => {
|
||||
await assert.rejects(
|
||||
() => plan({ inputAttemptId: 'attempt-unrelated' }),
|
||||
|
||||
@@ -20,6 +20,7 @@ const {
|
||||
LOCAL_MODEL_PRICE_CATALOG_AUTHORIZATION_MIGRATION_ID,
|
||||
LOCAL_MODEL_PRICE_CATALOG_MIGRATION_ID,
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||
POSTGRES_MODEL_INVOCATION_MIGRATION_ID,
|
||||
POSTGRES_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE,
|
||||
POSTGRES_MODEL_INVOCATION_MIGRATION_STREAM_ID,
|
||||
@@ -426,6 +427,10 @@ test('PostgreSQL AI schema is an independent reviewed feature stream', async ()
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
||||
'pg-9018-ai-copilot-failure-diagnosis-admissions',
|
||||
);
|
||||
assert.equal(
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||
'pg-9019-ai-copilot-failure-diagnosis-tool-unlocks',
|
||||
);
|
||||
assert.equal(
|
||||
POSTGRES_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE,
|
||||
'ai_schema_migrations',
|
||||
@@ -507,9 +512,13 @@ test('PostgreSQL AI schema is an independent reviewed feature stream', async ()
|
||||
postgresModelInvocationMigrationDefinition.migrations[17].checksum,
|
||||
'cd0837c68ecc6c2bce58d048308d0239397b6347b4483f02372f966c05ae7ad6',
|
||||
);
|
||||
assert.equal(
|
||||
postgresModelInvocationMigrationDefinition.migrations[18].checksum,
|
||||
'341733fde691e81efafabc9749197514d83af6c851e91ea55d85074d2dec6b4f',
|
||||
);
|
||||
assert.equal(
|
||||
postgresModelInvocationMigrationDefinition.migrations.length,
|
||||
18,
|
||||
19,
|
||||
);
|
||||
|
||||
const diagnosisAdmissionStatements = [];
|
||||
@@ -537,6 +546,25 @@ test('PostgreSQL AI schema is an independent reviewed feature stream', async ()
|
||||
assert.match(diagnosisAdmissionSql, /TO ql3_runtime/);
|
||||
assert.doesNotMatch(diagnosisAdmissionSql, /GRANT[^;]*(?:UPDATE|DELETE)/);
|
||||
|
||||
const diagnosisToolUnlockStatements = [];
|
||||
await postgresModelInvocationMigrationDefinition.migrations[18].up({
|
||||
async query(statement) {
|
||||
diagnosisToolUnlockStatements.push(statement);
|
||||
return { rows: [] };
|
||||
},
|
||||
});
|
||||
const diagnosisToolUnlockSql = diagnosisToolUnlockStatements.join('\n');
|
||||
assert.match(
|
||||
diagnosisToolUnlockSql,
|
||||
/CREATE TABLE "ql3_ai"\."copilot_failure_diagnosis_tool_unlocks"/,
|
||||
);
|
||||
assert.match(
|
||||
diagnosisToolUnlockSql,
|
||||
/FOREIGN KEY \(start_id\)[\s\S]*tool_execution_completions/,
|
||||
);
|
||||
assert.match(diagnosisToolUnlockSql, /TO ql3_runtime/);
|
||||
assert.doesNotMatch(diagnosisToolUnlockSql, /GRANT[^;]*(?:UPDATE|DELETE)/);
|
||||
|
||||
const retirementStatements = [];
|
||||
await postgresModelInvocationMigrationDefinition.migrations[10].up({
|
||||
async query(statement) {
|
||||
|
||||
+1
-1
@@ -175,7 +175,7 @@ test('readiness binds exact migration history and least-privilege primary author
|
||||
assert.equal(report.ready, true);
|
||||
assert.equal(
|
||||
report.migrationIds.at(-1),
|
||||
'pg-9018-ai-copilot-failure-diagnosis-admissions',
|
||||
'pg-9019-ai-copilot-failure-diagnosis-tool-unlocks',
|
||||
);
|
||||
assert.match(
|
||||
queries[1],
|
||||
|
||||
@@ -277,7 +277,7 @@ test('tester readiness freezes migration history and least privilege', async ()
|
||||
assert.equal(ready.ready, true);
|
||||
assert.equal(
|
||||
ready.migrationIds.at(-1),
|
||||
'pg-9018-ai-copilot-failure-diagnosis-admissions',
|
||||
'pg-9019-ai-copilot-failure-diagnosis-tool-unlocks',
|
||||
);
|
||||
|
||||
await assert.rejects(
|
||||
|
||||
@@ -125,6 +125,11 @@
|
||||
"require": "./dist/trusted-tool/key-management/toolResultProjectedKeyring.js",
|
||||
"default": "./dist/trusted-tool/key-management/toolResultProjectedKeyring.js"
|
||||
},
|
||||
"./trusted-tool-invocation-keyring": {
|
||||
"types": "./dist/trusted-tool/key-management/toolInvocationProjectedKeyring.d.ts",
|
||||
"require": "./dist/trusted-tool/key-management/toolInvocationProjectedKeyring.js",
|
||||
"default": "./dist/trusted-tool/key-management/toolInvocationProjectedKeyring.js"
|
||||
},
|
||||
"./remote-completion": {
|
||||
"types": "./dist/remote-execution/remoteWorkerCompletionService.d.ts",
|
||||
"require": "./dist/remote-execution/remoteWorkerCompletionService.js",
|
||||
|
||||
+178
@@ -0,0 +1,178 @@
|
||||
import { Buffer } from 'node:buffer';
|
||||
import { createHash } from 'node:crypto';
|
||||
|
||||
export const CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA =
|
||||
'qinglong/cluster-tool-invocation-projected-keyring@v1' as const;
|
||||
export const MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES = 64 * 1024;
|
||||
export const MAX_CLUSTER_TOOL_INVOCATION_PROJECTED_KEYS = 16;
|
||||
|
||||
const KEY_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/;
|
||||
const BASE64URL_PATTERN = /^[A-Za-z0-9_-]+$/;
|
||||
const PROJECTION_DIGEST_DOMAIN = Buffer.from(
|
||||
'qinglong/cluster-tool-invocation-projected-keyring-digest@v1\0',
|
||||
'utf8',
|
||||
);
|
||||
|
||||
export interface ClusterToolInvocationKeyringManifest {
|
||||
readonly schema: typeof CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA;
|
||||
readonly activeKeyId: string;
|
||||
readonly keys: Readonly<Record<string, string>>;
|
||||
}
|
||||
|
||||
export interface ClusterToolInvocationKeyringSummary {
|
||||
readonly schemaVersion: 1;
|
||||
readonly activeKeyId: string;
|
||||
readonly keyIds: readonly string[];
|
||||
readonly projectionDigest: string;
|
||||
}
|
||||
|
||||
export class InvalidClusterToolInvocationKeyringManifestError extends TypeError {
|
||||
readonly code = 'QL3_CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_INVALID';
|
||||
|
||||
constructor() {
|
||||
super('Cluster Tool invocation keyring manifest is invalid');
|
||||
this.name = 'InvalidClusterToolInvocationKeyringManifestError';
|
||||
}
|
||||
}
|
||||
|
||||
function invalid(): never {
|
||||
throw new InvalidClusterToolInvocationKeyringManifestError();
|
||||
}
|
||||
|
||||
function dataRecord(value: unknown): Record<string, unknown> {
|
||||
if (
|
||||
!value ||
|
||||
typeof value !== 'object' ||
|
||||
Array.isArray(value) ||
|
||||
Object.getPrototypeOf(value) !== Object.prototype
|
||||
) {
|
||||
return invalid();
|
||||
}
|
||||
return value as Record<string, unknown>;
|
||||
}
|
||||
|
||||
function exactKeys(value: object, expected: readonly string[]): boolean {
|
||||
const actual = Object.keys(value).sort();
|
||||
const canonical = [...expected].sort();
|
||||
return (
|
||||
actual.length === canonical.length &&
|
||||
actual.every((key, index) => key === canonical[index])
|
||||
);
|
||||
}
|
||||
|
||||
export function normalizeClusterToolInvocationKeyringManifest(
|
||||
value: unknown,
|
||||
): Readonly<ClusterToolInvocationKeyringManifest> {
|
||||
const manifest = dataRecord(value);
|
||||
if (
|
||||
!exactKeys(manifest, ['activeKeyId', 'keys', 'schema']) ||
|
||||
manifest.schema !== CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA ||
|
||||
typeof manifest.activeKeyId !== 'string' ||
|
||||
!KEY_ID_PATTERN.test(manifest.activeKeyId)
|
||||
) {
|
||||
return invalid();
|
||||
}
|
||||
const keys = dataRecord(manifest.keys);
|
||||
const entries = Object.entries(keys).sort(([left], [right]) =>
|
||||
left < right ? -1 : left > right ? 1 : 0,
|
||||
);
|
||||
if (
|
||||
entries.length < 1 ||
|
||||
entries.length > MAX_CLUSTER_TOOL_INVOCATION_PROJECTED_KEYS
|
||||
) {
|
||||
return invalid();
|
||||
}
|
||||
const normalized: (readonly [string, string])[] = [];
|
||||
for (const [keyId, encoded] of entries) {
|
||||
let material: Buffer | undefined;
|
||||
try {
|
||||
if (
|
||||
!KEY_ID_PATTERN.test(keyId) ||
|
||||
typeof encoded !== 'string' ||
|
||||
!BASE64URL_PATTERN.test(encoded)
|
||||
) {
|
||||
return invalid();
|
||||
}
|
||||
material = Buffer.from(encoded, 'base64url');
|
||||
if (
|
||||
material.byteLength !== 32 ||
|
||||
material.toString('base64url') !== encoded
|
||||
) {
|
||||
return invalid();
|
||||
}
|
||||
normalized.push(Object.freeze([keyId, encoded] as const));
|
||||
} finally {
|
||||
material?.fill(0);
|
||||
}
|
||||
}
|
||||
const normalizedKeys = Object.freeze(Object.fromEntries(normalized));
|
||||
if (normalizedKeys[manifest.activeKeyId] === undefined) return invalid();
|
||||
return Object.freeze({
|
||||
schema: CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA,
|
||||
activeKeyId: manifest.activeKeyId,
|
||||
keys: normalizedKeys,
|
||||
});
|
||||
}
|
||||
|
||||
export function parseClusterToolInvocationKeyringManifest(
|
||||
bytes: Buffer,
|
||||
): Readonly<ClusterToolInvocationKeyringManifest> {
|
||||
try {
|
||||
if (
|
||||
!Buffer.isBuffer(bytes) ||
|
||||
bytes.byteLength < 1 ||
|
||||
bytes.byteLength > MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES
|
||||
) {
|
||||
return invalid();
|
||||
}
|
||||
return normalizeClusterToolInvocationKeyringManifest(
|
||||
JSON.parse(bytes.toString('utf8')),
|
||||
);
|
||||
} catch (error) {
|
||||
if (error instanceof InvalidClusterToolInvocationKeyringManifestError) {
|
||||
throw error;
|
||||
}
|
||||
return invalid();
|
||||
}
|
||||
}
|
||||
|
||||
export function canonicalClusterToolInvocationKeyringManifest(
|
||||
value: ClusterToolInvocationKeyringManifest,
|
||||
): Buffer {
|
||||
const manifest = normalizeClusterToolInvocationKeyringManifest(value);
|
||||
return Buffer.from(`${JSON.stringify(manifest)}\n`, 'utf8');
|
||||
}
|
||||
|
||||
export function resolveClusterToolInvocationKeyringMaterial(
|
||||
value: ClusterToolInvocationKeyringManifest,
|
||||
keyId: string,
|
||||
): Readonly<{ keyId: string; key: Uint8Array }> | null {
|
||||
if (typeof keyId !== 'string' || !KEY_ID_PATTERN.test(keyId)) {
|
||||
return invalid();
|
||||
}
|
||||
const manifest = normalizeClusterToolInvocationKeyringManifest(value);
|
||||
const encoded = manifest.keys[keyId];
|
||||
if (encoded === undefined) return null;
|
||||
const key = Buffer.from(encoded, 'base64url');
|
||||
if (key.byteLength !== 32 || key.toString('base64url') !== encoded) {
|
||||
key.fill(0);
|
||||
return invalid();
|
||||
}
|
||||
return Object.freeze({ keyId, key });
|
||||
}
|
||||
|
||||
export function summarizeClusterToolInvocationKeyringManifest(
|
||||
value: ClusterToolInvocationKeyringManifest,
|
||||
): Readonly<ClusterToolInvocationKeyringSummary> {
|
||||
const manifest = normalizeClusterToolInvocationKeyringManifest(value);
|
||||
const keyIds = Object.freeze(Object.keys(manifest.keys).sort());
|
||||
return Object.freeze({
|
||||
schemaVersion: 1 as const,
|
||||
activeKeyId: manifest.activeKeyId,
|
||||
keyIds,
|
||||
projectionDigest: createHash('sha256')
|
||||
.update(PROJECTION_DIGEST_DOMAIN)
|
||||
.update(JSON.stringify(manifest))
|
||||
.digest('hex'),
|
||||
});
|
||||
}
|
||||
+151
@@ -0,0 +1,151 @@
|
||||
import { Buffer } from 'node:buffer';
|
||||
|
||||
import type { ToolInvocationArtifactKeyProvider } from '@qinglong/runtime-core/tool-invocation-artifact';
|
||||
|
||||
import { PrivateProjectedFileReader } from '../../security/privateProjectedFile';
|
||||
import {
|
||||
MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES,
|
||||
canonicalClusterToolInvocationKeyringManifest,
|
||||
parseClusterToolInvocationKeyringManifest,
|
||||
resolveClusterToolInvocationKeyringMaterial,
|
||||
summarizeClusterToolInvocationKeyringManifest,
|
||||
type ClusterToolInvocationKeyringManifest,
|
||||
type ClusterToolInvocationKeyringSummary,
|
||||
} from './toolInvocationKeyringManifest';
|
||||
|
||||
const DATA_FILE_NAME = /^[A-Za-z0-9][A-Za-z0-9._-]{0,252}$/;
|
||||
|
||||
export {
|
||||
CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA,
|
||||
MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES,
|
||||
MAX_CLUSTER_TOOL_INVOCATION_PROJECTED_KEYS,
|
||||
InvalidClusterToolInvocationKeyringManifestError,
|
||||
canonicalClusterToolInvocationKeyringManifest,
|
||||
normalizeClusterToolInvocationKeyringManifest,
|
||||
parseClusterToolInvocationKeyringManifest,
|
||||
resolveClusterToolInvocationKeyringMaterial,
|
||||
summarizeClusterToolInvocationKeyringManifest,
|
||||
type ClusterToolInvocationKeyringManifest,
|
||||
type ClusterToolInvocationKeyringSummary,
|
||||
} from './toolInvocationKeyringManifest';
|
||||
|
||||
export interface ClusterToolInvocationProjectedKeyringOptions {
|
||||
readonly rootDirectory: string;
|
||||
readonly dataFileName?: string;
|
||||
}
|
||||
|
||||
export class ClusterToolInvocationProjectedKeyringUnavailableError extends Error {
|
||||
readonly code = 'QL3_CLUSTER_TOOL_INVOCATION_PROJECTED_KEYRING_UNAVAILABLE';
|
||||
|
||||
constructor(options?: ErrorOptions) {
|
||||
super('Projected Cluster Tool invocation keyring is unavailable', options);
|
||||
this.name = 'ClusterToolInvocationProjectedKeyringUnavailableError';
|
||||
}
|
||||
}
|
||||
|
||||
function unavailable(
|
||||
cause?: unknown,
|
||||
): ClusterToolInvocationProjectedKeyringUnavailableError {
|
||||
return new ClusterToolInvocationProjectedKeyringUnavailableError({
|
||||
cause: cause instanceof Error ? cause : undefined,
|
||||
});
|
||||
}
|
||||
|
||||
function dataFileName(value: unknown): string {
|
||||
if (typeof value !== 'string' || !DATA_FILE_NAME.test(value)) {
|
||||
throw unavailable();
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
async function readManifest(
|
||||
reader: PrivateProjectedFileReader,
|
||||
fileName: string,
|
||||
): Promise<Readonly<ClusterToolInvocationKeyringManifest>> {
|
||||
let bytes: Buffer | undefined;
|
||||
let canonical: Buffer | undefined;
|
||||
try {
|
||||
bytes = await reader.read(fileName);
|
||||
const manifest = parseClusterToolInvocationKeyringManifest(bytes);
|
||||
canonical = canonicalClusterToolInvocationKeyringManifest(manifest);
|
||||
if (!canonical.equals(bytes)) throw unavailable();
|
||||
return manifest;
|
||||
} catch (cause) {
|
||||
throw cause instanceof ClusterToolInvocationProjectedKeyringUnavailableError
|
||||
? cause
|
||||
: unavailable(cause);
|
||||
} finally {
|
||||
bytes?.fill(0);
|
||||
canonical?.fill(0);
|
||||
}
|
||||
}
|
||||
|
||||
/** Read-only, no-cache invocation Artifact key authority for projections. */
|
||||
export class ClusterToolInvocationProjectedKeyring
|
||||
implements ToolInvocationArtifactKeyProvider
|
||||
{
|
||||
readonly #reader: PrivateProjectedFileReader;
|
||||
readonly #dataFileName: string;
|
||||
|
||||
constructor(options: ClusterToolInvocationProjectedKeyringOptions) {
|
||||
if (!options || typeof options !== 'object' || Array.isArray(options)) {
|
||||
throw unavailable();
|
||||
}
|
||||
try {
|
||||
this.#reader = new PrivateProjectedFileReader({
|
||||
rootDirectory: options.rootDirectory,
|
||||
minimumBytes: 1,
|
||||
maximumBytes: MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES,
|
||||
access: 'read_only_keyring',
|
||||
});
|
||||
this.#dataFileName = dataFileName(options.dataFileName ?? 'keyring.json');
|
||||
} catch (cause) {
|
||||
throw unavailable(cause);
|
||||
}
|
||||
}
|
||||
|
||||
async verify(): Promise<Readonly<ClusterToolInvocationKeyringSummary>> {
|
||||
return summarizeClusterToolInvocationKeyringManifest(
|
||||
await readManifest(this.#reader, this.#dataFileName),
|
||||
);
|
||||
}
|
||||
|
||||
async active(): ReturnType<ToolInvocationArtifactKeyProvider['active']> {
|
||||
try {
|
||||
const manifest = await readManifest(this.#reader, this.#dataFileName);
|
||||
return resolveClusterToolInvocationKeyringMaterial(
|
||||
manifest,
|
||||
manifest.activeKeyId,
|
||||
)!;
|
||||
} catch (cause) {
|
||||
throw cause instanceof
|
||||
ClusterToolInvocationProjectedKeyringUnavailableError
|
||||
? cause
|
||||
: unavailable(cause);
|
||||
}
|
||||
}
|
||||
|
||||
async resolve(
|
||||
keyId: string,
|
||||
): ReturnType<ToolInvocationArtifactKeyProvider['resolve']> {
|
||||
try {
|
||||
return resolveClusterToolInvocationKeyringMaterial(
|
||||
await readManifest(this.#reader, this.#dataFileName),
|
||||
keyId,
|
||||
);
|
||||
} catch (cause) {
|
||||
throw cause instanceof
|
||||
ClusterToolInvocationProjectedKeyringUnavailableError
|
||||
? cause
|
||||
: unavailable(cause);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export async function createClusterToolInvocationProjectedKeyring(
|
||||
options: ClusterToolInvocationProjectedKeyringOptions,
|
||||
): Promise<Readonly<ClusterToolInvocationProjectedKeyring>> {
|
||||
const provider = new ClusterToolInvocationProjectedKeyring(options);
|
||||
await provider.verify();
|
||||
return provider;
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs/promises');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const { afterEach, test } = require('node:test');
|
||||
|
||||
const {
|
||||
CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA,
|
||||
ClusterToolInvocationProjectedKeyring,
|
||||
ClusterToolInvocationProjectedKeyringUnavailableError,
|
||||
canonicalClusterToolInvocationKeyringManifest,
|
||||
createClusterToolInvocationProjectedKeyring,
|
||||
} = require('@qinglong/cluster-control/trusted-tool-invocation-keyring');
|
||||
|
||||
const roots = [];
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(
|
||||
roots
|
||||
.splice(0)
|
||||
.map((root) => fs.rm(root, { recursive: true, force: true })),
|
||||
);
|
||||
});
|
||||
|
||||
async function tempRoot() {
|
||||
const root = await fs.mkdtemp(
|
||||
path.join(os.tmpdir(), 'ql3-tool-invocation-keyring-'),
|
||||
);
|
||||
roots.push(root);
|
||||
return root;
|
||||
}
|
||||
|
||||
async function publish(root, generationName, manifest, mode = 0o440) {
|
||||
const generation = path.join(root, generationName);
|
||||
await fs.mkdir(generation, { mode: 0o750 });
|
||||
const target = path.join(generation, 'keyring.json');
|
||||
await fs.writeFile(
|
||||
target,
|
||||
canonicalClusterToolInvocationKeyringManifest(manifest),
|
||||
{ mode },
|
||||
);
|
||||
await fs.chmod(target, mode);
|
||||
const next = path.join(root, '..data-next');
|
||||
await fs.symlink(generationName, next);
|
||||
await fs.rename(next, path.join(root, '..data'));
|
||||
try {
|
||||
await fs.symlink('..data/keyring.json', path.join(root, 'keyring.json'));
|
||||
} catch (error) {
|
||||
if (error.code !== 'EEXIST') throw error;
|
||||
}
|
||||
}
|
||||
|
||||
function manifest(activeKeyId, keys) {
|
||||
return Object.freeze({
|
||||
schema: CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA,
|
||||
activeKeyId,
|
||||
keys: Object.freeze(keys),
|
||||
});
|
||||
}
|
||||
|
||||
test('projected Tool invocation keyring rotates active material without caching', async () => {
|
||||
const root = await tempRoot();
|
||||
const keyOne = Buffer.alloc(32, 0x41);
|
||||
const keyTwo = Buffer.alloc(32, 0x42);
|
||||
await publish(
|
||||
root,
|
||||
'..2026_08_15_01',
|
||||
manifest('invocation-key-one', {
|
||||
'invocation-key-one': keyOne.toString('base64url'),
|
||||
}),
|
||||
);
|
||||
const provider = await createClusterToolInvocationProjectedKeyring({
|
||||
rootDirectory: root,
|
||||
});
|
||||
const first = await provider.active();
|
||||
assert.equal(first.keyId, 'invocation-key-one');
|
||||
assert.deepEqual(Buffer.from(first.key), keyOne);
|
||||
first.key.fill(0);
|
||||
|
||||
await publish(
|
||||
root,
|
||||
'..2026_08_15_02',
|
||||
manifest('invocation-key-two', {
|
||||
'invocation-key-one': keyOne.toString('base64url'),
|
||||
'invocation-key-two': keyTwo.toString('base64url'),
|
||||
}),
|
||||
);
|
||||
const second = await provider.active();
|
||||
assert.equal(second.keyId, 'invocation-key-two');
|
||||
assert.deepEqual(Buffer.from(second.key), keyTwo);
|
||||
second.key.fill(0);
|
||||
const historical = await provider.resolve('invocation-key-one');
|
||||
assert.ok(historical);
|
||||
assert.deepEqual(Buffer.from(historical.key), keyOne);
|
||||
historical.key.fill(0);
|
||||
assert.equal(await provider.resolve('missing-key'), null);
|
||||
});
|
||||
|
||||
test('projected Tool invocation keyring rejects missing active and writable material', async () => {
|
||||
const root = await tempRoot();
|
||||
const key = Buffer.alloc(32, 0x51).toString('base64url');
|
||||
assert.throws(
|
||||
() =>
|
||||
canonicalClusterToolInvocationKeyringManifest(
|
||||
manifest('missing-key', { 'invocation-key-one': key }),
|
||||
),
|
||||
TypeError,
|
||||
);
|
||||
|
||||
await publish(
|
||||
root,
|
||||
'..2026_08_15_01',
|
||||
manifest('invocation-key-one', { 'invocation-key-one': key }),
|
||||
0o640,
|
||||
);
|
||||
await assert.rejects(
|
||||
() =>
|
||||
new ClusterToolInvocationProjectedKeyring({
|
||||
rootDirectory: root,
|
||||
}).verify(),
|
||||
ClusterToolInvocationProjectedKeyringUnavailableError,
|
||||
);
|
||||
});
|
||||
@@ -26,7 +26,13 @@ const {
|
||||
PostgresRemoteWorkerCompletionRepository,
|
||||
PostgresRemoteWorkerLeaseControlRepository,
|
||||
PostgresRunAttemptLogRetentionClaimRepository,
|
||||
PostgresRunRepository,
|
||||
PostgresToolExecutionCompletionRepository,
|
||||
PostgresToolExecutionFailureCompletionRepository,
|
||||
PostgresToolExecutionStartBarrierRepository,
|
||||
PostgresToolInvocationArtifactRepository,
|
||||
PostgresToolResultKeyCatalogReader,
|
||||
PostgresToolResultRekeyReader,
|
||||
PostgresWorkerSessionRepository,
|
||||
} = require('../packages/ql3-cluster-postgres/dist/entrypoints/runtime.js');
|
||||
const {
|
||||
@@ -34,6 +40,7 @@ const {
|
||||
} = require('../packages/ql3-cluster-postgres/dist/migration/migration.js');
|
||||
const {
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||
POSTGRES_MODEL_INVOCATION_MIGRATION_ID,
|
||||
POSTGRES_MODEL_INVOCATION_PRICING_MIGRATION_ID,
|
||||
POSTGRES_MODEL_INVOCATION_QUOTA_MIGRATION_ID,
|
||||
@@ -59,6 +66,12 @@ const {
|
||||
const {
|
||||
PostgresCopilotFailureDiagnosisAdmissionRepository,
|
||||
} = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/postgresAdmissionRepository.js');
|
||||
const {
|
||||
executeCopilotFailureDiagnosisTool,
|
||||
} = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/toolExecution.js');
|
||||
const {
|
||||
PostgresCopilotFailureDiagnosisToolUnlockRepository,
|
||||
} = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/postgresToolExecutionRepository.js');
|
||||
const {
|
||||
BUILTIN_RUN_LOG_EXCERPT_TOOL,
|
||||
BUILTIN_RUN_LOG_EXCERPT_TOOL_DEFINITION,
|
||||
@@ -1472,6 +1485,25 @@ async function modelInvocationFeatureFacts(pool) {
|
||||
'SELECT,INSERT,UPDATE,DELETE'
|
||||
)
|
||||
) AS "copilotDiagnosisAdmissionRuntimeOnly",
|
||||
has_table_privilege(
|
||||
'ql3_runtime',
|
||||
'ql3_ai.copilot_failure_diagnosis_tool_unlocks', 'SELECT,INSERT'
|
||||
) AND NOT has_table_privilege(
|
||||
'ql3_runtime',
|
||||
'ql3_ai.copilot_failure_diagnosis_tool_unlocks', 'UPDATE,DELETE'
|
||||
) AND NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM unnest(ARRAY[
|
||||
'ql3_admin', 'ql3_package_manager', 'ql3_package_executor',
|
||||
'ql3_worker_ingress', 'ql3_ai_maintenance',
|
||||
'ql3_ai_credential_manager', 'ql3_ai_credential_tester'
|
||||
]::text[]) AS denied(role_name)
|
||||
WHERE has_table_privilege(
|
||||
denied.role_name,
|
||||
'ql3_ai.copilot_failure_diagnosis_tool_unlocks',
|
||||
'SELECT,INSERT,UPDATE,DELETE'
|
||||
)
|
||||
) AS "copilotDiagnosisToolUnlockRuntimeOnly",
|
||||
has_function_privilege(
|
||||
'ql3_runtime',
|
||||
'ql3_ai.copilot_failure_diagnosis_admission_source_snapshot(varchar,varchar,varchar,integer,integer,varchar,varchar)',
|
||||
@@ -1598,6 +1630,24 @@ async function copilotFailureDiagnosisAdmissionFacts(pool, requestId) {
|
||||
WHERE run_id = admission.run_id) AS "eventCount",
|
||||
(SELECT count(*)::integer FROM "ql3"."step_run_mutations"
|
||||
WHERE run_id = admission.run_id) AS "mutationCount",
|
||||
(SELECT count(*)::integer
|
||||
FROM "ql3"."tool_execution_start_barriers"
|
||||
WHERE run_id = admission.run_id
|
||||
AND step_run_id = admission.tool_step_run_id)
|
||||
AS "startCount",
|
||||
(SELECT count(*)::integer
|
||||
FROM "ql3"."tool_execution_completions"
|
||||
WHERE run_id = admission.run_id
|
||||
AND step_run_id = admission.tool_step_run_id)
|
||||
AS "completionCount",
|
||||
(SELECT count(*)::integer
|
||||
FROM "ql3_ai"."copilot_failure_diagnosis_tool_unlocks"
|
||||
WHERE request_id = admission.request_id) AS "unlockCount",
|
||||
(SELECT completion.artifact_json::text
|
||||
FROM "ql3"."tool_execution_completions" AS completion
|
||||
WHERE completion.run_id = admission.run_id
|
||||
AND completion.step_run_id = admission.tool_step_run_id
|
||||
LIMIT 1) AS "resultArtifactJson",
|
||||
tool.status AS "toolStatus", tool.kind AS "toolKind",
|
||||
model.status AS "modelStatus", model.kind AS "modelKind",
|
||||
model.parent_step_run_id AS "modelParentStepRunId",
|
||||
@@ -1615,11 +1665,20 @@ async function copilotFailureDiagnosisAdmissionFacts(pool, requestId) {
|
||||
[requestId],
|
||||
);
|
||||
assert.equal(result.rowCount, 1);
|
||||
const { planJson, receiptJson, ...facts } = result.rows[0];
|
||||
const { planJson, receiptJson, resultArtifactJson, ...facts } =
|
||||
result.rows[0];
|
||||
assert.equal(typeof planJson, 'string');
|
||||
assert.equal(typeof receiptJson, 'string');
|
||||
return {
|
||||
...facts,
|
||||
resultArtifactPresent: typeof resultArtifactJson === 'string',
|
||||
resultArtifactJsonDigest:
|
||||
typeof resultArtifactJson === 'string'
|
||||
? createHash('sha256').update(resultArtifactJson).digest('hex')
|
||||
: null,
|
||||
resultArtifactContainsPlaintext:
|
||||
typeof resultArtifactJson === 'string' &&
|
||||
resultArtifactJson.includes('diagnosis-sensitive-line'),
|
||||
planJsonDigest: createHash('sha256').update(planJson).digest('hex'),
|
||||
receiptJsonDigest: createHash('sha256').update(receiptJson).digest('hex'),
|
||||
};
|
||||
@@ -1717,7 +1776,7 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) {
|
||||
subject: { type: 'user', id: subjectId },
|
||||
authenticationId: `diagnosis-auth-${suffix}`,
|
||||
authenticatedAtMs: baseTimeMs - 1_000,
|
||||
expiresAtMs: baseTimeMs + 60_000,
|
||||
expiresAtMs: baseTimeMs + 600_000,
|
||||
assurance: 'multi_factor',
|
||||
});
|
||||
const invocation = await prepareToolInvocation(
|
||||
@@ -1742,7 +1801,7 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) {
|
||||
const bindings = new TrustedToolHandlerBindingRegistry(snapshot, [
|
||||
createBuiltInRunLogExcerptToolHandlerBinding(snapshot, ['cluster-control']),
|
||||
]);
|
||||
const toolPlan = createTrustedToolInvocationPlan(bindings, invocation, {
|
||||
const toolBundle = createTrustedToolInvocationPlan(bindings, invocation, {
|
||||
actionRef: `diagnosis-log-${suffix}`,
|
||||
inputArtifactId: `diagnosis-input-${suffix}`,
|
||||
previewArtifactId: `diagnosis-preview-${suffix}`,
|
||||
@@ -1760,7 +1819,8 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) {
|
||||
warnings: ['potentially_sensitive_output'],
|
||||
},
|
||||
sealedAtMs: baseTimeMs + 200,
|
||||
}).plan;
|
||||
});
|
||||
const toolPlan = toolBundle.plan;
|
||||
const plan = prepareCopilotFailureDiagnosisExecution({
|
||||
requestId,
|
||||
traceId: `diagnosis-trace-${suffix}`,
|
||||
@@ -1793,6 +1853,12 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) {
|
||||
const repository = new PostgresCopilotFailureDiagnosisAdmissionRepository(
|
||||
runtimeDatabase.pool,
|
||||
);
|
||||
assert.deepEqual(
|
||||
await new PostgresToolInvocationArtifactRepository(
|
||||
runtimeDatabase.pool,
|
||||
).put(toolBundle.inputArtifact, toolBundle.previewArtifact),
|
||||
{ status: 'inserted' },
|
||||
);
|
||||
const first = await repository.admit(plan);
|
||||
const replay = await repository.admit(plan);
|
||||
assert.equal(first.status, 'created');
|
||||
@@ -1836,20 +1902,221 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) {
|
||||
},
|
||||
);
|
||||
return {
|
||||
requestId,
|
||||
sourceRunId,
|
||||
sourceAttemptId,
|
||||
runId: plan.runId,
|
||||
toolStepRunId: plan.toolStepRunId,
|
||||
modelStepRunId: plan.modelStepRunId,
|
||||
exactReplay: true,
|
||||
beforePromotion: facts,
|
||||
report: {
|
||||
requestId,
|
||||
sourceRunId,
|
||||
sourceAttemptId,
|
||||
runId: plan.runId,
|
||||
toolStepRunId: plan.toolStepRunId,
|
||||
modelStepRunId: plan.modelStepRunId,
|
||||
exactReplay: true,
|
||||
toolExecutionExactReplay: false,
|
||||
encryptedCompletion: false,
|
||||
beforePromotion: facts,
|
||||
},
|
||||
fixture: {
|
||||
baseTimeMs,
|
||||
plan,
|
||||
principal,
|
||||
snapshot,
|
||||
invocationKeyId: toolBundle.inputArtifact.keyId,
|
||||
invocationKey: Buffer.alloc(32, 0x21),
|
||||
},
|
||||
};
|
||||
} finally {
|
||||
await runtimeDatabase.close();
|
||||
}
|
||||
}
|
||||
|
||||
async function executeCopilotFailureDiagnosisToolHaEvidence(options) {
|
||||
const {
|
||||
port,
|
||||
report,
|
||||
fixture,
|
||||
resultKeyFixture,
|
||||
expectedExisting = false,
|
||||
} = options;
|
||||
const runtimeDatabase = await databaseOpener(
|
||||
'runtime',
|
||||
databaseUrl(RUNTIME_USER, RUNTIME_PASSWORD, port),
|
||||
expectedExisting
|
||||
? 'ql3-ha-copilot-diagnosis-tool-promoted'
|
||||
: 'ql3-ha-copilot-diagnosis-tool-primary',
|
||||
)();
|
||||
let logReads = 0;
|
||||
const logContent = Buffer.from(
|
||||
'diagnosis-sensitive-line password=must-be-redacted',
|
||||
'utf8',
|
||||
);
|
||||
const keys = {
|
||||
async resolve(keyId) {
|
||||
if (keyId !== fixture.invocationKeyId) return null;
|
||||
return { keyId, key: Buffer.from(fixture.invocationKey) };
|
||||
},
|
||||
};
|
||||
const resultKeys = {
|
||||
async resolve(keyId) {
|
||||
if (keyId !== resultKeyFixture.resultKeyId) return null;
|
||||
return {
|
||||
keyId,
|
||||
key: Buffer.alloc(32, resultKeyFixture.resultKeyByte),
|
||||
};
|
||||
},
|
||||
};
|
||||
const dependencies = {
|
||||
admissions: new PostgresCopilotFailureDiagnosisAdmissionRepository(
|
||||
runtimeDatabase.pool,
|
||||
),
|
||||
snapshots: {
|
||||
async findCurrent(projectId) {
|
||||
return projectId === fixture.plan.projectId
|
||||
? { snapshot: fixture.snapshot }
|
||||
: null;
|
||||
},
|
||||
},
|
||||
artifacts: new PostgresToolInvocationArtifactRepository(
|
||||
runtimeDatabase.pool,
|
||||
),
|
||||
invocationKeys: keys,
|
||||
resultKeys,
|
||||
stepRuns: new PostgresStepRunRepository(runtimeDatabase.pool),
|
||||
runs: new PostgresRunRepository(runtimeDatabase.pool),
|
||||
barriers: new PostgresToolExecutionStartBarrierRepository(
|
||||
runtimeDatabase.pool,
|
||||
),
|
||||
completions: new PostgresToolExecutionCompletionRepository(
|
||||
runtimeDatabase.pool,
|
||||
),
|
||||
failureCompletions: new PostgresToolExecutionFailureCompletionRepository(
|
||||
runtimeDatabase.pool,
|
||||
),
|
||||
resultKeyCatalog: new PostgresToolResultKeyCatalogReader(
|
||||
runtimeDatabase.pool,
|
||||
),
|
||||
resultRekeys: new PostgresToolResultRekeyReader(runtimeDatabase.pool),
|
||||
logs: {
|
||||
async read(request) {
|
||||
logReads += 1;
|
||||
assert.equal(request.projectId, fixture.plan.projectId);
|
||||
assert.equal(request.runId, report.sourceRunId);
|
||||
assert.equal(request.attemptId, report.sourceAttemptId);
|
||||
const totalBytes = logContent.byteLength;
|
||||
const start = Math.min(request.range.offset, totalBytes);
|
||||
const endExclusive = Math.min(start + request.range.length, totalBytes);
|
||||
return {
|
||||
status: 'available',
|
||||
projectId: request.projectId,
|
||||
runId: request.runId,
|
||||
attemptId: request.attemptId,
|
||||
logArtifactId: fixture.plan.source.logArtifactId,
|
||||
content: logContent.subarray(start, endExclusive),
|
||||
start,
|
||||
endExclusive,
|
||||
totalBytes,
|
||||
...(endExclusive < totalBytes ? { nextOffset: endExclusive } : {}),
|
||||
truncation: { truncated: false, maximumBytes: 4_194_304 },
|
||||
};
|
||||
},
|
||||
},
|
||||
unlocks: new PostgresCopilotFailureDiagnosisToolUnlockRepository(
|
||||
runtimeDatabase.pool,
|
||||
),
|
||||
now: () => fixture.baseTimeMs + 400,
|
||||
nonceFactory: () => Buffer.alloc(12, 0x61),
|
||||
};
|
||||
const authorizer = {
|
||||
async authorize() {
|
||||
return {
|
||||
effect: 'allow',
|
||||
reasons: ['role_grant'],
|
||||
fence: { projectVersion: 1, bindingVersion: 1 },
|
||||
};
|
||||
},
|
||||
};
|
||||
try {
|
||||
const first = await executeCopilotFailureDiagnosisTool(
|
||||
{
|
||||
requestId: report.requestId,
|
||||
principal: fixture.principal,
|
||||
authorizer,
|
||||
},
|
||||
dependencies,
|
||||
);
|
||||
assert.equal(first.outcome, 'succeeded');
|
||||
assert.equal(
|
||||
first.completionStatus,
|
||||
expectedExisting ? 'existing' : 'created',
|
||||
);
|
||||
assert.equal(first.unlockStatus, expectedExisting ? 'existing' : 'created');
|
||||
if (!expectedExisting) {
|
||||
const replay = await executeCopilotFailureDiagnosisTool(
|
||||
{
|
||||
requestId: report.requestId,
|
||||
principal: fixture.principal,
|
||||
authorizer,
|
||||
},
|
||||
dependencies,
|
||||
);
|
||||
assert.equal(replay.outcome, 'succeeded');
|
||||
assert.equal(replay.completionStatus, 'existing');
|
||||
assert.equal(replay.unlockStatus, 'existing');
|
||||
assert.deepEqual(replay.completion, first.completion);
|
||||
assert.deepEqual(replay.unlock, first.unlock);
|
||||
assert.equal(logReads, 2);
|
||||
} else {
|
||||
assert.equal(logReads, 0);
|
||||
}
|
||||
const facts = await copilotFailureDiagnosisAdmissionFacts(
|
||||
runtimeDatabase.pool,
|
||||
report.requestId,
|
||||
);
|
||||
assert.deepEqual(
|
||||
{
|
||||
status: facts.status,
|
||||
version: facts.version,
|
||||
eventSequence: facts.eventSequence,
|
||||
eventCount: facts.eventCount,
|
||||
mutationCount: facts.mutationCount,
|
||||
startCount: facts.startCount,
|
||||
completionCount: facts.completionCount,
|
||||
unlockCount: facts.unlockCount,
|
||||
toolStatus: facts.toolStatus,
|
||||
modelStatus: facts.modelStatus,
|
||||
resultArtifactPresent: facts.resultArtifactPresent,
|
||||
resultArtifactContainsPlaintext: facts.resultArtifactContainsPlaintext,
|
||||
},
|
||||
{
|
||||
status: 'running',
|
||||
version: 6,
|
||||
eventSequence: 6,
|
||||
eventCount: 6,
|
||||
mutationCount: 5,
|
||||
startCount: 1,
|
||||
completionCount: 1,
|
||||
unlockCount: 1,
|
||||
toolStatus: 'succeeded',
|
||||
modelStatus: 'ready',
|
||||
resultArtifactPresent: true,
|
||||
resultArtifactContainsPlaintext: false,
|
||||
},
|
||||
);
|
||||
if (!expectedExisting) {
|
||||
report.beforePromotion = facts;
|
||||
report.toolExecutionExactReplay = true;
|
||||
report.encryptedCompletion = true;
|
||||
report.primaryToolAdapterReads = logReads;
|
||||
} else {
|
||||
report.promotedToolReplayReads = logReads;
|
||||
report.toolExecutionSurvivedPromotion = true;
|
||||
}
|
||||
return facts;
|
||||
} finally {
|
||||
fixture.invocationKey.fill(0);
|
||||
logContent.fill(0);
|
||||
await runtimeDatabase.close();
|
||||
}
|
||||
}
|
||||
|
||||
async function runModelProviderCredentialCatalogMatrix(options) {
|
||||
const { primaryPort, migrationPool } = options;
|
||||
const suffix = `${process.pid}-${randomBytes(3).toString('hex')}`;
|
||||
@@ -10300,6 +10567,7 @@ async function main(argv = process.argv.slice(2)) {
|
||||
let oldPrimaryRejoin;
|
||||
let modelInvocationFeaturePromotion;
|
||||
let copilotFailureDiagnosisAdmission;
|
||||
let copilotFailureDiagnosisToolFixture;
|
||||
let modelProviderCredentialCatalog;
|
||||
let modelProviderCredentialTestConnection;
|
||||
let runAttemptLogRetentionEvidence;
|
||||
@@ -10398,6 +10666,7 @@ async function main(argv = process.argv.slice(2)) {
|
||||
);
|
||||
assert.deepEqual(beforePromotion.tables, [
|
||||
'copilot_failure_diagnosis_admissions',
|
||||
'copilot_failure_diagnosis_tool_unlocks',
|
||||
'model_invocation_completions',
|
||||
'model_invocation_price_quotes',
|
||||
'model_invocation_price_settlements',
|
||||
@@ -10448,6 +10717,7 @@ async function main(argv = process.argv.slice(2)) {
|
||||
POSTGRES_PLUGIN_PACKAGE_PROMPT_OUTPUT_KEY_ROTATION_MIGRATION_ID,
|
||||
POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID,
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID,
|
||||
POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID,
|
||||
],
|
||||
);
|
||||
assert.deepEqual(beforePromotion.privileges, {
|
||||
@@ -10472,16 +10742,19 @@ async function main(argv = process.argv.slice(2)) {
|
||||
promptOutputKeyRotationAuthoritySplit: true,
|
||||
promptSnapshotRuntimeOnly: true,
|
||||
copilotDiagnosisAdmissionRuntimeOnly: true,
|
||||
copilotDiagnosisToolUnlockRuntimeOnly: true,
|
||||
copilotDiagnosisSnapshotRuntimeOnly: true,
|
||||
migrationHistoryRuntimeReadOnly: true,
|
||||
modelProviderCredentialManagementAuthoritySplit: true,
|
||||
});
|
||||
modelInvocationFeaturePromotion = { beforePromotion };
|
||||
copilotFailureDiagnosisAdmission =
|
||||
const copilotDiagnosis =
|
||||
await runCopilotFailureDiagnosisAdmissionHaEvidence({
|
||||
primaryPort,
|
||||
migrationPool: migrationDatabase.pool,
|
||||
});
|
||||
copilotFailureDiagnosisAdmission = copilotDiagnosis.report;
|
||||
copilotFailureDiagnosisToolFixture = copilotDiagnosis.fixture;
|
||||
} finally {
|
||||
await migrationDatabase.close();
|
||||
}
|
||||
@@ -10866,6 +11139,26 @@ async function main(argv = process.argv.slice(2)) {
|
||||
toolResultRekeyFaultPool.pool.end(),
|
||||
]);
|
||||
}
|
||||
await executeCopilotFailureDiagnosisToolHaEvidence({
|
||||
port: primaryPort,
|
||||
report: copilotFailureDiagnosisAdmission,
|
||||
fixture: copilotFailureDiagnosisToolFixture,
|
||||
resultKeyFixture: toolResultKeyFixture,
|
||||
});
|
||||
await waitFor(async () => {
|
||||
const replicated = await copilotFailureDiagnosisAdmissionFacts(
|
||||
standbyDatabase.pool,
|
||||
copilotFailureDiagnosisAdmission.requestId,
|
||||
);
|
||||
return JSON.stringify(replicated) ===
|
||||
JSON.stringify(copilotFailureDiagnosisAdmission.beforePromotion)
|
||||
? replicated
|
||||
: null;
|
||||
}, 'Copilot failure diagnosis Tool completion WAL replay');
|
||||
timeline.push({
|
||||
state: 'copilot_failure_diagnosis_tool_completed',
|
||||
atMs: Number((performance.now() - startedAt).toFixed(3)),
|
||||
});
|
||||
await waitFor(async () => {
|
||||
const facts = await standbyDatabase.pool.query(
|
||||
`SELECT
|
||||
@@ -11759,6 +12052,13 @@ async function main(argv = process.argv.slice(2)) {
|
||||
copilotFailureDiagnosisAdmission.afterPromotion =
|
||||
promotedCopilotFailureDiagnosisAdmission;
|
||||
copilotFailureDiagnosisAdmission.survivedPromotion = true;
|
||||
await executeCopilotFailureDiagnosisToolHaEvidence({
|
||||
port: standbyPort,
|
||||
report: copilotFailureDiagnosisAdmission,
|
||||
fixture: copilotFailureDiagnosisToolFixture,
|
||||
resultKeyFixture: toolResultKeyFixture,
|
||||
expectedExisting: true,
|
||||
});
|
||||
await verifyModelProviderCredentialCatalogAfterPromotion({
|
||||
promotedPort: standbyPort,
|
||||
promotedPool: promotedDatabase.pool,
|
||||
@@ -12513,7 +12813,7 @@ async function main(argv = process.argv.slice(2)) {
|
||||
FROM "ql3"."worker_credential_deliveries") AS "credentialDeliveries"`,
|
||||
);
|
||||
assert.deepEqual(sideEffects.rows, [
|
||||
{ runs: 78, runEvents: 179, credentialDeliveries: 4 },
|
||||
{ runs: 78, runEvents: 182, credentialDeliveries: 4 },
|
||||
]);
|
||||
timeline.push({
|
||||
state: 'two_fresh_control_replicas_ready',
|
||||
@@ -12644,14 +12944,29 @@ async function main(argv = process.argv.slice(2)) {
|
||||
optionalAiFeatureSchemaSurvivesPromotion: true,
|
||||
copilotFailureDiagnosisAdmissionExactlyReplays:
|
||||
copilotFailureDiagnosisAdmission.exactReplay &&
|
||||
copilotFailureDiagnosisAdmission.beforePromotion.eventCount === 3 &&
|
||||
copilotFailureDiagnosisAdmission.beforePromotion.mutationCount === 2,
|
||||
copilotFailureDiagnosisAdmission.beforePromotion.eventCount === 6 &&
|
||||
copilotFailureDiagnosisAdmission.beforePromotion.mutationCount === 5,
|
||||
copilotFailureDiagnosisToolCompletesEncryptedAndUnlocksModel:
|
||||
copilotFailureDiagnosisAdmission.toolExecutionExactReplay &&
|
||||
copilotFailureDiagnosisAdmission.encryptedCompletion &&
|
||||
copilotFailureDiagnosisAdmission.beforePromotion.toolStatus ===
|
||||
'succeeded' &&
|
||||
copilotFailureDiagnosisAdmission.beforePromotion.modelStatus ===
|
||||
'ready' &&
|
||||
copilotFailureDiagnosisAdmission.beforePromotion.unlockCount === 1 &&
|
||||
copilotFailureDiagnosisAdmission.beforePromotion
|
||||
.resultArtifactContainsPlaintext === false,
|
||||
copilotFailureDiagnosisToolReplaysAfterPromotionWithoutExecution:
|
||||
copilotFailureDiagnosisAdmission.toolExecutionSurvivedPromotion &&
|
||||
copilotFailureDiagnosisAdmission.promotedToolReplayReads === 0,
|
||||
copilotFailureDiagnosisAdmissionReplicatesAndSurvivesPromotion:
|
||||
copilotFailureDiagnosisAdmission.replicatedBeforePromotion &&
|
||||
copilotFailureDiagnosisAdmission.survivedPromotion,
|
||||
copilotFailureDiagnosisAdmissionUsesLeastPrivilegeRuntime:
|
||||
modelInvocationFeaturePromotion.beforePromotion.privileges
|
||||
.copilotDiagnosisAdmissionRuntimeOnly &&
|
||||
modelInvocationFeaturePromotion.beforePromotion.privileges
|
||||
.copilotDiagnosisToolUnlockRuntimeOnly &&
|
||||
modelInvocationFeaturePromotion.beforePromotion.privileges
|
||||
.copilotDiagnosisSnapshotRuntimeOnly,
|
||||
modelProviderCredentialCatalogSurvivesPromotion:
|
||||
|
||||
@@ -268,10 +268,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
|
||||
{
|
||||
path: 'packages/ql3-ai',
|
||||
name: '@qinglong/ai',
|
||||
sourceFiles: 168,
|
||||
sourceFiles: 175,
|
||||
rootSourceFiles: 1,
|
||||
rootSourceLines: 16,
|
||||
nestedSourceFiles: 167,
|
||||
nestedSourceFiles: 174,
|
||||
rootSourceFileHardCap: 1,
|
||||
rootSourceLineHardCap: 16,
|
||||
rootSourceFileRoles: {
|
||||
@@ -385,10 +385,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
|
||||
rootSourceFileRoles: clusterControl.rootSourceFileRoles,
|
||||
},
|
||||
{
|
||||
sourceFiles: 54,
|
||||
sourceFiles: 56,
|
||||
rootSourceFiles: 2,
|
||||
rootSourceLines: 195,
|
||||
nestedSourceFiles: 52,
|
||||
nestedSourceFiles: 54,
|
||||
rootSourceFileRoles: {
|
||||
'aiCli.ts': 'binary_entry',
|
||||
'cli.ts': 'binary_entry',
|
||||
|
||||
Reference in New Issue
Block a user