diff --git a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md index 56a2b53d..d45a54e6 100644 --- a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md +++ b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md @@ -11,6 +11,7 @@ 最新增量证据（2026-08-15）： +- D-316/ADR-0408（已接受）：Cluster Copilot 现在能够从 ADR-0407 的 durable admission 恢复 exact `qinglong.run.log.excerpt@1.0.0` authority，复用通用 Trusted Tool start barrier、加密 success/failure completion、result catalog/rekey 和内建日志 adapter；确定性 start/completion identity 让 response-loss replay 直接打开既有证据，不重复读取日志或执行 adapter。Cluster invocation Artifact 使用独立 projected keyring，提供 active+historical material，但每次读取均重新执行 canonical path/symlink/mode/inode/realpath fence，且不取得 PostgreSQL Tool result generation authority。只有 exact `succeeded` completion 才能通过 `pg-9019` 的 SERIALIZABLE 事务把 Model Step 从 `pending` 原子推进到 `ready`，同时写 RunEvent、StepRunMutation 和 append-only unlock receipt；`failed|timed_out` 不解锁。本 Gate 不执行模型、不终态化 diagnosis Run，下一 Gate 是 ADR-0405 builder + Model Gateway + Copilot encrypted model completion/terminalization。实现仍为 18 个 package，无单文件/浅平 package；AI 175 个源码中 174 个、Cluster Control 56 个中 54 个位于嵌套目录，不新增依赖、进程、连接、timer/watcher/cache 或默认 Edge 成本。18-package clean build/test 全绿，AI 229 pass/3 条件 skip、backend 1,207 pass/2 条件 skip，四项架构审计零 finding；14 档 Local artifact 全通过，默认 Edge/Standalone 为 2,589,890/2,589,968 bytes。PostgreSQL 18.4 arm64 HA 130/130、timeline `1→2`，首次执行只读两次日志、密文不含敏感 fixture，提升后 exact replay 零日志读取；报告 SHA-256 为 `d525a303696e178d777b021b376729bd2c5382fb5eb7bc98466a2b79d3940517`，独立审计与 Docker 清理通过。 - D-315/ADR-0407（已接受）：Cluster Copilot 故障诊断不修改或重新打开终态源 Run，而是用源 Run/version、最新已结束 Attempt/status 与日志 Artifact 建立 exact fence，原子创建独立 `copilot_failure_diagnosis` Run、admission event、`ready` 的 `qinglong.run.log.excerpt@1.0.0` Tool Step、以其为父节点的 `pending` Model Step 和不可变 receipt。计划只接受 `cluster-control` 的 reviewed snapshot/binding/Policy/subject authority；`approval_required`、Tool contract/输入漂移或未显式允许的模型出口均在写库前失败关闭。`ql3_ai` 新增有界 admission ledger 与仅授予 runtime 的 `SECURITY DEFINER` source snapshot，在 SERIALIZABLE 事务内重验 active Project/binding 和源 Run/Attempt；response-loss replay 对 JSONB 做结构 exact 比较而不依赖无语义的对象键序。Copilot/Plugin 这类 StepRun 聚合从通用 Task orphan recovery 排除，后续由各自状态机恢复，不能伪造顶层 Attempt 或被错误终态化。本阶段只 admission，不执行 Tool/模型、不授予行动权；Tool encrypted completion、Model 解锁和 Copilot encrypted model completion 是下一 Gate。最终 18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，四项边界审计零 finding且 workspace 仍为 18 package、无单文件/浅平 package；14 档 Local Profile artifact 全通过，默认 Edge/Standalone 不引入 Cluster admission。PostgreSQL 18.4 arm64 HA 128/128、timeline `1→2`，报告 SHA-256 为 `a4ed1edec783e3f5b42507c0f8e11b94c59dbe44a57e691017d1445ec9d115e2`，证据审计与 Docker 清理通过。 - D-314/ADR-0406（已接受）：Cluster Trusted Tool 的 encrypted completion 不再停留在 storage port。`@qinglong/cluster-control/trusted-tool-result-keyring` 新增只读 projected material authority：canonical v1 manifest 只含最多 16 个 canonical 32-byte key，不含 generation、active/state/retirement，provider 只有 `resolve(keyId)` 而没有 `active()`；因此 PostgreSQL `trusted-tool-results` catalog 仍是 active/decryptable 状态唯一 authority，completion 会以 catalog material proof 再次校验。runtime 每次调用重新执行 direct-root、in-root atomic symlink、single-link、64 KiB、只读/不可执行/other-inaccessible mode、dev/inode/size/mtime 与双 realpath fence，不持有 Kubernetes API、cache、watcher 或 timer。新增能力位于 Cluster Control 既有 `trusted-tool/key-management/`，并把 mounted Secret 与 keyring 的 projected-file/TOCTOU 逻辑收敛到 package-private `security/privateProjectedFile` 真源；公开 mounted Secret 行为不变，不新增 package、依赖、migration、连接、route 或默认 Profile importer。定向共享回归 7/7，Cluster Control 完整 234 pass/2 条条件 skip/0 fail；最终 18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，package/dependency/Edge import/Cluster deployment 四项审计零 finding。workspace 仍为 18 package、无单文件或浅平 package；Cluster Control 54 个源码中仅 2 个 binary entry 位于根层，52 个处于嵌套领域目录。14 档 Local Profile artifact 全部通过，默认 Edge/Standalone 保持 2,589,812/2,589,890 bytes，AI 保持 3,121,108/3,121,198 bytes，MCP 保持 7,315,930/7,316,038 bytes。PostgreSQL 18.4 arm64 HA 125/125、timeline `1→2`，报告 SHA-256 为 `26c817647ed984d8d4627a7cae1c95de06017a5d6d32dd3dfd01414ba029e542`，证据审计与 Docker 容器/网络/卷零残留。下一 Gate 是独立 diagnosis Run 的 Tool/Model Step admission 与 Copilot encrypted model completion，不能借用终态源 Run 或 Plugin Prompt plan。 - D-313/ADR-0405（已接受）：新增 `@qinglong/ai/failure-diagnosis-prompt`，把 ADR-0403 的潜在敏感日志投影收敛为固定 system instruction + canonical JSON data envelope；日志只存在于 `log.content` JSON string value，不能通过引号、换行、伪造 role/schema 或 delimiter 拼接出新 message。builder 重新校验完整 trust/redaction/profile byte 契约，拒绝伪造 `safe`、行动权、未知字段与 byte/signal drift；envelope 不带 Run/Attempt、Artifact、path、cursor 或 content digest。部署者必须通过 `qinglong/copilot-model-egress-policy@v1` 显式允许 `potentially_sensitive` 数据进入 `on_device|external` 边界并提供输入/output token 双预算，空 allowlist 与 external 未授权均在 Model Gateway/Provider I/O 前失败关闭。输出只含 content-free egress evidence，并固定要求模型 completion 继承潜在敏感、仅加密持久化、禁止明文审计且无行动权；真正 Cluster Trusted Tool/model completion 仍需后续组合门。能力以 `ql3-ai/src/copilot/failure-diagnosis/` 三个内聚文件和精确 subpath 交付，不新增 package、依赖、迁移、连接或常驻组件。定向 12/12、AI 221 pass/3 条件 skip/0 fail；最终 18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail，四项结构/部署审计零 finding，14 档 Local Profile artifact 全部通过。默认 Edge/Standalone 保持 2,589,812/2,589,890 bytes、315 files、56 modules，Edge/Standalone AI 保持 3,121,108/3,121,198 bytes、368 files、61 modules，MCP 保持 7,315,930/7,316,038 bytes、801 files、226 modules，证明未装配 subpath 被完全裁掉。PostgreSQL 18.4 arm64 HA 125/125、timeline `1→2`，报告 SHA-256 为 `2bbc8bdd0d90e6ec9ce82d2afcaec817679dddb82860c5d405a09d5e5458bece`，证据审计与 Docker 零残留。 @@ -6942,9 +6943,12 @@ ADR-0407 已完成该 admission Gate：源失败/超时 Run 和最新已结束 A 独立 diagnosis Run 在一个 SERIALIZABLE 事务内同时获得 admission event、ready Tool Step、依赖它的 pending Model Step 与 receipt；Cluster runtime 的 source snapshot authority 会重验 active Project/binding 和日志 Artifact，response-loss replay 以结构语义比较 JSONB。该 Run 是 StepRun 编排聚合，不进入通用 -Task orphan recovery。当前阶段仍没有执行 Tool 或模型；下一 Gate 必须先形成可信 encrypted Tool -completion，再原子解锁 Model Step，并以 ADR-0405 egress policy 和 Copilot 专用 encrypted model -completion 收敛最终 Run。 +Task orphan recovery。ADR-0408 已进一步关闭 Tool execution Gate：coordinator 从 durable admission 恢复 +exact Tool authority，复用通用 start barrier 与 encrypted completion；首次执行有界读取日志，response-loss +或主库提升后的 exact replay 不再执行 adapter。只有成功 completion 才由 `pg-9019` 的原子 unlock ledger +把 Model Step 推进到 ready，失败或超时保持 pending。当前阶段仍没有执行模型或终态化 diagnosis Run； +下一 Gate 必须以 ADR-0405 builder 和 Model Gateway 形成 Copilot 专用 encrypted model completion，随后 +原子收敛 Model Step、Run、取消、deadline 与恢复语义。 ## 17. Tool Registry diff --git a/docs/adr/ADR-0408-cluster-copilot-failure-diagnosis-tool-execution.md b/docs/adr/ADR-0408-cluster-copilot-failure-diagnosis-tool-execution.md new file mode 100644 index 00000000..048a9961 --- /dev/null +++ b/docs/adr/ADR-0408-cluster-copilot-failure-diagnosis-tool-execution.md @@ -0,0 +1,91 @@ +# ADR-0408：Cluster Copilot Failure Diagnosis Tool Execution + +- 状态：Accepted +- 日期：2026-08-15 +- 关联 RFC：QL-RFC-0001 D-316、Phase 2 +- 关联 ADR：ADR-0163、ADR-0226、ADR-0403、ADR-0405、ADR-0406、ADR-0407 + +## 问题 + +ADR-0407 只把故障诊断原子接纳为独立 Run、一个 `ready` Tool Step 和一个依赖它的 +`pending` Model Step。系统仍不能执行该 Tool，也不能在崩溃、响应丢失或 PostgreSQL +主库切换后证明 Tool 是否已经产生副作用、结果是否已经加密持久化，以及 Model Step +是否可以安全解锁。 + +若 Copilot 绕过共享 Trusted Tool start/completion 协议直接读取日志，重放可能重复执行 +adapter；若先解锁 Model 再写 completion，模型可能读取缺失、失败或未验证的结果；若复用 +Prompt output key 或让 projected file 决定 Tool result active generation，又会合并本应独立的 +密钥域和 durable authority。 + +## 决策 + +1. 新增 Copilot failure-diagnosis Tool execution coordinator。它只接受 ADR-0407 已持久化的 + `requestId`，重新读取并规范化 exact admission plan/receipt、当前 Project Tool snapshot 和 + invocation input/preview Artifact；任何 plan、Definition、binding、Artifact 或 source fence + 漂移均在日志读取前失败关闭。 +2. coordinator 复用 Runtime Core 的通用 Trusted Tool execution evidence、start barrier、 + encrypted success/failure completion、result catalog 和 rekey 协议，并只装配内建 + `qinglong.run.log.excerpt@1.0.0` adapter。start/completion 使用从 plan 稳定派生的 identity； + exact replay 首先打开 durable barrier/completion，不再次读取日志或执行 adapter。 +3. invocation Artifact 使用独立的 Cluster projected keyring。它包含一个 active key 和最多 + 16 个历史 key，提供 `active()` 与 `resolve(keyId)`，每次调用重新读取 canonical 32-byte + material，不使用 cache、watcher 或 timer，并复用既有 projected-file 的 canonical path、 + symlink、权限、inode/size/mtime 和双 realpath fence。Tool result keyring 仍保持 resolve-only， + PostgreSQL result catalog 继续独占 active/decryptable generation authority；两个密钥域不得互换。 +4. 只有规范化且与 admission exact 绑定的 `succeeded` Tool completion 才能解锁 Model Step。 + `pg-9019-ai-copilot-failure-diagnosis-tool-unlocks` 新增 append-only unlock ledger；一个 + SERIALIZABLE 事务同时把 Model Step 从 `pending` 推进到 `ready`、递增 Run version/event + sequence、写 RunEvent、StepRunMutation 和不可变 unlock receipt。Tool `failed|timed_out` + completion 不解锁 Model，也不伪造最终 diagnosis 结论。 +5. unlock transaction 重新锁定并验证 admission、success completion、当前 Model Step 和 running + diagnosis Run；serialization/deadlock 只在事务提交前有界重试。response-loss replay 必须返回 + 结构完全一致的 receipt；缺失关联事实、旧 fence、不同 digest 或部分写入全部视为冲突/损坏。 +6. `ql3_runtime` 对 unlock ledger 只有 `SELECT, INSERT`，没有 `UPDATE, DELETE`；数据库外调用者 + 不能直接把 Model Step 改为 ready。Tool execution 与 PostgreSQL storage 分别通过精确 AI + subpath 发布，纯 execution 入口不隐式 re-export PostgreSQL repository。 +7. 本 Gate 到 Model Step `ready` 为止，不调用模型、不持久化模型明文、不终态化 diagnosis Run。 + ADR-0405 prompt builder、Model Gateway、Copilot 专用 encrypted model completion、失败/取消/ + deadline terminalization 和产品 API 属于后续 Gate。 +8. 能力内聚在既有 `@qinglong/ai` 与 `@qinglong/cluster-control` 的领域子目录，不新增 workspace + package、依赖、进程、Pod、连接池、listener、daemon、timer、watcher 或 cache。默认 Edge/ + Standalone 不导入 Cluster execution；显式本地 AI 制品也不会取得 PostgreSQL/S3 authority。 + +## 被否决方案 + +1. **Copilot 自建一套 Tool completion**：会分叉 start barrier、加密 Artifact、rekey 与恢复语义。 +2. **admission 后直接调用日志 reader**：响应丢失时无法区分“未执行”和“已执行但未返回”。 +3. **Tool 一启动就解锁 Model**：允许模型消费未完成、失败或未认证的结果。 +4. **把失败 Tool completion 当成可诊断输入**：当前没有经过审定的失败 Prompt/terminalization + 协议，会把基础设施错误伪装成业务诊断。 +5. **共用 Prompt output/result/invocation keyring**：混淆密钥用途、active authority 与退役范围。 +6. **为 keyring 或 coordinator 新建 package**：没有独立部署/ownership 边界，并会继续加剧包碎片化。 + +## 当前验证 + +1. admission/execution/unlock 与 migration/readiness 聚焦契约 21/21，projected invocation keyring + 2/2；AI package 最终为 229 pass、3 条件 skip、0 fail，类型检查通过。 +2. 18 个 QL3 package 的 clean build/test 全部退出 0；backend 为 1,207 pass、2 条件 skip、 + 0 fail。 +3. package boundary、Cluster dependency、Edge import 和 Cluster deployment 四项审计均 + compatible 且零 finding。workspace 仍为 18 个 package，`singleSourcePackages=[]`、 + `shallowSourcePackages=[]`；AI 175 个源码中 174 个、Cluster Control 56 个中 54 个位于 + 嵌套领域目录。 +4. 14 档 Local Profile artifact 全部通过。默认 Edge/Standalone 为 2,589,890 / 2,589,968 + bytes；显式 Edge/Standalone AI 为 3,135,809 / 3,135,899 bytes；最大 Standalone MCP 为 + 7,316,038 bytes，均在各自门内。 +5. PostgreSQL 18.4 arm64 physical HA 为 130/130 Gate、timeline `1→2`。门证明首次 Tool 执行只 + 读取两次日志、密文 JSON 不含敏感 fixture 文本、Model Step 原子进入 ready；主库提升后 exact + replay 的日志读取为零。报告 SHA-256 为 + `d525a303696e178d777b021b376729bd2c5382fb5eb7bc98466a2b79d3940517`，独立审计零 finding， + 临时 Docker 资源已清理。 + +## 后续门禁 + +1. 从 durable Tool completion 解密受信投影，经 ADR-0405 builder 生成 prompt，并通过 Model + Gateway 执行 provider credential、egress policy、quota、price 和 deadline fence。 +2. 建立 Copilot 专用 encrypted model completion 和 response-loss recovery；模型输出不得进入 + 普通 completion、数据库 JSON、审计或日志明文。 +3. 原子终态化 Model Step 与 diagnosis Run，并覆盖 Tool/Model 失败、取消、超时、未知结果和 + 主库切换后的恢复状态机。 +4. 最后再开放默认关闭的 Cluster API/CLI/UI/MCP 产品入口，并补真实 S3、外部 Provider fault + injection、多副本并发和 plaintext negative evidence。 diff --git a/docs/adr/README.md b/docs/adr/README.md index dd0d7db3..298d8d0a 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -411,6 +411,7 @@ | [ADR-0405](./ADR-0405-bounded-failure-diagnosis-prompt-and-model-egress-policy.md) | 有界故障诊断 Prompt 与显式模型出口策略 | Accepted | | [ADR-0406](./ADR-0406-cluster-projected-tool-result-key-authority.md) | Cluster Projected Tool Result Key Authority | Accepted | | [ADR-0407](./ADR-0407-cluster-copilot-failure-diagnosis-run-admission.md) | Cluster Copilot Failure Diagnosis Run Admission | Accepted | +| [ADR-0408](./ADR-0408-cluster-copilot-failure-diagnosis-tool-execution.md) | Cluster Copilot Failure Diagnosis Tool Execution 与原子 Model 解锁 | Accepted | ## 规则 diff --git a/packages/ql3-ai/package.json b/packages/ql3-ai/package.json index 825b2eb8..f82e1a95 100644 --- a/packages/ql3-ai/package.json +++ b/packages/ql3-ai/package.json @@ -40,6 +40,16 @@ "require": "./dist/copilot/failure-diagnosis/postgresAdmissionRepository.js", "default": "./dist/copilot/failure-diagnosis/postgresAdmissionRepository.js" }, + "./failure-diagnosis-tool-execution": { + "types": "./dist/copilot/failure-diagnosis/toolExecution.d.ts", + "require": "./dist/copilot/failure-diagnosis/toolExecution.js", + "default": "./dist/copilot/failure-diagnosis/toolExecution.js" + }, + "./postgres-failure-diagnosis-tool-execution-storage": { + "types": "./dist/copilot/failure-diagnosis/postgresToolExecutionRepository.d.ts", + "require": "./dist/copilot/failure-diagnosis/postgresToolExecutionRepository.js", + "default": "./dist/copilot/failure-diagnosis/postgresToolExecutionRepository.js" + }, "./model-invocation": { "types": "./dist/model-invocation/modelInvocation.d.ts", "require": "./dist/model-invocation/modelInvocation.js", diff --git a/packages/ql3-ai/src/copilot/failure-diagnosis/admission/durableEvidence.ts b/packages/ql3-ai/src/copilot/failure-diagnosis/admission/durableEvidence.ts index b529ae08..6a8a99fc 100644 --- a/packages/ql3-ai/src/copilot/failure-diagnosis/admission/durableEvidence.ts +++ b/packages/ql3-ai/src/copilot/failure-diagnosis/admission/durableEvidence.ts @@ -175,7 +175,7 @@ export function createCopilotFailureDiagnosisAdmissionBundle( runId: plan.runId, stepKey: 'collect-log', kind: 'tool', - definitionRef: `trusted-tool-plan:${plan.tool.planDigest}`, + definitionRef: 'tool:qinglong.run.log.excerpt@1.0.0', definitionDigest: plan.tool.definitionDigest, required: true, initialStatus: 'ready', diff --git a/packages/ql3-ai/src/copilot/failure-diagnosis/postgresToolExecutionRepository.ts b/packages/ql3-ai/src/copilot/failure-diagnosis/postgresToolExecutionRepository.ts new file mode 100644 index 00000000..e5d3a329 --- /dev/null +++ b/packages/ql3-ai/src/copilot/failure-diagnosis/postgresToolExecutionRepository.ts @@ -0,0 +1 @@ +export * from './tool-execution/postgresUnlockRepository'; diff --git a/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/contracts.ts b/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/contracts.ts new file mode 100644 index 00000000..7a740fdb --- /dev/null +++ b/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/contracts.ts @@ -0,0 +1,131 @@ +import type { RunRecord } from '@qinglong/runtime-core'; +import type { SecurityPrincipal } from '@qinglong/runtime-core/security'; +import type { StepRunMutation } from '@qinglong/runtime-core/step-run'; +import type { + ToolExecutionCompletionRecord, + ToolExecutionResultArtifactReference, +} from '@qinglong/runtime-core/tool-execution-completion'; +import type { ToolPolicyAuthorizer } from '@qinglong/runtime-core/tool-registry'; + +import type { + CopilotFailureDiagnosisAdmissionRepository, + CopilotFailureDiagnosisExecutionPlan, +} from '../admission/contracts'; + +export const COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA = + 'qinglong/copilot-failure-diagnosis-tool-unlock-receipt@v1' as const; +export const COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA = + 'qinglong/copilot-failure-diagnosis-tool-unlock-command@v1' as const; +export const MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_BYTES = + 16 * 1024; +export const MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_BYTES = + 48 * 1024; + +export interface CopilotFailureDiagnosisToolUnlockReceipt { + readonly schema: typeof COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA; + readonly requestId: string; + readonly planDigest: string; + readonly runId: string; + readonly startId: string; + readonly barrierDigest: string; + readonly toolStepRunId: string; + readonly toolCompletionDigest: string; + readonly resultArtifact: Readonly; + readonly modelStepRunId: string; + readonly modelStepRunVersion: number; + readonly modelStepRunDigest: string; + readonly modelMutationId: string; + readonly modelMutationDigest: string; + readonly modelEventId: string; + readonly finalRunVersion: number; + readonly finalRunEventSequence: number; + readonly unlockedAtMs: number; + readonly receiptDigest: string; +} + +export interface CopilotFailureDiagnosisToolUnlockCommand { + readonly schema: typeof COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA; + readonly plan: Readonly; + readonly completion: Readonly; + readonly modelStepRunMutation: Readonly; + readonly receipt: Readonly; + readonly commandDigest: string; +} + +export interface CopilotFailureDiagnosisToolUnlockRepository { + findByRequestId( + requestId: string, + ): Promise | null>; + commit(command: CopilotFailureDiagnosisToolUnlockCommand): Promise< + Readonly<{ + status: 'created' | 'existing'; + receipt: Readonly; + }> + >; +} + +export interface ExecuteCopilotFailureDiagnosisToolInput { + readonly requestId: string; + readonly principal: Readonly; + readonly authorizer: ToolPolicyAuthorizer; +} + +export type CopilotFailureDiagnosisToolExecutionResult = + | Readonly<{ + outcome: 'succeeded'; + completionStatus: 'created' | 'existing'; + unlockStatus: 'created' | 'existing'; + completion: Readonly; + unlock: Readonly; + }> + | Readonly<{ + outcome: 'failed' | 'timed_out'; + completionStatus: 'created' | 'existing'; + unlockStatus: null; + }>; + +export class InvalidCopilotFailureDiagnosisToolExecutionError extends TypeError { + readonly code = 'COPILOT_FAILURE_DIAGNOSIS_TOOL_EXECUTION_INVALID'; + + constructor(message: string) { + super(`Copilot failure diagnosis Tool execution is invalid: ${message}`); + this.name = 'InvalidCopilotFailureDiagnosisToolExecutionError'; + } +} + +export class CopilotFailureDiagnosisToolExecutionConflictError extends Error { + readonly code = 'COPILOT_FAILURE_DIAGNOSIS_TOOL_EXECUTION_CONFLICT'; + + constructor(message = 'durable Tool execution facts changed') { + super(`Copilot failure diagnosis Tool execution conflicts: ${message}`); + this.name = 'CopilotFailureDiagnosisToolExecutionConflictError'; + } +} + +export class CopilotFailureDiagnosisToolExecutionUnavailableError extends Error { + readonly code = 'COPILOT_FAILURE_DIAGNOSIS_TOOL_EXECUTION_UNAVAILABLE'; + + constructor(options?: ErrorOptions) { + super('Copilot failure diagnosis Tool execution is unavailable', options); + this.name = 'CopilotFailureDiagnosisToolExecutionUnavailableError'; + } +} + +export class CopilotFailureDiagnosisToolExecutionDeadlineExceededError extends Error { + readonly code = 'COPILOT_FAILURE_DIAGNOSIS_TOOL_EXECUTION_DEADLINE_EXCEEDED'; + + constructor() { + super('Copilot failure diagnosis Tool execution deadline was exceeded'); + this.name = 'CopilotFailureDiagnosisToolExecutionDeadlineExceededError'; + } +} + +export type CopilotFailureDiagnosisToolExecutionAdmissionReader = Pick< + CopilotFailureDiagnosisAdmissionRepository, + 'findByRequestId' | 'findPlanByRequestId' +>; + +export type CopilotFailureDiagnosisRunAuthority = Pick< + RunRecord, + 'id' | 'projectId' | 'status' | 'version' | 'eventSequence' +>; diff --git a/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/coordinator.ts b/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/coordinator.ts new file mode 100644 index 00000000..c1fe559f --- /dev/null +++ b/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/coordinator.ts @@ -0,0 +1,599 @@ +import { Buffer } from 'node:buffer'; +import { createHash } from 'node:crypto'; + +import type { RunRepositoryReader } from '@qinglong/runtime-core/run-repository'; +import { + transitionStepRunMutation, + type StepRunRepository, +} from '@qinglong/runtime-core/step-run'; +import { + createToolExecutionEvidenceBundle, + toolExecutionAdmissionEvidence, + TOOL_EXECUTION_START_AUDIT_OPERATION, +} from '@qinglong/runtime-core/tool-execution-evidence'; +import { + createToolExecutionStartCommand, + normalizeToolExecutionStartBarrierRecord, + type ToolExecutionStartBarrierRecord, + type ToolExecutionStartBarrierRepository, +} from '@qinglong/runtime-core/tool-execution-start-barrier'; +import type { ToolExecutionCompletionRepository } from '@qinglong/runtime-core/tool-execution-completion'; +import type { ToolExecutionFailureCompletionRepository } from '@qinglong/runtime-core/tool-execution-failure-completion'; +import type { ToolExecutionResultRekeyReader } from '@qinglong/runtime-core/tool-result-rekey'; +import type { ToolResultKeyCatalogReader } from '@qinglong/runtime-core/tool-result-key-catalog'; +import type { ProjectToolDefinitionSnapshotRepository } from '@qinglong/runtime-core/project-tool-definition-snapshot'; +import { + toolInvocationInputArtifactReference, + toolInvocationPreviewArtifactReference, + type ToolInvocationArtifactKeyProvider, + type ToolInvocationArtifactRepository, +} from '@qinglong/runtime-core/tool-invocation-artifact'; +import { + admitTrustedToolExecution, + trustedToolContractIdentityDigest, +} from '@qinglong/runtime-core/trusted-tool-invocation'; +import { TrustedToolExecutionAdapterRegistry } from '@qinglong/runtime-core/trusted-tool-execution'; +import { executeAndCompleteTrustedTool } from '@qinglong/runtime-core/trusted-tool-completion'; +import { BuiltInRunLogExcerptToolAdapter } from '@qinglong/runtime-core/builtin-run-log-excerpt-tool'; +import type { RunAttemptLogReadPort } from '@qinglong/runtime-core/builtin-run-log-excerpt-projection'; + +import { + CopilotFailureDiagnosisToolExecutionConflictError, + CopilotFailureDiagnosisToolExecutionDeadlineExceededError, + CopilotFailureDiagnosisToolExecutionUnavailableError, + InvalidCopilotFailureDiagnosisToolExecutionError, + type CopilotFailureDiagnosisToolExecutionAdmissionReader, + type CopilotFailureDiagnosisToolExecutionResult, + type CopilotFailureDiagnosisToolUnlockRepository, + type ExecuteCopilotFailureDiagnosisToolInput, +} from './contracts'; +import { restoreCopilotFailureDiagnosisTrustedToolAuthority } from './planAuthority'; +import { createCopilotFailureDiagnosisToolUnlockCommand } from './unlockProtocol'; + +export interface CopilotFailureDiagnosisToolExecutionDependencies { + readonly admissions: CopilotFailureDiagnosisToolExecutionAdmissionReader; + readonly snapshots: Pick< + ProjectToolDefinitionSnapshotRepository, + 'findCurrent' + >; + readonly artifacts: ToolInvocationArtifactRepository; + readonly invocationKeys: Pick; + readonly resultKeys: Pick; + readonly stepRuns: Pick; + readonly runs: Pick; + readonly barriers: ToolExecutionStartBarrierRepository; + readonly completions: ToolExecutionCompletionRepository; + readonly failureCompletions: ToolExecutionFailureCompletionRepository; + readonly resultKeyCatalog: ToolResultKeyCatalogReader; + readonly resultRekeys: ToolExecutionResultRekeyReader; + readonly logs: RunAttemptLogReadPort; + readonly unlocks: CopilotFailureDiagnosisToolUnlockRepository; + readonly now?: () => number; + readonly nonceFactory?: () => Uint8Array; +} + +const IDENTITY_DOMAIN = Buffer.from( + 'qinglong/copilot-failure-diagnosis-tool-execution-identity@v1\0', + 'utf8', +); + +function invalid(message: string): never { + throw new InvalidCopilotFailureDiagnosisToolExecutionError(message); +} + +function unavailable(cause?: unknown): never { + throw new CopilotFailureDiagnosisToolExecutionUnavailableError({ + cause: cause instanceof Error ? cause : undefined, + }); +} + +function hash(value: unknown): string { + return createHash('sha256') + .update(IDENTITY_DOMAIN) + .update(JSON.stringify(value)) + .digest('hex'); +} + +function identity(prefix: string, planDigest: string): string { + const maximumDigestLength = 35 - prefix.length; + return `${prefix}:${hash({ prefix, planDigest }).slice( + 0, + maximumDigestLength, + )}`; +} + +function traceIdentity(planDigest: string): string { + return hash({ prefix: 'trace', planDigest }).slice(0, 32); +} + +function spanIdentity(planDigest: string): string { + return hash({ prefix: 'span', planDigest }).slice(0, 16); +} + +function auditEventIdentity(planDigest: string): string { + const value = hash({ prefix: 'audit', planDigest }).slice(0, 32).split(''); + value[12] = '4'; + value[16] = '8'; + const hex = value.join(''); + return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice( + 12, + 16, + )}-${hex.slice(16, 20)}-${hex.slice(20)}`; +} + +function clock(now: (() => number) | undefined): number { + let value: number; + try { + value = (now ?? Date.now)(); + } catch (cause) { + return unavailable(cause); + } + if (!Number.isSafeInteger(value) || value < 0) return unavailable(); + return value; +} + +function sameValue(left: unknown, right: unknown): boolean { + return JSON.stringify(left) === JSON.stringify(right); +} + +function assertDependencies( + dependencies: CopilotFailureDiagnosisToolExecutionDependencies, +): void { + if ( + !dependencies || + typeof dependencies !== 'object' || + typeof dependencies.admissions?.findByRequestId !== 'function' || + typeof dependencies.admissions?.findPlanByRequestId !== 'function' || + typeof dependencies.snapshots?.findCurrent !== 'function' || + typeof dependencies.artifacts?.findInput !== 'function' || + typeof dependencies.artifacts?.findPreview !== 'function' || + typeof dependencies.invocationKeys?.resolve !== 'function' || + typeof dependencies.resultKeys?.resolve !== 'function' || + typeof dependencies.stepRuns?.findById !== 'function' || + typeof dependencies.runs?.findRunById !== 'function' || + typeof dependencies.barriers?.findByStartId !== 'function' || + typeof dependencies.barriers?.prepare !== 'function' || + typeof dependencies.completions?.findByStartId !== 'function' || + typeof dependencies.failureCompletions?.findByStartId !== 'function' || + typeof dependencies.resultKeyCatalog?.findCurrent !== 'function' || + typeof dependencies.resultRekeys?.findHeadByArtifactId !== 'function' || + typeof dependencies.logs?.read !== 'function' || + typeof dependencies.unlocks?.findByRequestId !== 'function' || + typeof dependencies.unlocks?.commit !== 'function' || + (dependencies.now !== undefined && + typeof dependencies.now !== 'function') || + (dependencies.nonceFactory !== undefined && + typeof dependencies.nonceFactory !== 'function') + ) { + return invalid('dependencies are invalid'); + } +} + +async function durablePlan( + requestId: string, + dependencies: CopilotFailureDiagnosisToolExecutionDependencies, +) { + let plan; + let receipt; + try { + [plan, receipt] = await Promise.all([ + dependencies.admissions.findPlanByRequestId(requestId), + dependencies.admissions.findByRequestId(requestId), + ]); + } catch (cause) { + return unavailable(cause); + } + if ( + !plan || + !receipt || + plan.requestId !== requestId || + receipt.requestId !== requestId || + receipt.planDigest !== plan.planDigest || + receipt.runId !== plan.runId || + receipt.toolStepRunId !== plan.toolStepRunId || + receipt.modelStepRunId !== plan.modelStepRunId + ) { + throw new CopilotFailureDiagnosisToolExecutionConflictError( + 'diagnosis admission evidence is incomplete', + ); + } + return plan; +} + +async function currentAuthority( + plan: Awaited>, + dependencies: CopilotFailureDiagnosisToolExecutionDependencies, +) { + let record; + try { + record = await dependencies.snapshots.findCurrent(plan.projectId); + } catch (cause) { + return unavailable(cause); + } + if (!record) { + throw new CopilotFailureDiagnosisToolExecutionConflictError( + 'the current Project Tool snapshot is absent', + ); + } + return restoreCopilotFailureDiagnosisTrustedToolAuthority( + plan, + record.snapshot, + ); +} + +async function assertArtifacts( + plan: Awaited>, + dependencies: CopilotFailureDiagnosisToolExecutionDependencies, +): Promise { + let input; + let preview; + try { + [input, preview] = await Promise.all([ + dependencies.artifacts.findInput(plan.tool.invocationArtifact.artifactId), + dependencies.artifacts.findPreview(plan.tool.previewArtifact.artifactId), + ]); + } catch (cause) { + return unavailable(cause); + } + if ( + !input || + !preview || + !sameValue( + toolInvocationInputArtifactReference(input), + plan.tool.invocationArtifact, + ) || + !sameValue( + toolInvocationPreviewArtifactReference(preview), + plan.tool.previewArtifact, + ) || + input.projectId !== plan.projectId || + input.actionRef !== plan.tool.actionRef || + preview.projectId !== plan.projectId || + preview.actionRef !== plan.tool.actionRef || + input.sealedAtMs !== plan.tool.sealedAtMs || + preview.sealedAtMs !== plan.tool.sealedAtMs + ) { + throw new CopilotFailureDiagnosisToolExecutionConflictError( + 'the durable Tool invocation Artifacts changed', + ); + } +} + +function barrierMatches( + barrierValue: ToolExecutionStartBarrierRecord, + plan: Awaited>, + startId: string, +): Readonly { + const barrier = normalizeToolExecutionStartBarrierRecord(barrierValue); + if ( + barrier.startId !== startId || + barrier.projectId !== plan.projectId || + barrier.runId !== plan.runId || + barrier.stepRunId !== plan.toolStepRunId || + barrier.actionRef !== plan.tool.actionRef || + barrier.planDigest !== plan.tool.planDigest || + barrier.actionDigest !== plan.tool.actionDigest || + barrier.snapshotDigest !== plan.tool.snapshotDigest || + barrier.definitionDigest !== plan.tool.definitionDigest || + barrier.bindingDigest !== plan.tool.bindingDigest || + !sameValue(barrier.invocationArtifact, plan.tool.invocationArtifact) || + !sameValue(barrier.previewArtifact, plan.tool.previewArtifact) || + !sameValue(barrier.requestedBy, plan.requestedBySubject) || + barrier.profile !== 'cluster-control' || + !sameValue(barrier.policyFence, plan.policyFence) || + barrier.approvalRequestId !== null || + barrier.approvalDispatchId !== null || + barrier.approvalDispatchDigest !== null || + barrier.previousStepRunVersion !== 1 || + barrier.startedStepRunVersion !== 2 + ) { + throw new CopilotFailureDiagnosisToolExecutionConflictError( + 'the durable Tool start barrier changed', + ); + } + return barrier; +} + +async function prepareStart( + plan: Awaited>, + input: ExecuteCopilotFailureDiagnosisToolInput, + authority: Awaited>, + dependencies: CopilotFailureDiagnosisToolExecutionDependencies, + startId: string, +): Promise> { + let existing; + try { + existing = await dependencies.barriers.findByStartId(startId); + } catch (cause) { + return unavailable(cause); + } + if (existing) return barrierMatches(existing, plan, startId); + + await assertArtifacts(plan, dependencies); + let stepRun; + let run; + try { + [stepRun, run] = await Promise.all([ + dependencies.stepRuns.findById(plan.toolStepRunId), + dependencies.runs.findRunById(plan.runId), + ]); + } catch (cause) { + return unavailable(cause); + } + if ( + !stepRun || + !run || + stepRun.id !== plan.toolStepRunId || + stepRun.runId !== plan.runId || + stepRun.kind !== 'tool' || + stepRun.status !== 'ready' || + stepRun.version !== 1 || + stepRun.definitionRef !== 'tool:qinglong.run.log.excerpt@1.0.0' || + stepRun.definitionDigest !== plan.tool.definitionDigest || + run.id !== plan.runId || + run.projectId !== plan.projectId || + run.status !== 'running' || + run.version !== run.eventSequence + ) { + throw new CopilotFailureDiagnosisToolExecutionConflictError( + 'the Tool StepRun is not startable', + ); + } + const startedAtMs = clock(dependencies.now); + if ( + startedAtMs < plan.plannedAtMs || + startedAtMs + authority.plan.timeoutSeconds * 1_000 > plan.deadlineAtMs + ) { + throw new CopilotFailureDiagnosisToolExecutionDeadlineExceededError(); + } + const evidence = createToolExecutionEvidenceBundle({ + traceId: traceIdentity(plan.planDigest), + spanId: spanIdentity(plan.planDigest), + projectId: plan.projectId, + runId: plan.runId, + stepRunId: plan.toolStepRunId, + invocationPlanDigest: authority.plan.planDigest, + bindingDigest: authority.binding.bindingDigest, + adapterDigest: trustedToolContractIdentityDigest(authority.binding.adapter), + redactionContractDigest: trustedToolContractIdentityDigest( + authority.binding.redactionContract, + ), + auditContractDigest: trustedToolContractIdentityDigest( + authority.binding.auditContract, + ), + audit: { + eventId: auditEventIdentity(plan.planDigest), + requestId: plan.requestId, + operationId: TOOL_EXECUTION_START_AUDIT_OPERATION, + projectId: plan.projectId, + subject: plan.requestedBySubject, + authenticationId: input.principal.authenticationId, + outcome: 'allowed', + reasons: ['copilot_failure_diagnosis_tool_start'], + fence: plan.policyFence, + occurredAtMs: startedAtMs, + }, + createdAtMs: startedAtMs, + }); + const admission = await admitTrustedToolExecution( + authority.bindings, + authority.plan, + { + principal: input.principal, + profile: 'cluster-control', + nowMs: startedAtMs, + authorizer: input.authorizer, + evidence: { + stepRun: { + id: stepRun.id, + version: stepRun.version, + digest: stepRun.stepRunDigest, + }, + ...toolExecutionAdmissionEvidence(evidence), + }, + }, + ); + const mutationId = identity('cdstm', plan.planDigest); + const eventId = identity('cdste', plan.planDigest); + const mutation = transitionStepRunMutation( + stepRun, + { + expectedVersion: stepRun.version, + expectedDigest: stepRun.stepRunDigest, + mutationId, + to: 'running', + atMs: startedAtMs, + }, + { + expectedRunVersion: run.version, + expectedRunEventSequence: run.eventSequence, + eventId, + dedupeKey: eventId, + actor: plan.requestedBySubject, + }, + ); + const command = createToolExecutionStartCommand({ + startId, + admission, + evidence, + stepRunMutation: mutation, + }); + try { + const prepared = await dependencies.barriers.prepare(command); + if (!['created', 'existing'].includes(prepared.status)) { + return unavailable(); + } + return barrierMatches(prepared.barrier, plan, startId); + } catch (cause) { + let recovered; + try { + recovered = await dependencies.barriers.findByStartId(startId); + } catch { + throw cause; + } + if (recovered) return barrierMatches(recovered, plan, startId); + throw cause; + } +} + +function completionIdentities(planDigest: string) { + return Object.freeze({ + success: Object.freeze({ + create() { + return Object.freeze({ + artifactId: identity('cdra', planDigest), + mutationId: identity('cdscm', planDigest), + eventId: identity('cdsce', planDigest), + }); + }, + }), + failure: Object.freeze({ + create() { + return Object.freeze({ + mutationId: identity('cdfcm', planDigest), + eventId: identity('cdfce', planDigest), + }); + }, + }), + }); +} + +async function unlockModel( + plan: Awaited>, + completion: Parameters< + typeof createCopilotFailureDiagnosisToolUnlockCommand + >[0]['completion'], + dependencies: CopilotFailureDiagnosisToolExecutionDependencies, +) { + let existing; + try { + existing = await dependencies.unlocks.findByRequestId(plan.requestId); + } catch (cause) { + return unavailable(cause); + } + if (existing) { + if ( + existing.planDigest !== plan.planDigest || + existing.toolCompletionDigest !== completion.completionDigest + ) { + throw new CopilotFailureDiagnosisToolExecutionConflictError( + 'the durable model unlock changed', + ); + } + return Object.freeze({ status: 'existing' as const, receipt: existing }); + } + let modelStepRun; + let run; + try { + [modelStepRun, run] = await Promise.all([ + dependencies.stepRuns.findById(plan.modelStepRunId), + dependencies.runs.findRunById(plan.runId), + ]); + } catch (cause) { + return unavailable(cause); + } + if (!modelStepRun || !run) return unavailable(); + const command = createCopilotFailureDiagnosisToolUnlockCommand({ + plan, + completion, + modelStepRun, + run: { + id: run.id, + projectId: run.projectId, + status: run.status, + version: run.version, + eventSequence: run.eventSequence, + }, + }); + try { + return await dependencies.unlocks.commit(command); + } catch (cause) { + let recovered; + try { + recovered = await dependencies.unlocks.findByRequestId(plan.requestId); + } catch { + throw cause; + } + if ( + recovered && + recovered.planDigest === plan.planDigest && + recovered.toolCompletionDigest === completion.completionDigest + ) { + return Object.freeze({ + status: 'existing' as const, + receipt: recovered, + }); + } + throw cause; + } +} + +export async function executeCopilotFailureDiagnosisTool( + input: ExecuteCopilotFailureDiagnosisToolInput, + dependencies: CopilotFailureDiagnosisToolExecutionDependencies, +): Promise> { + assertDependencies(dependencies); + if ( + !input || + typeof input !== 'object' || + Array.isArray(input) || + typeof input.requestId !== 'string' || + typeof input.authorizer?.authorize !== 'function' + ) { + return invalid('input is invalid'); + } + const plan = await durablePlan(input.requestId, dependencies); + const authority = await currentAuthority(plan, dependencies); + const startId = identity('cds', plan.planDigest); + const barrier = await prepareStart( + plan, + input, + authority, + dependencies, + startId, + ); + const definitions = authority.bindings.definitionRegistry(); + const adapters = new TrustedToolExecutionAdapterRegistry(authority.bindings, [ + new BuiltInRunLogExcerptToolAdapter( + authority.binding, + 'cluster-control', + definitions, + dependencies.logs, + ), + ]); + const ids = completionIdentities(plan.planDigest); + const completed = await executeAndCompleteTrustedTool(barrier.startId, { + barriers: dependencies.barriers, + artifacts: dependencies.artifacts, + keys: dependencies.invocationKeys, + adapters, + completions: dependencies.completions, + failureCompletions: dependencies.failureCompletions, + stepRuns: dependencies.stepRuns, + runs: dependencies.runs, + resultKeyCatalog: dependencies.resultKeyCatalog, + resultRekeys: dependencies.resultRekeys, + resultKeys: dependencies.resultKeys, + identities: ids.success, + failureIdentities: ids.failure, + ...(dependencies.now === undefined ? {} : { now: dependencies.now }), + ...(dependencies.nonceFactory === undefined + ? {} + : { nonceFactory: dependencies.nonceFactory }), + }); + if (completed.outcome !== 'succeeded') { + return Object.freeze({ + outcome: completed.outcome, + completionStatus: completed.status, + unlockStatus: null, + }); + } + const unlock = await unlockModel(plan, completed.completion, dependencies); + return Object.freeze({ + outcome: 'succeeded' as const, + completionStatus: completed.status, + unlockStatus: unlock.status, + completion: completed.completion, + unlock: unlock.receipt, + }); +} diff --git a/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/planAuthority.ts b/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/planAuthority.ts new file mode 100644 index 00000000..ba4b2798 --- /dev/null +++ b/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/planAuthority.ts @@ -0,0 +1,101 @@ +import { + BUILTIN_RUN_LOG_EXCERPT_TIMEOUT_SECONDS, + BUILTIN_RUN_LOG_EXCERPT_TOOL, + createBuiltInRunLogExcerptToolHandlerBinding, +} from '@qinglong/runtime-core/builtin-run-log-excerpt-tool'; +import { + normalizeProjectToolDefinitionSnapshot, + type ProjectToolDefinitionSnapshot, +} from '@qinglong/runtime-core/project-tool-definition-snapshot'; +import { + TrustedToolHandlerBindingRegistry, + normalizeTrustedToolInvocationPlan, + type TrustedToolHandlerBinding, + type TrustedToolInvocationPlan, +} from '@qinglong/runtime-core/trusted-tool-invocation'; + +import type { CopilotFailureDiagnosisExecutionPlan } from '../admission/contracts'; +import { normalizeCopilotFailureDiagnosisExecutionPlan } from '../admission/plan'; +import { + CopilotFailureDiagnosisToolExecutionConflictError, + InvalidCopilotFailureDiagnosisToolExecutionError, +} from './contracts'; + +export interface CopilotFailureDiagnosisTrustedToolAuthority { + readonly plan: Readonly; + readonly binding: Readonly; + readonly bindings: TrustedToolHandlerBindingRegistry; +} + +function invalid(message: string): never { + throw new InvalidCopilotFailureDiagnosisToolExecutionError(message); +} + +export function restoreCopilotFailureDiagnosisTrustedToolAuthority( + executionPlanValue: CopilotFailureDiagnosisExecutionPlan, + snapshotValue: ProjectToolDefinitionSnapshot, +): Readonly { + const executionPlan = + normalizeCopilotFailureDiagnosisExecutionPlan(executionPlanValue); + const snapshot = normalizeProjectToolDefinitionSnapshot(snapshotValue); + if ( + snapshot.projectId !== executionPlan.projectId || + snapshot.snapshotDigest !== executionPlan.tool.snapshotDigest + ) { + throw new CopilotFailureDiagnosisToolExecutionConflictError( + 'the current Project Tool snapshot changed', + ); + } + const binding = createBuiltInRunLogExcerptToolHandlerBinding(snapshot, [ + 'cluster-control', + ]); + if ( + binding.bindingDigest !== executionPlan.tool.bindingDigest || + binding.definitionDigest !== executionPlan.tool.definitionDigest + ) { + throw new CopilotFailureDiagnosisToolExecutionConflictError( + 'the reviewed log Tool binding changed', + ); + } + const bindings = new TrustedToolHandlerBindingRegistry(snapshot, [binding]); + let plan: Readonly; + try { + plan = normalizeTrustedToolInvocationPlan( + { + schema: 'qinglong/trusted-tool-invocation-plan@v1', + status: 'ready', + actionType: 'tool.invoke', + actionRef: executionPlan.tool.actionRef, + projectId: executionPlan.projectId, + requestedBy: executionPlan.requestedBySubject, + tool: BUILTIN_RUN_LOG_EXCERPT_TOOL, + permission: 'tool.call:qinglong.run.log.excerpt', + requiredPermissions: ['artifact.read'], + effect: 'read', + risk: 'medium', + policyFence: executionPlan.policyFence, + profile: 'cluster-control', + snapshotDigest: executionPlan.tool.snapshotDigest, + definitionDigest: executionPlan.tool.definitionDigest, + binding, + timeoutSeconds: BUILTIN_RUN_LOG_EXCERPT_TIMEOUT_SECONDS, + invocationArtifact: executionPlan.tool.invocationArtifact, + invocationActionDigest: executionPlan.tool.invocationActionDigest, + previewArtifact: executionPlan.tool.previewArtifact, + actionDigest: executionPlan.tool.actionDigest, + sealedAtMs: executionPlan.tool.sealedAtMs, + planDigest: executionPlan.tool.planDigest, + }, + bindings, + ); + } catch (cause) { + if (cause instanceof CopilotFailureDiagnosisToolExecutionConflictError) { + throw cause; + } + return invalid('the admitted trusted Tool plan cannot be restored'); + } + if (plan.planDigest !== executionPlan.tool.planDigest) { + return invalid('the restored trusted Tool plan digest changed'); + } + return Object.freeze({ plan, binding, bindings }); +} diff --git a/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/postgresUnlockRepository.ts b/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/postgresUnlockRepository.ts new file mode 100644 index 00000000..5ca79d1d --- /dev/null +++ b/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/postgresUnlockRepository.ts @@ -0,0 +1,548 @@ +import { isDeepStrictEqual } from 'node:util'; + +import type { + PostgresClient, + PostgresPool, + PostgresQueryable, +} from '@qinglong/runtime-core'; +import { + normalizeToolExecutionCompletionRecord, + type ToolExecutionCompletionRecord, +} from '@qinglong/runtime-core/tool-execution-completion'; + +import { POSTGRES_MODEL_INVOCATION_SCHEMA } from '../../../migration/modelInvocationMigration'; +import type { CopilotFailureDiagnosisExecutionPlan } from '../admission/contracts'; +import { normalizeCopilotFailureDiagnosisExecutionPlan } from '../admission/plan'; +import { + CopilotFailureDiagnosisToolExecutionConflictError, + CopilotFailureDiagnosisToolExecutionUnavailableError, + type CopilotFailureDiagnosisToolUnlockCommand, + type CopilotFailureDiagnosisToolUnlockReceipt, + type CopilotFailureDiagnosisToolUnlockRepository, +} from './contracts'; +import { + normalizeCopilotFailureDiagnosisToolUnlockCommand, + normalizeCopilotFailureDiagnosisToolUnlockReceipt, +} from './unlockProtocol'; + +const UNLOCK_TABLE = 'copilot_failure_diagnosis_tool_unlocks'; +const RETRYABLE_SQL_STATES = new Set(['40001', '40P01']); +const MAX_TRANSACTION_ATTEMPTS = 3; + +type Row = Readonly>; + +function unavailable( + cause?: unknown, +): CopilotFailureDiagnosisToolExecutionUnavailableError { + return new CopilotFailureDiagnosisToolExecutionUnavailableError({ + cause: cause instanceof Error ? cause : undefined, + }); +} + +function sqlState(error: unknown): string | undefined { + if (!error || typeof error !== 'object') return undefined; + const value = (error as { code?: unknown }).code; + return typeof value === 'string' ? value : undefined; +} + +function mapStorageError(error: unknown): Error { + if ( + error instanceof CopilotFailureDiagnosisToolExecutionConflictError || + error instanceof CopilotFailureDiagnosisToolExecutionUnavailableError + ) { + return error; + } + if (['23503', '23505', '23514'].includes(sqlState(error) ?? '')) { + return new CopilotFailureDiagnosisToolExecutionConflictError( + 'a durable Tool unlock identity is already bound', + ); + } + return unavailable(error); +} + +function text(row: Row, key: string): string { + const value = row[key]; + if (typeof value !== 'string' || value.length === 0) throw unavailable(); + return value; +} + +function integer(row: Row, key: string): number { + const value = row[key]; + if (typeof value === 'number' && Number.isSafeInteger(value)) return value; + if (typeof value === 'string' && /^(0|[1-9]\d*)$/.test(value)) { + const parsed = Number(value); + if (Number.isSafeInteger(parsed)) return parsed; + } + throw unavailable(); +} + +function jsonObject(value: unknown): Record { + let parsed = value; + if (typeof value === 'string') { + try { + parsed = JSON.parse(value); + } catch (cause) { + throw unavailable(cause); + } + } + if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) { + throw unavailable(); + } + return parsed as Record; +} + +function same(left: unknown, right: unknown): boolean { + return isDeepStrictEqual(left, right); +} + +async function begin(client: PostgresClient): Promise { + await client.query('BEGIN ISOLATION LEVEL SERIALIZABLE'); + await client.query(`SELECT set_config('statement_timeout', $1, true)`, [ + '5s', + ]); + await client.query(`SELECT set_config('lock_timeout', $1, true)`, ['2s']); + await client.query( + `SELECT set_config('idle_in_transaction_session_timeout', $1, true)`, + ['5s'], + ); +} + +async function rollback(client: PostgresClient): Promise { + try { + await client.query('ROLLBACK'); + } catch { + // Preserve the original transaction failure. + } +} + +type StoredUnlock = Readonly<{ + receipt: Readonly; + commandDigest: string; +}>; + +const UNLOCK_SELECT = ` + unlock.request_id AS "requestId", + unlock.plan_digest AS "planDigest", + unlock.start_id AS "startId", + unlock.tool_completion_digest AS "toolCompletionDigest", + unlock.model_step_run_id AS "modelStepRunId", + unlock.model_step_run_version AS "modelStepRunVersion", + unlock.model_step_run_digest AS "modelStepRunDigest", + unlock.model_mutation_id AS "modelMutationId", + unlock.model_mutation_digest AS "modelMutationDigest", + unlock.model_event_id AS "modelEventId", + unlock.final_run_version AS "finalRunVersion", + unlock.final_run_event_sequence AS "finalRunEventSequence", + unlock.unlocked_at_ms AS "unlockedAtMs", + unlock.receipt_digest AS "receiptDigest", + unlock.command_digest AS "commandDigest", + unlock.receipt_json AS "receiptJson", + admission.plan_digest AS "joinedPlanDigest", + completion.completion_digest AS "joinedCompletionDigest", + mutation.mutation_digest AS "joinedMutationDigest", + mutation.step_run_digest AS "joinedModelStepRunDigest", + mutation.event_id AS "joinedEventId", + mutation.event_sequence AS "joinedEventSequence", + mutation.run_version AS "joinedRunVersion"`; + +async function findRows( + queryable: PostgresQueryable, + where: string, + values: readonly unknown[], +): Promise { + const result = await queryable.query( + `SELECT ${UNLOCK_SELECT} + FROM "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${UNLOCK_TABLE}" AS unlock + JOIN "${POSTGRES_MODEL_INVOCATION_SCHEMA}". + "copilot_failure_diagnosis_admissions" AS admission + ON admission.request_id = unlock.request_id + JOIN "ql3"."tool_execution_completions" AS completion + ON completion.start_id = unlock.start_id + JOIN "ql3"."step_run_mutations" AS mutation + ON mutation.mutation_id = unlock.model_mutation_id + WHERE ${where} + LIMIT 2`, + values, + ); + return result.rows; +} + +function storedUnlock(row: Row): StoredUnlock { + let receipt: Readonly; + try { + receipt = normalizeCopilotFailureDiagnosisToolUnlockReceipt( + jsonObject( + row.receiptJson, + ) as unknown as CopilotFailureDiagnosisToolUnlockReceipt, + ); + } catch (cause) { + throw unavailable(cause); + } + if ( + text(row, 'requestId') !== receipt.requestId || + text(row, 'planDigest') !== receipt.planDigest || + text(row, 'startId') !== receipt.startId || + text(row, 'toolCompletionDigest') !== receipt.toolCompletionDigest || + text(row, 'modelStepRunId') !== receipt.modelStepRunId || + integer(row, 'modelStepRunVersion') !== receipt.modelStepRunVersion || + text(row, 'modelStepRunDigest') !== receipt.modelStepRunDigest || + text(row, 'modelMutationId') !== receipt.modelMutationId || + text(row, 'modelMutationDigest') !== receipt.modelMutationDigest || + text(row, 'modelEventId') !== receipt.modelEventId || + integer(row, 'finalRunVersion') !== receipt.finalRunVersion || + integer(row, 'finalRunEventSequence') !== receipt.finalRunEventSequence || + integer(row, 'unlockedAtMs') !== receipt.unlockedAtMs || + text(row, 'receiptDigest') !== receipt.receiptDigest || + text(row, 'joinedPlanDigest') !== receipt.planDigest || + text(row, 'joinedCompletionDigest') !== receipt.toolCompletionDigest || + text(row, 'joinedMutationDigest') !== receipt.modelMutationDigest || + text(row, 'joinedModelStepRunDigest') !== receipt.modelStepRunDigest || + text(row, 'joinedEventId') !== receipt.modelEventId || + integer(row, 'joinedEventSequence') !== receipt.finalRunEventSequence || + integer(row, 'joinedRunVersion') !== receipt.finalRunVersion + ) { + throw unavailable(); + } + return Object.freeze({ + receipt, + commandDigest: text(row, 'commandDigest'), + }); +} + +async function updateModelStepRun( + client: PostgresClient, + command: Readonly, +): Promise { + const mutation = command.modelStepRunMutation; + const step = mutation.stepRun; + const result = await client.query( + `UPDATE "ql3"."step_runs" + SET status = $1, version = $2, attempt_count = $3, + output_ref = $4, approval_request_id = $5, ready_at_ms = $6, + started_at_ms = $7, finished_at_ms = $8, result_code = $9, + error_summary = $10, updated_at_ms = $11, + last_mutation_id = $12, step_run_digest = $13, + step_run_json = $14::jsonb + WHERE id = $15 AND run_id = $16 AND version = $17 + AND step_run_digest = $18 AND status = $19`, + [ + step.status, + step.version, + step.attemptCount, + step.outputRef, + step.approvalRequestId, + step.readyAtMs, + step.startedAtMs, + step.finishedAtMs, + step.resultCode, + step.errorSummary, + step.updatedAtMs, + step.lastMutationId, + step.stepRunDigest, + JSON.stringify(step), + step.id, + step.runId, + mutation.expectedStepRunVersion, + mutation.expectedStepRunDigest, + mutation.previousStatus, + ], + ); + if (result.rowCount !== 1) { + throw new CopilotFailureDiagnosisToolExecutionConflictError( + 'the Model StepRun unlock fence changed', + ); + } +} + +async function updateRun( + client: PostgresClient, + command: Readonly, +): Promise { + const mutation = command.modelStepRunMutation; + const result = await client.query( + `UPDATE "ql3"."runs" + SET version = version + 1, event_sequence = event_sequence + 1 + WHERE id = $1 AND status = 'running' + AND version = $2 AND event_sequence = $3`, + [ + mutation.runId, + mutation.expectedRunVersion, + mutation.expectedRunEventSequence, + ], + ); + if (result.rowCount !== 1) { + throw new CopilotFailureDiagnosisToolExecutionConflictError( + 'the diagnosis Run unlock fence changed', + ); + } +} + +async function insertEventAndMutation( + client: PostgresClient, + command: Readonly, +): Promise { + const mutation = command.modelStepRunMutation; + const event = mutation.event; + await client.query( + `INSERT INTO "ql3"."run_events" ( + id, run_id, sequence, type, dedupe_key, actor_type, actor_id, + attempt_id, step_run_id, payload, created_at_ms + ) VALUES ($1, $2, $3, $4, $5, $6, $7, NULL, $8, $9::jsonb, $10)`, + [ + event.id, + event.runId, + event.sequence, + event.type, + event.dedupeKey, + event.actorType, + event.actorId ?? null, + mutation.stepRun.id, + JSON.stringify(event.payload), + event.createdAtMs, + ], + ); + await client.query( + `INSERT INTO "ql3"."step_run_mutations" ( + mutation_id, mutation_digest, run_id, step_run_id, + step_run_digest, event_id, event_sequence, run_version, + step_run_json, committed_at_ms + ) VALUES ( + $1, $2, $3, $4, $5, $6, $7, $8, $9::jsonb, + floor(extract(epoch FROM transaction_timestamp()) * 1000)::bigint + )`, + [ + mutation.mutationId, + mutation.mutationDigest, + mutation.runId, + mutation.stepRun.id, + mutation.stepRun.stepRunDigest, + mutation.event.id, + mutation.event.sequence, + mutation.expectedRunVersion + 1, + JSON.stringify(mutation.stepRun), + ], + ); +} + +async function insertUnlock( + client: PostgresClient, + command: Readonly, +): Promise { + const receipt = command.receipt; + await client.query( + `INSERT INTO "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${UNLOCK_TABLE}" ( + request_id, plan_digest, run_id, start_id, tool_step_run_id, + tool_completion_digest, + model_step_run_id, model_step_run_version, model_step_run_digest, + model_mutation_id, model_mutation_digest, model_event_id, + final_run_version, final_run_event_sequence, unlocked_at_ms, + receipt_digest, command_digest, receipt_json + ) VALUES ( + $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, + $14, $15, $16, $17, $18::jsonb + )`, + [ + receipt.requestId, + receipt.planDigest, + receipt.runId, + receipt.startId, + receipt.toolStepRunId, + receipt.toolCompletionDigest, + receipt.modelStepRunId, + receipt.modelStepRunVersion, + receipt.modelStepRunDigest, + receipt.modelMutationId, + receipt.modelMutationDigest, + receipt.modelEventId, + receipt.finalRunVersion, + receipt.finalRunEventSequence, + receipt.unlockedAtMs, + receipt.receiptDigest, + command.commandDigest, + JSON.stringify(receipt), + ], + ); +} + +async function currentEvidence( + client: PostgresClient, + command: Readonly, +): Promise { + const mutation = command.modelStepRunMutation; + const result = await client.query( + `SELECT admission.plan_json AS "planJson", + completion.completion_json AS "completionJson", + model_step.kind AS "modelKind", + model_step.status AS "modelStatus", + model_step.version AS "modelVersion", + model_step.step_run_digest AS "modelDigest", + run.project_id AS "projectId", run.status AS "runStatus", + run.version AS "runVersion", + run.event_sequence AS "runEventSequence" + FROM "${POSTGRES_MODEL_INVOCATION_SCHEMA}". + "copilot_failure_diagnosis_admissions" AS admission + JOIN "ql3"."tool_execution_completions" AS completion + ON completion.start_id = $2 + AND completion.run_id = admission.run_id + AND completion.step_run_id = admission.tool_step_run_id + JOIN "ql3"."step_runs" AS model_step + ON model_step.run_id = admission.run_id + AND model_step.id = admission.model_step_run_id + JOIN "ql3"."runs" AS run ON run.id = admission.run_id + WHERE admission.request_id = $1 + LIMIT 2 + FOR UPDATE OF model_step, run`, + [command.plan.requestId, command.completion.startId], + ); + const row = result.rows[0]; + let plan: Readonly; + let completion: Readonly; + try { + plan = row + ? normalizeCopilotFailureDiagnosisExecutionPlan( + jsonObject( + row.planJson, + ) as unknown as CopilotFailureDiagnosisExecutionPlan, + ) + : command.plan; + completion = row + ? normalizeToolExecutionCompletionRecord( + jsonObject( + row.completionJson, + ) as unknown as ToolExecutionCompletionRecord, + ) + : command.completion; + } catch (cause) { + throw unavailable(cause); + } + if ( + result.rows.length !== 1 || + !row || + !same(plan, command.plan) || + !same(completion, command.completion) || + text(row, 'modelKind') !== 'model' || + text(row, 'modelStatus') !== mutation.previousStatus || + integer(row, 'modelVersion') !== mutation.expectedStepRunVersion || + text(row, 'modelDigest') !== mutation.expectedStepRunDigest || + text(row, 'projectId') !== command.plan.projectId || + text(row, 'runStatus') !== 'running' || + integer(row, 'runVersion') !== mutation.expectedRunVersion || + integer(row, 'runEventSequence') !== mutation.expectedRunEventSequence + ) { + throw new CopilotFailureDiagnosisToolExecutionConflictError( + 'the admitted Tool completion or Model StepRun fence changed', + ); + } +} + +export class PostgresCopilotFailureDiagnosisToolUnlockRepository + implements CopilotFailureDiagnosisToolUnlockRepository +{ + constructor(private readonly pool: PostgresPool) { + if ( + !pool || + typeof pool.query !== 'function' || + typeof pool.connect !== 'function' + ) { + throw unavailable(); + } + } + + async findByRequestId( + requestId: string, + ): Promise | null> { + if ( + typeof requestId !== 'string' || + !/^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/.test(requestId) + ) { + throw new TypeError('Tool unlock request id is invalid'); + } + try { + const rows = await findRows(this.pool, 'unlock.request_id = $1', [ + requestId, + ]); + if (rows.length > 1) throw unavailable(); + return rows[0] ? storedUnlock(rows[0]).receipt : null; + } catch (error) { + throw mapStorageError(error); + } + } + + async commit(commandValue: CopilotFailureDiagnosisToolUnlockCommand): Promise< + Readonly<{ + status: 'created' | 'existing'; + receipt: Readonly; + }> + > { + const command = + normalizeCopilotFailureDiagnosisToolUnlockCommand(commandValue); + for (let attempt = 0; attempt < MAX_TRANSACTION_ATTEMPTS; attempt += 1) { + let client: PostgresClient; + try { + client = await this.pool.connect(); + } catch (cause) { + throw unavailable(cause); + } + let began = false; + try { + await begin(client); + began = true; + const existingRows = await findRows( + client, + `unlock.request_id = $1 + OR unlock.tool_completion_digest = $2 + OR unlock.model_mutation_id = $3 + OR unlock.model_event_id = $4`, + [ + command.receipt.requestId, + command.receipt.toolCompletionDigest, + command.receipt.modelMutationId, + command.receipt.modelEventId, + ], + ); + if (existingRows.length > 1) { + throw new CopilotFailureDiagnosisToolExecutionConflictError(); + } + if (existingRows[0]) { + const stored = storedUnlock(existingRows[0]); + if ( + stored.commandDigest !== command.commandDigest || + !same(stored.receipt, command.receipt) + ) { + throw new CopilotFailureDiagnosisToolExecutionConflictError(); + } + await client.query('COMMIT'); + began = false; + return Object.freeze({ + status: 'existing' as const, + receipt: stored.receipt, + }); + } + + await currentEvidence(client, command); + await updateModelStepRun(client, command); + await updateRun(client, command); + await insertEventAndMutation(client, command); + await insertUnlock(client, command); + await client.query('COMMIT'); + began = false; + return Object.freeze({ + status: 'created' as const, + receipt: command.receipt, + }); + } catch (error) { + if (began) await rollback(client); + if ( + RETRYABLE_SQL_STATES.has(sqlState(error) ?? '') && + attempt + 1 < MAX_TRANSACTION_ATTEMPTS + ) { + continue; + } + throw mapStorageError(error); + } finally { + client.release(); + } + } + throw unavailable(); + } +} diff --git a/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/unlockProtocol.ts b/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/unlockProtocol.ts new file mode 100644 index 00000000..81753dfa --- /dev/null +++ b/packages/ql3-ai/src/copilot/failure-diagnosis/tool-execution/unlockProtocol.ts @@ -0,0 +1,424 @@ +import { Buffer } from 'node:buffer'; +import { createHash } from 'node:crypto'; + +import { + normalizeStepRunMutation, + normalizeStepRunRecord, + transitionStepRunMutation, + type StepRunRecord, +} from '@qinglong/runtime-core/step-run'; +import { + normalizeToolExecutionCompletionRecord, + type ToolExecutionCompletionRecord, +} from '@qinglong/runtime-core/tool-execution-completion'; + +import type { CopilotFailureDiagnosisExecutionPlan } from '../admission/contracts'; +import { normalizeCopilotFailureDiagnosisExecutionPlan } from '../admission/plan'; +import { + COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA, + COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA, + InvalidCopilotFailureDiagnosisToolExecutionError, + MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_BYTES, + MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_BYTES, + type CopilotFailureDiagnosisRunAuthority, + type CopilotFailureDiagnosisToolUnlockCommand, + type CopilotFailureDiagnosisToolUnlockReceipt, +} from './contracts'; + +const RECEIPT_DIGEST_DOMAIN = Buffer.from( + 'qinglong/copilot-failure-diagnosis-tool-unlock-receipt-digest@v1\0', + 'utf8', +); +const COMMAND_DIGEST_DOMAIN = Buffer.from( + 'qinglong/copilot-failure-diagnosis-tool-unlock-command-digest@v1\0', + 'utf8', +); +const IDENTITY_DOMAIN = Buffer.from( + 'qinglong/copilot-failure-diagnosis-tool-unlock-identity@v1\0', + 'utf8', +); +const DIGEST_PATTERN = /^[0-9a-f]{64}$/; +const IDENTITY_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/; + +function invalid(message: string): never { + throw new InvalidCopilotFailureDiagnosisToolExecutionError(message); +} + +function hash(domain: Uint8Array, value: unknown): string { + return createHash('sha256') + .update(domain) + .update(JSON.stringify(value)) + .digest('hex'); +} + +function dataRecord(value: unknown, label: string): Record { + if ( + !value || + typeof value !== 'object' || + Array.isArray(value) || + Object.getPrototypeOf(value) !== Object.prototype + ) { + return invalid(`${label} must be a plain object`); + } + return value as Record; +} + +function exactKeys( + value: object, + keys: readonly string[], + label: string, +): void { + const actual = Object.keys(value).sort(); + const expected = [...keys].sort(); + if ( + actual.length !== expected.length || + actual.some((key, index) => key !== expected[index]) + ) { + invalid(`${label} shape is invalid`); + } +} + +function identity(value: unknown, label: string): string { + if (typeof value !== 'string' || !IDENTITY_PATTERN.test(value)) { + return invalid(`${label} is invalid`); + } + return value; +} + +function digest(value: unknown, label: string): string { + if (typeof value !== 'string' || !DIGEST_PATTERN.test(value)) { + return invalid(`${label} is invalid`); + } + return value; +} + +function integer(value: unknown, minimum: number, label: string): number { + if (!Number.isSafeInteger(value) || (value as number) < minimum) { + return invalid(`${label} is invalid`); + } + return value as number; +} + +function evidenceIdentity( + prefix: 'cdum' | 'cdue', + planDigest: string, + completionDigest: string, +): string { + const maximumDigestLength = 35 - prefix.length; + return `${prefix}:${hash(IDENTITY_DOMAIN, { + prefix, + planDigest, + completionDigest, + }).slice(0, maximumDigestLength)}`; +} + +export function copilotFailureDiagnosisToolUnlockReceiptDigest( + value: Omit, +): string { + return hash(RECEIPT_DIGEST_DOMAIN, value); +} + +export function normalizeCopilotFailureDiagnosisToolUnlockReceipt( + value: CopilotFailureDiagnosisToolUnlockReceipt, +): Readonly { + const candidate = dataRecord(value, 'Tool unlock receipt'); + exactKeys( + candidate, + [ + 'barrierDigest', + 'finalRunEventSequence', + 'finalRunVersion', + 'modelEventId', + 'modelMutationDigest', + 'modelMutationId', + 'modelStepRunDigest', + 'modelStepRunId', + 'modelStepRunVersion', + 'planDigest', + 'receiptDigest', + 'requestId', + 'resultArtifact', + 'runId', + 'schema', + 'startId', + 'toolCompletionDigest', + 'toolStepRunId', + 'unlockedAtMs', + ], + 'Tool unlock receipt', + ); + if (value.schema !== COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA) { + return invalid('Tool unlock receipt schema is unsupported'); + } + const artifact = dataRecord(value.resultArtifact, 'result Artifact'); + exactKeys( + artifact, + ['artifactDigest', 'artifactId', 'executionResultDigest', 'outputDigest'], + 'result Artifact', + ); + const unsigned = Object.freeze({ + schema: COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA, + requestId: identity(value.requestId, 'request id'), + planDigest: digest(value.planDigest, 'plan digest'), + runId: identity(value.runId, 'Run id'), + startId: identity(value.startId, 'start id'), + barrierDigest: digest(value.barrierDigest, 'barrier digest'), + toolStepRunId: identity(value.toolStepRunId, 'Tool StepRun id'), + toolCompletionDigest: digest( + value.toolCompletionDigest, + 'Tool completion digest', + ), + resultArtifact: Object.freeze({ + artifactId: identity(value.resultArtifact.artifactId, 'Artifact id'), + artifactDigest: digest( + value.resultArtifact.artifactDigest, + 'Artifact digest', + ), + outputDigest: digest(value.resultArtifact.outputDigest, 'output digest'), + executionResultDigest: digest( + value.resultArtifact.executionResultDigest, + 'execution result digest', + ), + }), + modelStepRunId: identity(value.modelStepRunId, 'model StepRun id'), + modelStepRunVersion: integer( + value.modelStepRunVersion, + 2, + 'model StepRun version', + ), + modelStepRunDigest: digest( + value.modelStepRunDigest, + 'model StepRun digest', + ), + modelMutationId: identity(value.modelMutationId, 'model mutation id'), + modelMutationDigest: digest( + value.modelMutationDigest, + 'model mutation digest', + ), + modelEventId: identity(value.modelEventId, 'model event id'), + finalRunVersion: integer(value.finalRunVersion, 1, 'final Run version'), + finalRunEventSequence: integer( + value.finalRunEventSequence, + 1, + 'final Run event sequence', + ), + unlockedAtMs: integer(value.unlockedAtMs, 0, 'unlock time'), + } satisfies Omit); + if (unsigned.finalRunVersion !== unsigned.finalRunEventSequence) { + return invalid('Tool unlock Run fence is invalid'); + } + const receiptDigest = digest(value.receiptDigest, 'receipt digest'); + if ( + copilotFailureDiagnosisToolUnlockReceiptDigest(unsigned) !== receiptDigest + ) { + return invalid('Tool unlock receipt digest does not match'); + } + const normalized = Object.freeze({ ...unsigned, receiptDigest }); + if ( + Buffer.byteLength(JSON.stringify(normalized), 'utf8') > + MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_BYTES + ) { + return invalid('Tool unlock receipt exceeds its byte budget'); + } + return normalized; +} + +function commandUnsigned( + value: Readonly, +): Omit { + const { commandDigest: _commandDigest, ...unsigned } = value; + return unsigned; +} + +function validateCommandBindings( + plan: Readonly, + completion: Readonly, + mutation: ReturnType, + receipt: Readonly, +): void { + if ( + completion.projectId !== plan.projectId || + completion.runId !== plan.runId || + completion.stepRunId !== plan.toolStepRunId || + mutation.runId !== plan.runId || + mutation.previousStatus !== 'pending' || + mutation.stepRun.id !== plan.modelStepRunId || + mutation.stepRun.runId !== plan.runId || + mutation.stepRun.parentStepRunId !== plan.toolStepRunId || + mutation.stepRun.kind !== 'model' || + mutation.stepRun.status !== 'ready' || + mutation.stepRun.inputRef !== `tool-result-step:${plan.toolStepRunId}` || + mutation.stepRun.updatedAtMs !== completion.completedAtMs || + receipt.requestId !== plan.requestId || + receipt.planDigest !== plan.planDigest || + receipt.runId !== plan.runId || + receipt.startId !== completion.startId || + receipt.barrierDigest !== completion.barrierDigest || + receipt.toolStepRunId !== completion.stepRunId || + receipt.toolCompletionDigest !== completion.completionDigest || + JSON.stringify(receipt.resultArtifact) !== + JSON.stringify(completion.resultArtifact) || + receipt.modelStepRunId !== mutation.stepRun.id || + receipt.modelStepRunVersion !== mutation.stepRun.version || + receipt.modelStepRunDigest !== mutation.stepRun.stepRunDigest || + receipt.modelMutationId !== mutation.mutationId || + receipt.modelMutationDigest !== mutation.mutationDigest || + receipt.modelEventId !== mutation.event.id || + receipt.finalRunVersion !== mutation.expectedRunVersion + 1 || + receipt.finalRunEventSequence !== mutation.expectedRunEventSequence + 1 || + receipt.unlockedAtMs !== mutation.stepRun.updatedAtMs + ) { + return invalid('Tool unlock command bindings are inconsistent'); + } +} + +export function normalizeCopilotFailureDiagnosisToolUnlockCommand( + value: CopilotFailureDiagnosisToolUnlockCommand, +): Readonly { + const candidate = dataRecord(value, 'Tool unlock command'); + exactKeys( + candidate, + [ + 'commandDigest', + 'completion', + 'modelStepRunMutation', + 'plan', + 'receipt', + 'schema', + ], + 'Tool unlock command', + ); + if (value.schema !== COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA) { + return invalid('Tool unlock command schema is unsupported'); + } + const plan = normalizeCopilotFailureDiagnosisExecutionPlan(value.plan); + const completion = normalizeToolExecutionCompletionRecord(value.completion); + const modelStepRunMutation = normalizeStepRunMutation( + value.modelStepRunMutation, + ); + const receipt = normalizeCopilotFailureDiagnosisToolUnlockReceipt( + value.receipt, + ); + validateCommandBindings(plan, completion, modelStepRunMutation, receipt); + const unsigned = Object.freeze({ + schema: COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA, + plan, + completion, + modelStepRunMutation, + receipt, + } satisfies Omit); + const commandDigest = digest(value.commandDigest, 'command digest'); + if (hash(COMMAND_DIGEST_DOMAIN, unsigned) !== commandDigest) { + return invalid('Tool unlock command digest does not match'); + } + const normalized = Object.freeze({ ...unsigned, commandDigest }); + if ( + Buffer.byteLength(JSON.stringify(normalized), 'utf8') > + MAX_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_BYTES + ) { + return invalid('Tool unlock command exceeds its byte budget'); + } + return normalized; +} + +export function createCopilotFailureDiagnosisToolUnlockCommand(input: { + readonly plan: CopilotFailureDiagnosisExecutionPlan; + readonly completion: ToolExecutionCompletionRecord; + readonly modelStepRun: StepRunRecord; + readonly run: CopilotFailureDiagnosisRunAuthority; +}): Readonly { + const plan = normalizeCopilotFailureDiagnosisExecutionPlan(input.plan); + const completion = normalizeToolExecutionCompletionRecord(input.completion); + const modelStepRun = normalizeStepRunRecord(input.modelStepRun); + const run = dataRecord(input.run, 'Run authority'); + exactKeys( + run, + ['eventSequence', 'id', 'projectId', 'status', 'version'], + 'Run authority', + ); + if ( + input.run.id !== plan.runId || + input.run.projectId !== plan.projectId || + input.run.status !== 'running' || + !Number.isSafeInteger(input.run.version) || + input.run.version < 1 || + !Number.isSafeInteger(input.run.eventSequence) || + input.run.eventSequence !== input.run.version || + completion.projectId !== plan.projectId || + completion.runId !== plan.runId || + completion.stepRunId !== plan.toolStepRunId || + modelStepRun.id !== plan.modelStepRunId || + modelStepRun.runId !== plan.runId || + modelStepRun.parentStepRunId !== plan.toolStepRunId || + modelStepRun.kind !== 'model' || + modelStepRun.status !== 'pending' + ) { + return invalid('Tool completion cannot unlock the model StepRun'); + } + const mutationId = evidenceIdentity( + 'cdum', + plan.planDigest, + completion.completionDigest, + ); + const eventId = evidenceIdentity( + 'cdue', + plan.planDigest, + completion.completionDigest, + ); + const modelStepRunMutation = transitionStepRunMutation( + modelStepRun, + { + expectedVersion: modelStepRun.version, + expectedDigest: modelStepRun.stepRunDigest, + mutationId, + to: 'ready', + atMs: completion.completedAtMs, + }, + { + expectedRunVersion: input.run.version, + expectedRunEventSequence: input.run.eventSequence, + eventId, + dedupeKey: eventId, + actor: Object.freeze({ + type: 'system' as const, + id: 'copilot-runtime', + }), + }, + ); + const receiptUnsigned = Object.freeze({ + schema: COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA, + requestId: plan.requestId, + planDigest: plan.planDigest, + runId: plan.runId, + startId: completion.startId, + barrierDigest: completion.barrierDigest, + toolStepRunId: completion.stepRunId, + toolCompletionDigest: completion.completionDigest, + resultArtifact: completion.resultArtifact, + modelStepRunId: modelStepRunMutation.stepRun.id, + modelStepRunVersion: modelStepRunMutation.stepRun.version, + modelStepRunDigest: modelStepRunMutation.stepRun.stepRunDigest, + modelMutationId: modelStepRunMutation.mutationId, + modelMutationDigest: modelStepRunMutation.mutationDigest, + modelEventId: modelStepRunMutation.event.id, + finalRunVersion: modelStepRunMutation.expectedRunVersion + 1, + finalRunEventSequence: modelStepRunMutation.expectedRunEventSequence + 1, + unlockedAtMs: completion.completedAtMs, + } satisfies Omit); + const receipt = normalizeCopilotFailureDiagnosisToolUnlockReceipt({ + ...receiptUnsigned, + receiptDigest: + copilotFailureDiagnosisToolUnlockReceiptDigest(receiptUnsigned), + }); + const unsigned = Object.freeze({ + schema: COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_COMMAND_SCHEMA, + plan, + completion, + modelStepRunMutation, + receipt, + } satisfies Omit); + return normalizeCopilotFailureDiagnosisToolUnlockCommand({ + ...unsigned, + commandDigest: hash(COMMAND_DIGEST_DOMAIN, unsigned), + }); +} diff --git a/packages/ql3-ai/src/copilot/failure-diagnosis/toolExecution.ts b/packages/ql3-ai/src/copilot/failure-diagnosis/toolExecution.ts new file mode 100644 index 00000000..2079d5d7 --- /dev/null +++ b/packages/ql3-ai/src/copilot/failure-diagnosis/toolExecution.ts @@ -0,0 +1,4 @@ +export * from './tool-execution/contracts'; +export * from './tool-execution/coordinator'; +export * from './tool-execution/planAuthority'; +export * from './tool-execution/unlockProtocol'; diff --git a/packages/ql3-ai/src/migration/model-invocation/identities.ts b/packages/ql3-ai/src/migration/model-invocation/identities.ts index 6bd877cf..610a3cb8 100644 --- a/packages/ql3-ai/src/migration/model-invocation/identities.ts +++ b/packages/ql3-ai/src/migration/model-invocation/identities.ts @@ -63,6 +63,8 @@ export const POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID = 'pg-9017-ai-plugin-package-prompt-product-authorization'; export const POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID = 'pg-9018-ai-copilot-failure-diagnosis-admissions'; +export const POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID = + 'pg-9019-ai-copilot-failure-diagnosis-tool-unlocks'; export const POSTGRES_MODEL_INVOCATION_SCHEMA = 'ql3_ai'; export const LOCAL_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE = 'QingLong3AiSchemaMigrations'; diff --git a/packages/ql3-ai/src/migration/model-invocation/postgres.ts b/packages/ql3-ai/src/migration/model-invocation/postgres.ts index aab56723..1b4202a2 100644 --- a/packages/ql3-ai/src/migration/model-invocation/postgres.ts +++ b/packages/ql3-ai/src/migration/model-invocation/postgres.ts @@ -28,6 +28,7 @@ import { POSTGRES_PLUGIN_PACKAGE_PROMPT_OUTPUT_KEY_ROTATION_MIGRATION_ID, POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID, POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID, + POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID, POSTGRES_MODEL_INVOCATION_SCHEMA, POSTGRES_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE, } from './identities'; @@ -63,6 +64,7 @@ const POSTGRES_HISTORY_IDENTITY = Object.freeze({ POSTGRES_MODEL_PROVIDER_CREDENTIAL_TEST_CONNECTION_MIGRATION_ID, POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID, POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID, + POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID, ]), streamId: POSTGRES_MODEL_INVOCATION_MIGRATION_STREAM_ID, dialect: 'postgresql' as const, diff --git a/packages/ql3-ai/src/migration/model-invocation/postgres/copilot.ts b/packages/ql3-ai/src/migration/model-invocation/postgres/copilot.ts index 764b75d3..b37cbbb6 100644 --- a/packages/ql3-ai/src/migration/model-invocation/postgres/copilot.ts +++ b/packages/ql3-ai/src/migration/model-invocation/postgres/copilot.ts @@ -2,6 +2,7 @@ import type { PostgresQueryable } from '@qinglong/runtime-core'; import { POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID, + POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID, POSTGRES_MODEL_INVOCATION_SCHEMA, } from '../identities'; import { defineSqlMigration } from '../shared'; @@ -9,6 +10,7 @@ import { defineSqlMigration } from '../shared'; const ADMISSION_TABLE = 'copilot_failure_diagnosis_admissions'; const SOURCE_SNAPSHOT_FUNCTION = 'copilot_failure_diagnosis_admission_source_snapshot'; +const TOOL_UNLOCK_TABLE = 'copilot_failure_diagnosis_tool_unlocks'; const POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_TABLE_SQL = ` CREATE TABLE "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${ADMISSION_TABLE}" ( @@ -215,6 +217,109 @@ const postgresCopilotFailureDiagnosisAdmissionMigration = (context, statement) => context.query(statement).then(() => undefined), ); +const POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_TABLE_SQL = ` +CREATE TABLE "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${TOOL_UNLOCK_TABLE}" ( + request_id varchar(128) PRIMARY KEY, + plan_digest char(64) NOT NULL UNIQUE, + run_id varchar(36) NOT NULL, + start_id varchar(36) NOT NULL UNIQUE, + tool_step_run_id varchar(128) NOT NULL, + tool_completion_digest char(64) NOT NULL UNIQUE, + model_step_run_id varchar(128) NOT NULL UNIQUE, + model_step_run_version integer NOT NULL, + model_step_run_digest char(64) NOT NULL, + model_mutation_id varchar(36) NOT NULL UNIQUE, + model_mutation_digest char(64) NOT NULL UNIQUE, + model_event_id varchar(36) NOT NULL UNIQUE, + final_run_version integer NOT NULL, + final_run_event_sequence integer NOT NULL, + unlocked_at_ms bigint NOT NULL, + receipt_digest char(64) NOT NULL UNIQUE, + command_digest char(64) NOT NULL UNIQUE, + receipt_json jsonb NOT NULL, + CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_admission_fk + FOREIGN KEY (request_id) + REFERENCES "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${ADMISSION_TABLE}" + (request_id) ON DELETE RESTRICT, + CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_completion_fk + FOREIGN KEY (start_id) + REFERENCES "ql3"."tool_execution_completions" (start_id) + ON DELETE RESTRICT, + CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_model_step_fk + FOREIGN KEY (run_id, model_step_run_id) + REFERENCES "ql3"."step_runs" (run_id, id) ON DELETE RESTRICT, + CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_mutation_fk + FOREIGN KEY (model_mutation_id) + REFERENCES "ql3"."step_run_mutations" (mutation_id) + ON DELETE RESTRICT, + CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_event_fk + FOREIGN KEY (model_event_id) + REFERENCES "ql3"."run_events" (id) ON DELETE RESTRICT, + CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_identity_check CHECK ( + request_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' AND + run_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$' AND + start_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$' AND + tool_step_run_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' AND + model_step_run_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' AND + model_mutation_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$' AND + model_event_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$' + ), + CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_version_check CHECK ( + model_step_run_version >= 2 AND + final_run_version >= 1 AND + final_run_event_sequence = final_run_version AND + unlocked_at_ms >= 0 + ), + CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_digest_check CHECK ( + plan_digest ~ '^[0-9a-f]{64}$' AND + tool_completion_digest ~ '^[0-9a-f]{64}$' AND + model_step_run_digest ~ '^[0-9a-f]{64}$' AND + model_mutation_digest ~ '^[0-9a-f]{64}$' AND + receipt_digest ~ '^[0-9a-f]{64}$' AND + command_digest ~ '^[0-9a-f]{64}$' + ), + CONSTRAINT ql3_ai_copilot_diagnosis_tool_unlock_json_check CHECK ( + jsonb_typeof(receipt_json) = 'object' AND + octet_length(receipt_json::text) BETWEEN 2 AND 16384 AND + receipt_json @> jsonb_build_object( + 'schema', + 'qinglong/copilot-failure-diagnosis-tool-unlock-receipt@v1', + 'requestId', request_id, 'planDigest', plan_digest, + 'runId', run_id, 'startId', start_id, + 'toolStepRunId', tool_step_run_id, + 'toolCompletionDigest', tool_completion_digest, + 'modelStepRunId', model_step_run_id, + 'modelStepRunVersion', model_step_run_version, + 'modelStepRunDigest', model_step_run_digest, + 'modelMutationId', model_mutation_id, + 'modelMutationDigest', model_mutation_digest, + 'modelEventId', model_event_id, + 'finalRunVersion', final_run_version, + 'finalRunEventSequence', final_run_event_sequence, + 'unlockedAtMs', unlocked_at_ms, 'receiptDigest', receipt_digest + ) + ) +)`; + +const postgresCopilotFailureDiagnosisToolUnlockMigration = + defineSqlMigration( + POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID, + [ + POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_TABLE_SQL, + `CREATE UNIQUE INDEX ql3_ai_copilot_diagnosis_tool_unlock_fence_uidx + ON "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${TOOL_UNLOCK_TABLE}" + (run_id, tool_step_run_id, model_step_run_id)`, + `REVOKE ALL ON TABLE + "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${TOOL_UNLOCK_TABLE}" + FROM PUBLIC`, + `GRANT SELECT, INSERT ON TABLE + "${POSTGRES_MODEL_INVOCATION_SCHEMA}"."${TOOL_UNLOCK_TABLE}" + TO ql3_runtime`, + ], + (context, statement) => context.query(statement).then(() => undefined), + ); + export const postgresCopilotMigrations = Object.freeze([ postgresCopilotFailureDiagnosisAdmissionMigration, + postgresCopilotFailureDiagnosisToolUnlockMigration, ]); diff --git a/packages/ql3-ai/test/failureDiagnosisExecutionAdmission.test.cjs b/packages/ql3-ai/test/failureDiagnosisExecutionAdmission.test.cjs index 4c0cd9a8..c4a033fe 100644 --- a/packages/ql3-ai/test/failureDiagnosisExecutionAdmission.test.cjs +++ b/packages/ql3-ai/test/failureDiagnosisExecutionAdmission.test.cjs @@ -1,4 +1,5 @@ const assert = require('node:assert/strict'); +const { createHash } = require('node:crypto'); const { test } = require('node:test'); const { @@ -33,6 +34,12 @@ const { const { storedJsonEquals, } = require('../dist/copilot/failure-diagnosis/admission/postgresRepository.js'); +const { + COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA, + createCopilotFailureDiagnosisToolUnlockCommand, + normalizeCopilotFailureDiagnosisToolUnlockCommand, + restoreCopilotFailureDiagnosisTrustedToolAuthority, +} = require('../dist/copilot/failure-diagnosis/toolExecution.js'); const DIGEST_A = 'a'.repeat(64); const DIGEST_B = 'b'.repeat(64); @@ -189,6 +196,40 @@ async function plan(options = {}) { }); } +function successfulToolCompletion(current) { + const unsigned = Object.freeze({ + schema: 'qinglong/tool-execution-completion@v1', + startId: 'cds:diagnosis-tool-start', + projectId: current.projectId, + runId: current.runId, + stepRunId: current.toolStepRunId, + startedStepRunVersion: 2, + completedStepRunVersion: 3, + barrierDigest: DIGEST_A, + adapterDigest: DIGEST_B, + resultArtifact: Object.freeze({ + artifactId: 'cdra:diagnosis-result', + artifactDigest: DIGEST_C, + outputDigest: DIGEST_A, + executionResultDigest: DIGEST_B, + }), + stepRunMutationId: 'cdscm:diagnosis-tool', + stepRunMutationDigest: DIGEST_C, + completedStepRunDigest: DIGEST_A, + runEventId: 'cdsce:diagnosis-tool', + completedAtMs: 3_000, + }); + return Object.freeze({ + ...unsigned, + completionDigest: createHash('sha256') + .update( + Buffer.from('qinglong/tool-execution-completion-digest@v1\0', 'utf8'), + ) + .update(JSON.stringify(unsigned)) + .digest('hex'), + }); +} + test('creates one independent diagnosis Run with ready Tool and pending Model Steps', async () => { const current = await plan(); assert.equal(current.schema, COPILOT_FAILURE_DIAGNOSIS_EXECUTION_PLAN_SCHEMA); @@ -224,6 +265,10 @@ test('creates one independent diagnosis Run with ready Tool and pending Model St }, { kind: 'tool', status: 'ready', sequence: 2 }, ); + assert.equal( + bundle.toolStepMutation.stepRun.definitionRef, + 'tool:qinglong.run.log.excerpt@1.0.0', + ); assert.deepEqual( { kind: bundle.modelStepMutation.stepRun.kind, @@ -257,6 +302,63 @@ test('creates one independent diagnosis Run with ready Tool and pending Model St ); }); +test('restores the exact admitted Tool and binds success to one Model unlock mutation', async () => { + const current = await plan(); + const admission = createCopilotFailureDiagnosisAdmissionBundle(current); + const authority = restoreCopilotFailureDiagnosisTrustedToolAuthority( + current, + snapshot(), + ); + assert.equal(authority.plan.planDigest, current.tool.planDigest); + assert.equal(authority.binding.bindingDigest, current.tool.bindingDigest); + + const command = createCopilotFailureDiagnosisToolUnlockCommand({ + plan: current, + completion: successfulToolCompletion(current), + modelStepRun: admission.modelStepMutation.stepRun, + run: { + id: current.runId, + projectId: current.projectId, + status: 'running', + version: 5, + eventSequence: 5, + }, + }); + assert.equal( + command.receipt.schema, + COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_RECEIPT_SCHEMA, + ); + assert.equal(command.modelStepRunMutation.previousStatus, 'pending'); + assert.equal(command.modelStepRunMutation.stepRun.status, 'ready'); + assert.equal(command.receipt.finalRunVersion, 6); + assert.deepEqual( + normalizeCopilotFailureDiagnosisToolUnlockCommand( + JSON.parse(JSON.stringify(command)), + ), + command, + ); + assert.throws( + () => + normalizeCopilotFailureDiagnosisToolUnlockCommand({ + ...command, + receipt: { ...command.receipt, finalRunVersion: 7 }, + }), + TypeError, + ); +}); + +test('publishes Tool execution only through explicit AI subpaths', () => { + const root = require('../dist'); + const execution = require('@qinglong/ai/failure-diagnosis-tool-execution'); + const storage = require('@qinglong/ai/postgres-failure-diagnosis-tool-execution-storage'); + assert.equal(root.executeCopilotFailureDiagnosisTool, undefined); + assert.equal(typeof execution.executeCopilotFailureDiagnosisTool, 'function'); + assert.equal( + typeof storage.PostgresCopilotFailureDiagnosisToolUnlockRepository, + 'function', + ); +}); + test('rejects a Tool input detached from the source failure fence', async () => { await assert.rejects( () => plan({ inputAttemptId: 'attempt-unrelated' }), diff --git a/packages/ql3-ai/test/modelInvocationMigration.test.cjs b/packages/ql3-ai/test/modelInvocationMigration.test.cjs index bc3fa5fa..54c024dc 100644 --- a/packages/ql3-ai/test/modelInvocationMigration.test.cjs +++ b/packages/ql3-ai/test/modelInvocationMigration.test.cjs @@ -20,6 +20,7 @@ const { LOCAL_MODEL_PRICE_CATALOG_AUTHORIZATION_MIGRATION_ID, LOCAL_MODEL_PRICE_CATALOG_MIGRATION_ID, POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID, + POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID, POSTGRES_MODEL_INVOCATION_MIGRATION_ID, POSTGRES_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE, POSTGRES_MODEL_INVOCATION_MIGRATION_STREAM_ID, @@ -426,6 +427,10 @@ test('PostgreSQL AI schema is an independent reviewed feature stream', async () POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID, 'pg-9018-ai-copilot-failure-diagnosis-admissions', ); + assert.equal( + POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID, + 'pg-9019-ai-copilot-failure-diagnosis-tool-unlocks', + ); assert.equal( POSTGRES_MODEL_INVOCATION_MIGRATION_HISTORY_TABLE, 'ai_schema_migrations', @@ -507,9 +512,13 @@ test('PostgreSQL AI schema is an independent reviewed feature stream', async () postgresModelInvocationMigrationDefinition.migrations[17].checksum, 'cd0837c68ecc6c2bce58d048308d0239397b6347b4483f02372f966c05ae7ad6', ); + assert.equal( + postgresModelInvocationMigrationDefinition.migrations[18].checksum, + '341733fde691e81efafabc9749197514d83af6c851e91ea55d85074d2dec6b4f', + ); assert.equal( postgresModelInvocationMigrationDefinition.migrations.length, - 18, + 19, ); const diagnosisAdmissionStatements = []; @@ -537,6 +546,25 @@ test('PostgreSQL AI schema is an independent reviewed feature stream', async () assert.match(diagnosisAdmissionSql, /TO ql3_runtime/); assert.doesNotMatch(diagnosisAdmissionSql, /GRANT[^;]*(?:UPDATE|DELETE)/); + const diagnosisToolUnlockStatements = []; + await postgresModelInvocationMigrationDefinition.migrations[18].up({ + async query(statement) { + diagnosisToolUnlockStatements.push(statement); + return { rows: [] }; + }, + }); + const diagnosisToolUnlockSql = diagnosisToolUnlockStatements.join('\n'); + assert.match( + diagnosisToolUnlockSql, + /CREATE TABLE "ql3_ai"\."copilot_failure_diagnosis_tool_unlocks"/, + ); + assert.match( + diagnosisToolUnlockSql, + /FOREIGN KEY \(start_id\)[\s\S]*tool_execution_completions/, + ); + assert.match(diagnosisToolUnlockSql, /TO ql3_runtime/); + assert.doesNotMatch(diagnosisToolUnlockSql, /GRANT[^;]*(?:UPDATE|DELETE)/); + const retirementStatements = []; await postgresModelInvocationMigrationDefinition.migrations[10].up({ async query(statement) { diff --git a/packages/ql3-ai/test/postgresModelProviderCredentialManagementIdentityLedger.test.cjs b/packages/ql3-ai/test/postgresModelProviderCredentialManagementIdentityLedger.test.cjs index 2c75c65d..eabe1595 100644 --- a/packages/ql3-ai/test/postgresModelProviderCredentialManagementIdentityLedger.test.cjs +++ b/packages/ql3-ai/test/postgresModelProviderCredentialManagementIdentityLedger.test.cjs @@ -175,7 +175,7 @@ test('readiness binds exact migration history and least-privilege primary author assert.equal(report.ready, true); assert.equal( report.migrationIds.at(-1), - 'pg-9018-ai-copilot-failure-diagnosis-admissions', + 'pg-9019-ai-copilot-failure-diagnosis-tool-unlocks', ); assert.match( queries[1], diff --git a/packages/ql3-ai/test/postgresModelProviderCredentialTestExecution.test.cjs b/packages/ql3-ai/test/postgresModelProviderCredentialTestExecution.test.cjs index 94206a78..78a998de 100644 --- a/packages/ql3-ai/test/postgresModelProviderCredentialTestExecution.test.cjs +++ b/packages/ql3-ai/test/postgresModelProviderCredentialTestExecution.test.cjs @@ -277,7 +277,7 @@ test('tester readiness freezes migration history and least privilege', async () assert.equal(ready.ready, true); assert.equal( ready.migrationIds.at(-1), - 'pg-9018-ai-copilot-failure-diagnosis-admissions', + 'pg-9019-ai-copilot-failure-diagnosis-tool-unlocks', ); await assert.rejects( diff --git a/packages/ql3-cluster-control/package.json b/packages/ql3-cluster-control/package.json index 55643668..53f29168 100644 --- a/packages/ql3-cluster-control/package.json +++ b/packages/ql3-cluster-control/package.json @@ -125,6 +125,11 @@ "require": "./dist/trusted-tool/key-management/toolResultProjectedKeyring.js", "default": "./dist/trusted-tool/key-management/toolResultProjectedKeyring.js" }, + "./trusted-tool-invocation-keyring": { + "types": "./dist/trusted-tool/key-management/toolInvocationProjectedKeyring.d.ts", + "require": "./dist/trusted-tool/key-management/toolInvocationProjectedKeyring.js", + "default": "./dist/trusted-tool/key-management/toolInvocationProjectedKeyring.js" + }, "./remote-completion": { "types": "./dist/remote-execution/remoteWorkerCompletionService.d.ts", "require": "./dist/remote-execution/remoteWorkerCompletionService.js", diff --git a/packages/ql3-cluster-control/src/trusted-tool/key-management/toolInvocationKeyringManifest.ts b/packages/ql3-cluster-control/src/trusted-tool/key-management/toolInvocationKeyringManifest.ts new file mode 100644 index 00000000..ae22e268 --- /dev/null +++ b/packages/ql3-cluster-control/src/trusted-tool/key-management/toolInvocationKeyringManifest.ts @@ -0,0 +1,178 @@ +import { Buffer } from 'node:buffer'; +import { createHash } from 'node:crypto'; + +export const CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA = + 'qinglong/cluster-tool-invocation-projected-keyring@v1' as const; +export const MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES = 64 * 1024; +export const MAX_CLUSTER_TOOL_INVOCATION_PROJECTED_KEYS = 16; + +const KEY_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/; +const BASE64URL_PATTERN = /^[A-Za-z0-9_-]+$/; +const PROJECTION_DIGEST_DOMAIN = Buffer.from( + 'qinglong/cluster-tool-invocation-projected-keyring-digest@v1\0', + 'utf8', +); + +export interface ClusterToolInvocationKeyringManifest { + readonly schema: typeof CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA; + readonly activeKeyId: string; + readonly keys: Readonly>; +} + +export interface ClusterToolInvocationKeyringSummary { + readonly schemaVersion: 1; + readonly activeKeyId: string; + readonly keyIds: readonly string[]; + readonly projectionDigest: string; +} + +export class InvalidClusterToolInvocationKeyringManifestError extends TypeError { + readonly code = 'QL3_CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_INVALID'; + + constructor() { + super('Cluster Tool invocation keyring manifest is invalid'); + this.name = 'InvalidClusterToolInvocationKeyringManifestError'; + } +} + +function invalid(): never { + throw new InvalidClusterToolInvocationKeyringManifestError(); +} + +function dataRecord(value: unknown): Record { + if ( + !value || + typeof value !== 'object' || + Array.isArray(value) || + Object.getPrototypeOf(value) !== Object.prototype + ) { + return invalid(); + } + return value as Record; +} + +function exactKeys(value: object, expected: readonly string[]): boolean { + const actual = Object.keys(value).sort(); + const canonical = [...expected].sort(); + return ( + actual.length === canonical.length && + actual.every((key, index) => key === canonical[index]) + ); +} + +export function normalizeClusterToolInvocationKeyringManifest( + value: unknown, +): Readonly { + const manifest = dataRecord(value); + if ( + !exactKeys(manifest, ['activeKeyId', 'keys', 'schema']) || + manifest.schema !== CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA || + typeof manifest.activeKeyId !== 'string' || + !KEY_ID_PATTERN.test(manifest.activeKeyId) + ) { + return invalid(); + } + const keys = dataRecord(manifest.keys); + const entries = Object.entries(keys).sort(([left], [right]) => + left < right ? -1 : left > right ? 1 : 0, + ); + if ( + entries.length < 1 || + entries.length > MAX_CLUSTER_TOOL_INVOCATION_PROJECTED_KEYS + ) { + return invalid(); + } + const normalized: (readonly [string, string])[] = []; + for (const [keyId, encoded] of entries) { + let material: Buffer | undefined; + try { + if ( + !KEY_ID_PATTERN.test(keyId) || + typeof encoded !== 'string' || + !BASE64URL_PATTERN.test(encoded) + ) { + return invalid(); + } + material = Buffer.from(encoded, 'base64url'); + if ( + material.byteLength !== 32 || + material.toString('base64url') !== encoded + ) { + return invalid(); + } + normalized.push(Object.freeze([keyId, encoded] as const)); + } finally { + material?.fill(0); + } + } + const normalizedKeys = Object.freeze(Object.fromEntries(normalized)); + if (normalizedKeys[manifest.activeKeyId] === undefined) return invalid(); + return Object.freeze({ + schema: CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA, + activeKeyId: manifest.activeKeyId, + keys: normalizedKeys, + }); +} + +export function parseClusterToolInvocationKeyringManifest( + bytes: Buffer, +): Readonly { + try { + if ( + !Buffer.isBuffer(bytes) || + bytes.byteLength < 1 || + bytes.byteLength > MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES + ) { + return invalid(); + } + return normalizeClusterToolInvocationKeyringManifest( + JSON.parse(bytes.toString('utf8')), + ); + } catch (error) { + if (error instanceof InvalidClusterToolInvocationKeyringManifestError) { + throw error; + } + return invalid(); + } +} + +export function canonicalClusterToolInvocationKeyringManifest( + value: ClusterToolInvocationKeyringManifest, +): Buffer { + const manifest = normalizeClusterToolInvocationKeyringManifest(value); + return Buffer.from(`${JSON.stringify(manifest)}\n`, 'utf8'); +} + +export function resolveClusterToolInvocationKeyringMaterial( + value: ClusterToolInvocationKeyringManifest, + keyId: string, +): Readonly<{ keyId: string; key: Uint8Array }> | null { + if (typeof keyId !== 'string' || !KEY_ID_PATTERN.test(keyId)) { + return invalid(); + } + const manifest = normalizeClusterToolInvocationKeyringManifest(value); + const encoded = manifest.keys[keyId]; + if (encoded === undefined) return null; + const key = Buffer.from(encoded, 'base64url'); + if (key.byteLength !== 32 || key.toString('base64url') !== encoded) { + key.fill(0); + return invalid(); + } + return Object.freeze({ keyId, key }); +} + +export function summarizeClusterToolInvocationKeyringManifest( + value: ClusterToolInvocationKeyringManifest, +): Readonly { + const manifest = normalizeClusterToolInvocationKeyringManifest(value); + const keyIds = Object.freeze(Object.keys(manifest.keys).sort()); + return Object.freeze({ + schemaVersion: 1 as const, + activeKeyId: manifest.activeKeyId, + keyIds, + projectionDigest: createHash('sha256') + .update(PROJECTION_DIGEST_DOMAIN) + .update(JSON.stringify(manifest)) + .digest('hex'), + }); +} diff --git a/packages/ql3-cluster-control/src/trusted-tool/key-management/toolInvocationProjectedKeyring.ts b/packages/ql3-cluster-control/src/trusted-tool/key-management/toolInvocationProjectedKeyring.ts new file mode 100644 index 00000000..64a3df3e --- /dev/null +++ b/packages/ql3-cluster-control/src/trusted-tool/key-management/toolInvocationProjectedKeyring.ts @@ -0,0 +1,151 @@ +import { Buffer } from 'node:buffer'; + +import type { ToolInvocationArtifactKeyProvider } from '@qinglong/runtime-core/tool-invocation-artifact'; + +import { PrivateProjectedFileReader } from '../../security/privateProjectedFile'; +import { + MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES, + canonicalClusterToolInvocationKeyringManifest, + parseClusterToolInvocationKeyringManifest, + resolveClusterToolInvocationKeyringMaterial, + summarizeClusterToolInvocationKeyringManifest, + type ClusterToolInvocationKeyringManifest, + type ClusterToolInvocationKeyringSummary, +} from './toolInvocationKeyringManifest'; + +const DATA_FILE_NAME = /^[A-Za-z0-9][A-Za-z0-9._-]{0,252}$/; + +export { + CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA, + MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES, + MAX_CLUSTER_TOOL_INVOCATION_PROJECTED_KEYS, + InvalidClusterToolInvocationKeyringManifestError, + canonicalClusterToolInvocationKeyringManifest, + normalizeClusterToolInvocationKeyringManifest, + parseClusterToolInvocationKeyringManifest, + resolveClusterToolInvocationKeyringMaterial, + summarizeClusterToolInvocationKeyringManifest, + type ClusterToolInvocationKeyringManifest, + type ClusterToolInvocationKeyringSummary, +} from './toolInvocationKeyringManifest'; + +export interface ClusterToolInvocationProjectedKeyringOptions { + readonly rootDirectory: string; + readonly dataFileName?: string; +} + +export class ClusterToolInvocationProjectedKeyringUnavailableError extends Error { + readonly code = 'QL3_CLUSTER_TOOL_INVOCATION_PROJECTED_KEYRING_UNAVAILABLE'; + + constructor(options?: ErrorOptions) { + super('Projected Cluster Tool invocation keyring is unavailable', options); + this.name = 'ClusterToolInvocationProjectedKeyringUnavailableError'; + } +} + +function unavailable( + cause?: unknown, +): ClusterToolInvocationProjectedKeyringUnavailableError { + return new ClusterToolInvocationProjectedKeyringUnavailableError({ + cause: cause instanceof Error ? cause : undefined, + }); +} + +function dataFileName(value: unknown): string { + if (typeof value !== 'string' || !DATA_FILE_NAME.test(value)) { + throw unavailable(); + } + return value; +} + +async function readManifest( + reader: PrivateProjectedFileReader, + fileName: string, +): Promise> { + let bytes: Buffer | undefined; + let canonical: Buffer | undefined; + try { + bytes = await reader.read(fileName); + const manifest = parseClusterToolInvocationKeyringManifest(bytes); + canonical = canonicalClusterToolInvocationKeyringManifest(manifest); + if (!canonical.equals(bytes)) throw unavailable(); + return manifest; + } catch (cause) { + throw cause instanceof ClusterToolInvocationProjectedKeyringUnavailableError + ? cause + : unavailable(cause); + } finally { + bytes?.fill(0); + canonical?.fill(0); + } +} + +/** Read-only, no-cache invocation Artifact key authority for projections. */ +export class ClusterToolInvocationProjectedKeyring + implements ToolInvocationArtifactKeyProvider +{ + readonly #reader: PrivateProjectedFileReader; + readonly #dataFileName: string; + + constructor(options: ClusterToolInvocationProjectedKeyringOptions) { + if (!options || typeof options !== 'object' || Array.isArray(options)) { + throw unavailable(); + } + try { + this.#reader = new PrivateProjectedFileReader({ + rootDirectory: options.rootDirectory, + minimumBytes: 1, + maximumBytes: MAX_CLUSTER_TOOL_INVOCATION_KEYRING_BYTES, + access: 'read_only_keyring', + }); + this.#dataFileName = dataFileName(options.dataFileName ?? 'keyring.json'); + } catch (cause) { + throw unavailable(cause); + } + } + + async verify(): Promise> { + return summarizeClusterToolInvocationKeyringManifest( + await readManifest(this.#reader, this.#dataFileName), + ); + } + + async active(): ReturnType { + try { + const manifest = await readManifest(this.#reader, this.#dataFileName); + return resolveClusterToolInvocationKeyringMaterial( + manifest, + manifest.activeKeyId, + )!; + } catch (cause) { + throw cause instanceof + ClusterToolInvocationProjectedKeyringUnavailableError + ? cause + : unavailable(cause); + } + } + + async resolve( + keyId: string, + ): ReturnType { + try { + return resolveClusterToolInvocationKeyringMaterial( + await readManifest(this.#reader, this.#dataFileName), + keyId, + ); + } catch (cause) { + throw cause instanceof + ClusterToolInvocationProjectedKeyringUnavailableError + ? cause + : unavailable(cause); + } + } +} + +export async function createClusterToolInvocationProjectedKeyring( + options: ClusterToolInvocationProjectedKeyringOptions, +): Promise> { + const provider = new ClusterToolInvocationProjectedKeyring(options); + await provider.verify(); + return provider; +} diff --git a/packages/ql3-cluster-control/test/tool-invocation-projected-keyring.test.cjs b/packages/ql3-cluster-control/test/tool-invocation-projected-keyring.test.cjs new file mode 100644 index 00000000..99a6f21b --- /dev/null +++ b/packages/ql3-cluster-control/test/tool-invocation-projected-keyring.test.cjs @@ -0,0 +1,123 @@ +const assert = require('node:assert/strict'); +const fs = require('node:fs/promises'); +const os = require('node:os'); +const path = require('node:path'); +const { afterEach, test } = require('node:test'); + +const { + CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA, + ClusterToolInvocationProjectedKeyring, + ClusterToolInvocationProjectedKeyringUnavailableError, + canonicalClusterToolInvocationKeyringManifest, + createClusterToolInvocationProjectedKeyring, +} = require('@qinglong/cluster-control/trusted-tool-invocation-keyring'); + +const roots = []; + +afterEach(async () => { + await Promise.all( + roots + .splice(0) + .map((root) => fs.rm(root, { recursive: true, force: true })), + ); +}); + +async function tempRoot() { + const root = await fs.mkdtemp( + path.join(os.tmpdir(), 'ql3-tool-invocation-keyring-'), + ); + roots.push(root); + return root; +} + +async function publish(root, generationName, manifest, mode = 0o440) { + const generation = path.join(root, generationName); + await fs.mkdir(generation, { mode: 0o750 }); + const target = path.join(generation, 'keyring.json'); + await fs.writeFile( + target, + canonicalClusterToolInvocationKeyringManifest(manifest), + { mode }, + ); + await fs.chmod(target, mode); + const next = path.join(root, '..data-next'); + await fs.symlink(generationName, next); + await fs.rename(next, path.join(root, '..data')); + try { + await fs.symlink('..data/keyring.json', path.join(root, 'keyring.json')); + } catch (error) { + if (error.code !== 'EEXIST') throw error; + } +} + +function manifest(activeKeyId, keys) { + return Object.freeze({ + schema: CLUSTER_TOOL_INVOCATION_KEYRING_MANIFEST_SCHEMA, + activeKeyId, + keys: Object.freeze(keys), + }); +} + +test('projected Tool invocation keyring rotates active material without caching', async () => { + const root = await tempRoot(); + const keyOne = Buffer.alloc(32, 0x41); + const keyTwo = Buffer.alloc(32, 0x42); + await publish( + root, + '..2026_08_15_01', + manifest('invocation-key-one', { + 'invocation-key-one': keyOne.toString('base64url'), + }), + ); + const provider = await createClusterToolInvocationProjectedKeyring({ + rootDirectory: root, + }); + const first = await provider.active(); + assert.equal(first.keyId, 'invocation-key-one'); + assert.deepEqual(Buffer.from(first.key), keyOne); + first.key.fill(0); + + await publish( + root, + '..2026_08_15_02', + manifest('invocation-key-two', { + 'invocation-key-one': keyOne.toString('base64url'), + 'invocation-key-two': keyTwo.toString('base64url'), + }), + ); + const second = await provider.active(); + assert.equal(second.keyId, 'invocation-key-two'); + assert.deepEqual(Buffer.from(second.key), keyTwo); + second.key.fill(0); + const historical = await provider.resolve('invocation-key-one'); + assert.ok(historical); + assert.deepEqual(Buffer.from(historical.key), keyOne); + historical.key.fill(0); + assert.equal(await provider.resolve('missing-key'), null); +}); + +test('projected Tool invocation keyring rejects missing active and writable material', async () => { + const root = await tempRoot(); + const key = Buffer.alloc(32, 0x51).toString('base64url'); + assert.throws( + () => + canonicalClusterToolInvocationKeyringManifest( + manifest('missing-key', { 'invocation-key-one': key }), + ), + TypeError, + ); + + await publish( + root, + '..2026_08_15_01', + manifest('invocation-key-one', { 'invocation-key-one': key }), + 0o640, + ); + await assert.rejects( + () => + new ClusterToolInvocationProjectedKeyring({ + rootDirectory: root, + }).verify(), + ClusterToolInvocationProjectedKeyringUnavailableError, + ); +}); diff --git a/scripts/ql3-postgres-ha-contract.cjs b/scripts/ql3-postgres-ha-contract.cjs index fca0b515..006c5e83 100644 --- a/scripts/ql3-postgres-ha-contract.cjs +++ b/scripts/ql3-postgres-ha-contract.cjs @@ -26,7 +26,13 @@ const { PostgresRemoteWorkerCompletionRepository, PostgresRemoteWorkerLeaseControlRepository, PostgresRunAttemptLogRetentionClaimRepository, + PostgresRunRepository, + PostgresToolExecutionCompletionRepository, + PostgresToolExecutionFailureCompletionRepository, + PostgresToolExecutionStartBarrierRepository, PostgresToolInvocationArtifactRepository, + PostgresToolResultKeyCatalogReader, + PostgresToolResultRekeyReader, PostgresWorkerSessionRepository, } = require('../packages/ql3-cluster-postgres/dist/entrypoints/runtime.js'); const { @@ -34,6 +40,7 @@ const { } = require('../packages/ql3-cluster-postgres/dist/migration/migration.js'); const { POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID, + POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID, POSTGRES_MODEL_INVOCATION_MIGRATION_ID, POSTGRES_MODEL_INVOCATION_PRICING_MIGRATION_ID, POSTGRES_MODEL_INVOCATION_QUOTA_MIGRATION_ID, @@ -59,6 +66,12 @@ const { const { PostgresCopilotFailureDiagnosisAdmissionRepository, } = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/postgresAdmissionRepository.js'); +const { + executeCopilotFailureDiagnosisTool, +} = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/toolExecution.js'); +const { + PostgresCopilotFailureDiagnosisToolUnlockRepository, +} = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/postgresToolExecutionRepository.js'); const { BUILTIN_RUN_LOG_EXCERPT_TOOL, BUILTIN_RUN_LOG_EXCERPT_TOOL_DEFINITION, @@ -1472,6 +1485,25 @@ async function modelInvocationFeatureFacts(pool) { 'SELECT,INSERT,UPDATE,DELETE' ) ) AS "copilotDiagnosisAdmissionRuntimeOnly", + has_table_privilege( + 'ql3_runtime', + 'ql3_ai.copilot_failure_diagnosis_tool_unlocks', 'SELECT,INSERT' + ) AND NOT has_table_privilege( + 'ql3_runtime', + 'ql3_ai.copilot_failure_diagnosis_tool_unlocks', 'UPDATE,DELETE' + ) AND NOT EXISTS ( + SELECT 1 + FROM unnest(ARRAY[ + 'ql3_admin', 'ql3_package_manager', 'ql3_package_executor', + 'ql3_worker_ingress', 'ql3_ai_maintenance', + 'ql3_ai_credential_manager', 'ql3_ai_credential_tester' + ]::text[]) AS denied(role_name) + WHERE has_table_privilege( + denied.role_name, + 'ql3_ai.copilot_failure_diagnosis_tool_unlocks', + 'SELECT,INSERT,UPDATE,DELETE' + ) + ) AS "copilotDiagnosisToolUnlockRuntimeOnly", has_function_privilege( 'ql3_runtime', 'ql3_ai.copilot_failure_diagnosis_admission_source_snapshot(varchar,varchar,varchar,integer,integer,varchar,varchar)', @@ -1598,6 +1630,24 @@ async function copilotFailureDiagnosisAdmissionFacts(pool, requestId) { WHERE run_id = admission.run_id) AS "eventCount", (SELECT count(*)::integer FROM "ql3"."step_run_mutations" WHERE run_id = admission.run_id) AS "mutationCount", + (SELECT count(*)::integer + FROM "ql3"."tool_execution_start_barriers" + WHERE run_id = admission.run_id + AND step_run_id = admission.tool_step_run_id) + AS "startCount", + (SELECT count(*)::integer + FROM "ql3"."tool_execution_completions" + WHERE run_id = admission.run_id + AND step_run_id = admission.tool_step_run_id) + AS "completionCount", + (SELECT count(*)::integer + FROM "ql3_ai"."copilot_failure_diagnosis_tool_unlocks" + WHERE request_id = admission.request_id) AS "unlockCount", + (SELECT completion.artifact_json::text + FROM "ql3"."tool_execution_completions" AS completion + WHERE completion.run_id = admission.run_id + AND completion.step_run_id = admission.tool_step_run_id + LIMIT 1) AS "resultArtifactJson", tool.status AS "toolStatus", tool.kind AS "toolKind", model.status AS "modelStatus", model.kind AS "modelKind", model.parent_step_run_id AS "modelParentStepRunId", @@ -1615,11 +1665,20 @@ async function copilotFailureDiagnosisAdmissionFacts(pool, requestId) { [requestId], ); assert.equal(result.rowCount, 1); - const { planJson, receiptJson, ...facts } = result.rows[0]; + const { planJson, receiptJson, resultArtifactJson, ...facts } = + result.rows[0]; assert.equal(typeof planJson, 'string'); assert.equal(typeof receiptJson, 'string'); return { ...facts, + resultArtifactPresent: typeof resultArtifactJson === 'string', + resultArtifactJsonDigest: + typeof resultArtifactJson === 'string' + ? createHash('sha256').update(resultArtifactJson).digest('hex') + : null, + resultArtifactContainsPlaintext: + typeof resultArtifactJson === 'string' && + resultArtifactJson.includes('diagnosis-sensitive-line'), planJsonDigest: createHash('sha256').update(planJson).digest('hex'), receiptJsonDigest: createHash('sha256').update(receiptJson).digest('hex'), }; @@ -1717,7 +1776,7 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) { subject: { type: 'user', id: subjectId }, authenticationId: `diagnosis-auth-${suffix}`, authenticatedAtMs: baseTimeMs - 1_000, - expiresAtMs: baseTimeMs + 60_000, + expiresAtMs: baseTimeMs + 600_000, assurance: 'multi_factor', }); const invocation = await prepareToolInvocation( @@ -1742,7 +1801,7 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) { const bindings = new TrustedToolHandlerBindingRegistry(snapshot, [ createBuiltInRunLogExcerptToolHandlerBinding(snapshot, ['cluster-control']), ]); - const toolPlan = createTrustedToolInvocationPlan(bindings, invocation, { + const toolBundle = createTrustedToolInvocationPlan(bindings, invocation, { actionRef: `diagnosis-log-${suffix}`, inputArtifactId: `diagnosis-input-${suffix}`, previewArtifactId: `diagnosis-preview-${suffix}`, @@ -1760,7 +1819,8 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) { warnings: ['potentially_sensitive_output'], }, sealedAtMs: baseTimeMs + 200, - }).plan; + }); + const toolPlan = toolBundle.plan; const plan = prepareCopilotFailureDiagnosisExecution({ requestId, traceId: `diagnosis-trace-${suffix}`, @@ -1793,6 +1853,12 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) { const repository = new PostgresCopilotFailureDiagnosisAdmissionRepository( runtimeDatabase.pool, ); + assert.deepEqual( + await new PostgresToolInvocationArtifactRepository( + runtimeDatabase.pool, + ).put(toolBundle.inputArtifact, toolBundle.previewArtifact), + { status: 'inserted' }, + ); const first = await repository.admit(plan); const replay = await repository.admit(plan); assert.equal(first.status, 'created'); @@ -1836,20 +1902,221 @@ async function runCopilotFailureDiagnosisAdmissionHaEvidence(options) { }, ); return { - requestId, - sourceRunId, - sourceAttemptId, - runId: plan.runId, - toolStepRunId: plan.toolStepRunId, - modelStepRunId: plan.modelStepRunId, - exactReplay: true, - beforePromotion: facts, + report: { + requestId, + sourceRunId, + sourceAttemptId, + runId: plan.runId, + toolStepRunId: plan.toolStepRunId, + modelStepRunId: plan.modelStepRunId, + exactReplay: true, + toolExecutionExactReplay: false, + encryptedCompletion: false, + beforePromotion: facts, + }, + fixture: { + baseTimeMs, + plan, + principal, + snapshot, + invocationKeyId: toolBundle.inputArtifact.keyId, + invocationKey: Buffer.alloc(32, 0x21), + }, }; } finally { await runtimeDatabase.close(); } } +async function executeCopilotFailureDiagnosisToolHaEvidence(options) { + const { + port, + report, + fixture, + resultKeyFixture, + expectedExisting = false, + } = options; + const runtimeDatabase = await databaseOpener( + 'runtime', + databaseUrl(RUNTIME_USER, RUNTIME_PASSWORD, port), + expectedExisting + ? 'ql3-ha-copilot-diagnosis-tool-promoted' + : 'ql3-ha-copilot-diagnosis-tool-primary', + )(); + let logReads = 0; + const logContent = Buffer.from( + 'diagnosis-sensitive-line password=must-be-redacted', + 'utf8', + ); + const keys = { + async resolve(keyId) { + if (keyId !== fixture.invocationKeyId) return null; + return { keyId, key: Buffer.from(fixture.invocationKey) }; + }, + }; + const resultKeys = { + async resolve(keyId) { + if (keyId !== resultKeyFixture.resultKeyId) return null; + return { + keyId, + key: Buffer.alloc(32, resultKeyFixture.resultKeyByte), + }; + }, + }; + const dependencies = { + admissions: new PostgresCopilotFailureDiagnosisAdmissionRepository( + runtimeDatabase.pool, + ), + snapshots: { + async findCurrent(projectId) { + return projectId === fixture.plan.projectId + ? { snapshot: fixture.snapshot } + : null; + }, + }, + artifacts: new PostgresToolInvocationArtifactRepository( + runtimeDatabase.pool, + ), + invocationKeys: keys, + resultKeys, + stepRuns: new PostgresStepRunRepository(runtimeDatabase.pool), + runs: new PostgresRunRepository(runtimeDatabase.pool), + barriers: new PostgresToolExecutionStartBarrierRepository( + runtimeDatabase.pool, + ), + completions: new PostgresToolExecutionCompletionRepository( + runtimeDatabase.pool, + ), + failureCompletions: new PostgresToolExecutionFailureCompletionRepository( + runtimeDatabase.pool, + ), + resultKeyCatalog: new PostgresToolResultKeyCatalogReader( + runtimeDatabase.pool, + ), + resultRekeys: new PostgresToolResultRekeyReader(runtimeDatabase.pool), + logs: { + async read(request) { + logReads += 1; + assert.equal(request.projectId, fixture.plan.projectId); + assert.equal(request.runId, report.sourceRunId); + assert.equal(request.attemptId, report.sourceAttemptId); + const totalBytes = logContent.byteLength; + const start = Math.min(request.range.offset, totalBytes); + const endExclusive = Math.min(start + request.range.length, totalBytes); + return { + status: 'available', + projectId: request.projectId, + runId: request.runId, + attemptId: request.attemptId, + logArtifactId: fixture.plan.source.logArtifactId, + content: logContent.subarray(start, endExclusive), + start, + endExclusive, + totalBytes, + ...(endExclusive < totalBytes ? { nextOffset: endExclusive } : {}), + truncation: { truncated: false, maximumBytes: 4_194_304 }, + }; + }, + }, + unlocks: new PostgresCopilotFailureDiagnosisToolUnlockRepository( + runtimeDatabase.pool, + ), + now: () => fixture.baseTimeMs + 400, + nonceFactory: () => Buffer.alloc(12, 0x61), + }; + const authorizer = { + async authorize() { + return { + effect: 'allow', + reasons: ['role_grant'], + fence: { projectVersion: 1, bindingVersion: 1 }, + }; + }, + }; + try { + const first = await executeCopilotFailureDiagnosisTool( + { + requestId: report.requestId, + principal: fixture.principal, + authorizer, + }, + dependencies, + ); + assert.equal(first.outcome, 'succeeded'); + assert.equal( + first.completionStatus, + expectedExisting ? 'existing' : 'created', + ); + assert.equal(first.unlockStatus, expectedExisting ? 'existing' : 'created'); + if (!expectedExisting) { + const replay = await executeCopilotFailureDiagnosisTool( + { + requestId: report.requestId, + principal: fixture.principal, + authorizer, + }, + dependencies, + ); + assert.equal(replay.outcome, 'succeeded'); + assert.equal(replay.completionStatus, 'existing'); + assert.equal(replay.unlockStatus, 'existing'); + assert.deepEqual(replay.completion, first.completion); + assert.deepEqual(replay.unlock, first.unlock); + assert.equal(logReads, 2); + } else { + assert.equal(logReads, 0); + } + const facts = await copilotFailureDiagnosisAdmissionFacts( + runtimeDatabase.pool, + report.requestId, + ); + assert.deepEqual( + { + status: facts.status, + version: facts.version, + eventSequence: facts.eventSequence, + eventCount: facts.eventCount, + mutationCount: facts.mutationCount, + startCount: facts.startCount, + completionCount: facts.completionCount, + unlockCount: facts.unlockCount, + toolStatus: facts.toolStatus, + modelStatus: facts.modelStatus, + resultArtifactPresent: facts.resultArtifactPresent, + resultArtifactContainsPlaintext: facts.resultArtifactContainsPlaintext, + }, + { + status: 'running', + version: 6, + eventSequence: 6, + eventCount: 6, + mutationCount: 5, + startCount: 1, + completionCount: 1, + unlockCount: 1, + toolStatus: 'succeeded', + modelStatus: 'ready', + resultArtifactPresent: true, + resultArtifactContainsPlaintext: false, + }, + ); + if (!expectedExisting) { + report.beforePromotion = facts; + report.toolExecutionExactReplay = true; + report.encryptedCompletion = true; + report.primaryToolAdapterReads = logReads; + } else { + report.promotedToolReplayReads = logReads; + report.toolExecutionSurvivedPromotion = true; + } + return facts; + } finally { + fixture.invocationKey.fill(0); + logContent.fill(0); + await runtimeDatabase.close(); + } +} + async function runModelProviderCredentialCatalogMatrix(options) { const { primaryPort, migrationPool } = options; const suffix = `${process.pid}-${randomBytes(3).toString('hex')}`; @@ -10300,6 +10567,7 @@ async function main(argv = process.argv.slice(2)) { let oldPrimaryRejoin; let modelInvocationFeaturePromotion; let copilotFailureDiagnosisAdmission; + let copilotFailureDiagnosisToolFixture; let modelProviderCredentialCatalog; let modelProviderCredentialTestConnection; let runAttemptLogRetentionEvidence; @@ -10398,6 +10666,7 @@ async function main(argv = process.argv.slice(2)) { ); assert.deepEqual(beforePromotion.tables, [ 'copilot_failure_diagnosis_admissions', + 'copilot_failure_diagnosis_tool_unlocks', 'model_invocation_completions', 'model_invocation_price_quotes', 'model_invocation_price_settlements', @@ -10448,6 +10717,7 @@ async function main(argv = process.argv.slice(2)) { POSTGRES_PLUGIN_PACKAGE_PROMPT_OUTPUT_KEY_ROTATION_MIGRATION_ID, POSTGRES_PLUGIN_PACKAGE_PROMPT_PRODUCT_AUTHORIZATION_MIGRATION_ID, POSTGRES_COPILOT_FAILURE_DIAGNOSIS_ADMISSION_MIGRATION_ID, + POSTGRES_COPILOT_FAILURE_DIAGNOSIS_TOOL_UNLOCK_MIGRATION_ID, ], ); assert.deepEqual(beforePromotion.privileges, { @@ -10472,16 +10742,19 @@ async function main(argv = process.argv.slice(2)) { promptOutputKeyRotationAuthoritySplit: true, promptSnapshotRuntimeOnly: true, copilotDiagnosisAdmissionRuntimeOnly: true, + copilotDiagnosisToolUnlockRuntimeOnly: true, copilotDiagnosisSnapshotRuntimeOnly: true, migrationHistoryRuntimeReadOnly: true, modelProviderCredentialManagementAuthoritySplit: true, }); modelInvocationFeaturePromotion = { beforePromotion }; - copilotFailureDiagnosisAdmission = + const copilotDiagnosis = await runCopilotFailureDiagnosisAdmissionHaEvidence({ primaryPort, migrationPool: migrationDatabase.pool, }); + copilotFailureDiagnosisAdmission = copilotDiagnosis.report; + copilotFailureDiagnosisToolFixture = copilotDiagnosis.fixture; } finally { await migrationDatabase.close(); } @@ -10866,6 +11139,26 @@ async function main(argv = process.argv.slice(2)) { toolResultRekeyFaultPool.pool.end(), ]); } + await executeCopilotFailureDiagnosisToolHaEvidence({ + port: primaryPort, + report: copilotFailureDiagnosisAdmission, + fixture: copilotFailureDiagnosisToolFixture, + resultKeyFixture: toolResultKeyFixture, + }); + await waitFor(async () => { + const replicated = await copilotFailureDiagnosisAdmissionFacts( + standbyDatabase.pool, + copilotFailureDiagnosisAdmission.requestId, + ); + return JSON.stringify(replicated) === + JSON.stringify(copilotFailureDiagnosisAdmission.beforePromotion) + ? replicated + : null; + }, 'Copilot failure diagnosis Tool completion WAL replay'); + timeline.push({ + state: 'copilot_failure_diagnosis_tool_completed', + atMs: Number((performance.now() - startedAt).toFixed(3)), + }); await waitFor(async () => { const facts = await standbyDatabase.pool.query( `SELECT @@ -11759,6 +12052,13 @@ async function main(argv = process.argv.slice(2)) { copilotFailureDiagnosisAdmission.afterPromotion = promotedCopilotFailureDiagnosisAdmission; copilotFailureDiagnosisAdmission.survivedPromotion = true; + await executeCopilotFailureDiagnosisToolHaEvidence({ + port: standbyPort, + report: copilotFailureDiagnosisAdmission, + fixture: copilotFailureDiagnosisToolFixture, + resultKeyFixture: toolResultKeyFixture, + expectedExisting: true, + }); await verifyModelProviderCredentialCatalogAfterPromotion({ promotedPort: standbyPort, promotedPool: promotedDatabase.pool, @@ -12513,7 +12813,7 @@ async function main(argv = process.argv.slice(2)) { FROM "ql3"."worker_credential_deliveries") AS "credentialDeliveries"`, ); assert.deepEqual(sideEffects.rows, [ - { runs: 78, runEvents: 179, credentialDeliveries: 4 }, + { runs: 78, runEvents: 182, credentialDeliveries: 4 }, ]); timeline.push({ state: 'two_fresh_control_replicas_ready', @@ -12644,14 +12944,29 @@ async function main(argv = process.argv.slice(2)) { optionalAiFeatureSchemaSurvivesPromotion: true, copilotFailureDiagnosisAdmissionExactlyReplays: copilotFailureDiagnosisAdmission.exactReplay && - copilotFailureDiagnosisAdmission.beforePromotion.eventCount === 3 && - copilotFailureDiagnosisAdmission.beforePromotion.mutationCount === 2, + copilotFailureDiagnosisAdmission.beforePromotion.eventCount === 6 && + copilotFailureDiagnosisAdmission.beforePromotion.mutationCount === 5, + copilotFailureDiagnosisToolCompletesEncryptedAndUnlocksModel: + copilotFailureDiagnosisAdmission.toolExecutionExactReplay && + copilotFailureDiagnosisAdmission.encryptedCompletion && + copilotFailureDiagnosisAdmission.beforePromotion.toolStatus === + 'succeeded' && + copilotFailureDiagnosisAdmission.beforePromotion.modelStatus === + 'ready' && + copilotFailureDiagnosisAdmission.beforePromotion.unlockCount === 1 && + copilotFailureDiagnosisAdmission.beforePromotion + .resultArtifactContainsPlaintext === false, + copilotFailureDiagnosisToolReplaysAfterPromotionWithoutExecution: + copilotFailureDiagnosisAdmission.toolExecutionSurvivedPromotion && + copilotFailureDiagnosisAdmission.promotedToolReplayReads === 0, copilotFailureDiagnosisAdmissionReplicatesAndSurvivesPromotion: copilotFailureDiagnosisAdmission.replicatedBeforePromotion && copilotFailureDiagnosisAdmission.survivedPromotion, copilotFailureDiagnosisAdmissionUsesLeastPrivilegeRuntime: modelInvocationFeaturePromotion.beforePromotion.privileges .copilotDiagnosisAdmissionRuntimeOnly && + modelInvocationFeaturePromotion.beforePromotion.privileges + .copilotDiagnosisToolUnlockRuntimeOnly && modelInvocationFeaturePromotion.beforePromotion.privileges .copilotDiagnosisSnapshotRuntimeOnly, modelProviderCredentialCatalogSurvivesPromotion: diff --git a/test/back/ql3PackageBoundaryAudit.test.cjs b/test/back/ql3PackageBoundaryAudit.test.cjs index 451ae378..c13099f4 100644 --- a/test/back/ql3PackageBoundaryAudit.test.cjs +++ b/test/back/ql3PackageBoundaryAudit.test.cjs @@ -268,10 +268,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( { path: 'packages/ql3-ai', name: '@qinglong/ai', - sourceFiles: 168, + sourceFiles: 175, rootSourceFiles: 1, rootSourceLines: 16, - nestedSourceFiles: 167, + nestedSourceFiles: 174, rootSourceFileHardCap: 1, rootSourceLineHardCap: 16, rootSourceFileRoles: { @@ -385,10 +385,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: clusterControl.rootSourceFileRoles, }, { - sourceFiles: 54, + sourceFiles: 56, rootSourceFiles: 2, rootSourceLines: 195, - nestedSourceFiles: 52, + nestedSourceFiles: 54, rootSourceFileRoles: { 'aiCli.ts': 'binary_entry', 'cli.ts': 'binary_entry',