feat(ql3): expose fenced copilot diagnosis API

This commit is contained in:
whyour
2026-08-15 23:45:13 +08:00
parent 0a31c3364b
commit 774af4a4c9
10 changed files with 845 additions and 5 deletions
+1
View File
@@ -11,6 +11,7 @@
最新增量证据(2026-08-15): 最新增量证据(2026-08-15):
- D-321/ADR-0413(已接受):Cluster Copilot failure diagnosis 已开放唯一、默认关闭的产品写入口 `POST /api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses`。该 route 只在显式 AI 进程且 `QL3_CLUSTER_AI_COPILOT_ENABLED=true` 时注入既有 Cluster Control route registry,完整复用同一认证器、`model.invoke` Project Policy、fail-closed 同步安全审计、HTTP/TLS body/response/concurrency 上限和生命周期;内部只读 Tool 仍独立复验其 exact `tool.call:*` Policy。耐久幂等身份直接绑定 `x-request-id`body 只允许 schema 与 `traceId`Project/source Run/principal 来自 canonical path 和认证,Attempt、日志范围、Tool、provider/model、预算、deadline、Policy fence、reason/outcome 均由服务端从数据库与只读配置派生。响应只投影 created/existing、source/diagnosis Run、终态 stage/reason/outcome 和加密输出 Artifact id/digest,不返回日志、Tool/Model plaintext、prompt、模型信息、密钥或内部异常。普通 Cluster Control、Edge、Standalone 和未启用 Copilot 的 AI 进程没有 route,也没有新增 package、进程、监听器、Pool/连接、timer、watcher、队列、cache、Pod、Service 或 Kubernetes 权限;workspace 仍为 18 package、无单文件/浅平 package,新增源码位于既有嵌套 Copilot 领域目录。Cluster Control 250 pass/2 条件 skip18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail,四项架构审计和 14 档 Local artifact 全部通过;默认 Edge/Standalone 保持 2,589,890/2,589,968 bytes。PostgreSQL 18.6 arm64 HA 137/137、timeline `1→2`,报告 SHA-256 为 `0a12b5c1102555823d43b5a93dd7868b98b194b491840a5242bab6fa2da26123`,离线审计零 finding。本 Gate 不改 migration/schema/role/SQL/HA 拓扑。下一 Gate 优先补同一 Policy 下的加密诊断输出读取与费用/取消可观测性;CLI/UI/MCP 只能复用该 API,不得直连 application capability。
- D-320/ADR-0412(已接受):Cluster Copilot failure diagnosis 已补齐产品入口前的非成功收敛边界。新增的 pre-Model terminalizer 只从 durable Tool failure、受审日志 projection 或数据库观察到的 deadline/cancellation 派生封闭 reason,在 Model start 不存在时以一个 SERIALIZABLE 事务原子提交 StepRun mutations、RunEvents、父 Run 终态和 `pg-9021` append-only content-free receiptTool `failed|timed_out`、日志 `not_found|pending|missing|retired`、五秒 Tool budget 不足、deadline 和 cancellation 均可 exact replay。Model 已开始后的 `outcome_unknown` 继续要求强 User 显式 `fail|cancel|retry` resolutionCopilot finalizer 现在对 `fail|cancel` 精确校验 resolution mutation 的 resolved Step digest,既不伪造失败也不自动重试 Provider。实现复用既有 AI package、Cluster AI 进程、Pool 与 ledger,不增加 package、进程、连接、timer、watcher、队列、cache 或产品 routeworkspace 仍为 18 package、无单文件/浅平 packageAI 192 个源码中 191 个位于嵌套领域目录。AI 完整测试 244 pass/3 条件 skip18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail,四项架构审计和 14 档 Local artifact 全部通过。PostgreSQL 18.6 arm64 HA 137/137、timeline `1→2`,覆盖日志不可用 pre-Model terminalization、unknown completion→人工 fail resolution→Run finalization 及晋升后零外部副作用 exact replay;报告 SHA-256 为 `6eaeb20615a62d153c5a69687344f41f31351c6ecf111cfb9cbafad115538c83`,离线审计零 finding。下一 Gate 才增加认证、Policy、audit 与 source fence 保护的 Cluster API,并由 CLI/UI/MCP 复用,仍不得建立旁路执行器。 - D-320/ADR-0412(已接受):Cluster Copilot failure diagnosis 已补齐产品入口前的非成功收敛边界。新增的 pre-Model terminalizer 只从 durable Tool failure、受审日志 projection 或数据库观察到的 deadline/cancellation 派生封闭 reason,在 Model start 不存在时以一个 SERIALIZABLE 事务原子提交 StepRun mutations、RunEvents、父 Run 终态和 `pg-9021` append-only content-free receiptTool `failed|timed_out`、日志 `not_found|pending|missing|retired`、五秒 Tool budget 不足、deadline 和 cancellation 均可 exact replay。Model 已开始后的 `outcome_unknown` 继续要求强 User 显式 `fail|cancel|retry` resolutionCopilot finalizer 现在对 `fail|cancel` 精确校验 resolution mutation 的 resolved Step digest,既不伪造失败也不自动重试 Provider。实现复用既有 AI package、Cluster AI 进程、Pool 与 ledger,不增加 package、进程、连接、timer、watcher、队列、cache 或产品 routeworkspace 仍为 18 package、无单文件/浅平 packageAI 192 个源码中 191 个位于嵌套领域目录。AI 完整测试 244 pass/3 条件 skip18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail,四项架构审计和 14 档 Local artifact 全部通过。PostgreSQL 18.6 arm64 HA 137/137、timeline `1→2`,覆盖日志不可用 pre-Model terminalization、unknown completion→人工 fail resolution→Run finalization 及晋升后零外部副作用 exact replay;报告 SHA-256 为 `6eaeb20615a62d153c5a69687344f41f31351c6ecf111cfb9cbafad115538c83`,离线审计零 finding。下一 Gate 才增加认证、Policy、audit 与 source fence 保护的 Cluster API,并由 CLI/UI/MCP 复用,仍不得建立旁路执行器。
- D-319/ADR-0411(已接受):Cluster Copilot failure diagnosis 已在既有 `ql3-cluster-control-ai` 进程内完成默认关闭的 production composition。Prompt 与 Copilot 共享同一个 PostgreSQL AI Pool、Model Gateway、Provider client、恢复扫描、quota/pricing ledger 与 `maxConcurrent` 预算;有界 successful-completion router 只向声明 exact invocation 的 durable sink 分发,不复制 Gateway、连接或隐藏队列。application service 只接受 source Run/request identity,从数据库当前 Run、latest Attempt、Project Tool snapshot、Policy 和 canonical read-only egress config 派生计划,先 admission,再以 durable plan、historical key 与确定性 nonce 修复 admission→Artifact crash window,随后复用 Trusted Tool、Worker Artifact range reader、Model Gateway 及独立的 invocation/result/model-output 三域 keyring。能力保持 caller-driven,不增加 timer、watcher、队列、HTTP/CLI/UI/MCP route 或 Kubernetes API 权限;Kubernetes 独立可选 component 仅投影 config 与三个 0440 keyring。最终 AI 238 pass/3 条件 skip、Cluster Control 240 pass/2 条件 skip、18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail,四项架构审计、14 档 Local artifact 全部通过;workspace 仍为 18 package、无单文件/浅平 packageAI 187 个源码中 186 个、Cluster Control 59 个中 57 个位于嵌套领域目录。默认 Edge/Standalone 保持 2,589,890/2,589,968 bytesEdge/Standalone AI 为 3,064,454/3,064,544 bytes,证明 Cluster-only composition 未进入小设备闭包。PostgreSQL 18.6 arm64 HA 130/130、timeline `1→2`,报告 SHA-256 为 `981299b454dce5541e9596450b85816dc40559cba8dc42adf3d5fea571c3d3a6`;本 Gate 不改 migration/schema/role/SQL/HA 拓扑。下一 Gate 是 Tool failure、日志 missing/retired/pending、deadline/cancel 与 Model outcome-unknown 的 durable terminalization/recovery,完成前不开放产品入口。 - D-319/ADR-0411(已接受):Cluster Copilot failure diagnosis 已在既有 `ql3-cluster-control-ai` 进程内完成默认关闭的 production composition。Prompt 与 Copilot 共享同一个 PostgreSQL AI Pool、Model Gateway、Provider client、恢复扫描、quota/pricing ledger 与 `maxConcurrent` 预算;有界 successful-completion router 只向声明 exact invocation 的 durable sink 分发,不复制 Gateway、连接或隐藏队列。application service 只接受 source Run/request identity,从数据库当前 Run、latest Attempt、Project Tool snapshot、Policy 和 canonical read-only egress config 派生计划,先 admission,再以 durable plan、historical key 与确定性 nonce 修复 admission→Artifact crash window,随后复用 Trusted Tool、Worker Artifact range reader、Model Gateway 及独立的 invocation/result/model-output 三域 keyring。能力保持 caller-driven,不增加 timer、watcher、队列、HTTP/CLI/UI/MCP route 或 Kubernetes API 权限;Kubernetes 独立可选 component 仅投影 config 与三个 0440 keyring。最终 AI 238 pass/3 条件 skip、Cluster Control 240 pass/2 条件 skip、18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail,四项架构审计、14 档 Local artifact 全部通过;workspace 仍为 18 package、无单文件/浅平 packageAI 187 个源码中 186 个、Cluster Control 59 个中 57 个位于嵌套领域目录。默认 Edge/Standalone 保持 2,589,890/2,589,968 bytesEdge/Standalone AI 为 3,064,454/3,064,544 bytes,证明 Cluster-only composition 未进入小设备闭包。PostgreSQL 18.6 arm64 HA 130/130、timeline `1→2`,报告 SHA-256 为 `981299b454dce5541e9596450b85816dc40559cba8dc42adf3d5fea571c3d3a6`;本 Gate 不改 migration/schema/role/SQL/HA 拓扑。下一 Gate 是 Tool failure、日志 missing/retired/pending、deadline/cancel 与 Model outcome-unknown 的 durable terminalization/recovery,完成前不开放产品入口。
- D-318/ADR-0410(已接受):Cluster Copilot diagnosis Model output 获得独立的只读 projected key authority。canonical `qinglong/copilot-failure-diagnosis-output-projected-keyring@v1` manifest 只允许一个 active key 与最多 16 个 historical 32-byte key;每次 `active()`/`resolve()` 都重新执行 Cluster 私有投影文件的 direct-root、根内 atomic symlink、single-link、mode、dev/inode/size/mtime 与双 realpath fence,不使用 cache、watcher、timer 或 Kubernetes API。该 authority 以结构兼容的本地窄端口位于既有 `cluster-control/copilot/failure-diagnosis/`,只通过 `failure-diagnosis-output-keyring` subpath 发布,避免 Cluster Control 默认源码反向依赖 AI;它不复用 Prompt output、Tool invocation/result 或 Provider credential key domain,也不提前声称 Copilot 产品入口已可达。定向回归 13/13、Cluster Control 239 pass/2 条件 skip、backend 1,207 pass/2 条件 skip、18-package clean build/test、四项架构审计与 14 档 Local artifact 全部通过;workspace 仍无单文件/浅平 packageCluster Control 58 个源码中 56 个位于嵌套领域目录。默认 Edge/Standalone 保持 2,589,890/2,589,968 bytesEdge/Standalone AI 保持 3,061,009/3,061,099 bytes,证明 Cluster-only subpath 被裁掉。本 Gate 不改 migration/schema/role/SQL/连接/HA 拓扑,因此数据库基线继续引用 ADR-0409 的 PostgreSQL 18.6 arm64 HA 130/130、timeline `1→2`;下一 Gate 是默认关闭的完整 Cluster Copilot composition,随后补齐 Tool failure、日志 missing/retired、Model admission 前 deadline/cancel 与 outcome-unknown 的 durable terminalization/recovery。 - D-318/ADR-0410(已接受):Cluster Copilot diagnosis Model output 获得独立的只读 projected key authority。canonical `qinglong/copilot-failure-diagnosis-output-projected-keyring@v1` manifest 只允许一个 active key 与最多 16 个 historical 32-byte key;每次 `active()`/`resolve()` 都重新执行 Cluster 私有投影文件的 direct-root、根内 atomic symlink、single-link、mode、dev/inode/size/mtime 与双 realpath fence,不使用 cache、watcher、timer 或 Kubernetes API。该 authority 以结构兼容的本地窄端口位于既有 `cluster-control/copilot/failure-diagnosis/`,只通过 `failure-diagnosis-output-keyring` subpath 发布,避免 Cluster Control 默认源码反向依赖 AI;它不复用 Prompt output、Tool invocation/result 或 Provider credential key domain,也不提前声称 Copilot 产品入口已可达。定向回归 13/13、Cluster Control 239 pass/2 条件 skip、backend 1,207 pass/2 条件 skip、18-package clean build/test、四项架构审计与 14 档 Local artifact 全部通过;workspace 仍无单文件/浅平 packageCluster Control 58 个源码中 56 个位于嵌套领域目录。默认 Edge/Standalone 保持 2,589,890/2,589,968 bytesEdge/Standalone AI 保持 3,061,009/3,061,099 bytes,证明 Cluster-only subpath 被裁掉。本 Gate 不改 migration/schema/role/SQL/连接/HA 拓扑,因此数据库基线继续引用 ADR-0409 的 PostgreSQL 18.6 arm64 HA 130/130、timeline `1→2`;下一 Gate 是默认关闭的完整 Cluster Copilot composition,随后补齐 Tool failure、日志 missing/retired、Model admission 前 deadline/cancel 与 outcome-unknown 的 durable terminalization/recovery。
@@ -0,0 +1,109 @@
# ADR-0413:默认关闭的 Cluster Copilot 故障诊断产品 API
- 状态:Accepted
- 日期:2026-08-15
- 关联 RFCQL-RFC-0001 D-321、Phase 2
- 关联 ADRADR-0087、ADR-0407、ADR-0408、ADR-0409、ADR-0411、ADR-0412
## 问题
ADR-0412 已使 Copilot failure diagnosis 的成功、Tool/日志失败和 Model unknown
路径都能耐久收敛,但 capability 仍没有产品入口。直接增加独立 AI HTTP 服务、让 CLI/UI/MCP
调用 application service,或允许调用者提交 Model、Tool、Attempt、日志范围和 outcome,都会绕过
Cluster Control 已有的认证、Project Policy、同步安全审计、资源预算或 source fence。
路由器上的低配 Edge/Standalone 与集群节点还必须保持不同部署闭包:默认部署不能因为一个可选
Copilot API 增加 AI 依赖、进程、监听器、连接池、队列或常驻任务;集群多副本则必须依靠耐久
request identity 和数据库 fence 收敛,而不是依靠单进程锁。
## 决策
1. 唯一产品写入口为
`POST /api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses`operation 是
`copilot.failure_diagnosis.execute`Project permission 是 `model.invoke`。它只注入既有
Cluster Control route registry,完整复用同一认证器、Project Policy 和 fail-closed 同步安全
审计;内部受信 Tool 仍独立复验其精确 `tool.call:*` permission。
2. 路由只在显式 AI 进程且 `QL3_CLUSTER_AI_COPILOT_ENABLED=true` 时注入。普通
`ql3-cluster-control`、Edge、Standalone 和未启用 Copilot 的 AI 进程都返回 `404`,并且不
import/构造 Copilot capability。
3. HTTP `x-request-id` 是耐久 diagnosis request identity;缺省时由既有 HTTP surface 生成并
回传。客户端重试必须复用它。JSON body 采用 exact-key schema,只包含
`schema=qinglong/cluster-copilot-failure-diagnosis-request@v1``traceId`。调用者不能提交
project、source Attempt、日志 Artifact/range、Tool、provider、model、预算、deadline、输出
key、reason 或 outcome。
4. `projectId``sourceRunId` 只从已匹配的 canonical path 获取,principal 只从认证结果获取。
application service 再从当前 Run、latest Attempt、Project Tool snapshot、配置投影和数据库时间
派生 exact source fence、Tool plan、Model intent 与 deadline。HTTP Policy fence 负责产品入口
admission;耐久执行计划仍记录 application 内部重新读取的当前 Policy fence,不能信任调用者
提交的 fence。
5. 响应是 content-free 投影,只返回 schema、request identity、created/existing replay 状态、
source/diagnosis Run identity、终态 outcome/stage/reason,以及成功时的加密输出 Artifact
`artifactId/artifactDigest`。不得返回日志、Tool output、Model plaintext、prompt、provider、
model、密钥材料、内部异常、Policy reasons 或数据库细节。
6. 请求体错误返回稳定 `400`source/idempotency/durable conflict 返回 `409`Tool approval/
policy deny 返回低敏 `403`;容量耗尽返回 `429`Model budget/egress policy 拒绝返回 `422`
caller abort 返回 `408`deadline 返回 `504`;其余依赖和存储错误统一为 `503`。任何响应都
不透传内部 message。
7. 路由是 caller-driven 的薄 transport adapter,不增加 package。实现放入既有
`cluster-control/src/copilot/failure-diagnosis` 领域目录;AI composition 只把已经创建的同一个
application capability 注入 control composition。不得增加进程、监听器、数据库 Pool/连接、
timer、watcher、队列、cache、Pod、Service 或 Kubernetes API 权限。
8. 多副本并发继续以 application request coalescing 加 PostgreSQL exact replay 收敛。进程内
coalescing 仅降低同实例重复工作,不是正确性边界;不同实例对同一 `x-request-id` 的相同请求
必须得到 `created|existing` 同一 diagnosis Run,对不同 source/trace 的复用必须 conflict。
9. 后续 CLI、UI、MCP 只能调用该 HTTP API,不得直接导入 Copilot application/composition
也不得各自定义更宽的请求契约。
## 被否决方案
1. **独立 Copilot HTTP 进程或端口**:复制认证、审计、TLS、限流和生命周期,增加低配与集群
运维成本。
2. **把路由放进默认 Cluster Control**:即使运行时返回 disabled,也会扩大默认制品和依赖闭包。
3. **body 自带 requestId 且与 HTTP requestId 分离**:会产生两个重试/审计身份,难以可靠关联。
4. **调用者选择 provider/model/Tool/日志范围**:扩大数据外发和权限边界,破坏 canonical config
与 source fence。
5. **只检查 `model.invoke`,跳过内部 Tool Policy**:把组合 capability 变成 Tool 权限旁路。
6. **同步返回诊断正文**:使重放、响应大小、日志泄露和生命周期依赖于一次 HTTP 连接。
7. **新增 Copilot API package**:没有独立部署或复用边界,只会制造薄 package。
## 验证标准
1. 路由单元测试覆盖 exact body、HTTP request identity 绑定、path-derived project/source Run、
principal 传递、created/existing replay、成功/非成功 content-free projection、错误低敏映射和
非法输入零 capability call。
2. production composition 测试证明默认 route 不存在;显式 capability 注入后只增加一个 exact
route,并先完成 authentication、`model.invoke` Policy 和同步 `allowed` audit 才读取 body/
调 capability。拒绝路径不得调用 capability。
3. AI composition 测试证明只有显式 Copilot 配置才注入同一个 capabilityPrompt 与 Copilot
仍共享原 Pool/Gatewaystop 顺序和 availability ownership 不变。
4. package/dependency/import/deployment 审计证明 package 数、默认 Edge/Standalone 闭包、进程、
端口、Pool、Pod、Service 与 Kubernetes 权限无增长。
5. 18-package clean build/test、完整 backend、Local artifact、PostgreSQL 18 physical HA 与
GitNexus staged/change detection 全通过后才允许阶段性提交。
## 当前验证
1. 路由与 composition 新增测试覆盖 HTTP request identity、path-derived Project/source Run、认证
principal、exact body、created/existing replay、成功和 pre-Model content-free projection、内部
错误低敏映射、默认 route absence、`model.invoke` deny 零 capability call,以及 AI composition
的同一 capability 注入。Cluster Control 全包 250 pass/2 条件 skip/0 fail;受影响测试格式化后
独立复跑 27/27。
2. 18 个 QL3 package 从清空全部 `dist` 开始完成拓扑构建和全包测试;backend 1,207 pass/2 条件
skip/0 fail。Edge import、Cluster dependency、package boundary 与 Cluster deployment 四项审计
全部兼容且零 finding。
3. workspace 仍为 18 个 package`singleSourcePackages=[]``shallowSourcePackages=[]`。Cluster
Control 60 个源码中 58 个位于嵌套领域目录;唯一新增源码位于既有
`copilot/failure-diagnosis`,没有增加 package 或根部平铺文件。
4. 14 档 Local artifact 全部 `compatible=true`;默认 Edge/Standalone 分别保持
2,589,890/2,589,968 bytesEdge/Standalone AI 为 3,069,143/3,069,233 bytes,证明
Cluster-only 产品路由未进入路由设备或 Local AI 闭包。
5. PostgreSQL 18.6 arm64 physical HA 137/137、timeline `1→2`;既有成功、pre-Model
terminalization、unknown resolution 和晋升后 exact replay 全部通过。私有报告 SHA-256 为
`0a12b5c1102555823d43b5a93dd7868b98b194b491840a5242bab6fa2da26123`,独立离线审计
`compatible=true`、零 finding。本 Gate 未改 migration、schema、role、SQL 或 HA 拓扑。
## 后续门禁
本 Gate 不开放诊断正文读取、列表、取消、unknown resolution、CLI、UI 或 MCP。下一 Gate 应优先
提供同一 Project Policy 下的加密输出 Artifact 读取与费用/取消可观测性;真实外部 Provider 和
多副本并发证据完成前,Copilot 继续保持默认关闭。
@@ -45,6 +45,11 @@
"require": "./dist/copilot/failure-diagnosis/outputProjectedKeyring.js", "require": "./dist/copilot/failure-diagnosis/outputProjectedKeyring.js",
"default": "./dist/copilot/failure-diagnosis/outputProjectedKeyring.js" "default": "./dist/copilot/failure-diagnosis/outputProjectedKeyring.js"
}, },
"./copilot-routes": {
"types": "./dist/copilot/failure-diagnosis/failureDiagnosisRoute.d.ts",
"require": "./dist/copilot/failure-diagnosis/failureDiagnosisRoute.js",
"default": "./dist/copilot/failure-diagnosis/failureDiagnosisRoute.js"
},
"./http": { "./http": {
"types": "./dist/transport/httpSurface.d.ts", "types": "./dist/transport/httpSurface.d.ts",
"require": "./dist/transport/httpSurface.js", "require": "./dist/transport/httpSurface.js",
@@ -445,6 +445,13 @@ export async function startProductionClusterAiControlApplication(
capability: promptApplication.promptExecutionOutputs, capability: promptApplication.promptExecutionOutputs,
}, },
}), }),
...(copilotApplication === undefined
? {}
: {
copilotFailureDiagnosis: {
capability: copilotApplication,
},
}),
}); });
if (controlApplication.status !== 'active') { if (controlApplication.status !== 'active') {
throw new Error('AI-enabled cluster-control did not activate'); throw new Error('AI-enabled cluster-control did not activate');
@@ -61,6 +61,10 @@ import {
} from '../worker-ingress/productionWorkerIngress'; } from '../worker-ingress/productionWorkerIngress';
import type { ClusterWorkerIngressApplicationResult } from '../worker-ingress/workerIngressApplication'; import type { ClusterWorkerIngressApplicationResult } from '../worker-ingress/workerIngressApplication';
import { createClusterControlPluginPackageWorkflowRoutes } from '../plugin-package/workflow/pluginPackageWorkflowRoute'; import { createClusterControlPluginPackageWorkflowRoutes } from '../plugin-package/workflow/pluginPackageWorkflowRoute';
import {
createClusterControlCopilotFailureDiagnosisRoute,
type ClusterCopilotFailureDiagnosisCapability,
} from '../copilot/failure-diagnosis/failureDiagnosisRoute';
export const PRODUCTION_CLUSTER_CONTROL_ROUTE_OPERATIONS = Object.freeze([ export const PRODUCTION_CLUSTER_CONTROL_ROUTE_OPERATIONS = Object.freeze([
'task.get', 'task.get',
@@ -87,6 +91,7 @@ export const PRODUCTION_CLUSTER_CONTROL_OPTIONAL_ROUTE_OPERATIONS =
'prompt.execution.read', 'prompt.execution.read',
'prompt.execution.output.read', 'prompt.execution.output.read',
'prompt.output.read', 'prompt.output.read',
'copilot.failure_diagnosis.execute',
] as const); ] as const);
export interface ProductionClusterControlAssemblyOptions { export interface ProductionClusterControlAssemblyOptions {
@@ -109,6 +114,9 @@ export interface ProductionClusterControlAssemblyOptions {
readonly promptExecutionOutputRead?: Readonly<{ readonly promptExecutionOutputRead?: Readonly<{
readonly capability: ClusterPluginPackagePromptExecutionOutputReadCapability; readonly capability: ClusterPluginPackagePromptExecutionOutputReadCapability;
}>; }>;
readonly copilotFailureDiagnosis?: Readonly<{
readonly capability: ClusterCopilotFailureDiagnosisCapability;
}>;
readonly workerIngress?: Readonly<{ readonly workerIngress?: Readonly<{
readonly config: EnabledClusterWorkerIngressConfig; readonly config: EnabledClusterWorkerIngressConfig;
readonly onDiagnostic?: (error: unknown) => void | Promise<void>; readonly onDiagnostic?: (error: unknown) => void | Promise<void>;
@@ -154,6 +162,9 @@ export interface ProductionClusterControlApplicationOptions
readonly promptExecutionOutputRead?: Readonly<{ readonly promptExecutionOutputRead?: Readonly<{
readonly capability: ClusterPluginPackagePromptExecutionOutputReadCapability; readonly capability: ClusterPluginPackagePromptExecutionOutputReadCapability;
}>; }>;
readonly copilotFailureDiagnosis?: Readonly<{
readonly capability: ClusterCopilotFailureDiagnosisCapability;
}>;
readonly workerIngress?: ProductionClusterWorkerIngressOptions; readonly workerIngress?: ProductionClusterWorkerIngressOptions;
} }
@@ -261,6 +272,13 @@ export function createProductionClusterControlApplicationStack(
options.promptExecutionOutputRead.capability, options.promptExecutionOutputRead.capability,
), ),
]), ]),
...(options.copilotFailureDiagnosis === undefined
? []
: [
createClusterControlCopilotFailureDiagnosisRoute(
options.copilotFailureDiagnosis.capability,
),
]),
]; ];
const routes = createClusterControlRouteRegistry(routeDefinitions); const routes = createClusterControlRouteRegistry(routeDefinitions);
const expectedRouteCount = const expectedRouteCount =
@@ -269,7 +287,8 @@ export function createProductionClusterControlApplicationStack(
(options.promptExecution === undefined ? 0 : 1) + (options.promptExecution === undefined ? 0 : 1) +
(options.promptExecutionInspection === undefined ? 0 : 1) + (options.promptExecutionInspection === undefined ? 0 : 1) +
(options.promptOutputRead === undefined ? 0 : 1) + (options.promptOutputRead === undefined ? 0 : 1) +
(options.promptExecutionOutputRead === undefined ? 0 : 1); (options.promptExecutionOutputRead === undefined ? 0 : 1) +
(options.copilotFailureDiagnosis === undefined ? 0 : 1);
if (routes.size !== expectedRouteCount) { if (routes.size !== expectedRouteCount) {
throw new Error('Production cluster-control route allowlist is incomplete'); throw new Error('Production cluster-control route allowlist is incomplete');
} }
@@ -359,6 +378,7 @@ export function startProductionClusterControlApplication(
promptExecutionInspection, promptExecutionInspection,
promptOutputRead, promptOutputRead,
promptExecutionOutputRead, promptExecutionOutputRead,
copilotFailureDiagnosis,
...applicationOptions ...applicationOptions
} = options; } = options;
const database = createClusterControlDatabaseBinding(config); const database = createClusterControlDatabaseBinding(config);
@@ -402,6 +422,9 @@ export function startProductionClusterControlApplication(
...(promptExecutionOutputRead === undefined ...(promptExecutionOutputRead === undefined
? {} ? {}
: { promptExecutionOutputRead }), : { promptExecutionOutputRead }),
...(copilotFailureDiagnosis === undefined
? {}
: { copilotFailureDiagnosis }),
...(workerIngress === undefined ...(workerIngress === undefined
? {} ? {}
: { : {
@@ -0,0 +1,297 @@
// Cluster Copilot owns one bounded, Policy-fenced diagnosis admission route.
import type { SecurityPrincipal } from '@qinglong/runtime-core/security';
import type { ClusterControlAdmissionResponse } from '../../transport/httpSurface';
import type {
ClusterControlAuthorizedOperationRequest,
ClusterControlRouteDefinition,
ClusterControlRouteParameters,
} from '../../transport/routeRegistry';
export const CLUSTER_COPILOT_FAILURE_DIAGNOSIS_REQUEST_SCHEMA =
'qinglong/cluster-copilot-failure-diagnosis-request@v1' as const;
export const CLUSTER_COPILOT_FAILURE_DIAGNOSIS_RESPONSE_SCHEMA =
'qinglong/cluster-copilot-failure-diagnosis-response@v1' as const;
export const CLUSTER_CONTROL_COPILOT_FAILURE_DIAGNOSIS_ROUTE = Object.freeze({
method: 'POST' as const,
path: '/api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses',
operationId: 'copilot.failure_diagnosis.execute',
permission: 'model.invoke',
projectParameter: 'projectId',
});
export interface ClusterCopilotFailureDiagnosisCommand {
readonly requestId: string;
readonly traceId: string;
readonly projectId: string;
readonly sourceRunId: string;
readonly principal: Readonly<SecurityPrincipal>;
}
interface ClusterCopilotFailureDiagnosisOutputReference {
readonly artifactId: string;
readonly artifactDigest: string;
}
export interface ClusterCopilotFailureDiagnosisCapability {
execute(command: Readonly<ClusterCopilotFailureDiagnosisCommand>): Promise<
Readonly<{
readonly admissionStatus: 'created' | 'existing';
readonly admission: Readonly<{
readonly requestId: string;
readonly runId: string;
readonly sourceRunId: string;
}>;
readonly tool: Readonly<{ readonly outcome: string }> | null;
readonly model: Readonly<{
readonly outcome: string;
readonly output: Readonly<ClusterCopilotFailureDiagnosisOutputReference> | null;
}> | null;
readonly terminalization: Readonly<{
readonly stage: string;
readonly reason: string;
readonly outcome: string;
}> | null;
readonly terminalizationRequired: boolean;
}>
>;
}
class InvalidClusterCopilotFailureDiagnosisRequestError extends TypeError {}
const ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const RUN_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$/;
const DIGEST_PATTERN = /^[0-9a-f]{64}$/;
const OUTCOMES = new Set(['succeeded', 'failed', 'timed_out', 'cancelled']);
const TERMINAL_STAGES = new Set(['tool', 'log', 'deadline', 'cancellation']);
const TERMINAL_REASONS = new Set([
'tool_failed',
'tool_timed_out',
'log_not_found',
'log_pending',
'log_missing',
'log_retired',
'tool_budget_exhausted',
'deadline_exceeded',
'cancellation_requested',
]);
function invalid(): never {
throw new InvalidClusterCopilotFailureDiagnosisRequestError();
}
function response(
statusCode: number,
body: Readonly<Record<string, unknown>>,
): ClusterControlAdmissionResponse {
return Object.freeze({ statusCode, body: Object.freeze(body) });
}
function parseBody(value: unknown): Readonly<{ traceId: string }> {
if (
!value ||
typeof value !== 'object' ||
Array.isArray(value) ||
Object.getPrototypeOf(value) !== Object.prototype ||
Object.keys(value).sort().join('\0') !== ['schema', 'traceId'].join('\0')
) {
return invalid();
}
const body = value as Record<string, unknown>;
if (
body.schema !== CLUSTER_COPILOT_FAILURE_DIAGNOSIS_REQUEST_SCHEMA ||
typeof body.traceId !== 'string' ||
!ID_PATTERN.test(body.traceId)
) {
return invalid();
}
return Object.freeze({ traceId: body.traceId });
}
function errorCode(error: unknown): string | null {
if (
!error ||
typeof error !== 'object' ||
!('code' in error) ||
typeof error.code !== 'string'
) {
return null;
}
return error.code;
}
function executionError(error: unknown): ClusterControlAdmissionResponse {
const code = errorCode(error);
if (
code === 'COPILOT_FAILURE_DIAGNOSIS_APPLICATION_CONFLICT' ||
code === 'COPILOT_FAILURE_DIAGNOSIS_ADMISSION_CONFLICT' ||
code === 'COPILOT_FAILURE_DIAGNOSIS_ADMISSION_NOT_ALLOWED' ||
code === 'COPILOT_FAILURE_DIAGNOSIS_TOOL_EXECUTION_CONFLICT' ||
code === 'COPILOT_FAILURE_DIAGNOSIS_MODEL_EXECUTION_CONFLICT' ||
code === 'COPILOT_FAILURE_DIAGNOSIS_FINALIZATION_CONFLICT' ||
code === 'COPILOT_FAILURE_DIAGNOSIS_MODEL_EXECUTION_IN_PROGRESS' ||
code === 'COPILOT_FAILURE_DIAGNOSIS_MODEL_RESOLUTION_REQUIRED' ||
code === 'COPILOT_FAILURE_DIAGNOSIS_PRE_MODEL_TERMINALIZATION_CONFLICT'
) {
return response(409, { code: 'copilot_failure_diagnosis_conflict' });
}
if (
code === 'TRUSTED_TOOL_EXECUTION_POLICY_DENIED' ||
code === 'TRUSTED_TOOL_EXECUTION_APPROVAL_REQUIRED'
) {
return response(403, { code: 'copilot_failure_diagnosis_forbidden' });
}
if (
code === 'COPILOT_FAILURE_DIAGNOSIS_APPLICATION_BUSY' ||
code === 'MODEL_GATEWAY_BUSY'
) {
return response(429, {
code: 'copilot_failure_diagnosis_capacity_exceeded',
});
}
if (
code === 'MODEL_POLICY_DENIED' ||
code === 'MODEL_BUDGET_EXCEEDED' ||
code === 'COPILOT_MODEL_EGRESS_DENIED' ||
code === 'COPILOT_FAILURE_DIAGNOSIS_BUDGET_EXCEEDED'
) {
return response(422, { code: 'copilot_failure_diagnosis_policy_rejected' });
}
if (
code === 'MODEL_INVOCATION_DEADLINE_EXCEEDED' ||
code === 'COPILOT_FAILURE_DIAGNOSIS_TOOL_EXECUTION_DEADLINE_EXCEEDED'
) {
return response(504, {
code: 'copilot_failure_diagnosis_deadline_exceeded',
});
}
if (code === 'MODEL_INVOCATION_ABORTED') {
return response(408, { code: 'copilot_failure_diagnosis_aborted' });
}
if (
code === 'COPILOT_FAILURE_DIAGNOSIS_APPLICATION_INVALID' ||
code === 'COPILOT_FAILURE_DIAGNOSIS_EXECUTION_PLAN_INVALID'
) {
return response(400, { code: 'invalid_copilot_failure_diagnosis_request' });
}
return response(503, { code: 'copilot_failure_diagnosis_unavailable' });
}
function projectResult(
result: Awaited<
ReturnType<ClusterCopilotFailureDiagnosisCapability['execute']>
>,
requestId: string,
sourceRunId: string,
): ClusterControlAdmissionResponse {
const model = result?.model;
const terminalization = result?.terminalization;
const outcome = model?.outcome ?? terminalization?.outcome;
if (
!result ||
(result.admissionStatus !== 'created' &&
result.admissionStatus !== 'existing') ||
!result.admission ||
result.admission.requestId !== requestId ||
result.admission.sourceRunId !== sourceRunId ||
!RUN_ID_PATTERN.test(result.admission.runId) ||
result.terminalizationRequired !== false ||
(model === null) === (terminalization === null) ||
typeof outcome !== 'string' ||
!OUTCOMES.has(outcome)
) {
return response(503, { code: 'copilot_failure_diagnosis_unavailable' });
}
let stage: string;
let reason: string | null;
let outputArtifact: Readonly<ClusterCopilotFailureDiagnosisOutputReference> | null;
if (model) {
if (
!OUTCOMES.has(model.outcome) ||
(model.outcome === 'succeeded') !== (model.output !== null) ||
(model.output !== null &&
(!ID_PATTERN.test(model.output.artifactId) ||
!DIGEST_PATTERN.test(model.output.artifactDigest)))
) {
return response(503, { code: 'copilot_failure_diagnosis_unavailable' });
}
stage = 'model';
reason = null;
outputArtifact = model.output;
} else {
if (
!terminalization ||
!TERMINAL_STAGES.has(terminalization.stage) ||
!TERMINAL_REASONS.has(terminalization.reason)
) {
return response(503, { code: 'copilot_failure_diagnosis_unavailable' });
}
stage = terminalization.stage;
reason = terminalization.reason;
outputArtifact = null;
}
return response(result.admissionStatus === 'created' ? 201 : 200, {
schema: CLUSTER_COPILOT_FAILURE_DIAGNOSIS_RESPONSE_SCHEMA,
requestId,
status: result.admissionStatus,
replayed: result.admissionStatus === 'existing',
sourceRunId,
diagnosisRunId: result.admission.runId,
outcome,
stage,
reason,
outputArtifact:
outputArtifact === null
? null
: Object.freeze({
artifactId: outputArtifact.artifactId,
artifactDigest: outputArtifact.artifactDigest,
}),
});
}
export function createClusterControlCopilotFailureDiagnosisRoute(
capability: ClusterCopilotFailureDiagnosisCapability,
): Readonly<ClusterControlRouteDefinition> {
if (!capability || typeof capability.execute !== 'function') {
throw new TypeError(
'Cluster-control Copilot failure diagnosis capability is invalid',
);
}
return Object.freeze({
...CLUSTER_CONTROL_COPILOT_FAILURE_DIAGNOSIS_ROUTE,
async handle(
authorized: ClusterControlAuthorizedOperationRequest,
routeParameters: ClusterControlRouteParameters,
) {
let body: Readonly<{ traceId: string }>;
try {
body = parseBody(authorized.request.body);
} catch {
return response(400, {
code: 'invalid_copilot_failure_diagnosis_request',
});
}
const projectId = authorized.projectId;
const sourceRunId = routeParameters.runId;
if (
projectId === null ||
typeof sourceRunId !== 'string' ||
!RUN_ID_PATTERN.test(sourceRunId)
) {
return response(503, { code: 'copilot_failure_diagnosis_unavailable' });
}
try {
const result = await capability.execute({
requestId: authorized.request.requestId,
traceId: body.traceId,
projectId,
sourceRunId,
principal: authorized.principal,
});
return projectResult(result, authorized.request.requestId, sourceRunId);
} catch (error) {
return executionError(error);
}
},
});
}
@@ -117,7 +117,7 @@ async function projectedFile(root, name, bytes) {
await chmod(join(root, name), 0o440); await chmod(join(root, name), 0o440);
} }
test('Copilot composition is explicit, shares the Prompt gateway and exposes no route', async () => { test('Copilot composition is explicit, shares the Prompt gateway and injects one route capability', async () => {
const secretRoot = await mkdtemp(join(tmpdir(), 'ql3-cluster-ai-secret-')); const secretRoot = await mkdtemp(join(tmpdir(), 'ql3-cluster-ai-secret-'));
const configRoot = await mkdtemp(join(tmpdir(), 'ql3-copilot-config-')); const configRoot = await mkdtemp(join(tmpdir(), 'ql3-copilot-config-'));
const invocationRoot = await mkdtemp(join(tmpdir(), 'ql3-copilot-invocation-')); const invocationRoot = await mkdtemp(join(tmpdir(), 'ql3-copilot-invocation-'));
@@ -229,7 +229,7 @@ test('Copilot composition is explicit, shares the Prompt gateway and exposes no
assert.equal(created.gateway, gateway); assert.equal(created.gateway, gateway);
assert.equal(created.successfulCompletion, registeredSink); assert.equal(created.successfulCompletion, registeredSink);
assert.equal(created.artifactStore, artifactStore); assert.equal(created.artifactStore, artifactStore);
assert.equal('copilot' in controlOptions, false); assert.equal(controlOptions.copilotFailureDiagnosis.capability, copilot);
assert.equal(await application.stop(), 'stopped'); assert.equal(await application.stop(), 'stopped');
} finally { } finally {
config.fill(0); invocation.fill(0); result.fill(0); output.fill(0); config.fill(0); invocation.fill(0); result.fill(0); output.fill(0);
@@ -0,0 +1,269 @@
const assert = require('node:assert/strict');
const { test } = require('node:test');
const {
CLUSTER_COPILOT_FAILURE_DIAGNOSIS_REQUEST_SCHEMA,
CLUSTER_COPILOT_FAILURE_DIAGNOSIS_RESPONSE_SCHEMA,
CLUSTER_CONTROL_COPILOT_FAILURE_DIAGNOSIS_ROUTE,
createClusterControlCopilotFailureDiagnosisRoute,
} = require('@qinglong/cluster-control/copilot-routes');
function authorized(body, overrides = {}) {
return {
request: {
requestId: 'diagnosis-request-1',
method: 'POST',
path: '/api/v3/projects/project-1/runs/source-run-1/copilot/failure-diagnoses',
query: {},
headers: {},
signal: new AbortController().signal,
body,
},
principal: {
subject: { type: 'api_app', id: 'app-1' },
authenticationId: 'credential-1',
authenticatedAtMs: 1,
expiresAtMs: 10_000,
assurance: 'service',
},
operationId: 'copilot.failure_diagnosis.execute',
permission: 'model.invoke',
projectId: 'project-1',
policyFence: { projectVersion: 3, bindingVersion: 7 },
...overrides,
};
}
function body(overrides = {}) {
return {
schema: CLUSTER_COPILOT_FAILURE_DIAGNOSIS_REQUEST_SCHEMA,
traceId: 'trace-1',
...overrides,
};
}
function succeeded(admissionStatus = 'created') {
return {
admissionStatus,
admission: {
requestId: 'diagnosis-request-1',
runId: 'diagnosis-run-1',
sourceRunId: 'source-run-1',
},
tool: { outcome: 'succeeded', output: { private: 'must not cross' } },
model: {
outcome: 'succeeded',
output: {
artifactId: 'cdo:artifact-1',
artifactDigest: 'a'.repeat(64),
provider: 'private-provider',
},
plaintext: 'private diagnosis',
},
terminalization: null,
terminalizationRequired: false,
};
}
test('defines one exact model.invoke route and binds HTTP request identity', async () => {
let command;
const route = createClusterControlCopilotFailureDiagnosisRoute({
async execute(value) {
command = value;
return succeeded();
},
});
const request = authorized(body());
const result = await route.handle(request, {
projectId: 'project-1',
runId: 'source-run-1',
});
assert.deepEqual(CLUSTER_CONTROL_COPILOT_FAILURE_DIAGNOSIS_ROUTE, {
method: 'POST',
path: '/api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses',
operationId: 'copilot.failure_diagnosis.execute',
permission: 'model.invoke',
projectParameter: 'projectId',
});
assert.deepEqual(command, {
requestId: 'diagnosis-request-1',
traceId: 'trace-1',
projectId: 'project-1',
sourceRunId: 'source-run-1',
principal: request.principal,
});
assert.equal('policyFence' in command, false);
assert.equal('model' in command, false);
assert.equal('attemptId' in command, false);
assert.equal(result.statusCode, 201);
assert.deepEqual(result.body, {
schema: CLUSTER_COPILOT_FAILURE_DIAGNOSIS_RESPONSE_SCHEMA,
requestId: 'diagnosis-request-1',
status: 'created',
replayed: false,
sourceRunId: 'source-run-1',
diagnosisRunId: 'diagnosis-run-1',
outcome: 'succeeded',
stage: 'model',
reason: null,
outputArtifact: {
artifactId: 'cdo:artifact-1',
artifactDigest: 'a'.repeat(64),
},
});
assert.equal(JSON.stringify(result).includes('private'), false);
});
test('returns a content-free existing receipt for exact replay', async () => {
const route = createClusterControlCopilotFailureDiagnosisRoute({
async execute() {
return succeeded('existing');
},
});
const result = await route.handle(authorized(body()), {
projectId: 'project-1',
runId: 'source-run-1',
});
assert.equal(result.statusCode, 200);
assert.equal(result.body.status, 'existing');
assert.equal(result.body.replayed, true);
});
test('projects pre-Model terminalization without Tool or log content', async () => {
const route = createClusterControlCopilotFailureDiagnosisRoute({
async execute() {
return {
admissionStatus: 'created',
admission: {
requestId: 'diagnosis-request-1',
runId: 'diagnosis-run-1',
sourceRunId: 'source-run-1',
},
tool: { outcome: 'failed', privateLog: 'must not cross' },
model: null,
terminalization: {
stage: 'log',
reason: 'log_retired',
outcome: 'failed',
privateEvidence: 'must not cross',
},
terminalizationRequired: false,
};
},
});
const result = await route.handle(authorized(body()), {
projectId: 'project-1',
runId: 'source-run-1',
});
assert.equal(result.statusCode, 201);
assert.equal(result.body.stage, 'log');
assert.equal(result.body.reason, 'log_retired');
assert.equal(result.body.outcome, 'failed');
assert.equal(result.body.outputArtifact, null);
assert.equal(JSON.stringify(result).includes('private'), false);
});
test('rejects non-exact bodies before invoking the capability', async () => {
let calls = 0;
const route = createClusterControlCopilotFailureDiagnosisRoute({
async execute() {
calls += 1;
return succeeded();
},
});
for (const invalid of [
null,
{},
body({ requestId: 'body-request-must-not-exist' }),
body({ provider: 'caller-selected' }),
body({ traceId: '' }),
Object.assign(Object.create(null), body()),
]) {
const result = await route.handle(authorized(invalid), {
projectId: 'project-1',
runId: 'source-run-1',
});
assert.equal(result.statusCode, 400);
assert.deepEqual(result.body, {
code: 'invalid_copilot_failure_diagnosis_request',
});
}
assert.equal(calls, 0);
});
test('fails closed on capability responses that do not bind the durable identity', async () => {
for (const mutation of [
(value) => ({
...value,
admission: { ...value.admission, requestId: 'other' },
}),
(value) => ({
...value,
admission: { ...value.admission, sourceRunId: 'other' },
}),
(value) => ({ ...value, terminalizationRequired: true }),
(value) => ({ ...value, model: null }),
(value) => ({ ...value, model: { ...value.model, output: null } }),
]) {
const route = createClusterControlCopilotFailureDiagnosisRoute({
async execute() {
return mutation(succeeded());
},
});
const result = await route.handle(authorized(body()), {
projectId: 'project-1',
runId: 'source-run-1',
});
assert.deepEqual(result, {
statusCode: 503,
body: { code: 'copilot_failure_diagnosis_unavailable' },
});
}
});
test('maps internal failures to stable low-sensitive transport codes', async () => {
for (const [internal, statusCode, external] of [
[
'COPILOT_FAILURE_DIAGNOSIS_APPLICATION_CONFLICT',
409,
'copilot_failure_diagnosis_conflict',
],
[
'TRUSTED_TOOL_EXECUTION_POLICY_DENIED',
403,
'copilot_failure_diagnosis_forbidden',
],
[
'COPILOT_FAILURE_DIAGNOSIS_APPLICATION_BUSY',
429,
'copilot_failure_diagnosis_capacity_exceeded',
],
[
'COPILOT_MODEL_EGRESS_DENIED',
422,
'copilot_failure_diagnosis_policy_rejected',
],
[
'MODEL_INVOCATION_DEADLINE_EXCEEDED',
504,
'copilot_failure_diagnosis_deadline_exceeded',
],
['MODEL_INVOCATION_ABORTED', 408, 'copilot_failure_diagnosis_aborted'],
['PRIVATE_STORAGE_FAILURE', 503, 'copilot_failure_diagnosis_unavailable'],
]) {
const route = createClusterControlCopilotFailureDiagnosisRoute({
async execute() {
throw Object.assign(new Error('private internal detail'), {
code: internal,
});
},
});
const result = await route.handle(authorized(body()), {
projectId: 'project-1',
runId: 'source-run-1',
});
assert.equal(result.statusCode, statusCode, internal);
assert.deepEqual(result.body, { code: external });
assert.equal(JSON.stringify(result).includes('private'), false);
}
});
@@ -729,6 +729,7 @@ test('optionally exposes Prompt execution behind shared admission and policy', a
'prompt.execution.read', 'prompt.execution.read',
'prompt.execution.output.read', 'prompt.execution.output.read',
'prompt.output.read', 'prompt.output.read',
'copilot.failure_diagnosis.execute',
]); ]);
const response = await invoke( const response = await invoke(
stack, stack,
@@ -756,6 +757,134 @@ test('optionally exposes Prompt execution behind shared admission and policy', a
assert.equal(events.includes('audit:prompt.execute:allowed'), true); assert.equal(events.includes('audit:prompt.execute:allowed'), true);
}); });
test('optionally exposes Copilot diagnosis behind shared authentication, Policy and audit', async () => {
const { events, input } = fixture();
let command;
const capability = {
async execute(value) {
command = value;
events.push(`diagnose:${value.sourceRunId}`);
return {
admissionStatus: 'created',
admission: {
requestId: value.requestId,
runId: 'diagnosis-run-1',
sourceRunId: value.sourceRunId,
},
tool: { outcome: 'succeeded' },
model: {
outcome: 'succeeded',
output: {
artifactId: 'cdo:artifact-1',
artifactDigest: 'a'.repeat(64),
},
},
terminalization: null,
terminalizationRequired: false,
};
},
};
const stack = createProductionClusterControlApplicationStack(input, {
copilotFailureDiagnosis: { capability },
});
const result = await invoke(
stack,
metadata(
'/api/v3/projects/project-1/runs/run-1/copilot/failure-diagnoses',
'POST',
{
schema: 'qinglong/cluster-copilot-failure-diagnosis-request@v1',
traceId: 'trace-production-1',
},
),
);
assert.equal(result.statusCode, 201);
assert.equal(command.requestId, 'request-production-1');
assert.equal(command.projectId, 'project-1');
assert.equal(command.sourceRunId, 'run-1');
assert.equal(command.principal.subject.id, 'app-production');
assert.equal('policyFence' in command, false);
assert.deepEqual(events.slice(-4), [
'authenticate',
'authorize',
'audit:copilot.failure_diagnosis.execute:allowed',
'diagnose:run-1',
]);
});
test('keeps the Copilot route absent by default and never invokes it after Policy denial', async () => {
const defaultFixture = fixture();
const defaultStack = createProductionClusterControlApplicationStack(
defaultFixture.input,
);
const request = metadata(
'/api/v3/projects/project-1/runs/run-1/copilot/failure-diagnoses',
'POST',
{
schema: 'qinglong/cluster-copilot-failure-diagnosis-request@v1',
traceId: 'trace-production-1',
},
);
await assert.rejects(
defaultStack.admission.prepare(request),
(error) => error?.statusCode === 404 && error?.code === 'route_not_found',
);
let calls = 0;
const deniedFixture = fixture({
policies: {
async resolve() {
deniedFixture.events.push('authorize');
return {
project: {
id: 'project-1',
name: 'Denied Project',
slug: 'denied-project',
status: 'active',
version: 3,
createdAtMs: 1,
updatedAtMs: 2,
},
binding: {
projectId: 'project-1',
subject: { type: 'api_app', id: 'app-production' },
state: 'active',
role: 'viewer',
version: 7,
mutationId: 'binding-denied-1',
changedBy: { type: 'system', id: 'bootstrap' },
createdAtMs: 1,
},
};
},
},
});
const deniedStack = createProductionClusterControlApplicationStack(
deniedFixture.input,
{
copilotFailureDiagnosis: {
capability: {
async execute() {
calls += 1;
},
},
},
},
);
await assert.rejects(
deniedStack.admission.prepare(request),
(error) => error?.statusCode === 403 && error?.code === 'forbidden',
);
assert.equal(calls, 0);
assert.equal(
deniedFixture.events.includes(
'audit:copilot.failure_diagnosis.execute:denied',
),
true,
);
});
test('optionally exposes the redacted Prompt catalog behind shared admission and policy', async () => { test('optionally exposes the redacted Prompt catalog behind shared admission and policy', async () => {
const { events, input } = fixture(); const { events, input } = fixture();
const stack = createProductionClusterControlApplicationStack(input, { const stack = createProductionClusterControlApplicationStack(input, {
+2 -2
View File
@@ -385,10 +385,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
rootSourceFileRoles: clusterControl.rootSourceFileRoles, rootSourceFileRoles: clusterControl.rootSourceFileRoles,
}, },
{ {
sourceFiles: 59, sourceFiles: 60,
rootSourceFiles: 2, rootSourceFiles: 2,
rootSourceLines: 195, rootSourceLines: 195,
nestedSourceFiles: 57, nestedSourceFiles: 58,
rootSourceFileRoles: { rootSourceFileRoles: {
'aiCli.ts': 'binary_entry', 'aiCli.ts': 'binary_entry',
'cli.ts': 'binary_entry', 'cli.ts': 'binary_entry',