feat(ql3): add fenced copilot diagnosis cancellation

This commit is contained in:
whyour
2026-08-16 01:34:55 +08:00
parent f77b19e615
commit e5a3d83a8d
17 changed files with 1685 additions and 8 deletions
+1
View File
@@ -11,6 +11,7 @@
最新增量证据(2026-08-16):
- D-323/ADR-0415(已接受):Cluster Copilot failure diagnosis 已补齐受围栏的 request-keyed 取消 mutation。`POST /api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses/{requestId}/cancellation` 复用 exact `qinglong/run-cancellation@v1` body、`run.stop` 当前 Policy、durable audit 与通用 PostgreSQL Run cancellation transaction;调用方只能提供 `mutationId`diagnosis Run、Event、reason、Provider 与终态均由服务端从 admission plan/receipt 和 durable authority 派生。pre-Model 取消立即与 Model start 通过 Run/Step/version fence 竞争并原子 terminalizeModel start 获胜时只返回 `model_in_flight` durable intent,不伪造 Provider abort、终态、usage 或 cost,真实 completion/finalization 仍可成为最终 winner。相同或不同 mutation 都不能追加第二个 intent/Event 或覆盖首次 reason/time。能力只注入默认关闭的 Cluster AI profile,复用既有 AI Pool、repository、Policy pipeline 与进程,不新增 package、schema、Pool、连接、timer、watcher、queue、cache、端口、Pod、Service 或 Kubernetes 权限,Edge/Standalone 与普通 Cluster Control 均无该 route。目录审计曾拒绝 `failure-diagnosis` 达到 12 个直属源码文件;最终删除仅做 re-export 的单文件 façade,让公开 package subpath 直接指向 `cancellation/service`,而不是再拆 package 或放宽阈值。workspace 保持 18 package`singleSourcePackages=[]``shallowSourcePackages=[]`AI 194 个源码中 193 个、Cluster Control 64 个源码中 62 个位于嵌套职责目录。AI 254 pass/3 条件 skip、Cluster Control 261 pass/2 条件 skip、18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail;四项架构审计和 14 档 Local artifact 全部通过。默认 Edge/Standalone 为 2,589,890/2,589,968 bytesEdge/Standalone AI application 为 4,493,043/4,493,175 bytes,证明 Cluster-only mutation 未进入低配闭包。PostgreSQL 18.6 arm64 physical HA 142/142、timeline `1→2`,覆盖 intent/Event/terminal receipt 同步复制、promotion 后 exact replay 与 in-flight 不伪造终态;报告 SHA-256 为 `5dbcffb74a3181aabee66a8f68ecfa7a65e0491a6f2ba24e2bc903c83da9d766`,离线审计零 finding。下一独立 Gate 可选择 CLI/UI/MCP 客户端或带 Provider abort acknowledgement 的取消语义,不得在没有 durable acknowledgement 时宣称外部调用已停止。
- D-322/ADR-0414(已接受):Cluster Copilot failure diagnosis 已补齐 request-keyed 产品读模型。`GET /api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses/{requestId}` 只需 `run.read`,投影 running/terminal、取消/失败 stage/reason、authoritative admission/finalization time 与 durable usage/settled cost;同路径 `/output` 独立要求 `artifact.read`,只在 admission plan/receipt、Project/source Run、finalization、invocation 和 encrypted Artifact 全部 exact binding 且 current Project Policy 再授权后解析 historical key。调用者不能提交 Artifact、diagnosis Run、Model invocation、provider/model、价格、usage、outcome 或 key identitydeny/approval/absent/cross-target 统一 404,存储/Policy/key/decrypt/cipher 冲突统一 503。写 `capability``readCapability` 接口隔离,production 复用既有 AI PostgreSQL Pool、repository、Project Policy 与 projected output keyring,不新增 package、进程、端口、Pool、timer、watcher、queue、cache、Pod、Service 或 Kubernetes 权限;普通 Cluster Control、Edge/Standalone 均无这些 route。workspace 保持 18 package、`singleSourcePackages=[]``shallowSourcePackages=[]`AI 193 个源码中 192 个、Cluster Control 62 个源码中 60 个位于嵌套领域目录。AI 249 pass/3 条件 skip、Cluster Control 256 pass/2 条件 skip、18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail,四项架构审计和 14 档 Local artifact 全部通过;默认 Edge/Standalone 仍为 2,589,890/2,589,968 bytesEdge/Standalone AI 为 3,069,143/3,069,233 bytes。PostgreSQL 18.6 arm64 physical HA 139/139、timeline `1→2`,成功密文输出在 standby 可读,提升后 request-keyed exact replay 且 provider 调用为零;报告 SHA-256 为 `22decb54cfb8735bf787fe0665c877c201fc7b44d3c3de16fdbfdab31b7ac2cd`,离线审计零 finding。下一独立 Gate 可评审取消 mutation、CLI/UI/MCP 客户端或真实 Provider,不得把它们混入读边界。
- D-321/ADR-0413(已接受):Cluster Copilot failure diagnosis 已开放唯一、默认关闭的产品写入口 `POST /api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses`。该 route 只在显式 AI 进程且 `QL3_CLUSTER_AI_COPILOT_ENABLED=true` 时注入既有 Cluster Control route registry,完整复用同一认证器、`model.invoke` Project Policy、fail-closed 同步安全审计、HTTP/TLS body/response/concurrency 上限和生命周期;内部只读 Tool 仍独立复验其 exact `tool.call:*` Policy。耐久幂等身份直接绑定 `x-request-id`body 只允许 schema 与 `traceId`Project/source Run/principal 来自 canonical path 和认证,Attempt、日志范围、Tool、provider/model、预算、deadline、Policy fence、reason/outcome 均由服务端从数据库与只读配置派生。响应只投影 created/existing、source/diagnosis Run、终态 stage/reason/outcome 和加密输出 Artifact id/digest,不返回日志、Tool/Model plaintext、prompt、模型信息、密钥或内部异常。普通 Cluster Control、Edge、Standalone 和未启用 Copilot 的 AI 进程没有 route,也没有新增 package、进程、监听器、Pool/连接、timer、watcher、队列、cache、Pod、Service 或 Kubernetes 权限;workspace 仍为 18 package、无单文件/浅平 package,新增源码位于既有嵌套 Copilot 领域目录。Cluster Control 250 pass/2 条件 skip18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail,四项架构审计和 14 档 Local artifact 全部通过;默认 Edge/Standalone 保持 2,589,890/2,589,968 bytes。PostgreSQL 18.6 arm64 HA 137/137、timeline `1→2`,报告 SHA-256 为 `0a12b5c1102555823d43b5a93dd7868b98b194b491840a5242bab6fa2da26123`,离线审计零 finding。本 Gate 不改 migration/schema/role/SQL/HA 拓扑。下一 Gate 优先补同一 Policy 下的加密诊断输出读取与费用/取消可观测性;CLI/UI/MCP 只能复用该 API,不得直连 application capability。
- D-320/ADR-0412(已接受):Cluster Copilot failure diagnosis 已补齐产品入口前的非成功收敛边界。新增的 pre-Model terminalizer 只从 durable Tool failure、受审日志 projection 或数据库观察到的 deadline/cancellation 派生封闭 reason,在 Model start 不存在时以一个 SERIALIZABLE 事务原子提交 StepRun mutations、RunEvents、父 Run 终态和 `pg-9021` append-only content-free receiptTool `failed|timed_out`、日志 `not_found|pending|missing|retired`、五秒 Tool budget 不足、deadline 和 cancellation 均可 exact replay。Model 已开始后的 `outcome_unknown` 继续要求强 User 显式 `fail|cancel|retry` resolutionCopilot finalizer 现在对 `fail|cancel` 精确校验 resolution mutation 的 resolved Step digest,既不伪造失败也不自动重试 Provider。实现复用既有 AI package、Cluster AI 进程、Pool 与 ledger,不增加 package、进程、连接、timer、watcher、队列、cache 或产品 routeworkspace 仍为 18 package、无单文件/浅平 packageAI 192 个源码中 191 个位于嵌套领域目录。AI 完整测试 244 pass/3 条件 skip18-package clean build/test 与 backend 1,207 pass/2 条件 skip/0 fail,四项架构审计和 14 档 Local artifact 全部通过。PostgreSQL 18.6 arm64 HA 137/137、timeline `1→2`,覆盖日志不可用 pre-Model terminalization、unknown completion→人工 fail resolution→Run finalization 及晋升后零外部副作用 exact replay;报告 SHA-256 为 `6eaeb20615a62d153c5a69687344f41f31351c6ecf111cfb9cbafad115538c83`,离线审计零 finding。下一 Gate 才增加认证、Policy、audit 与 source fence 保护的 Cluster API,并由 CLI/UI/MCP 复用,仍不得建立旁路执行器。
@@ -0,0 +1,39 @@
# ADR-0415:受围栏的 Copilot 故障诊断取消入口
- 状态:Accepted
- 日期:2026-08-16
- 关联 RFCQL-RFC-0001 D-323、Phase 2
## 背景
D-321 已提供受认证的故障诊断执行入口,D-322 已提供 request-keyed 状态与密文输出读取,但用户还不能按原始诊断 `requestId` 停止不再需要的诊断。直接要求调用者取消内部 diagnosis Run 会泄露实现 identity;直接复用普通 Run cancellation route 又无法验证 source Run、原始 request 与 diagnosis Run 的三方绑定。
现有 durable authority 已经足够:admission plan/receipt 唯一绑定 Project、source Run、request 与 diagnosis Run;通用 Run cancellation transaction 会重验当前 Project/RoleBinding Policy fence,并以数据库时间写唯一 cancellation intent/Eventpre-Model terminalizer 能在 Model start 不存在时原子终结 Tool/Model Step、父 Run 与 receipt。Model start 之后目前没有 Provider abort acknowledgement,因而不能把“已写停止意图”伪装成“Provider 已停止”。
## 决策
1. Cluster AI profile 新增:
`POST /api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses/{requestId}/cancellation`,固定 operation `copilot.failure_diagnosis.cancel`、permission `run.stop`,不接受 query。
2. 请求复用 profile-neutral exact body `qinglong/run-cancellation@v1`,仅含 `schema` 与调用方生成的 `mutationId`。调用方不能提交 diagnosis Run、Event ID、取消原因、Model invocation、Provider 或终态;Event ID 由服务端生成。
3. admission plan 与 receipt 必须同时存在,并 exact 绑定 path 中的 Project、source Run 和 request。服务端只使用 plan 中的 diagnosis Run ID 调用既有 PostgreSQL Run cancellation repository;不存在、跨 Project、跨 source Run 或漂移统一返回 404。
4. route admission 负责 Bearer authentication、`run.stop` 当前 Policy 判定与 durable auditRun cancellation transaction 在写 intent/Event 的同一 SERIALIZABLE 事务中再次锁定并验证当前 Project/RoleBinding fence。撤权竞态返回 409,存储或 durable evidence 冲突返回 503。
5. Model start 尚不存在时,取消 intent 提交后立即调用既有 pre-Model terminalizer。terminalizer 与 Model start 通过 PostgreSQL Run/Step/version fence 竞争:terminalizer 获胜则 Run/未完成 Step 原子进入 `cancelled`Model start 获胜则进入 `model_in_flight`
6. Model 已开始时只承诺 durable cancellation intent,不伪造 Provider abort、usage、cost 或终态。真实 Provider completion 可以成为最终结果;`outcome_unknown` 仍必须走既有强 User resolution。后续若增加 Provider abort,必须另立包含 abort acknowledgement、late completion 与计费语义的 ADR。
7. 响应 schema 为 `qinglong/cluster-copilot-failure-diagnosis-cancellation-response@v1`,投影 `accepted|already_requested|already_terminal``terminal|model_in_flight`、exact target、diagnosis Run、当前/终态 outcome 与 cancellation fact。首次 intent 为 202,其余幂等重放为 200。
8. 相同 mutation 的响应丢失重放不得追加 Event 或再次递增版本;不同 mutation 也不得覆盖已有 reason/time。取消与 Model start/finalization 的任一竞态只能有一个合法 durable winner。
9. 实现留在既有 `@qinglong/ai``@qinglong/cluster-control` 的嵌套 Copilot 领域目录,复用同一 AI PostgreSQL Pool、repositories、Policy pipeline 与进程;不新增 package、schema、Pool、连接、timer、watcher、队列、cache、端口、Pod、Service 或 Kubernetes 权限。普通 Cluster Control、Edge 与 Standalone 不注册该 route。
## 不选择
- **让客户端直接调用 diagnosis Run 的通用 cancellation route**:暴露内部 Run identity,且无法证明 source/request binding。
- **Model start 后直接写 cancelled**:没有 Provider abort acknowledgement,会伪造外部副作用与计费事实,并允许迟到 completion 覆盖假终态。
- **为 Copilot cancellation 新建 package、表或 supervisor**:现有 admission、Run intent、terminalization 与 Model recovery ledger 已覆盖所需 authority;新资源只会复制状态机并增加低配设备成本。
- **在 MCP/UI/CLI 中同时实现写入口**:客户端需要独立的强认证、确认交互与错误展示门,不能与服务端 authority 混成一个不可审计改动。
## 验收
1. 单元/集成测试覆盖 invalid body、权限、跨 Project/source/request、Policy 撤权、首次/相同 mutation/不同 mutation、终态重放、pre-Model 原子 terminalization、Model-start winner、finalization race 与故障映射。
2. PostgreSQL HA 覆盖 intent/Event/terminal receipt 复制、promotion 后 exact replay、pre-Model winner 和 in-flight 不伪造终态。
3. `@qinglong/ai``@qinglong/cluster-control`、18-package clean build/test、backend、四项架构审计、14 档 Local artifact 与离线 HA 报告审计全部通过后才允许 D-323 阶段提交。
最终验收(2026-08-16):AI 254 pass/3 条件 skipCluster Control 261 pass/2 条件 skip18-package clean build/test 退出 0backend 1,207 pass/2 条件 skip/0 failworkspace 保持 18 package`singleSourcePackages=[]``shallowSourcePackages=[]`。目录密度门拒绝新增的第 12 个 failure-diagnosis 直属文件后,删除了仅做 re-export 的单文件 façade,公开 subpath 直接指向内聚的 `cancellation/service`,未新增 package 或放宽目录阈值。四项架构审计、14 档 Edge/Standalone artifact 与 PostgreSQL HA 离线证据审计均通过;PostgreSQL 18.6 arm64 physical HA 为 142/142、timeline `1→2`,报告 SHA-256 为 `5dbcffb74a3181aabee66a8f68ecfa7a65e0491a6f2ba24e2bc903c83da9d766`
+3
View File
@@ -416,6 +416,9 @@
| [ADR-0410](./ADR-0410-cluster-copilot-failure-diagnosis-output-key-authority.md) | Cluster Copilot Failure Diagnosis Output Projected Key Authority | Accepted |
| [ADR-0411](./ADR-0411-default-off-cluster-copilot-composition.md) | 默认关闭的 Cluster Copilot Failure Diagnosis Composition | Accepted |
| [ADR-0412](./ADR-0412-durable-copilot-pre-model-terminalization-and-unknown-resolution.md) | 耐久 Copilot Pre-Model 终态化与 Model Unknown Resolution | Accepted |
| [ADR-0413](./ADR-0413-default-off-cluster-copilot-failure-diagnosis-api.md) | 默认关闭的 Cluster Copilot Failure Diagnosis API | Accepted |
| [ADR-0414](./ADR-0414-copilot-failure-diagnosis-read-model.md) | Copilot 故障诊断请求键读模型 | Accepted |
| [ADR-0415](./ADR-0415-fenced-copilot-failure-diagnosis-cancellation.md) | 受围栏的 Copilot 故障诊断取消入口 | Accepted |
## 规则
+5
View File
@@ -65,6 +65,11 @@
"require": "./dist/copilot/failure-diagnosis/read-model/service.js",
"default": "./dist/copilot/failure-diagnosis/read-model/service.js"
},
"./failure-diagnosis-cancellation": {
"types": "./dist/copilot/failure-diagnosis/cancellation/service.d.ts",
"require": "./dist/copilot/failure-diagnosis/cancellation/service.js",
"default": "./dist/copilot/failure-diagnosis/cancellation/service.js"
},
"./failure-diagnosis-pre-model-terminalization": {
"types": "./dist/copilot/failure-diagnosis/preModelTerminalization.d.ts",
"require": "./dist/copilot/failure-diagnosis/preModelTerminalization.js",
@@ -0,0 +1,415 @@
import { RUN_STATUSES, type RunStatus } from '@qinglong/runtime-core';
import type {
ClusterRunCancellationRepository,
ClusterRunCancellationResult,
} from '@qinglong/runtime-core/cluster-run-cancellation';
import type {
SecurityPolicyFence,
SecuritySubject,
} from '@qinglong/runtime-core/security';
import type {
CopilotFailureDiagnosisAdmissionRepository,
CopilotFailureDiagnosisAdmissionReceipt,
CopilotFailureDiagnosisExecutionPlan,
} from '../admission/contracts';
import { normalizeCopilotFailureDiagnosisAdmissionReceipt } from '../admission/durableEvidence';
import { normalizeCopilotFailureDiagnosisExecutionPlan } from '../admission/plan';
import {
terminalizeCopilotFailureDiagnosisBeforeModel,
type CopilotFailureDiagnosisPreModelTerminalizationDependencies,
} from '../terminalization/coordinator';
import {
CopilotFailureDiagnosisPreModelTerminalizationConflictError,
CopilotFailureDiagnosisPreModelTerminalizationNotReadyError,
CopilotFailureDiagnosisPreModelTerminalizationUnavailableError,
} from '../terminalization/contracts';
export const COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_RESULT_SCHEMA =
'qinglong/copilot-failure-diagnosis-cancellation-result@v1' as const;
export interface CopilotFailureDiagnosisCancellationCommand {
readonly projectId: string;
readonly sourceRunId: string;
readonly requestId: string;
readonly mutationId: string;
readonly eventId: string;
readonly subject: Readonly<SecuritySubject>;
readonly policyFence: Readonly<SecurityPolicyFence>;
}
export interface CopilotFailureDiagnosisCancellationResult {
readonly schema: typeof COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_RESULT_SCHEMA;
readonly status: ClusterRunCancellationResult['status'];
readonly convergence: 'terminal' | 'model_in_flight';
readonly projectId: string;
readonly sourceRunId: string;
readonly requestId: string;
readonly diagnosisRunId: string;
readonly runStatus: RunStatus;
readonly outcome: 'succeeded' | 'failed' | 'timed_out' | 'cancelled' | null;
readonly runVersion: number;
readonly eventSequence: number;
readonly cancelRequestedAtMs: number | null;
readonly cancelReason:
| 'user'
| 'policy'
| 'shutdown'
| 'reconcile'
| 'timeout'
| null;
}
export interface CopilotFailureDiagnosisCancellationDependencies {
readonly admissions: Pick<
CopilotFailureDiagnosisAdmissionRepository,
'findByRequestId' | 'findPlanByRequestId'
>;
readonly cancellations: ClusterRunCancellationRepository;
readonly terminalizations: CopilotFailureDiagnosisPreModelTerminalizationDependencies;
readonly terminalizeBeforeModel?: typeof terminalizeCopilotFailureDiagnosisBeforeModel;
}
export class InvalidCopilotFailureDiagnosisCancellationError extends TypeError {
readonly code = 'COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_INVALID';
constructor() {
super('Copilot failure diagnosis cancellation is invalid');
this.name = 'InvalidCopilotFailureDiagnosisCancellationError';
}
}
export class CopilotFailureDiagnosisCancellationNotFoundError extends Error {
readonly code = 'COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_NOT_FOUND';
constructor() {
super('Copilot failure diagnosis cancellation target does not exist');
this.name = 'CopilotFailureDiagnosisCancellationNotFoundError';
}
}
export class CopilotFailureDiagnosisCancellationUnavailableError extends Error {
readonly code = 'COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_UNAVAILABLE';
constructor(options?: ErrorOptions) {
super('Copilot failure diagnosis cancellation is unavailable', options);
this.name = 'CopilotFailureDiagnosisCancellationUnavailableError';
}
}
const IDENTITY = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const RUN_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$/;
const TERMINAL = new Set<RunStatus>([
'succeeded',
'failed',
'cancelled',
'timed_out',
]);
function unavailable(cause?: unknown): never {
throw new CopilotFailureDiagnosisCancellationUnavailableError({
cause: cause instanceof Error ? cause : undefined,
});
}
function exactKeys(value: object, expected: readonly string[]): boolean {
const actual = Object.keys(value).sort();
const canonical = [...expected].sort();
return (
actual.length === canonical.length &&
actual.every((key, index) => key === canonical[index])
);
}
function command(
value: CopilotFailureDiagnosisCancellationCommand,
): Readonly<CopilotFailureDiagnosisCancellationCommand> {
if (
!value ||
typeof value !== 'object' ||
Array.isArray(value) ||
!exactKeys(value, [
'eventId',
'mutationId',
'policyFence',
'projectId',
'requestId',
'sourceRunId',
'subject',
]) ||
!IDENTITY.test(value.projectId) ||
!RUN_ID.test(value.sourceRunId) ||
!IDENTITY.test(value.requestId) ||
!IDENTITY.test(value.mutationId) ||
!IDENTITY.test(value.eventId) ||
!value.subject ||
typeof value.subject !== 'object' ||
!value.policyFence ||
typeof value.policyFence !== 'object'
) {
throw new InvalidCopilotFailureDiagnosisCancellationError();
}
return Object.freeze({ ...value });
}
function evidenceMatches(
plan: Readonly<CopilotFailureDiagnosisExecutionPlan>,
receipt: Readonly<CopilotFailureDiagnosisAdmissionReceipt>,
): boolean {
return (
receipt.requestId === plan.requestId &&
receipt.planDigest === plan.planDigest &&
receipt.runId === plan.runId &&
receipt.sourceRunId === plan.source.runId &&
receipt.sourceRunVersion === plan.source.runVersion &&
receipt.sourceAttemptId === plan.source.attemptId &&
receipt.toolStepRunId === plan.toolStepRunId &&
receipt.modelStepRunId === plan.modelStepRunId
);
}
function targetMatches(
plan: Readonly<CopilotFailureDiagnosisExecutionPlan>,
value: Readonly<CopilotFailureDiagnosisCancellationCommand>,
): boolean {
return (
plan.projectId === value.projectId &&
plan.source.runId === value.sourceRunId &&
plan.requestId === value.requestId
);
}
function result(
target: Readonly<CopilotFailureDiagnosisCancellationCommand>,
cancellation: Readonly<ClusterRunCancellationResult>,
state: Readonly<{
convergence: CopilotFailureDiagnosisCancellationResult['convergence'];
runStatus: RunStatus;
runVersion: number;
eventSequence: number;
cancelRequestedAtMs?: number;
cancelReason?: NonNullable<
CopilotFailureDiagnosisCancellationResult['cancelReason']
>;
}>,
): Readonly<CopilotFailureDiagnosisCancellationResult> {
if (
cancellation.projectId !== target.projectId ||
!RUN_STATUSES.includes(state.runStatus) ||
!Number.isSafeInteger(state.runVersion) ||
state.runVersion < 0 ||
!Number.isSafeInteger(state.eventSequence) ||
state.eventSequence < 0 ||
(state.convergence === 'terminal') !== TERMINAL.has(state.runStatus) ||
(state.convergence === 'model_in_flight' && state.runStatus !== 'running')
) {
return unavailable();
}
return Object.freeze({
schema: COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_RESULT_SCHEMA,
status: cancellation.status,
convergence: state.convergence,
projectId: target.projectId,
sourceRunId: target.sourceRunId,
requestId: target.requestId,
diagnosisRunId: cancellation.runId,
runStatus: state.runStatus,
outcome: TERMINAL.has(state.runStatus)
? (state.runStatus as 'succeeded' | 'failed' | 'timed_out' | 'cancelled')
: null,
runVersion: state.runVersion,
eventSequence: state.eventSequence,
cancelRequestedAtMs: state.cancelRequestedAtMs ?? null,
cancelReason: state.cancelReason ?? null,
});
}
/**
* Resolves an external request key to its server-owned diagnosis Run, writes
* one ordinary Run cancellation intent, then converges only while no Model
* invocation start exists.
*/
export class CopilotFailureDiagnosisCancellationService {
readonly #dependencies: Readonly<CopilotFailureDiagnosisCancellationDependencies>;
readonly #terminalize: typeof terminalizeCopilotFailureDiagnosisBeforeModel;
constructor(dependencies: CopilotFailureDiagnosisCancellationDependencies) {
if (
!dependencies ||
typeof dependencies !== 'object' ||
Array.isArray(dependencies) ||
typeof dependencies.admissions?.findByRequestId !== 'function' ||
typeof dependencies.admissions?.findPlanByRequestId !== 'function' ||
typeof dependencies.cancellations?.requestUserCancellation !==
'function' ||
typeof dependencies.terminalizations?.repository?.findByRequestId !==
'function' ||
typeof dependencies.terminalizations?.repository?.readAuthority !==
'function' ||
typeof dependencies.terminalizations?.repository?.commit !== 'function' ||
(dependencies.terminalizeBeforeModel !== undefined &&
typeof dependencies.terminalizeBeforeModel !== 'function')
) {
throw new InvalidCopilotFailureDiagnosisCancellationError();
}
this.#dependencies = Object.freeze({ ...dependencies });
this.#terminalize =
dependencies.terminalizeBeforeModel ??
terminalizeCopilotFailureDiagnosisBeforeModel;
}
async cancel(
value: CopilotFailureDiagnosisCancellationCommand,
): Promise<Readonly<CopilotFailureDiagnosisCancellationResult>> {
const target = command(value);
let plan: Readonly<CopilotFailureDiagnosisExecutionPlan> | null;
let receipt: Readonly<CopilotFailureDiagnosisAdmissionReceipt> | null;
try {
const located = await Promise.all([
this.#dependencies.admissions.findPlanByRequestId(target.requestId),
this.#dependencies.admissions.findByRequestId(target.requestId),
]);
plan = located[0]
? normalizeCopilotFailureDiagnosisExecutionPlan(located[0])
: null;
receipt = located[1]
? normalizeCopilotFailureDiagnosisAdmissionReceipt(located[1])
: null;
} catch (cause) {
return unavailable(cause);
}
if (!plan || !receipt || !targetMatches(plan, target)) {
throw new CopilotFailureDiagnosisCancellationNotFoundError();
}
if (!evidenceMatches(plan, receipt)) return unavailable();
const cancellation =
await this.#dependencies.cancellations.requestUserCancellation({
projectId: target.projectId,
runId: plan.runId,
mutationId: target.mutationId,
eventId: target.eventId,
subject: target.subject,
policyFence: target.policyFence,
});
if (
cancellation.projectId !== target.projectId ||
cancellation.runId !== plan.runId
) {
return unavailable();
}
if (TERMINAL.has(cancellation.runStatus)) {
return result(target, cancellation, {
convergence: 'terminal',
runStatus: cancellation.runStatus,
runVersion: cancellation.runVersion,
eventSequence: cancellation.eventSequence,
...(cancellation.cancelRequestedAtMs === undefined
? {}
: { cancelRequestedAtMs: cancellation.cancelRequestedAtMs }),
...(cancellation.cancelReason === undefined
? {}
: { cancelReason: cancellation.cancelReason }),
});
}
for (let attempt = 0; attempt < 2; attempt += 1) {
try {
const terminalized = await this.#terminalize(
target.requestId,
{ kind: 'boundary' },
this.#dependencies.terminalizations,
);
return result(target, cancellation, {
convergence: 'terminal',
runStatus: terminalized.receipt.outcome,
runVersion: terminalized.receipt.finalRunVersion,
eventSequence: terminalized.receipt.finalRunEventSequence,
...(cancellation.cancelRequestedAtMs === undefined
? {}
: { cancelRequestedAtMs: cancellation.cancelRequestedAtMs }),
...(cancellation.cancelReason === undefined
? {}
: { cancelReason: cancellation.cancelReason }),
});
} catch (cause) {
if (
!(
cause instanceof
CopilotFailureDiagnosisPreModelTerminalizationConflictError
) &&
!(
cause instanceof
CopilotFailureDiagnosisPreModelTerminalizationNotReadyError
)
) {
if (
cause instanceof
CopilotFailureDiagnosisPreModelTerminalizationUnavailableError
) {
return unavailable(cause);
}
return unavailable(cause);
}
}
try {
const authority =
await this.#dependencies.terminalizations.repository.readAuthority(
target.requestId,
);
if (
!targetMatches(authority.plan, target) ||
authority.plan.planDigest !== plan.planDigest ||
authority.run.id !== plan.runId ||
authority.run.projectId !== target.projectId ||
authority.run.version !== authority.run.eventSequence
) {
return unavailable();
}
if (TERMINAL.has(authority.run.status)) {
return result(target, cancellation, {
convergence: 'terminal',
runStatus: authority.run.status,
runVersion: authority.run.version,
eventSequence: authority.run.eventSequence,
...(authority.run.cancelRequestedAtMs === undefined
? {}
: { cancelRequestedAtMs: authority.run.cancelRequestedAtMs }),
...(authority.run.cancelReason === undefined
? {}
: { cancelReason: authority.run.cancelReason }),
});
}
if (
authority.run.status === 'running' &&
authority.modelStartExists &&
authority.run.cancelRequestedAtMs !== undefined &&
authority.run.cancelReason !== undefined
) {
return result(target, cancellation, {
convergence: 'model_in_flight',
runStatus: 'running',
runVersion: authority.run.version,
eventSequence: authority.run.eventSequence,
cancelRequestedAtMs: authority.run.cancelRequestedAtMs,
cancelReason: authority.run.cancelReason,
});
}
if (
attempt === 0 &&
authority.run.status === 'running' &&
!authority.modelStartExists &&
authority.run.cancelRequestedAtMs !== undefined &&
authority.run.cancelReason !== undefined
) {
continue;
}
return unavailable();
} catch (cause) {
return unavailable(cause);
}
}
return unavailable();
}
}
@@ -0,0 +1,341 @@
const assert = require('node:assert/strict');
const { test } = require('node:test');
const {
BUILTIN_RUN_LOG_EXCERPT_TOOL,
BUILTIN_RUN_LOG_EXCERPT_TOOL_DEFINITION,
createBuiltInRunLogExcerptToolHandlerBinding,
} = require('@qinglong/runtime-core/builtin-run-log-excerpt-tool');
const {
createPluginPackageResourceGenerationFromReferences,
} = require('@qinglong/runtime-core/plugin-package-resource-generation');
const {
createProjectToolDefinitionSnapshot,
projectToolDefinitionRegistry,
} = require('@qinglong/runtime-core/project-tool-definition-snapshot');
const {
prepareToolInvocation,
} = require('@qinglong/runtime-core/tool-registry');
const {
TrustedToolHandlerBindingRegistry,
createTrustedToolInvocationPlan,
} = require('@qinglong/runtime-core/trusted-tool-invocation');
const {
CopilotFailureDiagnosisCancellationNotFoundError,
CopilotFailureDiagnosisCancellationService,
CopilotFailureDiagnosisCancellationUnavailableError,
} = require('@qinglong/ai/failure-diagnosis-cancellation');
const {
createCopilotFailureDiagnosisAdmissionBundle,
prepareCopilotFailureDiagnosisExecution,
} = require('@qinglong/ai/failure-diagnosis-execution-admission');
const {
CopilotFailureDiagnosisPreModelTerminalizationConflictError,
} = require('@qinglong/ai/failure-diagnosis-pre-model-terminalization');
async function durablePlan() {
const generation = createPluginPackageResourceGenerationFromReferences({
installationId: 'install-cancel-test',
projectId: 'project-cancel',
packageName: 'qinglong',
lockDigest: 'a'.repeat(64),
generation: 1,
previousActiveLockDigest: null,
contentDigest: 'b'.repeat(64),
resources: [],
});
const snapshot = createProjectToolDefinitionSnapshot({
projectId: 'project-cancel',
contributions: [
{
generation,
revisionDigest: 'c'.repeat(64),
definitions: [BUILTIN_RUN_LOG_EXCERPT_TOOL_DEFINITION],
},
],
});
const binding = createBuiltInRunLogExcerptToolHandlerBinding(snapshot, [
'cluster-control',
]);
const bindings = new TrustedToolHandlerBindingRegistry(snapshot, [binding]);
const principal = {
subject: { type: 'user', id: 'owner-cancel' },
authenticationId: 'auth-cancel',
authenticatedAtMs: 100,
expiresAtMs: 10_000,
assurance: 'multi_factor',
};
const invocation = await prepareToolInvocation(
projectToolDefinitionRegistry(snapshot),
{
projectId: 'project-cancel',
principal,
nowMs: 200,
tool: BUILTIN_RUN_LOG_EXCERPT_TOOL,
input: { runId: 'source-run-cancel', attemptId: 'source-attempt-cancel' },
},
{
async authorize() {
return {
effect: 'allow',
reasons: ['role_grant'],
fence: { projectVersion: 2, bindingVersion: 3 },
};
},
},
);
const tool = createTrustedToolInvocationPlan(bindings, invocation, {
actionRef: 'cancel-log-tool',
inputArtifactId: 'cancel-input',
previewArtifactId: 'cancel-preview',
artifactKeyId: 'cancel-key',
artifactKey: Buffer.alloc(32, 0x11),
artifactNonce: Buffer.alloc(12, 0x22),
profile: 'cluster-control',
preview: {
title: 'Read failed Run log',
summary: 'Read bounded evidence',
fields: [
{ kind: 'identifier', label: 'Run', value: 'source-run-cancel' },
{
kind: 'identifier',
label: 'Attempt',
value: 'source-attempt-cancel',
},
],
warnings: ['potentially_sensitive_output'],
},
sealedAtMs: 300,
});
return prepareCopilotFailureDiagnosisExecution({
requestId: 'diagnosis-request-cancel',
traceId: 'diagnosis-trace-cancel',
source: {
runId: 'source-run-cancel',
runVersion: 4,
runStatus: 'failed',
attemptId: 'source-attempt-cancel',
attemptStatus: 'failed',
attemptFinishedAtMs: 250,
logArtifactId: 'source-log-cancel',
},
toolPlan: tool.plan,
bindings,
model: {
provider: 'provider-primary',
model: 'model-diagnosis',
modelBoundary: 'external',
responseLanguage: 'zh-CN',
maxOutputTokens: 256,
egressPolicy: {
schema: 'qinglong/copilot-model-egress-policy@v1',
revision: 'cancel-policy-v1',
potentiallySensitiveDataBoundaries: ['external'],
maxInputBytes: 64 * 1024,
maxOutputTokens: 512,
},
},
plannedAtMs: 400,
deadlineAtMs: 8_000,
});
}
function command(overrides = {}) {
return {
projectId: 'project-cancel',
sourceRunId: 'source-run-cancel',
requestId: 'diagnosis-request-cancel',
mutationId: '11111111-1111-4111-8111-111111111111',
eventId: '22222222-2222-4222-8222-222222222222',
subject: { type: 'user', id: 'owner-cancel' },
policyFence: { projectVersion: 2, bindingVersion: 3 },
...overrides,
};
}
function fixture(plan, overrides = {}) {
const receipt = createCopilotFailureDiagnosisAdmissionBundle(plan).receipt;
const state = {
cancellationCalls: [],
terminalizeCalls: 0,
cancellation: {
status: 'accepted',
projectId: plan.projectId,
runId: plan.runId,
runStatus: 'running',
runVersion: 4,
eventSequence: 4,
cancelRequestedAtMs: 500,
cancelReason: 'user',
},
authority: {
plan,
run: {
id: plan.runId,
projectId: plan.projectId,
status: 'running',
version: 4,
eventSequence: 4,
cancelRequestedAtMs: 500,
cancelReason: 'user',
},
toolStep: {},
modelStep: {},
modelStartExists: false,
observedAtMs: 501,
},
...overrides,
};
const terminalizations = {
repository: {
async findByRequestId() {
return null;
},
async readAuthority() {
return state.authority;
},
async commit() {
throw new Error('not called by injected terminalizer');
},
},
};
const service = new CopilotFailureDiagnosisCancellationService({
admissions: {
async findPlanByRequestId() {
return state.plan === undefined ? plan : state.plan;
},
async findByRequestId() {
return state.receipt === undefined ? receipt : state.receipt;
},
},
cancellations: {
async requestUserCancellation(value) {
state.cancellationCalls.push(value);
return state.cancellation;
},
},
terminalizations,
async terminalizeBeforeModel(requestId, trigger) {
state.terminalizeCalls += 1;
if (state.terminalizeError) throw state.terminalizeError;
assert.equal(requestId, plan.requestId);
assert.deepEqual(trigger, { kind: 'boundary' });
return {
status: 'created',
receipt: {
outcome: 'cancelled',
finalRunVersion: 7,
finalRunEventSequence: 7,
},
};
},
});
return { service, state, receipt };
}
test('resolves the request key to a server-owned Run and terminalizes pre-Model', async () => {
const plan = await durablePlan();
const { service, state } = fixture(plan);
const result = await service.cancel(command());
assert.deepEqual(state.cancellationCalls, [
{
projectId: 'project-cancel',
runId: plan.runId,
mutationId: '11111111-1111-4111-8111-111111111111',
eventId: '22222222-2222-4222-8222-222222222222',
subject: { type: 'user', id: 'owner-cancel' },
policyFence: { projectVersion: 2, bindingVersion: 3 },
},
]);
assert.equal(state.terminalizeCalls, 1);
assert.equal(result.status, 'accepted');
assert.equal(result.convergence, 'terminal');
assert.equal(result.runStatus, 'cancelled');
assert.equal(result.outcome, 'cancelled');
assert.equal(result.diagnosisRunId, plan.runId);
assert.equal(result.cancelRequestedAtMs, 500);
});
test('masks cross-target requests before writing a cancellation intent', async () => {
const plan = await durablePlan();
const { service, state } = fixture(plan);
await assert.rejects(
service.cancel(command({ sourceRunId: 'other-source-run' })),
CopilotFailureDiagnosisCancellationNotFoundError,
);
assert.equal(state.cancellationCalls.length, 0);
});
test('keeps a durable intent pending when Model start wins the race', async () => {
const plan = await durablePlan();
const conflict =
new CopilotFailureDiagnosisPreModelTerminalizationConflictError(
'Model started',
);
const { service, state } = fixture(plan, {
terminalizeError: conflict,
authority: {
plan,
run: {
id: plan.runId,
projectId: plan.projectId,
status: 'running',
version: 5,
eventSequence: 5,
cancelRequestedAtMs: 500,
cancelReason: 'user',
},
toolStep: {},
modelStep: {},
modelStartExists: true,
observedAtMs: 501,
},
});
const result = await service.cancel(command());
assert.equal(state.terminalizeCalls, 1);
assert.equal(result.convergence, 'model_in_flight');
assert.equal(result.runStatus, 'running');
assert.equal(result.outcome, null);
assert.equal(result.runVersion, 5);
});
test('reports the real terminal winner instead of forging cancellation', async () => {
const plan = await durablePlan();
const { service } = fixture(plan, {
terminalizeError:
new CopilotFailureDiagnosisPreModelTerminalizationConflictError(),
authority: {
plan,
run: {
id: plan.runId,
projectId: plan.projectId,
status: 'succeeded',
version: 7,
eventSequence: 7,
cancelRequestedAtMs: 500,
cancelReason: 'user',
},
toolStep: {},
modelStep: {},
modelStartExists: true,
observedAtMs: 600,
},
});
const result = await service.cancel(command());
assert.equal(result.convergence, 'terminal');
assert.equal(result.runStatus, 'succeeded');
assert.equal(result.outcome, 'succeeded');
});
test('fails closed when plan and admission receipt drift', async () => {
const plan = await durablePlan();
const base = fixture(plan);
const { service, state } = fixture(plan, {
receipt: { ...base.receipt, runId: 'diagnosis-run-drift' },
});
await assert.rejects(
service.cancel(command()),
CopilotFailureDiagnosisCancellationUnavailableError,
);
assert.equal(state.cancellationCalls.length, 0);
});
+10
View File
@@ -40,6 +40,11 @@
"require": "./dist/application-runtime/copilot/failureDiagnosisComposition.js",
"default": "./dist/application-runtime/copilot/failureDiagnosisComposition.js"
},
"./copilot-cancellation-production": {
"types": "./dist/application-runtime/copilot/failureDiagnosisCancellationComposition.d.ts",
"require": "./dist/application-runtime/copilot/failureDiagnosisCancellationComposition.js",
"default": "./dist/application-runtime/copilot/failureDiagnosisCancellationComposition.js"
},
"./failure-diagnosis-output-keyring": {
"types": "./dist/copilot/failure-diagnosis/outputProjectedKeyring.d.ts",
"require": "./dist/copilot/failure-diagnosis/outputProjectedKeyring.js",
@@ -55,6 +60,11 @@
"require": "./dist/copilot/failure-diagnosis/failureDiagnosisReadRoutes.js",
"default": "./dist/copilot/failure-diagnosis/failureDiagnosisReadRoutes.js"
},
"./copilot-cancellation-route": {
"types": "./dist/copilot/failure-diagnosis/failureDiagnosisCancellationRoute.d.ts",
"require": "./dist/copilot/failure-diagnosis/failureDiagnosisCancellationRoute.js",
"default": "./dist/copilot/failure-diagnosis/failureDiagnosisCancellationRoute.js"
},
"./http": {
"types": "./dist/transport/httpSurface.d.ts",
"require": "./dist/transport/httpSurface.js",
@@ -2,6 +2,7 @@ import type { ModelGatewayProfileAudit } from '@qinglong/ai/profile';
import type { DurableModelInvocationCoordinator } from '@qinglong/ai/durable-model-invocation';
import { CopilotFailureDiagnosisModelCompletionCoordinator } from '@qinglong/ai/failure-diagnosis-model-execution';
import type { CopilotFailureDiagnosisApplicationService } from '@qinglong/ai/failure-diagnosis-application';
import type { CopilotFailureDiagnosisCancellationService } from '@qinglong/ai/failure-diagnosis-cancellation';
import { BoundModelProviderCredentialProvider } from '@qinglong/ai/provider-credential';
import { PostgresModelProviderCredentialReader } from '@qinglong/ai/postgres-model-provider-credential-storage';
import { loadProjectedModelGatewayProviderAuthority } from '@qinglong/ai/projected-model-gateway-authority';
@@ -35,6 +36,10 @@ import {
createProductionClusterCopilotFailureDiagnosisReadService,
type CreateProductionClusterCopilotFailureDiagnosisReadServiceOptions,
} from './copilot/failureDiagnosisReadComposition';
import {
createProductionClusterCopilotFailureDiagnosisCancellation,
type CreateProductionClusterCopilotFailureDiagnosisCancellationOptions,
} from './copilot/failureDiagnosisCancellationComposition';
export interface EnabledProductionClusterAiConfig {
readonly enabled: true;
@@ -63,6 +68,9 @@ export interface ProductionClusterAiControlApplicationOptions {
) => ReturnType<
typeof createProductionClusterCopilotFailureDiagnosisReadService
>;
readonly createCopilotCancellation?: (
options: CreateProductionClusterCopilotFailureDiagnosisCancellationOptions,
) => Readonly<CopilotFailureDiagnosisCancellationService>;
readonly openAiDatabase?: ReturnType<typeof createPostgresDatabaseOpener>;
}
@@ -263,11 +271,15 @@ export async function startProductionClusterAiControlApplication(
const createCopilotRead =
options.createCopilotRead ??
createProductionClusterCopilotFailureDiagnosisReadService;
const createCopilotCancellation =
options.createCopilotCancellation ??
createProductionClusterCopilotFailureDiagnosisCancellation;
if (
typeof startControl !== 'function' ||
typeof bootstrapPrompt !== 'function' ||
typeof createCopilot !== 'function' ||
typeof createCopilotRead !== 'function' ||
typeof createCopilotCancellation !== 'function' ||
(options.openAiDatabase !== undefined &&
typeof options.openAiDatabase !== 'function')
) {
@@ -325,6 +337,9 @@ export async function startProductionClusterAiControlApplication(
typeof createProductionClusterCopilotFailureDiagnosisReadService
>
| undefined;
let copilotCancellationApplication:
| Readonly<CopilotFailureDiagnosisCancellationService>
| undefined;
let copilotSuccessfulCompletion:
| CopilotFailureDiagnosisModelCompletionCoordinator
| undefined;
@@ -455,6 +470,9 @@ export async function startProductionClusterAiControlApplication(
pool: aiDatabase.pool,
prepared: preparedCopilot,
});
copilotCancellationApplication = createCopilotCancellation({
pool: aiDatabase.pool,
});
}
controlApplication = await startControl({
...options.control,
@@ -482,12 +500,14 @@ export async function startProductionClusterAiControlApplication(
},
}),
...(copilotApplication === undefined ||
copilotReadApplication === undefined
copilotReadApplication === undefined ||
copilotCancellationApplication === undefined
? {}
: {
copilotFailureDiagnosis: {
capability: copilotApplication,
readCapability: copilotReadApplication,
cancellationCapability: copilotCancellationApplication,
},
}),
});
@@ -0,0 +1,47 @@
import { CopilotFailureDiagnosisCancellationService } from '@qinglong/ai/failure-diagnosis-cancellation';
import { PostgresCopilotFailureDiagnosisAdmissionRepository } from '@qinglong/ai/postgres-failure-diagnosis-admission-storage';
import {
PostgresCopilotFailureDiagnosisPreModelTerminalizationRepository,
terminalizeCopilotFailureDiagnosisBeforeModel,
} from '@qinglong/ai/failure-diagnosis-pre-model-terminalization';
import {
PostgresClusterRunCancellationRepository,
type QingLongPostgresPool,
} from '@qinglong/cluster-postgres/runtime';
export interface CreateProductionClusterCopilotFailureDiagnosisCancellationOptions {
readonly pool: QingLongPostgresPool;
}
/**
* Reuses the AI Pool and existing Run/admission ledgers. The cancellation
* capability owns no connection, timer, listener or background lifecycle.
*/
export function createProductionClusterCopilotFailureDiagnosisCancellation(
options: CreateProductionClusterCopilotFailureDiagnosisCancellationOptions,
): Readonly<CopilotFailureDiagnosisCancellationService> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
typeof options.pool?.query !== 'function' ||
typeof options.pool?.connect !== 'function'
) {
throw new TypeError(
'Production Cluster Copilot failure diagnosis cancellation dependencies are invalid',
);
}
const admissions = new PostgresCopilotFailureDiagnosisAdmissionRepository(
options.pool,
);
const terminalizations =
new PostgresCopilotFailureDiagnosisPreModelTerminalizationRepository(
options.pool,
);
return new CopilotFailureDiagnosisCancellationService({
admissions,
cancellations: new PostgresClusterRunCancellationRepository(options.pool),
terminalizations: Object.freeze({ repository: terminalizations }),
terminalizeBeforeModel: terminalizeCopilotFailureDiagnosisBeforeModel,
});
}
@@ -71,6 +71,10 @@ import {
type ClusterCopilotFailureDiagnosisInspectionCapability,
type ClusterCopilotFailureDiagnosisOutputReadCapability,
} from '../copilot/failure-diagnosis/failureDiagnosisReadRoutes';
import {
createClusterControlCopilotFailureDiagnosisCancellationRoute,
type ClusterCopilotFailureDiagnosisCancellationCapability,
} from '../copilot/failure-diagnosis/failureDiagnosisCancellationRoute';
export const PRODUCTION_CLUSTER_CONTROL_ROUTE_OPERATIONS = Object.freeze([
'task.get',
@@ -100,6 +104,7 @@ export const PRODUCTION_CLUSTER_CONTROL_OPTIONAL_ROUTE_OPERATIONS =
'copilot.failure_diagnosis.execute',
'copilot.failure_diagnosis.read',
'copilot.failure_diagnosis.output.read',
'copilot.failure_diagnosis.cancel',
] as const);
export interface ClusterCopilotFailureDiagnosisReadCapability
@@ -129,6 +134,7 @@ export interface ProductionClusterControlAssemblyOptions {
readonly copilotFailureDiagnosis?: Readonly<{
readonly capability: ClusterCopilotFailureDiagnosisCapability;
readonly readCapability?: ClusterCopilotFailureDiagnosisReadCapability;
readonly cancellationCapability?: ClusterCopilotFailureDiagnosisCancellationCapability;
}>;
readonly workerIngress?: Readonly<{
readonly config: EnabledClusterWorkerIngressConfig;
@@ -178,6 +184,7 @@ export interface ProductionClusterControlApplicationOptions
readonly copilotFailureDiagnosis?: Readonly<{
readonly capability: ClusterCopilotFailureDiagnosisCapability;
readonly readCapability?: ClusterCopilotFailureDiagnosisReadCapability;
readonly cancellationCapability?: ClusterCopilotFailureDiagnosisCancellationCapability;
}>;
readonly workerIngress?: ProductionClusterWorkerIngressOptions;
}
@@ -303,6 +310,14 @@ export function createProductionClusterControlApplicationStack(
options.copilotFailureDiagnosis.readCapability,
),
]),
...(options.copilotFailureDiagnosis?.cancellationCapability === undefined
? []
: [
createClusterControlCopilotFailureDiagnosisCancellationRoute(
options.copilotFailureDiagnosis.cancellationCapability,
createEventId,
),
]),
];
const routes = createClusterControlRouteRegistry(routeDefinitions);
const expectedRouteCount =
@@ -313,7 +328,10 @@ export function createProductionClusterControlApplicationStack(
(options.promptOutputRead === undefined ? 0 : 1) +
(options.promptExecutionOutputRead === undefined ? 0 : 1) +
(options.copilotFailureDiagnosis === undefined ? 0 : 1) +
(options.copilotFailureDiagnosis?.readCapability === undefined ? 0 : 2);
(options.copilotFailureDiagnosis?.readCapability === undefined ? 0 : 2) +
(options.copilotFailureDiagnosis?.cancellationCapability === undefined
? 0
: 1);
if (routes.size !== expectedRouteCount) {
throw new Error('Production cluster-control route allowlist is incomplete');
}
@@ -0,0 +1,261 @@
// Cluster Copilot resolves an external request key before cancelling its Run.
import {
CopilotFailureDiagnosisCancellationNotFoundError,
CopilotFailureDiagnosisCancellationUnavailableError,
InvalidCopilotFailureDiagnosisCancellationError,
type CopilotFailureDiagnosisCancellationCommand,
} from '@qinglong/ai/failure-diagnosis-cancellation';
import {
CLUSTER_RUN_CANCELLATION_SCHEMA,
ClusterRunCancellationFenceRejectedError,
ClusterRunCancellationNotFoundError,
ClusterRunCancellationUnavailableError,
InvalidClusterRunCancellationError,
parseClusterRunCancellationRequestBody,
} from '@qinglong/runtime-core/cluster-run-cancellation';
import type { ClusterControlAdmissionResponse } from '../../transport/httpSurface';
import type {
ClusterControlAuthorizedOperationRequest,
ClusterControlRouteDefinition,
ClusterControlRouteParameters,
} from '../../transport/routeRegistry';
export const CLUSTER_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_RESPONSE_SCHEMA =
'qinglong/cluster-copilot-failure-diagnosis-cancellation-response@v1' as const;
export const CLUSTER_CONTROL_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_ROUTE =
Object.freeze({
method: 'POST' as const,
path: '/api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses/{requestId}/cancellation',
operationId: 'copilot.failure_diagnosis.cancel',
permission: 'run.stop',
projectParameter: 'projectId',
});
export interface ClusterCopilotFailureDiagnosisCancellationCapability {
cancel(
command: Readonly<CopilotFailureDiagnosisCancellationCommand>,
): Promise<unknown>;
}
export type ClusterCopilotFailureDiagnosisCancellationEventIdFactory =
() => string;
const IDENTITY = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const RUN_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,35}$/;
const OUTCOMES = new Set(['succeeded', 'failed', 'timed_out', 'cancelled']);
const STATUSES = new Set(['accepted', 'already_requested', 'already_terminal']);
const CANCEL_REASONS = new Set([
'user',
'policy',
'shutdown',
'reconcile',
'timeout',
]);
function response(
statusCode: number,
body: Readonly<Record<string, unknown>>,
): ClusterControlAdmissionResponse {
return Object.freeze({ statusCode, body: Object.freeze(body) });
}
function exactRecord(
value: unknown,
expected: readonly string[],
): Record<string, unknown> | null {
if (!value || typeof value !== 'object' || Array.isArray(value)) return null;
const record = value as Record<string, unknown>;
const actual = Object.keys(record).sort();
const canonical = [...expected].sort();
return actual.length === canonical.length &&
actual.every((key, index) => key === canonical[index])
? record
: null;
}
function safeInteger(value: unknown): value is number {
return Number.isSafeInteger(value) && (value as number) >= 0;
}
function projectResult(
value: unknown,
target: Readonly<{
projectId: string;
sourceRunId: string;
requestId: string;
}>,
): Readonly<Record<string, unknown>> | null {
const candidate = exactRecord(value, [
'cancelReason',
'cancelRequestedAtMs',
'convergence',
'diagnosisRunId',
'eventSequence',
'outcome',
'projectId',
'requestId',
'runStatus',
'runVersion',
'schema',
'sourceRunId',
'status',
]);
if (
!candidate ||
candidate.schema !==
'qinglong/copilot-failure-diagnosis-cancellation-result@v1' ||
candidate.projectId !== target.projectId ||
candidate.sourceRunId !== target.sourceRunId ||
candidate.requestId !== target.requestId ||
typeof candidate.diagnosisRunId !== 'string' ||
!RUN_ID.test(candidate.diagnosisRunId) ||
typeof candidate.status !== 'string' ||
!STATUSES.has(candidate.status) ||
!safeInteger(candidate.runVersion) ||
!safeInteger(candidate.eventSequence) ||
candidate.runVersion !== candidate.eventSequence ||
!(
(candidate.cancelRequestedAtMs === null &&
candidate.cancelReason === null) ||
(safeInteger(candidate.cancelRequestedAtMs) &&
typeof candidate.cancelReason === 'string' &&
CANCEL_REASONS.has(candidate.cancelReason))
)
) {
return null;
}
if (candidate.convergence === 'model_in_flight') {
if (
candidate.runStatus !== 'running' ||
candidate.outcome !== null ||
candidate.cancelRequestedAtMs === null
) {
return null;
}
} else if (candidate.convergence === 'terminal') {
if (
typeof candidate.runStatus !== 'string' ||
!OUTCOMES.has(candidate.runStatus) ||
candidate.outcome !== candidate.runStatus
) {
return null;
}
} else {
return null;
}
return Object.freeze({
schema: CLUSTER_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_RESPONSE_SCHEMA,
status: candidate.status,
convergence: candidate.convergence,
projectId: target.projectId,
sourceRunId: target.sourceRunId,
requestId: target.requestId,
diagnosisRunId: candidate.diagnosisRunId,
runStatus: candidate.runStatus,
outcome: candidate.outcome,
runVersion: candidate.runVersion,
eventSequence: candidate.eventSequence,
cancelRequestedAtMs: candidate.cancelRequestedAtMs,
cancelReason: candidate.cancelReason,
});
}
export function createClusterControlCopilotFailureDiagnosisCancellationRoute(
capability: ClusterCopilotFailureDiagnosisCancellationCapability,
createEventId: ClusterCopilotFailureDiagnosisCancellationEventIdFactory,
): Readonly<ClusterControlRouteDefinition> {
if (
!capability ||
typeof capability.cancel !== 'function' ||
typeof createEventId !== 'function'
) {
throw new TypeError(
'Cluster-control Copilot failure diagnosis cancellation route is invalid',
);
}
return Object.freeze({
...CLUSTER_CONTROL_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_ROUTE,
async handle(
authorized: ClusterControlAuthorizedOperationRequest,
parameters: ClusterControlRouteParameters,
) {
let body;
try {
body = parseClusterRunCancellationRequestBody(authorized.request.body);
} catch (error) {
return error instanceof InvalidClusterRunCancellationError
? response(400, {
code: 'invalid_copilot_failure_diagnosis_cancellation_request',
schema: CLUSTER_RUN_CANCELLATION_SCHEMA,
})
: response(503, {
code: 'copilot_failure_diagnosis_cancellation_unavailable',
});
}
const projectId = authorized.projectId;
const sourceRunId = parameters.runId;
const requestId = parameters.requestId;
if (
projectId === null ||
typeof sourceRunId !== 'string' ||
!RUN_ID.test(sourceRunId) ||
typeof requestId !== 'string' ||
!IDENTITY.test(requestId) ||
!authorized.policyFence ||
authorized.policyFence.bindingVersion === null
) {
return response(503, {
code: 'copilot_failure_diagnosis_cancellation_unavailable',
});
}
const target = Object.freeze({ projectId, sourceRunId, requestId });
try {
const result = await capability.cancel({
...target,
mutationId: body.mutationId,
eventId: createEventId(),
subject: authorized.principal.subject,
policyFence: authorized.policyFence,
});
const view = projectResult(result, target);
if (!view) {
return response(503, {
code: 'copilot_failure_diagnosis_cancellation_unavailable',
});
}
return response(view.status === 'accepted' ? 202 : 200, view);
} catch (error) {
if (
error instanceof CopilotFailureDiagnosisCancellationNotFoundError ||
error instanceof ClusterRunCancellationNotFoundError
) {
return response(404, {
code: 'copilot_failure_diagnosis_not_found',
});
}
if (error instanceof ClusterRunCancellationFenceRejectedError) {
return response(409, {
code: 'copilot_failure_diagnosis_cancellation_fence_rejected',
reason: error.reason,
});
}
if (
error instanceof InvalidCopilotFailureDiagnosisCancellationError ||
error instanceof
CopilotFailureDiagnosisCancellationUnavailableError ||
error instanceof InvalidClusterRunCancellationError ||
error instanceof ClusterRunCancellationUnavailableError
) {
return response(503, {
code: 'copilot_failure_diagnosis_cancellation_unavailable',
});
}
return response(503, {
code: 'copilot_failure_diagnosis_cancellation_unavailable',
});
}
},
});
}
@@ -169,9 +169,11 @@ test('Copilot composition is explicit, shares the Prompt gateway and injects one
const artifactStore = { put() {}, inspect() {}, readLogRange() {} };
const copilot = Object.freeze({ execute() {} });
const copilotRead = Object.freeze({ inspect() {}, readOutput() {} });
const copilotCancellation = Object.freeze({ cancel() {} });
let registeredSink;
let created;
let createdRead;
let createdCancellation;
let controlOptions;
try {
await Promise.all([
@@ -233,6 +235,10 @@ test('Copilot composition is explicit, shares the Prompt gateway and injects one
createdRead = options;
return copilotRead;
},
createCopilotCancellation(options) {
createdCancellation = options;
return copilotCancellation;
},
async startControl(options) {
controlOptions = options;
return {
@@ -257,11 +263,16 @@ test('Copilot composition is explicit, shares the Prompt gateway and injects one
assert.equal(created.artifactStore, artifactStore);
assert.equal(createdRead.pool, fakePool);
assert.equal(typeof createdRead.prepared.outputKeys.resolve, 'function');
assert.equal(createdCancellation.pool, fakePool);
assert.equal(controlOptions.copilotFailureDiagnosis.capability, copilot);
assert.equal(
controlOptions.copilotFailureDiagnosis.readCapability,
copilotRead,
);
assert.equal(
controlOptions.copilotFailureDiagnosis.cancellationCapability,
copilotCancellation,
);
assert.equal(await application.stop(), 'stopped');
} finally {
config.fill(0);
@@ -0,0 +1,232 @@
const assert = require('node:assert/strict');
const { test } = require('node:test');
const {
CopilotFailureDiagnosisCancellationNotFoundError,
} = require('@qinglong/ai/failure-diagnosis-cancellation');
const {
CLUSTER_RUN_CANCELLATION_SCHEMA,
ClusterRunCancellationFenceRejectedError,
} = require('@qinglong/runtime-core/cluster-run-cancellation');
const {
CLUSTER_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_RESPONSE_SCHEMA,
CLUSTER_CONTROL_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_ROUTE,
createClusterControlCopilotFailureDiagnosisCancellationRoute,
} = require('@qinglong/cluster-control/copilot-cancellation-route');
function authorized(body, overrides = {}) {
return {
request: {
requestId: 'transport-request-1',
method: 'POST',
path: '/api/v3/projects/project-1/runs/source-run-1/copilot/failure-diagnoses/diagnosis-request-1/cancellation',
query: {},
headers: {},
signal: new AbortController().signal,
body,
},
principal: {
subject: { type: 'user', id: 'owner-1' },
authenticationId: 'credential-1',
authenticatedAtMs: 1,
expiresAtMs: 10_000,
assurance: 'multi_factor',
},
operationId: 'copilot.failure_diagnosis.cancel',
permission: 'run.stop',
projectId: 'project-1',
policyFence: { projectVersion: 3, bindingVersion: 7 },
...overrides,
};
}
const parameters = {
projectId: 'project-1',
runId: 'source-run-1',
requestId: 'diagnosis-request-1',
};
function body(overrides = {}) {
return {
schema: CLUSTER_RUN_CANCELLATION_SCHEMA,
mutationId: '11111111-1111-4111-8111-111111111111',
...overrides,
};
}
function result(overrides = {}) {
return {
schema: 'qinglong/copilot-failure-diagnosis-cancellation-result@v1',
status: 'accepted',
convergence: 'terminal',
projectId: 'project-1',
sourceRunId: 'source-run-1',
requestId: 'diagnosis-request-1',
diagnosisRunId: 'diagnosis-run-1',
runStatus: 'cancelled',
outcome: 'cancelled',
runVersion: 7,
eventSequence: 7,
cancelRequestedAtMs: 500,
cancelReason: 'user',
...overrides,
};
}
test('defines an exact run.stop route and passes only fenced target facts', async () => {
let command;
const request = authorized(body());
const route = createClusterControlCopilotFailureDiagnosisCancellationRoute(
{
async cancel(value) {
command = value;
return result();
},
},
() => '22222222-2222-4222-8222-222222222222',
);
const response = await route.handle(request, parameters);
assert.deepEqual(
CLUSTER_CONTROL_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_ROUTE,
{
method: 'POST',
path: '/api/v3/projects/{projectId}/runs/{runId}/copilot/failure-diagnoses/{requestId}/cancellation',
operationId: 'copilot.failure_diagnosis.cancel',
permission: 'run.stop',
projectParameter: 'projectId',
},
);
assert.deepEqual(command, {
projectId: 'project-1',
sourceRunId: 'source-run-1',
requestId: 'diagnosis-request-1',
mutationId: '11111111-1111-4111-8111-111111111111',
eventId: '22222222-2222-4222-8222-222222222222',
subject: request.principal.subject,
policyFence: request.policyFence,
});
assert.deepEqual(response, {
statusCode: 202,
body: {
schema: CLUSTER_COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_RESPONSE_SCHEMA,
status: 'accepted',
convergence: 'terminal',
projectId: 'project-1',
sourceRunId: 'source-run-1',
requestId: 'diagnosis-request-1',
diagnosisRunId: 'diagnosis-run-1',
runStatus: 'cancelled',
outcome: 'cancelled',
runVersion: 7,
eventSequence: 7,
cancelRequestedAtMs: 500,
cancelReason: 'user',
},
});
});
test('projects an in-flight durable intent without claiming Provider abort', async () => {
const route = createClusterControlCopilotFailureDiagnosisCancellationRoute(
{
async cancel() {
return result({
status: 'already_requested',
convergence: 'model_in_flight',
runStatus: 'running',
outcome: null,
runVersion: 6,
eventSequence: 6,
});
},
},
() => '22222222-2222-4222-8222-222222222222',
);
const response = await route.handle(authorized(body()), parameters);
assert.equal(response.statusCode, 200);
assert.equal(response.body.convergence, 'model_in_flight');
assert.equal(response.body.runStatus, 'running');
assert.equal(response.body.outcome, null);
assert.equal('providerAborted' in response.body, false);
});
test('rejects non-exact bodies before invoking the capability', async () => {
let calls = 0;
const route = createClusterControlCopilotFailureDiagnosisCancellationRoute(
{
async cancel() {
calls += 1;
return result();
},
},
() => '22222222-2222-4222-8222-222222222222',
);
for (const value of [
null,
{},
body({ runId: 'caller-selected' }),
body({ reason: 'timeout' }),
body({ mutationId: '' }),
]) {
const response = await route.handle(authorized(value), parameters);
assert.equal(response.statusCode, 400);
}
assert.equal(calls, 0);
});
test('fails closed on widened or identity-drifted capability results', async () => {
for (const value of [
result({ sourceRunId: 'other' }),
result({ privateProvider: 'must-not-cross' }),
result({ runVersion: 8 }),
result({ convergence: 'model_in_flight' }),
]) {
const route = createClusterControlCopilotFailureDiagnosisCancellationRoute(
{
async cancel() {
return value;
},
},
() => '22222222-2222-4222-8222-222222222222',
);
const response = await route.handle(authorized(body()), parameters);
assert.deepEqual(response, {
statusCode: 503,
body: {
code: 'copilot_failure_diagnosis_cancellation_unavailable',
},
});
}
});
test('maps hidden targets, Policy races and storage failures to stable codes', async () => {
for (const [error, statusCode, code] of [
[
new CopilotFailureDiagnosisCancellationNotFoundError(),
404,
'copilot_failure_diagnosis_not_found',
],
[
new ClusterRunCancellationFenceRejectedError('authorization_changed'),
409,
'copilot_failure_diagnosis_cancellation_fence_rejected',
],
[
new Error('private storage detail'),
503,
'copilot_failure_diagnosis_cancellation_unavailable',
],
]) {
const route = createClusterControlCopilotFailureDiagnosisCancellationRoute(
{
async cancel() {
throw error;
},
},
() => '22222222-2222-4222-8222-222222222222',
);
const response = await route.handle(authorized(body()), parameters);
assert.equal(response.statusCode, statusCode);
assert.equal(response.body.code, code);
assert.equal(JSON.stringify(response).includes('private'), false);
}
});
@@ -732,6 +732,7 @@ test('optionally exposes Prompt execution behind shared admission and policy', a
'copilot.failure_diagnosis.execute',
'copilot.failure_diagnosis.read',
'copilot.failure_diagnosis.output.read',
'copilot.failure_diagnosis.cancel',
]);
const response = await invoke(
stack,
@@ -762,6 +763,7 @@ test('optionally exposes Prompt execution behind shared admission and policy', a
test('optionally exposes Copilot diagnosis behind shared authentication, Policy and audit', async () => {
const { events, input } = fixture();
let command;
let cancellationCommand;
const capability = {
async execute(value) {
command = value;
@@ -803,11 +805,30 @@ test('optionally exposes Copilot diagnosis behind shared authentication, Policy
requestId: value.requestId,
};
},
async cancel(value) {
cancellationCommand = value;
return {
schema: 'qinglong/copilot-failure-diagnosis-cancellation-result@v1',
status: 'accepted',
convergence: 'terminal',
projectId: value.projectId,
sourceRunId: value.sourceRunId,
requestId: value.requestId,
diagnosisRunId: 'diagnosis-run-1',
runStatus: 'cancelled',
outcome: 'cancelled',
runVersion: 7,
eventSequence: 7,
cancelRequestedAtMs: 2_000,
cancelReason: 'user',
};
},
};
const stack = createProductionClusterControlApplicationStack(input, {
copilotFailureDiagnosis: {
capability,
readCapability: capability,
cancellationCapability: capability,
},
});
const result = await invoke(
@@ -849,6 +870,25 @@ test('optionally exposes Copilot diagnosis behind shared authentication, Policy
);
assert.equal(inspection.statusCode, 404);
assert.equal(output.statusCode, 404);
const cancellation = await invoke(
stack,
metadata(
'/api/v3/projects/project-1/runs/run-1/copilot/failure-diagnoses/diagnosis-request-1/cancellation',
'POST',
{
schema: 'qinglong/run-cancellation@v1',
mutationId: '00000000-0000-4000-8000-000000000099',
},
),
);
assert.equal(cancellation.statusCode, 202);
assert.equal(cancellationCommand.projectId, 'project-1');
assert.equal(cancellationCommand.sourceRunId, 'run-1');
assert.equal(cancellationCommand.requestId, 'diagnosis-request-1');
assert.deepEqual(cancellationCommand.policyFence, {
projectVersion: 3,
bindingVersion: 7,
});
assert.equal(
events.includes('audit:copilot.failure_diagnosis.read:allowed'),
true,
@@ -857,6 +897,10 @@ test('optionally exposes Copilot diagnosis behind shared authentication, Policy
events.includes('audit:copilot.failure_diagnosis.output.read:allowed'),
true,
);
assert.equal(
events.includes('audit:copilot.failure_diagnosis.cancel:allowed'),
true,
);
});
test('keeps the Copilot route absent by default and never invokes it after Policy denial', async () => {
@@ -879,6 +923,7 @@ test('keeps the Copilot route absent by default and never invokes it after Polic
for (const path of [
'/api/v3/projects/project-1/runs/run-1/copilot/failure-diagnoses/diagnosis-request-1',
'/api/v3/projects/project-1/runs/run-1/copilot/failure-diagnoses/diagnosis-request-1/output',
'/api/v3/projects/project-1/runs/run-1/copilot/failure-diagnoses/diagnosis-request-1/cancellation',
]) {
await assert.rejects(
defaultStack.admission.prepare(metadata(path)),
+9
View File
@@ -1165,6 +1165,7 @@ function auditSourceImports(root, packagePath, findings) {
'src/application-runtime/aiProductionApplication.ts': Object.freeze([
'@qinglong/ai/durable-model-invocation',
'@qinglong/ai/failure-diagnosis-application',
'@qinglong/ai/failure-diagnosis-cancellation',
'@qinglong/ai/failure-diagnosis-model-execution',
'@qinglong/ai/plugin-package-prompt-output-projected-keyring',
'@qinglong/ai/postgres-model-provider-credential-storage',
@@ -1193,6 +1194,14 @@ function auditSourceImports(root, packagePath, findings) {
'@qinglong/ai/postgres-failure-diagnosis-admission-storage',
'@qinglong/ai/postgres-failure-diagnosis-model-execution-storage',
]),
'src/application-runtime/copilot/failureDiagnosisCancellationComposition.ts':
Object.freeze([
'@qinglong/ai/failure-diagnosis-cancellation',
'@qinglong/ai/failure-diagnosis-pre-model-terminalization',
'@qinglong/ai/postgres-failure-diagnosis-admission-storage',
]),
'src/copilot/failure-diagnosis/failureDiagnosisCancellationRoute.ts':
Object.freeze(['@qinglong/ai/failure-diagnosis-cancellation']),
});
const sourceRelative = path.relative(packageDirectory, filePath);
if (!clusterAiImports[sourceRelative]?.includes(specifier)) {
+222 -2
View File
@@ -88,6 +88,9 @@ const {
const {
CopilotFailureDiagnosisReadService,
} = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/read-model/service.js');
const {
CopilotFailureDiagnosisCancellationService,
} = require('../packages/ql3-ai/dist/copilot/failure-diagnosis/cancellation/service.js');
const {
BUILTIN_RUN_LOG_EXCERPT_TOOL,
BUILTIN_RUN_LOG_EXCERPT_TOOL_DEFINITION,
@@ -1807,6 +1810,34 @@ async function copilotFailureDiagnosisTerminalizationFacts(pool, requestId) {
};
}
async function copilotFailureDiagnosisCancellationFacts(pool, requestId) {
const terminalization = await copilotFailureDiagnosisTerminalizationFacts(
pool,
requestId,
);
const result = await pool.query(
`SELECT run.cancel_requested_at_ms AS "cancelRequestedAtMs",
run.cancel_reason AS "cancelReason",
(SELECT count(*)::integer FROM "ql3"."run_events"
WHERE run_id = admission.run_id
AND type = 'run.cancel_requested') AS "cancelEventCount",
(SELECT count(*)::integer
FROM "ql3_ai"."model_invocation_starts"
WHERE invocation_id = admission.plan_json->>'modelInvocationId')
AS "modelStartCount",
(SELECT count(*)::integer
FROM "ql3_ai"."model_invocation_completions"
WHERE invocation_id = admission.plan_json->>'modelInvocationId')
AS "modelCompletionCount"
FROM "ql3_ai"."copilot_failure_diagnosis_admissions" AS admission
JOIN "ql3"."runs" AS run ON run.id = admission.run_id
WHERE admission.request_id = $1`,
[requestId],
);
assert.equal(result.rowCount, 1);
return { ...terminalization, ...result.rows[0] };
}
async function copilotFailureDiagnosisModelResolutionFacts(pool, requestId) {
const result = await pool.query(
`SELECT admission.request_id AS "requestId",
@@ -2646,6 +2677,134 @@ async function terminalizeCopilotFailureDiagnosisLogUnavailableHaEvidence(
}
}
async function cancelCopilotFailureDiagnosisHaEvidence(options) {
const { port, report, fixture, expectedExisting = false } = options;
const runtimeDatabase = await databaseOpener(
'runtime',
databaseUrl(RUNTIME_USER, RUNTIME_PASSWORD, port),
expectedExisting
? 'ql3-ha-copilot-cancellation-promoted'
: 'ql3-ha-copilot-cancellation-primary',
)();
let promotedClient;
try {
const pool = expectedExisting
? await (async () => {
promotedClient = await runtimeDatabase.pool.connect();
await promotedClient.query(`SET synchronous_commit = 'local'`);
return {
query: promotedClient.query.bind(promotedClient),
async connect() {
return {
query: promotedClient.query.bind(promotedClient),
release() {},
};
},
};
})()
: runtimeDatabase.pool;
const admissions = new PostgresCopilotFailureDiagnosisAdmissionRepository(
pool,
);
const terminalizations =
new PostgresCopilotFailureDiagnosisPreModelTerminalizationRepository(
pool,
);
const service = new CopilotFailureDiagnosisCancellationService({
admissions,
cancellations: new PostgresClusterRunCancellationRepository(pool),
terminalizations: { repository: terminalizations },
terminalizeBeforeModel: terminalizeCopilotFailureDiagnosisBeforeModel,
});
const mutationId = report.cancellationMutationId ?? randomUUID();
const command = {
projectId: fixture.plan.projectId,
sourceRunId: fixture.plan.source.runId,
requestId: fixture.plan.requestId,
mutationId,
eventId: randomUUID(),
subject: fixture.principal.subject,
policyFence: fixture.plan.policyFence,
};
if (!expectedExisting) {
await assert.rejects(
service.cancel({ ...command, sourceRunId: 'other-source-run' }),
(error) =>
error?.code === 'COPILOT_FAILURE_DIAGNOSIS_CANCELLATION_NOT_FOUND',
);
}
const first = await service.cancel(command);
const replay = await service.cancel({ ...command, eventId: randomUUID() });
assert.equal(
first.status,
expectedExisting ? 'already_terminal' : 'accepted',
);
assert.equal(first.convergence, 'terminal');
assert.equal(first.runStatus, 'cancelled');
assert.equal(first.outcome, 'cancelled');
assert.equal(replay.status, 'already_terminal');
assert.equal(replay.convergence, 'terminal');
assert.equal(replay.runStatus, 'cancelled');
const facts = await copilotFailureDiagnosisCancellationFacts(
runtimeDatabase.pool,
report.requestId,
);
assert.deepEqual(
{
reason: facts.reason,
outcome: facts.outcome,
runStatus: facts.runStatus,
runVersion: facts.runVersion,
runEventSequence: facts.runEventSequence,
eventCount: facts.eventCount,
mutationCount: facts.mutationCount,
terminalizationCount: facts.terminalizationCount,
toolStatus: facts.toolStatus,
modelStatus: facts.modelStatus,
cancelReason: facts.cancelReason,
cancelEventCount: facts.cancelEventCount,
modelStartCount: facts.modelStartCount,
modelCompletionCount: facts.modelCompletionCount,
receiptContainsLogContent: facts.receiptContainsLogContent,
},
{
reason: 'cancellation_requested',
outcome: 'cancelled',
runStatus: 'cancelled',
runVersion: 7,
runEventSequence: 7,
eventCount: 7,
mutationCount: 4,
terminalizationCount: 1,
toolStatus: 'cancelled',
modelStatus: 'cancelled',
cancelReason: 'user',
cancelEventCount: 1,
modelStartCount: 0,
modelCompletionCount: 0,
receiptContainsLogContent: false,
},
);
if (expectedExisting) {
report.afterPromotion = facts;
report.survivedPromotion = true;
report.promotedExactReplay = true;
report.promotedProviderCalls = 0;
} else {
report.cancellationMutationId = mutationId;
report.beforePromotion = facts;
report.cancellationExactReplay = true;
report.crossTargetHidden = true;
report.providerCalls = 0;
report.contentFree = true;
}
return facts;
} finally {
promotedClient?.release();
await runtimeDatabase.close();
}
}
async function resolveCopilotFailureDiagnosisUnknownModelHaEvidence(options) {
const {
port,
@@ -11233,6 +11392,8 @@ async function main(argv = process.argv.slice(2)) {
let copilotFailureDiagnosisModelResolutionFixture;
let copilotFailureDiagnosisRead;
let copilotFailureDiagnosisReadFixture;
let copilotFailureDiagnosisCancellation;
let copilotFailureDiagnosisCancellationFixture;
let modelProviderCredentialCatalog;
let modelProviderCredentialTestConnection;
let runAttemptLogRetentionEvidence;
@@ -11449,6 +11610,13 @@ async function main(argv = process.argv.slice(2)) {
});
copilotFailureDiagnosisRead = copilotRead.report;
copilotFailureDiagnosisReadFixture = copilotRead.fixture;
const copilotCancellation =
await runCopilotFailureDiagnosisAdmissionHaEvidence({
primaryPort,
migrationPool: migrationDatabase.pool,
});
copilotFailureDiagnosisCancellation = copilotCancellation.report;
copilotFailureDiagnosisCancellationFixture = copilotCancellation.fixture;
} finally {
await migrationDatabase.close();
}
@@ -11903,6 +12071,26 @@ async function main(argv = process.argv.slice(2)) {
state: 'copilot_failure_diagnosis_pre_model_terminalized',
atMs: Number((performance.now() - startedAt).toFixed(3)),
});
await cancelCopilotFailureDiagnosisHaEvidence({
port: primaryPort,
report: copilotFailureDiagnosisCancellation,
fixture: copilotFailureDiagnosisCancellationFixture,
});
await waitFor(async () => {
const replicated = await copilotFailureDiagnosisCancellationFacts(
standbyDatabase.pool,
copilotFailureDiagnosisCancellation.requestId,
);
return JSON.stringify(replicated) ===
JSON.stringify(copilotFailureDiagnosisCancellation.beforePromotion)
? replicated
: null;
}, 'Copilot failure diagnosis cancellation WAL replay');
copilotFailureDiagnosisCancellation.replicatedBeforePromotion = true;
timeline.push({
state: 'copilot_failure_diagnosis_cancellation_terminalized',
atMs: Number((performance.now() - startedAt).toFixed(3)),
});
await resolveCopilotFailureDiagnosisUnknownModelHaEvidence({
port: primaryPort,
report: copilotFailureDiagnosisModelResolution,
@@ -12846,6 +13034,12 @@ async function main(argv = process.argv.slice(2)) {
outputKeyByte: 0x74,
expectedExisting: true,
});
await cancelCopilotFailureDiagnosisHaEvidence({
port: standbyPort,
report: copilotFailureDiagnosisCancellation,
fixture: copilotFailureDiagnosisCancellationFixture,
expectedExisting: true,
});
await verifyModelProviderCredentialCatalogAfterPromotion({
promotedPort: standbyPort,
promotedPool: promotedDatabase.pool,
@@ -13611,8 +13805,8 @@ async function main(argv = process.argv.slice(2)) {
);
assert.deepEqual(sideEffects.rows, [
{
runs: 84,
runEvents: 209,
runs: 86,
runEvents: 216,
diagnosisReadRuns: 2,
diagnosisReadRunEvents: 9,
credentialDeliveries: 4,
@@ -13734,6 +13928,7 @@ async function main(argv = process.argv.slice(2)) {
copilotFailureDiagnosisTerminalization,
copilotFailureDiagnosisModelResolution,
copilotFailureDiagnosisRead,
copilotFailureDiagnosisCancellation,
modelProviderCredentialCatalog,
modelProviderCredentialTestConnection,
runAttemptLogRetention,
@@ -13843,6 +14038,31 @@ async function main(argv = process.argv.slice(2)) {
copilotFailureDiagnosisRead.promotedProviderCalls === 0 &&
JSON.stringify(copilotFailureDiagnosisRead.afterPromotion) ===
JSON.stringify(copilotFailureDiagnosisRead.beforePromotion),
copilotFailureDiagnosisCancellationExactlyReplaysWithoutProviderCall:
copilotFailureDiagnosisCancellation.cancellationExactReplay &&
copilotFailureDiagnosisCancellation.crossTargetHidden &&
copilotFailureDiagnosisCancellation.providerCalls === 0 &&
copilotFailureDiagnosisCancellation.beforePromotion
.cancelEventCount === 1 &&
copilotFailureDiagnosisCancellation.beforePromotion
.terminalizationCount === 1 &&
copilotFailureDiagnosisCancellation.beforePromotion
.modelStartCount === 0 &&
copilotFailureDiagnosisCancellation.beforePromotion
.modelCompletionCount === 0 &&
copilotFailureDiagnosisCancellation.beforePromotion.runStatus ===
'cancelled',
copilotFailureDiagnosisCancellationIsContentFree:
copilotFailureDiagnosisCancellation.contentFree &&
copilotFailureDiagnosisCancellation.beforePromotion
.receiptContainsLogContent === false,
copilotFailureDiagnosisCancellationReplicatesAndSurvivesPromotion:
copilotFailureDiagnosisCancellation.replicatedBeforePromotion &&
copilotFailureDiagnosisCancellation.survivedPromotion &&
copilotFailureDiagnosisCancellation.promotedExactReplay &&
copilotFailureDiagnosisCancellation.promotedProviderCalls === 0 &&
JSON.stringify(copilotFailureDiagnosisCancellation.afterPromotion) ===
JSON.stringify(copilotFailureDiagnosisCancellation.beforePromotion),
modelProviderCredentialCatalogSurvivesPromotion:
modelProviderCredentialCatalog.survivedPromotion,
modelProviderCredentialManagementIdentityLedgerSurvivesPromotion:
+4 -4
View File
@@ -268,10 +268,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
{
path: 'packages/ql3-ai',
name: '@qinglong/ai',
sourceFiles: 193,
sourceFiles: 194,
rootSourceFiles: 1,
rootSourceLines: 16,
nestedSourceFiles: 192,
nestedSourceFiles: 193,
rootSourceFileHardCap: 1,
rootSourceLineHardCap: 16,
rootSourceFileRoles: {
@@ -385,10 +385,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
rootSourceFileRoles: clusterControl.rootSourceFileRoles,
},
{
sourceFiles: 62,
sourceFiles: 64,
rootSourceFiles: 2,
rootSourceLines: 195,
nestedSourceFiles: 60,
nestedSourceFiles: 62,
rootSourceFileRoles: {
'aiCli.ts': 'binary_entry',
'cli.ts': 'binary_entry',