mirror of
https://github.com/whyour/qinglong.git
synced 2026-09-20 16:07:11 +08:00
feat(ql3): add strong local run stop
This commit is contained in:
@@ -11,6 +11,17 @@
|
||||
|
||||
最新增量证据(2026-08-12):
|
||||
|
||||
- D-297/ADR-0385(已接受)
|
||||
Local Edge/Standalone 已补齐强认证 `run.stop` 产品入口,并与既有 `run.retry` 统一为同一个 caller-driven `ql3-run retry|stop`
|
||||
binary,不新增 package、migration、表、索引、进程、listener、timer、watcher、连接、cache 或 sidecar。stop 只接受 POSIX 私有命令文件
|
||||
中的 Project/Run、UUID mutation、request/audit identity 与既有 deployment authority 路径;Event ID、`local_console` User、Policy
|
||||
fence、取消原因和数据库时间均由进程/服务端建立。SQLite audited cancellation 在一个 `BEGIN IMMEDIATE` 中重验 Owner
|
||||
credential/pepper/material、Project/RoleBinding,原子提交 cancellation intent、immutable Event 与 `run.stop` allowed audit;相同命令
|
||||
精确重放不重复 Event/audit,实际终态继续由既有 cancellation lifecycle 收敛。常驻 runtime 保留原轻量 repository 方法,只有短生命周期
|
||||
CLI 加载 Owner/Policy/audit authority;单因子 Local HTTP、MCP 与 AI Tool 不因此获得强 stop authority。完整 18-package clean
|
||||
build/test 与 backend 1,166 pass/2 conditional skip/0 fail;workspace 保持 18 package/1,072 source/1,054 nested,无 single-source 或
|
||||
shallow package;14 档 Local Profile artifact 全部 compatible,最小 Edge 为 2,467,343 bytes/295 files/53 loaded modules,RSS delta
|
||||
11,141,120 bytes,低于 4 MiB/512 files/16 MiB 门限。完整验收记录见 ADR-0385。
|
||||
- D-296/ADR-0384(已接受)
|
||||
Cluster Run Management Plane 已在既有内聚领域内增加强认证 `run.stop`,不新增 package、进程、listener、Pool、timer、watcher、cache、
|
||||
sidecar 或部署 overlay。`run.retry | run.stop` 共用 mTLS、purpose-bound OIDC、五分钟内 `multi_factor|hardware` User、固定
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
# ADR-0385:强认证的 Local Run Stop 产品入口
|
||||
|
||||
- 状态:Accepted
|
||||
- 日期:2026-08-12
|
||||
- 关联 RFC:QL-RFC-0001 D-297
|
||||
- 前置决策:ADR-0005、ADR-0072、ADR-0086、ADR-0365、ADR-0372、ADR-0381、ADR-0384
|
||||
|
||||
## 上下文
|
||||
|
||||
Local Edge/Standalone 已有 cancellation repository、执行期 convergence 和单进程 HTTP `run.stop`,但 HTTP bearer 只能建立 `single_factor` User。ADR-0381 为人工 `run.retry` 建立了 Owner credential、pepper provenance、POSIX 私有命令文件和五分钟 `local_console` 强认证入口;`run.stop` 尚未进入同一产品面,导致 Local 与 Cluster 的强管理命令不对称。
|
||||
|
||||
QingLong 的 Local 部署可能运行在内存和 CPU 很小的路由设备上。补齐 stop 不能新增 package、数据库 migration、表、索引、常驻进程、listener、timer、watcher、连接、cache 或 sidecar,也不能让未调用管理命令的 runtime 多加载 Owner 管理依赖。
|
||||
|
||||
## 决策
|
||||
|
||||
### 1. 统一既有 `ql3-run`,不新增 package 或 binary
|
||||
|
||||
`@qinglong/local-owner-cli` 的既有 `ql3-run` binary 改为严格判别的 `retry | stop` 命令面:
|
||||
|
||||
```text
|
||||
ql3-run retry --command-file /absolute/private-command.json
|
||||
ql3-run stop --command-file /absolute/private-command.json
|
||||
```
|
||||
|
||||
`run.stop` 只接受私有 regular command file 中的 Project/Run identity、UUID mutation identity、request/audit identity、请求时间和既有 Local deployment authority 路径;Event ID、Principal、Policy fence、取消原因与数据库时间均由进程或服务端生成。产品根命令 `ql3 run` 仍无 shell 拼接地委派同一个 binary。
|
||||
|
||||
### 2. 强认证与事务内 credential fence
|
||||
|
||||
stop 复用 Owner credential、versioned pepper keyring 与 POSIX proof,建立不超过五分钟的 `local_console` User,并执行 `run.stop` Policy。命令进入 SQLite 事务前再次确认 presentation;事务内再次验证 active credential/pepper/material fence、Project version 和最新 RoleBinding。Owner、Admin、Operator 允许,Viewer 与其他主体失败关闭。
|
||||
|
||||
现有 Local HTTP route 继续保持原兼容能力,但其 `single_factor` bearer 不得被描述为本 ADR 的强产品入口;MCP 与 AI Tool 不获得 stop authority。
|
||||
|
||||
### 3. cancellation intent、Event 与 allowed audit 原子提交
|
||||
|
||||
`LocalSqliteRunCancellationRepository` 保留常驻 runtime 使用的轻量 `requestUserCancellation`,并新增只供短生命周期 Run Management 数据库暴露的 audited 方法。audited 方法在一个 `BEGIN IMMEDIATE` 事务中使用 SQLite 数据库时钟,完成:
|
||||
|
||||
1. 五分钟强认证和 credential fence;
|
||||
2. Project/RoleBinding Policy fence;
|
||||
3. Run cancellation intent、version/event sequence CAS;
|
||||
4. immutable `run.cancel_requested` Event;
|
||||
5. `run.stop` allowed security audit。
|
||||
|
||||
任一写入失败整体回滚。相同 command file 重放返回 `already_requested`,复验同一 allowed audit 且不产生重复 Event/audit。不存在、撤权、credential drift 与存储失败由 CLI 写一个独立 failure audit。实际进程 signal、Attempt/Run 终态和崩溃恢复仍由 ADR-0072 的既有 cancellation lifecycle 收敛。
|
||||
|
||||
### 4. 低配设备按调用付费
|
||||
|
||||
Run Management 数据库仍是 caller-driven、短生命周期、单 SQLite authority;只有执行 `ql3-run` 时才加载 Owner console、pepper、Policy 和 audit adapter。Edge/Standalone application 启动路径、timer 数、连接数、默认 artifact 与运行制品 package 集合保持不变。领域代码继续位于既有 `run-management/` 与 SQLite `run/` 目录,不创建单文件或浅 package。
|
||||
|
||||
## 验收
|
||||
|
||||
- repository 测试证明 intent/Event/allowed audit 原子性、exact replay、credential fence 回滚和原 runtime API 兼容;
|
||||
- 真实 SQLite + Owner credential + 私有命令文件测试证明 `ql3-run stop` accepted/replay、Viewer 拒绝、missing 遮蔽与低敏输出;
|
||||
- `ql3-run retry` 兼容回归、产品 CLI 无 shell 委派、package boundary 与内部 layout ratchet 全部通过;
|
||||
- 完整 Local SQLite/Owner CLI package、backend、18-package、dependency/Edge import、Local image 和 14 档 Profile artifact 门全部通过后才允许阶段性提交。
|
||||
|
||||
## 被否决的替代方案
|
||||
|
||||
1. **开放 Local HTTP 强 stop**:现有 bearer 只有单因子,增加独立 MFA HTTP adapter 又会扩大常驻攻击面与路由设备资源。
|
||||
2. **新增 `ql3-run-stop` binary/package**:与 retry 共享认证、存储和生命周期,拆分只会增加制品与浅边界。
|
||||
3. **CLI 直接更新 Runs**:会绕过 Policy、credential fence、Event 和 cancellation convergence。
|
||||
4. **allowed audit 在事务外写入**:可能留下意图成功但审计缺失,或审计成功但意图回滚。
|
||||
5. **调用方提供 Event ID、Principal 或 cancel reason**:扩大重放碰撞与权限注入表面。
|
||||
|
||||
## 影响
|
||||
|
||||
- `ql3-run retry` 的参数与结果保持兼容,help 扩展为 `retry | stop`;内部 binary 文件名改为领域中性的 `runManagementCli`;
|
||||
- Local SQLite schema contract 不变,不需要 migration;
|
||||
- `@qinglong/local-owner-cli` 增加一个公开 `run-stop-command` subpath,但 workspace package 数保持不变;
|
||||
- Cluster Run Management Kubernetes 多节点组合证据、UI 与固定物理路由设备门继续由后续阶段完成。
|
||||
|
||||
## 验收记录(2026-08-12)
|
||||
|
||||
- Local SQLite package:227/227;Local Owner CLI:163 pass、5 个条件 skip、0 fail;
|
||||
- backend:1,166 pass、2 个平台条件 skip、0 fail;其中依赖边界测试 53/53,包含 exact `runRetryCommand`/`runStopCommand` authority allowlist 及 widened sibling 反向拒绝;
|
||||
- 18-package clean build/test 通过;workspace 仍为 18 package、1,072 source、1,054 nested source,`singleSourcePackages=[]`、`shallowSourcePackages=[]`;
|
||||
- dependency audit、Edge import audit 与 Local application image audit 均 `compatible=true`;Edge import 闭包仍为 121 module 且无 Cluster/PostgreSQL 越界;
|
||||
- 14 档 Local Profile artifact 全部 `compatible=true`。最小 Edge 为 2,467,343 bytes/295 files/53 loaded modules,RSS delta 11,141,120 bytes,分别低于 4 MiB/512 files/16 MiB 门限;完整 application、application+AI、MCP 档位也均在各自固定预算内;
|
||||
- 本增量没有修改 lockfile、依赖版本、SQLite schema 或 migration,也没有增加 package、常驻进程、listener、timer、watcher、连接、cache 或 sidecar。
|
||||
@@ -388,6 +388,7 @@
|
||||
| [ADR-0382](./ADR-0382-postgresql-cluster-manual-run-retry-authority.md) | PostgreSQL Cluster 手动 Run retry 原子 Authority | Accepted |
|
||||
| [ADR-0383](./ADR-0383-strong-cluster-run-management-plane.md) | 强认证的 Cluster Run Management Plane 与专用数据库角色 | Accepted |
|
||||
| [ADR-0384](./ADR-0384-strong-cluster-run-stop-management.md) | 强认证的 Cluster Run Stop Management 与列级数据库权限 | Accepted |
|
||||
| [ADR-0385](./ADR-0385-strong-local-run-stop-product-entry.md) | 强认证的 Local Run Stop 产品入口与原子审计 | Accepted |
|
||||
|
||||
## 规则
|
||||
|
||||
|
||||
@@ -65,6 +65,11 @@
|
||||
"require": "./dist/run-management/runRetryCommand.js",
|
||||
"default": "./dist/run-management/runRetryCommand.js"
|
||||
},
|
||||
"./run-stop-command": {
|
||||
"types": "./dist/run-management/runStopCommand.d.ts",
|
||||
"require": "./dist/run-management/runStopCommand.js",
|
||||
"default": "./dist/run-management/runStopCommand.js"
|
||||
},
|
||||
"./plugin-package-workflow-command": {
|
||||
"types": "./dist/plugin-package/pluginPackageWorkflowCommand.d.ts",
|
||||
"require": "./dist/plugin-package/pluginPackageWorkflowCommand.js",
|
||||
@@ -130,7 +135,7 @@
|
||||
"ql3-ai-feature": "dist/ai-management/aiFeatureCli.js",
|
||||
"ql3-secret": "dist/security-management/secretCli.js",
|
||||
"ql3-task": "dist/automation-management/taskDefinitionCli.js",
|
||||
"ql3-run": "dist/run-management/runRetryCli.js",
|
||||
"ql3-run": "dist/run-management/runManagementCli.js",
|
||||
"ql3-workflow": "dist/plugin-package/pluginPackageWorkflowCli.js",
|
||||
"ql3-trigger": "dist/automation-management/triggerCli.js",
|
||||
"ql3-policy": "dist/security-management/projectPolicyCli.js",
|
||||
|
||||
@@ -87,8 +87,8 @@ export const QINGLONG3_PRODUCT_COMMANDS: readonly QingLong3ProductCommandDefinit
|
||||
Object.freeze({
|
||||
name: 'run',
|
||||
binary: 'ql3-run',
|
||||
target: 'run-management/runRetryCli.js',
|
||||
description: 'retry terminal Runs under strong local authentication',
|
||||
target: 'run-management/runManagementCli.js',
|
||||
description: 'retry or stop Runs under strong local authentication',
|
||||
}),
|
||||
Object.freeze({
|
||||
name: 'trigger',
|
||||
|
||||
+19
-9
@@ -1,25 +1,26 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { runLocalRunRetryCommandFile } from './runRetryCommand';
|
||||
|
||||
const USAGE =
|
||||
'Usage: ql3-run retry --command-file /absolute/private-command.json';
|
||||
const USAGE = [
|
||||
'Usage: ql3-run retry --command-file /absolute/private-command.json',
|
||||
' ql3-run stop --command-file /absolute/private-command.json',
|
||||
].join('\n');
|
||||
|
||||
async function main(argv: readonly string[]): Promise<void> {
|
||||
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
|
||||
process.stdout.write(`${USAGE}\n`);
|
||||
return;
|
||||
}
|
||||
const operation = argv[0];
|
||||
const commandFilePath = argv[2];
|
||||
if (
|
||||
argv.length !== 3 ||
|
||||
argv[0] !== 'retry' ||
|
||||
(operation !== 'retry' && operation !== 'stop') ||
|
||||
argv[1] !== '--command-file' ||
|
||||
commandFilePath === undefined
|
||||
) {
|
||||
process.stderr.write(
|
||||
`${JSON.stringify({
|
||||
code: 'LOCAL_RUN_RETRY_CLI_USAGE_INVALID',
|
||||
code: 'LOCAL_RUN_MANAGEMENT_CLI_USAGE_INVALID',
|
||||
message: USAGE,
|
||||
})}\n`,
|
||||
);
|
||||
@@ -27,7 +28,16 @@ async function main(argv: readonly string[]): Promise<void> {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const result = await runLocalRunRetryCommandFile(commandFilePath);
|
||||
const result =
|
||||
operation === 'retry'
|
||||
? await import('./runRetryCommand.js').then(
|
||||
({ runLocalRunRetryCommandFile }) =>
|
||||
runLocalRunRetryCommandFile(commandFilePath),
|
||||
)
|
||||
: await import('./runStopCommand.js').then(
|
||||
({ runLocalRunStopCommandFile }) =>
|
||||
runLocalRunStopCommandFile(commandFilePath),
|
||||
);
|
||||
process.stdout.write(`${JSON.stringify(result)}\n`);
|
||||
} catch (error) {
|
||||
const candidate = error as {
|
||||
@@ -41,12 +51,12 @@ async function main(argv: readonly string[]): Promise<void> {
|
||||
code:
|
||||
typeof candidate.code === 'string'
|
||||
? candidate.code
|
||||
: 'LOCAL_RUN_RETRY_CLI_FAILED',
|
||||
: 'LOCAL_RUN_MANAGEMENT_CLI_FAILED',
|
||||
name: typeof candidate.name === 'string' ? candidate.name : 'Error',
|
||||
message:
|
||||
typeof candidate.message === 'string'
|
||||
? candidate.message
|
||||
: 'Local Run retry command failed',
|
||||
: 'Local Run management command failed',
|
||||
...(Number.isSafeInteger(candidate.retryAfterMs)
|
||||
? { retryAfterMs: candidate.retryAfterMs }
|
||||
: {}),
|
||||
@@ -0,0 +1,482 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import path from 'node:path';
|
||||
|
||||
import {
|
||||
PrivateLocalCommandFileError,
|
||||
readPrivateLocalCommandFile,
|
||||
} from '@qinglong/local-command-file';
|
||||
import {
|
||||
AuthenticatedLocalCommandAuthenticationError,
|
||||
establishAuthenticatedLocalCommand,
|
||||
type AuthenticatedLocalCommand,
|
||||
} from '@qinglong/local-owner-console/authenticated-command';
|
||||
import {
|
||||
LocalSqliteAuthenticatedManagementFenceError,
|
||||
type LocalSqliteAuthenticatedUserCredentialFence,
|
||||
} from '@qinglong/local-sqlite/authenticated-management';
|
||||
import {
|
||||
openLocalSqliteRunManagementDatabase,
|
||||
type LocalSqliteRunManagementDatabase,
|
||||
} from '@qinglong/local-sqlite/run-management';
|
||||
import { ProjectPolicyEngine } from '@qinglong/runtime-core/project-policy';
|
||||
import {
|
||||
RUN_CANCELLATION_SCHEMA,
|
||||
RunCancellationFenceRejectedError,
|
||||
RunCancellationNotFoundError,
|
||||
RunCancellationUnavailableError,
|
||||
parseRunCancellationRequestBody,
|
||||
type RunCancellationResult,
|
||||
} from '@qinglong/runtime-core/run-cancellation';
|
||||
import {
|
||||
normalizeSecurityAuditRecord,
|
||||
type SecurityAuditRecord,
|
||||
} from '@qinglong/runtime-core/security-audit';
|
||||
|
||||
const MAX_PATH_BYTES = 4_096;
|
||||
const UUID_V4_PATTERN =
|
||||
/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/;
|
||||
const IDENTIFIER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
|
||||
|
||||
export interface LocalRunStopCommandOptions {
|
||||
readonly deploymentRoot: string;
|
||||
readonly databasePath: string;
|
||||
readonly profile: 'edge' | 'standalone';
|
||||
readonly ownerPepperKeyringDirectory: string;
|
||||
readonly credentialFilePath: string;
|
||||
readonly busyTimeoutMs?: number;
|
||||
}
|
||||
|
||||
export interface LocalRunStopCommand {
|
||||
readonly schemaVersion: 1;
|
||||
readonly operation: 'run.stop';
|
||||
readonly options: LocalRunStopCommandOptions;
|
||||
readonly request: Readonly<{
|
||||
projectId: string;
|
||||
runId: string;
|
||||
mutationId: string;
|
||||
requestId: string;
|
||||
auditEventId: string;
|
||||
failureAuditEventId: string;
|
||||
occurredAtMs: number;
|
||||
}>;
|
||||
}
|
||||
|
||||
export interface LocalRunStopCommandResult {
|
||||
readonly schemaVersion: 1;
|
||||
readonly operation: 'run.stop';
|
||||
readonly stop: Readonly<RunCancellationResult>;
|
||||
}
|
||||
|
||||
export interface LocalRunStopCommandRunner {
|
||||
run(commandFilePath: string): Promise<Readonly<LocalRunStopCommandResult>>;
|
||||
}
|
||||
|
||||
export interface LocalRunStopCommandRunnerDependencies {
|
||||
readonly openDatabase: typeof openLocalSqliteRunManagementDatabase;
|
||||
readonly authenticate: typeof establishAuthenticatedLocalCommand;
|
||||
readonly now: () => number;
|
||||
readonly randomUuid: () => string;
|
||||
}
|
||||
|
||||
export class LocalRunStopCommandConfigurationError extends TypeError {
|
||||
readonly code = 'LOCAL_RUN_STOP_COMMAND_CONFIGURATION_INVALID';
|
||||
|
||||
constructor(message: string, options?: ErrorOptions) {
|
||||
super(
|
||||
`Local Run stop command configuration is invalid: ${message}`,
|
||||
options,
|
||||
);
|
||||
this.name = 'LocalRunStopCommandConfigurationError';
|
||||
}
|
||||
}
|
||||
|
||||
export class LocalRunStopCommandAuthorizationError extends Error {
|
||||
readonly code = 'LOCAL_RUN_STOP_COMMAND_AUTHORIZATION_REJECTED';
|
||||
|
||||
constructor() {
|
||||
super('Local Run stop command authorization was rejected');
|
||||
this.name = 'LocalRunStopCommandAuthorizationError';
|
||||
}
|
||||
}
|
||||
|
||||
export class LocalRunStopCommandUnavailableError extends Error {
|
||||
readonly code = 'LOCAL_RUN_STOP_COMMAND_UNAVAILABLE';
|
||||
|
||||
constructor(options?: ErrorOptions) {
|
||||
super('Local Run stop command is unavailable', options);
|
||||
this.name = 'LocalRunStopCommandUnavailableError';
|
||||
}
|
||||
}
|
||||
|
||||
function exactRecord(
|
||||
value: unknown,
|
||||
required: readonly string[],
|
||||
optional: readonly string[],
|
||||
label: string,
|
||||
): asserts value is Record<string, unknown> {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value)) {
|
||||
throw new LocalRunStopCommandConfigurationError(
|
||||
`${label} must be an object`,
|
||||
);
|
||||
}
|
||||
const keys = Object.keys(value).sort();
|
||||
const allowed = new Set([...required, ...optional]);
|
||||
if (
|
||||
required.some((key) => !keys.includes(key)) ||
|
||||
keys.some((key) => !allowed.has(key))
|
||||
) {
|
||||
throw new LocalRunStopCommandConfigurationError(
|
||||
`${label} shape is invalid`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function boundedPath(value: unknown, label: string): string {
|
||||
if (
|
||||
typeof value !== 'string' ||
|
||||
value.length < 1 ||
|
||||
Buffer.byteLength(value, 'utf8') > MAX_PATH_BYTES ||
|
||||
value.includes('\0') ||
|
||||
!path.isAbsolute(value) ||
|
||||
path.normalize(value) !== value ||
|
||||
path.parse(value).root === value
|
||||
) {
|
||||
throw new LocalRunStopCommandConfigurationError(
|
||||
`${label} must be a normalized bounded absolute non-root path`,
|
||||
);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function descendant(root: string, value: string, label: string): void {
|
||||
const relative = path.relative(root, value);
|
||||
if (
|
||||
relative.length === 0 ||
|
||||
relative === '..' ||
|
||||
relative.startsWith(`..${path.sep}`) ||
|
||||
path.isAbsolute(relative)
|
||||
) {
|
||||
throw new LocalRunStopCommandConfigurationError(
|
||||
`${label} must be a descendant of deploymentRoot`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function identifier(value: unknown, label: string): string {
|
||||
if (typeof value !== 'string' || !IDENTIFIER_PATTERN.test(value)) {
|
||||
throw new LocalRunStopCommandConfigurationError(`${label} is invalid`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function uuid(value: unknown, label: string): string {
|
||||
if (typeof value !== 'string' || !UUID_V4_PATTERN.test(value)) {
|
||||
throw new LocalRunStopCommandConfigurationError(`${label} is invalid`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function options(value: unknown): Readonly<LocalRunStopCommandOptions> {
|
||||
exactRecord(
|
||||
value,
|
||||
[
|
||||
'deploymentRoot',
|
||||
'databasePath',
|
||||
'profile',
|
||||
'ownerPepperKeyringDirectory',
|
||||
'credentialFilePath',
|
||||
],
|
||||
['busyTimeoutMs'],
|
||||
'options',
|
||||
);
|
||||
const deploymentRoot = boundedPath(value.deploymentRoot, 'deploymentRoot');
|
||||
const databasePath = boundedPath(value.databasePath, 'databasePath');
|
||||
const ownerPepperKeyringDirectory = boundedPath(
|
||||
value.ownerPepperKeyringDirectory,
|
||||
'ownerPepperKeyringDirectory',
|
||||
);
|
||||
const credentialFilePath = boundedPath(
|
||||
value.credentialFilePath,
|
||||
'credentialFilePath',
|
||||
);
|
||||
for (const [target, label] of [
|
||||
[databasePath, 'databasePath'],
|
||||
[ownerPepperKeyringDirectory, 'ownerPepperKeyringDirectory'],
|
||||
[credentialFilePath, 'credentialFilePath'],
|
||||
] as const) {
|
||||
descendant(deploymentRoot, target, label);
|
||||
}
|
||||
if (
|
||||
new Set([databasePath, ownerPepperKeyringDirectory, credentialFilePath])
|
||||
.size !== 3 ||
|
||||
(value.profile !== 'edge' && value.profile !== 'standalone') ||
|
||||
(value.busyTimeoutMs !== undefined &&
|
||||
(!Number.isSafeInteger(value.busyTimeoutMs) ||
|
||||
(value.busyTimeoutMs as number) < 1 ||
|
||||
(value.busyTimeoutMs as number) > 60_000))
|
||||
) {
|
||||
throw new LocalRunStopCommandConfigurationError('options are invalid');
|
||||
}
|
||||
return Object.freeze({
|
||||
deploymentRoot,
|
||||
databasePath,
|
||||
profile: value.profile,
|
||||
ownerPepperKeyringDirectory,
|
||||
credentialFilePath,
|
||||
...(value.busyTimeoutMs === undefined
|
||||
? {}
|
||||
: { busyTimeoutMs: value.busyTimeoutMs as number }),
|
||||
});
|
||||
}
|
||||
|
||||
function normalizeCommand(value: unknown): Readonly<LocalRunStopCommand> {
|
||||
exactRecord(
|
||||
value,
|
||||
['schemaVersion', 'operation', 'options', 'request'],
|
||||
[],
|
||||
'command',
|
||||
);
|
||||
if (value.schemaVersion !== 1 || value.operation !== 'run.stop') {
|
||||
throw new LocalRunStopCommandConfigurationError(
|
||||
'schemaVersion or operation is invalid',
|
||||
);
|
||||
}
|
||||
exactRecord(
|
||||
value.request,
|
||||
[
|
||||
'projectId',
|
||||
'runId',
|
||||
'mutationId',
|
||||
'requestId',
|
||||
'auditEventId',
|
||||
'failureAuditEventId',
|
||||
'occurredAtMs',
|
||||
],
|
||||
[],
|
||||
'request',
|
||||
);
|
||||
const mutationId = uuid(value.request.mutationId, 'mutationId');
|
||||
try {
|
||||
parseRunCancellationRequestBody({
|
||||
schema: RUN_CANCELLATION_SCHEMA,
|
||||
mutationId,
|
||||
});
|
||||
} catch (error) {
|
||||
throw new LocalRunStopCommandConfigurationError('stop request is invalid', {
|
||||
cause: error,
|
||||
});
|
||||
}
|
||||
const auditEventId = uuid(value.request.auditEventId, 'auditEventId');
|
||||
const failureAuditEventId = uuid(
|
||||
value.request.failureAuditEventId,
|
||||
'failureAuditEventId',
|
||||
);
|
||||
if (
|
||||
auditEventId === failureAuditEventId ||
|
||||
!Number.isSafeInteger(value.request.occurredAtMs) ||
|
||||
(value.request.occurredAtMs as number) < 0
|
||||
) {
|
||||
throw new LocalRunStopCommandConfigurationError('request is invalid');
|
||||
}
|
||||
return Object.freeze({
|
||||
schemaVersion: 1,
|
||||
operation: 'run.stop',
|
||||
options: options(value.options),
|
||||
request: Object.freeze({
|
||||
projectId: identifier(value.request.projectId, 'projectId'),
|
||||
runId: identifier(value.request.runId, 'runId'),
|
||||
mutationId,
|
||||
requestId: identifier(value.request.requestId, 'requestId'),
|
||||
auditEventId,
|
||||
failureAuditEventId,
|
||||
occurredAtMs: value.request.occurredAtMs as number,
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
function readCommandFile(
|
||||
commandFilePath: string,
|
||||
): Readonly<LocalRunStopCommand> {
|
||||
try {
|
||||
return normalizeCommand(readPrivateLocalCommandFile(commandFilePath));
|
||||
} catch (error) {
|
||||
if (error instanceof LocalRunStopCommandConfigurationError) throw error;
|
||||
if (error instanceof PrivateLocalCommandFileError) {
|
||||
throw new LocalRunStopCommandConfigurationError(
|
||||
'private command file cannot be read',
|
||||
{ cause: error },
|
||||
);
|
||||
}
|
||||
throw new LocalRunStopCommandConfigurationError('command file is invalid', {
|
||||
cause: error,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
function dependencies(
|
||||
value: LocalRunStopCommandRunnerDependencies,
|
||||
): LocalRunStopCommandRunnerDependencies {
|
||||
if (
|
||||
!value ||
|
||||
typeof value !== 'object' ||
|
||||
Array.isArray(value) ||
|
||||
Object.keys(value).sort().join('\0') !==
|
||||
['authenticate', 'now', 'openDatabase', 'randomUuid'].sort().join('\0') ||
|
||||
typeof value.openDatabase !== 'function' ||
|
||||
typeof value.authenticate !== 'function' ||
|
||||
typeof value.now !== 'function' ||
|
||||
typeof value.randomUuid !== 'function'
|
||||
) {
|
||||
throw new LocalRunStopCommandConfigurationError(
|
||||
'runner dependencies are invalid',
|
||||
);
|
||||
}
|
||||
return Object.freeze({ ...value });
|
||||
}
|
||||
|
||||
function failureReason(error: unknown): string {
|
||||
if (error instanceof RunCancellationNotFoundError) return 'run_not_found';
|
||||
if (
|
||||
error instanceof LocalRunStopCommandAuthorizationError ||
|
||||
error instanceof RunCancellationFenceRejectedError ||
|
||||
error instanceof LocalSqliteAuthenticatedManagementFenceError
|
||||
) {
|
||||
return 'run_stop_fence_rejected';
|
||||
}
|
||||
return 'run_stop_unavailable';
|
||||
}
|
||||
|
||||
function failureAudit(
|
||||
command: Readonly<LocalRunStopCommand>,
|
||||
authenticated: Readonly<AuthenticatedLocalCommand> | undefined,
|
||||
error: unknown,
|
||||
): Readonly<SecurityAuditRecord> {
|
||||
const unauthenticated = authenticated === undefined;
|
||||
return normalizeSecurityAuditRecord({
|
||||
eventId: command.request.failureAuditEventId,
|
||||
requestId: command.request.requestId,
|
||||
operationId: 'run.stop',
|
||||
projectId: command.request.projectId,
|
||||
subject: authenticated?.principal.subject ?? null,
|
||||
authenticationId: authenticated?.principal.authenticationId ?? null,
|
||||
outcome: unauthenticated ? 'authentication_rejected' : 'denied',
|
||||
reasons: [
|
||||
unauthenticated ? 'local_console_required' : failureReason(error),
|
||||
],
|
||||
fence: null,
|
||||
occurredAtMs: command.request.occurredAtMs,
|
||||
});
|
||||
}
|
||||
|
||||
export function createLocalRunStopCommandRunner(
|
||||
candidateDependencies: LocalRunStopCommandRunnerDependencies = {
|
||||
openDatabase: openLocalSqliteRunManagementDatabase,
|
||||
authenticate: establishAuthenticatedLocalCommand,
|
||||
now: Date.now,
|
||||
randomUuid: randomUUID,
|
||||
},
|
||||
): Readonly<LocalRunStopCommandRunner> {
|
||||
const adapters = dependencies(candidateDependencies);
|
||||
return Object.freeze({
|
||||
async run(commandFilePath: string) {
|
||||
const command = readCommandFile(commandFilePath);
|
||||
const nowMs = adapters.now();
|
||||
if (
|
||||
!Number.isSafeInteger(nowMs) ||
|
||||
nowMs < command.request.occurredAtMs ||
|
||||
nowMs - command.request.occurredAtMs > 5 * 60_000
|
||||
) {
|
||||
throw new LocalRunStopCommandConfigurationError(
|
||||
'request time is outside the accepted window',
|
||||
);
|
||||
}
|
||||
const database: LocalSqliteRunManagementDatabase =
|
||||
await adapters.openDatabase({
|
||||
databasePath: command.options.databasePath,
|
||||
profile: command.options.profile,
|
||||
...(command.options.busyTimeoutMs === undefined
|
||||
? {}
|
||||
: { busyTimeoutMs: command.options.busyTimeoutMs }),
|
||||
});
|
||||
let authenticated: Readonly<AuthenticatedLocalCommand> | undefined;
|
||||
try {
|
||||
try {
|
||||
authenticated = await adapters.authenticate(database, {
|
||||
deploymentRoot: command.options.deploymentRoot,
|
||||
databasePath: command.options.databasePath,
|
||||
ownerPepperKeyringDirectory:
|
||||
command.options.ownerPepperKeyringDirectory,
|
||||
credentialFilePath: command.options.credentialFilePath,
|
||||
authenticationNamespace: 'local_run_stop',
|
||||
});
|
||||
database.activateUserCredentialFence(
|
||||
authenticated.databaseFence as Readonly<LocalSqliteAuthenticatedUserCredentialFence>,
|
||||
);
|
||||
const decision = await new ProjectPolicyEngine(
|
||||
database.projectPolicy,
|
||||
).authorize(
|
||||
authenticated.principal,
|
||||
command.request.projectId,
|
||||
'run.stop',
|
||||
);
|
||||
if (
|
||||
decision.effect !== 'allow' ||
|
||||
!decision.fence ||
|
||||
decision.fence.bindingVersion === null
|
||||
) {
|
||||
throw new LocalRunStopCommandAuthorizationError();
|
||||
}
|
||||
await authenticated.confirm();
|
||||
const stop =
|
||||
await database.runCancellation.requestUserCancellationAudited({
|
||||
projectId: command.request.projectId,
|
||||
runId: command.request.runId,
|
||||
mutationId: command.request.mutationId,
|
||||
eventId: adapters.randomUuid(),
|
||||
requestId: command.request.requestId,
|
||||
auditEventId: command.request.auditEventId,
|
||||
principal: authenticated.principal,
|
||||
policyFence: decision.fence,
|
||||
});
|
||||
return Object.freeze({
|
||||
schemaVersion: 1 as const,
|
||||
operation: 'run.stop' as const,
|
||||
stop,
|
||||
});
|
||||
} catch (error) {
|
||||
try {
|
||||
await database.securityAudit.record(
|
||||
failureAudit(command, authenticated, error),
|
||||
);
|
||||
} catch (auditError) {
|
||||
throw new LocalRunStopCommandUnavailableError({
|
||||
cause: auditError,
|
||||
});
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
} finally {
|
||||
await database.close();
|
||||
}
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
export function runLocalRunStopCommandFile(
|
||||
commandFilePath: string,
|
||||
): Promise<Readonly<LocalRunStopCommandResult>> {
|
||||
return createLocalRunStopCommandRunner().run(commandFilePath);
|
||||
}
|
||||
|
||||
export function isLocalRunStopCommandError(error: unknown): boolean {
|
||||
return (
|
||||
error instanceof LocalRunStopCommandConfigurationError ||
|
||||
error instanceof LocalRunStopCommandAuthorizationError ||
|
||||
error instanceof LocalRunStopCommandUnavailableError ||
|
||||
error instanceof AuthenticatedLocalCommandAuthenticationError ||
|
||||
error instanceof LocalSqliteAuthenticatedManagementFenceError ||
|
||||
error instanceof RunCancellationNotFoundError ||
|
||||
error instanceof RunCancellationFenceRejectedError ||
|
||||
error instanceof RunCancellationUnavailableError
|
||||
);
|
||||
}
|
||||
@@ -60,7 +60,10 @@ test('help and version are bounded installation-derived product facts', () => {
|
||||
const help = qingLong3ProductHelp();
|
||||
assert.match(help, /^Usage: ql3 <command> \[arguments\]/);
|
||||
assert.match(help, /\n task\s+manage Task definitions\n/);
|
||||
assert.match(help, /\n run\s+retry terminal Runs/);
|
||||
assert.match(
|
||||
help,
|
||||
/\n run\s+retry or stop Runs under strong local authentication/,
|
||||
);
|
||||
assert.match(help, /Root service mutation remains isolated/);
|
||||
assert.equal(help.includes('ql3-service-bridge '), false);
|
||||
assert.equal(loadQingLong3ProductVersion(moduleDirectory), manifest.version);
|
||||
@@ -158,7 +161,10 @@ test('product binary exposes help/version and delegates without a shell', () =>
|
||||
assert.equal(delegatedRunHelp.status, 0);
|
||||
assert.equal(
|
||||
delegatedRunHelp.stdout.trim(),
|
||||
'Usage: ql3-run retry --command-file /absolute/private-command.json',
|
||||
[
|
||||
'Usage: ql3-run retry --command-file /absolute/private-command.json',
|
||||
' ql3-run stop --command-file /absolute/private-command.json',
|
||||
].join('\n'),
|
||||
);
|
||||
assert.equal(delegatedRunHelp.stderr, '');
|
||||
|
||||
|
||||
@@ -195,7 +195,7 @@ test('binary exposes only the private command-file retry interface', () => {
|
||||
'..',
|
||||
'dist',
|
||||
'run-management',
|
||||
'runRetryCli.js',
|
||||
'runManagementCli.js',
|
||||
);
|
||||
const help = spawnSync(process.execPath, [cli, '--help'], {
|
||||
encoding: 'utf8',
|
||||
@@ -203,7 +203,10 @@ test('binary exposes only the private command-file retry interface', () => {
|
||||
assert.equal(help.status, 0);
|
||||
assert.equal(
|
||||
help.stdout.trim(),
|
||||
'Usage: ql3-run retry --command-file /absolute/private-command.json',
|
||||
[
|
||||
'Usage: ql3-run retry --command-file /absolute/private-command.json',
|
||||
' ql3-run stop --command-file /absolute/private-command.json',
|
||||
].join('\n'),
|
||||
);
|
||||
const invalid = spawnSync(process.execPath, [cli, 'retry'], {
|
||||
encoding: 'utf8',
|
||||
@@ -211,6 +214,6 @@ test('binary exposes only the private command-file retry interface', () => {
|
||||
assert.equal(invalid.status, 64);
|
||||
assert.equal(
|
||||
JSON.parse(invalid.stderr).code,
|
||||
'LOCAL_RUN_RETRY_CLI_USAGE_INVALID',
|
||||
'LOCAL_RUN_MANAGEMENT_CLI_USAGE_INVALID',
|
||||
);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,245 @@
|
||||
const assert = require('node:assert/strict');
|
||||
const { spawnSync } = require('node:child_process');
|
||||
const { DatabaseSync } = require('node:sqlite');
|
||||
const path = require('node:path');
|
||||
const { test } = require('node:test');
|
||||
|
||||
const {
|
||||
RunCancellationNotFoundError,
|
||||
} = require('@qinglong/runtime-core/run-cancellation');
|
||||
const {
|
||||
openLocalSqliteRuntimeDatabase,
|
||||
} = require('@qinglong/local-sqlite/runtime');
|
||||
const {
|
||||
runLocalRunStopCommandFile,
|
||||
} = require('../dist/run-management/runStopCommand.js');
|
||||
const {
|
||||
auditRows,
|
||||
localManagementFixture,
|
||||
writeCommand,
|
||||
} = require('./localManagementFixture.cjs');
|
||||
|
||||
function uuid(value) {
|
||||
return `019f9300-0000-4000-8000-${value.toString(16).padStart(12, '0')}`;
|
||||
}
|
||||
|
||||
async function createRunningSource(value, suffix = 1) {
|
||||
const runtime = await openLocalSqliteRuntimeDatabase({
|
||||
databasePath: value.databasePath,
|
||||
profile: 'edge',
|
||||
});
|
||||
try {
|
||||
const task = (
|
||||
await runtime.taskDefinitions.appendTaskDefinitionRevision({
|
||||
projectId: 'default',
|
||||
taskId: `stop-product-task-${suffix}`,
|
||||
expectedRevision: null,
|
||||
mutationId: uuid(10 + suffix),
|
||||
name: 'Stop product task',
|
||||
kind: 'command',
|
||||
spec: {
|
||||
schema: 'qinglong/command@v1',
|
||||
config: {
|
||||
command: {
|
||||
kind: 'argv',
|
||||
file: '/bin/echo',
|
||||
args: ['stop-product'],
|
||||
},
|
||||
},
|
||||
},
|
||||
labels: {},
|
||||
enabled: true,
|
||||
occurredAtMs: value.now,
|
||||
})
|
||||
).definition;
|
||||
return await (
|
||||
await runtime.taskStartRepository()
|
||||
).startTask({
|
||||
projectId: 'default',
|
||||
taskId: task.taskId,
|
||||
mutationId: uuid(20 + suffix),
|
||||
expectedRevision: task.revision,
|
||||
expectedContentDigest: task.contentDigest,
|
||||
runId: uuid(30 + suffix),
|
||||
attemptId: uuid(40 + suffix),
|
||||
createdEventId: uuid(50 + suffix),
|
||||
queuedEventId: uuid(60 + suffix),
|
||||
subject: { type: 'user', id: 'automation-user' },
|
||||
policyFence: { projectVersion: 1, bindingVersion: 1 },
|
||||
});
|
||||
} finally {
|
||||
await runtime.close();
|
||||
}
|
||||
}
|
||||
|
||||
function request(value, runId, overrides = {}) {
|
||||
return {
|
||||
projectId: 'default',
|
||||
runId,
|
||||
mutationId: uuid(100),
|
||||
requestId: 'local-run-stop-product-1',
|
||||
auditEventId: uuid(101),
|
||||
failureAuditEventId: uuid(102),
|
||||
occurredAtMs: value.now,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
test('strongly stops one Local Run and replays through the unified CLI', async (t) => {
|
||||
const value = await localManagementFixture(t);
|
||||
const source = await createRunningSource(value);
|
||||
const commandFile = writeCommand(
|
||||
value,
|
||||
'run.stop',
|
||||
request(value, source.runId),
|
||||
'run-stop',
|
||||
);
|
||||
|
||||
const accepted = await runLocalRunStopCommandFile(commandFile);
|
||||
assert.equal(accepted.schemaVersion, 1);
|
||||
assert.equal(accepted.operation, 'run.stop');
|
||||
assert.equal(accepted.stop.status, 'accepted');
|
||||
assert.equal(accepted.stop.runId, source.runId);
|
||||
assert.equal(accepted.stop.cancelReason, 'user');
|
||||
|
||||
const cli = path.join(
|
||||
__dirname,
|
||||
'..',
|
||||
'dist',
|
||||
'run-management',
|
||||
'runManagementCli.js',
|
||||
);
|
||||
const replay = spawnSync(
|
||||
process.execPath,
|
||||
[cli, 'stop', '--command-file', commandFile],
|
||||
{ encoding: 'utf8' },
|
||||
);
|
||||
assert.equal(replay.status, 0, replay.stderr);
|
||||
assert.equal(JSON.parse(replay.stdout).stop.status, 'already_requested');
|
||||
|
||||
assert.deepEqual(
|
||||
auditRows(value.databasePath)
|
||||
.filter(({ operationId }) => operationId === 'run.stop')
|
||||
.map((row) => ({ ...row })),
|
||||
[
|
||||
{
|
||||
eventId: uuid(101),
|
||||
operationId: 'run.stop',
|
||||
outcome: 'allowed',
|
||||
reasonsJson: '["role_grant","strong_authentication"]',
|
||||
},
|
||||
],
|
||||
);
|
||||
const database = new DatabaseSync(value.databasePath, { readOnly: true });
|
||||
try {
|
||||
assert.deepEqual(
|
||||
{
|
||||
...database
|
||||
.prepare(
|
||||
`SELECT "cancel_reason" AS "cancelReason",
|
||||
(SELECT count(*) FROM "RunEvents"
|
||||
WHERE "run_id" = ? AND "type" = 'run.cancel_requested')
|
||||
AS "eventCount"
|
||||
FROM "Runs" WHERE "id" = ?`,
|
||||
)
|
||||
.get(source.runId, source.runId),
|
||||
},
|
||||
{ cancelReason: 'user', eventCount: 1 },
|
||||
);
|
||||
} finally {
|
||||
database.close();
|
||||
}
|
||||
});
|
||||
|
||||
test('audits a missing stop target without creating Run state', async (t) => {
|
||||
const value = await localManagementFixture(t);
|
||||
const commandFile = writeCommand(
|
||||
value,
|
||||
'run.stop',
|
||||
request(value, 'missing-run', {
|
||||
mutationId: uuid(110),
|
||||
requestId: 'local-run-stop-missing',
|
||||
auditEventId: uuid(111),
|
||||
failureAuditEventId: uuid(112),
|
||||
}),
|
||||
'run-stop-missing',
|
||||
);
|
||||
await assert.rejects(
|
||||
runLocalRunStopCommandFile(commandFile),
|
||||
RunCancellationNotFoundError,
|
||||
);
|
||||
assert.deepEqual(
|
||||
auditRows(value.databasePath)
|
||||
.filter(({ operationId }) => operationId === 'run.stop')
|
||||
.map((row) => ({ ...row })),
|
||||
[
|
||||
{
|
||||
eventId: uuid(112),
|
||||
operationId: 'run.stop',
|
||||
outcome: 'denied',
|
||||
reasonsJson: '["run_not_found"]',
|
||||
},
|
||||
],
|
||||
);
|
||||
});
|
||||
|
||||
test('denies a Viewer and preserves the running Run', async (t) => {
|
||||
const value = await localManagementFixture(t);
|
||||
const source = await createRunningSource(value, 2);
|
||||
const policyDatabase = new DatabaseSync(value.databasePath);
|
||||
try {
|
||||
policyDatabase
|
||||
.prepare(
|
||||
`INSERT INTO "QingLong3ProjectRoleBindings" (
|
||||
"project_id", "subject_type", "subject_id", "version", "state",
|
||||
"role", "mutation_id", "changed_by_type", "changed_by_id",
|
||||
"created_at_ms"
|
||||
) VALUES ('default', 'user', 'automation-user', 2, 'active', 'viewer',
|
||||
'run-stop-viewer-binding', 'user', 'automation-user', ?)`,
|
||||
)
|
||||
.run(value.now + 1);
|
||||
} finally {
|
||||
policyDatabase.close();
|
||||
}
|
||||
const commandFile = writeCommand(
|
||||
value,
|
||||
'run.stop',
|
||||
request(value, source.runId, {
|
||||
mutationId: uuid(120),
|
||||
requestId: 'local-run-stop-viewer',
|
||||
auditEventId: uuid(121),
|
||||
failureAuditEventId: uuid(122),
|
||||
}),
|
||||
'run-stop-viewer',
|
||||
);
|
||||
await assert.rejects(
|
||||
runLocalRunStopCommandFile(commandFile),
|
||||
(error) => error?.code === 'LOCAL_RUN_STOP_COMMAND_AUTHORIZATION_REJECTED',
|
||||
);
|
||||
assert.deepEqual(
|
||||
auditRows(value.databasePath)
|
||||
.filter(({ operationId }) => operationId === 'run.stop')
|
||||
.map((row) => ({ ...row })),
|
||||
[
|
||||
{
|
||||
eventId: uuid(122),
|
||||
operationId: 'run.stop',
|
||||
outcome: 'denied',
|
||||
reasonsJson: '["run_stop_fence_rejected"]',
|
||||
},
|
||||
],
|
||||
);
|
||||
const database = new DatabaseSync(value.databasePath, { readOnly: true });
|
||||
try {
|
||||
assert.equal(
|
||||
database
|
||||
.prepare(
|
||||
`SELECT "cancel_requested_at_ms" AS value FROM "Runs" WHERE "id" = ?`,
|
||||
)
|
||||
.get(source.runId).value,
|
||||
null,
|
||||
);
|
||||
} finally {
|
||||
database.close();
|
||||
}
|
||||
});
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
LocalSqliteRunManualRetryRepository,
|
||||
STANDALONE_RUN_MANUAL_RETRY_RATE_LIMIT,
|
||||
} from '../run/runManualRetryRepository';
|
||||
import { LocalSqliteRunCancellationRepository } from '../run/runCancellationRepository';
|
||||
import { LocalSqliteApiCredentialRepository } from '../security/apiCredentialRepository';
|
||||
import { LocalSqliteProjectPolicyRepository } from '../security/projectPolicyRepository';
|
||||
import { LocalSqliteSecurityAuthorityStore } from '../security/securityAuthorityStore';
|
||||
@@ -37,6 +38,10 @@ export interface LocalSqliteRunManagementDatabase {
|
||||
readonly apiCredentials: ApiCredentialRepository;
|
||||
readonly ownerPepper: Pick<LocalOwnerPepperRepository, 'resolveKey'>;
|
||||
readonly projectPolicy: ProjectPolicyRepository;
|
||||
readonly runCancellation: Pick<
|
||||
LocalSqliteRunCancellationRepository,
|
||||
'requestUserCancellationAudited'
|
||||
>;
|
||||
readonly runManualRetry: RunManualRetryRepository;
|
||||
readonly securityAudit: SecurityAuditSink;
|
||||
activateUserCredentialFence(
|
||||
@@ -95,6 +100,25 @@ export async function openLocalSqliteRunManagementDatabase(
|
||||
);
|
||||
},
|
||||
});
|
||||
const runCancellation = new LocalSqliteRunCancellationRepository(
|
||||
authority,
|
||||
Date.now,
|
||||
{
|
||||
beforeMutation(actor) {
|
||||
if (
|
||||
!activeFence ||
|
||||
actor.type !== activeFence.subjectType ||
|
||||
actor.id !== activeFence.subjectId
|
||||
) {
|
||||
throw new LocalSqliteAuthenticatedManagementFenceError();
|
||||
}
|
||||
confirmLocalSqliteAuthenticatedUserCredentialFence(
|
||||
authority,
|
||||
activeFence,
|
||||
);
|
||||
},
|
||||
},
|
||||
);
|
||||
let closePromise: Promise<void> | undefined;
|
||||
return Object.freeze({
|
||||
profile: options.profile,
|
||||
@@ -102,6 +126,7 @@ export async function openLocalSqliteRunManagementDatabase(
|
||||
apiCredentials: new LocalSqliteApiCredentialRepository(authority),
|
||||
ownerPepper: new LocalSqliteOwnerPepperRepository(authority),
|
||||
projectPolicy: new LocalSqliteProjectPolicyRepository(authority),
|
||||
runCancellation,
|
||||
runManualRetry,
|
||||
securityAudit: securityAuthority,
|
||||
activateUserCredentialFence(
|
||||
|
||||
@@ -12,8 +12,23 @@ import {
|
||||
type RunCancellationResult,
|
||||
} from '@qinglong/runtime-core/run-cancellation';
|
||||
import { RUN_STATUSES, type RunStatus } from '@qinglong/runtime-core/run';
|
||||
import {
|
||||
normalizeSecurityPrincipal,
|
||||
type SecurityPolicyFence,
|
||||
type SecurityPrincipal,
|
||||
type SecuritySubject,
|
||||
} from '@qinglong/runtime-core/security';
|
||||
import {
|
||||
normalizeSecurityAuditRecord,
|
||||
type SecurityAuditRecord,
|
||||
} from '@qinglong/runtime-core/security-audit';
|
||||
|
||||
import { LocalSqliteOperationAuthority } from '../authority/operationAuthority';
|
||||
import {
|
||||
insertLocalSecurityAudit,
|
||||
localSecurityAuditFromRow,
|
||||
sameSecurityAuditSemantic,
|
||||
} from '../security/securityPersistence';
|
||||
import {
|
||||
optionalInteger,
|
||||
optionalString,
|
||||
@@ -22,6 +37,15 @@ import {
|
||||
type QueryRow,
|
||||
} from './runPersistence';
|
||||
|
||||
const MAX_AUTHENTICATION_AGE_MS = 5 * 60 * 1_000;
|
||||
const STRONG_ASSURANCES = new Set([
|
||||
'multi_factor',
|
||||
'hardware',
|
||||
'local_console',
|
||||
]);
|
||||
const IDENTIFIER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
|
||||
const UUID_PATTERN =
|
||||
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
|
||||
const ALLOWED_ROLES = new Set<RunCancellationAllowedRole>([
|
||||
'owner',
|
||||
'admin',
|
||||
@@ -40,6 +64,119 @@ const CANCEL_REASONS = new Set([
|
||||
'reconcile',
|
||||
'timeout',
|
||||
]);
|
||||
const AUDIT_SELECT = `
|
||||
"event_id" AS "eventId",
|
||||
"request_id" AS "requestId",
|
||||
"operation_id" AS "operationId",
|
||||
"project_id" AS "auditProjectId",
|
||||
"subject_type" AS "subjectType",
|
||||
"subject_id" AS "subjectId",
|
||||
"authentication_id" AS "authenticationId",
|
||||
"outcome" AS "outcome",
|
||||
"reasons_json" AS "reasonsJson",
|
||||
"fence_project_version" AS "fenceProjectVersion",
|
||||
"fence_binding_version" AS "fenceBindingVersion",
|
||||
"occurred_at_ms" AS "occurredAtMs"`;
|
||||
|
||||
export interface LocalSqliteRunManagementCancellationCommand {
|
||||
readonly projectId: string;
|
||||
readonly runId: string;
|
||||
readonly mutationId: string;
|
||||
readonly eventId: string;
|
||||
readonly requestId: string;
|
||||
readonly auditEventId: string;
|
||||
readonly principal: Readonly<SecurityPrincipal>;
|
||||
readonly policyFence: Readonly<SecurityPolicyFence>;
|
||||
}
|
||||
|
||||
export interface LocalSqliteRunCancellationRepositoryOptions {
|
||||
readonly beforeMutation?: (actor: Readonly<SecuritySubject>) => void;
|
||||
}
|
||||
|
||||
interface CancellationAudit {
|
||||
readonly requestId: string;
|
||||
readonly auditEventId: string;
|
||||
readonly principal: Readonly<SecurityPrincipal>;
|
||||
}
|
||||
|
||||
function exact(
|
||||
value: unknown,
|
||||
keys: readonly string[],
|
||||
): Record<string, unknown> {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value)) {
|
||||
throw new InvalidRunCancellationError('management command is invalid');
|
||||
}
|
||||
const actual = Object.keys(value).sort();
|
||||
const expected = [...keys].sort();
|
||||
if (
|
||||
actual.length !== expected.length ||
|
||||
actual.some((key, index) => key !== expected[index])
|
||||
) {
|
||||
throw new InvalidRunCancellationError(
|
||||
'management command shape is invalid',
|
||||
);
|
||||
}
|
||||
return value as Record<string, unknown>;
|
||||
}
|
||||
|
||||
function identifier(value: unknown, label: string): string {
|
||||
if (typeof value !== 'string' || !IDENTIFIER_PATTERN.test(value)) {
|
||||
throw new InvalidRunCancellationError(`${label} is invalid`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function uuid(value: unknown, label: string): string {
|
||||
if (typeof value !== 'string' || !UUID_PATTERN.test(value)) {
|
||||
throw new InvalidRunCancellationError(`${label} is invalid`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function normalizeManagementCommand(
|
||||
value: Readonly<LocalSqliteRunManagementCancellationCommand>,
|
||||
): Readonly<{
|
||||
command: Readonly<RunCancellationCommand>;
|
||||
audit: Readonly<CancellationAudit>;
|
||||
}> {
|
||||
const input = exact(value, [
|
||||
'projectId',
|
||||
'runId',
|
||||
'mutationId',
|
||||
'eventId',
|
||||
'requestId',
|
||||
'auditEventId',
|
||||
'principal',
|
||||
'policyFence',
|
||||
]);
|
||||
const principal = exact(input.principal, [
|
||||
'subject',
|
||||
'authenticationId',
|
||||
'authenticatedAtMs',
|
||||
'expiresAtMs',
|
||||
'assurance',
|
||||
]) as unknown as SecurityPrincipal;
|
||||
const projectId = identifier(input.projectId, 'projectId');
|
||||
const runId = identifier(input.runId, 'runId');
|
||||
const mutationId = uuid(input.mutationId, 'mutationId');
|
||||
const eventId = uuid(input.eventId, 'eventId');
|
||||
const requestId = identifier(input.requestId, 'requestId');
|
||||
const auditEventId = uuid(input.auditEventId, 'auditEventId');
|
||||
if (eventId === auditEventId) {
|
||||
throw new InvalidRunCancellationError('event identities must differ');
|
||||
}
|
||||
return Object.freeze({
|
||||
command: normalizeRunCancellationCommand({
|
||||
projectId,
|
||||
runId,
|
||||
mutationId,
|
||||
eventId,
|
||||
subject: principal.subject,
|
||||
policyFence: input.policyFence as SecurityPolicyFence,
|
||||
}),
|
||||
audit: Object.freeze({ requestId, auditEventId, principal }),
|
||||
});
|
||||
}
|
||||
|
||||
function timestamp(value: number): number {
|
||||
if (!Number.isSafeInteger(value) || value < 0) {
|
||||
@@ -99,18 +236,28 @@ function rollback(authority: LocalSqliteOperationAuthority): void {
|
||||
export class LocalSqliteRunCancellationRepository
|
||||
implements RunCancellationRepository
|
||||
{
|
||||
private readonly beforeMutation: (actor: Readonly<SecuritySubject>) => void;
|
||||
|
||||
constructor(
|
||||
private readonly authority: LocalSqliteOperationAuthority,
|
||||
private readonly now: () => number = Date.now,
|
||||
options: LocalSqliteRunCancellationRepositoryOptions = {},
|
||||
) {
|
||||
if (
|
||||
!(authority instanceof LocalSqliteOperationAuthority) ||
|
||||
typeof now !== 'function'
|
||||
typeof now !== 'function' ||
|
||||
!options ||
|
||||
typeof options !== 'object' ||
|
||||
Array.isArray(options) ||
|
||||
Object.keys(options).some((key) => key !== 'beforeMutation') ||
|
||||
(options.beforeMutation !== undefined &&
|
||||
typeof options.beforeMutation !== 'function')
|
||||
) {
|
||||
throw new TypeError(
|
||||
'Local SQLite Run cancellation dependencies are invalid',
|
||||
);
|
||||
}
|
||||
this.beforeMutation = options.beforeMutation ?? (() => undefined);
|
||||
}
|
||||
|
||||
requestUserCancellation(
|
||||
@@ -122,51 +269,59 @@ export class LocalSqliteRunCancellationRepository
|
||||
} catch (error) {
|
||||
return Promise.reject(error);
|
||||
}
|
||||
return this.requestCancellation(command);
|
||||
}
|
||||
|
||||
requestUserCancellationAudited(
|
||||
value: Readonly<LocalSqliteRunManagementCancellationCommand>,
|
||||
): Promise<Readonly<RunCancellationResult>> {
|
||||
let normalized: ReturnType<typeof normalizeManagementCommand>;
|
||||
try {
|
||||
normalized = normalizeManagementCommand(value);
|
||||
} catch (error) {
|
||||
return Promise.reject(error);
|
||||
}
|
||||
return this.requestCancellation(normalized.command, normalized.audit);
|
||||
}
|
||||
|
||||
private requestCancellation(
|
||||
command: Readonly<RunCancellationCommand>,
|
||||
audit?: Readonly<CancellationAudit>,
|
||||
): Promise<Readonly<RunCancellationResult>> {
|
||||
return this.authority.enqueue(
|
||||
async () => {
|
||||
const client = this.authority.client;
|
||||
try {
|
||||
client.exec('BEGIN IMMEDIATE');
|
||||
const project = client
|
||||
.prepare(
|
||||
`SELECT "status" AS "projectStatus", "version" AS "projectVersion"
|
||||
FROM "QingLong3Projects" WHERE "id" = ?`,
|
||||
)
|
||||
.get(command.projectId) as QueryRow | undefined;
|
||||
if (!project) throw new RunCancellationNotFoundError();
|
||||
const binding = client
|
||||
.prepare(
|
||||
`SELECT "version" AS "bindingVersion", "state" AS "bindingState",
|
||||
"role" AS "bindingRole"
|
||||
FROM "QingLong3ProjectRoleBindings"
|
||||
WHERE "project_id" = ? AND "subject_type" = ?
|
||||
AND "subject_id" = ?
|
||||
ORDER BY "version" DESC LIMIT 1`,
|
||||
)
|
||||
.get(
|
||||
command.projectId,
|
||||
command.subject.type,
|
||||
command.subject.id,
|
||||
) as QueryRow | undefined;
|
||||
const observedAtMs = audit ? this.databaseTime() : undefined;
|
||||
const confirmedAudit = audit
|
||||
? Object.freeze({
|
||||
...audit,
|
||||
principal: this.confirmStrongAuthentication(
|
||||
audit.principal,
|
||||
observedAtMs!,
|
||||
),
|
||||
})
|
||||
: undefined;
|
||||
if (
|
||||
requiredString(project, 'projectStatus') !== 'active' ||
|
||||
requiredInteger(project, 'projectVersion') !==
|
||||
command.policyFence.projectVersion ||
|
||||
!binding ||
|
||||
requiredInteger(binding, 'bindingVersion') !==
|
||||
command.policyFence.bindingVersion ||
|
||||
requiredString(binding, 'bindingState') !== 'active' ||
|
||||
!ALLOWED_ROLES.has(
|
||||
requiredString(
|
||||
binding,
|
||||
'bindingRole',
|
||||
) as RunCancellationAllowedRole,
|
||||
)
|
||||
confirmedAudit &&
|
||||
(confirmedAudit.principal.subject.type !== command.subject.type ||
|
||||
confirmedAudit.principal.subject.id !== command.subject.id)
|
||||
) {
|
||||
throw new RunCancellationFenceRejectedError(
|
||||
'authorization_changed',
|
||||
);
|
||||
}
|
||||
if (confirmedAudit) {
|
||||
try {
|
||||
this.beforeMutation(confirmedAudit.principal.subject);
|
||||
} catch {
|
||||
throw new RunCancellationFenceRejectedError(
|
||||
'authorization_changed',
|
||||
);
|
||||
}
|
||||
}
|
||||
this.confirmAuthorization(command);
|
||||
|
||||
const run = client
|
||||
.prepare(
|
||||
@@ -178,84 +333,83 @@ export class LocalSqliteRunCancellationRepository
|
||||
FROM "Runs" WHERE "id" = ?`,
|
||||
)
|
||||
.get(command.runId) as QueryRow | undefined;
|
||||
if (
|
||||
!run ||
|
||||
requiredString(run, 'projectId') !== command.projectId
|
||||
) {
|
||||
if (!run || requiredString(run, 'projectId') !== command.projectId) {
|
||||
throw new RunCancellationNotFoundError();
|
||||
}
|
||||
const runStatus = status(run);
|
||||
let outcome: Readonly<RunCancellationResult>;
|
||||
if (TERMINAL.has(runStatus)) {
|
||||
const outcome = result('already_terminal', command, run);
|
||||
client.exec('COMMIT');
|
||||
return outcome;
|
||||
}
|
||||
if (optionalInteger(run, 'cancelRequestedAtMs') !== undefined) {
|
||||
const outcome = result('already_requested', command, run);
|
||||
client.exec('COMMIT');
|
||||
return outcome;
|
||||
}
|
||||
if (optionalString(run, 'cancelReason') !== undefined) {
|
||||
outcome = result('already_terminal', command, run);
|
||||
} else if (
|
||||
optionalInteger(run, 'cancelRequestedAtMs') !== undefined
|
||||
) {
|
||||
outcome = result('already_requested', command, run);
|
||||
} else if (optionalString(run, 'cancelReason') !== undefined) {
|
||||
throw new RunCancellationFenceRejectedError('state_mismatch');
|
||||
} else {
|
||||
const runVersion = requiredInteger(run, 'runVersion');
|
||||
const eventSequence = requiredInteger(run, 'eventSequence');
|
||||
if (runVersion >= 2_147_483_647 || eventSequence >= 2_147_483_647) {
|
||||
throw new RunCancellationFenceRejectedError('state_mismatch');
|
||||
}
|
||||
const mutationObservedAtMs = observedAtMs ?? timestamp(this.now());
|
||||
const updated = client
|
||||
.prepare(
|
||||
`UPDATE "Runs"
|
||||
SET "cancel_requested_at_ms" = ?, "cancel_reason" = 'user',
|
||||
"version" = ?, "event_sequence" = ?
|
||||
WHERE "id" = ? AND "project_id" = ? AND "version" = ?
|
||||
AND "cancel_requested_at_ms" IS NULL
|
||||
RETURNING "project_id" AS "projectId",
|
||||
"status" AS "runStatus", "version" AS "runVersion",
|
||||
"event_sequence" AS "eventSequence",
|
||||
"cancel_requested_at_ms" AS "cancelRequestedAtMs",
|
||||
"cancel_reason" AS "cancelReason"`,
|
||||
)
|
||||
.get(
|
||||
mutationObservedAtMs,
|
||||
runVersion + 1,
|
||||
eventSequence + 1,
|
||||
command.runId,
|
||||
command.projectId,
|
||||
runVersion,
|
||||
) as QueryRow | undefined;
|
||||
if (!updated) {
|
||||
throw new RunCancellationFenceRejectedError('state_mismatch');
|
||||
}
|
||||
client
|
||||
.prepare(
|
||||
`INSERT INTO "RunEvents" (
|
||||
"id", "run_id", "sequence", "type", "dedupe_key",
|
||||
"actor_type", "actor_id", "attempt_id", "step_run_id",
|
||||
"payload", "created_at_ms"
|
||||
) VALUES (?, ?, ?, 'run.cancel_requested', ?, ?, ?,
|
||||
NULL, NULL, ?, ?)`,
|
||||
)
|
||||
.run(
|
||||
command.eventId,
|
||||
command.runId,
|
||||
eventSequence + 1,
|
||||
`user-cancel:${command.mutationId}`,
|
||||
command.subject.type,
|
||||
command.subject.id,
|
||||
JSON.stringify({
|
||||
reason: 'user',
|
||||
mutation_id: command.mutationId,
|
||||
policy_fence: {
|
||||
project_version: command.policyFence.projectVersion,
|
||||
binding_version: command.policyFence.bindingVersion,
|
||||
},
|
||||
}),
|
||||
mutationObservedAtMs,
|
||||
);
|
||||
outcome = result('accepted', command, updated);
|
||||
}
|
||||
|
||||
const runVersion = requiredInteger(run, 'runVersion');
|
||||
const eventSequence = requiredInteger(run, 'eventSequence');
|
||||
if (runVersion >= 2_147_483_647 || eventSequence >= 2_147_483_647) {
|
||||
throw new RunCancellationFenceRejectedError('state_mismatch');
|
||||
}
|
||||
const observedAtMs = timestamp(this.now());
|
||||
const updated = client
|
||||
.prepare(
|
||||
`UPDATE "Runs"
|
||||
SET "cancel_requested_at_ms" = ?, "cancel_reason" = 'user',
|
||||
"version" = ?, "event_sequence" = ?
|
||||
WHERE "id" = ? AND "project_id" = ? AND "version" = ?
|
||||
AND "cancel_requested_at_ms" IS NULL
|
||||
RETURNING "project_id" AS "projectId",
|
||||
"status" AS "runStatus", "version" AS "runVersion",
|
||||
"event_sequence" AS "eventSequence",
|
||||
"cancel_requested_at_ms" AS "cancelRequestedAtMs",
|
||||
"cancel_reason" AS "cancelReason"`,
|
||||
)
|
||||
.get(
|
||||
observedAtMs,
|
||||
runVersion + 1,
|
||||
eventSequence + 1,
|
||||
command.runId,
|
||||
command.projectId,
|
||||
runVersion,
|
||||
) as QueryRow | undefined;
|
||||
if (!updated) {
|
||||
throw new RunCancellationFenceRejectedError('state_mismatch');
|
||||
}
|
||||
client
|
||||
.prepare(
|
||||
`INSERT INTO "RunEvents" (
|
||||
"id", "run_id", "sequence", "type", "dedupe_key",
|
||||
"actor_type", "actor_id", "attempt_id", "step_run_id",
|
||||
"payload", "created_at_ms"
|
||||
) VALUES (?, ?, ?, 'run.cancel_requested', ?, ?, ?,
|
||||
NULL, NULL, ?, ?)`,
|
||||
)
|
||||
.run(
|
||||
command.eventId,
|
||||
command.runId,
|
||||
eventSequence + 1,
|
||||
`user-cancel:${command.mutationId}`,
|
||||
command.subject.type,
|
||||
command.subject.id,
|
||||
JSON.stringify({
|
||||
reason: 'user',
|
||||
mutation_id: command.mutationId,
|
||||
policy_fence: {
|
||||
project_version: command.policyFence.projectVersion,
|
||||
binding_version: command.policyFence.bindingVersion,
|
||||
},
|
||||
}),
|
||||
observedAtMs,
|
||||
if (confirmedAudit) {
|
||||
this.commitAudit(
|
||||
this.allowedAudit(command, confirmedAudit, observedAtMs!),
|
||||
);
|
||||
const outcome = result('accepted', command, updated);
|
||||
}
|
||||
client.exec('COMMIT');
|
||||
return outcome;
|
||||
} catch (error) {
|
||||
@@ -273,4 +427,114 @@ export class LocalSqliteRunCancellationRepository
|
||||
() => new RunCancellationUnavailableError(),
|
||||
);
|
||||
}
|
||||
|
||||
private databaseTime(): number {
|
||||
const row = this.authority.client
|
||||
.prepare(
|
||||
`SELECT CAST(unixepoch('subsec') * 1000 AS INTEGER) AS "observedAtMs"`,
|
||||
)
|
||||
.get() as QueryRow | undefined;
|
||||
return timestamp(row?.observedAtMs as number);
|
||||
}
|
||||
|
||||
private confirmStrongAuthentication(
|
||||
value: Readonly<SecurityPrincipal>,
|
||||
observedAtMs: number,
|
||||
): Readonly<SecurityPrincipal> {
|
||||
let principal: Readonly<SecurityPrincipal>;
|
||||
try {
|
||||
principal = normalizeSecurityPrincipal(value, observedAtMs);
|
||||
} catch {
|
||||
throw new RunCancellationFenceRejectedError('authorization_changed');
|
||||
}
|
||||
if (
|
||||
principal.subject.type !== 'user' ||
|
||||
!STRONG_ASSURANCES.has(principal.assurance) ||
|
||||
principal.authenticatedAtMs > observedAtMs ||
|
||||
principal.expiresAtMs <= observedAtMs ||
|
||||
observedAtMs - principal.authenticatedAtMs > MAX_AUTHENTICATION_AGE_MS
|
||||
) {
|
||||
throw new RunCancellationFenceRejectedError('authorization_changed');
|
||||
}
|
||||
return principal;
|
||||
}
|
||||
|
||||
private confirmAuthorization(
|
||||
command: Readonly<RunCancellationCommand>,
|
||||
): void {
|
||||
const client = this.authority.client;
|
||||
const project = client
|
||||
.prepare(
|
||||
`SELECT "status" AS "projectStatus", "version" AS "projectVersion"
|
||||
FROM "QingLong3Projects" WHERE "id" = ?`,
|
||||
)
|
||||
.get(command.projectId) as QueryRow | undefined;
|
||||
if (!project) throw new RunCancellationNotFoundError();
|
||||
const binding = client
|
||||
.prepare(
|
||||
`SELECT "version" AS "bindingVersion", "state" AS "bindingState",
|
||||
"role" AS "bindingRole"
|
||||
FROM "QingLong3ProjectRoleBindings"
|
||||
WHERE "project_id" = ? AND "subject_type" = ?
|
||||
AND "subject_id" = ?
|
||||
ORDER BY "version" DESC LIMIT 1`,
|
||||
)
|
||||
.get(command.projectId, command.subject.type, command.subject.id) as
|
||||
| QueryRow
|
||||
| undefined;
|
||||
if (
|
||||
requiredString(project, 'projectStatus') !== 'active' ||
|
||||
requiredInteger(project, 'projectVersion') !==
|
||||
command.policyFence.projectVersion ||
|
||||
!binding ||
|
||||
requiredInteger(binding, 'bindingVersion') !==
|
||||
command.policyFence.bindingVersion ||
|
||||
requiredString(binding, 'bindingState') !== 'active' ||
|
||||
!ALLOWED_ROLES.has(
|
||||
requiredString(binding, 'bindingRole') as RunCancellationAllowedRole,
|
||||
)
|
||||
) {
|
||||
throw new RunCancellationFenceRejectedError('authorization_changed');
|
||||
}
|
||||
}
|
||||
|
||||
private allowedAudit(
|
||||
command: Readonly<RunCancellationCommand>,
|
||||
audit: Readonly<CancellationAudit>,
|
||||
observedAtMs: number,
|
||||
): Readonly<SecurityAuditRecord> {
|
||||
return normalizeSecurityAuditRecord({
|
||||
eventId: audit.auditEventId,
|
||||
requestId: audit.requestId,
|
||||
operationId: 'run.stop',
|
||||
projectId: command.projectId,
|
||||
subject: audit.principal.subject,
|
||||
authenticationId: audit.principal.authenticationId,
|
||||
outcome: 'allowed',
|
||||
reasons: ['role_grant', 'strong_authentication'],
|
||||
fence: command.policyFence,
|
||||
occurredAtMs: observedAtMs,
|
||||
});
|
||||
}
|
||||
|
||||
private commitAudit(audit: Readonly<SecurityAuditRecord>): void {
|
||||
const row = this.authority.client
|
||||
.prepare(
|
||||
`SELECT ${AUDIT_SELECT}
|
||||
FROM "QingLong3SecurityAuditEvents" WHERE "event_id" = ?`,
|
||||
)
|
||||
.get(audit.eventId) as QueryRow | undefined;
|
||||
if (row) {
|
||||
const stored = localSecurityAuditFromRow(row);
|
||||
const storedWithoutTime = Object.freeze({
|
||||
...stored,
|
||||
occurredAtMs: audit.occurredAtMs,
|
||||
});
|
||||
if (!sameSecurityAuditSemantic(storedWithoutTime, audit)) {
|
||||
throw new RunCancellationFenceRejectedError('state_mismatch');
|
||||
}
|
||||
return;
|
||||
}
|
||||
insertLocalSecurityAudit(this.authority.client, audit);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -56,6 +56,27 @@ function command(overrides = {}) {
|
||||
};
|
||||
}
|
||||
|
||||
function auditedCommand(overrides = {}) {
|
||||
const now = Date.now();
|
||||
return {
|
||||
projectId: 'default',
|
||||
runId: 'run-1',
|
||||
mutationId: '018f0000-0000-4000-8000-000000000010',
|
||||
eventId: '018f0000-0000-4000-8000-000000000011',
|
||||
requestId: 'local-run-stop-request',
|
||||
auditEventId: '018f0000-0000-4000-8000-000000000012',
|
||||
principal: {
|
||||
subject: { type: 'user', id: 'user-1' },
|
||||
authenticationId: 'local-console:run-stop',
|
||||
authenticatedAtMs: now - 1_000,
|
||||
expiresAtMs: now + 60_000,
|
||||
assurance: 'local_console',
|
||||
},
|
||||
policyFence: { projectVersion: 1, bindingVersion: 1 },
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
async function fixture(t, status = 'running') {
|
||||
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'ql3-run-cancel-'));
|
||||
t.after(() => fs.rmSync(root, { recursive: true, force: true }));
|
||||
@@ -68,6 +89,7 @@ async function fixture(t, status = 'running') {
|
||||
t.after(() => authority.close());
|
||||
return {
|
||||
client,
|
||||
authority,
|
||||
repository: new LocalSqliteRunCancellationRepository(authority, () => NOW),
|
||||
};
|
||||
}
|
||||
@@ -85,9 +107,11 @@ test('atomically publishes one durable cancellation intent and exact replay', as
|
||||
cancelReason: 'user',
|
||||
});
|
||||
assert.equal(
|
||||
(await repository.requestUserCancellation(
|
||||
command({ eventId: '018f0000-0000-7000-8000-000000000002' }),
|
||||
)).status,
|
||||
(
|
||||
await repository.requestUserCancellation(
|
||||
command({ eventId: '018f0000-0000-7000-8000-000000000002' }),
|
||||
)
|
||||
).status,
|
||||
'already_requested',
|
||||
);
|
||||
assert.deepEqual(
|
||||
@@ -108,6 +132,75 @@ test('atomically publishes one durable cancellation intent and exact replay', as
|
||||
);
|
||||
});
|
||||
|
||||
test('atomically binds strong management cancellation to one allowed audit', async (t) => {
|
||||
const { authority, client } = await fixture(t);
|
||||
let fenceChecks = 0;
|
||||
const repository = new LocalSqliteRunCancellationRepository(
|
||||
authority,
|
||||
Date.now,
|
||||
{
|
||||
beforeMutation(subject) {
|
||||
assert.deepEqual(subject, { type: 'user', id: 'user-1' });
|
||||
fenceChecks += 1;
|
||||
},
|
||||
},
|
||||
);
|
||||
const value = auditedCommand();
|
||||
const accepted = await repository.requestUserCancellationAudited(value);
|
||||
assert.equal(accepted.status, 'accepted');
|
||||
assert.equal(
|
||||
(await repository.requestUserCancellationAudited(value)).status,
|
||||
'already_requested',
|
||||
);
|
||||
assert.equal(fenceChecks, 2);
|
||||
assert.deepEqual(
|
||||
{
|
||||
...client
|
||||
.prepare(
|
||||
`SELECT "operation_id" AS "operationId", outcome, reasons_json AS "reasonsJson"
|
||||
FROM "QingLong3SecurityAuditEvents" WHERE "event_id" = ?`,
|
||||
)
|
||||
.get(value.auditEventId),
|
||||
},
|
||||
{
|
||||
operationId: 'run.stop',
|
||||
outcome: 'allowed',
|
||||
reasonsJson: '["role_grant","strong_authentication"]',
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test('rolls cancellation back when the authenticated credential fence changes', async (t) => {
|
||||
const { authority, client } = await fixture(t);
|
||||
const repository = new LocalSqliteRunCancellationRepository(
|
||||
authority,
|
||||
Date.now,
|
||||
{
|
||||
beforeMutation() {
|
||||
throw new Error('credential changed');
|
||||
},
|
||||
},
|
||||
);
|
||||
await assert.rejects(
|
||||
repository.requestUserCancellationAudited(auditedCommand()),
|
||||
(error) =>
|
||||
error instanceof RunCancellationFenceRejectedError &&
|
||||
error.reason === 'authorization_changed',
|
||||
);
|
||||
assert.deepEqual(
|
||||
{
|
||||
...client
|
||||
.prepare(
|
||||
`SELECT "cancel_requested_at_ms" AS "cancelRequestedAtMs",
|
||||
(SELECT count(*) FROM "QingLong3SecurityAuditEvents") AS audits
|
||||
FROM "Runs" WHERE "id" = 'run-1'`,
|
||||
)
|
||||
.get(),
|
||||
},
|
||||
{ cancelRequestedAtMs: null, audits: 0 },
|
||||
);
|
||||
});
|
||||
|
||||
test('returns terminal and masks missing or cross-Project Runs', async (t) => {
|
||||
const { repository } = await fixture(t, 'succeeded');
|
||||
assert.deepEqual(await repository.requestUserCancellation(command()), {
|
||||
@@ -148,7 +241,9 @@ test('revalidates the latest RoleBinding inside the mutation transaction', async
|
||||
);
|
||||
assert.equal(
|
||||
client
|
||||
.prepare(`SELECT "cancel_requested_at_ms" AS value FROM "Runs" WHERE "id" = 'run-1'`)
|
||||
.prepare(
|
||||
`SELECT "cancel_requested_at_ms" AS value FROM "Runs" WHERE "id" = 'run-1'`,
|
||||
)
|
||||
.get().value,
|
||||
null,
|
||||
);
|
||||
|
||||
@@ -1675,8 +1675,10 @@ function auditSourceImports(root, packagePath, findings) {
|
||||
specifier.startsWith(`${packageName}/`),
|
||||
) &&
|
||||
!(
|
||||
path.relative(packageDirectory, filePath) ===
|
||||
'src/run-management/runRetryCommand.ts' &&
|
||||
[
|
||||
'src/run-management/runRetryCommand.ts',
|
||||
'src/run-management/runStopCommand.ts',
|
||||
].includes(path.relative(packageDirectory, filePath)) &&
|
||||
[
|
||||
'@qinglong/local-owner-console/authenticated-command',
|
||||
'@qinglong/local-sqlite/authenticated-management',
|
||||
@@ -2539,6 +2541,18 @@ function auditSourceImports(root, packagePath, findings) {
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
if (
|
||||
packagePath === 'packages/ql3-local-owner-cli' &&
|
||||
path.relative(packageDirectory, filePath) ===
|
||||
'src/run-management/runStopCommand.ts' &&
|
||||
[
|
||||
'@qinglong/runtime-core/project-policy',
|
||||
'@qinglong/runtime-core/run-cancellation',
|
||||
'@qinglong/runtime-core/security-audit',
|
||||
].includes(specifier)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
if (
|
||||
packagePath === 'packages/ql3-local-owner-cli' &&
|
||||
path.relative(packageDirectory, filePath) ===
|
||||
|
||||
@@ -119,6 +119,65 @@ test('accepts package-local and declared forward source imports', (t) => {
|
||||
assert.deepEqual(findings, []);
|
||||
});
|
||||
|
||||
test('confines local Run management authority to reviewed retry and stop commands', (t) => {
|
||||
const root = fs.mkdtempSync(
|
||||
path.join(os.tmpdir(), 'ql3-run-management-boundary-'),
|
||||
);
|
||||
const sourceDirectory = path.join(
|
||||
root,
|
||||
'packages/ql3-local-owner-cli/src/run-management',
|
||||
);
|
||||
fs.mkdirSync(sourceDirectory, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(sourceDirectory, 'runRetryCommand.ts'),
|
||||
[
|
||||
"import { authenticate } from '@qinglong/local-owner-console/authenticated-command';",
|
||||
"import { fence } from '@qinglong/local-sqlite/authenticated-management';",
|
||||
"import { database } from '@qinglong/local-sqlite/run-management';",
|
||||
"import { policy } from '@qinglong/runtime-core/project-policy';",
|
||||
"import { retry } from '@qinglong/runtime-core/run-manual-retry';",
|
||||
"import { audit } from '@qinglong/runtime-core/security-audit';",
|
||||
].join('\n'),
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(sourceDirectory, 'runStopCommand.ts'),
|
||||
[
|
||||
"import { authenticate } from '@qinglong/local-owner-console/authenticated-command';",
|
||||
"import { fence } from '@qinglong/local-sqlite/authenticated-management';",
|
||||
"import { database } from '@qinglong/local-sqlite/run-management';",
|
||||
"import { policy } from '@qinglong/runtime-core/project-policy';",
|
||||
"import { stop } from '@qinglong/runtime-core/run-cancellation';",
|
||||
"import { audit } from '@qinglong/runtime-core/security-audit';",
|
||||
].join('\n'),
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(sourceDirectory, 'widenedRunCommand.ts'),
|
||||
[
|
||||
"import { database } from '@qinglong/local-sqlite/run-management';",
|
||||
"import { stop } from '@qinglong/runtime-core/run-cancellation';",
|
||||
].join('\n'),
|
||||
);
|
||||
t.after(() => fs.rmSync(root, { recursive: true, force: true }));
|
||||
|
||||
const findings = [];
|
||||
auditSourceImports(root, 'packages/ql3-local-owner-cli', findings);
|
||||
assert.deepEqual(
|
||||
findings.map(({ code, file, specifier }) => ({ code, file, specifier })),
|
||||
[
|
||||
{
|
||||
code: 'FORBIDDEN_LOCAL_ADOPTION_CLI_AUTHORITY_IMPORT',
|
||||
file: 'packages/ql3-local-owner-cli/src/run-management/widenedRunCommand.ts',
|
||||
specifier: '@qinglong/local-sqlite/run-management',
|
||||
},
|
||||
{
|
||||
code: 'FORBIDDEN_PACKAGE_SOURCE_IMPORT',
|
||||
file: 'packages/ql3-local-owner-cli/src/run-management/widenedRunCommand.ts',
|
||||
specifier: '@qinglong/runtime-core/run-cancellation',
|
||||
},
|
||||
],
|
||||
);
|
||||
});
|
||||
|
||||
test('confines local MCP to its reviewed protocol and read-authority subpaths', (t) => {
|
||||
const root = fixture(
|
||||
t,
|
||||
@@ -1973,8 +2032,7 @@ test('confines Local API identity authentication to its exact read-only adapter'
|
||||
[
|
||||
{
|
||||
code: 'FORBIDDEN_LOCAL_OWNER_PEPPER_DESTRUCTIVE_ENTRYPOINT',
|
||||
file:
|
||||
'packages/ql3-local-api/src/authentication/destructive.ts',
|
||||
file: 'packages/ql3-local-api/src/authentication/destructive.ts',
|
||||
},
|
||||
{
|
||||
code: 'FORBIDDEN_LOCAL_IDENTITY_AUTHORITY_IMPORT',
|
||||
|
||||
@@ -207,10 +207,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
|
||||
rootSourceFileRoles: localOwnerCli.rootSourceFileRoles,
|
||||
},
|
||||
{
|
||||
sourceFiles: 104,
|
||||
sourceFiles: 105,
|
||||
rootSourceFiles: 1,
|
||||
rootSourceLines: 50,
|
||||
nestedSourceFiles: 103,
|
||||
nestedSourceFiles: 104,
|
||||
rootSourceFileRoles: { 'cli.ts': 'binary_entry' },
|
||||
},
|
||||
);
|
||||
|
||||
Reference in New Issue
Block a user