diff --git a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md index 9d603952..fc0d198a 100644 --- a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md +++ b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md @@ -11,6 +11,16 @@ 最新增量证据(2026-08-12): +- D-292/ADR-0380(已接受) + Local Edge/Standalone 已补齐 admission-safe lost Run retry 纵向闭环,并修正“共享纯策略却暴露 Cluster-only 名称”的边界。 + Runtime Core 在既有 Run 域发布 canonical `RunLostRetry*`/`run-lost-retry` subpath,同时保留早期 Cluster 名称兼容;SQLite + adapter 复用唯一 operation authority 与 `BEGIN IMMEDIATE` Run aggregate transaction,有界选择 runtime-owned、非 Workflow、 + 无 cancellation intent 且 latest Attempt 为 lost 的候选,原子 CAS Run/RetryPolicy、插入新 Attempt 与 Events。Local application + 把一页 retry 接入现有 execution-control 周期,顺序为 completion→control→lost retry→慢速 cleanup;启动时在 recovery 后、Scheduler + 前执行首轮。Edge 为 2 条/5 秒,Standalone 为 16 条/1 秒,不新增 package、migration、表、索引、timer、连接、listener 或 sidecar。 + 无 policy/禁用、unknown safety、attempt 耗尽均终态失败关闭;只有 admission 时已证明 `idempotent|deduplicated` 的策略才创建全新 + Attempt。workspace 仍为 18 package/1055 source/1037 nested,`singleSourcePackages=[]`、`shallowSourcePackages=[]`。人工 + `run.retry` API/CLI/UI、强认证、审计与 rate limit 仍未包含,不能把自动 lost retry 误报为人工恢复完成。 - D-291/ADR-0379(已接受) Cluster Run Attempt 日志 retention 已完成多副本纵向闭环,且没有新增 package:共享 claim contract 位于 Runtime Core 既有 Run log-retention 目录;PostgreSQL v54 提供 durable control、immutable tombstone、terminal remote Worker candidate index 与最小权限 @@ -8485,7 +8495,7 @@ flowchart LR > ADR-0058 至 ADR-0092 的以下段落是历史快照,其 PostgreSQL 数字与“下一切片”已由 ADR-0104/0105 及后续 Accepted ADR 取代;资源、Package 和物理证据边界仍保留作为演进记录:PostgreSQL 当时 baseline 为 11 条 reviewed migration、capability v10、19 张表和 migration/runtime/admin/worker-ingress 四角色;本机 SQLite 为二十八条 reviewed migration、capability v14、二十六张 owned table,并新增正式 TaskDefinition head/immutable revision Repository。当前有二十七个受审 3.0 importer:Owner maintenance 已合并为两个权限隔离 subpath,execution/control/recovery/dispatch 已合并为 `@qinglong/local-execution` 四个单向 subpath,bootstrap/credential-recovery 已合并为 `@qinglong/local-owner-ceremony` 两个互不依赖 subpath;三个 package 都不提供聚合根入口,hard cap 已同步降为 27。一次拓扑 build/test 会先清理已登记 QL3 package 的 stale dist;27 包全量测试、dependency/source boundary、联网 vulnerability audit 与六种 Profile 制品门禁均通过。ADR-0090 让 production packlist 只发布 JS、`.d.ts` 和受审 assets/drizzle,不再携带开发 map;ADR-0092 后当前最大 application 为 1,691,009 bytes、267 files、61 loaded modules,最大抽样 RSS delta 11,780,096 bytes,低于 4 MiB/512 files/16 MiB。所有 package build 已变为 self-only,全量从约 198 次编译降至精确 27 次,clean 状态单包测试仍按依赖闭包工作。资源门禁已拆为原生 Linux x64/arm64 的 128 MiB router stress、256 MiB Edge release guard 与 512 MiB Cluster control guard,并从容器内部验证 cgroup v2、零 swap/OOM、非 root、只读挂载、seccomp 和 `NoNewPrivs`;本轮本地原生 arm64 三档已通过,x64 等待远端 CI,所有档位均明确不是物理设备或生产容量承诺。物理 Edge candidate recorder 已绑定设备 manifest、实际 Linux/存储环境、Edge/SQLite 基准与 no-replace SHA-256 报告,并会拒绝容器/VM;同设备同 boot 的 idle sampler、不主动填盘的专用文件系统 fault probe,以及通过正式 Repository 写入/扫描 100/1000/10000 个 `qinglong/command@v1` TaskDefinition 的规模记录协议均已具备契约门禁,但尚未取得固定实机报告或 signature。ADR-0091 已冻结 1–32 个 exact descriptor 的不可变 TaskSpec semantic registry、内建 command v1 和本机写前门禁;历史 revision 在 provider 缺失时仍可读。ADR-0092 已实现绑定 source revision/content digest 的 Profile-neutral command plan 与确定性本机 context/execution 映射,且保持 subpath-only。 -> 下表 PR-1 的旧“未完成”累计文字中,`completion/cancellation/timeout` 已由 ADR-0072 取代并闭环,Artifact range read 已由 ADR-0377 闭环;当前本机剩余项是人工 recovery、retry 产品策略、Artifact retention/tombstone、HTTP/CLI/UI 与部署 controller。远端 Worker completion 与 Workflow cancellation 的核心数据库链和 HA 重放已闭环;仍缺 production ingress/internal runtime port、expiry/retry lifecycle、部署启动装配与真实 Kubernetes 故障证据,不能与本机结论混用。 +> 下表 PR-1 的旧“未完成”累计文字中,`completion/cancellation/timeout` 已由 ADR-0072 取代并闭环,Artifact range read 与 Local/Cluster retention 已由 ADR-0377/0378/0379 闭环,Local admission-safe lost retry lifecycle 已由 ADR-0380 闭环;当前本机剩余项是人工 recovery、手工 retry 产品策略/API、HTTP/CLI/UI、部署 controller 与固定路由设备实机门。远端 Worker completion、Workflow cancellation 和 Cluster expiry/retry 的核心数据库链、HA 重放与启动装配也已闭环;仍缺真实 Kubernetes 多节点故障证据,不能与本机结论混用。 > > 下表 PR-0 的累积长文本仍含“四角色、21 条 migration、capability v20、21 项 HA”历史短语;当前权威基线应读取为六角色、23 条 migration、capability v22/36 表和 23 项 physical HA gate。ADR-0145/0146 又增加默认关闭的 manager-only TLS 1.3 management process、可选双副本 operation 与 durable distributed quota;它仍缺全副本重启 keyset anti-rollback 和真实 IdP/live ingress,所以“受认证管理入口”保持“已孵化但生产失败关闭”。 diff --git a/docs/adr/ADR-0380-local-lost-run-retry-single-control-cadence.md b/docs/adr/ADR-0380-local-lost-run-retry-single-control-cadence.md new file mode 100644 index 00000000..26b47ff9 --- /dev/null +++ b/docs/adr/ADR-0380-local-lost-run-retry-single-control-cadence.md @@ -0,0 +1,75 @@ +# ADR-0380:Local lost Run retry 复用单一 execution-control cadence + +- 状态:Accepted +- 日期:2026-08-12 +- 关联 RFC:QL-RFC-0001 D-292 +- 前置决策:ADR-0021、ADR-0066、ADR-0067、ADR-0232、ADR-0366 + +## 上下文 + +Cluster 已消费 admission-time `RunRetryPolicy`,但 Edge/Standalone 的启动恢复只能把确定不再运行的本地 Attempt 与 Run 标记为 `lost`。此后没有生产 consumer 处理该状态:安全且显式启用的重试不会创建新 Attempt,禁用、unsafe 或耗尽的 Run 也不会终态关闭。 + +直接复制一套 Local retry policy、建立独立 lifecycle package,或为 retry 新增 timer/SQLite 连接都会形成不合理架构。纯 transition 本来就是 profile-neutral,只因最初由 Cluster 接入而使用 `Cluster*` 名称;本地运行时也已经拥有单 SQLite operation authority、`BEGIN IMMEDIATE` Run aggregate transaction 和唯一 execution-control timer。 + +## 决策 + +### 1. 通用契约留在现有 Runtime Core Run 域 + +不新增 workspace package,也不复制状态机。`runtime-core` 对既有纯 transition、page contract、coordinator 与错误发布 canonical `RunLostRetry*` 名称和 `run-lost-retry` subpath;早期 `ClusterRunLostRetry*` 名称继续兼容。新代码只能依赖 profile-neutral 名称。 + +该兼容层不创建第二份实现。后续在具备 call-graph-aware rename 工具和独立废弃周期后,才允许移动物理文件或删除旧名称;本切片不以文本替换破坏已有 Cluster consumer。 + +### 2. SQLite adapter 复用唯一 operation authority + +`LocalSqliteRunLostRetryRepository` 位于 `ql3-local-sqlite/src/run`,不形成单文件微包。它: + +- 只选择 runtime-owned、非 Workflow、无 cancellation intent、latest Attempt 为 `lost` 的 Run; +- `lost` 优先,`retry_wait` 仅在 durable `next_attempt_at_ms` 到期后进入候选; +- 候选发现是一次有界 authority 操作,每个候选在同一既有 `LocalSqliteRunRepository.transaction()` 中重读并原子 CAS Run、RetryPolicy,插入新 Attempt 与 Events; +- 已变化的候选返回 `raced`,响应丢失后的重放由当前 durable aggregate 自然收敛; +- 不拥有连接、cursor、timer、listener、watcher 或常驻内存队列。 + +没有 policy、未启用或 `maxAttempts<=1` 时以 `RUN_LOST_RETRY_DISABLED` 失败关闭;`safety=unknown` 以 `RUN_LOST_RETRY_UNSAFE` 失败关闭;耗尽以 `RUN_LOST_RETRY_EXHAUSTED` 失败关闭。只有 admission 时已证明 `idempotent|deduplicated` 的 policy 才能先进入 `retry_wait`,到期后创建一个全新、无执行 handle/lease 的 claimed Attempt。 + +### 3. Local application 复用现有控制周期 + +lost retry 注入 `LocalExecutionControlLifecycle`,每轮顺序固定为: + +1. 消费内存中的 completion 通知; +2. 扫描 cancellation/deadline control; +3. 执行一页 lost retry; +4. 到慢周期时清理 completion receipt 与 Run log Artifact。 + +启动顺序仍为 startup recovery → 首次 execution-control cycle → scheduler。因而启动恢复刚写入的 `lost` 会在 Scheduler 首轮前进入安全 retry/终态。停机的 control drain 不再创建 retry work。 + +Edge 每轮最多 2 条,沿用 5 秒 control cadence;Standalone 每轮最多 16 条,沿用 1 秒 cadence。两者均复用已有 `unref` timer 和 SQLite authority,没有增加低配路由设备的空闲连接、常驻 sidecar 或独立唤醒源。 + +### 4. 明确不包含人工重试产品入口 + +本 ADR 只消费 admission 时冻结的自动 retry policy,不授予用户新的 mutation authority。`run.retry` Policy permission、人工 recovery API/CLI/UI、强认证、rate limit、审计与“对失败 Run 手工重跑”的产品语义仍需独立 ADR;不得把自动 lost retry 误报为人工恢复完成。 + +## 验收 + +- Runtime Core 验证通用名称与旧 Cluster 名称指向同一实现; +- SQLite 真库验证安全 lost → retry_wait → 新 Attempt 的原子闭环、禁用 policy 失败关闭、页上限、`hasMore` 与重放不重复; +- Local lifecycle 验证 control → lost retry → cleanup 顺序及重叠 `runOnce` 合并; +- Local application 真启动验证 startup recovery 后、Scheduler 前完成首次 lost retry; +- 受影响 package build/test、完整 QL3 package/backend 回归、dependency/package/Edge import audit 必须通过后才允许阶段性提交; +- package 数、migration 数、表/索引数、timer 数和 SQLite connection 数不得增加。 + +### 验收证据(2026-08-12) + +- `pnpm run test:packages:ql3`:18 个 QL3 package 全量清理、构建与测试通过,退出码 0; +- backend 回归:1165 项中 1163 通过、2 项按环境条件跳过、0 失败; +- package boundary audit:18 个 package、1055 个 source file、1037 个 nested source file,`singleSourcePackages=[]`、`shallowSourcePackages=[]`; +- cluster dependency audit:`findings=[]`;Edge import audit:121 个 imported module,root dependency 与 import 违规均为空; +- 本切片未新增 package、migration、表、索引、timer 或 SQLite connection。 + +## 被否决的替代方案 + +1. **新增 `ql3-local-retry` package**:只有一个 SQLite adapter,继续放大 package 碎片化,拒绝。 +2. **复制 Cluster transition 为 Local 版本**:会让安全与终态规则漂移,拒绝。 +3. **独立 retry timer**:增加路由设备唤醒、竞态和 shutdown 故障域,拒绝。 +4. **启动时递归清空全部 lost backlog**:无法约束启动延迟与写放大,拒绝。 +5. **`safety=unknown` 也自动重试**:可能复制不可逆外部副作用,拒绝。 +6. **把人工 retry API 一并塞入本切片**:认证、Policy、审计与新 mutation 语义未冻结,拒绝。 diff --git a/docs/adr/README.md b/docs/adr/README.md index 372c7b32..c3a3f7b4 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -383,6 +383,7 @@ | [ADR-0377](./ADR-0377-profile-aware-run-attempt-log-range-read.md) | Profile-aware Run Attempt 日志 Range 读取 | Accepted | | [ADR-0378](./ADR-0378-local-run-attempt-log-retention-and-tombstones.md) | Local Run Attempt 日志有界保留与 durable tombstone | Accepted | | [ADR-0379](./ADR-0379-cluster-run-attempt-log-retention.md) | Cluster Run Attempt 日志多副本保留与条件删除 | Accepted | +| [ADR-0380](./ADR-0380-local-lost-run-retry-single-control-cadence.md) | Local lost Run retry 复用单一 execution-control cadence | Accepted | ## 规则 diff --git a/packages/ql3-local-admin/src/adopted-profile/localAdoptedProfile.ts b/packages/ql3-local-admin/src/adopted-profile/localAdoptedProfile.ts index 525aa0f0..a5f902f9 100644 --- a/packages/ql3-local-admin/src/adopted-profile/localAdoptedProfile.ts +++ b/packages/ql3-local-admin/src/adopted-profile/localAdoptedProfile.ts @@ -88,6 +88,7 @@ export type LocalAdoptedProfileBootstrapResult = readonly executionControl: ReadyLocalStorage['executionControl']; readonly completionReceipts: ReadyLocalStorage['completionReceipts']; readonly runAttemptLogRetention: ReadyLocalStorage['runAttemptLogRetention']; + readonly runLostRetry: ReadyLocalStorage['runLostRetry']; readonly localSecrets: LocalSecretEnvelopeRepository; readonly localSecretAdministration: LocalSecretAdministrationRepository; readonly projectPolicy: ProjectPolicyRepository; @@ -233,6 +234,7 @@ export async function bootstrapLocalAdoptedProfileStorage( executionControl: readyStorage.executionControl, completionReceipts: readyStorage.completionReceipts, runAttemptLogRetention: readyStorage.runAttemptLogRetention, + runLostRetry: readyStorage.runLostRetry, localSecrets: readyStorage.localSecrets, localSecretAdministration: readyStorage.localSecretAdministration, projectPolicy: readyStorage.projectPolicy, diff --git a/packages/ql3-local-application/src/application-runtime/activation.ts b/packages/ql3-local-application/src/application-runtime/activation.ts index 9bfca581..6566d70c 100644 --- a/packages/ql3-local-application/src/application-runtime/activation.ts +++ b/packages/ql3-local-application/src/application-runtime/activation.ts @@ -73,6 +73,7 @@ const EXECUTION_CONTROL_POLICIES = Object.freeze({ cleanupPageSize: 8, controlIntervalMs: 5_000, controlPageSize: 4, + lostRetryPageSize: 2, maxDrainPages: 2, retentionMs: 24 * 60 * 60_000, artifactNormalRetentionMs: 7 * 24 * 60 * 60_000, @@ -87,6 +88,7 @@ const EXECUTION_CONTROL_POLICIES = Object.freeze({ cleanupPageSize: 32, controlIntervalMs: 1_000, controlPageSize: 32, + lostRetryPageSize: 16, maxDrainPages: 8, retentionMs: 60 * 60_000, artifactNormalRetentionMs: 30 * 24 * 60 * 60_000, @@ -345,6 +347,8 @@ export async function bootstrapLocalApplication( stopTimeoutMs: executionPolicy.stopTimeoutMs, maxDrainPages: executionPolicy.maxDrainPages, artifactRetention, + lostRetry: storage.runLostRetry, + lostRetryPageSize: executionPolicy.lostRetryPageSize, onDiagnostic: async (error) => { if (error === undefined) return; await bestEffortAudit(options, { diff --git a/packages/ql3-local-application/test/activation.test.cjs b/packages/ql3-local-application/test/activation.test.cjs index 47bb383b..b8392781 100644 --- a/packages/ql3-local-application/test/activation.test.cjs +++ b/packages/ql3-local-application/test/activation.test.cjs @@ -87,6 +87,8 @@ const RECEIPT_ATTEMPT_ID = '019f70c0-0000-7000-8000-000000000002'; const RECEIPT_TOKEN = 'A'.repeat(32); const CLEANUP_RUN_ID = '019f70c0-0000-7000-8000-000000000011'; const CLEANUP_ATTEMPT_ID = '019f70c0-0000-7000-8000-000000000012'; +const LOST_RETRY_RUN_ID = '019f70c0-0000-7000-8000-000000000021'; +const LOST_RETRY_ATTEMPT_ID = '019f70c0-0000-7000-8000-000000000022'; const WORKFLOW_CANCELLATION_CREDENTIAL_ID = 'application-workflow-owner'; const WORKFLOW_CANCELLATION_PEPPER_KEY_ID = 'application-workflow-owner-v1'; const WORKFLOW_CANCELLATION_PEPPER_BYTES = Buffer.alloc(32, 141); @@ -478,6 +480,58 @@ function insertActiveTargetRun(value, id, status = 'running') { target.close(); } +function insertSafeLostTargetRun(value) { + const now = Date.now(); + const target = new DatabaseSync(value.targetPath); + target.exec('BEGIN IMMEDIATE'); + try { + target + .prepare( + `INSERT INTO "Runs" ( + id, project_id, task_id, task_revision, trigger_type, + execution_origin, execution_owner, status, version, event_sequence, + priority, created_at_ms, queued_at_ms, started_at_ms, + error_code, error_summary + ) VALUES (?, 'default', 'task-lost-retry', 'revision-lost-retry', + 'manual', 'manual', 'runtime', 'lost', 1, 0, 0, ?, ?, ?, + 'LOCAL_RECOVERY_EXECUTION_NOT_RUNNING', 'lost')`, + ) + .run(LOST_RETRY_RUN_ID, now - 300, now - 250, now - 200); + target + .prepare( + `INSERT INTO "RunAttempts" ( + id, run_id, attempt, status, executor_type, callback_sequence, + created_at_ms, started_at_ms, finished_at_ms, + error_code, error_summary + ) VALUES (?, ?, 1, 'lost', 'local_process', 0, ?, ?, ?, + 'LOCAL_RECOVERY_EXECUTION_NOT_RUNNING', 'lost')`, + ) + .run( + LOST_RETRY_ATTEMPT_ID, + LOST_RETRY_RUN_ID, + now - 250, + now - 200, + now - 100, + ); + target + .prepare( + `INSERT INTO "RunRetryPolicies" ( + run_id, max_attempts, retry_on_lost, safety, + backoff_base_ms, backoff_max_ms, next_attempt_at_ms, + version, created_at_ms, updated_at_ms + ) VALUES (?, 3, 1, 'idempotent', 86400000, 86400000, + NULL, 0, ?, ?)`, + ) + .run(LOST_RETRY_RUN_ID, now - 300, now - 300); + target.exec('COMMIT'); + } catch (error) { + if (target.isTransaction) target.exec('ROLLBACK'); + throw error; + } finally { + target.close(); + } +} + function insertManyActiveTargetRuns(value, count) { const target = new DatabaseSync(value.targetPath); const statement = target.prepare( @@ -2179,6 +2233,39 @@ test('durable Run candidates block lifecycle activation', async (t) => { assertSourceWritable(value, 2); }); +test('reconciles a safe lost Run before the first Local scheduler pass', async (t) => { + const value = await prepare(t, 'edge'); + insertSafeLostTargetRun(value); + const audits = []; + const result = await bootstrapLocalApplication( + options(value, { + applicationAudit: (record) => audits.push(record), + }), + ); + + assert.equal(result.status, 'active'); + assert.equal( + (await result.runs.findRunById(LOST_RETRY_RUN_ID)).status, + 'retry_wait', + ); + const reconciled = audits.find( + (record) => record.state === 'receipts_reconciled', + ); + assert.deepEqual(reconciled.executionControl.lostRetry, { + scanned: 1, + scheduled: 1, + requeued: 0, + failed: 0, + raced: 0, + hasMore: false, + }); + assert.equal( + (await result.runs.findLatestAttemptByRunId(LOST_RETRY_RUN_ID)).attempt, + 1, + ); + assert.equal(await result.stop(), 'stopped'); +}); + test('startup recovery candidate overflow fails closed at the hard page bound', async (t) => { const value = await prepare(t, 'edge'); insertManyActiveTargetRuns(value, MAX_LOCAL_RUN_RECOVERY_ITEMS + 1); diff --git a/packages/ql3-local-execution/src/control/lifecycle.ts b/packages/ql3-local-execution/src/control/lifecycle.ts index 1f77537d..6c0dbea9 100644 --- a/packages/ql3-local-execution/src/control/lifecycle.ts +++ b/packages/ql3-local-execution/src/control/lifecycle.ts @@ -1,6 +1,11 @@ import type { LocalCompletionReceiptJournalCursor } from '@qinglong/runtime-core/local-completion-receipt-journal'; import { assertLocalExecutionControlLimit } from '@qinglong/runtime-core/local-execution-control'; import type { RunAttemptLogRetentionSweepSummary } from '@qinglong/runtime-core/run-attempt-log-retention'; +import { + RunLostRetryCoordinator, + type RunLostRetryPageResult, + type RunLostRetryRepository, +} from '@qinglong/runtime-core/run-lost-retry'; import type { LocalCompletionReceiptCleanupScanner, LocalCompletionReceiptCleanupSummary, @@ -25,6 +30,8 @@ export interface LocalExecutionControlLifecycleOptions { readonly artifactRetention?: Readonly<{ sweep(): Promise; }>; + readonly lostRetry?: RunLostRetryRepository; + readonly lostRetryPageSize?: number; readonly clock?: { now(): number }; readonly onDiagnostic?: ( error: unknown, @@ -38,6 +45,7 @@ export interface LocalExecutionControlCycleSummary { readonly control: LocalExecutionControlScanSummary; readonly cleanup?: LocalCompletionReceiptCleanupSummary; readonly artifactRetention?: RunAttemptLogRetentionSweepSummary; + readonly lostRetry?: Readonly; } export interface LocalExecutionControlStopSummary { @@ -49,6 +57,7 @@ export interface LocalExecutionControlStopSummary { export class LocalExecutionControlLifecycle { private readonly clock: { now(): number }; private readonly maxNotifications: number; + private readonly lostRetry: RunLostRetryCoordinator | undefined; private timer: NodeJS.Timeout | undefined; private inFlight: Promise | undefined; private stopPromise: Promise | undefined; @@ -122,6 +131,18 @@ export class LocalExecutionControlLifecycle { ) { throw new TypeError('Local Artifact retention lifecycle is invalid'); } + if ( + (options.lostRetry === undefined) !== + (options.lostRetryPageSize === undefined) + ) { + throw new TypeError('Local Run lost retry lifecycle is invalid'); + } + this.lostRetry = + options.lostRetry && options.lostRetryPageSize !== undefined + ? new RunLostRetryCoordinator(options.lostRetry, { + pageSize: options.lostRetryPageSize, + }) + : undefined; this.clock = options.clock ?? { now: Date.now }; } @@ -238,6 +259,7 @@ export class LocalExecutionControlLifecycle { : { cursor: this.controlCursor }), }); this.controlCursor = control.truncated ? control.nextCursor : undefined; + const lostRetry = await this.lostRetry?.reconcile(); const now = this.clock.now(); if (!Number.isSafeInteger(now) || now < 0) { throw new RangeError( @@ -266,6 +288,7 @@ export class LocalExecutionControlLifecycle { completions: completion.processed, completionFailures: completion.failed, control, + ...(lostRetry === undefined ? {} : { lostRetry }), ...(cleanup === undefined ? {} : { cleanup }), ...(artifactRetention === undefined ? {} : { artifactRetention }), }); diff --git a/packages/ql3-local-execution/test/controlLostRetry.test.cjs b/packages/ql3-local-execution/test/controlLostRetry.test.cjs new file mode 100644 index 00000000..ab1d2c92 --- /dev/null +++ b/packages/ql3-local-execution/test/controlLostRetry.test.cjs @@ -0,0 +1,92 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const { test } = require('node:test'); + +const { LocalExecutionControlLifecycle } = require('../dist/control'); + +test('runs lost retry inside the existing control cadence before cleanup', async () => { + const calls = []; + const lifecycle = new LocalExecutionControlLifecycle( + { + async process() { + throw new Error('no completion notifications are expected'); + }, + }, + { + async scan() { + calls.push('control'); + return { + observedAtMs: 100, + scanned: 0, + terminal: 0, + deferred: 0, + failed: 0, + truncated: false, + }; + }, + async drain() { + return { + observedAtMs: 100, + scanned: 0, + terminal: 0, + deferred: 0, + failed: 0, + remaining: 0, + pages: 0, + truncated: false, + }; + }, + }, + { + async scan() { + calls.push('cleanup'); + return { + scanned: 0, + removed: 0, + missing: 0, + deferred: 0, + failed: 0, + truncated: false, + }; + }, + }, + { + intervalMs: 5_000, + pageSize: 4, + cleanupIntervalMs: 60_000, + cleanupPageSize: 4, + stopTimeoutMs: 1_000, + maxDrainPages: 1, + lostRetry: { + async reconcilePage(command) { + calls.push('lost_retry'); + assert.deepEqual(command, { limit: 2 }); + return { + scanned: 1, + scheduled: 1, + requeued: 0, + failed: 0, + raced: 0, + hasMore: false, + }; + }, + }, + lostRetryPageSize: 2, + clock: { now: () => 100 }, + }, + ); + + const first = lifecycle.runOnce(true); + const second = lifecycle.runOnce(true); + assert.equal(first, second); + assert.deepEqual((await first).lostRetry, { + scanned: 1, + scheduled: 1, + requeued: 0, + failed: 0, + raced: 0, + hasMore: false, + }); + assert.deepEqual(calls, ['control', 'lost_retry', 'cleanup']); +}); diff --git a/packages/ql3-local-sqlite/package.json b/packages/ql3-local-sqlite/package.json index c2270d38..bba6d4f2 100644 --- a/packages/ql3-local-sqlite/package.json +++ b/packages/ql3-local-sqlite/package.json @@ -250,6 +250,11 @@ "require": "./dist/run/runAttemptLogRetentionRepository.js", "default": "./dist/run/runAttemptLogRetentionRepository.js" }, + "./run-lost-retry": { + "types": "./dist/run/runLostRetryRepository.d.ts", + "require": "./dist/run/runLostRetryRepository.js", + "default": "./dist/run/runLostRetryRepository.js" + }, "./trigger-administration": { "types": "./dist/scheduling/triggerAdministration.d.ts", "require": "./dist/scheduling/triggerAdministration.js", diff --git a/packages/ql3-local-sqlite/src/profile/localProfile.ts b/packages/ql3-local-sqlite/src/profile/localProfile.ts index d35ffb25..95cfa88a 100644 --- a/packages/ql3-local-sqlite/src/profile/localProfile.ts +++ b/packages/ql3-local-sqlite/src/profile/localProfile.ts @@ -47,6 +47,7 @@ export type LocalProfileStorageBootstrapResult = readonly executionControl: LocalSqliteRuntimeDatabase['executionControl']; readonly completionReceipts: LocalSqliteRuntimeDatabase['completionReceipts']; readonly runAttemptLogRetention: LocalSqliteRuntimeDatabase['runAttemptLogRetention']; + readonly runLostRetry: LocalSqliteRuntimeDatabase['runLostRetry']; readonly localSecrets: LocalSqliteRuntimeDatabase['localSecrets']; readonly localSecretAdministration: LocalSqliteRuntimeDatabase['localSecretAdministration']; readonly projectPolicy: LocalSqliteRuntimeDatabase['projectPolicy']; @@ -140,6 +141,7 @@ export async function bootstrapLocalProfileStorage( executionControl: database.executionControl, completionReceipts: database.completionReceipts, runAttemptLogRetention: database.runAttemptLogRetention, + runLostRetry: database.runLostRetry, localSecrets: database.localSecrets, localSecretAdministration: database.localSecretAdministration, projectPolicy: database.projectPolicy, diff --git a/packages/ql3-local-sqlite/src/run/runLostRetryRepository.ts b/packages/ql3-local-sqlite/src/run/runLostRetryRepository.ts new file mode 100644 index 00000000..673825cd --- /dev/null +++ b/packages/ql3-local-sqlite/src/run/runLostRetryRepository.ts @@ -0,0 +1,251 @@ +import { randomUUID } from 'node:crypto'; + +import { + RunLostRetryUnavailableError, + buildRunLostRetryTransition, + normalizeRunLostRetryPageCommand, + normalizeRunLostRetryPageResult, + type RunLostRetryDisposition, + type RunLostRetryPageCommand, + type RunLostRetryPageResult, + type RunLostRetryRepository, + type RunLostRetryTransition, +} from '@qinglong/runtime-core/run-lost-retry'; +import type { + RunRecord, + RunRepositoryTransaction, + RunRetryPolicyRecord, +} from '@qinglong/runtime-core/run-repository'; + +import { LocalSqliteOperationAuthority } from '../authority/operationAuthority'; +import { LocalSqliteRunRepository } from './runRepository'; + +type Row = Record; + +interface Candidate { + readonly runId: string; + readonly attemptId: string; +} + +function identifier(row: Row, key: string): string { + const value = row[key]; + if ( + typeof value !== 'string' || + value.length < 1 || + value.length > 128 || + /[\u0000-\u001f\u007f]/.test(value) + ) { + throw new TypeError(`Local SQLite lost retry ${key} is invalid`); + } + return value; +} + +function generatedId(factory: () => string): string { + const value = factory(); + if ( + typeof value !== 'string' || + value.length < 1 || + value.length > 36 || + /[\u0000-\u001f\u007f]/.test(value) + ) { + throw new TypeError('Local SQLite lost retry generated ID is invalid'); + } + return value; +} + +/** + * Local Profiles share one SQLite authority. Candidate discovery and each + * BEGIN IMMEDIATE aggregate mutation are bounded, serialized operations; this + * repository owns no connection, timer, cursor, or background task. + */ +export class LocalSqliteRunLostRetryRepository + implements RunLostRetryRepository +{ + constructor( + private readonly authority: LocalSqliteOperationAuthority, + private readonly runs: LocalSqliteRunRepository, + private readonly createId: () => string = randomUUID, + private readonly clock: { now(): number } = { now: Date.now }, + ) { + if ( + !(authority instanceof LocalSqliteOperationAuthority) || + !(runs instanceof LocalSqliteRunRepository) || + typeof createId !== 'function' || + typeof clock?.now !== 'function' + ) { + throw new TypeError('Local SQLite lost retry repository is invalid'); + } + } + + async reconcilePage( + input: Readonly, + ): Promise> { + const command = normalizeRunLostRetryPageCommand(input); + try { + const candidates = await this.listCandidates(command.limit + 1); + const page = candidates.slice(0, command.limit); + const counts: Record = { + scheduled: 0, + requeued: 0, + failed_disabled: 0, + failed_unsafe: 0, + failed_exhausted: 0, + raced: 0, + }; + for (const candidate of page) { + counts[await this.reconcileCandidate(candidate)] += 1; + } + return normalizeRunLostRetryPageResult( + { + scanned: page.length, + scheduled: counts.scheduled, + requeued: counts.requeued, + failed: + counts.failed_disabled + + counts.failed_unsafe + + counts.failed_exhausted, + raced: counts.raced, + hasMore: candidates.length > command.limit, + }, + command.limit, + ); + } catch (error) { + if (error instanceof RunLostRetryUnavailableError) throw error; + throw new RunLostRetryUnavailableError({ cause: error }); + } + } + + private listCandidates(limit: number): Promise { + return this.authority.enqueue( + async () => { + const observedAtMs = this.observedAtMs(); + const rows = this.authority.client + .prepare( + `SELECT run."id" AS "runId", attempt."id" AS "attemptId" + FROM "Runs" AS run + JOIN "RunAttempts" AS attempt + ON attempt."run_id" = run."id" + AND attempt."attempt" = ( + SELECT MAX(latest."attempt") + FROM "RunAttempts" AS latest + WHERE latest."run_id" = run."id" + ) + LEFT JOIN "RunRetryPolicies" AS policy + ON policy."run_id" = run."id" + WHERE run."execution_owner" = 'runtime' + AND run."trigger_type" <> 'plugin_package_workflow' + AND run."cancel_requested_at_ms" IS NULL + AND attempt."status" = 'lost' + AND ( + run."status" = 'lost' + OR ( + run."status" = 'retry_wait' + AND policy."next_attempt_at_ms" IS NOT NULL + AND policy."next_attempt_at_ms" <= ? + ) + ) + ORDER BY + CASE WHEN run."status" = 'lost' THEN 0 + ELSE policy."next_attempt_at_ms" END, + run."id" + LIMIT ?`, + ) + .all(observedAtMs, limit) as Row[]; + if (rows.length > limit) { + throw new TypeError('Local SQLite lost retry exceeded its page size'); + } + return Object.freeze( + rows.map((row) => + Object.freeze({ + runId: identifier(row, 'runId'), + attemptId: identifier(row, 'attemptId'), + }), + ), + ); + }, + () => new RunLostRetryUnavailableError(), + ); + } + + private async reconcileCandidate( + candidate: Readonly, + ): Promise { + return this.runs.transaction(async (transaction) => { + const run = await transaction.findRunById(candidate.runId); + const attempt = await transaction.findLatestAttemptByRunId( + candidate.runId, + ); + const policy = await transaction.findRetryPolicyByRunId(candidate.runId); + if ( + !run || + !attempt || + attempt.id !== candidate.attemptId || + attempt.status !== 'lost' || + run.executionOwner !== 'runtime' || + run.triggerType === 'plugin_package_workflow' || + run.cancelRequestedAtMs !== undefined || + (run.status !== 'lost' && run.status !== 'retry_wait') || + (run.status === 'retry_wait' && + (policy?.nextAttemptAtMs === undefined || + policy.nextAttemptAtMs > this.observedAtMs())) + ) { + return 'raced'; + } + const transition = buildRunLostRetryTransition({ + run, + attempt, + policy, + observedAtMs: this.observedAtMs(), + runEventId: generatedId(this.createId), + ...(run.status === 'retry_wait' + ? { + attemptId: generatedId(this.createId), + attemptEventId: generatedId(this.createId), + } + : {}), + }); + await this.persistTransition(transaction, run, policy, transition); + return transition.disposition; + }); + } + + private observedAtMs(): number { + const value = this.clock.now(); + if (!Number.isSafeInteger(value) || value < 0) { + throw new TypeError('Local SQLite lost retry clock is invalid'); + } + return value; + } + + private async persistTransition( + transaction: RunRepositoryTransaction, + currentRun: Readonly, + currentPolicy: Readonly | null, + transition: Readonly, + ): Promise { + let version = currentRun.version; + for (const run of transition.runTransitions) { + if (!(await transaction.compareAndSetRun(run, version))) { + throw new TypeError('Local SQLite lost retry Run fence changed'); + } + version = run.version; + } + if (transition.policy && transition.policy !== currentPolicy) { + if ( + !currentPolicy || + !(await transaction.compareAndSetRetryPolicy( + transition.policy, + currentPolicy.version, + )) + ) { + throw new TypeError('Local SQLite lost retry policy fence changed'); + } + } + if (transition.attempt) { + await transaction.insertAttempt(transition.attempt); + } + for (const event of transition.events) { + await transaction.appendEvent(event); + } + } +} diff --git a/packages/ql3-local-sqlite/src/runtime/runtimeDatabase.ts b/packages/ql3-local-sqlite/src/runtime/runtimeDatabase.ts index 9b169cbb..561c34b9 100644 --- a/packages/ql3-local-sqlite/src/runtime/runtimeDatabase.ts +++ b/packages/ql3-local-sqlite/src/runtime/runtimeDatabase.ts @@ -63,6 +63,7 @@ import type { LocalSqlitePluginPackageWorkflowFrontierRepository } from '../plug import type { LocalSqlitePluginPackageWorkflowTaskAttemptAdmissionRepository } from '../plugin-package/workflow/pluginPackageWorkflowTaskAttemptAdmissionRepository'; import type { LocalSqlitePluginPackageWorkflowCancellationConvergenceRepository } from '../plugin-package/workflow/pluginPackageWorkflowCancellationConvergenceRepository'; import { LocalSqliteRunAttemptLogRetentionRepository } from '../run/runAttemptLogRetentionRepository'; +import { LocalSqliteRunLostRetryRepository } from '../run/runLostRetryRepository'; export interface LocalSqliteRuntimeDependencies { readonly taskSpecSemanticRegistry?: TaskSpecSemanticRegistry; @@ -99,6 +100,7 @@ export interface LocalSqliteRuntimeDatabase { readonly executionControl: LocalExecutionControlSource; readonly completionReceipts: LocalCompletionReceiptJournal; readonly runAttemptLogRetention: LocalSqliteRunAttemptLogRetentionRepository; + readonly runLostRetry: LocalSqliteRunLostRetryRepository; readonly localSecrets: LocalSecretEnvelopeRepository; readonly localSecretAdministration: LocalSecretAdministrationRepository; readonly projectPolicy: ProjectPolicyRepository; @@ -183,6 +185,10 @@ export async function openLocalSqliteRuntimeDatabase( const ownerPepper = new LocalSqliteOwnerPepperRepository(authority); const runAttemptLogRetention = new LocalSqliteRunAttemptLogRetentionRepository(authority); + const runLostRetry = new LocalSqliteRunLostRetryRepository( + authority, + runRepository, + ); let pluginPackageInstallsPromise: | Promise | undefined; @@ -237,6 +243,7 @@ export async function openLocalSqliteRuntimeDatabase( executionControl: runRuntimeCapabilities.executionControl, completionReceipts: runRuntimeCapabilities.completionReceipts, runAttemptLogRetention, + runLostRetry, localSecrets: securityAuthority, localSecretAdministration: securityAuthority, projectPolicy, diff --git a/packages/ql3-local-sqlite/test/runLostRetryRepository.test.cjs b/packages/ql3-local-sqlite/test/runLostRetryRepository.test.cjs new file mode 100644 index 00000000..35fd35ec --- /dev/null +++ b/packages/ql3-local-sqlite/test/runLostRetryRepository.test.cjs @@ -0,0 +1,149 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const { test } = require('node:test'); + +const { + migrateLocalSqlitePath, + openLocalSqliteRuntimeDatabase, +} = require('../dist'); + +async function fixture(t) { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'ql3-lost-retry-')); + const databasePath = path.join(directory, 'qinglong3.sqlite'); + await migrateLocalSqlitePath({ databasePath, profile: 'edge' }); + const runtime = await openLocalSqliteRuntimeDatabase({ + databasePath, + profile: 'edge', + }); + t.after(async () => { + await runtime.close(); + fs.rmSync(directory, { recursive: true, force: true }); + }); + return runtime; +} + +async function insertLostRun(runtime, suffix, retryPolicy) { + const runId = `run-lost-${suffix}`; + const attemptId = `attempt-lost-${suffix}`; + await runtime.runRepository.transaction(async (transaction) => { + await transaction.insertRun({ + id: runId, + projectId: 'default', + taskId: `task-${suffix}`, + taskRevision: 'revision-1', + triggerType: 'manual', + executionOrigin: 'manual', + executionOwner: 'runtime', + status: 'lost', + version: 1, + eventSequence: 1, + priority: 0, + createdAtMs: 100, + queuedAtMs: 110, + startedAtMs: 120, + errorCode: 'LOCAL_RECOVERY_EXECUTION_NOT_RUNNING', + errorSummary: 'lost', + }); + await transaction.insertAttempt({ + id: attemptId, + runId, + attempt: 1, + status: 'lost', + executorType: 'local_process', + callbackSequence: 0, + createdAtMs: 110, + startedAtMs: 120, + finishedAtMs: 130, + errorCode: 'LOCAL_RECOVERY_EXECUTION_NOT_RUNNING', + errorSummary: 'lost', + }); + if (retryPolicy) { + await transaction.insertRetryPolicy({ + runId, + maxAttempts: 3, + retryOnLost: true, + safety: 'idempotent', + backoffBaseMs: 0, + backoffMaxMs: 0, + version: 0, + createdAtMs: 100, + updatedAtMs: 100, + }); + } + }); + return { runId, attemptId }; +} + +test('atomically schedules and requeues one safe Local lost Run exactly once', async (t) => { + const runtime = await fixture(t); + const ids = await insertLostRun(runtime, 'safe', true); + + assert.deepEqual(await runtime.runLostRetry.reconcilePage({ limit: 1 }), { + scanned: 1, + scheduled: 1, + requeued: 0, + failed: 0, + raced: 0, + hasMore: false, + }); + assert.equal( + (await runtime.runRepository.findRunById(ids.runId)).status, + 'retry_wait', + ); + + assert.deepEqual(await runtime.runLostRetry.reconcilePage({ limit: 1 }), { + scanned: 1, + scheduled: 0, + requeued: 1, + failed: 0, + raced: 0, + hasMore: false, + }); + const run = await runtime.runRepository.findRunById(ids.runId); + const attempt = await runtime.runRepository.findLatestAttemptByRunId( + ids.runId, + ); + assert.equal(run.status, 'queued'); + assert.equal(run.version, 4); + assert.equal(attempt.attempt, 2); + assert.equal(attempt.status, 'claimed'); + assert.equal(attempt.executorType, 'local_process'); + assert.deepEqual( + (await runtime.runRepository.listEvents(ids.runId)).map( + (event) => event.type, + ), + ['run.retry_wait', 'run.queued', 'attempt.claimed'], + ); + assert.equal( + (await runtime.runLostRetry.reconcilePage({ limit: 1 })).scanned, + 0, + ); +}); + +test('fails closed without an admitted retry policy and preserves a bounded page', async (t) => { + const runtime = await fixture(t); + const first = await insertLostRun(runtime, 'disabled-a', false); + await insertLostRun(runtime, 'disabled-b', false); + + const result = await runtime.runLostRetry.reconcilePage({ limit: 1 }); + assert.deepEqual(result, { + scanned: 1, + scheduled: 0, + requeued: 0, + failed: 1, + raced: 0, + hasMore: true, + }); + assert.equal( + (await runtime.runRepository.findRunById(first.runId)).errorCode, + 'RUN_LOST_RETRY_DISABLED', + ); + await assert.rejects( + runtime.runLostRetry.reconcilePage({ limit: 65 }), + /page size/, + ); +}); diff --git a/packages/ql3-runtime-core/package.json b/packages/ql3-runtime-core/package.json index 89b2b528..384bf6f4 100644 --- a/packages/ql3-runtime-core/package.json +++ b/packages/ql3-runtime-core/package.json @@ -218,6 +218,12 @@ "run": [ "dist/run/run.d.ts" ], + "run-lost-retry": [ + "dist/run/clusterRunLostRetry.d.ts" + ], + "cluster-run-lost-retry": [ + "dist/run/clusterRunLostRetry.d.ts" + ], "project-run-list": [ "dist/run/projectRunList.d.ts" ], @@ -265,6 +271,16 @@ "require": "./dist/run/runRepositoryContract.js", "default": "./dist/run/runRepositoryContract.js" }, + "./run-lost-retry": { + "types": "./dist/run/clusterRunLostRetry.d.ts", + "require": "./dist/run/clusterRunLostRetry.js", + "default": "./dist/run/clusterRunLostRetry.js" + }, + "./cluster-run-lost-retry": { + "types": "./dist/run/clusterRunLostRetry.d.ts", + "require": "./dist/run/clusterRunLostRetry.js", + "default": "./dist/run/clusterRunLostRetry.js" + }, "./project-run-list": { "types": "./dist/run/projectRunList.d.ts", "require": "./dist/run/projectRunList.js", diff --git a/packages/ql3-runtime-core/src/run/clusterRunLostRetry.ts b/packages/ql3-runtime-core/src/run/clusterRunLostRetry.ts index d2492c51..28301b40 100644 --- a/packages/ql3-runtime-core/src/run/clusterRunLostRetry.ts +++ b/packages/ql3-runtime-core/src/run/clusterRunLostRetry.ts @@ -1,12 +1,5 @@ -import type { - RunAttemptRecord, - RunEventRecord, - RunRecord, -} from './run'; -import { - runRetryDelayMs, - type RunRetryPolicyRecord, -} from './runRetryPolicy'; +import type { RunAttemptRecord, RunEventRecord, RunRecord } from './run'; +import { runRetryDelayMs, type RunRetryPolicyRecord } from './runRetryPolicy'; export const MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE = 64; @@ -62,7 +55,7 @@ export class ClusterRunLostRetryUnavailableError extends Error { readonly code = 'CLUSTER_RUN_LOST_RETRY_UNAVAILABLE'; constructor(options?: ErrorOptions) { - super('Cluster Run lost retry is unavailable', options); + super('Run lost retry is unavailable', options); this.name = 'ClusterRunLostRetryUnavailableError'; } } @@ -71,7 +64,7 @@ export class InvalidClusterRunLostRetryTransitionError extends TypeError { readonly code = 'CLUSTER_RUN_LOST_RETRY_INVALID'; constructor(message: string) { - super(`Cluster Run lost retry is invalid: ${message}`); + super(`Run lost retry is invalid: ${message}`); this.name = 'InvalidClusterRunLostRetryTransitionError'; } } @@ -182,10 +175,7 @@ function transitionTime( function finish( input: Readonly, atMs: number, - disposition: - | 'failed_disabled' - | 'failed_unsafe' - | 'failed_exhausted', + disposition: 'failed_disabled' | 'failed_unsafe' | 'failed_exhausted', error: Readonly<{ code: string; summary: string }>, ): Readonly { const run = reserve(input.run, 'failed', atMs, error); @@ -209,7 +199,7 @@ function finish( run, input.attempt.id, 'run.failed', - `cluster-lost-retry:${disposition}:${input.attempt.id}`, + `run-lost-retry:${disposition}:${input.attempt.id}`, atMs, { from_status: input.run.status, @@ -271,8 +261,11 @@ export function buildClusterRunLostRetryTransition( if (run.status === 'lost') { const nextAttemptAtMs = - Math.max(run.createdAtMs, attempt.createdAtMs, attempt.finishedAtMs ?? 0) + - runRetryDelayMs(policy, attempt.attempt); + Math.max( + run.createdAtMs, + attempt.createdAtMs, + attempt.finishedAtMs ?? 0, + ) + runRetryDelayMs(policy, attempt.attempt); if (!Number.isSafeInteger(nextAttemptAtMs)) { return invalid('next Attempt time overflowed'); } @@ -297,7 +290,7 @@ export function buildClusterRunLostRetryTransition( nextRun, attempt.id, 'run.retry_wait', - `cluster-lost-retry:scheduled:${attempt.id}`, + `run-lost-retry:scheduled:${attempt.id}`, atMs, { from_status: 'lost', @@ -313,10 +306,7 @@ export function buildClusterRunLostRetryTransition( }); } - if ( - policy.nextAttemptAtMs === undefined || - policy.nextAttemptAtMs > atMs - ) { + if (policy.nextAttemptAtMs === undefined || policy.nextAttemptAtMs > atMs) { return invalid('retry_wait policy is not due'); } const nextAttemptId = identifier('replacement Attempt ID', input.attemptId); @@ -352,7 +342,7 @@ export function buildClusterRunLostRetryTransition( queued, replacement.id, 'run.queued', - `cluster-lost-retry:queued:${replacement.id}`, + `run-lost-retry:queued:${replacement.id}`, atMs, { from_status: 'retry_wait', @@ -365,7 +355,7 @@ export function buildClusterRunLostRetryTransition( claimed, replacement.id, 'attempt.claimed', - `cluster-lost-retry:attempt-claimed:${replacement.id}`, + `run-lost-retry:attempt-claimed:${replacement.id}`, atMs, { attempt: replacement.attempt, @@ -387,11 +377,11 @@ export function normalizeClusterRunLostRetryPageCommand( Object.keys(value).length !== 1 || !Object.prototype.hasOwnProperty.call(value, 'limit') ) { - throw new TypeError('Cluster Run lost retry command is invalid'); + throw new TypeError('Run lost retry command is invalid'); } return Object.freeze({ limit: boundedInteger( - 'Cluster Run lost retry page size', + 'Run lost retry page size', value.limit, 1, MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE, @@ -410,40 +400,40 @@ export function normalizeClusterRunLostRetryPageResult( Object.keys(value).sort().join(',') !== 'failed,hasMore,raced,requeued,scanned,scheduled' ) { - throw new TypeError('Cluster Run lost retry result is invalid'); + throw new TypeError('Run lost retry result is invalid'); } const maximum = boundedInteger( - 'Cluster Run lost retry result limit', + 'Run lost retry result limit', limit, 1, MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE, ); const scanned = boundedInteger( - 'Cluster Run lost retry scanned count', + 'Run lost retry scanned count', value.scanned, 0, maximum, ); const scheduled = boundedInteger( - 'Cluster Run lost retry scheduled count', + 'Run lost retry scheduled count', value.scheduled, 0, scanned, ); const requeued = boundedInteger( - 'Cluster Run lost retry requeued count', + 'Run lost retry requeued count', value.requeued, 0, scanned, ); const failed = boundedInteger( - 'Cluster Run lost retry failed count', + 'Run lost retry failed count', value.failed, 0, scanned, ); const raced = boundedInteger( - 'Cluster Run lost retry raced count', + 'Run lost retry raced count', value.raced, 0, scanned, @@ -452,7 +442,7 @@ export function normalizeClusterRunLostRetryPageResult( scheduled + requeued + failed + raced !== scanned || typeof value.hasMore !== 'boolean' ) { - throw new TypeError('Cluster Run lost retry result counts are invalid'); + throw new TypeError('Run lost retry result counts are invalid'); } return Object.freeze({ scanned, @@ -481,10 +471,10 @@ export class ClusterRunLostRetryCoordinator { typeof options !== 'object' || Array.isArray(options) ) { - throw new TypeError('Cluster Run lost retry coordinator is invalid'); + throw new TypeError('Run lost retry coordinator is invalid'); } this.pageSize = boundedInteger( - 'Cluster Run lost retry page size', + 'Run lost retry page size', options.pageSize ?? 16, 1, MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE, @@ -494,9 +484,7 @@ export class ClusterRunLostRetryCoordinator { reconcile(): Promise> { if (this.inFlight) return this.inFlight; const operation = Promise.resolve() - .then(() => - this.repository.reconcilePage({ limit: this.pageSize }), - ) + .then(() => this.repository.reconcilePage({ limit: this.pageSize })) .then((result) => normalizeClusterRunLostRetryPageResult(result, this.pageSize), ) @@ -511,3 +499,32 @@ export class ClusterRunLostRetryCoordinator { return operation; } } + +/** + * Profile-neutral names are the canonical API for new adapters. The Cluster + * names remain exported because early 3.0 incubator consumers used them before + * Local Profiles acquired the same reconciliation lifecycle. + */ +export const MAX_RUN_LOST_RETRY_PAGE_SIZE = + MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE; +export type RunLostRetryDisposition = ClusterRunLostRetryDisposition; +export type RunLostRetryPageCommand = ClusterRunLostRetryPageCommand; +export type RunLostRetryPageResult = ClusterRunLostRetryPageResult; +export type RunLostRetryRepository = ClusterRunLostRetryRepository; +export type RunLostRetryCoordinatorOptions = + ClusterRunLostRetryCoordinatorOptions; +export type RunLostRetryTransitionInput = ClusterRunLostRetryTransitionInput; +export type RunLostRetryTransition = ClusterRunLostRetryTransition; +export type RunLostRetryUnavailableError = ClusterRunLostRetryUnavailableError; +export type InvalidRunLostRetryTransitionError = + InvalidClusterRunLostRetryTransitionError; +export type RunLostRetryCoordinator = ClusterRunLostRetryCoordinator; +export const RunLostRetryUnavailableError = ClusterRunLostRetryUnavailableError; +export const InvalidRunLostRetryTransitionError = + InvalidClusterRunLostRetryTransitionError; +export const buildRunLostRetryTransition = buildClusterRunLostRetryTransition; +export const normalizeRunLostRetryPageCommand = + normalizeClusterRunLostRetryPageCommand; +export const normalizeRunLostRetryPageResult = + normalizeClusterRunLostRetryPageResult; +export const RunLostRetryCoordinator = ClusterRunLostRetryCoordinator; diff --git a/packages/ql3-runtime-core/test/runLostRetryProfileNeutral.test.cjs b/packages/ql3-runtime-core/test/runLostRetryProfileNeutral.test.cjs new file mode 100644 index 00000000..b50c8b0c --- /dev/null +++ b/packages/ql3-runtime-core/test/runLostRetryProfileNeutral.test.cjs @@ -0,0 +1,25 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const { test } = require('node:test'); + +const generic = require('../dist/run/clusterRunLostRetry'); + +test('publishes profile-neutral lost retry names without breaking Cluster consumers', () => { + assert.equal( + generic.MAX_RUN_LOST_RETRY_PAGE_SIZE, + generic.MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE, + ); + assert.equal( + generic.buildRunLostRetryTransition, + generic.buildClusterRunLostRetryTransition, + ); + assert.equal( + generic.RunLostRetryCoordinator, + generic.ClusterRunLostRetryCoordinator, + ); + assert.equal( + generic.RunLostRetryUnavailableError, + generic.ClusterRunLostRetryUnavailableError, + ); +}); diff --git a/test/back/ql3PackageBoundaryAudit.test.cjs b/test/back/ql3PackageBoundaryAudit.test.cjs index d973c393..f2177cfc 100644 --- a/test/back/ql3PackageBoundaryAudit.test.cjs +++ b/test/back/ql3PackageBoundaryAudit.test.cjs @@ -540,10 +540,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: localSqlite.rootSourceFileRoles, }, { - sourceFiles: 176, + sourceFiles: 177, rootSourceFiles: 1, rootSourceLines: 31, - nestedSourceFiles: 175, + nestedSourceFiles: 176, rootSourceFileRoles: { 'index.ts': 'public_export' }, }, );