mirror of
https://github.com/whyour/qinglong.git
synced 2026-09-20 16:07:11 +08:00
feat(ql3): reconcile local lost run retries
This commit is contained in:
@@ -11,6 +11,16 @@
|
|||||||
|
|
||||||
最新增量证据(2026-08-12):
|
最新增量证据(2026-08-12):
|
||||||
|
|
||||||
|
- D-292/ADR-0380(已接受)
|
||||||
|
Local Edge/Standalone 已补齐 admission-safe lost Run retry 纵向闭环,并修正“共享纯策略却暴露 Cluster-only 名称”的边界。
|
||||||
|
Runtime Core 在既有 Run 域发布 canonical `RunLostRetry*`/`run-lost-retry` subpath,同时保留早期 Cluster 名称兼容;SQLite
|
||||||
|
adapter 复用唯一 operation authority 与 `BEGIN IMMEDIATE` Run aggregate transaction,有界选择 runtime-owned、非 Workflow、
|
||||||
|
无 cancellation intent 且 latest Attempt 为 lost 的候选,原子 CAS Run/RetryPolicy、插入新 Attempt 与 Events。Local application
|
||||||
|
把一页 retry 接入现有 execution-control 周期,顺序为 completion→control→lost retry→慢速 cleanup;启动时在 recovery 后、Scheduler
|
||||||
|
前执行首轮。Edge 为 2 条/5 秒,Standalone 为 16 条/1 秒,不新增 package、migration、表、索引、timer、连接、listener 或 sidecar。
|
||||||
|
无 policy/禁用、unknown safety、attempt 耗尽均终态失败关闭;只有 admission 时已证明 `idempotent|deduplicated` 的策略才创建全新
|
||||||
|
Attempt。workspace 仍为 18 package/1055 source/1037 nested,`singleSourcePackages=[]`、`shallowSourcePackages=[]`。人工
|
||||||
|
`run.retry` API/CLI/UI、强认证、审计与 rate limit 仍未包含,不能把自动 lost retry 误报为人工恢复完成。
|
||||||
- D-291/ADR-0379(已接受)
|
- D-291/ADR-0379(已接受)
|
||||||
Cluster Run Attempt 日志 retention 已完成多副本纵向闭环,且没有新增 package:共享 claim contract 位于 Runtime Core 既有 Run
|
Cluster Run Attempt 日志 retention 已完成多副本纵向闭环,且没有新增 package:共享 claim contract 位于 Runtime Core 既有 Run
|
||||||
log-retention 目录;PostgreSQL v54 提供 durable control、immutable tombstone、terminal remote Worker candidate index 与最小权限
|
log-retention 目录;PostgreSQL v54 提供 durable control、immutable tombstone、terminal remote Worker candidate index 与最小权限
|
||||||
@@ -8485,7 +8495,7 @@ flowchart LR
|
|||||||
|
|
||||||
> ADR-0058 至 ADR-0092 的以下段落是历史快照,其 PostgreSQL 数字与“下一切片”已由 ADR-0104/0105 及后续 Accepted ADR 取代;资源、Package 和物理证据边界仍保留作为演进记录:PostgreSQL 当时 baseline 为 11 条 reviewed migration、capability v10、19 张表和 migration/runtime/admin/worker-ingress 四角色;本机 SQLite 为二十八条 reviewed migration、capability v14、二十六张 owned table,并新增正式 TaskDefinition head/immutable revision Repository。当前有二十七个受审 3.0 importer:Owner maintenance 已合并为两个权限隔离 subpath,execution/control/recovery/dispatch 已合并为 `@qinglong/local-execution` 四个单向 subpath,bootstrap/credential-recovery 已合并为 `@qinglong/local-owner-ceremony` 两个互不依赖 subpath;三个 package 都不提供聚合根入口,hard cap 已同步降为 27。一次拓扑 build/test 会先清理已登记 QL3 package 的 stale dist;27 包全量测试、dependency/source boundary、联网 vulnerability audit 与六种 Profile 制品门禁均通过。ADR-0090 让 production packlist 只发布 JS、`.d.ts` 和受审 assets/drizzle,不再携带开发 map;ADR-0092 后当前最大 application 为 1,691,009 bytes、267 files、61 loaded modules,最大抽样 RSS delta 11,780,096 bytes,低于 4 MiB/512 files/16 MiB。所有 package build 已变为 self-only,全量从约 198 次编译降至精确 27 次,clean 状态单包测试仍按依赖闭包工作。资源门禁已拆为原生 Linux x64/arm64 的 128 MiB router stress、256 MiB Edge release guard 与 512 MiB Cluster control guard,并从容器内部验证 cgroup v2、零 swap/OOM、非 root、只读挂载、seccomp 和 `NoNewPrivs`;本轮本地原生 arm64 三档已通过,x64 等待远端 CI,所有档位均明确不是物理设备或生产容量承诺。物理 Edge candidate recorder 已绑定设备 manifest、实际 Linux/存储环境、Edge/SQLite 基准与 no-replace SHA-256 报告,并会拒绝容器/VM;同设备同 boot 的 idle sampler、不主动填盘的专用文件系统 fault probe,以及通过正式 Repository 写入/扫描 100/1000/10000 个 `qinglong/command@v1` TaskDefinition 的规模记录协议均已具备契约门禁,但尚未取得固定实机报告或 signature。ADR-0091 已冻结 1–32 个 exact descriptor 的不可变 TaskSpec semantic registry、内建 command v1 和本机写前门禁;历史 revision 在 provider 缺失时仍可读。ADR-0092 已实现绑定 source revision/content digest 的 Profile-neutral command plan 与确定性本机 context/execution 映射,且保持 subpath-only。
|
> ADR-0058 至 ADR-0092 的以下段落是历史快照,其 PostgreSQL 数字与“下一切片”已由 ADR-0104/0105 及后续 Accepted ADR 取代;资源、Package 和物理证据边界仍保留作为演进记录:PostgreSQL 当时 baseline 为 11 条 reviewed migration、capability v10、19 张表和 migration/runtime/admin/worker-ingress 四角色;本机 SQLite 为二十八条 reviewed migration、capability v14、二十六张 owned table,并新增正式 TaskDefinition head/immutable revision Repository。当前有二十七个受审 3.0 importer:Owner maintenance 已合并为两个权限隔离 subpath,execution/control/recovery/dispatch 已合并为 `@qinglong/local-execution` 四个单向 subpath,bootstrap/credential-recovery 已合并为 `@qinglong/local-owner-ceremony` 两个互不依赖 subpath;三个 package 都不提供聚合根入口,hard cap 已同步降为 27。一次拓扑 build/test 会先清理已登记 QL3 package 的 stale dist;27 包全量测试、dependency/source boundary、联网 vulnerability audit 与六种 Profile 制品门禁均通过。ADR-0090 让 production packlist 只发布 JS、`.d.ts` 和受审 assets/drizzle,不再携带开发 map;ADR-0092 后当前最大 application 为 1,691,009 bytes、267 files、61 loaded modules,最大抽样 RSS delta 11,780,096 bytes,低于 4 MiB/512 files/16 MiB。所有 package build 已变为 self-only,全量从约 198 次编译降至精确 27 次,clean 状态单包测试仍按依赖闭包工作。资源门禁已拆为原生 Linux x64/arm64 的 128 MiB router stress、256 MiB Edge release guard 与 512 MiB Cluster control guard,并从容器内部验证 cgroup v2、零 swap/OOM、非 root、只读挂载、seccomp 和 `NoNewPrivs`;本轮本地原生 arm64 三档已通过,x64 等待远端 CI,所有档位均明确不是物理设备或生产容量承诺。物理 Edge candidate recorder 已绑定设备 manifest、实际 Linux/存储环境、Edge/SQLite 基准与 no-replace SHA-256 报告,并会拒绝容器/VM;同设备同 boot 的 idle sampler、不主动填盘的专用文件系统 fault probe,以及通过正式 Repository 写入/扫描 100/1000/10000 个 `qinglong/command@v1` TaskDefinition 的规模记录协议均已具备契约门禁,但尚未取得固定实机报告或 signature。ADR-0091 已冻结 1–32 个 exact descriptor 的不可变 TaskSpec semantic registry、内建 command v1 和本机写前门禁;历史 revision 在 provider 缺失时仍可读。ADR-0092 已实现绑定 source revision/content digest 的 Profile-neutral command plan 与确定性本机 context/execution 映射,且保持 subpath-only。
|
||||||
|
|
||||||
> 下表 PR-1 的旧“未完成”累计文字中,`completion/cancellation/timeout` 已由 ADR-0072 取代并闭环,Artifact range read 已由 ADR-0377 闭环;当前本机剩余项是人工 recovery、retry 产品策略、Artifact retention/tombstone、HTTP/CLI/UI 与部署 controller。远端 Worker completion 与 Workflow cancellation 的核心数据库链和 HA 重放已闭环;仍缺 production ingress/internal runtime port、expiry/retry lifecycle、部署启动装配与真实 Kubernetes 故障证据,不能与本机结论混用。
|
> 下表 PR-1 的旧“未完成”累计文字中,`completion/cancellation/timeout` 已由 ADR-0072 取代并闭环,Artifact range read 与 Local/Cluster retention 已由 ADR-0377/0378/0379 闭环,Local admission-safe lost retry lifecycle 已由 ADR-0380 闭环;当前本机剩余项是人工 recovery、手工 retry 产品策略/API、HTTP/CLI/UI、部署 controller 与固定路由设备实机门。远端 Worker completion、Workflow cancellation 和 Cluster expiry/retry 的核心数据库链、HA 重放与启动装配也已闭环;仍缺真实 Kubernetes 多节点故障证据,不能与本机结论混用。
|
||||||
>
|
>
|
||||||
> 下表 PR-0 的累积长文本仍含“四角色、21 条 migration、capability v20、21 项 HA”历史短语;当前权威基线应读取为六角色、23 条 migration、capability v22/36 表和 23 项 physical HA gate。ADR-0145/0146 又增加默认关闭的 manager-only TLS 1.3 management process、可选双副本 operation 与 durable distributed quota;它仍缺全副本重启 keyset anti-rollback 和真实 IdP/live ingress,所以“受认证管理入口”保持“已孵化但生产失败关闭”。
|
> 下表 PR-0 的累积长文本仍含“四角色、21 条 migration、capability v20、21 项 HA”历史短语;当前权威基线应读取为六角色、23 条 migration、capability v22/36 表和 23 项 physical HA gate。ADR-0145/0146 又增加默认关闭的 manager-only TLS 1.3 management process、可选双副本 operation 与 durable distributed quota;它仍缺全副本重启 keyset anti-rollback 和真实 IdP/live ingress,所以“受认证管理入口”保持“已孵化但生产失败关闭”。
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,75 @@
|
|||||||
|
# ADR-0380:Local lost Run retry 复用单一 execution-control cadence
|
||||||
|
|
||||||
|
- 状态:Accepted
|
||||||
|
- 日期:2026-08-12
|
||||||
|
- 关联 RFC:QL-RFC-0001 D-292
|
||||||
|
- 前置决策:ADR-0021、ADR-0066、ADR-0067、ADR-0232、ADR-0366
|
||||||
|
|
||||||
|
## 上下文
|
||||||
|
|
||||||
|
Cluster 已消费 admission-time `RunRetryPolicy`,但 Edge/Standalone 的启动恢复只能把确定不再运行的本地 Attempt 与 Run 标记为 `lost`。此后没有生产 consumer 处理该状态:安全且显式启用的重试不会创建新 Attempt,禁用、unsafe 或耗尽的 Run 也不会终态关闭。
|
||||||
|
|
||||||
|
直接复制一套 Local retry policy、建立独立 lifecycle package,或为 retry 新增 timer/SQLite 连接都会形成不合理架构。纯 transition 本来就是 profile-neutral,只因最初由 Cluster 接入而使用 `Cluster*` 名称;本地运行时也已经拥有单 SQLite operation authority、`BEGIN IMMEDIATE` Run aggregate transaction 和唯一 execution-control timer。
|
||||||
|
|
||||||
|
## 决策
|
||||||
|
|
||||||
|
### 1. 通用契约留在现有 Runtime Core Run 域
|
||||||
|
|
||||||
|
不新增 workspace package,也不复制状态机。`runtime-core` 对既有纯 transition、page contract、coordinator 与错误发布 canonical `RunLostRetry*` 名称和 `run-lost-retry` subpath;早期 `ClusterRunLostRetry*` 名称继续兼容。新代码只能依赖 profile-neutral 名称。
|
||||||
|
|
||||||
|
该兼容层不创建第二份实现。后续在具备 call-graph-aware rename 工具和独立废弃周期后,才允许移动物理文件或删除旧名称;本切片不以文本替换破坏已有 Cluster consumer。
|
||||||
|
|
||||||
|
### 2. SQLite adapter 复用唯一 operation authority
|
||||||
|
|
||||||
|
`LocalSqliteRunLostRetryRepository` 位于 `ql3-local-sqlite/src/run`,不形成单文件微包。它:
|
||||||
|
|
||||||
|
- 只选择 runtime-owned、非 Workflow、无 cancellation intent、latest Attempt 为 `lost` 的 Run;
|
||||||
|
- `lost` 优先,`retry_wait` 仅在 durable `next_attempt_at_ms` 到期后进入候选;
|
||||||
|
- 候选发现是一次有界 authority 操作,每个候选在同一既有 `LocalSqliteRunRepository.transaction()` 中重读并原子 CAS Run、RetryPolicy,插入新 Attempt 与 Events;
|
||||||
|
- 已变化的候选返回 `raced`,响应丢失后的重放由当前 durable aggregate 自然收敛;
|
||||||
|
- 不拥有连接、cursor、timer、listener、watcher 或常驻内存队列。
|
||||||
|
|
||||||
|
没有 policy、未启用或 `maxAttempts<=1` 时以 `RUN_LOST_RETRY_DISABLED` 失败关闭;`safety=unknown` 以 `RUN_LOST_RETRY_UNSAFE` 失败关闭;耗尽以 `RUN_LOST_RETRY_EXHAUSTED` 失败关闭。只有 admission 时已证明 `idempotent|deduplicated` 的 policy 才能先进入 `retry_wait`,到期后创建一个全新、无执行 handle/lease 的 claimed Attempt。
|
||||||
|
|
||||||
|
### 3. Local application 复用现有控制周期
|
||||||
|
|
||||||
|
lost retry 注入 `LocalExecutionControlLifecycle`,每轮顺序固定为:
|
||||||
|
|
||||||
|
1. 消费内存中的 completion 通知;
|
||||||
|
2. 扫描 cancellation/deadline control;
|
||||||
|
3. 执行一页 lost retry;
|
||||||
|
4. 到慢周期时清理 completion receipt 与 Run log Artifact。
|
||||||
|
|
||||||
|
启动顺序仍为 startup recovery → 首次 execution-control cycle → scheduler。因而启动恢复刚写入的 `lost` 会在 Scheduler 首轮前进入安全 retry/终态。停机的 control drain 不再创建 retry work。
|
||||||
|
|
||||||
|
Edge 每轮最多 2 条,沿用 5 秒 control cadence;Standalone 每轮最多 16 条,沿用 1 秒 cadence。两者均复用已有 `unref` timer 和 SQLite authority,没有增加低配路由设备的空闲连接、常驻 sidecar 或独立唤醒源。
|
||||||
|
|
||||||
|
### 4. 明确不包含人工重试产品入口
|
||||||
|
|
||||||
|
本 ADR 只消费 admission 时冻结的自动 retry policy,不授予用户新的 mutation authority。`run.retry` Policy permission、人工 recovery API/CLI/UI、强认证、rate limit、审计与“对失败 Run 手工重跑”的产品语义仍需独立 ADR;不得把自动 lost retry 误报为人工恢复完成。
|
||||||
|
|
||||||
|
## 验收
|
||||||
|
|
||||||
|
- Runtime Core 验证通用名称与旧 Cluster 名称指向同一实现;
|
||||||
|
- SQLite 真库验证安全 lost → retry_wait → 新 Attempt 的原子闭环、禁用 policy 失败关闭、页上限、`hasMore` 与重放不重复;
|
||||||
|
- Local lifecycle 验证 control → lost retry → cleanup 顺序及重叠 `runOnce` 合并;
|
||||||
|
- Local application 真启动验证 startup recovery 后、Scheduler 前完成首次 lost retry;
|
||||||
|
- 受影响 package build/test、完整 QL3 package/backend 回归、dependency/package/Edge import audit 必须通过后才允许阶段性提交;
|
||||||
|
- package 数、migration 数、表/索引数、timer 数和 SQLite connection 数不得增加。
|
||||||
|
|
||||||
|
### 验收证据(2026-08-12)
|
||||||
|
|
||||||
|
- `pnpm run test:packages:ql3`:18 个 QL3 package 全量清理、构建与测试通过,退出码 0;
|
||||||
|
- backend 回归:1165 项中 1163 通过、2 项按环境条件跳过、0 失败;
|
||||||
|
- package boundary audit:18 个 package、1055 个 source file、1037 个 nested source file,`singleSourcePackages=[]`、`shallowSourcePackages=[]`;
|
||||||
|
- cluster dependency audit:`findings=[]`;Edge import audit:121 个 imported module,root dependency 与 import 违规均为空;
|
||||||
|
- 本切片未新增 package、migration、表、索引、timer 或 SQLite connection。
|
||||||
|
|
||||||
|
## 被否决的替代方案
|
||||||
|
|
||||||
|
1. **新增 `ql3-local-retry` package**:只有一个 SQLite adapter,继续放大 package 碎片化,拒绝。
|
||||||
|
2. **复制 Cluster transition 为 Local 版本**:会让安全与终态规则漂移,拒绝。
|
||||||
|
3. **独立 retry timer**:增加路由设备唤醒、竞态和 shutdown 故障域,拒绝。
|
||||||
|
4. **启动时递归清空全部 lost backlog**:无法约束启动延迟与写放大,拒绝。
|
||||||
|
5. **`safety=unknown` 也自动重试**:可能复制不可逆外部副作用,拒绝。
|
||||||
|
6. **把人工 retry API 一并塞入本切片**:认证、Policy、审计与新 mutation 语义未冻结,拒绝。
|
||||||
@@ -383,6 +383,7 @@
|
|||||||
| [ADR-0377](./ADR-0377-profile-aware-run-attempt-log-range-read.md) | Profile-aware Run Attempt 日志 Range 读取 | Accepted |
|
| [ADR-0377](./ADR-0377-profile-aware-run-attempt-log-range-read.md) | Profile-aware Run Attempt 日志 Range 读取 | Accepted |
|
||||||
| [ADR-0378](./ADR-0378-local-run-attempt-log-retention-and-tombstones.md) | Local Run Attempt 日志有界保留与 durable tombstone | Accepted |
|
| [ADR-0378](./ADR-0378-local-run-attempt-log-retention-and-tombstones.md) | Local Run Attempt 日志有界保留与 durable tombstone | Accepted |
|
||||||
| [ADR-0379](./ADR-0379-cluster-run-attempt-log-retention.md) | Cluster Run Attempt 日志多副本保留与条件删除 | Accepted |
|
| [ADR-0379](./ADR-0379-cluster-run-attempt-log-retention.md) | Cluster Run Attempt 日志多副本保留与条件删除 | Accepted |
|
||||||
|
| [ADR-0380](./ADR-0380-local-lost-run-retry-single-control-cadence.md) | Local lost Run retry 复用单一 execution-control cadence | Accepted |
|
||||||
|
|
||||||
## 规则
|
## 规则
|
||||||
|
|
||||||
|
|||||||
@@ -88,6 +88,7 @@ export type LocalAdoptedProfileBootstrapResult =
|
|||||||
readonly executionControl: ReadyLocalStorage['executionControl'];
|
readonly executionControl: ReadyLocalStorage['executionControl'];
|
||||||
readonly completionReceipts: ReadyLocalStorage['completionReceipts'];
|
readonly completionReceipts: ReadyLocalStorage['completionReceipts'];
|
||||||
readonly runAttemptLogRetention: ReadyLocalStorage['runAttemptLogRetention'];
|
readonly runAttemptLogRetention: ReadyLocalStorage['runAttemptLogRetention'];
|
||||||
|
readonly runLostRetry: ReadyLocalStorage['runLostRetry'];
|
||||||
readonly localSecrets: LocalSecretEnvelopeRepository;
|
readonly localSecrets: LocalSecretEnvelopeRepository;
|
||||||
readonly localSecretAdministration: LocalSecretAdministrationRepository;
|
readonly localSecretAdministration: LocalSecretAdministrationRepository;
|
||||||
readonly projectPolicy: ProjectPolicyRepository;
|
readonly projectPolicy: ProjectPolicyRepository;
|
||||||
@@ -233,6 +234,7 @@ export async function bootstrapLocalAdoptedProfileStorage(
|
|||||||
executionControl: readyStorage.executionControl,
|
executionControl: readyStorage.executionControl,
|
||||||
completionReceipts: readyStorage.completionReceipts,
|
completionReceipts: readyStorage.completionReceipts,
|
||||||
runAttemptLogRetention: readyStorage.runAttemptLogRetention,
|
runAttemptLogRetention: readyStorage.runAttemptLogRetention,
|
||||||
|
runLostRetry: readyStorage.runLostRetry,
|
||||||
localSecrets: readyStorage.localSecrets,
|
localSecrets: readyStorage.localSecrets,
|
||||||
localSecretAdministration: readyStorage.localSecretAdministration,
|
localSecretAdministration: readyStorage.localSecretAdministration,
|
||||||
projectPolicy: readyStorage.projectPolicy,
|
projectPolicy: readyStorage.projectPolicy,
|
||||||
|
|||||||
@@ -73,6 +73,7 @@ const EXECUTION_CONTROL_POLICIES = Object.freeze({
|
|||||||
cleanupPageSize: 8,
|
cleanupPageSize: 8,
|
||||||
controlIntervalMs: 5_000,
|
controlIntervalMs: 5_000,
|
||||||
controlPageSize: 4,
|
controlPageSize: 4,
|
||||||
|
lostRetryPageSize: 2,
|
||||||
maxDrainPages: 2,
|
maxDrainPages: 2,
|
||||||
retentionMs: 24 * 60 * 60_000,
|
retentionMs: 24 * 60 * 60_000,
|
||||||
artifactNormalRetentionMs: 7 * 24 * 60 * 60_000,
|
artifactNormalRetentionMs: 7 * 24 * 60 * 60_000,
|
||||||
@@ -87,6 +88,7 @@ const EXECUTION_CONTROL_POLICIES = Object.freeze({
|
|||||||
cleanupPageSize: 32,
|
cleanupPageSize: 32,
|
||||||
controlIntervalMs: 1_000,
|
controlIntervalMs: 1_000,
|
||||||
controlPageSize: 32,
|
controlPageSize: 32,
|
||||||
|
lostRetryPageSize: 16,
|
||||||
maxDrainPages: 8,
|
maxDrainPages: 8,
|
||||||
retentionMs: 60 * 60_000,
|
retentionMs: 60 * 60_000,
|
||||||
artifactNormalRetentionMs: 30 * 24 * 60 * 60_000,
|
artifactNormalRetentionMs: 30 * 24 * 60 * 60_000,
|
||||||
@@ -345,6 +347,8 @@ export async function bootstrapLocalApplication(
|
|||||||
stopTimeoutMs: executionPolicy.stopTimeoutMs,
|
stopTimeoutMs: executionPolicy.stopTimeoutMs,
|
||||||
maxDrainPages: executionPolicy.maxDrainPages,
|
maxDrainPages: executionPolicy.maxDrainPages,
|
||||||
artifactRetention,
|
artifactRetention,
|
||||||
|
lostRetry: storage.runLostRetry,
|
||||||
|
lostRetryPageSize: executionPolicy.lostRetryPageSize,
|
||||||
onDiagnostic: async (error) => {
|
onDiagnostic: async (error) => {
|
||||||
if (error === undefined) return;
|
if (error === undefined) return;
|
||||||
await bestEffortAudit(options, {
|
await bestEffortAudit(options, {
|
||||||
|
|||||||
@@ -87,6 +87,8 @@ const RECEIPT_ATTEMPT_ID = '019f70c0-0000-7000-8000-000000000002';
|
|||||||
const RECEIPT_TOKEN = 'A'.repeat(32);
|
const RECEIPT_TOKEN = 'A'.repeat(32);
|
||||||
const CLEANUP_RUN_ID = '019f70c0-0000-7000-8000-000000000011';
|
const CLEANUP_RUN_ID = '019f70c0-0000-7000-8000-000000000011';
|
||||||
const CLEANUP_ATTEMPT_ID = '019f70c0-0000-7000-8000-000000000012';
|
const CLEANUP_ATTEMPT_ID = '019f70c0-0000-7000-8000-000000000012';
|
||||||
|
const LOST_RETRY_RUN_ID = '019f70c0-0000-7000-8000-000000000021';
|
||||||
|
const LOST_RETRY_ATTEMPT_ID = '019f70c0-0000-7000-8000-000000000022';
|
||||||
const WORKFLOW_CANCELLATION_CREDENTIAL_ID = 'application-workflow-owner';
|
const WORKFLOW_CANCELLATION_CREDENTIAL_ID = 'application-workflow-owner';
|
||||||
const WORKFLOW_CANCELLATION_PEPPER_KEY_ID = 'application-workflow-owner-v1';
|
const WORKFLOW_CANCELLATION_PEPPER_KEY_ID = 'application-workflow-owner-v1';
|
||||||
const WORKFLOW_CANCELLATION_PEPPER_BYTES = Buffer.alloc(32, 141);
|
const WORKFLOW_CANCELLATION_PEPPER_BYTES = Buffer.alloc(32, 141);
|
||||||
@@ -478,6 +480,58 @@ function insertActiveTargetRun(value, id, status = 'running') {
|
|||||||
target.close();
|
target.close();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function insertSafeLostTargetRun(value) {
|
||||||
|
const now = Date.now();
|
||||||
|
const target = new DatabaseSync(value.targetPath);
|
||||||
|
target.exec('BEGIN IMMEDIATE');
|
||||||
|
try {
|
||||||
|
target
|
||||||
|
.prepare(
|
||||||
|
`INSERT INTO "Runs" (
|
||||||
|
id, project_id, task_id, task_revision, trigger_type,
|
||||||
|
execution_origin, execution_owner, status, version, event_sequence,
|
||||||
|
priority, created_at_ms, queued_at_ms, started_at_ms,
|
||||||
|
error_code, error_summary
|
||||||
|
) VALUES (?, 'default', 'task-lost-retry', 'revision-lost-retry',
|
||||||
|
'manual', 'manual', 'runtime', 'lost', 1, 0, 0, ?, ?, ?,
|
||||||
|
'LOCAL_RECOVERY_EXECUTION_NOT_RUNNING', 'lost')`,
|
||||||
|
)
|
||||||
|
.run(LOST_RETRY_RUN_ID, now - 300, now - 250, now - 200);
|
||||||
|
target
|
||||||
|
.prepare(
|
||||||
|
`INSERT INTO "RunAttempts" (
|
||||||
|
id, run_id, attempt, status, executor_type, callback_sequence,
|
||||||
|
created_at_ms, started_at_ms, finished_at_ms,
|
||||||
|
error_code, error_summary
|
||||||
|
) VALUES (?, ?, 1, 'lost', 'local_process', 0, ?, ?, ?,
|
||||||
|
'LOCAL_RECOVERY_EXECUTION_NOT_RUNNING', 'lost')`,
|
||||||
|
)
|
||||||
|
.run(
|
||||||
|
LOST_RETRY_ATTEMPT_ID,
|
||||||
|
LOST_RETRY_RUN_ID,
|
||||||
|
now - 250,
|
||||||
|
now - 200,
|
||||||
|
now - 100,
|
||||||
|
);
|
||||||
|
target
|
||||||
|
.prepare(
|
||||||
|
`INSERT INTO "RunRetryPolicies" (
|
||||||
|
run_id, max_attempts, retry_on_lost, safety,
|
||||||
|
backoff_base_ms, backoff_max_ms, next_attempt_at_ms,
|
||||||
|
version, created_at_ms, updated_at_ms
|
||||||
|
) VALUES (?, 3, 1, 'idempotent', 86400000, 86400000,
|
||||||
|
NULL, 0, ?, ?)`,
|
||||||
|
)
|
||||||
|
.run(LOST_RETRY_RUN_ID, now - 300, now - 300);
|
||||||
|
target.exec('COMMIT');
|
||||||
|
} catch (error) {
|
||||||
|
if (target.isTransaction) target.exec('ROLLBACK');
|
||||||
|
throw error;
|
||||||
|
} finally {
|
||||||
|
target.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
function insertManyActiveTargetRuns(value, count) {
|
function insertManyActiveTargetRuns(value, count) {
|
||||||
const target = new DatabaseSync(value.targetPath);
|
const target = new DatabaseSync(value.targetPath);
|
||||||
const statement = target.prepare(
|
const statement = target.prepare(
|
||||||
@@ -2179,6 +2233,39 @@ test('durable Run candidates block lifecycle activation', async (t) => {
|
|||||||
assertSourceWritable(value, 2);
|
assertSourceWritable(value, 2);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('reconciles a safe lost Run before the first Local scheduler pass', async (t) => {
|
||||||
|
const value = await prepare(t, 'edge');
|
||||||
|
insertSafeLostTargetRun(value);
|
||||||
|
const audits = [];
|
||||||
|
const result = await bootstrapLocalApplication(
|
||||||
|
options(value, {
|
||||||
|
applicationAudit: (record) => audits.push(record),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
assert.equal(result.status, 'active');
|
||||||
|
assert.equal(
|
||||||
|
(await result.runs.findRunById(LOST_RETRY_RUN_ID)).status,
|
||||||
|
'retry_wait',
|
||||||
|
);
|
||||||
|
const reconciled = audits.find(
|
||||||
|
(record) => record.state === 'receipts_reconciled',
|
||||||
|
);
|
||||||
|
assert.deepEqual(reconciled.executionControl.lostRetry, {
|
||||||
|
scanned: 1,
|
||||||
|
scheduled: 1,
|
||||||
|
requeued: 0,
|
||||||
|
failed: 0,
|
||||||
|
raced: 0,
|
||||||
|
hasMore: false,
|
||||||
|
});
|
||||||
|
assert.equal(
|
||||||
|
(await result.runs.findLatestAttemptByRunId(LOST_RETRY_RUN_ID)).attempt,
|
||||||
|
1,
|
||||||
|
);
|
||||||
|
assert.equal(await result.stop(), 'stopped');
|
||||||
|
});
|
||||||
|
|
||||||
test('startup recovery candidate overflow fails closed at the hard page bound', async (t) => {
|
test('startup recovery candidate overflow fails closed at the hard page bound', async (t) => {
|
||||||
const value = await prepare(t, 'edge');
|
const value = await prepare(t, 'edge');
|
||||||
insertManyActiveTargetRuns(value, MAX_LOCAL_RUN_RECOVERY_ITEMS + 1);
|
insertManyActiveTargetRuns(value, MAX_LOCAL_RUN_RECOVERY_ITEMS + 1);
|
||||||
|
|||||||
@@ -1,6 +1,11 @@
|
|||||||
import type { LocalCompletionReceiptJournalCursor } from '@qinglong/runtime-core/local-completion-receipt-journal';
|
import type { LocalCompletionReceiptJournalCursor } from '@qinglong/runtime-core/local-completion-receipt-journal';
|
||||||
import { assertLocalExecutionControlLimit } from '@qinglong/runtime-core/local-execution-control';
|
import { assertLocalExecutionControlLimit } from '@qinglong/runtime-core/local-execution-control';
|
||||||
import type { RunAttemptLogRetentionSweepSummary } from '@qinglong/runtime-core/run-attempt-log-retention';
|
import type { RunAttemptLogRetentionSweepSummary } from '@qinglong/runtime-core/run-attempt-log-retention';
|
||||||
|
import {
|
||||||
|
RunLostRetryCoordinator,
|
||||||
|
type RunLostRetryPageResult,
|
||||||
|
type RunLostRetryRepository,
|
||||||
|
} from '@qinglong/runtime-core/run-lost-retry';
|
||||||
import type {
|
import type {
|
||||||
LocalCompletionReceiptCleanupScanner,
|
LocalCompletionReceiptCleanupScanner,
|
||||||
LocalCompletionReceiptCleanupSummary,
|
LocalCompletionReceiptCleanupSummary,
|
||||||
@@ -25,6 +30,8 @@ export interface LocalExecutionControlLifecycleOptions {
|
|||||||
readonly artifactRetention?: Readonly<{
|
readonly artifactRetention?: Readonly<{
|
||||||
sweep(): Promise<RunAttemptLogRetentionSweepSummary>;
|
sweep(): Promise<RunAttemptLogRetentionSweepSummary>;
|
||||||
}>;
|
}>;
|
||||||
|
readonly lostRetry?: RunLostRetryRepository;
|
||||||
|
readonly lostRetryPageSize?: number;
|
||||||
readonly clock?: { now(): number };
|
readonly clock?: { now(): number };
|
||||||
readonly onDiagnostic?: (
|
readonly onDiagnostic?: (
|
||||||
error: unknown,
|
error: unknown,
|
||||||
@@ -38,6 +45,7 @@ export interface LocalExecutionControlCycleSummary {
|
|||||||
readonly control: LocalExecutionControlScanSummary;
|
readonly control: LocalExecutionControlScanSummary;
|
||||||
readonly cleanup?: LocalCompletionReceiptCleanupSummary;
|
readonly cleanup?: LocalCompletionReceiptCleanupSummary;
|
||||||
readonly artifactRetention?: RunAttemptLogRetentionSweepSummary;
|
readonly artifactRetention?: RunAttemptLogRetentionSweepSummary;
|
||||||
|
readonly lostRetry?: Readonly<RunLostRetryPageResult>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface LocalExecutionControlStopSummary {
|
export interface LocalExecutionControlStopSummary {
|
||||||
@@ -49,6 +57,7 @@ export interface LocalExecutionControlStopSummary {
|
|||||||
export class LocalExecutionControlLifecycle {
|
export class LocalExecutionControlLifecycle {
|
||||||
private readonly clock: { now(): number };
|
private readonly clock: { now(): number };
|
||||||
private readonly maxNotifications: number;
|
private readonly maxNotifications: number;
|
||||||
|
private readonly lostRetry: RunLostRetryCoordinator | undefined;
|
||||||
private timer: NodeJS.Timeout | undefined;
|
private timer: NodeJS.Timeout | undefined;
|
||||||
private inFlight: Promise<LocalExecutionControlCycleSummary> | undefined;
|
private inFlight: Promise<LocalExecutionControlCycleSummary> | undefined;
|
||||||
private stopPromise: Promise<LocalExecutionControlStopSummary> | undefined;
|
private stopPromise: Promise<LocalExecutionControlStopSummary> | undefined;
|
||||||
@@ -122,6 +131,18 @@ export class LocalExecutionControlLifecycle {
|
|||||||
) {
|
) {
|
||||||
throw new TypeError('Local Artifact retention lifecycle is invalid');
|
throw new TypeError('Local Artifact retention lifecycle is invalid');
|
||||||
}
|
}
|
||||||
|
if (
|
||||||
|
(options.lostRetry === undefined) !==
|
||||||
|
(options.lostRetryPageSize === undefined)
|
||||||
|
) {
|
||||||
|
throw new TypeError('Local Run lost retry lifecycle is invalid');
|
||||||
|
}
|
||||||
|
this.lostRetry =
|
||||||
|
options.lostRetry && options.lostRetryPageSize !== undefined
|
||||||
|
? new RunLostRetryCoordinator(options.lostRetry, {
|
||||||
|
pageSize: options.lostRetryPageSize,
|
||||||
|
})
|
||||||
|
: undefined;
|
||||||
this.clock = options.clock ?? { now: Date.now };
|
this.clock = options.clock ?? { now: Date.now };
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -238,6 +259,7 @@ export class LocalExecutionControlLifecycle {
|
|||||||
: { cursor: this.controlCursor }),
|
: { cursor: this.controlCursor }),
|
||||||
});
|
});
|
||||||
this.controlCursor = control.truncated ? control.nextCursor : undefined;
|
this.controlCursor = control.truncated ? control.nextCursor : undefined;
|
||||||
|
const lostRetry = await this.lostRetry?.reconcile();
|
||||||
const now = this.clock.now();
|
const now = this.clock.now();
|
||||||
if (!Number.isSafeInteger(now) || now < 0) {
|
if (!Number.isSafeInteger(now) || now < 0) {
|
||||||
throw new RangeError(
|
throw new RangeError(
|
||||||
@@ -266,6 +288,7 @@ export class LocalExecutionControlLifecycle {
|
|||||||
completions: completion.processed,
|
completions: completion.processed,
|
||||||
completionFailures: completion.failed,
|
completionFailures: completion.failed,
|
||||||
control,
|
control,
|
||||||
|
...(lostRetry === undefined ? {} : { lostRetry }),
|
||||||
...(cleanup === undefined ? {} : { cleanup }),
|
...(cleanup === undefined ? {} : { cleanup }),
|
||||||
...(artifactRetention === undefined ? {} : { artifactRetention }),
|
...(artifactRetention === undefined ? {} : { artifactRetention }),
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -0,0 +1,92 @@
|
|||||||
|
'use strict';
|
||||||
|
|
||||||
|
const assert = require('node:assert/strict');
|
||||||
|
const { test } = require('node:test');
|
||||||
|
|
||||||
|
const { LocalExecutionControlLifecycle } = require('../dist/control');
|
||||||
|
|
||||||
|
test('runs lost retry inside the existing control cadence before cleanup', async () => {
|
||||||
|
const calls = [];
|
||||||
|
const lifecycle = new LocalExecutionControlLifecycle(
|
||||||
|
{
|
||||||
|
async process() {
|
||||||
|
throw new Error('no completion notifications are expected');
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
async scan() {
|
||||||
|
calls.push('control');
|
||||||
|
return {
|
||||||
|
observedAtMs: 100,
|
||||||
|
scanned: 0,
|
||||||
|
terminal: 0,
|
||||||
|
deferred: 0,
|
||||||
|
failed: 0,
|
||||||
|
truncated: false,
|
||||||
|
};
|
||||||
|
},
|
||||||
|
async drain() {
|
||||||
|
return {
|
||||||
|
observedAtMs: 100,
|
||||||
|
scanned: 0,
|
||||||
|
terminal: 0,
|
||||||
|
deferred: 0,
|
||||||
|
failed: 0,
|
||||||
|
remaining: 0,
|
||||||
|
pages: 0,
|
||||||
|
truncated: false,
|
||||||
|
};
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
async scan() {
|
||||||
|
calls.push('cleanup');
|
||||||
|
return {
|
||||||
|
scanned: 0,
|
||||||
|
removed: 0,
|
||||||
|
missing: 0,
|
||||||
|
deferred: 0,
|
||||||
|
failed: 0,
|
||||||
|
truncated: false,
|
||||||
|
};
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
intervalMs: 5_000,
|
||||||
|
pageSize: 4,
|
||||||
|
cleanupIntervalMs: 60_000,
|
||||||
|
cleanupPageSize: 4,
|
||||||
|
stopTimeoutMs: 1_000,
|
||||||
|
maxDrainPages: 1,
|
||||||
|
lostRetry: {
|
||||||
|
async reconcilePage(command) {
|
||||||
|
calls.push('lost_retry');
|
||||||
|
assert.deepEqual(command, { limit: 2 });
|
||||||
|
return {
|
||||||
|
scanned: 1,
|
||||||
|
scheduled: 1,
|
||||||
|
requeued: 0,
|
||||||
|
failed: 0,
|
||||||
|
raced: 0,
|
||||||
|
hasMore: false,
|
||||||
|
};
|
||||||
|
},
|
||||||
|
},
|
||||||
|
lostRetryPageSize: 2,
|
||||||
|
clock: { now: () => 100 },
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
const first = lifecycle.runOnce(true);
|
||||||
|
const second = lifecycle.runOnce(true);
|
||||||
|
assert.equal(first, second);
|
||||||
|
assert.deepEqual((await first).lostRetry, {
|
||||||
|
scanned: 1,
|
||||||
|
scheduled: 1,
|
||||||
|
requeued: 0,
|
||||||
|
failed: 0,
|
||||||
|
raced: 0,
|
||||||
|
hasMore: false,
|
||||||
|
});
|
||||||
|
assert.deepEqual(calls, ['control', 'lost_retry', 'cleanup']);
|
||||||
|
});
|
||||||
@@ -250,6 +250,11 @@
|
|||||||
"require": "./dist/run/runAttemptLogRetentionRepository.js",
|
"require": "./dist/run/runAttemptLogRetentionRepository.js",
|
||||||
"default": "./dist/run/runAttemptLogRetentionRepository.js"
|
"default": "./dist/run/runAttemptLogRetentionRepository.js"
|
||||||
},
|
},
|
||||||
|
"./run-lost-retry": {
|
||||||
|
"types": "./dist/run/runLostRetryRepository.d.ts",
|
||||||
|
"require": "./dist/run/runLostRetryRepository.js",
|
||||||
|
"default": "./dist/run/runLostRetryRepository.js"
|
||||||
|
},
|
||||||
"./trigger-administration": {
|
"./trigger-administration": {
|
||||||
"types": "./dist/scheduling/triggerAdministration.d.ts",
|
"types": "./dist/scheduling/triggerAdministration.d.ts",
|
||||||
"require": "./dist/scheduling/triggerAdministration.js",
|
"require": "./dist/scheduling/triggerAdministration.js",
|
||||||
|
|||||||
@@ -47,6 +47,7 @@ export type LocalProfileStorageBootstrapResult =
|
|||||||
readonly executionControl: LocalSqliteRuntimeDatabase['executionControl'];
|
readonly executionControl: LocalSqliteRuntimeDatabase['executionControl'];
|
||||||
readonly completionReceipts: LocalSqliteRuntimeDatabase['completionReceipts'];
|
readonly completionReceipts: LocalSqliteRuntimeDatabase['completionReceipts'];
|
||||||
readonly runAttemptLogRetention: LocalSqliteRuntimeDatabase['runAttemptLogRetention'];
|
readonly runAttemptLogRetention: LocalSqliteRuntimeDatabase['runAttemptLogRetention'];
|
||||||
|
readonly runLostRetry: LocalSqliteRuntimeDatabase['runLostRetry'];
|
||||||
readonly localSecrets: LocalSqliteRuntimeDatabase['localSecrets'];
|
readonly localSecrets: LocalSqliteRuntimeDatabase['localSecrets'];
|
||||||
readonly localSecretAdministration: LocalSqliteRuntimeDatabase['localSecretAdministration'];
|
readonly localSecretAdministration: LocalSqliteRuntimeDatabase['localSecretAdministration'];
|
||||||
readonly projectPolicy: LocalSqliteRuntimeDatabase['projectPolicy'];
|
readonly projectPolicy: LocalSqliteRuntimeDatabase['projectPolicy'];
|
||||||
@@ -140,6 +141,7 @@ export async function bootstrapLocalProfileStorage(
|
|||||||
executionControl: database.executionControl,
|
executionControl: database.executionControl,
|
||||||
completionReceipts: database.completionReceipts,
|
completionReceipts: database.completionReceipts,
|
||||||
runAttemptLogRetention: database.runAttemptLogRetention,
|
runAttemptLogRetention: database.runAttemptLogRetention,
|
||||||
|
runLostRetry: database.runLostRetry,
|
||||||
localSecrets: database.localSecrets,
|
localSecrets: database.localSecrets,
|
||||||
localSecretAdministration: database.localSecretAdministration,
|
localSecretAdministration: database.localSecretAdministration,
|
||||||
projectPolicy: database.projectPolicy,
|
projectPolicy: database.projectPolicy,
|
||||||
|
|||||||
@@ -0,0 +1,251 @@
|
|||||||
|
import { randomUUID } from 'node:crypto';
|
||||||
|
|
||||||
|
import {
|
||||||
|
RunLostRetryUnavailableError,
|
||||||
|
buildRunLostRetryTransition,
|
||||||
|
normalizeRunLostRetryPageCommand,
|
||||||
|
normalizeRunLostRetryPageResult,
|
||||||
|
type RunLostRetryDisposition,
|
||||||
|
type RunLostRetryPageCommand,
|
||||||
|
type RunLostRetryPageResult,
|
||||||
|
type RunLostRetryRepository,
|
||||||
|
type RunLostRetryTransition,
|
||||||
|
} from '@qinglong/runtime-core/run-lost-retry';
|
||||||
|
import type {
|
||||||
|
RunRecord,
|
||||||
|
RunRepositoryTransaction,
|
||||||
|
RunRetryPolicyRecord,
|
||||||
|
} from '@qinglong/runtime-core/run-repository';
|
||||||
|
|
||||||
|
import { LocalSqliteOperationAuthority } from '../authority/operationAuthority';
|
||||||
|
import { LocalSqliteRunRepository } from './runRepository';
|
||||||
|
|
||||||
|
type Row = Record<string, unknown>;
|
||||||
|
|
||||||
|
interface Candidate {
|
||||||
|
readonly runId: string;
|
||||||
|
readonly attemptId: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
function identifier(row: Row, key: string): string {
|
||||||
|
const value = row[key];
|
||||||
|
if (
|
||||||
|
typeof value !== 'string' ||
|
||||||
|
value.length < 1 ||
|
||||||
|
value.length > 128 ||
|
||||||
|
/[\u0000-\u001f\u007f]/.test(value)
|
||||||
|
) {
|
||||||
|
throw new TypeError(`Local SQLite lost retry ${key} is invalid`);
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
function generatedId(factory: () => string): string {
|
||||||
|
const value = factory();
|
||||||
|
if (
|
||||||
|
typeof value !== 'string' ||
|
||||||
|
value.length < 1 ||
|
||||||
|
value.length > 36 ||
|
||||||
|
/[\u0000-\u001f\u007f]/.test(value)
|
||||||
|
) {
|
||||||
|
throw new TypeError('Local SQLite lost retry generated ID is invalid');
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Local Profiles share one SQLite authority. Candidate discovery and each
|
||||||
|
* BEGIN IMMEDIATE aggregate mutation are bounded, serialized operations; this
|
||||||
|
* repository owns no connection, timer, cursor, or background task.
|
||||||
|
*/
|
||||||
|
export class LocalSqliteRunLostRetryRepository
|
||||||
|
implements RunLostRetryRepository
|
||||||
|
{
|
||||||
|
constructor(
|
||||||
|
private readonly authority: LocalSqliteOperationAuthority,
|
||||||
|
private readonly runs: LocalSqliteRunRepository,
|
||||||
|
private readonly createId: () => string = randomUUID,
|
||||||
|
private readonly clock: { now(): number } = { now: Date.now },
|
||||||
|
) {
|
||||||
|
if (
|
||||||
|
!(authority instanceof LocalSqliteOperationAuthority) ||
|
||||||
|
!(runs instanceof LocalSqliteRunRepository) ||
|
||||||
|
typeof createId !== 'function' ||
|
||||||
|
typeof clock?.now !== 'function'
|
||||||
|
) {
|
||||||
|
throw new TypeError('Local SQLite lost retry repository is invalid');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async reconcilePage(
|
||||||
|
input: Readonly<RunLostRetryPageCommand>,
|
||||||
|
): Promise<Readonly<RunLostRetryPageResult>> {
|
||||||
|
const command = normalizeRunLostRetryPageCommand(input);
|
||||||
|
try {
|
||||||
|
const candidates = await this.listCandidates(command.limit + 1);
|
||||||
|
const page = candidates.slice(0, command.limit);
|
||||||
|
const counts: Record<RunLostRetryDisposition | 'raced', number> = {
|
||||||
|
scheduled: 0,
|
||||||
|
requeued: 0,
|
||||||
|
failed_disabled: 0,
|
||||||
|
failed_unsafe: 0,
|
||||||
|
failed_exhausted: 0,
|
||||||
|
raced: 0,
|
||||||
|
};
|
||||||
|
for (const candidate of page) {
|
||||||
|
counts[await this.reconcileCandidate(candidate)] += 1;
|
||||||
|
}
|
||||||
|
return normalizeRunLostRetryPageResult(
|
||||||
|
{
|
||||||
|
scanned: page.length,
|
||||||
|
scheduled: counts.scheduled,
|
||||||
|
requeued: counts.requeued,
|
||||||
|
failed:
|
||||||
|
counts.failed_disabled +
|
||||||
|
counts.failed_unsafe +
|
||||||
|
counts.failed_exhausted,
|
||||||
|
raced: counts.raced,
|
||||||
|
hasMore: candidates.length > command.limit,
|
||||||
|
},
|
||||||
|
command.limit,
|
||||||
|
);
|
||||||
|
} catch (error) {
|
||||||
|
if (error instanceof RunLostRetryUnavailableError) throw error;
|
||||||
|
throw new RunLostRetryUnavailableError({ cause: error });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private listCandidates(limit: number): Promise<readonly Candidate[]> {
|
||||||
|
return this.authority.enqueue(
|
||||||
|
async () => {
|
||||||
|
const observedAtMs = this.observedAtMs();
|
||||||
|
const rows = this.authority.client
|
||||||
|
.prepare(
|
||||||
|
`SELECT run."id" AS "runId", attempt."id" AS "attemptId"
|
||||||
|
FROM "Runs" AS run
|
||||||
|
JOIN "RunAttempts" AS attempt
|
||||||
|
ON attempt."run_id" = run."id"
|
||||||
|
AND attempt."attempt" = (
|
||||||
|
SELECT MAX(latest."attempt")
|
||||||
|
FROM "RunAttempts" AS latest
|
||||||
|
WHERE latest."run_id" = run."id"
|
||||||
|
)
|
||||||
|
LEFT JOIN "RunRetryPolicies" AS policy
|
||||||
|
ON policy."run_id" = run."id"
|
||||||
|
WHERE run."execution_owner" = 'runtime'
|
||||||
|
AND run."trigger_type" <> 'plugin_package_workflow'
|
||||||
|
AND run."cancel_requested_at_ms" IS NULL
|
||||||
|
AND attempt."status" = 'lost'
|
||||||
|
AND (
|
||||||
|
run."status" = 'lost'
|
||||||
|
OR (
|
||||||
|
run."status" = 'retry_wait'
|
||||||
|
AND policy."next_attempt_at_ms" IS NOT NULL
|
||||||
|
AND policy."next_attempt_at_ms" <= ?
|
||||||
|
)
|
||||||
|
)
|
||||||
|
ORDER BY
|
||||||
|
CASE WHEN run."status" = 'lost' THEN 0
|
||||||
|
ELSE policy."next_attempt_at_ms" END,
|
||||||
|
run."id"
|
||||||
|
LIMIT ?`,
|
||||||
|
)
|
||||||
|
.all(observedAtMs, limit) as Row[];
|
||||||
|
if (rows.length > limit) {
|
||||||
|
throw new TypeError('Local SQLite lost retry exceeded its page size');
|
||||||
|
}
|
||||||
|
return Object.freeze(
|
||||||
|
rows.map((row) =>
|
||||||
|
Object.freeze({
|
||||||
|
runId: identifier(row, 'runId'),
|
||||||
|
attemptId: identifier(row, 'attemptId'),
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
},
|
||||||
|
() => new RunLostRetryUnavailableError(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async reconcileCandidate(
|
||||||
|
candidate: Readonly<Candidate>,
|
||||||
|
): Promise<RunLostRetryDisposition | 'raced'> {
|
||||||
|
return this.runs.transaction(async (transaction) => {
|
||||||
|
const run = await transaction.findRunById(candidate.runId);
|
||||||
|
const attempt = await transaction.findLatestAttemptByRunId(
|
||||||
|
candidate.runId,
|
||||||
|
);
|
||||||
|
const policy = await transaction.findRetryPolicyByRunId(candidate.runId);
|
||||||
|
if (
|
||||||
|
!run ||
|
||||||
|
!attempt ||
|
||||||
|
attempt.id !== candidate.attemptId ||
|
||||||
|
attempt.status !== 'lost' ||
|
||||||
|
run.executionOwner !== 'runtime' ||
|
||||||
|
run.triggerType === 'plugin_package_workflow' ||
|
||||||
|
run.cancelRequestedAtMs !== undefined ||
|
||||||
|
(run.status !== 'lost' && run.status !== 'retry_wait') ||
|
||||||
|
(run.status === 'retry_wait' &&
|
||||||
|
(policy?.nextAttemptAtMs === undefined ||
|
||||||
|
policy.nextAttemptAtMs > this.observedAtMs()))
|
||||||
|
) {
|
||||||
|
return 'raced';
|
||||||
|
}
|
||||||
|
const transition = buildRunLostRetryTransition({
|
||||||
|
run,
|
||||||
|
attempt,
|
||||||
|
policy,
|
||||||
|
observedAtMs: this.observedAtMs(),
|
||||||
|
runEventId: generatedId(this.createId),
|
||||||
|
...(run.status === 'retry_wait'
|
||||||
|
? {
|
||||||
|
attemptId: generatedId(this.createId),
|
||||||
|
attemptEventId: generatedId(this.createId),
|
||||||
|
}
|
||||||
|
: {}),
|
||||||
|
});
|
||||||
|
await this.persistTransition(transaction, run, policy, transition);
|
||||||
|
return transition.disposition;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private observedAtMs(): number {
|
||||||
|
const value = this.clock.now();
|
||||||
|
if (!Number.isSafeInteger(value) || value < 0) {
|
||||||
|
throw new TypeError('Local SQLite lost retry clock is invalid');
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
private async persistTransition(
|
||||||
|
transaction: RunRepositoryTransaction,
|
||||||
|
currentRun: Readonly<RunRecord>,
|
||||||
|
currentPolicy: Readonly<RunRetryPolicyRecord> | null,
|
||||||
|
transition: Readonly<RunLostRetryTransition>,
|
||||||
|
): Promise<void> {
|
||||||
|
let version = currentRun.version;
|
||||||
|
for (const run of transition.runTransitions) {
|
||||||
|
if (!(await transaction.compareAndSetRun(run, version))) {
|
||||||
|
throw new TypeError('Local SQLite lost retry Run fence changed');
|
||||||
|
}
|
||||||
|
version = run.version;
|
||||||
|
}
|
||||||
|
if (transition.policy && transition.policy !== currentPolicy) {
|
||||||
|
if (
|
||||||
|
!currentPolicy ||
|
||||||
|
!(await transaction.compareAndSetRetryPolicy(
|
||||||
|
transition.policy,
|
||||||
|
currentPolicy.version,
|
||||||
|
))
|
||||||
|
) {
|
||||||
|
throw new TypeError('Local SQLite lost retry policy fence changed');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (transition.attempt) {
|
||||||
|
await transaction.insertAttempt(transition.attempt);
|
||||||
|
}
|
||||||
|
for (const event of transition.events) {
|
||||||
|
await transaction.appendEvent(event);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -63,6 +63,7 @@ import type { LocalSqlitePluginPackageWorkflowFrontierRepository } from '../plug
|
|||||||
import type { LocalSqlitePluginPackageWorkflowTaskAttemptAdmissionRepository } from '../plugin-package/workflow/pluginPackageWorkflowTaskAttemptAdmissionRepository';
|
import type { LocalSqlitePluginPackageWorkflowTaskAttemptAdmissionRepository } from '../plugin-package/workflow/pluginPackageWorkflowTaskAttemptAdmissionRepository';
|
||||||
import type { LocalSqlitePluginPackageWorkflowCancellationConvergenceRepository } from '../plugin-package/workflow/pluginPackageWorkflowCancellationConvergenceRepository';
|
import type { LocalSqlitePluginPackageWorkflowCancellationConvergenceRepository } from '../plugin-package/workflow/pluginPackageWorkflowCancellationConvergenceRepository';
|
||||||
import { LocalSqliteRunAttemptLogRetentionRepository } from '../run/runAttemptLogRetentionRepository';
|
import { LocalSqliteRunAttemptLogRetentionRepository } from '../run/runAttemptLogRetentionRepository';
|
||||||
|
import { LocalSqliteRunLostRetryRepository } from '../run/runLostRetryRepository';
|
||||||
|
|
||||||
export interface LocalSqliteRuntimeDependencies {
|
export interface LocalSqliteRuntimeDependencies {
|
||||||
readonly taskSpecSemanticRegistry?: TaskSpecSemanticRegistry;
|
readonly taskSpecSemanticRegistry?: TaskSpecSemanticRegistry;
|
||||||
@@ -99,6 +100,7 @@ export interface LocalSqliteRuntimeDatabase {
|
|||||||
readonly executionControl: LocalExecutionControlSource;
|
readonly executionControl: LocalExecutionControlSource;
|
||||||
readonly completionReceipts: LocalCompletionReceiptJournal;
|
readonly completionReceipts: LocalCompletionReceiptJournal;
|
||||||
readonly runAttemptLogRetention: LocalSqliteRunAttemptLogRetentionRepository;
|
readonly runAttemptLogRetention: LocalSqliteRunAttemptLogRetentionRepository;
|
||||||
|
readonly runLostRetry: LocalSqliteRunLostRetryRepository;
|
||||||
readonly localSecrets: LocalSecretEnvelopeRepository;
|
readonly localSecrets: LocalSecretEnvelopeRepository;
|
||||||
readonly localSecretAdministration: LocalSecretAdministrationRepository;
|
readonly localSecretAdministration: LocalSecretAdministrationRepository;
|
||||||
readonly projectPolicy: ProjectPolicyRepository;
|
readonly projectPolicy: ProjectPolicyRepository;
|
||||||
@@ -183,6 +185,10 @@ export async function openLocalSqliteRuntimeDatabase(
|
|||||||
const ownerPepper = new LocalSqliteOwnerPepperRepository(authority);
|
const ownerPepper = new LocalSqliteOwnerPepperRepository(authority);
|
||||||
const runAttemptLogRetention =
|
const runAttemptLogRetention =
|
||||||
new LocalSqliteRunAttemptLogRetentionRepository(authority);
|
new LocalSqliteRunAttemptLogRetentionRepository(authority);
|
||||||
|
const runLostRetry = new LocalSqliteRunLostRetryRepository(
|
||||||
|
authority,
|
||||||
|
runRepository,
|
||||||
|
);
|
||||||
let pluginPackageInstallsPromise:
|
let pluginPackageInstallsPromise:
|
||||||
| Promise<PluginPackageInstallRepository>
|
| Promise<PluginPackageInstallRepository>
|
||||||
| undefined;
|
| undefined;
|
||||||
@@ -237,6 +243,7 @@ export async function openLocalSqliteRuntimeDatabase(
|
|||||||
executionControl: runRuntimeCapabilities.executionControl,
|
executionControl: runRuntimeCapabilities.executionControl,
|
||||||
completionReceipts: runRuntimeCapabilities.completionReceipts,
|
completionReceipts: runRuntimeCapabilities.completionReceipts,
|
||||||
runAttemptLogRetention,
|
runAttemptLogRetention,
|
||||||
|
runLostRetry,
|
||||||
localSecrets: securityAuthority,
|
localSecrets: securityAuthority,
|
||||||
localSecretAdministration: securityAuthority,
|
localSecretAdministration: securityAuthority,
|
||||||
projectPolicy,
|
projectPolicy,
|
||||||
|
|||||||
@@ -0,0 +1,149 @@
|
|||||||
|
'use strict';
|
||||||
|
|
||||||
|
const assert = require('node:assert/strict');
|
||||||
|
const fs = require('node:fs');
|
||||||
|
const os = require('node:os');
|
||||||
|
const path = require('node:path');
|
||||||
|
const { test } = require('node:test');
|
||||||
|
|
||||||
|
const {
|
||||||
|
migrateLocalSqlitePath,
|
||||||
|
openLocalSqliteRuntimeDatabase,
|
||||||
|
} = require('../dist');
|
||||||
|
|
||||||
|
async function fixture(t) {
|
||||||
|
const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'ql3-lost-retry-'));
|
||||||
|
const databasePath = path.join(directory, 'qinglong3.sqlite');
|
||||||
|
await migrateLocalSqlitePath({ databasePath, profile: 'edge' });
|
||||||
|
const runtime = await openLocalSqliteRuntimeDatabase({
|
||||||
|
databasePath,
|
||||||
|
profile: 'edge',
|
||||||
|
});
|
||||||
|
t.after(async () => {
|
||||||
|
await runtime.close();
|
||||||
|
fs.rmSync(directory, { recursive: true, force: true });
|
||||||
|
});
|
||||||
|
return runtime;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function insertLostRun(runtime, suffix, retryPolicy) {
|
||||||
|
const runId = `run-lost-${suffix}`;
|
||||||
|
const attemptId = `attempt-lost-${suffix}`;
|
||||||
|
await runtime.runRepository.transaction(async (transaction) => {
|
||||||
|
await transaction.insertRun({
|
||||||
|
id: runId,
|
||||||
|
projectId: 'default',
|
||||||
|
taskId: `task-${suffix}`,
|
||||||
|
taskRevision: 'revision-1',
|
||||||
|
triggerType: 'manual',
|
||||||
|
executionOrigin: 'manual',
|
||||||
|
executionOwner: 'runtime',
|
||||||
|
status: 'lost',
|
||||||
|
version: 1,
|
||||||
|
eventSequence: 1,
|
||||||
|
priority: 0,
|
||||||
|
createdAtMs: 100,
|
||||||
|
queuedAtMs: 110,
|
||||||
|
startedAtMs: 120,
|
||||||
|
errorCode: 'LOCAL_RECOVERY_EXECUTION_NOT_RUNNING',
|
||||||
|
errorSummary: 'lost',
|
||||||
|
});
|
||||||
|
await transaction.insertAttempt({
|
||||||
|
id: attemptId,
|
||||||
|
runId,
|
||||||
|
attempt: 1,
|
||||||
|
status: 'lost',
|
||||||
|
executorType: 'local_process',
|
||||||
|
callbackSequence: 0,
|
||||||
|
createdAtMs: 110,
|
||||||
|
startedAtMs: 120,
|
||||||
|
finishedAtMs: 130,
|
||||||
|
errorCode: 'LOCAL_RECOVERY_EXECUTION_NOT_RUNNING',
|
||||||
|
errorSummary: 'lost',
|
||||||
|
});
|
||||||
|
if (retryPolicy) {
|
||||||
|
await transaction.insertRetryPolicy({
|
||||||
|
runId,
|
||||||
|
maxAttempts: 3,
|
||||||
|
retryOnLost: true,
|
||||||
|
safety: 'idempotent',
|
||||||
|
backoffBaseMs: 0,
|
||||||
|
backoffMaxMs: 0,
|
||||||
|
version: 0,
|
||||||
|
createdAtMs: 100,
|
||||||
|
updatedAtMs: 100,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return { runId, attemptId };
|
||||||
|
}
|
||||||
|
|
||||||
|
test('atomically schedules and requeues one safe Local lost Run exactly once', async (t) => {
|
||||||
|
const runtime = await fixture(t);
|
||||||
|
const ids = await insertLostRun(runtime, 'safe', true);
|
||||||
|
|
||||||
|
assert.deepEqual(await runtime.runLostRetry.reconcilePage({ limit: 1 }), {
|
||||||
|
scanned: 1,
|
||||||
|
scheduled: 1,
|
||||||
|
requeued: 0,
|
||||||
|
failed: 0,
|
||||||
|
raced: 0,
|
||||||
|
hasMore: false,
|
||||||
|
});
|
||||||
|
assert.equal(
|
||||||
|
(await runtime.runRepository.findRunById(ids.runId)).status,
|
||||||
|
'retry_wait',
|
||||||
|
);
|
||||||
|
|
||||||
|
assert.deepEqual(await runtime.runLostRetry.reconcilePage({ limit: 1 }), {
|
||||||
|
scanned: 1,
|
||||||
|
scheduled: 0,
|
||||||
|
requeued: 1,
|
||||||
|
failed: 0,
|
||||||
|
raced: 0,
|
||||||
|
hasMore: false,
|
||||||
|
});
|
||||||
|
const run = await runtime.runRepository.findRunById(ids.runId);
|
||||||
|
const attempt = await runtime.runRepository.findLatestAttemptByRunId(
|
||||||
|
ids.runId,
|
||||||
|
);
|
||||||
|
assert.equal(run.status, 'queued');
|
||||||
|
assert.equal(run.version, 4);
|
||||||
|
assert.equal(attempt.attempt, 2);
|
||||||
|
assert.equal(attempt.status, 'claimed');
|
||||||
|
assert.equal(attempt.executorType, 'local_process');
|
||||||
|
assert.deepEqual(
|
||||||
|
(await runtime.runRepository.listEvents(ids.runId)).map(
|
||||||
|
(event) => event.type,
|
||||||
|
),
|
||||||
|
['run.retry_wait', 'run.queued', 'attempt.claimed'],
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
(await runtime.runLostRetry.reconcilePage({ limit: 1 })).scanned,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('fails closed without an admitted retry policy and preserves a bounded page', async (t) => {
|
||||||
|
const runtime = await fixture(t);
|
||||||
|
const first = await insertLostRun(runtime, 'disabled-a', false);
|
||||||
|
await insertLostRun(runtime, 'disabled-b', false);
|
||||||
|
|
||||||
|
const result = await runtime.runLostRetry.reconcilePage({ limit: 1 });
|
||||||
|
assert.deepEqual(result, {
|
||||||
|
scanned: 1,
|
||||||
|
scheduled: 0,
|
||||||
|
requeued: 0,
|
||||||
|
failed: 1,
|
||||||
|
raced: 0,
|
||||||
|
hasMore: true,
|
||||||
|
});
|
||||||
|
assert.equal(
|
||||||
|
(await runtime.runRepository.findRunById(first.runId)).errorCode,
|
||||||
|
'RUN_LOST_RETRY_DISABLED',
|
||||||
|
);
|
||||||
|
await assert.rejects(
|
||||||
|
runtime.runLostRetry.reconcilePage({ limit: 65 }),
|
||||||
|
/page size/,
|
||||||
|
);
|
||||||
|
});
|
||||||
@@ -218,6 +218,12 @@
|
|||||||
"run": [
|
"run": [
|
||||||
"dist/run/run.d.ts"
|
"dist/run/run.d.ts"
|
||||||
],
|
],
|
||||||
|
"run-lost-retry": [
|
||||||
|
"dist/run/clusterRunLostRetry.d.ts"
|
||||||
|
],
|
||||||
|
"cluster-run-lost-retry": [
|
||||||
|
"dist/run/clusterRunLostRetry.d.ts"
|
||||||
|
],
|
||||||
"project-run-list": [
|
"project-run-list": [
|
||||||
"dist/run/projectRunList.d.ts"
|
"dist/run/projectRunList.d.ts"
|
||||||
],
|
],
|
||||||
@@ -265,6 +271,16 @@
|
|||||||
"require": "./dist/run/runRepositoryContract.js",
|
"require": "./dist/run/runRepositoryContract.js",
|
||||||
"default": "./dist/run/runRepositoryContract.js"
|
"default": "./dist/run/runRepositoryContract.js"
|
||||||
},
|
},
|
||||||
|
"./run-lost-retry": {
|
||||||
|
"types": "./dist/run/clusterRunLostRetry.d.ts",
|
||||||
|
"require": "./dist/run/clusterRunLostRetry.js",
|
||||||
|
"default": "./dist/run/clusterRunLostRetry.js"
|
||||||
|
},
|
||||||
|
"./cluster-run-lost-retry": {
|
||||||
|
"types": "./dist/run/clusterRunLostRetry.d.ts",
|
||||||
|
"require": "./dist/run/clusterRunLostRetry.js",
|
||||||
|
"default": "./dist/run/clusterRunLostRetry.js"
|
||||||
|
},
|
||||||
"./project-run-list": {
|
"./project-run-list": {
|
||||||
"types": "./dist/run/projectRunList.d.ts",
|
"types": "./dist/run/projectRunList.d.ts",
|
||||||
"require": "./dist/run/projectRunList.js",
|
"require": "./dist/run/projectRunList.js",
|
||||||
|
|||||||
@@ -1,12 +1,5 @@
|
|||||||
import type {
|
import type { RunAttemptRecord, RunEventRecord, RunRecord } from './run';
|
||||||
RunAttemptRecord,
|
import { runRetryDelayMs, type RunRetryPolicyRecord } from './runRetryPolicy';
|
||||||
RunEventRecord,
|
|
||||||
RunRecord,
|
|
||||||
} from './run';
|
|
||||||
import {
|
|
||||||
runRetryDelayMs,
|
|
||||||
type RunRetryPolicyRecord,
|
|
||||||
} from './runRetryPolicy';
|
|
||||||
|
|
||||||
export const MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE = 64;
|
export const MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE = 64;
|
||||||
|
|
||||||
@@ -62,7 +55,7 @@ export class ClusterRunLostRetryUnavailableError extends Error {
|
|||||||
readonly code = 'CLUSTER_RUN_LOST_RETRY_UNAVAILABLE';
|
readonly code = 'CLUSTER_RUN_LOST_RETRY_UNAVAILABLE';
|
||||||
|
|
||||||
constructor(options?: ErrorOptions) {
|
constructor(options?: ErrorOptions) {
|
||||||
super('Cluster Run lost retry is unavailable', options);
|
super('Run lost retry is unavailable', options);
|
||||||
this.name = 'ClusterRunLostRetryUnavailableError';
|
this.name = 'ClusterRunLostRetryUnavailableError';
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -71,7 +64,7 @@ export class InvalidClusterRunLostRetryTransitionError extends TypeError {
|
|||||||
readonly code = 'CLUSTER_RUN_LOST_RETRY_INVALID';
|
readonly code = 'CLUSTER_RUN_LOST_RETRY_INVALID';
|
||||||
|
|
||||||
constructor(message: string) {
|
constructor(message: string) {
|
||||||
super(`Cluster Run lost retry is invalid: ${message}`);
|
super(`Run lost retry is invalid: ${message}`);
|
||||||
this.name = 'InvalidClusterRunLostRetryTransitionError';
|
this.name = 'InvalidClusterRunLostRetryTransitionError';
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -182,10 +175,7 @@ function transitionTime(
|
|||||||
function finish(
|
function finish(
|
||||||
input: Readonly<ClusterRunLostRetryTransitionInput>,
|
input: Readonly<ClusterRunLostRetryTransitionInput>,
|
||||||
atMs: number,
|
atMs: number,
|
||||||
disposition:
|
disposition: 'failed_disabled' | 'failed_unsafe' | 'failed_exhausted',
|
||||||
| 'failed_disabled'
|
|
||||||
| 'failed_unsafe'
|
|
||||||
| 'failed_exhausted',
|
|
||||||
error: Readonly<{ code: string; summary: string }>,
|
error: Readonly<{ code: string; summary: string }>,
|
||||||
): Readonly<ClusterRunLostRetryTransition> {
|
): Readonly<ClusterRunLostRetryTransition> {
|
||||||
const run = reserve(input.run, 'failed', atMs, error);
|
const run = reserve(input.run, 'failed', atMs, error);
|
||||||
@@ -209,7 +199,7 @@ function finish(
|
|||||||
run,
|
run,
|
||||||
input.attempt.id,
|
input.attempt.id,
|
||||||
'run.failed',
|
'run.failed',
|
||||||
`cluster-lost-retry:${disposition}:${input.attempt.id}`,
|
`run-lost-retry:${disposition}:${input.attempt.id}`,
|
||||||
atMs,
|
atMs,
|
||||||
{
|
{
|
||||||
from_status: input.run.status,
|
from_status: input.run.status,
|
||||||
@@ -271,8 +261,11 @@ export function buildClusterRunLostRetryTransition(
|
|||||||
|
|
||||||
if (run.status === 'lost') {
|
if (run.status === 'lost') {
|
||||||
const nextAttemptAtMs =
|
const nextAttemptAtMs =
|
||||||
Math.max(run.createdAtMs, attempt.createdAtMs, attempt.finishedAtMs ?? 0) +
|
Math.max(
|
||||||
runRetryDelayMs(policy, attempt.attempt);
|
run.createdAtMs,
|
||||||
|
attempt.createdAtMs,
|
||||||
|
attempt.finishedAtMs ?? 0,
|
||||||
|
) + runRetryDelayMs(policy, attempt.attempt);
|
||||||
if (!Number.isSafeInteger(nextAttemptAtMs)) {
|
if (!Number.isSafeInteger(nextAttemptAtMs)) {
|
||||||
return invalid('next Attempt time overflowed');
|
return invalid('next Attempt time overflowed');
|
||||||
}
|
}
|
||||||
@@ -297,7 +290,7 @@ export function buildClusterRunLostRetryTransition(
|
|||||||
nextRun,
|
nextRun,
|
||||||
attempt.id,
|
attempt.id,
|
||||||
'run.retry_wait',
|
'run.retry_wait',
|
||||||
`cluster-lost-retry:scheduled:${attempt.id}`,
|
`run-lost-retry:scheduled:${attempt.id}`,
|
||||||
atMs,
|
atMs,
|
||||||
{
|
{
|
||||||
from_status: 'lost',
|
from_status: 'lost',
|
||||||
@@ -313,10 +306,7 @@ export function buildClusterRunLostRetryTransition(
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
if (
|
if (policy.nextAttemptAtMs === undefined || policy.nextAttemptAtMs > atMs) {
|
||||||
policy.nextAttemptAtMs === undefined ||
|
|
||||||
policy.nextAttemptAtMs > atMs
|
|
||||||
) {
|
|
||||||
return invalid('retry_wait policy is not due');
|
return invalid('retry_wait policy is not due');
|
||||||
}
|
}
|
||||||
const nextAttemptId = identifier('replacement Attempt ID', input.attemptId);
|
const nextAttemptId = identifier('replacement Attempt ID', input.attemptId);
|
||||||
@@ -352,7 +342,7 @@ export function buildClusterRunLostRetryTransition(
|
|||||||
queued,
|
queued,
|
||||||
replacement.id,
|
replacement.id,
|
||||||
'run.queued',
|
'run.queued',
|
||||||
`cluster-lost-retry:queued:${replacement.id}`,
|
`run-lost-retry:queued:${replacement.id}`,
|
||||||
atMs,
|
atMs,
|
||||||
{
|
{
|
||||||
from_status: 'retry_wait',
|
from_status: 'retry_wait',
|
||||||
@@ -365,7 +355,7 @@ export function buildClusterRunLostRetryTransition(
|
|||||||
claimed,
|
claimed,
|
||||||
replacement.id,
|
replacement.id,
|
||||||
'attempt.claimed',
|
'attempt.claimed',
|
||||||
`cluster-lost-retry:attempt-claimed:${replacement.id}`,
|
`run-lost-retry:attempt-claimed:${replacement.id}`,
|
||||||
atMs,
|
atMs,
|
||||||
{
|
{
|
||||||
attempt: replacement.attempt,
|
attempt: replacement.attempt,
|
||||||
@@ -387,11 +377,11 @@ export function normalizeClusterRunLostRetryPageCommand(
|
|||||||
Object.keys(value).length !== 1 ||
|
Object.keys(value).length !== 1 ||
|
||||||
!Object.prototype.hasOwnProperty.call(value, 'limit')
|
!Object.prototype.hasOwnProperty.call(value, 'limit')
|
||||||
) {
|
) {
|
||||||
throw new TypeError('Cluster Run lost retry command is invalid');
|
throw new TypeError('Run lost retry command is invalid');
|
||||||
}
|
}
|
||||||
return Object.freeze({
|
return Object.freeze({
|
||||||
limit: boundedInteger(
|
limit: boundedInteger(
|
||||||
'Cluster Run lost retry page size',
|
'Run lost retry page size',
|
||||||
value.limit,
|
value.limit,
|
||||||
1,
|
1,
|
||||||
MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE,
|
MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE,
|
||||||
@@ -410,40 +400,40 @@ export function normalizeClusterRunLostRetryPageResult(
|
|||||||
Object.keys(value).sort().join(',') !==
|
Object.keys(value).sort().join(',') !==
|
||||||
'failed,hasMore,raced,requeued,scanned,scheduled'
|
'failed,hasMore,raced,requeued,scanned,scheduled'
|
||||||
) {
|
) {
|
||||||
throw new TypeError('Cluster Run lost retry result is invalid');
|
throw new TypeError('Run lost retry result is invalid');
|
||||||
}
|
}
|
||||||
const maximum = boundedInteger(
|
const maximum = boundedInteger(
|
||||||
'Cluster Run lost retry result limit',
|
'Run lost retry result limit',
|
||||||
limit,
|
limit,
|
||||||
1,
|
1,
|
||||||
MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE,
|
MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE,
|
||||||
);
|
);
|
||||||
const scanned = boundedInteger(
|
const scanned = boundedInteger(
|
||||||
'Cluster Run lost retry scanned count',
|
'Run lost retry scanned count',
|
||||||
value.scanned,
|
value.scanned,
|
||||||
0,
|
0,
|
||||||
maximum,
|
maximum,
|
||||||
);
|
);
|
||||||
const scheduled = boundedInteger(
|
const scheduled = boundedInteger(
|
||||||
'Cluster Run lost retry scheduled count',
|
'Run lost retry scheduled count',
|
||||||
value.scheduled,
|
value.scheduled,
|
||||||
0,
|
0,
|
||||||
scanned,
|
scanned,
|
||||||
);
|
);
|
||||||
const requeued = boundedInteger(
|
const requeued = boundedInteger(
|
||||||
'Cluster Run lost retry requeued count',
|
'Run lost retry requeued count',
|
||||||
value.requeued,
|
value.requeued,
|
||||||
0,
|
0,
|
||||||
scanned,
|
scanned,
|
||||||
);
|
);
|
||||||
const failed = boundedInteger(
|
const failed = boundedInteger(
|
||||||
'Cluster Run lost retry failed count',
|
'Run lost retry failed count',
|
||||||
value.failed,
|
value.failed,
|
||||||
0,
|
0,
|
||||||
scanned,
|
scanned,
|
||||||
);
|
);
|
||||||
const raced = boundedInteger(
|
const raced = boundedInteger(
|
||||||
'Cluster Run lost retry raced count',
|
'Run lost retry raced count',
|
||||||
value.raced,
|
value.raced,
|
||||||
0,
|
0,
|
||||||
scanned,
|
scanned,
|
||||||
@@ -452,7 +442,7 @@ export function normalizeClusterRunLostRetryPageResult(
|
|||||||
scheduled + requeued + failed + raced !== scanned ||
|
scheduled + requeued + failed + raced !== scanned ||
|
||||||
typeof value.hasMore !== 'boolean'
|
typeof value.hasMore !== 'boolean'
|
||||||
) {
|
) {
|
||||||
throw new TypeError('Cluster Run lost retry result counts are invalid');
|
throw new TypeError('Run lost retry result counts are invalid');
|
||||||
}
|
}
|
||||||
return Object.freeze({
|
return Object.freeze({
|
||||||
scanned,
|
scanned,
|
||||||
@@ -481,10 +471,10 @@ export class ClusterRunLostRetryCoordinator {
|
|||||||
typeof options !== 'object' ||
|
typeof options !== 'object' ||
|
||||||
Array.isArray(options)
|
Array.isArray(options)
|
||||||
) {
|
) {
|
||||||
throw new TypeError('Cluster Run lost retry coordinator is invalid');
|
throw new TypeError('Run lost retry coordinator is invalid');
|
||||||
}
|
}
|
||||||
this.pageSize = boundedInteger(
|
this.pageSize = boundedInteger(
|
||||||
'Cluster Run lost retry page size',
|
'Run lost retry page size',
|
||||||
options.pageSize ?? 16,
|
options.pageSize ?? 16,
|
||||||
1,
|
1,
|
||||||
MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE,
|
MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE,
|
||||||
@@ -494,9 +484,7 @@ export class ClusterRunLostRetryCoordinator {
|
|||||||
reconcile(): Promise<Readonly<ClusterRunLostRetryPageResult>> {
|
reconcile(): Promise<Readonly<ClusterRunLostRetryPageResult>> {
|
||||||
if (this.inFlight) return this.inFlight;
|
if (this.inFlight) return this.inFlight;
|
||||||
const operation = Promise.resolve()
|
const operation = Promise.resolve()
|
||||||
.then(() =>
|
.then(() => this.repository.reconcilePage({ limit: this.pageSize }))
|
||||||
this.repository.reconcilePage({ limit: this.pageSize }),
|
|
||||||
)
|
|
||||||
.then((result) =>
|
.then((result) =>
|
||||||
normalizeClusterRunLostRetryPageResult(result, this.pageSize),
|
normalizeClusterRunLostRetryPageResult(result, this.pageSize),
|
||||||
)
|
)
|
||||||
@@ -511,3 +499,32 @@ export class ClusterRunLostRetryCoordinator {
|
|||||||
return operation;
|
return operation;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Profile-neutral names are the canonical API for new adapters. The Cluster
|
||||||
|
* names remain exported because early 3.0 incubator consumers used them before
|
||||||
|
* Local Profiles acquired the same reconciliation lifecycle.
|
||||||
|
*/
|
||||||
|
export const MAX_RUN_LOST_RETRY_PAGE_SIZE =
|
||||||
|
MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE;
|
||||||
|
export type RunLostRetryDisposition = ClusterRunLostRetryDisposition;
|
||||||
|
export type RunLostRetryPageCommand = ClusterRunLostRetryPageCommand;
|
||||||
|
export type RunLostRetryPageResult = ClusterRunLostRetryPageResult;
|
||||||
|
export type RunLostRetryRepository = ClusterRunLostRetryRepository;
|
||||||
|
export type RunLostRetryCoordinatorOptions =
|
||||||
|
ClusterRunLostRetryCoordinatorOptions;
|
||||||
|
export type RunLostRetryTransitionInput = ClusterRunLostRetryTransitionInput;
|
||||||
|
export type RunLostRetryTransition = ClusterRunLostRetryTransition;
|
||||||
|
export type RunLostRetryUnavailableError = ClusterRunLostRetryUnavailableError;
|
||||||
|
export type InvalidRunLostRetryTransitionError =
|
||||||
|
InvalidClusterRunLostRetryTransitionError;
|
||||||
|
export type RunLostRetryCoordinator = ClusterRunLostRetryCoordinator;
|
||||||
|
export const RunLostRetryUnavailableError = ClusterRunLostRetryUnavailableError;
|
||||||
|
export const InvalidRunLostRetryTransitionError =
|
||||||
|
InvalidClusterRunLostRetryTransitionError;
|
||||||
|
export const buildRunLostRetryTransition = buildClusterRunLostRetryTransition;
|
||||||
|
export const normalizeRunLostRetryPageCommand =
|
||||||
|
normalizeClusterRunLostRetryPageCommand;
|
||||||
|
export const normalizeRunLostRetryPageResult =
|
||||||
|
normalizeClusterRunLostRetryPageResult;
|
||||||
|
export const RunLostRetryCoordinator = ClusterRunLostRetryCoordinator;
|
||||||
|
|||||||
@@ -0,0 +1,25 @@
|
|||||||
|
'use strict';
|
||||||
|
|
||||||
|
const assert = require('node:assert/strict');
|
||||||
|
const { test } = require('node:test');
|
||||||
|
|
||||||
|
const generic = require('../dist/run/clusterRunLostRetry');
|
||||||
|
|
||||||
|
test('publishes profile-neutral lost retry names without breaking Cluster consumers', () => {
|
||||||
|
assert.equal(
|
||||||
|
generic.MAX_RUN_LOST_RETRY_PAGE_SIZE,
|
||||||
|
generic.MAX_CLUSTER_RUN_LOST_RETRY_PAGE_SIZE,
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
generic.buildRunLostRetryTransition,
|
||||||
|
generic.buildClusterRunLostRetryTransition,
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
generic.RunLostRetryCoordinator,
|
||||||
|
generic.ClusterRunLostRetryCoordinator,
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
generic.RunLostRetryUnavailableError,
|
||||||
|
generic.ClusterRunLostRetryUnavailableError,
|
||||||
|
);
|
||||||
|
});
|
||||||
@@ -540,10 +540,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
|
|||||||
rootSourceFileRoles: localSqlite.rootSourceFileRoles,
|
rootSourceFileRoles: localSqlite.rootSourceFileRoles,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
sourceFiles: 176,
|
sourceFiles: 177,
|
||||||
rootSourceFiles: 1,
|
rootSourceFiles: 1,
|
||||||
rootSourceLines: 31,
|
rootSourceLines: 31,
|
||||||
nestedSourceFiles: 175,
|
nestedSourceFiles: 176,
|
||||||
rootSourceFileRoles: { 'index.ts': 'public_export' },
|
rootSourceFileRoles: { 'index.ts': 'public_export' },
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|||||||
Reference in New Issue
Block a user