feat(ql3): add cluster log retention authority

This commit is contained in:
whyour
2026-08-12 03:38:27 +08:00
parent 2bfa8ca279
commit 40628c843d
20 changed files with 1737 additions and 19 deletions
+11
View File
@@ -11,6 +11,17 @@
最新增量证据(2026-08-12):
- D-291/ADR-0379(进行中,PostgreSQL authority 阶段完成)
Cluster Run Attempt 日志 retention 第一阶段已冻结并实现多副本权威边界,且没有新增 package:共享 claim contract 位于既有
Runtime Core Run log-retention 目录,PostgreSQL v54 增加 durable control 与 immutable tombstone、terminal remote Worker
candidate index 和 `run_attempt_log_retention` capability。候选只允许 runtime-owned、Run/Attempt 双终态、非 lost、canonical
`remote_worker/wlog-*` 且超过 cutoff;每批最多 16 条,通过短 `READ COMMITTED` 事务和 `FOR UPDATE ... SKIP LOCKED` 获取
owner/token/version/expiry fence。S3 网络调用明确位于数据库事务之外;finalize 在第二个短事务中重验完整 fence 与 durable
Run/Attempt identity,写 exact retirement record 后删除 control。retry/manual 持久化失败分类并提供最长 24 小时 backofflease
过期可由其他副本安全接管。`ql3_runtime` 可完整维护 control,但 tombstone 只有 SELECT/INSERT,其他角色保持零权限;repository
已提供 digest/identity 失败关闭的 retention state read,为 Cluster 410 wiring 提供权威来源。migration/schema/readiness/repository
定向 63/63 通过。ADR 暂保持 Proposed;下一阶段继续完成 validated S3 HEAD、ETag/VersionId 条件删除、bounded lifecycle、MinIO
failure matrix 与 PostgreSQL HA failover,全部完整门通过后才接受。
- D-290/ADR-0378(已接受)
Local Run Attempt 日志 retention 已形成真实纵向切片:Runtime Core 增加精确 identity、canonical SHA-256 的 immutable
retirement record、容量压力策略、有界 page/delete budget 与 durable cursor;日志读取在存储前检查 tombstone,并在 missing 后二次
@@ -0,0 +1,74 @@
# ADR-0379Cluster Run Attempt 日志多副本保留与条件删除
- 状态:ProposedPostgreSQL authority 已实现,S3/lifecycle/HA 验收待完成)
- 日期:2026-08-12
- 关联 RFCQL-RFC-0001 D-291
- 前置决策:ADR-0026、ADR-0027、ADR-0377、ADR-0378
## 上下文
Local retention 已证明 durable tombstone、删除后收敛和读取 410 的通用语义,但 Cluster 不能复用 Local cursor + unlink 模型。多个控制面副本可能同时发现同一 remote Worker 日志;对象删除需要跨 S3 网络,PostgreSQL 事务不能覆盖该网络调用;删除响应也可能丢失。若没有 durable ownership fence,副本会重复删除或把另一个副本的结果写成自己的证据;若删除前写 tombstone,则可能把仍存在的对象声明为 retired。
Cluster 还必须保持与低配 Local 部署的物理隔离:本能力不得让 Edge/Standalone closure 引入 PostgreSQL、AWS SDK、额外连接或后台任务。
## 决策
### 1. 不新增 package
共享多副本 claim contract 放在 `runtime-core` 的既有 Run log-retention 目录;PostgreSQL authority 放在 `cluster-postgres`S3 条件删除和调度 lifecycle 放在 `cluster-control`。不创建单文件 retention 微包,也不把 Cluster 依赖加入 Local closure。
### 2. PostgreSQL v54 是唯一 ownership authority
`pg-0055-run-attempt-log-retention` 将 control-core 升至 v54,并声明 `run_attempt_log_retention` capability
- `run_attempt_log_retention_controls` 保存精确 Project/Run/Attempt/Artifact identity、eligibility、claim owner/token/version/expiry、retry time、failure count 与最后失败分类;
- `run_attempt_log_artifact_tombstones` 保存 immutable `qinglong/run-attempt-log-retirement@v1` 标量证据与 canonical digest
- terminal remote Worker Attempt 增加局部候选索引;
- `ql3_runtime` 对 control 拥有 `SELECT/INSERT/UPDATE/DELETE`,对 tombstone 仅拥有 `SELECT/INSERT`,其他运行角色继续零权限。
候选必须同时满足 runtime-owned Run、Run/Attempt 均为非 `lost` 终态、`remote_worker` executor、canonical `wlog-*` identity、Run/Attempt 均超过 retention cutoff 且不存在 tombstone。claim 在短 `READ COMMITTED` 事务中以 `FOR UPDATE ... SKIP LOCKED` 获取;冲突更新再次校验 immutable identity、retry due/lease expiry 与单调 version。
### 3. S3 调用不进入数据库事务
每个副本先取得有界 durable claim,提交并释放 PostgreSQL client 后才执行 S3
1. validated HEAD 校验 content type、metadata identity、checksum、byte length,并取得 ETag 与可用的 VersionId
2. 对 versioned object 使用精确 VersionId,对未版本化对象使用 ETag `If-Match` 条件删除;
3. 412/对象身份变化失败关闭并进入 retry/manual,不得删除新对象;
4. 删除成功或 HEAD 已不存在后,开启第二个短 PostgreSQL 事务;
5. 事务重验 owner/token/version/expiry、terminal Run/Attempt 与 immutable identity,插入 exact tombstone 后删除 control
6. 删除响应丢失时,lease 过期后的新 claim 以 HEAD absent 写入 `already_absent`,最终收敛。
### 4. 有界调度与退避
每轮 claim 不超过 16 条,lease 范围 5 秒至 5 分钟;retry delay 最大 24 小时。副本不得持有跨 sweep cursor,也不得为每个 Attempt 建 timer。调度复用 Cluster control application 既有 lifecycle,并受每轮 claim 数、删除数和 wall-clock budget 共同限制。`artifact_unavailable``artifact_integrity_mismatch``retirement_record_unavailable` 是持久化失败分类;达到策略阈值后转 manual,避免坏对象形成热循环。
### 5. 读取收敛
PostgreSQL claim repository 同时实现 retention state reader。Cluster 日志读取在 S3 前检查 tombstone,并在 S3 missing 后二次检查:已 retired 返回 410;没有 tombstone 的 missing 继续保持 503,而不是伪造 retention。tombstone identity 或 digest 漂移失败关闭。
## 阶段验收
第一阶段已完成 PostgreSQL authority:共享 claim contract、v54 migration、typed Drizzle/schema/readiness、最小权限、短事务 claim、完整 lease fence、retry/manual settlement、exact tombstone finalize/replay、tombstone state read。定向 63 项 migration/schema/readiness/repository 门通过;`runtime-core` 498 项全通过,`cluster-postgres` 302 项通过、1 项条件跳过,`cluster-control` 216 项通过、2 项条件跳过。
阶段收口还通过 18 个 QL3 workspace package 的完整 build/test 门,以及后端兼容回归 1163 项通过、2 项条件跳过、0 失败/取消。v54 readiness 引入的两张表已同步进入 `cluster-control` 测试数据库的最小权限 fixture,避免旧 v53 fixture 把正常启动误判为 `runtime_role_invalid`
结构门保持 18 个 workspace package`singleSourcePackages=[]``shallowSourcePackages=[]`。新 repository 只从明确的 Cluster runtime entrypoint 导出,不扩大 package rootPostgreSQL migration append-only `ordered_ledger` 的 reviewed hard cap 随 pg-0055 由 57 精确推进到 58,不把版本账本伪拆为子目录。
ADR 保持 Proposed,只有以下剩余项全部完成后才转 Accepted:
1. S3 validated HEAD + ETag/VersionId 条件删除和失败矩阵;
2. Cluster service 的 bounded claim/delete/backoff/manual 策略;
3. production composition、lifecycle drain 与读取 410
4. MinIO versioned/unversioned 集成、响应丢失重放;
5. PostgreSQL 18 HA failover 中 lease takeover/tombstone 收敛;
6. 完整 package/backend/boundary/Profile/image gates,证明 Local closure 无 PostgreSQL/AWS SDK 回归。
## 被否决的替代方案
1. **复用 Local durable cursor**:无法形成多副本 ownership,拒绝。
2. **在 PostgreSQL 事务中调用 S3**:把网络停顿扩大为数据库锁和故障域,拒绝。
3. **删除前写 tombstone**:可能把仍存在的对象声明为 retired,拒绝。
4. **无条件 DeleteObject**:无法防止对象身份变化或运维侧替换,拒绝。
5. **授予 tombstone UPDATE/DELETE**:破坏 immutable evidence,拒绝。
6. **新增 Cluster retention 微包**:现有聚合边界足够,且会继续放大 package 碎片化,拒绝。
+4
View File
@@ -379,6 +379,10 @@
| [ADR-0373](./ADR-0373-profile-reachable-runtime-javascript-projection.md) | Profile 可达的 Runtime JavaScript 投影 | Accepted |
| [ADR-0374](./ADR-0374-shared-bounded-task-discovery-http-api.md) | 共享、有界的 Task Discovery HTTP API | Accepted |
| [ADR-0375](./ADR-0375-shared-current-task-point-read-api.md) | 共享的 current Task point-read API | Proposed(设计冻结,实现中) |
| [ADR-0376](./ADR-0376-policy-and-digest-fenced-task-start.md) | Policy 与 digest fenced 的 Task Start | Accepted |
| [ADR-0377](./ADR-0377-profile-aware-run-attempt-log-range-read.md) | Profile-aware Run Attempt 日志 Range 读取 | Accepted |
| [ADR-0378](./ADR-0378-local-run-attempt-log-retention-and-tombstones.md) | Local Run Attempt 日志有界保留与 durable tombstone | Accepted |
| [ADR-0379](./ADR-0379-cluster-run-attempt-log-retention.md) | Cluster Run Attempt 日志多副本保留与条件删除 | ProposedPostgreSQL authority 已完成) |
## 规则
+1 -1
View File
@@ -6,7 +6,7 @@
"reviewedDenseDirectories": [
{
"kind": "ordered_ledger",
"maxDirectSourceFiles": 57,
"maxDirectSourceFiles": 58,
"path": "packages/ql3-cluster-postgres/src/migrations",
"rationale": "PostgreSQL migrations are an append-only version ledger whose ordering and discoverability are safer in one reviewed directory."
},
@@ -137,6 +137,8 @@ function runtimePrivileges() {
tool_invocation_input_artifacts: [true, true, false, false],
tool_invocation_preview_artifacts: [true, true, false, false],
run_attempts: [true, true, true, false],
run_attempt_log_retention_controls: [true, true, true, true],
run_attempt_log_artifact_tombstones: [true, true, false, false],
run_recovery_controls: [true, true, true, false],
worker_sessions: [true, true, true, false],
run_dispatch_leases: [true, true, true, false],
@@ -51,6 +51,8 @@ function runtimePrivileges() {
tool_invocation_input_artifacts: [true, true, false, false],
tool_invocation_preview_artifacts: [true, true, false, false],
run_attempts: [true, true, true, false],
run_attempt_log_retention_controls: [true, true, true, true],
run_attempt_log_artifact_tombstones: [true, true, false, false],
run_recovery_controls: [true, true, true, false],
worker_sessions: [true, true, true, false],
run_dispatch_leases: [true, true, true, false],
@@ -54,6 +54,7 @@ export { PostgresToolResultKeyCatalogReader } from '../tool-execution/toolResult
export { PostgresToolResultRekeyReader } from '../tool-execution/toolResultRekeyRepository';
export * from '../run/runRepository';
export * from '../run/runAttemptLogRetentionClaimRepository';
export * from '../security/projectPolicyRepository';
export * from '../security/apiCredentialRepository';
export * from '../security/securityAuditRepository';
@@ -278,5 +278,10 @@ export const postgresqlMainMigrationManifest: MigrationStreamManifest =
checksum:
'5e3e6b222269f095e0d7a985fdeb0ea154510e59dfe15873192af8c8d603fca3',
}),
Object.freeze({
id: 'pg-0055-run-attempt-log-retention',
checksum:
'c775c65ec03ae3a1606f899064d2d38fa63fd136ce52cbd1b1172c3a51e6bf30',
}),
]),
});
@@ -57,6 +57,7 @@ import { pg0051AutomationManagementBoundaryMigration } from './pg-0051-automatio
import { pg0052AutomationManagementIdentityKeysetLedgerMigration } from './pg-0052-automation-management-identity-keyset-ledger';
import { pg0053PluginPackageWorkflowRunListIndexMigration } from './pg-0053-plugin-package-workflow-run-list-index';
import { pg0054ApprovalManagementBoundaryMigration } from './pg-0054-approval-management-boundary';
import { pg0055RunAttemptLogRetentionMigration } from './pg-0055-run-attempt-log-retention';
export const postgresqlMainMigrationStream: MigrationStreamDefinition<PostgresMigrationContext> =
Object.freeze({
@@ -119,5 +120,6 @@ export const postgresqlMainMigrationStream: MigrationStreamDefinition<PostgresMi
pg0052AutomationManagementIdentityKeysetLedgerMigration,
pg0053PluginPackageWorkflowRunListIndexMigration,
pg0054ApprovalManagementBoundaryMigration,
pg0055RunAttemptLogRetentionMigration,
]),
});
@@ -0,0 +1,189 @@
import { CAPABILITIES_V53 } from './pg-0054-approval-management-boundary';
import { definePostgresSqlMigration } from './sqlMigration';
export const CAPABILITIES_V54 = CAPABILITIES_V53.replace(
'"run_core":1,',
'"run_attempt_log_retention":1,"run_core":1,',
);
export const pg0055RunAttemptLogRetentionMigration =
definePostgresSqlMigration({
id: 'pg-0055-run-attempt-log-retention',
statements: [
`
CREATE TABLE "ql3"."run_attempt_log_retention_controls" (
attempt_id varchar(36) PRIMARY KEY,
project_id varchar(128) NOT NULL,
run_id varchar(36) NOT NULL,
log_artifact_id varchar(36) NOT NULL,
executor_type varchar(32) NOT NULL,
finished_at_ms bigint NOT NULL,
eligible_at_ms bigint NOT NULL,
state varchar(16) NOT NULL,
claim_owner varchar(128),
claim_token varchar(64),
claim_version integer NOT NULL DEFAULT 1,
claim_expires_at_ms bigint,
next_claim_at_ms bigint,
failure_count integer NOT NULL DEFAULT 0,
last_failure_code varchar(64),
created_at_ms bigint NOT NULL,
updated_at_ms bigint NOT NULL,
CONSTRAINT ql3_run_log_retention_control_artifact_key
UNIQUE (log_artifact_id),
CONSTRAINT ql3_run_log_retention_control_identity_check CHECK (
char_length(project_id) BETWEEN 1 AND 128
AND char_length(run_id) BETWEEN 1 AND 36
AND char_length(attempt_id) BETWEEN 1 AND 36
AND log_artifact_id ~ '^wlog-[a-f0-9]{30}$'
AND executor_type = 'remote_worker'
),
CONSTRAINT ql3_run_log_retention_control_time_check CHECK (
finished_at_ms >= 0
AND eligible_at_ms >= finished_at_ms
AND created_at_ms >= 0
AND updated_at_ms >= created_at_ms
),
CONSTRAINT ql3_run_log_retention_control_state_check
CHECK (state IN ('claimed', 'retry', 'manual')),
CONSTRAINT ql3_run_log_retention_control_claim_owner_check CHECK (
claim_owner IS NULL
OR claim_owner ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$'
),
CONSTRAINT ql3_run_log_retention_control_claim_token_check CHECK (
claim_token IS NULL
OR claim_token ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{15,63}$'
),
CONSTRAINT ql3_run_log_retention_control_claim_version_check
CHECK (claim_version BETWEEN 1 AND 2147483647),
CONSTRAINT ql3_run_log_retention_control_claim_expiry_check
CHECK (claim_expires_at_ms IS NULL OR claim_expires_at_ms >= 0),
CONSTRAINT ql3_run_log_retention_control_next_claim_check
CHECK (next_claim_at_ms IS NULL OR next_claim_at_ms >= 0),
CONSTRAINT ql3_run_log_retention_control_failure_count_check
CHECK (failure_count BETWEEN 0 AND 2147483647),
CONSTRAINT ql3_run_log_retention_control_failure_code_check CHECK (
last_failure_code IS NULL
OR last_failure_code IN (
'artifact_unavailable',
'artifact_integrity_mismatch',
'retirement_record_unavailable'
)
),
CONSTRAINT ql3_run_log_retention_control_state_shape_check CHECK (
(
state = 'claimed'
AND claim_owner IS NOT NULL
AND claim_token IS NOT NULL
AND claim_expires_at_ms IS NOT NULL
AND next_claim_at_ms IS NULL
)
OR (
state = 'retry'
AND claim_owner IS NULL
AND claim_token IS NULL
AND claim_expires_at_ms IS NULL
AND next_claim_at_ms IS NOT NULL
AND last_failure_code IS NOT NULL
)
OR (
state = 'manual'
AND claim_owner IS NULL
AND claim_token IS NULL
AND claim_expires_at_ms IS NULL
AND next_claim_at_ms IS NULL
AND last_failure_code IS NOT NULL
)
),
CONSTRAINT ql3_run_log_retention_control_attempt_fk
FOREIGN KEY (attempt_id) REFERENCES "ql3"."run_attempts" (id)
ON DELETE CASCADE,
CONSTRAINT ql3_run_log_retention_control_run_fk
FOREIGN KEY (run_id) REFERENCES "ql3"."runs" (id)
ON DELETE CASCADE
)
`.trim(),
`
CREATE TABLE "ql3"."run_attempt_log_artifact_tombstones" (
log_artifact_id varchar(36) PRIMARY KEY,
project_id varchar(128) NOT NULL,
run_id varchar(36) NOT NULL,
attempt_id varchar(36) NOT NULL,
executor_type varchar(32) NOT NULL,
finished_at_ms bigint NOT NULL,
eligible_at_ms bigint NOT NULL,
retired_at_ms bigint NOT NULL,
disposition varchar(16) NOT NULL,
byte_length bigint NOT NULL,
truncated varchar(16) NOT NULL,
maximum_bytes bigint,
truncation_observed_at_ms bigint,
record_digest char(64) NOT NULL,
CONSTRAINT ql3_run_log_tombstone_attempt_key UNIQUE (attempt_id),
CONSTRAINT ql3_run_log_tombstone_identity_check CHECK (
char_length(project_id) BETWEEN 1 AND 128
AND char_length(run_id) BETWEEN 1 AND 36
AND char_length(attempt_id) BETWEEN 1 AND 36
AND log_artifact_id ~ '^wlog-[a-f0-9]{30}$'
AND executor_type = 'remote_worker'
),
CONSTRAINT ql3_run_log_tombstone_time_check CHECK (
finished_at_ms >= 0
AND eligible_at_ms >= finished_at_ms
AND retired_at_ms >= eligible_at_ms
),
CONSTRAINT ql3_run_log_tombstone_disposition_check CHECK (
disposition IN ('deleted', 'already_absent')
AND (disposition <> 'already_absent' OR byte_length = 0)
),
CONSTRAINT ql3_run_log_tombstone_size_check
CHECK (byte_length BETWEEN 0 AND 1073741824),
CONSTRAINT ql3_run_log_tombstone_truncation_check CHECK (
(
truncated = 'unknown'
AND maximum_bytes IS NULL
AND truncation_observed_at_ms IS NULL
)
OR (
truncated IN ('true', 'false')
AND maximum_bytes >= 1
AND truncation_observed_at_ms >= 0
)
),
CONSTRAINT ql3_run_log_tombstone_digest_check
CHECK (record_digest ~ '^[a-f0-9]{64}$'),
CONSTRAINT ql3_run_log_tombstone_attempt_fk
FOREIGN KEY (attempt_id) REFERENCES "ql3"."run_attempts" (id)
ON DELETE CASCADE,
CONSTRAINT ql3_run_log_tombstone_run_fk
FOREIGN KEY (run_id) REFERENCES "ql3"."runs" (id)
ON DELETE CASCADE
)
`.trim(),
`CREATE INDEX ql3_run_log_retention_retry_idx ON "ql3"."run_attempt_log_retention_controls" (next_claim_at_ms, finished_at_ms, attempt_id) WHERE state = 'retry'`,
`CREATE INDEX ql3_run_log_retention_claim_expiry_idx ON "ql3"."run_attempt_log_retention_controls" (claim_expires_at_ms, finished_at_ms, attempt_id) WHERE state = 'claimed'`,
`CREATE INDEX ql3_run_log_tombstone_retired_idx ON "ql3"."run_attempt_log_artifact_tombstones" (retired_at_ms, attempt_id)`,
`CREATE INDEX ql3_run_log_retention_candidate_idx ON "ql3"."run_attempts" (finished_at_ms, id) WHERE executor_type = 'remote_worker' AND log_artifact_id IS NOT NULL AND status IN ('succeeded', 'failed', 'cancelled', 'timed_out')`,
`GRANT SELECT, INSERT, UPDATE, DELETE ON "ql3"."run_attempt_log_retention_controls" TO ql3_runtime`,
`GRANT SELECT, INSERT ON "ql3"."run_attempt_log_artifact_tombstones" TO ql3_runtime`,
`
DO $ql3$
BEGIN
UPDATE "ql3"."schema_capabilities"
SET contract_version = 54,
migration_id = 'pg-0055-run-attempt-log-retention',
capabilities = '${CAPABILITIES_V54}'::jsonb,
updated_at_ms = floor(extract(epoch FROM transaction_timestamp()) * 1000)::bigint
WHERE contract_name = 'control-core'
AND contract_version = 53
AND migration_id = 'pg-0054-approval-management-boundary'
AND capabilities = '${CAPABILITIES_V53}'::jsonb;
IF NOT FOUND THEN
RAISE EXCEPTION 'control-core capability is not at version 53'
USING ERRCODE = 'check_violation';
END IF;
END
$ql3$
`.trim(),
],
});
@@ -0,0 +1,746 @@
import { randomUUID } from 'node:crypto';
import {
MAX_CLUSTER_RUN_ATTEMPT_LOG_RETENTION_CLAIMS,
MAX_CLUSTER_RUN_ATTEMPT_LOG_RETENTION_LEASE_MS,
MAX_CLUSTER_RUN_ATTEMPT_LOG_RETENTION_RETRY_DELAY_MS,
MIN_CLUSTER_RUN_ATTEMPT_LOG_RETENTION_LEASE_MS,
type ClusterRunAttemptLogRetentionClaim,
type ClusterRunAttemptLogRetentionClaimPage,
type ClusterRunAttemptLogRetentionClaimRepository,
type ClusterRunAttemptLogRetentionFailureCode,
type ClusterRunAttemptLogRetentionSettlement,
} from '@qinglong/runtime-core/cluster-run-attempt-log-retention';
import {
MAX_RUN_ATTEMPT_LOG_RETENTION_MS,
MIN_RUN_ATTEMPT_LOG_RETENTION_MS,
RunAttemptLogRetentionUnavailableError,
normalizeRunAttemptLogRetentionCandidate,
normalizeRunAttemptLogRetirementRecord,
type RunAttemptLogRetentionState,
type RunAttemptLogRetirementRecord,
} from '@qinglong/runtime-core/run-attempt-log-retention';
import type { RunAttemptLogReadIdentity } from '@qinglong/runtime-core/run-attempt-log-read';
import type {
PostgresPool,
PostgresQueryable,
} from '@qinglong/runtime-core';
type Row = Record<string, unknown>;
const CLAIM_SQL = `
WITH observation AS (
SELECT floor(extract(epoch FROM statement_timestamp()) * 1000)::bigint
AS observed_at_ms
), eligible AS (
SELECT attempt.id,
run.project_id,
attempt.run_id,
attempt.log_artifact_id,
attempt.executor_type,
attempt.finished_at_ms,
observation.observed_at_ms
FROM "ql3"."run_attempts" AS attempt
JOIN "ql3"."runs" AS run ON run.id = attempt.run_id
CROSS JOIN observation
LEFT JOIN "ql3"."run_attempt_log_retention_controls" AS control
ON control.attempt_id = attempt.id
LEFT JOIN "ql3"."run_attempt_log_artifact_tombstones" AS tombstone
ON tombstone.attempt_id = attempt.id
OR tombstone.log_artifact_id = attempt.log_artifact_id
WHERE run.execution_owner = 'runtime'
AND run.status IN ('succeeded', 'failed', 'cancelled', 'timed_out')
AND run.finished_at_ms IS NOT NULL
AND attempt.status IN ('succeeded', 'failed', 'cancelled', 'timed_out')
AND attempt.executor_type = 'remote_worker'
AND attempt.finished_at_ms IS NOT NULL
AND attempt.log_artifact_id ~ '^wlog-[a-f0-9]{30}$'
AND attempt.finished_at_ms <= observation.observed_at_ms - $1::bigint
AND run.finished_at_ms <= observation.observed_at_ms - $1::bigint
AND tombstone.attempt_id IS NULL
AND (
control.attempt_id IS NULL
OR (
control.claim_version < 2147483647
AND (
(control.state = 'retry'
AND control.next_claim_at_ms <= observation.observed_at_ms)
OR (control.state = 'claimed'
AND control.claim_expires_at_ms <= observation.observed_at_ms)
)
)
)
ORDER BY attempt.finished_at_ms, attempt.id
FOR UPDATE OF attempt SKIP LOCKED
LIMIT $2
)
INSERT INTO "ql3"."run_attempt_log_retention_controls" (
attempt_id, project_id, run_id, log_artifact_id, executor_type,
finished_at_ms, eligible_at_ms, state, claim_owner, claim_token,
claim_version, claim_expires_at_ms, failure_count, last_failure_code,
created_at_ms, updated_at_ms
)
SELECT id, project_id, run_id, log_artifact_id, executor_type,
finished_at_ms, finished_at_ms + $1::bigint, 'claimed', $3, $4,
1, observed_at_ms + $5::bigint, 0, NULL,
observed_at_ms, observed_at_ms
FROM eligible
ON CONFLICT (attempt_id) DO UPDATE
SET state = 'claimed',
claim_owner = EXCLUDED.claim_owner,
claim_token = EXCLUDED.claim_token,
claim_version = "ql3"."run_attempt_log_retention_controls".claim_version + 1,
claim_expires_at_ms = EXCLUDED.claim_expires_at_ms,
next_claim_at_ms = NULL,
updated_at_ms = GREATEST(
"ql3"."run_attempt_log_retention_controls".updated_at_ms,
EXCLUDED.updated_at_ms
)
WHERE "ql3"."run_attempt_log_retention_controls".project_id = EXCLUDED.project_id
AND "ql3"."run_attempt_log_retention_controls".run_id = EXCLUDED.run_id
AND "ql3"."run_attempt_log_retention_controls".log_artifact_id = EXCLUDED.log_artifact_id
AND "ql3"."run_attempt_log_retention_controls".executor_type = EXCLUDED.executor_type
AND "ql3"."run_attempt_log_retention_controls".finished_at_ms = EXCLUDED.finished_at_ms
AND "ql3"."run_attempt_log_retention_controls".eligible_at_ms = EXCLUDED.eligible_at_ms
AND "ql3"."run_attempt_log_retention_controls".claim_version < 2147483647
AND (
("ql3"."run_attempt_log_retention_controls".state = 'retry'
AND "ql3"."run_attempt_log_retention_controls".next_claim_at_ms <= EXCLUDED.updated_at_ms)
OR ("ql3"."run_attempt_log_retention_controls".state = 'claimed'
AND "ql3"."run_attempt_log_retention_controls".claim_expires_at_ms <= EXCLUDED.updated_at_ms)
)
RETURNING project_id AS "projectId", run_id AS "runId",
attempt_id AS "attemptId", log_artifact_id AS "logArtifactId",
executor_type AS "executorType", finished_at_ms AS "finishedAtMs",
eligible_at_ms AS "eligibleAtMs", updated_at_ms AS "observedAtMs",
claim_owner AS "claimOwner", claim_token AS "claimToken",
claim_version AS "claimVersion", claim_expires_at_ms AS "claimExpiresAtMs",
failure_count AS "failureCount"
`.trim();
const LOCK_CLAIM_SQL = `
WITH observation AS (
SELECT floor(extract(epoch FROM statement_timestamp()) * 1000)::bigint
AS observed_at_ms
)
SELECT control.project_id AS "projectId", control.run_id AS "runId",
control.attempt_id AS "attemptId", control.log_artifact_id AS "logArtifactId",
control.executor_type AS "executorType", control.finished_at_ms AS "finishedAtMs",
control.eligible_at_ms AS "eligibleAtMs",
control.claim_owner AS "claimOwner", control.claim_token AS "claimToken",
control.claim_version AS "claimVersion",
control.claim_expires_at_ms AS "claimExpiresAtMs",
control.failure_count AS "failureCount",
observation.observed_at_ms AS "observedAtMs"
FROM "ql3"."run_attempt_log_retention_controls" AS control
JOIN "ql3"."run_attempts" AS attempt ON attempt.id = control.attempt_id
JOIN "ql3"."runs" AS run ON run.id = control.run_id
CROSS JOIN observation
WHERE control.attempt_id = $1
AND control.state = 'claimed'
AND control.claim_owner = $2
AND control.claim_token = $3
AND control.claim_version = $4
AND control.claim_expires_at_ms = $5::bigint
AND control.claim_expires_at_ms > observation.observed_at_ms
AND attempt.run_id = control.run_id
AND attempt.log_artifact_id = control.log_artifact_id
AND attempt.executor_type = 'remote_worker'
AND attempt.finished_at_ms = control.finished_at_ms
AND attempt.status IN ('succeeded', 'failed', 'cancelled', 'timed_out')
AND run.project_id = control.project_id
AND run.execution_owner = 'runtime'
AND run.status IN ('succeeded', 'failed', 'cancelled', 'timed_out')
FOR UPDATE OF control
`.trim();
const INSERT_TOMBSTONE_SQL = `
INSERT INTO "ql3"."run_attempt_log_artifact_tombstones" (
log_artifact_id, project_id, run_id, attempt_id, executor_type,
finished_at_ms, eligible_at_ms, retired_at_ms, disposition,
byte_length, truncated, maximum_bytes, truncation_observed_at_ms,
record_digest
) VALUES ($1, $2, $3, $4, $5, $6::bigint, $7::bigint, $8::bigint,
$9, $10::bigint, $11, $12::bigint, $13::bigint, $14)
ON CONFLICT DO NOTHING
RETURNING record_digest AS "recordDigest"
`.trim();
const READ_TOMBSTONE_SQL = `
SELECT record_digest AS "recordDigest"
FROM "ql3"."run_attempt_log_artifact_tombstones"
WHERE attempt_id = $1 OR log_artifact_id = $2
`.trim();
const DELETE_CONTROL_SQL = `
DELETE FROM "ql3"."run_attempt_log_retention_controls"
WHERE attempt_id = $1 AND state = 'claimed'
AND claim_owner = $2 AND claim_token = $3 AND claim_version = $4
AND claim_expires_at_ms = $5::bigint
`.trim();
const SET_FAILURE_SQL = `
WITH observation AS (
SELECT floor(extract(epoch FROM statement_timestamp()) * 1000)::bigint
AS observed_at_ms
)
UPDATE "ql3"."run_attempt_log_retention_controls" AS control
SET state = $6,
claim_owner = NULL,
claim_token = NULL,
claim_expires_at_ms = NULL,
next_claim_at_ms = CASE WHEN $6 = 'retry'
THEN observation.observed_at_ms + $7::bigint ELSE NULL END,
failure_count = LEAST(control.failure_count + 1, 2147483647),
last_failure_code = $8,
updated_at_ms = GREATEST(control.updated_at_ms, observation.observed_at_ms)
FROM observation
WHERE control.attempt_id = $1
AND control.state = 'claimed'
AND control.claim_owner = $2
AND control.claim_token = $3
AND control.claim_version = $4
AND control.claim_expires_at_ms = $5::bigint
AND control.claim_expires_at_ms > observation.observed_at_ms
RETURNING control.attempt_id AS "attemptId"
`.trim();
const READ_TOMBSTONE_STATE_SQL = `
SELECT project_id AS "projectId", run_id AS "runId",
attempt_id AS "attemptId", log_artifact_id AS "logArtifactId",
executor_type AS "executorType", finished_at_ms AS "finishedAtMs",
eligible_at_ms AS "eligibleAtMs", retired_at_ms AS "retiredAtMs",
disposition, byte_length AS "byteLength", truncated,
maximum_bytes AS "maximumBytes",
truncation_observed_at_ms AS "truncationObservedAtMs",
record_digest AS "recordDigest"
FROM "ql3"."run_attempt_log_artifact_tombstones"
WHERE log_artifact_id = $1
`.trim();
const FAILURE_CODES = new Set<ClusterRunAttemptLogRetentionFailureCode>([
'artifact_unavailable',
'artifact_integrity_mismatch',
'retirement_record_unavailable',
]);
function integer(
name: string,
value: unknown,
minimum: number,
maximum = Number.MAX_SAFE_INTEGER,
): number {
const converted =
typeof value === 'string' && /^\d+$/.test(value) ? Number(value) : value;
if (
typeof converted !== 'number' ||
!Number.isSafeInteger(converted) ||
converted < minimum ||
converted > maximum
) {
throw new TypeError(`${name} is invalid`);
}
return converted;
}
function identifier(name: string, value: unknown, maximum = 128): string {
if (
typeof value !== 'string' ||
value.length > maximum ||
!/^[A-Za-z0-9][A-Za-z0-9._:-]*$/.test(value)
) {
throw new TypeError(`${name} is invalid`);
}
return value;
}
function optionalInteger(
name: string,
value: unknown,
minimum: number,
): number | undefined {
return value === null ? undefined : integer(name, value, minimum);
}
function readIdentity(
value: Readonly<RunAttemptLogReadIdentity>,
): Readonly<RunAttemptLogReadIdentity> {
const normalized = Object.freeze({
projectId: identifier('Cluster log retention projectId', value?.projectId),
runId: identifier('Cluster log retention runId', value?.runId),
attemptId: identifier('Cluster log retention attemptId', value?.attemptId),
logArtifactId: identifier(
'Cluster log retention logArtifactId',
value?.logArtifactId,
36,
),
});
if (
!value ||
typeof value !== 'object' ||
Array.isArray(value) ||
Object.keys(value).sort().join(',') !==
'attemptId,logArtifactId,projectId,runId' ||
!/^wlog-[a-f0-9]{30}$/.test(normalized.logArtifactId)
) {
throw new TypeError('Cluster Run Attempt log retention identity is invalid');
}
return normalized;
}
function tombstoneFromRow(
row: Row,
): Readonly<RunAttemptLogRetirementRecord> {
const truncated = row.truncated;
if (
truncated !== 'true' &&
truncated !== 'false' &&
truncated !== 'unknown'
) {
throw new TypeError('PostgreSQL retention tombstone is invalid');
}
return normalizeRunAttemptLogRetirementRecord({
schema: 'qinglong/run-attempt-log-retirement@v1',
projectId: identifier('PostgreSQL tombstone projectId', row.projectId),
runId: identifier('PostgreSQL tombstone runId', row.runId),
attemptId: identifier('PostgreSQL tombstone attemptId', row.attemptId),
logArtifactId: identifier(
'PostgreSQL tombstone logArtifactId',
row.logArtifactId,
36,
),
executorType: row.executorType as 'remote_worker',
finishedAtMs: integer(
'PostgreSQL tombstone finishedAtMs',
row.finishedAtMs,
0,
),
eligibleAtMs: integer(
'PostgreSQL tombstone eligibleAtMs',
row.eligibleAtMs,
0,
),
retiredAtMs: integer(
'PostgreSQL tombstone retiredAtMs',
row.retiredAtMs,
0,
),
disposition: row.disposition as 'deleted' | 'already_absent',
byteLength: integer(
'PostgreSQL tombstone byteLength',
row.byteLength,
0,
),
truncation:
truncated === 'unknown'
? Object.freeze({ truncated: 'unknown' as const })
: Object.freeze({
truncated: truncated === 'true',
maximumBytes: optionalInteger(
'PostgreSQL tombstone maximumBytes',
row.maximumBytes,
1,
)!,
observedAtMs: optionalInteger(
'PostgreSQL tombstone truncationObservedAtMs',
row.truncationObservedAtMs,
0,
)!,
}),
recordDigest: identifier(
'PostgreSQL tombstone recordDigest',
row.recordDigest,
64,
),
});
}
function failureCode(
value: unknown,
): ClusterRunAttemptLogRetentionFailureCode {
if (!FAILURE_CODES.has(value as ClusterRunAttemptLogRetentionFailureCode)) {
throw new TypeError('Cluster Run Attempt log retention failure code is invalid');
}
return value as ClusterRunAttemptLogRetentionFailureCode;
}
function claimFromRow(row: Row): Readonly<ClusterRunAttemptLogRetentionClaim> {
const candidate = normalizeRunAttemptLogRetentionCandidate({
projectId: identifier('PostgreSQL retention projectId', row.projectId),
runId: identifier('PostgreSQL retention runId', row.runId),
attemptId: identifier('PostgreSQL retention attemptId', row.attemptId),
logArtifactId: identifier(
'PostgreSQL retention logArtifactId',
row.logArtifactId,
),
executorType: row.executorType as 'remote_worker',
finishedAtMs: integer(
'PostgreSQL retention finishedAtMs',
row.finishedAtMs,
0,
),
});
if (
candidate.executorType !== 'remote_worker' ||
!/^wlog-[a-f0-9]{30}$/.test(candidate.logArtifactId)
) {
throw new TypeError('PostgreSQL retention candidate is invalid');
}
const observedAtMs = integer(
'PostgreSQL retention observedAtMs',
row.observedAtMs,
0,
);
const eligibleAtMs = integer(
'PostgreSQL retention eligibleAtMs',
row.eligibleAtMs,
candidate.finishedAtMs,
);
return Object.freeze({
candidate,
eligibleAtMs,
observedAtMs,
ownerId: identifier('PostgreSQL retention claimOwner', row.claimOwner),
token: identifier('PostgreSQL retention claimToken', row.claimToken, 64),
version: integer(
'PostgreSQL retention claimVersion',
row.claimVersion,
1,
2147483647,
),
expiresAtMs: integer(
'PostgreSQL retention claimExpiresAtMs',
row.claimExpiresAtMs,
observedAtMs + 1,
),
failureCount: integer(
'PostgreSQL retention failureCount',
row.failureCount,
0,
2147483647,
),
});
}
function assertClaim(
value: Readonly<ClusterRunAttemptLogRetentionClaim>,
): Readonly<ClusterRunAttemptLogRetentionClaim> {
const normalized = claimFromRow({
projectId: value?.candidate?.projectId,
runId: value?.candidate?.runId,
attemptId: value?.candidate?.attemptId,
logArtifactId: value?.candidate?.logArtifactId,
executorType: value?.candidate?.executorType,
finishedAtMs: value?.candidate?.finishedAtMs,
eligibleAtMs: value?.eligibleAtMs,
observedAtMs: value?.observedAtMs,
claimOwner: value?.ownerId,
claimToken: value?.token,
claimVersion: value?.version,
claimExpiresAtMs: value?.expiresAtMs,
failureCount: value?.failureCount,
});
if (normalized.expiresAtMs !== value.expiresAtMs) {
throw new TypeError('Cluster Run Attempt log retention claim is invalid');
}
return normalized;
}
function sameClaim(
actual: Readonly<ClusterRunAttemptLogRetentionClaim>,
expected: Readonly<ClusterRunAttemptLogRetentionClaim>,
): boolean {
return (
actual.candidate.projectId === expected.candidate.projectId &&
actual.candidate.runId === expected.candidate.runId &&
actual.candidate.attemptId === expected.candidate.attemptId &&
actual.candidate.logArtifactId === expected.candidate.logArtifactId &&
actual.candidate.executorType === expected.candidate.executorType &&
actual.candidate.finishedAtMs === expected.candidate.finishedAtMs &&
actual.eligibleAtMs === expected.eligibleAtMs &&
actual.ownerId === expected.ownerId &&
actual.token === expected.token &&
actual.version === expected.version &&
actual.expiresAtMs === expected.expiresAtMs
);
}
function sameRecordClaim(
record: Readonly<RunAttemptLogRetirementRecord>,
claim: Readonly<ClusterRunAttemptLogRetentionClaim>,
): boolean {
const candidate = claim.candidate;
return (
record.projectId === candidate.projectId &&
record.runId === candidate.runId &&
record.attemptId === candidate.attemptId &&
record.logArtifactId === candidate.logArtifactId &&
record.executorType === candidate.executorType &&
record.finishedAtMs === candidate.finishedAtMs &&
record.eligibleAtMs === claim.eligibleAtMs
);
}
function unavailable(error: unknown): RunAttemptLogRetentionUnavailableError {
return error instanceof RunAttemptLogRetentionUnavailableError
? error
: new RunAttemptLogRetentionUnavailableError({ cause: error });
}
async function rollback(queryable: PostgresQueryable): Promise<void> {
try {
await queryable.query('ROLLBACK');
} catch {
// Preserve the originating authority failure.
}
}
export class PostgresRunAttemptLogRetentionClaimRepository
implements ClusterRunAttemptLogRetentionClaimRepository
{
constructor(
private readonly pool: PostgresPool,
private readonly createToken: () => string = randomUUID,
) {
if (
!pool ||
typeof pool.query !== 'function' ||
typeof pool.connect !== 'function' ||
typeof createToken !== 'function'
) {
throw new TypeError(
'PostgreSQL Run Attempt log retention repository is invalid',
);
}
}
async inspect(
rawIdentity: Readonly<RunAttemptLogReadIdentity>,
): Promise<RunAttemptLogRetentionState> {
const expected = readIdentity(rawIdentity);
try {
const result = await this.pool.query<Row>(READ_TOMBSTONE_STATE_SQL, [
expected.logArtifactId,
]);
if (result.rows.length === 0) {
return Object.freeze({ status: 'active' as const });
}
if (result.rows.length !== 1) {
throw new TypeError('PostgreSQL retention tombstone is not unique');
}
const record = tombstoneFromRow(result.rows[0]!);
if (
record.projectId !== expected.projectId ||
record.runId !== expected.runId ||
record.attemptId !== expected.attemptId ||
record.logArtifactId !== expected.logArtifactId
) {
throw new TypeError('PostgreSQL retention tombstone identity changed');
}
return Object.freeze({ status: 'retired' as const, record });
} catch (error) {
throw unavailable(error);
}
}
async claim(options: Readonly<{
ownerId: string;
retentionMs: number;
limit: number;
leaseMs: number;
}>): Promise<Readonly<ClusterRunAttemptLogRetentionClaimPage>> {
const ownerId = identifier(
'Cluster Run Attempt log retention ownerId',
options?.ownerId,
);
const retentionMs = integer(
'Cluster Run Attempt log retention duration',
options?.retentionMs,
MIN_RUN_ATTEMPT_LOG_RETENTION_MS,
MAX_RUN_ATTEMPT_LOG_RETENTION_MS,
);
const limit = integer(
'Cluster Run Attempt log retention claim limit',
options?.limit,
1,
MAX_CLUSTER_RUN_ATTEMPT_LOG_RETENTION_CLAIMS,
);
const leaseMs = integer(
'Cluster Run Attempt log retention lease',
options?.leaseMs,
MIN_CLUSTER_RUN_ATTEMPT_LOG_RETENTION_LEASE_MS,
MAX_CLUSTER_RUN_ATTEMPT_LOG_RETENTION_LEASE_MS,
);
const token = identifier(
'Cluster Run Attempt log retention generated token',
this.createToken(),
64,
);
if (token.length < 16) {
throw new TypeError(
'Cluster Run Attempt log retention generated token is invalid',
);
}
const client = await this.pool.connect().catch((error: unknown) => {
throw unavailable(error);
});
let transactionOpen = false;
try {
await client.query('BEGIN ISOLATION LEVEL READ COMMITTED');
transactionOpen = true;
await client.query("SET LOCAL statement_timeout = '5000ms'");
await client.query("SET LOCAL lock_timeout = '1000ms'");
const result = await client.query<Row>(CLAIM_SQL, [
retentionMs,
limit,
ownerId,
token,
leaseMs,
]);
const claims = result.rows.map(claimFromRow);
if (claims.length > limit) {
throw new TypeError('PostgreSQL retention claim bound was violated');
}
await client.query('COMMIT');
transactionOpen = false;
return Object.freeze({
claims: Object.freeze(claims),
hasMore: claims.length === limit,
});
} catch (error) {
if (transactionOpen) await rollback(client);
throw unavailable(error);
} finally {
client.release();
}
}
async settle(
rawClaim: Readonly<ClusterRunAttemptLogRetentionClaim>,
settlement: Readonly<ClusterRunAttemptLogRetentionSettlement>,
): Promise<'settled' | 'fenced'> {
const claim = assertClaim(rawClaim);
if (!settlement || typeof settlement !== 'object') {
throw new TypeError(
'Cluster Run Attempt log retention settlement is invalid',
);
}
if (settlement.status === 'retired') {
return this.recordRetirement(claim, settlement.record);
}
if (settlement.status !== 'retry' && settlement.status !== 'manual') {
throw new TypeError(
'Cluster Run Attempt log retention settlement is invalid',
);
}
const delayMs =
settlement.status === 'retry'
? integer(
'Cluster Run Attempt log retention retry delay',
settlement.delayMs,
0,
MAX_CLUSTER_RUN_ATTEMPT_LOG_RETENTION_RETRY_DELAY_MS,
)
: 0;
const code = failureCode(settlement.failureCode);
try {
const result = await this.pool.query<Row>(SET_FAILURE_SQL, [
claim.candidate.attemptId,
claim.ownerId,
claim.token,
claim.version,
claim.expiresAtMs,
settlement.status,
delayMs,
code,
]);
return result.rows.length === 1 ? 'settled' : 'fenced';
} catch (error) {
throw unavailable(error);
}
}
private async recordRetirement(
claim: Readonly<ClusterRunAttemptLogRetentionClaim>,
rawRecord: Readonly<RunAttemptLogRetirementRecord>,
): Promise<'settled' | 'fenced'> {
const record = normalizeRunAttemptLogRetirementRecord(rawRecord);
if (!sameRecordClaim(record, claim)) {
throw new TypeError(
'Cluster Run Attempt log retirement record does not match its claim',
);
}
const client = await this.pool.connect().catch((error: unknown) => {
throw unavailable(error);
});
let transactionOpen = false;
try {
await client.query('BEGIN ISOLATION LEVEL READ COMMITTED');
transactionOpen = true;
await client.query("SET LOCAL statement_timeout = '5000ms'");
await client.query("SET LOCAL lock_timeout = '1000ms'");
const locked = await client.query<Row>(LOCK_CLAIM_SQL, [
claim.candidate.attemptId,
claim.ownerId,
claim.token,
claim.version,
claim.expiresAtMs,
]);
if (locked.rows.length !== 1) {
await client.query('COMMIT');
transactionOpen = false;
return 'fenced';
}
if (!sameClaim(claimFromRow(locked.rows[0]!), claim)) {
throw new TypeError('PostgreSQL retention claim authority changed');
}
const truncation = record.truncation;
const inserted = await client.query<Row>(INSERT_TOMBSTONE_SQL, [
record.logArtifactId,
record.projectId,
record.runId,
record.attemptId,
record.executorType,
record.finishedAtMs,
record.eligibleAtMs,
record.retiredAtMs,
record.disposition,
record.byteLength,
String(truncation.truncated),
truncation.maximumBytes ?? null,
truncation.observedAtMs ?? null,
record.recordDigest,
]);
if (inserted.rows.length === 0) {
const existing = await client.query<Row>(READ_TOMBSTONE_SQL, [
record.attemptId,
record.logArtifactId,
]);
if (
existing.rows.length !== 1 ||
existing.rows[0]?.recordDigest !== record.recordDigest
) {
throw new TypeError('PostgreSQL retention tombstone conflicts');
}
}
const removed = await client.query<Row>(DELETE_CONTROL_SQL, [
claim.candidate.attemptId,
claim.ownerId,
claim.token,
claim.version,
claim.expiresAtMs,
]);
if (removed.rowCount !== 1) {
throw new TypeError('PostgreSQL retention claim fence was lost');
}
await client.query('COMMIT');
transactionOpen = false;
return 'settled';
} catch (error) {
if (transactionOpen) await rollback(client);
throw unavailable(error);
} finally {
client.release();
}
}
}
@@ -4474,6 +4474,162 @@ export const runAttempts = ql3Schema.table(
index('ql3_run_attempts_lease_idx')
.on(table.leaseExpiresAtMs, table.id)
.where(sql`${table.leaseExpiresAtMs} is not null`),
index('ql3_run_log_retention_candidate_idx')
.on(table.finishedAtMs, table.id)
.where(
sql`${table.executorType} = 'remote_worker' and ${table.logArtifactId} is not null and ${table.status} in ('succeeded', 'failed', 'cancelled', 'timed_out')`,
),
],
);
export const runAttemptLogRetentionControls = ql3Schema.table(
'run_attempt_log_retention_controls',
{
attemptId: varchar('attempt_id', { length: 36 }).primaryKey(),
projectId: varchar('project_id', { length: 128 }).notNull(),
runId: varchar('run_id', { length: 36 }).notNull(),
logArtifactId: varchar('log_artifact_id', { length: 36 }).notNull(),
executorType: varchar('executor_type', { length: 32 }).notNull(),
finishedAtMs: bigint('finished_at_ms', { mode: 'number' }).notNull(),
eligibleAtMs: bigint('eligible_at_ms', { mode: 'number' }).notNull(),
state: varchar('state', { length: 16 }).notNull(),
claimOwner: varchar('claim_owner', { length: 128 }),
claimToken: varchar('claim_token', { length: 64 }),
claimVersion: integer('claim_version').default(1).notNull(),
claimExpiresAtMs: bigint('claim_expires_at_ms', { mode: 'number' }),
nextClaimAtMs: bigint('next_claim_at_ms', { mode: 'number' }),
failureCount: integer('failure_count').default(0).notNull(),
lastFailureCode: varchar('last_failure_code', { length: 64 }),
createdAtMs: bigint('created_at_ms', { mode: 'number' }).notNull(),
updatedAtMs: bigint('updated_at_ms', { mode: 'number' }).notNull(),
},
(table) => [
uniqueIndex('ql3_run_log_retention_control_artifact_key').on(
table.logArtifactId,
),
check(
'ql3_run_log_retention_control_identity_check',
sql`char_length(${table.projectId}) between 1 and 128 and char_length(${table.runId}) between 1 and 36 and char_length(${table.attemptId}) between 1 and 36 and ${table.logArtifactId} ~ '^wlog-[a-f0-9]{30}$' and ${table.executorType} = 'remote_worker'`,
),
check(
'ql3_run_log_retention_control_time_check',
sql`${table.finishedAtMs} >= 0 and ${table.eligibleAtMs} >= ${table.finishedAtMs} and ${table.createdAtMs} >= 0 and ${table.updatedAtMs} >= ${table.createdAtMs}`,
),
check(
'ql3_run_log_retention_control_state_check',
sql`${table.state} in ('claimed', 'retry', 'manual')`,
),
check(
'ql3_run_log_retention_control_claim_owner_check',
sql`${table.claimOwner} is null or ${table.claimOwner} ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$'`,
),
check(
'ql3_run_log_retention_control_claim_token_check',
sql`${table.claimToken} is null or ${table.claimToken} ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{15,63}$'`,
),
check(
'ql3_run_log_retention_control_claim_version_check',
sql`${table.claimVersion} between 1 and 2147483647`,
),
check(
'ql3_run_log_retention_control_claim_expiry_check',
sql`${table.claimExpiresAtMs} is null or ${table.claimExpiresAtMs} >= 0`,
),
check(
'ql3_run_log_retention_control_next_claim_check',
sql`${table.nextClaimAtMs} is null or ${table.nextClaimAtMs} >= 0`,
),
check(
'ql3_run_log_retention_control_failure_count_check',
sql`${table.failureCount} between 0 and 2147483647`,
),
check(
'ql3_run_log_retention_control_failure_code_check',
sql`${table.lastFailureCode} is null or ${table.lastFailureCode} in ('artifact_unavailable', 'artifact_integrity_mismatch', 'retirement_record_unavailable')`,
),
check(
'ql3_run_log_retention_control_state_shape_check',
sql`(${table.state} = 'claimed' and ${table.claimOwner} is not null and ${table.claimToken} is not null and ${table.claimExpiresAtMs} is not null and ${table.nextClaimAtMs} is null) or (${table.state} = 'retry' and ${table.claimOwner} is null and ${table.claimToken} is null and ${table.claimExpiresAtMs} is null and ${table.nextClaimAtMs} is not null and ${table.lastFailureCode} is not null) or (${table.state} = 'manual' and ${table.claimOwner} is null and ${table.claimToken} is null and ${table.claimExpiresAtMs} is null and ${table.nextClaimAtMs} is null and ${table.lastFailureCode} is not null)`,
),
foreignKey({
name: 'ql3_run_log_retention_control_attempt_fk',
columns: [table.attemptId],
foreignColumns: [runAttempts.id],
}).onDelete('cascade'),
foreignKey({
name: 'ql3_run_log_retention_control_run_fk',
columns: [table.runId],
foreignColumns: [runs.id],
}).onDelete('cascade'),
index('ql3_run_log_retention_retry_idx')
.on(table.nextClaimAtMs, table.finishedAtMs, table.attemptId)
.where(sql`${table.state} = 'retry'`),
index('ql3_run_log_retention_claim_expiry_idx')
.on(table.claimExpiresAtMs, table.finishedAtMs, table.attemptId)
.where(sql`${table.state} = 'claimed'`),
],
);
export const runAttemptLogArtifactTombstones = ql3Schema.table(
'run_attempt_log_artifact_tombstones',
{
logArtifactId: varchar('log_artifact_id', { length: 36 }).primaryKey(),
projectId: varchar('project_id', { length: 128 }).notNull(),
runId: varchar('run_id', { length: 36 }).notNull(),
attemptId: varchar('attempt_id', { length: 36 }).notNull(),
executorType: varchar('executor_type', { length: 32 }).notNull(),
finishedAtMs: bigint('finished_at_ms', { mode: 'number' }).notNull(),
eligibleAtMs: bigint('eligible_at_ms', { mode: 'number' }).notNull(),
retiredAtMs: bigint('retired_at_ms', { mode: 'number' }).notNull(),
disposition: varchar('disposition', { length: 16 }).notNull(),
byteLength: bigint('byte_length', { mode: 'number' }).notNull(),
truncated: varchar('truncated', { length: 16 }).notNull(),
maximumBytes: bigint('maximum_bytes', { mode: 'number' }),
truncationObservedAtMs: bigint('truncation_observed_at_ms', {
mode: 'number',
}),
recordDigest: char('record_digest', { length: 64 }).notNull(),
},
(table) => [
uniqueIndex('ql3_run_log_tombstone_attempt_key').on(table.attemptId),
check(
'ql3_run_log_tombstone_identity_check',
sql`char_length(${table.projectId}) between 1 and 128 and char_length(${table.runId}) between 1 and 36 and char_length(${table.attemptId}) between 1 and 36 and ${table.logArtifactId} ~ '^wlog-[a-f0-9]{30}$' and ${table.executorType} = 'remote_worker'`,
),
check(
'ql3_run_log_tombstone_time_check',
sql`${table.finishedAtMs} >= 0 and ${table.eligibleAtMs} >= ${table.finishedAtMs} and ${table.retiredAtMs} >= ${table.eligibleAtMs}`,
),
check(
'ql3_run_log_tombstone_disposition_check',
sql`${table.disposition} in ('deleted', 'already_absent') and (${table.disposition} <> 'already_absent' or ${table.byteLength} = 0)`,
),
check(
'ql3_run_log_tombstone_size_check',
sql`${table.byteLength} between 0 and 1073741824`,
),
check(
'ql3_run_log_tombstone_truncation_check',
sql`(${table.truncated} = 'unknown' and ${table.maximumBytes} is null and ${table.truncationObservedAtMs} is null) or (${table.truncated} in ('true', 'false') and ${table.maximumBytes} >= 1 and ${table.truncationObservedAtMs} >= 0)`,
),
check(
'ql3_run_log_tombstone_digest_check',
sql`${table.recordDigest} ~ '^[a-f0-9]{64}$'`,
),
foreignKey({
name: 'ql3_run_log_tombstone_attempt_fk',
columns: [table.attemptId],
foreignColumns: [runAttempts.id],
}).onDelete('cascade'),
foreignKey({
name: 'ql3_run_log_tombstone_run_fk',
columns: [table.runId],
foreignColumns: [runs.id],
}).onDelete('cascade'),
index('ql3_run_log_tombstone_retired_idx').on(
table.retiredAtMs,
table.attemptId,
),
],
);
@@ -5700,6 +5856,8 @@ export const ql3PostgresTables = [
toolExecutionResultRekeyHeads,
toolResultKeyRetirementReceipts,
runAttempts,
runAttemptLogRetentionControls,
runAttemptLogArtifactTombstones,
workerSessions,
runDispatchLeases,
workerCredentials,
@@ -15,12 +15,13 @@ export interface PostgresSchemaContractFunction {
export interface PostgresSchemaContract {
readonly schema: 'ql3';
readonly contractName: 'control-core';
readonly contractVersion: 53;
readonly migrationId: 'pg-0054-approval-management-boundary';
readonly contractVersion: 54;
readonly migrationId: 'pg-0055-run-attempt-log-retention';
readonly minimumServerMajor: 16;
readonly maximumServerMajor: 18;
readonly capabilities: Readonly<{
run_core: 1;
run_attempt_log_retention: 1;
run_dispatch_lease: 1;
run_retry_policy: 1;
project_policy: 1;
@@ -100,8 +101,8 @@ export const postgresqlControlSchemaContract: PostgresSchemaContract =
Object.freeze({
schema: 'ql3',
contractName: 'control-core',
contractVersion: 53,
migrationId: 'pg-0054-approval-management-boundary',
contractVersion: 54,
migrationId: 'pg-0055-run-attempt-log-retention',
minimumServerMajor: 16,
maximumServerMajor: 18,
capabilities: Object.freeze({
@@ -141,6 +142,7 @@ export const postgresqlControlSchemaContract: PostgresSchemaContract =
project_policy: 1,
project_tool_definition_snapshot: 1,
run_core: 1,
run_attempt_log_retention: 1,
run_dispatch_lease: 1,
run_retry_policy: 1,
security_audit: 1,
@@ -1166,6 +1168,41 @@ export const postgresqlControlSchemaContract: PostgresSchemaContract =
'error_code',
'error_summary',
]),
table('run_attempt_log_retention_controls', [
'attempt_id',
'project_id',
'run_id',
'log_artifact_id',
'executor_type',
'finished_at_ms',
'eligible_at_ms',
'state',
'claim_owner',
'claim_token',
'claim_version',
'claim_expires_at_ms',
'next_claim_at_ms',
'failure_count',
'last_failure_code',
'created_at_ms',
'updated_at_ms',
]),
table('run_attempt_log_artifact_tombstones', [
'log_artifact_id',
'project_id',
'run_id',
'attempt_id',
'executor_type',
'finished_at_ms',
'eligible_at_ms',
'retired_at_ms',
'disposition',
'byte_length',
'truncated',
'maximum_bytes',
'truncation_observed_at_ms',
'record_digest',
]),
table('worker_sessions', [
'worker_id',
'session_id',
@@ -1594,6 +1631,14 @@ export const postgresqlControlSchemaContract: PostgresSchemaContract =
'ql3_run_attempts_dispatch_candidates_idx',
'ql3_run_attempts_recovery_idx',
'ql3_run_attempts_lease_idx',
'run_attempt_log_retention_controls_pkey',
'ql3_run_log_retention_control_artifact_key',
'ql3_run_log_retention_retry_idx',
'ql3_run_log_retention_claim_expiry_idx',
'run_attempt_log_artifact_tombstones_pkey',
'ql3_run_log_tombstone_attempt_key',
'ql3_run_log_tombstone_retired_idx',
'ql3_run_log_retention_candidate_idx',
'worker_sessions_pkey',
'ql3_worker_sessions_available_idx',
'run_dispatch_leases_pkey',
@@ -1913,6 +1958,23 @@ export const postgresqlControlSchemaContract: PostgresSchemaContract =
'ql3_run_attempts_created_at_check',
'ql3_run_attempts_started_at_check',
'ql3_run_attempts_finished_at_check',
'ql3_run_log_retention_control_identity_check',
'ql3_run_log_retention_control_time_check',
'ql3_run_log_retention_control_state_check',
'ql3_run_log_retention_control_claim_owner_check',
'ql3_run_log_retention_control_claim_token_check',
'ql3_run_log_retention_control_claim_version_check',
'ql3_run_log_retention_control_claim_expiry_check',
'ql3_run_log_retention_control_next_claim_check',
'ql3_run_log_retention_control_failure_count_check',
'ql3_run_log_retention_control_failure_code_check',
'ql3_run_log_retention_control_state_shape_check',
'ql3_run_log_tombstone_identity_check',
'ql3_run_log_tombstone_time_check',
'ql3_run_log_tombstone_disposition_check',
'ql3_run_log_tombstone_size_check',
'ql3_run_log_tombstone_truncation_check',
'ql3_run_log_tombstone_digest_check',
'ql3_worker_sessions_worker_id_check',
'ql3_worker_sessions_session_id_check',
'ql3_worker_sessions_generation_check',
@@ -2211,6 +2273,10 @@ export const postgresqlControlSchemaContract: PostgresSchemaContract =
'ql3_result_retirement_catalog_fk',
'ql3_run_attempts_run_fk',
'ql3_run_attempts_step_run_fk',
'ql3_run_log_retention_control_attempt_fk',
'ql3_run_log_retention_control_run_fk',
'ql3_run_log_tombstone_attempt_fk',
'ql3_run_log_tombstone_run_fk',
'ql3_run_dispatch_leases_attempt_fk',
'ql3_run_dispatch_leases_run_fk',
'ql3_run_dispatch_leases_worker_fk',
@@ -539,6 +539,18 @@ const REQUIRED_RUNTIME_PRIVILEGES = Object.freeze({
update: true,
delete: false,
}),
run_attempt_log_retention_controls: Object.freeze({
select: true,
insert: true,
update: true,
delete: true,
}),
run_attempt_log_artifact_tombstones: Object.freeze({
select: true,
insert: true,
update: false,
delete: false,
}),
worker_sessions: Object.freeze({
select: true,
insert: true,
@@ -1034,6 +1046,18 @@ const REQUIRED_ADMIN_PRIVILEGES = Object.freeze({
update: false,
delete: false,
}),
run_attempt_log_retention_controls: Object.freeze({
select: false,
insert: false,
update: false,
delete: false,
}),
run_attempt_log_artifact_tombstones: Object.freeze({
select: false,
insert: false,
update: false,
delete: false,
}),
worker_sessions: Object.freeze({
select: false,
insert: false,
@@ -103,6 +103,7 @@ test('defines the immutable PostgreSQL capability and Run core stream', async ()
'pg-0052-automation-management-identity-keyset-ledger',
'pg-0053-plugin-package-workflow-run-list-index',
'pg-0054-approval-management-boundary',
'pg-0055-run-attempt-log-retention',
],
);
for (const migration of postgresqlMainMigrationStream.migrations) {
@@ -508,6 +509,11 @@ test('freezes every published PostgreSQL migration checksum', () => {
checksum:
'5e3e6b222269f095e0d7a985fdeb0ea154510e59dfe15873192af8c8d603fca3',
},
{
id: 'pg-0055-run-attempt-log-retention',
checksum:
'c775c65ec03ae3a1606f899064d2d38fa63fd136ce52cbd1b1172c3a51e6bf30',
},
];
assert.deepEqual(
postgresqlMainMigrationStream.migrations.map(({ id, checksum }) => ({
@@ -1875,3 +1881,26 @@ test('advances capability v53 with isolated human Approval management authority'
/migration_id = 'pg-0053-plugin-package-workflow-run-list-index'/,
);
});
test('advances capability v54 with durable Cluster log retention authority', async () => {
const migration = migrationById('pg-0055-run-attempt-log-retention');
const statements = [];
await migration.up({
async query(statement) {
statements.push(statement);
return { rows: [] };
},
});
const sql = statements.join('\n');
assert.match(sql, /run_attempt_log_retention_controls/);
assert.match(sql, /run_attempt_log_artifact_tombstones/);
assert.match(sql, /FOR UPDATE|SKIP LOCKED|claim_expires_at_ms/);
assert.match(sql, /TO ql3_runtime/);
assert.match(sql, /contract_version = 54/);
assert.match(sql, /"run_attempt_log_retention":1/);
assert.match(sql, /contract_version = 53/);
assert.match(
sql,
/migration_id = 'pg-0054-approval-management-boundary'/,
);
});
@@ -58,6 +58,8 @@ function validPrivileges() {
tool_invocation_input_artifacts: [true, true, false, false],
tool_invocation_preview_artifacts: [true, true, false, false],
run_attempts: [true, true, true, false],
run_attempt_log_retention_controls: [true, true, true, true],
run_attempt_log_artifact_tombstones: [true, true, false, false],
worker_sessions: [true, true, true, false],
run_dispatch_leases: [true, true, true, false],
worker_credentials: [false, false, false, false],
@@ -178,6 +180,8 @@ function validAdminPrivileges() {
tool_invocation_input_artifacts: [false, false, false, false],
tool_invocation_preview_artifacts: [false, false, false, false],
run_attempts: [false, false, false, false],
run_attempt_log_retention_controls: [false, false, false, false],
run_attempt_log_artifact_tombstones: [false, false, false, false],
worker_sessions: [false, false, false, false],
run_dispatch_leases: [false, false, false, false],
worker_credentials: [true, true, true, false],
@@ -692,7 +696,7 @@ test('accepts the exact PostgreSQL control schema and least-privilege runtime ro
serverMajor: 16,
currentUser: 'ql3_runtime',
contractName: 'control-core',
contractVersion: 53,
contractVersion: 54,
migrationIds: [
'pg-0001-schema-capability',
'pg-0002-run-core',
@@ -748,6 +752,7 @@ test('accepts the exact PostgreSQL control schema and least-privilege runtime ro
'pg-0052-automation-management-identity-keyset-ledger',
'pg-0053-plugin-package-workflow-run-list-index',
'pg-0054-approval-management-boundary',
'pg-0055-run-attempt-log-retention',
],
});
});
@@ -778,10 +783,10 @@ test('accepts the exact schema and isolated least-privilege admin role', async (
}),
);
assert.equal(report.currentUser, 'ql3_admin');
assert.equal(report.contractVersion, 53);
assert.equal(report.contractVersion, 54);
assert.equal(
report.migrationIds.at(-1),
'pg-0054-approval-management-boundary',
'pg-0055-run-attempt-log-retention',
);
});
@@ -794,10 +799,10 @@ test('accepts the isolated least-privilege automation manager role', async () =>
}),
);
assert.equal(report.currentUser, 'ql3_automation_manager');
assert.equal(report.contractVersion, 53);
assert.equal(report.contractVersion, 54);
assert.equal(
report.migrationIds.at(-1),
'pg-0054-approval-management-boundary',
'pg-0055-run-attempt-log-retention',
);
const widened = automationManagerPrivileges();
@@ -826,10 +831,10 @@ test('accepts the isolated least-privilege human Approval manager role', async (
}),
);
assert.equal(report.currentUser, 'ql3_approval_manager');
assert.equal(report.contractVersion, 53);
assert.equal(report.contractVersion, 54);
assert.equal(
report.migrationIds.at(-1),
'pg-0054-approval-management-boundary',
'pg-0055-run-attempt-log-retention',
);
const widened = approvalManagerPrivileges();
@@ -936,10 +941,10 @@ test('accepts the exact schema and isolated Worker ingress role', async () => {
}),
);
assert.equal(report.currentUser, 'ql3_worker_ingress');
assert.equal(report.contractVersion, 53);
assert.equal(report.contractVersion, 54);
assert.equal(
report.migrationIds.at(-1),
'pg-0054-approval-management-boundary',
'pg-0055-run-attempt-log-retention',
);
});
@@ -0,0 +1,330 @@
'use strict';
const assert = require('node:assert/strict');
const { test } = require('node:test');
const {
createRunAttemptLogRetirementRecord,
RunAttemptLogRetentionUnavailableError,
} = require('@qinglong/runtime-core/run-attempt-log-retention');
const {
PostgresRunAttemptLogRetentionClaimRepository,
} = require('../dist/entrypoints/runtime');
const TOKEN = '00000000-0000-4000-8000-000000000055';
const ARTIFACT_ID = `wlog-${'a'.repeat(30)}`;
function claimRow(overrides = {}) {
return {
projectId: 'project-1',
runId: 'run-1',
attemptId: 'attempt-1',
logArtifactId: ARTIFACT_ID,
executorType: 'remote_worker',
finishedAtMs: '1000',
eligibleAtMs: '61000',
observedAtMs: '70000',
claimOwner: 'replica-a',
claimToken: TOKEN,
claimVersion: 1,
claimExpiresAtMs: '100000',
failureCount: 0,
...overrides,
};
}
function claim() {
return Object.freeze({
candidate: Object.freeze({
projectId: 'project-1',
runId: 'run-1',
attemptId: 'attempt-1',
logArtifactId: ARTIFACT_ID,
executorType: 'remote_worker',
finishedAtMs: 1000,
}),
eligibleAtMs: 61000,
observedAtMs: 70000,
ownerId: 'replica-a',
token: TOKEN,
version: 1,
expiresAtMs: 100000,
failureCount: 0,
});
}
test('claims one bounded remote log page under a short database lease', async () => {
const calls = [];
let released = false;
const repository = new PostgresRunAttemptLogRetentionClaimRepository(
{
async connect() {
return {
async query(text, values = []) {
calls.push({ text, values });
if (text.includes('FOR UPDATE OF attempt SKIP LOCKED')) {
return { rows: [claimRow()], rowCount: 1 };
}
return { rows: [], rowCount: 0 };
},
release() {
released = true;
},
};
},
async query() {
throw new Error('pool query not expected');
},
},
() => TOKEN,
);
const page = await repository.claim({
ownerId: 'replica-a',
retentionMs: 60000,
limit: 4,
leaseMs: 30000,
});
assert.deepEqual(page, { claims: [claim()], hasMore: false });
assert.deepEqual(
calls.map(({ text }) => text.split('\n', 1)[0]),
[
'BEGIN ISOLATION LEVEL READ COMMITTED',
"SET LOCAL statement_timeout = '5000ms'",
"SET LOCAL lock_timeout = '1000ms'",
'WITH observation AS (',
'COMMIT',
],
);
assert.deepEqual(calls[3].values, [
60000,
4,
'replica-a',
TOKEN,
30000,
]);
assert.match(calls[3].text, /ON CONFLICT \(attempt_id\) DO UPDATE/);
assert.match(calls[3].text, /claim_expires_at_ms <= EXCLUDED\.updated_at_ms/);
assert.equal(released, true);
});
test('fences retry settlement by owner token version and database expiry', async () => {
const calls = [];
const repository = new PostgresRunAttemptLogRetentionClaimRepository({
async connect() {
throw new Error('not expected');
},
async query(text, values) {
calls.push({ text, values });
return { rows: [{ attemptId: 'attempt-1' }], rowCount: 1 };
},
});
assert.equal(
await repository.settle(claim(), {
status: 'retry',
delayMs: 2500,
failureCode: 'artifact_unavailable',
}),
'settled',
);
assert.deepEqual(calls[0].values, [
'attempt-1',
'replica-a',
TOKEN,
1,
100000,
'retry',
2500,
'artifact_unavailable',
]);
assert.match(calls[0].text, /claim_expires_at_ms > observation\.observed_at_ms/);
});
test('reads an exact durable tombstone for the profile-aware log route', async () => {
const record = createRunAttemptLogRetirementRecord({
...claim().candidate,
eligibleAtMs: 61000,
retiredAtMs: 80000,
disposition: 'already_absent',
byteLength: 0,
truncation: { truncated: 'unknown' },
});
const rows = [
{
...record,
finishedAtMs: String(record.finishedAtMs),
eligibleAtMs: String(record.eligibleAtMs),
retiredAtMs: String(record.retiredAtMs),
byteLength: String(record.byteLength),
truncated: 'unknown',
maximumBytes: null,
truncationObservedAtMs: null,
},
];
const repository = new PostgresRunAttemptLogRetentionClaimRepository({
async connect() {
throw new Error('not expected');
},
async query(text, values) {
assert.match(text, /artifact_tombstones/);
assert.deepEqual(values, [ARTIFACT_ID]);
return { rows: rows.splice(0), rowCount: 1 };
},
});
const identity = {
projectId: 'project-1',
runId: 'run-1',
attemptId: 'attempt-1',
logArtifactId: ARTIFACT_ID,
};
assert.deepEqual(await repository.inspect(identity), {
status: 'retired',
record,
});
assert.deepEqual(await repository.inspect(identity), { status: 'active' });
});
test('records the exact tombstone and removes its claim atomically', async () => {
const record = createRunAttemptLogRetirementRecord({
...claim().candidate,
eligibleAtMs: 61000,
retiredAtMs: 80000,
disposition: 'deleted',
byteLength: 11,
truncation: {
truncated: false,
maximumBytes: 1048576,
observedAtMs: 80000,
},
});
const calls = [];
let released = false;
const repository = new PostgresRunAttemptLogRetentionClaimRepository({
async connect() {
return {
async query(text, values = []) {
calls.push({ text, values });
if (text.includes('FOR UPDATE OF control')) {
return {
rows: [claimRow({ observedAtMs: '80000' })],
rowCount: 1,
};
}
if (text.startsWith('INSERT INTO "ql3"."run_attempt_log_artifact_tombstones"')) {
return {
rows: [{ recordDigest: record.recordDigest }],
rowCount: 1,
};
}
if (text.startsWith('DELETE FROM "ql3"."run_attempt_log_retention_controls"')) {
return { rows: [], rowCount: 1 };
}
return { rows: [], rowCount: 0 };
},
release() {
released = true;
},
};
},
async query() {
throw new Error('not expected');
},
});
assert.equal(
await repository.settle(claim(), { status: 'retired', record }),
'settled',
);
assert.deepEqual(
calls.map(({ text }) => text.split('\n', 1)[0]),
[
'BEGIN ISOLATION LEVEL READ COMMITTED',
"SET LOCAL statement_timeout = '5000ms'",
"SET LOCAL lock_timeout = '1000ms'",
'WITH observation AS (',
'INSERT INTO "ql3"."run_attempt_log_artifact_tombstones" (',
'DELETE FROM "ql3"."run_attempt_log_retention_controls"',
'COMMIT',
],
);
assert.equal(calls[4].values.at(-1), record.recordDigest);
assert.equal(released, true);
});
test('returns fenced without writing when the durable lease changed', async () => {
const calls = [];
const repository = new PostgresRunAttemptLogRetentionClaimRepository({
async connect() {
return {
async query(text) {
calls.push(text);
if (text.includes('FOR UPDATE OF control')) {
return { rows: [], rowCount: 0 };
}
return { rows: [], rowCount: 0 };
},
release() {},
};
},
async query() {
throw new Error('not expected');
},
});
const record = createRunAttemptLogRetirementRecord({
...claim().candidate,
eligibleAtMs: 61000,
retiredAtMs: 80000,
disposition: 'already_absent',
byteLength: 0,
truncation: { truncated: 'unknown' },
});
assert.equal(
await repository.settle(claim(), { status: 'retired', record }),
'fenced',
);
assert.equal(
calls.some((text) => text.includes('artifact_tombstones" (')),
false,
);
});
test('rolls back and wraps claim failures without leaking the client', async () => {
const calls = [];
let released = false;
const repository = new PostgresRunAttemptLogRetentionClaimRepository(
{
async connect() {
return {
async query(text) {
calls.push(text);
if (text.includes('FOR UPDATE OF attempt')) throw new Error('offline');
return { rows: [], rowCount: 0 };
},
release() {
released = true;
},
};
},
async query() {
throw new Error('not expected');
},
},
() => TOKEN,
);
await assert.rejects(
repository.claim({
ownerId: 'replica-a',
retentionMs: 60000,
limit: 1,
leaseMs: 5000,
}),
RunAttemptLogRetentionUnavailableError,
);
assert.equal(calls.at(-1), 'ROLLBACK');
assert.equal(released, true);
});
+8
View File
@@ -238,6 +238,9 @@
],
"run-attempt-log-retention": [
"dist/run/log-retention/runAttemptLogRetention.d.ts"
],
"cluster-run-attempt-log-retention": [
"dist/run/log-retention/clusterRunAttemptLogRetention.d.ts"
]
}
},
@@ -297,6 +300,11 @@
"require": "./dist/run/log-retention/runAttemptLogRetention.js",
"default": "./dist/run/log-retention/runAttemptLogRetention.js"
},
"./cluster-run-attempt-log-retention": {
"types": "./dist/run/log-retention/clusterRunAttemptLogRetention.d.ts",
"require": "./dist/run/log-retention/clusterRunAttemptLogRetention.js",
"default": "./dist/run/log-retention/clusterRunAttemptLogRetention.js"
},
"./task-definition": {
"types": "./dist/task-definition/taskDefinition.d.ts",
"require": "./dist/task-definition/taskDefinition.js",
@@ -0,0 +1,62 @@
import type {
RunAttemptLogRetentionCandidate,
RunAttemptLogRetentionStateReader,
RunAttemptLogRetirementRecord,
} from './runAttemptLogRetention';
export const MAX_CLUSTER_RUN_ATTEMPT_LOG_RETENTION_CLAIMS = 16;
export const MIN_CLUSTER_RUN_ATTEMPT_LOG_RETENTION_LEASE_MS = 5_000;
export const MAX_CLUSTER_RUN_ATTEMPT_LOG_RETENTION_LEASE_MS = 5 * 60_000;
export const MAX_CLUSTER_RUN_ATTEMPT_LOG_RETENTION_RETRY_DELAY_MS =
24 * 60 * 60_000;
export type ClusterRunAttemptLogRetentionFailureCode =
| 'artifact_unavailable'
| 'artifact_integrity_mismatch'
| 'retirement_record_unavailable';
export interface ClusterRunAttemptLogRetentionClaim {
readonly candidate: Readonly<RunAttemptLogRetentionCandidate>;
readonly eligibleAtMs: number;
readonly observedAtMs: number;
readonly ownerId: string;
readonly token: string;
readonly version: number;
readonly expiresAtMs: number;
readonly failureCount: number;
}
export interface ClusterRunAttemptLogRetentionClaimPage {
readonly claims: readonly Readonly<ClusterRunAttemptLogRetentionClaim>[];
readonly hasMore: boolean;
}
export type ClusterRunAttemptLogRetentionSettlement =
| Readonly<{
readonly status: 'retired';
readonly record: Readonly<RunAttemptLogRetirementRecord>;
}>
| Readonly<{
readonly status: 'retry';
readonly delayMs: number;
readonly failureCode: ClusterRunAttemptLogRetentionFailureCode;
}>
| Readonly<{
readonly status: 'manual';
readonly failureCode: ClusterRunAttemptLogRetentionFailureCode;
}>;
export interface ClusterRunAttemptLogRetentionClaimRepository
extends RunAttemptLogRetentionStateReader {
claim(options: Readonly<{
readonly ownerId: string;
readonly retentionMs: number;
readonly limit: number;
readonly leaseMs: number;
}>): Promise<Readonly<ClusterRunAttemptLogRetentionClaimPage>>;
settle(
claim: Readonly<ClusterRunAttemptLogRetentionClaim>,
settlement: Readonly<ClusterRunAttemptLogRetentionSettlement>,
): Promise<'settled' | 'fenced'>;
}
+5 -5
View File
@@ -74,7 +74,7 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
[
{
directory: 'packages/ql3-cluster-postgres/src/migrations',
directSourceFiles: 57,
directSourceFiles: 58,
reviewKind: 'ordered_ledger',
},
{
@@ -299,10 +299,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
rootSourceFileRoles: runtimeCore.rootSourceFileRoles,
},
{
sourceFiles: 150,
sourceFiles: 151,
rootSourceFiles: 1,
rootSourceLines: 160,
nestedSourceFiles: 149,
nestedSourceFiles: 150,
rootSourceFileRoles: { 'index.ts': 'public_export' },
},
);
@@ -421,10 +421,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
rootSourceFileRoles: clusterPostgres.rootSourceFileRoles,
},
{
sourceFiles: 147,
sourceFiles: 149,
rootSourceFiles: 1,
rootSourceLines: 125,
nestedSourceFiles: 146,
nestedSourceFiles: 148,
rootSourceFileRoles: { 'index.ts': 'public_export' },
},
);