diff --git a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md index bd9bc01a..32d45bdf 100644 --- a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md +++ b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md @@ -11,6 +11,22 @@ 最新增量证据（2026-08-12）： +- D-289/ADR-0377（已接受） + Local/Cluster 已增加同构 `GET /api/v3/projects/{projectId}/runs/{runId}/attempts/{attemptId}/log`、 + `run.log.read`/`artifact.read`。请求只接受 Project/Run/Attempt identity 与 offset/length，不接受 Artifact ID、路径、URI、bucket + 或 object key；Policy、durable audit 与 credential confirmation 先于 metadata 和存储访问，拒绝、不存在与跨 Project 对外统一 + 遮蔽为 404。Runtime Core 复用现有 Run/Attempt authority；Local 使用私有文件 Range 与 canonical truncation fact，Cluster 在既有 + immutable S3 HEAD/checksum 后执行 ETag-fenced Range GET；缺少 COPY 后 `ChecksumSHA256` 的 S3-compatible provider 失败关闭。共享硬 + 上限 256 KiB，但 Edge 默认 16/最大 32 KiB、Standalone 最大 32 KiB、Cluster 默认 64/最大 256 KiB，以满足 Local 64 KiB JSON + response 和低配设备固定内存门。Runtime Core 494/494、Local Execution 35/35、Local API 45/45、Local Application 45 pass/4 + skip、Cluster Control 215 pass/2 skip；完整 18-package 门退出 0，backend 1,163 pass/2 skip/0 fail。workspace 保持 18 package/ + 1,045 source、1,027 nested/18 reviewed root entry，无 single-source/shallow package，package/dependency/Edge import boundary 全绿。 + 14 个 artifact 全部 compatible；Edge Application API 为 3,596,457 bytes/423 files/84 modules，仍不包含 Cluster、PostgreSQL 或 + AWS SDK。当前 arm64 Local image 为 10 package/385 files/3,331,225 bytes；Edge 128 MiB/64 PIDs 与 Standalone 256 MiB/256 PIDs + 均 graceful stop、20 events、SQLite integrity `ok`。固定摘要 MinIO KMS/SSE 条件 promotion、重放、HEAD/Range 集成通过； + PostgreSQL 18.4 arm64 HA 通过 112 gates、timeline `1→2`，报告 SHA-256 为 + `61bea38e3a4f66884b9642c2fc1944dd7084f70ceb358d6d3fea8c11d5d33b65`。本批未新增 package、第三方依赖、migration、table/index、 + 连接、listener、timer、watcher 或 cache；headless Local 与 MCP 不加载该能力，retention/tombstone 仍由 ADR-0027 后续闭环。 - D-288/ADR-0376（已接受） Local/Cluster 固定同构 `POST /api/v3/projects/{projectId}/tasks/{taskId}/runs`、`task.start`/`run.start`，共享 `qinglong/task-start@v1` 只接受 canonical mutation UUID、expected Task revision 与 contentDigest。Project/RoleBinding @@ -5574,7 +5590,7 @@ Dispatcher 只能把任务分配给满足 required 条件的 Worker。当没有 - ADR-0120 已实现具体 S3-compatible shared Artifact store：body 先一次流式写入带 SHA-256/SSE 的共享临时 key，复验后用 destination `If-None-Match: *` server-side Copy 发布 opaque 永久 key；最终 HEAD 同时校验 S3 checksum、identity metadata、length 与 truncation。Put/Copy 丢响应和并发 winner 只由 inspect 裁决，临时 owner nonce 防止碰撞误删；未配置 SSE/KMS 不提供明文 fallback。该 adapter 已通过启用 static KMS 的真实 MinIO 验证，但不会因安装依赖而自动打开 Worker ingress 或 execution lifecycle。 - ADR-0121 已实现默认关闭的 `@qinglong/worker-runtime/production`：它只在 `worker` Profile 下装配单 journal owner、单 mTLS Agent、完整 Offer-to-completion execution graph、Edge/Node 有界预算与一个 `unref` cadence，并在返回 active 前完成 startup reconciliation。shutdown 先 Abort Pull 但保留 owner，只有 Session 已 durable draining 且所有记录 settled 才释放资源；timeout、recovery 或 Session drain 失败均保留 authority 供重试。完整 Session heartbeat、证书/`ql3w` credential 与部署配置仍由外层产品生命周期补齐。 - `next` 已按 ADR-0021 增加默认不可达的 Worker offer inbox：私有原子文件 journal 默认/硬上限为 64/1024 条、单条 192 KiB、分页最多 64 条；相同 offer 在进程内并发合并、跨重启按稳定 ID/digest 去重，过期 Lease、替换 Session、draining 新 claim 和冲突 payload 在 spawn 前拒绝。状态按 `accepted → starting_acknowledged → launching → started → running_acknowledged` 持久化；`launching` 是禁止自动二次 spawn 的 crash barrier，`started` 只重放 running ACK。文件 root 必须显式取得单 owner lease，默认 stale 30 秒且只有一个 `unref` 刷新 timer；双 owner、未持锁操作和 lock compromise 全部 fail closed。Inbox lifecycle 固定 acquire→audit→hold，审计失败或预算耗尽先释放再拒绝启动。启动审计默认扫描 4×16 条，硬上限 16×64，输出不含 token/命令，并把 `launching/recovery_required/started/running_acknowledged` 阻断为待恢复 ownership。Receiver 已在 spawn barrier 前校验 ephemeral completion callback，只把 sequence 与 token SHA-256 成对持久化，原 token 不进入 journal；旧记录继续可读但不能认证 receipt。其后的 recovery evidence pass 固定 receipt-first：SHA-256 authenticator 同时校验 sequence 并 constant-time 比较 digest，随后校验 Run/Attempt/start time，receipt 缺失才探测 durable process identity；进程 exited 后立即重读并可做最多 5 秒的有界 grace，invalid/mismatch/unsupported/I/O 均保守分类。结果不含 capability，不调用 start/stop/ACK/remove，并明确区分 current、替换 Session、offline、Worker/Run Lease 过期及控制面 terminal。该 adapter 不增加数据库连接、watcher 或每任务 sidecar。 -- 当前已具备认证 ExecutionSpec delivery、跨重启 delivery attempt/退避、真实 activation/Artifact/completion/lease-control HTTPS client、批量 Secret delivery、Secret-before-Artifact materializer、受审 POSIX Executor、流式本地 Artifact source、S3-compatible shared Artifact store、PostgreSQL completion/lease-control/cancellation/convergence repository、固定 Cluster Control 生产路由组合，以及在同一 package journal 上推进 `starting_acknowledged → launching → started → running_acknowledged`、从可信回执收敛 completion、按 durable deadline/失租停止精确进程的 Processor/协调器和具体 Worker production execution composition，不再要求把 accepted record 双写到旧 Receiver。具体 KMS/Vault Secret provider、Artifact range read/retention、对象存储 credential/lifecycle 产品配置，以及完整 Worker Session heartbeat、证书/`ql3w` credential 与 deploy bootstrap 仍未完成；execution subpath 因此继续默认关闭。HTTP 成功或本地 delivery receipt 仍不得冒充执行已经开始。 +- 当前已具备认证 ExecutionSpec delivery、跨重启 delivery attempt/退避、真实 activation/Artifact/completion/lease-control HTTPS client、批量 Secret delivery、Secret-before-Artifact materializer、受审 POSIX Executor、流式本地 Artifact source、S3-compatible shared Artifact store、Project-scoped Artifact range read、PostgreSQL completion/lease-control/cancellation/convergence repository、固定 Cluster Control 生产路由组合，以及在同一 package journal 上推进 `starting_acknowledged → launching → started → running_acknowledged`、从可信回执收敛 completion、按 durable deadline/失租停止精确进程的 Processor/协调器和具体 Worker production execution composition，不再要求把 accepted record 双写到旧 Receiver。具体 KMS/Vault Secret provider、Artifact retention/tombstone、对象存储 credential/lifecycle 产品配置，以及完整 Worker Session heartbeat、证书/`ql3w` credential 与 deploy bootstrap 仍未完成；execution subpath 因此继续默认关闭。HTTP 成功或本地 delivery receipt 仍不得冒充执行已经开始。 SQLite `Workers` adapter 只用于单控制面协议孵化、竞争测试和 standalone 未来连接 Worker 的兼容路径；它不是多副本 cluster-control 的存储方案。完整决策见 ADR-0012。 @@ -6285,7 +6301,7 @@ content-addressed bundle 与 entry 以 no-replace 发布，edge/standalone 每 authenticated publisher trust provision、addition-only overlap rotation 与 generation 恢复，并统一按不可变 lock creation time 校验签名 key lifetime；旧 key retirement/紧急 revoke、在线 OCI fetch、HTTP/UI、备份/rekey、Approval/retry、 -Artifact retention/read、外部 +Artifact retention/tombstone、外部 deployment/cutover controller 仍未接入；因此可以宣称 base headless target executable 及本机 queued package recovery consumer/publisher 已可达，但不能宣称 2.x cutover、 在线插件市场或完整产品 surface 已完成。 @@ -6296,6 +6312,8 @@ ADR-0068/0087 增量：`@qinglong/local-execution/recovery` 只依赖 runtime-co ADR-0069/0070/0071/0072/0073/0074/0076/0077/0078/0079/0083/0084/0085/0086 增量：`@qinglong/local-process` 只依赖 runtime-core，拥有 receipt/identity 协议、digest-bound POSIX launcher、exact controller 与数据库索引 cleanup；local-execution 与 local-execution-control 单向依赖 local-process，local-run-recovery 再依赖统一控制包，local-dispatch 单向依赖 local-execution，local-secret 与 local-identity 生产均只依赖 runtime-core，短生命周期 local-secret-admin 只依赖 local-secret/runtime-core，local-owner-bootstrap 只依赖 local-identity/runtime-core，local-owner-console 再以专用 SQLite bootstrap 子入口组合 POSIX proof 与有界 staged secret outbox，且 authority 依赖均受审计隔离。launcher 在 spawn 前登记 exact Run/Attempt，身份捕获失败停止进程组，wrapper 清除 callback 环境并 no-overwrite 发布回执。独立 coordinator 已在 spawn 前原子提交 dispatching/starting 与 callback digest，spawn 后原子提交 handle/PID/running；ownership 写失败先复验身份并 TERM/KILL，stop 不确定时保留 starting。SQLite reviewed migration `0003/0004` 建立 journal，`0005/0006` 加入 dispatch plan并推进 capability v3，`0007/0008` 加入 encrypted Secret envelope并推进 v4，`0009/0010` 加入 ownerless Project/RoleBinding/security audit 和原子授权 mutation并推进 v5，`0011/0012` 加入 ownerless stable Identity/append-only API credential 并推进 v6，`0013/0014` 加入一次性 provisioning/digest-only challenge 并推进 v7；非法回执先持久化 quarantine intent，终态文件清理后才 resolve。application 只拥有一个 `unref` execution-control timer，同时驱动 completion notification、deadline/cancellation 单页扫描与低频 cleanup：edge 控制 5 秒×4、cleanup 5 分钟×8，standalone 控制 1 秒×32、cleanup 1 分钟×32；显式 dispatcher 每次只处理 edge 4/standalone 16 个候选并最多启动一个。Artifact allocator、Secret-first materializer、AES-256-GCM provider、数据库外显式 keyring preflight 与 4/64 MiB output hard quota已接入；稳定本机 credential authenticator、一次性 provisioning、原子首 Owner claim、POSIX 私有文件证明组合根以及 credential/challenge 的 prepare→commit→publish→recovery 协议已孵化，pepper/credential rotation/recovery 和两个 GC 核心已完成，`ql3-owner-gc` 与 transport-free `ql3-owner` 产品 CLI 已提供；但 fresh database/pepper setup、Secret/Project/Role/Approval 管理入口、备份/rekey、retry、Artifact retention/read、HTTP admission 与部署 controller 完成前仍保持默认生产不可达。 +ADR-0377 已取代上段的 Artifact read/HTTP 缺口：Local/Cluster 的 Project-scoped、Profile-aware range read 已进入可选产品面；仍未闭环的是 retention/tombstone、管理 UI/CLI 与部署 controller。 + ADR-0087 execution 合并增量：application 常驻闭包只安装 `@qinglong/local-execution` 的四个显式 subpath，不存在 package 根入口；内部依赖只允许 recovery→control 与 dispatch→execution。构建前 stale-dist 清理后最大 application 为 2,351,747 bytes、489 files、58 modules，仍低于 4 MiB/512 files/16 MiB 门禁，并较合并前减少 38,051 bytes 与 6 files。 ### 11.5 LocalProcess completion receipt 与日志存活 @@ -8432,7 +8450,7 @@ flowchart LR > ADR-0058 至 ADR-0092 的以下段落是历史快照，其 PostgreSQL 数字与“下一切片”已由 ADR-0104/0105 及后续 Accepted ADR 取代；资源、Package 和物理证据边界仍保留作为演进记录：PostgreSQL 当时 baseline 为 11 条 reviewed migration、capability v10、19 张表和 migration/runtime/admin/worker-ingress 四角色；本机 SQLite 为二十八条 reviewed migration、capability v14、二十六张 owned table，并新增正式 TaskDefinition head/immutable revision Repository。当前有二十七个受审 3.0 importer：Owner maintenance 已合并为两个权限隔离 subpath，execution/control/recovery/dispatch 已合并为 `@qinglong/local-execution` 四个单向 subpath，bootstrap/credential-recovery 已合并为 `@qinglong/local-owner-ceremony` 两个互不依赖 subpath；三个 package 都不提供聚合根入口，hard cap 已同步降为 27。一次拓扑 build/test 会先清理已登记 QL3 package 的 stale dist；27 包全量测试、dependency/source boundary、联网 vulnerability audit 与六种 Profile 制品门禁均通过。ADR-0090 让 production packlist 只发布 JS、`.d.ts` 和受审 assets/drizzle，不再携带开发 map；ADR-0092 后当前最大 application 为 1,691,009 bytes、267 files、61 loaded modules，最大抽样 RSS delta 11,780,096 bytes，低于 4 MiB/512 files/16 MiB。所有 package build 已变为 self-only，全量从约 198 次编译降至精确 27 次，clean 状态单包测试仍按依赖闭包工作。资源门禁已拆为原生 Linux x64/arm64 的 128 MiB router stress、256 MiB Edge release guard 与 512 MiB Cluster control guard，并从容器内部验证 cgroup v2、零 swap/OOM、非 root、只读挂载、seccomp 和 `NoNewPrivs`；本轮本地原生 arm64 三档已通过，x64 等待远端 CI，所有档位均明确不是物理设备或生产容量承诺。物理 Edge candidate recorder 已绑定设备 manifest、实际 Linux/存储环境、Edge/SQLite 基准与 no-replace SHA-256 报告，并会拒绝容器/VM；同设备同 boot 的 idle sampler、不主动填盘的专用文件系统 fault probe，以及通过正式 Repository 写入/扫描 100/1000/10000 个 `qinglong/command@v1` TaskDefinition 的规模记录协议均已具备契约门禁，但尚未取得固定实机报告或 signature。ADR-0091 已冻结 1–32 个 exact descriptor 的不可变 TaskSpec semantic registry、内建 command v1 和本机写前门禁；历史 revision 在 provider 缺失时仍可读。ADR-0092 已实现绑定 source revision/content digest 的 Profile-neutral command plan 与确定性本机 context/execution 映射，且保持 subpath-only。 -> 下表 PR-1 的旧“未完成”累计文字中，`completion/cancellation/timeout` 已由 ADR-0072 取代并闭环；当前本机剩余项是人工 recovery、retry 产品策略、Artifact retention/read、HTTP/CLI/UI 与部署 controller。远端 Worker completion 与 Workflow cancellation 的核心数据库链和 HA 重放已闭环；仍缺 production ingress/internal runtime port、expiry/retry lifecycle、部署启动装配与真实 Kubernetes 故障证据，不能与本机结论混用。 +> 下表 PR-1 的旧“未完成”累计文字中，`completion/cancellation/timeout` 已由 ADR-0072 取代并闭环，Artifact range read 已由 ADR-0377 闭环；当前本机剩余项是人工 recovery、retry 产品策略、Artifact retention/tombstone、HTTP/CLI/UI 与部署 controller。远端 Worker completion 与 Workflow cancellation 的核心数据库链和 HA 重放已闭环；仍缺 production ingress/internal runtime port、expiry/retry lifecycle、部署启动装配与真实 Kubernetes 故障证据，不能与本机结论混用。 > > 下表 PR-0 的累积长文本仍含“四角色、21 条 migration、capability v20、21 项 HA”历史短语；当前权威基线应读取为六角色、23 条 migration、capability v22/36 表和 23 项 physical HA gate。ADR-0145/0146 又增加默认关闭的 manager-only TLS 1.3 management process、可选双副本 operation 与 durable distributed quota；它仍缺全副本重启 keyset anti-rollback 和真实 IdP/live ingress，所以“受认证管理入口”保持“已孵化但生产失败关闭”。 @@ -8468,12 +8486,12 @@ flowchart LR | 切片 | 当前状态 | 已具备 | 尚未接入/门禁 | | --- | --- | --- | --- | | PR-0 Migration/Cluster 基线 | In progress | SQLite/PostgreSQL reviewed migration、checksum/rollback/schema ownership/readiness 与 Node 20/24 × x64/arm64 CI 基线；`runtime-core` 提供共享领域 contract，`cluster-postgres` 提供 exact-pinned Pool、四角色、21 条 migration、capability v20/35 表、可写主库、受审 function/role activation check、完整 cluster repository 及显式 admin-only Plugin Package installation和 Approved Action authority，`cluster-admin` 提供短生命周期发布 authority、POSIX/Kubernetes Secret adapter、标准 OCI stage verifier、exact-registry credential provider、有界恢复与 caller-driven Package Approved Action dispatcher，`local-admin` 以相同 handler 组合 edge/standalone 单 SQLite authority；`cluster-control` 提供 readiness-first HTTP/mTLS admission、startup recovery、数据库时钟 row-lease scheduler 和 Remote Worker 控制面。Package admission 已在两个方言的同一事务中强绑定 immutable proposal、exact executing snapshot、Project Policy fence 与数据库观察时钟，并以 receipt 收敛 COMMIT response loss；dispatcher 固定 start 前 retry、start 后 durable inspection，edge/standalone/cluster 默认单批 1/4/16 且无 timer。PostgreSQL 18 physical-promotion 门禁已在本机 arm64 完成 streaming、`remote_apply`、受控复制链分区、fence-before-promote、timeline 1→2、新 generation 同步策略、新 slot、旧主 `pg_rewind`/只读 sync rejoin、端点延迟切换、双 control 摘流/激活，以及 scheduler、credential v1/v2/v3/v4、completion、cancellation 和通用事务不确定窗口的精确收敛。ADR-0126/0127 进一步完成独立 process/migration binary、三 workspace package/五 production root 的 builder/production 双 lock、实际非 root/只读 Docker smoke、双副本 QL3 Kustomize、migration/runtime authority 分离与有界私有 CA 文件绑定；本地 PostgreSQL 18 TLS 1.3 门禁已验证旧 CA、overlap、新 CA、SIGHUP 端点证书切换和逆序回滚，并接入 x64/arm64 CI；ADR-0128 已完成 control 43 外部+3 内部与 admin 85 外部+3 内部 component 的精确 CycloneDX、两种真实 arm64 镜像 inventory 对账、control 真实本地 amd64/arm64 OCI + 逐平台 SPDX/SLSA digest graph、双镜像原生 x64/arm64 image/OCI CI matrix、两个独立 GHCR repository 的 digest-bound Cosign/GitHub attestation 发布契约与发布后 manifest/signature/provenance/CycloneDX 四重自验证门；ADR-0129 已完成 CloudNativePG 1.30.0/PostgreSQL 18.4 digest lock、三实例同步 HA、四 DatabaseRole/Database owner、离散凭据、`-rw` endpoint、`ca.crt` 投影、Kustomize/audit 门及四节点 Kind worker-loss live contract/CI 定义；ADR-0137 至 ADR-0141 已完成 Plugin Package PostgreSQL parity、Approved Action 双方言 durable authority、immutable Package proposal、execution/start-barrier、Package 双方言原子 admission receipt、单用途 Project/Policy lock authority、跨 adapter 合同、标准 OCI resolver、exact-registry credential provider、ConfigMap CAS publisher、独立 admin image、最小权限 Job/RBAC 与真实 Kind 双 Pod resourceVersion/RBAC live gate，并由 21 项本机物理 HA 门复验 PostgreSQL authority。ADR-0230 把 Workflow frontier 与 Task Attempt admission 接入既有 production Scheduler cadence；ADR-0231 又在同一 production process 中以独立 mTLS listener/Worker Pool 和内部 runtime capability port 装入 Remote Worker 全套无 Secret transport，保持 20 包、单 Scheduler timer 与显式权限 subpath | PostgreSQL 18 × x64/arm64 远端 CI 成功记录；取得一次 CloudNativePG live job 成功证据并继续完成 CA overlap rotation/rollback、Barman/WAL archive restore、Kubernetes HA control-plane/failover、基础设施 STONITH、真实 Pod/raw-wire 分区、多 Pod 与 Session replacement 竞争；stale lock/temp 修复与 pepper retire/reference/GC；PostgreSQL Package admission 过期租约真库门、资源 generation consumer 和受认证管理入口；admin 真实双架构 OCI 记录、两个 image 的实际远端 GHCR manifest digest、Cosign/GitHub attestation 独立 verify 记录、漏洞/容量证明；Cluster Secret provider、Remote Worker expiry/retry lifecycle 与启动装配；管理 CLI/API/UI、审批与 rate limit；schedule 指标/告警；正式停写与双数据库 backup manifest；固定物理 edge 基线和完整 2.x API 契约样本 | -| PR-1 Run Schema | Incubating | Run/RunAttempt/RunEvent schema、nullable cancel request 与 Attempt deadline 字段及恢复索引、CancellationDispatch 状态/version/lease/backoff schema、Repository port、临时 Sequelize adapter、统一事件大小/分页上限、跨 adapter RunRepository contract suite（原子事务、回滚、Run/Attempt/RetryPolicy CAS、唯一错误、分页与取消恢复）；ADR-0041 的 `pg-0003-run-retry-policy`、capability v2、driver-neutral PostgreSQL Run Repository 与真实 `pg.Pool` 上的共享 Repository/rollback/SQLSTATE contract；ADR-0063/0069/0071/0073/0074/0076 的独立 Node 24 local-sqlite typed schema、十二条 reviewed migration、capability v6、共享 operation authority、readiness/RunRepository/API credential repository/receipt journal/dispatch plan/encrypted Secret envelope/Project Policy/security audit/authorized mutation/stable Identity catalog、Drizzle↔真实 catalog table/column/index/CHECK/FK lockstep、base/adopted/application edge/standalone 产物门禁；ADR-0064 的 legacy baseline/plan digest、Online Backup recovery、side-by-side target migration、staged manifest、双库栅栏 activation、source 生命周期写栅栏、target stable identity 和重启语义；ADR-0065 的独立 cutover authority、外部副作用停机 evidence、append-only journal、start/restart/stop barrier 与 unknown→manual_required 收敛；ADR-0066 的 adopted storage→Run reconciliation→receipt maintenance→domain recovery→lifecycle→admission application gate、严格有界 recovery summary 与 admission-first reverse stop；ADR-0067 的 SQLite 事实驱动 Run 候选源、256 条硬上限、截断失败关闭和唯一 Repository authority；ADR-0068 的 receipt-first Reconciler、callback token/sequence fence、exact local-process identity、Attempt/Run/双 Event 原子终态推进和最终 verifier；ADR-0069 的 local-process 单向包边界、pre-spawn journal、受审 POSIX launcher、immutable receipt、exact identity 和 Profile-aware cleanup lifecycle；ADR-0070 的独立 local-execution、spawn 前后双 transaction CAS、callback digest、exact stop 补偿与 fail-closed starting 保留；ADR-0071 的独立 local-dispatch、不可变 revision/context、Secret-first materializer、Profile Artifact admission、4/64 MiB output hard quota 和窄 application facade；ADR-0073/0074 的 Project-bound SecretRef、AES-256-GCM、外置 keyring 生命周期、双 SQLite authority CAS、application preflight、强 Principal/Policy 和 envelope+audit 原子提交；ADR-0086 的本机 Owner provisioning/challenge/claim/delivery acknowledgement/credential recovery CLI | fresh database/pepper setup、credential rotation/GC 运维编排与 Secret/Project/Role/Approval 管理 CLI/API/UI、备份/rekey、2.x/target process controller、人工 recovery、target 写后 reconciliation 与完整 cutover/rollback 演练；retry 产品策略、Artifact retention/read stack、具体本机 lifecycle 和 target executable；Linux x64/arm64、PID namespace、断电与固定路由设备门禁；PostgreSQL 16/18 双连接并发与 failover integration；Task revision/context 跨方言 contract/并发压力与引用感知 retention、Keyv 数据迁移 | +| PR-1 Run Schema | Incubating | Run/RunAttempt/RunEvent schema、nullable cancel request 与 Attempt deadline 字段及恢复索引、CancellationDispatch 状态/version/lease/backoff schema、Repository port、临时 Sequelize adapter、统一事件大小/分页上限、跨 adapter RunRepository contract suite（原子事务、回滚、Run/Attempt/RetryPolicy CAS、唯一错误、分页与取消恢复）；ADR-0041 的 `pg-0003-run-retry-policy`、capability v2、driver-neutral PostgreSQL Run Repository 与真实 `pg.Pool` 上的共享 Repository/rollback/SQLSTATE contract；ADR-0063/0069/0071/0073/0074/0076 的独立 Node 24 local-sqlite typed schema、十二条 reviewed migration、capability v6、共享 operation authority、readiness/RunRepository/API credential repository/receipt journal/dispatch plan/encrypted Secret envelope/Project Policy/security audit/authorized mutation/stable Identity catalog、Drizzle↔真实 catalog table/column/index/CHECK/FK lockstep、base/adopted/application edge/standalone 产物门禁；ADR-0064 的 legacy baseline/plan digest、Online Backup recovery、side-by-side target migration、staged manifest、双库栅栏 activation、source 生命周期写栅栏、target stable identity 和重启语义；ADR-0065 的独立 cutover authority、外部副作用停机 evidence、append-only journal、start/restart/stop barrier 与 unknown→manual_required 收敛；ADR-0066 的 adopted storage→Run reconciliation→receipt maintenance→domain recovery→lifecycle→admission application gate、严格有界 recovery summary 与 admission-first reverse stop；ADR-0067 的 SQLite 事实驱动 Run 候选源、256 条硬上限、截断失败关闭和唯一 Repository authority；ADR-0068 的 receipt-first Reconciler、callback token/sequence fence、exact local-process identity、Attempt/Run/双 Event 原子终态推进和最终 verifier；ADR-0069 的 local-process 单向包边界、pre-spawn journal、受审 POSIX launcher、immutable receipt、exact identity 和 Profile-aware cleanup lifecycle；ADR-0070 的独立 local-execution、spawn 前后双 transaction CAS、callback digest、exact stop 补偿与 fail-closed starting 保留；ADR-0071 的独立 local-dispatch、不可变 revision/context、Secret-first materializer、Profile Artifact admission、4/64 MiB output hard quota 和窄 application facade；ADR-0073/0074 的 Project-bound SecretRef、AES-256-GCM、外置 keyring 生命周期、双 SQLite authority CAS、application preflight、强 Principal/Policy 和 envelope+audit 原子提交；ADR-0086 的本机 Owner provisioning/challenge/claim/delivery acknowledgement/credential recovery CLI；ADR-0377 的 Local/Cluster 同构、Profile-aware、Project-scoped Artifact range read | fresh database/pepper setup、credential rotation/GC 运维编排与 Secret/Project/Role/Approval 管理 CLI/API/UI、备份/rekey、2.x/target process controller、人工 recovery、target 写后 reconciliation 与完整 cutover/rollback 演练；retry 产品策略、Artifact retention/tombstone stack、具体本机 lifecycle 和 target executable；Linux x64/arm64、PID namespace、断电与固定路由设备门禁；PostgreSQL 16/18 双连接并发与 failover integration；Task revision/context 跨方言 contract/并发压力与引用感知 retention、Keyv 数据迁移 | | PR-2 Run 状态机 | Incubating | 纯转换表、终态/时间/错误/执行器元数据规则、Run version 与 event sequence CAS、事务性 RunCommandService、回滚测试 | 重复 Worker callback/fencing、并发数据库压力测试、Primary 执行链接入 | | PR-3 Executor 端口 | Incubating | ADR-0003、ExecutionSpec/Context/Handle/Result、Executor port、LocalProcessExecutor、进程组取消/超时升级、流式背压、Legacy Cron spec builder、真实进程 contract tests、可复现 edge 基准入口 | 固定 edge/多架构设备基线、Legacy builder 与 makeCommand 差异审计、Primary 生产流量接入 | | PR-4 Shadow Run | Incubating | origin 三态策略；默认关闭的 `QL3_SHADOW_ORIGINS`；manual 与 scheduled_node 现有 ChildProcess 旁路观察；每个 worker 懒加载；Run/Attempt/Event 影子生命周期；稳定 task revision 与有界日志引用；同 worker 有界注册表和跨 worker 持久化候选关联；stop all/stop instance、Shell callback、乱序/迟到/歧义处理；失败开放和契约测试 | `ScheduleService.runTask` 覆盖的 scheduled_system/subscription/system/script 等高风险入口；启动后 Reconciler、差异报表、可采集指标、资源压力、回滚演练和 Primary 门禁 | | PR-5 Primary LocalExecutor | Incubating（默认不激活，仅 manifest-gated manual） | runtime-owned Run 创建器；持久化先于 spawn；Run/Attempt 完整成功、失败、取消、超时与 lost 闭环；Executor handle 身份校验；spawn 后激活写失败的 stop+lost 补偿；completion rejection 安全收敛；独立 Primary 幂等查询与唯一索引竞态裁决；durable `run.cancel_requested`、stop-before-signal、首次请求幂等、晚到完成裁决与待取消有界恢复查询；最多 64 条一页的 cross-worker cancellation source；独立 CancellationDispatch Repository 原子 claim/result、lease expiry 接管、owner/token/version fencing、指数退避与结果 RunEvent；最多 64 页的单周期 cancel supervisor；显式 start/stop、无重叠、错误隔离、停止等待有上限且 timer unref 的 lifecycle runner；Linux durable handle 的 PID/boot/start ticks/process-group 复验与 TERM/KILL controller；完整有界分页且 fail-closed 的 startup Reconcile supervisor；RunningInstance nullable `run_id/attempt_id` 关联；Primary 专用组合 Repository 在同一 SQLite 事务提交前投影 Crontab/RunningInstance，失败整体回滚；有界且防穿越的 legacy log output ref；manual owner seam、真实本机装配、单 spawn/fail-closed；严格 manual-only rollout manifest loader、短期审批/gate、配置哈希审计；HTTP worker 已接轻量 lazy bootstrap，accepted 后按 receipt-first reconcile→completion receipt lifecycle→timeout intent lifecycle→cancel dispatch lifecycle→router 顺序激活，失败撤销，监听失败和 shutdown 有界停止；Primary timeout 在 spawn 前持久化绝对 deadline，有界 source/requester/supervisor 只提交 timeout 意图并复用 CancellationDispatch；代码级 edge/standalone Profile 为各 lifecycle 提供不同 cadence 与页上限，cluster-control/worker 拒绝误装本机 SQLite Primary；统一 CompletionService 原子提交 Attempt/Run/双 Event，spawn 前保存 callback token hash、终态推进 sequence，实时回调与 receipt consumer 共享入口并覆盖两个清理 crash window；manual Primary 已接入受限 POSIX launcher、`0600` direct-file stdout/stderr、父进程退出后续写、不可覆盖 receipt 生产、回执环境清除、TERM 转发等待及 live transaction 后清理；Startup Reconciler receipt-first 双检查并在确定 exited 后执行 profile 化的单次 50/100 ms publish grace；`0007` 独立 CompletionReceiptJournal 在 spawn 前登记、为升级前 active Attempt 补登记并驱动周期扫描，使终态残留继续可发现；确定无效的已知 Attempt receipt 先持久化隔离状态，再进入确定性私有分片 quarantine；终态 missing 与 quarantine 按 edge/standalone retention 有界清理；非 Journal 文件具备只读优先、固定分片/条目上限、overflow fail-closed、显式同盘隔离的 Node 24 运维 CLI；扫描具备页上限、resume cursor、timer unref、无重叠、有界 stop 和低敏计数；ENOSPC 与 launcher receipt 存储失败有代码门禁 | 部署配置写入/审批入口与用户可见状态；PostgreSQL CancellationDispatch adapter；cluster-control 生产启动拓扑；固定 edge/Linux 多架构与真实磁盘压力基线、完整 2.x API 契约和回滚演练 | -| PR-7 Worker Session、Run Lease 与启动协议基础 | Incubating（默认关闭，独立入口显式 opt-in） | ADR-0012/0013/0014/0021/0057–0061/0108–0121/0231–0239；有界 capability/Placement/Dispatcher；SQLite 协议孵化与 PostgreSQL v9 Session/Run Lease/credential/attestation authority；immutable revision Placement、数据库时钟 keyset candidate、认证 Worker Pull、digest-only offer recovery；versioned capability-free ExecutionSpec response、stable claim 跨重启退避、单 owner 原子 inbox 准入与 TLS 1.3 mTLS/`ql3w` HTTPS client；同一 package journal 上 revision-fenced starting/spawn/started/running/completion 状态、callback digest、tagged no-spawn 与 ambiguous recovery；PostgreSQL starting/running/start-failure/completion 数据库权威事务、精确重放与 cancellation/timeout 优先终态；batch Secret delivery 在 Attempt advisory lock 下复验 Session/Lease/revision 完整围栏并复用单 Agent，Secret-before-Artifact materializer 将同一 log ID 交给 Executor/journal/running ACK；offer-scoped `wlog-*` 私有文件 spool、Edge/Node 容量策略、append/quota/path 防护、barrier 后 output ownership、受审 POSIX Executor、truncation fact、固定内存流式 source、认证 Artifact stream、共享 immutable store port、S3-compatible SSE/checksum/条件 promotion adapter 与 upload-before-completion 协调；用户取消 run.stop mutation 以数据库时间写 intent/Event 并在事务内复验 Project/RoleBinding fence；非执行取消 convergence lifecycle、运行期 expiry 与安全 lost retry 已接入 cluster-control 单一全局 cadence；完整 generation/version/token/Attempt fencing；独立最小权限 Worker ingress、CA/CRL 与连接 generation 热重载；offer journal、spawn barrier、receipt-first recovery；独立 `@qinglong/worker-runtime` 的本地 P-256 CSR、key/chain/trust 验证、generation + active pointer 安装和持久退避；默认关闭的 production process 已装配具体 execution graph、完整 Session heartbeat/drain/offline、direct-file bootstrap、单 Agent/单 cadence、startup reconciliation、证书 maintenance、transport fail-close/recovery 与 Edge/Node 有界预算；真实 PostgreSQL 18 + Linux Node 合约已覆盖 Run completion、credential 和 CA 双轮换且保持同一 Session；真实 K3s 合约已覆盖 TLS/credential Secret 分权、双对象 CAS、Recreate 顺序、identity generation 与单节点 PVC recovery；所有能力默认不可达且受 edge/cluster import audit 约束 | 具体 cert-manager/Vault/SPIFFE/离线 CA adapter 与模板、ingress reload controller、生产 RBAC、证书到期告警和 `ql3w` credential recovery 产品面；具体 KMS/Vault Secret provider、对象存储 credential/temporary lifecycle、range read/retention；Worker 管理 API；真实 Kubernetes 多节点 CSI/node-loss/production 360 秒 drain 与固定 edge 文件系统 suspend/时钟/断电、x64/arm64 资源门禁 | +| PR-7 Worker Session、Run Lease 与启动协议基础 | Incubating（默认关闭，独立入口显式 opt-in） | ADR-0012/0013/0014/0021/0057–0061/0108–0121/0231–0239/0377；有界 capability/Placement/Dispatcher；SQLite 协议孵化与 PostgreSQL v9 Session/Run Lease/credential/attestation authority；immutable revision Placement、数据库时钟 keyset candidate、认证 Worker Pull、digest-only offer recovery；versioned capability-free ExecutionSpec response、stable claim 跨重启退避、单 owner 原子 inbox 准入与 TLS 1.3 mTLS/`ql3w` HTTPS client；同一 package journal 上 revision-fenced starting/spawn/started/running/completion 状态、callback digest、tagged no-spawn 与 ambiguous recovery；PostgreSQL starting/running/start-failure/completion 数据库权威事务、精确重放与 cancellation/timeout 优先终态；batch Secret delivery 在 Attempt advisory lock 下复验 Session/Lease/revision 完整围栏并复用单 Agent，Secret-before-Artifact materializer 将同一 log ID 交给 Executor/journal/running ACK；offer-scoped `wlog-*` 私有文件 spool、Edge/Node 容量策略、append/quota/path 防护、barrier 后 output ownership、受审 POSIX Executor、truncation fact、固定内存流式 source、认证 Artifact stream、共享 immutable store port、S3-compatible SSE/checksum/条件 promotion adapter、upload-before-completion 协调，以及 Local/Cluster 同构、Profile-aware、ETag-fenced range read；用户取消 run.stop mutation 以数据库时间写 intent/Event 并在事务内复验 Project/RoleBinding fence；非执行取消 convergence lifecycle、运行期 expiry 与安全 lost retry 已接入 cluster-control 单一全局 cadence；完整 generation/version/token/Attempt fencing；独立最小权限 Worker ingress、CA/CRL 与连接 generation 热重载；offer journal、spawn barrier、receipt-first recovery；独立 `@qinglong/worker-runtime` 的本地 P-256 CSR、key/chain/trust 验证、generation + active pointer 安装和持久退避；默认关闭的 production process 已装配具体 execution graph、完整 Session heartbeat/drain/offline、direct-file bootstrap、单 Agent/单 cadence、startup reconciliation、证书 maintenance、transport fail-close/recovery 与 Edge/Node 有界预算；真实 PostgreSQL 18 + Linux Node 合约已覆盖 Run completion、credential 和 CA 双轮换且保持同一 Session；真实 K3s 合约已覆盖 TLS/credential Secret 分权、双对象 CAS、Recreate 顺序、identity generation 与单节点 PVC recovery；所有能力默认不可达且受 edge/cluster import audit 约束 | 具体 cert-manager/Vault/SPIFFE/离线 CA adapter 与模板、ingress reload controller、生产 RBAC、证书到期告警和 `ql3w` credential recovery 产品面；具体 KMS/Vault Secret provider、对象存储 credential/temporary lifecycle 与 retention/tombstone；Worker 管理 API；真实 Kubernetes 多节点 CSI/node-loss/production 360 秒 drain 与固定 edge 文件系统 suspend/时钟/断电、x64/arm64 资源门禁 | | PR-8 Project/Policy/Approval Core | Incubating（默认拒绝、无生产业务执行入口） | ADR-0028；统一六类 ActorRef 与 exact-shape 校验；`0017` ownerless default Project 和 append-only versioned RoleBinding；owner/admin/operator/viewer 固定矩阵；Project 内 mutation 幂等、expected-version CAS、双 SQLite 连接竞争门禁；archived read-only、revocation、存储损坏 fail-closed；Agent 写/Secret/Tool `require_approval`；ADR-0047 把六类 subject、role/permission matrix 与 fence 抽到 runtime-core，`pg-0004-project-policy`/capability v3 建立 ownerless PostgreSQL baseline、严格 role/state CHECK、append-only runtime 权限、SERIALIZABLE Project lock、mutation replay、双连接单 winner 和 cluster admission authorizer；ADR-0049/`pg-0005` capability v4 建立 stable IdentitySubject、append-only digest-only API credential、真实 cluster bearer authenticator、write-only durable security audit 与最小权限 runtime role，且已验证 HTTP→credential→Policy→audit→handler 纵向链路；ADR-0051 建立 `/api/v3` 认证前 peer/global 双预算、transport-peer-only、无 timer 且有界内存的 overload shield；ADR-0027 Artifact authorizer adapter；ADR-0029 `AuthenticatedPrincipal` contract、`0018` digest-only versioned challenge、CSPRNG/TTL、同事务消费 challenge + 写首 owner、精确重放与双连接竞争/崩溃回滚门禁；ADR-0030 `0019` stable identity/binding、legacy HS384 + current-session membership、logout/platform/revoke/disable、single-factor 与损坏 fail-closed 门禁；ADR-0031 `0020` digest-bound ApprovalRequest、User-only decision、Project/Role version fence、精确 expiry/重放/并发裁决及同事务 immutable dispatch；ADR-0032 `0021` execution backfill、三表原子 consume、稳定 due keyset、claim/renew/start/result fencing、pre-start takeover/post-start recovery-required、attempt budget、handler inspect/digest barrier 和 bounded dispatcher；ADR-0033/`0022` control/resolution backfill、start/renew/completion 原子联动、稳定 recovery keyset、双 resolver claim/takeover、finding/result 精确重放、自动/人工终结、迟到 completion 单 winner 和 evidence-only bounded reconciler；ADR-0034/`0023` 首个 `run.create` canonical plan、Run/Attempt/Event/receipt 同事务、幂等 collision fail-closed、renew/终态 fence、真实 SQLite handler 与 automatic evidence provider；ADR-0035/`0024` 独立 `approval.recover` 矩阵、稳定 User + 五分钟强认证、Project/RoleBinding fence、human resolution + authorization fact 原子提交、撤权竞态与回滚门禁；ADR-0036 recovery-first 单 timer lifecycle、edge/standalone 独立 cadence/页预算、跨周期 cursor、非重叠与有界 stop；ADR-0074 以新的 Node 24 SQLite v5 ownerless Project/RoleBinding/audit authority 和独立 local-secret-admin 提供强 Principal、`secret.manage`、撤权 fence、envelope+allowed audit 原子提交及不回显语义；ADR-0086 以可信 POSIX console 和 staged delivery 完成本机首 Owner 产品 ceremony | fresh database/pepper setup 与安全迁移向导；`shareStore`/Express 到 authentication core 的 production migration；credential rotation/revocation API、mTLS/Worker enrollment、恢复码；Project/Role/Approval/Secret 管理 CLI/API/UI、audit retention/query/export/alert、preview Artifact/digest/immutable plan builder、真实 MFA/hardware adapter、人工 recovery API/UI/独立 rate limit 与审计事件、handler/provider registry、lifecycle startup/shutdown/指标/admission gate；PostgreSQL action/receipt/provider/recovery-authorization 与 OPA adapter、缓存 version 失效；Tool/Package/Secret/Shell 各自的 handler/evidence contract；Secret/Run/Tool/Workflow waiting_approval 全入口装配；完整回滚演练 | PR-8 的本机最新增量由 ADR-0075/0076/0077/0078/0079/0080/0081/0082/0083/0084/0085/0086/0087 约束：Node 24 SQLite capability v13 已有 ownerless stable Identity、append-only digest-only API credential、credential-version pepper provenance binding、versioned pepper catalog/append-only activation generation、credential recovery/material GC ledger、delivery acknowledgement tombstone GC、共享 operation authority、Runtime exact-key authenticator、一次性 provisioning、digest-only challenge 与无 secret delivery acknowledgement ledger；共享 `ApiCredentialRecord` 与 PostgreSQL capability v10 也具备同名 key provenance。独立 `@qinglong/local-owner-keyring` 提供最多 8 key 的 no-replace provision、低敏 inspect、独立 backup、absence-only restore、exact-ID load 和仅供 GC 的 destructive 子入口；认证只接受 record 指定且 catalog 状态为 active/retired、摘要一致的 material，staged/recovery-required/缺失均 fail closed。独立 `@qinglong/local-owner-credential-recovery` 以不同 ID 签发新 credential，复用 staged outbox；只有 exact delivery digest 已持久化确认，才允许在 SQLite 事务中 append revoked 旧 version、pepper binding 与 audit。短生命周期 `@qinglong/local-owner-maintenance/pepper-gc` 负责双材料 crash-safe 销毁；`/acknowledgement-gc` 以内建 clock、文件 bridge-clear proof、source 终态与 30 天 replay/audit 保留裁决，在单事务写 audit/tombstone 并删除完整行，bootstrap 随后从 immutable source 验证 semantic digest 后重建 exact replay，credential/challenge 均不重新请求 entropy。ADR-0085/0086 抽出共享 `@qinglong/local-command-file` 叶子协议；`ql3-owner-gc` 开放两个 GC ceremony，`ql3-owner` 开放 provisioning/challenge/transport-free claim/delivery acknowledgement/credential recovery，均只输出低敏 JSON。专用 `@qinglong/local-sqlite/bootstrap|pepper-gc|acknowledgement-gc`、`@qinglong/local-owner-bootstrap` 与短生命周期 console/maintenance authority 仍保持默认不可达，产品 CLI 只能经 console facade 到达 Owner ceremony。fresh database/pepper setup、Secret/Project/Role/Approval 管理产品入口、Linux/容器/断电/真实 ENOSPC 实机证据和 legacy adapter 未完成；两个本机 `bin` 都不是可远程使用的 cluster 管理入口。 diff --git a/docs/adr/ADR-0377-profile-aware-run-attempt-log-range-read.md b/docs/adr/ADR-0377-profile-aware-run-attempt-log-range-read.md new file mode 100644 index 00000000..b62e78da --- /dev/null +++ b/docs/adr/ADR-0377-profile-aware-run-attempt-log-range-read.md @@ -0,0 +1,103 @@ +# ADR-0377：Profile-aware Run Attempt 日志范围读取 + +- 状态：Accepted +- 日期：2026-08-12 +- 关联 RFC：QL-RFC-0001 D-289 +- 前置决策：ADR-0027、ADR-0367、ADR-0376 + +## 上下文 + +Task Start 已能在 Local 与 Cluster 创建可执行 Run，产品入口也能读取 Run、Event、Step 并请求取消，但调用方仍无法从同一 Project-scoped 产品面读取某个 Attempt 的执行日志。早期 `back/runtime` 已验证本机 metadata-first 授权、私有文件 Range 与截断三态，但它不在 QingLong 3.0 package 生产图中，也没有 Cluster 对象存储实现。 + +直接恢复 2.x 的 `path + file` 会把路径当作外部 capability；把整份日志编码进 JSON 会让内存与日志大小线性增长；把 Local 与 Cluster 强行放到同一存储 adapter，又会使控制面依赖 Worker 本地路径。低配路由设备和集群节点需要共享语义，但必须拥有不同的单次窗口与存储实现。 + +## 决策 + +### 1. 同一产品路由与只读权限 + +Local 与 Cluster 提供同构入口： + +`GET /api/v3/projects/{projectId}/runs/{runId}/attempts/{attemptId}/log?offset={n}&length={n}` + +- operation：`run.log.read` +- permission：`artifact.read` +- response schema：`qinglong/run-attempt-log-read-result@v1` +- `offset` 默认为 0；`length` 默认为当前 Profile 窗口 +- 请求只接受 Project、Run、Attempt identity 和有界范围，不接受 Artifact ID、路径、URI、bucket 或 object key + +Policy、durable security audit 与 credential confirmation 必须先于 Run/Attempt metadata 和 Artifact 存储访问。对非可信 HTTP 调用方，Policy deny、require-approval、跨 Project、不属于该 Run、legacy owner、错误 executor 和不存在均返回相同的 404，不形成存在性 oracle；真实结果只进入低敏审计。 + +### 2. 共享 application contract，不共享存储 adapter + +`runtime-core` 定义唯一的 Run Attempt 日志读取服务和 byte-range port。服务复用现有 `RunRepositoryReader.findRunById/findAttemptById`，依次验证： + +1. Run 属于请求 Project 且 `executionOwner=runtime`； +2. Attempt 属于请求 Run； +3. executor 与当前 Profile 一致； +4. Attempt 绑定 canonical `local-*` 或 `wlog-*` Artifact ID； +5. range 规范化且不超过调用 Profile 的上限； +6. 只有上述数据库事实成立后才调用存储 reader。 + +Local adapter 从 opaque `local-*` ID 派生私有 shard，拒绝 symlink、非普通文件、非当前进程 owner、多 hard link、异常短读和 identity 不一致的 truncation fact。Cluster adapter 先用现有 immutable S3 HEAD 校验 identity digest、length、content type、checksum metadata，再执行单次有界 Range GET；控制面从不拼接 Worker 路径。 + +S3-compatible provider 必须在最终对象 COPY 后，通过启用 checksum 的 HEAD 返回 canonical `ChecksumSHA256`；只返回调用方可写 metadata 而不能独立证明对象内容的实现不在兼容集合内，adapter 必须失败关闭，不能退化为信任 metadata 或 ETag。 + +### 3. Profile 窗口和资源上限 + +共享协议硬上限为 256 KiB，但部署档位收紧实际窗口： + +- Local Edge：默认 16 KiB，最大 32 KiB； +- Local Standalone：默认/最大 32 KiB； +- Cluster Control：默认 64 KiB，最大 256 KiB。 + +当前 HTTP transport 只返回 JSON，因此 content 使用 Base64，并同时返回 byte range、总长度和下一 offset。Local 的 32 KiB 上限保证编码和元数据仍低于既有 64 KiB response hard limit；Cluster 保持低于既有 1 MiB 默认 response limit。reader 只分配请求窗口，不启动 watcher、tail timer、cache、额外 listener、Pool 或 S3 client。 + +### 4. 状态与失败语义 + +- `200 available`：返回当前文件/对象快照内的有界 bytes、`start`、`endExclusive`、`totalBytes` 和可选 `nextOffset`； +- `202 pending`：合法远端 Attempt 已绑定日志，但对象尚未在 upload-before-completion 协议中发布，或 Attempt 尚未产生 Artifact identity； +- `404 artifact_not_found`：不存在、越权或 identity/executor 不匹配的统一外部遮蔽； +- `503 artifact_unavailable`：数据库、文件安全检查、对象存储或证据一致性无法证明。 + +`truncated` 保持 `true | false | unknown`。Local 只相信 launcher canonical fact；Cluster 只相信 immutable S3 metadata；禁止从 size、尾部文本或退出状态推断。offset 等于或超过当前快照末尾时返回空的 available range，并把 start 收敛到 totalBytes，便于轮询而不制造 416 竞态。 + +### 5. 本批明确不闭环 retention + +本批不新增 tombstone table、delete marker、retention lifecycle 或对象删除权限。ADR-0027 的 retained/missing 竞态和 ADR-0026 retention 仍是独立后续切片。没有 tombstone 时，Local 的已绑定文件缺失和终态 Cluster 对象缺失都按 503 处理，不能伪装成已执行 retention；因此 ADR-0027 在 retention 完成前保持 Proposed。 + +### 6. package 与部署边界 + +- 不新增 workspace package、第三方依赖、migration、table/index、数据库连接或常驻资源； +- contract 放入 `runtime-core` 既有 Run 能力目录；Local reader 放入 `local-execution` 既有 Artifact 能力目录；S3 reader 扩展 `cluster-control` 既有 immutable store； +- 默认 headless Local 不加载文件 reader；只有可选 Local API product surface 通过 application authority 获得已构造 capability； +- Cluster 复用 production Worker artifact binding 的同一 S3 client/store；未配置对象存储时 route 保持存在并失败关闭为 503； +- MCP 本批不读取日志，避免在 Agent 面扩大敏感数据能力。 + +## 被否决的替代方案 + +1. **恢复 `path + file`**：路径成为跨 Project capability，拒绝。 +2. **整文件返回或服务端 tail 长连接**：内存、连接和后台资源无界，不适合 Edge，拒绝。 +3. **固定所有 Profile 为 256 KiB**：Base64 后突破 Local 64 KiB transport 门，拒绝。 +4. **Cluster 读取 Worker 共享挂载**：绕过 immutable upload、checksum 与 fencing，拒绝。 +5. **为日志读取新建微包或 metadata 表**：既有 Run/Attempt authority 足够，本批不制造薄 package，拒绝。 +6. **将 missing 当 retention**：没有 durable tombstone 无法证明，拒绝。 + +## 验收 + +1. shared contract 覆盖 identity/range、Project/Run/Attempt/executor 绑定、pending/available/missing 与截断三态； +2. Local adapter 覆盖 symlink、owner/link count、append snapshot、短读、空/越尾 range 和 fact identity 冲突； +3. S3 adapter 覆盖 HEAD identity/checksum、精确 Range、abort、404、响应长度/Content-Range 漂移和终态缺失； +4. Local/Cluster HTTP 覆盖 route/query、`artifact.read`、durable audit、deny/not-found 遮蔽、202/200/400/404/503； +5. 默认 headless Edge closure 不增加；Local API 增量仍满足 64 KiB response、4 in-flight 与 128 MiB/64 PID 门； +6. 完整 package/backend、dependency/package boundary、Profile artifact、真实 SQLite+HTTP、S3-compatible integration、PostgreSQL HA 与 Local image 门全部通过后，状态才可改为 Accepted。 + +## Accepted 证据 + +- Runtime Core 494/494、Local Execution 35/35、Local API 45/45、Local Application 45 pass/4 platform skip、Cluster Control 215 pass/2 conditional skip；完整 18-package build/test 退出 0。 +- backend 1,163 pass/2 skip/0 fail；package boundary、dependency boundary 与 Edge import audit 全部 compatible。workspace 仍为 18 package、1,045 source、1,027 nested/18 reviewed root entry，无 single-source 或 shallow package。 +- 真实 SQLite+HTTP 已覆盖认证、Policy、durable audit、credential confirmation、Attempt metadata 与本地私有文件 Range 的完整顺序；Local 文件 adapter 的 symlink、权限、hard-link、短读、append snapshot、越尾空读与 truncation identity drift 全部通过。 +- S3 单元门覆盖 checksum/metadata/ETag/Content-Range 漂移、短体与超长体；固定摘要 `minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e` 的真实 KMS/SSE、条件 promotion、重放、HEAD 与 Range GET 通过。不能在 COPY 后返回 `ChecksumSHA256` 的旧 provider 被实测失败关闭。 +- PostgreSQL 18.4 arm64 HA 通过 112 gates，timeline `1→2`，报告 SHA-256 为 `61bea38e3a4f66884b9642c2fc1944dd7084f70ceb358d6d3fea8c11d5d33b65`。 +- 14 个 Edge/Standalone Profile artifact 全部 compatible。Edge Application API 为 3,596,457 bytes/423 files/84 loaded modules，距 6 MiB 门仍有 2,694,999 bytes；依赖闭包不包含 Cluster、PostgreSQL 或 AWS SDK。 +- 当前源码构建的 arm64 Local image 为 10 package/385 files/3,331,225 bytes。Edge 128 MiB/64 PIDs 与 Standalone 256 MiB/256 PIDs 均在 non-root、read-only root、network-none 下 active、20 events、graceful stop，SQLite integrity 为 `ok`。 +- 本批没有新增 package、第三方依赖、migration、table/index、连接、listener、timer、watcher 或 cache；Artifact retention/tombstone 仍由 ADR-0027 后续闭环。 diff --git a/packages/ql3-cluster-control/src/application-runtime/productionApplication.ts b/packages/ql3-cluster-control/src/application-runtime/productionApplication.ts index 3b3e0b7b..bcf2e2b8 100644 --- a/packages/ql3-cluster-control/src/application-runtime/productionApplication.ts +++ b/packages/ql3-cluster-control/src/application-runtime/productionApplication.ts @@ -24,6 +24,7 @@ import { createClusterControlRunReadRoute } from '../run/runReadRoute'; import { createClusterControlRunListRoute } from '../run/runListRoute'; import { createClusterControlRunEventListRoute } from '../run/runEventListRoute'; import { createClusterControlRunStepListRoute } from '../run/runStepListRoute'; +import { createClusterControlRunAttemptLogReadRoute } from '../run/runAttemptLogReadRoute'; import { createClusterControlTaskListRoute } from '../task/taskListRoute'; import { createClusterControlTaskReadRoute } from '../task/taskReadRoute'; import { createClusterControlTaskStartRoute } from '../task/taskStartRoute'; @@ -69,6 +70,7 @@ export const PRODUCTION_CLUSTER_CONTROL_ROUTE_OPERATIONS = Object.freeze([ 'run.list', 'run.events.list', 'run.steps.list', + 'run.log.read', 'run.cancel', 'workflow.read', 'workflow.run.read', @@ -195,6 +197,10 @@ export function createProductionClusterControlApplicationStack( input.runs, input.trustedToolStorage.stepRuns, ), + createClusterControlRunAttemptLogReadRoute( + input.runs, + input.workerRuntime?.runAttemptLogRead, + ), createClusterControlRunCancellationRoute( input.runCancellation, createEventId, diff --git a/packages/ql3-cluster-control/src/artifact/s3ArtifactStore.ts b/packages/ql3-cluster-control/src/artifact/s3ArtifactStore.ts index 8a0e9bd4..9025f9f4 100644 --- a/packages/ql3-cluster-control/src/artifact/s3ArtifactStore.ts +++ b/packages/ql3-cluster-control/src/artifact/s3ArtifactStore.ts @@ -6,12 +6,19 @@ import { ChecksumMode, CopyObjectCommand, DeleteObjectCommand, + GetObjectCommand, HeadObjectCommand, MetadataDirective, PutObjectCommand, S3Client, ServerSideEncryption, } from '@aws-sdk/client-s3'; +import { + normalizeRunAttemptLogReadRange, + type RunAttemptLogRangeReadResult, + type RunAttemptLogReadIdentity, + type RunAttemptLogReadRange, +} from '@qinglong/runtime-core/run-attempt-log-read'; import { MAX_REMOTE_WORKER_ARTIFACT_BYTES, REMOTE_WORKER_ARTIFACT_CONTENT_TYPE, @@ -31,7 +38,8 @@ const TEMPORARY_METADATA_SCHEMA = const SHA256_PATTERN = /^[a-f0-9]{64}$/; const BUCKET_PATTERN = /^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/; const PREFIX_PATTERN = /^[A-Za-z0-9][A-Za-z0-9/_=-]{0,254}$/; -const TEMPORARY_ID_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; +const TEMPORARY_ID_PATTERN = + /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; type S3SendClient = Pick; @@ -70,8 +78,7 @@ export function createS3ClusterRemoteWorkerArtifactClient( typeof options !== 'object' || Array.isArray(options) || !/^[a-z0-9][a-z0-9-]{0,62}$/.test(options.region) || - (options.endpoint !== undefined && - typeof options.endpoint !== 'string') || + (options.endpoint !== undefined && typeof options.endpoint !== 'string') || (options.forcePathStyle !== undefined && typeof options.forcePathStyle !== 'boolean') ) { @@ -79,9 +86,7 @@ export function createS3ClusterRemoteWorkerArtifactClient( } return new S3Client({ region: options.region, - ...(options.endpoint === undefined - ? {} - : { endpoint: options.endpoint }), + ...(options.endpoint === undefined ? {} : { endpoint: options.endpoint }), forcePathStyle: options.forcePathStyle ?? false, }); } @@ -116,6 +121,11 @@ type ArtifactAuthority = Readonly<{ logArtifactId: string; }>; +type StoredArtifactHead = Readonly<{ + receipt: Readonly; + eTag?: string; +}>; + type NormalizedStorageCommand = ArtifactAuthority & Readonly<{ byteLength: number; @@ -127,7 +137,9 @@ const DIAGNOSTIC_CONTEXT = Object.freeze({ }); function configurationError(message: string): TypeError { - return new TypeError(`S3 Remote Worker Artifact store is invalid: ${message}`); + return new TypeError( + `S3 Remote Worker Artifact store is invalid: ${message}`, + ); } function prepareOptions( @@ -175,14 +187,15 @@ function prepareOptions( throw configurationError('expected bucket owner is invalid'); } const encryption = options.encryption; - if (!encryption || typeof encryption !== 'object' || Array.isArray(encryption)) { + if ( + !encryption || + typeof encryption !== 'object' || + Array.isArray(encryption) + ) { throw configurationError('encryption is required'); } let preparedEncryption: PreparedOptions['encryption']; - if ( - encryption.mode === 's3' && - Object.keys(encryption).length === 1 - ) { + if (encryption.mode === 's3' && Object.keys(encryption).length === 1) { preparedEncryption = Object.freeze({ ServerSideEncryption: ServerSideEncryption.AES256, }); @@ -383,9 +396,8 @@ function finalMetadata( ), 'ql3-byte-length': String(command.byteLength), 'ql3-content-sha256': sha256, - 'ql3-truncated': command.truncated === undefined - ? 'omitted' - : String(command.truncated), + 'ql3-truncated': + command.truncated === undefined ? 'omitted' : String(command.truncated), }); } @@ -460,6 +472,99 @@ function parseStoredReceipt( ); } +function canonicalETag(value: unknown): string { + if ( + typeof value !== 'string' || + value.length < 3 || + value.length > 256 || + !/^"[^"\u0000-\u001f\u007f]+"$/.test(value) + ) { + throw new S3ClusterRemoteWorkerArtifactStoreError('integrity_mismatch'); + } + return value; +} + +function assertRangeMetadata( + authority: ArtifactAuthority, + receipt: Readonly, + output: Readonly<{ + ContentLength?: number | undefined; + ContentRange?: string | undefined; + ContentType?: string | undefined; + ETag?: string | undefined; + Metadata?: Readonly> | undefined; + }>, + eTag: string, + start: number, + endExclusive: number, +): void { + const metadata = output.Metadata; + const truncated = + receipt.truncated === undefined ? 'omitted' : String(receipt.truncated); + if ( + output.ContentLength !== endExclusive - start || + output.ContentRange !== + `bytes ${start}-${endExclusive - 1}/${receipt.byteLength}` || + output.ContentType !== REMOTE_WORKER_ARTIFACT_CONTENT_TYPE || + canonicalETag(output.ETag) !== eTag || + metadataValue(metadata, 'ql3-schema') !== METADATA_SCHEMA || + metadataValue(metadata, 'ql3-project-sha256') !== + fieldDigest('project', authority.projectId) || + metadataValue(metadata, 'ql3-run-sha256') !== + fieldDigest('run', authority.runId) || + metadataValue(metadata, 'ql3-attempt-sha256') !== + fieldDigest('attempt', authority.attemptId) || + metadataValue(metadata, 'ql3-log-artifact-sha256') !== + fieldDigest('log-artifact', authority.logArtifactId) || + metadataValue(metadata, 'ql3-byte-length') !== String(receipt.byteLength) || + metadataValue(metadata, 'ql3-content-sha256') !== receipt.sha256 || + metadataValue(metadata, 'ql3-truncated') !== truncated + ) { + throw new S3ClusterRemoteWorkerArtifactStoreError('integrity_mismatch'); + } +} + +async function readBoundedRangeBody( + body: unknown, + expectedBytes: number, + signal?: AbortSignal, +): Promise { + if ( + !body || + typeof body !== 'object' || + !(Symbol.asyncIterator in body) || + typeof (body as AsyncIterable)[Symbol.asyncIterator] !== + 'function' + ) { + throw new S3ClusterRemoteWorkerArtifactStoreError('integrity_mismatch'); + } + const content = Buffer.allocUnsafe(expectedBytes); + let received = 0; + try { + for await (const chunk of body as AsyncIterable) { + if (signal?.aborted) throw signal.reason; + if (!(chunk instanceof Uint8Array) || chunk.byteLength === 0) { + throw new S3ClusterRemoteWorkerArtifactStoreError('integrity_mismatch'); + } + if (received + chunk.byteLength > expectedBytes) { + throw new S3ClusterRemoteWorkerArtifactStoreError('integrity_mismatch'); + } + Buffer.from(chunk.buffer, chunk.byteOffset, chunk.byteLength).copy( + content, + received, + ); + received += chunk.byteLength; + } + if (received !== expectedBytes) { + throw new S3ClusterRemoteWorkerArtifactStoreError('integrity_mismatch'); + } + return content; + } catch (error) { + content.fill(0); + throw error; + } +} + function isNotFound(error: unknown): boolean { if (!error || typeof error !== 'object') return false; const value = error as { @@ -467,13 +572,17 @@ function isNotFound(error: unknown): boolean { Code?: unknown; $metadata?: { httpStatusCode?: unknown }; }; - return value.name === 'NotFound' || + return ( + value.name === 'NotFound' || value.name === 'NoSuchKey' || value.Code === 'NoSuchKey' || - value.$metadata?.httpStatusCode === 404; + value.$metadata?.httpStatusCode === 404 + ); } -function requestOptions(signal?: AbortSignal): { abortSignal: AbortSignal } | undefined { +function requestOptions( + signal?: AbortSignal, +): { abortSignal: AbortSignal } | undefined { return signal === undefined ? undefined : { abortSignal: signal }; } @@ -549,7 +658,8 @@ class ArtifactContentDigest { * objects are never overwritten or deleted by this adapter. */ export class S3ClusterRemoteWorkerArtifactStore - implements ClusterRemoteWorkerArtifactStore { + implements ClusterRemoteWorkerArtifactStore +{ private readonly options: PreparedOptions; constructor(options: S3ClusterRemoteWorkerArtifactStoreOptions) { @@ -561,6 +671,90 @@ export class S3ClusterRemoteWorkerArtifactStore signal?: AbortSignal, ): Promise | undefined> { const authority = normalizeLookup(lookup); + return (await this.head(authority, signal))?.receipt; + } + + async readLogRange( + rawIdentity: Readonly, + rawRange: Readonly, + signal?: AbortSignal, + ): Promise { + const authority = normalizeLookup(rawIdentity); + const range = normalizeRunAttemptLogReadRange(rawRange); + const stored = await this.head(authority, signal); + if (!stored) return Object.freeze({ status: 'missing' as const }); + const start = Math.min(range.offset, stored.receipt.byteLength); + const endExclusive = Math.min( + start + range.length, + stored.receipt.byteLength, + ); + const truncation = Object.freeze({ + truncated: stored.receipt.truncated ?? ('unknown' as const), + }); + if (start === endExclusive) { + return Object.freeze({ + status: 'available' as const, + content: Buffer.alloc(0), + start, + endExclusive, + totalBytes: stored.receipt.byteLength, + truncation, + }); + } + const eTag = canonicalETag(stored.eTag); + if (signal?.aborted) throw signal.reason; + try { + const output = await this.options.client.send( + new GetObjectCommand({ + Bucket: this.options.bucket, + Key: finalObjectKey(this.options.prefix, authority), + IfMatch: eTag, + Range: `bytes=${start}-${endExclusive - 1}`, + ...(this.options.expectedBucketOwner === undefined + ? {} + : { ExpectedBucketOwner: this.options.expectedBucketOwner }), + }), + requestOptions(signal), + ); + assertRangeMetadata( + authority, + stored.receipt, + output, + eTag, + start, + endExclusive, + ); + const bytes = await readBoundedRangeBody( + output.Body, + endExclusive - start, + signal, + ); + return Object.freeze({ + status: 'available' as const, + content: bytes, + start, + endExclusive, + totalBytes: stored.receipt.byteLength, + ...(endExclusive < stored.receipt.byteLength + ? { nextOffset: endExclusive } + : {}), + truncation, + }); + } catch (error) { + if (isNotFound(error)) { + return Object.freeze({ status: 'missing' as const }); + } + if (error instanceof S3ClusterRemoteWorkerArtifactStoreError) throw error; + throw new S3ClusterRemoteWorkerArtifactStoreError('unavailable', { + cause: error, + }); + } + } + + private async head( + authority: ArtifactAuthority, + signal?: AbortSignal, + ): Promise { if (signal?.aborted) throw signal.reason; try { const output = await this.options.client.send( @@ -574,7 +768,10 @@ export class S3ClusterRemoteWorkerArtifactStore }), requestOptions(signal), ); - return parseStoredReceipt(authority, output); + return Object.freeze({ + receipt: parseStoredReceipt(authority, output), + ...(output.ETag === undefined ? {} : { eTag: output.ETag }), + }); } catch (error) { if (isNotFound(error)) return undefined; if (error instanceof S3ClusterRemoteWorkerArtifactStoreError) throw error; @@ -604,9 +801,7 @@ export class S3ClusterRemoteWorkerArtifactStore existing.truncated !== command.truncated || existing.sha256 !== incomingSha256 ) { - throw new S3ClusterRemoteWorkerArtifactStoreError( - 'integrity_mismatch', - ); + throw new S3ClusterRemoteWorkerArtifactStoreError('integrity_mismatch'); } return existing; } @@ -693,13 +888,11 @@ export class S3ClusterRemoteWorkerArtifactStore stored.truncated !== command.truncated || stored.sha256 !== sha256 ) { - throw new S3ClusterRemoteWorkerArtifactStoreError( - 'integrity_mismatch', - ); + throw new S3ClusterRemoteWorkerArtifactStoreError('integrity_mismatch'); } result = Object.freeze({ ...stored, - status: copied ? 'stored' as const : 'already_stored' as const, + status: copied ? ('stored' as const) : ('already_stored' as const), }); } catch (error) { primaryError = error; @@ -770,9 +963,7 @@ export class S3ClusterRemoteWorkerArtifactStore output.Metadata?.['ql3-owner-sha256'] !== ownerSha256 || canonicalChecksum(output.ChecksumSHA256) !== sha256 ) { - throw new S3ClusterRemoteWorkerArtifactStoreError( - 'integrity_mismatch', - ); + throw new S3ClusterRemoteWorkerArtifactStoreError('integrity_mismatch'); } } } diff --git a/packages/ql3-cluster-control/src/remote-execution/remoteWorkerCompletionService.ts b/packages/ql3-cluster-control/src/remote-execution/remoteWorkerCompletionService.ts index e6cb8505..8cc738a7 100644 --- a/packages/ql3-cluster-control/src/remote-execution/remoteWorkerCompletionService.ts +++ b/packages/ql3-cluster-control/src/remote-execution/remoteWorkerCompletionService.ts @@ -16,6 +16,11 @@ import { type RemoteWorkerCompletionRepository, type RemoteWorkerCompletionResult, } from '@qinglong/runtime-core/remote-worker-completion'; +import type { + RunAttemptLogRangeReadResult, + RunAttemptLogReadIdentity, + RunAttemptLogReadRange, +} from '@qinglong/runtime-core/run-attempt-log-read'; export interface ClusterRemoteWorkerArtifactStorageCommand { readonly projectId: string; @@ -47,6 +52,12 @@ export interface ClusterRemoteWorkerArtifactStore { lookup: Readonly, signal?: AbortSignal, ): Promise | undefined>; + /** Optional during Alpha so upload-only test and alternate stores remain compatible. */ + readLogRange?( + identity: Readonly, + range: Readonly, + signal?: AbortSignal, + ): Promise; } export interface ClusterRemoteWorkerArtifactUploadInput { @@ -353,11 +364,13 @@ export class ClusterRemoteWorkerCompletionService { } try { const result = normalizeRemoteWorkerCompletionResult( - await this.repository.complete(Object.freeze({ - ...command, - attemptEventId: eventId(this.createEventId), - runEventId: eventId(this.createEventId), - })), + await this.repository.complete( + Object.freeze({ + ...command, + attemptEventId: eventId(this.createEventId), + runEventId: eventId(this.createEventId), + }), + ), ); if ( result.runId !== command.runId || diff --git a/packages/ql3-cluster-control/src/remote-execution/workerRuntimePort.ts b/packages/ql3-cluster-control/src/remote-execution/workerRuntimePort.ts index 4872ba7b..1875d865 100644 --- a/packages/ql3-cluster-control/src/remote-execution/workerRuntimePort.ts +++ b/packages/ql3-cluster-control/src/remote-execution/workerRuntimePort.ts @@ -1,6 +1,7 @@ // Remote execution owns the least-privilege assembly of Worker-facing runtime capabilities. import type { PostgresPool } from '@qinglong/runtime-core'; import type { RemoteWorkerSecretValueProvider } from '@qinglong/runtime-core/remote-secret-delivery'; +import type { RunAttemptLogRangeReader } from '@qinglong/runtime-core/run-attempt-log-read'; import { PostgresClusterDispatchSource, PostgresRemoteRunActivationRepository, @@ -11,23 +12,15 @@ import { PostgresTaskExecutionRevisionSource, PostgresWorkerSessionRepository, } from '@qinglong/cluster-postgres/runtime'; -import { - ClusterRemoteWorkerOfferClaimService, -} from './remoteWorkerDispatcher'; -import { - ClusterRemoteRunActivationService, -} from './remoteRunActivationService'; -import { - ClusterRemoteWorkerSecretDeliveryService, -} from './remoteWorkerSecretDeliveryService'; +import { ClusterRemoteWorkerOfferClaimService } from './remoteWorkerDispatcher'; +import { ClusterRemoteRunActivationService } from './remoteRunActivationService'; +import { ClusterRemoteWorkerSecretDeliveryService } from './remoteWorkerSecretDeliveryService'; import { ClusterRemoteWorkerArtifactService, ClusterRemoteWorkerCompletionService, type ClusterRemoteWorkerArtifactStore, } from './remoteWorkerCompletionService'; -import { - ClusterRemoteWorkerLeaseControlService, -} from './remoteWorkerLeaseControlService'; +import { ClusterRemoteWorkerLeaseControlService } from './remoteWorkerLeaseControlService'; import type { WorkerIngressPipelineOptions } from '../worker-ingress/workerIngressPipeline'; export interface ClusterWorkerRuntimeDependencies { @@ -49,6 +42,7 @@ export interface ClusterWorkerRuntimePort { readonly leaseControl: NonNullable< WorkerIngressPipelineOptions['leaseControl'] >; + readonly runAttemptLogRead?: RunAttemptLogRangeReader; } export function createClusterWorkerRuntimePort( @@ -67,9 +61,11 @@ export function createClusterWorkerRuntimePort( } const workerSessions = new PostgresWorkerSessionRepository(pool); - const completionRepository = - new PostgresRemoteWorkerCompletionRepository(pool); + const completionRepository = new PostgresRemoteWorkerCompletionRepository( + pool, + ); const secretProvider = dependencies.secretProvider; + const readLogRange = dependencies.artifactStore.readLogRange; return Object.freeze({ offers: new ClusterRemoteWorkerOfferClaimService( new PostgresClusterDispatchSource(pool), @@ -99,5 +95,12 @@ export function createClusterWorkerRuntimePort( leaseControl: new ClusterRemoteWorkerLeaseControlService( new PostgresRemoteWorkerLeaseControlRepository(pool), ), + ...(readLogRange === undefined + ? {} + : { + runAttemptLogRead: Object.freeze({ + read: readLogRange.bind(dependencies.artifactStore), + }), + }), }); } diff --git a/packages/ql3-cluster-control/src/run/runAttemptLogReadRoute.ts b/packages/ql3-cluster-control/src/run/runAttemptLogReadRoute.ts new file mode 100644 index 00000000..0e98d0ec --- /dev/null +++ b/packages/ql3-cluster-control/src/run/runAttemptLogReadRoute.ts @@ -0,0 +1,176 @@ +import { + InvalidRunAttemptLogReadError, + RunAttemptLogReadService, + RunAttemptLogReadUnavailableError, + type RunAttemptLogRangeReader, + type RunAttemptLogReadResult, +} from '@qinglong/runtime-core/run-attempt-log-read'; +import type { RunRepositoryReader } from '@qinglong/runtime-core/run-repository'; + +import type { ClusterControlAdmissionResponse } from '../transport/httpSurface'; +import type { + ClusterControlAuthorizedOperationRequest, + ClusterControlRouteDefinition, + ClusterControlRouteParameters, +} from '../transport/routeRegistry'; + +export const CLUSTER_CONTROL_RUN_ATTEMPT_LOG_READ_ROUTE = Object.freeze({ + method: 'GET' as const, + path: '/api/v3/projects/{projectId}/runs/{runId}/attempts/{attemptId}/log', + operationId: 'run.log.read', + permission: 'artifact.read', + projectParameter: 'projectId', + allowedQuery: Object.freeze(['length', 'offset']), +}); + +const DEFAULT_READ_BYTES = 64 * 1024; +const MAXIMUM_READ_BYTES = 256 * 1024; + +function response( + statusCode: number, + body: Readonly>, +): ClusterControlAdmissionResponse { + return Object.freeze({ statusCode, body: Object.freeze(body) }); +} + +function parseQuery( + query: Readonly>, +): Readonly<{ offset: number; length: number }> { + const offsetValues = query.offset; + const lengthValues = query.length; + if ( + (offsetValues !== undefined && offsetValues.length !== 1) || + (lengthValues !== undefined && lengthValues.length !== 1) + ) { + throw new TypeError(); + } + const rawOffset = offsetValues?.[0]; + const offset = rawOffset === undefined ? 0 : Number(rawOffset); + const rawLength = lengthValues?.[0]; + const length = + rawLength === undefined ? DEFAULT_READ_BYTES : Number(rawLength); + if ( + !Number.isSafeInteger(offset) || + offset < 0 || + (rawOffset !== undefined && String(offset) !== rawOffset) || + !Number.isSafeInteger(length) || + length < 1 || + length > MAXIMUM_READ_BYTES || + (rawLength !== undefined && String(length) !== rawLength) + ) { + throw new TypeError(); + } + return Object.freeze({ offset, length }); +} + +function validateQuery( + query: Readonly>, +): void { + parseQuery(query); +} + +function projection( + result: Extract, +): Readonly> { + return Object.freeze({ + schema: 'qinglong/run-attempt-log-read-result@v1', + status: 'available', + projectId: result.projectId, + runId: result.runId, + attemptId: result.attemptId, + range: Object.freeze({ + start: result.start, + endExclusive: result.endExclusive, + totalBytes: result.totalBytes, + ...(result.nextOffset === undefined + ? {} + : { nextOffset: result.nextOffset }), + }), + encoding: 'base64', + content: Buffer.from( + result.content.buffer, + result.content.byteOffset, + result.content.byteLength, + ).toString('base64'), + truncation: result.truncation, + }); +} + +export function createClusterControlRunAttemptLogReadRoute( + runs: Pick, + reader?: RunAttemptLogRangeReader, +): Readonly { + if ( + !runs || + typeof runs.findRunById !== 'function' || + typeof runs.findAttemptById !== 'function' || + (reader !== undefined && typeof reader.read !== 'function') + ) { + throw new TypeError( + 'Cluster-control Run Attempt log read dependencies are invalid', + ); + } + const service = + reader === undefined + ? undefined + : new RunAttemptLogReadService(runs, reader, { + executorType: 'remote_worker', + artifactIdPattern: /^wlog-[a-f0-9]{30}$/, + maximumReadBytes: MAXIMUM_READ_BYTES, + activeMissingIsPending: true, + }); + return Object.freeze({ + ...CLUSTER_CONTROL_RUN_ATTEMPT_LOG_READ_ROUTE, + validateQuery, + async handle( + authorized: ClusterControlAuthorizedOperationRequest, + parameters: ClusterControlRouteParameters, + ) { + if (authorized.request.body !== null) { + return response(400, { code: 'invalid_request_body' }); + } + if (authorized.projectId === null || service === undefined) { + return response(503, { code: 'artifact_unavailable' }); + } + let range; + try { + range = parseQuery(authorized.request.query); + } catch { + return response(400, { code: 'invalid_run_log_read_query' }); + } + try { + const result = await service.read({ + projectId: authorized.projectId, + runId: parameters.runId!, + attemptId: parameters.attemptId!, + range, + signal: authorized.request.signal, + }); + if (result.status === 'not_found') { + return response(404, { code: 'artifact_not_found' }); + } + if (result.status === 'pending') { + return response(202, { + schema: 'qinglong/run-attempt-log-read-result@v1', + status: 'pending', + projectId: result.projectId, + runId: result.runId, + attemptId: result.attemptId, + }); + } + if (result.status === 'missing') { + return response(503, { code: 'artifact_unavailable' }); + } + return response(200, projection(result)); + } catch (error) { + if (error instanceof InvalidRunAttemptLogReadError) { + return response(400, { code: 'invalid_run_log_read_request' }); + } + if (error instanceof RunAttemptLogReadUnavailableError) { + return response(503, { code: 'artifact_unavailable' }); + } + return response(503, { code: 'artifact_unavailable' }); + } + }, + }); +} diff --git a/packages/ql3-cluster-control/src/transport/admissionPipeline.ts b/packages/ql3-cluster-control/src/transport/admissionPipeline.ts index 3cb06305..e01e9b0d 100644 --- a/packages/ql3-cluster-control/src/transport/admissionPipeline.ts +++ b/packages/ql3-cluster-control/src/transport/admissionPipeline.ts @@ -308,6 +308,13 @@ export function createClusterControlAdmissionPipeline( now, ); if (decision.effect === 'deny') { + if (route.permission === 'artifact.read') { + throw securityError( + 404, + 'artifact_not_found', + 'Cluster-control Artifact is not available', + ); + } throw securityError( 403, 'forbidden', @@ -315,6 +322,13 @@ export function createClusterControlAdmissionPipeline( ); } if (decision.effect === 'require_approval') { + if (route.permission === 'artifact.read') { + throw securityError( + 404, + 'artifact_not_found', + 'Cluster-control Artifact is not available', + ); + } throw securityError( 403, 'approval_required', diff --git a/packages/ql3-cluster-control/test/bootstrap.test.cjs b/packages/ql3-cluster-control/test/bootstrap.test.cjs index 6f1d524b..20c95410 100644 --- a/packages/ql3-cluster-control/test/bootstrap.test.cjs +++ b/packages/ql3-cluster-control/test/bootstrap.test.cjs @@ -573,6 +573,9 @@ test('injects reviewed Worker operations without exposing the runtime Pool', asy async inspect() { throw new Error('not invoked during assembly'); }, + async readLogRange() { + throw new Error('not invoked during assembly'); + }, }; const result = await bootstrapClusterControlRuntime( bootstrapOptions(events, { @@ -596,6 +599,10 @@ test('injects reviewed Worker operations without exposing the runtime Pool', asy typeof input.workerRuntime.leaseControl.control, 'function', ); + assert.equal( + typeof input.workerRuntime.runAttemptLogRead.read, + 'function', + ); return activationStack(events); }, }), diff --git a/packages/ql3-cluster-control/test/productionApplication.test.cjs b/packages/ql3-cluster-control/test/productionApplication.test.cjs index 2ba8d9b6..46976ccf 100644 --- a/packages/ql3-cluster-control/test/productionApplication.test.cjs +++ b/packages/ql3-cluster-control/test/productionApplication.test.cjs @@ -143,6 +143,9 @@ function fixture(overrides = {}) { }, ]; }, + async findAttemptById() { + return null; + }, }, trustedToolStorage: { stepRuns: { @@ -190,7 +193,8 @@ function fixture(overrides = {}) { taskDefinitions: { async findCurrentTaskDefinition(projectId, taskId) { events.push(`task-get:${projectId}:${taskId}`); - return projectId === currentTask.projectId && taskId === currentTask.taskId + return projectId === currentTask.projectId && + taskId === currentTask.taskId ? currentTask : null; }, @@ -328,6 +332,7 @@ test('production composition exposes the reviewed Run and Workflow routes', asyn 'run.list', 'run.events.list', 'run.steps.list', + 'run.log.read', 'run.cancel', 'workflow.read', 'workflow.run.read', @@ -401,6 +406,15 @@ test('production composition exposes the reviewed Run and Workflow routes', asyn body: { steps: [], hasMore: false, next: null }, }); + const log = await invoke( + stack, + metadata('/api/v3/projects/project-1/runs/run-1/attempts/attempt-1/log'), + ); + assert.deepEqual(log, { + statusCode: 503, + body: { code: 'artifact_unavailable' }, + }); + const cancellation = await invoke( stack, metadata('/api/v3/projects/project-1/runs/run-1/cancellation', 'POST', { @@ -414,6 +428,7 @@ test('production composition exposes the reviewed Run and Workflow routes', asyn assert.equal(events.includes('audit:run.get:allowed'), true); assert.equal(events.includes('audit:run.events.list:allowed'), true); assert.equal(events.includes('audit:run.steps.list:allowed'), true); + assert.equal(events.includes('audit:run.log.read:allowed'), true); assert.equal(events.includes('audit:run.cancel:allowed'), true); const workflows = await invoke( @@ -539,6 +554,69 @@ test('production composition fails closed for an unreviewed route', async () => ); }); +test('wires the production Worker object reader into the Project-scoped log route', async () => { + const { input } = fixture(); + const run = await input.runs.findRunById('run-1'); + const logArtifactId = `wlog-${'a'.repeat(30)}`; + const stack = createProductionClusterControlApplicationStack({ + ...input, + runs: { + ...input.runs, + async findRunById() { + return { ...run, status: 'running' }; + }, + async findAttemptById() { + return { + id: 'attempt-1', + runId: 'run-1', + attempt: 1, + status: 'running', + executorType: 'remote_worker', + logArtifactId, + callbackSequence: 0, + createdAtMs: 1, + }; + }, + }, + workerRuntime: { + offers: { claimNext() {} }, + activation: { + acknowledgeStarting() {}, + acknowledgeRunning() {}, + failStart() {}, + }, + artifacts: { upload() {} }, + completion: { complete() {} }, + leaseControl: { control() {} }, + runAttemptLogRead: { + async read(identity, range) { + assert.equal(identity.logArtifactId, logArtifactId); + assert.deepEqual(range, { offset: 1, length: 4 }); + return { + status: 'available', + content: Buffer.from('prod'), + start: 1, + endExclusive: 5, + totalBytes: 5, + truncation: { truncated: false }, + }; + }, + }, + }, + }); + const result = await invoke( + stack, + metadata( + '/api/v3/projects/project-1/runs/run-1/attempts/attempt-1/log', + 'GET', + null, + { offset: ['1'], length: ['4'] }, + ), + ); + assert.equal(result.statusCode, 200); + assert.equal(Buffer.from(result.body.content, 'base64').toString(), 'prod'); +}); + test('optionally exposes Prompt execution behind shared admission and policy', async () => { const { events, input } = fixture(); let command; diff --git a/packages/ql3-cluster-control/test/runAttemptLogReadRoute.test.cjs b/packages/ql3-cluster-control/test/runAttemptLogReadRoute.test.cjs new file mode 100644 index 00000000..80e606ba --- /dev/null +++ b/packages/ql3-cluster-control/test/runAttemptLogReadRoute.test.cjs @@ -0,0 +1,251 @@ +const assert = require('node:assert/strict'); +const { test } = require('node:test'); + +const { + createClusterControlAdmissionPipeline, +} = require('@qinglong/cluster-control/admission'); +const { + createClusterControlRouteRegistry, +} = require('@qinglong/cluster-control/routes'); +const { + CLUSTER_CONTROL_RUN_ATTEMPT_LOG_READ_ROUTE, + createClusterControlRunAttemptLogReadRoute, +} = require('../dist/run/runAttemptLogReadRoute.js'); + +const PRINCIPAL = Object.freeze({ + subject: Object.freeze({ type: 'user', id: 'usr_viewer' }), + authenticationId: 'session:viewer', + authenticatedAtMs: 9_000, + expiresAtMs: 11_000, + assurance: 'single_factor', +}); + +function run(overrides = {}) { + return { + id: 'run_123', + projectId: 'prj_default', + taskId: 'task_1', + taskRevision: 'revision_1', + triggerType: 'task_start', + executionOrigin: 'manual', + executionOwner: 'runtime', + status: 'running', + version: 2, + eventSequence: 2, + priority: 0, + createdAtMs: 1, + ...overrides, + }; +} + +function attempt(overrides = {}) { + return { + id: 'attempt_123', + runId: 'run_123', + attempt: 1, + status: 'running', + executorType: 'remote_worker', + logArtifactId: `wlog-${'a'.repeat(30)}`, + callbackSequence: 0, + createdAtMs: 1, + ...overrides, + }; +} + +function metadata(query = {}) { + return Object.freeze({ + requestId: 'request-log-read', + method: 'GET', + path: '/api/v3/projects/prj_default/runs/run_123/attempts/attempt_123/log', + query: Object.freeze(query), + headers: Object.freeze({ authorization: 'Bearer opaque' }), + signal: new AbortController().signal, + }); +} + +function pipeline(options = {}) { + const events = options.events ?? []; + const repository = options.repository ?? { + async findRunById() { + events.push('run'); + return run(); + }, + async findAttemptById() { + events.push('attempt'); + return attempt(); + }, + }; + const reader = options.reader ?? { + async read(identity, range) { + events.push(`storage:${range.offset}:${range.length}`); + return { + status: 'available', + content: Buffer.from('cluster-log'), + start: range.offset, + endExclusive: range.offset + 11, + totalBytes: range.offset + 20, + nextOffset: range.offset + 11, + truncation: { truncated: false }, + }; + }, + }; + return createClusterControlAdmissionPipeline({ + routes: createClusterControlRouteRegistry([ + createClusterControlRunAttemptLogReadRoute(repository, reader), + ]), + authenticator: { + authenticate() { + events.push('authenticate'); + return PRINCIPAL; + }, + }, + policy: { + authorize(request) { + events.push(`authorize:${request.permission}`); + return options.effect + ? { effect: options.effect, reasons: ['masked'], fence: null } + : { + effect: 'allow', + reasons: ['role_grant'], + fence: { projectVersion: 2, bindingVersion: 3 }, + }; + }, + }, + audit: { + record(record) { + events.push(`audit:${record.outcome}`); + }, + }, + now: () => 10_000, + }); +} + +async function invoke(value, query = {}) { + const prepared = await value.prepare(metadata(query)); + return prepared.handle(null); +} + +test('publishes the immutable Artifact-scoped route contract', () => { + assert.deepEqual(CLUSTER_CONTROL_RUN_ATTEMPT_LOG_READ_ROUTE, { + method: 'GET', + path: '/api/v3/projects/{projectId}/runs/{runId}/attempts/{attemptId}/log', + operationId: 'run.log.read', + permission: 'artifact.read', + projectParameter: 'projectId', + allowedQuery: ['length', 'offset'], + }); +}); + +test('authorizes and audits before metadata and one bounded range read', async () => { + const events = []; + const result = await invoke(pipeline({ events }), { + offset: ['4'], + length: ['16'], + }); + assert.equal(result.statusCode, 200); + assert.equal(result.body.schema, 'qinglong/run-attempt-log-read-result@v1'); + assert.equal( + Buffer.from(result.body.content, 'base64').toString(), + 'cluster-log', + ); + assert.deepEqual(result.body.range, { + start: 4, + endExclusive: 15, + totalBytes: 24, + nextOffset: 15, + }); + assert.deepEqual(events, [ + 'authenticate', + 'authorize:artifact.read', + 'audit:allowed', + 'run', + 'attempt', + 'storage:4:16', + ]); +}); + +test('uses the Cluster default window and rejects unbounded query values', async () => { + const events = []; + assert.equal((await invoke(pipeline({ events }))).statusCode, 200); + assert.equal(events.at(-1), `storage:0:${64 * 1024}`); + for (const query of [ + { offset: ['-1'] }, + { offset: ['04'] }, + { length: ['0'] }, + { length: [String(256 * 1024 + 1)] }, + { length: ['1', '2'] }, + ]) { + await assert.rejects( + pipeline().prepare(metadata(query)), + (error) => + error.statusCode === 400 && error.code === 'invalid_route_query', + ); + } +}); + +test('masks deny and approval without reading Run or object storage', async () => { + for (const effect of ['deny', 'require_approval']) { + let touched = false; + await assert.rejects( + pipeline({ + effect, + repository: { + async findRunById() { + touched = true; + return run(); + }, + async findAttemptById() { + touched = true; + return attempt(); + }, + }, + reader: { + async read() { + touched = true; + return { status: 'missing' }; + }, + }, + }).prepare(metadata()), + (error) => + error.statusCode === 404 && error.code === 'artifact_not_found', + ); + assert.equal(touched, false); + } +}); + +test('returns pending during upload and fails closed without an object reader', async () => { + const pending = pipeline({ + reader: { + async read() { + return { status: 'missing' }; + }, + }, + }); + assert.equal((await invoke(pending)).statusCode, 202); + + const unavailableRoute = createClusterControlRunAttemptLogReadRoute({ + async findRunById() { + return run(); + }, + async findAttemptById() { + return attempt(); + }, + }); + const prepared = await createClusterControlAdmissionPipeline({ + routes: createClusterControlRouteRegistry([unavailableRoute]), + authenticator: { authenticate: () => PRINCIPAL }, + policy: { + authorize: () => ({ + effect: 'allow', + reasons: ['role_grant'], + fence: { projectVersion: 1, bindingVersion: 1 }, + }), + }, + audit: { record() {} }, + now: () => 10_000, + }).prepare(metadata()); + assert.deepEqual(await prepared.handle(null), { + statusCode: 503, + body: { code: 'artifact_unavailable' }, + }); +}); diff --git a/packages/ql3-cluster-control/test/s3ArtifactStore.integration.test.cjs b/packages/ql3-cluster-control/test/s3ArtifactStore.integration.test.cjs index ba1e52f5..480e523e 100644 --- a/packages/ql3-cluster-control/test/s3ArtifactStore.integration.test.cjs +++ b/packages/ql3-cluster-control/test/s3ArtifactStore.integration.test.cjs @@ -19,80 +19,111 @@ const endpoint = process.env.QL3_TEST_S3_ENDPOINT; const accessKeyId = process.env.QL3_TEST_S3_ACCESS_KEY_ID; const secretAccessKey = process.env.QL3_TEST_S3_SECRET_ACCESS_KEY; -test('real S3-compatible service preserves immutable Artifact evidence', { - skip: endpoint && accessKeyId && secretAccessKey - ? false - : 'requires QL3_TEST_S3_ENDPOINT and credentials', -}, async () => { - const client = new S3Client({ - endpoint, - region: 'us-east-1', - forcePathStyle: true, - credentials: { accessKeyId, secretAccessKey }, - }); - const bucket = `ql3-artifact-${process.pid}-${Date.now()}`.slice(0, 63); - const command = Object.freeze({ - projectId: 'project-s3-integration', - runId: 'run-s3-integration', - attemptId: 'attempt-s3-integration', - logArtifactId: `wlog-${'c'.repeat(30)}`, - byteLength: 17, - truncated: true, - }); - const content = Buffer.from('real object bytes'); - const body = (value) => Object.freeze({ - async *[Symbol.asyncIterator]() { - yield value.subarray(0, 4); - yield value.subarray(4); - }, - }); - - try { - await client.send(new CreateBucketCommand({ Bucket: bucket })); - const store = new S3ClusterRemoteWorkerArtifactStore({ - client, - bucket, - prefix: 'qinglong/integration', - encryption: { mode: 's3' }, +test( + 'real S3-compatible service preserves immutable Artifact evidence', + { + skip: + endpoint && accessKeyId && secretAccessKey + ? false + : 'requires QL3_TEST_S3_ENDPOINT and credentials', + }, + async () => { + const client = new S3Client({ + endpoint, + region: 'us-east-1', + forcePathStyle: true, + credentials: { accessKeyId, secretAccessKey }, }); - const stored = await store.put(command, body(content)); - assert.equal(stored.status, 'stored'); - assert.equal( - stored.sha256, - createHash('sha256').update(content).digest('hex'), - ); - const replay = await store.put(command, body(content)); - assert.equal(replay.status, 'already_stored'); - await assert.rejects( - store.put(command, body(Buffer.from('REAL OBJECT BYTES'))), - (error) => - error instanceof S3ClusterRemoteWorkerArtifactStoreError && - error.reason === 'integrity_mismatch', - ); - const objects = await client.send(new ListObjectsV2Command({ - Bucket: bucket, - Prefix: 'qinglong/integration/', - })); - assert.equal(objects.KeyCount, 1); - assert.match(objects.Contents[0].Key, /\/objects\//); - } finally { + const bucket = `ql3-artifact-${process.pid}-${Date.now()}`.slice(0, 63); + const command = Object.freeze({ + projectId: 'project-s3-integration', + runId: 'run-s3-integration', + attemptId: 'attempt-s3-integration', + logArtifactId: `wlog-${'c'.repeat(30)}`, + byteLength: 17, + truncated: true, + }); + const content = Buffer.from('real object bytes'); + const body = (value) => + Object.freeze({ + async *[Symbol.asyncIterator]() { + yield value.subarray(0, 4); + yield value.subarray(4); + }, + }); + try { - const objects = await client.send(new ListObjectsV2Command({ - Bucket: bucket, - })); - if (objects.Contents?.length) { - await client.send(new DeleteObjectsCommand({ + await client.send(new CreateBucketCommand({ Bucket: bucket })); + const store = new S3ClusterRemoteWorkerArtifactStore({ + client, + bucket, + prefix: 'qinglong/integration', + encryption: { mode: 's3' }, + }); + const stored = await store.put(command, body(content)); + assert.equal(stored.status, 'stored'); + assert.equal( + stored.sha256, + createHash('sha256').update(content).digest('hex'), + ); + const replay = await store.put(command, body(content)); + assert.equal(replay.status, 'already_stored'); + const range = await store.readLogRange( + { + projectId: command.projectId, + runId: command.runId, + attemptId: command.attemptId, + logArtifactId: command.logArtifactId, + }, + { + offset: 5, + length: 6, + }, + ); + assert.equal(range.status, 'available'); + assert.equal(Buffer.from(range.content).toString(), 'object'); + assert.equal(range.start, 5); + assert.equal(range.endExclusive, 11); + assert.equal(range.totalBytes, content.byteLength); + assert.equal(range.nextOffset, 11); + assert.deepEqual(range.truncation, { truncated: true }); + await assert.rejects( + store.put(command, body(Buffer.from('REAL OBJECT BYTES'))), + (error) => + error instanceof S3ClusterRemoteWorkerArtifactStoreError && + error.reason === 'integrity_mismatch', + ); + const objects = await client.send( + new ListObjectsV2Command({ Bucket: bucket, - Delete: { - Objects: objects.Contents.map(({ Key }) => ({ Key })), - Quiet: true, - }, - })); + Prefix: 'qinglong/integration/', + }), + ); + assert.equal(objects.KeyCount, 1); + assert.match(objects.Contents[0].Key, /\/objects\//); + } finally { + try { + const objects = await client.send( + new ListObjectsV2Command({ + Bucket: bucket, + }), + ); + if (objects.Contents?.length) { + await client.send( + new DeleteObjectsCommand({ + Bucket: bucket, + Delete: { + Objects: objects.Contents.map(({ Key }) => ({ Key })), + Quiet: true, + }, + }), + ); + } + await client.send(new DeleteBucketCommand({ Bucket: bucket })); + } catch { + // Preserve the integration assertion; the ephemeral container is removed. } - await client.send(new DeleteBucketCommand({ Bucket: bucket })); - } catch { - // Preserve the integration assertion; the ephemeral container is removed. + client.destroy(); } - client.destroy(); - } -}); + }, +); diff --git a/packages/ql3-cluster-control/test/s3ArtifactStore.test.cjs b/packages/ql3-cluster-control/test/s3ArtifactStore.test.cjs index e4b682e0..1ee8a3b6 100644 --- a/packages/ql3-cluster-control/test/s3ArtifactStore.test.cjs +++ b/packages/ql3-cluster-control/test/s3ArtifactStore.test.cjs @@ -6,6 +6,7 @@ const { test } = require('node:test'); const { CopyObjectCommand, DeleteObjectCommand, + GetObjectCommand, HeadObjectCommand, PutObjectCommand, } = require('@aws-sdk/client-s3'); @@ -57,19 +58,58 @@ class MemoryS3Client { const object = this.objects.get(input.Key); if (!object) throw notFound(); const metadata = { ...object.metadata }; - if (this.options.corruptFinalMetadata && input.Key.includes('/objects/')) { + if ( + this.options.corruptFinalMetadata && + input.Key.includes('/objects/') + ) { metadata['ql3-content-sha256'] = '0'.repeat(64); } return { ContentLength: object.content.byteLength, ContentType: object.contentType, - ChecksumSHA256: this.options.corruptFinalChecksum && - input.Key.includes('/objects/') - ? Buffer.alloc(32, 9).toString('base64') - : checksum(object.content), + ETag: `"${checksum(object.content).slice(0, 32)}"`, + ChecksumSHA256: + this.options.corruptFinalChecksum && input.Key.includes('/objects/') + ? Buffer.alloc(32, 9).toString('base64') + : checksum(object.content), Metadata: metadata, }; } + if (command instanceof GetObjectCommand) { + const object = this.objects.get(input.Key); + if (!object || this.options.rangeNotFound) throw notFound(); + const eTag = `"${checksum(object.content).slice(0, 32)}"`; + assert.equal(input.IfMatch, eTag); + const match = /^bytes=(\d+)-(\d+)$/.exec(input.Range); + assert.ok(match); + const start = Number(match[1]); + const end = Number(match[2]); + let content = object.content.subarray(start, end + 1); + if (this.options.shortRangeBody) content = content.subarray(0, -1); + if (this.options.oversizedRangeBody) { + content = Buffer.concat([content, Buffer.from('x')]); + } + const metadata = { ...object.metadata }; + if (this.options.corruptRangeMetadata) { + metadata['ql3-run-sha256'] = '0'.repeat(64); + } + return { + ContentLength: end - start + 1, + ContentRange: this.options.corruptContentRange + ? `bytes ${start}-${end}/${object.content.byteLength + 1}` + : `bytes ${start}-${end}/${object.content.byteLength}`, + ContentType: object.contentType, + ETag: this.options.corruptRangeETag ? '"other"' : eTag, + Metadata: metadata, + Body: { + async *[Symbol.asyncIterator]() { + const split = Math.min(2, content.byteLength); + if (split > 0) yield content.subarray(0, split); + if (split < content.byteLength) yield content.subarray(split); + }, + }, + }; + } if (command instanceof PutObjectCommand) { assert.equal(input.IfNoneMatch, '*'); assert.equal(input.ChecksumAlgorithm, 'SHA256'); @@ -183,14 +223,25 @@ test('streams to a checksummed temporary object then conditionally promotes it', ], ); const key = permanentKey(client); - assert.match(key, /^tenant-a\/worker-artifacts\/objects\/[a-f0-9]{2}\/[a-f0-9]{64}$/); + assert.match( + key, + /^tenant-a\/worker-artifacts\/objects\/[a-f0-9]{2}\/[a-f0-9]{64}$/, + ); assert.equal(key.includes(COMMAND.runId), false); assert.equal(client.objects.size, 1); - const copy = client.commands.find((command) => command instanceof CopyObjectCommand); + const copy = client.commands.find( + (command) => command instanceof CopyObjectCommand, + ); assert.equal(copy.input.Metadata['ql3-content-sha256'], CONTENT_SHA256); - assert.equal(JSON.stringify(copy.input.Metadata).includes(COMMAND.projectId), false); - assert.equal(JSON.stringify(copy.input.Metadata).includes(COMMAND.runId), false); + assert.equal( + JSON.stringify(copy.input.Metadata).includes(COMMAND.projectId), + false, + ); + assert.equal( + JSON.stringify(copy.input.Metadata).includes(COMMAND.runId), + false, + ); const inspected = await adapter.inspect(LOOKUP); assert.deepEqual(inspected, { ...receipt, status: 'already_stored' }); @@ -219,6 +270,78 @@ test('exact replay consumes and hashes the whole body without another write', as ); }); +test('reads only one ETag-fenced immutable byte range and stable end snapshot', async () => { + const client = new MemoryS3Client(); + const adapter = store(client); + await adapter.put(COMMAND, chunks()); + client.commands.length = 0; + + const result = await adapter.readLogRange(LOOKUP, { offset: 2, length: 4 }); + assert.equal(result.status, 'available'); + assert.equal(Buffer.from(result.content).toString(), 'llo '); + assert.deepEqual( + { + start: result.start, + endExclusive: result.endExclusive, + totalBytes: result.totalBytes, + nextOffset: result.nextOffset, + truncation: result.truncation, + }, + { + start: 2, + endExclusive: 6, + totalBytes: 11, + nextOffset: 6, + truncation: { truncated: false }, + }, + ); + assert.deepEqual( + client.commands.map((command) => command.constructor.name), + ['HeadObjectCommand', 'GetObjectCommand'], + ); + assert.equal(client.commands[1].input.Range, 'bytes=2-5'); + + client.commands.length = 0; + const ended = await adapter.readLogRange(LOOKUP, { + offset: 999, + length: 4, + }); + assert.equal(ended.status, 'available'); + assert.equal(ended.content.byteLength, 0); + assert.equal(ended.start, 11); + assert.equal(ended.totalBytes, 11); + assert.deepEqual( + client.commands.map((command) => command.constructor.name), + ['HeadObjectCommand'], + ); +}); + +test('maps absent objects and fails closed on range evidence drift', async () => { + const absent = new MemoryS3Client(); + assert.deepEqual( + await store(absent).readLogRange(LOOKUP, { offset: 0, length: 1 }), + { status: 'missing' }, + ); + for (const option of [ + 'corruptContentRange', + 'corruptRangeETag', + 'corruptRangeMetadata', + 'shortRangeBody', + 'oversizedRangeBody', + ]) { + const client = new MemoryS3Client(); + const adapter = store(client); + await adapter.put(COMMAND, chunks()); + client.options[option] = true; + await assert.rejects( + adapter.readLogRange(LOOKUP, { offset: 0, length: 4 }), + (error) => + error instanceof S3ClusterRemoteWorkerArtifactStoreError && + error.reason === 'integrity_mismatch', + ); + } +}); + test('resolves a concurrent conditional-copy winner by immutable inspect', async () => { const client = new MemoryS3Client({ raceOnCopy: true }); const receipt = await store(client).put(COMMAND, chunks()); @@ -270,38 +393,40 @@ test('temporary cleanup failure is diagnostic and never reverses promotion', asy }, }).put(COMMAND, chunks()); assert.equal(receipt.status, 'stored'); - assert.deepEqual(diagnostics, [[ - 'delete unavailable', - 'temporary_object_cleanup', - ]]); + assert.deepEqual(diagnostics, [ + ['delete unavailable', 'temporary_object_cleanup'], + ]); assert.equal(client.objects.size, 2); }); test('requires exact bucket, prefix, encryption and temporary ID configuration', async () => { const client = new MemoryS3Client(); assert.throws( - () => new S3ClusterRemoteWorkerArtifactStore({ - client, - bucket: 'Invalid_Bucket', - encryption: { mode: 's3' }, - }), + () => + new S3ClusterRemoteWorkerArtifactStore({ + client, + bucket: 'Invalid_Bucket', + encryption: { mode: 's3' }, + }), /bucket is invalid/, ); assert.throws( - () => new S3ClusterRemoteWorkerArtifactStore({ - client, - bucket: 'valid-bucket', - prefix: '../escape', - encryption: { mode: 's3' }, - }), + () => + new S3ClusterRemoteWorkerArtifactStore({ + client, + bucket: 'valid-bucket', + prefix: '../escape', + encryption: { mode: 's3' }, + }), /prefix is invalid/, ); assert.throws( - () => new S3ClusterRemoteWorkerArtifactStore({ - client, - bucket: 'valid-bucket', - encryption: { mode: 'kms' }, - }), + () => + new S3ClusterRemoteWorkerArtifactStore({ + client, + bucket: 'valid-bucket', + encryption: { mode: 'kms' }, + }), /encryption is invalid/, ); await assert.rejects( @@ -355,8 +480,7 @@ test('propagates KMS and expected-owner fences to both sides of promotion', asyn test('never deletes a colliding temporary object it cannot prove it owns', async () => { const client = new MemoryS3Client(); - const temporaryKey = - `tenant-a/worker-artifacts/temporary/${TEMPORARY_ID}`; + const temporaryKey = `tenant-a/worker-artifacts/temporary/${TEMPORARY_ID}`; client.objects.set(temporaryKey, { content: Buffer.from('other operation'), contentType: 'application/octet-stream', diff --git a/packages/ql3-local-api/src/admission/localApiAdmission.ts b/packages/ql3-local-api/src/admission/localApiAdmission.ts index 0754b046..6af7ed09 100644 --- a/packages/ql3-local-api/src/admission/localApiAdmission.ts +++ b/packages/ql3-local-api/src/admission/localApiAdmission.ts @@ -22,6 +22,7 @@ import type { LocalApiRunListRoute } from '../run/runListRoute'; import type { LocalApiRunReadRoute } from '../run/runReadRoute'; import type { LocalApiRunStepListRoute } from '../run/runStepListRoute'; import type { LocalApiRunCancellationRoute } from '../run/runCancellationRoute'; +import type { LocalApiRunAttemptLogReadRoute } from '../run/runAttemptLogReadRoute'; import type { LocalApiTaskListRoute } from '../task/taskListRoute'; import type { LocalApiTaskReadRoute } from '../task/taskReadRoute'; import type { LocalApiTaskStartRoute } from '../task/taskStartRoute'; @@ -50,6 +51,14 @@ export type LocalApiAdmissionOperation = runId: string; input: Readonly; }> + | Readonly<{ + operationId: 'run.log.read'; + projectId: string; + runId: string; + attemptId: string; + offset: number; + length: number; + }> | Readonly<{ operationId: 'run.cancel'; projectId: string; @@ -99,6 +108,7 @@ export interface LocalApiAdmissionOptions { readonly runEventListRoute: LocalApiRunEventListRoute; readonly runStepListRoute: LocalApiRunStepListRoute; readonly runCancellationRoute: LocalApiRunCancellationRoute; + readonly runAttemptLogReadRoute: LocalApiRunAttemptLogReadRoute; readonly taskListRoute: LocalApiTaskListRoute; readonly taskReadRoute: LocalApiTaskReadRoute; readonly taskStartRoute: LocalApiTaskStartRoute; @@ -176,6 +186,7 @@ export function createLocalApiAdmission( typeof options.runEventListRoute?.handle !== 'function' || typeof options.runStepListRoute?.handle !== 'function' || typeof options.runCancellationRoute?.handle !== 'function' || + typeof options.runAttemptLogReadRoute?.handle !== 'function' || typeof options.taskListRoute?.handle !== 'function' || typeof options.taskReadRoute?.handle !== 'function' || typeof options.taskStartRoute?.handle !== 'function' || @@ -236,12 +247,14 @@ export function createLocalApiAdmission( request.operation.projectId, request.operation.operationId === 'run.cancel' ? 'run.stop' + : request.operation.operationId === 'run.log.read' + ? 'artifact.read' : request.operation.operationId === 'task.start' - ? 'run.start' + ? 'run.start' : request.operation.operationId === 'task.list' || - request.operation.operationId === 'task.get' - ? 'task.read' - : 'run.read', + request.operation.operationId === 'task.get' + ? 'task.read' + : 'run.read', ), ); } catch { @@ -279,9 +292,15 @@ export function createLocalApiAdmission( ), ); if (auditFailure) return auditFailure; - if (decision.effect === 'deny') return response(403, 'forbidden'); + if (decision.effect === 'deny') { + return request.operation.operationId === 'run.log.read' + ? response(404, 'artifact_not_found') + : response(403, 'forbidden'); + } if (decision.effect === 'require_approval') { - return response(403, 'approval_required'); + return request.operation.operationId === 'run.log.read' + ? response(404, 'artifact_not_found') + : response(403, 'approval_required'); } if (request.signal.aborted) return response(503, 'request_unavailable'); try { @@ -337,6 +356,16 @@ export function createLocalApiAdmission( principal: authenticated.principal, policyFence: decision.fence, }); + case 'run.log.read': + if (body !== null) return response(400, 'invalid_request_body'); + return options.runAttemptLogReadRoute.handle({ + projectId: request.operation.projectId, + runId: request.operation.runId, + attemptId: request.operation.attemptId, + offset: request.operation.offset, + length: request.operation.length, + signal: request.signal, + }); case 'task.list': if (body !== null) return response(400, 'invalid_request_body'); return options.taskListRoute.handle({ diff --git a/packages/ql3-local-api/src/application-runtime/localApiProductSurface.ts b/packages/ql3-local-api/src/application-runtime/localApiProductSurface.ts index 05e4c224..ba8ba41e 100644 --- a/packages/ql3-local-api/src/application-runtime/localApiProductSurface.ts +++ b/packages/ql3-local-api/src/application-runtime/localApiProductSurface.ts @@ -15,6 +15,7 @@ import { createLocalApiRunReadRoute } from '../run/runReadRoute'; import { createLocalApiRunEventListRoute } from '../run/runEventListRoute'; import { createLocalApiRunStepListRoute } from '../run/runStepListRoute'; import { createLocalApiRunCancellationRoute } from '../run/runCancellationRoute'; +import { createLocalApiRunAttemptLogReadRoute } from '../run/runAttemptLogReadRoute'; import { createLocalApiTaskListRoute } from '../task/taskListRoute'; import { createLocalApiTaskReadRoute } from '../task/taskReadRoute'; import { createLocalApiTaskStartRoute } from '../task/taskStartRoute'; @@ -109,6 +110,9 @@ export function createLocalApiProductSurface( authority.runCancellation, options.randomUuid ?? randomUUID, ); + const runAttemptLogReadRoute = createLocalApiRunAttemptLogReadRoute( + authority.runAttemptLogRead, + ); const taskListRoute = createLocalApiTaskListRoute( authority.taskDefinitions, ); @@ -128,6 +132,7 @@ export function createLocalApiProductSurface( runEventListRoute, runStepListRoute, runCancellationRoute, + runAttemptLogReadRoute, taskListRoute, taskReadRoute, taskStartRoute, diff --git a/packages/ql3-local-api/src/run/runAttemptLogReadRoute.ts b/packages/ql3-local-api/src/run/runAttemptLogReadRoute.ts new file mode 100644 index 00000000..69eafca8 --- /dev/null +++ b/packages/ql3-local-api/src/run/runAttemptLogReadRoute.ts @@ -0,0 +1,111 @@ +import { + InvalidRunAttemptLogReadError, + RunAttemptLogReadUnavailableError, + type RunAttemptLogReadRequest, + type RunAttemptLogReadResult, +} from '@qinglong/runtime-core/run-attempt-log-read'; + +import type { LocalApiResponse } from '../transport/contract'; + +export interface LocalApiRunAttemptLogReadCapability { + read( + request: Readonly, + ): Promise; +} + +export interface LocalApiRunAttemptLogReadRequest { + readonly projectId: string; + readonly runId: string; + readonly attemptId: string; + readonly offset: number; + readonly length: number; + readonly signal?: AbortSignal; +} + +export interface LocalApiRunAttemptLogReadRoute { + handle( + request: Readonly, + ): Promise; +} + +function response( + statusCode: number, + body: Readonly>, +): LocalApiResponse { + return Object.freeze({ statusCode, body: Object.freeze(body) }); +} + +function projection( + result: Extract, +): Readonly> { + return Object.freeze({ + schema: 'qinglong/run-attempt-log-read-result@v1', + status: 'available', + projectId: result.projectId, + runId: result.runId, + attemptId: result.attemptId, + range: Object.freeze({ + start: result.start, + endExclusive: result.endExclusive, + totalBytes: result.totalBytes, + ...(result.nextOffset === undefined + ? {} + : { nextOffset: result.nextOffset }), + }), + encoding: 'base64', + content: Buffer.from( + result.content.buffer, + result.content.byteOffset, + result.content.byteLength, + ).toString('base64'), + truncation: result.truncation, + }); +} + +export function createLocalApiRunAttemptLogReadRoute( + capability: LocalApiRunAttemptLogReadCapability, +): Readonly { + if (!capability || typeof capability.read !== 'function') { + throw new TypeError('Local API Run Attempt log read capability is invalid'); + } + return Object.freeze({ + async handle(request: Readonly) { + try { + const result = await capability.read({ + projectId: request.projectId, + runId: request.runId, + attemptId: request.attemptId, + range: Object.freeze({ + offset: request.offset, + length: request.length, + }), + ...(request.signal === undefined ? {} : { signal: request.signal }), + }); + if (result.status === 'not_found') { + return response(404, { code: 'artifact_not_found' }); + } + if (result.status === 'pending') { + return response(202, { + schema: 'qinglong/run-attempt-log-read-result@v1', + status: 'pending', + projectId: result.projectId, + runId: result.runId, + attemptId: result.attemptId, + }); + } + if (result.status === 'missing') { + return response(503, { code: 'artifact_unavailable' }); + } + return response(200, projection(result)); + } catch (error) { + if (error instanceof InvalidRunAttemptLogReadError) { + return response(400, { code: 'invalid_run_log_read_request' }); + } + if (error instanceof RunAttemptLogReadUnavailableError) { + return response(503, { code: 'artifact_unavailable' }); + } + return response(503, { code: 'artifact_unavailable' }); + } + }, + }); +} diff --git a/packages/ql3-local-api/src/transport/httpSurface.ts b/packages/ql3-local-api/src/transport/httpSurface.ts index 4f086abb..3ab9ac7f 100644 --- a/packages/ql3-local-api/src/transport/httpSurface.ts +++ b/packages/ql3-local-api/src/transport/httpSurface.ts @@ -28,6 +28,8 @@ const RUN_STEP_LIST_ROUTE_PATTERN = /^\/api\/v3\/projects\/([A-Za-z0-9][A-Za-z0-9._:-]{0,127})\/runs\/([A-Za-z0-9][A-Za-z0-9._:-]{0,127})\/steps$/; const RUN_CANCELLATION_ROUTE_PATTERN = /^\/api\/v3\/projects\/([A-Za-z0-9][A-Za-z0-9._:-]{0,127})\/runs\/([A-Za-z0-9][A-Za-z0-9._:-]{0,127})\/cancellation$/; +const RUN_ATTEMPT_LOG_READ_ROUTE_PATTERN = + /^\/api\/v3\/projects\/([A-Za-z0-9][A-Za-z0-9._:-]{0,127})\/runs\/([A-Za-z0-9][A-Za-z0-9._:-]{0,127})\/attempts\/([A-Za-z0-9][A-Za-z0-9._:-]{0,127})\/log$/; const TASK_LIST_ROUTE_PATTERN = /^\/api\/v3\/projects\/([A-Za-z0-9][A-Za-z0-9._:-]{0,127})\/tasks$/; const TASK_READ_ROUTE_PATTERN = @@ -44,6 +46,7 @@ type LocalApiRouteResolution = | 'invalid_run_list_query' | 'invalid_run_event_list_query' | 'invalid_run_step_list_query' + | 'invalid_run_log_read_query' | 'invalid_task_list_query'; }>; @@ -340,10 +343,7 @@ function parseTaskListQuery( } const name = field.slice(0, separator); const value = field.slice(separator + 1); - if ( - values.has(name) || - (name !== 'limit' && name !== 'after_task_id') - ) { + if (values.has(name) || (name !== 'limit' && name !== 'after_task_id')) { throw new TypeError(); } values.set(name, value); @@ -363,13 +363,58 @@ function parseTaskListQuery( } return Object.freeze({ ...(limit === undefined ? {} : { limit }), - ...(taskId === undefined - ? {} - : { after: Object.freeze({ taskId }) }), + ...(taskId === undefined ? {} : { after: Object.freeze({ taskId }) }), }); } -function route(request: IncomingMessage): LocalApiRouteResolution | null { +function parseRunAttemptLogReadQuery( + rawQuery: string | undefined, + profile: LocalApplicationProfile, +): Readonly<{ offset: number; length: number }> { + const defaultLength = profile === 'edge' ? 16 * 1024 : 32 * 1024; + if (rawQuery === undefined) { + return Object.freeze({ offset: 0, length: defaultLength }); + } + if (rawQuery.length === 0) throw new TypeError(); + const values = new Map(); + for (const field of rawQuery.split('&')) { + const separator = field.indexOf('='); + if ( + separator < 1 || + separator !== field.lastIndexOf('=') || + separator === field.length - 1 + ) { + throw new TypeError(); + } + const name = field.slice(0, separator); + const value = field.slice(separator + 1); + if (values.has(name) || (name !== 'offset' && name !== 'length')) { + throw new TypeError(); + } + values.set(name, value); + } + const rawOffset = values.get('offset'); + const offset = rawOffset === undefined ? 0 : Number(rawOffset); + const rawLength = values.get('length'); + const length = rawLength === undefined ? defaultLength : Number(rawLength); + if ( + !Number.isSafeInteger(offset) || + offset < 0 || + (rawOffset !== undefined && String(offset) !== rawOffset) || + !Number.isSafeInteger(length) || + length < 1 || + length > 32 * 1024 || + (rawLength !== undefined && String(length) !== rawLength) + ) { + throw new TypeError(); + } + return Object.freeze({ offset, length }); +} + +function route( + request: IncomingMessage, + profile: LocalApplicationProfile, +): LocalApiRouteResolution | null { const rawUrl = request.url; if ( typeof rawUrl !== 'string' || @@ -403,6 +448,20 @@ function route(request: IncomingMessage): LocalApiRouteResolution | null { : null; } if (request.method !== 'GET') return null; + const runAttemptLogReadMatch = RUN_ATTEMPT_LOG_READ_ROUTE_PATTERN.exec(path); + if (runAttemptLogReadMatch) { + try { + return Object.freeze({ + operationId: 'run.log.read', + projectId: runAttemptLogReadMatch[1]!, + runId: runAttemptLogReadMatch[2]!, + attemptId: runAttemptLogReadMatch[3]!, + ...parseRunAttemptLogReadQuery(rawQuery, profile), + }); + } catch { + return Object.freeze({ errorCode: 'invalid_run_log_read_query' }); + } + } const taskReadMatch = TASK_READ_ROUTE_PATTERN.exec(path); if (taskReadMatch) { return rawQuery === undefined @@ -552,7 +611,7 @@ export async function startLocalApiHttpSurface( send(response, requestId, errorResponse(503, 'server_overloaded')); return; } - const resolvedRoute = route(request); + const resolvedRoute = route(request, options.profile); if (!resolvedRoute) { send(response, requestId, errorResponse(404, 'route_not_found')); return; @@ -608,9 +667,9 @@ export async function startLocalApiHttpSurface( error instanceof RangeError ? 'request_body_too_large' : error instanceof Error && - error.message === 'request_unavailable' - ? 'request_unavailable' - : 'invalid_request_body'; + error.message === 'request_unavailable' + ? 'request_unavailable' + : 'invalid_request_body'; send( response, requestId, @@ -618,8 +677,8 @@ export async function startLocalApiHttpSurface( code === 'request_body_too_large' ? 413 : code === 'request_unavailable' - ? 503 - : 400, + ? 503 + : 400, code, ), ); diff --git a/packages/ql3-local-api/test/admission.test.cjs b/packages/ql3-local-api/test/admission.test.cjs index 98ad3ced..f49f038d 100644 --- a/packages/ql3-local-api/test/admission.test.cjs +++ b/packages/ql3-local-api/test/admission.test.cjs @@ -101,6 +101,14 @@ function fixture(overrides = {}) { return { statusCode: 202, body: { status: 'accepted' } }; }, }, + runAttemptLogReadRoute: { + async handle(value) { + events.push( + `log:${value.projectId}:${value.runId}:${value.attemptId}:${value.offset}:${value.length}`, + ); + return { statusCode: 200, body: { status: 'available' } }; + }, + }, taskListRoute: { async handle(value) { events.push(`tasks:${value.projectId}:${value.input.limit ?? 32}`); @@ -148,6 +156,51 @@ test('authenticates, authorizes, durably audits and re-confirms before reading', ]); }); +test('uses artifact.read and masks denied or approval-fenced log existence', async () => { + const operation = Object.freeze({ + operationId: 'run.log.read', + projectId: 'prj_default', + runId: 'run_123', + attemptId: 'attempt_123', + offset: 4, + length: 16, + }); + const allowed = fixture(); + assert.deepEqual(await execute(allowed.admission, request({ operation })), { + statusCode: 200, + body: { status: 'available' }, + }); + assert.deepEqual(allowed.events, [ + 'authenticate', + 'authorize:artifact.read:prj_default', + 'audit:allowed:run.log.read', + 'confirm', + 'log:prj_default:run_123:attempt_123:4:16', + ]); + + for (const effect of ['deny', 'require_approval']) { + let routed = false; + const denied = fixture({ + policy: { + async authorize() { + return { effect, reasons: ['masked'], fence: null }; + }, + }, + runAttemptLogReadRoute: { + async handle() { + routed = true; + throw new Error('must not route'); + }, + }, + }); + assert.deepEqual(await execute(denied.admission, request({ operation })), { + statusCode: 404, + body: { code: 'artifact_not_found' }, + }); + assert.equal(routed, false); + } +}); + test('uses the same admission chain with a route-owned run.list audit identity', async () => { const { admission, events } = fixture(); assert.deepEqual( diff --git a/packages/ql3-local-api/test/httpSurface.test.cjs b/packages/ql3-local-api/test/httpSurface.test.cjs index f4761f75..0719a193 100644 --- a/packages/ql3-local-api/test/httpSurface.test.cjs +++ b/packages/ql3-local-api/test/httpSurface.test.cjs @@ -74,60 +74,71 @@ test('serves only the fixed canonical loopback Run route and drains idempotently host: '127.0.0.1', port, admission: preparedAdmission(async (value, body) => { - observed.push(value); - if ( - value.operation.operationId === 'run.cancel' || - value.operation.operationId === 'task.start' - ) { - return { statusCode: 202, body: { accepted: body } }; - } - if (value.operation.operationId === 'run.get') { - return { - statusCode: 200, - body: { run: { id: value.operation.runId } }, - }; - } - if (value.operation.operationId === 'run.events.list') { - return { - statusCode: 200, - body: { - events: [], - hasMore: false, - nextAfterSequence: value.operation.input.afterSequence ?? 0, - }, - }; - } - if (value.operation.operationId === 'run.steps.list') { - return { - statusCode: 200, - body: { - steps: [], - hasMore: false, - next: value.operation.input.after ?? null, - }, - }; - } - if (value.operation.operationId === 'task.list') { - return { - statusCode: 200, - body: { - tasks: [], - hasMore: false, - input: value.operation.input, - }, - }; - } - if (value.operation.operationId === 'task.get') { - return { - statusCode: 200, - body: { task: { taskId: value.operation.taskId } }, - }; - } + observed.push(value); + if ( + value.operation.operationId === 'run.cancel' || + value.operation.operationId === 'task.start' + ) { + return { statusCode: 202, body: { accepted: body } }; + } + if (value.operation.operationId === 'run.get') { return { statusCode: 200, - body: { runs: [], hasMore: false, input: value.operation.input }, + body: { run: { id: value.operation.runId } }, }; - }), + } + if (value.operation.operationId === 'run.events.list') { + return { + statusCode: 200, + body: { + events: [], + hasMore: false, + nextAfterSequence: value.operation.input.afterSequence ?? 0, + }, + }; + } + if (value.operation.operationId === 'run.steps.list') { + return { + statusCode: 200, + body: { + steps: [], + hasMore: false, + next: value.operation.input.after ?? null, + }, + }; + } + if (value.operation.operationId === 'run.log.read') { + return { + statusCode: 200, + body: { + range: { + offset: value.operation.offset, + length: value.operation.length, + }, + }, + }; + } + if (value.operation.operationId === 'task.list') { + return { + statusCode: 200, + body: { + tasks: [], + hasMore: false, + input: value.operation.input, + }, + }; + } + if (value.operation.operationId === 'task.get') { + return { + statusCode: 200, + body: { task: { taskId: value.operation.taskId } }, + }; + } + return { + statusCode: 200, + body: { runs: [], hasMore: false, input: value.operation.input }, + }; + }), randomUuid: () => '019f70c0-0000-4000-8000-000000000003', }); t.after(() => surface.stopAndDrain()); @@ -212,10 +223,7 @@ test('serves only the fixed canonical loopback Run route and drains idempotently input: { after: { taskId: 'task_100' }, limit: 8 }, }); - const task = await request( - port, - '/api/v3/projects/prj_default/tasks/task_1', - ); + const task = await request(port, '/api/v3/projects/prj_default/tasks/task_1'); assert.deepEqual(task.body, { task: { taskId: 'task_1' } }); assert.deepEqual(observed[5].operation, { operationId: 'task.get', @@ -275,6 +283,28 @@ test('serves only the fixed canonical loopback Run route and drains idempotently taskId: 'task_1', }); + const log = await request( + port, + '/api/v3/projects/prj_default/runs/run_123/attempts/attempt_1/log?offset=4&length=32', + ); + assert.deepEqual(log.body, { range: { offset: 4, length: 32 } }); + assert.deepEqual(observed[8].operation, { + operationId: 'run.log.read', + projectId: 'prj_default', + runId: 'run_123', + attemptId: 'attempt_1', + offset: 4, + length: 32, + }); + + const defaultLog = await request( + port, + '/api/v3/projects/prj_default/runs/run_123/attempts/attempt_1/log', + ); + assert.deepEqual(defaultLog.body, { + range: { offset: 0, length: 16 * 1024 }, + }); + for (const invalidPath of [ '/api/v3/projects/prj_default/runs/run_123?expanded=true', '/api/v3/projects/prj_default/runs/run%5f123', @@ -299,6 +329,18 @@ test('serves only the fixed canonical loopback Run route and drains idempotently assert.equal(invalid.statusCode, 400); assert.deepEqual(invalid.body, { code: 'invalid_run_list_query' }); } + for (const invalidQuery of [ + '/api/v3/projects/prj_default/runs/run_123/attempts/attempt_1/log?', + '/api/v3/projects/prj_default/runs/run_123/attempts/attempt_1/log?offset=-1', + '/api/v3/projects/prj_default/runs/run_123/attempts/attempt_1/log?offset=04', + '/api/v3/projects/prj_default/runs/run_123/attempts/attempt_1/log?length=0', + '/api/v3/projects/prj_default/runs/run_123/attempts/attempt_1/log?length=32769', + '/api/v3/projects/prj_default/runs/run_123/attempts/attempt_1/log?unknown=1', + ]) { + const invalid = await request(port, invalidQuery); + assert.equal(invalid.statusCode, 400); + assert.deepEqual(invalid.body, { code: 'invalid_run_log_read_query' }); + } for (const invalidQuery of [ '/api/v3/projects/prj_default/tasks?', '/api/v3/projects/prj_default/tasks?limit=08', @@ -332,7 +374,7 @@ test('serves only the fixed canonical loopback Run route and drains idempotently assert.equal(invalid.statusCode, 400); assert.deepEqual(invalid.body, { code: 'invalid_run_step_list_query' }); } - assert.equal(observed.length, 8); + assert.equal(observed.length, 10); assert.deepEqual( await Promise.all([surface.stopAndDrain(), surface.stopAndDrain()]), ['stopped', 'stopped'], @@ -347,9 +389,9 @@ test('rejects GET bodies without invoking the prepared route handler', async (t) host: '127.0.0.1', port, admission: preparedAdmission(async () => { - handlers += 1; - return { statusCode: 200, body: {} }; - }), + handlers += 1; + return { statusCode: 200, body: {} }; + }), }); t.after(() => surface.stopAndDrain()); const response = await request( @@ -477,15 +519,15 @@ test('serves the reviewed worst-case 64-item Run list inside the fixed response host: '127.0.0.1', port, admission: preparedAdmission(async () => { - return { - statusCode: 200, - body: { - runs: Object.freeze(Array.from({ length: 64 }, () => item)), - hasMore: true, - next: { createdAtMs: Number.MAX_SAFE_INTEGER, runId: id128 }, - }, - }; - }), + return { + statusCode: 200, + body: { + runs: Object.freeze(Array.from({ length: 64 }, () => item)), + hasMore: true, + next: { createdAtMs: Number.MAX_SAFE_INTEGER, runId: id128 }, + }, + }; + }), }); t.after(() => surface.stopAndDrain()); const response = await request( @@ -545,15 +587,15 @@ test('serves the reviewed worst-case 64-item RunEvent list inside the fixed resp host: '127.0.0.1', port, admission: preparedAdmission(async () => { - return { - statusCode: 200, - body: { - events: Object.freeze(Array.from({ length: 64 }, () => event)), - hasMore: true, - nextAfterSequence: event.sequence, - }, - }; - }), + return { + statusCode: 200, + body: { + events: Object.freeze(Array.from({ length: 64 }, () => event)), + hasMore: true, + nextAfterSequence: event.sequence, + }, + }; + }), }); t.after(() => surface.stopAndDrain()); const response = await request( @@ -589,15 +631,15 @@ test('serves the reviewed worst-case 64-item Run Step list inside the fixed resp host: '127.0.0.1', port, admission: preparedAdmission(async () => { - return { - statusCode: 200, - body: { - steps: Object.freeze(Array.from({ length: 64 }, () => item)), - hasMore: true, - next: { stepKey: id128, stepRunId: id128 }, - }, - }; - }), + return { + statusCode: 200, + body: { + steps: Object.freeze(Array.from({ length: 64 }, () => item)), + hasMore: true, + next: { stepKey: id128, stepRunId: id128 }, + }, + }; + }), }); t.after(() => surface.stopAndDrain()); const response = await request( @@ -621,13 +663,13 @@ test('bounds Edge admission concurrency and drains accepted work', async (t) => host: '127.0.0.1', port, admission: preparedAdmission(async (value) => { - admissions += 1; - await barrier; - return { - statusCode: 200, - body: { run: { id: value.operation.runId } }, - }; - }), + admissions += 1; + await barrier; + return { + statusCode: 200, + body: { run: { id: value.operation.runId } }, + }; + }), }); t.after(() => surface.stopAndDrain()); const accepted = Array.from({ length: 4 }, () => diff --git a/packages/ql3-local-api/test/runAttemptLogReadRoute.test.cjs b/packages/ql3-local-api/test/runAttemptLogReadRoute.test.cjs new file mode 100644 index 00000000..28923451 --- /dev/null +++ b/packages/ql3-local-api/test/runAttemptLogReadRoute.test.cjs @@ -0,0 +1,109 @@ +const assert = require('node:assert/strict'); +const { test } = require('node:test'); + +const { + RunAttemptLogReadUnavailableError, +} = require('@qinglong/runtime-core/run-attempt-log-read'); +const { + createLocalApiRunAttemptLogReadRoute, +} = require('../dist/run/runAttemptLogReadRoute.js'); + +function request(overrides = {}) { + return { + projectId: 'prj_default', + runId: 'run_123', + attemptId: 'attempt_123', + offset: 2, + length: 16, + ...overrides, + }; +} + +test('projects an available byte range as bounded base64 JSON', async () => { + const route = createLocalApiRunAttemptLogReadRoute({ + async read(value) { + assert.deepEqual(value.range, { offset: 2, length: 16 }); + return { + status: 'available', + projectId: value.projectId, + runId: value.runId, + attemptId: value.attemptId, + logArtifactId: `local-${'a'.repeat(30)}`, + content: Buffer.from('hello'), + start: 2, + endExclusive: 7, + totalBytes: 9, + nextOffset: 7, + truncation: { truncated: 'unknown' }, + }; + }, + }); + assert.deepEqual(await route.handle(request()), { + statusCode: 200, + body: { + schema: 'qinglong/run-attempt-log-read-result@v1', + status: 'available', + projectId: 'prj_default', + runId: 'run_123', + attemptId: 'attempt_123', + range: { start: 2, endExclusive: 7, totalBytes: 9, nextOffset: 7 }, + encoding: 'base64', + content: Buffer.from('hello').toString('base64'), + truncation: { truncated: 'unknown' }, + }, + }); +}); + +test('maps pending, masked absence, missing storage and unavailable evidence', async () => { + const cases = [ + [ + { + status: 'pending', + projectId: 'prj_default', + runId: 'run_123', + attemptId: 'attempt_123', + }, + { + statusCode: 202, + body: { + schema: 'qinglong/run-attempt-log-read-result@v1', + status: 'pending', + projectId: 'prj_default', + runId: 'run_123', + attemptId: 'attempt_123', + }, + }, + ], + [ + { status: 'not_found' }, + { statusCode: 404, body: { code: 'artifact_not_found' } }, + ], + [ + { + status: 'missing', + projectId: 'prj_default', + runId: 'run_123', + attemptId: 'attempt_123', + logArtifactId: `local-${'a'.repeat(30)}`, + }, + { statusCode: 503, body: { code: 'artifact_unavailable' } }, + ], + ]; + for (const [result, expected] of cases) { + const route = createLocalApiRunAttemptLogReadRoute({ + async read() { + return result; + }, + }); + assert.deepEqual(await route.handle(request()), expected); + } + const unavailable = createLocalApiRunAttemptLogReadRoute({ + async read() { + throw new RunAttemptLogReadUnavailableError(); + }, + }); + assert.deepEqual(await unavailable.handle(request()), { + statusCode: 503, + body: { code: 'artifact_unavailable' }, + }); +}); diff --git a/packages/ql3-local-api/test/sqliteHttp.integration.test.cjs b/packages/ql3-local-api/test/sqliteHttp.integration.test.cjs index 731a6310..52a44dee 100644 --- a/packages/ql3-local-api/test/sqliteHttp.integration.test.cjs +++ b/packages/ql3-local-api/test/sqliteHttp.integration.test.cjs @@ -24,6 +24,9 @@ const { const { compileLocalCommandTaskDefinition, } = require('@qinglong/runtime-core/task-definition-execution-compiler'); +const { + RunAttemptLogReadService, +} = require('@qinglong/runtime-core/run-attempt-log-read'); const { createBuiltInTaskSpecSemanticRegistry, } = require('@qinglong/runtime-core/task-spec-semantic'); @@ -34,11 +37,16 @@ const { const { createLocalApiProductSurface, } = require('../dist/application-runtime/localApiProductSurface.js'); +const { + LocalRunAttemptLogRangeReader, +} = require('../../ql3-local-execution/dist/artifact-read/localRunAttemptLogRangeReader.js'); const NOW = 1_800_000_000_000; const PEPPER_KEY_ID = 'local-api-pepper-v1'; const CREDENTIAL_ID = 'local-api-owner'; const RUN_ID = 'run_local_api_1'; +const ATTEMPT_ID = 'attempt_local_api_1'; +const LOG_ARTIFACT_ID = `local-${'a'.repeat(30)}`; const SECRET = Buffer.alloc(32, 81).toString('base64url'); const PEPPER = Buffer.alloc(32, 82).toString('base64url'); const TOKEN = formatApiCredentialToken(CREDENTIAL_ID, SECRET); @@ -206,15 +214,18 @@ function seed(databasePath, materialDigest) { enabled: true, occurredAtMs: NOW - 200, }; - const taskDefinition = createTaskDefinitionRecord({ - ...taskCommand, - spec: taskSemantics.normalize({ - projectId: taskCommand.projectId, - taskId: taskCommand.taskId, - kind: taskCommand.kind, - spec: taskCommand.spec, - }), - }, NOW - 200); + const taskDefinition = createTaskDefinitionRecord( + { + ...taskCommand, + spec: taskSemantics.normalize({ + projectId: taskCommand.projectId, + taskId: taskCommand.taskId, + kind: taskCommand.kind, + spec: taskCommand.spec, + }), + }, + NOW - 200, + ); const taskExecution = compileLocalCommandTaskDefinition( taskDefinition, taskSemantics, @@ -298,6 +309,15 @@ function seed(databasePath, materialDigest) { 'manual', 'runtime', 'running', 1, 1, 0, ?)`, ) .run(RUN_ID, NOW - 100); + client + .prepare( + `INSERT INTO "RunAttempts" ( + "id", "run_id", "attempt", "status", "executor_type", + "log_artifact_id", "callback_sequence", "created_at_ms", + "started_at_ms" + ) VALUES (?, ?, 1, 'running', 'local_process', ?, 0, ?, ?)`, + ) + .run(ATTEMPT_ID, RUN_ID, LOG_ARTIFACT_ID, NOW - 90, NOW - 80); client .prepare( `INSERT INTO "StepRuns" ( @@ -357,6 +377,15 @@ test('serves an authenticated Run through one real SQLite authority and durable fs.chmodSync(root, 0o700); t.after(() => fs.rmSync(root, { recursive: true, force: true })); const databasePath = path.join(root, 'qinglong3.sqlite'); + const artifactRoot = path.join(root, 'artifacts'); + const artifactShard = path.join(artifactRoot, 'aa'); + fs.mkdirSync(artifactShard, { recursive: true, mode: 0o700 }); + fs.chmodSync(artifactRoot, 0o700); + fs.chmodSync(artifactShard, 0o700); + const logContent = Buffer.from('local-api-log-line\n', 'utf8'); + const logPath = path.join(artifactShard, `${LOG_ARTIFACT_ID}.log`); + fs.writeFileSync(logPath, logContent, { mode: 0o600 }); + fs.chmodSync(logPath, 0o600); const keyringDirectory = path.join(root, 'owner-pepper'); fs.mkdirSync(keyringDirectory, { mode: 0o700 }); const summary = provisionLocalOwnerPepperKey({ @@ -398,6 +427,15 @@ test('serves an authenticated Run through one real SQLite authority and durable stepRuns: await runtime.stepRunReader(), runCancellation: await runtime.runCancellationRepository(), taskStart: await runtime.taskStartRepository(), + runAttemptLogRead: new RunAttemptLogReadService( + runtime.runRepository, + new LocalRunAttemptLogRangeReader(artifactRoot), + { + executorType: 'local_process', + artifactIdPattern: /^local-[a-f0-9]{30}$/, + maximumReadBytes: 32 * 1024, + }, + ), taskDefinitions: runtime.taskDefinitions, apiCredentials: runtime.apiCredentials, ownerPepper: runtime.ownerPepper, @@ -572,12 +610,29 @@ test('serves an authenticated Run through one real SQLite authority and durable }); assert.equal(JSON.stringify(steps).includes('private'), false); + const log = await request( + port, + `Bearer ${TOKEN}`, + `/api/v3/projects/default/runs/${RUN_ID}/attempts/${ATTEMPT_ID}/log?offset=0&length=8`, + ); + assert.equal(log.statusCode, 200); + assert.equal(log.body.schema, 'qinglong/run-attempt-log-read-result@v1'); + assert.equal(log.body.status, 'available'); + assert.equal(log.body.encoding, 'base64'); + assert.equal(Buffer.from(log.body.content, 'base64').toString(), 'local-ap'); + assert.deepEqual(log.body.range, { + start: 0, + endExclusive: 8, + totalBytes: logContent.byteLength, + nextOffset: 8, + }); + assert.deepEqual(log.body.truncation, { truncated: 'unknown' }); + const cancellationBody = JSON.stringify({ schema: 'qinglong/run-cancellation@v1', mutationId: 'cancel-local-api-1', }); - const cancellationPath = - `/api/v3/projects/default/runs/${RUN_ID}/cancellation`; + const cancellationPath = `/api/v3/projects/default/runs/${RUN_ID}/cancellation`; const cancellationOptions = { method: 'POST', headers: { @@ -623,7 +678,7 @@ test('serves an authenticated Run through one real SQLite authority and durable WHERE operation_id IN ( 'run.get', 'run.list', 'run.events.list', 'run.steps.list', 'run.cancel', 'task.get', 'task.list' - , 'task.start' + , 'task.start', 'run.log.read' ) ORDER BY operation_id, outcome`, ) @@ -636,6 +691,7 @@ test('serves an authenticated Run through one real SQLite authority and durable 'run.get:allowed', 'run.get:authentication_rejected', 'run.list:allowed', + 'run.log.read:allowed', 'run.steps.list:allowed', 'task.get:allowed', 'task.get:allowed', diff --git a/packages/ql3-local-application/src/application-runtime/activation.ts b/packages/ql3-local-application/src/application-runtime/activation.ts index e1bc866c..a4e0157c 100644 --- a/packages/ql3-local-application/src/application-runtime/activation.ts +++ b/packages/ql3-local-application/src/application-runtime/activation.ts @@ -455,11 +455,28 @@ export async function bootstrapLocalApplication( }); if (options.productSurface) { - const [stepRuns, runCancellation, taskStart] = await Promise.all([ + const [ + stepRuns, + runCancellation, + taskStart, + { LocalRunAttemptLogRangeReader }, + { RunAttemptLogReadService }, + ] = await Promise.all([ storage.stepRunReader(), storage.runCancellationRepository(), storage.taskStartRepository(), + import('@qinglong/local-execution/artifact-read'), + import('@qinglong/runtime-core/run-attempt-log-read'), ]); + const runAttemptLogRead = new RunAttemptLogReadService( + storage.runs, + new LocalRunAttemptLogRangeReader(options.artifactRoot), + { + executorType: 'local_process', + artifactIdPattern: /^local-[a-f0-9]{30}$/, + maximumReadBytes: 32 * 1024, + }, + ); productSurfaceLifecycle = await options.productSurface.start( Object.freeze({ profile: options.profile, @@ -467,6 +484,7 @@ export async function bootstrapLocalApplication( stepRuns, runCancellation, taskStart, + runAttemptLogRead, taskDefinitions: storage.taskDefinitions, apiCredentials: storage.apiCredentials, ownerPepper: storage.ownerPepper, diff --git a/packages/ql3-local-application/src/application-runtime/contract.ts b/packages/ql3-local-application/src/application-runtime/contract.ts index acd7189e..2f43ac1b 100644 --- a/packages/ql3-local-application/src/application-runtime/contract.ts +++ b/packages/ql3-local-application/src/application-runtime/contract.ts @@ -18,6 +18,10 @@ import type { PluginPackageRecoveryCycleResult } from '@qinglong/runtime-core/pl import type { PluginPackageAutomationPublicationRecoveryCycleResult } from '@qinglong/runtime-core/plugin-package-automation-publication'; import type { PluginPackageTaskPublicationRecoveryCycleResult } from '@qinglong/runtime-core/plugin-package-task-publication'; import type { ProjectToolDefinitionSnapshotRecoveryCycleResult } from '@qinglong/runtime-core/project-tool-definition-snapshot'; +import type { + RunAttemptLogReadRequest, + RunAttemptLogReadResult, +} from '@qinglong/runtime-core/run-attempt-log-read'; export type LocalApplicationProfile = 'edge' | 'standalone'; @@ -59,6 +63,11 @@ export interface LocalApplicationProductSurfaceAuthority { ReadyFreshStorage['taskDefinitions'], 'findCurrentTaskDefinition' | 'listTaskDefinitions' >; + readonly runAttemptLogRead: Readonly<{ + read( + request: Readonly, + ): Promise; + }>; readonly apiCredentials: ReadyFreshStorage['apiCredentials']; readonly ownerPepper: ReadyFreshStorage['ownerPepper']; readonly projectPolicy: ReadyFreshStorage['projectPolicy']; diff --git a/packages/ql3-local-application/test/activation.test.cjs b/packages/ql3-local-application/test/activation.test.cjs index fd1e16c2..9eb69afb 100644 --- a/packages/ql3-local-application/test/activation.test.cjs +++ b/packages/ql3-local-application/test/activation.test.cjs @@ -88,8 +88,7 @@ const RECEIPT_TOKEN = 'A'.repeat(32); const CLEANUP_RUN_ID = '019f70c0-0000-7000-8000-000000000011'; const CLEANUP_ATTEMPT_ID = '019f70c0-0000-7000-8000-000000000012'; const WORKFLOW_CANCELLATION_CREDENTIAL_ID = 'application-workflow-owner'; -const WORKFLOW_CANCELLATION_PEPPER_KEY_ID = - 'application-workflow-owner-v1'; +const WORKFLOW_CANCELLATION_PEPPER_KEY_ID = 'application-workflow-owner-v1'; const WORKFLOW_CANCELLATION_PEPPER_BYTES = Buffer.alloc(32, 141); const WORKFLOW_CANCELLATION_SECRET = Buffer.alloc(32, 142).toString( 'base64url', @@ -322,8 +321,7 @@ function promptResourceConfiguration() { const prefix = 'private router model output:'; const minimumOutputBytes = Buffer.byteLength(prefix, 'utf8'); const rawOutputBytes = - process.env.QL3_PROMPT_RESOURCE_OUTPUT_BYTES ?? - String(minimumOutputBytes); + process.env.QL3_PROMPT_RESOURCE_OUTPUT_BYTES ?? String(minimumOutputBytes); const outputBytes = Number(rawOutputBytes); if ( !Number.isSafeInteger(outputBytes) || @@ -338,8 +336,7 @@ function promptResourceConfiguration() { profile, output: prefix + 'x'.repeat(outputBytes - Buffer.byteLength(prefix)), outputBytes, - resourceProbe: - process.env.QL3_PROMPT_RESOURCE_OUTPUT_BYTES !== undefined, + resourceProbe: process.env.QL3_PROMPT_RESOURCE_OUTPUT_BYTES !== undefined, }); } @@ -1263,7 +1260,10 @@ test('executes one active Package Prompt through local AI composition with conte promptOutputKeys: { async active() { metrics.keyLoads += 1; - return { keyId: 'edge-prompt-output-key-1', key: Buffer.alloc(32, 7) }; + return { + keyId: 'edge-prompt-output-key-1', + key: Buffer.alloc(32, 7), + }; }, async resolve(keyId) { metrics.keyResolutions += 1; @@ -1274,7 +1274,10 @@ test('executes one active Package Prompt through local AI composition with conte promptOutputRead: { authorizer: { async authorize(request) { - assert.equal(request.projectId, 'project-application-prompt-resource'); + assert.equal( + request.projectId, + 'project-application-prompt-resource', + ); assert.equal(request.principal.subject.id, 'edge-resource-owner'); return { effect: 'allow' }; }, @@ -1376,7 +1379,10 @@ test('executes one active Package Prompt through local AI composition with conte assert.equal(recoveredByRequest.result.text, metrics.privateOutput); assert.deepEqual(recoveredByRequest.reference, durableFirst.outputArtifact); assert.equal(metrics.keyResolutions, 2); - assert.equal(metrics.lastResolvedKey.every((byte) => byte === 0), true); + assert.equal( + metrics.lastResolvedKey.every((byte) => byte === 0), + true, + ); const durableStorageAfter = sqliteStorageSnapshot(value.targetPath); const durableStorageGrowth = sqliteStorageGrowth( durableStorageBefore, @@ -1450,9 +1456,8 @@ test('executes one active Package Prompt through local AI composition with conte ) .get(durableFirst.admission.invocationId), }, - integrityCheck: reader - .prepare('PRAGMA integrity_check') - .get().integrity_check, + integrityCheck: reader.prepare('PRAGMA integrity_check').get() + .integrity_check, }; } finally { reader.close(); @@ -1532,8 +1537,7 @@ test('executes one active Package Prompt through local AI composition with conte databaseLogicalWriteAmplificationPermille, databaseAllocatedWriteAmplificationPermille, walWriteAmplificationPermille, - journalMode: - resource.profile === 'edge' ? 'delete' : 'wal', + journalMode: resource.profile === 'edge' ? 'delete' : 'wal', providerCalls: metrics.providerCalls, keyLoads: metrics.keyLoads, keyResolutions: metrics.keyResolutions, @@ -1700,6 +1704,7 @@ test('starts an optional product surface after recovery and drains it before own typeof authority.taskDefinitions.listTaskDefinitions, 'function', ); + assert.equal(typeof authority.runAttemptLogRead.read, 'function'); assert.equal(typeof authority.apiCredentials.resolve, 'function'); assert.equal(typeof authority.ownerPepper.resolveKey, 'function'); assert.equal(typeof authority.projectPolicy.resolve, 'function'); diff --git a/packages/ql3-local-execution/package.json b/packages/ql3-local-execution/package.json index f40d3622..d1ff9a56 100644 --- a/packages/ql3-local-execution/package.json +++ b/packages/ql3-local-execution/package.json @@ -28,6 +28,11 @@ "require": "./dist/dispatch/index.js", "default": "./dist/dispatch/index.js" }, + "./artifact-read": { + "types": "./dist/artifact-read/index.d.ts", + "require": "./dist/artifact-read/index.js", + "default": "./dist/artifact-read/index.js" + }, "./scheduler": { "types": "./dist/scheduler/index.d.ts", "require": "./dist/scheduler/index.js", diff --git a/packages/ql3-local-execution/src/artifact-read/index.ts b/packages/ql3-local-execution/src/artifact-read/index.ts new file mode 100644 index 00000000..6cb8526a --- /dev/null +++ b/packages/ql3-local-execution/src/artifact-read/index.ts @@ -0,0 +1 @@ +export * from './localRunAttemptLogRangeReader'; diff --git a/packages/ql3-local-execution/src/artifact-read/localRunAttemptLogRangeReader.ts b/packages/ql3-local-execution/src/artifact-read/localRunAttemptLogRangeReader.ts new file mode 100644 index 00000000..6206df64 --- /dev/null +++ b/packages/ql3-local-execution/src/artifact-read/localRunAttemptLogRangeReader.ts @@ -0,0 +1,285 @@ +import { constants, type Stats } from 'node:fs'; +import fs, { type FileHandle } from 'node:fs/promises'; +import path from 'node:path'; + +import { + normalizeRunAttemptLogReadRange, + type RunAttemptLogRangeReader, + type RunAttemptLogRangeReadResult, + type RunAttemptLogReadIdentity, + type RunAttemptLogReadRange, + type RunAttemptLogTruncationView, +} from '@qinglong/runtime-core/run-attempt-log-read'; + +const LOCAL_ARTIFACT_ID = /^local-[a-f0-9]{30}$/; +const MAXIMUM_ARTIFACT_BYTES = 1024 * 1024 * 1024; +const MAXIMUM_FACT_BYTES = 1024; + +export class LocalRunAttemptLogRangeReadError extends Error { + constructor( + readonly reason: + | 'invalid_configuration' + | 'unsafe_path' + | 'integrity_mismatch', + options?: ErrorOptions, + ) { + super(`Local Run Attempt log range read failed: ${reason}`, options); + this.name = 'LocalRunAttemptLogRangeReadError'; + } +} + +function isCode(error: unknown, code: string): boolean { + return ( + !!error && + typeof error === 'object' && + 'code' in error && + (error as { code?: unknown }).code === code + ); +} + +function currentUid(): number | undefined { + return typeof process.getuid === 'function' ? process.getuid() : undefined; +} + +function root(value: string): string { + if ( + typeof value !== 'string' || + !path.isAbsolute(value) || + path.parse(value).root === value || + value.includes('\0') || + Buffer.byteLength(value, 'utf8') > 4096 + ) { + throw new LocalRunAttemptLogRangeReadError('invalid_configuration'); + } + return path.resolve(value); +} + +function identity( + value: Readonly, +): Readonly { + if ( + !value || + typeof value !== 'object' || + Array.isArray(value) || + !LOCAL_ARTIFACT_ID.test(value.logArtifactId) + ) { + throw new LocalRunAttemptLogRangeReadError('integrity_mismatch'); + } + return value; +} + +function assertOwnedDirectory(stat: Stats): void { + const uid = currentUid(); + if ( + !stat.isDirectory() || + stat.isSymbolicLink() || + (stat.mode & 0o777) !== 0o700 || + (uid !== undefined && stat.uid !== uid) + ) { + throw new LocalRunAttemptLogRangeReadError('unsafe_path'); + } +} + +function assertOwnedFile(stat: Stats): void { + const uid = currentUid(); + if ( + !stat.isFile() || + stat.nlink !== 1 || + (stat.mode & 0o777) !== 0o600 || + (uid !== undefined && stat.uid !== uid) || + !Number.isSafeInteger(stat.size) || + stat.size < 0 || + stat.size > MAXIMUM_ARTIFACT_BYTES + ) { + throw new LocalRunAttemptLogRangeReadError('unsafe_path'); + } +} + +async function optionalPrivateDirectory(directory: string): Promise { + try { + assertOwnedDirectory(await fs.lstat(directory)); + return true; + } catch (error) { + if (isCode(error, 'ENOENT')) return false; + if (error instanceof LocalRunAttemptLogRangeReadError) throw error; + throw new LocalRunAttemptLogRangeReadError('unsafe_path', { cause: error }); + } +} + +async function openPrivateFile( + filePath: string, +): Promise { + try { + return await fs.open( + filePath, + constants.O_RDONLY | (constants.O_NOFOLLOW ?? 0), + ); + } catch (error) { + if (isCode(error, 'ENOENT')) return undefined; + throw new LocalRunAttemptLogRangeReadError('unsafe_path', { cause: error }); + } +} + +function exactFact( + value: unknown, + expected: Readonly, +): Readonly { + if (!value || typeof value !== 'object' || Array.isArray(value)) { + throw new LocalRunAttemptLogRangeReadError('integrity_mismatch'); + } + const fact = value as Record; + const keys = Object.keys(fact).sort(); + if ( + keys.join(',') !== + 'attemptId,logArtifactId,maximumBytes,observedAtMs,quotaReached,runId,schemaVersion' || + fact.schemaVersion !== 1 || + fact.runId !== expected.runId || + fact.attemptId !== expected.attemptId || + fact.logArtifactId !== expected.logArtifactId || + !Number.isSafeInteger(fact.maximumBytes) || + Number(fact.maximumBytes) < 64 * 1024 || + Number(fact.maximumBytes) > MAXIMUM_ARTIFACT_BYTES || + typeof fact.quotaReached !== 'boolean' || + !Number.isSafeInteger(fact.observedAtMs) || + Number(fact.observedAtMs) < 0 + ) { + throw new LocalRunAttemptLogRangeReadError('integrity_mismatch'); + } + return Object.freeze({ + truncated: fact.quotaReached, + maximumBytes: fact.maximumBytes as number, + observedAtMs: fact.observedAtMs as number, + }); +} + +async function readTruncationFact( + directory: string, + expected: Readonly, + signal?: AbortSignal, +): Promise> { + if (signal?.aborted) throw signal.reason; + const factPath = path.join( + directory, + `.${expected.logArtifactId}.log.truncated.json`, + ); + const handle = await openPrivateFile(factPath); + if (!handle) return Object.freeze({ truncated: 'unknown' as const }); + try { + const before = await handle.stat(); + assertOwnedFile(before); + if (before.size < 2 || before.size > MAXIMUM_FACT_BYTES) { + throw new LocalRunAttemptLogRangeReadError('integrity_mismatch'); + } + const content = Buffer.allocUnsafe(before.size); + let read = 0; + while (read < content.byteLength) { + if (signal?.aborted) throw signal.reason; + const result = await handle.read( + content, + read, + content.byteLength - read, + read, + ); + if (result.bytesRead < 1) { + throw new LocalRunAttemptLogRangeReadError('integrity_mismatch'); + } + read += result.bytesRead; + } + const after = await handle.stat(); + assertOwnedFile(after); + if ( + after.dev !== before.dev || + after.ino !== before.ino || + after.size !== before.size + ) { + throw new LocalRunAttemptLogRangeReadError('integrity_mismatch'); + } + try { + const text = new TextDecoder('utf-8', { fatal: true }).decode(content); + return exactFact(JSON.parse(text), expected); + } catch (error) { + if (error instanceof LocalRunAttemptLogRangeReadError) throw error; + throw new LocalRunAttemptLogRangeReadError('integrity_mismatch', { + cause: error, + }); + } finally { + content.fill(0); + } + } finally { + await handle.close().catch(() => undefined); + } +} + +export class LocalRunAttemptLogRangeReader implements RunAttemptLogRangeReader { + private readonly root: string; + + constructor(artifactRoot: string) { + this.root = root(artifactRoot); + } + + async read( + rawIdentity: Readonly, + rawRange: Readonly, + signal?: AbortSignal, + ): Promise { + const expected = identity(rawIdentity); + const range = normalizeRunAttemptLogReadRange(rawRange); + if (signal?.aborted) throw signal.reason; + if (!(await optionalPrivateDirectory(this.root))) { + return Object.freeze({ status: 'missing' as const }); + } + const directory = path.join( + this.root, + expected.logArtifactId.slice('local-'.length, 'local-'.length + 2), + ); + if (!(await optionalPrivateDirectory(directory))) { + return Object.freeze({ status: 'missing' as const }); + } + const target = path.join(directory, `${expected.logArtifactId}.log`); + const handle = await openPrivateFile(target); + if (!handle) return Object.freeze({ status: 'missing' as const }); + try { + const before = await handle.stat(); + assertOwnedFile(before); + const start = Math.min(range.offset, before.size); + const expectedBytes = Math.min(range.length, before.size - start); + const content = Buffer.allocUnsafe(expectedBytes); + let read = 0; + while (read < expectedBytes) { + if (signal?.aborted) throw signal.reason; + const result = await handle.read( + content, + read, + expectedBytes - read, + start + read, + ); + if (result.bytesRead < 1) { + throw new LocalRunAttemptLogRangeReadError('integrity_mismatch'); + } + read += result.bytesRead; + } + const after = await handle.stat(); + assertOwnedFile(after); + if ( + after.dev !== before.dev || + after.ino !== before.ino || + after.size < before.size + ) { + throw new LocalRunAttemptLogRangeReadError('integrity_mismatch'); + } + const endExclusive = start + content.byteLength; + const truncation = await readTruncationFact(directory, expected, signal); + return Object.freeze({ + status: 'available' as const, + content, + start, + endExclusive, + totalBytes: before.size, + ...(endExclusive < before.size ? { nextOffset: endExclusive } : {}), + truncation, + }); + } finally { + await handle.close().catch(() => undefined); + } + } +} diff --git a/packages/ql3-local-execution/test/localRunAttemptLogRangeReader.test.cjs b/packages/ql3-local-execution/test/localRunAttemptLogRangeReader.test.cjs new file mode 100644 index 00000000..3ad4b9e5 --- /dev/null +++ b/packages/ql3-local-execution/test/localRunAttemptLogRangeReader.test.cjs @@ -0,0 +1,158 @@ +const assert = require('node:assert/strict'); +const fs = require('node:fs/promises'); +const os = require('node:os'); +const path = require('node:path'); +const { test } = require('node:test'); + +const { + LocalRunAttemptLogRangeReadError, + LocalRunAttemptLogRangeReader, +} = require('../dist/artifact-read/localRunAttemptLogRangeReader.js'); + +const artifactId = `local-${'a'.repeat(30)}`; +const identity = Object.freeze({ + projectId: 'prj_default', + runId: 'run_123', + attemptId: 'attempt_123', + logArtifactId: artifactId, +}); + +async function fixture(t, content = Buffer.from('0123456789')) { + const parent = await fs.mkdtemp(path.join(os.tmpdir(), 'ql3-log-read-')); + t.after(() => fs.rm(parent, { recursive: true, force: true })); + const root = path.join(parent, 'artifacts'); + const shard = path.join(root, 'aa'); + await fs.mkdir(shard, { recursive: true, mode: 0o700 }); + await fs.chmod(root, 0o700); + await fs.chmod(shard, 0o700); + const log = path.join(shard, `${artifactId}.log`); + await fs.writeFile(log, content, { mode: 0o600 }); + await fs.chmod(log, 0o600); + return { parent, root, shard, log }; +} + +async function fact(shard, overrides = {}) { + const file = path.join(shard, `.${artifactId}.log.truncated.json`); + await fs.writeFile( + file, + JSON.stringify({ + schemaVersion: 1, + runId: identity.runId, + attemptId: identity.attemptId, + logArtifactId: identity.logArtifactId, + maximumBytes: 64 * 1024, + quotaReached: false, + observedAtMs: 9, + ...overrides, + }), + { mode: 0o600 }, + ); + await fs.chmod(file, 0o600); + return file; +} + +test('reads one bounded private-file snapshot and canonical truncation fact', async (t) => { + const value = await fixture(t); + await fact(value.shard, { quotaReached: true }); + const result = await new LocalRunAttemptLogRangeReader(value.root).read( + identity, + { offset: 2, length: 4 }, + ); + assert.equal(result.status, 'available'); + assert.equal(Buffer.from(result.content).toString(), '2345'); + assert.deepEqual( + { + start: result.start, + endExclusive: result.endExclusive, + totalBytes: result.totalBytes, + nextOffset: result.nextOffset, + truncation: result.truncation, + }, + { + start: 2, + endExclusive: 6, + totalBytes: 10, + nextOffset: 6, + truncation: { + truncated: true, + maximumBytes: 64 * 1024, + observedAtMs: 9, + }, + }, + ); +}); + +test('returns unknown truncation and a stable empty range beyond the snapshot', async (t) => { + const value = await fixture(t); + const result = await new LocalRunAttemptLogRangeReader(value.root).read( + identity, + { offset: 99, length: 4 }, + ); + assert.equal(result.status, 'available'); + assert.equal(result.content.byteLength, 0); + assert.equal(result.start, 10); + assert.equal(result.endExclusive, 10); + assert.equal(result.totalBytes, 10); + assert.equal(result.nextOffset, undefined); + assert.deepEqual(result.truncation, { truncated: 'unknown' }); +}); + +test('treats absent root, shard and log as missing', async (t) => { + const parent = await fs.mkdtemp(path.join(os.tmpdir(), 'ql3-log-missing-')); + t.after(() => fs.rm(parent, { recursive: true, force: true })); + const root = path.join(parent, 'artifacts'); + const reader = new LocalRunAttemptLogRangeReader(root); + assert.deepEqual(await reader.read(identity, { offset: 0, length: 1 }), { + status: 'missing', + }); + await fs.mkdir(root, { mode: 0o700 }); + assert.deepEqual(await reader.read(identity, { offset: 0, length: 1 }), { + status: 'missing', + }); + const shard = path.join(root, 'aa'); + await fs.mkdir(shard, { mode: 0o700 }); + assert.deepEqual(await reader.read(identity, { offset: 0, length: 1 }), { + status: 'missing', + }); +}); + +test('fails closed for symlink targets and widened file permissions', async (t) => { + const symlink = await fixture(t); + await fs.rm(symlink.log); + await fs.symlink(path.join(symlink.parent, 'outside'), symlink.log); + await assert.rejects( + new LocalRunAttemptLogRangeReader(symlink.root).read(identity, { + offset: 0, + length: 1, + }), + LocalRunAttemptLogRangeReadError, + ); + + const widened = await fixture(t); + await fs.chmod(widened.log, 0o644); + await assert.rejects( + new LocalRunAttemptLogRangeReader(widened.root).read(identity, { + offset: 0, + length: 1, + }), + LocalRunAttemptLogRangeReadError, + ); +}); + +test('fails closed for truncation identity drift and an aborted request', async (t) => { + const value = await fixture(t); + await fact(value.shard, { attemptId: 'attempt_other' }); + const reader = new LocalRunAttemptLogRangeReader(value.root); + await assert.rejects( + reader.read(identity, { offset: 0, length: 1 }), + (error) => + error instanceof LocalRunAttemptLogRangeReadError && + error.reason === 'integrity_mismatch', + ); + const abort = new AbortController(); + abort.abort(new Error('cancelled')); + await assert.rejects( + reader.read(identity, { offset: 0, length: 1 }, abort.signal), + /cancelled/, + ); +}); diff --git a/packages/ql3-runtime-core/package.json b/packages/ql3-runtime-core/package.json index 55bee8ce..f5cf5772 100644 --- a/packages/ql3-runtime-core/package.json +++ b/packages/ql3-runtime-core/package.json @@ -232,6 +232,9 @@ ], "bounded-run-step-list-projection": [ "dist/run/projection/boundedRunStepListProjection.d.ts" + ], + "run-attempt-log-read": [ + "dist/run/log-read/runAttemptLogRead.d.ts" ] } }, @@ -281,6 +284,11 @@ "require": "./dist/run/projection/boundedRunStepListProjection.js", "default": "./dist/run/projection/boundedRunStepListProjection.js" }, + "./run-attempt-log-read": { + "types": "./dist/run/log-read/runAttemptLogRead.d.ts", + "require": "./dist/run/log-read/runAttemptLogRead.js", + "default": "./dist/run/log-read/runAttemptLogRead.js" + }, "./task-definition": { "types": "./dist/task-definition/taskDefinition.d.ts", "require": "./dist/task-definition/taskDefinition.js", diff --git a/packages/ql3-runtime-core/src/run/log-read/runAttemptLogRead.ts b/packages/ql3-runtime-core/src/run/log-read/runAttemptLogRead.ts new file mode 100644 index 00000000..11bef44c --- /dev/null +++ b/packages/ql3-runtime-core/src/run/log-read/runAttemptLogRead.ts @@ -0,0 +1,343 @@ +import { RUN_ATTEMPT_STATUSES, type RunAttemptStatus } from '../run'; +import type { RunRepositoryReader } from '../runRepository'; + +export const MAX_RUN_ATTEMPT_LOG_READ_BYTES = 256 * 1024; + +const ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/; +const TERMINAL_ATTEMPT_STATUSES = new Set([ + 'succeeded', + 'failed', + 'cancelled', + 'timed_out', + 'lost', +]); + +export interface RunAttemptLogReadRange { + readonly offset: number; + readonly length: number; +} + +export interface RunAttemptLogReadIdentity { + readonly projectId: string; + readonly runId: string; + readonly attemptId: string; + readonly logArtifactId: string; +} + +export interface RunAttemptLogTruncationView { + readonly truncated: boolean | 'unknown'; + readonly maximumBytes?: number; + readonly observedAtMs?: number; +} + +export type RunAttemptLogRangeReadResult = + | Readonly<{ readonly status: 'missing' }> + | Readonly<{ + readonly status: 'available'; + readonly content: Uint8Array; + readonly start: number; + readonly endExclusive: number; + readonly totalBytes: number; + readonly nextOffset?: number; + readonly truncation: Readonly; + }>; + +export interface RunAttemptLogRangeReader { + read( + identity: Readonly, + range: Readonly, + signal?: AbortSignal, + ): Promise; +} + +export interface RunAttemptLogReadRequest { + readonly projectId: string; + readonly runId: string; + readonly attemptId: string; + readonly range: Readonly; + readonly signal?: AbortSignal; +} + +export type RunAttemptLogReadResult = + | Readonly<{ readonly status: 'not_found' }> + | Readonly<{ + readonly status: 'pending'; + readonly projectId: string; + readonly runId: string; + readonly attemptId: string; + readonly logArtifactId?: string; + }> + | (Readonly & + Readonly<{ readonly status: 'missing' }>) + | (Readonly & + Extract); + +export interface RunAttemptLogReadServiceOptions { + readonly executorType: 'local_process' | 'remote_worker'; + readonly artifactIdPattern: RegExp; + readonly maximumReadBytes: number; + readonly activeMissingIsPending?: boolean; +} + +export class InvalidRunAttemptLogReadError extends TypeError { + constructor(message: string) { + super(`Run Attempt log read is invalid: ${message}`); + this.name = 'InvalidRunAttemptLogReadError'; + } +} + +export class RunAttemptLogReadUnavailableError extends Error { + constructor(options?: ErrorOptions) { + super('Run Attempt log read is unavailable', options); + this.name = 'RunAttemptLogReadUnavailableError'; + } +} + +function canonicalId(name: string, value: unknown): string { + if (typeof value !== 'string' || !ID_PATTERN.test(value)) { + throw new InvalidRunAttemptLogReadError(`${name} is invalid`); + } + return value; +} + +function exactKeys( + value: object, + required: readonly string[], + optional: readonly string[], + name: string, +): void { + const keys = Object.keys(value); + const allowed = new Set([...required, ...optional]); + if ( + required.some((key) => !Object.hasOwn(value, key)) || + keys.some((key) => !allowed.has(key)) + ) { + throw new InvalidRunAttemptLogReadError(`${name} shape is invalid`); + } +} + +export function normalizeRunAttemptLogReadRange( + value: Readonly, + maximumReadBytes = MAX_RUN_ATTEMPT_LOG_READ_BYTES, +): Readonly { + if ( + !Number.isSafeInteger(maximumReadBytes) || + maximumReadBytes < 1 || + maximumReadBytes > MAX_RUN_ATTEMPT_LOG_READ_BYTES + ) { + throw new InvalidRunAttemptLogReadError('maximum read bytes is invalid'); + } + if (!value || typeof value !== 'object' || Array.isArray(value)) { + throw new InvalidRunAttemptLogReadError('range is invalid'); + } + exactKeys(value, ['length', 'offset'], [], 'range'); + if (!Number.isSafeInteger(value.offset) || value.offset < 0) { + throw new InvalidRunAttemptLogReadError('offset is invalid'); + } + if ( + !Number.isSafeInteger(value.length) || + value.length < 1 || + value.length > maximumReadBytes + ) { + throw new InvalidRunAttemptLogReadError('length is invalid'); + } + return Object.freeze({ offset: value.offset, length: value.length }); +} + +function prepareOptions( + options: RunAttemptLogReadServiceOptions, +): Readonly { + if ( + !options || + typeof options !== 'object' || + Array.isArray(options) || + (options.executorType !== 'local_process' && + options.executorType !== 'remote_worker') || + !(options.artifactIdPattern instanceof RegExp) || + options.artifactIdPattern.global || + options.artifactIdPattern.sticky || + (options.activeMissingIsPending !== undefined && + typeof options.activeMissingIsPending !== 'boolean') + ) { + throw new InvalidRunAttemptLogReadError('service options are invalid'); + } + exactKeys( + options, + ['artifactIdPattern', 'executorType', 'maximumReadBytes'], + ['activeMissingIsPending'], + 'service options', + ); + normalizeRunAttemptLogReadRange( + { offset: 0, length: options.maximumReadBytes }, + options.maximumReadBytes, + ); + return Object.freeze({ ...options }); +} + +function truncation( + value: Readonly, +): Readonly { + if ( + !value || + typeof value !== 'object' || + Array.isArray(value) || + (value.truncated !== true && + value.truncated !== false && + value.truncated !== 'unknown') || + (value.maximumBytes !== undefined && + (!Number.isSafeInteger(value.maximumBytes) || value.maximumBytes < 1)) || + (value.observedAtMs !== undefined && + (!Number.isSafeInteger(value.observedAtMs) || value.observedAtMs < 0)) || + (value.truncated === 'unknown' && + (value.maximumBytes !== undefined || value.observedAtMs !== undefined)) + ) { + throw new RunAttemptLogReadUnavailableError(); + } + return Object.freeze({ ...value }); +} + +function available( + identity: Readonly, + range: Readonly, + result: Extract< + RunAttemptLogRangeReadResult, + { readonly status: 'available' } + >, +): RunAttemptLogReadResult { + if ( + !(result.content instanceof Uint8Array) || + !Number.isSafeInteger(result.start) || + !Number.isSafeInteger(result.endExclusive) || + !Number.isSafeInteger(result.totalBytes) || + result.start !== Math.min(range.offset, result.totalBytes) || + result.endExclusive !== result.start + result.content.byteLength || + result.endExclusive > result.totalBytes || + result.content.byteLength > range.length || + (result.nextOffset === undefined) !== + (result.endExclusive === result.totalBytes) || + (result.nextOffset !== undefined && + result.nextOffset !== result.endExclusive) + ) { + throw new RunAttemptLogReadUnavailableError(); + } + return Object.freeze({ + status: 'available' as const, + ...identity, + content: result.content, + start: result.start, + endExclusive: result.endExclusive, + totalBytes: result.totalBytes, + ...(result.nextOffset === undefined + ? {} + : { nextOffset: result.nextOffset }), + truncation: truncation(result.truncation), + }); +} + +export class RunAttemptLogReadService { + private readonly options: Readonly; + + constructor( + private readonly runs: Pick< + RunRepositoryReader, + 'findRunById' | 'findAttemptById' + >, + private readonly reader: RunAttemptLogRangeReader, + options: RunAttemptLogReadServiceOptions, + ) { + if ( + !runs || + typeof runs.findRunById !== 'function' || + typeof runs.findAttemptById !== 'function' || + !reader || + typeof reader.read !== 'function' + ) { + throw new InvalidRunAttemptLogReadError('dependencies are invalid'); + } + this.options = prepareOptions(options); + } + + async read( + request: Readonly, + ): Promise { + if (!request || typeof request !== 'object' || Array.isArray(request)) { + throw new InvalidRunAttemptLogReadError('request is invalid'); + } + exactKeys( + request, + ['attemptId', 'projectId', 'range', 'runId'], + ['signal'], + 'request', + ); + const projectId = canonicalId('projectId', request.projectId); + const runId = canonicalId('runId', request.runId); + const attemptId = canonicalId('attemptId', request.attemptId); + const range = normalizeRunAttemptLogReadRange( + request.range, + this.options.maximumReadBytes, + ); + if (request.signal?.aborted) { + throw new RunAttemptLogReadUnavailableError({ + cause: request.signal.reason, + }); + } + + try { + const run = await this.runs.findRunById(runId); + if ( + !run || + run.id !== runId || + run.projectId !== projectId || + run.executionOwner !== 'runtime' + ) { + return Object.freeze({ status: 'not_found' as const }); + } + const attempt = await this.runs.findAttemptById(attemptId); + if ( + !attempt || + attempt.id !== attemptId || + attempt.runId !== runId || + attempt.executorType !== this.options.executorType || + !RUN_ATTEMPT_STATUSES.includes(attempt.status) + ) { + return Object.freeze({ status: 'not_found' as const }); + } + if (attempt.logArtifactId === undefined) { + return Object.freeze({ + status: 'pending' as const, + projectId, + runId, + attemptId, + }); + } + if (!this.options.artifactIdPattern.test(attempt.logArtifactId)) { + return Object.freeze({ status: 'not_found' as const }); + } + const identity = Object.freeze({ + projectId, + runId, + attemptId, + logArtifactId: attempt.logArtifactId, + }); + const result = await this.reader.read(identity, range, request.signal); + if (result.status === 'missing') { + if ( + this.options.activeMissingIsPending === true && + !TERMINAL_ATTEMPT_STATUSES.has(attempt.status) + ) { + return Object.freeze({ status: 'pending' as const, ...identity }); + } + return Object.freeze({ status: 'missing' as const, ...identity }); + } + return available(identity, range, result); + } catch (error) { + if ( + error instanceof InvalidRunAttemptLogReadError || + error instanceof RunAttemptLogReadUnavailableError + ) { + throw error; + } + throw new RunAttemptLogReadUnavailableError({ cause: error }); + } + } +} diff --git a/packages/ql3-runtime-core/test/runAttemptLogRead.test.cjs b/packages/ql3-runtime-core/test/runAttemptLogRead.test.cjs new file mode 100644 index 00000000..01d30f25 --- /dev/null +++ b/packages/ql3-runtime-core/test/runAttemptLogRead.test.cjs @@ -0,0 +1,261 @@ +const assert = require('node:assert/strict'); +const { test } = require('node:test'); + +const { + InvalidRunAttemptLogReadError, + MAX_RUN_ATTEMPT_LOG_READ_BYTES, + RunAttemptLogReadService, + RunAttemptLogReadUnavailableError, + normalizeRunAttemptLogReadRange, +} = require('../dist/run/log-read/runAttemptLogRead.js'); + +function run(overrides = {}) { + return { + id: 'run_123', + projectId: 'prj_default', + taskId: 'task_1', + taskRevision: 'revision_1', + triggerType: 'task_start', + executionOrigin: 'manual', + executionOwner: 'runtime', + status: 'running', + version: 2, + eventSequence: 2, + priority: 0, + createdAtMs: 1, + ...overrides, + }; +} + +function attempt(overrides = {}) { + return { + id: 'attempt_123', + runId: 'run_123', + attempt: 1, + status: 'running', + executorType: 'local_process', + logArtifactId: `local-${'a'.repeat(30)}`, + callbackSequence: 0, + createdAtMs: 1, + ...overrides, + }; +} + +function service(overrides = {}) { + const calls = []; + const runs = overrides.runs ?? { + async findRunById() { + return run(); + }, + async findAttemptById() { + return attempt(); + }, + }; + const reader = overrides.reader ?? { + async read(identity, range, signal) { + calls.push({ identity, range, signal }); + return { + status: 'available', + content: Buffer.from('log'), + start: range.offset, + endExclusive: range.offset + 3, + totalBytes: range.offset + 5, + nextOffset: range.offset + 3, + truncation: { truncated: false, maximumBytes: 1024, observedAtMs: 9 }, + }; + }, + }; + return { + calls, + value: new RunAttemptLogReadService(runs, reader, { + executorType: overrides.executorType ?? 'local_process', + artifactIdPattern: overrides.artifactIdPattern ?? /^local-[a-f0-9]{30}$/, + maximumReadBytes: overrides.maximumReadBytes ?? 32 * 1024, + ...(overrides.activeMissingIsPending === undefined + ? {} + : { activeMissingIsPending: overrides.activeMissingIsPending }), + }), + }; +} + +function request(overrides = {}) { + return { + projectId: 'prj_default', + runId: 'run_123', + attemptId: 'attempt_123', + range: { offset: 4, length: 16 }, + ...overrides, + }; +} + +test('normalizes only bounded safe ranges', () => { + assert.deepEqual(normalizeRunAttemptLogReadRange({ offset: 0, length: 1 }), { + offset: 0, + length: 1, + }); + assert.deepEqual( + normalizeRunAttemptLogReadRange({ + offset: Number.MAX_SAFE_INTEGER, + length: MAX_RUN_ATTEMPT_LOG_READ_BYTES, + }), + { offset: Number.MAX_SAFE_INTEGER, length: MAX_RUN_ATTEMPT_LOG_READ_BYTES }, + ); + for (const range of [ + { offset: -1, length: 1 }, + { offset: 0.5, length: 1 }, + { offset: 0, length: 0 }, + { offset: 0, length: MAX_RUN_ATTEMPT_LOG_READ_BYTES + 1 }, + ]) { + assert.throws( + () => normalizeRunAttemptLogReadRange(range), + InvalidRunAttemptLogReadError, + ); + } +}); + +test('validates Project, Run, Attempt, owner and executor before storage access', async () => { + const cases = [ + { run: null }, + { run: run({ projectId: 'prj_other' }) }, + { run: run({ executionOwner: 'legacy' }) }, + { attempt: null }, + { attempt: attempt({ runId: 'run_other' }) }, + { attempt: attempt({ executorType: 'remote_worker' }) }, + { attempt: attempt({ logArtifactId: `wlog-${'a'.repeat(30)}` }) }, + ]; + for (const values of cases) { + let reads = 0; + const { value } = service({ + runs: { + async findRunById() { + return values.run === undefined ? run() : values.run; + }, + async findAttemptById() { + return values.attempt === undefined ? attempt() : values.attempt; + }, + }, + reader: { + async read() { + reads += 1; + return { status: 'missing' }; + }, + }, + }); + assert.deepEqual(await value.read(request()), { status: 'not_found' }); + assert.equal(reads, 0); + } +}); + +test('returns pending before Artifact binding and for active remote publication lag', async () => { + const unbound = service({ + runs: { + async findRunById() { + return run(); + }, + async findAttemptById() { + return attempt({ logArtifactId: undefined }); + }, + }, + }); + assert.deepEqual(await unbound.value.read(request()), { + status: 'pending', + projectId: 'prj_default', + runId: 'run_123', + attemptId: 'attempt_123', + }); + assert.equal(unbound.calls.length, 0); + + const remote = service({ + executorType: 'remote_worker', + artifactIdPattern: /^wlog-[a-f0-9]{30}$/, + activeMissingIsPending: true, + runs: { + async findRunById() { + return run(); + }, + async findAttemptById() { + return attempt({ + executorType: 'remote_worker', + logArtifactId: `wlog-${'b'.repeat(30)}`, + }); + }, + }, + reader: { + async read() { + return { status: 'missing' }; + }, + }, + }); + assert.equal((await remote.value.read(request())).status, 'pending'); +}); + +test('returns a validated bounded snapshot without copying storage bytes', async () => { + const content = Buffer.from('log'); + const abort = new AbortController(); + const { value, calls } = service({ + reader: { + async read(identity, range, signal) { + assert.equal(signal, abort.signal); + return { + status: 'available', + content, + start: 4, + endExclusive: 7, + totalBytes: 9, + nextOffset: 7, + truncation: { + truncated: true, + maximumBytes: 64 * 1024, + observedAtMs: 10, + }, + }; + }, + }, + }); + const result = await value.read(request({ signal: abort.signal })); + assert.equal(result.status, 'available'); + assert.equal(result.content, content); + assert.equal(result.nextOffset, 7); + assert.deepEqual(result.truncation, { + truncated: true, + maximumBytes: 64 * 1024, + observedAtMs: 10, + }); + assert.equal(calls.length, 0); +}); + +test('fails closed on malformed storage results and dependency failures', async () => { + const malformed = service({ + reader: { + async read() { + return { + status: 'available', + content: Buffer.from('too-long'), + start: 4, + endExclusive: 12, + totalBytes: 9, + truncation: { truncated: 'unknown' }, + }; + }, + }, + }); + await assert.rejects( + malformed.value.read(request()), + RunAttemptLogReadUnavailableError, + ); + + const failed = service({ + runs: { + async findRunById() { + throw new Error('database detail'); + }, + async findAttemptById() { + return null; + }, + }, + }); + await assert.rejects( + failed.value.read(request()), + RunAttemptLogReadUnavailableError, + ); +}); diff --git a/scripts/ql3-cluster-dependency-audit.cjs b/scripts/ql3-cluster-dependency-audit.cjs index 0d9884eb..14ba8c13 100644 --- a/scripts/ql3-cluster-dependency-audit.cjs +++ b/scripts/ql3-cluster-dependency-audit.cjs @@ -1461,6 +1461,7 @@ function auditSourceImports(root, packagePath, findings) { specifier.startsWith('@qinglong/local-execution/')) && (packagePath !== 'packages/ql3-local-application' || ![ + '@qinglong/local-execution/artifact-read', '@qinglong/local-execution/control', '@qinglong/local-execution/dispatch', '@qinglong/local-execution/execution', @@ -2418,6 +2419,7 @@ function auditSourceImports(root, packagePath, findings) { '@qinglong/runtime-core/plugin-package-recovery', '@qinglong/runtime-core/plugin-package-task-publication', '@qinglong/runtime-core/project-tool-definition-snapshot', + '@qinglong/runtime-core/run-attempt-log-read', '@qinglong/runtime-core/task-spec-semantic', ].includes(specifier) && !( diff --git a/test/back/ql3PackageBoundaryAudit.test.cjs b/test/back/ql3PackageBoundaryAudit.test.cjs index e127f9bc..239e72ca 100644 --- a/test/back/ql3PackageBoundaryAudit.test.cjs +++ b/test/back/ql3PackageBoundaryAudit.test.cjs @@ -130,10 +130,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: localApi.rootSourceFileRoles, }, { - sourceFiles: 16, + sourceFiles: 17, rootSourceFiles: 1, rootSourceLines: 84, - nestedSourceFiles: 15, + nestedSourceFiles: 16, rootSourceFileRoles: { 'cli.ts': 'binary_entry' }, }, ); @@ -146,10 +146,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( { path: 'packages/ql3-local-execution', name: '@qinglong/local-execution', - sourceFiles: 19, + sourceFiles: 21, rootSourceFiles: 0, rootSourceLines: 0, - nestedSourceFiles: 19, + nestedSourceFiles: 21, rootSourceFileHardCap: 0, rootSourceLineHardCap: 0, rootSourceFileRoles: {}, @@ -299,10 +299,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: runtimeCore.rootSourceFileRoles, }, { - sourceFiles: 148, + sourceFiles: 149, rootSourceFiles: 1, rootSourceLines: 160, - nestedSourceFiles: 147, + nestedSourceFiles: 148, rootSourceFileRoles: { 'index.ts': 'public_export' }, }, ); @@ -385,10 +385,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: clusterControl.rootSourceFileRoles, }, { - sourceFiles: 49, + sourceFiles: 50, rootSourceFiles: 2, rootSourceLines: 195, - nestedSourceFiles: 47, + nestedSourceFiles: 48, rootSourceFileRoles: { 'aiCli.ts': 'binary_entry', 'cli.ts': 'binary_entry',