mirror of
https://github.com/whyour/qinglong.git
synced 2026-09-20 16:07:11 +08:00
feat(ql3): add opaque cluster environment bundle delivery
This commit is contained in:
@@ -120,6 +120,34 @@
|
|||||||
migration receipt、HA promotion replay 和固定低性能设备证据仍是后续门禁。ADR-0491 因此继续
|
migration receipt、HA promotion replay 和固定低性能设备证据仍是后续门禁。ADR-0491 因此继续
|
||||||
Proposed。
|
Proposed。
|
||||||
|
|
||||||
|
- D-401/ADR-0496(前置切片已验收,D-401 整体仍进行中):在真正执行 Cluster Task/Trigger mutation 前,先关闭一个原架构会泄漏
|
||||||
|
Legacy Env 名称的缺口。`qinglong/command@v1` 与 Cluster execution revision 只增加可选、同
|
||||||
|
Project、固定 version 的 `environmentBundleRef`;PostgreSQL Task/plan 历史只保存该引用,不保存
|
||||||
|
bundle schema、Env name/value、key ID 或 provider path。新的纯数据
|
||||||
|
`qinglong/environment-bundle@v1` 固定 1~256 项、单值 16 KiB、最终 name+value 64 KiB、carrier
|
||||||
|
96 KiB,并拒绝重复/保留名称、NUL、未知字段和非 canonical authority。ADR-0495 计划中的
|
||||||
|
`effectiveBindingCount` 实现 cap 也从误用的 100,000 source-row 上限收紧到 256。
|
||||||
|
|
||||||
|
Remote Worker Secret delivery 升级为 v2,将最多 64 个普通 `secretRefs` 与最多一个
|
||||||
|
`environmentBundleRefs` 分栏且禁止角色重叠;普通值仍为 16 KiB/项、64 KiB 合计,bundle carrier
|
||||||
|
为 96 KiB,response cap 为 256 KiB。PostgreSQL 在同一个 Session/Lease/Attempt/offer/revision
|
||||||
|
authority fence 中比较两组完整引用;mounted-files provider 只对 bundle authority 放宽单文件
|
||||||
|
carrier 上限,普通 Secret 仍为 16 KiB。Worker 从 durable inbox 内部装配 capability,经共享
|
||||||
|
mTLS client 获取 bundle,只在 Artifact 分配与 spawn 前的内存 materializer 中展开,并对 Task
|
||||||
|
原有环境名称冲突、256 项和最终 64 KiB 再次失败关闭。
|
||||||
|
|
||||||
|
本切片不增加 package、dependency、PostgreSQL migration/table/role、daemon、timer 或 watcher,
|
||||||
|
Local execution 对 bundle 继续拒绝并沿用 D-397 的逐 Secret SQLite authority。它只建立 D-401
|
||||||
|
mutation 的安全数据面前置,不声明 Task/Trigger 已改写、Secret 已由 direct Vault/KMS custody、
|
||||||
|
receipt 已完成或 migration receipt 已在 HA promotion 后重放;这些仍由下一切片关闭。Runtime Core
|
||||||
|
`586/586`、Worker `135/135`、Cluster Control `273 total / 271 pass / 2 skip`、Cluster PostgreSQL
|
||||||
|
`359 total / 356 pass / 3 skip`、18-package clean build/test 与 backend
|
||||||
|
`1568 total / 1566 pass / 2 skip` 均为 0 fail。package/dependency/Edge/service bridge/Local image 与基础
|
||||||
|
Edge/Standalone artifact audit 全部 compatible,workspace 保持 18 packages、无单文件或浅层 package;
|
||||||
|
基础 Edge/Standalone 为 `2,650,564 / 2,650,642 bytes`、324 files、58 modules。PostgreSQL 18.6
|
||||||
|
arm64 physical HA 通过 146 gates、timeline `1→2`,报告 SHA-256 为
|
||||||
|
`0ee2199d0a52a02025bff017a07477d707353d12aefc3811f474e3775f2bb86b`。
|
||||||
|
|
||||||
- D-396/ADR-0490(已验收):Run History 不再只有永久 `manual_external`,但也没有被错误实现为 Legacy 日志到 3.0 Run ledger 的回灌。
|
- D-396/ADR-0490(已验收):Run History 不再只有永久 `manual_external`,但也没有被错误实现为 Legacy 日志到 3.0 Run ledger 的回灌。
|
||||||
新的 Local adapter 以 ADR-0482 sealed capture bundle 作为 append-only 保全资产:Legacy history 必须逐事实选择 `retain_both`,Target history
|
新的 Local adapter 以 ADR-0482 sealed capture bundle 作为 append-only 保全资产:Legacy history 必须逐事实选择 `retain_both`,Target history
|
||||||
必须选择 `retain_target`;receipt 只绑定 signed review、application、bundle fingerprint、领域 inventory 与有界 fact counts,不保存表名、Run ID、
|
必须选择 `retain_target`;receipt 只绑定 signed review、application、bundle fingerprint、领域 inventory 与有界 fact counts,不保存表名、Run ID、
|
||||||
@@ -5700,7 +5728,7 @@
|
|||||||
| D-110 | package delivery inbox 必须原位成为唯一 Worker execution authority,不得复制到 legacy/第二 journal。状态以 revision CAS 经过 `accepted → starting_acknowledged → launching → started → running_acknowledged`;callback token 原文只在内存,spawn 前仅持久化 sequence/digest。只有 Executor tagged `rejected` 可证明 no-spawn 并上报 start-failure;异常、未知结果、spawn 后持久化不确定和重启可见 `launching` 全部进入 recovery,绝不自动二次 spawn。Processor 只消费注入 port,不新增 package、timer、数据库或 cluster 依赖 | Accepted(ADR-0111/0121:单 journal transition/page、ACK/spawn barrier、环境闭合、completion capability 擦除、ambiguous spawn recovery、restart no-respawn 与默认关闭的 production execution composition 已实现;完整 Worker 产品生命周期仍关闭) | adapter 双写不能原子覆盖两个文件 authority,旧/新 Offer digest 也不等价;原位状态机把 delivery receipt 与执行事实明确分开,同时让低配节点只维护原有文件/锁资源,让大节点按 Worker 实例扩展 |
|
| D-110 | package delivery inbox 必须原位成为唯一 Worker execution authority,不得复制到 legacy/第二 journal。状态以 revision CAS 经过 `accepted → starting_acknowledged → launching → started → running_acknowledged`;callback token 原文只在内存,spawn 前仅持久化 sequence/digest。只有 Executor tagged `rejected` 可证明 no-spawn 并上报 start-failure;异常、未知结果、spawn 后持久化不确定和重启可见 `launching` 全部进入 recovery,绝不自动二次 spawn。Processor 只消费注入 port,不新增 package、timer、数据库或 cluster 依赖 | Accepted(ADR-0111/0121:单 journal transition/page、ACK/spawn barrier、环境闭合、completion capability 擦除、ambiguous spawn recovery、restart no-respawn 与默认关闭的 production execution composition 已实现;完整 Worker 产品生命周期仍关闭) | adapter 双写不能原子覆盖两个文件 authority,旧/新 Offer digest 也不等价;原位状态机把 delivery receipt 与执行事实明确分开,同时让低配节点只维护原有文件/锁资源,让大节点按 Worker 实例扩展 |
|
||||||
| D-111 | Remote activation response 必须使用 `qinglong/remote-run-activation@v1` exact schema 与 16 KiB 上限,完整返回 Run/Attempt/Lease/callback fence,拒绝未知字段和空终态 snapshot。Offer 与 Activation 必须共用同一受限 Worker ingress HTTPS client、Agent、mTLS credential provider 与错误/字节预算,不得复制 TLS authority;Worker/event identity 由 path/服务端生成,不在 body 扩权。headless lifecycle 必须默认关闭、显式取得单 journal owner、无自有 timer,并在完整分页 startup reconciliation 前禁止 Pull;任何 recovery fact 都停止接收新工作。不得新增 package | Accepted(ADR-0112/0121:versioned wire、Cluster projection、共享单 Agent client、稳定凭据指纹池键、真实 TLS 1.3 mTLS、authority mismatch 负向测试、显式 lifecycle、分页 reconciliation、coalescing、drain-before-release 与完整 execution graph 已实现;外层 Session/credential/deploy gate 仍关闭) | 裸内部 object 会把 repository 漂移暴露到 wire;两套 HTTPS client 会让路由器重复 socket/证书状态;自动 timer 会让禁用不再等于零资源;扫描未收敛就 Pull 会在 ambiguous spawn 后继续扩大故障面。单 contract、单 socket authority 和 caller-driven tick 同时适配路由设备与实例级水平扩展 Worker |
|
| D-111 | Remote activation response 必须使用 `qinglong/remote-run-activation@v1` exact schema 与 16 KiB 上限,完整返回 Run/Attempt/Lease/callback fence,拒绝未知字段和空终态 snapshot。Offer 与 Activation 必须共用同一受限 Worker ingress HTTPS client、Agent、mTLS credential provider 与错误/字节预算,不得复制 TLS authority;Worker/event identity 由 path/服务端生成,不在 body 扩权。headless lifecycle 必须默认关闭、显式取得单 journal owner、无自有 timer,并在完整分页 startup reconciliation 前禁止 Pull;任何 recovery fact 都停止接收新工作。不得新增 package | Accepted(ADR-0112/0121:versioned wire、Cluster projection、共享单 Agent client、稳定凭据指纹池键、真实 TLS 1.3 mTLS、authority mismatch 负向测试、显式 lifecycle、分页 reconciliation、coalescing、drain-before-release 与完整 execution graph 已实现;外层 Session/credential/deploy gate 仍关闭) | 裸内部 object 会把 repository 漂移暴露到 wire;两套 HTTPS client 会让路由器重复 socket/证书状态;自动 timer 会让禁用不再等于零资源;扫描未收敛就 Pull 会在 ambiguous spawn 后继续扩大故障面。单 contract、单 socket authority 和 caller-driven tick 同时适配路由设备与实例级水平扩展 Worker |
|
||||||
| D-112 | Worker execution materializer 必须从 canonical Offer 提取最小 provider identity,不得把 lease/callback/credential capability、command 或完整 Offer 交给 Secret/Artifact provider。Secret ref 去重后最多 64 个,response 以 ref 显式配对且 exact;单值 16 KiB、解析后环境总量 64 KiB。必须 Secret 全部闭合后才分配 Attempt Artifact;所有失败路径执行幂等 cleanup。materializer、Executor、journal 与 running ACK 必须使用同一 `logArtifactId`。不得持久化 Secret 明文、新增 package/数据库/timer | Accepted(ADR-0113/0114/0121:最小 identity projection、Secret-before-Artifact、exact ref pairing、值/总量预算、dispose、log ID handoff、认证 HTTPS provider、Executor/upload composition 与负向测试已实现;具体 KMS/Vault 和 retention 仍关闭) | 完整 Offer 会扩散 bearer capability;位置数组会静默错配 Secret;先建日志会在 Secret 缺失时制造 orphan;Executor 自建日志 ID 会让控制面与实际输出分叉。最小显式配对和单日志身份同时限制路由器写放大并允许集群 Worker 用 provider 实现水平扩展 |
|
| D-112 | Worker execution materializer 必须从 canonical Offer 提取最小 provider identity,不得把 lease/callback/credential capability、command 或完整 Offer 交给 Secret/Artifact provider。Secret ref 去重后最多 64 个,response 以 ref 显式配对且 exact;单值 16 KiB、解析后环境总量 64 KiB。必须 Secret 全部闭合后才分配 Attempt Artifact;所有失败路径执行幂等 cleanup。materializer、Executor、journal 与 running ACK 必须使用同一 `logArtifactId`。不得持久化 Secret 明文、新增 package/数据库/timer | Accepted(ADR-0113/0114/0121:最小 identity projection、Secret-before-Artifact、exact ref pairing、值/总量预算、dispose、log ID handoff、认证 HTTPS provider、Executor/upload composition 与负向测试已实现;具体 KMS/Vault 和 retention 仍关闭) | 完整 Offer 会扩散 bearer capability;位置数组会静默错配 Secret;先建日志会在 Secret 缺失时制造 orphan;Executor 自建日志 ID 会让控制面与实际输出分叉。最小显式配对和单日志身份同时限制路由器写放大并允许集群 Worker 用 provider 实现水平扩展 |
|
||||||
| D-113 | Remote Worker Secret delivery 必须使用 `qinglong/remote-secret-delivery@v1` exact batch wire:最多 64 KiB request、128 KiB response、64 个唯一 Project-bound ref、单值 16 KiB且值总量 64 KiB。控制面必须在 Attempt advisory lock 下用 PostgreSQL 时钟复验认证 Worker/Session、Run/Attempt starting、Lease token/generation/version/expiry、offer、pinned revision、execution digest 与完整 ref 集合,成功后才调用 plaintext provider;旧 Session/Lease/version/Attempt 重放必须拒绝,同一有效 fence 允许网络幂等重试。Worker 通用 provider 不接收 lease capability,由 inbox-bound HTTPS adapter 内部装配并复用单 Agent mTLS client。runtime repository 通过受保护 service 注入,`ql3_worker_ingress` role 不扩权;不得持久化/记录 Secret、新增 package/schema/timer | Accepted(ADR-0114/0121:wire、runtime authority repository、受保护 service、ingress route、Worker HTTPS provider、共享 Agent、默认 4 KiB兼容、production execution composition 与 replay/leak/size/fence 负向测试已实现;具体 KMS/Vault 和外层产品部署仍关闭) | 逐 Secret 请求会放大路由器 TLS/JSON 成本,只按 Project/ref 验证会允许旧 Attempt 取密,把 token 交给通用 provider 或扩大 ingress 数据库权限都会扩散 authority。单批次完整围栏同时给低配节点确定资源上限,并允许集群 Worker 以无状态服务和外置 Secret backend 水平扩展 |
|
| D-113 | Remote Worker Secret delivery 当前使用 `qinglong/remote-secret-delivery@v2` exact batch wire:最多 64 KiB request、256 KiB response、64 个唯一普通 Project-bound ref 与一个独立 environment-bundle ref;普通单值 16 KiB且总量 64 KiB,bundle carrier 96 KiB,两个角色不得重叠。控制面必须在 Attempt advisory lock 下用 PostgreSQL 时钟复验认证 Worker/Session、Run/Attempt starting、Lease token/generation/version/expiry、offer、pinned revision、execution digest 与完整双 ref 集合,成功后才调用 plaintext provider;旧 Session/Lease/version/Attempt 重放必须拒绝,同一有效 fence 允许网络幂等重试。Worker 通用 provider 不接收 lease capability,由 inbox-bound HTTPS adapter 内部装配并复用单 Agent mTLS client。runtime repository 通过受保护 service 注入,`ql3_worker_ingress` role 不扩权;不得持久化/记录 Secret、新增 package/schema/timer | Accepted(ADR-0114/0121/0496:普通 Secret v1 围栏、v2 typed bundle、runtime authority repository、受保护 service、ingress route、Worker HTTPS provider、共享 Agent、默认 4 KiB兼容、production execution composition 与 replay/leak/size/fence 负向测试已实现;具体 KMS/Vault 和外层产品部署仍关闭) | 逐 Secret 请求会放大路由器 TLS/JSON 成本,只按 Project/ref 验证会允许旧 Attempt 取密,把 token 交给通用 provider 或扩大 ingress 数据库权限都会扩散 authority。单批次完整围栏同时给低配节点确定资源上限,并允许集群 Worker 以无状态服务和外置 Secret backend 水平扩展 |
|
||||||
| D-114 | Worker 本地日志必须在现有 worker-runtime 内使用 offer-scoped opaque `wlog-*` 单一身份和私有固定 shard;root/shard/file 强制 `0700/0700/0600`,`O_APPEND|O_NOFOLLOW` 后复验 owner、单 link、fd/path inode 与目录 identity。Edge/Node 分别使用 4/64 MiB Attempt 上限与 32/256 MiB reserve,单 write 最多 1 MiB;capacity 不明、symlink、identity 漂移、非法 chunk 和 quota 必须 fail closed。output 只能在 durable launching barrier 后 take 一次,调用 Executor 即转移 ownership;tagged reject 幂等 close,unknown spawn 禁止 Processor 误关。不得新增 package/schema/timer;本地 spool 未获 upload 与 control-plane ACK 前不得删除 | Accepted(ADR-0115–0117/0120/0121:文件 allocator/writer、Profile policy、capacity admission、append/fdatasync、单次 handoff、受审 POSIX Executor、truncation fact、固定 64 KiB source、认证中央 upload、S3-compatible immutable store 与 production execution-plane composition 已实现;range read/retention 仍默认关闭) | 日志 ID 与 writer 分离会产生不可路由字节,barrier 前交接或 finally 无条件 close 会制造泄漏/截断,无界输出会耗尽路由器磁盘。单实现加显式 Profile policy 同时适配低配设备和集群 Worker;上传/保留必须通过 receipt 闭环,不能从本地 close 推断完成 |
|
| D-114 | Worker 本地日志必须在现有 worker-runtime 内使用 offer-scoped opaque `wlog-*` 单一身份和私有固定 shard;root/shard/file 强制 `0700/0700/0600`,`O_APPEND|O_NOFOLLOW` 后复验 owner、单 link、fd/path inode 与目录 identity。Edge/Node 分别使用 4/64 MiB Attempt 上限与 32/256 MiB reserve,单 write 最多 1 MiB;capacity 不明、symlink、identity 漂移、非法 chunk 和 quota 必须 fail closed。output 只能在 durable launching barrier 后 take 一次,调用 Executor 即转移 ownership;tagged reject 幂等 close,unknown spawn 禁止 Processor 误关。不得新增 package/schema/timer;本地 spool 未获 upload 与 control-plane ACK 前不得删除 | Accepted(ADR-0115–0117/0120/0121:文件 allocator/writer、Profile policy、capacity admission、append/fdatasync、单次 handoff、受审 POSIX Executor、truncation fact、固定 64 KiB source、认证中央 upload、S3-compatible immutable store 与 production execution-plane composition 已实现;range read/retention 仍默认关闭) | 日志 ID 与 writer 分离会产生不可路由字节,barrier 前交接或 finally 无条件 close 会制造泄漏/截断,无界输出会耗尽路由器磁盘。单实现加显式 Profile policy 同时适配低配设备和集群 Worker;上传/保留必须通过 receipt 闭环,不能从本地 close 推断完成 |
|
||||||
| D-115 | Worker POSIX 执行必须复用 local-process 的受审 launcher/receipt/durable identity,不复制实现。`launching` 必须在 spawn 前持久化 exact start time、log ID 与 callback sequence/digest;launcher journal、receipt、Executor result 和 inbox 必须一致,使 spawn 后/started 前崩溃可从 authenticated receipt 恢复。completion receipt 的 base64url capability 必须 canonical decode 为 32 bytes 后 constant-time 校验原始 digest 并清零;提交顺序固定为流式 Artifact upload + byte/digest ACK → lease-fenced completion → durable inbox terminal ACK → receipt cleanup,本地 spool 只能由后续 retention 删除。timeout 未有 durable evidence 时 fail closed。实现留在现有 worker-runtime subpath,不新增 package/schema/timer | Accepted(ADR-0116/0117/0118/0121:reviewed-fd Executor、durable pre-spawn time、launching crash recovery、认证流式 Artifact upload、PostgreSQL completion、数据库 deadline、lease-control stop、用户取消 intent、非执行状态 convergence、cleanup ordering 与具体 production execution stack 已实现;本地 spool retention 仍关闭) | 把启动时间延后保存会留下不可认证崩溃窗口;对编码字符串 hash 会产生错误 authority;先完成后上传或上传后删 spool 都会丢日志。显式可重放顺序让 Edge 以固定内存工作,也允许 Node/集群提高容量而共享同一状态机 |
|
| D-115 | Worker POSIX 执行必须复用 local-process 的受审 launcher/receipt/durable identity,不复制实现。`launching` 必须在 spawn 前持久化 exact start time、log ID 与 callback sequence/digest;launcher journal、receipt、Executor result 和 inbox 必须一致,使 spawn 后/started 前崩溃可从 authenticated receipt 恢复。completion receipt 的 base64url capability 必须 canonical decode 为 32 bytes 后 constant-time 校验原始 digest 并清零;提交顺序固定为流式 Artifact upload + byte/digest ACK → lease-fenced completion → durable inbox terminal ACK → receipt cleanup,本地 spool 只能由后续 retention 删除。timeout 未有 durable evidence 时 fail closed。实现留在现有 worker-runtime subpath,不新增 package/schema/timer | Accepted(ADR-0116/0117/0118/0121:reviewed-fd Executor、durable pre-spawn time、launching crash recovery、认证流式 Artifact upload、PostgreSQL completion、数据库 deadline、lease-control stop、用户取消 intent、非执行状态 convergence、cleanup ordering 与具体 production execution stack 已实现;本地 spool retention 仍关闭) | 把启动时间延后保存会留下不可认证崩溃窗口;对编码字符串 hash 会产生错误 authority;先完成后上传或上传后删 spool 都会丢日志。显式可重放顺序让 Edge 以固定内存工作,也允许 Node/集群提高容量而共享同一状态机 |
|
||||||
| D-116 | Remote Worker Artifact/completion 必须复用同一 TLS 1.3 mTLS Worker ingress client 与 Agent。Artifact 使用 `qinglong/remote-worker-artifact-upload@v1`:4-byte big-endian header length、最多 4 KiB exact JSON header、最多 64 MiB 原始内容、固定 media type、精确 Content-Length、identity encoding;普通 JSON 上限不得因此放宽。控制面必须在读取内容前认证 path-bound Worker/Session,在共享 Attempt fence 下授权后才写入跨副本可见的 immutable digest store,禁止 controller 本地文件冒充 cluster store;完成前必须重新 inspect 同一 Artifact 并精确匹配 ID/length/SHA-256/truncation。`qinglong/remote-worker-completion@v1` 必须在一个 PostgreSQL transaction 中按既定顺序锁定 authority,允许从 durable `starting|running` 崩溃窗口收敛,以数据库时间和 cancellation/timeout 优先级原子完成 Lease、Attempt、Run 与双 Event;服务端生成 Event ID,raw lease token 不入库/事件,只有完整终态与 Event payload 一致才允许 exact replay。缺少共享 Artifact store 的生产组合必须默认关闭;不得新增 workspace package/schema/timer/队列 | Accepted(ADR-0117/0120/0121:runtime wire、streaming ingress、Worker HTTPS adapters、共享 store port、S3-compatible checksum/conditional-copy adapter、Artifact re-inspection、PostgreSQL terminal transaction、真实 TLS 1.3、真实 MinIO SSE-S3、完整 fence/replay 与 concrete Worker execution composition 已实现;range read/retention 仍关闭) | JSON base64 会复制 64 MiB 日志并抬高路由器 RSS;controller 本地盘在多副本下不可见且会产生假 ACK;先信任 Worker digest 或把 upload 与终态放进同一长事务会分别允许内容漂移或跨网络持锁。认证前置、流式固定上限、共享 immutable store 与短终态事务同时适配低配 Worker 和水平扩展控制面 |
|
| D-116 | Remote Worker Artifact/completion 必须复用同一 TLS 1.3 mTLS Worker ingress client 与 Agent。Artifact 使用 `qinglong/remote-worker-artifact-upload@v1`:4-byte big-endian header length、最多 4 KiB exact JSON header、最多 64 MiB 原始内容、固定 media type、精确 Content-Length、identity encoding;普通 JSON 上限不得因此放宽。控制面必须在读取内容前认证 path-bound Worker/Session,在共享 Attempt fence 下授权后才写入跨副本可见的 immutable digest store,禁止 controller 本地文件冒充 cluster store;完成前必须重新 inspect 同一 Artifact 并精确匹配 ID/length/SHA-256/truncation。`qinglong/remote-worker-completion@v1` 必须在一个 PostgreSQL transaction 中按既定顺序锁定 authority,允许从 durable `starting|running` 崩溃窗口收敛,以数据库时间和 cancellation/timeout 优先级原子完成 Lease、Attempt、Run 与双 Event;服务端生成 Event ID,raw lease token 不入库/事件,只有完整终态与 Event payload 一致才允许 exact replay。缺少共享 Artifact store 的生产组合必须默认关闭;不得新增 workspace package/schema/timer/队列 | Accepted(ADR-0117/0120/0121:runtime wire、streaming ingress、Worker HTTPS adapters、共享 store port、S3-compatible checksum/conditional-copy adapter、Artifact re-inspection、PostgreSQL terminal transaction、真实 TLS 1.3、真实 MinIO SSE-S3、完整 fence/replay 与 concrete Worker execution composition 已实现;range read/retention 仍关闭) | JSON base64 会复制 64 MiB 日志并抬高路由器 RSS;controller 本地盘在多副本下不可见且会产生假 ACK;先信任 Worker digest 或把 upload 与终态放进同一长事务会分别允许内容漂移或跨网络持锁。认证前置、流式固定上限、共享 immutable store 与短终态事务同时适配低配 Worker 和水平扩展控制面 |
|
||||||
@@ -6982,7 +7010,7 @@ Dispatcher 只能把任务分配给满足 required 条件的 Worker。当没有
|
|||||||
- ADR-0111 已让该 package inbox 原位承担唯一 execution authority:同一 offer 文件用 revision CAS 持久化 starting ACK、spawn crash barrier、started handle 与 running ACK;materialized environment 必须与 immutable revision 精确闭合,completion token 只在内存且局部副本随后清零。Executor 只有 tagged `rejected` 才能上报 start-failure,异常或未知 spawn 进入 recovery;重启可见 `launching` 不调用任何 Executor。该 Processor 仍由显式 subpath 暴露,不创建 timer 或生产连接。
|
- ADR-0111 已让该 package inbox 原位承担唯一 execution authority:同一 offer 文件用 revision CAS 持久化 starting ACK、spawn crash barrier、started handle 与 running ACK;materialized environment 必须与 immutable revision 精确闭合,completion token 只在内存且局部副本随后清零。Executor 只有 tagged `rejected` 才能上报 start-failure,异常或未知 spawn 进入 recovery;重启可见 `launching` 不调用任何 Executor。该 Processor 仍由显式 subpath 暴露,不创建 timer 或生产连接。
|
||||||
- ADR-0112 已把 starting/running/start-failure 接入 exact versioned response,并让 Offer、Activation 与后续 Secret delivery 复用同一 TLS 1.3 mTLS client、credential authority 和单 keep-alive Agent;显式 headless lifecycle 在完整 inbox reconciliation 收敛前不 Pull,且不创建自有 timer。
|
- ADR-0112 已把 starting/running/start-failure 接入 exact versioned response,并让 Offer、Activation 与后续 Secret delivery 复用同一 TLS 1.3 mTLS client、credential authority 和单 keep-alive Agent;显式 headless lifecycle 在完整 inbox reconciliation 收敛前不 Pull,且不创建自有 timer。
|
||||||
- ADR-0113 已实现 Secret-before-Artifact 的有界 materializer:provider 只得到 capability-free 最小 identity,Secret ref 去重并 exact 配对,解析环境总量 64 KiB;同一 `logArtifactId` 从 allocator 交给 Executor、journal 与 running ACK,所有失败路径幂等 cleanup。
|
- ADR-0113 已实现 Secret-before-Artifact 的有界 materializer:provider 只得到 capability-free 最小 identity,Secret ref 去重并 exact 配对,解析环境总量 64 KiB;同一 `logArtifactId` 从 allocator 交给 Executor、journal 与 running ACK,所有失败路径幂等 cleanup。
|
||||||
- ADR-0114 已把批量 Secret provider 接入真实认证边界:`qinglong/remote-secret-delivery@v1` 以 64 KiB request/128 KiB response 复验 Session/Lease/Attempt/revision 完整围栏,runtime repository 在数据库权威锁内通过后才调用 plaintext provider;Worker 从 canonical inbox 内部装配 token,通用 provider 不接收 capability。`ql3_worker_ingress` 数据库权限、package 数、schema 与 timer 均不增加。
|
- ADR-0114/0496 已把批量 Secret provider 接入真实认证边界:`qinglong/remote-secret-delivery@v2` 以 64 KiB request/256 KiB response 复验 Session/Lease/Attempt/revision 和普通/bundle 双 ref-set 完整围栏,runtime repository 在数据库权威锁内通过后才调用 plaintext provider;Worker 从 canonical inbox 内部装配 token,通用 provider 不接收 capability。普通 Secret 仍为 16 KiB/项与 64 KiB 合计,唯一 bundle carrier 为 96 KiB。`ql3_worker_ingress` 数据库权限、package 数、schema 与 timer 均不增加。
|
||||||
- ADR-0115 已在现有 Worker package 内实现 offer-scoped `wlog-*` 私有文件 spool:Edge/Node 显式容量策略、append/reopen、symlink/owner/inode/link 防护、单 write/总量硬上限,并在 durable launching barrier 后一次性交接 output ownership;tagged reject 与 unknown spawn 的 close 语义分离。它不冒充具体 Executor、上传 ACK 或 retention。
|
- ADR-0115 已在现有 Worker package 内实现 offer-scoped `wlog-*` 私有文件 spool:Edge/Node 显式容量策略、append/reopen、symlink/owner/inode/link 防护、单 write/总量硬上限,并在 durable launching barrier 后一次性交接 output ownership;tagged reject 与 unknown spawn 的 close 语义分离。它不冒充具体 Executor、上传 ACK 或 retention。
|
||||||
- ADR-0116 已让 Worker 复用 `@qinglong/local-process` 的受审 verified-fd launcher、completion receipt 与 durable identity,不复制进程实现或新增 package。`launching` 现在在 spawn 前同时持久化 exact start time、log ID 和 callback digest,因此 spawn 后/started 前崩溃可由回执严格恢复;日志 source 以最多 64 KiB chunk 读取并验证 truncation fact,完成协调器固定执行 upload byte/digest ACK → lease-fenced completion → inbox terminal ACK → receipt cleanup,且从不删除本地 spool。
|
- ADR-0116 已让 Worker 复用 `@qinglong/local-process` 的受审 verified-fd launcher、completion receipt 与 durable identity,不复制进程实现或新增 package。`launching` 现在在 spawn 前同时持久化 exact start time、log ID 和 callback digest,因此 spawn 后/started 前崩溃可由回执严格恢复;日志 source 以最多 64 KiB chunk 读取并验证 truncation fact,完成协调器固定执行 upload byte/digest ACK → lease-fenced completion → inbox terminal ACK → receipt cleanup,且从不删除本地 spool。
|
||||||
- ADR-0117 已补齐认证中央链路:Worker 复用单一 mTLS Agent,以 4 KiB header + 最多 64 MiB 原始内容流式上传;control ingress 在 body 前认证并用共享 Attempt fence 授权,只向跨副本 immutable Artifact store 写入。completion 前重新 inspect exact digest/length/truncation,再在一个 PostgreSQL transaction 中从 `starting|running` 收敛 Lease、Attempt、Run 与双 Event,支持 Event-authenticated exact replay。Artifact range read 与 retention 仍默认关闭;具体 production execution composition 由 ADR-0121 补齐。
|
- ADR-0117 已补齐认证中央链路:Worker 复用单一 mTLS Agent,以 4 KiB header + 最多 64 MiB 原始内容流式上传;control ingress 在 body 前认证并用共享 Attempt fence 授权,只向跨副本 immutable Artifact store 写入。completion 前重新 inspect exact digest/length/truncation,再在一个 PostgreSQL transaction 中从 `starting|running` 收敛 Lease、Attempt、Run 与双 Event,支持 Event-authenticated exact replay。Artifact range read 与 retention 仍默认关闭;具体 production execution composition 由 ADR-0121 补齐。
|
||||||
|
|||||||
@@ -5,6 +5,10 @@
|
|||||||
- 关联 RFC:QL-RFC-0001 D-24、D-72、D-85、D-109、D-111、D-112、D-113
|
- 关联 RFC:QL-RFC-0001 D-24、D-72、D-85、D-109、D-111、D-112、D-113
|
||||||
- 关联 ADR:ADR-0058、ADR-0073、ADR-0108、ADR-0109、ADR-0110、ADR-0113
|
- 关联 ADR:ADR-0058、ADR-0073、ADR-0108、ADR-0109、ADR-0110、ADR-0113
|
||||||
|
|
||||||
|
> 2026-08-24:ADR-0496 为 Cluster Legacy Env 增加 typed environment bundle,wire 已升级为
|
||||||
|
> `qinglong/remote-secret-delivery@v2`。本 ADR 的 v1 普通 Secret 围栏、顺序与预算语义继续有效;
|
||||||
|
> v2 的双 ref-set、96 KiB bundle carrier 和 256 KiB response cap 以 ADR-0496 为准。
|
||||||
|
|
||||||
## 背景
|
## 背景
|
||||||
|
|
||||||
ADR-0113 已固定 Secret-before-Artifact 的 Worker materializer,但只留下抽象 provider。
|
ADR-0113 已固定 Secret-before-Artifact 的 Worker materializer,但只留下抽象 provider。
|
||||||
@@ -20,11 +24,12 @@ Attempt 或被篡改 execution revision 都可能重放同一 Secret capability
|
|||||||
|
|
||||||
### 1. 使用一个 exact versioned 批量协议
|
### 1. 使用一个 exact versioned 批量协议
|
||||||
|
|
||||||
协议固定为 `qinglong/remote-secret-delivery@v1`。单次 request 最多 64 KiB、64 个唯一
|
当前协议为 `qinglong/remote-secret-delivery@v2`;原 v1 是普通 Secret-only 的历史基线。
|
||||||
`qlsecret:v1` reference;response 最多 128 KiB,单值最多 16 KiB、值总量最多 64 KiB。
|
单次 request 最多 64 KiB,分为最多 64 个唯一普通 `secretRefs` 与最多一个
|
||||||
response 必须按请求顺序返回 exact `{secretRef,value}`,并回绑 Run、Attempt、offer 与
|
`environmentBundleRefs`,两组不得重叠;response 最多 256 KiB。普通单值最多 16 KiB、值总量
|
||||||
execution digest;未知字段、缺项、乱序、重复、跨 Project reference、NUL、超限或身份
|
最多 64 KiB,bundle carrier 最多 96 KiB。response 必须按两组请求顺序分别返回 exact
|
||||||
漂移全部 fail closed。
|
`{secretRef,value}`,并回绑 Run、Attempt、offer 与 execution digest;未知字段、缺项、乱序、
|
||||||
|
重复、跨 Project reference、NUL、超限或身份漂移全部 fail closed。
|
||||||
|
|
||||||
Worker 继续复用 ADR-0112 的单一 `WorkerIngressHttpsClient`、TLS 1.3 mTLS credential
|
Worker 继续复用 ADR-0112 的单一 `WorkerIngressHttpsClient`、TLS 1.3 mTLS credential
|
||||||
provider 与最多一个 keep-alive socket。现有 offer/activation 的默认 request 上限保持
|
provider 与最多一个 keep-alive socket。现有 offer/activation 的默认 request 上限保持
|
||||||
@@ -39,13 +44,13 @@ provider 与最多一个 keep-alive socket。现有 offer/activation 的默认 r
|
|||||||
- Run、Attempt、Project、Task 与 pinned Task revision;
|
- Run、Attempt、Project、Task 与 pinned Task revision;
|
||||||
- offer ID 与 execution digest;
|
- offer ID 与 execution digest;
|
||||||
- Lease generation、token 与 expected version;
|
- Lease generation、token 与 expected version;
|
||||||
- 完整且有序的 SecretRef 集合。
|
- 完整且有序的普通 SecretRef 与 environment bundle SecretRef 两组集合。
|
||||||
|
|
||||||
runtime authority repository 在 Attempt advisory transaction lock 下使用 PostgreSQL 时钟,
|
runtime authority repository 在 Attempt advisory transaction lock 下使用 PostgreSQL 时钟,
|
||||||
要求 Run=`dispatching`、Attempt=`starting`、executor=`remote_worker`、Session 为当前
|
要求 Run=`dispatching`、Attempt=`starting`、executor=`remote_worker`、Session 为当前
|
||||||
`online|draining` 且未过期、Lease=`leased` 且未过期,并逐项比较 Attempt/Session/Lease
|
`online|draining` 且未过期、Lease=`leased` 且未过期,并逐项比较 Attempt/Session/Lease
|
||||||
中的 worker、generation、offer、token digest 与 version。随后重新规范化 immutable
|
中的 worker、generation、offer、token digest 与 version。随后重新规范化 immutable
|
||||||
`task_execution_revisions.plan_json`,要求 execution digest 与 SecretRef 集合完全一致。
|
`task_execution_revisions.plan_json`,要求 execution digest 与两组 SecretRef 集合完全一致。
|
||||||
任一漂移都在调用明文 provider 前拒绝。
|
任一漂移都在调用明文 provider 前拒绝。
|
||||||
|
|
||||||
同一仍然有效的 starting fence 可以因网络丢包进行幂等重试;它不产生“已消费”事实。
|
同一仍然有效的 starting fence 可以因网络丢包进行幂等重试;它不产生“已消费”事实。
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# ADR-0491:有界 Secret/Config Reconciliation 与任务环境绑定
|
# ADR-0491:有界 Secret/Config Reconciliation 与任务环境绑定
|
||||||
|
|
||||||
- 状态:Proposed(D-397 已实现 Legacy Env inspection、私有有界 row plan、durable plan publication、独立 signed decision、逐项 Automation adoption provenance、Local SQLite 原子 application publisher、Owner prepared/apply/rollback 编排、ADR-0492 completion v3;ADR-0494 完成 Cluster mounted-files provider live 子门,ADR-0495 完成 content-free Cluster plan ledger baseline;真实 Edge 空间证据、Cluster Task/Trigger mutation/receipt、HA replay 与直接外部 custody gate 尚未完成)
|
- 状态:Proposed(D-397 已实现 Legacy Env inspection、私有有界 row plan、durable plan publication、独立 signed decision、逐项 Automation adoption provenance、Local SQLite 原子 application publisher、Owner prepared/apply/rollback 编排、ADR-0492 completion v3;ADR-0494 完成 Cluster mounted-files provider live 子门,ADR-0495 完成 content-free Cluster plan ledger baseline,ADR-0496 完成 opaque environment bundle 的 Worker 内存展开与 HA-validated 数据面;真实 Edge 空间证据、Cluster Task/Trigger mutation/receipt、promotion 后 receipt replay 与直接外部 custody gate 尚未完成)
|
||||||
- 日期:2026-08-23
|
- 日期:2026-08-23
|
||||||
- 决策:D-397
|
- 决策:D-397
|
||||||
- 关联:ADR-0073、ADR-0074、ADR-0092、ADR-0094、ADR-0480、ADR-0482、ADR-0483、ADR-0484、ADR-0485、ADR-0486、ADR-0487、ADR-0488、ADR-0490
|
- 关联:ADR-0073、ADR-0074、ADR-0092、ADR-0094、ADR-0480、ADR-0482、ADR-0483、ADR-0484、ADR-0485、ADR-0486、ADR-0487、ADR-0488、ADR-0490
|
||||||
@@ -147,4 +147,4 @@ D-397 当前八切片已经实现:absent、unsupported、Edge over-budget、2.
|
|||||||
|
|
||||||
ADR-0494 已完成 Cluster `mounted-files` provider live 子门:真实三节点 K3s 中两个 management replica、direct exact-key executor 和两个跨节点 provider observer 完成 PostgreSQL durable approval/binding、Kubernetes atomic projection rotation、无 Secret API 权限/ServiceAccount token、只读 `0440`、内容脱敏及删除后 fail-closed;v2 私有报告 24/24 gates 为 true,并保持 v1 verifier 兼容。该门不增加 Edge 闭包,也不等于直接 Vault/KMS/HSM custody。
|
ADR-0494 已完成 Cluster `mounted-files` provider live 子门:真实三节点 K3s 中两个 management replica、direct exact-key executor 和两个跨节点 provider observer 完成 PostgreSQL durable approval/binding、Kubernetes atomic projection rotation、无 Secret API 权限/ServiceAccount token、只读 `0440`、内容脱敏及删除后 fail-closed;v2 私有报告 24/24 gates 为 true,并保持 v1 verifier 兼容。该门不增加 Edge 闭包,也不等于直接 Vault/KMS/HSM custody。
|
||||||
|
|
||||||
转为 Accepted 前仍必须完成:固定低性能 Edge 设备的真实空间/写放大/断电恢复证据,以及 Cluster Legacy Env migration 的逐项 Task/Trigger current-head revalidation、revision mutation/receipt、外部 custody adapter 和 HA promotion 后 receipt replay。ADR-0495 已完成专用 PostgreSQL SERIALIZABLE plan ledger baseline,但它只保存摘要、计数和 pinned SecretRef,不执行 Task/Trigger DML,也不接触 Secret material。ADR-0492 已完成本机 completion schema 演进和 completed-head 后 rollback material 回收,ADR-0493 又让没有 Legacy 身份输入的 fresh v52 目标身份经 signed `retain_target` 正确形成 no-effect,并精确消除六张已知目标表的 `unknown` 误判。Legacy `Auths/Users` 或真正未知表仍保持 manual;本切片的 Local Owner 编排、ADR-0494 的 mounted-files gate、ADR-0495 的 plan ledger 或 PostgreSQL HA 证据都不得冒充完整 Cluster migration 与外部密钥托管。
|
转为 Accepted 前仍必须完成:固定低性能 Edge 设备的真实空间/写放大/断电恢复证据,以及 Cluster Legacy Env migration 的逐项 Task/Trigger current-head revalidation、revision mutation/receipt、外部 custody adapter 和 HA promotion 后 receipt replay。ADR-0495 已完成专用 PostgreSQL SERIALIZABLE plan ledger baseline,但它只保存摘要、计数和 pinned SecretRef,不执行 Task/Trigger DML,也不接触 Secret material。ADR-0496 已验收只保存 pinned bundle ref、通过 fenced remote delivery 取回 typed carrier、在 Worker 内存展开的安全数据面,并通过 146-gate PostgreSQL HA;它仍不执行 migration DML 或生成 migration receipt。ADR-0492 已完成本机 completion schema 演进和 completed-head 后 rollback material 回收,ADR-0493 又让没有 Legacy 身份输入的 fresh v52 目标身份经 signed `retain_target` 正确形成 no-effect,并精确消除六张已知目标表的 `unknown` 误判。Legacy `Auths/Users` 或真正未知表仍保持 manual;本切片的 Local Owner 编排、ADR-0494 的 mounted-files gate、ADR-0495 的 plan ledger、ADR-0496 的数据面或 PostgreSQL HA 证据都不得冒充完整 Cluster migration 与外部密钥托管。
|
||||||
|
|||||||
@@ -0,0 +1,131 @@
|
|||||||
|
# ADR-0496:Cluster 不透明环境 Bundle 的有界交付与 Worker 内存展开
|
||||||
|
|
||||||
|
- 状态:Accepted(D-401 的安全执行数据面前置切片;Cluster Task/Trigger mutation 与 migration receipt 仍未完成)
|
||||||
|
- 日期:2026-08-24
|
||||||
|
- 决策:D-401 前置切片
|
||||||
|
- 关联:ADR-0091、ADR-0092、ADR-0104、ADR-0113、ADR-0114、ADR-0491、ADR-0494、ADR-0495
|
||||||
|
|
||||||
|
## 背景
|
||||||
|
|
||||||
|
ADR-0495 的 Cluster plan 只保存一个同 Project、固定 version 的 SecretRef,并明确禁止把
|
||||||
|
Legacy Env 名称、值、密文或 key ID 写入 PostgreSQL。现有 `command@v1` 环境模型却要求
|
||||||
|
每个 Secret binding 同时保存 `{name,secretRef}`;如果 D-401 直接把 Legacy Env 逐项追加到
|
||||||
|
Task revision,`task_definition_revisions.spec_json` 和
|
||||||
|
`task_execution_revisions.plan_json` 会永久保存全部 Legacy Env 名称,违反 ADR-0491。
|
||||||
|
|
||||||
|
现有 Remote Worker Secret delivery 还假设一个 SecretRef 对应一个最多 16 KiB 的普通值。
|
||||||
|
Legacy active environment 的合法上限是 256 项、单值 16 KiB、最终 name+value 总计 64 KiB,
|
||||||
|
因此不能把一个最多约 96 KiB 的 JSON carrier 冒充普通 Secret,也不能在 Worker 启动前写
|
||||||
|
ConfigMap、Pod environment、临时 export 文件或 command。
|
||||||
|
|
||||||
|
## 决策
|
||||||
|
|
||||||
|
### 1. Task 与执行修订只保存一个固定版本引用
|
||||||
|
|
||||||
|
`qinglong/command@v1` 增加可选 `environmentBundleRef`。它必须是 canonical、同 Project 且
|
||||||
|
显式固定 version 的 `qlsecret:v1` reference。Task semantic registry、profile-neutral compiler、
|
||||||
|
Cluster execution revision digest 和 PostgreSQL plan JSON 都只携带这个引用;不携带 bundle
|
||||||
|
schema、Env 名称、值、行数、key ID 或 provider path。
|
||||||
|
|
||||||
|
这是 `command@v1` 与 `qinglong/command-execution@v1` 的严格可选扩展:没有该字段的历史
|
||||||
|
revision 按原字节与原摘要解释,不执行回填或静默重写。Local execution compiler 当前拒绝带
|
||||||
|
bundle 的 Task;现有 Local D-397 继续使用逐 Secret SQLite 原子绑定,不能意外获得 Cluster
|
||||||
|
external-custody authority。
|
||||||
|
|
||||||
|
### 2. Bundle 是独立、纯数据、严格有界的 carrier
|
||||||
|
|
||||||
|
`qinglong/environment-bundle@v1` 固定为:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"schema":"qinglong/environment-bundle@v1","entries":[{"name":"NAME","value":"opaque"}]}
|
||||||
|
```
|
||||||
|
|
||||||
|
- entry 为 exact `{name,value}`,按 name canonical 排序;
|
||||||
|
- 1~256 项,名称符合 shell 共同子集、不得使用 `QL3_`、不得重复;
|
||||||
|
- 单值最多 16 KiB,最终 name+value 总计最多 64 KiB;
|
||||||
|
- UTF-8 JSON carrier 最多 96 KiB,拒绝 NUL、accessor、symbol、未知字段和无效 JSON;
|
||||||
|
- parser 是 profile-neutral 纯函数,不读取文件、网络、数据库、时钟或 Secret provider。
|
||||||
|
|
||||||
|
ADR-0495 的 `effectiveBindingCount` 实现上限同时从错误复用的 100,000 source-row cap 修正为
|
||||||
|
256,使 plan contract 与 ADR-0491 的真实执行预算一致。
|
||||||
|
|
||||||
|
### 3. Secret delivery v2 把普通 Secret 与 bundle 分权
|
||||||
|
|
||||||
|
wire 升级为 `qinglong/remote-secret-delivery@v2`。请求与数据库 authority 分别携带:
|
||||||
|
|
||||||
|
- `secretRefs`:最多 64 个普通 SecretRef;
|
||||||
|
- `environmentBundleRefs`:最多 1 个 bundle SecretRef;
|
||||||
|
- 两组不得重叠,合计不得为空。
|
||||||
|
|
||||||
|
response 分别返回 `values` 与 `environmentBundles`。普通值继续保持单项 16 KiB、总计
|
||||||
|
64 KiB;bundle carrier 最多 96 KiB;response cap 从 128 KiB 提升到 256 KiB,request cap
|
||||||
|
仍为 64 KiB。Run、Attempt、Session、Lease、offer、execution digest 和完整有序的两组 ref
|
||||||
|
仍在同一个 PostgreSQL authority transaction 中复验,bundle 不获得更弱的授权路径。
|
||||||
|
|
||||||
|
mounted-files provider 对普通 Secret 继续执行 16 KiB 上限,只对 authority 明确标记的唯一
|
||||||
|
bundle ref 允许 96 KiB。它仍逐请求重读 projection、无缓存、timer、watcher 或 Kubernetes API
|
||||||
|
权限;provider 不解析或记录 Env 内容。
|
||||||
|
|
||||||
|
### 4. Worker 只在进程上下文准备阶段展开
|
||||||
|
|
||||||
|
Worker 从 durable inbox 重建 exact Offer,由同一个 mTLS client 获取 v2 response。materializer
|
||||||
|
先解析普通 bindings,再在内存中解析唯一 bundle,并拒绝 bundle 内重复名称、与 Task 原有
|
||||||
|
environment 名称冲突、畸形 response、超过 256 项或最终 64 KiB 的环境。全部验证完成后才
|
||||||
|
分配 Artifact,随后把扁平 `{name,value}` 交给 Executor;失败时释放 provider material,且不
|
||||||
|
spawn。
|
||||||
|
|
||||||
|
Bundle 明文不得进入 inbox、Offer、PostgreSQL、Artifact、日志、audit、diagnostic、错误消息或
|
||||||
|
临时文件。JavaScript string 不能可靠清零,因此实现不缓存 bundle;transport Buffer 在解析后
|
||||||
|
清零,并在 execution context dispose 后释放引用。
|
||||||
|
|
||||||
|
### 5. Profile 与部署成本
|
||||||
|
|
||||||
|
实现复用 `runtime-core`、`worker-runtime`、`cluster-control` 和 `cluster-postgres`,不新增 workspace
|
||||||
|
package、生产依赖、数据库 migration、表、role grant、daemon、线程、timer 或 watcher。
|
||||||
|
Edge/Standalone 默认 import graph 不导入 Cluster provider/transport;纯 bundle parser 只在
|
||||||
|
实际调用时按 96 KiB carrier 上限付费。Cluster 可继续通过 mounted Secret、CSI、Vault/KMS
|
||||||
|
sidecar 或后续 direct adapter 横向扩展。
|
||||||
|
|
||||||
|
## 被拒绝的替代方案
|
||||||
|
|
||||||
|
### 把每个 Env 名称写入 Cluster Task environment
|
||||||
|
|
||||||
|
拒绝。Secret 值虽不在数据库,名称仍会进入 Task 历史、execution revision、备份与 HA 副本。
|
||||||
|
|
||||||
|
### 使用保留名称的普通 Secret binding 并让 Worker 猜 JSON
|
||||||
|
|
||||||
|
拒绝。payload sniffing 会让普通 Secret 意外获得 bundle 语义,也无法在 authority 层区分
|
||||||
|
16 KiB 与 96 KiB 的不同预算。
|
||||||
|
|
||||||
|
### 把 bundle 放进 ConfigMap、Pod env 或 Job command
|
||||||
|
|
||||||
|
拒绝。它把 Secret material 交给 Kubernetes API、etcd、进程元数据或命令审计面,并破坏
|
||||||
|
一次性、Attempt-bound 的 remote delivery fence。
|
||||||
|
|
||||||
|
### 为 bundle 单独新增服务或 workspace package
|
||||||
|
|
||||||
|
拒绝。它没有独立部署责任;新 endpoint/package 会增加低配设备与集群的维护面,现有 Secret
|
||||||
|
delivery authority 已能安全承载 typed payload。
|
||||||
|
|
||||||
|
## 当前验证与后续门禁
|
||||||
|
|
||||||
|
当前实现已完成纯 bundle contract、Task/Cluster execution optional ref、v2 wire、PostgreSQL
|
||||||
|
exact ref-set authority、mounted-files typed budget、Worker HTTPS transport 与内存展开;测试覆盖
|
||||||
|
canonicalization、跨 Project/未固定 version、普通/bundle role overlap、bundle-only Offer、名称冲突、
|
||||||
|
最终环境预算及普通 Secret 仍为 16 KiB。Runtime Core `586/586`、Worker `135/135`、Cluster Control
|
||||||
|
`273 total / 271 pass / 2 conditional skip / 0 fail`、Cluster PostgreSQL
|
||||||
|
`359 total / 356 pass / 3 conditional skip / 0 fail`,18-package clean build/test 退出 0;完整 backend
|
||||||
|
为 `1568 total / 1566 pass / 2 conditional skip / 0 fail`。
|
||||||
|
|
||||||
|
package boundary、精确 Cluster dependency、122-module Edge import、service-manager bridge import、
|
||||||
|
Local image 与基础 Edge/Standalone artifact audit 全部 compatible;workspace 保持 18 packages、
|
||||||
|
`singleSourcePackages=[]`、`shallowSourcePackages=[]`。基础 Edge/Standalone 制品为
|
||||||
|
`2,650,564 / 2,650,642 bytes`、324 files、58 loaded modules,峰值 RSS 增量分别为
|
||||||
|
`11,190,272 / 11,173,888 bytes`,没有把 Cluster provider/transport 带入低配设备默认闭包。
|
||||||
|
PostgreSQL 18.6 arm64 physical HA 通过 146 gates,timeline `1→2`,报告 SHA-256 为
|
||||||
|
`0ee2199d0a52a02025bff017a07477d707353d12aefc3811f474e3775f2bb86b`。
|
||||||
|
|
||||||
|
本 ADR 只关闭 D-401 的执行数据面前置条件,不声明 Legacy Env migration 已完成。下一切片仍须
|
||||||
|
在同一个 Automation Manager SERIALIZABLE transaction 中逐项复验 Task/Trigger current head,
|
||||||
|
追加只含 `environmentBundleRef` 的 Task revision、重定向 Trigger revision,并写 content-free
|
||||||
|
receipt;随后补 direct external custody 和 PostgreSQL promotion 后 exact replay。
|
||||||
@@ -1,6 +1,7 @@
|
|||||||
// Remote Execution owns mounted Secret resolution for authenticated delivery.
|
// Remote Execution owns mounted Secret resolution for authenticated delivery.
|
||||||
import {
|
import {
|
||||||
MAX_REMOTE_SECRET_DELIVERY_TOTAL_VALUE_BYTES,
|
MAX_REMOTE_SECRET_DELIVERY_TOTAL_VALUE_BYTES,
|
||||||
|
MAX_REMOTE_ENVIRONMENT_BUNDLE_VALUE_BYTES,
|
||||||
MAX_REMOTE_SECRET_VALUE_BYTES,
|
MAX_REMOTE_SECRET_VALUE_BYTES,
|
||||||
normalizeRemoteWorkerSecretDeliveryAuthority,
|
normalizeRemoteWorkerSecretDeliveryAuthority,
|
||||||
type RemoteWorkerSecretDeliveryAuthority,
|
type RemoteWorkerSecretDeliveryAuthority,
|
||||||
@@ -82,7 +83,7 @@ export class ClusterMountedSecretProvider
|
|||||||
this.reader = new PrivateProjectedFileReader({
|
this.reader = new PrivateProjectedFileReader({
|
||||||
rootDirectory: options.rootDirectory,
|
rootDirectory: options.rootDirectory,
|
||||||
minimumBytes: 0,
|
minimumBytes: 0,
|
||||||
maximumBytes: MAX_REMOTE_SECRET_VALUE_BYTES,
|
maximumBytes: MAX_REMOTE_ENVIRONMENT_BUNDLE_VALUE_BYTES,
|
||||||
access: 'private_material',
|
access: 'private_material',
|
||||||
});
|
});
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
@@ -116,6 +117,7 @@ export class ClusterMountedSecretProvider
|
|||||||
const buffers: Buffer[] = [];
|
const buffers: Buffer[] = [];
|
||||||
try {
|
try {
|
||||||
const values = [];
|
const values = [];
|
||||||
|
const environmentBundles = [];
|
||||||
let totalBytes = 0;
|
let totalBytes = 0;
|
||||||
for (const secretRef of normalized.secretRefs) {
|
for (const secretRef of normalized.secretRefs) {
|
||||||
const bytes = await this.reader
|
const bytes = await this.reader
|
||||||
@@ -127,6 +129,9 @@ export class ClusterMountedSecretProvider
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
buffers.push(bytes);
|
buffers.push(bytes);
|
||||||
|
if (bytes.byteLength > MAX_REMOTE_SECRET_VALUE_BYTES) {
|
||||||
|
throw new ClusterMountedSecretProviderError('material_unavailable');
|
||||||
|
}
|
||||||
totalBytes += bytes.byteLength;
|
totalBytes += bytes.byteLength;
|
||||||
if (totalBytes > MAX_REMOTE_SECRET_DELIVERY_TOTAL_VALUE_BYTES) {
|
if (totalBytes > MAX_REMOTE_SECRET_DELIVERY_TOTAL_VALUE_BYTES) {
|
||||||
throw new ClusterMountedSecretProviderError('material_unavailable');
|
throw new ClusterMountedSecretProviderError('material_unavailable');
|
||||||
@@ -138,9 +143,27 @@ export class ClusterMountedSecretProvider
|
|||||||
}),
|
}),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
for (const secretRef of normalized.environmentBundleRefs) {
|
||||||
|
const bytes = await this.reader
|
||||||
|
.read(clusterMountedSecretFileName(secretRef))
|
||||||
|
.catch((error) => {
|
||||||
|
throw new ClusterMountedSecretProviderError(
|
||||||
|
'material_unavailable',
|
||||||
|
{ cause: error },
|
||||||
|
);
|
||||||
|
});
|
||||||
|
buffers.push(bytes);
|
||||||
|
environmentBundles.push(
|
||||||
|
Object.freeze({
|
||||||
|
secretRef,
|
||||||
|
value: secretValue(bytes),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}
|
||||||
let disposed = false;
|
let disposed = false;
|
||||||
return Object.freeze({
|
return Object.freeze({
|
||||||
values: Object.freeze(values),
|
values: Object.freeze(values),
|
||||||
|
environmentBundles: Object.freeze(environmentBundles),
|
||||||
dispose() {
|
dispose() {
|
||||||
if (disposed) return;
|
if (disposed) return;
|
||||||
disposed = true;
|
disposed = true;
|
||||||
|
|||||||
+11
-3
@@ -66,7 +66,9 @@ export class ClusterRemoteWorkerSecretDeliveryService {
|
|||||||
authorized.offerId !== command.offerId ||
|
authorized.offerId !== command.offerId ||
|
||||||
authorized.leaseGeneration !== command.leaseGeneration ||
|
authorized.leaseGeneration !== command.leaseGeneration ||
|
||||||
authorized.leaseVersion !== command.expectedLeaseVersion ||
|
authorized.leaseVersion !== command.expectedLeaseVersion ||
|
||||||
JSON.stringify(authorized.secretRefs) !== JSON.stringify(command.secretRefs)
|
JSON.stringify(authorized.secretRefs) !== JSON.stringify(command.secretRefs) ||
|
||||||
|
JSON.stringify(authorized.environmentBundleRefs) !==
|
||||||
|
JSON.stringify(command.environmentBundleRefs)
|
||||||
) throw new InvalidRemoteWorkerSecretDeliveryError(
|
) throw new InvalidRemoteWorkerSecretDeliveryError(
|
||||||
'repository authority does not match command',
|
'repository authority does not match command',
|
||||||
);
|
);
|
||||||
@@ -88,7 +90,8 @@ export class ClusterRemoteWorkerSecretDeliveryService {
|
|||||||
if (
|
if (
|
||||||
typeof resolution !== 'object' ||
|
typeof resolution !== 'object' ||
|
||||||
Array.isArray(resolution) ||
|
Array.isArray(resolution) ||
|
||||||
Object.keys(resolution).some((key) => key !== 'values' && key !== 'dispose') ||
|
Object.keys(resolution).some((key) =>
|
||||||
|
key !== 'values' && key !== 'environmentBundles' && key !== 'dispose') ||
|
||||||
(resolution.dispose !== undefined &&
|
(resolution.dispose !== undefined &&
|
||||||
typeof resolution.dispose !== 'function')
|
typeof resolution.dispose !== 'function')
|
||||||
) throw new InvalidRemoteWorkerSecretDeliveryError(
|
) throw new InvalidRemoteWorkerSecretDeliveryError(
|
||||||
@@ -100,13 +103,18 @@ export class ClusterRemoteWorkerSecretDeliveryService {
|
|||||||
offerId: authorized.offerId,
|
offerId: authorized.offerId,
|
||||||
executionDigest: authorized.executionDigest,
|
executionDigest: authorized.executionDigest,
|
||||||
values: resolution.values,
|
values: resolution.values,
|
||||||
}, authorized.secretRefs);
|
environmentBundles: resolution.environmentBundles,
|
||||||
|
}, {
|
||||||
|
secretRefs: authorized.secretRefs,
|
||||||
|
environmentBundleRefs: authorized.environmentBundleRefs,
|
||||||
|
});
|
||||||
return Object.freeze({
|
return Object.freeze({
|
||||||
runId: body.runId,
|
runId: body.runId,
|
||||||
attemptId: body.attemptId,
|
attemptId: body.attemptId,
|
||||||
offerId: body.offerId,
|
offerId: body.offerId,
|
||||||
executionDigest: body.executionDigest,
|
executionDigest: body.executionDigest,
|
||||||
values: body.values,
|
values: body.values,
|
||||||
|
environmentBundles: body.environmentBundles,
|
||||||
...(resolution.dispose === undefined
|
...(resolution.dispose === undefined
|
||||||
? {}
|
? {}
|
||||||
: { dispose: resolution.dispose }),
|
: { dispose: resolution.dispose }),
|
||||||
|
|||||||
@@ -519,9 +519,9 @@ export function loadClusterWorkerIngressConfig(
|
|||||||
maxResponseBytes: integerValue(
|
maxResponseBytes: integerValue(
|
||||||
environment,
|
environment,
|
||||||
'QL3_WORKER_INGRESS_MAX_RESPONSE_BYTES',
|
'QL3_WORKER_INGRESS_MAX_RESPONSE_BYTES',
|
||||||
64 * 1024,
|
256 * 1024,
|
||||||
1024,
|
1024,
|
||||||
64 * 1024,
|
256 * 1024,
|
||||||
),
|
),
|
||||||
maxInFlightRequests: integerValue(
|
maxInFlightRequests: integerValue(
|
||||||
environment,
|
environment,
|
||||||
|
|||||||
@@ -465,7 +465,7 @@ export function createWorkerIngressAdmissionPipeline(
|
|||||||
'schema', 'runId', 'attemptId', 'projectId', 'taskId',
|
'schema', 'runId', 'attemptId', 'projectId', 'taskId',
|
||||||
'taskRevision', 'executionDigest', 'workerGeneration',
|
'taskRevision', 'executionDigest', 'workerGeneration',
|
||||||
'offerId', 'leaseGeneration', 'leaseToken',
|
'offerId', 'leaseGeneration', 'leaseToken',
|
||||||
'expectedLeaseVersion', 'secretRefs',
|
'expectedLeaseVersion', 'secretRefs', 'environmentBundleRefs',
|
||||||
]);
|
]);
|
||||||
if (value.schema !== REMOTE_SECRET_DELIVERY_SCHEMA) {
|
if (value.schema !== REMOTE_SECRET_DELIVERY_SCHEMA) {
|
||||||
throw failure(400, 'invalid_worker_request');
|
throw failure(400, 'invalid_worker_request');
|
||||||
@@ -486,12 +486,16 @@ export function createWorkerIngressAdmissionPipeline(
|
|||||||
leaseToken: value.leaseToken as string,
|
leaseToken: value.leaseToken as string,
|
||||||
expectedLeaseVersion: value.expectedLeaseVersion as number,
|
expectedLeaseVersion: value.expectedLeaseVersion as number,
|
||||||
secretRefs: value.secretRefs as string[],
|
secretRefs: value.secretRefs as string[],
|
||||||
|
environmentBundleRefs: value.environmentBundleRefs as string[],
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
try {
|
try {
|
||||||
const responseBody = createRemoteWorkerSecretDeliveryResponseBody(
|
const responseBody = createRemoteWorkerSecretDeliveryResponseBody(
|
||||||
delivered,
|
delivered,
|
||||||
value.secretRefs as string[],
|
{
|
||||||
|
secretRefs: value.secretRefs as string[],
|
||||||
|
environmentBundleRefs: value.environmentBundleRefs as string[],
|
||||||
|
},
|
||||||
);
|
);
|
||||||
if (
|
if (
|
||||||
responseBody.runId !== value.runId ||
|
responseBody.runId !== value.runId ||
|
||||||
|
|||||||
@@ -30,6 +30,11 @@ const VERSIONED_SECRET_REF = createSecretRef({
|
|||||||
name: 'certificate',
|
name: 'certificate',
|
||||||
version: 3,
|
version: 3,
|
||||||
});
|
});
|
||||||
|
const ENVIRONMENT_BUNDLE_REF = createSecretRef({
|
||||||
|
projectId: 'project-1',
|
||||||
|
name: 'legacy-env-bundle',
|
||||||
|
version: 4,
|
||||||
|
});
|
||||||
|
|
||||||
function authority(secretRefs = [SECRET_REF]) {
|
function authority(secretRefs = [SECRET_REF]) {
|
||||||
return {
|
return {
|
||||||
@@ -46,6 +51,7 @@ function authority(secretRefs = [SECRET_REF]) {
|
|||||||
leaseGeneration: 1,
|
leaseGeneration: 1,
|
||||||
leaseVersion: 1,
|
leaseVersion: 1,
|
||||||
secretRefs,
|
secretRefs,
|
||||||
|
environmentBundleRefs: [],
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -58,10 +64,7 @@ test('maps canonical SecretRef to a stable path-free Kubernetes key', () => {
|
|||||||
const first = clusterMountedSecretFileName(SECRET_REF);
|
const first = clusterMountedSecretFileName(SECRET_REF);
|
||||||
assert.match(first, /^[0-9a-f]{64}$/);
|
assert.match(first, /^[0-9a-f]{64}$/);
|
||||||
assert.equal(clusterMountedSecretFileName(SECRET_REF), first);
|
assert.equal(clusterMountedSecretFileName(SECRET_REF), first);
|
||||||
assert.notEqual(
|
assert.notEqual(clusterMountedSecretFileName(VERSIONED_SECRET_REF), first);
|
||||||
clusterMountedSecretFileName(VERSIONED_SECRET_REF),
|
|
||||||
first,
|
|
||||||
);
|
|
||||||
assert.throws(
|
assert.throws(
|
||||||
() => clusterMountedSecretFileName('not-a-secret-ref'),
|
() => clusterMountedSecretFileName('not-a-secret-ref'),
|
||||||
ClusterMountedSecretProviderError,
|
ClusterMountedSecretProviderError,
|
||||||
@@ -82,6 +85,7 @@ test('resolves every request again and observes atomic material rotation', async
|
|||||||
assert.deepEqual(first.values, [
|
assert.deepEqual(first.values, [
|
||||||
{ secretRef: SECRET_REF, value: 'generation-one' },
|
{ secretRef: SECRET_REF, value: 'generation-one' },
|
||||||
]);
|
]);
|
||||||
|
assert.deepEqual(first.environmentBundles, []);
|
||||||
await first.dispose();
|
await first.dispose();
|
||||||
|
|
||||||
const replacement = `${file}.replacement`;
|
const replacement = `${file}.replacement`;
|
||||||
@@ -155,3 +159,35 @@ test('fails readiness for a missing or symlinked provider root', async (t) => {
|
|||||||
ClusterMountedSecretProviderError,
|
ClusterMountedSecretProviderError,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('delivers one larger opaque environment bundle without widening normal Secrets', async (t) => {
|
||||||
|
const root = await mkdtemp(path.join(os.tmpdir(), 'ql3-mounted-bundle-'));
|
||||||
|
t.after(() => rm(root, { recursive: true, force: true }));
|
||||||
|
await chmod(root, 0o700);
|
||||||
|
const value = JSON.stringify({
|
||||||
|
schema: 'qinglong/environment-bundle@v1',
|
||||||
|
entries: [{ name: 'LEGACY_VALUE', value: 'x'.repeat(20 * 1024) }],
|
||||||
|
});
|
||||||
|
await privateFile(
|
||||||
|
path.join(root, clusterMountedSecretFileName(ENVIRONMENT_BUNDLE_REF)),
|
||||||
|
value,
|
||||||
|
);
|
||||||
|
const provider = await createClusterMountedSecretProvider({
|
||||||
|
rootDirectory: root,
|
||||||
|
});
|
||||||
|
const resolution = await provider.resolve({
|
||||||
|
...authority([]),
|
||||||
|
environmentBundleRefs: [ENVIRONMENT_BUNDLE_REF],
|
||||||
|
});
|
||||||
|
assert.deepEqual(resolution.values, []);
|
||||||
|
assert.deepEqual(resolution.environmentBundles, [
|
||||||
|
{ secretRef: ENVIRONMENT_BUNDLE_REF, value },
|
||||||
|
]);
|
||||||
|
await resolution.dispose();
|
||||||
|
|
||||||
|
await privateFile(
|
||||||
|
path.join(root, clusterMountedSecretFileName(SECRET_REF)),
|
||||||
|
'x'.repeat(16 * 1024 + 1),
|
||||||
|
);
|
||||||
|
await assert.rejects(provider.resolve(authority()), /material_unavailable/);
|
||||||
|
});
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ function command() {
|
|||||||
taskId: 'task-1', taskRevision: 'revision-1', executionDigest: DIGEST,
|
taskId: 'task-1', taskRevision: 'revision-1', executionDigest: DIGEST,
|
||||||
offerId: 'offer-1', leaseGeneration: 3,
|
offerId: 'offer-1', leaseGeneration: 3,
|
||||||
leaseToken: 'worker_generated_lease_capability_0000000000000001',
|
leaseToken: 'worker_generated_lease_capability_0000000000000001',
|
||||||
expectedLeaseVersion: 4, secretRefs: [SECRET_REF],
|
expectedLeaseVersion: 4, secretRefs: [SECRET_REF], environmentBundleRefs: [],
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -47,6 +47,7 @@ test('resolves plaintext only after repository authority succeeds', async () =>
|
|||||||
assert.equal('leaseToken' in input, false);
|
assert.equal('leaseToken' in input, false);
|
||||||
return {
|
return {
|
||||||
values: [{ secretRef: SECRET_REF, value: 'resolved-value' }],
|
values: [{ secretRef: SECRET_REF, value: 'resolved-value' }],
|
||||||
|
environmentBundles: [],
|
||||||
dispose() { events.push('dispose'); },
|
dispose() { events.push('dispose'); },
|
||||||
};
|
};
|
||||||
},
|
},
|
||||||
@@ -56,6 +57,7 @@ test('resolves plaintext only after repository authority succeeds', async () =>
|
|||||||
assert.deepEqual(result.values, [
|
assert.deepEqual(result.values, [
|
||||||
{ secretRef: SECRET_REF, value: 'resolved-value' },
|
{ secretRef: SECRET_REF, value: 'resolved-value' },
|
||||||
]);
|
]);
|
||||||
|
assert.deepEqual(result.environmentBundles, []);
|
||||||
assert.deepEqual(events, ['authorize', 'resolve']);
|
assert.deepEqual(events, ['authorize', 'resolve']);
|
||||||
await result.dispose();
|
await result.dispose();
|
||||||
assert.deepEqual(events, ['authorize', 'resolve', 'dispose']);
|
assert.deepEqual(events, ['authorize', 'resolve', 'dispose']);
|
||||||
@@ -101,6 +103,7 @@ test('disposes malformed provider output and converts it to unavailable', async
|
|||||||
async resolve() {
|
async resolve() {
|
||||||
return {
|
return {
|
||||||
values: [{ secretRef: SECRET_REF, value: 'x'.repeat(17 * 1024) }],
|
values: [{ secretRef: SECRET_REF, value: 'x'.repeat(17 * 1024) }],
|
||||||
|
environmentBundles: [],
|
||||||
dispose() { disposed += 1; },
|
dispose() { disposed += 1; },
|
||||||
};
|
};
|
||||||
},
|
},
|
||||||
@@ -120,6 +123,7 @@ test('rejects extensible provider output and still invokes valid cleanup', async
|
|||||||
async resolve() {
|
async resolve() {
|
||||||
return {
|
return {
|
||||||
values: [{ secretRef: SECRET_REF, value: 'resolved-value' }],
|
values: [{ secretRef: SECRET_REF, value: 'resolved-value' }],
|
||||||
|
environmentBundles: [],
|
||||||
dispose() { disposed += 1; },
|
dispose() { disposed += 1; },
|
||||||
diagnostic: 'must-not-cross-boundary',
|
diagnostic: 'must-not-cross-boundary',
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -93,7 +93,7 @@ test('builds exact bounded Worker ingress and least-privilege Pool config', asyn
|
|||||||
host: '127.0.0.1',
|
host: '127.0.0.1',
|
||||||
port: 5901,
|
port: 5901,
|
||||||
maxBodyBytes: 65_536,
|
maxBodyBytes: 65_536,
|
||||||
maxResponseBytes: 65_536,
|
maxResponseBytes: 262_144,
|
||||||
maxInFlightRequests: 32,
|
maxInFlightRequests: 32,
|
||||||
authenticationRateWindowMs: 60_000,
|
authenticationRateWindowMs: 60_000,
|
||||||
authenticationRatePerPeer: 20,
|
authenticationRatePerPeer: 20,
|
||||||
|
|||||||
@@ -592,6 +592,7 @@ test('binds one Secret batch to path identity and never echoes capabilities', as
|
|||||||
offerId: command.offerId,
|
offerId: command.offerId,
|
||||||
executionDigest: command.executionDigest,
|
executionDigest: command.executionDigest,
|
||||||
values: [{ secretRef, value: 'resolved-value' }],
|
values: [{ secretRef, value: 'resolved-value' }],
|
||||||
|
environmentBundles: [],
|
||||||
dispose() { disposed += 1; },
|
dispose() { disposed += 1; },
|
||||||
};
|
};
|
||||||
},
|
},
|
||||||
@@ -599,18 +600,20 @@ test('binds one Secret batch to path identity and never echoes capabilities', as
|
|||||||
});
|
});
|
||||||
const leaseToken = 'worker_generated_lease_capability_0000000000000001';
|
const leaseToken = 'worker_generated_lease_capability_0000000000000001';
|
||||||
const body = {
|
const body = {
|
||||||
schema: 'qinglong/remote-secret-delivery@v1',
|
schema: 'qinglong/remote-secret-delivery@v2',
|
||||||
runId: 'run-1', attemptId: 'attempt-1', projectId: 'project-1',
|
runId: 'run-1', attemptId: 'attempt-1', projectId: 'project-1',
|
||||||
taskId: 'task-1', taskRevision: 'revision-1', executionDigest,
|
taskId: 'task-1', taskRevision: 'revision-1', executionDigest,
|
||||||
workerGeneration: 2, offerId: 'offer-1', leaseGeneration: 3,
|
workerGeneration: 2, offerId: 'offer-1', leaseGeneration: 3,
|
||||||
leaseToken, expectedLeaseVersion: 4, secretRefs: [secretRef],
|
leaseToken, expectedLeaseVersion: 4, secretRefs: [secretRef],
|
||||||
|
environmentBundleRefs: [],
|
||||||
};
|
};
|
||||||
const result = await (await pipeline.prepare(metadata('secrets'))).handle(body);
|
const result = await (await pipeline.prepare(metadata('secrets'))).handle(body);
|
||||||
assert.equal(result.statusCode, 200);
|
assert.equal(result.statusCode, 200);
|
||||||
assert.equal(result.body.schema, 'qinglong/remote-secret-delivery@v1');
|
assert.equal(result.body.schema, 'qinglong/remote-secret-delivery@v2');
|
||||||
assert.deepEqual(result.body.values, [
|
assert.deepEqual(result.body.values, [
|
||||||
{ secretRef, value: 'resolved-value' },
|
{ secretRef, value: 'resolved-value' },
|
||||||
]);
|
]);
|
||||||
|
assert.deepEqual(result.body.environmentBundles, []);
|
||||||
assert.equal(JSON.stringify(result.body).includes(leaseToken), false);
|
assert.equal(JSON.stringify(result.body).includes(leaseToken), false);
|
||||||
const { schema: _schema, ...commandBody } = body;
|
const { schema: _schema, ...commandBody } = body;
|
||||||
assert.deepEqual(observed, {
|
assert.deepEqual(observed, {
|
||||||
@@ -634,13 +637,14 @@ test('maps stale Secret delivery authority to conflict before any response', asy
|
|||||||
});
|
});
|
||||||
await assert.rejects(
|
await assert.rejects(
|
||||||
(await pipeline.prepare(metadata('secrets'))).handle({
|
(await pipeline.prepare(metadata('secrets'))).handle({
|
||||||
schema: 'qinglong/remote-secret-delivery@v1',
|
schema: 'qinglong/remote-secret-delivery@v2',
|
||||||
runId: 'run-1', attemptId: 'attempt-1', projectId: 'project-1',
|
runId: 'run-1', attemptId: 'attempt-1', projectId: 'project-1',
|
||||||
taskId: 'task-1', taskRevision: 'revision-1',
|
taskId: 'task-1', taskRevision: 'revision-1',
|
||||||
executionDigest: 'c'.repeat(64), workerGeneration: 2,
|
executionDigest: 'c'.repeat(64), workerGeneration: 2,
|
||||||
offerId: 'offer-1', leaseGeneration: 3,
|
offerId: 'offer-1', leaseGeneration: 3,
|
||||||
leaseToken: 'worker_generated_lease_capability_0000000000000001',
|
leaseToken: 'worker_generated_lease_capability_0000000000000001',
|
||||||
expectedLeaseVersion: 4, secretRefs: [secretRef],
|
expectedLeaseVersion: 4, secretRefs: [secretRef],
|
||||||
|
environmentBundleRefs: [],
|
||||||
}),
|
}),
|
||||||
(error) =>
|
(error) =>
|
||||||
error.statusCode === 409 && error.code === 'worker_secret_delivery_fenced',
|
error.statusCode === 409 && error.code === 'worker_secret_delivery_fenced',
|
||||||
@@ -656,19 +660,21 @@ test('rejects a Secret service response whose authority drifts', async () => {
|
|||||||
runId: 'run-other', attemptId: 'attempt-1', offerId: 'offer-1',
|
runId: 'run-other', attemptId: 'attempt-1', offerId: 'offer-1',
|
||||||
executionDigest: 'c'.repeat(64),
|
executionDigest: 'c'.repeat(64),
|
||||||
values: [{ secretRef, value: 'must-not-escape' }],
|
values: [{ secretRef, value: 'must-not-escape' }],
|
||||||
|
environmentBundles: [],
|
||||||
};
|
};
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
await assert.rejects(
|
await assert.rejects(
|
||||||
(await pipeline.prepare(metadata('secrets'))).handle({
|
(await pipeline.prepare(metadata('secrets'))).handle({
|
||||||
schema: 'qinglong/remote-secret-delivery@v1',
|
schema: 'qinglong/remote-secret-delivery@v2',
|
||||||
runId: 'run-1', attemptId: 'attempt-1', projectId: 'project-1',
|
runId: 'run-1', attemptId: 'attempt-1', projectId: 'project-1',
|
||||||
taskId: 'task-1', taskRevision: 'revision-1',
|
taskId: 'task-1', taskRevision: 'revision-1',
|
||||||
executionDigest: 'c'.repeat(64), workerGeneration: 2,
|
executionDigest: 'c'.repeat(64), workerGeneration: 2,
|
||||||
offerId: 'offer-1', leaseGeneration: 3,
|
offerId: 'offer-1', leaseGeneration: 3,
|
||||||
leaseToken: 'worker_generated_lease_capability_0000000000000001',
|
leaseToken: 'worker_generated_lease_capability_0000000000000001',
|
||||||
expectedLeaseVersion: 4, secretRefs: [secretRef],
|
expectedLeaseVersion: 4, secretRefs: [secretRef],
|
||||||
|
environmentBundleRefs: [],
|
||||||
}),
|
}),
|
||||||
(error) =>
|
(error) =>
|
||||||
error.statusCode === 503 && error.code === 'worker_ingress_unavailable',
|
error.statusCode === 503 && error.code === 'worker_ingress_unavailable',
|
||||||
|
|||||||
@@ -77,9 +77,7 @@ function unavailable(): TaskDefinitionUnavailableError {
|
|||||||
return new TaskDefinitionUnavailableError();
|
return new TaskDefinitionUnavailableError();
|
||||||
}
|
}
|
||||||
|
|
||||||
function taskDefinitionRecord(
|
function taskDefinitionRecord(row: TaskDefinitionRow): TaskDefinitionRecord {
|
||||||
row: TaskDefinitionRow,
|
|
||||||
): TaskDefinitionRecord {
|
|
||||||
try {
|
try {
|
||||||
const description = row.description;
|
const description = row.description;
|
||||||
if (description !== null && typeof description !== 'string') {
|
if (description !== null && typeof description !== 'string') {
|
||||||
@@ -128,6 +126,9 @@ function executionPlanJson(
|
|||||||
return Object.freeze({
|
return Object.freeze({
|
||||||
command: revision.command,
|
command: revision.command,
|
||||||
environment: revision.environment,
|
environment: revision.environment,
|
||||||
|
...(revision.environmentBundleRef === undefined
|
||||||
|
? {}
|
||||||
|
: { environmentBundleRef: revision.environmentBundleRef }),
|
||||||
...(revision.workingDirectory === undefined
|
...(revision.workingDirectory === undefined
|
||||||
? {}
|
? {}
|
||||||
: { workingDirectory: revision.workingDirectory }),
|
: { workingDirectory: revision.workingDirectory }),
|
||||||
@@ -140,7 +141,9 @@ function executionPlanJson(
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
function executionRevision(row: TaskDefinitionRow): ClusterTaskExecutionRevision {
|
function executionRevision(
|
||||||
|
row: TaskDefinitionRow,
|
||||||
|
): ClusterTaskExecutionRevision {
|
||||||
try {
|
try {
|
||||||
const plan = postgresRequiredJsonObject(row.planJson, unavailable);
|
const plan = postgresRequiredJsonObject(row.planJson, unavailable);
|
||||||
const keys = Object.keys(plan);
|
const keys = Object.keys(plan);
|
||||||
@@ -149,9 +152,14 @@ function executionRevision(row: TaskDefinitionRow): ClusterTaskExecutionRevision
|
|||||||
!keys.includes('environment') ||
|
!keys.includes('environment') ||
|
||||||
keys.some(
|
keys.some(
|
||||||
(key) =>
|
(key) =>
|
||||||
!['command', 'environment', 'placement', 'timeoutMs', 'workingDirectory'].includes(
|
![
|
||||||
key,
|
'command',
|
||||||
),
|
'environment',
|
||||||
|
'environmentBundleRef',
|
||||||
|
'placement',
|
||||||
|
'timeoutMs',
|
||||||
|
'workingDirectory',
|
||||||
|
].includes(key),
|
||||||
)
|
)
|
||||||
) {
|
) {
|
||||||
throw unavailable();
|
throw unavailable();
|
||||||
@@ -176,6 +184,9 @@ function executionRevision(row: TaskDefinitionRow): ClusterTaskExecutionRevision
|
|||||||
command: plan.command as ClusterTaskExecutionRevision['command'],
|
command: plan.command as ClusterTaskExecutionRevision['command'],
|
||||||
environment:
|
environment:
|
||||||
plan.environment as ClusterTaskExecutionRevision['environment'],
|
plan.environment as ClusterTaskExecutionRevision['environment'],
|
||||||
|
...(plan.environmentBundleRef === undefined
|
||||||
|
? {}
|
||||||
|
: { environmentBundleRef: plan.environmentBundleRef as string }),
|
||||||
...(plan.workingDirectory === undefined
|
...(plan.workingDirectory === undefined
|
||||||
? {}
|
? {}
|
||||||
: { workingDirectory: plan.workingDirectory as string }),
|
: { workingDirectory: plan.workingDirectory as string }),
|
||||||
@@ -185,10 +196,9 @@ function executionRevision(row: TaskDefinitionRow): ClusterTaskExecutionRevision
|
|||||||
...(plan.placement === undefined
|
...(plan.placement === undefined
|
||||||
? {}
|
? {}
|
||||||
: {
|
: {
|
||||||
placement:
|
placement: plan.placement as unknown as NonNullable<
|
||||||
plan.placement as unknown as NonNullable<
|
ClusterTaskExecutionRevision['placement']
|
||||||
ClusterTaskExecutionRevision['placement']
|
>,
|
||||||
>,
|
|
||||||
}),
|
}),
|
||||||
contentDigest: postgresRequiredString(row.contentDigest, unavailable),
|
contentDigest: postgresRequiredString(row.contentDigest, unavailable),
|
||||||
createdAtMs: postgresRequiredInteger(row.createdAtMs, unavailable),
|
createdAtMs: postgresRequiredInteger(row.createdAtMs, unavailable),
|
||||||
@@ -319,7 +329,6 @@ export class PostgresTaskDefinitionSource implements TaskDefinitionSource {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
async findCurrentTaskDefinition(
|
async findCurrentTaskDefinition(
|
||||||
projectId: string,
|
projectId: string,
|
||||||
taskId: string,
|
taskId: string,
|
||||||
@@ -422,7 +431,9 @@ export class PostgresTaskExecutionRevisionSource
|
|||||||
readonly sourceRevision: number;
|
readonly sourceRevision: number;
|
||||||
}): Promise<ClusterTaskExecutionRevision | null> {
|
}): Promise<ClusterTaskExecutionRevision | null> {
|
||||||
if (!identity || typeof identity !== 'object' || Array.isArray(identity)) {
|
if (!identity || typeof identity !== 'object' || Array.isArray(identity)) {
|
||||||
throw new TypeError('Cluster Task execution revision identity is invalid');
|
throw new TypeError(
|
||||||
|
'Cluster Task execution revision identity is invalid',
|
||||||
|
);
|
||||||
}
|
}
|
||||||
assertTaskDefinitionIdentifier(identity.projectId, 'projectId');
|
assertTaskDefinitionIdentifier(identity.projectId, 'projectId');
|
||||||
assertTaskDefinitionIdentifier(identity.taskId, 'taskId');
|
assertTaskDefinitionIdentifier(identity.taskId, 'taskId');
|
||||||
@@ -447,8 +458,7 @@ export class PostgresTaskDefinitionRepository
|
|||||||
{
|
{
|
||||||
constructor(
|
constructor(
|
||||||
pool: PostgresPool,
|
pool: PostgresPool,
|
||||||
private readonly semanticRegistry: TaskSpecSemanticRegistry =
|
private readonly semanticRegistry: TaskSpecSemanticRegistry = createBuiltInTaskSpecSemanticRegistry(),
|
||||||
createBuiltInTaskSpecSemanticRegistry(),
|
|
||||||
) {
|
) {
|
||||||
super(pool);
|
super(pool);
|
||||||
}
|
}
|
||||||
@@ -483,10 +493,7 @@ export class PostgresTaskDefinitionRepository
|
|||||||
command.kind === 'command' &&
|
command.kind === 'command' &&
|
||||||
command.spec.schema === BUILT_IN_COMMAND_TASK_SPEC_SCHEMA
|
command.spec.schema === BUILT_IN_COMMAND_TASK_SPEC_SCHEMA
|
||||||
? compileClusterCommandTaskDefinition(
|
? compileClusterCommandTaskDefinition(
|
||||||
createTaskDefinitionRecord(
|
createTaskDefinitionRecord(command, command.occurredAtMs),
|
||||||
command,
|
|
||||||
command.occurredAtMs,
|
|
||||||
),
|
|
||||||
this.semanticRegistry,
|
this.semanticRegistry,
|
||||||
)
|
)
|
||||||
: null;
|
: null;
|
||||||
@@ -560,10 +567,7 @@ export class PostgresTaskDefinitionRepository
|
|||||||
WHERE id = $1`,
|
WHERE id = $1`,
|
||||||
[command.projectId],
|
[command.projectId],
|
||||||
);
|
);
|
||||||
if (
|
if (project.rows.length !== 1 || project.rows[0]?.status !== 'active') {
|
||||||
project.rows.length !== 1 ||
|
|
||||||
project.rows[0]?.status !== 'active'
|
|
||||||
) {
|
|
||||||
throw new TaskDefinitionConflictError();
|
throw new TaskDefinitionConflictError();
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -600,8 +604,9 @@ export class PostgresTaskDefinitionRepository
|
|||||||
unavailable,
|
unavailable,
|
||||||
);
|
);
|
||||||
if (
|
if (
|
||||||
(created ? command.expectedRevision !== null :
|
(created
|
||||||
currentRevision !== command.expectedRevision) ||
|
? command.expectedRevision !== null
|
||||||
|
: currentRevision !== command.expectedRevision) ||
|
||||||
command.occurredAtMs < previousUpdatedAtMs
|
command.occurredAtMs < previousUpdatedAtMs
|
||||||
) {
|
) {
|
||||||
throw new TaskDefinitionConflictError();
|
throw new TaskDefinitionConflictError();
|
||||||
|
|||||||
+56
-16
@@ -27,7 +27,8 @@ function text(row: Row, key: string): string {
|
|||||||
|
|
||||||
function integer(row: Row, key: string): number {
|
function integer(row: Row, key: string): number {
|
||||||
const raw = row[key];
|
const raw = row[key];
|
||||||
const value = typeof raw === 'string' && /^\d+$/.test(raw) ? Number(raw) : raw;
|
const value =
|
||||||
|
typeof raw === 'string' && /^\d+$/.test(raw) ? Number(raw) : raw;
|
||||||
if (typeof value !== 'number' || !Number.isSafeInteger(value)) {
|
if (typeof value !== 'number' || !Number.isSafeInteger(value)) {
|
||||||
throw new RemoteWorkerSecretDeliveryUnavailableError();
|
throw new RemoteWorkerSecretDeliveryUnavailableError();
|
||||||
}
|
}
|
||||||
@@ -44,10 +45,19 @@ function executionRevision(row: Row): ClusterTaskExecutionRevision {
|
|||||||
if (
|
if (
|
||||||
!keys.includes('command') ||
|
!keys.includes('command') ||
|
||||||
!keys.includes('environment') ||
|
!keys.includes('environment') ||
|
||||||
keys.some((key) =>
|
keys.some(
|
||||||
!['command', 'environment', 'placement', 'timeoutMs', 'workingDirectory']
|
(key) =>
|
||||||
.includes(key))
|
![
|
||||||
) throw new RemoteWorkerSecretDeliveryUnavailableError();
|
'command',
|
||||||
|
'environment',
|
||||||
|
'environmentBundleRef',
|
||||||
|
'placement',
|
||||||
|
'timeoutMs',
|
||||||
|
'workingDirectory',
|
||||||
|
].includes(key),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
throw new RemoteWorkerSecretDeliveryUnavailableError();
|
||||||
return normalizeClusterTaskExecutionRevision({
|
return normalizeClusterTaskExecutionRevision({
|
||||||
projectId: text(row, 'revisionProjectId'),
|
projectId: text(row, 'revisionProjectId'),
|
||||||
taskId: text(row, 'revisionTaskId'),
|
taskId: text(row, 'revisionTaskId'),
|
||||||
@@ -57,7 +67,11 @@ function executionRevision(row: Row): ClusterTaskExecutionRevision {
|
|||||||
executorType: text(row, 'revisionExecutorType') as 'remote_worker',
|
executorType: text(row, 'revisionExecutorType') as 'remote_worker',
|
||||||
planSchema: text(row, 'planSchema') as 'qinglong/command-execution@v1',
|
planSchema: text(row, 'planSchema') as 'qinglong/command-execution@v1',
|
||||||
command: value.command as ClusterTaskExecutionRevision['command'],
|
command: value.command as ClusterTaskExecutionRevision['command'],
|
||||||
environment: value.environment as ClusterTaskExecutionRevision['environment'],
|
environment:
|
||||||
|
value.environment as ClusterTaskExecutionRevision['environment'],
|
||||||
|
...(value.environmentBundleRef === undefined
|
||||||
|
? {}
|
||||||
|
: { environmentBundleRef: value.environmentBundleRef as string }),
|
||||||
...(value.workingDirectory === undefined
|
...(value.workingDirectory === undefined
|
||||||
? {}
|
? {}
|
||||||
: { workingDirectory: value.workingDirectory as string }),
|
: { workingDirectory: value.workingDirectory as string }),
|
||||||
@@ -66,7 +80,11 @@ function executionRevision(row: Row): ClusterTaskExecutionRevision {
|
|||||||
: { timeoutMs: value.timeoutMs as number }),
|
: { timeoutMs: value.timeoutMs as number }),
|
||||||
...(value.placement === undefined
|
...(value.placement === undefined
|
||||||
? {}
|
? {}
|
||||||
: { placement: value.placement as NonNullable<ClusterTaskExecutionRevision['placement']> }),
|
: {
|
||||||
|
placement: value.placement as NonNullable<
|
||||||
|
ClusterTaskExecutionRevision['placement']
|
||||||
|
>,
|
||||||
|
}),
|
||||||
contentDigest: text(row, 'revisionContentDigest'),
|
contentDigest: text(row, 'revisionContentDigest'),
|
||||||
createdAtMs: integer(row, 'revisionCreatedAtMs'),
|
createdAtMs: integer(row, 'revisionCreatedAtMs'),
|
||||||
});
|
});
|
||||||
@@ -80,7 +98,8 @@ async function begin(client: PostgresClient): Promise<void> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export class PostgresRemoteWorkerSecretDeliveryAuthorityRepository
|
export class PostgresRemoteWorkerSecretDeliveryAuthorityRepository
|
||||||
implements RemoteWorkerSecretDeliveryAuthorityRepository {
|
implements RemoteWorkerSecretDeliveryAuthorityRepository
|
||||||
|
{
|
||||||
constructor(private readonly pool: PostgresPool) {
|
constructor(private readonly pool: PostgresPool) {
|
||||||
if (!pool || typeof pool.connect !== 'function') {
|
if (!pool || typeof pool.connect !== 'function') {
|
||||||
throw new TypeError('PostgreSQL remote Secret delivery pool is invalid');
|
throw new TypeError('PostgreSQL remote Secret delivery pool is invalid');
|
||||||
@@ -156,7 +175,9 @@ export class PostgresRemoteWorkerSecretDeliveryAuthorityRepository
|
|||||||
}
|
}
|
||||||
const row = result.rows[0];
|
const row = result.rows[0];
|
||||||
if (!row) {
|
if (!row) {
|
||||||
throw new RemoteWorkerSecretDeliveryFenceRejectedError('authority_mismatch');
|
throw new RemoteWorkerSecretDeliveryFenceRejectedError(
|
||||||
|
'authority_mismatch',
|
||||||
|
);
|
||||||
}
|
}
|
||||||
const observedAtMs = integer(row, 'observedAtMs');
|
const observedAtMs = integer(row, 'observedAtMs');
|
||||||
const tokenDigest = digestRunDispatchLeaseToken(command.leaseToken);
|
const tokenDigest = digestRunDispatchLeaseToken(command.leaseToken);
|
||||||
@@ -192,25 +213,38 @@ export class PostgresRemoteWorkerSecretDeliveryAuthorityRepository
|
|||||||
row.leaseOfferId === command.offerId &&
|
row.leaseOfferId === command.offerId &&
|
||||||
integer(row, 'leaseExpiresAtMs') > observedAtMs;
|
integer(row, 'leaseExpiresAtMs') > observedAtMs;
|
||||||
if (!matches) {
|
if (!matches) {
|
||||||
throw new RemoteWorkerSecretDeliveryFenceRejectedError('authority_mismatch');
|
throw new RemoteWorkerSecretDeliveryFenceRejectedError(
|
||||||
|
'authority_mismatch',
|
||||||
|
);
|
||||||
}
|
}
|
||||||
let revision: ClusterTaskExecutionRevision;
|
let revision: ClusterTaskExecutionRevision;
|
||||||
try {
|
try {
|
||||||
revision = executionRevision(row);
|
revision = executionRevision(row);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
if (error instanceof RemoteWorkerSecretDeliveryUnavailableError) throw error;
|
if (error instanceof RemoteWorkerSecretDeliveryUnavailableError)
|
||||||
|
throw error;
|
||||||
throw new RemoteWorkerSecretDeliveryUnavailableError();
|
throw new RemoteWorkerSecretDeliveryUnavailableError();
|
||||||
}
|
}
|
||||||
const expectedRefs = Object.freeze([
|
const expectedRefs = Object.freeze([
|
||||||
...new Set(revision.environment.flatMap((binding) =>
|
...new Set(
|
||||||
binding.kind === 'secret' ? [binding.secretRef] : [])),
|
revision.environment.flatMap((binding) =>
|
||||||
|
binding.kind === 'secret' ? [binding.secretRef] : [],
|
||||||
|
),
|
||||||
|
),
|
||||||
]);
|
]);
|
||||||
|
const expectedEnvironmentBundleRefs = Object.freeze(
|
||||||
|
revision.environmentBundleRef === undefined
|
||||||
|
? []
|
||||||
|
: [revision.environmentBundleRef],
|
||||||
|
);
|
||||||
if (
|
if (
|
||||||
revision.projectId !== command.projectId ||
|
revision.projectId !== command.projectId ||
|
||||||
revision.taskId !== command.taskId ||
|
revision.taskId !== command.taskId ||
|
||||||
revision.taskRevision !== command.taskRevision ||
|
revision.taskRevision !== command.taskRevision ||
|
||||||
revision.contentDigest !== command.executionDigest ||
|
revision.contentDigest !== command.executionDigest ||
|
||||||
JSON.stringify(expectedRefs) !== JSON.stringify(command.secretRefs)
|
JSON.stringify(expectedRefs) !== JSON.stringify(command.secretRefs) ||
|
||||||
|
JSON.stringify(expectedEnvironmentBundleRefs) !==
|
||||||
|
JSON.stringify(command.environmentBundleRefs)
|
||||||
) {
|
) {
|
||||||
throw new RemoteWorkerSecretDeliveryFenceRejectedError(
|
throw new RemoteWorkerSecretDeliveryFenceRejectedError(
|
||||||
'secret_scope_mismatch',
|
'secret_scope_mismatch',
|
||||||
@@ -230,15 +264,21 @@ export class PostgresRemoteWorkerSecretDeliveryAuthorityRepository
|
|||||||
leaseGeneration: command.leaseGeneration,
|
leaseGeneration: command.leaseGeneration,
|
||||||
leaseVersion: command.expectedLeaseVersion,
|
leaseVersion: command.expectedLeaseVersion,
|
||||||
secretRefs: expectedRefs,
|
secretRefs: expectedRefs,
|
||||||
|
environmentBundleRefs: expectedEnvironmentBundleRefs,
|
||||||
});
|
});
|
||||||
await client.query('COMMIT');
|
await client.query('COMMIT');
|
||||||
return authority;
|
return authority;
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
try { await client.query('ROLLBACK'); } catch { /* preserve root */ }
|
try {
|
||||||
|
await client.query('ROLLBACK');
|
||||||
|
} catch {
|
||||||
|
/* preserve root */
|
||||||
|
}
|
||||||
if (
|
if (
|
||||||
error instanceof RemoteWorkerSecretDeliveryFenceRejectedError ||
|
error instanceof RemoteWorkerSecretDeliveryFenceRejectedError ||
|
||||||
error instanceof RemoteWorkerSecretDeliveryUnavailableError
|
error instanceof RemoteWorkerSecretDeliveryUnavailableError
|
||||||
) throw error;
|
)
|
||||||
|
throw error;
|
||||||
throw new RemoteWorkerSecretDeliveryUnavailableError();
|
throw new RemoteWorkerSecretDeliveryUnavailableError();
|
||||||
} finally {
|
} finally {
|
||||||
client.release();
|
client.release();
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ const {
|
|||||||
const {
|
const {
|
||||||
compileClusterCommandTaskDefinition,
|
compileClusterCommandTaskDefinition,
|
||||||
} = require('@qinglong/runtime-core/cluster-execution-revision');
|
} = require('@qinglong/runtime-core/cluster-execution-revision');
|
||||||
|
const { createSecretRef } = require('@qinglong/runtime-core/secret-reference');
|
||||||
const {
|
const {
|
||||||
TaskDefinitionAdministrationAuthorizationFenceConflictError,
|
TaskDefinitionAdministrationAuthorizationFenceConflictError,
|
||||||
TaskDefinitionAdministrationMutationConflictError,
|
TaskDefinitionAdministrationMutationConflictError,
|
||||||
@@ -447,6 +448,34 @@ test('publishes TaskDefinition atomically and replays the exact mutation', async
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('persists only the pinned environment bundle reference in an execution plan', async () => {
|
||||||
|
const environmentBundleRef = createSecretRef({
|
||||||
|
projectId: TASK_COMMAND.projectId,
|
||||||
|
name: 'legacy-env-bundle',
|
||||||
|
version: 7,
|
||||||
|
});
|
||||||
|
const command = {
|
||||||
|
...TASK_COMMAND,
|
||||||
|
taskId: 'task-bundle-00001',
|
||||||
|
mutationId: '123e4567-e89b-42d3-a456-426614174021',
|
||||||
|
spec: {
|
||||||
|
...TASK_COMMAND.spec,
|
||||||
|
config: { ...TASK_COMMAND.spec.config, environmentBundleRef },
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const fixture = appendPool('task');
|
||||||
|
await new PostgresTaskDefinitionRepository(
|
||||||
|
fixture.pool,
|
||||||
|
).appendTaskDefinitionRevision(command);
|
||||||
|
const insert = fixture.queries.find(({ text }) =>
|
||||||
|
text.includes('INSERT INTO "ql3"."task_execution_revisions"'),
|
||||||
|
);
|
||||||
|
const plan = JSON.parse(insert.values[7]);
|
||||||
|
assert.equal(plan.environmentBundleRef, environmentBundleRef);
|
||||||
|
assert.deepEqual(plan.environment, []);
|
||||||
|
assert.equal(JSON.stringify(plan).includes('LEGACY_ENV_NAME'), false);
|
||||||
|
});
|
||||||
|
|
||||||
test('runs TaskDefinition transaction hooks for create and replay before COMMIT', async () => {
|
test('runs TaskDefinition transaction hooks for create and replay before COMMIT', async () => {
|
||||||
const createdFixture = appendPool('task');
|
const createdFixture = appendPool('task');
|
||||||
const createdHook = [];
|
const createdHook = [];
|
||||||
|
|||||||
@@ -18,12 +18,21 @@ const SOURCE_DIGEST = 'a'.repeat(64);
|
|||||||
const TASK_REVISION = `qltd:v1:1:${SOURCE_DIGEST}`;
|
const TASK_REVISION = `qltd:v1:1:${SOURCE_DIGEST}`;
|
||||||
const LEASE_TOKEN = 'worker_generated_lease_capability_0000000000000001';
|
const LEASE_TOKEN = 'worker_generated_lease_capability_0000000000000001';
|
||||||
const SECRET_REF = createSecretRef({ projectId: 'project-1', name: 'token' });
|
const SECRET_REF = createSecretRef({ projectId: 'project-1', name: 'token' });
|
||||||
|
const ENVIRONMENT_BUNDLE_REF = createSecretRef({
|
||||||
|
projectId: 'project-1',
|
||||||
|
name: 'legacy-env-bundle',
|
||||||
|
version: 4,
|
||||||
|
});
|
||||||
|
|
||||||
function revision() {
|
function revision() {
|
||||||
return createClusterTaskExecutionRevision({
|
return createClusterTaskExecutionRevision({
|
||||||
projectId: 'project-1', taskId: 'task-1', taskRevision: TASK_REVISION,
|
projectId: 'project-1',
|
||||||
sourceRevision: 1, sourceContentDigest: SOURCE_DIGEST,
|
taskId: 'task-1',
|
||||||
executorType: 'remote_worker', planSchema: 'qinglong/command-execution@v1',
|
taskRevision: TASK_REVISION,
|
||||||
|
sourceRevision: 1,
|
||||||
|
sourceContentDigest: SOURCE_DIGEST,
|
||||||
|
executorType: 'remote_worker',
|
||||||
|
planSchema: 'qinglong/command-execution@v1',
|
||||||
command: { kind: 'argv', file: '/bin/true', args: [] },
|
command: { kind: 'argv', file: '/bin/true', args: [] },
|
||||||
environment: [{ name: 'TOKEN', kind: 'secret', secretRef: SECRET_REF }],
|
environment: [{ name: 'TOKEN', kind: 'secret', secretRef: SECRET_REF }],
|
||||||
createdAtMs: 1,
|
createdAtMs: 1,
|
||||||
@@ -32,40 +41,74 @@ function revision() {
|
|||||||
|
|
||||||
function command(executionDigest, overrides = {}) {
|
function command(executionDigest, overrides = {}) {
|
||||||
return {
|
return {
|
||||||
workerId: 'edge-1', workerSessionId: SESSION_ID, workerGeneration: 2,
|
workerId: 'edge-1',
|
||||||
runId: 'run-1', attemptId: 'attempt-1', projectId: 'project-1',
|
workerSessionId: SESSION_ID,
|
||||||
taskId: 'task-1', taskRevision: TASK_REVISION, executionDigest,
|
workerGeneration: 2,
|
||||||
offerId: 'offer-1', leaseGeneration: 3, leaseToken: LEASE_TOKEN,
|
runId: 'run-1',
|
||||||
expectedLeaseVersion: 4, secretRefs: [SECRET_REF], ...overrides,
|
attemptId: 'attempt-1',
|
||||||
|
projectId: 'project-1',
|
||||||
|
taskId: 'task-1',
|
||||||
|
taskRevision: TASK_REVISION,
|
||||||
|
executionDigest,
|
||||||
|
offerId: 'offer-1',
|
||||||
|
leaseGeneration: 3,
|
||||||
|
leaseToken: LEASE_TOKEN,
|
||||||
|
expectedLeaseVersion: 4,
|
||||||
|
secretRefs: [SECRET_REF],
|
||||||
|
environmentBundleRefs: [],
|
||||||
|
...overrides,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
function authorityRow(plan, overrides = {}) {
|
function authorityRow(plan, overrides = {}) {
|
||||||
return {
|
return {
|
||||||
observedAtMs: '1000', runId: 'run-1', runProjectId: 'project-1',
|
observedAtMs: '1000',
|
||||||
runTaskId: 'task-1', runTaskRevision: TASK_REVISION,
|
runId: 'run-1',
|
||||||
runStatus: 'dispatching', executionOwner: 'runtime',
|
runProjectId: 'project-1',
|
||||||
cancelRequestedAtMs: null, attemptStatus: 'starting',
|
runTaskId: 'task-1',
|
||||||
attemptExecutorType: 'remote_worker', attemptWorkerId: 'edge-1',
|
runTaskRevision: TASK_REVISION,
|
||||||
attemptWorkerSessionId: SESSION_ID, attemptWorkerGeneration: 2,
|
runStatus: 'dispatching',
|
||||||
|
executionOwner: 'runtime',
|
||||||
|
cancelRequestedAtMs: null,
|
||||||
|
attemptStatus: 'starting',
|
||||||
|
attemptExecutorType: 'remote_worker',
|
||||||
|
attemptWorkerId: 'edge-1',
|
||||||
|
attemptWorkerSessionId: SESSION_ID,
|
||||||
|
attemptWorkerGeneration: 2,
|
||||||
attemptLeaseTokenDigest: digestRunDispatchLeaseToken(LEASE_TOKEN),
|
attemptLeaseTokenDigest: digestRunDispatchLeaseToken(LEASE_TOKEN),
|
||||||
attemptLeaseGeneration: 3, attemptLeaseVersion: 4,
|
attemptLeaseGeneration: 3,
|
||||||
attemptOfferId: 'offer-1', sessionId: SESSION_ID, sessionGeneration: 2,
|
attemptLeaseVersion: 4,
|
||||||
sessionStatus: 'online', sessionExpiresAtMs: '5000',
|
attemptOfferId: 'offer-1',
|
||||||
leaseRunId: 'run-1', leaseStatus: 'leased', leaseVersion: 4,
|
sessionId: SESSION_ID,
|
||||||
leaseGeneration: 3, leaseWorkerId: 'edge-1',
|
sessionGeneration: 2,
|
||||||
leaseWorkerSessionId: SESSION_ID, leaseWorkerGeneration: 2,
|
sessionStatus: 'online',
|
||||||
|
sessionExpiresAtMs: '5000',
|
||||||
|
leaseRunId: 'run-1',
|
||||||
|
leaseStatus: 'leased',
|
||||||
|
leaseVersion: 4,
|
||||||
|
leaseGeneration: 3,
|
||||||
|
leaseWorkerId: 'edge-1',
|
||||||
|
leaseWorkerSessionId: SESSION_ID,
|
||||||
|
leaseWorkerGeneration: 2,
|
||||||
leaseTokenDigest: digestRunDispatchLeaseToken(LEASE_TOKEN),
|
leaseTokenDigest: digestRunDispatchLeaseToken(LEASE_TOKEN),
|
||||||
leaseOfferId: 'offer-1', leaseExpiresAtMs: '5000',
|
leaseOfferId: 'offer-1',
|
||||||
revisionProjectId: plan.projectId, revisionTaskId: plan.taskId,
|
leaseExpiresAtMs: '5000',
|
||||||
|
revisionProjectId: plan.projectId,
|
||||||
|
revisionTaskId: plan.taskId,
|
||||||
sourceRevision: plan.sourceRevision,
|
sourceRevision: plan.sourceRevision,
|
||||||
revisionTaskRevision: plan.taskRevision,
|
revisionTaskRevision: plan.taskRevision,
|
||||||
sourceContentDigest: plan.sourceContentDigest,
|
sourceContentDigest: plan.sourceContentDigest,
|
||||||
revisionExecutorType: plan.executorType, planSchema: plan.planSchema,
|
revisionExecutorType: plan.executorType,
|
||||||
|
planSchema: plan.planSchema,
|
||||||
planJson: {
|
planJson: {
|
||||||
command: plan.command,
|
command: plan.command,
|
||||||
environment: plan.environment,
|
environment: plan.environment,
|
||||||
...(plan.workingDirectory === undefined ? {} : { workingDirectory: plan.workingDirectory }),
|
...(plan.environmentBundleRef === undefined
|
||||||
|
? {}
|
||||||
|
: { environmentBundleRef: plan.environmentBundleRef }),
|
||||||
|
...(plan.workingDirectory === undefined
|
||||||
|
? {}
|
||||||
|
: { workingDirectory: plan.workingDirectory }),
|
||||||
...(plan.timeoutMs === undefined ? {} : { timeoutMs: plan.timeoutMs }),
|
...(plan.timeoutMs === undefined ? {} : { timeoutMs: plan.timeoutMs }),
|
||||||
...(plan.placement === undefined ? {} : { placement: plan.placement }),
|
...(plan.placement === undefined ? {} : { placement: plan.placement }),
|
||||||
},
|
},
|
||||||
@@ -84,11 +127,15 @@ function fixture(row) {
|
|||||||
if (sql.includes('FROM observation')) return { rows: row ? [row] : [] };
|
if (sql.includes('FROM observation')) return { rows: row ? [row] : [] };
|
||||||
return { rows: [] };
|
return { rows: [] };
|
||||||
},
|
},
|
||||||
release() { released += 1; },
|
release() {
|
||||||
|
released += 1;
|
||||||
|
},
|
||||||
};
|
};
|
||||||
return {
|
return {
|
||||||
repository: new PostgresRemoteWorkerSecretDeliveryAuthorityRepository({
|
repository: new PostgresRemoteWorkerSecretDeliveryAuthorityRepository({
|
||||||
async connect() { return client; },
|
async connect() {
|
||||||
|
return client;
|
||||||
|
},
|
||||||
}),
|
}),
|
||||||
queries,
|
queries,
|
||||||
released: () => released,
|
released: () => released,
|
||||||
@@ -100,9 +147,13 @@ test('authorizes exact Session, Lease and immutable execution revision fences',
|
|||||||
const { repository, queries, released } = fixture(authorityRow(plan));
|
const { repository, queries, released } = fixture(authorityRow(plan));
|
||||||
const result = await repository.authorize(command(plan.contentDigest));
|
const result = await repository.authorize(command(plan.contentDigest));
|
||||||
assert.deepEqual(result.secretRefs, [SECRET_REF]);
|
assert.deepEqual(result.secretRefs, [SECRET_REF]);
|
||||||
|
assert.deepEqual(result.environmentBundleRefs, []);
|
||||||
assert.equal(result.executionDigest, plan.contentDigest);
|
assert.equal(result.executionDigest, plan.contentDigest);
|
||||||
assert.equal('leaseToken' in result, false);
|
assert.equal('leaseToken' in result, false);
|
||||||
assert.equal(queries.some(({ sql }) => sql.includes('pg_advisory_xact_lock')), true);
|
assert.equal(
|
||||||
|
queries.some(({ sql }) => sql.includes('pg_advisory_xact_lock')),
|
||||||
|
true,
|
||||||
|
);
|
||||||
assert.equal(queries.at(-1).sql, 'COMMIT');
|
assert.equal(queries.at(-1).sql, 'COMMIT');
|
||||||
assert.equal(released(), 1);
|
assert.equal(released(), 1);
|
||||||
});
|
});
|
||||||
@@ -129,9 +180,11 @@ test('rejects partial Secret scope and execution digest drift', async () => {
|
|||||||
const extra = createSecretRef({ projectId: 'project-1', name: 'other' });
|
const extra = createSecretRef({ projectId: 'project-1', name: 'other' });
|
||||||
const partial = fixture(authorityRow(plan));
|
const partial = fixture(authorityRow(plan));
|
||||||
await assert.rejects(
|
await assert.rejects(
|
||||||
partial.repository.authorize(command(plan.contentDigest, {
|
partial.repository.authorize(
|
||||||
secretRefs: [extra],
|
command(plan.contentDigest, {
|
||||||
})),
|
secretRefs: [extra],
|
||||||
|
}),
|
||||||
|
),
|
||||||
/secret_scope_mismatch/,
|
/secret_scope_mismatch/,
|
||||||
);
|
);
|
||||||
const digestDrift = fixture(authorityRow(plan));
|
const digestDrift = fixture(authorityRow(plan));
|
||||||
@@ -140,3 +193,29 @@ test('rejects partial Secret scope and execution digest drift', async () => {
|
|||||||
/secret_scope_mismatch/,
|
/secret_scope_mismatch/,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('authorizes a bundle-only execution without exposing environment names', async () => {
|
||||||
|
const plan = createClusterTaskExecutionRevision({
|
||||||
|
projectId: 'project-1',
|
||||||
|
taskId: 'task-1',
|
||||||
|
taskRevision: TASK_REVISION,
|
||||||
|
sourceRevision: 1,
|
||||||
|
sourceContentDigest: SOURCE_DIGEST,
|
||||||
|
executorType: 'remote_worker',
|
||||||
|
planSchema: 'qinglong/command-execution@v1',
|
||||||
|
command: { kind: 'argv', file: '/bin/true', args: [] },
|
||||||
|
environment: [],
|
||||||
|
environmentBundleRef: ENVIRONMENT_BUNDLE_REF,
|
||||||
|
createdAtMs: 1,
|
||||||
|
});
|
||||||
|
const { repository } = fixture(authorityRow(plan));
|
||||||
|
const result = await repository.authorize(
|
||||||
|
command(plan.contentDigest, {
|
||||||
|
secretRefs: [],
|
||||||
|
environmentBundleRefs: [ENVIRONMENT_BUNDLE_REF],
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.deepEqual(result.secretRefs, []);
|
||||||
|
assert.deepEqual(result.environmentBundleRefs, [ENVIRONMENT_BUNDLE_REF]);
|
||||||
|
assert.equal(JSON.stringify(result).includes('LEGACY_VALUE'), false);
|
||||||
|
});
|
||||||
|
|||||||
@@ -110,6 +110,9 @@
|
|||||||
"secret-projection": [
|
"secret-projection": [
|
||||||
"dist/secret/secretProjection.d.ts"
|
"dist/secret/secretProjection.d.ts"
|
||||||
],
|
],
|
||||||
|
"environment-bundle": [
|
||||||
|
"dist/secret/environmentBundle.d.ts"
|
||||||
|
],
|
||||||
"plugin-package-task-reconciliation": [
|
"plugin-package-task-reconciliation": [
|
||||||
"dist/plugin-package/pluginPackageTaskReconciliation.d.ts"
|
"dist/plugin-package/pluginPackageTaskReconciliation.d.ts"
|
||||||
],
|
],
|
||||||
@@ -470,6 +473,11 @@
|
|||||||
"require": "./dist/secret/secretProjection.js",
|
"require": "./dist/secret/secretProjection.js",
|
||||||
"default": "./dist/secret/secretProjection.js"
|
"default": "./dist/secret/secretProjection.js"
|
||||||
},
|
},
|
||||||
|
"./environment-bundle": {
|
||||||
|
"types": "./dist/secret/environmentBundle.d.ts",
|
||||||
|
"require": "./dist/secret/environmentBundle.js",
|
||||||
|
"default": "./dist/secret/environmentBundle.js"
|
||||||
|
},
|
||||||
"./plugin-package-task-reconciliation": {
|
"./plugin-package-task-reconciliation": {
|
||||||
"types": "./dist/plugin-package/pluginPackageTaskReconciliation.d.ts",
|
"types": "./dist/plugin-package/pluginPackageTaskReconciliation.d.ts",
|
||||||
"require": "./dist/plugin-package/pluginPackageTaskReconciliation.js",
|
"require": "./dist/plugin-package/pluginPackageTaskReconciliation.js",
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ export const CLUSTER_LEGACY_ENV_MIGRATION_PLAN_SCHEMA =
|
|||||||
export const MAX_CLUSTER_LEGACY_ENV_SOURCE_ROWS = 100_000;
|
export const MAX_CLUSTER_LEGACY_ENV_SOURCE_ROWS = 100_000;
|
||||||
export const MAX_CLUSTER_LEGACY_ENV_TASKS = 100_000;
|
export const MAX_CLUSTER_LEGACY_ENV_TASKS = 100_000;
|
||||||
export const MAX_CLUSTER_LEGACY_ENV_TRIGGERS = 500_000;
|
export const MAX_CLUSTER_LEGACY_ENV_TRIGGERS = 500_000;
|
||||||
|
export const MAX_CLUSTER_LEGACY_ENV_EFFECTIVE_BINDINGS = 256;
|
||||||
export const MAX_CLUSTER_LEGACY_ENV_EFFECTIVE_BYTES = 64 * 1024;
|
export const MAX_CLUSTER_LEGACY_ENV_EFFECTIVE_BYTES = 64 * 1024;
|
||||||
export const MAX_CLUSTER_LEGACY_ENV_MIGRATION_PLAN_JSON_BYTES = 8 * 1024;
|
export const MAX_CLUSTER_LEGACY_ENV_MIGRATION_PLAN_JSON_BYTES = 8 * 1024;
|
||||||
|
|
||||||
@@ -207,7 +208,7 @@ function sourceEvidence(
|
|||||||
const effectiveBindingCount = count(
|
const effectiveBindingCount = count(
|
||||||
value.effectiveBindingCount,
|
value.effectiveBindingCount,
|
||||||
'effectiveBindingCount',
|
'effectiveBindingCount',
|
||||||
MAX_CLUSTER_LEGACY_ENV_SOURCE_ROWS,
|
MAX_CLUSTER_LEGACY_ENV_EFFECTIVE_BINDINGS,
|
||||||
);
|
);
|
||||||
if (
|
if (
|
||||||
sourceRowCount < 1 ||
|
sourceRowCount < 1 ||
|
||||||
|
|||||||
@@ -1,14 +1,19 @@
|
|||||||
import { digestRunDispatchLeaseToken, assertRunDispatchId } from '../run/runDispatchLease';
|
import {
|
||||||
|
digestRunDispatchLeaseToken,
|
||||||
|
assertRunDispatchId,
|
||||||
|
} from '../run/runDispatchLease';
|
||||||
import { parseSecretRef } from '../secret/secretReference';
|
import { parseSecretRef } from '../secret/secretReference';
|
||||||
import { assertWorkerId, assertWorkerSessionId } from '../worker/workerSession';
|
import { assertWorkerId, assertWorkerSessionId } from '../worker/workerSession';
|
||||||
|
|
||||||
export const REMOTE_SECRET_DELIVERY_SCHEMA =
|
export const REMOTE_SECRET_DELIVERY_SCHEMA =
|
||||||
'qinglong/remote-secret-delivery@v1';
|
'qinglong/remote-secret-delivery@v2';
|
||||||
export const MAX_REMOTE_SECRET_DELIVERY_REFS = 64;
|
export const MAX_REMOTE_SECRET_DELIVERY_REFS = 64;
|
||||||
|
export const MAX_REMOTE_ENVIRONMENT_BUNDLE_REFS = 1;
|
||||||
export const MAX_REMOTE_SECRET_DELIVERY_REQUEST_BYTES = 64 * 1024;
|
export const MAX_REMOTE_SECRET_DELIVERY_REQUEST_BYTES = 64 * 1024;
|
||||||
export const MAX_REMOTE_SECRET_DELIVERY_RESPONSE_BYTES = 128 * 1024;
|
export const MAX_REMOTE_SECRET_DELIVERY_RESPONSE_BYTES = 256 * 1024;
|
||||||
export const MAX_REMOTE_SECRET_VALUE_BYTES = 16 * 1024;
|
export const MAX_REMOTE_SECRET_VALUE_BYTES = 16 * 1024;
|
||||||
export const MAX_REMOTE_SECRET_DELIVERY_TOTAL_VALUE_BYTES = 64 * 1024;
|
export const MAX_REMOTE_SECRET_DELIVERY_TOTAL_VALUE_BYTES = 64 * 1024;
|
||||||
|
export const MAX_REMOTE_ENVIRONMENT_BUNDLE_VALUE_BYTES = 96 * 1024;
|
||||||
|
|
||||||
export interface RemoteWorkerSecretDeliveryCommand {
|
export interface RemoteWorkerSecretDeliveryCommand {
|
||||||
readonly workerId: string;
|
readonly workerId: string;
|
||||||
@@ -25,6 +30,7 @@ export interface RemoteWorkerSecretDeliveryCommand {
|
|||||||
readonly leaseToken: string;
|
readonly leaseToken: string;
|
||||||
readonly expectedLeaseVersion: number;
|
readonly expectedLeaseVersion: number;
|
||||||
readonly secretRefs: readonly string[];
|
readonly secretRefs: readonly string[];
|
||||||
|
readonly environmentBundleRefs: readonly string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
export type RemoteWorkerSecretDeliveryRequestBody = Readonly<
|
export type RemoteWorkerSecretDeliveryRequestBody = Readonly<
|
||||||
@@ -47,6 +53,7 @@ export interface RemoteWorkerSecretDeliveryAuthority {
|
|||||||
readonly leaseGeneration: number;
|
readonly leaseGeneration: number;
|
||||||
readonly leaseVersion: number;
|
readonly leaseVersion: number;
|
||||||
readonly secretRefs: readonly string[];
|
readonly secretRefs: readonly string[];
|
||||||
|
readonly environmentBundleRefs: readonly string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface RemoteWorkerSecretValue {
|
export interface RemoteWorkerSecretValue {
|
||||||
@@ -56,6 +63,7 @@ export interface RemoteWorkerSecretValue {
|
|||||||
|
|
||||||
export interface RemoteWorkerSecretResolution {
|
export interface RemoteWorkerSecretResolution {
|
||||||
readonly values: readonly RemoteWorkerSecretValue[];
|
readonly values: readonly RemoteWorkerSecretValue[];
|
||||||
|
readonly environmentBundles: readonly RemoteWorkerSecretValue[];
|
||||||
readonly dispose?: () => Promise<void> | void;
|
readonly dispose?: () => Promise<void> | void;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -77,6 +85,7 @@ export interface RemoteWorkerSecretDeliveryResult {
|
|||||||
readonly offerId: string;
|
readonly offerId: string;
|
||||||
readonly executionDigest: string;
|
readonly executionDigest: string;
|
||||||
readonly values: readonly RemoteWorkerSecretValue[];
|
readonly values: readonly RemoteWorkerSecretValue[];
|
||||||
|
readonly environmentBundles: readonly RemoteWorkerSecretValue[];
|
||||||
readonly dispose?: () => Promise<void> | void;
|
readonly dispose?: () => Promise<void> | void;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -134,7 +143,8 @@ function exactKeys(
|
|||||||
if (
|
if (
|
||||||
actual.length !== sorted.length ||
|
actual.length !== sorted.length ||
|
||||||
actual.some((key, index) => key !== sorted[index])
|
actual.some((key, index) => key !== sorted[index])
|
||||||
) invalid(`${label} shape is invalid`);
|
)
|
||||||
|
invalid(`${label} shape is invalid`);
|
||||||
}
|
}
|
||||||
|
|
||||||
function identifier(label: string, value: unknown, maximum = 128): string {
|
function identifier(label: string, value: unknown, maximum = 128): string {
|
||||||
@@ -143,7 +153,8 @@ function identifier(label: string, value: unknown, maximum = 128): string {
|
|||||||
value.length < 1 ||
|
value.length < 1 ||
|
||||||
Buffer.byteLength(value, 'utf8') > maximum ||
|
Buffer.byteLength(value, 'utf8') > maximum ||
|
||||||
/[\u0000-\u001f\u007f]/.test(value)
|
/[\u0000-\u001f\u007f]/.test(value)
|
||||||
) return invalid(`${label} is invalid`);
|
)
|
||||||
|
return invalid(`${label} is invalid`);
|
||||||
return value;
|
return value;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -152,19 +163,18 @@ function positiveInteger(label: string, value: unknown, minimum = 1): number {
|
|||||||
!Number.isSafeInteger(value) ||
|
!Number.isSafeInteger(value) ||
|
||||||
(value as number) < minimum ||
|
(value as number) < minimum ||
|
||||||
(value as number) > 2_147_483_647
|
(value as number) > 2_147_483_647
|
||||||
) return invalid(`${label} is invalid`);
|
)
|
||||||
|
return invalid(`${label} is invalid`);
|
||||||
return value as number;
|
return value as number;
|
||||||
}
|
}
|
||||||
|
|
||||||
function normalizeSecretRefs(
|
function normalizeSecretRefs(
|
||||||
value: unknown,
|
value: unknown,
|
||||||
projectId: string,
|
projectId: string,
|
||||||
|
maximum: number,
|
||||||
): readonly string[] {
|
): readonly string[] {
|
||||||
if (
|
if (!Array.isArray(value) || value.length > maximum)
|
||||||
!Array.isArray(value) ||
|
return invalid('secretRefs are invalid');
|
||||||
value.length < 1 ||
|
|
||||||
value.length > MAX_REMOTE_SECRET_DELIVERY_REFS
|
|
||||||
) return invalid('secretRefs are invalid');
|
|
||||||
const seen = new Set<string>();
|
const seen = new Set<string>();
|
||||||
const refs = value.map((entry) => {
|
const refs = value.map((entry) => {
|
||||||
if (typeof entry !== 'string' || seen.has(entry)) {
|
if (typeof entry !== 'string' || seen.has(entry)) {
|
||||||
@@ -188,11 +198,27 @@ export function normalizeRemoteWorkerSecretDeliveryCommand(
|
|||||||
value: RemoteWorkerSecretDeliveryCommand,
|
value: RemoteWorkerSecretDeliveryCommand,
|
||||||
): Readonly<RemoteWorkerSecretDeliveryCommand> {
|
): Readonly<RemoteWorkerSecretDeliveryCommand> {
|
||||||
const command = object(value, 'command');
|
const command = object(value, 'command');
|
||||||
exactKeys(command, [
|
exactKeys(
|
||||||
'attemptId', 'executionDigest', 'expectedLeaseVersion', 'leaseGeneration',
|
command,
|
||||||
'leaseToken', 'offerId', 'projectId', 'runId', 'secretRefs', 'taskId',
|
[
|
||||||
'taskRevision', 'workerGeneration', 'workerId', 'workerSessionId',
|
'attemptId',
|
||||||
], 'command');
|
'environmentBundleRefs',
|
||||||
|
'executionDigest',
|
||||||
|
'expectedLeaseVersion',
|
||||||
|
'leaseGeneration',
|
||||||
|
'leaseToken',
|
||||||
|
'offerId',
|
||||||
|
'projectId',
|
||||||
|
'runId',
|
||||||
|
'secretRefs',
|
||||||
|
'taskId',
|
||||||
|
'taskRevision',
|
||||||
|
'workerGeneration',
|
||||||
|
'workerId',
|
||||||
|
'workerSessionId',
|
||||||
|
],
|
||||||
|
'command',
|
||||||
|
);
|
||||||
try {
|
try {
|
||||||
assertWorkerId(command.workerId as string);
|
assertWorkerId(command.workerId as string);
|
||||||
assertWorkerSessionId(command.workerSessionId as string);
|
assertWorkerSessionId(command.workerSessionId as string);
|
||||||
@@ -206,7 +232,10 @@ export function normalizeRemoteWorkerSecretDeliveryCommand(
|
|||||||
const normalized = Object.freeze({
|
const normalized = Object.freeze({
|
||||||
workerId: command.workerId as string,
|
workerId: command.workerId as string,
|
||||||
workerSessionId: command.workerSessionId as string,
|
workerSessionId: command.workerSessionId as string,
|
||||||
workerGeneration: positiveInteger('workerGeneration', command.workerGeneration),
|
workerGeneration: positiveInteger(
|
||||||
|
'workerGeneration',
|
||||||
|
command.workerGeneration,
|
||||||
|
),
|
||||||
runId: command.runId as string,
|
runId: command.runId as string,
|
||||||
attemptId: command.attemptId as string,
|
attemptId: command.attemptId as string,
|
||||||
projectId,
|
projectId,
|
||||||
@@ -214,13 +243,38 @@ export function normalizeRemoteWorkerSecretDeliveryCommand(
|
|||||||
taskRevision: identifier('taskRevision', command.taskRevision),
|
taskRevision: identifier('taskRevision', command.taskRevision),
|
||||||
executionDigest: identifier('executionDigest', command.executionDigest, 64),
|
executionDigest: identifier('executionDigest', command.executionDigest, 64),
|
||||||
offerId: command.offerId as string,
|
offerId: command.offerId as string,
|
||||||
leaseGeneration: positiveInteger('leaseGeneration', command.leaseGeneration),
|
leaseGeneration: positiveInteger(
|
||||||
|
'leaseGeneration',
|
||||||
|
command.leaseGeneration,
|
||||||
|
),
|
||||||
leaseToken: identifier('leaseToken', command.leaseToken, 128),
|
leaseToken: identifier('leaseToken', command.leaseToken, 128),
|
||||||
expectedLeaseVersion: positiveInteger(
|
expectedLeaseVersion: positiveInteger(
|
||||||
'expectedLeaseVersion', command.expectedLeaseVersion, 0,
|
'expectedLeaseVersion',
|
||||||
|
command.expectedLeaseVersion,
|
||||||
|
0,
|
||||||
|
),
|
||||||
|
secretRefs: normalizeSecretRefs(
|
||||||
|
command.secretRefs,
|
||||||
|
projectId,
|
||||||
|
MAX_REMOTE_SECRET_DELIVERY_REFS,
|
||||||
|
),
|
||||||
|
environmentBundleRefs: normalizeSecretRefs(
|
||||||
|
command.environmentBundleRefs,
|
||||||
|
projectId,
|
||||||
|
MAX_REMOTE_ENVIRONMENT_BUNDLE_REFS,
|
||||||
),
|
),
|
||||||
secretRefs: normalizeSecretRefs(command.secretRefs, projectId),
|
|
||||||
});
|
});
|
||||||
|
if (
|
||||||
|
normalized.secretRefs.length + normalized.environmentBundleRefs.length <
|
||||||
|
1
|
||||||
|
)
|
||||||
|
return invalid('Secret reference set is empty');
|
||||||
|
if (
|
||||||
|
normalized.secretRefs.some((reference) =>
|
||||||
|
normalized.environmentBundleRefs.includes(reference),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return invalid('Secret reference roles overlap');
|
||||||
if (!/^[0-9a-f]{64}$/.test(normalized.executionDigest)) {
|
if (!/^[0-9a-f]{64}$/.test(normalized.executionDigest)) {
|
||||||
return invalid('executionDigest is invalid');
|
return invalid('executionDigest is invalid');
|
||||||
}
|
}
|
||||||
@@ -236,11 +290,26 @@ export function normalizeRemoteWorkerSecretDeliveryAuthority(
|
|||||||
value: RemoteWorkerSecretDeliveryAuthority,
|
value: RemoteWorkerSecretDeliveryAuthority,
|
||||||
): Readonly<RemoteWorkerSecretDeliveryAuthority> {
|
): Readonly<RemoteWorkerSecretDeliveryAuthority> {
|
||||||
const authority = object(value, 'authority');
|
const authority = object(value, 'authority');
|
||||||
exactKeys(authority, [
|
exactKeys(
|
||||||
'attemptId', 'executionDigest', 'leaseGeneration', 'leaseVersion',
|
authority,
|
||||||
'offerId', 'projectId', 'runId', 'secretRefs', 'taskId', 'taskRevision',
|
[
|
||||||
'workerGeneration', 'workerId', 'workerSessionId',
|
'attemptId',
|
||||||
], 'authority');
|
'environmentBundleRefs',
|
||||||
|
'executionDigest',
|
||||||
|
'leaseGeneration',
|
||||||
|
'leaseVersion',
|
||||||
|
'offerId',
|
||||||
|
'projectId',
|
||||||
|
'runId',
|
||||||
|
'secretRefs',
|
||||||
|
'taskId',
|
||||||
|
'taskRevision',
|
||||||
|
'workerGeneration',
|
||||||
|
'workerId',
|
||||||
|
'workerSessionId',
|
||||||
|
],
|
||||||
|
'authority',
|
||||||
|
);
|
||||||
try {
|
try {
|
||||||
assertWorkerId(authority.workerId as string);
|
assertWorkerId(authority.workerId as string);
|
||||||
assertWorkerSessionId(authority.workerSessionId as string);
|
assertWorkerSessionId(authority.workerSessionId as string);
|
||||||
@@ -255,7 +324,8 @@ export function normalizeRemoteWorkerSecretDeliveryAuthority(
|
|||||||
workerId: authority.workerId as string,
|
workerId: authority.workerId as string,
|
||||||
workerSessionId: authority.workerSessionId as string,
|
workerSessionId: authority.workerSessionId as string,
|
||||||
workerGeneration: positiveInteger(
|
workerGeneration: positiveInteger(
|
||||||
'workerGeneration', authority.workerGeneration,
|
'workerGeneration',
|
||||||
|
authority.workerGeneration,
|
||||||
),
|
),
|
||||||
runId: authority.runId as string,
|
runId: authority.runId as string,
|
||||||
attemptId: authority.attemptId as string,
|
attemptId: authority.attemptId as string,
|
||||||
@@ -263,15 +333,38 @@ export function normalizeRemoteWorkerSecretDeliveryAuthority(
|
|||||||
taskId: identifier('taskId', authority.taskId),
|
taskId: identifier('taskId', authority.taskId),
|
||||||
taskRevision: identifier('taskRevision', authority.taskRevision),
|
taskRevision: identifier('taskRevision', authority.taskRevision),
|
||||||
executionDigest: identifier(
|
executionDigest: identifier(
|
||||||
'executionDigest', authority.executionDigest, 64,
|
'executionDigest',
|
||||||
|
authority.executionDigest,
|
||||||
|
64,
|
||||||
),
|
),
|
||||||
offerId: authority.offerId as string,
|
offerId: authority.offerId as string,
|
||||||
leaseGeneration: positiveInteger(
|
leaseGeneration: positiveInteger(
|
||||||
'leaseGeneration', authority.leaseGeneration,
|
'leaseGeneration',
|
||||||
|
authority.leaseGeneration,
|
||||||
),
|
),
|
||||||
leaseVersion: positiveInteger('leaseVersion', authority.leaseVersion, 0),
|
leaseVersion: positiveInteger('leaseVersion', authority.leaseVersion, 0),
|
||||||
secretRefs: normalizeSecretRefs(authority.secretRefs, projectId),
|
secretRefs: normalizeSecretRefs(
|
||||||
|
authority.secretRefs,
|
||||||
|
projectId,
|
||||||
|
MAX_REMOTE_SECRET_DELIVERY_REFS,
|
||||||
|
),
|
||||||
|
environmentBundleRefs: normalizeSecretRefs(
|
||||||
|
authority.environmentBundleRefs,
|
||||||
|
projectId,
|
||||||
|
MAX_REMOTE_ENVIRONMENT_BUNDLE_REFS,
|
||||||
|
),
|
||||||
});
|
});
|
||||||
|
if (
|
||||||
|
normalized.secretRefs.length + normalized.environmentBundleRefs.length <
|
||||||
|
1
|
||||||
|
)
|
||||||
|
return invalid('Secret reference set is empty');
|
||||||
|
if (
|
||||||
|
normalized.secretRefs.some((reference) =>
|
||||||
|
normalized.environmentBundleRefs.includes(reference),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return invalid('Secret reference roles overlap');
|
||||||
if (!/^[0-9a-f]{64}$/.test(normalized.executionDigest)) {
|
if (!/^[0-9a-f]{64}$/.test(normalized.executionDigest)) {
|
||||||
return invalid('executionDigest is invalid');
|
return invalid('executionDigest is invalid');
|
||||||
}
|
}
|
||||||
@@ -282,59 +375,98 @@ export function createRemoteWorkerSecretDeliveryRequestBody(
|
|||||||
command: RemoteWorkerSecretDeliveryCommand,
|
command: RemoteWorkerSecretDeliveryCommand,
|
||||||
): RemoteWorkerSecretDeliveryRequestBody {
|
): RemoteWorkerSecretDeliveryRequestBody {
|
||||||
const normalized = normalizeRemoteWorkerSecretDeliveryCommand(command);
|
const normalized = normalizeRemoteWorkerSecretDeliveryCommand(command);
|
||||||
const { workerId: _workerId, workerSessionId: _sessionId, ...request } = normalized;
|
const {
|
||||||
|
workerId: _workerId,
|
||||||
|
workerSessionId: _sessionId,
|
||||||
|
...request
|
||||||
|
} = normalized;
|
||||||
return Object.freeze({ schema: REMOTE_SECRET_DELIVERY_SCHEMA, ...request });
|
return Object.freeze({ schema: REMOTE_SECRET_DELIVERY_SCHEMA, ...request });
|
||||||
}
|
}
|
||||||
|
|
||||||
function normalizeValues(
|
function normalizeValues(
|
||||||
value: unknown,
|
value: unknown,
|
||||||
expectedRefs: readonly string[],
|
expectedRefs: readonly string[],
|
||||||
|
maximumValueBytes: number,
|
||||||
|
maximumTotalBytes: number,
|
||||||
|
label: string,
|
||||||
): readonly RemoteWorkerSecretValue[] {
|
): readonly RemoteWorkerSecretValue[] {
|
||||||
if (!Array.isArray(value) || value.length !== expectedRefs.length) {
|
if (!Array.isArray(value) || value.length !== expectedRefs.length) {
|
||||||
return invalid('Secret values are invalid');
|
return invalid(`${label} are invalid`);
|
||||||
}
|
}
|
||||||
let totalValueBytes = 0;
|
let totalValueBytes = 0;
|
||||||
return Object.freeze(value.map((entry, index) => {
|
return Object.freeze(
|
||||||
const item = object(entry, `values[${index}]`);
|
value.map((entry, index) => {
|
||||||
exactKeys(item, ['secretRef', 'value'], `values[${index}]`);
|
const item = object(entry, `values[${index}]`);
|
||||||
if (
|
exactKeys(item, ['secretRef', 'value'], `values[${index}]`);
|
||||||
item.secretRef !== expectedRefs[index] ||
|
if (
|
||||||
typeof item.value !== 'string' ||
|
item.secretRef !== expectedRefs[index] ||
|
||||||
item.value.includes('\0') ||
|
typeof item.value !== 'string' ||
|
||||||
Buffer.byteLength(item.value, 'utf8') > MAX_REMOTE_SECRET_VALUE_BYTES
|
item.value.includes('\0') ||
|
||||||
) return invalid(`values[${index}] is invalid`);
|
Buffer.byteLength(item.value, 'utf8') > maximumValueBytes
|
||||||
totalValueBytes += Buffer.byteLength(item.value, 'utf8');
|
)
|
||||||
if (totalValueBytes > MAX_REMOTE_SECRET_DELIVERY_TOTAL_VALUE_BYTES) {
|
return invalid(`values[${index}] is invalid`);
|
||||||
return invalid('Secret value byte budget exceeded');
|
totalValueBytes += Buffer.byteLength(item.value, 'utf8');
|
||||||
}
|
if (totalValueBytes > maximumTotalBytes) {
|
||||||
return Object.freeze({
|
return invalid('Secret value byte budget exceeded');
|
||||||
secretRef: item.secretRef as string,
|
}
|
||||||
value: item.value,
|
return Object.freeze({
|
||||||
});
|
secretRef: item.secretRef as string,
|
||||||
}));
|
value: item.value,
|
||||||
|
});
|
||||||
|
}),
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
export function createRemoteWorkerSecretDeliveryResponseBody(
|
export function createRemoteWorkerSecretDeliveryResponseBody(
|
||||||
result: Readonly<RemoteWorkerSecretDeliveryResult>,
|
result: Readonly<RemoteWorkerSecretDeliveryResult>,
|
||||||
expectedRefs: readonly string[],
|
expected: Readonly<{
|
||||||
|
secretRefs: readonly string[];
|
||||||
|
environmentBundleRefs: readonly string[];
|
||||||
|
}>,
|
||||||
): RemoteWorkerSecretDeliveryResponseBody {
|
): RemoteWorkerSecretDeliveryResponseBody {
|
||||||
const value = object(result, 'result');
|
const value = object(result, 'result');
|
||||||
const allowed = ['attemptId', 'dispose', 'executionDigest', 'offerId', 'runId', 'values'];
|
const allowed = [
|
||||||
|
'attemptId',
|
||||||
|
'dispose',
|
||||||
|
'environmentBundles',
|
||||||
|
'executionDigest',
|
||||||
|
'offerId',
|
||||||
|
'runId',
|
||||||
|
'values',
|
||||||
|
];
|
||||||
if (Object.keys(value).some((key) => !allowed.includes(key))) {
|
if (Object.keys(value).some((key) => !allowed.includes(key))) {
|
||||||
return invalid('result shape is invalid');
|
return invalid('result shape is invalid');
|
||||||
}
|
}
|
||||||
const runId = identifier('runId', value.runId, 36);
|
const runId = identifier('runId', value.runId, 36);
|
||||||
const attemptId = identifier('attemptId', value.attemptId, 36);
|
const attemptId = identifier('attemptId', value.attemptId, 36);
|
||||||
const offerId = identifier('offerId', value.offerId, 128);
|
const offerId = identifier('offerId', value.offerId, 128);
|
||||||
const executionDigest = identifier('executionDigest', value.executionDigest, 64);
|
const executionDigest = identifier(
|
||||||
if (!/^[0-9a-f]{64}$/.test(executionDigest)) invalid('executionDigest is invalid');
|
'executionDigest',
|
||||||
|
value.executionDigest,
|
||||||
|
64,
|
||||||
|
);
|
||||||
|
if (!/^[0-9a-f]{64}$/.test(executionDigest))
|
||||||
|
invalid('executionDigest is invalid');
|
||||||
return Object.freeze({
|
return Object.freeze({
|
||||||
schema: REMOTE_SECRET_DELIVERY_SCHEMA,
|
schema: REMOTE_SECRET_DELIVERY_SCHEMA,
|
||||||
runId,
|
runId,
|
||||||
attemptId,
|
attemptId,
|
||||||
offerId,
|
offerId,
|
||||||
executionDigest,
|
executionDigest,
|
||||||
values: normalizeValues(value.values, expectedRefs),
|
values: normalizeValues(
|
||||||
|
value.values,
|
||||||
|
expected.secretRefs,
|
||||||
|
MAX_REMOTE_SECRET_VALUE_BYTES,
|
||||||
|
MAX_REMOTE_SECRET_DELIVERY_TOTAL_VALUE_BYTES,
|
||||||
|
'Secret values',
|
||||||
|
),
|
||||||
|
environmentBundles: normalizeValues(
|
||||||
|
value.environmentBundles,
|
||||||
|
expected.environmentBundleRefs,
|
||||||
|
MAX_REMOTE_ENVIRONMENT_BUNDLE_VALUE_BYTES,
|
||||||
|
MAX_REMOTE_ENVIRONMENT_BUNDLE_VALUE_BYTES,
|
||||||
|
'environment bundles',
|
||||||
|
),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -346,15 +478,18 @@ export function parseRemoteWorkerSecretDeliveryResponse(
|
|||||||
offerId: string;
|
offerId: string;
|
||||||
executionDigest: string;
|
executionDigest: string;
|
||||||
secretRefs: readonly string[];
|
secretRefs: readonly string[];
|
||||||
|
environmentBundleRefs: readonly string[];
|
||||||
}>,
|
}>,
|
||||||
): Readonly<RemoteWorkerSecretDeliveryResult> {
|
): Readonly<RemoteWorkerSecretDeliveryResult> {
|
||||||
const bytes = typeof serialized === 'string'
|
const bytes =
|
||||||
? Buffer.from(serialized, 'utf8')
|
typeof serialized === 'string'
|
||||||
: Buffer.from(serialized);
|
? Buffer.from(serialized, 'utf8')
|
||||||
|
: Buffer.from(serialized);
|
||||||
if (
|
if (
|
||||||
bytes.byteLength < 2 ||
|
bytes.byteLength < 2 ||
|
||||||
bytes.byteLength > MAX_REMOTE_SECRET_DELIVERY_RESPONSE_BYTES
|
bytes.byteLength > MAX_REMOTE_SECRET_DELIVERY_RESPONSE_BYTES
|
||||||
) return invalid('response byte size is outside the allowed range');
|
)
|
||||||
|
return invalid('response byte size is outside the allowed range');
|
||||||
let parsed: unknown;
|
let parsed: unknown;
|
||||||
try {
|
try {
|
||||||
parsed = JSON.parse(bytes.toString('utf8')) as unknown;
|
parsed = JSON.parse(bytes.toString('utf8')) as unknown;
|
||||||
@@ -364,30 +499,47 @@ export function parseRemoteWorkerSecretDeliveryResponse(
|
|||||||
bytes.fill(0);
|
bytes.fill(0);
|
||||||
}
|
}
|
||||||
const response = object(parsed, 'response');
|
const response = object(parsed, 'response');
|
||||||
exactKeys(response, [
|
exactKeys(
|
||||||
'attemptId', 'executionDigest', 'offerId', 'runId', 'schema', 'values',
|
response,
|
||||||
], 'response');
|
[
|
||||||
|
'attemptId',
|
||||||
|
'environmentBundles',
|
||||||
|
'executionDigest',
|
||||||
|
'offerId',
|
||||||
|
'runId',
|
||||||
|
'schema',
|
||||||
|
'values',
|
||||||
|
],
|
||||||
|
'response',
|
||||||
|
);
|
||||||
if (response.schema !== REMOTE_SECRET_DELIVERY_SCHEMA) {
|
if (response.schema !== REMOTE_SECRET_DELIVERY_SCHEMA) {
|
||||||
return invalid('response schema is invalid');
|
return invalid('response schema is invalid');
|
||||||
}
|
}
|
||||||
const result = createRemoteWorkerSecretDeliveryResponseBody({
|
const result = createRemoteWorkerSecretDeliveryResponseBody(
|
||||||
runId: response.runId as string,
|
{
|
||||||
attemptId: response.attemptId as string,
|
runId: response.runId as string,
|
||||||
offerId: response.offerId as string,
|
attemptId: response.attemptId as string,
|
||||||
executionDigest: response.executionDigest as string,
|
offerId: response.offerId as string,
|
||||||
values: response.values as readonly RemoteWorkerSecretValue[],
|
executionDigest: response.executionDigest as string,
|
||||||
}, expected.secretRefs);
|
values: response.values as readonly RemoteWorkerSecretValue[],
|
||||||
|
environmentBundles:
|
||||||
|
response.environmentBundles as readonly RemoteWorkerSecretValue[],
|
||||||
|
},
|
||||||
|
expected,
|
||||||
|
);
|
||||||
if (
|
if (
|
||||||
result.runId !== expected.runId ||
|
result.runId !== expected.runId ||
|
||||||
result.attemptId !== expected.attemptId ||
|
result.attemptId !== expected.attemptId ||
|
||||||
result.offerId !== expected.offerId ||
|
result.offerId !== expected.offerId ||
|
||||||
result.executionDigest !== expected.executionDigest
|
result.executionDigest !== expected.executionDigest
|
||||||
) return invalid('response authority does not match request');
|
)
|
||||||
|
return invalid('response authority does not match request');
|
||||||
return Object.freeze({
|
return Object.freeze({
|
||||||
runId: result.runId,
|
runId: result.runId,
|
||||||
attemptId: result.attemptId,
|
attemptId: result.attemptId,
|
||||||
offerId: result.offerId,
|
offerId: result.offerId,
|
||||||
executionDigest: result.executionDigest,
|
executionDigest: result.executionDigest,
|
||||||
values: result.values,
|
values: result.values,
|
||||||
|
environmentBundles: result.environmentBundles,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,164 @@
|
|||||||
|
export const ENVIRONMENT_BUNDLE_SCHEMA =
|
||||||
|
'qinglong/environment-bundle@v1' as const;
|
||||||
|
export const MAX_ENVIRONMENT_BUNDLE_ENTRIES = 256;
|
||||||
|
export const MAX_ENVIRONMENT_BUNDLE_VALUE_BYTES = 16 * 1024;
|
||||||
|
export const MAX_ENVIRONMENT_BUNDLE_TOTAL_BYTES = 64 * 1024;
|
||||||
|
export const MAX_ENVIRONMENT_BUNDLE_ENCODED_BYTES = 96 * 1024;
|
||||||
|
|
||||||
|
const ENVIRONMENT_NAME_PATTERN = /^[A-Za-z_][A-Za-z0-9_]{0,127}$/;
|
||||||
|
|
||||||
|
export interface EnvironmentBundleEntry {
|
||||||
|
readonly name: string;
|
||||||
|
readonly value: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface EnvironmentBundle {
|
||||||
|
readonly schema: typeof ENVIRONMENT_BUNDLE_SCHEMA;
|
||||||
|
readonly entries: readonly EnvironmentBundleEntry[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export class InvalidEnvironmentBundleError extends TypeError {
|
||||||
|
readonly code = 'ENVIRONMENT_BUNDLE_INVALID';
|
||||||
|
|
||||||
|
constructor(message: string) {
|
||||||
|
super(`Environment bundle is invalid: ${message}`);
|
||||||
|
this.name = 'InvalidEnvironmentBundleError';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function invalid(message: string): never {
|
||||||
|
throw new InvalidEnvironmentBundleError(message);
|
||||||
|
}
|
||||||
|
|
||||||
|
function dataObject(value: unknown, label: string): Record<string, unknown> {
|
||||||
|
if (
|
||||||
|
!value ||
|
||||||
|
typeof value !== 'object' ||
|
||||||
|
Array.isArray(value) ||
|
||||||
|
(Object.getPrototypeOf(value) !== Object.prototype &&
|
||||||
|
Object.getPrototypeOf(value) !== null)
|
||||||
|
) {
|
||||||
|
return invalid(`${label} must be an object`);
|
||||||
|
}
|
||||||
|
const descriptors = Object.getOwnPropertyDescriptors(value);
|
||||||
|
if (
|
||||||
|
Object.values(descriptors).some(
|
||||||
|
(descriptor) =>
|
||||||
|
descriptor.get !== undefined ||
|
||||||
|
descriptor.set !== undefined ||
|
||||||
|
descriptor.enumerable !== true,
|
||||||
|
)
|
||||||
|
) {
|
||||||
|
return invalid(`${label} must contain enumerable data properties`);
|
||||||
|
}
|
||||||
|
return value as Record<string, unknown>;
|
||||||
|
}
|
||||||
|
|
||||||
|
function exactKeys(
|
||||||
|
value: object,
|
||||||
|
expected: readonly string[],
|
||||||
|
label: string,
|
||||||
|
): void {
|
||||||
|
const actual = Reflect.ownKeys(value);
|
||||||
|
const canonical = [...expected].sort();
|
||||||
|
if (
|
||||||
|
actual.some((key) => typeof key !== 'string') ||
|
||||||
|
actual.length !== canonical.length ||
|
||||||
|
actual
|
||||||
|
.map(String)
|
||||||
|
.sort()
|
||||||
|
.some((key, index) => key !== canonical[index])
|
||||||
|
) {
|
||||||
|
return invalid(`${label} shape is invalid`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function normalizeEnvironmentBundle(
|
||||||
|
value: EnvironmentBundle,
|
||||||
|
): Readonly<EnvironmentBundle> {
|
||||||
|
const bundle = dataObject(value, 'bundle');
|
||||||
|
exactKeys(bundle, ['entries', 'schema'], 'bundle');
|
||||||
|
if (bundle.schema !== ENVIRONMENT_BUNDLE_SCHEMA) {
|
||||||
|
return invalid('schema is invalid');
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
!Array.isArray(bundle.entries) ||
|
||||||
|
bundle.entries.length < 1 ||
|
||||||
|
bundle.entries.length > MAX_ENVIRONMENT_BUNDLE_ENTRIES
|
||||||
|
) {
|
||||||
|
return invalid('entry count is invalid');
|
||||||
|
}
|
||||||
|
const names = new Set<string>();
|
||||||
|
let totalBytes = 0;
|
||||||
|
const entries = bundle.entries.map((value, index) => {
|
||||||
|
const entry = dataObject(value, `entries[${index}]`);
|
||||||
|
exactKeys(entry, ['name', 'value'], `entries[${index}]`);
|
||||||
|
if (
|
||||||
|
typeof entry.name !== 'string' ||
|
||||||
|
!ENVIRONMENT_NAME_PATTERN.test(entry.name) ||
|
||||||
|
entry.name.startsWith('QL3_') ||
|
||||||
|
names.has(entry.name)
|
||||||
|
) {
|
||||||
|
return invalid(`entries[${index}].name is invalid or duplicated`);
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
typeof entry.value !== 'string' ||
|
||||||
|
entry.value.includes('\0') ||
|
||||||
|
Buffer.byteLength(entry.value, 'utf8') >
|
||||||
|
MAX_ENVIRONMENT_BUNDLE_VALUE_BYTES
|
||||||
|
) {
|
||||||
|
return invalid(`entries[${index}].value is invalid`);
|
||||||
|
}
|
||||||
|
names.add(entry.name);
|
||||||
|
totalBytes +=
|
||||||
|
Buffer.byteLength(entry.name, 'utf8') +
|
||||||
|
Buffer.byteLength(entry.value, 'utf8');
|
||||||
|
if (totalBytes > MAX_ENVIRONMENT_BUNDLE_TOTAL_BYTES) {
|
||||||
|
return invalid('environment byte budget exceeded');
|
||||||
|
}
|
||||||
|
return Object.freeze({ name: entry.name, value: entry.value });
|
||||||
|
});
|
||||||
|
entries.sort((left, right) =>
|
||||||
|
left.name < right.name ? -1 : left.name > right.name ? 1 : 0,
|
||||||
|
);
|
||||||
|
const normalized = Object.freeze({
|
||||||
|
schema: ENVIRONMENT_BUNDLE_SCHEMA,
|
||||||
|
entries: Object.freeze(entries),
|
||||||
|
});
|
||||||
|
if (
|
||||||
|
Buffer.byteLength(JSON.stringify(normalized), 'utf8') >
|
||||||
|
MAX_ENVIRONMENT_BUNDLE_ENCODED_BYTES
|
||||||
|
) {
|
||||||
|
return invalid('encoded byte budget exceeded');
|
||||||
|
}
|
||||||
|
return normalized;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function serializeEnvironmentBundle(value: EnvironmentBundle): string {
|
||||||
|
return JSON.stringify(normalizeEnvironmentBundle(value));
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parseEnvironmentBundle(
|
||||||
|
serialized: string | Uint8Array,
|
||||||
|
): Readonly<EnvironmentBundle> {
|
||||||
|
const bytes =
|
||||||
|
typeof serialized === 'string'
|
||||||
|
? Buffer.from(serialized, 'utf8')
|
||||||
|
: Buffer.from(serialized);
|
||||||
|
if (
|
||||||
|
bytes.byteLength < 2 ||
|
||||||
|
bytes.byteLength > MAX_ENVIRONMENT_BUNDLE_ENCODED_BYTES
|
||||||
|
) {
|
||||||
|
bytes.fill(0);
|
||||||
|
return invalid('encoded byte size is outside the allowed range');
|
||||||
|
}
|
||||||
|
let parsed: unknown;
|
||||||
|
try {
|
||||||
|
parsed = JSON.parse(bytes.toString('utf8')) as unknown;
|
||||||
|
} catch {
|
||||||
|
return invalid('payload is not valid JSON');
|
||||||
|
} finally {
|
||||||
|
bytes.fill(0);
|
||||||
|
}
|
||||||
|
return normalizeEnvironmentBundle(parsed as EnvironmentBundle);
|
||||||
|
}
|
||||||
@@ -32,6 +32,7 @@ export interface ClusterTaskExecutionRevisionContent {
|
|||||||
readonly planSchema: typeof CLUSTER_EXECUTION_PLAN_SCHEMA;
|
readonly planSchema: typeof CLUSTER_EXECUTION_PLAN_SCHEMA;
|
||||||
readonly command: LocalDispatchCommand;
|
readonly command: LocalDispatchCommand;
|
||||||
readonly environment: readonly LocalExecutionEnvironmentBinding[];
|
readonly environment: readonly LocalExecutionEnvironmentBinding[];
|
||||||
|
readonly environmentBundleRef?: string;
|
||||||
readonly workingDirectory?: string;
|
readonly workingDirectory?: string;
|
||||||
readonly timeoutMs?: number;
|
readonly timeoutMs?: number;
|
||||||
readonly placement?: RemoteWorkerPlacementSpec;
|
readonly placement?: RemoteWorkerPlacementSpec;
|
||||||
@@ -79,9 +80,7 @@ function revision(value: unknown): number {
|
|||||||
(value as number) < 1 ||
|
(value as number) < 1 ||
|
||||||
(value as number) > 2_147_483_647
|
(value as number) > 2_147_483_647
|
||||||
) {
|
) {
|
||||||
throw new InvalidClusterExecutionRevisionError(
|
throw new InvalidClusterExecutionRevisionError('sourceRevision is invalid');
|
||||||
'sourceRevision is invalid',
|
|
||||||
);
|
|
||||||
}
|
}
|
||||||
return value as number;
|
return value as number;
|
||||||
}
|
}
|
||||||
@@ -103,6 +102,7 @@ function normalizeContent(
|
|||||||
'command',
|
'command',
|
||||||
'createdAtMs',
|
'createdAtMs',
|
||||||
'environment',
|
'environment',
|
||||||
|
'environmentBundleRef',
|
||||||
'executorType',
|
'executorType',
|
||||||
'planSchema',
|
'planSchema',
|
||||||
'placement',
|
'placement',
|
||||||
@@ -155,6 +155,7 @@ function normalizeContent(
|
|||||||
}
|
}
|
||||||
let command: LocalDispatchCommand;
|
let command: LocalDispatchCommand;
|
||||||
let environment: readonly LocalExecutionEnvironmentBinding[];
|
let environment: readonly LocalExecutionEnvironmentBinding[];
|
||||||
|
let environmentBundleRef: string | undefined;
|
||||||
try {
|
try {
|
||||||
command = normalizeLocalDispatchCommand(value.command);
|
command = normalizeLocalDispatchCommand(value.command);
|
||||||
environment = createLocalExecutionContextRecipe({
|
environment = createLocalExecutionContextRecipe({
|
||||||
@@ -169,6 +170,16 @@ function normalizeContent(
|
|||||||
throw new Error('cross-project Secret reference');
|
throw new Error('cross-project Secret reference');
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if (value.environmentBundleRef !== undefined) {
|
||||||
|
const reference = parseSecretRef(value.environmentBundleRef);
|
||||||
|
if (
|
||||||
|
reference.projectId !== projectId ||
|
||||||
|
reference.version === undefined
|
||||||
|
) {
|
||||||
|
throw new Error('invalid environment bundle Secret reference');
|
||||||
|
}
|
||||||
|
environmentBundleRef = value.environmentBundleRef;
|
||||||
|
}
|
||||||
} catch {
|
} catch {
|
||||||
throw new InvalidClusterExecutionRevisionError(
|
throw new InvalidClusterExecutionRevisionError(
|
||||||
'command or environment is invalid',
|
'command or environment is invalid',
|
||||||
@@ -200,9 +211,10 @@ function normalizeContent(
|
|||||||
timeoutMs = value.timeoutMs;
|
timeoutMs = value.timeoutMs;
|
||||||
}
|
}
|
||||||
const createdAtMs = timestamp(value.createdAtMs);
|
const createdAtMs = timestamp(value.createdAtMs);
|
||||||
const placement = value.placement === undefined
|
const placement =
|
||||||
? undefined
|
value.placement === undefined
|
||||||
: effectiveRemoteWorkerPlacement(value.placement);
|
? undefined
|
||||||
|
: effectiveRemoteWorkerPlacement(value.placement);
|
||||||
const normalized = Object.freeze({
|
const normalized = Object.freeze({
|
||||||
projectId,
|
projectId,
|
||||||
taskId,
|
taskId,
|
||||||
@@ -213,20 +225,22 @@ function normalizeContent(
|
|||||||
planSchema: CLUSTER_EXECUTION_PLAN_SCHEMA,
|
planSchema: CLUSTER_EXECUTION_PLAN_SCHEMA,
|
||||||
command,
|
command,
|
||||||
environment,
|
environment,
|
||||||
|
...(environmentBundleRef === undefined ? {} : { environmentBundleRef }),
|
||||||
...(workingDirectory === undefined ? {} : { workingDirectory }),
|
...(workingDirectory === undefined ? {} : { workingDirectory }),
|
||||||
...(timeoutMs === undefined ? {} : { timeoutMs }),
|
...(timeoutMs === undefined ? {} : { timeoutMs }),
|
||||||
...(placement === undefined ? {} : { placement }),
|
...(placement === undefined ? {} : { placement }),
|
||||||
createdAtMs,
|
createdAtMs,
|
||||||
});
|
});
|
||||||
if (Buffer.byteLength(JSON.stringify(normalized), 'utf8') > MAX_CLUSTER_EXECUTION_PLAN_BYTES) {
|
if (
|
||||||
|
Buffer.byteLength(JSON.stringify(normalized), 'utf8') >
|
||||||
|
MAX_CLUSTER_EXECUTION_PLAN_BYTES
|
||||||
|
) {
|
||||||
throw new InvalidClusterExecutionRevisionError('plan byte budget exceeded');
|
throw new InvalidClusterExecutionRevisionError('plan byte budget exceeded');
|
||||||
}
|
}
|
||||||
return normalized;
|
return normalized;
|
||||||
}
|
}
|
||||||
|
|
||||||
function digest(
|
function digest(content: ClusterTaskExecutionRevisionContent): string {
|
||||||
content: ClusterTaskExecutionRevisionContent,
|
|
||||||
): string {
|
|
||||||
const { createdAtMs: _createdAtMs, ...immutable } = content;
|
const { createdAtMs: _createdAtMs, ...immutable } = content;
|
||||||
return createHash('sha256')
|
return createHash('sha256')
|
||||||
.update('qinglong.cluster-task-execution-revision.v1\0', 'utf8')
|
.update('qinglong.cluster-task-execution-revision.v1\0', 'utf8')
|
||||||
@@ -273,6 +287,9 @@ export function compileClusterCommandTaskDefinition(
|
|||||||
planSchema: CLUSTER_EXECUTION_PLAN_SCHEMA,
|
planSchema: CLUSTER_EXECUTION_PLAN_SCHEMA,
|
||||||
command: plan.command,
|
command: plan.command,
|
||||||
environment: plan.environment,
|
environment: plan.environment,
|
||||||
|
...(plan.environmentBundleRef === undefined
|
||||||
|
? {}
|
||||||
|
: { environmentBundleRef: plan.environmentBundleRef }),
|
||||||
...(plan.workingDirectory === undefined
|
...(plan.workingDirectory === undefined
|
||||||
? {}
|
? {}
|
||||||
: { workingDirectory: plan.workingDirectory }),
|
: { workingDirectory: plan.workingDirectory }),
|
||||||
|
|||||||
@@ -36,6 +36,7 @@ export interface CommandTaskExecutionPlan {
|
|||||||
readonly sourceContentDigest: string;
|
readonly sourceContentDigest: string;
|
||||||
readonly command: LocalDispatchCommand;
|
readonly command: LocalDispatchCommand;
|
||||||
readonly environment: readonly LocalExecutionEnvironmentBinding[];
|
readonly environment: readonly LocalExecutionEnvironmentBinding[];
|
||||||
|
readonly environmentBundleRef?: string;
|
||||||
readonly workingDirectory?: string;
|
readonly workingDirectory?: string;
|
||||||
readonly timeoutMs?: number;
|
readonly timeoutMs?: number;
|
||||||
readonly placement?: RemoteWorkerPlacementSpec;
|
readonly placement?: RemoteWorkerPlacementSpec;
|
||||||
@@ -112,13 +113,13 @@ export function parseTaskDefinitionRevisionRef(
|
|||||||
return Object.freeze({ revision, contentDigest });
|
return Object.freeze({ revision, contentDigest });
|
||||||
}
|
}
|
||||||
|
|
||||||
function canonicalRecord(definition: TaskDefinitionRecord): TaskDefinitionRecord {
|
function canonicalRecord(
|
||||||
|
definition: TaskDefinitionRecord,
|
||||||
|
): TaskDefinitionRecord {
|
||||||
try {
|
try {
|
||||||
return normalizeTaskDefinitionRecord(definition);
|
return normalizeTaskDefinitionRecord(definition);
|
||||||
} catch {
|
} catch {
|
||||||
throw new InvalidTaskDefinitionCompilationError(
|
throw new InvalidTaskDefinitionCompilationError('source record is invalid');
|
||||||
'source record is invalid',
|
|
||||||
);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -172,6 +173,7 @@ export function compileCommandTaskDefinition(
|
|||||||
const config = semanticSpec.config as unknown as Readonly<{
|
const config = semanticSpec.config as unknown as Readonly<{
|
||||||
command: LocalDispatchCommand;
|
command: LocalDispatchCommand;
|
||||||
environment: readonly LocalExecutionEnvironmentBinding[];
|
environment: readonly LocalExecutionEnvironmentBinding[];
|
||||||
|
environmentBundleRef?: string;
|
||||||
workingDirectory?: string;
|
workingDirectory?: string;
|
||||||
timeoutMs?: number;
|
timeoutMs?: number;
|
||||||
placement?: RemoteWorkerPlacementSpec;
|
placement?: RemoteWorkerPlacementSpec;
|
||||||
@@ -188,6 +190,9 @@ export function compileCommandTaskDefinition(
|
|||||||
sourceContentDigest: source.contentDigest,
|
sourceContentDigest: source.contentDigest,
|
||||||
command: config.command,
|
command: config.command,
|
||||||
environment: config.environment,
|
environment: config.environment,
|
||||||
|
...(config.environmentBundleRef === undefined
|
||||||
|
? {}
|
||||||
|
: { environmentBundleRef: config.environmentBundleRef }),
|
||||||
...(config.workingDirectory === undefined
|
...(config.workingDirectory === undefined
|
||||||
? {}
|
? {}
|
||||||
: { workingDirectory: config.workingDirectory }),
|
: { workingDirectory: config.workingDirectory }),
|
||||||
@@ -202,6 +207,9 @@ export function compileLocalCommandTaskDefinition(
|
|||||||
semanticRegistry: TaskSpecSemanticRegistry,
|
semanticRegistry: TaskSpecSemanticRegistry,
|
||||||
): LocalCommandTaskExecutionPlan {
|
): LocalCommandTaskExecutionPlan {
|
||||||
const source = compileCommandTaskDefinition(definition, semanticRegistry);
|
const source = compileCommandTaskDefinition(definition, semanticRegistry);
|
||||||
|
if (source.environmentBundleRef !== undefined) {
|
||||||
|
throw new UnsupportedTaskDefinitionCompilationError();
|
||||||
|
}
|
||||||
const contextRecipe = createLocalExecutionContextRecipe({
|
const contextRecipe = createLocalExecutionContextRecipe({
|
||||||
environment: source.environment,
|
environment: source.environment,
|
||||||
createdAtMs: source.createdAtMs,
|
createdAtMs: source.createdAtMs,
|
||||||
|
|||||||
@@ -196,9 +196,7 @@ function normalizeEnvironment(
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
if (bytes > MAX_COMMAND_TASK_ENVIRONMENT_BYTES) {
|
if (bytes > MAX_COMMAND_TASK_ENVIRONMENT_BYTES) {
|
||||||
throw new InvalidTaskSpecSemanticError(
|
throw new InvalidTaskSpecSemanticError('environment byte budget exceeded');
|
||||||
'environment byte budget exceeded',
|
|
||||||
);
|
|
||||||
}
|
}
|
||||||
environment.sort((left, right) =>
|
environment.sort((left, right) =>
|
||||||
(left as { name: string }).name.localeCompare(
|
(left as { name: string }).name.localeCompare(
|
||||||
@@ -215,14 +213,46 @@ function normalizeCommandConfig(
|
|||||||
exactKeys(
|
exactKeys(
|
||||||
config,
|
config,
|
||||||
['command'],
|
['command'],
|
||||||
['environment', 'placement', 'timeoutMs', 'workingDirectory'],
|
[
|
||||||
|
'environment',
|
||||||
|
'environmentBundleRef',
|
||||||
|
'placement',
|
||||||
|
'timeoutMs',
|
||||||
|
'workingDirectory',
|
||||||
|
],
|
||||||
'command config',
|
'command config',
|
||||||
);
|
);
|
||||||
const command = normalizeCommand(config.command);
|
const command = normalizeCommand(config.command);
|
||||||
const environment = normalizeEnvironment(config.environment ?? [], context.projectId);
|
const environment = normalizeEnvironment(
|
||||||
const placement = config.placement === undefined
|
config.environment ?? [],
|
||||||
? undefined
|
context.projectId,
|
||||||
: normalizeRemoteWorkerPlacement(config.placement);
|
);
|
||||||
|
let environmentBundleRef: string | undefined;
|
||||||
|
if (config.environmentBundleRef !== undefined) {
|
||||||
|
environmentBundleRef = boundedText(
|
||||||
|
config.environmentBundleRef,
|
||||||
|
'environmentBundleRef',
|
||||||
|
512,
|
||||||
|
);
|
||||||
|
let reference;
|
||||||
|
try {
|
||||||
|
reference = parseSecretRef(environmentBundleRef);
|
||||||
|
} catch {
|
||||||
|
throw new InvalidTaskSpecSemanticError('environmentBundleRef is invalid');
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
reference.projectId !== context.projectId ||
|
||||||
|
reference.version === undefined
|
||||||
|
) {
|
||||||
|
throw new InvalidTaskSpecSemanticError(
|
||||||
|
'environmentBundleRef must pin a version in the same Project',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const placement =
|
||||||
|
config.placement === undefined
|
||||||
|
? undefined
|
||||||
|
: normalizeRemoteWorkerPlacement(config.placement);
|
||||||
let workingDirectory: string | undefined;
|
let workingDirectory: string | undefined;
|
||||||
if (config.workingDirectory !== undefined) {
|
if (config.workingDirectory !== undefined) {
|
||||||
workingDirectory = boundedText(
|
workingDirectory = boundedText(
|
||||||
@@ -250,6 +280,7 @@ function normalizeCommandConfig(
|
|||||||
return Object.freeze({
|
return Object.freeze({
|
||||||
command,
|
command,
|
||||||
environment,
|
environment,
|
||||||
|
...(environmentBundleRef === undefined ? {} : { environmentBundleRef }),
|
||||||
...(placement === undefined
|
...(placement === undefined
|
||||||
? {}
|
? {}
|
||||||
: { placement: placement as unknown as TaskDefinitionJson }),
|
: { placement: placement as unknown as TaskDefinitionJson }),
|
||||||
@@ -268,10 +299,7 @@ const BUILT_IN_DESCRIPTORS: readonly TaskSpecSemanticDescriptor[] =
|
|||||||
]);
|
]);
|
||||||
|
|
||||||
export class TaskSpecSemanticRegistry {
|
export class TaskSpecSemanticRegistry {
|
||||||
readonly #descriptors: ReadonlyMap<
|
readonly #descriptors: ReadonlyMap<string, TaskSpecSemanticDescriptor>;
|
||||||
string,
|
|
||||||
TaskSpecSemanticDescriptor
|
|
||||||
>;
|
|
||||||
readonly #metadata: readonly TaskSpecSemanticMetadata[];
|
readonly #metadata: readonly TaskSpecSemanticMetadata[];
|
||||||
|
|
||||||
constructor(descriptors: readonly TaskSpecSemanticDescriptor[]) {
|
constructor(descriptors: readonly TaskSpecSemanticDescriptor[]) {
|
||||||
@@ -393,8 +421,5 @@ export function createTaskSpecSemanticRegistry(
|
|||||||
'extension descriptor uses the reserved qinglong namespace',
|
'extension descriptor uses the reserved qinglong namespace',
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
return new TaskSpecSemanticRegistry([
|
return new TaskSpecSemanticRegistry([...BUILT_IN_DESCRIPTORS, ...extensions]);
|
||||||
...BUILT_IN_DESCRIPTORS,
|
|
||||||
...extensions,
|
|
||||||
]);
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -44,15 +44,55 @@ function definition() {
|
|||||||
});
|
});
|
||||||
return {
|
return {
|
||||||
registry,
|
registry,
|
||||||
record: createTaskDefinitionRecord({
|
record: createTaskDefinitionRecord(
|
||||||
...command,
|
{
|
||||||
spec: registry.normalize({
|
...command,
|
||||||
projectId: command.projectId,
|
spec: registry.normalize({
|
||||||
taskId: command.taskId,
|
projectId: command.projectId,
|
||||||
kind: command.kind,
|
taskId: command.taskId,
|
||||||
spec: command.spec,
|
kind: command.kind,
|
||||||
}),
|
spec: command.spec,
|
||||||
}, 90),
|
}),
|
||||||
|
},
|
||||||
|
90,
|
||||||
|
),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function definitionWithBundle() {
|
||||||
|
const input = definition();
|
||||||
|
const environmentBundleRef = createSecretRef({
|
||||||
|
projectId: 'default',
|
||||||
|
name: 'legacy-env-bundle',
|
||||||
|
version: 4,
|
||||||
|
});
|
||||||
|
const spec = input.registry.normalize({
|
||||||
|
projectId: input.record.projectId,
|
||||||
|
taskId: input.record.taskId,
|
||||||
|
kind: input.record.kind,
|
||||||
|
spec: {
|
||||||
|
...input.record.spec,
|
||||||
|
config: { ...input.record.spec.config, environmentBundleRef },
|
||||||
|
},
|
||||||
|
});
|
||||||
|
return {
|
||||||
|
registry: input.registry,
|
||||||
|
environmentBundleRef,
|
||||||
|
record: createTaskDefinitionRecord(
|
||||||
|
{
|
||||||
|
projectId: input.record.projectId,
|
||||||
|
taskId: input.record.taskId,
|
||||||
|
expectedRevision: null,
|
||||||
|
mutationId: input.record.mutationId,
|
||||||
|
name: input.record.name,
|
||||||
|
kind: input.record.kind,
|
||||||
|
spec,
|
||||||
|
labels: input.record.labels,
|
||||||
|
enabled: input.record.enabled,
|
||||||
|
occurredAtMs: input.record.updatedAtMs,
|
||||||
|
},
|
||||||
|
input.record.createdAtMs,
|
||||||
|
),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -70,6 +110,17 @@ test('compiles one digest-bound remote Worker execution revision', () => {
|
|||||||
assert.deepEqual(normalizeClusterTaskExecutionRevision(revision), revision);
|
assert.deepEqual(normalizeClusterTaskExecutionRevision(revision), revision);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('carries only the pinned environment bundle reference into Cluster plans', () => {
|
||||||
|
const input = definitionWithBundle();
|
||||||
|
const revision = compileClusterCommandTaskDefinition(
|
||||||
|
input.record,
|
||||||
|
input.registry,
|
||||||
|
);
|
||||||
|
assert.equal(revision.environmentBundleRef, input.environmentBundleRef);
|
||||||
|
assert.equal(JSON.stringify(revision).includes('legacy env value'), false);
|
||||||
|
assert.deepEqual(normalizeClusterTaskExecutionRevision(revision), revision);
|
||||||
|
});
|
||||||
|
|
||||||
test('rejects digest drift and cross-Project Secret references', () => {
|
test('rejects digest drift and cross-Project Secret references', () => {
|
||||||
const input = definition();
|
const input = definition();
|
||||||
const revision = compileClusterCommandTaskDefinition(
|
const revision = compileClusterCommandTaskDefinition(
|
||||||
@@ -77,21 +128,25 @@ test('rejects digest drift and cross-Project Secret references', () => {
|
|||||||
input.registry,
|
input.registry,
|
||||||
);
|
);
|
||||||
assert.throws(
|
assert.throws(
|
||||||
() => normalizeClusterTaskExecutionRevision({
|
() =>
|
||||||
...revision,
|
normalizeClusterTaskExecutionRevision({
|
||||||
contentDigest: '0'.repeat(64),
|
...revision,
|
||||||
}),
|
contentDigest: '0'.repeat(64),
|
||||||
|
}),
|
||||||
InvalidClusterExecutionRevisionError,
|
InvalidClusterExecutionRevisionError,
|
||||||
);
|
);
|
||||||
assert.throws(
|
assert.throws(
|
||||||
() => normalizeClusterTaskExecutionRevision({
|
() =>
|
||||||
...revision,
|
normalizeClusterTaskExecutionRevision({
|
||||||
environment: [{
|
...revision,
|
||||||
kind: 'secret',
|
environment: [
|
||||||
name: 'TOKEN',
|
{
|
||||||
secretRef: createSecretRef({ projectId: 'another', name: 'TOKEN' }),
|
kind: 'secret',
|
||||||
}],
|
name: 'TOKEN',
|
||||||
}),
|
secretRef: createSecretRef({ projectId: 'another', name: 'TOKEN' }),
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}),
|
||||||
InvalidClusterExecutionRevisionError,
|
InvalidClusterExecutionRevisionError,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ const { test } = require('node:test');
|
|||||||
const {
|
const {
|
||||||
CLUSTER_LEGACY_ENV_MIGRATION_PLAN_SCHEMA,
|
CLUSTER_LEGACY_ENV_MIGRATION_PLAN_SCHEMA,
|
||||||
MAX_CLUSTER_LEGACY_ENV_EFFECTIVE_BYTES,
|
MAX_CLUSTER_LEGACY_ENV_EFFECTIVE_BYTES,
|
||||||
|
MAX_CLUSTER_LEGACY_ENV_EFFECTIVE_BINDINGS,
|
||||||
MAX_CLUSTER_LEGACY_ENV_SOURCE_ROWS,
|
MAX_CLUSTER_LEGACY_ENV_SOURCE_ROWS,
|
||||||
MAX_CLUSTER_LEGACY_ENV_TASKS,
|
MAX_CLUSTER_LEGACY_ENV_TASKS,
|
||||||
MAX_CLUSTER_LEGACY_ENV_TRIGGERS,
|
MAX_CLUSTER_LEGACY_ENV_TRIGGERS,
|
||||||
@@ -111,6 +112,15 @@ test('enforces source consistency and router-safe bounded targets', () => {
|
|||||||
sourceRowCount: MAX_CLUSTER_LEGACY_ENV_SOURCE_ROWS + 1,
|
sourceRowCount: MAX_CLUSTER_LEGACY_ENV_SOURCE_ROWS + 1,
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
source: {
|
||||||
|
...intent().source,
|
||||||
|
sourceRowCount: MAX_CLUSTER_LEGACY_ENV_EFFECTIVE_BINDINGS + 1,
|
||||||
|
activeRowCount: MAX_CLUSTER_LEGACY_ENV_EFFECTIVE_BINDINGS + 1,
|
||||||
|
disabledRowCount: 0,
|
||||||
|
effectiveBindingCount: MAX_CLUSTER_LEGACY_ENV_EFFECTIVE_BINDINGS + 1,
|
||||||
|
},
|
||||||
|
},
|
||||||
{
|
{
|
||||||
target: {
|
target: {
|
||||||
...intent().target,
|
...intent().target,
|
||||||
|
|||||||
@@ -0,0 +1,58 @@
|
|||||||
|
'use strict';
|
||||||
|
|
||||||
|
const assert = require('node:assert/strict');
|
||||||
|
const { test } = require('node:test');
|
||||||
|
const {
|
||||||
|
ENVIRONMENT_BUNDLE_SCHEMA,
|
||||||
|
InvalidEnvironmentBundleError,
|
||||||
|
parseEnvironmentBundle,
|
||||||
|
serializeEnvironmentBundle,
|
||||||
|
} = require('../dist/secret/environmentBundle');
|
||||||
|
|
||||||
|
test('canonicalizes one opaque environment bundle without external authority', () => {
|
||||||
|
const serialized = serializeEnvironmentBundle({
|
||||||
|
schema: ENVIRONMENT_BUNDLE_SCHEMA,
|
||||||
|
entries: [
|
||||||
|
{ name: 'TOKEN', value: 'secret' },
|
||||||
|
{ name: 'EMPTY', value: '' },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
assert.deepEqual(parseEnvironmentBundle(serialized), {
|
||||||
|
schema: ENVIRONMENT_BUNDLE_SCHEMA,
|
||||||
|
entries: [
|
||||||
|
{ name: 'EMPTY', value: '' },
|
||||||
|
{ name: 'TOKEN', value: 'secret' },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('rejects duplicate, reserved, widened and over-budget bundle entries', () => {
|
||||||
|
const values = [
|
||||||
|
{ schema: ENVIRONMENT_BUNDLE_SCHEMA, entries: [] },
|
||||||
|
{
|
||||||
|
schema: ENVIRONMENT_BUNDLE_SCHEMA,
|
||||||
|
entries: [
|
||||||
|
{ name: 'TOKEN', value: 'a' },
|
||||||
|
{ name: 'TOKEN', value: 'b' },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
schema: ENVIRONMENT_BUNDLE_SCHEMA,
|
||||||
|
entries: [{ name: 'QL3_TOKEN', value: 'a' }],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
schema: ENVIRONMENT_BUNDLE_SCHEMA,
|
||||||
|
entries: [{ name: 'TOKEN', value: 'x'.repeat(16 * 1024 + 1) }],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
schema: ENVIRONMENT_BUNDLE_SCHEMA,
|
||||||
|
entries: [{ name: 'TOKEN', value: 'a', secretRef: 'forbidden' }],
|
||||||
|
},
|
||||||
|
];
|
||||||
|
for (const value of values) {
|
||||||
|
assert.throws(
|
||||||
|
() => serializeEnvironmentBundle(value),
|
||||||
|
InvalidEnvironmentBundleError,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
@@ -14,6 +14,11 @@ const { createSecretRef } = require('../dist/secret/secretReference');
|
|||||||
const SESSION_ID = '018f0000-0000-7000-8000-000000000001';
|
const SESSION_ID = '018f0000-0000-7000-8000-000000000001';
|
||||||
const DIGEST = 'a'.repeat(64);
|
const DIGEST = 'a'.repeat(64);
|
||||||
const SECRET_REF = createSecretRef({ projectId: 'project-1', name: 'token' });
|
const SECRET_REF = createSecretRef({ projectId: 'project-1', name: 'token' });
|
||||||
|
const BUNDLE_REF = createSecretRef({
|
||||||
|
projectId: 'project-1',
|
||||||
|
name: 'legacy-env-bundle',
|
||||||
|
version: 7,
|
||||||
|
});
|
||||||
|
|
||||||
function command(overrides = {}) {
|
function command(overrides = {}) {
|
||||||
return {
|
return {
|
||||||
@@ -31,13 +36,14 @@ function command(overrides = {}) {
|
|||||||
leaseToken: 'worker_generated_lease_capability_0000000000000001',
|
leaseToken: 'worker_generated_lease_capability_0000000000000001',
|
||||||
expectedLeaseVersion: 4,
|
expectedLeaseVersion: 4,
|
||||||
secretRefs: [SECRET_REF],
|
secretRefs: [SECRET_REF],
|
||||||
|
environmentBundleRefs: [],
|
||||||
...overrides,
|
...overrides,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
test('creates a versioned request without duplicating path-bound identity', () => {
|
test('creates a versioned request without duplicating path-bound identity', () => {
|
||||||
const body = createRemoteWorkerSecretDeliveryRequestBody(command());
|
const body = createRemoteWorkerSecretDeliveryRequestBody(command());
|
||||||
assert.equal(body.schema, 'qinglong/remote-secret-delivery@v1');
|
assert.equal(body.schema, 'qinglong/remote-secret-delivery@v2');
|
||||||
assert.equal('workerId' in body, false);
|
assert.equal('workerId' in body, false);
|
||||||
assert.equal('workerSessionId' in body, false);
|
assert.equal('workerSessionId' in body, false);
|
||||||
assert.deepEqual(body.secretRefs, [SECRET_REF]);
|
assert.deepEqual(body.secretRefs, [SECRET_REF]);
|
||||||
@@ -45,67 +51,129 @@ test('creates a versioned request without duplicating path-bound identity', () =
|
|||||||
});
|
});
|
||||||
|
|
||||||
test('parses only an exact authority and ordered Secret set', () => {
|
test('parses only an exact authority and ordered Secret set', () => {
|
||||||
const response = createRemoteWorkerSecretDeliveryResponseBody({
|
const response = createRemoteWorkerSecretDeliveryResponseBody(
|
||||||
runId: 'run-1',
|
{
|
||||||
attemptId: 'attempt-1',
|
runId: 'run-1',
|
||||||
offerId: 'offer-1',
|
attemptId: 'attempt-1',
|
||||||
executionDigest: DIGEST,
|
offerId: 'offer-1',
|
||||||
values: [{ secretRef: SECRET_REF, value: 'private-value' }],
|
executionDigest: DIGEST,
|
||||||
}, [SECRET_REF]);
|
values: [{ secretRef: SECRET_REF, value: 'private-value' }],
|
||||||
|
environmentBundles: [],
|
||||||
|
},
|
||||||
|
{ secretRefs: [SECRET_REF], environmentBundleRefs: [] },
|
||||||
|
);
|
||||||
const parsed = parseRemoteWorkerSecretDeliveryResponse(
|
const parsed = parseRemoteWorkerSecretDeliveryResponse(
|
||||||
JSON.stringify(response),
|
JSON.stringify(response),
|
||||||
{
|
{
|
||||||
runId: 'run-1', attemptId: 'attempt-1', offerId: 'offer-1',
|
runId: 'run-1',
|
||||||
executionDigest: DIGEST, secretRefs: [SECRET_REF],
|
attemptId: 'attempt-1',
|
||||||
|
offerId: 'offer-1',
|
||||||
|
executionDigest: DIGEST,
|
||||||
|
secretRefs: [SECRET_REF],
|
||||||
|
environmentBundleRefs: [],
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
assert.deepEqual(parsed.values, [
|
assert.deepEqual(parsed.values, [
|
||||||
{ secretRef: SECRET_REF, value: 'private-value' },
|
{ secretRef: SECRET_REF, value: 'private-value' },
|
||||||
]);
|
]);
|
||||||
assert.throws(
|
assert.throws(
|
||||||
() => parseRemoteWorkerSecretDeliveryResponse(JSON.stringify(response), {
|
() =>
|
||||||
runId: 'run-other', attemptId: 'attempt-1', offerId: 'offer-1',
|
parseRemoteWorkerSecretDeliveryResponse(JSON.stringify(response), {
|
||||||
executionDigest: DIGEST, secretRefs: [SECRET_REF],
|
runId: 'run-other',
|
||||||
}),
|
attemptId: 'attempt-1',
|
||||||
|
offerId: 'offer-1',
|
||||||
|
executionDigest: DIGEST,
|
||||||
|
secretRefs: [SECRET_REF],
|
||||||
|
environmentBundleRefs: [],
|
||||||
|
}),
|
||||||
/authority does not match/,
|
/authority does not match/,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
test('rejects duplicate, cross-project and oversized delivery input', () => {
|
test('rejects duplicate, cross-project and oversized delivery input', () => {
|
||||||
assert.throws(
|
assert.throws(
|
||||||
() => normalizeRemoteWorkerSecretDeliveryCommand(command({
|
() =>
|
||||||
secretRefs: [SECRET_REF, SECRET_REF],
|
normalizeRemoteWorkerSecretDeliveryCommand(
|
||||||
})),
|
command({
|
||||||
|
secretRefs: [SECRET_REF, SECRET_REF],
|
||||||
|
}),
|
||||||
|
),
|
||||||
/secretRefs are invalid/,
|
/secretRefs are invalid/,
|
||||||
);
|
);
|
||||||
const foreign = createSecretRef({ projectId: 'project-2', name: 'token' });
|
const foreign = createSecretRef({ projectId: 'project-2', name: 'token' });
|
||||||
assert.throws(
|
assert.throws(
|
||||||
() => normalizeRemoteWorkerSecretDeliveryCommand(command({
|
() =>
|
||||||
secretRefs: [foreign],
|
normalizeRemoteWorkerSecretDeliveryCommand(
|
||||||
})),
|
command({
|
||||||
|
secretRefs: [foreign],
|
||||||
|
}),
|
||||||
|
),
|
||||||
/project is invalid/,
|
/project is invalid/,
|
||||||
);
|
);
|
||||||
assert.throws(
|
assert.throws(
|
||||||
() => parseRemoteWorkerSecretDeliveryResponse(
|
() =>
|
||||||
Buffer.alloc(MAX_REMOTE_SECRET_DELIVERY_RESPONSE_BYTES + 1),
|
parseRemoteWorkerSecretDeliveryResponse(
|
||||||
{
|
Buffer.alloc(MAX_REMOTE_SECRET_DELIVERY_RESPONSE_BYTES + 1),
|
||||||
runId: 'run-1', attemptId: 'attempt-1', offerId: 'offer-1',
|
{
|
||||||
executionDigest: DIGEST, secretRefs: [SECRET_REF],
|
runId: 'run-1',
|
||||||
},
|
attemptId: 'attempt-1',
|
||||||
),
|
offerId: 'offer-1',
|
||||||
|
executionDigest: DIGEST,
|
||||||
|
secretRefs: [SECRET_REF],
|
||||||
|
environmentBundleRefs: [],
|
||||||
|
},
|
||||||
|
),
|
||||||
/byte size/,
|
/byte size/,
|
||||||
);
|
);
|
||||||
const refs = Array.from({ length: 5 }, (_, index) =>
|
const refs = Array.from({ length: 5 }, (_, index) =>
|
||||||
createSecretRef({ projectId: 'project-1', name: `item-${index}` }));
|
createSecretRef({ projectId: 'project-1', name: `item-${index}` }),
|
||||||
|
);
|
||||||
assert.throws(
|
assert.throws(
|
||||||
() => createRemoteWorkerSecretDeliveryResponseBody({
|
() =>
|
||||||
runId: 'run-1', attemptId: 'attempt-1', offerId: 'offer-1',
|
createRemoteWorkerSecretDeliveryResponseBody(
|
||||||
executionDigest: DIGEST,
|
{
|
||||||
values: refs.map((secretRef) => ({
|
runId: 'run-1',
|
||||||
secretRef,
|
attemptId: 'attempt-1',
|
||||||
value: 'x'.repeat(16 * 1024),
|
offerId: 'offer-1',
|
||||||
})),
|
executionDigest: DIGEST,
|
||||||
}, refs),
|
values: refs.map((secretRef) => ({
|
||||||
|
secretRef,
|
||||||
|
value: 'x'.repeat(16 * 1024),
|
||||||
|
})),
|
||||||
|
environmentBundles: [],
|
||||||
|
},
|
||||||
|
{ secretRefs: refs, environmentBundleRefs: [] },
|
||||||
|
),
|
||||||
/byte budget/,
|
/byte budget/,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('keeps one environment bundle in a distinct bounded authority role', () => {
|
||||||
|
const normalized = normalizeRemoteWorkerSecretDeliveryCommand(
|
||||||
|
command({
|
||||||
|
secretRefs: [],
|
||||||
|
environmentBundleRefs: [BUNDLE_REF],
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert.deepEqual(normalized.environmentBundleRefs, [BUNDLE_REF]);
|
||||||
|
assert.throws(
|
||||||
|
() =>
|
||||||
|
normalizeRemoteWorkerSecretDeliveryCommand(
|
||||||
|
command({
|
||||||
|
secretRefs: [BUNDLE_REF],
|
||||||
|
environmentBundleRefs: [BUNDLE_REF],
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
/roles overlap/,
|
||||||
|
);
|
||||||
|
assert.throws(
|
||||||
|
() =>
|
||||||
|
normalizeRemoteWorkerSecretDeliveryCommand(
|
||||||
|
command({
|
||||||
|
secretRefs: [],
|
||||||
|
environmentBundleRefs: [],
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
/set is empty/,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
const assert = require('node:assert/strict');
|
const assert = require('node:assert/strict');
|
||||||
const { test } = require('node:test');
|
const { test } = require('node:test');
|
||||||
const { createLocalSecretRef } = require('../dist/secret/localSecret');
|
const { createLocalSecretRef } = require('../dist/secret/localSecret');
|
||||||
|
const { createSecretRef } = require('../dist/secret/secretReference');
|
||||||
const {
|
const {
|
||||||
BUILT_IN_COMMAND_TASK_SPEC_SCHEMA,
|
BUILT_IN_COMMAND_TASK_SPEC_SCHEMA,
|
||||||
InvalidTaskSpecSemanticError,
|
InvalidTaskSpecSemanticError,
|
||||||
@@ -231,17 +232,67 @@ test('canonicalizes an optional bounded Remote Worker PlacementSpec in command s
|
|||||||
preferred: [{ labels: { tier: 'edge' }, weight: 5 }],
|
preferred: [{ labels: { tier: 'edge' }, weight: 5 }],
|
||||||
});
|
});
|
||||||
assert.throws(
|
assert.throws(
|
||||||
() => registry.normalize(context({
|
() =>
|
||||||
|
registry.normalize(
|
||||||
|
context({
|
||||||
|
spec: {
|
||||||
|
schema: BUILT_IN_COMMAND_TASK_SPEC_SCHEMA,
|
||||||
|
config: {
|
||||||
|
command: { kind: 'argv', file: '/bin/echo', args: [] },
|
||||||
|
placement: {
|
||||||
|
required: {
|
||||||
|
runtimes: [{ name: 'node', versionRange: 'not-semver' }],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
InvalidTaskSpecSemanticError,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('accepts only a same-Project version-pinned environment bundle reference', () => {
|
||||||
|
const registry = createBuiltInTaskSpecSemanticRegistry();
|
||||||
|
const environmentBundleRef = createSecretRef({
|
||||||
|
projectId: 'default',
|
||||||
|
name: 'legacy-env-bundle',
|
||||||
|
version: 3,
|
||||||
|
});
|
||||||
|
const normalized = registry.normalize(
|
||||||
|
context({
|
||||||
spec: {
|
spec: {
|
||||||
schema: BUILT_IN_COMMAND_TASK_SPEC_SCHEMA,
|
schema: BUILT_IN_COMMAND_TASK_SPEC_SCHEMA,
|
||||||
config: {
|
config: {
|
||||||
command: { kind: 'argv', file: '/bin/echo', args: [] },
|
command: { kind: 'argv', file: '/bin/echo', args: [] },
|
||||||
placement: {
|
environmentBundleRef,
|
||||||
required: { runtimes: [{ name: 'node', versionRange: 'not-semver' }] },
|
|
||||||
},
|
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
})),
|
}),
|
||||||
InvalidTaskSpecSemanticError,
|
|
||||||
);
|
);
|
||||||
|
assert.equal(normalized.config.environmentBundleRef, environmentBundleRef);
|
||||||
|
for (const invalidRef of [
|
||||||
|
createSecretRef({ projectId: 'default', name: 'legacy-env-bundle' }),
|
||||||
|
createSecretRef({
|
||||||
|
projectId: 'other',
|
||||||
|
name: 'legacy-env-bundle',
|
||||||
|
version: 3,
|
||||||
|
}),
|
||||||
|
]) {
|
||||||
|
assert.throws(
|
||||||
|
() =>
|
||||||
|
registry.normalize(
|
||||||
|
context({
|
||||||
|
spec: {
|
||||||
|
schema: BUILT_IN_COMMAND_TASK_SPEC_SCHEMA,
|
||||||
|
config: {
|
||||||
|
command: { kind: 'argv', file: '/bin/echo', args: [] },
|
||||||
|
environmentBundleRef: invalidRef,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
/environmentBundleRef/,
|
||||||
|
);
|
||||||
|
}
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -1,8 +1,10 @@
|
|||||||
// Remote Execution owns bounded Secret and Artifact context materialization.
|
// Remote Execution owns bounded Secret and Artifact context materialization.
|
||||||
import {
|
import {
|
||||||
MAX_LOCAL_DISPATCH_ENVIRONMENT_BYTES,
|
MAX_LOCAL_DISPATCH_ENVIRONMENT_BYTES,
|
||||||
|
MAX_LOCAL_DISPATCH_ENVIRONMENT_ENTRIES,
|
||||||
MAX_LOCAL_DISPATCH_SECRET_REFS,
|
MAX_LOCAL_DISPATCH_SECRET_REFS,
|
||||||
} from '@qinglong/runtime-core/local-dispatch';
|
} from '@qinglong/runtime-core/local-dispatch';
|
||||||
|
import { parseEnvironmentBundle } from '@qinglong/runtime-core/environment-bundle';
|
||||||
import type { ClusterRemoteExecutionOffer } from '@qinglong/runtime-core/remote-dispatch';
|
import type { ClusterRemoteExecutionOffer } from '@qinglong/runtime-core/remote-dispatch';
|
||||||
import { createClusterRemoteExecutionOffer } from '@qinglong/runtime-core/remote-dispatch';
|
import { createClusterRemoteExecutionOffer } from '@qinglong/runtime-core/remote-dispatch';
|
||||||
import { assertRunDispatchId } from '@qinglong/runtime-core/run-dispatch-lease';
|
import { assertRunDispatchId } from '@qinglong/runtime-core/run-dispatch-lease';
|
||||||
@@ -17,20 +19,27 @@ export interface WorkerRemoteSecretResolution {
|
|||||||
secretRef: string;
|
secretRef: string;
|
||||||
value: string;
|
value: string;
|
||||||
}>[];
|
}>[];
|
||||||
|
readonly environmentBundles: readonly Readonly<{
|
||||||
|
secretRef: string;
|
||||||
|
value: string;
|
||||||
|
}>[];
|
||||||
readonly dispose?: () => Promise<void> | void;
|
readonly dispose?: () => Promise<void> | void;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface WorkerRemoteSecretEnvironmentProvider {
|
export interface WorkerRemoteSecretEnvironmentProvider {
|
||||||
resolve(request: Readonly<{
|
resolve(
|
||||||
projectId: string;
|
request: Readonly<{
|
||||||
taskId: string;
|
projectId: string;
|
||||||
taskRevision: string;
|
taskId: string;
|
||||||
runId: string;
|
taskRevision: string;
|
||||||
attemptId: string;
|
runId: string;
|
||||||
offerId: string;
|
attemptId: string;
|
||||||
executionDigest: string;
|
offerId: string;
|
||||||
secretRefs: readonly string[];
|
executionDigest: string;
|
||||||
}>): Promise<WorkerRemoteSecretResolution | undefined>;
|
secretRefs: readonly string[];
|
||||||
|
environmentBundleRefs: readonly string[];
|
||||||
|
}>,
|
||||||
|
): Promise<WorkerRemoteSecretResolution | undefined>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface WorkerRemoteLogArtifactPreparation {
|
export interface WorkerRemoteLogArtifactPreparation {
|
||||||
@@ -42,12 +51,14 @@ export interface WorkerRemoteLogArtifactPreparation {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export interface WorkerRemoteLogArtifactAllocator {
|
export interface WorkerRemoteLogArtifactAllocator {
|
||||||
prepare(request: Readonly<{
|
prepare(
|
||||||
projectId: string;
|
request: Readonly<{
|
||||||
runId: string;
|
projectId: string;
|
||||||
attemptId: string;
|
runId: string;
|
||||||
offerId: string;
|
attemptId: string;
|
||||||
}>): Promise<WorkerRemoteLogArtifactPreparation | undefined>;
|
offerId: string;
|
||||||
|
}>,
|
||||||
|
): Promise<WorkerRemoteLogArtifactPreparation | undefined>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface BoundedWorkerRemoteExecutionContextMaterializerOptions {
|
export interface BoundedWorkerRemoteExecutionContextMaterializerOptions {
|
||||||
@@ -86,11 +97,14 @@ function environmentValue(value: unknown): string {
|
|||||||
async function disposeQuietly(
|
async function disposeQuietly(
|
||||||
operation: (() => Promise<void> | void) | undefined,
|
operation: (() => Promise<void> | void) | undefined,
|
||||||
): Promise<void> {
|
): Promise<void> {
|
||||||
await Promise.resolve().then(() => operation?.()).catch(() => undefined);
|
await Promise.resolve()
|
||||||
|
.then(() => operation?.())
|
||||||
|
.catch(() => undefined);
|
||||||
}
|
}
|
||||||
|
|
||||||
export class BoundedWorkerRemoteExecutionContextMaterializer
|
export class BoundedWorkerRemoteExecutionContextMaterializer
|
||||||
implements WorkerRemoteExecutionContextMaterializer {
|
implements WorkerRemoteExecutionContextMaterializer
|
||||||
|
{
|
||||||
private readonly artifacts: WorkerRemoteLogArtifactAllocator;
|
private readonly artifacts: WorkerRemoteLogArtifactAllocator;
|
||||||
private readonly secrets?: WorkerRemoteSecretEnvironmentProvider;
|
private readonly secrets?: WorkerRemoteSecretEnvironmentProvider;
|
||||||
|
|
||||||
@@ -109,9 +123,11 @@ export class BoundedWorkerRemoteExecutionContextMaterializer
|
|||||||
this.secrets = options.secrets;
|
this.secrets = options.secrets;
|
||||||
}
|
}
|
||||||
|
|
||||||
async prepare(input: Readonly<{
|
async prepare(
|
||||||
offer: ClusterRemoteExecutionOffer;
|
input: Readonly<{
|
||||||
}>): Promise<MaterializedWorkerRemoteExecutionContext> {
|
offer: ClusterRemoteExecutionOffer;
|
||||||
|
}>,
|
||||||
|
): Promise<MaterializedWorkerRemoteExecutionContext> {
|
||||||
let offer: ClusterRemoteExecutionOffer;
|
let offer: ClusterRemoteExecutionOffer;
|
||||||
try {
|
try {
|
||||||
offer = createClusterRemoteExecutionOffer(input?.offer);
|
offer = createClusterRemoteExecutionOffer(input?.offer);
|
||||||
@@ -122,9 +138,17 @@ export class BoundedWorkerRemoteExecutionContextMaterializer
|
|||||||
}
|
}
|
||||||
const bindings = offer.executionRevision.environment;
|
const bindings = offer.executionRevision.environment;
|
||||||
const secretRefs = Object.freeze([
|
const secretRefs = Object.freeze([
|
||||||
...new Set(bindings.flatMap((binding) =>
|
...new Set(
|
||||||
binding.kind === 'secret' ? [binding.secretRef] : [])),
|
bindings.flatMap((binding) =>
|
||||||
|
binding.kind === 'secret' ? [binding.secretRef] : [],
|
||||||
|
),
|
||||||
|
),
|
||||||
]);
|
]);
|
||||||
|
const environmentBundleRefs = Object.freeze(
|
||||||
|
offer.executionRevision.environmentBundleRef === undefined
|
||||||
|
? []
|
||||||
|
: [offer.executionRevision.environmentBundleRef],
|
||||||
|
);
|
||||||
if (secretRefs.length > MAX_LOCAL_DISPATCH_SECRET_REFS) {
|
if (secretRefs.length > MAX_LOCAL_DISPATCH_SECRET_REFS) {
|
||||||
throw new WorkerRemoteExecutionMaterializationError(
|
throw new WorkerRemoteExecutionMaterializationError(
|
||||||
'environment_budget_exceeded',
|
'environment_budget_exceeded',
|
||||||
@@ -132,23 +156,26 @@ export class BoundedWorkerRemoteExecutionContextMaterializer
|
|||||||
}
|
}
|
||||||
let secretResolution: WorkerRemoteSecretResolution | undefined;
|
let secretResolution: WorkerRemoteSecretResolution | undefined;
|
||||||
const secretByRef = new Map<string, string>();
|
const secretByRef = new Map<string, string>();
|
||||||
if (secretRefs.length > 0) {
|
if (secretRefs.length > 0 || environmentBundleRefs.length > 0) {
|
||||||
if (!this.secrets) {
|
if (!this.secrets) {
|
||||||
throw new WorkerRemoteExecutionMaterializationError(
|
throw new WorkerRemoteExecutionMaterializationError(
|
||||||
'secret_unavailable',
|
'secret_unavailable',
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
secretResolution = await this.secrets.resolve(Object.freeze({
|
secretResolution = await this.secrets.resolve(
|
||||||
projectId: offer.candidate.projectId,
|
Object.freeze({
|
||||||
taskId: offer.candidate.taskId,
|
projectId: offer.candidate.projectId,
|
||||||
taskRevision: offer.candidate.taskRevision,
|
taskId: offer.candidate.taskId,
|
||||||
runId: offer.candidate.runId,
|
taskRevision: offer.candidate.taskRevision,
|
||||||
attemptId: offer.candidate.attemptId,
|
runId: offer.candidate.runId,
|
||||||
offerId: offer.offerId,
|
attemptId: offer.candidate.attemptId,
|
||||||
executionDigest: offer.executionDigest,
|
offerId: offer.offerId,
|
||||||
secretRefs,
|
executionDigest: offer.executionDigest,
|
||||||
}));
|
secretRefs,
|
||||||
|
environmentBundleRefs,
|
||||||
|
}),
|
||||||
|
);
|
||||||
} catch {
|
} catch {
|
||||||
throw new WorkerRemoteExecutionMaterializationError(
|
throw new WorkerRemoteExecutionMaterializationError(
|
||||||
'secret_unavailable',
|
'secret_unavailable',
|
||||||
@@ -160,10 +187,17 @@ export class BoundedWorkerRemoteExecutionContextMaterializer
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
if (
|
if (
|
||||||
Object.keys(secretResolution).some((key) =>
|
Object.keys(secretResolution).some(
|
||||||
key !== 'values' && key !== 'dispose') ||
|
(key) =>
|
||||||
|
key !== 'values' &&
|
||||||
|
key !== 'environmentBundles' &&
|
||||||
|
key !== 'dispose',
|
||||||
|
) ||
|
||||||
!Array.isArray(secretResolution.values) ||
|
!Array.isArray(secretResolution.values) ||
|
||||||
secretResolution.values.length !== secretRefs.length ||
|
secretResolution.values.length !== secretRefs.length ||
|
||||||
|
!Array.isArray(secretResolution.environmentBundles) ||
|
||||||
|
secretResolution.environmentBundles.length !==
|
||||||
|
environmentBundleRefs.length ||
|
||||||
(secretResolution.dispose !== undefined &&
|
(secretResolution.dispose !== undefined &&
|
||||||
typeof secretResolution.dispose !== 'function')
|
typeof secretResolution.dispose !== 'function')
|
||||||
) {
|
) {
|
||||||
@@ -198,36 +232,89 @@ export class BoundedWorkerRemoteExecutionContextMaterializer
|
|||||||
let environmentBytes = 0;
|
let environmentBytes = 0;
|
||||||
let environment: MaterializedWorkerRemoteExecutionContext['environment'];
|
let environment: MaterializedWorkerRemoteExecutionContext['environment'];
|
||||||
try {
|
try {
|
||||||
environment = Object.freeze(bindings.map((binding) => {
|
const names = new Set<string>();
|
||||||
const value = binding.kind === 'public'
|
const materialized = bindings.map((binding) => {
|
||||||
? binding.value
|
const value =
|
||||||
: secretByRef.get(binding.secretRef);
|
binding.kind === 'public'
|
||||||
|
? binding.value
|
||||||
|
: secretByRef.get(binding.secretRef);
|
||||||
if (value === undefined) {
|
if (value === undefined) {
|
||||||
throw new WorkerRemoteExecutionMaterializationError(
|
throw new WorkerRemoteExecutionMaterializationError(
|
||||||
'secret_response_invalid',
|
'secret_response_invalid',
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
environmentBytes += Buffer.byteLength(binding.name, 'utf8') +
|
environmentBytes +=
|
||||||
|
Buffer.byteLength(binding.name, 'utf8') +
|
||||||
Buffer.byteLength(value, 'utf8');
|
Buffer.byteLength(value, 'utf8');
|
||||||
|
names.add(binding.name);
|
||||||
if (environmentBytes > MAX_LOCAL_DISPATCH_ENVIRONMENT_BYTES) {
|
if (environmentBytes > MAX_LOCAL_DISPATCH_ENVIRONMENT_BYTES) {
|
||||||
throw new WorkerRemoteExecutionMaterializationError(
|
throw new WorkerRemoteExecutionMaterializationError(
|
||||||
'environment_budget_exceeded',
|
'environment_budget_exceeded',
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
return Object.freeze({ name: binding.name, value });
|
return Object.freeze({ name: binding.name, value });
|
||||||
}));
|
});
|
||||||
|
for (const entry of secretResolution?.environmentBundles ?? []) {
|
||||||
|
if (
|
||||||
|
!entry ||
|
||||||
|
typeof entry !== 'object' ||
|
||||||
|
Object.keys(entry).length !== 2 ||
|
||||||
|
!Object.hasOwn(entry, 'secretRef') ||
|
||||||
|
!Object.hasOwn(entry, 'value') ||
|
||||||
|
typeof entry.secretRef !== 'string' ||
|
||||||
|
!environmentBundleRefs.includes(entry.secretRef) ||
|
||||||
|
typeof entry.value !== 'string'
|
||||||
|
) {
|
||||||
|
throw new WorkerRemoteExecutionMaterializationError(
|
||||||
|
'secret_response_invalid',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let bundle;
|
||||||
|
try {
|
||||||
|
bundle = parseEnvironmentBundle(entry.value);
|
||||||
|
} catch {
|
||||||
|
throw new WorkerRemoteExecutionMaterializationError(
|
||||||
|
'secret_response_invalid',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
for (const binding of bundle.entries) {
|
||||||
|
if (names.has(binding.name)) {
|
||||||
|
throw new WorkerRemoteExecutionMaterializationError(
|
||||||
|
'secret_response_invalid',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
names.add(binding.name);
|
||||||
|
environmentBytes +=
|
||||||
|
Buffer.byteLength(binding.name, 'utf8') +
|
||||||
|
Buffer.byteLength(binding.value, 'utf8');
|
||||||
|
if (
|
||||||
|
materialized.length >= MAX_LOCAL_DISPATCH_ENVIRONMENT_ENTRIES ||
|
||||||
|
environmentBytes > MAX_LOCAL_DISPATCH_ENVIRONMENT_BYTES
|
||||||
|
) {
|
||||||
|
throw new WorkerRemoteExecutionMaterializationError(
|
||||||
|
'environment_budget_exceeded',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
materialized.push(
|
||||||
|
Object.freeze({ name: binding.name, value: binding.value }),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
environment = Object.freeze(materialized);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
await disposeQuietly(secretResolution?.dispose);
|
await disposeQuietly(secretResolution?.dispose);
|
||||||
throw error;
|
throw error;
|
||||||
}
|
}
|
||||||
let artifact: WorkerRemoteLogArtifactPreparation | undefined;
|
let artifact: WorkerRemoteLogArtifactPreparation | undefined;
|
||||||
try {
|
try {
|
||||||
artifact = await this.artifacts.prepare(Object.freeze({
|
artifact = await this.artifacts.prepare(
|
||||||
projectId: offer.candidate.projectId,
|
Object.freeze({
|
||||||
runId: offer.candidate.runId,
|
projectId: offer.candidate.projectId,
|
||||||
attemptId: offer.candidate.attemptId,
|
runId: offer.candidate.runId,
|
||||||
offerId: offer.offerId,
|
attemptId: offer.candidate.attemptId,
|
||||||
}));
|
offerId: offer.offerId,
|
||||||
|
}),
|
||||||
|
);
|
||||||
} catch {
|
} catch {
|
||||||
await disposeQuietly(secretResolution?.dispose);
|
await disposeQuietly(secretResolution?.dispose);
|
||||||
throw new WorkerRemoteExecutionMaterializationError(
|
throw new WorkerRemoteExecutionMaterializationError(
|
||||||
|
|||||||
+37
-11
@@ -36,7 +36,8 @@ export interface WorkerRemoteSecretHttpsProviderOptions {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export class WorkerRemoteSecretHttpsProvider
|
export class WorkerRemoteSecretHttpsProvider
|
||||||
implements WorkerRemoteSecretEnvironmentProvider {
|
implements WorkerRemoteSecretEnvironmentProvider
|
||||||
|
{
|
||||||
private readonly client: Pick<WorkerIngressHttpsClient, 'postJson'>;
|
private readonly client: Pick<WorkerIngressHttpsClient, 'postJson'>;
|
||||||
private readonly inbox: Pick<WorkerRemoteExecutionInbox, 'readOffer'>;
|
private readonly inbox: Pick<WorkerRemoteExecutionInbox, 'readOffer'>;
|
||||||
|
|
||||||
@@ -45,13 +46,15 @@ export class WorkerRemoteSecretHttpsProvider
|
|||||||
!options ||
|
!options ||
|
||||||
typeof options.client?.postJson !== 'function' ||
|
typeof options.client?.postJson !== 'function' ||
|
||||||
typeof options.inbox?.readOffer !== 'function'
|
typeof options.inbox?.readOffer !== 'function'
|
||||||
) throw new WorkerRemoteSecretHttpsProviderError('invalid_configuration');
|
)
|
||||||
|
throw new WorkerRemoteSecretHttpsProviderError('invalid_configuration');
|
||||||
this.client = options.client;
|
this.client = options.client;
|
||||||
this.inbox = options.inbox;
|
this.inbox = options.inbox;
|
||||||
}
|
}
|
||||||
|
|
||||||
async resolve(request: Parameters<WorkerRemoteSecretEnvironmentProvider['resolve']>[0])
|
async resolve(
|
||||||
: Promise<WorkerRemoteSecretResolution | undefined> {
|
request: Parameters<WorkerRemoteSecretEnvironmentProvider['resolve']>[0],
|
||||||
|
): Promise<WorkerRemoteSecretResolution | undefined> {
|
||||||
let record;
|
let record;
|
||||||
try {
|
try {
|
||||||
record = await this.inbox.readOffer(request.offerId);
|
record = await this.inbox.readOffer(request.offerId);
|
||||||
@@ -68,9 +71,17 @@ export class WorkerRemoteSecretHttpsProvider
|
|||||||
throw new WorkerRemoteSecretHttpsProviderError('authority_mismatch');
|
throw new WorkerRemoteSecretHttpsProviderError('authority_mismatch');
|
||||||
}
|
}
|
||||||
const expectedRefs = Object.freeze([
|
const expectedRefs = Object.freeze([
|
||||||
...new Set(offer.executionRevision.environment.flatMap((binding) =>
|
...new Set(
|
||||||
binding.kind === 'secret' ? [binding.secretRef] : [])),
|
offer.executionRevision.environment.flatMap((binding) =>
|
||||||
|
binding.kind === 'secret' ? [binding.secretRef] : [],
|
||||||
|
),
|
||||||
|
),
|
||||||
]);
|
]);
|
||||||
|
const expectedEnvironmentBundleRefs = Object.freeze(
|
||||||
|
offer.executionRevision.environmentBundleRef === undefined
|
||||||
|
? []
|
||||||
|
: [offer.executionRevision.environmentBundleRef],
|
||||||
|
);
|
||||||
if (
|
if (
|
||||||
offer.offerId !== request.offerId ||
|
offer.offerId !== request.offerId ||
|
||||||
offer.executionDigest !== request.executionDigest ||
|
offer.executionDigest !== request.executionDigest ||
|
||||||
@@ -79,10 +90,14 @@ export class WorkerRemoteSecretHttpsProvider
|
|||||||
offer.candidate.taskRevision !== request.taskRevision ||
|
offer.candidate.taskRevision !== request.taskRevision ||
|
||||||
offer.candidate.runId !== request.runId ||
|
offer.candidate.runId !== request.runId ||
|
||||||
offer.candidate.attemptId !== request.attemptId ||
|
offer.candidate.attemptId !== request.attemptId ||
|
||||||
JSON.stringify(expectedRefs) !== JSON.stringify(request.secretRefs)
|
JSON.stringify(expectedRefs) !== JSON.stringify(request.secretRefs) ||
|
||||||
) throw new WorkerRemoteSecretHttpsProviderError('authority_mismatch');
|
JSON.stringify(expectedEnvironmentBundleRefs) !==
|
||||||
|
JSON.stringify(request.environmentBundleRefs)
|
||||||
|
)
|
||||||
|
throw new WorkerRemoteSecretHttpsProviderError('authority_mismatch');
|
||||||
|
|
||||||
const path = `/api/v3/worker-ingress/workers/${offer.worker.workerId}` +
|
const path =
|
||||||
|
`/api/v3/worker-ingress/workers/${offer.worker.workerId}` +
|
||||||
`/sessions/${offer.worker.sessionId}/secrets`;
|
`/sessions/${offer.worker.sessionId}/secrets`;
|
||||||
const body = createRemoteWorkerSecretDeliveryRequestBody({
|
const body = createRemoteWorkerSecretDeliveryRequestBody({
|
||||||
workerId: offer.worker.workerId,
|
workerId: offer.worker.workerId,
|
||||||
@@ -99,6 +114,7 @@ export class WorkerRemoteSecretHttpsProvider
|
|||||||
leaseToken: offer.leaseToken,
|
leaseToken: offer.leaseToken,
|
||||||
expectedLeaseVersion: offer.lease.version,
|
expectedLeaseVersion: offer.lease.version,
|
||||||
secretRefs: expectedRefs,
|
secretRefs: expectedRefs,
|
||||||
|
environmentBundleRefs: expectedEnvironmentBundleRefs,
|
||||||
});
|
});
|
||||||
let serialized: Uint8Array;
|
let serialized: Uint8Array;
|
||||||
try {
|
try {
|
||||||
@@ -119,11 +135,21 @@ export class WorkerRemoteSecretHttpsProvider
|
|||||||
offerId: offer.offerId,
|
offerId: offer.offerId,
|
||||||
executionDigest: offer.executionDigest,
|
executionDigest: offer.executionDigest,
|
||||||
secretRefs: expectedRefs,
|
secretRefs: expectedRefs,
|
||||||
|
environmentBundleRefs: expectedEnvironmentBundleRefs,
|
||||||
});
|
});
|
||||||
const values = Object.freeze(delivered.values.map((entry) =>
|
const values = Object.freeze(
|
||||||
Object.freeze({ secretRef: entry.secretRef, value: entry.value })));
|
delivered.values.map((entry) =>
|
||||||
|
Object.freeze({ secretRef: entry.secretRef, value: entry.value }),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
const environmentBundles = Object.freeze(
|
||||||
|
delivered.environmentBundles.map((entry) =>
|
||||||
|
Object.freeze({ secretRef: entry.secretRef, value: entry.value }),
|
||||||
|
),
|
||||||
|
);
|
||||||
return Object.freeze({
|
return Object.freeze({
|
||||||
values,
|
values,
|
||||||
|
environmentBundles,
|
||||||
dispose() {
|
dispose() {
|
||||||
// JavaScript strings cannot be zeroized. Drop all retained references;
|
// JavaScript strings cannot be zeroized. Drop all retained references;
|
||||||
// the transport bytes were already scrubbed by the parser.
|
// the transport bytes were already scrubbed by the parser.
|
||||||
|
|||||||
+3
-1
@@ -17,7 +17,9 @@ const MAX_TLS_MATERIAL_BYTES = 1024 * 1024;
|
|||||||
const MAX_REQUEST_BYTES = 4096;
|
const MAX_REQUEST_BYTES = 4096;
|
||||||
const HARD_MAX_REQUEST_BYTES = 64 * 1024;
|
const HARD_MAX_REQUEST_BYTES = 64 * 1024;
|
||||||
const HARD_MAX_STREAM_REQUEST_BYTES = 64 * 1024 * 1024 + 4 * 1024 + 4;
|
const HARD_MAX_STREAM_REQUEST_BYTES = 64 * 1024 * 1024 + 4 * 1024 + 4;
|
||||||
const MAX_RESPONSE_BYTES = 128 * 1024;
|
// JSON routes retain their own smaller requested caps. The shared ceiling must
|
||||||
|
// also admit one bounded environment bundle response.
|
||||||
|
const MAX_RESPONSE_BYTES = 256 * 1024;
|
||||||
const CREDENTIAL_POOL_KEY = Symbol('qinglong.worker-ingress-credential-pool-key');
|
const CREDENTIAL_POOL_KEY = Symbol('qinglong.worker-ingress-credential-pool-key');
|
||||||
|
|
||||||
export const WORKER_INGRESS_ARTIFACT_CONTENT_TYPE =
|
export const WORKER_INGRESS_ARTIFACT_CONTENT_TYPE =
|
||||||
|
|||||||
@@ -11,9 +11,11 @@ const {
|
|||||||
const {
|
const {
|
||||||
digestRunDispatchLeaseToken,
|
digestRunDispatchLeaseToken,
|
||||||
} = require('@qinglong/runtime-core/run-dispatch-lease');
|
} = require('@qinglong/runtime-core/run-dispatch-lease');
|
||||||
|
const { createSecretRef } = require('@qinglong/runtime-core/secret-reference');
|
||||||
const {
|
const {
|
||||||
createSecretRef,
|
ENVIRONMENT_BUNDLE_SCHEMA,
|
||||||
} = require('@qinglong/runtime-core/secret-reference');
|
serializeEnvironmentBundle,
|
||||||
|
} = require('@qinglong/runtime-core/environment-bundle');
|
||||||
const {
|
const {
|
||||||
BoundedWorkerRemoteExecutionContextMaterializer,
|
BoundedWorkerRemoteExecutionContextMaterializer,
|
||||||
} = require('../dist/remote-execution/remoteOfferDeliveryEntrypoint');
|
} = require('../dist/remote-execution/remoteOfferDeliveryEntrypoint');
|
||||||
@@ -27,7 +29,7 @@ function secret(name) {
|
|||||||
return createSecretRef({ projectId: 'project-1', name });
|
return createSecretRef({ projectId: 'project-1', name });
|
||||||
}
|
}
|
||||||
|
|
||||||
function offer(environment) {
|
function offer(environment, environmentBundleRef) {
|
||||||
const executionRevision = createClusterTaskExecutionRevision({
|
const executionRevision = createClusterTaskExecutionRevision({
|
||||||
projectId: 'project-1',
|
projectId: 'project-1',
|
||||||
taskId: 'task-1',
|
taskId: 'task-1',
|
||||||
@@ -38,6 +40,7 @@ function offer(environment) {
|
|||||||
planSchema: 'qinglong/command-execution@v1',
|
planSchema: 'qinglong/command-execution@v1',
|
||||||
command: { kind: 'argv', file: '/bin/true', args: [] },
|
command: { kind: 'argv', file: '/bin/true', args: [] },
|
||||||
environment,
|
environment,
|
||||||
|
...(environmentBundleRef === undefined ? {} : { environmentBundleRef }),
|
||||||
createdAtMs: 1,
|
createdAtMs: 1,
|
||||||
});
|
});
|
||||||
return createClusterRemoteExecutionOffer({
|
return createClusterRemoteExecutionOffer({
|
||||||
@@ -100,7 +103,10 @@ test('resolves deduplicated Secrets before allocating one Attempt log', async ()
|
|||||||
secretRequest = request;
|
secretRequest = request;
|
||||||
return {
|
return {
|
||||||
values: [{ secretRef, value: 'resolved-value' }],
|
values: [{ secretRef, value: 'resolved-value' }],
|
||||||
dispose() { events.push('dispose-secrets'); },
|
environmentBundles: [],
|
||||||
|
dispose() {
|
||||||
|
events.push('dispose-secrets');
|
||||||
|
},
|
||||||
};
|
};
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
@@ -110,8 +116,13 @@ test('resolves deduplicated Secrets before allocating one Attempt log', async ()
|
|||||||
artifactRequest = request;
|
artifactRequest = request;
|
||||||
return {
|
return {
|
||||||
logArtifactId: 'remote-log-1',
|
logArtifactId: 'remote-log-1',
|
||||||
takeOutput() { events.push('take-output'); return output; },
|
takeOutput() {
|
||||||
release() { events.push('release-artifact'); },
|
events.push('take-output');
|
||||||
|
return output;
|
||||||
|
},
|
||||||
|
release() {
|
||||||
|
events.push('release-artifact');
|
||||||
|
},
|
||||||
};
|
};
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
@@ -129,6 +140,7 @@ test('resolves deduplicated Secrets before allocating one Attempt log', async ()
|
|||||||
offerId: 'offer-materializer-1',
|
offerId: 'offer-materializer-1',
|
||||||
executionDigest: acceptedOffer.executionDigest,
|
executionDigest: acceptedOffer.executionDigest,
|
||||||
secretRefs: [secretRef],
|
secretRefs: [secretRef],
|
||||||
|
environmentBundleRefs: [],
|
||||||
});
|
});
|
||||||
assert.deepEqual(artifactRequest, {
|
assert.deepEqual(artifactRequest, {
|
||||||
projectId: 'project-1',
|
projectId: 'project-1',
|
||||||
@@ -136,7 +148,10 @@ test('resolves deduplicated Secrets before allocating one Attempt log', async ()
|
|||||||
attemptId: 'attempt-1',
|
attemptId: 'attempt-1',
|
||||||
offerId: 'offer-materializer-1',
|
offerId: 'offer-materializer-1',
|
||||||
});
|
});
|
||||||
assert.equal(JSON.stringify([secretRequest, artifactRequest]).includes(LEASE_TOKEN), false);
|
assert.equal(
|
||||||
|
JSON.stringify([secretRequest, artifactRequest]).includes(LEASE_TOKEN),
|
||||||
|
false,
|
||||||
|
);
|
||||||
assert.deepEqual(context.environment, [
|
assert.deepEqual(context.environment, [
|
||||||
{ name: 'PUBLIC', value: 'visible' },
|
{ name: 'PUBLIC', value: 'visible' },
|
||||||
{ name: 'SECRET_A', value: 'resolved-value' },
|
{ name: 'SECRET_A', value: 'resolved-value' },
|
||||||
@@ -149,18 +164,26 @@ test('resolves deduplicated Secrets before allocating one Attempt log', async ()
|
|||||||
await context.dispose();
|
await context.dispose();
|
||||||
await context.dispose();
|
await context.dispose();
|
||||||
assert.deepEqual(events.slice(2).sort(), [
|
assert.deepEqual(events.slice(2).sort(), [
|
||||||
'dispose-secrets', 'release-artifact', 'take-output',
|
'dispose-secrets',
|
||||||
|
'release-artifact',
|
||||||
|
'take-output',
|
||||||
]);
|
]);
|
||||||
});
|
});
|
||||||
|
|
||||||
test('fails before Artifact allocation when Secret authority is unavailable', async () => {
|
test('fails before Artifact allocation when Secret authority is unavailable', async () => {
|
||||||
let artifacts = 0;
|
let artifacts = 0;
|
||||||
const materializer = new BoundedWorkerRemoteExecutionContextMaterializer({
|
const materializer = new BoundedWorkerRemoteExecutionContextMaterializer({
|
||||||
artifacts: { async prepare() { artifacts += 1; } },
|
artifacts: {
|
||||||
|
async prepare() {
|
||||||
|
artifacts += 1;
|
||||||
|
},
|
||||||
|
},
|
||||||
});
|
});
|
||||||
await assert.rejects(
|
await assert.rejects(
|
||||||
materializer.prepare({
|
materializer.prepare({
|
||||||
offer: offer([{ name: 'SECRET', kind: 'secret', secretRef: secret('one') }]),
|
offer: offer([
|
||||||
|
{ name: 'SECRET', kind: 'secret', secretRef: secret('one') },
|
||||||
|
]),
|
||||||
}),
|
}),
|
||||||
/secret_unavailable/,
|
/secret_unavailable/,
|
||||||
);
|
);
|
||||||
@@ -179,11 +202,18 @@ test('disposes malformed Secret and Artifact responses without exposing values',
|
|||||||
{ secretRef, value: 'first' },
|
{ secretRef, value: 'first' },
|
||||||
{ secretRef, value: 'duplicate' },
|
{ secretRef, value: 'duplicate' },
|
||||||
],
|
],
|
||||||
dispose() { disposedSecrets += 1; },
|
environmentBundles: [],
|
||||||
|
dispose() {
|
||||||
|
disposedSecrets += 1;
|
||||||
|
},
|
||||||
};
|
};
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
artifacts: { async prepare() { throw new Error('must not allocate'); } },
|
artifacts: {
|
||||||
|
async prepare() {
|
||||||
|
throw new Error('must not allocate');
|
||||||
|
},
|
||||||
|
},
|
||||||
});
|
});
|
||||||
await assert.rejects(
|
await assert.rejects(
|
||||||
malformedSecrets.prepare({
|
malformedSecrets.prepare({
|
||||||
@@ -196,16 +226,20 @@ test('disposes malformed Secret and Artifact responses without exposing values',
|
|||||||
);
|
);
|
||||||
assert.equal(disposedSecrets, 1);
|
assert.equal(disposedSecrets, 1);
|
||||||
|
|
||||||
const malformedArtifact = new BoundedWorkerRemoteExecutionContextMaterializer({
|
const malformedArtifact = new BoundedWorkerRemoteExecutionContextMaterializer(
|
||||||
artifacts: {
|
{
|
||||||
async prepare() {
|
artifacts: {
|
||||||
return {
|
async prepare() {
|
||||||
logArtifactId: 'x'.repeat(37),
|
return {
|
||||||
release() { releasedArtifact += 1; },
|
logArtifactId: 'x'.repeat(37),
|
||||||
};
|
release() {
|
||||||
|
releasedArtifact += 1;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
},
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
});
|
);
|
||||||
await assert.rejects(
|
await assert.rejects(
|
||||||
malformedArtifact.prepare({
|
malformedArtifact.prepare({
|
||||||
offer: offer([{ name: 'PUBLIC', kind: 'public', value: 'visible' }]),
|
offer: offer([{ name: 'PUBLIC', kind: 'public', value: 'visible' }]),
|
||||||
@@ -231,11 +265,18 @@ test('enforces the resolved environment byte budget before Artifact allocation',
|
|||||||
secretRef,
|
secretRef,
|
||||||
value: 'x'.repeat(16 * 1024),
|
value: 'x'.repeat(16 * 1024),
|
||||||
})),
|
})),
|
||||||
dispose() { disposed += 1; },
|
environmentBundles: [],
|
||||||
|
dispose() {
|
||||||
|
disposed += 1;
|
||||||
|
},
|
||||||
};
|
};
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
artifacts: { async prepare() { artifacts += 1; } },
|
artifacts: {
|
||||||
|
async prepare() {
|
||||||
|
artifacts += 1;
|
||||||
|
},
|
||||||
|
},
|
||||||
});
|
});
|
||||||
await assert.rejects(
|
await assert.rejects(
|
||||||
materializer.prepare({ offer: offer(bindings) }),
|
materializer.prepare({ offer: offer(bindings) }),
|
||||||
@@ -244,3 +285,69 @@ test('enforces the resolved environment byte budget before Artifact allocation',
|
|||||||
assert.equal(disposed, 1);
|
assert.equal(disposed, 1);
|
||||||
assert.equal(artifacts, 0);
|
assert.equal(artifacts, 0);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('expands one opaque bundle in memory and rejects name collisions', async () => {
|
||||||
|
const environmentBundleRef = createSecretRef({
|
||||||
|
projectId: 'project-1',
|
||||||
|
name: 'legacy-env-bundle',
|
||||||
|
version: 2,
|
||||||
|
});
|
||||||
|
const bundle = serializeEnvironmentBundle({
|
||||||
|
schema: ENVIRONMENT_BUNDLE_SCHEMA,
|
||||||
|
entries: [
|
||||||
|
{ name: 'LEGACY_TOKEN', value: 'private' },
|
||||||
|
{ name: 'LEGACY_MODE', value: 'compat' },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
const materializer = new BoundedWorkerRemoteExecutionContextMaterializer({
|
||||||
|
secrets: {
|
||||||
|
async resolve(request) {
|
||||||
|
assert.deepEqual(request.secretRefs, []);
|
||||||
|
assert.deepEqual(request.environmentBundleRefs, [environmentBundleRef]);
|
||||||
|
return {
|
||||||
|
values: [],
|
||||||
|
environmentBundles: [
|
||||||
|
{ secretRef: environmentBundleRef, value: bundle },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
},
|
||||||
|
},
|
||||||
|
artifacts: {
|
||||||
|
async prepare() {
|
||||||
|
return {
|
||||||
|
logArtifactId: 'remote-log-bundle',
|
||||||
|
takeOutput() {
|
||||||
|
return {
|
||||||
|
logArtifactId: 'remote-log-bundle',
|
||||||
|
async write() {},
|
||||||
|
async close() {},
|
||||||
|
};
|
||||||
|
},
|
||||||
|
async release() {},
|
||||||
|
};
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const context = await materializer.prepare({
|
||||||
|
offer: offer(
|
||||||
|
[{ name: 'PUBLIC', kind: 'public', value: 'visible' }],
|
||||||
|
environmentBundleRef,
|
||||||
|
),
|
||||||
|
});
|
||||||
|
assert.deepEqual(context.environment, [
|
||||||
|
{ name: 'PUBLIC', value: 'visible' },
|
||||||
|
{ name: 'LEGACY_MODE', value: 'compat' },
|
||||||
|
{ name: 'LEGACY_TOKEN', value: 'private' },
|
||||||
|
]);
|
||||||
|
await context.dispose();
|
||||||
|
|
||||||
|
await assert.rejects(
|
||||||
|
materializer.prepare({
|
||||||
|
offer: offer(
|
||||||
|
[{ name: 'LEGACY_MODE', kind: 'public', value: 'current' }],
|
||||||
|
environmentBundleRef,
|
||||||
|
),
|
||||||
|
}),
|
||||||
|
/secret_response_invalid/,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|||||||
@@ -24,31 +24,52 @@ const SECRET_REF = createSecretRef({ projectId: 'project-1', name: 'token' });
|
|||||||
|
|
||||||
function acceptedOffer() {
|
function acceptedOffer() {
|
||||||
const executionRevision = createClusterTaskExecutionRevision({
|
const executionRevision = createClusterTaskExecutionRevision({
|
||||||
projectId: 'project-1', taskId: 'task-1', taskRevision: TASK_REVISION,
|
projectId: 'project-1',
|
||||||
sourceRevision: 1, sourceContentDigest: SOURCE_DIGEST,
|
taskId: 'task-1',
|
||||||
executorType: 'remote_worker', planSchema: 'qinglong/command-execution@v1',
|
taskRevision: TASK_REVISION,
|
||||||
|
sourceRevision: 1,
|
||||||
|
sourceContentDigest: SOURCE_DIGEST,
|
||||||
|
executorType: 'remote_worker',
|
||||||
|
planSchema: 'qinglong/command-execution@v1',
|
||||||
command: { kind: 'argv', file: '/bin/true', args: [] },
|
command: { kind: 'argv', file: '/bin/true', args: [] },
|
||||||
environment: [{ name: 'TOKEN', kind: 'secret', secretRef: SECRET_REF }],
|
environment: [{ name: 'TOKEN', kind: 'secret', secretRef: SECRET_REF }],
|
||||||
createdAtMs: 1,
|
createdAtMs: 1,
|
||||||
});
|
});
|
||||||
return createClusterRemoteExecutionOffer({
|
return createClusterRemoteExecutionOffer({
|
||||||
offerId: 'offer-1', deliveryKind: 'new_claim',
|
offerId: 'offer-1',
|
||||||
|
deliveryKind: 'new_claim',
|
||||||
executionDigest: executionRevision.contentDigest,
|
executionDigest: executionRevision.contentDigest,
|
||||||
candidate: {
|
candidate: {
|
||||||
runId: 'run-1', attemptId: 'attempt-1', projectId: 'project-1',
|
runId: 'run-1',
|
||||||
taskId: 'task-1', taskRevision: TASK_REVISION, priority: 1,
|
attemptId: 'attempt-1',
|
||||||
queuedAtMs: 10, attemptCreatedAtMs: 11, attemptNumber: 1,
|
projectId: 'project-1',
|
||||||
|
taskId: 'task-1',
|
||||||
|
taskRevision: TASK_REVISION,
|
||||||
|
priority: 1,
|
||||||
|
queuedAtMs: 10,
|
||||||
|
attemptCreatedAtMs: 11,
|
||||||
|
attemptNumber: 1,
|
||||||
executorType: 'remote_worker',
|
executorType: 'remote_worker',
|
||||||
},
|
},
|
||||||
worker: { workerId: 'edge-1', sessionId: SESSION_ID, generation: 2 },
|
worker: { workerId: 'edge-1', sessionId: SESSION_ID, generation: 2 },
|
||||||
lease: {
|
lease: {
|
||||||
attemptId: 'attempt-1', runId: 'run-1', status: 'leased', version: 4,
|
attemptId: 'attempt-1',
|
||||||
leaseGeneration: 3, workerId: 'edge-1', workerSessionId: SESSION_ID,
|
runId: 'run-1',
|
||||||
workerGeneration: 2, leaseTokenDigest: digestRunDispatchLeaseToken(LEASE_TOKEN),
|
status: 'leased',
|
||||||
acquiredAtMs: 20, renewedAtMs: 20, expiresAtMs: 30_020,
|
version: 4,
|
||||||
|
leaseGeneration: 3,
|
||||||
|
workerId: 'edge-1',
|
||||||
|
workerSessionId: SESSION_ID,
|
||||||
|
workerGeneration: 2,
|
||||||
|
leaseTokenDigest: digestRunDispatchLeaseToken(LEASE_TOKEN),
|
||||||
|
acquiredAtMs: 20,
|
||||||
|
renewedAtMs: 20,
|
||||||
|
expiresAtMs: 30_020,
|
||||||
updatedAtMs: 20,
|
updatedAtMs: 20,
|
||||||
},
|
},
|
||||||
leaseToken: LEASE_TOKEN, executionRevision, placementScore: 0,
|
leaseToken: LEASE_TOKEN,
|
||||||
|
executionRevision,
|
||||||
|
placementScore: 0,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -62,6 +83,7 @@ function requestFor(offer) {
|
|||||||
offerId: offer.offerId,
|
offerId: offer.offerId,
|
||||||
executionDigest: offer.executionDigest,
|
executionDigest: offer.executionDigest,
|
||||||
secretRefs: [SECRET_REF],
|
secretRefs: [SECRET_REF],
|
||||||
|
environmentBundleRefs: [],
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -78,12 +100,17 @@ test('rehydrates lease authority from inbox and delivers one exact Secret batch'
|
|||||||
client: {
|
client: {
|
||||||
async postJson(request) {
|
async postJson(request) {
|
||||||
transport = request;
|
transport = request;
|
||||||
return Buffer.from(JSON.stringify({
|
return Buffer.from(
|
||||||
schema: 'qinglong/remote-secret-delivery@v1',
|
JSON.stringify({
|
||||||
runId: 'run-1', attemptId: 'attempt-1', offerId: 'offer-1',
|
schema: 'qinglong/remote-secret-delivery@v2',
|
||||||
executionDigest: offer.executionDigest,
|
runId: 'run-1',
|
||||||
values: [{ secretRef: SECRET_REF, value: 'resolved-value' }],
|
attemptId: 'attempt-1',
|
||||||
}));
|
offerId: 'offer-1',
|
||||||
|
executionDigest: offer.executionDigest,
|
||||||
|
values: [{ secretRef: SECRET_REF, value: 'resolved-value' }],
|
||||||
|
environmentBundles: [],
|
||||||
|
}),
|
||||||
|
);
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
@@ -91,7 +118,11 @@ test('rehydrates lease authority from inbox and delivers one exact Secret batch'
|
|||||||
assert.deepEqual(resolution.values, [
|
assert.deepEqual(resolution.values, [
|
||||||
{ secretRef: SECRET_REF, value: 'resolved-value' },
|
{ secretRef: SECRET_REF, value: 'resolved-value' },
|
||||||
]);
|
]);
|
||||||
assert.equal(transport.path.endsWith(`/sessions/${SESSION_ID}/secrets`), true);
|
assert.deepEqual(resolution.environmentBundles, []);
|
||||||
|
assert.equal(
|
||||||
|
transport.path.endsWith(`/sessions/${SESSION_ID}/secrets`),
|
||||||
|
true,
|
||||||
|
);
|
||||||
assert.equal(transport.body.leaseToken, LEASE_TOKEN);
|
assert.equal(transport.body.leaseToken, LEASE_TOKEN);
|
||||||
assert.equal(transport.maximumRequestBytes, 64 * 1024);
|
assert.equal(transport.maximumRequestBytes, 64 * 1024);
|
||||||
assert.equal(JSON.stringify(requestFor(offer)).includes(LEASE_TOKEN), false);
|
assert.equal(JSON.stringify(requestFor(offer)).includes(LEASE_TOKEN), false);
|
||||||
@@ -102,9 +133,15 @@ test('rejects a stale inbox identity before sending the capability', async () =>
|
|||||||
let calls = 0;
|
let calls = 0;
|
||||||
const provider = new WorkerRemoteSecretHttpsProvider({
|
const provider = new WorkerRemoteSecretHttpsProvider({
|
||||||
inbox: {
|
inbox: {
|
||||||
async readOffer() { return { state: 'starting_acknowledged', offer }; },
|
async readOffer() {
|
||||||
|
return { state: 'starting_acknowledged', offer };
|
||||||
|
},
|
||||||
|
},
|
||||||
|
client: {
|
||||||
|
async postJson() {
|
||||||
|
calls += 1;
|
||||||
|
},
|
||||||
},
|
},
|
||||||
client: { async postJson() { calls += 1; } },
|
|
||||||
});
|
});
|
||||||
await assert.rejects(
|
await assert.rejects(
|
||||||
provider.resolve({ ...requestFor(offer), executionDigest: 'b'.repeat(64) }),
|
provider.resolve({ ...requestFor(offer), executionDigest: 'b'.repeat(64) }),
|
||||||
@@ -117,16 +154,23 @@ test('rejects response authority drift and does not return plaintext', async ()
|
|||||||
const offer = acceptedOffer();
|
const offer = acceptedOffer();
|
||||||
const provider = new WorkerRemoteSecretHttpsProvider({
|
const provider = new WorkerRemoteSecretHttpsProvider({
|
||||||
inbox: {
|
inbox: {
|
||||||
async readOffer() { return { state: 'starting_acknowledged', offer }; },
|
async readOffer() {
|
||||||
|
return { state: 'starting_acknowledged', offer };
|
||||||
|
},
|
||||||
},
|
},
|
||||||
client: {
|
client: {
|
||||||
async postJson() {
|
async postJson() {
|
||||||
return Buffer.from(JSON.stringify({
|
return Buffer.from(
|
||||||
schema: 'qinglong/remote-secret-delivery@v1',
|
JSON.stringify({
|
||||||
runId: 'run-other', attemptId: 'attempt-1', offerId: 'offer-1',
|
schema: 'qinglong/remote-secret-delivery@v2',
|
||||||
executionDigest: offer.executionDigest,
|
runId: 'run-other',
|
||||||
values: [{ secretRef: SECRET_REF, value: 'must-not-escape' }],
|
attemptId: 'attempt-1',
|
||||||
}));
|
offerId: 'offer-1',
|
||||||
|
executionDigest: offer.executionDigest,
|
||||||
|
values: [{ secretRef: SECRET_REF, value: 'must-not-escape' }],
|
||||||
|
environmentBundles: [],
|
||||||
|
}),
|
||||||
|
);
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
@@ -138,10 +182,21 @@ test('does not fetch Secrets before starting ACK or after the launch barrier', a
|
|||||||
for (const state of ['accepted', 'launching']) {
|
for (const state of ['accepted', 'launching']) {
|
||||||
let calls = 0;
|
let calls = 0;
|
||||||
const provider = new WorkerRemoteSecretHttpsProvider({
|
const provider = new WorkerRemoteSecretHttpsProvider({
|
||||||
inbox: { async readOffer() { return { state, offer }; } },
|
inbox: {
|
||||||
client: { async postJson() { calls += 1; } },
|
async readOffer() {
|
||||||
|
return { state, offer };
|
||||||
|
},
|
||||||
|
},
|
||||||
|
client: {
|
||||||
|
async postJson() {
|
||||||
|
calls += 1;
|
||||||
|
},
|
||||||
|
},
|
||||||
});
|
});
|
||||||
await assert.rejects(provider.resolve(requestFor(offer)), /offer_unavailable/);
|
await assert.rejects(
|
||||||
|
provider.resolve(requestFor(offer)),
|
||||||
|
/offer_unavailable/,
|
||||||
|
);
|
||||||
assert.equal(calls, 0);
|
assert.equal(calls, 0);
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -267,11 +267,20 @@ test('disposes provider-owned credential material on success and rejection', asy
|
|||||||
rejected.postJson({
|
rejected.postJson({
|
||||||
path: COMPLETION_PATH,
|
path: COMPLETION_PATH,
|
||||||
body: {},
|
body: {},
|
||||||
maximumResponseBytes: 1024,
|
maximumResponseBytes: 256 * 1024,
|
||||||
}),
|
}),
|
||||||
/credentials_unavailable/,
|
/credentials_unavailable/,
|
||||||
);
|
);
|
||||||
assert.equal(rejectedDisposals, 1);
|
assert.equal(rejectedDisposals, 1);
|
||||||
|
await assert.rejects(
|
||||||
|
rejected.postJson({
|
||||||
|
path: COMPLETION_PATH,
|
||||||
|
body: {},
|
||||||
|
maximumResponseBytes: 256 * 1024 + 1,
|
||||||
|
}),
|
||||||
|
/request_rejected/,
|
||||||
|
);
|
||||||
|
assert.equal(rejectedDisposals, 1);
|
||||||
} finally {
|
} finally {
|
||||||
rejected.close();
|
rejected.close();
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -6,9 +6,7 @@ const assert = require('node:assert/strict');
|
|||||||
const { createHash } = require('node:crypto');
|
const { createHash } = require('node:crypto');
|
||||||
const fs = require('node:fs');
|
const fs = require('node:fs');
|
||||||
|
|
||||||
const {
|
const { createSecretRef } = require('@qinglong/runtime-core/secret-reference');
|
||||||
createSecretRef,
|
|
||||||
} = require('@qinglong/runtime-core/secret-reference');
|
|
||||||
const {
|
const {
|
||||||
ClusterMountedSecretProviderError,
|
ClusterMountedSecretProviderError,
|
||||||
createClusterMountedSecretProvider,
|
createClusterMountedSecretProvider,
|
||||||
@@ -37,6 +35,7 @@ function authority() {
|
|||||||
leaseGeneration: 1,
|
leaseGeneration: 1,
|
||||||
leaseVersion: 1,
|
leaseVersion: 1,
|
||||||
secretRefs: Object.freeze([SECRET_REF]),
|
secretRefs: Object.freeze([SECRET_REF]),
|
||||||
|
environmentBundleRefs: Object.freeze([]),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -299,10 +299,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
|
|||||||
rootSourceFileRoles: runtimeCore.rootSourceFileRoles,
|
rootSourceFileRoles: runtimeCore.rootSourceFileRoles,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
sourceFiles: 172,
|
sourceFiles: 173,
|
||||||
rootSourceFiles: 1,
|
rootSourceFiles: 1,
|
||||||
rootSourceLines: 160,
|
rootSourceLines: 160,
|
||||||
nestedSourceFiles: 171,
|
nestedSourceFiles: 172,
|
||||||
rootSourceFileRoles: { 'index.ts': 'public_export' },
|
rootSourceFileRoles: { 'index.ts': 'public_export' },
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|||||||
Reference in New Issue
Block a user