diff --git a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md index c1f6b0a1..48fe269d 100644 --- a/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md +++ b/docs/QINGLONG_3_0_ARCHITECTURE_RFC.md @@ -11,6 +11,26 @@ 最新增量证据(2026-08-20): +- D-373/ADR-0466(已接受):完成 D-372 的首个 caller-driven Worker 管理只读面。在既有 Worker management + service/transport/client 内增加 `worker-session.inspect|list`,继续复用同一个 TLS 1.3/mTLS/OIDC listener、强 User + `worker.manage` 与耐久 `worker-session.observe` quota,不新增 package、服务、端口、连接池、timer、watcher、queue、 + cache 或 Deployment。`pg-0069-worker-session-management-observation` 把 `control-core` 升至 v68,只给 + `ql3_worker_credential_manager` 增加 `SELECT worker_sessions`;point 只读一个 canonical Worker,list 固定查询 17 行并 + 最多返回 16 行 `workerId` keyset。响应只投影 Session/generation/version、`online|draining|offline|lease_expired`、 + `default_placement|explicit_placement_required|protocol_incompatible`、architecture/Tier/protocol、OS、并发与有界 + runtime/capacity,不返回 raw capability、label、GPU model、credential、Secret 或 lease capability。alpha 继续复用历史 + `/api/v3/worker-credentials/management` 是明确的命名债务,Beta 前须决定通用 Worker management alias/CLI 迁移,不能 + 复制第二个 listener。PostgreSQL 聚焦 migration/readiness `76/76`,Cluster PostgreSQL 全量 + `351 total / 348 pass / 3 conditional skip / 0 fail`,Cluster Admin 全量 + `421 total / 418 pass / 3 conditional skip / 0 fail`;package boundary、Cluster dependency、Edge import、Cluster + deployment 与 Worker deployment 全部 compatible,18 packages 仍无 single/shallow package且没有新增外部 dependency。 + 完整 backend 为 `1,503 total / 1,501 pass / 2 conditional skip / 0 fail`(包含一条不进入本阶段提交的既有用户测试)。 + 14 档 Local artifact audit 全部 compatible;基础 Edge/Standalone 为 `2,598,669 / 2,598,747` bytes、57 loaded modules, + Application+AI 为 `4,501,822 / 4,501,954` bytes,MCP 为 `7,324,601 / 7,324,709` bytes。 + PostgreSQL 18.6 arm64 HA `146/146`、timeline `1→2`,报告 SHA-256 为 + `c4cf0189b68d7af18169cd8f8de726e26a970af703f28d4ffe0ad3ace96fa596`,真实 Worker manager role readiness、promotion、 + fencing、`pg_rewind` 与只读 rejoin 全绿。Edge/Standalone 和禁用 Worker manager 的 Cluster 路径保持零新增常驻成本。 + - D-372/ADR-0465(已接受):把 D-371 的发布架构分层推进到 Remote Worker 的实际注册与调度边界。Worker canonical capability 现在必须声明 `protocolVersion` 与 `supportTier`,架构/Tier 必须与根 release identity v2 一致,Node `x64/ppc64/ia32/arm` 会归一化到发布词汇并与实际 Worker 进程复验。普通 Task 即使未声明 Placement 也只匹配 Tier 1 @@ -6457,6 +6477,12 @@ ARMv6/386 的 2.x 进程已经能直接连接 3.0,也不授予 Plugin Host、 Maintainers 要求 ARMv6、ARMv7 或 386 成为 3.0 正式支持项,必须先修改 D-16 或长期维护对应 Node 24 构建链, 升级 release identity schema,并通过与现有 Tier 1 同等级的发布门禁。 +D-373/ADR-0466 已把该分层投影到 caller-driven Worker 管理只读面:operator 可以 point inspect 或显式读取固定 16 项 +keyset page,区分 Session lifecycle 与 Placement compatibility,并查看有界 runtime/capacity;不能请求任意 filter、 +limit、自动翻页或轮询。该能力复用现有 Worker manager 进程和 `worker.manage`/耐久 quota,PostgreSQL role 只新增 +`SELECT worker_sessions`,所以不会进入 Edge/Standalone 常驻闭包。alpha 的 credential-named URI/CLI 是待 Beta 迁移的 +命名债务,不得用新建第二套 listener、Deployment 或 package 的方式修复。 + ### 7.10 资源基准与发布门禁 每个 3.x 版本至少记录以下基准: diff --git a/docs/adr/ADR-0466-bounded-worker-session-compatibility-observation.md b/docs/adr/ADR-0466-bounded-worker-session-compatibility-observation.md new file mode 100644 index 00000000..fef60170 --- /dev/null +++ b/docs/adr/ADR-0466-bounded-worker-session-compatibility-observation.md @@ -0,0 +1,117 @@ +# ADR-0466:有界 Worker Session 兼容性观察 + +- 状态:Accepted +- 日期:2026-08-20 +- 关联 RFC:QL-RFC-0001 D-373、D-14、D-16、D-107 +- 关联 ADR:ADR-0012、ADR-0146、ADR-0464、ADR-0465 +- Amends:ADR-0465 的 Worker 管理只读面后续边界 + +## 上下文 + +ADR-0465 已让 Worker Session 持久化 canonical architecture、support Tier 与 protocol version,并让 Scheduler +按 Tier 1 默认和显式 Placement 执行准入。但 operator 仍只能从数据库或日志拼接状态,无法通过受认证管理面回答: + +- 当前 Session 是在线、draining、offline,还是 lease 已过期; +- Worker 是否可进入默认 Placement、必须显式选择,还是协议不兼容; +- 运行时、并发和声明容量是否足以解释“注册成功但没有领取任务”; +- 一个指定 Worker 的 generation/session version,或按稳定顺序查看下一小页 Worker。 + +另建 package、服务或轮询器会扩大低配部署和 Cluster 运维成本;直接返回 `capabilities_json`、labels、GPU model、 +credential 或 lease token 又会把不必要的调度与 Secret 事实暴露到管理客户端。现有 Worker credential manager 已拥有 +独立 TLS 1.3/mTLS/OIDC、强 User、`worker.manage`、耐久 quota、单独 PostgreSQL role 与固定 HTTP 路径,可以承载 +同一 Worker 管理域的按需只读观察,但它的历史命名仍偏向 credential。 + +## 决策 + +1. 在既有三个职责边界内实现,不新增 workspace package: + - `@qinglong/runtime-core/worker-session-observation` 拥有低敏投影和状态推导; + - `@qinglong/cluster-postgres` 的 `remote-execution` 领域拥有只读 repository; + - `@qinglong/cluster-admin` 的既有 Worker management service/transport/client 暴露管理操作。 +2. 增加两个精确 operation:`worker-session.inspect` 与 `worker-session.list`。alpha 阶段继续复用固定 + `/api/v3/worker-credentials/management` listener、进程、连接池和客户端协议,不创建第二个 Worker 管理进程。 + 该 URI 与 binary 的 credential 命名是明确的产品债务;Beta 前必须决定兼容 alias/通用 Worker management 路径, + 不能长期让通用 Worker 观察依附于误导性名称。 +3. 两个操作都要求 fresh 强认证 User、authority Project 上的 `worker.manage`,并使用单一耐久 quota operation + `worker-session.observe`。默认窗口上限为 600;inspect/list 以各自 `inspectionId` 派生幂等 receipt,caller 不能 + 选择 quota operation、窗口或 limit。 +4. PostgreSQL `pg-0069-worker-session-management-observation` 把 `control-core` 升至 capability v68,并只给 + `ql3_worker_credential_manager` 增加 `SELECT` on `ql3.worker_sessions`。迁移先 `REVOKE ALL` 再精确 `GRANT SELECT`; + manager 不取得 INSERT/UPDATE/DELETE/TRUNCATE,executor、runtime 及其它管理角色不因本能力扩权。 +5. point inspect 只按 canonical `workerId` 查询,最多接受一条 Session;list 固定读取 17 行、最多返回 16 行,按 + `workerId` 升序 keyset 分页。caller 只能提供 nullable `afterWorkerId`,不能提供 limit、排序、过滤表达式、自动翻页 + 或任意 SQL 字段。 +6. 每次 repository 调用用 PostgreSQL `statement_timestamp()` 取得一个数据库观察时间。生命周期与兼容性是投影, + 不是新的持久状态机: + - lifecycle 为 `online|draining|offline|lease_expired`; + - compatibility 为 `default_placement|explicit_placement_required|protocol_incompatible`; + - protocol v1 且 Tier 1 才是 default placement;其它受支持 Tier 必须显式 Placement;不满足全局协议范围则不兼容。 +7. list 只返回 Worker/Session/generation/version、lifecycle/compatibility、architecture/Tier/protocol、OS、并发与时间; + point inspect 额外返回最多 32 个 runtime 及 `{cpuCores,memoryBytes,diskBytes,gpuCount}`。任何响应都不得返回 raw + capability JSON/hash、labels、features、GPU model、credential、Secret、authentication identity 或 lease capability。 +8. PostgreSQL row、runtime-core projection 和远程 client 分别执行严格校验。客户端拒绝未知字段、负数/越界容量、 + 时间关系漂移、无序 page、cursor 漂移、observedAt 不一致和 secret-bearing widening,不能把受攻击的管理响应作为 + 可信事实呈现。 +9. 本能力是 caller-driven one-shot read:不新增 listener、Deployment、Service、连接池、timer、watcher、queue、cache + 或后台扫描。Edge/Standalone 与未启用 Worker manager 的 Cluster 路径不加载这些入口,也不创建 PostgreSQL authority。 + +## 升级与回滚 + +- 先用 migration role 执行 `pg-0069`,再滚动升级 Worker manager;旧 manager 不使用新增 SELECT,混合窗口安全。 +- 回滚应用时可以保留 v68 schema 与只读 GRANT;旧应用 readiness 不接受 ahead-of-code schema,因此完整回滚必须使用 + 对应数据库备份/受审降级流程,不能手工改写 migration history 或 capability JSON。 +- 两个 operation 不修改 Session,也不影响 Worker heartbeat、Scheduler placement 或 execution revision digest;关闭 + Worker manager 即可撤销网络可达性。 + +## 被拒绝的替代方案 + +### 新建 `worker-observability` package 和常驻服务 + +拒绝。这里只有一个投影契约、一个 repository 与两个管理操作,没有独立部署、扩缩容或权限生命周期;拆包会重新 +制造用户已指出的单文件/浅层 package,并让路由设备与供应链承担无收益边界。 + +### 直接返回 capability JSON 或 labels + +拒绝。它会扩大响应、把未来字段默认公开,并泄露调度标签、硬件型号或可用于定向攻击的细节。管理面只返回明确评审的 +固定投影。 + +### list 接受 caller limit、任意 filter 和自动轮询 + +拒绝。caller-controlled query 会使低配管理节点承担不可预测扫描,自动轮询会引入空闲网络、数据库和 CPU 成本。固定 +16 项 keyset 页面需要 operator 显式发起下一次读取。 + +### 为观察结果新建数据库表 + +拒绝。lifecycle 与 compatibility 都能由 immutable capability、Session 状态、lease 和数据库时钟确定;复制一份状态会 +产生跨副本漂移和额外写放大。 + +## 验证与证据 + +- runtime-core 聚焦测试覆盖四种 lifecycle、三种 compatibility、Tier/协议、详细/摘要投影和非法数据库时间。 +- Cluster PostgreSQL repository 测试覆盖 point、masked absence、固定 16+1 keyset、同一数据库时钟、身份/排序/行上限 + 与 canonical capability fail-closed。 +- Cluster Admin service/transport/client 测试覆盖强 User + `worker.manage`、耐久 quota、六种现有 Worker management + operation、严格响应交叉不变量与 secret-bearing widening 拒绝。 +- PostgreSQL migration/readiness 聚焦门 `76/76`;Cluster PostgreSQL 全量 `351 total / 348 pass / 3 conditional skip / + 0 fail`;Cluster Admin 全量 `421 total / 418 pass / 3 conditional skip / 0 fail`。 +- 完整 backend 工作区为 `1,503 total / 1,501 pass / 2 conditional skip / 0 fail`;总数包含工作树中既有且不进入 + 本阶段提交的用户测试,本提交没有修改或暂存该文件。 +- PostgreSQL 18.6 arm64 physical HA 门 `146/146`、timeline `1→2`,报告 SHA-256 为 + `c4cf0189b68d7af18169cd8f8de726e26a970af703f28d4ffe0ad3ace96fa596`。主库 readiness 使用真实 + `ql3_worker_credential_manager` 连接复验 v68 migration 与精确 Worker manager ACL;promotion、旧主 fencing、 + `pg_rewind` 和只读 rejoin 全绿。 +- package boundary、Cluster dependency、Edge import、Cluster deployment 与 Worker deployment 审计全部 compatible; + workspace 保持 18 packages,`singleSourcePackages=[]`、`shallowSourcePackages=[]`。runtime-core 为 `171 source / 170 + nested`,Cluster PostgreSQL 为 `175 source / 174 nested`,没有新增外部 dependency。 +- 14 档 Local artifact audit 全部 compatible;基础 Edge/Standalone 为 `2,598,669 / 2,598,747` bytes 且都只加载 + 57 modules,Application+AI 为 `4,501,822 / 4,501,954` bytes,MCP 为 `7,324,601 / 7,324,709` bytes。新增子路径被 + artifact projection 裁剪,没有进入低配 Profile 启动闭包。 + +## 后续边界 + +- Beta 前提供不误导的通用 Worker management 路径/CLI 命名及兼容迁移,不复制 listener 或 authority。 +- Console/UI 若接入,必须保持显式点击、固定 16 项、无轮询/自动翻页,并隔离短期 assertion;不能把管理 capability + 放入普通 Project API credential。 +- 如需按 Project、label、资源池或健康历史查询,必须先定义 ownership、索引、保留期、响应预算与隐私边界;不得把 + 本次 cluster-wide 当前 Session 读取悄悄扩成无界 inventory/metrics 系统。 +- 当前沿用既有 Worker credential management 的 quota/认证模型;若统一管理面引入持久化 read-access audit,应对 + credential inspect 与 Session observe 一并设计事务和 retention,不能只给其中一个操作制造不一致的审计语义。 diff --git a/docs/adr/README.md b/docs/adr/README.md index bc928c95..1d5bd89e 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -469,6 +469,7 @@ | [ADR-0463](./ADR-0463-native-console-capacity-and-assertion-lifecycle-evidence.md) | 原生双架构 Console 容量与 Assertion 生命周期证据 | Accepted(首份原生 CI 双架构报告待实际运行) | | [ADR-0464](./ADR-0464-machine-enforced-node24-architecture-support-tiers.md) | 机器化 Node 24 架构支持分层 | Accepted | | [ADR-0465](./ADR-0465-versioned-worker-support-tier-admission.md) | 版本化 Worker 支持等级准入 | Accepted | +| [ADR-0466](./ADR-0466-bounded-worker-session-compatibility-observation.md) | 有界 Worker Session 兼容性观察 | Accepted | ## 规则 diff --git a/packages/ql3-cluster-admin/src/worker-credential/management-server/workerCredentialManagement.ts b/packages/ql3-cluster-admin/src/worker-credential/management-server/workerCredentialManagement.ts index f30c213a..f8522d6b 100644 --- a/packages/ql3-cluster-admin/src/worker-credential/management-server/workerCredentialManagement.ts +++ b/packages/ql3-cluster-admin/src/worker-credential/management-server/workerCredentialManagement.ts @@ -3,6 +3,7 @@ import { PostgresApprovalRequestRepository, PostgresProjectPolicyRepository, PostgresWorkerCredentialManagementPlanRepository, + PostgresWorkerSessionObservationRepository, } from '@qinglong/cluster-postgres/worker-credential-manager'; import type { PostgresPool } from '@qinglong/runtime-core'; import { @@ -20,6 +21,10 @@ import { type SecuritySubject, } from '@qinglong/runtime-core/security'; import type { SecurityAuditRecord } from '@qinglong/runtime-core/security-audit'; +import type { + WorkerSessionInspection, + WorkerSessionObservationPage, +} from '@qinglong/runtime-core/worker-session-observation'; import { InvalidWorkerCredentialManagementPlanError, MAX_WORKER_CREDENTIAL_MANAGEMENT_PLAN_LIFETIME_MS, @@ -93,7 +98,8 @@ export type WorkerCredentialManagementQuotaOperation = | 'worker-credential.plan' | 'worker-credential.propose' | 'worker-credential.decide' - | 'worker-credential.inspect'; + | 'worker-credential.inspect' + | 'worker-session.observe'; export interface WorkerCredentialManagementQuotaPort { consume( @@ -166,6 +172,20 @@ export interface InspectClusterWorkerCredentialResult { readonly stale: boolean; } +export interface InspectClusterWorkerSessionRequest { + readonly authorityProjectId: string; + readonly workerId: string; + readonly inspectionId: string; + readonly principal: SecurityPrincipal; +} + +export interface ListClusterWorkerSessionsRequest { + readonly authorityProjectId: string; + readonly afterWorkerId: string | null; + readonly inspectionId: string; + readonly principal: SecurityPrincipal; +} + export interface ClusterWorkerCredentialManagementService { plan( request: PlanClusterWorkerCredentialRequest, @@ -179,6 +199,12 @@ export interface ClusterWorkerCredentialManagementService { inspectAuthorized( request: InspectClusterWorkerCredentialRequest, ): Promise>; + inspectSession( + request: InspectClusterWorkerSessionRequest, + ): Promise>; + listSessions( + request: ListClusterWorkerSessionsRequest, + ): Promise>; } export interface ClusterWorkerCredentialManagementOptions { @@ -324,6 +350,7 @@ export function createClusterWorkerCredentialManagementService( const plans = new PostgresWorkerCredentialManagementPlanRepository( options.pool, ); + const sessions = new PostgresWorkerSessionObservationRepository(options.pool); const approvals = new PostgresApprovalRequestRepository(options.pool); const policy = new ProjectPolicyEngine( new PostgresProjectPolicyRepository(options.pool), @@ -796,5 +823,74 @@ export function createClusterWorkerCredentialManagementService( observedAtMs > plan.expiresAtMs, }); }, + + async inspectSession(request: InspectClusterWorkerSessionRequest) { + exact( + request, + ['authorityProjectId', 'inspectionId', 'principal', 'workerId'], + 'Session inspection request', + ); + const projectId = identifier( + request.authorityProjectId, + 'authorityProjectId', + ); + const inspectionId = identifier(request.inspectionId, 'inspectionId'); + const workerId = identifier(request.workerId, 'workerId'); + const authorization = await authorize( + request.principal, + projectId, + 'worker.manage', + currentTime(now), + ); + await consumeQuota( + projectId, + authorization.principal, + 'worker-session.observe', + `session-inspect:${inspectionId}`, + ); + try { + return await sessions.inspect(workerId); + } catch (error) { + throw new WorkerCredentialManagementUnavailableError({ + cause: error instanceof Error ? error : undefined, + }); + } + }, + + async listSessions(request: ListClusterWorkerSessionsRequest) { + exact( + request, + ['afterWorkerId', 'authorityProjectId', 'inspectionId', 'principal'], + 'Session list request', + ); + const projectId = identifier( + request.authorityProjectId, + 'authorityProjectId', + ); + const inspectionId = identifier(request.inspectionId, 'inspectionId'); + const afterWorkerId = + request.afterWorkerId === null + ? null + : identifier(request.afterWorkerId, 'afterWorkerId'); + const authorization = await authorize( + request.principal, + projectId, + 'worker.manage', + currentTime(now), + ); + await consumeQuota( + projectId, + authorization.principal, + 'worker-session.observe', + `session-list:${inspectionId}`, + ); + try { + return await sessions.list(afterWorkerId); + } catch (error) { + throw new WorkerCredentialManagementUnavailableError({ + cause: error instanceof Error ? error : undefined, + }); + } + }, }); } diff --git a/packages/ql3-cluster-admin/src/worker-credential/management-server/workerCredentialManagementTransport.ts b/packages/ql3-cluster-admin/src/worker-credential/management-server/workerCredentialManagementTransport.ts index d5dfd045..a342d37a 100644 --- a/packages/ql3-cluster-admin/src/worker-credential/management-server/workerCredentialManagementTransport.ts +++ b/packages/ql3-cluster-admin/src/worker-credential/management-server/workerCredentialManagementTransport.ts @@ -74,11 +74,33 @@ export interface InspectClusterWorkerCredentialCommand { }; } +export interface InspectClusterWorkerSessionCommand { + readonly schemaVersion: 1; + readonly operation: 'worker-session.inspect'; + readonly request: { + readonly authorityProjectId: string; + readonly workerId: string; + readonly inspectionId: string; + }; +} + +export interface ListClusterWorkerSessionsCommand { + readonly schemaVersion: 1; + readonly operation: 'worker-session.list'; + readonly request: { + readonly authorityProjectId: string; + readonly afterWorkerId: string | null; + readonly inspectionId: string; + }; +} + export type ClusterWorkerCredentialManagementCommand = | PlanClusterWorkerCredentialCommand | ProposeClusterWorkerCredentialCommand | DecideClusterWorkerCredentialCommand - | InspectClusterWorkerCredentialCommand; + | InspectClusterWorkerCredentialCommand + | InspectClusterWorkerSessionCommand + | ListClusterWorkerSessionsCommand; type PlanSummary = ReturnType; type ApprovalSummary = ReturnType; @@ -109,6 +131,23 @@ export type ClusterWorkerCredentialManagementTransportResult = plan: PlanSummary | null; approval: ApprovalSummary | null; stale: boolean; + }> + | Readonly<{ + schemaVersion: 1; + operation: 'worker-session.inspect'; + observedAtMs: number; + worker: Awaited< + ReturnType + >['worker']; + }> + | Readonly<{ + schemaVersion: 1; + operation: 'worker-session.list'; + observedAtMs: number; + workers: Awaited< + ReturnType + >['workers']; + nextCursor: string | null; }>; export interface ClusterWorkerCredentialManagementTransport { @@ -254,6 +293,20 @@ export function normalizeClusterWorkerCredentialManagementCommand( 'inspection request', ); break; + case 'worker-session.inspect': + exactObject( + value.request, + ['authorityProjectId', 'inspectionId', 'workerId'], + 'Session inspection request', + ); + break; + case 'worker-session.list': + exactObject( + value.request, + ['afterWorkerId', 'authorityProjectId', 'inspectionId'], + 'Session list request', + ); + break; default: throw new ClusterWorkerCredentialManagementTransportRequestError( 'operation is not publicly available', @@ -317,6 +370,8 @@ export function createClusterWorkerCredentialManagementTransport( typeof options.service.propose !== 'function' || typeof options.service.decide !== 'function' || typeof options.service.inspectAuthorized !== 'function' || + typeof options.service.inspectSession !== 'function' || + typeof options.service.listSessions !== 'function' || (options.now !== undefined && typeof options.now !== 'function') ) { throw new ClusterWorkerCredentialManagementTransportConfigurationError( @@ -420,6 +475,31 @@ export function createClusterWorkerCredentialManagementTransport( stale: result.stale, }); } + case 'worker-session.inspect': { + const result = await options.service.inspectSession({ + ...command.request, + principal, + }); + return Object.freeze({ + schemaVersion: 1 as const, + operation: command.operation, + observedAtMs: result.observedAtMs, + worker: result.worker, + }); + } + case 'worker-session.list': { + const result = await options.service.listSessions({ + ...command.request, + principal, + }); + return Object.freeze({ + schemaVersion: 1 as const, + operation: command.operation, + observedAtMs: result.observedAtMs, + workers: result.workers, + nextCursor: result.nextCursor, + }); + } } }, }); diff --git a/packages/ql3-cluster-admin/src/worker-credential/workerCredentialManagementClient.ts b/packages/ql3-cluster-admin/src/worker-credential/workerCredentialManagementClient.ts index aff20605..e8820a17 100644 --- a/packages/ql3-cluster-admin/src/worker-credential/workerCredentialManagementClient.ts +++ b/packages/ql3-cluster-admin/src/worker-credential/workerCredentialManagementClient.ts @@ -57,6 +57,23 @@ function boundedScalar(value: unknown): void { } } +function boundedText(value: unknown, nullable = false): void { + if (value === null && nullable) return; + if ( + typeof value !== 'string' || + value.length === 0 || + value.length > 2_048 || + CONTROL_PATTERN.test(value) + ) { + invalid(); + } +} + +function nonNegativeInteger(value: unknown, nullable = false): void { + if (value === null && nullable) return; + if (!Number.isSafeInteger(value) || (value as number) < 0) invalid(); +} + function subject(value: unknown): void { const record = exactRecord(value, ['type', 'id']); if (record.type !== 'user') invalid(); @@ -131,6 +148,106 @@ function approval(value: unknown): void { } } +const WORKER_OBSERVATION_KEYS = [ + 'architecture', + 'availableSlots', + 'compatibility', + 'generation', + 'lastHeartbeatAtMs', + 'leaseExpiresAtMs', + 'lifecycle', + 'maxConcurrentRuns', + 'observedAtMs', + 'operatingSystem', + 'protocolVersion', + 'registeredAtMs', + 'sessionId', + 'sessionVersion', + 'supportTier', + 'updatedAtMs', + 'workerId', +] as const; + +function workerObservation(value: unknown, detailed: boolean): void { + const record = exactRecord( + value, + detailed + ? [...WORKER_OBSERVATION_KEYS, 'declaredCapacity', 'runtimes'] + : WORKER_OBSERVATION_KEYS, + ); + boundedText(record.workerId); + boundedText(record.sessionId); + boundedText(record.protocolVersion); + boundedText(record.operatingSystem, true); + for (const key of [ + 'generation', + 'sessionVersion', + 'maxConcurrentRuns', + 'availableSlots', + 'registeredAtMs', + 'lastHeartbeatAtMs', + 'leaseExpiresAtMs', + 'updatedAtMs', + 'observedAtMs', + ] as const) { + nonNegativeInteger(record[key]); + } + if ( + !['online', 'draining', 'offline', 'lease_expired'].includes( + String(record.lifecycle), + ) || + ![ + 'default_placement', + 'explicit_placement_required', + 'protocol_incompatible', + ].includes(String(record.compatibility)) || + !['tier1', 'candidate', 'experimental', 'legacy-only'].includes( + String(record.supportTier), + ) || + !['amd64', 'arm64', 'ppc64le', 's390x', 'arm/v7', 'arm/v6', '386'].includes( + String(record.architecture), + ) + ) { + invalid(); + } + if ( + (record.generation as number) < 1 || + (record.maxConcurrentRuns as number) < 1 || + (record.availableSlots as number) > (record.maxConcurrentRuns as number) || + (record.lastHeartbeatAtMs as number) < (record.registeredAtMs as number) || + (record.updatedAtMs as number) < (record.lastHeartbeatAtMs as number) || + (record.observedAtMs as number) < (record.updatedAtMs as number) || + (record.lifecycle !== 'offline' && + (record.leaseExpiresAtMs as number) <= + (record.lastHeartbeatAtMs as number)) || + (['online', 'draining'].includes(String(record.lifecycle)) && + (record.leaseExpiresAtMs as number) <= (record.observedAtMs as number)) || + (record.lifecycle === 'lease_expired' && + (record.leaseExpiresAtMs as number) > (record.observedAtMs as number)) || + (['draining', 'offline'].includes(String(record.lifecycle)) && + (record.availableSlots as number) !== 0) + ) { + invalid(); + } + if (!detailed) return; + if (!Array.isArray(record.runtimes) || record.runtimes.length > 32) invalid(); + for (const runtime of record.runtimes) { + const item = exactRecord(runtime, ['name', 'version']); + boundedText(item.name); + boundedText(item.version); + } + const capacity = exactRecord(record.declaredCapacity, [ + 'cpuCores', + 'diskBytes', + 'gpuCount', + 'memoryBytes', + ]); + nonNegativeInteger(capacity.cpuCores, true); + nonNegativeInteger(capacity.memoryBytes, true); + nonNegativeInteger(capacity.diskBytes, true); + nonNegativeInteger(capacity.gpuCount); +} + export function validateClusterWorkerCredentialManagementClientResult( value: unknown, command: Readonly, @@ -143,7 +260,17 @@ export function validateClusterWorkerCredentialManagementClientResult( ? ['schemaVersion', 'operation', 'approvalStatus', 'plan', 'approval'] : operation === 'worker-credential.decide' ? ['schemaVersion', 'operation', 'status', 'approval'] - : ['schemaVersion', 'operation', 'plan', 'approval', 'stale']; + : operation === 'worker-credential.inspect' + ? ['schemaVersion', 'operation', 'plan', 'approval', 'stale'] + : operation === 'worker-session.inspect' + ? ['schemaVersion', 'operation', 'observedAtMs', 'worker'] + : [ + 'schemaVersion', + 'operation', + 'observedAtMs', + 'workers', + 'nextCursor', + ]; const record = exactRecord(value, keys); if (record.schemaVersion !== 1 || record.operation !== operation) invalid(); if (operation === 'worker-credential.plan') { @@ -157,10 +284,45 @@ export function validateClusterWorkerCredentialManagementClientResult( } else if (operation === 'worker-credential.decide') { if (!['decided', 'existing'].includes(String(record.status))) invalid(); approval(record.approval); - } else { + } else if (operation === 'worker-credential.inspect') { if (typeof record.stale !== 'boolean') invalid(); if (record.plan !== null) plan(record.plan); if (record.approval !== null) approval(record.approval); + } else if (operation === 'worker-session.inspect') { + nonNegativeInteger(record.observedAtMs); + if (record.worker !== null) { + workerObservation(record.worker, true); + if ( + (record.worker as Record).observedAtMs !== + record.observedAtMs + ) { + invalid(); + } + } + } else { + nonNegativeInteger(record.observedAtMs); + boundedText(record.nextCursor, true); + if (!Array.isArray(record.workers) || record.workers.length > 16) invalid(); + let previous = ''; + for (const worker of record.workers) { + workerObservation(worker, false); + const item = worker as Record; + if ( + item.observedAtMs !== record.observedAtMs || + typeof item.workerId !== 'string' || + item.workerId <= previous + ) { + invalid(); + } + previous = item.workerId; + } + if ( + (record.nextCursor !== null && + (record.workers.length === 0 || record.nextCursor !== previous)) || + (record.nextCursor === null && record.workers.length > 16) + ) { + invalid(); + } } return Object.freeze( record as unknown as ClusterWorkerCredentialManagementTransportResult, diff --git a/packages/ql3-cluster-admin/test/workerCredentialManagementApproval.test.cjs b/packages/ql3-cluster-admin/test/workerCredentialManagementApproval.test.cjs index 386d74f5..20e19470 100644 --- a/packages/ql3-cluster-admin/test/workerCredentialManagementApproval.test.cjs +++ b/packages/ql3-cluster-admin/test/workerCredentialManagementApproval.test.cjs @@ -4,6 +4,9 @@ const { test } = require('node:test'); const { createClusterWorkerCredentialManagementService, } = require('@qinglong/cluster-admin/worker-credential-management'); +const { + canonicalRemoteWorkerCapabilities, +} = require('@qinglong/runtime-core/remote-dispatch'); const REQUESTER = Object.freeze({ subject: Object.freeze({ type: 'user', id: 'operator-a' }), @@ -20,6 +23,32 @@ const REVIEWER = Object.freeze({ assurance: 'hardware', }); +function sessionRow() { + const capabilities = canonicalRemoteWorkerCapabilities({ + architecture: 'arm64', + executors: ['remote-worker'], + protocolVersion: '1.0.0', + supportTier: 'tier1', + runtimes: [{ name: 'node', version: '24.18.0' }], + }); + return { + observedAtMs: 2_000, + workerId: 'worker-a', + sessionId: '018f0f5d-7b6a-7a11-8f4d-2f7b4f477001', + generation: 2, + status: 'online', + version: 5, + capabilitiesJson: capabilities.json, + capabilitiesHash: capabilities.hash, + maxConcurrentRuns: 2, + availableSlots: 1, + registeredAtMs: 1_000, + lastHeartbeatAtMs: 1_900, + leaseExpiresAtMs: 3_000, + updatedAtMs: 1_900, + }; +} + function approvalFixture() { const plans = new Map(); const approvals = new Map(); @@ -76,6 +105,9 @@ function approvalFixture() { const stored = audits.get(values[0]); return { rows: stored ? [stored] : [], rowCount: stored ? 1 : 0 }; } + if (text.includes('FROM "ql3"."worker_sessions"')) { + return { rows: [sessionRow()], rowCount: 1 }; + } if (text.includes('"ql3"."lock_approval_policy_fence"')) { return { rows: [{ matches: true }], rowCount: 1 }; } @@ -277,3 +309,50 @@ test('authorizes and consumes durable quota before reading management state', as assert.equal(quotaCalls[1].operation, 'worker-credential.propose'); assert.equal(quotaCalls[1].planReads, readsBefore); }); + +test('observes one Session and one bounded page only after worker.manage quota', async () => { + const state = approvalFixture(); + const quotaCalls = []; + const service = createClusterWorkerCredentialManagementService({ + pool: state.pool, + now: () => 2_000, + quota: { + async consume(command) { + quotaCalls.push(command); + return { admitted: true, retryAfterMs: null }; + }, + }, + }); + const inspection = await service.inspectSession({ + authorityProjectId: 'cluster-authority', + workerId: 'worker-a', + inspectionId: 'worker-session-inspection-a', + principal: REQUESTER, + }); + const page = await service.listSessions({ + authorityProjectId: 'cluster-authority', + afterWorkerId: null, + inspectionId: 'worker-session-list-a', + principal: REQUESTER, + }); + assert.equal(inspection.worker.workerId, 'worker-a'); + assert.equal(inspection.worker.compatibility, 'default_placement'); + assert.equal(page.workers.length, 1); + assert.equal(page.workers[0].workerId, 'worker-a'); + assert.deepEqual( + quotaCalls.map(({ operation, idempotencyKey }) => ({ + operation, + idempotencyKey, + })), + [ + { + operation: 'worker-session.observe', + idempotencyKey: 'session-inspect:worker-session-inspection-a', + }, + { + operation: 'worker-session.observe', + idempotencyKey: 'session-list:worker-session-list-a', + }, + ], + ); +}); diff --git a/packages/ql3-cluster-admin/test/workerCredentialManagementClient.test.cjs b/packages/ql3-cluster-admin/test/workerCredentialManagementClient.test.cjs index e01f9d0f..299cd206 100644 --- a/packages/ql3-cluster-admin/test/workerCredentialManagementClient.test.cjs +++ b/packages/ql3-cluster-admin/test/workerCredentialManagementClient.test.cjs @@ -82,6 +82,39 @@ const approval = Object.freeze({ actionDigest: 'd'.repeat(64), previewDigest: plan.previewDigest, }); +const session = Object.freeze({ + workerId: 'worker-1', + sessionId: '018f0f5d-7b6a-7a11-8f4d-2f7b4f477001', + generation: 2, + sessionVersion: 5, + lifecycle: 'online', + compatibility: 'default_placement', + architecture: 'arm64', + supportTier: 'tier1', + protocolVersion: '1.0.0', + operatingSystem: 'linux', + maxConcurrentRuns: 2, + availableSlots: 1, + registeredAtMs: 900, + lastHeartbeatAtMs: 1_050, + leaseExpiresAtMs: 2_000, + updatedAtMs: 1_050, + observedAtMs: 1_100, + runtimes: Object.freeze([{ name: 'node', version: '24.18.0' }]), + declaredCapacity: Object.freeze({ + cpuCores: 1, + memoryBytes: 268_435_456, + diskBytes: 1_073_741_824, + gpuCount: 0, + }), +}); +const sessionSummary = Object.freeze( + Object.fromEntries( + Object.entries(session).filter( + ([key]) => key !== 'runtimes' && key !== 'declaredCapacity', + ), + ), +); function command(operation) { return { @@ -152,7 +185,7 @@ afterEach(() => { } }); -test('validates all four low-sensitive Worker management results', () => { +test('validates all six low-sensitive Worker management results', () => { const fixtures = [ [ 'worker-credential.plan', @@ -187,6 +220,25 @@ test('validates all four low-sensitive Worker management results', () => { stale: false, }, ], + [ + 'worker-session.inspect', + { + schemaVersion: 1, + operation: 'worker-session.inspect', + observedAtMs: 1_100, + worker: session, + }, + ], + [ + 'worker-session.list', + { + schemaVersion: 1, + operation: 'worker-session.list', + observedAtMs: 1_100, + workers: [sessionSummary], + nextCursor: null, + }, + ], ]; for (const [operation, result] of fixtures) { assert.equal( @@ -223,6 +275,51 @@ test('rejects widened and secret-bearing response shapes', () => { command('worker-credential.inspect'), ), ); + assert.throws(() => + validateClusterWorkerCredentialManagementClientResult( + { + schemaVersion: 1, + operation: 'worker-session.inspect', + observedAtMs: 1_100, + worker: { ...session, labels: { secret: 'value' } }, + }, + command('worker-session.inspect'), + ), + ); + assert.throws(() => + validateClusterWorkerCredentialManagementClientResult( + { + schemaVersion: 1, + operation: 'worker-session.list', + observedAtMs: 1_100, + workers: [ + { ...sessionSummary, workerId: 'worker-b' }, + { ...sessionSummary, workerId: 'worker-a' }, + ], + nextCursor: null, + }, + command('worker-session.list'), + ), + ); + for (const worker of [ + { ...session, workerId: null }, + { ...session, generation: -1 }, + { ...session, availableSlots: 3 }, + { ...session, observedAtMs: 1_000 }, + { ...session, declaredCapacity: { ...session.declaredCapacity, gpuCount: -1 } }, + ]) { + assert.throws(() => + validateClusterWorkerCredentialManagementClientResult( + { + schemaVersion: 1, + operation: 'worker-session.inspect', + observedAtMs: worker.observedAtMs, + worker, + }, + command('worker-session.inspect'), + ), + ); + } }); test('requires one matching private client certificate identity before connect', async () => { diff --git a/packages/ql3-cluster-admin/test/workerCredentialManagementTransport.test.cjs b/packages/ql3-cluster-admin/test/workerCredentialManagementTransport.test.cjs index 5fd3b145..df76c2dd 100644 --- a/packages/ql3-cluster-admin/test/workerCredentialManagementTransport.test.cjs +++ b/packages/ql3-cluster-admin/test/workerCredentialManagementTransport.test.cjs @@ -69,6 +69,41 @@ function approval(planValue) { }); } +const SESSION_OBSERVATION = Object.freeze({ + workerId: 'worker-a', + sessionId: '018f0f5d-7b6a-7a11-8f4d-2f7b4f477001', + generation: 2, + sessionVersion: 5, + lifecycle: 'online', + compatibility: 'default_placement', + architecture: 'arm64', + supportTier: 'tier1', + protocolVersion: '1.0.0', + operatingSystem: 'linux', + maxConcurrentRuns: 2, + availableSlots: 1, + registeredAtMs: 900, + lastHeartbeatAtMs: 1_050, + leaseExpiresAtMs: 2_000, + updatedAtMs: 1_050, + observedAtMs: 1_100, + runtimes: Object.freeze([{ name: 'node', version: '24.18.0' }]), + declaredCapacity: Object.freeze({ + cpuCores: 1, + memoryBytes: 268_435_456, + diskBytes: 1_073_741_824, + gpuCount: 0, + }), +}); + +const SESSION_SUMMARY = Object.freeze( + Object.fromEntries( + Object.entries(SESSION_OBSERVATION).filter( + ([key]) => key !== 'runtimes' && key !== 'declaredCapacity', + ), + ), +); + function commands() { return [ { @@ -122,10 +157,28 @@ function commands() { inspectionId: 'inspection-worker-a-generation-2', }, }, + { + schemaVersion: 1, + operation: 'worker-session.inspect', + request: { + authorityProjectId: 'cluster-authority', + workerId: 'worker-a', + inspectionId: 'inspection-worker-session-a', + }, + }, + { + schemaVersion: 1, + operation: 'worker-session.list', + request: { + authorityProjectId: 'cluster-authority', + afterWorkerId: null, + inspectionId: 'inspection-worker-session-list-a', + }, + }, ]; } -test('routes the four public commands with strong User authority and low-sensitive results', async () => { +test('routes six public commands with strong User authority and low-sensitive results', async () => { const planValue = plan(); const approvalValue = approval(planValue); const calls = []; @@ -154,6 +207,18 @@ test('routes the four public commands with strong User authority and low-sensiti stale: false, }; }, + async inspectSession(request) { + calls.push(['inspectSession', request]); + return { observedAtMs: 1_100, worker: SESSION_OBSERVATION }; + }, + async listSessions(request) { + calls.push(['listSessions', request]); + return { + observedAtMs: 1_100, + workers: [SESSION_SUMMARY], + nextCursor: null, + }; + }, }; const transport = createClusterWorkerCredentialManagementTransport({ service, @@ -170,7 +235,7 @@ test('routes the four public commands with strong User authority and low-sensiti } assert.deepEqual( calls.map(([kind]) => kind), - ['plan', 'propose', 'decide', 'inspect'], + ['plan', 'propose', 'decide', 'inspect', 'inspectSession', 'listSessions'], ); for (const [, request] of calls) { assert.deepEqual(request.principal, principal()); @@ -182,11 +247,15 @@ test('routes the four public commands with strong User authority and low-sensiti 'worker-credential.propose', 'worker-credential.decide', 'worker-credential.inspect', + 'worker-session.inspect', + 'worker-session.list', ], ); assert.equal(results[0].plan.planDigest, planValue.planDigest); assert.equal(results[1].approval.actionDigest, planValue.planDigest); assert.equal(results[3].stale, false); + assert.equal(results[4].worker.compatibility, 'default_placement'); + assert.equal(results[5].workers[0].workerId, 'worker-a'); const serialized = JSON.stringify(results); assert.doesNotMatch(serialized, /authenticationId|credential-token|secret/i); }); @@ -194,7 +263,14 @@ test('routes the four public commands with strong User authority and low-sensiti test('rejects weak or unavailable identity before management authority', async () => { let calls = 0; const service = Object.fromEntries( - ['plan', 'propose', 'decide', 'inspectAuthorized'].map((name) => [ + [ + 'plan', + 'propose', + 'decide', + 'inspectAuthorized', + 'inspectSession', + 'listSessions', + ].map((name) => [ name, async () => { calls += 1; @@ -232,6 +308,8 @@ test('rejects widened and internal commands before authentication', async () => async propose() {}, async decide() {}, async inspectAuthorized() {}, + async inspectSession() {}, + async listSessions() {}, }, }); let authentications = 0; diff --git a/packages/ql3-cluster-postgres/src/migration/migrationManifest.ts b/packages/ql3-cluster-postgres/src/migration/migrationManifest.ts index 947c1c7f..5810215b 100644 --- a/packages/ql3-cluster-postgres/src/migration/migrationManifest.ts +++ b/packages/ql3-cluster-postgres/src/migration/migrationManifest.ts @@ -348,5 +348,10 @@ export const postgresqlMainMigrationManifest: MigrationStreamManifest = checksum: '2fcac38386581189db63faacff325356f11c4529a8db9cef6be1a1ca706aaf10', }), + Object.freeze({ + id: 'pg-0069-worker-session-management-observation', + checksum: + '1191255575589abc2686b391827607abddb4edb78007245dbaaf45dc1c4e5e8b', + }), ]), }); diff --git a/packages/ql3-cluster-postgres/src/migrations/index.ts b/packages/ql3-cluster-postgres/src/migrations/index.ts index 64faae2a..bf3888ea 100644 --- a/packages/ql3-cluster-postgres/src/migrations/index.ts +++ b/packages/ql3-cluster-postgres/src/migrations/index.ts @@ -71,6 +71,7 @@ import { pg0065ApprovedActionManualRecoveryMigration } from '../approved-action/ import { pg0066CancellationDispatchMigration } from '../run/migrations/pg-0066-cancellation-dispatch'; import { pg0067CancellationDispatchManagementMigration } from '../run-management/pg-0067-cancellation-dispatch-management'; import { pg0068CancellationDispatchProjectKeysetMigration } from '../run-management/pg-0068-cancellation-dispatch-project-keyset'; +import { pg0069WorkerSessionManagementObservationMigration } from '../remote-execution/pg-0069-worker-session-management-observation'; export const postgresqlMainMigrationStream: MigrationStreamDefinition = Object.freeze({ @@ -147,5 +148,6 @@ export const postgresqlMainMigrationStream: MigrationStreamDefinition; + +const SELECT_COLUMNS = ` + worker_id AS "workerId", + session_id AS "sessionId", + generation AS "generation", + status AS "status", + version AS "version", + capabilities_json AS "capabilitiesJson", + capabilities_hash AS "capabilitiesHash", + max_concurrent_runs AS "maxConcurrentRuns", + available_slots AS "availableSlots", + registered_at_ms AS "registeredAtMs", + last_heartbeat_at_ms AS "lastHeartbeatAtMs", + lease_expires_at_ms AS "leaseExpiresAtMs", + updated_at_ms AS "updatedAtMs" +`.trim(); + +function string(row: Row, key: string): string { + const value = row[key]; + if (typeof value !== 'string' || value.length === 0) { + throw new TypeError(`PostgreSQL Worker observation ${key} is invalid`); + } + return value; +} + +function integer(row: Row, key: string): number { + const value = row[key]; + const normalized = + typeof value === 'string' && /^(0|[1-9]\d*)$/.test(value) + ? Number(value) + : value; + if (!Number.isSafeInteger(normalized) || (normalized as number) < 0) { + throw new TypeError(`PostgreSQL Worker observation ${key} is invalid`); + } + return normalized as number; +} + +function record(row: Row): Readonly { + const status = string(row, 'status'); + if (!WORKER_SESSION_STATUSES.includes(status as WorkerSessionStatus)) { + throw new TypeError('PostgreSQL Worker observation status is invalid'); + } + const worker: WorkerSessionRecord = Object.freeze({ + workerId: string(row, 'workerId'), + sessionId: string(row, 'sessionId'), + generation: integer(row, 'generation'), + status: status as WorkerSessionStatus, + version: integer(row, 'version'), + capabilitiesJson: string(row, 'capabilitiesJson'), + capabilitiesHash: string(row, 'capabilitiesHash'), + maxConcurrentRuns: integer(row, 'maxConcurrentRuns'), + availableSlots: integer(row, 'availableSlots'), + registeredAtMs: integer(row, 'registeredAtMs'), + lastHeartbeatAtMs: integer(row, 'lastHeartbeatAtMs'), + leaseExpiresAtMs: integer(row, 'leaseExpiresAtMs'), + updatedAtMs: integer(row, 'updatedAtMs'), + }); + assertWorkerId(worker.workerId); + assertWorkerSessionId(worker.sessionId); + assertWorkerCapabilitiesSnapshot( + worker.capabilitiesJson, + worker.capabilitiesHash, + ); + assertWorkerConcurrency(worker.maxConcurrentRuns, worker.availableSlots); + return worker; +} + +function observedAtMs(row: Row): number { + return integer(row, 'observedAtMs'); +} + +export class PostgresWorkerSessionObservationRepository { + constructor(private readonly pool: PostgresPool) { + if (!pool || typeof pool.query !== 'function') { + throw new TypeError('PostgreSQL Worker observation Pool is invalid'); + } + } + + async inspect(workerId: string): Promise> { + assertWorkerId(workerId); + const result = await this.pool.query( + `WITH observation AS ( + SELECT floor(extract(epoch FROM statement_timestamp()) * 1000)::bigint AS observed_at_ms + ) + SELECT observation.observed_at_ms AS "observedAtMs", worker.* + FROM observation + LEFT JOIN LATERAL ( + SELECT ${SELECT_COLUMNS} + FROM "ql3"."worker_sessions" + WHERE worker_id = $1 + LIMIT 2 + ) AS worker ON TRUE`, + [workerId], + ); + if (result.rows.length !== 1) { + throw new TypeError('PostgreSQL Worker inspection violated its bound'); + } + const observed = observedAtMs(result.rows[0]!); + if ( + result.rows[0]!.workerId !== null && + string(result.rows[0]!, 'workerId') !== workerId + ) { + throw new TypeError('PostgreSQL Worker inspection identity drifted'); + } + return Object.freeze({ + observedAtMs: observed, + worker: + result.rows[0]!.workerId === null + ? null + : projectWorkerSessionObservation(record(result.rows[0]!), observed), + }); + } + + async list( + afterWorkerId: string | null, + ): Promise> { + if (afterWorkerId !== null) assertWorkerId(afterWorkerId); + const result = await this.pool.query( + `WITH observation AS ( + SELECT floor(extract(epoch FROM statement_timestamp()) * 1000)::bigint AS observed_at_ms + ), page AS ( + SELECT ${SELECT_COLUMNS} + FROM "ql3"."worker_sessions" + WHERE ($1::varchar IS NULL OR worker_id > $1) + ORDER BY worker_id + LIMIT $2 + ) + SELECT observation.observed_at_ms AS "observedAtMs", page.* + FROM observation LEFT JOIN page ON TRUE + ORDER BY page."workerId"`, + [afterWorkerId, MAX_WORKER_SESSION_OBSERVATION_PAGE_SIZE + 1], + ); + if ( + result.rows.length < 1 || + result.rows.length > MAX_WORKER_SESSION_OBSERVATION_PAGE_SIZE + 1 + ) { + throw new TypeError('PostgreSQL Worker observation page violated its bound'); + } + const observed = observedAtMs(result.rows[0]!); + if (result.rows.some((row) => observedAtMs(row) !== observed)) { + throw new TypeError('PostgreSQL Worker observation clock drifted'); + } + const observations = + result.rows[0]!.workerId === null + ? [] + : result.rows.map((row) => + projectWorkerSessionObservation(record(row), observed), + ); + for (let index = 0; index < observations.length; index += 1) { + const workerId = observations[index]!.workerId; + const previous = observations[index - 1]?.workerId ?? afterWorkerId; + if (previous !== null && workerId <= previous) { + throw new TypeError('PostgreSQL Worker observation page is unordered'); + } + } + const page = observations.slice( + 0, + MAX_WORKER_SESSION_OBSERVATION_PAGE_SIZE, + ); + return Object.freeze({ + observedAtMs: observed, + workers: Object.freeze(page.map(summarizeWorkerSessionObservation)), + nextCursor: + observations.length > MAX_WORKER_SESSION_OBSERVATION_PAGE_SIZE + ? page.at(-1)!.workerId + : null, + }); + } +} diff --git a/packages/ql3-cluster-postgres/src/schema/schemaContract.ts b/packages/ql3-cluster-postgres/src/schema/schemaContract.ts index e6ce6b5e..576a307c 100644 --- a/packages/ql3-cluster-postgres/src/schema/schemaContract.ts +++ b/packages/ql3-cluster-postgres/src/schema/schemaContract.ts @@ -21,8 +21,8 @@ export interface PostgresSchemaContractTrigger { export interface PostgresSchemaContract { readonly schema: 'ql3'; readonly contractName: 'control-core'; - readonly contractVersion: 67; - readonly migrationId: 'pg-0068-cancellation-dispatch-project-keyset'; + readonly contractVersion: 68; + readonly migrationId: 'pg-0069-worker-session-management-observation'; readonly minimumServerMajor: 16; readonly maximumServerMajor: 18; readonly capabilities: Readonly<{ @@ -92,6 +92,7 @@ export interface PostgresSchemaContract { tool_invocation_artifact: 1; trigger_definition: 1; worker_session: 1; + worker_session_observation: 1; worker_credential: 1; worker_credential_delivery: 1; worker_credential_execution_receipt: 1; @@ -121,8 +122,8 @@ export const postgresqlControlSchemaContract: PostgresSchemaContract = Object.freeze({ schema: 'ql3', contractName: 'control-core', - contractVersion: 67, - migrationId: 'pg-0068-cancellation-dispatch-project-keyset', + contractVersion: 68, + migrationId: 'pg-0069-worker-session-management-observation', minimumServerMajor: 16, maximumServerMajor: 18, capabilities: Object.freeze({ @@ -192,6 +193,7 @@ export const postgresqlControlSchemaContract: PostgresSchemaContract = tool_invocation_artifact: 1, trigger_definition: 1, worker_session: 1, + worker_session_observation: 1, worker_credential: 1, worker_credential_delivery: 1, worker_credential_execution_receipt: 1, diff --git a/packages/ql3-cluster-postgres/src/schema/schemaReadiness.ts b/packages/ql3-cluster-postgres/src/schema/schemaReadiness.ts index 8274e67d..022390b6 100644 --- a/packages/ql3-cluster-postgres/src/schema/schemaReadiness.ts +++ b/packages/ql3-cluster-postgres/src/schema/schemaReadiness.ts @@ -1509,7 +1509,8 @@ const REQUIRED_WORKER_CREDENTIAL_MANAGER_PRIVILEGES: RequiredPrivileges = name === 'schema_migrations' || name === 'schema_capabilities' || name === 'projects' || - name === 'project_role_bindings' + name === 'project_role_bindings' || + name === 'worker_sessions' ? { ...NO_TABLE_PRIVILEGES, select: true } : name === 'worker_credential_management_plans' || name === 'security_audit_events' diff --git a/packages/ql3-cluster-postgres/src/worker-credential/workerCredentialManagementQuotaRepository.ts b/packages/ql3-cluster-postgres/src/worker-credential/workerCredentialManagementQuotaRepository.ts index 6d45cf0b..5e9e27da 100644 --- a/packages/ql3-cluster-postgres/src/worker-credential/workerCredentialManagementQuotaRepository.ts +++ b/packages/ql3-cluster-postgres/src/worker-credential/workerCredentialManagementQuotaRepository.ts @@ -10,6 +10,7 @@ const OPERATIONS = [ 'worker-credential.propose', 'worker-credential.decide', 'worker-credential.inspect', + 'worker-session.observe', ] as const; type Operation = (typeof OPERATIONS)[number]; type Row = Record; @@ -31,6 +32,7 @@ const DEFAULT_LIMITS: Readonly> = Object.freeze({ 'worker-credential.propose': 30, 'worker-credential.decide': 60, 'worker-credential.inspect': 600, + 'worker-session.observe': 600, }); const ID = /^[A-Za-z0-9][A-Za-z0-9._:/-]{0,254}$/; const PROJECT = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/; diff --git a/packages/ql3-cluster-postgres/src/worker-credential/workerCredentialManager.ts b/packages/ql3-cluster-postgres/src/worker-credential/workerCredentialManager.ts index 960ecdc3..2b5b59f9 100644 --- a/packages/ql3-cluster-postgres/src/worker-credential/workerCredentialManager.ts +++ b/packages/ql3-cluster-postgres/src/worker-credential/workerCredentialManager.ts @@ -44,6 +44,7 @@ export { postgresqlMainMigrationManifest } from '../migration/migrationManifest' export { PostgresApprovalRequestRepository } from '../approved-action/approvalRequestRepository'; export { PostgresProjectPolicyRepository } from '../security/projectPolicyRepository'; export { PostgresWorkerCredentialManagementPlanRepository } from './workerCredentialManagementPlanRepository'; +export { PostgresWorkerSessionObservationRepository } from '../remote-execution/workerSessionObservationRepository'; export { PostgresPluginPackageIdentityKeysetLedgerRepository as PostgresWorkerCredentialManagementIdentityKeysetLedgerRepository, type ClusterManagementIdentityAuthority, diff --git a/packages/ql3-cluster-postgres/test/postgresqlMigrationDefinitions.test.cjs b/packages/ql3-cluster-postgres/test/postgresqlMigrationDefinitions.test.cjs index 8fa5b217..015d6698 100644 --- a/packages/ql3-cluster-postgres/test/postgresqlMigrationDefinitions.test.cjs +++ b/packages/ql3-cluster-postgres/test/postgresqlMigrationDefinitions.test.cjs @@ -119,6 +119,7 @@ test('defines the immutable PostgreSQL capability and Run core stream', async () 'pg-0066-cancellation-dispatch', 'pg-0067-cancellation-dispatch-management', 'pg-0068-cancellation-dispatch-project-keyset', + 'pg-0069-worker-session-management-observation', ], ); for (const migration of postgresqlMainMigrationStream.migrations) { @@ -597,6 +598,11 @@ test('freezes every published PostgreSQL migration checksum', () => { checksum: '2fcac38386581189db63faacff325356f11c4529a8db9cef6be1a1ca706aaf10', }, + { + id: 'pg-0069-worker-session-management-observation', + checksum: + '1191255575589abc2686b391827607abddb4edb78007245dbaaf45dc1c4e5e8b', + }, ]; assert.deepEqual( postgresqlMainMigrationStream.migrations.map(({ id, checksum }) => ({ @@ -2421,3 +2427,36 @@ test('advances capability v67 with a Project-scoped blocked keyset', async () => /migration_id = 'pg-0067-cancellation-dispatch-management'/, ); }); + +test('advances capability v68 with read-only Worker session observation', async () => { + const migration = migrationById( + 'pg-0069-worker-session-management-observation', + ); + const statements = []; + await migration.up({ + async query(statement) { + statements.push(statement); + return { rows: [] }; + }, + }); + const sql = statements.join('\n'); + assert.match( + sql, + /REVOKE ALL ON "ql3"\."worker_sessions" FROM ql3_worker_credential_manager/, + ); + assert.match( + sql, + /GRANT SELECT ON "ql3"\."worker_sessions" TO ql3_worker_credential_manager/, + ); + assert.doesNotMatch( + sql, + /GRANT (?:INSERT|UPDATE|DELETE|TRUNCATE)[^;]+worker_sessions[^;]+ql3_worker_credential_manager/, + ); + assert.match(sql, /contract_version = 68/); + assert.match(sql, /"worker_session_observation":1/); + assert.match(sql, /contract_version = 67/); + assert.match( + sql, + /migration_id = 'pg-0068-cancellation-dispatch-project-keyset'/, + ); +}); diff --git a/packages/ql3-cluster-postgres/test/postgresqlSchemaReadiness.test.cjs b/packages/ql3-cluster-postgres/test/postgresqlSchemaReadiness.test.cjs index d6b152a2..0900776b 100644 --- a/packages/ql3-cluster-postgres/test/postgresqlSchemaReadiness.test.cjs +++ b/packages/ql3-cluster-postgres/test/postgresqlSchemaReadiness.test.cjs @@ -561,6 +561,7 @@ function workerCredentialPrivileges(kind) { ? [ 'worker_credential_management_quota_buckets', 'plugin_package_identity_keyset_ledger', + 'worker_sessions', ] : []), ...(manager @@ -835,7 +836,7 @@ test('accepts the exact PostgreSQL control schema and least-privilege runtime ro serverMajor: 16, currentUser: 'ql3_runtime', contractName: 'control-core', - contractVersion: 67, + contractVersion: 68, migrationIds: [ 'pg-0001-schema-capability', 'pg-0002-run-core', @@ -905,6 +906,7 @@ test('accepts the exact PostgreSQL control schema and least-privilege runtime ro 'pg-0066-cancellation-dispatch', 'pg-0067-cancellation-dispatch-management', 'pg-0068-cancellation-dispatch-project-keyset', + 'pg-0069-worker-session-management-observation', ], }); }); @@ -935,10 +937,10 @@ test('accepts the exact schema and isolated least-privilege admin role', async ( }), ); assert.equal(report.currentUser, 'ql3_admin'); - assert.equal(report.contractVersion, 67); + assert.equal(report.contractVersion, 68); assert.equal( report.migrationIds.at(-1), - 'pg-0068-cancellation-dispatch-project-keyset', + 'pg-0069-worker-session-management-observation', ); }); @@ -951,10 +953,10 @@ test('accepts the isolated least-privilege automation manager role', async () => }), ); assert.equal(report.currentUser, 'ql3_automation_manager'); - assert.equal(report.contractVersion, 67); + assert.equal(report.contractVersion, 68); assert.equal( report.migrationIds.at(-1), - 'pg-0068-cancellation-dispatch-project-keyset', + 'pg-0069-worker-session-management-observation', ); const widened = automationManagerPrivileges(); @@ -983,10 +985,10 @@ test('accepts the isolated least-privilege human Approval manager role', async ( }), ); assert.equal(report.currentUser, 'ql3_approval_manager'); - assert.equal(report.contractVersion, 67); + assert.equal(report.contractVersion, 68); assert.equal( report.migrationIds.at(-1), - 'pg-0068-cancellation-dispatch-project-keyset', + 'pg-0069-worker-session-management-observation', ); const widened = approvalManagerPrivileges(); @@ -1017,10 +1019,10 @@ test('accepts the isolated least-privilege Run manager role', async () => { }), ); assert.equal(report.currentUser, 'ql3_run_manager'); - assert.equal(report.contractVersion, 67); + assert.equal(report.contractVersion, 68); assert.equal( report.migrationIds.at(-1), - 'pg-0068-cancellation-dispatch-project-keyset', + 'pg-0069-worker-session-management-observation', ); const widened = runManagerPrivileges(); @@ -1181,10 +1183,10 @@ test('accepts the exact schema and isolated Worker ingress role', async () => { }), ); assert.equal(report.currentUser, 'ql3_worker_ingress'); - assert.equal(report.contractVersion, 67); + assert.equal(report.contractVersion, 68); assert.equal( report.migrationIds.at(-1), - 'pg-0068-cancellation-dispatch-project-keyset', + 'pg-0069-worker-session-management-observation', ); }); diff --git a/packages/ql3-cluster-postgres/test/workerSessionObservationRepository.test.cjs b/packages/ql3-cluster-postgres/test/workerSessionObservationRepository.test.cjs new file mode 100644 index 00000000..61892b45 --- /dev/null +++ b/packages/ql3-cluster-postgres/test/workerSessionObservationRepository.test.cjs @@ -0,0 +1,119 @@ +const assert = require('node:assert/strict'); +const { test } = require('node:test'); + +const { + canonicalRemoteWorkerCapabilities, +} = require('@qinglong/runtime-core/remote-dispatch'); +const { + PostgresWorkerSessionObservationRepository, +} = require('@qinglong/cluster-postgres/worker-credential-manager'); + +function row(index, overrides = {}) { + const canonical = canonicalRemoteWorkerCapabilities( + overrides.capabilities ?? { + architecture: index % 2 === 0 ? 'arm64' : 's390x', + executors: ['remote-worker'], + protocolVersion: '1.0.0', + supportTier: index % 2 === 0 ? 'tier1' : 'candidate', + runtimes: [{ name: 'node', version: '24.18.0' }], + }, + ); + return { + observedAtMs: 2_000, + workerId: `worker-${String(index).padStart(2, '0')}`, + sessionId: `018f0f5d-7b6a-7a11-8f4d-${String(index + 1).padStart( + 12, + '0', + )}`, + generation: 1, + status: 'online', + version: 2, + capabilitiesJson: canonical.json, + capabilitiesHash: canonical.hash, + maxConcurrentRuns: 4, + availableSlots: 2, + registeredAtMs: 1_000, + lastHeartbeatAtMs: 1_900, + leaseExpiresAtMs: 3_000, + updatedAtMs: 1_900, + ...Object.fromEntries( + Object.entries(overrides).filter(([key]) => key !== 'capabilities'), + ), + }; +} + +test('inspects one exact Worker without exposing raw capability content', async () => { + const queries = []; + const repository = new PostgresWorkerSessionObservationRepository({ + async query(text, values) { + queries.push({ text, values }); + return { rows: [row(0)], rowCount: 1 }; + }, + }); + const result = await repository.inspect('worker-00'); + assert.equal(result.observedAtMs, 2_000); + assert.equal(result.worker.workerId, 'worker-00'); + assert.equal(result.worker.compatibility, 'default_placement'); + assert.deepEqual(result.worker.runtimes, [ + { name: 'node', version: '24.18.0' }, + ]); + assert.equal(Object.hasOwn(result.worker, 'capabilitiesJson'), false); + assert.deepEqual(queries[0].values, ['worker-00']); + assert.match(queries[0].text, /LIMIT 2/); +}); + +test('returns a masked absent inspection and a fixed sixteen-item keyset page', async () => { + let call = 0; + const repository = new PostgresWorkerSessionObservationRepository({ + async query(_text, values) { + call += 1; + if (call === 1) { + return { + rows: [{ observedAtMs: 2_000, workerId: null }], + rowCount: 1, + }; + } + assert.deepEqual(values, [null, 17]); + return { + rows: Array.from({ length: 17 }, (_, index) => row(index)), + rowCount: 17, + }; + }, + }); + assert.deepEqual(await repository.inspect('missing-worker'), { + observedAtMs: 2_000, + worker: null, + }); + const page = await repository.list(null); + assert.equal(page.workers.length, 16); + assert.equal(page.nextCursor, 'worker-15'); + assert.equal(page.workers[1].compatibility, 'explicit_placement_required'); + assert.equal(Object.hasOwn(page.workers[0], 'runtimes'), false); + assert.equal(Object.hasOwn(page.workers[0], 'declaredCapacity'), false); +}); + +test('fails closed on identity, ordering and database-clock drift', async () => { + const identity = new PostgresWorkerSessionObservationRepository({ + async query() { + return { rows: [row(1)], rowCount: 1 }; + }, + }); + await assert.rejects(identity.inspect('worker-00'), /identity drifted/); + + const unordered = new PostgresWorkerSessionObservationRepository({ + async query() { + return { rows: [row(1), row(0)], rowCount: 2 }; + }, + }); + await assert.rejects(unordered.list(null), /page is unordered/); + + const clock = new PostgresWorkerSessionObservationRepository({ + async query() { + return { + rows: [row(0), row(1, { observedAtMs: 2_001 })], + rowCount: 2, + }; + }, + }); + await assert.rejects(clock.list(null), /clock drifted/); +}); diff --git a/packages/ql3-runtime-core/package.json b/packages/ql3-runtime-core/package.json index 7bd20fcc..5af679fb 100644 --- a/packages/ql3-runtime-core/package.json +++ b/packages/ql3-runtime-core/package.json @@ -56,6 +56,9 @@ "worker-session": [ "dist/worker/workerSession.d.ts" ], + "worker-session-observation": [ + "dist/worker/workerSessionObservation.d.ts" + ], "worker-session-transport": [ "dist/worker/workerSessionTransport.d.ts" ], @@ -804,6 +807,11 @@ "require": "./dist/worker/workerSession.js", "default": "./dist/worker/workerSession.js" }, + "./worker-session-observation": { + "types": "./dist/worker/workerSessionObservation.d.ts", + "require": "./dist/worker/workerSessionObservation.js", + "default": "./dist/worker/workerSessionObservation.js" + }, "./worker-session-transport": { "types": "./dist/worker/workerSessionTransport.d.ts", "require": "./dist/worker/workerSessionTransport.js", diff --git a/packages/ql3-runtime-core/src/worker/workerSessionObservation.ts b/packages/ql3-runtime-core/src/worker/workerSessionObservation.ts new file mode 100644 index 00000000..827a7f5a --- /dev/null +++ b/packages/ql3-runtime-core/src/worker/workerSessionObservation.ts @@ -0,0 +1,144 @@ +import { + REMOTE_WORKER_PROTOCOL_RANGE, + parseRemoteWorkerCapabilities, + remoteWorkerProtocolIsCompatible, + type RemoteWorkerArchitecture, + type RemoteWorkerRuntimeCapability, + type RemoteWorkerSupportTier, +} from '../remote-execution/remoteWorkerPlacement'; +import { + assertWorkerSessionRecord, + type WorkerSessionRecord, +} from './workerSession'; + +export const MAX_WORKER_SESSION_OBSERVATION_PAGE_SIZE = 16; + +export type WorkerSessionObservedLifecycle = + | 'online' + | 'draining' + | 'offline' + | 'lease_expired'; + +export type WorkerSessionObservedCompatibility = + | 'default_placement' + | 'explicit_placement_required' + | 'protocol_incompatible'; + +export interface WorkerSessionObservationSummary { + readonly workerId: string; + readonly sessionId: string; + readonly generation: number; + readonly sessionVersion: number; + readonly lifecycle: WorkerSessionObservedLifecycle; + readonly compatibility: WorkerSessionObservedCompatibility; + readonly architecture: RemoteWorkerArchitecture; + readonly supportTier: RemoteWorkerSupportTier; + readonly protocolVersion: string; + readonly operatingSystem: string | null; + readonly maxConcurrentRuns: number; + readonly availableSlots: number; + readonly registeredAtMs: number; + readonly lastHeartbeatAtMs: number; + readonly leaseExpiresAtMs: number; + readonly updatedAtMs: number; + readonly observedAtMs: number; +} + +export interface WorkerSessionObservation + extends WorkerSessionObservationSummary { + readonly runtimes: readonly Readonly[]; + readonly declaredCapacity: Readonly<{ + readonly cpuCores: number | null; + readonly memoryBytes: number | null; + readonly diskBytes: number | null; + readonly gpuCount: number; + }>; +} + +export interface WorkerSessionObservationPage { + readonly observedAtMs: number; + readonly workers: readonly Readonly[]; + readonly nextCursor: string | null; +} + +export interface WorkerSessionInspection { + readonly observedAtMs: number; + readonly worker: Readonly | null; +} + +function invalid(message: string): never { + throw new TypeError(`Worker Session observation is invalid: ${message}`); +} + +function observedLifecycle( + worker: Readonly, + observedAtMs: number, +): WorkerSessionObservedLifecycle { + if (worker.status === 'offline') return 'offline'; + if (worker.leaseExpiresAtMs <= observedAtMs) return 'lease_expired'; + return worker.status; +} + +export function projectWorkerSessionObservation( + worker: Readonly, + observedAtMs: number, +): Readonly { + assertWorkerSessionRecord(worker); + if ( + !Number.isSafeInteger(observedAtMs) || + observedAtMs < worker.updatedAtMs + ) { + return invalid('observedAtMs is invalid'); + } + const capabilities = parseRemoteWorkerCapabilities(worker); + const protocolCompatible = remoteWorkerProtocolIsCompatible( + capabilities.protocolVersion, + REMOTE_WORKER_PROTOCOL_RANGE, + ); + const compatibility: WorkerSessionObservedCompatibility = protocolCompatible + ? capabilities.supportTier === 'tier1' + ? 'default_placement' + : 'explicit_placement_required' + : 'protocol_incompatible'; + return Object.freeze({ + workerId: worker.workerId, + sessionId: worker.sessionId, + generation: worker.generation, + sessionVersion: worker.version, + lifecycle: observedLifecycle(worker, observedAtMs), + compatibility, + architecture: capabilities.architecture, + supportTier: capabilities.supportTier, + protocolVersion: capabilities.protocolVersion, + operatingSystem: capabilities.operatingSystem ?? null, + maxConcurrentRuns: worker.maxConcurrentRuns, + availableSlots: worker.availableSlots, + registeredAtMs: worker.registeredAtMs, + lastHeartbeatAtMs: worker.lastHeartbeatAtMs, + leaseExpiresAtMs: worker.leaseExpiresAtMs, + updatedAtMs: worker.updatedAtMs, + observedAtMs, + runtimes: Object.freeze( + (capabilities.runtimes ?? []).map((runtime) => + Object.freeze({ ...runtime }), + ), + ), + declaredCapacity: Object.freeze({ + cpuCores: capabilities.capacity?.cpuCores ?? null, + memoryBytes: capabilities.capacity?.memoryBytes ?? null, + diskBytes: capabilities.capacity?.diskBytes ?? null, + gpuCount: capabilities.capacity?.gpu?.length ?? 0, + }), + }); +} + +export function summarizeWorkerSessionObservation( + observation: Readonly, +): Readonly { + const { + runtimes: _runtimes, + declaredCapacity: _declaredCapacity, + ...summary + } = observation; + return Object.freeze(summary); +} diff --git a/packages/ql3-runtime-core/test/workerSessionObservation.test.cjs b/packages/ql3-runtime-core/test/workerSessionObservation.test.cjs new file mode 100644 index 00000000..6b794f46 --- /dev/null +++ b/packages/ql3-runtime-core/test/workerSessionObservation.test.cjs @@ -0,0 +1,143 @@ +const assert = require('node:assert/strict'); +const { test } = require('node:test'); + +const { + canonicalRemoteWorkerCapabilities, +} = require('@qinglong/runtime-core/remote-dispatch'); +const { + projectWorkerSessionObservation, + summarizeWorkerSessionObservation, +} = require('@qinglong/runtime-core/worker-session-observation'); + +function worker(overrides = {}) { + const canonical = canonicalRemoteWorkerCapabilities( + overrides.capabilities ?? { + architecture: 'arm64', + executors: ['remote-worker'], + protocolVersion: '1.0.0', + supportTier: 'tier1', + operatingSystem: 'linux', + runtimes: [{ name: 'node', version: '24.18.0' }], + labels: { private: 'must-not-project' }, + capacity: { + cpuCores: 2, + memoryBytes: 512 * 1024 * 1024, + diskBytes: 4 * 1024 * 1024 * 1024, + gpu: [{ vendor: 'example', model: 'must-not-project' }], + }, + features: ['must-not-project'], + }, + ); + return Object.freeze({ + workerId: 'worker-a', + sessionId: '018f0f5d-7b6a-7a11-8f4d-2f7b4f477001', + generation: 3, + status: 'online', + version: 9, + capabilitiesJson: canonical.json, + capabilitiesHash: canonical.hash, + maxConcurrentRuns: 4, + availableSlots: 2, + registeredAtMs: 1_000, + lastHeartbeatAtMs: 1_900, + leaseExpiresAtMs: 3_000, + updatedAtMs: 1_900, + ...Object.fromEntries( + Object.entries(overrides).filter(([key]) => key !== 'capabilities'), + ), + }); +} + +test('projects only bounded Worker compatibility, runtime and capacity facts', () => { + const observation = projectWorkerSessionObservation(worker(), 2_000); + assert.deepEqual(observation, { + workerId: 'worker-a', + sessionId: '018f0f5d-7b6a-7a11-8f4d-2f7b4f477001', + generation: 3, + sessionVersion: 9, + lifecycle: 'online', + compatibility: 'default_placement', + architecture: 'arm64', + supportTier: 'tier1', + protocolVersion: '1.0.0', + operatingSystem: 'linux', + maxConcurrentRuns: 4, + availableSlots: 2, + registeredAtMs: 1_000, + lastHeartbeatAtMs: 1_900, + leaseExpiresAtMs: 3_000, + updatedAtMs: 1_900, + observedAtMs: 2_000, + runtimes: [{ name: 'node', version: '24.18.0' }], + declaredCapacity: { + cpuCores: 2, + memoryBytes: 512 * 1024 * 1024, + diskBytes: 4 * 1024 * 1024 * 1024, + gpuCount: 1, + }, + }); + assert.doesNotMatch( + JSON.stringify(observation), + /must-not-project|labels|features|model/, + ); + const summary = summarizeWorkerSessionObservation(observation); + assert.equal(Object.hasOwn(summary, 'runtimes'), false); + assert.equal(Object.hasOwn(summary, 'declaredCapacity'), false); +}); + +test('distinguishes explicit Tier, incompatible protocol and lifecycle state', () => { + const candidate = projectWorkerSessionObservation( + worker({ + capabilities: { + architecture: 's390x', + executors: ['remote-worker'], + protocolVersion: '1.0.0', + supportTier: 'candidate', + }, + }), + 2_000, + ); + assert.equal(candidate.compatibility, 'explicit_placement_required'); + + const incompatible = projectWorkerSessionObservation( + worker({ + capabilities: { + architecture: 'amd64', + executors: ['remote-worker'], + protocolVersion: '2.0.0', + supportTier: 'tier1', + }, + }), + 2_000, + ); + assert.equal(incompatible.compatibility, 'protocol_incompatible'); + assert.equal( + projectWorkerSessionObservation(worker(), 3_000).lifecycle, + 'lease_expired', + ); + assert.equal( + projectWorkerSessionObservation( + worker({ status: 'draining', availableSlots: 0 }), + 2_000, + ).lifecycle, + 'draining', + ); + assert.equal( + projectWorkerSessionObservation( + worker({ + status: 'offline', + availableSlots: 0, + leaseExpiresAtMs: 1_900, + }), + 2_000, + ).lifecycle, + 'offline', + ); +}); + +test('rejects an observation clock before the durable Session head', () => { + assert.throws( + () => projectWorkerSessionObservation(worker(), 1_899), + /observedAtMs is invalid/, + ); +}); diff --git a/test/back/ql3PackageBoundaryAudit.test.cjs b/test/back/ql3PackageBoundaryAudit.test.cjs index 7d635b7e..fbfde0b8 100644 --- a/test/back/ql3PackageBoundaryAudit.test.cjs +++ b/test/back/ql3PackageBoundaryAudit.test.cjs @@ -299,10 +299,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: runtimeCore.rootSourceFileRoles, }, { - sourceFiles: 170, + sourceFiles: 171, rootSourceFiles: 1, rootSourceLines: 160, - nestedSourceFiles: 169, + nestedSourceFiles: 170, rootSourceFileRoles: { 'index.ts': 'public_export' }, }, ); @@ -421,10 +421,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', ( rootSourceFileRoles: clusterPostgres.rootSourceFileRoles, }, { - sourceFiles: 173, + sourceFiles: 175, rootSourceFiles: 1, rootSourceLines: 126, - nestedSourceFiles: 172, + nestedSourceFiles: 174, rootSourceFileRoles: { 'index.ts': 'public_export' }, }, );