feat(ql3): prove legacy rollback readiness

This commit is contained in:
whyour
2026-08-20 18:24:52 +08:00
parent 2cd6ea1d55
commit 3c02ae2020
15 changed files with 1491 additions and 11 deletions
+21
View File
@@ -11,6 +11,27 @@
最新增量证据(2026-08-20):
- D-381/ADR-0474(已接受;OpenRC live actor 待镜像基础设施恢复后补跑):把 service-manager adopted rollback 从仅证明
init/process 的 `legacy_running` 推进到有界、可重放的 2.x core readiness。新增显式 Owner 私有命令
`cutover-legacy-readiness-probe`,绑定 exact cutover/profile/instance/generation、activation、当前 head、legacy-running source
record、port 与精确 2.x version;只允许 `GET 127.0.0.1:<port>/api/system`,没有 caller URL/header/credential、redirect、proxy
或 keep-alive agent。每次请求最多 2 秒/32 KiBEdge 总预算 30 秒/最多 60 次,Standalone 60 秒/最多 120 次;只有 HTTP/envelope
成功、`isInitialized=true` 且 version 精确相等才 no-replace 发布 `0600` 收据并 CAS 到 `legacy_ready`exact replay 发起零次网络
请求,所有 not-ready reason 保持 `legacy_running`。该状态只证明正确 2.x 版本的本机 HTTP core 与初始化,不宣称任务、订阅、
provider、通知、Worker 或全部 API 健康。实现内聚在现有 Local Owner 的 `cutover/legacy-readiness/`,不新增 package、production
dependency、binary、daemon、listener、watcher、timer、数据库连接或部署对象。readiness 聚焦 `6/6`,生产 `/api/system` Router
兼容门 `2/2`Local Owner 全量 `187 total / 182 pass / 5 conditional skip / 0 fail`backend 全量
`1,525 total / 1,523 pass / 2 conditional skip / 0 fail`18-package clean build/逐包测试单次退出 0。package boundary、Service
Bridge import、Edge import、Cluster dependency、Cluster/Worker deployment、Console 与 Console distribution 八项审计全部
compatible/passedworkspace 保持 18 packages、`singleSourcePackages=[]``shallowSourcePackages=[]`Local Owner 为
`113 source / 112 nested / 1 root binary entry`。14 档 Local artifact audit 全部 compatible;基础 Edge/Standalone 保持
`2,598,669 / 2,598,747` bytes、316 files、57 loaded modulesAdopted 为 `2,817,964 / 2,818,087` bytes、58 loaded modules
Application+AI 为 `4,501,822 / 4,501,954` bytesMCP 为 `7,324,601 / 7,324,709` bytes、227 loaded modules,证明 ceremony
未进入低配运行闭包。systemd Docker live gate 已覆盖 root/non-root success,真实 `/api/system` 推进到 `legacy_ready` 并验证收据/head
exact replaybarrier-crash 保持 `manual_required` 且不执行 readiness。OpenRC 因 `node:24-alpine` 拉取卡在 Docker credential
helper 而未执行,不能宣称全组合门闭合;挂起拉取已终止。本切片不改变 PostgreSQL schema、ACL、repository、role、Pool、连接或
failover 语义,因此不重跑且不重新占有 HA 证明。D-382 应扩展 System/Script/Open API/鉴权错误 envelope 兼容矩阵与真实升级/回滚
rehearsalOpenRC 门在镜像基础设施恢复后补跑。
- D-380/ADR-0473(已接受;OpenRC live actor 待镜像基础设施恢复后补跑):完成 systemd/OpenRC service-manager rollback 的
安全 commit 协议。`rollback_prepared` 不再被误当作 root 授权;Owner 重新绑定当前 head、preparation、Application 与
legacy-silence commitment 原始摘要、descriptor 摘要后,才 no-replace 发布 `legacy_restart_requested`。短生命周期 root bridge
@@ -0,0 +1,119 @@
# ADR-0474:有界 Legacy Core Readiness Proof
- 状态:Accepted
- 日期:2026-08-20
- 关联 RFCQL-RFC-0001 D-64、D-274、D-275、D-380、D-381
- 关联 ADRADR-0315、ADR-0471、ADR-0472、ADR-0473
## 背景
ADR-0473 已把 adopted systemd/OpenRC 回滚安全地收敛到 `legacy_running`,但该状态只证明 init/process running、
target stopped 以及进程身份绑定。它不能证明 QingLong 2.x 已完成数据库初始化并能通过 HTTP 服务核心请求。把
`legacy_running` 直接当作回滚完成,会在服务仍启动中、初始化失败或错误版本占用端口时产生假成功。
readiness 证明同时必须适配低配路由设备与集群节点:它不能引入常驻 watcher、连接池、队列或新部署对象,也不能把
调用方提供的 URL、header、代理或重定向变成新的网络能力。编辑前 GitNexus 对 instance lineage 返回 MEDIUM11 个累计影响、
9 个直接 importer、0 条已识别 execution flowLocal Owner CLI、live actor、Docker gate 和 package boundary 返回 LOW,未发现
HIGH/CRITICAL 风险。
## 决策
### 1. readiness 是显式、一次性 Owner ceremony
新增私有命令文件操作:
```text
ql3-local-deploy cutover-legacy-readiness-probe --command-file <private.json>
```
命令必须绑定 exact cutover/profile/instance/generation、activation digest、当前 `legacy_running` head digest、产生该 head 的
source record digest、legacy HTTP port、预期精确 2.x version 和请求时间。Owner 在任何网络请求前完成 instance head
compare-and-swap 复验;stale、漂移或非 `legacy_running|legacy_ready` 状态失败关闭。
该 ceremony 复用于 Docker、systemd 与 OpenRC,因为 readiness 所证明的是同一个 Owner instance lineage,而不是某个 init
manager 的私有事实。它仍是显式短生命周期进程,不自动挂接到 root bridge,也不在后台周期执行。
### 2. 网络能力固定且有界
生产探针只允许:
- `GET http://127.0.0.1:<bound-port>/api/system`
- `Accept: application/json``Connection: close`
- 每次请求最多 2 秒、响应最多 32 KiB;
- Edge 总预算 30 秒、最多 60 次,Standalone 总预算 60 秒、最多 120 次,固定 500 ms 间隔;
- 不使用 keep-alive agent、不跟随 redirect、不接受 caller URL/path/header/credential/proxy。
这些边界使失败成本由 Profile 固定,内存消耗为常数;集群节点不会因此获得新的外部网络、Worker 或控制面权限,低配设备也
不会新增常驻 RSS。
### 3. 只证明 2.x core readiness
成功必须同时满足 HTTP 200、现行 JSON envelope `code=200``data.isInitialized=true`,以及 `data.version` 与命令绑定的
精确 2.x version 相同。生产 Router 兼容测试直接运行 `/api/system`,证明当前 2.x 在已初始化与默认未初始化账户下保持该契约。
该结果称为 **core readiness**:它证明正确版本的本机 2.x HTTP core 已响应并完成初始化。它不证明任务执行、订阅、外部 provider、
通知、脚本依赖、Cluster Worker 或全部业务 API 健康,因此不得命名为 full health。
### 4. 成功持久化,失败不推进 lineage
成功后 Owner 以 no-replace、`0600` 发布 `legacy-readiness-gN.json`,收据绑定 prior head、legacy-running source record、endpoint、
预期/观察 version、attempts 与观察时间,再把 instance head 唯一合法地从 `legacy_running` CAS 到 `legacy_ready`
相同命令的 exact replay 读取并复验收据,返回相同 `legacy_ready` 事实且发起零次网络请求。`unavailable``http_rejected`
`response_too_large``response_invalid``not_initialized``version_mismatch` 都返回 `not_ready`,保持 head 为
`legacy_running`,不写失败收据,也不自动重启任何服务。
### 5. 保持 package 与部署边界
实现内聚在现有 `@qinglong/local-owner-cli`
```text
deployment/cutover/legacy-readiness/
contract.ts
probe.ts
```
不新增 workspace package、production dependency、binary、daemon、listener、timer、watcher、数据库连接、systemd/OpenRC unit、
容器或 Kubernetes workload。用于 Docker live gate 的 HTTP 服务只是测试 fixture,不进入产品制品。
## 被否决方案
1. **把 `legacy_running` 当作健康**:进程存在不能证明 HTTP 与数据库初始化,拒绝。
2. **探测 `/api/health`**:该端点的进程局部 gRPC 观察在多节点/Worker 语义下容易过度声明,不适合作为 2.x core 回滚证明。
3. **接受 caller URL、header 或 `curl` 参数**:会扩大 SSRF、凭据与代理面,且无法形成可重放的固定契约,拒绝。
4. **在 root bridge 内自动探测**:会混合 OS mutation 与 Owner lineage authority,并延长 root ceremony,拒绝。
5. **后台 watcher 或无限重试**:会给路由设备增加常驻成本,也让失败无法有界,拒绝。
6. **新建 readiness package**:两个同生命周期源码模块没有独立交付和依赖理由,拒绝。
## 升级与回退
`legacy_ready` 是 additive lineage state;新版本只在 exact `legacy_running` 后写入。升级前应记录实际 2.x version 与 port,并以
Owner 私有命令文件传入。若回退到不认识 `legacy_ready` 的旧 3.0 孵化构建,应保留收据和 instance journal、停止自动 ceremony
由 operator 审核后从受支持版本恢复;不得手工改写 head 伪造旧状态。
## 验收证据
- readiness 聚焦门 `6/6`,覆盖闭合 contract、成功/零网络重放、Edge 有界失败、version mismatch/stale head、真实固定回环和
oversized/redirect 拒绝;生产 `/api/system` Router 兼容门 `2/2`
- Local Owner 全量 `187 total / 182 pass / 5 conditional skip / 0 fail`backend 全量
`1,525 total / 1,523 pass / 2 conditional skip / 0 fail`
- 18-package clean build 与逐包测试单次退出 0。package boundary、Service Bridge import、Edge import、Cluster dependency、
Cluster/Worker deployment、Console 与 Console distribution 八项审计全部 compatible/passedworkspace 仍为 18 packages、
`singleSourcePackages=[]``shallowSourcePackages=[]`Local Owner 为 `113 source / 112 nested / 1 root binary entry`
- 14 档 Local artifact audit 全部 compatible。基础 Edge/Standalone 保持 `2,598,669 / 2,598,747` bytes、316 files、
57 loaded modulesAdopted 为 `2,817,964 / 2,818,087` bytes、58 loaded modulesApplication+AI 为
`4,501,822 / 4,501,954` bytesMCP 为 `7,324,601 / 7,324,709` bytes、227 loaded modules。readiness ceremony
没有进入基础运行闭包。
- systemd Docker live gate 覆盖 root/non-root success,真实 `/api/system` 使 Owner 进入 `legacy_ready` 并完成零网络 exact replay
root barrier-crash 保持 `manual_required`,不会错误执行 readiness。
- OpenRC actor 因 `node:24-alpine` 拉取卡在本机 Docker credential helper 而未执行,所以本 ADR 不宣称 systemd/OpenRC 全组合门闭合。
- 本阶段不修改 SQL、migration、PostgreSQL ACL/repository/role/Pool、连接或 failover 语义,因此不重跑且不重新占有 PostgreSQL HA 证明。
## 未完成
- 镜像基础设施恢复后的 OpenRC root/non-root success 与 barrier-crash live actor
- 固定物理 Edge 的完整 rollback/readiness 证据;
- System、Script、Open API、鉴权与错误 envelope 的更完整 2.x 兼容矩阵;
- 真实 2.x 数据目录升级、Primary 双态和目标实例 rollback rehearsal。
本 ADR 关闭 `legacy_running` 到有界 `legacy_ready` 的 core readiness 证明,不代表 QingLong 3.0 升级/回退 Gate 已全部完成。
+1
View File
@@ -477,6 +477,7 @@
| [ADR-0471](./ADR-0471-legacy-core-api-compatibility-baseline.md) | Legacy 核心执行 API 兼容基线 | Accepted |
| [ADR-0472](./ADR-0472-service-manager-legacy-rollback-preparation.md) | Service Manager Legacy Rollback Preparation | Accepted |
| [ADR-0473](./ADR-0473-service-manager-legacy-rollback-commit.md) | Service Manager Legacy Rollback Commit | AcceptedOpenRC live actor 待补) |
| [ADR-0474](./ADR-0474-bounded-legacy-core-readiness-proof.md) | 有界 Legacy Core Readiness Proof | AcceptedOpenRC live actor 待补) |
## 规则
+46
View File
@@ -454,6 +454,52 @@ barrier 后崩溃时原样重放 commit。重放只 inspect,绝不盲目重复
补写 exact outcome;若无法证明,则安全收敛为 `manual_required`。整个流程不删除/覆盖 target,不把 target
数据库写回 source。`reconciliation_required``manual_review` 必须停止在 2.5,等待独立数据恢复流程。
### 2.7 Legacy core readiness proof
`legacy_running` 只证明 legacy init/process running 与 target stopped。要把回滚实例推进为 core ready,必须创建独立的私有
`0600` command file;不要把探针塞进 root bridge,也不要手工用可变 URL 的 `curl` 结果替代:
```json
{
"schemaVersion": 1,
"operation": "local.deployment.cutover.legacy-readiness-probe",
"options": {
"deploymentRoot": "/opt/qinglong3",
"allowRootService": false
},
"request": {
"cutoverId": "cutover-20260820-01",
"profile": "edge",
"instanceId": "local-default",
"generation": 1,
"expectedActivationDigest": "REPLACE_WITH_64_HEX_ACTIVATION_DIGEST",
"expectedInstanceHeadDigest": "REPLACE_WITH_LEGACY_RUNNING_HEAD_DIGEST",
"expectedLegacyRunningRecordDigest": "REPLACE_WITH_LEGACY_RUNNING_SOURCE_RECORD_DIGEST",
"legacyHttpPort": 5700,
"expectedLegacyVersion": "2.21.0",
"requestedAtMs": 1787236200000
}
}
```
```sh
ql3-local-deploy cutover-legacy-readiness-probe \
--command-file /secure/operator/qinglong3-legacy-readiness.json
```
`expectedInstanceHeadDigest` 取 2.6 Owner consume 返回的 `legacy_running` head
`expectedLegacyRunningRecordDigest` 取该 head 绑定的 source record digest。Docker 与 systemd/OpenRC 使用相同命令,因为它只操作
共同的 Owner instance lineage。
探针固定请求 `GET http://127.0.0.1:<legacyHttpPort>/api/system`,不接受其他 host/path/header/credential,不跟随重定向;
每次请求最多 2 秒、响应最多 32 KiB。Edge 总预算 30 秒/最多 60 次,Standalone 总预算 60 秒/最多 120 次。
只有 HTTP/envelope 成功、`isInitialized=true` 且 version 精确等于命令中的 2.x version 才会发布
`legacy-readiness-gN.json` 并 CAS 到 `legacy_ready`
`not_ready` 会给出 `unavailable|http_rejected|response_too_large|response_invalid|not_initialized|version_mismatch`,保持
`legacy_running` 不变;处理原因后必须原样重放,不要换 cutover/generation 绕过证据链。成功命令的 exact replay 只读取既有收据,
不会再次发网络请求。`legacy_ready` 是 2.x HTTP core 与初始化证明,不代表任务、订阅、外部 provider、通知或 Cluster Worker 全健康。
## 3. systemd
命令中使用 `"kind": "systemd"`。成功后检查:
@@ -26,6 +26,7 @@ export type LocalCutoverInstanceHeadState =
| 'rollback_prepared'
| 'legacy_restart_requested'
| 'legacy_running'
| 'legacy_ready'
| 'manual_required'
| 'resolution_authorized';
@@ -156,6 +157,7 @@ function parseHead(value: unknown): Readonly<LocalCutoverInstanceHead> {
head.state !== 'rollback_prepared' &&
head.state !== 'legacy_restart_requested' &&
head.state !== 'legacy_running' &&
head.state !== 'legacy_ready' &&
head.state !== 'manual_required' &&
head.state !== 'resolution_authorized') ||
!Number.isSafeInteger(head.generation) ||
@@ -321,6 +323,7 @@ export function advanceLocalCutoverInstanceHead(
| 'rollback_prepared'
| 'legacy_restart_requested'
| 'legacy_running'
| 'legacy_ready'
| 'manual_required',
generation: number,
sourceRecordDigest: string,
@@ -356,7 +359,8 @@ export function advanceLocalCutoverInstanceHead(
current.generation === generation &&
(current.state === 'rollback_prepared' ||
current.state === 'legacy_restart_requested' ||
current.state === 'legacy_running')
current.state === 'legacy_running' ||
current.state === 'legacy_ready')
) {
return current;
}
@@ -374,6 +378,7 @@ export function advanceLocalCutoverInstanceHead(
current.state === 'rollback_prepared') ||
(state === 'legacy_running' &&
current.state === 'legacy_restart_requested') ||
(state === 'legacy_ready' && current.state === 'legacy_running') ||
(state === 'manual_required' &&
(current.state === 'legacy_stopped' ||
current.state === 'target_active' ||
@@ -0,0 +1,198 @@
import path from 'node:path';
import {
currentIdentity,
LocalDeploymentConfigurationError,
type LocalDeploymentProfile,
} from '../../foundation/contract';
const MAX_PATH_BYTES = 4_096;
const SAFE_PATH_PATTERN = /^\/[A-Za-z0-9._/@-]+$/;
const INSTANCE_ID_PATTERN = /^[a-z0-9][a-z0-9._-]{0,127}$/;
const CUTOVER_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const DIGEST_PATTERN = /^[0-9a-f]{64}$/;
const LEGACY_VERSION_PATTERN =
/^2\.[0-9]+\.[0-9]+(?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?$/;
export interface LocalDeploymentLegacyReadinessCommand {
readonly schemaVersion: 1;
readonly operation: 'local.deployment.cutover.legacy-readiness-probe';
readonly options: Readonly<{
deploymentRoot: string;
allowRootService: boolean;
}>;
readonly request: Readonly<{
cutoverId: string;
profile: LocalDeploymentProfile;
instanceId: string;
generation: number;
expectedActivationDigest: string;
expectedInstanceHeadDigest: string;
expectedLegacyRunningRecordDigest: string;
legacyHttpPort: number;
expectedLegacyVersion: string;
requestedAtMs: number;
}>;
}
function object(value: unknown, label: string): Record<string, unknown> {
if (
!value ||
typeof value !== 'object' ||
Array.isArray(value) ||
(Object.getPrototypeOf(value) !== Object.prototype &&
Object.getPrototypeOf(value) !== null)
) {
throw new LocalDeploymentConfigurationError(`${label} must be an object`);
}
return value as Record<string, unknown>;
}
function exact(
value: Record<string, unknown>,
keys: readonly string[],
label: string,
): void {
const actual = Object.keys(value).sort();
const expected = [...keys].sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
throw new LocalDeploymentConfigurationError(`${label} shape is invalid`);
}
}
function safeAbsolutePath(value: unknown, label: string): string {
if (
typeof value !== 'string' ||
!path.isAbsolute(value) ||
path.normalize(value) !== value ||
path.parse(value).root === value ||
value.includes('\0') ||
value.includes('//') ||
!SAFE_PATH_PATTERN.test(value) ||
Buffer.byteLength(value, 'utf8') > MAX_PATH_BYTES
) {
throw new LocalDeploymentConfigurationError(
`${label} must be a supervisor-safe normalized absolute non-root path`,
);
}
return value;
}
function integer(
value: unknown,
minimum: number,
maximum: number,
label: string,
): number {
if (
!Number.isSafeInteger(value) ||
(value as number) < minimum ||
(value as number) > maximum
) {
throw new LocalDeploymentConfigurationError(`${label} is invalid`);
}
return value as number;
}
export function normalizeLocalDeploymentLegacyReadinessCommand(
value: unknown,
): Readonly<LocalDeploymentLegacyReadinessCommand> {
const command = object(value, 'command');
exact(
command,
['operation', 'options', 'request', 'schemaVersion'],
'command',
);
if (
command.schemaVersion !== 1 ||
command.operation !== 'local.deployment.cutover.legacy-readiness-probe'
) {
throw new LocalDeploymentConfigurationError(
'legacy readiness schemaVersion or operation is invalid',
);
}
const identity = currentIdentity();
const options = object(command.options, 'options');
exact(options, ['allowRootService', 'deploymentRoot'], 'options');
if (
typeof options.allowRootService !== 'boolean' ||
(identity.uid === 0) !== options.allowRootService
) {
throw new LocalDeploymentConfigurationError(
'allowRootService does not match the current identity',
);
}
const request = object(command.request, 'request');
exact(
request,
[
'cutoverId',
'expectedActivationDigest',
'expectedInstanceHeadDigest',
'expectedLegacyRunningRecordDigest',
'expectedLegacyVersion',
'generation',
'instanceId',
'legacyHttpPort',
'profile',
'requestedAtMs',
],
'request',
);
if (
typeof request.cutoverId !== 'string' ||
!CUTOVER_ID_PATTERN.test(request.cutoverId) ||
(request.profile !== 'edge' && request.profile !== 'standalone') ||
typeof request.instanceId !== 'string' ||
!INSTANCE_ID_PATTERN.test(request.instanceId) ||
typeof request.expectedActivationDigest !== 'string' ||
!DIGEST_PATTERN.test(request.expectedActivationDigest) ||
typeof request.expectedInstanceHeadDigest !== 'string' ||
!DIGEST_PATTERN.test(request.expectedInstanceHeadDigest) ||
typeof request.expectedLegacyRunningRecordDigest !== 'string' ||
!DIGEST_PATTERN.test(request.expectedLegacyRunningRecordDigest) ||
typeof request.expectedLegacyVersion !== 'string' ||
!LEGACY_VERSION_PATTERN.test(request.expectedLegacyVersion)
) {
throw new LocalDeploymentConfigurationError(
'legacy readiness request identity is invalid',
);
}
return Object.freeze({
schemaVersion: 1 as const,
operation: 'local.deployment.cutover.legacy-readiness-probe' as const,
options: Object.freeze({
deploymentRoot: safeAbsolutePath(
options.deploymentRoot,
'deploymentRoot',
),
allowRootService: options.allowRootService,
}),
request: Object.freeze({
cutoverId: request.cutoverId,
profile: request.profile,
instanceId: request.instanceId,
generation: integer(request.generation, 1, 15, 'generation'),
expectedActivationDigest: request.expectedActivationDigest,
expectedInstanceHeadDigest: request.expectedInstanceHeadDigest,
expectedLegacyRunningRecordDigest:
request.expectedLegacyRunningRecordDigest,
legacyHttpPort: integer(
request.legacyHttpPort,
1,
65_535,
'legacyHttpPort',
),
expectedLegacyVersion: request.expectedLegacyVersion,
requestedAtMs: integer(
request.requestedAtMs,
0,
Number.MAX_SAFE_INTEGER,
'requestedAtMs',
),
}),
});
}
@@ -0,0 +1,538 @@
import fs from 'node:fs';
import http from 'node:http';
import path from 'node:path';
import { readPrivateLocalCommandFile } from '@qinglong/local-command-file';
import {
currentIdentity,
LocalDeploymentConfigurationError,
} from '../../foundation/contract';
import {
preflightPublishedFile,
publishExactFile,
} from '../../foundation/files';
import {
advanceLocalCutoverInstanceHead,
localCutoverInstanceDirectory,
readLocalCutoverInstanceHead,
type LocalCutoverInstanceHead,
} from '../instanceLineage';
import { cutoverDigest } from '../targetEvidence';
import {
normalizeLocalDeploymentLegacyReadinessCommand,
type LocalDeploymentLegacyReadinessCommand,
} from './contract';
const RECEIPT_SCHEMA = 'qinglong3-local-legacy-readiness-receipt';
const LOOPBACK_HOST = '127.0.0.1';
const SYSTEM_PATH = '/api/system';
const MAX_RESPONSE_BYTES = 32 * 1024;
const DIGEST_PATTERN = /^[0-9a-f]{64}$/;
export type LocalLegacyReadinessReason =
| 'unavailable'
| 'http_rejected'
| 'response_too_large'
| 'response_invalid'
| 'not_initialized'
| 'version_mismatch';
export type LocalLegacyReadinessObservation =
| Readonly<{
ready: true;
initialized: true;
version: string;
}>
| Readonly<{
ready: false;
reason: LocalLegacyReadinessReason;
}>;
export interface LocalLegacyReadinessProbeInput {
readonly host: typeof LOOPBACK_HOST;
readonly port: number;
readonly path: typeof SYSTEM_PATH;
readonly timeoutMs: number;
readonly maxResponseBytes: number;
}
export interface LocalDeploymentLegacyReadinessDependencies {
readonly probe?: (
input: Readonly<LocalLegacyReadinessProbeInput>,
) => Promise<Readonly<LocalLegacyReadinessObservation>>;
readonly now?: () => number;
readonly wait?: (milliseconds: number) => Promise<void>;
}
interface LocalLegacyReadinessReceipt {
readonly schema: typeof RECEIPT_SCHEMA;
readonly schemaVersion: 1;
readonly state: 'legacy_ready';
readonly cutoverId: string;
readonly profile: 'edge' | 'standalone';
readonly instanceId: string;
readonly generation: number;
readonly activationDigest: string;
readonly previousHeadDigest: string;
readonly legacyRunningRecordDigest: string;
readonly endpoint: Readonly<{
host: typeof LOOPBACK_HOST;
port: number;
path: typeof SYSTEM_PATH;
}>;
readonly expectedVersion: string;
readonly observedVersion: string;
readonly initialized: true;
readonly attempts: number;
readonly observedAtMs: number;
readonly receiptDigest: string;
}
export type LocalDeploymentLegacyReadinessResult =
| Readonly<{
schemaVersion: 1;
operation: LocalDeploymentLegacyReadinessCommand['operation'];
status: 'prepared' | 'existing';
state: 'legacy_ready';
cutoverId: string;
generation: number;
attempts: number;
receiptDigest: string;
instanceHeadDigest: string;
}>
| Readonly<{
schemaVersion: 1;
operation: LocalDeploymentLegacyReadinessCommand['operation'];
status: 'not_ready';
state: 'legacy_running';
reason: LocalLegacyReadinessReason;
cutoverId: string;
generation: number;
attempts: number;
instanceHeadDigest: string;
}>;
function configurationError(message: string, cause?: unknown): never {
throw new LocalDeploymentConfigurationError(message, { cause });
}
function object(value: unknown, label: string): Record<string, unknown> {
if (
!value ||
typeof value !== 'object' ||
Array.isArray(value) ||
(Object.getPrototypeOf(value) !== Object.prototype &&
Object.getPrototypeOf(value) !== null)
) {
configurationError(`${label} must be an object`);
}
return value as Record<string, unknown>;
}
function exact(
value: Record<string, unknown>,
keys: readonly string[],
label: string,
): void {
const actual = Object.keys(value).sort();
const expected = [...keys].sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
configurationError(`${label} shape is invalid`);
}
}
function rejected(reason: LocalLegacyReadinessReason) {
return Object.freeze({ ready: false as const, reason });
}
export function probeLegacySystemEndpoint(
input: Readonly<LocalLegacyReadinessProbeInput>,
): Promise<Readonly<LocalLegacyReadinessObservation>> {
return new Promise((resolve) => {
let settled = false;
const finish = (observation: Readonly<LocalLegacyReadinessObservation>) => {
if (settled) return;
settled = true;
resolve(observation);
};
const request = http.request(
{
agent: false,
host: input.host,
port: input.port,
path: input.path,
method: 'GET',
headers: Object.freeze({
accept: 'application/json',
connection: 'close',
}),
},
(response) => {
if (response.statusCode !== 200) {
response.resume();
finish(rejected('http_rejected'));
return;
}
const chunks: Buffer[] = [];
let size = 0;
response.on('data', (chunk: Buffer | string) => {
const bytes = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
size += bytes.byteLength;
if (size > input.maxResponseBytes) {
finish(rejected('response_too_large'));
response.destroy();
return;
}
chunks.push(bytes);
});
response.on('end', () => {
if (settled) return;
try {
const envelope = object(
JSON.parse(Buffer.concat(chunks).toString('utf8')),
'legacy system response',
);
const data = object(envelope.data, 'legacy system response data');
if (envelope.code !== 200 || typeof data.version !== 'string') {
finish(rejected('response_invalid'));
return;
}
if (data.isInitialized !== true) {
finish(rejected('not_initialized'));
return;
}
finish(
Object.freeze({
ready: true as const,
initialized: true as const,
version: data.version,
}),
);
} catch {
finish(rejected('response_invalid'));
}
});
response.on('error', () => finish(rejected('unavailable')));
},
);
request.setTimeout(input.timeoutMs, () => request.destroy());
request.on('error', () => finish(rejected('unavailable')));
request.end();
});
}
function receiptPath(
command: Readonly<LocalDeploymentLegacyReadinessCommand>,
): string {
return path.join(
localCutoverInstanceDirectory(
command.options.deploymentRoot,
command.request.instanceId,
),
`legacy-readiness-g${command.request.generation}.json`,
);
}
function receiptContents(
receipt: Readonly<LocalLegacyReadinessReceipt>,
): string {
return `${JSON.stringify(receipt, null, 2)}\n`;
}
function parseReceipt(value: unknown): Readonly<LocalLegacyReadinessReceipt> {
const receipt = object(value, 'legacy readiness receipt');
exact(
receipt,
[
'activationDigest',
'attempts',
'cutoverId',
'endpoint',
'expectedVersion',
'generation',
'initialized',
'instanceId',
'legacyRunningRecordDigest',
'observedAtMs',
'observedVersion',
'previousHeadDigest',
'profile',
'receiptDigest',
'schema',
'schemaVersion',
'state',
],
'legacy readiness receipt',
);
const endpoint = object(receipt.endpoint, 'legacy readiness endpoint');
exact(endpoint, ['host', 'path', 'port'], 'legacy readiness endpoint');
const { receiptDigest, ...payload } = receipt;
if (
receipt.schema !== RECEIPT_SCHEMA ||
receipt.schemaVersion !== 1 ||
receipt.state !== 'legacy_ready' ||
typeof receipt.cutoverId !== 'string' ||
(receipt.profile !== 'edge' && receipt.profile !== 'standalone') ||
typeof receipt.instanceId !== 'string' ||
!Number.isSafeInteger(receipt.generation) ||
(receipt.generation as number) < 1 ||
typeof receipt.activationDigest !== 'string' ||
!DIGEST_PATTERN.test(receipt.activationDigest) ||
typeof receipt.previousHeadDigest !== 'string' ||
!DIGEST_PATTERN.test(receipt.previousHeadDigest) ||
typeof receipt.legacyRunningRecordDigest !== 'string' ||
!DIGEST_PATTERN.test(receipt.legacyRunningRecordDigest) ||
endpoint.host !== LOOPBACK_HOST ||
endpoint.path !== SYSTEM_PATH ||
!Number.isSafeInteger(endpoint.port) ||
(endpoint.port as number) < 1 ||
(endpoint.port as number) > 65_535 ||
typeof receipt.expectedVersion !== 'string' ||
typeof receipt.observedVersion !== 'string' ||
receipt.initialized !== true ||
!Number.isSafeInteger(receipt.attempts) ||
(receipt.attempts as number) < 1 ||
!Number.isSafeInteger(receipt.observedAtMs) ||
(receipt.observedAtMs as number) < 0 ||
typeof receiptDigest !== 'string' ||
!DIGEST_PATTERN.test(receiptDigest) ||
cutoverDigest(payload) !== receiptDigest
) {
configurationError('legacy readiness receipt drifted');
}
return receipt as unknown as Readonly<LocalLegacyReadinessReceipt>;
}
function verifyReceipt(
command: Readonly<LocalDeploymentLegacyReadinessCommand>,
receipt: Readonly<LocalLegacyReadinessReceipt>,
): void {
if (
receipt.cutoverId !== command.request.cutoverId ||
receipt.profile !== command.request.profile ||
receipt.instanceId !== command.request.instanceId ||
receipt.generation !== command.request.generation ||
receipt.activationDigest !== command.request.expectedActivationDigest ||
receipt.previousHeadDigest !== command.request.expectedInstanceHeadDigest ||
receipt.legacyRunningRecordDigest !==
command.request.expectedLegacyRunningRecordDigest ||
receipt.endpoint.port !== command.request.legacyHttpPort ||
receipt.expectedVersion !== command.request.expectedLegacyVersion ||
receipt.observedVersion !== command.request.expectedLegacyVersion
) {
configurationError('legacy readiness receipt does not match the command');
}
}
function verifyHead(
command: Readonly<LocalDeploymentLegacyReadinessCommand>,
head: Readonly<LocalCutoverInstanceHead>,
): void {
const replay = head.state === 'legacy_ready';
if (
head.profile !== command.request.profile ||
head.cutoverId !== command.request.cutoverId ||
head.instanceId !== command.request.instanceId ||
head.generation !== command.request.generation ||
head.activationDigest !== command.request.expectedActivationDigest ||
(replay
? head.previousHeadDigest !== command.request.expectedInstanceHeadDigest
: head.headDigest !== command.request.expectedInstanceHeadDigest) ||
(!replay &&
head.sourceRecordDigest !==
command.request.expectedLegacyRunningRecordDigest) ||
(head.state !== 'legacy_running' && head.state !== 'legacy_ready')
) {
configurationError(
'legacy readiness command lost the instance head compare-and-swap',
);
}
}
function successfulResult(
command: Readonly<LocalDeploymentLegacyReadinessCommand>,
status: 'prepared' | 'existing',
receipt: Readonly<LocalLegacyReadinessReceipt>,
head: Readonly<LocalCutoverInstanceHead>,
): Readonly<LocalDeploymentLegacyReadinessResult> {
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
status,
state: 'legacy_ready' as const,
cutoverId: command.request.cutoverId,
generation: command.request.generation,
attempts: receipt.attempts,
receiptDigest: receipt.receiptDigest,
instanceHeadDigest: head.headDigest,
});
}
function completeExistingReceipt(
command: Readonly<LocalDeploymentLegacyReadinessCommand>,
uid: number,
current: Readonly<LocalCutoverInstanceHead>,
): Readonly<LocalDeploymentLegacyReadinessResult> | undefined {
const filePath = receiptPath(command);
if (!fs.existsSync(filePath)) return undefined;
const receipt = parseReceipt(readPrivateLocalCommandFile(filePath));
verifyReceipt(command, receipt);
const head =
current.state === 'legacy_ready'
? current
: advanceLocalCutoverInstanceHead(
command,
uid,
'legacy_ready',
command.request.generation,
receipt.receiptDigest,
);
if (head.sourceRecordDigest !== receipt.receiptDigest) {
configurationError('legacy-ready instance head is not receipt-bound');
}
return successfulResult(command, 'existing', receipt, head);
}
function policy(profile: 'edge' | 'standalone') {
return profile === 'edge'
? Object.freeze({
totalTimeoutMs: 30_000,
requestTimeoutMs: 2_000,
pollIntervalMs: 500,
maximumAttempts: 60,
})
: Object.freeze({
totalTimeoutMs: 60_000,
requestTimeoutMs: 2_000,
pollIntervalMs: 500,
maximumAttempts: 120,
});
}
export async function proveLocalDeploymentLegacyReadiness(
input: unknown,
dependencies: Readonly<LocalDeploymentLegacyReadinessDependencies> = {},
): Promise<Readonly<LocalDeploymentLegacyReadinessResult>> {
const command = normalizeLocalDeploymentLegacyReadinessCommand(input);
const identity = currentIdentity();
const current = readLocalCutoverInstanceHead(
command.options.deploymentRoot,
command.request.instanceId,
identity.uid,
);
verifyHead(command, current);
const existing = completeExistingReceipt(command, identity.uid, current);
if (existing) return existing;
if (current.state !== 'legacy_running') {
configurationError('legacy-ready instance is missing its receipt');
}
const observe = dependencies.probe ?? probeLegacySystemEndpoint;
const now = dependencies.now ?? Date.now;
const wait =
dependencies.wait ??
((milliseconds: number) =>
new Promise<void>((resolve) => setTimeout(resolve, milliseconds)));
const limits = policy(command.request.profile);
const deadline = now() + limits.totalTimeoutMs;
let attempts = 0;
let lastReason: LocalLegacyReadinessReason = 'unavailable';
while (attempts < limits.maximumAttempts && now() <= deadline) {
attempts += 1;
const remaining = Math.max(1, deadline - now());
const observation = await observe(
Object.freeze({
host: LOOPBACK_HOST,
port: command.request.legacyHttpPort,
path: SYSTEM_PATH,
timeoutMs: Math.min(limits.requestTimeoutMs, remaining),
maxResponseBytes: MAX_RESPONSE_BYTES,
}),
);
if (observation.ready === true) {
if (observation.version !== command.request.expectedLegacyVersion) {
lastReason = 'version_mismatch';
break;
}
const payload = Object.freeze({
schema: RECEIPT_SCHEMA,
schemaVersion: 1 as const,
state: 'legacy_ready' as const,
cutoverId: command.request.cutoverId,
profile: command.request.profile,
instanceId: command.request.instanceId,
generation: command.request.generation,
activationDigest: command.request.expectedActivationDigest,
previousHeadDigest: command.request.expectedInstanceHeadDigest,
legacyRunningRecordDigest:
command.request.expectedLegacyRunningRecordDigest,
endpoint: Object.freeze({
host: LOOPBACK_HOST,
port: command.request.legacyHttpPort,
path: SYSTEM_PATH,
}),
expectedVersion: command.request.expectedLegacyVersion,
observedVersion: observation.version,
initialized: true as const,
attempts,
observedAtMs: now(),
});
const receipt = Object.freeze({
...payload,
receiptDigest: cutoverDigest(payload),
});
const filePath = receiptPath(command);
const contents = receiptContents(receipt);
preflightPublishedFile(
filePath,
contents,
0o600,
identity.uid,
'legacy readiness receipt',
);
const status = publishExactFile(
filePath,
contents,
0o600,
identity.uid,
'legacy readiness receipt',
);
const head = advanceLocalCutoverInstanceHead(
command,
identity.uid,
'legacy_ready',
command.request.generation,
receipt.receiptDigest,
);
return successfulResult(command, status, receipt, head);
}
lastReason = observation.reason;
if (attempts >= limits.maximumAttempts || now() >= deadline) break;
await wait(Math.min(limits.pollIntervalMs, deadline - now()));
}
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
status: 'not_ready' as const,
state: 'legacy_running' as const,
reason: lastReason,
cutoverId: command.request.cutoverId,
generation: command.request.generation,
attempts,
instanceHeadDigest: current.headDigest,
});
}
export function proveLocalDeploymentLegacyReadinessCommandFile(
filePath: string,
dependencies: Readonly<LocalDeploymentLegacyReadinessDependencies> = {},
): Promise<Readonly<LocalDeploymentLegacyReadinessResult>> {
return proveLocalDeploymentLegacyReadiness(
readPrivateLocalCommandFile(filePath),
dependencies,
);
}
@@ -69,6 +69,10 @@ import {
runLocalDeploymentLegacyRollback,
runLocalDeploymentLegacyRollbackCommandFile,
} from './cutover/legacyRollback';
import {
proveLocalDeploymentLegacyReadiness,
proveLocalDeploymentLegacyReadinessCommandFile,
} from './cutover/legacy-readiness/probe';
import {
consumeLocalServiceManagerOutcome,
consumeLocalServiceManagerOutcomeCommandFile,
@@ -148,6 +152,17 @@ export {
type LocalDeploymentLegacyRollbackResult,
} from './cutover/legacyRollbackContract';
export { type LocalDeploymentLegacyRollbackDependencies } from './cutover/legacyRollback';
export {
normalizeLocalDeploymentLegacyReadinessCommand,
type LocalDeploymentLegacyReadinessCommand,
} from './cutover/legacy-readiness/contract';
export {
type LocalDeploymentLegacyReadinessDependencies,
type LocalDeploymentLegacyReadinessResult,
type LocalLegacyReadinessObservation,
type LocalLegacyReadinessProbeInput,
type LocalLegacyReadinessReason,
} from './cutover/legacy-readiness/probe';
export {
EMPTY_RESOLUTION_DIGEST,
normalizeLocalDeploymentCutoverManualCommand,
@@ -177,6 +192,8 @@ export {
runLocalDeploymentCutoverManualCommandFile,
runLocalDeploymentLegacyRollback,
runLocalDeploymentLegacyRollbackCommandFile,
proveLocalDeploymentLegacyReadiness,
proveLocalDeploymentLegacyReadinessCommandFile,
runLocalDeploymentDockerTarget,
runLocalDeploymentDockerTargetCommandFile,
switchLocalDeploymentComposeRevision,
@@ -14,6 +14,7 @@ import {
prepareLocalServiceManagerIntentCommandFile,
prepareLocalServiceManagerLegacyRollbackCommandFile,
prepareLocalDeploymentCommandFile,
proveLocalDeploymentLegacyReadinessCommandFile,
restoreLocalDeploymentComposeCommitCommandFile,
restoreLocalDeploymentComposePrepareCommandFile,
runLocalDeploymentCutoverManualCommandFile,
@@ -25,7 +26,7 @@ import {
} from './localDeployment';
const USAGE =
'Usage: ql3-local-deploy <prepare|status|service-intent-prepare|service-outcome-consume|service-cutover-consume|service-legacy-rollback-prepare|service-legacy-rollback-authorize|service-legacy-rollback-consume|cutover-legacy-stop|cutover-target-start|cutover-target-restart|cutover-target-stop|cutover-legacy-rollback-prepare|cutover-legacy-rollback-commit|cutover-manual-diagnose|cutover-manual-resolution-prepare|cutover-manual-resolution-commit|compose-revision|compose-preflight|compose-apply|compose-restore-prepare|compose-restore-commit|compose-evidence-collect-prepare|compose-evidence-collect-commit> --command-file /absolute/private-command.json';
'Usage: ql3-local-deploy <prepare|status|service-intent-prepare|service-outcome-consume|service-cutover-consume|service-legacy-rollback-prepare|service-legacy-rollback-authorize|service-legacy-rollback-consume|cutover-legacy-stop|cutover-target-start|cutover-target-restart|cutover-target-stop|cutover-legacy-rollback-prepare|cutover-legacy-rollback-commit|cutover-legacy-readiness-probe|cutover-manual-diagnose|cutover-manual-resolution-prepare|cutover-manual-resolution-commit|compose-revision|compose-preflight|compose-apply|compose-restore-prepare|compose-restore-commit|compose-evidence-collect-prepare|compose-evidence-collect-commit> --command-file /absolute/private-command.json';
async function main(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
@@ -48,6 +49,7 @@ async function main(argv: readonly string[]): Promise<void> {
argv[0] !== 'cutover-target-stop' &&
argv[0] !== 'cutover-legacy-rollback-prepare' &&
argv[0] !== 'cutover-legacy-rollback-commit' &&
argv[0] !== 'cutover-legacy-readiness-probe' &&
argv[0] !== 'cutover-manual-diagnose' &&
argv[0] !== 'cutover-manual-resolution-prepare' &&
argv[0] !== 'cutover-manual-resolution-commit' &&
@@ -101,6 +103,8 @@ async function main(argv: readonly string[]): Promise<void> {
? 'local.deployment.cutover.legacy-rollback-prepare'
: 'local.deployment.cutover.legacy-rollback-commit',
)
: argv[0] === 'cutover-legacy-readiness-probe'
? proveLocalDeploymentLegacyReadinessCommandFile(argv[2]!)
: argv[0] === 'cutover-manual-diagnose' ||
argv[0] === 'cutover-manual-resolution-prepare' ||
argv[0] === 'cutover-manual-resolution-commit'
@@ -0,0 +1,321 @@
const assert = require('node:assert/strict');
const fs = require('node:fs');
const http = require('node:http');
const os = require('node:os');
const path = require('node:path');
const { test } = require('node:test');
const {
normalizeLocalDeploymentLegacyReadinessCommand,
proveLocalDeploymentLegacyReadiness,
} = require('../dist/deployment/localDeployment.js');
const {
probeLegacySystemEndpoint,
} = require('../dist/deployment/cutover/legacy-readiness/probe.js');
const {
advanceLocalCutoverInstanceHead,
claimLocalCutoverInstance,
readLocalCutoverInstanceHead,
} = require('../dist/deployment/cutover/instanceLineage.js');
function rootAcknowledgement() {
return typeof process.getuid === 'function' && process.getuid() === 0;
}
function fixture(t, profile = 'edge', instanceId = `${profile}-legacy-1`) {
const deploymentRoot = fs.realpathSync(
fs.mkdtempSync(path.join(os.tmpdir(), 'ql3-legacy-readiness-')),
);
fs.chmodSync(deploymentRoot, 0o700);
fs.mkdirSync(path.join(deploymentRoot, 'service'), { mode: 0o700 });
t.after(() => fs.rmSync(deploymentRoot, { recursive: true, force: true }));
const identity = {
options: { deploymentRoot },
request: {
cutoverId: `cutover-${instanceId}`,
profile,
instanceId,
expectedActivationDigest: 'a'.repeat(64),
requestedAtMs: 1_787_200_000_000,
},
};
const uid = process.getuid();
claimLocalCutoverInstance(identity, uid, '0'.repeat(64));
const transitions = [
['legacy_stopped', 0, '1'.repeat(64)],
['target_active', 1, '2'.repeat(64)],
['target_stopped', 1, '3'.repeat(64)],
['rollback_prepared', 1, '4'.repeat(64)],
['legacy_restart_requested', 1, '5'.repeat(64)],
['legacy_running', 1, '6'.repeat(64)],
];
let head;
for (const [state, generation, sourceDigest] of transitions) {
head = advanceLocalCutoverInstanceHead(
identity,
uid,
state,
generation,
sourceDigest,
);
}
const command = {
schemaVersion: 1,
operation: 'local.deployment.cutover.legacy-readiness-probe',
options: {
deploymentRoot,
allowRootService: rootAcknowledgement(),
},
request: {
cutoverId: identity.request.cutoverId,
profile,
instanceId,
generation: 1,
expectedActivationDigest: identity.request.expectedActivationDigest,
expectedInstanceHeadDigest: head.headDigest,
expectedLegacyRunningRecordDigest: '6'.repeat(64),
legacyHttpPort: 5700,
expectedLegacyVersion: '2.21.0',
requestedAtMs: 1_787_200_030_000,
},
};
return { command, deploymentRoot, head, uid };
}
test('normalizes a closed legacy readiness command', (t) => {
const state = fixture(t);
const normalized = normalizeLocalDeploymentLegacyReadinessCommand(
state.command,
);
assert.equal(normalized.request.legacyHttpPort, 5700);
assert.equal(normalized.request.expectedLegacyVersion, '2.21.0');
assert.throws(
() =>
normalizeLocalDeploymentLegacyReadinessCommand({
...state.command,
request: { ...state.command.request, endpoint: 'http://example.com' },
}),
/request shape is invalid/,
);
assert.throws(
() =>
normalizeLocalDeploymentLegacyReadinessCommand({
...state.command,
request: { ...state.command.request, expectedLegacyVersion: '3.0.0' },
}),
/request identity is invalid/,
);
assert.throws(
() =>
normalizeLocalDeploymentLegacyReadinessCommand({
...state.command,
request: { ...state.command.request, legacyHttpPort: 65_536 },
}),
/legacyHttpPort is invalid/,
);
});
test('persists a legacy-ready receipt and replays without network authority', async (t) => {
const state = fixture(t);
let attempts = 0;
let clock = state.command.request.requestedAtMs;
const result = await proveLocalDeploymentLegacyReadiness(state.command, {
now: () => clock,
wait: async (milliseconds) => {
clock += milliseconds;
},
async probe(input) {
attempts += 1;
assert.deepEqual(
{ host: input.host, port: input.port, path: input.path },
{ host: '127.0.0.1', port: 5700, path: '/api/system' },
);
return attempts < 3
? { ready: false, reason: 'not_initialized' }
: { ready: true, initialized: true, version: '2.21.0' };
},
});
assert.equal(result.status, 'prepared');
assert.equal(result.state, 'legacy_ready');
assert.equal(result.attempts, 3);
assert.match(result.receiptDigest, /^[0-9a-f]{64}$/);
const receiptPath = path.join(
state.deploymentRoot,
'service',
'cutover-instances',
state.command.request.instanceId,
'legacy-readiness-g1.json',
);
const receipt = JSON.parse(fs.readFileSync(receiptPath, 'utf8'));
assert.equal(receipt.state, 'legacy_ready');
assert.equal(receipt.endpoint.host, '127.0.0.1');
assert.equal(receipt.endpoint.path, '/api/system');
assert.equal(receipt.observedVersion, '2.21.0');
assert.equal(fs.statSync(receiptPath).mode & 0o777, 0o600);
const head = readLocalCutoverInstanceHead(
state.deploymentRoot,
state.command.request.instanceId,
state.uid,
);
assert.equal(head.state, 'legacy_ready');
assert.equal(head.previousHeadDigest, state.head.headDigest);
assert.equal(head.sourceRecordDigest, result.receiptDigest);
const replay = await proveLocalDeploymentLegacyReadiness(state.command, {
async probe() {
throw new Error('exact replay must not open loopback HTTP authority');
},
});
assert.equal(replay.status, 'existing');
assert.equal(replay.receiptDigest, result.receiptDigest);
assert.equal(replay.instanceHeadDigest, result.instanceHeadDigest);
});
test('bounds an unavailable edge probe without mutating lineage', async (t) => {
const state = fixture(t);
let clock = state.command.request.requestedAtMs;
let calls = 0;
const result = await proveLocalDeploymentLegacyReadiness(state.command, {
now: () => clock,
wait: async (milliseconds) => {
clock += milliseconds;
},
async probe() {
calls += 1;
return { ready: false, reason: 'unavailable' };
},
});
assert.equal(result.status, 'not_ready');
assert.equal(result.state, 'legacy_running');
assert.equal(result.reason, 'unavailable');
assert.equal(result.attempts, 60);
assert.equal(calls, 60);
assert.equal(
readLocalCutoverInstanceHead(
state.deploymentRoot,
state.command.request.instanceId,
state.uid,
).headDigest,
state.head.headDigest,
);
assert.equal(
fs.existsSync(
path.join(
state.deploymentRoot,
'service',
'cutover-instances',
state.command.request.instanceId,
'legacy-readiness-g1.json',
),
),
false,
);
});
test('fails a version mismatch and stale head before durable mutation', async (t) => {
const state = fixture(t);
const mismatch = await proveLocalDeploymentLegacyReadiness(state.command, {
async probe() {
return { ready: true, initialized: true, version: '2.20.0' };
},
});
assert.equal(mismatch.status, 'not_ready');
assert.equal(mismatch.reason, 'version_mismatch');
assert.equal(mismatch.attempts, 1);
let calls = 0;
await assert.rejects(
proveLocalDeploymentLegacyReadiness(
{
...state.command,
request: {
...state.command.request,
expectedInstanceHeadDigest: 'f'.repeat(64),
},
},
{
async probe() {
calls += 1;
return { ready: true, initialized: true, version: '2.21.0' };
},
},
),
/lost the instance head compare-and-swap/,
);
assert.equal(calls, 0);
});
test('probes only the fixed loopback system endpoint with bounded parsing', async (t) => {
const paths = [];
const server = http.createServer((request, response) => {
paths.push(request.url);
response.setHeader('content-type', 'application/json');
response.end(
JSON.stringify({
code: 200,
data: { isInitialized: true, version: '2.21.0' },
}),
);
});
await new Promise((resolve, reject) => {
server.once('error', reject);
server.listen(0, '127.0.0.1', resolve);
});
t.after(() => new Promise((resolve) => server.close(resolve)));
const address = server.address();
const result = await probeLegacySystemEndpoint({
host: '127.0.0.1',
port: address.port,
path: '/api/system',
timeoutMs: 2_000,
maxResponseBytes: 32 * 1024,
});
assert.deepEqual(result, {
ready: true,
initialized: true,
version: '2.21.0',
});
assert.deepEqual(paths, ['/api/system']);
});
test('rejects oversized and redirect responses without following them', async (t) => {
let redirected = false;
const server = http.createServer((request, response) => {
if (request.url === '/redirected') {
redirected = true;
response.end('unexpected');
return;
}
if (request.url === '/large') {
response.end('x'.repeat(2_048));
return;
}
response.writeHead(302, { location: '/redirected' });
response.end();
});
await new Promise((resolve, reject) => {
server.once('error', reject);
server.listen(0, '127.0.0.1', resolve);
});
t.after(() => new Promise((resolve) => server.close(resolve)));
const address = server.address();
const redirect = await probeLegacySystemEndpoint({
host: '127.0.0.1',
port: address.port,
path: '/api/system',
timeoutMs: 2_000,
maxResponseBytes: 1_024,
});
assert.deepEqual(redirect, { ready: false, reason: 'http_rejected' });
assert.equal(redirected, false);
const large = await probeLegacySystemEndpoint({
host: '127.0.0.1',
port: address.port,
path: '/large',
timeoutMs: 2_000,
maxResponseBytes: 1_024,
});
assert.deepEqual(large, {
ready: false,
reason: 'response_too_large',
});
});
@@ -95,6 +95,8 @@ function actorReport(name, kind, identityMode, scenario = 'success') {
'adoptedLegacyStarted',
'adoptedLegacyProcessIdentity',
'adoptedLegacyResponseLossInspected',
'adoptedLegacyReady',
'adoptedLegacyReadinessReplay',
]
: [
'adoptedLegacyBarrierCrash',
@@ -258,6 +260,12 @@ function main() {
adoptedLegacyResponseLossInspected: successReports.every(
(report) => report.gates.adoptedLegacyResponseLossInspected,
),
adoptedLegacyReady: successReports.every(
(report) => report.gates.adoptedLegacyReady,
),
adoptedLegacyReadinessReplay: successReports.every(
(report) => report.gates.adoptedLegacyReadinessReplay,
),
adoptedLegacyBarrierCrash: crashReports.length === 2,
adoptedLegacyInspectOnlyConvergence: crashReports.every(
(report) => report.gates.adoptedLegacyInspectOnlyConvergence,
@@ -136,7 +136,10 @@ function descriptor(kind, uid, gid, root, adopted = false) {
].join('\n');
}
function legacyDescriptor(kind, uid, gid) {
function legacyDescriptor(kind, uid, gid, port) {
const nodeExecutable = fs.realpathSync(process.execPath);
const commandArgs =
`/workspace/scripts/ql3-service-manager-legacy-live-service.cjs ${port}`;
if (kind === 'systemd') {
return [
'[Unit]',
@@ -146,7 +149,7 @@ function legacyDescriptor(kind, uid, gid) {
'Type=simple',
`User=${uid}`,
`Group=${gid}`,
'ExecStart=/bin/sleep 300',
`ExecStart=${nodeExecutable} ${commandArgs}`,
'Restart=no',
'',
'[Install]',
@@ -158,8 +161,8 @@ function legacyDescriptor(kind, uid, gid) {
'#!/sbin/openrc-run',
'name="qinglong"',
'description="QingLong 2 legacy rollback live gate"',
'command="/bin/sleep"',
'command_args="300"',
`command="${nodeExecutable}"`,
`command_args="${commandArgs}"`,
`command_user="${uid}:${gid}"`,
'supervisor="supervise-daemon"',
'retry="TERM/5/KILL/2"',
@@ -532,7 +535,7 @@ function authorizeAdoptedRollback(root, adopted, prepared, uid, gid) {
};
}
function installLegacyService(kind, managerOptions, uid, gid) {
function installLegacyService(kind, managerOptions, uid, gid, port) {
const descriptorPath =
kind === 'systemd'
? '/etc/systemd/system/qinglong.service'
@@ -540,7 +543,7 @@ function installLegacyService(kind, managerOptions, uid, gid) {
fs.rmSync(descriptorPath, { force: true });
writePrivate(
descriptorPath,
legacyDescriptor(kind, uid, gid),
legacyDescriptor(kind, uid, gid, port),
kind === 'systemd' ? 0o644 : 0o755,
0,
0,
@@ -622,6 +625,54 @@ function executeLegacyRollback(
return { commandPath, result, replay, consumed, outcome };
}
function proveLegacyReadiness(
root,
adopted,
consumed,
uid,
gid,
legacyHttpPort,
) {
const command = {
schemaVersion: 1,
operation: 'local.deployment.cutover.legacy-readiness-probe',
options: { deploymentRoot: root, allowRootService: uid === 0 },
request: {
cutoverId: adopted.cutoverId,
profile: 'edge',
instanceId: adopted.instanceId,
generation: 1,
expectedActivationDigest: adopted.activationDigest,
expectedInstanceHeadDigest: consumed.instanceHeadDigest,
expectedLegacyRunningRecordDigest: consumed.completionDigest,
legacyHttpPort,
expectedLegacyVersion: '2.21.0',
requestedAtMs: Date.now(),
},
};
const commandPath = path.join(root, 'owner-legacy-readiness-probe.json');
writePrivate(
commandPath,
`${JSON.stringify(command, null, 2)}\n`,
0o600,
uid,
gid,
);
const result = ownerCli(
uid,
gid,
'cutover-legacy-readiness-probe',
commandPath,
);
const replay = ownerCli(
uid,
gid,
'cutover-legacy-readiness-probe',
commandPath,
);
return { commandPath, result, replay };
}
function executeLegacyRollbackBarrierCrash(
root,
controllerRoot,
@@ -830,6 +881,7 @@ async function main(argv) {
}
const uid = identityMode === 'root' ? 0 : NON_ROOT_SERVICE_UID;
const gid = uid;
const legacyHttpPort = 15_700;
const root = `/var/lib/ql3-service-manager-${kind}-${identityMode}`;
const controllerRoot = `/var/lib/ql3-service-bridge-${kind}-${identityMode}`;
fs.rmSync(root, { force: true, recursive: true });
@@ -1035,7 +1087,7 @@ async function main(argv) {
) {
fail('adopted rollback authorization did not converge exactly');
}
installLegacyService(kind, managerOptions, uid, gid);
installLegacyService(kind, managerOptions, uid, gid, legacyHttpPort);
const adoptedLegacyStarted =
scenario === 'success'
? executeLegacyRollback(
@@ -1058,6 +1110,7 @@ async function main(argv) {
uid,
gid,
);
let adoptedLegacyReadiness;
if (scenario === 'success') {
if (
adoptedLegacyStarted.result.state !== 'legacy_running' ||
@@ -1078,6 +1131,26 @@ async function main(argv) {
if (legacyUid !== uid) {
fail(`legacy service process UID drifted: ${legacyUid} != ${uid}`);
}
adoptedLegacyReadiness = proveLegacyReadiness(
root,
adopted,
adoptedLegacyStarted.consumed,
uid,
gid,
legacyHttpPort,
);
if (
adoptedLegacyReadiness.result.status !== 'prepared' ||
adoptedLegacyReadiness.result.state !== 'legacy_ready' ||
adoptedLegacyReadiness.replay.status !== 'existing' ||
adoptedLegacyReadiness.replay.state !== 'legacy_ready' ||
adoptedLegacyReadiness.replay.receiptDigest !==
adoptedLegacyReadiness.result.receiptDigest ||
adoptedLegacyReadiness.replay.instanceHeadDigest !==
adoptedLegacyReadiness.result.instanceHeadDigest
) {
fail('legacy readiness proof did not converge exactly');
}
} else if (
adoptedLegacyStarted.result.state !== 'manual_required' ||
adoptedLegacyStarted.result.status !== 'prepared' ||
@@ -1116,6 +1189,8 @@ async function main(argv) {
legacyStarted: adoptedLegacyStarted.result,
legacyStartReplay: adoptedLegacyStarted.replay,
legacyConsumed: adoptedLegacyStarted.consumed,
legacyReadiness: adoptedLegacyReadiness?.result ?? null,
legacyReadinessReplay: adoptedLegacyReadiness?.replay ?? null,
},
gates: {
rootCommandFile: true,
@@ -1140,6 +1215,8 @@ async function main(argv) {
adoptedTargetRemainedStopped: true,
adoptedLegacyProcessIdentity: scenario === 'success',
adoptedLegacyResponseLossInspected: scenario === 'success',
adoptedLegacyReady: scenario === 'success',
adoptedLegacyReadinessReplay: scenario === 'success',
adoptedLegacyBarrierCrash: scenario === 'barrier-crash',
adoptedLegacyInspectOnlyConvergence: scenario === 'barrier-crash',
adoptedLegacyRemainedStopped: scenario === 'barrier-crash',
@@ -0,0 +1,44 @@
#!/usr/bin/env node
'use strict';
const http = require('node:http');
const port = Number(process.argv[2]);
if (!Number.isSafeInteger(port) || port < 1 || port > 65_535) {
process.stderr.write('legacy live service port is invalid\n');
process.exit(64);
}
const server = http.createServer((request, response) => {
if (request.method !== 'GET' || request.url !== '/api/system') {
response.writeHead(404, { connection: 'close' });
response.end();
return;
}
const body = JSON.stringify({
code: 200,
data: {
isInitialized: true,
version: '2.21.0',
publishTime: 1_787_200_000,
branch: 'next-live-gate',
changeLog: '',
changeLogLink: '',
},
});
response.writeHead(200, {
'content-type': 'application/json',
'content-length': Buffer.byteLength(body),
connection: 'close',
});
response.end(body);
});
function shutdown() {
server.close(() => process.exit(0));
}
process.on('SIGINT', shutdown);
process.on('SIGTERM', shutdown);
server.listen(port, '127.0.0.1');
@@ -0,0 +1,81 @@
require('ts-node/register/transpile-only');
require('reflect-metadata');
const assert = require('node:assert/strict');
const http = require('node:http');
const { after, before, test } = require('node:test');
const express = require('express');
const { Container } = require('typedi');
const config = require('../../back/config').default;
const UserService = require('../../back/services/user').default;
const registerSystemRoutes = require('../../back/api/system').default;
let authInfo = {
username: 'operator',
password: 'changed',
token: 'redacted',
};
let origin;
let server;
const logger = {
debug() {},
error() {},
info() {},
warn() {},
};
before(async () => {
Container.set('logger', logger);
Container.set(UserService, {
async getAuthInfo() {
return authInfo;
},
});
const app = express();
app.set('case sensitive routing', true);
app.set('strict routing', true);
const api = express.Router();
registerSystemRoutes(api);
app.use('/api', api);
server = http.createServer(app);
await new Promise((resolve, reject) => {
server.once('error', reject);
server.listen(0, '127.0.0.1', resolve);
});
const address = server.address();
origin = `http://127.0.0.1:${address.port}`;
});
after(async () => {
Container.remove(UserService);
Container.remove('logger');
await new Promise((resolve, reject) =>
server.close((error) => (error ? reject(error) : resolve())),
);
});
test('preserves the public 2.x system readiness envelope', async () => {
const response = await fetch(`${origin}/api/system`);
const body = await response.json();
assert.equal(response.status, 200);
assert.equal(body.code, 200);
assert.equal(body.data.isInitialized, true);
assert.equal(body.data.version, '2.21.0');
assert.equal(body.data.branch, process.env.QL_BRANCH || 'master');
assert.equal(Number.isSafeInteger(body.data.publishTime), true);
assert.equal(typeof body.data.changeLog, 'string');
assert.equal(typeof body.data.changeLogLink, 'string');
assert.equal(config.apiWhiteList.includes('/api/system'), true);
});
test('keeps default admin credentials distinguishable from readiness', async () => {
authInfo = { username: 'admin', password: 'admin' };
const response = await fetch(`${origin}/api/system`);
const body = await response.json();
assert.equal(response.status, 200);
assert.equal(body.code, 200);
assert.equal(body.data.isInitialized, false);
assert.equal(body.data.version, '2.21.0');
});
+2 -2
View File
@@ -207,10 +207,10 @@ test('current QL3 workspace has exactly eighteen reviewed package boundaries', (
rootSourceFileRoles: localOwnerCli.rootSourceFileRoles,
},
{
sourceFiles: 111,
sourceFiles: 113,
rootSourceFiles: 1,
rootSourceLines: 50,
nestedSourceFiles: 110,
nestedSourceFiles: 112,
rootSourceFileRoles: { 'cli.ts': 'binary_entry' },
},
);