mirror of
https://github.com/whyour/qinglong.git
synced 2026-09-23 12:05:27 +08:00
feat(local): bind service completion restart
This commit is contained in:
@@ -109,6 +109,21 @@
|
|||||||
`268 total / 261 pass / 7 conditional skip / 0 fail`、18-package clean build/逐包测试、dependency/package boundary `70/70` 和 122-module Edge
|
`268 total / 261 pass / 7 conditional skip / 0 fail`、18-package clean build/逐包测试、dependency/package boundary `70/70` 和 122-module Edge
|
||||||
import audit 已通过;其余六个领域的 terminal adapter、Service Manager v2 binding、完整 target readiness/restart 演练和 Cluster 自有 completion
|
import audit 已通过;其余六个领域的 terminal adapter、Service Manager v2 binding、完整 target readiness/restart 演练和 Cluster 自有 completion
|
||||||
authority 仍是后续门禁。
|
authority 仍是后续门禁。
|
||||||
|
- D-395/ADR-0489(已接受):systemd/OpenRC 不再因 v1 `previousRecordDigest` 同时承担历史 service record 与 current head source 两种含义而永久拒绝
|
||||||
|
completed restart。Service Manager intent 新增严格版本化 schema v2,且只允许 adopted generation ≥ 2 的 `restart`:保留
|
||||||
|
`previousRecordDigest` 精确绑定上一代 active record,并以 `completionFence.expectedInstanceHeadDigest|expectedCompletionDigest` 同时绑定当前
|
||||||
|
`reconciliation_completed` head 和 completion receipt digest。Owner 在 intent publish 前及 root outcome consume 后都重验 head,并打开上一代 active
|
||||||
|
record 复验 Profile、instance、activation、generation 与 record digest;v1 completed restart、缺字段、额外字段、旧 record、stale head、时间倒序全部失败关闭。
|
||||||
|
|
||||||
|
v2 产生的 service cutover record 使用 schema v3 显式保存双谱系证据;发布仍为 record-first、head-second。record publish 后响应丢失时,重放只有在
|
||||||
|
v3 record、intent、manager outcome、completed head 和 completion fence 全部一致时才 CAS 到下一代 `target_active`,不重复 root manager mutation,也不接受
|
||||||
|
上一代 startup receipt。新 receipt 或进程 identity 无法证明时,从 completed head 窄化为 `manual_required` 并可 exact replay。旧 v1 intent、普通 active
|
||||||
|
restart 和 v1/v2 journal 保持原语义,没有静默升级。实现全部留在既有 Local Owner Service Manager 子域,没有新增 package、`src/` 根平铺、dependency、
|
||||||
|
SQL、daemon、timer、watcher、listener、Pool 或 Cluster workload;Cluster restart/completion 仍需独立事务与 HA evidence。focused 组合门为
|
||||||
|
`75 total / 73 pass / 2 conditional Docker skip / 0 fail`,完整 Local Owner 为 `271 total / 264 pass / 7 conditional Docker skip / 0 fail`,18-package
|
||||||
|
clean build/逐包测试为 `2917 total / 2895 pass / 22 conditional integration skip / 0 fail`;package/dependency boundary `70/70`、122-module Edge import audit
|
||||||
|
与 18-package non-shallow 边界保持通过。本机 root service bridge Docker 门因 Docker Desktop `ENOSPC` 条件跳过且未清理用户匿名卷,必须由同一提交的远程
|
||||||
|
fresh runner root/non-root 门补齐。
|
||||||
- D-392/ADR-0485(已接受):D-391 的 signed review 不能直接获得通用 DML authority;表级 `adopt_legacy/retain_both` 也不能证明
|
- D-392/ADR-0485(已接受):D-391 的 signed review 不能直接获得通用 DML authority;表级 `adopt_legacy/retain_both` 也不能证明
|
||||||
Automation 行级 command/trigger 兼容,更不能覆盖 Secret custody、append-only history、Plugin/AI 外部资产与 Identity/Policy 语义。
|
Automation 行级 command/trigger 兼容,更不能覆盖 Secret custody、append-only history、Plugin/AI 外部资产与 Identity/Policy 语义。
|
||||||
因此既有 Local Owner 新增 `reconciliation.application.prepare|commit|verify`,以
|
因此既有 Local Owner 新增 `reconciliation.application.prepare|commit|verify`,以
|
||||||
|
|||||||
@@ -51,7 +51,7 @@ receipt publish、seal、head advance 或 backup collection 任一响应丢失
|
|||||||
|
|
||||||
Docker target authority 继续拒绝 `reconciliation_application_planned`、`reconciliation_automation_applied` 和所有 manual/rolled-back 中间态;只有 `reconciliation_completed` 能重新进入 `target_active`,且调用方必须使用下一 generation。
|
Docker target authority 继续拒绝 `reconciliation_application_planned`、`reconciliation_automation_applied` 和所有 manual/rolled-back 中间态;只有 `reconciliation_completed` 能重新进入 `target_active`,且调用方必须使用下一 generation。
|
||||||
|
|
||||||
Service Manager 虽然复用同一个 head assertion,但既有 v1 intent 只绑定 prior service journal `previousRecordDigest`,没有独立的 expected completion-head digest;其后续 compare-and-swap 因而继续拒绝 completed head。D-394 不通过忽略该比较来伪造兼容性。systemd/OpenRC 的 restart-ready 必须由后续 v2 intent 同时绑定 prior service record 与 exact completion head 后才能开放。
|
Service Manager 虽然复用同一个 head assertion,但既有 v1 intent 只绑定 prior service journal `previousRecordDigest`,没有独立的 expected completion-head digest;其后续 compare-and-swap 因而继续拒绝 completed head。D-394 不通过忽略该比较来伪造兼容性。该后续边界现由 D-395/ADR-0489 的 v2 intent 完成:systemd/OpenRC 必须同时绑定 prior service record、exact completion head 与 completion digest 才能获得 restart-ready authority;v1 仍失败关闭。
|
||||||
|
|
||||||
### 5. 部署规模与代码边界
|
### 5. 部署规模与代码边界
|
||||||
|
|
||||||
@@ -84,4 +84,4 @@ Service Manager 虽然复用同一个 head assertion,但既有 v1 intent 只
|
|||||||
- 完整 Local Owner `268 total / 261 pass / 7 conditional skip / 0 fail`;18-package clean build 与逐包测试 exit 0。
|
- 完整 Local Owner `268 total / 261 pass / 7 conditional skip / 0 fail`;18-package clean build 与逐包测试 exit 0。
|
||||||
- dependency/package boundary 组合门 `70/70`,Edge import audit 为 122 modules、0 forbidden;workspace 保持 18 packages、无 single-source/shallow package;Local Owner 为 `172 source / 171 nested / 1 root binary entry`。
|
- dependency/package boundary 组合门 `70/70`,Edge import audit 为 122 modules、0 forbidden;workspace 保持 18 packages、无 single-source/shallow package;Local Owner 为 `172 source / 171 nested / 1 root binary entry`。
|
||||||
- 不新增 production dependency、SQL migration、daemon、timer、watcher、listener、Pool、PostgreSQL role/ACL 或 cluster workload。
|
- 不新增 production dependency、SQL migration、daemon、timer、watcher、listener、Pool、PostgreSQL role/ACL 或 cluster workload。
|
||||||
- Secret/Config、Run History、Plugin Package、AI/Tool、Identity/Policy/Audit、Unknown 的 terminal adapter,以及 Service Manager v2 completion-head binding/真实 restart 演练仍是后续工作;D-394 不把尚未实现的领域或部署方式宣称为完成。
|
- Secret/Config、Run History、Plugin Package、AI/Tool、Identity/Policy/Audit、Unknown 的 terminal adapter,以及固定真实 systemd/OpenRC completed restart 演练仍是后续工作;Service Manager v2 completion-head binding 已由 D-395/ADR-0489 完成,D-394 本身不倒填该结论。
|
||||||
|
|||||||
@@ -0,0 +1,61 @@
|
|||||||
|
# ADR-0489:Service Manager 完成围栏重启谱系
|
||||||
|
|
||||||
|
- 状态:Accepted
|
||||||
|
- 日期:2026-08-22
|
||||||
|
- 决策:D-395
|
||||||
|
- 关联:ADR-0487、ADR-0488
|
||||||
|
|
||||||
|
## 背景
|
||||||
|
|
||||||
|
ADR-0488 建立 `reconciliation_completed`,并允许直接 Docker target 在下一 generation 重启,但 systemd/OpenRC 仍保持失败关闭。既有 Service Manager v1 intent 的 `previousRecordDigest` 同时承担“上一代 service active record”和“当前 instance head source record”两种含义;在 reconciliation 开始前二者相等,在 completed head 上却必然分离:历史 active record 仍用于排除旧 startup receipt,当前 head source 已变成 completion receipt digest。
|
||||||
|
|
||||||
|
忽略其中任意一条比较都会形成越权:只认历史 active record 会跳过八领域 completion,只认 completed head 则无法证明新 startup receipt 相对上一代发生变化。
|
||||||
|
|
||||||
|
## 决策
|
||||||
|
|
||||||
|
### 1. v2 intent 只表示 completed restart
|
||||||
|
|
||||||
|
`qinglong3-local-service-manager-intent` 接受 schema v2,但 v2 只允许 adopted、generation ≥ 2 的 `restart`。其 lineage 保留 `previousRecordDigest` 作为上一代 Service Manager active record,并新增一个内聚 `completionFence`:
|
||||||
|
|
||||||
|
- `expectedInstanceHeadDigest`:精确绑定当前 `reconciliation_completed` head;
|
||||||
|
- `expectedCompletionDigest`:必须等于该 head 的 `sourceRecordDigest`。
|
||||||
|
|
||||||
|
Owner 在 intent publish 前和 root outcome consume 后都重新读取 head;同时按 generation 打开上一代 active record,复验 record digest、Profile、instance 与 activation。intent 时间不得早于 completed head。调用方不能用 v2 表示 fresh install、普通 active restart、stop 或第一代 start。
|
||||||
|
|
||||||
|
### 2. v1 不被静默升级
|
||||||
|
|
||||||
|
v1 intent、普通 `target_active → restart` 和既有 v1/v2 service journal 继续按原契约解析。v1 intent 即使携带额外 completion 字段也因 exact-shape 失败;v2 缺少 completion fence 同样失败。completed head 不对 v1 放宽,因此旧 command、stale intent 或 root bridge replay 不能意外获得新 authority。
|
||||||
|
|
||||||
|
### 3. v3 journal 保存双谱系证据
|
||||||
|
|
||||||
|
由 v2 intent 产生的 Service Manager cutover record 使用 schema v3,在既有 manager outcome、application/activation/commitment、target identity、legacy data application 与 startup receipt 证据外,显式保存 completion fence。record 自身 digest 和 intent digest 共同覆盖上一代 service record 与 completed head;普通路径仍写既有 schema。
|
||||||
|
|
||||||
|
发布顺序保持 record-first、head-second。若 record 已发布而进程丢失响应,重放只在 v3 record、intent、outcome、completed head 和 completion fence 全部一致时把 head CAS 到 generation N 的 `target_active`;不会再次执行 root manager mutation 或接受旧 startup receipt。若 manager active 但新 receipt 无法证明,则允许从 completed head 窄化到 `manual_required`,该终态也能精确重放。
|
||||||
|
|
||||||
|
### 4. 部署与代码边界
|
||||||
|
|
||||||
|
实现留在既有 `@qinglong/local-owner-cli/deployment/service-manager` 内,没有新增 workspace package、常驻进程、timer、watcher、listener、Pool、SQL migration 或 production dependency。Edge/Standalone 只在显式管理命令中读取一个 current head 和一个上一代 service record;systemd/OpenRC root bridge 仍只消费 content-bounded intent,不读取 SQLite 或 reconciliation 大对象。
|
||||||
|
|
||||||
|
Cluster 不复用本机文件 journal。Kubernetes rollout、PostgreSQL completion 和多副本 restart authority 仍需各自的事务/CAS 与 HA evidence。
|
||||||
|
|
||||||
|
## 被拒绝的替代方案
|
||||||
|
|
||||||
|
### 把 `previousRecordDigest` 改成 completion digest
|
||||||
|
|
||||||
|
拒绝。它会失去上一代 startup receipt 和 Service Manager journal 的连续性,无法证明 restart 产生了新进程证据。
|
||||||
|
|
||||||
|
### completed head 对所有 v1 restart 自动放行
|
||||||
|
|
||||||
|
拒绝。旧 intent 没有声明或 digest-bind completion head,放行会把协议升级变成隐式权限扩大。
|
||||||
|
|
||||||
|
### 先推进 head、再补 service record
|
||||||
|
|
||||||
|
拒绝。响应丢失会留下 restart-ready head,却没有 manager outcome、startup receipt 或 process identity 的 durable record。
|
||||||
|
|
||||||
|
## 验收证据
|
||||||
|
|
||||||
|
- Service Manager contract/cutover、legacy rollback 与 reconciliation focused 组合门为 `75 total / 73 pass / 2 conditional Docker skip / 0 fail`,覆盖 v1/v2 exact shape、systemd/OpenRC contract、上一代 record 与 completed head 双重漂移拒绝、新 startup receipt、record-first response-loss replay,以及缺失新 receipt 时的 `manual_required` 收敛。
|
||||||
|
- HIGH/CRITICAL 影响范围通过完整 Local Owner `271 total / 264 pass / 7 conditional Docker skip / 0 fail`;18-package clean build/逐包测试为 `2917 total / 2895 pass / 22 conditional integration skip / 0 fail`。Worker receipt 测试在完整矩阵中复现约 `1.004 s` 的合法完成,轮询预算由 1 秒修正为 5 秒后 Worker Runtime 为 `134/134`,生产执行语义不变。
|
||||||
|
- dependency/package boundary 组合门 `70/70`,Edge import audit 为 122 modules、0 forbidden;workspace 保持 18 packages、无 single-source/shallow package,Local Owner 为 `172 source / 171 nested / 1 root binary entry`。
|
||||||
|
- 本机 root systemd/OpenRC Docker 门因 Docker Desktop 存储 `ENOSPC` 条件跳过,未清理用户匿名卷;同一提交仍必须通过远程全新 runner 的 root/non-root service bridge 和多架构门,不能用聚焦测试替代。
|
||||||
|
- 不新增 production dependency、SQL migration、daemon、timer、watcher、listener、Pool、PostgreSQL role/ACL、cluster workload、package 或 `src/` 根平铺文件。
|
||||||
@@ -474,6 +474,7 @@ export function advanceLocalCutoverInstanceHead(
|
|||||||
(state === 'manual_required' &&
|
(state === 'manual_required' &&
|
||||||
(current.state === 'legacy_stopped' ||
|
(current.state === 'legacy_stopped' ||
|
||||||
current.state === 'target_active' ||
|
current.state === 'target_active' ||
|
||||||
|
current.state === 'reconciliation_completed' ||
|
||||||
current.state === 'rollback_prepared' ||
|
current.state === 'rollback_prepared' ||
|
||||||
current.state === 'legacy_restart_requested'));
|
current.state === 'legacy_restart_requested'));
|
||||||
if (!allowed)
|
if (!allowed)
|
||||||
|
|||||||
+51
-7
@@ -27,10 +27,14 @@ export type LocalServiceManagerIntentLineage =
|
|||||||
generation: number;
|
generation: number;
|
||||||
expectedActivationDigest: string;
|
expectedActivationDigest: string;
|
||||||
previousRecordDigest: string;
|
previousRecordDigest: string;
|
||||||
|
completionFence?: Readonly<{
|
||||||
|
expectedInstanceHeadDigest: string;
|
||||||
|
expectedCompletionDigest: string;
|
||||||
|
}>;
|
||||||
}>;
|
}>;
|
||||||
|
|
||||||
export interface LocalServiceManagerIntent {
|
export interface LocalServiceManagerIntent {
|
||||||
readonly schemaVersion: 1;
|
readonly schemaVersion: 1 | 2;
|
||||||
readonly kind: 'qinglong3-local-service-manager-intent';
|
readonly kind: 'qinglong3-local-service-manager-intent';
|
||||||
readonly actionId: string;
|
readonly actionId: string;
|
||||||
readonly action: LocalServiceManagerAction;
|
readonly action: LocalServiceManagerAction;
|
||||||
@@ -169,7 +173,10 @@ function safeInteger(value: unknown, label: string, maximum: number): number {
|
|||||||
return value as number;
|
return value as number;
|
||||||
}
|
}
|
||||||
|
|
||||||
function normalizedLineage(value: unknown): LocalServiceManagerIntentLineage {
|
function normalizedLineage(
|
||||||
|
value: unknown,
|
||||||
|
schemaVersion: 1 | 2,
|
||||||
|
): LocalServiceManagerIntentLineage {
|
||||||
const lineage = object(value, 'lineage');
|
const lineage = object(value, 'lineage');
|
||||||
if (lineage.mode === 'fresh') {
|
if (lineage.mode === 'fresh') {
|
||||||
exact(lineage, ['mode'], 'lineage');
|
exact(lineage, ['mode'], 'lineage');
|
||||||
@@ -179,6 +186,7 @@ function normalizedLineage(value: unknown): LocalServiceManagerIntentLineage {
|
|||||||
lineage,
|
lineage,
|
||||||
[
|
[
|
||||||
'cutoverId',
|
'cutoverId',
|
||||||
|
...(schemaVersion === 2 ? ['completionFence'] : []),
|
||||||
'expectedActivationDigest',
|
'expectedActivationDigest',
|
||||||
'generation',
|
'generation',
|
||||||
'mode',
|
'mode',
|
||||||
@@ -186,6 +194,17 @@ function normalizedLineage(value: unknown): LocalServiceManagerIntentLineage {
|
|||||||
],
|
],
|
||||||
'lineage',
|
'lineage',
|
||||||
);
|
);
|
||||||
|
const completionFence =
|
||||||
|
schemaVersion === 2
|
||||||
|
? object(lineage.completionFence, 'completion fence')
|
||||||
|
: undefined;
|
||||||
|
if (completionFence !== undefined) {
|
||||||
|
exact(
|
||||||
|
completionFence,
|
||||||
|
['expectedCompletionDigest', 'expectedInstanceHeadDigest'],
|
||||||
|
'completion fence',
|
||||||
|
);
|
||||||
|
}
|
||||||
if (
|
if (
|
||||||
lineage.mode !== 'adopted' ||
|
lineage.mode !== 'adopted' ||
|
||||||
typeof lineage.cutoverId !== 'string' ||
|
typeof lineage.cutoverId !== 'string' ||
|
||||||
@@ -196,7 +215,12 @@ function normalizedLineage(value: unknown): LocalServiceManagerIntentLineage {
|
|||||||
typeof lineage.expectedActivationDigest !== 'string' ||
|
typeof lineage.expectedActivationDigest !== 'string' ||
|
||||||
!DIGEST_PATTERN.test(lineage.expectedActivationDigest) ||
|
!DIGEST_PATTERN.test(lineage.expectedActivationDigest) ||
|
||||||
typeof lineage.previousRecordDigest !== 'string' ||
|
typeof lineage.previousRecordDigest !== 'string' ||
|
||||||
!DIGEST_PATTERN.test(lineage.previousRecordDigest)
|
!DIGEST_PATTERN.test(lineage.previousRecordDigest) ||
|
||||||
|
(completionFence !== undefined &&
|
||||||
|
(typeof completionFence.expectedInstanceHeadDigest !== 'string' ||
|
||||||
|
!DIGEST_PATTERN.test(completionFence.expectedInstanceHeadDigest) ||
|
||||||
|
typeof completionFence.expectedCompletionDigest !== 'string' ||
|
||||||
|
!DIGEST_PATTERN.test(completionFence.expectedCompletionDigest)))
|
||||||
) {
|
) {
|
||||||
configurationError('adopted lineage is invalid');
|
configurationError('adopted lineage is invalid');
|
||||||
}
|
}
|
||||||
@@ -206,6 +230,16 @@ function normalizedLineage(value: unknown): LocalServiceManagerIntentLineage {
|
|||||||
generation: lineage.generation as number,
|
generation: lineage.generation as number,
|
||||||
expectedActivationDigest: lineage.expectedActivationDigest,
|
expectedActivationDigest: lineage.expectedActivationDigest,
|
||||||
previousRecordDigest: lineage.previousRecordDigest,
|
previousRecordDigest: lineage.previousRecordDigest,
|
||||||
|
...(completionFence === undefined
|
||||||
|
? {}
|
||||||
|
: {
|
||||||
|
completionFence: Object.freeze({
|
||||||
|
expectedInstanceHeadDigest:
|
||||||
|
completionFence.expectedInstanceHeadDigest as string,
|
||||||
|
expectedCompletionDigest:
|
||||||
|
completionFence.expectedCompletionDigest as string,
|
||||||
|
}),
|
||||||
|
}),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -297,9 +331,12 @@ export function normalizeLocalServiceManagerIntent(
|
|||||||
: '/etc/init.d/qinglong3';
|
: '/etc/init.d/qinglong3';
|
||||||
const expectedSourceMode = service.kind === 'systemd' ? 0o600 : 0o700;
|
const expectedSourceMode = service.kind === 'systemd' ? 0o600 : 0o700;
|
||||||
const expectedDestinationMode = service.kind === 'systemd' ? 0o644 : 0o755;
|
const expectedDestinationMode = service.kind === 'systemd' ? 0o644 : 0o755;
|
||||||
const lineage = normalizedLineage(intent.lineage);
|
if (intent.schemaVersion !== 1 && intent.schemaVersion !== 2) {
|
||||||
|
configurationError('service manager intent schema version is invalid');
|
||||||
|
}
|
||||||
|
const schemaVersion = intent.schemaVersion;
|
||||||
|
const lineage = normalizedLineage(intent.lineage, schemaVersion);
|
||||||
if (
|
if (
|
||||||
intent.schemaVersion !== 1 ||
|
|
||||||
intent.kind !== 'qinglong3-local-service-manager-intent' ||
|
intent.kind !== 'qinglong3-local-service-manager-intent' ||
|
||||||
typeof intent.actionId !== 'string' ||
|
typeof intent.actionId !== 'string' ||
|
||||||
!UUID_V4_PATTERN.test(intent.actionId) ||
|
!UUID_V4_PATTERN.test(intent.actionId) ||
|
||||||
@@ -336,7 +373,9 @@ export function normalizeLocalServiceManagerIntent(
|
|||||||
}
|
}
|
||||||
if (
|
if (
|
||||||
lineage.mode === 'adopted' &&
|
lineage.mode === 'adopted' &&
|
||||||
((lineage.generation === 1 &&
|
(schemaVersion === 2
|
||||||
|
? lineage.generation < 2 || intent.action !== 'restart'
|
||||||
|
: (lineage.generation === 1 &&
|
||||||
intent.action !== 'install-enable-start' &&
|
intent.action !== 'install-enable-start' &&
|
||||||
intent.action !== 'start' &&
|
intent.action !== 'start' &&
|
||||||
intent.action !== 'stop') ||
|
intent.action !== 'stop') ||
|
||||||
@@ -348,8 +387,13 @@ export function normalizeLocalServiceManagerIntent(
|
|||||||
'service manager action does not match the adopted generation',
|
'service manager action does not match the adopted generation',
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
if (schemaVersion === 2 && lineage.mode !== 'adopted') {
|
||||||
|
configurationError(
|
||||||
|
'service manager v2 intent requires adopted completion lineage',
|
||||||
|
);
|
||||||
|
}
|
||||||
const payload = Object.freeze({
|
const payload = Object.freeze({
|
||||||
schemaVersion: 1 as const,
|
schemaVersion,
|
||||||
kind: 'qinglong3-local-service-manager-intent' as const,
|
kind: 'qinglong3-local-service-manager-intent' as const,
|
||||||
actionId: intent.actionId,
|
actionId: intent.actionId,
|
||||||
action: intent.action,
|
action: intent.action,
|
||||||
|
|||||||
+68
-7
@@ -84,6 +84,7 @@ export interface LocalServiceManagerCutoverDependencies {
|
|||||||
readonly procRoot?: string;
|
readonly procRoot?: string;
|
||||||
readonly now?: () => number;
|
readonly now?: () => number;
|
||||||
readonly wait?: (milliseconds: number) => Promise<void>;
|
readonly wait?: (milliseconds: number) => Promise<void>;
|
||||||
|
readonly afterRecordPublished?: () => void;
|
||||||
}
|
}
|
||||||
|
|
||||||
interface AdoptedBinding {
|
interface AdoptedBinding {
|
||||||
@@ -736,16 +737,28 @@ function replayResult(
|
|||||||
): Readonly<LocalServiceManagerCutoverConsumeResult> | undefined {
|
): Readonly<LocalServiceManagerCutoverConsumeResult> | undefined {
|
||||||
if (intent.lineage.mode !== 'adopted') return undefined;
|
if (intent.lineage.mode !== 'adopted') return undefined;
|
||||||
const state = desiredState(intent, outcome);
|
const state = desiredState(intent, outcome);
|
||||||
const recordPath = localServiceManagerCutoverRecordPath(intent, state);
|
const candidatePaths = [
|
||||||
if (!fs.existsSync(recordPath)) return undefined;
|
localServiceManagerCutoverRecordPath(intent, state),
|
||||||
|
...(state === 'manual_required'
|
||||||
|
? []
|
||||||
|
: [localServiceManagerCutoverRecordPath(intent, 'manual_required')]),
|
||||||
|
].filter((candidate) => fs.existsSync(candidate));
|
||||||
|
if (candidatePaths.length === 0) return undefined;
|
||||||
|
if (candidatePaths.length !== 1) {
|
||||||
|
configurationError('service manager cutover replay is ambiguous');
|
||||||
|
}
|
||||||
const record = normalizeLocalServiceManagerCutoverRecord(
|
const record = normalizeLocalServiceManagerCutoverRecord(
|
||||||
readPrivateLocalCommandFile(recordPath),
|
readPrivateLocalCommandFile(candidatePaths[0]!),
|
||||||
);
|
);
|
||||||
const head = readLocalCutoverInstanceHead(
|
let head = readLocalCutoverInstanceHead(
|
||||||
intent.deployment.root,
|
intent.deployment.root,
|
||||||
intent.instanceId,
|
intent.instanceId,
|
||||||
currentIdentity().uid,
|
currentIdentity().uid,
|
||||||
);
|
);
|
||||||
|
const completionFence =
|
||||||
|
intent.schemaVersion === 2 && intent.lineage.mode === 'adopted'
|
||||||
|
? intent.lineage.completionFence
|
||||||
|
: undefined;
|
||||||
const stoppedCaptureProgress =
|
const stoppedCaptureProgress =
|
||||||
record.state === 'target_stopped' &&
|
record.state === 'target_stopped' &&
|
||||||
(head.state === 'reconciliation_capture_prepared' ||
|
(head.state === 'reconciliation_capture_prepared' ||
|
||||||
@@ -759,17 +772,58 @@ function replayResult(
|
|||||||
head.state === 'reconciliation_automation_planned' ||
|
head.state === 'reconciliation_automation_planned' ||
|
||||||
head.state === 'reconciliation_automation_decision_prepared' ||
|
head.state === 'reconciliation_automation_decision_prepared' ||
|
||||||
head.state === 'reconciliation_automation_reviewed');
|
head.state === 'reconciliation_automation_reviewed');
|
||||||
|
const completionRestartPendingHead =
|
||||||
|
(record.state === 'target_active' || record.state === 'manual_required') &&
|
||||||
|
completionFence !== undefined &&
|
||||||
|
record.schemaVersion === 3 &&
|
||||||
|
record.evidence.completionFence?.expectedInstanceHeadDigest ===
|
||||||
|
completionFence.expectedInstanceHeadDigest &&
|
||||||
|
record.evidence.completionFence.expectedCompletionDigest ===
|
||||||
|
completionFence.expectedCompletionDigest &&
|
||||||
|
head.state === 'reconciliation_completed' &&
|
||||||
|
head.headDigest === completionFence.expectedInstanceHeadDigest &&
|
||||||
|
head.sourceRecordDigest === completionFence.expectedCompletionDigest;
|
||||||
if (
|
if (
|
||||||
record.actionId !== intent.actionId ||
|
record.actionId !== intent.actionId ||
|
||||||
record.intentDigest !== intent.intentDigest ||
|
record.intentDigest !== intent.intentDigest ||
|
||||||
record.evidence.managerOutcomeDigest !== outcome.outcomeDigest ||
|
record.evidence.managerOutcomeDigest !== outcome.outcomeDigest ||
|
||||||
|
record.profile !== intent.profile ||
|
||||||
|
record.instanceId !== intent.instanceId ||
|
||||||
|
record.activationDigest !== intent.lineage.expectedActivationDigest ||
|
||||||
|
record.generation !== intent.lineage.generation ||
|
||||||
|
record.previousRecordDigest !== intent.lineage.previousRecordDigest ||
|
||||||
head.cutoverId !== intent.lineage.cutoverId ||
|
head.cutoverId !== intent.lineage.cutoverId ||
|
||||||
head.generation !== intent.lineage.generation ||
|
head.generation !==
|
||||||
(!stoppedCaptureProgress && head.state !== record.state) ||
|
(completionRestartPendingHead
|
||||||
(!stoppedCaptureProgress && head.sourceRecordDigest !== record.recordDigest)
|
? intent.lineage.generation - 1
|
||||||
|
: intent.lineage.generation) ||
|
||||||
|
(!stoppedCaptureProgress &&
|
||||||
|
!completionRestartPendingHead &&
|
||||||
|
head.state !== record.state) ||
|
||||||
|
(!stoppedCaptureProgress &&
|
||||||
|
!completionRestartPendingHead &&
|
||||||
|
head.sourceRecordDigest !== record.recordDigest)
|
||||||
) {
|
) {
|
||||||
configurationError('service manager cutover replay drifted');
|
configurationError('service manager cutover replay drifted');
|
||||||
}
|
}
|
||||||
|
if (completionRestartPendingHead) {
|
||||||
|
head = advanceLocalCutoverInstanceHead(
|
||||||
|
{
|
||||||
|
options: { deploymentRoot: intent.deployment.root },
|
||||||
|
request: {
|
||||||
|
cutoverId: intent.lineage.cutoverId,
|
||||||
|
profile: intent.profile,
|
||||||
|
instanceId: intent.instanceId,
|
||||||
|
expectedActivationDigest: intent.lineage.expectedActivationDigest,
|
||||||
|
requestedAtMs: record.completedAtMs,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
currentIdentity().uid,
|
||||||
|
record.state,
|
||||||
|
intent.lineage.generation,
|
||||||
|
record.recordDigest,
|
||||||
|
);
|
||||||
|
}
|
||||||
return Object.freeze({
|
return Object.freeze({
|
||||||
schemaVersion: 1 as const,
|
schemaVersion: 1 as const,
|
||||||
operation: command.operation,
|
operation: command.operation,
|
||||||
@@ -895,6 +949,12 @@ export async function consumeLocalServiceManagerCutoverOutcome(
|
|||||||
shutdownReceiptDigest,
|
shutdownReceiptDigest,
|
||||||
processIdentityDigest,
|
processIdentityDigest,
|
||||||
manualReason,
|
manualReason,
|
||||||
|
...(intent.schemaVersion === 2 &&
|
||||||
|
intent.lineage.completionFence !== undefined
|
||||||
|
? {
|
||||||
|
completionFence: intent.lineage.completionFence,
|
||||||
|
}
|
||||||
|
: {}),
|
||||||
});
|
});
|
||||||
const record = localServiceManagerCutoverRecord(
|
const record = localServiceManagerCutoverRecord(
|
||||||
intent,
|
intent,
|
||||||
@@ -907,6 +967,7 @@ export async function consumeLocalServiceManagerCutoverOutcome(
|
|||||||
record,
|
record,
|
||||||
identity.uid,
|
identity.uid,
|
||||||
);
|
);
|
||||||
|
dependencies.afterRecordPublished?.();
|
||||||
const head = advanceLocalCutoverInstanceHead(
|
const head = advanceLocalCutoverInstanceHead(
|
||||||
{
|
{
|
||||||
options: { deploymentRoot: intent.deployment.root },
|
options: { deploymentRoot: intent.deployment.root },
|
||||||
|
|||||||
+51
-4
@@ -33,11 +33,15 @@ export interface LocalServiceManagerCutoverEvidence {
|
|||||||
readonly shutdownReceiptDigest: string | null;
|
readonly shutdownReceiptDigest: string | null;
|
||||||
readonly processIdentityDigest: string | null;
|
readonly processIdentityDigest: string | null;
|
||||||
readonly manualReason: string | null;
|
readonly manualReason: string | null;
|
||||||
|
readonly completionFence?: Readonly<{
|
||||||
|
expectedInstanceHeadDigest: string;
|
||||||
|
expectedCompletionDigest: string;
|
||||||
|
}>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface LocalServiceManagerCutoverRecord {
|
export interface LocalServiceManagerCutoverRecord {
|
||||||
readonly schema: typeof SCHEMA;
|
readonly schema: typeof SCHEMA;
|
||||||
readonly schemaVersion: 1 | 2;
|
readonly schemaVersion: 1 | 2 | 3;
|
||||||
readonly actionId: string;
|
readonly actionId: string;
|
||||||
readonly action: LocalServiceManagerAction;
|
readonly action: LocalServiceManagerAction;
|
||||||
readonly state: LocalServiceManagerCutoverState;
|
readonly state: LocalServiceManagerCutoverState;
|
||||||
@@ -145,6 +149,7 @@ export function localServiceManagerCutoverRecord(
|
|||||||
evidence,
|
evidence,
|
||||||
'legacyDataApplicationReceiptDigest',
|
'legacyDataApplicationReceiptDigest',
|
||||||
);
|
);
|
||||||
|
const hasCompletionFence = Object.hasOwn(evidence, 'completionFence');
|
||||||
if (
|
if (
|
||||||
hasCommitDigest !== hasReceiptDigest ||
|
hasCommitDigest !== hasReceiptDigest ||
|
||||||
(hasCommitDigest &&
|
(hasCommitDigest &&
|
||||||
@@ -153,9 +158,25 @@ export function localServiceManagerCutoverRecord(
|
|||||||
) {
|
) {
|
||||||
configurationError('legacy data application evidence is incomplete');
|
configurationError('legacy data application evidence is incomplete');
|
||||||
}
|
}
|
||||||
|
if (
|
||||||
|
hasCompletionFence !== (intent.schemaVersion === 2) ||
|
||||||
|
(hasCompletionFence &&
|
||||||
|
(intent.lineage.mode !== 'adopted' ||
|
||||||
|
intent.lineage.completionFence === undefined ||
|
||||||
|
evidence.completionFence?.expectedInstanceHeadDigest !==
|
||||||
|
intent.lineage.completionFence.expectedInstanceHeadDigest ||
|
||||||
|
evidence.completionFence?.expectedCompletionDigest !==
|
||||||
|
intent.lineage.completionFence.expectedCompletionDigest))
|
||||||
|
) {
|
||||||
|
configurationError('service manager completion evidence is incomplete');
|
||||||
|
}
|
||||||
const payload = Object.freeze({
|
const payload = Object.freeze({
|
||||||
schema: SCHEMA,
|
schema: SCHEMA,
|
||||||
schemaVersion: hasCommitDigest ? (2 as const) : (1 as const),
|
schemaVersion: hasCompletionFence
|
||||||
|
? (3 as const)
|
||||||
|
: hasCommitDigest
|
||||||
|
? (2 as const)
|
||||||
|
: (1 as const),
|
||||||
actionId: intent.actionId,
|
actionId: intent.actionId,
|
||||||
action: intent.action,
|
action: intent.action,
|
||||||
state,
|
state,
|
||||||
@@ -213,6 +234,12 @@ export function normalizeLocalServiceManagerCutoverRecord(
|
|||||||
'legacyDataApplicationCommitDigest',
|
'legacyDataApplicationCommitDigest',
|
||||||
'legacyDataApplicationReceiptDigest',
|
'legacyDataApplicationReceiptDigest',
|
||||||
]
|
]
|
||||||
|
: record.schemaVersion === 3
|
||||||
|
? [
|
||||||
|
'completionFence',
|
||||||
|
'legacyDataApplicationCommitDigest',
|
||||||
|
'legacyDataApplicationReceiptDigest',
|
||||||
|
]
|
||||||
: []),
|
: []),
|
||||||
'manualReason',
|
'manualReason',
|
||||||
'processIdentityDigest',
|
'processIdentityDigest',
|
||||||
@@ -226,9 +253,22 @@ export function normalizeLocalServiceManagerCutoverRecord(
|
|||||||
candidate === null ||
|
candidate === null ||
|
||||||
(typeof candidate === 'string' && DIGEST_PATTERN.test(candidate));
|
(typeof candidate === 'string' && DIGEST_PATTERN.test(candidate));
|
||||||
const { recordDigest, ...payload } = record;
|
const { recordDigest, ...payload } = record;
|
||||||
|
const completionFence =
|
||||||
|
record.schemaVersion === 3
|
||||||
|
? object(evidence.completionFence, 'service manager completion evidence')
|
||||||
|
: undefined;
|
||||||
|
if (completionFence !== undefined) {
|
||||||
|
exact(
|
||||||
|
completionFence,
|
||||||
|
['expectedCompletionDigest', 'expectedInstanceHeadDigest'],
|
||||||
|
'service manager completion evidence',
|
||||||
|
);
|
||||||
|
}
|
||||||
if (
|
if (
|
||||||
record.schema !== SCHEMA ||
|
record.schema !== SCHEMA ||
|
||||||
(record.schemaVersion !== 1 && record.schemaVersion !== 2) ||
|
(record.schemaVersion !== 1 &&
|
||||||
|
record.schemaVersion !== 2 &&
|
||||||
|
record.schemaVersion !== 3) ||
|
||||||
typeof record.actionId !== 'string' ||
|
typeof record.actionId !== 'string' ||
|
||||||
(record.action !== 'install-enable-start' &&
|
(record.action !== 'install-enable-start' &&
|
||||||
record.action !== 'start' &&
|
record.action !== 'start' &&
|
||||||
@@ -244,6 +284,8 @@ export function normalizeLocalServiceManagerCutoverRecord(
|
|||||||
!DIGEST_PATTERN.test(record.activationDigest) ||
|
!DIGEST_PATTERN.test(record.activationDigest) ||
|
||||||
!Number.isSafeInteger(record.generation) ||
|
!Number.isSafeInteger(record.generation) ||
|
||||||
(record.generation as number) < 1 ||
|
(record.generation as number) < 1 ||
|
||||||
|
(record.schemaVersion === 3 &&
|
||||||
|
(record.action !== 'restart' || (record.generation as number) < 2)) ||
|
||||||
typeof record.previousRecordDigest !== 'string' ||
|
typeof record.previousRecordDigest !== 'string' ||
|
||||||
!DIGEST_PATTERN.test(record.previousRecordDigest) ||
|
!DIGEST_PATTERN.test(record.previousRecordDigest) ||
|
||||||
typeof record.intentDigest !== 'string' ||
|
typeof record.intentDigest !== 'string' ||
|
||||||
@@ -264,11 +306,16 @@ export function normalizeLocalServiceManagerCutoverRecord(
|
|||||||
!DIGEST_PATTERN.test(evidence.commitmentDigest) ||
|
!DIGEST_PATTERN.test(evidence.commitmentDigest) ||
|
||||||
typeof evidence.targetDataIdentityDigest !== 'string' ||
|
typeof evidence.targetDataIdentityDigest !== 'string' ||
|
||||||
!DIGEST_PATTERN.test(evidence.targetDataIdentityDigest) ||
|
!DIGEST_PATTERN.test(evidence.targetDataIdentityDigest) ||
|
||||||
(record.schemaVersion === 2 &&
|
((record.schemaVersion === 2 || record.schemaVersion === 3) &&
|
||||||
(!nullableDigest(evidence.legacyDataApplicationCommitDigest) ||
|
(!nullableDigest(evidence.legacyDataApplicationCommitDigest) ||
|
||||||
!nullableDigest(evidence.legacyDataApplicationReceiptDigest) ||
|
!nullableDigest(evidence.legacyDataApplicationReceiptDigest) ||
|
||||||
(evidence.legacyDataApplicationCommitDigest === null) !==
|
(evidence.legacyDataApplicationCommitDigest === null) !==
|
||||||
(evidence.legacyDataApplicationReceiptDigest === null))) ||
|
(evidence.legacyDataApplicationReceiptDigest === null))) ||
|
||||||
|
(completionFence !== undefined &&
|
||||||
|
(typeof completionFence.expectedInstanceHeadDigest !== 'string' ||
|
||||||
|
!DIGEST_PATTERN.test(completionFence.expectedInstanceHeadDigest) ||
|
||||||
|
typeof completionFence.expectedCompletionDigest !== 'string' ||
|
||||||
|
!DIGEST_PATTERN.test(completionFence.expectedCompletionDigest))) ||
|
||||||
!nullableDigest(evidence.startupReceiptDigest) ||
|
!nullableDigest(evidence.startupReceiptDigest) ||
|
||||||
!nullableDigest(evidence.shutdownReceiptDigest) ||
|
!nullableDigest(evidence.shutdownReceiptDigest) ||
|
||||||
!nullableDigest(evidence.processIdentityDigest) ||
|
!nullableDigest(evidence.processIdentityDigest) ||
|
||||||
|
|||||||
+38
-4
@@ -34,6 +34,7 @@ import {
|
|||||||
normalizeLocalServiceManagerOutcome,
|
normalizeLocalServiceManagerOutcome,
|
||||||
type LocalServiceManagerOutcome,
|
type LocalServiceManagerOutcome,
|
||||||
} from './serviceOutcomeContract';
|
} from './serviceOutcomeContract';
|
||||||
|
import { readLocalServiceManagerActiveRecord } from './serviceCutoverJournal';
|
||||||
|
|
||||||
const MAX_PATH_BYTES = 4_096;
|
const MAX_PATH_BYTES = 4_096;
|
||||||
const MAX_DESCRIPTOR_BYTES = 64 * 1024;
|
const MAX_DESCRIPTOR_BYTES = 64 * 1024;
|
||||||
@@ -43,7 +44,7 @@ const UUID_V4_PATTERN =
|
|||||||
const DIGEST_PATTERN = /^[0-9a-f]{64}$/;
|
const DIGEST_PATTERN = /^[0-9a-f]{64}$/;
|
||||||
|
|
||||||
export interface LocalServiceManagerIntentPrepareCommand {
|
export interface LocalServiceManagerIntentPrepareCommand {
|
||||||
readonly schemaVersion: 1;
|
readonly schemaVersion: 1 | 2;
|
||||||
readonly operation: 'local.deployment.service-manager.intent.prepare';
|
readonly operation: 'local.deployment.service-manager.intent.prepare';
|
||||||
readonly options: Readonly<{
|
readonly options: Readonly<{
|
||||||
deploymentRoot: string;
|
deploymentRoot: string;
|
||||||
@@ -454,6 +455,39 @@ function assertIntentLineageHead(
|
|||||||
}),
|
}),
|
||||||
uid,
|
uid,
|
||||||
);
|
);
|
||||||
|
if (intent.schemaVersion === 2) {
|
||||||
|
if (
|
||||||
|
intent.action !== 'restart' ||
|
||||||
|
intent.lineage.completionFence === undefined ||
|
||||||
|
head.state !== 'reconciliation_completed' ||
|
||||||
|
head.generation !== intent.lineage.generation - 1 ||
|
||||||
|
head.headDigest !==
|
||||||
|
intent.lineage.completionFence.expectedInstanceHeadDigest ||
|
||||||
|
head.sourceRecordDigest !==
|
||||||
|
intent.lineage.completionFence.expectedCompletionDigest ||
|
||||||
|
intent.requestedAtMs < head.updatedAtMs
|
||||||
|
) {
|
||||||
|
configurationError(
|
||||||
|
'service manager completion restart lost the instance head compare-and-swap',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const previous = readLocalServiceManagerActiveRecord(
|
||||||
|
intent.deployment.root,
|
||||||
|
intent.lineage.cutoverId,
|
||||||
|
intent.lineage.generation - 1,
|
||||||
|
);
|
||||||
|
if (
|
||||||
|
previous.recordDigest !== intent.lineage.previousRecordDigest ||
|
||||||
|
previous.profile !== intent.profile ||
|
||||||
|
previous.instanceId !== intent.instanceId ||
|
||||||
|
previous.activationDigest !== intent.lineage.expectedActivationDigest
|
||||||
|
) {
|
||||||
|
configurationError(
|
||||||
|
'service manager completion restart lost the previous active record',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
const expected =
|
const expected =
|
||||||
intent.action === 'restart'
|
intent.action === 'restart'
|
||||||
? Object.freeze({
|
? Object.freeze({
|
||||||
@@ -495,7 +529,7 @@ function normalizePrepareCommand(
|
|||||||
'request',
|
'request',
|
||||||
);
|
);
|
||||||
if (
|
if (
|
||||||
command.schemaVersion !== 1 ||
|
(command.schemaVersion !== 1 && command.schemaVersion !== 2) ||
|
||||||
command.operation !== 'local.deployment.service-manager.intent.prepare' ||
|
command.operation !== 'local.deployment.service-manager.intent.prepare' ||
|
||||||
typeof request.actionId !== 'string' ||
|
typeof request.actionId !== 'string' ||
|
||||||
!UUID_V4_PATTERN.test(request.actionId) ||
|
!UUID_V4_PATTERN.test(request.actionId) ||
|
||||||
@@ -510,7 +544,7 @@ function normalizePrepareCommand(
|
|||||||
configurationError('service manager intent command is invalid');
|
configurationError('service manager intent command is invalid');
|
||||||
}
|
}
|
||||||
return Object.freeze({
|
return Object.freeze({
|
||||||
schemaVersion: 1 as const,
|
schemaVersion: command.schemaVersion,
|
||||||
operation: 'local.deployment.service-manager.intent.prepare' as const,
|
operation: 'local.deployment.service-manager.intent.prepare' as const,
|
||||||
options: Object.freeze({
|
options: Object.freeze({
|
||||||
deploymentRoot: safeAbsolutePath(
|
deploymentRoot: safeAbsolutePath(
|
||||||
@@ -576,7 +610,7 @@ export function prepareLocalServiceManagerIntent(
|
|||||||
const application = parseApplicationIdentity(applicationBytes);
|
const application = parseApplicationIdentity(applicationBytes);
|
||||||
const payload: Omit<LocalServiceManagerIntent, 'intentDigest'> =
|
const payload: Omit<LocalServiceManagerIntent, 'intentDigest'> =
|
||||||
Object.freeze({
|
Object.freeze({
|
||||||
schemaVersion: 1 as const,
|
schemaVersion: command.schemaVersion,
|
||||||
kind: 'qinglong3-local-service-manager-intent' as const,
|
kind: 'qinglong3-local-service-manager-intent' as const,
|
||||||
actionId: command.request.actionId,
|
actionId: command.request.actionId,
|
||||||
action: command.request.action,
|
action: command.request.action,
|
||||||
|
|||||||
@@ -92,6 +92,32 @@ test('normalizes exact systemd fresh and OpenRC adopted intents', () => {
|
|||||||
),
|
),
|
||||||
});
|
});
|
||||||
assert.deepEqual(normalizeLocalServiceManagerIntent(openrc), openrc);
|
assert.deepEqual(normalizeLocalServiceManagerIntent(openrc), openrc);
|
||||||
|
|
||||||
|
const completedRestart = intent({
|
||||||
|
schemaVersion: 2,
|
||||||
|
actionId: '123e4567-e89b-42d3-a456-426614174003',
|
||||||
|
action: 'restart',
|
||||||
|
lineage: {
|
||||||
|
mode: 'adopted',
|
||||||
|
cutoverId: 'router-edge-1-cutover',
|
||||||
|
generation: 2,
|
||||||
|
expectedActivationDigest: 'c'.repeat(64),
|
||||||
|
previousRecordDigest: 'd'.repeat(64),
|
||||||
|
completionFence: {
|
||||||
|
expectedInstanceHeadDigest: '1'.repeat(64),
|
||||||
|
expectedCompletionDigest: '2'.repeat(64),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
outcomePath: path.join(
|
||||||
|
'/opt/qinglong3',
|
||||||
|
'service/service-manager-outcomes',
|
||||||
|
'123e4567-e89b-42d3-a456-426614174003.json',
|
||||||
|
),
|
||||||
|
});
|
||||||
|
assert.deepEqual(
|
||||||
|
normalizeLocalServiceManagerIntent(completedRestart),
|
||||||
|
completedRestart,
|
||||||
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
test('rejects arbitrary destinations, root drift, digest drift and unknown fields', () => {
|
test('rejects arbitrary destinations, root drift, digest drift and unknown fields', () => {
|
||||||
@@ -127,6 +153,30 @@ test('rejects arbitrary destinations, root drift, digest drift and unknown field
|
|||||||
previousRecordDigest: 'd'.repeat(64),
|
previousRecordDigest: 'd'.repeat(64),
|
||||||
},
|
},
|
||||||
}),
|
}),
|
||||||
|
intent({
|
||||||
|
schemaVersion: 2,
|
||||||
|
action: 'restart',
|
||||||
|
lineage: {
|
||||||
|
mode: 'adopted',
|
||||||
|
cutoverId: 'router-edge-1-cutover',
|
||||||
|
generation: 2,
|
||||||
|
expectedActivationDigest: 'c'.repeat(64),
|
||||||
|
previousRecordDigest: 'd'.repeat(64),
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
intent({
|
||||||
|
lineage: {
|
||||||
|
mode: 'adopted',
|
||||||
|
cutoverId: 'router-edge-1-cutover',
|
||||||
|
generation: 2,
|
||||||
|
expectedActivationDigest: 'c'.repeat(64),
|
||||||
|
previousRecordDigest: 'd'.repeat(64),
|
||||||
|
completionFence: {
|
||||||
|
expectedInstanceHeadDigest: '1'.repeat(64),
|
||||||
|
expectedCompletionDigest: '2'.repeat(64),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}),
|
||||||
]) {
|
]) {
|
||||||
assert.throws(
|
assert.throws(
|
||||||
() => normalizeLocalServiceManagerIntent(candidate),
|
() => normalizeLocalServiceManagerIntent(candidate),
|
||||||
|
|||||||
@@ -403,6 +403,110 @@ async function stopAdoptedTarget(state, suffix = '031') {
|
|||||||
return { stoppedResult, head };
|
return { stoppedResult, head };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function completeStoppedServiceTarget(state, suffix = '061') {
|
||||||
|
const active = prepare(
|
||||||
|
state,
|
||||||
|
1,
|
||||||
|
'install-enable-start',
|
||||||
|
state.commitmentDigest,
|
||||||
|
`123e4567-e89b-42d3-a456-426614174${suffix}`,
|
||||||
|
);
|
||||||
|
publishOutcome(active, 'install-enable-start', 'active', 6123, 1786416000200);
|
||||||
|
const startupReceiptDigest = publishReceipt(state, 6123, '100061');
|
||||||
|
const activeResult = await consumeLocalServiceManagerCutoverOutcome(
|
||||||
|
consumeCommand(state, active),
|
||||||
|
{ procRoot: state.procRoot },
|
||||||
|
);
|
||||||
|
const stopped = prepare(
|
||||||
|
state,
|
||||||
|
1,
|
||||||
|
'stop',
|
||||||
|
activeResult.recordDigest,
|
||||||
|
`123e4567-e89b-42d3-a456-426614174${String(Number(suffix) + 1).padStart(
|
||||||
|
3,
|
||||||
|
'0',
|
||||||
|
)}`,
|
||||||
|
);
|
||||||
|
publishOutcome(stopped, 'stop', 'stopped', 0, 1786416000300);
|
||||||
|
publishShutdownReceipt(state, 6123, '100061', startupReceiptDigest);
|
||||||
|
fs.rmSync(path.join(state.procRoot, '6123'), {
|
||||||
|
recursive: true,
|
||||||
|
force: true,
|
||||||
|
});
|
||||||
|
const stoppedResult = await consumeLocalServiceManagerCutoverOutcome(
|
||||||
|
consumeCommand(state, stopped),
|
||||||
|
{ procRoot: state.procRoot },
|
||||||
|
);
|
||||||
|
const states = [
|
||||||
|
'reconciliation_capture_prepared',
|
||||||
|
'reconciliation_captured',
|
||||||
|
'reconciliation_plan_prepared',
|
||||||
|
'reconciliation_planned',
|
||||||
|
'reconciliation_review_prepared',
|
||||||
|
'reconciliation_reviewed',
|
||||||
|
'reconciliation_application_prepared',
|
||||||
|
'reconciliation_application_planned',
|
||||||
|
'reconciliation_completed',
|
||||||
|
];
|
||||||
|
let completedHead;
|
||||||
|
for (const [index, nextState] of states.entries()) {
|
||||||
|
completedHead = advanceLocalCutoverInstanceHead(
|
||||||
|
{
|
||||||
|
options: { deploymentRoot: state.root },
|
||||||
|
request: {
|
||||||
|
cutoverId: state.cutoverId,
|
||||||
|
profile: 'edge',
|
||||||
|
instanceId: 'edge-router-1',
|
||||||
|
expectedActivationDigest: state.activationDigest,
|
||||||
|
requestedAtMs: 1786416000400 + index,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
process.getuid(),
|
||||||
|
nextState,
|
||||||
|
1,
|
||||||
|
index === states.length - 1
|
||||||
|
? sha256('service-manager-completion-receipt')
|
||||||
|
: sha256(`service-manager-${nextState}`),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return { activeResult, stoppedResult, completedHead };
|
||||||
|
}
|
||||||
|
|
||||||
|
function prepareCompletionRestart(
|
||||||
|
state,
|
||||||
|
previousRecordDigest,
|
||||||
|
completedHead,
|
||||||
|
actionId,
|
||||||
|
lineageOverrides = {},
|
||||||
|
) {
|
||||||
|
return prepareLocalServiceManagerIntent({
|
||||||
|
schemaVersion: 2,
|
||||||
|
operation: 'local.deployment.service-manager.intent.prepare',
|
||||||
|
options: {
|
||||||
|
deploymentRoot: state.root,
|
||||||
|
allowRootService: process.getuid() === 0,
|
||||||
|
},
|
||||||
|
request: {
|
||||||
|
actionId,
|
||||||
|
action: 'restart',
|
||||||
|
serviceKind: 'systemd',
|
||||||
|
lineage: {
|
||||||
|
mode: 'adopted',
|
||||||
|
cutoverId: state.cutoverId,
|
||||||
|
generation: 2,
|
||||||
|
expectedActivationDigest: state.activationDigest,
|
||||||
|
previousRecordDigest,
|
||||||
|
completionFence: {
|
||||||
|
expectedInstanceHeadDigest: completedHead.headDigest,
|
||||||
|
expectedCompletionDigest: completedHead.sourceRecordDigest,
|
||||||
|
},
|
||||||
|
...lineageOverrides,
|
||||||
|
},
|
||||||
|
requestedAtMs: 1786416000600,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
function rollbackPrepareCommand(state, stoppedResult, head) {
|
function rollbackPrepareCommand(state, stoppedResult, head) {
|
||||||
return {
|
return {
|
||||||
schemaVersion: 1,
|
schemaVersion: 1,
|
||||||
@@ -646,6 +750,165 @@ test('restart cannot reuse the previous generation startup receipt', async (t) =
|
|||||||
assert.equal(head.state, 'manual_required');
|
assert.equal(head.state, 'manual_required');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('restarts a completed service lineage with dual evidence and replays record-first response loss', async (t) => {
|
||||||
|
const state = fixture(t);
|
||||||
|
const { activeResult, completedHead } = await completeStoppedServiceTarget(
|
||||||
|
state,
|
||||||
|
);
|
||||||
|
const restart = prepareCompletionRestart(
|
||||||
|
state,
|
||||||
|
activeResult.recordDigest,
|
||||||
|
completedHead,
|
||||||
|
'123e4567-e89b-42d3-a456-426614174063',
|
||||||
|
);
|
||||||
|
publishOutcome(restart, 'restart', 'active', 7123, 1786416000700);
|
||||||
|
const startupReceiptDigest = publishReceipt(state, 7123, '100063');
|
||||||
|
const command = consumeCommand(state, restart);
|
||||||
|
let injected = false;
|
||||||
|
await assert.rejects(
|
||||||
|
() =>
|
||||||
|
consumeLocalServiceManagerCutoverOutcome(command, {
|
||||||
|
procRoot: state.procRoot,
|
||||||
|
afterRecordPublished() {
|
||||||
|
if (!injected) {
|
||||||
|
injected = true;
|
||||||
|
throw new Error('simulated service record response loss');
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
/simulated service record response loss/,
|
||||||
|
);
|
||||||
|
const pendingHead = readLocalCutoverInstanceHead(
|
||||||
|
state.root,
|
||||||
|
'edge-router-1',
|
||||||
|
process.getuid(),
|
||||||
|
);
|
||||||
|
assert.equal(pendingHead.state, 'reconciliation_completed');
|
||||||
|
assert.equal(pendingHead.headDigest, completedHead.headDigest);
|
||||||
|
|
||||||
|
const record = JSON.parse(
|
||||||
|
fs.readFileSync(
|
||||||
|
path.join(
|
||||||
|
state.root,
|
||||||
|
'service',
|
||||||
|
'cutovers',
|
||||||
|
state.cutoverId,
|
||||||
|
'service-manager-g02-active.json',
|
||||||
|
),
|
||||||
|
'utf8',
|
||||||
|
),
|
||||||
|
);
|
||||||
|
assert.equal(record.schemaVersion, 3);
|
||||||
|
assert.equal(record.evidence.startupReceiptDigest, startupReceiptDigest);
|
||||||
|
assert.deepEqual(record.evidence.completionFence, {
|
||||||
|
expectedInstanceHeadDigest: completedHead.headDigest,
|
||||||
|
expectedCompletionDigest: completedHead.sourceRecordDigest,
|
||||||
|
});
|
||||||
|
|
||||||
|
const replay = await consumeLocalServiceManagerCutoverOutcome(command, {
|
||||||
|
procRoot: state.procRoot,
|
||||||
|
});
|
||||||
|
assert.equal(replay.status, 'existing');
|
||||||
|
assert.equal(replay.state, 'target_active');
|
||||||
|
const activeHead = readLocalCutoverInstanceHead(
|
||||||
|
state.root,
|
||||||
|
'edge-router-1',
|
||||||
|
process.getuid(),
|
||||||
|
);
|
||||||
|
assert.equal(activeHead.state, 'target_active');
|
||||||
|
assert.equal(activeHead.generation, 2);
|
||||||
|
assert.equal(activeHead.sourceRecordDigest, replay.recordDigest);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('rejects completed service restart without both the prior record and exact completion head', async (t) => {
|
||||||
|
const state = fixture(t);
|
||||||
|
const { activeResult, completedHead } = await completeStoppedServiceTarget(
|
||||||
|
state,
|
||||||
|
'071',
|
||||||
|
);
|
||||||
|
assert.throws(
|
||||||
|
() =>
|
||||||
|
prepare(
|
||||||
|
state,
|
||||||
|
2,
|
||||||
|
'restart',
|
||||||
|
activeResult.recordDigest,
|
||||||
|
'123e4567-e89b-42d3-a456-426614174073',
|
||||||
|
),
|
||||||
|
/lost the instance lineage compare-and-swap/,
|
||||||
|
);
|
||||||
|
assert.throws(
|
||||||
|
() =>
|
||||||
|
prepareCompletionRestart(
|
||||||
|
state,
|
||||||
|
activeResult.recordDigest,
|
||||||
|
completedHead,
|
||||||
|
'123e4567-e89b-42d3-a456-426614174074',
|
||||||
|
{
|
||||||
|
completionFence: {
|
||||||
|
expectedInstanceHeadDigest: completedHead.headDigest,
|
||||||
|
expectedCompletionDigest: '0'.repeat(64),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
),
|
||||||
|
/completion restart lost the instance head compare-and-swap/,
|
||||||
|
);
|
||||||
|
assert.throws(
|
||||||
|
() =>
|
||||||
|
prepareCompletionRestart(
|
||||||
|
state,
|
||||||
|
'0'.repeat(64),
|
||||||
|
completedHead,
|
||||||
|
'123e4567-e89b-42d3-a456-426614174075',
|
||||||
|
),
|
||||||
|
/completion restart lost the previous active record/,
|
||||||
|
);
|
||||||
|
const unchanged = readLocalCutoverInstanceHead(
|
||||||
|
state.root,
|
||||||
|
'edge-router-1',
|
||||||
|
process.getuid(),
|
||||||
|
);
|
||||||
|
assert.equal(unchanged.headDigest, completedHead.headDigest);
|
||||||
|
assert.equal(unchanged.state, 'reconciliation_completed');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('terminalizes and replays a completed service restart without a new startup receipt', async (t) => {
|
||||||
|
const state = fixture(t);
|
||||||
|
const { activeResult, completedHead } = await completeStoppedServiceTarget(
|
||||||
|
state,
|
||||||
|
'081',
|
||||||
|
);
|
||||||
|
const restart = prepareCompletionRestart(
|
||||||
|
state,
|
||||||
|
activeResult.recordDigest,
|
||||||
|
completedHead,
|
||||||
|
'123e4567-e89b-42d3-a456-426614174083',
|
||||||
|
);
|
||||||
|
publishOutcome(restart, 'restart', 'active', 8123, 1786416000700);
|
||||||
|
let clock = 0;
|
||||||
|
const command = consumeCommand(state, restart);
|
||||||
|
const result = await consumeLocalServiceManagerCutoverOutcome(command, {
|
||||||
|
procRoot: state.procRoot,
|
||||||
|
now: () => clock,
|
||||||
|
wait: async (milliseconds) => {
|
||||||
|
clock += milliseconds;
|
||||||
|
},
|
||||||
|
});
|
||||||
|
assert.equal(result.state, 'manual_required');
|
||||||
|
const terminal = readLocalCutoverInstanceHead(
|
||||||
|
state.root,
|
||||||
|
'edge-router-1',
|
||||||
|
process.getuid(),
|
||||||
|
);
|
||||||
|
assert.equal(terminal.state, 'manual_required');
|
||||||
|
assert.equal(terminal.sourceRecordDigest, result.recordDigest);
|
||||||
|
const replay = await consumeLocalServiceManagerCutoverOutcome(command, {
|
||||||
|
procRoot: state.procRoot,
|
||||||
|
});
|
||||||
|
assert.equal(replay.status, 'existing');
|
||||||
|
assert.equal(replay.state, 'manual_required');
|
||||||
|
});
|
||||||
|
|
||||||
test('stop advances only after the exact receipted process identity disappears', async (t) => {
|
test('stop advances only after the exact receipted process identity disappears', async (t) => {
|
||||||
const state = fixture(t);
|
const state = fixture(t);
|
||||||
const first = prepare(
|
const first = prepare(
|
||||||
|
|||||||
@@ -49,7 +49,11 @@ async function preparedOutput(artifactRoot, offerId = 'offer-posix-1') {
|
|||||||
const allocator = new WorkerFileLogArtifactAllocator({
|
const allocator = new WorkerFileLogArtifactAllocator({
|
||||||
root: artifactRoot,
|
root: artifactRoot,
|
||||||
policy: workerRemoteLogArtifactPolicy('edge'),
|
policy: workerRemoteLogArtifactPolicy('edge'),
|
||||||
capacity: { async availableBytes() { return 1024n ** 4n; } },
|
capacity: {
|
||||||
|
async availableBytes() {
|
||||||
|
return 1024n ** 4n;
|
||||||
|
},
|
||||||
|
},
|
||||||
});
|
});
|
||||||
const prepared = await allocator.prepare({
|
const prepared = await allocator.prepare({
|
||||||
projectId: 'project-1',
|
projectId: 'project-1',
|
||||||
@@ -84,7 +88,7 @@ function launch(prepared, output, overrides = {}) {
|
|||||||
|
|
||||||
async function waitForReceipt(root) {
|
async function waitForReceipt(root) {
|
||||||
const store = new CompletionReceiptFileStore(root);
|
const store = new CompletionReceiptFileStore(root);
|
||||||
for (let attempt = 0; attempt < 100; attempt += 1) {
|
for (let attempt = 0; attempt < 500; attempt += 1) {
|
||||||
const receipt = await store.read(ATTEMPT_ID);
|
const receipt = await store.read(ATTEMPT_ID);
|
||||||
if (receipt) return receipt;
|
if (receipt) return receipt;
|
||||||
await new Promise((resolve) => setTimeout(resolve, 10));
|
await new Promise((resolve) => setTimeout(resolve, 10));
|
||||||
@@ -97,7 +101,11 @@ test('verifies the Worker barrier, launches through the reviewed fd and writes a
|
|||||||
const { prepared, output } = await preparedOutput(roots.artifactRoot);
|
const { prepared, output } = await preparedOutput(roots.artifactRoot);
|
||||||
let barrier;
|
let barrier;
|
||||||
const executor = new WorkerPosixExecutionExecutor({
|
const executor = new WorkerPosixExecutionExecutor({
|
||||||
barrier: { async verify(input) { barrier = input; } },
|
barrier: {
|
||||||
|
async verify(input) {
|
||||||
|
barrier = input;
|
||||||
|
},
|
||||||
|
},
|
||||||
receiptRoot: roots.receiptRoot,
|
receiptRoot: roots.receiptRoot,
|
||||||
identityProvider: identityProvider(),
|
identityProvider: identityProvider(),
|
||||||
clock: { now: () => 100 },
|
clock: { now: () => 100 },
|
||||||
@@ -128,13 +136,19 @@ test('does not spawn when the durable Worker barrier rejects authority', async (
|
|||||||
const marker = path.join(roots.root, 'spawned');
|
const marker = path.join(roots.root, 'spawned');
|
||||||
const { prepared, output } = await preparedOutput(roots.artifactRoot);
|
const { prepared, output } = await preparedOutput(roots.artifactRoot);
|
||||||
const executor = new WorkerPosixExecutionExecutor({
|
const executor = new WorkerPosixExecutionExecutor({
|
||||||
barrier: { async verify() { throw new Error('stale inbox'); } },
|
barrier: {
|
||||||
|
async verify() {
|
||||||
|
throw new Error('stale inbox');
|
||||||
|
},
|
||||||
|
},
|
||||||
receiptRoot: roots.receiptRoot,
|
receiptRoot: roots.receiptRoot,
|
||||||
identityProvider: identityProvider(),
|
identityProvider: identityProvider(),
|
||||||
});
|
});
|
||||||
const result = await executor.start(launch(prepared, output, {
|
const result = await executor.start(
|
||||||
|
launch(prepared, output, {
|
||||||
command: { kind: 'argv', file: '/usr/bin/touch', args: [marker] },
|
command: { kind: 'argv', file: '/usr/bin/touch', args: [marker] },
|
||||||
}));
|
}),
|
||||||
|
);
|
||||||
assert.deepEqual(result, { status: 'rejected' });
|
assert.deepEqual(result, { status: 'rejected' });
|
||||||
await assert.rejects(fs.stat(marker), { code: 'ENOENT' });
|
await assert.rejects(fs.stat(marker), { code: 'ENOENT' });
|
||||||
});
|
});
|
||||||
@@ -144,13 +158,19 @@ test('rejects timeout without durable control-plane deadline before spawn', asyn
|
|||||||
const { prepared, output } = await preparedOutput(roots.artifactRoot);
|
const { prepared, output } = await preparedOutput(roots.artifactRoot);
|
||||||
let barriers = 0;
|
let barriers = 0;
|
||||||
const executor = new WorkerPosixExecutionExecutor({
|
const executor = new WorkerPosixExecutionExecutor({
|
||||||
barrier: { async verify() { barriers += 1; } },
|
barrier: {
|
||||||
|
async verify() {
|
||||||
|
barriers += 1;
|
||||||
|
},
|
||||||
|
},
|
||||||
receiptRoot: roots.receiptRoot,
|
receiptRoot: roots.receiptRoot,
|
||||||
identityProvider: identityProvider(),
|
identityProvider: identityProvider(),
|
||||||
});
|
});
|
||||||
const result = await executor.start(launch(prepared, output, {
|
const result = await executor.start(
|
||||||
|
launch(prepared, output, {
|
||||||
timeoutMs: 1_000,
|
timeoutMs: 1_000,
|
||||||
}));
|
}),
|
||||||
|
);
|
||||||
assert.deepEqual(result, { status: 'rejected' });
|
assert.deepEqual(result, { status: 'rejected' });
|
||||||
assert.equal(barriers, 0);
|
assert.equal(barriers, 0);
|
||||||
});
|
});
|
||||||
@@ -160,14 +180,20 @@ test('accepts timeout only when starting ACK supplied a durable deadline', async
|
|||||||
const { prepared, output } = await preparedOutput(roots.artifactRoot);
|
const { prepared, output } = await preparedOutput(roots.artifactRoot);
|
||||||
let barriers = 0;
|
let barriers = 0;
|
||||||
const executor = new WorkerPosixExecutionExecutor({
|
const executor = new WorkerPosixExecutionExecutor({
|
||||||
barrier: { async verify() { barriers += 1; } },
|
barrier: {
|
||||||
|
async verify() {
|
||||||
|
barriers += 1;
|
||||||
|
},
|
||||||
|
},
|
||||||
receiptRoot: roots.receiptRoot,
|
receiptRoot: roots.receiptRoot,
|
||||||
identityProvider: identityProvider(),
|
identityProvider: identityProvider(),
|
||||||
});
|
});
|
||||||
const result = await executor.start(launch(prepared, output, {
|
const result = await executor.start(
|
||||||
|
launch(prepared, output, {
|
||||||
timeoutMs: 1_000,
|
timeoutMs: 1_000,
|
||||||
executionDeadlineAtMs: 2_000,
|
executionDeadlineAtMs: 2_000,
|
||||||
}));
|
}),
|
||||||
|
);
|
||||||
assert.equal(result.status, 'started');
|
assert.equal(result.status, 'started');
|
||||||
assert.equal(barriers, 1);
|
assert.equal(barriers, 1);
|
||||||
await waitForReceipt(roots.receiptRoot);
|
await waitForReceipt(roots.receiptRoot);
|
||||||
@@ -180,14 +206,20 @@ test('propagates unknown outcome when durable identity capture fails after spawn
|
|||||||
barrier: { async verify() {} },
|
barrier: { async verify() {} },
|
||||||
receiptRoot: roots.receiptRoot,
|
receiptRoot: roots.receiptRoot,
|
||||||
identityProvider: {
|
identityProvider: {
|
||||||
async capture() { throw new Error('procfs unavailable'); },
|
async capture() {
|
||||||
async inspect() { return { status: 'unknown' }; },
|
throw new Error('procfs unavailable');
|
||||||
|
},
|
||||||
|
async inspect() {
|
||||||
|
return { status: 'unknown' };
|
||||||
|
},
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
await assert.rejects(
|
await assert.rejects(
|
||||||
executor.start(launch(prepared, output, {
|
executor.start(
|
||||||
|
launch(prepared, output, {
|
||||||
command: { kind: 'shell', command: 'sleep 5', shell: '/bin/sh' },
|
command: { kind: 'shell', command: 'sleep 5', shell: '/bin/sh' },
|
||||||
})),
|
}),
|
||||||
|
),
|
||||||
(error) => error?.spawnOutcome === 'unknown',
|
(error) => error?.spawnOutcome === 'unknown',
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|||||||
Reference in New Issue
Block a user