feat(ql3): establish 3.0 incubation baseline

This commit is contained in:
whyour
2026-08-12 00:25:26 +08:00
parent 4bf92dcfeb
commit c699c32461
2817 changed files with 779642 additions and 653 deletions
+133
View File
@@ -0,0 +1,133 @@
# `@qinglong/cluster-admin`
This private QingLong 3.0 package is a short-lived cluster administration
authority. It is intentionally separate from the resident `cluster-control`
artifact and requires a distinct PostgreSQL role.
The admin role can append Identity/API Credential mutations and their security
audit in one serializable transaction, and can perform bounded read-only audit
queries. It cannot read or mutate Run, Project or RoleBinding data, cannot
update/delete credential history, and cannot delete audit events.
Prompt output lifecycle work remains outside the resident control plane. The
explicit `prompt-output-gc-process` runs one bounded retention page, while
`prompt-output-key-retirement-process` coordinates one durable PostgreSQL
prepare/material-delete/complete operation through an injected material
authority. The latter intentionally does not select or dynamically load a KMS:
each deployment must bind a reviewed KMS/HSM adapter, and the process returns
only content-free identities, digests, status, and database time.
Credential issue/rotation returns the bearer token only for a newly inserted
mutation. Exact mutation replay returns `token: null`; a caller that loses the
one-time response must rotate again instead of recovering secret material from
storage. Replay equality covers caller-controlled semantics (actor,
authentication/request/mutation identity, subject, version, operation and
explicit absolute `notBeforeAtMs`/`expiresAtMs`), while generated secret/digest
and server timestamps are excluded. A collision in any semantic field fails
closed.
Production Worker credential rollout must use the explicit
`@qinglong/cluster-admin/worker-credential-delivery` subpath instead of exposing
that one-time response. It stages a new credential ID before one atomic
credential/mutation/audit/delivery commit, appends publication acknowledgement,
and never returns the token. The current slice is intentionally short-lived and
timer-free. Authenticated Session observation is appended in the Session
transaction; a bounded PostgreSQL-timed recovery page can resume publication,
wait for observation, and atomically revoke the previous credential while
appending delivery v4. The explicit
`@qinglong/cluster-admin/worker-credential-file-delivery` subpath supplies a
bounded POSIX implementation for Docker bind mounts, systemd deployments and
controlled single-writer volumes: private no-replace stages, generation-fenced
atomic token replacement, durable directory synchronization and bounded stage
enumeration. Capability v15 also provides a database-authorized orphan cleanup:
the inventory coordinator must append an exact immutable discard tombstone
before deleting a stage, while credential commit and discard authorization share
one PostgreSQL advisory transaction lock so exactly one can win. Authorization
and unfinished cleanup are recovered through separate bounded pages without a
resident timer. The explicit
`@qinglong/cluster-admin/worker-credential-kubernetes-delivery` subpath provides
the multi-writer deployment adapter without adding another package: immutable
per-delivery staging Secrets, target updates fenced by the resourceVersion from
the preceding GET, exact replay after 409, and UID + resourceVersion deletion
preconditions. It uses the exact-pinned official Kubernetes client through a
narrow injected Secret API, owns no watcher/timer/cache, and fails closed when
the stage inventory exceeds 128 objects; cleanup pages remain capped at 64.
A real k3s 1.34 API-server test with a dedicated ServiceAccount proves the
get/list/create/update/delete-only RBAC matrix, concurrent update single winner,
and preconditioned delete. Kubernetes HA/control-plane failover, stale-lock
repair for the POSIX adapter, PostgreSQL 18/failover evidence, and product-facing
administration remain gated.
Plugin Package recovery is exposed only through
`plugin-package-oci-stage`, `plugin-package-recovery` and
`plugin-package-recovery-process`. The process is the entrypoint for a
short-lived recovery Job; it opens one `ql3_package_executor` Pool, proves the
exact executor schema/role
readiness, serially converges a bounded recovery cycle, closes PostgreSQL and
exits. It is not imported by `cluster-control`.
The OCI stage authority implements the OCI Distribution API without another
workspace package. A PackageLock binds the OCI manifest digest in its locator
and independently binds the single bundle layer digest and byte length. The
resolver accepts only an explicit registry allowlist, HTTPS, no redirects and
no ambient credential provider. It validates the package manifest/config,
retrieves exactly one lock-annotated OCI referrer, verifies its Ed25519
publisher signature, and streams the bundle through the common inspector.
The deterministic evidence can be re-resolved after process restart; a bounded
64-entry cache only avoids fetching the same evidence twice during one Job.
The production Kustomize operation gives its dedicated ServiceAccount only
namespace-scoped ConfigMap `get/create/update`. It receives neither Secret API
access nor runtime/migration database credentials. A dedicated Kind 1.32.8
live gate runs two non-root, read-only-rootfs admin-image Pods with real
projected ServiceAccount tokens. It proves one API-confirmed create response
loss converges through durable inspect without a second create, and that two
processes attempting the same `resourceVersion` produce one winner and one
conflict. ConfigMap list/delete, Secret access and cross-namespace reads are
all denied. The response-loss injection is at the client boundary after the
API confirms create; it is not raw-wire packet loss or Kubernetes
control-plane HA evidence. Private registry authentication and package
resource-generation consumers remain gated.
Cluster Package management authentication is exposed only through the
`plugin-package-identity-assertion` and
`plugin-package-management-transport` subpaths. The identity verifier accepts
only a dedicated compact JWS purpose with one canonical HTTPS issuer, one exact
audience, an explicitly reviewed public-key set and deployment-specific
ACR-plus-required-AMR mappings to `multi_factor` or `hardware`. It supports
Ed25519, P-256 and bounded RSA public keys through Node 24 built-in crypto and
adds no dependency. Raw assertions and `jti` values do not enter management
requests or results. The shared verifier/file implementation is now composed
by two authority-specific processes: Plugin Package management and Worker
credential management. The Worker process is available only through
`worker-credential-management-process`; it opens the dedicated
`worker-credential-manager` PostgreSQL role, validates the v49 role contract,
pins keyset generations under authority `worker-credential-management`, uses
database-clock durable quota before management state reads, then exposes only
the fixed TLS 1.3 Worker management route. It does not import Worker delivery,
TokenRequest or executor authority. Its disabled gate opens no files, listener
or database, and the Kubernetes operation remains explicit opt-in.
The three-node K3s management gate runs two required-anti-affinity replicas and
the production TLS 1.3 client against each exact Pod. It proves shared durable
quota (8 admitted/8 limited), cross-Pod semantic replay, identity generation
overlap/revocation/rollback, and database-failure readiness withdrawal followed
by fresh-Pod-only recovery. Plan replay equality covers caller-controlled plan
semantics; server-authored plan times and their derived digests return the
original stored plan, while target or requester drift remains a conflict.
Approved Worker delivery execution is exposed separately through
`worker-credential-executor-process` and `ql3-worker-credential-execute`. The
process accepts one exact non-secret command file, a private canonical 32-byte
pepper file, the dedicated executor PostgreSQL role and one in-cluster issuer
session. It confirms the exact TokenRequest authorization before and during
execution, delegates the 8-allow/20-deny delivery proof to the restricted
session, clears issuer client credentials and exits after one result. The
Kubernetes operation is a caller-created `Job`, never a `CronJob` or resident
controller; its base NetworkPolicy remains DNS-only until a private overlay
adds the exact API-server and database destinations.
A real K3s `v1.34.3+k3s1` gate runs this production image and process: the first
Job publishes generation 3 with one 600-second TokenRequest, while a second
independent Job replays the exact command as `existing` without another token
request. Both use the executor-only one-connection PostgreSQL role,
`backoffLimit=0`, no automounted token and exact API/PostgreSQL egress.
+351
View File
@@ -0,0 +1,351 @@
{
"name": "@qinglong/cluster-admin",
"version": "3.0.0-alpha.0",
"private": true,
"description": "QingLong 3.0 short-lived cluster administration authority",
"license": "Apache-2.0",
"engines": {
"node": ">=24.18.0 <25"
},
"main": "dist/application-runtime/clusterAdminRuntime.js",
"types": "dist/application-runtime/clusterAdminRuntime.d.ts",
"exports": {
".": {
"types": "./dist/application-runtime/clusterAdminRuntime.d.ts",
"require": "./dist/application-runtime/clusterAdminRuntime.js",
"default": "./dist/application-runtime/clusterAdminRuntime.js"
},
"./approval-decision-management": {
"types": "./dist/approval-management/approvalDecisionManagement.d.ts",
"require": "./dist/approval-management/approvalDecisionManagement.js",
"default": "./dist/approval-management/approvalDecisionManagement.js"
},
"./approval-management": {
"types": "./dist/approval-management/approvalManagement.d.ts",
"require": "./dist/approval-management/approvalManagement.js",
"default": "./dist/approval-management/approvalManagement.js"
},
"./approval-management-transport": {
"types": "./dist/approval-management/approvalManagementTransport.d.ts",
"require": "./dist/approval-management/approvalManagementTransport.js",
"default": "./dist/approval-management/approvalManagementTransport.js"
},
"./approval-management-http": {
"types": "./dist/approval-management/approvalManagementHttp.d.ts",
"require": "./dist/approval-management/approvalManagementHttp.js",
"default": "./dist/approval-management/approvalManagementHttp.js"
},
"./approval-management-process": {
"types": "./dist/approval-management/approvalManagementProcess.d.ts",
"require": "./dist/approval-management/approvalManagementProcess.js",
"default": "./dist/approval-management/approvalManagementProcess.js"
},
"./approval-management-client": {
"types": "./dist/approval-management/approvalManagementClient.d.ts",
"require": "./dist/approval-management/approvalManagementClient.js",
"default": "./dist/approval-management/approvalManagementClient.js"
},
"./administration": {
"types": "./dist/security-administration/clusterAdministration.d.ts",
"require": "./dist/security-administration/clusterAdministration.js",
"default": "./dist/security-administration/clusterAdministration.js"
},
"./automation-management": {
"types": "./dist/automation-management/automationManagement.d.ts",
"require": "./dist/automation-management/automationManagement.js",
"default": "./dist/automation-management/automationManagement.js"
},
"./model-provider-credential-management": {
"types": "./dist/model-provider-credential/modelProviderCredentialManagement.d.ts",
"require": "./dist/model-provider-credential/modelProviderCredentialManagement.js",
"default": "./dist/model-provider-credential/modelProviderCredentialManagement.js"
},
"./model-provider-credential-management-transport": {
"types": "./dist/model-provider-credential/modelProviderCredentialManagementTransport.d.ts",
"require": "./dist/model-provider-credential/modelProviderCredentialManagementTransport.js",
"default": "./dist/model-provider-credential/modelProviderCredentialManagementTransport.js"
},
"./model-provider-credential-management-http": {
"types": "./dist/model-provider-credential/modelProviderCredentialManagementHttp.d.ts",
"require": "./dist/model-provider-credential/modelProviderCredentialManagementHttp.js",
"default": "./dist/model-provider-credential/modelProviderCredentialManagementHttp.js"
},
"./model-provider-credential-management-process": {
"types": "./dist/model-provider-credential/modelProviderCredentialManagementProcess.d.ts",
"require": "./dist/model-provider-credential/modelProviderCredentialManagementProcess.js",
"default": "./dist/model-provider-credential/modelProviderCredentialManagementProcess.js"
},
"./model-provider-credential-management-client": {
"types": "./dist/model-provider-credential/modelProviderCredentialManagementClient.d.ts",
"require": "./dist/model-provider-credential/modelProviderCredentialManagementClient.js",
"default": "./dist/model-provider-credential/modelProviderCredentialManagementClient.js"
},
"./model-provider-credential-test-executor": {
"types": "./dist/model-provider-credential/modelProviderCredentialTestExecutor.d.ts",
"require": "./dist/model-provider-credential/modelProviderCredentialTestExecutor.js",
"default": "./dist/model-provider-credential/modelProviderCredentialTestExecutor.js"
},
"./model-provider-credential-test-executor-process": {
"types": "./dist/model-provider-credential/modelProviderCredentialTestExecutorProcess.d.ts",
"require": "./dist/model-provider-credential/modelProviderCredentialTestExecutorProcess.js",
"default": "./dist/model-provider-credential/modelProviderCredentialTestExecutorProcess.js"
},
"./automation-management-transport": {
"types": "./dist/automation-management/automationManagementTransport.d.ts",
"require": "./dist/automation-management/automationManagementTransport.js",
"default": "./dist/automation-management/automationManagementTransport.js"
},
"./automation-management-http": {
"types": "./dist/automation-management/automationManagementHttp.d.ts",
"require": "./dist/automation-management/automationManagementHttp.js",
"default": "./dist/automation-management/automationManagementHttp.js"
},
"./automation-management-process": {
"types": "./dist/automation-management/automationManagementProcess.d.ts",
"require": "./dist/automation-management/automationManagementProcess.js",
"default": "./dist/automation-management/automationManagementProcess.js"
},
"./automation-management-client": {
"types": "./dist/automation-management/automationManagementClient.d.ts",
"require": "./dist/automation-management/automationManagementClient.js",
"default": "./dist/automation-management/automationManagementClient.js"
},
"./worker-credential-delivery": {
"types": "./dist/worker-credential/workerCredentialDelivery.d.ts",
"require": "./dist/worker-credential/workerCredentialDelivery.js",
"default": "./dist/worker-credential/workerCredentialDelivery.js"
},
"./worker-credential-file-delivery": {
"types": "./dist/worker-credential/workerCredentialFileDelivery.d.ts",
"require": "./dist/worker-credential/workerCredentialFileDelivery.js",
"default": "./dist/worker-credential/workerCredentialFileDelivery.js"
},
"./worker-credential-kubernetes-delivery": {
"types": "./dist/worker-credential/workerCredentialKubernetesDelivery.d.ts",
"require": "./dist/worker-credential/workerCredentialKubernetesDelivery.js",
"default": "./dist/worker-credential/workerCredentialKubernetesDelivery.js"
},
"./worker-credential-kubernetes-token-request": {
"types": "./dist/worker-credential/workerCredentialKubernetesTokenRequest.d.ts",
"require": "./dist/worker-credential/workerCredentialKubernetesTokenRequest.js",
"default": "./dist/worker-credential/workerCredentialKubernetesTokenRequest.js"
},
"./worker-credential-management": {
"types": "./dist/worker-credential/management-server/workerCredentialManagement.d.ts",
"require": "./dist/worker-credential/management-server/workerCredentialManagement.js",
"default": "./dist/worker-credential/management-server/workerCredentialManagement.js"
},
"./worker-credential-management-transport": {
"types": "./dist/worker-credential/management-server/workerCredentialManagementTransport.d.ts",
"require": "./dist/worker-credential/management-server/workerCredentialManagementTransport.js",
"default": "./dist/worker-credential/management-server/workerCredentialManagementTransport.js"
},
"./worker-credential-management-http": {
"types": "./dist/worker-credential/management-server/workerCredentialManagementHttp.d.ts",
"require": "./dist/worker-credential/management-server/workerCredentialManagementHttp.js",
"default": "./dist/worker-credential/management-server/workerCredentialManagementHttp.js"
},
"./worker-credential-management-process": {
"types": "./dist/worker-credential/management-server/workerCredentialManagementProcess.d.ts",
"require": "./dist/worker-credential/management-server/workerCredentialManagementProcess.js",
"default": "./dist/worker-credential/management-server/workerCredentialManagementProcess.js"
},
"./worker-credential-management-client": {
"types": "./dist/worker-credential/workerCredentialManagementClient.d.ts",
"require": "./dist/worker-credential/workerCredentialManagementClient.js",
"default": "./dist/worker-credential/workerCredentialManagementClient.js"
},
"./worker-credential-management-executor": {
"types": "./dist/worker-credential/workerCredentialManagementExecutor.d.ts",
"require": "./dist/worker-credential/workerCredentialManagementExecutor.js",
"default": "./dist/worker-credential/workerCredentialManagementExecutor.js"
},
"./worker-credential-executor-process": {
"types": "./dist/worker-credential/workerCredentialExecutorProcess.d.ts",
"require": "./dist/worker-credential/workerCredentialExecutorProcess.js",
"default": "./dist/worker-credential/workerCredentialExecutorProcess.js"
},
"./plugin-package-kubernetes-activation": {
"types": "./dist/plugin-package/recovery/pluginPackageKubernetesActivation.d.ts",
"require": "./dist/plugin-package/recovery/pluginPackageKubernetesActivation.js",
"default": "./dist/plugin-package/recovery/pluginPackageKubernetesActivation.js"
},
"./plugin-package-recovery": {
"types": "./dist/plugin-package/recovery/pluginPackageRecovery.d.ts",
"require": "./dist/plugin-package/recovery/pluginPackageRecovery.js",
"default": "./dist/plugin-package/recovery/pluginPackageRecovery.js"
},
"./plugin-package-oci-stage": {
"types": "./dist/plugin-package/recovery/pluginPackageOciStage.d.ts",
"require": "./dist/plugin-package/recovery/pluginPackageOciStage.js",
"default": "./dist/plugin-package/recovery/pluginPackageOciStage.js"
},
"./plugin-package-recovery-process": {
"types": "./dist/plugin-package/recovery/pluginPackageRecoveryProcess.d.ts",
"require": "./dist/plugin-package/recovery/pluginPackageRecoveryProcess.js",
"default": "./dist/plugin-package/recovery/pluginPackageRecoveryProcess.js"
},
"./plugin-package-approved-action": {
"types": "./dist/plugin-package/executor/pluginPackageApprovedAction.d.ts",
"require": "./dist/plugin-package/executor/pluginPackageApprovedAction.js",
"default": "./dist/plugin-package/executor/pluginPackageApprovedAction.js"
},
"./plugin-package-executor-process": {
"types": "./dist/plugin-package/executor/pluginPackageExecutorProcess.d.ts",
"require": "./dist/plugin-package/executor/pluginPackageExecutorProcess.js",
"default": "./dist/plugin-package/executor/pluginPackageExecutorProcess.js"
},
"./prompt-output-gc-process": {
"types": "./dist/prompt-output/retention/promptOutputGcProcess.d.ts",
"require": "./dist/prompt-output/retention/promptOutputGcProcess.js",
"default": "./dist/prompt-output/retention/promptOutputGcProcess.js"
},
"./prompt-output-key-retirement-process": {
"types": "./dist/prompt-output/key-management/promptOutputKeyRetirementProcess.d.ts",
"require": "./dist/prompt-output/key-management/promptOutputKeyRetirementProcess.js",
"default": "./dist/prompt-output/key-management/promptOutputKeyRetirementProcess.js"
},
"./prompt-output-key-rotation-process": {
"types": "./dist/prompt-output/key-management/promptOutputKeyRotationProcess.d.ts",
"require": "./dist/prompt-output/key-management/promptOutputKeyRotationProcess.js",
"default": "./dist/prompt-output/key-management/promptOutputKeyRotationProcess.js"
},
"./prompt-output-external-recovery-verifier": {
"types": "./dist/prompt-output/external-recovery/promptOutputExternalRecoveryVerifier.d.ts",
"require": "./dist/prompt-output/external-recovery/promptOutputExternalRecoveryVerifier.js",
"default": "./dist/prompt-output/external-recovery/promptOutputExternalRecoveryVerifier.js"
},
"./prompt-output-kubernetes-secret-keyring": {
"types": "./dist/prompt-output/key-management/promptOutputKubernetesSecretKeyring.d.ts",
"require": "./dist/prompt-output/key-management/promptOutputKubernetesSecretKeyring.js",
"default": "./dist/prompt-output/key-management/promptOutputKubernetesSecretKeyring.js"
},
"./plugin-package-management": {
"types": "./dist/plugin-package/management/pluginPackageManagement.d.ts",
"require": "./dist/plugin-package/management/pluginPackageManagement.js",
"default": "./dist/plugin-package/management/pluginPackageManagement.js"
},
"./plugin-package-lifecycle-management": {
"types": "./dist/plugin-package/lifecycle/pluginPackageLifecycleManagement.d.ts",
"require": "./dist/plugin-package/lifecycle/pluginPackageLifecycleManagement.js",
"default": "./dist/plugin-package/lifecycle/pluginPackageLifecycleManagement.js"
},
"./plugin-package-lifecycle-executor": {
"types": "./dist/plugin-package/lifecycle/pluginPackageLifecycleExecutor.d.ts",
"require": "./dist/plugin-package/lifecycle/pluginPackageLifecycleExecutor.js",
"default": "./dist/plugin-package/lifecycle/pluginPackageLifecycleExecutor.js"
},
"./plugin-package-management-transport": {
"types": "./dist/plugin-package/management/pluginPackageManagementTransport.d.ts",
"require": "./dist/plugin-package/management/pluginPackageManagementTransport.js",
"default": "./dist/plugin-package/management/pluginPackageManagementTransport.js"
},
"./plugin-package-identity-assertion": {
"types": "./dist/management-support/pluginPackageIdentityAssertion.d.ts",
"require": "./dist/management-support/pluginPackageIdentityAssertion.js",
"default": "./dist/management-support/pluginPackageIdentityAssertion.js"
},
"./plugin-package-identity-keyset": {
"types": "./dist/management-support/pluginPackageIdentityKeyset.d.ts",
"require": "./dist/management-support/pluginPackageIdentityKeyset.js",
"default": "./dist/management-support/pluginPackageIdentityKeyset.js"
},
"./plugin-package-management-http": {
"types": "./dist/management-support/pluginPackageManagementHttp.d.ts",
"require": "./dist/management-support/pluginPackageManagementHttp.js",
"default": "./dist/management-support/pluginPackageManagementHttp.js"
},
"./plugin-package-management-process": {
"types": "./dist/plugin-package/management/pluginPackageManagementProcess.d.ts",
"require": "./dist/plugin-package/management/pluginPackageManagementProcess.js",
"default": "./dist/plugin-package/management/pluginPackageManagementProcess.js"
},
"./plugin-package-management-client": {
"types": "./dist/management-support/pluginPackageManagementClient.d.ts",
"require": "./dist/management-support/pluginPackageManagementClient.js",
"default": "./dist/management-support/pluginPackageManagementClient.js"
},
"./plugin-package-management-kubernetes-client": {
"types": "./dist/plugin-package/management/pluginPackageManagementKubernetesClient.d.ts",
"require": "./dist/plugin-package/management/pluginPackageManagementKubernetesClient.js",
"default": "./dist/plugin-package/management/pluginPackageManagementKubernetesClient.js"
},
"./plugin-package-quarantine": {
"types": "./dist/plugin-package/lifecycle/pluginPackageQuarantine.d.ts",
"require": "./dist/plugin-package/lifecycle/pluginPackageQuarantine.js",
"default": "./dist/plugin-package/lifecycle/pluginPackageQuarantine.js"
},
"./plugin-package-publisher-revocation": {
"types": "./dist/plugin-package/publisher/pluginPackagePublisherRevocation.d.ts",
"require": "./dist/plugin-package/publisher/pluginPackagePublisherRevocation.js",
"default": "./dist/plugin-package/publisher/pluginPackagePublisherRevocation.js"
},
"./plugin-package-publisher-revocation-approved-action": {
"types": "./dist/plugin-package/publisher/pluginPackagePublisherRevocationApprovedAction.d.ts",
"require": "./dist/plugin-package/publisher/pluginPackagePublisherRevocationApprovedAction.js",
"default": "./dist/plugin-package/publisher/pluginPackagePublisherRevocationApprovedAction.js"
},
"./plugin-package-publisher-revocation-approval-consumer": {
"types": "./dist/plugin-package/publisher/pluginPackagePublisherRevocationApprovalConsumer.d.ts",
"require": "./dist/plugin-package/publisher/pluginPackagePublisherRevocationApprovalConsumer.js",
"default": "./dist/plugin-package/publisher/pluginPackagePublisherRevocationApprovalConsumer.js"
},
"./plugin-package-publisher-trust-transition-approved-action": {
"types": "./dist/plugin-package/publisher/pluginPackagePublisherTrustTransitionApprovedAction.d.ts",
"require": "./dist/plugin-package/publisher/pluginPackagePublisherTrustTransitionApprovedAction.js",
"default": "./dist/plugin-package/publisher/pluginPackagePublisherTrustTransitionApprovedAction.js"
},
"./plugin-package-publisher-trust-transition-approval-consumer": {
"types": "./dist/plugin-package/publisher/pluginPackagePublisherTrustTransitionApprovalConsumer.d.ts",
"require": "./dist/plugin-package/publisher/pluginPackagePublisherTrustTransitionApprovalConsumer.js",
"default": "./dist/plugin-package/publisher/pluginPackagePublisherTrustTransitionApprovalConsumer.js"
},
"./plugin-package-publisher-trust-management": {
"types": "./dist/plugin-package/publisher/pluginPackagePublisherTrustManagement.d.ts",
"require": "./dist/plugin-package/publisher/pluginPackagePublisherTrustManagement.js",
"default": "./dist/plugin-package/publisher/pluginPackagePublisherTrustManagement.js"
}
},
"files": [
"dist/**/*.js",
"dist/**/*.d.ts"
],
"scripts": {
"build": "tsc -p tsconfig.json",
"check": "node ../../scripts/ql3-build-package-closure.cjs && tsc -p tsconfig.json --noEmit",
"test": "node ../../scripts/ql3-build-package-closure.cjs && node --test test/*.test.cjs",
"test:automation-integration": "node ../../scripts/ql3-build-package-closure.cjs && node --test test/automationManagementPostgres.integration.test.cjs"
},
"bin": {
"ql3-plugin-package-recover": "dist/plugin-package/recovery/pluginPackageRecoveryCli.js",
"ql3-plugin-package-manage": "dist/plugin-package/management/pluginPackageManagementCli.js",
"ql3-plugin-package-client": "dist/plugin-package/management/pluginPackageManagementClientCli.js",
"ql3-plugin-package-client-kubernetes": "dist/plugin-package/management/pluginPackageManagementKubernetesClientCli.js",
"ql3-plugin-package-execute": "dist/plugin-package/executor/pluginPackageExecutorCli.js",
"ql3-prompt-output-gc": "dist/prompt-output/retention/promptOutputGcCli.js",
"ql3-prompt-output-key-retire": "dist/prompt-output/key-management/promptOutputKeyRetirementCli.js",
"ql3-prompt-output-key-rotate": "dist/prompt-output/key-management/promptOutputKeyRotationCli.js",
"ql3-prompt-output-key-recovery-verify": "dist/prompt-output/external-recovery/promptOutputExternalRecoveryCli.js",
"ql3-worker-credential-manage": "dist/worker-credential/management-server/workerCredentialManagementCli.js",
"ql3-worker-credential-execute": "dist/worker-credential/workerCredentialExecutorCli.js",
"ql3-worker-credential-client": "dist/worker-credential/workerCredentialManagementClientCli.js",
"ql3-approval-manage": "dist/approval-management/approvalManagementCli.js",
"ql3-approval-client": "dist/approval-management/approvalManagementClientCli.js",
"ql3-automation-manage": "dist/automation-management/automationManagementCli.js",
"ql3-automation-client": "dist/automation-management/automationManagementClientCli.js",
"ql3-provider-credential-manage": "dist/model-provider-credential/modelProviderCredentialManagementCli.js",
"ql3-provider-credential-client": "dist/model-provider-credential/modelProviderCredentialManagementClientCli.js",
"ql3-provider-credential-test-execute": "dist/model-provider-credential/modelProviderCredentialTestExecutorCli.js",
"ql3-ai-feature-migrate": "dist/modelInvocationMigrationCli.js"
},
"dependencies": {
"@kubernetes/client-node": "1.4.0",
"@qinglong/ai": "workspace:*",
"@qinglong/cluster-postgres": "workspace:*",
"@qinglong/runtime-core": "workspace:*"
},
"devDependencies": {
"@types/node": "24.13.3",
"typescript": "5.9.3"
}
}
@@ -0,0 +1,128 @@
import type {
OpenPostgresDatabase,
PostgresDatabaseResource,
} from '@qinglong/runtime-core';
import { assertApiCredentialPepper } from '@qinglong/runtime-core/api-credential-token';
import { assertWorkerCredentialPepper } from '@qinglong/runtime-core/worker-credential-token';
import type { SecurityAuditQueryRepository } from '@qinglong/runtime-core/security-audit-query';
import {
PostgresApiCredentialAdministrationRepository,
PostgresIdentityAdministrationRepository,
PostgresSecurityAuditQueryRepository,
PostgresWorkerCredentialAdministrationRepository,
assertPostgresAdminSchemaReady,
type PostgresSchemaReadinessReport,
} from '@qinglong/cluster-postgres/admin';
import {
createClusterAdministrationService,
type ClusterAdministrationOptions,
type ClusterAdministrationService,
} from '../security-administration/clusterAdministration';
import {
createWorkerCredentialAdministrationService,
type WorkerCredentialAdministrationService,
} from '../worker-credential/workerCredentialAdministration';
export interface ClusterAdminBootstrapOptions
extends ClusterAdministrationOptions {
readonly openDatabase: OpenPostgresDatabase;
readonly apiCredentialPepper: string;
readonly workerCredentialPepper: string;
}
export interface ClusterAdminRuntime {
readonly evidence: PostgresSchemaReadinessReport;
readonly administration: ClusterAdministrationService;
readonly audit: SecurityAuditQueryRepository;
readonly workerCredentials: WorkerCredentialAdministrationService;
close(): Promise<void>;
}
export async function bootstrapClusterAdmin(
options: ClusterAdminBootstrapOptions,
): Promise<ClusterAdminRuntime> {
if (!options || typeof options !== 'object' || Array.isArray(options)) {
throw new TypeError('Cluster admin bootstrap options are invalid');
}
if (typeof options.openDatabase !== 'function') {
throw new TypeError('Cluster admin database opener is invalid');
}
const optionKeys = new Set([
'openDatabase',
'apiCredentialPepper',
'workerCredentialPepper',
'now',
'randomBytes',
]);
if (Object.keys(options).some((key) => !optionKeys.has(key))) {
throw new TypeError('Cluster admin bootstrap options shape is invalid');
}
if (options.now !== undefined && typeof options.now !== 'function') {
throw new TypeError('Cluster admin clock is invalid');
}
if (
options.randomBytes !== undefined &&
typeof options.randomBytes !== 'function'
) {
throw new TypeError('Cluster admin random source is invalid');
}
try {
assertApiCredentialPepper(options.apiCredentialPepper);
} catch {
throw new TypeError('Cluster admin API credential pepper is invalid');
}
try {
assertWorkerCredentialPepper(options.workerCredentialPepper);
} catch {
throw new TypeError('Cluster admin Worker credential pepper is invalid');
}
let database: PostgresDatabaseResource | undefined;
let closePromise: Promise<void> | undefined;
const close = (): Promise<void> => {
if (!database) return Promise.resolve();
closePromise ??= database.close();
return closePromise;
};
try {
database = await options.openDatabase();
const evidence = await assertPostgresAdminSchemaReady(database.pool);
const identities = new PostgresIdentityAdministrationRepository(
database.pool,
);
const credentials = new PostgresApiCredentialAdministrationRepository(
database.pool,
);
return Object.freeze({
evidence,
administration: createClusterAdministrationService(
identities,
credentials,
options.apiCredentialPepper,
{
...(options.now ? { now: options.now } : {}),
...(options.randomBytes ? { randomBytes: options.randomBytes } : {}),
},
),
audit: new PostgresSecurityAuditQueryRepository(database.pool),
workerCredentials: createWorkerCredentialAdministrationService(
new PostgresWorkerCredentialAdministrationRepository(database.pool),
options.workerCredentialPepper,
{
...(options.now ? { now: options.now } : {}),
...(options.randomBytes ? { randomBytes: options.randomBytes } : {}),
},
),
close,
});
} catch (error) {
try {
await close();
} catch {
// Preserve configuration/readiness/assembly failure.
}
throw error;
}
}
export * from '../security-administration/clusterAdministration';
export * from '../worker-credential/workerCredentialAdministration';
@@ -0,0 +1,62 @@
import { PostgresApprovalRequestRepository } from '@qinglong/cluster-postgres/approved-action';
import { PostgresProjectPolicyRepository } from '@qinglong/cluster-postgres/project-policy';
import type { PostgresPool } from '@qinglong/runtime-core';
import {
createApprovalDecisionService,
type ApprovalDecisionService,
} from '@qinglong/runtime-core/approval-decision';
import { ProjectPolicyEngine } from '@qinglong/runtime-core/project-policy';
export interface ClusterApprovalDecisionManagementOptions {
readonly pool: PostgresPool;
readonly confirmAuthorization?: () => void | Promise<void>;
readonly now?: () => number;
}
export class ClusterApprovalDecisionManagementConfigurationError extends TypeError {
readonly code = 'CLUSTER_APPROVAL_DECISION_CONFIGURATION_INVALID';
constructor(message: string) {
super(`Cluster Approval decision configuration is invalid: ${message}`);
this.name = 'ClusterApprovalDecisionManagementConfigurationError';
}
}
/**
* Composes the profile-neutral human decision contract over a caller-owned
* PostgreSQL pool. Transport authentication and lifecycle remain outside this
* authority so cluster nodes can reuse their existing mTLS/OIDC boundary.
*/
export function createClusterApprovalDecisionManagementService(
options: ClusterApprovalDecisionManagementOptions,
): Readonly<ApprovalDecisionService> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
key !== 'pool' && key !== 'confirmAuthorization' && key !== 'now',
) ||
!options.pool ||
typeof options.pool.query !== 'function' ||
typeof options.pool.connect !== 'function' ||
(options.confirmAuthorization !== undefined &&
typeof options.confirmAuthorization !== 'function') ||
(options.now !== undefined && typeof options.now !== 'function')
) {
throw new ClusterApprovalDecisionManagementConfigurationError(
'options are invalid',
);
}
return createApprovalDecisionService({
approvals: new PostgresApprovalRequestRepository(options.pool),
policy: new ProjectPolicyEngine(
new PostgresProjectPolicyRepository(options.pool),
),
...(options.confirmAuthorization === undefined
? {}
: { confirmAuthorization: options.confirmAuthorization }),
...(options.now === undefined ? {} : { now: options.now }),
});
}
@@ -0,0 +1,102 @@
import {
PostgresApprovalManagementIdentityKeysetLedgerRepository,
PostgresApprovalRequestRepository,
PostgresApprovalRequestSource,
PostgresProjectPolicyRepository,
PostgresSecurityAuditRepository,
} from '@qinglong/cluster-postgres/approval-manager';
import type { PostgresPool } from '@qinglong/runtime-core';
import {
createApprovalDecisionService,
type ApprovalDecisionRequest,
type ApprovalDecisionService,
} from '@qinglong/runtime-core/approval-decision';
import {
createApprovalInspectionService,
type ApprovalInspectionRequest,
type ApprovalInspectionService,
} from '@qinglong/runtime-core/approval-inspection';
import { ProjectPolicyEngine } from '@qinglong/runtime-core/project-policy';
import type { SecurityAuditRecord } from '@qinglong/runtime-core/security-audit';
export interface ClusterApprovalManagementService {
inspect(
request: ApprovalInspectionRequest,
confirmAuthorization: () => void | Promise<void>,
): ReturnType<ApprovalInspectionService['inspect']>;
decide(
request: ApprovalDecisionRequest,
confirmAuthorization: () => void | Promise<void>,
): ReturnType<ApprovalDecisionService['decide']>;
recordFailure(record: SecurityAuditRecord): Promise<void>;
}
export interface ClusterApprovalManagementOptions {
readonly pool: PostgresPool;
readonly now?: () => number;
}
export class ClusterApprovalManagementConfigurationError extends TypeError {
readonly code = 'CLUSTER_APPROVAL_MANAGEMENT_CONFIGURATION_INVALID';
constructor(message: string) {
super(`Cluster Approval management configuration is invalid: ${message}`);
this.name = 'ClusterApprovalManagementConfigurationError';
}
}
/** Dedicated Approval manager composition over one caller-owned Pool. */
export function createClusterApprovalManagementService(
options: ClusterApprovalManagementOptions,
): Readonly<ClusterApprovalManagementService> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some((key) => key !== 'pool' && key !== 'now') ||
!options.pool ||
typeof options.pool.query !== 'function' ||
typeof options.pool.connect !== 'function' ||
(options.now !== undefined && typeof options.now !== 'function')
) {
throw new ClusterApprovalManagementConfigurationError(
'options are invalid',
);
}
const policy = new ProjectPolicyEngine(
new PostgresProjectPolicyRepository(options.pool),
);
const approvals = new PostgresApprovalRequestRepository(options.pool);
const source = new PostgresApprovalRequestSource(options.pool);
const audit = new PostgresSecurityAuditRepository(options.pool);
return Object.freeze({
inspect(
request: ApprovalInspectionRequest,
confirmAuthorization: () => void | Promise<void>,
) {
return createApprovalInspectionService({
source,
policy,
audit,
confirmAuthorization,
...(options.now === undefined ? {} : { now: options.now }),
}).inspect(request);
},
decide(
request: ApprovalDecisionRequest,
confirmAuthorization: () => void | Promise<void>,
) {
return createApprovalDecisionService({
approvals,
policy,
confirmAuthorization,
...(options.now === undefined ? {} : { now: options.now }),
}).decide(request);
},
recordFailure(record: SecurityAuditRecord) {
return audit.record(record);
},
});
}
export { PostgresApprovalManagementIdentityKeysetLedgerRepository };
@@ -0,0 +1,103 @@
#!/usr/bin/env node
import {
startClusterApprovalManagementProcess,
type ClusterApprovalManagementProcessRuntime,
} from './approvalManagementProcess';
const USAGE = 'Usage: ql3-approval-manage';
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as { readonly name?: unknown; readonly code?: unknown };
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-approval-management',
event: 'management_failed',
name:
typeof candidate?.name === 'string' && candidate.name.length <= 128
? candidate.name
: 'Error',
...(typeof candidate?.code === 'string' && candidate.code.length <= 128
? { code: candidate.code }
: {}),
});
}
function emit(value: Readonly<Record<string, unknown>>): void {
process.stdout.write(`${JSON.stringify(value)}\n`);
}
async function run(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
if (argv.length !== 0) {
process.stderr.write(`${JSON.stringify({
code: 'QL3_APPROVAL_MANAGEMENT_CLI_USAGE_INVALID',
message: USAGE,
})}\n`);
process.exitCode = 64;
return;
}
let runtime: Readonly<ClusterApprovalManagementProcessRuntime>;
try {
runtime = await startClusterApprovalManagementProcess({
environment: process.env,
onError() {
emit({
schemaVersion: 1,
component: 'qinglong3-approval-management',
event: 'management_unavailable',
});
},
});
} catch (error) {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
return;
}
if (runtime.status === 'disabled') {
emit({
schemaVersion: 1,
component: 'qinglong3-approval-management',
event: 'management_disabled',
});
return;
}
emit({
schemaVersion: 1,
component: 'qinglong3-approval-management',
event: 'management_started',
address: runtime.address,
identityGeneration: runtime.identity.generation,
databaseContractVersion: runtime.database.contractVersion,
databaseMigrationCount: runtime.database.migrationIds.length,
});
let stopping: Promise<void> | undefined;
const stop = (): Promise<void> => {
stopping ??= runtime.close().then(() => {
emit({
schemaVersion: 1,
component: 'qinglong3-approval-management',
event: 'management_stopped',
});
});
return stopping;
};
for (const signal of ['SIGINT', 'SIGTERM'] as const) {
process.once(signal, () => {
void stop().then(
() => {
process.exitCode = 0;
},
(error) => {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
},
);
});
}
}
void run(process.argv.slice(2));
@@ -0,0 +1,201 @@
import { normalizeApprovedActionBinding } from '@qinglong/runtime-core/approved-action';
import { normalizeApprovalDetailPreview } from '@qinglong/runtime-core/approval-discovery';
import {
ClusterPluginPackageManagementClientRequestError,
executeClusterAuthenticatedManagementClient,
type ClusterAuthenticatedManagementClientResult,
type ClusterPluginPackageManagementClientConnectionOptions,
type ClusterPluginPackageManagementClientPaths,
} from '../management-support/pluginPackageManagementClient';
import {
normalizeClusterApprovalManagementCommand,
type ClusterApprovalManagementCommand,
type ClusterApprovalManagementTransportResult,
} from './approvalManagementTransport';
const MANAGEMENT_PATH = '/api/v3/approvals/management';
const IDENTIFIER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
export type ClusterApprovalManagementClientPaths =
ClusterPluginPackageManagementClientPaths;
export type ClusterApprovalManagementClientConnectionOptions =
ClusterPluginPackageManagementClientConnectionOptions;
export type ClusterApprovalManagementClientResult =
ClusterAuthenticatedManagementClientResult<ClusterApprovalManagementTransportResult>;
function invalid(): never {
throw new ClusterPluginPackageManagementClientRequestError();
}
function exact(value: unknown, keys: readonly string[]): Record<string, unknown> {
if (!value || typeof value !== 'object' || Array.isArray(value)) invalid();
const actual = Object.keys(value as object).sort();
const expected = [...keys].sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
invalid();
}
return value as Record<string, unknown>;
}
function identifier(value: unknown): string {
if (typeof value !== 'string' || !IDENTIFIER_PATTERN.test(value)) invalid();
return value;
}
function subject(value: unknown): void {
const record = exact(value, ['type', 'id']);
if (
(record.type !== 'user' && record.type !== 'system' && record.type !== 'agent') ||
typeof record.id !== 'string'
) {
invalid();
}
identifier(record.id);
}
function safeTime(value: unknown, nullable = false): void {
if (nullable && value === null) return;
if (!Number.isSafeInteger(value) || Number(value) < 0) invalid();
}
function action(value: unknown): void {
try {
normalizeApprovedActionBinding(value as never);
} catch {
invalid();
}
}
function inspectApproval(
value: unknown,
command: Extract<ClusterApprovalManagementCommand, { operation: 'approval.inspect' }>,
): void {
const approval = exact(value, [
'projectId',
'approvalRequestId',
'version',
'state',
'risk',
'decisionMode',
'expectedAction',
'requestedBy',
'requestedAtMs',
'expiresAtMs',
'preview',
]);
if (
identifier(approval.projectId) !== command.request.projectId ||
identifier(approval.approvalRequestId) !== command.request.approvalRequestId ||
!Number.isSafeInteger(approval.version) ||
Number(approval.version) < 1 ||
typeof approval.state !== 'string' ||
typeof approval.risk !== 'string' ||
typeof approval.decisionMode !== 'string'
) {
invalid();
}
action(approval.expectedAction);
subject(approval.requestedBy);
safeTime(approval.requestedAtMs);
safeTime(approval.expiresAtMs);
if (approval.preview !== null) {
try {
normalizeApprovalDetailPreview(approval.preview as never);
} catch {
invalid();
}
}
}
function decisionApproval(
value: unknown,
command: Extract<ClusterApprovalManagementCommand, { operation: 'approval.decide' }>,
): void {
const approval = exact(value, [
'projectId',
'approvalRequestId',
'version',
'state',
'expectedAction',
'decisionId',
'decision',
'reasonCode',
'decidedBy',
'decidedAtMs',
]);
if (
identifier(approval.projectId) !== command.request.projectId ||
identifier(approval.approvalRequestId) !== command.request.approvalRequestId ||
approval.version !== 2 ||
approval.state !== command.request.decision ||
identifier(approval.decisionId) !== command.request.decisionId ||
approval.decision !== command.request.decision ||
approval.reasonCode !== command.request.reasonCode
) {
invalid();
}
action(approval.expectedAction);
subject(approval.decidedBy);
safeTime(approval.decidedAtMs);
}
export function validateClusterApprovalManagementClientResult(
value: unknown,
command: Readonly<ClusterApprovalManagementCommand>,
): Readonly<ClusterApprovalManagementTransportResult> {
const envelope = exact(value, [
'schemaVersion',
'operation',
'status',
'approval',
]);
if (
envelope.schemaVersion !== 1 ||
envelope.operation !== command.operation
) {
invalid();
}
if (command.operation === 'approval.inspect') {
if (
(envelope.status !== 'found' && envelope.status !== 'absent') ||
(envelope.status === 'absent') !== (envelope.approval === null)
) {
invalid();
}
if (envelope.approval !== null) {
inspectApproval(envelope.approval, command);
}
} else {
if (
envelope.status !== 'decided' &&
envelope.status !== 'existing'
) {
invalid();
}
decisionApproval(envelope.approval, command);
}
return Object.freeze(
envelope as unknown as ClusterApprovalManagementTransportResult,
);
}
const PROTOCOL = Object.freeze({
managementPath: MANAGEMENT_PATH,
clientCertificate: 'required' as const,
normalizeCommand: normalizeClusterApprovalManagementCommand,
validateResult: validateClusterApprovalManagementClientResult,
});
export async function executeClusterApprovalManagementClient(
paths: ClusterApprovalManagementClientPaths,
connectionOptions?: ClusterApprovalManagementClientConnectionOptions,
): Promise<Readonly<ClusterApprovalManagementClientResult>> {
return executeClusterAuthenticatedManagementClient(
paths,
PROTOCOL,
connectionOptions,
);
}
@@ -0,0 +1,89 @@
#!/usr/bin/env node
import { executeClusterApprovalManagementClient } from './approvalManagementClient';
import { ClusterPluginPackageManagementClientRemoteError } from '../management-support/pluginPackageManagementClient';
const USAGE =
'Usage: ql3-approval-client --config=/absolute/client.json --command=/absolute/command.json --assertion=/absolute/assertion.jwt';
function argumentsFrom(argv: readonly string[]): Readonly<{
configFile: string;
commandFile: string;
assertionFile: string;
}> | null {
if (argv.length !== 3) return null;
const values = new Map<string, string>();
for (const argument of argv) {
const match = /^--(config|command|assertion)=(\/.+)$/.exec(argument);
if (!match || values.has(match[1]!)) return null;
values.set(match[1]!, match[2]!);
}
if (
!values.has('config') ||
!values.has('command') ||
!values.has('assertion')
) {
return null;
}
return Object.freeze({
configFile: values.get('config')!,
commandFile: values.get('command')!,
assertionFile: values.get('assertion')!,
});
}
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as { readonly code?: unknown };
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-approval-management-client',
event: 'command_failed',
code:
typeof candidate?.code === 'string' && candidate.code.length <= 128
? candidate.code
: 'QL3_APPROVAL_MANAGEMENT_CLIENT_FAILED',
...(error instanceof ClusterPluginPackageManagementClientRemoteError
? {
statusCode: error.statusCode,
responseCode: error.responseCode,
requestId: error.requestId,
...(error.retryAfterSeconds === null
? {}
: { retryAfterSeconds: error.retryAfterSeconds }),
}
: {}),
});
}
async function run(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
const paths = argumentsFrom(argv);
if (!paths) {
process.stderr.write(`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-approval-management-client',
event: 'usage_invalid',
code: 'QL3_APPROVAL_MANAGEMENT_CLIENT_USAGE_INVALID',
})}\n`);
process.exitCode = 64;
return;
}
try {
const result = await executeClusterApprovalManagementClient(paths);
process.stdout.write(`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-approval-management-client',
event: 'command_completed',
requestId: result.requestId,
result: result.result,
})}\n`);
} catch (error) {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
}
}
void run(process.argv.slice(2));
@@ -0,0 +1,24 @@
import {
CLUSTER_APPROVAL_MANAGEMENT_PATH,
startClusterPluginPackageManagementHttp,
type ClusterPluginPackageManagementHttpApplication,
type StartClusterPluginPackageManagementHttpOptions,
} from '../management-support/pluginPackageManagementHttp';
export type ClusterApprovalManagementHttpApplication =
ClusterPluginPackageManagementHttpApplication;
export type StartClusterApprovalManagementHttpOptions = Omit<
StartClusterPluginPackageManagementHttpOptions,
'managementPath'
>;
/** Starts the shared bounded OIDC/mTLS HTTPS adapter on the Approval-only path. */
export function startClusterApprovalManagementHttp(
options: StartClusterApprovalManagementHttpOptions,
): Promise<Readonly<ClusterApprovalManagementHttpApplication>> {
return startClusterPluginPackageManagementHttp({
...options,
managementPath: CLUSTER_APPROVAL_MANAGEMENT_PATH,
});
}
@@ -0,0 +1,574 @@
import type {
OpenPostgresDatabase,
PostgresDatabaseResource,
} from '@qinglong/runtime-core';
import {
PostgresApprovalManagementIdentityKeysetLedgerRepository,
assertPostgresApprovalManagerSchemaReady,
createPostgresDatabaseOpener,
isPostgresTlsDnsServername,
loadPostgresCertificateAuthorityFile,
loadPostgresConnectionEnvironment,
type PostgresConnectionOptions,
type PostgresPoolOptions,
type PostgresSchemaReadinessReport,
} from '@qinglong/cluster-postgres/approval-manager';
import {
absoluteManagementEnvironmentFile,
booleanManagementEnvironmentValue,
boundedManagementEnvironmentValue,
integerManagementEnvironmentValue,
readManagementTlsFile,
} from '../management-support/managementProcessSupport';
import {
createClusterApprovalIdentityKeysetFile,
type ClusterPluginPackageIdentityKeysetFile,
type ClusterPluginPackageIdentityKeysetSnapshot,
} from '../management-support/pluginPackageIdentityKeyset';
import { validateClusterManagementClientTrust } from '../worker-credential/management-server/workerCredentialManagementMutualTls';
import { createClusterApprovalManagementService } from './approvalManagement';
import {
startClusterApprovalManagementHttp,
type ClusterApprovalManagementHttpApplication,
type StartClusterApprovalManagementHttpOptions,
} from './approvalManagementHttp';
import { createClusterApprovalManagementTransport } from './approvalManagementTransport';
const SAFE_HOST = /^[A-Za-z0-9][A-Za-z0-9.:-]{0,254}$/;
const SAFE_APPLICATION_NAME = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,62}$/;
export type ClusterApprovalManagementProcessEnvironment = Readonly<
Record<string, string | undefined>
>;
export type ClusterApprovalManagementProcessConfig =
| Readonly<{ enabled: false }>
| Readonly<{
enabled: true;
profile: 'cluster-admin';
host: string;
port: number;
certificateFile: string;
privateKeyFile: string;
clientCertificateAuthorityFile: string;
clientCertificateRevocationListFile: string;
identityKeysetFile: string;
http: Readonly<{
maxBodyBytes: number;
maxConnections: number;
maxConcurrentRequests: number;
requestTimeoutMs: number;
drainTimeoutMs: number;
rateWindowMs: number;
peerRequestLimit: number;
globalRequestLimit: number;
maxRateLimitPeers: number;
}>;
database: Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}>;
}>;
export type ClusterApprovalManagementProcessRuntime =
| Readonly<{
status: 'disabled';
close(): Promise<void>;
}>
| Readonly<{
status: 'active';
address: Readonly<{ host: string; port: number }>;
database: PostgresSchemaReadinessReport;
identity: ClusterPluginPackageIdentityKeysetSnapshot;
availabilityStatus(): 'ready' | 'unavailable' | 'stopped';
close(): Promise<void>;
}>;
export interface StartClusterApprovalManagementProcessOptions {
readonly environment: ClusterApprovalManagementProcessEnvironment;
readonly openDatabase?: OpenPostgresDatabase;
readonly identities?: ClusterPluginPackageIdentityKeysetFile;
readonly assertReady?: (
pool: PostgresDatabaseResource['pool'],
) => Promise<PostgresSchemaReadinessReport>;
readonly startHttp?: (
options: StartClusterApprovalManagementHttpOptions,
) => Promise<Readonly<ClusterApprovalManagementHttpApplication>>;
readonly now?: () => number;
readonly onError?: (error: unknown) => void;
}
export class ClusterApprovalManagementProcessConfigError extends TypeError {
readonly code = 'QL3_APPROVAL_MANAGEMENT_PROCESS_CONFIG_INVALID';
constructor(message: string) {
super(`Approval management process configuration is invalid: ${message}`);
this.name = 'ClusterApprovalManagementProcessConfigError';
}
}
function failure(message: string): ClusterApprovalManagementProcessConfigError {
return new ClusterApprovalManagementProcessConfigError(message);
}
function bounded(
environment: ClusterApprovalManagementProcessEnvironment,
name: string,
maximumLength: number,
required = false,
): string | undefined {
return boundedManagementEnvironmentValue(
environment,
name,
maximumLength,
failure,
required,
);
}
function bool(
environment: ClusterApprovalManagementProcessEnvironment,
name: string,
): boolean {
return booleanManagementEnvironmentValue(environment, name, failure);
}
function integer(
environment: ClusterApprovalManagementProcessEnvironment,
name: string,
fallback: number,
minimum: number,
maximum: number,
): number {
return integerManagementEnvironmentValue(
environment,
name,
fallback,
minimum,
maximum,
failure,
);
}
function absolute(
environment: ClusterApprovalManagementProcessEnvironment,
name: string,
): string {
return absoluteManagementEnvironmentFile(environment, name, failure);
}
function loadDatabase(
environment: ClusterApprovalManagementProcessEnvironment,
): Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}> {
let connection: PostgresConnectionOptions;
try {
connection = loadPostgresConnectionEnvironment(environment, {
connectionString: 'QL3_POSTGRES_APPROVAL_MANAGER_URL',
host: 'QL3_POSTGRES_APPROVAL_MANAGER_HOST',
port: 'QL3_POSTGRES_APPROVAL_MANAGER_PORT',
database: 'QL3_POSTGRES_APPROVAL_MANAGER_DATABASE',
user: 'QL3_POSTGRES_APPROVAL_MANAGER_USER',
password: 'QL3_POSTGRES_APPROVAL_MANAGER_PASSWORD',
});
} catch (error) {
throw failure(
error instanceof Error
? error.message
: 'PostgreSQL approval manager connection is invalid',
);
}
const mode =
environment.QL3_POSTGRES_APPROVAL_MANAGER_TLS_MODE ?? 'verify-full';
if (mode !== 'verify-full' && mode !== 'disable') {
throw failure(
'QL3_POSTGRES_APPROVAL_MANAGER_TLS_MODE must be verify-full or disable',
);
}
if (
mode === 'disable' &&
!bool(environment, 'QL3_POSTGRES_APPROVAL_MANAGER_ALLOW_INSECURE')
) {
throw failure(
'disabling approval manager PostgreSQL TLS requires QL3_POSTGRES_APPROVAL_MANAGER_ALLOW_INSECURE=true',
);
}
const servername = bounded(
environment,
'QL3_POSTGRES_APPROVAL_MANAGER_TLS_SERVERNAME',
253,
);
if (mode === 'verify-full' && !isPostgresTlsDnsServername(servername)) {
throw failure(
'QL3_POSTGRES_APPROVAL_MANAGER_TLS_SERVERNAME must be an explicit DNS name',
);
}
const caFile = bounded(
environment,
'QL3_POSTGRES_APPROVAL_MANAGER_TLS_CA_FILE',
4_096,
);
if (mode === 'disable' && caFile !== undefined) {
throw failure(
'QL3_POSTGRES_APPROVAL_MANAGER_TLS_CA_FILE cannot be used when TLS is disabled',
);
}
let ca: string | undefined;
if (caFile !== undefined) {
try {
ca = loadPostgresCertificateAuthorityFile(caFile);
} catch {
throw failure('QL3_POSTGRES_APPROVAL_MANAGER_TLS_CA_FILE is invalid');
}
}
const applicationName =
bounded(
environment,
'QL3_POSTGRES_APPROVAL_MANAGER_APPLICATION_NAME',
63,
) ?? 'qinglong3-approval-manager';
if (!SAFE_APPLICATION_NAME.test(applicationName)) {
throw failure('QL3_POSTGRES_APPROVAL_MANAGER_APPLICATION_NAME is invalid');
}
return Object.freeze({
connection: Object.freeze({
...connection,
tls:
mode === 'disable'
? Object.freeze({ mode: 'disable' as const })
: Object.freeze({
mode: 'verify-full' as const,
servername: servername!,
...(ca === undefined ? {} : { ca }),
}),
}),
pool: Object.freeze({
applicationName,
maxConnections: integer(
environment,
'QL3_POSTGRES_APPROVAL_MANAGER_POOL_MAX',
2,
1,
4,
),
idleTimeoutMs: integer(
environment,
'QL3_POSTGRES_APPROVAL_MANAGER_IDLE_TIMEOUT_MS',
10_000,
1_000,
60_000,
),
connectionTimeoutMs: integer(
environment,
'QL3_POSTGRES_APPROVAL_MANAGER_CONNECTION_TIMEOUT_MS',
5_000,
100,
60_000,
),
}),
});
}
export function loadClusterApprovalManagementProcessConfig(
environment: ClusterApprovalManagementProcessEnvironment,
): Readonly<ClusterApprovalManagementProcessConfig> {
if (!environment || typeof environment !== 'object') {
throw failure('environment is invalid');
}
if (!bool(environment, 'QL3_APPROVAL_MANAGEMENT_ENABLED')) {
return Object.freeze({ enabled: false as const });
}
if (environment.QL3_PROFILE !== 'cluster-admin') {
throw failure(
'QL3_PROFILE must be cluster-admin when Approval management is enabled',
);
}
const host =
bounded(environment, 'QL3_APPROVAL_MANAGEMENT_HOST', 255) ?? '0.0.0.0';
if (!SAFE_HOST.test(host)) {
throw failure('QL3_APPROVAL_MANAGEMENT_HOST is invalid');
}
const http = Object.freeze({
maxBodyBytes: integer(
environment,
'QL3_APPROVAL_MANAGEMENT_MAX_BODY_BYTES',
64 * 1024,
1_024,
256 * 1024,
),
maxConnections: integer(
environment,
'QL3_APPROVAL_MANAGEMENT_MAX_CONNECTIONS',
32,
1,
512,
),
maxConcurrentRequests: integer(
environment,
'QL3_APPROVAL_MANAGEMENT_MAX_CONCURRENT_REQUESTS',
16,
1,
256,
),
requestTimeoutMs: integer(
environment,
'QL3_APPROVAL_MANAGEMENT_REQUEST_TIMEOUT_MS',
10_000,
1_000,
60_000,
),
drainTimeoutMs: integer(
environment,
'QL3_APPROVAL_MANAGEMENT_DRAIN_TIMEOUT_MS',
5_000,
100,
60_000,
),
rateWindowMs: integer(
environment,
'QL3_APPROVAL_MANAGEMENT_RATE_WINDOW_MS',
60_000,
1_000,
5 * 60_000,
),
peerRequestLimit: integer(
environment,
'QL3_APPROVAL_MANAGEMENT_PEER_REQUEST_LIMIT',
30,
1,
10_000,
),
globalRequestLimit: integer(
environment,
'QL3_APPROVAL_MANAGEMENT_GLOBAL_REQUEST_LIMIT',
300,
1,
100_000,
),
maxRateLimitPeers: integer(
environment,
'QL3_APPROVAL_MANAGEMENT_MAX_RATE_LIMIT_PEERS',
1_024,
1,
16_384,
),
});
if (http.globalRequestLimit < http.peerRequestLimit) {
throw failure('global request limit cannot be below peer request limit');
}
return Object.freeze({
enabled: true as const,
profile: 'cluster-admin' as const,
host,
port: integer(
environment,
'QL3_APPROVAL_MANAGEMENT_PORT',
8_447,
1,
65_535,
),
certificateFile: absolute(
environment,
'QL3_APPROVAL_MANAGEMENT_TLS_CERT_FILE',
),
privateKeyFile: absolute(
environment,
'QL3_APPROVAL_MANAGEMENT_TLS_KEY_FILE',
),
clientCertificateAuthorityFile: absolute(
environment,
'QL3_APPROVAL_MANAGEMENT_CLIENT_CA_FILE',
),
clientCertificateRevocationListFile: absolute(
environment,
'QL3_APPROVAL_MANAGEMENT_CLIENT_CRL_FILE',
),
identityKeysetFile: absolute(
environment,
'QL3_APPROVAL_MANAGEMENT_IDENTITY_KEYSET_FILE',
),
http,
database: loadDatabase(environment),
});
}
export async function startClusterApprovalManagementProcess(
options: StartClusterApprovalManagementProcessOptions,
): Promise<Readonly<ClusterApprovalManagementProcessRuntime>> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
![
'environment',
'openDatabase',
'identities',
'assertReady',
'startHttp',
'now',
'onError',
].includes(key),
) ||
!options.environment ||
typeof options.environment !== 'object' ||
(options.openDatabase !== undefined &&
typeof options.openDatabase !== 'function') ||
(options.identities !== undefined &&
(typeof options.identities.reload !== 'function' ||
typeof options.identities.bind !== 'function')) ||
(options.assertReady !== undefined &&
typeof options.assertReady !== 'function') ||
(options.startHttp !== undefined &&
typeof options.startHttp !== 'function') ||
(options.now !== undefined && typeof options.now !== 'function') ||
(options.onError !== undefined && typeof options.onError !== 'function')
) {
throw failure('options are invalid');
}
const config = loadClusterApprovalManagementProcessConfig(
options.environment,
);
if (!config.enabled) {
return Object.freeze({
status: 'disabled' as const,
close: () => Promise.resolve(),
});
}
const now = options.now ?? Date.now;
let http: Readonly<ClusterApprovalManagementHttpApplication> | undefined;
let database: PostgresDatabaseResource | undefined;
let unavailableError: unknown;
let closePromise: Promise<void> | undefined;
const report = (error: unknown): void => {
try {
options.onError?.(error);
} catch {
// Diagnostics never own availability or cleanup.
}
};
const openDatabase =
options.openDatabase ??
createPostgresDatabaseOpener({
role: 'approval-manager',
connection: config.database.connection,
pool: config.database.pool,
onPoolError(error) {
const first = unavailableError === undefined;
unavailableError ??= error;
http?.withdraw(error);
if (first) report(error);
},
});
try {
database = await openDatabase();
const evidence = await (
options.assertReady ?? assertPostgresApprovalManagerSchemaReady
)(database.pool);
if (unavailableError !== undefined) throw unavailableError;
const identities =
options.identities ??
createClusterApprovalIdentityKeysetFile({
filePath: config.identityKeysetFile,
now,
ledger: new PostgresApprovalManagementIdentityKeysetLedgerRepository(
database.pool,
'approval-management',
),
});
const identity = await identities.reload();
const service = createClusterApprovalManagementService({
pool: database.pool,
now,
});
const transport = createClusterApprovalManagementTransport({
service,
now,
});
const privateKey = readManagementTlsFile(
config.privateKeyFile,
true,
failure,
);
try {
const certificate = readManagementTlsFile(
config.certificateFile,
false,
failure,
);
const clientCertificateAuthority = readManagementTlsFile(
config.clientCertificateAuthorityFile,
false,
failure,
);
const clientCertificateRevocationList = readManagementTlsFile(
config.clientCertificateRevocationListFile,
false,
failure,
);
validateClusterManagementClientTrust(
clientCertificateAuthority,
clientCertificateRevocationList,
now(),
failure,
);
http = await (options.startHttp ?? startClusterApprovalManagementHttp)({
host: config.host,
port: config.port,
tls: {
privateKey,
certificate,
clientCertificateAuthority,
clientCertificateRevocationList,
},
transport,
identities,
limits: config.http,
now,
onError: report,
});
} finally {
privateKey.fill(0);
}
if (unavailableError !== undefined) http.withdraw(unavailableError);
return Object.freeze({
status: 'active' as const,
address: http.address,
database: evidence,
identity,
availabilityStatus: () => http?.availabilityStatus() ?? 'stopped',
close(): Promise<void> {
closePromise ??= (async () => {
let primaryError: unknown;
try {
await http?.close();
} catch (error) {
primaryError = error;
}
try {
await database?.close();
} catch (error) {
primaryError ??= error;
}
if (primaryError) throw primaryError;
})();
return closePromise;
},
});
} catch (error) {
try {
await http?.close();
} catch {
// Preserve startup failure.
}
try {
await database?.close();
} catch {
// Preserve startup failure.
}
throw error;
}
}
@@ -0,0 +1,505 @@
import {
ApprovalMutationConflictError,
ApprovalPolicyFenceConflictError,
ApprovalRequestExpiredError,
ApprovalRequestStateConflictError,
ApprovalRequestVersionConflictError,
normalizeApprovedActionBinding,
type ApprovedActionBinding,
} from '@qinglong/runtime-core/approved-action';
import {
ApprovalDecisionAuthorizationError,
ApprovalDecisionBindingConflictError,
ApprovalDecisionTargetUnavailableError,
ApprovalDecisionUnavailableError,
} from '@qinglong/runtime-core/approval-decision';
import {
ApprovalInspectionAuthorizationError,
ApprovalInspectionUnavailableError,
} from '@qinglong/runtime-core/approval-inspection';
import {
normalizeSecurityPrincipal,
type SecurityPrincipal,
} from '@qinglong/runtime-core/security';
import type { SecurityAuditRecord } from '@qinglong/runtime-core/security-audit';
import type { ClusterApprovalManagementService } from './approvalManagement';
const IDENTIFIER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const REASON_PATTERN = /^[a-z][a-z0-9_]{0,63}$/;
const UUID_V4_PATTERN =
/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/;
const STRONG_ASSURANCES = new Set(['multi_factor', 'hardware']);
interface BaseRequest {
readonly projectId: string;
readonly approvalRequestId: string;
readonly requestId: string;
readonly auditEventId: string;
readonly failureAuditEventId: string;
}
export type ClusterApprovalManagementCommand =
| Readonly<{
schemaVersion: 1;
operation: 'approval.inspect';
request: BaseRequest;
}>
| Readonly<{
schemaVersion: 1;
operation: 'approval.decide';
request: BaseRequest & {
readonly expectedVersion: 1;
readonly expectedAction: Readonly<ApprovedActionBinding>;
readonly decisionId: string;
readonly decision: 'approved' | 'rejected';
readonly reasonCode: string;
};
}>;
export type ClusterApprovalManagementTransportResult = Readonly<
Record<string, unknown> & {
readonly schemaVersion: 1;
readonly operation: ClusterApprovalManagementCommand['operation'];
}
>;
export interface ClusterApprovalManagementAuthentication {
authenticate(): Promise<Readonly<SecurityPrincipal> | null>;
}
export interface ClusterApprovalManagementTransport {
execute(
command: unknown,
authentication: ClusterApprovalManagementAuthentication,
): Promise<Readonly<ClusterApprovalManagementTransportResult>>;
}
export class ClusterApprovalManagementTransportConfigurationError extends TypeError {
readonly code = 'CLUSTER_APPROVAL_TRANSPORT_CONFIGURATION_INVALID';
constructor() {
super('Cluster Approval transport configuration is invalid');
this.name = 'ClusterApprovalManagementTransportConfigurationError';
}
}
export class ClusterApprovalManagementTransportRequestError extends TypeError {
readonly code = 'CLUSTER_APPROVAL_TRANSPORT_REQUEST_INVALID';
constructor() {
super('Cluster Approval transport request is invalid');
this.name = 'ClusterApprovalManagementTransportRequestError';
}
}
export class ClusterApprovalManagementTransportAuthenticationError extends Error {
readonly code = 'CLUSTER_APPROVAL_TRANSPORT_AUTHENTICATION_REQUIRED';
constructor() {
super('Cluster Approval transport requires a strong User principal');
this.name = 'ClusterApprovalManagementTransportAuthenticationError';
}
}
export class ClusterApprovalManagementTransportAuthorizationError extends Error {
readonly code = 'CLUSTER_APPROVAL_TRANSPORT_AUTHORIZATION_REJECTED';
constructor() {
super('Cluster Approval transport authorization was rejected');
this.name = 'ClusterApprovalManagementTransportAuthorizationError';
}
}
export class ClusterApprovalManagementTransportTargetUnavailableError extends Error {
readonly code = 'CLUSTER_APPROVAL_TRANSPORT_TARGET_UNAVAILABLE';
constructor() {
super('Cluster Approval target is unavailable');
this.name = 'ClusterApprovalManagementTransportTargetUnavailableError';
}
}
export class ClusterApprovalManagementTransportConflictError extends Error {
readonly code = 'CLUSTER_APPROVAL_TRANSPORT_CONFLICT';
constructor() {
super('Cluster Approval transport observed a conflict');
this.name = 'ClusterApprovalManagementTransportConflictError';
}
}
export class ClusterApprovalManagementTransportUnavailableError extends Error {
readonly code = 'CLUSTER_APPROVAL_TRANSPORT_UNAVAILABLE';
constructor() {
super('Cluster Approval transport is unavailable');
this.name = 'ClusterApprovalManagementTransportUnavailableError';
}
}
function invalid(): never {
throw new ClusterApprovalManagementTransportRequestError();
}
function exact(value: unknown, keys: readonly string[]): Record<string, unknown> {
if (!value || typeof value !== 'object' || Array.isArray(value)) invalid();
const actual = Object.keys(value as object).sort();
const expected = [...keys].sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
invalid();
}
return value as Record<string, unknown>;
}
function identifier(value: unknown): string {
if (typeof value !== 'string' || !IDENTIFIER_PATTERN.test(value)) invalid();
return value;
}
function uuid(value: unknown): string {
if (typeof value !== 'string' || !UUID_V4_PATTERN.test(value)) invalid();
return value;
}
export function normalizeClusterApprovalManagementCommand(
value: unknown,
): Readonly<ClusterApprovalManagementCommand> {
const envelope = exact(value, ['schemaVersion', 'operation', 'request']);
if (
envelope.schemaVersion !== 1 ||
(envelope.operation !== 'approval.inspect' &&
envelope.operation !== 'approval.decide')
) {
invalid();
}
const operation = envelope.operation;
const base = [
'projectId',
'approvalRequestId',
'requestId',
'auditEventId',
'failureAuditEventId',
];
const request = exact(
envelope.request,
operation === 'approval.inspect'
? base
: [
...base,
'expectedVersion',
'expectedAction',
'decisionId',
'decision',
'reasonCode',
],
);
const normalizedBase = {
projectId: identifier(request.projectId),
approvalRequestId: identifier(request.approvalRequestId),
requestId: identifier(request.requestId),
auditEventId: uuid(request.auditEventId),
failureAuditEventId: uuid(request.failureAuditEventId),
};
if (normalizedBase.auditEventId === normalizedBase.failureAuditEventId) invalid();
if (operation === 'approval.inspect') {
return Object.freeze({
schemaVersion: 1,
operation,
request: Object.freeze(normalizedBase),
});
}
if (
request.expectedVersion !== 1 ||
(request.decision !== 'approved' && request.decision !== 'rejected') ||
typeof request.reasonCode !== 'string' ||
!REASON_PATTERN.test(request.reasonCode)
) {
invalid();
}
let expectedAction: Readonly<ApprovedActionBinding>;
try {
expectedAction = normalizeApprovedActionBinding(
request.expectedAction as ApprovedActionBinding,
);
} catch {
invalid();
}
return Object.freeze({
schemaVersion: 1,
operation,
request: Object.freeze({
...normalizedBase,
expectedVersion: 1,
expectedAction,
decisionId: identifier(request.decisionId),
decision: request.decision,
reasonCode: request.reasonCode,
}),
});
}
function authenticatedPrincipal(
candidate: Readonly<SecurityPrincipal> | null,
nowMs: number,
): Readonly<SecurityPrincipal> {
let principal: Readonly<SecurityPrincipal>;
try {
principal = normalizeSecurityPrincipal(candidate as SecurityPrincipal, nowMs);
} catch {
throw new ClusterApprovalManagementTransportAuthenticationError();
}
if (
principal.subject.type !== 'user' ||
!STRONG_ASSURANCES.has(principal.assurance)
) {
throw new ClusterApprovalManagementTransportAuthenticationError();
}
return principal;
}
function samePrincipal(
left: Readonly<SecurityPrincipal>,
right: Readonly<SecurityPrincipal>,
): boolean {
return (
left.subject.type === right.subject.type &&
left.subject.id === right.subject.id &&
left.authenticationId === right.authenticationId &&
left.authenticatedAtMs === right.authenticatedAtMs &&
left.expiresAtMs === right.expiresAtMs &&
left.assurance === right.assurance
);
}
function failureReason(error: unknown, authenticated: boolean): Readonly<{
outcome: SecurityAuditRecord['outcome'];
reason: string;
}> {
if (error instanceof ClusterApprovalManagementTransportAuthenticationError) {
return authenticated
? Object.freeze({
outcome: 'denied',
reason: 'identity_confirmation_rejected',
})
: Object.freeze({
outcome: 'authentication_rejected',
reason: 'identity_assertion_rejected',
});
}
if (
error instanceof ApprovalInspectionAuthorizationError ||
error instanceof ApprovalDecisionAuthorizationError
) {
return Object.freeze({ outcome: 'denied', reason: 'policy_rejected' });
}
if (error instanceof ApprovalDecisionTargetUnavailableError) {
return Object.freeze({ outcome: 'denied', reason: 'approval_target_unavailable' });
}
if (error instanceof ApprovalDecisionBindingConflictError) {
return Object.freeze({ outcome: 'denied', reason: 'approval_binding_conflict' });
}
if (
error instanceof ApprovalRequestVersionConflictError ||
error instanceof ApprovalRequestStateConflictError ||
error instanceof ApprovalRequestExpiredError ||
error instanceof ApprovalMutationConflictError ||
error instanceof ApprovalPolicyFenceConflictError
) {
return Object.freeze({ outcome: 'denied', reason: 'approval_state_or_fence_conflict' });
}
return Object.freeze({
outcome: 'authorization_unavailable',
reason: 'approval_authority_unavailable',
});
}
function observedTime(now: () => number): number {
const value = now();
if (!Number.isSafeInteger(value) || value < 0) {
throw new ClusterApprovalManagementTransportUnavailableError();
}
return value;
}
function mapped(error: unknown): Error {
if (
error instanceof ApprovalInspectionAuthorizationError ||
error instanceof ApprovalDecisionAuthorizationError
) {
return new ClusterApprovalManagementTransportAuthorizationError();
}
if (error instanceof ApprovalDecisionTargetUnavailableError) {
return new ClusterApprovalManagementTransportTargetUnavailableError();
}
if (
error instanceof ApprovalDecisionBindingConflictError ||
error instanceof ApprovalRequestVersionConflictError ||
error instanceof ApprovalRequestStateConflictError ||
error instanceof ApprovalRequestExpiredError ||
error instanceof ApprovalMutationConflictError ||
error instanceof ApprovalPolicyFenceConflictError
) {
return new ClusterApprovalManagementTransportConflictError();
}
if (
error instanceof ApprovalInspectionUnavailableError ||
error instanceof ApprovalDecisionUnavailableError
) {
return new ClusterApprovalManagementTransportUnavailableError();
}
return error instanceof Error
? error
: new ClusterApprovalManagementTransportUnavailableError();
}
export function createClusterApprovalManagementTransport(options: Readonly<{
service: ClusterApprovalManagementService;
now?: () => number;
}>): Readonly<ClusterApprovalManagementTransport> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some((key) => key !== 'service' && key !== 'now') ||
typeof options.service?.inspect !== 'function' ||
typeof options.service?.decide !== 'function' ||
typeof options.service?.recordFailure !== 'function' ||
(options.now !== undefined && typeof options.now !== 'function')
) {
throw new ClusterApprovalManagementTransportConfigurationError();
}
const now = options.now ?? Date.now;
return Object.freeze({
async execute(
commandValue: unknown,
authentication: ClusterApprovalManagementAuthentication,
) {
const command = normalizeClusterApprovalManagementCommand(commandValue);
if (
!authentication ||
typeof authentication !== 'object' ||
Array.isArray(authentication) ||
Object.keys(authentication).length !== 1 ||
typeof authentication.authenticate !== 'function'
) {
throw new ClusterApprovalManagementTransportConfigurationError();
}
let principal: Readonly<SecurityPrincipal> | undefined;
try {
try {
principal = authenticatedPrincipal(
await authentication.authenticate(),
observedTime(now),
);
} catch (error) {
if (error instanceof ClusterApprovalManagementTransportAuthenticationError) {
throw error;
}
throw new ClusterApprovalManagementTransportUnavailableError();
}
const confirmAuthorization = async (): Promise<void> => {
let confirmed: Readonly<SecurityPrincipal>;
try {
confirmed = authenticatedPrincipal(
await authentication.authenticate(),
observedTime(now),
);
} catch {
throw new ClusterApprovalManagementTransportAuthenticationError();
}
if (!samePrincipal(principal!, confirmed)) {
throw new ClusterApprovalManagementTransportAuthenticationError();
}
};
if (command.operation === 'approval.inspect') {
const detail = await options.service.inspect(
{
projectId: command.request.projectId,
approvalRequestId: command.request.approvalRequestId,
auditEventId: command.request.auditEventId,
requestId: command.request.requestId,
principal,
},
confirmAuthorization,
);
if (!detail) {
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
status: 'absent' as const,
approval: null,
});
}
const request = detail.request;
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
status: 'found' as const,
approval: Object.freeze({
projectId: request.projectId,
approvalRequestId: request.id,
version: request.version,
state: request.state,
risk: request.risk,
decisionMode: request.decisionMode,
expectedAction: request.action,
requestedBy: request.requestedBy,
requestedAtMs: request.requestedAtMs,
expiresAtMs: request.expiresAtMs,
preview: detail.preview,
}),
});
}
const result = await options.service.decide(
{
projectId: command.request.projectId,
approvalRequestId: command.request.approvalRequestId,
expectedVersion: command.request.expectedVersion,
expectedAction: command.request.expectedAction,
decisionId: command.request.decisionId,
decision: command.request.decision,
reasonCode: command.request.reasonCode,
auditEventId: command.request.auditEventId,
requestId: command.request.requestId,
principal,
},
confirmAuthorization,
);
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
status: result.status,
approval: Object.freeze({
projectId: result.request.projectId,
approvalRequestId: result.request.id,
version: result.request.version,
state: result.request.state,
expectedAction: result.request.action,
decisionId: result.request.decisionId,
decision: result.request.decision,
reasonCode: result.request.decisionReasonCode,
decidedBy: result.request.decidedBy,
decidedAtMs: result.request.decidedAtMs,
}),
});
} catch (error) {
if (!(error instanceof ClusterApprovalManagementTransportConfigurationError)) {
const fact = failureReason(error, principal !== undefined);
try {
await options.service.recordFailure({
eventId: command.request.failureAuditEventId,
requestId: command.request.requestId,
operationId: command.operation,
projectId: command.request.projectId,
subject: principal?.subject ?? null,
authenticationId: principal?.authenticationId ?? null,
outcome: fact.outcome,
reasons: Object.freeze([fact.reason]),
fence: null,
occurredAtMs: observedTime(now),
});
} catch {
throw new ClusterApprovalManagementTransportUnavailableError();
}
}
throw mapped(error);
}
},
});
}
@@ -0,0 +1,593 @@
/** Automation-management application service boundary. */
import {
InvalidTaskDefinitionError,
TaskDefinitionConflictError,
TaskDefinitionUnavailableError,
assertTaskDefinitionIdentifier,
assertTaskDefinitionPageSize,
normalizeAppendTaskDefinitionRevisionCommand,
normalizeTaskDefinitionCursor,
type AppendTaskDefinitionRevisionCommand,
type TaskDefinitionCursor,
type TaskDefinitionPage,
type TaskDefinitionRecord,
} from '@qinglong/runtime-core/task-definition';
import {
InvalidTaskDefinitionAdministrationReadError,
InvalidTaskDefinitionAdministrationMutationError,
TaskDefinitionAdministrationAuthorizationFenceConflictError,
TaskDefinitionAdministrationMutationConflictError,
TaskDefinitionAdministrationReadConflictError,
type TaskDefinitionAdministrationRepository,
type TaskDefinitionAdministrationSource,
} from '@qinglong/runtime-core/task-definition-administration';
import {
InvalidTriggerError,
TriggerConflictError,
TriggerUnavailableError,
assertTriggerIdentifier,
assertTriggerPageSize,
normalizeAppendTriggerRevisionCommand,
normalizeTriggerCursor,
type AppendTriggerRevisionCommand,
type TriggerCursor,
type TriggerPage,
type TriggerRecord,
} from '@qinglong/runtime-core/trigger';
import {
InvalidTriggerAdministrationReadError,
InvalidTriggerAdministrationMutationError,
TriggerAdministrationAuthorizationFenceConflictError,
TriggerAdministrationMutationConflictError,
TriggerAdministrationReadConflictError,
type TriggerAdministrationRepository,
type TriggerAdministrationSource,
} from '@qinglong/runtime-core/trigger-administration';
import type { ProjectPermission } from '@qinglong/runtime-core/project-policy';
import {
normalizeSecurityPrincipal,
type SecurityPolicyDecision,
type SecurityPrincipal,
} from '@qinglong/runtime-core/security';
const REQUEST_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const AUDIT_EVENT_ID_PATTERN =
/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/;
const STRONG_USER_ASSURANCES = new Set(['multi_factor', 'hardware']);
export interface ClusterAutomationManagementPolicy {
authorize(
principal: Readonly<SecurityPrincipal>,
projectId: string,
permission: ProjectPermission,
): Promise<Readonly<SecurityPolicyDecision>>;
}
export interface ClusterAutomationManagementService {
publishTask(request: Readonly<{
requestId: string;
command: AppendTaskDefinitionRevisionCommand;
principal: SecurityPrincipal;
}>): Promise<Readonly<{
status: 'created' | 'updated' | 'existing';
definition: TaskDefinitionRecord;
}>>;
publishTrigger(request: Readonly<{
requestId: string;
command: AppendTriggerRevisionCommand;
principal: SecurityPrincipal;
}>): Promise<Readonly<{
status: 'created' | 'updated' | 'existing';
trigger: TriggerRecord;
}>>;
inspectTask(request: Readonly<{
requestId: string;
auditEventId: string;
projectId: string;
taskId: string;
principal: SecurityPrincipal;
}>): Promise<TaskDefinitionRecord | null>;
listTasks(request: Readonly<{
requestId: string;
auditEventId: string;
projectId: string;
limit: number;
after?: TaskDefinitionCursor;
principal: SecurityPrincipal;
}>): Promise<TaskDefinitionPage>;
inspectTrigger(request: Readonly<{
requestId: string;
auditEventId: string;
projectId: string;
triggerId: string;
principal: SecurityPrincipal;
}>): Promise<TriggerRecord | null>;
listTriggers(request: Readonly<{
requestId: string;
auditEventId: string;
projectId: string;
limit: number;
after?: TriggerCursor;
principal: SecurityPrincipal;
}>): Promise<TriggerPage>;
}
export interface ClusterAutomationManagementOptions {
readonly policy: ClusterAutomationManagementPolicy;
readonly taskDefinitions: TaskDefinitionAdministrationRepository &
TaskDefinitionAdministrationSource;
readonly triggers: TriggerAdministrationRepository &
TriggerAdministrationSource;
readonly now?: () => number;
}
export class ClusterAutomationManagementRequestError extends TypeError {
readonly code = 'CLUSTER_AUTOMATION_MANAGEMENT_REQUEST_INVALID';
constructor() {
super('Cluster automation management request is invalid');
this.name = 'ClusterAutomationManagementRequestError';
}
}
export class ClusterAutomationManagementAuthorizationError extends Error {
readonly code = 'CLUSTER_AUTOMATION_MANAGEMENT_FORBIDDEN';
constructor() {
super('Cluster automation management is forbidden');
this.name = 'ClusterAutomationManagementAuthorizationError';
}
}
export class ClusterAutomationManagementConflictError extends Error {
readonly code = 'CLUSTER_AUTOMATION_MANAGEMENT_CONFLICT';
constructor() {
super('Cluster automation management conflicts with durable state');
this.name = 'ClusterAutomationManagementConflictError';
}
}
export class ClusterAutomationManagementUnavailableError extends Error {
readonly code = 'CLUSTER_AUTOMATION_MANAGEMENT_UNAVAILABLE';
constructor() {
super('Cluster automation management is unavailable');
this.name = 'ClusterAutomationManagementUnavailableError';
}
}
function exactRequest(value: unknown): void {
if (
!value ||
typeof value !== 'object' ||
Array.isArray(value) ||
Object.keys(value).length !== 3 ||
!Object.hasOwn(value, 'requestId') ||
!Object.hasOwn(value, 'command') ||
!Object.hasOwn(value, 'principal') ||
typeof (value as { requestId?: unknown }).requestId !== 'string' ||
!REQUEST_ID_PATTERN.test((value as { requestId: string }).requestId)
) {
throw new ClusterAutomationManagementRequestError();
}
}
function mapMutationError(error: unknown): never {
if (
error instanceof InvalidTaskDefinitionError ||
error instanceof InvalidTaskDefinitionAdministrationMutationError ||
error instanceof InvalidTriggerError ||
error instanceof InvalidTriggerAdministrationMutationError
) {
throw new ClusterAutomationManagementRequestError();
}
if (
error instanceof TaskDefinitionConflictError ||
error instanceof TaskDefinitionAdministrationAuthorizationFenceConflictError ||
error instanceof TaskDefinitionAdministrationMutationConflictError ||
error instanceof TriggerConflictError ||
error instanceof TriggerAdministrationAuthorizationFenceConflictError ||
error instanceof TriggerAdministrationMutationConflictError
) {
throw new ClusterAutomationManagementConflictError();
}
if (
error instanceof TaskDefinitionUnavailableError ||
error instanceof TriggerUnavailableError
) {
throw new ClusterAutomationManagementUnavailableError();
}
throw new ClusterAutomationManagementUnavailableError();
}
function mapReadError(error: unknown): never {
if (
error instanceof InvalidTaskDefinitionError ||
error instanceof InvalidTaskDefinitionAdministrationReadError ||
error instanceof InvalidTriggerError ||
error instanceof InvalidTriggerAdministrationReadError
) {
throw new ClusterAutomationManagementRequestError();
}
if (
error instanceof TaskDefinitionAdministrationAuthorizationFenceConflictError ||
error instanceof TaskDefinitionAdministrationReadConflictError ||
error instanceof TriggerAdministrationAuthorizationFenceConflictError ||
error instanceof TriggerAdministrationReadConflictError
) {
throw new ClusterAutomationManagementConflictError();
}
throw new ClusterAutomationManagementUnavailableError();
}
function exactReadBase(
value: unknown,
required: readonly string[],
optional: readonly string[] = [],
): asserts value is Record<string, unknown> {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw new ClusterAutomationManagementRequestError();
}
const keys = Object.keys(value);
if (
!required.every((key) => keys.includes(key)) ||
keys.some((key) => !required.includes(key) && !optional.includes(key)) ||
typeof (value as { requestId?: unknown }).requestId !== 'string' ||
!REQUEST_ID_PATTERN.test((value as { requestId: string }).requestId) ||
typeof (value as { auditEventId?: unknown }).auditEventId !== 'string' ||
!AUDIT_EVENT_ID_PATTERN.test(
(value as { auditEventId: string }).auditEventId,
)
) {
throw new ClusterAutomationManagementRequestError();
}
}
function readAudit(
request: Readonly<{
requestId: string;
auditEventId: string;
projectId: string;
}>,
operationId: 'task.read' | 'trigger.read',
authority: Readonly<{
principal: Readonly<SecurityPrincipal>;
decision: Readonly<SecurityPolicyDecision>;
observedAtMs: number;
}>,
) {
return Object.freeze({
eventId: request.auditEventId,
requestId: request.requestId,
operationId,
projectId: request.projectId,
subject: authority.principal.subject,
authenticationId: authority.principal.authenticationId,
outcome: 'allowed' as const,
reasons: authority.decision.reasons,
fence: authority.decision.fence,
occurredAtMs: authority.observedAtMs,
});
}
export function createClusterAutomationManagementService(
options: ClusterAutomationManagementOptions,
): Readonly<ClusterAutomationManagementService> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
key !== 'policy' &&
key !== 'taskDefinitions' &&
key !== 'triggers' &&
key !== 'now',
) ||
!options.policy ||
typeof options.policy.authorize !== 'function' ||
!options.taskDefinitions ||
typeof options.taskDefinitions.appendAuthorizedTaskDefinitionRevision !==
'function' ||
typeof options.taskDefinitions.findAuthorizedCurrentTaskDefinition !==
'function' ||
typeof options.taskDefinitions.listAuthorizedTaskDefinitions !==
'function' ||
!options.triggers ||
typeof options.triggers.appendAuthorizedTriggerRevision !== 'function' ||
typeof options.triggers.findAuthorizedCurrentTrigger !== 'function' ||
typeof options.triggers.listAuthorizedTriggers !== 'function' ||
(options.now !== undefined && typeof options.now !== 'function')
) {
throw new TypeError('Cluster automation management options are invalid');
}
const now = options.now ?? Date.now;
const authorize = async (
principalValue: SecurityPrincipal,
projectId: string,
permission: ProjectPermission,
) => {
const observedAtMs = now();
if (!Number.isSafeInteger(observedAtMs) || observedAtMs < 0) {
throw new ClusterAutomationManagementUnavailableError();
}
let principal: Readonly<SecurityPrincipal>;
try {
principal = normalizeSecurityPrincipal(principalValue, observedAtMs);
} catch {
throw new ClusterAutomationManagementAuthorizationError();
}
if (
principal.subject.type !== 'user' ||
!STRONG_USER_ASSURANCES.has(principal.assurance)
) {
throw new ClusterAutomationManagementAuthorizationError();
}
let decision: Readonly<SecurityPolicyDecision>;
try {
decision = await options.policy.authorize(
principal,
projectId,
permission,
);
} catch {
throw new ClusterAutomationManagementUnavailableError();
}
if (
decision.effect !== 'allow' ||
decision.fence === null ||
decision.fence.bindingVersion === null
) {
throw new ClusterAutomationManagementAuthorizationError();
}
return Object.freeze({ principal, decision, observedAtMs });
};
return Object.freeze({
async publishTask(
request: Parameters<ClusterAutomationManagementService['publishTask']>[0],
) {
exactRequest(request);
let command: Readonly<AppendTaskDefinitionRevisionCommand>;
try {
command = normalizeAppendTaskDefinitionRevisionCommand(request.command);
} catch (error) {
return mapMutationError(error);
}
const operation =
command.expectedRevision === null ? 'task.create' : 'task.update';
const authority = await authorize(
request.principal,
command.projectId,
operation,
);
try {
return await options.taskDefinitions.appendAuthorizedTaskDefinitionRevision(
{
command,
actor: authority.principal.subject,
fence: authority.decision.fence!,
audit: {
eventId: command.mutationId,
requestId: request.requestId,
operationId: operation,
projectId: command.projectId,
subject: authority.principal.subject,
authenticationId: authority.principal.authenticationId,
outcome: 'allowed',
reasons: authority.decision.reasons,
fence: authority.decision.fence,
occurredAtMs: authority.observedAtMs,
},
},
);
} catch (error) {
return mapMutationError(error);
}
},
async publishTrigger(
request: Parameters<ClusterAutomationManagementService['publishTrigger']>[0],
) {
exactRequest(request);
let command: Readonly<AppendTriggerRevisionCommand>;
try {
command = normalizeAppendTriggerRevisionCommand(request.command);
} catch (error) {
return mapMutationError(error);
}
const operation =
command.expectedRevision === null
? 'trigger.create'
: 'trigger.update';
const authority = await authorize(
request.principal,
command.projectId,
operation,
);
try {
return await options.triggers.appendAuthorizedTriggerRevision({
command,
actor: authority.principal.subject,
fence: authority.decision.fence!,
audit: {
eventId: command.mutationId,
requestId: request.requestId,
operationId: operation,
projectId: command.projectId,
subject: authority.principal.subject,
authenticationId: authority.principal.authenticationId,
outcome: 'allowed',
reasons: authority.decision.reasons,
fence: authority.decision.fence,
occurredAtMs: authority.observedAtMs,
},
});
} catch (error) {
return mapMutationError(error);
}
},
async inspectTask(
request: Parameters<ClusterAutomationManagementService['inspectTask']>[0],
) {
exactReadBase(request, [
'auditEventId',
'principal',
'projectId',
'requestId',
'taskId',
]);
try {
assertTaskDefinitionIdentifier(request.projectId, 'projectId');
assertTaskDefinitionIdentifier(request.taskId, 'taskId');
} catch {
throw new ClusterAutomationManagementRequestError();
}
const authority = await authorize(
request.principal,
request.projectId,
'task.read',
);
try {
return await options.taskDefinitions.findAuthorizedCurrentTaskDefinition(
{
projectId: request.projectId,
taskId: request.taskId,
actor: authority.principal.subject,
fence: authority.decision.fence!,
audit: readAudit(request, 'task.read', authority),
},
);
} catch (error) {
return mapReadError(error);
}
},
async listTasks(
request: Parameters<ClusterAutomationManagementService['listTasks']>[0],
) {
exactReadBase(
request,
[
'auditEventId',
'limit',
'principal',
'projectId',
'requestId',
],
['after'],
);
let after: TaskDefinitionCursor | undefined;
try {
assertTaskDefinitionIdentifier(request.projectId, 'projectId');
assertTaskDefinitionPageSize(request.limit);
after = Object.hasOwn(request, 'after')
? normalizeTaskDefinitionCursor(request.after as TaskDefinitionCursor)
: undefined;
} catch {
throw new ClusterAutomationManagementRequestError();
}
const authority = await authorize(
request.principal,
request.projectId,
'task.read',
);
try {
return await options.taskDefinitions.listAuthorizedTaskDefinitions({
projectId: request.projectId,
limit: request.limit,
...(after ? { after } : {}),
actor: authority.principal.subject,
fence: authority.decision.fence!,
audit: readAudit(request, 'task.read', authority),
});
} catch (error) {
return mapReadError(error);
}
},
async inspectTrigger(
request: Parameters<
ClusterAutomationManagementService['inspectTrigger']
>[0],
) {
exactReadBase(request, [
'auditEventId',
'principal',
'projectId',
'requestId',
'triggerId',
]);
try {
assertTriggerIdentifier(request.projectId, 'projectId');
assertTriggerIdentifier(request.triggerId, 'triggerId');
} catch {
throw new ClusterAutomationManagementRequestError();
}
const authority = await authorize(
request.principal,
request.projectId,
'trigger.read',
);
try {
return await options.triggers.findAuthorizedCurrentTrigger({
projectId: request.projectId,
triggerId: request.triggerId,
actor: authority.principal.subject,
fence: authority.decision.fence!,
audit: readAudit(request, 'trigger.read', authority),
});
} catch (error) {
return mapReadError(error);
}
},
async listTriggers(
request: Parameters<
ClusterAutomationManagementService['listTriggers']
>[0],
) {
exactReadBase(
request,
[
'auditEventId',
'limit',
'principal',
'projectId',
'requestId',
],
['after'],
);
let after: TriggerCursor | undefined;
try {
assertTriggerIdentifier(request.projectId, 'projectId');
assertTriggerPageSize(request.limit);
after = Object.hasOwn(request, 'after')
? normalizeTriggerCursor(request.after as TriggerCursor)
: undefined;
} catch {
throw new ClusterAutomationManagementRequestError();
}
const authority = await authorize(
request.principal,
request.projectId,
'trigger.read',
);
try {
return await options.triggers.listAuthorizedTriggers({
projectId: request.projectId,
limit: request.limit,
...(after ? { after } : {}),
actor: authority.principal.subject,
fence: authority.decision.fence!,
audit: readAudit(request, 'trigger.read', authority),
});
} catch (error) {
return mapReadError(error);
}
},
});
}
@@ -0,0 +1,107 @@
#!/usr/bin/env node
import {
startClusterAutomationManagementProcess,
type ClusterAutomationManagementProcessRuntime,
} from './automationManagementProcess';
const USAGE = 'Usage: ql3-automation-manage';
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as { readonly name?: unknown; readonly code?: unknown };
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-automation-management',
event: 'management_failed',
name:
typeof candidate?.name === 'string' && candidate.name.length <= 128
? candidate.name
: 'Error',
...(typeof candidate?.code === 'string' && candidate.code.length <= 128
? { code: candidate.code }
: {}),
});
}
function emit(value: Readonly<Record<string, unknown>>): void {
process.stdout.write(`${JSON.stringify(value)}\n`);
}
async function run(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
if (argv.length !== 0) {
process.stderr.write(
`${JSON.stringify({
code: 'QL3_AUTOMATION_MANAGEMENT_CLI_USAGE_INVALID',
message: USAGE,
})}\n`,
);
process.exitCode = 64;
return;
}
let runtime: Readonly<ClusterAutomationManagementProcessRuntime>;
try {
runtime = await startClusterAutomationManagementProcess({
environment: process.env,
onError() {
emit({
schemaVersion: 1,
component: 'qinglong3-automation-management',
event: 'management_unavailable',
});
},
});
} catch (error) {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
return;
}
if (runtime.status === 'disabled') {
emit({
schemaVersion: 1,
component: 'qinglong3-automation-management',
event: 'management_disabled',
});
return;
}
emit({
schemaVersion: 1,
component: 'qinglong3-automation-management',
event: 'management_started',
address: runtime.address,
identityGeneration: runtime.identity.generation,
databaseContractVersion: runtime.database.contractVersion,
databaseMigrationCount: runtime.database.migrationIds.length,
});
let stopping: Promise<void> | undefined;
const stop = (): Promise<void> => {
stopping ??= runtime.close().then(() => {
emit({
schemaVersion: 1,
component: 'qinglong3-automation-management',
event: 'management_stopped',
});
});
return stopping;
};
for (const signal of ['SIGINT', 'SIGTERM'] as const) {
process.once(signal, () => {
void stop().then(
() => {
process.exitCode = 0;
},
(error) => {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
},
);
});
}
}
void run(process.argv.slice(2));
@@ -0,0 +1,318 @@
import {
ClusterPluginPackageManagementClientRequestError,
executeClusterAuthenticatedManagementClient,
type ClusterAuthenticatedManagementClientResult,
type ClusterPluginPackageManagementClientConnectionOptions,
type ClusterPluginPackageManagementClientPaths,
} from '../management-support/pluginPackageManagementClient';
import {
normalizeClusterAutomationManagementCommand,
type ClusterAutomationManagementCommand,
type ClusterAutomationManagementTransportResult,
} from './automationManagementTransport';
const MANAGEMENT_PATH = '/api/v3/automations/management';
const CONTROL_PATTERN = /[\u0000-\u001f\u007f]/;
const DIGEST_PATTERN = /^[0-9a-f]{64}$/;
export type ClusterAutomationManagementClientPaths =
ClusterPluginPackageManagementClientPaths;
export type ClusterAutomationManagementClientConnectionOptions =
ClusterPluginPackageManagementClientConnectionOptions;
export type ClusterAutomationManagementClientResult =
ClusterAuthenticatedManagementClientResult<ClusterAutomationManagementTransportResult>;
function invalid(): never {
throw new ClusterPluginPackageManagementClientRequestError();
}
function exactRecord(
value: unknown,
keys: readonly string[],
): Record<string, unknown> {
if (!value || typeof value !== 'object' || Array.isArray(value)) invalid();
const record = value as Record<string, unknown>;
const actual = Object.keys(record).sort();
const expected = [...keys].sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
invalid();
}
return record;
}
function identifier(value: unknown): string {
if (
typeof value !== 'string' ||
value.length < 1 ||
value.length > 128 ||
CONTROL_PATTERN.test(value)
) {
invalid();
}
return value;
}
function positiveRevision(value: unknown): number {
if (!Number.isSafeInteger(value) || (value as number) < 1) invalid();
return value as number;
}
function digest(value: unknown): string {
if (typeof value !== 'string' || !DIGEST_PATTERN.test(value)) invalid();
return value;
}
function taskResultSummary(
value: unknown,
projectId: string,
taskId?: string,
): Record<string, unknown> {
const task = exactRecord(value, [
'projectId',
'taskId',
'revision',
'kind',
'enabled',
'contentDigest',
'updatedAtMs',
]);
if (
identifier(task.projectId) !== projectId ||
(taskId !== undefined && identifier(task.taskId) !== taskId) ||
identifier(task.kind).length > 64 ||
typeof task.enabled !== 'boolean' ||
!Number.isSafeInteger(task.updatedAtMs) ||
(task.updatedAtMs as number) < 0
) {
invalid();
}
identifier(task.taskId);
positiveRevision(task.revision);
digest(task.contentDigest);
return task;
}
function triggerResultSummary(
value: unknown,
projectId: string,
triggerId?: string,
): Record<string, unknown> {
const trigger = exactRecord(value, [
'projectId',
'triggerId',
'revision',
'taskId',
'taskRevision',
'taskContentDigest',
'enabled',
'contentDigest',
'updatedAtMs',
]);
if (
identifier(trigger.projectId) !== projectId ||
(triggerId !== undefined && identifier(trigger.triggerId) !== triggerId) ||
typeof trigger.enabled !== 'boolean' ||
!Number.isSafeInteger(trigger.updatedAtMs) ||
(trigger.updatedAtMs as number) < 0
) {
invalid();
}
identifier(trigger.triggerId);
identifier(trigger.taskId);
positiveRevision(trigger.revision);
positiveRevision(trigger.taskRevision);
digest(trigger.taskContentDigest);
digest(trigger.contentDigest);
return trigger;
}
function validateListPage(
envelope: Record<string, unknown>,
itemsKey: 'tasks' | 'triggers',
limit: number,
afterId: string | undefined,
idKey: 'taskId' | 'triggerId',
validateItem: (value: unknown) => Record<string, unknown>,
): void {
if (
!Array.isArray(envelope[itemsKey]) ||
(envelope[itemsKey] as unknown[]).length > limit ||
typeof envelope.truncated !== 'boolean'
) {
invalid();
}
let previous = afterId;
for (const itemValue of envelope[itemsKey] as unknown[]) {
const item = validateItem(itemValue);
const current = identifier(item[idKey]);
if (previous !== undefined && current <= previous) invalid();
previous = current;
}
if (envelope.truncated) {
const next = exactRecord(envelope.next, [idKey]);
const nextId = identifier(next[idKey]);
if (previous === undefined || nextId !== previous) invalid();
} else if (envelope.next !== null) {
invalid();
}
}
export function validateClusterAutomationManagementClientResult(
value: unknown,
command: Readonly<ClusterAutomationManagementCommand>,
): Readonly<ClusterAutomationManagementTransportResult> {
const operation = command.operation;
if (operation === 'task.inspect') {
const envelope = exactRecord(value, [
'schemaVersion',
'operation',
'status',
'task',
]);
if (
envelope.schemaVersion !== 1 ||
envelope.operation !== operation ||
!['found', 'absent'].includes(String(envelope.status)) ||
(envelope.status === 'absent') !== (envelope.task === null)
) {
invalid();
}
if (envelope.task !== null) {
taskResultSummary(
envelope.task,
command.request.projectId,
command.request.taskId,
);
}
return Object.freeze(
envelope as unknown as ClusterAutomationManagementTransportResult,
);
}
if (operation === 'trigger.inspect') {
const envelope = exactRecord(value, [
'schemaVersion',
'operation',
'status',
'trigger',
]);
if (
envelope.schemaVersion !== 1 ||
envelope.operation !== operation ||
!['found', 'absent'].includes(String(envelope.status)) ||
(envelope.status === 'absent') !== (envelope.trigger === null)
) {
invalid();
}
if (envelope.trigger !== null) {
triggerResultSummary(
envelope.trigger,
command.request.projectId,
command.request.triggerId,
);
}
return Object.freeze(
envelope as unknown as ClusterAutomationManagementTransportResult,
);
}
if (operation === 'task.list') {
const envelope = exactRecord(value, [
'schemaVersion',
'operation',
'tasks',
'truncated',
'next',
]);
if (envelope.schemaVersion !== 1 || envelope.operation !== operation) {
invalid();
}
validateListPage(
envelope,
'tasks',
command.request.limit,
command.request.after?.taskId,
'taskId',
(item) => taskResultSummary(item, command.request.projectId),
);
return Object.freeze(
envelope as unknown as ClusterAutomationManagementTransportResult,
);
}
if (operation === 'trigger.list') {
const envelope = exactRecord(value, [
'schemaVersion',
'operation',
'triggers',
'truncated',
'next',
]);
if (envelope.schemaVersion !== 1 || envelope.operation !== operation) {
invalid();
}
validateListPage(
envelope,
'triggers',
command.request.limit,
command.request.after?.triggerId,
'triggerId',
(item) => triggerResultSummary(item, command.request.projectId),
);
return Object.freeze(
envelope as unknown as ClusterAutomationManagementTransportResult,
);
}
const envelope = exactRecord(
value,
operation === 'task.publish'
? ['schemaVersion', 'operation', 'status', 'task']
: ['schemaVersion', 'operation', 'status', 'trigger'],
);
if (
envelope.schemaVersion !== 1 ||
envelope.operation !== operation ||
!['created', 'updated', 'existing'].includes(String(envelope.status))
) {
invalid();
}
if (command.operation === 'task.publish') {
const requested = command.request.command;
taskResultSummary(envelope.task, requested.projectId, requested.taskId);
} else {
const requested = command.request.command;
const trigger = triggerResultSummary(
envelope.trigger,
requested.projectId,
requested.triggerId,
);
if (
identifier(trigger.taskId) !== requested.taskId ||
positiveRevision(trigger.taskRevision) !== requested.taskRevision ||
digest(trigger.taskContentDigest) !== requested.taskContentDigest
) {
invalid();
}
}
return Object.freeze(
envelope as unknown as ClusterAutomationManagementTransportResult,
);
}
const PROTOCOL = Object.freeze({
managementPath: MANAGEMENT_PATH,
clientCertificate: 'required' as const,
normalizeCommand: normalizeClusterAutomationManagementCommand,
validateResult: validateClusterAutomationManagementClientResult,
});
export async function executeClusterAutomationManagementClient(
paths: ClusterAutomationManagementClientPaths,
connectionOptions?: ClusterAutomationManagementClientConnectionOptions,
): Promise<Readonly<ClusterAutomationManagementClientResult>> {
return executeClusterAuthenticatedManagementClient(
paths,
PROTOCOL,
connectionOptions,
);
}
@@ -0,0 +1,95 @@
#!/usr/bin/env node
import { executeClusterAutomationManagementClient } from './automationManagementClient';
import { ClusterPluginPackageManagementClientRemoteError } from '../management-support/pluginPackageManagementClient';
const USAGE =
'Usage: ql3-automation-client --config=/absolute/client.json --command=/absolute/command.json --assertion=/absolute/assertion.jwt';
function parseArguments(
argv: readonly string[],
): Readonly<{
configFile: string;
commandFile: string;
assertionFile: string;
}> | null {
if (argv.length !== 3) return null;
const values = new Map<string, string>();
for (const argument of argv) {
const match = /^--(config|command|assertion)=(\/.+)$/.exec(argument);
if (!match || values.has(match[1]!)) return null;
values.set(match[1]!, match[2]!);
}
if (
!values.has('config') ||
!values.has('command') ||
!values.has('assertion')
) {
return null;
}
return Object.freeze({
configFile: values.get('config')!,
commandFile: values.get('command')!,
assertionFile: values.get('assertion')!,
});
}
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as { readonly code?: unknown };
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-automation-management-client',
event: 'command_failed',
code:
typeof candidate?.code === 'string' && candidate.code.length <= 128
? candidate.code
: 'QL3_AUTOMATION_MANAGEMENT_CLIENT_FAILED',
...(error instanceof ClusterPluginPackageManagementClientRemoteError
? {
statusCode: error.statusCode,
responseCode: error.responseCode,
requestId: error.requestId,
...(error.retryAfterSeconds === null
? {}
: { retryAfterSeconds: error.retryAfterSeconds }),
}
: {}),
});
}
async function run(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
const paths = parseArguments(argv);
if (!paths) {
process.stderr.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-automation-management-client',
event: 'usage_invalid',
code: 'QL3_AUTOMATION_MANAGEMENT_CLIENT_USAGE_INVALID',
})}\n`,
);
process.exitCode = 64;
return;
}
try {
const result = await executeClusterAutomationManagementClient(paths);
process.stdout.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-automation-management-client',
event: 'command_completed',
requestId: result.requestId,
result: result.result,
})}\n`,
);
} catch (error) {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
}
}
void run(process.argv.slice(2));
@@ -0,0 +1,24 @@
import {
CLUSTER_AUTOMATION_MANAGEMENT_PATH,
startClusterPluginPackageManagementHttp,
type ClusterPluginPackageManagementHttpApplication,
type StartClusterPluginPackageManagementHttpOptions,
} from '../management-support/pluginPackageManagementHttp';
export type ClusterAutomationManagementHttpApplication =
ClusterPluginPackageManagementHttpApplication;
export type StartClusterAutomationManagementHttpOptions = Omit<
StartClusterPluginPackageManagementHttpOptions,
'managementPath'
>;
/** Starts the shared bounded OIDC/mTLS HTTPS adapter on the automation-only path. */
export function startClusterAutomationManagementHttp(
options: StartClusterAutomationManagementHttpOptions,
): Promise<Readonly<ClusterAutomationManagementHttpApplication>> {
return startClusterPluginPackageManagementHttp({
...options,
managementPath: CLUSTER_AUTOMATION_MANAGEMENT_PATH,
});
}
@@ -0,0 +1,590 @@
import type {
OpenPostgresDatabase,
PostgresDatabaseResource,
} from '@qinglong/runtime-core';
import { ProjectPolicyEngine } from '@qinglong/runtime-core/project-policy';
import {
PostgresAutomationManagementIdentityKeysetLedgerRepository,
PostgresProjectPolicyRepository,
PostgresTaskDefinitionAdministrationRepository,
PostgresTriggerAdministrationRepository,
assertPostgresAutomationManagerSchemaReady,
createPostgresDatabaseOpener,
isPostgresTlsDnsServername,
loadPostgresCertificateAuthorityFile,
loadPostgresConnectionEnvironment,
type PostgresConnectionOptions,
type PostgresPoolOptions,
type PostgresSchemaReadinessReport,
} from '@qinglong/cluster-postgres/automation-manager';
import {
absoluteManagementEnvironmentFile,
booleanManagementEnvironmentValue,
boundedManagementEnvironmentValue,
integerManagementEnvironmentValue,
readManagementTlsFile,
} from '../management-support/managementProcessSupport';
import {
createClusterAutomationIdentityKeysetFile,
type ClusterPluginPackageIdentityKeysetFile,
type ClusterPluginPackageIdentityKeysetSnapshot,
} from '../management-support/pluginPackageIdentityKeyset';
import { createClusterAutomationManagementService } from './automationManagement';
import {
startClusterAutomationManagementHttp,
type ClusterAutomationManagementHttpApplication,
type StartClusterAutomationManagementHttpOptions,
} from './automationManagementHttp';
import { createClusterAutomationManagementTransport } from './automationManagementTransport';
import { validateClusterManagementClientTrust } from '../worker-credential/management-server/workerCredentialManagementMutualTls';
const SAFE_HOST = /^[A-Za-z0-9][A-Za-z0-9.:-]{0,254}$/;
const SAFE_APPLICATION_NAME = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,62}$/;
export type ClusterAutomationManagementProcessEnvironment = Readonly<
Record<string, string | undefined>
>;
export type ClusterAutomationManagementProcessConfig =
| Readonly<{ enabled: false }>
| Readonly<{
enabled: true;
profile: 'cluster-admin';
host: string;
port: number;
certificateFile: string;
privateKeyFile: string;
clientCertificateAuthorityFile: string;
clientCertificateRevocationListFile: string;
identityKeysetFile: string;
http: Readonly<{
maxBodyBytes: number;
maxConnections: number;
maxConcurrentRequests: number;
requestTimeoutMs: number;
drainTimeoutMs: number;
rateWindowMs: number;
peerRequestLimit: number;
globalRequestLimit: number;
maxRateLimitPeers: number;
}>;
database: Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}>;
}>;
export type ClusterAutomationManagementProcessRuntime =
| Readonly<{
status: 'disabled';
close(): Promise<void>;
}>
| Readonly<{
status: 'active';
address: Readonly<{ host: string; port: number }>;
database: PostgresSchemaReadinessReport;
identity: ClusterPluginPackageIdentityKeysetSnapshot;
availabilityStatus(): 'ready' | 'unavailable' | 'stopped';
close(): Promise<void>;
}>;
export interface StartClusterAutomationManagementProcessOptions {
readonly environment: ClusterAutomationManagementProcessEnvironment;
readonly openDatabase?: OpenPostgresDatabase;
readonly identities?: ClusterPluginPackageIdentityKeysetFile;
readonly assertReady?: (
pool: PostgresDatabaseResource['pool'],
) => Promise<PostgresSchemaReadinessReport>;
readonly startHttp?: (
options: StartClusterAutomationManagementHttpOptions,
) => Promise<Readonly<ClusterAutomationManagementHttpApplication>>;
readonly now?: () => number;
readonly onError?: (error: unknown) => void;
}
export class ClusterAutomationManagementProcessConfigError extends TypeError {
readonly code = 'QL3_AUTOMATION_MANAGEMENT_PROCESS_CONFIG_INVALID';
constructor(message: string) {
super(`Automation management process configuration is invalid: ${message}`);
this.name = 'ClusterAutomationManagementProcessConfigError';
}
}
function configFailure(
message: string,
): ClusterAutomationManagementProcessConfigError {
return new ClusterAutomationManagementProcessConfigError(message);
}
function boundedValue(
environment: ClusterAutomationManagementProcessEnvironment,
name: string,
maximumLength: number,
required = false,
): string | undefined {
return boundedManagementEnvironmentValue(
environment,
name,
maximumLength,
configFailure,
required,
);
}
function booleanValue(
environment: ClusterAutomationManagementProcessEnvironment,
name: string,
): boolean {
return booleanManagementEnvironmentValue(environment, name, configFailure);
}
function integerValue(
environment: ClusterAutomationManagementProcessEnvironment,
name: string,
fallback: number,
minimum: number,
maximum: number,
): number {
return integerManagementEnvironmentValue(
environment,
name,
fallback,
minimum,
maximum,
configFailure,
);
}
function absoluteFile(
environment: ClusterAutomationManagementProcessEnvironment,
name: string,
): string {
return absoluteManagementEnvironmentFile(environment, name, configFailure);
}
function loadConnection(
environment: ClusterAutomationManagementProcessEnvironment,
): Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}> {
let connection: PostgresConnectionOptions;
try {
connection = loadPostgresConnectionEnvironment(environment, {
connectionString: 'QL3_POSTGRES_AUTOMATION_MANAGER_URL',
host: 'QL3_POSTGRES_AUTOMATION_MANAGER_HOST',
port: 'QL3_POSTGRES_AUTOMATION_MANAGER_PORT',
database: 'QL3_POSTGRES_AUTOMATION_MANAGER_DATABASE',
user: 'QL3_POSTGRES_AUTOMATION_MANAGER_USER',
password: 'QL3_POSTGRES_AUTOMATION_MANAGER_PASSWORD',
});
} catch (error) {
throw configFailure(
error instanceof Error
? error.message
: 'PostgreSQL automation manager connection is invalid',
);
}
const mode =
environment.QL3_POSTGRES_AUTOMATION_MANAGER_TLS_MODE ?? 'verify-full';
if (mode !== 'verify-full' && mode !== 'disable') {
throw configFailure(
'QL3_POSTGRES_AUTOMATION_MANAGER_TLS_MODE must be verify-full or disable',
);
}
if (
mode === 'disable' &&
!booleanValue(environment, 'QL3_POSTGRES_AUTOMATION_MANAGER_ALLOW_INSECURE')
) {
throw configFailure(
'disabling automation manager PostgreSQL TLS requires QL3_POSTGRES_AUTOMATION_MANAGER_ALLOW_INSECURE=true',
);
}
const servername = boundedValue(
environment,
'QL3_POSTGRES_AUTOMATION_MANAGER_TLS_SERVERNAME',
253,
);
if (mode === 'verify-full' && !isPostgresTlsDnsServername(servername)) {
throw configFailure(
'QL3_POSTGRES_AUTOMATION_MANAGER_TLS_SERVERNAME must be an explicit DNS name',
);
}
const caFile = boundedValue(
environment,
'QL3_POSTGRES_AUTOMATION_MANAGER_TLS_CA_FILE',
4_096,
);
if (mode === 'disable' && caFile !== undefined) {
throw configFailure(
'QL3_POSTGRES_AUTOMATION_MANAGER_TLS_CA_FILE cannot be used when TLS is disabled',
);
}
let ca: string | undefined;
if (caFile !== undefined) {
try {
ca = loadPostgresCertificateAuthorityFile(caFile);
} catch {
throw configFailure(
'QL3_POSTGRES_AUTOMATION_MANAGER_TLS_CA_FILE is invalid',
);
}
}
const applicationName =
boundedValue(
environment,
'QL3_POSTGRES_AUTOMATION_MANAGER_APPLICATION_NAME',
63,
) ?? 'qinglong3-automation-manager';
if (!SAFE_APPLICATION_NAME.test(applicationName)) {
throw configFailure(
'QL3_POSTGRES_AUTOMATION_MANAGER_APPLICATION_NAME is invalid',
);
}
return Object.freeze({
connection: Object.freeze({
...connection,
tls:
mode === 'disable'
? Object.freeze({ mode: 'disable' as const })
: Object.freeze({
mode: 'verify-full' as const,
servername: servername!,
...(ca === undefined ? {} : { ca }),
}),
}),
pool: Object.freeze({
applicationName,
maxConnections: integerValue(
environment,
'QL3_POSTGRES_AUTOMATION_MANAGER_POOL_MAX',
2,
1,
4,
),
idleTimeoutMs: integerValue(
environment,
'QL3_POSTGRES_AUTOMATION_MANAGER_IDLE_TIMEOUT_MS',
10_000,
1_000,
60_000,
),
connectionTimeoutMs: integerValue(
environment,
'QL3_POSTGRES_AUTOMATION_MANAGER_CONNECTION_TIMEOUT_MS',
5_000,
100,
60_000,
),
}),
});
}
export function loadClusterAutomationManagementProcessConfig(
environment: ClusterAutomationManagementProcessEnvironment,
): Readonly<ClusterAutomationManagementProcessConfig> {
if (!environment || typeof environment !== 'object') {
throw configFailure('environment is invalid');
}
if (!booleanValue(environment, 'QL3_AUTOMATION_MANAGEMENT_ENABLED')) {
return Object.freeze({ enabled: false as const });
}
if (environment.QL3_PROFILE !== 'cluster-admin') {
throw configFailure(
'QL3_PROFILE must be cluster-admin when automation management is enabled',
);
}
const host =
boundedValue(environment, 'QL3_AUTOMATION_MANAGEMENT_HOST', 255) ??
'0.0.0.0';
if (!SAFE_HOST.test(host)) {
throw configFailure('QL3_AUTOMATION_MANAGEMENT_HOST is invalid');
}
const http = Object.freeze({
maxBodyBytes: integerValue(
environment,
'QL3_AUTOMATION_MANAGEMENT_MAX_BODY_BYTES',
64 * 1024,
1_024,
256 * 1024,
),
maxConnections: integerValue(
environment,
'QL3_AUTOMATION_MANAGEMENT_MAX_CONNECTIONS',
32,
1,
512,
),
maxConcurrentRequests: integerValue(
environment,
'QL3_AUTOMATION_MANAGEMENT_MAX_CONCURRENT_REQUESTS',
16,
1,
256,
),
requestTimeoutMs: integerValue(
environment,
'QL3_AUTOMATION_MANAGEMENT_REQUEST_TIMEOUT_MS',
10_000,
1_000,
60_000,
),
drainTimeoutMs: integerValue(
environment,
'QL3_AUTOMATION_MANAGEMENT_DRAIN_TIMEOUT_MS',
5_000,
100,
60_000,
),
rateWindowMs: integerValue(
environment,
'QL3_AUTOMATION_MANAGEMENT_RATE_WINDOW_MS',
60_000,
1_000,
5 * 60_000,
),
peerRequestLimit: integerValue(
environment,
'QL3_AUTOMATION_MANAGEMENT_PEER_REQUEST_LIMIT',
60,
1,
10_000,
),
globalRequestLimit: integerValue(
environment,
'QL3_AUTOMATION_MANAGEMENT_GLOBAL_REQUEST_LIMIT',
600,
1,
100_000,
),
maxRateLimitPeers: integerValue(
environment,
'QL3_AUTOMATION_MANAGEMENT_MAX_RATE_LIMIT_PEERS',
1_024,
1,
16_384,
),
});
if (http.globalRequestLimit < http.peerRequestLimit) {
throw configFailure(
'global request limit cannot be below the peer request limit',
);
}
return Object.freeze({
enabled: true as const,
profile: 'cluster-admin' as const,
host,
port: integerValue(
environment,
'QL3_AUTOMATION_MANAGEMENT_PORT',
8_445,
1,
65_535,
),
certificateFile: absoluteFile(
environment,
'QL3_AUTOMATION_MANAGEMENT_TLS_CERT_FILE',
),
privateKeyFile: absoluteFile(
environment,
'QL3_AUTOMATION_MANAGEMENT_TLS_KEY_FILE',
),
clientCertificateAuthorityFile: absoluteFile(
environment,
'QL3_AUTOMATION_MANAGEMENT_CLIENT_CA_FILE',
),
clientCertificateRevocationListFile: absoluteFile(
environment,
'QL3_AUTOMATION_MANAGEMENT_CLIENT_CRL_FILE',
),
identityKeysetFile: absoluteFile(
environment,
'QL3_AUTOMATION_MANAGEMENT_IDENTITY_KEYSET_FILE',
),
http,
database: loadConnection(environment),
});
}
function readTlsFile(filePath: string, privateMaterial: boolean): Buffer {
return readManagementTlsFile(filePath, privateMaterial, configFailure);
}
export async function startClusterAutomationManagementProcess(
options: StartClusterAutomationManagementProcessOptions,
): Promise<Readonly<ClusterAutomationManagementProcessRuntime>> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
![
'environment',
'openDatabase',
'identities',
'assertReady',
'startHttp',
'now',
'onError',
].includes(key),
) ||
!options.environment ||
typeof options.environment !== 'object' ||
(options.openDatabase !== undefined &&
typeof options.openDatabase !== 'function') ||
(options.identities !== undefined &&
(typeof options.identities.reload !== 'function' ||
typeof options.identities.bind !== 'function')) ||
(options.assertReady !== undefined &&
typeof options.assertReady !== 'function') ||
(options.startHttp !== undefined &&
typeof options.startHttp !== 'function') ||
(options.now !== undefined && typeof options.now !== 'function') ||
(options.onError !== undefined && typeof options.onError !== 'function')
) {
throw configFailure('options are invalid');
}
const config = loadClusterAutomationManagementProcessConfig(
options.environment,
);
if (!config.enabled) {
return Object.freeze({
status: 'disabled' as const,
close: () => Promise.resolve(),
});
}
const now = options.now ?? Date.now;
let http: Readonly<ClusterAutomationManagementHttpApplication> | undefined;
let database: PostgresDatabaseResource | undefined;
let unavailableError: unknown;
let closePromise: Promise<void> | undefined;
const report = (error: unknown): void => {
try {
options.onError?.(error);
} catch {
// Diagnostics do not own availability or cleanup.
}
};
const openDatabase =
options.openDatabase ??
createPostgresDatabaseOpener({
role: 'automation-manager',
connection: config.database.connection,
pool: config.database.pool,
onPoolError(error) {
const firstAvailabilityError = unavailableError === undefined;
unavailableError ??= error;
http?.withdraw(error);
if (firstAvailabilityError) report(error);
},
});
try {
database = await openDatabase();
const evidence = await (
options.assertReady ?? assertPostgresAutomationManagerSchemaReady
)(database.pool);
if (unavailableError !== undefined) throw unavailableError;
const identities =
options.identities ??
createClusterAutomationIdentityKeysetFile({
filePath: config.identityKeysetFile,
now,
ledger: new PostgresAutomationManagementIdentityKeysetLedgerRepository(
database.pool,
'automation-management',
),
});
const identity = await identities.reload();
const policy = new ProjectPolicyEngine(
new PostgresProjectPolicyRepository(database.pool),
);
const service = createClusterAutomationManagementService({
policy,
taskDefinitions: new PostgresTaskDefinitionAdministrationRepository(
database.pool,
),
triggers: new PostgresTriggerAdministrationRepository(database.pool),
now,
});
const transport = createClusterAutomationManagementTransport({
service,
now,
});
const privateKey = readTlsFile(config.privateKeyFile, true);
try {
const certificate = readTlsFile(config.certificateFile, false);
const clientCertificateAuthority = readTlsFile(
config.clientCertificateAuthorityFile,
false,
);
const clientCertificateRevocationList = readTlsFile(
config.clientCertificateRevocationListFile,
false,
);
validateClusterManagementClientTrust(
clientCertificateAuthority,
clientCertificateRevocationList,
now(),
configFailure,
);
http = await (options.startHttp ?? startClusterAutomationManagementHttp)({
host: config.host,
port: config.port,
tls: {
privateKey,
certificate,
clientCertificateAuthority,
clientCertificateRevocationList,
},
transport,
identities,
limits: config.http,
now,
onError: report,
});
} finally {
privateKey.fill(0);
}
if (unavailableError !== undefined) http.withdraw(unavailableError);
return Object.freeze({
status: 'active' as const,
address: http.address,
database: evidence,
identity,
availabilityStatus: () => http?.availabilityStatus() ?? 'stopped',
close(): Promise<void> {
if (closePromise) return closePromise;
closePromise = (async () => {
let primaryError: unknown;
try {
await http?.close();
} catch (error) {
primaryError = error;
}
try {
await database?.close();
} catch (error) {
primaryError ??= error;
}
if (primaryError) throw primaryError;
})();
return closePromise;
},
});
} catch (error) {
try {
await http?.close();
} catch {
// Preserve startup failure.
}
try {
await database?.close();
} catch {
// Preserve startup failure.
}
throw error;
}
}
@@ -0,0 +1,423 @@
import {
assertTaskDefinitionIdentifier,
assertTaskDefinitionPageSize,
normalizeTaskDefinitionCursor,
type AppendTaskDefinitionRevisionCommand,
type TaskDefinitionRecord,
} from '@qinglong/runtime-core/task-definition';
import {
assertTriggerIdentifier,
assertTriggerPageSize,
normalizeTriggerCursor,
type AppendTriggerRevisionCommand,
type TriggerRecord,
} from '@qinglong/runtime-core/trigger';
import {
normalizeSecurityPrincipal,
type SecurityPrincipal,
} from '@qinglong/runtime-core/security';
import type { ClusterAutomationManagementService } from './automationManagement';
const STRONG_CLUSTER_ASSURANCES = new Set(['multi_factor', 'hardware']);
const REQUEST_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const AUDIT_EVENT_ID_PATTERN =
/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/;
export interface ClusterAutomationManagementAuthentication {
authenticate(): Promise<Readonly<SecurityPrincipal> | null>;
}
export type ClusterAutomationManagementCommand =
| Readonly<{
schemaVersion: 1;
operation: 'task.publish';
request: Readonly<{
requestId: string;
command: AppendTaskDefinitionRevisionCommand;
}>;
}>
| Readonly<{
schemaVersion: 1;
operation: 'trigger.publish';
request: Readonly<{
requestId: string;
command: AppendTriggerRevisionCommand;
}>;
}>
| Readonly<{
schemaVersion: 1;
operation: 'task.inspect';
request: Readonly<{
requestId: string;
auditEventId: string;
projectId: string;
taskId: string;
}>;
}>
| Readonly<{
schemaVersion: 1;
operation: 'task.list';
request: Readonly<{
requestId: string;
auditEventId: string;
projectId: string;
limit: number;
after?: Readonly<{ taskId: string }>;
}>;
}>
| Readonly<{
schemaVersion: 1;
operation: 'trigger.inspect';
request: Readonly<{
requestId: string;
auditEventId: string;
projectId: string;
triggerId: string;
}>;
}>
| Readonly<{
schemaVersion: 1;
operation: 'trigger.list';
request: Readonly<{
requestId: string;
auditEventId: string;
projectId: string;
limit: number;
after?: Readonly<{ triggerId: string }>;
}>;
}>;
export type ClusterAutomationManagementTransportResult =
| Readonly<{
schemaVersion: 1;
operation: 'task.publish';
status: 'created' | 'updated' | 'existing';
task: ReturnType<typeof taskSummary>;
}>
| Readonly<{
schemaVersion: 1;
operation: 'trigger.publish';
status: 'created' | 'updated' | 'existing';
trigger: ReturnType<typeof triggerSummary>;
}>
| Readonly<{
schemaVersion: 1;
operation: 'task.inspect';
status: 'found' | 'absent';
task: ReturnType<typeof taskSummary> | null;
}>
| Readonly<{
schemaVersion: 1;
operation: 'task.list';
tasks: readonly ReturnType<typeof taskSummary>[];
truncated: boolean;
next: Readonly<{ taskId: string }> | null;
}>
| Readonly<{
schemaVersion: 1;
operation: 'trigger.inspect';
status: 'found' | 'absent';
trigger: ReturnType<typeof triggerSummary> | null;
}>
| Readonly<{
schemaVersion: 1;
operation: 'trigger.list';
triggers: readonly ReturnType<typeof triggerSummary>[];
truncated: boolean;
next: Readonly<{ triggerId: string }> | null;
}>;
export interface ClusterAutomationManagementTransport {
execute(
command: unknown,
authentication: ClusterAutomationManagementAuthentication,
): Promise<Readonly<ClusterAutomationManagementTransportResult>>;
}
export class ClusterAutomationManagementTransportConfigurationError extends TypeError {
readonly code = 'CLUSTER_AUTOMATION_TRANSPORT_CONFIGURATION_INVALID';
constructor() {
super('Cluster automation transport configuration is invalid');
this.name = 'ClusterAutomationManagementTransportConfigurationError';
}
}
export class ClusterAutomationManagementTransportRequestError extends TypeError {
readonly code = 'CLUSTER_AUTOMATION_TRANSPORT_REQUEST_INVALID';
constructor() {
super('Cluster automation transport request is invalid');
this.name = 'ClusterAutomationManagementTransportRequestError';
}
}
export class ClusterAutomationManagementTransportAuthenticationError extends Error {
readonly code = 'CLUSTER_AUTOMATION_TRANSPORT_AUTHENTICATION_REQUIRED';
constructor() {
super('Cluster automation transport requires a strong User principal');
this.name = 'ClusterAutomationManagementTransportAuthenticationError';
}
}
export class ClusterAutomationManagementTransportUnavailableError extends Error {
readonly code = 'CLUSTER_AUTOMATION_TRANSPORT_UNAVAILABLE';
constructor() {
super('Cluster automation transport is unavailable');
this.name = 'ClusterAutomationManagementTransportUnavailableError';
}
}
function normalizeCommand(value: unknown): ClusterAutomationManagementCommand {
if (
!value ||
typeof value !== 'object' ||
Array.isArray(value) ||
Object.keys(value).length !== 3 ||
(value as { schemaVersion?: unknown }).schemaVersion !== 1 ||
![
'task.publish',
'trigger.publish',
'task.inspect',
'task.list',
'trigger.inspect',
'trigger.list',
].includes(String((value as { operation?: unknown }).operation)) ||
!Object.hasOwn(value, 'request') ||
!(value as { request?: unknown }).request ||
typeof (value as { request: unknown }).request !== 'object' ||
Array.isArray((value as { request: unknown }).request) ||
!Object.hasOwn((value as { request: object }).request, 'requestId')
) {
throw new ClusterAutomationManagementTransportRequestError();
}
const operation = (value as { operation: string }).operation;
const request = (value as { request: Record<string, unknown> }).request;
const keys = Object.keys(request);
const required = operation.endsWith('.publish')
? ['command', 'requestId']
: operation.endsWith('.inspect')
? [
'auditEventId',
operation.startsWith('task.') ? 'taskId' : 'triggerId',
'projectId',
'requestId',
]
: ['auditEventId', 'limit', 'projectId', 'requestId'];
const optional = operation.endsWith('.list') ? ['after'] : [];
if (
!required.every((key) => keys.includes(key)) ||
keys.some((key) => !required.includes(key) && !optional.includes(key))
) {
throw new ClusterAutomationManagementTransportRequestError();
}
if (
typeof request.requestId !== 'string' ||
!REQUEST_ID_PATTERN.test(request.requestId)
) {
throw new ClusterAutomationManagementTransportRequestError();
}
if (!operation.endsWith('.publish')) {
if (
typeof request.auditEventId !== 'string' ||
!AUDIT_EVENT_ID_PATTERN.test(request.auditEventId) ||
typeof request.projectId !== 'string'
) {
throw new ClusterAutomationManagementTransportRequestError();
}
try {
if (operation.startsWith('task.')) {
assertTaskDefinitionIdentifier(request.projectId, 'projectId');
if (operation === 'task.inspect') {
assertTaskDefinitionIdentifier(request.taskId as string, 'taskId');
} else {
assertTaskDefinitionPageSize(request.limit as number);
if (Object.hasOwn(request, 'after')) {
normalizeTaskDefinitionCursor(
request.after as Readonly<{ taskId: string }>,
);
}
}
} else {
assertTriggerIdentifier(request.projectId, 'projectId');
if (operation === 'trigger.inspect') {
assertTriggerIdentifier(request.triggerId as string, 'triggerId');
} else {
assertTriggerPageSize(request.limit as number);
if (Object.hasOwn(request, 'after')) {
normalizeTriggerCursor(
request.after as Readonly<{ triggerId: string }>,
);
}
}
}
} catch {
throw new ClusterAutomationManagementTransportRequestError();
}
}
return value as ClusterAutomationManagementCommand;
}
export function normalizeClusterAutomationManagementCommand(
value: unknown,
): Readonly<ClusterAutomationManagementCommand> {
return normalizeCommand(value);
}
function taskSummary(definition: Readonly<TaskDefinitionRecord>) {
return Object.freeze({
projectId: definition.projectId,
taskId: definition.taskId,
revision: definition.revision,
kind: definition.kind,
enabled: definition.enabled,
contentDigest: definition.contentDigest,
updatedAtMs: definition.updatedAtMs,
});
}
function triggerSummary(trigger: Readonly<TriggerRecord>) {
return Object.freeze({
projectId: trigger.projectId,
triggerId: trigger.triggerId,
revision: trigger.revision,
taskId: trigger.taskId,
taskRevision: trigger.taskRevision,
taskContentDigest: trigger.taskContentDigest,
enabled: trigger.enabled,
contentDigest: trigger.contentDigest,
updatedAtMs: trigger.updatedAtMs,
});
}
export function createClusterAutomationManagementTransport(options: Readonly<{
service: ClusterAutomationManagementService;
now?: () => number;
}>): Readonly<ClusterAutomationManagementTransport> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some((key) => key !== 'service' && key !== 'now') ||
!options.service ||
typeof options.service.publishTask !== 'function' ||
typeof options.service.publishTrigger !== 'function' ||
typeof options.service.inspectTask !== 'function' ||
typeof options.service.listTasks !== 'function' ||
typeof options.service.inspectTrigger !== 'function' ||
typeof options.service.listTriggers !== 'function' ||
(options.now !== undefined && typeof options.now !== 'function')
) {
throw new ClusterAutomationManagementTransportConfigurationError();
}
const now = options.now ?? Date.now;
return Object.freeze({
async execute(
commandValue: unknown,
authentication: ClusterAutomationManagementAuthentication,
) {
const command = normalizeCommand(commandValue);
if (
!authentication ||
typeof authentication !== 'object' ||
Array.isArray(authentication) ||
Object.keys(authentication).length !== 1 ||
typeof authentication.authenticate !== 'function'
) {
throw new ClusterAutomationManagementTransportConfigurationError();
}
let candidate: Readonly<SecurityPrincipal> | null;
try {
candidate = await authentication.authenticate();
} catch {
throw new ClusterAutomationManagementTransportUnavailableError();
}
let principal: Readonly<SecurityPrincipal>;
try {
principal = normalizeSecurityPrincipal(candidate as SecurityPrincipal, now());
} catch {
throw new ClusterAutomationManagementTransportAuthenticationError();
}
if (
principal.subject.type !== 'user' ||
!STRONG_CLUSTER_ASSURANCES.has(principal.assurance)
) {
throw new ClusterAutomationManagementTransportAuthenticationError();
}
if (command.operation === 'task.publish') {
const result = await options.service.publishTask({
...command.request,
principal,
});
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
status: result.status,
task: taskSummary(result.definition),
});
}
if (command.operation === 'trigger.publish') {
const result = await options.service.publishTrigger({
...command.request,
principal,
});
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
status: result.status,
trigger: triggerSummary(result.trigger),
});
}
if (command.operation === 'task.inspect') {
const task = await options.service.inspectTask({
...command.request,
principal,
});
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
status: task ? ('found' as const) : ('absent' as const),
task: task ? taskSummary(task) : null,
});
}
if (command.operation === 'task.list') {
const page = await options.service.listTasks({
...command.request,
principal,
});
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
tasks: Object.freeze(page.definitions.map(taskSummary)),
truncated: page.truncated,
next: page.next ?? null,
});
}
if (command.operation === 'trigger.inspect') {
const trigger = await options.service.inspectTrigger({
...command.request,
principal,
});
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
status: trigger ? ('found' as const) : ('absent' as const),
trigger: trigger ? triggerSummary(trigger) : null,
});
}
const page = await options.service.listTriggers({
...command.request,
principal,
});
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
triggers: Object.freeze(page.triggers.map(triggerSummary)),
truncated: page.truncated,
next: page.next ?? null,
});
},
});
}
@@ -0,0 +1,135 @@
/** Shared bounded process-configuration authority for cluster management planes. */
import { closeSync, constants, fstatSync, openSync, readSync } from 'node:fs';
import { isAbsolute } from 'node:path';
const MAX_TLS_FILE_BYTES = 256 * 1024;
const CONTROL_PATTERN = /[\u0000-\u001f\u007f]/;
export type ClusterManagementProcessEnvironment = Readonly<
Record<string, string | undefined>
>;
export type ClusterManagementProcessConfigurationFailure = (
message: string,
) => Error;
export function boundedManagementEnvironmentValue(
environment: ClusterManagementProcessEnvironment,
name: string,
maximumLength: number,
failure: ClusterManagementProcessConfigurationFailure,
required = false,
): string | undefined {
const value = environment[name];
if (value === undefined || value === '') {
if (required) throw failure(`${name} is required`);
return undefined;
}
if (value.length > maximumLength || CONTROL_PATTERN.test(value)) {
throw failure(`${name} is invalid`);
}
return value;
}
export function booleanManagementEnvironmentValue(
environment: ClusterManagementProcessEnvironment,
name: string,
failure: ClusterManagementProcessConfigurationFailure,
): boolean {
const value = environment[name];
if (value === undefined || value === '') return false;
if (value === 'true') return true;
if (value === 'false') return false;
throw failure(`${name} must be true or false`);
}
export function integerManagementEnvironmentValue(
environment: ClusterManagementProcessEnvironment,
name: string,
fallback: number,
minimum: number,
maximum: number,
failure: ClusterManagementProcessConfigurationFailure,
): number {
const value = environment[name];
if (value === undefined || value === '') return fallback;
if (!/^(?:0|[1-9][0-9]*)$/.test(value)) {
throw failure(`${name} must be an integer`);
}
const parsed = Number(value);
if (!Number.isSafeInteger(parsed) || parsed < minimum || parsed > maximum) {
throw failure(`${name} must be between ${minimum} and ${maximum}`);
}
return parsed;
}
export function absoluteManagementEnvironmentFile(
environment: ClusterManagementProcessEnvironment,
name: string,
failure: ClusterManagementProcessConfigurationFailure,
): string {
const value = boundedManagementEnvironmentValue(
environment,
name,
4_096,
failure,
true,
)!;
if (!isAbsolute(value)) {
throw failure(`${name} must be an absolute path`);
}
return value;
}
export function readManagementTlsFile(
filePath: string,
privateMaterial: boolean,
failure: ClusterManagementProcessConfigurationFailure,
): Buffer {
let descriptor: number | undefined;
let bytes: Buffer | undefined;
try {
descriptor = openSync(filePath, constants.O_RDONLY);
const stat = fstatSync(descriptor);
if (
!stat.isFile() ||
stat.size < 1 ||
stat.size > MAX_TLS_FILE_BYTES ||
(stat.mode & 0o022) !== 0 ||
(privateMaterial && (stat.mode & 0o007) !== 0)
) {
throw failure('TLS file authority is invalid');
}
bytes = Buffer.alloc(stat.size + 1);
let offset = 0;
while (offset < bytes.length) {
const read = readSync(
descriptor,
bytes,
offset,
bytes.length - offset,
offset,
);
if (read === 0) break;
offset += read;
}
const after = fstatSync(descriptor);
if (
offset !== stat.size ||
offset > MAX_TLS_FILE_BYTES ||
stat.dev !== after.dev ||
stat.ino !== after.ino ||
stat.size !== after.size ||
stat.mtimeMs !== after.mtimeMs ||
stat.ctimeMs !== after.ctimeMs
) {
throw failure('TLS file changed while being read');
}
return bytes.subarray(0, offset);
} catch (error) {
if (privateMaterial) bytes?.fill(0);
throw error;
} finally {
if (descriptor !== undefined) closeSync(descriptor);
}
}
@@ -0,0 +1,755 @@
/** Shared authenticated identity assertion boundary for cluster management planes. */
import {
constants,
createHash,
createPublicKey,
verify as verifySignature,
type KeyObject,
} from 'node:crypto';
import {
normalizeSecurityPrincipal,
type SecurityAuthenticationAssurance,
type SecurityPrincipal,
} from '@qinglong/runtime-core/security';
const ASSERTION_ALGORITHMS = ['EdDSA', 'ES256', 'RS256'] as const;
const MAX_KEYS = 8;
const MAX_ASSURANCE_MAPPINGS = 8;
const MAX_AMR_VALUES = 8;
const MIN_ASSERTION_BYTES = 512;
const MAX_ASSERTION_BYTES = 16 * 1024;
const DEFAULT_ASSERTION_BYTES = 8 * 1024;
const MIN_LIFETIME_MS = 30_000;
const MAX_LIFETIME_MS = 15 * 60_000;
const DEFAULT_LIFETIME_MS = 5 * 60_000;
const MAX_AUTHENTICATION_AGE_MS = 15 * 60_000;
const DEFAULT_AUTHENTICATION_AGE_MS = 5 * 60_000;
const MAX_CLOCK_SKEW_MS = 60_000;
const DEFAULT_CLOCK_SKEW_MS = 5_000;
const TOKEN_VALUE_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const ASSERTION_TYPE_PATTERN = /^ql3-[a-z0-9]+(?:-[a-z0-9]+)*\+jwt$/;
const ASSERTION_PURPOSE_PATTERN = /^[a-z0-9]+(?:-[a-z0-9]+)*$/;
const BASE64URL_PATTERN = /^[A-Za-z0-9_-]+$/;
const CONTROL_PATTERN = /[\u0000-\u001f\u007f]/;
type AssertionAlgorithm = (typeof ASSERTION_ALGORITHMS)[number];
type AssertionAssurance = Extract<
SecurityAuthenticationAssurance,
'multi_factor' | 'hardware'
>;
export interface ClusterPluginPackageIdentityAssertionAssuranceMapping {
readonly acr: string;
readonly assurance: AssertionAssurance;
readonly requiredAmr: readonly string[];
}
export interface ClusterManagementIdentityAssertionProfile {
readonly type: string;
readonly purpose: string;
}
export const CLUSTER_PLUGIN_PACKAGE_MANAGEMENT_IDENTITY_ASSERTION_PROFILE =
Object.freeze({
type: 'ql3-plugin-package-management+jwt',
purpose: 'plugin-package-management',
});
export const CLUSTER_WORKER_CREDENTIAL_MANAGEMENT_IDENTITY_ASSERTION_PROFILE =
Object.freeze({
type: 'ql3-worker-credential-management+jwt',
purpose: 'worker-credential-management',
});
export const CLUSTER_AUTOMATION_MANAGEMENT_IDENTITY_ASSERTION_PROFILE =
Object.freeze({
type: 'ql3-automation-management+jwt',
purpose: 'automation-management',
});
export const CLUSTER_APPROVAL_MANAGEMENT_IDENTITY_ASSERTION_PROFILE =
Object.freeze({
type: 'ql3-approval-management+jwt',
purpose: 'approval-management',
});
export const CLUSTER_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_IDENTITY_ASSERTION_PROFILE =
Object.freeze({
type: 'ql3-model-provider-credential-management+jwt',
purpose: 'model-provider-credential-management',
});
export interface ClusterPluginPackageIdentityAssertionVerifierOptions {
readonly issuer: string;
readonly audience: string;
readonly keys: readonly Readonly<Record<string, unknown>>[];
readonly assuranceMappings: readonly ClusterPluginPackageIdentityAssertionAssuranceMapping[];
readonly assertionProfile?: Readonly<ClusterManagementIdentityAssertionProfile>;
readonly maxAssertionBytes?: number;
readonly maxLifetimeMs?: number;
readonly maxAuthenticationAgeMs?: number;
readonly clockSkewMs?: number;
readonly now?: () => number;
}
export interface ClusterPluginPackageIdentityAssertionAuthentication {
authenticate(): Promise<Readonly<SecurityPrincipal>>;
}
export interface ClusterPluginPackageIdentityAssertionVerifier {
verify(assertion: unknown): Readonly<SecurityPrincipal>;
bind(
assertion: unknown,
): Readonly<ClusterPluginPackageIdentityAssertionAuthentication>;
}
export class ClusterPluginPackageIdentityAssertionConfigurationError extends TypeError {
readonly code =
'CLUSTER_PLUGIN_PACKAGE_IDENTITY_ASSERTION_CONFIGURATION_INVALID';
constructor(message: string) {
super(
`Cluster Plugin Package identity assertion configuration is invalid: ${message}`,
);
this.name = 'ClusterPluginPackageIdentityAssertionConfigurationError';
}
}
export class ClusterPluginPackageIdentityAssertionAuthenticationError extends Error {
readonly code = 'CLUSTER_PLUGIN_PACKAGE_IDENTITY_ASSERTION_INVALID';
constructor() {
super('Cluster Plugin Package identity assertion is invalid');
this.name = 'ClusterPluginPackageIdentityAssertionAuthenticationError';
}
}
interface ReviewedAssertionKey {
readonly kid: string;
readonly algorithm: AssertionAlgorithm;
readonly key: KeyObject;
}
interface ReviewedAssuranceMapping {
readonly assurance: AssertionAssurance;
readonly requiredAmr: ReadonlySet<string>;
}
function configurationFailure(
message: string,
): ClusterPluginPackageIdentityAssertionConfigurationError {
return new ClusterPluginPackageIdentityAssertionConfigurationError(message);
}
function exactObject(
value: unknown,
expectedKeys: readonly string[],
label: string,
): asserts value is Record<string, unknown> {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw configurationFailure(`${label} must be an object`);
}
const actual = Object.keys(value).sort();
const expected = [...expectedKeys].sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
throw configurationFailure(`${label} shape is invalid`);
}
}
function reviewedAssertionProfile(
value: unknown,
): Readonly<ClusterManagementIdentityAssertionProfile> {
exactObject(value, ['type', 'purpose'], 'assertion profile');
if (
typeof value.type !== 'string' ||
value.type.length > 128 ||
!ASSERTION_TYPE_PATTERN.test(value.type) ||
typeof value.purpose !== 'string' ||
value.purpose.length > 96 ||
!ASSERTION_PURPOSE_PATTERN.test(value.purpose)
) {
throw configurationFailure('assertion profile is invalid');
}
return Object.freeze({ type: value.type, purpose: value.purpose });
}
function boundedInteger(
value: number | undefined,
fallback: number,
minimum: number,
maximum: number,
label: string,
): number {
const candidate = value ?? fallback;
if (
!Number.isSafeInteger(candidate) ||
candidate < minimum ||
candidate > maximum
) {
throw configurationFailure(`${label} is invalid`);
}
return candidate;
}
function reviewedIssuer(value: unknown): string {
if (
typeof value !== 'string' ||
value.length < 1 ||
value.length > 512 ||
CONTROL_PATTERN.test(value)
) {
throw configurationFailure('issuer is invalid');
}
let parsed: URL;
try {
parsed = new URL(value);
} catch {
throw configurationFailure('issuer is invalid');
}
if (
parsed.protocol !== 'https:' ||
parsed.username !== '' ||
parsed.password !== '' ||
parsed.search !== '' ||
parsed.hash !== '' ||
parsed.toString() !== value
) {
throw configurationFailure('issuer must be one canonical HTTPS URL');
}
return value;
}
function reviewedTokenValue(
value: unknown,
label: string,
maximumLength = 128,
): string {
if (
typeof value !== 'string' ||
value.length < 1 ||
value.length > maximumLength ||
CONTROL_PATTERN.test(value)
) {
throw configurationFailure(`${label} is invalid`);
}
return value;
}
function reviewedJwkComponent(
value: unknown,
minimumBytes: number,
maximumBytes: number,
label: string,
): void {
if (
typeof value !== 'string' ||
value.length < 1 ||
!BASE64URL_PATTERN.test(value)
) {
throw configurationFailure(`${label} is invalid`);
}
const bytes = Buffer.from(value, 'base64url');
if (
bytes.length < minimumBytes ||
bytes.length > maximumBytes ||
bytes.toString('base64url') !== value
) {
throw configurationFailure(`${label} is invalid`);
}
}
function reviewedJwk(
value: unknown,
seenKids: Set<string>,
): Readonly<ReviewedAssertionKey> {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw configurationFailure('key must be an object');
}
const candidate = value as Record<string, unknown>;
const algorithm = candidate.alg;
if (
typeof algorithm !== 'string' ||
!ASSERTION_ALGORITHMS.includes(algorithm as AssertionAlgorithm)
) {
throw configurationFailure('key algorithm is invalid');
}
const reviewedAlgorithm = algorithm as AssertionAlgorithm;
const expectedKeys =
reviewedAlgorithm === 'RS256'
? ['alg', 'e', 'kid', 'kty', 'n', 'use']
: reviewedAlgorithm === 'ES256'
? ['alg', 'crv', 'kid', 'kty', 'use', 'x', 'y']
: ['alg', 'crv', 'kid', 'kty', 'use', 'x'];
exactObject(candidate, expectedKeys, 'key');
const kid = candidate.kid;
if (
typeof kid !== 'string' ||
!TOKEN_VALUE_PATTERN.test(kid) ||
seenKids.has(kid)
) {
throw configurationFailure('key id is invalid or duplicated');
}
if (candidate.use !== 'sig') {
throw configurationFailure('key use must be sig');
}
if (
(reviewedAlgorithm === 'RS256' && candidate.kty !== 'RSA') ||
(reviewedAlgorithm === 'ES256' &&
(candidate.kty !== 'EC' || candidate.crv !== 'P-256')) ||
(reviewedAlgorithm === 'EdDSA' &&
(candidate.kty !== 'OKP' || candidate.crv !== 'Ed25519'))
) {
throw configurationFailure('key type does not match its algorithm');
}
for (const name of expectedKeys) {
if (
['alg', 'kid', 'kty', 'use', 'crv'].includes(name) ||
(typeof candidate[name] === 'string' &&
BASE64URL_PATTERN.test(candidate[name] as string))
) {
continue;
}
throw configurationFailure(`key ${name} is invalid`);
}
if (reviewedAlgorithm === 'RS256') {
reviewedJwkComponent(candidate.n, 256, 512, 'RSA modulus');
reviewedJwkComponent(candidate.e, 3, 4, 'RSA exponent');
} else if (reviewedAlgorithm === 'ES256') {
reviewedJwkComponent(candidate.x, 32, 32, 'EC x coordinate');
reviewedJwkComponent(candidate.y, 32, 32, 'EC y coordinate');
} else {
reviewedJwkComponent(candidate.x, 32, 32, 'Ed25519 public key');
}
let key: KeyObject;
try {
key = createPublicKey({
key: candidate,
format: 'jwk',
});
} catch {
throw configurationFailure('key material is invalid');
}
if (
reviewedAlgorithm === 'RS256' &&
(key.asymmetricKeyType !== 'rsa' ||
(key.asymmetricKeyDetails?.modulusLength ?? 0) < 2048 ||
(key.asymmetricKeyDetails?.modulusLength ?? 0) > 4096 ||
key.asymmetricKeyDetails?.publicExponent !== 65_537n)
) {
throw configurationFailure('RSA key strength is invalid');
}
if (
reviewedAlgorithm === 'ES256' &&
(key.asymmetricKeyType !== 'ec' ||
key.asymmetricKeyDetails?.namedCurve !== 'prime256v1')
) {
throw configurationFailure('EC key strength is invalid');
}
if (reviewedAlgorithm === 'EdDSA' && key.asymmetricKeyType !== 'ed25519') {
throw configurationFailure('Ed25519 key is invalid');
}
seenKids.add(kid);
return Object.freeze({ kid, algorithm: reviewedAlgorithm, key });
}
function reviewedKeys(
value: unknown,
): ReadonlyMap<string, Readonly<ReviewedAssertionKey>> {
if (!Array.isArray(value) || value.length < 1 || value.length > MAX_KEYS) {
throw configurationFailure('keys must contain between one and eight keys');
}
const seenKids = new Set<string>();
const keys = new Map<string, Readonly<ReviewedAssertionKey>>();
for (const candidate of value) {
const reviewed = reviewedJwk(candidate, seenKids);
keys.set(reviewed.kid, reviewed);
}
return keys;
}
function reviewedAssuranceMappings(
value: unknown,
): ReadonlyMap<string, Readonly<ReviewedAssuranceMapping>> {
if (
!Array.isArray(value) ||
value.length < 1 ||
value.length > MAX_ASSURANCE_MAPPINGS
) {
throw configurationFailure(
'assurance mappings must contain between one and eight entries',
);
}
const mappings = new Map<string, Readonly<ReviewedAssuranceMapping>>();
for (const candidate of value) {
exactObject(candidate, ['acr', 'assurance', 'requiredAmr'], 'mapping');
const acr = reviewedTokenValue(candidate.acr, 'mapping acr', 256);
if (
mappings.has(acr) ||
(candidate.assurance !== 'multi_factor' &&
candidate.assurance !== 'hardware') ||
!Array.isArray(candidate.requiredAmr) ||
candidate.requiredAmr.length < 1 ||
candidate.requiredAmr.length > MAX_AMR_VALUES
) {
throw configurationFailure('assurance mapping is invalid');
}
const requiredAmr = new Set<string>();
for (const entry of candidate.requiredAmr) {
if (
typeof entry !== 'string' ||
!TOKEN_VALUE_PATTERN.test(entry) ||
requiredAmr.has(entry)
) {
throw configurationFailure('mapping AMR is invalid or duplicated');
}
requiredAmr.add(entry);
}
mappings.set(
acr,
Object.freeze({
assurance: candidate.assurance,
requiredAmr,
}),
);
}
return mappings;
}
function canonicalBase64Url(segment: string, maximumBytes: number): Buffer {
if (
segment.length < 1 ||
segment.length > Math.ceil((maximumBytes * 4) / 3) ||
!BASE64URL_PATTERN.test(segment)
) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
const decoded = Buffer.from(segment, 'base64url');
if (
decoded.length < 1 ||
decoded.length > maximumBytes ||
decoded.toString('base64url') !== segment
) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
return decoded;
}
function jsonObject(
segment: string,
maximumBytes: number,
): Record<string, unknown> {
const bytes = canonicalBase64Url(segment, maximumBytes);
let value: unknown;
try {
value = JSON.parse(bytes.toString('utf8'));
} catch {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
return value as Record<string, unknown>;
}
function assertionExactObject(
value: Record<string, unknown>,
expectedKeys: readonly string[],
): void {
const actual = Object.keys(value).sort();
const expected = [...expectedKeys].sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
}
function numericDate(value: unknown): number {
if (
!Number.isSafeInteger(value) ||
(value as number) < 0 ||
!Number.isSafeInteger((value as number) * 1_000)
) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
return (value as number) * 1_000;
}
function assertionTokenValue(value: unknown, maximumLength: number): string {
if (
typeof value !== 'string' ||
value.length < 1 ||
value.length > maximumLength ||
CONTROL_PATTERN.test(value)
) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
return value;
}
function verifyReviewedSignature(
algorithm: AssertionAlgorithm,
key: KeyObject,
signed: Buffer,
signature: Buffer,
): boolean {
switch (algorithm) {
case 'EdDSA':
return (
signature.length === 64 && verifySignature(null, signed, key, signature)
);
case 'ES256':
return (
signature.length === 64 &&
verifySignature(
'sha256',
signed,
{ key, dsaEncoding: 'ieee-p1363' },
signature,
)
);
case 'RS256':
return verifySignature(
'RSA-SHA256',
signed,
{ key, padding: constants.RSA_PKCS1_PADDING },
signature,
);
}
}
function authenticationId(issuer: string, jti: string): string {
return `ql3oidc.${createHash('sha256')
.update(issuer)
.update('\0')
.update(jti)
.digest('base64url')}`;
}
export function createClusterPluginPackageIdentityAssertionVerifier(
options: ClusterPluginPackageIdentityAssertionVerifierOptions,
): Readonly<ClusterPluginPackageIdentityAssertionVerifier> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
![
'issuer',
'audience',
'keys',
'assuranceMappings',
'assertionProfile',
'maxAssertionBytes',
'maxLifetimeMs',
'maxAuthenticationAgeMs',
'clockSkewMs',
'now',
].includes(key),
)
) {
throw configurationFailure('options shape is invalid');
}
const issuer = reviewedIssuer(options.issuer);
const audience = reviewedTokenValue(options.audience, 'audience', 256);
const assertionProfile = reviewedAssertionProfile(
options.assertionProfile ??
CLUSTER_PLUGIN_PACKAGE_MANAGEMENT_IDENTITY_ASSERTION_PROFILE,
);
const keys = reviewedKeys(options.keys);
const assuranceMappings = reviewedAssuranceMappings(
options.assuranceMappings,
);
const maxAssertionBytes = boundedInteger(
options.maxAssertionBytes,
DEFAULT_ASSERTION_BYTES,
MIN_ASSERTION_BYTES,
MAX_ASSERTION_BYTES,
'max assertion bytes',
);
const maxLifetimeMs = boundedInteger(
options.maxLifetimeMs,
DEFAULT_LIFETIME_MS,
MIN_LIFETIME_MS,
MAX_LIFETIME_MS,
'max lifetime',
);
const maxAuthenticationAgeMs = boundedInteger(
options.maxAuthenticationAgeMs,
DEFAULT_AUTHENTICATION_AGE_MS,
MIN_LIFETIME_MS,
MAX_AUTHENTICATION_AGE_MS,
'max authentication age',
);
const clockSkewMs = boundedInteger(
options.clockSkewMs,
DEFAULT_CLOCK_SKEW_MS,
0,
MAX_CLOCK_SKEW_MS,
'clock skew',
);
if (options.now !== undefined && typeof options.now !== 'function') {
throw configurationFailure('clock is invalid');
}
const now = options.now ?? Date.now;
const verify = (assertion: unknown): Readonly<SecurityPrincipal> => {
try {
if (
typeof assertion !== 'string' ||
Buffer.byteLength(assertion, 'utf8') > maxAssertionBytes ||
CONTROL_PATTERN.test(assertion)
) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
const segments = assertion.split('.');
if (segments.length !== 3) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
const [protectedSegment, payloadSegment, signatureSegment] = segments;
if (
protectedSegment === undefined ||
payloadSegment === undefined ||
signatureSegment === undefined
) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
const header = jsonObject(protectedSegment, 1_024);
assertionExactObject(header, ['alg', 'kid', 'typ']);
if (
header.typ !== assertionProfile.type ||
typeof header.kid !== 'string' ||
typeof header.alg !== 'string'
) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
const reviewedKey = keys.get(header.kid);
if (!reviewedKey || reviewedKey.algorithm !== header.alg) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
const signature = canonicalBase64Url(signatureSegment, 512);
const signed = Buffer.from(
`${protectedSegment}.${payloadSegment}`,
'ascii',
);
if (
!verifyReviewedSignature(
reviewedKey.algorithm,
reviewedKey.key,
signed,
signature,
)
) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
const claims = jsonObject(payloadSegment, 8 * 1_024);
const claimKeys = [
'acr',
'amr',
'aud',
'auth_time',
'exp',
'iat',
'iss',
'jti',
'ql3_purpose',
'sub',
];
if (Object.hasOwn(claims, 'nbf')) claimKeys.push('nbf');
assertionExactObject(claims, claimKeys);
if (
claims.iss !== issuer ||
claims.aud !== audience ||
claims.ql3_purpose !== assertionProfile.purpose
) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
const subjectId = assertionTokenValue(claims.sub, 255);
const jti = assertionTokenValue(claims.jti, 255);
const issuedAtMs = numericDate(claims.iat);
const authenticatedAtMs = numericDate(claims.auth_time);
const expiresAtMs = numericDate(claims.exp);
const notBeforeAtMs = Object.hasOwn(claims, 'nbf')
? numericDate(claims.nbf)
: issuedAtMs;
const observedAtMs = now();
if (
!Number.isSafeInteger(observedAtMs) ||
observedAtMs < 0 ||
authenticatedAtMs > issuedAtMs ||
issuedAtMs > observedAtMs + clockSkewMs ||
authenticatedAtMs > observedAtMs ||
notBeforeAtMs < issuedAtMs ||
notBeforeAtMs >= expiresAtMs ||
observedAtMs + clockSkewMs < notBeforeAtMs ||
expiresAtMs <= observedAtMs ||
expiresAtMs - issuedAtMs > maxLifetimeMs ||
observedAtMs - authenticatedAtMs > maxAuthenticationAgeMs
) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
const acr = assertionTokenValue(claims.acr, 256);
const mapping = assuranceMappings.get(acr);
if (
!mapping ||
!Array.isArray(claims.amr) ||
claims.amr.length < 1 ||
claims.amr.length > MAX_AMR_VALUES
) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
const amr = new Set<string>();
for (const entry of claims.amr) {
if (
typeof entry !== 'string' ||
!TOKEN_VALUE_PATTERN.test(entry) ||
amr.has(entry)
) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
amr.add(entry);
}
if ([...mapping.requiredAmr].some((entry) => !amr.has(entry))) {
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
return normalizeSecurityPrincipal(
{
subject: { type: 'user', id: subjectId },
authenticationId: authenticationId(issuer, jti),
authenticatedAtMs,
expiresAtMs,
assurance: mapping.assurance,
},
observedAtMs,
);
} catch (error) {
if (
error instanceof
ClusterPluginPackageIdentityAssertionAuthenticationError
) {
throw error;
}
throw new ClusterPluginPackageIdentityAssertionAuthenticationError();
}
};
return Object.freeze({
verify,
bind(
assertion: unknown,
): Readonly<ClusterPluginPackageIdentityAssertionAuthentication> {
return Object.freeze({
async authenticate(): Promise<Readonly<SecurityPrincipal>> {
return verify(assertion);
},
});
},
});
}
@@ -0,0 +1,509 @@
/** Shared bounded identity keyset and rotation boundary for cluster management planes. */
import { constants } from 'node:fs';
import { open } from 'node:fs/promises';
import { createHash } from 'node:crypto';
import { isAbsolute } from 'node:path';
import {
CLUSTER_WORKER_CREDENTIAL_MANAGEMENT_IDENTITY_ASSERTION_PROFILE,
CLUSTER_AUTOMATION_MANAGEMENT_IDENTITY_ASSERTION_PROFILE,
CLUSTER_APPROVAL_MANAGEMENT_IDENTITY_ASSERTION_PROFILE,
CLUSTER_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_IDENTITY_ASSERTION_PROFILE,
createClusterPluginPackageIdentityAssertionVerifier,
type ClusterManagementIdentityAssertionProfile,
type ClusterPluginPackageIdentityAssertionAuthentication,
type ClusterPluginPackageIdentityAssertionVerifier,
} from './pluginPackageIdentityAssertion';
const DEFAULT_MAX_FILE_BYTES = 64 * 1024;
const MIN_MAX_FILE_BYTES = 4 * 1024;
const HARD_MAX_FILE_BYTES = 256 * 1024;
const MAX_REVOKED_KEYS = 64;
const KEY_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const CONTROL_PATTERN = /[\u0000-\u001f\u007f]/;
export interface ClusterPluginPackageIdentityKeysetFileOptions {
readonly filePath: string;
readonly maxFileBytes?: number;
readonly now?: () => number;
readonly ledger?: ClusterPluginPackageIdentityKeysetLedger;
readonly assertionProfile?: Readonly<ClusterManagementIdentityAssertionProfile>;
}
export type ClusterWorkerCredentialIdentityKeysetFileOptions = Omit<
ClusterPluginPackageIdentityKeysetFileOptions,
'assertionProfile'
>;
export interface ClusterPluginPackageIdentityKeysetSnapshot {
readonly schemaVersion: 1;
readonly generation: number;
readonly digest: string;
readonly issuer: string;
readonly audience: string;
readonly activeKeyIds: readonly string[];
readonly revokedKeyIds: readonly string[];
}
export interface ClusterPluginPackageIdentityKeysetFile {
reload(): Promise<Readonly<ClusterPluginPackageIdentityKeysetSnapshot>>;
bind(
assertion: unknown,
): Readonly<ClusterPluginPackageIdentityAssertionAuthentication>;
}
export interface ClusterPluginPackageIdentityKeysetLedger {
observe(
snapshot: Readonly<ClusterPluginPackageIdentityKeysetSnapshot>,
): Promise<void>;
}
export class ClusterPluginPackageIdentityKeysetConfigurationError extends TypeError {
readonly code =
'CLUSTER_PLUGIN_PACKAGE_IDENTITY_KEYSET_CONFIGURATION_INVALID';
constructor(message: string) {
super(
`Cluster Plugin Package identity keyset configuration is invalid: ${message}`,
);
this.name = 'ClusterPluginPackageIdentityKeysetConfigurationError';
}
}
export class ClusterPluginPackageIdentityKeysetUnavailableError extends Error {
readonly code = 'CLUSTER_PLUGIN_PACKAGE_IDENTITY_KEYSET_UNAVAILABLE';
constructor(readonly cause?: unknown) {
super('Cluster Plugin Package identity keyset is unavailable');
this.name = 'ClusterPluginPackageIdentityKeysetUnavailableError';
}
}
interface LoadedKeyset {
readonly generation: number;
readonly digest: string;
readonly verifier: Readonly<ClusterPluginPackageIdentityAssertionVerifier>;
readonly activeKeyIds: ReadonlySet<string>;
readonly revokedKeyIds: ReadonlySet<string>;
readonly snapshot: Readonly<ClusterPluginPackageIdentityKeysetSnapshot>;
}
function configurationFailure(
message: string,
): ClusterPluginPackageIdentityKeysetConfigurationError {
return new ClusterPluginPackageIdentityKeysetConfigurationError(message);
}
function exactObject(
value: unknown,
expectedKeys: readonly string[],
label: string,
): asserts value is Record<string, unknown> {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw configurationFailure(`${label} must be an object`);
}
const actual = Object.keys(value).sort();
const expected = [...expectedKeys].sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
throw configurationFailure(`${label} shape is invalid`);
}
}
function boundedInteger(
value: unknown,
minimum: number,
maximum: number,
label: string,
): number {
if (
!Number.isSafeInteger(value) ||
(value as number) < minimum ||
(value as number) > maximum
) {
throw configurationFailure(`${label} is invalid`);
}
return value as number;
}
function sameFileState(
left: Readonly<{
dev: number;
ino: number;
size: number;
mtimeMs: number;
ctimeMs: number;
}>,
right: Readonly<{
dev: number;
ino: number;
size: number;
mtimeMs: number;
ctimeMs: number;
}>,
): boolean {
return (
left.dev === right.dev &&
left.ino === right.ino &&
left.size === right.size &&
left.mtimeMs === right.mtimeMs &&
left.ctimeMs === right.ctimeMs
);
}
async function readBoundedRegularFile(
filePath: string,
maxFileBytes: number,
): Promise<Buffer> {
const handle = await open(filePath, constants.O_RDONLY);
try {
const before = await handle.stat();
if (
!before.isFile() ||
before.size < 1 ||
before.size > maxFileBytes ||
(before.mode & 0o022) !== 0
) {
throw configurationFailure(
'keyset file must be a bounded non-writable regular file',
);
}
const buffer = Buffer.allocUnsafe(maxFileBytes + 1);
let offset = 0;
while (offset < buffer.length) {
const { bytesRead } = await handle.read(
buffer,
offset,
buffer.length - offset,
offset,
);
if (bytesRead === 0) break;
offset += bytesRead;
}
const after = await handle.stat();
if (
offset !== before.size ||
offset > maxFileBytes ||
!sameFileState(before, after)
) {
throw configurationFailure('keyset file changed while being read');
}
return buffer.subarray(0, offset);
} finally {
await handle.close().catch(() => undefined);
}
}
function parseJson(bytes: Buffer): Record<string, unknown> {
let text: string;
try {
text = new TextDecoder('utf-8', { fatal: true }).decode(bytes);
} catch {
throw configurationFailure('keyset file must be strict UTF-8');
}
let value: unknown;
try {
value = JSON.parse(text);
} catch {
throw configurationFailure('keyset file must contain JSON');
}
exactObject(
value,
[
'schemaVersion',
'generation',
'issuer',
'audience',
'keys',
'revokedKids',
'assuranceMappings',
'constraints',
],
'keyset',
);
return value;
}
function reviewedRevokedKeyIds(value: unknown): ReadonlySet<string> {
if (!Array.isArray(value) || value.length > MAX_REVOKED_KEYS) {
throw configurationFailure('revoked key ids are invalid');
}
const ids = new Set<string>();
for (const candidate of value) {
if (
typeof candidate !== 'string' ||
!KEY_ID_PATTERN.test(candidate) ||
ids.has(candidate)
) {
throw configurationFailure('revoked key id is invalid or duplicated');
}
ids.add(candidate);
}
return ids;
}
function activeKeys(
value: unknown,
revokedKeyIds: ReadonlySet<string>,
): {
readonly all: readonly Readonly<Record<string, unknown>>[];
readonly active: readonly Readonly<Record<string, unknown>>[];
readonly activeKeyIds: ReadonlySet<string>;
} {
if (!Array.isArray(value)) {
throw configurationFailure('keys must be an array');
}
const all = value as readonly Readonly<Record<string, unknown>>[];
const active: Readonly<Record<string, unknown>>[] = [];
const activeKeyIds = new Set<string>();
for (const candidate of all) {
if (
!candidate ||
typeof candidate !== 'object' ||
Array.isArray(candidate)
) {
throw configurationFailure('key must be an object');
}
const kid = candidate.kid;
if (typeof kid !== 'string') {
throw configurationFailure('key id is invalid');
}
if (!revokedKeyIds.has(kid)) {
active.push(candidate);
activeKeyIds.add(kid);
}
}
if (active.length < 1) {
throw configurationFailure('at least one key must remain active');
}
return Object.freeze({ all, active, activeKeyIds });
}
function loadDocument(
bytes: Buffer,
now: (() => number) | undefined,
digest: string,
assertionProfile:
| Readonly<ClusterManagementIdentityAssertionProfile>
| undefined,
): LoadedKeyset {
const document = parseJson(bytes);
if (document.schemaVersion !== 1) {
throw configurationFailure('schemaVersion is invalid');
}
const generation = boundedInteger(
document.generation,
1,
Number.MAX_SAFE_INTEGER,
'generation',
);
const revokedKeyIds = reviewedRevokedKeyIds(document.revokedKids);
const keySelection = activeKeys(document.keys, revokedKeyIds);
exactObject(
document.constraints,
[
'maxAssertionBytes',
'maxLifetimeMs',
'maxAuthenticationAgeMs',
'clockSkewMs',
],
'constraints',
);
const verifierOptions = {
issuer: document.issuer as string,
audience: document.audience as string,
assuranceMappings: document.assuranceMappings as never,
maxAssertionBytes: document.constraints.maxAssertionBytes as number,
maxLifetimeMs: document.constraints.maxLifetimeMs as number,
maxAuthenticationAgeMs: document.constraints
.maxAuthenticationAgeMs as number,
clockSkewMs: document.constraints.clockSkewMs as number,
...(assertionProfile === undefined ? {} : { assertionProfile }),
...(now === undefined ? {} : { now }),
};
// Validate revoked definitions too; revocation must not become a channel for
// retaining malformed or private key material in the trust document.
createClusterPluginPackageIdentityAssertionVerifier({
...verifierOptions,
keys: keySelection.all,
});
const verifier = createClusterPluginPackageIdentityAssertionVerifier({
...verifierOptions,
keys: keySelection.active,
});
const issuer = document.issuer as string;
const audience = document.audience as string;
const snapshot = Object.freeze({
schemaVersion: 1 as const,
generation,
digest,
issuer,
audience,
activeKeyIds: Object.freeze([...keySelection.activeKeyIds].sort()),
revokedKeyIds: Object.freeze([...revokedKeyIds].sort()),
});
return Object.freeze({
generation,
digest,
verifier,
activeKeyIds: keySelection.activeKeyIds,
revokedKeyIds,
snapshot,
});
}
function assertForwardRotation(
current: LoadedKeyset,
candidate: LoadedKeyset,
): void {
if (candidate.generation < current.generation) {
throw configurationFailure('keyset generation rollback is forbidden');
}
if (
candidate.generation === current.generation &&
candidate.digest !== current.digest
) {
throw configurationFailure('keyset generation rewrite is forbidden');
}
if (candidate.generation === current.generation) return;
for (const kid of current.revokedKeyIds) {
if (!candidate.revokedKeyIds.has(kid)) {
throw configurationFailure('revoked key ids are append-only');
}
}
for (const kid of current.activeKeyIds) {
if (!candidate.activeKeyIds.has(kid) && !candidate.revokedKeyIds.has(kid)) {
throw configurationFailure(
'removed active keys must be explicitly revoked',
);
}
}
}
export function createClusterPluginPackageIdentityKeysetFile(
options: ClusterPluginPackageIdentityKeysetFileOptions,
): Readonly<ClusterPluginPackageIdentityKeysetFile> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
key !== 'filePath' &&
key !== 'maxFileBytes' &&
key !== 'now' &&
key !== 'ledger' &&
key !== 'assertionProfile',
) ||
typeof options.filePath !== 'string' ||
options.filePath.length < 1 ||
options.filePath.length > 4_096 ||
CONTROL_PATTERN.test(options.filePath) ||
!isAbsolute(options.filePath) ||
(options.now !== undefined && typeof options.now !== 'function') ||
(options.ledger !== undefined &&
(!options.ledger ||
typeof options.ledger !== 'object' ||
typeof options.ledger.observe !== 'function'))
) {
throw configurationFailure('options are invalid');
}
const maxFileBytes =
options.maxFileBytes === undefined
? DEFAULT_MAX_FILE_BYTES
: boundedInteger(
options.maxFileBytes,
MIN_MAX_FILE_BYTES,
HARD_MAX_FILE_BYTES,
'maximum file bytes',
);
let current: LoadedKeyset | undefined;
const reload = async (): Promise<
Readonly<ClusterPluginPackageIdentityKeysetSnapshot>
> => {
try {
const bytes = await readBoundedRegularFile(
options.filePath,
maxFileBytes,
);
const digest = createHash('sha256').update(bytes).digest('base64url');
if (current?.digest === digest) {
await options.ledger?.observe(current.snapshot);
return current.snapshot;
}
const candidate = loadDocument(
bytes,
options.now,
digest,
options.assertionProfile,
);
if (current) {
assertForwardRotation(current, candidate);
}
await options.ledger?.observe(candidate.snapshot);
current = candidate;
return candidate.snapshot;
} catch (error) {
if (error instanceof ClusterPluginPackageIdentityKeysetUnavailableError) {
throw error;
}
throw new ClusterPluginPackageIdentityKeysetUnavailableError(error);
}
};
return Object.freeze({
reload,
bind(assertion: unknown) {
return Object.freeze({
async authenticate() {
await reload();
if (!current) {
throw new ClusterPluginPackageIdentityKeysetUnavailableError();
}
return current.verifier.verify(assertion);
},
});
},
});
}
export function createClusterWorkerCredentialIdentityKeysetFile(
options: ClusterWorkerCredentialIdentityKeysetFileOptions,
): Readonly<ClusterPluginPackageIdentityKeysetFile> {
return createClusterPluginPackageIdentityKeysetFile({
...options,
assertionProfile:
CLUSTER_WORKER_CREDENTIAL_MANAGEMENT_IDENTITY_ASSERTION_PROFILE,
});
}
export function createClusterAutomationIdentityKeysetFile(
options: ClusterWorkerCredentialIdentityKeysetFileOptions,
): Readonly<ClusterPluginPackageIdentityKeysetFile> {
return createClusterPluginPackageIdentityKeysetFile({
...options,
assertionProfile: CLUSTER_AUTOMATION_MANAGEMENT_IDENTITY_ASSERTION_PROFILE,
});
}
export function createClusterApprovalIdentityKeysetFile(
options: ClusterWorkerCredentialIdentityKeysetFileOptions,
): Readonly<ClusterPluginPackageIdentityKeysetFile> {
return createClusterPluginPackageIdentityKeysetFile({
...options,
assertionProfile: CLUSTER_APPROVAL_MANAGEMENT_IDENTITY_ASSERTION_PROFILE,
});
}
export function createClusterModelProviderCredentialIdentityKeysetFile(
options: ClusterWorkerCredentialIdentityKeysetFileOptions,
): Readonly<ClusterPluginPackageIdentityKeysetFile> {
return createClusterPluginPackageIdentityKeysetFile({
...options,
assertionProfile:
CLUSTER_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_IDENTITY_ASSERTION_PROFILE,
});
}
@@ -0,0 +1,919 @@
/** Shared bounded TLS HTTP host boundary for cluster management planes. */
import { randomUUID } from 'node:crypto';
import { createServer, type Server as HttpsServer } from 'node:https';
import type { IncomingMessage, ServerResponse } from 'node:http';
import type { Duplex } from 'node:stream';
import type { AddressInfo } from 'node:net';
import type { TLSSocket } from 'node:tls';
import {
PluginPackageManagementAuthorizationError,
PluginPackageManagementConflictError,
PluginPackageManagementQuotaExceededError,
PluginPackageManagementRequestError,
PluginPackageManagementUnavailableError,
} from '@qinglong/runtime-core/plugin-package-management';
import { ClusterPluginPackageIdentityAssertionAuthenticationError } from './pluginPackageIdentityAssertion';
import {
ClusterPluginPackageIdentityKeysetUnavailableError,
type ClusterPluginPackageIdentityKeysetFile,
} from './pluginPackageIdentityKeyset';
import {
ClusterPluginPackageManagementTransportAuthenticationError,
ClusterPluginPackageManagementTransportRequestError,
ClusterPluginPackageManagementTransportUnavailableError,
} from '../plugin-package/management/pluginPackageManagementTransport';
import {
WorkerCredentialManagementAuthorizationError,
WorkerCredentialManagementConflictError,
WorkerCredentialManagementQuotaExceededError,
WorkerCredentialManagementRequestError,
WorkerCredentialManagementUnavailableError,
} from '../worker-credential/management-server/workerCredentialManagement';
import {
ClusterWorkerCredentialManagementTransportAuthenticationError,
ClusterWorkerCredentialManagementTransportRequestError,
ClusterWorkerCredentialManagementTransportUnavailableError,
} from '../worker-credential/management-server/workerCredentialManagementTransport';
import {
ClusterAutomationManagementAuthorizationError,
ClusterAutomationManagementConflictError,
ClusterAutomationManagementRequestError,
ClusterAutomationManagementUnavailableError,
} from '../automation-management/automationManagement';
import {
ClusterAutomationManagementTransportAuthenticationError,
ClusterAutomationManagementTransportRequestError,
ClusterAutomationManagementTransportUnavailableError,
} from '../automation-management/automationManagementTransport';
import {
ClusterApprovalManagementTransportAuthenticationError,
ClusterApprovalManagementTransportAuthorizationError,
ClusterApprovalManagementTransportConflictError,
ClusterApprovalManagementTransportRequestError,
ClusterApprovalManagementTransportTargetUnavailableError,
ClusterApprovalManagementTransportUnavailableError,
} from '../approval-management/approvalManagementTransport';
import {
ClusterModelProviderCredentialManagementAuthenticationError,
ClusterModelProviderCredentialManagementAuthorizationError,
ClusterModelProviderCredentialManagementConflictError,
ClusterModelProviderCredentialManagementQuotaExceededError,
ClusterModelProviderCredentialManagementRequestError,
ClusterModelProviderCredentialManagementUnavailableError,
} from '../model-provider-credential/modelProviderCredentialManagement';
import {
ClusterModelProviderCredentialManagementTransportAuthenticationError,
ClusterModelProviderCredentialManagementTransportRequestError,
ClusterModelProviderCredentialManagementTransportUnavailableError,
} from '../model-provider-credential/modelProviderCredentialManagementTransport';
export const CLUSTER_PLUGIN_PACKAGE_MANAGEMENT_PATH =
'/api/v3/plugin-packages/management';
export const CLUSTER_WORKER_CREDENTIAL_MANAGEMENT_PATH =
'/api/v3/worker-credentials/management';
export const CLUSTER_AUTOMATION_MANAGEMENT_PATH =
'/api/v3/automations/management';
export const CLUSTER_APPROVAL_MANAGEMENT_PATH = '/api/v3/approvals/management';
export const CLUSTER_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_PATH =
'/api/v3/provider-credentials/management';
export type ClusterAuthenticatedManagementPath =
| typeof CLUSTER_PLUGIN_PACKAGE_MANAGEMENT_PATH
| typeof CLUSTER_WORKER_CREDENTIAL_MANAGEMENT_PATH
| typeof CLUSTER_AUTOMATION_MANAGEMENT_PATH
| typeof CLUSTER_APPROVAL_MANAGEMENT_PATH
| typeof CLUSTER_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_PATH;
const MANAGEMENT_PATHS = new Set<ClusterAuthenticatedManagementPath>([
CLUSTER_PLUGIN_PACKAGE_MANAGEMENT_PATH,
CLUSTER_WORKER_CREDENTIAL_MANAGEMENT_PATH,
CLUSTER_AUTOMATION_MANAGEMENT_PATH,
CLUSTER_APPROVAL_MANAGEMENT_PATH,
CLUSTER_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_PATH,
]);
const DEFAULT_MAX_BODY_BYTES = 64 * 1024;
const DEFAULT_MAX_CONNECTIONS = 64;
const DEFAULT_MAX_CONCURRENT_REQUESTS = 32;
const DEFAULT_REQUEST_TIMEOUT_MS = 10_000;
const DEFAULT_DRAIN_TIMEOUT_MS = 5_000;
const DEFAULT_RATE_WINDOW_MS = 60_000;
const DEFAULT_PEER_REQUEST_LIMIT = 60;
const DEFAULT_GLOBAL_REQUEST_LIMIT = 600;
const DEFAULT_MAX_RATE_LIMIT_PEERS = 1_024;
const MAX_AUTHORIZATION_BYTES = 16 * 1024;
const MAX_RESPONSE_BYTES = 128 * 1024;
const CONTROL_PATTERN = /[\u0000-\u001f\u007f]/;
export interface ClusterPluginPackageManagementHttpLimits {
readonly maxBodyBytes?: number;
readonly maxConnections?: number;
readonly maxConcurrentRequests?: number;
readonly requestTimeoutMs?: number;
readonly drainTimeoutMs?: number;
readonly rateWindowMs?: number;
readonly peerRequestLimit?: number;
readonly globalRequestLimit?: number;
readonly maxRateLimitPeers?: number;
}
export interface StartClusterPluginPackageManagementHttpOptions {
readonly host: string;
readonly port: number;
readonly tls: Readonly<{
readonly privateKey: Buffer;
readonly certificate: Buffer;
readonly clientCertificateAuthority?: Buffer;
readonly clientCertificateRevocationList?: Buffer;
}>;
readonly transport: ClusterAuthenticatedManagementTransport;
readonly identities: ClusterPluginPackageIdentityKeysetFile;
readonly managementPath?: ClusterAuthenticatedManagementPath;
readonly limits?: ClusterPluginPackageManagementHttpLimits;
readonly now?: () => number;
readonly createRequestId?: () => string;
readonly onError?: (error: unknown) => void;
}
export interface ClusterAuthenticatedManagementTransport {
execute(
command: unknown,
authentication: Readonly<{
authenticate(): Promise<unknown>;
}>,
): Promise<unknown>;
}
export interface ClusterPluginPackageManagementHttpApplication {
readonly status: 'active';
readonly address: Readonly<{ host: string; port: number }>;
availabilityStatus(): 'ready' | 'unavailable' | 'stopped';
withdraw(error?: unknown): void;
close(): Promise<void>;
}
export class ClusterPluginPackageManagementHttpConfigurationError extends TypeError {
readonly code = 'CLUSTER_PLUGIN_PACKAGE_MANAGEMENT_HTTP_CONFIG_INVALID';
constructor(message: string) {
super(
`Cluster Plugin Package management HTTP configuration is invalid: ${message}`,
);
this.name = 'ClusterPluginPackageManagementHttpConfigurationError';
}
}
class HttpRequestError extends Error {
constructor(
readonly statusCode: number,
readonly responseCode: string,
readonly retryAfterMs?: number,
) {
super(responseCode);
}
}
interface ReviewedLimits {
readonly maxBodyBytes: number;
readonly maxConnections: number;
readonly maxConcurrentRequests: number;
readonly requestTimeoutMs: number;
readonly drainTimeoutMs: number;
readonly rateWindowMs: number;
readonly peerRequestLimit: number;
readonly globalRequestLimit: number;
readonly maxRateLimitPeers: number;
}
interface RateBucket {
windowStartedAtMs: number;
count: number;
lastSeenAtMs: number;
}
class BoundedRateLimiter {
readonly #peers = new Map<string, RateBucket>();
#global: RateBucket;
constructor(
private readonly limits: ReviewedLimits,
private readonly now: () => number,
) {
const nowMs = this.currentTime();
this.#global = {
windowStartedAtMs: nowMs,
count: 0,
lastSeenAtMs: nowMs,
};
}
private currentTime(): number {
const value = this.now();
if (!Number.isSafeInteger(value) || value < 0) {
throw new Error('HTTP rate-limit clock is invalid');
}
return value;
}
private retryAfter(
bucket: RateBucket,
limit: number,
nowMs: number,
): number | null {
if (nowMs >= bucket.windowStartedAtMs + this.limits.rateWindowMs) {
bucket.windowStartedAtMs = nowMs;
bucket.count = 0;
}
bucket.lastSeenAtMs = nowMs;
if (bucket.count >= limit) {
return Math.max(
1,
bucket.windowStartedAtMs + this.limits.rateWindowMs - nowMs,
);
}
return null;
}
private evictOldestPeer(): void {
let oldestKey: string | undefined;
let oldestAtMs = Number.POSITIVE_INFINITY;
for (const [key, bucket] of this.#peers) {
if (bucket.lastSeenAtMs < oldestAtMs) {
oldestAtMs = bucket.lastSeenAtMs;
oldestKey = key;
}
}
if (oldestKey !== undefined) this.#peers.delete(oldestKey);
}
consume(peerValue: string | undefined): number | null {
const nowMs = this.currentTime();
const globalRetry = this.retryAfter(
this.#global,
this.limits.globalRequestLimit,
nowMs,
);
if (globalRetry !== null) return globalRetry;
const peer =
typeof peerValue === 'string' &&
peerValue.length >= 1 &&
peerValue.length <= 128 &&
!CONTROL_PATTERN.test(peerValue)
? peerValue
: '<unknown>';
let bucket = this.#peers.get(peer);
if (!bucket) {
if (this.#peers.size >= this.limits.maxRateLimitPeers) {
this.evictOldestPeer();
}
bucket = {
windowStartedAtMs: nowMs,
count: 0,
lastSeenAtMs: nowMs,
};
this.#peers.set(peer, bucket);
}
const peerRetry = this.retryAfter(
bucket,
this.limits.peerRequestLimit,
nowMs,
);
if (peerRetry !== null) return peerRetry;
this.#global.count += 1;
bucket.count += 1;
return null;
}
}
function configurationFailure(
message: string,
): ClusterPluginPackageManagementHttpConfigurationError {
return new ClusterPluginPackageManagementHttpConfigurationError(message);
}
function integer(
value: number | undefined,
fallback: number,
minimum: number,
maximum: number,
label: string,
): number {
const candidate = value ?? fallback;
if (
!Number.isSafeInteger(candidate) ||
candidate < minimum ||
candidate > maximum
) {
throw configurationFailure(`${label} is invalid`);
}
return candidate;
}
function reviewedLimits(
value: ClusterPluginPackageManagementHttpLimits | undefined,
): ReviewedLimits {
if (
value !== undefined &&
(!value ||
typeof value !== 'object' ||
Array.isArray(value) ||
Object.keys(value).some(
(key) =>
![
'maxBodyBytes',
'maxConnections',
'maxConcurrentRequests',
'requestTimeoutMs',
'drainTimeoutMs',
'rateWindowMs',
'peerRequestLimit',
'globalRequestLimit',
'maxRateLimitPeers',
].includes(key),
))
) {
throw configurationFailure('limits are invalid');
}
const limits = value ?? {};
const reviewed = {
maxBodyBytes: integer(
limits.maxBodyBytes,
DEFAULT_MAX_BODY_BYTES,
1_024,
256 * 1024,
'maximum body bytes',
),
maxConnections: integer(
limits.maxConnections,
DEFAULT_MAX_CONNECTIONS,
1,
512,
'maximum connections',
),
maxConcurrentRequests: integer(
limits.maxConcurrentRequests,
DEFAULT_MAX_CONCURRENT_REQUESTS,
1,
256,
'maximum concurrent requests',
),
requestTimeoutMs: integer(
limits.requestTimeoutMs,
DEFAULT_REQUEST_TIMEOUT_MS,
1_000,
60_000,
'request timeout',
),
drainTimeoutMs: integer(
limits.drainTimeoutMs,
DEFAULT_DRAIN_TIMEOUT_MS,
100,
60_000,
'drain timeout',
),
rateWindowMs: integer(
limits.rateWindowMs,
DEFAULT_RATE_WINDOW_MS,
1_000,
5 * 60_000,
'rate window',
),
peerRequestLimit: integer(
limits.peerRequestLimit,
DEFAULT_PEER_REQUEST_LIMIT,
1,
10_000,
'peer request limit',
),
globalRequestLimit: integer(
limits.globalRequestLimit,
DEFAULT_GLOBAL_REQUEST_LIMIT,
1,
100_000,
'global request limit',
),
maxRateLimitPeers: integer(
limits.maxRateLimitPeers,
DEFAULT_MAX_RATE_LIMIT_PEERS,
1,
16_384,
'maximum rate-limit peers',
),
};
if (reviewed.globalRequestLimit < reviewed.peerRequestLimit) {
throw configurationFailure(
'global request limit cannot be below the peer limit',
);
}
return Object.freeze(reviewed);
}
function rawHeaderCount(request: IncomingMessage, name: string): number {
let count = 0;
for (let index = 0; index < request.rawHeaders.length; index += 2) {
if (request.rawHeaders[index]?.toLowerCase() === name) count += 1;
}
return count;
}
function bearerAssertion(request: IncomingMessage): string {
if (
rawHeaderCount(request, 'authorization') !== 1 ||
typeof request.headers.authorization !== 'string'
) {
throw new HttpRequestError(401, 'authentication_required');
}
const value = request.headers.authorization;
if (
!value.startsWith('Bearer ') ||
value.length <= 7 ||
Buffer.byteLength(value, 'utf8') > MAX_AUTHORIZATION_BYTES ||
CONTROL_PATTERN.test(value)
) {
throw new HttpRequestError(401, 'authentication_required');
}
return value.slice(7);
}
function assertRequestHeaders(
request: IncomingMessage,
maxBodyBytes: number,
): void {
if (
rawHeaderCount(request, 'content-type') !== 1 ||
request.headers['content-type'] !== 'application/json'
) {
throw new HttpRequestError(415, 'unsupported_media_type');
}
if (
request.headers['content-encoding'] !== undefined ||
request.headers.expect !== undefined
) {
throw new HttpRequestError(400, 'request_invalid');
}
if (rawHeaderCount(request, 'content-length') > 1) {
throw new HttpRequestError(400, 'request_invalid');
}
const contentLength = request.headers['content-length'];
if (contentLength !== undefined) {
if (
!/^(?:0|[1-9][0-9]*)$/.test(contentLength) ||
Number(contentLength) > maxBodyBytes
) {
throw new HttpRequestError(413, 'request_too_large');
}
}
}
async function readJsonBody(
request: IncomingMessage,
maxBodyBytes: number,
): Promise<unknown> {
const chunks: Buffer[] = [];
let length = 0;
for await (const value of request) {
const chunk = Buffer.isBuffer(value) ? value : Buffer.from(value);
length += chunk.length;
if (length > maxBodyBytes) {
throw new HttpRequestError(413, 'request_too_large');
}
chunks.push(chunk);
}
if (length < 1) {
throw new HttpRequestError(400, 'request_invalid');
}
let text: string;
try {
text = new TextDecoder('utf-8', { fatal: true }).decode(
Buffer.concat(chunks, length),
);
} catch {
throw new HttpRequestError(400, 'request_invalid');
}
try {
return JSON.parse(text);
} catch {
throw new HttpRequestError(400, 'request_invalid');
}
}
function writeJson(
response: ServerResponse,
statusCode: number,
value: Readonly<Record<string, unknown>>,
retryAfterMs?: number,
): void {
const body = Buffer.from(JSON.stringify(value));
if (body.length > MAX_RESPONSE_BYTES) {
throw new Error('management response exceeds its hard limit');
}
response.statusCode = statusCode;
response.setHeader('content-type', 'application/json; charset=utf-8');
response.setHeader('content-length', String(body.length));
response.setHeader('cache-control', 'no-store');
response.setHeader('x-content-type-options', 'nosniff');
if (retryAfterMs !== undefined) {
response.setHeader(
'retry-after',
String(Math.max(1, Math.ceil(retryAfterMs / 1_000))),
);
}
response.end(body);
}
function responseError(error: unknown): HttpRequestError {
if (error instanceof HttpRequestError) return error;
if (
error instanceof ClusterPluginPackageIdentityAssertionAuthenticationError ||
error instanceof
ClusterPluginPackageManagementTransportAuthenticationError ||
error instanceof
ClusterWorkerCredentialManagementTransportAuthenticationError ||
error instanceof ClusterAutomationManagementTransportAuthenticationError ||
error instanceof ClusterApprovalManagementTransportAuthenticationError ||
error instanceof
ClusterModelProviderCredentialManagementTransportAuthenticationError ||
error instanceof ClusterModelProviderCredentialManagementAuthenticationError
) {
return new HttpRequestError(401, 'authentication_required');
}
if (
error instanceof ClusterPluginPackageManagementTransportRequestError ||
error instanceof ClusterWorkerCredentialManagementTransportRequestError ||
error instanceof ClusterAutomationManagementTransportRequestError ||
error instanceof ClusterApprovalManagementTransportRequestError ||
error instanceof ClusterAutomationManagementRequestError ||
error instanceof
ClusterModelProviderCredentialManagementTransportRequestError ||
error instanceof ClusterModelProviderCredentialManagementRequestError ||
error instanceof PluginPackageManagementRequestError ||
error instanceof WorkerCredentialManagementRequestError
) {
return new HttpRequestError(400, 'request_invalid');
}
if (
error instanceof PluginPackageManagementAuthorizationError ||
error instanceof WorkerCredentialManagementAuthorizationError ||
error instanceof ClusterAutomationManagementAuthorizationError ||
error instanceof ClusterApprovalManagementTransportAuthorizationError ||
error instanceof ClusterModelProviderCredentialManagementAuthorizationError
) {
return new HttpRequestError(403, 'forbidden');
}
if (
error instanceof PluginPackageManagementConflictError ||
error instanceof WorkerCredentialManagementConflictError ||
error instanceof ClusterAutomationManagementConflictError ||
error instanceof ClusterApprovalManagementTransportConflictError ||
error instanceof ClusterModelProviderCredentialManagementConflictError
) {
return new HttpRequestError(409, 'conflict');
}
if (error instanceof PluginPackageManagementQuotaExceededError) {
return new HttpRequestError(429, 'quota_exceeded', error.retryAfterMs);
}
if (error instanceof WorkerCredentialManagementQuotaExceededError) {
return new HttpRequestError(429, 'quota_exceeded', error.retryAfterMs);
}
if (
error instanceof ClusterModelProviderCredentialManagementQuotaExceededError
) {
return new HttpRequestError(429, 'quota_exceeded', error.retryAfterMs);
}
if (
error instanceof ClusterApprovalManagementTransportTargetUnavailableError
) {
return new HttpRequestError(404, 'not_found');
}
if (
error instanceof ClusterPluginPackageIdentityKeysetUnavailableError ||
error instanceof ClusterPluginPackageManagementTransportUnavailableError ||
error instanceof
ClusterWorkerCredentialManagementTransportUnavailableError ||
error instanceof ClusterAutomationManagementTransportUnavailableError ||
error instanceof ClusterApprovalManagementTransportUnavailableError ||
error instanceof ClusterAutomationManagementUnavailableError ||
error instanceof
ClusterModelProviderCredentialManagementTransportUnavailableError ||
error instanceof ClusterModelProviderCredentialManagementUnavailableError ||
error instanceof PluginPackageManagementUnavailableError ||
error instanceof WorkerCredentialManagementUnavailableError
) {
return new HttpRequestError(503, 'unavailable');
}
return new HttpRequestError(500, 'internal_error');
}
async function listen(
server: HttpsServer,
port: number,
host: string,
): Promise<void> {
await new Promise<void>((resolve, reject) => {
const onError = (error: Error) => {
server.removeListener('listening', onListening);
reject(error);
};
const onListening = () => {
server.removeListener('error', onError);
resolve();
};
server.once('error', onError);
server.once('listening', onListening);
server.listen(port, host);
});
}
export async function startClusterPluginPackageManagementHttp(
options: StartClusterPluginPackageManagementHttpOptions,
): Promise<Readonly<ClusterPluginPackageManagementHttpApplication>> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
![
'host',
'port',
'tls',
'transport',
'identities',
'managementPath',
'limits',
'now',
'createRequestId',
'onError',
].includes(key),
) ||
typeof options.host !== 'string' ||
options.host.length < 1 ||
options.host.length > 255 ||
CONTROL_PATTERN.test(options.host) ||
!Number.isInteger(options.port) ||
options.port < 0 ||
options.port > 65_535 ||
!options.tls ||
typeof options.tls !== 'object' ||
Array.isArray(options.tls) ||
Object.keys(options.tls).some(
(key) =>
key !== 'privateKey' &&
key !== 'certificate' &&
key !== 'clientCertificateAuthority' &&
key !== 'clientCertificateRevocationList',
) ||
!Buffer.isBuffer(options.tls.privateKey) ||
options.tls.privateKey.length < 1 ||
options.tls.privateKey.length > 256 * 1024 ||
!Buffer.isBuffer(options.tls.certificate) ||
options.tls.certificate.length < 1 ||
options.tls.certificate.length > 256 * 1024 ||
(options.tls.clientCertificateAuthority !== undefined &&
(!Buffer.isBuffer(options.tls.clientCertificateAuthority) ||
options.tls.clientCertificateAuthority.length < 1 ||
options.tls.clientCertificateAuthority.length > 256 * 1024)) ||
(options.tls.clientCertificateRevocationList !== undefined &&
(!Buffer.isBuffer(options.tls.clientCertificateRevocationList) ||
options.tls.clientCertificateRevocationList.length < 1 ||
options.tls.clientCertificateRevocationList.length > 256 * 1024)) ||
(options.tls.clientCertificateAuthority === undefined) !==
(options.tls.clientCertificateRevocationList === undefined) ||
!options.transport ||
typeof options.transport.execute !== 'function' ||
!options.identities ||
typeof options.identities.bind !== 'function' ||
typeof options.identities.reload !== 'function' ||
(options.managementPath !== undefined &&
!MANAGEMENT_PATHS.has(options.managementPath)) ||
(options.now !== undefined && typeof options.now !== 'function') ||
(options.createRequestId !== undefined &&
typeof options.createRequestId !== 'function') ||
(options.onError !== undefined && typeof options.onError !== 'function')
) {
throw configurationFailure('options are invalid');
}
const limits = reviewedLimits(options.limits);
const managementPath =
options.managementPath ?? CLUSTER_PLUGIN_PACKAGE_MANAGEMENT_PATH;
const now = options.now ?? Date.now;
const createRequestId = options.createRequestId ?? randomUUID;
const clientCertificateRequired =
options.tls.clientCertificateAuthority !== undefined;
const rateLimiter = new BoundedRateLimiter(limits, now);
let availability: 'ready' | 'unavailable' | 'stopped' = 'ready';
let inFlight = 0;
const sockets = new Set<Duplex>();
let server: HttpsServer;
try {
server = createServer({
key: options.tls.privateKey,
cert: options.tls.certificate,
minVersion: 'TLSv1.3',
maxVersion: 'TLSv1.3',
honorCipherOrder: true,
...(clientCertificateRequired
? {
ca: options.tls.clientCertificateAuthority,
crl: options.tls.clientCertificateRevocationList,
}
: {}),
requestCert: clientCertificateRequired,
// Health probes intentionally remain reachable without a client
// certificate. Every non-health route checks TLSSocket.authorized before
// reading Authorization or request body bytes.
rejectUnauthorized: false,
});
} finally {
options.tls.privateKey.fill(0);
}
server.maxHeadersCount = 32;
server.maxConnections = limits.maxConnections;
server.requestTimeout = limits.requestTimeoutMs;
server.headersTimeout = Math.min(5_000, limits.requestTimeoutMs);
server.keepAliveTimeout = 5_000;
server.maxRequestsPerSocket = 100;
server.on('connection', (socket) => {
sockets.add(socket);
socket.once('close', () => sockets.delete(socket));
});
const report = (error: unknown): void => {
try {
options.onError?.(error);
} catch {
// Diagnostics must never replace the stable HTTP response.
}
};
server.on('checkContinue', (request, response) => {
response.setHeader('connection', 'close');
response.once('finish', () => request.destroy());
writeJson(response, 417, {
schemaVersion: 1,
error: { code: 'request_invalid' },
});
});
server.on('clientError', (_error, socket) => {
if (socket.writable) {
socket.end(
'HTTP/1.1 400 Bad Request\r\nConnection: close\r\nContent-Length: 0\r\n\r\n',
);
}
});
server.on('tlsClientError', () => {
// TLS failures are unauthenticated network noise, not diagnostics.
});
server.on('request', (request, response) => {
void (async () => {
const requestId = createRequestId();
if (
typeof requestId !== 'string' ||
requestId.length < 1 ||
requestId.length > 128 ||
CONTROL_PATTERN.test(requestId)
) {
throw new Error('HTTP request id is invalid');
}
response.setHeader('x-request-id', requestId);
const url = request.url;
if (request.method === 'GET' && url === '/livez') {
writeJson(response, 200, {
schemaVersion: 1,
status: 'live',
});
return;
}
if (request.method === 'GET' && url === '/readyz') {
writeJson(response, availability === 'ready' ? 200 : 503, {
schemaVersion: 1,
status: availability === 'ready' ? 'ready' : 'not_ready',
});
return;
}
if (
clientCertificateRequired &&
!(request.socket as TLSSocket).authorized
) {
throw new HttpRequestError(401, 'client_certificate_required');
}
if (request.method !== 'POST' || url !== managementPath) {
throw new HttpRequestError(404, 'not_found');
}
if (availability !== 'ready') {
throw new HttpRequestError(503, 'unavailable');
}
const retryAfterMs = rateLimiter.consume(request.socket.remoteAddress);
if (retryAfterMs !== null) {
writeJson(
response,
429,
{
schemaVersion: 1,
requestId,
error: { code: 'rate_limited' },
},
retryAfterMs,
);
return;
}
if (inFlight >= limits.maxConcurrentRequests) {
throw new HttpRequestError(503, 'overloaded');
}
inFlight += 1;
try {
const assertion = bearerAssertion(request);
const authentication = options.identities.bind(assertion);
const principal = await authentication.authenticate();
assertRequestHeaders(request, limits.maxBodyBytes);
const command = await readJsonBody(request, limits.maxBodyBytes);
const result = await options.transport.execute(
command,
Object.freeze({
async authenticate() {
return principal;
},
}),
);
writeJson(response, 200, {
schemaVersion: 1,
requestId,
result,
});
} finally {
inFlight -= 1;
}
})().catch((error) => {
const mapped = responseError(error);
if (mapped.statusCode === 500) report(error);
if (!request.destroyed) {
response.once('finish', () => request.destroy());
}
if (!response.headersSent && !response.destroyed) {
response.setHeader('connection', 'close');
writeJson(
response,
mapped.statusCode,
{
schemaVersion: 1,
requestId:
typeof response.getHeader('x-request-id') === 'string'
? response.getHeader('x-request-id')
: 'unavailable',
error: { code: mapped.responseCode },
},
mapped.retryAfterMs,
);
} else if (!response.destroyed) {
response.destroy();
}
});
});
try {
await listen(server, options.port, options.host);
} catch (error) {
for (const socket of sockets) socket.destroy();
throw error;
}
const address = server.address();
if (!address || typeof address === 'string') {
for (const socket of sockets) socket.destroy();
throw new Error('management HTTP server address is unavailable');
}
const networkAddress = address as AddressInfo;
let closePromise: Promise<void> | undefined;
return Object.freeze({
status: 'active' as const,
address: Object.freeze({
host: networkAddress.address,
port: networkAddress.port,
}),
availabilityStatus: () => availability,
withdraw(error?: unknown) {
if (availability !== 'ready') return;
availability = 'unavailable';
if (error !== undefined) report(error);
},
close(): Promise<void> {
if (closePromise) return closePromise;
availability = 'stopped';
closePromise = new Promise<void>((resolve) => {
let settled = false;
const finish = () => {
if (settled) return;
settled = true;
clearTimeout(timer);
resolve();
};
const timer = setTimeout(() => {
for (const socket of sockets) socket.destroy();
finish();
}, limits.drainTimeoutMs);
server.close(finish);
});
return closePromise;
},
});
}
@@ -0,0 +1,608 @@
/** Model-provider credential management application service boundary. */
import {
InvalidModelProviderCredentialAdministrationMutationError,
ModelProviderCredentialAdministrationAuthorizationFenceConflictError,
ModelProviderCredentialAdministrationMutationConflictError,
modelProviderCredentialAdministrationOperationId,
type ModelProviderCredentialAdministrationRepository,
} from '@qinglong/ai/model-provider-credential-administration';
import {
MODEL_PROVIDER_CREDENTIAL_TRANSITION_COMMAND_SCHEMA,
InvalidModelProviderCredentialTransitionError,
ModelProviderCredentialCatalogUnavailableError,
ModelProviderCredentialTransitionConflictError,
createModelProviderCredentialTransitionCommand,
type CommitModelProviderCredentialTransitionResult,
} from '@qinglong/ai/model-provider-credential-catalog';
import { MODEL_PROVIDER_CREDENTIAL_BINDING_SCHEMA } from '@qinglong/ai/provider-credential';
import {
MAX_MODEL_PROVIDER_CREDENTIAL_TEST_PLAN_LIFETIME_MS,
InvalidModelProviderCredentialTestConnectionError,
createModelProviderCredentialTestPlan,
normalizeModelProviderCredentialTestAllowlist,
resolveModelProviderCredentialTestEndpoint,
type ModelProviderCredentialTestAllowlist,
} from '@qinglong/ai/model-provider-credential-test-connection';
import {
InvalidModelProviderCredentialManagementAuditQueryError,
MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_AUDIT_QUERY_OPERATION_ID,
ModelProviderCredentialManagementAuditAuthorizationFenceConflictError,
ModelProviderCredentialManagementAuditConflictError,
ModelProviderCredentialManagementAuditUnavailableError,
normalizeModelProviderCredentialManagementAuditQuery,
type ModelProviderCredentialManagementAuditCursor,
type ModelProviderCredentialManagementAuditPage,
type ModelProviderCredentialManagementAuditQueryRepository,
} from '@qinglong/ai/postgres-model-provider-credential-management-audit-query';
import {
MODEL_PROVIDER_CREDENTIAL_TEST_PLAN_OPERATION_ID,
ModelProviderCredentialTestPlanAuthorizationFenceConflictError,
ModelProviderCredentialTestPlanConflictError,
ModelProviderCredentialTestPlanQuotaExceededError,
ModelProviderCredentialTestPlanUnavailableError,
type CreateModelProviderCredentialTestPlanResult,
type ModelProviderCredentialTestPlanRepository,
} from '@qinglong/ai/postgres-model-provider-credential-test-connection';
import type { ProjectPermission } from '@qinglong/runtime-core/project-policy';
import {
normalizeSecurityPrincipal,
type SecurityPolicyDecision,
type SecurityPrincipal,
} from '@qinglong/runtime-core/security';
const REQUEST_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const IDENTITY_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:/-]{0,127}$/;
const UUID_PATTERN =
/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/;
const STRONG_USER_ASSURANCES = new Set(['multi_factor', 'hardware']);
const MAX_PRINCIPAL_AGE_MS = 5 * 60 * 1_000;
interface BaseMutationRequest {
readonly requestId: string;
readonly mutationId: string;
readonly projectId: string;
readonly provider: string;
readonly expectedGeneration: number;
readonly principal: SecurityPrincipal;
}
export interface BindModelProviderCredentialRequest
extends BaseMutationRequest {
readonly revision: string;
readonly secretRef: string;
}
export interface RevokeModelProviderCredentialRequest
extends BaseMutationRequest {}
export interface ListModelProviderCredentialManagementAuditRequest {
readonly requestId: string;
readonly queryId: string;
readonly projectId: string;
readonly limit: number;
readonly before?: ModelProviderCredentialManagementAuditCursor;
readonly principal: SecurityPrincipal;
}
export interface PlanModelProviderCredentialTestRequest {
readonly requestId: string;
readonly testId: string;
readonly projectId: string;
readonly provider: string;
readonly principal: SecurityPrincipal;
}
export interface ClusterModelProviderCredentialManagementPolicy {
authorize(
principal: Readonly<SecurityPrincipal>,
projectId: string,
permission: ProjectPermission,
): Promise<Readonly<SecurityPolicyDecision>>;
}
export interface ClusterModelProviderCredentialManagementService {
bind(
request: Readonly<BindModelProviderCredentialRequest>,
): Promise<Readonly<CommitModelProviderCredentialTransitionResult>>;
revoke(
request: Readonly<RevokeModelProviderCredentialRequest>,
): Promise<Readonly<CommitModelProviderCredentialTransitionResult>>;
listAudit(
request: Readonly<ListModelProviderCredentialManagementAuditRequest>,
): Promise<Readonly<ModelProviderCredentialManagementAuditPage>>;
planTestConnection(
request: Readonly<PlanModelProviderCredentialTestRequest>,
): Promise<Readonly<CreateModelProviderCredentialTestPlanResult>>;
}
export interface ClusterModelProviderCredentialManagementOptions {
readonly policy: ClusterModelProviderCredentialManagementPolicy;
readonly credentials: ModelProviderCredentialAdministrationRepository;
readonly audit: ModelProviderCredentialManagementAuditQueryRepository;
readonly testPlans: ModelProviderCredentialTestPlanRepository;
readonly testAllowlist: ModelProviderCredentialTestAllowlist;
readonly testPlanLifetimeMs?: number;
readonly now?: () => number;
}
export class ClusterModelProviderCredentialManagementRequestError extends TypeError {
readonly code =
'CLUSTER_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_REQUEST_INVALID';
constructor() {
super('Cluster model provider credential management request is invalid');
this.name = 'ClusterModelProviderCredentialManagementRequestError';
}
}
export class ClusterModelProviderCredentialManagementAuthenticationError extends Error {
readonly code =
'CLUSTER_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_AUTHENTICATION_REQUIRED';
constructor() {
super(
'Cluster model provider credential management requires a recent strong User',
);
this.name = 'ClusterModelProviderCredentialManagementAuthenticationError';
}
}
export class ClusterModelProviderCredentialManagementAuthorizationError extends Error {
readonly code = 'CLUSTER_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_FORBIDDEN';
constructor() {
super('Cluster model provider credential management is forbidden');
this.name = 'ClusterModelProviderCredentialManagementAuthorizationError';
}
}
export class ClusterModelProviderCredentialManagementConflictError extends Error {
readonly code = 'CLUSTER_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_CONFLICT';
constructor() {
super(
'Cluster model provider credential management conflicts with durable state',
);
this.name = 'ClusterModelProviderCredentialManagementConflictError';
}
}
export class ClusterModelProviderCredentialManagementUnavailableError extends Error {
readonly code = 'CLUSTER_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_UNAVAILABLE';
constructor() {
super('Cluster model provider credential management is unavailable');
this.name = 'ClusterModelProviderCredentialManagementUnavailableError';
}
}
export class ClusterModelProviderCredentialManagementQuotaExceededError extends Error {
readonly code = 'CLUSTER_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_QUOTA_EXCEEDED';
constructor(readonly retryAfterMs: number) {
super('Cluster model provider credential management quota is exceeded');
this.name = 'ClusterModelProviderCredentialManagementQuotaExceededError';
}
}
function exactKeys(value: object, expected: readonly string[]): boolean {
const actual = Object.keys(value).sort();
const canonical = [...expected].sort();
return (
actual.length === canonical.length &&
actual.every((key, index) => key === canonical[index])
);
}
function normalizeBaseRequest(
value: BaseMutationRequest,
expectedKeys: readonly string[],
): Readonly<BaseMutationRequest> {
if (
!value ||
typeof value !== 'object' ||
Array.isArray(value) ||
!exactKeys(value, expectedKeys) ||
typeof value.requestId !== 'string' ||
!REQUEST_ID_PATTERN.test(value.requestId) ||
typeof value.mutationId !== 'string' ||
!UUID_PATTERN.test(value.mutationId) ||
typeof value.projectId !== 'string' ||
!IDENTITY_PATTERN.test(value.projectId) ||
typeof value.provider !== 'string' ||
!IDENTITY_PATTERN.test(value.provider) ||
!Number.isSafeInteger(value.expectedGeneration) ||
value.expectedGeneration < 0 ||
value.expectedGeneration > 2_147_483_646
) {
throw new ClusterModelProviderCredentialManagementRequestError();
}
return value;
}
function mapMutationError(error: unknown): never {
if (
error instanceof
InvalidModelProviderCredentialAdministrationMutationError ||
error instanceof InvalidModelProviderCredentialTransitionError
) {
throw new ClusterModelProviderCredentialManagementRequestError();
}
if (
error instanceof
ModelProviderCredentialAdministrationAuthorizationFenceConflictError ||
error instanceof
ModelProviderCredentialAdministrationMutationConflictError ||
error instanceof ModelProviderCredentialTransitionConflictError
) {
throw new ClusterModelProviderCredentialManagementConflictError();
}
if (error instanceof ModelProviderCredentialCatalogUnavailableError) {
throw new ClusterModelProviderCredentialManagementUnavailableError();
}
throw new ClusterModelProviderCredentialManagementUnavailableError();
}
function mapAuditError(error: unknown): never {
if (
error instanceof InvalidModelProviderCredentialManagementAuditQueryError
) {
throw new ClusterModelProviderCredentialManagementRequestError();
}
if (
error instanceof
ModelProviderCredentialManagementAuditAuthorizationFenceConflictError ||
error instanceof ModelProviderCredentialManagementAuditConflictError
) {
throw new ClusterModelProviderCredentialManagementConflictError();
}
if (error instanceof ModelProviderCredentialManagementAuditUnavailableError) {
throw new ClusterModelProviderCredentialManagementUnavailableError();
}
throw new ClusterModelProviderCredentialManagementUnavailableError();
}
function mapTestPlanError(error: unknown): never {
if (error instanceof InvalidModelProviderCredentialTestConnectionError) {
throw new ClusterModelProviderCredentialManagementRequestError();
}
if (
error instanceof
ModelProviderCredentialTestPlanAuthorizationFenceConflictError ||
error instanceof ModelProviderCredentialTestPlanConflictError
) {
throw new ClusterModelProviderCredentialManagementConflictError();
}
if (error instanceof ModelProviderCredentialTestPlanQuotaExceededError) {
throw new ClusterModelProviderCredentialManagementQuotaExceededError(
error.retryAfterMs,
);
}
if (error instanceof ModelProviderCredentialTestPlanUnavailableError) {
throw new ClusterModelProviderCredentialManagementUnavailableError();
}
throw new ClusterModelProviderCredentialManagementUnavailableError();
}
export function createClusterModelProviderCredentialManagementService(
options: ClusterModelProviderCredentialManagementOptions,
): Readonly<ClusterModelProviderCredentialManagementService> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
key !== 'policy' &&
key !== 'credentials' &&
key !== 'audit' &&
key !== 'testPlans' &&
key !== 'testAllowlist' &&
key !== 'testPlanLifetimeMs' &&
key !== 'now',
) ||
!options.policy ||
typeof options.policy.authorize !== 'function' ||
!options.credentials ||
typeof options.credentials.commitAuthorized !== 'function' ||
typeof options.credentials.findCurrentTransition !== 'function' ||
typeof options.credentials.commit !== 'function' ||
!options.audit ||
typeof options.audit.listAuthorized !== 'function' ||
!options.testPlans ||
typeof options.testPlans.createAuthorized !== 'function' ||
(options.now !== undefined && typeof options.now !== 'function')
) {
throw new TypeError(
'Cluster model provider credential management options are invalid',
);
}
const now = options.now ?? Date.now;
let testAllowlist: Readonly<ModelProviderCredentialTestAllowlist>;
try {
testAllowlist = normalizeModelProviderCredentialTestAllowlist(
options.testAllowlist,
);
} catch {
throw new TypeError(
'Cluster model provider credential management options are invalid',
);
}
const testPlanLifetimeMs =
options.testPlanLifetimeMs ??
MAX_MODEL_PROVIDER_CREDENTIAL_TEST_PLAN_LIFETIME_MS;
if (
!Number.isSafeInteger(testPlanLifetimeMs) ||
testPlanLifetimeMs < 1_000 ||
testPlanLifetimeMs > MAX_MODEL_PROVIDER_CREDENTIAL_TEST_PLAN_LIFETIME_MS
) {
throw new TypeError(
'Cluster model provider credential management options are invalid',
);
}
const authorize = async (
principalValue: SecurityPrincipal,
projectId: string,
) => {
const observedAtMs = now();
if (!Number.isSafeInteger(observedAtMs) || observedAtMs < 0) {
throw new ClusterModelProviderCredentialManagementUnavailableError();
}
let principal: Readonly<SecurityPrincipal>;
try {
principal = normalizeSecurityPrincipal(principalValue, observedAtMs);
} catch {
throw new ClusterModelProviderCredentialManagementAuthenticationError();
}
if (
principal.subject.type !== 'user' ||
!STRONG_USER_ASSURANCES.has(principal.assurance) ||
observedAtMs - principal.authenticatedAtMs > MAX_PRINCIPAL_AGE_MS
) {
throw new ClusterModelProviderCredentialManagementAuthenticationError();
}
let decision: Readonly<SecurityPolicyDecision>;
try {
decision = await options.policy.authorize(
principal,
projectId,
'secret.manage',
);
} catch {
throw new ClusterModelProviderCredentialManagementUnavailableError();
}
if (
decision.effect !== 'allow' ||
decision.fence === null ||
decision.fence.bindingVersion === null
) {
throw new ClusterModelProviderCredentialManagementAuthorizationError();
}
return Object.freeze({ principal, decision, observedAtMs });
};
const commit = async (
request:
| Readonly<BindModelProviderCredentialRequest>
| Readonly<RevokeModelProviderCredentialRequest>,
action: 'bind' | 'revoke',
) => {
const authority = await authorize(request.principal, request.projectId);
let command;
try {
command = createModelProviderCredentialTransitionCommand({
schema: MODEL_PROVIDER_CREDENTIAL_TRANSITION_COMMAND_SCHEMA,
mutationId: request.mutationId,
projectId: request.projectId,
provider: request.provider,
expectedGeneration: request.expectedGeneration,
action,
binding:
action === 'bind'
? {
schema: MODEL_PROVIDER_CREDENTIAL_BINDING_SCHEMA,
projectId: request.projectId,
provider: request.provider,
revision: (request as BindModelProviderCredentialRequest)
.revision,
secretRef: (request as BindModelProviderCredentialRequest)
.secretRef,
scheme: 'bearer',
}
: null,
changedBy: authority.principal.subject,
});
} catch (error) {
return mapMutationError(error);
}
try {
return await options.credentials.commitAuthorized({
command,
actor: authority.principal.subject,
fence: authority.decision.fence!,
audit: {
eventId: command.mutationId,
requestId: request.requestId,
operationId: modelProviderCredentialAdministrationOperationId(
command.action,
),
projectId: command.projectId,
subject: authority.principal.subject,
authenticationId: authority.principal.authenticationId,
outcome: 'allowed',
reasons: authority.decision.reasons,
fence: authority.decision.fence,
occurredAtMs: authority.observedAtMs,
},
});
} catch (error) {
return mapMutationError(error);
}
};
return Object.freeze({
async bind(request: Readonly<BindModelProviderCredentialRequest>) {
normalizeBaseRequest(request, [
'expectedGeneration',
'mutationId',
'principal',
'projectId',
'provider',
'requestId',
'revision',
'secretRef',
]);
if (
typeof request.revision !== 'string' ||
!IDENTITY_PATTERN.test(request.revision) ||
typeof request.secretRef !== 'string'
) {
throw new ClusterModelProviderCredentialManagementRequestError();
}
return commit(request, 'bind');
},
async revoke(request: Readonly<RevokeModelProviderCredentialRequest>) {
normalizeBaseRequest(request, [
'expectedGeneration',
'mutationId',
'principal',
'projectId',
'provider',
'requestId',
]);
return commit(request, 'revoke');
},
async listAudit(
request: Readonly<ListModelProviderCredentialManagementAuditRequest>,
) {
if (
!request ||
typeof request !== 'object' ||
Array.isArray(request) ||
!exactKeys(
request,
request.before === undefined
? ['limit', 'principal', 'projectId', 'queryId', 'requestId']
: [
'before',
'limit',
'principal',
'projectId',
'queryId',
'requestId',
],
)
) {
throw new ClusterModelProviderCredentialManagementRequestError();
}
let query;
try {
query = normalizeModelProviderCredentialManagementAuditQuery({
schemaVersion: 1,
queryId: request.queryId,
requestId: request.requestId,
projectId: request.projectId,
limit: request.limit,
...(request.before === undefined ? {} : { before: request.before }),
});
} catch {
throw new ClusterModelProviderCredentialManagementRequestError();
}
const authority = await authorize(request.principal, request.projectId);
try {
return await options.audit.listAuthorized({
query,
actor: authority.principal.subject,
fence: authority.decision.fence!,
audit: {
eventId: request.queryId,
requestId: request.requestId,
operationId:
MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_AUDIT_QUERY_OPERATION_ID,
projectId: request.projectId,
subject: authority.principal.subject,
authenticationId: authority.principal.authenticationId,
outcome: 'allowed',
reasons: authority.decision.reasons,
fence: authority.decision.fence,
occurredAtMs: authority.observedAtMs,
},
});
} catch (error) {
return mapAuditError(error);
}
},
async planTestConnection(
request: Readonly<PlanModelProviderCredentialTestRequest>,
) {
if (
!request ||
typeof request !== 'object' ||
Array.isArray(request) ||
!exactKeys(request, [
'principal',
'projectId',
'provider',
'requestId',
'testId',
]) ||
typeof request.requestId !== 'string' ||
!REQUEST_ID_PATTERN.test(request.requestId) ||
typeof request.testId !== 'string' ||
!UUID_PATTERN.test(request.testId) ||
typeof request.projectId !== 'string' ||
!IDENTITY_PATTERN.test(request.projectId) ||
typeof request.provider !== 'string' ||
!IDENTITY_PATTERN.test(request.provider)
) {
throw new ClusterModelProviderCredentialManagementRequestError();
}
const authority = await authorize(request.principal, request.projectId);
let plan;
try {
const endpoint = resolveModelProviderCredentialTestEndpoint(
testAllowlist,
request.provider,
);
if (
authority.observedAtMs >
Number.MAX_SAFE_INTEGER - testPlanLifetimeMs
) {
throw new ModelProviderCredentialTestPlanUnavailableError();
}
plan = createModelProviderCredentialTestPlan({
testId: request.testId,
requestId: request.requestId,
projectId: request.projectId,
provider: request.provider,
endpoint,
requestedBy: authority.principal.subject,
fence: authority.decision.fence!,
plannedAtMs: authority.observedAtMs,
expiresAtMs: authority.observedAtMs + testPlanLifetimeMs,
});
return await options.testPlans.createAuthorized({
plan,
audit: {
eventId: plan.testId,
requestId: plan.requestId,
operationId: MODEL_PROVIDER_CREDENTIAL_TEST_PLAN_OPERATION_ID,
projectId: plan.projectId,
subject: authority.principal.subject,
authenticationId: authority.principal.authenticationId,
outcome: 'allowed',
reasons: authority.decision.reasons,
fence: authority.decision.fence,
occurredAtMs: authority.observedAtMs,
},
});
} catch (error) {
return mapTestPlanError(error);
}
},
});
}
@@ -0,0 +1,105 @@
#!/usr/bin/env node
import {
startClusterModelProviderCredentialManagementProcess,
type ClusterModelProviderCredentialManagementProcessRuntime,
} from './modelProviderCredentialManagementProcess';
const USAGE = 'Usage: ql3-provider-credential-manage';
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as { readonly name?: unknown; readonly code?: unknown };
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-model-provider-credential-management',
event: 'management_failed',
name:
typeof candidate?.name === 'string' && candidate.name.length <= 128
? candidate.name
: 'Error',
...(typeof candidate?.code === 'string' && candidate.code.length <= 128
? { code: candidate.code }
: {}),
});
}
function emit(value: Readonly<Record<string, unknown>>): void {
process.stdout.write(`${JSON.stringify(value)}\n`);
}
async function run(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
if (argv.length !== 0) {
process.stderr.write(
`${JSON.stringify({
code: 'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_CLI_USAGE_INVALID',
message: USAGE,
})}\n`,
);
process.exitCode = 64;
return;
}
let runtime: Readonly<ClusterModelProviderCredentialManagementProcessRuntime>;
try {
runtime = await startClusterModelProviderCredentialManagementProcess({
environment: process.env,
onError() {
emit({
schemaVersion: 1,
component: 'qinglong3-model-provider-credential-management',
event: 'management_unavailable',
});
},
});
} catch (error) {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
return;
}
if (runtime.status === 'disabled') {
emit({
schemaVersion: 1,
component: 'qinglong3-model-provider-credential-management',
event: 'management_disabled',
});
return;
}
emit({
schemaVersion: 1,
component: 'qinglong3-model-provider-credential-management',
event: 'management_started',
address: runtime.address,
identityGeneration: runtime.identity.generation,
databaseMigrationCount: runtime.database.migrationIds.length,
});
let stopping: Promise<void> | undefined;
const stop = (): Promise<void> => {
stopping ??= runtime.close().then(() => {
emit({
schemaVersion: 1,
component: 'qinglong3-model-provider-credential-management',
event: 'management_stopped',
});
});
return stopping;
};
for (const signal of ['SIGINT', 'SIGTERM'] as const) {
process.once(signal, () => {
void stop().then(
() => {
process.exitCode = 0;
},
(error) => {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
},
);
});
}
}
void run(process.argv.slice(2));
@@ -0,0 +1,312 @@
import {
ClusterPluginPackageManagementClientRequestError,
executeClusterAuthenticatedManagementClient,
type ClusterAuthenticatedManagementClientResult,
type ClusterPluginPackageManagementClientConnectionOptions,
type ClusterPluginPackageManagementClientPaths,
} from '../management-support/pluginPackageManagementClient';
import {
normalizeClusterModelProviderCredentialManagementCommand,
type ClusterModelProviderCredentialManagementCommand,
} from './modelProviderCredentialManagementTransport';
import {
normalizeModelProviderCredentialTestPlan,
type ModelProviderCredentialTestPlan,
} from '@qinglong/ai/model-provider-credential-test-connection';
const MANAGEMENT_PATH = '/api/v3/provider-credentials/management';
const CONTROL_PATTERN = /[\u0000-\u001f\u007f]/;
const DIGEST_PATTERN = /^[0-9a-f]{64}$/;
const UUID_V4_PATTERN =
/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/;
export type ClusterModelProviderCredentialManagementClientPaths =
ClusterPluginPackageManagementClientPaths;
export type ClusterModelProviderCredentialManagementClientConnectionOptions =
ClusterPluginPackageManagementClientConnectionOptions;
export type ClusterModelProviderCredentialManagementClientTransportResult =
| Readonly<{
schemaVersion: 1;
operation: 'provider-credential.bind' | 'provider-credential.revoke';
status: 'created' | 'existing';
credential: Readonly<{
projectId: string;
provider: string;
generation: number;
action: 'bind' | 'revoke';
activeBindingRevision: string | null;
activeBindingDigest: string | null;
transitionDigest: string;
changedAtMs: number;
}>;
}>
| Readonly<{
schemaVersion: 1;
operation: 'provider-credential.audit.list';
audit: Readonly<{
projectId: string;
records: readonly Readonly<{
eventId: string;
requestId: string;
operation: 'provider-credential.bind' | 'provider-credential.revoke';
actor: Readonly<{ type: 'user'; id: string }>;
fence: Readonly<{
projectVersion: number;
bindingVersion: number;
}>;
occurredAtMs: number;
}>[];
nextCursor: Readonly<{
occurredAtMs: number;
eventId: string;
}> | null;
}>;
}>
| Readonly<{
schemaVersion: 1;
operation: 'provider-credential.test.plan';
status: 'created' | 'existing';
plan: Readonly<ModelProviderCredentialTestPlan>;
}>;
export type ClusterModelProviderCredentialManagementClientResult =
ClusterAuthenticatedManagementClientResult<ClusterModelProviderCredentialManagementClientTransportResult>;
function invalid(): never {
throw new ClusterPluginPackageManagementClientRequestError();
}
function exactRecord(
value: unknown,
keys: readonly string[],
): Record<string, unknown> {
if (!value || typeof value !== 'object' || Array.isArray(value)) invalid();
const record = value as Record<string, unknown>;
const actual = Object.keys(record).sort();
const expected = [...keys].sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
invalid();
}
return record;
}
function identifier(value: unknown): string {
if (
typeof value !== 'string' ||
value.length < 1 ||
value.length > 128 ||
CONTROL_PATTERN.test(value)
) {
invalid();
}
return value;
}
function digest(value: unknown): string {
if (typeof value !== 'string' || !DIGEST_PATTERN.test(value)) invalid();
return value;
}
function uuid(value: unknown): string {
if (typeof value !== 'string' || !UUID_V4_PATTERN.test(value)) invalid();
return value;
}
function positiveInteger(value: unknown): number {
if (!Number.isSafeInteger(value) || (value as number) < 1) invalid();
return value as number;
}
function nonNegativeInteger(value: unknown): number {
if (!Number.isSafeInteger(value) || (value as number) < 0) invalid();
return value as number;
}
export function validateClusterModelProviderCredentialManagementClientResult(
value: unknown,
command: Readonly<ClusterModelProviderCredentialManagementCommand>,
): Readonly<ClusterModelProviderCredentialManagementClientTransportResult> {
if (command.operation === 'provider-credential.test.plan') {
const envelope = exactRecord(value, [
'operation',
'plan',
'schemaVersion',
'status',
]);
if (
envelope.schemaVersion !== 1 ||
envelope.operation !== command.operation ||
(envelope.status !== 'created' && envelope.status !== 'existing')
) {
invalid();
}
let plan: Readonly<ModelProviderCredentialTestPlan>;
try {
plan = normalizeModelProviderCredentialTestPlan(
envelope.plan as ModelProviderCredentialTestPlan,
);
} catch {
return invalid();
}
if (
plan.testId !== command.request.testId ||
plan.requestId !== command.request.requestId ||
plan.projectId !== command.request.projectId ||
plan.provider !== command.request.provider
) {
invalid();
}
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
status: envelope.status,
plan,
});
}
if (command.operation === 'provider-credential.audit.list') {
const envelope = exactRecord(value, [
'schemaVersion',
'operation',
'audit',
]);
if (
envelope.schemaVersion !== 1 ||
envelope.operation !== command.operation
) {
invalid();
}
const audit = exactRecord(envelope.audit, [
'projectId',
'records',
'nextCursor',
]);
if (
identifier(audit.projectId) !== command.request.projectId ||
!Array.isArray(audit.records) ||
audit.records.length > command.request.limit
) {
invalid();
}
const records = audit.records.map((value) => {
const record = exactRecord(value, [
'actor',
'eventId',
'fence',
'occurredAtMs',
'operation',
'requestId',
]);
uuid(record.eventId);
identifier(record.requestId);
if (
record.operation !== 'provider-credential.bind' &&
record.operation !== 'provider-credential.revoke'
) {
invalid();
}
const actor = exactRecord(record.actor, ['id', 'type']);
if (actor.type !== 'user') invalid();
identifier(actor.id);
const fence = exactRecord(record.fence, [
'bindingVersion',
'projectVersion',
]);
positiveInteger(fence.projectVersion);
positiveInteger(fence.bindingVersion);
nonNegativeInteger(record.occurredAtMs);
return record;
});
if (audit.nextCursor !== null) {
const cursor = exactRecord(audit.nextCursor, ['eventId', 'occurredAtMs']);
uuid(cursor.eventId);
nonNegativeInteger(cursor.occurredAtMs);
const last = records.at(-1);
if (
records.length !== command.request.limit ||
!last ||
cursor.eventId !== last.eventId ||
cursor.occurredAtMs !== last.occurredAtMs
) {
invalid();
}
}
return Object.freeze(
envelope as unknown as ClusterModelProviderCredentialManagementClientTransportResult,
);
}
const envelope = exactRecord(value, [
'schemaVersion',
'operation',
'status',
'credential',
]);
if (
envelope.schemaVersion !== 1 ||
envelope.operation !== command.operation ||
!['created', 'existing'].includes(String(envelope.status))
) {
invalid();
}
const credential = exactRecord(envelope.credential, [
'projectId',
'provider',
'generation',
'action',
'activeBindingRevision',
'activeBindingDigest',
'transitionDigest',
'changedAtMs',
]);
const expectedAction =
command.operation === 'provider-credential.bind' ? 'bind' : 'revoke';
if (
identifier(credential.projectId) !== command.request.projectId ||
identifier(credential.provider) !== command.request.provider ||
!Number.isSafeInteger(credential.generation) ||
(credential.generation as number) < 1 ||
credential.action !== expectedAction ||
!Number.isSafeInteger(credential.changedAtMs) ||
(credential.changedAtMs as number) < 0
) {
invalid();
}
digest(credential.transitionDigest);
if (command.operation === 'provider-credential.bind') {
if (
identifier(credential.activeBindingRevision) !== command.request.revision
) {
invalid();
}
digest(credential.activeBindingDigest);
} else if (
credential.activeBindingRevision !== null ||
credential.activeBindingDigest !== null
) {
invalid();
}
return Object.freeze(
envelope as unknown as ClusterModelProviderCredentialManagementClientTransportResult,
);
}
const PROTOCOL = Object.freeze({
managementPath: MANAGEMENT_PATH,
clientCertificate: 'required' as const,
normalizeCommand: normalizeClusterModelProviderCredentialManagementCommand,
validateResult: validateClusterModelProviderCredentialManagementClientResult,
});
export async function executeClusterModelProviderCredentialManagementClient(
paths: ClusterModelProviderCredentialManagementClientPaths,
connectionOptions?: ClusterModelProviderCredentialManagementClientConnectionOptions,
): Promise<Readonly<ClusterModelProviderCredentialManagementClientResult>> {
return executeClusterAuthenticatedManagementClient(
paths,
PROTOCOL,
connectionOptions,
);
}
@@ -0,0 +1,96 @@
#!/usr/bin/env node
import { executeClusterModelProviderCredentialManagementClient } from './modelProviderCredentialManagementClient';
import { ClusterPluginPackageManagementClientRemoteError } from '../management-support/pluginPackageManagementClient';
const USAGE =
'Usage: ql3-provider-credential-client --config=/absolute/client.json --command=/absolute/command.json --assertion=/absolute/assertion.jwt';
function parseArguments(
argv: readonly string[],
): Readonly<{
configFile: string;
commandFile: string;
assertionFile: string;
}> | null {
if (argv.length !== 3) return null;
const values = new Map<string, string>();
for (const argument of argv) {
const match = /^--(config|command|assertion)=(\/.+)$/.exec(argument);
if (!match || values.has(match[1]!)) return null;
values.set(match[1]!, match[2]!);
}
if (
!values.has('config') ||
!values.has('command') ||
!values.has('assertion')
) {
return null;
}
return Object.freeze({
configFile: values.get('config')!,
commandFile: values.get('command')!,
assertionFile: values.get('assertion')!,
});
}
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as { readonly code?: unknown };
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-model-provider-credential-management-client',
event: 'command_failed',
code:
typeof candidate?.code === 'string' && candidate.code.length <= 128
? candidate.code
: 'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_CLIENT_FAILED',
...(error instanceof ClusterPluginPackageManagementClientRemoteError
? {
statusCode: error.statusCode,
responseCode: error.responseCode,
requestId: error.requestId,
...(error.retryAfterSeconds === null
? {}
: { retryAfterSeconds: error.retryAfterSeconds }),
}
: {}),
});
}
async function run(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
const paths = parseArguments(argv);
if (!paths) {
process.stderr.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-model-provider-credential-management-client',
event: 'usage_invalid',
code: 'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_CLIENT_USAGE_INVALID',
})}\n`,
);
process.exitCode = 64;
return;
}
try {
const result =
await executeClusterModelProviderCredentialManagementClient(paths);
process.stdout.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-model-provider-credential-management-client',
event: 'command_completed',
requestId: result.requestId,
result: result.result,
})}\n`,
);
} catch (error) {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
}
}
void run(process.argv.slice(2));
@@ -0,0 +1,24 @@
import {
CLUSTER_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_PATH,
startClusterPluginPackageManagementHttp,
type ClusterPluginPackageManagementHttpApplication,
type StartClusterPluginPackageManagementHttpOptions,
} from '../management-support/pluginPackageManagementHttp';
export type ClusterModelProviderCredentialManagementHttpApplication =
ClusterPluginPackageManagementHttpApplication;
export type StartClusterModelProviderCredentialManagementHttpOptions = Omit<
StartClusterPluginPackageManagementHttpOptions,
'managementPath'
>;
/** Starts the bounded OIDC/mTLS adapter on the provider-credential-only path. */
export function startClusterModelProviderCredentialManagementHttp(
options: StartClusterModelProviderCredentialManagementHttpOptions,
): Promise<Readonly<ClusterModelProviderCredentialManagementHttpApplication>> {
return startClusterPluginPackageManagementHttp({
...options,
managementPath: CLUSTER_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_PATH,
});
}
@@ -0,0 +1,685 @@
import type {
OpenPostgresDatabase,
PostgresDatabaseResource,
} from '@qinglong/runtime-core';
import { ProjectPolicyEngine } from '@qinglong/runtime-core/project-policy';
import {
PostgresProjectPolicyRepository,
createPostgresDatabaseOpener,
isPostgresTlsDnsServername,
loadPostgresCertificateAuthorityFile,
loadPostgresConnectionEnvironment,
type PostgresConnectionOptions,
type PostgresPoolOptions,
} from '@qinglong/cluster-postgres/ai-credential-manager';
import { PostgresModelProviderCredentialRepository } from '@qinglong/ai/postgres-model-provider-credential-storage';
import { PostgresModelProviderCredentialManagementAuditQueryRepository } from '@qinglong/ai/postgres-model-provider-credential-management-audit-query';
import {
PostgresModelProviderCredentialManagementIdentityLedgerRepository,
assertPostgresModelProviderCredentialManagerReady,
type PostgresModelProviderCredentialManagerReadinessReport,
} from '@qinglong/ai/postgres-model-provider-credential-management-identity-ledger';
import {
MAX_MODEL_PROVIDER_CREDENTIAL_TEST_PLAN_LIFETIME_MS,
normalizeModelProviderCredentialTestAllowlist,
type ModelProviderCredentialTestAllowlist,
} from '@qinglong/ai/model-provider-credential-test-connection';
import { PostgresModelProviderCredentialTestPlanRepository } from '@qinglong/ai/postgres-model-provider-credential-test-connection';
import {
absoluteManagementEnvironmentFile,
booleanManagementEnvironmentValue,
boundedManagementEnvironmentValue,
integerManagementEnvironmentValue,
readManagementTlsFile,
} from '../management-support/managementProcessSupport';
import {
createClusterModelProviderCredentialIdentityKeysetFile,
type ClusterPluginPackageIdentityKeysetFile,
type ClusterPluginPackageIdentityKeysetSnapshot,
} from '../management-support/pluginPackageIdentityKeyset';
import { createClusterModelProviderCredentialManagementService } from './modelProviderCredentialManagement';
import {
startClusterModelProviderCredentialManagementHttp,
type ClusterModelProviderCredentialManagementHttpApplication,
type StartClusterModelProviderCredentialManagementHttpOptions,
} from './modelProviderCredentialManagementHttp';
import { createClusterModelProviderCredentialManagementTransport } from './modelProviderCredentialManagementTransport';
import { validateClusterManagementClientTrust } from '../worker-credential/management-server/workerCredentialManagementMutualTls';
const SAFE_HOST = /^[A-Za-z0-9][A-Za-z0-9.:-]{0,254}$/;
const SAFE_APPLICATION_NAME = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,62}$/;
export type ClusterModelProviderCredentialManagementProcessEnvironment =
Readonly<Record<string, string | undefined>>;
export type ClusterModelProviderCredentialManagementProcessConfig =
| Readonly<{ enabled: false }>
| Readonly<{
enabled: true;
profile: 'cluster-admin';
host: string;
port: number;
certificateFile: string;
privateKeyFile: string;
clientCertificateAuthorityFile: string;
clientCertificateRevocationListFile: string;
identityKeysetFile: string;
testConnection: Readonly<{
allowlistFile: string;
planLifetimeMs: number;
quotaWindowMs: number;
quotaLimit: number;
}>;
http: Readonly<{
maxBodyBytes: number;
maxConnections: number;
maxConcurrentRequests: number;
requestTimeoutMs: number;
drainTimeoutMs: number;
rateWindowMs: number;
peerRequestLimit: number;
globalRequestLimit: number;
maxRateLimitPeers: number;
}>;
database: Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}>;
}>;
export type ClusterModelProviderCredentialManagementProcessRuntime =
| Readonly<{
status: 'disabled';
close(): Promise<void>;
}>
| Readonly<{
status: 'active';
address: Readonly<{ host: string; port: number }>;
database: PostgresModelProviderCredentialManagerReadinessReport;
identity: ClusterPluginPackageIdentityKeysetSnapshot;
availabilityStatus(): 'ready' | 'unavailable' | 'stopped';
close(): Promise<void>;
}>;
export interface StartClusterModelProviderCredentialManagementProcessOptions {
readonly environment: ClusterModelProviderCredentialManagementProcessEnvironment;
readonly openDatabase?: OpenPostgresDatabase;
readonly identities?: ClusterPluginPackageIdentityKeysetFile;
readonly assertReady?: (
pool: PostgresDatabaseResource['pool'],
) => Promise<PostgresModelProviderCredentialManagerReadinessReport>;
readonly startHttp?: (
options: StartClusterModelProviderCredentialManagementHttpOptions,
) => Promise<
Readonly<ClusterModelProviderCredentialManagementHttpApplication>
>;
readonly now?: () => number;
readonly onError?: (error: unknown) => void;
}
export class ClusterModelProviderCredentialManagementProcessConfigError extends TypeError {
readonly code =
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_PROCESS_CONFIG_INVALID';
constructor(message: string) {
super(
`Model provider credential management process configuration is invalid: ${message}`,
);
this.name = 'ClusterModelProviderCredentialManagementProcessConfigError';
}
}
function configFailure(
message: string,
): ClusterModelProviderCredentialManagementProcessConfigError {
return new ClusterModelProviderCredentialManagementProcessConfigError(
message,
);
}
function boundedValue(
environment: ClusterModelProviderCredentialManagementProcessEnvironment,
name: string,
maximumLength: number,
): string | undefined {
return boundedManagementEnvironmentValue(
environment,
name,
maximumLength,
configFailure,
);
}
function booleanValue(
environment: ClusterModelProviderCredentialManagementProcessEnvironment,
name: string,
): boolean {
return booleanManagementEnvironmentValue(environment, name, configFailure);
}
function integerValue(
environment: ClusterModelProviderCredentialManagementProcessEnvironment,
name: string,
fallback: number,
minimum: number,
maximum: number,
): number {
return integerManagementEnvironmentValue(
environment,
name,
fallback,
minimum,
maximum,
configFailure,
);
}
function absoluteFile(
environment: ClusterModelProviderCredentialManagementProcessEnvironment,
name: string,
): string {
return absoluteManagementEnvironmentFile(environment, name, configFailure);
}
function loadConnection(
environment: ClusterModelProviderCredentialManagementProcessEnvironment,
): Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}> {
let connection: PostgresConnectionOptions;
try {
connection = loadPostgresConnectionEnvironment(environment, {
connectionString: 'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_URL',
host: 'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_HOST',
port: 'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_PORT',
database: 'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_DATABASE',
user: 'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_USER',
password: 'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_PASSWORD',
});
} catch (error) {
throw configFailure(
error instanceof Error
? error.message
: 'PostgreSQL AI credential manager connection is invalid',
);
}
const mode =
environment.QL3_POSTGRES_AI_CREDENTIAL_MANAGER_TLS_MODE ?? 'verify-full';
if (mode !== 'verify-full' && mode !== 'disable') {
throw configFailure(
'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_TLS_MODE must be verify-full or disable',
);
}
if (
mode === 'disable' &&
!booleanValue(
environment,
'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_ALLOW_INSECURE',
)
) {
throw configFailure(
'disabling AI credential manager PostgreSQL TLS requires QL3_POSTGRES_AI_CREDENTIAL_MANAGER_ALLOW_INSECURE=true',
);
}
const servername = boundedValue(
environment,
'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_TLS_SERVERNAME',
253,
);
if (mode === 'verify-full' && !isPostgresTlsDnsServername(servername)) {
throw configFailure(
'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_TLS_SERVERNAME must be an explicit DNS name',
);
}
const caFile = boundedValue(
environment,
'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_TLS_CA_FILE',
4_096,
);
if (mode === 'disable' && caFile !== undefined) {
throw configFailure(
'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_TLS_CA_FILE cannot be used when TLS is disabled',
);
}
let ca: string | undefined;
if (caFile !== undefined) {
try {
ca = loadPostgresCertificateAuthorityFile(caFile);
} catch {
throw configFailure(
'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_TLS_CA_FILE is invalid',
);
}
}
const applicationName =
boundedValue(
environment,
'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_APPLICATION_NAME',
63,
) ?? 'qinglong3-ai-credential-manager';
if (!SAFE_APPLICATION_NAME.test(applicationName)) {
throw configFailure(
'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_APPLICATION_NAME is invalid',
);
}
return Object.freeze({
connection: Object.freeze({
...connection,
tls:
mode === 'disable'
? Object.freeze({ mode: 'disable' as const })
: Object.freeze({
mode: 'verify-full' as const,
servername: servername!,
...(ca === undefined ? {} : { ca }),
}),
}),
pool: Object.freeze({
applicationName,
maxConnections: integerValue(
environment,
'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_POOL_MAX',
2,
1,
2,
),
idleTimeoutMs: integerValue(
environment,
'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_IDLE_TIMEOUT_MS',
10_000,
1_000,
60_000,
),
connectionTimeoutMs: integerValue(
environment,
'QL3_POSTGRES_AI_CREDENTIAL_MANAGER_CONNECTION_TIMEOUT_MS',
5_000,
100,
60_000,
),
}),
});
}
export function loadClusterModelProviderCredentialManagementProcessConfig(
environment: ClusterModelProviderCredentialManagementProcessEnvironment,
): Readonly<ClusterModelProviderCredentialManagementProcessConfig> {
if (!environment || typeof environment !== 'object') {
throw configFailure('environment is invalid');
}
if (
!booleanValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_ENABLED',
)
) {
return Object.freeze({ enabled: false as const });
}
if (environment.QL3_PROFILE !== 'cluster-admin') {
throw configFailure(
'QL3_PROFILE must be cluster-admin when model provider credential management is enabled',
);
}
const host =
boundedValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_HOST',
255,
) ?? '0.0.0.0';
if (!SAFE_HOST.test(host)) throw configFailure('host is invalid');
const http = Object.freeze({
maxBodyBytes: integerValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_MAX_BODY_BYTES',
32 * 1024,
1_024,
64 * 1024,
),
maxConnections: integerValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_MAX_CONNECTIONS',
32,
1,
128,
),
maxConcurrentRequests: integerValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_MAX_CONCURRENT_REQUESTS',
8,
1,
32,
),
requestTimeoutMs: integerValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_REQUEST_TIMEOUT_MS',
10_000,
1_000,
30_000,
),
drainTimeoutMs: integerValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_DRAIN_TIMEOUT_MS',
5_000,
100,
30_000,
),
rateWindowMs: integerValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_RATE_WINDOW_MS',
60_000,
1_000,
5 * 60_000,
),
peerRequestLimit: integerValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_PEER_REQUEST_LIMIT',
30,
1,
1_000,
),
globalRequestLimit: integerValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_GLOBAL_REQUEST_LIMIT',
120,
1,
10_000,
),
maxRateLimitPeers: integerValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_MAX_RATE_LIMIT_PEERS',
256,
1,
4_096,
),
});
if (http.globalRequestLimit < http.peerRequestLimit) {
throw configFailure(
'global request limit cannot be below the peer request limit',
);
}
return Object.freeze({
enabled: true as const,
profile: 'cluster-admin' as const,
host,
port: integerValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_PORT',
8_446,
1,
65_535,
),
certificateFile: absoluteFile(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_TLS_CERT_FILE',
),
privateKeyFile: absoluteFile(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_TLS_KEY_FILE',
),
clientCertificateAuthorityFile: absoluteFile(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_CLIENT_CA_FILE',
),
clientCertificateRevocationListFile: absoluteFile(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_CLIENT_CRL_FILE',
),
identityKeysetFile: absoluteFile(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_MANAGEMENT_IDENTITY_KEYSET_FILE',
),
testConnection: Object.freeze({
allowlistFile: absoluteFile(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_TEST_ALLOWLIST_FILE',
),
planLifetimeMs: integerValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_TEST_PLAN_LIFETIME_MS',
60_000,
1_000,
MAX_MODEL_PROVIDER_CREDENTIAL_TEST_PLAN_LIFETIME_MS,
),
quotaWindowMs: integerValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_TEST_QUOTA_WINDOW_MS',
60_000,
1_000,
5 * 60_000,
),
quotaLimit: integerValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_TEST_QUOTA_LIMIT',
5,
1,
32,
),
}),
http,
database: loadConnection(environment),
});
}
function readTlsFile(filePath: string, privateMaterial: boolean): Buffer {
return readManagementTlsFile(filePath, privateMaterial, configFailure);
}
function readTestAllowlist(
filePath: string,
): Readonly<ModelProviderCredentialTestAllowlist> {
const bytes = readManagementTlsFile(filePath, false, configFailure);
try {
if (bytes.length > 64 * 1_024) {
throw configFailure(
'model provider credential test allowlist is too large',
);
}
return normalizeModelProviderCredentialTestAllowlist(
JSON.parse(
bytes.toString('utf8'),
) as ModelProviderCredentialTestAllowlist,
);
} catch (error) {
if (
error instanceof
ClusterModelProviderCredentialManagementProcessConfigError
) {
throw error;
}
throw configFailure('model provider credential test allowlist is invalid');
}
}
export async function startClusterModelProviderCredentialManagementProcess(
options: StartClusterModelProviderCredentialManagementProcessOptions,
): Promise<Readonly<ClusterModelProviderCredentialManagementProcessRuntime>> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
![
'environment',
'openDatabase',
'identities',
'assertReady',
'startHttp',
'now',
'onError',
].includes(key),
) ||
!options.environment ||
typeof options.environment !== 'object' ||
(options.openDatabase !== undefined &&
typeof options.openDatabase !== 'function') ||
(options.identities !== undefined &&
(typeof options.identities.reload !== 'function' ||
typeof options.identities.bind !== 'function')) ||
(options.assertReady !== undefined &&
typeof options.assertReady !== 'function') ||
(options.startHttp !== undefined &&
typeof options.startHttp !== 'function') ||
(options.now !== undefined && typeof options.now !== 'function') ||
(options.onError !== undefined && typeof options.onError !== 'function')
) {
throw configFailure('options are invalid');
}
const config = loadClusterModelProviderCredentialManagementProcessConfig(
options.environment,
);
if (!config.enabled) {
return Object.freeze({
status: 'disabled' as const,
close: () => Promise.resolve(),
});
}
const now = options.now ?? Date.now;
let http:
| Readonly<ClusterModelProviderCredentialManagementHttpApplication>
| undefined;
let database: PostgresDatabaseResource | undefined;
let unavailableError: unknown;
let closePromise: Promise<void> | undefined;
const report = (error: unknown): void => {
try {
options.onError?.(error);
} catch {
// Diagnostics do not own availability or cleanup.
}
};
const openDatabase =
options.openDatabase ??
createPostgresDatabaseOpener({
role: 'ai-credential-manager',
connection: config.database.connection,
pool: config.database.pool,
onPoolError(error) {
const firstAvailabilityError = unavailableError === undefined;
unavailableError ??= error;
http?.withdraw(error);
if (firstAvailabilityError) report(error);
},
});
try {
database = await openDatabase();
const evidence = await (
options.assertReady ?? assertPostgresModelProviderCredentialManagerReady
)(database.pool);
if (unavailableError !== undefined) throw unavailableError;
const identities =
options.identities ??
createClusterModelProviderCredentialIdentityKeysetFile({
filePath: config.identityKeysetFile,
now,
ledger:
new PostgresModelProviderCredentialManagementIdentityLedgerRepository(
database.pool,
),
});
const identity = await identities.reload();
const testAllowlist = readTestAllowlist(
config.testConnection.allowlistFile,
);
const service = createClusterModelProviderCredentialManagementService({
policy: new ProjectPolicyEngine(
new PostgresProjectPolicyRepository(database.pool),
),
credentials: new PostgresModelProviderCredentialRepository(database.pool),
audit: new PostgresModelProviderCredentialManagementAuditQueryRepository(
database.pool,
),
testPlans: new PostgresModelProviderCredentialTestPlanRepository(
database.pool,
{
quotaWindowMs: config.testConnection.quotaWindowMs,
quotaLimit: config.testConnection.quotaLimit,
},
),
testAllowlist,
testPlanLifetimeMs: config.testConnection.planLifetimeMs,
now,
});
const transport = createClusterModelProviderCredentialManagementTransport({
service,
now,
});
const privateKey = readTlsFile(config.privateKeyFile, true);
try {
const certificate = readTlsFile(config.certificateFile, false);
const clientCertificateAuthority = readTlsFile(
config.clientCertificateAuthorityFile,
false,
);
const clientCertificateRevocationList = readTlsFile(
config.clientCertificateRevocationListFile,
false,
);
validateClusterManagementClientTrust(
clientCertificateAuthority,
clientCertificateRevocationList,
now(),
configFailure,
);
http = await (
options.startHttp ?? startClusterModelProviderCredentialManagementHttp
)({
host: config.host,
port: config.port,
tls: {
privateKey,
certificate,
clientCertificateAuthority,
clientCertificateRevocationList,
},
transport,
identities,
limits: config.http,
now,
onError: report,
});
} finally {
privateKey.fill(0);
}
if (unavailableError !== undefined) http.withdraw(unavailableError);
return Object.freeze({
status: 'active' as const,
address: http.address,
database: evidence,
identity,
availabilityStatus: () => http?.availabilityStatus() ?? 'stopped',
close(): Promise<void> {
if (closePromise) return closePromise;
closePromise = (async () => {
let primaryError: unknown;
try {
await http?.close();
} catch (error) {
primaryError = error;
}
try {
await database?.close();
} catch (error) {
primaryError ??= error;
}
if (primaryError) throw primaryError;
})();
return closePromise;
},
});
} catch (error) {
try {
await http?.close();
} catch {
// Preserve startup failure.
}
try {
await database?.close();
} catch {
// Preserve startup failure.
}
throw error;
}
}
@@ -0,0 +1,357 @@
import {
normalizeSecurityPrincipal,
type SecurityPrincipal,
} from '@qinglong/runtime-core/security';
import type { ModelProviderCredentialTestPlan } from '@qinglong/ai/model-provider-credential-test-connection';
import type {
BindModelProviderCredentialRequest,
ClusterModelProviderCredentialManagementService,
ListModelProviderCredentialManagementAuditRequest,
PlanModelProviderCredentialTestRequest,
RevokeModelProviderCredentialRequest,
} from './modelProviderCredentialManagement';
const STRONG_CLUSTER_ASSURANCES = new Set(['multi_factor', 'hardware']);
const MAX_PRINCIPAL_AGE_MS = 5 * 60 * 1_000;
interface BindTransportRequest
extends Omit<BindModelProviderCredentialRequest, 'principal'> {}
interface RevokeTransportRequest
extends Omit<RevokeModelProviderCredentialRequest, 'principal'> {}
interface AuditTransportRequest
extends Omit<
ListModelProviderCredentialManagementAuditRequest,
'principal'
> {}
interface TestPlanTransportRequest
extends Omit<PlanModelProviderCredentialTestRequest, 'principal'> {}
export type ClusterModelProviderCredentialManagementCommand =
| Readonly<{
schemaVersion: 1;
operation: 'provider-credential.bind';
request: BindTransportRequest;
}>
| Readonly<{
schemaVersion: 1;
operation: 'provider-credential.revoke';
request: RevokeTransportRequest;
}>
| Readonly<{
schemaVersion: 1;
operation: 'provider-credential.audit.list';
request: AuditTransportRequest;
}>
| Readonly<{
schemaVersion: 1;
operation: 'provider-credential.test.plan';
request: TestPlanTransportRequest;
}>;
export interface ClusterModelProviderCredentialManagementAuthentication {
authenticate(): Promise<Readonly<SecurityPrincipal> | null>;
}
export interface ClusterModelProviderCredentialManagementTransport {
execute(
command: unknown,
authentication: ClusterModelProviderCredentialManagementAuthentication,
): Promise<Readonly<ClusterModelProviderCredentialManagementTransportResult>>;
}
export type ClusterModelProviderCredentialManagementTransportResult =
| Readonly<{
schemaVersion: 1;
operation: 'provider-credential.bind' | 'provider-credential.revoke';
status: 'created' | 'existing';
credential: Readonly<{
projectId: string;
provider: string;
generation: number;
action: 'bind' | 'revoke';
activeBindingRevision: string | null;
activeBindingDigest: string | null;
transitionDigest: string;
changedAtMs: number;
}>;
}>
| Readonly<{
schemaVersion: 1;
operation: 'provider-credential.audit.list';
audit: Readonly<{
projectId: string;
records: readonly Readonly<{
eventId: string;
requestId: string;
operation: 'provider-credential.bind' | 'provider-credential.revoke';
actor: Readonly<{ type: 'user'; id: string }>;
fence: Readonly<{
projectVersion: number;
bindingVersion: number;
}>;
occurredAtMs: number;
}>[];
nextCursor: Readonly<{
occurredAtMs: number;
eventId: string;
}> | null;
}>;
}>
| Readonly<{
schemaVersion: 1;
operation: 'provider-credential.test.plan';
status: 'created' | 'existing';
plan: Readonly<ModelProviderCredentialTestPlan>;
}>;
export class ClusterModelProviderCredentialManagementTransportConfigurationError extends TypeError {
readonly code =
'CLUSTER_MODEL_PROVIDER_CREDENTIAL_TRANSPORT_CONFIGURATION_INVALID';
constructor() {
super(
'Cluster model provider credential transport configuration is invalid',
);
this.name =
'ClusterModelProviderCredentialManagementTransportConfigurationError';
}
}
export class ClusterModelProviderCredentialManagementTransportRequestError extends TypeError {
readonly code = 'CLUSTER_MODEL_PROVIDER_CREDENTIAL_TRANSPORT_REQUEST_INVALID';
constructor() {
super('Cluster model provider credential transport request is invalid');
this.name = 'ClusterModelProviderCredentialManagementTransportRequestError';
}
}
export class ClusterModelProviderCredentialManagementTransportAuthenticationError extends Error {
readonly code =
'CLUSTER_MODEL_PROVIDER_CREDENTIAL_TRANSPORT_AUTHENTICATION_REQUIRED';
constructor() {
super(
'Cluster model provider credential transport requires a recent strong User',
);
this.name =
'ClusterModelProviderCredentialManagementTransportAuthenticationError';
}
}
export class ClusterModelProviderCredentialManagementTransportUnavailableError extends Error {
readonly code = 'CLUSTER_MODEL_PROVIDER_CREDENTIAL_TRANSPORT_UNAVAILABLE';
constructor() {
super('Cluster model provider credential transport is unavailable');
this.name =
'ClusterModelProviderCredentialManagementTransportUnavailableError';
}
}
function exactKeys(value: object, expected: readonly string[]): boolean {
const actual = Object.keys(value).sort();
const canonical = [...expected].sort();
return (
actual.length === canonical.length &&
actual.every((key, index) => key === canonical[index])
);
}
function normalizeCommand(
value: unknown,
): Readonly<ClusterModelProviderCredentialManagementCommand> {
if (
!value ||
typeof value !== 'object' ||
Array.isArray(value) ||
!exactKeys(value, ['operation', 'request', 'schemaVersion']) ||
(value as { schemaVersion?: unknown }).schemaVersion !== 1 ||
((value as { operation?: unknown }).operation !==
'provider-credential.bind' &&
(value as { operation?: unknown }).operation !==
'provider-credential.revoke' &&
(value as { operation?: unknown }).operation !==
'provider-credential.audit.list' &&
(value as { operation?: unknown }).operation !==
'provider-credential.test.plan') ||
!(value as { request?: unknown }).request ||
typeof (value as { request: unknown }).request !== 'object' ||
Array.isArray((value as { request: unknown }).request)
) {
throw new ClusterModelProviderCredentialManagementTransportRequestError();
}
const command = value as ClusterModelProviderCredentialManagementCommand;
const common = [
'expectedGeneration',
'mutationId',
'projectId',
'provider',
'requestId',
];
if (command.operation === 'provider-credential.audit.list') {
const auditKeys = ['limit', 'projectId', 'queryId', 'requestId'];
if ('before' in command.request) auditKeys.push('before');
if (!exactKeys(command.request, auditKeys)) {
throw new ClusterModelProviderCredentialManagementTransportRequestError();
}
return command;
}
if (command.operation === 'provider-credential.test.plan') {
if (
!exactKeys(command.request, [
'projectId',
'provider',
'requestId',
'testId',
])
) {
throw new ClusterModelProviderCredentialManagementTransportRequestError();
}
return command;
}
if (
!exactKeys(
command.request,
command.operation === 'provider-credential.bind'
? [...common, 'revision', 'secretRef']
: common,
)
) {
throw new ClusterModelProviderCredentialManagementTransportRequestError();
}
return command;
}
export function normalizeClusterModelProviderCredentialManagementCommand(
value: unknown,
): Readonly<ClusterModelProviderCredentialManagementCommand> {
return normalizeCommand(value);
}
function summary(
transition: Readonly<{
projectId: string;
provider: string;
generation: number;
action: 'bind' | 'revoke';
activeBindingRevision: string | null;
activeBindingDigest: string | null;
transitionDigest: string;
changedAtMs: number;
}>,
) {
return Object.freeze({
projectId: transition.projectId,
provider: transition.provider,
generation: transition.generation,
action: transition.action,
activeBindingRevision: transition.activeBindingRevision,
activeBindingDigest: transition.activeBindingDigest,
transitionDigest: transition.transitionDigest,
changedAtMs: transition.changedAtMs,
});
}
export function createClusterModelProviderCredentialManagementTransport(
options: Readonly<{
service: ClusterModelProviderCredentialManagementService;
now?: () => number;
}>,
): Readonly<ClusterModelProviderCredentialManagementTransport> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some((key) => key !== 'service' && key !== 'now') ||
!options.service ||
typeof options.service.bind !== 'function' ||
typeof options.service.revoke !== 'function' ||
typeof options.service.listAudit !== 'function' ||
typeof options.service.planTestConnection !== 'function' ||
(options.now !== undefined && typeof options.now !== 'function')
) {
throw new ClusterModelProviderCredentialManagementTransportConfigurationError();
}
const now = options.now ?? Date.now;
return Object.freeze({
async execute(
commandValue: unknown,
authentication: ClusterModelProviderCredentialManagementAuthentication,
) {
const command = normalizeCommand(commandValue);
if (
!authentication ||
typeof authentication !== 'object' ||
Array.isArray(authentication) ||
!exactKeys(authentication, ['authenticate']) ||
typeof authentication.authenticate !== 'function'
) {
throw new ClusterModelProviderCredentialManagementTransportConfigurationError();
}
const observedAtMs = now();
if (!Number.isSafeInteger(observedAtMs) || observedAtMs < 0) {
throw new ClusterModelProviderCredentialManagementTransportUnavailableError();
}
let candidate: Readonly<SecurityPrincipal> | null;
try {
candidate = await authentication.authenticate();
} catch {
throw new ClusterModelProviderCredentialManagementTransportUnavailableError();
}
let principal: Readonly<SecurityPrincipal>;
try {
principal = normalizeSecurityPrincipal(
candidate as SecurityPrincipal,
observedAtMs,
);
} catch {
throw new ClusterModelProviderCredentialManagementTransportAuthenticationError();
}
if (
principal.subject.type !== 'user' ||
!STRONG_CLUSTER_ASSURANCES.has(principal.assurance) ||
observedAtMs - principal.authenticatedAtMs > MAX_PRINCIPAL_AGE_MS
) {
throw new ClusterModelProviderCredentialManagementTransportAuthenticationError();
}
if (command.operation === 'provider-credential.audit.list') {
const audit = await options.service.listAudit({
...command.request,
principal,
});
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
audit,
});
}
if (command.operation === 'provider-credential.test.plan') {
const result = await options.service.planTestConnection({
...command.request,
principal,
});
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
status: result.status,
plan: result.plan,
});
}
const result =
command.operation === 'provider-credential.bind'
? await options.service.bind({ ...command.request, principal })
: await options.service.revoke({ ...command.request, principal });
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
status: result.status,
credential: summary(result.transition),
});
},
});
}
@@ -0,0 +1,255 @@
import { performance } from 'node:perf_hooks';
/** One-shot model-provider credential connectivity executor. */
import {
createModelProviderCredentialTestResult,
type ModelProviderCredentialTestAllowlist,
type ModelProviderCredentialTestExecution,
type ModelProviderCredentialTestPlan,
type ModelProviderCredentialTestResult,
} from '@qinglong/ai/model-provider-credential-test-connection';
import {
ModelProviderCredentialTestExecutionUnavailableError,
type BeginModelProviderCredentialTestExecutionInput,
type ModelProviderCredentialTestExecutionRepository,
} from '@qinglong/ai/postgres-model-provider-credential-test-connection';
import { OpenAiCompatibleProvider } from '@qinglong/ai/openai-compatible';
import {
BoundModelProviderCredentialProvider,
type ModelProviderCredentialAuditSink,
type ModelProviderCredentialBindingSource,
type ModelProviderSecretMaterialProvider,
} from '@qinglong/ai/provider-credential';
export interface ExecuteModelProviderCredentialTestInput
extends BeginModelProviderCredentialTestExecutionInput {}
interface ModelProviderCredentialTestExecutorEvidence {
readonly plan: Readonly<ModelProviderCredentialTestPlan>;
readonly execution: Readonly<ModelProviderCredentialTestExecution>;
}
export type ExecuteModelProviderCredentialTestResult =
| Readonly<
ModelProviderCredentialTestExecutorEvidence & {
status: 'completed' | 'existing';
result: Readonly<ModelProviderCredentialTestResult>;
}
>
| Readonly<
ModelProviderCredentialTestExecutorEvidence & {
status: 'outcome_unknown';
result: null;
}
>;
export interface ModelProviderCredentialTestExecutor {
execute(
input: Readonly<ExecuteModelProviderCredentialTestInput>,
): Promise<Readonly<ExecuteModelProviderCredentialTestResult>>;
}
export interface ModelProviderCredentialTestExecutorOptions {
readonly repository: ModelProviderCredentialTestExecutionRepository;
readonly credentials: ModelProviderCredentialBindingSource &
ModelProviderCredentialAuditSink;
readonly secrets: ModelProviderSecretMaterialProvider;
readonly fetch?: typeof globalThis.fetch;
readonly now?: () => number;
readonly monotonicNow?: () => number;
readonly transportReady?: (
baseUrl: string,
signal: AbortSignal,
) => Promise<void>;
}
export class ModelProviderCredentialTestExecutorConfigurationError extends TypeError {
readonly code =
'MODEL_PROVIDER_CREDENTIAL_TEST_EXECUTOR_CONFIGURATION_INVALID';
constructor() {
super('Model provider credential test executor configuration is invalid');
this.name = 'ModelProviderCredentialTestExecutorConfigurationError';
}
}
export class ModelProviderCredentialTestExecutorUnavailableError extends Error {
readonly code = 'MODEL_PROVIDER_CREDENTIAL_TEST_EXECUTOR_UNAVAILABLE';
constructor(options?: ErrorOptions) {
super('Model provider credential test executor is unavailable', options);
this.name = 'ModelProviderCredentialTestExecutorUnavailableError';
}
}
function exact(value: unknown, keys: readonly string[]): boolean {
if (!value || typeof value !== 'object' || Array.isArray(value)) return false;
const actual = Object.keys(value).sort();
const expected = [...keys].sort();
return (
actual.length === expected.length &&
actual.every((key, index) => key === expected[index])
);
}
function currentTime(now: () => number): number {
const value = now();
if (!Number.isSafeInteger(value) || value < 0) {
throw new ModelProviderCredentialTestExecutorUnavailableError();
}
return value;
}
function elapsedMs(startedAt: number, monotonicNow: () => number): number {
const value = monotonicNow() - startedAt;
if (!Number.isFinite(value) || value < 0) {
throw new ModelProviderCredentialTestExecutorUnavailableError();
}
return Math.floor(value);
}
async function completeExactly(
repository: ModelProviderCredentialTestExecutionRepository,
result: Readonly<ModelProviderCredentialTestResult>,
): Promise<void> {
try {
await repository.complete(result);
} catch (error) {
if (
!(error instanceof ModelProviderCredentialTestExecutionUnavailableError)
) {
throw error;
}
await repository.complete(result);
}
}
export function createModelProviderCredentialTestExecutor(
options: ModelProviderCredentialTestExecutorOptions,
): Readonly<ModelProviderCredentialTestExecutor> {
const expectedKeys = ['credentials', 'repository', 'secrets'];
if (options?.fetch !== undefined) expectedKeys.push('fetch');
if (options?.monotonicNow !== undefined) expectedKeys.push('monotonicNow');
if (options?.now !== undefined) expectedKeys.push('now');
if (options?.transportReady !== undefined)
expectedKeys.push('transportReady');
if (
!exact(options, expectedKeys) ||
typeof options.repository?.beginExecution !== 'function' ||
typeof options.repository?.complete !== 'function' ||
typeof options.credentials?.resolveModelProviderCredentialBinding !==
'function' ||
typeof options.credentials?.record !== 'function' ||
typeof options.secrets?.resolveProjectSecretMaterial !== 'function' ||
(options.fetch !== undefined && typeof options.fetch !== 'function') ||
(options.now !== undefined && typeof options.now !== 'function') ||
(options.monotonicNow !== undefined &&
typeof options.monotonicNow !== 'function') ||
(options.transportReady !== undefined &&
typeof options.transportReady !== 'function')
) {
throw new ModelProviderCredentialTestExecutorConfigurationError();
}
const now = options.now ?? Date.now;
const monotonicNow = options.monotonicNow ?? (() => performance.now());
const credentials = new BoundModelProviderCredentialProvider({
bindings: options.credentials,
secrets: options.secrets,
audit: options.credentials,
now,
});
return Object.freeze({
async execute(input: Readonly<ExecuteModelProviderCredentialTestInput>) {
if (!exact(input, ['allowlist', 'executionId', 'testId'])) {
throw new ModelProviderCredentialTestExecutorConfigurationError();
}
const begun = await options.repository.beginExecution(input);
if (begun.status === 'existing') {
return begun.result === null
? Object.freeze({
status: 'outcome_unknown' as const,
plan: begun.plan,
execution: begun.execution,
result: null,
})
: Object.freeze({
status: 'existing' as const,
plan: begun.plan,
execution: begun.execution,
result: begun.result,
});
}
const startedAt = monotonicNow();
if (!Number.isFinite(startedAt) || startedAt < 0) {
throw new ModelProviderCredentialTestExecutorUnavailableError();
}
const controller = new AbortController();
const timeout = setTimeout(
() => controller.abort(),
begun.plan.endpoint.deadlineMs,
);
timeout.unref?.();
let outcome: 'reachable' | 'unreachable' = 'unreachable';
let modelCount: number | null = null;
try {
await options.transportReady?.(
begun.plan.endpoint.baseUrl,
controller.signal,
);
const provider = new OpenAiCompatibleProvider({
type: begun.plan.provider,
baseUrl: begun.plan.endpoint.baseUrl,
credentials,
maxResponseBytes: begun.plan.endpoint.maxResponseBytes,
...(options.fetch === undefined ? {} : { fetch: options.fetch }),
});
const models = await provider.listModels({
projectId: begun.plan.projectId,
requestId: begun.execution.executionId,
signal: controller.signal,
});
if (
!controller.signal.aborted &&
models.length <= begun.plan.endpoint.maxModels &&
elapsedMs(startedAt, monotonicNow) <= begun.plan.endpoint.deadlineMs
) {
outcome = 'reachable';
modelCount = models.length;
}
} catch {
outcome = 'unreachable';
modelCount = null;
} finally {
clearTimeout(timeout);
}
const durationMs = Math.min(
begun.plan.endpoint.deadlineMs,
elapsedMs(startedAt, monotonicNow),
);
const result = createModelProviderCredentialTestResult({
executionId: begun.execution.executionId,
testId: begun.plan.testId,
planDigest: begun.plan.planDigest,
outcome,
modelCount,
durationMs,
completedAtMs: currentTime(now),
});
try {
await completeExactly(options.repository, result);
} catch (error) {
throw new ModelProviderCredentialTestExecutorUnavailableError({
cause: error instanceof Error ? error : undefined,
});
}
return Object.freeze({
status: 'completed' as const,
plan: begun.plan,
execution: begun.execution,
result,
});
},
});
}
@@ -0,0 +1,110 @@
#!/usr/bin/env node
import { writeFileSync } from 'node:fs';
import { runModelProviderCredentialTestExecutorProcess } from './modelProviderCredentialTestExecutorProcess';
const USAGE = 'Usage: ql3-provider-credential-test-execute';
const TERMINATION_MESSAGE_FILE = '/dev/termination-log';
function writeFact(
value: Readonly<Record<string, unknown>>,
stream: NodeJS.WriteStream,
): void {
const serialized = JSON.stringify(value);
stream.write(`${serialized}\n`);
if (
process.env.QL3_MODEL_PROVIDER_CREDENTIAL_TEST_WRITE_TERMINATION_MESSAGE ===
'true'
) {
writeFileSync(TERMINATION_MESSAGE_FILE, serialized, {
encoding: 'utf8',
mode: 0o600,
});
}
}
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as {
readonly name?: unknown;
readonly code?: unknown;
};
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-model-provider-credential-test-executor',
event: 'execution_failed',
name:
typeof candidate?.name === 'string' && candidate.name.length <= 128
? candidate.name
: 'Error',
...(typeof candidate?.code === 'string' && candidate.code.length <= 128
? { code: candidate.code }
: {}),
});
}
async function run(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
if (argv.length !== 0) {
process.stderr.write(
`${JSON.stringify({
code: 'QL3_MODEL_PROVIDER_CREDENTIAL_TEST_EXECUTOR_CLI_USAGE_INVALID',
message: USAGE,
})}\n`,
);
process.exitCode = 64;
return;
}
try {
const result = await runModelProviderCredentialTestExecutorProcess({
environment: process.env,
});
writeFact(
result.status === 'disabled'
? {
schemaVersion: 1,
component: 'qinglong3-model-provider-credential-test-executor',
event: 'execution_disabled',
}
: {
schemaVersion: 1,
component: 'qinglong3-model-provider-credential-test-executor',
event:
result.test.status === 'outcome_unknown'
? 'execution_outcome_unknown'
: 'execution_completed',
testId: result.test.plan.testId,
executionId: result.test.execution.executionId,
status: result.test.status,
...(result.test.result === null
? {}
: {
outcome: result.test.result.outcome,
modelCount: result.test.result.modelCount,
durationMs: result.test.result.durationMs,
...(result.transportFailureCode === undefined
? {}
: {
transportFailureCode: result.transportFailureCode,
transportRequestDigest: result.transportRequestDigest,
...(result.transportAddressSha256 === undefined
? {}
: {
transportAddressSha256:
result.transportAddressSha256,
transportPort: result.transportPort,
}),
}),
}),
},
process.stdout,
);
} catch (error) {
writeFact(failureFact(error), process.stderr);
process.exitCode = 1;
}
}
void run(process.argv.slice(2));
@@ -0,0 +1,694 @@
import { Buffer } from 'node:buffer';
import { createHash } from 'node:crypto';
import { createConnection } from 'node:net';
import { normalize, parse } from 'node:path';
import { setTimeout as delay } from 'node:timers/promises';
import {
normalizeModelProviderCredentialTestAllowlist,
type ModelProviderCredentialTestAllowlist,
} from '@qinglong/ai/model-provider-credential-test-connection';
import {
PostgresModelProviderCredentialTestExecutionRepository,
assertPostgresModelProviderCredentialTesterReady,
type PostgresModelProviderCredentialTesterReadinessReport,
} from '@qinglong/ai/postgres-model-provider-credential-test-connection';
import { PostgresModelProviderCredentialReader } from '@qinglong/ai/postgres-model-provider-credential-storage';
import {
createProjectedModelProviderSecretMaterialProvider,
type ProjectedModelProviderSecretMaterialProvider,
} from '@qinglong/ai/projected-model-provider-secret-material';
import type { OpenPostgresDatabase } from '@qinglong/runtime-core';
import {
createPostgresDatabaseOpener,
isPostgresTlsDnsServername,
loadPostgresCertificateAuthorityFile,
loadPostgresConnectionEnvironment,
type PostgresConnectionOptions,
type PostgresPoolOptions,
} from '@qinglong/cluster-postgres/ai-credential-tester';
import {
absoluteManagementEnvironmentFile,
booleanManagementEnvironmentValue,
boundedManagementEnvironmentValue,
integerManagementEnvironmentValue,
readManagementTlsFile,
} from '../management-support/managementProcessSupport';
import {
createModelProviderCredentialTestExecutor,
type ExecuteModelProviderCredentialTestResult,
type ModelProviderCredentialTestExecutor,
} from './modelProviderCredentialTestExecutor';
const UUID_V4_PATTERN =
/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/;
const SAFE_APPLICATION_NAME = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,62}$/;
const SAFE_TRANSPORT_FAILURE_CODE = /^[A-Z][A-Z0-9_]{0,63}$/;
const MAX_COMMAND_BYTES = 4 * 1_024;
const MAX_ALLOWLIST_BYTES = 64 * 1_024;
export interface ModelProviderCredentialTestExecutorCommand {
readonly schemaVersion: 1;
readonly executionId: string;
readonly testId: string;
}
export type ModelProviderCredentialTestExecutorProcessEnvironment = Readonly<
Record<string, string | undefined>
>;
export type ModelProviderCredentialTestExecutorProcessConfig =
| Readonly<{ enabled: false }>
| Readonly<{
enabled: true;
profile: 'cluster-admin';
commandFile: string;
allowlistFile: string;
secretRootDirectory: string;
networkPolicyDenyCanary?: Readonly<{
host: string;
port: number;
}>;
database: Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}>;
}>;
export type ModelProviderCredentialTestExecutorProcessResult =
| Readonly<{ status: 'disabled' }>
| Readonly<{
status: 'completed';
database: PostgresModelProviderCredentialTesterReadinessReport;
test: Readonly<ExecuteModelProviderCredentialTestResult>;
readonly transportFailureCode?: string;
readonly transportRequestDigest?: string;
readonly transportAddressSha256?: string;
readonly transportPort?: number;
}>;
export interface RunModelProviderCredentialTestExecutorProcessOptions {
readonly environment: ModelProviderCredentialTestExecutorProcessEnvironment;
readonly command?: Readonly<ModelProviderCredentialTestExecutorCommand>;
readonly allowlist?: Readonly<ModelProviderCredentialTestAllowlist>;
readonly openDatabase?: OpenPostgresDatabase;
readonly assertReady?: typeof assertPostgresModelProviderCredentialTesterReady;
readonly secrets?: Readonly<ProjectedModelProviderSecretMaterialProvider>;
readonly executor?: Readonly<ModelProviderCredentialTestExecutor>;
readonly fetch?: typeof globalThis.fetch;
readonly now?: () => number;
readonly monotonicNow?: () => number;
readonly transportReady?: (
baseUrl: string,
signal: AbortSignal,
) => Promise<void>;
}
export class ModelProviderCredentialTestExecutorProcessConfigError extends TypeError {
readonly code = 'QL3_MODEL_PROVIDER_CREDENTIAL_TEST_EXECUTOR_CONFIG_INVALID';
constructor(message: string) {
super(
`Model provider credential test executor config is invalid: ${message}`,
);
this.name = 'ModelProviderCredentialTestExecutorProcessConfigError';
}
}
function configFailure(
message: string,
): ModelProviderCredentialTestExecutorProcessConfigError {
return new ModelProviderCredentialTestExecutorProcessConfigError(message);
}
function transportFailureCode(error: unknown): string {
let candidate: unknown = error;
for (let depth = 0; depth < 4; depth += 1) {
if (!candidate || typeof candidate !== 'object') break;
const value = candidate as {
readonly cause?: unknown;
readonly code?: unknown;
readonly message?: unknown;
};
if (
typeof value.code === 'string' &&
SAFE_TRANSPORT_FAILURE_CODE.test(value.code)
) {
return value.code;
}
const message =
typeof value.message === 'string' ? value.message.toLowerCase() : '';
if (
message.includes('certificate') ||
message.includes('self signed') ||
message.includes('unable to verify')
) {
return 'TLS_VALIDATION_FAILED';
}
if (message.includes('getaddrinfo')) return 'DNS_LOOKUP_FAILED';
if (message.includes('bad port')) return 'PORT_REJECTED';
candidate = value.cause;
}
return 'TRANSPORT_FAILED';
}
function transportRequestDigest(
input: Parameters<typeof globalThis.fetch>[0],
init?: Parameters<typeof globalThis.fetch>[1],
): string {
const request = input instanceof Request ? input : undefined;
const url = request?.url ?? String(input);
const method = init?.method ?? request?.method ?? 'GET';
return `sha256:${createHash('sha256')
.update(method.toUpperCase(), 'utf8')
.update('\0', 'utf8')
.update(url, 'utf8')
.digest('hex')}`;
}
function transportFailureEndpoint(
error: unknown,
): Readonly<{ addressSha256: string; port: number }> | undefined {
let candidate: unknown = error;
for (let depth = 0; depth < 4; depth += 1) {
if (!candidate || typeof candidate !== 'object') break;
const value = candidate as {
readonly address?: unknown;
readonly cause?: unknown;
readonly port?: unknown;
};
if (
typeof value.address === 'string' &&
value.address.length >= 1 &&
value.address.length <= 128 &&
Number.isInteger(value.port) &&
(value.port as number) >= 1 &&
(value.port as number) <= 65_535
) {
return Object.freeze({
addressSha256: `sha256:${createHash('sha256')
.update(value.address, 'utf8')
.digest('hex')}`,
port: value.port as number,
});
}
candidate = value.cause;
}
return undefined;
}
async function connectOnce(
host: string,
port: number,
signal: AbortSignal,
timeoutMs: number,
): Promise<void> {
await new Promise<void>((resolve, reject) => {
let settled = false;
const socket = createConnection({ host, port });
const finish = (error?: Error): void => {
if (settled) return;
settled = true;
signal.removeEventListener('abort', onAbort);
socket.destroy();
if (error) reject(error);
else resolve();
};
const onAbort = (): void =>
finish(
signal.reason instanceof Error
? signal.reason
: new Error('transport readiness aborted'),
);
signal.addEventListener('abort', onAbort, { once: true });
socket.setTimeout(timeoutMs);
socket.once('connect', () => finish());
socket.once('timeout', () => finish(new Error('transport timeout')));
socket.once('error', (error) => finish(error));
});
}
async function waitForTransportReady(
baseUrl: string,
signal: AbortSignal,
denyCanary?: Readonly<{ host: string; port: number }>,
): Promise<void> {
const endpoint = new URL(baseUrl);
const hostname = endpoint.hostname.replace(/^\[|\]$/g, '');
const port = endpoint.port
? Number(endpoint.port)
: endpoint.protocol === 'https:'
? 443
: 80;
while (true) {
if (signal.aborted) throw signal.reason;
try {
await connectOnce(hostname, port, signal, 500);
if (denyCanary !== undefined) {
let denied = false;
try {
await connectOnce(denyCanary.host, denyCanary.port, signal, 150);
} catch (error) {
if (signal.aborted) throw signal.reason ?? error;
denied = true;
}
if (!denied) {
await delay(50, undefined, { signal });
continue;
}
}
return;
} catch (error) {
if (signal.aborted) throw signal.reason ?? error;
await delay(50, undefined, { signal });
}
}
}
function boundedValue(
environment: ModelProviderCredentialTestExecutorProcessEnvironment,
name: string,
maximumLength: number,
): string | undefined {
return boundedManagementEnvironmentValue(
environment,
name,
maximumLength,
configFailure,
);
}
function booleanValue(
environment: ModelProviderCredentialTestExecutorProcessEnvironment,
name: string,
): boolean {
return booleanManagementEnvironmentValue(environment, name, configFailure);
}
function integerValue(
environment: ModelProviderCredentialTestExecutorProcessEnvironment,
name: string,
fallback: number,
minimum: number,
maximum: number,
): number {
return integerManagementEnvironmentValue(
environment,
name,
fallback,
minimum,
maximum,
configFailure,
);
}
function absolutePath(
environment: ModelProviderCredentialTestExecutorProcessEnvironment,
name: string,
): string {
const value = absoluteManagementEnvironmentFile(
environment,
name,
configFailure,
);
if (normalize(value) !== value || parse(value).root === value) {
throw configFailure(`${name} is invalid`);
}
return value;
}
function loadConnection(
environment: ModelProviderCredentialTestExecutorProcessEnvironment,
): Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}> {
let connection: PostgresConnectionOptions;
try {
connection = loadPostgresConnectionEnvironment(environment, {
connectionString: 'QL3_POSTGRES_AI_CREDENTIAL_TESTER_URL',
host: 'QL3_POSTGRES_AI_CREDENTIAL_TESTER_HOST',
port: 'QL3_POSTGRES_AI_CREDENTIAL_TESTER_PORT',
database: 'QL3_POSTGRES_AI_CREDENTIAL_TESTER_DATABASE',
user: 'QL3_POSTGRES_AI_CREDENTIAL_TESTER_USER',
password: 'QL3_POSTGRES_AI_CREDENTIAL_TESTER_PASSWORD',
});
} catch (error) {
throw configFailure(
error instanceof Error
? error.message
: 'PostgreSQL AI credential tester connection is invalid',
);
}
const mode =
environment.QL3_POSTGRES_AI_CREDENTIAL_TESTER_TLS_MODE ?? 'verify-full';
if (mode !== 'verify-full' && mode !== 'disable') {
throw configFailure('PostgreSQL TLS mode must be verify-full or disable');
}
if (
mode === 'disable' &&
!booleanValue(
environment,
'QL3_POSTGRES_AI_CREDENTIAL_TESTER_ALLOW_INSECURE',
)
) {
throw configFailure('disabling PostgreSQL TLS requires explicit opt-in');
}
const servername = boundedValue(
environment,
'QL3_POSTGRES_AI_CREDENTIAL_TESTER_TLS_SERVERNAME',
253,
);
if (mode === 'verify-full' && !isPostgresTlsDnsServername(servername)) {
throw configFailure(
'PostgreSQL TLS servername must be an explicit DNS name',
);
}
const caFile = boundedValue(
environment,
'QL3_POSTGRES_AI_CREDENTIAL_TESTER_TLS_CA_FILE',
4_096,
);
if (mode === 'disable' && caFile !== undefined) {
throw configFailure('PostgreSQL CA file cannot be used with disabled TLS');
}
let ca: string | undefined;
if (caFile !== undefined) {
try {
ca = loadPostgresCertificateAuthorityFile(caFile);
} catch {
throw configFailure('PostgreSQL CA file is invalid');
}
}
const applicationName =
boundedValue(
environment,
'QL3_POSTGRES_AI_CREDENTIAL_TESTER_APPLICATION_NAME',
63,
) ?? 'qinglong3-ai-credential-tester';
if (!SAFE_APPLICATION_NAME.test(applicationName)) {
throw configFailure('PostgreSQL application name is invalid');
}
return Object.freeze({
connection: Object.freeze({
...connection,
tls:
mode === 'disable'
? Object.freeze({ mode: 'disable' as const })
: Object.freeze({
mode: 'verify-full' as const,
servername: servername!,
...(ca === undefined ? {} : { ca }),
}),
}),
pool: Object.freeze({
applicationName,
maxConnections: integerValue(
environment,
'QL3_POSTGRES_AI_CREDENTIAL_TESTER_POOL_MAX',
1,
1,
1,
),
idleTimeoutMs: integerValue(
environment,
'QL3_POSTGRES_AI_CREDENTIAL_TESTER_IDLE_TIMEOUT_MS',
1_000,
100,
10_000,
),
connectionTimeoutMs: integerValue(
environment,
'QL3_POSTGRES_AI_CREDENTIAL_TESTER_CONNECTION_TIMEOUT_MS',
5_000,
100,
60_000,
),
}),
});
}
function loadNetworkPolicyDenyCanary(
environment: ModelProviderCredentialTestExecutorProcessEnvironment,
): Readonly<{ host: string; port: number }> | undefined {
const host = boundedValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_TEST_DENY_CANARY_HOST',
253,
);
const portValue = boundedValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_TEST_DENY_CANARY_PORT',
5,
);
if (host === undefined && portValue === undefined) return undefined;
if (
!isPostgresTlsDnsServername(host) ||
portValue === undefined ||
!/^[1-9][0-9]{0,4}$/.test(portValue)
) {
throw configFailure('network policy deny canary is invalid');
}
const port = Number(portValue);
if (port > 65_535) {
throw configFailure('network policy deny canary is invalid');
}
return Object.freeze({ host, port });
}
export function loadModelProviderCredentialTestExecutorProcessConfig(
environment: ModelProviderCredentialTestExecutorProcessEnvironment,
): Readonly<ModelProviderCredentialTestExecutorProcessConfig> {
if (!environment || typeof environment !== 'object') {
throw configFailure('environment is invalid');
}
if (
!booleanValue(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_TEST_EXECUTOR_ENABLED',
)
) {
return Object.freeze({ enabled: false as const });
}
if (environment.QL3_PROFILE !== 'cluster-admin') {
throw configFailure('QL3_PROFILE must be cluster-admin');
}
const networkPolicyDenyCanary = loadNetworkPolicyDenyCanary(environment);
return Object.freeze({
enabled: true as const,
profile: 'cluster-admin' as const,
commandFile: absolutePath(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_TEST_COMMAND_FILE',
),
allowlistFile: absolutePath(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_TEST_ALLOWLIST_FILE',
),
secretRootDirectory: absolutePath(
environment,
'QL3_MODEL_PROVIDER_CREDENTIAL_TEST_SECRET_ROOT',
),
...(networkPolicyDenyCanary === undefined
? {}
: { networkPolicyDenyCanary }),
database: loadConnection(environment),
});
}
function exact(value: unknown, keys: readonly string[]): boolean {
if (!value || typeof value !== 'object' || Array.isArray(value)) return false;
const actual = Object.keys(value).sort();
const expected = [...keys].sort();
return (
actual.length === expected.length &&
actual.every((key, index) => key === expected[index])
);
}
function normalizeCommand(
value: unknown,
): Readonly<ModelProviderCredentialTestExecutorCommand> {
if (
!exact(value, ['executionId', 'schemaVersion', 'testId']) ||
(value as { schemaVersion?: unknown }).schemaVersion !== 1 ||
typeof (value as { executionId?: unknown }).executionId !== 'string' ||
!UUID_V4_PATTERN.test((value as { executionId: string }).executionId) ||
typeof (value as { testId?: unknown }).testId !== 'string' ||
!UUID_V4_PATTERN.test((value as { testId: string }).testId)
) {
throw configFailure('command is invalid');
}
const command = value as ModelProviderCredentialTestExecutorCommand;
return Object.freeze({
schemaVersion: 1 as const,
executionId: command.executionId,
testId: command.testId,
});
}
function readJson(filePath: string, maximumBytes: number): unknown {
const bytes = readManagementTlsFile(filePath, false, configFailure);
try {
if (bytes.length > maximumBytes)
throw configFailure('authority file is too large');
return JSON.parse(new TextDecoder('utf8', { fatal: true }).decode(bytes));
} catch (error) {
if (
error instanceof ModelProviderCredentialTestExecutorProcessConfigError
) {
throw error;
}
throw configFailure('authority file is invalid');
} finally {
bytes.fill(0);
}
}
function readCommand(
filePath: string,
): Readonly<ModelProviderCredentialTestExecutorCommand> {
return normalizeCommand(readJson(filePath, MAX_COMMAND_BYTES));
}
function readAllowlist(
filePath: string,
): Readonly<ModelProviderCredentialTestAllowlist> {
try {
return normalizeModelProviderCredentialTestAllowlist(
readJson(
filePath,
MAX_ALLOWLIST_BYTES,
) as ModelProviderCredentialTestAllowlist,
);
} catch (error) {
if (
error instanceof ModelProviderCredentialTestExecutorProcessConfigError
) {
throw error;
}
throw configFailure('allowlist is invalid');
}
}
export async function runModelProviderCredentialTestExecutorProcess(
options: RunModelProviderCredentialTestExecutorProcessOptions,
): Promise<Readonly<ModelProviderCredentialTestExecutorProcessResult>> {
const expectedKeys = ['environment'];
for (const key of [
'allowlist',
'assertReady',
'command',
'executor',
'fetch',
'monotonicNow',
'now',
'openDatabase',
'secrets',
'transportReady',
]) {
if (
(options as unknown as Record<string, unknown> | undefined)?.[key] !==
undefined
) {
expectedKeys.push(key);
}
}
if (!exact(options, expectedKeys)) throw configFailure('options are invalid');
const config = loadModelProviderCredentialTestExecutorProcessConfig(
options.environment,
);
if (!config.enabled) return Object.freeze({ status: 'disabled' as const });
const command = options.command
? normalizeCommand(options.command)
: readCommand(config.commandFile);
const allowlist = options.allowlist
? normalizeModelProviderCredentialTestAllowlist(options.allowlist)
: readAllowlist(config.allowlistFile);
const openDatabase =
options.openDatabase ??
createPostgresDatabaseOpener({
role: 'ai-credential-tester',
connection: config.database.connection,
pool: config.database.pool,
onPoolError() {
// In-flight queries own one-shot availability; there is no listener to withdraw.
},
});
const database = await openDatabase();
try {
const evidence = await (
options.assertReady ?? assertPostgresModelProviderCredentialTesterReady
)(database.pool);
const secrets =
options.secrets ??
(await createProjectedModelProviderSecretMaterialProvider({
rootDirectory: config.secretRootDirectory,
}));
let observedTransportFailureCode: string | undefined;
let observedTransportRequestDigest: string | undefined;
let observedTransportEndpoint:
| Readonly<{ addressSha256: string; port: number }>
| undefined;
const configuredFetch = options.fetch ?? globalThis.fetch;
const observedFetch: typeof globalThis.fetch = async (input, init) => {
const requestDigest = transportRequestDigest(input, init);
try {
return await configuredFetch(input, init);
} catch (error) {
observedTransportFailureCode = transportFailureCode(error);
observedTransportRequestDigest = requestDigest;
observedTransportEndpoint = transportFailureEndpoint(error);
throw error;
}
};
const executor =
options.executor ??
createModelProviderCredentialTestExecutor({
repository: new PostgresModelProviderCredentialTestExecutionRepository(
database.pool,
),
credentials: new PostgresModelProviderCredentialReader(database.pool),
secrets,
fetch: observedFetch,
transportReady:
options.transportReady ??
((baseUrl, signal) =>
waitForTransportReady(
baseUrl,
signal,
config.networkPolicyDenyCanary,
)),
...(options.now === undefined ? {} : { now: options.now }),
...(options.monotonicNow === undefined
? {}
: { monotonicNow: options.monotonicNow }),
});
const result = await executor.execute({
executionId: command.executionId,
testId: command.testId,
allowlist,
});
return Object.freeze({
status: 'completed' as const,
database: evidence,
test: result,
...(observedTransportFailureCode === undefined
? {}
: {
transportFailureCode: observedTransportFailureCode,
transportRequestDigest: observedTransportRequestDigest!,
...(observedTransportEndpoint === undefined
? {}
: {
transportAddressSha256:
observedTransportEndpoint.addressSha256,
transportPort: observedTransportEndpoint.port,
}),
}),
});
} finally {
await database.close();
}
}
@@ -0,0 +1,60 @@
#!/usr/bin/env node
import { migratePostgresModelInvocationFeature } from '@qinglong/ai/model-invocation-migration';
import { runPostgresMigrationProcess } from '@qinglong/cluster-postgres/migration-process';
const USAGE = 'Usage: ql3-ai-feature-migrate';
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as {
readonly name?: unknown;
readonly code?: unknown;
};
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-ai-feature-migration',
event: 'migration_failed',
name:
typeof candidate?.name === 'string' && candidate.name.length <= 128
? candidate.name
: 'Error',
...(typeof candidate?.code === 'string' && candidate.code.length <= 128
? { code: candidate.code }
: {}),
});
}
async function main(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
if (argv.length !== 0) {
process.stderr.write(
`${JSON.stringify({
code: 'QL3_AI_FEATURE_MIGRATION_CLI_USAGE_INVALID',
message: USAGE,
})}\n`,
);
process.exitCode = 64;
return;
}
try {
await runPostgresMigrationProcess({
environment: process.env,
migrate: ({ pool }) => migratePostgresModelInvocationFeature(pool),
});
process.stdout.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-ai-feature-migration',
event: 'migration_completed',
})}\n`,
);
} catch (error) {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
}
}
void main(process.argv.slice(2));
@@ -0,0 +1,83 @@
// Cluster Plugin Package executor boundary; keep approved-action dispatch explicit.
import type { PostgresPool } from '@qinglong/runtime-core';
import {
ApprovedActionDispatcher,
type ApprovedActionDispatcherOptions,
} from '@qinglong/runtime-core/approved-action-dispatcher';
import { PluginPackageApprovedActionHandler } from '@qinglong/runtime-core/plugin-package-approved-action';
import { PostgresApprovedActionExecutionRepository } from '@qinglong/cluster-postgres/approved-action-execution';
import { PostgresPluginPackageInstallRepository } from '@qinglong/cluster-postgres/plugin-package-install';
import { PostgresPluginPackageInstallProposalRepository } from '@qinglong/cluster-postgres/plugin-package-proposal';
import {
PostgresPluginPackagePublisherRevocationProposalRepository,
PostgresPluginPackagePublisherTrustTransitionProposalRepository,
PostgresPluginPackagePublisherTrustTransitionRepository,
} from '@qinglong/cluster-postgres/package-executor';
import {
ClusterPluginPackagePublisherRevocationApprovedActionHandler,
type ClusterPluginPackagePublisherRevocationExecutionPort,
} from '../publisher/pluginPackagePublisherRevocationApprovedAction';
import {
ClusterPluginPackagePublisherTrustTransitionApprovedActionHandler,
type ClusterPluginPackagePublisherTrustTransitionExecutionPort,
} from '../publisher/pluginPackagePublisherTrustTransitionApprovedAction';
export const CLUSTER_PLUGIN_PACKAGE_DISPATCH_BATCH_LIMIT = 16;
export interface ClusterPluginPackageApprovedActionDispatcherOptions
extends Omit<ApprovedActionDispatcherOptions, 'defaultBatchSize'> {
readonly pool: PostgresPool;
readonly defaultBatchSize?: number;
readonly publisherRevocations?: ClusterPluginPackagePublisherRevocationExecutionPort;
readonly publisherTrustTransitions?: ClusterPluginPackagePublisherTrustTransitionExecutionPort;
}
export function createClusterPluginPackageApprovedActionDispatcher(
options: ClusterPluginPackageApprovedActionDispatcherOptions,
): ApprovedActionDispatcher {
if (!options || typeof options !== 'object') {
throw new TypeError('cluster Package Approved Action options are invalid');
}
const {
pool,
defaultBatchSize,
publisherRevocations,
publisherTrustTransitions,
...dispatcherOptions
} = options;
const executions = new PostgresApprovedActionExecutionRepository(pool);
const handler = new PluginPackageApprovedActionHandler(
new PostgresPluginPackageInstallProposalRepository(pool),
new PostgresPluginPackageInstallRepository(pool),
);
const handlers = [
handler,
...(['overlap_add', 'safe_retire'] as const).map(
(mode) =>
new ClusterPluginPackagePublisherTrustTransitionApprovedActionHandler(
mode,
new PostgresPluginPackagePublisherTrustTransitionProposalRepository(
pool,
),
publisherTrustTransitions ??
new PostgresPluginPackagePublisherTrustTransitionRepository(pool),
),
),
...(publisherRevocations
? [
new ClusterPluginPackagePublisherRevocationApprovedActionHandler(
new PostgresPluginPackagePublisherRevocationProposalRepository(
pool,
),
publisherRevocations,
),
]
: []),
];
return new ApprovedActionDispatcher(executions, handlers, {
...dispatcherOptions,
defaultBatchSize:
defaultBatchSize ?? CLUSTER_PLUGIN_PACKAGE_DISPATCH_BATCH_LIMIT,
});
}
@@ -0,0 +1,70 @@
#!/usr/bin/env node
// Cluster Plugin Package executor boundary; keep the operational CLI explicit.
import { runClusterPluginPackageExecutorProcess } from './pluginPackageExecutorProcess';
const USAGE = 'Usage: ql3-plugin-package-execute';
function emit(value: Readonly<Record<string, unknown>>): void {
process.stdout.write(`${JSON.stringify(value)}\n`);
}
async function run(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
if (argv.length !== 0) {
process.stderr.write(
`${JSON.stringify({
code: 'QL3_PLUGIN_PACKAGE_EXECUTOR_CLI_USAGE_INVALID',
message: USAGE,
})}\n`,
);
process.exitCode = 64;
return;
}
try {
const result = await runClusterPluginPackageExecutorProcess({
environment: process.env,
});
if (result.status === 'disabled') {
emit({
schemaVersion: 1,
component: 'qinglong3-plugin-package-executor',
event: 'executor_disabled',
});
return;
}
emit({
schemaVersion: 1,
component: 'qinglong3-plugin-package-executor',
event: 'executor_completed',
databaseContractVersion: result.database.contractVersion,
databaseMigrationCount: result.database.migrationIds.length,
batches: result.batches,
});
} catch (error) {
const candidate = error as {
readonly name?: unknown;
readonly code?: unknown;
};
process.stderr.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-plugin-package-executor',
event: 'executor_failed',
name:
typeof candidate?.name === 'string'
? candidate.name.slice(0, 128)
: 'Error',
...(typeof candidate?.code === 'string'
? { code: candidate.code.slice(0, 128) }
: {}),
})}\n`,
);
process.exitCode = 1;
}
}
void run(process.argv.slice(2));
@@ -0,0 +1,464 @@
// Cluster Plugin Package executor boundary; keep process composition explicit.
import type {
OpenPostgresDatabase,
PostgresDatabaseResource,
PostgresPool,
} from '@qinglong/runtime-core';
import type {
ApprovedActionDispatchBatchSummary,
ApprovedActionDispatcher,
} from '@qinglong/runtime-core/approved-action-dispatcher';
import {
assertPostgresPackageExecutorSchemaReady,
createPostgresDatabaseOpener,
isPostgresTlsDnsServername,
loadPostgresCertificateAuthorityFile,
loadPostgresConnectionEnvironment,
type PostgresConnectionOptions,
type PostgresPoolOptions,
type PostgresSchemaReadinessReport,
} from '@qinglong/cluster-postgres/package-executor';
import {
createClusterPluginPackageApprovedActionDispatcher,
type ClusterPluginPackageApprovedActionDispatcherOptions,
} from './pluginPackageApprovedAction';
import {
consumeClusterPluginPackagePublisherRevocationApprovals,
type ClusterPluginPackagePublisherRevocationApprovalSummary,
type ConsumeClusterPluginPackagePublisherRevocationApprovalsOptions,
} from '../publisher/pluginPackagePublisherRevocationApprovalConsumer';
import {
consumeClusterPluginPackagePublisherTrustTransitionApprovals,
type ClusterPluginPackagePublisherTrustTransitionApprovalSummary,
type ConsumeClusterPluginPackagePublisherTrustTransitionApprovalsOptions,
} from '../publisher/pluginPackagePublisherTrustTransitionApprovalConsumer';
import {
runClusterPluginPackagePublisherRevocation,
} from '../publisher/pluginPackagePublisherRevocation';
export type ClusterPluginPackageExecutorProcessEnvironment = Readonly<
Record<string, string | undefined>
>;
export type ClusterPluginPackageExecutorProcessConfig =
| Readonly<{ enabled: false }>
| Readonly<{
enabled: true;
owner: string;
approvalBatchSize: number;
dispatchBatchSize: number;
maxBatches: number;
leaseDurationMs: number;
revocationPageSize: number;
revocationMaxPages: number;
database: Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}>;
}>;
export interface ClusterPluginPackageExecutorBatchResult {
readonly approvals: Readonly<ClusterPluginPackagePublisherRevocationApprovalSummary>;
readonly trustTransitionApprovals: Readonly<ClusterPluginPackagePublisherTrustTransitionApprovalSummary>;
readonly dispatch: Readonly<ApprovedActionDispatchBatchSummary>;
}
export type ClusterPluginPackageExecutorProcessResult =
| Readonly<{ status: 'disabled' }>
| Readonly<{
status: 'completed';
database: PostgresSchemaReadinessReport;
batches: readonly Readonly<ClusterPluginPackageExecutorBatchResult>[];
}>;
export interface RunClusterPluginPackageExecutorProcessOptions {
readonly environment: ClusterPluginPackageExecutorProcessEnvironment;
readonly openDatabase?: OpenPostgresDatabase;
readonly consumeApprovals?: (
options: ConsumeClusterPluginPackagePublisherRevocationApprovalsOptions,
) => Promise<
Readonly<ClusterPluginPackagePublisherRevocationApprovalSummary>
>;
readonly consumeTrustTransitionApprovals?: (
options: ConsumeClusterPluginPackagePublisherTrustTransitionApprovalsOptions,
) => Promise<
Readonly<ClusterPluginPackagePublisherTrustTransitionApprovalSummary>
>;
readonly createDispatcher?: (
options: ClusterPluginPackageApprovedActionDispatcherOptions,
) => ApprovedActionDispatcher;
readonly now?: () => number;
}
export class ClusterPluginPackageExecutorProcessConfigError extends TypeError {
readonly code = 'QL3_PLUGIN_PACKAGE_EXECUTOR_PROCESS_CONFIG_INVALID';
constructor(message: string) {
super(
`Plugin Package executor process configuration is invalid: ${message}`,
);
this.name = 'ClusterPluginPackageExecutorProcessConfigError';
}
}
const SAFE_OWNER = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
function enabledValue(
environment: ClusterPluginPackageExecutorProcessEnvironment,
): boolean {
const value = environment.QL3_PLUGIN_PACKAGE_EXECUTOR_ENABLED;
if (value === undefined || value === '' || value === 'false') return false;
if (value === 'true') return true;
throw new ClusterPluginPackageExecutorProcessConfigError(
'QL3_PLUGIN_PACKAGE_EXECUTOR_ENABLED must be true or false',
);
}
function boundedValue(
environment: ClusterPluginPackageExecutorProcessEnvironment,
name: string,
maximumLength: number,
): string | undefined {
const value = environment[name];
if (value === undefined || value === '') return undefined;
if (value.length > maximumLength || /[\0\r\n]/.test(value)) {
throw new ClusterPluginPackageExecutorProcessConfigError(
`${name} is invalid`,
);
}
return value;
}
function integerValue(
environment: ClusterPluginPackageExecutorProcessEnvironment,
name: string,
defaultValue: number,
minimum: number,
maximum: number,
): number {
const value = environment[name];
if (value === undefined || value === '') return defaultValue;
if (!/^\d+$/.test(value)) {
throw new ClusterPluginPackageExecutorProcessConfigError(
`${name} must be an integer`,
);
}
const parsed = Number(value);
if (!Number.isSafeInteger(parsed) || parsed < minimum || parsed > maximum) {
throw new ClusterPluginPackageExecutorProcessConfigError(
`${name} must be between ${minimum} and ${maximum}`,
);
}
return parsed;
}
function databaseConfig(
environment: ClusterPluginPackageExecutorProcessEnvironment,
): Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}> {
let connection: PostgresConnectionOptions;
try {
connection = loadPostgresConnectionEnvironment(environment, {
connectionString: 'QL3_POSTGRES_PACKAGE_EXECUTOR_URL',
host: 'QL3_POSTGRES_PACKAGE_EXECUTOR_HOST',
port: 'QL3_POSTGRES_PACKAGE_EXECUTOR_PORT',
database: 'QL3_POSTGRES_PACKAGE_EXECUTOR_DATABASE',
user: 'QL3_POSTGRES_PACKAGE_EXECUTOR_USER',
password: 'QL3_POSTGRES_PACKAGE_EXECUTOR_PASSWORD',
});
} catch (error) {
throw new ClusterPluginPackageExecutorProcessConfigError(
error instanceof Error
? error.message
: 'PostgreSQL Package executor connection is invalid',
);
}
const tlsMode = environment.QL3_POSTGRES_TLS_MODE ?? 'verify-full';
let tls: PostgresConnectionOptions['tls'];
if (tlsMode === 'disable') {
if (environment.QL3_POSTGRES_ALLOW_INSECURE !== 'true') {
throw new ClusterPluginPackageExecutorProcessConfigError(
'disabling PostgreSQL TLS requires QL3_POSTGRES_ALLOW_INSECURE=true',
);
}
tls = Object.freeze({ mode: 'disable' });
} else if (tlsMode === 'verify-full') {
const servername = boundedValue(
environment,
'QL3_POSTGRES_TLS_SERVERNAME',
253,
);
if (!isPostgresTlsDnsServername(servername)) {
throw new ClusterPluginPackageExecutorProcessConfigError(
'QL3_POSTGRES_TLS_SERVERNAME must be an explicit DNS name for verify-full',
);
}
const caFile = boundedValue(
environment,
'QL3_POSTGRES_TLS_CA_FILE',
4096,
);
let ca: string | undefined;
if (caFile !== undefined) {
try {
ca = loadPostgresCertificateAuthorityFile(caFile);
} catch {
throw new ClusterPluginPackageExecutorProcessConfigError(
'QL3_POSTGRES_TLS_CA_FILE must contain a bounded trusted CA bundle',
);
}
}
tls = Object.freeze({
mode: 'verify-full',
servername,
...(ca === undefined ? {} : { ca }),
});
} else {
throw new ClusterPluginPackageExecutorProcessConfigError(
'QL3_POSTGRES_TLS_MODE must be verify-full or disable',
);
}
const applicationName =
boundedValue(environment, 'QL3_POSTGRES_APPLICATION_NAME', 63) ??
'qinglong3-plugin-package-executor';
if (!/^[A-Za-z0-9][A-Za-z0-9._:-]{0,62}$/.test(applicationName)) {
throw new ClusterPluginPackageExecutorProcessConfigError(
'QL3_POSTGRES_APPLICATION_NAME is invalid',
);
}
return Object.freeze({
connection: Object.freeze({ ...connection, tls }),
pool: Object.freeze({
applicationName,
maxConnections: integerValue(
environment,
'QL3_POSTGRES_MAX_CONNECTIONS',
2,
1,
4,
),
connectionTimeoutMs: integerValue(
environment,
'QL3_POSTGRES_CONNECTION_TIMEOUT_MS',
5_000,
100,
60_000,
),
idleTimeoutMs: integerValue(
environment,
'QL3_POSTGRES_IDLE_TIMEOUT_MS',
10_000,
1_000,
300_000,
),
}),
});
}
export function loadClusterPluginPackageExecutorProcessConfig(
environment: ClusterPluginPackageExecutorProcessEnvironment,
): ClusterPluginPackageExecutorProcessConfig {
if (!environment || typeof environment !== 'object') {
throw new ClusterPluginPackageExecutorProcessConfigError(
'environment is required',
);
}
if (!enabledValue(environment)) return Object.freeze({ enabled: false });
const owner =
boundedValue(environment, 'QL3_PLUGIN_PACKAGE_EXECUTOR_OWNER', 128) ??
'cluster_package_executor_1';
if (!SAFE_OWNER.test(owner)) {
throw new ClusterPluginPackageExecutorProcessConfigError(
'QL3_PLUGIN_PACKAGE_EXECUTOR_OWNER is invalid',
);
}
return Object.freeze({
enabled: true,
owner,
approvalBatchSize: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_EXECUTOR_APPROVAL_BATCH_SIZE',
8,
1,
64,
),
dispatchBatchSize: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_EXECUTOR_DISPATCH_BATCH_SIZE',
8,
1,
64,
),
maxBatches: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_EXECUTOR_MAX_BATCHES',
4,
1,
64,
),
leaseDurationMs: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_EXECUTOR_LEASE_DURATION_MS',
600_000,
1,
600_000,
),
revocationPageSize: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_EXECUTOR_REVOCATION_PAGE_SIZE',
16,
1,
128,
),
revocationMaxPages: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_EXECUTOR_REVOCATION_MAX_PAGES',
16,
1,
64,
),
database: databaseConfig(environment),
});
}
function borrowedDatabase(
database: PostgresDatabaseResource,
): OpenPostgresDatabase {
return async () =>
Object.freeze({
pool: database.pool,
close: async () => undefined,
});
}
function isIdleBatch(
batch: Readonly<ClusterPluginPackageExecutorBatchResult>,
): boolean {
return (
batch.approvals.scanned === 0 &&
batch.trustTransitionApprovals.scanned === 0 &&
batch.dispatch.scanned === 0
);
}
export async function runClusterPluginPackageExecutorProcess(
options: RunClusterPluginPackageExecutorProcessOptions,
): Promise<ClusterPluginPackageExecutorProcessResult> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
!options.environment ||
(options.openDatabase !== undefined &&
typeof options.openDatabase !== 'function') ||
(options.consumeApprovals !== undefined &&
typeof options.consumeApprovals !== 'function') ||
(options.consumeTrustTransitionApprovals !== undefined &&
typeof options.consumeTrustTransitionApprovals !== 'function') ||
(options.createDispatcher !== undefined &&
typeof options.createDispatcher !== 'function') ||
(options.now !== undefined && typeof options.now !== 'function')
) {
throw new TypeError('Plugin Package executor process options are invalid');
}
const config = loadClusterPluginPackageExecutorProcessConfig(
options.environment,
);
if (!config.enabled) return Object.freeze({ status: 'disabled' });
const openDatabase =
options.openDatabase ??
createPostgresDatabaseOpener({
role: 'package-executor',
connection: config.database.connection,
pool: config.database.pool,
onPoolError: () => undefined,
});
const database = await openDatabase();
let failure: unknown;
try {
const evidence = await assertPostgresPackageExecutorSchemaReady(
database.pool,
);
const dispatcherFactory =
options.createDispatcher ??
createClusterPluginPackageApprovedActionDispatcher;
const consumeApprovals =
options.consumeApprovals ??
consumeClusterPluginPackagePublisherRevocationApprovals;
const consumeTrustTransitionApprovals =
options.consumeTrustTransitionApprovals ??
consumeClusterPluginPackagePublisherTrustTransitionApprovals;
const dispatcher = dispatcherFactory({
pool: database.pool,
owner: config.owner,
leaseDurationMs: config.leaseDurationMs,
defaultBatchSize: config.dispatchBatchSize,
...(options.now ? { clock: options.now } : {}),
publisherRevocations: {
async run(receipt) {
const result = await runClusterPluginPackagePublisherRevocation({
openDatabase: borrowedDatabase(database),
receipt,
// Durable proposal, dispatch, Project Policy fence and trust-head
// generation are revalidated in the same SERIALIZABLE mutation.
confirmAuthorization: () => undefined,
pageSize: config.revocationPageSize,
maxPages: config.revocationMaxPages,
});
return Object.freeze({
safeToAdmit: result.safeToAdmit,
receiptDigest: result.receiptDigest,
impactDigest: result.impactDigest,
});
},
},
});
const batches: Readonly<ClusterPluginPackageExecutorBatchResult>[] = [];
for (let index = 0; index < config.maxBatches; index += 1) {
const approvals = await consumeApprovals({
pool: database.pool,
limit: config.approvalBatchSize,
...(options.now ? { now: options.now } : {}),
});
const trustTransitionApprovals =
await consumeTrustTransitionApprovals({
pool: database.pool,
limit: config.approvalBatchSize,
...(options.now ? { now: options.now } : {}),
});
const dispatch = await dispatcher.dispatchBatch({
limit: config.dispatchBatchSize,
});
const batch = Object.freeze({
approvals,
trustTransitionApprovals,
dispatch,
});
batches.push(batch);
if (isIdleBatch(batch)) break;
}
return Object.freeze({
status: 'completed',
database: evidence,
batches: Object.freeze([...batches]),
});
} catch (error) {
failure = error;
throw error;
} finally {
try {
await database.close();
} catch (closeError) {
if (failure !== undefined) {
throw new AggregateError(
[failure, closeError],
'Plugin Package executor process failed and PostgreSQL did not close',
);
}
throw closeError;
}
}
}
@@ -0,0 +1,450 @@
// Cluster Plugin Package lifecycle boundary; keep execution authority explicit.
import {
PostgresPluginPackageLifecyclePlanRepository,
PostgresPluginPackageLifecycleRepository,
assertPostgresPackageExecutorSchemaReady,
type PostgresSchemaReadinessReport,
} from '@qinglong/cluster-postgres/package-executor';
import { PostgresApprovalRequestRepository } from '@qinglong/cluster-postgres/approved-action';
import { PostgresProjectPolicyRepository } from '@qinglong/cluster-postgres/project-policy';
import type {
OpenPostgresDatabase,
PostgresDatabaseResource,
PostgresPool,
} from '@qinglong/runtime-core';
import {
normalizeApprovalRequestRecord,
type ApprovedActionBinding,
type ApprovedActionDispatchRecord,
} from '@qinglong/runtime-core/approved-action';
import {
createPluginPackageLifecycleEvent,
pluginPackageLifecycleActionDigest,
PluginPackageLifecycleConflictError,
type PluginPackageLifecycleAction,
type PluginPackageLifecycleReceipt,
} from '@qinglong/runtime-core/plugin-package-lifecycle';
import {
MAX_PLUGIN_PACKAGE_LIFECYCLE_PLAN_LIFETIME_MS,
PluginPackageLifecyclePlanConflictError,
createPluginPackageLifecyclePlan,
normalizePluginPackageLifecyclePlan,
type PluginPackageLifecyclePlan,
} from '@qinglong/runtime-core/plugin-package-lifecycle-plan';
import { ProjectPolicyEngine } from '@qinglong/runtime-core/project-policy';
import type {
SecurityPolicyFence,
SecuritySubject,
} from '@qinglong/runtime-core/security';
import type { SecurityAuditRecord } from '@qinglong/runtime-core/security-audit';
const IDENTIFIER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const ACTION_REF_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:/-]{0,254}$/;
const PACKAGE_NAME_PATTERN = /^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$/;
const PROJECT_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const CLUSTER_LIFECYCLE_CONSUMER = Object.freeze({
subject: Object.freeze({
type: 'system' as const,
id: 'cluster_plugin_package_lifecycle_executor',
}),
authenticationId: 'cluster_plugin_package_lifecycle_executor_v1',
});
export interface RunClusterPluginPackageLifecyclePlanOptions {
readonly openDatabase: OpenPostgresDatabase;
readonly actionRef: string;
readonly action: PluginPackageLifecycleAction;
readonly projectId: string;
readonly packageName: string;
readonly requestedBy: SecuritySubject;
readonly confirmAuthorization: () => void | Promise<void>;
readonly lifetimeMs?: number;
}
export interface ClusterPluginPackageLifecyclePlanRun {
readonly database: PostgresSchemaReadinessReport;
readonly status: 'created' | 'existing';
readonly plan: Readonly<PluginPackageLifecyclePlan>;
}
export interface RunClusterPluginPackageLifecycleExecutionOptions {
readonly openDatabase: OpenPostgresDatabase;
readonly actionRef: string;
readonly approvalRequestId: string;
readonly consumptionId: string;
readonly dispatchId: string;
readonly auditEventId: string;
readonly confirmAuthorization: () => void | Promise<void>;
}
export interface ClusterPluginPackageLifecycleExecutionRun {
readonly database: PostgresSchemaReadinessReport;
readonly status: 'created' | 'existing';
readonly receipt: Readonly<PluginPackageLifecycleReceipt>;
}
type Row = Record<string, unknown>;
function identifier(value: unknown, label: string): string {
if (typeof value !== 'string' || !IDENTIFIER_PATTERN.test(value)) {
throw new TypeError(`${label} is invalid`);
}
return value;
}
function actionRef(value: unknown): string {
if (typeof value !== 'string' || !ACTION_REF_PATTERN.test(value)) {
throw new TypeError('actionRef is invalid');
}
return value;
}
function same(left: unknown, right: unknown): boolean {
return JSON.stringify(left) === JSON.stringify(right);
}
async function databaseNowMs(pool: PostgresPool): Promise<number> {
const result = await pool.query<Row>(
`SELECT floor(extract(epoch FROM clock_timestamp()) * 1000)::bigint
AS "nowMs"`,
);
const value = result.rows[0]?.nowMs;
const parsed =
typeof value === 'number'
? value
: typeof value === 'string' && /^(0|[1-9]\d*)$/.test(value)
? Number(value)
: Number.NaN;
if (
result.rows.length !== 1 ||
!Number.isSafeInteger(parsed) ||
parsed < 0
) {
throw new Error('PostgreSQL lifecycle clock is unavailable');
}
return parsed;
}
function binding(
plan: Readonly<PluginPackageLifecyclePlan>,
): Readonly<ApprovedActionBinding> {
return Object.freeze({
permission: 'package.manage',
actionType: `plugin_package.lifecycle.${plan.impact.action}`,
actionRef: plan.actionRef,
actionDigest: pluginPackageLifecycleActionDigest(plan.impact),
previewDigest: plan.impact.impactDigest,
});
}
function audit(
eventId: string,
approvalRequestId: string,
projectId: string,
fence: Readonly<SecurityPolicyFence>,
occurredAtMs: number,
): Readonly<SecurityAuditRecord> {
return Object.freeze({
eventId,
requestId: approvalRequestId,
operationId: 'approval.consume',
projectId,
subject: CLUSTER_LIFECYCLE_CONSUMER.subject,
authenticationId: CLUSTER_LIFECYCLE_CONSUMER.authenticationId,
outcome: 'allowed',
reasons: Object.freeze(['package_lifecycle_review']),
fence,
occurredAtMs,
});
}
async function closeDatabase(
database: PostgresDatabaseResource | undefined,
failure: unknown,
): Promise<void> {
if (!database) {
if (failure !== undefined) throw failure;
return;
}
try {
await database.close();
} catch (closeError) {
if (failure !== undefined) {
throw new AggregateError(
[failure, closeError],
'Cluster Plugin Package lifecycle failed and PostgreSQL did not close',
);
}
throw closeError;
}
if (failure !== undefined) throw failure;
}
export async function runClusterPluginPackageLifecyclePlan(
options: RunClusterPluginPackageLifecyclePlanOptions,
): Promise<Readonly<ClusterPluginPackageLifecyclePlanRun>> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
typeof options.openDatabase !== 'function' ||
typeof options.confirmAuthorization !== 'function' ||
typeof options.projectId !== 'string' ||
!PROJECT_ID_PATTERN.test(options.projectId) ||
typeof options.packageName !== 'string' ||
!PACKAGE_NAME_PATTERN.test(options.packageName)
) {
throw new TypeError(
'Cluster Plugin Package lifecycle plan options are invalid',
);
}
const requestedActionRef = actionRef(options.actionRef);
const lifetimeMs =
options.lifetimeMs ?? MAX_PLUGIN_PACKAGE_LIFECYCLE_PLAN_LIFETIME_MS;
if (
!Number.isSafeInteger(lifetimeMs) ||
lifetimeMs < 1_000 ||
lifetimeMs > MAX_PLUGIN_PACKAGE_LIFECYCLE_PLAN_LIFETIME_MS
) {
throw new TypeError(
'Cluster Plugin Package lifecycle plan lifetime is invalid',
);
}
let database: PostgresDatabaseResource | undefined;
let failure: unknown;
let result: Readonly<ClusterPluginPackageLifecyclePlanRun> | undefined;
try {
await options.confirmAuthorization();
database = await options.openDatabase();
const evidence = await assertPostgresPackageExecutorSchemaReady(
database.pool,
);
const lifecycles = new PostgresPluginPackageLifecycleRepository(
database.pool,
);
const plans = new PostgresPluginPackageLifecyclePlanRepository(
database.pool,
);
const existingValue = await plans.findByActionRef(requestedActionRef);
if (existingValue) {
const existing = normalizePluginPackageLifecyclePlan(existingValue);
if (
existing.impact.action !== options.action ||
existing.impact.target.projectId !== options.projectId ||
existing.impact.target.packageName !== options.packageName ||
!same(existing.requestedBy, options.requestedBy) ||
existing.expiresAtMs - existing.plannedAtMs !== lifetimeMs
) {
throw new PluginPackageLifecyclePlanConflictError(
'actionRef is bound to another lifecycle request',
);
}
await options.confirmAuthorization();
result = Object.freeze({
database: evidence,
status: 'existing' as const,
plan: existing,
});
} else {
const impact = await lifecycles.plan(
options.action,
options.projectId,
options.packageName,
);
const plannedAtMs = await databaseNowMs(database.pool);
const plan = createPluginPackageLifecyclePlan({
actionRef: requestedActionRef,
impact,
requestedBy: options.requestedBy,
plannedAtMs,
expiresAtMs: plannedAtMs + lifetimeMs,
});
await options.confirmAuthorization();
const created = await plans.create(plan);
result = Object.freeze({
database: evidence,
status: created.status,
plan: created.plan,
});
}
} catch (error) {
failure = error;
}
await closeDatabase(database, failure);
if (!result) {
throw new Error('Cluster Plugin Package lifecycle plan produced no result');
}
return result;
}
export async function runClusterPluginPackageLifecycleExecution(
options: RunClusterPluginPackageLifecycleExecutionOptions,
): Promise<Readonly<ClusterPluginPackageLifecycleExecutionRun>> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
typeof options.openDatabase !== 'function' ||
typeof options.confirmAuthorization !== 'function'
) {
throw new TypeError(
'Cluster Plugin Package lifecycle execution options are invalid',
);
}
const requestedActionRef = actionRef(options.actionRef);
const approvalRequestId = identifier(
options.approvalRequestId,
'approvalRequestId',
);
const consumptionId = identifier(options.consumptionId, 'consumptionId');
const dispatchId = identifier(options.dispatchId, 'dispatchId');
const auditEventId = identifier(options.auditEventId, 'auditEventId');
let database: PostgresDatabaseResource | undefined;
let failure: unknown;
let result: Readonly<ClusterPluginPackageLifecycleExecutionRun> | undefined;
try {
await options.confirmAuthorization();
database = await options.openDatabase();
const evidence = await assertPostgresPackageExecutorSchemaReady(
database.pool,
);
const plans = new PostgresPluginPackageLifecyclePlanRepository(
database.pool,
);
const planValue = await plans.findByActionRef(requestedActionRef);
if (!planValue) {
throw new PluginPackageLifecycleConflictError(
'durable lifecycle plan is absent',
);
}
const plan = normalizePluginPackageLifecyclePlan(planValue);
const approvals = new PostgresApprovalRequestRepository(database.pool);
let approvalValue = await approvals.findById(approvalRequestId);
if (!approvalValue) {
throw new PluginPackageLifecycleConflictError(
'lifecycle approval is absent',
);
}
let approval = normalizeApprovalRequestRecord(approvalValue);
const approvedAction = binding(plan);
if (
approval.projectId !== plan.impact.target.projectId ||
approval.decisionMode !== 'separation_of_duty' ||
!same(approval.action, approvedAction) ||
!same(approval.requestedBy, plan.requestedBy)
) {
throw new PluginPackageLifecycleConflictError(
'lifecycle approval does not match durable plan',
);
}
let dispatch: Readonly<ApprovedActionDispatchRecord> | null = null;
if (approval.version === 2 && approval.state === 'approved') {
const policy = new ProjectPolicyEngine(
new PostgresProjectPolicyRepository(database.pool),
);
const decision = await policy.decide({
subject: plan.requestedBy,
projectId: plan.impact.target.projectId,
permission: 'package.manage',
});
if (
(decision.effect !== 'allow' &&
decision.effect !== 'require_approval') ||
decision.fence === null
) {
throw new PluginPackageLifecycleConflictError(
'lifecycle requester is no longer authorized',
);
}
const consumedAtMs = await databaseNowMs(database.pool);
const consumed = await approvals.consume({
requestId: approvalRequestId,
expectedVersion: 2,
consumptionId,
dispatchId,
action: approvedAction,
requestedBy: plan.requestedBy,
consumedBy: CLUSTER_LIFECYCLE_CONSUMER.subject,
consumedAtMs,
authorizationFence: decision.fence,
audit: audit(
auditEventId,
approvalRequestId,
plan.impact.target.projectId,
decision.fence,
consumedAtMs,
),
});
approval = consumed.request;
dispatch = consumed.dispatch;
} else if (approval.version === 3 && approval.state === 'consumed') {
dispatch = await approvals.findDispatchById(dispatchId);
}
if (
approval.version !== 3 ||
approval.state !== 'consumed' ||
approval.consumptionId !== consumptionId ||
approval.dispatchId !== dispatchId ||
!dispatch ||
!same(dispatch.action, approvedAction) ||
!same(dispatch.requestedBy, plan.requestedBy) ||
!same(dispatch.approvedBy, approval.decidedBy) ||
!same(dispatch.consumedBy, CLUSTER_LIFECYCLE_CONSUMER.subject)
) {
throw new PluginPackageLifecycleConflictError(
'lifecycle dispatch does not match durable approval',
);
}
const lifecycles = new PostgresPluginPackageLifecycleRepository(
database.pool,
);
const event = createPluginPackageLifecycleEvent({
dispatchId: dispatch.id,
impact: plan.impact,
requestedBy: dispatch.requestedBy,
approvedBy: dispatch.approvedBy,
authorizationMode: 'separation_of_duty',
occurredAtMs: dispatch.createdAtMs,
});
const existingReceipt = await lifecycles.findByEventDigest(
event.eventDigest,
);
if (existingReceipt) {
await options.confirmAuthorization();
result = Object.freeze({
database: evidence,
status: 'existing' as const,
receipt: existingReceipt,
});
} else {
const currentImpact = await lifecycles.plan(
plan.impact.action,
plan.impact.target.projectId,
plan.impact.target.packageName,
);
if (!same(currentImpact, plan.impact)) {
throw new PluginPackageLifecycleConflictError(
'approved lifecycle impact is stale',
);
}
const transitioned = await lifecycles.transition(
event,
options.confirmAuthorization,
);
result = Object.freeze({
database: evidence,
status: transitioned.status,
receipt: transitioned.receipt,
});
}
} catch (error) {
failure = error;
}
await closeDatabase(database, failure);
if (!result) {
throw new Error(
'Cluster Plugin Package lifecycle execution produced no result',
);
}
return result;
}
@@ -0,0 +1,525 @@
// Cluster Plugin Package lifecycle boundary; keep approval management authority explicit.
import { PostgresApprovalRequestRepository } from '@qinglong/cluster-postgres/approved-action';
import { PostgresPluginPackageLifecyclePlanReader } from '@qinglong/cluster-postgres/package-manager';
import { PostgresProjectPolicyRepository } from '@qinglong/cluster-postgres/project-policy';
import type { PostgresPool } from '@qinglong/runtime-core';
import {
createApprovalRequest,
normalizeApprovalRequestRecord,
type ApprovalRequestRecord,
type CreateApprovalRequestResult,
type DecideApprovalRequestResult,
} from '@qinglong/runtime-core/approved-action';
import { pluginPackageLifecycleActionDigest } from '@qinglong/runtime-core/plugin-package-lifecycle';
import {
normalizePluginPackageLifecyclePlan,
type PluginPackageLifecyclePlan,
} from '@qinglong/runtime-core/plugin-package-lifecycle-plan';
import {
PluginPackageManagementAuthorizationError,
PluginPackageManagementConflictError,
PluginPackageManagementRequestError,
PluginPackageManagementUnavailableError,
} from '@qinglong/runtime-core/plugin-package-management';
import { ProjectPolicyEngine } from '@qinglong/runtime-core/project-policy';
import {
normalizeSecurityPrincipal,
type SecurityPolicyFence,
type SecurityPrincipal,
type SecuritySubject,
} from '@qinglong/runtime-core/security';
import type { SecurityAuditRecord } from '@qinglong/runtime-core/security-audit';
const DEFAULT_APPROVAL_LIFETIME_MS = 15 * 60 * 1000;
const IDENTIFIER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const ACTION_REF_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:/-]{0,254}$/;
const REASON_PATTERN = /^[a-z][a-z0-9_]{0,63}$/;
export interface ClusterPluginPackageLifecycleManagementOptions {
readonly pool: PostgresPool;
readonly now?: () => number;
readonly approvalLifetimeMs?: number;
}
export interface ProposeClusterPluginPackageLifecycleRequest {
readonly actionRef: string;
readonly approvalRequestId: string;
readonly approvalAuditEventId: string;
readonly principal: SecurityPrincipal;
}
export interface ProposeClusterPluginPackageLifecycleResult {
readonly plan: Readonly<PluginPackageLifecyclePlan>;
readonly approvalStatus: CreateApprovalRequestResult['status'];
readonly approvalRequest: Readonly<ApprovalRequestRecord>;
}
export interface DecideClusterPluginPackageLifecycleRequest {
readonly actionRef: string;
readonly approvalRequestId: string;
readonly expectedVersion: number;
readonly decisionId: string;
readonly auditEventId: string;
readonly decision: 'approved' | 'rejected';
readonly reasonCode: string;
readonly principal: SecurityPrincipal;
}
export interface InspectClusterPluginPackageLifecycleRequest {
readonly actionRef: string;
readonly approvalRequestId: string;
readonly inspectionId: string;
readonly principal: SecurityPrincipal;
}
export interface InspectClusterPluginPackageLifecycleResult {
readonly plan: Readonly<PluginPackageLifecyclePlan> | null;
readonly approvalRequest: Readonly<ApprovalRequestRecord> | null;
readonly stale: boolean;
}
export interface ClusterPluginPackageLifecycleManagementService {
propose(
request: ProposeClusterPluginPackageLifecycleRequest,
): Promise<Readonly<ProposeClusterPluginPackageLifecycleResult>>;
decide(
request: DecideClusterPluginPackageLifecycleRequest,
): Promise<Readonly<DecideApprovalRequestResult>>;
inspectAuthorized(
request: InspectClusterPluginPackageLifecycleRequest,
): Promise<Readonly<InspectClusterPluginPackageLifecycleResult>>;
}
function identifier(value: unknown, label: string): string {
if (typeof value !== 'string' || !IDENTIFIER_PATTERN.test(value)) {
throw new PluginPackageManagementRequestError(`${label} is invalid`);
}
return value;
}
function actionRef(value: unknown): string {
if (typeof value !== 'string' || !ACTION_REF_PATTERN.test(value)) {
throw new PluginPackageManagementRequestError('actionRef is invalid');
}
return value;
}
function currentTime(now: () => number): number {
const value = now();
if (!Number.isSafeInteger(value) || value < 0) {
throw new PluginPackageManagementUnavailableError();
}
return value;
}
function sameSubject(
left: Readonly<SecuritySubject>,
right: Readonly<SecuritySubject>,
): boolean {
return left.type === right.type && left.id === right.id;
}
function action(plan: Readonly<PluginPackageLifecyclePlan>) {
return Object.freeze({
permission: 'package.manage' as const,
actionType: `plugin_package.lifecycle.${plan.impact.action}`,
actionRef: plan.actionRef,
actionDigest: pluginPackageLifecycleActionDigest(plan.impact),
previewDigest: plan.impact.impactDigest,
});
}
function audit(
eventId: string,
requestId: string,
operationId: 'approval.request' | 'approval.decide',
projectId: string,
subject: Readonly<SecuritySubject>,
authenticationId: string,
outcome: 'allowed' | 'approval_required',
fence: Readonly<SecurityPolicyFence>,
occurredAtMs: number,
): Readonly<SecurityAuditRecord> {
return Object.freeze({
eventId,
requestId,
operationId,
projectId,
subject,
authenticationId,
outcome,
reasons: Object.freeze(['package_lifecycle_review']),
fence,
occurredAtMs,
});
}
export function createClusterPluginPackageLifecycleManagementService(
options: ClusterPluginPackageLifecycleManagementOptions,
): Readonly<ClusterPluginPackageLifecycleManagementService> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
key !== 'pool' &&
key !== 'now' &&
key !== 'approvalLifetimeMs',
) ||
(options.now !== undefined && typeof options.now !== 'function')
) {
throw new TypeError(
'Cluster Plugin Package lifecycle management options are invalid',
);
}
const approvalLifetimeMs =
options.approvalLifetimeMs ?? DEFAULT_APPROVAL_LIFETIME_MS;
if (
!Number.isSafeInteger(approvalLifetimeMs) ||
approvalLifetimeMs < 1_000 ||
approvalLifetimeMs > DEFAULT_APPROVAL_LIFETIME_MS
) {
throw new TypeError(
'Cluster Plugin Package lifecycle approval lifetime is invalid',
);
}
const now = options.now ?? Date.now;
const plans = new PostgresPluginPackageLifecyclePlanReader(options.pool);
const approvals = new PostgresApprovalRequestRepository(options.pool);
const policy = new ProjectPolicyEngine(
new PostgresProjectPolicyRepository(options.pool),
);
const authorize = async (
principalValue: SecurityPrincipal,
projectId: string,
permission: 'package.manage' | 'approval.decide',
observedAtMs: number,
): Promise<
Readonly<{
principal: Readonly<SecurityPrincipal>;
fence: Readonly<SecurityPolicyFence>;
}>
> => {
let principal;
try {
principal = normalizeSecurityPrincipal(principalValue, observedAtMs);
} catch {
throw new PluginPackageManagementAuthorizationError();
}
if (
principal.subject.type !== 'user' ||
(principal.assurance !== 'multi_factor' &&
principal.assurance !== 'hardware')
) {
throw new PluginPackageManagementAuthorizationError();
}
let decision;
try {
decision = await policy.authorize(principal, projectId, permission);
} catch (error) {
throw new PluginPackageManagementUnavailableError({
cause: error instanceof Error ? error : undefined,
});
}
if (decision.effect !== 'allow' || decision.fence === null) {
throw new PluginPackageManagementAuthorizationError();
}
return Object.freeze({ principal, fence: decision.fence });
};
const loadPlan = async (
requestedActionRef: string,
): Promise<Readonly<PluginPackageLifecyclePlan>> => {
let plan;
try {
plan = await plans.findByActionRef(actionRef(requestedActionRef));
} catch (error) {
throw new PluginPackageManagementUnavailableError({
cause: error instanceof Error ? error : undefined,
});
}
if (!plan) {
throw new PluginPackageManagementConflictError(
'Plugin Package lifecycle plan does not exist',
);
}
return normalizePluginPackageLifecyclePlan(plan);
};
return Object.freeze({
async propose(request: ProposeClusterPluginPackageLifecycleRequest) {
if (
!request ||
typeof request !== 'object' ||
Array.isArray(request) ||
Object.keys(request).sort().join('\0') !==
[
'actionRef',
'approvalAuditEventId',
'approvalRequestId',
'principal',
]
.sort()
.join('\0')
) {
throw new PluginPackageManagementRequestError(
'lifecycle proposal request is invalid',
);
}
const approvalRequestId = identifier(
request.approvalRequestId,
'approvalRequestId',
);
const approvalAuditEventId = identifier(
request.approvalAuditEventId,
'approvalAuditEventId',
);
const plan = await loadPlan(request.actionRef);
const observedAtMs = currentTime(now);
if (observedAtMs > plan.expiresAtMs) {
throw new PluginPackageManagementConflictError(
'Plugin Package lifecycle plan expired',
);
}
const authorization = await authorize(
request.principal,
plan.impact.target.projectId,
'package.manage',
observedAtMs,
);
if (!sameSubject(plan.requestedBy, authorization.principal.subject)) {
throw new PluginPackageManagementAuthorizationError();
}
const binding = action(plan);
const existing = await approvals.findById(approvalRequestId);
if (existing) {
const normalized = normalizeApprovalRequestRecord(existing);
if (
normalized.projectId !== plan.impact.target.projectId ||
normalized.decisionMode !== 'separation_of_duty' ||
!sameSubject(normalized.requestedBy, plan.requestedBy) ||
JSON.stringify(normalized.action) !== JSON.stringify(binding)
) {
throw new PluginPackageManagementConflictError(
'Approval request is bound to another lifecycle plan',
);
}
return Object.freeze({
plan,
approvalStatus: 'existing' as const,
approvalRequest: normalized,
});
}
const expiresAtMs = Math.min(
observedAtMs + approvalLifetimeMs,
plan.expiresAtMs,
);
if (expiresAtMs <= observedAtMs) {
throw new PluginPackageManagementConflictError(
'Plugin Package lifecycle plan has no approval lifetime',
);
}
const result = await approvals.create({
request: createApprovalRequest({
id: approvalRequestId,
projectId: plan.impact.target.projectId,
action: binding,
risk: 'high',
decisionMode: 'separation_of_duty',
requestedBy: authorization.principal.subject,
requestedAtMs: observedAtMs,
expiresAtMs,
requestFence: authorization.fence,
}),
audit: audit(
approvalAuditEventId,
approvalRequestId,
'approval.request',
plan.impact.target.projectId,
authorization.principal.subject,
authorization.principal.authenticationId,
'approval_required',
authorization.fence,
observedAtMs,
),
});
return Object.freeze({
plan,
approvalStatus: result.status,
approvalRequest: result.request,
});
},
async decide(request: DecideClusterPluginPackageLifecycleRequest) {
if (
!request ||
typeof request !== 'object' ||
Array.isArray(request) ||
Object.keys(request).sort().join('\0') !==
[
'actionRef',
'approvalRequestId',
'auditEventId',
'decision',
'decisionId',
'expectedVersion',
'principal',
'reasonCode',
]
.sort()
.join('\0') ||
(request.decision !== 'approved' &&
request.decision !== 'rejected') ||
typeof request.reasonCode !== 'string' ||
!REASON_PATTERN.test(request.reasonCode) ||
!Number.isSafeInteger(request.expectedVersion) ||
request.expectedVersion < 1
) {
throw new PluginPackageManagementRequestError(
'lifecycle decision request is invalid',
);
}
const plan = await loadPlan(request.actionRef);
const approvalRequestId = identifier(
request.approvalRequestId,
'approvalRequestId',
);
const decisionId = identifier(request.decisionId, 'decisionId');
const auditEventId = identifier(request.auditEventId, 'auditEventId');
const current = await approvals.findById(approvalRequestId);
if (!current) {
throw new PluginPackageManagementConflictError(
'Approval request does not exist',
);
}
const approval = normalizeApprovalRequestRecord(current);
if (
approval.action.actionRef !== plan.actionRef ||
JSON.stringify(approval.action) !== JSON.stringify(action(plan))
) {
throw new PluginPackageManagementConflictError(
'Approval request does not match lifecycle plan',
);
}
const observedAtMs = currentTime(now);
const authorization = await authorize(
request.principal,
approval.projectId,
'approval.decide',
observedAtMs,
);
if (
approval.decisionId === decisionId &&
approval.decision === request.decision &&
approval.decisionReasonCode === request.reasonCode &&
approval.decidedBy &&
sameSubject(approval.decidedBy, authorization.principal.subject)
) {
return Object.freeze({
status: 'existing' as const,
request: approval,
});
}
return approvals.decide({
requestId: approvalRequestId,
expectedVersion: request.expectedVersion,
decisionId,
decision: request.decision,
reasonCode: request.reasonCode,
principal: authorization.principal,
decidedAtMs: observedAtMs,
authorizationFence: authorization.fence,
audit: audit(
auditEventId,
approvalRequestId,
'approval.decide',
approval.projectId,
authorization.principal.subject,
authorization.principal.authenticationId,
'allowed',
authorization.fence,
observedAtMs,
),
});
},
async inspectAuthorized(
request: InspectClusterPluginPackageLifecycleRequest,
) {
if (
!request ||
typeof request !== 'object' ||
Array.isArray(request) ||
Object.keys(request).sort().join('\0') !==
[
'actionRef',
'approvalRequestId',
'inspectionId',
'principal',
]
.sort()
.join('\0')
) {
throw new PluginPackageManagementRequestError(
'lifecycle inspection request is invalid',
);
}
identifier(request.inspectionId, 'inspectionId');
const requestedActionRef = actionRef(request.actionRef);
const approvalRequestId = identifier(
request.approvalRequestId,
'approvalRequestId',
);
const [planValue, approvalValue] = await Promise.all([
plans.findByActionRef(requestedActionRef),
approvals.findById(approvalRequestId),
]);
if (!planValue && !approvalValue) {
throw new PluginPackageManagementConflictError(
'Plugin Package lifecycle state does not exist',
);
}
const plan = planValue
? normalizePluginPackageLifecyclePlan(planValue)
: null;
const approval = approvalValue
? normalizeApprovalRequestRecord(approvalValue)
: null;
const projectId = plan?.impact.target.projectId ?? approval?.projectId;
if (!projectId) {
throw new PluginPackageManagementUnavailableError();
}
const observedAtMs = currentTime(now);
try {
await authorize(
request.principal,
projectId,
'package.manage',
observedAtMs,
);
} catch (error) {
if (!(error instanceof PluginPackageManagementAuthorizationError)) {
throw error;
}
await authorize(
request.principal,
projectId,
'approval.decide',
observedAtMs,
);
}
return Object.freeze({
plan,
approvalRequest: approval,
stale:
plan === null ||
approval === null ||
approval.action.actionRef !== plan.actionRef ||
JSON.stringify(approval.action) !== JSON.stringify(action(plan)) ||
observedAtMs > plan.expiresAtMs,
});
},
});
}
@@ -0,0 +1,174 @@
// Cluster Plugin Package lifecycle boundary; keep quarantine authority explicit.
import type {
OpenPostgresDatabase,
PostgresDatabaseResource,
} from '@qinglong/runtime-core';
import {
InvalidPluginPackageQuarantineError,
normalizePluginPackageQuarantineEvent,
type PluginPackageQuarantineEvent,
type PluginPackageQuarantineRepository,
type PluginPackageWithdrawalReceipt,
} from '@qinglong/runtime-core/plugin-package-quarantine';
import {
PostgresPluginPackageQuarantineRepository,
assertPostgresPackageExecutorSchemaReady,
type PostgresSchemaReadinessReport,
} from '@qinglong/cluster-postgres/package-executor';
export const CLUSTER_PLUGIN_PACKAGE_QUARANTINE_BATCH_LIMIT = 128;
export interface ClusterPluginPackageQuarantineService {
quarantine(
events: readonly Readonly<PluginPackageQuarantineEvent>[],
confirmAuthorization: (
event: Readonly<PluginPackageQuarantineEvent>,
) => void | Promise<void>,
): Promise<
readonly Readonly<{
status: 'created' | 'existing';
eventDigest: string;
receipt: Readonly<PluginPackageWithdrawalReceipt>;
}>[]
>;
}
export interface RunClusterPluginPackageQuarantineOptions {
readonly openDatabase: OpenPostgresDatabase;
readonly events: readonly Readonly<PluginPackageQuarantineEvent>[];
readonly confirmAuthorization: (
event: Readonly<PluginPackageQuarantineEvent>,
) => void | Promise<void>;
}
export interface ClusterPluginPackageQuarantineRun {
readonly database: PostgresSchemaReadinessReport;
readonly results: readonly Readonly<{
status: 'created' | 'existing';
eventDigest: string;
receipt: Readonly<PluginPackageWithdrawalReceipt>;
}>[];
}
function targetKey(event: Readonly<PluginPackageQuarantineEvent>): string {
return [
event.target.projectId,
event.target.packageName,
event.target.installationId,
event.target.lockDigest,
].join('\0');
}
function normalizedBatch(
events: readonly Readonly<PluginPackageQuarantineEvent>[],
): readonly Readonly<PluginPackageQuarantineEvent>[] {
if (
!Array.isArray(events) ||
events.length < 1 ||
events.length > CLUSTER_PLUGIN_PACKAGE_QUARANTINE_BATCH_LIMIT ||
Object.keys(events).some((key, index) => key !== String(index))
) {
throw new InvalidPluginPackageQuarantineError(
`events must contain 1-${CLUSTER_PLUGIN_PACKAGE_QUARANTINE_BATCH_LIMIT} dense items`,
);
}
const normalized = events.map(normalizePluginPackageQuarantineEvent);
const eventDigests = new Set<string>();
const targets = new Set<string>();
for (const event of normalized) {
const target = targetKey(event);
if (eventDigests.has(event.eventDigest) || targets.has(target)) {
throw new InvalidPluginPackageQuarantineError(
'batch event digests and targets must be unique',
);
}
eventDigests.add(event.eventDigest);
targets.add(target);
}
return Object.freeze(normalized);
}
export function createClusterPluginPackageQuarantineService(
repository: PluginPackageQuarantineRepository,
): Readonly<ClusterPluginPackageQuarantineService> {
if (
!repository ||
typeof repository.findTargetsByLockDigest !== 'function' ||
typeof repository.findByEventDigest !== 'function' ||
typeof repository.quarantine !== 'function'
) {
throw new TypeError(
'Cluster Plugin Package quarantine repository is invalid',
);
}
return Object.freeze({
async quarantine(
events: readonly Readonly<PluginPackageQuarantineEvent>[],
confirmAuthorization: (
event: Readonly<PluginPackageQuarantineEvent>,
) => void | Promise<void>,
) {
const batch = normalizedBatch(events);
if (typeof confirmAuthorization !== 'function') {
throw new InvalidPluginPackageQuarantineError(
'confirmAuthorization is invalid',
);
}
const results = [];
for (const event of batch) {
const result = await repository.quarantine(event, () =>
confirmAuthorization(event),
);
results.push(
Object.freeze({
status: result.status,
eventDigest: event.eventDigest,
receipt: result.receipt,
}),
);
}
return Object.freeze(results);
},
});
}
export async function runClusterPluginPackageQuarantine(
options: RunClusterPluginPackageQuarantineOptions,
): Promise<Readonly<ClusterPluginPackageQuarantineRun>> {
if (!options || typeof options !== 'object' || Array.isArray(options)) {
throw new TypeError(
'Cluster Plugin Package quarantine options are invalid',
);
}
if (
Object.keys(options).some(
(key) =>
!['openDatabase', 'events', 'confirmAuthorization'].includes(key),
) ||
typeof options.openDatabase !== 'function'
) {
throw new TypeError(
'Cluster Plugin Package quarantine options shape is invalid',
);
}
const events = normalizedBatch(options.events);
if (typeof options.confirmAuthorization !== 'function') {
throw new InvalidPluginPackageQuarantineError(
'confirmAuthorization is invalid',
);
}
let database: PostgresDatabaseResource | undefined;
try {
database = await options.openDatabase();
const evidence = await assertPostgresPackageExecutorSchemaReady(
database.pool,
);
const results =
await createClusterPluginPackageQuarantineService(
new PostgresPluginPackageQuarantineRepository(database.pool),
).quarantine(events, options.confirmAuthorization);
return Object.freeze({ database: evidence, results });
} finally {
await database?.close();
}
}
@@ -0,0 +1,409 @@
/** Plugin Package management service boundary. */
import { PostgresApprovalRequestRepository } from '@qinglong/cluster-postgres/approved-action';
import { PostgresPluginPackageInstallInventoryReader } from '@qinglong/cluster-postgres/package-manager';
import { PostgresPluginPackageInstallProposalRepository } from '@qinglong/cluster-postgres/plugin-package-proposal';
import { PostgresProjectPolicyRepository } from '@qinglong/cluster-postgres/project-policy';
import type { PostgresPool } from '@qinglong/runtime-core';
import {
PluginPackageManagementAuthorizationError,
PluginPackageManagementConflictError,
PluginPackageManagementQuotaExceededError,
PluginPackageManagementRequestError,
PluginPackageManagementUnavailableError,
createPluginPackageManagementService,
type InspectPluginPackageInstallResult,
type PluginPackageManagementQuotaPort,
type PluginPackageManagementService as RuntimePluginPackageManagementService,
} from '@qinglong/runtime-core/plugin-package-management';
import {
MAX_PLUGIN_PACKAGE_INSTALL_INVENTORY_PAGE_SIZE,
normalizePluginPackageInstallInventoryCursor,
type PluginPackageInstallInventoryItem,
type PluginPackageInstallInventoryPage,
} from '@qinglong/runtime-core/plugin-package-install';
import { ProjectPolicyEngine } from '@qinglong/runtime-core/project-policy';
import {
normalizeSecurityPrincipal,
type SecurityPolicyDecision,
type SecurityPrincipal,
} from '@qinglong/runtime-core/security';
export const CLUSTER_PLUGIN_PACKAGE_MANAGEMENT_DECISION_MODE =
'separation_of_duty' as const;
type ClusterPluginPackageManagementMutationService = Pick<
RuntimePluginPackageManagementService,
'propose' | 'decide' | 'inspect'
>;
export interface InspectAuthorizedClusterPluginPackageRequest {
readonly actionRef: string;
readonly approvalRequestId: string;
readonly inspectionId: string;
readonly principal: SecurityPrincipal;
}
export interface InspectAuthorizedClusterPluginPackageInstallationRequest {
readonly projectId: string;
readonly packageName: string;
readonly inspectionId: string;
readonly principal: SecurityPrincipal;
}
export interface ListAuthorizedClusterPluginPackageInstallationsRequest {
readonly projectId: string;
readonly limit: number;
readonly after?: Readonly<{ packageName: string }>;
readonly inspectionId: string;
readonly principal: SecurityPrincipal;
}
export type ClusterPluginPackageManagementService =
ClusterPluginPackageManagementMutationService &
Readonly<{
inspectAuthorized(
request: InspectAuthorizedClusterPluginPackageRequest,
): Promise<Readonly<InspectPluginPackageInstallResult>>;
inspectInstallationAuthorized(
request: InspectAuthorizedClusterPluginPackageInstallationRequest,
): Promise<Readonly<PluginPackageInstallInventoryItem> | null>;
listInstallationsAuthorized(
request: ListAuthorizedClusterPluginPackageInstallationsRequest,
): Promise<Readonly<PluginPackageInstallInventoryPage>>;
}>;
export interface ClusterPluginPackageManagementOptions {
readonly pool: PostgresPool;
readonly approvalLifetimeMs?: number;
readonly now?: () => number;
readonly quota?: PluginPackageManagementQuotaPort;
}
const INSPECTION_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:/-]{0,254}$/;
const PROJECT_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const PACKAGE_NAME_PATTERN = /^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$/;
export function createClusterPluginPackageManagementService(
options: ClusterPluginPackageManagementOptions,
): Readonly<ClusterPluginPackageManagementService> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
key !== 'pool' &&
key !== 'approvalLifetimeMs' &&
key !== 'now' &&
key !== 'quota',
)
) {
throw new TypeError(
'cluster Plugin Package management options are invalid',
);
}
const now = options.now ?? Date.now;
const policy = new ProjectPolicyEngine(
new PostgresProjectPolicyRepository(options.pool),
);
const installations = new PostgresPluginPackageInstallInventoryReader(
options.pool,
);
const service = createPluginPackageManagementService(
policy,
new PostgresPluginPackageInstallProposalRepository(options.pool),
new PostgresApprovalRequestRepository(options.pool),
Object.freeze({
async dispatchBatch(): Promise<never> {
throw new Error(
'cluster Plugin Package management cannot execute approved actions',
);
},
}),
{
decisionMode: CLUSTER_PLUGIN_PACKAGE_MANAGEMENT_DECISION_MODE,
consumer: Object.freeze({
subject: Object.freeze({
type: 'system' as const,
id: 'cluster_package_management_unreachable_consumer',
}),
authenticationId: 'cluster-package-management-unreachable-consumer',
}),
...(options.approvalLifetimeMs === undefined
? {}
: { approvalLifetimeMs: options.approvalLifetimeMs }),
now,
...(options.quota === undefined ? {} : { quota: options.quota }),
},
);
const allowed = (
decision: Readonly<SecurityPolicyDecision>,
allowApproval: boolean,
): boolean =>
decision.fence !== null &&
(decision.effect === 'allow' ||
(allowApproval && decision.effect === 'require_approval'));
const authorizeInstallationInventory = async (
projectId: string,
inspectionId: string,
principalValue: SecurityPrincipal,
): Promise<Readonly<SecurityPrincipal>> => {
const observedAtMs = now();
if (!Number.isSafeInteger(observedAtMs) || observedAtMs < 0) {
throw new PluginPackageManagementUnavailableError();
}
let principal: Readonly<SecurityPrincipal>;
try {
principal = normalizeSecurityPrincipal(principalValue, observedAtMs);
} catch {
throw new PluginPackageManagementAuthorizationError();
}
let decision: Readonly<SecurityPolicyDecision>;
try {
decision = await policy.authorize(principal, projectId, 'package.manage');
} catch (error) {
throw new PluginPackageManagementUnavailableError({
cause: error instanceof Error ? error : undefined,
});
}
if (!allowed(decision, true)) {
throw new PluginPackageManagementAuthorizationError();
}
if (options.quota) {
try {
await options.quota.consume({
projectId,
subject: principal.subject,
operation: 'plugin-package.inspect',
idempotencyKey: inspectionId,
});
} catch (error) {
if (error instanceof PluginPackageManagementQuotaExceededError) {
throw error;
}
throw new PluginPackageManagementUnavailableError({
cause: error instanceof Error ? error : undefined,
});
}
}
return principal;
};
return Object.freeze({
propose: service.propose,
decide: service.decide,
inspect: service.inspect,
async inspectInstallationAuthorized(
request: InspectAuthorizedClusterPluginPackageInstallationRequest,
): Promise<Readonly<PluginPackageInstallInventoryItem> | null> {
if (
!request ||
typeof request !== 'object' ||
Array.isArray(request) ||
Object.keys(request).sort().join('\0') !==
['inspectionId', 'packageName', 'principal', 'projectId']
.sort()
.join('\0') ||
typeof request.projectId !== 'string' ||
!PROJECT_ID_PATTERN.test(request.projectId) ||
typeof request.packageName !== 'string' ||
!PACKAGE_NAME_PATTERN.test(request.packageName) ||
typeof request.inspectionId !== 'string' ||
!INSPECTION_ID_PATTERN.test(request.inspectionId)
) {
throw new PluginPackageManagementRequestError(
'installation inspection request is invalid',
);
}
await authorizeInstallationInventory(
request.projectId,
request.inspectionId,
request.principal,
);
try {
return await installations.findCurrent(
request.projectId,
request.packageName,
);
} catch (error) {
throw new PluginPackageManagementUnavailableError({
cause: error instanceof Error ? error : undefined,
});
}
},
async listInstallationsAuthorized(
request: ListAuthorizedClusterPluginPackageInstallationsRequest,
): Promise<Readonly<PluginPackageInstallInventoryPage>> {
const keys =
request && typeof request === 'object' && !Array.isArray(request)
? Object.keys(request)
: [];
if (
!request ||
typeof request !== 'object' ||
Array.isArray(request) ||
!keys.includes('projectId') ||
!keys.includes('limit') ||
!keys.includes('inspectionId') ||
!keys.includes('principal') ||
keys.some(
(key) =>
![
'after',
'inspectionId',
'limit',
'principal',
'projectId',
].includes(key),
) ||
typeof request.projectId !== 'string' ||
!PROJECT_ID_PATTERN.test(request.projectId) ||
!Number.isSafeInteger(request.limit) ||
request.limit < 1 ||
request.limit > MAX_PLUGIN_PACKAGE_INSTALL_INVENTORY_PAGE_SIZE ||
typeof request.inspectionId !== 'string' ||
!INSPECTION_ID_PATTERN.test(request.inspectionId)
) {
throw new PluginPackageManagementRequestError(
'installation list request is invalid',
);
}
let after: Readonly<{ packageName: string }> | undefined;
try {
after =
request.after === undefined
? undefined
: normalizePluginPackageInstallInventoryCursor(request.after);
} catch {
throw new PluginPackageManagementRequestError(
'installation list cursor is invalid',
);
}
await authorizeInstallationInventory(
request.projectId,
request.inspectionId,
request.principal,
);
try {
return await installations.listCurrentPage({
projectId: request.projectId,
limit: request.limit,
...(after === undefined ? {} : { after }),
});
} catch (error) {
throw new PluginPackageManagementUnavailableError({
cause: error instanceof Error ? error : undefined,
});
}
},
async inspectAuthorized(
request: InspectAuthorizedClusterPluginPackageRequest,
): Promise<Readonly<InspectPluginPackageInstallResult>> {
if (
!request ||
typeof request !== 'object' ||
Array.isArray(request) ||
Object.keys(request).length !== 4 ||
Object.keys(request).some(
(key) =>
![
'actionRef',
'approvalRequestId',
'inspectionId',
'principal',
].includes(key),
) ||
typeof request.inspectionId !== 'string' ||
!INSPECTION_ID_PATTERN.test(request.inspectionId)
) {
throw new PluginPackageManagementRequestError(
'inspection request is invalid',
);
}
const observedAtMs = now();
if (!Number.isSafeInteger(observedAtMs) || observedAtMs < 0) {
throw new PluginPackageManagementUnavailableError();
}
let principal: Readonly<SecurityPrincipal>;
try {
principal = normalizeSecurityPrincipal(request.principal, observedAtMs);
} catch {
throw new PluginPackageManagementAuthorizationError();
}
const current = await service.inspect(
request.actionRef,
request.approvalRequestId,
);
const projectId =
current.proposal?.projectId ?? current.approvalRequest?.projectId;
if (!projectId) {
throw new PluginPackageManagementConflictError(
'Plugin Package management state does not exist',
);
}
if (
(current.proposal &&
current.approvalRequest &&
(current.proposal.projectId !== current.approvalRequest.projectId ||
current.approvalRequest.action.actionRef !==
current.proposal.actionRef ||
current.approvalRequest.action.actionDigest !==
current.proposal.actionDigest ||
current.approvalRequest.action.previewDigest !==
current.proposal.previewDigest)) ||
(current.proposal &&
current.proposal.actionRef !== request.actionRef) ||
(current.approvalRequest &&
current.approvalRequest.id !== request.approvalRequestId)
) {
throw new PluginPackageManagementUnavailableError();
}
let packageDecision: Readonly<SecurityPolicyDecision>;
let approvalDecision: Readonly<SecurityPolicyDecision> | undefined;
try {
packageDecision = await policy.authorize(
principal,
projectId,
'package.manage',
);
if (!allowed(packageDecision, true)) {
approvalDecision = await policy.authorize(
principal,
projectId,
'approval.decide',
);
}
} catch (error) {
throw new PluginPackageManagementUnavailableError({
cause: error instanceof Error ? error : undefined,
});
}
if (
!allowed(packageDecision, true) &&
(!approvalDecision || !allowed(approvalDecision, false))
) {
throw new PluginPackageManagementAuthorizationError();
}
if (options.quota) {
try {
await options.quota.consume({
projectId,
subject: principal.subject,
operation: 'plugin-package.inspect',
idempotencyKey: request.inspectionId,
});
} catch (error) {
if (error instanceof PluginPackageManagementQuotaExceededError) {
throw error;
}
throw new PluginPackageManagementUnavailableError({
cause: error instanceof Error ? error : undefined,
});
}
}
return current;
},
});
}
@@ -0,0 +1,111 @@
#!/usr/bin/env node
/** One-shot Plugin Package management process CLI boundary. */
import {
startClusterPluginPackageManagementProcess,
type ClusterPluginPackageManagementProcessRuntime,
} from './pluginPackageManagementProcess';
const USAGE = 'Usage: ql3-plugin-package-manage';
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as {
readonly name?: unknown;
readonly code?: unknown;
};
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-plugin-package-management',
event: 'management_failed',
name:
typeof candidate?.name === 'string' && candidate.name.length <= 128
? candidate.name
: 'Error',
...(typeof candidate?.code === 'string' && candidate.code.length <= 128
? { code: candidate.code }
: {}),
});
}
function emit(value: Readonly<Record<string, unknown>>): void {
process.stdout.write(`${JSON.stringify(value)}\n`);
}
async function run(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
if (argv.length !== 0) {
process.stderr.write(
`${JSON.stringify({
code: 'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLI_USAGE_INVALID',
message: USAGE,
})}\n`,
);
process.exitCode = 64;
return;
}
let runtime: Readonly<ClusterPluginPackageManagementProcessRuntime>;
try {
runtime = await startClusterPluginPackageManagementProcess({
environment: process.env,
onError() {
emit({
schemaVersion: 1,
component: 'qinglong3-plugin-package-management',
event: 'management_unavailable',
});
},
});
} catch (error) {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
return;
}
if (runtime.status === 'disabled') {
emit({
schemaVersion: 1,
component: 'qinglong3-plugin-package-management',
event: 'management_disabled',
});
return;
}
emit({
schemaVersion: 1,
component: 'qinglong3-plugin-package-management',
event: 'management_started',
address: runtime.address,
identityGeneration: runtime.identity.generation,
databaseContractVersion: runtime.database.contractVersion,
databaseMigrationCount: runtime.database.migrationIds.length,
});
let stopping: Promise<void> | undefined;
const stop = (): Promise<void> => {
stopping ??= runtime.close().then(() => {
emit({
schemaVersion: 1,
component: 'qinglong3-plugin-package-management',
event: 'management_stopped',
});
});
return stopping;
};
for (const signal of ['SIGINT', 'SIGTERM'] as const) {
process.once(signal, () => {
void stop().then(
() => {
process.exitCode = 0;
},
(error) => {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
},
);
});
}
}
void run(process.argv.slice(2));
@@ -0,0 +1,102 @@
#!/usr/bin/env node
/** One-shot Plugin Package management client CLI boundary. */
import {
ClusterPluginPackageManagementClientRemoteError,
executeClusterPluginPackageManagementClient,
} from '../../management-support/pluginPackageManagementClient';
const USAGE =
'Usage: ql3-plugin-package-client --config=/absolute/client.json --command=/absolute/command.json --assertion=/absolute/assertion.jwt';
function parseArguments(
argv: readonly string[],
): Readonly<{
configFile: string;
commandFile: string;
assertionFile: string;
}> | null {
if (argv.length !== 3) return null;
const values = new Map<string, string>();
for (const argument of argv) {
const match = /^--(config|command|assertion)=(\/.+)$/.exec(argument);
if (!match || values.has(match[1]!)) return null;
values.set(match[1]!, match[2]!);
}
if (
!values.has('config') ||
!values.has('command') ||
!values.has('assertion')
) {
return null;
}
return Object.freeze({
configFile: values.get('config')!,
commandFile: values.get('command')!,
assertionFile: values.get('assertion')!,
});
}
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as {
readonly code?: unknown;
};
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-plugin-package-management-client',
event: 'command_failed',
code:
typeof candidate?.code === 'string' &&
candidate.code.length <= 128
? candidate.code
: 'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_FAILED',
...(error instanceof ClusterPluginPackageManagementClientRemoteError
? {
statusCode: error.statusCode,
responseCode: error.responseCode,
requestId: error.requestId,
...(error.retryAfterSeconds === null
? {}
: { retryAfterSeconds: error.retryAfterSeconds }),
}
: {}),
});
}
async function run(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
const paths = parseArguments(argv);
if (!paths) {
process.stderr.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-plugin-package-management-client',
event: 'usage_invalid',
code: 'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_USAGE_INVALID',
})}\n`,
);
process.exitCode = 64;
return;
}
try {
const result =
await executeClusterPluginPackageManagementClient(paths);
process.stdout.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-plugin-package-management-client',
event: 'command_completed',
requestId: result.requestId,
result: result.result,
})}\n`,
);
} catch (error) {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
}
}
void run(process.argv.slice(2));
@@ -0,0 +1,796 @@
/** Explicit Kubernetes PortForward client boundary for Plugin Package management. */
import {
createPrivateKey,
X509Certificate,
} from 'node:crypto';
import { Duplex, PassThrough, Writable } from 'node:stream';
import { TextDecoder } from 'node:util';
import {
ClusterPluginPackageManagementClientConfigurationError,
ClusterPluginPackageManagementClientRemoteError,
ClusterPluginPackageManagementClientRequestError,
executeClusterPluginPackageManagementClient,
readCanonicalFile,
type ClusterPluginPackageManagementClientPaths,
type ClusterPluginPackageManagementClientRawConnection,
type ClusterPluginPackageManagementClientResult,
} from '../../management-support/pluginPackageManagementClient';
const MAX_KUBERNETES_CONFIG_BYTES = 16 * 1024;
const MAX_KUBECONFIG_BYTES = 256 * 1024;
const MAX_KUBERNETES_CA_BYTES = 256 * 1024;
const MAX_KUBERNETES_CLIENT_MATERIAL_BYTES = 256 * 1024;
const MAX_KUBERNETES_TOKEN_BYTES = 16 * 1024;
const MANAGEMENT_NAME = 'ql3-plugin-package-management';
const MANAGEMENT_PORT = 8443;
const MANAGEMENT_LABEL_SELECTOR =
'app.kubernetes.io/name=ql3-plugin-package-management,' +
'app.kubernetes.io/component=plugin-package-management';
const CONTROL_PATTERN = /[\u0000-\u001f\u007f]/;
const DNS_LABEL_PATTERN =
/^[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?$/;
const CONTEXT_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:/@-]{0,255}$/;
const POD_NAME_PATTERN =
/^ql3-plugin-package-management-[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?(?:-[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?)?$/;
const TOKEN_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._~+/-]{0,16383}$/;
type JsonObject = Record<string, unknown>;
type KubernetesModule = typeof import('@kubernetes/client-node', {
with: { 'resolution-mode': 'import' }
});
type KubernetesConfig = InstanceType<KubernetesModule['KubeConfig']>;
interface ReviewedKubernetesClientConfig {
readonly schemaVersion: 1;
readonly kubeconfigFile: string;
readonly context: string;
readonly namespace: string;
readonly apiTimeoutMs: number;
}
interface KubernetesPod {
readonly metadata?: {
readonly name?: string;
readonly namespace?: string;
readonly uid?: string;
readonly deletionTimestamp?: unknown;
readonly labels?: Readonly<Record<string, string>>;
};
readonly spec?: {
readonly serviceAccountName?: string;
readonly automountServiceAccountToken?: boolean;
readonly containers?: readonly Readonly<{ readonly name?: string }>[];
};
readonly status?: {
readonly phase?: string;
readonly conditions?: readonly Readonly<{
readonly type?: string;
readonly status?: string;
}>[];
readonly containerStatuses?: readonly Readonly<{
readonly name?: string;
readonly ready?: boolean;
}>[];
};
}
interface KubernetesPodList {
readonly metadata?: {
readonly continue?: string;
};
readonly items?: readonly KubernetesPod[];
}
export interface ClusterPluginPackageManagementKubernetesPodApi {
listNamespacedPod(
request: Readonly<{
namespace: string;
labelSelector: string;
limit: number;
timeoutSeconds: number;
watch: false;
}>,
): Promise<KubernetesPodList>;
}
export interface ClusterPluginPackageManagementKubernetesRuntime {
readonly pods: ClusterPluginPackageManagementKubernetesPodApi;
openPortForward(
request: Readonly<{
namespace: string;
podName: string;
port: 8443;
}>,
): Promise<ClusterPluginPackageManagementClientRawConnection>;
}
export interface ClusterPluginPackageManagementPortForwardWebSocket {
addEventListener(
type: 'close' | 'error',
listener: () => void,
): void;
close(): void;
}
export interface ClusterPluginPackageManagementPortForwardApi {
portForward(
namespace: string,
podName: string,
targetPorts: number[],
output: Writable,
error: Writable,
input: PassThrough,
retryCount: 0,
): Promise<
| ClusterPluginPackageManagementPortForwardWebSocket
| (() => ClusterPluginPackageManagementPortForwardWebSocket | null)
>;
}
export interface ClusterPluginPackageManagementKubernetesClientPaths
extends ClusterPluginPackageManagementClientPaths {
readonly kubernetesFile: string;
}
export interface ClusterPluginPackageManagementKubernetesClientOptions {
readonly createRuntime?: (
kubeConfig: KubernetesConfig,
kubernetes: KubernetesModule,
) => ClusterPluginPackageManagementKubernetesRuntime;
}
export class ClusterPluginPackageManagementKubernetesClientConfigurationError extends TypeError {
readonly code =
'QL3_PLUGIN_PACKAGE_MANAGEMENT_KUBERNETES_CLIENT_CONFIG_INVALID';
constructor() {
super('Kubernetes Plugin Package management client configuration is invalid');
this.name =
'ClusterPluginPackageManagementKubernetesClientConfigurationError';
}
}
export class ClusterPluginPackageManagementKubernetesClientTunnelError extends Error {
readonly code =
'QL3_PLUGIN_PACKAGE_MANAGEMENT_KUBERNETES_CLIENT_TUNNEL_FAILED';
constructor(readonly cause?: unknown) {
super('Kubernetes Plugin Package management tunnel failed');
this.name = 'ClusterPluginPackageManagementKubernetesClientTunnelError';
}
}
function configurationFailure(): ClusterPluginPackageManagementKubernetesClientConfigurationError {
return new ClusterPluginPackageManagementKubernetesClientConfigurationError();
}
function exactObject(
value: unknown,
expectedKeys: readonly string[],
): asserts value is JsonObject {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw configurationFailure();
}
const actual = Object.keys(value).sort();
const expected = [...expectedKeys].sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
throw configurationFailure();
}
}
function decodeUtf8(bytes: Buffer): string {
try {
return new TextDecoder('utf-8', { fatal: true }).decode(bytes);
} catch {
throw configurationFailure();
}
}
function parseJson(bytes: Buffer): unknown {
try {
return JSON.parse(decodeUtf8(bytes));
} catch (error) {
if (
error instanceof
ClusterPluginPackageManagementKubernetesClientConfigurationError
) {
throw error;
}
throw configurationFailure();
}
}
function readPrivateFile(filePath: string, maximumBytes: number): Buffer {
try {
return readCanonicalFile(filePath, maximumBytes, 'private');
} catch {
throw configurationFailure();
}
}
function normalizeConfig(
value: unknown,
): Readonly<ReviewedKubernetesClientConfig> {
exactObject(value, [
'schemaVersion',
'kubeconfigFile',
'context',
'namespace',
'apiTimeoutMs',
]);
if (
value.schemaVersion !== 1 ||
typeof value.kubeconfigFile !== 'string' ||
typeof value.context !== 'string' ||
!CONTEXT_PATTERN.test(value.context) ||
typeof value.namespace !== 'string' ||
!DNS_LABEL_PATTERN.test(value.namespace) ||
!Number.isSafeInteger(value.apiTimeoutMs) ||
(value.apiTimeoutMs as number) < 1_000 ||
(value.apiTimeoutMs as number) > 30_000
) {
throw configurationFailure();
}
return Object.freeze({
schemaVersion: 1,
kubeconfigFile: value.kubeconfigFile,
context: value.context,
namespace: value.namespace,
apiTimeoutMs: value.apiTimeoutMs as number,
});
}
function decodeCanonicalBase64(
value: unknown,
maximumBytes: number,
): Buffer {
if (
typeof value !== 'string' ||
value.length < 4 ||
value.length > maximumBytes * 2 ||
!/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/.test(
value,
)
) {
throw configurationFailure();
}
const bytes = Buffer.from(value, 'base64');
if (
bytes.length < 1 ||
bytes.length > maximumBytes ||
bytes.toString('base64') !== value
) {
bytes.fill(0);
throw configurationFailure();
}
return bytes;
}
function validateRawKubeconfig(
value: unknown,
config: Readonly<ReviewedKubernetesClientConfig>,
): void {
exactObject(value, [
'apiVersion',
'kind',
'clusters',
'users',
'contexts',
'current-context',
]);
if (
value.apiVersion !== 'v1' ||
value.kind !== 'Config' ||
value['current-context'] !== config.context ||
!Array.isArray(value.clusters) ||
value.clusters.length !== 1 ||
!Array.isArray(value.users) ||
value.users.length !== 1 ||
!Array.isArray(value.contexts) ||
value.contexts.length !== 1
) {
throw configurationFailure();
}
const clusterEntry = value.clusters[0];
const userEntry = value.users[0];
const contextEntry = value.contexts[0];
exactObject(clusterEntry, ['name', 'cluster']);
const rawCluster = clusterEntry.cluster;
exactObject(rawCluster, [
'server',
'certificate-authority-data',
]);
exactObject(userEntry, ['name', 'user']);
const rawUser = userEntry.user;
if (!rawUser || typeof rawUser !== 'object' || Array.isArray(rawUser)) {
throw configurationFailure();
}
exactObject(contextEntry, ['name', 'context']);
const rawContext = contextEntry.context;
exactObject(rawContext, [
'cluster',
'user',
'namespace',
]);
if (
typeof clusterEntry.name !== 'string' ||
!CONTEXT_PATTERN.test(clusterEntry.name) ||
typeof userEntry.name !== 'string' ||
!CONTEXT_PATTERN.test(userEntry.name) ||
contextEntry.name !== config.context ||
rawContext.cluster !== clusterEntry.name ||
rawContext.user !== userEntry.name ||
rawContext.namespace !== config.namespace
) {
throw configurationFailure();
}
const userKeys = Object.keys(rawUser).sort();
if (
JSON.stringify(userKeys) !== JSON.stringify(['token']) &&
JSON.stringify(userKeys) !==
JSON.stringify(
['client-certificate-data', 'client-key-data'].sort(),
)
) {
throw configurationFailure();
}
}
function validateKubeConfig(
kubeConfig: KubernetesConfig,
config: Readonly<ReviewedKubernetesClientConfig>,
): void {
kubeConfig.setCurrentContext(config.context);
if (kubeConfig.getCurrentContext() !== config.context) {
throw configurationFailure();
}
const context = kubeConfig.getContextObject(config.context);
const cluster = kubeConfig.getCurrentCluster();
const user = kubeConfig.getCurrentUser();
if (
!context ||
context.namespace !== config.namespace ||
!cluster ||
!user
) {
throw configurationFailure();
}
let server: URL;
try {
server = new URL(cluster.server);
} catch {
throw configurationFailure();
}
if (
server.protocol !== 'https:' ||
server.username !== '' ||
server.password !== '' ||
(server.pathname !== '' && server.pathname !== '/') ||
server.search !== '' ||
server.hash !== '' ||
server.hostname.length < 1 ||
cluster.skipTLSVerify !== false ||
cluster.proxyUrl != null ||
cluster.caFile != null ||
typeof cluster.caData !== 'string' ||
(cluster.tlsServerName != null &&
cluster.tlsServerName !== server.hostname)
) {
throw configurationFailure();
}
const ca = decodeCanonicalBase64(
cluster.caData,
MAX_KUBERNETES_CA_BYTES,
);
try {
new X509Certificate(ca);
} catch {
throw configurationFailure();
} finally {
ca.fill(0);
}
if (
user.exec != null ||
user.authProvider != null ||
user.certFile != null ||
user.keyFile != null ||
user.username != null ||
user.password != null ||
user.impersonateUser != null
) {
throw configurationFailure();
}
const hasToken = user.token != null;
const hasCertificate =
user.certData != null || user.keyData != null;
if (
hasToken === hasCertificate ||
(hasToken &&
(typeof user.token !== 'string' ||
Buffer.byteLength(user.token, 'utf8') >
MAX_KUBERNETES_TOKEN_BYTES ||
CONTROL_PATTERN.test(user.token) ||
!TOKEN_PATTERN.test(user.token)))
) {
throw configurationFailure();
}
if (hasCertificate) {
const certificate = decodeCanonicalBase64(
user.certData,
MAX_KUBERNETES_CLIENT_MATERIAL_BYTES,
);
const privateKey = decodeCanonicalBase64(
user.keyData,
MAX_KUBERNETES_CLIENT_MATERIAL_BYTES,
);
try {
const parsedCertificate = new X509Certificate(certificate);
const parsedPrivateKey = createPrivateKey(privateKey);
if (!parsedCertificate.checkPrivateKey(parsedPrivateKey)) {
throw configurationFailure();
}
} catch (error) {
if (
error instanceof
ClusterPluginPackageManagementKubernetesClientConfigurationError
) {
throw error;
}
throw configurationFailure();
} finally {
certificate.fill(0);
privateKey.fill(0);
}
}
}
function isReviewedPod(
value: KubernetesPod,
namespace: string,
): value is KubernetesPod & {
readonly metadata: {
readonly name: string;
readonly namespace: string;
readonly uid: string;
};
} {
const labels = value.metadata?.labels;
return (
typeof value.metadata?.name === 'string' &&
POD_NAME_PATTERN.test(value.metadata.name) &&
value.metadata.namespace === namespace &&
typeof value.metadata.uid === 'string' &&
value.metadata.uid.length >= 8 &&
value.metadata.uid.length <= 128 &&
!CONTROL_PATTERN.test(value.metadata.uid) &&
value.metadata.deletionTimestamp === undefined &&
labels?.['app.kubernetes.io/name'] === MANAGEMENT_NAME &&
labels?.['app.kubernetes.io/component'] ===
'plugin-package-management' &&
value.spec?.serviceAccountName === MANAGEMENT_NAME &&
value.spec?.automountServiceAccountToken === false &&
value.spec?.containers?.some(({ name }) => name === 'management') ===
true &&
value.status?.phase === 'Running' &&
value.status.conditions?.some(
({ type, status }) => type === 'Ready' && status === 'True',
) === true &&
value.status.containerStatuses?.some(
({ name, ready }) => name === 'management' && ready === true,
) === true
);
}
function selectManagementPod(
value: KubernetesPodList,
namespace: string,
): string {
if (
!value ||
typeof value !== 'object' ||
!Array.isArray(value.items) ||
value.items.length < 1 ||
value.items.length > 3 ||
(value.metadata?.continue !== undefined &&
value.metadata.continue !== '')
) {
throw new ClusterPluginPackageManagementKubernetesClientTunnelError();
}
const current = value.items.filter(
({ metadata }) => metadata?.deletionTimestamp === undefined,
);
if (
current.length < 1 ||
current.length > 2 ||
!current.every((pod) => isReviewedPod(pod, namespace))
) {
throw new ClusterPluginPackageManagementKubernetesClientTunnelError();
}
return current
.map(({ metadata }) => metadata!.name!)
.sort()[0]!;
}
function deadline<T>(
operation: Promise<T>,
timeoutMs: number,
disposeLate?: (value: T) => void | Promise<void>,
): Promise<T> {
return new Promise((resolve, reject) => {
let settled = false;
const timer = setTimeout(() => {
if (settled) return;
settled = true;
reject(
new ClusterPluginPackageManagementKubernetesClientTunnelError(),
);
}, timeoutMs);
operation.then(
(value) => {
if (settled) {
void Promise.resolve(disposeLate?.(value)).catch(() => {});
return;
}
settled = true;
clearTimeout(timer);
resolve(value);
},
(error) => {
if (settled) return;
settled = true;
clearTimeout(timer);
reject(
error instanceof
ClusterPluginPackageManagementKubernetesClientTunnelError
? error
: new ClusterPluginPackageManagementKubernetesClientTunnelError(
error,
),
);
},
);
});
}
export async function openClusterPluginPackageManagementPortForward(
forward: ClusterPluginPackageManagementPortForwardApi,
request: Readonly<{
namespace: string;
podName: string;
port: 8443;
}>,
): Promise<ClusterPluginPackageManagementClientRawConnection> {
const incoming = new PassThrough();
const outgoing = new PassThrough();
let connection: Duplex | undefined;
let pendingError = false;
const errors = new Writable({
write(chunk: Buffer | string, _encoding, callback) {
const bytes = Buffer.isBuffer(chunk)
? chunk
: Buffer.from(chunk);
const failed = bytes.length > 0;
bytes.fill(0);
if (failed) {
pendingError = true;
connection?.destroy(
new ClusterPluginPackageManagementKubernetesClientTunnelError(),
);
}
callback();
},
});
const handle = await forward.portForward(
request.namespace,
request.podName,
[request.port],
incoming,
errors,
outgoing,
0,
);
const webSocket =
typeof handle === 'function' ? handle() : handle;
if (!webSocket) {
throw new ClusterPluginPackageManagementKubernetesClientTunnelError();
}
const nodeStreamPair = {
readable: incoming,
writable: outgoing,
};
// Node supports a { readable, writable } pair of Node streams here, while
// @types/node@24.13.3 currently models only the equivalent Web Streams pair.
connection = Duplex.from(
nodeStreamPair as unknown as Parameters<typeof Duplex.from>[0],
);
if (pendingError) {
connection.destroy(
new ClusterPluginPackageManagementKubernetesClientTunnelError(),
);
}
let closed = false;
const tunnelFailure = () => {
if (!closed) {
connection?.destroy(
new ClusterPluginPackageManagementKubernetesClientTunnelError(),
);
}
};
webSocket.addEventListener('close', tunnelFailure);
webSocket.addEventListener('error', tunnelFailure);
return Object.freeze({
stream: connection,
close() {
if (closed) return;
closed = true;
connection?.end();
incoming.end();
outgoing.end();
errors.end();
webSocket.close();
},
});
}
function productionRuntime(
kubeConfig: KubernetesConfig,
kubernetes: KubernetesModule,
): ClusterPluginPackageManagementKubernetesRuntime {
const pods = kubeConfig.makeApiClient(
kubernetes.CoreV1Api,
) as unknown as ClusterPluginPackageManagementKubernetesPodApi;
const forward = new kubernetes.PortForward(
kubeConfig,
true,
) as unknown as ClusterPluginPackageManagementPortForwardApi;
const runtime: ClusterPluginPackageManagementKubernetesRuntime = {
pods,
openPortForward: (request) =>
openClusterPluginPackageManagementPortForward(
forward,
request,
),
};
return Object.freeze(runtime);
}
export async function executeClusterPluginPackageManagementKubernetesClient(
paths: ClusterPluginPackageManagementKubernetesClientPaths,
options: ClusterPluginPackageManagementKubernetesClientOptions = {},
): Promise<Readonly<ClusterPluginPackageManagementClientResult>> {
exactObject(paths, [
'configFile',
'commandFile',
'assertionFile',
'kubernetesFile',
]);
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some((key) => key !== 'createRuntime') ||
(options.createRuntime !== undefined &&
typeof options.createRuntime !== 'function')
) {
throw configurationFailure();
}
let kubernetesConfigBytes: Buffer | undefined;
let kubeconfigBytes: Buffer | undefined;
try {
kubernetesConfigBytes = readPrivateFile(
paths.kubernetesFile,
MAX_KUBERNETES_CONFIG_BYTES,
);
const config = normalizeConfig(parseJson(kubernetesConfigBytes));
kubeconfigBytes = readPrivateFile(
config.kubeconfigFile,
MAX_KUBECONFIG_BYTES,
);
const kubernetes = await import('@kubernetes/client-node');
const kubeConfig = new kubernetes.KubeConfig();
try {
const kubeconfigText = decodeUtf8(kubeconfigBytes);
validateRawKubeconfig(parseJson(kubeconfigBytes), config);
kubeConfig.loadFromString(kubeconfigText);
validateKubeConfig(kubeConfig, config);
} catch (error) {
if (
error instanceof
ClusterPluginPackageManagementKubernetesClientConfigurationError
) {
throw error;
}
throw configurationFailure();
}
const runtime = (options.createRuntime ?? productionRuntime)(
kubeConfig,
kubernetes,
);
if (
!runtime ||
typeof runtime !== 'object' ||
typeof runtime.pods?.listNamespacedPod !== 'function' ||
typeof runtime.openPortForward !== 'function'
) {
throw configurationFailure();
}
const expectedHostname =
`${MANAGEMENT_NAME}.${config.namespace}.svc`;
return await executeClusterPluginPackageManagementClient(
{
configFile: paths.configFile,
commandFile: paths.commandFile,
assertionFile: paths.assertionFile,
},
{
async connect(target) {
if (
target.hostname !== expectedHostname ||
target.port !== MANAGEMENT_PORT
) {
throw configurationFailure();
}
const list = await deadline(
runtime.pods.listNamespacedPod({
namespace: config.namespace,
labelSelector: MANAGEMENT_LABEL_SELECTOR,
limit: 3,
timeoutSeconds: Math.ceil(config.apiTimeoutMs / 1_000),
watch: false,
}),
config.apiTimeoutMs,
);
const podName = selectManagementPod(
list,
config.namespace,
);
return await deadline(
runtime.openPortForward({
namespace: config.namespace,
podName,
port: MANAGEMENT_PORT,
}),
config.apiTimeoutMs,
async (connection) => {
await connection.close();
},
);
},
},
);
} catch (error) {
if (
error instanceof ClusterPluginPackageManagementClientRequestError &&
error.cause instanceof
ClusterPluginPackageManagementKubernetesClientTunnelError
) {
throw error.cause;
}
if (
error instanceof
ClusterPluginPackageManagementKubernetesClientConfigurationError ||
error instanceof
ClusterPluginPackageManagementKubernetesClientTunnelError ||
error instanceof
ClusterPluginPackageManagementClientConfigurationError ||
error instanceof ClusterPluginPackageManagementClientRequestError ||
error instanceof ClusterPluginPackageManagementClientRemoteError
) {
throw error;
}
throw new ClusterPluginPackageManagementKubernetesClientTunnelError(
error,
);
} finally {
kubernetesConfigBytes?.fill(0);
kubeconfigBytes?.fill(0);
}
}
@@ -0,0 +1,113 @@
#!/usr/bin/env node
/** One-shot Kubernetes-tunneled Plugin Package management client CLI boundary. */
import {
ClusterPluginPackageManagementClientRemoteError,
} from '../../management-support/pluginPackageManagementClient';
import {
executeClusterPluginPackageManagementKubernetesClient,
} from './pluginPackageManagementKubernetesClient';
const USAGE =
'Usage: ql3-plugin-package-client-kubernetes ' +
'--config=/absolute/client.json --command=/absolute/command.json ' +
'--assertion=/absolute/assertion.jwt ' +
'--kubernetes=/absolute/kubernetes.json';
function parseArguments(
argv: readonly string[],
): Readonly<{
configFile: string;
commandFile: string;
assertionFile: string;
kubernetesFile: string;
}> | null {
if (argv.length !== 4) return null;
const values = new Map<string, string>();
for (const argument of argv) {
const match =
/^--(config|command|assertion|kubernetes)=(\/.+)$/.exec(argument);
if (!match || values.has(match[1]!)) return null;
values.set(match[1]!, match[2]!);
}
if (
!values.has('config') ||
!values.has('command') ||
!values.has('assertion') ||
!values.has('kubernetes')
) {
return null;
}
return Object.freeze({
configFile: values.get('config')!,
commandFile: values.get('command')!,
assertionFile: values.get('assertion')!,
kubernetesFile: values.get('kubernetes')!,
});
}
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as { readonly code?: unknown };
return Object.freeze({
schemaVersion: 1,
component:
'qinglong3-plugin-package-management-kubernetes-client',
event: 'command_failed',
code:
typeof candidate?.code === 'string' &&
candidate.code.length <= 128
? candidate.code
: 'QL3_PLUGIN_PACKAGE_MANAGEMENT_KUBERNETES_CLIENT_FAILED',
...(error instanceof ClusterPluginPackageManagementClientRemoteError
? {
statusCode: error.statusCode,
responseCode: error.responseCode,
requestId: error.requestId,
...(error.retryAfterSeconds === null
? {}
: { retryAfterSeconds: error.retryAfterSeconds }),
}
: {}),
});
}
async function run(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
const paths = parseArguments(argv);
if (!paths) {
process.stderr.write(
`${JSON.stringify({
schemaVersion: 1,
component:
'qinglong3-plugin-package-management-kubernetes-client',
event: 'usage_invalid',
code:
'QL3_PLUGIN_PACKAGE_MANAGEMENT_KUBERNETES_CLIENT_USAGE_INVALID',
})}\n`,
);
process.exitCode = 64;
return;
}
try {
const result =
await executeClusterPluginPackageManagementKubernetesClient(paths);
process.stdout.write(
`${JSON.stringify({
schemaVersion: 1,
component:
'qinglong3-plugin-package-management-kubernetes-client',
event: 'command_completed',
requestId: result.requestId,
result: result.result,
})}\n`,
);
} catch (error) {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
}
}
void run(process.argv.slice(2));
@@ -0,0 +1,710 @@
/** Optional bounded Plugin Package management process composition boundary. */
import type {
OpenPostgresDatabase,
PostgresDatabaseResource,
} from '@qinglong/runtime-core';
import type {
ObservePluginPackagePublisherTrustSnapshotInput,
ObservePluginPackagePublisherTrustSnapshotResult,
} from '@qinglong/runtime-core/plugin-package-publisher-trust';
import {
assertPostgresPackageManagerSchemaReady,
createPostgresDatabaseOpener,
isPostgresTlsDnsServername,
loadPostgresCertificateAuthorityFile,
loadPostgresConnectionEnvironment,
PostgresPluginPackageIdentityKeysetLedgerRepository,
PostgresPluginPackageManagementQuotaRepository,
PostgresPluginPackagePublisherTrustAuthorityRepository,
type PostgresConnectionOptions,
type PostgresPoolOptions,
type PostgresSchemaReadinessReport,
} from '@qinglong/cluster-postgres/package-manager';
import {
createClusterPluginPackageIdentityKeysetFile,
type ClusterPluginPackageIdentityKeysetFile,
type ClusterPluginPackageIdentityKeysetSnapshot,
} from '../../management-support/pluginPackageIdentityKeyset';
import { createClusterPluginPackageManagementService } from './pluginPackageManagement';
import { createClusterPluginPackageLifecycleManagementService } from '../lifecycle/pluginPackageLifecycleManagement';
import {
loadClusterPluginPackagePublisherTrustFileEvidence,
type ClusterPluginPackagePublisherTrustFileEvidence,
} from '../recovery/pluginPackageRecoveryProcess';
import { createClusterPluginPackagePublisherTrustManagementService } from '../publisher/pluginPackagePublisherTrustManagement';
import {
startClusterPluginPackageManagementHttp,
type ClusterPluginPackageManagementHttpApplication,
type StartClusterPluginPackageManagementHttpOptions,
} from '../../management-support/pluginPackageManagementHttp';
import { createClusterPluginPackageManagementTransport } from './pluginPackageManagementTransport';
import {
absoluteManagementEnvironmentFile,
booleanManagementEnvironmentValue,
boundedManagementEnvironmentValue,
integerManagementEnvironmentValue,
readManagementTlsFile,
} from '../../management-support/managementProcessSupport';
const SAFE_HOST = /^[A-Za-z0-9][A-Za-z0-9.:-]{0,254}$/;
const SAFE_APPLICATION_NAME = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,62}$/;
export type ClusterPluginPackageManagementProcessEnvironment = Readonly<
Record<string, string | undefined>
>;
export type ClusterPluginPackageManagementProcessConfig =
| Readonly<{
enabled: false;
}>
| Readonly<{
enabled: true;
profile: 'cluster-admin';
host: string;
port: number;
certificateFile: string;
privateKeyFile: string;
identityKeysetFile: string;
publisherTrust: Readonly<{
file: string;
authorityProjectId: string;
authorityId: string;
observerId: string;
}>;
approvalLifetimeMs: number;
quota: Readonly<{
windowMs: number;
proposeLimit: number;
decideLimit: number;
inspectLimit: number;
}>;
http: Readonly<{
maxBodyBytes: number;
maxConnections: number;
maxConcurrentRequests: number;
requestTimeoutMs: number;
drainTimeoutMs: number;
rateWindowMs: number;
peerRequestLimit: number;
globalRequestLimit: number;
maxRateLimitPeers: number;
}>;
database: Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}>;
}>;
export type ClusterPluginPackageManagementProcessRuntime =
| Readonly<{
status: 'disabled';
close(): Promise<void>;
}>
| Readonly<{
status: 'active';
address: Readonly<{ host: string; port: number }>;
database: PostgresSchemaReadinessReport;
identity: ClusterPluginPackageIdentityKeysetSnapshot;
publisherTrust: Readonly<{
generation: number;
baseSnapshotDigest: string;
effectiveTrustDigest: string;
}>;
availabilityStatus(): 'ready' | 'unavailable' | 'stopped';
close(): Promise<void>;
}>;
export interface StartClusterPluginPackageManagementProcessOptions {
readonly environment: ClusterPluginPackageManagementProcessEnvironment;
readonly openDatabase?: OpenPostgresDatabase;
readonly identities?: ClusterPluginPackageIdentityKeysetFile;
readonly publisherTrustEvidence?: ClusterPluginPackagePublisherTrustFileEvidence;
readonly observePublisherTrust?: (
pool: PostgresDatabaseResource['pool'],
input: ObservePluginPackagePublisherTrustSnapshotInput,
) => Promise<Readonly<ObservePluginPackagePublisherTrustSnapshotResult>>;
readonly assertReady?: (
pool: PostgresDatabaseResource['pool'],
) => Promise<PostgresSchemaReadinessReport>;
readonly startHttp?: (
options: StartClusterPluginPackageManagementHttpOptions,
) => Promise<Readonly<ClusterPluginPackageManagementHttpApplication>>;
readonly now?: () => number;
readonly onError?: (error: unknown) => void;
}
export class ClusterPluginPackageManagementProcessConfigError extends TypeError {
readonly code = 'QL3_PLUGIN_PACKAGE_MANAGEMENT_PROCESS_CONFIG_INVALID';
constructor(message: string) {
super(
`Plugin Package management process configuration is invalid: ${message}`,
);
this.name = 'ClusterPluginPackageManagementProcessConfigError';
}
}
function configFailure(
message: string,
): ClusterPluginPackageManagementProcessConfigError {
return new ClusterPluginPackageManagementProcessConfigError(message);
}
function boundedValue(
environment: ClusterPluginPackageManagementProcessEnvironment,
name: string,
maximumLength: number,
required = false,
): string | undefined {
return boundedManagementEnvironmentValue(
environment,
name,
maximumLength,
configFailure,
required,
);
}
function booleanValue(
environment: ClusterPluginPackageManagementProcessEnvironment,
name: string,
): boolean {
return booleanManagementEnvironmentValue(environment, name, configFailure);
}
function integerValue(
environment: ClusterPluginPackageManagementProcessEnvironment,
name: string,
fallback: number,
minimum: number,
maximum: number,
): number {
return integerManagementEnvironmentValue(
environment,
name,
fallback,
minimum,
maximum,
configFailure,
);
}
function absoluteFile(
environment: ClusterPluginPackageManagementProcessEnvironment,
name: string,
): string {
return absoluteManagementEnvironmentFile(environment, name, configFailure);
}
function loadConnection(
environment: ClusterPluginPackageManagementProcessEnvironment,
): Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}> {
let connection: PostgresConnectionOptions;
try {
connection = loadPostgresConnectionEnvironment(environment, {
connectionString: 'QL3_POSTGRES_PACKAGE_MANAGER_URL',
host: 'QL3_POSTGRES_PACKAGE_MANAGER_HOST',
port: 'QL3_POSTGRES_PACKAGE_MANAGER_PORT',
database: 'QL3_POSTGRES_PACKAGE_MANAGER_DATABASE',
user: 'QL3_POSTGRES_PACKAGE_MANAGER_USER',
password: 'QL3_POSTGRES_PACKAGE_MANAGER_PASSWORD',
});
} catch (error) {
throw configFailure(
error instanceof Error
? error.message
: 'PostgreSQL Package manager connection is invalid',
);
}
const mode =
environment.QL3_POSTGRES_PACKAGE_MANAGER_TLS_MODE ?? 'verify-full';
if (mode !== 'verify-full' && mode !== 'disable') {
throw configFailure(
'QL3_POSTGRES_PACKAGE_MANAGER_TLS_MODE must be verify-full or disable',
);
}
if (
mode === 'disable' &&
!booleanValue(environment, 'QL3_POSTGRES_PACKAGE_MANAGER_ALLOW_INSECURE')
) {
throw configFailure(
'disabling Package manager PostgreSQL TLS requires QL3_POSTGRES_PACKAGE_MANAGER_ALLOW_INSECURE=true',
);
}
const servername = boundedValue(
environment,
'QL3_POSTGRES_PACKAGE_MANAGER_TLS_SERVERNAME',
253,
);
if (mode === 'verify-full' && !isPostgresTlsDnsServername(servername)) {
throw configFailure(
'QL3_POSTGRES_PACKAGE_MANAGER_TLS_SERVERNAME must be an explicit DNS name',
);
}
const caFile = boundedValue(
environment,
'QL3_POSTGRES_PACKAGE_MANAGER_TLS_CA_FILE',
4_096,
);
if (mode === 'disable' && caFile !== undefined) {
throw configFailure(
'QL3_POSTGRES_PACKAGE_MANAGER_TLS_CA_FILE cannot be used when TLS is disabled',
);
}
let ca: string | undefined;
if (caFile !== undefined) {
try {
ca = loadPostgresCertificateAuthorityFile(caFile);
} catch {
throw configFailure(
'QL3_POSTGRES_PACKAGE_MANAGER_TLS_CA_FILE is invalid',
);
}
}
const applicationName =
boundedValue(
environment,
'QL3_POSTGRES_PACKAGE_MANAGER_APPLICATION_NAME',
63,
) ?? 'qinglong3-plugin-package-manager';
if (!SAFE_APPLICATION_NAME.test(applicationName)) {
throw configFailure(
'QL3_POSTGRES_PACKAGE_MANAGER_APPLICATION_NAME is invalid',
);
}
return Object.freeze({
connection: Object.freeze({
...connection,
tls:
mode === 'disable'
? { mode: 'disable' as const }
: {
mode: 'verify-full' as const,
servername: servername!,
...(ca === undefined ? {} : { ca }),
},
}),
pool: Object.freeze({
applicationName,
maxConnections: integerValue(
environment,
'QL3_POSTGRES_PACKAGE_MANAGER_MAX_CONNECTIONS',
2,
1,
4,
),
connectionTimeoutMs: integerValue(
environment,
'QL3_POSTGRES_PACKAGE_MANAGER_CONNECTION_TIMEOUT_MS',
5_000,
100,
60_000,
),
}),
});
}
export function loadClusterPluginPackageManagementProcessConfig(
environment: ClusterPluginPackageManagementProcessEnvironment,
): Readonly<ClusterPluginPackageManagementProcessConfig> {
if (!environment || typeof environment !== 'object') {
throw configFailure('environment is invalid');
}
if (!booleanValue(environment, 'QL3_PLUGIN_PACKAGE_MANAGEMENT_ENABLED')) {
return Object.freeze({ enabled: false as const });
}
if (environment.QL3_PROFILE !== 'cluster-admin') {
throw configFailure(
'QL3_PROFILE must be cluster-admin when management is enabled',
);
}
const host =
boundedValue(environment, 'QL3_PLUGIN_PACKAGE_MANAGEMENT_HOST', 255) ??
'0.0.0.0';
if (!SAFE_HOST.test(host)) {
throw configFailure('QL3_PLUGIN_PACKAGE_MANAGEMENT_HOST is invalid');
}
const http = Object.freeze({
maxBodyBytes: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_MAX_BODY_BYTES',
64 * 1024,
1_024,
256 * 1024,
),
maxConnections: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_MAX_CONNECTIONS',
64,
1,
512,
),
maxConcurrentRequests: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_MAX_CONCURRENT_REQUESTS',
32,
1,
256,
),
requestTimeoutMs: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_REQUEST_TIMEOUT_MS',
10_000,
1_000,
60_000,
),
drainTimeoutMs: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_DRAIN_TIMEOUT_MS',
5_000,
100,
60_000,
),
rateWindowMs: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_RATE_WINDOW_MS',
60_000,
1_000,
5 * 60_000,
),
peerRequestLimit: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_PEER_REQUEST_LIMIT',
60,
1,
10_000,
),
globalRequestLimit: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_GLOBAL_REQUEST_LIMIT',
600,
1,
100_000,
),
maxRateLimitPeers: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_MAX_RATE_LIMIT_PEERS',
1_024,
1,
16_384,
),
});
if (http.globalRequestLimit < http.peerRequestLimit) {
throw configFailure(
'global request limit cannot be below the peer request limit',
);
}
return Object.freeze({
enabled: true as const,
profile: 'cluster-admin' as const,
host,
port: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_PORT',
8_443,
1,
65_535,
),
certificateFile: absoluteFile(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_TLS_CERT_FILE',
),
privateKeyFile: absoluteFile(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_TLS_KEY_FILE',
),
identityKeysetFile: absoluteFile(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_IDENTITY_KEYSET_FILE',
),
publisherTrust: Object.freeze({
file: absoluteFile(
environment,
'QL3_PLUGIN_PACKAGE_PUBLISHER_TRUST_FILE',
),
authorityProjectId: boundedValue(
environment,
'QL3_PLUGIN_PACKAGE_TRUST_AUTHORITY_PROJECT_ID',
128,
true,
)!,
authorityId:
boundedValue(
environment,
'QL3_PLUGIN_PACKAGE_TRUST_AUTHORITY_ID',
128,
) ?? 'cluster',
observerId:
boundedValue(
environment,
'QL3_PLUGIN_PACKAGE_TRUST_OBSERVER_ID',
128,
) ?? 'cluster-package-manager',
}),
approvalLifetimeMs: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_APPROVAL_LIFETIME_MS',
15 * 60_000,
1_000,
24 * 60 * 60_000,
),
quota: Object.freeze({
windowMs: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_QUOTA_WINDOW_MS',
60_000,
1_000,
5 * 60_000,
),
proposeLimit: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_PROPOSE_QUOTA',
30,
1,
1_000,
),
decideLimit: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_DECIDE_QUOTA',
60,
1,
1_000,
),
inspectLimit: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_MANAGEMENT_INSPECT_QUOTA',
600,
1,
1_000,
),
}),
http,
database: loadConnection(environment),
});
}
function readTlsFile(filePath: string, privateMaterial: boolean): Buffer {
return readManagementTlsFile(filePath, privateMaterial, configFailure);
}
export async function startClusterPluginPackageManagementProcess(
options: StartClusterPluginPackageManagementProcessOptions,
): Promise<Readonly<ClusterPluginPackageManagementProcessRuntime>> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
![
'environment',
'openDatabase',
'identities',
'publisherTrustEvidence',
'observePublisherTrust',
'assertReady',
'startHttp',
'now',
'onError',
].includes(key),
) ||
!options.environment ||
typeof options.environment !== 'object' ||
(options.openDatabase !== undefined &&
typeof options.openDatabase !== 'function') ||
(options.identities !== undefined &&
(typeof options.identities.reload !== 'function' ||
typeof options.identities.bind !== 'function')) ||
(options.publisherTrustEvidence !== undefined &&
(!options.publisherTrustEvidence ||
typeof options.publisherTrustEvidence !== 'object' ||
!options.publisherTrustEvidence.registry ||
!options.publisherTrustEvidence.snapshot)) ||
(options.observePublisherTrust !== undefined &&
typeof options.observePublisherTrust !== 'function') ||
(options.assertReady !== undefined &&
typeof options.assertReady !== 'function') ||
(options.startHttp !== undefined &&
typeof options.startHttp !== 'function') ||
(options.now !== undefined && typeof options.now !== 'function') ||
(options.onError !== undefined && typeof options.onError !== 'function')
) {
throw configFailure('options are invalid');
}
const config = loadClusterPluginPackageManagementProcessConfig(
options.environment,
);
if (!config.enabled) {
return Object.freeze({
status: 'disabled' as const,
close: () => Promise.resolve(),
});
}
const now = options.now ?? Date.now;
let http: Readonly<ClusterPluginPackageManagementHttpApplication> | undefined;
let database: PostgresDatabaseResource | undefined;
let unavailableError: unknown;
let closePromise: Promise<void> | undefined;
const report = (error: unknown): void => {
try {
options.onError?.(error);
} catch {
// Diagnostics do not own availability or cleanup.
}
};
const openDatabase =
options.openDatabase ??
createPostgresDatabaseOpener({
role: 'package-manager',
connection: config.database.connection,
pool: config.database.pool,
onPoolError(error) {
const firstAvailabilityError = unavailableError === undefined;
unavailableError ??= error;
http?.withdraw(error);
if (firstAvailabilityError) report(error);
},
});
try {
database = await openDatabase();
const evidence = await (
options.assertReady ?? assertPostgresPackageManagerSchemaReady
)(database.pool);
if (unavailableError !== undefined) throw unavailableError;
const identities =
options.identities ??
createClusterPluginPackageIdentityKeysetFile({
filePath: config.identityKeysetFile,
now,
ledger: new PostgresPluginPackageIdentityKeysetLedgerRepository(
database.pool,
),
});
const identity = await identities.reload();
const publisherTrustEvidence =
options.publisherTrustEvidence ??
loadClusterPluginPackagePublisherTrustFileEvidence(
config.publisherTrust.file,
);
const publisherTrustObservation = await (
options.observePublisherTrust ??
(async (pool, input) =>
new PostgresPluginPackagePublisherTrustAuthorityRepository(
pool,
).observeSnapshot(input))
)(database.pool, {
authorityId: config.publisherTrust.authorityId,
snapshot: publisherTrustEvidence.snapshot,
observedBy: config.publisherTrust.observerId,
observedAtMs: now(),
});
const quota = new PostgresPluginPackageManagementQuotaRepository(
database.pool,
{
windowMs: config.quota.windowMs,
limits: {
'plugin-package.propose': config.quota.proposeLimit,
'plugin-package.decide': config.quota.decideLimit,
'plugin-package.inspect': config.quota.inspectLimit,
},
},
);
const service = createClusterPluginPackageManagementService({
pool: database.pool,
approvalLifetimeMs: config.approvalLifetimeMs,
now,
quota,
});
const lifecycle =
createClusterPluginPackageLifecycleManagementService({
pool: database.pool,
approvalLifetimeMs: config.approvalLifetimeMs,
now,
});
const publisherTrust =
createClusterPluginPackagePublisherTrustManagementService({
pool: database.pool,
authorityProjectId: config.publisherTrust.authorityProjectId,
trustAuthorityId: config.publisherTrust.authorityId,
materialSnapshot: publisherTrustEvidence.snapshot,
approvalLifetimeMs: config.approvalLifetimeMs,
now,
quota,
});
const transport = createClusterPluginPackageManagementTransport({
service,
lifecycle,
publisherTrust,
now,
});
const privateKey = readTlsFile(config.privateKeyFile, true);
try {
const certificate = readTlsFile(config.certificateFile, false);
http = await (
options.startHttp ?? startClusterPluginPackageManagementHttp
)({
host: config.host,
port: config.port,
tls: { privateKey, certificate },
transport,
identities,
limits: config.http,
now,
onError: report,
});
} finally {
privateKey.fill(0);
}
if (unavailableError !== undefined) {
http.withdraw(unavailableError);
}
return Object.freeze({
status: 'active' as const,
address: http.address,
database: evidence,
identity,
publisherTrust: Object.freeze({
generation: publisherTrustObservation.head.generation,
baseSnapshotDigest:
publisherTrustObservation.head.baseSnapshotDigest,
effectiveTrustDigest:
publisherTrustObservation.head.effectiveTrustDigest,
}),
availabilityStatus: () => http?.availabilityStatus() ?? 'stopped',
close(): Promise<void> {
if (closePromise) return closePromise;
closePromise = (async () => {
let primaryError: unknown;
try {
await http?.close();
} catch (error) {
primaryError = error;
}
try {
await database?.close();
} catch (error) {
primaryError ??= error;
}
if (primaryError) throw primaryError;
})();
return closePromise;
},
});
} catch (error) {
try {
await http?.close();
} catch {
// Preserve startup failure.
}
try {
await database?.close();
} catch {
// Preserve startup failure.
}
throw error;
}
}
@@ -0,0 +1,234 @@
// Cluster Plugin Package publisher boundary; keep provenance recovery authority explicit.
import {
assertPluginPackageInstallMatchesLock,
type PluginPackageInstallCommit,
type PluginPackageInstallCreate,
type PluginPackageInstallRecord,
type PluginPackageInstallRecoveryCursor,
type PluginPackageInstallRecoveryPage,
type PluginPackageInstallRepository,
type PluginPackageLock,
} from '@qinglong/runtime-core/plugin-package-install';
import {
normalizePluginPackageStageEvidence,
type PluginPackageStageEvidence,
} from '@qinglong/runtime-core/plugin-package-installation';
import {
createPluginPackagePublisherProvenance,
type PluginPackagePublisherProvenance,
} from '@qinglong/runtime-core/plugin-package-publisher-provenance';
import {
PostgresPluginPackageInstallRepository,
} from '@qinglong/cluster-postgres/plugin-package-install';
import {
POSTGRES_PLUGIN_PACKAGE_PROVENANCE_RECOVERY_PAGE_LIMIT,
PostgresPluginPackagePublisherProvenanceRepository,
type PluginPackagePublisherProvenanceRecoveryCursor,
} from '@qinglong/cluster-postgres/package-executor';
import type { ClusterPluginPackageStageAuthority } from '../recovery/pluginPackageOciStage';
export const MAX_CLUSTER_PLUGIN_PACKAGE_PROVENANCE_RECOVERY_PAGES = 64;
export interface ClusterPluginPackagePublisherProvenanceRecoveryResult {
readonly pages: number;
readonly scanned: number;
readonly created: number;
readonly existing: number;
readonly remaining: boolean;
readonly safeToAdmit: boolean;
}
function stageEvidence(
record: Readonly<PluginPackageInstallRecord>,
): Readonly<PluginPackageStageEvidence> {
if (record.stageReceipt === null) {
throw new TypeError(
'Plugin Package install lacks durable stage evidence for provenance',
);
}
return normalizePluginPackageStageEvidence({
stageRef: record.stageReceipt.stageRef,
artifactDigest: record.stageReceipt.artifactDigest,
manifestDigest: record.stageReceipt.manifestDigest,
contentDigest: record.stageReceipt.contentDigest,
evidenceDigest: record.stageReceipt.evidenceDigest,
});
}
async function provenanceFor(
authority: ClusterPluginPackageStageAuthority,
lock: Readonly<PluginPackageLock>,
record: Readonly<PluginPackageInstallRecord>,
): Promise<Readonly<PluginPackagePublisherProvenance>> {
assertPluginPackageInstallMatchesLock(lock, record);
const stage = stageEvidence(record);
const signature = await authority.publisherEvidence(lock, stage);
return createPluginPackagePublisherProvenance({
projectId: record.projectId,
packageName: record.packageName,
installationId: record.installationId,
lockDigest: record.lockDigest,
artifactDigest: stage.artifactDigest,
manifestDigest: stage.manifestDigest,
contentDigest: stage.contentDigest,
stageEvidenceDigest: stage.evidenceDigest,
signature,
});
}
export class ClusterPluginPackageProvenanceInstallRepository
implements PluginPackageInstallRepository
{
constructor(
private readonly installs: PostgresPluginPackageInstallRepository,
private readonly provenance: PostgresPluginPackagePublisherProvenanceRepository,
private readonly authority: ClusterPluginPackageStageAuthority,
private readonly trustAuthorityId: string,
) {
if (
!(installs instanceof PostgresPluginPackageInstallRepository) ||
!(provenance instanceof
PostgresPluginPackagePublisherProvenanceRepository) ||
!authority ||
typeof authority.publisherEvidence !== 'function' ||
typeof trustAuthorityId !== 'string' ||
!/^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/.test(trustAuthorityId)
) {
throw new TypeError(
'Cluster Plugin Package provenance install repository is invalid',
);
}
}
find(
projectId: string,
packageName: string,
): Promise<Readonly<PluginPackageInstallRecord> | null> {
return this.installs.find(projectId, packageName);
}
findLock(
lockDigest: string,
): Promise<Readonly<PluginPackageLock> | null> {
return this.installs.findLock(lockDigest);
}
create(command: Readonly<PluginPackageInstallCreate>): Promise<
Readonly<{
status: 'created' | 'existing';
record: Readonly<PluginPackageInstallRecord>;
}>
> {
return this.installs.create(command);
}
async commit(command: Readonly<PluginPackageInstallCommit>): Promise<
Readonly<{
status: 'committed' | 'existing';
record: Readonly<PluginPackageInstallRecord>;
}>
> {
if (command.record.state !== 'staged') {
return this.installs.commit(command);
}
const lock = await this.installs.findLock(command.record.lockDigest);
if (!lock) {
throw new TypeError('Plugin Package stage lock is unavailable');
}
return this.provenance.commitStage(
command,
await provenanceFor(this.authority, lock, command.record),
this.trustAuthorityId,
);
}
listRecoveryPage(options: {
readonly limit: number;
readonly after?: Readonly<PluginPackageInstallRecoveryCursor>;
}): Promise<Readonly<PluginPackageInstallRecoveryPage>> {
return this.installs.listRecoveryPage(options);
}
}
export async function recoverClusterPluginPackagePublisherProvenance(
installs: PostgresPluginPackageInstallRepository,
repository: PostgresPluginPackagePublisherProvenanceRepository,
authority: ClusterPluginPackageStageAuthority,
options: Readonly<{
trustAuthorityId: string;
pageSize?: number;
maxPages?: number;
}>,
): Promise<Readonly<ClusterPluginPackagePublisherProvenanceRecoveryResult>> {
const pageSize = options.pageSize ?? 16;
const maxPages = options.maxPages ?? 16;
if (
!(installs instanceof PostgresPluginPackageInstallRepository) ||
!(repository instanceof
PostgresPluginPackagePublisherProvenanceRepository) ||
!authority ||
typeof authority.verify !== 'function' ||
typeof authority.publisherEvidence !== 'function' ||
typeof options.trustAuthorityId !== 'string' ||
!/^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/.test(
options.trustAuthorityId,
) ||
!Number.isSafeInteger(pageSize) ||
pageSize < 1 ||
pageSize > POSTGRES_PLUGIN_PACKAGE_PROVENANCE_RECOVERY_PAGE_LIMIT ||
!Number.isSafeInteger(maxPages) ||
maxPages < 1 ||
maxPages > MAX_CLUSTER_PLUGIN_PACKAGE_PROVENANCE_RECOVERY_PAGES
) {
throw new TypeError(
'Cluster Plugin Package provenance recovery configuration is invalid',
);
}
let after:
| Readonly<PluginPackagePublisherProvenanceRecoveryCursor>
| undefined;
const counts = {
pages: 0,
scanned: 0,
created: 0,
existing: 0,
};
let exhausted = false;
while (counts.pages < maxPages) {
const page = await repository.listMissingPage({
limit: pageSize,
...(after ? { after } : {}),
});
counts.pages += 1;
for (const record of page.records) {
const lock = await installs.findLock(record.lockDigest);
if (!lock || record.stageReceipt === null) {
throw new TypeError(
'Cluster Plugin Package provenance recovery source is incomplete',
);
}
assertPluginPackageInstallMatchesLock(lock, record);
await authority.verify(lock, record.stageReceipt);
const result = await repository.recordExisting(
record,
await provenanceFor(authority, lock, record),
options.trustAuthorityId,
);
counts.scanned += 1;
counts[result.status] += 1;
}
if (!page.truncated) {
exhausted = true;
break;
}
after = page.next;
}
const probe = await repository.listMissingPage({ limit: 1 });
const remaining = !exhausted || probe.records.length > 0;
return Object.freeze({
...counts,
remaining,
safeToAdmit: !remaining,
});
}
@@ -0,0 +1,203 @@
// Cluster Plugin Package publisher boundary; keep revocation authority explicit.
import type {
OpenPostgresDatabase,
PostgresDatabaseResource,
} from '@qinglong/runtime-core';
import {
normalizePluginPackagePublisherRevocationReceipt,
type PluginPackagePublisherRevocationReceipt,
} from '@qinglong/runtime-core/plugin-package-publisher-provenance';
import {
createPluginPackageQuarantineEvent,
pluginPackageQuarantineMutationId,
} from '@qinglong/runtime-core/plugin-package-quarantine';
import {
PostgresPluginPackagePublisherProvenanceRepository,
PostgresPluginPackageQuarantineRepository,
assertPostgresPackageExecutorSchemaReady,
type PostgresSchemaReadinessReport,
} from '@qinglong/cluster-postgres/package-executor';
import {
CLUSTER_PLUGIN_PACKAGE_QUARANTINE_BATCH_LIMIT,
createClusterPluginPackageQuarantineService,
} from '../lifecycle/pluginPackageQuarantine';
export const MAX_CLUSTER_PLUGIN_PACKAGE_REVOCATION_PAGES = 64;
export interface RunClusterPluginPackagePublisherRevocationOptions {
readonly openDatabase: OpenPostgresDatabase;
readonly receipt: Readonly<PluginPackagePublisherRevocationReceipt>;
readonly confirmAuthorization: (
receipt: Readonly<PluginPackagePublisherRevocationReceipt>,
) => void | Promise<void>;
readonly pageSize?: number;
readonly maxPages?: number;
}
export interface ClusterPluginPackagePublisherRevocationRun {
readonly database: PostgresSchemaReadinessReport;
readonly receiptStatus: 'created' | 'existing';
readonly receiptDigest: string;
readonly impactDigest: string;
readonly impacted: number;
readonly pages: number;
readonly quarantined: number;
readonly existing: number;
readonly remaining: boolean;
readonly safeToAdmit: boolean;
}
function normalizedOptions(
options: RunClusterPluginPackagePublisherRevocationOptions,
): Readonly<{
receipt: Readonly<PluginPackagePublisherRevocationReceipt>;
pageSize: number;
maxPages: number;
}> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
![
'openDatabase',
'receipt',
'confirmAuthorization',
'pageSize',
'maxPages',
].includes(key),
) ||
typeof options.openDatabase !== 'function' ||
typeof options.confirmAuthorization !== 'function'
) {
throw new TypeError(
'Cluster Plugin Package publisher revocation options are invalid',
);
}
const pageSize = options.pageSize ?? 64;
const maxPages = options.maxPages ?? 32;
if (
!Number.isSafeInteger(pageSize) ||
pageSize < 1 ||
pageSize > CLUSTER_PLUGIN_PACKAGE_QUARANTINE_BATCH_LIMIT ||
!Number.isSafeInteger(maxPages) ||
maxPages < 1 ||
maxPages > MAX_CLUSTER_PLUGIN_PACKAGE_REVOCATION_PAGES
) {
throw new TypeError(
'Cluster Plugin Package publisher revocation bounds are invalid',
);
}
return Object.freeze({
receipt: normalizePluginPackagePublisherRevocationReceipt(options.receipt),
pageSize,
maxPages,
});
}
/**
* Short-lived administration composition. The immutable revocation receipt
* and its bounded impact are committed before quarantine materialization.
* Re-running the same receipt converges on the same facts and skips targets
* already quarantined or superseded by a newer installation head.
*/
export async function runClusterPluginPackagePublisherRevocation(
options: RunClusterPluginPackagePublisherRevocationOptions,
): Promise<Readonly<ClusterPluginPackagePublisherRevocationRun>> {
const normalized = normalizedOptions(options);
let database: PostgresDatabaseResource | undefined;
let result: Readonly<ClusterPluginPackagePublisherRevocationRun> | undefined;
let failure: unknown;
try {
database = await options.openDatabase();
const evidence = await assertPostgresPackageExecutorSchemaReady(
database.pool,
);
const provenance =
new PostgresPluginPackagePublisherProvenanceRepository(database.pool);
const quarantine = createClusterPluginPackageQuarantineService(
new PostgresPluginPackageQuarantineRepository(database.pool),
);
const impactResult = await provenance.recordRevocationImpact(
normalized.receipt,
() => options.confirmAuthorization(normalized.receipt),
);
let pages = 0;
let quarantined = 0;
let existing = 0;
while (pages < normalized.maxPages) {
const page = await provenance.listPendingQuarantineTargets(
impactResult.impact.impactDigest,
normalized.pageSize,
);
if (page.targets.length === 0) break;
const events = page.targets.map((target) =>
createPluginPackageQuarantineEvent({
mutationId: pluginPackageQuarantineMutationId(
normalized.receipt.receiptDigest,
target,
),
revocationReceiptDigest: normalized.receipt.receiptDigest,
impactDigest: impactResult.impact.impactDigest,
target,
proposer: normalized.receipt.proposer,
confirmer: normalized.receipt.confirmer,
authorizationMode: normalized.receipt.authorizationMode,
reasonCode: normalized.receipt.reasonCode,
occurredAtMs: normalized.receipt.revokedAtMs,
}),
);
const quarantineResults = await quarantine.quarantine(
events,
() => options.confirmAuthorization(normalized.receipt),
);
pages += 1;
for (const item of quarantineResults) {
if (item.status === 'created') quarantined += 1;
else existing += 1;
}
}
const probe = await provenance.listPendingQuarantineTargets(
impactResult.impact.impactDigest,
1,
);
const remaining = probe.targets.length > 0;
result = Object.freeze({
database: evidence,
receiptStatus: impactResult.status,
receiptDigest: normalized.receipt.receiptDigest,
impactDigest: impactResult.impact.impactDigest,
impacted: impactResult.impact.items.length,
pages,
quarantined,
existing,
remaining,
safeToAdmit: !remaining,
});
} catch (error) {
failure = error;
}
if (database) {
try {
await database.close();
} catch (closeError) {
if (failure !== undefined) {
throw new AggregateError(
[failure, closeError],
'Cluster Plugin Package publisher revocation failed and PostgreSQL did not close',
);
}
throw closeError;
}
}
if (failure !== undefined) throw failure;
if (!result) {
throw new Error(
'Cluster Plugin Package publisher revocation produced no result',
);
}
return result;
}
@@ -0,0 +1,172 @@
// Cluster Plugin Package publisher boundary; keep approval consumption authority explicit.
import { createHash } from 'node:crypto';
import { PostgresApprovalRequestRepository } from '@qinglong/cluster-postgres/approved-action';
import {
PostgresPluginPackagePublisherRevocationProposalRepository,
} from '@qinglong/cluster-postgres/package-executor';
import { PostgresProjectPolicyRepository } from '@qinglong/cluster-postgres/project-policy';
import type { PostgresPool } from '@qinglong/runtime-core';
import { ProjectPolicyEngine } from '@qinglong/runtime-core/project-policy';
export const CLUSTER_PLUGIN_PACKAGE_PUBLISHER_APPROVAL_BATCH_LIMIT = 16;
export interface ConsumeClusterPluginPackagePublisherRevocationApprovalsOptions {
readonly pool: PostgresPool;
readonly now?: () => number;
readonly limit?: number;
}
export interface ClusterPluginPackagePublisherRevocationApprovalSummary {
readonly scanned: number;
readonly consumed: number;
readonly existing: number;
readonly expired: number;
readonly blocked: number;
}
function stableDigest(domain: string, value: string): string {
return createHash('sha256')
.update(domain)
.update('\0')
.update(value)
.digest('hex');
}
function stableId(prefix: string, domain: string, value: string): string {
return `${prefix}-${stableDigest(domain, value)}`;
}
function stableAuditEventId(requestId: string): string {
const bytes = Buffer.from(
stableDigest(
'qinglong/plugin-package-publisher-revocation-consume-audit@v1',
requestId,
),
'hex',
);
bytes[6] = (bytes[6]! & 0x0f) | 0x40;
bytes[8] = (bytes[8]! & 0x3f) | 0x80;
const hex = bytes.toString('hex');
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(
12,
16,
)}-${hex.slice(16, 20)}-${hex.slice(20, 32)}`;
}
export async function consumeClusterPluginPackagePublisherRevocationApprovals(
options: ConsumeClusterPluginPackagePublisherRevocationApprovalsOptions,
): Promise<
Readonly<ClusterPluginPackagePublisherRevocationApprovalSummary>
> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) => !['pool', 'now', 'limit'].includes(key),
) ||
!options.pool ||
typeof options.pool.query !== 'function' ||
typeof options.pool.connect !== 'function' ||
(options.now !== undefined && typeof options.now !== 'function')
) {
throw new TypeError(
'publisher revocation approval consumer options are invalid',
);
}
const limit =
options.limit ?? CLUSTER_PLUGIN_PACKAGE_PUBLISHER_APPROVAL_BATCH_LIMIT;
if (!Number.isSafeInteger(limit) || limit < 1 || limit > 64) {
throw new TypeError(
'publisher revocation approval consumer limit is invalid',
);
}
const now = options.now ?? Date.now;
const observedAtMs = now();
if (!Number.isSafeInteger(observedAtMs) || observedAtMs < 0) {
throw new TypeError(
'publisher revocation approval consumer clock is invalid',
);
}
const proposals =
new PostgresPluginPackagePublisherRevocationProposalRepository(
options.pool,
);
const approvals = new PostgresApprovalRequestRepository(options.pool);
const policy = new ProjectPolicyEngine(
new PostgresProjectPolicyRepository(options.pool),
);
const requests = await proposals.listApprovedRequests(limit);
let consumed = 0;
let existing = 0;
let expired = 0;
let blocked = 0;
for (const request of requests) {
if (observedAtMs >= request.expiresAtMs) {
expired += 1;
continue;
}
const decision = await policy.decide({
subject: request.requestedBy,
projectId: request.projectId,
permission: 'package.manage',
});
if (
decision.fence === null ||
(decision.effect !== 'allow' &&
decision.effect !== 'require_approval')
) {
blocked += 1;
continue;
}
const consumptionId = stableId(
'pprc',
'qinglong/plugin-package-publisher-revocation-consumption@v1',
request.id,
);
const dispatchId = stableId(
'pprd',
'qinglong/plugin-package-publisher-revocation-dispatch@v1',
request.id,
);
const result = await approvals.consume({
requestId: request.id,
expectedVersion: request.version,
consumptionId,
dispatchId,
action: request.action,
requestedBy: request.requestedBy,
consumedBy: {
type: 'system',
id: 'cluster_package_executor',
},
consumedAtMs: observedAtMs,
authorizationFence: decision.fence,
audit: {
eventId: stableAuditEventId(request.id),
requestId: request.id,
operationId: 'approval.consume',
projectId: request.projectId,
subject: {
type: 'system',
id: 'cluster_package_executor',
},
authenticationId: 'cluster-package-executor',
outcome: 'allowed',
reasons: ['publisher_revocation_execution'],
fence: decision.fence,
occurredAtMs: observedAtMs,
},
});
if (result.status === 'consumed') consumed += 1;
else existing += 1;
}
return Object.freeze({
scanned: requests.length,
consumed,
existing,
expired,
blocked,
});
}
@@ -0,0 +1,160 @@
// Cluster Plugin Package publisher boundary; keep Approved Action authority explicit.
import {
type ApprovedActionHandler,
type ApprovedActionHandlerExecutionContext,
type ApprovedActionHandlerInspection,
type ApprovedActionHandlerResult,
} from '@qinglong/runtime-core/approved-action-dispatcher';
import {
PLUGIN_PACKAGE_PUBLISHER_REVOCATION_ACTION_TYPE,
PluginPackagePublisherRevocationProposalBindingConflictError,
normalizePluginPackagePublisherRevocationProposal,
resolvePluginPackagePublisherRevocationProposal,
type PluginPackagePublisherRevocationProposalRepository,
} from '@qinglong/runtime-core/plugin-package-publisher-revocation-proposal';
import type {
PluginPackagePublisherRevocationReceipt,
} from '@qinglong/runtime-core/plugin-package-publisher-provenance';
export interface ClusterPluginPackagePublisherRevocationExecutionResult {
readonly safeToAdmit: boolean;
readonly receiptDigest: string;
readonly impactDigest: string;
}
export interface ClusterPluginPackagePublisherRevocationExecutionPort {
run(
receipt: Readonly<PluginPackagePublisherRevocationReceipt>,
): Promise<
Readonly<ClusterPluginPackagePublisherRevocationExecutionResult>
>;
}
export class ClusterPluginPackagePublisherRevocationApprovedActionHandler
implements ApprovedActionHandler
{
readonly actionType = PLUGIN_PACKAGE_PUBLISHER_REVOCATION_ACTION_TYPE;
constructor(
readonly proposals: PluginPackagePublisherRevocationProposalRepository,
readonly revocations: ClusterPluginPackagePublisherRevocationExecutionPort,
) {
if (
!proposals ||
typeof proposals.findProposalByActionRef !== 'function' ||
!revocations ||
typeof revocations.run !== 'function'
) {
throw new TypeError(
'publisher revocation Approved Action authority is invalid',
);
}
}
async inspect(
dispatch: ApprovedActionHandlerExecutionContext['dispatch'],
): Promise<ApprovedActionHandlerInspection> {
let proposal;
try {
proposal = await this.proposals.findProposalByActionRef(
dispatch.action.actionRef,
);
} catch {
return Object.freeze({
status: 'retry',
resultCode: 'publisher_revocation_proposal_unavailable',
});
}
if (!proposal) {
return Object.freeze({
status: 'blocked',
resultCode: 'publisher_revocation_proposal_missing',
});
}
try {
const normalized =
normalizePluginPackagePublisherRevocationProposal(proposal);
resolvePluginPackagePublisherRevocationProposal(
normalized,
dispatch,
dispatch.createdAtMs,
);
return Object.freeze({
status: 'ready',
actionDigest: normalized.actionDigest,
});
} catch {
return Object.freeze({
status: 'blocked',
resultCode: 'publisher_revocation_proposal_rejected',
});
}
}
async execute(
context: Readonly<ApprovedActionHandlerExecutionContext>,
): Promise<Readonly<ApprovedActionHandlerResult>> {
const startedAtMs = context.execution.startedAtMs;
if (
context.execution.status !== 'executing' ||
startedAtMs === null ||
context.execution.leaseOwner !== context.fence.owner ||
context.execution.leaseToken !== context.fence.leaseToken ||
context.execution.version !== context.fence.version
) {
return Object.freeze({
outcome: 'failed',
resultCode: 'publisher_revocation_execution_rejected',
});
}
const proposal = await this.proposals.findProposalByActionRef(
context.dispatch.action.actionRef,
);
if (!proposal) {
return Object.freeze({
outcome: 'failed',
resultCode: 'publisher_revocation_proposal_missing',
});
}
let receipt;
try {
receipt = resolvePluginPackagePublisherRevocationProposal(
proposal,
context.dispatch,
startedAtMs,
);
} catch (error) {
if (
error instanceof
PluginPackagePublisherRevocationProposalBindingConflictError
) {
return Object.freeze({
outcome: 'failed',
resultCode: 'publisher_revocation_proposal_rejected',
});
}
throw error;
}
const result = await this.revocations.run(receipt);
if (!result.safeToAdmit) {
return Object.freeze({
outcome: 'indeterminate',
resultCode: 'publisher_revocation_convergence_incomplete',
});
}
if (
result.receiptDigest !== receipt.receiptDigest ||
!/^[0-9a-f]{64}$/.test(result.impactDigest)
) {
return Object.freeze({
outcome: 'failed',
resultCode: 'publisher_revocation_result_rejected',
});
}
return Object.freeze({
outcome: 'succeeded',
resultCode: 'publisher_revocation_converged',
resultDigest: result.impactDigest,
});
}
}
@@ -0,0 +1,178 @@
// Cluster Plugin Package publisher boundary; keep transition approval authority explicit.
import { createHash } from 'node:crypto';
import { PostgresApprovalRequestRepository } from '@qinglong/cluster-postgres/approved-action';
import {
PostgresPluginPackagePublisherTrustTransitionProposalRepository,
} from '@qinglong/cluster-postgres/package-executor';
import { PostgresProjectPolicyRepository } from '@qinglong/cluster-postgres/project-policy';
import type { PostgresPool } from '@qinglong/runtime-core';
import { ProjectPolicyEngine } from '@qinglong/runtime-core/project-policy';
export const CLUSTER_PLUGIN_PACKAGE_PUBLISHER_TRUST_TRANSITION_APPROVAL_BATCH_LIMIT =
16;
export interface ConsumeClusterPluginPackagePublisherTrustTransitionApprovalsOptions {
readonly pool: PostgresPool;
readonly now?: () => number;
readonly limit?: number;
}
export interface ClusterPluginPackagePublisherTrustTransitionApprovalSummary {
readonly scanned: number;
readonly consumed: number;
readonly existing: number;
readonly expired: number;
readonly blocked: number;
}
function stableDigest(domain: string, value: string): string {
return createHash('sha256')
.update(domain)
.update('\0')
.update(value)
.digest('hex');
}
function stableId(prefix: string, domain: string, value: string): string {
return `${prefix}-${stableDigest(domain, value)}`;
}
function stableAuditEventId(requestId: string): string {
const bytes = Buffer.from(
stableDigest(
'qinglong/plugin-package-publisher-trust-transition-consume-audit@v1',
requestId,
),
'hex',
);
bytes[6] = (bytes[6]! & 0x0f) | 0x40;
bytes[8] = (bytes[8]! & 0x3f) | 0x80;
const hex = bytes.toString('hex');
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(
12,
16,
)}-${hex.slice(16, 20)}-${hex.slice(20, 32)}`;
}
export async function consumeClusterPluginPackagePublisherTrustTransitionApprovals(
options: ConsumeClusterPluginPackagePublisherTrustTransitionApprovalsOptions,
): Promise<
Readonly<ClusterPluginPackagePublisherTrustTransitionApprovalSummary>
> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) => !['pool', 'now', 'limit'].includes(key),
) ||
!options.pool ||
typeof options.pool.query !== 'function' ||
typeof options.pool.connect !== 'function' ||
(options.now !== undefined && typeof options.now !== 'function')
) {
throw new TypeError(
'publisher trust transition approval consumer options are invalid',
);
}
const limit =
options.limit ??
CLUSTER_PLUGIN_PACKAGE_PUBLISHER_TRUST_TRANSITION_APPROVAL_BATCH_LIMIT;
if (!Number.isSafeInteger(limit) || limit < 1 || limit > 64) {
throw new TypeError(
'publisher trust transition approval consumer limit is invalid',
);
}
const now = options.now ?? Date.now;
const observedAtMs = now();
if (!Number.isSafeInteger(observedAtMs) || observedAtMs < 0) {
throw new TypeError(
'publisher trust transition approval consumer clock is invalid',
);
}
const proposals =
new PostgresPluginPackagePublisherTrustTransitionProposalRepository(
options.pool,
);
const approvals = new PostgresApprovalRequestRepository(options.pool);
const policy = new ProjectPolicyEngine(
new PostgresProjectPolicyRepository(options.pool),
);
const requests = await proposals.listApprovedRequests(limit);
let consumed = 0;
let existing = 0;
let expired = 0;
let blocked = 0;
for (const request of requests) {
if (observedAtMs >= request.expiresAtMs) {
expired += 1;
continue;
}
if (request.decisionMode !== 'separation_of_duty') {
blocked += 1;
continue;
}
const decision = await policy.decide({
subject: request.requestedBy,
projectId: request.projectId,
permission: 'package.manage',
});
if (
decision.fence === null ||
(decision.effect !== 'allow' &&
decision.effect !== 'require_approval')
) {
blocked += 1;
continue;
}
const consumptionId = stableId(
'ppttc',
'qinglong/plugin-package-publisher-trust-transition-consumption@v1',
request.id,
);
const dispatchId = stableId(
'ppttd',
'qinglong/plugin-package-publisher-trust-transition-dispatch@v1',
request.id,
);
const result = await approvals.consume({
requestId: request.id,
expectedVersion: request.version,
consumptionId,
dispatchId,
action: request.action,
requestedBy: request.requestedBy,
consumedBy: {
type: 'system',
id: 'cluster_package_executor',
},
consumedAtMs: observedAtMs,
authorizationFence: decision.fence,
audit: {
eventId: stableAuditEventId(request.id),
requestId: request.id,
operationId: 'approval.consume',
projectId: request.projectId,
subject: {
type: 'system',
id: 'cluster_package_executor',
},
authenticationId: 'cluster-package-executor',
outcome: 'allowed',
reasons: ['publisher_trust_transition_execution'],
fence: decision.fence,
occurredAtMs: observedAtMs,
},
});
if (result.status === 'consumed') consumed += 1;
else existing += 1;
}
return Object.freeze({
scanned: requests.length,
consumed,
existing,
expired,
blocked,
});
}
@@ -0,0 +1,167 @@
// Cluster Plugin Package publisher boundary; keep transition execution authority explicit.
import {
type ApprovedActionHandler,
type ApprovedActionHandlerExecutionContext,
type ApprovedActionHandlerInspection,
type ApprovedActionHandlerResult,
} from '@qinglong/runtime-core/approved-action-dispatcher';
import {
PLUGIN_PACKAGE_PUBLISHER_TRUST_TRANSITION_ACTION_TYPES,
PluginPackagePublisherTrustTransitionBindingConflictError,
PluginPackagePublisherTrustTransitionConflictError,
normalizePluginPackagePublisherTrustTransitionProposal,
resolvePluginPackagePublisherTrustTransitionProposal,
type PluginPackagePublisherTrustTransitionMode,
type PluginPackagePublisherTrustTransitionProposalRepository,
type PluginPackagePublisherTrustTransitionReceipt,
} from '@qinglong/runtime-core/plugin-package-publisher-trust-transition-proposal';
export interface ClusterPluginPackagePublisherTrustTransitionExecutionResult {
readonly status: 'created' | 'existing';
readonly receipt: Readonly<PluginPackagePublisherTrustTransitionReceipt>;
readonly head: Readonly<{
generation: number;
effectiveTrustDigest: string;
}>;
}
export interface ClusterPluginPackagePublisherTrustTransitionExecutionPort {
applyApprovedTransition(
input: Readonly<{
dispatch: ApprovedActionHandlerExecutionContext['dispatch'];
executedAtMs: number;
}>,
): Promise<
Readonly<ClusterPluginPackagePublisherTrustTransitionExecutionResult>
>;
}
export class ClusterPluginPackagePublisherTrustTransitionApprovedActionHandler
implements ApprovedActionHandler
{
readonly actionType:
(typeof PLUGIN_PACKAGE_PUBLISHER_TRUST_TRANSITION_ACTION_TYPES)[PluginPackagePublisherTrustTransitionMode];
constructor(
readonly mode: PluginPackagePublisherTrustTransitionMode,
readonly proposals: PluginPackagePublisherTrustTransitionProposalRepository,
readonly transitions: ClusterPluginPackagePublisherTrustTransitionExecutionPort,
) {
this.actionType =
PLUGIN_PACKAGE_PUBLISHER_TRUST_TRANSITION_ACTION_TYPES[mode];
if (
(mode !== 'overlap_add' && mode !== 'safe_retire') ||
!proposals ||
typeof proposals.findProposalByActionRef !== 'function' ||
!transitions ||
typeof transitions.applyApprovedTransition !== 'function'
) {
throw new TypeError(
'publisher trust transition Approved Action authority is invalid',
);
}
}
async inspect(
dispatch: ApprovedActionHandlerExecutionContext['dispatch'],
): Promise<ApprovedActionHandlerInspection> {
let proposal;
try {
proposal = await this.proposals.findProposalByActionRef(
dispatch.action.actionRef,
);
} catch {
return Object.freeze({
status: 'retry',
resultCode: 'publisher_trust_transition_proposal_unavailable',
});
}
if (!proposal) {
return Object.freeze({
status: 'blocked',
resultCode: 'publisher_trust_transition_proposal_missing',
});
}
try {
const normalized =
normalizePluginPackagePublisherTrustTransitionProposal(proposal);
if (
normalized.actionInput.mode !== this.mode ||
normalized.actionType !== this.actionType
) {
throw new PluginPackagePublisherTrustTransitionBindingConflictError();
}
resolvePluginPackagePublisherTrustTransitionProposal(
normalized,
dispatch,
dispatch.createdAtMs,
this.mode === 'safe_retire' ? 0 : null,
);
return Object.freeze({
status: 'ready',
actionDigest: normalized.actionDigest,
});
} catch {
return Object.freeze({
status: 'blocked',
resultCode: 'publisher_trust_transition_proposal_rejected',
});
}
}
async execute(
context: Readonly<ApprovedActionHandlerExecutionContext>,
): Promise<Readonly<ApprovedActionHandlerResult>> {
const startedAtMs = context.execution.startedAtMs;
if (
context.execution.status !== 'executing' ||
startedAtMs === null ||
context.execution.leaseOwner !== context.fence.owner ||
context.execution.leaseToken !== context.fence.leaseToken ||
context.execution.version !== context.fence.version
) {
return Object.freeze({
outcome: 'failed',
resultCode: 'publisher_trust_transition_execution_rejected',
});
}
try {
const result = await this.transitions.applyApprovedTransition({
dispatch: context.dispatch,
executedAtMs: startedAtMs,
});
if (
result.receipt.mode !== this.mode ||
result.receipt.mutationId !== context.dispatch.id ||
result.head.generation !== result.receipt.currentGeneration ||
result.head.effectiveTrustDigest !==
result.receipt.currentTrustDigest
) {
return Object.freeze({
outcome: 'failed',
resultCode: 'publisher_trust_transition_result_rejected',
});
}
return Object.freeze({
outcome: 'succeeded',
resultCode:
this.mode === 'overlap_add'
? 'publisher_trust_overlap_added'
: 'publisher_trust_key_retired',
resultDigest: result.receipt.receiptDigest,
});
} catch (error) {
if (
error instanceof
PluginPackagePublisherTrustTransitionBindingConflictError ||
error instanceof PluginPackagePublisherTrustTransitionConflictError
) {
return Object.freeze({
outcome: 'failed',
resultCode: 'publisher_trust_transition_conflict',
});
}
throw error;
}
}
}
@@ -0,0 +1,565 @@
// Cluster Plugin Package recovery boundary; keep Kubernetes activation authority explicit.
import { createHash } from 'node:crypto';
import {
PluginPackageActivationConflictError,
PluginPackageActivationUnavailableError,
normalizePluginPackageActivationIntent,
type PluginPackageActivationIntent,
type PluginPackageActivationObservation,
type PluginPackageActivationPublisher,
} from '@qinglong/runtime-core/plugin-package-activation';
import type {
PluginPackageResourceGeneration,
PluginPackageResourceGenerationSource,
} from '@qinglong/runtime-core/plugin-package-resource-generation';
import {
createPluginPackageActivationReceipt,
normalizePluginPackageActivationReceipt,
type PluginPackageActivationReceipt,
} from '@qinglong/runtime-core/plugin-package-install';
const ACTIVE_POINTER_SCHEMA =
'qinglong/plugin-package-kubernetes-active-pointer@v2';
const ACTIVE_POINTER_KEY = 'active.json';
const MANAGED_BY_LABEL = 'app.kubernetes.io/managed-by';
const MANAGED_BY_VALUE = 'qinglong3';
const ACTIVE_LABEL = 'qinglong.io/plugin-package-active';
const TARGET_LABEL = 'qinglong.io/plugin-package-target';
const INTENT_ANNOTATION = 'qinglong.io/plugin-package-intent';
const FIELD_MANAGER = 'qinglong-plugin-package-activation';
const MAX_ACTIVE_POINTER_BYTES = 512 * 1024;
const DNS_LABEL = /^[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?$/;
const SAFE_IDENTITY = /^[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}$/;
const RESOURCE_ID = /^[A-Za-z0-9][A-Za-z0-9._:/+=-]{0,511}$/;
const DIGEST = /^[0-9a-f]{64}$/;
const TARGET_DIGEST_DOMAIN = Buffer.from(
'qinglong/plugin-package-kubernetes-target@v1\0',
'utf8',
);
export interface ClusterPluginPackageStageEvidence {
readonly lockDigest: string;
readonly stageRef: string;
readonly stageReceiptDigest: string;
readonly stageEvidenceDigest: string;
readonly contentDigest: string;
}
export interface ClusterPluginPackageStageEvidenceVerifier {
verify(
intent: Readonly<PluginPackageActivationIntent>,
): Promise<Readonly<ClusterPluginPackageStageEvidence>>;
}
export interface PluginPackageKubernetesActivationPublisherOptions {
/** Stable operator-reviewed identity for one Kubernetes API cluster. */
readonly clusterIdentity: string;
readonly namespace: string;
/** Explicit authoritative clock called only for a new publication attempt. */
readonly now: () => number | Promise<number>;
}
export interface PluginPackageKubernetesConfigMap {
readonly apiVersion?: string;
readonly kind?: string;
readonly immutable?: boolean;
readonly data?: Readonly<Record<string, string>>;
readonly binaryData?: Readonly<Record<string, string>>;
readonly metadata?: Readonly<{
name?: string;
namespace?: string;
uid?: string;
resourceVersion?: string;
deletionTimestamp?: Date;
finalizers?: readonly string[];
ownerReferences?: readonly Readonly<Record<string, unknown>>[];
labels?: Readonly<Record<string, string>>;
annotations?: Readonly<Record<string, string>>;
}>;
}
interface ConfigMapWrite extends PluginPackageKubernetesConfigMap {
readonly metadata: NonNullable<PluginPackageKubernetesConfigMap['metadata']>;
readonly data: Readonly<Record<string, string>>;
}
export interface PluginPackageKubernetesConfigMapApi {
readNamespacedConfigMap(
request: Readonly<{
name: string;
namespace: string;
}>,
): Promise<PluginPackageKubernetesConfigMap>;
createNamespacedConfigMap(
request: Readonly<{
namespace: string;
body: ConfigMapWrite;
fieldManager: string;
fieldValidation: 'Strict';
}>,
): Promise<PluginPackageKubernetesConfigMap>;
replaceNamespacedConfigMap(
request: Readonly<{
name: string;
namespace: string;
body: ConfigMapWrite;
fieldManager: string;
fieldValidation: 'Strict';
}>,
): Promise<PluginPackageKubernetesConfigMap>;
}
interface ActivePointer {
readonly schema: typeof ACTIVE_POINTER_SCHEMA;
readonly clusterIdentityDigest: string;
readonly intent: Readonly<PluginPackageActivationIntent>;
readonly receipt: Readonly<PluginPackageActivationReceipt>;
}
interface StoredPointer extends ActivePointer {
readonly resourceVersion: string;
}
function apiStatus(error: unknown): number | null {
if (!error || typeof error !== 'object') return null;
if ('code' in error && typeof error.code === 'number') return error.code;
if (
'response' in error &&
error.response &&
typeof error.response === 'object' &&
'statusCode' in error.response &&
typeof error.response.statusCode === 'number'
) {
return error.response.statusCode;
}
return null;
}
function preserveDomainError(error: unknown): never {
if (
error instanceof PluginPackageActivationConflictError ||
error instanceof PluginPackageActivationUnavailableError
) {
throw error;
}
throw new PluginPackageActivationUnavailableError({
cause: error instanceof Error ? error : undefined,
});
}
function dataRecord(value: unknown): Record<string, unknown> {
if (
!value ||
typeof value !== 'object' ||
Array.isArray(value) ||
(Object.getPrototypeOf(value) !== Object.prototype &&
Object.getPrototypeOf(value) !== null)
) {
throw new PluginPackageActivationConflictError();
}
const descriptors = Object.getOwnPropertyDescriptors(value);
if (
Object.values(descriptors).some(
(descriptor) =>
descriptor.get !== undefined ||
descriptor.set !== undefined ||
descriptor.enumerable !== true,
)
) {
throw new PluginPackageActivationConflictError();
}
return value as Record<string, unknown>;
}
function exactKeys(value: object, expected: readonly string[]): void {
const actual = Object.keys(value).sort();
const canonical = [...expected].sort();
if (
actual.length !== canonical.length ||
actual.some((key, index) => key !== canonical[index])
) {
throw new PluginPackageActivationConflictError();
}
}
function boundedResourceId(value: unknown): string {
if (typeof value !== 'string' || !RESOURCE_ID.test(value)) {
throw new PluginPackageActivationUnavailableError();
}
return value;
}
function normalizeIntent(
value: Readonly<PluginPackageActivationIntent>,
): Readonly<PluginPackageActivationIntent> {
try {
return normalizePluginPackageActivationIntent(value);
} catch {
throw new PluginPackageActivationConflictError();
}
}
function same(left: unknown, right: unknown): boolean {
return JSON.stringify(left) === JSON.stringify(right);
}
/**
* Short-lived Kubernetes ConfigMap publisher. It owns no timer, watcher,
* database connection or cache; every replacement is resourceVersion fenced.
*/
export class PluginPackageKubernetesActivationPublisher
implements
PluginPackageActivationPublisher,
PluginPackageResourceGenerationSource
{
readonly #clusterIdentityDigest: string;
constructor(
private readonly api: PluginPackageKubernetesConfigMapApi,
private readonly stageEvidence: ClusterPluginPackageStageEvidenceVerifier,
private readonly options: PluginPackageKubernetesActivationPublisherOptions,
) {
if (
!api ||
typeof api.readNamespacedConfigMap !== 'function' ||
typeof api.createNamespacedConfigMap !== 'function' ||
typeof api.replaceNamespacedConfigMap !== 'function'
) {
throw new TypeError('Plugin Package Kubernetes ConfigMap API is invalid');
}
if (!stageEvidence || typeof stageEvidence.verify !== 'function') {
throw new TypeError(
'Plugin Package cluster stage evidence verifier is invalid',
);
}
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).sort().join(',') !==
'clusterIdentity,namespace,now' ||
!SAFE_IDENTITY.test(options.clusterIdentity) ||
!DNS_LABEL.test(options.namespace) ||
typeof options.now !== 'function'
) {
throw new TypeError(
'Plugin Package Kubernetes activation options are invalid',
);
}
this.#clusterIdentityDigest = createHash('sha256')
.update('qinglong/plugin-package-kubernetes-cluster@v1\0', 'utf8')
.update(options.clusterIdentity, 'utf8')
.digest('hex');
}
#targetDigest(
identity: Readonly<
Pick<PluginPackageActivationIntent, 'projectId' | 'packageName'>
>,
): string {
return createHash('sha256')
.update(TARGET_DIGEST_DOMAIN)
.update(this.#clusterIdentityDigest, 'utf8')
.update('\0', 'utf8')
.update(this.options.namespace, 'utf8')
.update('\0', 'utf8')
.update(identity.projectId, 'utf8')
.update('\0', 'utf8')
.update(identity.packageName, 'utf8')
.digest('hex');
}
#name(
identity: Readonly<
Pick<PluginPackageActivationIntent, 'projectId' | 'packageName'>
>,
): string {
return `ql3p-${this.#targetDigest(identity).slice(0, 52)}`;
}
async #verifyStage(
intent: Readonly<PluginPackageActivationIntent>,
): Promise<void> {
let value: unknown;
try {
value = await this.stageEvidence.verify(intent);
} catch (error) {
return preserveDomainError(error);
}
const evidence = dataRecord(value);
exactKeys(evidence, [
'lockDigest',
'stageRef',
'stageReceiptDigest',
'stageEvidenceDigest',
'contentDigest',
]);
if (
evidence.lockDigest !== intent.lockDigest ||
evidence.stageRef !== intent.stageRef ||
evidence.stageReceiptDigest !== intent.stageReceiptDigest ||
evidence.stageEvidenceDigest !== intent.stageEvidenceDigest ||
evidence.contentDigest !== intent.contentDigest
) {
throw new PluginPackageActivationConflictError();
}
}
#parsePointer(
configMap: PluginPackageKubernetesConfigMap,
expectedName: string,
): Readonly<StoredPointer> {
try {
const metadata = configMap?.metadata;
if (
configMap.apiVersion !== 'v1' ||
configMap.kind !== 'ConfigMap' ||
configMap.immutable === true ||
configMap.binaryData !== undefined ||
!metadata ||
metadata.name !== expectedName ||
metadata.namespace !== this.options.namespace ||
metadata.deletionTimestamp !== undefined ||
(metadata.finalizers?.length ?? 0) !== 0 ||
(metadata.ownerReferences?.length ?? 0) !== 0 ||
!configMap.data
) {
throw new PluginPackageActivationConflictError();
}
const labels = dataRecord(metadata.labels);
exactKeys(labels, [MANAGED_BY_LABEL, ACTIVE_LABEL, TARGET_LABEL]);
const annotations = dataRecord(metadata.annotations);
exactKeys(annotations, [INTENT_ANNOTATION]);
const data = dataRecord(configMap.data);
exactKeys(data, [ACTIVE_POINTER_KEY]);
const serialized = data[ACTIVE_POINTER_KEY];
if (
labels[MANAGED_BY_LABEL] !== MANAGED_BY_VALUE ||
labels[ACTIVE_LABEL] !== 'v2' ||
typeof serialized !== 'string' ||
Buffer.byteLength(serialized, 'utf8') > MAX_ACTIVE_POINTER_BYTES
) {
throw new PluginPackageActivationConflictError();
}
const pointer = dataRecord(JSON.parse(serialized));
exactKeys(pointer, [
'schema',
'clusterIdentityDigest',
'intent',
'receipt',
]);
const intent = normalizeIntent(
pointer.intent as PluginPackageActivationIntent,
);
const receipt = normalizePluginPackageActivationReceipt(pointer.receipt);
const normalized: ActivePointer = Object.freeze({
schema: ACTIVE_POINTER_SCHEMA,
clusterIdentityDigest: this.#clusterIdentityDigest,
intent,
receipt,
});
if (
pointer.schema !== ACTIVE_POINTER_SCHEMA ||
pointer.clusterIdentityDigest !== this.#clusterIdentityDigest ||
this.#name(intent) !== expectedName ||
labels[TARGET_LABEL] !==
Buffer.from(this.#targetDigest(intent), 'hex').toString(
'base64url',
) ||
annotations[INTENT_ANNOTATION] !== intent.intentDigest ||
receipt.intentDigest !== intent.intentDigest ||
receipt.generation !== intent.targetGeneration ||
receipt.contentDigest !== intent.contentDigest ||
`${JSON.stringify(normalized)}\n` !== serialized
) {
throw new PluginPackageActivationConflictError();
}
boundedResourceId(metadata.uid);
return Object.freeze({
...normalized,
resourceVersion: boundedResourceId(metadata.resourceVersion),
});
} catch (error) {
return preserveDomainError(error);
}
}
async #optionalPointer(
identity: Readonly<
Pick<PluginPackageActivationIntent, 'projectId' | 'packageName'>
>,
): Promise<Readonly<StoredPointer> | null> {
const name = this.#name(identity);
try {
return this.#parsePointer(
await this.api.readNamespacedConfigMap({
name,
namespace: this.options.namespace,
}),
name,
);
} catch (error) {
if (apiStatus(error) === 404) return null;
return preserveDomainError(error);
}
}
async #observe(
intent: Readonly<PluginPackageActivationIntent>,
): Promise<Readonly<PluginPackageActivationObservation>> {
await this.#verifyStage(intent);
const pointer = await this.#optionalPointer(intent);
if (!pointer) {
if (intent.previousActiveLockDigest !== null) {
throw new PluginPackageActivationConflictError();
}
return Object.freeze({ status: 'not_published' });
}
if (same(pointer.intent, intent)) {
return Object.freeze({ status: 'published', receipt: pointer.receipt });
}
if (
pointer.intent.projectId === intent.projectId &&
pointer.intent.packageName === intent.packageName &&
pointer.intent.lockDigest === intent.previousActiveLockDigest
) {
return Object.freeze({ status: 'not_published' });
}
throw new PluginPackageActivationConflictError();
}
#body(
intent: Readonly<PluginPackageActivationIntent>,
receipt: Readonly<PluginPackageActivationReceipt>,
current: Readonly<StoredPointer> | null,
): ConfigMapWrite {
const targetDigest = this.#targetDigest(intent);
const pointer: Readonly<ActivePointer> = Object.freeze({
schema: ACTIVE_POINTER_SCHEMA,
clusterIdentityDigest: this.#clusterIdentityDigest,
intent,
receipt,
});
const serialized = `${JSON.stringify(pointer)}\n`;
if (Buffer.byteLength(serialized, 'utf8') > MAX_ACTIVE_POINTER_BYTES) {
throw new PluginPackageActivationUnavailableError();
}
return Object.freeze({
apiVersion: 'v1',
kind: 'ConfigMap',
immutable: false,
metadata: Object.freeze({
name: this.#name(intent),
namespace: this.options.namespace,
...(current ? { resourceVersion: current.resourceVersion } : {}),
labels: Object.freeze({
[MANAGED_BY_LABEL]: MANAGED_BY_VALUE,
[ACTIVE_LABEL]: 'v2',
[TARGET_LABEL]: Buffer.from(targetDigest, 'hex').toString(
'base64url',
),
}),
annotations: Object.freeze({
[INTENT_ANNOTATION]: intent.intentDigest,
}),
}),
data: Object.freeze({ [ACTIVE_POINTER_KEY]: serialized }),
});
}
async inspect(
value: Readonly<PluginPackageActivationIntent>,
): Promise<Readonly<PluginPackageActivationObservation>> {
try {
return await this.#observe(normalizeIntent(value));
} catch (error) {
return preserveDomainError(error);
}
}
async findActiveResourceGeneration(
projectId: string,
packageName: string,
): Promise<Readonly<PluginPackageResourceGeneration> | null> {
if (
typeof projectId !== 'string' ||
!/^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/.test(projectId) ||
typeof packageName !== 'string' ||
!DNS_LABEL.test(packageName)
) {
throw new TypeError('Plugin Package active resource identity is invalid');
}
try {
return (
(await this.#optionalPointer(Object.freeze({ projectId, packageName })))
?.intent.resourceGeneration ?? null
);
} catch (error) {
return preserveDomainError(error);
}
}
async publish(
value: Readonly<PluginPackageActivationIntent>,
): Promise<Readonly<PluginPackageActivationReceipt>> {
const intent = normalizeIntent(value);
try {
const first = await this.#observe(intent);
if (first.status === 'published') return first.receipt;
const current = await this.#optionalPointer(intent);
if (current && same(current.intent, intent)) return current.receipt;
if (
(!current && intent.previousActiveLockDigest !== null) ||
(current &&
(current.intent.projectId !== intent.projectId ||
current.intent.packageName !== intent.packageName ||
current.intent.lockDigest !== intent.previousActiveLockDigest))
) {
throw new PluginPackageActivationConflictError();
}
const activatedAtMs = await this.options.now();
if (!Number.isSafeInteger(activatedAtMs) || activatedAtMs < 0) {
throw new PluginPackageActivationUnavailableError();
}
const receipt = createPluginPackageActivationReceipt({
activationRef: `k8s-configmap:${this.#targetDigest(intent)}`,
intentDigest: intent.intentDigest,
generation: intent.targetGeneration,
contentDigest: intent.contentDigest,
activatedAtMs,
});
const body = this.#body(intent, receipt, current);
try {
if (current) {
await this.api.replaceNamespacedConfigMap({
name: this.#name(intent),
namespace: this.options.namespace,
body,
fieldManager: FIELD_MANAGER,
fieldValidation: 'Strict',
});
} else {
await this.api.createNamespacedConfigMap({
namespace: this.options.namespace,
body,
fieldManager: FIELD_MANAGER,
fieldValidation: 'Strict',
});
}
} catch (error) {
if (apiStatus(error) !== 409) return preserveDomainError(error);
const winner = await this.#observe(intent);
if (winner.status === 'published') return winner.receipt;
throw new PluginPackageActivationConflictError();
}
const final = await this.#observe(intent);
if (final.status !== 'published') {
throw new PluginPackageActivationUnavailableError();
}
return final.receipt;
} catch (error) {
return preserveDomainError(error);
}
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,450 @@
// Cluster Plugin Package recovery boundary; keep recovery coordination authority explicit.
import type {
OpenPostgresDatabase,
PostgresDatabaseResource,
PostgresPool,
} from '@qinglong/runtime-core';
import {
MAX_PLUGIN_PACKAGE_INSTALL_RECOVERY_PAGE_SIZE,
assertPluginPackageInstallMatchesLock,
} from '@qinglong/runtime-core/plugin-package-install';
import {
MAX_PLUGIN_PACKAGE_RECOVERY_PAGES,
PluginPackageRecoveryCoordinator,
type PluginPackageRecoveryCycleResult,
} from '@qinglong/runtime-core/plugin-package-recovery';
import {
PluginPackageAutomationPublicationCoordinator,
PluginPackageAutomationPublicationRecoveryCoordinator,
type PluginPackageAutomationPublicationRecoveryCycleResult,
} from '@qinglong/runtime-core/plugin-package-automation-publication';
import type { PluginPackageResourceByteSource } from '@qinglong/runtime-core/plugin-package-resource-materialization';
import {
MAX_PLUGIN_PACKAGE_TASK_PUBLICATION_RECOVERY_PAGES,
MAX_PLUGIN_PACKAGE_TASK_PUBLICATION_RECOVERY_PAGE_SIZE,
PluginPackageTaskPublicationCoordinator,
PluginPackageTaskPublicationRecoveryCoordinator,
type PluginPackageTaskPublicationRecoveryCycleResult,
} from '@qinglong/runtime-core/plugin-package-task-publication';
import {
MAX_PROJECT_TOOL_SNAPSHOT_RECOVERY_PAGES,
MAX_PROJECT_TOOL_SNAPSHOT_RECOVERY_PAGE_SIZE,
MAX_PROJECT_TOOL_SNAPSHOT_SOURCE_PAGE_SIZE,
ProjectToolDefinitionSnapshotPublicationCoordinator,
ProjectToolDefinitionSnapshotRecoveryCoordinator,
type ProjectToolDefinitionSnapshotRecoveryCycleResult,
} from '@qinglong/runtime-core/project-tool-definition-snapshot';
import { createBuiltInTaskSpecSemanticRegistry } from '@qinglong/runtime-core/task-spec-semantic';
import {
assertPostgresPackageExecutorSchemaReady,
PostgresPluginPackageAutomationPublicationRepository,
PostgresPluginPackageMaterializedRevisionRepository,
PostgresPluginPackagePublisherProvenanceRepository,
PostgresPluginPackageTaskReconciliationRepository,
PostgresProjectToolDefinitionSnapshotRepository,
type PostgresSchemaReadinessReport,
} from '@qinglong/cluster-postgres/package-executor';
import { PostgresPluginPackageInstallRepository } from '@qinglong/cluster-postgres/plugin-package-install';
import {
PluginPackageKubernetesActivationPublisher,
type PluginPackageKubernetesConfigMapApi,
} from './pluginPackageKubernetesActivation';
import {
ClusterPluginPackageOciResourceByteSource,
ClusterPluginPackageOciStageAuthority,
clusterPluginPackageActivationEvidence,
pluginPackageStageVerificationFailure,
type ClusterPluginPackageStageAuthority,
} from './pluginPackageOciStage';
import {
ClusterPluginPackageProvenanceInstallRepository,
recoverClusterPluginPackagePublisherProvenance,
type ClusterPluginPackagePublisherProvenanceRecoveryResult,
} from '../publisher/pluginPackagePublisherProvenanceRecovery';
export interface ClusterPluginPackageRecoveryOptions {
readonly openDatabase: OpenPostgresDatabase;
readonly api: PluginPackageKubernetesConfigMapApi;
readonly stageAuthority?: ClusterPluginPackageStageAuthority;
readonly stageAuthorityFactory?: (
pool: PostgresPool,
) =>
| ClusterPluginPackageStageAuthority
| Promise<ClusterPluginPackageStageAuthority>;
readonly resourceByteSource?: PluginPackageResourceByteSource;
readonly trustAuthorityId: string;
readonly clusterIdentity: string;
readonly namespace: string;
readonly now: () => number | Promise<number>;
readonly pageSize?: number;
readonly maxPages?: number;
}
export interface ClusterPluginPackageRecoveryResult {
readonly evidence: PostgresSchemaReadinessReport;
readonly provenanceRecovery: Readonly<ClusterPluginPackagePublisherProvenanceRecoveryResult>;
readonly recovery: Readonly<PluginPackageRecoveryCycleResult>;
readonly taskPublicationRecovery: Readonly<PluginPackageTaskPublicationRecoveryCycleResult>;
readonly automationPublicationRecovery: Readonly<PluginPackageAutomationPublicationRecoveryCycleResult>;
readonly toolSnapshotRecovery: Readonly<ProjectToolDefinitionSnapshotRecoveryCycleResult>;
}
export class ClusterPluginPackageRecoveryRequiredError extends Error {
constructor(readonly recovery: Readonly<PluginPackageRecoveryCycleResult>) {
super('Cluster has unresolved Plugin Package recovery work');
this.name = 'ClusterPluginPackageRecoveryRequiredError';
}
}
export class ClusterPluginPackagePublisherProvenanceRecoveryRequiredError extends Error {
constructor(
readonly recovery: Readonly<ClusterPluginPackagePublisherProvenanceRecoveryResult>,
) {
super('Cluster has unresolved Plugin Package publisher provenance work');
this.name =
'ClusterPluginPackagePublisherProvenanceRecoveryRequiredError';
}
}
export class ClusterPluginPackageTaskPublicationRequiredError extends Error {
constructor(
readonly recovery: Readonly<PluginPackageTaskPublicationRecoveryCycleResult>,
) {
super('Cluster has unresolved Plugin Package Task publication work');
this.name = 'ClusterPluginPackageTaskPublicationRequiredError';
}
}
export class ClusterPluginPackageAutomationPublicationRequiredError extends Error {
constructor(
readonly recovery: Readonly<PluginPackageAutomationPublicationRecoveryCycleResult>,
) {
super(
'Cluster has unresolved Plugin Package Workflow/Prompt publication work',
);
this.name = 'ClusterPluginPackageAutomationPublicationRequiredError';
}
}
export class ClusterPluginPackageToolSnapshotRequiredError extends Error {
constructor(
readonly recovery: Readonly<ProjectToolDefinitionSnapshotRecoveryCycleResult>,
) {
super('Cluster has unresolved Plugin Package Tool snapshot work');
this.name = 'ClusterPluginPackageToolSnapshotRequiredError';
}
}
function assertOptions(options: ClusterPluginPackageRecoveryOptions): void {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
![
'openDatabase',
'api',
'stageAuthority',
'stageAuthorityFactory',
'resourceByteSource',
'trustAuthorityId',
'clusterIdentity',
'namespace',
'now',
'pageSize',
'maxPages',
].includes(key),
) ||
typeof options.openDatabase !== 'function' ||
(options.stageAuthority === undefined) ===
(options.stageAuthorityFactory === undefined) ||
(options.stageAuthorityFactory !== undefined &&
typeof options.stageAuthorityFactory !== 'function') ||
(options.resourceByteSource !== undefined &&
(!options.resourceByteSource ||
typeof options.resourceByteSource.open !== 'function')) ||
typeof options.trustAuthorityId !== 'string' ||
!/^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/.test(
options.trustAuthorityId,
) ||
typeof options.now !== 'function' ||
(options.pageSize !== undefined &&
(!Number.isSafeInteger(options.pageSize) ||
options.pageSize < 1 ||
options.pageSize >
Math.min(
MAX_PLUGIN_PACKAGE_INSTALL_RECOVERY_PAGE_SIZE,
MAX_PLUGIN_PACKAGE_TASK_PUBLICATION_RECOVERY_PAGE_SIZE,
MAX_PROJECT_TOOL_SNAPSHOT_RECOVERY_PAGE_SIZE,
))) ||
(options.maxPages !== undefined &&
(!Number.isSafeInteger(options.maxPages) ||
options.maxPages < 1 ||
options.maxPages >
Math.min(
MAX_PLUGIN_PACKAGE_RECOVERY_PAGES,
MAX_PLUGIN_PACKAGE_TASK_PUBLICATION_RECOVERY_PAGES,
MAX_PROJECT_TOOL_SNAPSHOT_RECOVERY_PAGES,
)))
) {
throw new TypeError(
'Cluster Plugin Package recovery configuration is invalid',
);
}
}
function assertStageAuthority(
stageAuthority: ClusterPluginPackageStageAuthority,
hasResourceByteSource: boolean,
): void {
if (
!stageAuthority ||
typeof stageAuthority.stage !== 'function' ||
typeof stageAuthority.publisherEvidence !== 'function' ||
typeof stageAuthority.verify !== 'function' ||
(!hasResourceByteSource &&
!(stageAuthority instanceof ClusterPluginPackageOciStageAuthority))
) {
throw new TypeError(
'Cluster Plugin Package recovery stage authority is invalid',
);
}
}
/**
* One-shot admin Job composition. The database is always closed before this
* function settles, and no repository or Kubernetes authority escapes.
*/
export async function recoverClusterPluginPackages(
options: ClusterPluginPackageRecoveryOptions,
): Promise<Readonly<ClusterPluginPackageRecoveryResult>> {
assertOptions(options);
let database: PostgresDatabaseResource | undefined;
let result: Readonly<ClusterPluginPackageRecoveryResult> | undefined;
let failure: unknown;
try {
database = await options.openDatabase();
const evidence = await assertPostgresPackageExecutorSchemaReady(
database.pool,
);
const stageAuthority =
options.stageAuthority ??
(await options.stageAuthorityFactory!(database.pool));
assertStageAuthority(
stageAuthority,
options.resourceByteSource !== undefined,
);
const installRepository = new PostgresPluginPackageInstallRepository(
database.pool,
);
const provenanceRepository =
new PostgresPluginPackagePublisherProvenanceRepository(database.pool);
const provenanceRecovery =
await recoverClusterPluginPackagePublisherProvenance(
installRepository,
provenanceRepository,
stageAuthority,
{
trustAuthorityId: options.trustAuthorityId,
...(options.pageSize === undefined
? {}
: { pageSize: options.pageSize }),
...(options.maxPages === undefined
? {}
: { maxPages: options.maxPages }),
},
);
if (!provenanceRecovery.safeToAdmit) {
throw new ClusterPluginPackagePublisherProvenanceRecoveryRequiredError(
provenanceRecovery,
);
}
const repository = new ClusterPluginPackageProvenanceInstallRepository(
installRepository,
provenanceRepository,
stageAuthority,
options.trustAuthorityId,
);
const publisher = new PluginPackageKubernetesActivationPublisher(
options.api,
{
async verify(intent) {
try {
const [record, lock] = await Promise.all([
repository.find(intent.projectId, intent.packageName),
repository.findLock(intent.lockDigest),
]);
if (
!record ||
!lock ||
record.installationId !== intent.installationId ||
record.lockDigest !== intent.lockDigest ||
record.stageReceipt === null ||
record.stageReceipt.stageRef !== intent.stageRef ||
record.stageReceipt.receiptDigest !== intent.stageReceiptDigest ||
record.stageReceipt.evidenceDigest !==
intent.stageEvidenceDigest ||
record.stageReceipt.contentDigest !== intent.contentDigest
) {
return pluginPackageStageVerificationFailure(
new Error('durable stage identity conflict'),
);
}
assertPluginPackageInstallMatchesLock(lock, record);
await stageAuthority.verify(lock, record.stageReceipt);
await provenanceRepository.assertInstallationNotRevoked(
record.installationId,
);
return clusterPluginPackageActivationEvidence(intent);
} catch (error) {
return pluginPackageStageVerificationFailure(error);
}
},
},
{
clusterIdentity: options.clusterIdentity,
namespace: options.namespace,
now: options.now,
},
);
const resourceByteSource =
options.resourceByteSource ??
new ClusterPluginPackageOciResourceByteSource({
authority:
stageAuthority as ClusterPluginPackageOciStageAuthority,
lockSource: repository,
});
const recovery = await new PluginPackageRecoveryCoordinator({
repository,
stageProvider: stageAuthority,
publisher,
now: options.now,
}).recover({
...(options.pageSize === undefined ? {} : { pageSize: options.pageSize }),
...(options.maxPages === undefined ? {} : { maxPages: options.maxPages }),
});
if (!recovery.safeToAdmit) {
throw new ClusterPluginPackageRecoveryRequiredError(recovery);
}
const taskSpecSemanticRegistry = createBuiltInTaskSpecSemanticRegistry();
const taskReconciliationRepository =
new PostgresPluginPackageTaskReconciliationRepository(
database.pool,
taskSpecSemanticRegistry,
);
const materializedRepository =
new PostgresPluginPackageMaterializedRevisionRepository(
database.pool,
taskSpecSemanticRegistry,
);
const taskPublicationRecovery =
await new PluginPackageTaskPublicationRecoveryCoordinator({
source: taskReconciliationRepository,
publisher: new PluginPackageTaskPublicationCoordinator({
generationSource: publisher,
lockSource: repository,
byteSource: resourceByteSource,
materializedRepository,
reconciliationRepository: taskReconciliationRepository,
taskSpecSemanticRegistry,
}),
}).recover({
...(options.pageSize === undefined
? {}
: { pageSize: options.pageSize }),
...(options.maxPages === undefined
? {}
: { maxPages: options.maxPages }),
});
if (!taskPublicationRecovery.safeToAdmit) {
throw new ClusterPluginPackageTaskPublicationRequiredError(
taskPublicationRecovery,
);
}
const automationPublicationRepository =
new PostgresPluginPackageAutomationPublicationRepository(database.pool);
const automationPublicationRecovery =
await new PluginPackageAutomationPublicationRecoveryCoordinator({
source: automationPublicationRepository,
publisher: new PluginPackageAutomationPublicationCoordinator({
generationSource: publisher,
materializedRepository,
repository: automationPublicationRepository,
taskSpecSemanticRegistry,
now: options.now,
}),
}).recover({
...(options.pageSize === undefined
? {}
: { pageSize: options.pageSize }),
...(options.maxPages === undefined
? {}
: { maxPages: options.maxPages }),
});
if (!automationPublicationRecovery.safeToAdmit) {
throw new ClusterPluginPackageAutomationPublicationRequiredError(
automationPublicationRecovery,
);
}
const toolSnapshotRepository =
new PostgresProjectToolDefinitionSnapshotRepository(database.pool);
const toolSnapshotRecovery =
await new ProjectToolDefinitionSnapshotRecoveryCoordinator({
source: toolSnapshotRepository,
publisher: new ProjectToolDefinitionSnapshotPublicationCoordinator({
source: toolSnapshotRepository,
materializedRepository,
repository: toolSnapshotRepository,
taskSpecSemanticRegistry,
pageSize: Math.min(
options.pageSize ?? 16,
MAX_PROJECT_TOOL_SNAPSHOT_SOURCE_PAGE_SIZE,
),
}),
}).recover({
...(options.pageSize === undefined
? {}
: { pageSize: options.pageSize }),
...(options.maxPages === undefined
? {}
: { maxPages: options.maxPages }),
});
if (!toolSnapshotRecovery.safeToAdmit) {
throw new ClusterPluginPackageToolSnapshotRequiredError(
toolSnapshotRecovery,
);
}
result = Object.freeze({
evidence,
provenanceRecovery,
recovery,
taskPublicationRecovery,
automationPublicationRecovery,
toolSnapshotRecovery,
});
} catch (error) {
failure = error;
}
if (database) {
try {
await database.close();
} catch (closeError) {
if (failure !== undefined) {
throw new AggregateError(
[failure, closeError],
'Cluster Plugin Package recovery failed and PostgreSQL did not close',
);
}
throw closeError;
}
}
if (failure !== undefined) throw failure;
if (!result) {
throw new Error('Cluster Plugin Package recovery produced no result');
}
return result;
}
@@ -0,0 +1,55 @@
#!/usr/bin/env node
// Cluster Plugin Package recovery boundary; keep the operational CLI explicit.
import { runClusterPluginPackageRecoveryProcess } from './pluginPackageRecoveryProcess';
const USAGE = 'Usage: ql3-plugin-package-recover';
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as {
readonly name?: unknown;
readonly code?: unknown;
};
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-plugin-package-recovery',
event: 'recovery_failed',
name:
typeof candidate?.name === 'string' && candidate.name.length <= 128
? candidate.name
: 'Error',
...(typeof candidate?.code === 'string' && candidate.code.length <= 128
? { code: candidate.code }
: {}),
});
}
async function main(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
if (argv.length !== 0) {
process.stderr.write(
`${JSON.stringify({
code: 'QL3_PLUGIN_PACKAGE_RECOVERY_CLI_USAGE_INVALID',
message: USAGE,
})}\n`,
);
process.exitCode = 64;
return;
}
try {
await runClusterPluginPackageRecoveryProcess({
environment: process.env,
emit(record) {
process.stdout.write(`${JSON.stringify(record)}\n`);
},
});
} catch (error) {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
}
}
void main(process.argv.slice(2));
@@ -0,0 +1,885 @@
// Cluster Plugin Package recovery boundary; keep process composition explicit.
import { closeSync, constants, fstatSync, openSync, readSync } from 'node:fs';
import { isAbsolute } from 'node:path';
import type {
OpenPostgresDatabase,
PostgresPool,
} from '@qinglong/runtime-core';
import { MAX_PLUGIN_PACKAGE_INSTALL_RECOVERY_PAGE_SIZE } from '@qinglong/runtime-core/plugin-package-install';
import {
PluginPackagePublisherTrustRegistry,
type PluginPackagePublisherKeyDefinition,
} from '@qinglong/runtime-core/plugin-package-bundle';
import {
createPluginPackagePublisherTrustSnapshot,
createPluginPackagePublisherEffectiveTrustRegistry,
type PluginPackagePublisherTrustSnapshot,
} from '@qinglong/runtime-core/plugin-package-publisher-trust';
import {
MAX_PLUGIN_PACKAGE_RECOVERY_PAGES,
type PluginPackageRecoveryCycleResult,
} from '@qinglong/runtime-core/plugin-package-recovery';
import type { PluginPackageAutomationPublicationRecoveryCycleResult } from '@qinglong/runtime-core/plugin-package-automation-publication';
import type { PluginPackageResourceByteSource } from '@qinglong/runtime-core/plugin-package-resource-materialization';
import type { PluginPackageTaskPublicationRecoveryCycleResult } from '@qinglong/runtime-core/plugin-package-task-publication';
import type { ProjectToolDefinitionSnapshotRecoveryCycleResult } from '@qinglong/runtime-core/project-tool-definition-snapshot';
import {
createPostgresDatabaseOpener,
isPostgresTlsDnsServername,
loadPostgresCertificateAuthorityFile,
loadPostgresConnectionEnvironment,
PostgresPluginPackagePublisherTrustAuthorityRepository,
type PostgresConnectionOptions,
type PostgresPoolOptions,
} from '@qinglong/cluster-postgres/package-executor';
import {
recoverClusterPluginPackages,
type ClusterPluginPackageRecoveryResult,
} from './pluginPackageRecovery';
import type { ClusterPluginPackagePublisherProvenanceRecoveryResult } from '../publisher/pluginPackagePublisherProvenanceRecovery';
import {
ClusterPluginPackageOciStageAuthority,
type ClusterPluginPackageOciFetch,
type ClusterPluginPackageRegistryCredentialProvider,
type ClusterPluginPackageStageAuthority,
} from './pluginPackageOciStage';
import type { PluginPackageKubernetesConfigMapApi } from './pluginPackageKubernetesActivation';
export type ClusterPluginPackageRecoveryProcessEnvironment = Readonly<
Record<string, string | undefined>
>;
export interface ClusterPluginPackageRecoveryProcessConfig {
readonly clusterIdentity: string;
readonly namespace: string;
readonly allowedRegistries: readonly string[];
readonly publisherTrustFile: string;
readonly publisherTrustAuthorityId: string;
readonly registryCredentialFile?: string;
readonly requestTimeoutMs: number;
readonly pageSize: number;
readonly maxPages: number;
readonly database: Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}>;
}
export interface ClusterPluginPackageRecoveryProcessEvent {
readonly schemaVersion: 1;
readonly component: 'qinglong3-plugin-package-recovery';
readonly event: 'recovery_started' | 'recovery_completed';
readonly clusterIdentity: string;
readonly provenanceRecovery?: Readonly<ClusterPluginPackagePublisherProvenanceRecoveryResult>;
readonly recovery?: Readonly<PluginPackageRecoveryCycleResult>;
readonly taskPublicationRecovery?: Readonly<PluginPackageTaskPublicationRecoveryCycleResult>;
readonly automationPublicationRecovery?: Readonly<PluginPackageAutomationPublicationRecoveryCycleResult>;
readonly toolSnapshotRecovery?: Readonly<ProjectToolDefinitionSnapshotRecoveryCycleResult>;
}
export interface RunClusterPluginPackageRecoveryProcessOptions {
readonly environment: ClusterPluginPackageRecoveryProcessEnvironment;
readonly emit?: (
event: ClusterPluginPackageRecoveryProcessEvent,
) => void | Promise<void>;
readonly openDatabase?: OpenPostgresDatabase;
readonly api?: PluginPackageKubernetesConfigMapApi;
readonly stageAuthority?: ClusterPluginPackageStageAuthority;
readonly resourceByteSource?: PluginPackageResourceByteSource;
readonly trust?: PluginPackagePublisherTrustRegistry;
readonly fetch?: ClusterPluginPackageOciFetch;
}
export interface ClusterPluginPackageRegistryCredentialFile
extends ClusterPluginPackageRegistryCredentialProvider {
dispose(): void;
}
export interface ClusterPluginPackagePublisherTrustFileEvidence {
readonly registry: PluginPackagePublisherTrustRegistry;
readonly snapshot: Readonly<PluginPackagePublisherTrustSnapshot>;
readonly definitions: readonly Readonly<PluginPackagePublisherKeyDefinition>[];
}
export class ClusterPluginPackageRecoveryProcessConfigError extends TypeError {
readonly code = 'QL3_PLUGIN_PACKAGE_RECOVERY_PROCESS_CONFIG_INVALID';
constructor(message: string) {
super(
`Plugin Package recovery process configuration is invalid: ${message}`,
);
this.name = 'ClusterPluginPackageRecoveryProcessConfigError';
}
}
const TRUST_SCHEMA = 'qinglong/plugin-package-publisher-trust@v1';
const REGISTRY_CREDENTIAL_SCHEMA =
'qinglong/plugin-package-registry-credentials@v1';
const MAX_TRUST_FILE_BYTES = 256 * 1024;
const MAX_REGISTRY_CREDENTIAL_FILE_BYTES = 256 * 1024;
const MAX_REGISTRY_CREDENTIALS = 32;
const SAFE_IDENTITY = /^[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}$/;
const DNS_LABEL = /^[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?$/;
const REGISTRY =
/^([a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?(?:\.[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?)*)(?::([1-9][0-9]{0,4}))?$/;
const BEARER_TOKEN = /^[A-Za-z0-9._~+/-]+={0,2}$/;
class LoadedClusterPluginPackageRegistryCredentialFile
implements ClusterPluginPackageRegistryCredentialFile
{
readonly #authorizations: Map<string, Buffer>;
constructor(authorizations: Map<string, Buffer>) {
this.#authorizations = authorizations;
}
authorizationFor(registry: string): string | undefined {
if (typeof registry !== 'string' || !REGISTRY.test(registry)) {
return undefined;
}
return this.#authorizations.get(registry)?.toString('ascii');
}
dispose(): void {
for (const authorization of this.#authorizations.values()) {
authorization.fill(0);
}
this.#authorizations.clear();
}
}
function boundedValue(
environment: ClusterPluginPackageRecoveryProcessEnvironment,
name: string,
maximumLength: number,
required = false,
): string | undefined {
const value = environment[name];
if (value === undefined || value === '') {
if (required) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
`${name} is required`,
);
}
return undefined;
}
if (value.length > maximumLength || /[\0\r\n]/.test(value)) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
`${name} is invalid`,
);
}
return value;
}
function booleanValue(
environment: ClusterPluginPackageRecoveryProcessEnvironment,
name: string,
): boolean {
const value = environment[name];
if (value === undefined || value === '') return false;
if (value === 'true') return true;
if (value === 'false') return false;
throw new ClusterPluginPackageRecoveryProcessConfigError(
`${name} must be true or false`,
);
}
function integerValue(
environment: ClusterPluginPackageRecoveryProcessEnvironment,
name: string,
defaultValue: number,
minimum: number,
maximum: number,
): number {
const value = environment[name];
if (value === undefined || value === '') return defaultValue;
if (!/^\d+$/.test(value)) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
`${name} must be an integer`,
);
}
const parsed = Number(value);
if (!Number.isSafeInteger(parsed) || parsed < minimum || parsed > maximum) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
`${name} must be between ${minimum} and ${maximum}`,
);
}
return parsed;
}
function loadConnection(
environment: ClusterPluginPackageRecoveryProcessEnvironment,
): Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}> {
let connection: PostgresConnectionOptions;
try {
connection = loadPostgresConnectionEnvironment(environment, {
connectionString: 'QL3_POSTGRES_PACKAGE_EXECUTOR_URL',
host: 'QL3_POSTGRES_PACKAGE_EXECUTOR_HOST',
port: 'QL3_POSTGRES_PACKAGE_EXECUTOR_PORT',
database: 'QL3_POSTGRES_PACKAGE_EXECUTOR_DATABASE',
user: 'QL3_POSTGRES_PACKAGE_EXECUTOR_USER',
password: 'QL3_POSTGRES_PACKAGE_EXECUTOR_PASSWORD',
});
} catch (error) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
error instanceof Error
? error.message
: 'PostgreSQL Package executor connection is invalid',
);
}
const mode = environment.QL3_POSTGRES_TLS_MODE ?? 'verify-full';
if (mode !== 'verify-full' && mode !== 'disable') {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'QL3_POSTGRES_TLS_MODE must be verify-full or disable',
);
}
if (
mode === 'disable' &&
!booleanValue(environment, 'QL3_POSTGRES_ALLOW_INSECURE')
) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'disabling PostgreSQL TLS requires QL3_POSTGRES_ALLOW_INSECURE=true',
);
}
const servername = boundedValue(
environment,
'QL3_POSTGRES_TLS_SERVERNAME',
253,
);
if (mode === 'verify-full' && !isPostgresTlsDnsServername(servername)) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'QL3_POSTGRES_TLS_SERVERNAME must be an explicit DNS name for verify-full',
);
}
const certificateAuthorityFile = boundedValue(
environment,
'QL3_POSTGRES_TLS_CA_FILE',
4096,
);
if (mode === 'disable' && certificateAuthorityFile !== undefined) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'QL3_POSTGRES_TLS_CA_FILE cannot be used when TLS is disabled',
);
}
let certificateAuthority: string | undefined;
if (certificateAuthorityFile !== undefined) {
try {
certificateAuthority = loadPostgresCertificateAuthorityFile(
certificateAuthorityFile,
);
} catch {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'QL3_POSTGRES_TLS_CA_FILE must contain a bounded trusted CA bundle',
);
}
}
const applicationName =
boundedValue(environment, 'QL3_POSTGRES_APPLICATION_NAME', 63) ??
'qinglong3-plugin-package-recovery';
if (!/^[A-Za-z0-9][A-Za-z0-9._:-]{0,62}$/.test(applicationName)) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'QL3_POSTGRES_APPLICATION_NAME is invalid',
);
}
return Object.freeze({
connection: Object.freeze({
...connection,
tls:
mode === 'disable'
? Object.freeze({ mode: 'disable' as const })
: Object.freeze({
mode: 'verify-full' as const,
...(certificateAuthority === undefined
? {}
: { ca: certificateAuthority }),
servername: servername!,
}),
}),
pool: Object.freeze({
applicationName,
maxConnections: 1,
connectionTimeoutMs: 15_000,
}),
});
}
export function loadClusterPluginPackageRecoveryProcessConfig(
environment: ClusterPluginPackageRecoveryProcessEnvironment,
): Readonly<ClusterPluginPackageRecoveryProcessConfig> {
if (
!environment ||
typeof environment !== 'object' ||
Array.isArray(environment)
) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'environment must be an object',
);
}
const clusterIdentity = boundedValue(
environment,
'QL3_CLUSTER_IDENTITY',
256,
true,
)!;
const namespace = boundedValue(
environment,
'QL3_KUBERNETES_NAMESPACE',
63,
true,
)!;
const registryValue = boundedValue(
environment,
'QL3_PLUGIN_PACKAGE_OCI_REGISTRIES',
4096,
true,
)!;
const allowedRegistries = registryValue.split(',');
const publisherTrustFile = boundedValue(
environment,
'QL3_PLUGIN_PACKAGE_PUBLISHER_TRUST_FILE',
4096,
true,
)!;
const registryCredentialFile = boundedValue(
environment,
'QL3_PLUGIN_PACKAGE_REGISTRY_CREDENTIAL_FILE',
4096,
);
if (
!SAFE_IDENTITY.test(clusterIdentity) ||
!DNS_LABEL.test(namespace) ||
allowedRegistries.length < 1 ||
allowedRegistries.length > 32 ||
allowedRegistries.some((registry) => !REGISTRY.test(registry)) ||
new Set(allowedRegistries).size !== allowedRegistries.length ||
!isAbsolute(publisherTrustFile) ||
(registryCredentialFile !== undefined &&
!isAbsolute(registryCredentialFile))
) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'cluster, namespace, registry or publisher trust binding is invalid',
);
}
return Object.freeze({
clusterIdentity,
namespace,
allowedRegistries: Object.freeze(allowedRegistries),
publisherTrustFile,
publisherTrustAuthorityId:
boundedValue(
environment,
'QL3_PLUGIN_PACKAGE_TRUST_AUTHORITY_ID',
128,
) ?? 'cluster',
...(registryCredentialFile === undefined ? {} : { registryCredentialFile }),
requestTimeoutMs: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_OCI_TIMEOUT_MS',
15_000,
1_000,
60_000,
),
pageSize: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_RECOVERY_PAGE_SIZE',
16,
1,
MAX_PLUGIN_PACKAGE_INSTALL_RECOVERY_PAGE_SIZE,
),
maxPages: integerValue(
environment,
'QL3_PLUGIN_PACKAGE_RECOVERY_MAX_PAGES',
16,
1,
MAX_PLUGIN_PACKAGE_RECOVERY_PAGES,
),
database: loadConnection(environment),
});
}
function readClusterPluginPackageRegistryCredentialFile(
filePath: string,
): Buffer {
if (
typeof filePath !== 'string' ||
filePath.length < 1 ||
filePath.length > 4096 ||
/[\0\r\n]/.test(filePath) ||
!isAbsolute(filePath)
) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'registry credential file path is invalid',
);
}
let descriptor: number;
try {
descriptor = openSync(filePath, constants.O_RDONLY);
} catch {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'registry credential file is unavailable',
);
}
try {
const stat = fstatSync(descriptor);
if (
!stat.isFile() ||
(stat.mode & 0o027) !== 0 ||
!Number.isSafeInteger(stat.size) ||
stat.size < 1 ||
stat.size > MAX_REGISTRY_CREDENTIAL_FILE_BYTES
) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'registry credential file is not a bounded private regular file',
);
}
const bytes = Buffer.alloc(stat.size + 1);
let offset = 0;
while (offset < bytes.byteLength) {
const count = readSync(
descriptor,
bytes,
offset,
bytes.byteLength - offset,
null,
);
if (count === 0) break;
offset += count;
}
if (offset !== stat.size) {
bytes.fill(0);
throw new ClusterPluginPackageRecoveryProcessConfigError(
'registry credential file changed while reading',
);
}
return bytes.subarray(0, offset);
} finally {
closeSync(descriptor);
}
}
function credentialRecord(
value: unknown,
label: string,
): Record<string, unknown> {
if (
!value ||
typeof value !== 'object' ||
Array.isArray(value) ||
Object.getPrototypeOf(value) !== Object.prototype
) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
`${label} must be an object`,
);
}
return value as Record<string, unknown>;
}
function scrubCredentialSource(value: unknown): void {
if (!value || typeof value !== 'object') return;
const credentials = (value as { credentials?: unknown }).credentials;
if (!Array.isArray(credentials)) return;
for (const candidate of credentials) {
if (!candidate || typeof candidate !== 'object') continue;
const record = candidate as Record<string, unknown>;
if (typeof record.password === 'string') record.password = '';
if (typeof record.token === 'string') record.token = '';
}
}
function zeroAuthorizations(authorizations: Map<string, Buffer>): void {
for (const authorization of authorizations.values()) {
authorization.fill(0);
}
authorizations.clear();
}
export function loadClusterPluginPackageRegistryCredentialFile(
filePath: string,
allowedRegistries: readonly string[],
): ClusterPluginPackageRegistryCredentialFile {
if (
!Array.isArray(allowedRegistries) ||
allowedRegistries.length < 1 ||
allowedRegistries.length > MAX_REGISTRY_CREDENTIALS ||
allowedRegistries.some(
(registry) => typeof registry !== 'string' || !REGISTRY.test(registry),
) ||
new Set(allowedRegistries).size !== allowedRegistries.length
) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'registry credential allowlist is invalid',
);
}
const bytes = readClusterPluginPackageRegistryCredentialFile(filePath);
let value: unknown;
try {
value = JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(bytes));
} catch {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'registry credential file is not valid JSON',
);
} finally {
bytes.fill(0);
}
const authorizations = new Map<string, Buffer>();
try {
const root = credentialRecord(value, 'registry credential file');
if (
Object.keys(root).sort().join(',') !== 'credentials,schema' ||
root.schema !== REGISTRY_CREDENTIAL_SCHEMA ||
!Array.isArray(root.credentials) ||
root.credentials.length < 1 ||
root.credentials.length > MAX_REGISTRY_CREDENTIALS
) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'registry credential file shape is invalid',
);
}
const allowed = new Set(allowedRegistries);
for (const [index, candidate] of root.credentials.entries()) {
const entry = credentialRecord(candidate, `registry credential ${index}`);
const registry = entry.registry;
const scheme = entry.scheme;
if (
typeof registry !== 'string' ||
!REGISTRY.test(registry) ||
!allowed.has(registry) ||
authorizations.has(registry)
) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
`registry credential ${index} binding is invalid`,
);
}
let authorization: Buffer;
if (scheme === 'basic') {
if (
Object.keys(entry).sort().join(',') !==
'password,registry,scheme,username' ||
typeof entry.username !== 'string' ||
Buffer.byteLength(entry.username, 'utf8') < 1 ||
Buffer.byteLength(entry.username, 'utf8') > 256 ||
/[\0-\x1f\x7f:]/.test(entry.username) ||
typeof entry.password !== 'string' ||
Buffer.byteLength(entry.password, 'utf8') < 1 ||
Buffer.byteLength(entry.password, 'utf8') > 4096 ||
/[\0-\x1f\x7f]/.test(entry.password)
) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
`registry credential ${index} basic value is invalid`,
);
}
const userPassword = Buffer.from(
`${entry.username}:${entry.password}`,
'utf8',
);
try {
authorization = Buffer.from(
`Basic ${userPassword.toString('base64')}`,
'ascii',
);
} finally {
userPassword.fill(0);
}
} else if (scheme === 'bearer') {
if (
Object.keys(entry).sort().join(',') !== 'registry,scheme,token' ||
typeof entry.token !== 'string' ||
entry.token.length < 1 ||
entry.token.length > 8192 ||
!BEARER_TOKEN.test(entry.token)
) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
`registry credential ${index} bearer value is invalid`,
);
}
authorization = Buffer.from(`Bearer ${entry.token}`, 'ascii');
} else {
throw new ClusterPluginPackageRecoveryProcessConfigError(
`registry credential ${index} scheme is invalid`,
);
}
authorizations.set(registry, authorization);
}
return new LoadedClusterPluginPackageRegistryCredentialFile(authorizations);
} catch (error) {
zeroAuthorizations(authorizations);
throw error;
} finally {
scrubCredentialSource(value);
}
}
export function loadClusterPluginPackagePublisherTrustFileEvidence(
filePath: string,
): Readonly<ClusterPluginPackagePublisherTrustFileEvidence> {
if (
typeof filePath !== 'string' ||
filePath.length < 1 ||
filePath.length > 4096 ||
/[\0\r\n]/.test(filePath) ||
!isAbsolute(filePath)
) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'publisher trust file path is invalid',
);
}
let descriptor: number;
try {
descriptor = openSync(filePath, constants.O_RDONLY);
} catch {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'publisher trust file is unavailable',
);
}
let bytes: Buffer;
try {
const stat = fstatSync(descriptor);
if (
!stat.isFile() ||
(stat.mode & 0o022) !== 0 ||
!Number.isSafeInteger(stat.size) ||
stat.size < 1 ||
stat.size > MAX_TRUST_FILE_BYTES
) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'publisher trust file is not a bounded read-only regular file',
);
}
bytes = Buffer.alloc(stat.size + 1);
let offset = 0;
while (offset < bytes.byteLength) {
const count = readSync(
descriptor,
bytes,
offset,
bytes.byteLength - offset,
null,
);
if (count === 0) break;
offset += count;
}
if (offset !== stat.size) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'publisher trust file changed while reading',
);
}
bytes = bytes.subarray(0, offset);
} finally {
closeSync(descriptor);
}
let value: unknown;
try {
value = JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(bytes));
} catch {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'publisher trust file is not valid JSON',
);
} finally {
bytes.fill(0);
}
if (
!value ||
typeof value !== 'object' ||
Array.isArray(value) ||
Object.keys(value).sort().join(',') !== 'keys,schema' ||
(value as { schema?: unknown }).schema !== TRUST_SCHEMA ||
!Array.isArray((value as { keys?: unknown }).keys)
) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'publisher trust file shape is invalid',
);
}
try {
const definitions = (
value as { keys: PluginPackagePublisherKeyDefinition[] }
).keys;
const frozenDefinitions = Object.freeze(
definitions.map((definition) => Object.freeze({ ...definition })),
);
return Object.freeze({
registry: new PluginPackagePublisherTrustRegistry(frozenDefinitions),
snapshot:
createPluginPackagePublisherTrustSnapshot(frozenDefinitions),
definitions: frozenDefinitions,
});
} catch {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'publisher trust keys are invalid',
);
}
}
export function loadClusterPluginPackagePublisherTrustFile(
filePath: string,
): PluginPackagePublisherTrustRegistry {
return loadClusterPluginPackagePublisherTrustFileEvidence(filePath).registry;
}
async function productionKubernetesApi(): Promise<PluginPackageKubernetesConfigMapApi> {
const kubernetes = await import('@kubernetes/client-node');
const config = new kubernetes.KubeConfig();
config.loadFromCluster();
return config.makeApiClient(
kubernetes.CoreV1Api,
) as unknown as PluginPackageKubernetesConfigMapApi;
}
function processEvent(
config: Readonly<ClusterPluginPackageRecoveryProcessConfig>,
event: ClusterPluginPackageRecoveryProcessEvent['event'],
provenanceRecovery?: Readonly<ClusterPluginPackagePublisherProvenanceRecoveryResult>,
recovery?: Readonly<PluginPackageRecoveryCycleResult>,
taskPublicationRecovery?: Readonly<PluginPackageTaskPublicationRecoveryCycleResult>,
automationPublicationRecovery?: Readonly<PluginPackageAutomationPublicationRecoveryCycleResult>,
toolSnapshotRecovery?: Readonly<ProjectToolDefinitionSnapshotRecoveryCycleResult>,
): Readonly<ClusterPluginPackageRecoveryProcessEvent> {
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-plugin-package-recovery',
event,
clusterIdentity: config.clusterIdentity,
...(provenanceRecovery === undefined ? {} : { provenanceRecovery }),
...(recovery === undefined ? {} : { recovery }),
...(taskPublicationRecovery === undefined
? {}
: { taskPublicationRecovery }),
...(automationPublicationRecovery === undefined
? {}
: { automationPublicationRecovery }),
...(toolSnapshotRecovery === undefined ? {} : { toolSnapshotRecovery }),
});
}
async function emit(
sink: RunClusterPluginPackageRecoveryProcessOptions['emit'],
value: Readonly<ClusterPluginPackageRecoveryProcessEvent>,
): Promise<void> {
if (!sink) return;
try {
await sink(value);
} catch {
// Diagnostics cannot replace recovery or database close outcomes.
}
}
/** Runs exactly one admin recovery cycle and owns no resident authority. */
export async function runClusterPluginPackageRecoveryProcess(
options: RunClusterPluginPackageRecoveryProcessOptions,
): Promise<Readonly<ClusterPluginPackageRecoveryResult>> {
if (
!options ||
typeof options !== 'object' ||
(options.emit !== undefined && typeof options.emit !== 'function') ||
(options.openDatabase !== undefined &&
typeof options.openDatabase !== 'function') ||
(options.resourceByteSource !== undefined &&
(!options.resourceByteSource ||
typeof options.resourceByteSource.open !== 'function')) ||
(options.fetch !== undefined && typeof options.fetch !== 'function')
) {
throw new TypeError('Plugin Package recovery process options are invalid');
}
const config = loadClusterPluginPackageRecoveryProcessConfig(
options.environment,
);
const trustEvidence =
options.trust === undefined && options.stageAuthority === undefined
? loadClusterPluginPackagePublisherTrustFileEvidence(
config.publisherTrustFile,
)
: undefined;
const registryCredentials =
options.stageAuthority !== undefined ||
config.registryCredentialFile === undefined
? undefined
: loadClusterPluginPackageRegistryCredentialFile(
config.registryCredentialFile,
config.allowedRegistries,
);
try {
const api = options.api ?? (await productionKubernetesApi());
const openDatabase =
options.openDatabase ??
createPostgresDatabaseOpener({
role: 'package-executor',
connection: config.database.connection,
pool: config.database.pool,
onPoolError() {
// Awaited recovery queries and final close remain authoritative.
},
});
await emit(options.emit, processEvent(config, 'recovery_started'));
const result = await recoverClusterPluginPackages({
openDatabase,
api,
...(options.stageAuthority === undefined
? {
stageAuthorityFactory: async (pool: PostgresPool) => {
let effectiveTrust = options.trust;
if (effectiveTrust === undefined && trustEvidence !== undefined) {
const authority =
await new PostgresPluginPackagePublisherTrustAuthorityRepository(
pool,
).findAuthority(config.publisherTrustAuthorityId);
if (!authority) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'durable publisher trust authority is unavailable',
);
}
effectiveTrust =
createPluginPackagePublisherEffectiveTrustRegistry(
trustEvidence.definitions,
authority.effectiveSnapshot,
);
}
if (effectiveTrust === undefined) {
throw new ClusterPluginPackageRecoveryProcessConfigError(
'publisher trust evidence is unavailable',
);
}
return new ClusterPluginPackageOciStageAuthority({
allowedRegistries: config.allowedRegistries,
trust: effectiveTrust,
...(registryCredentials === undefined
? {}
: { credentialProvider: registryCredentials }),
...(options.fetch === undefined
? {}
: { fetch: options.fetch }),
requestTimeoutMs: config.requestTimeoutMs,
});
},
}
: { stageAuthority: options.stageAuthority }),
...(options.resourceByteSource === undefined
? {}
: { resourceByteSource: options.resourceByteSource }),
trustAuthorityId: config.publisherTrustAuthorityId,
clusterIdentity: config.clusterIdentity,
namespace: config.namespace,
now: Date.now,
pageSize: config.pageSize,
maxPages: config.maxPages,
});
await emit(
options.emit,
processEvent(
config,
'recovery_completed',
result.provenanceRecovery,
result.recovery,
result.taskPublicationRecovery,
result.automationPublicationRecovery,
result.toolSnapshotRecovery,
),
);
return result;
} finally {
registryCredentials?.dispose();
}
}
@@ -0,0 +1,73 @@
#!/usr/bin/env node
/** Offline Prompt Output external recovery verification CLI boundary. */
import {
disposeClusterPromptOutputExternalRecoveryInput,
readClusterPromptOutputExternalRecoveryCommand,
readClusterPromptOutputExternalRecoveryInput,
type ClusterPromptOutputExternalRecoveryInput,
} from './promptOutputExternalRecoveryInput';
import { runClusterPromptOutputExternalRecoveryVerifier } from './promptOutputExternalRecoveryVerifier';
const USAGE =
'Usage: ql3-prompt-output-key-recovery-verify run --command-file /absolute/recovery.json';
async function main(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
if (
argv.length !== 3 ||
argv[0] !== 'run' ||
argv[1] !== '--command-file' ||
!argv[2]
) {
process.stderr.write(
`${JSON.stringify({
code: 'QL3_PROMPT_OUTPUT_EXTERNAL_RECOVERY_CLI_USAGE_INVALID',
message: USAGE,
})}\n`,
);
process.exitCode = 64;
return;
}
let input: Readonly<ClusterPromptOutputExternalRecoveryInput> | undefined;
try {
const command = readClusterPromptOutputExternalRecoveryCommand(argv[2]);
input = readClusterPromptOutputExternalRecoveryInput(command);
const proof = runClusterPromptOutputExternalRecoveryVerifier(input);
process.stdout.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-prompt-output-external-recovery-verifier',
event: 'recovery_verified',
...proof,
})}\n`,
);
} catch (error) {
const candidate = error as {
readonly code?: unknown;
readonly name?: unknown;
};
process.stderr.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-prompt-output-external-recovery-verifier',
event: 'recovery_rejected',
name:
typeof candidate.name === 'string'
? candidate.name.slice(0, 128)
: 'Error',
...(typeof candidate.code === 'string'
? { code: candidate.code.slice(0, 128) }
: {}),
})}\n`,
);
process.exitCode = 1;
} finally {
if (input) disposeClusterPromptOutputExternalRecoveryInput(input);
}
}
void main(process.argv.slice(2));
@@ -0,0 +1,325 @@
/** Private Prompt Output external recovery input authority boundary. */
import {
constants,
closeSync,
fstatSync,
openSync,
readFileSync,
} from 'node:fs';
import path from 'node:path';
import type { PluginPackagePromptOutputArtifact } from '@qinglong/ai/plugin-package-prompt-output-artifact';
import type {
PluginPackagePromptOutputDurableKeyFact,
PluginPackagePromptOutputExternalCustodyReceipt,
} from '@qinglong/ai/plugin-package-prompt-output-external-custody';
import {
openPluginPackagePromptOutputExternalCustodyBundle,
type OpenPluginPackagePromptOutputExternalCustodyBundle,
type PluginPackagePromptOutputExternalCustodyBundle,
} from '@qinglong/ai/plugin-package-prompt-output-external-custody-bundle';
import type { PluginPackagePromptOutputExternalRecoveryAuthorization } from '@qinglong/ai/plugin-package-prompt-output-external-recovery-authorization';
const MAX_COMMAND_FILE_BYTES = 16 * 1024;
const MAX_JSON_INPUT_BYTES = 2 * 1024 * 1024;
const MAX_CUSTODY_BUNDLE_BYTES = 128 * 1024;
const MAX_PUBLIC_KEY_BYTES = 8 * 1024;
const ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:@/-]{0,127}$/;
export interface ClusterPromptOutputExternalRecoveryCommand {
readonly schemaVersion: 1;
readonly operation: 'cluster.prompt-output-key.verify-recovery';
readonly authorizationFile: string;
readonly custodyBundleFile: string;
readonly recoveredMaterialFile: string;
readonly durableKeyFactFile: string;
readonly artifactFile: string;
readonly custodyPublicKeyFile: string;
readonly approverPublicKeyFiles: readonly [
Readonly<{ userId: string; filePath: string }>,
Readonly<{ userId: string; filePath: string }>,
];
}
export interface ClusterPromptOutputExternalRecoveryInput {
readonly authorization: PluginPackagePromptOutputExternalRecoveryAuthorization;
readonly receipt: PluginPackagePromptOutputExternalCustodyReceipt;
readonly wrappedMaterial: Buffer;
readonly material: Buffer;
readonly durableKeyFact: Readonly<PluginPackagePromptOutputDurableKeyFact>;
readonly artifact: PluginPackagePromptOutputArtifact;
readonly custodyPublicKey: Buffer;
readonly approverPublicKeys: readonly [
Readonly<{ userId: string; publicKey: Buffer }>,
Readonly<{ userId: string; publicKey: Buffer }>,
];
}
function exactKeys(value: object, expected: readonly string[]): boolean {
const keys = Object.keys(value).sort();
const canonical = [...expected].sort();
return (
keys.length === canonical.length &&
keys.every((key, index) => key === canonical[index])
);
}
function absolutePath(value: unknown, label: string): string {
if (
typeof value !== 'string' ||
!path.isAbsolute(value) ||
path.normalize(value) !== value ||
value.includes('\0') ||
Buffer.byteLength(value, 'utf8') > 4096
) {
throw new TypeError(`${label} is invalid`);
}
return value;
}
function stableReadOnlyFile(
filePathValue: string,
options: Readonly<{
label: string;
minimumBytes: number;
maximumBytes: number;
privateFile: boolean;
}>,
): Buffer {
const filePath = absolutePath(filePathValue, `${options.label} path`);
let descriptor: number | undefined;
try {
descriptor = openSync(
filePath,
constants.O_RDONLY | (constants.O_NOFOLLOW ?? 0),
);
const before = fstatSync(descriptor);
if (
!before.isFile() ||
before.nlink !== 1 ||
before.size < options.minimumBytes ||
before.size > options.maximumBytes ||
(before.mode & 0o222) !== 0 ||
(before.mode & 0o111) !== 0 ||
(before.mode & 0o440) === 0 ||
(options.privateFile && (before.mode & 0o007) !== 0)
) {
throw new TypeError(`${options.label} is unavailable`);
}
const value = readFileSync(descriptor);
const after = fstatSync(descriptor);
if (
value.byteLength !== before.size ||
after.dev !== before.dev ||
after.ino !== before.ino ||
after.size !== before.size ||
after.mtimeMs !== before.mtimeMs
) {
value.fill(0);
throw new TypeError(`${options.label} changed during read`);
}
return value;
} finally {
if (descriptor !== undefined) closeSync(descriptor);
}
}
function jsonFile(filePath: string, label: string): unknown {
const bytes = stableReadOnlyFile(filePath, {
label,
minimumBytes: 2,
maximumBytes: MAX_JSON_INPUT_BYTES,
privateFile: true,
});
try {
return JSON.parse(bytes.toString('utf8')) as unknown;
} finally {
bytes.fill(0);
}
}
export function readClusterPromptOutputExternalRecoveryCommand(
filePathValue: string,
): Readonly<ClusterPromptOutputExternalRecoveryCommand> {
const bytes = stableReadOnlyFile(filePathValue, {
label: 'Recovery command file',
minimumBytes: 2,
maximumBytes: MAX_COMMAND_FILE_BYTES,
privateFile: false,
});
try {
const value = JSON.parse(bytes.toString('utf8')) as unknown;
if (
!value ||
typeof value !== 'object' ||
Array.isArray(value) ||
!exactKeys(value, [
'approverPublicKeyFiles',
'artifactFile',
'authorizationFile',
'custodyBundleFile',
'custodyPublicKeyFile',
'durableKeyFactFile',
'operation',
'recoveredMaterialFile',
'schemaVersion',
])
) {
throw new TypeError('Recovery command shape is invalid');
}
const candidate = value as Record<string, unknown>;
if (
candidate.schemaVersion !== 1 ||
candidate.operation !== 'cluster.prompt-output-key.verify-recovery' ||
!Array.isArray(candidate.approverPublicKeyFiles) ||
candidate.approverPublicKeyFiles.length !== 2
) {
throw new TypeError('Recovery command value is invalid');
}
const approvers = candidate.approverPublicKeyFiles
.map((entry, index) => {
if (
!entry ||
typeof entry !== 'object' ||
Array.isArray(entry) ||
!exactKeys(entry, ['filePath', 'userId'])
) {
throw new TypeError(`Approver public key ${index} is invalid`);
}
const item = entry as Record<string, unknown>;
if (typeof item.userId !== 'string' || !ID_PATTERN.test(item.userId)) {
throw new TypeError(`Approver public key ${index} user is invalid`);
}
return Object.freeze({
userId: item.userId,
filePath: absolutePath(
item.filePath,
`Approver public key ${index} file`,
),
});
})
.sort((left, right) => left.userId.localeCompare(right.userId));
if (approvers[0]!.userId === approvers[1]!.userId) {
throw new TypeError('Approver public key users must be distinct');
}
return Object.freeze({
schemaVersion: 1,
operation: 'cluster.prompt-output-key.verify-recovery',
authorizationFile: absolutePath(
candidate.authorizationFile,
'Authorization file',
),
custodyBundleFile: absolutePath(
candidate.custodyBundleFile,
'Custody bundle file',
),
recoveredMaterialFile: absolutePath(
candidate.recoveredMaterialFile,
'Recovered material file',
),
durableKeyFactFile: absolutePath(
candidate.durableKeyFactFile,
'Durable key fact file',
),
artifactFile: absolutePath(candidate.artifactFile, 'Artifact file'),
custodyPublicKeyFile: absolutePath(
candidate.custodyPublicKeyFile,
'Custody public key file',
),
approverPublicKeyFiles: approvers as unknown as readonly [
Readonly<{ userId: string; filePath: string }>,
Readonly<{ userId: string; filePath: string }>,
],
});
} finally {
bytes.fill(0);
}
}
export function readClusterPromptOutputExternalRecoveryInput(
command: Readonly<ClusterPromptOutputExternalRecoveryCommand>,
): Readonly<ClusterPromptOutputExternalRecoveryInput> {
let custodyBundle:
| Readonly<OpenPluginPackagePromptOutputExternalCustodyBundle>
| undefined;
let material: Buffer | undefined;
let custodyPublicKey: Buffer | undefined;
const approverPublicKeys: Buffer[] = [];
try {
material = stableReadOnlyFile(command.recoveredMaterialFile, {
label: 'Recovered material',
minimumBytes: 32,
maximumBytes: 32,
privateFile: true,
});
custodyPublicKey = stableReadOnlyFile(command.custodyPublicKeyFile, {
label: 'Custody public key',
minimumBytes: 32,
maximumBytes: MAX_PUBLIC_KEY_BYTES,
privateFile: true,
});
const custodyBundleBytes = stableReadOnlyFile(command.custodyBundleFile, {
label: 'Custody bundle',
minimumBytes: 2,
maximumBytes: MAX_CUSTODY_BUNDLE_BYTES,
privateFile: true,
});
try {
custodyBundle = openPluginPackagePromptOutputExternalCustodyBundle(
JSON.parse(
custodyBundleBytes.toString('utf8'),
) as PluginPackagePromptOutputExternalCustodyBundle,
custodyPublicKey,
);
} finally {
custodyBundleBytes.fill(0);
}
const approvers = command.approverPublicKeyFiles.map((entry) => {
const publicKey = stableReadOnlyFile(entry.filePath, {
label: `Approver ${entry.userId} public key`,
minimumBytes: 32,
maximumBytes: MAX_PUBLIC_KEY_BYTES,
privateFile: true,
});
approverPublicKeys.push(publicKey);
return Object.freeze({ userId: entry.userId, publicKey });
}) as unknown as readonly [
Readonly<{ userId: string; publicKey: Buffer }>,
Readonly<{ userId: string; publicKey: Buffer }>,
];
return Object.freeze({
authorization: jsonFile(
command.authorizationFile,
'Recovery authorization',
) as PluginPackagePromptOutputExternalRecoveryAuthorization,
receipt: custodyBundle.receipt,
wrappedMaterial: custodyBundle.wrappedMaterial,
material,
durableKeyFact: jsonFile(
command.durableKeyFactFile,
'Durable key fact',
) as PluginPackagePromptOutputDurableKeyFact,
artifact: jsonFile(
command.artifactFile,
'Prompt output Artifact',
) as PluginPackagePromptOutputArtifact,
custodyPublicKey,
approverPublicKeys: approvers,
});
} catch (cause) {
custodyBundle?.wrappedMaterial.fill(0);
material?.fill(0);
custodyPublicKey?.fill(0);
approverPublicKeys.forEach((publicKey) => publicKey.fill(0));
throw cause;
}
}
export function disposeClusterPromptOutputExternalRecoveryInput(
value: Readonly<ClusterPromptOutputExternalRecoveryInput>,
): void {
value.wrappedMaterial.fill(0);
value.material.fill(0);
value.custodyPublicKey.fill(0);
value.approverPublicKeys.forEach(({ publicKey }) => publicKey.fill(0));
}
@@ -0,0 +1,47 @@
/** Content-free Prompt Output external recovery verifier boundary. */
import {
verifyAuthorizedPluginPackagePromptOutputRecoveredMaterial,
type PluginPackagePromptOutputAuthorizedExternalRecoveryProof,
} from '@qinglong/ai/plugin-package-prompt-output-external-recovery-authorization';
import type { ClusterPromptOutputExternalRecoveryInput } from './promptOutputExternalRecoveryInput';
export class ClusterPromptOutputExternalRecoveryVerifierConfigError extends TypeError {
readonly code = 'QL3_PROMPT_OUTPUT_EXTERNAL_RECOVERY_VERIFIER_CONFIG_INVALID';
constructor(message: string, readonly cause?: unknown) {
super(
`Prompt output external recovery verifier configuration is invalid: ${message}`,
);
this.name = 'ClusterPromptOutputExternalRecoveryVerifierConfigError';
}
}
export function runClusterPromptOutputExternalRecoveryVerifier(
input: Readonly<ClusterPromptOutputExternalRecoveryInput>,
verifiedAtMs = Date.now(),
): Readonly<PluginPackagePromptOutputAuthorizedExternalRecoveryProof> {
if (!Number.isSafeInteger(verifiedAtMs) || verifiedAtMs < 0) {
throw new ClusterPromptOutputExternalRecoveryVerifierConfigError(
'verification time is invalid',
);
}
try {
return verifyAuthorizedPluginPackagePromptOutputRecoveredMaterial({
authorization: input.authorization,
trustedApprovers: input.approverPublicKeys,
receipt: input.receipt,
trustedCustodyPublicKey: input.custodyPublicKey,
wrappedMaterial: input.wrappedMaterial,
durableKeyFact: input.durableKeyFact,
material: input.material,
artifact: input.artifact,
verifiedAtMs,
});
} catch (cause) {
throw new ClusterPromptOutputExternalRecoveryVerifierConfigError(
'recovery evidence is untrusted',
cause,
);
}
}
@@ -0,0 +1,185 @@
#!/usr/bin/env node
/** One-shot Prompt Output key retirement CLI boundary. */
import {
constants,
closeSync,
fstatSync,
openSync,
readFileSync,
} from 'node:fs';
import path from 'node:path';
import { type ClusterPromptOutputKubernetesSecretKeyringOptions } from './promptOutputKubernetesSecretKeyring';
import { openPromptOutputKubernetesSecretAuthority } from './promptOutputKubernetesSecretAuthority';
import { runClusterPromptOutputKeyRetirementProcess } from './promptOutputKeyRetirementProcess';
import { loadPromptOutputPostgresMaintenanceConnection } from './promptOutputPostgresMaintenanceConnection';
const USAGE =
'Usage: ql3-prompt-output-key-retire run --command-file /absolute/retirement.json';
const MAX_COMMAND_FILE_BYTES = 16 * 1024;
interface RetirementCommand {
readonly schemaVersion: 1;
readonly operation: 'cluster.prompt-output-key.retire';
readonly kubernetes: ClusterPromptOutputKubernetesSecretKeyringOptions;
readonly request: Readonly<{
keyId: string;
retirementId: string;
requestId: string;
mutationId: string;
}>;
}
function exactKeys(value: object, expected: readonly string[]): boolean {
const actual = Object.keys(value).sort();
const canonical = [...expected].sort();
return (
actual.length === canonical.length &&
actual.every((key, index) => key === canonical[index])
);
}
function commandFile(filePath: string): RetirementCommand {
if (
typeof filePath !== 'string' ||
!path.isAbsolute(filePath) ||
filePath.includes('\0') ||
Buffer.byteLength(filePath, 'utf8') > 4096
) {
throw new TypeError('Retirement command file path is invalid');
}
let descriptor: number | undefined;
try {
descriptor = openSync(
filePath,
constants.O_RDONLY | (constants.O_NOFOLLOW ?? 0),
);
const stat = fstatSync(descriptor);
if (!stat.isFile() || stat.size < 1 || stat.size > MAX_COMMAND_FILE_BYTES) {
throw new TypeError('Retirement command file is invalid');
}
const parsed = JSON.parse(readFileSync(descriptor, 'utf8')) as unknown;
if (
!parsed ||
typeof parsed !== 'object' ||
Array.isArray(parsed) ||
!exactKeys(parsed, [
'kubernetes',
'operation',
'request',
'schemaVersion',
])
) {
throw new TypeError('Retirement command shape is invalid');
}
const candidate = parsed as Record<string, unknown>;
if (
candidate.schemaVersion !== 1 ||
candidate.operation !== 'cluster.prompt-output-key.retire' ||
!candidate.kubernetes ||
typeof candidate.kubernetes !== 'object' ||
Array.isArray(candidate.kubernetes) ||
!candidate.request ||
typeof candidate.request !== 'object' ||
Array.isArray(candidate.request)
) {
throw new TypeError('Retirement command value is invalid');
}
const kubernetes = candidate.kubernetes as Record<string, unknown>;
const request = candidate.request as Record<string, unknown>;
if (
!exactKeys(kubernetes, [
'dataKey',
'expectedSecretUid',
'namespace',
'secretName',
]) ||
!exactKeys(request, ['keyId', 'mutationId', 'requestId', 'retirementId'])
) {
throw new TypeError('Retirement command nested shape is invalid');
}
return Object.freeze(parsed as RetirementCommand);
} finally {
if (descriptor !== undefined) closeSync(descriptor);
}
}
async function main(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
if (
argv.length !== 3 ||
argv[0] !== 'run' ||
argv[1] !== '--command-file' ||
!argv[2]
) {
process.stderr.write(
`${JSON.stringify({
code: 'QL3_PROMPT_OUTPUT_KEY_RETIREMENT_CLI_USAGE_INVALID',
message: USAGE,
})}\n`,
);
process.exitCode = 64;
return;
}
let authority:
| Awaited<ReturnType<typeof openPromptOutputKubernetesSecretAuthority>>
| undefined;
try {
const command = commandFile(argv[2]);
authority = await openPromptOutputKubernetesSecretAuthority(
command.kubernetes,
);
const result = await runClusterPromptOutputKeyRetirementProcess({
database: {
connection: loadPromptOutputPostgresMaintenanceConnection(process.env),
pool: {
applicationName: 'qinglong3-prompt-output-key-retirement',
maxConnections: 1,
},
},
request: command.request,
materials: authority.materials,
});
process.stdout.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-prompt-output-key-retirement',
event: 'key_retirement_completed',
status: result.status,
keyId: result.keyId,
retirementId: result.retirementId,
preparationDigest: result.preparationDigest,
completionDigest: result.completionDigest,
completedAtMs: result.completedAtMs,
})}\n`,
);
} catch (error) {
const candidate = error as {
readonly code?: unknown;
readonly name?: unknown;
};
process.stderr.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-prompt-output-key-retirement',
event: 'key_retirement_failed',
name:
typeof candidate.name === 'string'
? candidate.name.slice(0, 128)
: 'Error',
...(typeof candidate.code === 'string'
? { code: candidate.code.slice(0, 128) }
: {}),
})}\n`,
);
process.exitCode = 1;
} finally {
authority?.dispose();
}
}
void main(process.argv.slice(2));
@@ -0,0 +1,137 @@
/** Prompt Output key retirement transaction composition boundary. */
import type { OpenPostgresDatabase } from '@qinglong/runtime-core';
import {
createPostgresDatabaseOpener,
type PostgresConnectionOptions,
type PostgresPoolOptions,
} from '@qinglong/cluster-postgres/ai-maintenance';
import {
PluginPackagePromptOutputKeyRetirementCoordinator,
normalizePluginPackagePromptOutputKeyRetirementRequest,
type PluginPackagePromptOutputKeyRetirementMaterialAuthority,
} from '@qinglong/ai/plugin-package-prompt-output-key-retirement';
import { PostgresPluginPackagePromptOutputKeyRetirementRepository } from '@qinglong/ai/postgres-plugin-package-prompt-output-key-retirement-storage';
import {
assertPostgresPluginPackagePromptOutputMaintenanceReady,
type PostgresPluginPackagePromptOutputMaintenanceReadinessReport,
} from '@qinglong/ai/postgres-plugin-package-prompt-output-retention-storage';
export interface RunClusterPromptOutputKeyRetirementProcessOptions {
readonly database: Readonly<{
readonly connection: PostgresConnectionOptions;
readonly pool?: PostgresPoolOptions;
}>;
readonly request: Readonly<{
readonly keyId: string;
readonly retirementId: string;
readonly requestId: string;
readonly mutationId: string;
}>;
readonly materials: PluginPackagePromptOutputKeyRetirementMaterialAuthority;
readonly openDatabase?: OpenPostgresDatabase;
}
export interface ClusterPromptOutputKeyRetirementProcessResult {
readonly readiness: PostgresPluginPackagePromptOutputMaintenanceReadinessReport;
readonly status: 'completed' | 'existing';
readonly keyId: string;
readonly retirementId: string;
readonly preparationDigest: string;
readonly completionDigest: string;
readonly completedAtMs: number;
}
export class ClusterPromptOutputKeyRetirementProcessConfigError extends TypeError {
readonly code = 'QL3_PROMPT_OUTPUT_KEY_RETIREMENT_PROCESS_CONFIG_INVALID';
constructor(message: string, readonly cause?: unknown) {
super(
`Prompt output key retirement process configuration is invalid: ${message}`,
);
this.name = 'ClusterPromptOutputKeyRetirementProcessConfigError';
}
}
export async function runClusterPromptOutputKeyRetirementProcess(
options: RunClusterPromptOutputKeyRetirementProcessOptions,
): Promise<Readonly<ClusterPromptOutputKeyRetirementProcessResult>> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
!options.database ||
typeof options.database !== 'object' ||
Array.isArray(options.database) ||
!options.materials ||
typeof options.materials !== 'object' ||
typeof options.materials.inspect !== 'function' ||
typeof options.materials.retire !== 'function'
) {
throw new ClusterPromptOutputKeyRetirementProcessConfigError(
'options are invalid',
);
}
let request;
try {
request = normalizePluginPackagePromptOutputKeyRetirementRequest(
options.request,
);
} catch (cause) {
throw new ClusterPromptOutputKeyRetirementProcessConfigError(
'request is invalid',
cause,
);
}
let poolError: Error | undefined;
const openDatabase =
options.openDatabase ??
createPostgresDatabaseOpener({
role: 'ai-maintenance',
connection: options.database.connection,
...(options.database.pool === undefined
? {}
: { pool: options.database.pool }),
onPoolError(error) {
poolError ??= error;
},
});
const database = await openDatabase();
let failure: unknown;
try {
const readiness =
await assertPostgresPluginPackagePromptOutputMaintenanceReady(
database.pool,
);
const result = await new PluginPackagePromptOutputKeyRetirementCoordinator({
repository: new PostgresPluginPackagePromptOutputKeyRetirementRepository({
pool: database.pool,
}),
materials: options.materials,
}).retire(request);
if (poolError) throw poolError;
return Object.freeze({
readiness,
status: result.status,
keyId: result.preparation.keyId,
retirementId: result.preparation.retirementId,
preparationDigest: result.preparation.preparationDigest,
completionDigest: result.completion.completionDigest,
completedAtMs: result.completion.completedAtMs,
});
} catch (cause) {
failure = cause;
throw cause;
} finally {
try {
await database.close();
} catch (closeError) {
if (failure !== undefined) {
throw new AggregateError(
[failure, closeError],
'Prompt output key retirement failed and PostgreSQL did not close',
);
}
throw closeError;
}
}
}
@@ -0,0 +1,107 @@
#!/usr/bin/env node
/** One-shot Prompt Output key rotation CLI boundary. */
import { openPromptOutputKubernetesSecretAuthority } from './promptOutputKubernetesSecretAuthority';
import { runClusterPromptOutputKeyRotationProcess } from './promptOutputKeyRotationProcess';
import {
readClusterPromptOutputKeyRotationCommand,
readClusterPromptOutputKeyRotationMaterial,
} from './promptOutputKeyRotationInput';
import { loadPromptOutputPostgresMaintenanceConnection } from './promptOutputPostgresMaintenanceConnection';
const USAGE =
'Usage: ql3-prompt-output-key-rotate run --command-file /absolute/rotation.json';
async function main(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
if (
argv.length !== 3 ||
argv[0] !== 'run' ||
argv[1] !== '--command-file' ||
!argv[2]
) {
process.stderr.write(
`${JSON.stringify({
code: 'QL3_PROMPT_OUTPUT_KEY_ROTATION_CLI_USAGE_INVALID',
message: USAGE,
})}\n`,
);
process.exitCode = 64;
return;
}
let authority:
| Awaited<ReturnType<typeof openPromptOutputKubernetesSecretAuthority>>
| undefined;
let material: Buffer | undefined;
try {
const command = readClusterPromptOutputKeyRotationCommand(argv[2]);
authority = await openPromptOutputKubernetesSecretAuthority(
command.kubernetes,
);
material = readClusterPromptOutputKeyRotationMaterial(
command.stagedMaterialFile,
);
const result = await runClusterPromptOutputKeyRotationProcess({
database: {
connection: loadPromptOutputPostgresMaintenanceConnection(process.env),
pool: {
applicationName: 'qinglong3-prompt-output-key-rotation',
maxConnections: 1,
},
},
request: {
...command.request,
expectedSecretUid: command.kubernetes.expectedSecretUid,
},
material,
materials: authority.materials,
});
process.stdout.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-prompt-output-key-rotation',
event: 'key_rotation_completed',
status: result.status,
rotationId: result.rotationId,
requestId: result.requestId,
mutationId: result.mutationId,
preparationDigest: result.preparationDigest,
completionDigest: result.completionDigest,
generation: result.generation,
previousActiveKeyId: result.previousActiveKeyId,
activeKeyId: result.activeKeyId,
catalogDigest: result.catalogDigest,
materialProof: result.materialProof,
completedAtMs: result.completedAtMs,
})}\n`,
);
} catch (error) {
const candidate = error as {
readonly code?: unknown;
readonly name?: unknown;
};
process.stderr.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-prompt-output-key-rotation',
event: 'key_rotation_failed',
name:
typeof candidate.name === 'string'
? candidate.name.slice(0, 128)
: 'Error',
...(typeof candidate.code === 'string'
? { code: candidate.code.slice(0, 128) }
: {}),
})}\n`,
);
process.exitCode = 1;
} finally {
material?.fill(0);
authority?.dispose();
}
}
void main(process.argv.slice(2));
@@ -0,0 +1,179 @@
/** Private staged Prompt Output key rotation input authority boundary. */
import {
constants,
closeSync,
fstatSync,
openSync,
readFileSync,
} from 'node:fs';
import path from 'node:path';
import type { ClusterPromptOutputKubernetesSecretKeyringOptions } from './promptOutputKubernetesSecretKeyring';
const MAX_COMMAND_FILE_BYTES = 16 * 1024;
const ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const DIGEST = /^[0-9a-f]{64}$/;
export interface ClusterPromptOutputKeyRotationCommand {
readonly schemaVersion: 1;
readonly operation: 'cluster.prompt-output-key.rotate';
readonly kubernetes: ClusterPromptOutputKubernetesSecretKeyringOptions;
readonly stagedMaterialFile: string;
readonly request: Readonly<{
rotationId: string;
requestId: string;
mutationId: string;
expectedActiveKeyId: string;
expectedCatalogDigest: string;
newKeyId: string;
}>;
}
function exactKeys(value: object, expected: readonly string[]): boolean {
const actual = Object.keys(value).sort();
const canonical = [...expected].sort();
return (
actual.length === canonical.length &&
actual.every((key, index) => key === canonical[index])
);
}
function absoluteFilePath(value: unknown, label: string): string {
if (
typeof value !== 'string' ||
!path.isAbsolute(value) ||
path.normalize(value) !== value ||
value.includes('\0') ||
Buffer.byteLength(value, 'utf8') > 4096
) {
throw new TypeError(`${label} is invalid`);
}
return value;
}
export function readClusterPromptOutputKeyRotationCommand(
filePathValue: string,
): Readonly<ClusterPromptOutputKeyRotationCommand> {
const filePath = absoluteFilePath(
filePathValue,
'Rotation command file path',
);
let descriptor: number | undefined;
try {
descriptor = openSync(
filePath,
constants.O_RDONLY | (constants.O_NOFOLLOW ?? 0),
);
const stat = fstatSync(descriptor);
if (!stat.isFile() || stat.size < 1 || stat.size > MAX_COMMAND_FILE_BYTES) {
throw new TypeError('Rotation command file is invalid');
}
const parsed = JSON.parse(readFileSync(descriptor, 'utf8')) as unknown;
if (
!parsed ||
typeof parsed !== 'object' ||
Array.isArray(parsed) ||
!exactKeys(parsed, [
'kubernetes',
'operation',
'request',
'schemaVersion',
'stagedMaterialFile',
])
) {
throw new TypeError('Rotation command shape is invalid');
}
const candidate = parsed as Record<string, unknown>;
if (
candidate.schemaVersion !== 1 ||
candidate.operation !== 'cluster.prompt-output-key.rotate' ||
!candidate.kubernetes ||
typeof candidate.kubernetes !== 'object' ||
Array.isArray(candidate.kubernetes) ||
!candidate.request ||
typeof candidate.request !== 'object' ||
Array.isArray(candidate.request)
) {
throw new TypeError('Rotation command value is invalid');
}
const kubernetes = candidate.kubernetes as Record<string, unknown>;
const request = candidate.request as Record<string, unknown>;
if (
!exactKeys(kubernetes, [
'dataKey',
'expectedSecretUid',
'namespace',
'secretName',
]) ||
!exactKeys(request, [
'expectedActiveKeyId',
'expectedCatalogDigest',
'mutationId',
'newKeyId',
'requestId',
'rotationId',
]) ||
![
request.rotationId,
request.requestId,
request.mutationId,
request.expectedActiveKeyId,
request.newKeyId,
].every((value) => typeof value === 'string' && ID.test(value)) ||
typeof request.expectedCatalogDigest !== 'string' ||
!DIGEST.test(request.expectedCatalogDigest)
) {
throw new TypeError('Rotation command nested value is invalid');
}
const stagedMaterialFile = absoluteFilePath(
candidate.stagedMaterialFile,
'Staged material file path',
);
return Object.freeze({
...(parsed as ClusterPromptOutputKeyRotationCommand),
stagedMaterialFile,
});
} finally {
if (descriptor !== undefined) closeSync(descriptor);
}
}
export function readClusterPromptOutputKeyRotationMaterial(
filePathValue: string,
): Buffer {
const filePath = absoluteFilePath(filePathValue, 'Staged material file path');
let descriptor: number | undefined;
try {
descriptor = openSync(
filePath,
constants.O_RDONLY | (constants.O_NOFOLLOW ?? 0),
);
const before = fstatSync(descriptor);
if (
!before.isFile() ||
before.nlink !== 1 ||
before.size !== 32 ||
(before.mode & 0o222) !== 0 ||
(before.mode & 0o111) !== 0 ||
(before.mode & 0o007) !== 0 ||
(before.mode & 0o440) === 0
) {
throw new TypeError('Staged rotation material is unavailable');
}
const material = readFileSync(descriptor);
const after = fstatSync(descriptor);
if (
material.byteLength !== 32 ||
after.dev !== before.dev ||
after.ino !== before.ino ||
after.size !== before.size ||
after.mtimeMs !== before.mtimeMs
) {
material.fill(0);
throw new TypeError('Staged rotation material changed during read');
}
return material;
} finally {
if (descriptor !== undefined) closeSync(descriptor);
}
}
@@ -0,0 +1,147 @@
/** Prompt Output key rotation transaction composition boundary. */
import type { OpenPostgresDatabase } from '@qinglong/runtime-core';
import {
createPostgresDatabaseOpener,
type PostgresConnectionOptions,
type PostgresPoolOptions,
} from '@qinglong/cluster-postgres/ai-maintenance';
import {
PluginPackagePromptOutputKeyRotationCoordinator,
normalizePluginPackagePromptOutputKeyRotationRequest,
type PluginPackagePromptOutputKeyRotationMaterialAuthority,
type PluginPackagePromptOutputKeyRotationRequest,
} from '@qinglong/ai/plugin-package-prompt-output-key-rotation';
import { PostgresPluginPackagePromptOutputKeyRotationRepository } from '@qinglong/ai/postgres-plugin-package-prompt-output-key-rotation-storage';
import {
assertPostgresPluginPackagePromptOutputMaintenanceReady,
type PostgresPluginPackagePromptOutputMaintenanceReadinessReport,
} from '@qinglong/ai/postgres-plugin-package-prompt-output-retention-storage';
export interface RunClusterPromptOutputKeyRotationProcessOptions {
readonly database: Readonly<{
readonly connection: PostgresConnectionOptions;
readonly pool?: PostgresPoolOptions;
}>;
readonly request: Readonly<PluginPackagePromptOutputKeyRotationRequest>;
readonly material: Uint8Array;
readonly materials: PluginPackagePromptOutputKeyRotationMaterialAuthority;
readonly openDatabase?: OpenPostgresDatabase;
}
export interface ClusterPromptOutputKeyRotationProcessResult {
readonly readiness: PostgresPluginPackagePromptOutputMaintenanceReadinessReport;
readonly status: 'completed' | 'existing';
readonly rotationId: string;
readonly requestId: string;
readonly mutationId: string;
readonly preparationDigest: string;
readonly completionDigest: string;
readonly generation: number;
readonly previousActiveKeyId: string;
readonly activeKeyId: string;
readonly catalogDigest: string;
readonly materialProof: string;
readonly completedAtMs: number;
}
export class ClusterPromptOutputKeyRotationProcessConfigError extends TypeError {
readonly code = 'QL3_PROMPT_OUTPUT_KEY_ROTATION_PROCESS_CONFIG_INVALID';
constructor(message: string, readonly cause?: unknown) {
super(
`Prompt output key rotation process configuration is invalid: ${message}`,
);
this.name = 'ClusterPromptOutputKeyRotationProcessConfigError';
}
}
export async function runClusterPromptOutputKeyRotationProcess(
options: RunClusterPromptOutputKeyRotationProcessOptions,
): Promise<Readonly<ClusterPromptOutputKeyRotationProcessResult>> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
!options.database ||
typeof options.database !== 'object' ||
Array.isArray(options.database) ||
!(options.material instanceof Uint8Array) ||
options.material.byteLength !== 32 ||
!options.materials ||
typeof options.materials !== 'object' ||
typeof options.materials.rotate !== 'function'
) {
throw new ClusterPromptOutputKeyRotationProcessConfigError(
'options are invalid',
);
}
let request;
try {
request = normalizePluginPackagePromptOutputKeyRotationRequest(
options.request,
);
} catch (cause) {
throw new ClusterPromptOutputKeyRotationProcessConfigError(
'request is invalid',
cause,
);
}
let poolError: Error | undefined;
const openDatabase =
options.openDatabase ??
createPostgresDatabaseOpener({
role: 'ai-maintenance',
connection: options.database.connection,
...(options.database.pool === undefined
? {}
: { pool: options.database.pool }),
onPoolError(error) {
poolError ??= error;
},
});
const database = await openDatabase();
let failure: unknown;
try {
const readiness =
await assertPostgresPluginPackagePromptOutputMaintenanceReady(
database.pool,
);
const result = await new PluginPackagePromptOutputKeyRotationCoordinator({
repository: new PostgresPluginPackagePromptOutputKeyRotationRepository({
pool: database.pool,
}),
materials: options.materials,
}).rotate({ request, material: options.material });
if (poolError) throw poolError;
return Object.freeze({
readiness,
status: result.status,
rotationId: result.preparation.rotationId,
requestId: result.preparation.requestId,
mutationId: result.preparation.mutationId,
preparationDigest: result.preparation.preparationDigest,
completionDigest: result.completion.completionDigest,
generation: result.completion.generation,
previousActiveKeyId: result.completion.previousActiveKeyId,
activeKeyId: result.completion.activeKeyId,
catalogDigest: result.completion.catalogDigest,
materialProof: result.completion.materialProof,
completedAtMs: result.completion.completedAtMs,
});
} catch (cause) {
failure = cause;
throw cause;
} finally {
try {
await database.close();
} catch (closeError) {
if (failure !== undefined) {
throw new AggregateError(
[failure, closeError],
'Prompt output key rotation failed and PostgreSQL did not close',
);
}
throw closeError;
}
}
}
@@ -0,0 +1,157 @@
/** Least-privilege Kubernetes Secret authority assembly boundary. */
import {
ClusterPromptOutputKubernetesSecretKeyring,
type ClusterPromptOutputKubernetesSecretApi,
type ClusterPromptOutputKubernetesSecretKeyringOptions,
} from './promptOutputKubernetesSecretKeyring';
interface AccessReviewAttributes {
readonly namespace?: string;
readonly verb: string;
readonly resource: string;
readonly name?: string;
}
interface AuthorizationApi {
createSelfSubjectAccessReview(
request: Readonly<{
body: Readonly<{
apiVersion: 'authorization.k8s.io/v1';
kind: 'SelfSubjectAccessReview';
spec: Readonly<{ resourceAttributes: AccessReviewAttributes }>;
}>;
}>,
): Promise<
Readonly<{
status?: Readonly<{ allowed?: boolean; denied?: boolean }>;
}>
>;
}
function accessMatrix(
options: ClusterPromptOutputKubernetesSecretKeyringOptions,
): Readonly<{
allowed: readonly AccessReviewAttributes[];
denied: readonly AccessReviewAttributes[];
}> {
return Object.freeze({
allowed: Object.freeze([
{
namespace: options.namespace,
verb: 'get',
resource: 'secrets',
name: options.secretName,
},
{
namespace: options.namespace,
verb: 'update',
resource: 'secrets',
name: options.secretName,
},
]),
denied: Object.freeze([
{ namespace: options.namespace, verb: 'list', resource: 'secrets' },
{ namespace: options.namespace, verb: 'watch', resource: 'secrets' },
{ namespace: options.namespace, verb: 'create', resource: 'secrets' },
{
namespace: options.namespace,
verb: 'delete',
resource: 'secrets',
name: options.secretName,
},
{
namespace: options.namespace,
verb: 'patch',
resource: 'secrets',
name: options.secretName,
},
{
namespace: options.namespace,
verb: 'get',
resource: 'secrets',
name: `${options.secretName}-other`,
},
{ namespace: options.namespace, verb: 'get', resource: 'configmaps' },
{ namespace: options.namespace, verb: 'get', resource: 'pods' },
]),
});
}
async function assertExactKubernetesAuthority(
api: AuthorizationApi,
options: ClusterPromptOutputKubernetesSecretKeyringOptions,
): Promise<void> {
const matrix = accessMatrix(options);
for (const [expected, checks] of [
[true, matrix.allowed],
[false, matrix.denied],
] as const) {
for (const attributes of checks) {
const review = await api.createSelfSubjectAccessReview({
body: {
apiVersion: 'authorization.k8s.io/v1',
kind: 'SelfSubjectAccessReview',
spec: { resourceAttributes: attributes },
},
});
if (
review.status?.allowed !== expected ||
(expected && review.status.denied === true)
) {
throw new TypeError(
'Kubernetes Secret lifecycle authority is not exact',
);
}
}
}
}
export async function openPromptOutputKubernetesSecretAuthority(
options: ClusterPromptOutputKubernetesSecretKeyringOptions,
): Promise<
Readonly<{
materials: ClusterPromptOutputKubernetesSecretKeyring;
dispose(): void;
}>
> {
const kubernetes = await import('@kubernetes/client-node');
const config = new kubernetes.KubeConfig();
config.loadFromCluster();
const cluster = config.getCurrentCluster();
let server: URL;
try {
server = new URL(cluster?.server ?? '');
} catch {
throw new TypeError('Kubernetes cluster authority is invalid');
}
if (
!cluster ||
server.protocol !== 'https:' ||
server.username !== '' ||
server.password !== '' ||
server.hash !== '' ||
cluster.skipTLSVerify === true ||
(typeof cluster.caData !== 'string' && typeof cluster.caFile !== 'string')
) {
throw new TypeError('Kubernetes cluster authority is invalid');
}
const secrets = config.makeApiClient(
kubernetes.CoreV1Api,
) as unknown as ClusterPromptOutputKubernetesSecretApi;
const authorization = config.makeApiClient(
kubernetes.AuthorizationV1Api,
) as unknown as AuthorizationApi;
await assertExactKubernetesAuthority(authorization, options);
let active = true;
return Object.freeze({
materials: new ClusterPromptOutputKubernetesSecretKeyring(secrets, options),
dispose() {
if (!active) return;
active = false;
for (const user of config.getUsers()) {
(user as { token?: string }).token = '';
}
config.setCurrentContext('disposed');
},
});
}
@@ -0,0 +1,475 @@
/** ResourceVersion-fenced Prompt Output Kubernetes keyring boundary. */
import { Buffer } from 'node:buffer';
import {
canonicalPluginPackagePromptOutputKeyringManifest,
inspectPluginPackagePromptOutputKeyringManifest,
parsePluginPackagePromptOutputKeyringManifest,
pluginPackagePromptOutputKeyringCatalogDigest,
retirePluginPackagePromptOutputKeyringManifest,
rotatePluginPackagePromptOutputKeyringManifest,
type PluginPackagePromptOutputKeyringManifest,
type PluginPackagePromptOutputKeyringRotationMutation,
} from '@qinglong/ai/plugin-package-prompt-output-keyring-manifest';
import {
InvalidPluginPackagePromptOutputKeyRetirementError,
PluginPackagePromptOutputKeyRetirementConflictError,
PluginPackagePromptOutputKeyRetirementUnavailableError,
type PluginPackagePromptOutputKeyMaterialState,
type PluginPackagePromptOutputKeyRetirementMaterialAuthority,
type PluginPackagePromptOutputKeyRetirementPreparation,
} from '@qinglong/ai/plugin-package-prompt-output-key-retirement';
const DNS_LABEL = /^[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?$/;
const DNS_SUBDOMAIN = /^[a-z0-9](?:[-a-z0-9.]{0,251}[a-z0-9])?$/;
const DATA_KEY = /^[A-Za-z0-9._-]{1,253}$/;
const UID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const RESOURCE_VERSION = /^[1-9][0-9]{0,31}$/;
const BASE64 =
/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/;
const FIELD_MANAGER = 'qinglong-prompt-output-key-retirement';
const ROTATION_FIELD_MANAGER = 'qinglong-prompt-output-key-rotation';
const MANAGED_BY_LABEL = 'app.kubernetes.io/managed-by';
const MANAGED_BY_VALUE = 'qinglong3';
const KEYRING_LABEL = 'qinglong.io/prompt-output-keyring';
const KEYRING_LABEL_VALUE = 'v1';
const GENERATION_ANNOTATION = 'qinglong.io/prompt-output-keyring-generation';
const CATALOG_DIGEST_ANNOTATION =
'qinglong.io/prompt-output-keyring-catalog-digest';
const LAST_APPLIED_ANNOTATION =
'kubectl.kubernetes.io/last-applied-configuration';
const MAX_SECRET_DATA_BYTES = 384 * 1024;
export interface ClusterPromptOutputKubernetesSecretKeyringOptions {
readonly namespace: string;
readonly secretName: string;
readonly expectedSecretUid: string;
readonly dataKey?: string;
}
export interface ClusterPromptOutputKubernetesSecret {
readonly apiVersion?: string;
readonly kind?: string;
readonly type?: string;
readonly immutable?: boolean;
readonly stringData?: Readonly<Record<string, string>>;
readonly data?: Readonly<Record<string, string>>;
readonly metadata?: Readonly<{
name?: string;
namespace?: string;
uid?: string;
resourceVersion?: string;
deletionTimestamp?: Date | string;
labels?: Readonly<Record<string, string>>;
annotations?: Readonly<Record<string, string>>;
[key: string]: unknown;
}>;
readonly [key: string]: unknown;
}
interface ClusterPromptOutputKubernetesSecretWrite
extends ClusterPromptOutputKubernetesSecret {
readonly apiVersion: 'v1';
readonly kind: 'Secret';
readonly type: 'Opaque';
readonly immutable: false;
readonly metadata: NonNullable<
ClusterPromptOutputKubernetesSecret['metadata']
>;
readonly data: Readonly<Record<string, string>>;
}
export interface ClusterPromptOutputKubernetesSecretApi {
readNamespacedSecret(
request: Readonly<{
name: string;
namespace: string;
}>,
): Promise<ClusterPromptOutputKubernetesSecret>;
replaceNamespacedSecret(
request: Readonly<{
name: string;
namespace: string;
body: ClusterPromptOutputKubernetesSecretWrite;
fieldManager: typeof FIELD_MANAGER | typeof ROTATION_FIELD_MANAGER;
fieldValidation: 'Strict';
}>,
): Promise<ClusterPromptOutputKubernetesSecret>;
}
interface StoredKeyring {
readonly secret: ClusterPromptOutputKubernetesSecret;
readonly manifest: Readonly<PluginPackagePromptOutputKeyringManifest>;
readonly resourceVersion: string;
}
function unavailable(
cause?: unknown,
): PluginPackagePromptOutputKeyRetirementUnavailableError {
return new PluginPackagePromptOutputKeyRetirementUnavailableError({
cause: cause instanceof Error ? cause : undefined,
});
}
function apiStatus(cause: unknown): number | null {
if (!cause || typeof cause !== 'object') return null;
if ('code' in cause && typeof cause.code === 'number') return cause.code;
if (
'response' in cause &&
cause.response &&
typeof cause.response === 'object' &&
'statusCode' in cause.response &&
typeof cause.response.statusCode === 'number'
) {
return cause.response.statusCode;
}
return null;
}
function exactKeys(value: object, expected: readonly string[]): boolean {
const actual = Object.keys(value).sort();
const canonical = [...expected].sort();
return (
actual.length === canonical.length &&
actual.every((key, index) => key === canonical[index])
);
}
function option(value: unknown, pattern: RegExp, label: string): string {
if (typeof value !== 'string' || !pattern.test(value)) {
throw new InvalidPluginPackagePromptOutputKeyRetirementError(
`${label} is invalid`,
);
}
return value;
}
function sameAbsentState(
left: Readonly<{
state: 'absent';
keyId: string;
catalogDigest: string;
absenceProof: string;
}>,
right: PluginPackagePromptOutputKeyMaterialState,
): boolean {
return (
right.state === 'absent' &&
right.keyId === left.keyId &&
right.catalogDigest === left.catalogDigest &&
right.absenceProof === left.absenceProof
);
}
/**
* Short-lived, resourceVersion-fenced adapter for one dedicated mutable Secret.
* It owns no timer, watcher, cache, Secret creation, or runtime key resolution.
*/
export class ClusterPromptOutputKubernetesSecretKeyring
implements PluginPackagePromptOutputKeyRetirementMaterialAuthority
{
readonly #namespace: string;
readonly #secretName: string;
readonly #expectedSecretUid: string;
readonly #dataKey: string;
constructor(
private readonly api: ClusterPromptOutputKubernetesSecretApi,
options: ClusterPromptOutputKubernetesSecretKeyringOptions,
) {
if (
!api ||
typeof api !== 'object' ||
typeof api.readNamespacedSecret !== 'function' ||
typeof api.replaceNamespacedSecret !== 'function' ||
!options ||
typeof options !== 'object' ||
Array.isArray(options)
) {
throw new InvalidPluginPackagePromptOutputKeyRetirementError(
'Kubernetes Secret keyring options are invalid',
);
}
this.#namespace = option(options.namespace, DNS_LABEL, 'namespace');
this.#secretName = option(options.secretName, DNS_SUBDOMAIN, 'secretName');
this.#expectedSecretUid = option(
options.expectedSecretUid,
UID,
'expectedSecretUid',
);
this.#dataKey = option(
options.dataKey ?? 'keyring.json',
DATA_KEY,
'dataKey',
);
}
async inspect(
keyId: string,
): Promise<PluginPackagePromptOutputKeyMaterialState> {
return inspectPluginPackagePromptOutputKeyringManifest(
(await this.#read()).manifest,
keyId,
);
}
async retire(
command: Readonly<{
preparation: Readonly<PluginPackagePromptOutputKeyRetirementPreparation>;
}>,
): Promise<
Readonly<{
state: 'absent';
keyId: string;
catalogDigest: string;
absenceProof: string;
}>
> {
const current = await this.#read();
const mutation = retirePluginPackagePromptOutputKeyringManifest(
current.manifest,
command.preparation,
);
if (!mutation.changed) return mutation.state;
try {
const written = this.#parse(
await this.api.replaceNamespacedSecret({
name: this.#secretName,
namespace: this.#namespace,
body: this.#body(current, mutation.manifest),
fieldManager: FIELD_MANAGER,
fieldValidation: 'Strict',
}),
);
const state = inspectPluginPackagePromptOutputKeyringManifest(
written.manifest,
mutation.state.keyId,
);
if (!sameAbsentState(mutation.state, state)) {
throw new PluginPackagePromptOutputKeyRetirementConflictError();
}
return mutation.state;
} catch (cause) {
const status = apiStatus(cause);
if (status !== 409 && status !== null) {
if (
cause instanceof
PluginPackagePromptOutputKeyRetirementConflictError ||
cause instanceof
PluginPackagePromptOutputKeyRetirementUnavailableError
) {
throw cause;
}
throw unavailable(cause);
}
try {
const winner = retirePluginPackagePromptOutputKeyringManifest(
(await this.#read()).manifest,
command.preparation,
);
if (!winner.changed && sameAbsentState(mutation.state, winner.state)) {
return winner.state;
}
} catch (replayCause) {
if (status === null) throw unavailable(cause);
throw replayCause;
}
if (status === null) throw unavailable(cause);
throw new PluginPackagePromptOutputKeyRetirementConflictError();
}
}
async rotate(
command: Readonly<{
expectedActiveKeyId: string;
expectedCatalogDigest: string;
newKeyId: string;
material: Uint8Array;
}>,
): Promise<
Readonly<PluginPackagePromptOutputKeyringRotationMutation['state']>
> {
const current = await this.#read();
const mutation = rotatePluginPackagePromptOutputKeyringManifest(
current.manifest,
command,
);
if (!mutation.changed) return mutation.state;
try {
const written = this.#parse(
await this.api.replaceNamespacedSecret({
name: this.#secretName,
namespace: this.#namespace,
body: this.#body(current, mutation.manifest),
fieldManager: ROTATION_FIELD_MANAGER,
fieldValidation: 'Strict',
}),
);
const winner = rotatePluginPackagePromptOutputKeyringManifest(
written.manifest,
command,
);
if (winner.changed) {
throw new PluginPackagePromptOutputKeyRetirementConflictError();
}
return winner.state;
} catch (cause) {
const status = apiStatus(cause);
if (status !== 409 && status !== null) {
if (
cause instanceof
PluginPackagePromptOutputKeyRetirementConflictError ||
cause instanceof
PluginPackagePromptOutputKeyRetirementUnavailableError
) {
throw cause;
}
throw unavailable(cause);
}
try {
const winner = rotatePluginPackagePromptOutputKeyringManifest(
(await this.#read()).manifest,
command,
);
if (!winner.changed) return winner.state;
} catch (replayCause) {
if (status === null) throw unavailable(cause);
throw replayCause;
}
if (status === null) throw unavailable(cause);
throw new PluginPackagePromptOutputKeyRetirementConflictError();
}
}
async #read(): Promise<StoredKeyring> {
try {
return this.#parse(
await this.api.readNamespacedSecret({
name: this.#secretName,
namespace: this.#namespace,
}),
);
} catch (cause) {
if (
cause instanceof PluginPackagePromptOutputKeyRetirementConflictError ||
cause instanceof PluginPackagePromptOutputKeyRetirementUnavailableError
) {
throw cause;
}
throw unavailable(cause);
}
}
#parse(secret: ClusterPromptOutputKubernetesSecret): StoredKeyring {
const metadata = secret?.metadata;
const data = secret?.data;
const encoded = data?.[this.#dataKey];
let bytes: Buffer | undefined;
let canonical: Buffer | undefined;
try {
if (
secret.apiVersion !== 'v1' ||
secret.kind !== 'Secret' ||
secret.type !== 'Opaque' ||
secret.immutable !== false ||
secret.stringData !== undefined ||
!metadata ||
metadata.name !== this.#secretName ||
metadata.namespace !== this.#namespace ||
metadata.uid !== this.#expectedSecretUid ||
typeof metadata.resourceVersion !== 'string' ||
!RESOURCE_VERSION.test(metadata.resourceVersion) ||
metadata.deletionTimestamp !== undefined ||
metadata.labels?.[MANAGED_BY_LABEL] !== MANAGED_BY_VALUE ||
metadata.labels?.[KEYRING_LABEL] !== KEYRING_LABEL_VALUE ||
metadata.annotations?.[LAST_APPLIED_ANNOTATION] !== undefined ||
!data ||
!exactKeys(data, [this.#dataKey]) ||
typeof encoded !== 'string' ||
encoded.length < 1 ||
encoded.length > MAX_SECRET_DATA_BYTES ||
!BASE64.test(encoded)
) {
throw unavailable();
}
bytes = Buffer.from(encoded, 'base64');
if (bytes.toString('base64') !== encoded) throw unavailable();
const manifest = parsePluginPackagePromptOutputKeyringManifest(bytes);
canonical = canonicalPluginPackagePromptOutputKeyringManifest(manifest);
if (
!bytes.equals(canonical) ||
metadata.annotations?.[GENERATION_ANNOTATION] !==
String(manifest.generation) ||
metadata.annotations?.[CATALOG_DIGEST_ANNOTATION] !==
pluginPackagePromptOutputKeyringCatalogDigest(manifest)
) {
throw unavailable();
}
return Object.freeze({
secret,
manifest,
resourceVersion: metadata.resourceVersion,
});
} catch (cause) {
throw cause instanceof
PluginPackagePromptOutputKeyRetirementUnavailableError
? cause
: unavailable(cause);
} finally {
bytes?.fill(0);
canonical?.fill(0);
}
}
#body(
current: StoredKeyring,
manifest: Readonly<PluginPackagePromptOutputKeyringManifest>,
): ClusterPromptOutputKubernetesSecretWrite {
const bytes = canonicalPluginPackagePromptOutputKeyringManifest(manifest);
const { stringData: _ignoredStringData, ...secretWithoutStringData } =
current.secret;
try {
return Object.freeze({
...secretWithoutStringData,
apiVersion: 'v1' as const,
kind: 'Secret' as const,
type: 'Opaque' as const,
immutable: false as const,
metadata: Object.freeze({
...current.secret.metadata,
name: this.#secretName,
namespace: this.#namespace,
uid: this.#expectedSecretUid,
resourceVersion: current.resourceVersion,
labels: Object.freeze({
...current.secret.metadata?.labels,
[MANAGED_BY_LABEL]: MANAGED_BY_VALUE,
[KEYRING_LABEL]: KEYRING_LABEL_VALUE,
}),
annotations: Object.freeze({
...current.secret.metadata?.annotations,
[GENERATION_ANNOTATION]: String(manifest.generation),
[CATALOG_DIGEST_ANNOTATION]:
pluginPackagePromptOutputKeyringCatalogDigest(manifest),
}),
}),
data: Object.freeze({ [this.#dataKey]: bytes.toString('base64') }),
});
} finally {
bytes.fill(0);
}
}
}
export const clusterPromptOutputKubernetesSecretKeyringMetadata = Object.freeze(
{
fieldManager: FIELD_MANAGER,
rotationFieldManager: ROTATION_FIELD_MANAGER,
managedByLabel: MANAGED_BY_LABEL,
managedByValue: MANAGED_BY_VALUE,
keyringLabel: KEYRING_LABEL,
keyringLabelValue: KEYRING_LABEL_VALUE,
generationAnnotation: GENERATION_ANNOTATION,
catalogDigestAnnotation: CATALOG_DIGEST_ANNOTATION,
},
);
@@ -0,0 +1,42 @@
/** Bounded Prompt Output maintenance PostgreSQL connection boundary. */
import {
loadPostgresCertificateAuthorityFile,
loadPostgresConnectionEnvironment,
type PostgresConnectionOptions,
} from '@qinglong/cluster-postgres/ai-maintenance';
export function loadPromptOutputPostgresMaintenanceConnection(
environment: NodeJS.ProcessEnv,
): PostgresConnectionOptions {
const base = loadPostgresConnectionEnvironment(environment, {
connectionString: 'QL3_POSTGRES_AI_MAINTENANCE_URL',
host: 'QL3_POSTGRES_AI_MAINTENANCE_HOST',
port: 'QL3_POSTGRES_AI_MAINTENANCE_PORT',
database: 'QL3_POSTGRES_AI_MAINTENANCE_DATABASE',
user: 'QL3_POSTGRES_AI_MAINTENANCE_USER',
password: 'QL3_POSTGRES_AI_MAINTENANCE_PASSWORD',
});
const mode = environment.QL3_POSTGRES_TLS_MODE ?? 'verify-full';
if (mode === 'disable') {
if (environment.QL3_POSTGRES_ALLOW_INSECURE !== 'true') {
throw new TypeError('Insecure PostgreSQL requires an explicit gate');
}
return Object.freeze({ ...base, tls: { mode: 'disable' as const } });
}
if (mode !== 'verify-full') {
throw new TypeError('PostgreSQL TLS mode is invalid');
}
const caFile = environment.QL3_POSTGRES_TLS_CA_FILE;
const servername = environment.QL3_POSTGRES_TLS_SERVERNAME;
if (!caFile || !servername) {
throw new TypeError('PostgreSQL TLS CA and servername are required');
}
return Object.freeze({
...base,
tls: {
mode: 'verify-full' as const,
ca: loadPostgresCertificateAuthorityFile(caFile),
servername,
},
});
}
@@ -0,0 +1,144 @@
#!/usr/bin/env node
/** One-shot Prompt Output garbage collection CLI boundary. */
import { readFileSync } from 'node:fs';
import path from 'node:path';
import {
loadPostgresCertificateAuthorityFile,
loadPostgresConnectionEnvironment,
type PostgresConnectionOptions,
} from '@qinglong/cluster-postgres/ai-maintenance';
import type { PluginPackagePromptOutputRetentionPolicyCatalog } from '@qinglong/ai/plugin-package-prompt-output-retention';
import { runClusterPromptOutputGcProcess } from './promptOutputGcProcess';
const USAGE =
'Usage: ql3-prompt-output-gc run --policy-file /absolute/retention-policies.json';
const MAX_POLICY_FILE_BYTES = 65_536;
function policyCatalog(
filePath: string,
): PluginPackagePromptOutputRetentionPolicyCatalog {
if (!path.isAbsolute(filePath) || filePath.includes('\0')) {
throw new TypeError('Policy file path is invalid');
}
const bytes = readFileSync(filePath);
if (bytes.length < 1 || bytes.length > MAX_POLICY_FILE_BYTES) {
throw new TypeError('Policy file size is invalid');
}
return JSON.parse(
bytes.toString('utf8'),
) as PluginPackagePromptOutputRetentionPolicyCatalog;
}
function limit(): number {
const raw = process.env.QL3_PROMPT_OUTPUT_GC_LIMIT ?? '32';
if (!/^\d+$/.test(raw)) throw new TypeError('GC limit is invalid');
const value = Number(raw);
if (!Number.isSafeInteger(value) || value < 1 || value > 128) {
throw new TypeError('GC limit is invalid');
}
return value;
}
function connection(): PostgresConnectionOptions {
const base = loadPostgresConnectionEnvironment(process.env, {
connectionString: 'QL3_POSTGRES_AI_MAINTENANCE_URL',
host: 'QL3_POSTGRES_AI_MAINTENANCE_HOST',
port: 'QL3_POSTGRES_AI_MAINTENANCE_PORT',
database: 'QL3_POSTGRES_AI_MAINTENANCE_DATABASE',
user: 'QL3_POSTGRES_AI_MAINTENANCE_USER',
password: 'QL3_POSTGRES_AI_MAINTENANCE_PASSWORD',
});
const mode = process.env.QL3_POSTGRES_TLS_MODE ?? 'verify-full';
if (mode === 'disable') {
if (process.env.QL3_POSTGRES_ALLOW_INSECURE !== 'true') {
throw new TypeError('Insecure PostgreSQL requires an explicit gate');
}
return Object.freeze({ ...base, tls: { mode: 'disable' as const } });
}
if (mode !== 'verify-full')
throw new TypeError('PostgreSQL TLS mode is invalid');
const caFile = process.env.QL3_POSTGRES_TLS_CA_FILE;
const servername = process.env.QL3_POSTGRES_TLS_SERVERNAME;
if (!caFile || !servername) {
throw new TypeError('PostgreSQL TLS CA and servername are required');
}
return Object.freeze({
...base,
tls: {
mode: 'verify-full' as const,
ca: loadPostgresCertificateAuthorityFile(caFile),
servername,
},
});
}
async function main(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
if (
argv.length !== 3 ||
argv[0] !== 'run' ||
argv[1] !== '--policy-file' ||
!argv[2]
) {
process.stderr.write(
`${JSON.stringify({
code: 'QL3_PROMPT_OUTPUT_GC_CLI_USAGE_INVALID',
message: USAGE,
})}\n`,
);
process.exitCode = 64;
return;
}
try {
const result = await runClusterPromptOutputGcProcess({
database: {
connection: connection(),
pool: {
applicationName: 'qinglong3-prompt-output-gc',
maxConnections: 1,
},
},
retentionPolicyCatalog: policyCatalog(argv[2]),
limit: limit(),
});
process.stdout.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-prompt-output-gc',
event: 'gc_completed',
scanned: result.scanned,
tombstoned: result.tombstoned,
skipped: result.skipped,
hasMore: result.hasMore,
})}\n`,
);
} catch (error) {
const candidate = error as {
readonly code?: unknown;
readonly name?: unknown;
};
process.stderr.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-prompt-output-gc',
event: 'gc_failed',
name:
typeof candidate.name === 'string'
? candidate.name.slice(0, 128)
: 'Error',
...(typeof candidate.code === 'string'
? { code: candidate.code.slice(0, 128) }
: {}),
})}\n`,
);
process.exitCode = 1;
}
}
void main(process.argv.slice(2));
@@ -0,0 +1,112 @@
/** Prompt Output garbage collection PostgreSQL process boundary. */
import type { OpenPostgresDatabase } from '@qinglong/runtime-core';
import {
createPostgresDatabaseOpener,
type PostgresConnectionOptions,
type PostgresPoolOptions,
} from '@qinglong/cluster-postgres/ai-maintenance';
import {
PostgresPluginPackagePromptOutputGarbageCollector,
assertPostgresPluginPackagePromptOutputMaintenanceReady,
type PostgresPluginPackagePromptOutputMaintenanceReadinessReport,
} from '@qinglong/ai/postgres-plugin-package-prompt-output-retention-storage';
import {
createPluginPackagePromptOutputRetentionPolicyCatalogResolver,
type PluginPackagePromptOutputRetentionPolicyCatalog,
} from '@qinglong/ai/plugin-package-prompt-output-retention';
export interface RunClusterPromptOutputGcProcessOptions {
readonly database: Readonly<{
readonly connection: PostgresConnectionOptions;
readonly pool?: PostgresPoolOptions;
}>;
readonly retentionPolicyCatalog: PluginPackagePromptOutputRetentionPolicyCatalog;
readonly limit?: number;
readonly openDatabase?: OpenPostgresDatabase;
}
export interface ClusterPromptOutputGcProcessResult {
readonly readiness: PostgresPluginPackagePromptOutputMaintenanceReadinessReport;
readonly scanned: number;
readonly tombstoned: number;
readonly skipped: number;
readonly hasMore: boolean;
}
export class ClusterPromptOutputGcProcessConfigError extends TypeError {
readonly code = 'QL3_PROMPT_OUTPUT_GC_PROCESS_CONFIG_INVALID';
constructor(message: string, readonly cause?: unknown) {
super(`Prompt output GC process configuration is invalid: ${message}`);
this.name = 'ClusterPromptOutputGcProcessConfigError';
}
}
export async function runClusterPromptOutputGcProcess(
options: RunClusterPromptOutputGcProcessOptions,
): Promise<Readonly<ClusterPromptOutputGcProcessResult>> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
!options.database ||
typeof options.database !== 'object' ||
Array.isArray(options.database)
) {
throw new ClusterPromptOutputGcProcessConfigError('options are invalid');
}
let policies;
try {
policies = createPluginPackagePromptOutputRetentionPolicyCatalogResolver(
options.retentionPolicyCatalog,
);
} catch (cause) {
throw new ClusterPromptOutputGcProcessConfigError(
'retention policy catalog is invalid',
cause,
);
}
let poolError: Error | undefined;
const openDatabase =
options.openDatabase ??
createPostgresDatabaseOpener({
role: 'ai-maintenance',
connection: options.database.connection,
...(options.database.pool === undefined
? {}
: { pool: options.database.pool }),
onPoolError(error) {
poolError ??= error;
},
});
const database = await openDatabase();
let failure: unknown;
try {
const readiness =
await assertPostgresPluginPackagePromptOutputMaintenanceReady(
database.pool,
);
const result = await new PostgresPluginPackagePromptOutputGarbageCollector({
pool: database.pool,
policies,
...(options.limit === undefined ? {} : { limit: options.limit }),
}).collect();
if (poolError) throw poolError;
return Object.freeze({ readiness, ...result });
} catch (cause) {
failure = cause;
throw cause;
} finally {
try {
await database.close();
} catch (closeError) {
if (failure !== undefined) {
throw new AggregateError(
[failure, closeError],
'Prompt output GC failed and PostgreSQL did not close',
);
}
throw closeError;
}
}
}
@@ -0,0 +1,510 @@
// Security Administration owns identity and API credential mutations.
import { randomBytes as nodeRandomBytes } from 'node:crypto';
import {
REVOKED_API_CREDENTIAL_DIGEST,
ApiCredentialAdministrationMutationConflictError,
type ApiCredentialAdministrationRepository,
type AppendApiCredentialResult,
type ResolvedApiCredentialMutation,
} from '@qinglong/runtime-core/api-credential-administration';
import {
apiCredentialSecretDigest,
assertApiCredentialPepper,
formatApiCredentialToken,
} from '@qinglong/runtime-core/api-credential-token';
import {
LEGACY_API_CREDENTIAL_PEPPER_KEY_ID,
type ApiCredentialRecord,
} from '@qinglong/runtime-core/api-credential';
import {
type AppendIdentitySubjectResult,
IdentityAdministrationMutationConflictError,
type IdentityAdministrationOperation,
type IdentityAdministrationRepository,
type ResolvedIdentitySubjectMutation,
} from '@qinglong/runtime-core/identity-administration';
import {
normalizeSecurityPrincipal,
type SecurityPrincipal,
type SecuritySubject,
} from '@qinglong/runtime-core/security';
import type { SecurityAuditRecord } from '@qinglong/runtime-core/security-audit';
const MAX_CREDENTIAL_LIFETIME_MS = 2 * 365 * 24 * 60 * 60 * 1000;
const STRONG_USER_ASSURANCES = new Set([
'multi_factor',
'hardware',
'local_console',
]);
export interface ClusterAdministrationOptions {
readonly now?: () => number;
readonly randomBytes?: (size: number) => Buffer;
}
export interface IdentityAdministrationRequest {
readonly mutationId: string;
readonly requestId: string;
readonly expectedCurrentVersion: number;
readonly subject: SecuritySubject;
readonly principal: SecurityPrincipal;
}
export interface CredentialAdministrationRequest {
readonly mutationId: string;
readonly requestId: string;
readonly expectedCurrentVersion: number;
readonly credentialId: string;
readonly subject: SecuritySubject;
readonly principal: SecurityPrincipal;
}
export interface ActiveCredentialAdministrationRequest
extends CredentialAdministrationRequest {
readonly notBeforeAtMs: number;
readonly expiresAtMs: number;
}
export interface CredentialAdministrationResult
extends AppendApiCredentialResult {
readonly token: string | null;
}
export interface ClusterAdministrationService {
registerIdentity(
request: IdentityAdministrationRequest,
): Promise<AppendIdentitySubjectResult>;
enableIdentity(
request: IdentityAdministrationRequest,
): Promise<AppendIdentitySubjectResult>;
disableIdentity(
request: IdentityAdministrationRequest,
): Promise<AppendIdentitySubjectResult>;
issueCredential(
request: ActiveCredentialAdministrationRequest,
): Promise<CredentialAdministrationResult>;
rotateCredential(
request: ActiveCredentialAdministrationRequest,
): Promise<CredentialAdministrationResult>;
revokeCredential(
request: CredentialAdministrationRequest,
): Promise<CredentialAdministrationResult>;
}
export class ClusterAdministrationConfigurationError extends TypeError {
constructor(message: string) {
super(`Cluster administration configuration is invalid: ${message}`);
this.name = 'ClusterAdministrationConfigurationError';
}
}
export class ClusterAdministrationAuthenticationError extends Error {
readonly code = 'CLUSTER_ADMINISTRATION_AUTHENTICATION_REQUIRED';
constructor() {
super('Cluster administration requires a strong principal');
this.name = 'ClusterAdministrationAuthenticationError';
}
}
export class ClusterAdministrationSubjectUnavailableError extends Error {
readonly code = 'CLUSTER_ADMINISTRATION_SUBJECT_UNAVAILABLE';
constructor() {
super('Cluster administration subject is unavailable');
this.name = 'ClusterAdministrationSubjectUnavailableError';
}
}
function exactObject(value: unknown, name: string): asserts value is object {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw new ClusterAdministrationConfigurationError(`${name} is invalid`);
}
}
function exactKeys(
value: object,
name: string,
allowed: ReadonlySet<string>,
): void {
if (Object.keys(value).some((key) => !allowed.has(key))) {
throw new ClusterAdministrationConfigurationError(
`${name} shape is invalid`,
);
}
}
const IDENTITY_REQUEST_KEYS = new Set([
'mutationId',
'requestId',
'expectedCurrentVersion',
'subject',
'principal',
]);
const CREDENTIAL_REQUEST_KEYS = new Set([
...IDENTITY_REQUEST_KEYS,
'credentialId',
]);
const ACTIVE_CREDENTIAL_REQUEST_KEYS = new Set([
...CREDENTIAL_REQUEST_KEYS,
'notBeforeAtMs',
'expiresAtMs',
]);
function administrationPrincipal(
principal: SecurityPrincipal,
nowMs: number,
): Readonly<SecurityPrincipal> {
let normalized: Readonly<SecurityPrincipal>;
try {
normalized = normalizeSecurityPrincipal(principal, nowMs);
} catch {
throw new ClusterAdministrationAuthenticationError();
}
const human =
normalized.subject.type === 'user' &&
STRONG_USER_ASSURANCES.has(normalized.assurance);
const system =
normalized.subject.type === 'system' && normalized.assurance === 'service';
if (!human && !system) {
throw new ClusterAdministrationAuthenticationError();
}
return normalized;
}
function audit(
mutationId: string,
requestId: string,
operationId: string,
principal: Readonly<SecurityPrincipal>,
reason: 'identity_admin' | 'credential_admin',
nowMs: number,
): SecurityAuditRecord {
return {
eventId: mutationId,
requestId,
operationId,
projectId: null,
subject: principal.subject,
authenticationId: principal.authenticationId,
outcome: 'allowed',
reasons: [reason],
fence: null,
occurredAtMs: nowMs,
};
}
function sameSubject(
left: Readonly<SecuritySubject>,
right: Readonly<SecuritySubject>,
): boolean {
return left.type === right.type && left.id === right.id;
}
function sameReplayAudit(
stored: Readonly<SecurityAuditRecord>,
expected: Readonly<SecurityAuditRecord>,
): boolean {
const { occurredAtMs: _storedOccurredAtMs, ...storedSemantic } = stored;
const { occurredAtMs: _expectedOccurredAtMs, ...expectedSemantic } = expected;
return JSON.stringify(storedSemantic) === JSON.stringify(expectedSemantic);
}
function sameIdentityReplay(
stored: ResolvedIdentitySubjectMutation,
operation: IdentityAdministrationOperation,
request: IdentityAdministrationRequest,
principal: Readonly<SecurityPrincipal>,
nowMs: number,
): boolean {
return (
stored.mutation.operation === operation &&
sameSubject(stored.mutation.subject, request.subject) &&
stored.mutation.subjectVersion === request.expectedCurrentVersion + 1 &&
stored.mutation.expectedPreviousVersion ===
request.expectedCurrentVersion &&
stored.mutation.status ===
(operation === 'disable' ? 'disabled' : 'active') &&
sameSubject(stored.mutation.changedBy, principal.subject) &&
sameReplayAudit(
stored.audit,
audit(
request.mutationId,
request.requestId,
`identity.${operation}`,
principal,
'identity_admin',
nowMs,
),
)
);
}
function sameCredentialReplay(
stored: ResolvedApiCredentialMutation,
operation: 'issue' | 'rotate' | 'revoke',
request:
| ActiveCredentialAdministrationRequest
| CredentialAdministrationRequest,
principal: Readonly<SecurityPrincipal>,
nowMs: number,
): boolean {
const active =
operation === 'revoke'
? null
: (request as ActiveCredentialAdministrationRequest);
return (
stored.mutation.operation === operation &&
stored.mutation.credentialId === request.credentialId &&
stored.mutation.credentialVersion === request.expectedCurrentVersion + 1 &&
stored.mutation.expectedPreviousVersion ===
request.expectedCurrentVersion &&
sameSubject(stored.mutation.changedBy, principal.subject) &&
stored.credential.state ===
(operation === 'revoke' ? 'revoked' : 'active') &&
sameSubject(stored.credential.subject, request.subject) &&
(active === null ||
(stored.credential.notBeforeAtMs === active.notBeforeAtMs &&
stored.credential.expiresAtMs === active.expiresAtMs)) &&
sameReplayAudit(
stored.audit,
audit(
request.mutationId,
request.requestId,
`credential.${operation}`,
principal,
'credential_admin',
nowMs,
),
)
);
}
export function createClusterAdministrationService(
identities: IdentityAdministrationRepository,
credentials: ApiCredentialAdministrationRepository,
pepper: string,
options: ClusterAdministrationOptions = {},
): ClusterAdministrationService {
if (
!identities ||
typeof identities.resolve !== 'function' ||
typeof identities.resolveMutation !== 'function' ||
typeof identities.append !== 'function'
) {
throw new ClusterAdministrationConfigurationError(
'identity repository is invalid',
);
}
if (
!credentials ||
typeof credentials.resolveMutation !== 'function' ||
typeof credentials.append !== 'function'
) {
throw new ClusterAdministrationConfigurationError(
'credential repository is invalid',
);
}
try {
assertApiCredentialPepper(pepper);
} catch {
throw new ClusterAdministrationConfigurationError('pepper is invalid');
}
exactObject(options, 'options');
const optionKeys = Object.keys(options);
if (optionKeys.some((key) => key !== 'now' && key !== 'randomBytes')) {
throw new ClusterAdministrationConfigurationError(
'options shape is invalid',
);
}
if (options.now !== undefined && typeof options.now !== 'function') {
throw new ClusterAdministrationConfigurationError('now is invalid');
}
if (
options.randomBytes !== undefined &&
typeof options.randomBytes !== 'function'
) {
throw new ClusterAdministrationConfigurationError('randomBytes is invalid');
}
const now = options.now ?? Date.now;
const randomBytes = options.randomBytes ?? nodeRandomBytes;
const mutateIdentity = async (
operation: IdentityAdministrationOperation,
request: IdentityAdministrationRequest,
): Promise<AppendIdentitySubjectResult> => {
exactObject(request, 'identity request');
exactKeys(request, 'identity request', IDENTITY_REQUEST_KEYS);
const nowMs = now();
if (!Number.isSafeInteger(nowMs) || nowMs < 0) {
throw new ClusterAdministrationConfigurationError('clock is invalid');
}
const principal = administrationPrincipal(request.principal, nowMs);
const existing = await identities.resolveMutation(request.mutationId);
if (existing) {
if (!sameIdentityReplay(existing, operation, request, principal, nowMs)) {
throw new IdentityAdministrationMutationConflictError();
}
return Object.freeze({
status: 'existing',
identity: existing.identity,
mutation: existing.mutation,
});
}
return identities.append({
expectedCurrentVersion: request.expectedCurrentVersion,
mutation: {
mutationId: request.mutationId,
operation,
subject: request.subject,
subjectVersion: request.expectedCurrentVersion + 1,
expectedPreviousVersion: request.expectedCurrentVersion,
status: operation === 'disable' ? 'disabled' : 'active',
changedBy: principal.subject,
createdAtMs: nowMs,
},
audit: audit(
request.mutationId,
request.requestId,
`identity.${operation}`,
principal,
'identity_admin',
nowMs,
),
});
};
const mutateCredential = async (
operation: 'issue' | 'rotate' | 'revoke',
request:
| ActiveCredentialAdministrationRequest
| CredentialAdministrationRequest,
): Promise<CredentialAdministrationResult> => {
exactObject(request, 'credential request');
exactKeys(
request,
'credential request',
operation === 'revoke'
? CREDENTIAL_REQUEST_KEYS
: ACTIVE_CREDENTIAL_REQUEST_KEYS,
);
const nowMs = now();
if (!Number.isSafeInteger(nowMs) || nowMs < 0) {
throw new ClusterAdministrationConfigurationError('clock is invalid');
}
const principal = administrationPrincipal(request.principal, nowMs);
const existing = await credentials.resolveMutation(request.mutationId);
if (existing) {
if (
!sameCredentialReplay(existing, operation, request, principal, nowMs)
) {
throw new ApiCredentialAdministrationMutationConflictError();
}
return Object.freeze({
status: 'existing',
credential: existing.credential,
mutation: existing.mutation,
token: null,
});
}
const identity = await identities.resolve(request.subject);
if (!identity || (operation !== 'revoke' && identity.status !== 'active')) {
throw new ClusterAdministrationSubjectUnavailableError();
}
let secret: Buffer | undefined;
let secretBase64Url: string | null = null;
let secretDigest = REVOKED_API_CREDENTIAL_DIGEST;
let notBeforeAtMs = nowMs;
let expiresAtMs = nowMs + 1;
if (operation !== 'revoke') {
const activeRequest = request as ActiveCredentialAdministrationRequest;
notBeforeAtMs = activeRequest.notBeforeAtMs;
expiresAtMs = activeRequest.expiresAtMs;
if (
!Number.isSafeInteger(notBeforeAtMs) ||
notBeforeAtMs < nowMs ||
!Number.isSafeInteger(expiresAtMs) ||
expiresAtMs <= notBeforeAtMs ||
expiresAtMs - nowMs > MAX_CREDENTIAL_LIFETIME_MS
) {
throw new ClusterAdministrationConfigurationError(
'credential lifetime is invalid',
);
}
secret = randomBytes(32);
if (!Buffer.isBuffer(secret) || secret.byteLength !== 32) {
if (Buffer.isBuffer(secret)) secret.fill(0);
throw new ClusterAdministrationConfigurationError(
'randomBytes returned invalid secret material',
);
}
try {
secretBase64Url = secret.toString('base64url');
secretDigest = apiCredentialSecretDigest(
pepper,
request.credentialId,
secretBase64Url,
);
} finally {
secret.fill(0);
}
}
const credential: ApiCredentialRecord = {
credentialId: request.credentialId,
version: request.expectedCurrentVersion + 1,
pepperKeyId: LEGACY_API_CREDENTIAL_PEPPER_KEY_ID,
state: operation === 'revoke' ? 'revoked' : 'active',
subject: request.subject,
subjectStatus: identity.status,
secretDigest,
createdAtMs: nowMs,
notBeforeAtMs,
expiresAtMs,
};
const result = await credentials.append({
expectedCurrentVersion: request.expectedCurrentVersion,
credential,
mutation: {
mutationId: request.mutationId,
operation,
credentialId: request.credentialId,
credentialVersion: request.expectedCurrentVersion + 1,
expectedPreviousVersion: request.expectedCurrentVersion,
changedBy: principal.subject,
createdAtMs: nowMs,
},
audit: audit(
request.mutationId,
request.requestId,
`credential.${operation}`,
principal,
'credential_admin',
nowMs,
),
});
return Object.freeze({
...result,
token:
result.status === 'inserted' && secretBase64Url
? formatApiCredentialToken(request.credentialId, secretBase64Url)
: null,
});
};
return Object.freeze({
registerIdentity: (request: IdentityAdministrationRequest) =>
mutateIdentity('register', request),
enableIdentity: (request: IdentityAdministrationRequest) =>
mutateIdentity('enable', request),
disableIdentity: (request: IdentityAdministrationRequest) =>
mutateIdentity('disable', request),
issueCredential: (request: ActiveCredentialAdministrationRequest) =>
mutateCredential('issue', request),
rotateCredential: (request: ActiveCredentialAdministrationRequest) =>
mutateCredential('rotate', request),
revokeCredential: (request: CredentialAdministrationRequest) =>
mutateCredential('revoke', request),
});
}
@@ -0,0 +1,800 @@
/** Worker credential management application service boundary. */
import {
PostgresApprovalRequestRepository,
PostgresProjectPolicyRepository,
PostgresWorkerCredentialManagementPlanRepository,
} from '@qinglong/cluster-postgres/worker-credential-manager';
import type { PostgresPool } from '@qinglong/runtime-core';
import {
createApprovalRequest,
normalizeApprovalRequestRecord,
type ApprovalRequestRecord,
type CreateApprovalRequestResult,
type DecideApprovalRequestResult,
} from '@qinglong/runtime-core/approved-action';
import { ProjectPolicyEngine } from '@qinglong/runtime-core/project-policy';
import {
normalizeSecurityPrincipal,
type SecurityPolicyFence,
type SecurityPrincipal,
type SecuritySubject,
} from '@qinglong/runtime-core/security';
import type { SecurityAuditRecord } from '@qinglong/runtime-core/security-audit';
import {
InvalidWorkerCredentialManagementPlanError,
MAX_WORKER_CREDENTIAL_MANAGEMENT_PLAN_LIFETIME_MS,
WorkerCredentialManagementPlanConflictError,
WorkerCredentialManagementPlanUnavailableError,
createWorkerCredentialManagementPlan,
normalizeWorkerCredentialManagementPlan,
type CreateWorkerCredentialManagementPlanResult,
type WorkerCredentialManagementAction,
type WorkerCredentialManagementPlan,
} from '@qinglong/runtime-core/worker-credential-management-plan';
const DEFAULT_APPROVAL_LIFETIME_MS = 15 * 60 * 1000;
const ACTION_REF_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:/-]{0,254}$/;
const IDENTIFIER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const REASON_PATTERN = /^[a-z][a-z0-9_]{0,63}$/;
const STRONG_USER_ASSURANCES = new Set([
'multi_factor',
'hardware',
'local_console',
]);
export class WorkerCredentialManagementRequestError extends TypeError {
readonly code = 'WORKER_CREDENTIAL_MANAGEMENT_REQUEST_INVALID';
constructor(message: string) {
super(`Worker credential management request is invalid: ${message}`);
this.name = 'WorkerCredentialManagementRequestError';
}
}
export class WorkerCredentialManagementAuthorizationError extends Error {
readonly code = 'WORKER_CREDENTIAL_MANAGEMENT_FORBIDDEN';
constructor() {
super('Worker credential management is not authorized');
this.name = 'WorkerCredentialManagementAuthorizationError';
}
}
export class WorkerCredentialManagementConflictError extends Error {
readonly code = 'WORKER_CREDENTIAL_MANAGEMENT_CONFLICT';
constructor(message: string) {
super(
`Worker credential management conflicts with durable state: ${message}`,
);
this.name = 'WorkerCredentialManagementConflictError';
}
}
export class WorkerCredentialManagementUnavailableError extends Error {
readonly code = 'WORKER_CREDENTIAL_MANAGEMENT_UNAVAILABLE';
constructor(options?: ErrorOptions) {
super('Worker credential management is unavailable', options);
this.name = 'WorkerCredentialManagementUnavailableError';
}
}
export class WorkerCredentialManagementQuotaExceededError extends Error {
readonly code = 'WORKER_CREDENTIAL_MANAGEMENT_QUOTA_EXCEEDED';
constructor(readonly retryAfterMs: number) {
super('Worker credential management quota is exceeded');
this.name = 'WorkerCredentialManagementQuotaExceededError';
}
}
export type WorkerCredentialManagementQuotaOperation =
| 'worker-credential.plan'
| 'worker-credential.propose'
| 'worker-credential.decide'
| 'worker-credential.inspect';
export interface WorkerCredentialManagementQuotaPort {
consume(
command: Readonly<{
projectId: string;
subject: Readonly<SecuritySubject>;
operation: WorkerCredentialManagementQuotaOperation;
idempotencyKey: string;
}>,
): Promise<
Readonly<{
admitted: boolean;
retryAfterMs: number | null;
}>
>;
}
export interface PlanClusterWorkerCredentialRequest {
readonly actionRef: string;
readonly authorityProjectId: string;
readonly action: WorkerCredentialManagementAction;
readonly deliveryId: string;
readonly workerId: string;
readonly credentialId: string;
readonly previousCredentialId: string | null;
readonly credentialNotBeforeAtMs: number;
readonly credentialExpiresAtMs: number;
readonly deploymentTargetDigest: string;
readonly deploymentGeneration: string;
readonly principal: SecurityPrincipal;
}
export interface ProposeClusterWorkerCredentialRequest {
readonly actionRef: string;
readonly authorityProjectId: string;
readonly approvalRequestId: string;
readonly approvalAuditEventId: string;
readonly principal: SecurityPrincipal;
}
export interface ProposeClusterWorkerCredentialResult {
readonly plan: Readonly<WorkerCredentialManagementPlan>;
readonly approvalStatus: CreateApprovalRequestResult['status'];
readonly approvalRequest: Readonly<ApprovalRequestRecord>;
}
export interface DecideClusterWorkerCredentialRequest {
readonly actionRef: string;
readonly authorityProjectId: string;
readonly approvalRequestId: string;
readonly expectedVersion: number;
readonly decisionId: string;
readonly auditEventId: string;
readonly decision: 'approved' | 'rejected';
readonly reasonCode: string;
readonly principal: SecurityPrincipal;
}
export interface InspectClusterWorkerCredentialRequest {
readonly actionRef: string;
readonly authorityProjectId: string;
readonly approvalRequestId: string;
readonly inspectionId: string;
readonly principal: SecurityPrincipal;
}
export interface InspectClusterWorkerCredentialResult {
readonly plan: Readonly<WorkerCredentialManagementPlan> | null;
readonly approvalRequest: Readonly<ApprovalRequestRecord> | null;
readonly stale: boolean;
}
export interface ClusterWorkerCredentialManagementService {
plan(
request: PlanClusterWorkerCredentialRequest,
): Promise<Readonly<CreateWorkerCredentialManagementPlanResult>>;
propose(
request: ProposeClusterWorkerCredentialRequest,
): Promise<Readonly<ProposeClusterWorkerCredentialResult>>;
decide(
request: DecideClusterWorkerCredentialRequest,
): Promise<Readonly<DecideApprovalRequestResult>>;
inspectAuthorized(
request: InspectClusterWorkerCredentialRequest,
): Promise<Readonly<InspectClusterWorkerCredentialResult>>;
}
export interface ClusterWorkerCredentialManagementOptions {
readonly pool: PostgresPool;
readonly quota?: WorkerCredentialManagementQuotaPort;
readonly now?: () => number;
readonly planLifetimeMs?: number;
readonly approvalLifetimeMs?: number;
}
function exact(value: unknown, keys: readonly string[], label: string): void {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw new WorkerCredentialManagementRequestError(
`${label} must be an object`,
);
}
const actual = Object.keys(value).sort();
const expected = [...keys].sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
throw new WorkerCredentialManagementRequestError(
`${label} shape is invalid`,
);
}
}
function identifier(value: unknown, label: string): string {
if (typeof value !== 'string' || !IDENTIFIER_PATTERN.test(value)) {
throw new WorkerCredentialManagementRequestError(`${label} is invalid`);
}
return value;
}
function actionRef(value: unknown): string {
if (typeof value !== 'string' || !ACTION_REF_PATTERN.test(value)) {
throw new WorkerCredentialManagementRequestError('actionRef is invalid');
}
return value;
}
function currentTime(now: () => number): number {
const value = now();
if (!Number.isSafeInteger(value) || value < 0) {
throw new WorkerCredentialManagementUnavailableError();
}
return value;
}
function same(left: unknown, right: unknown): boolean {
return JSON.stringify(left) === JSON.stringify(right);
}
function sameSubject(
left: Readonly<SecuritySubject>,
right: Readonly<SecuritySubject>,
): boolean {
return left.type === right.type && left.id === right.id;
}
function binding(plan: Readonly<WorkerCredentialManagementPlan>) {
return Object.freeze({
permission: 'worker.manage' as const,
actionType: `worker_credential.delivery.${plan.action}`,
actionRef: plan.actionRef,
actionDigest: plan.planDigest,
previewDigest: plan.previewDigest,
});
}
function audit(
eventId: string,
requestId: string,
operationId: 'approval.request' | 'approval.decide',
projectId: string,
principal: Readonly<SecurityPrincipal>,
outcome: 'allowed' | 'approval_required',
fence: Readonly<SecurityPolicyFence>,
occurredAtMs: number,
): Readonly<SecurityAuditRecord> {
return Object.freeze({
eventId,
requestId,
operationId,
projectId,
subject: principal.subject,
authenticationId: principal.authenticationId,
outcome,
reasons: Object.freeze(['worker_credential_review']),
fence,
occurredAtMs,
});
}
export function createClusterWorkerCredentialManagementService(
options: ClusterWorkerCredentialManagementOptions,
): Readonly<ClusterWorkerCredentialManagementService> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
key !== 'approvalLifetimeMs' &&
key !== 'now' &&
key !== 'planLifetimeMs' &&
key !== 'pool' &&
key !== 'quota',
)
) {
throw new WorkerCredentialManagementRequestError(
'options shape is invalid',
);
}
if (!options.pool || typeof options.pool.query !== 'function') {
throw new WorkerCredentialManagementRequestError('pool is invalid');
}
if (options.now !== undefined && typeof options.now !== 'function') {
throw new WorkerCredentialManagementRequestError('now is invalid');
}
if (
options.quota !== undefined &&
(!options.quota || typeof options.quota.consume !== 'function')
) {
throw new WorkerCredentialManagementRequestError('quota is invalid');
}
const planLifetimeMs =
options.planLifetimeMs ?? MAX_WORKER_CREDENTIAL_MANAGEMENT_PLAN_LIFETIME_MS;
const approvalLifetimeMs =
options.approvalLifetimeMs ?? DEFAULT_APPROVAL_LIFETIME_MS;
if (
!Number.isSafeInteger(planLifetimeMs) ||
planLifetimeMs < 1_000 ||
planLifetimeMs > MAX_WORKER_CREDENTIAL_MANAGEMENT_PLAN_LIFETIME_MS ||
!Number.isSafeInteger(approvalLifetimeMs) ||
approvalLifetimeMs < 1_000 ||
approvalLifetimeMs > DEFAULT_APPROVAL_LIFETIME_MS
) {
throw new WorkerCredentialManagementRequestError('lifetime is invalid');
}
const now = options.now ?? Date.now;
const plans = new PostgresWorkerCredentialManagementPlanRepository(
options.pool,
);
const approvals = new PostgresApprovalRequestRepository(options.pool);
const policy = new ProjectPolicyEngine(
new PostgresProjectPolicyRepository(options.pool),
);
const consumeQuota = async (
projectId: string,
principal: Readonly<SecurityPrincipal>,
operation: WorkerCredentialManagementQuotaOperation,
idempotencyKey: string,
): Promise<void> => {
if (!options.quota) return;
try {
const result = await options.quota.consume({
projectId,
subject: principal.subject,
operation,
idempotencyKey,
});
if (
!result ||
typeof result !== 'object' ||
typeof result.admitted !== 'boolean' ||
(result.retryAfterMs !== null &&
(!Number.isSafeInteger(result.retryAfterMs) ||
result.retryAfterMs < 1))
) {
throw new Error('quota result is invalid');
}
if (!result.admitted) {
if (result.retryAfterMs === null) {
throw new Error('quota rejection has no retry bound');
}
throw new WorkerCredentialManagementQuotaExceededError(
result.retryAfterMs,
);
}
} catch (error) {
if (error instanceof WorkerCredentialManagementQuotaExceededError) {
throw error;
}
throw new WorkerCredentialManagementUnavailableError({
cause: error instanceof Error ? error : undefined,
});
}
};
const authorize = async (
principalValue: SecurityPrincipal,
projectId: string,
permission: 'worker.manage' | 'approval.decide',
observedAtMs: number,
): Promise<
Readonly<{
principal: Readonly<SecurityPrincipal>;
fence: Readonly<SecurityPolicyFence>;
}>
> => {
let principal: Readonly<SecurityPrincipal>;
try {
principal = normalizeSecurityPrincipal(principalValue, observedAtMs);
} catch {
throw new WorkerCredentialManagementAuthorizationError();
}
if (
principal.subject.type !== 'user' ||
!STRONG_USER_ASSURANCES.has(principal.assurance)
) {
throw new WorkerCredentialManagementAuthorizationError();
}
let decision;
try {
decision = await policy.authorize(principal, projectId, permission);
} catch (error) {
throw new WorkerCredentialManagementUnavailableError({
cause: error instanceof Error ? error : undefined,
});
}
if (decision.effect !== 'allow' || decision.fence === null) {
throw new WorkerCredentialManagementAuthorizationError();
}
return Object.freeze({ principal, fence: decision.fence });
};
const loadPlan = async (
requestedActionRef: string,
): Promise<Readonly<WorkerCredentialManagementPlan>> => {
let value;
try {
value = await plans.findByActionRef(actionRef(requestedActionRef));
} catch (error) {
throw new WorkerCredentialManagementUnavailableError({
cause: error instanceof Error ? error : undefined,
});
}
if (!value) {
throw new WorkerCredentialManagementConflictError('plan does not exist');
}
return normalizeWorkerCredentialManagementPlan(value);
};
return Object.freeze({
async plan(request: PlanClusterWorkerCredentialRequest) {
exact(
request,
[
'action',
'actionRef',
'authorityProjectId',
'credentialExpiresAtMs',
'credentialId',
'credentialNotBeforeAtMs',
'deliveryId',
'deploymentGeneration',
'deploymentTargetDigest',
'previousCredentialId',
'principal',
'workerId',
],
'plan request',
);
const observedAtMs = currentTime(now);
const authorization = await authorize(
request.principal,
identifier(request.authorityProjectId, 'authorityProjectId'),
'worker.manage',
observedAtMs,
);
await consumeQuota(
request.authorityProjectId,
authorization.principal,
'worker-credential.plan',
actionRef(request.actionRef),
);
try {
const plan = createWorkerCredentialManagementPlan({
actionRef: actionRef(request.actionRef),
authorityProjectId: request.authorityProjectId,
action: request.action,
target: {
deliveryId: request.deliveryId,
workerId: request.workerId,
credentialId: request.credentialId,
previousCredentialId: request.previousCredentialId,
credentialNotBeforeAtMs: request.credentialNotBeforeAtMs,
credentialExpiresAtMs: request.credentialExpiresAtMs,
deploymentTargetDigest: request.deploymentTargetDigest,
deploymentGeneration: request.deploymentGeneration,
},
requestedBy: authorization.principal.subject,
plannedAtMs: observedAtMs,
expiresAtMs: observedAtMs + planLifetimeMs,
});
return await plans.create(plan);
} catch (error) {
if (error instanceof InvalidWorkerCredentialManagementPlanError) {
throw new WorkerCredentialManagementRequestError('plan is invalid');
}
if (error instanceof WorkerCredentialManagementPlanConflictError) {
throw new WorkerCredentialManagementConflictError(
'plan identity is already bound',
);
}
throw new WorkerCredentialManagementUnavailableError({
cause:
error instanceof WorkerCredentialManagementPlanUnavailableError
? error
: error instanceof Error
? error
: undefined,
});
}
},
async propose(request: ProposeClusterWorkerCredentialRequest) {
exact(
request,
[
'actionRef',
'authorityProjectId',
'approvalAuditEventId',
'approvalRequestId',
'principal',
],
'proposal request',
);
const approvalRequestId = identifier(
request.approvalRequestId,
'approvalRequestId',
);
const approvalAuditEventId = identifier(
request.approvalAuditEventId,
'approvalAuditEventId',
);
const observedAtMs = currentTime(now);
const authorityProjectId = identifier(
request.authorityProjectId,
'authorityProjectId',
);
const authorization = await authorize(
request.principal,
authorityProjectId,
'worker.manage',
observedAtMs,
);
await consumeQuota(
authorityProjectId,
authorization.principal,
'worker-credential.propose',
approvalRequestId,
);
const plan = await loadPlan(request.actionRef);
if (plan.authorityProjectId !== authorityProjectId) {
throw new WorkerCredentialManagementConflictError(
'plan belongs to another authority Project',
);
}
if (observedAtMs > plan.expiresAtMs) {
throw new WorkerCredentialManagementConflictError('plan expired');
}
if (!sameSubject(plan.requestedBy, authorization.principal.subject)) {
throw new WorkerCredentialManagementAuthorizationError();
}
const action = binding(plan);
let existing;
try {
existing = await approvals.findById(approvalRequestId);
} catch (error) {
throw new WorkerCredentialManagementUnavailableError({
cause: error instanceof Error ? error : undefined,
});
}
if (existing) {
const normalized = normalizeApprovalRequestRecord(existing);
if (
normalized.projectId !== plan.authorityProjectId ||
normalized.decisionMode !== 'separation_of_duty' ||
!sameSubject(normalized.requestedBy, plan.requestedBy) ||
!same(normalized.action, action)
) {
throw new WorkerCredentialManagementConflictError(
'approval is bound to another plan',
);
}
return Object.freeze({
plan,
approvalStatus: 'existing' as const,
approvalRequest: normalized,
});
}
const expiresAtMs = Math.min(
observedAtMs + approvalLifetimeMs,
plan.expiresAtMs,
);
if (expiresAtMs <= observedAtMs) {
throw new WorkerCredentialManagementConflictError(
'plan has no approval lifetime',
);
}
const created = await approvals.create({
request: createApprovalRequest({
id: approvalRequestId,
projectId: plan.authorityProjectId,
action,
risk: 'high',
decisionMode: 'separation_of_duty',
requestedBy: authorization.principal.subject,
requestedAtMs: observedAtMs,
expiresAtMs,
requestFence: authorization.fence,
}),
audit: audit(
approvalAuditEventId,
approvalRequestId,
'approval.request',
plan.authorityProjectId,
authorization.principal,
'approval_required',
authorization.fence,
observedAtMs,
),
});
return Object.freeze({
plan,
approvalStatus: created.status,
approvalRequest: created.request,
});
},
async decide(request: DecideClusterWorkerCredentialRequest) {
exact(
request,
[
'actionRef',
'authorityProjectId',
'approvalRequestId',
'auditEventId',
'decision',
'decisionId',
'expectedVersion',
'principal',
'reasonCode',
],
'decision request',
);
if (
(request.decision !== 'approved' && request.decision !== 'rejected') ||
typeof request.reasonCode !== 'string' ||
!REASON_PATTERN.test(request.reasonCode) ||
!Number.isSafeInteger(request.expectedVersion) ||
request.expectedVersion < 1
) {
throw new WorkerCredentialManagementRequestError(
'decision tuple is invalid',
);
}
const approvalRequestId = identifier(
request.approvalRequestId,
'approvalRequestId',
);
const observedAtMs = currentTime(now);
const authorityProjectId = identifier(
request.authorityProjectId,
'authorityProjectId',
);
const authorization = await authorize(
request.principal,
authorityProjectId,
'approval.decide',
observedAtMs,
);
const decisionId = identifier(request.decisionId, 'decisionId');
await consumeQuota(
authorityProjectId,
authorization.principal,
'worker-credential.decide',
decisionId,
);
const plan = await loadPlan(request.actionRef);
if (plan.authorityProjectId !== authorityProjectId) {
throw new WorkerCredentialManagementConflictError(
'plan belongs to another authority Project',
);
}
const current = await approvals.findById(approvalRequestId);
if (!current) {
throw new WorkerCredentialManagementConflictError(
'approval does not exist',
);
}
const approval = normalizeApprovalRequestRecord(current);
if (
approval.projectId !== plan.authorityProjectId ||
!same(approval.action, binding(plan))
) {
throw new WorkerCredentialManagementConflictError(
'approval does not match plan',
);
}
if (
approval.decisionId === decisionId &&
approval.decision === request.decision &&
approval.decisionReasonCode === request.reasonCode &&
approval.decidedBy &&
sameSubject(approval.decidedBy, authorization.principal.subject)
) {
return Object.freeze({
status: 'existing' as const,
request: approval,
});
}
return approvals.decide({
requestId: approvalRequestId,
expectedVersion: request.expectedVersion,
decisionId,
decision: request.decision,
reasonCode: request.reasonCode,
principal: authorization.principal,
decidedAtMs: observedAtMs,
authorizationFence: authorization.fence,
audit: audit(
identifier(request.auditEventId, 'auditEventId'),
approvalRequestId,
'approval.decide',
approval.projectId,
authorization.principal,
'allowed',
authorization.fence,
observedAtMs,
),
});
},
async inspectAuthorized(request: InspectClusterWorkerCredentialRequest) {
exact(
request,
[
'actionRef',
'authorityProjectId',
'approvalRequestId',
'inspectionId',
'principal',
],
'inspection request',
);
const inspectionId = identifier(request.inspectionId, 'inspectionId');
const projectId = identifier(
request.authorityProjectId,
'authorityProjectId',
);
const observedAtMs = currentTime(now);
let authorization;
try {
authorization = await authorize(
request.principal,
projectId,
'worker.manage',
observedAtMs,
);
} catch (error) {
if (!(error instanceof WorkerCredentialManagementAuthorizationError)) {
throw error;
}
authorization = await authorize(
request.principal,
projectId,
'approval.decide',
observedAtMs,
);
}
await consumeQuota(
projectId,
authorization.principal,
'worker-credential.inspect',
inspectionId,
);
const [planValue, approvalValue] = await Promise.all([
plans.findByActionRef(actionRef(request.actionRef)),
approvals.findById(
identifier(request.approvalRequestId, 'approvalRequestId'),
),
]);
if (!planValue && !approvalValue) {
throw new WorkerCredentialManagementConflictError(
'management state does not exist',
);
}
const plan = planValue
? normalizeWorkerCredentialManagementPlan(planValue)
: null;
const approval = approvalValue
? normalizeApprovalRequestRecord(approvalValue)
: null;
if (
(plan && plan.authorityProjectId !== projectId) ||
(approval && approval.projectId !== projectId)
) {
throw new WorkerCredentialManagementConflictError(
'management state belongs to another authority Project',
);
}
return Object.freeze({
plan,
approvalRequest: approval,
stale:
plan === null ||
approval === null ||
approval.projectId !== plan.authorityProjectId ||
!same(approval.action, binding(plan)) ||
observedAtMs > plan.expiresAtMs,
});
},
});
}
@@ -0,0 +1,111 @@
#!/usr/bin/env node
/** Worker credential management service CLI boundary. */
import {
startClusterWorkerCredentialManagementProcess,
type ClusterWorkerCredentialManagementProcessRuntime,
} from './workerCredentialManagementProcess';
const USAGE = 'Usage: ql3-worker-credential-manage';
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as {
readonly name?: unknown;
readonly code?: unknown;
};
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-worker-credential-management',
event: 'management_failed',
name:
typeof candidate?.name === 'string' && candidate.name.length <= 128
? candidate.name
: 'Error',
...(typeof candidate?.code === 'string' && candidate.code.length <= 128
? { code: candidate.code }
: {}),
});
}
function emit(value: Readonly<Record<string, unknown>>): void {
process.stdout.write(`${JSON.stringify(value)}\n`);
}
async function run(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
if (argv.length !== 0) {
process.stderr.write(
`${JSON.stringify({
code: 'QL3_WORKER_CREDENTIAL_MANAGEMENT_CLI_USAGE_INVALID',
message: USAGE,
})}\n`,
);
process.exitCode = 64;
return;
}
let runtime: Readonly<ClusterWorkerCredentialManagementProcessRuntime>;
try {
runtime = await startClusterWorkerCredentialManagementProcess({
environment: process.env,
onError() {
emit({
schemaVersion: 1,
component: 'qinglong3-worker-credential-management',
event: 'management_unavailable',
});
},
});
} catch (error) {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
return;
}
if (runtime.status === 'disabled') {
emit({
schemaVersion: 1,
component: 'qinglong3-worker-credential-management',
event: 'management_disabled',
});
return;
}
emit({
schemaVersion: 1,
component: 'qinglong3-worker-credential-management',
event: 'management_started',
address: runtime.address,
identityGeneration: runtime.identity.generation,
databaseContractVersion: runtime.database.contractVersion,
databaseMigrationCount: runtime.database.migrationIds.length,
});
let stopping: Promise<void> | undefined;
const stop = (): Promise<void> => {
stopping ??= runtime.close().then(() => {
emit({
schemaVersion: 1,
component: 'qinglong3-worker-credential-management',
event: 'management_stopped',
});
});
return stopping;
};
for (const signal of ['SIGINT', 'SIGTERM'] as const) {
process.once(signal, () => {
void stop().then(
() => {
process.exitCode = 0;
},
(error) => {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
},
);
});
}
}
void run(process.argv.slice(2));
@@ -0,0 +1,46 @@
/** TLS 1.3 Worker credential management HTTP adapter boundary. */
import {
CLUSTER_WORKER_CREDENTIAL_MANAGEMENT_PATH,
startClusterPluginPackageManagementHttp,
type ClusterPluginPackageManagementHttpApplication,
type ClusterPluginPackageManagementHttpLimits,
} from '../../management-support/pluginPackageManagementHttp';
import type { ClusterPluginPackageIdentityKeysetFile } from '../../management-support/pluginPackageIdentityKeyset';
import type { ClusterWorkerCredentialManagementTransport } from './workerCredentialManagementTransport';
export type ClusterWorkerCredentialManagementHttpLimits =
ClusterPluginPackageManagementHttpLimits;
export type ClusterWorkerCredentialManagementHttpApplication =
ClusterPluginPackageManagementHttpApplication;
export interface StartClusterWorkerCredentialManagementHttpOptions {
readonly host: string;
readonly port: number;
readonly tls: Readonly<{
readonly privateKey: Buffer;
readonly certificate: Buffer;
readonly clientCertificateAuthority: Buffer;
readonly clientCertificateRevocationList: Buffer;
}>;
readonly transport: ClusterWorkerCredentialManagementTransport;
readonly identities: ClusterPluginPackageIdentityKeysetFile;
readonly limits?: ClusterWorkerCredentialManagementHttpLimits;
readonly now?: () => number;
readonly createRequestId?: () => string;
readonly onError?: (error: unknown) => void;
}
/**
* Starts the Worker credential management endpoint on the shared Cluster Admin
* TLS 1.3/OIDC boundary. The public manager process never receives credential
* delivery or Kubernetes execution capabilities.
*/
export async function startClusterWorkerCredentialManagementHttp(
options: StartClusterWorkerCredentialManagementHttpOptions,
): Promise<Readonly<ClusterWorkerCredentialManagementHttpApplication>> {
return startClusterPluginPackageManagementHttp({
...options,
managementPath: CLUSTER_WORKER_CREDENTIAL_MANAGEMENT_PATH,
});
}
@@ -0,0 +1,132 @@
/** Shared Cluster management mutual-TLS trust validation boundary. */
import { createHash, X509Certificate } from 'node:crypto';
import { createSecureContext } from 'node:tls';
import { TextDecoder } from 'node:util';
import type { ClusterManagementProcessConfigurationFailure } from '../../management-support/managementProcessSupport';
const MAX_PEM_BLOCKS = 16;
const STRICT_UTF8 = new TextDecoder('utf-8', { fatal: true });
function exactPemBlocks(
bytes: Buffer,
label: 'CERTIFICATE' | 'X509 CRL',
description: string,
failure: ClusterManagementProcessConfigurationFailure,
): readonly Buffer[] {
let value: string;
try {
value = STRICT_UTF8.decode(bytes);
} catch {
throw failure(`${description} bundle must be strict UTF-8`);
}
const pattern = new RegExp(
`-----BEGIN ${label}-----[\\s\\S]*?-----END ${label}-----`,
'g',
);
const matches = value.match(pattern);
if (!matches || matches.length < 1 || matches.length > MAX_PEM_BLOCKS) {
throw failure(
`${description} bundle must contain 1 to ${MAX_PEM_BLOCKS} PEM blocks`,
);
}
if (value.replace(pattern, '').trim() !== '') {
throw failure(`${description} bundle contains unsupported data`);
}
return Object.freeze(
matches.map((match) => Buffer.from(`${match}\n`, 'utf8')),
);
}
function validateCertificateAuthorities(
authorities: readonly Buffer[],
now: number,
failure: ClusterManagementProcessConfigurationFailure,
): void {
const fingerprints = new Set<string>();
for (const authorityBytes of authorities) {
let authority: X509Certificate;
try {
authority = new X509Certificate(authorityBytes);
} catch {
throw failure('client certificate authority is not an X.509 certificate');
}
const validFrom = Date.parse(authority.validFrom);
const validTo = Date.parse(authority.validTo);
if (
!Number.isFinite(validFrom) ||
!Number.isFinite(validTo) ||
now < validFrom ||
now >= validTo
) {
throw failure('client certificate authority is not currently valid');
}
if (!authority.ca) {
throw failure('client certificate authority is not a CA');
}
if (fingerprints.has(authority.fingerprint256)) {
throw failure('client certificate authority bundle contains a duplicate');
}
fingerprints.add(authority.fingerprint256);
}
}
function rejectDuplicateRevocationLists(
revocationLists: readonly Buffer[],
failure: ClusterManagementProcessConfigurationFailure,
): void {
const digests = new Set<string>();
for (const revocationList of revocationLists) {
const digest = createHash('sha256').update(revocationList).digest('hex');
if (digests.has(digest)) {
throw failure(
'client certificate revocation list bundle contains a duplicate',
);
}
digests.add(digest);
}
}
export function validateWorkerCredentialManagementClientTrust(
certificateAuthorityBundle: Buffer,
certificateRevocationListBundle: Buffer,
now: number,
failure: ClusterManagementProcessConfigurationFailure,
): void {
if (!Number.isSafeInteger(now) || now < 0) {
throw failure('TLS observation time is invalid');
}
const authorities = exactPemBlocks(
certificateAuthorityBundle,
'CERTIFICATE',
'client certificate authority',
failure,
);
let revocationLists: readonly Buffer[] = Object.freeze([]);
try {
revocationLists = exactPemBlocks(
certificateRevocationListBundle,
'X509 CRL',
'client certificate revocation list',
failure,
);
validateCertificateAuthorities(authorities, now, failure);
rejectDuplicateRevocationLists(revocationLists, failure);
try {
createSecureContext({
ca: [...authorities],
crl: [...revocationLists],
minVersion: 'TLSv1.3',
maxVersion: 'TLSv1.3',
});
} catch {
throw failure('client trust or revocation bundle is invalid');
}
} finally {
for (const authority of authorities) authority.fill(0);
for (const revocationList of revocationLists) revocationList.fill(0);
}
}
export const validateClusterManagementClientTrust =
validateWorkerCredentialManagementClientTrust;
@@ -0,0 +1,673 @@
/** Worker credential management PostgreSQL process composition boundary. */
import type {
OpenPostgresDatabase,
PostgresDatabaseResource,
} from '@qinglong/runtime-core';
import {
assertPostgresWorkerCredentialManagerSchemaReady,
createPostgresDatabaseOpener,
isPostgresTlsDnsServername,
loadPostgresCertificateAuthorityFile,
loadPostgresConnectionEnvironment,
PostgresWorkerCredentialManagementIdentityKeysetLedgerRepository,
PostgresWorkerCredentialManagementQuotaRepository,
type PostgresConnectionOptions,
type PostgresPoolOptions,
type PostgresSchemaReadinessReport,
} from '@qinglong/cluster-postgres/worker-credential-manager';
import {
absoluteManagementEnvironmentFile,
booleanManagementEnvironmentValue,
boundedManagementEnvironmentValue,
integerManagementEnvironmentValue,
readManagementTlsFile,
} from '../../management-support/managementProcessSupport';
import {
createClusterWorkerCredentialIdentityKeysetFile,
type ClusterPluginPackageIdentityKeysetFile,
type ClusterPluginPackageIdentityKeysetSnapshot,
} from '../../management-support/pluginPackageIdentityKeyset';
import { createClusterWorkerCredentialManagementService } from './workerCredentialManagement';
import {
startClusterWorkerCredentialManagementHttp,
type ClusterWorkerCredentialManagementHttpApplication,
type StartClusterWorkerCredentialManagementHttpOptions,
} from './workerCredentialManagementHttp';
import { createClusterWorkerCredentialManagementTransport } from './workerCredentialManagementTransport';
import { validateWorkerCredentialManagementClientTrust } from './workerCredentialManagementMutualTls';
const SAFE_HOST = /^[A-Za-z0-9][A-Za-z0-9.:-]{0,254}$/;
const SAFE_APPLICATION_NAME = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,62}$/;
export type ClusterWorkerCredentialManagementProcessEnvironment = Readonly<
Record<string, string | undefined>
>;
export type ClusterWorkerCredentialManagementProcessConfig =
| Readonly<{ enabled: false }>
| Readonly<{
enabled: true;
profile: 'cluster-admin';
host: string;
port: number;
certificateFile: string;
privateKeyFile: string;
clientCertificateAuthorityFile: string;
clientCertificateRevocationListFile: string;
identityKeysetFile: string;
planLifetimeMs: number;
approvalLifetimeMs: number;
quota: Readonly<{
windowMs: number;
planLimit: number;
proposeLimit: number;
decideLimit: number;
inspectLimit: number;
}>;
http: Readonly<{
maxBodyBytes: number;
maxConnections: number;
maxConcurrentRequests: number;
requestTimeoutMs: number;
drainTimeoutMs: number;
rateWindowMs: number;
peerRequestLimit: number;
globalRequestLimit: number;
maxRateLimitPeers: number;
}>;
database: Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}>;
}>;
export type ClusterWorkerCredentialManagementProcessRuntime =
| Readonly<{
status: 'disabled';
close(): Promise<void>;
}>
| Readonly<{
status: 'active';
address: Readonly<{ host: string; port: number }>;
database: PostgresSchemaReadinessReport;
identity: ClusterPluginPackageIdentityKeysetSnapshot;
availabilityStatus(): 'ready' | 'unavailable' | 'stopped';
close(): Promise<void>;
}>;
export interface StartClusterWorkerCredentialManagementProcessOptions {
readonly environment: ClusterWorkerCredentialManagementProcessEnvironment;
readonly openDatabase?: OpenPostgresDatabase;
readonly identities?: ClusterPluginPackageIdentityKeysetFile;
readonly assertReady?: (
pool: PostgresDatabaseResource['pool'],
) => Promise<PostgresSchemaReadinessReport>;
readonly startHttp?: (
options: StartClusterWorkerCredentialManagementHttpOptions,
) => Promise<Readonly<ClusterWorkerCredentialManagementHttpApplication>>;
readonly now?: () => number;
readonly onError?: (error: unknown) => void;
}
export class ClusterWorkerCredentialManagementProcessConfigError extends TypeError {
readonly code = 'QL3_WORKER_CREDENTIAL_MANAGEMENT_PROCESS_CONFIG_INVALID';
constructor(message: string) {
super(
`Worker credential management process configuration is invalid: ${message}`,
);
this.name = 'ClusterWorkerCredentialManagementProcessConfigError';
}
}
function configFailure(
message: string,
): ClusterWorkerCredentialManagementProcessConfigError {
return new ClusterWorkerCredentialManagementProcessConfigError(message);
}
function boundedValue(
environment: ClusterWorkerCredentialManagementProcessEnvironment,
name: string,
maximumLength: number,
required = false,
): string | undefined {
return boundedManagementEnvironmentValue(
environment,
name,
maximumLength,
configFailure,
required,
);
}
function booleanValue(
environment: ClusterWorkerCredentialManagementProcessEnvironment,
name: string,
): boolean {
return booleanManagementEnvironmentValue(environment, name, configFailure);
}
function integerValue(
environment: ClusterWorkerCredentialManagementProcessEnvironment,
name: string,
fallback: number,
minimum: number,
maximum: number,
): number {
return integerManagementEnvironmentValue(
environment,
name,
fallback,
minimum,
maximum,
configFailure,
);
}
function absoluteFile(
environment: ClusterWorkerCredentialManagementProcessEnvironment,
name: string,
): string {
return absoluteManagementEnvironmentFile(environment, name, configFailure);
}
function loadConnection(
environment: ClusterWorkerCredentialManagementProcessEnvironment,
): Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}> {
let connection: PostgresConnectionOptions;
try {
connection = loadPostgresConnectionEnvironment(environment, {
connectionString: 'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_URL',
host: 'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_HOST',
port: 'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_PORT',
database: 'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_DATABASE',
user: 'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_USER',
password: 'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_PASSWORD',
});
} catch (error) {
throw configFailure(
error instanceof Error
? error.message
: 'PostgreSQL Worker credential manager connection is invalid',
);
}
const mode =
environment.QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_TLS_MODE ??
'verify-full';
if (mode !== 'verify-full' && mode !== 'disable') {
throw configFailure(
'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_TLS_MODE must be verify-full or disable',
);
}
if (
mode === 'disable' &&
!booleanValue(
environment,
'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_ALLOW_INSECURE',
)
) {
throw configFailure(
'disabling Worker credential manager PostgreSQL TLS requires QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_ALLOW_INSECURE=true',
);
}
const servername = boundedValue(
environment,
'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_TLS_SERVERNAME',
253,
);
if (mode === 'verify-full' && !isPostgresTlsDnsServername(servername)) {
throw configFailure(
'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_TLS_SERVERNAME must be an explicit DNS name',
);
}
const caFile = boundedValue(
environment,
'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_TLS_CA_FILE',
4_096,
);
if (mode === 'disable' && caFile !== undefined) {
throw configFailure(
'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_TLS_CA_FILE cannot be used when TLS is disabled',
);
}
let ca: string | undefined;
if (caFile !== undefined) {
try {
ca = loadPostgresCertificateAuthorityFile(caFile);
} catch {
throw configFailure(
'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_TLS_CA_FILE is invalid',
);
}
}
const applicationName =
boundedValue(
environment,
'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_APPLICATION_NAME',
63,
) ?? 'qinglong3-worker-credential-manager';
if (!SAFE_APPLICATION_NAME.test(applicationName)) {
throw configFailure(
'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_APPLICATION_NAME is invalid',
);
}
return Object.freeze({
connection: Object.freeze({
...connection,
tls:
mode === 'disable'
? { mode: 'disable' as const }
: {
mode: 'verify-full' as const,
servername: servername!,
...(ca === undefined ? {} : { ca }),
},
}),
pool: Object.freeze({
applicationName,
maxConnections: integerValue(
environment,
'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_MAX_CONNECTIONS',
2,
1,
4,
),
connectionTimeoutMs: integerValue(
environment,
'QL3_POSTGRES_WORKER_CREDENTIAL_MANAGER_CONNECTION_TIMEOUT_MS',
5_000,
100,
60_000,
),
}),
});
}
export function loadClusterWorkerCredentialManagementProcessConfig(
environment: ClusterWorkerCredentialManagementProcessEnvironment,
): Readonly<ClusterWorkerCredentialManagementProcessConfig> {
if (!environment || typeof environment !== 'object') {
throw configFailure('environment is invalid');
}
if (!booleanValue(environment, 'QL3_WORKER_CREDENTIAL_MANAGEMENT_ENABLED')) {
return Object.freeze({ enabled: false as const });
}
if (environment.QL3_PROFILE !== 'cluster-admin') {
throw configFailure(
'QL3_PROFILE must be cluster-admin when Worker credential management is enabled',
);
}
const host =
boundedValue(environment, 'QL3_WORKER_CREDENTIAL_MANAGEMENT_HOST', 255) ??
'0.0.0.0';
if (!SAFE_HOST.test(host)) {
throw configFailure('QL3_WORKER_CREDENTIAL_MANAGEMENT_HOST is invalid');
}
const http = Object.freeze({
maxBodyBytes: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_MAX_BODY_BYTES',
64 * 1024,
1_024,
256 * 1024,
),
maxConnections: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_MAX_CONNECTIONS',
64,
1,
512,
),
maxConcurrentRequests: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_MAX_CONCURRENT_REQUESTS',
32,
1,
256,
),
requestTimeoutMs: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_REQUEST_TIMEOUT_MS',
10_000,
1_000,
60_000,
),
drainTimeoutMs: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_DRAIN_TIMEOUT_MS',
5_000,
100,
60_000,
),
rateWindowMs: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_RATE_WINDOW_MS',
60_000,
1_000,
5 * 60_000,
),
peerRequestLimit: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_PEER_REQUEST_LIMIT',
60,
1,
10_000,
),
globalRequestLimit: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_GLOBAL_REQUEST_LIMIT',
600,
1,
100_000,
),
maxRateLimitPeers: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_MAX_RATE_LIMIT_PEERS',
1_024,
1,
16_384,
),
});
if (http.globalRequestLimit < http.peerRequestLimit) {
throw configFailure(
'global request limit cannot be below the peer request limit',
);
}
return Object.freeze({
enabled: true as const,
profile: 'cluster-admin' as const,
host,
port: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_PORT',
8_444,
1,
65_535,
),
certificateFile: absoluteFile(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_TLS_CERT_FILE',
),
privateKeyFile: absoluteFile(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_TLS_KEY_FILE',
),
clientCertificateAuthorityFile: absoluteFile(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_CLIENT_CA_FILE',
),
clientCertificateRevocationListFile: absoluteFile(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_CLIENT_CRL_FILE',
),
identityKeysetFile: absoluteFile(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_IDENTITY_KEYSET_FILE',
),
planLifetimeMs: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_PLAN_LIFETIME_MS',
15 * 60_000,
1_000,
15 * 60_000,
),
approvalLifetimeMs: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_APPROVAL_LIFETIME_MS',
15 * 60_000,
1_000,
15 * 60_000,
),
quota: Object.freeze({
windowMs: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_QUOTA_WINDOW_MS',
60_000,
1_000,
5 * 60_000,
),
planLimit: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_PLAN_QUOTA',
30,
1,
1_000,
),
proposeLimit: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_PROPOSE_QUOTA',
30,
1,
1_000,
),
decideLimit: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_DECIDE_QUOTA',
60,
1,
1_000,
),
inspectLimit: integerValue(
environment,
'QL3_WORKER_CREDENTIAL_MANAGEMENT_INSPECT_QUOTA',
600,
1,
1_000,
),
}),
http,
database: loadConnection(environment),
});
}
export async function startClusterWorkerCredentialManagementProcess(
options: StartClusterWorkerCredentialManagementProcessOptions,
): Promise<Readonly<ClusterWorkerCredentialManagementProcessRuntime>> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
![
'environment',
'openDatabase',
'identities',
'assertReady',
'startHttp',
'now',
'onError',
].includes(key),
) ||
!options.environment ||
typeof options.environment !== 'object' ||
(options.openDatabase !== undefined &&
typeof options.openDatabase !== 'function') ||
(options.identities !== undefined &&
(typeof options.identities.reload !== 'function' ||
typeof options.identities.bind !== 'function')) ||
(options.assertReady !== undefined &&
typeof options.assertReady !== 'function') ||
(options.startHttp !== undefined &&
typeof options.startHttp !== 'function') ||
(options.now !== undefined && typeof options.now !== 'function') ||
(options.onError !== undefined && typeof options.onError !== 'function')
) {
throw configFailure('options are invalid');
}
const config = loadClusterWorkerCredentialManagementProcessConfig(
options.environment,
);
if (!config.enabled) {
return Object.freeze({
status: 'disabled' as const,
close: () => Promise.resolve(),
});
}
const now = options.now ?? Date.now;
let http:
| Readonly<ClusterWorkerCredentialManagementHttpApplication>
| undefined;
let database: PostgresDatabaseResource | undefined;
let unavailableError: unknown;
let closePromise: Promise<void> | undefined;
const report = (error: unknown): void => {
try {
options.onError?.(error);
} catch {
// Diagnostics do not own availability or cleanup.
}
};
const openDatabase =
options.openDatabase ??
createPostgresDatabaseOpener({
role: 'worker-credential-manager',
connection: config.database.connection,
pool: config.database.pool,
onPoolError(error) {
const firstAvailabilityError = unavailableError === undefined;
unavailableError ??= error;
http?.withdraw(error);
if (firstAvailabilityError) report(error);
},
});
try {
database = await openDatabase();
const evidence = await (
options.assertReady ?? assertPostgresWorkerCredentialManagerSchemaReady
)(database.pool);
if (unavailableError !== undefined) throw unavailableError;
const identities =
options.identities ??
createClusterWorkerCredentialIdentityKeysetFile({
filePath: config.identityKeysetFile,
now,
ledger:
new PostgresWorkerCredentialManagementIdentityKeysetLedgerRepository(
database.pool,
'worker-credential-management',
),
});
const identity = await identities.reload();
const quota = new PostgresWorkerCredentialManagementQuotaRepository(
database.pool,
{
windowMs: config.quota.windowMs,
limits: {
'worker-credential.plan': config.quota.planLimit,
'worker-credential.propose': config.quota.proposeLimit,
'worker-credential.decide': config.quota.decideLimit,
'worker-credential.inspect': config.quota.inspectLimit,
},
},
);
const service = createClusterWorkerCredentialManagementService({
pool: database.pool,
planLifetimeMs: config.planLifetimeMs,
approvalLifetimeMs: config.approvalLifetimeMs,
quota,
now,
});
const transport = createClusterWorkerCredentialManagementTransport({
service,
now,
});
const privateKey = readManagementTlsFile(
config.privateKeyFile,
true,
configFailure,
);
try {
const certificate = readManagementTlsFile(
config.certificateFile,
false,
configFailure,
);
const clientCertificateAuthority = readManagementTlsFile(
config.clientCertificateAuthorityFile,
false,
configFailure,
);
const clientCertificateRevocationList = readManagementTlsFile(
config.clientCertificateRevocationListFile,
false,
configFailure,
);
try {
validateWorkerCredentialManagementClientTrust(
clientCertificateAuthority,
clientCertificateRevocationList,
now(),
configFailure,
);
http = await (
options.startHttp ?? startClusterWorkerCredentialManagementHttp
)({
host: config.host,
port: config.port,
tls: {
privateKey,
certificate,
clientCertificateAuthority,
clientCertificateRevocationList,
},
transport,
identities,
limits: config.http,
now,
onError: report,
});
} finally {
clientCertificateAuthority.fill(0);
clientCertificateRevocationList.fill(0);
}
} finally {
privateKey.fill(0);
}
if (unavailableError !== undefined) {
http.withdraw(unavailableError);
}
return Object.freeze({
status: 'active' as const,
address: http.address,
database: evidence,
identity,
availabilityStatus: () => http?.availabilityStatus() ?? 'stopped',
close(): Promise<void> {
if (closePromise) return closePromise;
closePromise = (async () => {
let primaryError: unknown;
try {
await http?.close();
} catch (error) {
primaryError = error;
}
try {
await database?.close();
} catch (error) {
primaryError ??= error;
}
if (primaryError) throw primaryError;
})();
return closePromise;
},
});
} catch (error) {
try {
await http?.close();
} catch {
// Preserve startup failure.
}
try {
await database?.close();
} catch {
// Preserve startup failure.
}
throw error;
}
}
@@ -0,0 +1,426 @@
/** Authenticated Worker credential management transport boundary. */
import type { ApprovalRequestRecord } from '@qinglong/runtime-core/approved-action';
import {
normalizeSecurityPrincipal,
type SecurityPrincipal,
} from '@qinglong/runtime-core/security';
import type {
CreateWorkerCredentialManagementPlanResult,
WorkerCredentialManagementPlan,
} from '@qinglong/runtime-core/worker-credential-management-plan';
import type {
ClusterWorkerCredentialManagementService,
ProposeClusterWorkerCredentialResult,
} from './workerCredentialManagement';
const STRONG_CLUSTER_ASSURANCES = new Set(['multi_factor', 'hardware']);
export interface ClusterWorkerCredentialManagementAuthentication {
authenticate(): Promise<Readonly<SecurityPrincipal> | null>;
}
export interface PlanClusterWorkerCredentialCommand {
readonly schemaVersion: 1;
readonly operation: 'worker-credential.plan';
readonly request: {
readonly actionRef: string;
readonly authorityProjectId: string;
readonly action: 'issue' | 'rotate';
readonly deliveryId: string;
readonly workerId: string;
readonly credentialId: string;
readonly previousCredentialId: string | null;
readonly credentialNotBeforeAtMs: number;
readonly credentialExpiresAtMs: number;
readonly deploymentTargetDigest: string;
readonly deploymentGeneration: string;
};
}
export interface ProposeClusterWorkerCredentialCommand {
readonly schemaVersion: 1;
readonly operation: 'worker-credential.propose';
readonly request: {
readonly actionRef: string;
readonly authorityProjectId: string;
readonly approvalRequestId: string;
readonly approvalAuditEventId: string;
};
}
export interface DecideClusterWorkerCredentialCommand {
readonly schemaVersion: 1;
readonly operation: 'worker-credential.decide';
readonly request: {
readonly actionRef: string;
readonly authorityProjectId: string;
readonly approvalRequestId: string;
readonly expectedVersion: number;
readonly decisionId: string;
readonly auditEventId: string;
readonly decision: 'approved' | 'rejected';
readonly reasonCode: string;
};
}
export interface InspectClusterWorkerCredentialCommand {
readonly schemaVersion: 1;
readonly operation: 'worker-credential.inspect';
readonly request: {
readonly actionRef: string;
readonly authorityProjectId: string;
readonly approvalRequestId: string;
readonly inspectionId: string;
};
}
export type ClusterWorkerCredentialManagementCommand =
| PlanClusterWorkerCredentialCommand
| ProposeClusterWorkerCredentialCommand
| DecideClusterWorkerCredentialCommand
| InspectClusterWorkerCredentialCommand;
type PlanSummary = ReturnType<typeof planSummary>;
type ApprovalSummary = ReturnType<typeof approvalSummary>;
export type ClusterWorkerCredentialManagementTransportResult =
| Readonly<{
schemaVersion: 1;
operation: 'worker-credential.plan';
status: 'created' | 'existing';
plan: PlanSummary;
}>
| Readonly<{
schemaVersion: 1;
operation: 'worker-credential.propose';
approvalStatus: 'created' | 'existing';
plan: PlanSummary;
approval: ApprovalSummary;
}>
| Readonly<{
schemaVersion: 1;
operation: 'worker-credential.decide';
status: 'decided' | 'existing';
approval: ApprovalSummary;
}>
| Readonly<{
schemaVersion: 1;
operation: 'worker-credential.inspect';
plan: PlanSummary | null;
approval: ApprovalSummary | null;
stale: boolean;
}>;
export interface ClusterWorkerCredentialManagementTransport {
execute(
command: unknown,
authentication: ClusterWorkerCredentialManagementAuthentication,
): Promise<Readonly<ClusterWorkerCredentialManagementTransportResult>>;
}
export interface ClusterWorkerCredentialManagementTransportOptions {
readonly service: ClusterWorkerCredentialManagementService;
readonly now?: () => number;
}
export class ClusterWorkerCredentialManagementTransportConfigurationError extends TypeError {
readonly code = 'CLUSTER_WORKER_CREDENTIAL_TRANSPORT_CONFIGURATION_INVALID';
constructor(message: string) {
super(
`Cluster Worker credential transport configuration is invalid: ${message}`,
);
this.name = 'ClusterWorkerCredentialManagementTransportConfigurationError';
}
}
export class ClusterWorkerCredentialManagementTransportRequestError extends TypeError {
readonly code = 'CLUSTER_WORKER_CREDENTIAL_TRANSPORT_REQUEST_INVALID';
constructor(message: string) {
super(`Cluster Worker credential transport request is invalid: ${message}`);
this.name = 'ClusterWorkerCredentialManagementTransportRequestError';
}
}
export class ClusterWorkerCredentialManagementTransportAuthenticationError extends Error {
readonly code = 'CLUSTER_WORKER_CREDENTIAL_TRANSPORT_AUTHENTICATION_REQUIRED';
constructor() {
super(
'Cluster Worker credential transport requires a strong User principal',
);
this.name = 'ClusterWorkerCredentialManagementTransportAuthenticationError';
}
}
export class ClusterWorkerCredentialManagementTransportUnavailableError extends Error {
readonly code = 'CLUSTER_WORKER_CREDENTIAL_TRANSPORT_UNAVAILABLE';
constructor(readonly cause?: unknown) {
super('Cluster Worker credential transport is unavailable');
this.name = 'ClusterWorkerCredentialManagementTransportUnavailableError';
}
}
function exactObject(
value: unknown,
expectedKeys: readonly string[],
label: string,
): asserts value is Record<string, unknown> {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw new ClusterWorkerCredentialManagementTransportRequestError(
`${label} must be an object`,
);
}
const actual = Object.keys(value).sort();
const expected = [...expectedKeys].sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
throw new ClusterWorkerCredentialManagementTransportRequestError(
`${label} shape is invalid`,
);
}
}
export function normalizeClusterWorkerCredentialManagementCommand(
value: unknown,
): Readonly<ClusterWorkerCredentialManagementCommand> {
exactObject(value, ['schemaVersion', 'operation', 'request'], 'command');
if (value.schemaVersion !== 1) {
throw new ClusterWorkerCredentialManagementTransportRequestError(
'schemaVersion is invalid',
);
}
switch (value.operation) {
case 'worker-credential.plan':
exactObject(
value.request,
[
'action',
'actionRef',
'authorityProjectId',
'credentialExpiresAtMs',
'credentialId',
'credentialNotBeforeAtMs',
'deliveryId',
'deploymentGeneration',
'deploymentTargetDigest',
'previousCredentialId',
'workerId',
],
'plan request',
);
break;
case 'worker-credential.propose':
exactObject(
value.request,
[
'actionRef',
'authorityProjectId',
'approvalRequestId',
'approvalAuditEventId',
],
'proposal request',
);
break;
case 'worker-credential.decide':
exactObject(
value.request,
[
'actionRef',
'authorityProjectId',
'approvalRequestId',
'expectedVersion',
'decisionId',
'auditEventId',
'decision',
'reasonCode',
],
'decision request',
);
break;
case 'worker-credential.inspect':
exactObject(
value.request,
[
'actionRef',
'authorityProjectId',
'approvalRequestId',
'inspectionId',
],
'inspection request',
);
break;
default:
throw new ClusterWorkerCredentialManagementTransportRequestError(
'operation is not publicly available',
);
}
return Object.freeze(
value as unknown as ClusterWorkerCredentialManagementCommand,
);
}
function planSummary(plan: Readonly<WorkerCredentialManagementPlan>) {
return Object.freeze({
actionRef: plan.actionRef,
authorityProjectId: plan.authorityProjectId,
action: plan.action,
target: Object.freeze({ ...plan.target }),
requestedBy: Object.freeze({ ...plan.requestedBy }),
plannedAtMs: plan.plannedAtMs,
expiresAtMs: plan.expiresAtMs,
previewDigest: plan.previewDigest,
planDigest: plan.planDigest,
});
}
function approvalSummary(approval: Readonly<ApprovalRequestRecord>) {
return Object.freeze({
id: approval.id,
projectId: approval.projectId,
version: approval.version,
state: approval.state,
risk: approval.risk,
decisionMode: approval.decisionMode,
requestedBy: Object.freeze({ ...approval.requestedBy }),
requestedAtMs: approval.requestedAtMs,
expiresAtMs: approval.expiresAtMs,
decision: approval.decision,
decisionReasonCode: approval.decisionReasonCode,
decidedBy: approval.decidedBy
? Object.freeze({ ...approval.decidedBy })
: null,
decidedAtMs: approval.decidedAtMs,
dispatchId: approval.dispatchId,
consumedAtMs: approval.consumedAtMs,
actionType: approval.action.actionType,
actionRef: approval.action.actionRef,
actionDigest: approval.action.actionDigest,
previewDigest: approval.action.previewDigest,
});
}
export function createClusterWorkerCredentialManagementTransport(
options: ClusterWorkerCredentialManagementTransportOptions,
): Readonly<ClusterWorkerCredentialManagementTransport> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some((key) => key !== 'service' && key !== 'now') ||
!options.service ||
typeof options.service.plan !== 'function' ||
typeof options.service.propose !== 'function' ||
typeof options.service.decide !== 'function' ||
typeof options.service.inspectAuthorized !== 'function' ||
(options.now !== undefined && typeof options.now !== 'function')
) {
throw new ClusterWorkerCredentialManagementTransportConfigurationError(
'options are invalid',
);
}
const now = options.now ?? Date.now;
return Object.freeze({
async execute(
commandValue: unknown,
authentication: ClusterWorkerCredentialManagementAuthentication,
): Promise<Readonly<ClusterWorkerCredentialManagementTransportResult>> {
const command =
normalizeClusterWorkerCredentialManagementCommand(commandValue);
if (
!authentication ||
typeof authentication !== 'object' ||
Array.isArray(authentication) ||
Object.keys(authentication).some((key) => key !== 'authenticate') ||
typeof authentication.authenticate !== 'function'
) {
throw new ClusterWorkerCredentialManagementTransportConfigurationError(
'authentication authority is invalid',
);
}
let candidate: Readonly<SecurityPrincipal> | null;
try {
candidate = await authentication.authenticate();
} catch (error) {
throw new ClusterWorkerCredentialManagementTransportUnavailableError(
error,
);
}
const observedAtMs = now();
if (!Number.isSafeInteger(observedAtMs) || observedAtMs < 0) {
throw new ClusterWorkerCredentialManagementTransportUnavailableError();
}
let principal: Readonly<SecurityPrincipal>;
try {
principal = normalizeSecurityPrincipal(
candidate as SecurityPrincipal,
observedAtMs,
);
} catch {
throw new ClusterWorkerCredentialManagementTransportAuthenticationError();
}
if (
principal.subject.type !== 'user' ||
!STRONG_CLUSTER_ASSURANCES.has(principal.assurance)
) {
throw new ClusterWorkerCredentialManagementTransportAuthenticationError();
}
switch (command.operation) {
case 'worker-credential.plan': {
const result: Readonly<CreateWorkerCredentialManagementPlanResult> =
await options.service.plan({ ...command.request, principal });
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
status: result.status,
plan: planSummary(result.plan),
});
}
case 'worker-credential.propose': {
const result: Readonly<ProposeClusterWorkerCredentialResult> =
await options.service.propose({ ...command.request, principal });
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
approvalStatus: result.approvalStatus,
plan: planSummary(result.plan),
approval: approvalSummary(result.approvalRequest),
});
}
case 'worker-credential.decide': {
const result = await options.service.decide({
...command.request,
principal,
});
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
status: result.status,
approval: approvalSummary(result.request),
});
}
case 'worker-credential.inspect': {
const result = await options.service.inspectAuthorized({
...command.request,
principal,
});
return Object.freeze({
schemaVersion: 1 as const,
operation: command.operation,
plan: result.plan ? planSummary(result.plan) : null,
approval: result.approvalRequest
? approvalSummary(result.approvalRequest)
: null,
stale: result.stale,
});
}
}
},
});
}
@@ -0,0 +1,268 @@
/** Worker credential administration application boundary. */
import { randomBytes as nodeRandomBytes } from 'node:crypto';
import {
WorkerCredentialMutationConflictError,
normalizeWorkerCredentialId,
normalizeWorkerCredentialMutationId,
type AppendWorkerCredentialResult,
type WorkerCredentialAdministrationOperation,
type WorkerCredentialAdministrationRepository,
} from '@qinglong/runtime-core/worker-credential';
import {
assertWorkerCredentialPepper,
formatWorkerCredentialToken,
workerCredentialSecretDigest,
} from '@qinglong/runtime-core/worker-credential-token';
import {
normalizeSecurityPrincipal,
type SecurityPrincipal,
} from '@qinglong/runtime-core/security';
const MAX_LIFETIME_MS = 2 * 365 * 24 * 60 * 60 * 1000;
const REVOKED_DIGEST = '0'.repeat(64);
const STRONG = new Set(['multi_factor', 'hardware', 'local_console']);
const SAFE_WORKER_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const SAFE_REQUEST_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
export interface WorkerCredentialAdministrationRequest {
readonly mutationId: string;
readonly requestId: string;
readonly expectedCurrentVersion: number;
readonly credentialId: string;
readonly workerId: string;
readonly principal: SecurityPrincipal;
}
export interface ActiveWorkerCredentialAdministrationRequest
extends WorkerCredentialAdministrationRequest {
readonly notBeforeAtMs: number;
readonly expiresAtMs: number;
}
export interface WorkerCredentialAdministrationResult
extends AppendWorkerCredentialResult {
readonly token: string | null;
}
export interface WorkerCredentialAdministrationService {
issue(
request: ActiveWorkerCredentialAdministrationRequest,
): Promise<WorkerCredentialAdministrationResult>;
rotate(
request: ActiveWorkerCredentialAdministrationRequest,
): Promise<WorkerCredentialAdministrationResult>;
revoke(
request: WorkerCredentialAdministrationRequest,
): Promise<WorkerCredentialAdministrationResult>;
}
export interface WorkerCredentialAdministrationOptions {
readonly now?: () => number;
readonly randomBytes?: (size: number) => Buffer;
readonly returnToken?: boolean;
}
function exactRequest(
value: WorkerCredentialAdministrationRequest,
active: boolean,
): void {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw new TypeError('Worker credential administration request is invalid');
}
const expected = [
'mutationId',
'requestId',
'expectedCurrentVersion',
'credentialId',
'workerId',
'principal',
...(active ? ['notBeforeAtMs', 'expiresAtMs'] : []),
].sort();
const actual = Object.keys(value).sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
throw new TypeError('Worker credential administration request shape is invalid');
}
normalizeWorkerCredentialMutationId(value.mutationId);
normalizeWorkerCredentialId(value.credentialId);
if (
typeof value.requestId !== 'string' ||
!SAFE_REQUEST_ID.test(value.requestId) ||
typeof value.workerId !== 'string' ||
!SAFE_WORKER_ID.test(value.workerId)
) {
throw new TypeError('Worker credential administration request identity is invalid');
}
}
function principal(value: SecurityPrincipal, nowMs: number) {
const normalized = normalizeSecurityPrincipal(value, nowMs);
if (
!(
(normalized.subject.type === 'user' && STRONG.has(normalized.assurance)) ||
(normalized.subject.type === 'system' && normalized.assurance === 'service')
)
) {
throw new TypeError('Worker credential administration requires a strong principal');
}
return normalized;
}
function sameReplay(
operation: WorkerCredentialAdministrationOperation,
existing: Awaited<ReturnType<WorkerCredentialAdministrationRepository['resolveMutation']>>,
request: WorkerCredentialAdministrationRequest | ActiveWorkerCredentialAdministrationRequest,
): boolean {
if (!existing) return false;
const active = operation === 'revoke'
? null
: request as ActiveWorkerCredentialAdministrationRequest;
return (
existing.mutation.operation === operation &&
existing.mutation.credentialId === request.credentialId &&
existing.mutation.expectedPreviousVersion === request.expectedCurrentVersion &&
existing.credential.workerId === request.workerId &&
existing.credential.state === (operation === 'revoke' ? 'revoked' : 'active') &&
(active === null ||
(existing.credential.notBeforeAtMs === active.notBeforeAtMs &&
existing.credential.expiresAtMs === active.expiresAtMs)) &&
existing.audit.requestId === request.requestId &&
existing.audit.subject?.type === request.principal.subject.type &&
existing.audit.subject.id === request.principal.subject.id
);
}
export function createWorkerCredentialAdministrationService(
repository: WorkerCredentialAdministrationRepository,
pepper: string,
options: WorkerCredentialAdministrationOptions = {},
): WorkerCredentialAdministrationService {
if (
!repository ||
typeof repository.resolveMutation !== 'function' ||
typeof repository.append !== 'function'
) {
throw new TypeError('Worker credential administration repository is invalid');
}
assertWorkerCredentialPepper(pepper);
const now = options.now ?? Date.now;
const randomBytes = options.randomBytes ?? nodeRandomBytes;
const mutate = async (
operation: WorkerCredentialAdministrationOperation,
request: WorkerCredentialAdministrationRequest | ActiveWorkerCredentialAdministrationRequest,
): Promise<WorkerCredentialAdministrationResult> => {
exactRequest(request, operation !== 'revoke');
const nowMs = now();
if (!Number.isSafeInteger(nowMs) || nowMs < 0) {
throw new TypeError('Worker credential administration clock is invalid');
}
const actor = principal(request.principal, nowMs);
if (
(operation === 'issue' && request.expectedCurrentVersion !== 0) ||
(operation !== 'issue' &&
(!Number.isSafeInteger(request.expectedCurrentVersion) ||
request.expectedCurrentVersion < 1))
) {
throw new RangeError('Worker credential administration operation fence is invalid');
}
const existing = await repository.resolveMutation(request.mutationId);
if (existing) {
if (!sameReplay(operation, existing, request)) {
throw new WorkerCredentialMutationConflictError();
}
return Object.freeze({
status: 'existing',
credential: existing.credential,
mutation: existing.mutation,
token: null,
});
}
const active = operation === 'revoke'
? null
: request as ActiveWorkerCredentialAdministrationRequest;
const notBeforeAtMs = active?.notBeforeAtMs ?? nowMs;
const expiresAtMs = active?.expiresAtMs ?? Math.max(nowMs + 1, nowMs + 1_000);
if (
!Number.isSafeInteger(request.expectedCurrentVersion) ||
request.expectedCurrentVersion < 0 ||
!Number.isSafeInteger(notBeforeAtMs) ||
!Number.isSafeInteger(expiresAtMs) ||
notBeforeAtMs < 0 ||
expiresAtMs <= Math.max(nowMs, notBeforeAtMs) ||
expiresAtMs - notBeforeAtMs > MAX_LIFETIME_MS
) {
throw new RangeError('Worker credential administration lifetime or fence is invalid');
}
let secret: Buffer | undefined;
let secretText: string | undefined;
try {
if (operation !== 'revoke') {
secret = randomBytes(32);
if (!Buffer.isBuffer(secret) || secret.byteLength !== 32) {
throw new TypeError('Worker credential administration entropy is invalid');
}
secretText = secret.toString('base64url');
}
const credential = {
credentialId: request.credentialId,
version: request.expectedCurrentVersion + 1,
state: operation === 'revoke' ? 'revoked' as const : 'active' as const,
workerId: request.workerId,
secretDigest: secretText
? workerCredentialSecretDigest(pepper, request.credentialId, secretText)
: REVOKED_DIGEST,
createdAtMs: nowMs,
notBeforeAtMs,
expiresAtMs,
};
const result = await repository.append({
expectedCurrentVersion: request.expectedCurrentVersion,
credential,
mutation: {
mutationId: request.mutationId,
operation,
credentialId: request.credentialId,
credentialVersion: request.expectedCurrentVersion + 1,
expectedPreviousVersion: request.expectedCurrentVersion,
changedBy: actor.subject,
createdAtMs: nowMs,
},
audit: {
eventId: request.mutationId,
requestId: request.requestId,
operationId: `worker_credential.${operation}`,
projectId: null,
subject: actor.subject,
authenticationId: actor.authenticationId,
outcome: 'allowed',
reasons: ['worker_credential_admin'],
fence: null,
occurredAtMs: nowMs,
},
});
return Object.freeze({
...result,
token:
options.returnToken !== false &&
result.status === 'created' && secretText
? formatWorkerCredentialToken(request.credentialId, secretText)
: null,
});
} finally {
secret?.fill(0);
secretText = undefined;
}
};
return Object.freeze({
issue: (request: ActiveWorkerCredentialAdministrationRequest) =>
mutate('issue', request),
rotate: (request: ActiveWorkerCredentialAdministrationRequest) =>
mutate('rotate', request),
revoke: (request: WorkerCredentialAdministrationRequest) =>
mutate('revoke', request),
});
}
@@ -0,0 +1,944 @@
/** Recoverable Worker credential delivery application boundary. */
import {
createHash,
randomBytes as nodeRandomBytes,
} from 'node:crypto';
import {
normalizeWorkerCredentialId,
normalizeWorkerCredentialMutationId,
type AppendWorkerCredentialCommand,
} from '@qinglong/runtime-core/worker-credential';
import {
WorkerCredentialDeliveryConflictError,
WorkerCredentialDeliveryUnavailableError,
MAX_WORKER_CREDENTIAL_STAGE_DISCARD_PAGE_SIZE,
normalizeCommitWorkerCredentialDeliveryCommand,
normalizeRevokePreviousWorkerCredentialDeliveryCommand,
normalizeWorkerCredentialDeliveryIntent,
normalizeWorkerCredentialDeliveryRecoveryPage,
normalizeWorkerCredentialDeliveryRecord,
normalizeWorkerCredentialStageDiscardRecord,
normalizeWorkerCredentialStageDiscardRecoveryPage,
workerCredentialDeliveryTokenDigest,
type ResolvedWorkerCredentialDelivery,
type WorkerCredentialDeliveryAdministrationRepository,
type WorkerCredentialDeliveryIntent,
type WorkerCredentialDeliveryRecoveryPage,
type WorkerCredentialDeliveryRecord,
type WorkerCredentialStageDiscardRecord,
type WorkerCredentialStageDiscardRecoveryPage,
} from '@qinglong/runtime-core/worker-credential-delivery';
import {
formatWorkerCredentialToken,
} from '@qinglong/runtime-core/worker-credential-token';
import {
normalizeSecurityPrincipal,
type SecurityPrincipal,
} from '@qinglong/runtime-core/security';
import {
createWorkerCredentialAdministrationService,
type ActiveWorkerCredentialAdministrationRequest,
} from './workerCredentialAdministration';
const MAX_LIFETIME_MS = 2 * 365 * 24 * 60 * 60 * 1000;
const STRONG = new Set(['multi_factor', 'hardware', 'local_console']);
const SAFE_WORKER_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const SAFE_REQUEST_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const SAFE_GENERATION = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const SHA256 = /^[0-9a-f]{64}$/;
export interface RecoverableWorkerCredentialIssueRequest
extends ActiveWorkerCredentialAdministrationRequest {
readonly previousCredentialId: string | null;
readonly deploymentTargetDigest: string;
readonly deploymentGeneration: string;
}
export interface WorkerCredentialStagedSecretAdapter {
inspect(
deliveryId: string,
): Promise<Readonly<WorkerCredentialDeliveryIntent> | null>;
stage(
delivery: Readonly<WorkerCredentialDeliveryIntent>,
token: Buffer,
): Promise<void>;
publish(
delivery: Readonly<WorkerCredentialDeliveryRecord>,
): Promise<Readonly<{ publicationDigest: string }>>;
discard(delivery: Readonly<WorkerCredentialDeliveryIntent>): Promise<void>;
}
export interface WorkerCredentialStagedSecretPage {
readonly stages: readonly Readonly<WorkerCredentialDeliveryIntent>[];
readonly truncated: boolean;
readonly nextCursor?: string;
}
export interface WorkerCredentialStagedSecretInventoryAdapter
extends WorkerCredentialStagedSecretAdapter {
listStaged(options?: Readonly<{
afterDeliveryId?: string;
limit?: number;
}>): Promise<Readonly<WorkerCredentialStagedSecretPage>>;
}
export interface WorkerCredentialStageCleanupPageResult {
readonly outcomes: readonly Readonly<{
deliveryId: string;
result: 'discarded' | 'already_discarded';
}>[];
readonly truncated: boolean;
readonly nextCursor?: string;
}
export interface WorkerCredentialStageCleanupRecoveryResult
extends WorkerCredentialStageCleanupPageResult {
readonly observedAtMs: number;
}
export interface WorkerCredentialStageCleanupService {
cleanupInventoryPage(options?: Readonly<{
afterDeliveryId?: string;
limit?: number;
}>): Promise<Readonly<WorkerCredentialStageCleanupPageResult>>;
recoverAuthorizedPage(options?: Readonly<{
afterDeliveryId?: string;
limit?: number;
}>): Promise<Readonly<WorkerCredentialStageCleanupRecoveryResult>>;
}
export interface RecoverableWorkerCredentialIssueResult {
readonly status:
| 'published'
| 'existing'
| 'orphaned_stage_discarded';
readonly delivery: Readonly<WorkerCredentialDeliveryRecord> | null;
}
export interface RecoverableWorkerCredentialIssuer {
issue(
request: RecoverableWorkerCredentialIssueRequest,
): Promise<RecoverableWorkerCredentialIssueResult>;
}
export interface RecoverableWorkerCredentialIssuerOptions {
readonly now?: () => number;
readonly randomBytes?: (size: number) => Buffer;
}
export interface WorkerCredentialDeliveryRecoveryResult {
readonly observedAtMs: number;
readonly outcomes: readonly Readonly<{
deliveryId: string;
state: WorkerCredentialDeliveryRecord['state'];
result: 'published' | 'waiting_observation' | 'previous_revoked';
}>[];
readonly truncated: boolean;
readonly nextCursor?: string;
}
export interface WorkerCredentialDeliveryRecoveryService {
recoverPage(options?: Readonly<{
afterDeliveryId?: string;
limit?: number;
}>): Promise<Readonly<WorkerCredentialDeliveryRecoveryResult>>;
}
function exactRequest(value: RecoverableWorkerCredentialIssueRequest): void {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw new TypeError('Recoverable Worker credential issue request is invalid');
}
const expected = [
'mutationId',
'requestId',
'expectedCurrentVersion',
'credentialId',
'workerId',
'principal',
'notBeforeAtMs',
'expiresAtMs',
'previousCredentialId',
'deploymentTargetDigest',
'deploymentGeneration',
].sort();
const actual = Object.keys(value).sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
throw new TypeError(
'Recoverable Worker credential issue request shape is invalid',
);
}
}
function validateRequest(
request: RecoverableWorkerCredentialIssueRequest,
nowMs: number,
): void {
exactRequest(request);
normalizeWorkerCredentialMutationId(request.mutationId);
normalizeWorkerCredentialId(request.credentialId);
if (request.previousCredentialId !== null) {
normalizeWorkerCredentialId(request.previousCredentialId);
}
if (
request.expectedCurrentVersion !== 0 ||
request.previousCredentialId === request.credentialId ||
typeof request.workerId !== 'string' ||
!SAFE_WORKER_ID.test(request.workerId) ||
typeof request.requestId !== 'string' ||
!SAFE_REQUEST_ID.test(request.requestId) ||
typeof request.deploymentTargetDigest !== 'string' ||
!SHA256.test(request.deploymentTargetDigest) ||
typeof request.deploymentGeneration !== 'string' ||
!SAFE_GENERATION.test(request.deploymentGeneration)
) {
throw new TypeError('Recoverable Worker credential issue identity is invalid');
}
if (
!Number.isSafeInteger(nowMs) ||
nowMs < 0 ||
!Number.isSafeInteger(request.notBeforeAtMs) ||
request.notBeforeAtMs < 0 ||
!Number.isSafeInteger(request.expiresAtMs) ||
request.expiresAtMs <= Math.max(nowMs, request.notBeforeAtMs) ||
request.expiresAtMs - request.notBeforeAtMs > MAX_LIFETIME_MS
) {
throw new RangeError('Recoverable Worker credential issue lifetime is invalid');
}
const actor = normalizeSecurityPrincipal(request.principal, nowMs);
if (
!(
(actor.subject.type === 'user' && STRONG.has(actor.assurance)) ||
(actor.subject.type === 'system' && actor.assurance === 'service')
)
) {
throw new TypeError(
'Recoverable Worker credential issue requires a strong principal',
);
}
}
function requestForAdministration(
request: RecoverableWorkerCredentialIssueRequest,
): ActiveWorkerCredentialAdministrationRequest {
return {
mutationId: request.mutationId,
requestId: request.requestId,
expectedCurrentVersion: request.expectedCurrentVersion,
credentialId: request.credentialId,
workerId: request.workerId,
principal: request.principal,
notBeforeAtMs: request.notBeforeAtMs,
expiresAtMs: request.expiresAtMs,
};
}
function sameRequest(
resolved: ResolvedWorkerCredentialDelivery,
request: RecoverableWorkerCredentialIssueRequest,
): boolean {
const { delivery, credential, mutation, audit } = resolved;
return (
delivery.deliveryId === request.mutationId &&
delivery.credentialId === request.credentialId &&
delivery.previousCredentialId === request.previousCredentialId &&
delivery.workerId === request.workerId &&
delivery.deploymentTargetDigest === request.deploymentTargetDigest &&
delivery.deploymentGeneration === request.deploymentGeneration &&
credential.notBeforeAtMs === request.notBeforeAtMs &&
credential.expiresAtMs === request.expiresAtMs &&
mutation.operation === 'issue' &&
mutation.expectedPreviousVersion === 0 &&
audit.requestId === request.requestId &&
audit.subject?.type === request.principal.subject.type &&
audit.subject.id === request.principal.subject.id
);
}
function sameStage(
delivery: Readonly<WorkerCredentialDeliveryIntent>,
staged: Readonly<WorkerCredentialDeliveryIntent>,
): boolean {
return (
delivery.deliveryId === staged.deliveryId &&
delivery.workerId === staged.workerId &&
delivery.credentialId === staged.credentialId &&
delivery.credentialVersion === staged.credentialVersion &&
delivery.previousCredentialId === staged.previousCredentialId &&
delivery.secretDigest === staged.secretDigest &&
delivery.tokenDigest === staged.tokenDigest &&
delivery.deploymentTargetDigest === staged.deploymentTargetDigest &&
delivery.deploymentGeneration === staged.deploymentGeneration &&
delivery.stagedAtMs === staged.stagedAtMs
);
}
function mapDeliveryAdapterError(error: unknown): never {
if (error instanceof WorkerCredentialDeliveryConflictError) throw error;
throw new WorkerCredentialDeliveryUnavailableError();
}
function deliveryIntent(
command: Readonly<AppendWorkerCredentialCommand>,
request: RecoverableWorkerCredentialIssueRequest,
digest: string,
): Readonly<WorkerCredentialDeliveryIntent> {
return normalizeWorkerCredentialDeliveryIntent({
deliveryId: command.mutation.mutationId,
workerId: command.credential.workerId,
credentialId: command.credential.credentialId,
credentialVersion: command.credential.version,
previousCredentialId: request.previousCredentialId,
secretDigest: command.credential.secretDigest,
tokenDigest: digest,
deploymentTargetDigest: request.deploymentTargetDigest,
deploymentGeneration: request.deploymentGeneration,
stagedAtMs: command.credential.createdAtMs,
});
}
function committedDelivery(
intent: Readonly<WorkerCredentialDeliveryIntent>,
credentialCommittedAtMs: number,
): Readonly<WorkerCredentialDeliveryRecord> {
return normalizeWorkerCredentialDeliveryRecord({
...intent,
version: 1,
state: 'credential_committed',
credentialCommittedAtMs,
publishedAtMs: null,
publicationDigest: null,
observedAtMs: null,
observedSessionId: null,
observedSessionVersion: null,
previousRevokedAtMs: null,
});
}
export function createRecoverableWorkerCredentialIssuer(
authority: WorkerCredentialDeliveryAdministrationRepository,
deliveryAdapter: WorkerCredentialStagedSecretAdapter,
pepper: string,
options: RecoverableWorkerCredentialIssuerOptions = {},
): RecoverableWorkerCredentialIssuer {
if (
!authority ||
typeof authority.resolveMutation !== 'function' ||
typeof authority.resolveDelivered !== 'function' ||
typeof authority.commitDelivered !== 'function' ||
typeof authority.markPublished !== 'function' ||
typeof authority.authorizeStageDiscard !== 'function' ||
typeof authority.markStageDiscarded !== 'function'
) {
throw new TypeError('Worker credential delivery authority is invalid');
}
if (
!deliveryAdapter ||
typeof deliveryAdapter.inspect !== 'function' ||
typeof deliveryAdapter.stage !== 'function' ||
typeof deliveryAdapter.publish !== 'function' ||
typeof deliveryAdapter.discard !== 'function'
) {
throw new TypeError('Worker credential delivery adapter is invalid');
}
const now = options.now ?? Date.now;
const randomBytes = options.randomBytes ?? nodeRandomBytes;
const publish = async (
delivery: Readonly<WorkerCredentialDeliveryRecord>,
): Promise<Readonly<WorkerCredentialDeliveryRecord>> => {
if (delivery.state !== 'credential_committed') return delivery;
let publication: Readonly<{ publicationDigest: string }>;
try {
publication = await deliveryAdapter.publish(delivery);
} catch (error) {
mapDeliveryAdapterError(error);
}
if (
!publication ||
typeof publication !== 'object' ||
Array.isArray(publication) ||
Object.keys(publication).length !== 1 ||
typeof publication.publicationDigest !== 'string' ||
!SHA256.test(publication.publicationDigest)
) {
throw new WorkerCredentialDeliveryUnavailableError();
}
const publishedAtMs = now();
if (
!Number.isSafeInteger(publishedAtMs) ||
publishedAtMs < delivery.credentialCommittedAtMs
) {
throw new WorkerCredentialDeliveryUnavailableError();
}
try {
return normalizeWorkerCredentialDeliveryRecord(
await authority.markPublished({
deliveryId: delivery.deliveryId,
expectedVersion: delivery.version,
publicationDigest: publication.publicationDigest,
publishedAtMs,
}),
);
} catch (error) {
if (error instanceof WorkerCredentialDeliveryConflictError) throw error;
throw new WorkerCredentialDeliveryUnavailableError();
}
};
return Object.freeze({
async issue(request: RecoverableWorkerCredentialIssueRequest) {
const operationNowMs = now();
validateRequest(request, operationNowMs);
let capturedSecret: Buffer | undefined;
try {
let resolved: ResolvedWorkerCredentialDelivery | null;
try {
resolved = await authority.resolveDelivered(request.mutationId);
} catch {
throw new WorkerCredentialDeliveryUnavailableError();
}
if (resolved) {
if (!sameRequest(resolved, request)) {
throw new WorkerCredentialDeliveryConflictError();
}
let staged: Readonly<WorkerCredentialDeliveryIntent> | null;
try {
const inspected = await deliveryAdapter.inspect(request.mutationId);
staged = inspected
? normalizeWorkerCredentialDeliveryIntent(inspected)
: null;
} catch (error) {
mapDeliveryAdapterError(error);
}
if (!staged || !sameStage(resolved.delivery, staged)) {
throw new WorkerCredentialDeliveryConflictError();
}
const delivery = await publish(resolved.delivery);
return Object.freeze({
status: 'existing' as const,
delivery,
});
}
let rawMutation;
let orphanedStage: Readonly<WorkerCredentialDeliveryIntent> | null;
try {
rawMutation = await authority.resolveMutation(request.mutationId);
const inspected = await deliveryAdapter.inspect(request.mutationId);
orphanedStage = inspected
? normalizeWorkerCredentialDeliveryIntent(inspected)
: null;
} catch (error) {
mapDeliveryAdapterError(error);
}
if (rawMutation) {
throw new WorkerCredentialDeliveryConflictError();
}
if (orphanedStage) {
let authorized: Readonly<WorkerCredentialStageDiscardRecord>;
try {
authorized = normalizeWorkerCredentialStageDiscardRecord(
await authority.authorizeStageDiscard(orphanedStage),
);
} catch (error) {
if (error instanceof WorkerCredentialDeliveryConflictError) throw error;
throw new WorkerCredentialDeliveryUnavailableError();
}
if (!sameStage(authorized, orphanedStage)) {
throw new WorkerCredentialDeliveryConflictError();
}
try {
await deliveryAdapter.discard(orphanedStage);
} catch (error) {
mapDeliveryAdapterError(error);
}
if (authorized.state === 'discard_authorized') {
let completed: Readonly<WorkerCredentialStageDiscardRecord>;
try {
completed = normalizeWorkerCredentialStageDiscardRecord(
await authority.markStageDiscarded({
deliveryId: authorized.deliveryId,
expectedVersion: authorized.version,
}),
);
} catch (error) {
if (error instanceof WorkerCredentialDeliveryConflictError) throw error;
throw new WorkerCredentialDeliveryUnavailableError();
}
if (
completed.state !== 'discarded' ||
completed.authorizedAtMs !== authorized.authorizedAtMs ||
!sameStage(completed, authorized)
) {
throw new WorkerCredentialDeliveryConflictError();
}
}
return Object.freeze({
status: 'orphaned_stage_discarded' as const,
delivery: null,
});
}
const repository = {
resolveMutation: authority.resolveMutation.bind(authority),
async append(command: AppendWorkerCredentialCommand) {
if (!capturedSecret) {
throw new WorkerCredentialDeliveryUnavailableError();
}
let token: Buffer | undefined;
try {
const secretText = capturedSecret.toString('base64url');
token = Buffer.from(
formatWorkerCredentialToken(
command.credential.credentialId,
secretText,
),
'utf8',
);
const intent = deliveryIntent(
command,
request,
workerCredentialDeliveryTokenDigest(token),
);
try {
await deliveryAdapter.stage(intent, token);
} catch (error) {
mapDeliveryAdapterError(error);
}
const delivery = committedDelivery(
intent,
command.credential.createdAtMs,
);
return await authority.commitDelivered(
normalizeCommitWorkerCredentialDeliveryCommand({
credential: command,
delivery,
}),
);
} finally {
token?.fill(0);
capturedSecret.fill(0);
capturedSecret = undefined;
}
},
};
const administration = createWorkerCredentialAdministrationService(
repository,
pepper,
{
now: () => operationNowMs,
randomBytes(size) {
const secret = randomBytes(size);
if (Buffer.isBuffer(secret)) capturedSecret = Buffer.from(secret);
return secret;
},
returnToken: false,
},
);
await administration.issue(requestForAdministration(request));
let committed: ResolvedWorkerCredentialDelivery | null;
try {
committed = await authority.resolveDelivered(request.mutationId);
} catch {
throw new WorkerCredentialDeliveryUnavailableError();
}
if (!committed || !sameRequest(committed, request)) {
throw new WorkerCredentialDeliveryUnavailableError();
}
const delivery = await publish(committed.delivery);
return Object.freeze({ status: 'published' as const, delivery });
} finally {
capturedSecret?.fill(0);
}
},
});
}
const REVOKE_MUTATION_DOMAIN = Buffer.from(
'qinglong/worker-credential-delivery-revoke@v1\0',
'utf8',
);
function revokeMutationId(deliveryId: string): string {
const value = createHash('sha256')
.update(REVOKE_MUTATION_DOMAIN)
.update(deliveryId, 'utf8')
.digest();
value[6] = (value[6]! & 0x0f) | 0x40;
value[8] = (value[8]! & 0x3f) | 0x80;
const hex = value.subarray(0, 16).toString('hex');
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20)}`;
}
export function createWorkerCredentialDeliveryRecoveryService(
authority: WorkerCredentialDeliveryAdministrationRepository,
deliveryAdapter: WorkerCredentialStagedSecretAdapter,
pepper: string,
principal: SecurityPrincipal,
): WorkerCredentialDeliveryRecoveryService {
if (
!authority ||
typeof authority.resolveMutation !== 'function' ||
typeof authority.resolveDelivery !== 'function' ||
typeof authority.markPublished !== 'function' ||
typeof authority.listRecoveryPage !== 'function' ||
typeof authority.revokePreviousDelivered !== 'function'
) {
throw new TypeError('Worker credential recovery authority is invalid');
}
if (
!deliveryAdapter ||
typeof deliveryAdapter.inspect !== 'function' ||
typeof deliveryAdapter.publish !== 'function'
) {
throw new TypeError('Worker credential recovery adapter is invalid');
}
const publish = async (
delivery: Readonly<WorkerCredentialDeliveryRecord>,
operationNowMs: number,
): Promise<Readonly<WorkerCredentialDeliveryRecord>> => {
let staged: Readonly<WorkerCredentialDeliveryIntent> | null;
try {
const inspected = await deliveryAdapter.inspect(delivery.deliveryId);
staged = inspected
? normalizeWorkerCredentialDeliveryIntent(inspected)
: null;
} catch (error) {
mapDeliveryAdapterError(error);
}
if (!staged || !sameStage(delivery, staged)) {
throw new WorkerCredentialDeliveryConflictError();
}
let published: Readonly<{ publicationDigest: string }>;
try {
published = await deliveryAdapter.publish(delivery);
} catch (error) {
mapDeliveryAdapterError(error);
}
if (
!published ||
typeof published !== 'object' ||
Array.isArray(published) ||
Object.keys(published).length !== 1 ||
typeof published.publicationDigest !== 'string' ||
!SHA256.test(published.publicationDigest)
) {
throw new WorkerCredentialDeliveryUnavailableError();
}
if (operationNowMs < delivery.credentialCommittedAtMs) {
throw new WorkerCredentialDeliveryUnavailableError();
}
try {
return normalizeWorkerCredentialDeliveryRecord(
await authority.markPublished({
deliveryId: delivery.deliveryId,
expectedVersion: 1,
publicationDigest: published.publicationDigest,
publishedAtMs: operationNowMs,
}),
);
} catch (error) {
if (error instanceof WorkerCredentialDeliveryConflictError) throw error;
throw new WorkerCredentialDeliveryUnavailableError();
}
};
const revoke = async (
delivery: Readonly<WorkerCredentialDeliveryRecord>,
operationNowMs: number,
): Promise<Readonly<WorkerCredentialDeliveryRecord>> => {
if (delivery.state !== 'observed' || !delivery.previousCredentialId) {
throw new WorkerCredentialDeliveryConflictError();
}
if (operationNowMs < (delivery.observedAtMs ?? 0)) {
throw new WorkerCredentialDeliveryUnavailableError();
}
const repository = {
resolveMutation: authority.resolveMutation.bind(authority),
append(command: AppendWorkerCredentialCommand) {
return authority.revokePreviousDelivered(
normalizeRevokePreviousWorkerCredentialDeliveryCommand({
credential: command,
delivery: normalizeWorkerCredentialDeliveryRecord({
...delivery,
version: 4,
state: 'previous_revoked',
previousRevokedAtMs: command.credential.createdAtMs,
}),
}),
);
},
};
const administration = createWorkerCredentialAdministrationService(
repository,
pepper,
{ now: () => operationNowMs, returnToken: false },
);
await administration.revoke({
mutationId: revokeMutationId(delivery.deliveryId),
requestId: `worker-delivery-revoke:${delivery.deliveryId}`,
expectedCurrentVersion: 1,
credentialId: delivery.previousCredentialId,
workerId: delivery.workerId,
principal,
});
let resolved: Readonly<WorkerCredentialDeliveryRecord> | null;
try {
resolved = await authority.resolveDelivery(delivery.deliveryId);
} catch {
throw new WorkerCredentialDeliveryUnavailableError();
}
if (
!resolved ||
resolved.state !== 'previous_revoked' ||
!sameStage(resolved, delivery)
) {
throw new WorkerCredentialDeliveryConflictError();
}
return resolved;
};
return Object.freeze({
async recoverPage(
requested: Readonly<{
afterDeliveryId?: string;
limit?: number;
}> = {},
) {
let page: Readonly<WorkerCredentialDeliveryRecoveryPage>;
try {
page = normalizeWorkerCredentialDeliveryRecoveryPage(
await authority.listRecoveryPage(requested),
);
} catch (error) {
if (error instanceof WorkerCredentialDeliveryConflictError) throw error;
throw new WorkerCredentialDeliveryUnavailableError();
}
const outcomes = [];
for (const candidate of page.deliveries) {
const delivery = candidate.state === 'credential_committed'
? await publish(candidate, page.observedAtMs)
: candidate.state === 'observed'
? await revoke(candidate, page.observedAtMs)
: candidate;
outcomes.push(Object.freeze({
deliveryId: delivery.deliveryId,
state: delivery.state,
result: candidate.state === 'credential_committed'
? 'published' as const
: candidate.state === 'observed'
? 'previous_revoked' as const
: 'waiting_observation' as const,
}));
}
return Object.freeze({
observedAtMs: page.observedAtMs,
outcomes: Object.freeze(outcomes),
truncated: page.truncated,
...(page.nextCursor === undefined
? {}
: { nextCursor: page.nextCursor }),
});
},
});
}
function normalizeStagedSecretPage(
value: WorkerCredentialStagedSecretPage,
): Readonly<WorkerCredentialStagedSecretPage> {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw new WorkerCredentialDeliveryUnavailableError();
}
const expected = [
'stages',
'truncated',
...(value.nextCursor === undefined ? [] : ['nextCursor']),
].sort();
const actual = Object.keys(value).sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index]) ||
!Array.isArray(value.stages) ||
value.stages.length > MAX_WORKER_CREDENTIAL_STAGE_DISCARD_PAGE_SIZE ||
typeof value.truncated !== 'boolean'
) {
throw new WorkerCredentialDeliveryUnavailableError();
}
let stages: Readonly<WorkerCredentialDeliveryIntent>[];
try {
stages = value.stages.map((stage) =>
normalizeWorkerCredentialDeliveryIntent(stage));
} catch {
throw new WorkerCredentialDeliveryUnavailableError();
}
for (let index = 1; index < stages.length; index += 1) {
if (stages[index - 1]!.deliveryId >= stages[index]!.deliveryId) {
throw new WorkerCredentialDeliveryUnavailableError();
}
}
const last = stages[stages.length - 1];
if (
value.truncated !== (value.nextCursor !== undefined) ||
(value.nextCursor !== undefined &&
(!last || value.nextCursor !== last.deliveryId))
) {
throw new WorkerCredentialDeliveryUnavailableError();
}
return Object.freeze({
stages: Object.freeze(stages),
truncated: value.truncated,
...(value.nextCursor === undefined
? {}
: { nextCursor: value.nextCursor }),
});
}
export function createWorkerCredentialStageCleanupService(
authority: WorkerCredentialDeliveryAdministrationRepository,
deliveryAdapter: WorkerCredentialStagedSecretInventoryAdapter,
): WorkerCredentialStageCleanupService {
if (
!authority ||
typeof authority.authorizeStageDiscard !== 'function' ||
typeof authority.markStageDiscarded !== 'function' ||
typeof authority.listStageDiscardRecoveryPage !== 'function'
) {
throw new TypeError('Worker credential stage cleanup authority is invalid');
}
if (
!deliveryAdapter ||
typeof deliveryAdapter.inspect !== 'function' ||
typeof deliveryAdapter.discard !== 'function' ||
typeof deliveryAdapter.listStaged !== 'function'
) {
throw new TypeError('Worker credential stage cleanup adapter is invalid');
}
const discard = async (
record: Readonly<WorkerCredentialStageDiscardRecord>,
staged: Readonly<WorkerCredentialDeliveryIntent> | null,
): Promise<'discarded' | 'already_discarded'> => {
if (!sameStage(record, staged ?? record)) {
throw new WorkerCredentialDeliveryConflictError();
}
if (staged) {
try {
await deliveryAdapter.discard(staged);
} catch (error) {
mapDeliveryAdapterError(error);
}
}
if (record.state === 'discarded') return 'already_discarded';
let completed: Readonly<WorkerCredentialStageDiscardRecord>;
try {
completed = normalizeWorkerCredentialStageDiscardRecord(
await authority.markStageDiscarded({
deliveryId: record.deliveryId,
expectedVersion: record.version,
}),
);
} catch (error) {
if (error instanceof WorkerCredentialDeliveryConflictError) throw error;
throw new WorkerCredentialDeliveryUnavailableError();
}
if (
completed.state !== 'discarded' ||
completed.authorizedAtMs !== record.authorizedAtMs ||
!sameStage(completed, record)
) {
throw new WorkerCredentialDeliveryConflictError();
}
return 'discarded';
};
return Object.freeze({
async cleanupInventoryPage(
options: Readonly<{
afterDeliveryId?: string;
limit?: number;
}> = {},
) {
let page: Readonly<WorkerCredentialStagedSecretPage>;
try {
page = normalizeStagedSecretPage(
await deliveryAdapter.listStaged(options),
);
} catch (error) {
mapDeliveryAdapterError(error);
}
const outcomes = [];
for (const staged of page.stages) {
let authorized: Readonly<WorkerCredentialStageDiscardRecord>;
try {
authorized = normalizeWorkerCredentialStageDiscardRecord(
await authority.authorizeStageDiscard(staged),
);
} catch (error) {
if (error instanceof WorkerCredentialDeliveryConflictError) throw error;
throw new WorkerCredentialDeliveryUnavailableError();
}
if (!sameStage(authorized, staged)) {
throw new WorkerCredentialDeliveryConflictError();
}
outcomes.push(Object.freeze({
deliveryId: staged.deliveryId,
result: await discard(authorized, staged),
}));
}
return Object.freeze({
outcomes: Object.freeze(outcomes),
truncated: page.truncated,
...(page.nextCursor === undefined
? {}
: { nextCursor: page.nextCursor }),
});
},
async recoverAuthorizedPage(
options: Readonly<{
afterDeliveryId?: string;
limit?: number;
}> = {},
) {
let page: Readonly<WorkerCredentialStageDiscardRecoveryPage>;
try {
page = normalizeWorkerCredentialStageDiscardRecoveryPage(
await authority.listStageDiscardRecoveryPage(options),
);
} catch (error) {
if (error instanceof WorkerCredentialDeliveryConflictError) throw error;
throw new WorkerCredentialDeliveryUnavailableError();
}
const outcomes = [];
for (const authorized of page.discards) {
let staged: Readonly<WorkerCredentialDeliveryIntent> | null;
try {
const inspected = await deliveryAdapter.inspect(authorized.deliveryId);
staged = inspected
? normalizeWorkerCredentialDeliveryIntent(inspected)
: null;
} catch (error) {
mapDeliveryAdapterError(error);
}
if (staged && !sameStage(authorized, staged)) {
throw new WorkerCredentialDeliveryConflictError();
}
outcomes.push(Object.freeze({
deliveryId: authorized.deliveryId,
result: await discard(authorized, staged),
}));
}
return Object.freeze({
observedAtMs: page.observedAtMs,
outcomes: Object.freeze(outcomes),
truncated: page.truncated,
...(page.nextCursor === undefined
? {}
: { nextCursor: page.nextCursor }),
});
},
});
}
@@ -0,0 +1,69 @@
#!/usr/bin/env node
/** One-shot Worker credential executor CLI boundary. */
import { runClusterWorkerCredentialExecutorProcess } from './workerCredentialExecutorProcess';
const USAGE = 'Usage: ql3-worker-credential-execute';
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as { readonly name?: unknown; readonly code?: unknown };
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-worker-credential-executor',
event: 'execution_failed',
name:
typeof candidate?.name === 'string' && candidate.name.length <= 128
? candidate.name
: 'Error',
...(typeof candidate?.code === 'string' && candidate.code.length <= 128
? { code: candidate.code }
: {}),
});
}
async function run(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
if (argv.length !== 0) {
process.stderr.write(
`${JSON.stringify({
code: 'QL3_WORKER_CREDENTIAL_EXECUTOR_CLI_USAGE_INVALID',
message: USAGE,
})}\n`,
);
process.exitCode = 64;
return;
}
try {
const result = await runClusterWorkerCredentialExecutorProcess({
environment: process.env,
});
process.stdout.write(
`${JSON.stringify(
result.status === 'disabled'
? {
schemaVersion: 1,
component: 'qinglong3-worker-credential-executor',
event: 'execution_disabled',
}
: {
schemaVersion: 1,
component: 'qinglong3-worker-credential-executor',
event: 'execution_completed',
actionRef: result.command.actionRef,
dispatchId: result.command.dispatchId,
executionStatus: result.run.execution.status,
deliveryStatus: result.run.result.status,
tokenRequestUsed: result.run.tokenRequest !== null,
},
)}\n`,
);
} catch (error) {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
}
}
void run(process.argv.slice(2));
@@ -0,0 +1,640 @@
/** One-shot Worker credential executor process composition boundary. */
import { constants } from 'node:fs';
import { open } from 'node:fs/promises';
import type { OpenPostgresDatabase } from '@qinglong/runtime-core';
import {
createPostgresDatabaseOpener,
isPostgresTlsDnsServername,
loadPostgresCertificateAuthorityFile,
loadPostgresConnectionEnvironment,
type PostgresConnectionOptions,
type PostgresPoolOptions,
} from '@qinglong/cluster-postgres/worker-credential-executor';
import {
absoluteManagementEnvironmentFile,
booleanManagementEnvironmentValue,
boundedManagementEnvironmentValue,
integerManagementEnvironmentValue,
} from '../management-support/managementProcessSupport';
import {
runClusterWorkerCredentialExecution,
type ClusterWorkerCredentialExecutionRun,
type RunClusterWorkerCredentialExecutionOptions,
} from './workerCredentialManagementExecutor';
import type { WorkerCredentialKubernetesDeliveryAdapterOptions } from './workerCredentialKubernetesDelivery';
import {
createWorkerCredentialKubernetesKubeConfigTokenRequestSession,
type WorkerCredentialKubernetesAuthorizationApi,
type WorkerCredentialKubernetesTokenRequestSession,
} from './workerCredentialKubernetesTokenRequest';
const COMMAND_MAX_BYTES = 16 * 1024;
const PEPPER_MAX_BYTES = 256;
const CONTROL_PATTERN = /[\u0000-\u001f\u007f]/;
const SAFE_APPLICATION_NAME = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,62}$/;
const ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const ACTION_REF = /^[A-Za-z0-9][A-Za-z0-9._:/-]{0,254}$/;
type KubernetesModule = typeof import('@kubernetes/client-node', {
with: { 'resolution-mode': 'import' }
});
export interface ClusterWorkerCredentialExecutorCommand {
readonly schemaVersion: 1;
readonly actionRef: string;
readonly approvalRequestId: string;
readonly consumptionId: string;
readonly dispatchId: string;
readonly auditEventId: string;
}
export type ClusterWorkerCredentialExecutorProcessEnvironment = Readonly<
Record<string, string | undefined>
>;
export type ClusterWorkerCredentialExecutorProcessConfig =
| Readonly<{ enabled: false }>
| Readonly<{
enabled: true;
profile: 'cluster-admin';
commandFile: string;
pepperFile: string;
serviceAccountName: string;
identitySecretName: string;
delivery: WorkerCredentialKubernetesDeliveryAdapterOptions;
database: Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}>;
}>;
export type ClusterWorkerCredentialExecutorProcessResult =
| Readonly<{ status: 'disabled' }>
| Readonly<{
status: 'completed';
command: Readonly<ClusterWorkerCredentialExecutorCommand>;
run: Readonly<ClusterWorkerCredentialExecutionRun>;
}>;
interface KubernetesExecutionAuthority {
readonly session: WorkerCredentialKubernetesTokenRequestSession;
confirmAuthorization(): Promise<void>;
dispose(): void;
}
export interface RunClusterWorkerCredentialExecutorProcessOptions {
readonly environment: ClusterWorkerCredentialExecutorProcessEnvironment;
readonly command?: Readonly<ClusterWorkerCredentialExecutorCommand>;
readonly openDatabase?: OpenPostgresDatabase;
readonly kubernetesAuthority?: KubernetesExecutionAuthority;
readonly createKubernetesAuthority?: (
config: Readonly<ClusterWorkerCredentialExecutorProcessConfig & { enabled: true }>,
) => Promise<KubernetesExecutionAuthority>;
readonly execute?: (
options: RunClusterWorkerCredentialExecutionOptions,
) => Promise<Readonly<ClusterWorkerCredentialExecutionRun>>;
readonly now?: () => number;
}
export class ClusterWorkerCredentialExecutorProcessConfigError extends TypeError {
readonly code = 'QL3_WORKER_CREDENTIAL_EXECUTOR_PROCESS_CONFIG_INVALID';
constructor(message: string) {
super(`Worker credential executor process configuration is invalid: ${message}`);
this.name = 'ClusterWorkerCredentialExecutorProcessConfigError';
}
}
function configFailure(
message: string,
): ClusterWorkerCredentialExecutorProcessConfigError {
return new ClusterWorkerCredentialExecutorProcessConfigError(message);
}
function boundedValue(
environment: ClusterWorkerCredentialExecutorProcessEnvironment,
name: string,
maximumLength: number,
required = false,
): string | undefined {
return boundedManagementEnvironmentValue(
environment,
name,
maximumLength,
configFailure,
required,
);
}
function booleanValue(
environment: ClusterWorkerCredentialExecutorProcessEnvironment,
name: string,
): boolean {
return booleanManagementEnvironmentValue(environment, name, configFailure);
}
function integerValue(
environment: ClusterWorkerCredentialExecutorProcessEnvironment,
name: string,
fallback: number,
minimum: number,
maximum: number,
): number {
return integerManagementEnvironmentValue(
environment,
name,
fallback,
minimum,
maximum,
configFailure,
);
}
function identifier(value: unknown, label: string, pattern = ID): string {
if (typeof value !== 'string' || !pattern.test(value)) {
throw configFailure(`${label} is invalid`);
}
return value;
}
function loadConnection(
environment: ClusterWorkerCredentialExecutorProcessEnvironment,
): Readonly<{
connection: PostgresConnectionOptions;
pool: PostgresPoolOptions;
}> {
let connection: PostgresConnectionOptions;
try {
connection = loadPostgresConnectionEnvironment(environment, {
connectionString: 'QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_URL',
host: 'QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_HOST',
port: 'QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_PORT',
database: 'QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_DATABASE',
user: 'QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_USER',
password: 'QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_PASSWORD',
});
} catch (error) {
throw configFailure(
error instanceof Error
? error.message
: 'PostgreSQL Worker credential executor connection is invalid',
);
}
const mode =
environment.QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_TLS_MODE ??
'verify-full';
if (mode !== 'verify-full' && mode !== 'disable') {
throw configFailure(
'QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_TLS_MODE must be verify-full or disable',
);
}
if (
mode === 'disable' &&
!booleanValue(
environment,
'QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_ALLOW_INSECURE',
)
) {
throw configFailure(
'disabling Worker credential executor PostgreSQL TLS requires explicit opt-in',
);
}
const servername = boundedValue(
environment,
'QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_TLS_SERVERNAME',
253,
);
if (mode === 'verify-full' && !isPostgresTlsDnsServername(servername)) {
throw configFailure(
'QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_TLS_SERVERNAME must be an explicit DNS name',
);
}
const caFile = boundedValue(
environment,
'QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_TLS_CA_FILE',
4_096,
);
if (mode === 'disable' && caFile !== undefined) {
throw configFailure('PostgreSQL CA file cannot be used when TLS is disabled');
}
let ca: string | undefined;
if (caFile !== undefined) {
try {
ca = loadPostgresCertificateAuthorityFile(caFile);
} catch {
throw configFailure('PostgreSQL CA file is invalid');
}
}
const applicationName =
boundedValue(
environment,
'QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_APPLICATION_NAME',
63,
) ?? 'qinglong3-worker-credential-executor';
if (!SAFE_APPLICATION_NAME.test(applicationName)) {
throw configFailure('PostgreSQL application name is invalid');
}
return Object.freeze({
connection: Object.freeze({
...connection,
tls:
mode === 'disable'
? { mode: 'disable' as const }
: {
mode: 'verify-full' as const,
servername: servername!,
...(ca === undefined ? {} : { ca }),
},
}),
pool: Object.freeze({
applicationName,
maxConnections: integerValue(
environment,
'QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_MAX_CONNECTIONS',
1,
1,
2,
),
connectionTimeoutMs: integerValue(
environment,
'QL3_POSTGRES_WORKER_CREDENTIAL_EXECUTOR_CONNECTION_TIMEOUT_MS',
5_000,
100,
60_000,
),
}),
});
}
export function loadClusterWorkerCredentialExecutorProcessConfig(
environment: ClusterWorkerCredentialExecutorProcessEnvironment,
): Readonly<ClusterWorkerCredentialExecutorProcessConfig> {
if (!environment || typeof environment !== 'object') {
throw configFailure('environment is invalid');
}
if (!booleanValue(environment, 'QL3_WORKER_CREDENTIAL_EXECUTOR_ENABLED')) {
return Object.freeze({ enabled: false as const });
}
if (environment.QL3_PROFILE !== 'cluster-admin') {
throw configFailure('QL3_PROFILE must be cluster-admin when executor is enabled');
}
const delivery = Object.freeze({
clusterIdentity: identifier(
boundedValue(
environment,
'QL3_WORKER_CREDENTIAL_EXECUTOR_CLUSTER_IDENTITY',
128,
true,
),
'cluster identity',
),
stageNamespace: identifier(
boundedValue(
environment,
'QL3_WORKER_CREDENTIAL_EXECUTOR_STAGE_NAMESPACE',
63,
true,
),
'stage namespace',
/^[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?$/,
),
namespace: identifier(
boundedValue(
environment,
'QL3_WORKER_CREDENTIAL_EXECUTOR_TARGET_NAMESPACE',
63,
true,
),
'target namespace',
/^[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?$/,
),
targetSecretName: identifier(
boundedValue(
environment,
'QL3_WORKER_CREDENTIAL_EXECUTOR_TARGET_SECRET',
253,
true,
),
'target Secret',
/^[a-z0-9](?:[-a-z0-9.]{0,251}[a-z0-9])?$/,
),
targetDeploymentName: identifier(
boundedValue(
environment,
'QL3_WORKER_CREDENTIAL_EXECUTOR_TARGET_DEPLOYMENT',
253,
true,
),
'target Deployment',
/^[a-z0-9](?:[-a-z0-9.]{0,251}[a-z0-9])?$/,
),
targetDataKey: identifier(
boundedValue(
environment,
'QL3_WORKER_CREDENTIAL_EXECUTOR_TARGET_DATA_KEY',
253,
true,
),
'target data key',
/^[A-Za-z0-9._-]{1,253}$/,
),
});
return Object.freeze({
enabled: true as const,
profile: 'cluster-admin' as const,
commandFile: absoluteManagementEnvironmentFile(
environment,
'QL3_WORKER_CREDENTIAL_EXECUTOR_COMMAND_FILE',
configFailure,
),
pepperFile: absoluteManagementEnvironmentFile(
environment,
'QL3_WORKER_CREDENTIAL_EXECUTOR_PEPPER_FILE',
configFailure,
),
serviceAccountName: identifier(
boundedValue(
environment,
'QL3_WORKER_CREDENTIAL_EXECUTOR_DELIVERY_SERVICE_ACCOUNT',
63,
true,
),
'delivery ServiceAccount',
/^[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?$/,
),
identitySecretName: identifier(
boundedValue(
environment,
'QL3_WORKER_CREDENTIAL_EXECUTOR_IDENTITY_SECRET',
253,
true,
),
'identity Secret',
/^[a-z0-9](?:[-a-z0-9.]{0,251}[a-z0-9])?$/,
),
delivery,
database: loadConnection(environment),
});
}
async function readBoundedFile(
filePath: string,
maximumBytes: number,
privateMaterial: boolean,
): Promise<Buffer> {
const handle = await open(filePath, constants.O_RDONLY);
try {
const before = await handle.stat();
if (
!before.isFile() ||
before.size < 1 ||
before.size > maximumBytes ||
(before.mode & 0o022) !== 0 ||
(privateMaterial && (before.mode & 0o007) !== 0)
) {
throw configFailure('authority file permissions or size are invalid');
}
const bytes = Buffer.alloc(before.size + 1);
let offset = 0;
while (offset < bytes.length) {
const result = await handle.read(
bytes,
offset,
bytes.length - offset,
offset,
);
if (result.bytesRead === 0) break;
offset += result.bytesRead;
}
const after = await handle.stat();
if (
offset !== before.size ||
before.dev !== after.dev ||
before.ino !== after.ino ||
before.size !== after.size ||
before.mtimeMs !== after.mtimeMs ||
before.ctimeMs !== after.ctimeMs
) {
throw configFailure('authority file changed while being read');
}
return bytes.subarray(0, offset);
} finally {
await handle.close().catch(() => undefined);
}
}
function normalizeCommand(
value: unknown,
): Readonly<ClusterWorkerCredentialExecutorCommand> {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw configFailure('command must be an object');
}
const command = value as Record<string, unknown>;
const keys = [
'actionRef',
'approvalRequestId',
'auditEventId',
'consumptionId',
'dispatchId',
'schemaVersion',
];
if (
Object.keys(command).sort().join('\0') !== keys.sort().join('\0') ||
command.schemaVersion !== 1
) {
throw configFailure('command shape is invalid');
}
return Object.freeze({
schemaVersion: 1 as const,
actionRef: identifier(command.actionRef, 'actionRef', ACTION_REF),
approvalRequestId: identifier(command.approvalRequestId, 'approvalRequestId'),
consumptionId: identifier(command.consumptionId, 'consumptionId'),
dispatchId: identifier(command.dispatchId, 'dispatchId'),
auditEventId: identifier(command.auditEventId, 'auditEventId'),
});
}
async function loadCommand(
filePath: string,
): Promise<Readonly<ClusterWorkerCredentialExecutorCommand>> {
const bytes = await readBoundedFile(filePath, COMMAND_MAX_BYTES, false);
try {
const text = new TextDecoder('utf-8', { fatal: true }).decode(bytes);
return normalizeCommand(JSON.parse(text));
} catch (error) {
if (error instanceof ClusterWorkerCredentialExecutorProcessConfigError) {
throw error;
}
throw configFailure('command file is invalid');
}
}
async function loadPepper(filePath: string): Promise<string> {
const bytes = await readBoundedFile(filePath, PEPPER_MAX_BYTES, true);
try {
const value = new TextDecoder('utf-8', { fatal: true }).decode(bytes);
if (
CONTROL_PATTERN.test(value) ||
!/^[A-Za-z0-9_-]{43}$/.test(value) ||
Buffer.from(value, 'base64url').length !== 32 ||
Buffer.from(value, 'base64url').toString('base64url') !== value
) {
throw configFailure('Worker credential pepper is invalid');
}
return value;
} finally {
bytes.fill(0);
}
}
async function createDefaultKubernetesAuthority(
config: Readonly<ClusterWorkerCredentialExecutorProcessConfig & { enabled: true }>,
): Promise<KubernetesExecutionAuthority> {
const kubernetes = (await import('@kubernetes/client-node')) as KubernetesModule;
const issuer = new kubernetes.KubeConfig();
issuer.loadFromCluster();
const authorization = issuer.makeApiClient(
kubernetes.AuthorizationV1Api,
) as unknown as WorkerCredentialKubernetesAuthorizationApi;
const confirmAuthorization = async (): Promise<void> => {
const result = await authorization.createSelfSubjectAccessReview({
body: {
apiVersion: 'authorization.k8s.io/v1',
kind: 'SelfSubjectAccessReview',
spec: {
resourceAttributes: {
namespace: config.delivery.stageNamespace,
verb: 'create',
resource: 'serviceaccounts',
subresource: 'token',
name: config.serviceAccountName,
},
},
},
});
if (result?.status?.allowed !== true || result.status.denied === true) {
throw configFailure('executor Kubernetes authorization is unavailable');
}
};
return Object.freeze({
session: createWorkerCredentialKubernetesKubeConfigTokenRequestSession(
issuer,
kubernetes,
{
serviceAccountName: config.serviceAccountName,
identitySecretName: config.identitySecretName,
delivery: config.delivery,
},
),
confirmAuthorization,
dispose() {
for (const user of issuer.getUsers()) {
const mutable = user as {
token?: string;
certData?: string;
keyData?: string;
};
mutable.token = '';
mutable.certData = '';
mutable.keyData = '';
}
issuer.setCurrentContext('disposed');
},
});
}
export async function runClusterWorkerCredentialExecutorProcess(
options: RunClusterWorkerCredentialExecutorProcessOptions,
): Promise<Readonly<ClusterWorkerCredentialExecutorProcessResult>> {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) =>
![
'environment',
'command',
'openDatabase',
'kubernetesAuthority',
'createKubernetesAuthority',
'execute',
'now',
].includes(key),
) ||
!options.environment ||
typeof options.environment !== 'object' ||
(options.openDatabase !== undefined &&
typeof options.openDatabase !== 'function') ||
(options.createKubernetesAuthority !== undefined &&
typeof options.createKubernetesAuthority !== 'function') ||
(options.execute !== undefined && typeof options.execute !== 'function') ||
(options.now !== undefined && typeof options.now !== 'function')
) {
throw configFailure('options are invalid');
}
const config = loadClusterWorkerCredentialExecutorProcessConfig(
options.environment,
);
if (!config.enabled) return Object.freeze({ status: 'disabled' as const });
const command = normalizeCommand(
options.command ?? (await loadCommand(config.commandFile)),
);
const pepper = await loadPepper(config.pepperFile);
const openDatabase =
options.openDatabase ??
createPostgresDatabaseOpener({
role: 'worker-credential-executor',
connection: config.database.connection,
pool: config.database.pool,
onPoolError() {},
});
const authority =
options.kubernetesAuthority ??
(await (
options.createKubernetesAuthority ?? createDefaultKubernetesAuthority
)(config));
if (
!authority ||
typeof authority !== 'object' ||
!authority.session ||
typeof authority.session.withDelivery !== 'function' ||
typeof authority.confirmAuthorization !== 'function' ||
typeof authority.dispose !== 'function'
) {
throw configFailure('Kubernetes execution authority is invalid');
}
let failure: unknown;
try {
const run = await (options.execute ?? runClusterWorkerCredentialExecution)({
openDatabase,
tokenRequestSession: authority.session,
workerCredentialPepper: pepper,
actionRef: command.actionRef,
approvalRequestId: command.approvalRequestId,
consumptionId: command.consumptionId,
dispatchId: command.dispatchId,
auditEventId: command.auditEventId,
confirmAuthorization: authority.confirmAuthorization,
...(options.now === undefined ? {} : { now: options.now }),
});
return Object.freeze({ status: 'completed' as const, command, run });
} catch (error) {
failure = error;
throw error;
} finally {
try {
authority.dispose();
} catch (disposeError) {
if (failure !== undefined) {
throw new AggregateError(
[failure, disposeError],
'Worker credential executor failed and Kubernetes authority did not dispose',
);
}
throw disposeError;
}
}
}
@@ -0,0 +1,794 @@
/** POSIX file-backed Worker credential delivery adapter boundary. */
import { createHash, randomUUID } from 'node:crypto';
import fs from 'node:fs';
import path from 'node:path';
import {
WorkerCredentialDeliveryConflictError,
WorkerCredentialDeliveryUnavailableError,
normalizeWorkerCredentialDeliveryIntent,
normalizeWorkerCredentialDeliveryRecord,
workerCredentialDeliveryTokenDigest,
type WorkerCredentialDeliveryIntent,
type WorkerCredentialDeliveryRecord,
} from '@qinglong/runtime-core/worker-credential-delivery';
import type {
WorkerCredentialStagedSecretInventoryAdapter,
WorkerCredentialStagedSecretPage,
} from './workerCredentialDelivery';
const MAX_PATH_BYTES = 4096;
const MAX_STAGE_BYTES = 8192;
const MAX_STAGE_HEADER_BYTES = 4096;
const MAX_TOKEN_BYTES = 256;
export const MAX_WORKER_CREDENTIAL_FILE_STAGES = 128;
export const MAX_WORKER_CREDENTIAL_FILE_STAGE_PAGE_SIZE = 64;
const STAGE_MAGIC = Buffer.from(
'qinglong/worker-credential-file-stage@v1\n',
'ascii',
);
const TARGET_DIGEST_DOMAIN = Buffer.from(
'qinglong/worker-credential-file-target@v1\0',
'utf8',
);
const PUBLICATION_DIGEST_DOMAIN = Buffer.from(
'qinglong/worker-credential-file-publication@v1\0',
'utf8',
);
const UUID_V4 =
/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/;
const STAGE_NAME =
/^([0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12})\.stage$/;
const STAGE_TEMP_NAME =
/^\.([0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12})\.[0-9a-f-]{36}\.tmp$/;
const TOKEN =
/^ql3w_([A-Za-z0-9][A-Za-z0-9._:-]{0,63})_([A-Za-z0-9_-]{43})$/;
const TARGET_LOCK_NAME = '.ql3-worker-credential-delivery.lock';
export interface WorkerCredentialFileDeliveryAdapterOptions {
/** Dedicated private 0700 directory containing bounded durable stages. */
readonly stageDirectory: string;
/** Atomically replaceable 0600 ql3w token file read by worker-runtime. */
readonly targetTokenFile: string;
}
export type WorkerCredentialFileStagePage = WorkerCredentialStagedSecretPage;
interface ParsedToken {
readonly credentialId: string;
readonly tokenDigest: string;
}
interface StagedSecret {
readonly intent: Readonly<WorkerCredentialDeliveryIntent>;
readonly token: Buffer;
}
interface OwnedTargetLock {
readonly device: bigint;
readonly inode: bigint;
}
function isMissing(error: unknown): boolean {
return Boolean(
error &&
typeof error === 'object' &&
'code' in error &&
error.code === 'ENOENT',
);
}
function isExists(error: unknown): boolean {
return Boolean(
error &&
typeof error === 'object' &&
'code' in error &&
error.code === 'EEXIST',
);
}
function boundedAbsolutePath(value: string, name: string): string {
if (
typeof value !== 'string' ||
!path.isAbsolute(value) ||
path.parse(value).root === value ||
path.normalize(value) !== value ||
value.includes('\0') ||
Buffer.byteLength(value, 'utf8') > MAX_PATH_BYTES
) {
throw new TypeError(`${name} must be a bounded canonical absolute path`);
}
return value;
}
class PrivateDirectoryAuthority {
readonly directory: string;
readonly uid: number;
readonly device: bigint;
readonly inode: bigint;
constructor(directory: string, name: string) {
this.directory = boundedAbsolutePath(directory, name);
if (typeof process.getuid !== 'function') {
throw new TypeError(`${name} requires a POSIX process identity`);
}
this.uid = process.getuid();
const stat = fs.lstatSync(this.directory, { bigint: true });
if (
!stat.isDirectory() ||
stat.isSymbolicLink() ||
Number(stat.uid) !== this.uid ||
(Number(stat.mode) & 0o777) !== 0o700
) {
throw new TypeError(`${name} must be a private owned real directory`);
}
this.device = stat.dev;
this.inode = stat.ino;
}
verify(): void {
const stat = fs.lstatSync(this.directory, { bigint: true });
if (
!stat.isDirectory() ||
stat.isSymbolicLink() ||
Number(stat.uid) !== this.uid ||
(Number(stat.mode) & 0o777) !== 0o700 ||
stat.dev !== this.device ||
stat.ino !== this.inode
) {
throw new WorkerCredentialDeliveryUnavailableError();
}
}
sync(): void {
this.verify();
const descriptor = fs.openSync(this.directory, 'r');
try {
fs.fsyncSync(descriptor);
} finally {
fs.closeSync(descriptor);
}
}
}
function readPrivateFile(
authority: PrivateDirectoryAuthority,
filePath: string,
maximumBytes: number,
): Buffer {
authority.verify();
const before = fs.lstatSync(filePath, { bigint: true });
if (
!before.isFile() ||
before.isSymbolicLink() ||
Number(before.uid) !== authority.uid ||
(Number(before.mode) & 0o777) !== 0o600 ||
before.size < 1n ||
before.size > BigInt(maximumBytes)
) {
throw new WorkerCredentialDeliveryUnavailableError();
}
const descriptor = fs.openSync(
filePath,
fs.constants.O_RDONLY | (fs.constants.O_NOFOLLOW ?? 0),
);
try {
const opened = fs.fstatSync(descriptor, { bigint: true });
if (
!opened.isFile() ||
opened.dev !== before.dev ||
opened.ino !== before.ino ||
opened.size !== before.size
) {
throw new WorkerCredentialDeliveryUnavailableError();
}
const bytes = fs.readFileSync(descriptor);
if (bytes.byteLength < 1 || bytes.byteLength > maximumBytes) {
bytes.fill(0);
throw new WorkerCredentialDeliveryUnavailableError();
}
return bytes;
} finally {
fs.closeSync(descriptor);
}
}
function parseToken(bytes: Buffer): ParsedToken {
let visible = bytes;
if (visible[visible.byteLength - 1] === 0x0a) {
visible = visible.subarray(0, -1);
}
if (
visible.byteLength < 1 ||
visible.byteLength > MAX_TOKEN_BYTES ||
visible.includes(0x0a) ||
visible.some((byte) => byte > 0x7f)
) {
throw new WorkerCredentialDeliveryUnavailableError();
}
const match = TOKEN.exec(visible.toString('ascii'));
if (!match) throw new WorkerCredentialDeliveryUnavailableError();
return Object.freeze({
credentialId: match[1]!,
tokenDigest: workerCredentialDeliveryTokenDigest(visible),
});
}
function sameIntent(
left: Readonly<WorkerCredentialDeliveryIntent>,
right: Readonly<WorkerCredentialDeliveryIntent>,
): boolean {
return (
left.deliveryId === right.deliveryId &&
left.workerId === right.workerId &&
left.credentialId === right.credentialId &&
left.credentialVersion === right.credentialVersion &&
left.previousCredentialId === right.previousCredentialId &&
left.secretDigest === right.secretDigest &&
left.tokenDigest === right.tokenDigest &&
left.deploymentTargetDigest === right.deploymentTargetDigest &&
left.deploymentGeneration === right.deploymentGeneration &&
left.stagedAtMs === right.stagedAtMs
);
}
function recordMatchesIntent(
record: Readonly<WorkerCredentialDeliveryRecord>,
intent: Readonly<WorkerCredentialDeliveryIntent>,
): boolean {
return sameIntent(record, intent);
}
function preserveDomainError(error: unknown): never {
if (
error instanceof WorkerCredentialDeliveryConflictError ||
error instanceof WorkerCredentialDeliveryUnavailableError
) {
throw error;
}
throw new WorkerCredentialDeliveryUnavailableError();
}
/**
* Concrete short-lived POSIX adapter for Docker bind mounts, systemd services,
* and controlled shared volumes. It owns no timer, socket, database or cache.
*/
export class WorkerCredentialFileDeliveryAdapter
implements WorkerCredentialStagedSecretInventoryAdapter {
readonly deploymentTargetDigest: string;
private readonly stages: PrivateDirectoryAuthority;
private readonly targetParent: PrivateDirectoryAuthority;
private readonly targetTokenFile: string;
private readonly targetLockFile: string;
constructor(options: WorkerCredentialFileDeliveryAdapterOptions) {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).length !== 2 ||
!Object.prototype.hasOwnProperty.call(options, 'stageDirectory') ||
!Object.prototype.hasOwnProperty.call(options, 'targetTokenFile')
) {
throw new TypeError('Worker credential file delivery options are invalid');
}
this.stages = new PrivateDirectoryAuthority(
options.stageDirectory,
'Worker credential stage directory',
);
this.targetTokenFile = boundedAbsolutePath(
options.targetTokenFile,
'Worker credential target token file',
);
const targetName = path.basename(this.targetTokenFile);
if (targetName === '.' || targetName === '..' || targetName.startsWith('.ql3w-')) {
throw new TypeError('Worker credential target token name is invalid');
}
this.targetParent = new PrivateDirectoryAuthority(
path.dirname(this.targetTokenFile),
'Worker credential target directory',
);
if (
this.stages.device === this.targetParent.device &&
this.stages.inode === this.targetParent.inode
) {
throw new TypeError('Worker credential stage and target directories must differ');
}
this.targetLockFile = path.join(
this.targetParent.directory,
TARGET_LOCK_NAME,
);
this.deploymentTargetDigest = createHash('sha256')
.update(TARGET_DIGEST_DOMAIN)
.update(this.targetTokenFile, 'utf8')
.update('\0', 'utf8')
.update(String(this.targetParent.uid), 'utf8')
.update('\0', 'utf8')
.update(this.targetParent.device.toString(), 'utf8')
.update('\0', 'utf8')
.update(this.targetParent.inode.toString(), 'utf8')
.digest('hex');
}
private stagePath(deliveryId: string): string {
if (!UUID_V4.test(deliveryId)) {
throw new WorkerCredentialDeliveryConflictError();
}
return path.join(this.stages.directory, `${deliveryId}.stage`);
}
private verifyStageCapacity(): void {
this.stages.verify();
const directory = fs.opendirSync(this.stages.directory);
let count = 0;
try {
for (;;) {
const entry = directory.readSync();
if (!entry) break;
count += 1;
if (
count >= MAX_WORKER_CREDENTIAL_FILE_STAGES ||
!STAGE_NAME.test(entry.name)
) {
throw new WorkerCredentialDeliveryUnavailableError();
}
}
} finally {
directory.closeSync();
}
}
private stableStageNames(): readonly string[] {
this.stages.verify();
const directory = fs.opendirSync(this.stages.directory);
const names: string[] = [];
let entries = 0;
try {
for (;;) {
const entry = directory.readSync();
if (!entry) break;
entries += 1;
if (entries > MAX_WORKER_CREDENTIAL_FILE_STAGES) {
throw new WorkerCredentialDeliveryUnavailableError();
}
const match = STAGE_NAME.exec(entry.name);
if (match) {
names.push(match[1]!);
continue;
}
if (STAGE_TEMP_NAME.test(entry.name)) {
throw new WorkerCredentialDeliveryUnavailableError();
}
throw new WorkerCredentialDeliveryUnavailableError();
}
} finally {
directory.closeSync();
}
return Object.freeze(names.sort());
}
private normalizeIntent(
value: WorkerCredentialDeliveryIntent,
): Readonly<WorkerCredentialDeliveryIntent> {
const intent = normalizeWorkerCredentialDeliveryIntent(value);
if (intent.deploymentTargetDigest !== this.deploymentTargetDigest) {
throw new WorkerCredentialDeliveryConflictError();
}
return intent;
}
private readStage(deliveryId: string): StagedSecret {
const material = readPrivateFile(
this.stages,
this.stagePath(deliveryId),
MAX_STAGE_BYTES,
);
let token: Buffer | undefined;
try {
if (!material.subarray(0, STAGE_MAGIC.byteLength).equals(STAGE_MAGIC)) {
throw new WorkerCredentialDeliveryUnavailableError();
}
const headerEnd = material.indexOf(0x0a, STAGE_MAGIC.byteLength);
const headerBytes = headerEnd - STAGE_MAGIC.byteLength;
if (
headerEnd < STAGE_MAGIC.byteLength ||
headerBytes < 2 ||
headerBytes > MAX_STAGE_HEADER_BYTES ||
headerEnd + 1 >= material.byteLength
) {
throw new WorkerCredentialDeliveryUnavailableError();
}
const intent = this.normalizeIntent(
JSON.parse(
material
.subarray(STAGE_MAGIC.byteLength, headerEnd)
.toString('utf8'),
),
);
if (intent.deliveryId !== deliveryId) {
throw new WorkerCredentialDeliveryConflictError();
}
token = Buffer.from(material.subarray(headerEnd + 1));
const parsed = parseToken(token);
if (
parsed.credentialId !== intent.credentialId ||
parsed.tokenDigest !== intent.tokenDigest
) {
throw new WorkerCredentialDeliveryConflictError();
}
const result = Object.freeze({ intent, token });
token = undefined;
return result;
} finally {
token?.fill(0);
material.fill(0);
}
}
private optionalStage(deliveryId: string): StagedSecret | null {
try {
return this.readStage(deliveryId);
} catch (error) {
if (isMissing(error)) return null;
throw error;
}
}
private optionalTarget(): ParsedToken | null {
let material: Buffer | undefined;
try {
material = readPrivateFile(
this.targetParent,
this.targetTokenFile,
MAX_TOKEN_BYTES,
);
return parseToken(material);
} catch (error) {
if (isMissing(error)) return null;
throw error;
} finally {
material?.fill(0);
}
}
private publicationDigest(
delivery: Readonly<WorkerCredentialDeliveryRecord>,
): string {
return createHash('sha256')
.update(PUBLICATION_DIGEST_DOMAIN)
.update(JSON.stringify({
deliveryId: delivery.deliveryId,
workerId: delivery.workerId,
credentialId: delivery.credentialId,
credentialVersion: delivery.credentialVersion,
previousCredentialId: delivery.previousCredentialId,
tokenDigest: delivery.tokenDigest,
deploymentTargetDigest: delivery.deploymentTargetDigest,
deploymentGeneration: delivery.deploymentGeneration,
}), 'utf8')
.digest('hex');
}
private assertTargetFence(
delivery: Readonly<WorkerCredentialDeliveryRecord>,
target: ParsedToken | null,
): 'published' | 'replace' {
if (
target?.credentialId === delivery.credentialId &&
target.tokenDigest === delivery.tokenDigest
) {
return 'published';
}
if (
target?.credentialId === delivery.credentialId ||
(delivery.previousCredentialId === null && target !== null) ||
(delivery.previousCredentialId !== null &&
target?.credentialId !== delivery.previousCredentialId)
) {
throw new WorkerCredentialDeliveryConflictError();
}
return 'replace';
}
private acquireTargetLock(deliveryId: string): OwnedTargetLock {
this.targetParent.verify();
let descriptor: number | undefined;
try {
descriptor = fs.openSync(
this.targetLockFile,
fs.constants.O_WRONLY |
fs.constants.O_CREAT |
fs.constants.O_EXCL |
(fs.constants.O_NOFOLLOW ?? 0),
0o600,
);
fs.writeFileSync(descriptor, `${JSON.stringify({ deliveryId })}\n`, 'utf8');
fs.fsyncSync(descriptor);
const stat = fs.fstatSync(descriptor, { bigint: true });
fs.closeSync(descriptor);
descriptor = undefined;
this.targetParent.sync();
return Object.freeze({ device: stat.dev, inode: stat.ino });
} catch (error) {
if (descriptor !== undefined) fs.closeSync(descriptor);
if (isExists(error)) throw new WorkerCredentialDeliveryUnavailableError();
preserveDomainError(error);
}
}
private releaseTargetLock(lock: OwnedTargetLock): void {
try {
const stat = fs.lstatSync(this.targetLockFile, { bigint: true });
if (
!stat.isFile() ||
stat.isSymbolicLink() ||
Number(stat.uid) !== this.targetParent.uid ||
(Number(stat.mode) & 0o777) !== 0o600 ||
stat.dev !== lock.device ||
stat.ino !== lock.inode
) {
return;
}
fs.unlinkSync(this.targetLockFile);
this.targetParent.sync();
} catch {
// A stale lock fails future rotations closed and requires explicit repair.
}
}
async inspect(
deliveryId: string,
): Promise<Readonly<WorkerCredentialDeliveryIntent> | null> {
let staged: StagedSecret | null = null;
try {
staged = this.optionalStage(deliveryId);
return staged?.intent ?? null;
} catch (error) {
return preserveDomainError(error);
} finally {
staged?.token.fill(0);
}
}
async listStaged(
options: Readonly<{ afterDeliveryId?: string; limit?: number }> = {},
): Promise<Readonly<WorkerCredentialFileStagePage>> {
try {
if (
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some(
(key) => key !== 'afterDeliveryId' && key !== 'limit',
)
) {
throw new WorkerCredentialDeliveryConflictError();
}
const limit = options.limit ?? 16;
if (
!Number.isInteger(limit) ||
limit < 1 ||
limit > MAX_WORKER_CREDENTIAL_FILE_STAGE_PAGE_SIZE ||
(options.afterDeliveryId !== undefined &&
!UUID_V4.test(options.afterDeliveryId))
) {
throw new WorkerCredentialDeliveryConflictError();
}
const names = this.stableStageNames().filter(
(name) =>
options.afterDeliveryId === undefined ||
name > options.afterDeliveryId,
);
const selected = names.slice(0, limit + 1);
const stages: Readonly<WorkerCredentialDeliveryIntent>[] = [];
for (const deliveryId of selected.slice(0, limit)) {
const staged = this.readStage(deliveryId);
try {
stages.push(staged.intent);
} finally {
staged.token.fill(0);
}
}
const truncated = selected.length > limit;
return Object.freeze({
stages: Object.freeze(stages),
truncated,
...(truncated
? { nextCursor: stages[stages.length - 1]!.deliveryId }
: {}),
});
} catch (error) {
return preserveDomainError(error);
}
}
async stage(
value: Readonly<WorkerCredentialDeliveryIntent>,
token: Buffer,
): Promise<void> {
let serialized: Buffer | undefined;
let descriptor: number | undefined;
const intent = this.normalizeIntent(value);
const parsed = Buffer.isBuffer(token) ? parseToken(token) : null;
if (
!parsed ||
parsed.credentialId !== intent.credentialId ||
parsed.tokenDigest !== intent.tokenDigest
) {
throw new WorkerCredentialDeliveryConflictError();
}
const targetPath = this.stagePath(intent.deliveryId);
const temporaryPath = path.join(
this.stages.directory,
`.${intent.deliveryId}.${randomUUID()}.tmp`,
);
try {
const existing = this.optionalStage(intent.deliveryId);
if (existing) {
try {
if (!sameIntent(existing.intent, intent)) {
throw new WorkerCredentialDeliveryConflictError();
}
return;
} finally {
existing.token.fill(0);
}
}
this.verifyStageCapacity();
const header = Buffer.from(JSON.stringify(intent), 'utf8');
if (header.byteLength > MAX_STAGE_HEADER_BYTES) {
throw new WorkerCredentialDeliveryUnavailableError();
}
serialized = Buffer.concat([
STAGE_MAGIC,
header,
Buffer.from('\n', 'ascii'),
token,
]);
header.fill(0);
if (serialized.byteLength > MAX_STAGE_BYTES) {
throw new WorkerCredentialDeliveryUnavailableError();
}
descriptor = fs.openSync(
temporaryPath,
fs.constants.O_WRONLY |
fs.constants.O_CREAT |
fs.constants.O_EXCL |
(fs.constants.O_NOFOLLOW ?? 0),
0o600,
);
fs.writeFileSync(descriptor, serialized);
fs.fsyncSync(descriptor);
fs.closeSync(descriptor);
descriptor = undefined;
try {
fs.linkSync(temporaryPath, targetPath);
this.stages.sync();
} catch (error) {
if (!isExists(error)) throw error;
}
try {
fs.unlinkSync(temporaryPath);
this.stages.sync();
} catch (error) {
if (!isMissing(error)) throw error;
}
const winner = this.readStage(intent.deliveryId);
try {
if (!sameIntent(winner.intent, intent)) {
throw new WorkerCredentialDeliveryConflictError();
}
} finally {
winner.token.fill(0);
}
} catch (error) {
return preserveDomainError(error);
} finally {
serialized?.fill(0);
if (descriptor !== undefined) fs.closeSync(descriptor);
try {
fs.unlinkSync(temporaryPath);
this.stages.sync();
} catch {
// The no-replace stage, if published, remains authoritative.
}
}
}
async publish(
value: Readonly<WorkerCredentialDeliveryRecord>,
): Promise<Readonly<{ publicationDigest: string }>> {
const delivery = normalizeWorkerCredentialDeliveryRecord(value);
if (
delivery.state !== 'credential_committed' ||
delivery.version !== 1 ||
delivery.deploymentTargetDigest !== this.deploymentTargetDigest
) {
throw new WorkerCredentialDeliveryConflictError();
}
let staged: StagedSecret | null = null;
let temporaryPath: string | undefined;
let descriptor: number | undefined;
let targetLock: OwnedTargetLock | undefined;
try {
staged = this.optionalStage(delivery.deliveryId);
if (!staged || !recordMatchesIntent(delivery, staged.intent)) {
throw new WorkerCredentialDeliveryConflictError();
}
if (this.assertTargetFence(delivery, this.optionalTarget()) === 'published') {
return Object.freeze({
publicationDigest: this.publicationDigest(delivery),
});
}
targetLock = this.acquireTargetLock(delivery.deliveryId);
if (this.assertTargetFence(delivery, this.optionalTarget()) === 'published') {
return Object.freeze({
publicationDigest: this.publicationDigest(delivery),
});
}
temporaryPath = path.join(
this.targetParent.directory,
`.ql3w-${delivery.deliveryId}-${randomUUID()}.tmp`,
);
descriptor = fs.openSync(
temporaryPath,
fs.constants.O_WRONLY |
fs.constants.O_CREAT |
fs.constants.O_EXCL |
(fs.constants.O_NOFOLLOW ?? 0),
0o600,
);
fs.writeFileSync(descriptor, staged.token);
fs.fsyncSync(descriptor);
fs.closeSync(descriptor);
descriptor = undefined;
fs.renameSync(temporaryPath, this.targetTokenFile);
temporaryPath = undefined;
this.targetParent.sync();
if (this.assertTargetFence(delivery, this.optionalTarget()) !== 'published') {
throw new WorkerCredentialDeliveryUnavailableError();
}
return Object.freeze({
publicationDigest: this.publicationDigest(delivery),
});
} catch (error) {
return preserveDomainError(error);
} finally {
staged?.token.fill(0);
if (descriptor !== undefined) fs.closeSync(descriptor);
if (temporaryPath) {
try {
fs.unlinkSync(temporaryPath);
this.targetParent.sync();
} catch {
// The target was never published from this temporary path.
}
}
if (targetLock) this.releaseTargetLock(targetLock);
}
}
async discard(value: Readonly<WorkerCredentialDeliveryIntent>): Promise<void> {
const intent = this.normalizeIntent(value);
let staged: StagedSecret | null = null;
try {
staged = this.optionalStage(intent.deliveryId);
if (!staged) return;
if (!sameIntent(staged.intent, intent)) {
throw new WorkerCredentialDeliveryConflictError();
}
const target = this.optionalTarget();
if (
target?.credentialId === intent.credentialId &&
target.tokenDigest === intent.tokenDigest
) {
throw new WorkerCredentialDeliveryConflictError();
}
fs.unlinkSync(this.stagePath(intent.deliveryId));
this.stages.sync();
} catch (error) {
preserveDomainError(error);
} finally {
staged?.token.fill(0);
}
}
}
@@ -0,0 +1,549 @@
/** Short-lived Kubernetes TokenRequest delivery session boundary. */
import {
WorkerCredentialKubernetesDeliveryAdapter,
type WorkerCredentialKubernetesDeliveryAdapterOptions,
type WorkerCredentialKubernetesDeploymentApi,
type WorkerCredentialKubernetesSecretApi,
} from './workerCredentialKubernetesDelivery';
export const WORKER_CREDENTIAL_KUBERNETES_TOKEN_REQUEST_SECONDS = 600;
const MIN_USEFUL_TOKEN_SECONDS = 30;
const MAX_TOKEN_BYTES = 16 * 1024;
const DNS_LABEL = /^[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?$/;
const DNS_SUBDOMAIN =
/^[a-z0-9](?:[-a-z0-9.]{0,251}[a-z0-9])?$/;
const SAFE_JWT_ALGORITHM = /^[A-Za-z0-9_-]{2,32}$/;
const JWT = /^([A-Za-z0-9_-]+)\.([A-Za-z0-9_-]+)\.([A-Za-z0-9_-]+)$/;
type JsonObject = Record<string, unknown>;
type KubernetesModule = typeof import('@kubernetes/client-node', {
with: { 'resolution-mode': 'import' }
});
type KubernetesConfig = InstanceType<KubernetesModule['KubeConfig']>;
interface TokenRequestResponse {
apiVersion?: string;
kind?: string;
status?: {
token?: string;
expirationTimestamp?: Date | string;
};
}
interface AccessReviewAttributes {
readonly namespace?: string;
readonly verb: string;
readonly group?: string;
readonly resource: string;
readonly subresource?: string;
readonly name?: string;
}
export interface WorkerCredentialKubernetesTokenRequestApi {
createNamespacedServiceAccountToken(request: Readonly<{
name: string;
namespace: string;
body: Readonly<{
apiVersion: 'authentication.k8s.io/v1';
kind: 'TokenRequest';
spec: Readonly<{ expirationSeconds: 600 }>;
}>;
}>): Promise<TokenRequestResponse>;
}
export interface WorkerCredentialKubernetesAuthorizationApi {
createSelfSubjectAccessReview(request: Readonly<{
body: Readonly<{
apiVersion: 'authorization.k8s.io/v1';
kind: 'SelfSubjectAccessReview';
spec: Readonly<{
resourceAttributes: AccessReviewAttributes;
}>;
}>;
}>): Promise<Readonly<{
status?: Readonly<{
allowed?: boolean;
denied?: boolean;
reason?: string;
}>;
}>>;
}
export interface WorkerCredentialKubernetesRestrictedClients {
readonly secrets: WorkerCredentialKubernetesSecretApi;
readonly deployments: WorkerCredentialKubernetesDeploymentApi;
readonly authorization: WorkerCredentialKubernetesAuthorizationApi;
dispose(): void | Promise<void>;
}
export interface WorkerCredentialKubernetesTokenRequestSessionOptions {
readonly serviceAccountName: string;
readonly identitySecretName: string;
readonly delivery: WorkerCredentialKubernetesDeliveryAdapterOptions;
readonly now?: () => number;
}
export interface WorkerCredentialKubernetesTokenRequestEvidence {
readonly tokenLifetimeSeconds: number;
readonly issuerAllowedChecks: number;
readonly issuerDeniedChecks: number;
readonly allowedChecks: number;
readonly deniedChecks: number;
}
export interface WorkerCredentialKubernetesTokenRequestContext {
readonly delivery: WorkerCredentialKubernetesDeliveryAdapter;
readonly evidence: WorkerCredentialKubernetesTokenRequestEvidence;
}
export interface WorkerCredentialKubernetesTokenRequestSession {
withDelivery<T>(
operation: (
context: Readonly<WorkerCredentialKubernetesTokenRequestContext>,
) => Promise<T>,
): Promise<T>;
}
export class WorkerCredentialKubernetesTokenRequestUnavailableError
extends Error {
readonly code = 'QL3_WORKER_CREDENTIAL_KUBERNETES_TOKEN_REQUEST_UNAVAILABLE';
constructor() {
super('Worker credential Kubernetes TokenRequest session is unavailable');
this.name = 'WorkerCredentialKubernetesTokenRequestUnavailableError';
}
}
const VALIDATION_SECRET_API: WorkerCredentialKubernetesSecretApi = {
async readNamespacedSecret() { throw new Error('validation only'); },
async createNamespacedSecret() { throw new Error('validation only'); },
async replaceNamespacedSecret() { throw new Error('validation only'); },
async deleteNamespacedSecret() { throw new Error('validation only'); },
async listNamespacedSecret() { throw new Error('validation only'); },
};
const VALIDATION_DEPLOYMENT_API: WorkerCredentialKubernetesDeploymentApi = {
async readNamespacedDeployment() { throw new Error('validation only'); },
async replaceNamespacedDeployment() { throw new Error('validation only'); },
};
function jsonObject(value: unknown): JsonObject {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
}
return value as JsonObject;
}
function decodeJwtSegment(value: string): JsonObject {
try {
const bytes = Buffer.from(value, 'base64url');
if (bytes.toString('base64url') !== value) {
throw new Error('non-canonical base64url');
}
return jsonObject(JSON.parse(bytes.toString('utf8')));
} catch (error) {
if (error instanceof WorkerCredentialKubernetesTokenRequestUnavailableError) {
throw error;
}
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
}
}
function tokenEvidence(
response: TokenRequestResponse,
namespace: string,
serviceAccountName: string,
observedAtMs: number,
): Readonly<{ token: string; lifetimeSeconds: number }> {
if (
response?.apiVersion !== 'authentication.k8s.io/v1' ||
response.kind !== 'TokenRequest' ||
typeof response.status?.token !== 'string' ||
response.status.token.length < 1 ||
Buffer.byteLength(response.status.token, 'utf8') > MAX_TOKEN_BYTES
) {
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
}
const match = JWT.exec(response.status.token);
if (!match) {
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
}
const header = decodeJwtSegment(match[1]!);
const claims = decodeJwtSegment(match[2]!);
if (
typeof header.alg !== 'string' ||
!SAFE_JWT_ALGORITHM.test(header.alg) ||
header.alg.toLowerCase() === 'none' ||
claims.sub !== `system:serviceaccount:${namespace}:${serviceAccountName}` ||
!Number.isSafeInteger(claims.iat) ||
!Number.isSafeInteger(claims.exp)
) {
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
}
const issuedAtSeconds = claims.iat as number;
const expiresAtSeconds = claims.exp as number;
const lifetimeSeconds = expiresAtSeconds - issuedAtSeconds;
const expiration = response.status.expirationTimestamp;
const expirationMs = expiration instanceof Date
? expiration.getTime()
: typeof expiration === 'string'
? Date.parse(expiration)
: Number.NaN;
if (
lifetimeSeconds < MIN_USEFUL_TOKEN_SECONDS ||
lifetimeSeconds > WORKER_CREDENTIAL_KUBERNETES_TOKEN_REQUEST_SECONDS ||
!Number.isSafeInteger(expirationMs) ||
expirationMs !== expiresAtSeconds * 1_000 ||
expirationMs - observedAtMs < MIN_USEFUL_TOKEN_SECONDS * 1_000
) {
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
}
return Object.freeze({
token: response.status.token,
lifetimeSeconds,
});
}
function accessMatrix(
delivery: WorkerCredentialKubernetesDeliveryAdapterOptions,
identitySecretName: string,
serviceAccountName: string,
): Readonly<{
allowed: readonly AccessReviewAttributes[];
denied: readonly AccessReviewAttributes[];
}> {
const allowed: readonly AccessReviewAttributes[] = [
{ namespace: delivery.stageNamespace, verb: 'get', resource: 'secrets', name: 'stage' },
{ namespace: delivery.stageNamespace, verb: 'list', resource: 'secrets' },
{ namespace: delivery.stageNamespace, verb: 'create', resource: 'secrets' },
{ namespace: delivery.stageNamespace, verb: 'delete', resource: 'secrets', name: 'stage' },
{ namespace: delivery.namespace, verb: 'get', resource: 'secrets', name: delivery.targetSecretName },
{ namespace: delivery.namespace, verb: 'update', resource: 'secrets', name: delivery.targetSecretName },
{ namespace: delivery.namespace, verb: 'get', group: 'apps', resource: 'deployments', name: delivery.targetDeploymentName },
{ namespace: delivery.namespace, verb: 'update', group: 'apps', resource: 'deployments', name: delivery.targetDeploymentName },
];
const denied: readonly AccessReviewAttributes[] = [
{ namespace: delivery.stageNamespace, verb: 'update', resource: 'secrets', name: 'stage' },
{ namespace: delivery.stageNamespace, verb: 'patch', resource: 'secrets', name: 'stage' },
{ namespace: delivery.stageNamespace, verb: 'watch', resource: 'secrets' },
{ namespace: delivery.stageNamespace, verb: 'get', resource: 'configmaps', name: 'any' },
{ namespace: delivery.namespace, verb: 'list', resource: 'secrets' },
{ namespace: delivery.namespace, verb: 'get', resource: 'secrets', name: identitySecretName },
{ namespace: delivery.namespace, verb: 'create', resource: 'secrets' },
{ namespace: delivery.namespace, verb: 'delete', resource: 'secrets', name: delivery.targetSecretName },
{ namespace: delivery.namespace, verb: 'patch', resource: 'secrets', name: delivery.targetSecretName },
{ namespace: delivery.namespace, verb: 'watch', resource: 'secrets' },
{ namespace: delivery.namespace, verb: 'list', group: 'apps', resource: 'deployments' },
{ namespace: delivery.namespace, verb: 'get', group: 'apps', resource: 'deployments', name: 'other' },
{ namespace: delivery.namespace, verb: 'patch', group: 'apps', resource: 'deployments', name: delivery.targetDeploymentName },
{ namespace: delivery.namespace, verb: 'delete', group: 'apps', resource: 'deployments', name: delivery.targetDeploymentName },
{ namespace: delivery.namespace, verb: 'get', resource: 'pods' },
{ namespace: delivery.namespace, verb: 'list', resource: 'pods' },
{ namespace: delivery.namespace, verb: 'create', resource: 'pods', subresource: 'exec' },
{ namespace: delivery.namespace, verb: 'delete', resource: 'pods', name: 'any' },
{ namespace: delivery.stageNamespace, verb: 'create', resource: 'serviceaccounts', subresource: 'token', name: serviceAccountName },
{ verb: 'get', resource: 'namespaces', name: delivery.namespace },
];
return Object.freeze({ allowed, denied });
}
function issuerAccessMatrix(
delivery: WorkerCredentialKubernetesDeliveryAdapterOptions,
serviceAccountName: string,
): Readonly<{
allowed: readonly AccessReviewAttributes[];
denied: readonly AccessReviewAttributes[];
}> {
return Object.freeze({
allowed: [{
namespace: delivery.stageNamespace,
verb: 'create',
resource: 'serviceaccounts',
subresource: 'token',
name: serviceAccountName,
}],
denied: [
{
namespace: delivery.stageNamespace,
verb: 'create',
resource: 'serviceaccounts',
subresource: 'token',
name: 'other',
},
{ namespace: delivery.stageNamespace, verb: 'get', resource: 'secrets', name: 'any' },
{ namespace: delivery.stageNamespace, verb: 'list', resource: 'secrets' },
{ namespace: delivery.stageNamespace, verb: 'create', resource: 'secrets' },
{ namespace: delivery.namespace, verb: 'get', resource: 'secrets', name: delivery.targetSecretName },
{ namespace: delivery.namespace, verb: 'get', group: 'apps', resource: 'deployments', name: delivery.targetDeploymentName },
{ namespace: delivery.namespace, verb: 'get', resource: 'pods' },
{ verb: 'get', resource: 'namespaces', name: delivery.namespace },
],
});
}
async function assertAccess(
authorization: WorkerCredentialKubernetesAuthorizationApi,
expected: boolean,
checks: readonly AccessReviewAttributes[],
): Promise<void> {
for (const attributes of checks) {
let result;
try {
result = await authorization.createSelfSubjectAccessReview({
body: {
apiVersion: 'authorization.k8s.io/v1',
kind: 'SelfSubjectAccessReview',
spec: { resourceAttributes: attributes },
},
});
} catch {
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
}
if (result?.status?.allowed !== expected) {
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
}
}
}
export function createWorkerCredentialKubernetesTokenRequestSession(
tokenRequests: WorkerCredentialKubernetesTokenRequestApi,
issuerAuthorization: WorkerCredentialKubernetesAuthorizationApi,
createRestrictedClients: (
token: string,
) => WorkerCredentialKubernetesRestrictedClients,
options: WorkerCredentialKubernetesTokenRequestSessionOptions,
): WorkerCredentialKubernetesTokenRequestSession {
if (
!tokenRequests ||
typeof tokenRequests.createNamespacedServiceAccountToken !== 'function' ||
!issuerAuthorization ||
typeof issuerAuthorization.createSelfSubjectAccessReview !== 'function' ||
typeof createRestrictedClients !== 'function' ||
!options ||
typeof options !== 'object' ||
Array.isArray(options) ||
Object.keys(options).some((key) =>
!['serviceAccountName', 'identitySecretName', 'delivery', 'now'].includes(key)) ||
typeof options.serviceAccountName !== 'string' ||
!DNS_LABEL.test(options.serviceAccountName) ||
typeof options.identitySecretName !== 'string' ||
!DNS_SUBDOMAIN.test(options.identitySecretName) ||
Buffer.byteLength(options.identitySecretName, 'utf8') > 253 ||
(options.now !== undefined && typeof options.now !== 'function')
) {
throw new TypeError('Worker credential Kubernetes TokenRequest options are invalid');
}
const validation = new WorkerCredentialKubernetesDeliveryAdapter(
VALIDATION_SECRET_API,
VALIDATION_DEPLOYMENT_API,
options.delivery,
);
const now = options.now ?? Date.now;
const matrix = accessMatrix(
options.delivery,
options.identitySecretName,
options.serviceAccountName,
);
const issuerMatrix = issuerAccessMatrix(
options.delivery,
options.serviceAccountName,
);
return Object.freeze({
async withDelivery<T>(
operation: (
context: Readonly<WorkerCredentialKubernetesTokenRequestContext>,
) => Promise<T>,
): Promise<T> {
if (typeof operation !== 'function') {
throw new TypeError('Worker credential Kubernetes operation is invalid');
}
const requestedAtMs = now();
if (!Number.isSafeInteger(requestedAtMs) || requestedAtMs < 0) {
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
}
let response: TokenRequestResponse | undefined;
let issuedToken = '';
let clients: WorkerCredentialKubernetesRestrictedClients | undefined;
try {
await assertAccess(issuerAuthorization, true, issuerMatrix.allowed);
await assertAccess(issuerAuthorization, false, issuerMatrix.denied);
try {
response = await tokenRequests.createNamespacedServiceAccountToken({
name: options.serviceAccountName,
namespace: options.delivery.stageNamespace,
body: {
apiVersion: 'authentication.k8s.io/v1',
kind: 'TokenRequest',
spec: {
expirationSeconds:
WORKER_CREDENTIAL_KUBERNETES_TOKEN_REQUEST_SECONDS,
},
},
});
} catch {
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
}
const observedAtMs = now();
if (
!Number.isSafeInteger(observedAtMs) ||
observedAtMs < requestedAtMs
) {
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
}
const evidence = tokenEvidence(
response,
options.delivery.stageNamespace,
options.serviceAccountName,
observedAtMs,
);
issuedToken = evidence.token;
try {
clients = createRestrictedClients(issuedToken);
} catch {
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
} finally {
if (response.status) response.status.token = '';
issuedToken = '';
}
if (
!clients ||
!clients.authorization ||
typeof clients.authorization.createSelfSubjectAccessReview !== 'function' ||
typeof clients.dispose !== 'function'
) {
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
}
const delivery = new WorkerCredentialKubernetesDeliveryAdapter(
clients.secrets,
clients.deployments,
options.delivery,
);
if (delivery.deploymentTargetDigest !== validation.deploymentTargetDigest) {
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
}
await assertAccess(clients.authorization, true, matrix.allowed);
await assertAccess(clients.authorization, false, matrix.denied);
return await operation(Object.freeze({
delivery,
evidence: Object.freeze({
tokenLifetimeSeconds: evidence.lifetimeSeconds,
issuerAllowedChecks: issuerMatrix.allowed.length,
issuerDeniedChecks: issuerMatrix.denied.length,
allowedChecks: matrix.allowed.length,
deniedChecks: matrix.denied.length,
}),
}));
} finally {
if (response?.status) response.status.token = '';
issuedToken = '';
try {
await clients?.dispose();
} catch {
throw new WorkerCredentialKubernetesTokenRequestUnavailableError();
} finally {
clients = undefined;
response = undefined;
}
}
},
});
}
export function createWorkerCredentialKubernetesKubeConfigTokenRequestSession(
issuerKubeConfig: KubernetesConfig,
kubernetes: KubernetesModule,
options: WorkerCredentialKubernetesTokenRequestSessionOptions,
): WorkerCredentialKubernetesTokenRequestSession {
if (
!issuerKubeConfig ||
!kubernetes ||
typeof kubernetes !== 'object' ||
typeof kubernetes.KubeConfig !== 'function' ||
typeof kubernetes.CoreV1Api !== 'function' ||
typeof kubernetes.AppsV1Api !== 'function' ||
typeof kubernetes.AuthorizationV1Api !== 'function' ||
typeof issuerKubeConfig.getCurrentCluster !== 'function' ||
typeof issuerKubeConfig.makeApiClient !== 'function'
) {
throw new TypeError('Worker credential Kubernetes issuer kubeconfig is invalid');
}
const cluster = issuerKubeConfig.getCurrentCluster();
let server: URL;
try {
server = new URL(cluster?.server ?? '');
} catch {
throw new TypeError('Worker credential Kubernetes issuer cluster is invalid');
}
if (
!cluster ||
server.protocol !== 'https:' ||
server.username !== '' ||
server.password !== '' ||
server.hash !== '' ||
cluster.skipTLSVerify === true ||
(typeof cluster.caData !== 'string' && typeof cluster.caFile !== 'string')
) {
throw new TypeError('Worker credential Kubernetes issuer cluster is invalid');
}
const tokenRequestClient = issuerKubeConfig.makeApiClient(
kubernetes.CoreV1Api,
);
const issuerAuthorization = issuerKubeConfig.makeApiClient(
kubernetes.AuthorizationV1Api,
) as unknown as WorkerCredentialKubernetesAuthorizationApi;
const tokenRequests: WorkerCredentialKubernetesTokenRequestApi = {
async createNamespacedServiceAccountToken(request) {
return await tokenRequestClient.createNamespacedServiceAccountToken({
...request,
body: {
...request.body,
spec: {
audiences: [],
expirationSeconds: request.body.spec.expirationSeconds,
},
},
}) as unknown as TokenRequestResponse;
},
};
return createWorkerCredentialKubernetesTokenRequestSession(
tokenRequests,
issuerAuthorization,
(token) => {
const restricted = new kubernetes.KubeConfig();
restricted.loadFromOptions({
clusters: [{ ...cluster, name: 'ql3-worker-credential-delivery' }],
users: [{ name: 'ql3-worker-credential-delivery', token }],
contexts: [{
name: 'ql3-worker-credential-delivery',
cluster: 'ql3-worker-credential-delivery',
user: 'ql3-worker-credential-delivery',
namespace: options.delivery.stageNamespace,
}],
currentContext: 'ql3-worker-credential-delivery',
});
let active = true;
return {
secrets: restricted.makeApiClient(
kubernetes.CoreV1Api,
) as unknown as WorkerCredentialKubernetesSecretApi,
deployments: restricted.makeApiClient(
kubernetes.AppsV1Api,
) as unknown as WorkerCredentialKubernetesDeploymentApi,
authorization: restricted.makeApiClient(
kubernetes.AuthorizationV1Api,
) as unknown as WorkerCredentialKubernetesAuthorizationApi,
dispose() {
if (!active) return;
active = false;
for (const user of restricted.getUsers()) {
(user as { token?: string }).token = '';
}
restricted.setCurrentContext('disposed');
},
};
},
options,
);
}
@@ -0,0 +1,186 @@
/** Worker credential management client boundary. */
import {
executeClusterAuthenticatedManagementClient,
type ClusterAuthenticatedManagementClientResult,
type ClusterPluginPackageManagementClientConnectionOptions,
type ClusterPluginPackageManagementClientPaths,
} from '../management-support/pluginPackageManagementClient';
import {
normalizeClusterWorkerCredentialManagementCommand,
type ClusterWorkerCredentialManagementCommand,
type ClusterWorkerCredentialManagementTransportResult,
} from './management-server/workerCredentialManagementTransport';
const MANAGEMENT_PATH = '/api/v3/worker-credentials/management';
const CONTROL_PATTERN = /[\u0000-\u001f\u007f]/;
export type ClusterWorkerCredentialManagementClientPaths =
ClusterPluginPackageManagementClientPaths;
export type ClusterWorkerCredentialManagementClientConnectionOptions =
ClusterPluginPackageManagementClientConnectionOptions;
export type ClusterWorkerCredentialManagementClientResult =
ClusterAuthenticatedManagementClientResult<ClusterWorkerCredentialManagementTransportResult>;
function invalid(): never {
throw new Error('Worker credential management response is invalid');
}
function exactRecord(
value: unknown,
keys: readonly string[],
): Record<string, unknown> {
if (!value || typeof value !== 'object' || Array.isArray(value)) invalid();
const record = value as Record<string, unknown>;
const actual = Object.keys(record).sort();
const expected = [...keys].sort();
if (
actual.length !== expected.length ||
actual.some((key, index) => key !== expected[index])
) {
invalid();
}
return record;
}
function boundedScalar(value: unknown): void {
if (
value !== null &&
!(typeof value === 'boolean') &&
!(typeof value === 'number' && Number.isSafeInteger(value)) &&
!(
typeof value === 'string' &&
value.length <= 2_048 &&
!CONTROL_PATTERN.test(value)
)
) {
invalid();
}
}
function subject(value: unknown): void {
const record = exactRecord(value, ['type', 'id']);
if (record.type !== 'user') invalid();
boundedScalar(record.id);
}
function plan(value: unknown): void {
const record = exactRecord(value, [
'actionRef',
'authorityProjectId',
'action',
'target',
'requestedBy',
'plannedAtMs',
'expiresAtMs',
'previewDigest',
'planDigest',
]);
const target = exactRecord(record.target, [
'deliveryId',
'workerId',
'credentialId',
'previousCredentialId',
'credentialNotBeforeAtMs',
'credentialExpiresAtMs',
'deploymentTargetDigest',
'deploymentGeneration',
]);
for (const entry of Object.values(record)) {
if (entry !== record.target && entry !== record.requestedBy)
boundedScalar(entry);
}
for (const entry of Object.values(target)) boundedScalar(entry);
subject(record.requestedBy);
if (!['issue', 'rotate'].includes(String(record.action))) invalid();
}
function approval(value: unknown): void {
const record = exactRecord(value, [
'id',
'projectId',
'version',
'state',
'risk',
'decisionMode',
'requestedBy',
'requestedAtMs',
'expiresAtMs',
'decision',
'decisionReasonCode',
'decidedBy',
'decidedAtMs',
'dispatchId',
'consumedAtMs',
'actionType',
'actionRef',
'actionDigest',
'previewDigest',
]);
for (const entry of Object.values(record)) {
if (entry !== record.requestedBy && entry !== record.decidedBy)
boundedScalar(entry);
}
subject(record.requestedBy);
if (record.decidedBy !== null) subject(record.decidedBy);
if (
!/^worker_credential\.delivery\.(?:issue|rotate)$/.test(
String(record.actionType),
)
) {
invalid();
}
}
export function validateClusterWorkerCredentialManagementClientResult(
value: unknown,
command: Readonly<ClusterWorkerCredentialManagementCommand>,
): Readonly<ClusterWorkerCredentialManagementTransportResult> {
const operation = command.operation;
const keys =
operation === 'worker-credential.plan'
? ['schemaVersion', 'operation', 'status', 'plan']
: operation === 'worker-credential.propose'
? ['schemaVersion', 'operation', 'approvalStatus', 'plan', 'approval']
: operation === 'worker-credential.decide'
? ['schemaVersion', 'operation', 'status', 'approval']
: ['schemaVersion', 'operation', 'plan', 'approval', 'stale'];
const record = exactRecord(value, keys);
if (record.schemaVersion !== 1 || record.operation !== operation) invalid();
if (operation === 'worker-credential.plan') {
if (!['created', 'existing'].includes(String(record.status))) invalid();
plan(record.plan);
} else if (operation === 'worker-credential.propose') {
if (!['created', 'existing'].includes(String(record.approvalStatus)))
invalid();
plan(record.plan);
approval(record.approval);
} else if (operation === 'worker-credential.decide') {
if (!['decided', 'existing'].includes(String(record.status))) invalid();
approval(record.approval);
} else {
if (typeof record.stale !== 'boolean') invalid();
if (record.plan !== null) plan(record.plan);
if (record.approval !== null) approval(record.approval);
}
return Object.freeze(
record as unknown as ClusterWorkerCredentialManagementTransportResult,
);
}
const PROTOCOL = Object.freeze({
managementPath: MANAGEMENT_PATH,
clientCertificate: 'required' as const,
normalizeCommand: normalizeClusterWorkerCredentialManagementCommand,
validateResult: validateClusterWorkerCredentialManagementClientResult,
});
export async function executeClusterWorkerCredentialManagementClient(
paths: ClusterWorkerCredentialManagementClientPaths,
connectionOptions?: ClusterWorkerCredentialManagementClientConnectionOptions,
): Promise<Readonly<ClusterWorkerCredentialManagementClientResult>> {
return executeClusterAuthenticatedManagementClient(
paths,
PROTOCOL,
connectionOptions,
);
}
@@ -0,0 +1,96 @@
#!/usr/bin/env node
/** One-shot Worker credential management client CLI boundary. */
import { ClusterPluginPackageManagementClientRemoteError } from '../management-support/pluginPackageManagementClient';
import { executeClusterWorkerCredentialManagementClient } from './workerCredentialManagementClient';
const USAGE =
'Usage: ql3-worker-credential-client --config=/absolute/client.json --command=/absolute/command.json --assertion=/absolute/assertion.jwt';
function parseArguments(
argv: readonly string[],
): Readonly<{
configFile: string;
commandFile: string;
assertionFile: string;
}> | null {
if (argv.length !== 3) return null;
const values = new Map<string, string>();
for (const argument of argv) {
const match = /^--(config|command|assertion)=(\/.+)$/.exec(argument);
if (!match || values.has(match[1]!)) return null;
values.set(match[1]!, match[2]!);
}
if (
!values.has('config') ||
!values.has('command') ||
!values.has('assertion')
) {
return null;
}
return Object.freeze({
configFile: values.get('config')!,
commandFile: values.get('command')!,
assertionFile: values.get('assertion')!,
});
}
function failureFact(error: unknown): Readonly<Record<string, unknown>> {
const candidate = error as { readonly code?: unknown };
return Object.freeze({
schemaVersion: 1,
component: 'qinglong3-worker-credential-management-client',
event: 'command_failed',
code:
typeof candidate?.code === 'string' && candidate.code.length <= 128
? candidate.code
: 'QL3_WORKER_CREDENTIAL_MANAGEMENT_CLIENT_FAILED',
...(error instanceof ClusterPluginPackageManagementClientRemoteError
? {
statusCode: error.statusCode,
responseCode: error.responseCode,
requestId: error.requestId,
...(error.retryAfterSeconds === null
? {}
: { retryAfterSeconds: error.retryAfterSeconds }),
}
: {}),
});
}
async function run(argv: readonly string[]): Promise<void> {
if (argv.length === 1 && (argv[0] === '--help' || argv[0] === '-h')) {
process.stdout.write(`${USAGE}\n`);
return;
}
const paths = parseArguments(argv);
if (!paths) {
process.stderr.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-worker-credential-management-client',
event: 'usage_invalid',
code: 'QL3_WORKER_CREDENTIAL_MANAGEMENT_CLIENT_USAGE_INVALID',
})}\n`,
);
process.exitCode = 64;
return;
}
try {
const result = await executeClusterWorkerCredentialManagementClient(paths);
process.stdout.write(
`${JSON.stringify({
schemaVersion: 1,
component: 'qinglong3-worker-credential-management-client',
event: 'command_completed',
requestId: result.requestId,
result: result.result,
})}\n`,
);
} catch (error) {
process.stderr.write(`${JSON.stringify(failureFact(error))}\n`);
process.exitCode = 1;
}
}
void run(process.argv.slice(2));
@@ -0,0 +1,546 @@
/** Approved Worker credential management execution boundary. */
import { createHash } from 'node:crypto';
import {
PostgresApprovedActionExecutionRepository,
PostgresApprovalRequestRepository,
PostgresProjectPolicyRepository,
PostgresWorkerCredentialAdministrationRepository,
PostgresWorkerCredentialManagementPlanReader,
assertPostgresWorkerCredentialExecutorSchemaReady,
type PostgresSchemaReadinessReport,
} from '@qinglong/cluster-postgres/worker-credential-executor';
import type {
OpenPostgresDatabase,
PostgresDatabaseResource,
PostgresPool,
} from '@qinglong/runtime-core';
import {
normalizeApprovalRequestRecord,
type ApprovedActionBinding,
type ApprovedActionDispatchRecord,
} from '@qinglong/runtime-core/approved-action';
import type { ApprovedActionExecutionRecord } from '@qinglong/runtime-core/approved-action-execution';
import { ProjectPolicyEngine } from '@qinglong/runtime-core/project-policy';
import type {
SecurityPolicyFence,
SecurityPrincipal,
SecuritySubject,
} from '@qinglong/runtime-core/security';
import type { SecurityAuditRecord } from '@qinglong/runtime-core/security-audit';
import {
normalizeWorkerCredentialManagementPlan,
type WorkerCredentialManagementPlan,
} from '@qinglong/runtime-core/worker-credential-management-plan';
import {
createRecoverableWorkerCredentialIssuer,
type RecoverableWorkerCredentialIssueResult,
} from './workerCredentialDelivery';
import type {
WorkerCredentialKubernetesTokenRequestEvidence,
WorkerCredentialKubernetesTokenRequestSession,
} from './workerCredentialKubernetesTokenRequest';
import {
WorkerCredentialManagementConflictError,
WorkerCredentialManagementRequestError,
WorkerCredentialManagementUnavailableError,
} from './management-server/workerCredentialManagement';
const IDENTIFIER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/;
const ACTION_REF_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:/-]{0,254}$/;
const EXECUTOR_SUBJECT = Object.freeze({
type: 'system' as const,
id: 'cluster_worker_credential_executor',
});
const EXECUTOR_AUTHENTICATION_ID = 'cluster_worker_credential_executor_v1';
const EXECUTOR_PRINCIPAL_LIFETIME_MS = 15 * 60 * 1000;
const EXECUTION_LEASE_DURATION_MS = 10 * 60 * 1000;
const EXECUTION_OWNER = 'cluster_worker_credential_executor';
const EXECUTION_RESULT_CODE = 'worker_credential_published';
export interface RunClusterWorkerCredentialExecutionOptions {
readonly openDatabase: OpenPostgresDatabase;
readonly tokenRequestSession: WorkerCredentialKubernetesTokenRequestSession;
readonly workerCredentialPepper: string;
readonly actionRef: string;
readonly approvalRequestId: string;
readonly consumptionId: string;
readonly dispatchId: string;
readonly auditEventId: string;
readonly confirmAuthorization: () => void | Promise<void>;
readonly now?: () => number;
readonly randomBytes?: (size: number) => Buffer;
}
export interface ClusterWorkerCredentialExecutionRun {
readonly database: PostgresSchemaReadinessReport;
readonly approval: Readonly<ApprovedActionDispatchRecord>;
readonly execution: Readonly<ApprovedActionExecutionRecord>;
readonly result: Readonly<RecoverableWorkerCredentialIssueResult>;
readonly tokenRequest: Readonly<WorkerCredentialKubernetesTokenRequestEvidence> | null;
}
type Row = Record<string, unknown>;
function exactOptions(value: RunClusterWorkerCredentialExecutionOptions): void {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
throw new WorkerCredentialManagementRequestError(
'execution options must be an object',
);
}
const allowed = new Set([
'actionRef',
'approvalRequestId',
'auditEventId',
'confirmAuthorization',
'consumptionId',
'dispatchId',
'now',
'openDatabase',
'randomBytes',
'tokenRequestSession',
'workerCredentialPepper',
]);
if (Object.keys(value).some((key) => !allowed.has(key))) {
throw new WorkerCredentialManagementRequestError(
'execution options shape is invalid',
);
}
}
function identifier(value: unknown, label: string): string {
if (typeof value !== 'string' || !IDENTIFIER_PATTERN.test(value)) {
throw new WorkerCredentialManagementRequestError(`${label} is invalid`);
}
return value;
}
function actionRef(value: unknown): string {
if (typeof value !== 'string' || !ACTION_REF_PATTERN.test(value)) {
throw new WorkerCredentialManagementRequestError('actionRef is invalid');
}
return value;
}
function currentTime(now: () => number): number {
const value = now();
if (!Number.isSafeInteger(value) || value < 0) {
throw new WorkerCredentialManagementUnavailableError();
}
return value;
}
function same(left: unknown, right: unknown): boolean {
return JSON.stringify(left) === JSON.stringify(right);
}
function binding(
plan: Readonly<WorkerCredentialManagementPlan>,
): Readonly<ApprovedActionBinding> {
return Object.freeze({
permission: 'worker.manage',
actionType: `worker_credential.delivery.${plan.action}`,
actionRef: plan.actionRef,
actionDigest: plan.planDigest,
previewDigest: plan.previewDigest,
});
}
function audit(
eventId: string,
requestId: string,
projectId: string,
fence: Readonly<SecurityPolicyFence>,
occurredAtMs: number,
): Readonly<SecurityAuditRecord> {
return Object.freeze({
eventId,
requestId,
operationId: 'approval.consume',
projectId,
subject: EXECUTOR_SUBJECT,
authenticationId: EXECUTOR_AUTHENTICATION_ID,
outcome: 'allowed',
reasons: Object.freeze(['worker_credential_review']),
fence,
occurredAtMs,
});
}
function executorPrincipal(nowMs: number): Readonly<SecurityPrincipal> {
return Object.freeze({
subject: EXECUTOR_SUBJECT,
authenticationId: EXECUTOR_AUTHENTICATION_ID,
authenticatedAtMs: nowMs,
expiresAtMs: nowMs + EXECUTOR_PRINCIPAL_LIFETIME_MS,
assurance: 'service' as const,
});
}
function executionResultDigest(
plan: Readonly<WorkerCredentialManagementPlan>,
result: Readonly<RecoverableWorkerCredentialIssueResult>,
): string {
const delivery = result.delivery;
if (
!delivery ||
(delivery.state !== 'published' &&
delivery.state !== 'observed' &&
delivery.state !== 'previous_revoked') ||
delivery.deliveryId !== plan.target.deliveryId ||
delivery.workerId !== plan.target.workerId ||
delivery.credentialId !== plan.target.credentialId ||
delivery.previousCredentialId !== plan.target.previousCredentialId ||
delivery.deploymentTargetDigest !== plan.target.deploymentTargetDigest ||
delivery.deploymentGeneration !== plan.target.deploymentGeneration ||
typeof delivery.publicationDigest !== 'string'
) {
throw new WorkerCredentialManagementConflictError(
'credential delivery does not match approved execution',
);
}
return createHash('sha256')
.update('qinglong/worker-credential-execution-result@v1\0', 'utf8')
.update(
JSON.stringify({
deliveryId: delivery.deliveryId,
credentialId: delivery.credentialId,
workerId: delivery.workerId,
deploymentGeneration: delivery.deploymentGeneration,
publicationDigest: delivery.publicationDigest,
}),
'utf8',
)
.digest('hex');
}
async function assertPredecessor(
pool: PostgresPool,
plan: Readonly<WorkerCredentialManagementPlan>,
observedAtMs: number,
): Promise<void> {
if (plan.action === 'issue') return;
const result = await pool.query<Row>(
`SELECT state, worker_id AS "workerId", expires_at_ms AS "expiresAtMs"
FROM "ql3"."worker_credentials"
WHERE credential_id = $1
ORDER BY version DESC
LIMIT 1`,
[plan.target.previousCredentialId],
);
const row = result.rows[0];
const expiresAtMs =
typeof row?.expiresAtMs === 'number'
? row.expiresAtMs
: typeof row?.expiresAtMs === 'string' && /^\d+$/.test(row.expiresAtMs)
? Number(row.expiresAtMs)
: Number.NaN;
if (
result.rows.length !== 1 ||
row?.state !== 'active' ||
row.workerId !== plan.target.workerId ||
!Number.isSafeInteger(expiresAtMs) ||
expiresAtMs <= observedAtMs
) {
throw new WorkerCredentialManagementConflictError(
'rotation predecessor is not active for the target Worker',
);
}
}
async function closeDatabase(
database: PostgresDatabaseResource | undefined,
failure: unknown,
): Promise<void> {
if (!database) {
if (failure !== undefined) throw failure;
return;
}
try {
await database.close();
} catch (closeError) {
if (failure !== undefined) {
throw new AggregateError(
[failure, closeError],
'Worker credential execution failed and PostgreSQL did not close',
);
}
throw closeError;
}
if (failure !== undefined) throw failure;
}
export async function runClusterWorkerCredentialExecution(
options: RunClusterWorkerCredentialExecutionOptions,
): Promise<Readonly<ClusterWorkerCredentialExecutionRun>> {
exactOptions(options);
if (
typeof options.openDatabase !== 'function' ||
!options.tokenRequestSession ||
typeof options.tokenRequestSession.withDelivery !== 'function' ||
typeof options.confirmAuthorization !== 'function' ||
typeof options.workerCredentialPepper !== 'string' ||
(options.now !== undefined && typeof options.now !== 'function') ||
(options.randomBytes !== undefined &&
typeof options.randomBytes !== 'function')
) {
throw new WorkerCredentialManagementRequestError(
'execution dependency is invalid',
);
}
const requestedActionRef = actionRef(options.actionRef);
const approvalRequestId = identifier(
options.approvalRequestId,
'approvalRequestId',
);
const consumptionId = identifier(options.consumptionId, 'consumptionId');
const dispatchId = identifier(options.dispatchId, 'dispatchId');
const auditEventId = identifier(options.auditEventId, 'auditEventId');
const now = options.now ?? Date.now;
let database: PostgresDatabaseResource | undefined;
let failure: unknown;
let run: Readonly<ClusterWorkerCredentialExecutionRun> | undefined;
try {
await options.confirmAuthorization();
database = await options.openDatabase();
const evidence = await assertPostgresWorkerCredentialExecutorSchemaReady(
database.pool,
);
const plans = new PostgresWorkerCredentialManagementPlanReader(
database.pool,
);
const planValue = await plans.findByActionRef(requestedActionRef);
if (!planValue) {
throw new WorkerCredentialManagementConflictError('plan does not exist');
}
const plan = normalizeWorkerCredentialManagementPlan(planValue);
const approvals = new PostgresApprovalRequestRepository(database.pool);
const approvalValue = await approvals.findById(approvalRequestId);
if (!approvalValue) {
throw new WorkerCredentialManagementConflictError(
'approval does not exist',
);
}
let approval = normalizeApprovalRequestRecord(approvalValue);
const approvedAction = binding(plan);
if (
approval.projectId !== plan.authorityProjectId ||
approval.decisionMode !== 'separation_of_duty' ||
!same(approval.action, approvedAction) ||
!same(approval.requestedBy, plan.requestedBy)
) {
throw new WorkerCredentialManagementConflictError(
'approval does not match durable plan',
);
}
let dispatch: Readonly<ApprovedActionDispatchRecord> | null = null;
if (approval.version === 2 && approval.state === 'approved') {
const policy = new ProjectPolicyEngine(
new PostgresProjectPolicyRepository(database.pool),
);
const decision = await policy.decide({
subject: plan.requestedBy,
projectId: plan.authorityProjectId,
permission: 'worker.manage',
});
if (
(decision.effect !== 'allow' &&
decision.effect !== 'require_approval') ||
decision.fence === null
) {
throw new WorkerCredentialManagementConflictError(
'requester is no longer authorized',
);
}
const consumedAtMs = currentTime(now);
const consumed = await approvals.consume({
requestId: approvalRequestId,
expectedVersion: 2,
consumptionId,
dispatchId,
action: approvedAction,
requestedBy: plan.requestedBy,
consumedBy: EXECUTOR_SUBJECT,
consumedAtMs,
authorizationFence: decision.fence,
audit: audit(
auditEventId,
approvalRequestId,
plan.authorityProjectId,
decision.fence,
consumedAtMs,
),
});
approval = consumed.request;
dispatch = consumed.dispatch;
} else if (approval.version === 3 && approval.state === 'consumed') {
dispatch = await approvals.findDispatchById(dispatchId);
}
if (
approval.version !== 3 ||
approval.state !== 'consumed' ||
approval.consumptionId !== consumptionId ||
approval.dispatchId !== dispatchId ||
!dispatch ||
dispatch.id !== dispatchId ||
!same(dispatch.action, approvedAction) ||
!same(dispatch.requestedBy, plan.requestedBy) ||
!same(dispatch.approvedBy, approval.decidedBy) ||
!same(dispatch.consumedBy, EXECUTOR_SUBJECT)
) {
throw new WorkerCredentialManagementConflictError(
'approval consumption does not match execution',
);
}
const authority = new PostgresWorkerCredentialAdministrationRepository(
database.pool,
);
const executions = new PostgresApprovedActionExecutionRepository(
database.pool,
);
let executionSnapshot = await executions.findExecutionByDispatchId(
dispatchId,
);
if (!executionSnapshot || !same(executionSnapshot.dispatch, dispatch)) {
throw new WorkerCredentialManagementConflictError(
'durable execution baseline does not match dispatch',
);
}
if (executionSnapshot.execution.status === 'succeeded') {
const resolved = await authority.resolveDelivered(plan.target.deliveryId);
const result = Object.freeze({
status: 'existing' as const,
delivery: resolved?.delivery ?? null,
});
const resultDigest = executionResultDigest(plan, result);
if (
executionSnapshot.execution.resultMutationId !==
plan.target.deliveryId ||
executionSnapshot.execution.resultCode !== EXECUTION_RESULT_CODE ||
executionSnapshot.execution.resultDigest !== resultDigest
) {
throw new WorkerCredentialManagementConflictError(
'durable execution result does not match credential delivery',
);
}
run = Object.freeze({
database: evidence,
approval: dispatch,
execution: executionSnapshot.execution,
result,
tokenRequest: null,
});
await closeDatabase(database, undefined);
return run;
}
const executionNowMs = currentTime(now);
if (
executionNowMs > plan.expiresAtMs ||
executionNowMs >= dispatch.expiresAtMs ||
executionNowMs >= plan.target.credentialExpiresAtMs
) {
throw new WorkerCredentialManagementConflictError(
'approved execution window expired',
);
}
await assertPredecessor(database.pool, plan, executionNowMs);
await options.confirmAuthorization();
if (
executionSnapshot.execution.status === 'pending' ||
executionSnapshot.execution.status === 'retry_wait' ||
(executionSnapshot.execution.status === 'leased' &&
executionSnapshot.execution.leaseExpiresAtMs !== null &&
executionSnapshot.execution.leaseExpiresAtMs <= executionNowMs)
) {
const claimed = await executions.claimExecution({
dispatchId,
owner: EXECUTION_OWNER,
leaseToken: consumptionId,
nowMs: executionNowMs,
leaseDurationMs: EXECUTION_LEASE_DURATION_MS,
});
if (claimed.status !== 'claimed') {
throw new WorkerCredentialManagementConflictError(
'approved execution could not be claimed',
);
}
executionSnapshot = claimed.snapshot;
}
if (
(executionSnapshot.execution.status !== 'leased' &&
executionSnapshot.execution.status !== 'executing') ||
executionSnapshot.execution.leaseOwner !== EXECUTION_OWNER ||
executionSnapshot.execution.leaseToken !== consumptionId ||
executionSnapshot.execution.leaseExpiresAtMs === null ||
executionSnapshot.execution.leaseExpiresAtMs <= executionNowMs
) {
throw new WorkerCredentialManagementConflictError(
'approved execution lease does not match caller',
);
}
if (executionSnapshot.execution.status === 'leased') {
executionSnapshot = await executions.startExecution({
dispatchId,
approvalRequestId,
actionDigest: approvedAction.actionDigest,
owner: EXECUTION_OWNER,
leaseToken: consumptionId,
expectedVersion: executionSnapshot.execution.version,
startedAtMs: executionNowMs,
});
}
const sessionResult = await options.tokenRequestSession.withDelivery(
async ({ delivery, evidence: tokenRequest }) => {
const issuer = createRecoverableWorkerCredentialIssuer(
authority,
delivery,
options.workerCredentialPepper,
{
now,
...(options.randomBytes
? { randomBytes: options.randomBytes }
: {}),
},
);
const result = await issuer.issue({
mutationId: plan.target.deliveryId,
requestId: approvalRequestId,
expectedCurrentVersion: 0,
credentialId: plan.target.credentialId,
workerId: plan.target.workerId,
principal: executorPrincipal(currentTime(now)),
notBeforeAtMs: plan.target.credentialNotBeforeAtMs,
expiresAtMs: plan.target.credentialExpiresAtMs,
previousCredentialId: plan.target.previousCredentialId,
deploymentTargetDigest: plan.target.deploymentTargetDigest,
deploymentGeneration: plan.target.deploymentGeneration,
});
return Object.freeze({ result, tokenRequest });
},
);
const completed = await executions.completeExecution({
dispatchId,
owner: EXECUTION_OWNER,
leaseToken: consumptionId,
expectedVersion: executionSnapshot.execution.version,
resultMutationId: plan.target.deliveryId,
outcome: 'succeeded',
resultCode: EXECUTION_RESULT_CODE,
resultDigest: executionResultDigest(plan, sessionResult.result),
completedAtMs: currentTime(now),
});
run = Object.freeze({
database: evidence,
approval: dispatch,
execution: completed.execution,
result: sessionResult.result,
tokenRequest: sessionResult.tokenRequest,
});
} catch (error) {
failure = error;
}
await closeDatabase(database, failure);
if (!run) {
throw new WorkerCredentialManagementUnavailableError();
}
return run;
}
@@ -0,0 +1,187 @@
const assert = require('node:assert/strict');
const { test } = require('node:test');
const {
ApiCredentialAdministrationMutationConflictError,
} = require('@qinglong/runtime-core/api-credential-administration');
const {
apiCredentialSecretDigest,
} = require('@qinglong/runtime-core/api-credential-token');
const {
ClusterAdministrationAuthenticationError,
createClusterAdministrationService,
} = require('@qinglong/cluster-admin/administration');
const NOW = 1_000;
const PEPPER = 'A'.repeat(43);
const SUBJECT = { type: 'api_app', id: 'app_primary' };
const PRINCIPAL = {
subject: { type: 'user', id: 'usr_admin' },
authenticationId: 'session:admin:1',
authenticatedAtMs: 900,
expiresAtMs: 2_000,
assurance: 'multi_factor',
};
function request(overrides = {}) {
return {
mutationId: '123e4567-e89b-42d3-a456-426614174301',
requestId: 'request-credential-issue-1',
expectedCurrentVersion: 0,
credentialId: 'credential_primary',
subject: SUBJECT,
principal: PRINCIPAL,
notBeforeAtMs: NOW,
expiresAtMs: 2_000,
...overrides,
};
}
function repositories() {
let credentialMutation = null;
const credentialCommands = [];
const identityCommands = [];
const identity = {
subject: SUBJECT,
status: 'active',
version: 1,
createdAtMs: 100,
updatedAtMs: 100,
};
return {
identityCommands,
credentialCommands,
identities: {
async resolve() {
return identity;
},
async resolveMutation() {
return null;
},
async append(command) {
identityCommands.push(command);
return {
status: 'inserted',
identity: {
subject: command.mutation.subject,
status: command.mutation.status,
version: command.mutation.subjectVersion,
createdAtMs: command.mutation.createdAtMs,
updatedAtMs: command.mutation.createdAtMs,
},
mutation: command.mutation,
};
},
},
credentials: {
async resolveMutation() {
return credentialMutation;
},
async append(command) {
credentialCommands.push(command);
credentialMutation = {
credential: command.credential,
mutation: command.mutation,
audit: command.audit,
};
return {
status: 'inserted',
credential: command.credential,
mutation: command.mutation,
};
},
conflictWith(stored) {
credentialMutation = stored;
},
},
};
}
test('issues one token, stores only its digest and clears mutable secret bytes', async () => {
const repos = repositories();
const generated = Buffer.alloc(32, 7);
const expectedSecret = generated.toString('base64url');
const service = createClusterAdministrationService(
repos.identities,
repos.credentials,
PEPPER,
{ now: () => NOW, randomBytes: () => generated },
);
const result = await service.issueCredential(request());
assert.equal(result.status, 'inserted');
assert.equal(result.token, `ql3c_credential_primary_${expectedSecret}`);
assert.equal(repos.credentialCommands.length, 1);
assert.equal(
repos.credentialCommands[0].credential.secretDigest,
apiCredentialSecretDigest(PEPPER, 'credential_primary', expectedSecret),
);
assert.equal(repos.credentialCommands[0].audit.eventId, request().mutationId);
assert.equal(
generated.every((value) => value === 0),
true,
);
});
test('semantic mutation replay returns no token and does not generate a new secret', async () => {
const repos = repositories();
let randomCalls = 0;
const service = createClusterAdministrationService(
repos.identities,
repos.credentials,
PEPPER,
{
now: () => NOW,
randomBytes() {
randomCalls += 1;
return Buffer.alloc(32, randomCalls);
},
},
);
assert.ok((await service.issueCredential(request())).token);
const replay = await service.issueCredential(request());
assert.equal(replay.status, 'existing');
assert.equal(replay.token, null);
assert.equal(randomCalls, 1);
await assert.rejects(
service.issueCredential(request({ requestId: 'different-request' })),
ApiCredentialAdministrationMutationConflictError,
);
});
test('rejects weak principals and widened requests before repository mutation', async () => {
const repos = repositories();
const service = createClusterAdministrationService(
repos.identities,
repos.credentials,
PEPPER,
{ now: () => NOW, randomBytes: () => Buffer.alloc(32, 1) },
);
await assert.rejects(
service.issueCredential(
request({ principal: { ...PRINCIPAL, assurance: 'single_factor' } }),
),
ClusterAdministrationAuthenticationError,
);
await assert.rejects(
service.issueCredential(request({ debug: true })),
/request shape is invalid/,
);
assert.equal(repos.credentialCommands.length, 0);
});
test('maps malformed entropy output to a stable configuration error', async () => {
const repos = repositories();
const service = createClusterAdministrationService(
repos.identities,
repos.credentials,
PEPPER,
{ now: () => NOW, randomBytes: () => 'not-a-buffer' },
);
await assert.rejects(
service.issueCredential(request()),
/randomBytes returned invalid secret material/,
);
assert.equal(repos.credentialCommands.length, 0);
});
@@ -0,0 +1,34 @@
const assert = require('node:assert/strict');
const { test } = require('node:test');
const {
ClusterApprovalDecisionManagementConfigurationError,
createClusterApprovalDecisionManagementService,
} = require('@qinglong/cluster-admin/approval-decision-management');
test('composes the shared Approval decision authority over a caller-owned pool', () => {
const pool = {
async query() {
throw new Error('not used');
},
async connect() {
throw new Error('not used');
},
};
const service = createClusterApprovalDecisionManagementService({ pool });
assert.equal(typeof service.decide, 'function');
assert.equal(Object.isFrozen(service), true);
});
test('rejects missing pool authority and widened options', () => {
for (const options of [
{},
{ pool: { query() {} } },
{ pool: { query() {}, connect() {} }, transport: {} },
]) {
assert.throws(
() => createClusterApprovalDecisionManagementService(options),
ClusterApprovalDecisionManagementConfigurationError,
);
}
});
@@ -0,0 +1,223 @@
const assert = require('node:assert/strict');
const {
chmodSync,
mkdtempSync,
readFileSync,
realpathSync,
rmSync,
writeFileSync,
} = require('node:fs');
const { tmpdir } = require('node:os');
const { join, resolve } = require('node:path');
const { afterEach, test } = require('node:test');
const {
ClusterPluginPackageManagementClientRequestError,
} = require('@qinglong/cluster-admin/plugin-package-management-client');
const {
executeClusterApprovalManagementClient,
validateClusterApprovalManagementClientResult,
} = require('@qinglong/cluster-admin/approval-management-client');
const FIXTURES = resolve(
__dirname,
'../../ql3-cluster-control/test/fixtures/mtls',
);
const temporaryDirectories = [];
const ACTION = Object.freeze({
permission: 'run.start',
actionType: 'tool.invoke',
actionRef: 'tool:task-1',
actionDigest: 'a'.repeat(64),
previewDigest: 'b'.repeat(64),
});
const BASE_REQUEST = Object.freeze({
projectId: 'default',
approvalRequestId: 'approval-1',
requestId: 'approval-command-1',
auditEventId: '60000000-0000-4000-8000-000000000001',
failureAuditEventId: '60000000-0000-4000-8000-000000000002',
});
function privateWrite(filePath, value) {
writeFileSync(filePath, value, { mode: 0o600 });
chmodSync(filePath, 0o600);
}
function clientFiles() {
const directory = realpathSync(
mkdtempSync(join(tmpdir(), 'ql3-approval-client-')),
);
temporaryDirectories.push(directory);
const paths = {
configFile: join(directory, 'client.json'),
commandFile: join(directory, 'command.json'),
assertionFile: join(directory, 'assertion.jwt'),
};
const caFile = join(directory, 'ca.crt');
const clientCertificateFile = join(directory, 'client.crt');
const clientPrivateKeyFile = join(directory, 'client.key');
privateWrite(caFile, readFileSync(join(FIXTURES, 'ca-cert.pem')));
privateWrite(
clientCertificateFile,
readFileSync(join(FIXTURES, 'client-cert.pem')),
);
privateWrite(
clientPrivateKeyFile,
readFileSync(join(FIXTURES, 'client-key.pem')),
);
privateWrite(
paths.configFile,
`${JSON.stringify({
schemaVersion: 1,
endpoint:
'https://approval.example.test:8447/api/v3/approvals/management',
servername: 'approval.example.test',
caFile,
clientCertificateFile,
clientPrivateKeyFile,
requestTimeoutMs: 1_000,
})}\n`,
);
privateWrite(
paths.commandFile,
`${JSON.stringify({
schemaVersion: 1,
operation: 'approval.inspect',
request: BASE_REQUEST,
})}\n`,
);
privateWrite(
paths.assertionFile,
'eyJhbGciOiJFZERTQSJ9.eyJzdWIiOiJ1In0.c2lnbmF0dXJl',
);
return paths;
}
afterEach(() => {
for (const directory of temporaryDirectories.splice(0)) {
rmSync(directory, { recursive: true, force: true });
}
});
test('validates exact inspect results and binds them to the requested Approval', () => {
const command = {
schemaVersion: 1,
operation: 'approval.inspect',
request: BASE_REQUEST,
};
const found = {
schemaVersion: 1,
operation: 'approval.inspect',
status: 'found',
approval: {
projectId: 'default',
approvalRequestId: 'approval-1',
version: 1,
state: 'pending',
risk: 'high',
decisionMode: 'human_confirmation',
expectedAction: ACTION,
requestedBy: { type: 'agent', id: 'agent-1' },
requestedAtMs: 1_000,
expiresAtMs: 10_000,
preview: {
title: 'Run task',
summary: 'Runs one reviewed task.',
fields: [{ kind: 'identifier', label: 'Task', value: 'task-1' }],
warnings: ['external_effect'],
},
},
};
assert.deepEqual(
validateClusterApprovalManagementClientResult(found, command),
found,
);
assert.deepEqual(
validateClusterApprovalManagementClientResult(
{ ...found, status: 'absent', approval: null },
command,
),
{ ...found, status: 'absent', approval: null },
);
assert.throws(
() =>
validateClusterApprovalManagementClientResult(
{
...found,
approval: { ...found.approval, projectId: 'other' },
},
command,
),
ClusterPluginPackageManagementClientRequestError,
);
});
test('validates the durable decision tuple and rejects server-side drift', () => {
const command = {
schemaVersion: 1,
operation: 'approval.decide',
request: {
...BASE_REQUEST,
expectedVersion: 1,
expectedAction: ACTION,
decisionId: 'decision-1',
decision: 'approved',
reasonCode: 'reviewed',
},
};
const result = {
schemaVersion: 1,
operation: 'approval.decide',
status: 'decided',
approval: {
projectId: 'default',
approvalRequestId: 'approval-1',
version: 2,
state: 'approved',
expectedAction: ACTION,
decisionId: 'decision-1',
decision: 'approved',
reasonCode: 'reviewed',
decidedBy: { type: 'user', id: 'owner-1' },
decidedAtMs: 2_000,
},
};
assert.deepEqual(
validateClusterApprovalManagementClientResult(result, command),
result,
);
for (const approval of [
{ ...result.approval, version: 1 },
{ ...result.approval, state: 'rejected' },
{ ...result.approval, decisionId: 'decision-2' },
]) {
assert.throws(
() =>
validateClusterApprovalManagementClientResult(
{ ...result, approval },
command,
),
ClusterPluginPackageManagementClientRequestError,
);
}
});
test('accepts only the exact Approval route before opening one mTLS connection', async () => {
let connects = 0;
await assert.rejects(
executeClusterApprovalManagementClient(clientFiles(), {
async connect(target) {
connects += 1;
assert.deepEqual(target, {
hostname: 'approval.example.test',
port: 8447,
});
throw new Error('expected-connect-stop');
},
}),
{ code: 'QL3_PLUGIN_PACKAGE_MANAGEMENT_CLIENT_REQUEST_FAILED' },
);
assert.equal(connects, 1);
});
@@ -0,0 +1,134 @@
const assert = require('node:assert/strict');
const { readFileSync } = require('node:fs');
const { request: httpsRequest } = require('node:https');
const { resolve } = require('node:path');
const { test } = require('node:test');
const {
startClusterApprovalManagementHttp,
} = require('@qinglong/cluster-admin/approval-management-http');
const {
ClusterApprovalManagementTransportConflictError,
} = require('@qinglong/cluster-admin/approval-management-transport');
const SERVER_KEY = resolve(
__dirname,
'../../ql3-cluster-control/test/fixtures/mtls/server-key.pem',
);
const SERVER_CERT = resolve(
__dirname,
'../../ql3-cluster-control/test/fixtures/mtls/server-cert.pem',
);
const PATH = '/api/v3/approvals/management';
function post(port, path = PATH) {
const body = Buffer.from(
JSON.stringify({
schemaVersion: 1,
operation: 'approval.inspect',
request: {
projectId: 'default',
approvalRequestId: 'approval-1',
requestId: 'approval-command-1',
auditEventId: '50000000-0000-4000-8000-000000000001',
failureAuditEventId: '50000000-0000-4000-8000-000000000002',
},
}),
);
return new Promise((resolvePromise, reject) => {
const outgoing = httpsRequest(
{
host: '127.0.0.1',
port,
path,
method: 'POST',
rejectUnauthorized: false,
agent: false,
headers: {
authorization: 'Bearer assertion',
'content-type': 'application/json',
'content-length': String(body.length),
},
},
(incoming) => {
const chunks = [];
incoming.on('data', (chunk) => chunks.push(chunk));
incoming.once('end', () => {
const bytes = Buffer.concat(chunks);
resolvePromise({
statusCode: incoming.statusCode,
body: bytes.length ? JSON.parse(bytes.toString('utf8')) : null,
});
});
},
);
outgoing.once('error', reject);
outgoing.end(body);
});
}
test('serves only the human Approval path and maps conflicts to low-sensitive facts', async () => {
const privateKey = Buffer.from(readFileSync(SERVER_KEY));
const application = await startClusterApprovalManagementHttp({
host: '127.0.0.1',
port: 0,
tls: {
privateKey,
certificate: Buffer.from(readFileSync(SERVER_CERT)),
},
identities: {
async reload() {
return {
schemaVersion: 1,
generation: 1,
digest: 'digest',
issuer: 'https://identity.example.test/',
audience: 'qinglong3-approval-management',
activeKeyIds: ['key-1'],
revokedKeyIds: [],
};
},
bind() {
return {
async authenticate() {
return {
subject: { type: 'user', id: 'owner-1' },
authenticationId: 'session-owner-1',
authenticatedAtMs: 900,
expiresAtMs: 10_000,
assurance: 'hardware',
};
},
};
},
},
transport: {
async execute(_command, authentication) {
await authentication.authenticate();
throw new ClusterApprovalManagementTransportConflictError();
},
},
now: () => 1_000,
});
try {
assert.equal(privateKey.every((value) => value === 0), true);
const conflict = await post(application.address.port);
assert.deepEqual(conflict, {
statusCode: 409,
body: {
schemaVersion: 1,
requestId: conflict.body.requestId,
error: { code: 'conflict' },
},
});
assert.match(conflict.body.requestId, /^[0-9a-f-]{36}$/);
const absent = await post(
application.address.port,
'/api/v3/automations/management',
);
assert.equal(absent.statusCode, 404);
assert.equal(absent.body.error.code, 'not_found');
} finally {
await application.close();
}
});
@@ -0,0 +1,209 @@
const assert = require('node:assert/strict');
const { readFile, writeFile, mkdtemp, rm } = require('node:fs/promises');
const { tmpdir } = require('node:os');
const { join, resolve } = require('node:path');
const { test } = require('node:test');
const {
ClusterApprovalManagementProcessConfigError,
loadClusterApprovalManagementProcessConfig,
startClusterApprovalManagementProcess,
} = require('@qinglong/cluster-admin/approval-management-process');
const FIXTURES = resolve(
__dirname,
'../../ql3-cluster-control/test/fixtures/mtls',
);
const NOW_MS = Date.UTC(2030, 0, 1);
function enabledEnvironment(paths, overrides = {}) {
return {
QL3_APPROVAL_MANAGEMENT_ENABLED: 'true',
QL3_PROFILE: 'cluster-admin',
QL3_APPROVAL_MANAGEMENT_HOST: '127.0.0.1',
QL3_APPROVAL_MANAGEMENT_PORT: '8447',
QL3_APPROVAL_MANAGEMENT_TLS_CERT_FILE: paths.certificateFile,
QL3_APPROVAL_MANAGEMENT_TLS_KEY_FILE: paths.privateKeyFile,
QL3_APPROVAL_MANAGEMENT_CLIENT_CA_FILE:
paths.clientCertificateAuthorityFile,
QL3_APPROVAL_MANAGEMENT_CLIENT_CRL_FILE:
paths.clientCertificateRevocationListFile,
QL3_APPROVAL_MANAGEMENT_IDENTITY_KEYSET_FILE: paths.identityKeysetFile,
QL3_POSTGRES_APPROVAL_MANAGER_URL:
'postgresql://ql3_approval_manager:secret@postgres.example.test/ql3',
QL3_POSTGRES_APPROVAL_MANAGER_TLS_MODE: 'disable',
QL3_POSTGRES_APPROVAL_MANAGER_ALLOW_INSECURE: 'true',
...overrides,
};
}
async function tlsFixture(run) {
const directory = await mkdtemp(join(tmpdir(), 'ql3-approval-manager-'));
const paths = {
certificateFile: join(directory, 'tls.crt'),
privateKeyFile: join(directory, 'tls.key'),
clientCertificateAuthorityFile: join(directory, 'client-ca.crt'),
clientCertificateRevocationListFile: join(directory, 'client.crl'),
identityKeysetFile: join(directory, 'keyset.json'),
};
try {
await writeFile(
paths.certificateFile,
await readFile(join(FIXTURES, 'server-cert.pem')),
{ mode: 0o644 },
);
await writeFile(
paths.privateKeyFile,
await readFile(join(FIXTURES, 'server-key.pem')),
{ mode: 0o640 },
);
await writeFile(
paths.clientCertificateAuthorityFile,
await readFile(join(FIXTURES, 'ca-cert.pem')),
{ mode: 0o644 },
);
await writeFile(
paths.clientCertificateRevocationListFile,
await readFile(join(FIXTURES, 'empty-crl.pem')),
{ mode: 0o644 },
);
await writeFile(paths.identityKeysetFile, '{}\n', { mode: 0o644 });
return await run(paths);
} finally {
await rm(directory, { recursive: true, force: true });
}
}
test('disabled Approval manager acquires no PostgreSQL or file authority', async () => {
let opened = 0;
const runtime = await startClusterApprovalManagementProcess({
environment: { QL3_APPROVAL_MANAGEMENT_ENABLED: 'false' },
async openDatabase() {
opened += 1;
throw new Error('must not open');
},
});
assert.equal(runtime.status, 'disabled');
assert.equal(opened, 0);
await runtime.close();
});
test('loads bounded low-footprint Approval-only HTTPS and PostgreSQL configuration', async () => {
await tlsFixture(async (paths) => {
const config = loadClusterApprovalManagementProcessConfig(
enabledEnvironment(paths),
);
assert.equal(config.enabled, true);
assert.equal(config.port, 8447);
assert.equal(config.database.pool.maxConnections, 2);
assert.equal(
config.database.pool.applicationName,
'qinglong3-approval-manager',
);
assert.equal(config.http.maxConnections, 32);
assert.equal(config.http.maxConcurrentRequests, 16);
assert.match(
config.database.connection.connectionString,
/^postgresql:\/\/ql3_approval_manager:/,
);
assert.equal(config.database.connection.tls.mode, 'disable');
});
assert.throws(
() =>
loadClusterApprovalManagementProcessConfig({
QL3_APPROVAL_MANAGEMENT_ENABLED: 'true',
QL3_PROFILE: 'cluster-admin',
QL3_POSTGRES_APPROVAL_MANAGER_TLS_MODE: 'disable',
}),
ClusterApprovalManagementProcessConfigError,
);
});
test('starts after dedicated readiness and identity validation then closes in order', async () => {
await tlsFixture(async (paths) => {
const order = [];
let privateKey;
let transport;
let httpClosed = 0;
let databaseClosed = 0;
const pool = {
async query() {
throw new Error('repositories must remain lazy during composition');
},
async connect() {
throw new Error('repositories must remain lazy during composition');
},
};
const runtime = await startClusterApprovalManagementProcess({
environment: enabledEnvironment(paths),
now: () => NOW_MS,
async openDatabase() {
order.push('open');
return {
pool,
async close() {
order.push('database-close');
databaseClosed += 1;
},
};
},
async assertReady(candidate) {
order.push('ready');
assert.equal(candidate, pool);
return {
ready: true,
writablePrimary: true,
serverVersionNum: 180004,
serverMajor: 18,
currentUser: 'ql3_approval_manager',
contractName: 'control-core',
contractVersion: 53,
migrationIds: ['pg-0054-approval-management-boundary'],
};
},
identities: {
async reload() {
order.push('identity');
return {
schemaVersion: 1,
generation: 3,
digest: 'identity-digest',
issuer: 'https://identity.example.test/',
audience: 'qinglong3-approval-management',
activeKeyIds: ['key-3'],
revokedKeyIds: ['key-2'],
};
},
bind() {
throw new Error('HTTP stub does not authenticate');
},
},
async startHttp(options) {
order.push('http');
privateKey = options.tls.privateKey;
transport = options.transport;
assert.ok(options.tls.clientCertificateAuthority);
assert.ok(options.tls.clientCertificateRevocationList);
return {
status: 'active',
address: { host: options.host, port: options.port },
availabilityStatus: () => 'ready',
withdraw() {},
async close() {
order.push('http-close');
httpClosed += 1;
},
};
},
});
assert.equal(runtime.status, 'active');
assert.deepEqual(order, ['open', 'ready', 'identity', 'http']);
assert.equal(typeof transport.execute, 'function');
assert.equal(privateKey.every((byte) => byte === 0), true);
assert.equal(runtime.database.contractVersion, 53);
await Promise.all([runtime.close(), runtime.close()]);
assert.equal(httpClosed, 1);
assert.equal(databaseClosed, 1);
assert.deepEqual(order.slice(-2), ['http-close', 'database-close']);
});
});
@@ -0,0 +1,231 @@
const assert = require('node:assert/strict');
const { test } = require('node:test');
const {
createApprovalRequest,
decideApprovalRequest,
} = require('@qinglong/runtime-core/approved-action');
const {
ClusterApprovalManagementTransportAuthenticationError,
ClusterApprovalManagementTransportRequestError,
createClusterApprovalManagementTransport,
} = require('@qinglong/cluster-admin/approval-management-transport');
const ACTION = Object.freeze({
permission: 'run.start',
actionType: 'tool.invoke',
actionRef: 'tool:task-1',
actionDigest: 'a'.repeat(64),
previewDigest: 'b'.repeat(64),
});
const PRINCIPAL = Object.freeze({
subject: Object.freeze({ type: 'user', id: 'owner-1' }),
authenticationId: 'oidc:session-1',
authenticatedAtMs: 1_000,
expiresAtMs: 20_000,
assurance: 'hardware',
});
const BASE_REQUEST = Object.freeze({
projectId: 'default',
approvalRequestId: 'approval-1',
requestId: 'approval-command-1',
auditEventId: '40000000-0000-4000-8000-000000000001',
failureAuditEventId: '40000000-0000-4000-8000-000000000002',
});
function pending() {
return createApprovalRequest({
id: 'approval-1',
projectId: 'default',
action: ACTION,
risk: 'high',
decisionMode: 'human_confirmation',
requestedBy: { type: 'agent', id: 'agent-1' },
requestedAtMs: 900,
expiresAtMs: 10_000,
requestFence: { projectVersion: 1, bindingVersion: 2 },
});
}
function inspectCommand() {
return {
schemaVersion: 1,
operation: 'approval.inspect',
request: BASE_REQUEST,
};
}
function decideCommand() {
return {
schemaVersion: 1,
operation: 'approval.decide',
request: {
...BASE_REQUEST,
expectedVersion: 1,
expectedAction: ACTION,
decisionId: 'decision-1',
decision: 'approved',
reasonCode: 'reviewed',
},
};
}
test('inspects and decides through fresh strong authentication without leaking principal facts', async () => {
const calls = [];
const failures = [];
const transport = createClusterApprovalManagementTransport({
service: {
async inspect(request, confirmAuthorization) {
calls.push(['inspect', request]);
await confirmAuthorization();
return {
request: pending(),
preview: {
title: 'Run task',
summary: 'Runs one reviewed task.',
fields: [{ kind: 'identifier', label: 'Task', value: 'task-1' }],
warnings: ['external_effect'],
},
};
},
async decide(request, confirmAuthorization) {
calls.push(['decide', request]);
await confirmAuthorization();
return {
status: 'decided',
request: decideApprovalRequest(pending(), {
expectedVersion: 1,
decisionId: request.decisionId,
decision: request.decision,
reasonCode: request.reasonCode,
principal: request.principal,
decidedAtMs: 2_000,
authorizationFence: { projectVersion: 1, bindingVersion: 2 },
}),
};
},
async recordFailure(record) {
failures.push(record);
},
},
now: () => 2_000,
});
let authenticationCalls = 0;
const authentication = {
async authenticate() {
authenticationCalls += 1;
return PRINCIPAL;
},
};
const inspected = await transport.execute(inspectCommand(), authentication);
assert.equal(authenticationCalls, 2);
assert.equal(inspected.status, 'found');
assert.equal(inspected.approval.preview.title, 'Run task');
assert.equal(inspected.approval.expectedAction.actionDigest, 'a'.repeat(64));
const decided = await transport.execute(decideCommand(), authentication);
assert.equal(authenticationCalls, 4);
assert.equal(decided.status, 'decided');
assert.equal(decided.approval.state, 'approved');
assert.equal(decided.approval.version, 2);
assert.deepEqual(calls.map(([operation]) => operation), ['inspect', 'decide']);
assert.equal(failures.length, 0);
assert.doesNotMatch(
JSON.stringify([inspected, decided]),
/authenticationId|authenticatedAtMs|assurance/,
);
});
test('records unauthenticated and reauthentication failures with schema-valid identities', async () => {
const failures = [];
const service = {
async inspect(_request, confirmAuthorization) {
await confirmAuthorization();
return null;
},
async decide() {
throw new Error('not used');
},
async recordFailure(record) {
failures.push(record);
},
};
const transport = createClusterApprovalManagementTransport({
service,
now: () => 2_000,
});
await assert.rejects(
transport.execute(inspectCommand(), {
async authenticate() {
return { ...PRINCIPAL, assurance: 'single_factor' };
},
}),
ClusterApprovalManagementTransportAuthenticationError,
);
assert.equal(failures[0].outcome, 'authentication_rejected');
assert.equal(failures[0].subject, null);
assert.equal(failures[0].authenticationId, null);
let calls = 0;
await assert.rejects(
transport.execute(inspectCommand(), {
async authenticate() {
calls += 1;
return calls === 1
? PRINCIPAL
: { ...PRINCIPAL, authenticationId: 'oidc:session-2' };
},
}),
ClusterApprovalManagementTransportAuthenticationError,
);
assert.equal(failures[1].outcome, 'denied');
assert.deepEqual(failures[1].subject, PRINCIPAL.subject);
assert.equal(failures[1].authenticationId, PRINCIPAL.authenticationId);
});
test('rejects widened or ambiguously audited commands before authentication', async () => {
let authenticationCalls = 0;
const transport = createClusterApprovalManagementTransport({
service: {
async inspect() {},
async decide() {},
async recordFailure() {},
},
});
await assert.rejects(
transport.execute(
{
...inspectCommand(),
extra: true,
},
{
async authenticate() {
authenticationCalls += 1;
return PRINCIPAL;
},
},
),
ClusterApprovalManagementTransportRequestError,
);
await assert.rejects(
transport.execute(
{
...inspectCommand(),
request: {
...BASE_REQUEST,
failureAuditEventId: BASE_REQUEST.auditEventId,
},
},
{
async authenticate() {
authenticationCalls += 1;
return PRINCIPAL;
},
},
),
ClusterApprovalManagementTransportRequestError,
);
assert.equal(authenticationCalls, 0);
});
@@ -0,0 +1,286 @@
const assert = require('node:assert/strict');
const { test } = require('node:test');
const {
createTaskDefinitionRecord,
} = require('@qinglong/runtime-core/task-definition');
const { createTriggerRecord } = require('@qinglong/runtime-core/trigger');
const {
ClusterAutomationManagementAuthorizationError,
ClusterAutomationManagementConflictError,
createClusterAutomationManagementService,
} = require('@qinglong/cluster-admin/automation-management');
const {
TaskDefinitionAdministrationMutationConflictError,
} = require('@qinglong/runtime-core/task-definition-administration');
const principal = Object.freeze({
subject: Object.freeze({ type: 'user', id: 'operator-a' }),
authenticationId: 'session-operator-a',
authenticatedAtMs: 900,
expiresAtMs: 10_000,
assurance: 'multi_factor',
});
function taskCommand(expectedRevision = null) {
return {
projectId: 'project-a',
taskId: 'task-a',
expectedRevision,
mutationId:
expectedRevision === null
? '123e4567-e89b-42d3-a456-426614174000'
: '123e4567-e89b-42d3-a456-426614174001',
name: 'Sensitive command name',
kind: 'script',
spec: {
schema: 'qinglong/script@v1',
config: { source: 'sensitive source text' },
},
labels: { environment: 'test' },
enabled: true,
occurredAtMs: 1_000,
};
}
function triggerCommand(expectedRevision = null) {
return {
projectId: 'project-a',
triggerId: 'trigger-a',
expectedRevision,
mutationId:
expectedRevision === null
? '123e4567-e89b-42d3-a456-426614174002'
: '123e4567-e89b-42d3-a456-426614174003',
taskId: 'task-a',
taskRevision: 1,
taskContentDigest: 'a'.repeat(64),
spec: {
schema: 'qinglong/cron@v1',
config: {
expression: '*/5 * * * *',
timezone: 'UTC',
misfirePolicy: 'fire_once',
},
},
enabled: true,
occurredAtMs: 1_000,
};
}
function fixture(effect = 'allow') {
const calls = [];
const service = createClusterAutomationManagementService({
policy: {
async authorize(candidate, projectId, permission) {
calls.push(['authorize', candidate, projectId, permission]);
return {
effect,
reasons: [effect === 'allow' ? 'role_grant' : 'permission_missing'],
fence: { projectVersion: 3, bindingVersion: 5 },
};
},
},
taskDefinitions: {
async appendAuthorizedTaskDefinitionRevision(mutation) {
calls.push(['task', mutation]);
return {
status:
mutation.command.expectedRevision === null ? 'created' : 'updated',
definition: createTaskDefinitionRecord(mutation.command, 900),
};
},
async findAuthorizedCurrentTaskDefinition(read) {
calls.push(['task-inspect', read]);
return createTaskDefinitionRecord(taskCommand(), 900);
},
async listAuthorizedTaskDefinitions(read) {
calls.push(['task-list', read]);
return {
definitions: [createTaskDefinitionRecord(taskCommand(), 900)],
truncated: true,
next: { taskId: 'task-a' },
};
},
},
triggers: {
async appendAuthorizedTriggerRevision(mutation) {
calls.push(['trigger', mutation]);
return {
status:
mutation.command.expectedRevision === null ? 'created' : 'updated',
trigger: createTriggerRecord(mutation.command, 900),
};
},
async findAuthorizedCurrentTrigger(read) {
calls.push(['trigger-inspect', read]);
return createTriggerRecord(triggerCommand(), 900);
},
async listAuthorizedTriggers(read) {
calls.push(['trigger-list', read]);
return {
triggers: [createTriggerRecord(triggerCommand(), 900)],
truncated: false,
};
},
},
now: () => 1_100,
});
return { service, calls };
}
test('authorizes Task and Trigger create/update and binds allowed audit to the repository fence', async () => {
const { service, calls } = fixture();
await service.publishTask({
requestId: 'request-task-create',
command: taskCommand(),
principal,
});
await service.publishTask({
requestId: 'request-task-update',
command: taskCommand(1),
principal,
});
await service.publishTrigger({
requestId: 'request-trigger-create',
command: triggerCommand(),
principal,
});
await service.publishTrigger({
requestId: 'request-trigger-update',
command: triggerCommand(1),
principal,
});
assert.deepEqual(
calls.filter(([kind]) => kind === 'authorize').map((call) => call[3]),
['task.create', 'task.update', 'trigger.create', 'trigger.update'],
);
for (const [kind, mutation] of calls.filter(([kind]) =>
['task', 'trigger'].includes(kind),
)) {
assert.deepEqual(mutation.actor, principal.subject);
assert.deepEqual(mutation.fence, {
projectVersion: 3,
bindingVersion: 5,
});
assert.equal(mutation.audit.outcome, 'allowed');
assert.equal(mutation.audit.authenticationId, principal.authenticationId);
assert.equal(mutation.audit.eventId, mutation.command.mutationId);
assert.equal(
mutation.audit.operationId,
`${kind}.${mutation.command.expectedRevision === null ? 'create' : 'update'}`,
);
}
});
test('fails closed before mutation on weak identity or denied policy', async () => {
const denied = fixture('deny');
await assert.rejects(
denied.service.publishTask({
requestId: 'request-denied',
command: taskCommand(),
principal,
}),
ClusterAutomationManagementAuthorizationError,
);
assert.equal(denied.calls.some(([kind]) => kind === 'task'), false);
const allowed = fixture();
await assert.rejects(
allowed.service.publishTrigger({
requestId: 'request-weak',
command: triggerCommand(),
principal: { ...principal, assurance: 'single_factor' },
}),
ClusterAutomationManagementAuthorizationError,
);
assert.equal(allowed.calls.length, 0);
});
test('authorizes bounded Task and Trigger reads and binds each durable audit fence', async () => {
const { service, calls } = fixture();
const task = await service.inspectTask({
requestId: 'request-task-inspect',
auditEventId: '123e4567-e89b-42d3-a456-426614174010',
projectId: 'project-a',
taskId: 'task-a',
principal,
});
const tasks = await service.listTasks({
requestId: 'request-task-list',
auditEventId: '123e4567-e89b-42d3-a456-426614174011',
projectId: 'project-a',
limit: 1,
principal,
});
const trigger = await service.inspectTrigger({
requestId: 'request-trigger-inspect',
auditEventId: '123e4567-e89b-42d3-a456-426614174012',
projectId: 'project-a',
triggerId: 'trigger-a',
principal,
});
const triggers = await service.listTriggers({
requestId: 'request-trigger-list',
auditEventId: '123e4567-e89b-42d3-a456-426614174013',
projectId: 'project-a',
limit: 1,
after: { triggerId: 'trigger-0' },
principal,
});
assert.equal(task.taskId, 'task-a');
assert.equal(tasks.truncated, true);
assert.equal(trigger.triggerId, 'trigger-a');
assert.equal(triggers.truncated, false);
assert.deepEqual(
calls.filter(([kind]) => kind === 'authorize').map((call) => call[3]),
['task.read', 'task.read', 'trigger.read', 'trigger.read'],
);
for (const [kind, read] of calls.filter(([kind]) => kind.includes('-'))) {
assert.equal(read.audit.operationId, kind.startsWith('task') ? 'task.read' : 'trigger.read');
assert.equal(read.audit.outcome, 'allowed');
assert.deepEqual(read.actor, principal.subject);
assert.deepEqual(read.fence, { projectVersion: 3, bindingVersion: 5 });
}
});
test('maps durable mutation conflicts without leaking repository details', async () => {
const conflicting = createClusterAutomationManagementService({
policy: {
async authorize() {
return {
effect: 'allow',
reasons: ['role_grant'],
fence: { projectVersion: 1, bindingVersion: 1 },
};
},
},
taskDefinitions: {
async appendAuthorizedTaskDefinitionRevision() {
throw new TaskDefinitionAdministrationMutationConflictError();
},
async findAuthorizedCurrentTaskDefinition() { return null; },
async listAuthorizedTaskDefinitions() {
return { definitions: [], truncated: false };
},
},
triggers: {
async appendAuthorizedTriggerRevision() {
throw new Error('unused');
},
async findAuthorizedCurrentTrigger() { return null; },
async listAuthorizedTriggers() {
return { triggers: [], truncated: false };
},
},
now: () => 1_100,
});
await assert.rejects(
conflicting.publishTask({
requestId: 'request-conflict',
command: taskCommand(),
principal,
}),
ClusterAutomationManagementConflictError,
);
});
@@ -0,0 +1,374 @@
const assert = require('node:assert/strict');
const { spawnSync } = require('node:child_process');
const {
chmodSync,
mkdtempSync,
readFileSync,
realpathSync,
rmSync,
writeFileSync,
} = require('node:fs');
const { createServer } = require('node:https');
const { tmpdir } = require('node:os');
const { join, resolve } = require('node:path');
const { afterEach, test } = require('node:test');
const {
executeClusterAutomationManagementClient,
validateClusterAutomationManagementClientResult,
} = require('@qinglong/cluster-admin/automation-management-client');
const FIXTURES = resolve(
__dirname,
'../../ql3-cluster-control/test/fixtures/mtls',
);
const CLI = resolve(
__dirname,
'../dist/automation-management/automationManagementClientCli.js',
);
const temporaryDirectories = [];
const taskCommand = Object.freeze({
projectId: 'project-a',
taskId: 'task-a',
expectedRevision: null,
mutationId: '123e4567-e89b-42d3-a456-426614174000',
name: 'Sensitive task name',
kind: 'script',
spec: {
schema: 'qinglong/script@v1',
config: { source: 'sensitive script body' },
},
labels: {},
enabled: true,
occurredAtMs: 1_000,
});
const triggerCommand = Object.freeze({
projectId: 'project-a',
triggerId: 'trigger-a',
expectedRevision: null,
mutationId: '123e4567-e89b-42d3-a456-426614174002',
taskId: 'task-a',
taskRevision: 1,
taskContentDigest: 'a'.repeat(64),
spec: {
schema: 'qinglong/cron@v1',
config: {
expression: '*/5 * * * *',
timezone: 'UTC',
misfirePolicy: 'fire_once',
},
},
enabled: true,
occurredAtMs: 1_000,
});
function envelope(operation, command) {
return Object.freeze({
schemaVersion: 1,
operation,
request: Object.freeze({
requestId: `request-${operation}`,
command,
}),
});
}
function taskResult(overrides = {}) {
return {
schemaVersion: 1,
operation: 'task.publish',
status: 'created',
task: {
projectId: 'project-a',
taskId: 'task-a',
revision: 1,
kind: 'script',
enabled: true,
contentDigest: 'b'.repeat(64),
updatedAtMs: 1_001,
...overrides,
},
};
}
function triggerResult(overrides = {}) {
return {
schemaVersion: 1,
operation: 'trigger.publish',
status: 'created',
trigger: {
projectId: 'project-a',
triggerId: 'trigger-a',
revision: 1,
taskId: 'task-a',
taskRevision: 1,
taskContentDigest: 'a'.repeat(64),
enabled: true,
contentDigest: 'c'.repeat(64),
updatedAtMs: 1_001,
...overrides,
},
};
}
function privateWrite(filePath, value) {
writeFileSync(filePath, value, { mode: 0o600 });
chmodSync(filePath, 0o600);
}
function clientFiles(port, command) {
const directory = realpathSync(
mkdtempSync(join(tmpdir(), 'ql3-automation-client-')),
);
temporaryDirectories.push(directory);
const paths = {
configFile: join(directory, 'client.json'),
commandFile: join(directory, 'command.json'),
assertionFile: join(directory, 'assertion.jwt'),
};
const caFile = join(directory, 'ca.crt');
const clientCertificateFile = join(directory, 'client.crt');
const clientPrivateKeyFile = join(directory, 'client.key');
privateWrite(caFile, readFileSync(join(FIXTURES, 'ca-cert.pem')));
privateWrite(
clientCertificateFile,
readFileSync(join(FIXTURES, 'client-cert.pem')),
);
privateWrite(
clientPrivateKeyFile,
readFileSync(join(FIXTURES, 'client-key.pem')),
);
privateWrite(
paths.configFile,
`${JSON.stringify({
schemaVersion: 1,
endpoint: `https://localhost:${port}/api/v3/automations/management`,
servername: 'localhost',
caFile,
clientCertificateFile,
clientPrivateKeyFile,
requestTimeoutMs: 2_000,
})}\n`,
);
privateWrite(paths.commandFile, `${JSON.stringify(command)}\n`);
privateWrite(
paths.assertionFile,
'eyJhbGciOiJFZERTQSJ9.eyJzdWIiOiJ1In0.c2lnbmF0dXJl',
);
return paths;
}
function listen(server) {
return new Promise((resolvePromise, reject) => {
server.once('error', reject);
server.listen(0, 'localhost', () => resolvePromise(server.address()));
});
}
function close(server) {
return new Promise((resolvePromise, reject) => {
server.close((error) => (error ? reject(error) : resolvePromise()));
});
}
afterEach(() => {
for (const directory of temporaryDirectories.splice(0)) {
rmSync(directory, { recursive: true, force: true });
}
});
test('validates Task and Trigger low-sensitive results against the request fence', () => {
const task = validateClusterAutomationManagementClientResult(
taskResult(),
envelope('task.publish', taskCommand),
);
const trigger = validateClusterAutomationManagementClientResult(
triggerResult(),
envelope('trigger.publish', triggerCommand),
);
assert.equal(task.task.taskId, 'task-a');
assert.equal(trigger.trigger.triggerId, 'trigger-a');
assert.doesNotMatch(
JSON.stringify([task, trigger]),
/Sensitive|script body|expression|mutationId|authenticationId/,
);
assert.throws(() =>
validateClusterAutomationManagementClientResult(
taskResult({ projectId: 'project-b' }),
envelope('task.publish', taskCommand),
),
);
assert.throws(() =>
validateClusterAutomationManagementClientResult(
{ ...triggerResult(), credential: 'must-not-leak' },
envelope('trigger.publish', triggerCommand),
),
);
});
test('validates bounded inspect/list results, absent state and stable cursors', () => {
const taskInspectCommand = {
schemaVersion: 1,
operation: 'task.inspect',
request: {
requestId: 'request-task-inspect',
auditEventId: '123e4567-e89b-42d3-a456-426614174010',
projectId: 'project-a',
taskId: 'task-a',
},
};
const taskInspect = validateClusterAutomationManagementClientResult(
{
schemaVersion: 1,
operation: 'task.inspect',
status: 'found',
task: taskResult().task,
},
taskInspectCommand,
);
assert.equal(taskInspect.task.taskId, 'task-a');
const triggerInspectCommand = {
schemaVersion: 1,
operation: 'trigger.inspect',
request: {
requestId: 'request-trigger-inspect',
auditEventId: '123e4567-e89b-42d3-a456-426614174011',
projectId: 'project-a',
triggerId: 'trigger-a',
},
};
const absent = validateClusterAutomationManagementClientResult(
{
schemaVersion: 1,
operation: 'trigger.inspect',
status: 'absent',
trigger: null,
},
triggerInspectCommand,
);
assert.equal(absent.trigger, null);
const taskListCommand = {
schemaVersion: 1,
operation: 'task.list',
request: {
requestId: 'request-task-list',
auditEventId: '123e4567-e89b-42d3-a456-426614174012',
projectId: 'project-a',
limit: 1,
after: { taskId: 'task-0' },
},
};
const taskList = validateClusterAutomationManagementClientResult(
{
schemaVersion: 1,
operation: 'task.list',
tasks: [taskResult().task],
truncated: true,
next: { taskId: 'task-a' },
},
taskListCommand,
);
assert.equal(taskList.next.taskId, 'task-a');
assert.throws(() =>
validateClusterAutomationManagementClientResult(
{
schemaVersion: 1,
operation: 'task.list',
tasks: [{ ...taskResult().task, spec: { secret: true } }],
truncated: false,
next: null,
},
taskListCommand,
),
);
assert.throws(() =>
validateClusterAutomationManagementClientResult(
{
schemaVersion: 1,
operation: 'task.list',
tasks: [taskResult().task],
truncated: true,
next: { taskId: 'task-b' },
},
taskListCommand,
),
);
});
test('sends one mTLS 1.3 automation command and validates its response', async () => {
let observed;
const server = createServer(
{
key: readFileSync(join(FIXTURES, 'server-key.pem')),
cert: readFileSync(join(FIXTURES, 'server-cert.pem')),
ca: readFileSync(join(FIXTURES, 'ca-cert.pem')),
requestCert: true,
rejectUnauthorized: true,
minVersion: 'TLSv1.3',
maxVersion: 'TLSv1.3',
},
(request, response) => {
const chunks = [];
request.on('data', (chunk) => chunks.push(chunk));
request.once('end', () => {
observed = {
method: request.method,
path: request.url,
authorization: request.headers.authorization,
authorized: request.socket.authorized,
protocol: request.socket.getProtocol(),
command: JSON.parse(Buffer.concat(chunks).toString('utf8')),
};
const body = Buffer.from(
JSON.stringify({
schemaVersion: 1,
requestId: 'http-request-1',
result: taskResult(),
}),
);
response.writeHead(200, {
'content-type': 'application/json; charset=utf-8',
'content-length': String(body.length),
});
response.end(body);
});
},
);
const address = await listen(server);
try {
const command = envelope('task.publish', taskCommand);
const result = await executeClusterAutomationManagementClient(
clientFiles(address.port, command),
);
assert.equal(result.requestId, 'http-request-1');
assert.equal(result.result.task.taskId, 'task-a');
assert.deepEqual(observed, {
method: 'POST',
path: '/api/v3/automations/management',
authorization:
'Bearer eyJhbGciOiJFZERTQSJ9.eyJzdWIiOiJ1In0.c2lnbmF0dXJl',
authorized: true,
protocol: 'TLSv1.3',
command,
});
} finally {
await close(server);
}
});
test('CLI exposes only private file paths and stable low-sensitive errors', () => {
const help = spawnSync(process.execPath, [CLI, '--help'], {
encoding: 'utf8',
});
assert.equal(help.status, 0);
assert.match(help.stdout, /^Usage: ql3-automation-client /);
assert.doesNotMatch(help.stdout, /token|secret|command body/i);
const invalid = spawnSync(process.execPath, [CLI, '--assertion=value'], {
encoding: 'utf8',
});
assert.equal(invalid.status, 64);
assert.match(invalid.stderr, /USAGE_INVALID/);
assert.doesNotMatch(invalid.stderr, /assertion=value/);
});
@@ -0,0 +1,128 @@
const assert = require('node:assert/strict');
const { readFileSync } = require('node:fs');
const { request: httpsRequest } = require('node:https');
const { resolve } = require('node:path');
const { test } = require('node:test');
const {
ClusterAutomationManagementConflictError,
} = require('@qinglong/cluster-admin/automation-management');
const {
startClusterAutomationManagementHttp,
} = require('@qinglong/cluster-admin/automation-management-http');
const SERVER_KEY = resolve(
__dirname,
'../../ql3-cluster-control/test/fixtures/mtls/server-key.pem',
);
const SERVER_CERT = resolve(
__dirname,
'../../ql3-cluster-control/test/fixtures/mtls/server-cert.pem',
);
const PATH = '/api/v3/automations/management';
function post(port, path = PATH) {
const body = Buffer.from(
JSON.stringify({
schemaVersion: 1,
operation: 'task.publish',
request: { requestId: 'request-task', command: {} },
}),
);
return new Promise((resolvePromise, reject) => {
const outgoing = httpsRequest(
{
host: '127.0.0.1',
port,
path,
method: 'POST',
rejectUnauthorized: false,
agent: false,
headers: {
authorization: 'Bearer assertion',
'content-type': 'application/json',
'content-length': String(body.length),
},
},
(incoming) => {
const chunks = [];
incoming.on('data', (chunk) => chunks.push(chunk));
incoming.once('end', () => {
const bytes = Buffer.concat(chunks);
resolvePromise({
statusCode: incoming.statusCode,
body: bytes.length ? JSON.parse(bytes.toString('utf8')) : null,
});
});
},
);
outgoing.once('error', reject);
outgoing.end(body);
});
}
test('serves only the automation path and maps automation conflicts to low-sensitive HTTP facts', async () => {
const privateKey = Buffer.from(readFileSync(SERVER_KEY));
const application = await startClusterAutomationManagementHttp({
host: '127.0.0.1',
port: 0,
tls: {
privateKey,
certificate: Buffer.from(readFileSync(SERVER_CERT)),
},
identities: {
async reload() {
return {
schemaVersion: 1,
generation: 1,
digest: 'digest',
issuer: 'https://identity.example.test/',
audience: 'qinglong3-automation-management',
activeKeyIds: ['key-1'],
revokedKeyIds: [],
};
},
bind() {
return {
async authenticate() {
return {
subject: { type: 'user', id: 'operator-a' },
authenticationId: 'session-operator-a',
authenticatedAtMs: 900,
expiresAtMs: 10_000,
assurance: 'multi_factor',
};
},
};
},
},
transport: {
async execute(_command, authentication) {
await authentication.authenticate();
throw new ClusterAutomationManagementConflictError();
},
},
now: () => 1_000,
});
try {
assert.equal(privateKey.every((value) => value === 0), true);
const conflict = await post(application.address.port);
assert.deepEqual(conflict, {
statusCode: 409,
body: {
schemaVersion: 1,
requestId: conflict.body.requestId,
error: { code: 'conflict' },
},
});
assert.match(conflict.body.requestId, /^[0-9a-f-]{36}$/);
const absent = await post(
application.address.port,
'/api/v3/plugin-packages/management',
);
assert.equal(absent.statusCode, 404);
assert.equal(absent.body.error.code, 'not_found');
} finally {
await application.close();
}
});
@@ -0,0 +1,506 @@
const assert = require('node:assert/strict');
const {
chmodSync,
mkdtempSync,
readFileSync,
realpathSync,
rmSync,
writeFileSync,
} = require('node:fs');
const { tmpdir } = require('node:os');
const { join, resolve } = require('node:path');
const { test } = require('node:test');
const { ProjectPolicyEngine } = require('@qinglong/runtime-core/project-policy');
const {
PostgresProjectPolicyRepository,
PostgresTaskDefinitionAdministrationRepository,
PostgresTriggerAdministrationRepository,
assertPostgresAutomationManagerSchemaReady,
createPostgresDatabaseOpener,
} = require('@qinglong/cluster-postgres/automation-manager');
const {
runPostgresMigrations,
} = require('@qinglong/cluster-postgres/migration');
const {
createClusterAutomationManagementService,
} = require('@qinglong/cluster-admin/automation-management');
const {
executeClusterAutomationManagementClient,
} = require('@qinglong/cluster-admin/automation-management-client');
const {
startClusterAutomationManagementHttp,
} = require('@qinglong/cluster-admin/automation-management-http');
const {
createClusterAutomationManagementTransport,
} = require('@qinglong/cluster-admin/automation-management-transport');
const MIGRATION_URL = process.env.QL3_TEST_POSTGRES_URL;
const AUTOMATION_URL =
process.env.QL3_TEST_POSTGRES_AUTOMATION_MANAGER_URL;
const FIXTURES = resolve(
__dirname,
'../../ql3-cluster-control/test/fixtures/mtls',
);
function opener(role, connectionString, applicationName) {
return createPostgresDatabaseOpener({
role,
connection: {
connectionString,
tls: { mode: 'disable' },
},
pool: { maxConnections: 2, applicationName },
onPoolError(error) {
throw error;
},
});
}
const principal = Object.freeze({
subject: Object.freeze({ type: 'user', id: 'automation-operator' }),
authenticationId: 'oidc:automation-live-contract',
authenticatedAtMs: 1,
expiresAtMs: 4_102_444_800_000,
assurance: 'hardware',
});
const identities = Object.freeze({
async reload() {
return Object.freeze({
schemaVersion: 1,
generation: 1,
digest: 'automation-live-contract',
issuer: 'https://identity.example.test/',
audience: 'qinglong3-automation-management',
activeKeyIds: Object.freeze(['automation-live-key']),
revokedKeyIds: Object.freeze([]),
});
},
bind() {
return Object.freeze({
async authenticate() {
return principal;
},
});
},
});
function privateWrite(filePath, value) {
writeFileSync(filePath, value, { mode: 0o600 });
chmodSync(filePath, 0o600);
}
function clientFiles(port, command, directories) {
const directory = realpathSync(
mkdtempSync(join(tmpdir(), 'ql3-automation-pg-client-')),
);
directories.push(directory);
const caFile = join(directory, 'ca.crt');
const clientCertificateFile = join(directory, 'client.crt');
const clientPrivateKeyFile = join(directory, 'client.key');
const paths = {
configFile: join(directory, 'client.json'),
commandFile: join(directory, 'command.json'),
assertionFile: join(directory, 'assertion.jwt'),
};
privateWrite(caFile, readFileSync(join(FIXTURES, 'ca-cert.pem')));
privateWrite(
clientCertificateFile,
readFileSync(join(FIXTURES, 'client-cert.pem')),
);
privateWrite(
clientPrivateKeyFile,
readFileSync(join(FIXTURES, 'client-key.pem')),
);
privateWrite(
paths.configFile,
`${JSON.stringify({
schemaVersion: 1,
endpoint: `https://localhost:${port}/api/v3/automations/management`,
servername: 'localhost',
caFile,
clientCertificateFile,
clientPrivateKeyFile,
requestTimeoutMs: 5_000,
})}\n`,
);
privateWrite(paths.commandFile, `${JSON.stringify(command)}\n`);
privateWrite(
paths.assertionFile,
'eyJhbGciOiJFZERTQSJ9.eyJzdWIiOiJ1In0.c2lnbmF0dXJl',
);
return paths;
}
function envelope(operation, requestId, command) {
return Object.freeze({
schemaVersion: 1,
operation,
request: Object.freeze({ requestId, command }),
});
}
function readEnvelope(operation, request) {
return Object.freeze({
schemaVersion: 1,
operation,
request: Object.freeze(request),
});
}
function taskCommand(projectId, revision, suffix, value) {
return Object.freeze({
projectId,
taskId: 'managed-task',
expectedRevision: revision,
mutationId: `123e4567-e89b-42d3-a456-426614176${suffix}`,
name: 'Managed Task',
kind: 'command',
spec: Object.freeze({
schema: 'qinglong/command@v1',
config: Object.freeze({
command: Object.freeze({
kind: 'argv',
file: '/bin/echo',
args: Object.freeze([value]),
}),
}),
}),
labels: Object.freeze({}),
enabled: true,
occurredAtMs: 1_000 + Number(suffix),
});
}
function triggerCommand(projectId, revision, task, suffix) {
return Object.freeze({
projectId,
triggerId: 'managed-trigger',
expectedRevision: revision,
mutationId: `123e4567-e89b-42d3-a456-426614177${suffix}`,
taskId: task.taskId,
taskRevision: task.revision,
taskContentDigest: task.contentDigest,
spec: Object.freeze({
schema: 'qinglong/cron@v1',
config: Object.freeze({
expression: '*/5 * * * *',
timezone: 'UTC',
misfirePolicy: 'skip',
}),
}),
enabled: true,
occurredAtMs: 2_000 + Number(suffix),
});
}
function buildTransport(database) {
return createClusterAutomationManagementTransport({
service: createClusterAutomationManagementService({
policy: new ProjectPolicyEngine(
new PostgresProjectPolicyRepository(database.pool),
),
taskDefinitions: new PostgresTaskDefinitionAdministrationRepository(
database.pool,
),
triggers: new PostgresTriggerAdministrationRepository(database.pool),
}),
});
}
async function startManager(transport, sequence) {
return startClusterAutomationManagementHttp({
host: 'localhost',
port: 0,
tls: {
privateKey: Buffer.from(readFileSync(join(FIXTURES, 'server-key.pem'))),
certificate: Buffer.from(
readFileSync(join(FIXTURES, 'server-cert.pem')),
),
clientCertificateAuthority: Buffer.from(
readFileSync(join(FIXTURES, 'ca-cert.pem')),
),
clientCertificateRevocationList: Buffer.from(
readFileSync(join(FIXTURES, 'empty-crl.pem')),
),
},
transport,
identities,
createRequestId: () => `manager-${sequence.value++}`,
});
}
if (!MIGRATION_URL || !AUTOMATION_URL) {
test('automation PostgreSQL HTTPS integration requires migration and automation-manager URLs', {
skip: true,
});
} else {
test(
'two HTTPS managers converge concurrent publish and commit-response loss on real PostgreSQL',
{ timeout: 60_000 },
async () => {
const directories = [];
const sequence = { value: 1 };
const migration = await opener(
'migration',
MIGRATION_URL,
'ql3-automation-live-migration',
)();
const firstDatabase = await opener(
'automation-manager',
AUTOMATION_URL,
'ql3-automation-live-first',
)();
const secondDatabase = await opener(
'automation-manager',
AUTOMATION_URL,
'ql3-automation-live-second',
)();
let first;
let second;
let responseLoss;
try {
await runPostgresMigrations({ pool: migration.pool });
const readiness = await Promise.all([
assertPostgresAutomationManagerSchemaReady(firstDatabase.pool),
assertPostgresAutomationManagerSchemaReady(secondDatabase.pool),
]);
assert.deepEqual(
readiness.map((entry) => entry.currentUser),
['ql3_automation_manager', 'ql3_automation_manager'],
);
const observed = await migration.pool.query(
`SELECT floor(extract(epoch FROM clock_timestamp()) * 1000)::bigint
AS "observedAtMs"`,
);
const occurredAtMs = Number(observed.rows[0].observedAtMs);
const projectId = `automation-https-${process.pid}`;
await migration.pool.query(
`DELETE FROM "ql3"."projects" WHERE id = $1`,
[projectId],
);
await migration.pool.query(
`INSERT INTO "ql3"."projects" (
id, name, slug, status, version, created_at_ms, updated_at_ms
) VALUES ($1, 'Automation HTTPS', $1, 'active', 1, $2, $2)`,
[projectId, occurredAtMs],
);
await migration.pool.query(
`INSERT INTO "ql3"."project_role_bindings" (
project_id, subject_type, subject_id, version, state, role,
mutation_id, changed_by_type, changed_by_id, created_at_ms
) VALUES ($1, 'user', $2, 1, 'active', 'owner', $3,
'user', $2, $4)`,
[
projectId,
principal.subject.id,
'123e4567-e89b-42d3-a456-426614176000',
occurredAtMs,
],
);
const firstTransport = buildTransport(firstDatabase);
const secondTransport = buildTransport(secondDatabase);
first = await startManager(firstTransport, sequence);
second = await startManager(secondTransport, sequence);
const createTask = envelope(
'task.publish',
'concurrent-task-create',
taskCommand(projectId, null, '001', 'v1'),
);
const concurrent = await Promise.all([
executeClusterAutomationManagementClient(
clientFiles(first.address.port, createTask, directories),
),
executeClusterAutomationManagementClient(
clientFiles(second.address.port, createTask, directories),
),
]);
assert.deepEqual(
concurrent.map((entry) => entry.result.status).sort(),
['created', 'existing'],
);
const taskV1 = concurrent[0].result.task;
let dropCommittedResponse = true;
responseLoss = await startManager(
Object.freeze({
async execute(command, authentication) {
const result = await firstTransport.execute(
command,
authentication,
);
if (dropCommittedResponse) {
dropCommittedResponse = false;
throw new Error('simulated post-commit response loss');
}
return result;
},
}),
sequence,
);
const updateTask = envelope(
'task.publish',
'response-loss-task-update',
taskCommand(projectId, 1, '002', 'v2'),
);
await assert.rejects(
executeClusterAutomationManagementClient(
clientFiles(responseLoss.address.port, updateTask, directories),
),
(error) =>
error?.statusCode === 500 &&
error?.responseCode === 'internal_error',
);
const converged = await executeClusterAutomationManagementClient(
clientFiles(second.address.port, updateTask, directories),
);
assert.equal(converged.result.status, 'existing');
assert.equal(converged.result.task.revision, 2);
const createTrigger = envelope(
'trigger.publish',
'trigger-create-v2',
triggerCommand(projectId, null, converged.result.task, '001'),
);
const triggerV1 = await executeClusterAutomationManagementClient(
clientFiles(first.address.port, createTrigger, directories),
);
assert.equal(triggerV1.result.status, 'created');
assert.equal(triggerV1.result.trigger.taskRevision, 2);
const taskV3 = await executeClusterAutomationManagementClient(
clientFiles(
second.address.port,
envelope(
'task.publish',
'task-update-v3',
taskCommand(projectId, 2, '003', 'v3'),
),
directories,
),
);
const triggerV2 = await executeClusterAutomationManagementClient(
clientFiles(
first.address.port,
envelope(
'trigger.publish',
'trigger-repin-v3',
triggerCommand(projectId, 1, taskV3.result.task, '002'),
),
directories,
),
);
assert.equal(triggerV2.result.status, 'updated');
assert.equal(triggerV2.result.trigger.taskRevision, 3);
const taskInspection = await executeClusterAutomationManagementClient(
clientFiles(
second.address.port,
readEnvelope('task.inspect', {
requestId: 'task-inspect-v3',
auditEventId: '123e4567-e89b-42d3-a456-426614178001',
projectId,
taskId: 'managed-task',
}),
directories,
),
);
assert.equal(taskInspection.result.status, 'found');
assert.equal(taskInspection.result.task.revision, 3);
const taskPage = await executeClusterAutomationManagementClient(
clientFiles(
first.address.port,
readEnvelope('task.list', {
requestId: 'task-list-v3',
auditEventId: '123e4567-e89b-42d3-a456-426614178002',
projectId,
limit: 1,
}),
directories,
),
);
assert.equal(taskPage.result.tasks.length, 1);
assert.equal(taskPage.result.next, null);
const triggerInspection =
await executeClusterAutomationManagementClient(
clientFiles(
first.address.port,
readEnvelope('trigger.inspect', {
requestId: 'trigger-inspect-v2',
auditEventId: '123e4567-e89b-42d3-a456-426614178003',
projectId,
triggerId: 'managed-trigger',
}),
directories,
),
);
assert.equal(triggerInspection.result.trigger.revision, 2);
const triggerPage = await executeClusterAutomationManagementClient(
clientFiles(
second.address.port,
readEnvelope('trigger.list', {
requestId: 'trigger-list-v2',
auditEventId: '123e4567-e89b-42d3-a456-426614178004',
projectId,
limit: 1,
}),
directories,
),
);
assert.equal(triggerPage.result.triggers.length, 1);
assert.equal(triggerPage.result.next, null);
assert.doesNotMatch(
JSON.stringify([
taskInspection.result,
taskPage.result,
triggerInspection.result,
triggerPage.result,
]),
/Managed Task|\/bin\/echo|expression|authenticationId|mutationId/,
);
const durable = await migration.pool.query(
`SELECT
(SELECT count(*)::integer FROM "ql3"."task_definition_revisions"
WHERE project_id = $1 AND task_id = 'managed-task') AS "taskRevisions",
(SELECT current_revision FROM "ql3"."task_definitions"
WHERE project_id = $1 AND task_id = 'managed-task') AS "taskHead",
(SELECT count(*)::integer FROM "ql3"."trigger_revisions"
WHERE project_id = $1 AND trigger_id = 'managed-trigger') AS "triggerRevisions",
(SELECT current_revision FROM "ql3"."triggers"
WHERE project_id = $1 AND trigger_id = 'managed-trigger') AS "triggerHead",
(SELECT count(*)::integer FROM "ql3"."security_audit_events"
WHERE project_id = $1 AND outcome = 'allowed') AS "allowedAudits"`,
[projectId],
);
assert.deepEqual(durable.rows, [
{
taskRevisions: 3,
taskHead: 3,
triggerRevisions: 2,
triggerHead: 2,
allowedAudits: 9,
},
]);
assert.equal(taskV1.revision, 1);
} finally {
await Promise.allSettled([
responseLoss?.close(),
first?.close(),
second?.close(),
]);
await Promise.allSettled([
firstDatabase.close(),
secondDatabase.close(),
migration.close(),
]);
for (const directory of directories) {
rmSync(directory, { recursive: true, force: true });
}
}
},
);
}
@@ -0,0 +1,211 @@
const assert = require('node:assert/strict');
const { readFile, writeFile, mkdtemp, rm } = require('node:fs/promises');
const { tmpdir } = require('node:os');
const { join, resolve } = require('node:path');
const { test } = require('node:test');
const {
ClusterAutomationManagementProcessConfigError,
loadClusterAutomationManagementProcessConfig,
startClusterAutomationManagementProcess,
} = require('@qinglong/cluster-admin/automation-management-process');
const FIXTURES = resolve(
__dirname,
'../../ql3-cluster-control/test/fixtures/mtls',
);
const NOW_MS = Date.UTC(2030, 0, 1);
function enabledEnvironment(paths, overrides = {}) {
return {
QL3_AUTOMATION_MANAGEMENT_ENABLED: 'true',
QL3_PROFILE: 'cluster-admin',
QL3_AUTOMATION_MANAGEMENT_HOST: '127.0.0.1',
QL3_AUTOMATION_MANAGEMENT_PORT: '8445',
QL3_AUTOMATION_MANAGEMENT_TLS_CERT_FILE: paths.certificateFile,
QL3_AUTOMATION_MANAGEMENT_TLS_KEY_FILE: paths.privateKeyFile,
QL3_AUTOMATION_MANAGEMENT_CLIENT_CA_FILE:
paths.clientCertificateAuthorityFile,
QL3_AUTOMATION_MANAGEMENT_CLIENT_CRL_FILE:
paths.clientCertificateRevocationListFile,
QL3_AUTOMATION_MANAGEMENT_IDENTITY_KEYSET_FILE: paths.identityKeysetFile,
QL3_POSTGRES_AUTOMATION_MANAGER_URL:
'postgresql://ql3_automation_manager:secret@postgres.example.test/ql3',
QL3_POSTGRES_AUTOMATION_MANAGER_TLS_MODE: 'disable',
QL3_POSTGRES_AUTOMATION_MANAGER_ALLOW_INSECURE: 'true',
...overrides,
};
}
async function tlsFixture(run) {
const directory = await mkdtemp(join(tmpdir(), 'ql3-automation-manager-'));
const paths = {
certificateFile: join(directory, 'tls.crt'),
privateKeyFile: join(directory, 'tls.key'),
clientCertificateAuthorityFile: join(directory, 'client-ca.crt'),
clientCertificateRevocationListFile: join(directory, 'client.crl'),
identityKeysetFile: join(directory, 'keyset.json'),
};
try {
await writeFile(
paths.certificateFile,
await readFile(join(FIXTURES, 'server-cert.pem')),
{ mode: 0o644 },
);
await writeFile(
paths.privateKeyFile,
await readFile(join(FIXTURES, 'server-key.pem')),
{ mode: 0o640 },
);
await writeFile(
paths.clientCertificateAuthorityFile,
await readFile(join(FIXTURES, 'ca-cert.pem')),
{ mode: 0o644 },
);
await writeFile(
paths.clientCertificateRevocationListFile,
await readFile(join(FIXTURES, 'empty-crl.pem')),
{ mode: 0o644 },
);
await writeFile(paths.identityKeysetFile, '{}\n', { mode: 0o644 });
return await run(paths);
} finally {
await rm(directory, { recursive: true, force: true });
}
}
test('disabled automation manager opens no PostgreSQL or file authority', async () => {
let opened = 0;
const runtime = await startClusterAutomationManagementProcess({
environment: { QL3_AUTOMATION_MANAGEMENT_ENABLED: 'false' },
async openDatabase() {
opened += 1;
throw new Error('must not open');
},
});
assert.equal(runtime.status, 'disabled');
assert.equal(opened, 0);
await runtime.close();
});
test('loads bounded low-footprint automation-only HTTPS and PostgreSQL configuration', async () => {
await tlsFixture(async (paths) => {
const config = loadClusterAutomationManagementProcessConfig(
enabledEnvironment(paths),
);
assert.equal(config.enabled, true);
assert.equal(config.port, 8445);
assert.equal(config.database.pool.maxConnections, 2);
assert.equal(
config.database.pool.applicationName,
'qinglong3-automation-manager',
);
assert.equal(config.http.maxConnections, 32);
assert.equal(config.http.maxConcurrentRequests, 16);
assert.match(
config.database.connection.connectionString,
/^postgresql:\/\/ql3_automation_manager:/,
);
assert.equal(config.database.connection.tls.mode, 'disable');
});
assert.throws(
() =>
loadClusterAutomationManagementProcessConfig({
QL3_AUTOMATION_MANAGEMENT_ENABLED: 'true',
QL3_PROFILE: 'cluster-admin',
QL3_POSTGRES_AUTOMATION_MANAGER_TLS_MODE: 'disable',
}),
ClusterAutomationManagementProcessConfigError,
);
});
test('starts only after automation readiness and identity validation then closes in order', async () => {
await tlsFixture(async (paths) => {
const order = [];
let privateKey;
let transport;
let httpClosed = 0;
let databaseClosed = 0;
const pool = {
async query() {
throw new Error('repositories must remain lazy during composition');
},
async connect() {
throw new Error('repositories must remain lazy during composition');
},
};
const runtime = await startClusterAutomationManagementProcess({
environment: enabledEnvironment(paths),
now: () => NOW_MS,
async openDatabase() {
order.push('open');
return {
pool,
async close() {
order.push('database-close');
databaseClosed += 1;
},
};
},
async assertReady(candidate) {
order.push('ready');
assert.equal(candidate, pool);
return {
ready: true,
writablePrimary: true,
serverVersionNum: 180004,
serverMajor: 18,
currentUser: 'ql3_automation_manager',
contractName: 'control-core',
contractVersion: 53,
migrationIds: [
'pg-0052-automation-management-identity-keyset-ledger',
],
};
},
identities: {
async reload() {
order.push('identity');
return {
schemaVersion: 1,
generation: 3,
digest: 'identity-digest',
issuer: 'https://identity.example.test/',
audience: 'qinglong3-automation-management',
activeKeyIds: ['key-3'],
revokedKeyIds: ['key-2'],
};
},
bind() {
throw new Error('HTTP stub does not authenticate');
},
},
async startHttp(options) {
order.push('http');
privateKey = options.tls.privateKey;
transport = options.transport;
assert.ok(options.tls.clientCertificateAuthority);
assert.ok(options.tls.clientCertificateRevocationList);
return {
status: 'active',
address: { host: options.host, port: options.port },
availabilityStatus: () => 'ready',
withdraw() {},
async close() {
order.push('http-close');
httpClosed += 1;
},
};
},
});
assert.equal(runtime.status, 'active');
assert.deepEqual(order, ['open', 'ready', 'identity', 'http']);
assert.equal(typeof transport.execute, 'function');
assert.equal(privateKey.every((byte) => byte === 0), true);
assert.equal(runtime.database.contractVersion, 53);
await Promise.all([runtime.close(), runtime.close()]);
assert.equal(httpClosed, 1);
assert.equal(databaseClosed, 1);
assert.deepEqual(order.slice(-2), ['http-close', 'database-close']);
});
});

Some files were not shown because too many files have changed in this diff Show More