qinglong/packages/ql3-cluster-admin/README.md

# `@qinglong/cluster-admin`

This private QingLong 3.0 package owns explicit cluster operations and the
bounded Cluster Copilot MCP plus Cluster field Console product surfaces.
Database/Kubernetes administration remains short-lived and requires distinct
purpose-bound authority. The MCP subpath has only the remote API client; the
Console is a loopback-only read BFF serving digest-bound static assets. Neither
opens database or Kubernetes authority, enters the legacy 2.x Web application,
or resides in `cluster-control`.

The Console accepts thirteen default Run, Task, Workflow and Copilot reads,
plus independently enabled Run cancellation, Worker and Plugin Package
installation observations, for at most twenty exact operations. The browser
cannot provide an upstream path or HTTP method, and every list page and
detail/evidence read requires an explicit click. Its Cluster API credential
stays in a canonical owner-private file and is reread for each upstream
request; browser JavaScript receives only a separate session token which cannot
call Cluster APIs. It binds `127.0.0.1`, enforces exact Host/Origin, no-store
responses and a closed CSP, renders all returned data only via `textContent`,
and keeps start/diagnose/cancel, polling, cache, WebSocket, ServiceWorker and
legacy session authority absent.

The fourth digest-bound Console asset builds an explicit redacted evidence
bundle from only the current page's completed reads. The in-memory ledger is
bounded to the newest 16 entries and 8 MiB; export makes no network request and
produces at most 512 KiB of UTF-8 JSON. Identifiers are correlated only through
per-bundle typed aliases, while free text, names, paths, commands, inputs,
outputs, environment, errors, credentials, unknown fields and Copilot model
text are omitted. Per-fact and top-level SHA-256 values provide self-integrity,
not a server signature, durable audit or action authority.

The same package provides an independently implemented offline verifier:

```sh
ql3-cluster-admin evidence-verify \
  --bundle=/absolute/qinglong-cluster-evidence.json
```

It performs one read-only, no-follow file read, revalidates the complete fixed
schema/redaction/alias contract and recomputes the top-level digest without a
network request or file write. A verified result does not prove the server
origin, attestation or durable audit, and cannot recompute the per-fact hashes
because the deliberately omitted raw facts are not present.

The reviewed operator-workstation setup, private-file ceremony, release
verification, preflight and session lifecycle are documented in
`deploy/console/ql3-cluster-copilot/README.md`. Native execution binds host
loopback directly. The signed Admin OCI also carries an exact launcher which
uses a container-internal listener only with a fixed publication on host
`127.0.0.1`; arbitrary port mappings, Kubernetes workloads and shared-network
listeners remain forbidden.

The admin role can append Identity/API Credential mutations and their security
audit in one serializable transaction, and can perform bounded read-only audit
queries. It cannot read or mutate Run, Project or RoleBinding data, cannot
update/delete credential history, and cannot delete audit events.

Prompt output lifecycle work remains outside the resident control plane. The
explicit `prompt-output-gc-process` runs one bounded retention page, while
`prompt-output-key-retirement-process` coordinates one durable PostgreSQL
prepare/material-delete/complete operation through an injected material
authority. The latter intentionally does not select or dynamically load a KMS:
each deployment must bind a reviewed KMS/HSM adapter, and the process returns
only content-free identities, digests, status, and database time.

Credential issue/rotation returns the bearer token only for a newly inserted
mutation. Exact mutation replay returns `token: null`; a caller that loses the
one-time response must rotate again instead of recovering secret material from
storage. Replay equality covers caller-controlled semantics (actor,
authentication/request/mutation identity, subject, version, operation and
explicit absolute `notBeforeAtMs`/`expiresAtMs`), while generated secret/digest
and server timestamps are excluded. A collision in any semantic field fails
closed.

Production Worker credential rollout must use the explicit
`@qinglong/cluster-admin/worker-credential-delivery` subpath instead of exposing
that one-time response. It stages a new credential ID before one atomic
credential/mutation/audit/delivery commit, appends publication acknowledgement,
and never returns the token. The current slice is intentionally short-lived and
timer-free. Authenticated Session observation is appended in the Session
transaction; a bounded PostgreSQL-timed recovery page can resume publication,
wait for observation, and atomically revoke the previous credential while
appending delivery v4. The explicit
`@qinglong/cluster-admin/worker-credential-file-delivery` subpath supplies a
bounded POSIX implementation for Docker bind mounts, systemd deployments and
controlled single-writer volumes: private no-replace stages, generation-fenced
atomic token replacement, durable directory synchronization and bounded stage
enumeration. Capability v15 also provides a database-authorized orphan cleanup:
the inventory coordinator must append an exact immutable discard tombstone
before deleting a stage, while credential commit and discard authorization share
one PostgreSQL advisory transaction lock so exactly one can win. Authorization
and unfinished cleanup are recovered through separate bounded pages without a
resident timer. The explicit
`@qinglong/cluster-admin/worker-credential-kubernetes-delivery` subpath provides
the multi-writer deployment adapter without adding another package: immutable
per-delivery staging Secrets, target updates fenced by the resourceVersion from
the preceding GET, exact replay after 409, and UID + resourceVersion deletion
preconditions. It uses the exact-pinned official Kubernetes client through a
narrow injected Secret API, owns no watcher/timer/cache, and fails closed when
the stage inventory exceeds 128 objects; cleanup pages remain capped at 64.
A real k3s 1.34 API-server test with a dedicated ServiceAccount proves the
get/list/create/update/delete-only RBAC matrix, concurrent update single winner,
and preconditioned delete. Kubernetes HA/control-plane failover, stale-lock
repair for the POSIX adapter, PostgreSQL 18/failover evidence, and product-facing
administration remain gated.

Plugin Package recovery is exposed only through
`plugin-package-oci-stage`, `plugin-package-recovery` and
`plugin-package-recovery-process`. The process is the entrypoint for a
short-lived recovery Job; it opens one `ql3_package_executor` Pool, proves the
exact executor schema/role
readiness, serially converges a bounded recovery cycle, closes PostgreSQL and
exits. It is not imported by `cluster-control`.

The OCI stage authority implements the OCI Distribution API without another
workspace package. A PackageLock binds the OCI manifest digest in its locator
and independently binds the single bundle layer digest and byte length. The
resolver accepts only an explicit registry allowlist, HTTPS, no redirects and
no ambient credential provider. It validates the package manifest/config,
retrieves exactly one lock-annotated OCI referrer, verifies its Ed25519
publisher signature, and streams the bundle through the common inspector.
The deterministic evidence can be re-resolved after process restart; a bounded
64-entry cache only avoids fetching the same evidence twice during one Job.

The production Kustomize operation gives its dedicated ServiceAccount only
namespace-scoped ConfigMap `get/create/update`. It receives neither Secret API
access nor runtime/migration database credentials. A dedicated Kind 1.32.8
live gate runs two non-root, read-only-rootfs admin-image Pods with real
projected ServiceAccount tokens. It proves one API-confirmed create response
loss converges through durable inspect without a second create, and that two
processes attempting the same `resourceVersion` produce one winner and one
conflict. ConfigMap list/delete, Secret access and cross-namespace reads are
all denied. The response-loss injection is at the client boundary after the
API confirms create; it is not raw-wire packet loss or Kubernetes
control-plane HA evidence. Private registry authentication and package
resource-generation consumers remain gated.

Cluster Package management authentication is exposed only through the
`plugin-package-identity-assertion` and
`plugin-package-management-transport` subpaths. The identity verifier accepts
only a dedicated compact JWS purpose with one canonical HTTPS issuer, one exact
audience, an explicitly reviewed public-key set and deployment-specific
ACR-plus-required-AMR mappings to `multi_factor` or `hardware`. It supports
Ed25519, P-256 and bounded RSA public keys through Node 24 built-in crypto and
adds no dependency. Raw assertions and `jti` values do not enter management
requests or results. The shared verifier/file implementation is now composed
by two authority-specific processes: Plugin Package management and Worker
credential management. The Worker process is available only through
`worker-credential-management-process`; it opens the dedicated
`worker-credential-manager` PostgreSQL role, validates the v49 role contract,
pins keyset generations under authority `worker-credential-management`, uses
database-clock durable quota before management state reads, then exposes only
the fixed TLS 1.3 Worker management route. It does not import Worker delivery,
TokenRequest or executor authority. Its disabled gate opens no files, listener
or database, and the Kubernetes operation remains explicit opt-in.

The three-node K3s management gate runs two required-anti-affinity replicas and
the production TLS 1.3 client against each exact Pod. It proves shared durable
quota (8 admitted/8 limited), cross-Pod semantic replay, identity generation
overlap/revocation/rollback, and database-failure readiness withdrawal followed
by fresh-Pod-only recovery. Plan replay equality covers caller-controlled plan
semantics; server-authored plan times and their derived digests return the
original stored plan, while target or requester drift remains a conflict.

Approved Worker delivery execution is exposed separately through
`worker-credential-executor-process` and `ql3-worker-credential-execute`. The
process accepts one exact non-secret command file, a private canonical 32-byte
pepper file, the dedicated executor PostgreSQL role and one in-cluster issuer
session. It confirms the exact TokenRequest authorization before and during
execution, delegates the 8-allow/20-deny delivery proof to the restricted
session, clears issuer client credentials and exits after one result. The
Kubernetes operation is a caller-created `Job`, never a `CronJob` or resident
controller; its base NetworkPolicy remains DNS-only until a private overlay
adds the exact API-server and database destinations.
A real K3s `v1.34.3+k3s1` gate runs this production image and process: the first
Job publishes generation 3 with one 600-second TokenRequest, while a second
independent Job replays the exact command as `existing` without another token
request. Both use the executor-only one-connection PostgreSQL role,
`backoffLimit=0`, no automounted token and exact API/PostgreSQL egress.