qinglong/docs/operations/ql3-worker-credential-execution.md

QingLong 3.0 caller-driven Worker credential execution

This operation consumes one already approved Worker credential action. It is a short-lived Kubernetes Job, not a scheduler, controller or rotation loop. The management Deployment never receives this Job's PostgreSQL credential, pepper or Kubernetes issuer token.

Preconditions

• PostgreSQL is migrated through pg-0050 and the dedicated ql3_worker_credential_executor role is ready.
• The Worker credential-admin overlay is installed. Its delivery ServiceAccount has the reviewed 8-allow/20-deny RBAC matrix.
• Two distinct strong Users created and approved the exact immutable plan.
• The caller has the resulting action, approval, consumption, dispatch and audit IDs without any credential token.
• The independently verified cluster-admin image digest is available.

Copy deploy/kubernetes/ql3-cluster/operations/worker-credential-executor/config.example.yaml to a private per-dispatch overlay. Replace every placeholder. Keep command.json at schema version 1 with exactly these six fields:

{"schemaVersion":1,"actionRef":"…","approvalRequestId":"…","consumptionId":"…","dispatchId":"…","auditEventId":"…"}

Create the pepper through a Secret manager. It must be a distinct canonical 32-byte base64url value and must not appear in the command, argv, logs, Git or the management Deployment. Treat the immutable command ConfigMap as per-dispatch input; do not rewrite and reuse it for a different approval.

Private network and image overlay

The committed base deliberately permits only DNS. The CloudNativePG overlay adds only the exact ql3-postgres Pods on TCP 5432. Copy api-server-egress-patch.example.yaml into the private overlay and replace its TEST-NET address and port with the exact Kubernetes API destination observed by Pods under the deployed CNI. Some CNIs enforce policy before Service DNAT and others after it; verify the enforced destination rather than assuming the kubernetes.default ClusterIP is sufficient. Never use 0.0.0.0/0, an empty egress peer, or a namespace-wide API destination.

The private Kustomization must also replace the all-zero qinglong3-cluster-admin digest with the independently verified digest. Keep the Job backoffLimit at zero: a caller inspects durable execution state before explicitly deciding whether to retry the same command.

Render and audit before creating anything:

pnpm audit:cluster-deployment:ql3
kubectl kustomize /absolute/path/to/private-worker-executor-overlay >/tmp/ql3-worker-executor.yaml

Review the rendered file for exactly one Job, no CronJob or Deployment, the executor-only PostgreSQL Secret, a 600-second projected ServiceAccount token and the exact TokenRequest RoleBinding. Then create and observe it:

kubectl create -k /absolute/path/to/private-worker-executor-overlay
kubectl -n qinglong3-system wait \
  --for=condition=Complete job/ql3-worker-credential-executor \
  --timeout=10m
kubectl -n qinglong3-system logs job/ql3-worker-credential-executor

Successful stdout contains only low-sensitive execution and delivery status. It never contains the issued Worker token. On failure, inspect the durable approval dispatch and execution receipt before recreating the same exact Job; do not generate a new command identity or broaden RBAC to force progress.

Live evidence

The test:worker-kubernetes-rollout-live:ql3 gate runs current-source cluster-admin, cluster-control and Worker images in K3s v1.34.3+k3s1 against digest-bound PostgreSQL 18.4. Its first caller-driven Job completed a third credential rotation with deliveryStatus=published and tokenRequestUsed=true. A second independent Job replayed the exact same command and completed with deliveryStatus=existing and tokenRequestUsed=false. The durable database facts converged to four plans, consumed approvals, dispatches, successful executions, credentials and published deliveries plus sixteen management security audit events. Three host-side fresh execution/exact-replay pairs perform exactly nine authorization rechecks; the separate Job keeps its own authorization boundary.

The product phase then composes the real Cluster Worker ingress and production Worker over TLS 1.3 mutual authentication. Credential generation 4 and a new client identity each force a Recreate replacement on the same RWO PVC. The three distinct Worker Sessions must each persist online, at least one heartbeat audit, draining and offline; their generations must increase strictly. The final scale-to-zero must complete graceful drain before the 360-second Kubernetes termination grace expires.

That gate also demonstrates why private overlays must account for the active CNI enforcement point: K3s-in-Docker required exact egress to both the kubernetes.default Service /32:443 and its post-DNAT API backend /32:6443. This is evidence for exact destination discovery, not permission to copy those fixture addresses into another cluster or widen the destination CIDR.

Run this destructive fixture only with explicit opt-in and a new private report path. The parent directory must already exist, must not be a symlink and should be mode 0700:

QL3_WORKER_KUBERNETES_ROLLOUT_LIVE=1 \
QL3_KUBECTL_BIN=/absolute/path/to/kubectl \
pnpm test:worker-kubernetes-rollout-live:ql3 \
  --report=/absolute/private/worker-kubernetes-report.json

pnpm audit:worker-kubernetes-rollout-live:ql3 \
  --report=/absolute/private/worker-kubernetes-report.json

The producer refuses to overwrite a report and writes it with mode 0600. The auditor is process-independent, rejects non-exact schemas and secret material, and must return compatible=true. The corresponding GitHub Actions workflow is manual-only; it does not add K3s, PostgreSQL or Cluster packages to an Edge/Standalone deployment.

After retaining the required audit evidence, delete the per-dispatch command ConfigMap and pepper Secret according to the deployment retention policy. The Job TTL removes only the finished Job; it does not remove those caller-owned inputs.