feat(ql3): establish 3.0 incubation baseline

2026-09-22 02:27:44 +08:00 · 2026-08-12 00:25:26 +08:00
parent 4bf92dcfeb
commit c699c32461
2817 changed files with 779642 additions and 653 deletions
@@ -0,0 +1,165 @@
+# QingLong 3.0 Worker deployment baseline
+
+The Worker is an outbound-only, headless process. It opens one shared TLS 1.3
+client authority to the Cluster Worker ingress and owns one local execution
+journal. It does not expose an HTTP Service, database connection or Kubernetes
+API credential.
+
+The committed base intentionally references, but does not create,
+`ql3-worker-config`, `ql3-worker-identity` or the mutable credential target.
+The separate `credential-bootstrap` asset creates an empty, labelled
+`ql3-worker-credential` exactly once so the delivery authority needs only
+exact-name `get` and `update`, never broad Secret creation in the Worker
+namespace. Copy the ConfigMap and TLS identity examples into a private overlay,
+replace every placeholder and use a separately issued Worker ID, certificate,
+`ql3w` token and PVC for every deployed Worker. Do not put the `ql3w` token in
+the TLS identity Secret.
+
+Never scale this Deployment above one replica. A replica is not a stateless
+copy: the Worker ID, Session, credential, journal, logs, receipts and running
+POSIX processes form one authority. Horizontal capacity is added by deploying
+another independently named Worker with another identity and volume.
+
+Build the image from the repository root:
+
+```bash
+node scripts/ql3-worker-image-lock.cjs
+docker build \
+  --file deploy/containers/ql3-worker/Dockerfile \
+  --tag qinglong3-worker:3.0.0-alpha.0 \
+  --build-arg SOURCE_REVISION="$(git rev-parse HEAD)" \
+  .
+```
+
+The builder and runtime dependency roots have separate npm v3 locks generated
+from the reviewed workspace `pnpm-lock.yaml`. The runtime image contains only
+`runtime-core`, `local-process`, `worker-runtime` and their 24 external runtime
+packages. The bundled durable launcher is copied explicitly and made
+executable; no QingLong 2.x server, UI, SQLite or PostgreSQL package is present.
+
+Each Worker identity needs two namespaces: one Worker namespace containing only
+that identity's Deployment, PVC, ConfigMap, TLS Secret and credential target,
+and a separate staging namespace containing only immutable credential delivery
+stages. Never deploy either into the shared `qinglong3-system` namespace. A
+single Worker namespace is not sufficient because the stage authority must not
+be able to list the TLS private-key Secret.
+
+Before applying, create both namespaces and materialize the private
+ConfigMap/TLS Secret. The committed names are `qinglong3-worker` and
+`qinglong3-worker-credential-staging`; a private overlay must replace both for
+every independently named Worker. It must also replace
+`qinglong.io/worker-identity-generation` with a stable, non-secret generation
+identifier derived by the operator's release process. Then apply either the
+edge base or the node resource overlay and the credential-admin RBAC. Render
+the private credential-bootstrap overlay and submit it with create-only
+semantics before the first delivery:
+
+```bash
+kubectl apply -k deploy/kubernetes/ql3-worker/base
+# or:
+kubectl apply -k deploy/kubernetes/ql3-worker/overlays/node
+kubectl apply -k deploy/kubernetes/ql3-worker/credential-admin
+kubectl kustomize deploy/kubernetes/ql3-worker/credential-bootstrap \
+  | kubectl create -f -
+```
+
+An `AlreadyExists` response is a safety stop, not an instruction to apply,
+replace or delete the target. The normal base/GitOps reconciliation set must
+exclude this mutable Secret; if an external reconciler imports it, configure
+that reconciler to ignore the target data, labels and annotations owned by
+credential delivery. Never use server-side apply to continuously reconcile the
+bootstrap manifest.
+
+The first Pod remains pending until credential delivery populates the prepared
+`ql3-worker-credential`. Publication is not complete after only replacing that
+Secret. `WorkerCredentialKubernetesDeliveryAdapter` stages immutable Secrets in
+the separate staging namespace, then reads the exact
+single-replica `Recreate` Deployment, verifies that it projects the dedicated
+Secret, and replaces the PodTemplate with delivery ID, credential ID,
+generation, token digest and publication digest annotations under the same
+`resourceVersion` fence. A crash between Secret and Deployment updates is
+replayed from the durable delivery ledger; the adapter returns success only
+after the PodTemplate converges. It owns no watcher, timer, controller loop or
+Worker-side Kubernetes credential.
+
+Run a delivery through
+`createWorkerCredentialKubernetesKubeConfigTokenRequestSession`, using an
+external OIDC/client-certificate identity mapped to the dedicated
+`qinglong:worker-credential-operators` Kubernetes group. That group can only
+create the `serviceaccounts/token` subresource for the exact delivery
+ServiceAccount; it cannot use the delivery RBAC itself. The session requests a
+TokenRequest credential, not an automatically mounted or persisted
+ServiceAccount token:
+
+```bash
+kubectl -n qinglong3-worker-credential-staging create token \
+  ql3-worker-credential-admin --duration=10m
+```
+
+The command above is an operator troubleshooting equivalent, not the product
+execution path: it prints the bearer token and therefore must never be used by
+automation. The production session receives only the issuer kubeconfig,
+constructs the delivery client in memory, validates the issued JWT subject and
+at-most-600-second lifetime, runs 8 required-allow and 20 required-deny
+SelfSubjectAccessReview checks before mutation, and invalidates the restricted
+client in `finally`. It never returns the token or adapter. The ServiceAccount
+has `automountServiceAccountToken: false`. In the staging namespace it can only
+`get/list/create/delete` Secrets.
+In the Worker namespace it can only `get/update` the exact prepared credential
+Secret and exact Worker Deployment. It cannot read the TLS Secret, list Worker
+Secrets, access Pods, create a target Secret, patch/delete workloads, mint a
+replacement token or inspect cluster-scoped resources. Stage `create` and
+`list` cannot safely be narrowed by `resourceNames`, which is why the empty
+staging namespace is a required isolation boundary.
+
+The issuer kubeconfig remains external operator authority and must use strong,
+short-lived authentication; do not bind the token-issuer Role to the delivery
+ServiceAccount, Worker Pod, control Pod or a broad system group. Approval and
+durable product command binding are separate management-plane gates and are not
+implied by possession of this Kubernetes role.
+
+The projected TLS Secret, credential Secret and ConfigMap are not consumed directly. Kubernetes
+projects them through symlinks and group-readable modes, while the Worker
+credential boundary requires direct files, a `0700` parent and `0400` private
+material. The non-root init container creates its own `private/` subdirectory
+inside the fsGroup-writable tmpfs, copies the bounded inputs there and
+initializes `0700` recovery directories on the PVC before the main process
+starts. It never relies on changing the root-owned mount-point mode.
+
+CA/key/certificate rotation uses the same `Recreate` boundary but remains an
+explicit deployment operation: update `ql3-worker-identity`, advance
+`qinglong.io/worker-identity-generation` in the private overlay and wait for the
+old Pod to terminate before accepting the replacement Session. Never reuse the
+credential adapter as TLS-key authority; separating the Secrets prevents its
+least-privilege API client from reading the Worker private key.
+
+The PVC is mandatory. Replacing it with `emptyDir` can erase the only evidence
+needed to decide whether a process was spawned or completed after a restart.
+The Pod uses `Recreate` and a 360-second termination grace so the node profile's
+five-minute drain can keep heartbeating and settle completion evidence. An
+incomplete drain deliberately does not exit successfully.
+
+The process entrypoint itself retains one ref'ed lifecycle handle while it
+waits for SIGINT/SIGTERM. A pending JavaScript Promise alone does not keep Node
+alive; removing this handle makes an otherwise active Worker exit with code 0
+and puts an `Always`-restart Pod into `CrashLoopBackOff`. The handle is cleared
+with signal authority after proof-bearing stop, does not poll Kubernetes and
+does not replace the application's single execution cadence.
+
+There is no synthetic Kubernetes readiness or liveness probe. A running PID
+does not prove that startup reconciliation completed or that the Worker Session
+is schedulable, and an automatic liveness kill can destroy the only drain
+owner. Operational readiness is the current durable Worker Session observed
+through the Cluster control plane.
+
+The manual `ql3-worker-kubernetes-rollout-live` workflow and
+`qinglong/worker-kubernetes-rollout-live-contract@v2` prove the current Worker
+image through three distinct Pod/Session generations, per-Session heartbeat,
+credential and mTLS identity Recreate, startup reconciliation, shared PVC and
+graceful offline transition. This single-node K3s/local-path evidence is not a
+claim about multi-node CSI detach/attach or physical node loss.
+
+For routers or other small non-Kubernetes devices, run the same
+`ql3-worker` binary with `QL3_WORKER_CAPACITY_PROFILE=edge`, one concurrent run,
+the 2-second cadence defaults and private persistent directories. Do not install
+the Cluster PostgreSQL, control-plane, admin or UI closure on that device.
@@ -0,0 +1,176 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ql3-worker
+  labels:
+    app.kubernetes.io/name: ql3-worker
+    app.kubernetes.io/component: worker
+    app.kubernetes.io/part-of: qinglong3
+spec:
+  replicas: 1
+  minReadySeconds: 10
+  revisionHistoryLimit: 2
+  progressDeadlineSeconds: 600
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: ql3-worker
+      app.kubernetes.io/component: worker
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: ql3-worker
+        app.kubernetes.io/component: worker
+        app.kubernetes.io/part-of: qinglong3
+      annotations:
+        # A private overlay advances this value with every CA/key/cert
+        # generation. Credential delivery owns separate qinglong.io/*
+        # annotations and advances them with resourceVersion CAS.
+        qinglong.io/worker-identity-generation: replace-in-private-overlay
+    spec:
+      automountServiceAccountToken: false
+      terminationGracePeriodSeconds: 360
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65532
+        runAsGroup: 65532
+        fsGroup: 65532
+        fsGroupChangePolicy: OnRootMismatch
+        seccompProfile:
+          type: RuntimeDefault
+      initContainers:
+        - name: materialize-worker-authority
+          image: qinglong3-worker:3.0.0-alpha.0
+          imagePullPolicy: IfNotPresent
+          command:
+            - /bin/sh
+            - -ec
+          args:
+            - |
+              umask 077
+              mkdir -p /authority/private /state/journal /state/logs /state/receipts /state/identity
+              chmod 0700 /authority/private /state/journal /state/logs /state/receipts /state/identity
+              cp /projected/ca.crt /authority/private/ca.crt
+              cp /projected/tls.key /authority/private/tls.key
+              cp /projected/tls.crt /authority/private/tls.crt
+              cp /projected/credential-token /authority/private/credential-token
+              cp /projected/capabilities.json /authority/private/capabilities.json
+              chmod 0400 /authority/private/ca.crt /authority/private/tls.key /authority/private/tls.crt
+              chmod 0400 /authority/private/credential-token /authority/private/capabilities.json
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+          resources:
+            requests:
+              cpu: 5m
+              memory: 16Mi
+            limits:
+              cpu: 100m
+              memory: 64Mi
+          volumeMounts:
+            - name: projected-authority
+              mountPath: /projected
+              readOnly: true
+            - name: materialized-authority
+              mountPath: /authority
+            - name: worker-state
+              mountPath: /state
+      containers:
+        - name: worker
+          image: qinglong3-worker:3.0.0-alpha.0
+          imagePullPolicy: IfNotPresent
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+          env:
+            - name: QL_DEPLOYMENT_PROFILE
+              value: worker
+            - name: QL3_WORKER_RUNTIME_ENABLED
+              value: "true"
+            - name: QL3_WORKER_ID
+              valueFrom:
+                configMapKeyRef:
+                  name: ql3-worker-config
+                  key: worker-id
+            - name: QL3_WORKER_CONTROL_ORIGIN
+              valueFrom:
+                configMapKeyRef:
+                  name: ql3-worker-config
+                  key: control-origin
+            - name: QL3_WORKER_CAPACITY_PROFILE
+              value: edge
+            - name: QL3_WORKER_CAPABILITIES_FILE
+              value: /var/run/qinglong3-worker/private/capabilities.json
+            - name: QL3_WORKER_JOURNAL_ROOT
+              value: /var/lib/qinglong3-worker/journal
+            - name: QL3_WORKER_LOG_ROOT
+              value: /var/lib/qinglong3-worker/logs
+            - name: QL3_WORKER_RECEIPT_ROOT
+              value: /var/lib/qinglong3-worker/receipts
+            - name: QL3_WORKER_CERTIFICATE_STORE_ROOT
+              value: /var/lib/qinglong3-worker/identity
+            - name: QL3_WORKER_TRUST_ANCHOR_FILE
+              value: /var/run/qinglong3-worker/private/ca.crt
+            - name: QL3_WORKER_CREDENTIAL_TOKEN_FILE
+              value: /var/run/qinglong3-worker/private/credential-token
+            - name: QL3_WORKER_IDENTITY_BOOTSTRAP_PRIVATE_KEY_FILE
+              value: /var/run/qinglong3-worker/private/tls.key
+            - name: QL3_WORKER_IDENTITY_BOOTSTRAP_CERTIFICATE_FILE
+              value: /var/run/qinglong3-worker/private/tls.crt
+          resources:
+            requests:
+              cpu: 25m
+              memory: 64Mi
+            limits:
+              cpu: 500m
+              memory: 256Mi
+          volumeMounts:
+            - name: materialized-authority
+              mountPath: /var/run/qinglong3-worker
+              readOnly: true
+            - name: worker-state
+              mountPath: /var/lib/qinglong3-worker
+            - name: tmp
+              mountPath: /tmp
+      volumes:
+        - name: projected-authority
+          projected:
+            defaultMode: 288
+            sources:
+              - secret:
+                  name: ql3-worker-identity
+                  items:
+                    - key: ca.crt
+                      path: ca.crt
+                    - key: tls.key
+                      path: tls.key
+                    - key: tls.crt
+                      path: tls.crt
+              - secret:
+                  name: ql3-worker-credential
+                  items:
+                    - key: credential-token
+                      path: credential-token
+              - configMap:
+                  name: ql3-worker-config
+                  items:
+                    - key: capabilities.json
+                      path: capabilities.json
+        - name: materialized-authority
+          emptyDir:
+            medium: Memory
+            sizeLimit: 4Mi
+        - name: worker-state
+          persistentVolumeClaim:
+            claimName: ql3-worker-state
+        - name: tmp
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
@@ -0,0 +1,8 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: qinglong3-worker
+
+resources:
+  - deployment.yaml
+  - persistent-volume-claim.yaml
@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: ql3-worker-state
+  labels:
+    app.kubernetes.io/name: ql3-worker
+    app.kubernetes.io/component: worker
+    app.kubernetes.io/part-of: qinglong3
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 2Gi
@@ -0,0 +1,20 @@
+# Example only. Copy into a private overlay and bind the exact architecture,
+# runtime inventory and control-plane origin of this one Worker identity.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ql3-worker-config
+  namespace: qinglong3-worker
+data:
+  worker-id: REPLACE_WITH_UNIQUE_WORKER_ID
+  control-origin: https://ql3-cluster-control.qinglong3-system.svc:5801
+  capabilities.json: |
+    {
+      "architecture": "arm64",
+      "operatingSystem": "linux",
+      "executors": ["local_process"],
+      "runtimes": [{"name": "node", "version": "24.18.0"}],
+      "labels": {"site": "REPLACE_WITH_SITE"},
+      "capacity": {"cpuCores": 1, "memoryBytes": 268435456},
+      "features": []
+    }
@@ -0,0 +1,11 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - service-account.yaml
+  - stage-role.yaml
+  - stage-role-binding.yaml
+  - target-role.yaml
+  - target-role-binding.yaml
+  - token-issuer-role.yaml
+  - token-issuer-role-binding.yaml
@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: ql3-worker-credential-admin
+  namespace: qinglong3-worker-credential-staging
+automountServiceAccountToken: false
@@ -0,0 +1,13 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: ql3-worker-credential-stage-admin
+  namespace: qinglong3-worker-credential-staging
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: ql3-worker-credential-stage-admin
+subjects:
+  - kind: ServiceAccount
+    name: ql3-worker-credential-admin
+    namespace: qinglong3-worker-credential-staging
@@ -0,0 +1,15 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: ql3-worker-credential-stage-admin
+  namespace: qinglong3-worker-credential-staging
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - secrets
+    verbs:
+      - get
+      - list
+      - create
+      - delete
@@ -0,0 +1,13 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: ql3-worker-credential-target-admin
+  namespace: qinglong3-worker
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: ql3-worker-credential-target-admin
+subjects:
+  - kind: ServiceAccount
+    name: ql3-worker-credential-admin
+    namespace: qinglong3-worker-credential-staging
@@ -0,0 +1,24 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: ql3-worker-credential-target-admin
+  namespace: qinglong3-worker
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - secrets
+    resourceNames:
+      - ql3-worker-credential
+    verbs:
+      - get
+      - update
+  - apiGroups:
+      - apps
+    resources:
+      - deployments
+    resourceNames:
+      - ql3-worker
+    verbs:
+      - get
+      - update
@@ -0,0 +1,13 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: ql3-worker-credential-token-issuer
+  namespace: qinglong3-worker-credential-staging
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: ql3-worker-credential-token-issuer
+subjects:
+  - apiGroup: rbac.authorization.k8s.io
+    kind: Group
+    name: qinglong:worker-credential-operators
@@ -0,0 +1,14 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: ql3-worker-credential-token-issuer
+  namespace: qinglong3-worker-credential-staging
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - serviceaccounts/token
+    resourceNames:
+      - ql3-worker-credential-admin
+    verbs:
+      - create
@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ql3-worker-credential
+  labels:
+    app.kubernetes.io/name: ql3-worker
+    app.kubernetes.io/component: worker
+    app.kubernetes.io/part-of: qinglong3
+    app.kubernetes.io/managed-by: qinglong3
+    qinglong.io/worker-credential-target: prepared-v3
+type: Opaque
+data: {}
@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: qinglong3-worker
+
+resources:
+  - credential-secret.yaml
@@ -0,0 +1,22 @@
+# Example only. Populate through a Secret manager and never commit a real
+# Worker private key. The ql3w credential is delivered into the separate
+# ql3-worker-credential Secret by the short-lived cluster-admin adapter.
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ql3-worker-identity
+  namespace: qinglong3-worker
+type: Opaque
+stringData:
+  ca.crt: |
+    -----BEGIN CERTIFICATE-----
+    REPLACE_WITH_WORKER_INGRESS_CLIENT_CA
+    -----END CERTIFICATE-----
+  tls.key: |
+    -----BEGIN PRIVATE KEY-----
+    REPLACE_WITH_THIS_WORKER_PRIVATE_KEY
+    -----END PRIVATE KEY-----
+  tls.crt: |
+    -----BEGIN CERTIFICATE-----
+    REPLACE_WITH_THIS_WORKER_CERTIFICATE_CHAIN
+    -----END CERTIFICATE-----
@@ -0,0 +1,19 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ql3-worker
+spec:
+  template:
+    spec:
+      containers:
+        - name: worker
+          env:
+            - name: QL3_WORKER_CAPACITY_PROFILE
+              value: node
+          resources:
+            requests:
+              cpu: 250m
+              memory: 256Mi
+            limits:
+              cpu: "2"
+              memory: 1Gi
@@ -0,0 +1,8 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - ../../base
+
+patches:
+  - path: deployment-patch.yaml