Compare commits

...
Author SHA1 Message Date
whyour ac8567e537 fix(runtime): resolve develop merge and restore frozen lockfile compatibility 2026-10-13 02:20:59 +08:00
whyour be877cfddc fix: 修复完整镜像源地址及 CRLF 任务导入 (#3093)
* fix: address six verified legacy issue regressions

* fix: limit issue patch to mirror URLs and CRLF imports

* fix: preserve APT formats and third-party repositories
2026-10-13 01:11:35 +08:00
Copilotwhyourcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>whyour
fdeffebba8 Fix catch-all route serving HTML for API requests (#2885)
* Initial plan

* Fix env.js loading error by preventing catch-all from matching API routes

Co-authored-by: whyour <22700758+whyour@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: whyour <22700758+whyour@users.noreply.github.com>
Co-authored-by: whyour <imwhyour@gmail.com>
2026-10-13 01:07:50 +08:00
dependabot[bot]dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>whyour
3c61e933a1 chore(deps): bump the npm_and_yarn group across 1 directory with 8 updates (#3075)
* chore(deps): bump the npm_and_yarn group across 1 directory with 8 updates

Bumps the npm_and_yarn group with 8 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [body-parser](https://github.com/expressjs/body-parser) | `1.20.3` | `1.20.6` |
| [http-proxy-middleware](https://github.com/chimurai/http-proxy-middleware) | `3.0.3` | `3.0.7` |
| [js-yaml](https://github.com/nodeca/js-yaml) | `4.1.0` | `4.3.2` |
| [multer](https://github.com/expressjs/multer) | `2.3.0` | `2.4.0` |
| [nodemailer](https://github.com/nodemailer/nodemailer) | `8.0.1` | `10.0.9` |
| [undici](https://github.com/nodejs/undici) | `7.9.0` | `7.29.1` |
| [axios](https://github.com/axios/axios) | `1.7.8` | `1.20.0` |
| [brace-expansion](https://github.com/juliangruber/brace-expansion) | `1.1.12` | `1.1.21` |



Updates `body-parser` from 1.20.3 to 1.20.6
- [Release notes](https://github.com/expressjs/body-parser/releases)
- [Changelog](https://github.com/expressjs/body-parser/blob/master/HISTORY.md)
- [Commits](https://github.com/expressjs/body-parser/compare/1.20.3...1.20.6)

Updates `http-proxy-middleware` from 3.0.3 to 3.0.7
- [Release notes](https://github.com/chimurai/http-proxy-middleware/releases)
- [Changelog](https://github.com/chimurai/http-proxy-middleware/blob/v3.0.7/CHANGELOG.md)
- [Commits](https://github.com/chimurai/http-proxy-middleware/compare/v3.0.3...v3.0.7)

Updates `js-yaml` from 4.1.0 to 4.3.2
- [Changelog](https://github.com/nodeca/js-yaml/blob/4.3.2/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/compare/4.1.0...4.3.2)

Updates `multer` from 2.3.0 to 2.4.0
- [Release notes](https://github.com/expressjs/multer/releases)
- [Changelog](https://github.com/expressjs/multer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/expressjs/multer/compare/v2.3.0...v2.4.0)

Updates `nodemailer` from 8.0.1 to 10.0.9
- [Release notes](https://github.com/nodemailer/nodemailer/releases)
- [Changelog](https://github.com/nodemailer/nodemailer/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodemailer/nodemailer/compare/v8.0.1...v10.0.9)

Updates `undici` from 7.9.0 to 7.29.1
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v7.9.0...v7.29.1)

Updates `axios` from 1.7.8 to 1.20.0
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v1.7.8...v1.20.0)

Updates `brace-expansion` from 1.1.12 to 1.1.21
- [Release notes](https://github.com/juliangruber/brace-expansion/releases)
- [Commits](https://github.com/juliangruber/brace-expansion/compare/v1.1.12...v1.1.21)

---
updated-dependencies:
- dependency-name: axios
  dependency-version: 1.18.0
  dependency-type: direct:development
- dependency-name: body-parser
  dependency-version: 1.20.6
  dependency-type: direct:production
- dependency-name: brace-expansion
  dependency-version: 1.1.21
  dependency-type: indirect
- dependency-name: http-proxy-middleware
  dependency-version: 3.0.7
  dependency-type: direct:production
- dependency-name: js-yaml
  dependency-version: 4.3.2
  dependency-type: direct:production
- dependency-name: multer
  dependency-version: 2.3.0
  dependency-type: direct:production
- dependency-name: nodemailer
  dependency-version: 9.1.1
  dependency-type: direct:production
- dependency-name: undici
  dependency-version: 7.29.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix(ci): verify the updated multer runtime version

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: whyour <imwhyour@gmail.com>
2026-10-13 01:06:29 +08:00
whyour 59979ee061 fix(build): pin frontend compilation to Node 22 on newer hosts 2026-10-12 01:34:21 +08:00
whyour 449a8a471c chore(runtime): reconcile future migration with current develop 2026-10-12 01:13:53 +08:00
whyour 70636900b2 chore: finalize 2.23.0 publication metadata 2026-10-11 12:27:51 +08:00
whyour 331f4f9138 Prepare Qinglong 2.23.0 and CLI 0.1.2 release (#3090)
* fix: patch reachable release-blocking dependencies

* docs: keep release risk notes in PR description

* chore: prepare Qinglong 2.23.0 and CLI 0.1.2 release metadata
2026-10-11 12:24:50 +08:00
whyour b06cb72334 fix: apply minimal production dependency security patches 2026-10-11 02:09:20 +08:00
whyour 89c6355635 fix: renew system tokens and serialize authentication cache updates 2026-10-11 01:56:50 +08:00
whyour 4b4e485305 ci: probe legacy pnpm without unsupported config options 2026-10-10 06:10:58 +08:00
whyour 2f359c6bf0 fix: freeze release runtime tools and npm consumer artifacts 2026-10-10 06:02:39 +08:00
whyour a70ec66fe3 fix: prepare supported runtime images for 2.23.0 2026-10-10 03:17:52 +08:00
whyour 5038d2d9a0 fix: patch production dependencies and add release validation 2026-10-10 01:16:07 +08:00
whyour 68ffa106f8 fix: resolve Debian regression findings (#3087)
* fix: resolve Debian regression findings

* fix: reject failed APT index refreshes and simplify SSH home lookup
2026-10-09 01:41:29 +08:00
whyour 9d5f58ef99 ci: verify Gitee Pull mirrors with resilient GitHub notifications
Use native Gitee Pull mirrors for code and static artifacts, and verify branch and tag SHAs alongside image builds. Trigger stale mirrors through small GitHub REST ref updates, retry transient errors with backoff, and keep polling when a notification fails.
2026-10-08 01:16:43 +08:00
whyour ef1c54a620 fix: 补齐前端缺失的中英文翻译 2026-10-07 20:35:59 +08:00
whyour 0218404d9f fix: 修复页面升级、ql update 与静态资源版本同步 (#3086)
* fix: validate panel upgrades and synchronize versioned static builds

* fix: add static-only mirror backfill workflow

* fix: avoid lifecycle self-kill and stop cancelled upgrade groups

* fix: build images when the shared upgrade engine changes

* fix: dispatch upgrade lifecycle through a fixed script

* fix: reclaim upgrade locks safely and normalize health probes

* fix: match backend base URL normalization exactly
2026-10-07 17:43:03 +08:00
whyour 24f7d1a46f fix: resolve dev panel and CLI regressions
Synchronize scheduler concurrency across API and gRPC processes and restore persisted limits during recovery.

Serialize subscription mutations and preserve shared storage while cleaning Shell and CLI namespaces, historical references, and filesystem-specific naming candidates.

Handle CLI text responses correctly and include script history in backups. Add regression coverage for concurrency, cleanup ownership, long URLs, Unicode paths, and history exports.
2026-10-07 12:35:10 +08:00
whyour fa8910b3d6 feat: add script version history (#3085)
* feat: add script version history

* fix: validate canonical save-as paths before backup

* fix: stack script history footer on small screens
2026-10-06 22:00:11 +08:00
whyour 2cab112391 fix: 加固安全输入边界并修复订阅兼容性 (#3084)
* fix: harden scoped API inputs and isolate login throttling

* fix: close script path gaps and harden security boundaries

* fix: preserve subscription aliases and avoid extension eval

* fix: allow cancelling invalid historical subscriptions

* fix: make script parent validation explicit for CodeQL

* fix: limit command execution and clarify trusted path branches

* fix: separate diff tool script access from config APIs

* fix: clean up obsolete subscription SSH credentials
2026-10-06 18:21:54 +08:00
whyour 2ab9c702e6 ci: skip Docker builds for CLI-only changes 2026-10-01 12:25:16 +08:00
whyour 6dd98a8565 fix(ci): isolate CLI release concurrency from validation 2026-10-01 12:17:26 +08:00
whyour 06c8d8b112 fix: report resource conflicts and accurate CLI API errors 2026-10-01 11:26:40 +08:00
whyour a5406bedca ci: allow verified CLI prereleases from develop 2026-10-01 00:26:42 +08:00
whyour 3d2b98a442 fix: scope file access restrictions and honor custom data directories 2026-09-30 21:21:35 +08:00
whyour 93d7ec69dc fix: scope config file blacklist to config directory (#3082) 2026-09-30 20:31:20 +08:00
whyour a303e33b4e fix: invalidate queued cron revisions and enforce task process timeouts 2026-09-29 23:37:00 +08:00
whyour 44129ca088 fix: align CLI shell completion and cancellation semantics 2026-09-29 01:06:47 +08:00
whyour def4917447 fix: finalize exited shell tasks and preserve command environments 2026-09-29 00:52:33 +08:00
whyour bc0f35e3f7 fix: reduce task startup overhead and lazy-load notifications (#3080) 2026-09-28 23:41:31 +08:00
whyour f168efcdae refactor: migrate cron scheduling to node-cron (#3079)
* refactor: migrate cron scheduling to node-cron

* feat: support annually midnight and minutely cron macros

* fix: harden cron recovery and system scheduler compatibility
2026-09-27 22:18:39 +08:00
217 changed files with 58276 additions and 10168 deletions
+230 -106
View File
@@ -2,23 +2,48 @@ name: Build And Push Docker Image
on:
push:
paths-ignore:
- "*.md"
paths:
- "**"
- "!*.md"
# CLI is verified separately; its shared Bash upgrade engine ships in images.
- "!cli/**"
- "cli/src/internal/maintenance/upgradeArtifacts.cjs"
- "!.github/workflows/cli-package.yml"
branches:
- "master"
- "develop"
tags:
- "v*"
workflow_dispatch:
inputs:
static_sync_only:
description: Synchronize an existing static snapshot to GitLab and verify Gitee
type: boolean
default: false
static_ref:
description: Static branch or version name, for example v2.22.0
type: string
default: ''
static_ref_type:
description: Synchronize a branch, or a version branch and Git tag
type: choice
options: [branch, tag]
default: branch
permissions:
contents: read
jobs:
validate:
if: ${{ !inputs.static_sync_only }}
uses: ./.github/workflows/validate.yml
runtime-tools:
if: ${{ !inputs.static_sync_only }}
uses: ./.github/workflows/runtime-tools.yml
code_gitlab:
if: ${{ !inputs.static_sync_only }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
@@ -35,48 +60,19 @@ jobs:
git push --force --all gitlab 2>&1 || echo "::warning::GitLab push failed"
git push --force --tags gitlab 2>&1 || echo "::warning::GitLab tags push failed"
code_gitee:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Push to Gitee
env:
GITEE_TOKEN: ${{ secrets.GITEE_TOKEN }}
run: |
set +e
mkdir -p ~/.ssh
printf '%s\n' "${{ secrets.GITLAB_SSH_PK }}" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan -T 10 gitee.com >> ~/.ssh/known_hosts 2>/dev/null
git remote add gitee git@gitee.com:whyour/qinglong.git 2>/dev/null
if git push --force --all gitee 2>&1; then
echo "::notice::Gitee push --all succeeded"
else
echo "::warning::Push failed, trying to create repo via API..."
curl -sS --connect-timeout 30 --max-time 60 \
-X POST "https://gitee.com/api/v5/user/repos" \
-H "Content-Type: application/json" \
-d '{"name":"qinglong","private":"false"}' \
"?access_token=$GITEE_TOKEN" 2>/dev/null
git push --force --all gitee 2>&1 && echo "::notice::Gitee push succeeded after repo creation" || echo "::warning::Gitee push failed after retry"
fi
git push --force --tags gitee 2>&1 || echo "::warning::Gitee tags push failed"
build-static:
needs: validate
if: ${{ !inputs.static_sync_only }}
needs: [validate, runtime-tools]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: pnpm/action-setup@v6
with:
version: "8.3.1"
version: "10.34.6"
- uses: actions/setup-node@v6
with:
node-version: "20"
node-version: "22"
cache: "pnpm"
cache-dependency-path: pnpm-lock.yaml
@@ -85,7 +81,12 @@ jobs:
pnpm install --frozen-lockfile
pnpm build:front
pnpm build:back
pnpm build:info
- uses: actions/download-artifact@v7
with:
name: qinglong-runtime-tools-${{ github.sha }}
path: static/
- run: pnpm build:info
- uses: actions/upload-artifact@v6
with:
@@ -98,62 +99,116 @@ jobs:
env:
GITHUB_REPO: github.com/${{ github.repository_owner }}/qinglong-static
GITHUB_BRANCH: ${{ github.ref_name }}
GITHUB_REF_TYPE: ${{ github.ref_type }}
run: |
mkdir -p tmp
cd ./tmp
cp -rf ../static/* ./
git init -b ${GITHUB_BRANCH} && git add .
git config --local user.name 'github-actions[bot]'
git config --local user.email 'github-actions[bot]@users.noreply.github.com'
git commit --allow-empty -m "copy static at $(date +'%Y-%m-%d %H:%M:%S')"
git push --force --quiet "https://${{ secrets.API_TOKEN }}@${GITHUB_REPO}.git" ${GITHUB_BRANCH}:${GITHUB_BRANCH}
bash scripts/publish-static.sh "https://${{ secrets.API_TOKEN }}@${GITHUB_REPO}.git" "$GITHUB_BRANCH" "$GITHUB_REF_TYPE"
npm-package:
if: ${{ !inputs.static_sync_only }}
needs: build-static
uses: ./.github/workflows/npm-package.yml
with:
static-artifact: qinglong-static-${{ github.sha }}
static_gitlab:
needs: build-static
runs-on: ubuntu-latest
concurrency:
group: static-gitlab-${{ github.ref_name }}
cancel-in-progress: false
steps:
- uses: actions/checkout@v6
- name: Push qinglong-static to GitLab
env:
STATIC_BRANCH: ${{ github.ref_name }}
STATIC_REF_TYPE: ${{ github.ref_type }}
MIRROR_SSH_KEY: ${{ secrets.GITLAB_SSH_PK }}
run: |
set +e
set -euo pipefail
mkdir -p ~/.ssh
printf '%s\n' "${{ secrets.GITLAB_SSH_PK }}" > ~/.ssh/id_rsa
printf '%s\n' "$MIRROR_SSH_KEY" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan -T 10 gitlab.com >> ~/.ssh/known_hosts 2>/dev/null
git clone --depth=1 --single-branch https://github.com/whyour/qinglong-static.git static-mirror
cd static-mirror
git remote add gitlab git@gitlab.com:whyour/qinglong-static.git 2>/dev/null
git push --force --all gitlab 2>&1 || echo "::warning::GitLab static push failed"
git push --force --tags gitlab 2>&1 || echo "::warning::GitLab static tags push failed"
bash scripts/sync-static.sh git@gitlab.com:whyour/qinglong-static.git "$STATIC_BRANCH" "$STATIC_REF_TYPE"
static_gitee:
verify-gitee-mirrors:
needs: build-static
runs-on: ubuntu-latest
timeout-minutes: 14
permissions:
contents: write
strategy:
fail-fast: false
matrix:
repository: [qinglong, qinglong-static]
steps:
- name: Push qinglong-static to Gitee
- uses: actions/checkout@v6
# Gitee Pull mirrors and GitHub push webhooks own synchronization.
# Verification runs alongside Docker builds and needs no Gitee credentials.
- name: Wait for Gitee Pull mirror
if: ${{ matrix.repository == 'qinglong' }}
env:
GITEE_TOKEN: ${{ secrets.GITEE_TOKEN }}
GH_TOKEN: ${{ github.token }}
QL_MIRROR_NOTIFY_REPO: whyour/qinglong
MIRROR_REPOSITORY: ${{ matrix.repository }}
MIRROR_REF: ${{ github.ref_name }}
MIRROR_REF_TYPE: ${{ github.ref_type }}
QL_MIRROR_INCLUDE_VERSION_BRANCH: ${{ matrix.repository == 'qinglong-static' }}
QL_MIRROR_NOTIFY_ON_START: 'false'
run: |
set +e
mkdir -p ~/.ssh
printf '%s\n' "${{ secrets.GITLAB_SSH_PK }}" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan -T 10 gitee.com >> ~/.ssh/known_hosts 2>/dev/null
git clone --depth=1 --single-branch https://github.com/whyour/qinglong-static.git static-mirror
cd static-mirror
git remote add gitee git@gitee.com:whyour/qinglong-static.git 2>/dev/null
bash scripts/wait-for-mirror.sh "https://github.com/whyour/$MIRROR_REPOSITORY.git" "https://gitee.com/whyour/$MIRROR_REPOSITORY.git" "$MIRROR_REF" "$MIRROR_REF_TYPE"
- name: Notify and wait for Gitee static mirror
if: ${{ matrix.repository == 'qinglong-static' }}
env:
GH_TOKEN: ${{ secrets.API_TOKEN }}
QL_MIRROR_NOTIFY_REPO: whyour/qinglong-static
MIRROR_REF: ${{ github.ref_name }}
MIRROR_REF_TYPE: ${{ github.ref_type }}
QL_MIRROR_INCLUDE_VERSION_BRANCH: 'true'
run: |
bash scripts/wait-for-mirror.sh https://github.com/whyour/qinglong-static.git https://gitee.com/whyour/qinglong-static.git "$MIRROR_REF" "$MIRROR_REF_TYPE"
if git push --force --all gitee 2>&1; then
echo "::notice::Gitee static push succeeded"
else
echo "::warning::Push failed, trying to create repo via API..."
curl -sS --connect-timeout 30 --max-time 60 \
-X POST "https://gitee.com/api/v5/user/repos" \
-H "Content-Type: application/json" \
-d '{"name":"qinglong-static","private":"false"}' \
"?access_token=$GITEE_TOKEN" 2>/dev/null
git push --force --all gitee 2>&1 && echo "::notice::Gitee static push succeeded after repo creation" || echo "::warning::Gitee static push failed after retry"
fi
git push --force --tags gitee 2>&1 || echo "::warning::Gitee static tags push failed"
sync-existing-static:
if: ${{ github.event_name == 'workflow_dispatch' && inputs.static_sync_only }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
mirror: [gitlab]
concurrency:
group: static-${{ matrix.mirror }}-${{ inputs.static_ref }}
cancel-in-progress: false
steps:
- uses: actions/checkout@v6
- name: Synchronize existing static snapshot
env:
STATIC_REF: ${{ inputs.static_ref }}
STATIC_REF_TYPE: ${{ inputs.static_ref_type }}
MIRROR_HOST: ${{ matrix.mirror }}.com
MIRROR_SSH_KEY: ${{ secrets.GITLAB_SSH_PK }}
run: |
set -euo pipefail
mkdir -p ~/.ssh
printf '%s\n' "$MIRROR_SSH_KEY" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan -T 10 "$MIRROR_HOST" >> ~/.ssh/known_hosts 2>/dev/null
bash scripts/sync-static.sh "git@$MIRROR_HOST:whyour/qinglong-static.git" "$STATIC_REF" "$STATIC_REF_TYPE"
verify-existing-static-gitee:
if: ${{ github.event_name == 'workflow_dispatch' && inputs.static_sync_only }}
runs-on: ubuntu-latest
timeout-minutes: 14
steps:
- uses: actions/checkout@v6
- name: Wait for existing Gitee static snapshot
env:
GH_TOKEN: ${{ secrets.API_TOKEN }}
QL_MIRROR_NOTIFY_REPO: whyour/qinglong-static
MIRROR_REF: ${{ inputs.static_ref }}
MIRROR_REF_TYPE: ${{ inputs.static_ref_type }}
QL_MIRROR_INCLUDE_VERSION_BRANCH: 'true'
run: |
bash scripts/wait-for-mirror.sh https://github.com/whyour/qinglong-static.git https://gitee.com/whyour/qinglong-static.git "$MIRROR_REF" "$MIRROR_REF_TYPE"
build-alpine:
if: ${{ !startsWith(github.ref, 'refs/tags/') }}
@@ -170,10 +225,10 @@ jobs:
path: static/
- uses: pnpm/action-setup@v6
with:
version: "8.3.1"
version: "10.34.6"
- uses: actions/setup-node@v6
with:
node-version: "20"
node-version: "22"
cache: "pnpm"
cache-dependency-path: pnpm-lock.yaml
@@ -256,10 +311,10 @@ jobs:
path: static/
- uses: pnpm/action-setup@v6
with:
version: "8.3.1"
version: "10.34.6"
- uses: actions/setup-node@v6
with:
node-version: "20"
node-version: "22"
cache: "pnpm"
cache-dependency-path: pnpm-lock.yaml
@@ -342,10 +397,10 @@ jobs:
path: static/
- uses: pnpm/action-setup@v6
with:
version: "8.3.1"
version: "10.34.6"
- uses: actions/setup-node@v6
with:
node-version: "20"
node-version: "22"
cache: "pnpm"
cache-dependency-path: pnpm-lock.yaml
@@ -414,10 +469,10 @@ jobs:
path: static/
- uses: pnpm/action-setup@v6
with:
version: "8.3.1"
version: "10.34.6"
- uses: actions/setup-node@v6
with:
node-version: "20"
node-version: "22"
cache: "pnpm"
cache-dependency-path: pnpm-lock.yaml
@@ -474,46 +529,115 @@ jobs:
publish:
if: ${{ github.repository == 'whyour/qinglong' && github.ref == 'refs/heads/master' }}
needs: [build-alpine, build-debian, build-alpine310, build-debian310]
needs: [build-alpine, build-debian, build-alpine310, build-debian310, npm-package]
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
id-token: write
env:
SOURCE_SHA: ${{ github.sha }}
VERIFIED_TARBALL_SHA256: ${{ needs.npm-package.outputs.tarball-sha256 }}
VERIFIED_PACKAGE_VERSION: ${{ needs.npm-package.outputs.package-version }}
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v3
- uses: actions/checkout@v6
with:
version: "8.3.1"
- uses: actions/setup-node@v3
ref: ${{ github.sha }}
- uses: actions/setup-node@v6
with:
node-version: "20"
cache: "pnpm"
- name: build front and back
run: |
pnpm install --frozen-lockfile
pnpm build:front
pnpm build:back
- name: Set up Node for trusted publishing
uses: actions/setup-node@v6
with:
node-version: '24'
registry-url: https://registry.npmjs.org
node-version: '22.23.3'
package-manager-cache: false
- name: Publish npm package with OIDC
- uses: actions/download-artifact@v7
with:
name: ${{ needs.npm-package.outputs.artifact-name }}
path: ${{ runner.temp }}/main-npm-publish/
- uses: actions/download-artifact@v7
with:
name: qinglong-runtime-tools-${{ github.sha }}
path: ${{ runner.temp }}/npm-publisher-runtime-tools/
- name: Verify the gate-approved archive and its source receipts
shell: bash
run: |
set -euo pipefail
node <<'NODE'
const fs = require('node:fs'), path = require('node:path'), crypto = require('node:crypto'), assert = require('node:assert/strict');
const { spawnSync } = require('node:child_process');
const hash = (bytes) => crypto.createHash('sha256').update(bytes).digest('hex');
const fileHash = (file) => hash(fs.readFileSync(file));
const directory = path.join(process.env.RUNNER_TEMP, 'main-npm-publish');
const summary = JSON.parse(fs.readFileSync(path.join(directory, 'package-build-summary.json')));
const pkg = JSON.parse(fs.readFileSync('package.json'));
assert.match(process.env.VERIFIED_TARBALL_SHA256 || '', /^[a-f0-9]{64}$/);
assert.equal(summary.sourceCommit, process.env.SOURCE_SHA); assert.equal(summary.previewMetadata, false);
assert.deepEqual(summary.sourceMetadataChanges, []); assert(summary.helperProvenance.length > 0);
assert(summary.helperProvenance.every((item) => item.matchesGitSource));
assert.equal(summary.package, pkg.name); assert.equal(summary.packageVersion, pkg.version);
assert.equal(summary.packageVersion, process.env.VERIFIED_PACKAGE_VERSION);
assert.equal(summary.productionManifestSHA256, fileHash('package.json')); assert.equal(summary.productionLockSHA256, fileHash('pnpm-lock.yaml'));
const source = spawnSync('git', ['archive', process.env.SOURCE_SHA], { maxBuffer: 128 * 1024 * 1024 });
assert.equal(source.status, 0); assert.equal(hash(source.stdout), summary.sourceArchiveSHA256);
assert.equal(summary.runtimeToolsIncluded, true); assert.equal(summary.runtimeTools.sourceCommit, process.env.SOURCE_SHA);
assert.equal(summary.runtimeTools.builder.platform, 'linux'); assert.equal(summary.runtimeTools.builder.releaseBuild, true);
assert.equal(path.basename(summary.tarball.filename), summary.tarball.filename); assert(summary.tarball.filename.endsWith('.tgz'));
assert.deepEqual(fs.readdirSync(directory).filter((name) => name.endsWith('.tgz')), [summary.tarball.filename]);
const tarball = path.join(directory, summary.tarball.filename);
assert.equal(summary.tarball.sha256, process.env.VERIFIED_TARBALL_SHA256);
assert.equal(fileHash(tarball), process.env.VERIFIED_TARBALL_SHA256); assert.equal(fs.statSync(tarball).size, summary.tarball.bytes);
const tar = (args) => {
const result = spawnSync('tar', args, { maxBuffer: 128 * 1024 * 1024 });
assert.equal(result.status, 0, 'Cannot read the approved archive'); return result.stdout;
};
const entries = tar(['-tzf', tarball]).toString().split('\n').filter(Boolean);
assert.equal(entries.length, new Set(entries).size);
for (const name of entries) assert(name.startsWith('package/') && !name.split('/').includes('..') && !name.endsWith('.node') && !name.split('/').some((part) => ['.npmrc', '.git', '.pnpm', '_cacache'].includes(part)), 'Forbidden archive item ' + name);
const read = (relative) => tar(['-xOf', tarball, 'package/' + relative]);
const receipt = JSON.parse(read('qinglong-npm-dependency-proof.json'));
const { tarball: ignoredTarball, ...producerReceipt } = summary; assert.deepEqual(receipt, producerReceipt);
assert.deepEqual(entries.map((name) => name.slice('package/'.length)).sort(), receipt.publishedFilePaths);
const published = JSON.parse(read('package.json')); assert.equal(published.name, pkg.name); assert.equal(published.version, pkg.version);
assert.equal(hash(read('package.json')), receipt.publishedRootManifestSHA256);
const buildInfo = read('static/build-info.json'), build = JSON.parse(buildInfo);
assert.equal(hash(buildInfo), summary.ciBuildInfoSHA256); assert.equal(build.sourceCommit, process.env.SOURCE_SHA); assert.equal(build.dirty, false);
assert.equal(build.lockfileSha256, summary.productionLockSHA256);
assert.deepEqual(entries.filter((name) => name.startsWith('package/static/')).map((name) => name.slice('package/static/'.length)).sort(), [...Object.keys(build.files), 'build-info.json'].sort());
const toolsRoot = path.join(process.env.RUNNER_TEMP, 'npm-publisher-runtime-tools');
for (const [name, expected] of [['runtime-tools.tgz', summary.runtimeTools.archiveSHA256], ['runtime-tools-proof.json', summary.runtimeTools.proofSHA256]]) {
assert.equal(fileHash(path.join(toolsRoot, name)), expected); assert.equal(hash(read('static/' + name)), expected); assert.equal(build.files[name], expected);
}
const tools = JSON.parse(fs.readFileSync(path.join(toolsRoot, 'runtime-tools-proof.json')));
assert.equal(tools.sourceCommit, process.env.SOURCE_SHA); assert.equal(tools.archive.sha256, summary.runtimeTools.archiveSHA256);
assert.equal(tools.builder.platform, 'linux'); assert.equal(tools.builder.releaseBuild, true);
assert.equal(tools.sourceAttestation.verified, true); assert.equal(tools.sourceAttestation.head, process.env.SOURCE_SHA); assert(tools.inputs.length > 0);
for (const input of tools.inputs) {
assert(input.path && !path.isAbsolute(input.path) && !input.path.split(/[\\/]/).includes('..'));
assert.equal(fileHash(input.path), input.sha256, 'Tool source input changed: ' + input.path);
}
fs.appendFileSync(process.env.GITHUB_ENV, 'QINGLONG_PUBLISH_TARBALL=' + tarball + '\n');
console.log(JSON.stringify({ sourceCommit: summary.sourceCommit, packageVersion: summary.packageVersion, tarballSHA256: summary.tarball.sha256, cleanSource: true, runtimeToolsIncluded: true }));
NODE
- name: Install the same fixed npm CLI for trusted publishing
run: |
node scripts/install-runtime-tools.cjs \
--archive "$RUNNER_TEMP/npm-publisher-runtime-tools/runtime-tools.tgz" \
--proof "$RUNNER_TEMP/npm-publisher-runtime-tools/runtime-tools-proof.json" \
--prefix "$RUNNER_TEMP/npm-publisher-tools"
- name: Publish the verified npm archive with OIDC
shell: bash
run: |
set -euo pipefail
npm_cli="$RUNNER_TEMP/npm-publisher-tools/lib/node_modules/npm/bin/npm-cli.js"
export NPM_CONFIG_USERCONFIG="$RUNNER_TEMP/npm-publish-public.npmrc"
export NPM_CONFIG_GLOBALCONFIG=/dev/null
export NPM_CONFIG_CACHE="$RUNNER_TEMP/npm-publish-cache"
printf 'registry=https://registry.npmjs.org/\n' > "$NPM_CONFIG_USERCONFIG"
test "$(node "$npm_cli" --version)" = '12.2.0'
name=$(node -p 'require("./package.json").name')
version=$(node -p 'require("./package.json").version')
if npm view "$name@$version" version --json --registry=https://registry.npmjs.org > "$RUNNER_TEMP/npm-version.json" 2> "$RUNNER_TEMP/npm-version-error.log"; then
if node "$npm_cli" view "$name@$version" version --json --registry=https://registry.npmjs.org > "$RUNNER_TEMP/npm-version.json" 2> "$RUNNER_TEMP/npm-version-error.log"; then
jq -e --arg version "$version" '. == $version' "$RUNNER_TEMP/npm-version.json" > /dev/null
echo "::notice::$name@$version is already published; skipping npm publication for this image rebuild."
elif jq -e '.error.code == "E404"' "$RUNNER_TEMP/npm-version.json" > /dev/null; then
npm publish --access public --registry=https://registry.npmjs.org
node "$npm_cli" publish "$QINGLONG_PUBLISH_TARBALL" --ignore-scripts --access public --registry=https://registry.npmjs.org
else
echo '::error::Could not check the published npm version; refusing to publish.'
exit 1
+54 -8
View File
@@ -21,16 +21,27 @@ on:
workflow_dispatch:
inputs:
publish:
description: Publish the verified CLI package (master only)
description: Publish the verified CLI package (master stable / develop prerelease)
type: boolean
default: false
prerelease_version:
description: Develop only, e.g. 0.1.2-beta.0 (alpha/beta/rc)
type: string
default: ''
permissions:
contents: read
concurrency:
group: cli-package-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}
# Keep validation runs separate from releases; never interrupt a running release.
# Master push releases share the same group as manually requested releases.
group: >-
cli-package-${{ github.workflow }}-${{ github.ref }}-${{
((github.event_name == 'workflow_dispatch' && inputs.publish) ||
(github.event_name == 'push' && github.ref == 'refs/heads/master')) && 'publish' || 'check' }}
cancel-in-progress: >-
${{ github.ref != 'refs/heads/master' &&
!(github.event_name == 'workflow_dispatch' && inputs.publish) }}
jobs:
package:
@@ -52,6 +63,20 @@ jobs:
sudo apt-get update
sudo apt-get install -y --no-install-recommends bash ca-certificates curl git jq perl procps python3 unzip zip
npm install --global --ignore-scripts --no-audit --no-fund ts-node@10.9.2 typescript@5.2.2
- name: Set requested prerelease version before verification
if: github.event_name == 'workflow_dispatch' && inputs.prerelease_version != ''
working-directory: cli
env:
PRERELEASE_VERSION: ${{ inputs.prerelease_version }}
shell: bash
run: |
set -euo pipefail
if [[ "$GITHUB_REF" != refs/heads/develop ||
! "$PRERELEASE_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+-(alpha|beta|rc)\.[0-9]+$ ]]; then
echo '::error::Prereleases require develop and an explicit alpha/beta/rc version.'
exit 1
fi
npm version "$PRERELEASE_VERSION" --no-git-tag-version --ignore-scripts
- run: npm ci --prefix cli --no-audit --no-fund
- run: npm run check:cli
- run: npm run test:cli
@@ -72,9 +97,11 @@ jobs:
needs: package
if: >-
github.repository == 'whyour/qinglong' &&
github.ref == 'refs/heads/master' &&
(github.event_name == 'push' ||
(github.event_name == 'workflow_dispatch' && inputs.publish))
((github.ref == 'refs/heads/master' &&
(github.event_name == 'push' ||
(github.event_name == 'workflow_dispatch' && inputs.publish))) ||
(github.ref == 'refs/heads/develop' && github.event_name == 'workflow_dispatch' &&
inputs.publish && inputs.prerelease_version != ''))
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
@@ -91,6 +118,8 @@ jobs:
name: qinglong-cli-${{ github.sha }}
path: cli-package
- name: Publish verified npm archive
env:
PRERELEASE_VERSION: ${{ inputs.prerelease_version }}
shell: bash
run: |
set -euo pipefail
@@ -103,13 +132,30 @@ jobs:
archive="${archives[0]}"
metadata=$(tar -xOf "$archive" package/package.json)
name=$(jq -er '.name | select(. == "@whyour/qinglong-cli")' <<< "$metadata")
version=$(jq -er '.version | select(type == "string" and test("^[0-9]+\\.[0-9]+\\.[0-9]+$"))' <<< "$metadata")
version=$(jq -er '.version | select(type == "string")' <<< "$metadata")
if [[ "$GITHUB_REF" == refs/heads/develop ]]; then
if [[ "$version" != "$PRERELEASE_VERSION" ||
! "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+-(alpha|beta|rc)\.[0-9]+$ ]]; then
echo '::error::Archive version does not match the requested prerelease.'
exit 1
fi
tag="${BASH_REMATCH[1]}"
elif [[ "$GITHUB_REF" == refs/heads/master && "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
tag=latest
else
echo '::error::Refusing to publish this branch/version combination.'
exit 1
fi
if npm view "$name@$version" version --json --registry=https://registry.npmjs.org > version.json 2> version-error.log; then
jq -e --arg version "$version" '. == $version' version.json > /dev/null
if [[ "$tag" != latest ]]; then
echo '::error::Prerelease version already exists; choose a new version to test this build.'
exit 1
fi
echo "::notice::$name@$version is already published; bump cli/package.json and its lockfile to release changes."
exit 0
elif ! jq -e '.error.code == "E404"' version.json > /dev/null; then
echo '::error::Could not check the published CLI version; refusing to publish.'
exit 1
fi
npm publish "./$archive" --access public --tag latest --ignore-scripts --registry=https://registry.npmjs.org
npm publish "./$archive" --access public --tag "$tag" --ignore-scripts --registry=https://registry.npmjs.org
+312
View File
@@ -0,0 +1,312 @@
name: Verify main npm archive
on:
workflow_call:
inputs:
static-artifact:
description: Complete static artifact including the same-source runtime tools
required: true
type: string
outputs:
artifact-name:
description: Main npm archive and producer summary, after every consumer passes
value: ${{ jobs.verified.outputs.artifact-name }}
tarball-sha256:
description: SHA256 of the exact archive consumed by all six cases
value: ${{ jobs.verified.outputs.tarball-sha256 }}
package-version:
description: Source package version verified against the archive
value: ${{ jobs.verified.outputs.package-version }}
permissions:
contents: read
env:
SOURCE_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
jobs:
archive:
runs-on: ubuntu-latest
timeout-minutes: 20
outputs:
artifact-name: ${{ steps.archive.outputs.artifact-name }}
tarball-sha256: ${{ steps.archive.outputs.tarball-sha256 }}
package-version: ${{ steps.archive.outputs.package-version }}
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- uses: actions/setup-node@v6
with:
node-version: '22.23.3'
package-manager-cache: false
- uses: actions/download-artifact@v7
with:
name: ${{ inputs.static-artifact }}
path: static/
- name: Verify tool artifact source before executing the producer tools
shell: bash
run: |
set -euo pipefail
node <<'NODE'
const fs = require('node:fs'), crypto = require('node:crypto'), assert = require('node:assert/strict');
const hash = (file) => crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex');
const build = JSON.parse(fs.readFileSync('static/build-info.json'));
const tools = JSON.parse(fs.readFileSync('static/runtime-tools-proof.json'));
assert.equal(build.sourceCommit, process.env.SOURCE_SHA); assert.equal(build.dirty, false);
assert.equal(build.lockfileSha256, hash('pnpm-lock.yaml'));
assert.equal(tools.sourceCommit, process.env.SOURCE_SHA);
assert.equal(tools.builder.platform, 'linux'); assert.equal(tools.builder.releaseBuild, true);
assert.equal(tools.sourceAttestation.verified, true); assert.equal(tools.sourceAttestation.head, process.env.SOURCE_SHA);
for (const name of ['runtime-tools.tgz', 'runtime-tools-proof.json']) assert.equal(hash(`static/${name}`), build.files[name]);
assert.equal(hash('static/runtime-tools.tgz'), tools.archive.sha256);
for (const input of tools.inputs) {
assert(input.path && !input.path.startsWith('/') && !input.path.split(/[\\/]/).includes('..'));
assert.equal(hash(input.path), input.sha256, `Runtime source input changed: ${input.path}`);
}
NODE
- name: Install fixed producer tools into an isolated prefix
run: |
node scripts/install-runtime-tools.cjs \
--archive "$GITHUB_WORKSPACE/static/runtime-tools.tgz" \
--proof "$GITHUB_WORKSPACE/static/runtime-tools-proof.json" \
--prefix "$RUNNER_TEMP/npm-producer-tools"
- name: Export the single frozen production graph
run: |
node scripts/build-npm-package.cjs \
--repo "$GITHUB_WORKSPACE" \
--source-ref "$SOURCE_SHA" \
--artifact-dir "$GITHUB_WORKSPACE/static" \
--pnpm-cli "$RUNNER_TEMP/npm-producer-tools/lib/node_modules/qinglong-runtime-tools/node_modules/pnpm/bin/pnpm.cjs" \
--npm-cli "$RUNNER_TEMP/npm-producer-tools/lib/node_modules/npm/bin/npm-cli.js" \
--output "$RUNNER_TEMP/qinglong-npm-export"
- name: Verify and retain the exact consumer archive
id: archive
shell: bash
run: |
set -euo pipefail
node <<'NODE'
const fs = require('node:fs'), path = require('node:path'), crypto = require('node:crypto'), assert = require('node:assert/strict');
const root = path.join(process.env.RUNNER_TEMP, 'qinglong-npm-export');
const summary = JSON.parse(fs.readFileSync(path.join(root, 'package-build-summary.json')));
const pkg = JSON.parse(fs.readFileSync('package.json'));
const hash = (file) => crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex');
assert.equal(summary.sourceCommit, process.env.SOURCE_SHA); assert.equal(summary.previewMetadata, false);
assert.deepEqual(summary.sourceMetadataChanges, []); assert(summary.helperProvenance.every((item) => item.matchesGitSource));
assert.equal(summary.package, pkg.name); assert.equal(summary.packageVersion, pkg.version);
assert.equal(summary.productionManifestSHA256, hash('package.json')); assert.equal(summary.productionLockSHA256, hash('pnpm-lock.yaml'));
assert.equal(summary.runtimeToolsIncluded, true); assert.equal(summary.runtimeTools.sourceCommit, process.env.SOURCE_SHA);
assert.equal(summary.runtimeTools.builder.releaseBuild, true);
assert.equal(path.basename(summary.tarball.filename), summary.tarball.filename);
const tarball = path.join(root, 'artifacts', summary.tarball.filename);
assert.equal(hash(tarball), summary.tarball.sha256); assert.equal(fs.statSync(tarball).size, summary.tarball.bytes);
const delivery = path.join(process.env.RUNNER_TEMP, 'main-npm-delivery'); fs.mkdirSync(delivery);
fs.copyFileSync(tarball, path.join(delivery, summary.tarball.filename));
fs.copyFileSync(path.join(root, 'package-build-summary.json'), path.join(delivery, 'package-build-summary.json'));
fs.appendFileSync(process.env.GITHUB_OUTPUT, `artifact-name=qinglong-main-npm-${process.env.SOURCE_SHA}\ntarball-sha256=${summary.tarball.sha256}\npackage-version=${pkg.version}\n`);
NODE
- uses: actions/upload-artifact@v6
with:
name: ${{ steps.archive.outputs.artifact-name }}
path: ${{ runner.temp }}/main-npm-delivery/
compression-level: 0
if-no-files-found: error
- name: Preserve producer diagnostics
if: always()
uses: actions/upload-artifact@v6
with:
name: qinglong-main-npm-producer-${{ github.event.pull_request.head.sha || github.sha }}
path: ${{ runner.temp }}/qinglong-npm-export/logs/
if-no-files-found: ignore
consumer:
needs: archive
runs-on: ubuntu-latest
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
client:
- {npm: '10', mode: local}
- {npm: '12', mode: local}
- {npm: '12', mode: global}
sqlite: [prebuilt, source]
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- uses: actions/setup-node@v6
with:
node-version: '22.23.3'
package-manager-cache: false
- uses: actions/download-artifact@v7
with:
name: ${{ needs.archive.outputs.artifact-name }}
path: ${{ runner.temp }}/main-npm-input/
- uses: actions/download-artifact@v7
with:
name: qinglong-runtime-tools-${{ github.event.pull_request.head.sha || github.sha }}
path: ${{ runner.temp }}/runtime-tools-input/
- name: Verify archive and client tools against the producer source
env:
VERIFIED_TARBALL_SHA256: ${{ needs.archive.outputs.tarball-sha256 }}
shell: bash
run: |
set -euo pipefail
node <<'NODE'
const fs = require('node:fs'), path = require('node:path'), crypto = require('node:crypto'), assert = require('node:assert/strict');
const directory = path.join(process.env.RUNNER_TEMP, 'main-npm-input');
const summary = JSON.parse(fs.readFileSync(path.join(directory, 'package-build-summary.json')));
const pkg = JSON.parse(fs.readFileSync('package.json'));
const hash = (file) => crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex');
assert.equal(summary.sourceCommit, process.env.SOURCE_SHA); assert.equal(summary.previewMetadata, false);
assert.equal(summary.package, pkg.name); assert.equal(summary.packageVersion, pkg.version);
assert.equal(summary.productionManifestSHA256, hash('package.json')); assert.equal(summary.productionLockSHA256, hash('pnpm-lock.yaml'));
assert.equal(summary.runtimeToolsIncluded, true); assert.equal(summary.runtimeTools.sourceCommit, process.env.SOURCE_SHA);
assert.equal(summary.tarball.sha256, process.env.VERIFIED_TARBALL_SHA256);
assert.equal(path.basename(summary.tarball.filename), summary.tarball.filename);
assert.deepEqual(fs.readdirSync(directory).filter((name) => name.endsWith('.tgz')), [summary.tarball.filename]);
assert.equal(hash(path.join(directory, summary.tarball.filename)), process.env.VERIFIED_TARBALL_SHA256);
const tools = path.join(process.env.RUNNER_TEMP, 'runtime-tools-input');
assert.equal(hash(path.join(tools, 'runtime-tools.tgz')), summary.runtimeTools.archiveSHA256);
assert.equal(hash(path.join(tools, 'runtime-tools-proof.json')), summary.runtimeTools.proofSHA256);
NODE
- name: Install native compiler prerequisites
if: matrix.sqlite == 'source'
run: sudo apt-get update && sudo apt-get install -y --no-install-recommends build-essential python3
- name: Select the isolated npm client
env:
CLIENT_NPM: ${{ matrix.client.npm }}
shell: bash
run: |
set -euo pipefail
npm10_cli="$(node -p "require('node:path').join(require('node:path').dirname(process.execPath), '../lib/node_modules/npm/bin/npm-cli.js')")"
if [[ "$(node "$npm10_cli" --version)" != 10.* ]]; then
echo '::error::The pinned Node22 client must supply npm10.'
exit 1
fi
node scripts/install-runtime-tools.cjs \
--archive "$RUNNER_TEMP/runtime-tools-input/runtime-tools.tgz" \
--proof "$RUNNER_TEMP/runtime-tools-input/runtime-tools-proof.json" \
--prefix "$RUNNER_TEMP/npm-consumer-tools"
if [[ "$CLIENT_NPM" == 10 ]]; then
npm_cli="$npm10_cli"
else
npm_cli="$RUNNER_TEMP/npm-consumer-tools/lib/node_modules/npm/bin/npm-cli.js"
fi
echo "QINGLONG_CONSUMER_NPM_CLI=$npm_cli" >> "$GITHUB_ENV"
- name: Consume the same archive and verify the selected SQLite installation
env:
CONSUMER_MODE: ${{ matrix.client.mode }}
SQLITE_INSTALL: ${{ matrix.sqlite }}
shell: bash
run: |
set -euo pipefail
extra_args=()
if [[ "$CONSUMER_MODE" == global ]]; then extra_args+=(--global); fi
if [[ "$SQLITE_INSTALL" == source ]]; then extra_args+=(--source-build); else extra_args+=(--require-prebuilt); fi
tarball="$(node -e 'const fs=require("node:fs"),p=require("node:path"); const d=process.argv[1]; const s=JSON.parse(fs.readFileSync(p.join(d,"package-build-summary.json"))); process.stdout.write(p.join(d,s.tarball.filename))' "$RUNNER_TEMP/main-npm-input")"
node scripts/verify-npm-package.cjs \
--tarball "$tarball" \
--build-summary "$RUNNER_TEMP/main-npm-input/package-build-summary.json" \
--npm-cli "$QINGLONG_CONSUMER_NPM_CLI" \
--output "$RUNNER_TEMP/qinglong-npm-consumer" \
"${extra_args[@]}"
- name: Preserve actual consumer evidence
if: always()
uses: actions/upload-artifact@v6
with:
name: qinglong-main-npm-consumer-${{ matrix.client.npm }}-${{ matrix.client.mode }}-${{ matrix.sqlite }}-${{ github.event.pull_request.head.sha || github.sha }}
path: |
${{ runner.temp }}/qinglong-npm-consumer/*.log
${{ runner.temp }}/qinglong-npm-consumer/npm-consumer-validation.json
if-no-files-found: ignore
source-fallback:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- uses: actions/setup-node@v6
with:
node-version: '22.23.3'
package-manager-cache: false
- uses: actions/download-artifact@v7
with:
name: ${{ inputs.static-artifact }}
path: static/
- name: Install native compiler prerequisites
run: sudo apt-get update && sudo apt-get install -y --no-install-recommends build-essential python3
- name: Prepare source using an older manager and the same verified static tools
env:
npm_config_build_from_source: 'true'
shell: bash
run: |
set -euo pipefail
mkdir "$RUNNER_TEMP/source-tool-fallback"
git archive "$SOURCE_SHA" | tar -x -C "$RUNNER_TEMP/source-tool-fallback"
npm10_cli="$(node -p "require('node:path').join(require('node:path').dirname(process.execPath), '../lib/node_modules/npm/bin/npm-cli.js')")"
printf 'registry=https://registry.npmjs.org/\n' > "$RUNNER_TEMP/legacy-pnpm-public.npmrc"
export NPM_CONFIG_USERCONFIG="$RUNNER_TEMP/legacy-pnpm-public.npmrc"
export npm_config_userconfig="$NPM_CONFIG_USERCONFIG"
export NPM_CONFIG_GLOBALCONFIG=/dev/null npm_config_globalconfig=/dev/null
export NPM_CONFIG_CACHE="$RUNNER_TEMP/legacy-pnpm-cache"
export PNPM_HOME="$RUNNER_TEMP/legacy-pnpm-home"
export XDG_CACHE_HOME="$RUNNER_TEMP/legacy-pnpm-xdg-cache" XDG_STATE_HOME="$RUNNER_TEMP/legacy-pnpm-xdg-state"
node "$npm10_cli" install --global pnpm@8.3.1 \
--prefix "$RUNNER_TEMP/legacy-pnpm" --ignore-scripts --no-audit --no-fund \
--userconfig "$RUNNER_TEMP/legacy-pnpm-public.npmrc" --globalconfig /dev/null \
--cache "$RUNNER_TEMP/legacy-pnpm-cache" --registry=https://registry.npmjs.org \
2>&1 | tee "$RUNNER_TEMP/legacy-pnpm-install.log"
legacy_cli="$RUNNER_TEMP/legacy-pnpm/lib/node_modules/pnpm/bin/pnpm.cjs"
node "$legacy_cli" --version | tee "$RUNNER_TEMP/legacy-pnpm-version.log"
test "$(cat "$RUNNER_TEMP/legacy-pnpm-version.log")" = '8.3.1'
node -e 'const fs=require("node:fs"),os=require("node:os"); fs.writeFileSync(process.argv[1],JSON.stringify(fs.readdirSync(os.tmpdir()).filter(n=>n.startsWith("qinglong-source-tools-")).sort()))' "$RUNNER_TEMP/source-tool-prefixes-before.json"
node scripts/install-panel-dependencies.cjs \
--root "$RUNNER_TEMP/source-tool-fallback" \
--pnpm-cli "$legacy_cli" \
--static-dir "$GITHUB_WORKSPACE/static" \
--source-commit "$SOURCE_SHA" \
2>&1 | tee "$RUNNER_TEMP/source-tool-fallback.log"
grep -q 'SOLINK_MODULE' "$RUNNER_TEMP/source-tool-fallback.log"
grep -q 'node-gyp@12.4.0' "$RUNNER_TEMP/source-tool-fallback.log"
grep -q '"prefixScope":"fresh temporary only"' "$RUNNER_TEMP/source-tool-fallback.log"
node <<'NODE'
const fs = require('node:fs'), path = require('node:path'), os = require('node:os'), crypto = require('node:crypto'), assert = require('node:assert/strict');
const { createRequire } = require('node:module');
const root = path.join(process.env.RUNNER_TEMP, 'source-tool-fallback');
const hash = (file) => crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex');
assert.equal(hash(path.join(root, 'pnpm-lock.yaml')), hash('pnpm-lock.yaml'));
const before = JSON.parse(fs.readFileSync(path.join(process.env.RUNNER_TEMP, 'source-tool-prefixes-before.json')));
assert.deepEqual(fs.readdirSync(os.tmpdir()).filter((name) => name.startsWith('qinglong-source-tools-')).sort(), before);
const requireSource = createRequire(path.join(root, 'package.json')); requireSource('sqlite3');
const binding = Object.keys(require.cache).find((file) => file.endsWith('.node'));
assert(binding); assert.equal(fs.readFileSync(binding).subarray(0, 4).toString('hex'), '7f454c46');
fs.writeFileSync(path.join(process.env.RUNNER_TEMP, 'source-tool-native-proof.json'), JSON.stringify({ sourceCommit: process.env.SOURCE_SHA, productionLockSHA256: hash('pnpm-lock.yaml'), legacyManagerVersion: '8.3.1', legacyManagerCliSHA256: hash(path.join(process.env.RUNNER_TEMP, 'legacy-pnpm/lib/node_modules/pnpm/bin/pnpm.cjs')), binding: path.relative(root, binding), bindingSHA256: hash(binding), temporaryPrefixRemoved: true }, null, 2));
NODE
node "$RUNNER_TEMP/source-tool-fallback/scripts/verify-production-dependencies.cjs" \
2>&1 | tee "$RUNNER_TEMP/source-tool-fallback-protocols.log"
- name: Preserve source migration evidence
if: always()
uses: actions/upload-artifact@v6
with:
name: qinglong-source-tool-fallback-${{ github.event.pull_request.head.sha || github.sha }}
path: |
${{ runner.temp }}/legacy-pnpm-install.log
${{ runner.temp }}/legacy-pnpm-version.log
${{ runner.temp }}/source-tool-fallback.log
${{ runner.temp }}/source-tool-fallback-protocols.log
${{ runner.temp }}/source-tool-native-proof.json
if-no-files-found: ignore
verified:
needs: [archive, consumer, source-fallback]
runs-on: ubuntu-latest
outputs:
artifact-name: ${{ needs.archive.outputs.artifact-name }}
tarball-sha256: ${{ needs.archive.outputs.tarball-sha256 }}
package-version: ${{ needs.archive.outputs.package-version }}
steps:
- run: echo 'The exact main npm archive passed six Linux consumers and the older-manager source migration.'
+194
View File
@@ -0,0 +1,194 @@
name: Release readiness
on:
pull_request:
paths:
- '.github/workflows/release-readiness.yml'
- 'package.json'
- 'pnpm-lock.yaml'
workflow_dispatch:
permissions:
contents: read
concurrency:
group: release-readiness-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
regression:
uses: ./.github/workflows/validate.yml
runtime-tools:
uses: ./.github/workflows/runtime-tools.yml
static:
needs: runtime-tools
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- uses: pnpm/action-setup@v6
with:
version: '10.34.6'
- uses: actions/setup-node@v6
with:
node-version: '22'
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: pnpm audit --prod --audit-level high
- run: pnpm build:front && pnpm build:back
- uses: actions/download-artifact@v7
with:
name: qinglong-runtime-tools-${{ github.event.pull_request.head.sha || github.sha }}
path: static/
- run: pnpm build:info
- uses: actions/upload-artifact@v6
with:
name: release-readiness-static
path: static/
include-hidden-files: true
if-no-files-found: error
sqlite-install:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- uses: pnpm/action-setup@v6
with:
version: '10.34.6'
- uses: actions/setup-node@v6
with:
node-version: '22'
cache: pnpm
- run: pnpm install --prod --frozen-lockfile
- name: Verify precompiled SQLite and production protocols
run: node scripts/verify-production-dependencies.cjs
- name: Rebuild SQLite from source
env:
npm_config_build_from_source: 'true'
PYTHON: /usr/bin/python3
run: |
export npm_config_node_gyp="$(node -p 'require("node:module").createRequire(require.resolve("sqlite3/package.json")).resolve("node-gyp/bin/node-gyp.js")')"
pnpm rebuild @whyour/sqlite3
- name: Verify source-built SQLite and production protocols
run: node scripts/verify-production-dependencies.cjs
npm-package:
needs: static
uses: ./.github/workflows/npm-package.yml
with:
static-artifact: release-readiness-static
images:
needs: [regression, static, sqlite-install]
runs-on: ubuntu-22.04
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
include:
- image: alpine
file: docker/Dockerfile
cache: alpine
platforms: linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x,linux/386
- image: debian
file: docker/Dockerfile.debian
cache: debian
platforms: linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x
- image: alpine-python3.10
file: docker/Dockerfile.310
cache: alpine-python3.10
platforms: linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x,linux/386
- image: debian-python3.10
file: docker/Dockerfile.debian310
cache: debian-python3.10
platforms: linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Record source
id: source
env:
SOURCE_URL: ${{ github.event.pull_request.head.repo.clone_url || format('https://github.com/{0}.git', github.repository) }}
SOURCE_BRANCH: ${{ github.head_ref || github.ref_name }}
run: |
echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
echo "url=$SOURCE_URL" >> "$GITHUB_OUTPUT"
echo "branch=$SOURCE_BRANCH" >> "$GITHUB_OUTPUT"
- uses: actions/download-artifact@v7
with:
name: release-readiness-static
path: static/
- uses: docker/setup-qemu-action@v4
with:
cache-image: false
- uses: docker/setup-buildx-action@v4
- name: Build every release architecture
uses: docker/build-push-action@v7
with:
context: .
file: ${{ matrix.file }}
platforms: ${{ matrix.platforms }}
build-args: |
QL_URL=${{ steps.source.outputs.url }}
QL_BRANCH=${{ steps.source.outputs.branch }}
SOURCE_COMMIT=${{ steps.source.outputs.sha }}
push: false
provenance: false
sbom: false
cache-from: type=registry,ref=whyour/qinglong:cache-${{ matrix.cache }}
- name: Load native image for runtime verification
uses: docker/build-push-action@v7
with:
context: .
file: ${{ matrix.file }}
platforms: linux/amd64
build-args: |
QL_URL=${{ steps.source.outputs.url }}
QL_BRANCH=${{ steps.source.outputs.branch }}
SOURCE_COMMIT=${{ steps.source.outputs.sha }}
tags: qinglong-readiness:${{ matrix.image }}
load: true
push: false
provenance: false
sbom: false
- name: Verify native production runtime without external networking
run: |
docker run --rm --network none --entrypoint node \
qinglong-readiness:${{ matrix.image }} \
--eval 'console.log(JSON.stringify({node: process.version, arch: process.arch, platform: process.platform}))'
docker run --rm --network none --entrypoint python3 \
qinglong-readiness:${{ matrix.image }} --version
docker run --rm --network none --entrypoint node \
qinglong-readiness:${{ matrix.image }} \
scripts/verify-system-runtime.cjs
docker run --rm --network none --entrypoint node \
qinglong-readiness:${{ matrix.image }} \
scripts/verify-production-dependencies.cjs
- name: Archive verified native Debian image for isolated dev validation
if: matrix.image == 'debian'
shell: bash
run: |
docker image inspect qinglong-readiness:debian \
--format '{"id":{{json .Id}},"revision":{{json (index .Config.Labels "org.opencontainers.image.revision")}},"architecture":{{json .Architecture}},"os":{{json .Os}}}' \
> release-readiness-debian-image.json
docker save qinglong-readiness:debian | gzip > release-readiness-debian-image.tar.gz
sha256sum release-readiness-debian-image.tar.gz > release-readiness-debian-image.sha256
- uses: actions/upload-artifact@v6
if: matrix.image == 'debian'
with:
name: release-readiness-debian-image
path: |
release-readiness-debian-image.tar.gz
release-readiness-debian-image.sha256
release-readiness-debian-image.json
compression-level: 0
retention-days: 3
if-no-files-found: error
+40
View File
@@ -0,0 +1,40 @@
name: Build verified runtime tools
on:
workflow_call:
workflow_dispatch:
permissions:
contents: read
jobs:
tools:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- uses: actions/setup-node@v6
with:
node-version: '22.23.3'
package-manager-cache: false
- name: Build the pinned universal Linux tool archive
env:
QL_TOOLS_SOURCE_COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
run: bash scripts/build-runtime-tools.sh "$RUNNER_TEMP/qinglong-runtime-tools"
- uses: actions/upload-artifact@v6
with:
name: qinglong-runtime-tools-${{ github.event.pull_request.head.sha || github.sha }}
path: |
${{ runner.temp }}/qinglong-runtime-tools/runtime-tools.tgz
${{ runner.temp }}/qinglong-runtime-tools/runtime-tools-proof.json
compression-level: 0
if-no-files-found: error
- name: Preserve build diagnostics
if: always()
uses: actions/upload-artifact@v6
with:
name: qinglong-runtime-tools-diagnostics-${{ github.event.pull_request.head.sha || github.sha }}
path: ${{ runner.temp }}/qinglong-runtime-tools/logs/
if-no-files-found: ignore
+8 -3
View File
@@ -11,21 +11,26 @@ permissions:
jobs:
regression:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
node: ['22', '24']
steps:
- uses: actions/checkout@v6
- uses: pnpm/action-setup@v6
with:
version: '8.3.1'
version: '10.34.6'
- uses: actions/setup-node@v6
with:
node-version: '20'
node-version: ${{ matrix.node }}
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: cp .env.example .env
- run: pnpm test
- run: pnpm build:back
- run: pnpm build:front
- run: pnpm benchmark:execution -- --runs=10 --output=execution-baseline.json
- uses: actions/upload-artifact@v6
with:
name: execution-baseline
name: execution-baseline-node-${{ matrix.node }}
path: execution-baseline.json
+2 -1
View File
@@ -1 +1,2 @@
strict-peer-dependencies=false
strict-peer-dependencies=false
strict-dep-builds=true
+10 -4
View File
@@ -62,10 +62,14 @@ docker run -d \
### npm
The npm version supports `debian/ubuntu/alpine` systems and requires `node/npm/python3/pip3/pnpm` to be installed.
The npm deployment supports `debian/ubuntu/alpine` with Node.js 22.x >=22.22.2 or 24.x >=24.15.0, npm, Python 3 and pip. The 2.23.0 archive includes frozen production dependencies and verified runtime tools; a separate pnpm installation is unnecessary. After this version is published, use the command below. With npm 12, approve this exact package lifecycle in your project's `package.json` first:
```json
{"allowScripts":{"@whyour/qinglong@2.23.0":true}}
```
```bash
npm i @whyour/qinglong
npm i @whyour/qinglong@2.23.0
```
## Deployment
@@ -105,13 +109,15 @@ The package includes a [remote management skill](cli/skills/qinglong-cli/SKILL.m
## Development
Use Node.js 22.x starting at 22.22.2, or Node.js 24.x starting at 24.15.0.
```bash
git clone https://github.com/whyour/qinglong.git
cd qinglong
cp .env.example .env
# Recommended use pnpm https://pnpm.io/zh/installation
npm install -g pnpm@8.3.1
pnpm install
npm install -g pnpm@10.34.6
pnpm install --frozen-lockfile
pnpm start
```
+10 -4
View File
@@ -64,10 +64,14 @@ docker run -d \
### npm
npm 版本支持 `debian/ubuntu/alpine` 系统,需要自行安装 `node/npm/python3/pip3/pnpm`
npm 版本支持 `debian/ubuntu/alpine` 系统,需要自行安装 Node.js 22.x >=22.22.2 或 24.x >=24.15.0,以及 npm、Python 3 和 pip。2.23.0 归档携带冻结生产依赖和受控工具,不需要单独安装 pnpm。该版本发布后使用以下命令;npm 12 的项目安装需要在当前项目 `package.json` 中精确批准安装脚本:
```json
{"allowScripts":{"@whyour/qinglong@2.23.0":true}}
```
```bash
npm i @whyour/qinglong
npm i @whyour/qinglong@2.23.0
```
## 部署
@@ -107,13 +111,15 @@ npm CLI 用于远程管理。面板内置 `ql` 同样使用这个命令名,已
## 开发
使用 Node.js 22.22.2 及以上的 22.x,或 Node.js 24.x >=24.15.0。
```bash
git clone https://github.com/whyour/qinglong.git
cd qinglong
cp .env.example .env
# 推荐使用 pnpm https://pnpm.io/zh/installation
npm install -g pnpm@8.3.1
pnpm install
npm install -g pnpm@10.34.6
pnpm install --frozen-lockfile
pnpm start
```
+3 -7
View File
@@ -4,7 +4,6 @@ import { Logger } from 'winston';
import config from '../config';
import * as fs from 'fs/promises';
import { celebrate, Joi } from 'celebrate';
import { join, basename } from 'path';
import { SAMPLE_FILES } from '../config/const';
import { t } from '../shared/i18n';
import ConfigService from '../services/config';
@@ -73,15 +72,12 @@ export default (app: Router) => {
const logger: Logger = Container.get('logger');
try {
const { name, content } = req.body;
// Resolve path first to prevent traversal attacks
let basePath = config.configPath;
if (name.startsWith('data/scripts/')) {
basePath = join(config.rootPath, 'data/scripts');
return res.send({ code: 403, message: t('文件无法访问') });
}
const cleanName = name.replace(/^data\/scripts\//, '');
const normalized = resolveFileAccess(
basePath,
[cleanName],
config.configPath,
[name],
config.blackFileList,
);
if (!normalized) {
+17 -14
View File
@@ -1,24 +1,16 @@
import { Joi, celebrate } from 'celebrate';
import { NextFunction, Request, Response, Router } from 'express';
import fs from 'fs';
import multer from 'multer';
import { Container } from 'typedi';
import { Logger } from 'winston';
import config from '../config';
import { safeJSONParse } from '../config/util';
import { t } from '../shared/i18n';
import EnvService from '../services/env';
const route = Router();
const storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, config.scriptPath);
},
filename: function (req, file, cb) {
cb(null, file.originalname);
},
const upload = multer({
storage: multer.memoryStorage(),
limits: { fileSize: 5 * 1024 * 1024, files: 1 },
});
const upload = multer({ storage: storage });
const labelSchema = Joi.array()
.items(Joi.string().trim().required())
.min(1)
@@ -282,12 +274,23 @@ export default (app: Router) => {
const logger: Logger = Container.get('logger');
try {
const envService = Container.get(EnvService);
const fileContent = await fs.promises.readFile(req!.file!.path, 'utf8');
const parseContent = safeJSONParse(fileContent);
let parseContent: unknown;
try {
parseContent = JSON.parse(req.file?.buffer.toString('utf8') || '');
} catch {
return res.status(400).send({ code: 400, message: t('参数错误') });
}
const data = Array.isArray(parseContent)
? parseContent
: [parseContent];
if (data.every((x) => x.name && x.value)) {
if (
data.length > 0 &&
data.every((x) =>
x && typeof x.name === 'string' &&
/^[a-zA-Z_][0-9a-zA-Z_]*$/.test(x.name) &&
typeof x.value === 'string' && x.value.length > 0,
)
) {
const result = await envService.create(
data.map((x) => ({
name: x.name,
+2
View File
@@ -3,6 +3,7 @@ import { Container } from 'typedi';
import OpenService from '../services/open';
import { Logger } from 'winston';
import { celebrate, Joi } from 'celebrate';
import { rateLimit } from 'express-rate-limit';
const route = Router();
export default (app: Router) => {
@@ -100,6 +101,7 @@ export default (app: Router) => {
route.get(
'/auth/token',
rateLimit({ windowMs: 15 * 60 * 1000, limit: 100 }),
celebrate({
query: {
client_id: Joi.string().required(),
+199 -14
View File
@@ -7,20 +7,62 @@ import { Logger } from 'winston';
import config from '../config';
import * as fs from 'fs/promises';
import { celebrate, Joi } from 'celebrate';
import path, { join, parse } from 'path';
import path, { dirname, join, parse, resolve, sep } from 'path';
import ScriptService from '../services/script';
import ScriptHistoryService from '../services/scriptHistory';
import { t } from '../shared/i18n';
import multer from 'multer';
import { writeFileWithLock } from '../shared/utils';
import {
ScriptHistoryError,
HistoryUnavailableError,
} from '../shared/scriptHistory';
const route = Router();
function isPathAllowed(targetPath: string): boolean {
const resolved = path.resolve(targetPath);
return config.writePathList.some((x) =>
Boolean(resolveFileAccess(x, [resolved], config.blackFileList)),
Boolean(
resolveFileAccess(
x,
[resolved],
// Panel configuration secrets must not restrict user script filenames.
path.resolve(x) === path.resolve(config.configPath)
? config.blackFileList
: [],
),
),
);
}
async function existingWritableRealPath(targetPath: string): Promise<string> {
if (!isPathAllowed(targetPath)) {
throw new ScriptHistoryError(403, '暂无权限');
}
const resolved = resolve(targetPath);
for (const writableRoot of config.writePathList) {
const root = resolve(writableRoot);
const rootPrefix = root.endsWith(sep) ? root : root + sep;
// Keep an explicit normalized boundary check at the filesystem operation.
if (!resolved.startsWith(rootPrefix)) continue;
const realPath = await fs.realpath(resolved);
const realRoot = await fs.realpath(root);
const realRootPrefix = realRoot.endsWith(sep) ? realRoot : realRoot + sep;
if (
!realPath.startsWith(realRootPrefix) ||
!resolveFileAccess(
realRoot,
[realPath],
root === resolve(config.configPath) ? config.blackFileList : [],
)
) {
throw new ScriptHistoryError(403, '暂无权限');
}
return realPath;
}
throw new ScriptHistoryError(403, '暂无权限');
}
const storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, config.tmpPath);
@@ -55,11 +97,7 @@ export default (app: Router) => {
];
if (req.query.path) {
if (
!resolveFileAccess(
config.scriptPath,
[req.query.path as string],
config.blackFileList,
)
!resolveFileAccess(config.scriptPath, [req.query.path as string])
) {
return res.send({ code: 403, message: t('暂无权限') });
}
@@ -115,6 +153,82 @@ export default (app: Router) => {
},
);
const historyError = (error: unknown, res: Response, next: NextFunction) => {
if (error instanceof ScriptHistoryError) {
return res.status(error.status).send({
code: error.status,
message: t(error.message),
...(error instanceof HistoryUnavailableError
? { historyUnavailable: true, currentHash: error.currentHash }
: {}),
});
}
return next(error);
};
const historyQuery = {
filename: Joi.string().required(),
path: Joi.string().optional().allow(''),
};
route.get(
'/history',
celebrate({ query: Joi.object(historyQuery).unknown(true) }),
async (req: Request, res: Response, next: NextFunction) => {
try {
const service = Container.get(ScriptHistoryService);
const data = await service.list(
(req.query.path as string) || '',
req.query.filename as string,
);
res.send({ code: 200, data });
} catch (error) {
historyError(error, res, next);
}
},
);
route.get(
'/history/detail',
celebrate({
query: Joi.object({
...historyQuery,
id: Joi.string().uuid().required(),
}).unknown(true),
}),
async (req: Request, res: Response, next: NextFunction) => {
try {
const service = Container.get(ScriptHistoryService);
const data = await service.detail(
(req.query.path as string) || '',
req.query.filename as string,
req.query.id as string,
);
res.send({ code: 200, data });
} catch (error) {
historyError(error, res, next);
}
},
);
route.put(
'/history/restore',
celebrate({
body: Joi.object({
...historyQuery,
id: Joi.string().uuid().required(),
expectedHash: Joi.string().hex().length(64).required(),
}),
}),
async (req: Request, res: Response, next: NextFunction) => {
try {
const { path = '', filename, id, expectedHash } = req.body;
const service = Container.get(ScriptHistoryService);
const data = await service.restore(path, filename, id, expectedHash);
res.send({ code: 200, data });
} catch (error) {
historyError(error, res, next);
}
},
);
route.get('/:file', (req: Request, res: Response) => {
return res.send({
code: 410,
@@ -136,6 +250,11 @@ export default (app: Router) => {
filename: Joi.string().required(),
path: Joi.string().optional().allow(''),
content: Joi.string().optional().allow(''),
skipHistory: Joi.boolean().optional(),
expectedHash: Joi.string()
.hex()
.length(64)
.when('skipHistory', { is: true, then: Joi.required() }),
originFilename: Joi.string().optional().allow(''),
directory: Joi.string().optional().allow(''),
file: Joi.string().optional().allow(''),
@@ -195,8 +314,56 @@ export default (app: Router) => {
if (!isPathAllowed(filePath) || !isPathAllowed(originFilePath)) {
return res.send({ code: 403, message: t('暂无权限') });
}
await fs.mkdir(path, { recursive: true });
// Check the canonical parent directly at the directory creation boundary.
const parentPath = resolve(dirname(filePath));
let parentCreated = false;
for (const writableRoot of config.writePathList) {
const root = resolve(writableRoot);
const rootPrefix = root.endsWith(sep) ? root : root + sep;
if (parentPath === root) {
await fs.mkdir(root, { recursive: true });
} else if (parentPath.startsWith(rootPrefix)) {
await fs.mkdir(parentPath, { recursive: true });
} else {
continue;
}
parentCreated = true;
break;
}
if (!parentCreated) {
return res.send({ code: 403, message: t('暂无权限') });
}
const fileExists = await fileExist(filePath);
if (fileExists && resolveFileAccess(config.scriptPath, [filePath])) {
let removeSource = filename !== originFilename;
let backupSource = '';
if (removeSource) {
const [originRealPath, targetRealPath] = await Promise.all([
existingWritableRealPath(originFilePath),
existingWritableRealPath(filePath),
]);
// Aliases of one script are an in-place save. Removing the source
// would also remove the destination behind a target symlink.
removeSource = originRealPath !== targetRealPath;
backupSource = originRealPath;
}
// Save-as removes the source after committing the destination. Keep
// its original content too: destination history only protects the
// file being overwritten, not the source being deleted.
if (removeSource) {
await fs.copyFile(
backupSource,
join(config.bakPath, originFilename.replace(/\//g, '')),
);
}
const service = Container.get(ScriptHistoryService);
const data = await service.save(path, filename, content, {
skipHistory: req.body.skipHistory,
expectedHash: req.body.expectedHash,
});
if (removeSource) await rmPath(originFilePath);
return res.send({ code: 200, data });
}
if (fileExists) {
await fs.copyFile(
originFilePath,
@@ -209,7 +376,7 @@ export default (app: Router) => {
await writeFileWithLock(filePath, content);
return res.send({ code: 200 });
} catch (e) {
return next(e);
return historyError(e, res, next);
}
},
);
@@ -221,6 +388,11 @@ export default (app: Router) => {
filename: Joi.string().required(),
path: Joi.string().optional().allow(''),
content: Joi.string().required().allow(''),
skipHistory: Joi.boolean().optional(),
expectedHash: Joi.string()
.hex()
.length(64)
.when('skipHistory', { is: true, then: Joi.required() }),
}),
}),
async (req: Request, res: Response, next: NextFunction) => {
@@ -238,10 +410,14 @@ export default (app: Router) => {
message: t('暂无权限'),
});
}
await writeFileWithLock(filePath, content);
return res.send({ code: 200 });
const service = Container.get(ScriptHistoryService);
const data = await service.save(path || '', filename, content, {
skipHistory: req.body.skipHistory,
expectedHash: req.body.expectedHash,
});
return res.send({ code: 200, data });
} catch (e) {
return next(e);
return historyError(e, res, next);
}
},
);
@@ -368,12 +544,21 @@ export default (app: Router) => {
if (!isPathAllowed(filePath)) {
return res.send({ code: 403, message: t('暂无权限') });
}
const logPath = join(config.logPath, path, `${name}.swap`);
const logPath = resolveFileAccess(config.logPath, [
path,
`${name}.swap`,
]);
if (!logPath) {
return res.send({ code: 403, message: t('暂无权限') });
}
const scriptService = Container.get(ScriptService);
const result = await scriptService.stopScript(filePath, pid);
setTimeout(() => {
rmPath(logPath);
const cleanupPath = resolveFileAccess(config.logPath, [logPath]);
if (cleanupPath) {
void rmPath(cleanupPath);
}
}, 3000);
res.send(result);
} catch (e) {
+3 -10
View File
@@ -3,7 +3,7 @@ import { Container } from 'typedi';
import { Logger } from 'winston';
import SubscriptionService from '../services/subscription';
import { celebrate, Joi } from 'celebrate';
import CronExpressionParser from 'cron-parser';
import { isValidCronSchedule } from '../shared/cronSchedule';
const route = Router();
export default (app: Router) => {
@@ -58,10 +58,7 @@ export default (app: Router) => {
async (req: Request, res: Response, next: NextFunction) => {
const logger: Logger = Container.get('logger');
try {
if (
!req.body.schedule ||
CronExpressionParser.parse(req.body.schedule).hasNext()
) {
if (!req.body.schedule || isValidCronSchedule(req.body.schedule)) {
const subscriptionService = Container.get(SubscriptionService);
const data = await subscriptionService.create(req.body);
return res.send({ code: 200, data });
@@ -204,11 +201,7 @@ export default (app: Router) => {
async (req: Request, res: Response, next: NextFunction) => {
const logger: Logger = Container.get('logger');
try {
if (
!req.body.schedule ||
typeof req.body.schedule === 'object' ||
CronExpressionParser.parse(req.body.schedule).hasNext()
) {
if (!req.body.schedule || isValidCronSchedule(req.body.schedule)) {
const subscriptionService = Container.get(SubscriptionService);
const data = await subscriptionService.update(req.body);
return res.send({ code: 200, data });
+14 -9
View File
@@ -8,15 +8,12 @@ import { celebrate, Joi } from 'celebrate';
import UserService from '../services/user';
import { t } from '../shared/i18n';
import { isDefaultAuthInfo } from '../shared/auth';
import {
getUniqPath,
handleLogPath,
parseVersion,
promiseExec,
} from '../config/util';
import { getUniqPath, handleLogPath, parseVersion } from '../config/util';
import { findCronId } from '../shared/cronCommand';
import dayjs from 'dayjs';
import multer from 'multer';
import { logStreamManager } from '../shared/logStreamManager';
import { rateLimit } from 'express-rate-limit';
const route = Router();
const storage = multer.diskStorage({
@@ -249,6 +246,7 @@ export default (app: Router) => {
route.put(
'/command-run',
rateLimit({ windowMs: 60 * 1000, limit: 60 }),
celebrate({
body: Joi.object({
command: Joi.string().required(),
@@ -258,8 +256,13 @@ export default (app: Router) => {
try {
const systemService = Container.get(SystemService);
const command = req.body.command;
const idStr = `cat ${config.crontabFile} | grep -E "${command}" | perl -pe "s|.*ID=(.*) ${command}.*|\\1|" | head -1 | awk -F " " '{print $1}' | xargs echo -n`;
let id = await promiseExec(idStr);
const crontab = await fs
.readFile(config.crontabFile, 'utf8')
.catch((e) => {
if (e.code === 'ENOENT') return '';
throw e;
});
const id = findCronId(crontab, command);
const uniqPath = await getUniqPath(command, id);
const logTime = dayjs().format('YYYY-MM-DD-HH-mm-ss-SSS');
const logPath = `${uniqPath}/${logTime}.log`;
@@ -274,7 +277,9 @@ export default (app: Router) => {
onEnd: async (cp, endTime, diff) => {
// Close the stream after task completion
try {
await logStreamManager.closeStream(await handleLogPath(logPath));
await logStreamManager.closeStream(
await handleLogPath(logPath),
);
} finally {
res.end();
}
+93 -45
View File
@@ -1,6 +1,5 @@
import { execSync } from 'child_process';
import { execFileSync } from 'child_process';
import * as fs from 'fs/promises';
import * as os from 'os';
import path from 'path';
import config from './index';
import { fileExist } from './util';
@@ -24,71 +23,120 @@ const clientCertPath = path.join(certDir, 'client.crt');
let cachedConfig: GrpcTlsConfig | null = null;
function run(cmd: string, execOpts?: Record<string, unknown>): string {
const opts = { stdio: 'pipe', timeout: 30000, encoding: 'utf-8', ...execOpts } as any;
return (execSync(cmd, opts) as string).trim();
}
async function tmpFile(prefix: string): Promise<string> {
const dir = (await fileExist(certDir)) ? certDir : os.tmpdir();
await fs.mkdir(dir, { recursive: true });
return path.join(dir, `.${prefix}_${Date.now()}_${Math.random().toString(36).slice(2)}.pem`);
function run(args: string[]): string {
return execFileSync('openssl', args, {
stdio: 'pipe',
timeout: 30000,
encoding: 'utf-8',
}).trim();
}
async function generateAllCerts(): Promise<GrpcTlsConfig> {
Logger.info('[boot] Generating gRPC mTLS certificates...');
const caKeyTmp = await tmpFile('ca_key');
const caCertTmp = await tmpFile('ca_cert');
const serverKeyTmp = await tmpFile('server_key');
const serverCsrTmp = await tmpFile('server_csr');
const serverExtTmp = await tmpFile('server_ext');
const clientKeyTmp = await tmpFile('client_key');
const clientCsrTmp = await tmpFile('client_csr');
const clientExtTmp = await tmpFile('client_ext');
const srlTmp = path.join(path.dirname(caKeyTmp), '.grpc_ca.srl');
await fs.mkdir(certDir, { recursive: true, mode: 0o700 });
// Atomically reserve a private workspace for keys, CSRs and OpenSSL side files.
const workspace = await fs.mkdtemp(path.join(certDir, '.generate-'));
const caKeyTmp = path.join(workspace, 'ca.key');
const caCertTmp = path.join(workspace, 'ca.crt');
const serverKeyTmp = path.join(workspace, 'server.key');
const serverCsrTmp = path.join(workspace, 'server.csr');
const serverExtTmp = path.join(workspace, 'server.ext');
const clientKeyTmp = path.join(workspace, 'client.key');
const clientCsrTmp = path.join(workspace, 'client.csr');
const clientExtTmp = path.join(workspace, 'client.ext');
const cleanup = async () => {
for (const f of [caKeyTmp, caCertTmp, serverKeyTmp, serverCsrTmp, serverExtTmp,
clientKeyTmp, clientCsrTmp, clientExtTmp, srlTmp]) {
try { await fs.unlink(f); } catch {}
}
await fs.rm(workspace, { recursive: true, force: true });
};
try {
// 1. CA(私钥直接存盘,证书写入临时文件供签发使用)
run(`openssl genrsa -out '${caKeyTmp}' 2048 2>/dev/null`);
run(`openssl req -new -x509 -days 3650 -key '${caKeyTmp}' -out '${caCertTmp}' -subj '/CN=qinglong-ca/O=qinglong/C=CN' 2>/dev/null`);
run(['genrsa', '-out', caKeyTmp, '2048']);
run([
'req',
'-new',
'-x509',
'-days',
'3650',
'-key',
caKeyTmp,
'-out',
caCertTmp,
'-subj',
'/CN=qinglong-ca/O=qinglong/C=CN',
]);
const caKey = await fs.readFile(caKeyTmp, 'utf-8');
const caCert = await fs.readFile(caCertTmp, 'utf-8');
await fs.mkdir(certDir, { recursive: true });
await fs.writeFile(caKeyPath, caKey, { mode: 0o600 });
// 2. 服务端
run(`openssl genrsa -out '${serverKeyTmp}' 2048 2>/dev/null`);
run(`openssl req -new -key '${serverKeyTmp}' -out '${serverCsrTmp}' -subj '/CN=grpc-server' 2>/dev/null`);
await fs.writeFile(serverExtTmp, 'subjectAltName=DNS:localhost,IP:127.0.0.1,IP:::1\n');
const serverCert = run(
`openssl x509 -req -days 3650 -in '${serverCsrTmp}' -CA '${caCertTmp}' -CAkey '${caKeyTmp}' -CAcreateserial -extfile '${serverExtTmp}' 2>/dev/null`,
run(['genrsa', '-out', serverKeyTmp, '2048']);
run([
'req',
'-new',
'-key',
serverKeyTmp,
'-out',
serverCsrTmp,
'-subj',
'/CN=grpc-server',
]);
await fs.writeFile(
serverExtTmp,
'subjectAltName=DNS:localhost,IP:127.0.0.1,IP:::1\n',
);
const serverCert = run([
'x509',
'-req',
'-days',
'3650',
'-in',
serverCsrTmp,
'-CA',
caCertTmp,
'-CAkey',
caKeyTmp,
'-CAserial',
path.join(workspace, 'ca.srl'),
'-CAcreateserial',
'-extfile',
serverExtTmp,
]);
const serverKey = await fs.readFile(serverKeyTmp, 'utf-8');
// 3. 客户端
run(`openssl genrsa -out '${clientKeyTmp}' 2048 2>/dev/null`);
run(`openssl req -new -key '${clientKeyTmp}' -out '${clientCsrTmp}' -subj '/CN=grpc-client' 2>/dev/null`);
run(['genrsa', '-out', clientKeyTmp, '2048']);
run([
'req',
'-new',
'-key',
clientKeyTmp,
'-out',
clientCsrTmp,
'-subj',
'/CN=grpc-client',
]);
await fs.writeFile(clientExtTmp, 'extendedKeyUsage=clientAuth\n');
const clientCert = run(
`openssl x509 -req -days 3650 -in '${clientCsrTmp}' -CA '${caCertTmp}' -CAkey '${caKeyTmp}' -CAcreateserial -extfile '${clientExtTmp}' 2>/dev/null`,
);
const clientCert = run([
'x509',
'-req',
'-days',
'3650',
'-in',
clientCsrTmp,
'-CA',
caCertTmp,
'-CAkey',
caKeyTmp,
'-CAserial',
path.join(workspace, 'ca.srl'),
'-CAcreateserial',
'-extfile',
clientExtTmp,
]);
const clientKey = await fs.readFile(clientKeyTmp, 'utf-8');
await cleanup();
Logger.info('[boot] gRPC mTLS certificates generated successfully');
return { caCert, serverCert, serverKey, clientCert, clientKey };
} catch (e) {
} finally {
await cleanup();
throw e;
}
}
+23 -26
View File
@@ -1,5 +1,7 @@
import { Subscription } from '../data/subscription';
import isNil from 'lodash/isNil';
import { shellQuote } from '../shared/shellQuote';
import { getSubscriptionSshAlias } from '../shared/subscriptionPath';
export function formatUrl(doc: Subscription) {
let url = doc.url;
@@ -7,7 +9,7 @@ export function formatUrl(doc: Subscription) {
if (doc.type === 'private-repo') {
if (doc.pull_type === 'ssh-key') {
host = doc.url!.replace(/.*\@([^\:]+)\:.*/, '$1');
url = doc.url!.replace(host, doc.alias);
url = doc.url!.replace(host, getSubscriptionSshAlias(doc.alias));
} else {
host = doc.url!.replace(/.*\:\/\/([^\/]+)\/.*/, '$1');
const { username, password } = doc.pull_option as any;
@@ -18,29 +20,24 @@ export function formatUrl(doc: Subscription) {
}
export function formatCommand(doc: Subscription, url?: string) {
let command = `SUB_ID=${doc.id} ql `;
let _url = url || formatUrl(doc).url;
const {
type,
whitelist,
blacklist,
dependences,
branch,
extensions,
proxy,
autoAddCron,
autoDelCron,
} = doc;
if (type === 'file') {
command += `raw "${_url}" "${proxy || ''}" "${
isNil(autoAddCron) ? true : Boolean(autoAddCron)
}" "${isNil(autoDelCron) ? true : Boolean(autoDelCron)}"`;
} else {
command += `repo "${_url}" "${whitelist || ''}" "${blacklist || ''}" "${
dependences || ''
}" "${branch || ''}" "${extensions || ''}" "${proxy || ''}" "${
isNil(autoAddCron) ? true : Boolean(autoAddCron)
}" "${isNil(autoDelCron) ? true : Boolean(autoDelCron)}"`;
}
return command;
const args =
doc.type === 'file'
? ['raw', url || formatUrl(doc).url, doc.proxy || '']
: [
'repo',
url || formatUrl(doc).url,
doc.whitelist || '',
doc.blacklist || '',
doc.dependences || '',
doc.branch || '',
doc.extensions || '',
doc.proxy || '',
];
args.push(
String(isNil(doc.autoAddCron) ? true : Boolean(doc.autoAddCron)),
String(isNil(doc.autoDelCron) ? true : Boolean(doc.autoDelCron)),
);
return `SUB_ID=${shellQuote(String(doc.id))} ql ${args
.map((arg) => shellQuote(arg!))
.join(' ')}`;
}
+62 -63
View File
@@ -13,6 +13,7 @@ import { FormData } from 'undici';
import os from 'os';
import { maybeSudo, isInContainer } from './container';
import { resolveFileAccess } from '../shared/fileAccess';
import { mirrorWriteCommand, normalizeMirror, rewriteMirrorSources } from '../shared/linuxMirror';
export * from './share';
@@ -806,75 +807,73 @@ export async function detectOS(): Promise<
return osType;
}
async function getCurrentMirrorDomain(
filePath: string,
): Promise<string | null> {
const fileContent = await fs.readFile(filePath, 'utf8');
const lines = fileContent.split('\n');
for (const line of lines) {
if (line.trim().startsWith('#')) {
continue;
}
const match = line.match(/https?:\/\/[^\/]+/);
if (match) {
return match[0];
}
}
return null;
}
async function replaceDomainInFile(
filePath: string,
oldDomainWithScheme: string,
newDomainWithScheme: string,
): Promise<void> {
let fileContent = await fs.readFile(filePath, 'utf8');
let updatedContent = fileContent.replace(
new RegExp(oldDomainWithScheme, 'g'),
newDomainWithScheme,
);
if (!newDomainWithScheme.endsWith('/')) {
newDomainWithScheme += '/';
}
await writeFileWithLock(filePath, updatedContent);
}
async function _updateLinuxMirror(
osType: string,
mirrorDomainWithScheme: string,
mirror: string,
): Promise<string> {
const defaults: Record<string, string> = {
Debian: 'http://deb.debian.org/debian',
Ubuntu: 'http://archive.ubuntu.com/ubuntu',
Alpine: 'http://dl-cdn.alpinelinux.org/alpine',
};
if (!defaults[osType])
throw new Error('Unsupported OS type for updating mirrors.');
const target = mirror || defaults[osType];
normalizeMirror(target);
const S = isInContainer() ? 'sudo ' : '';
let filePath: string, currentDomainWithScheme: string | null;
switch (osType) {
case 'Debian':
filePath = '/etc/apt/sources.list.d/debian.sources';
currentDomainWithScheme = await getCurrentMirrorDomain(filePath);
if (currentDomainWithScheme) {
return `${S}sed -i 's|${currentDomainWithScheme}|${mirrorDomainWithScheme || 'http://deb.debian.org'}|g' ${filePath} || (${S}mkdir -p /etc/apt/sources.list.d && echo -e "Types: deb\\nURIs: ${mirrorDomainWithScheme || 'http://deb.debian.org'}\\nSuites: \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2) \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2)-updates\\nComponents: main\\nSigned-By: /usr/share/keyrings/debian-archive-keyring.gpg" | ${S}tee ${filePath}) && ${S}apt-get update`;
} else {
return `${S}mkdir -p /etc/apt/sources.list.d && echo -e "Types: deb\\nURIs: ${mirrorDomainWithScheme || 'http://deb.debian.org'}\\nSuites: \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2) \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2)-updates\\nComponents: main\\nSigned-By: /usr/share/keyrings/debian-archive-keyring.gpg" | ${S}tee ${filePath} && ${S}apt-get update`;
const update =
osType === 'Alpine' ? 'apk update' : `${S}apt-get update --error-on=any`;
let file =
osType === 'Alpine'
? '/etc/apk/repositories'
: `/etc/apt/sources.list.d/${osType.toLowerCase()}.sources`;
let content = '';
try {
content = await fs.readFile(file, 'utf8');
} catch (error: any) {
if (error.code !== 'ENOENT') throw error;
if (osType !== 'Alpine') {
try {
file = '/etc/apt/sources.list';
content = await fs.readFile(file, 'utf8');
} catch (legacyError: any) {
if (legacyError.code !== 'ENOENT') throw legacyError;
file = `/etc/apt/sources.list.d/${osType.toLowerCase()}.sources`;
}
case 'Ubuntu':
filePath = '/etc/apt/sources.list.d/ubuntu.sources';
currentDomainWithScheme = await getCurrentMirrorDomain(filePath);
if (currentDomainWithScheme) {
return `${S}sed -i 's|${currentDomainWithScheme}|${mirrorDomainWithScheme || 'http://archive.ubuntu.com'}|g' ${filePath} || (${S}mkdir -p /etc/apt/sources.list.d && echo -e "Types: deb\\nURIs: ${mirrorDomainWithScheme || 'http://archive.ubuntu.com'}\\nSuites: \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2) \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2)-updates \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2)-backports\\nComponents: main restricted universe multiverse\\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg" | ${S}tee ${filePath}) && ${S}apt-get update`;
} else {
return `${S}mkdir -p /etc/apt/sources.list.d && echo -e "Types: deb\\nURIs: ${mirrorDomainWithScheme || 'http://archive.ubuntu.com'}\\nSuites: \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2) \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2)-updates \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2)-backports\\nComponents: main restricted universe multiverse\\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg" | ${S}tee ${filePath} && ${S}apt-get update`;
}
case 'Alpine':
filePath = '/etc/apk/repositories';
currentDomainWithScheme = await getCurrentMirrorDomain(filePath);
if (currentDomainWithScheme) {
return `sed -i 's|${currentDomainWithScheme}|${mirrorDomainWithScheme || 'http://dl-cdn.alpinelinux.org'}|g' ${filePath} || (mkdir -p /etc/apk && echo -e "\\$(grep VERSION_ID /etc/os-release | cut -d= -f2 | cut -d. -f1,2)/main\\n\\$(grep VERSION_ID /etc/os-release | cut -d= -f2 | cut -d. -f1,2)/community" | sed "s|^|${mirrorDomainWithScheme || 'http://dl-cdn.alpinelinux.org'}/alpine/v|" | tee ${filePath}) && apk update`;
} else {
return `mkdir -p /etc/apk && echo -e "\\$(grep VERSION_ID /etc/os-release | cut -d= -f2 | cut -d. -f1,2)/main\\n\\$(grep VERSION_ID /etc/os-release | cut -d= -f2 | cut -d. -f1,2)/community" | sed "s|^|${mirrorDomainWithScheme || 'http://dl-cdn.alpinelinux.org'}/alpine/v|" | tee ${filePath} && apk update`;
}
default:
throw Error('Unsupported OS type for updating mirrors.');
}
}
if (!content.trim()) {
const release = await fs.readFile('/etc/os-release', 'utf8');
if (osType === 'Alpine') {
const version = release.match(/^VERSION_ID=["']?(\d+\.\d+)/m)?.[1];
if (!version)
throw new Error('Cannot determine Alpine repository version');
content = `${defaults.Alpine}/v${version}/main\n${defaults.Alpine}/v${version}/community\n`;
} else {
// The fallback may have read an empty legacy .list. Deb822 stanzas must
// be written to a .sources file, leaving the legacy file untouched.
file = `/etc/apt/sources.list.d/${osType.toLowerCase()}.sources`;
const codename = release.match(/^VERSION_CODENAME=["']?([\w-]+)/m)?.[1];
if (!codename)
throw new Error('Cannot determine APT repository codename');
const ubuntu = osType === 'Ubuntu';
content = `Types: deb\nURIs: ${
defaults[osType]
}\nSuites: ${codename} ${codename}-updates${
ubuntu ? ` ${codename}-backports` : ''
}\nComponents: ${
ubuntu ? 'main restricted universe multiverse' : 'main'
}\nSigned-By: /usr/share/keyrings/${osType.toLowerCase()}-archive-keyring.gpg\n`;
if (!ubuntu)
content += `\nTypes: deb\nURIs: http://security.debian.org/debian-security\nSuites: ${codename}-security\nComponents: main\nSigned-By: /usr/share/keyrings/debian-archive-keyring.gpg\n`;
}
}
return mirrorWriteCommand(
file,
rewriteMirrorSources(content, target),
S,
update,
);
}
export async function updateLinuxMirrorFile(mirror: string): Promise<string> {
+10 -1
View File
@@ -14,6 +14,7 @@ import { AuthInfo } from '../data/system';
import path from 'path';
import { t } from '../shared/i18n';
import { AppScope } from '../data/open';
import uniqueConstraintError from '../middlewares/uniqueConstraintError';
import protectedPathCase from '../middlewares/protectedPathCase';
function resolveTrustProxy(value = process.env.QL_TRUST_PROXY) {
@@ -178,7 +179,14 @@ export default ({ app }: { app: Application }) => {
app.use(rewrite('/open/*', '/api/$1'));
app.use(config.api.prefix, routes());
app.get('*', (_, res, next) => {
app.get('*', (req, res, next) => {
// Don't serve index.html for API routes
if (
req.path === config.api.prefix ||
req.path.startsWith(`${config.api.prefix}/`)
) {
return res.status(404).json({ code: 404, message: 'Not Found' });
}
const indexPath = path.join(frontendPath, 'index.html');
res.sendFile(indexPath, (err) => {
if (err) {
@@ -190,6 +198,7 @@ export default ({ app }: { app: Application }) => {
});
app.use(errors());
app.use(uniqueConstraintError);
app.use(
(
+12 -26
View File
@@ -13,12 +13,11 @@ import { AuthDataType, SystemModel } from '../data/system';
import SystemService from '../services/system';
import UserService from '../services/user';
import { writeFile, readFile } from 'fs/promises';
import { createRandomString, fileExist, isDemoEnv, safeJSONParse } from '../config/util';
import { fileExist, isDemoEnv, safeJSONParse } from '../config/util';
import OpenService from '../services/open';
import { shareStore } from '../shared/store';
import Logger from './logger';
import cronClient from '../schedule/client';
import { AppModel } from '../data/open';
import { InstanceStatus, RunningInstanceModel } from '../data/runningInstance';
import { setLang, systemLang } from '../shared/i18n';
@@ -31,24 +30,7 @@ export default async () => {
const openService = Container.get(OpenService);
// 初始化增加系统配置
let systemApp = (
await AppModel.findOne({
where: { name: 'system' },
})
)?.get({ plain: true });
if (!systemApp) {
systemApp = await AppModel.create({
name: 'system',
scopes: ['crons', 'system', 'dashboard'],
client_id: createRandomString(12, 12),
client_secret: createRandomString(24, 24),
});
} else if (!systemApp.scopes.includes('dashboard')) {
await AppModel.update(
{ scopes: [...systemApp.scopes, 'dashboard'] },
{ where: { name: 'system' } },
);
}
await openService.initializeSystemApp();
const [systemConfig] = await SystemModel.findOrCreate({
where: { type: AuthDataType.systemConfig },
});
@@ -144,7 +126,11 @@ export default async () => {
// 清空所有运行中的实例记录(服务重启后进程已不存在)
await RunningInstanceModel.update(
{ status: InstanceStatus.stopped },
{
status: InstanceStatus.stopped,
finished_at: Math.floor(Date.now() / 1000),
// The previous process's exit code cannot be recovered after a restart.
},
{ where: { status: InstanceStatus.running } },
);
@@ -237,15 +223,15 @@ export default async () => {
} catch { }
// 初始化保存一次ck和定时任务数据
cronClient.readiness.configure(() => cronService.autosave_crontab(true));
cronClient.readiness.configure(async () => {
await cronService.autosave_crontab(true);
await systemService.restoreCronConcurrency();
});
await cronClient.readiness.recover();
await envService.set_envs();
const authInfo = await userService.getAuthInfo();
const apps = await openService.findApps();
await shareStore.updateAuthInfo(authInfo);
if (apps?.length) {
await shareStore.updateApps(apps);
}
await openService.refreshApps();
};
+10 -1
View File
@@ -7,6 +7,8 @@ import config from '../config';
import { fileExist } from '../config/util';
import { join } from 'path';
import { t } from '../shared/i18n';
import Logger from './logger';
import { assertSubscriptionAlias } from '../shared/subscriptionPath';
export default async () => {
const systemService = Container.get(SystemService);
@@ -24,6 +26,7 @@ export default async () => {
if (await fileExist(tokenFile)) {
tokenCommand = `node ${tokenFile}`;
}
tokenCommand += ' --renew';
const cron = {
id: NaN,
name: t('生成token'),
@@ -70,6 +73,12 @@ export default async () => {
await subscriptionService.setSshConfig();
const subs = await subscriptionService.list();
for (const sub of subs) {
subscriptionService.handleTask(sub.get({ plain: true }), !sub.is_disabled);
try {
const doc = sub.get({ plain: true });
assertSubscriptionAlias(doc.alias);
await subscriptionService.handleTask(doc, !sub.is_disabled);
} catch (error) {
Logger.warn('跳过订阅 %s 的无效定时配置: %s', sub.id, error);
}
}
};
+14
View File
@@ -0,0 +1,14 @@
import { ErrorRequestHandler } from 'express';
import { UniqueConstraintError } from 'sequelize';
import { t } from '../shared/i18n';
// Handle the database constraint itself so concurrent writes are covered too.
const uniqueConstraintError: ErrorRequestHandler = (err, req, res, next) => {
if (!(err instanceof UniqueConstraintError)) return next(err);
// Constraint details can contain environment values or application credentials.
res
.status(409)
.json({ code: 409, message: t('资源已存在,请检查重复的名称或值') });
};
export default uniqueConstraintError;
+4
View File
@@ -5,6 +5,7 @@ package com.ql.cron;
service Cron {
rpc addCron(AddCronRequest) returns (AddCronResponse);
rpc delCron(DeleteCronRequest) returns (DeleteCronResponse);
rpc setConcurrency(SetConcurrencyRequest) returns (AddCronResponse);
}
message ISchedule { string schedule = 1; }
@@ -27,3 +28,6 @@ message AddCronResponse {}
message DeleteCronRequest { repeated string ids = 1; }
message DeleteCronResponse {}
// Zero restores the default limit.
message SetConcurrencyRequest { int32 concurrency = 1; }
+89 -1
View File
@@ -1,7 +1,7 @@
// Code generated by protoc-gen-ts_proto. DO NOT EDIT.
// versions:
// protoc-gen-ts_proto v2.6.1
// protoc v3.17.3
// protoc v5.28.3
// source: back/protos/cron.proto
/* eslint-disable */
@@ -48,6 +48,11 @@ export interface DeleteCronRequest {
export interface DeleteCronResponse {
}
/** Zero restores the default limit. */
export interface SetConcurrencyRequest {
concurrency: number;
}
function createBaseISchedule(): ISchedule {
return { schedule: "" };
}
@@ -452,6 +457,64 @@ export const DeleteCronResponse: MessageFns<DeleteCronResponse> = {
},
};
function createBaseSetConcurrencyRequest(): SetConcurrencyRequest {
return { concurrency: 0 };
}
export const SetConcurrencyRequest: MessageFns<SetConcurrencyRequest> = {
encode(message: SetConcurrencyRequest, writer: BinaryWriter = new BinaryWriter()): BinaryWriter {
if (message.concurrency !== 0) {
writer.uint32(8).int32(message.concurrency);
}
return writer;
},
decode(input: BinaryReader | Uint8Array, length?: number): SetConcurrencyRequest {
const reader = input instanceof BinaryReader ? input : new BinaryReader(input);
let end = length === undefined ? reader.len : reader.pos + length;
const message = createBaseSetConcurrencyRequest();
while (reader.pos < end) {
const tag = reader.uint32();
switch (tag >>> 3) {
case 1: {
if (tag !== 8) {
break;
}
message.concurrency = reader.int32();
continue;
}
}
if ((tag & 7) === 4 || tag === 0) {
break;
}
reader.skip(tag & 7);
}
return message;
},
fromJSON(object: any): SetConcurrencyRequest {
return { concurrency: isSet(object.concurrency) ? globalThis.Number(object.concurrency) : 0 };
},
toJSON(message: SetConcurrencyRequest): unknown {
const obj: any = {};
if (message.concurrency !== 0) {
obj.concurrency = Math.round(message.concurrency);
}
return obj;
},
create<I extends Exact<DeepPartial<SetConcurrencyRequest>, I>>(base?: I): SetConcurrencyRequest {
return SetConcurrencyRequest.fromPartial(base ?? ({} as any));
},
fromPartial<I extends Exact<DeepPartial<SetConcurrencyRequest>, I>>(object: I): SetConcurrencyRequest {
const message = createBaseSetConcurrencyRequest();
message.concurrency = object.concurrency ?? 0;
return message;
},
};
export type CronService = typeof CronService;
export const CronService = {
addCron: {
@@ -472,11 +535,21 @@ export const CronService = {
responseSerialize: (value: DeleteCronResponse) => Buffer.from(DeleteCronResponse.encode(value).finish()),
responseDeserialize: (value: Buffer) => DeleteCronResponse.decode(value),
},
setConcurrency: {
path: "/com.ql.cron.Cron/setConcurrency",
requestStream: false,
responseStream: false,
requestSerialize: (value: SetConcurrencyRequest) => Buffer.from(SetConcurrencyRequest.encode(value).finish()),
requestDeserialize: (value: Buffer) => SetConcurrencyRequest.decode(value),
responseSerialize: (value: AddCronResponse) => Buffer.from(AddCronResponse.encode(value).finish()),
responseDeserialize: (value: Buffer) => AddCronResponse.decode(value),
},
} as const;
export interface CronServer extends UntypedServiceImplementation {
addCron: handleUnaryCall<AddCronRequest, AddCronResponse>;
delCron: handleUnaryCall<DeleteCronRequest, DeleteCronResponse>;
setConcurrency: handleUnaryCall<SetConcurrencyRequest, AddCronResponse>;
}
export interface CronClient extends Client {
@@ -510,6 +583,21 @@ export interface CronClient extends Client {
options: Partial<CallOptions>,
callback: (error: ServiceError | null, response: DeleteCronResponse) => void,
): ClientUnaryCall;
setConcurrency(
request: SetConcurrencyRequest,
callback: (error: ServiceError | null, response: AddCronResponse) => void,
): ClientUnaryCall;
setConcurrency(
request: SetConcurrencyRequest,
metadata: Metadata,
callback: (error: ServiceError | null, response: AddCronResponse) => void,
): ClientUnaryCall;
setConcurrency(
request: SetConcurrencyRequest,
metadata: Metadata,
options: Partial<CallOptions>,
callback: (error: ServiceError | null, response: AddCronResponse) => void,
): ClientUnaryCall;
}
export const CronClient = makeGenericClientConstructor(CronService, "com.ql.cron.Cron") as unknown as {
+52 -73
View File
@@ -1,6 +1,6 @@
import { ServerUnaryCall, sendUnaryData, status } from '@grpc/grpc-js';
import { AddCronRequest, AddCronResponse } from '../protos/cron';
import nodeSchedule from 'node-schedule';
import { createCronJob, CronJob } from '../shared/cronScheduler';
import { isValidCronSchedule } from '../shared/cronSchedule';
import { scheduleStacks } from './data';
import { runCron } from '../shared/runCron';
@@ -24,11 +24,7 @@ const addCron = (
if (!isValidCronField(schedule)) {
validationErrors.push(
tf(
'任务ID %s: 无效的 cron 表达式 "%s"',
String(id),
schedule,
),
tf('任务ID %s: 无效的 cron 表达式 "%s"', String(id), schedule),
);
}
@@ -56,79 +52,62 @@ const addCron = (
return;
}
// Recovery replaces the whole snapshot, including deletions and disabled jobs.
// Validation above must finish before touching the previous schedule.
if (call.request.replace) {
for (const jobs of scheduleStacks.values()) {
for (const job of jobs) job?.cancel();
// Prepare stopped jobs first: construction errors must preserve the old snapshot.
const prepared = new Map<string, CronJob[]>();
try {
for (const item of call.request.crons) {
const jobs: CronJob[] = [];
prepared.get(item.id)?.forEach((job) => job.cancel());
prepared.set(item.id, jobs);
for (const schedule of [
item.schedule,
...(item.extra_schedules || []).map((x) => x.schedule),
]) {
jobs.push(
createCronJob(
schedule,
async () => {
Logger.info('[schedule][准备运行任务] 命令: %s', item.command);
await runCron(
item.command,
item,
() => scheduleStacks.get(item.id) === jobs,
);
},
{
name: `${item.id}: ${item.name || ''}`,
logger: Logger,
start: false,
},
),
);
}
}
scheduleStacks.clear();
} catch (error) {
for (const jobs of prepared.values()) jobs.forEach((job) => job.cancel());
const err: any = new Error(
error instanceof Error ? error.message : String(error),
);
err.code = status.INVALID_ARGUMENT;
err.details = err.message;
callback(err, null);
return;
}
// ===== 第二遍:注册所有任务 =====
for (const item of call.request.crons) {
const { id, schedule, command, extra_schedules, name } = item;
// 取消该 id 已有的旧任务
if (scheduleStacks.has(id)) {
scheduleStacks.get(id)?.forEach((x) => x.cancel());
}
if (call.request.replace) {
for (const jobs of scheduleStacks.values())
jobs.forEach((job) => job?.cancel());
scheduleStacks.clear();
}
for (const [id, jobs] of prepared) {
scheduleStacks.get(id)?.forEach((job) => job.cancel());
scheduleStacks.set(id, jobs);
jobs.forEach((job) => job.start());
Logger.info(
'[schedule][创建定时任务] 任务ID: %s, 名称: %s, cron: %s, 执行命令: %s',
'[schedule][创建定时任务] 任务ID: %s, 规则数: %s',
id,
name,
schedule,
command,
jobs.length,
);
if (extra_schedules?.length) {
extra_schedules.forEach((x) => {
Logger.info(
'[schedule][创建定时任务] 任务ID: %s, 名称: %s, cron: %s, 执行命令: %s',
id,
name,
x.schedule,
command,
);
});
}
const mainJob = nodeSchedule.scheduleJob(id, schedule, async () => {
Logger.info(`[schedule][准备运行任务] 命令: ${command}`);
runCron(command, item);
});
if (!mainJob) {
Logger.warn(
'[schedule][创建定时任务] scheduleJob 返回 null(不符合预期,已通过预校验): 任务ID: %s, cron: %s',
id,
schedule,
);
}
const extraJobs = extra_schedules?.length
? extra_schedules.map((x) => {
const job = nodeSchedule.scheduleJob(id, x.schedule, async () => {
Logger.info(`[schedule][准备运行任务] 命令: ${command}`);
runCron(command, item);
});
if (!job) {
Logger.warn(
'[schedule][创建定时任务] scheduleJob 返回 null(不符合预期,已通过预校验): 任务ID: %s, cron: %s',
id,
x.schedule,
);
}
return job;
})
: [];
// 过滤 null(兜底保护,正常情况下预校验已拦截)
const jobs = [mainJob, ...extraJobs].filter((x) => x != null);
if (jobs.length > 0) {
scheduleStacks.set(id, jobs);
}
}
callback(null, null);
+26 -1
View File
@@ -62,16 +62,41 @@ class Client {
return this._client;
}
async setConcurrency(concurrency: number): Promise<void> {
await this.waitForReady(2000);
return new Promise((resolve, reject) => {
this.client.setConcurrency(
{ concurrency }, new Metadata(), { deadline: Date.now() + 5000 },
(err) => {
if (err) {
if (err.code === status.UNAVAILABLE || err.code === status.DEADLINE_EXCEEDED) {
this.readiness.invalidate();
Object.assign(err, { status: 503 });
}
return reject(err);
}
resolve();
},
);
});
}
async addCron(
request: AddCronRequest['crons'],
replace = false
): Promise<AddCronResponse> {
await this.waitForReady(2000);
// Include every rule in a recovery snapshot, but bound a stuck worker.
const ruleCount = request.reduce(
(count, item) => count + 1 + (item.extra_schedules?.length || 0),
0,
);
const registrationTimeoutMs = Math.min(120000, 5000 + ruleCount * 5);
return new Promise((resolve, reject) => {
this.client.addCron(
{ crons: request, replace },
new Metadata(),
{ deadline: Date.now() + 5000 },
{ deadline: Date.now() + registrationTimeoutMs },
(err, res) => {
if (err) {
if (err.code === status.UNAVAILABLE || err.code === status.DEADLINE_EXCEEDED) {
+2 -2
View File
@@ -1,6 +1,6 @@
import nodeSchedule from 'node-schedule';
import { CronJob } from '../shared/cronScheduler';
import { ToadScheduler } from 'toad-scheduler';
export const scheduleStacks = new Map<string, nodeSchedule.Job[]>();
export const scheduleStacks = new Map<string, CronJob[]>();
export const intervalSchedule = new ToadScheduler();
+1 -1
View File
@@ -13,7 +13,7 @@ const delCron = (
'[schedule][取消定时任务] 任务ID: %s',
id,
);
// 过滤掉 nodeSchedule.scheduleJob() 对无效表达式返回的 null,
// 防御性过滤历史调度栈中的空任务,
// 否则对 null 调 cancel() 会让整个取消流程抛出 UNKNOWN 错误,
// 进而导致 HTTP 端的 remove() 跳过 setCrontab(),造成 crontab.list 残留。
scheduleStacks.get(id)?.filter((x) => x != null).forEach((x) => {
+26
View File
@@ -0,0 +1,26 @@
import { ServerUnaryCall, sendUnaryData, status } from '@grpc/grpc-js';
import os from 'os';
import { AddCronResponse, SetConcurrencyRequest } from '../protos/cron';
import taskLimit from '../shared/pLimit';
export async function setConcurrency(
call: ServerUnaryCall<SetConcurrencyRequest, AddCronResponse>,
callback: sendUnaryData<AddCronResponse>,
) {
const limit = call.request.concurrency;
if (!Number.isInteger(limit) || limit < 0 || limit > 2147483647) {
callback(
Object.assign(new Error('Invalid concurrency'), {
code: status.INVALID_ARGUMENT,
}),
null,
);
return;
}
try {
await taskLimit.setCustomLimit(limit || Math.max(os.cpus().length, 4));
callback(null, {});
} catch (error) {
callback(error as Error, null);
}
}
+3 -4
View File
@@ -12,13 +12,12 @@ export default class ConfigService {
public async getFile(filePath: string, res: Response) {
let content = '';
if (!filePath) {
if (!filePath || filePath.startsWith('data/scripts/')) {
return res.send({ code: 403, message: t('文件无法访问') });
}
const scriptFile = filePath.startsWith('data/scripts/');
const resolved = resolveFileAccess(
scriptFile ? config.scriptPath : config.configPath,
[scriptFile ? filePath.slice('data/scripts/'.length) : filePath],
config.configPath,
[filePath],
config.blackFileList,
);
if (!resolved) {
+22 -9
View File
@@ -1,5 +1,5 @@
import { randomUUID } from 'crypto';
import { getInvalidCronSchedules } from '../shared/cronSchedule';
import { getInvalidCronSchedules, isValidCronSchedule } from '../shared/cronSchedule';
import {
withSchedulerMutation,
schedulerRegistrationError,
@@ -7,6 +7,7 @@ import {
import { Service, Inject } from 'typedi';
import winston from 'winston';
import config from '../config';
import { resolveLogPath } from '../shared/logPath';
import { Crontab, CrontabModel, CrontabStatus } from '../data/cron';
import {
RunningInstanceModel,
@@ -14,7 +15,6 @@ import {
} from '../data/runningInstance';
import { exec, execSync } from 'child_process';
import fs from 'fs/promises';
import CronExpressionParser from 'cron-parser';
import {
getFileContentByName,
fileExist,
@@ -47,12 +47,20 @@ export default class CronService {
private isNodeCron(cron: Crontab) {
const { schedule, extra_schedules } = cron;
const fields = schedule?.trim().split(/\s+/) || [];
// System crontab only receives portable numeric five-field expressions.
// Extended syntax, macros and legacy shorthand belong to node-schedule.
// Extended syntax, macros and legacy shorthand use the Node scheduler.
return (
schedule?.trim().split(/\s+/).length !== 5 ||
fields.length !== 5 ||
/[^\d\s*,/\-]/.test(schedule || '') ||
Boolean(extra_schedules?.length)
Boolean(extra_schedules?.length) ||
// BusyBox treats N/step as a single value and doesn't support Sunday=7.
fields.some((field) => field.split(',').some((part) => /^\d+\//.test(part))) ||
fields[4].includes('7') ||
// BusyBox steps DOM from zero; its full-range day/week wildcard
// handling also differs from the legacy Node parser's OR semantics.
fields[2].includes('/') ||
(fields[2] !== '*' && fields[4] !== '*')
);
}
@@ -239,6 +247,9 @@ export default class CronService {
last_execution_time: number;
exit_code?: number;
}) {
if (log_path && log_path !== '/dev/null') {
resolveLogPath(config.logPath, log_path);
}
let options: any = {
status,
pid,
@@ -918,7 +929,9 @@ export default class CronService {
truncated: false,
};
}
const absolutePath = path.resolve(config.logPath, `${doc.log_path}`);
const absolutePath = doc.log_path
? resolveLogPath(config.logPath, doc.log_path)
: '';
const logFileExist = doc.log_path && (await fileExist(absolutePath));
if (logFileExist) {
const chunk = await readLogChunk(`${absolutePath}`, options);
@@ -948,12 +961,12 @@ export default class CronService {
public async logs(id: number) {
const doc = await this.getDb({ id });
if (!doc || !doc.log_path) {
if (!doc || !doc.log_path || doc.log_path === '/dev/null') {
return [];
}
const relativeDir = path.dirname(`${doc.log_path}`);
const dir = path.resolve(config.logPath, relativeDir);
const dir = path.dirname(resolveLogPath(config.logPath, doc.log_path));
const dirExist = await fileExist(dir);
if (dirExist) {
let files = await fs.readdir(dir);
@@ -1057,7 +1070,7 @@ export default class CronService {
if (
command &&
schedule &&
CronExpressionParser.parse(schedule).hasNext()
isValidCronSchedule(schedule)
) {
const name = namePrefix + '_' + index;
+3 -4
View File
@@ -262,10 +262,9 @@ export default class EnvService {
.map((x) => x.value)
.join('&')
.replace(/\\/g, '\\\\')}`;
js_env_string += `process.env.${key}=\`${_env_value.replace(
/\`/g,
'\\`',
)}\`;\n`;
js_env_string += `process.env[${JSON.stringify(key)}]=${JSON.stringify(
group.map((x) => x.value).join('&'),
)};\n`;
py_env_string += `os.environ['${key}']='''${_env_value.replace(
/\'/g,
"\\'",
+2 -1
View File
@@ -4,6 +4,7 @@ import { HealthService } from '../protos/health';
import { ApiService } from '../protos/api';
import { addCron } from '../schedule/addCron';
import { delCron } from '../schedule/delCron';
import { setConcurrency } from '../schedule/setConcurrency';
import { check } from '../schedule/health';
import * as Api from '../schedule/api';
import Logger from '../loaders/logger';
@@ -27,7 +28,7 @@ export class GrpcServerService {
async initialize() {
try {
this.server.addService(HealthService, { check });
this.server.addService(CronService, { addCron, delCron });
this.server.addService(CronService, { addCron, delCron, setConcurrency });
this.server.addService(ApiService, Api);
const tlsConfig = await initGrpcCerts();
+1 -6
View File
@@ -1,5 +1,4 @@
import { resolveFileAccess } from '../shared/fileAccess';
import path from 'path';
import { Inject, Service } from 'typedi';
import winston from 'winston';
import config from '../config';
@@ -9,10 +8,6 @@ export default class LogService {
constructor(@Inject('logger') private logger: winston.Logger) {}
public checkFilePath(filePath: string, fileName: string) {
return resolveFileAccess(
config.logPath,
[filePath || '', fileName],
config.blackFileList,
);
return resolveFileAccess(config.logPath, [filePath || '', fileName]);
}
}
+116 -34
View File
@@ -6,27 +6,77 @@ import { v4 as uuidV4 } from 'uuid';
import sequelize, { Op } from 'sequelize';
import { shareStore } from '../shared/store';
import { t } from '../shared/i18n';
import { lock } from 'proper-lockfile';
import path from 'path';
import os from 'os';
import { writeFile, rename, rm, stat, chmod } from 'fs/promises';
import config from '../config';
@Service()
export default class OpenService {
constructor(@Inject('logger') private logger: winston.Logger) {}
// Every database mutation and full authentication-cache publication shares
// this cross-process lock. Lock a separate marker, not token.json itself.
private async withAppsLock<T>(operation: () => Promise<T>): Promise<T> {
const target = path.join(config.configPath, '.apps-state');
const release = await lock(target, {
realpath: false,
stale: 10_000,
retries: { retries: 10, factor: 2, minTimeout: 100, maxTimeout: 3000 },
});
try {
return await operation();
} finally {
await release();
}
}
public async initializeSystemApp(): Promise<void> {
await this.withAppsLock(async () => {
const doc = await AppModel.findOne({ where: { name: 'system' } });
const app = doc?.get({ plain: true });
if (!app) {
await this.insert({
name: 'system',
scopes: ['crons', 'system', 'dashboard'],
client_id: createRandomString(12, 12),
client_secret: createRandomString(24, 24),
});
} else if (!app.scopes.includes('dashboard')) {
await AppModel.update(
{ scopes: [...app.scopes, 'dashboard'] },
{ where: { id: app.id } },
);
}
await shareStore.updateApps(await this.find({}));
});
}
public async refreshApps(): Promise<void> {
await this.withAppsLock(async () => {
await shareStore.updateApps(await this.find({}));
});
}
public async findApps(): Promise<App[] | null> {
const docs = await this.find({});
return docs;
}
public async create(payload: App): Promise<App> {
const tab = { ...payload };
tab.client_id = createRandomString(12, 12);
tab.client_secret = createRandomString(24, 24);
const doc = await this.insert(tab);
const apps = await this.find({});
await shareStore.updateApps(apps);
return { ...doc, tokens: [] };
return this.withAppsLock(async () => {
const tab = { ...payload };
tab.client_id = createRandomString(12, 12);
tab.client_secret = createRandomString(24, 24);
const doc = await this.insert(tab);
const apps = await this.find({});
await shareStore.updateApps(apps);
return { ...doc, tokens: [] };
});
}
public async insert(payload: App): Promise<App> {
private async insert(payload: App): Promise<App> {
const doc = await AppModel.create(payload, { returning: true });
return doc.get({ plain: true });
}
@@ -41,10 +91,12 @@ export default class OpenService {
}
private async updateDb(payload: Partial<App>): Promise<App> {
await AppModel.update(payload, { where: { id: payload.id } });
const apps = await this.find({});
await shareStore.updateApps(apps);
return apps?.find((x) => x.id === payload.id) as App;
return this.withAppsLock(async () => {
await AppModel.update(payload, { where: { id: payload.id } });
const apps = await this.find({});
await shareStore.updateApps(apps);
return apps?.find((x) => x.id === payload.id) as App;
});
}
public async getDb(query: Record<string, any>): Promise<App> {
@@ -56,9 +108,11 @@ export default class OpenService {
}
public async remove(ids: number[]) {
await AppModel.destroy({ where: { id: ids } });
const apps = await this.find({});
await shareStore.updateApps(apps);
return this.withAppsLock(async () => {
await AppModel.destroy({ where: { id: ids } });
const apps = await this.find({});
await shareStore.updateApps(apps);
});
}
public async resetSecret(id: number): Promise<App> {
@@ -112,7 +166,15 @@ export default class OpenService {
return docs.map((x) => x.get({ plain: true }));
}
public async authToken({
public async authToken(credentials: {
client_id: string;
client_secret: string;
}): Promise<any> {
return this.withAppsLock(() => this.issueToken(credentials));
}
// Caller holds withAppsLock through database update and cache publication.
private async issueToken({
client_id,
client_secret,
}: {
@@ -156,27 +218,47 @@ export default class OpenService {
}
}
public async generateSystemToken(): Promise<{
public async generateSystemToken(renew = false): Promise<{
value: string;
expiration: number;
}> {
const apps = await shareStore.getApps();
const systemApp = apps?.find((x) => x.name === 'system');
if (!systemApp) {
throw new Error('system app not found');
}
const now = Math.round(Date.now() / 1000);
const currentToken = systemApp.tokens?.find((x) => x.expiration > now);
if (currentToken) {
return currentToken;
}
const { data } = await this.authToken({
client_id: systemApp.client_id,
client_secret: systemApp.client_secret,
return this.withAppsLock(async () => {
// Read after acquiring the lock; a concurrent issuer may have renewed.
const systemApp = await this.getDb({ name: 'system' });
const now = Math.round(Date.now() / 1000);
let appToken = systemApp.tokens
?.filter((token) => token.expiration > now)
.sort((left, right) => right.expiration - left.expiration)[0];
if (renew || !appToken) {
const { data } = await this.issueToken({
client_id: systemApp.client_id,
client_secret: systemApp.client_secret,
});
appToken = { value: data.token, expiration: data.expiration };
} else {
// Recover a prior database commit whose cache publication failed.
await shareStore.updateApps(await this.find({}));
}
const tokenFile = path.join(config.configPath, 'token.json');
const temporaryFile = `${tokenFile}.${process.pid}.tmp`;
const mode = await stat(tokenFile).then(
(file) => file.mode & 0o777,
(error: NodeJS.ErrnoException) => {
if (error.code === 'ENOENT') return 0o600;
throw error;
},
);
try {
await writeFile(temporaryFile, `${JSON.stringify(appToken)}${os.EOL}`, {
mode,
});
await chmod(temporaryFile, mode);
// Shell readers see a complete old or new document, never a partial one.
await rename(temporaryFile, tokenFile);
} finally {
await rm(temporaryFile, { force: true });
}
return appToken;
});
return {
...data,
value: data.token,
};
}
}
+33 -14
View File
@@ -1,6 +1,6 @@
import { Service, Inject } from 'typedi';
import winston from 'winston';
import nodeSchedule from 'node-schedule';
import { createCronJob, CronJob } from '../shared/cronScheduler';
import { ChildProcessWithoutNullStreams } from 'child_process';
import {
ToadScheduler,
@@ -11,7 +11,11 @@ import {
import dayjs from 'dayjs';
import taskLimit from '../shared/pLimit';
import { spawn } from 'cross-spawn';
import { observeChildProcess, asError, ProcessResult } from '../shared/childProcess';
import {
observeChildProcess,
asError,
ProcessResult,
} from '../shared/childProcess';
export interface ScheduleTaskType {
id?: number;
@@ -38,7 +42,7 @@ export interface TaskCallbacks {
@Service()
export default class ScheduleService {
private scheduleStacks = new Map<string, nodeSchedule.Job>();
private scheduleStacks = new Map<string, CronJob>();
private intervalSchedule = new ToadScheduler();
@@ -159,18 +163,33 @@ export default class ScheduleService {
command,
);
this.scheduleStacks.set(
_id,
nodeSchedule.scheduleJob(_id, schedule, async () => {
this.runTask(command, callbacks, {
name,
if (schedule) {
try {
const job = createCronJob(
schedule,
command,
id: _id,
runOrigin,
});
}),
);
() =>
this.runTask(command, callbacks, {
name,
schedule,
command,
id: _id,
runOrigin,
}),
{ name: `${_id}: ${name || ''}`, logger: this.logger, start: false },
);
this.scheduleStacks.get(_id)?.cancel();
this.scheduleStacks.set(_id, job);
job.start();
} catch (error) {
// A persisted invalid subscription must not become an unhandled startup rejection.
this.logger.warn(
'[panel][跳过无效定时任务] 任务ID: %s, cron: %s, 错误: %s',
_id,
schedule,
asError(error).message,
);
}
}
if (runImmediately) {
this.runTask(command, callbacks, {
+1 -5
View File
@@ -66,11 +66,7 @@ export default class ScriptService {
}
public checkFilePath(filePath: string, fileName: string) {
return resolveFileAccess(
config.scriptPath,
[filePath || '', fileName],
config.blackFileList,
);
return resolveFileAccess(config.scriptPath, [filePath || '', fileName]);
}
public async getFile(filePath: string, fileName: string) {
+12
View File
@@ -0,0 +1,12 @@
import path from 'path';
import { Service } from 'typedi';
import config from '../config';
import { ScriptHistory } from '../shared/scriptHistory';
/** Panel configuration and lifecycle; the storage core stays independently testable. */
@Service()
export default class ScriptHistoryService extends ScriptHistory {
constructor() {
super(config.scriptPath, path.join(config.dataPath, 'script-history'));
}
}
+43 -23
View File
@@ -8,10 +8,22 @@ import { formatUrl } from '../config/subscription';
import config from '../config';
import { fileExist, rmPath } from '../config/util';
import { writeFileWithLock } from '../shared/utils';
import {
getSubscriptionSshAlias,
resolveSubscriptionPath,
} from '../shared/subscriptionPath';
@Service()
export default class SshKeyService {
private homedir = os.homedir();
// OpenSSH uses the effective user's passwd entry, not the HOME override.
private get homedir(): string {
try {
return os.userInfo().homedir;
} catch {
// Custom container UIDs may have no passwd entry. Keep startup working.
return os.homedir();
}
}
private sshPath = config.sshdPath;
private sshConfigFilePath = path.resolve(this.homedir, '.ssh', 'config');
private sshConfigHeader = `Include ${path.join(this.sshPath, '*.config')}`;
@@ -21,6 +33,10 @@ export default class SshKeyService {
}
private async initSshConfigFile() {
await fs.mkdir(path.dirname(this.sshConfigFilePath), {
recursive: true,
mode: 0o700,
});
let config = '';
const _exist = await fileExist(this.sshConfigFilePath);
if (_exist) {
@@ -42,7 +58,7 @@ export default class SshKeyService {
key: string,
): Promise<void> {
try {
const filePath = path.join(this.sshPath, alias);
const filePath = resolveSubscriptionPath(this.sshPath, alias);
try {
await rmPath(filePath);
} catch { }
@@ -54,7 +70,7 @@ export default class SshKeyService {
private async removePrivateKeyFile(alias: string): Promise<void> {
try {
const filePath = path.join(this.sshPath, alias);
const filePath = resolveSubscriptionPath(this.sshPath, alias);
await rmPath(filePath);
} catch (error) {
this.logger.error('删除私钥文件失败', error);
@@ -66,18 +82,19 @@ export default class SshKeyService {
host: string,
proxy?: string,
) {
if (!/^[a-zA-Z0-9.-]+$/.test(host) ||
(proxy && !/^[a-zA-Z0-9.:[\]-]+$/.test(proxy))) {
throw Object.assign(new Error('Invalid SSH host or proxy'), { status: 400 });
}
if (host === 'github.com') {
host = `ssh.github.com\n Port 443\n HostkeyAlgorithms +ssh-rsa`;
}
const proxyStr = proxy
? ` ProxyCommand nc -v -x ${proxy} %h %p 2>/dev/null\n`
: '';
const config = `Host ${alias}\n Hostname ${host}\n IdentityFile ${path.join(
this.sshPath,
alias,
)}\n StrictHostKeyChecking no\n${proxyStr}`;
const config = `Host ${alias}\n Hostname ${host}\n IdentityFile ${resolveSubscriptionPath(this.sshPath, alias)}\n StrictHostKeyChecking no\n${proxyStr}`;
await writeFileWithLock(
`${path.join(this.sshPath, `${alias}.config`)}`,
`${resolveSubscriptionPath(this.sshPath, alias, '.config')}`,
config,
{
encoding: 'utf8',
@@ -88,7 +105,7 @@ export default class SshKeyService {
private async removeSshConfig(alias: string) {
try {
const filePath = path.join(this.sshPath, `${alias}.config`);
const filePath = resolveSubscriptionPath(this.sshPath, alias, '.config');
await rmPath(filePath);
} catch (error) {
this.logger.error(`删除ssh配置文件${alias}失败`, error);
@@ -101,6 +118,7 @@ export default class SshKeyService {
host: string,
proxy?: string,
): Promise<void> {
alias = getSubscriptionSshAlias(alias);
await this.generatePrivateKeyFile(alias, key);
await this.generateSingleSshConfig(alias, host, proxy);
}
@@ -110,6 +128,7 @@ export default class SshKeyService {
host: string,
proxy?: string,
): Promise<void> {
alias = getSubscriptionSshAlias(alias);
await this.removePrivateKeyFile(alias);
await this.removeSshConfig(alias);
}
@@ -117,15 +136,19 @@ export default class SshKeyService {
public async setSshConfig(docs: Subscription[]) {
for (const doc of docs) {
if (doc.type === 'private-repo' && doc.pull_type === 'ssh-key') {
const { alias, proxy } = doc;
const { host } = formatUrl(doc);
await this.removePrivateKeyFile(alias);
await this.removeSshConfig(alias);
await this.generatePrivateKeyFile(
alias,
(doc.pull_option as any).private_key,
);
await this.generateSingleSshConfig(alias, host, proxy);
try {
const alias = getSubscriptionSshAlias(doc.alias);
const { host } = formatUrl(doc);
await this.removePrivateKeyFile(alias);
await this.removeSshConfig(alias);
await this.generatePrivateKeyFile(
alias,
(doc.pull_option as any).private_key,
);
await this.generateSingleSshConfig(alias, host, doc.proxy);
} catch (error) {
this.logger.warn('跳过订阅 %s 的无效 SSH 配置: %s', doc.id, error);
}
}
}
}
@@ -144,12 +167,9 @@ export default class SshKeyService {
private async generateGlobalSshConfig(alias: string) {
// Create a config that matches all hosts, making this key globally available
const config = `Host *\n IdentityFile ${path.join(
this.sshPath,
alias,
)}\n StrictHostKeyChecking no\n`;
const config = `Host *\n IdentityFile ${resolveSubscriptionPath(this.sshPath, alias)}\n StrictHostKeyChecking no\n`;
await writeFileWithLock(
`${path.join(this.sshPath, `${alias}.config`)}`,
`${resolveSubscriptionPath(this.sshPath, alias, '.config')}`,
config,
{
encoding: 'utf8',
+107 -32
View File
@@ -20,7 +20,7 @@ import {
} from '../config/util';
import fs from 'fs/promises';
import { FindOptions, Op } from 'sequelize';
import path, { join } from 'path';
import path from 'path';
import ScheduleService, { TaskCallbacks } from './schedule';
import { SimpleIntervalSchedule } from 'toad-scheduler';
import SockService from './sock';
@@ -30,6 +30,14 @@ import dayjs from 'dayjs';
import { LOG_END_SYMBOL } from '../config/const';
import { formatCommand, formatUrl } from '../config/subscription';
import { CrontabModel } from '../data/cron';
import { assertSubscriptionAlias } from '../shared/subscriptionPath';
import { resolveLogPath } from '../shared/logPath';
import {
getSubscriptionStorageCandidates,
isSubscriptionStorageName,
resolveSubscriptionStoragePath,
} from '../shared/subscriptionStorage';
import { withSubscriptionMutation } from '../shared/subscriptionMutationLock';
import CrontabService from './cron';
import taskLimit from '../shared/pLimit';
import { logStreamManager } from '../shared/logStreamManager';
@@ -88,9 +96,11 @@ export default class SubscriptionService {
needCreate = true,
runImmediately = false,
) {
const { url } = formatUrl(doc);
doc.command = formatCommand(doc, url as string);
// Cancelling by ID must also work for invalid historical subscriptions.
if (needCreate) {
const { url } = formatUrl(doc);
doc.command = formatCommand(doc, url as string);
}
if (doc.schedule_type === 'crontab') {
this.scheduleService.cancelCronTask(doc as any);
@@ -222,11 +232,14 @@ export default class SubscriptionService {
}
public async create(payload: Subscription): Promise<Subscription> {
const tab = new Subscription(payload);
const doc = await this.insert(tab);
await this.handleTask(doc.get({ plain: true }));
await this.setSshConfig();
return doc;
assertSubscriptionAlias(payload.alias);
return withSubscriptionMutation(async () => {
const tab = new Subscription(payload);
const doc = await this.insert(tab);
await this.handleTask(doc.get({ plain: true }));
await this.setSshConfig();
return doc;
});
}
public async insert(payload: Subscription): Promise<SubscriptionInstance> {
@@ -234,12 +247,27 @@ export default class SubscriptionService {
}
public async update(payload: Subscription): Promise<Subscription> {
const doc = await this.getDb({ id: payload.id });
const tab = new Subscription({ ...doc, ...payload });
const newDoc = await this.updateDb(tab);
await this.handleTask(newDoc, !newDoc.is_disabled);
await this.setSshConfig();
return newDoc;
assertSubscriptionAlias(payload.alias);
return withSubscriptionMutation(async () => {
const doc = await this.getDb({ id: payload.id });
const tab = new Subscription({ ...doc, ...payload });
const newDoc = await this.updateDb(tab);
await this.handleTask(newDoc, !newDoc.is_disabled);
await this.removeSshConfigForSubscription(doc);
await this.setSshConfig();
return newDoc;
});
}
private async removeSshConfigForSubscription(doc: Subscription) {
if (doc.type !== 'private-repo' || doc.pull_type !== 'ssh-key') return;
// Invalid historical rows must remain deletable without deriving unsafe paths.
try {
assertSubscriptionAlias(doc.alias);
} catch {
return;
}
await this.sshKeyService.removeSSHKey(doc.alias, '');
}
public async updateDb(payload: Subscription): Promise<Subscription> {
@@ -262,6 +290,7 @@ export default class SubscriptionService {
last_running_time: number;
last_execution_time: number;
}) {
if (log_path) resolveLogPath(config.logPath, log_path);
const options: any = {
status,
pid,
@@ -279,25 +308,71 @@ export default class SubscriptionService {
}
public async remove(ids: number[], query: { force?: boolean }) {
const docs = await SubscriptionModel.findAll({ where: { id: ids } });
for (const doc of docs) {
await this.handleTask(doc.get({ plain: true }), false);
}
await SubscriptionModel.destroy({ where: { id: ids } });
await this.setSshConfig();
if (query?.force === true) {
const crons = await CrontabModel.findAll({ where: { sub_id: ids } });
if (crons?.length) {
await this.crontabService.remove(crons.map((x) => x.id!));
return withSubscriptionMutation(async () => {
const docs = await SubscriptionModel.findAll({ where: { id: ids } });
const storagePaths: string[] = [];
const retainedPaths: string[] = [];
const collectPaths = (
doc: Subscription,
targets: string[],
deleting = false,
) => {
for (const names of getSubscriptionStorageCandidates(doc)) {
// A URL accepted by one engine can produce an invalid name in another.
// Validate each candidate independently, retaining the safe candidate.
if (!Object.values(names).every(isSubscriptionStorageName)) continue;
const resolve = deleting ? resolveSubscriptionStoragePath : path.resolve;
const scriptPath = resolve(config.scriptPath, names.script);
if (scriptPath) targets.push(scriptPath);
if (names.repo) {
const repoPath = resolve(config.repoPath, names.repo);
if (repoPath) targets.push(repoPath);
}
if (names.raw) {
const rawPath = resolve(path.join(config.dataPath, 'raw'), names.raw);
if (rawPath) targets.push(rawPath);
}
}
};
// Validate all deletion targets before changing rows or removing any files.
if (query?.force === true) {
for (const doc of docs) {
assertSubscriptionAlias(doc.alias);
resolveSubscriptionStoragePath(config.scriptPath, doc.alias);
resolveSubscriptionStoragePath(config.repoPath, doc.alias);
collectPaths(doc, storagePaths, true);
}
const remaining = await SubscriptionModel.findAll({ where: {} });
for (const doc of remaining) {
if (!ids.includes(doc.id!)) collectPaths(doc, retainedPaths);
}
}
for (const doc of docs) {
const filePath = join(config.scriptPath, doc.alias);
const repoPath = join(config.repoPath, doc.alias);
await rmPath(filePath);
await rmPath(repoPath);
await this.handleTask(doc.get({ plain: true }), false);
}
}
await SubscriptionModel.destroy({ where: { id: ids } });
for (const doc of docs) {
await this.removeSshConfigForSubscription(doc);
}
await this.setSshConfig();
if (query?.force === true) {
const crons = await CrontabModel.findAll({ where: { sub_id: ids } });
if (crons?.length) {
await this.crontabService.remove(crons.map((x) => x.id!));
}
for (const storagePath of new Set(storagePaths)) {
// rm is recursive: an ancestor must also be retained if a surviving
// subscription uses a nested branch, and vice versa.
const shared = retainedPaths.some((retained) =>
retained === storagePath ||
retained.startsWith(`${storagePath}${path.sep}`) ||
storagePath.startsWith(`${retained}${path.sep}`),
);
if (!shared) await rmPath(storagePath);
}
}
});
}
public async getDb(
@@ -399,7 +474,7 @@ export default class SubscriptionService {
if (doc.log_path) {
const relativeDir = path.dirname(`${doc.log_path}`);
const dir = path.resolve(config.logPath, relativeDir);
const dir = path.dirname(resolveLogPath(config.logPath, doc.log_path));
const _exist = await fileExist(dir);
if (_exist) {
let files = await fs.readdir(dir);
+52 -22
View File
@@ -1,4 +1,7 @@
import { spawn } from 'cross-spawn';
import { execFile } from 'child_process';
import { promisify } from 'util';
import { shellQuote } from '../shared/shellQuote';
import { Response } from 'express';
import fs from 'fs';
import { Agent, request } from 'undici';
@@ -31,6 +34,8 @@ import {
SystemModelInfo,
} from '../data/system';
import taskLimit from '../shared/pLimit';
import cronClient from '../schedule/client';
import { withSchedulerMutation } from '../shared/schedulerMutationLock';
import NotificationService from './notify';
import ScheduleService, { TaskCallbacks } from './schedule';
import SockService from './sock';
@@ -119,15 +124,29 @@ export default class SystemService {
}
public async updateCronConcurrency(info: SystemModelInfo) {
const oDoc = await this.getSystemConfig();
await this.updateAuthDb({
...oDoc,
info: { ...oDoc.info, ...info },
});
if (info.cronConcurrency) {
await taskLimit.setCustomLimit(info.cronConcurrency);
const limit = info.cronConcurrency ?? 0;
if (!Number.isInteger(limit) || limit < 0 || limit > 2147483647) {
throw Object.assign(new Error(t('参数错误')), { status: 400 });
}
return { code: 200, data: info };
return withSchedulerMutation(async () => {
const oDoc = await this.getSystemConfig();
await this.updateAuthDb({
...oDoc,
info: { ...oDoc.info, ...info, cronConcurrency: limit },
});
await taskLimit.setCustomLimit(limit || Math.max(os.cpus().length, 4));
await cronClient.setConcurrency(limit);
return { code: 200, data: info };
});
}
public async restoreCronConcurrency() {
await withSchedulerMutation(async () => {
const doc = await this.getSystemConfig();
const limit = doc.info?.cronConcurrency ?? 0;
await taskLimit.setCustomLimit(limit || Math.max(os.cpus().length, 4));
await cronClient.setConcurrency(limit);
});
}
public async updateDependenceProxy(info: SystemModelInfo) {
@@ -139,7 +158,7 @@ export default class SystemService {
if (info.dependenceProxy) {
await fs.promises.writeFile(
config.dependenceProxyFile,
`export http_proxy="${info.dependenceProxy}"\nexport https_proxy="${info.dependenceProxy}"`,
`export http_proxy=${shellQuote(info.dependenceProxy)}\nexport https_proxy=${shellQuote(info.dependenceProxy)}`,
);
} else {
await fs.promises.rm(config.dependenceProxyFile);
@@ -155,7 +174,7 @@ export default class SystemService {
});
let cmd = 'pnpm config delete registry';
if (info.nodeMirror) {
cmd = `pnpm config set registry ${info.nodeMirror}`;
cmd = `pnpm config set registry ${shellQuote(info.nodeMirror)}`;
}
let command = `cd && ${cmd}`;
const docs = await DependenceModel.findAll({
@@ -204,7 +223,7 @@ export default class SystemService {
});
let cmd = 'pip config unset global.index-url';
if (info.pythonMirror) {
cmd = `pip3 config set global.index-url ${info.pythonMirror}`;
cmd = `pip3 config set global.index-url ${shellQuote(info.pythonMirror)}`;
}
await promiseExec(cmd);
return { code: 200, data: info };
@@ -220,7 +239,6 @@ export default class SystemService {
return;
}
const command = await updateLinuxMirrorFile(info.linuxMirror || '');
let hasError = false;
this.scheduleService.runTask(
command,
{
@@ -228,14 +246,15 @@ export default class SystemService {
res?.setHeader('QL-Task-Pid', `${cp.pid}`);
res?.end();
},
onEnd: async () => {
onEnd: async (cp) => {
this.sockService.sendMessage({
type: 'updateLinuxMirror',
message: 'update linux mirror end',
status: 'completed',
});
onEnd?.();
if (!hasError) {
// Tools such as sudo and apt may write warnings to stderr on success.
if (cp?.exitCode === 0 && cp.signalCode === null) {
await this.updateAuthDb({
...oDoc,
info: { ...oDoc.info, ...info },
@@ -243,7 +262,6 @@ export default class SystemService {
}
},
onError: async (message: string) => {
hasError = true;
this.sockService.sendMessage({ type: 'updateLinuxMirror', message });
},
onLog: async (message: string) => {
@@ -357,7 +375,10 @@ export default class SystemService {
}
public async reloadSystem(target?: 'system' | 'data') {
const cmd = `real_time=true ql reload ${target || ''}`;
if (target && !['system', 'data'].includes(target)) {
return { code: 400, message: t('参数错误') };
}
const cmd = `real_time=true ql reload ${shellQuote(target || '')}`;
const cp = spawn(cmd, {
shell: '/bin/bash',
detached: true,
@@ -438,11 +459,20 @@ export default class SystemService {
if (type && type.length) {
dataDirs = dataDirs.concat(type.filter((x) => x !== 'base'));
}
const dataPaths = dataDirs.map((dir) => `data/${dir}`);
await promiseExec(
`cd ${config.dataPath} && cd ../ && tar -zcvf ${config.dataTgzFile
} ${dataPaths.join(' ')}`,
);
const allowed = new Set(['db', 'upload', 'config', 'scripts', 'log', 'deps',
'syslog', 'dep_cache', 'raw', 'repo', 'ssh.d', 'script-history']);
if (dataDirs.some((dir) => !allowed.has(dir))) {
return res.status(400).send({ code: 400, message: t('参数错误') });
}
if (dataDirs.includes('scripts')) dataDirs.push('script-history');
// History is created lazily; old/fresh installations must still export.
if (!fs.existsSync(path.join(config.dataPath, 'script-history'))) {
dataDirs = dataDirs.filter((dir) => dir !== 'script-history');
}
const dataPaths = [...new Set(dataDirs)].map((dir) => `data/${dir}`);
await promisify(execFile)('tar', ['-zcf', config.dataTgzFile, '--', ...dataPaths], {
cwd: path.dirname(config.dataPath),
});
res.download(config.dataTgzFile);
} catch (error: any) {
return res.send({ code: 400, message: error.message });
@@ -565,7 +595,7 @@ export default class SystemService {
try {
await fs.promises.writeFile(
config.langEnvFile,
`export QL_LANG='${lang}'\n`,
`export QL_LANG=${shellQuote(lang)}\n`,
);
} catch (error) {
this.logger.error(`Failed to write lang_env.sh: ${error}`);
+15 -12
View File
@@ -33,9 +33,11 @@ import {
verifyPassword,
} from '../shared/password';
import { serializeAuthMutation } from '../shared/authMutation';
import { LoginThrottle } from '../shared/loginThrottle';
@Service()
export default class UserService {
private loginThrottle = new LoginThrottle();
@Inject((type) => NotificationService)
private notificationService!: NotificationService;
@@ -106,11 +108,8 @@ export default class UserService {
return { code: 403, message: t('该 IP 已被列入黑名单') };
}
const retriesTime = Math.pow(3, retries) * 1000;
if (retries > 2 && timestamp - lastlogon < retriesTime) {
const waitTime = Math.ceil(
(retriesTime - (timestamp - lastlogon)) / 1000,
);
const waitTime = this.loginThrottle.retryAfter(ip, timestamp);
if (waitTime > 0) {
return {
code: 410,
message: tf('失败次数过多,请%s秒后重试', waitTime),
@@ -133,6 +132,7 @@ export default class UserService {
}
if (passwordMatches) {
this.loginThrottle.reset(ip);
const data = createRandomString(50, 100);
const expiration = twoFactorActivated ? '60d' : '20d';
let token = jwt.sign({ data }, config.jwt.secret, {
@@ -206,8 +206,9 @@ export default class UserService {
},
};
} else {
const failures = this.loginThrottle.fail(ip, timestamp);
await this.updateAuthInfo(content, {
retries: retries + 1,
retries: failures,
lastlogon: timestamp,
lastip: ip,
lastaddr: address,
@@ -239,8 +240,8 @@ export default class UserService {
},
});
this.getLoginLog();
if (retries > 2) {
const waitTime = Math.round(Math.pow(3, retries + 1));
if (failures > 3) {
const waitTime = this.loginThrottle.retryAfter(ip, timestamp);
return {
code: 410,
message: tf('失败次数过多,请%s秒后重试', waitTime),
@@ -439,8 +440,8 @@ export default class UserService {
const authInfo = await this.getAuthInfo();
const { isTwoFactorChecking, twoFactorSecret } = authInfo;
const now = Date.now();
const retries = authInfo.retries || 0;
if (retries > 2 && now - authInfo.lastlogon < Math.pow(3, retries) * 1000) {
const source = getClientIp(req);
if (this.loginThrottle.retryAfter(source, now) > 0) {
return { code: 410, message: t('失败次数过多,请稍后重试') };
}
if (
@@ -472,9 +473,8 @@ export default class UserService {
.join(' ');
}
await this.updateAuthInfo(authInfo, {
retries: retries + 1,
retries: this.loginThrottle.fail(source, now),
lastlogon: now,
isTwoFactorChecking: retries + 1 < 5,
lastip: ip,
lastaddr: address,
platform: req.platform,
@@ -690,5 +690,8 @@ export default class UserService {
});
}
await this.updateAuthInfo(authInfo, payload);
if (retries === 0 || password !== undefined || username !== undefined) {
this.loginThrottle.clear();
}
}
}
+11
View File
@@ -0,0 +1,11 @@
// Match command text literally: looking up a task must never execute its input.
export function findCronId(crontab: string, command: string): string {
if (!command) return '';
for (const line of crontab.split('\n')) {
const id = /\bID=(\d+)\s/.exec(line);
if (id && line.includes(command, id.index + id[0].length)) {
return id[1];
}
}
return '';
}
+90 -7
View File
@@ -1,12 +1,93 @@
import cronParser from 'cron-parser-v4';
import CronExpressionParser from 'cron-parser';
import { validate } from 'node-cron';
import { ScheduleType } from '../interface/schedule';
// Keep validation aligned with node-schedule's locked cron-parser version.
// cron-parser 5 accepts syntax (e.g. H) that the scheduler cannot execute.
const aliases: Record<string, string> = {
'@yearly': '0 0 0 1 1 *',
'@annually': '0 0 0 1 1 *',
'@monthly': '0 0 0 1 * *',
'@weekly': '0 0 0 * * 0',
'@daily': '0 0 0 * * *',
'@midnight': '0 0 0 * * *',
'@hourly': '0 0 * * * *',
'@minutely': '0 * * * * *',
};
export interface CronSchedule {
patterns: string[];
dayCount: number;
nthDay: number;
}
// Validation and registration repeatedly see identical rules in large snapshots.
// Cache only immutable syntax, never dates or running tasks, and bound user input.
const scheduleCache = new Map<string, CronSchedule>();
// Canonicalize legacy shorthand, aliases, names and numeric-start steps before
// node-cron sees them. Extend legacy aliases explicitly; reject H and bare /N.
export function parseCronSchedule(schedule: unknown): CronSchedule {
if (typeof schedule !== 'string' || !schedule.trim()) {
throw new Error('Invalid cron schedule');
}
let source = schedule.trim();
const cacheKey = source;
const cached = scheduleCache.get(cacheKey);
if (cached) return cached;
if (source.startsWith('@')) {
if (!Object.hasOwn(aliases, source)) throw new Error('Invalid cron alias');
source = aliases[source];
}
let parts = source.split(/\s+/);
if (
parts.length > 6 ||
parts.some((part) => /(^|,)\//.test(part) || /H(?:\(|\/|$)/i.test(part))
) {
throw new Error('Unsupported cron syntax');
}
parts = [
...['0', '*', '*', '*', '*', '*'].slice(0, 6 - parts.length),
...parts,
];
if (
parts.some(
(part, index) => index !== 3 && index !== 5 && part.includes('?'),
)
) {
throw new Error('Question mark is only valid in day fields');
}
const expression = CronExpressionParser.parse(parts.join(' '));
if (!expression.hasNext())
throw new Error('Cron schedule has no next execution');
const normalized = expression.stringify(true).replace(/\?/g, '*').split(' ');
const dayCount = expression.fields.dayOfMonth.values.length;
const weekCount = expression.fields.dayOfWeek.values.length;
const patterns =
dayCount < 31 && weekCount < 8
? [
[...normalized.slice(0, 5), '*'].join(' '),
[...normalized.slice(0, 3), '*', ...normalized.slice(4)].join(' '),
]
: [normalized.join(' ')];
if (!patterns.every((pattern) => validate(pattern))) {
throw new Error('Unsupported cron schedule');
}
const parsed = {
patterns,
dayCount,
nthDay: expression.fields.dayOfWeek.nthDay,
};
Object.freeze(patterns);
Object.freeze(parsed);
if (scheduleCache.size >= 512)
scheduleCache.delete(scheduleCache.keys().next().value);
scheduleCache.set(cacheKey, parsed);
return parsed;
}
export function isValidCronSchedule(schedule: unknown): schedule is string {
if (typeof schedule !== 'string' || !schedule.trim()) return false;
try {
return cronParser.parseExpression(schedule).hasNext();
parseCronSchedule(schedule);
return true;
} catch {
return false;
}
@@ -22,6 +103,8 @@ export function getInvalidCronSchedules(cron: {
) {
return [];
}
return [cron.schedule, ...(cron.extra_schedules || []).map((x) => x.schedule)]
.filter((schedule) => !isValidCronSchedule(schedule));
return [
cron.schedule,
...(cron.extra_schedules || []).map((x) => x.schedule),
].filter((schedule) => !isValidCronSchedule(schedule));
}
+89
View File
@@ -0,0 +1,89 @@
import { createTask, ScheduledTask } from 'node-cron';
import { parseCronSchedule } from './cronSchedule';
interface SchedulerLogger {
warn(message: string, ...args: unknown[]): unknown;
error(message: string, ...args: unknown[]): unknown;
}
export interface CronJob {
start(): void;
cancel(): void;
}
// node-cron owns timers and calendar calculation; Qinglong owns execution
// concurrency and process lifetime. Missed in-process slots join that same queue.
export function createCronJob(
schedule: string,
callback: (date: Date) => unknown | Promise<unknown>,
options: { name: string; logger: SchedulerLogger; start?: boolean },
): CronJob {
const parsed = parseCronSchedule(schedule);
const tasks: ScheduledTask[] = [];
let cancelled = false;
let started = false;
const accepts = (index: number, date: Date) => {
if (tasks.length === 1) return true;
// Preserve cron-parser 4's day/weekday OR semantics, including its
// month-dependent full-day range and the global nth-week constraint.
if (parsed.nthDay && Math.ceil(date.getDate() / 7) !== parsed.nthDay)
return false;
if (index === 1) return !tasks[0].match(date);
const monthDays = [31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31];
return parsed.dayCount < monthDays[date.getMonth()] || tasks[1].match(date);
};
const dispatch = async (index: number, date: Date, missed: boolean) => {
if (cancelled || !started || !accepts(index, date)) return;
if (missed) {
options.logger.warn(
'[schedule][补执行迟到任务] 任务: %s, 计划时间: %s',
options.name,
date.toISOString(),
);
}
try {
await callback(date);
} catch (error) {
options.logger.error(
'[schedule][定时回调失败] 任务: %s, 错误: %s',
options.name,
error instanceof Error ? error.message : String(error),
);
}
};
try {
for (const [index, pattern] of parsed.patterns.entries()) {
const task = createTask(
pattern,
(context) => dispatch(index, context.date, false),
{
noOverlap: false,
missedExecutionTolerance: 5000,
},
);
tasks.push(task);
task.on('execution:missed', (context) =>
dispatch(index, context.date, true),
);
// Fail before an existing schedule is cancelled, including impossible dates.
task.getNextRuns(1);
}
} catch (error) {
tasks.forEach((task) => task.destroy());
throw error;
}
const job: CronJob = {
start() {
if (started || cancelled) return;
started = true;
tasks.forEach((task) => task.start());
},
cancel() {
if (cancelled) return;
cancelled = true;
tasks.forEach((task) => task.destroy());
},
};
if (options.start !== false) job.start();
return job;
}
+1
View File
@@ -3,6 +3,7 @@ import { shareStore } from './store';
const messages: Record<string, Record<string, string>> = {
zh: {},
en: {
'资源已存在,请检查重复的名称或值': 'Resource already exists; check for duplicate names or values',
'暂无权限': 'Access denied',
'参数错误': 'Invalid parameter',
'参数不正确': 'Invalid parameter',
+1
View File
@@ -18,6 +18,7 @@ export interface IDependencyFn<T> {
export interface ICronFn<T> {
(): Promise<T>;
cron?: TCron;
isCurrent?: () => boolean;
}
export interface ISchedule {
+121
View File
@@ -0,0 +1,121 @@
import { shellQuote } from './shellQuote';
const repositoryNames =
/^(debian(?:-security)?|ubuntu(?:-ports)?|alpine|armbian)$/;
export function normalizeMirror(mirror: string): URL {
if (/\s/.test(mirror)) throw new Error('Invalid Linux mirror URL');
const url = new URL(mirror);
if (
!['http:', 'https:'].includes(url.protocol) ||
url.username ||
url.password ||
url.search ||
url.hash
) {
throw new Error(
'Linux mirror must be an HTTP(S) URL without credentials, query or fragment',
);
}
return url;
}
export function replaceMirrorUrl(current: string, mirror: string): string {
const target = normalizeMirror(mirror);
const previous = new URL(current);
const oldSegments = previous.pathname.split('/').filter(Boolean);
const newSegments = target.pathname.split('/').filter(Boolean);
const repositoryIndex = oldSegments.findIndex((part) =>
repositoryNames.test(part),
);
const targetIndex = newSegments.findIndex((part) =>
repositoryNames.test(part),
);
const prefix =
targetIndex >= 0 ? newSegments.slice(0, targetIndex) : newSegments;
if (repositoryIndex >= 0) {
const repository = oldSegments[repositoryIndex];
let suffix = oldSegments.slice(repositoryIndex + 1);
// Repair paths already corrupted by repeated authority-only replacement.
while (suffix[0] === repository) suffix.shift();
return `${target.origin}/${[...prefix, repository, ...suffix].join('/')}`;
}
// Unknown repository paths: a full URL replaces the old root, a hostname
// preserves it. Never append the newly entered root to the existing root.
return `${target.origin}/${(newSegments.length
? newSegments
: oldSegments
).join('/')}`;
}
export function rewriteMirrorSources(content: string, mirror: string): string {
normalizeMirror(mirror);
return content
.split(/(\r?\n[ \t]*\r?\n)/)
.map((stanza) => {
const signedBy = stanza.match(/^Signed-By:\s*([^\r\n]*)/im)?.[1];
return stanza
.split('\n')
.map((line) => {
if (!/^\s*(?:URIs:|deb(?:-src)?\s|https?:\/\/)/.test(line))
return line;
const legacy = /^\s*deb(?:-src)?\s/.test(line);
const key = legacy
? line.match(/\bsigned-by=([^\]\s]+)/i)?.[1]
: signedBy;
// In .list files a # starts an inline comment; its URLs are metadata.
const comment = legacy ? line.indexOf('#') : -1;
const source = comment < 0 ? line : line.slice(0, comment);
const rewritten = source.replace(/https?:\/\/[^\s#]+/g, (url) =>
isSystemRepository(url, key) ? replaceMirrorUrl(url, mirror) : url,
);
return rewritten + (comment < 0 ? '' : line.slice(comment));
})
.join('\n');
})
.join('');
}
function isSystemRepository(url: string, signedBy?: string): boolean {
const keys =
signedBy
?.trim()
.split(/[\s,]+/)
.filter(Boolean) || [];
const archiveKeyring =
keys.length > 0 &&
keys.every((key) =>
/\/(?:debian|ubuntu|armbian)-archive-keyring\.(?:gpg|asc)$/.test(key),
);
// A separate signing key identifies a third-party archive, even if its
// repository happens to be called /debian or /ubuntu.
if (keys.length > 0 && !archiveKeyring) return false;
const segments = new URL(url).pathname.split('/').filter(Boolean);
const repositoryIndex = segments.findIndex((part) =>
repositoryNames.test(part),
);
if (repositoryIndex < 0) return false;
// Without archive metadata, accept only conventional archive roots. A
// vendor path such as Docker's /linux/debian is not the Debian archive.
return (
archiveKeyring ||
repositoryIndex === 0 ||
segments
.slice(0, repositoryIndex)
.every((part) => /^mirrors?$/.test(part)) ||
segments[repositoryIndex] === 'alpine'
);
}
export function mirrorWriteCommand(
file: string,
content: string,
prefix: string,
update: string,
): string {
return `${prefix}mkdir -p ${shellQuote(
file.slice(0, file.lastIndexOf('/')),
)} && printf '%s' ${shellQuote(content)} | ${prefix}tee ${shellQuote(
file,
)} >/dev/null && ${update}`;
}
+11
View File
@@ -0,0 +1,11 @@
import { resolveFileAccess } from './fileAccess';
/** Check stored paths too: old database rows are not trusted input. */
export function resolveLogPath(root: string, value: string): string {
const target = resolveFileAccess(root, [value]);
if (!target)
throw Object.assign(new Error('Log path is outside the log directory'), {
status: 400,
});
return target;
}
+47
View File
@@ -0,0 +1,47 @@
const RETRY_TTL = 15 * 60 * 1000;
const MAX_SOURCES = 10000;
const MAX_RETRY_SECONDS = 5 * 60;
/** Source-specific, bounded backoff; persisted account history is not a lock. */
export class LoginThrottle {
private sources = new Map<
string,
{ failures: number; lastFailure: number }
>();
private get(source: string, now: number) {
const state = this.sources.get(source);
if (state && now - state.lastFailure >= RETRY_TTL) {
this.sources.delete(source);
return undefined;
}
return state;
}
retryAfter(source: string, now: number): number {
const state = this.get(source, now);
if (!state || state.failures < 3) return 0;
const seconds = Math.min(3 ** state.failures, MAX_RETRY_SECONDS);
return Math.max(
0,
Math.ceil((seconds * 1000 - (now - state.lastFailure)) / 1000),
);
}
fail(source: string, now: number): number {
const failures = Math.min((this.get(source, now)?.failures || 0) + 1, 6);
this.sources.delete(source);
if (this.sources.size >= MAX_SOURCES) {
this.sources.delete(this.sources.keys().next().value!);
}
this.sources.set(source, { failures, lastFailure: now });
return failures;
}
reset(source: string) {
this.sources.delete(source);
}
clear() {
this.sources.clear();
}
}
+16 -9
View File
@@ -50,11 +50,9 @@ class TaskLimit {
Buffer.from(tlsConfig.clientCert),
)
: credentials.createInsecure();
this._client = new ApiClient(
`localhost:${config.grpcPort}`,
creds,
{ 'grpc.enable_http_proxy': 0 },
);
this._client = new ApiClient(`localhost:${config.grpcPort}`, creds, {
'grpc.enable_http_proxy': 0,
});
}
return this._client;
}
@@ -113,12 +111,15 @@ class TaskLimit {
}
}
public removeQueuedCron(id: string) {
public removeQueuedCron(id: string, completed?: ICronFn<any>) {
if (this.queuedCrons.has(id)) {
const runs = this.queuedCrons.get(id);
if (runs && runs.length > 0) {
runs.pop();
this.queuedCrons.set(id, runs);
const remaining = completed
? runs.filter((run) => run !== completed)
: runs.slice(0, -1);
if (remaining.length) this.queuedCrons.set(id, remaining);
else this.queuedCrons.delete(id);
}
}
}
@@ -143,9 +144,15 @@ class TaskLimit {
cron: TCron,
fn: ICronFn<T>,
options?: Partial<QueueAddOptions>,
isCurrent: () => boolean = () => true,
): Promise<T | void> {
if (!isCurrent()) return;
fn.cron = cron;
let runs = this.queuedCrons.get(cron.id);
fn.isCurrent = isCurrent;
// Invalidated snapshots must not consume the new revision's repeat budget.
const runs = this.queuedCrons
.get(cron.id)
?.filter((run) => run.isCurrent?.() !== false);
const result = runs?.length ? [...runs, fn] : [fn];
const repeatTimes = this.repeatCronNotifyMap.get(cron.id) || 0;
if (result?.length > 5) {
+15 -4
View File
@@ -8,15 +8,23 @@ import { RunningInstanceModel, InstanceStatus } from '../data/runningInstance';
import dayjs from 'dayjs';
import { observeChildProcess, asError } from './childProcess';
export function runCron(cmd: string, cron: ICron): Promise<number | void> {
return taskLimit.runWithCronLimit(cron, async () => {
export function runCron(
cmd: string,
cron: ICron,
isCurrent: () => boolean = () => true,
): Promise<number | void> {
const execute = async () => {
try {
if (!isCurrent()) return;
// Check if the cron is already running and stop it (only if multiple instances are not allowed)
try {
const existingCron = await CrontabModel.findOne({
where: { id: Number(cron.id) },
});
// A mutation may commit while this queued run is reading the database.
if (!isCurrent() || !existingCron || existingCron.isDisabled) return;
// Default to single instance mode (0) for backward compatibility
const allowSingleInstances =
existingCron?.allow_multiple_instances === 0;
@@ -56,6 +64,8 @@ export function runCron(cmd: string, cron: ICron): Promise<number | void> {
throw error;
}
// In particular, invalidation can happen while stopping a previous run.
if (!isCurrent()) return;
Logger.info(
`[schedule][开始执行任务] 参数 ${JSON.stringify({
...cron,
@@ -94,7 +104,8 @@ export function runCron(cmd: string, cron: ICron): Promise<number | void> {
asError(error).message,
);
} finally {
taskLimit.removeQueuedCron(cron.id);
taskLimit.removeQueuedCron(cron.id, execute);
}
});
};
return taskLimit.runWithCronLimit(cron, execute, undefined, isCurrent);
}
+467
View File
@@ -0,0 +1,467 @@
import * as fs from 'fs/promises';
import { createReadStream } from 'fs';
import path from 'path';
import { createHash, randomUUID } from 'crypto';
import { promisify } from 'util';
import { brotliCompress, brotliDecompress, constants } from 'zlib';
import { lock } from 'proper-lockfile';
import { resolveFileAccess } from './fileAccess';
import { getUniqueLockPath } from './utils';
const compress = promisify(brotliCompress);
const decompress = promisify(brotliDecompress);
const hash = (content: string) =>
createHash('sha256').update(content).digest('hex');
export const SCRIPT_HISTORY_LIMITS = {
versions: 20,
fileBytes: 1024 * 1024,
unpackedBytes: 8 * 1024 * 1024,
archiveBytes: 2 * 1024 * 1024,
totalBytes: 32 * 1024 * 1024,
files: 1000,
};
type Source = 'baseline' | 'save' | 'restore' | 'external';
interface Version {
id: string;
createdAt: string;
source: Source;
hash: string;
content: string;
}
interface Archive {
format: 1;
path: string;
versions: Version[];
head?: { hash: string; createdAt: string; source: Source };
}
export class ScriptHistoryError extends Error {
constructor(public status: number, message: string) {
super(message);
}
}
export class HistoryUnavailableError extends ScriptHistoryError {
constructor(public currentHash: string) {
super(413, '历史版本仅支持不超过 1 MiB 的 UTF-8 文本脚本');
}
}
interface SaveOptions {
skipHistory?: boolean;
expectedHash?: string;
}
/** Lazy, bounded storage: no watcher, database, daemon, or in-memory cache.
* Full snapshots share a Brotli window, so similar revisions compress together
* without a patch chain. All mutations use the existing script lock and a
* cross-process archive lock. History is durable before overwriting the script. */
export class ScriptHistory {
constructor(
private scriptRoot: string,
private historyRoot: string,
private blacklist: string[] = [],
private limits = SCRIPT_HISTORY_LIMITS,
) {}
private async target(directory: string, filename: string) {
const candidate = resolveFileAccess(
this.scriptRoot,
[directory, filename],
this.blacklist,
);
if (!candidate) throw new ScriptHistoryError(403, '暂无权限');
try {
const file = await fs.realpath(candidate);
const root = await fs.realpath(this.scriptRoot);
const relative = path.relative(root, file);
const key = hash(relative);
return {
file,
relative,
archive: path.join(this.historyRoot, key + '.br'),
};
} catch (error: any) {
if (error.code === 'ENOENT')
throw new ScriptHistoryError(404, '脚本不存在');
throw error;
}
}
private async readText(file: string) {
const handle = await fs.open(file, 'r');
try {
const stat = await handle.stat();
if (!stat.isFile()) throw new ScriptHistoryError(400, '请选择脚本文件');
if (stat.size > this.limits.fileBytes)
throw new ScriptHistoryError(
413,
'历史版本仅支持不超过 1 MiB 的 UTF-8 文本脚本',
);
// Bounded read even if an external process grows the file after stat.
const bytes = Buffer.alloc(this.limits.fileBytes + 1);
let size = 0;
while (size < bytes.length) {
const { bytesRead } = await handle.read(
bytes,
size,
bytes.length - size,
null,
);
if (!bytesRead) break;
size += bytesRead;
}
const buffer = bytes.subarray(0, size);
const content = buffer.toString('utf8');
if (
size > this.limits.fileBytes ||
content.includes('\0') ||
!Buffer.from(content).equals(buffer)
) {
throw new ScriptHistoryError(
413,
'历史版本仅支持不超过 1 MiB 的 UTF-8 文本脚本',
);
}
return content;
} finally {
await handle.close();
}
}
private async readArchive(file: string, relative: string): Promise<Archive> {
try {
const stat = await fs.lstat(file);
if (!stat.isFile() || stat.size > this.limits.archiveBytes)
throw new Error('Invalid archive');
const bytes = await decompress(await fs.readFile(file), {
maxOutputLength: this.limits.unpackedBytes,
});
const archive: Archive = JSON.parse(bytes.toString('utf8'));
if (
archive.format !== 1 ||
archive.path !== relative ||
!Array.isArray(archive.versions) ||
archive.versions.length > this.limits.versions ||
archive.versions.some(
(v) =>
typeof v.content !== 'string' ||
typeof v.id !== 'string' ||
typeof v.createdAt !== 'string' ||
!['baseline', 'save', 'restore', 'external'].includes(v.source) ||
Buffer.byteLength(v.content) > this.limits.fileBytes ||
hash(v.content) !== v.hash,
)
) {
throw new Error('Invalid archive');
}
return archive;
} catch (error: any) {
if (error.code === 'ENOENT')
return { format: 1, path: relative, versions: [] };
throw new ScriptHistoryError(500, '历史版本读取失败,请检查历史文件');
}
}
async list(directory: string, filename: string) {
const target = await this.target(directory, filename);
const [current, archive] = await Promise.all([
this.readText(target.file),
this.readArchive(target.archive, target.relative),
]);
const currentHash = hash(current);
const versions = archive.versions
.slice()
.reverse()
.map(({ content, ...v }) => ({
...v,
size: Buffer.byteLength(content),
identical: v.hash === currentHash,
}));
return { versions, currentHash, limit: this.limits.versions };
}
async detail(directory: string, filename: string, id: string) {
const target = await this.target(directory, filename);
const [current, archive] = await Promise.all([
this.readText(target.file),
this.readArchive(target.archive, target.relative),
]);
const version = archive.versions.find((v) => v.id === id);
if (!version) throw new ScriptHistoryError(404, '历史版本不存在或已清理');
return { version, current, currentHash: hash(current) };
}
async save(
directory: string,
filename: string,
content: string,
options: SaveOptions = {},
) {
return this.mutate(directory, filename, { content, ...options });
}
async restore(
directory: string,
filename: string,
id: string,
expectedHash: string,
) {
return this.mutate(directory, filename, { id, expectedHash });
}
private async mutate(
directory: string,
filename: string,
change:
| ({ content: string } & SaveOptions)
| { id: string; expectedHash: string },
) {
const target = await this.target(directory, filename);
const releaseFile = await lock(target.file, {
lockfilePath: getUniqueLockPath(target.file),
retries: { retries: 20, minTimeout: 50, maxTimeout: 250 },
});
try {
let current: string;
try {
current = await this.readText(target.file);
if (
'content' in change &&
(Buffer.byteLength(change.content) > this.limits.fileBytes ||
change.content.includes('\0'))
) {
throw new ScriptHistoryError(
413,
'历史版本仅支持不超过 1 MiB 的 UTF-8 文本脚本',
);
}
} catch (error) {
if (
!('content' in change) ||
!(error instanceof ScriptHistoryError) ||
error.status !== 413
)
throw error;
// Hash oversized or non-UTF-8 files without reading them all into memory.
const digest = createHash('sha256');
for await (const chunk of createReadStream(target.file))
digest.update(chunk);
const currentHash = digest.digest('hex');
if (!change.skipHistory) throw new HistoryUnavailableError(currentHash);
if (change.expectedHash !== currentHash)
throw new ScriptHistoryError(409, '文件已变化,请刷新历史版本后重试');
await this.preparedWrite(target.file, change.content, (commit) =>
commit(),
);
return {
content: change.content,
changed: true,
historyRecorded: false,
};
}
if (
('id' in change || change.skipHistory) &&
hash(current) !== change.expectedHash
) {
throw new ScriptHistoryError(409, '文件已变化,请刷新历史版本后重试');
}
if ('content' in change && change.content === current)
return { content: current, changed: false };
await fs.mkdir(this.historyRoot, { recursive: true, mode: 0o700 });
const releaseHistory = await lock(this.historyRoot, {
retries: { retries: 20, minTimeout: 50, maxTimeout: 250 },
});
try {
const archive = await this.readArchive(target.archive, target.relative);
const restored =
'id' in change
? archive.versions.find((v) => v.id === change.id)
: undefined;
if ('id' in change && !restored)
throw new ScriptHistoryError(404, '历史版本不存在或已清理');
const content =
'content' in change ? change.content : restored!.content;
if (content === current) return { content, changed: false };
if (
Buffer.byteLength(content) > this.limits.fileBytes ||
content.includes('\0')
) {
throw new ScriptHistoryError(
413,
'历史版本仅支持不超过 1 MiB 的 UTF-8 文本脚本',
);
}
const currentHash = hash(current);
if (
archive.versions[archive.versions.length - 1]?.hash !== currentHash
) {
const head =
archive.head?.hash === currentHash ? archive.head : undefined;
archive.versions.push({
id: randomUUID(),
createdAt: head?.createdAt || new Date().toISOString(),
source:
head?.source ||
(archive.versions.length ? 'external' : 'baseline'),
hash: currentHash,
content: current,
});
}
// Only prior contents are stored. Head metadata labels the next snapshot,
// but is used only if its hash matches the file. A failed file write can
// never introduce a phantom content version or lose the old snapshot.
archive.head = {
hash: hash(content),
createdAt: new Date().toISOString(),
source: 'id' in change ? 'restore' : 'save',
};
const packed = await this.pack(archive);
const cleanupPending = await this.preparedWrite(
target.file,
content,
(commit) => this.store(target.archive, packed, commit),
);
return {
content,
changed: true,
historyRecorded: true,
cleanupPending,
};
} finally {
await releaseHistory();
}
} finally {
await releaseFile();
}
}
private async pack(archive: Archive) {
archive.versions = archive.versions.slice(-this.limits.versions);
while (true) {
const serialized = Buffer.from(JSON.stringify(archive));
if (serialized.length <= this.limits.unpackedBytes) {
const compressed = await compress(serialized, {
params: {
[constants.BROTLI_PARAM_QUALITY]: 4,
[constants.BROTLI_PARAM_LGWIN]: 22,
},
});
if (compressed.length <= this.limits.archiveBytes) return compressed;
}
if (archive.versions.length <= 1)
throw new ScriptHistoryError(413, '历史版本容量不足,无法安全保存');
archive.versions.shift();
}
}
/** Stage the script before publishing history. A failed write/fsync never
* truncates the original. Preserve the existing file's ownership and mode. */
private async preparedWrite<T>(
file: string,
content: string,
run: (commit: () => Promise<void>) => Promise<T>,
): Promise<T> {
const stat = await fs.stat(file);
const temporary = path.join(
path.dirname(file),
`.ql-save-${randomUUID()}.tmp`,
);
try {
await fs.writeFile(temporary, content, {
encoding: 'utf8',
mode: 0o600,
flag: 'wx',
});
if (
typeof process.getuid === 'function' &&
typeof process.getgid === 'function' &&
(stat.uid !== process.getuid() || stat.gid !== process.getgid())
) {
await fs.chown(temporary, stat.uid, stat.gid);
}
await fs.chmod(temporary, stat.mode & 0o7777);
const handle = await fs.open(temporary, 'r');
try {
await handle.sync();
} finally {
await handle.close();
}
return await run(() => fs.rename(temporary, file));
} finally {
await fs.unlink(temporary).catch(() => undefined);
}
}
private async store(
file: string,
content: Buffer,
commit: () => Promise<void>,
) {
const entries = [];
for (const name of await fs.readdir(this.historyRoot)) {
if (!/^[a-f0-9]{64}\.br$/.test(name)) continue;
const fullPath = path.join(this.historyRoot, name);
if (fullPath === file) continue;
const stat = await fs.lstat(fullPath);
entries.push({ file: fullPath, size: stat.size, time: stat.mtimeMs });
}
entries.sort((a, b) => a.time - b.time);
let total = entries.reduce((sum, v) => sum + v.size, content.length);
const victims: string[] = [];
while (
entries.length &&
(total > this.limits.totalBytes || entries.length >= this.limits.files)
) {
const oldest = entries.shift()!;
victims.push(oldest.file);
total -= oldest.size;
}
if (total > this.limits.totalBytes)
throw new ScriptHistoryError(413, '历史版本容量不足,无法安全保存');
const previous = await fs.readFile(file).catch((error) => {
if (error.code === 'ENOENT') return undefined;
throw error;
});
const temporary = path.join(this.historyRoot, '.pending.tmp');
const publish = async (bytes: Buffer) => {
await fs.unlink(temporary).catch((error) => {
if (error.code !== 'ENOENT') throw error;
});
await fs.writeFile(temporary, bytes, { mode: 0o600, flag: 'wx' });
const handle = await fs.open(temporary, 'r');
try {
await handle.sync();
} finally {
await handle.close();
}
await fs.rename(temporary, file);
};
try {
await publish(content);
try {
await commit();
} catch (error) {
// The original script is still intact. Restore its exact history before
// returning an error; no unrelated archive has been removed yet.
if (previous) await publish(previous);
else await fs.unlink(file);
throw error;
}
// Quota eviction is post-commit maintenance. Report cleanup failures as a
// warning, not a failed save (which would encourage a misleading retry).
let cleanupPending = false;
for (const victim of victims) {
try {
await fs.unlink(victim);
} catch (error: any) {
if (error.code !== 'ENOENT') cleanupPending = true;
}
}
return cleanupPending;
} finally {
await fs.unlink(temporary).catch(() => undefined);
}
}
}
+5
View File
@@ -0,0 +1,5 @@
/** Quote one literal POSIX shell argument, including quotes and substitutions. */
export function shellQuote(value: string): string {
if (value.includes('\0')) throw new Error('Invalid shell argument');
return "'" + value.replace(/'/g, "'\\''") + "'";
}
+30
View File
@@ -0,0 +1,30 @@
import lockfile from 'proper-lockfile';
import path from 'path';
import config from '../config';
// Protect subscription storage ownership from the first database read through
// cleanup. Creation and URL/branch updates must use the same cross-process lock.
export async function withSubscriptionMutation<T>(
operation: () => Promise<T>,
): Promise<T> {
const target = `${path.resolve(config.scriptPath)}.subscriptions`;
let release: () => Promise<void>;
try {
release = await lockfile.lock(target, {
realpath: false,
stale: 30000,
update: 10000,
retries: { retries: 50, factor: 1, minTimeout: 100, maxTimeout: 100 },
});
} catch (cause) {
throw Object.assign(
new Error('Subscription configuration is busy', { cause }),
{ status: 503 },
);
}
try {
return await operation();
} finally {
await release();
}
}
+45
View File
@@ -0,0 +1,45 @@
import { resolveFileAccess } from './fileAccess';
import { createHash } from 'crypto';
export function assertSubscriptionAlias(alias: string): void {
if (
typeof alias !== 'string' ||
alias.length > 200 ||
!/^[^\s\\\p{C}]+$/u.test(alias) ||
alias.startsWith('~') ||
alias.split('/').some((part) => !part || part === '.' || part === '..')
) {
throw Object.assign(new Error('Invalid subscription alias'), {
status: 400,
});
}
}
/** Keep ordinary SSH names and use a reserved namespace for path-like aliases.
* The database alias and its script/log directories remain unchanged. */
export function getSubscriptionSshAlias(alias: string): string {
assertSubscriptionAlias(alias);
if (/^[\p{L}\p{N}_.+-]+$/u.test(alias) && !alias.endsWith('.config')) {
return alias;
}
return `~sub_${createHash('sha256').update(alias).digest('hex')}`;
}
export function resolveSubscriptionPath(
root: string,
alias: string,
suffix = '',
): string {
// Internal SSH filenames cannot collide with user-supplied aliases.
if (!/^~sub_[a-f0-9]{64}$/.test(alias)) {
assertSubscriptionAlias(
alias.startsWith('~global_') ? alias.slice(8) : alias,
);
}
const target = resolveFileAccess(root, [alias + suffix]);
if (!target)
throw Object.assign(new Error('Invalid subscription path'), {
status: 400,
});
return target;
}
+106
View File
@@ -0,0 +1,106 @@
import type { Subscription } from '../data/subscription';
import { formatUrl } from '../config/subscription';
import path from 'path';
import fs from 'fs';
import { resolveFileAccess } from './fileAccess';
// Storage namespaces come from URLs and branches, not user aliases. In
// particular, a legal Git directory can exceed the alias's 200-character limit.
export function isSubscriptionStorageName(name: string): boolean {
return (
typeof name === 'string' &&
!!name &&
!path.isAbsolute(name) &&
!/[\0\r\n\\]/.test(name) &&
!name.split('/').some((part) => !part || part === '.' || part === '..')
);
}
export function resolveSubscriptionStoragePath(
root: string,
name: string,
): string | undefined {
const target = isSubscriptionStorageName(name)
? resolveFileAccess(root, [name])
: '';
if (!target) {
if (isSubscriptionStorageName(name)) {
try {
fs.lstatSync(path.resolve(root, name));
} catch (error: any) {
// A long query can make the Shell candidate unrepresentable while
// the CLI's pathname-based file exists. Use the actual filesystem
// limit: macOS permits Unicode names exceeding 255 UTF-8 bytes.
if (error.code === 'ENAMETOOLONG') return undefined;
}
}
throw Object.assign(new Error('Invalid subscription path'), {
status: 400,
});
}
return target;
}
/** Match shell/update.sh get_uniq_path and update_raw, including SCP URLs. */
export function getSubscriptionStorageNames(
doc: Subscription,
engine: 'shell' | 'cli' = 'shell',
): {
script: string;
repo?: string;
raw?: string;
} {
// Old imported rows may only have an alias and no URL.
if (!doc.url) return { script: doc.alias, repo: doc.alias };
const url = formatUrl(doc).url!;
const trimmed = url.endsWith('/') ? url.slice(0, -1) : url;
const basename = trimmed.slice(trimmed.lastIndexOf('/') + 1);
const dot = basename.lastIndexOf('.');
const repo = dot < 0 ? basename : basename.slice(0, dot);
const parentUrl = engine === 'cli' ? trimmed : url;
const slash = parentUrl.lastIndexOf('/');
const parent =
engine === 'shell' && slash < 0 ? url : parentUrl.slice(0, slash);
const owner = parent.slice(parent.lastIndexOf('/') + 1);
const author = owner.slice(owner.lastIndexOf(':') + 1);
const stem = `${author.slice(author.lastIndexOf('.') + 1)}_${repo}`;
if (doc.type === 'file') {
const suffix =
engine === 'cli'
? path.extname(new URL(url).pathname)
: `.${url.slice(url.lastIndexOf('.') + 1)}`;
const raw = `${stem}${suffix}`;
return { script: `raw_${raw}`, raw };
}
const directory = doc.branch ? `${stem}_${doc.branch}` : stem;
return { script: directory, repo: directory };
}
// A subscription may have run through either engine before being deleted.
export function getSubscriptionStorageCandidates(doc: Subscription) {
const candidates: ReturnType<typeof getSubscriptionStorageNames>[] = [];
for (const engine of ['shell', 'cli'] as const) {
try {
candidates.push(getSubscriptionStorageNames(doc, engine));
} catch {
// Historical rows may have an invalid SSH alias or missing credentials.
// Retain their URL-derived storage without needing an executable SSH URL.
try {
candidates.push(
getSubscriptionStorageNames(
{
url: doc.url,
alias: doc.alias,
branch: doc.branch,
type: doc.type === 'private-repo' ? 'public-repo' : doc.type,
},
engine,
),
);
} catch {
// One engine's invalid URL must not hide the other engine's candidate.
}
}
}
return candidates;
}
+1 -1
View File
@@ -4,7 +4,7 @@ import path from 'path';
import { writeFile, open, chmod } from 'fs/promises';
import { fileExist } from '../config/util';
function getUniqueLockPath(filePath: string) {
export function getUniqueLockPath(filePath: string) {
const sanitizedPath = filePath
.replace(/[<>:"/\\|?*]/g, '_')
.replace(/^_/, '');
+5 -16
View File
@@ -2,30 +2,19 @@ import 'reflect-metadata';
import OpenService from './services/open';
import { Container } from 'typedi';
import LoggerInstance from './loaders/logger';
import config from './config';
import path from 'path';
import os from 'os';
import { writeFileWithLock } from './shared/utils';
const tokenFile = path.join(config.configPath, 'token.json');
async function getToken() {
try {
Container.set('logger', LoggerInstance);
const openService = Container.get(OpenService);
const appToken = await openService.generateSystemToken();
const appToken = await openService.generateSystemToken(
process.argv.includes('--renew'),
);
console.log(appToken.value);
await writeFile({
value: appToken.value,
expiration: appToken.expiration,
});
} catch (error) {
console.log(error);
console.error(error);
process.exitCode = 1;
}
}
async function writeFile(data: any) {
await writeFileWithLock(tokenFile, `${JSON.stringify(data)}${os.EOL}`);
}
getToken();
+16 -4
View File
@@ -5,7 +5,7 @@
The npm and panel-internal entries both use the name `ql`, but have separate Commander command trees. The npm entry only calls remote APIs; the internal entry only runs local tools. Verify the absolute executable path and `--help` before use. Installing the npm package does not migrate the built-in Shell commands.
`@whyour/qinglong-cli` is a standalone npm client for the panel's open API. It registers only `ql`, for the currently supported OpenAPI resources. Local execution, repo/raw workers, reload/update/account recovery belong to the panel's internal tools and are excluded from npm. Development publishing is outside both command sets.
`@whyour/qinglong-cli` is a standalone npm client for the panel's open API. It registers only `ql`, for the currently supported OpenAPI resources. Local execution, repo/raw workers, reload/update/account recovery belong to the panel's internal tools and are excluded from npm. Remote CLI commands do not manage development publishing.
## Installation and commands
@@ -43,7 +43,7 @@ Use command `--help` and `QL_LANG=en` for English. Command names and JSON fields
## Complete OpenAPI management
The CLI also supports task/subscription CRUD, application and secret management, environment variables, configuration, scripts, logs, dependencies, system, dashboard and user APIs. `ql api routes --json` lists all 143 active routes; three retired file-reading endpoints are excluded. CI compares the catalogue against backend routes. See the [complete bilingual reference](skills/qinglong-cli/references/openapi.md) for every command and payload.
The CLI also supports task/subscription CRUD, application and secret management, environment variables, configuration, scripts, logs, dependencies, system, dashboard and user APIs. `ql api routes --json` lists all 146 active routes; three retired file-reading endpoints are excluded. CI compares the catalogue against backend routes. See the [complete bilingual reference](skills/qinglong-cli/references/openapi.md) for every command and payload.
```sh
ql task create --name demo --command 'task demo.js' --schedule '0 0 * * *' --json
@@ -53,8 +53,20 @@ ql env create --data @envs.json --json
ql api request PUT /open/crons/run --data '[12,13]' --json
```
CLI 0.1.2 adds script history listing, detail and confirmed restore commands, requiring the history APIs in QingLong 2.23.0. Restore uses the current hash from the detail response; on 409, read and review the current state again.
```sh
ql script history-list --query '{"filename":"demo.js","path":""}' --json
ql script history-detail --query '{"filename":"demo.js","path":"","id":"VERSION_UUID"}' --json
ql script history-restore --data '{"filename":"demo.js","path":"","id":"VERSION_UUID","expectedHash":"CURRENT_SHA256"}' --json
```
Replace `VERSION_UUID` and `CURRENT_SHA256` with values returned by the list and detail commands.
Use --data JSON/@file/- for bodies, --query for query objects, --file for uploads and --output for downloads. New commands support --timeout seconds. Existing command contracts remain; raw API requests expose all fields and batch operations. Downloads do not overwrite files. App secrets require --show-secrets; other raw resources may contain sensitive data.
`--timeout` only limits waiting for the HTTP response; it does not stop server-side tasks. A failed write may already have taken effect, so inspect the resource before retrying.
Remote ql system commands call the target panel API. Local reload/reset tools remain excluded from npm.
## Authentication
@@ -130,9 +142,9 @@ node cli/scripts/verify-package.cjs
Tests cover requests, token refresh, output, errors, no automatic retries, permissions and isolated installation. Package verification installs the archive offline, verifies the sole ql executable and rejects local commands.
The CLI package workflow checks, builds and tests Node 22.12/24 on relevant PRs, develop/master pushes and manual runs. Node 24 uploads the archive verified by offline installation as `qinglong-cli-<commit>`. After both matrix jobs succeed, master pushes publish that exact archive to npm as latest, using GitHub Actions OIDC trusted publishing without an NPM_TOKEN secret. Manual runs publish only when run on master with publish enabled; PRs, develop and forks never publish. Configure the npm Trusted Publisher for repository `whyour/qinglong`, workflow `cli-package.yml`, with `npm publish` allowed. The panel package `@whyour/qinglong` separately trusts `build-docker-image.yml`. Publishing uses Node 24 with `id-token: write` granted only to the publish job. New packages need an initial publication before configuring their trusted publisher.
The CLI package workflow checks, builds and tests Node 22.12/24 on relevant PRs, develop/master pushes and manual runs. Node 24 uploads the archive verified by offline installation as `qinglong-cli-<commit>`. After both matrix jobs succeed, master pushes publish that exact archive to npm as latest, using GitHub Actions OIDC trusted publishing without an NPM_TOKEN secret. A manual release requires publish on master, or publish plus an explicit `X.Y.Z-alpha.N`, `X.Y.Z-beta.N` or `X.Y.Z-rc.N` version on develop. Develop prereleases set the same requested version before both Node verification jobs and publish to alpha/beta/rc without changing latest. Ordinary PRs, develop pushes, validation-only manual runs and forks do not publish. Releases and validation use separate concurrency groups; new validation does not cancel an active release. Configure the npm Trusted Publisher for repository `whyour/qinglong`, workflow `cli-package.yml`, with `npm publish` allowed. The panel package `@whyour/qinglong` separately trusts `build-docker-image.yml`. Publishing uses Node 24 with `id-token: write` granted only to the publish job. New packages need an initial publication before configuring their trusted publisher.
The CLI has an independent stable version in cli/package.json and cli/package-lock.json. Before releasing changes, run `npm version patch --prefix cli --no-git-tag-version` (or minor/major) and commit both files. Existing versions are skipped with a notice; registry failures stop publication. Only stable X.Y.Z versions are published by this workflow. Publication does not rebuild the verified archive or run package lifecycle scripts.
The CLI has an independent stable version in cli/package.json and cli/package-lock.json. Before releasing changes, run `npm version patch --prefix cli --no-git-tag-version` (or minor/major) and commit both files. Existing stable versions are skipped with a notice; an existing prerelease is an error and requires a new version. Registry failures stop publication. Manual develop prereleases only update the version in that verification workspace and do not replace stable source metadata maintenance. Publication does not rebuild the verified archive or run package lifecycle scripts.
## Panel-internal tools
+16 -4
View File
@@ -5,7 +5,7 @@
npm 与面板内部入口都叫 `ql`,但使用独立的 Commander 命令树:npm 入口只调用远程 API,内部入口只运行本机工具。使用前确认可执行文件的绝对路径和 `--help`;安装 npm 包不会迁移内置 Shell 命令。
`@whyour/qinglong-cli` 是独立 npm 包,覆盖当前 develop 的有效 OpenAPI,只注册一个 `ql` 命令。它不包含脚本执行器或本机运维实现;`task exec`、`repo/raw`、`reload/update/reset*` 等属于面板内部工具,不随 npm 包分发。开发发布也不属于 CLI 范围。
`@whyour/qinglong-cli` 是独立 npm 包,覆盖当前 develop 的有效 OpenAPI,只注册一个 `ql` 命令。它不包含脚本执行器或本机运维实现;`task exec`、`repo/raw`、`reload/update/reset*` 等属于面板内部工具,不随 npm 包分发。远程 CLI 命令不提供开发发布操作。
## 安装与使用
@@ -43,7 +43,7 @@ ql auth logout --json
## 全量 OpenAPI 管理
现在还支持任务/订阅创建、修改、删除,应用管理与密钥重置,以及环境变量、配置、脚本、日志、依赖、系统、仪表盘和用户管理。`ql api routes --json` 列出全部 143 条有效路由;3 条已下线文件读取接口不包含在内。新增命令在 [完整双语参考](skills/qinglong-cli/references/openapi.md) 中逐项列出,路由覆盖由 CI 与后端代码核对。
现在还支持任务/订阅创建、修改、删除,应用管理与密钥重置,以及环境变量、配置、脚本、日志、依赖、系统、仪表盘和用户管理。`ql api routes --json` 列出全部 146 条有效路由;3 条已下线文件读取接口不包含在内。新增命令在 [完整双语参考](skills/qinglong-cli/references/openapi.md) 中逐项列出,路由覆盖由 CI 与后端代码核对。
```sh
ql task create --name demo --command 'task demo.js' --schedule '0 0 * * *' --json
@@ -53,8 +53,20 @@ ql env create --data @envs.json --json
ql api request PUT /open/crons/run --data '[12,13]' --json
```
CLI 0.1.2 增加脚本历史列表、详情和确认恢复命令,需要青龙 2.23.0 的历史 API。恢复使用详情返回的当前哈希,409 后应重新读取并确认。
```sh
ql script history-list --query '{"filename":"demo.js","path":""}' --json
ql script history-detail --query '{"filename":"demo.js","path":"","id":"VERSION_UUID"}' --json
ql script history-restore --data '{"filename":"demo.js","path":"","id":"VERSION_UUID","expectedHash":"CURRENT_SHA256"}' --json
```
将 `VERSION_UUID` 和 `CURRENT_SHA256` 替换为列表和详情实际返回值。
请求体使用 --data JSON/@file/-,查询使用 --query,上传 --file,下载 --output。新命令支持 --timeout 秒数;旧命令行为保留,完整参数可用 api request。下载不覆盖现有文件,应用密钥默认隐藏,明确加 --show-secrets 才输出。新增资源通常保留原始返回字段,注意环境、配置和会话信息可能敏感。
`--timeout` 只限制等待 HTTP 响应,不停止服务器上的任务;失败的写入可能已经生效,应先查询状态。
远程 `ql system ...` 调用面板 API;本机 reload/reset 等仍不在 npm 包中。不要将远程 API 覆盖理解为本机运维重新混入包。
## 认证
@@ -130,9 +142,9 @@ node cli/scripts/verify-package.cjs
测试覆盖请求格式、认证刷新、输出、错误、禁止自动重试、权限和独立安装。打包验证会离线安装 tgz,并确认唯一入口是 `ql`,本机命令不可调用。
CLI package 工作流在相关 PR、develop/master 推送和手动触发时执行 Node 22.12/24 类型检查、构建及测试。Node 24 上传通过离线安装验证的 `qinglong-cli-<commit>` artifact。两个矩阵任务成功后,master 推送会将这份已验证的 tgz 发布到 npm 的 latest 标签,通过 GitHub Actions OIDC 可信发布,无需 `NPM_TOKEN` Secret。手动运行需选择 master 并勾选 publish;PR、develop 和 fork 不发布。npm 的 Trusted Publisher 需绑定仓库 `whyour/qinglong` 和工作流 `cli-package.yml`,允许 `npm publish`;面板包 `@whyour/qinglong` 单独绑定 `build-docker-image.yml`。发布 job 使用 Node 24,并仅在该 job 授予 `id-token: write`。新包需先完成首次发布,再配置对应包的可信发布关系。
CLI package 工作流在相关 PR、develop/master 推送和手动触发时执行 Node 22.12/24 类型检查、构建及测试。Node 24 上传通过离线安装验证的 `qinglong-cli-<commit>` artifact。两个矩阵任务成功后,master 推送会将这份已验证的 tgz 发布到 npm 的 latest 标签,通过 GitHub Actions OIDC 可信发布,无需 `NPM_TOKEN` Secret。手动发布在 master 勾选 publish,或在 develop 勾选 publish 并填写 `X.Y.Z-alpha.N`、`X.Y.Z-beta.N` 或 `X.Y.Z-rc.N` 版本。develop 预发布在两个 Node 矩阵中先设置相同版本并验证,发布对应 alpha/beta/rc 标签,不更新 latest。普通 PR、develop 推送、仅验证的手动运行和 fork 不发布。发布与验证使用独立并发组,正在运行的发布不会被新验证取消。npm 的 Trusted Publisher 需绑定仓库 `whyour/qinglong` 和工作流 `cli-package.yml`,允许 `npm publish`;面板包 `@whyour/qinglong` 单独绑定 `build-docker-image.yml`。发布 job 使用 Node 24,并仅在该 job 授予 `id-token: write`。新包需先完成首次发布,再配置对应包的可信发布关系。
CLI 版本独立维护在 cli/package.json 和 cli/package-lock.json。发布改动前执行 `npm version patch --prefix cli --no-git-tag-version`(也可用 minor/major),提交这两个文件。已发布的版本会提示并跳过;registry 查询失败则停止发布。该流程仅发布 X.Y.Z 稳定版本,不重新构建产物或执行包生命周期脚本。
CLI 版本独立维护在 cli/package.json 和 cli/package-lock.json。发布改动前执行 `npm version patch --prefix cli --no-git-tag-version`(也可用 minor/major),提交这两个文件。已存在的稳定版本提示并跳过;预发布版本已存在则失败,需选择新版本。registry 查询失败停止发布。手动 develop 预发布只临时修改本轮验证工作区的版本,不代替稳定版本元数据维护。发布不重新构建已验证归档,也不执行包生命周期脚本。
## 面板内部工具
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "@whyour/qinglong-cli",
"version": "0.1.1",
"version": "0.1.2",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@whyour/qinglong-cli",
"version": "0.1.1",
"version": "0.1.2",
"bin": {
"ql": "dist/npm/ql.js"
},
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@whyour/qinglong-cli",
"version": "0.1.1",
"version": "0.1.2",
"description": "Authenticated remote management CLI for QingLong 2.x",
"type": "commonjs",
"bin": {
+7 -1
View File
@@ -13,6 +13,10 @@ execFileSync(
],
{ stdio: 'inherit' },
);
fs.copyFileSync(
path.join(root, 'src/internal/maintenance/upgradeArtifacts.cjs'),
path.join(root, 'dist/internal/maintenance/upgradeArtifacts.cjs'),
);
require('esbuild').buildSync({
entryPoints: [path.join(root, 'src/shared/cli/commander.ts')],
outfile: path.join(root, 'dist/shared/cli/commander.js'),
@@ -45,7 +49,9 @@ const remote = require('esbuild').buildSync({
metafile: true,
});
for (const input of Object.keys(remote.metafile.inputs)) {
if (/(?:^|[\\/])src[\\/](?:internal|compatibility|developer)[\\/]/.test(input))
if (
/(?:^|[\\/])src[\\/](?:internal|compatibility|developer)[\\/]/.test(input)
)
throw new Error(
'Local implementation leaked into remote npm bundle: ' + input,
);
@@ -1,8 +1,8 @@
# OpenAPI 全量参考 / Complete OpenAPI reference
CLI 对应当前 develop 的 143 条有效路由。通过 `ql api routes --json` 查看方法、路径、对应命令、位置参数、请求体与上传字段;CLI 测试与 back/api 路由逐项核对。GET configs/:file、scripts/:file、logs/:file 已返回 410,改用 detail 命令,不再暴露旧入口。旧版面板可能没有新路由,以服务端响应为准。
CLI 对应本版本后端的 146 条有效路由。通过 `ql api routes --json` 查看方法、路径、对应命令、位置参数、请求体与上传字段;CLI 测试与 back/api 路由逐项核对。GET configs/:file、scripts/:file、logs/:file 已返回 410,改用 detail 命令,不再暴露旧入口。旧版面板可能没有新路由,以服务端响应为准。
The catalogue covers 143 active routes in the current develop backend. `ql api routes --json` lists methods, paths, commands, positional parameters, bodies and upload fields. A route-coverage test detects drift. Three retired GET :file routes return 410; use detail-based commands. Older panel versions may not implement newer routes.
The catalogue covers 146 active routes in the current backend. `ql api routes --json` lists methods, paths, commands, positional parameters, bodies and upload fields. A route-coverage test detects drift. Three retired GET :file routes return 410; use detail-based commands. Older panel versions may not implement newer routes.
## 认证与权限 / Authentication and permissions
@@ -46,6 +46,9 @@ ql env create --data @envs.json --json
ql env update 8 --data '{"name":"EXAMPLE","value":"value"}' --json
ql script create --file ./demo.js --data '{"filename":"demo.js","path":""}' --json
ql script get --query '{"file":"demo.js","path":""}' --json
ql script history-list --query '{"filename":"demo.js","path":""}' --json
ql script history-detail --query '{"filename":"demo.js","path":"","id":"VERSION_UUID"}' --json
ql script history-restore --data '{"filename":"demo.js","path":"","id":"VERSION_UUID","expectedHash":"CURRENT_SHA256"}' --json
ql config save --data '{"name":"example.sh","content":"# example"}' --json
ql config get --query '{"path":"example.sh"}' --json
ql log download --data '{"filename":"example.log","path":"example"}' --output ./example.log --json
@@ -133,6 +136,9 @@ System/user operations below execute remotely through the panel API. They do not
| `ql config save ` | POST | `configs/save` | --data, --query |
| `ql script list ` | GET | `scripts` | --query |
| `ql script get ` | GET | `scripts/detail` | --query |
| `ql script history-list ` | GET | `scripts/history` | --query |
| `ql script history-detail ` | GET | `scripts/history/detail` | --query |
| `ql script history-restore ` | PUT | `scripts/history/restore` | --data, --query |
| `ql script create ` | POST | `scripts` | --data, --file (file), --query |
| `ql script update ` | PUT | `scripts` | --data, --file (file), --query |
| `ql script delete ` | DELETE | `scripts` | --data, --query |
+18 -2
View File
@@ -70,10 +70,26 @@ if [ -n "$__ql_timeout" ]; then
fi
if [ "$__ql_command" = true ]; then
"\${__ql_args[@]}" "\${__ql_script_args[@]}"
_task_exit_code=$?
else
. "\${__ql_args[0]}" "\${__ql_script_args[@]}"
# Match task.sh: keep script exit/exec/options inside a child shell while
# retaining before-hook variables and functions for the script.
# Keep the supervisor alive until the child's signal/EXIT traps finish, but
# do not turn user cancellation into an ordinary completion with after hooks.
__ql_saved_signals=$(trap -p INT TERM HUP QUIT ALRM TSTP)
__ql_cancel_code=''
trap '__ql_cancel_code=$((128 + $(kill -l INT)))' INT
trap '__ql_cancel_code=$((128 + $(kill -l TERM)))' TERM
trap '__ql_cancel_code=$((128 + $(kill -l HUP)))' HUP
trap '__ql_cancel_code=$((128 + $(kill -l QUIT)))' QUIT
trap '__ql_cancel_code=$((128 + $(kill -l ALRM)))' ALRM
trap '__ql_cancel_code=$((128 + $(kill -l TSTP)))' TSTP
( . "\${__ql_args[0]}" "\${__ql_script_args[@]}" )
_task_exit_code=$?
trap - INT TERM HUP QUIT ALRM TSTP
eval "$__ql_saved_signals"
if [ -n "$__ql_cancel_code" ] && ! [ -f "$__ql_timeout" ]; then exit "$__ql_cancel_code"; fi
fi
_task_exit_code=$?
if [ -n "$__ql_timeout" ] && [ -f "$__ql_timeout" ]; then _task_exit_code=124; fi
__ql_run_after
exit "$_task_exit_code"
+6 -6
View File
@@ -8,6 +8,7 @@ import { repairConfiguration, startPanel } from './operator';
import { operatingSystem } from './bot';
import { fail } from '../../shared/errors';
import { registerHostServices } from '../runtime/hostServices';
import { installRuntimeTools } from './tools';
export function bootstrapPackages(
os: string,
@@ -92,10 +93,13 @@ export function runtimeEnvironment(
.filter(Boolean)
.join(path.delimiter),
NODE_PATH: [
context.env.NODE_PATH,
'/usr/local/bin',
'/usr/local/lib/node_modules',
path.join(node, 'global/5/node_modules'),
].join(path.delimiter),
]
.filter(Boolean)
.join(path.delimiter),
PYTHONPATH: [
python,
path.join(python, `lib/python${pythonVersion}`),
@@ -176,11 +180,7 @@ export async function bootstrapPanel(
root ? args : [packages.program, ...args],
{ cwd: context.root, env: context.env },
);
await checkedProcess(
'npm',
['install', '-g', 'pnpm@8.3.1', 'pm2', 'ts-node', 'typescript@5'],
{ cwd: context.root, env: context.env },
);
await installRuntimeTools(context);
}
const python = await checkedProcess(
'python3',
+2 -6
View File
@@ -4,7 +4,7 @@ import path from 'node:path';
import http from 'node:http';
import type { LocalContext } from '../runtime/context';
import { atomicWrite } from '../runtime/files';
import { checkedProcess } from '../runtime/process';
import { installRuntimeTools } from './tools';
import {
repairConfiguration,
installPanelDependencies,
@@ -152,11 +152,7 @@ export async function diagnosticLog(file: string): Promise<{
}
export async function checkAndRepair(context: LocalContext): Promise<unknown> {
await checkedProcess(
'npm',
['i', '-g', 'pnpm@8.3.1', 'pm2', 'ts-node', 'typescript@5'],
{ cwd: context.root, env: context.env },
);
await installRuntimeTools(context);
const restored = await repairConfiguration(context);
await installPanelDependencies(context);
const copied: string[] = [];
+10 -6
View File
@@ -71,14 +71,18 @@ export async function executableAvailable(
export async function installPanelDependencies(
context: LocalContext,
cwd = context.root,
staticRoot?: string,
sourceCommit?: string,
): Promise<void> {
const termux = context.env.is_termux === '1';
const pnpm = !termux && (await executableAvailable('pnpm', context.env));
await checkedProcess(
pnpm ? 'pnpm' : 'npm',
pnpm
? ['install', '--loglevel', 'error', '--production']
: ['install', '--production', ...(termux ? ['--no-bin-links'] : [])],
process.execPath,
[
path.join(context.root, 'scripts/install-panel-dependencies.cjs'),
'--root',
cwd,
...(staticRoot ? ['--static-dir', staticRoot] : []),
...(sourceCommit ? ['--source-commit', sourceCommit] : []),
],
{ cwd, env: context.env },
);
}
+38
View File
@@ -0,0 +1,38 @@
import path from 'node:path';
import type { LocalContext } from '../runtime/context';
import { checkedProcess } from '../runtime/process';
// Preserve the installation's configured global prefix, without resolving new
// registry packages or replacing the user's persisted script dependencies.
export async function installRuntimeTools(context: LocalContext): Promise<void> {
const options = { cwd: context.root, env: context.env };
const result = await checkedProcess('npm', ['prefix', '--global'], {
...options,
capture: true,
});
const prefix = result.stdout.trim();
if (!path.isAbsolute(prefix) || prefix.includes('\n'))
throw new Error('npm returned an invalid global prefix.');
await checkedProcess(
process.execPath,
[
path.join(context.root, 'scripts/install-runtime-tools.cjs'),
'--archive',
path.join(context.root, 'static/runtime-tools.tgz'),
'--proof',
path.join(context.root, 'static/runtime-tools-proof.json'),
'--prefix',
prefix,
],
options,
);
context.env.PATH = [path.join(prefix, 'bin'), context.env.PATH]
.filter(Boolean)
.join(path.delimiter);
context.env.NODE_PATH = [
path.join(prefix, 'lib/node_modules'),
context.env.NODE_PATH,
]
.filter(Boolean)
.join(path.delimiter);
}
+52 -129
View File
@@ -8,12 +8,33 @@ import { checkedProcess } from '../runtime/process';
import { operationSignal, withoutCancellation } from '../runtime/cancellation';
import { installPanelDependencies, startPanel, stopPanel } from './operator';
// The Bash image updater and local CLI must enforce the same build provenance.
const artifacts = require('./upgradeArtifacts.cjs') as {
releaseBranch(env: NodeJS.ProcessEnv): string;
verifyUpgrade(source: string, staticRoot: string): Promise<unknown>;
waitForHealth(
env: NodeJS.ProcessEnv,
timeout?: number,
signal?: AbortSignal,
): Promise<void>;
withUpgradeLock<T>(tmp: string, operation: () => Promise<T>): Promise<T>;
stageUpgrade(options: {
root: string;
tmp: string;
env: NodeJS.ProcessEnv;
mirror: 'github' | 'gitee';
signal?: AbortSignal;
run(program: string, args: string[], capture?: boolean): Promise<unknown>;
install(source: string, staticRoot: string, sourceCommit: string): Promise<void>;
}): Promise<{ source: string; static: string }>;
selectedUpgrade(
tmp: string,
branch: string,
): Promise<{ source: string; static: string }>;
};
export function releaseBranch(context: LocalContext): string {
return ['develop', 'debian', 'debian-dev'].includes(
context.env.QL_BRANCH ?? '',
)
? context.env.QL_BRANCH!
: 'master';
return artifacts.releaseBranch(context.env);
}
async function validateTree(
@@ -48,134 +69,28 @@ export async function stageUpgrade(
context: LocalContext,
mirror: 'github' | 'gitee',
): Promise<{ source: string; static: string }> {
await fs.mkdir(context.paths.dir_tmp!, { recursive: true });
const workspace = await fs.mkdtemp(
path.join(context.paths.dir_tmp!, 'upgrade-'),
return artifacts.withUpgradeLock(context.paths.dir_tmp!, () =>
artifacts.stageUpgrade({
root: context.root,
tmp: context.paths.dir_tmp!,
env: context.env,
mirror,
signal: operationSignal(),
run: (program, args, capture) =>
checkedProcess(program, args, { env: context.env, capture }),
install: (source, staticRoot, sourceCommit) =>
installPanelDependencies(context, source, staticRoot, sourceCommit),
}),
);
const branch = releaseBranch(context);
try {
for (const repository of ['qinglong', 'qinglong-static']) {
const url =
mirror === 'github'
? `https://github.com/whyour/${repository}/archive/refs/heads/${branch}.zip`
: `https://gitee.com/whyour/${repository}/repository/archive/${branch}.zip`;
const archive = path.join(workspace, `${repository}.zip`);
await checkedProcess(
'curl',
[
'--fail',
'--location',
'--proto',
'=https',
'--proto-redir',
'=https',
'--output',
archive,
url,
],
{ env: context.env },
);
const listing = await checkedProcess('unzip', ['-Z1', archive], {
env: context.env,
capture: true,
});
const expected = `${repository}-${branch}`;
for (const name of listing.stdout.split('\n').filter(Boolean)) {
if (
name.includes('\\') ||
name.includes('\r') ||
name.startsWith('/') ||
name.split('/').includes('..') ||
name.split('/')[0] !== expected
)
throw new Error(translate(context.env, '升级归档包含无效路径。'));
}
// Reject Unix symlinks before extraction, including links that could redirect
// a later archive entry outside the staging directory.
const modes = await checkedProcess('unzip', ['-Z', '-l', archive], {
env: context.env,
capture: true,
});
if (modes.stdout.split('\n').some((line) => /^l[rwx-]{9}\s/.test(line)))
throw new Error(translate(context.env, '升级归档包含符号链接。'));
await checkedProcess('unzip', ['-oq', archive, '-d', workspace], {
env: context.env,
});
await validateTree(path.join(workspace, expected), context.env);
}
const source = path.join(workspace, `qinglong-${branch}`);
const staticRoot = path.join(workspace, `qinglong-static-${branch}`);
await fs.access(path.join(staticRoot, 'build/app.js'));
const manifest = await fs.readFile(path.join(source, 'package.json'));
const existing = await fs.readFile(path.join(context.root, 'package.json'));
if (!manifest.equals(existing))
await installPanelDependencies(context, source);
// Publish a marker only after both archives and dependency installation succeed.
await fs.writeFile(
path.join(workspace, 'ready.json'),
JSON.stringify({ source, static: staticRoot }),
{ flag: 'wx', mode: 0o600 },
);
const pointer = path.join(workspace, 'pointer.json');
await fs.writeFile(
pointer,
JSON.stringify({ directory: path.basename(workspace) }),
{
flag: 'wx',
mode: 0o600,
},
);
await fs.rename(
pointer,
path.join(context.paths.dir_tmp!, `upgrade-ready-${branch}.json`),
);
return { source, static: staticRoot };
} catch (error) {
await fs.rm(workspace, { recursive: true, force: true });
throw error;
}
}
async function selectedUpgrade(
context: LocalContext,
): Promise<{ source: string; static: string }> {
const branch = releaseBranch(context);
const fallback = {
source: path.join(context.paths.dir_tmp!, `qinglong-${branch}`),
static: path.join(context.paths.dir_tmp!, `qinglong-static-${branch}`),
};
const pointer = await fs
.readFile(
path.join(context.paths.dir_tmp!, `upgrade-ready-${branch}.json`),
'utf8',
)
.catch((error: NodeJS.ErrnoException) => {
if (error.code === 'ENOENT') return undefined;
throw error;
});
if (pointer === undefined) return fallback;
const directory: unknown = JSON.parse(pointer)?.directory;
if (
typeof directory !== 'string' ||
!/^upgrade-[A-Za-z0-9_-]+$/.test(directory)
)
throw new Error(translate(context.env, '暂存升级指针无效。'));
const workspace = path.join(context.paths.dir_tmp!, directory);
const selected = {
source: path.join(workspace, `qinglong-${branch}`),
static: path.join(workspace, `qinglong-static-${branch}`),
};
for (const location of [workspace, selected.source, selected.static]) {
const stat = await fs.lstat(location);
if (!stat.isDirectory() || stat.isSymbolicLink())
throw new Error(translate(context.env, '暂存升级目录无效。'));
}
const ready = JSON.parse(
await fs.readFile(path.join(workspace, 'ready.json'), 'utf8'),
return artifacts.selectedUpgrade(
context.paths.dir_tmp!,
releaseBranch(context),
);
if (ready?.source !== selected.source || ready?.static !== selected.static)
throw new Error(translate(context.env, '暂存升级就绪记录与目录不匹配。'));
return selected;
}
export interface ReloadLifecycle {
@@ -345,8 +260,7 @@ export async function reloadPanel(
const staticRoot = selected.static;
await validateTree(source, context.env);
await validateTree(staticRoot, context.env);
await fs.access(path.join(source, 'package.json'));
await fs.access(path.join(staticRoot, 'build/app.js'));
await artifacts.verifyUpgrade(source, staticRoot);
const entries = await fs.readdir(source);
const reserved = new Set(['data', '.tmp', 'static', '.git', '.env']);
const relativeData = path.relative(context.root, context.data);
@@ -371,7 +285,16 @@ export async function reloadPanel(
if (selection)
replacements.push({ source: selection.source, target: selection.target });
try {
return await replaceAndReload(context, replacements);
return await artifacts.withUpgradeLock(context.paths.dir_tmp!, () =>
replaceAndReload(context, replacements, {
stop: stopPanel,
start: async (ctx) => {
const service = await startPanel(ctx);
await artifacts.waitForHealth(ctx.env, 60000, operationSignal());
return service;
},
}),
);
} finally {
if (selection)
await fs.rm(selection.directory, { recursive: true, force: true });
@@ -0,0 +1,664 @@
// Shared by the image's Bash updater and the optional local CLI. Use only
// installed code to validate downloads; never execute a downloaded verifier.
const fs = require('node:fs/promises');
const path = require('node:path');
const { createHash, randomUUID } = require('node:crypto');
const { spawn } = require('node:child_process');
const http = require('node:http');
const { setTimeout: delay } = require('node:timers/promises');
const hash = (bytes) => createHash('sha256').update(bytes).digest('hex');
const releaseBranch = (env) =>
['develop', 'debian-dev'].includes(env.QL_BRANCH) ? 'develop' : 'master';
async function manifestAt(staticRoot) {
const manifest = JSON.parse(
await fs.readFile(path.join(staticRoot, 'build-info.json'), 'utf8'),
);
if (
manifest.version !== 1 ||
manifest.dirty !== false ||
!/^[a-f0-9]{40}$/.test(manifest.sourceCommit) ||
!/^[a-f0-9]{64}$/.test(manifest.lockfileSha256) ||
!manifest.files ||
typeof manifest.files !== 'object' ||
Array.isArray(manifest.files)
)
throw new Error('Invalid or dirty build manifest / 构建清单无效。');
for (const required of ['build/app.js', 'dist/index.html'])
if (!Object.hasOwn(manifest.files, required))
throw new Error(`Missing build output / 缺少构建产物: ${required}`);
return manifest;
}
async function verifyUpgrade(source, staticRoot) {
const manifest = await manifestAt(staticRoot);
if (
hash(await fs.readFile(path.join(source, 'pnpm-lock.yaml'))) !==
manifest.lockfileSha256
)
throw new Error(
'Source and build lockfiles differ / 源码与构建依赖不一致。',
);
await fs.access(path.join(source, 'package.json'));
await fs.access(path.join(source, 'sample/config.sample.sh'));
const files = {};
async function visit(directory, prefix = '') {
for (const entry of await fs.readdir(directory, { withFileTypes: true })) {
const name = prefix + entry.name;
if (name === 'build-info.json') continue;
const filename = path.join(directory, entry.name);
if (entry.isDirectory()) await visit(filename, `${name}/`);
else if (entry.isFile()) files[name] = hash(await fs.readFile(filename));
else
throw new Error(`Unsupported build entry / 构建包含非法文件: ${name}`);
}
}
await visit(staticRoot);
const actual = Object.keys(files).sort(),
expected = Object.keys(manifest.files).sort();
if (
JSON.stringify(actual) !== JSON.stringify(expected) ||
actual.some((name) => files[name] !== manifest.files[name])
)
throw new Error(
'Build checksum mismatch / 构建文件缺失、损坏或混入旧文件。',
);
return manifest;
}
async function withUpgradeLock(tmp, operation) {
await fs.mkdir(tmp, { recursive: true });
const lock = path.join(tmp, 'upgrade.lock');
const token = randomUUID();
const owner = `owner-${process.pid}-${token}`;
const candidate = path.join(tmp, `upgrade-lock-${token}`);
let acquired = false;
await fs.mkdir(candidate, { mode: 0o700 });
try {
// Publish a nonempty directory atomically: a live lock can never be
// replaced, and a crash before publication cannot leave an ownerless lock.
await fs.writeFile(path.join(candidate, owner), '', {
flag: 'wx',
mode: 0o600,
});
for (;;) {
try {
await fs.rename(candidate, lock);
acquired = true;
break;
} catch (error) {
if (!['ENOTEMPTY', 'EEXIST'].includes(error.code)) throw error;
}
let entries;
try {
entries = await fs.readdir(lock);
} catch (error) {
if (error.code === 'ENOENT') continue;
throw error;
}
if (!entries.length) continue;
const heldOwner = entries[0];
const match = /^owner-([1-9]\d*)-[a-f0-9-]{36}$/.exec(heldOwner);
let pid = match ? Number(match[1]) : 0;
if (entries.length === 1 && heldOwner === 'pid') {
try {
pid = Number(await fs.readFile(path.join(lock, heldOwner), 'utf8'));
} catch (error) {
if (error.code === 'ENOENT') continue;
throw error;
}
}
let stale = false;
if (
entries.length === 1 &&
Number.isSafeInteger(pid) &&
pid > 0 &&
pid <= 0x7fffffff
) {
try {
process.kill(pid, 0);
} catch (error) {
if (error.code === 'ESRCH') stale = true;
}
}
if (!stale)
throw new Error(
'Another upgrade is running / 已有更新或重载正在执行。',
);
// Removing the observed unique marker is a compare-and-delete. A
// contender cannot unlink the marker belonging to a replacement lock.
// Legacy pid files are supported; new owners never reuse that filename.
try {
await fs.unlink(path.join(lock, heldOwner));
} catch (error) {
if (error.code !== 'ENOENT') throw error;
}
}
return await operation();
} finally {
if (acquired) {
await fs.unlink(path.join(lock, owner)).catch((error) => {
if (error.code !== 'ENOENT') throw error;
});
await fs.rmdir(lock).catch((error) => {
if (!['ENOENT', 'ENOTEMPTY', 'EEXIST'].includes(error.code))
throw error;
});
}
await fs.rm(candidate, { recursive: true, force: true });
}
}
async function stageUpgrade({ root, tmp, env, mirror, run, install, signal }) {
const workspace = await fs.mkdtemp(path.join(tmp, 'upgrade-'));
const branch = releaseBranch(env);
try {
async function extract(repository, ref) {
const archive = path.join(workspace, `${repository}.zip`);
const url =
mirror === 'github'
? `https://github.com/whyour/${repository}/archive/${ref}.zip`
: `https://gitee.com/whyour/${repository}/repository/archive/${ref}.zip`;
await run('curl', [
'--fail',
'--location',
'--proto',
'=https',
'--proto-redir',
'=https',
'--connect-timeout',
'30',
'--max-time',
'600',
'--output',
archive,
url,
]);
const listing = await run('unzip', ['-Z1', archive], true);
const expected = `${repository}-${ref}`;
const names = listing.stdout.split('\n').filter(Boolean);
if (
!names.length ||
names.some(
(name) =>
name.includes('\\') ||
name.includes('\r') ||
name.startsWith('/') ||
name.split('/').includes('..') ||
name.split('/')[0] !== expected,
)
)
throw new Error('Invalid archive paths / 升级归档路径无效。');
const modes = await run('unzip', ['-Z', '-l', archive], true);
if (modes.stdout.split('\n').some((line) => /^l[rwx-]{9}\s/.test(line)))
throw new Error('Archive contains symlinks / 升级归档包含符号链接。');
await run('unzip', ['-oq', archive, '-d', workspace]);
return path.join(workspace, expected);
}
// Static branches are mutable snapshots. Their manifest pins the exact
// source revision, avoiding a race between two independent branch reads.
const staticRoot = await extract('qinglong-static', branch);
const manifest = await manifestAt(staticRoot);
const downloadedSource = await extract('qinglong', manifest.sourceCommit);
const source = path.join(workspace, `qinglong-${branch}`);
await fs.rename(downloadedSource, source);
await verifyUpgrade(source, staticRoot);
let dependenciesChanged = false;
for (const name of ['package.json', 'pnpm-lock.yaml']) {
const current = await fs
.readFile(path.join(root, name))
.catch((error) => {
if (error.code !== 'ENOENT') throw error;
return Buffer.alloc(0);
});
if (!(await fs.readFile(path.join(source, name))).equals(current))
dependenciesChanged = true;
}
if (
!(await fs.stat(path.join(root, 'node_modules')).catch(() => undefined))
)
dependenciesChanged = true;
if (dependenciesChanged) {
await install(source, staticRoot, manifest.sourceCommit);
await fs.access(path.join(source, 'node_modules'));
await verifyUpgrade(source, staticRoot);
}
signal?.throwIfAborted();
const selected = { source, static: staticRoot };
await fs.writeFile(
path.join(workspace, 'ready.json'),
JSON.stringify({
...selected,
sourceCommit: manifest.sourceCommit,
dependenciesChanged,
}),
{ flag: 'wx', mode: 0o600 },
);
const pointer = path.join(workspace, 'pointer.json');
await fs.writeFile(
pointer,
JSON.stringify({ directory: path.basename(workspace) }),
{ flag: 'wx', mode: 0o600 },
);
const readyPointer = path.join(tmp, `upgrade-ready-${branch}.json`);
const previous = await fs
.readFile(readyPointer, 'utf8')
.catch(() => undefined);
signal?.throwIfAborted();
await fs.rename(pointer, readyPointer);
// Retire only our previous successful stage, after publishing its replacement.
try {
const directory = previous && JSON.parse(previous).directory;
if (
typeof directory === 'string' &&
/^upgrade-[A-Za-z0-9_-]+$/.test(directory) &&
directory !== path.basename(workspace)
)
await fs.rm(path.join(tmp, directory), {
recursive: true,
force: true,
});
} catch {
/* Cleanup cannot invalidate the newly published stage. */
}
return selected;
} catch (error) {
await fs.rm(workspace, { recursive: true, force: true });
throw error;
}
}
async function selectedUpgrade(tmp, branch) {
const pointer = JSON.parse(
await fs.readFile(path.join(tmp, `upgrade-ready-${branch}.json`), 'utf8'),
);
if (
typeof pointer.directory !== 'string' ||
!/^upgrade-[A-Za-z0-9_-]+$/.test(pointer.directory)
)
throw new Error('Invalid staged upgrade pointer / 暂存升级指针无效。');
const workspace = path.join(tmp, pointer.directory);
const selected = {
source: path.join(workspace, `qinglong-${branch}`),
static: path.join(workspace, `qinglong-static-${branch}`),
};
for (const directory of [workspace, selected.source, selected.static]) {
const stat = await fs.lstat(directory);
if (!stat.isDirectory() || stat.isSymbolicLink())
throw new Error('Invalid staging directory / 暂存目录无效。');
}
const ready = JSON.parse(
await fs.readFile(path.join(workspace, 'ready.json'), 'utf8'),
);
if (ready.source !== selected.source || ready.static !== selected.static)
throw new Error('Invalid upgrade readiness / 暂存升级就绪记录不匹配。');
const manifest = await verifyUpgrade(selected.source, selected.static);
if (ready.sourceCommit !== manifest.sourceCommit)
throw new Error('Invalid upgrade readiness / 暂存升级就绪记录不匹配。');
if (ready.dependenciesChanged)
await fs.access(path.join(selected.source, 'node_modules'));
return selected;
}
async function replaceAndReload(replacements, lifecycle) {
const changed = [];
let attemptedStart = false;
lifecycle.signal?.throwIfAborted();
try {
await lifecycle.stop();
for (const { source, target } of replacements) {
lifecycle.signal?.throwIfAborted();
await fs.mkdir(path.dirname(target), { recursive: true });
const backup = `${target}.ql-backup-${randomUUID()}`;
const existed = !!(await fs.lstat(target).catch((error) => {
if (error.code !== 'ENOENT') throw error;
return undefined;
}));
if (existed) {
try {
await fs.rename(target, backup);
} catch (error) {
if (error.code !== 'EXDEV') throw error;
try {
await fs.cp(target, backup, {
recursive: true,
verbatimSymlinks: true,
});
} catch (copyError) {
await fs.rm(backup, { recursive: true, force: true });
throw copyError;
}
changed.push({ target, backup, existed });
await fs.rm(target, { recursive: true });
}
}
if (changed.at(-1)?.target !== target)
changed.push({ target, backup, existed });
await fs.cp(source, target, {
recursive: true,
verbatimSymlinks: true,
force: false,
errorOnExist: true,
});
}
lifecycle.signal?.throwIfAborted();
attemptedStart = true;
await lifecycle.start();
lifecycle.signal?.throwIfAborted();
} catch (error) {
// Do not modify files under a still-running failed startup. Keep backups
// for recovery if stopping the new process itself fails.
if (attemptedStart) await lifecycle.stop();
for (const item of changed.reverse()) {
await fs.rm(item.target, { recursive: true, force: true });
if (item.existed) {
try {
await fs.rename(item.backup, item.target);
} catch (restoreError) {
if (restoreError.code !== 'EXDEV') throw restoreError;
await fs.cp(item.backup, item.target, {
recursive: true,
verbatimSymlinks: true,
});
await fs.rm(item.backup, { recursive: true });
}
}
}
await lifecycle.start(true);
throw error;
}
for (const item of changed)
if (item.existed)
await fs
.rm(item.backup, { recursive: true, force: true })
.catch(() =>
console.error(`Retained backup / 保留备份: ${item.backup}`),
);
}
async function reloadSystem({
root,
data,
tmp,
staticRoot,
config,
env,
lifecycle,
}) {
const selected = await selectedUpgrade(tmp, releaseBranch(env));
const reserved = new Set(['data', '.tmp', 'static', '.git', '.env']);
const relative = path.relative(root, data);
if (relative && !relative.startsWith('..') && !path.isAbsolute(relative))
reserved.add(relative.split(path.sep)[0]);
const replacements = (await fs.readdir(selected.source))
.filter((name) => !reserved.has(name))
.map((name) => ({
source: path.join(selected.source, name),
target: path.join(root, name),
}));
replacements.push(
{ source: selected.static, target: staticRoot },
{
source: path.join(selected.source, 'sample/config.sample.sh'),
target: path.join(config, 'config.sample.sh'),
},
);
await replaceAndReload(replacements, lifecycle);
// Cleanup after a committed installation must not report a failed update.
await fs
.rm(path.join(tmp, `upgrade-ready-${releaseBranch(env)}.json`), {
force: true,
})
.catch(() => {});
await fs
.rm(path.dirname(selected.source), { recursive: true, force: true })
.catch(() => {});
}
async function waitForHealth(env, timeout = 60000, signal) {
const port = Number(env.QlPort || 5700);
let base = env.QlBaseUrl || '';
if (base && !base.startsWith('/')) base = `/${base}`;
if (base.endsWith('/')) base = base.slice(0, -1);
const deadline = Date.now() + timeout;
while (Date.now() < deadline) {
signal?.throwIfAborted();
const healthy = await new Promise((resolve) => {
let timer;
const finish = (healthy) => {
clearTimeout(timer);
resolve(healthy);
};
const req = http.get(
{ hostname: '127.0.0.1', port, path: `${base}/api/health`, signal },
(res) => {
let body = '';
res.on('data', (chunk) => {
body += chunk;
if (body.length > 65536) req.destroy();
});
res.on('end', () => {
try {
const result = JSON.parse(body);
finish(
res.statusCode === 200 &&
result.code === 200 &&
result.data?.status === 'ok',
);
} catch {
finish(false);
}
});
res.on('error', () => finish(false));
},
);
// Socket inactivity timeouts do not bound a response that keeps streaming.
timer = setTimeout(
() => req.destroy(new Error('Health probe timed out')),
Math.min(1000, Math.max(1, deadline - Date.now())),
);
req.on('error', () => finish(false));
});
signal?.throwIfAborted();
if (healthy) return;
const remaining = deadline - Date.now();
if (remaining > 0) {
try {
await delay(Math.min(500, remaining), undefined, { signal });
} catch (error) {
signal?.throwIfAborted();
throw error;
}
}
}
throw new Error('Backend failed health check / 后端启动检查失败。');
}
function runProcess(env, program, args, capture = false, signal) {
if (signal?.aborted) return Promise.reject(signal.reason);
return new Promise((resolve, reject) => {
const child = spawn(program, args, {
env,
detached: process.platform !== 'win32',
// Pipes keep close pending until children have released their output.
stdio: ['ignore', 'pipe', 'pipe'],
});
let stdout = '';
let aborted = false;
let failure;
let escalation;
const kill = (name) => {
if (!child.pid) return;
try {
process.kill(
process.platform === 'win32' ? child.pid : -child.pid,
name,
);
} catch (error) {
if (error.code === 'EPERM') {
child.kill(name);
return;
}
if (error.code !== 'ESRCH') throw error;
}
};
const abort = () => {
if (aborted) return;
aborted = true;
kill('SIGTERM');
escalation = setTimeout(() => kill('SIGKILL'), 1000);
escalation.unref();
};
child.stdout.on('data', (chunk) => {
if (capture) stdout += chunk;
else process.stdout.write(chunk);
});
child.stderr.on('data', (chunk) => process.stderr.write(chunk));
child.once('error', (error) => {
failure = error;
});
child.once('exit', () => {
// The shell can exit before a descendant that ignores SIGTERM.
if (aborted) kill('SIGKILL');
});
child.once('close', (code) => {
if (aborted) kill('SIGKILL');
clearTimeout(escalation);
signal?.removeEventListener('abort', abort);
if (aborted) reject(signal.reason);
else if (failure) reject(failure);
else if (code === 0) resolve({ stdout });
else reject(new Error(`${program} failed (exit ${code})`));
});
signal?.addEventListener('abort', abort, { once: true });
if (signal?.aborted) abort();
});
}
async function withLifecycleHooks(root, tmp, env, operation) {
const helpers = await fs.readFile(path.join(root, 'shell/share.sh'), 'utf8');
const environment = await fs.readFile(
path.join(root, 'shell/env.sh'),
'utf8',
);
const directory = await fs.mkdtemp(path.join(tmp, 'upgrade-hooks-'));
try {
// Keep the installed functions available through replacement and rollback.
// Their pkill patterns must never become part of the bash command line.
const snapshot = path.join(directory, 'lifecycle.sh');
await fs.writeFile(
snapshot,
`${helpers}\n${environment}\nimport_config
case "$1" in
stop) delete_pm2 ;;
start) reload_pm2 ;;
install)
exit_status=0
npm_install_2 "$2" "$3" "$4"
hook_install_status=$?
[[ "$hook_install_status" -eq 0 ]] || exit "$hook_install_status"
exit "$exit_status"
;;
*) printf '%s\\n' 'Unknown lifecycle operation' >&2; exit 2 ;;
esac
`,
{
flag: 'wx',
mode: 0o600,
},
);
const hook = (operation, args = [], signal) =>
runProcess(
env,
'bash',
['--', snapshot, operation, ...args],
false,
signal,
);
return await operation(hook);
} finally {
await fs
.rm(directory, { recursive: true, force: true })
.catch(() =>
console.error(
`Retained lifecycle snapshot / 保留生命周期快照: ${directory}`,
),
);
}
}
async function main() {
const env = process.env,
root = env.QL_DIR;
if (!root || !path.isAbsolute(root))
throw new Error('QL_DIR must be absolute.');
const tmp = env.dir_tmp || path.join(root, '.tmp');
const controller = new AbortController();
for (const name of ['SIGINT', 'SIGTERM'])
process.on(name, () =>
controller.abort(new Error(`Upgrade interrupted (${name})`)),
);
const run = (program, args, capture) =>
runProcess(env, program, args, capture, controller.signal);
await withUpgradeLock(tmp, () =>
withLifecycleHooks(root, tmp, env, async (hook) => {
const lifecycle = {
signal: controller.signal,
stop: () => hook('stop'),
start: async (recover) => {
await hook('start', [], recover ? undefined : controller.signal);
await waitForHealth(
env,
60000,
recover ? undefined : controller.signal,
);
},
};
if (process.argv[2] === 'update') {
await stageUpgrade({
root,
tmp,
env,
mirror: process.argv[3],
run,
signal: controller.signal,
install: (source, staticRoot, sourceCommit) =>
hook('install', [source, staticRoot, sourceCommit], controller.signal),
});
console.log('更新包下载及校验成功...');
if (process.argv[4] !== 'true') return;
} else if (process.argv[2] !== 'reload')
throw new Error('Unknown upgrade command.');
await reloadSystem({
root,
tmp,
env,
data: env.dir_data || path.join(root, 'data'),
staticRoot: env.dir_static || path.join(root, 'static'),
config: env.dir_config || path.join(root, 'data/config'),
lifecycle,
});
console.log('更新及启动检查成功...');
}),
);
}
module.exports = {
releaseBranch,
verifyUpgrade,
withUpgradeLock,
stageUpgrade,
selectedUpgrade,
replaceAndReload,
reloadSystem,
waitForHealth,
runProcess,
withLifecycleHooks,
main,
};
if (require.main === module)
main().catch((error) => {
console.error(`更新失败: ${error.message}`);
process.exitCode = 1;
});
+52 -28
View File
@@ -53,23 +53,12 @@ export async function request(
// Check HTTP status before decoding (proxies may return HTML on denial).
if (!response.ok) {
await response.body?.cancel();
const suffix =
method !== 'GET' && response.status >= 500
? translate(
process.env,
' 执行结果可能未知,请先检查%s状态再考虑重试。',
resource,
)
: '';
fail(
translate(
process.env,
'API 请求被拒绝(HTTP %s),请检查%s。%s',
response.status,
credentialsHint,
suffix,
),
[401, 403].includes(response.status) ? 3 : 1,
rejectResponse(
response.status,
'HTTP',
method,
resource,
credentialsHint,
);
}
if (
@@ -100,28 +89,63 @@ export async function request(
),
);
}
if (endpoint.split('?')[0] === 'user/login' && isRecord(result) && result.code === 420)
fail(translate(process.env, '需要双因素验证,请调用 user two-factor-login。'), 3);
if (
endpoint.split('?')[0] === 'user/login' &&
isRecord(result) &&
result.code === 420
)
fail(
translate(process.env, '需要双因素验证,请调用 user two-factor-login。'),
3,
);
if (!isRecord(result) || result.code !== 200) {
const status =
isRecord(result) && typeof result.code === 'number'
? result.code
: undefined;
fail(
translate(
process.env,
'API 请求被拒绝(code %s),请检查%s。',
status ?? translate(process.env, '未知'),
credentialsHint,
),
status === 401 || status === 403 ? 3 : 1,
);
rejectResponse(status, 'code', method, resource, credentialsHint);
}
if (options.output)
fail(translate(process.env, '接口返回 JSON,未保存下载文件。'));
return result;
}
function rejectResponse(
status: number | undefined,
kind: 'HTTP' | 'code',
method: string,
resource: string,
credentialsHint: string,
): never {
const denied = status === 401 || status === 403;
const reason = denied
? translate(process.env, '请检查%s。', credentialsHint)
: status === 409
? translate(process.env, '资源冲突,请检查重复的名称或值。')
: status !== undefined && status >= 500
? translate(process.env, '服务端错误,请检查服务状态和日志。')
: translate(process.env, '请求未成功,请检查请求参数和服务日志。');
const suffix =
method !== 'GET' && (status === undefined || status >= 500)
? translate(
process.env,
' 执行结果可能未知,请先检查%s状态再考虑重试。',
resource,
)
: '';
fail(
translate(
process.env,
'API 请求失败(%s %s)。%s%s',
kind,
status ?? translate(process.env, '未知'),
reason,
suffix,
),
denied ? 3 : 1,
);
}
export async function authenticate(config: Credentials): Promise<Session> {
const query = new URLSearchParams({
client_id: config.clientId,
+12
View File
@@ -275,6 +275,18 @@ export const openOperations: readonly OpenOperation[] = [
{ name: 'config save', method: 'POST', path: 'configs/save', body: 'json' },
{ name: 'script list', method: 'GET', path: 'scripts' },
{ name: 'script get', method: 'GET', path: 'scripts/detail' },
{ name: 'script history-list', method: 'GET', path: 'scripts/history' },
{
name: 'script history-detail',
method: 'GET',
path: 'scripts/history/detail',
},
{
name: 'script history-restore',
method: 'PUT',
path: 'scripts/history/restore',
body: 'json',
},
{
name: 'script create',
method: 'POST',
+1 -1
View File
@@ -241,7 +241,7 @@ export async function openCommand(
body,
authenticated: !op.anonymous,
output: values.output as string | undefined,
text: op.path === 'system/log',
text: ['system/log', 'system/config/node-mirror'].includes(op.path),
timeoutMs: values.timeout ? Number(values.timeout) * 1000 : undefined,
});
const data =
+6
View File
@@ -151,6 +151,12 @@ export const english: Record<string, string> = {
应用凭据: 'application credentials',
'应用凭据和 %s 权限': 'application credentials and %s permission',
未知: 'unknown',
'请检查%s。': 'Check %s.',
'资源冲突,请检查重复的名称或值。': 'Resource conflict; check for duplicate names or values.',
'服务端错误,请检查服务状态和日志。': 'Server error; check service status and logs.',
'请求未成功,请检查请求参数和服务日志。': 'Request unsuccessful; check request parameters and service logs.',
'API 请求失败(%s %s)。%s%s': 'API request failed (%s %s). %s%s',
' 执行结果可能未知,请先检查%s状态再考虑重试。':
' Execution outcome may be unknown; check %s status before retrying.',
'API 请求被拒绝(HTTP %s),请检查%s。%s':
+9 -8
View File
@@ -7,7 +7,7 @@ const { execFileSync, spawnSync } = require('node:child_process');
const { createContext, sourceEnvironment } = require('../../dist/internal/runtime/context');
const { executeTask } = require('../../dist/internal/execution/taskRunner');
test('unmodified legacy Shell and TS agree on shell hook state and account modes', async (t) => {
test('Shell and CLI agree on isolated script state and account modes', async (t) => {
const root = await fs.realpath(
await fs.mkdtemp(path.join(os.tmpdir(), 'ql-differential-')),
);
@@ -31,7 +31,7 @@ test('unmodified legacy Shell and TS agree on shell hook state and account modes
path.join(root, 'static/build'),
])
await fs.mkdir(directory, { recursive: true });
for (const name of ['task.sh', 'otask.sh', 'share.sh', 'api.sh', 'env.sh'])
for (const name of ['task.sh', 'task-timeout.sh', 'otask.sh', 'share.sh', 'api.sh', 'env.sh'])
await fs.copyFile(
path.resolve(__dirname, '../../../shell', name),
path.join(context.paths.dir_shell, name),
@@ -110,13 +110,13 @@ test('unmodified legacy Shell and TS agree on shell hook state and account modes
async () => {
const cases = [
{ name: 'return', script: 'return 7', code: 7, legacy: 0, after: true },
{ name: 'exit', script: 'exit 7', code: 7, legacy: 7, after: false },
{ name: 'exit', script: 'exit 7', code: 7, legacy: 0, after: true },
{
name: 'errexit',
script: 'set -e; false',
code: 1,
legacy: 1,
after: false,
legacy: 0,
after: true,
},
{
name: 'trap-return',
@@ -130,8 +130,8 @@ test('unmodified legacy Shell and TS agree on shell hook state and account modes
name: 'trap-exit',
script: 'trap \'printf "exit-trap\\n" >> "$TRACE"\' EXIT; exit 7',
code: 7,
legacy: 7,
after: false,
legacy: 0,
after: true,
trap: true,
},
];
@@ -174,8 +174,8 @@ test('unmodified legacy Shell and TS agree on shell hook state and account modes
assert.equal(result.exitCode, item.code, item.name);
const expected = [
'before',
...(item.after ? [`after:${item.code}`] : []),
...(item.trap ? ['exit-trap'] : []),
...(item.after ? [`after:${item.code}`] : []),
];
for (const file of [legacyTrace, modernTrace])
assert.deepEqual(
@@ -355,6 +355,7 @@ t() { :; }
sleep() { printf DELAYED; }
enter_script_workdir() { :; }
run_else() { :; }
run_task_command() { "$@"; }
which_program=true
main "$@"
`,
@@ -10,7 +10,7 @@ test('legacy Shell and TS resolve executable paths after selecting the working d
t.after(() => fs.rmSync(root, { recursive: true, force: true }));
for (const dir of ['shell/preload', 'shell/lang', 'data/config', 'data/scripts/bin', 'data/scripts/custom', 'data/log', 'static/build', 'bin'])
fs.mkdirSync(path.join(root, dir), { recursive: true });
for (const name of ['task.sh', 'otask.sh', 'share.sh', 'api.sh', 'env.sh'])
for (const name of ['task.sh', 'task-timeout.sh', 'otask.sh', 'share.sh', 'api.sh', 'env.sh'])
fs.copyFileSync(path.resolve(__dirname, '../../../shell', name), path.join(root, 'shell', name));
for (const name of ['zh.sh', 'en.sh'])
fs.copyFileSync(path.resolve(__dirname, '../../../shell/lang', name), path.join(root, 'shell/lang', name));
+1 -1
View File
@@ -10,7 +10,7 @@ test('legacy Shell and TS preserve task stdin in each execution mode', t => {
t.after(() => fs.rmSync(root, { recursive: true, force: true }));
for (const dir of ['shell/preload', 'shell/lang', 'data/config', 'data/scripts', 'data/log', 'static/build', 'bin'])
fs.mkdirSync(path.join(root, dir), { recursive: true });
for (const name of ['task.sh', 'otask.sh', 'share.sh', 'api.sh', 'env.sh'])
for (const name of ['task.sh', 'task-timeout.sh', 'otask.sh', 'share.sh', 'api.sh', 'env.sh'])
fs.copyFileSync(path.resolve(__dirname, '../../../shell', name), path.join(root, 'shell', name));
for (const name of ['zh.sh', 'en.sh'])
fs.copyFileSync(path.resolve(__dirname, '../../../shell/lang', name), path.join(root, 'shell/lang', name));
+76 -6
View File
@@ -46,6 +46,7 @@ async function fixture(t) {
const fs=require('node:fs'),p=require('node:path');
const program=p.basename(process.argv[1]),args=process.argv.slice(2);
fs.appendFileSync(process.env.CALLS,JSON.stringify({program,args,python:process.env.PYTHON_HOME,node:process.env.PNPM_HOME})+'\\n');
if(program==='npm' && args[0]==='prefix') process.stdout.write(p.join(p.dirname(process.env.CALLS),'global-prefix')+'\\n');
if(program==='python3') process.stdout.write('3.12\\n');
if(program==='nginx' && args.includes('-s')) process.exit(1);
`;
@@ -64,6 +65,11 @@ async function fixture(t) {
`#!${process.execPath}\n${program}`,
{ mode: 0o755 },
);
await fs.mkdir(path.join(root, 'scripts'));
await fs.writeFile(
path.join(root, 'scripts/install-runtime-tools.cjs'),
program,
);
const hooks = [];
const registrations = [];
const system = {
@@ -82,7 +88,7 @@ async function fixture(t) {
}
test('bootstrap installs prerequisites, uses the configured data directory and starts services in order', async (t) => {
const { ctx, system, hooks, registrations } = await fixture(t);
const { ctx, system, hooks, base, registrations } = await fixture(t);
await fs.writeFile(path.join(ctx.root, '.env'), 'KEEP=true');
const result = await bootstrapPanel(ctx, false, system);
assert.equal(result.mode, 'install');
@@ -107,6 +113,7 @@ test('bootstrap installs prerequisites, uses the configured data directory and s
process.getuid() === 0 ? 'apt-get' : 'sudo',
process.getuid() === 0 ? 'apt-get' : 'sudo',
'npm',
'install-runtime-tools.cjs',
'python3',
'pip3',
'pm2',
@@ -118,14 +125,23 @@ test('bootstrap installs prerequisites, uses the configured data directory and s
'pm2',
],
);
assert.deepEqual(calls[4].args, [
assert.deepEqual(calls[2].args, ['prefix', '--global']);
assert.deepEqual(calls[3].args, [
'--archive',
path.join(ctx.root, 'static/runtime-tools.tgz'),
'--proof',
path.join(ctx.root, 'static/runtime-tools-proof.json'),
'--prefix',
path.join(base, 'global-prefix'),
]);
assert.deepEqual(calls[5].args, [
'install',
'--prefix',
path.join(ctx.data, 'dep_cache/python3'),
'requests',
]);
assert.deepEqual(calls[6].args, ['-c', system.nginxConfig, '-s', 'reload']);
assert.deepEqual(calls[7].args, ['-c', system.nginxConfig]);
assert.deepEqual(calls[7].args, ['-c', system.nginxConfig, '-s', 'reload']);
assert.deepEqual(calls[8].args, ['-c', system.nginxConfig]);
assert.deepEqual(
calls.slice(-4).map((c) => c.args),
[
@@ -137,7 +153,7 @@ test('bootstrap installs prerequisites, uses the configured data directory and s
);
assert.ok(
calls
.slice(4)
.slice(5)
.every(
(c) =>
c.python === path.join(ctx.data, 'dep_cache/python3') &&
@@ -240,7 +256,9 @@ for (const language of ['zh', 'en', 'unsupported']) {
return true;
},
);
const { runtimeEnvironment } = require('../../dist/internal/maintenance/bootstrap');
const {
runtimeEnvironment,
} = require('../../dist/internal/maintenance/bootstrap');
assert.throws(
() => runtimeEnvironment(ctx, 'bad'),
(error) => {
@@ -255,3 +273,55 @@ for (const language of ['zh', 'en', 'unsupported']) {
assert.deepEqual(await fs.readdir(root), []);
});
}
test('failed reviewed runtime installation stops before service registration and startup', async (t) => {
const { ctx, system, hooks, registrations } = await fixture(t);
await fs.writeFile(path.join(ctx.root, '.env'), 'KEEP=true');
await fs.writeFile(
path.join(ctx.root, 'scripts/install-runtime-tools.cjs'),
'process.exit(19);',
);
await assert.rejects(bootstrapPanel(ctx, false, system), /exit 19|退出码 19/);
assert.deepEqual(registrations, []);
assert.deepEqual(hooks, []);
assert.equal(
await fs.readFile(path.join(ctx.root, '.env'), 'utf8'),
'KEEP=true',
);
const calls = (await fs.readFile(ctx.env.CALLS, 'utf8'))
.trim()
.split('\n')
.map(JSON.parse);
assert.ok(
calls.every(
(call) => !['python3', 'pip3', 'nginx', 'pm2'].includes(call.program),
),
);
});
test('runtime environment preserves the controlled tools prefix and persistent global dependencies', async (t) => {
const { ctx, base } = await fixture(t);
const {
runtimeEnvironment,
} = require('../../dist/internal/maintenance/bootstrap');
const toolsModules = path.join(base, 'global-prefix/lib/node_modules');
const customModules = path.join(base, 'operator-modules');
const env = runtimeEnvironment(
{
...ctx,
env: {
...ctx.env,
NODE_PATH: [toolsModules, customModules].join(path.delimiter),
},
},
'3.12',
);
const modules = env.NODE_PATH.split(path.delimiter);
assert.deepEqual(modules.slice(0, 2), [toolsModules, customModules]);
assert.ok(modules.includes('/usr/local/lib/node_modules'));
assert.ok(
modules.includes(
path.join(ctx.data, 'dep_cache/node/global/5/node_modules'),
),
);
});
+4 -4
View File
@@ -445,7 +445,7 @@ test('hook exports reach scripts and after hooks share shell state with inline c
assert.match(log, /hook:before:inline:after:7/);
});
test('ordinary shell tasks share unexported variables and functions across before script and after hooks', async (t) => {
test('shell tasks inherit before-hook state without leaking script state into after hooks', async (t) => {
const { executeTask } = require('../../dist/internal/execution/taskRunner');
const root = sandbox(t);
const context = createContext(
@@ -464,7 +464,7 @@ test('ordinary shell tasks share unexported variables and functions across befor
);
fs.writeFileSync(
context.paths.file_task_after,
'script_function\nprintf "after:%s:%s:%s\\n" "$before_value" "$script_value" "$_task_exit_code"\n',
'if declare -F script_function >/dev/null; then exit 99; fi\nprintf "after:%s:%s:%s\\n" "$before_value" "${script_value-unset}" "$_task_exit_code"\n',
);
const result = await executeTask(context, {
argv: ['shared.sh'],
@@ -478,7 +478,7 @@ test('ordinary shell tasks share unexported variables and functions across befor
);
assert.match(
log,
/before:private\narg:space value\nscript:changed\nafter:private:changed:7/,
/before:private\narg:space value\nafter:private:unset:7/,
);
});
@@ -681,7 +681,7 @@ test('designated shell accounts retain hook functions and script state in a shar
path.join(context.paths.dir_log, result.logPath),
'utf8',
);
assert.match(log, /selected:third&first\nafter:third&first:from_script:4/);
assert.match(log, /selected:third&first\nafter:third&first::4/);
});
test('lifecycle reports share execution identity and statistics survive a rejected final status', async (t) => {
+278 -46
View File
@@ -15,8 +15,17 @@ const {
reloadPanel,
} = require('../../dist/internal/maintenance/upgrade');
const { parse } = require('../helpers/commands.cjs');
const {
writeManifest,
sourceCommit,
} = require('../../../test/helpers/upgrade-fixture.cjs');
async function fixture(t) {
t.mock.method(
require('../../dist/internal/maintenance/upgradeArtifacts.cjs'),
'waitForHealth',
async () => {},
);
const root = await fs.realpath(
await fs.mkdtemp(path.join(os.tmpdir(), 'ql-operator-')),
);
@@ -50,6 +59,24 @@ async function stub(context, program, body) {
context.env.PATH = `${bin}:/usr/bin:/bin`;
}
async function stubRuntimeTools(context, record) {
const prefix = path.join(context.root, 'global-prefix');
await stub(
context,
'npm',
`${record}
if(process.argv[2]==='prefix') process.stdout.write(${JSON.stringify(
prefix,
)}+'\\n');`,
);
await fs.mkdir(path.join(context.root, 'scripts'), { recursive: true });
await fs.writeFile(
path.join(context.root, 'scripts/install-runtime-tools.cjs'),
record,
);
return prefix;
}
test('configuration repair preserves custom content and intentionally empty hooks', async (t) => {
const ctx = await fixture(t);
await fs.mkdir(ctx.paths.dir_config, { recursive: true });
@@ -66,29 +93,51 @@ test('configuration repair preserves custom content and intentionally empty hook
assert.deepEqual(await repairConfiguration(ctx), [ctx.paths.file_notify_py]);
});
test('dependency installation selects pnpm or Termux npm argv without shell interpolation', async (t) => {
async function stubPanelDependencies(context, body) {
await fs.mkdir(path.join(context.root, 'scripts'), { recursive: true });
await fs.writeFile(
path.join(context.root, 'scripts/install-panel-dependencies.cjs'),
body,
);
}
test('dependency installation uses the installed helper and preserves root, static and commit arguments', async (t) => {
const ctx = await fixture(t);
const output = path.join(ctx.root, 'calls.jsonl');
ctx.env.CALLS = output;
ctx.env.CALLS = path.join(ctx.root, 'dependencies.jsonl');
const record =
'require("node:fs").appendFileSync(process.env.CALLS, JSON.stringify({args:process.argv.slice(2),cwd:process.cwd()})+"\\n");';
await stub(ctx, 'pnpm', record);
await stub(ctx, 'npm', record);
'require("node:fs").appendFileSync(process.env.CALLS,JSON.stringify({args:process.argv.slice(2),cwd:process.cwd(),termux:process.env.is_termux||""})+"\\n");';
await stubPanelDependencies(ctx, record);
await installPanelDependencies(ctx);
ctx.env.is_termux = '1';
await installPanelDependencies(ctx);
const calls = (await fs.readFile(output, 'utf8'))
const source = path.join(ctx.root, 'source ;$(touch SHOULD_NOT_RUN)');
const staticRoot = path.join(ctx.root, 'static ;$(touch SHOULD_NOT_RUN)');
await fs.mkdir(source);
await installPanelDependencies(ctx, source, staticRoot, sourceCommit);
const calls = (await fs.readFile(ctx.env.CALLS, 'utf8'))
.trim()
.split('\n')
.map(JSON.parse);
assert.deepEqual(
calls.map((c) => c.args),
[
['install', '--loglevel', 'error', '--production'],
['install', '--production', '--no-bin-links'],
],
);
assert.ok(calls.every((c) => c.cwd === ctx.root));
assert.deepEqual(calls, [
{ args: ['--root', ctx.root], cwd: ctx.root, termux: '0' },
{ args: ['--root', ctx.root], cwd: ctx.root, termux: '1' },
{
args: [
'--root',
source,
'--static-dir',
staticRoot,
'--source-commit',
sourceCommit,
],
cwd: source,
termux: '1',
},
]);
for (const directory of [ctx.root, source])
await assert.rejects(fs.stat(path.join(directory, 'SHOULD_NOT_RUN')), {
code: 'ENOENT',
});
});
test('PM2 reload uses installation cwd and exact process-manager arguments', async (t) => {
@@ -119,14 +168,27 @@ test('PM2 restarts retain configured ports without mutating legacy environment',
ctx.env.CALLS = path.join(ctx.root, 'ports.jsonl');
ctx.env.BACK_PORT = '5600';
ctx.env.GRPC_PORT = '5400';
await stub(ctx, 'pm2', 'require("node:fs").appendFileSync(process.env.CALLS, JSON.stringify([process.env.BACK_PORT,process.env.GRPC_PORT])+"\\n");');
await stub(
ctx,
'pm2',
'require("node:fs").appendFileSync(process.env.CALLS, JSON.stringify([process.env.BACK_PORT,process.env.GRPC_PORT])+"\\n");',
);
await startPanel(ctx);
ctx.env.QlPort = '5799';
ctx.env.QlGrpcPort = '5599';
await startPanel(ctx);
assert.deepEqual((await fs.readFile(ctx.env.CALLS, 'utf8')).trim().split('\n').map(JSON.parse), [
['5700', '5500'], ['5700', '5500'], ['5799', '5599'], ['5799', '5599'],
]);
assert.deepEqual(
(await fs.readFile(ctx.env.CALLS, 'utf8'))
.trim()
.split('\n')
.map(JSON.parse),
[
['5700', '5500'],
['5700', '5500'],
['5799', '5599'],
['5799', '5599'],
],
);
assert.equal(ctx.env.BACK_PORT, '5600');
assert.equal(ctx.env.GRPC_PORT, '5400');
});
@@ -199,24 +261,41 @@ test('upgrade staging extracts both archives before publishing readiness and rej
const payloads = path.join(ctx.root, 'payloads');
const archives = path.join(ctx.root, 'archives');
await fs.mkdir(archives);
await fs.mkdir(path.join(payloads, 'qinglong-master'), { recursive: true });
await fs.mkdir(path.join(payloads, `qinglong-${sourceCommit}`), {
recursive: true,
});
await fs.mkdir(path.join(payloads, 'qinglong-static-master/build'), {
recursive: true,
});
const manifest = '{"name":"fixture"}';
await fs.writeFile(path.join(ctx.root, 'package.json'), manifest);
await fs.writeFile(
path.join(payloads, 'qinglong-master/package.json'),
path.join(payloads, `qinglong-${sourceCommit}/package.json`),
manifest,
);
await fs.writeFile(
path.join(payloads, 'qinglong-static-master/build/app.js'),
'console.log("fixture")',
);
await fs.mkdir(path.join(payloads, `qinglong-${sourceCommit}/sample`));
await fs.writeFile(
path.join(payloads, `qinglong-${sourceCommit}/sample/config.sample.sh`),
'new sample',
);
writeManifest(
path.join(payloads, `qinglong-${sourceCommit}`),
path.join(payloads, 'qinglong-static-master'),
);
await fs.writeFile(path.join(ctx.root, 'pnpm-lock.yaml'), 'fixture lock\n');
await fs.mkdir(path.join(ctx.root, 'node_modules'));
for (const repo of ['qinglong', 'qinglong-static'])
execFileSync(
'/usr/bin/zip',
['-qry', path.join(archives, `${repo}.zip`), `${repo}-master`],
[
'-qry',
path.join(archives, `${repo}.zip`),
`${repo}-${repo === 'qinglong' ? sourceCommit : 'master'}`,
],
{ cwd: payloads },
);
ctx.env.ARCHIVES = archives;
@@ -225,18 +304,6 @@ test('upgrade staging extracts both archives before publishing readiness and rej
'curl',
'const fs=require("node:fs"),p=require("node:path"),args=process.argv.slice(2),out=args[args.indexOf("--output")+1]; fs.copyFileSync(p.join(process.env.ARCHIVES,p.basename(out)),out);',
);
await fs.mkdir(path.join(payloads, 'qinglong-master/sample'), {
recursive: true,
});
await fs.writeFile(
path.join(payloads, 'qinglong-master/sample/config.sample.sh'),
'new sample',
);
execFileSync(
'/usr/bin/zip',
['-qry', path.join(archives, 'qinglong.zip'), 'qinglong-master'],
{ cwd: payloads },
);
const staged = await stageUpgrade(ctx, 'github');
assert.equal(
await fs.readFile(path.join(staged.source, 'package.json'), 'utf8'),
@@ -249,22 +316,65 @@ test('upgrade staging extracts both archives before publishing readiness and rej
'utf8',
),
),
staged,
{ ...staged, sourceCommit, dependenciesChanged: false },
);
await fs.writeFile(
path.join(ctx.root, 'package.json'),
'{"name":"older-installed"}',
);
ctx.env.CALLS = path.join(ctx.root, 'upgrade-dependencies.jsonl');
await stubPanelDependencies(ctx, 'process.exit(23);');
await assert.rejects(stageUpgrade(ctx, 'github'), /exit 23/);
const ready = path.join(ctx.paths.dir_tmp, 'upgrade-ready-master.json');
assert.deepEqual(JSON.parse(await fs.readFile(ready, 'utf8')), {
directory: path.basename(path.dirname(staged.source)),
});
assert.equal(
await fs.readFile(path.join(ctx.root, 'package.json'), 'utf8'),
'{"name":"older-installed"}',
);
await stubPanelDependencies(
ctx,
'const fs=require("node:fs"),p=require("node:path"),args=process.argv.slice(2);fs.appendFileSync(process.env.CALLS,JSON.stringify({args,cwd:process.cwd()})+"\\n");fs.mkdirSync(p.join(args[args.indexOf("--root")+1],"node_modules"),{recursive:true});',
);
const changedStage = await stageUpgrade(ctx, 'github');
assert.deepEqual(
JSON.parse((await fs.readFile(ctx.env.CALLS, 'utf8')).trim()),
{
args: [
'--root',
changedStage.source,
'--static-dir',
changedStage.static,
'--source-commit',
sourceCommit,
],
cwd: changedStage.source,
},
);
assert.equal(
JSON.parse(
await fs.readFile(
path.join(path.dirname(changedStage.source), 'ready.json'),
'utf8',
),
).dependenciesChanged,
true,
);
await fs.symlink(
'../../outside',
path.join(payloads, 'qinglong-master/escape'),
path.join(payloads, `qinglong-${sourceCommit}/escape`),
);
execFileSync(
'/usr/bin/zip',
['-qry', path.join(archives, 'qinglong.zip'), 'qinglong-master'],
['-qry', path.join(archives, 'qinglong.zip'), `qinglong-${sourceCommit}`],
{ cwd: payloads },
);
await assert.rejects(stageUpgrade(ctx, 'github'), /symlinks/);
assert.deepEqual(
(await fs.readdir(ctx.paths.dir_tmp)).sort(),
[
path.basename(path.dirname(staged.source)),
path.basename(path.dirname(changedStage.source)),
'upgrade-ready-master.json',
].sort(),
);
@@ -309,8 +419,9 @@ test('check repairs dependencies and notifications, probes loopback and reloads
ctx.env.HTTP_PROXY = 'http://127.0.0.1:1';
const record =
'require("node:fs").appendFileSync(process.env.CALLS,JSON.stringify({program:require("node:path").basename(process.argv[1]),args:process.argv.slice(2)})+"\\n");';
for (const executable of ['npm', 'pnpm', 'pm2'])
await stub(ctx, executable, record);
for (const executable of ['pnpm', 'pm2']) await stub(ctx, executable, record);
const toolsPrefix = await stubRuntimeTools(ctx, record);
await stubPanelDependencies(ctx, record);
await repairConfiguration(ctx);
await fs.writeFile(ctx.paths.file_notify_py, 'custom old notify');
await fs.mkdir(path.join(ctx.env.PM2_HOME, 'logs'), { recursive: true });
@@ -337,11 +448,22 @@ test('check repairs dependencies and notifications, probes loopback and reloads
assert.deepEqual(calls, [
{
program: 'npm',
args: ['i', '-g', 'pnpm@8.3.1', 'pm2', 'ts-node', 'typescript@5'],
args: ['prefix', '--global'],
},
{
program: 'pnpm',
args: ['install', '--loglevel', 'error', '--production'],
program: 'install-runtime-tools.cjs',
args: [
'--archive',
path.join(ctx.root, 'static/runtime-tools.tgz'),
'--proof',
path.join(ctx.root, 'static/runtime-tools-proof.json'),
'--prefix',
toolsPrefix,
],
},
{
program: 'install-panel-dependencies.cjs',
args: ['--root', ctx.root],
},
{ program: 'pm2', args: ['flush'] },
{
@@ -410,6 +532,28 @@ test('repair stops after installer failure before replacing user files or reload
});
});
test('dependency helper failure preserves notifications and prevents a service reload', async (t) => {
const { checkAndRepair } = require('../../dist/internal/maintenance/check');
const ctx = await fixture(t);
await repairConfiguration(ctx);
await fs.writeFile(ctx.paths.file_notify_js, 'keep installed notification');
await stubRuntimeTools(ctx, '');
await stubPanelDependencies(ctx, 'process.exit(19);');
await stub(
ctx,
'pm2',
'require("node:fs").writeFileSync(process.env.QL_DIR+"/unexpected-reload", "bad");',
);
await assert.rejects(checkAndRepair(ctx), /exit 19/);
assert.equal(
await fs.readFile(ctx.paths.file_notify_js, 'utf8'),
'keep installed notification',
);
await assert.rejects(fs.stat(path.join(ctx.root, 'unexpected-reload')), {
code: 'ENOENT',
});
});
test('legacy check workflow and TypeScript repair agree on dependency and notification operations', async (t) => {
const { execFileSync } = require('node:child_process');
const ctx = await fixture(t);
@@ -417,7 +561,8 @@ test('legacy check workflow and TypeScript repair agree on dependency and notifi
const trace = path.join(ctx.root, 'legacy-trace');
const shell = `
t() { :; }
npm() { printf 'npm %s\\n' "$*" >> "$TRACE"; }
npm() { printf 'npm %s\\n' "$*" >> "$TRACE"; printf '%s\\n' "$dir_root/global-prefix"; }
node() { printf 'tools %s\\n' "$*" >> "$TRACE"; }
fix_config() { printf 'repair-config\\n' >> "$TRACE"; }
npm_install_2() { printf 'dependencies %s\\n' "$1" >> "$TRACE"; }
cp() { printf 'copy %s\\n' "$*" >> "$TRACE"; command cp "$@"; }
@@ -441,7 +586,8 @@ test('legacy check workflow and TypeScript repair agree on dependency and notifi
});
const operations = (await fs.readFile(trace, 'utf8')).trim().split('\n');
assert.deepEqual(operations, [
'npm i -g pnpm@8.3.1 pm2 ts-node typescript@5',
'npm prefix --global',
`tools ${ctx.root}/scripts/install-runtime-tools.cjs --archive ${ctx.root}/static/runtime-tools.tgz --proof ${ctx.root}/static/runtime-tools-proof.json --prefix ${ctx.root}/global-prefix`,
'repair-config',
`dependencies ${ctx.root}`,
`copy -fv ${ctx.paths.file_notify_py_sample} ${ctx.paths.file_notify_py}`,
@@ -458,6 +604,91 @@ test('legacy check workflow and TypeScript repair agree on dependency and notifi
);
});
test('legacy check aborts dependency failure before copying notifications or reloading', async (t) => {
const { execFileSync } = require('node:child_process');
const ctx = await fixture(t);
await repairConfiguration(ctx);
await fs.writeFile(ctx.paths.file_notify_js, 'keep legacy notification');
const shell = `
t() { :; }
npm() { printf '%s\\n' "$dir_root/global-prefix"; }
node() { :; }
fix_config() { :; }
npm_install_2() { return 19; }
reload_pm2() { printf unexpected > "$dir_root/unexpected-reload"; }
curl() { printf '<div id="root"></div>{"code":200,"status":"ok"}'; }
. "$CHECK_SOURCE"
`;
assert.throws(
() =>
execFileSync('/bin/bash', ['--noprofile', '--norc', '-c', shell], {
env: {
...ctx.env,
CHECK_SOURCE: path.resolve(__dirname, '../../../shell/check.sh'),
},
stdio: ['ignore', 'pipe', 'pipe'],
}),
(error) => error.status === 19,
);
assert.equal(
await fs.readFile(ctx.paths.file_notify_js, 'utf8'),
'keep legacy notification',
);
await assert.rejects(fs.stat(path.join(ctx.root, 'unexpected-reload')), {
code: 'ENOENT',
});
});
test('installed upgrade hooks propagate returned failures and preserve legacy installer status', async (t) => {
const {
withLifecycleHooks,
} = require('../../dist/internal/maintenance/upgradeArtifacts.cjs');
for (const [returned, legacy, expected] of [
[19, 0, 19],
[0, 23, 23],
[0, 0, 0],
]) {
const ctx = await fixture(t);
const tmp = path.join(ctx.root, 'tmp');
await fs.mkdir(tmp);
await fs.mkdir(path.join(ctx.root, 'shell'));
const trace = path.join(ctx.root, 'install-args');
await fs.writeFile(path.join(ctx.root, 'shell/env.sh'), '');
await fs.writeFile(
path.join(ctx.root, 'shell/share.sh'),
`import_config() { :; }
npm_install_2() {
printf '%s\\n' "$@" > "$QL_INSTALL_TRACE"
exit_status=${legacy}
return ${returned}
}
`,
);
const args = [
path.join(ctx.root, 'staged source;$(touch SHOULD_NOT_RUN)'),
path.join(ctx.root, 'staged static;$(touch SHOULD_NOT_RUN)'),
sourceCommit,
];
const operation = withLifecycleHooks(
ctx.root,
tmp,
{ ...ctx.env, QL_INSTALL_TRACE: trace },
(hook) => hook('install', args),
);
if (expected)
await assert.rejects(operation, new RegExp(`exit ${expected}`));
else await operation;
assert.deepEqual(
(await fs.readFile(trace, 'utf8')).trim().split('\n'),
args,
);
await assert.rejects(fs.stat(path.join(ctx.root, 'SHOULD_NOT_RUN')), {
code: 'ENOENT',
});
assert.deepEqual(await fs.readdir(tmp), []);
}
});
test('interrupted reload restores old files and recovery subprocesses ignore the cancelled operation', async (t) => {
const {
cancellableOperation,
@@ -728,7 +959,7 @@ test('invalid or incomplete staged pointers cannot stop the installed panel', as
reloadPanel(ctx, 'system'),
(error) =>
error.code === 'ENOENT' &&
error.path === path.join(ctx.paths.dir_tmp, 'qinglong-master'),
error.path === path.join(ctx.paths.dir_tmp, 'upgrade-ready-master.json'),
);
});
@@ -783,6 +1014,7 @@ test('system upgrade preserves installed dotenv bytes and permissions even when
);
await fs.writeFile(path.join(staticRoot, 'build/app.js'), '');
await fs.writeFile(path.join(source, '.env'), 'PORT=9999\nREPLACE_ME=true\n');
writeManifest(source, staticRoot);
const target = path.join(ctx.root, '.env');
const original = 'PORT=5700\nPRIVATE_VALUE=preserved\n';
await fs.writeFile(target, original, { mode: 0o600 });
@@ -409,7 +409,7 @@ test('cron metadata precedence agrees with unchanged legacy add_cron', (t) => {
'--noprofile',
'--norc',
'-c',
`${definition}\nt() { :; }\nnotify_api() { :; }\nadd_cron_api() { printf '%s' "$1"; }\nadd_cron "$dir_scripts/list" repo`,
`${definition}\nt() { :; }\nnotify_api() { :; }\nadd_cron_api() { printf '%s:%s:%s:%s' "$1" "$2" "$3" "$4"; }\nadd_cron "$dir_scripts/list" repo`,
],
{
encoding: 'utf8',
+3 -2
View File
@@ -27,13 +27,14 @@ test('shared Shell lifecycle scopes timeouts to the task and runs cleanup exactl
{ name: 'failure', task: 'return 7', code: 7 },
{ name: 'function timeout preserves state', task: 'STATE=changed; sleep 10', code: 124, timeout: true, state: 'changed' },
{ name: 'external timeout', args: ['sleep', '10'], code: 124, timeout: true },
{ name: 'Shell script timeout preserves state', task: 'STATE=changed; sleep 10', shell: true, code: 124, timeout: true, state: 'changed' },
{ name: 'Shell script timeout preserves parent hook state', task: 'STATE=changed; sleep 10', shell: true, code: 124, timeout: true },
{ name: 'slow before is outside task timeout', before: 'sleep 0.2', task: 'return 0', code: 0 },
{ name: 'slow after is outside task timeout', after: 'sleep 0.2', task: 'return 0', code: 0 },
{ name: 'background child cannot hold pipes after timeout cleanup', task: '(trap "" INT; sleep 30) & sleep 30', after: 'sleep 0.3', code: 124, timeout: true },
{ name: 'script may redirect fd 3', shell: true, task: 'exec 3> "$TRACE.fd"; return 0', after: 'sleep 0.3', code: 0 },
{ name: 'before may close fd 3 and redirect fd 9', before: 'exec 3>&- 9> "$TRACE.lock"', task: 'return 0', after: 'sleep 0.3', code: 0 },
{ name: 'timeout cleanup survives script fd 3 redirection', shell: true, task: 'exec 3> "$TRACE.fd"; STATE=changed; sleep 30', after: 'sleep 0.3', code: 124, timeout: true, state: 'changed' },
{ name: 'timeout cleanup survives script fd 3 redirection', shell: true, task: 'exec 3> "$TRACE.fd"; STATE=changed; sleep 30', after: 'sleep 0.3', code: 124, timeout: true },
{ name: 'Shell exit preserves user trap and runs after once', shell: true, task: 'trap \'printf "exit\\n" >> "$TRACE"\' EXIT; exit 9', code: 9, userExit: true },
{ name: 'explicit exit preserves user trap and skips after', task: 'trap \'printf "exit\\n" >> "$TRACE"\' EXIT; exit 9', code: 9, noAfter: true, userExit: true },
{ name: 'user EXIT trap survives timeout with parent cleanup', task: 'trap \'printf "exit\\n" >> "$TRACE"\' EXIT; sleep 10', code: 124, timeout: true, userExit: true },
{ name: 'forced termination falls back to cleanup', task: 'trap "" INT; sleep 30', code: 124, timeout: true },
@@ -6,10 +6,16 @@ const os = require('node:os');
const { createContext } = require('../../dist/internal/runtime/context');
const { reloadPanel } = require('../../dist/internal/maintenance/upgrade');
const operator = require('../../dist/internal/maintenance/operator');
const { writeManifest } = require('../../../test/helpers/upgrade-fixture.cjs');
const selectedLoader =
'// QL_CLI_ROOT dist/local/entrypoints.js dist/local/cronEntrypoint.js\n';
async function fixture(t) {
t.mock.method(
require('../../dist/internal/maintenance/upgradeArtifacts.cjs'),
'waitForHealth',
async () => {},
);
const base = await fs.mkdtemp(
path.join(os.tmpdir(), 'ql-upgrade-selection-'),
);
@@ -43,6 +49,7 @@ async function fixture(t) {
await fs.writeFile(incoming, '// original released loader');
await fs.writeFile(path.join(ctx.paths.dir_static, 'build/app.js'), 'old');
await fs.writeFile(path.join(staticRoot, 'build/app.js'), 'new');
writeManifest(source, staticRoot);
return { ctx, source, staticRoot, loader, incoming };
}
@@ -104,3 +111,33 @@ test('unsupported target and unrecognized loader fail before service stop', asyn
assert.equal(stopped, false);
assert.equal(await fs.readFile(f.loader, 'utf8'), 'unrecognized');
});
test('successful process start with failed health check rolls back the installed version', async (t) => {
const f = await fixture(t);
t.mock.method(operator, 'stopPanel', async () => {});
t.mock.method(operator, 'startPanel', async () => ({ manager: 'fixture' }));
let probes = 0;
t.mock.method(
require('../../dist/internal/maintenance/upgradeArtifacts.cjs'),
'waitForHealth',
async () => {
if (++probes === 1) throw new Error('health check failed');
assert.equal(
await fs.readFile(
path.join(f.ctx.paths.dir_static, 'build/app.js'),
'utf8',
),
'old',
);
},
);
await assert.rejects(
reloadPanel(f.ctx, 'system', { source: f.source, static: f.staticRoot }),
/health check failed/,
);
assert.equal(probes, 2);
assert.equal(
await fs.readFile(path.join(f.ctx.root, 'package.json'), 'utf8'),
'{"version":"2.19.0"}',
);
});
+13 -2
View File
@@ -2,7 +2,9 @@ const assert = require('node:assert/strict');
const fs = require('node:fs/promises');
const { randomUUID } = require('node:crypto');
const { createContext } = require('../../dist/internal/runtime/context');
const { installPanelDependencies } = require('../../dist/internal/maintenance/operator');
const {
installPanelDependencies,
} = require('../../dist/internal/maintenance/operator');
const { reloadPanel } = require('../../dist/internal/maintenance/upgrade');
assert.equal(process.env.QL_PANEL_INTEGRATION, '1');
(async () => {
@@ -65,7 +67,16 @@ assert.equal(process.env.QL_PANEL_INTEGRATION, '1');
const config = await fs.readFile('/ql/data/config/config.sh');
const dotenv = await fs.readFile('/ql/.env');
const context = createContext({ root: '/ql' }, process.env);
await installPanelDependencies(context, '/stage/source');
const stagedManifest = JSON.parse(
await fs.readFile('/stage/static/build-info.json', 'utf8'),
);
assert.match(stagedManifest.sourceCommit, /^[0-9a-f]{40}$/);
await installPanelDependencies(
context,
'/stage/source',
'/stage/static',
stagedManifest.sourceCommit,
);
const replacement = await reloadPanel(context, 'system', {
source: '/stage/source',
static: '/stage/static',
+52 -1
View File
@@ -54,7 +54,7 @@ test('API diagnostics preserve resource scope, uncertainty, codes and secret sup
language === 'en' ? /outcome.*unknown/ : /结果.*未知/,
);
}
if (failure !== 'network') {
if (failure === 'permission' || failure === 'api') {
assert.ok(
error.message.includes(
endpoint.startsWith('subscriptions')
@@ -89,3 +89,54 @@ test('API diagnostics preserve resource scope, uncertainty, codes and secret sup
);
}
});
test('HTTP and body errors distinguish conflicts, authentication and server failures without leaking bodies', async (t) => {
const previous = process.env.QL_LANG;
t.after(() => {
if (previous === undefined) delete process.env.QL_LANG;
else process.env.QL_LANG = previous;
});
for (const language of ['en', 'zh']) {
process.env.QL_LANG = language;
for (const kind of ['HTTP', 'code'])
for (const status of [400, 401, 403, 409, 500, 503])
for (const method of ['GET', 'PUT']) {
t.mock.method(
global,
'fetch',
async () =>
new Response(
JSON.stringify({ code: status, message: 'secret-value' }),
{ status: kind === 'HTTP' ? status : 200 },
),
);
await assert.rejects(
request(
{ url: 'https://fixture.invalid', token: 'secret-token' },
'envs',
{ method },
),
(error) => {
assert.equal(error.exitCode, [401, 403].includes(status) ? 3 : 1);
assert.match(error.message, new RegExp(`${kind} ${status}`));
assert.doesNotMatch(error.message, /secret-|fixture/);
if ([401, 403].includes(status))
assert.match(error.message, /credentials|凭据/);
else
assert.doesNotMatch(
error.message,
/credentials|permission|凭据|权限/,
);
if (status === 409) assert.match(error.message, /conflict|冲突/);
if (status >= 500)
assert.match(error.message, /Server error|服务端错误/);
if (status >= 500 && method === 'PUT')
assert.match(error.message, /unknown|未知/);
else assert.doesNotMatch(error.message, /unknown|未知/);
return true;
},
);
t.mock.restoreAll();
}
}
});
+51
View File
@@ -58,6 +58,31 @@ test('dashboard record sends execution statistics and rejects a missing body bef
assert.equal(f.requests.length, 1);
});
test('script history commands send exact query parameters and restore payloads', async t => {
const f = await fixture(t);
const identity = { filename: '中文 script.js', path: 'nested folder' };
const version = { ...identity, id: '32ec8b22-2c48-4bcf-85d4-b394198417ba' };
for (const [command, endpoint, query] of [
['history-list', 'scripts/history', identity],
['history-detail', 'scripts/history/detail', version],
]) {
success(await f.run(['script', command, '--query', JSON.stringify(query)]));
const req = f.requests.at(-1), url = new URL(req.url, 'http://localhost');
assert.equal(req.method, 'GET');
assert.equal(url.pathname, '/panel/open/' + endpoint);
for (const [key, value] of Object.entries(query)) assert.equal(url.searchParams.get(key), value);
assert.equal(req.body, '');
}
const payload = { ...version, expectedHash: 'a'.repeat(64) };
success(await f.run(['script', 'history-restore', '--data', '-'], JSON.stringify(payload)));
assert.equal(f.requests.at(-1).method, 'PUT');
assert.equal(f.requests.at(-1).url, '/panel/open/scripts/history/restore');
assert.deepEqual(JSON.parse(f.requests.at(-1).body), payload);
const count = f.requests.length;
assert.equal((await f.run(['script', 'history-restore'])).code, 2);
assert.equal(f.requests.length, count, 'missing restore payload must not issue a request');
});
test('OpenAPI catalogue covers every active registered backend route; retired 410 routes are explicit', async () => {
const root = path.resolve(__dirname, '../../../back/api');
const expected = [];
@@ -186,3 +211,29 @@ test('owner two-factor challenge remains actionable without exposing server mess
assert.equal(result.code, 3); assert.match(result.err, /two-factor-login/); assert.doesNotMatch(result.err, /SERVER-SECRET/);
assert.equal(f.requests.length, 1); assert.equal(f.requests[0].headers.authorization, undefined);
});
test('node mirror configuration accepts the real empty response for named and raw API commands', async t => {
const f = await fixture(t);
f.reply((_req, res) => { res.writeHead(200, { 'content-type': 'application/octet-stream' }); res.end(); });
for (const args of [
['system', 'config-node-mirror'],
['api', 'request', 'PUT', '/open/system/config/node-mirror'],
]) {
const reply = success(await f.run([...args, '--data', '{"nodeMirror":"https://registry.npmmirror.com"}']));
assert.equal(reply.code, 200);
assert.equal(f.requests.at(-1).method, 'PUT');
assert.equal(f.requests.at(-1).url, '/panel/open/system/config/node-mirror');
}
assert.equal(f.requests.length, 2, 'configuration writes must not be retried');
});
test('empty node mirror responses still reject authentication and server failures without retries', async t => {
const f = await fixture(t);
for (const status of [401, 403, 500]) {
f.reply((_req, res) => { res.writeHead(status, { 'content-type': 'application/octet-stream' }); res.end(); });
const result = await f.run(['system', 'config-node-mirror', '--data', '{}']);
assert.notEqual(result.code, 0);
assert.equal(result.out, '');
}
assert.equal(f.requests.length, 3);
});
+6 -5
View File
@@ -1,15 +1,17 @@
# Run Node package installation natively on the builder. Node/npm can spin at
# 100% CPU when Alpine s390x is emulated through QEMU.
FROM --platform=$BUILDPLATFORM node:18-alpine3.18 AS builder
FROM --platform=$BUILDPLATFORM node:22.23.3-alpine3.24 AS builder
ARG TARGETARCH
ENV NPM_CONFIG_PREFIX=/opt/node-global
ENV PATH=/opt/node-global/bin:${PATH}
COPY package.json .npmrc pnpm-lock.yaml /tmp/build/
COPY static/runtime-tools.tgz static/runtime-tools-proof.json /tmp/runtime-tools/
COPY scripts/install-runtime-tools.cjs /tmp/install-runtime-tools.cjs
RUN set -x \
&& apk add --no-cache git \
&& npm i -g pnpm@8.3.1 pm2 ts-node typescript@5 \
&& node /tmp/install-runtime-tools.cjs --archive /tmp/runtime-tools/runtime-tools.tgz --proof /tmp/runtime-tools/runtime-tools-proof.json --prefix /opt/node-global \
&& cd /tmp/build \
&& case "${TARGETARCH}" in \
amd64) NODE_ARCH=x64 ;; \
@@ -22,7 +24,7 @@ RUN set -x \
npm_config_target_libc=musl \
pnpm install --prod --frozen-lockfile
FROM python:3.11-alpine
FROM python:3.11.17-alpine3.24@sha256:d9368b3a5ac59afea7b5d4f2e2aea0941dbf9fdee9c369c5bec00b98244bc929
ARG QL_MAINTAINER="whyour"
LABEL maintainer="${QL_MAINTAINER}"
@@ -41,7 +43,7 @@ VOLUME /ql/data
EXPOSE 5700
COPY --from=builder /opt/node-global/lib/node_modules/. /usr/local/lib/node_modules/
COPY --from=builder /opt/node-global/lib/. /usr/local/lib/
COPY --from=builder /opt/node-global/bin/. /usr/local/bin/
RUN set -x \
@@ -61,7 +63,6 @@ RUN set -x \
procps \
netcat-openbsd \
unzip \
npm \
&& rm -rf /var/cache/apk/* \
&& apk update \
&& ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
+5 -4
View File
@@ -1,15 +1,17 @@
# Run Node package installation natively on the builder. Node/npm can spin at
# 100% CPU when Alpine s390x is emulated through QEMU.
FROM --platform=$BUILDPLATFORM node:18-alpine3.18 AS builder
FROM --platform=$BUILDPLATFORM node:22.23.3-alpine3.24 AS builder
ARG TARGETARCH
ENV NPM_CONFIG_PREFIX=/opt/node-global
ENV PATH=/opt/node-global/bin:${PATH}
COPY package.json .npmrc pnpm-lock.yaml /tmp/build/
COPY static/runtime-tools.tgz static/runtime-tools-proof.json /tmp/runtime-tools/
COPY scripts/install-runtime-tools.cjs /tmp/install-runtime-tools.cjs
RUN set -x \
&& apk add --no-cache git \
&& npm i -g pnpm@8.3.1 pm2 ts-node typescript@5 \
&& node /tmp/install-runtime-tools.cjs --archive /tmp/runtime-tools/runtime-tools.tgz --proof /tmp/runtime-tools/runtime-tools-proof.json --prefix /opt/node-global \
&& cd /tmp/build \
&& case "${TARGETARCH}" in \
amd64) NODE_ARCH=x64 ;; \
@@ -41,7 +43,7 @@ VOLUME /ql/data
EXPOSE 5700
COPY --from=builder /opt/node-global/lib/node_modules/. /usr/local/lib/node_modules/
COPY --from=builder /opt/node-global/lib/. /usr/local/lib/
COPY --from=builder /opt/node-global/bin/. /usr/local/bin/
RUN set -x \
@@ -61,7 +63,6 @@ RUN set -x \
procps \
netcat-openbsd \
unzip \
npm \
&& rm -rf /var/cache/apk/* \
&& apk update \
&& ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
+19 -11
View File
@@ -1,20 +1,26 @@
# Node 20 Bookworm is the latest official Node image variant that covers the
# full Debian build matrix, including arm/v7, ppc64le, and s390x.
FROM node:20-bookworm-slim AS nodebuilder
# Download authenticated target binaries natively; Node 22 Bookworm images do
# not include s390x, while official Node tarballs retain all five targets.
FROM --platform=$BUILDPLATFORM node:22.23.3-bookworm-slim AS nodebuilder
ARG TARGETARCH
ARG TARGETVARIANT
COPY docker/install-node.sh docker/node-release-key.asc docker/node-v22.23.3-SHASUMS256.txt.asc /tmp/
RUN apt-get update && apt-get install --no-install-recommends -y ca-certificates curl gnupg xz-utils \
&& sh /tmp/install-node.sh
FROM python:3.11.14-slim-bookworm AS builder
FROM python:3.11.17-slim-trixie@sha256:e88e9763f943ec1834f992a4b51e0f24500486803e8bc534e5767af9ea65f6ce AS builder
COPY package.json .npmrc pnpm-lock.yaml /tmp/build/
COPY --from=nodebuilder /usr/local/bin/node /usr/local/bin/
COPY --from=nodebuilder /usr/local/lib/node_modules/. /usr/local/lib/node_modules/
COPY static/runtime-tools.tgz static/runtime-tools-proof.json /tmp/runtime-tools/
COPY scripts/install-runtime-tools.cjs /tmp/install-runtime-tools.cjs
COPY --from=nodebuilder /opt/node-target/bin/node /usr/local/bin/
RUN set -x && \
ln -s /usr/local/lib/node_modules/npm/bin/npm-cli.js /usr/local/bin/npm && \
apt-get update && \
apt-get install --no-install-recommends -y libatomic1 && \
npm i -g pnpm@8.3.1 && \
node /tmp/install-runtime-tools.cjs --archive /tmp/runtime-tools/runtime-tools.tgz --proof /tmp/runtime-tools/runtime-tools-proof.json --prefix /usr/local && \
cd /tmp/build && \
pnpm install --prod --frozen-lockfile
FROM python:3.11.14-slim-bookworm
FROM python:3.11.17-slim-trixie@sha256:e88e9763f943ec1834f992a4b51e0f24500486803e8bc534e5767af9ea65f6ce
ARG QL_MAINTAINER="whyour"
LABEL maintainer="${QL_MAINTAINER}"
@@ -42,8 +48,9 @@ RUN groupadd -g ${QL_GID} qinglong && \
ENV QL_USER=qinglong
ENV QL_HOME=/home/$QL_USER
COPY --from=nodebuilder /usr/local/bin/node /usr/local/bin/
COPY --from=nodebuilder /usr/local/lib/node_modules/. /usr/local/lib/node_modules/
COPY --from=nodebuilder /opt/node-target/bin/node /usr/local/bin/
COPY static/runtime-tools.tgz static/runtime-tools-proof.json /tmp/runtime-tools/
COPY scripts/install-runtime-tools.cjs /tmp/install-runtime-tools.cjs
RUN set -x && \
ln -s /usr/local/lib/node_modules/npm/bin/npm-cli.js /usr/local/bin/npm && \
@@ -69,7 +76,8 @@ RUN set -x && \
git config --global user.email "qinglong@users.noreply.github.com" && \
git config --global user.name "qinglong" && \
git config --global http.postBuffer 524288000 && \
npm install -g pnpm@8.3.1 pm2 ts-node typescript@5 && \
node /tmp/install-runtime-tools.cjs --archive /tmp/runtime-tools/runtime-tools.tgz --proof /tmp/runtime-tools/runtime-tools-proof.json --prefix /usr/local && \
rm -rf /tmp/runtime-tools /tmp/install-runtime-tools.cjs && \
npm cache clean --force && \
rm -rf /root/.cache && \
rm -rf /root/.npm && \
+17 -9
View File
@@ -1,18 +1,24 @@
# Node 20 Bookworm is the latest official Node image variant that covers the
# full Debian build matrix, including arm/v7, ppc64le, and s390x.
FROM node:20-bookworm-slim AS nodebuilder
# Download authenticated target binaries natively; Node 22 Bookworm images do
# not include s390x, while official Node tarballs retain all five targets.
FROM --platform=$BUILDPLATFORM node:22.23.3-bookworm-slim AS nodebuilder
ARG TARGETARCH
ARG TARGETVARIANT
COPY docker/install-node.sh docker/node-release-key.asc docker/node-v22.23.3-SHASUMS256.txt.asc /tmp/
RUN apt-get update && apt-get install --no-install-recommends -y ca-certificates curl gnupg xz-utils \
&& sh /tmp/install-node.sh
# Keep the full release matrix, including s390x, on the same immutable base.
# The floating 3.10-slim-bookworm tag no longer includes s390x.
FROM python:3.10.19-slim-bookworm@sha256:23f63358922e79a794f71be8f3723c84e5ccca9638af3f74456dc73d6184499e AS builder
COPY package.json .npmrc pnpm-lock.yaml /tmp/build/
COPY --from=nodebuilder /usr/local/bin/node /usr/local/bin/
COPY --from=nodebuilder /usr/local/lib/node_modules/. /usr/local/lib/node_modules/
COPY static/runtime-tools.tgz static/runtime-tools-proof.json /tmp/runtime-tools/
COPY scripts/install-runtime-tools.cjs /tmp/install-runtime-tools.cjs
COPY --from=nodebuilder /opt/node-target/bin/node /usr/local/bin/
RUN set -x && \
ln -s /usr/local/lib/node_modules/npm/bin/npm-cli.js /usr/local/bin/npm && \
apt-get update && \
apt-get install --no-install-recommends -y libatomic1 && \
npm i -g pnpm@8.3.1 && \
node /tmp/install-runtime-tools.cjs --archive /tmp/runtime-tools/runtime-tools.tgz --proof /tmp/runtime-tools/runtime-tools-proof.json --prefix /usr/local && \
cd /tmp/build && \
pnpm install --prod --frozen-lockfile
@@ -44,8 +50,9 @@ RUN groupadd -g ${QL_GID} qinglong && \
ENV QL_USER=qinglong
ENV QL_HOME=/home/$QL_USER
COPY --from=nodebuilder /usr/local/bin/node /usr/local/bin/
COPY --from=nodebuilder /usr/local/lib/node_modules/. /usr/local/lib/node_modules/
COPY --from=nodebuilder /opt/node-target/bin/node /usr/local/bin/
COPY static/runtime-tools.tgz static/runtime-tools-proof.json /tmp/runtime-tools/
COPY scripts/install-runtime-tools.cjs /tmp/install-runtime-tools.cjs
RUN set -x && \
ln -s /usr/local/lib/node_modules/npm/bin/npm-cli.js /usr/local/bin/npm && \
@@ -70,7 +77,8 @@ RUN set -x && \
git config --global user.email "qinglong@users.noreply.github.com" && \
git config --global user.name "qinglong" && \
git config --global http.postBuffer 524288000 && \
npm install -g pnpm@8.3.1 pm2 ts-node typescript@5 && \
node /tmp/install-runtime-tools.cjs --archive /tmp/runtime-tools/runtime-tools.tgz --proof /tmp/runtime-tools/runtime-tools-proof.json --prefix /usr/local && \
rm -rf /tmp/runtime-tools /tmp/install-runtime-tools.cjs && \
npm cache clean --force && \
rm -rf /root/.cache && \
rm -rf /root/.npm && \
+38
View File
@@ -0,0 +1,38 @@
#!/bin/sh
set -eu
# These release files are committed together so every target architecture uses
# the same official Node release and authenticated checksum list.
NODE_VERSION=22.23.3
NODE_SIGNER=5BE8A3F6C8A5C01D106C0AD820B1A390B168D356
case "${TARGETARCH:?}/${TARGETVARIANT:-}" in
amd64/) NODE_ARCH=x64 ;;
arm/v7) NODE_ARCH=armv7l ;;
arm64/|arm64/v8) NODE_ARCH=arm64 ;;
ppc64le/) NODE_ARCH=ppc64le ;;
s390x/) NODE_ARCH=s390x ;;
*) printf 'Unsupported Debian Node architecture: %s/%s\n' "$TARGETARCH" "${TARGETVARIANT:-}" >&2; exit 1 ;;
esac
DOWNLOAD_DIR=$(mktemp -d)
KEY_DIR=$(mktemp -d)
trap 'rm -rf "$DOWNLOAD_DIR" "$KEY_DIR"' EXIT HUP INT TERM
chmod 700 "$KEY_DIR"
gpg --homedir "$KEY_DIR" --batch --import /tmp/node-release-key.asc
FINGERPRINT=$(gpg --homedir "$KEY_DIR" --batch --with-colons --fingerprint "$NODE_SIGNER" | awk -F: '$1 == "fpr" { print $10; exit }')
test "$FINGERPRINT" = "$NODE_SIGNER"
gpg --homedir "$KEY_DIR" --batch --status-fd 3 --output "$DOWNLOAD_DIR/SHASUMS256.txt" \
--decrypt "/tmp/node-v${NODE_VERSION}-SHASUMS256.txt.asc" 3> "$DOWNLOAD_DIR/signature.status"
grep -q "^\[GNUPG:\] VALIDSIG ${NODE_SIGNER} " "$DOWNLOAD_DIR/signature.status"
NODE_FILE="node-v${NODE_VERSION}-linux-${NODE_ARCH}.tar.xz"
cd "$DOWNLOAD_DIR"
grep " ${NODE_FILE}$" SHASUMS256.txt > expected.sha256
test "$(wc -l < expected.sha256 | tr -d ' ')" = 1
curl --fail --show-error --silent --location --proto '=https' --proto-redir '=https' --tlsv1.2 --retry 3 \
"https://nodejs.org/dist/v${NODE_VERSION}/${NODE_FILE}" --output "$NODE_FILE"
sha256sum --check --strict expected.sha256
mkdir -p /opt/node-target
tar --extract --xz --file "$NODE_FILE" --directory /opt/node-target --strip-components=1
test -x /opt/node-target/bin/node
test -f /opt/node-target/lib/node_modules/npm/package.json
+17
View File
@@ -0,0 +1,17 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mDMEaGA63BYJKwYBBAHaRw8BAQdAo/yU+MutacFmmn0CEX495goNrBxR24235XLM
cvHYjfq0L0FudG9pbmUgZHUgSGFtZWwgPGR1aGFtZWxhbnRvaW5lMTk5NUBnbWFp
bC5jb20+iI4EExYKADYWIQRb6KP2yKXAHRBsCtggsaOQsWjTVgUCaGA63AIbAwQL
CQgHBBUKCQgFFgIDAQACHgUCF4AACgkQILGjkLFo01afgwEA/sLHqsj7ml2vyDoT
KDPE8n9a80ZOh14OfnlOe0cCZA8BAMEOOk7QFI69DIlV1nMiqcFCqQFoSzBU2LkI
R17p/j4NtDNBbnRvaW5lIGR1IEhhbWVsIDxhbnRvaW5lLmR1aGFtZWxAcGxhdGZv
cm1hdGljLmRldj6IjgQTFgoANhYhBFvoo/bIpcAdEGwK2CCxo5CxaNNWBQJpsCMx
AhsDBAsJCAcEFQoJCAUWAgMBAAIeAQIXgAAKCRAgsaOQsWjTVr/sAPwIBsG8g6ND
zoNRTX1wPKBvfZg1NP7tYCyM5sxQfrpuLAEA05AhG4xBILfhL/f0pqR5jXfxg6gz
T6WfeVeS6zeHZwe4OARoYDrcEgorBgEEAZdVAQUBAQdAQVmtih8AO3ryBQMR/22x
WHVKLjAbCiH2cMxNH+iy1RQDAQgHiHgEGBYKACAWIQRb6KP2yKXAHRBsCtggsaOQ
sWjTVgUCaGA63AIbDAAKCRAgsaOQsWjTVu8oAP9Bc+QY+9FikX3YvMgWAqiDlVOy
o0y6UIZGBMSQlF80wAD/d34LqtVIVe9oe5NO3xA75+6Ew8tGeAjUq/ovagr5dAU=
=JsVv
-----END PGP PUBLIC KEY BLOCK-----
+54
View File
@@ -0,0 +1,54 @@
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
b85dc5f0d7a9be8038c4e73450d9125b241c792ec7ec495229345f86d1e3f6fa node-v22.23.3-aix-ppc64.tar.gz
a6fcbecafce9c8fbf86ffcb5fb12a66af5f465c449ef5e2fcf7c4e651c1a9b14 node-v22.23.3-arm64.msi
23b25245dcfb9af7262f8ff142e9e2e0af025368117329e7a7458a51e5922f53 node-v22.23.3-darwin-arm64.tar.gz
72d5d8832b41c9d9646197af614ffd751406ea4d215060eb91b98864e1919a3e node-v22.23.3-darwin-arm64.tar.xz
8a677b0219178efd6eb0e475457c4afb452b521a92f6e67845a73bd85727f2a8 node-v22.23.3-darwin-x64.tar.gz
ac41874c3352937119cfec39e1a98c578fe58ce8a86d7e785b89a4706015f305 node-v22.23.3-darwin-x64.tar.xz
eff1a7e67736bdf25db421ddaa613ed605f751a7b9fe07fb473ea880d1348987 node-v22.23.3-headers.tar.gz
8448fe9f94d65aece4df26f266358644d6f41da4ce23aab09675e206edcff1d4 node-v22.23.3-headers.tar.xz
5ced2d48d1d7198739b7f86804de0171aefb6823b684b12341d3321afc3cb0b2 node-v22.23.3-linux-arm64.tar.gz
a44aeb94849a299b22df10b9e622ec2f605c2183501bc40590705131de7c740f node-v22.23.3-linux-arm64.tar.xz
8b99058ec6f4e372b9887f8ecb577a292adb132d59e5104356428a758fdb126c node-v22.23.3-linux-armv7l.tar.gz
590a2768199bdd0b848648a81cd7f07f88a5ceb27949f0c70d5e6652c3eaca69 node-v22.23.3-linux-armv7l.tar.xz
236712ddef73f56c0e0590a2c49f69c8393a2c2d82c3d62c7c165605b472b38d node-v22.23.3-linux-ppc64le.tar.gz
0214e542c73686134856784c0cbd2762b5414097640762b4afc2e33e6e499207 node-v22.23.3-linux-ppc64le.tar.xz
51f64794f5f61bdfc80aa186c8273021473997605cf90ed31ed148bee650a554 node-v22.23.3-linux-s390x.tar.gz
a311c8f0b352e7dedc13958b612ac190a56456fc837dd26cd5e5845713a1eb1d node-v22.23.3-linux-s390x.tar.xz
633f49a4f388c5f9564d56d9c0f5ffa394346d92ca0076fe21cf776ece1988bc node-v22.23.3-linux-x64-musl.tar.gz
0564c678817e6b6fd7860689283185cecdf42e644135b45636094757b9a7922b node-v22.23.3-linux-x64-musl.tar.xz
1084aa36196bba4c3a5e69a1ee388a6e4ff729dad09445fbcd434b28fe3c24af node-v22.23.3-linux-x64.tar.gz
df450af89261115ef9f9e3830c3eeb2cc9213b63c720b1af623cb5dcbe2e02de node-v22.23.3-linux-x64.tar.xz
55c853a53b4bf88c1d8fd70e6add0b22b49c0e76352d4044caaa90512734ab7e node-v22.23.3.pkg
9436c81b284889303d39f5b8bd629bf19e790c665ccb94b4994aa87220d9799a node-v22.23.3.tar.gz
bd97093e1a1e9243338950c174a693a64d4e0926a9c6ce259962bc58d5e96909 node-v22.23.3.tar.xz
33fbc3458ce5cddd694f319bcb47490fbce720c492e25573b594e205c189c89a node-v22.23.3-win-arm64.7z
33dad22e4cef5ee8f9fbb1b0d037fdacd0e56d12a4580f0d63f68b894deab535 node-v22.23.3-win-arm64.zip
fa7a98510f2edf181733dee15a200a5039a0459511366150e4c64da72a53ab94 node-v22.23.3-win-x64.7z
2b0ff57b049cda1bbcea2240eec20467018713c1efe1f7360c2681859b90ed71 node-v22.23.3-win-x64.zip
ff5ffeb0cc42d76b4bfe453638dbe9a69738fdaec967a14cd84f3b11225bb523 node-v22.23.3-win-x86.7z
1e07ac00e61b41b7ecf242de8e38883d237d5ad3fdcad7c16f2dd3a88670e2b6 node-v22.23.3-win-x86.zip
1c0efc8449987e7da5d184786a0a96da83ffa11d334421201e5c09b93017cb8d node-v22.23.3-x64.msi
c71bbe31029c99d6f9cffa4841056ecc49dcd1fb09ef739b7e85dc7b207242b4 node-v22.23.3-x86.msi
b5a3165ec6f24c0b1fe2a10fed58e03f6c1d7d141beb3b225ff030ceac6c5eb5 win-arm64/node.exe
5378f5bad777b4929d1cd4f58974e0ef03815ce5c159863f1875c4b434460c27 win-arm64/node.lib
3f21d9cee58b7168ee46827e04d381dcdeb2d1cc5974bc226ba3a1bcd6bf8988 win-arm64/node_pdb.7z
bb155f2ecad21c98cd909831cb26320e123dd010a654f3649e8ba54a7de9f433 win-arm64/node_pdb.zip
9c9245166b4a8e182e0b797da9c20136117ff24368eaff1fec8343a123c8db0e win-x64/node.exe
c243185f2347870cadb637c909bf123fc703dc65dbfa769b17ec45a4b661d782 win-x64/node.lib
1babb45798cf3a2d4e70574ba926e7ff26b71487ed940a98cdd4971f32af4cae win-x64/node_pdb.7z
ab9fe384f04575fab7b2915b02d109a1d415b184cd6b28239fdb67e12c260321 win-x64/node_pdb.zip
a8aa72dda43af5357d0a502548a54e402d4f7643be221662624a2cc876cd37bb win-x86/node.exe
eff47cc18fe2d3e40886cd49f04a85134681657d8b0f2afc55f363743cf6ddd7 win-x86/node.lib
18963f55b0ebda23c72ab023990e68bce5302645f9643b6f0ac52dc809e3c501 win-x86/node_pdb.7z
95d75dce4db28836ef7ff2ddada21c1df67cc07f7d9a727ac38b021ee3afdeaa win-x86/node_pdb.zip
-----BEGIN PGP SIGNATURE-----
iHUEARYIAB0WIQRb6KP2yKXAHRBsCtggsaOQsWjTVgUCarQJ5gAKCRAgsaOQsWjT
Vi6XAQD18cb42T8Iokdj3jgAfoB41fqECt7DeN43VsgOoSNDZwEAxqn3W+nDSkFo
aQyV6A9gETWcFz27zSzMVA+vDrzbvQM=
=2yJE
-----END PGP SIGNATURE-----
+49
View File
@@ -0,0 +1,49 @@
# Node cron 调度器迁移
Node 调度路径使用固定版本 `node-cron@4.6.0`,普通任务(含额外规则)、订阅和系统 cron 共用 `back/shared/cronScheduler.ts`。启动/一次性任务、间隔任务、脚本并发限制和进程执行链路保持原有实现。系统模式只将语义一致的普通五字段规则交给 crond。
移除 `node-schedule`、其类型包以及兼容别名 `cron-parser-v4`。保留 `cron-parser@5` 做旧表达式的校验与标准化;计时器和日历推进由 node-cron 管理。
## 兼容行为
- 保留旧版 1–6 字段补全、五种宏、月份/星期名称、数字起点步长、L 和 #。另外明确新增 `@annually`(等同 `@yearly`)、`@midnight`(等同 `@daily`)、`@minutely`(每分钟第 0 秒)。宏按进程时区解释。`?` 仅允许日期和星期字段;裸 `/N`、H 和其他未列出的宏仍拒绝。
- 日期与星期同时指定时保留 OR 语义,不采用 node-cron 默认的 AND。两个原生任务分别处理日期和星期,并按计划时间过滤交集,避免重复执行;保留旧解析器的月份全日期范围及全局 nth-week 限制。
- 原生任务先暂停创建并验证,再替换旧任务。构造失败时销毁已准备的实例并保留旧快照。
- 进程内迟到的触发通过 `execution:missed` 加入既有执行队列,记录补执行日志。并发和单任务积压上限仍由青龙队列控制;这不提供进程退出期间的持久化补跑或分布式 exactly-once 保证。
- 标准化结果使用上限 512 条的只读缓存,仅缓存语法,不缓存运行实例或下次执行时间。避免大批相同表达式重复解析导致恢复 RPC 超时。
- 回调异常记录日志,后续调度继续;取消时调用原生 `destroy()` 释放计时器和注册表。
- 持久化无效任务在恢复时跳过并记录日志,无效订阅不会产生未处理的启动拒绝。API、导入和执行使用同一套表达式校验。
## 复审修复
- 移除没有生产调用的 `nextInvocation()` 接口,避免对 OR/# 等过滤规则报告不会实际执行的原生日期。回归测试直接观察实际执行和原生实例状态。
- 批量注册 RPC 时限为 5 秒加每条规则 5 毫秒,计入额外规则并封顶 120 秒;保留超时后的恢复机制,不立即重放结果不确定的写入。
- Alpine/系统模式下,数字起点步长(如 `0/5`)、星期包含 `7`、日期步长、日期和星期同时受限的表达式转入 Node 路径。BusyBox 的这些解析或通配语义与旧 Node 解析器不同,不能仅因是纯数字五字段就交给 crond。
- 这些规则在 `crontab.list` 中仍保留为注释,避免同时被系统和 Node 执行。其他可移植规则仍使用系统 crond。
## 夏令时差异
采用 node-cron 的当地时间语义:春季跳时不存在的时间跳过,秋季重复小时不再执行第二次。旧 cron-parser 对部分春季时间会顺延,并可能重复秋季高频任务。使用有夏令时的时区时需要考虑该行为变化;Asia/Shanghai 不受影响。回归测试固定 America/New_York 的 2026 年转换日期。
## 验证
`npm test` 和 `npm run build:back`。
- `test/fixtures/legacy-cron.json` 是迁移前使用 node-schedule 2.1.1 / cron-parser 4.9.0 保存的快照:269 条语法判定,18 组日历序列,每组 10 次执行。
- 真正的 node-cron 实例配合虚拟时钟验证日期/星期 OR、交集去重、闰年、L/#、8 秒迟到补执行、取消释放、回调拒绝和夏令时。
- 注册构造失败回滚、无效订阅隔离、空表达式立即运行,以及现有调度恢复/503/子进程测试覆盖接入流程。
- 隔离的正式 Debian 镜像使用实际编译产物验证 10,000 条有效任务和 2 条错误任务的恢复、300 个同时到期的 Python 脚本、并发上限 16 及 HTTP 健康检查。具体结果见本次测试报告;这是生产镜像隔离验证,不是线上长期运行证明。
### 本次验收结果(2026-09-27)
完整回归 260 通过、3 项原有跳过,后端编译成功。Node 20 生产运行时的 21 项兼容测试通过。隔离镜像中 300/300 脚本完成,无重复或遗漏,最大并发 16,270 个成功及 30 个预设失败均落库;完成耗时 52.36 秒。恢复后健康检查 1,870/1,870 成功,任务列表 38/38 成功。重启恢复阶段另有 50 次健康检查失败。
初版重复解析曾导致恢复超过 5 秒 RPC 时限,加入有上限的语法缓存后重跑通过。本次 10,000 个任务共享两个表达式,不能视为一万种复杂表达式的容量保证;实际脚本耗时也不代表纯调度性能。
新增宏在 UTC 和 Asia/Shanghai 下验证连续两次实际触发及不提前触发;原有语法快照仅对这三个新增宏允许预期变化。
### 复审后验收
完整回归 267 通过、3 项原有跳过;Node 20 的 10 项修复回归及 Alpine 镜像的 24 项兼容/宏测试通过。
额外使用 9,700 个不同表达式和 300 个同时到期任务验证恢复(另有 2 个无效任务),不再局限于共享两个表达式。300/300 脚本完成,无重复或遗漏,最大并发 16,270 个退出码 0、30 个预设退出码 3 均落库。恢复后健康检查 1,650/1,650 成功、任务列表 33/33 成功;启动恢复阶段另有 260 次健康检查失败,整批脚本耗时 60.27 秒。较大快照恢复仍需等待,不承诺启动阶段始终可用或任意规模都能在时限内恢复。
@@ -0,0 +1,35 @@
# Production dependency validation
Date: 2026-10-10. Source baseline: `68ffa106f80a2080ddd9b48affe0d7a7489e4d05` on `develop`.
The baseline frozen production lockfile reported 4 Critical, 48 High, 56 Moderate and 14 Low advisory matches across 410 dependencies. After the targeted updates, `pnpm audit --prod --json` reports zero matches at every severity across 368 dependencies. These counts describe the dependency advisory database at the time of verification; they are not a replacement for application security review.
## Changes
- Update the HTTP parser, compression, upload, YAML, Lodash, Sequelize, gRPC and Undici dependencies to patched versions while retaining their supported release lines. Use Undici 7 for the existing Node 20 runtime, rather than Undici 8's newer Node requirement.
- Upgrade Nodemailer to 10.0.16, including the address parser and OAuth2 TLS fixes. Nodemailer 10 requires Node 20; backend type checking and local message generation are included in validation.
- Pin patched indirect dependencies with scoped pnpm overrides. Keep old path-to-regexp, Joi, brace-expansion, picomatch and Undici consumers on compatible release lines.
- Remove the unused direct `http-proxy-middleware` production dependency. There are no imports or calls in the backend, frontend, CLI or shell sources. This removes its `micromatch -> braces` chain from the production graph; braces 3.0.3 has no upstream patch. Umi's development proxy remains managed by Umi's own development dependencies.
- Upgrade the SQLite installation chain to `@mapbox/node-pre-gyp@2.0.3` and `tar@7.5.22`. Retain the existing `@whyour/sqlite3@1.1.2` binary package and N-API ABI. The SQLite source extractor uses the supported synchronous `tar.extract` API.
- Replace nested UUID 8 with UUID 11.1.1 for Sequelize and SockJS. Database transactions and a real SockJS/WebSocket exchange verify their production use of the UUID API.
No advisory ignore list or risk acceptance exception was added. Development dependencies are outside the production audit claim.
## Local verification
- Independent Node 20.20.2 installation using the frozen lockfile; no shared node_modules.
- SQLite precompiled download and extraction through the upgraded node-pre-gyp/tar chain succeeded on macOS arm64.
- Forced SQLite source compilation using node-gyp 12.4.0 succeeded on macOS arm64, followed by the same production protocol checks. The local SDK required its C++ include directory to be supplied explicitly; no global toolchain setting was changed.
- Complete regression: 497 passed, 7 platform-dependent skips, zero failures.
- Backend TypeScript and frontend production builds passed. The existing frontend bundle-size advisory remains.
- `scripts/verify-production-dependencies.cjs` exercises real SQLite reads, Sequelize transactions and updates, protobuf serialization, a local gRPC request, a local SockJS/WebSocket exchange, the Undici ProxyAgent API, JWT signature verification and Nodemailer message generation. Mail uses an in-memory transport; it does not contact recipients or an SMTP service.
## Release verification
The `Release readiness` workflow builds the same four Dockerfile variants and all 24 variant/architecture combinations used by the release workflow, including both Python 3.10 variants. It uses the candidate source SHA and the candidate static artifact, then loads each amd64 image and runs the production dependency check inside it with external networking disabled.
A separate Linux job tests both precompiled SQLite installation and forced source compilation using the SQLite package's declared node-gyp, rather than pnpm 8's bundled older node-gyp. This distinction matters on Python 3.12 and newer.
The workflow does not push images, tags, npm packages, CDN metadata or static mirrors. It runs when a pull request introduces or changes the readiness workflow; subsequent candidates can invoke it manually. A successful run is required evidence for this dependency update, not evidence that a formal version has already been published.
Linux source compilation and the complete image matrix still require CI confirmation; results will be recorded after the candidate run completes.
+31
View File
@@ -0,0 +1,31 @@
# 发布运行时支持
2026-10-10 的依赖验收区分 npm 锁文件审计与镜像运行时维护。生产 npm 审计零公告命中不能证明 Node、Python、操作系统或用户脚本依赖没有漏洞。
默认 Debian 镜像使用 Node 22.23.3、Python 3.11.17 和 Debian 13(Trixie),保留 amd64、arm/v7、arm64、ppc64le、s390x 五种架构。Node 22 官方 Bookworm 镜像缺少 s390x,因此在原生构建阶段下载官方目标架构发布包,只将目标 Node 二进制复制到 Python 阶段;npm 等工具来自同源码 CI 构建的受控归档。不要将下载阶段自身的原生 Node 或附带的旧工具树复制到目标镜像。
`docker/install-node.sh` 使用仓库中固定的 Node 22.23.3 签名摘要清单和官方发布者公钥,通过 GPG 检查签名、发布者指纹和下载包 SHA256。升级 Node 时一起更新版本、签名清单和发布者公钥,并重新核验全部架构。arm/v7 需要 `libatomic1`。已读取五种实际 Node ELF:GLIBC 需求不超过 2.28,arm/v7 的 GLIBCXX 需求不超过 3.4.26;Trixie 满足这些要求,实际构建仍由完整镜像矩阵验证。
默认 Alpine 镜像固定 Python 3.11.17 / Alpine 3.24 基础镜像摘要,保留原有七种架构;运行时 Node 由 Alpine APK 提供。2026-10-10 的七架构 APK 索引均为 Node 24.18.1-r0。原生包安装阶段使用 Node 22.23.3,替换已结束维护的 Node 18。源码安装要求 Node 22.x >=22.22.2 或 Node 24.x >=24.15.0,CI 在 Node 22 和 24 上验证回归;镜像验证同时输出实际 Node、Python 和机器架构。
Python 3.10 已于 2026-10-01 结束维护。两个兼容 Dockerfile 仍单独验证,继续发布还是停止新增兼容标签需在正式发版前明确;不能将这些兼容镜像描述为受支持的默认运行时。已有历史镜像不会因这次准备工作删除。
运行时变更后必须重新执行镜像矩阵,并在使用实际新 Node 的独立 dev 容器中重新做至少两小时功能验证。原 Node 20 环境中的依赖长跑单独保留,不能替代新运行时结果。镜像准备与 CI 构建不会自动执行正式发布。
镜像中的全局 pnpm、PM2、npm 等工具不属于应用 `pnpm-lock.yaml` 的生产树。a70ec66 候选的实际全局工具补审发现 High/Critical 公告匹配。本轮收尾使用固定来源重构 npm 12.2.0 和 pnpm 10.34.6,更新实际 node-gyp 12.4.0、tar 7.5.22、PM2 的 js-yaml 4.3.2 等间接依赖;源码锁迁移到 PNPM10 格式,不改变已锁定的版本、依赖边或 peer 上下文。只供前端构建的 plots 移到开发依赖,应用生产树从 367 个减到 282 个(按 PNPM10 审计口径)。
受控工具归档是 Linux CI 生成的纯 JavaScript 产物。静态构建清单覆盖归档及 proof,proof 记录源码提交、所有构建输入 Git blob 与 SHA256、实际工具文件和安全相对链接。安装器只用 Node 标准库,先验收再原子替换已知工具,失败回退;不解析浮动 registry 版本,也不改变持久化 `global/5` 脚本依赖。启动、修复、源码升级共用这套安装路径。普通项目安装保留 PNPM10 的脚本批准规则,全局用户脚本依赖维持原有原生安装行为。
实际工具树的本机审计仍有 **一项 High:braces 3.0.3,GHSA-vfj7-8cjw-p6xm**,当前没有已发布修复版本。默认 PM2 watch 关闭;四个正常安装、冻结离线安装、workspace 和 pack 场景的 V8 覆盖未执行其 parse/compile/expand,正向控制确认覆盖有效。这仅限定这些已测场景,不证明所有命令不可达;用户启用 watch、维护 glob 或调用 Shell 模式仍属于相关输入面。[公告](https://github.com/advisories/GHSA-vfj7-8cjw-p6xm)。最终 Linux CI、镜像实际树和 dev 维护行为仍须验收,不宣称全局依赖漏洞为零。
主 npm 包按单一冻结 PNPM 图生成 `bundleDependencies`,记录实际发货源文件、每个依赖 manifest 和 Node 解析边。消费者修复只校验随包树并处理 SQLite,不重新解析 npm 依赖。npm 12 默认阻止依赖安装脚本;全局安装需精确批准 `npm install -g @whyour/qinglong@2.23.0 --allow-scripts=@whyour/qinglong@2.23.0`,项目安装在自己的 `package.json` 设置 `allowScripts`。实际 SQLite 编译由主包安装脚本转发。CI 必须独立验证 npm10 本地、npm12 本地/全局的预编译和源码编译六种组合,正式发布只消费通过验收的同一 TGZ。[npm12 脚本配置](https://docs.npmjs.com/cli/v12/using-npm/config/#allow-scripts)。
源码安装固定 pnpm 10.34.6 和 frozen lock。若升级暂存源码遇到旧 manager,只有同提交、同锁和全部输入哈希匹配的静态工具归档才可在临时 prefix 完成安装;无受控归档时明确失败,不再回退到 npm 无锁解析。旧 Node20/PNPM8 容器应拉取新镜像并重建;宿主机先升级 Node,再重新安装完整 npm 包。Termux 的无锁源码回退也不再支持。CLI 的远程 API 模式仍仅需 Node >=22.12。
依据:[Node 发布计划](https://github.com/nodejs/Release/blob/main/schedule.json)、[Node 22.23.3 发布与下载](https://nodejs.org/en/blog/release/v22.23.3)、[官方 Node 发布者公钥](https://github.com/nodejs/release-keys)、[Python 支持周期](https://devguide.python.org/versions/)、[Python 3.11.17 Trixie 镜像元数据](https://github.com/docker-library/repo-info/blob/master/repos/python/remote/3.11.17-slim-trixie.md)、[Python 3.11.17 Alpine 镜像元数据](https://github.com/docker-library/repo-info/blob/master/repos/python/remote/3.11.17-alpine3.24.md)。
## 前端构建运行时
后端继续在 Node 22 和 24 上验收。现有 Umi 4.4.4 构建依赖加载 spdy/http-deceiver,仍使用 Node 24 已移除的 `http_parser`。因此 `build:front` 通过 pnpm 单次命令参数固定使用 Node 22.23.3;这不会改变后端、全局 Node 或其他脚本的运行时。使用 `pnpm build:front` 或 `npm run build:front`,不要直接在 Node 24 下调用 `max build`。pnpm 会从官方 Node 发布站下载构建运行时,首次前端构建需要网络访问;前端构建不属于容器运行时离线验收阶段。Node 22/24 CI 均执行该构建入口,覆盖宿主 Node 24 的场景。此范围待前端工具链升级后再放宽。
相关设置:[pnpm 10 Node.js 运行时设置](https://github.com/pnpm/pnpm.io/blob/main/versioned_docs/version-10.x/settings.md#usenodeversion)。
+78
View File
@@ -0,0 +1,78 @@
# Task startup and notification loading
`task` performs shell configuration, status reporting and dependency-path discovery
before invoking the selected runtime. These costs are separate from cron callback
latency and time spent waiting for an execution slot.
## Dependency-path cache
Node scheduling usually starts in `/ql`, while Alpine crond starts in `/root`.
The pnpm discovery key includes the working directory, npm/pnpm configuration,
relevant environment and executable metadata. Each key now has its own cached
record, so alternating scheduler environments do not evict each other.
Records expire after 60 seconds. Refreshes share a persistent per-user lock;
records older than an hour are removed on refresh. Warm lookups do not scan the
cache directory. The lock is never removed while another process might use it.
The opt-out `QL_NODE_PATH_CACHE=0`, failed-discovery fallback, and package resolution
order remain available. This caches the dependency directory, not its contents.
## Language startup paths
| Invocation | Environment and before hooks | Automatic notification loading |
| --- | --- | --- |
| `task script.py` / `.pyc` | Python preload imports generated environment, runs the shell/command before hooks, imports `task_before.py`, then applies account selection | `QLAPI.notify(...)` imports `__ql_notify__` on first call |
| `task script.js` / `.mjs` / `.ts` | Node preload imports generated environment, runs shell/command before hooks, requires `task_before.js`, then applies account selection | `QLAPI.notify(...)` requires `__ql_notify__.js` on first call |
| `task script.sh` | Shell sources the generated environment and runs shell before hooks | No automatic Python/Node notification module |
| Explicit interpreter or other command, e.g. `task python3 script.py` | Retains generated environment and shell before-hook exports; it does not automatically install the Python/Node preload | No automatic notification module unless configured by the caller |
TypeScript uses `ts-node-transpile-only`; MJS uses the same Node preload plus the
ESM loader. There are no separate built-in Ruby, Go, Java or PHP preload modules.
All these task paths still pass through shared dependency-path setup: non-JS
scripts and their hooks may themselves call Node or use the QLAPI bridge.
The before hooks are deliberately still synchronous: they can export environment
variables or perform work required by the user's script. Python's preload still
starts a child shell/Python to capture that environment; Node's preload starts a
child shell/Node. Node also still loads its gRPC client eagerly. These are remaining
startup costs, not work performed by the cron library, and are not removed by this
change.
Shell scripts run in a child shell that inherits the wrapper's environment and
before-hook functions. This lets the wrapper run its after hooks and report the
exit status even when a script calls `exit`, replaces itself with `exec`, or fails
under `set -e`. The script's own EXIT trap still runs in the child. Changes made
inside the script (variables, working directory, functions and shell options)
do not propagate back to the wrapper or its after hooks.
## Lazy notifications
Scripts keep calling `QLAPI.notify` with the same arguments. When no notification
is sent, the notification provider and its channel dependencies are never imported
by the preload. First use loads the provider; subsequent uses reuse the standard
Python/Node module cache. JS forwards the receiver and return value, including
promises, and Python forwards positional and keyword arguments. Import/send errors
surface at the notification call rather than preventing unrelated scripts from
obtaining QLAPI during startup. A failed import can be retried after its cause is
resolved.
Provider initialization, including provider configuration reads, now happens at
first use. A script's explicit `import __ql_notify__`, `require(...)`, or independent
notification helper still loads that module immediately.
## Task timeouts
`task -m DURATION ...` also applies to sourced Shell scripts. The timeout worker
inherits before-hook variables, arrays and functions without serializing them.
At the deadline, the worker's process group receives TERM, followed by KILL after
0.5 seconds, so ordinary child processes cannot continue after the timeout.
Explicit interpreters and other commands use the same mechanism. Processes that
intentionally create a separate session/process group are outside this boundary.
The wrapper records status 124 and runs its after hook; concurrent account mode
retains its existing aggregate status behavior. Zero disables the timeout.
Cron executions waiting for a concurrency slot retain their schedule revision.
Disabling, deleting or replacing that revision prevents its queued executions
from starting when a slot becomes available. Already running executions retain
the existing stop behavior. Invalidated callbacks do not count against a new
revision's limit of five pending/running executions.
+97
View File
@@ -0,0 +1,97 @@
# 青龙生产依赖候选 dev 验收
> 此文档记录先前依赖迁移候选构建的验收。2.23.0 已采用最小生产依赖修复方案发布;完整 Node/pnpm 和依赖迁移仍留在草稿中,等待后续版本。历史提交与构建证据不能作为当前合并后提交的验收结果,需重新运行 CI。
**结论:全部完成审计通过。**
完成审计:57/57 项通过,0 项未通过。
## 候选与打包来源
- 源码:`5038d2d9a046b629dfe01e8162d48fe7143f5ef4`。
- dev 测试镜像:`sha256:141bdc3e3281cc22cd4bf407feb782fe969b92cde48f724c5e9d5bdc7078e8e1`;容器:`ql-dependency-soak-20261010`。
- 打包方法:`native-debian-runtime-repack`。本轮在既有原生 Debian runtime `sha256:9326cfda470960877b537463fc9158964356521586b58ebb1373a270fcfcd533`(基础源码 `68ffa106f80a2080ddd9b48affe0d7a7489e4d05`)上重新打包候选源码、全新冻结锁文件生产依赖包树及同 SHA 的 CI 静态产物。
- dev 镜像是本轮依赖功能验收用重打包产物;正式发行 Dockerfile 的完整四种镜像与多架构构建由下面的 CI 独立验证。
- 镜像内 Git HEAD、源/镜像 package.json 与锁文件哈希、全部静态文件哈希及实际依赖版本在开始、长跑期间、重启后和结束时复核。
运行环境:Node `20.20.2`,`Python 3.11.14`,`linux/x64`。静态产物复核 729 个文件;来源复核 7 次。
| 运行依赖 | 版本 |
|---|---|
| nodemailer | `10.0.16` |
| sequelize | `6.37.8` |
| @grpc/grpc-js | `1.14.6` |
| undici | `7.30.0` |
| multer | `2.4.0` |
| lodash | `4.18.1` |
| sqlite3 | `1.1.2` |
## 完整发布 CI
[Release readiness run 37965143236](https://github.com/whyour/qinglong/actions/runs/37965143236);源码 `5038d2d9a046b629dfe01e8162d48fe7143f5ef4`;结论 `success`。
| 镜像 | 架构数 | 构建 | 原生 amd64 协议检查 |
|---|---:|---|---|
| alpine | 7 | 通过 | 通过 |
| debian | 5 | 通过 | 通过 |
| alpine-python3.10 | 7 | 通过 | 通过 |
| debian-python3.10 | 5 | 通过 | 通过 |
留存 CI 证明记录 4 种镜像、24 个镜像/架构组合及 4 个通过的原生协议检查;该 readiness 流水线未推送正式镜像、版本标签、npm 包或 CDN。
## 至少两小时的实际功能运行
开始:2026-10-10 02:33:01 CST;结束:2026-10-10 04:35:30 CST。
有效运行 7324.418 秒;总墙钟 7348.53 秒;计划重启停机合计 24.112 秒已扣除。
健康/CLI 采样 578 次,失败 0 次;功能阶段 64 项,失败 0 项。
| 脚本语言 | 实际执行次数 | 执行时间跨度(秒) | 最大间隔(秒) |
|---|---:|---:|---:|
| JS | 733 | 7338.495 | 21.054 |
| PY | 727 | 7338.195 | 70.081 |
| SH | 469 | 6884.0 | 26.0 |
数据库实例记录:成功 1929 次,计划重启中断 0 次;中断的结束时间和未知退出码另行断言。
覆盖秒级任务调度、三种语言执行、禁用/恢复/修改/删除、并发与排队失效、任务停止/失败、环境变量更新;Git/SSH/HTTP/raw 订阅、过滤/自动任务、间隔与定时拉取、共享目录删除保护;脚本上传/编辑/历史/hash 冲突/回滚/路径限制;Node/Python/Linux 依赖安装、执行、重装与卸载;认证、应用 scope/secret、TOTP 和备份导入。
| 预检套件 | 真实断言 | 失败 |
|---|---:|---:|
| full | 18 | 0 |
| other | 15 | 0 |
| advanced-retry | 3 | 0 |
| queue | 4 | 0 |
初始化/安全/历史/CLI 断言:26 项。API 依赖专项:8 轮、129 项,失败 0 项;包括真实请求解析、gzip、反向代理 IP、上传临时文件和授权 SockJS XHR/WebSocket。
## 通知与真实邮件收件
Nodemailer `10.0.16`:通知 8 轮、224 项;真实收到邮件 164 封,其中 TLS 124 封。
传输模式:`{"tls": 82, "plain": 40, "starttls": 42}`;TLS 协议:`{"TLSv1.3": 124}`;认证机制:`{"PLAIN": 148, "LOGIN": 16}`。
预期错误反馈:认证拒绝 8 次,收件人拒绝 8 次;夹具内部错误 0 次。
本地 HTTP 通知接收 112 次;认证 CONNECT 代理 16 次。
覆盖面板 Plain/LOGIN/STARTTLS/SSL、保存配置、错误配置不覆盖原设置、空收件人回退、官方 JS/Python 样例、gRPC、UTF-8 正文/附件、四种 webhook body、gotify、ntfy 和 Telegram HTTP 代理。全部收件及 HTTP 请求由容器 localhost 夹具接收。报告只保留计数和协议摘要。
真实任务队列通知:1 轮;JS/Python 各执行官方 sample 和 QLAPI,SMTP 收件证据 4 封;实例退出码、任务日志 SHA、TLS/认证及原始收件 SHA 单独核对。任务和脚本清理错误 0 项。
## 资源与原有服务
资源采样 104 次;最高内存比例 23.73%(486.1MiB / 2GiB);最高 CPU 126.66%。
| 原有容器 | ID保持 | StartedAt保持 | 运行状态保持 |
|---|---|---|---|
| demo | 是 | 是 | 是 |
| ql | 是 | 是 | 是 |
## 完成审计与剩余事项
所有明确验收项及来源/时长/失败/服务保护检查通过。正式发版和公告仍按发布流程执行。
早期预检记录:fixture connection reuse failure; targeted fresh-agent export+upload passed。备份夹具首次连接失败记录保存于 `failed-preflight-1`;随后使用独立新 Undici Agent 的真实导出和上传均通过,没有自动重试或修改候选产品代码。首次异常未采集 cause;服务保持健康时闭合连接的 UND_ERR_SOCKET 复现提供时序佐证。
夹具修正状态:`True`;原始失败断言 1 项;留存证明文件 3 份。
原长跑窗口 `hour-attempt1` 独立留存:失败阶段 `[{"phase": "dependency-integration", "status": "fail", "errorType": "RuntimeError"}]`,DI 失败断言 `["actual SockJS XHR polling opens and exchanges authenticated messages"]`;归档原文件哈希复核 21/21。旧 SMTP journal 记录 43 封邮件,仅计入旧窗口。
首次 SockJS XHR 接收帧同时包含订阅广播和正确的本次 echo,原夹具对整个帧作单消息比较而失败。修正后只识别合法 a 帧内的唯一测试 marker,同时继续拒绝 marker 缺失、改变或重复。旧周期记录与邮件 journal 已归档,新窗口重新执行 DI/通知预检和任务通知;最终计时与当前周期统计不合并旧窗口。
原始 owner 凭据、私钥、数据库、邮件正文及原始日志没有进入本报告或安全汇总。仅在主代理保存本报告、完成审计和安全证据至另一目录并审阅 cleanup-owned.py 后,可决定清理本轮容器/镜像和测试私密数据。

Some files were not shown because too many files have changed in this diff Show More