Compare commits

...
Author SHA1 Message Date
whyour 0aa521a288 fix(python): integrate venv startup with source dependency support 2026-10-13 02:24:12 +08:00
whyour 1ac04a8062 fix(python): keep source dependency installs and removals in selected venv 2026-10-13 02:21:10 +08:00
whyour db88d037db fix(python): integrate latest develop and support older venv pip 2026-10-13 02:20:59 +08:00
whyour be877cfddc fix: 修复完整镜像源地址及 CRLF 任务导入 (#3093)
* fix: address six verified legacy issue regressions

* fix: limit issue patch to mirror URLs and CRLF imports

* fix: preserve APT formats and third-party repositories
2026-10-13 01:11:35 +08:00
Copilotwhyourcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>whyour
fdeffebba8 Fix catch-all route serving HTML for API requests (#2885)
* Initial plan

* Fix env.js loading error by preventing catch-all from matching API routes

Co-authored-by: whyour <22700758+whyour@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: whyour <22700758+whyour@users.noreply.github.com>
Co-authored-by: whyour <imwhyour@gmail.com>
2026-10-13 01:07:50 +08:00
dependabot[bot]dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>whyour
3c61e933a1 chore(deps): bump the npm_and_yarn group across 1 directory with 8 updates (#3075)
* chore(deps): bump the npm_and_yarn group across 1 directory with 8 updates

Bumps the npm_and_yarn group with 8 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [body-parser](https://github.com/expressjs/body-parser) | `1.20.3` | `1.20.6` |
| [http-proxy-middleware](https://github.com/chimurai/http-proxy-middleware) | `3.0.3` | `3.0.7` |
| [js-yaml](https://github.com/nodeca/js-yaml) | `4.1.0` | `4.3.2` |
| [multer](https://github.com/expressjs/multer) | `2.3.0` | `2.4.0` |
| [nodemailer](https://github.com/nodemailer/nodemailer) | `8.0.1` | `10.0.9` |
| [undici](https://github.com/nodejs/undici) | `7.9.0` | `7.29.1` |
| [axios](https://github.com/axios/axios) | `1.7.8` | `1.20.0` |
| [brace-expansion](https://github.com/juliangruber/brace-expansion) | `1.1.12` | `1.1.21` |



Updates `body-parser` from 1.20.3 to 1.20.6
- [Release notes](https://github.com/expressjs/body-parser/releases)
- [Changelog](https://github.com/expressjs/body-parser/blob/master/HISTORY.md)
- [Commits](https://github.com/expressjs/body-parser/compare/1.20.3...1.20.6)

Updates `http-proxy-middleware` from 3.0.3 to 3.0.7
- [Release notes](https://github.com/chimurai/http-proxy-middleware/releases)
- [Changelog](https://github.com/chimurai/http-proxy-middleware/blob/v3.0.7/CHANGELOG.md)
- [Commits](https://github.com/chimurai/http-proxy-middleware/compare/v3.0.3...v3.0.7)

Updates `js-yaml` from 4.1.0 to 4.3.2
- [Changelog](https://github.com/nodeca/js-yaml/blob/4.3.2/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/compare/4.1.0...4.3.2)

Updates `multer` from 2.3.0 to 2.4.0
- [Release notes](https://github.com/expressjs/multer/releases)
- [Changelog](https://github.com/expressjs/multer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/expressjs/multer/compare/v2.3.0...v2.4.0)

Updates `nodemailer` from 8.0.1 to 10.0.9
- [Release notes](https://github.com/nodemailer/nodemailer/releases)
- [Changelog](https://github.com/nodemailer/nodemailer/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodemailer/nodemailer/compare/v8.0.1...v10.0.9)

Updates `undici` from 7.9.0 to 7.29.1
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v7.9.0...v7.29.1)

Updates `axios` from 1.7.8 to 1.20.0
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v1.7.8...v1.20.0)

Updates `brace-expansion` from 1.1.12 to 1.1.21
- [Release notes](https://github.com/juliangruber/brace-expansion/releases)
- [Commits](https://github.com/juliangruber/brace-expansion/compare/v1.1.12...v1.1.21)

---
updated-dependencies:
- dependency-name: axios
  dependency-version: 1.18.0
  dependency-type: direct:development
- dependency-name: body-parser
  dependency-version: 1.20.6
  dependency-type: direct:production
- dependency-name: brace-expansion
  dependency-version: 1.1.21
  dependency-type: indirect
- dependency-name: http-proxy-middleware
  dependency-version: 3.0.7
  dependency-type: direct:production
- dependency-name: js-yaml
  dependency-version: 4.3.2
  dependency-type: direct:production
- dependency-name: multer
  dependency-version: 2.3.0
  dependency-type: direct:production
- dependency-name: nodemailer
  dependency-version: 9.1.1
  dependency-type: direct:production
- dependency-name: undici
  dependency-version: 7.29.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix(ci): verify the updated multer runtime version

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: whyour <imwhyour@gmail.com>
2026-10-13 01:06:29 +08:00
whyour cf236f8324 fix(dependencies): require existing sources and translate install hints 2026-10-12 01:13:53 +08:00
whyour 4cd6c6ad2e fix(dependencies): quote and validate Python source installs 2026-10-12 00:23:48 +08:00
whyour 81a60254a2 fix(python): share safe startup and honor explicit virtual environments 2026-10-12 00:23:20 +08:00
whyour 70636900b2 chore: finalize 2.23.0 publication metadata 2026-10-11 12:27:51 +08:00
whyour 331f4f9138 Prepare Qinglong 2.23.0 and CLI 0.1.2 release (#3090)
* fix: patch reachable release-blocking dependencies

* docs: keep release risk notes in PR description

* chore: prepare Qinglong 2.23.0 and CLI 0.1.2 release metadata
2026-10-11 12:24:50 +08:00
whyour b06cb72334 fix: apply minimal production dependency security patches 2026-10-11 02:09:20 +08:00
whyour 89c6355635 fix: renew system tokens and serialize authentication cache updates 2026-10-11 01:56:50 +08:00
whyour 68ffa106f8 fix: resolve Debian regression findings (#3087)
* fix: resolve Debian regression findings

* fix: reject failed APT index refreshes and simplify SSH home lookup
2026-10-09 01:41:29 +08:00
whyour 9d5f58ef99 ci: verify Gitee Pull mirrors with resilient GitHub notifications
Use native Gitee Pull mirrors for code and static artifacts, and verify branch and tag SHAs alongside image builds. Trigger stale mirrors through small GitHub REST ref updates, retry transient errors with backoff, and keep polling when a notification fails.
2026-10-08 01:16:43 +08:00
whyour ef1c54a620 fix: 补齐前端缺失的中英文翻译 2026-10-07 20:35:59 +08:00
whyour 0218404d9f fix: 修复页面升级、ql update 与静态资源版本同步 (#3086)
* fix: validate panel upgrades and synchronize versioned static builds

* fix: add static-only mirror backfill workflow

* fix: avoid lifecycle self-kill and stop cancelled upgrade groups

* fix: build images when the shared upgrade engine changes

* fix: dispatch upgrade lifecycle through a fixed script

* fix: reclaim upgrade locks safely and normalize health probes

* fix: match backend base URL normalization exactly
2026-10-07 17:43:03 +08:00
whyour 24f7d1a46f fix: resolve dev panel and CLI regressions
Synchronize scheduler concurrency across API and gRPC processes and restore persisted limits during recovery.

Serialize subscription mutations and preserve shared storage while cleaning Shell and CLI namespaces, historical references, and filesystem-specific naming candidates.

Handle CLI text responses correctly and include script history in backups. Add regression coverage for concurrency, cleanup ownership, long URLs, Unicode paths, and history exports.
2026-10-07 12:35:10 +08:00
whyour fa8910b3d6 feat: add script version history (#3085)
* feat: add script version history

* fix: validate canonical save-as paths before backup

* fix: stack script history footer on small screens
2026-10-06 22:00:11 +08:00
whyour 2cab112391 fix: 加固安全输入边界并修复订阅兼容性 (#3084)
* fix: harden scoped API inputs and isolate login throttling

* fix: close script path gaps and harden security boundaries

* fix: preserve subscription aliases and avoid extension eval

* fix: allow cancelling invalid historical subscriptions

* fix: make script parent validation explicit for CodeQL

* fix: limit command execution and clarify trusted path branches

* fix: separate diff tool script access from config APIs

* fix: clean up obsolete subscription SSH credentials
2026-10-06 18:21:54 +08:00
whyour 2ab9c702e6 ci: skip Docker builds for CLI-only changes 2026-10-01 12:25:16 +08:00
whyour 6dd98a8565 fix(ci): isolate CLI release concurrency from validation 2026-10-01 12:17:26 +08:00
whyour 06c8d8b112 fix: report resource conflicts and accurate CLI API errors 2026-10-01 11:26:40 +08:00
whyour a5406bedca ci: allow verified CLI prereleases from develop 2026-10-01 00:26:42 +08:00
whyour 3d2b98a442 fix: scope file access restrictions and honor custom data directories 2026-09-30 21:21:35 +08:00
whyour 93d7ec69dc fix: scope config file blacklist to config directory (#3082) 2026-09-30 20:31:20 +08:00
whyour a303e33b4e fix: invalidate queued cron revisions and enforce task process timeouts 2026-09-29 23:37:00 +08:00
whyour 44129ca088 fix: align CLI shell completion and cancellation semantics 2026-09-29 01:06:47 +08:00
whyour def4917447 fix: finalize exited shell tasks and preserve command environments 2026-09-29 00:52:33 +08:00
whyour bc0f35e3f7 fix: reduce task startup overhead and lazy-load notifications (#3080) 2026-09-28 23:41:31 +08:00
whyour f168efcdae refactor: migrate cron scheduling to node-cron (#3079)
* refactor: migrate cron scheduling to node-cron

* feat: support annually midnight and minutely cron macros

* fix: harden cron recovery and system scheduler compatibility
2026-09-27 22:18:39 +08:00
jiandanc 0af546cd1d feat: 支持 Python venv 隔离环境,适配 Node.js 原生部署
## 背景
原方案使用系统 Python + pip --prefix 安装依赖到 data/dep_cache/python3,
系统 Python 环境被污染,且无法隔离不同项目的依赖。

## 改动内容

### 核心:Python venv 支持
- shell/start.sh: 使用 python3 -m venv 创建虚拟环境,替代 --prefix 模式
  - 从 .env 文件读取 PYTHON_VENV_DIR 配置
  - 首次启动自动创建 .venv,已存在则跳过
  - 仅将 .venv/bin 加入 PATH 优先级,不设置 PYTHONHOME/PYTHONPATH(避免破坏 venv 机制)
  - .env 不再强制覆盖,仅首次从 .env.example 复制
  - 启动时自动修复 task/ql 软链接指向当前部署目录

### 后端适配
- back/config/const.ts: 新增 PYTHON_VENV_DIR 常量
- back/config/util.ts: venv 模式下跳过 pip --prefix,直接使用 venv 的 pip3

### 开发模式支持
- shell/dev-env.sh: pnpm start 时自动 source,将 .venv/bin 加入 PATH
- package.json: start:back 加入 source dev-env.sh

### 新增文件
- shell/start-simplify.sh: 精简版启动脚本(跳过系统依赖安装,适用于已预装环境的服务器)
- README-NODE.md: Node.js 原生部署完整文档

### 配置
- .env.example: 新增 PYTHON_VENV_DIR=./.venv 配置项(默认注释状态)

## 兼容性
- Docker 模式不受影响(使用独立的 docker-entrypoint.sh,走原有 --prefix 逻辑)
- 未配置 PYTHON_VENV_DIR 时默认使用系统 Python(向后兼容)
2026-06-07 21:03:51 +08:00
copilot-swe-agent[bot]andwhyour fcf0ee619b Improve URL pattern matching and remove redundant checks
Co-authored-by: whyour <22700758+whyour@users.noreply.github.com>
2025-12-27 08:51:12 +00:00
copilot-swe-agent[bot]andwhyour f3ec352066 Address code review feedback: improve pattern matching and error handling
Co-authored-by: whyour <22700758+whyour@users.noreply.github.com>
2025-12-27 08:49:41 +00:00
copilot-swe-agent[bot]andwhyour 023d0f1cc4 Fix pyproject.toml path handling and add error handling
Co-authored-by: whyour <22700758+whyour@users.noreply.github.com>
2025-12-27 08:46:12 +00:00
copilot-swe-agent[bot]andwhyour f526d3f972 Add support for GitHub URLs and requirements files in Python dependencies
Co-authored-by: whyour <22700758+whyour@users.noreply.github.com>
2025-12-27 08:41:43 +00:00
copilot-swe-agent[bot] e28f746294 Initial plan 2025-12-27 08:35:03 +00:00
167 changed files with 15879 additions and 1343 deletions
+3
View File
@@ -21,3 +21,6 @@ JWT_EXPIRES_IN=
QINIU_AK=
QINIU_SK=
QINIU_SCOPE=
# Python 虚拟环境路径,留空则使用系统 Python
# PYTHON_VENV_DIR=./.venv
+109 -70
View File
@@ -2,23 +2,44 @@ name: Build And Push Docker Image
on:
push:
paths-ignore:
- "*.md"
paths:
- "**"
- "!*.md"
# CLI is verified separately; its shared Bash upgrade engine ships in images.
- "!cli/**"
- "cli/src/internal/maintenance/upgradeArtifacts.cjs"
- "!.github/workflows/cli-package.yml"
branches:
- "master"
- "develop"
tags:
- "v*"
workflow_dispatch:
inputs:
static_sync_only:
description: Synchronize an existing static snapshot to GitLab and verify Gitee
type: boolean
default: false
static_ref:
description: Static branch or version name, for example v2.22.0
type: string
default: ''
static_ref_type:
description: Synchronize a branch, or a version branch and Git tag
type: choice
options: [branch, tag]
default: branch
permissions:
contents: read
jobs:
validate:
if: ${{ !inputs.static_sync_only }}
uses: ./.github/workflows/validate.yml
code_gitlab:
if: ${{ !inputs.static_sync_only }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
@@ -35,37 +56,8 @@ jobs:
git push --force --all gitlab 2>&1 || echo "::warning::GitLab push failed"
git push --force --tags gitlab 2>&1 || echo "::warning::GitLab tags push failed"
code_gitee:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Push to Gitee
env:
GITEE_TOKEN: ${{ secrets.GITEE_TOKEN }}
run: |
set +e
mkdir -p ~/.ssh
printf '%s\n' "${{ secrets.GITLAB_SSH_PK }}" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan -T 10 gitee.com >> ~/.ssh/known_hosts 2>/dev/null
git remote add gitee git@gitee.com:whyour/qinglong.git 2>/dev/null
if git push --force --all gitee 2>&1; then
echo "::notice::Gitee push --all succeeded"
else
echo "::warning::Push failed, trying to create repo via API..."
curl -sS --connect-timeout 30 --max-time 60 \
-X POST "https://gitee.com/api/v5/user/repos" \
-H "Content-Type: application/json" \
-d '{"name":"qinglong","private":"false"}' \
"?access_token=$GITEE_TOKEN" 2>/dev/null
git push --force --all gitee 2>&1 && echo "::notice::Gitee push succeeded after repo creation" || echo "::warning::Gitee push failed after retry"
fi
git push --force --tags gitee 2>&1 || echo "::warning::Gitee tags push failed"
build-static:
if: ${{ !inputs.static_sync_only }}
needs: validate
runs-on: ubuntu-latest
steps:
@@ -98,62 +90,109 @@ jobs:
env:
GITHUB_REPO: github.com/${{ github.repository_owner }}/qinglong-static
GITHUB_BRANCH: ${{ github.ref_name }}
GITHUB_REF_TYPE: ${{ github.ref_type }}
run: |
mkdir -p tmp
cd ./tmp
cp -rf ../static/* ./
git init -b ${GITHUB_BRANCH} && git add .
git config --local user.name 'github-actions[bot]'
git config --local user.email 'github-actions[bot]@users.noreply.github.com'
git commit --allow-empty -m "copy static at $(date +'%Y-%m-%d %H:%M:%S')"
git push --force --quiet "https://${{ secrets.API_TOKEN }}@${GITHUB_REPO}.git" ${GITHUB_BRANCH}:${GITHUB_BRANCH}
bash scripts/publish-static.sh "https://${{ secrets.API_TOKEN }}@${GITHUB_REPO}.git" "$GITHUB_BRANCH" "$GITHUB_REF_TYPE"
static_gitlab:
needs: build-static
runs-on: ubuntu-latest
concurrency:
group: static-gitlab-${{ github.ref_name }}
cancel-in-progress: false
steps:
- uses: actions/checkout@v6
- name: Push qinglong-static to GitLab
env:
STATIC_BRANCH: ${{ github.ref_name }}
STATIC_REF_TYPE: ${{ github.ref_type }}
MIRROR_SSH_KEY: ${{ secrets.GITLAB_SSH_PK }}
run: |
set +e
set -euo pipefail
mkdir -p ~/.ssh
printf '%s\n' "${{ secrets.GITLAB_SSH_PK }}" > ~/.ssh/id_rsa
printf '%s\n' "$MIRROR_SSH_KEY" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan -T 10 gitlab.com >> ~/.ssh/known_hosts 2>/dev/null
git clone --depth=1 --single-branch https://github.com/whyour/qinglong-static.git static-mirror
cd static-mirror
git remote add gitlab git@gitlab.com:whyour/qinglong-static.git 2>/dev/null
git push --force --all gitlab 2>&1 || echo "::warning::GitLab static push failed"
git push --force --tags gitlab 2>&1 || echo "::warning::GitLab static tags push failed"
bash scripts/sync-static.sh git@gitlab.com:whyour/qinglong-static.git "$STATIC_BRANCH" "$STATIC_REF_TYPE"
static_gitee:
verify-gitee-mirrors:
needs: build-static
runs-on: ubuntu-latest
timeout-minutes: 14
permissions:
contents: write
strategy:
fail-fast: false
matrix:
repository: [qinglong, qinglong-static]
steps:
- name: Push qinglong-static to Gitee
- uses: actions/checkout@v6
# Gitee Pull mirrors and GitHub push webhooks own synchronization.
# Verification runs alongside Docker builds and needs no Gitee credentials.
- name: Wait for Gitee Pull mirror
if: ${{ matrix.repository == 'qinglong' }}
env:
GITEE_TOKEN: ${{ secrets.GITEE_TOKEN }}
GH_TOKEN: ${{ github.token }}
QL_MIRROR_NOTIFY_REPO: whyour/qinglong
MIRROR_REPOSITORY: ${{ matrix.repository }}
MIRROR_REF: ${{ github.ref_name }}
MIRROR_REF_TYPE: ${{ github.ref_type }}
QL_MIRROR_INCLUDE_VERSION_BRANCH: ${{ matrix.repository == 'qinglong-static' }}
QL_MIRROR_NOTIFY_ON_START: 'false'
run: |
set +e
mkdir -p ~/.ssh
printf '%s\n' "${{ secrets.GITLAB_SSH_PK }}" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan -T 10 gitee.com >> ~/.ssh/known_hosts 2>/dev/null
git clone --depth=1 --single-branch https://github.com/whyour/qinglong-static.git static-mirror
cd static-mirror
git remote add gitee git@gitee.com:whyour/qinglong-static.git 2>/dev/null
bash scripts/wait-for-mirror.sh "https://github.com/whyour/$MIRROR_REPOSITORY.git" "https://gitee.com/whyour/$MIRROR_REPOSITORY.git" "$MIRROR_REF" "$MIRROR_REF_TYPE"
- name: Notify and wait for Gitee static mirror
if: ${{ matrix.repository == 'qinglong-static' }}
env:
GH_TOKEN: ${{ secrets.API_TOKEN }}
QL_MIRROR_NOTIFY_REPO: whyour/qinglong-static
MIRROR_REF: ${{ github.ref_name }}
MIRROR_REF_TYPE: ${{ github.ref_type }}
QL_MIRROR_INCLUDE_VERSION_BRANCH: 'true'
run: |
bash scripts/wait-for-mirror.sh https://github.com/whyour/qinglong-static.git https://gitee.com/whyour/qinglong-static.git "$MIRROR_REF" "$MIRROR_REF_TYPE"
if git push --force --all gitee 2>&1; then
echo "::notice::Gitee static push succeeded"
else
echo "::warning::Push failed, trying to create repo via API..."
curl -sS --connect-timeout 30 --max-time 60 \
-X POST "https://gitee.com/api/v5/user/repos" \
-H "Content-Type: application/json" \
-d '{"name":"qinglong-static","private":"false"}' \
"?access_token=$GITEE_TOKEN" 2>/dev/null
git push --force --all gitee 2>&1 && echo "::notice::Gitee static push succeeded after repo creation" || echo "::warning::Gitee static push failed after retry"
fi
git push --force --tags gitee 2>&1 || echo "::warning::Gitee static tags push failed"
sync-existing-static:
if: ${{ github.event_name == 'workflow_dispatch' && inputs.static_sync_only }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
mirror: [gitlab]
concurrency:
group: static-${{ matrix.mirror }}-${{ inputs.static_ref }}
cancel-in-progress: false
steps:
- uses: actions/checkout@v6
- name: Synchronize existing static snapshot
env:
STATIC_REF: ${{ inputs.static_ref }}
STATIC_REF_TYPE: ${{ inputs.static_ref_type }}
MIRROR_HOST: ${{ matrix.mirror }}.com
MIRROR_SSH_KEY: ${{ secrets.GITLAB_SSH_PK }}
run: |
set -euo pipefail
mkdir -p ~/.ssh
printf '%s\n' "$MIRROR_SSH_KEY" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan -T 10 "$MIRROR_HOST" >> ~/.ssh/known_hosts 2>/dev/null
bash scripts/sync-static.sh "git@$MIRROR_HOST:whyour/qinglong-static.git" "$STATIC_REF" "$STATIC_REF_TYPE"
verify-existing-static-gitee:
if: ${{ github.event_name == 'workflow_dispatch' && inputs.static_sync_only }}
runs-on: ubuntu-latest
timeout-minutes: 14
steps:
- uses: actions/checkout@v6
- name: Wait for existing Gitee static snapshot
env:
GH_TOKEN: ${{ secrets.API_TOKEN }}
QL_MIRROR_NOTIFY_REPO: whyour/qinglong-static
MIRROR_REF: ${{ inputs.static_ref }}
MIRROR_REF_TYPE: ${{ inputs.static_ref_type }}
QL_MIRROR_INCLUDE_VERSION_BRANCH: 'true'
run: |
bash scripts/wait-for-mirror.sh https://github.com/whyour/qinglong-static.git https://gitee.com/whyour/qinglong-static.git "$MIRROR_REF" "$MIRROR_REF_TYPE"
build-alpine:
if: ${{ !startsWith(github.ref, 'refs/tags/') }}
+54 -8
View File
@@ -21,16 +21,27 @@ on:
workflow_dispatch:
inputs:
publish:
description: Publish the verified CLI package (master only)
description: Publish the verified CLI package (master stable / develop prerelease)
type: boolean
default: false
prerelease_version:
description: Develop only, e.g. 0.1.2-beta.0 (alpha/beta/rc)
type: string
default: ''
permissions:
contents: read
concurrency:
group: cli-package-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}
# Keep validation runs separate from releases; never interrupt a running release.
# Master push releases share the same group as manually requested releases.
group: >-
cli-package-${{ github.workflow }}-${{ github.ref }}-${{
((github.event_name == 'workflow_dispatch' && inputs.publish) ||
(github.event_name == 'push' && github.ref == 'refs/heads/master')) && 'publish' || 'check' }}
cancel-in-progress: >-
${{ github.ref != 'refs/heads/master' &&
!(github.event_name == 'workflow_dispatch' && inputs.publish) }}
jobs:
package:
@@ -52,6 +63,20 @@ jobs:
sudo apt-get update
sudo apt-get install -y --no-install-recommends bash ca-certificates curl git jq perl procps python3 unzip zip
npm install --global --ignore-scripts --no-audit --no-fund ts-node@10.9.2 typescript@5.2.2
- name: Set requested prerelease version before verification
if: github.event_name == 'workflow_dispatch' && inputs.prerelease_version != ''
working-directory: cli
env:
PRERELEASE_VERSION: ${{ inputs.prerelease_version }}
shell: bash
run: |
set -euo pipefail
if [[ "$GITHUB_REF" != refs/heads/develop ||
! "$PRERELEASE_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+-(alpha|beta|rc)\.[0-9]+$ ]]; then
echo '::error::Prereleases require develop and an explicit alpha/beta/rc version.'
exit 1
fi
npm version "$PRERELEASE_VERSION" --no-git-tag-version --ignore-scripts
- run: npm ci --prefix cli --no-audit --no-fund
- run: npm run check:cli
- run: npm run test:cli
@@ -72,9 +97,11 @@ jobs:
needs: package
if: >-
github.repository == 'whyour/qinglong' &&
github.ref == 'refs/heads/master' &&
(github.event_name == 'push' ||
(github.event_name == 'workflow_dispatch' && inputs.publish))
((github.ref == 'refs/heads/master' &&
(github.event_name == 'push' ||
(github.event_name == 'workflow_dispatch' && inputs.publish))) ||
(github.ref == 'refs/heads/develop' && github.event_name == 'workflow_dispatch' &&
inputs.publish && inputs.prerelease_version != ''))
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
@@ -91,6 +118,8 @@ jobs:
name: qinglong-cli-${{ github.sha }}
path: cli-package
- name: Publish verified npm archive
env:
PRERELEASE_VERSION: ${{ inputs.prerelease_version }}
shell: bash
run: |
set -euo pipefail
@@ -103,13 +132,30 @@ jobs:
archive="${archives[0]}"
metadata=$(tar -xOf "$archive" package/package.json)
name=$(jq -er '.name | select(. == "@whyour/qinglong-cli")' <<< "$metadata")
version=$(jq -er '.version | select(type == "string" and test("^[0-9]+\\.[0-9]+\\.[0-9]+$"))' <<< "$metadata")
version=$(jq -er '.version | select(type == "string")' <<< "$metadata")
if [[ "$GITHUB_REF" == refs/heads/develop ]]; then
if [[ "$version" != "$PRERELEASE_VERSION" ||
! "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+-(alpha|beta|rc)\.[0-9]+$ ]]; then
echo '::error::Archive version does not match the requested prerelease.'
exit 1
fi
tag="${BASH_REMATCH[1]}"
elif [[ "$GITHUB_REF" == refs/heads/master && "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
tag=latest
else
echo '::error::Refusing to publish this branch/version combination.'
exit 1
fi
if npm view "$name@$version" version --json --registry=https://registry.npmjs.org > version.json 2> version-error.log; then
jq -e --arg version "$version" '. == $version' version.json > /dev/null
if [[ "$tag" != latest ]]; then
echo '::error::Prerelease version already exists; choose a new version to test this build.'
exit 1
fi
echo "::notice::$name@$version is already published; bump cli/package.json and its lockfile to release changes."
exit 0
elif ! jq -e '.error.code == "E404"' version.json > /dev/null; then
echo '::error::Could not check the published CLI version; refusing to publish.'
exit 1
fi
npm publish "./$archive" --access public --tag latest --ignore-scripts --registry=https://registry.npmjs.org
npm publish "./$archive" --access public --tag "$tag" --ignore-scripts --registry=https://registry.npmjs.org
+136
View File
@@ -0,0 +1,136 @@
name: Release readiness
on:
pull_request:
paths:
- '.github/workflows/release-readiness.yml'
- 'package.json'
- 'pnpm-lock.yaml'
workflow_dispatch:
permissions:
contents: read
concurrency:
group: release-readiness-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
regression:
uses: ./.github/workflows/validate.yml
static:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- uses: pnpm/action-setup@v6
with:
version: '8.3.1'
- uses: actions/setup-node@v6
with:
node-version: '20'
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: pnpm build:front && pnpm build:back && pnpm build:info
- uses: actions/upload-artifact@v6
with:
name: release-readiness-static
path: static/
include-hidden-files: true
if-no-files-found: error
images:
needs: [regression, static]
runs-on: ubuntu-22.04
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
include:
- image: alpine
file: docker/Dockerfile
cache: alpine
platforms: linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x,linux/386
- image: debian
file: docker/Dockerfile.debian
cache: debian
platforms: linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x
- image: alpine-python3.10
file: docker/Dockerfile.310
cache: alpine-python3.10
platforms: linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x,linux/386
- image: debian-python3.10
file: docker/Dockerfile.debian310
cache: debian-python3.10
platforms: linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Record source
id: source
env:
SOURCE_URL: ${{ github.event.pull_request.head.repo.clone_url || format('https://github.com/{0}.git', github.repository) }}
SOURCE_BRANCH: ${{ github.head_ref || github.ref_name }}
run: |
echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
echo "url=$SOURCE_URL" >> "$GITHUB_OUTPUT"
echo "branch=$SOURCE_BRANCH" >> "$GITHUB_OUTPUT"
- uses: actions/download-artifact@v7
with:
name: release-readiness-static
path: static/
- uses: docker/setup-qemu-action@v4
with:
cache-image: false
- uses: docker/setup-buildx-action@v4
- name: Build every release architecture
uses: docker/build-push-action@v7
with:
context: .
file: ${{ matrix.file }}
platforms: ${{ matrix.platforms }}
build-args: |
QL_URL=${{ steps.source.outputs.url }}
QL_BRANCH=${{ steps.source.outputs.branch }}
SOURCE_COMMIT=${{ steps.source.outputs.sha }}
push: false
provenance: false
sbom: false
cache-from: type=registry,ref=whyour/qinglong:cache-${{ matrix.cache }}
- name: Load native image for runtime verification
uses: docker/build-push-action@v7
with:
context: .
file: ${{ matrix.file }}
platforms: linux/amd64
build-args: |
QL_URL=${{ steps.source.outputs.url }}
QL_BRANCH=${{ steps.source.outputs.branch }}
SOURCE_COMMIT=${{ steps.source.outputs.sha }}
tags: qinglong-readiness:${{ matrix.image }}
load: true
push: false
provenance: false
sbom: false
- name: Verify native production runtime without external networking
run: |
docker run --rm --network none --entrypoint node \
qinglong-readiness:${{ matrix.image }} \
-e 'const assert=require("node:assert/strict"); const {createRequire}=require("node:module"); for(const [name,want] of Object.entries({"@grpc/grpc-js":"1.14.5",compression:"1.8.2",multer:"2.4.0",sequelize:"6.37.8"}))assert.equal(require(name+"/package.json").version,want); const expressRequire=createRequire(require.resolve("express")); assert.equal(expressRequire("proxy-addr/package.json").version,"2.0.8"); const sockRequire=createRequire(require.resolve("sockjs")); assert.equal(sockRequire("websocket-driver/package.json").version,"0.7.5"); require("sqlite3"); require("nodemailer"); require("undici"); console.log("Minimal production patches and runtime imports verified")'
- name: Export native Debian image for isolated dev validation
if: matrix.image == 'debian'
run: |
docker save qinglong-readiness:debian | gzip -1 > release-readiness-debian-image.tar.gz
sha256sum release-readiness-debian-image.tar.gz > release-readiness-debian-image.sha256
docker image inspect qinglong-readiness:debian > release-readiness-debian-image.json
- uses: actions/upload-artifact@v6
if: matrix.image == 'debian'
with:
name: release-readiness-debian-image
path: release-readiness-debian-image.*
compression-level: 0
if-no-files-found: error
+10
View File
@@ -0,0 +1,10 @@
# Node.js 原生部署与 Python venv
设置 `QL_DIR` 为安装目录,`QL_DATA_DIR` 为以 `/data` 结尾的绝对持久化数据路径。
如需隔离 Python,请在启动进程的环境中显式设置 `PYTHON_VENV_DIR`(绝对路径或相对 `QL_DIR` 的路径),例如 `export PYTHON_VENV_DIR=/srv/qinglong/.venv`。未设置时保留现有 Python prefix 行为。
- `qinglong`(`shell/start.sh`)安装系统依赖和 Node 工具,并安装 Python requests,然后调用公共启动脚本。
- `npm run start:simple`(`shell/start-simplify.sh`)使用已准备的 Node、pm2、nginx、jq 等依赖,不安装系统包、全局 npm 包或 requests;显式指定的 venv 不存在时仍会创建。请先在 venv 中安装 requests 和任务所需依赖。
- 两种启动方式均保留已有 `.env`;使用 venv 时清除继承的 `PYTHONPATH`/`PYTHONUSERBASE`,依赖查询、安装和卸载使用该 venv 的 Python。
该模式需具备现有启动流程所要求的 nginx 配置和 pm2 服务权限。生产环境设置变量应通过 systemd 的 `Environment` 或相应进程配置完成。已有系统 Python 依赖不自动迁移至 venv,需在 venv 中重新安装。
+3 -7
View File
@@ -4,7 +4,6 @@ import { Logger } from 'winston';
import config from '../config';
import * as fs from 'fs/promises';
import { celebrate, Joi } from 'celebrate';
import { join, basename } from 'path';
import { SAMPLE_FILES } from '../config/const';
import { t } from '../shared/i18n';
import ConfigService from '../services/config';
@@ -73,15 +72,12 @@ export default (app: Router) => {
const logger: Logger = Container.get('logger');
try {
const { name, content } = req.body;
// Resolve path first to prevent traversal attacks
let basePath = config.configPath;
if (name.startsWith('data/scripts/')) {
basePath = join(config.rootPath, 'data/scripts');
return res.send({ code: 403, message: t('文件无法访问') });
}
const cleanName = name.replace(/^data\/scripts\//, '');
const normalized = resolveFileAccess(
basePath,
[cleanName],
config.configPath,
[name],
config.blackFileList,
);
if (!normalized) {
+17 -14
View File
@@ -1,24 +1,16 @@
import { Joi, celebrate } from 'celebrate';
import { NextFunction, Request, Response, Router } from 'express';
import fs from 'fs';
import multer from 'multer';
import { Container } from 'typedi';
import { Logger } from 'winston';
import config from '../config';
import { safeJSONParse } from '../config/util';
import { t } from '../shared/i18n';
import EnvService from '../services/env';
const route = Router();
const storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, config.scriptPath);
},
filename: function (req, file, cb) {
cb(null, file.originalname);
},
const upload = multer({
storage: multer.memoryStorage(),
limits: { fileSize: 5 * 1024 * 1024, files: 1 },
});
const upload = multer({ storage: storage });
const labelSchema = Joi.array()
.items(Joi.string().trim().required())
.min(1)
@@ -282,12 +274,23 @@ export default (app: Router) => {
const logger: Logger = Container.get('logger');
try {
const envService = Container.get(EnvService);
const fileContent = await fs.promises.readFile(req!.file!.path, 'utf8');
const parseContent = safeJSONParse(fileContent);
let parseContent: unknown;
try {
parseContent = JSON.parse(req.file?.buffer.toString('utf8') || '');
} catch {
return res.status(400).send({ code: 400, message: t('参数错误') });
}
const data = Array.isArray(parseContent)
? parseContent
: [parseContent];
if (data.every((x) => x.name && x.value)) {
if (
data.length > 0 &&
data.every((x) =>
x && typeof x.name === 'string' &&
/^[a-zA-Z_][0-9a-zA-Z_]*$/.test(x.name) &&
typeof x.value === 'string' && x.value.length > 0,
)
) {
const result = await envService.create(
data.map((x) => ({
name: x.name,
+2
View File
@@ -3,6 +3,7 @@ import { Container } from 'typedi';
import OpenService from '../services/open';
import { Logger } from 'winston';
import { celebrate, Joi } from 'celebrate';
import { rateLimit } from 'express-rate-limit';
const route = Router();
export default (app: Router) => {
@@ -100,6 +101,7 @@ export default (app: Router) => {
route.get(
'/auth/token',
rateLimit({ windowMs: 15 * 60 * 1000, limit: 100 }),
celebrate({
query: {
client_id: Joi.string().required(),
+199 -14
View File
@@ -7,20 +7,62 @@ import { Logger } from 'winston';
import config from '../config';
import * as fs from 'fs/promises';
import { celebrate, Joi } from 'celebrate';
import path, { join, parse } from 'path';
import path, { dirname, join, parse, resolve, sep } from 'path';
import ScriptService from '../services/script';
import ScriptHistoryService from '../services/scriptHistory';
import { t } from '../shared/i18n';
import multer from 'multer';
import { writeFileWithLock } from '../shared/utils';
import {
ScriptHistoryError,
HistoryUnavailableError,
} from '../shared/scriptHistory';
const route = Router();
function isPathAllowed(targetPath: string): boolean {
const resolved = path.resolve(targetPath);
return config.writePathList.some((x) =>
Boolean(resolveFileAccess(x, [resolved], config.blackFileList)),
Boolean(
resolveFileAccess(
x,
[resolved],
// Panel configuration secrets must not restrict user script filenames.
path.resolve(x) === path.resolve(config.configPath)
? config.blackFileList
: [],
),
),
);
}
async function existingWritableRealPath(targetPath: string): Promise<string> {
if (!isPathAllowed(targetPath)) {
throw new ScriptHistoryError(403, '暂无权限');
}
const resolved = resolve(targetPath);
for (const writableRoot of config.writePathList) {
const root = resolve(writableRoot);
const rootPrefix = root.endsWith(sep) ? root : root + sep;
// Keep an explicit normalized boundary check at the filesystem operation.
if (!resolved.startsWith(rootPrefix)) continue;
const realPath = await fs.realpath(resolved);
const realRoot = await fs.realpath(root);
const realRootPrefix = realRoot.endsWith(sep) ? realRoot : realRoot + sep;
if (
!realPath.startsWith(realRootPrefix) ||
!resolveFileAccess(
realRoot,
[realPath],
root === resolve(config.configPath) ? config.blackFileList : [],
)
) {
throw new ScriptHistoryError(403, '暂无权限');
}
return realPath;
}
throw new ScriptHistoryError(403, '暂无权限');
}
const storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, config.tmpPath);
@@ -55,11 +97,7 @@ export default (app: Router) => {
];
if (req.query.path) {
if (
!resolveFileAccess(
config.scriptPath,
[req.query.path as string],
config.blackFileList,
)
!resolveFileAccess(config.scriptPath, [req.query.path as string])
) {
return res.send({ code: 403, message: t('暂无权限') });
}
@@ -115,6 +153,82 @@ export default (app: Router) => {
},
);
const historyError = (error: unknown, res: Response, next: NextFunction) => {
if (error instanceof ScriptHistoryError) {
return res.status(error.status).send({
code: error.status,
message: t(error.message),
...(error instanceof HistoryUnavailableError
? { historyUnavailable: true, currentHash: error.currentHash }
: {}),
});
}
return next(error);
};
const historyQuery = {
filename: Joi.string().required(),
path: Joi.string().optional().allow(''),
};
route.get(
'/history',
celebrate({ query: Joi.object(historyQuery).unknown(true) }),
async (req: Request, res: Response, next: NextFunction) => {
try {
const service = Container.get(ScriptHistoryService);
const data = await service.list(
(req.query.path as string) || '',
req.query.filename as string,
);
res.send({ code: 200, data });
} catch (error) {
historyError(error, res, next);
}
},
);
route.get(
'/history/detail',
celebrate({
query: Joi.object({
...historyQuery,
id: Joi.string().uuid().required(),
}).unknown(true),
}),
async (req: Request, res: Response, next: NextFunction) => {
try {
const service = Container.get(ScriptHistoryService);
const data = await service.detail(
(req.query.path as string) || '',
req.query.filename as string,
req.query.id as string,
);
res.send({ code: 200, data });
} catch (error) {
historyError(error, res, next);
}
},
);
route.put(
'/history/restore',
celebrate({
body: Joi.object({
...historyQuery,
id: Joi.string().uuid().required(),
expectedHash: Joi.string().hex().length(64).required(),
}),
}),
async (req: Request, res: Response, next: NextFunction) => {
try {
const { path = '', filename, id, expectedHash } = req.body;
const service = Container.get(ScriptHistoryService);
const data = await service.restore(path, filename, id, expectedHash);
res.send({ code: 200, data });
} catch (error) {
historyError(error, res, next);
}
},
);
route.get('/:file', (req: Request, res: Response) => {
return res.send({
code: 410,
@@ -136,6 +250,11 @@ export default (app: Router) => {
filename: Joi.string().required(),
path: Joi.string().optional().allow(''),
content: Joi.string().optional().allow(''),
skipHistory: Joi.boolean().optional(),
expectedHash: Joi.string()
.hex()
.length(64)
.when('skipHistory', { is: true, then: Joi.required() }),
originFilename: Joi.string().optional().allow(''),
directory: Joi.string().optional().allow(''),
file: Joi.string().optional().allow(''),
@@ -195,8 +314,56 @@ export default (app: Router) => {
if (!isPathAllowed(filePath) || !isPathAllowed(originFilePath)) {
return res.send({ code: 403, message: t('暂无权限') });
}
await fs.mkdir(path, { recursive: true });
// Check the canonical parent directly at the directory creation boundary.
const parentPath = resolve(dirname(filePath));
let parentCreated = false;
for (const writableRoot of config.writePathList) {
const root = resolve(writableRoot);
const rootPrefix = root.endsWith(sep) ? root : root + sep;
if (parentPath === root) {
await fs.mkdir(root, { recursive: true });
} else if (parentPath.startsWith(rootPrefix)) {
await fs.mkdir(parentPath, { recursive: true });
} else {
continue;
}
parentCreated = true;
break;
}
if (!parentCreated) {
return res.send({ code: 403, message: t('暂无权限') });
}
const fileExists = await fileExist(filePath);
if (fileExists && resolveFileAccess(config.scriptPath, [filePath])) {
let removeSource = filename !== originFilename;
let backupSource = '';
if (removeSource) {
const [originRealPath, targetRealPath] = await Promise.all([
existingWritableRealPath(originFilePath),
existingWritableRealPath(filePath),
]);
// Aliases of one script are an in-place save. Removing the source
// would also remove the destination behind a target symlink.
removeSource = originRealPath !== targetRealPath;
backupSource = originRealPath;
}
// Save-as removes the source after committing the destination. Keep
// its original content too: destination history only protects the
// file being overwritten, not the source being deleted.
if (removeSource) {
await fs.copyFile(
backupSource,
join(config.bakPath, originFilename.replace(/\//g, '')),
);
}
const service = Container.get(ScriptHistoryService);
const data = await service.save(path, filename, content, {
skipHistory: req.body.skipHistory,
expectedHash: req.body.expectedHash,
});
if (removeSource) await rmPath(originFilePath);
return res.send({ code: 200, data });
}
if (fileExists) {
await fs.copyFile(
originFilePath,
@@ -209,7 +376,7 @@ export default (app: Router) => {
await writeFileWithLock(filePath, content);
return res.send({ code: 200 });
} catch (e) {
return next(e);
return historyError(e, res, next);
}
},
);
@@ -221,6 +388,11 @@ export default (app: Router) => {
filename: Joi.string().required(),
path: Joi.string().optional().allow(''),
content: Joi.string().required().allow(''),
skipHistory: Joi.boolean().optional(),
expectedHash: Joi.string()
.hex()
.length(64)
.when('skipHistory', { is: true, then: Joi.required() }),
}),
}),
async (req: Request, res: Response, next: NextFunction) => {
@@ -238,10 +410,14 @@ export default (app: Router) => {
message: t('暂无权限'),
});
}
await writeFileWithLock(filePath, content);
return res.send({ code: 200 });
const service = Container.get(ScriptHistoryService);
const data = await service.save(path || '', filename, content, {
skipHistory: req.body.skipHistory,
expectedHash: req.body.expectedHash,
});
return res.send({ code: 200, data });
} catch (e) {
return next(e);
return historyError(e, res, next);
}
},
);
@@ -368,12 +544,21 @@ export default (app: Router) => {
if (!isPathAllowed(filePath)) {
return res.send({ code: 403, message: t('暂无权限') });
}
const logPath = join(config.logPath, path, `${name}.swap`);
const logPath = resolveFileAccess(config.logPath, [
path,
`${name}.swap`,
]);
if (!logPath) {
return res.send({ code: 403, message: t('暂无权限') });
}
const scriptService = Container.get(ScriptService);
const result = await scriptService.stopScript(filePath, pid);
setTimeout(() => {
rmPath(logPath);
const cleanupPath = resolveFileAccess(config.logPath, [logPath]);
if (cleanupPath) {
void rmPath(cleanupPath);
}
}, 3000);
res.send(result);
} catch (e) {
+3 -10
View File
@@ -3,7 +3,7 @@ import { Container } from 'typedi';
import { Logger } from 'winston';
import SubscriptionService from '../services/subscription';
import { celebrate, Joi } from 'celebrate';
import CronExpressionParser from 'cron-parser';
import { isValidCronSchedule } from '../shared/cronSchedule';
const route = Router();
export default (app: Router) => {
@@ -58,10 +58,7 @@ export default (app: Router) => {
async (req: Request, res: Response, next: NextFunction) => {
const logger: Logger = Container.get('logger');
try {
if (
!req.body.schedule ||
CronExpressionParser.parse(req.body.schedule).hasNext()
) {
if (!req.body.schedule || isValidCronSchedule(req.body.schedule)) {
const subscriptionService = Container.get(SubscriptionService);
const data = await subscriptionService.create(req.body);
return res.send({ code: 200, data });
@@ -204,11 +201,7 @@ export default (app: Router) => {
async (req: Request, res: Response, next: NextFunction) => {
const logger: Logger = Container.get('logger');
try {
if (
!req.body.schedule ||
typeof req.body.schedule === 'object' ||
CronExpressionParser.parse(req.body.schedule).hasNext()
) {
if (!req.body.schedule || isValidCronSchedule(req.body.schedule)) {
const subscriptionService = Container.get(SubscriptionService);
const data = await subscriptionService.update(req.body);
return res.send({ code: 200, data });
+14 -9
View File
@@ -8,15 +8,12 @@ import { celebrate, Joi } from 'celebrate';
import UserService from '../services/user';
import { t } from '../shared/i18n';
import { isDefaultAuthInfo } from '../shared/auth';
import {
getUniqPath,
handleLogPath,
parseVersion,
promiseExec,
} from '../config/util';
import { getUniqPath, handleLogPath, parseVersion } from '../config/util';
import { findCronId } from '../shared/cronCommand';
import dayjs from 'dayjs';
import multer from 'multer';
import { logStreamManager } from '../shared/logStreamManager';
import { rateLimit } from 'express-rate-limit';
const route = Router();
const storage = multer.diskStorage({
@@ -249,6 +246,7 @@ export default (app: Router) => {
route.put(
'/command-run',
rateLimit({ windowMs: 60 * 1000, limit: 60 }),
celebrate({
body: Joi.object({
command: Joi.string().required(),
@@ -258,8 +256,13 @@ export default (app: Router) => {
try {
const systemService = Container.get(SystemService);
const command = req.body.command;
const idStr = `cat ${config.crontabFile} | grep -E "${command}" | perl -pe "s|.*ID=(.*) ${command}.*|\\1|" | head -1 | awk -F " " '{print $1}' | xargs echo -n`;
let id = await promiseExec(idStr);
const crontab = await fs
.readFile(config.crontabFile, 'utf8')
.catch((e) => {
if (e.code === 'ENOENT') return '';
throw e;
});
const id = findCronId(crontab, command);
const uniqPath = await getUniqPath(command, id);
const logTime = dayjs().format('YYYY-MM-DD-HH-mm-ss-SSS');
const logPath = `${uniqPath}/${logTime}.log`;
@@ -274,7 +277,9 @@ export default (app: Router) => {
onEnd: async (cp, endTime, diff) => {
// Close the stream after task completion
try {
await logStreamManager.closeStream(await handleLogPath(logPath));
await logStreamManager.closeStream(
await handleLogPath(logPath),
);
} finally {
res.end();
}
+1
View File
@@ -27,6 +27,7 @@ export const SAMPLE_FILES = [
];
export const PYTHON_INSTALL_DIR = process.env.PYTHON_HOME;
export const PYTHON_VENV_DIR = process.env.PYTHON_VENV_DIR;
export const NotificationModeStringMap = {
0: 'gotify',
+93 -45
View File
@@ -1,6 +1,5 @@
import { execSync } from 'child_process';
import { execFileSync } from 'child_process';
import * as fs from 'fs/promises';
import * as os from 'os';
import path from 'path';
import config from './index';
import { fileExist } from './util';
@@ -24,71 +23,120 @@ const clientCertPath = path.join(certDir, 'client.crt');
let cachedConfig: GrpcTlsConfig | null = null;
function run(cmd: string, execOpts?: Record<string, unknown>): string {
const opts = { stdio: 'pipe', timeout: 30000, encoding: 'utf-8', ...execOpts } as any;
return (execSync(cmd, opts) as string).trim();
}
async function tmpFile(prefix: string): Promise<string> {
const dir = (await fileExist(certDir)) ? certDir : os.tmpdir();
await fs.mkdir(dir, { recursive: true });
return path.join(dir, `.${prefix}_${Date.now()}_${Math.random().toString(36).slice(2)}.pem`);
function run(args: string[]): string {
return execFileSync('openssl', args, {
stdio: 'pipe',
timeout: 30000,
encoding: 'utf-8',
}).trim();
}
async function generateAllCerts(): Promise<GrpcTlsConfig> {
Logger.info('[boot] Generating gRPC mTLS certificates...');
const caKeyTmp = await tmpFile('ca_key');
const caCertTmp = await tmpFile('ca_cert');
const serverKeyTmp = await tmpFile('server_key');
const serverCsrTmp = await tmpFile('server_csr');
const serverExtTmp = await tmpFile('server_ext');
const clientKeyTmp = await tmpFile('client_key');
const clientCsrTmp = await tmpFile('client_csr');
const clientExtTmp = await tmpFile('client_ext');
const srlTmp = path.join(path.dirname(caKeyTmp), '.grpc_ca.srl');
await fs.mkdir(certDir, { recursive: true, mode: 0o700 });
// Atomically reserve a private workspace for keys, CSRs and OpenSSL side files.
const workspace = await fs.mkdtemp(path.join(certDir, '.generate-'));
const caKeyTmp = path.join(workspace, 'ca.key');
const caCertTmp = path.join(workspace, 'ca.crt');
const serverKeyTmp = path.join(workspace, 'server.key');
const serverCsrTmp = path.join(workspace, 'server.csr');
const serverExtTmp = path.join(workspace, 'server.ext');
const clientKeyTmp = path.join(workspace, 'client.key');
const clientCsrTmp = path.join(workspace, 'client.csr');
const clientExtTmp = path.join(workspace, 'client.ext');
const cleanup = async () => {
for (const f of [caKeyTmp, caCertTmp, serverKeyTmp, serverCsrTmp, serverExtTmp,
clientKeyTmp, clientCsrTmp, clientExtTmp, srlTmp]) {
try { await fs.unlink(f); } catch {}
}
await fs.rm(workspace, { recursive: true, force: true });
};
try {
// 1. CA(私钥直接存盘,证书写入临时文件供签发使用)
run(`openssl genrsa -out '${caKeyTmp}' 2048 2>/dev/null`);
run(`openssl req -new -x509 -days 3650 -key '${caKeyTmp}' -out '${caCertTmp}' -subj '/CN=qinglong-ca/O=qinglong/C=CN' 2>/dev/null`);
run(['genrsa', '-out', caKeyTmp, '2048']);
run([
'req',
'-new',
'-x509',
'-days',
'3650',
'-key',
caKeyTmp,
'-out',
caCertTmp,
'-subj',
'/CN=qinglong-ca/O=qinglong/C=CN',
]);
const caKey = await fs.readFile(caKeyTmp, 'utf-8');
const caCert = await fs.readFile(caCertTmp, 'utf-8');
await fs.mkdir(certDir, { recursive: true });
await fs.writeFile(caKeyPath, caKey, { mode: 0o600 });
// 2. 服务端
run(`openssl genrsa -out '${serverKeyTmp}' 2048 2>/dev/null`);
run(`openssl req -new -key '${serverKeyTmp}' -out '${serverCsrTmp}' -subj '/CN=grpc-server' 2>/dev/null`);
await fs.writeFile(serverExtTmp, 'subjectAltName=DNS:localhost,IP:127.0.0.1,IP:::1\n');
const serverCert = run(
`openssl x509 -req -days 3650 -in '${serverCsrTmp}' -CA '${caCertTmp}' -CAkey '${caKeyTmp}' -CAcreateserial -extfile '${serverExtTmp}' 2>/dev/null`,
run(['genrsa', '-out', serverKeyTmp, '2048']);
run([
'req',
'-new',
'-key',
serverKeyTmp,
'-out',
serverCsrTmp,
'-subj',
'/CN=grpc-server',
]);
await fs.writeFile(
serverExtTmp,
'subjectAltName=DNS:localhost,IP:127.0.0.1,IP:::1\n',
);
const serverCert = run([
'x509',
'-req',
'-days',
'3650',
'-in',
serverCsrTmp,
'-CA',
caCertTmp,
'-CAkey',
caKeyTmp,
'-CAserial',
path.join(workspace, 'ca.srl'),
'-CAcreateserial',
'-extfile',
serverExtTmp,
]);
const serverKey = await fs.readFile(serverKeyTmp, 'utf-8');
// 3. 客户端
run(`openssl genrsa -out '${clientKeyTmp}' 2048 2>/dev/null`);
run(`openssl req -new -key '${clientKeyTmp}' -out '${clientCsrTmp}' -subj '/CN=grpc-client' 2>/dev/null`);
run(['genrsa', '-out', clientKeyTmp, '2048']);
run([
'req',
'-new',
'-key',
clientKeyTmp,
'-out',
clientCsrTmp,
'-subj',
'/CN=grpc-client',
]);
await fs.writeFile(clientExtTmp, 'extendedKeyUsage=clientAuth\n');
const clientCert = run(
`openssl x509 -req -days 3650 -in '${clientCsrTmp}' -CA '${caCertTmp}' -CAkey '${caKeyTmp}' -CAcreateserial -extfile '${clientExtTmp}' 2>/dev/null`,
);
const clientCert = run([
'x509',
'-req',
'-days',
'3650',
'-in',
clientCsrTmp,
'-CA',
caCertTmp,
'-CAkey',
caKeyTmp,
'-CAserial',
path.join(workspace, 'ca.srl'),
'-CAcreateserial',
'-extfile',
clientExtTmp,
]);
const clientKey = await fs.readFile(clientKeyTmp, 'utf-8');
await cleanup();
Logger.info('[boot] gRPC mTLS certificates generated successfully');
return { caCert, serverCert, serverKey, clientCert, clientKey };
} catch (e) {
} finally {
await cleanup();
throw e;
}
}
+23 -26
View File
@@ -1,5 +1,7 @@
import { Subscription } from '../data/subscription';
import isNil from 'lodash/isNil';
import { shellQuote } from '../shared/shellQuote';
import { getSubscriptionSshAlias } from '../shared/subscriptionPath';
export function formatUrl(doc: Subscription) {
let url = doc.url;
@@ -7,7 +9,7 @@ export function formatUrl(doc: Subscription) {
if (doc.type === 'private-repo') {
if (doc.pull_type === 'ssh-key') {
host = doc.url!.replace(/.*\@([^\:]+)\:.*/, '$1');
url = doc.url!.replace(host, doc.alias);
url = doc.url!.replace(host, getSubscriptionSshAlias(doc.alias));
} else {
host = doc.url!.replace(/.*\:\/\/([^\/]+)\/.*/, '$1');
const { username, password } = doc.pull_option as any;
@@ -18,29 +20,24 @@ export function formatUrl(doc: Subscription) {
}
export function formatCommand(doc: Subscription, url?: string) {
let command = `SUB_ID=${doc.id} ql `;
let _url = url || formatUrl(doc).url;
const {
type,
whitelist,
blacklist,
dependences,
branch,
extensions,
proxy,
autoAddCron,
autoDelCron,
} = doc;
if (type === 'file') {
command += `raw "${_url}" "${proxy || ''}" "${
isNil(autoAddCron) ? true : Boolean(autoAddCron)
}" "${isNil(autoDelCron) ? true : Boolean(autoDelCron)}"`;
} else {
command += `repo "${_url}" "${whitelist || ''}" "${blacklist || ''}" "${
dependences || ''
}" "${branch || ''}" "${extensions || ''}" "${proxy || ''}" "${
isNil(autoAddCron) ? true : Boolean(autoAddCron)
}" "${isNil(autoDelCron) ? true : Boolean(autoDelCron)}"`;
}
return command;
const args =
doc.type === 'file'
? ['raw', url || formatUrl(doc).url, doc.proxy || '']
: [
'repo',
url || formatUrl(doc).url,
doc.whitelist || '',
doc.blacklist || '',
doc.dependences || '',
doc.branch || '',
doc.extensions || '',
doc.proxy || '',
];
args.push(
String(isNil(doc.autoAddCron) ? true : Boolean(doc.autoAddCron)),
String(isNil(doc.autoDelCron) ? true : Boolean(doc.autoDelCron)),
);
return `SUB_ID=${shellQuote(String(doc.id))} ql ${args
.map((arg) => shellQuote(arg!))
.join(' ')}`;
}
+196 -100
View File
@@ -1,18 +1,25 @@
import * as fs from 'fs/promises';
import { statSync } from 'fs';
import * as path from 'path';
import { exec, execSync } from 'child_process';
import psTreeFun from 'ps-tree';
import { promisify } from 'util';
import { load } from 'js-yaml';
import config from './index';
import { PYTHON_INSTALL_DIR, TASK_COMMAND } from './const';
import { PYTHON_INSTALL_DIR, PYTHON_VENV_DIR, TASK_COMMAND } from './const';
import Logger from '../loaders/logger';
import { writeFileWithLock } from '../shared/utils';
import { DependenceTypes } from '../data/dependence';
import { FormData } from 'undici';
import os from 'os';
import { maybeSudo, isInContainer } from './container';
import { shellQuote } from '../shared/shellQuote';
import { resolveFileAccess } from '../shared/fileAccess';
import {
mirrorWriteCommand,
normalizeMirror,
rewriteMirrorSources,
} from '../shared/linuxMirror';
export * from './share';
@@ -670,7 +677,9 @@ export async function setSystemTimezone(timezone: string): Promise<boolean> {
throw new Error('Invalid timezone');
}
await promiseExec(maybeSudo(`ln -sf /usr/share/zoneinfo/${timezone} /etc/localtime`));
await promiseExec(
maybeSudo(`ln -sf /usr/share/zoneinfo/${timezone} /etc/localtime`),
);
await promiseExec(`echo "${timezone}" | ${maybeSudo('tee /etc/timezone')}`);
return true;
@@ -680,61 +689,150 @@ export async function setSystemTimezone(timezone: string): Promise<boolean> {
}
}
// Helper function to check if a name is a GitHub URL
function getPythonExecutable(): string {
return PYTHON_VENV_DIR
? shellQuote(path.resolve(config.rootPath, PYTHON_VENV_DIR, 'bin/python3'))
: 'python3';
}
function getPythonPip(): string {
return PYTHON_VENV_DIR ? `${getPythonExecutable()} -m pip` : 'pip3';
}
function isGitHubUrl(name: string): boolean {
// Support git+https://, git+http://, https://, and http:// URLs
// This covers GitHub URLs and other git-compatible repositories
return !!name.match(/^(git\+https?:\/\/|https?:\/\/)/i);
}
// Helper function to check if a name is a requirements file
function isRequirementsFile(name: string): boolean {
return !!name.match(/requirements.*\.(txt|in)$/i);
}
// Helper function to check if a name is a pyproject.toml file
function isPyprojectToml(name: string): boolean {
return name.endsWith('pyproject.toml');
}
export function getGetCommand(type: DependenceTypes, name: string): string {
const trimmedName = name.trim();
// For Python dependencies installed from GitHub or requirements files,
// we can't reliably check if they're installed, so skip the check
if (type === DependenceTypes.python3) {
if (
isGitHubUrl(trimmedName) ||
isRequirementsFile(trimmedName) ||
isPyprojectToml(trimmedName)
) {
// Return a command that will always indicate not installed
// This ensures GitHub URLs and requirements files are always installed
return 'echo ""';
}
}
const baseCommands = {
[DependenceTypes.nodejs]: `pnpm ls -g | grep "${name}" | head -1`,
[DependenceTypes.python3]: `
python3 -c "exec('''
name='${name}'
try:
from importlib.metadata import version
print(version(name))
except:
import importlib.util as u
import importlib.metadata as m
spec=u.find_spec(name)
print(name if spec else '')
''')"`,
[DependenceTypes.linux]: getOsTypeSync() === 'Alpine'
? `apk info -es ${name}`
: maybeSudo(`dpkg-query -s ${name}`),
[DependenceTypes.nodejs]: `pnpm ls -g | grep "${trimmedName}" | head -1`,
[DependenceTypes.python3]: `${getPythonExecutable()} -c ${shellQuote(
`from importlib.metadata import version\nimport importlib.util as u\nname=${JSON.stringify(
trimmedName,
)}\ntry:\n print(version(name))\nexcept Exception:\n print(name if u.find_spec(name) else '')`,
)}`,
[DependenceTypes.linux]:
getOsTypeSync() === 'Alpine'
? `apk info -es ${trimmedName}`
: maybeSudo(`dpkg-query -s ${trimmedName}`),
};
return baseCommands[type];
}
export function getInstallCommand(type: DependenceTypes, name: string): string {
if (type === DependenceTypes.python3) {
const prefix =
PYTHON_INSTALL_DIR && !PYTHON_VENV_DIR
? ` --prefix=${shellQuote(PYTHON_INSTALL_DIR)}`
: '';
const pip = `${getPythonPip()} install --disable-pip-version-check${
PYTHON_VENV_DIR ? '' : ' --root-user-action=ignore'
}${prefix}`;
name = name.trim();
if (isRequirementsFile(name) || isPyprojectToml(name)) {
const file = [config.scriptPath, config.repoPath]
.filter(Boolean)
.map((root) => resolveFileAccess(root, [name]))
.find((file) => {
if (!file) return false;
try {
return statSync(file).isFile();
} catch {
return false;
}
});
if (!file)
throw new Error(
'Python dependency file must be inside scripts or repo',
);
return isRequirementsFile(name)
? `${pip} -r ${shellQuote(file)}`
: `${pip} ${shellQuote(path.dirname(file))}`;
}
return `${pip} -- ${shellQuote(name)}`;
}
const baseCommands = {
[DependenceTypes.nodejs]: 'pnpm add -g',
[DependenceTypes.python3]:
'pip3 install --disable-pip-version-check --root-user-action=ignore',
[DependenceTypes.linux]: getOsTypeSync() === 'Alpine'
? 'apk add --no-check-certificate'
: maybeSudo('apt-get install -y'),
[DependenceTypes.linux]:
getOsTypeSync() === 'Alpine'
? 'apk add --no-check-certificate'
: maybeSudo('apt-get install -y'),
};
let command = baseCommands[type];
if (type === DependenceTypes.python3 && PYTHON_INSTALL_DIR) {
command = `${command} --prefix=${PYTHON_INSTALL_DIR}`;
}
return `${command} ${name.trim()}`;
return `${baseCommands[type]} ${name.trim()}`;
}
export function getUninstallCommand(
type: DependenceTypes,
name: string,
): string {
if (type === DependenceTypes.python3) {
name = name.trim();
if (isRequirementsFile(name)) {
const file = [config.scriptPath, config.repoPath]
.filter(Boolean)
.map((root) => resolveFileAccess(root, [name]))
.find((file) => {
if (!file) return false;
try {
return statSync(file).isFile();
} catch {
return false;
}
});
if (!file)
throw new Error(
'Python dependency file must be inside scripts or repo',
);
return `${getPythonPip()} uninstall --disable-pip-version-check${
PYTHON_VENV_DIR ? '' : ' --root-user-action=ignore'
} -y -r ${shellQuote(file)}`;
}
if (isPyprojectToml(name) || isGitHubUrl(name)) {
// A source URL/project can install several distributions and has no safe
// inverse. Removing its panel entry leaves installed packages intact.
return `printf '%s\\n' ${shellQuote(
'Remove installed Python distributions separately by package name.',
)}`;
}
return `${getPythonPip()} uninstall --disable-pip-version-check${
PYTHON_VENV_DIR ? '' : ' --root-user-action=ignore'
} -y -- ${shellQuote(name)}`;
}
const baseCommands = {
[DependenceTypes.nodejs]: 'pnpm remove -g',
[DependenceTypes.python3]:
'pip3 uninstall --disable-pip-version-check --root-user-action=ignore -y',
[DependenceTypes.linux]: getOsTypeSync() === 'Alpine'
? 'apk del'
: maybeSudo('apt-get remove -y'),
[DependenceTypes.linux]:
getOsTypeSync() === 'Alpine' ? 'apk del' : maybeSudo('apt-get remove -y'),
};
return `${baseCommands[type]} ${name.trim()}`;
}
@@ -806,75 +904,73 @@ export async function detectOS(): Promise<
return osType;
}
async function getCurrentMirrorDomain(
filePath: string,
): Promise<string | null> {
const fileContent = await fs.readFile(filePath, 'utf8');
const lines = fileContent.split('\n');
for (const line of lines) {
if (line.trim().startsWith('#')) {
continue;
}
const match = line.match(/https?:\/\/[^\/]+/);
if (match) {
return match[0];
}
}
return null;
}
async function replaceDomainInFile(
filePath: string,
oldDomainWithScheme: string,
newDomainWithScheme: string,
): Promise<void> {
let fileContent = await fs.readFile(filePath, 'utf8');
let updatedContent = fileContent.replace(
new RegExp(oldDomainWithScheme, 'g'),
newDomainWithScheme,
);
if (!newDomainWithScheme.endsWith('/')) {
newDomainWithScheme += '/';
}
await writeFileWithLock(filePath, updatedContent);
}
async function _updateLinuxMirror(
osType: string,
mirrorDomainWithScheme: string,
mirror: string,
): Promise<string> {
const defaults: Record<string, string> = {
Debian: 'http://deb.debian.org/debian',
Ubuntu: 'http://archive.ubuntu.com/ubuntu',
Alpine: 'http://dl-cdn.alpinelinux.org/alpine',
};
if (!defaults[osType])
throw new Error('Unsupported OS type for updating mirrors.');
const target = mirror || defaults[osType];
normalizeMirror(target);
const S = isInContainer() ? 'sudo ' : '';
let filePath: string, currentDomainWithScheme: string | null;
switch (osType) {
case 'Debian':
filePath = '/etc/apt/sources.list.d/debian.sources';
currentDomainWithScheme = await getCurrentMirrorDomain(filePath);
if (currentDomainWithScheme) {
return `${S}sed -i 's|${currentDomainWithScheme}|${mirrorDomainWithScheme || 'http://deb.debian.org'}|g' ${filePath} || (${S}mkdir -p /etc/apt/sources.list.d && echo -e "Types: deb\\nURIs: ${mirrorDomainWithScheme || 'http://deb.debian.org'}\\nSuites: \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2) \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2)-updates\\nComponents: main\\nSigned-By: /usr/share/keyrings/debian-archive-keyring.gpg" | ${S}tee ${filePath}) && ${S}apt-get update`;
} else {
return `${S}mkdir -p /etc/apt/sources.list.d && echo -e "Types: deb\\nURIs: ${mirrorDomainWithScheme || 'http://deb.debian.org'}\\nSuites: \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2) \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2)-updates\\nComponents: main\\nSigned-By: /usr/share/keyrings/debian-archive-keyring.gpg" | ${S}tee ${filePath} && ${S}apt-get update`;
const update =
osType === 'Alpine' ? 'apk update' : `${S}apt-get update --error-on=any`;
let file =
osType === 'Alpine'
? '/etc/apk/repositories'
: `/etc/apt/sources.list.d/${osType.toLowerCase()}.sources`;
let content = '';
try {
content = await fs.readFile(file, 'utf8');
} catch (error: any) {
if (error.code !== 'ENOENT') throw error;
if (osType !== 'Alpine') {
try {
file = '/etc/apt/sources.list';
content = await fs.readFile(file, 'utf8');
} catch (legacyError: any) {
if (legacyError.code !== 'ENOENT') throw legacyError;
file = `/etc/apt/sources.list.d/${osType.toLowerCase()}.sources`;
}
case 'Ubuntu':
filePath = '/etc/apt/sources.list.d/ubuntu.sources';
currentDomainWithScheme = await getCurrentMirrorDomain(filePath);
if (currentDomainWithScheme) {
return `${S}sed -i 's|${currentDomainWithScheme}|${mirrorDomainWithScheme || 'http://archive.ubuntu.com'}|g' ${filePath} || (${S}mkdir -p /etc/apt/sources.list.d && echo -e "Types: deb\\nURIs: ${mirrorDomainWithScheme || 'http://archive.ubuntu.com'}\\nSuites: \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2) \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2)-updates \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2)-backports\\nComponents: main restricted universe multiverse\\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg" | ${S}tee ${filePath}) && ${S}apt-get update`;
} else {
return `${S}mkdir -p /etc/apt/sources.list.d && echo -e "Types: deb\\nURIs: ${mirrorDomainWithScheme || 'http://archive.ubuntu.com'}\\nSuites: \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2) \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2)-updates \\$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2)-backports\\nComponents: main restricted universe multiverse\\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg" | ${S}tee ${filePath} && ${S}apt-get update`;
}
case 'Alpine':
filePath = '/etc/apk/repositories';
currentDomainWithScheme = await getCurrentMirrorDomain(filePath);
if (currentDomainWithScheme) {
return `sed -i 's|${currentDomainWithScheme}|${mirrorDomainWithScheme || 'http://dl-cdn.alpinelinux.org'}|g' ${filePath} || (mkdir -p /etc/apk && echo -e "\\$(grep VERSION_ID /etc/os-release | cut -d= -f2 | cut -d. -f1,2)/main\\n\\$(grep VERSION_ID /etc/os-release | cut -d= -f2 | cut -d. -f1,2)/community" | sed "s|^|${mirrorDomainWithScheme || 'http://dl-cdn.alpinelinux.org'}/alpine/v|" | tee ${filePath}) && apk update`;
} else {
return `mkdir -p /etc/apk && echo -e "\\$(grep VERSION_ID /etc/os-release | cut -d= -f2 | cut -d. -f1,2)/main\\n\\$(grep VERSION_ID /etc/os-release | cut -d= -f2 | cut -d. -f1,2)/community" | sed "s|^|${mirrorDomainWithScheme || 'http://dl-cdn.alpinelinux.org'}/alpine/v|" | tee ${filePath} && apk update`;
}
default:
throw Error('Unsupported OS type for updating mirrors.');
}
}
if (!content.trim()) {
const release = await fs.readFile('/etc/os-release', 'utf8');
if (osType === 'Alpine') {
const version = release.match(/^VERSION_ID=["']?(\d+\.\d+)/m)?.[1];
if (!version)
throw new Error('Cannot determine Alpine repository version');
content = `${defaults.Alpine}/v${version}/main\n${defaults.Alpine}/v${version}/community\n`;
} else {
// The fallback may have read an empty legacy .list. Deb822 stanzas must
// be written to a .sources file, leaving the legacy file untouched.
file = `/etc/apt/sources.list.d/${osType.toLowerCase()}.sources`;
const codename = release.match(/^VERSION_CODENAME=["']?([\w-]+)/m)?.[1];
if (!codename)
throw new Error('Cannot determine APT repository codename');
const ubuntu = osType === 'Ubuntu';
content = `Types: deb\nURIs: ${
defaults[osType]
}\nSuites: ${codename} ${codename}-updates${
ubuntu ? ` ${codename}-backports` : ''
}\nComponents: ${
ubuntu ? 'main restricted universe multiverse' : 'main'
}\nSigned-By: /usr/share/keyrings/${osType.toLowerCase()}-archive-keyring.gpg\n`;
if (!ubuntu)
content += `\nTypes: deb\nURIs: http://security.debian.org/debian-security\nSuites: ${codename}-security\nComponents: main\nSigned-By: /usr/share/keyrings/debian-archive-keyring.gpg\n`;
}
}
return mirrorWriteCommand(
file,
rewriteMirrorSources(content, target),
S,
update,
);
}
export async function updateLinuxMirrorFile(mirror: string): Promise<string> {
+10 -1
View File
@@ -14,6 +14,7 @@ import { AuthInfo } from '../data/system';
import path from 'path';
import { t } from '../shared/i18n';
import { AppScope } from '../data/open';
import uniqueConstraintError from '../middlewares/uniqueConstraintError';
import protectedPathCase from '../middlewares/protectedPathCase';
function resolveTrustProxy(value = process.env.QL_TRUST_PROXY) {
@@ -178,7 +179,14 @@ export default ({ app }: { app: Application }) => {
app.use(rewrite('/open/*', '/api/$1'));
app.use(config.api.prefix, routes());
app.get('*', (_, res, next) => {
app.get('*', (req, res, next) => {
// Don't serve index.html for API routes
if (
req.path === config.api.prefix ||
req.path.startsWith(`${config.api.prefix}/`)
) {
return res.status(404).json({ code: 404, message: 'Not Found' });
}
const indexPath = path.join(frontendPath, 'index.html');
res.sendFile(indexPath, (err) => {
if (err) {
@@ -190,6 +198,7 @@ export default ({ app }: { app: Application }) => {
});
app.use(errors());
app.use(uniqueConstraintError);
app.use(
(
+12 -26
View File
@@ -13,12 +13,11 @@ import { AuthDataType, SystemModel } from '../data/system';
import SystemService from '../services/system';
import UserService from '../services/user';
import { writeFile, readFile } from 'fs/promises';
import { createRandomString, fileExist, isDemoEnv, safeJSONParse } from '../config/util';
import { fileExist, isDemoEnv, safeJSONParse } from '../config/util';
import OpenService from '../services/open';
import { shareStore } from '../shared/store';
import Logger from './logger';
import cronClient from '../schedule/client';
import { AppModel } from '../data/open';
import { InstanceStatus, RunningInstanceModel } from '../data/runningInstance';
import { setLang, systemLang } from '../shared/i18n';
@@ -31,24 +30,7 @@ export default async () => {
const openService = Container.get(OpenService);
// 初始化增加系统配置
let systemApp = (
await AppModel.findOne({
where: { name: 'system' },
})
)?.get({ plain: true });
if (!systemApp) {
systemApp = await AppModel.create({
name: 'system',
scopes: ['crons', 'system', 'dashboard'],
client_id: createRandomString(12, 12),
client_secret: createRandomString(24, 24),
});
} else if (!systemApp.scopes.includes('dashboard')) {
await AppModel.update(
{ scopes: [...systemApp.scopes, 'dashboard'] },
{ where: { name: 'system' } },
);
}
await openService.initializeSystemApp();
const [systemConfig] = await SystemModel.findOrCreate({
where: { type: AuthDataType.systemConfig },
});
@@ -144,7 +126,11 @@ export default async () => {
// 清空所有运行中的实例记录(服务重启后进程已不存在)
await RunningInstanceModel.update(
{ status: InstanceStatus.stopped },
{
status: InstanceStatus.stopped,
finished_at: Math.floor(Date.now() / 1000),
// The previous process's exit code cannot be recovered after a restart.
},
{ where: { status: InstanceStatus.running } },
);
@@ -237,15 +223,15 @@ export default async () => {
} catch { }
// 初始化保存一次ck和定时任务数据
cronClient.readiness.configure(() => cronService.autosave_crontab(true));
cronClient.readiness.configure(async () => {
await cronService.autosave_crontab(true);
await systemService.restoreCronConcurrency();
});
await cronClient.readiness.recover();
await envService.set_envs();
const authInfo = await userService.getAuthInfo();
const apps = await openService.findApps();
await shareStore.updateAuthInfo(authInfo);
if (apps?.length) {
await shareStore.updateApps(apps);
}
await openService.refreshApps();
};
+10 -1
View File
@@ -7,6 +7,8 @@ import config from '../config';
import { fileExist } from '../config/util';
import { join } from 'path';
import { t } from '../shared/i18n';
import Logger from './logger';
import { assertSubscriptionAlias } from '../shared/subscriptionPath';
export default async () => {
const systemService = Container.get(SystemService);
@@ -24,6 +26,7 @@ export default async () => {
if (await fileExist(tokenFile)) {
tokenCommand = `node ${tokenFile}`;
}
tokenCommand += ' --renew';
const cron = {
id: NaN,
name: t('生成token'),
@@ -70,6 +73,12 @@ export default async () => {
await subscriptionService.setSshConfig();
const subs = await subscriptionService.list();
for (const sub of subs) {
subscriptionService.handleTask(sub.get({ plain: true }), !sub.is_disabled);
try {
const doc = sub.get({ plain: true });
assertSubscriptionAlias(doc.alias);
await subscriptionService.handleTask(doc, !sub.is_disabled);
} catch (error) {
Logger.warn('跳过订阅 %s 的无效定时配置: %s', sub.id, error);
}
}
};
+14
View File
@@ -0,0 +1,14 @@
import { ErrorRequestHandler } from 'express';
import { UniqueConstraintError } from 'sequelize';
import { t } from '../shared/i18n';
// Handle the database constraint itself so concurrent writes are covered too.
const uniqueConstraintError: ErrorRequestHandler = (err, req, res, next) => {
if (!(err instanceof UniqueConstraintError)) return next(err);
// Constraint details can contain environment values or application credentials.
res
.status(409)
.json({ code: 409, message: t('资源已存在,请检查重复的名称或值') });
};
export default uniqueConstraintError;
+4
View File
@@ -5,6 +5,7 @@ package com.ql.cron;
service Cron {
rpc addCron(AddCronRequest) returns (AddCronResponse);
rpc delCron(DeleteCronRequest) returns (DeleteCronResponse);
rpc setConcurrency(SetConcurrencyRequest) returns (AddCronResponse);
}
message ISchedule { string schedule = 1; }
@@ -27,3 +28,6 @@ message AddCronResponse {}
message DeleteCronRequest { repeated string ids = 1; }
message DeleteCronResponse {}
// Zero restores the default limit.
message SetConcurrencyRequest { int32 concurrency = 1; }
+89 -1
View File
@@ -1,7 +1,7 @@
// Code generated by protoc-gen-ts_proto. DO NOT EDIT.
// versions:
// protoc-gen-ts_proto v2.6.1
// protoc v3.17.3
// protoc v5.28.3
// source: back/protos/cron.proto
/* eslint-disable */
@@ -48,6 +48,11 @@ export interface DeleteCronRequest {
export interface DeleteCronResponse {
}
/** Zero restores the default limit. */
export interface SetConcurrencyRequest {
concurrency: number;
}
function createBaseISchedule(): ISchedule {
return { schedule: "" };
}
@@ -452,6 +457,64 @@ export const DeleteCronResponse: MessageFns<DeleteCronResponse> = {
},
};
function createBaseSetConcurrencyRequest(): SetConcurrencyRequest {
return { concurrency: 0 };
}
export const SetConcurrencyRequest: MessageFns<SetConcurrencyRequest> = {
encode(message: SetConcurrencyRequest, writer: BinaryWriter = new BinaryWriter()): BinaryWriter {
if (message.concurrency !== 0) {
writer.uint32(8).int32(message.concurrency);
}
return writer;
},
decode(input: BinaryReader | Uint8Array, length?: number): SetConcurrencyRequest {
const reader = input instanceof BinaryReader ? input : new BinaryReader(input);
let end = length === undefined ? reader.len : reader.pos + length;
const message = createBaseSetConcurrencyRequest();
while (reader.pos < end) {
const tag = reader.uint32();
switch (tag >>> 3) {
case 1: {
if (tag !== 8) {
break;
}
message.concurrency = reader.int32();
continue;
}
}
if ((tag & 7) === 4 || tag === 0) {
break;
}
reader.skip(tag & 7);
}
return message;
},
fromJSON(object: any): SetConcurrencyRequest {
return { concurrency: isSet(object.concurrency) ? globalThis.Number(object.concurrency) : 0 };
},
toJSON(message: SetConcurrencyRequest): unknown {
const obj: any = {};
if (message.concurrency !== 0) {
obj.concurrency = Math.round(message.concurrency);
}
return obj;
},
create<I extends Exact<DeepPartial<SetConcurrencyRequest>, I>>(base?: I): SetConcurrencyRequest {
return SetConcurrencyRequest.fromPartial(base ?? ({} as any));
},
fromPartial<I extends Exact<DeepPartial<SetConcurrencyRequest>, I>>(object: I): SetConcurrencyRequest {
const message = createBaseSetConcurrencyRequest();
message.concurrency = object.concurrency ?? 0;
return message;
},
};
export type CronService = typeof CronService;
export const CronService = {
addCron: {
@@ -472,11 +535,21 @@ export const CronService = {
responseSerialize: (value: DeleteCronResponse) => Buffer.from(DeleteCronResponse.encode(value).finish()),
responseDeserialize: (value: Buffer) => DeleteCronResponse.decode(value),
},
setConcurrency: {
path: "/com.ql.cron.Cron/setConcurrency",
requestStream: false,
responseStream: false,
requestSerialize: (value: SetConcurrencyRequest) => Buffer.from(SetConcurrencyRequest.encode(value).finish()),
requestDeserialize: (value: Buffer) => SetConcurrencyRequest.decode(value),
responseSerialize: (value: AddCronResponse) => Buffer.from(AddCronResponse.encode(value).finish()),
responseDeserialize: (value: Buffer) => AddCronResponse.decode(value),
},
} as const;
export interface CronServer extends UntypedServiceImplementation {
addCron: handleUnaryCall<AddCronRequest, AddCronResponse>;
delCron: handleUnaryCall<DeleteCronRequest, DeleteCronResponse>;
setConcurrency: handleUnaryCall<SetConcurrencyRequest, AddCronResponse>;
}
export interface CronClient extends Client {
@@ -510,6 +583,21 @@ export interface CronClient extends Client {
options: Partial<CallOptions>,
callback: (error: ServiceError | null, response: DeleteCronResponse) => void,
): ClientUnaryCall;
setConcurrency(
request: SetConcurrencyRequest,
callback: (error: ServiceError | null, response: AddCronResponse) => void,
): ClientUnaryCall;
setConcurrency(
request: SetConcurrencyRequest,
metadata: Metadata,
callback: (error: ServiceError | null, response: AddCronResponse) => void,
): ClientUnaryCall;
setConcurrency(
request: SetConcurrencyRequest,
metadata: Metadata,
options: Partial<CallOptions>,
callback: (error: ServiceError | null, response: AddCronResponse) => void,
): ClientUnaryCall;
}
export const CronClient = makeGenericClientConstructor(CronService, "com.ql.cron.Cron") as unknown as {
+52 -73
View File
@@ -1,6 +1,6 @@
import { ServerUnaryCall, sendUnaryData, status } from '@grpc/grpc-js';
import { AddCronRequest, AddCronResponse } from '../protos/cron';
import nodeSchedule from 'node-schedule';
import { createCronJob, CronJob } from '../shared/cronScheduler';
import { isValidCronSchedule } from '../shared/cronSchedule';
import { scheduleStacks } from './data';
import { runCron } from '../shared/runCron';
@@ -24,11 +24,7 @@ const addCron = (
if (!isValidCronField(schedule)) {
validationErrors.push(
tf(
'任务ID %s: 无效的 cron 表达式 "%s"',
String(id),
schedule,
),
tf('任务ID %s: 无效的 cron 表达式 "%s"', String(id), schedule),
);
}
@@ -56,79 +52,62 @@ const addCron = (
return;
}
// Recovery replaces the whole snapshot, including deletions and disabled jobs.
// Validation above must finish before touching the previous schedule.
if (call.request.replace) {
for (const jobs of scheduleStacks.values()) {
for (const job of jobs) job?.cancel();
// Prepare stopped jobs first: construction errors must preserve the old snapshot.
const prepared = new Map<string, CronJob[]>();
try {
for (const item of call.request.crons) {
const jobs: CronJob[] = [];
prepared.get(item.id)?.forEach((job) => job.cancel());
prepared.set(item.id, jobs);
for (const schedule of [
item.schedule,
...(item.extra_schedules || []).map((x) => x.schedule),
]) {
jobs.push(
createCronJob(
schedule,
async () => {
Logger.info('[schedule][准备运行任务] 命令: %s', item.command);
await runCron(
item.command,
item,
() => scheduleStacks.get(item.id) === jobs,
);
},
{
name: `${item.id}: ${item.name || ''}`,
logger: Logger,
start: false,
},
),
);
}
}
scheduleStacks.clear();
} catch (error) {
for (const jobs of prepared.values()) jobs.forEach((job) => job.cancel());
const err: any = new Error(
error instanceof Error ? error.message : String(error),
);
err.code = status.INVALID_ARGUMENT;
err.details = err.message;
callback(err, null);
return;
}
// ===== 第二遍:注册所有任务 =====
for (const item of call.request.crons) {
const { id, schedule, command, extra_schedules, name } = item;
// 取消该 id 已有的旧任务
if (scheduleStacks.has(id)) {
scheduleStacks.get(id)?.forEach((x) => x.cancel());
}
if (call.request.replace) {
for (const jobs of scheduleStacks.values())
jobs.forEach((job) => job?.cancel());
scheduleStacks.clear();
}
for (const [id, jobs] of prepared) {
scheduleStacks.get(id)?.forEach((job) => job.cancel());
scheduleStacks.set(id, jobs);
jobs.forEach((job) => job.start());
Logger.info(
'[schedule][创建定时任务] 任务ID: %s, 名称: %s, cron: %s, 执行命令: %s',
'[schedule][创建定时任务] 任务ID: %s, 规则数: %s',
id,
name,
schedule,
command,
jobs.length,
);
if (extra_schedules?.length) {
extra_schedules.forEach((x) => {
Logger.info(
'[schedule][创建定时任务] 任务ID: %s, 名称: %s, cron: %s, 执行命令: %s',
id,
name,
x.schedule,
command,
);
});
}
const mainJob = nodeSchedule.scheduleJob(id, schedule, async () => {
Logger.info(`[schedule][准备运行任务] 命令: ${command}`);
runCron(command, item);
});
if (!mainJob) {
Logger.warn(
'[schedule][创建定时任务] scheduleJob 返回 null(不符合预期,已通过预校验): 任务ID: %s, cron: %s',
id,
schedule,
);
}
const extraJobs = extra_schedules?.length
? extra_schedules.map((x) => {
const job = nodeSchedule.scheduleJob(id, x.schedule, async () => {
Logger.info(`[schedule][准备运行任务] 命令: ${command}`);
runCron(command, item);
});
if (!job) {
Logger.warn(
'[schedule][创建定时任务] scheduleJob 返回 null(不符合预期,已通过预校验): 任务ID: %s, cron: %s',
id,
x.schedule,
);
}
return job;
})
: [];
// 过滤 null(兜底保护,正常情况下预校验已拦截)
const jobs = [mainJob, ...extraJobs].filter((x) => x != null);
if (jobs.length > 0) {
scheduleStacks.set(id, jobs);
}
}
callback(null, null);
+26 -1
View File
@@ -62,16 +62,41 @@ class Client {
return this._client;
}
async setConcurrency(concurrency: number): Promise<void> {
await this.waitForReady(2000);
return new Promise((resolve, reject) => {
this.client.setConcurrency(
{ concurrency }, new Metadata(), { deadline: Date.now() + 5000 },
(err) => {
if (err) {
if (err.code === status.UNAVAILABLE || err.code === status.DEADLINE_EXCEEDED) {
this.readiness.invalidate();
Object.assign(err, { status: 503 });
}
return reject(err);
}
resolve();
},
);
});
}
async addCron(
request: AddCronRequest['crons'],
replace = false
): Promise<AddCronResponse> {
await this.waitForReady(2000);
// Include every rule in a recovery snapshot, but bound a stuck worker.
const ruleCount = request.reduce(
(count, item) => count + 1 + (item.extra_schedules?.length || 0),
0,
);
const registrationTimeoutMs = Math.min(120000, 5000 + ruleCount * 5);
return new Promise((resolve, reject) => {
this.client.addCron(
{ crons: request, replace },
new Metadata(),
{ deadline: Date.now() + 5000 },
{ deadline: Date.now() + registrationTimeoutMs },
(err, res) => {
if (err) {
if (err.code === status.UNAVAILABLE || err.code === status.DEADLINE_EXCEEDED) {
+2 -2
View File
@@ -1,6 +1,6 @@
import nodeSchedule from 'node-schedule';
import { CronJob } from '../shared/cronScheduler';
import { ToadScheduler } from 'toad-scheduler';
export const scheduleStacks = new Map<string, nodeSchedule.Job[]>();
export const scheduleStacks = new Map<string, CronJob[]>();
export const intervalSchedule = new ToadScheduler();
+1 -1
View File
@@ -13,7 +13,7 @@ const delCron = (
'[schedule][取消定时任务] 任务ID: %s',
id,
);
// 过滤掉 nodeSchedule.scheduleJob() 对无效表达式返回的 null,
// 防御性过滤历史调度栈中的空任务,
// 否则对 null 调 cancel() 会让整个取消流程抛出 UNKNOWN 错误,
// 进而导致 HTTP 端的 remove() 跳过 setCrontab(),造成 crontab.list 残留。
scheduleStacks.get(id)?.filter((x) => x != null).forEach((x) => {
+26
View File
@@ -0,0 +1,26 @@
import { ServerUnaryCall, sendUnaryData, status } from '@grpc/grpc-js';
import os from 'os';
import { AddCronResponse, SetConcurrencyRequest } from '../protos/cron';
import taskLimit from '../shared/pLimit';
export async function setConcurrency(
call: ServerUnaryCall<SetConcurrencyRequest, AddCronResponse>,
callback: sendUnaryData<AddCronResponse>,
) {
const limit = call.request.concurrency;
if (!Number.isInteger(limit) || limit < 0 || limit > 2147483647) {
callback(
Object.assign(new Error('Invalid concurrency'), {
code: status.INVALID_ARGUMENT,
}),
null,
);
return;
}
try {
await taskLimit.setCustomLimit(limit || Math.max(os.cpus().length, 4));
callback(null, {});
} catch (error) {
callback(error as Error, null);
}
}
+3 -4
View File
@@ -12,13 +12,12 @@ export default class ConfigService {
public async getFile(filePath: string, res: Response) {
let content = '';
if (!filePath) {
if (!filePath || filePath.startsWith('data/scripts/')) {
return res.send({ code: 403, message: t('文件无法访问') });
}
const scriptFile = filePath.startsWith('data/scripts/');
const resolved = resolveFileAccess(
scriptFile ? config.scriptPath : config.configPath,
[scriptFile ? filePath.slice('data/scripts/'.length) : filePath],
config.configPath,
[filePath],
config.blackFileList,
);
if (!resolved) {
+22 -9
View File
@@ -1,5 +1,5 @@
import { randomUUID } from 'crypto';
import { getInvalidCronSchedules } from '../shared/cronSchedule';
import { getInvalidCronSchedules, isValidCronSchedule } from '../shared/cronSchedule';
import {
withSchedulerMutation,
schedulerRegistrationError,
@@ -7,6 +7,7 @@ import {
import { Service, Inject } from 'typedi';
import winston from 'winston';
import config from '../config';
import { resolveLogPath } from '../shared/logPath';
import { Crontab, CrontabModel, CrontabStatus } from '../data/cron';
import {
RunningInstanceModel,
@@ -14,7 +15,6 @@ import {
} from '../data/runningInstance';
import { exec, execSync } from 'child_process';
import fs from 'fs/promises';
import CronExpressionParser from 'cron-parser';
import {
getFileContentByName,
fileExist,
@@ -47,12 +47,20 @@ export default class CronService {
private isNodeCron(cron: Crontab) {
const { schedule, extra_schedules } = cron;
const fields = schedule?.trim().split(/\s+/) || [];
// System crontab only receives portable numeric five-field expressions.
// Extended syntax, macros and legacy shorthand belong to node-schedule.
// Extended syntax, macros and legacy shorthand use the Node scheduler.
return (
schedule?.trim().split(/\s+/).length !== 5 ||
fields.length !== 5 ||
/[^\d\s*,/\-]/.test(schedule || '') ||
Boolean(extra_schedules?.length)
Boolean(extra_schedules?.length) ||
// BusyBox treats N/step as a single value and doesn't support Sunday=7.
fields.some((field) => field.split(',').some((part) => /^\d+\//.test(part))) ||
fields[4].includes('7') ||
// BusyBox steps DOM from zero; its full-range day/week wildcard
// handling also differs from the legacy Node parser's OR semantics.
fields[2].includes('/') ||
(fields[2] !== '*' && fields[4] !== '*')
);
}
@@ -239,6 +247,9 @@ export default class CronService {
last_execution_time: number;
exit_code?: number;
}) {
if (log_path && log_path !== '/dev/null') {
resolveLogPath(config.logPath, log_path);
}
let options: any = {
status,
pid,
@@ -918,7 +929,9 @@ export default class CronService {
truncated: false,
};
}
const absolutePath = path.resolve(config.logPath, `${doc.log_path}`);
const absolutePath = doc.log_path
? resolveLogPath(config.logPath, doc.log_path)
: '';
const logFileExist = doc.log_path && (await fileExist(absolutePath));
if (logFileExist) {
const chunk = await readLogChunk(`${absolutePath}`, options);
@@ -948,12 +961,12 @@ export default class CronService {
public async logs(id: number) {
const doc = await this.getDb({ id });
if (!doc || !doc.log_path) {
if (!doc || !doc.log_path || doc.log_path === '/dev/null') {
return [];
}
const relativeDir = path.dirname(`${doc.log_path}`);
const dir = path.resolve(config.logPath, relativeDir);
const dir = path.dirname(resolveLogPath(config.logPath, doc.log_path));
const dirExist = await fileExist(dir);
if (dirExist) {
let files = await fs.readdir(dir);
@@ -1057,7 +1070,7 @@ export default class CronService {
if (
command &&
schedule &&
CronExpressionParser.parse(schedule).hasNext()
isValidCronSchedule(schedule)
) {
const name = namePrefix + '_' + index;
+34 -12
View File
@@ -32,10 +32,11 @@ export default class DependenceService {
constructor(
@Inject('logger') private logger: winston.Logger,
private sockService: SockService,
) { }
) {}
public async create(payloads: Dependence[]): Promise<Dependence[]> {
const tabs = payloads.map((x) => {
getInstallCommand(x.type, x.name);
const tab = new Dependence({ ...x, status: DependenceStatus.queued });
return tab;
});
@@ -59,6 +60,7 @@ export default class DependenceService {
...other,
status: DependenceStatus.queued,
});
getInstallCommand(tab.type, tab.name);
const newDoc = await this.updateDb(tab);
this.installDependenceOneByOne([newDoc]);
return newDoc;
@@ -296,6 +298,23 @@ export default class DependenceService {
force: boolean = false,
) {
return taskLimit.runDependeny(dependency, () => {
let validatedCommand: string;
try {
validatedCommand = isInstall
? getInstallCommand(dependency.type, dependency.name.trim())
: getUninstallCommand(dependency.type, dependency.name.trim());
} catch (error) {
taskLimit.removeQueuedDependency(dependency);
return DependenceModel.update(
{
status: isInstall
? DependenceStatus.installFailed
: DependenceStatus.removeFailed,
log: [String(error)],
},
{ where: { id: dependency.id } },
).then(() => null);
}
return new Promise(async (resolve) => {
if (taskLimit.firstDependencyId !== dependency.id) {
return resolve(null);
@@ -345,9 +364,7 @@ export default class DependenceService {
: DependenceStatus.removing;
await DependenceModel.update({ status }, { where: { id: depIds } });
let command = isInstall
? getInstallCommand(dependency.type, depName)
: getUninstallCommand(dependency.type, depName);
let command = validatedCommand;
if (isLinuxDependence) {
command = isInstall
? `${linuxCommand.install} ${depName.trim()}`
@@ -470,15 +487,20 @@ export default class DependenceService {
const message =
'\n' +
tf('依赖%s%s,结束时间 %s,耗时 %s 秒',
actionText,
resultText,
endTime.format('YYYY-MM-DD HH:mm:ss'),
String(endTime.diff(startTime, 'second')),
);
tf(
'依赖%s%s,结束时间 %s,耗时 %s 秒',
actionText,
resultText,
endTime.format('YYYY-MM-DD HH:mm:ss'),
String(endTime.diff(startTime, 'second')),
);
const exitStatus = isSucceed
? (isInstall ? DependenceStatus.installed : DependenceStatus.removed)
: (isInstall ? DependenceStatus.installFailed : DependenceStatus.removeFailed);
? isInstall
? DependenceStatus.installed
: DependenceStatus.removed
: isInstall
? DependenceStatus.installFailed
: DependenceStatus.removeFailed;
this.sockService.sendMessage({
type: socketMessageType,
message,
+3 -4
View File
@@ -262,10 +262,9 @@ export default class EnvService {
.map((x) => x.value)
.join('&')
.replace(/\\/g, '\\\\')}`;
js_env_string += `process.env.${key}=\`${_env_value.replace(
/\`/g,
'\\`',
)}\`;\n`;
js_env_string += `process.env[${JSON.stringify(key)}]=${JSON.stringify(
group.map((x) => x.value).join('&'),
)};\n`;
py_env_string += `os.environ['${key}']='''${_env_value.replace(
/\'/g,
"\\'",
+2 -1
View File
@@ -4,6 +4,7 @@ import { HealthService } from '../protos/health';
import { ApiService } from '../protos/api';
import { addCron } from '../schedule/addCron';
import { delCron } from '../schedule/delCron';
import { setConcurrency } from '../schedule/setConcurrency';
import { check } from '../schedule/health';
import * as Api from '../schedule/api';
import Logger from '../loaders/logger';
@@ -27,7 +28,7 @@ export class GrpcServerService {
async initialize() {
try {
this.server.addService(HealthService, { check });
this.server.addService(CronService, { addCron, delCron });
this.server.addService(CronService, { addCron, delCron, setConcurrency });
this.server.addService(ApiService, Api);
const tlsConfig = await initGrpcCerts();
+1 -6
View File
@@ -1,5 +1,4 @@
import { resolveFileAccess } from '../shared/fileAccess';
import path from 'path';
import { Inject, Service } from 'typedi';
import winston from 'winston';
import config from '../config';
@@ -9,10 +8,6 @@ export default class LogService {
constructor(@Inject('logger') private logger: winston.Logger) {}
public checkFilePath(filePath: string, fileName: string) {
return resolveFileAccess(
config.logPath,
[filePath || '', fileName],
config.blackFileList,
);
return resolveFileAccess(config.logPath, [filePath || '', fileName]);
}
}
+116 -34
View File
@@ -6,27 +6,77 @@ import { v4 as uuidV4 } from 'uuid';
import sequelize, { Op } from 'sequelize';
import { shareStore } from '../shared/store';
import { t } from '../shared/i18n';
import { lock } from 'proper-lockfile';
import path from 'path';
import os from 'os';
import { writeFile, rename, rm, stat, chmod } from 'fs/promises';
import config from '../config';
@Service()
export default class OpenService {
constructor(@Inject('logger') private logger: winston.Logger) {}
// Every database mutation and full authentication-cache publication shares
// this cross-process lock. Lock a separate marker, not token.json itself.
private async withAppsLock<T>(operation: () => Promise<T>): Promise<T> {
const target = path.join(config.configPath, '.apps-state');
const release = await lock(target, {
realpath: false,
stale: 10_000,
retries: { retries: 10, factor: 2, minTimeout: 100, maxTimeout: 3000 },
});
try {
return await operation();
} finally {
await release();
}
}
public async initializeSystemApp(): Promise<void> {
await this.withAppsLock(async () => {
const doc = await AppModel.findOne({ where: { name: 'system' } });
const app = doc?.get({ plain: true });
if (!app) {
await this.insert({
name: 'system',
scopes: ['crons', 'system', 'dashboard'],
client_id: createRandomString(12, 12),
client_secret: createRandomString(24, 24),
});
} else if (!app.scopes.includes('dashboard')) {
await AppModel.update(
{ scopes: [...app.scopes, 'dashboard'] },
{ where: { id: app.id } },
);
}
await shareStore.updateApps(await this.find({}));
});
}
public async refreshApps(): Promise<void> {
await this.withAppsLock(async () => {
await shareStore.updateApps(await this.find({}));
});
}
public async findApps(): Promise<App[] | null> {
const docs = await this.find({});
return docs;
}
public async create(payload: App): Promise<App> {
const tab = { ...payload };
tab.client_id = createRandomString(12, 12);
tab.client_secret = createRandomString(24, 24);
const doc = await this.insert(tab);
const apps = await this.find({});
await shareStore.updateApps(apps);
return { ...doc, tokens: [] };
return this.withAppsLock(async () => {
const tab = { ...payload };
tab.client_id = createRandomString(12, 12);
tab.client_secret = createRandomString(24, 24);
const doc = await this.insert(tab);
const apps = await this.find({});
await shareStore.updateApps(apps);
return { ...doc, tokens: [] };
});
}
public async insert(payload: App): Promise<App> {
private async insert(payload: App): Promise<App> {
const doc = await AppModel.create(payload, { returning: true });
return doc.get({ plain: true });
}
@@ -41,10 +91,12 @@ export default class OpenService {
}
private async updateDb(payload: Partial<App>): Promise<App> {
await AppModel.update(payload, { where: { id: payload.id } });
const apps = await this.find({});
await shareStore.updateApps(apps);
return apps?.find((x) => x.id === payload.id) as App;
return this.withAppsLock(async () => {
await AppModel.update(payload, { where: { id: payload.id } });
const apps = await this.find({});
await shareStore.updateApps(apps);
return apps?.find((x) => x.id === payload.id) as App;
});
}
public async getDb(query: Record<string, any>): Promise<App> {
@@ -56,9 +108,11 @@ export default class OpenService {
}
public async remove(ids: number[]) {
await AppModel.destroy({ where: { id: ids } });
const apps = await this.find({});
await shareStore.updateApps(apps);
return this.withAppsLock(async () => {
await AppModel.destroy({ where: { id: ids } });
const apps = await this.find({});
await shareStore.updateApps(apps);
});
}
public async resetSecret(id: number): Promise<App> {
@@ -112,7 +166,15 @@ export default class OpenService {
return docs.map((x) => x.get({ plain: true }));
}
public async authToken({
public async authToken(credentials: {
client_id: string;
client_secret: string;
}): Promise<any> {
return this.withAppsLock(() => this.issueToken(credentials));
}
// Caller holds withAppsLock through database update and cache publication.
private async issueToken({
client_id,
client_secret,
}: {
@@ -156,27 +218,47 @@ export default class OpenService {
}
}
public async generateSystemToken(): Promise<{
public async generateSystemToken(renew = false): Promise<{
value: string;
expiration: number;
}> {
const apps = await shareStore.getApps();
const systemApp = apps?.find((x) => x.name === 'system');
if (!systemApp) {
throw new Error('system app not found');
}
const now = Math.round(Date.now() / 1000);
const currentToken = systemApp.tokens?.find((x) => x.expiration > now);
if (currentToken) {
return currentToken;
}
const { data } = await this.authToken({
client_id: systemApp.client_id,
client_secret: systemApp.client_secret,
return this.withAppsLock(async () => {
// Read after acquiring the lock; a concurrent issuer may have renewed.
const systemApp = await this.getDb({ name: 'system' });
const now = Math.round(Date.now() / 1000);
let appToken = systemApp.tokens
?.filter((token) => token.expiration > now)
.sort((left, right) => right.expiration - left.expiration)[0];
if (renew || !appToken) {
const { data } = await this.issueToken({
client_id: systemApp.client_id,
client_secret: systemApp.client_secret,
});
appToken = { value: data.token, expiration: data.expiration };
} else {
// Recover a prior database commit whose cache publication failed.
await shareStore.updateApps(await this.find({}));
}
const tokenFile = path.join(config.configPath, 'token.json');
const temporaryFile = `${tokenFile}.${process.pid}.tmp`;
const mode = await stat(tokenFile).then(
(file) => file.mode & 0o777,
(error: NodeJS.ErrnoException) => {
if (error.code === 'ENOENT') return 0o600;
throw error;
},
);
try {
await writeFile(temporaryFile, `${JSON.stringify(appToken)}${os.EOL}`, {
mode,
});
await chmod(temporaryFile, mode);
// Shell readers see a complete old or new document, never a partial one.
await rename(temporaryFile, tokenFile);
} finally {
await rm(temporaryFile, { force: true });
}
return appToken;
});
return {
...data,
value: data.token,
};
}
}
+33 -14
View File
@@ -1,6 +1,6 @@
import { Service, Inject } from 'typedi';
import winston from 'winston';
import nodeSchedule from 'node-schedule';
import { createCronJob, CronJob } from '../shared/cronScheduler';
import { ChildProcessWithoutNullStreams } from 'child_process';
import {
ToadScheduler,
@@ -11,7 +11,11 @@ import {
import dayjs from 'dayjs';
import taskLimit from '../shared/pLimit';
import { spawn } from 'cross-spawn';
import { observeChildProcess, asError, ProcessResult } from '../shared/childProcess';
import {
observeChildProcess,
asError,
ProcessResult,
} from '../shared/childProcess';
export interface ScheduleTaskType {
id?: number;
@@ -38,7 +42,7 @@ export interface TaskCallbacks {
@Service()
export default class ScheduleService {
private scheduleStacks = new Map<string, nodeSchedule.Job>();
private scheduleStacks = new Map<string, CronJob>();
private intervalSchedule = new ToadScheduler();
@@ -159,18 +163,33 @@ export default class ScheduleService {
command,
);
this.scheduleStacks.set(
_id,
nodeSchedule.scheduleJob(_id, schedule, async () => {
this.runTask(command, callbacks, {
name,
if (schedule) {
try {
const job = createCronJob(
schedule,
command,
id: _id,
runOrigin,
});
}),
);
() =>
this.runTask(command, callbacks, {
name,
schedule,
command,
id: _id,
runOrigin,
}),
{ name: `${_id}: ${name || ''}`, logger: this.logger, start: false },
);
this.scheduleStacks.get(_id)?.cancel();
this.scheduleStacks.set(_id, job);
job.start();
} catch (error) {
// A persisted invalid subscription must not become an unhandled startup rejection.
this.logger.warn(
'[panel][跳过无效定时任务] 任务ID: %s, cron: %s, 错误: %s',
_id,
schedule,
asError(error).message,
);
}
}
if (runImmediately) {
this.runTask(command, callbacks, {
+1 -5
View File
@@ -66,11 +66,7 @@ export default class ScriptService {
}
public checkFilePath(filePath: string, fileName: string) {
return resolveFileAccess(
config.scriptPath,
[filePath || '', fileName],
config.blackFileList,
);
return resolveFileAccess(config.scriptPath, [filePath || '', fileName]);
}
public async getFile(filePath: string, fileName: string) {
+12
View File
@@ -0,0 +1,12 @@
import path from 'path';
import { Service } from 'typedi';
import config from '../config';
import { ScriptHistory } from '../shared/scriptHistory';
/** Panel configuration and lifecycle; the storage core stays independently testable. */
@Service()
export default class ScriptHistoryService extends ScriptHistory {
constructor() {
super(config.scriptPath, path.join(config.dataPath, 'script-history'));
}
}
+43 -23
View File
@@ -8,10 +8,22 @@ import { formatUrl } from '../config/subscription';
import config from '../config';
import { fileExist, rmPath } from '../config/util';
import { writeFileWithLock } from '../shared/utils';
import {
getSubscriptionSshAlias,
resolveSubscriptionPath,
} from '../shared/subscriptionPath';
@Service()
export default class SshKeyService {
private homedir = os.homedir();
// OpenSSH uses the effective user's passwd entry, not the HOME override.
private get homedir(): string {
try {
return os.userInfo().homedir;
} catch {
// Custom container UIDs may have no passwd entry. Keep startup working.
return os.homedir();
}
}
private sshPath = config.sshdPath;
private sshConfigFilePath = path.resolve(this.homedir, '.ssh', 'config');
private sshConfigHeader = `Include ${path.join(this.sshPath, '*.config')}`;
@@ -21,6 +33,10 @@ export default class SshKeyService {
}
private async initSshConfigFile() {
await fs.mkdir(path.dirname(this.sshConfigFilePath), {
recursive: true,
mode: 0o700,
});
let config = '';
const _exist = await fileExist(this.sshConfigFilePath);
if (_exist) {
@@ -42,7 +58,7 @@ export default class SshKeyService {
key: string,
): Promise<void> {
try {
const filePath = path.join(this.sshPath, alias);
const filePath = resolveSubscriptionPath(this.sshPath, alias);
try {
await rmPath(filePath);
} catch { }
@@ -54,7 +70,7 @@ export default class SshKeyService {
private async removePrivateKeyFile(alias: string): Promise<void> {
try {
const filePath = path.join(this.sshPath, alias);
const filePath = resolveSubscriptionPath(this.sshPath, alias);
await rmPath(filePath);
} catch (error) {
this.logger.error('删除私钥文件失败', error);
@@ -66,18 +82,19 @@ export default class SshKeyService {
host: string,
proxy?: string,
) {
if (!/^[a-zA-Z0-9.-]+$/.test(host) ||
(proxy && !/^[a-zA-Z0-9.:[\]-]+$/.test(proxy))) {
throw Object.assign(new Error('Invalid SSH host or proxy'), { status: 400 });
}
if (host === 'github.com') {
host = `ssh.github.com\n Port 443\n HostkeyAlgorithms +ssh-rsa`;
}
const proxyStr = proxy
? ` ProxyCommand nc -v -x ${proxy} %h %p 2>/dev/null\n`
: '';
const config = `Host ${alias}\n Hostname ${host}\n IdentityFile ${path.join(
this.sshPath,
alias,
)}\n StrictHostKeyChecking no\n${proxyStr}`;
const config = `Host ${alias}\n Hostname ${host}\n IdentityFile ${resolveSubscriptionPath(this.sshPath, alias)}\n StrictHostKeyChecking no\n${proxyStr}`;
await writeFileWithLock(
`${path.join(this.sshPath, `${alias}.config`)}`,
`${resolveSubscriptionPath(this.sshPath, alias, '.config')}`,
config,
{
encoding: 'utf8',
@@ -88,7 +105,7 @@ export default class SshKeyService {
private async removeSshConfig(alias: string) {
try {
const filePath = path.join(this.sshPath, `${alias}.config`);
const filePath = resolveSubscriptionPath(this.sshPath, alias, '.config');
await rmPath(filePath);
} catch (error) {
this.logger.error(`删除ssh配置文件${alias}失败`, error);
@@ -101,6 +118,7 @@ export default class SshKeyService {
host: string,
proxy?: string,
): Promise<void> {
alias = getSubscriptionSshAlias(alias);
await this.generatePrivateKeyFile(alias, key);
await this.generateSingleSshConfig(alias, host, proxy);
}
@@ -110,6 +128,7 @@ export default class SshKeyService {
host: string,
proxy?: string,
): Promise<void> {
alias = getSubscriptionSshAlias(alias);
await this.removePrivateKeyFile(alias);
await this.removeSshConfig(alias);
}
@@ -117,15 +136,19 @@ export default class SshKeyService {
public async setSshConfig(docs: Subscription[]) {
for (const doc of docs) {
if (doc.type === 'private-repo' && doc.pull_type === 'ssh-key') {
const { alias, proxy } = doc;
const { host } = formatUrl(doc);
await this.removePrivateKeyFile(alias);
await this.removeSshConfig(alias);
await this.generatePrivateKeyFile(
alias,
(doc.pull_option as any).private_key,
);
await this.generateSingleSshConfig(alias, host, proxy);
try {
const alias = getSubscriptionSshAlias(doc.alias);
const { host } = formatUrl(doc);
await this.removePrivateKeyFile(alias);
await this.removeSshConfig(alias);
await this.generatePrivateKeyFile(
alias,
(doc.pull_option as any).private_key,
);
await this.generateSingleSshConfig(alias, host, doc.proxy);
} catch (error) {
this.logger.warn('跳过订阅 %s 的无效 SSH 配置: %s', doc.id, error);
}
}
}
}
@@ -144,12 +167,9 @@ export default class SshKeyService {
private async generateGlobalSshConfig(alias: string) {
// Create a config that matches all hosts, making this key globally available
const config = `Host *\n IdentityFile ${path.join(
this.sshPath,
alias,
)}\n StrictHostKeyChecking no\n`;
const config = `Host *\n IdentityFile ${resolveSubscriptionPath(this.sshPath, alias)}\n StrictHostKeyChecking no\n`;
await writeFileWithLock(
`${path.join(this.sshPath, `${alias}.config`)}`,
`${resolveSubscriptionPath(this.sshPath, alias, '.config')}`,
config,
{
encoding: 'utf8',
+107 -32
View File
@@ -20,7 +20,7 @@ import {
} from '../config/util';
import fs from 'fs/promises';
import { FindOptions, Op } from 'sequelize';
import path, { join } from 'path';
import path from 'path';
import ScheduleService, { TaskCallbacks } from './schedule';
import { SimpleIntervalSchedule } from 'toad-scheduler';
import SockService from './sock';
@@ -30,6 +30,14 @@ import dayjs from 'dayjs';
import { LOG_END_SYMBOL } from '../config/const';
import { formatCommand, formatUrl } from '../config/subscription';
import { CrontabModel } from '../data/cron';
import { assertSubscriptionAlias } from '../shared/subscriptionPath';
import { resolveLogPath } from '../shared/logPath';
import {
getSubscriptionStorageCandidates,
isSubscriptionStorageName,
resolveSubscriptionStoragePath,
} from '../shared/subscriptionStorage';
import { withSubscriptionMutation } from '../shared/subscriptionMutationLock';
import CrontabService from './cron';
import taskLimit from '../shared/pLimit';
import { logStreamManager } from '../shared/logStreamManager';
@@ -88,9 +96,11 @@ export default class SubscriptionService {
needCreate = true,
runImmediately = false,
) {
const { url } = formatUrl(doc);
doc.command = formatCommand(doc, url as string);
// Cancelling by ID must also work for invalid historical subscriptions.
if (needCreate) {
const { url } = formatUrl(doc);
doc.command = formatCommand(doc, url as string);
}
if (doc.schedule_type === 'crontab') {
this.scheduleService.cancelCronTask(doc as any);
@@ -222,11 +232,14 @@ export default class SubscriptionService {
}
public async create(payload: Subscription): Promise<Subscription> {
const tab = new Subscription(payload);
const doc = await this.insert(tab);
await this.handleTask(doc.get({ plain: true }));
await this.setSshConfig();
return doc;
assertSubscriptionAlias(payload.alias);
return withSubscriptionMutation(async () => {
const tab = new Subscription(payload);
const doc = await this.insert(tab);
await this.handleTask(doc.get({ plain: true }));
await this.setSshConfig();
return doc;
});
}
public async insert(payload: Subscription): Promise<SubscriptionInstance> {
@@ -234,12 +247,27 @@ export default class SubscriptionService {
}
public async update(payload: Subscription): Promise<Subscription> {
const doc = await this.getDb({ id: payload.id });
const tab = new Subscription({ ...doc, ...payload });
const newDoc = await this.updateDb(tab);
await this.handleTask(newDoc, !newDoc.is_disabled);
await this.setSshConfig();
return newDoc;
assertSubscriptionAlias(payload.alias);
return withSubscriptionMutation(async () => {
const doc = await this.getDb({ id: payload.id });
const tab = new Subscription({ ...doc, ...payload });
const newDoc = await this.updateDb(tab);
await this.handleTask(newDoc, !newDoc.is_disabled);
await this.removeSshConfigForSubscription(doc);
await this.setSshConfig();
return newDoc;
});
}
private async removeSshConfigForSubscription(doc: Subscription) {
if (doc.type !== 'private-repo' || doc.pull_type !== 'ssh-key') return;
// Invalid historical rows must remain deletable without deriving unsafe paths.
try {
assertSubscriptionAlias(doc.alias);
} catch {
return;
}
await this.sshKeyService.removeSSHKey(doc.alias, '');
}
public async updateDb(payload: Subscription): Promise<Subscription> {
@@ -262,6 +290,7 @@ export default class SubscriptionService {
last_running_time: number;
last_execution_time: number;
}) {
if (log_path) resolveLogPath(config.logPath, log_path);
const options: any = {
status,
pid,
@@ -279,25 +308,71 @@ export default class SubscriptionService {
}
public async remove(ids: number[], query: { force?: boolean }) {
const docs = await SubscriptionModel.findAll({ where: { id: ids } });
for (const doc of docs) {
await this.handleTask(doc.get({ plain: true }), false);
}
await SubscriptionModel.destroy({ where: { id: ids } });
await this.setSshConfig();
if (query?.force === true) {
const crons = await CrontabModel.findAll({ where: { sub_id: ids } });
if (crons?.length) {
await this.crontabService.remove(crons.map((x) => x.id!));
return withSubscriptionMutation(async () => {
const docs = await SubscriptionModel.findAll({ where: { id: ids } });
const storagePaths: string[] = [];
const retainedPaths: string[] = [];
const collectPaths = (
doc: Subscription,
targets: string[],
deleting = false,
) => {
for (const names of getSubscriptionStorageCandidates(doc)) {
// A URL accepted by one engine can produce an invalid name in another.
// Validate each candidate independently, retaining the safe candidate.
if (!Object.values(names).every(isSubscriptionStorageName)) continue;
const resolve = deleting ? resolveSubscriptionStoragePath : path.resolve;
const scriptPath = resolve(config.scriptPath, names.script);
if (scriptPath) targets.push(scriptPath);
if (names.repo) {
const repoPath = resolve(config.repoPath, names.repo);
if (repoPath) targets.push(repoPath);
}
if (names.raw) {
const rawPath = resolve(path.join(config.dataPath, 'raw'), names.raw);
if (rawPath) targets.push(rawPath);
}
}
};
// Validate all deletion targets before changing rows or removing any files.
if (query?.force === true) {
for (const doc of docs) {
assertSubscriptionAlias(doc.alias);
resolveSubscriptionStoragePath(config.scriptPath, doc.alias);
resolveSubscriptionStoragePath(config.repoPath, doc.alias);
collectPaths(doc, storagePaths, true);
}
const remaining = await SubscriptionModel.findAll({ where: {} });
for (const doc of remaining) {
if (!ids.includes(doc.id!)) collectPaths(doc, retainedPaths);
}
}
for (const doc of docs) {
const filePath = join(config.scriptPath, doc.alias);
const repoPath = join(config.repoPath, doc.alias);
await rmPath(filePath);
await rmPath(repoPath);
await this.handleTask(doc.get({ plain: true }), false);
}
}
await SubscriptionModel.destroy({ where: { id: ids } });
for (const doc of docs) {
await this.removeSshConfigForSubscription(doc);
}
await this.setSshConfig();
if (query?.force === true) {
const crons = await CrontabModel.findAll({ where: { sub_id: ids } });
if (crons?.length) {
await this.crontabService.remove(crons.map((x) => x.id!));
}
for (const storagePath of new Set(storagePaths)) {
// rm is recursive: an ancestor must also be retained if a surviving
// subscription uses a nested branch, and vice versa.
const shared = retainedPaths.some((retained) =>
retained === storagePath ||
retained.startsWith(`${storagePath}${path.sep}`) ||
storagePath.startsWith(`${retained}${path.sep}`),
);
if (!shared) await rmPath(storagePath);
}
}
});
}
public async getDb(
@@ -399,7 +474,7 @@ export default class SubscriptionService {
if (doc.log_path) {
const relativeDir = path.dirname(`${doc.log_path}`);
const dir = path.resolve(config.logPath, relativeDir);
const dir = path.dirname(resolveLogPath(config.logPath, doc.log_path));
const _exist = await fileExist(dir);
if (_exist) {
let files = await fs.readdir(dir);
+52 -22
View File
@@ -1,4 +1,7 @@
import { spawn } from 'cross-spawn';
import { execFile } from 'child_process';
import { promisify } from 'util';
import { shellQuote } from '../shared/shellQuote';
import { Response } from 'express';
import fs from 'fs';
import { Agent, request } from 'undici';
@@ -31,6 +34,8 @@ import {
SystemModelInfo,
} from '../data/system';
import taskLimit from '../shared/pLimit';
import cronClient from '../schedule/client';
import { withSchedulerMutation } from '../shared/schedulerMutationLock';
import NotificationService from './notify';
import ScheduleService, { TaskCallbacks } from './schedule';
import SockService from './sock';
@@ -119,15 +124,29 @@ export default class SystemService {
}
public async updateCronConcurrency(info: SystemModelInfo) {
const oDoc = await this.getSystemConfig();
await this.updateAuthDb({
...oDoc,
info: { ...oDoc.info, ...info },
});
if (info.cronConcurrency) {
await taskLimit.setCustomLimit(info.cronConcurrency);
const limit = info.cronConcurrency ?? 0;
if (!Number.isInteger(limit) || limit < 0 || limit > 2147483647) {
throw Object.assign(new Error(t('参数错误')), { status: 400 });
}
return { code: 200, data: info };
return withSchedulerMutation(async () => {
const oDoc = await this.getSystemConfig();
await this.updateAuthDb({
...oDoc,
info: { ...oDoc.info, ...info, cronConcurrency: limit },
});
await taskLimit.setCustomLimit(limit || Math.max(os.cpus().length, 4));
await cronClient.setConcurrency(limit);
return { code: 200, data: info };
});
}
public async restoreCronConcurrency() {
await withSchedulerMutation(async () => {
const doc = await this.getSystemConfig();
const limit = doc.info?.cronConcurrency ?? 0;
await taskLimit.setCustomLimit(limit || Math.max(os.cpus().length, 4));
await cronClient.setConcurrency(limit);
});
}
public async updateDependenceProxy(info: SystemModelInfo) {
@@ -139,7 +158,7 @@ export default class SystemService {
if (info.dependenceProxy) {
await fs.promises.writeFile(
config.dependenceProxyFile,
`export http_proxy="${info.dependenceProxy}"\nexport https_proxy="${info.dependenceProxy}"`,
`export http_proxy=${shellQuote(info.dependenceProxy)}\nexport https_proxy=${shellQuote(info.dependenceProxy)}`,
);
} else {
await fs.promises.rm(config.dependenceProxyFile);
@@ -155,7 +174,7 @@ export default class SystemService {
});
let cmd = 'pnpm config delete registry';
if (info.nodeMirror) {
cmd = `pnpm config set registry ${info.nodeMirror}`;
cmd = `pnpm config set registry ${shellQuote(info.nodeMirror)}`;
}
let command = `cd && ${cmd}`;
const docs = await DependenceModel.findAll({
@@ -204,7 +223,7 @@ export default class SystemService {
});
let cmd = 'pip config unset global.index-url';
if (info.pythonMirror) {
cmd = `pip3 config set global.index-url ${info.pythonMirror}`;
cmd = `pip3 config set global.index-url ${shellQuote(info.pythonMirror)}`;
}
await promiseExec(cmd);
return { code: 200, data: info };
@@ -220,7 +239,6 @@ export default class SystemService {
return;
}
const command = await updateLinuxMirrorFile(info.linuxMirror || '');
let hasError = false;
this.scheduleService.runTask(
command,
{
@@ -228,14 +246,15 @@ export default class SystemService {
res?.setHeader('QL-Task-Pid', `${cp.pid}`);
res?.end();
},
onEnd: async () => {
onEnd: async (cp) => {
this.sockService.sendMessage({
type: 'updateLinuxMirror',
message: 'update linux mirror end',
status: 'completed',
});
onEnd?.();
if (!hasError) {
// Tools such as sudo and apt may write warnings to stderr on success.
if (cp?.exitCode === 0 && cp.signalCode === null) {
await this.updateAuthDb({
...oDoc,
info: { ...oDoc.info, ...info },
@@ -243,7 +262,6 @@ export default class SystemService {
}
},
onError: async (message: string) => {
hasError = true;
this.sockService.sendMessage({ type: 'updateLinuxMirror', message });
},
onLog: async (message: string) => {
@@ -357,7 +375,10 @@ export default class SystemService {
}
public async reloadSystem(target?: 'system' | 'data') {
const cmd = `real_time=true ql reload ${target || ''}`;
if (target && !['system', 'data'].includes(target)) {
return { code: 400, message: t('参数错误') };
}
const cmd = `real_time=true ql reload ${shellQuote(target || '')}`;
const cp = spawn(cmd, {
shell: '/bin/bash',
detached: true,
@@ -438,11 +459,20 @@ export default class SystemService {
if (type && type.length) {
dataDirs = dataDirs.concat(type.filter((x) => x !== 'base'));
}
const dataPaths = dataDirs.map((dir) => `data/${dir}`);
await promiseExec(
`cd ${config.dataPath} && cd ../ && tar -zcvf ${config.dataTgzFile
} ${dataPaths.join(' ')}`,
);
const allowed = new Set(['db', 'upload', 'config', 'scripts', 'log', 'deps',
'syslog', 'dep_cache', 'raw', 'repo', 'ssh.d', 'script-history']);
if (dataDirs.some((dir) => !allowed.has(dir))) {
return res.status(400).send({ code: 400, message: t('参数错误') });
}
if (dataDirs.includes('scripts')) dataDirs.push('script-history');
// History is created lazily; old/fresh installations must still export.
if (!fs.existsSync(path.join(config.dataPath, 'script-history'))) {
dataDirs = dataDirs.filter((dir) => dir !== 'script-history');
}
const dataPaths = [...new Set(dataDirs)].map((dir) => `data/${dir}`);
await promisify(execFile)('tar', ['-zcf', config.dataTgzFile, '--', ...dataPaths], {
cwd: path.dirname(config.dataPath),
});
res.download(config.dataTgzFile);
} catch (error: any) {
return res.send({ code: 400, message: error.message });
@@ -565,7 +595,7 @@ export default class SystemService {
try {
await fs.promises.writeFile(
config.langEnvFile,
`export QL_LANG='${lang}'\n`,
`export QL_LANG=${shellQuote(lang)}\n`,
);
} catch (error) {
this.logger.error(`Failed to write lang_env.sh: ${error}`);
+15 -12
View File
@@ -33,9 +33,11 @@ import {
verifyPassword,
} from '../shared/password';
import { serializeAuthMutation } from '../shared/authMutation';
import { LoginThrottle } from '../shared/loginThrottle';
@Service()
export default class UserService {
private loginThrottle = new LoginThrottle();
@Inject((type) => NotificationService)
private notificationService!: NotificationService;
@@ -106,11 +108,8 @@ export default class UserService {
return { code: 403, message: t('该 IP 已被列入黑名单') };
}
const retriesTime = Math.pow(3, retries) * 1000;
if (retries > 2 && timestamp - lastlogon < retriesTime) {
const waitTime = Math.ceil(
(retriesTime - (timestamp - lastlogon)) / 1000,
);
const waitTime = this.loginThrottle.retryAfter(ip, timestamp);
if (waitTime > 0) {
return {
code: 410,
message: tf('失败次数过多,请%s秒后重试', waitTime),
@@ -133,6 +132,7 @@ export default class UserService {
}
if (passwordMatches) {
this.loginThrottle.reset(ip);
const data = createRandomString(50, 100);
const expiration = twoFactorActivated ? '60d' : '20d';
let token = jwt.sign({ data }, config.jwt.secret, {
@@ -206,8 +206,9 @@ export default class UserService {
},
};
} else {
const failures = this.loginThrottle.fail(ip, timestamp);
await this.updateAuthInfo(content, {
retries: retries + 1,
retries: failures,
lastlogon: timestamp,
lastip: ip,
lastaddr: address,
@@ -239,8 +240,8 @@ export default class UserService {
},
});
this.getLoginLog();
if (retries > 2) {
const waitTime = Math.round(Math.pow(3, retries + 1));
if (failures > 3) {
const waitTime = this.loginThrottle.retryAfter(ip, timestamp);
return {
code: 410,
message: tf('失败次数过多,请%s秒后重试', waitTime),
@@ -439,8 +440,8 @@ export default class UserService {
const authInfo = await this.getAuthInfo();
const { isTwoFactorChecking, twoFactorSecret } = authInfo;
const now = Date.now();
const retries = authInfo.retries || 0;
if (retries > 2 && now - authInfo.lastlogon < Math.pow(3, retries) * 1000) {
const source = getClientIp(req);
if (this.loginThrottle.retryAfter(source, now) > 0) {
return { code: 410, message: t('失败次数过多,请稍后重试') };
}
if (
@@ -472,9 +473,8 @@ export default class UserService {
.join(' ');
}
await this.updateAuthInfo(authInfo, {
retries: retries + 1,
retries: this.loginThrottle.fail(source, now),
lastlogon: now,
isTwoFactorChecking: retries + 1 < 5,
lastip: ip,
lastaddr: address,
platform: req.platform,
@@ -690,5 +690,8 @@ export default class UserService {
});
}
await this.updateAuthInfo(authInfo, payload);
if (retries === 0 || password !== undefined || username !== undefined) {
this.loginThrottle.clear();
}
}
}
+11
View File
@@ -0,0 +1,11 @@
// Match command text literally: looking up a task must never execute its input.
export function findCronId(crontab: string, command: string): string {
if (!command) return '';
for (const line of crontab.split('\n')) {
const id = /\bID=(\d+)\s/.exec(line);
if (id && line.includes(command, id.index + id[0].length)) {
return id[1];
}
}
return '';
}
+90 -7
View File
@@ -1,12 +1,93 @@
import cronParser from 'cron-parser-v4';
import CronExpressionParser from 'cron-parser';
import { validate } from 'node-cron';
import { ScheduleType } from '../interface/schedule';
// Keep validation aligned with node-schedule's locked cron-parser version.
// cron-parser 5 accepts syntax (e.g. H) that the scheduler cannot execute.
const aliases: Record<string, string> = {
'@yearly': '0 0 0 1 1 *',
'@annually': '0 0 0 1 1 *',
'@monthly': '0 0 0 1 * *',
'@weekly': '0 0 0 * * 0',
'@daily': '0 0 0 * * *',
'@midnight': '0 0 0 * * *',
'@hourly': '0 0 * * * *',
'@minutely': '0 * * * * *',
};
export interface CronSchedule {
patterns: string[];
dayCount: number;
nthDay: number;
}
// Validation and registration repeatedly see identical rules in large snapshots.
// Cache only immutable syntax, never dates or running tasks, and bound user input.
const scheduleCache = new Map<string, CronSchedule>();
// Canonicalize legacy shorthand, aliases, names and numeric-start steps before
// node-cron sees them. Extend legacy aliases explicitly; reject H and bare /N.
export function parseCronSchedule(schedule: unknown): CronSchedule {
if (typeof schedule !== 'string' || !schedule.trim()) {
throw new Error('Invalid cron schedule');
}
let source = schedule.trim();
const cacheKey = source;
const cached = scheduleCache.get(cacheKey);
if (cached) return cached;
if (source.startsWith('@')) {
if (!Object.hasOwn(aliases, source)) throw new Error('Invalid cron alias');
source = aliases[source];
}
let parts = source.split(/\s+/);
if (
parts.length > 6 ||
parts.some((part) => /(^|,)\//.test(part) || /H(?:\(|\/|$)/i.test(part))
) {
throw new Error('Unsupported cron syntax');
}
parts = [
...['0', '*', '*', '*', '*', '*'].slice(0, 6 - parts.length),
...parts,
];
if (
parts.some(
(part, index) => index !== 3 && index !== 5 && part.includes('?'),
)
) {
throw new Error('Question mark is only valid in day fields');
}
const expression = CronExpressionParser.parse(parts.join(' '));
if (!expression.hasNext())
throw new Error('Cron schedule has no next execution');
const normalized = expression.stringify(true).replace(/\?/g, '*').split(' ');
const dayCount = expression.fields.dayOfMonth.values.length;
const weekCount = expression.fields.dayOfWeek.values.length;
const patterns =
dayCount < 31 && weekCount < 8
? [
[...normalized.slice(0, 5), '*'].join(' '),
[...normalized.slice(0, 3), '*', ...normalized.slice(4)].join(' '),
]
: [normalized.join(' ')];
if (!patterns.every((pattern) => validate(pattern))) {
throw new Error('Unsupported cron schedule');
}
const parsed = {
patterns,
dayCount,
nthDay: expression.fields.dayOfWeek.nthDay,
};
Object.freeze(patterns);
Object.freeze(parsed);
if (scheduleCache.size >= 512)
scheduleCache.delete(scheduleCache.keys().next().value);
scheduleCache.set(cacheKey, parsed);
return parsed;
}
export function isValidCronSchedule(schedule: unknown): schedule is string {
if (typeof schedule !== 'string' || !schedule.trim()) return false;
try {
return cronParser.parseExpression(schedule).hasNext();
parseCronSchedule(schedule);
return true;
} catch {
return false;
}
@@ -22,6 +103,8 @@ export function getInvalidCronSchedules(cron: {
) {
return [];
}
return [cron.schedule, ...(cron.extra_schedules || []).map((x) => x.schedule)]
.filter((schedule) => !isValidCronSchedule(schedule));
return [
cron.schedule,
...(cron.extra_schedules || []).map((x) => x.schedule),
].filter((schedule) => !isValidCronSchedule(schedule));
}
+89
View File
@@ -0,0 +1,89 @@
import { createTask, ScheduledTask } from 'node-cron';
import { parseCronSchedule } from './cronSchedule';
interface SchedulerLogger {
warn(message: string, ...args: unknown[]): unknown;
error(message: string, ...args: unknown[]): unknown;
}
export interface CronJob {
start(): void;
cancel(): void;
}
// node-cron owns timers and calendar calculation; Qinglong owns execution
// concurrency and process lifetime. Missed in-process slots join that same queue.
export function createCronJob(
schedule: string,
callback: (date: Date) => unknown | Promise<unknown>,
options: { name: string; logger: SchedulerLogger; start?: boolean },
): CronJob {
const parsed = parseCronSchedule(schedule);
const tasks: ScheduledTask[] = [];
let cancelled = false;
let started = false;
const accepts = (index: number, date: Date) => {
if (tasks.length === 1) return true;
// Preserve cron-parser 4's day/weekday OR semantics, including its
// month-dependent full-day range and the global nth-week constraint.
if (parsed.nthDay && Math.ceil(date.getDate() / 7) !== parsed.nthDay)
return false;
if (index === 1) return !tasks[0].match(date);
const monthDays = [31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31];
return parsed.dayCount < monthDays[date.getMonth()] || tasks[1].match(date);
};
const dispatch = async (index: number, date: Date, missed: boolean) => {
if (cancelled || !started || !accepts(index, date)) return;
if (missed) {
options.logger.warn(
'[schedule][补执行迟到任务] 任务: %s, 计划时间: %s',
options.name,
date.toISOString(),
);
}
try {
await callback(date);
} catch (error) {
options.logger.error(
'[schedule][定时回调失败] 任务: %s, 错误: %s',
options.name,
error instanceof Error ? error.message : String(error),
);
}
};
try {
for (const [index, pattern] of parsed.patterns.entries()) {
const task = createTask(
pattern,
(context) => dispatch(index, context.date, false),
{
noOverlap: false,
missedExecutionTolerance: 5000,
},
);
tasks.push(task);
task.on('execution:missed', (context) =>
dispatch(index, context.date, true),
);
// Fail before an existing schedule is cancelled, including impossible dates.
task.getNextRuns(1);
}
} catch (error) {
tasks.forEach((task) => task.destroy());
throw error;
}
const job: CronJob = {
start() {
if (started || cancelled) return;
started = true;
tasks.forEach((task) => task.start());
},
cancel() {
if (cancelled) return;
cancelled = true;
tasks.forEach((task) => task.destroy());
},
};
if (options.start !== false) job.start();
return job;
}
+1
View File
@@ -3,6 +3,7 @@ import { shareStore } from './store';
const messages: Record<string, Record<string, string>> = {
zh: {},
en: {
'资源已存在,请检查重复的名称或值': 'Resource already exists; check for duplicate names or values',
'暂无权限': 'Access denied',
'参数错误': 'Invalid parameter',
'参数不正确': 'Invalid parameter',
+1
View File
@@ -18,6 +18,7 @@ export interface IDependencyFn<T> {
export interface ICronFn<T> {
(): Promise<T>;
cron?: TCron;
isCurrent?: () => boolean;
}
export interface ISchedule {
+121
View File
@@ -0,0 +1,121 @@
import { shellQuote } from './shellQuote';
const repositoryNames =
/^(debian(?:-security)?|ubuntu(?:-ports)?|alpine|armbian)$/;
export function normalizeMirror(mirror: string): URL {
if (/\s/.test(mirror)) throw new Error('Invalid Linux mirror URL');
const url = new URL(mirror);
if (
!['http:', 'https:'].includes(url.protocol) ||
url.username ||
url.password ||
url.search ||
url.hash
) {
throw new Error(
'Linux mirror must be an HTTP(S) URL without credentials, query or fragment',
);
}
return url;
}
export function replaceMirrorUrl(current: string, mirror: string): string {
const target = normalizeMirror(mirror);
const previous = new URL(current);
const oldSegments = previous.pathname.split('/').filter(Boolean);
const newSegments = target.pathname.split('/').filter(Boolean);
const repositoryIndex = oldSegments.findIndex((part) =>
repositoryNames.test(part),
);
const targetIndex = newSegments.findIndex((part) =>
repositoryNames.test(part),
);
const prefix =
targetIndex >= 0 ? newSegments.slice(0, targetIndex) : newSegments;
if (repositoryIndex >= 0) {
const repository = oldSegments[repositoryIndex];
let suffix = oldSegments.slice(repositoryIndex + 1);
// Repair paths already corrupted by repeated authority-only replacement.
while (suffix[0] === repository) suffix.shift();
return `${target.origin}/${[...prefix, repository, ...suffix].join('/')}`;
}
// Unknown repository paths: a full URL replaces the old root, a hostname
// preserves it. Never append the newly entered root to the existing root.
return `${target.origin}/${(newSegments.length
? newSegments
: oldSegments
).join('/')}`;
}
export function rewriteMirrorSources(content: string, mirror: string): string {
normalizeMirror(mirror);
return content
.split(/(\r?\n[ \t]*\r?\n)/)
.map((stanza) => {
const signedBy = stanza.match(/^Signed-By:\s*([^\r\n]*)/im)?.[1];
return stanza
.split('\n')
.map((line) => {
if (!/^\s*(?:URIs:|deb(?:-src)?\s|https?:\/\/)/.test(line))
return line;
const legacy = /^\s*deb(?:-src)?\s/.test(line);
const key = legacy
? line.match(/\bsigned-by=([^\]\s]+)/i)?.[1]
: signedBy;
// In .list files a # starts an inline comment; its URLs are metadata.
const comment = legacy ? line.indexOf('#') : -1;
const source = comment < 0 ? line : line.slice(0, comment);
const rewritten = source.replace(/https?:\/\/[^\s#]+/g, (url) =>
isSystemRepository(url, key) ? replaceMirrorUrl(url, mirror) : url,
);
return rewritten + (comment < 0 ? '' : line.slice(comment));
})
.join('\n');
})
.join('');
}
function isSystemRepository(url: string, signedBy?: string): boolean {
const keys =
signedBy
?.trim()
.split(/[\s,]+/)
.filter(Boolean) || [];
const archiveKeyring =
keys.length > 0 &&
keys.every((key) =>
/\/(?:debian|ubuntu|armbian)-archive-keyring\.(?:gpg|asc)$/.test(key),
);
// A separate signing key identifies a third-party archive, even if its
// repository happens to be called /debian or /ubuntu.
if (keys.length > 0 && !archiveKeyring) return false;
const segments = new URL(url).pathname.split('/').filter(Boolean);
const repositoryIndex = segments.findIndex((part) =>
repositoryNames.test(part),
);
if (repositoryIndex < 0) return false;
// Without archive metadata, accept only conventional archive roots. A
// vendor path such as Docker's /linux/debian is not the Debian archive.
return (
archiveKeyring ||
repositoryIndex === 0 ||
segments
.slice(0, repositoryIndex)
.every((part) => /^mirrors?$/.test(part)) ||
segments[repositoryIndex] === 'alpine'
);
}
export function mirrorWriteCommand(
file: string,
content: string,
prefix: string,
update: string,
): string {
return `${prefix}mkdir -p ${shellQuote(
file.slice(0, file.lastIndexOf('/')),
)} && printf '%s' ${shellQuote(content)} | ${prefix}tee ${shellQuote(
file,
)} >/dev/null && ${update}`;
}
+11
View File
@@ -0,0 +1,11 @@
import { resolveFileAccess } from './fileAccess';
/** Check stored paths too: old database rows are not trusted input. */
export function resolveLogPath(root: string, value: string): string {
const target = resolveFileAccess(root, [value]);
if (!target)
throw Object.assign(new Error('Log path is outside the log directory'), {
status: 400,
});
return target;
}
+47
View File
@@ -0,0 +1,47 @@
const RETRY_TTL = 15 * 60 * 1000;
const MAX_SOURCES = 10000;
const MAX_RETRY_SECONDS = 5 * 60;
/** Source-specific, bounded backoff; persisted account history is not a lock. */
export class LoginThrottle {
private sources = new Map<
string,
{ failures: number; lastFailure: number }
>();
private get(source: string, now: number) {
const state = this.sources.get(source);
if (state && now - state.lastFailure >= RETRY_TTL) {
this.sources.delete(source);
return undefined;
}
return state;
}
retryAfter(source: string, now: number): number {
const state = this.get(source, now);
if (!state || state.failures < 3) return 0;
const seconds = Math.min(3 ** state.failures, MAX_RETRY_SECONDS);
return Math.max(
0,
Math.ceil((seconds * 1000 - (now - state.lastFailure)) / 1000),
);
}
fail(source: string, now: number): number {
const failures = Math.min((this.get(source, now)?.failures || 0) + 1, 6);
this.sources.delete(source);
if (this.sources.size >= MAX_SOURCES) {
this.sources.delete(this.sources.keys().next().value!);
}
this.sources.set(source, { failures, lastFailure: now });
return failures;
}
reset(source: string) {
this.sources.delete(source);
}
clear() {
this.sources.clear();
}
}
+16 -9
View File
@@ -50,11 +50,9 @@ class TaskLimit {
Buffer.from(tlsConfig.clientCert),
)
: credentials.createInsecure();
this._client = new ApiClient(
`localhost:${config.grpcPort}`,
creds,
{ 'grpc.enable_http_proxy': 0 },
);
this._client = new ApiClient(`localhost:${config.grpcPort}`, creds, {
'grpc.enable_http_proxy': 0,
});
}
return this._client;
}
@@ -113,12 +111,15 @@ class TaskLimit {
}
}
public removeQueuedCron(id: string) {
public removeQueuedCron(id: string, completed?: ICronFn<any>) {
if (this.queuedCrons.has(id)) {
const runs = this.queuedCrons.get(id);
if (runs && runs.length > 0) {
runs.pop();
this.queuedCrons.set(id, runs);
const remaining = completed
? runs.filter((run) => run !== completed)
: runs.slice(0, -1);
if (remaining.length) this.queuedCrons.set(id, remaining);
else this.queuedCrons.delete(id);
}
}
}
@@ -143,9 +144,15 @@ class TaskLimit {
cron: TCron,
fn: ICronFn<T>,
options?: Partial<QueueAddOptions>,
isCurrent: () => boolean = () => true,
): Promise<T | void> {
if (!isCurrent()) return;
fn.cron = cron;
let runs = this.queuedCrons.get(cron.id);
fn.isCurrent = isCurrent;
// Invalidated snapshots must not consume the new revision's repeat budget.
const runs = this.queuedCrons
.get(cron.id)
?.filter((run) => run.isCurrent?.() !== false);
const result = runs?.length ? [...runs, fn] : [fn];
const repeatTimes = this.repeatCronNotifyMap.get(cron.id) || 0;
if (result?.length > 5) {
+15 -4
View File
@@ -8,15 +8,23 @@ import { RunningInstanceModel, InstanceStatus } from '../data/runningInstance';
import dayjs from 'dayjs';
import { observeChildProcess, asError } from './childProcess';
export function runCron(cmd: string, cron: ICron): Promise<number | void> {
return taskLimit.runWithCronLimit(cron, async () => {
export function runCron(
cmd: string,
cron: ICron,
isCurrent: () => boolean = () => true,
): Promise<number | void> {
const execute = async () => {
try {
if (!isCurrent()) return;
// Check if the cron is already running and stop it (only if multiple instances are not allowed)
try {
const existingCron = await CrontabModel.findOne({
where: { id: Number(cron.id) },
});
// A mutation may commit while this queued run is reading the database.
if (!isCurrent() || !existingCron || existingCron.isDisabled) return;
// Default to single instance mode (0) for backward compatibility
const allowSingleInstances =
existingCron?.allow_multiple_instances === 0;
@@ -56,6 +64,8 @@ export function runCron(cmd: string, cron: ICron): Promise<number | void> {
throw error;
}
// In particular, invalidation can happen while stopping a previous run.
if (!isCurrent()) return;
Logger.info(
`[schedule][开始执行任务] 参数 ${JSON.stringify({
...cron,
@@ -94,7 +104,8 @@ export function runCron(cmd: string, cron: ICron): Promise<number | void> {
asError(error).message,
);
} finally {
taskLimit.removeQueuedCron(cron.id);
taskLimit.removeQueuedCron(cron.id, execute);
}
});
};
return taskLimit.runWithCronLimit(cron, execute, undefined, isCurrent);
}
+467
View File
@@ -0,0 +1,467 @@
import * as fs from 'fs/promises';
import { createReadStream } from 'fs';
import path from 'path';
import { createHash, randomUUID } from 'crypto';
import { promisify } from 'util';
import { brotliCompress, brotliDecompress, constants } from 'zlib';
import { lock } from 'proper-lockfile';
import { resolveFileAccess } from './fileAccess';
import { getUniqueLockPath } from './utils';
const compress = promisify(brotliCompress);
const decompress = promisify(brotliDecompress);
const hash = (content: string) =>
createHash('sha256').update(content).digest('hex');
export const SCRIPT_HISTORY_LIMITS = {
versions: 20,
fileBytes: 1024 * 1024,
unpackedBytes: 8 * 1024 * 1024,
archiveBytes: 2 * 1024 * 1024,
totalBytes: 32 * 1024 * 1024,
files: 1000,
};
type Source = 'baseline' | 'save' | 'restore' | 'external';
interface Version {
id: string;
createdAt: string;
source: Source;
hash: string;
content: string;
}
interface Archive {
format: 1;
path: string;
versions: Version[];
head?: { hash: string; createdAt: string; source: Source };
}
export class ScriptHistoryError extends Error {
constructor(public status: number, message: string) {
super(message);
}
}
export class HistoryUnavailableError extends ScriptHistoryError {
constructor(public currentHash: string) {
super(413, '历史版本仅支持不超过 1 MiB 的 UTF-8 文本脚本');
}
}
interface SaveOptions {
skipHistory?: boolean;
expectedHash?: string;
}
/** Lazy, bounded storage: no watcher, database, daemon, or in-memory cache.
* Full snapshots share a Brotli window, so similar revisions compress together
* without a patch chain. All mutations use the existing script lock and a
* cross-process archive lock. History is durable before overwriting the script. */
export class ScriptHistory {
constructor(
private scriptRoot: string,
private historyRoot: string,
private blacklist: string[] = [],
private limits = SCRIPT_HISTORY_LIMITS,
) {}
private async target(directory: string, filename: string) {
const candidate = resolveFileAccess(
this.scriptRoot,
[directory, filename],
this.blacklist,
);
if (!candidate) throw new ScriptHistoryError(403, '暂无权限');
try {
const file = await fs.realpath(candidate);
const root = await fs.realpath(this.scriptRoot);
const relative = path.relative(root, file);
const key = hash(relative);
return {
file,
relative,
archive: path.join(this.historyRoot, key + '.br'),
};
} catch (error: any) {
if (error.code === 'ENOENT')
throw new ScriptHistoryError(404, '脚本不存在');
throw error;
}
}
private async readText(file: string) {
const handle = await fs.open(file, 'r');
try {
const stat = await handle.stat();
if (!stat.isFile()) throw new ScriptHistoryError(400, '请选择脚本文件');
if (stat.size > this.limits.fileBytes)
throw new ScriptHistoryError(
413,
'历史版本仅支持不超过 1 MiB 的 UTF-8 文本脚本',
);
// Bounded read even if an external process grows the file after stat.
const bytes = Buffer.alloc(this.limits.fileBytes + 1);
let size = 0;
while (size < bytes.length) {
const { bytesRead } = await handle.read(
bytes,
size,
bytes.length - size,
null,
);
if (!bytesRead) break;
size += bytesRead;
}
const buffer = bytes.subarray(0, size);
const content = buffer.toString('utf8');
if (
size > this.limits.fileBytes ||
content.includes('\0') ||
!Buffer.from(content).equals(buffer)
) {
throw new ScriptHistoryError(
413,
'历史版本仅支持不超过 1 MiB 的 UTF-8 文本脚本',
);
}
return content;
} finally {
await handle.close();
}
}
private async readArchive(file: string, relative: string): Promise<Archive> {
try {
const stat = await fs.lstat(file);
if (!stat.isFile() || stat.size > this.limits.archiveBytes)
throw new Error('Invalid archive');
const bytes = await decompress(await fs.readFile(file), {
maxOutputLength: this.limits.unpackedBytes,
});
const archive: Archive = JSON.parse(bytes.toString('utf8'));
if (
archive.format !== 1 ||
archive.path !== relative ||
!Array.isArray(archive.versions) ||
archive.versions.length > this.limits.versions ||
archive.versions.some(
(v) =>
typeof v.content !== 'string' ||
typeof v.id !== 'string' ||
typeof v.createdAt !== 'string' ||
!['baseline', 'save', 'restore', 'external'].includes(v.source) ||
Buffer.byteLength(v.content) > this.limits.fileBytes ||
hash(v.content) !== v.hash,
)
) {
throw new Error('Invalid archive');
}
return archive;
} catch (error: any) {
if (error.code === 'ENOENT')
return { format: 1, path: relative, versions: [] };
throw new ScriptHistoryError(500, '历史版本读取失败,请检查历史文件');
}
}
async list(directory: string, filename: string) {
const target = await this.target(directory, filename);
const [current, archive] = await Promise.all([
this.readText(target.file),
this.readArchive(target.archive, target.relative),
]);
const currentHash = hash(current);
const versions = archive.versions
.slice()
.reverse()
.map(({ content, ...v }) => ({
...v,
size: Buffer.byteLength(content),
identical: v.hash === currentHash,
}));
return { versions, currentHash, limit: this.limits.versions };
}
async detail(directory: string, filename: string, id: string) {
const target = await this.target(directory, filename);
const [current, archive] = await Promise.all([
this.readText(target.file),
this.readArchive(target.archive, target.relative),
]);
const version = archive.versions.find((v) => v.id === id);
if (!version) throw new ScriptHistoryError(404, '历史版本不存在或已清理');
return { version, current, currentHash: hash(current) };
}
async save(
directory: string,
filename: string,
content: string,
options: SaveOptions = {},
) {
return this.mutate(directory, filename, { content, ...options });
}
async restore(
directory: string,
filename: string,
id: string,
expectedHash: string,
) {
return this.mutate(directory, filename, { id, expectedHash });
}
private async mutate(
directory: string,
filename: string,
change:
| ({ content: string } & SaveOptions)
| { id: string; expectedHash: string },
) {
const target = await this.target(directory, filename);
const releaseFile = await lock(target.file, {
lockfilePath: getUniqueLockPath(target.file),
retries: { retries: 20, minTimeout: 50, maxTimeout: 250 },
});
try {
let current: string;
try {
current = await this.readText(target.file);
if (
'content' in change &&
(Buffer.byteLength(change.content) > this.limits.fileBytes ||
change.content.includes('\0'))
) {
throw new ScriptHistoryError(
413,
'历史版本仅支持不超过 1 MiB 的 UTF-8 文本脚本',
);
}
} catch (error) {
if (
!('content' in change) ||
!(error instanceof ScriptHistoryError) ||
error.status !== 413
)
throw error;
// Hash oversized or non-UTF-8 files without reading them all into memory.
const digest = createHash('sha256');
for await (const chunk of createReadStream(target.file))
digest.update(chunk);
const currentHash = digest.digest('hex');
if (!change.skipHistory) throw new HistoryUnavailableError(currentHash);
if (change.expectedHash !== currentHash)
throw new ScriptHistoryError(409, '文件已变化,请刷新历史版本后重试');
await this.preparedWrite(target.file, change.content, (commit) =>
commit(),
);
return {
content: change.content,
changed: true,
historyRecorded: false,
};
}
if (
('id' in change || change.skipHistory) &&
hash(current) !== change.expectedHash
) {
throw new ScriptHistoryError(409, '文件已变化,请刷新历史版本后重试');
}
if ('content' in change && change.content === current)
return { content: current, changed: false };
await fs.mkdir(this.historyRoot, { recursive: true, mode: 0o700 });
const releaseHistory = await lock(this.historyRoot, {
retries: { retries: 20, minTimeout: 50, maxTimeout: 250 },
});
try {
const archive = await this.readArchive(target.archive, target.relative);
const restored =
'id' in change
? archive.versions.find((v) => v.id === change.id)
: undefined;
if ('id' in change && !restored)
throw new ScriptHistoryError(404, '历史版本不存在或已清理');
const content =
'content' in change ? change.content : restored!.content;
if (content === current) return { content, changed: false };
if (
Buffer.byteLength(content) > this.limits.fileBytes ||
content.includes('\0')
) {
throw new ScriptHistoryError(
413,
'历史版本仅支持不超过 1 MiB 的 UTF-8 文本脚本',
);
}
const currentHash = hash(current);
if (
archive.versions[archive.versions.length - 1]?.hash !== currentHash
) {
const head =
archive.head?.hash === currentHash ? archive.head : undefined;
archive.versions.push({
id: randomUUID(),
createdAt: head?.createdAt || new Date().toISOString(),
source:
head?.source ||
(archive.versions.length ? 'external' : 'baseline'),
hash: currentHash,
content: current,
});
}
// Only prior contents are stored. Head metadata labels the next snapshot,
// but is used only if its hash matches the file. A failed file write can
// never introduce a phantom content version or lose the old snapshot.
archive.head = {
hash: hash(content),
createdAt: new Date().toISOString(),
source: 'id' in change ? 'restore' : 'save',
};
const packed = await this.pack(archive);
const cleanupPending = await this.preparedWrite(
target.file,
content,
(commit) => this.store(target.archive, packed, commit),
);
return {
content,
changed: true,
historyRecorded: true,
cleanupPending,
};
} finally {
await releaseHistory();
}
} finally {
await releaseFile();
}
}
private async pack(archive: Archive) {
archive.versions = archive.versions.slice(-this.limits.versions);
while (true) {
const serialized = Buffer.from(JSON.stringify(archive));
if (serialized.length <= this.limits.unpackedBytes) {
const compressed = await compress(serialized, {
params: {
[constants.BROTLI_PARAM_QUALITY]: 4,
[constants.BROTLI_PARAM_LGWIN]: 22,
},
});
if (compressed.length <= this.limits.archiveBytes) return compressed;
}
if (archive.versions.length <= 1)
throw new ScriptHistoryError(413, '历史版本容量不足,无法安全保存');
archive.versions.shift();
}
}
/** Stage the script before publishing history. A failed write/fsync never
* truncates the original. Preserve the existing file's ownership and mode. */
private async preparedWrite<T>(
file: string,
content: string,
run: (commit: () => Promise<void>) => Promise<T>,
): Promise<T> {
const stat = await fs.stat(file);
const temporary = path.join(
path.dirname(file),
`.ql-save-${randomUUID()}.tmp`,
);
try {
await fs.writeFile(temporary, content, {
encoding: 'utf8',
mode: 0o600,
flag: 'wx',
});
if (
typeof process.getuid === 'function' &&
typeof process.getgid === 'function' &&
(stat.uid !== process.getuid() || stat.gid !== process.getgid())
) {
await fs.chown(temporary, stat.uid, stat.gid);
}
await fs.chmod(temporary, stat.mode & 0o7777);
const handle = await fs.open(temporary, 'r');
try {
await handle.sync();
} finally {
await handle.close();
}
return await run(() => fs.rename(temporary, file));
} finally {
await fs.unlink(temporary).catch(() => undefined);
}
}
private async store(
file: string,
content: Buffer,
commit: () => Promise<void>,
) {
const entries = [];
for (const name of await fs.readdir(this.historyRoot)) {
if (!/^[a-f0-9]{64}\.br$/.test(name)) continue;
const fullPath = path.join(this.historyRoot, name);
if (fullPath === file) continue;
const stat = await fs.lstat(fullPath);
entries.push({ file: fullPath, size: stat.size, time: stat.mtimeMs });
}
entries.sort((a, b) => a.time - b.time);
let total = entries.reduce((sum, v) => sum + v.size, content.length);
const victims: string[] = [];
while (
entries.length &&
(total > this.limits.totalBytes || entries.length >= this.limits.files)
) {
const oldest = entries.shift()!;
victims.push(oldest.file);
total -= oldest.size;
}
if (total > this.limits.totalBytes)
throw new ScriptHistoryError(413, '历史版本容量不足,无法安全保存');
const previous = await fs.readFile(file).catch((error) => {
if (error.code === 'ENOENT') return undefined;
throw error;
});
const temporary = path.join(this.historyRoot, '.pending.tmp');
const publish = async (bytes: Buffer) => {
await fs.unlink(temporary).catch((error) => {
if (error.code !== 'ENOENT') throw error;
});
await fs.writeFile(temporary, bytes, { mode: 0o600, flag: 'wx' });
const handle = await fs.open(temporary, 'r');
try {
await handle.sync();
} finally {
await handle.close();
}
await fs.rename(temporary, file);
};
try {
await publish(content);
try {
await commit();
} catch (error) {
// The original script is still intact. Restore its exact history before
// returning an error; no unrelated archive has been removed yet.
if (previous) await publish(previous);
else await fs.unlink(file);
throw error;
}
// Quota eviction is post-commit maintenance. Report cleanup failures as a
// warning, not a failed save (which would encourage a misleading retry).
let cleanupPending = false;
for (const victim of victims) {
try {
await fs.unlink(victim);
} catch (error: any) {
if (error.code !== 'ENOENT') cleanupPending = true;
}
}
return cleanupPending;
} finally {
await fs.unlink(temporary).catch(() => undefined);
}
}
}
+5
View File
@@ -0,0 +1,5 @@
/** Quote one literal POSIX shell argument, including quotes and substitutions. */
export function shellQuote(value: string): string {
if (value.includes('\0')) throw new Error('Invalid shell argument');
return "'" + value.replace(/'/g, "'\\''") + "'";
}
+30
View File
@@ -0,0 +1,30 @@
import lockfile from 'proper-lockfile';
import path from 'path';
import config from '../config';
// Protect subscription storage ownership from the first database read through
// cleanup. Creation and URL/branch updates must use the same cross-process lock.
export async function withSubscriptionMutation<T>(
operation: () => Promise<T>,
): Promise<T> {
const target = `${path.resolve(config.scriptPath)}.subscriptions`;
let release: () => Promise<void>;
try {
release = await lockfile.lock(target, {
realpath: false,
stale: 30000,
update: 10000,
retries: { retries: 50, factor: 1, minTimeout: 100, maxTimeout: 100 },
});
} catch (cause) {
throw Object.assign(
new Error('Subscription configuration is busy', { cause }),
{ status: 503 },
);
}
try {
return await operation();
} finally {
await release();
}
}
+45
View File
@@ -0,0 +1,45 @@
import { resolveFileAccess } from './fileAccess';
import { createHash } from 'crypto';
export function assertSubscriptionAlias(alias: string): void {
if (
typeof alias !== 'string' ||
alias.length > 200 ||
!/^[^\s\\\p{C}]+$/u.test(alias) ||
alias.startsWith('~') ||
alias.split('/').some((part) => !part || part === '.' || part === '..')
) {
throw Object.assign(new Error('Invalid subscription alias'), {
status: 400,
});
}
}
/** Keep ordinary SSH names and use a reserved namespace for path-like aliases.
* The database alias and its script/log directories remain unchanged. */
export function getSubscriptionSshAlias(alias: string): string {
assertSubscriptionAlias(alias);
if (/^[\p{L}\p{N}_.+-]+$/u.test(alias) && !alias.endsWith('.config')) {
return alias;
}
return `~sub_${createHash('sha256').update(alias).digest('hex')}`;
}
export function resolveSubscriptionPath(
root: string,
alias: string,
suffix = '',
): string {
// Internal SSH filenames cannot collide with user-supplied aliases.
if (!/^~sub_[a-f0-9]{64}$/.test(alias)) {
assertSubscriptionAlias(
alias.startsWith('~global_') ? alias.slice(8) : alias,
);
}
const target = resolveFileAccess(root, [alias + suffix]);
if (!target)
throw Object.assign(new Error('Invalid subscription path'), {
status: 400,
});
return target;
}
+106
View File
@@ -0,0 +1,106 @@
import type { Subscription } from '../data/subscription';
import { formatUrl } from '../config/subscription';
import path from 'path';
import fs from 'fs';
import { resolveFileAccess } from './fileAccess';
// Storage namespaces come from URLs and branches, not user aliases. In
// particular, a legal Git directory can exceed the alias's 200-character limit.
export function isSubscriptionStorageName(name: string): boolean {
return (
typeof name === 'string' &&
!!name &&
!path.isAbsolute(name) &&
!/[\0\r\n\\]/.test(name) &&
!name.split('/').some((part) => !part || part === '.' || part === '..')
);
}
export function resolveSubscriptionStoragePath(
root: string,
name: string,
): string | undefined {
const target = isSubscriptionStorageName(name)
? resolveFileAccess(root, [name])
: '';
if (!target) {
if (isSubscriptionStorageName(name)) {
try {
fs.lstatSync(path.resolve(root, name));
} catch (error: any) {
// A long query can make the Shell candidate unrepresentable while
// the CLI's pathname-based file exists. Use the actual filesystem
// limit: macOS permits Unicode names exceeding 255 UTF-8 bytes.
if (error.code === 'ENAMETOOLONG') return undefined;
}
}
throw Object.assign(new Error('Invalid subscription path'), {
status: 400,
});
}
return target;
}
/** Match shell/update.sh get_uniq_path and update_raw, including SCP URLs. */
export function getSubscriptionStorageNames(
doc: Subscription,
engine: 'shell' | 'cli' = 'shell',
): {
script: string;
repo?: string;
raw?: string;
} {
// Old imported rows may only have an alias and no URL.
if (!doc.url) return { script: doc.alias, repo: doc.alias };
const url = formatUrl(doc).url!;
const trimmed = url.endsWith('/') ? url.slice(0, -1) : url;
const basename = trimmed.slice(trimmed.lastIndexOf('/') + 1);
const dot = basename.lastIndexOf('.');
const repo = dot < 0 ? basename : basename.slice(0, dot);
const parentUrl = engine === 'cli' ? trimmed : url;
const slash = parentUrl.lastIndexOf('/');
const parent =
engine === 'shell' && slash < 0 ? url : parentUrl.slice(0, slash);
const owner = parent.slice(parent.lastIndexOf('/') + 1);
const author = owner.slice(owner.lastIndexOf(':') + 1);
const stem = `${author.slice(author.lastIndexOf('.') + 1)}_${repo}`;
if (doc.type === 'file') {
const suffix =
engine === 'cli'
? path.extname(new URL(url).pathname)
: `.${url.slice(url.lastIndexOf('.') + 1)}`;
const raw = `${stem}${suffix}`;
return { script: `raw_${raw}`, raw };
}
const directory = doc.branch ? `${stem}_${doc.branch}` : stem;
return { script: directory, repo: directory };
}
// A subscription may have run through either engine before being deleted.
export function getSubscriptionStorageCandidates(doc: Subscription) {
const candidates: ReturnType<typeof getSubscriptionStorageNames>[] = [];
for (const engine of ['shell', 'cli'] as const) {
try {
candidates.push(getSubscriptionStorageNames(doc, engine));
} catch {
// Historical rows may have an invalid SSH alias or missing credentials.
// Retain their URL-derived storage without needing an executable SSH URL.
try {
candidates.push(
getSubscriptionStorageNames(
{
url: doc.url,
alias: doc.alias,
branch: doc.branch,
type: doc.type === 'private-repo' ? 'public-repo' : doc.type,
},
engine,
),
);
} catch {
// One engine's invalid URL must not hide the other engine's candidate.
}
}
}
return candidates;
}
+1 -1
View File
@@ -4,7 +4,7 @@ import path from 'path';
import { writeFile, open, chmod } from 'fs/promises';
import { fileExist } from '../config/util';
function getUniqueLockPath(filePath: string) {
export function getUniqueLockPath(filePath: string) {
const sanitizedPath = filePath
.replace(/[<>:"/\\|?*]/g, '_')
.replace(/^_/, '');
+5 -16
View File
@@ -2,30 +2,19 @@ import 'reflect-metadata';
import OpenService from './services/open';
import { Container } from 'typedi';
import LoggerInstance from './loaders/logger';
import config from './config';
import path from 'path';
import os from 'os';
import { writeFileWithLock } from './shared/utils';
const tokenFile = path.join(config.configPath, 'token.json');
async function getToken() {
try {
Container.set('logger', LoggerInstance);
const openService = Container.get(OpenService);
const appToken = await openService.generateSystemToken();
const appToken = await openService.generateSystemToken(
process.argv.includes('--renew'),
);
console.log(appToken.value);
await writeFile({
value: appToken.value,
expiration: appToken.expiration,
});
} catch (error) {
console.log(error);
console.error(error);
process.exitCode = 1;
}
}
async function writeFile(data: any) {
await writeFileWithLock(tokenFile, `${JSON.stringify(data)}${os.EOL}`);
}
getToken();
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "@whyour/qinglong-cli",
"version": "0.1.1",
"version": "0.1.2",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@whyour/qinglong-cli",
"version": "0.1.1",
"version": "0.1.2",
"bin": {
"ql": "dist/npm/ql.js"
},
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@whyour/qinglong-cli",
"version": "0.1.1",
"version": "0.1.2",
"description": "Authenticated remote management CLI for QingLong 2.x",
"type": "commonjs",
"bin": {
+7 -1
View File
@@ -13,6 +13,10 @@ execFileSync(
],
{ stdio: 'inherit' },
);
fs.copyFileSync(
path.join(root, 'src/internal/maintenance/upgradeArtifacts.cjs'),
path.join(root, 'dist/internal/maintenance/upgradeArtifacts.cjs'),
);
require('esbuild').buildSync({
entryPoints: [path.join(root, 'src/shared/cli/commander.ts')],
outfile: path.join(root, 'dist/shared/cli/commander.js'),
@@ -45,7 +49,9 @@ const remote = require('esbuild').buildSync({
metafile: true,
});
for (const input of Object.keys(remote.metafile.inputs)) {
if (/(?:^|[\\/])src[\\/](?:internal|compatibility|developer)[\\/]/.test(input))
if (
/(?:^|[\\/])src[\\/](?:internal|compatibility|developer)[\\/]/.test(input)
)
throw new Error(
'Local implementation leaked into remote npm bundle: ' + input,
);
@@ -1,8 +1,8 @@
# OpenAPI 全量参考 / Complete OpenAPI reference
CLI 对应当前 develop 的 143 条有效路由。通过 `ql api routes --json` 查看方法、路径、对应命令、位置参数、请求体与上传字段;CLI 测试与 back/api 路由逐项核对。GET configs/:file、scripts/:file、logs/:file 已返回 410,改用 detail 命令,不再暴露旧入口。旧版面板可能没有新路由,以服务端响应为准。
CLI 对应本版本后端的 146 条有效路由。通过 `ql api routes --json` 查看方法、路径、对应命令、位置参数、请求体与上传字段;CLI 测试与 back/api 路由逐项核对。GET configs/:file、scripts/:file、logs/:file 已返回 410,改用 detail 命令,不再暴露旧入口。旧版面板可能没有新路由,以服务端响应为准。
The catalogue covers 143 active routes in the current develop backend. `ql api routes --json` lists methods, paths, commands, positional parameters, bodies and upload fields. A route-coverage test detects drift. Three retired GET :file routes return 410; use detail-based commands. Older panel versions may not implement newer routes.
The catalogue covers 146 active routes in the current backend. `ql api routes --json` lists methods, paths, commands, positional parameters, bodies and upload fields. A route-coverage test detects drift. Three retired GET :file routes return 410; use detail-based commands. Older panel versions may not implement newer routes.
## 认证与权限 / Authentication and permissions
@@ -46,6 +46,9 @@ ql env create --data @envs.json --json
ql env update 8 --data '{"name":"EXAMPLE","value":"value"}' --json
ql script create --file ./demo.js --data '{"filename":"demo.js","path":""}' --json
ql script get --query '{"file":"demo.js","path":""}' --json
ql script history-list --query '{"filename":"demo.js","path":""}' --json
ql script history-detail --query '{"filename":"demo.js","path":"","id":"VERSION_UUID"}' --json
ql script history-restore --data '{"filename":"demo.js","path":"","id":"VERSION_UUID","expectedHash":"CURRENT_SHA256"}' --json
ql config save --data '{"name":"example.sh","content":"# example"}' --json
ql config get --query '{"path":"example.sh"}' --json
ql log download --data '{"filename":"example.log","path":"example"}' --output ./example.log --json
@@ -133,6 +136,9 @@ System/user operations below execute remotely through the panel API. They do not
| `ql config save ` | POST | `configs/save` | --data, --query |
| `ql script list ` | GET | `scripts` | --query |
| `ql script get ` | GET | `scripts/detail` | --query |
| `ql script history-list ` | GET | `scripts/history` | --query |
| `ql script history-detail ` | GET | `scripts/history/detail` | --query |
| `ql script history-restore ` | PUT | `scripts/history/restore` | --data, --query |
| `ql script create ` | POST | `scripts` | --data, --file (file), --query |
| `ql script update ` | PUT | `scripts` | --data, --file (file), --query |
| `ql script delete ` | DELETE | `scripts` | --data, --query |
+18 -2
View File
@@ -70,10 +70,26 @@ if [ -n "$__ql_timeout" ]; then
fi
if [ "$__ql_command" = true ]; then
"\${__ql_args[@]}" "\${__ql_script_args[@]}"
_task_exit_code=$?
else
. "\${__ql_args[0]}" "\${__ql_script_args[@]}"
# Match task.sh: keep script exit/exec/options inside a child shell while
# retaining before-hook variables and functions for the script.
# Keep the supervisor alive until the child's signal/EXIT traps finish, but
# do not turn user cancellation into an ordinary completion with after hooks.
__ql_saved_signals=$(trap -p INT TERM HUP QUIT ALRM TSTP)
__ql_cancel_code=''
trap '__ql_cancel_code=$((128 + $(kill -l INT)))' INT
trap '__ql_cancel_code=$((128 + $(kill -l TERM)))' TERM
trap '__ql_cancel_code=$((128 + $(kill -l HUP)))' HUP
trap '__ql_cancel_code=$((128 + $(kill -l QUIT)))' QUIT
trap '__ql_cancel_code=$((128 + $(kill -l ALRM)))' ALRM
trap '__ql_cancel_code=$((128 + $(kill -l TSTP)))' TSTP
( . "\${__ql_args[0]}" "\${__ql_script_args[@]}" )
_task_exit_code=$?
trap - INT TERM HUP QUIT ALRM TSTP
eval "$__ql_saved_signals"
if [ -n "$__ql_cancel_code" ] && ! [ -f "$__ql_timeout" ]; then exit "$__ql_cancel_code"; fi
fi
_task_exit_code=$?
if [ -n "$__ql_timeout" ] && [ -f "$__ql_timeout" ]; then _task_exit_code=124; fi
__ql_run_after
exit "$_task_exit_code"
+51 -129
View File
@@ -8,12 +8,33 @@ import { checkedProcess } from '../runtime/process';
import { operationSignal, withoutCancellation } from '../runtime/cancellation';
import { installPanelDependencies, startPanel, stopPanel } from './operator';
// The Bash image updater and local CLI must enforce the same build provenance.
const artifacts = require('./upgradeArtifacts.cjs') as {
releaseBranch(env: NodeJS.ProcessEnv): string;
verifyUpgrade(source: string, staticRoot: string): Promise<unknown>;
waitForHealth(
env: NodeJS.ProcessEnv,
timeout?: number,
signal?: AbortSignal,
): Promise<void>;
withUpgradeLock<T>(tmp: string, operation: () => Promise<T>): Promise<T>;
stageUpgrade(options: {
root: string;
tmp: string;
env: NodeJS.ProcessEnv;
mirror: 'github' | 'gitee';
signal?: AbortSignal;
run(program: string, args: string[], capture?: boolean): Promise<unknown>;
install(source: string): Promise<void>;
}): Promise<{ source: string; static: string }>;
selectedUpgrade(
tmp: string,
branch: string,
): Promise<{ source: string; static: string }>;
};
export function releaseBranch(context: LocalContext): string {
return ['develop', 'debian', 'debian-dev'].includes(
context.env.QL_BRANCH ?? '',
)
? context.env.QL_BRANCH!
: 'master';
return artifacts.releaseBranch(context.env);
}
async function validateTree(
@@ -48,134 +69,27 @@ export async function stageUpgrade(
context: LocalContext,
mirror: 'github' | 'gitee',
): Promise<{ source: string; static: string }> {
await fs.mkdir(context.paths.dir_tmp!, { recursive: true });
const workspace = await fs.mkdtemp(
path.join(context.paths.dir_tmp!, 'upgrade-'),
return artifacts.withUpgradeLock(context.paths.dir_tmp!, () =>
artifacts.stageUpgrade({
root: context.root,
tmp: context.paths.dir_tmp!,
env: context.env,
mirror,
signal: operationSignal(),
run: (program, args, capture) =>
checkedProcess(program, args, { env: context.env, capture }),
install: (source) => installPanelDependencies(context, source),
}),
);
const branch = releaseBranch(context);
try {
for (const repository of ['qinglong', 'qinglong-static']) {
const url =
mirror === 'github'
? `https://github.com/whyour/${repository}/archive/refs/heads/${branch}.zip`
: `https://gitee.com/whyour/${repository}/repository/archive/${branch}.zip`;
const archive = path.join(workspace, `${repository}.zip`);
await checkedProcess(
'curl',
[
'--fail',
'--location',
'--proto',
'=https',
'--proto-redir',
'=https',
'--output',
archive,
url,
],
{ env: context.env },
);
const listing = await checkedProcess('unzip', ['-Z1', archive], {
env: context.env,
capture: true,
});
const expected = `${repository}-${branch}`;
for (const name of listing.stdout.split('\n').filter(Boolean)) {
if (
name.includes('\\') ||
name.includes('\r') ||
name.startsWith('/') ||
name.split('/').includes('..') ||
name.split('/')[0] !== expected
)
throw new Error(translate(context.env, '升级归档包含无效路径。'));
}
// Reject Unix symlinks before extraction, including links that could redirect
// a later archive entry outside the staging directory.
const modes = await checkedProcess('unzip', ['-Z', '-l', archive], {
env: context.env,
capture: true,
});
if (modes.stdout.split('\n').some((line) => /^l[rwx-]{9}\s/.test(line)))
throw new Error(translate(context.env, '升级归档包含符号链接。'));
await checkedProcess('unzip', ['-oq', archive, '-d', workspace], {
env: context.env,
});
await validateTree(path.join(workspace, expected), context.env);
}
const source = path.join(workspace, `qinglong-${branch}`);
const staticRoot = path.join(workspace, `qinglong-static-${branch}`);
await fs.access(path.join(staticRoot, 'build/app.js'));
const manifest = await fs.readFile(path.join(source, 'package.json'));
const existing = await fs.readFile(path.join(context.root, 'package.json'));
if (!manifest.equals(existing))
await installPanelDependencies(context, source);
// Publish a marker only after both archives and dependency installation succeed.
await fs.writeFile(
path.join(workspace, 'ready.json'),
JSON.stringify({ source, static: staticRoot }),
{ flag: 'wx', mode: 0o600 },
);
const pointer = path.join(workspace, 'pointer.json');
await fs.writeFile(
pointer,
JSON.stringify({ directory: path.basename(workspace) }),
{
flag: 'wx',
mode: 0o600,
},
);
await fs.rename(
pointer,
path.join(context.paths.dir_tmp!, `upgrade-ready-${branch}.json`),
);
return { source, static: staticRoot };
} catch (error) {
await fs.rm(workspace, { recursive: true, force: true });
throw error;
}
}
async function selectedUpgrade(
context: LocalContext,
): Promise<{ source: string; static: string }> {
const branch = releaseBranch(context);
const fallback = {
source: path.join(context.paths.dir_tmp!, `qinglong-${branch}`),
static: path.join(context.paths.dir_tmp!, `qinglong-static-${branch}`),
};
const pointer = await fs
.readFile(
path.join(context.paths.dir_tmp!, `upgrade-ready-${branch}.json`),
'utf8',
)
.catch((error: NodeJS.ErrnoException) => {
if (error.code === 'ENOENT') return undefined;
throw error;
});
if (pointer === undefined) return fallback;
const directory: unknown = JSON.parse(pointer)?.directory;
if (
typeof directory !== 'string' ||
!/^upgrade-[A-Za-z0-9_-]+$/.test(directory)
)
throw new Error(translate(context.env, '暂存升级指针无效。'));
const workspace = path.join(context.paths.dir_tmp!, directory);
const selected = {
source: path.join(workspace, `qinglong-${branch}`),
static: path.join(workspace, `qinglong-static-${branch}`),
};
for (const location of [workspace, selected.source, selected.static]) {
const stat = await fs.lstat(location);
if (!stat.isDirectory() || stat.isSymbolicLink())
throw new Error(translate(context.env, '暂存升级目录无效。'));
}
const ready = JSON.parse(
await fs.readFile(path.join(workspace, 'ready.json'), 'utf8'),
return artifacts.selectedUpgrade(
context.paths.dir_tmp!,
releaseBranch(context),
);
if (ready?.source !== selected.source || ready?.static !== selected.static)
throw new Error(translate(context.env, '暂存升级就绪记录与目录不匹配。'));
return selected;
}
export interface ReloadLifecycle {
@@ -345,8 +259,7 @@ export async function reloadPanel(
const staticRoot = selected.static;
await validateTree(source, context.env);
await validateTree(staticRoot, context.env);
await fs.access(path.join(source, 'package.json'));
await fs.access(path.join(staticRoot, 'build/app.js'));
await artifacts.verifyUpgrade(source, staticRoot);
const entries = await fs.readdir(source);
const reserved = new Set(['data', '.tmp', 'static', '.git', '.env']);
const relativeData = path.relative(context.root, context.data);
@@ -371,7 +284,16 @@ export async function reloadPanel(
if (selection)
replacements.push({ source: selection.source, target: selection.target });
try {
return await replaceAndReload(context, replacements);
return await artifacts.withUpgradeLock(context.paths.dir_tmp!, () =>
replaceAndReload(context, replacements, {
stop: stopPanel,
start: async (ctx) => {
const service = await startPanel(ctx);
await artifacts.waitForHealth(ctx.env, 60000, operationSignal());
return service;
},
}),
);
} finally {
if (selection)
await fs.rm(selection.directory, { recursive: true, force: true });
@@ -0,0 +1,657 @@
// Shared by the image's Bash updater and the optional local CLI. Use only
// installed code to validate downloads; never execute a downloaded verifier.
const fs = require('node:fs/promises');
const path = require('node:path');
const { createHash, randomUUID } = require('node:crypto');
const { spawn } = require('node:child_process');
const http = require('node:http');
const { setTimeout: delay } = require('node:timers/promises');
const hash = (bytes) => createHash('sha256').update(bytes).digest('hex');
const releaseBranch = (env) =>
['develop', 'debian-dev'].includes(env.QL_BRANCH) ? 'develop' : 'master';
async function manifestAt(staticRoot) {
const manifest = JSON.parse(
await fs.readFile(path.join(staticRoot, 'build-info.json'), 'utf8'),
);
if (
manifest.version !== 1 ||
manifest.dirty !== false ||
!/^[a-f0-9]{40}$/.test(manifest.sourceCommit) ||
!/^[a-f0-9]{64}$/.test(manifest.lockfileSha256) ||
!manifest.files ||
typeof manifest.files !== 'object' ||
Array.isArray(manifest.files)
)
throw new Error('Invalid or dirty build manifest / 构建清单无效。');
for (const required of ['build/app.js', 'dist/index.html'])
if (!Object.hasOwn(manifest.files, required))
throw new Error(`Missing build output / 缺少构建产物: ${required}`);
return manifest;
}
async function verifyUpgrade(source, staticRoot) {
const manifest = await manifestAt(staticRoot);
if (
hash(await fs.readFile(path.join(source, 'pnpm-lock.yaml'))) !==
manifest.lockfileSha256
)
throw new Error(
'Source and build lockfiles differ / 源码与构建依赖不一致。',
);
await fs.access(path.join(source, 'package.json'));
await fs.access(path.join(source, 'sample/config.sample.sh'));
const files = {};
async function visit(directory, prefix = '') {
for (const entry of await fs.readdir(directory, { withFileTypes: true })) {
const name = prefix + entry.name;
if (name === 'build-info.json') continue;
const filename = path.join(directory, entry.name);
if (entry.isDirectory()) await visit(filename, `${name}/`);
else if (entry.isFile()) files[name] = hash(await fs.readFile(filename));
else
throw new Error(`Unsupported build entry / 构建包含非法文件: ${name}`);
}
}
await visit(staticRoot);
const actual = Object.keys(files).sort(),
expected = Object.keys(manifest.files).sort();
if (
JSON.stringify(actual) !== JSON.stringify(expected) ||
actual.some((name) => files[name] !== manifest.files[name])
)
throw new Error(
'Build checksum mismatch / 构建文件缺失、损坏或混入旧文件。',
);
return manifest;
}
async function withUpgradeLock(tmp, operation) {
await fs.mkdir(tmp, { recursive: true });
const lock = path.join(tmp, 'upgrade.lock');
const token = randomUUID();
const owner = `owner-${process.pid}-${token}`;
const candidate = path.join(tmp, `upgrade-lock-${token}`);
let acquired = false;
await fs.mkdir(candidate, { mode: 0o700 });
try {
// Publish a nonempty directory atomically: a live lock can never be
// replaced, and a crash before publication cannot leave an ownerless lock.
await fs.writeFile(path.join(candidate, owner), '', {
flag: 'wx',
mode: 0o600,
});
for (;;) {
try {
await fs.rename(candidate, lock);
acquired = true;
break;
} catch (error) {
if (!['ENOTEMPTY', 'EEXIST'].includes(error.code)) throw error;
}
let entries;
try {
entries = await fs.readdir(lock);
} catch (error) {
if (error.code === 'ENOENT') continue;
throw error;
}
if (!entries.length) continue;
const heldOwner = entries[0];
const match = /^owner-([1-9]\d*)-[a-f0-9-]{36}$/.exec(heldOwner);
let pid = match ? Number(match[1]) : 0;
if (entries.length === 1 && heldOwner === 'pid') {
try {
pid = Number(await fs.readFile(path.join(lock, heldOwner), 'utf8'));
} catch (error) {
if (error.code === 'ENOENT') continue;
throw error;
}
}
let stale = false;
if (
entries.length === 1 &&
Number.isSafeInteger(pid) &&
pid > 0 &&
pid <= 0x7fffffff
) {
try {
process.kill(pid, 0);
} catch (error) {
if (error.code === 'ESRCH') stale = true;
}
}
if (!stale)
throw new Error(
'Another upgrade is running / 已有更新或重载正在执行。',
);
// Removing the observed unique marker is a compare-and-delete. A
// contender cannot unlink the marker belonging to a replacement lock.
// Legacy pid files are supported; new owners never reuse that filename.
try {
await fs.unlink(path.join(lock, heldOwner));
} catch (error) {
if (error.code !== 'ENOENT') throw error;
}
}
return await operation();
} finally {
if (acquired) {
await fs.unlink(path.join(lock, owner)).catch((error) => {
if (error.code !== 'ENOENT') throw error;
});
await fs.rmdir(lock).catch((error) => {
if (!['ENOENT', 'ENOTEMPTY', 'EEXIST'].includes(error.code))
throw error;
});
}
await fs.rm(candidate, { recursive: true, force: true });
}
}
async function stageUpgrade({ root, tmp, env, mirror, run, install, signal }) {
const workspace = await fs.mkdtemp(path.join(tmp, 'upgrade-'));
const branch = releaseBranch(env);
try {
async function extract(repository, ref) {
const archive = path.join(workspace, `${repository}.zip`);
const url =
mirror === 'github'
? `https://github.com/whyour/${repository}/archive/${ref}.zip`
: `https://gitee.com/whyour/${repository}/repository/archive/${ref}.zip`;
await run('curl', [
'--fail',
'--location',
'--proto',
'=https',
'--proto-redir',
'=https',
'--connect-timeout',
'30',
'--max-time',
'600',
'--output',
archive,
url,
]);
const listing = await run('unzip', ['-Z1', archive], true);
const expected = `${repository}-${ref}`;
const names = listing.stdout.split('\n').filter(Boolean);
if (
!names.length ||
names.some(
(name) =>
name.includes('\\') ||
name.includes('\r') ||
name.startsWith('/') ||
name.split('/').includes('..') ||
name.split('/')[0] !== expected,
)
)
throw new Error('Invalid archive paths / 升级归档路径无效。');
const modes = await run('unzip', ['-Z', '-l', archive], true);
if (modes.stdout.split('\n').some((line) => /^l[rwx-]{9}\s/.test(line)))
throw new Error('Archive contains symlinks / 升级归档包含符号链接。');
await run('unzip', ['-oq', archive, '-d', workspace]);
return path.join(workspace, expected);
}
// Static branches are mutable snapshots. Their manifest pins the exact
// source revision, avoiding a race between two independent branch reads.
const staticRoot = await extract('qinglong-static', branch);
const manifest = await manifestAt(staticRoot);
const downloadedSource = await extract('qinglong', manifest.sourceCommit);
const source = path.join(workspace, `qinglong-${branch}`);
await fs.rename(downloadedSource, source);
await verifyUpgrade(source, staticRoot);
let dependenciesChanged = false;
for (const name of ['package.json', 'pnpm-lock.yaml']) {
const current = await fs
.readFile(path.join(root, name))
.catch((error) => {
if (error.code !== 'ENOENT') throw error;
return Buffer.alloc(0);
});
if (!(await fs.readFile(path.join(source, name))).equals(current))
dependenciesChanged = true;
}
if (
!(await fs.stat(path.join(root, 'node_modules')).catch(() => undefined))
)
dependenciesChanged = true;
if (dependenciesChanged) {
await install(source);
await fs.access(path.join(source, 'node_modules'));
await verifyUpgrade(source, staticRoot);
}
signal?.throwIfAborted();
const selected = { source, static: staticRoot };
await fs.writeFile(
path.join(workspace, 'ready.json'),
JSON.stringify({
...selected,
sourceCommit: manifest.sourceCommit,
dependenciesChanged,
}),
{ flag: 'wx', mode: 0o600 },
);
const pointer = path.join(workspace, 'pointer.json');
await fs.writeFile(
pointer,
JSON.stringify({ directory: path.basename(workspace) }),
{ flag: 'wx', mode: 0o600 },
);
const readyPointer = path.join(tmp, `upgrade-ready-${branch}.json`);
const previous = await fs
.readFile(readyPointer, 'utf8')
.catch(() => undefined);
signal?.throwIfAborted();
await fs.rename(pointer, readyPointer);
// Retire only our previous successful stage, after publishing its replacement.
try {
const directory = previous && JSON.parse(previous).directory;
if (
typeof directory === 'string' &&
/^upgrade-[A-Za-z0-9_-]+$/.test(directory) &&
directory !== path.basename(workspace)
)
await fs.rm(path.join(tmp, directory), {
recursive: true,
force: true,
});
} catch {
/* Cleanup cannot invalidate the newly published stage. */
}
return selected;
} catch (error) {
await fs.rm(workspace, { recursive: true, force: true });
throw error;
}
}
async function selectedUpgrade(tmp, branch) {
const pointer = JSON.parse(
await fs.readFile(path.join(tmp, `upgrade-ready-${branch}.json`), 'utf8'),
);
if (
typeof pointer.directory !== 'string' ||
!/^upgrade-[A-Za-z0-9_-]+$/.test(pointer.directory)
)
throw new Error('Invalid staged upgrade pointer / 暂存升级指针无效。');
const workspace = path.join(tmp, pointer.directory);
const selected = {
source: path.join(workspace, `qinglong-${branch}`),
static: path.join(workspace, `qinglong-static-${branch}`),
};
for (const directory of [workspace, selected.source, selected.static]) {
const stat = await fs.lstat(directory);
if (!stat.isDirectory() || stat.isSymbolicLink())
throw new Error('Invalid staging directory / 暂存目录无效。');
}
const ready = JSON.parse(
await fs.readFile(path.join(workspace, 'ready.json'), 'utf8'),
);
if (ready.source !== selected.source || ready.static !== selected.static)
throw new Error('Invalid upgrade readiness / 暂存升级就绪记录不匹配。');
const manifest = await verifyUpgrade(selected.source, selected.static);
if (ready.sourceCommit !== manifest.sourceCommit)
throw new Error('Invalid upgrade readiness / 暂存升级就绪记录不匹配。');
if (ready.dependenciesChanged)
await fs.access(path.join(selected.source, 'node_modules'));
return selected;
}
async function replaceAndReload(replacements, lifecycle) {
const changed = [];
let attemptedStart = false;
lifecycle.signal?.throwIfAborted();
try {
await lifecycle.stop();
for (const { source, target } of replacements) {
lifecycle.signal?.throwIfAborted();
await fs.mkdir(path.dirname(target), { recursive: true });
const backup = `${target}.ql-backup-${randomUUID()}`;
const existed = !!(await fs.lstat(target).catch((error) => {
if (error.code !== 'ENOENT') throw error;
return undefined;
}));
if (existed) {
try {
await fs.rename(target, backup);
} catch (error) {
if (error.code !== 'EXDEV') throw error;
try {
await fs.cp(target, backup, {
recursive: true,
verbatimSymlinks: true,
});
} catch (copyError) {
await fs.rm(backup, { recursive: true, force: true });
throw copyError;
}
changed.push({ target, backup, existed });
await fs.rm(target, { recursive: true });
}
}
if (changed.at(-1)?.target !== target)
changed.push({ target, backup, existed });
await fs.cp(source, target, {
recursive: true,
verbatimSymlinks: true,
force: false,
errorOnExist: true,
});
}
lifecycle.signal?.throwIfAborted();
attemptedStart = true;
await lifecycle.start();
lifecycle.signal?.throwIfAborted();
} catch (error) {
// Do not modify files under a still-running failed startup. Keep backups
// for recovery if stopping the new process itself fails.
if (attemptedStart) await lifecycle.stop();
for (const item of changed.reverse()) {
await fs.rm(item.target, { recursive: true, force: true });
if (item.existed) {
try {
await fs.rename(item.backup, item.target);
} catch (restoreError) {
if (restoreError.code !== 'EXDEV') throw restoreError;
await fs.cp(item.backup, item.target, {
recursive: true,
verbatimSymlinks: true,
});
await fs.rm(item.backup, { recursive: true });
}
}
}
await lifecycle.start(true);
throw error;
}
for (const item of changed)
if (item.existed)
await fs
.rm(item.backup, { recursive: true, force: true })
.catch(() =>
console.error(`Retained backup / 保留备份: ${item.backup}`),
);
}
async function reloadSystem({
root,
data,
tmp,
staticRoot,
config,
env,
lifecycle,
}) {
const selected = await selectedUpgrade(tmp, releaseBranch(env));
const reserved = new Set(['data', '.tmp', 'static', '.git', '.env']);
const relative = path.relative(root, data);
if (relative && !relative.startsWith('..') && !path.isAbsolute(relative))
reserved.add(relative.split(path.sep)[0]);
const replacements = (await fs.readdir(selected.source))
.filter((name) => !reserved.has(name))
.map((name) => ({
source: path.join(selected.source, name),
target: path.join(root, name),
}));
replacements.push(
{ source: selected.static, target: staticRoot },
{
source: path.join(selected.source, 'sample/config.sample.sh'),
target: path.join(config, 'config.sample.sh'),
},
);
await replaceAndReload(replacements, lifecycle);
// Cleanup after a committed installation must not report a failed update.
await fs
.rm(path.join(tmp, `upgrade-ready-${releaseBranch(env)}.json`), {
force: true,
})
.catch(() => {});
await fs
.rm(path.dirname(selected.source), { recursive: true, force: true })
.catch(() => {});
}
async function waitForHealth(env, timeout = 60000, signal) {
const port = Number(env.QlPort || 5700);
let base = env.QlBaseUrl || '';
if (base && !base.startsWith('/')) base = `/${base}`;
if (base.endsWith('/')) base = base.slice(0, -1);
const deadline = Date.now() + timeout;
while (Date.now() < deadline) {
signal?.throwIfAborted();
const healthy = await new Promise((resolve) => {
let timer;
const finish = (healthy) => {
clearTimeout(timer);
resolve(healthy);
};
const req = http.get(
{ hostname: '127.0.0.1', port, path: `${base}/api/health`, signal },
(res) => {
let body = '';
res.on('data', (chunk) => {
body += chunk;
if (body.length > 65536) req.destroy();
});
res.on('end', () => {
try {
const result = JSON.parse(body);
finish(
res.statusCode === 200 &&
result.code === 200 &&
result.data?.status === 'ok',
);
} catch {
finish(false);
}
});
res.on('error', () => finish(false));
},
);
// Socket inactivity timeouts do not bound a response that keeps streaming.
timer = setTimeout(
() => req.destroy(new Error('Health probe timed out')),
Math.min(1000, Math.max(1, deadline - Date.now())),
);
req.on('error', () => finish(false));
});
signal?.throwIfAborted();
if (healthy) return;
const remaining = deadline - Date.now();
if (remaining > 0) {
try {
await delay(Math.min(500, remaining), undefined, { signal });
} catch (error) {
signal?.throwIfAborted();
throw error;
}
}
}
throw new Error('Backend failed health check / 后端启动检查失败。');
}
function runProcess(env, program, args, capture = false, signal) {
if (signal?.aborted) return Promise.reject(signal.reason);
return new Promise((resolve, reject) => {
const child = spawn(program, args, {
env,
detached: process.platform !== 'win32',
// Pipes keep close pending until children have released their output.
stdio: ['ignore', 'pipe', 'pipe'],
});
let stdout = '';
let aborted = false;
let failure;
let escalation;
const kill = (name) => {
if (!child.pid) return;
try {
process.kill(
process.platform === 'win32' ? child.pid : -child.pid,
name,
);
} catch (error) {
if (error.code === 'EPERM') {
child.kill(name);
return;
}
if (error.code !== 'ESRCH') throw error;
}
};
const abort = () => {
if (aborted) return;
aborted = true;
kill('SIGTERM');
escalation = setTimeout(() => kill('SIGKILL'), 1000);
escalation.unref();
};
child.stdout.on('data', (chunk) => {
if (capture) stdout += chunk;
else process.stdout.write(chunk);
});
child.stderr.on('data', (chunk) => process.stderr.write(chunk));
child.once('error', (error) => {
failure = error;
});
child.once('exit', () => {
// The shell can exit before a descendant that ignores SIGTERM.
if (aborted) kill('SIGKILL');
});
child.once('close', (code) => {
if (aborted) kill('SIGKILL');
clearTimeout(escalation);
signal?.removeEventListener('abort', abort);
if (aborted) reject(signal.reason);
else if (failure) reject(failure);
else if (code === 0) resolve({ stdout });
else reject(new Error(`${program} failed (exit ${code})`));
});
signal?.addEventListener('abort', abort, { once: true });
if (signal?.aborted) abort();
});
}
async function withLifecycleHooks(root, tmp, env, operation) {
const helpers = await fs.readFile(path.join(root, 'shell/share.sh'), 'utf8');
const environment = await fs.readFile(
path.join(root, 'shell/env.sh'),
'utf8',
);
const directory = await fs.mkdtemp(path.join(tmp, 'upgrade-hooks-'));
try {
// Keep the installed functions available through replacement and rollback.
// Their pkill patterns must never become part of the bash command line.
const snapshot = path.join(directory, 'lifecycle.sh');
await fs.writeFile(
snapshot,
`${helpers}\n${environment}\nimport_config
case "$1" in
stop) delete_pm2 ;;
start) reload_pm2 ;;
install) npm_install_2 "$2"; exit "$exit_status" ;;
*) printf '%s\\n' 'Unknown lifecycle operation' >&2; exit 2 ;;
esac
`,
{
flag: 'wx',
mode: 0o600,
},
);
const hook = (operation, args = [], signal) =>
runProcess(
env,
'bash',
['--', snapshot, operation, ...args],
false,
signal,
);
return await operation(hook);
} finally {
await fs
.rm(directory, { recursive: true, force: true })
.catch(() =>
console.error(
`Retained lifecycle snapshot / 保留生命周期快照: ${directory}`,
),
);
}
}
async function main() {
const env = process.env,
root = env.QL_DIR;
if (!root || !path.isAbsolute(root))
throw new Error('QL_DIR must be absolute.');
const tmp = env.dir_tmp || path.join(root, '.tmp');
const controller = new AbortController();
for (const name of ['SIGINT', 'SIGTERM'])
process.on(name, () =>
controller.abort(new Error(`Upgrade interrupted (${name})`)),
);
const run = (program, args, capture) =>
runProcess(env, program, args, capture, controller.signal);
await withUpgradeLock(tmp, () =>
withLifecycleHooks(root, tmp, env, async (hook) => {
const lifecycle = {
signal: controller.signal,
stop: () => hook('stop'),
start: async (recover) => {
await hook('start', [], recover ? undefined : controller.signal);
await waitForHealth(
env,
60000,
recover ? undefined : controller.signal,
);
},
};
if (process.argv[2] === 'update') {
await stageUpgrade({
root,
tmp,
env,
mirror: process.argv[3],
run,
signal: controller.signal,
install: (source) => hook('install', [source], controller.signal),
});
console.log('更新包下载及校验成功...');
if (process.argv[4] !== 'true') return;
} else if (process.argv[2] !== 'reload')
throw new Error('Unknown upgrade command.');
await reloadSystem({
root,
tmp,
env,
data: env.dir_data || path.join(root, 'data'),
staticRoot: env.dir_static || path.join(root, 'static'),
config: env.dir_config || path.join(root, 'data/config'),
lifecycle,
});
console.log('更新及启动检查成功...');
}),
);
}
module.exports = {
releaseBranch,
verifyUpgrade,
withUpgradeLock,
stageUpgrade,
selectedUpgrade,
replaceAndReload,
reloadSystem,
waitForHealth,
runProcess,
withLifecycleHooks,
main,
};
if (require.main === module)
main().catch((error) => {
console.error(`更新失败: ${error.message}`);
process.exitCode = 1;
});
+52 -28
View File
@@ -53,23 +53,12 @@ export async function request(
// Check HTTP status before decoding (proxies may return HTML on denial).
if (!response.ok) {
await response.body?.cancel();
const suffix =
method !== 'GET' && response.status >= 500
? translate(
process.env,
' 执行结果可能未知,请先检查%s状态再考虑重试。',
resource,
)
: '';
fail(
translate(
process.env,
'API 请求被拒绝(HTTP %s),请检查%s。%s',
response.status,
credentialsHint,
suffix,
),
[401, 403].includes(response.status) ? 3 : 1,
rejectResponse(
response.status,
'HTTP',
method,
resource,
credentialsHint,
);
}
if (
@@ -100,28 +89,63 @@ export async function request(
),
);
}
if (endpoint.split('?')[0] === 'user/login' && isRecord(result) && result.code === 420)
fail(translate(process.env, '需要双因素验证,请调用 user two-factor-login。'), 3);
if (
endpoint.split('?')[0] === 'user/login' &&
isRecord(result) &&
result.code === 420
)
fail(
translate(process.env, '需要双因素验证,请调用 user two-factor-login。'),
3,
);
if (!isRecord(result) || result.code !== 200) {
const status =
isRecord(result) && typeof result.code === 'number'
? result.code
: undefined;
fail(
translate(
process.env,
'API 请求被拒绝(code %s),请检查%s。',
status ?? translate(process.env, '未知'),
credentialsHint,
),
status === 401 || status === 403 ? 3 : 1,
);
rejectResponse(status, 'code', method, resource, credentialsHint);
}
if (options.output)
fail(translate(process.env, '接口返回 JSON,未保存下载文件。'));
return result;
}
function rejectResponse(
status: number | undefined,
kind: 'HTTP' | 'code',
method: string,
resource: string,
credentialsHint: string,
): never {
const denied = status === 401 || status === 403;
const reason = denied
? translate(process.env, '请检查%s。', credentialsHint)
: status === 409
? translate(process.env, '资源冲突,请检查重复的名称或值。')
: status !== undefined && status >= 500
? translate(process.env, '服务端错误,请检查服务状态和日志。')
: translate(process.env, '请求未成功,请检查请求参数和服务日志。');
const suffix =
method !== 'GET' && (status === undefined || status >= 500)
? translate(
process.env,
' 执行结果可能未知,请先检查%s状态再考虑重试。',
resource,
)
: '';
fail(
translate(
process.env,
'API 请求失败(%s %s)。%s%s',
kind,
status ?? translate(process.env, '未知'),
reason,
suffix,
),
denied ? 3 : 1,
);
}
export async function authenticate(config: Credentials): Promise<Session> {
const query = new URLSearchParams({
client_id: config.clientId,
+12
View File
@@ -275,6 +275,18 @@ export const openOperations: readonly OpenOperation[] = [
{ name: 'config save', method: 'POST', path: 'configs/save', body: 'json' },
{ name: 'script list', method: 'GET', path: 'scripts' },
{ name: 'script get', method: 'GET', path: 'scripts/detail' },
{ name: 'script history-list', method: 'GET', path: 'scripts/history' },
{
name: 'script history-detail',
method: 'GET',
path: 'scripts/history/detail',
},
{
name: 'script history-restore',
method: 'PUT',
path: 'scripts/history/restore',
body: 'json',
},
{
name: 'script create',
method: 'POST',
+1 -1
View File
@@ -241,7 +241,7 @@ export async function openCommand(
body,
authenticated: !op.anonymous,
output: values.output as string | undefined,
text: op.path === 'system/log',
text: ['system/log', 'system/config/node-mirror'].includes(op.path),
timeoutMs: values.timeout ? Number(values.timeout) * 1000 : undefined,
});
const data =
+6
View File
@@ -151,6 +151,12 @@ export const english: Record<string, string> = {
应用凭据: 'application credentials',
'应用凭据和 %s 权限': 'application credentials and %s permission',
未知: 'unknown',
'请检查%s。': 'Check %s.',
'资源冲突,请检查重复的名称或值。': 'Resource conflict; check for duplicate names or values.',
'服务端错误,请检查服务状态和日志。': 'Server error; check service status and logs.',
'请求未成功,请检查请求参数和服务日志。': 'Request unsuccessful; check request parameters and service logs.',
'API 请求失败(%s %s)。%s%s': 'API request failed (%s %s). %s%s',
' 执行结果可能未知,请先检查%s状态再考虑重试。':
' Execution outcome may be unknown; check %s status before retrying.',
'API 请求被拒绝(HTTP %s),请检查%s。%s':
+9 -8
View File
@@ -7,7 +7,7 @@ const { execFileSync, spawnSync } = require('node:child_process');
const { createContext, sourceEnvironment } = require('../../dist/internal/runtime/context');
const { executeTask } = require('../../dist/internal/execution/taskRunner');
test('unmodified legacy Shell and TS agree on shell hook state and account modes', async (t) => {
test('Shell and CLI agree on isolated script state and account modes', async (t) => {
const root = await fs.realpath(
await fs.mkdtemp(path.join(os.tmpdir(), 'ql-differential-')),
);
@@ -31,7 +31,7 @@ test('unmodified legacy Shell and TS agree on shell hook state and account modes
path.join(root, 'static/build'),
])
await fs.mkdir(directory, { recursive: true });
for (const name of ['task.sh', 'otask.sh', 'share.sh', 'api.sh', 'env.sh'])
for (const name of ['task.sh', 'task-timeout.sh', 'otask.sh', 'share.sh', 'api.sh', 'env.sh'])
await fs.copyFile(
path.resolve(__dirname, '../../../shell', name),
path.join(context.paths.dir_shell, name),
@@ -110,13 +110,13 @@ test('unmodified legacy Shell and TS agree on shell hook state and account modes
async () => {
const cases = [
{ name: 'return', script: 'return 7', code: 7, legacy: 0, after: true },
{ name: 'exit', script: 'exit 7', code: 7, legacy: 7, after: false },
{ name: 'exit', script: 'exit 7', code: 7, legacy: 0, after: true },
{
name: 'errexit',
script: 'set -e; false',
code: 1,
legacy: 1,
after: false,
legacy: 0,
after: true,
},
{
name: 'trap-return',
@@ -130,8 +130,8 @@ test('unmodified legacy Shell and TS agree on shell hook state and account modes
name: 'trap-exit',
script: 'trap \'printf "exit-trap\\n" >> "$TRACE"\' EXIT; exit 7',
code: 7,
legacy: 7,
after: false,
legacy: 0,
after: true,
trap: true,
},
];
@@ -174,8 +174,8 @@ test('unmodified legacy Shell and TS agree on shell hook state and account modes
assert.equal(result.exitCode, item.code, item.name);
const expected = [
'before',
...(item.after ? [`after:${item.code}`] : []),
...(item.trap ? ['exit-trap'] : []),
...(item.after ? [`after:${item.code}`] : []),
];
for (const file of [legacyTrace, modernTrace])
assert.deepEqual(
@@ -355,6 +355,7 @@ t() { :; }
sleep() { printf DELAYED; }
enter_script_workdir() { :; }
run_else() { :; }
run_task_command() { "$@"; }
which_program=true
main "$@"
`,
@@ -10,7 +10,7 @@ test('legacy Shell and TS resolve executable paths after selecting the working d
t.after(() => fs.rmSync(root, { recursive: true, force: true }));
for (const dir of ['shell/preload', 'shell/lang', 'data/config', 'data/scripts/bin', 'data/scripts/custom', 'data/log', 'static/build', 'bin'])
fs.mkdirSync(path.join(root, dir), { recursive: true });
for (const name of ['task.sh', 'otask.sh', 'share.sh', 'api.sh', 'env.sh'])
for (const name of ['task.sh', 'task-timeout.sh', 'otask.sh', 'share.sh', 'api.sh', 'env.sh'])
fs.copyFileSync(path.resolve(__dirname, '../../../shell', name), path.join(root, 'shell', name));
for (const name of ['zh.sh', 'en.sh'])
fs.copyFileSync(path.resolve(__dirname, '../../../shell/lang', name), path.join(root, 'shell/lang', name));
+1 -1
View File
@@ -10,7 +10,7 @@ test('legacy Shell and TS preserve task stdin in each execution mode', t => {
t.after(() => fs.rmSync(root, { recursive: true, force: true }));
for (const dir of ['shell/preload', 'shell/lang', 'data/config', 'data/scripts', 'data/log', 'static/build', 'bin'])
fs.mkdirSync(path.join(root, dir), { recursive: true });
for (const name of ['task.sh', 'otask.sh', 'share.sh', 'api.sh', 'env.sh'])
for (const name of ['task.sh', 'task-timeout.sh', 'otask.sh', 'share.sh', 'api.sh', 'env.sh'])
fs.copyFileSync(path.resolve(__dirname, '../../../shell', name), path.join(root, 'shell', name));
for (const name of ['zh.sh', 'en.sh'])
fs.copyFileSync(path.resolve(__dirname, '../../../shell/lang', name), path.join(root, 'shell/lang', name));
+4 -4
View File
@@ -445,7 +445,7 @@ test('hook exports reach scripts and after hooks share shell state with inline c
assert.match(log, /hook:before:inline:after:7/);
});
test('ordinary shell tasks share unexported variables and functions across before script and after hooks', async (t) => {
test('shell tasks inherit before-hook state without leaking script state into after hooks', async (t) => {
const { executeTask } = require('../../dist/internal/execution/taskRunner');
const root = sandbox(t);
const context = createContext(
@@ -464,7 +464,7 @@ test('ordinary shell tasks share unexported variables and functions across befor
);
fs.writeFileSync(
context.paths.file_task_after,
'script_function\nprintf "after:%s:%s:%s\\n" "$before_value" "$script_value" "$_task_exit_code"\n',
'if declare -F script_function >/dev/null; then exit 99; fi\nprintf "after:%s:%s:%s\\n" "$before_value" "${script_value-unset}" "$_task_exit_code"\n',
);
const result = await executeTask(context, {
argv: ['shared.sh'],
@@ -478,7 +478,7 @@ test('ordinary shell tasks share unexported variables and functions across befor
);
assert.match(
log,
/before:private\narg:space value\nscript:changed\nafter:private:changed:7/,
/before:private\narg:space value\nafter:private:unset:7/,
);
});
@@ -681,7 +681,7 @@ test('designated shell accounts retain hook functions and script state in a shar
path.join(context.paths.dir_log, result.logPath),
'utf8',
);
assert.match(log, /selected:third&first\nafter:third&first:from_script:4/);
assert.match(log, /selected:third&first\nafter:third&first::4/);
});
test('lifecycle reports share execution identity and statistics survive a rejected final status', async (t) => {
+51 -23
View File
@@ -15,8 +15,17 @@ const {
reloadPanel,
} = require('../../dist/internal/maintenance/upgrade');
const { parse } = require('../helpers/commands.cjs');
const {
writeManifest,
sourceCommit,
} = require('../../../test/helpers/upgrade-fixture.cjs');
async function fixture(t) {
t.mock.method(
require('../../dist/internal/maintenance/upgradeArtifacts.cjs'),
'waitForHealth',
async () => {},
);
const root = await fs.realpath(
await fs.mkdtemp(path.join(os.tmpdir(), 'ql-operator-')),
);
@@ -119,14 +128,27 @@ test('PM2 restarts retain configured ports without mutating legacy environment',
ctx.env.CALLS = path.join(ctx.root, 'ports.jsonl');
ctx.env.BACK_PORT = '5600';
ctx.env.GRPC_PORT = '5400';
await stub(ctx, 'pm2', 'require("node:fs").appendFileSync(process.env.CALLS, JSON.stringify([process.env.BACK_PORT,process.env.GRPC_PORT])+"\\n");');
await stub(
ctx,
'pm2',
'require("node:fs").appendFileSync(process.env.CALLS, JSON.stringify([process.env.BACK_PORT,process.env.GRPC_PORT])+"\\n");',
);
await startPanel(ctx);
ctx.env.QlPort = '5799';
ctx.env.QlGrpcPort = '5599';
await startPanel(ctx);
assert.deepEqual((await fs.readFile(ctx.env.CALLS, 'utf8')).trim().split('\n').map(JSON.parse), [
['5700', '5500'], ['5700', '5500'], ['5799', '5599'], ['5799', '5599'],
]);
assert.deepEqual(
(await fs.readFile(ctx.env.CALLS, 'utf8'))
.trim()
.split('\n')
.map(JSON.parse),
[
['5700', '5500'],
['5700', '5500'],
['5799', '5599'],
['5799', '5599'],
],
);
assert.equal(ctx.env.BACK_PORT, '5600');
assert.equal(ctx.env.GRPC_PORT, '5400');
});
@@ -199,24 +221,41 @@ test('upgrade staging extracts both archives before publishing readiness and rej
const payloads = path.join(ctx.root, 'payloads');
const archives = path.join(ctx.root, 'archives');
await fs.mkdir(archives);
await fs.mkdir(path.join(payloads, 'qinglong-master'), { recursive: true });
await fs.mkdir(path.join(payloads, `qinglong-${sourceCommit}`), {
recursive: true,
});
await fs.mkdir(path.join(payloads, 'qinglong-static-master/build'), {
recursive: true,
});
const manifest = '{"name":"fixture"}';
await fs.writeFile(path.join(ctx.root, 'package.json'), manifest);
await fs.writeFile(
path.join(payloads, 'qinglong-master/package.json'),
path.join(payloads, `qinglong-${sourceCommit}/package.json`),
manifest,
);
await fs.writeFile(
path.join(payloads, 'qinglong-static-master/build/app.js'),
'console.log("fixture")',
);
await fs.mkdir(path.join(payloads, `qinglong-${sourceCommit}/sample`));
await fs.writeFile(
path.join(payloads, `qinglong-${sourceCommit}/sample/config.sample.sh`),
'new sample',
);
writeManifest(
path.join(payloads, `qinglong-${sourceCommit}`),
path.join(payloads, 'qinglong-static-master'),
);
await fs.writeFile(path.join(ctx.root, 'pnpm-lock.yaml'), 'fixture lock\n');
await fs.mkdir(path.join(ctx.root, 'node_modules'));
for (const repo of ['qinglong', 'qinglong-static'])
execFileSync(
'/usr/bin/zip',
['-qry', path.join(archives, `${repo}.zip`), `${repo}-master`],
[
'-qry',
path.join(archives, `${repo}.zip`),
`${repo}-${repo === 'qinglong' ? sourceCommit : 'master'}`,
],
{ cwd: payloads },
);
ctx.env.ARCHIVES = archives;
@@ -225,18 +264,6 @@ test('upgrade staging extracts both archives before publishing readiness and rej
'curl',
'const fs=require("node:fs"),p=require("node:path"),args=process.argv.slice(2),out=args[args.indexOf("--output")+1]; fs.copyFileSync(p.join(process.env.ARCHIVES,p.basename(out)),out);',
);
await fs.mkdir(path.join(payloads, 'qinglong-master/sample'), {
recursive: true,
});
await fs.writeFile(
path.join(payloads, 'qinglong-master/sample/config.sample.sh'),
'new sample',
);
execFileSync(
'/usr/bin/zip',
['-qry', path.join(archives, 'qinglong.zip'), 'qinglong-master'],
{ cwd: payloads },
);
const staged = await stageUpgrade(ctx, 'github');
assert.equal(
await fs.readFile(path.join(staged.source, 'package.json'), 'utf8'),
@@ -249,15 +276,15 @@ test('upgrade staging extracts both archives before publishing readiness and rej
'utf8',
),
),
staged,
{ ...staged, sourceCommit, dependenciesChanged: false },
);
await fs.symlink(
'../../outside',
path.join(payloads, 'qinglong-master/escape'),
path.join(payloads, `qinglong-${sourceCommit}/escape`),
);
execFileSync(
'/usr/bin/zip',
['-qry', path.join(archives, 'qinglong.zip'), 'qinglong-master'],
['-qry', path.join(archives, 'qinglong.zip'), `qinglong-${sourceCommit}`],
{ cwd: payloads },
);
await assert.rejects(stageUpgrade(ctx, 'github'), /symlinks/);
@@ -728,7 +755,7 @@ test('invalid or incomplete staged pointers cannot stop the installed panel', as
reloadPanel(ctx, 'system'),
(error) =>
error.code === 'ENOENT' &&
error.path === path.join(ctx.paths.dir_tmp, 'qinglong-master'),
error.path === path.join(ctx.paths.dir_tmp, 'upgrade-ready-master.json'),
);
});
@@ -783,6 +810,7 @@ test('system upgrade preserves installed dotenv bytes and permissions even when
);
await fs.writeFile(path.join(staticRoot, 'build/app.js'), '');
await fs.writeFile(path.join(source, '.env'), 'PORT=9999\nREPLACE_ME=true\n');
writeManifest(source, staticRoot);
const target = path.join(ctx.root, '.env');
const original = 'PORT=5700\nPRIVATE_VALUE=preserved\n';
await fs.writeFile(target, original, { mode: 0o600 });
@@ -409,7 +409,7 @@ test('cron metadata precedence agrees with unchanged legacy add_cron', (t) => {
'--noprofile',
'--norc',
'-c',
`${definition}\nt() { :; }\nnotify_api() { :; }\nadd_cron_api() { printf '%s' "$1"; }\nadd_cron "$dir_scripts/list" repo`,
`${definition}\nt() { :; }\nnotify_api() { :; }\nadd_cron_api() { printf '%s:%s:%s:%s' "$1" "$2" "$3" "$4"; }\nadd_cron "$dir_scripts/list" repo`,
],
{
encoding: 'utf8',
+3 -2
View File
@@ -27,13 +27,14 @@ test('shared Shell lifecycle scopes timeouts to the task and runs cleanup exactl
{ name: 'failure', task: 'return 7', code: 7 },
{ name: 'function timeout preserves state', task: 'STATE=changed; sleep 10', code: 124, timeout: true, state: 'changed' },
{ name: 'external timeout', args: ['sleep', '10'], code: 124, timeout: true },
{ name: 'Shell script timeout preserves state', task: 'STATE=changed; sleep 10', shell: true, code: 124, timeout: true, state: 'changed' },
{ name: 'Shell script timeout preserves parent hook state', task: 'STATE=changed; sleep 10', shell: true, code: 124, timeout: true },
{ name: 'slow before is outside task timeout', before: 'sleep 0.2', task: 'return 0', code: 0 },
{ name: 'slow after is outside task timeout', after: 'sleep 0.2', task: 'return 0', code: 0 },
{ name: 'background child cannot hold pipes after timeout cleanup', task: '(trap "" INT; sleep 30) & sleep 30', after: 'sleep 0.3', code: 124, timeout: true },
{ name: 'script may redirect fd 3', shell: true, task: 'exec 3> "$TRACE.fd"; return 0', after: 'sleep 0.3', code: 0 },
{ name: 'before may close fd 3 and redirect fd 9', before: 'exec 3>&- 9> "$TRACE.lock"', task: 'return 0', after: 'sleep 0.3', code: 0 },
{ name: 'timeout cleanup survives script fd 3 redirection', shell: true, task: 'exec 3> "$TRACE.fd"; STATE=changed; sleep 30', after: 'sleep 0.3', code: 124, timeout: true, state: 'changed' },
{ name: 'timeout cleanup survives script fd 3 redirection', shell: true, task: 'exec 3> "$TRACE.fd"; STATE=changed; sleep 30', after: 'sleep 0.3', code: 124, timeout: true },
{ name: 'Shell exit preserves user trap and runs after once', shell: true, task: 'trap \'printf "exit\\n" >> "$TRACE"\' EXIT; exit 9', code: 9, userExit: true },
{ name: 'explicit exit preserves user trap and skips after', task: 'trap \'printf "exit\\n" >> "$TRACE"\' EXIT; exit 9', code: 9, noAfter: true, userExit: true },
{ name: 'user EXIT trap survives timeout with parent cleanup', task: 'trap \'printf "exit\\n" >> "$TRACE"\' EXIT; sleep 10', code: 124, timeout: true, userExit: true },
{ name: 'forced termination falls back to cleanup', task: 'trap "" INT; sleep 30', code: 124, timeout: true },
@@ -6,10 +6,16 @@ const os = require('node:os');
const { createContext } = require('../../dist/internal/runtime/context');
const { reloadPanel } = require('../../dist/internal/maintenance/upgrade');
const operator = require('../../dist/internal/maintenance/operator');
const { writeManifest } = require('../../../test/helpers/upgrade-fixture.cjs');
const selectedLoader =
'// QL_CLI_ROOT dist/local/entrypoints.js dist/local/cronEntrypoint.js\n';
async function fixture(t) {
t.mock.method(
require('../../dist/internal/maintenance/upgradeArtifacts.cjs'),
'waitForHealth',
async () => {},
);
const base = await fs.mkdtemp(
path.join(os.tmpdir(), 'ql-upgrade-selection-'),
);
@@ -43,6 +49,7 @@ async function fixture(t) {
await fs.writeFile(incoming, '// original released loader');
await fs.writeFile(path.join(ctx.paths.dir_static, 'build/app.js'), 'old');
await fs.writeFile(path.join(staticRoot, 'build/app.js'), 'new');
writeManifest(source, staticRoot);
return { ctx, source, staticRoot, loader, incoming };
}
@@ -104,3 +111,33 @@ test('unsupported target and unrecognized loader fail before service stop', asyn
assert.equal(stopped, false);
assert.equal(await fs.readFile(f.loader, 'utf8'), 'unrecognized');
});
test('successful process start with failed health check rolls back the installed version', async (t) => {
const f = await fixture(t);
t.mock.method(operator, 'stopPanel', async () => {});
t.mock.method(operator, 'startPanel', async () => ({ manager: 'fixture' }));
let probes = 0;
t.mock.method(
require('../../dist/internal/maintenance/upgradeArtifacts.cjs'),
'waitForHealth',
async () => {
if (++probes === 1) throw new Error('health check failed');
assert.equal(
await fs.readFile(
path.join(f.ctx.paths.dir_static, 'build/app.js'),
'utf8',
),
'old',
);
},
);
await assert.rejects(
reloadPanel(f.ctx, 'system', { source: f.source, static: f.staticRoot }),
/health check failed/,
);
assert.equal(probes, 2);
assert.equal(
await fs.readFile(path.join(f.ctx.root, 'package.json'), 'utf8'),
'{"version":"2.19.0"}',
);
});
+52 -1
View File
@@ -54,7 +54,7 @@ test('API diagnostics preserve resource scope, uncertainty, codes and secret sup
language === 'en' ? /outcome.*unknown/ : /结果.*未知/,
);
}
if (failure !== 'network') {
if (failure === 'permission' || failure === 'api') {
assert.ok(
error.message.includes(
endpoint.startsWith('subscriptions')
@@ -89,3 +89,54 @@ test('API diagnostics preserve resource scope, uncertainty, codes and secret sup
);
}
});
test('HTTP and body errors distinguish conflicts, authentication and server failures without leaking bodies', async (t) => {
const previous = process.env.QL_LANG;
t.after(() => {
if (previous === undefined) delete process.env.QL_LANG;
else process.env.QL_LANG = previous;
});
for (const language of ['en', 'zh']) {
process.env.QL_LANG = language;
for (const kind of ['HTTP', 'code'])
for (const status of [400, 401, 403, 409, 500, 503])
for (const method of ['GET', 'PUT']) {
t.mock.method(
global,
'fetch',
async () =>
new Response(
JSON.stringify({ code: status, message: 'secret-value' }),
{ status: kind === 'HTTP' ? status : 200 },
),
);
await assert.rejects(
request(
{ url: 'https://fixture.invalid', token: 'secret-token' },
'envs',
{ method },
),
(error) => {
assert.equal(error.exitCode, [401, 403].includes(status) ? 3 : 1);
assert.match(error.message, new RegExp(`${kind} ${status}`));
assert.doesNotMatch(error.message, /secret-|fixture/);
if ([401, 403].includes(status))
assert.match(error.message, /credentials|凭据/);
else
assert.doesNotMatch(
error.message,
/credentials|permission|凭据|权限/,
);
if (status === 409) assert.match(error.message, /conflict|冲突/);
if (status >= 500)
assert.match(error.message, /Server error|服务端错误/);
if (status >= 500 && method === 'PUT')
assert.match(error.message, /unknown|未知/);
else assert.doesNotMatch(error.message, /unknown|未知/);
return true;
},
);
t.mock.restoreAll();
}
}
});
+51
View File
@@ -58,6 +58,31 @@ test('dashboard record sends execution statistics and rejects a missing body bef
assert.equal(f.requests.length, 1);
});
test('script history commands send exact query parameters and restore payloads', async t => {
const f = await fixture(t);
const identity = { filename: '中文 script.js', path: 'nested folder' };
const version = { ...identity, id: '32ec8b22-2c48-4bcf-85d4-b394198417ba' };
for (const [command, endpoint, query] of [
['history-list', 'scripts/history', identity],
['history-detail', 'scripts/history/detail', version],
]) {
success(await f.run(['script', command, '--query', JSON.stringify(query)]));
const req = f.requests.at(-1), url = new URL(req.url, 'http://localhost');
assert.equal(req.method, 'GET');
assert.equal(url.pathname, '/panel/open/' + endpoint);
for (const [key, value] of Object.entries(query)) assert.equal(url.searchParams.get(key), value);
assert.equal(req.body, '');
}
const payload = { ...version, expectedHash: 'a'.repeat(64) };
success(await f.run(['script', 'history-restore', '--data', '-'], JSON.stringify(payload)));
assert.equal(f.requests.at(-1).method, 'PUT');
assert.equal(f.requests.at(-1).url, '/panel/open/scripts/history/restore');
assert.deepEqual(JSON.parse(f.requests.at(-1).body), payload);
const count = f.requests.length;
assert.equal((await f.run(['script', 'history-restore'])).code, 2);
assert.equal(f.requests.length, count, 'missing restore payload must not issue a request');
});
test('OpenAPI catalogue covers every active registered backend route; retired 410 routes are explicit', async () => {
const root = path.resolve(__dirname, '../../../back/api');
const expected = [];
@@ -186,3 +211,29 @@ test('owner two-factor challenge remains actionable without exposing server mess
assert.equal(result.code, 3); assert.match(result.err, /two-factor-login/); assert.doesNotMatch(result.err, /SERVER-SECRET/);
assert.equal(f.requests.length, 1); assert.equal(f.requests[0].headers.authorization, undefined);
});
test('node mirror configuration accepts the real empty response for named and raw API commands', async t => {
const f = await fixture(t);
f.reply((_req, res) => { res.writeHead(200, { 'content-type': 'application/octet-stream' }); res.end(); });
for (const args of [
['system', 'config-node-mirror'],
['api', 'request', 'PUT', '/open/system/config/node-mirror'],
]) {
const reply = success(await f.run([...args, '--data', '{"nodeMirror":"https://registry.npmmirror.com"}']));
assert.equal(reply.code, 200);
assert.equal(f.requests.at(-1).method, 'PUT');
assert.equal(f.requests.at(-1).url, '/panel/open/system/config/node-mirror');
}
assert.equal(f.requests.length, 2, 'configuration writes must not be retried');
});
test('empty node mirror responses still reject authentication and server failures without retries', async t => {
const f = await fixture(t);
for (const status of [401, 403, 500]) {
f.reply((_req, res) => { res.writeHead(status, { 'content-type': 'application/octet-stream' }); res.end(); });
const result = await f.run(['system', 'config-node-mirror', '--data', '{}']);
assert.notEqual(result.code, 0);
assert.equal(result.out, '');
}
assert.equal(f.requests.length, 3);
});
+49
View File
@@ -0,0 +1,49 @@
# Node cron 调度器迁移
Node 调度路径使用固定版本 `node-cron@4.6.0`,普通任务(含额外规则)、订阅和系统 cron 共用 `back/shared/cronScheduler.ts`。启动/一次性任务、间隔任务、脚本并发限制和进程执行链路保持原有实现。系统模式只将语义一致的普通五字段规则交给 crond。
移除 `node-schedule`、其类型包以及兼容别名 `cron-parser-v4`。保留 `cron-parser@5` 做旧表达式的校验与标准化;计时器和日历推进由 node-cron 管理。
## 兼容行为
- 保留旧版 1–6 字段补全、五种宏、月份/星期名称、数字起点步长、L 和 #。另外明确新增 `@annually`(等同 `@yearly`)、`@midnight`(等同 `@daily`)、`@minutely`(每分钟第 0 秒)。宏按进程时区解释。`?` 仅允许日期和星期字段;裸 `/N`、H 和其他未列出的宏仍拒绝。
- 日期与星期同时指定时保留 OR 语义,不采用 node-cron 默认的 AND。两个原生任务分别处理日期和星期,并按计划时间过滤交集,避免重复执行;保留旧解析器的月份全日期范围及全局 nth-week 限制。
- 原生任务先暂停创建并验证,再替换旧任务。构造失败时销毁已准备的实例并保留旧快照。
- 进程内迟到的触发通过 `execution:missed` 加入既有执行队列,记录补执行日志。并发和单任务积压上限仍由青龙队列控制;这不提供进程退出期间的持久化补跑或分布式 exactly-once 保证。
- 标准化结果使用上限 512 条的只读缓存,仅缓存语法,不缓存运行实例或下次执行时间。避免大批相同表达式重复解析导致恢复 RPC 超时。
- 回调异常记录日志,后续调度继续;取消时调用原生 `destroy()` 释放计时器和注册表。
- 持久化无效任务在恢复时跳过并记录日志,无效订阅不会产生未处理的启动拒绝。API、导入和执行使用同一套表达式校验。
## 复审修复
- 移除没有生产调用的 `nextInvocation()` 接口,避免对 OR/# 等过滤规则报告不会实际执行的原生日期。回归测试直接观察实际执行和原生实例状态。
- 批量注册 RPC 时限为 5 秒加每条规则 5 毫秒,计入额外规则并封顶 120 秒;保留超时后的恢复机制,不立即重放结果不确定的写入。
- Alpine/系统模式下,数字起点步长(如 `0/5`)、星期包含 `7`、日期步长、日期和星期同时受限的表达式转入 Node 路径。BusyBox 的这些解析或通配语义与旧 Node 解析器不同,不能仅因是纯数字五字段就交给 crond。
- 这些规则在 `crontab.list` 中仍保留为注释,避免同时被系统和 Node 执行。其他可移植规则仍使用系统 crond。
## 夏令时差异
采用 node-cron 的当地时间语义:春季跳时不存在的时间跳过,秋季重复小时不再执行第二次。旧 cron-parser 对部分春季时间会顺延,并可能重复秋季高频任务。使用有夏令时的时区时需要考虑该行为变化;Asia/Shanghai 不受影响。回归测试固定 America/New_York 的 2026 年转换日期。
## 验证
`npm test` 和 `npm run build:back`。
- `test/fixtures/legacy-cron.json` 是迁移前使用 node-schedule 2.1.1 / cron-parser 4.9.0 保存的快照:269 条语法判定,18 组日历序列,每组 10 次执行。
- 真正的 node-cron 实例配合虚拟时钟验证日期/星期 OR、交集去重、闰年、L/#、8 秒迟到补执行、取消释放、回调拒绝和夏令时。
- 注册构造失败回滚、无效订阅隔离、空表达式立即运行,以及现有调度恢复/503/子进程测试覆盖接入流程。
- 隔离的正式 Debian 镜像使用实际编译产物验证 10,000 条有效任务和 2 条错误任务的恢复、300 个同时到期的 Python 脚本、并发上限 16 及 HTTP 健康检查。具体结果见本次测试报告;这是生产镜像隔离验证,不是线上长期运行证明。
### 本次验收结果(2026-09-27)
完整回归 260 通过、3 项原有跳过,后端编译成功。Node 20 生产运行时的 21 项兼容测试通过。隔离镜像中 300/300 脚本完成,无重复或遗漏,最大并发 16,270 个成功及 30 个预设失败均落库;完成耗时 52.36 秒。恢复后健康检查 1,870/1,870 成功,任务列表 38/38 成功。重启恢复阶段另有 50 次健康检查失败。
初版重复解析曾导致恢复超过 5 秒 RPC 时限,加入有上限的语法缓存后重跑通过。本次 10,000 个任务共享两个表达式,不能视为一万种复杂表达式的容量保证;实际脚本耗时也不代表纯调度性能。
新增宏在 UTC 和 Asia/Shanghai 下验证连续两次实际触发及不提前触发;原有语法快照仅对这三个新增宏允许预期变化。
### 复审后验收
完整回归 267 通过、3 项原有跳过;Node 20 的 10 项修复回归及 Alpine 镜像的 24 项兼容/宏测试通过。
额外使用 9,700 个不同表达式和 300 个同时到期任务验证恢复(另有 2 个无效任务),不再局限于共享两个表达式。300/300 脚本完成,无重复或遗漏,最大并发 16,270 个退出码 0、30 个预设退出码 3 均落库。恢复后健康检查 1,650/1,650 成功、任务列表 33/33 成功;启动恢复阶段另有 260 次健康检查失败,整批脚本耗时 60.27 秒。较大快照恢复仍需等待,不承诺启动阶段始终可用或任意规模都能在时限内恢复。
@@ -0,0 +1,7 @@
# Python 依赖来源
依赖管理中可手动添加 Python 包名、`git+https://` 等 pip 支持的源 URL,或脚本/订阅仓库目录内的 requirements 文件、pyproject.toml 路径。文件路径支持目录空格,必须位于持久化 scripts 或 repo 根目录内;越界与指向根目录外的符号链接会被拒绝。
requirements 使用 `pip install -r`,pyproject 使用完整项目目录安装,保留相对依赖文件和项目源码。拉取订阅不会自动安装仓库声明的依赖。
删除 requirements 条目会调用 `pip uninstall -r`。源 URL 或 pyproject 条目无法可靠逆向推断其安装的所有分发包,因此删除该条目只移除面板记录;需另外按包名卸载已安装的包,日志会明确提示。
+78
View File
@@ -0,0 +1,78 @@
# Task startup and notification loading
`task` performs shell configuration, status reporting and dependency-path discovery
before invoking the selected runtime. These costs are separate from cron callback
latency and time spent waiting for an execution slot.
## Dependency-path cache
Node scheduling usually starts in `/ql`, while Alpine crond starts in `/root`.
The pnpm discovery key includes the working directory, npm/pnpm configuration,
relevant environment and executable metadata. Each key now has its own cached
record, so alternating scheduler environments do not evict each other.
Records expire after 60 seconds. Refreshes share a persistent per-user lock;
records older than an hour are removed on refresh. Warm lookups do not scan the
cache directory. The lock is never removed while another process might use it.
The opt-out `QL_NODE_PATH_CACHE=0`, failed-discovery fallback, and package resolution
order remain available. This caches the dependency directory, not its contents.
## Language startup paths
| Invocation | Environment and before hooks | Automatic notification loading |
| --- | --- | --- |
| `task script.py` / `.pyc` | Python preload imports generated environment, runs the shell/command before hooks, imports `task_before.py`, then applies account selection | `QLAPI.notify(...)` imports `__ql_notify__` on first call |
| `task script.js` / `.mjs` / `.ts` | Node preload imports generated environment, runs shell/command before hooks, requires `task_before.js`, then applies account selection | `QLAPI.notify(...)` requires `__ql_notify__.js` on first call |
| `task script.sh` | Shell sources the generated environment and runs shell before hooks | No automatic Python/Node notification module |
| Explicit interpreter or other command, e.g. `task python3 script.py` | Retains generated environment and shell before-hook exports; it does not automatically install the Python/Node preload | No automatic notification module unless configured by the caller |
TypeScript uses `ts-node-transpile-only`; MJS uses the same Node preload plus the
ESM loader. There are no separate built-in Ruby, Go, Java or PHP preload modules.
All these task paths still pass through shared dependency-path setup: non-JS
scripts and their hooks may themselves call Node or use the QLAPI bridge.
The before hooks are deliberately still synchronous: they can export environment
variables or perform work required by the user's script. Python's preload still
starts a child shell/Python to capture that environment; Node's preload starts a
child shell/Node. Node also still loads its gRPC client eagerly. These are remaining
startup costs, not work performed by the cron library, and are not removed by this
change.
Shell scripts run in a child shell that inherits the wrapper's environment and
before-hook functions. This lets the wrapper run its after hooks and report the
exit status even when a script calls `exit`, replaces itself with `exec`, or fails
under `set -e`. The script's own EXIT trap still runs in the child. Changes made
inside the script (variables, working directory, functions and shell options)
do not propagate back to the wrapper or its after hooks.
## Lazy notifications
Scripts keep calling `QLAPI.notify` with the same arguments. When no notification
is sent, the notification provider and its channel dependencies are never imported
by the preload. First use loads the provider; subsequent uses reuse the standard
Python/Node module cache. JS forwards the receiver and return value, including
promises, and Python forwards positional and keyword arguments. Import/send errors
surface at the notification call rather than preventing unrelated scripts from
obtaining QLAPI during startup. A failed import can be retried after its cause is
resolved.
Provider initialization, including provider configuration reads, now happens at
first use. A script's explicit `import __ql_notify__`, `require(...)`, or independent
notification helper still loads that module immediately.
## Task timeouts
`task -m DURATION ...` also applies to sourced Shell scripts. The timeout worker
inherits before-hook variables, arrays and functions without serializing them.
At the deadline, the worker's process group receives TERM, followed by KILL after
0.5 seconds, so ordinary child processes cannot continue after the timeout.
Explicit interpreters and other commands use the same mechanism. Processes that
intentionally create a separate session/process group are outside this boundary.
The wrapper records status 124 and runs its after hook; concurrent account mode
retains its existing aggregate status behavior. Zero disables the timeout.
Cron executions waiting for a concurrency slot retain their schedule revision.
Disabling, deleting or replacing that revision prevents its queued executions
from starting when a slot becomes available. Already running executions retain
the existing stop behavior. Invalidated callbacks do not count against a new
revision's limit of five pending/running executions.
+33
View File
@@ -0,0 +1,33 @@
# 青龙 2.23.0
## 新增功能
- 脚本历史版本:查看、对比和恢复历史内容;每个脚本保留最多 20 个版本,支持不超过 1 MiB 的 UTF-8 文本脚本。恢复前校验当前内容哈希,发生并发修改时返回冲突,避免覆盖新内容。
- 独立远程 CLI 0.1.2 增加脚本历史查看、对比和恢复命令,并完善资源冲突及认证错误反馈。最低要求仍为 Node.js 22.12。
## 修复与优化
- 普通任务、额外规则和订阅的 Node 调度路径迁移到 node-cron。保留旧规则标准化、日期/星期 OR、L/# 等兼容行为,补充 `@annually`、`@midnight`、`@minutely`。
- 禁用、删除或修改任务后,旧规则对应的排队执行不会再启动。完善任务进程超时、停止回收、Shell before/after 环境传递和退出状态。
- 修复系统令牌未主动续期和认证缓存并发更新:定时生成与启动后重新计时会主动续期,避免长期运行后失效。Shell API 每次请求检查令牌,遇到 401 时刷新并重试一次;其他错误不自动重试。
- 降低任务启动开销;JS/Python 的 QLAPI 通知依赖在首次通知调用时加载,减少不发送通知的任务启动成本。
- 加固环境变量名和字面值、订阅参数、路径访问及 SSH 配置输入;保留合法的中文及分支别名兼容。
- 修复页面升级、`ql update` 和静态资源的版本同步,完善升级暂存校验及应用、完整备份与恢复流程。
- 修复 Debian 上的依赖安装、通知和调度回归,补齐前端缺失的中英文翻译,改进 Gitee/GitLab 静态资源镜像同步检查。
- 最小升级 compression、multer、@grpc/grpc-js、proxy-addr、websocket-driver、sequelize,修补现有 HTTP、上传、代理信任、WebSocket、gRPC 和 JSON 查询路径相关风险。完整依赖治理、Node/pnpm 迁移及工具打包调整延期。
## 升级说明
升级前停止面板与任务,备份完整持久化数据目录,包括数据库、脚本、配置、日志、依赖缓存及已有的 `script-history` 目录,并记录原镜像摘要和启动配置。使用面板备份接口时,确认选择了脚本、配置和日志;仅备份数据库不能恢复全部数据。
升级后试运行代表性的 JavaScript、Python、Shell 任务,检查订阅、环境变量、依赖安装以及通知。存在夏令时的时区采用 node-cron 行为:跳过春季不存在的当地时间,秋季重复小时不重复执行;Asia/Shanghai 不受影响。调度器不提供进程退出期间的持久化补跑保证。
任务超时会终止普通子进程所在的进程组并记录退出状态 124;主动创建独立会话/进程组的程序不在该边界内。并发账号执行模式保留既有汇总退出状态。
本次沿用现有 Dockerfile 和 pnpm 8.3.1。默认 Debian 镜像使用 Debian 12 与 Node.js 20,默认 Alpine 镜像的 Node.js 由 APK 提供;默认 Python 为 3.11。Node 20 与 Node 18 构建运行时已结束上游维护,相关迁移延期,功能验证通过不代表全部依赖风险清零。
2.23.0 是 Python 3.10 兼容镜像的最后一个版本。`2.23.0-python3.10` 和 `2.23.0-debian-python3.10` 发布后停止维护,不再提供后续更新;请验证脚本和依赖后迁移至默认 Python 3.11 镜像。
主包 `@whyour/qinglong` 与独立远程包 `@whyour/qinglong-cli` 的 `ql` 命令用途不同。在已有面板的主机上使用独立 npm prefix 或 `npm exec --package=@whyour/qinglong-cli -- ql ...`,避免覆盖面板内置入口。
发布时间:2026-10-11 12:27:02(Asia/Shanghai)。本版本已完成四类镜像验证及独立 dev 环境超过两小时的持续功能验收。
+53
View File
@@ -0,0 +1,53 @@
# 脚本编辑历史
脚本管理工具栏的“历史版本”打开右侧抽屉。选择版本后,对比历史内容与服务器上当前已保存文件;桌面使用双栏对比,手机使用行内对比。恢复需要确认,有未保存修改时必须先保存或取消。恢复前保留当前版本,因此恢复操作也可以撤销。普通保存不再弹出不可恢复确认。
## 记录范围
- 编辑器 `PUT /api/scripts` 和调试窗口 `POST /api/scripts` 覆盖已有脚本时,在内容变化时留档,首次修改前的内容也保留;相同内容不写历史。
- 历史只包含保存前真实存在的内容;不把失败写入当成一个内容版本。
- 上传、新增、重命名、删除、订阅更新和外部编辑不主动留档。若下次面板保存时检测到外部变更,会保留那一刻的内容,标记为“面板外修改后”。这不代表外部修改之前的内容受到保护。
- 按脚本规范化后的相对路径隔离历史;不同目录同名文件不会混淆,指向同一脚本的安全符号链接共用历史。重命名后不会自动迁移历史。调试器另存为覆盖已有脚本时,目标旧内容进入历史,删除源文件前将其旧内容保留到现有 bak 目录;备份失败则不覆盖目标、不删除源文件。源和目标实际指向同一脚本时(如安全符号链接或不同路径写法),按原地保存处理,保留两个路径,不删除源文件。历史不能用于回收已删除的文件。
- 历史记录支持至多 1 MiB 的 UTF-8 文本文件(含空文件)。当前文件或新内容超出范围时,界面要求明确确认“不记录历史直接保存”;取消则保持文件和草稿不变。确认后以 `skipHistory: true` 和服务端返回的当前哈希重试,锁内检查哈希以防确认期间覆盖他人修改。成功后明确提示本次未留档。大文件可以通过此流程缩小,后续恢复普通留档。未确认的 API 请求返回 413,并包含 `historyUnavailable` 和 `currentHash`;历史读取损坏等其他错误不允许用此选项绕过。
## 存储与资源边界
位置:`<QL_DATA_DIR>/script-history/<SHA-256(规范相对路径)>.br`。目录位于脚本目录之外,避免进入脚本树、订阅和任务扫描。目录权限 `0700`,历史文件权限 `0600`。目录随整个数据目录备份。
每个脚本仅一个 Brotli 压缩 JSON 包,存储完整的旧内容快照以及时间、来源、内容哈希。新文件内容不再重复存储,只保存头版本元数据,供下一次留档标注来源。面板服务通过 TypeDI 注册为单例,路由使用 `Container.get(ScriptHistoryService)` 获取;构造实例只初始化配置,首次发生保存时才创建历史目录。包内重复文本共享压缩窗口,不依赖补丁链,也不需要额外数据库、文件监听、后台任务或常驻缓存。使用 Node 内置 zlib 异步压缩,Brotli quality=4、window=4 MiB,不新增依赖。
| 限制 | 默认值 |
| --- | --- |
| 每脚本历史版本 | 最多 20 个 |
| 单个脚本内容 | 1 MiB |
| 单包解压后的 JSON | 8 MiB |
| 单个压缩历史包 | 2 MiB |
| 所有压缩历史包合计 | 32 MiB |
| 有历史的脚本数 | 最多 1000 个 |
| 历史写入暂存文件 | 共享一个,最多 2 MiB |
| 脚本写入暂存文件 | 每次保存一个,与本次保存内容等大,结束后清理 |
达到单文件限制时删除最旧版本,至少保留本次覆盖前的内容;无法保留则拒绝覆盖。达到全局限制时,在脚本成功提交后优先清理最久未更新的脚本历史包(该文件的整份历史),不删除脚本本身。历史暂存、提交或脚本提交失败时不清理任何其他历史;脚本提交失败会回滚本文件历史。如果提交成功但清理失败,返回 `cleanupPending`,界面提示检查存储权限和空间,后续保存重新尝试清理。配额是正常清理成功后的目标,事务期间和清理失败时会暂时超出。32 MiB 是文件内容字节数,不包含文件系统块分配、目录和锁的开销。清理只在发生保存时执行,通常最多扫描 1000 个包的元数据(清理失败期间可能更多),不解压其他脚本的历史。读取仅解压当前脚本,限制解压输出长度。
文件保存复用现有锁路径,同时以独立跨进程目录锁串行化历史写入和配额清理;`finally` 释放锁。先准备并同步脚本暂存文件,保留文件权限和所有者;随后写并同步历史暂存文件、原子提交历史,再原子替换脚本。任一步准备失败不会截断原脚本;最后替换脚本失败时还原此前历史;成功后才清理配额。超限确认保存不创建新历史包,使用同样的脚本暂存和替换方式,并流式计算旧文件哈希。外部进程不遵守该锁时,不承诺与其并发写入的事务一致性。
恢复请求携带预览时的当前内容哈希,锁内检查后才写入;文件变化则返回 409。历史读写与脚本读取采用相同的路径和符号链接边界检查。面板配置目录中的敏感文件仍受黑名单保护;脚本目录中的 auth.json、token.json 等普通脚本名称可以正常保存和留档。
## 压缩样本
运行 `node scripts/benchmark-script-history.cjs` 可复现比较。模拟同一脚本连续 20 次小修改,比较独立 gzip、合并 gzip 与合并 Brotli。首次本机测量:
| 样本 | 独立 gzip | 合并 gzip | 合并 Brotli(quality 4) |
| --- | ---: | ---: | ---: |
| sample/notify.js | 242,430 B | 221,832 B | 13,930 B |
| shell/task.sh | 30,159 B | 2,317 B | 1,737 B |
这是文本重复度较高的模拟样本,不是所有工作负载的压缩保证。Brotli 合包特别适合大小超过 gzip 32 KiB 窗口、但每次只改少量内容的脚本。增量补丁可能在部分样本更小,但增加补丁生成、链式恢复、裁剪和损坏恢复的复杂度;本实现选择压缩合包与硬上限,在磁盘、CPU 和可靠性之间取平衡。
## API
- `GET /api/scripts/history?filename=...&path=...`:版本元数据、当前哈希、保留数量。
- `GET /api/scripts/history/detail?filename=...&path=...&id=...`:历史内容、当前内容及当前哈希。
- `PUT /api/scripts/history/restore`:`{filename, path, id, expectedHash}`。
沿用现有 scripts 路由的认证与权限。历史 ID 使用 UUID;不存在或已清理返回 404,文件已变更返回 409,文本或容量限制返回 413。
+12 -11
View File
@@ -1,7 +1,7 @@
{
"name": "@whyour/qinglong",
"packageManager": "pnpm@8.3.1",
"version": "2.22.0",
"version": "2.23.0",
"description": "Timed task management platform supporting Python3, JavaScript, Shell, Typescript",
"repository": {
"type": "git",
@@ -28,7 +28,8 @@
"panel": "npm run build:back && node static/build/app.js",
"gen:proto": "protoc --experimental_allow_proto3_optional --plugin=./node_modules/.bin/protoc-gen-ts_proto ./back/protos/*.proto --ts_proto_out=./ --ts_proto_opt=outputServices=grpc-js,env=node,esModuleInterop=true,snakeToCamel=false",
"prettier": "prettier --write '**/*.{js,jsx,tsx,ts,less,md,json}'",
"postinstall": "max setup 2>/dev/null || true"
"postinstall": "max setup 2>/dev/null || true",
"start:simple": "bash shell/start-simplify.sh"
},
"gitHooks": {
"pre-commit": "lint-staged"
@@ -74,23 +75,24 @@
"overrides": {
"sqlite3": "npm:@whyour/sqlite3@1.1.2",
"@codemirror/state": "6.5.4",
"@codemirror/view": "6.39.16"
"@codemirror/view": "6.39.16",
"proxy-addr": "2.0.8",
"websocket-driver": "0.7.5"
}
},
"dependencies": {
"@ant-design/plots": "^2.6.8",
"@bufbuild/protobuf": "^2.10.0",
"@grpc/grpc-js": "^1.14.0",
"@grpc/grpc-js": "1.14.5",
"@grpc/proto-loader": "^0.8.0",
"@keyv/sqlite": "^4.0.1",
"@otplib/preset-default": "^12.0.1",
"body-parser": "^1.20.3",
"celebrate": "^15.0.3",
"chokidar": "^4.0.1",
"compression": "^1.7.4",
"compression": "1.8.2",
"cors": "^2.8.5",
"cron-parser": "^5.4.0",
"cron-parser-v4": "npm:cron-parser@4.9.0",
"cross-spawn": "^7.0.6",
"dayjs": "^1.11.13",
"dotenv": "^16.4.6",
@@ -107,15 +109,15 @@
"jsonwebtoken": "^9.0.2",
"keyv": "^5.2.3",
"lodash": "^4.17.21",
"multer": "2.1.1",
"node-schedule": "^2.1.0",
"nodemailer": "^8.0.1",
"multer": "2.4.0",
"node-cron": "4.6.0",
"nodemailer": "^10.0.9",
"p-queue-cjs": "7.3.4",
"proper-lockfile": "^4.1.2",
"ps-tree": "^1.2.0",
"reflect-metadata": "^0.2.2",
"request-ip": "3.3.0",
"sequelize": "^6.37.5",
"sequelize": "6.37.8",
"sockjs": "^0.3.24",
"sqlite3": "npm:@whyour/sqlite3@1.1.2",
"toad-scheduler": "^3.0.1",
@@ -145,7 +147,6 @@
"@types/lodash": "^4.14.185",
"@types/multer": "^1.4.7",
"@types/node": "^17.0.21",
"@types/node-schedule": "^1.3.2",
"@types/nodemailer": "^6.4.4",
"@types/proper-lockfile": "^4.1.4",
"@types/ps-tree": "^1.1.6",
+393 -91
View File
@@ -4,6 +4,8 @@ overrides:
sqlite3: npm:@whyour/sqlite3@1.1.2
'@codemirror/state': 6.5.4
'@codemirror/view': 6.39.16
proxy-addr: 2.0.8
websocket-driver: 0.7.5
dependencies:
'@ant-design/plots':
@@ -13,8 +15,8 @@ dependencies:
specifier: ^2.10.0
version: 2.10.0
'@grpc/grpc-js':
specifier: ^1.14.0
version: 1.14.0
specifier: 1.14.5
version: 1.14.5
'@grpc/proto-loader':
specifier: ^0.8.0
version: 0.8.0
@@ -26,7 +28,7 @@ dependencies:
version: 12.0.1
body-parser:
specifier: ^1.20.3
version: 1.20.3
version: 1.20.6
celebrate:
specifier: ^15.0.3
version: 15.0.3
@@ -34,17 +36,14 @@ dependencies:
specifier: ^4.0.1
version: 4.0.1
compression:
specifier: ^1.7.4
version: 1.7.5
specifier: 1.8.2
version: 1.8.2
cors:
specifier: ^2.8.5
version: 2.8.5
cron-parser:
specifier: ^5.4.0
version: 5.4.0
cron-parser-v4:
specifier: npm:cron-parser@4.9.0
version: /cron-parser@4.9.0
cross-spawn:
specifier: ^7.0.6
version: 7.0.6
@@ -74,7 +73,7 @@ dependencies:
version: 1.2.0
http-proxy-middleware:
specifier: ^3.0.3
version: 3.0.3
version: 3.0.7
iconv-lite:
specifier: ^0.6.3
version: 0.6.3
@@ -83,7 +82,7 @@ dependencies:
version: 2.3.0(@types/node@17.0.45)
js-yaml:
specifier: ^4.1.0
version: 4.1.0
version: 4.3.2
jsonwebtoken:
specifier: ^9.0.2
version: 9.0.2
@@ -94,14 +93,14 @@ dependencies:
specifier: ^4.17.21
version: 4.17.21
multer:
specifier: 2.1.1
version: 2.1.1
node-schedule:
specifier: ^2.1.0
version: 2.1.1
specifier: 2.4.0
version: 2.4.0
node-cron:
specifier: 4.6.0
version: 4.6.0
nodemailer:
specifier: ^8.0.1
version: 8.0.1
specifier: ^10.0.9
version: 10.0.9
p-queue-cjs:
specifier: 7.3.4
version: 7.3.4
@@ -118,8 +117,8 @@ dependencies:
specifier: 3.3.0
version: 3.3.0
sequelize:
specifier: ^6.37.5
version: 6.37.5(@whyour/sqlite3@1.1.2)
specifier: 6.37.8
version: 6.37.8(@whyour/sqlite3@1.1.2)
sockjs:
specifier: ^0.3.24
version: 0.3.24
@@ -134,7 +133,7 @@ dependencies:
version: 0.10.0
undici:
specifier: ^7.9.0
version: 7.9.0
version: 7.29.1
uuid:
specifier: ^11.0.3
version: 11.0.3
@@ -203,9 +202,6 @@ devDependencies:
'@types/node':
specifier: ^17.0.21
version: 17.0.45
'@types/node-schedule':
specifier: ^1.3.2
version: 1.3.2
'@types/nodemailer':
specifier: ^6.4.4
version: 6.4.17
@@ -268,7 +264,7 @@ devDependencies:
version: 4.23.0(antd@4.24.16)(react-dom@18.3.1)(react@18.3.1)
axios:
specifier: ^1.4.0
version: 1.7.8
version: 1.20.0
compression-webpack-plugin:
specifier: 9.2.0
version: 9.2.0
@@ -2883,7 +2879,7 @@ packages:
globals: 13.24.0
ignore: 5.3.2
import-fresh: 3.3.0
js-yaml: 4.1.0
js-yaml: 4.3.2
minimatch: 3.1.5
strip-json-comments: 3.1.1
transitivePeerDependencies:
@@ -2968,8 +2964,8 @@ packages:
deprecated: the package is rather renamed to @formatjs/ecma-abstract with some changes in functionality (primarily selectUnit is removed and we don't plan to make any further changes to this package
dev: true
/@grpc/grpc-js@1.14.0:
resolution: {integrity: sha512-N8Jx6PaYzcTRNzirReJCtADVoq4z7+1KQ4E70jTg/koQiMoUSN1kbNjPOqpPbhMFhfU1/l7ixspPl8dNY+FoUg==}
/@grpc/grpc-js@1.14.5:
resolution: {integrity: sha512-7VZM+SVdEcUUqSQeNI3zM8Qs/BhQKZndPo2h5VkYkAM8Iz0wJIa8mKV5ekQGqG8UUsnkQ0NMxIxwkIHYvj0qOw==}
engines: {node: '>=12.10.0'}
dependencies:
'@grpc/proto-loader': 0.8.0
@@ -3064,7 +3060,7 @@ packages:
camelcase: 5.3.1
find-up: 4.1.0
get-package-type: 0.1.0
js-yaml: 3.14.1
js-yaml: 3.15.2
resolve-from: 5.0.0
dev: true
@@ -4516,12 +4512,6 @@ packages:
'@types/express': 4.17.21
dev: true
/@types/node-schedule@1.3.2:
resolution: {integrity: sha512-Y0CqdAr+lCpArT8CJJjJq4U2v8Bb5e7ru2nV/NhDdaptCMCRdOL3Y7tAhen39HluQMaIKWvPbDuiFBUQpg7Srw==}
dependencies:
'@types/node': 17.0.45
dev: true
/@types/node@17.0.45:
resolution: {integrity: sha512-w+tIMs3rq2afQdsPJlODhoUEKzFP1ayaoyl1CcnwtIlsVe7K7bA1NGm4s3PraqTLlXnbIN84zuBlxBWo1u9BLw==}
@@ -4952,7 +4942,7 @@ packages:
'@umijs/bundler-utils': 4.4.4
'@umijs/mako': 0.11.1
chalk: 4.1.2
compression: 1.7.5
compression: 1.8.2
connect-history-api-fallback: 2.0.0
cors: 2.8.5
express: 4.21.2
@@ -5687,7 +5677,6 @@ packages:
debug: 4.4.3
transitivePeerDependencies:
- supports-color
dev: false
/agentkeepalive@4.5.0:
resolution: {integrity: sha512-5GG/5IbQQpC9FpkRGsSvZI5QYeSCzlJHdpBQntCsuTOxhKD8lqKhrleg2Yi7yvMIf82Ycmmqln9U8V9qwEiJew==}
@@ -6840,15 +6829,16 @@ packages:
serialize-javascript: 6.0.2
dev: true
/compression@1.7.5:
resolution: {integrity: sha512-bQJ0YRck5ak3LgtnpKkiabX5pNF7tMUh1BSy2ZBOTh0Dim0BUu6aPPwByIns6/A5Prh8PufSPerMDUklpzes2Q==}
/compression@1.8.2:
resolution: {integrity: sha512-o8vI5RE5A6EVVOd9o41jKp41aJom+QTEO/Bx8MYNjexMo/Bv2WOjUfZr+aL0WnYSgymUy6zeguqLTsIhV0gMvQ==}
engines: {node: '>= 0.8.0'}
dependencies:
bytes: 3.1.2
compressible: 2.0.18
debug: 2.6.9
destroy: 1.2.0
negotiator: 0.6.4
on-headers: 1.0.2
on-headers: 1.1.0
safe-buffer: 5.2.1
vary: 1.1.2
transitivePeerDependencies:
@@ -7046,13 +7036,6 @@ packages:
resolution: {integrity: sha512-VQ2MBenTq1fWZUH9DJNGti7kKv6EeAuYr3cLwxUWhIu1baTaXh4Ib5W2CqHVqib4/MqbYGJqiL3Zb8GJZr3l4g==}
dev: true
/cron-parser@4.9.0:
resolution: {integrity: sha512-p0SaNjrHOnQeR8/VnfGbmg9te2kfyYSQ7Sc/j/6DtPL3JQvKxmjO9TSjNFpujqV3vEYYBvNNvXSxzyksBWAx1Q==}
engines: {node: '>=12.0.0'}
dependencies:
luxon: 3.5.0
dev: false
/cron-parser@5.4.0:
resolution: {integrity: sha512-HxYB8vTvnQFx4dLsZpGRa0uHp6X3qIzS3ZJgJ9v6l/5TJMgeWQbLkR5yiJ5hOxGbc9+jCADDnydIe15ReLZnJA==}
engines: {node: '>=18'}
@@ -7750,6 +7733,7 @@ packages:
/dottie@2.0.6:
resolution: {integrity: sha512-iGCHkfUc5kFekGiqhe8B/mdaurD+lakO9txNnTvKtA6PISrw86LgqHvRzWYPyoE2Ph5aMIrCw9/uko6XHTKCwA==}
deprecated: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.
dev: false
/dprint-node@1.0.8:
@@ -8317,7 +8301,7 @@ packages:
is-glob: 4.0.3
is-path-inside: 3.0.3
js-sdsl: 4.4.2
js-yaml: 4.1.0
js-yaml: 4.3.2
json-stable-stringify-without-jsonify: 1.0.1
levn: 0.4.1
lodash.merge: 4.6.2
@@ -8528,7 +8512,7 @@ packages:
dependencies:
accepts: 1.3.8
array-flatten: 1.1.1
body-parser: 1.20.3
body-parser: 1.20.6
content-disposition: 0.5.4
content-type: 1.0.5
cookie: 0.7.1
@@ -8546,7 +8530,7 @@ packages:
on-finished: 2.4.1
parseurl: 1.3.3
path-to-regexp: 0.1.12
proxy-addr: 2.0.7
proxy-addr: 2.0.8
qs: 6.13.0
range-parser: 1.2.1
safe-buffer: 5.2.1
@@ -8630,7 +8614,7 @@ packages:
resolution: {integrity: sha512-CzbClwlXAuiRQAlUyfqPgvPoNKTckTPGfwZV4ZdAhVcP2lh9KUxJg2b5GkE7XbjKQ3YJnQ9z6D9ntLAlB+tP8g==}
engines: {node: '>=0.8.0'}
dependencies:
websocket-driver: 0.7.4
websocket-driver: 0.7.5
/fb-watchman@2.0.2:
resolution: {integrity: sha512-p5161BqbuCaSnB8jIbzQHOlpgsPmK5rJVDfDKO91Axs5NC1uu3HRQm6wt9cd9/+GtQQIO53JdGXXoyDpTAsgYA==}
@@ -9349,7 +9333,6 @@ packages:
debug: 4.4.3
transitivePeerDependencies:
- supports-color
dev: false
/human-signals@2.1.0:
resolution: {integrity: sha512-B4FFZ6q/T2jhhksgkbEW3HBvWIfDW85snkQgawt07S7J5QXTk6BkNV+0yAeZrM5QpMAdYlocGoljn0sJ/WQkFw==}
@@ -10445,10 +10428,6 @@ packages:
triple-beam: 1.4.1
dev: false
/long-timeout@0.1.1:
resolution: {integrity: sha512-BFRuQUqc7x2NWxfJBCyUrN8iYUYznzL9JROmRz1gZ6KlOIgmoD+njPVbb+VNn2nGMKggMsK79iUNErillsrx7w==}
dev: false
/long@5.2.3:
resolution: {integrity: sha512-lcHwpNoggQTObv5apGNCTdJrO69eHOZMi4BNC+rTLER8iHAqGrUVeLh/irVIM7zTw2bOXA8T6uNPeujwOLg/2Q==}
dev: false
@@ -10482,11 +10461,6 @@ packages:
yallist: 4.0.0
dev: true
/luxon@3.5.0:
resolution: {integrity: sha512-rh+Zjr6DNfUYR3bPwJEnuwDdqMbxZW7LOQfUN4B54+Cl+0o5zaU9RJ6bcidfDtC1cWCZXQ+nvX8bf6bAji37QQ==}
engines: {node: '>=12'}
dev: false
/luxon@3.7.2:
resolution: {integrity: sha512-vtEhXh/gNjI9Yg1u4jX/0YVPMvxzHuGgCm6tC5kZyb08yjGWGnqAjGJvcXbqQR2P3MyMEFnRbpcdFS6PBcLqew==}
engines: {node: '>=12'}
@@ -10611,7 +10585,7 @@ packages:
engines: {node: '>=8.6'}
dependencies:
braces: 3.0.3
picomatch: 2.3.1
picomatch: 2.3.2
/miller-rabin@4.0.1:
resolution: {integrity: sha512-115fLhvZVqWwHPbClyntxEVfVDfl9DLLTuJvq3g2O/Oxi8AiNouAHvDSzHS0viUJc+V5vm3eq91Xwqn9dp4jRA==}
@@ -10678,19 +10652,19 @@ packages:
/minimatch@3.1.2:
resolution: {integrity: sha512-J7p63hRiAjw1NDEww1W7i37+ByIrOWO5XQQAzZ3VOcL0PNybwpfmV/N05zFAzwQ9USyEcX6t3UO+K5aqBQOIHw==}
dependencies:
brace-expansion: 1.1.12
brace-expansion: 1.1.21
dev: true
/minimatch@3.1.5:
resolution: {integrity: sha512-VgjWUsnnT6n+NUk6eZq77zeFdpW2LWDzP6zFGrCbHXiYNul5Dzqk2HHQ5uFH2DNW5Xbp8+jVzaeNt94ssEEl4w==}
dependencies:
brace-expansion: 1.1.12
brace-expansion: 1.1.21
/minimatch@9.0.9:
resolution: {integrity: sha512-OBwBN9AL4dqmETlpS2zasx+vTeWclWzkblfZk7KTA5j3jeOONz/tRCnZomUyvNg83wL5Zv9Ss6HMJXAgL8R2Yg==}
engines: {node: '>=16 || 14 >=14.17'}
dependencies:
brace-expansion: 2.0.2
brace-expansion: 2.1.7
dev: true
/minimist-options@4.1.0:
@@ -10779,8 +10753,8 @@ packages:
/ms@2.1.3:
resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==}
/multer@2.1.1:
resolution: {integrity: sha512-mo+QTzKlx8R7E5ylSXxWzGoXoZbOsRMpyitcht8By2KHvMbf3tjwosZ/Mu/XYU6UuJ3VZnODIrak5ZrPiPyB6A==}
/multer@2.3.0:
resolution: {integrity: sha512-cjNbm3sttszgZeGfJR124D+jFEfkXCVAsoPBmFn9X7UxmDSFHWqE2CoEj0vrmSpuAFnqWR1Szcm9QTsiHr60Xw==}
engines: {node: '>= 10.16.0'}
dependencies:
append-field: 1.0.0
@@ -10849,6 +10823,11 @@ packages:
resolution: {integrity: sha512-73sE9+3UaLYYFmDsFZnqCInzPyh3MqIwZO9cw58yIqAZhONrrabrYyYe3TuIqtIiOuTXVhsGau8hcrhhwSsDIQ==}
dev: false
/node-cron@4.6.0:
resolution: {integrity: sha512-Si/bzYiKRHOB8/a99T2+SDGN582ONDMSTlJr5oCkT6GtnqPjZ2s10eoQRYkW9ZHwjVxONL+W8Fb+qR0AHMQsdg==}
engines: {node: '>=20'}
dev: false
/node-domexception@1.0.0:
resolution: {integrity: sha512-/jKZoMpw0F8GRwl4/eLROPA3cfcXtLApP0QzLmUT/HuPCZWyB7IY9ZrMeKw2O/nFIqPQB3PVM9aYm0F312AXDQ==}
engines: {node: '>=10.5.0'}
@@ -10889,7 +10868,7 @@ packages:
semver: 7.7.4
tar: 7.5.20
tinyglobby: 0.2.17
undici: 6.27.0
undici: 6.29.0
which: 6.0.1
dev: false
optional: true
@@ -10963,15 +10942,6 @@ packages:
resolution: {integrity: sha512-d9VeXT4SJ7ZeOqGX6R5EM022wpL+eWPooLI+5UpWn2jCT1aosUQEhQP214x33Wkwx3JQMvIm+tIoVOdodFS40g==}
dev: true
/node-schedule@2.1.1:
resolution: {integrity: sha512-OXdegQq03OmXEjt2hZP33W2YPs/E5BcFQks46+G2gAxs4gHOIVD1u7EqlYLYSKsaIpyKCK9Gbk0ta1/gjRSMRQ==}
engines: {node: '>=6'}
dependencies:
cron-parser: 4.9.0
long-timeout: 0.1.1
sorted-array-functions: 1.3.0
dev: false
/nodemailer@8.0.1:
resolution: {integrity: sha512-5kcldIXmaEjZcHR6F28IKGSgpmZHaF1IXLWFTG+Xh3S+Cce4MiakLtWY+PlBU69fLbRa8HlaGIrC/QolUpHkhg==}
engines: {node: '>=6.0.0'}
@@ -11172,8 +11142,8 @@ packages:
dependencies:
ee-first: 1.1.1
/on-headers@1.0.2:
resolution: {integrity: sha512-pZAE+FJLoyITytdqK0U5s+FIpjN0JP3OzFi/u8Rx+EV5/W+JTWGXG8xFzevE7AjBfDqHv/8vL8qQsIhHnqRkrA==}
/on-headers@1.1.0:
resolution: {integrity: sha512-737ZY3yNnXy37FHkQxPzt4UZ2UWPWiCZWLvFZ4fu5cueciegX0zGPnrlY6bwRg4FdQOe9YU8MkmJwGhoMybl8A==}
engines: {node: '>= 0.8'}
/once@1.4.0:
@@ -12209,8 +12179,8 @@ packages:
long: 5.2.3
dev: false
/proxy-addr@2.0.7:
resolution: {integrity: sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==}
/proxy-addr@2.0.8:
resolution: {integrity: sha512-5nnx0yGyVUcY6t9RnWcARWtwT9F1D8O9rt08htPvnd49W1IgZtmLkhu9WfMzQj1cFxjHIO6connUNVW5k7AVyQ==}
engines: {node: '>= 0.10'}
dependencies:
forwarded: 0.2.0
@@ -13942,8 +13912,8 @@ packages:
engines: {node: '>= 10.0.0'}
dev: false
/sequelize@6.37.5(@whyour/sqlite3@1.1.2):
resolution: {integrity: sha512-10WA4poUb3XWnUROThqL2Apq9C2NhyV1xHPMZuybNMCucDsbbFuKg51jhmyvvAUyUqCiimwTZamc3AHhMoBr2Q==}
/sequelize@6.37.8(@whyour/sqlite3@1.1.2):
resolution: {integrity: sha512-HJ0IQFqcTsTiqbEgiuioYFMSD00TP6Cz7zoTti+zVVBwVe9fEhev9cH6WnM3XU31+ABS356durAb99ZuOthnKw==}
engines: {node: '>=10.0.0'}
peerDependencies:
ibm_db: '*'
@@ -13977,7 +13947,7 @@ packages:
dependencies:
'@types/debug': 4.1.12
'@types/validator': 13.12.2
debug: 4.3.7(supports-color@5.5.0)
debug: 4.4.3
dottie: 2.0.6
inflection: 1.13.4
lodash: 4.17.21
@@ -13985,7 +13955,7 @@ packages:
moment-timezone: 0.5.46
pg-connection-string: 2.7.0
retry-as-promised: 7.0.4
semver: 7.6.3
semver: 7.7.4
sequelize-pool: 7.1.0
sqlite3: /@whyour/sqlite3@1.1.2
toposort-class: 1.0.1
@@ -14155,7 +14125,7 @@ packages:
dependencies:
faye-websocket: 0.11.4
uuid: 8.3.2
websocket-driver: 0.7.4
websocket-driver: 0.7.5
dev: false
/sonic-boom@2.8.0:
@@ -14180,10 +14150,6 @@ packages:
sort-object-keys: 1.1.3
dev: true
/sorted-array-functions@1.3.0:
resolution: {integrity: sha512-2sqgzeFlid6N4Z2fUQ1cvFmTOLRi/sEDzSQ0OKYchqgoPmQBVyM3959qYx3fpS6Esef80KjmpgPeEr028dP3OA==}
dev: false
/source-map-js@1.2.1:
resolution: {integrity: sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==}
engines: {node: '>=0.10.0'}
@@ -15564,8 +15530,8 @@ packages:
javascript-stringify: 2.1.0
dev: true
/websocket-driver@0.7.4:
resolution: {integrity: sha512-b17KeDIQVjvb0ssuSDF2cYXSg2iztliJ4B9WdsuB6J952qCPKmnVq4DyW5motImXHDC1cBT/1UezrJVsKw5zjg==}
/websocket-driver@0.7.5:
resolution: {integrity: sha512-ZL2+3c7kMBdIRCMz6l8jQMHyGVxj+UL+xVk74Ombiciboca8rHa15L86B19E5oh1pL9Ii/uj54gtsIrZGMo6zA==}
engines: {node: '>=0.8.0'}
dependencies:
http-parser-js: 0.5.8
@@ -15826,6 +15792,342 @@ packages:
resolution: {integrity: sha512-XBx9AXhXktjUqnepgTiE5flcKIYWi/rme0Eaj+5Y0lftuGBq+jyRu/md4WnuxqgP1ubdpNCsYEYPxrzVHD8d6g==}
dev: true
/brace-expansion@2.1.7:
resolution: {integrity: sha512-uZbew1NqdmPDTMJ8ah1y+b+9QEJrfkXFk3RcTQw3X0jW/xRUvFKsg1CfQdSYGdTbXZWExtU3J3ccxtnfw1Fi0g==}
dependencies:
balanced-match: 1.0.2
dev: true
/brace-expansion@1.1.21:
resolution: {integrity: sha512-9zeA+KLZNNzglF2TPKRQEDyx6Yby7daAkuy8MiPzpXPsYDWi/DRM8jmwUDxokQjYqBpv5DgPiwD4h4ZZSy1Ujw==}
dependencies:
balanced-match: 1.0.2
concat-map: 0.0.1
/axios@1.20.0:
resolution: {integrity: sha512-r8aOh8j9cGKpgQAqpzrUHnSIc6a59Y3Xf/cv8sy1DrHCkZHzQGEuoq1tARk6qSyDdtQGSDgpb9kFlruzPvrgwg==}
dependencies:
follow-redirects: 1.16.1(debug@4.4.3)
form-data: 4.0.6
https-proxy-agent: 5.0.1
proxy-from-env: 2.1.0
transitivePeerDependencies:
- debug
- supports-color
dev: true
/proxy-from-env@2.1.0:
resolution: {integrity: sha512-cJ+oHTW1VAEa8cJslgmUZrc+sjRKgAKl3Zyse6+PV38hZe/V6Z14TbCuXcan9F9ghlz4QrFr2c92TNF82UkYHA==}
engines: {node: '>=10'}
dev: true
/form-data@4.0.6:
resolution: {integrity: sha512-vKatAh4SlVfgbv+YtmhiRjhEMJsYpsG1Y2rMQtR+SVSbytsSD1YGzDIcrAJmdFec88u/+VoGmxnl+80gL1tRCQ==}
engines: {node: '>= 6'}
dependencies:
asynckit: 0.4.0
combined-stream: 1.0.8
es-set-tostringtag: 2.1.0
hasown: 2.0.4
mime-types: 2.1.35
dev: true
/hasown@2.0.4:
resolution: {integrity: sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==}
engines: {node: '>= 0.4'}
dependencies:
function-bind: 1.1.2
/es-set-tostringtag@2.1.0:
resolution: {integrity: sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA==}
engines: {node: '>= 0.4'}
dependencies:
es-errors: 1.3.0
get-intrinsic: 1.3.0
has-tostringtag: 1.0.2
hasown: 2.0.4
dev: true
/get-intrinsic@1.3.0:
resolution: {integrity: sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==}
engines: {node: '>= 0.4'}
dependencies:
call-bind-apply-helpers: 1.0.2
es-define-property: 1.0.1
es-errors: 1.3.0
es-object-atoms: 1.1.2
function-bind: 1.1.2
get-proto: 1.0.1
gopd: 1.2.0
has-symbols: 1.1.0
hasown: 2.0.4
math-intrinsics: 1.1.0
/math-intrinsics@1.1.0:
resolution: {integrity: sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==}
engines: {node: '>= 0.4'}
/gopd@1.2.0:
resolution: {integrity: sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==}
engines: {node: '>= 0.4'}
/get-proto@1.0.1:
resolution: {integrity: sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==}
engines: {node: '>= 0.4'}
dependencies:
dunder-proto: 1.0.1
es-object-atoms: 1.1.2
/es-object-atoms@1.1.2:
resolution: {integrity: sha512-HWcBoN6NileqtSydK2FqHbS/LoDd2pqrnQHLyJzBj4kOp/ky2MWMN694xOfkK8/SnUsW2DH7EfyVlydKCsm1Zw==}
engines: {node: '>= 0.4'}
dependencies:
es-errors: 1.3.0
/dunder-proto@1.0.1:
resolution: {integrity: sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==}
engines: {node: '>= 0.4'}
dependencies:
call-bind-apply-helpers: 1.0.2
es-errors: 1.3.0
gopd: 1.2.0
/call-bind-apply-helpers@1.0.2:
resolution: {integrity: sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==}
engines: {node: '>= 0.4'}
dependencies:
es-errors: 1.3.0
function-bind: 1.1.2
/es-define-property@1.0.1:
resolution: {integrity: sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==}
engines: {node: '>= 0.4'}
/follow-redirects@1.16.1(debug@4.4.3):
resolution: {integrity: sha512-FNvFGzoMLWmE6Yj9spb/zjd7yiNCHiAW9/Tg9CXrQ8wuu32HtlJOwWO11OJafl5FfY3DxTdQ0vj42zU1kvv5jg==}
engines: {node: '>=4.0'}
peerDependencies:
debug: '*'
peerDependenciesMeta:
debug:
optional: true
dependencies:
debug: 4.4.3
/undici@7.29.1:
resolution: {integrity: sha512-RYONW2MeafgYlkVOKYKkA/Ag7BmXqgIWCa8t1m0JcxrQg9pI9lEqRhAOruOBCbAohOa/gkCF+iPi9hrgvTzu6Q==}
engines: {node: '>=20.18.1'}
dev: false
/undici@6.29.0:
resolution: {integrity: sha512-R+RODBqp6i2pPflGdq+xIOUkl+RNfGgHwoinecKu/JCuf2uO06cOKoDbI2P7Dn6KcswdKwrczbU6IYJ6K8X+wg==}
engines: {node: '>=18.17'}
dev: false
optional: true
/nodemailer@10.0.9:
resolution: {integrity: sha512-BF0qcyplCwp+jMk6HCjFykBz/YhhZSsxrARhOldLwFWH+8kGjQsd2WIMIZhqEuyXRoGFi0ONbDeWDMDoL8MhLw==}
engines: {node: '>=20.0.0'}
dev: false
/multer@2.4.0:
resolution: {integrity: sha512-7dqa0ZcFfzbefdTuIkzOSMvZWC0J7FLqBOjJUZvDCXShIURWKxAyTT1wHhnE5q19c7jOJf43IYYKjBmZVZmvhg==}
engines: {node: '>= 10.16.0'}
dependencies:
append-field: 1.0.0
busboy: 1.6.0
type-is: 1.6.18
dev: false
/js-yaml@4.3.2:
resolution: {integrity: sha512-SFNOvSJ+Dgf/9An904Yx+CgSlIPCkIpao4qo51lpee25TIRejdH3rhR4EZMGoNx3/TP3O+wzWuiTFl4sqbltzA==}
hasBin: true
dependencies:
argparse: 2.0.1
/js-yaml@3.15.2:
resolution: {integrity: sha512-6EuL879VkRA+1Cz578mKMiKvjPNEuk6+r1JaFzoSWejZmtf7xWbIyw1e3KkxlkzTIt9Taw6JBhEppG7utc1P+w==}
hasBin: true
dependencies:
argparse: 1.0.10
esprima: 4.0.1
dev: true
/http-proxy-middleware@3.0.7:
resolution: {integrity: sha512-iwbQltVlx8bCrqePUM8C+hllHvdawVhQJaLrj1X7qllkvFQdXFsr16pW/mo9+JDVjN+QO2XUx9jd8SmoFkE5qw==}
engines: {node: ^14.18.0 || ^16.10.0 || >=18.0.0}
dependencies:
'@types/http-proxy': 1.17.17
debug: 4.4.3
http-proxy: 1.18.1(debug@4.4.3)
is-glob: 4.0.3
is-plain-object: 5.1.0
micromatch: 4.0.8
transitivePeerDependencies:
- supports-color
dev: false
/picomatch@2.3.2:
resolution: {integrity: sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA==}
engines: {node: '>=8.6'}
/is-plain-object@5.1.0:
resolution: {integrity: sha512-bUi/yjmtKYcRVUtWRGr0UA6xEFh2I6zWUwMrUXB3s7bmYCaZ8a+0ZsTRkrawh/mzlSD1Y0Ph8bp/U+TvBpWDNw==}
engines: {node: '>=0.10.0'}
/http-proxy@1.18.1(debug@4.4.3):
resolution: {integrity: sha512-7mz/721AbnJwIVbnaSv1Cz3Am0ZLT/UBwkC92VlxhXv/k/BBQfM2fXElQNC27BVGr0uwUpplYPQM9LnaBMR5NQ==}
engines: {node: '>=8.0.0'}
dependencies:
eventemitter3: 4.0.7
follow-redirects: 1.16.1(debug@4.4.3)
requires-port: 1.0.0
transitivePeerDependencies:
- debug
dev: false
/@types/http-proxy@1.17.17:
resolution: {integrity: sha512-ED6LB+Z1AVylNTu7hdzuBqOgMnvG/ld6wGCG8wFnAzKX5uyW2K3WD52v0gnLCTK/VLpXtKckgWuyScYK6cSPaw==}
dependencies:
'@types/node': 17.0.45
dev: false
/body-parser@1.20.6:
resolution: {integrity: sha512-p5tAzS57i5MV9fZFDj9LeIiTZEufbSe2eDozP+ElheSUq1m74CRq1jI4mYNDdVs9vQztXFLuk/Gd6BWTdwRJ5g==}
engines: {node: '>= 0.8', npm: 1.2.8000 || >= 1.4.16}
dependencies:
bytes: 3.1.2
content-type: 1.0.5
debug: 2.6.9
depd: 2.0.0
destroy: 1.2.0
http-errors: 2.0.1
iconv-lite: 0.4.24
on-finished: 2.4.1
qs: 6.15.3
raw-body: 2.5.3
type-is: 1.6.18
unpipe: 1.0.0
transitivePeerDependencies:
- supports-color
dev: false
/raw-body@2.5.3:
resolution: {integrity: sha512-s4VSOf6yN0rvbRZGxs8Om5CWj6seneMwK3oDb4lWDH0UPhWcxwOWw5+qk24bxq87szX1ydrwylIOp2uG1ojUpA==}
engines: {node: '>= 0.8'}
dependencies:
bytes: 3.1.2
http-errors: 2.0.1
iconv-lite: 0.4.24
unpipe: 1.0.0
/http-errors@2.0.1:
resolution: {integrity: sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==}
engines: {node: '>= 0.8'}
dependencies:
depd: 2.0.0
inherits: 2.0.4
setprototypeof: 1.2.0
statuses: 2.0.2
toidentifier: 1.0.1
/statuses@2.0.2:
resolution: {integrity: sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==}
engines: {node: '>= 0.8'}
/qs@6.15.3:
resolution: {integrity: sha512-O9gl3zCl5h5blw1KGUzQKhA5oUXSl8rwUIM5o0S3nCXMliSvy5Dzx7/DJcI+SwgICv+IneSZwhBh1oSyEHA71A==}
engines: {node: '>=0.6'}
dependencies:
es-define-property: 1.0.1
side-channel: 1.1.1
dev: false
/side-channel@1.1.1:
resolution: {integrity: sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ==}
engines: {node: '>= 0.4'}
dependencies:
es-errors: 1.3.0
object-inspect: 1.13.4
side-channel-list: 1.0.1
side-channel-map: 1.0.1
side-channel-weakmap: 1.0.2
/side-channel-weakmap@1.0.2:
resolution: {integrity: sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==}
engines: {node: '>= 0.4'}
dependencies:
call-bound: 1.0.4
es-errors: 1.3.0
get-intrinsic: 1.3.0
object-inspect: 1.13.4
side-channel-map: 1.0.1
/side-channel-map@1.0.1:
resolution: {integrity: sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==}
engines: {node: '>= 0.4'}
dependencies:
call-bound: 1.0.4
es-errors: 1.3.0
get-intrinsic: 1.3.0
object-inspect: 1.13.4
/object-inspect@1.13.4:
resolution: {integrity: sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==}
engines: {node: '>= 0.4'}
/call-bound@1.0.4:
resolution: {integrity: sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==}
engines: {node: '>= 0.4'}
dependencies:
call-bind-apply-helpers: 1.0.2
get-intrinsic: 1.3.0
/side-channel-list@1.0.1:
resolution: {integrity: sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==}
engines: {node: '>= 0.4'}
dependencies:
es-errors: 1.3.0
object-inspect: 1.13.4
settings:
autoInstallPeers: true
excludeLinksFromLockfile: false
+39
View File
@@ -0,0 +1,39 @@
// Reproducible storage comparison, without touching actual script history.
const fs = require('node:fs');
const path = require('node:path');
const zlib = require('node:zlib');
for (const file of ['sample/notify.js', 'shell/task.sh']) {
const source = fs.readFileSync(path.join(__dirname, '..', file), 'utf8');
const versions = Array.from({ length: 20 }, (_, i) => ({
content: source + '\n// edit ' + i,
}));
const raw = Buffer.from(JSON.stringify(versions));
const strategies = {
gzipSeparate: () =>
versions.reduce(
(sum, version) => sum + zlib.gzipSync(version.content).length,
0,
),
gzipBundle: () => zlib.gzipSync(raw).length,
brotliBundle: () =>
zlib.brotliCompressSync(raw, {
params: {
[zlib.constants.BROTLI_PARAM_QUALITY]: 4,
[zlib.constants.BROTLI_PARAM_LGWIN]: 22,
},
}).length,
};
for (const [strategy, compress] of Object.entries(strategies)) {
const start = performance.now();
const bytes = compress();
console.log(
JSON.stringify({
file,
strategy,
rawBytes: raw.length,
bytes,
ms: +(performance.now() - start).toFixed(2),
}),
);
}
}
+34
View File
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
set -euo pipefail
destination=$1
ref=$2
ref_type=$3
git check-ref-format "refs/heads/$ref"
script_dir=$(cd "$(dirname "$0")" && pwd)
# Publish only a complete artifact from the checked-out source commit.
node "$script_dir/../docker/verify-build.cjs"
if [[ "$ref_type" == 'tag' ]]; then
version=$(awk '$1 == "version:" { print $2; exit }' version.yaml)
if [[ "$ref" != "v$version" ]]; then
echo "::error::Release tag $ref does not match version.yaml ($version)"
exit 1
fi
elif [[ "$ref_type" != 'branch' ]]; then
echo "::error::Unsupported ref type: $ref_type"
exit 1
fi
workspace=$(mktemp -d)
trap 'rm -rf "$workspace"' EXIT
cp -a static/. "$workspace/"
git init -b "$ref" "$workspace"
git -C "$workspace" config user.name 'github-actions[bot]'
git -C "$workspace" config user.email 'github-actions[bot]@users.noreply.github.com'
git -C "$workspace" add -f .
git -C "$workspace" commit --allow-empty -m "Build static from $(git rev-parse HEAD)"
refs=("refs/heads/$ref:refs/heads/$ref")
if [[ "$ref_type" == 'tag' ]]; then
# Keep version branches for existing archive URLs and publish real Git tags.
git -C "$workspace" tag "$ref"
refs+=("refs/tags/$ref:refs/tags/$ref")
fi
git -C "$workspace" push --force --quiet "$destination" "${refs[@]}"
+20
View File
@@ -0,0 +1,20 @@
#!/usr/bin/env bash
# Static branches are orphan snapshots. Fetch complete commits for the branch
# that was built: a shallow/default-branch clone is rejected by mirror servers.
set -euo pipefail
destination=$1
branch=$2
ref_type=${3:-branch}
git check-ref-format "refs/heads/$branch"
workspace=$(mktemp -d)
trap 'rm -rf "$workspace"' EXIT
git init --bare "$workspace"
fetch_refs=("+refs/heads/$branch:refs/heads/$branch")
push_refs=("refs/heads/$branch:refs/heads/$branch")
if [[ "$ref_type" == 'tag' ]]; then
fetch_refs+=("+refs/tags/$branch:refs/tags/$branch")
push_refs+=("refs/tags/$branch:refs/tags/$branch")
fi
git -C "$workspace" fetch --no-tags "${QL_STATIC_SOURCE:-https://github.com/whyour/qinglong-static.git}" \
"${fetch_refs[@]}"
git -C "$workspace" push --force "$destination" "${push_refs[@]}"
+214
View File
@@ -0,0 +1,214 @@
#!/usr/bin/env bash
# Observe Pull mirrors; optionally notify them through a small GitHub push.
set -euo pipefail
if [[ $# -lt 3 || $# -gt 4 ]]; then
echo 'Usage: wait-for-mirror.sh SOURCE MIRROR REF [branch|tag]' >&2
exit 2
fi
source=$1
mirror=$2
ref=$3
ref_type=${4:-branch}
wait_seconds=${QL_MIRROR_WAIT_SECONDS:-600}
poll_interval=${QL_MIRROR_POLL_INTERVAL:-30}
query_timeout=${QL_MIRROR_QUERY_TIMEOUT:-20}
include_version_branch=${QL_MIRROR_INCLUDE_VERSION_BRANCH:-false}
notify_repo=${QL_MIRROR_NOTIFY_REPO:-}
retry_after=${QL_MIRROR_RETRY_AFTER_SECONDS:-310}
notify_on_start=${QL_MIRROR_NOTIFY_ON_START:-true}
timings=("$wait_seconds" "$poll_interval" "$query_timeout")
if [[ -n "$notify_repo" ]]; then timings+=("$retry_after"); fi
for value in "${timings[@]}"; do
if [[ ! "$value" =~ ^[1-9][0-9]{0,4}$ ]] || (( value > 86400 )); then
echo 'Mirror timing values must be integers between 1 and 86400 seconds' >&2
exit 2
fi
done
case "$ref_type" in
branch) refs=("refs/heads/$ref") ;;
tag)
refs=("refs/tags/$ref")
if [[ "$include_version_branch" == true ]]; then
refs+=("refs/heads/$ref")
fi
;;
*) echo 'Mirror ref type must be branch or tag' >&2; exit 2 ;;
esac
if [[ "$include_version_branch" != true && "$include_version_branch" != false ]]; then
echo 'QL_MIRROR_INCLUDE_VERSION_BRANCH must be true or false' >&2
exit 2
fi
if [[ "$notify_on_start" != true && "$notify_on_start" != false ]]; then
echo 'QL_MIRROR_NOTIFY_ON_START must be true or false' >&2
exit 2
fi
for requested_ref in "${refs[@]}"; do
git check-ref-format "$requested_ref"
done
command -v timeout >/dev/null || { echo 'GNU timeout is required' >&2; exit 2; }
if [[ -n "$notify_repo" ]]; then
if [[ ! "$notify_repo" =~ ^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$ ]]; then
echo 'QL_MIRROR_NOTIFY_REPO must be a GitHub owner/repository' >&2
exit 2
fi
command -v gh >/dev/null || { echo 'GitHub CLI is required for notifications' >&2; exit 2; }
fi
export GIT_TERMINAL_PROMPT=0
polling=false
notification_command() {
local budget=$((notification_deadline - SECONDS))
if (( budget <= 0 )); then return 1; fi
if (( budget > 10 )); then budget=10; fi
timeout --signal=KILL "${budget}s" "$@"
}
notify_static_mirror() {
local notification_deadline=$((SECONDS + 30))
if [[ "$polling" == true ]] && (( notification_deadline > wait_seconds )); then
notification_deadline=$wait_seconds
fi
local trigger_ref=refs/heads/gitee-sync-trigger
local attempt remote_refs source_sha parent tree commit delay remaining
local commit_args
if ! remote_refs=$(notification_command git ls-remote --refs "$source" "${refs[0]}"); then
echo '::warning::Could not read the published source; continuing mirror verification'
return 1
fi
source_sha=${remote_refs%%$'\t'*}
if [[ ! "$source_sha" =~ ^[0-9a-f]{40}$ ]]; then
echo "::error::Published source ref ${refs[0]} does not exist"
return 2
fi
for attempt in 1 2 3; do
if (( attempt > 1 )); then
delay=$((attempt * 2 - 2))
remaining=$((notification_deadline - SECONDS))
if (( remaining <= delay )); then break; fi
sleep "$delay"
fi
# Refresh the parent on every attempt. REST avoids the Git push endpoint's
# transient 500s, and non-forced updates preserve concurrent notifications.
if parent=$(notification_command gh api "repos/$notify_repo/git/ref/heads/gitee-sync-trigger" --jq '.object.sha' 2>&1); then
if ! tree=$(notification_command gh api "repos/$notify_repo/git/commits/$parent" --jq '.tree.sha'); then
echo "::warning::Could not read notification history (attempt $attempt)"
continue
fi
elif [[ "$parent" == *'(HTTP 404)'* ]]; then
parent=''
# A new notification branch starts with one tiny marker, never artifacts.
# Later commits reuse this tree and have no changed files in the webhook.
if ! tree=$(printf '%s' '{"tree":[{"path":".gitee-sync-trigger","mode":"100644","type":"blob","content":"Gitee Pull mirror notifications.\n"}]}' |
notification_command gh api --method POST "repos/$notify_repo/git/trees" --input - --jq '.sha'); then
echo "::warning::Could not create notification tree (attempt $attempt)"
continue
fi
else
echo "::warning::Could not read notification ref (attempt $attempt)"
continue
fi
commit_args=(-f "tree=$tree" -f "message=Trigger Gitee mirror for ${refs[0]} at $source_sha")
if [[ -n "$parent" ]]; then commit_args+=(-f "parents[]=$parent"); fi
if ! commit=$(notification_command gh api --method POST "repos/$notify_repo/git/commits" "${commit_args[@]}" --jq '.sha'); then
echo "::warning::Could not create notification commit (attempt $attempt)"
continue
fi
if [[ -n "$parent" ]]; then
if notification_command gh api --method PATCH "repos/$notify_repo/git/refs/heads/gitee-sync-trigger" -f "sha=$commit" -F force=false --silent; then
echo '::notice::Sent a small GitHub push event via REST to trigger the Gitee Pull mirror'
return 0
fi
elif notification_command gh api --method POST "repos/$notify_repo/git/refs" -f "ref=$trigger_ref" -f "sha=$commit" --silent; then
echo '::notice::Sent a small GitHub push event via REST to trigger the Gitee Pull mirror'
return 0
fi
echo "::warning::Mirror notification update failed (attempt $attempt)"
done
echo '::warning::Could not send the mirror notification; continuing mirror verification'
return 1
}
if [[ -n "$notify_repo" && "$notify_on_start" == true ]]; then
if notify_static_mirror; then :; elif [[ $? -eq 2 ]]; then exit 1; fi
fi
SECONDS=0
polling=true
attempt=0
retried=false
last_source='(not read)'
last_mirror='(not read)'
while (( SECONDS < wait_seconds )); do
attempt=$((attempt + 1))
outputs=()
ready=true
# Re-read the source after the mirror so a moving source cannot pass with
# the previous SHA. --refs compares annotated tag objects as well as branches.
urls=("$source" "$mirror" "$source")
labels=('source' 'mirror' 'source confirmation')
for index in 0 1 2; do
remaining=$((wait_seconds - SECONDS))
if (( remaining <= 0 )); then
ready=false
break
fi
budget=$query_timeout
if (( budget > remaining )); then budget=$remaining; fi
if ! output=$(timeout --signal=KILL "${budget}s" git ls-remote --refs \
"${urls[$index]}" "${refs[@]}" | LC_ALL=C sort); then
echo "::warning::Could not read ${labels[$index]} refs on attempt $attempt; retrying"
ready=false
break
fi
if (( index == 0 )); then last_source=$output; fi
if (( index == 1 )); then last_mirror=$output; fi
for requested_ref in "${refs[@]}"; do
found=false
while IFS=$'\t' read -r sha returned_ref; do
if [[ "$returned_ref" == "$requested_ref" && "$sha" =~ ^[0-9a-f]{40}([0-9a-f]{24})?$ ]]; then
found=true
fi
done <<< "$output"
if [[ "$found" == false ]]; then
echo "::notice::${labels[$index]} is missing $requested_ref; waiting"
ready=false
break
fi
done
if [[ "$ready" == false ]]; then break; fi
outputs+=("$output")
# Only confirm a source when the initial source and mirror reads match.
if (( index == 1 )) && [[ "${outputs[0]}" != "${outputs[1]}" ]]; then
ready=false
break
fi
done
if [[ "$ready" == true && "${outputs[0]}" == "${outputs[2]}" ]]; then
echo "::notice::Mirror matches the latest source refs after ${SECONDS}s"
printf '%s\n' "${outputs[2]}"
exit 0
fi
echo "Waiting for mirror (attempt $attempt, ${SECONDS}/${wait_seconds}s)"
remaining=$((wait_seconds - SECONDS))
if (( remaining <= 0 )); then break; fi
# Gitee requires five minutes between sync requests. Retry once after that
# interval without restarting the polling deadline.
if [[ -n "$notify_repo" && "$retried" == false ]] && (( SECONDS >= retry_after )); then
echo '::notice::Retrying the GitHub notification after the Gitee sync interval'
if notify_static_mirror; then :; elif [[ $? -eq 2 ]]; then exit 1; fi
retried=true
continue
fi
delay=$poll_interval
if (( delay > remaining )); then delay=$remaining; fi
if [[ -n "$notify_repo" && "$retried" == false ]] && (( SECONDS < retry_after && delay > retry_after - SECONDS )); then
delay=$((retry_after - SECONDS))
fi
sleep "$delay"
done
echo "::error::Mirror did not match the latest source refs within ${wait_seconds}s"
printf 'Last source refs:\n%s\nLast mirror refs:\n%s\n' "$last_source" "$last_mirror"
exit 1
+67 -70
View File
@@ -26,6 +26,28 @@ get_token() {
create_token
}
# Check the cached token before every authenticated API request. A rejected
# token is refreshed and retried once; other failures are never retried.
# Arguments are curl options, including the URL, method and optional body.
ql_api_request() {
local api=""
local attempt
get_token || return $?
for attempt in 1 2; do
api=$(curl -s --noproxy "*" \
-H "Authorization: Bearer ${__ql_token__}" \
-H "Content-Type: application/json;charset=UTF-8" \
"$@" \
--compressed)
if [[ "$attempt" == 2 || "$api" == '{"code":200}' ]] ||
[[ "$(jq -r '.code' <<< "$api" 2>/dev/null)" != 401 ]]; then
break
fi
create_token || return $?
done
printf '%s' "$api"
}
# Read the status response once; preserve multiline error messages and the
# legacy code/message variables used by the shell API callers.
ql_parse_status_response() {
@@ -63,13 +85,14 @@ add_cron_api() {
sub_id="null"
fi
local api=$(
curl -s --noproxy "*" "http://localhost:${ql_port}/open/crons?t=$currentTimeStamp" \
-H "Authorization: Bearer ${__ql_token__}" \
-H "Content-Type: application/json;charset=UTF-8" \
--data-raw "{\"name\":\"${name//\"/\\\"}\",\"command\":\"${command//\"/\\\"}\",\"schedule\":\"$schedule\",\"sub_id\":$sub_id}" \
--compressed
)
local payload
payload=$(jq -cn --arg name "$name" --arg command "$command" \
--arg schedule "$schedule" --argjson sub_id "$sub_id" \
'{name: $name, command: $command, schedule: $schedule, sub_id: $sub_id}') || return $?
local api=$(ql_api_request \
"http://localhost:${ql_port}/open/crons?t=$currentTimeStamp" \
--data-raw "$payload" \
-X POST)
code=$(echo "$api" | jq -r .code)
message=$(echo "$api" | jq -r .message)
if [[ $code == 200 ]]; then
@@ -93,14 +116,14 @@ update_cron_api() {
local id="$4"
fi
local api=$(
curl -s --noproxy "*" "http://localhost:${ql_port}/open/crons?t=$currentTimeStamp" \
-X 'PUT' \
-H "Authorization: Bearer ${__ql_token__}" \
-H "Content-Type: application/json;charset=UTF-8" \
--data-raw "{\"name\":\"${name//\"/\\\"}\",\"command\":\"${command//\"/\\\"}\",\"schedule\":\"$schedule\",\"id\":\"$id\"}" \
--compressed
)
local payload
payload=$(jq -cn --arg name "$name" --arg command "$command" \
--arg schedule "$schedule" --arg id "$id" \
'{name: $name, command: $command, schedule: $schedule, id: $id}') || return $?
local api=$(ql_api_request \
"http://localhost:${ql_port}/open/crons?t=$currentTimeStamp" \
-X PUT \
--data-raw "$payload")
code=$(echo "$api" | jq -r .code)
message=$(echo "$api" | jq -r .message)
if [[ $code == 200 ]]; then
@@ -120,14 +143,13 @@ update_cron_command_api() {
local id="$2"
fi
local api=$(
curl -s --noproxy "*" "http://localhost:${ql_port}/open/crons?t=$currentTimeStamp" \
-X 'PUT' \
-H "Authorization: Bearer ${__ql_token__}" \
-H "Content-Type: application/json;charset=UTF-8" \
--data-raw "{\"command\":\"${command//\"/\\\"}\",\"id\":\"$id\"}" \
--compressed
)
local payload
payload=$(jq -cn --arg command "$command" --arg id "$id" \
'{command: $command, id: $id}') || return $?
local api=$(ql_api_request \
"http://localhost:${ql_port}/open/crons?t=$currentTimeStamp" \
-X PUT \
--data-raw "$payload")
code=$(echo "$api" | jq -r .code)
message=$(echo "$api" | jq -r .message)
if [[ $code == 200 ]]; then
@@ -140,14 +162,10 @@ update_cron_command_api() {
del_cron_api() {
local ids="$1"
local currentTimeStamp=$(date +%s)
local api=$(
curl -s --noproxy "*" "http://localhost:${ql_port}/open/crons?t=$currentTimeStamp" \
-X 'DELETE' \
-H "Authorization: Bearer ${__ql_token__}" \
-H "Content-Type: application/json;charset=UTF-8" \
--data-raw "[$ids]" \
--compressed
)
local api=$(ql_api_request \
"http://localhost:${ql_port}/open/crons?t=$currentTimeStamp" \
-X DELETE \
--data-raw "[$ids]")
code=$(echo "$api" | jq -r .code)
message=$(echo "$api" | jq -r .message)
if [[ $code == 200 ]]; then
@@ -171,14 +189,10 @@ update_cron() {
dataRaw="${dataRaw},\"exit_code\":$exitCode"
fi
dataRaw="${dataRaw}}"
local api=$(
curl -s --noproxy "*" "http://localhost:${ql_port}/open/crons/status?t=$currentTimeStamp" \
-X 'PUT' \
-H "Authorization: Bearer ${__ql_token__}" \
-H "Content-Type: application/json;charset=UTF-8" \
--data-raw "$dataRaw" \
--compressed
)
local api=$(ql_api_request \
"http://localhost:${ql_port}/open/crons/status?t=$currentTimeStamp" \
-X PUT \
--data-raw "$dataRaw")
ql_parse_status_response "$api" || true
if [[ $code != 200 ]]; then
if [[ ! $message ]]; then
@@ -192,14 +206,10 @@ notify_api() {
local title="$1"
local content="$2"
local currentTimeStamp=$(date +%s)
local api=$(
curl -s --noproxy "*" "http://localhost:${ql_port}/open/system/notify?t=$currentTimeStamp" \
-X 'PUT' \
-H "Authorization: Bearer ${__ql_token__}" \
-H "Content-Type: application/json;charset=UTF-8" \
--data-raw "{\"title\":\"${title//\"/\\\"}\",\"content\":\"${content//\"/\\\"}\"}" \
--compressed
)
local api=$(ql_api_request \
"http://localhost:${ql_port}/open/system/notify?t=$currentTimeStamp" \
-X PUT \
--data-raw "{\"title\":\"${title//\"/\\\"}\",\"content\":\"${content//\"/\\\"}\"}")
code=$(echo "$api" | jq -r .code)
message=$(echo "$api" | jq -r .message)
if [[ $code == 200 ]]; then
@@ -212,12 +222,9 @@ notify_api() {
find_cron_api() {
local params="$1"
local currentTimeStamp=$(date +%s)
local api=$(
curl -s --noproxy "*" "http://localhost:${ql_port}/open/crons/detail?$params&t=$currentTimeStamp" \
-H "Authorization: Bearer ${__ql_token__}" \
-H "Content-Type: application/json;charset=UTF-8" \
--compressed
)
local api=$(ql_api_request \
"http://localhost:${ql_port}/open/crons/detail?$params&t=$currentTimeStamp" \
-X GET)
data=$(echo "$api" | jq -r .data)
if [[ $data == 'null' ]]; then
echo -e ""
@@ -231,14 +238,10 @@ update_auth_config() {
local body="$1"
local tip="$2"
local currentTimeStamp=$(date +%s)
local api=$(
curl -s --noproxy "*" "http://localhost:${ql_port}/open/system/auth/reset?t=$currentTimeStamp" \
-X 'PUT' \
-H "Authorization: Bearer ${__ql_token__}" \
-H "Content-Type: application/json;charset=UTF-8" \
--data-raw "{$body}" \
--compressed
)
local api=$(ql_api_request \
"http://localhost:${ql_port}/open/system/auth/reset?t=$currentTimeStamp" \
-X PUT \
--data-raw "{$body}")
code=$(echo "$api" | jq -r .code)
message=$(echo "$api" | jq -r .message)
if [[ $code == 200 ]]; then
@@ -254,14 +257,10 @@ record_cron_stat() {
local elapsed="${3:-0}"
[[ $ref_id ]] && [[ $ref_id -gt 0 ]] 2>/dev/null || return
local api=$(
curl -s --noproxy "*" "http://localhost:${ql_port:-5700}/open/dashboard/record" \
local api=$(ql_api_request \
"http://localhost:${ql_port:-5700}/open/dashboard/record" \
-X POST \
-H "Authorization: Bearer ${__ql_token__}" \
-H "Content-Type: application/json;charset=UTF-8" \
--data-raw "{\"ref_id\":$ref_id,\"code\":$exit_code,\"elapsed\":$elapsed}" \
--compressed
)
--data-raw "{\"ref_id\":$ref_id,\"code\":$exit_code,\"elapsed\":$elapsed}")
ql_parse_status_response "$api" || true
if [[ $code != 200 ]]; then
if [[ ! $message ]]; then
@@ -270,5 +269,3 @@ record_cron_stat() {
echo -e "${message}"
fi
}
get_token
+10 -2
View File
@@ -65,7 +65,9 @@ ql_read_node_path_cache() {
# the caller's descriptors. A crashed refresher releases the kernel lock.
ql_refresh_node_global_path() (
local cache="$1" key="$2" now result previous_umask
local lock="${cache}.lock"
# Keep one persistent lock per owner. Removing per-key lock files could
# split concurrent waiters across different inodes during cache cleanup.
local lock="${dir_tmp}/pnpm-root-${EUID}.lock"
previous_umask=$(umask)
umask 077
if type -P flock &>/dev/null && mkdir -p -- "$dir_tmp" 2>/dev/null; then
@@ -93,6 +95,10 @@ ql_refresh_node_global_path() (
# Private lock creation must not change pnpm's inherited creation mask.
umask "$previous_umask"
# Warm lookups never scan the directory. Retire old environment records on
# refresh, while preserving the shared lock and unrelated cache files.
find "$dir_tmp" -maxdepth 1 -type f -user "$EUID" \
-name "pnpm-root-${EUID}-*.cache" -mmin +60 -delete 2>/dev/null || true
now=${EPOCHSECONDS:-$(date +%s)}
result=$(pnpm root -g 9>&- 2>/dev/null) || return $?
# Never cache failed, empty, multiline or non-absolute answers.
@@ -119,8 +125,10 @@ ql_get_node_global_path() {
fi
local key cache
cache="${dir_tmp}/pnpm-root-${EUID}.cache"
key=$(ql_node_path_cache_key) || { pnpm root -g 2>/dev/null; return $?; }
# Node and crond may have different working directories/configuration.
# Preserve both discoveries instead of evicting each other on every run.
cache="${dir_tmp}/pnpm-root-${EUID}-${key// /-}.cache"
if ql_read_node_path_cache "$cache" "$key"; then
return 0
fi
+7 -9
View File
@@ -84,7 +84,9 @@ env_str_to_array() {
}
clear_non_sh_env() {
if [[ $file_param != *.sh ]]; then
# Only automatic Node/Python preloads reconstruct the generated environment.
# Explicit interpreters and other commands must keep the shell-loaded values.
if [[ $isJsOrPythonFile == 'true' ]]; then
clear_env
fi
}
@@ -162,7 +164,7 @@ run_normal() {
if [[ $isJsOrPythonFile == 'false' ]]; then
clear_non_sh_env
fi
$timeoutCmd $which_program $file_param "${script_params[@]}"
run_task_command $which_program $file_param "${script_params[@]}"
}
handle_env_split() {
@@ -201,7 +203,7 @@ run_concurrent() {
export "${env_param}=${array[$i - 1]}"
clear_non_sh_env
fi
eval envParam="${env_param}" numParam="${i}" $timeoutCmd $which_program $file_param "${script_params[@]}" &>$single_log_path &
eval envParam="${env_param}" numParam="${i}" run_task_command $which_program $file_param "${script_params[@]}" &>$single_log_path &
done
wait
@@ -241,7 +243,7 @@ run_designated() {
enter_script_workdir
envParam="${env_param}" numParam="${num_param}" $timeoutCmd $which_program $file_param "${script_params[@]}"
envParam="${env_param}" numParam="${num_param}" run_task_command $which_program $file_param "${script_params[@]}"
}
## 运行其他命令
@@ -297,7 +299,7 @@ run_else() {
fi
clear_non_sh_env
$timeoutCmd $which_program $file_param "$@"
run_task_command $which_program $file_param "$@"
}
check_file() {
@@ -334,10 +336,6 @@ check_nounset() {
main() {
if [[ $1 == *.js ]] || [[ $1 == *.mjs ]] || [[ $1 == *.py ]] || [[ $1 == *.pyc ]] || [[ $1 == *.sh ]] || [[ $1 == *.ts ]]; then
if [[ $1 == *.sh ]]; then
timeoutCmd=""
fi
case $# in
1)
run_normal "$1"
+6 -2
View File
@@ -159,9 +159,13 @@ try {
run();
const { sendNotify } = require('./__ql_notify__.js');
global.QLAPI = {
notify: sendNotify,
notify(...args) {
// Keep notification dependencies out of ordinary script startup.
// require caches the module after the first notification.
const { sendNotify } = require('./__ql_notify__.js');
return sendNotify.apply(this, args);
},
...client,
};
} catch (error) {
+4 -2
View File
@@ -129,10 +129,12 @@ try:
run()
from __ql_notify__ import send
class BaseApi(Client):
def notify(self, *args, **kwargs):
# Most scripts never send a notification. Python caches this
# module after the first call, including its channel dependencies.
from __ql_notify__ import send
return send(*args, **kwargs)
QLAPI = BaseApi()
+78
View File
@@ -0,0 +1,78 @@
#!/usr/bin/env bash
set -e
: "${QL_DIR:?Please set QL_DIR}"
: "${QL_DATA_DIR:?Please set QL_DATA_DIR}"
[[ "$QL_DIR" = /* && "$QL_DATA_DIR" = /*/data ]] || { echo "QL_DIR and QL_DATA_DIR must be absolute paths (data directory ends in /data)" >&2; exit 1; }
command="$1"
export PNPM_HOME="${PNPM_HOME:-${QL_DATA_DIR}/dep_cache/node}"
if [[ -n "${PYTHON_VENV_DIR:-}" ]]; then
[[ "$PYTHON_VENV_DIR" = /* ]] || PYTHON_VENV_DIR="${QL_DIR}/${PYTHON_VENV_DIR}"
export PYTHON_VENV_DIR
if [[ ! -x "$PYTHON_VENV_DIR/bin/python3" ]]; then
python3 -m venv "$PYTHON_VENV_DIR"
fi
[[ -x "$PYTHON_VENV_DIR/bin/python3" ]] || { echo "Invalid Python venv" >&2; exit 1; }
unset PYTHON_HOME PYTHONUSERBASE PYTHONPATH
export PATH="${PYTHON_VENV_DIR}/bin:${PATH}:${PNPM_HOME}"
export PIP_CACHE_DIR="${PYTHON_VENV_DIR}/pip"
else
export PYTHON_SHORT_VERSION=$(python3 -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')
export PYTHON_HOME="${PYTHON_HOME:-${QL_DATA_DIR}/dep_cache/python3}"
export PYTHONUSERBASE="$PYTHON_HOME"
export PATH="${PATH}:${PNPM_HOME}:${PYTHON_HOME}/bin"
export PIP_CACHE_DIR="${PYTHON_HOME}/pip"
export PYTHONPATH="${PYTHON_HOME}:${PYTHON_HOME}/lib/python${PYTHON_SHORT_VERSION}:${PYTHON_HOME}/lib/python${PYTHON_SHORT_VERSION}/site-packages"
fi
export NODE_PATH="${NODE_PATH:-/usr/local/bin:/usr/local/lib/node_modules:${PNPM_HOME}/global/5/node_modules}"
mkdir -p "$PIP_CACHE_DIR"
cd "${QL_DIR}"
if [[ ! -e .env ]]; then
# Preserve user configuration, including a file created concurrently.
(umask 077; set -C; cat .env.example > .env)
fi
chmod u+x "${QL_DIR}"/shell/*.sh
. "${QL_DIR}/shell/share.sh"
. "${QL_DIR}/shell/env.sh"
log_with_style() {
local level="$1"
local message="$2"
local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
printf "\n[%s] [%7s] %s\n" "${timestamp}" "${level}" "${message}"
}
log_with_style "INFO" "🚀 1. 检测配置文件..."
import_config "$@"
make_dir /etc/nginx/conf.d
make_dir /run/nginx
fix_config
pm2 l &>/dev/null
log_with_style "INFO" "🔄 2. 启动 nginx..."
nginx -s reload 2>/dev/null || nginx -c /etc/nginx/nginx.conf
log_with_style "INFO" "⚙️ 3. 启动 pm2 服务..."
reload_pm2
if [[ $command != "reload" ]]; then
if [[ $AutoStartBot == true ]]; then
log_with_style "INFO" "🤖 4. 启动 bot..."
nohup ql bot >$dir_log/bot.log 2>&1 &
fi
if [[ $EnableExtraShell == true ]]; then
log_with_style "INFO" "🛠️ 5. 执行自定义脚本..."
nohup ql extra >$dir_log/extra.log 2>&1 &
fi
pm2 startup
pm2 save
fi
log_with_style "SUCCESS" "🎉 启动成功!"
+8 -60
View File
@@ -77,65 +77,13 @@ if [[ $command != "reload" ]]; then
npm install -g pnpm@8.3.1 pm2 ts-node typescript@5
fi
export PYTHON_SHORT_VERSION=$(python3 -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')
export PNPM_HOME=${QL_DIR}/data/dep_cache/node
export PYTHON_HOME=${QL_DIR}/data/dep_cache/python3
export PYTHONUSERBASE=${QL_DIR}/data/dep_cache/python3
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:${PNPM_HOME}:${PYTHON_HOME}/bin
export NODE_PATH=/usr/local/bin:/usr/local/lib/node_modules:${PNPM_HOME}/global/5/node_modules
export PIP_CACHE_DIR=${PYTHON_HOME}/pip
export PYTHONPATH=${PYTHON_HOME}:${PYTHON_HOME}/lib/python${PYTHON_SHORT_VERSION}:${PYTHON_HOME}/lib/python${PYTHON_SHORT_VERSION}/site-packages
if [[ $command != "reload" ]]; then
pip3 install --prefix ${PYTHON_HOME} requests
fi
cd "${QL_DIR}"
if [[ ! -e .env ]]; then
# Preserve user configuration, including a file created concurrently.
(umask 077; set -C; cat .env.example > .env)
fi
chmod 777 ${QL_DIR}/shell/*.sh
. ${QL_DIR}/shell/share.sh
. ${QL_DIR}/shell/env.sh
log_with_style() {
local level="$1"
local message="$2"
local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
printf "\n[%s] [%7s] %s\n" "${timestamp}" "${level}" "${message}"
}
log_with_style "INFO" "🚀 1. 检测配置文件..."
import_config "$@"
make_dir /etc/nginx/conf.d
make_dir /run/nginx
fix_config
pm2 l &>/dev/null
log_with_style "INFO" "🔄 2. 启动 nginx..."
nginx -s reload 2>/dev/null || nginx -c /etc/nginx/nginx.conf
log_with_style "INFO" "⚙️ 3. 启动 pm2 服务..."
reload_pm2
if [[ $command != "reload" ]]; then
if [[ $AutoStartBot == true ]]; then
log_with_style "INFO" "🤖 4. 启动 bot..."
nohup ql bot >$dir_log/bot.log 2>&1 &
if [[ "$command" != "reload" ]]; then
if [[ -n "${PYTHON_VENV_DIR:-}" ]]; then
[[ "$PYTHON_VENV_DIR" = /* ]] || export PYTHON_VENV_DIR="${QL_DIR}/${PYTHON_VENV_DIR}"
[[ -x "$PYTHON_VENV_DIR/bin/python3" ]] || python3 -m venv "$PYTHON_VENV_DIR"
"$PYTHON_VENV_DIR/bin/python3" -m pip install requests
else
pip3 install --prefix "${PYTHON_HOME:-${QL_DATA_DIR}/dep_cache/python3}" requests
fi
if [[ $EnableExtraShell == true ]]; then
log_with_style "INFO" "🛠️ 5. 执行自定义脚本..."
nohup ql extra >$dir_log/extra.log 2>&1 &
fi
pm2 startup
pm2 save
fi
log_with_style "SUCCESS" "🎉 启动成功!"
exec /bin/bash "${QL_DIR}/shell/start-simplify.sh" "$@"
+63
View File
@@ -0,0 +1,63 @@
#!/usr/bin/env bash
# Fork from the current Bash state: an external `bash -c` would lose private
# before-hook variables, arrays and functions used by sourced Shell scripts.
run_task_command() {
local timeout_seconds
if [[ -z ${command_timeout_time:-} ]]; then
"$@"
return $?
fi
timeout_seconds=$(awk -v value="$command_timeout_time" 'BEGIN {
if (value !~ /^([0-9]+([.][0-9]*)?|[.][0-9]+)([eE][+-]?[0-9]+)?[smhd]?$/) exit 1
unit = substr(value, length(value), 1)
factor = unit == "d" ? 86400 : unit == "h" ? 3600 : unit == "m" ? 60 : 1
printf "%.9f", (value + 0) * factor
}') || { printf 'Invalid task timeout: %s\n' "$command_timeout_time" >&2; return 125; }
if [[ $timeout_seconds == '0.000000000' ]]; then
"$@"
return $?
fi
(
timeout_dir=$(mktemp -d "${TMPDIR:-/tmp}/ql-task-timeout.XXXXXXXX") || exit 125
# Both children get private process groups. The timer and the wrapper must
# survive terminating the task, and descendants must not outlive a timeout.
set -m
( set +m; "$@" ) <&0 &
timeout_worker=$!
(
set +m
: > "$timeout_dir/ready"
sleep "$timeout_seconds" || exit 125
: > "$timeout_dir/expired"
kill -TERM -- "-$timeout_worker" 2>/dev/null || :
sleep 0.5
kill -KILL -- "-$timeout_worker" 2>/dev/null || :
) &
timeout_watcher=$!
set +m
trap 'kill -KILL -- "-$timeout_watcher" 2>/dev/null || :
if kill -TERM -- "-$timeout_worker" 2>/dev/null; then sleep 0.1; fi
kill -KILL -- "-$timeout_worker" 2>/dev/null || :
wait "$timeout_worker" 2>/dev/null || :
wait "$timeout_watcher" 2>/dev/null || :
rm -rf -- "$timeout_dir"' EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
trap 'exit 129' HUP
trap 'exit 131' QUIT
# Do not clean up a timer whose process group has not initialized yet.
while [[ ! -f "$timeout_dir/ready" ]]; do
kill -0 "$timeout_watcher" 2>/dev/null || exit 125
sleep 0.001
done
wait "$timeout_worker"
timeout_code=$?
if [[ -f "$timeout_dir/expired" ]]; then
# Let the timer finish escalation even if the task leader exited first.
wait "$timeout_watcher" 2>/dev/null || :
exit 124
fi
exit "$timeout_code"
)
}
+8 -6
View File
@@ -3,6 +3,7 @@
dir_shell=$QL_DIR/shell
. $dir_shell/share.sh
. $dir_shell/api.sh
. $dir_shell/task-timeout.sh
trap 'single_hanle SIGINT' INT
trap 'single_hanle SIGTERM' TERM
@@ -18,6 +19,12 @@ single_hanle() {
exit 1
}
run_shell_script() {
# Preserve inherited variables/functions, but contain exit, exec and shell
# options so the task wrapper can run after hooks and report completion.
( . "$@" )
}
define_program() {
local file_param=$1
if [[ $file_param == *.js ]] || [[ $file_param == *.mjs ]]; then
@@ -25,7 +32,7 @@ define_program() {
elif [[ $file_param == *.py ]] || [[ $file_param == *.pyc ]]; then
which_program="python3"
elif [[ $file_param == *.sh ]]; then
which_program="."
which_program="run_shell_script"
elif [[ $file_param == *.ts ]]; then
which_program="ts-node-transpile-only"
else
@@ -106,11 +113,6 @@ format_params() {
mtime_format="%Y-%m-%d %H:%M:%S.%3N"
fi
timeoutCmd=""
if [[ $command_timeout_time ]]; then
if type timeout &>/dev/null; then
timeoutCmd="timeout --foreground -s 2 -k 10s $command_timeout_time "
fi
fi
# params=$(echo "$@" | sed -E 's/([^ ])&([^ ])/\1\\\&\2/g')
# 分割 task 内置参数和脚本参数
+48 -99
View File
@@ -99,7 +99,9 @@ add_cron() {
cron_name=$(grep "new Env" $file | awk -F "\(" '{print $2}' | awk -F "\)" '{print $1}' | sed 's:.*\('\''\|"\)\([^"'\'']*\)\('\''\|"\).*:\2:' | sed 's:"::g' | sed "s:'::g" | head -1)
[[ -z $cron_name ]] && cron_name=$(grep "name:" $file | awk -F ":" '{print $2}' | head -1 | xargs)
[[ -z $cron_name ]] && cron_name=$(basename "$file_name")
result=$(add_cron_api "${cron_line}:${cmd_task} ${file}:${cron_name}:${SUB_ID}")
cron_name=${cron_name//$'\r'/}
cron_line=${cron_line//$'\r'/}
result=$(add_cron_api "$cron_line" "${cmd_task} ${file}" "$cron_name" "$SUB_ID")
echo -e "$result"
if [[ $detail ]]; then
detail="${detail}${result}\n"
@@ -190,7 +192,9 @@ update_raw() {
[[ -z $cron_line ]] && cron_line=$(grep "cron:" $raw_file_name | awk -F ":" '{print $2}' | head -1 | xargs)
[[ -z $cron_line ]] && cron_line=$(grep "cron " $raw_file_name | awk -F "cron \"" '{print $2}' | awk -F "\" " '{print $1}' | head -1 | xargs)
[[ -z $cron_line ]] && cron_line="$default_cron"
result=$(add_cron_api "${cron_line}:${cmd_task} ${filename}:${cron_name}:${SUB_ID}")
cron_name=${cron_name//$'\r'/}
cron_line=${cron_line//$'\r'/}
result=$(add_cron_api "$cron_line" "${cmd_task} ${filename}" "$cron_name" "$SUB_ID")
echo -e "$result\n"
notify_api "新增任务通知" "\n$result"
# update_cron_api "$cron_line:$cmd_task $filename:$cron_name:$cron_id"
@@ -283,98 +287,44 @@ usage() {
}
reload_qinglong() {
echo -e "[reload_qinglong] deleting Triggered at $(date)" >>${dir_log}/reload.log
local reload_target="${1}"
if [[ "$reload_target" == 'system' ]]; then
QlPort="${ql_port:-${QlPort:-5700}}" QlBaseUrl="${ql_base_url:-${QlBaseUrl:-/}}" node "$dir_shell/upgrade.cjs" reload
return $?
fi
# Data import and a plain service restart retain their existing behavior.
echo -e "[reload_qinglong] deleting Triggered at $(date)" >>"${dir_log}/reload.log"
sleep 3
delete_pm2
echo -e "[reload_qinglong] deleted Triggered at $(date)" >>${dir_log}/reload.log
local reload_target="${1}"
local primary_branch="master"
if [[ "${QL_BRANCH}" == "develop" ]] || [[ "${QL_BRANCH}" == "debian" ]] || [[ "${QL_BRANCH}" == "debian-dev" ]]; then
primary_branch="${QL_BRANCH}"
fi
if [[ "$reload_target" == 'system' ]]; then
rm -rf ${dir_root}/back ${dir_root}/cli ${dir_root}/docker ${dir_root}/sample ${dir_root}/shell ${dir_root}/src
mv -f ${dir_tmp}/qinglong-${primary_branch}/* ${dir_root}/
rm -rf $dir_static/*
mv -f ${dir_tmp}/qinglong-static-${primary_branch}/* ${dir_static}/
cp -f $file_config_sample $dir_config/config.sample.sh
fi
if [[ "$reload_target" == 'data' ]]; then
rm -rf ${dir_data}/*
mv -f ${dir_tmp}/data/* ${dir_data}/
rm -rf "${dir_data:?}"/*
mv -f "$dir_tmp"/data/* "$dir_data"/ || return $?
fi
echo -e "[reload_qinglong] starting Triggered at $(date)" >>${dir_log}/reload.log
reload_pm2
echo -e "[reload_qinglong] started Triggered at $(date)\n" >>${dir_log}/reload.log
}
## 更新 qinglong
update_qinglong() {
rm -rf ${dir_tmp}/*
local mirror="gitee"
local downloadQLUrl="https://gitee.com/whyour/qinglong/repository/archive"
local downloadStaticUrl="https://gitee.com/whyour/qinglong-static/repository/archive"
local githubStatus=$(curl -s --noproxy "*" -m 2 -IL "https://google.com" | grep 200)
if [[ ! -z $githubStatus ]]; then
if curl --fail --silent --noproxy "*" --max-time 2 --head "https://github.com" >/dev/null; then
mirror="github"
downloadQLUrl="https://github.com/whyour/qinglong/archive/refs/heads"
downloadStaticUrl="https://github.com/whyour/qinglong-static/archive/refs/heads"
fi
t '使用 %s 源更新...\n' "${mirror}"
local primary_branch="master"
if [[ "${QL_BRANCH}" == "develop" ]] || [[ "${QL_BRANCH}" == "debian" ]] || [[ "${QL_BRANCH}" == "debian-dev" ]]; then
primary_branch="${QL_BRANCH}"
fi
wget -cqO "${dir_tmp}/ql.zip" "${downloadQLUrl}/${primary_branch}.zip"
exit_status=$?
if [[ $exit_status -eq 0 ]]; then
t '更新青龙源文件成功...\n'
unzip -oq ${dir_tmp}/ql.zip -d ${dir_tmp}
update_qinglong_static
else
t '更新青龙源文件失败,请检查网络...\n'
fi
t '使用 %s 源更新...\n' "$mirror"
QlPort="${ql_port:-${QlPort:-5700}}" QlBaseUrl="${ql_base_url:-${QlBaseUrl:-/}}" node "$dir_shell/upgrade.cjs" update "$mirror" "$needRestart"
}
update_qinglong_static() {
wget -cqO "${dir_tmp}/static.zip" "${downloadStaticUrl}/${primary_branch}.zip"
exit_status=$?
if [[ $exit_status -eq 0 ]]; then
t '更新青龙静态资源成功...\n'
unzip -oq ${dir_tmp}/static.zip -d ${dir_tmp}
check_update_dep
# Preserve the updater's status through both tee and the final timing output.
run_update_logged() (
set -o pipefail
if [[ "$real_time" == 'true' ]]; then
"$@"
elif [[ "$no_tee" == 'true' ]]; then
"$@" >>"$file_path" 2>&1
else
t '更新青龙静态资源失败,请检查网络...\n'
"$@" 2>&1 | tee -a "$file_path"
fi
}
check_update_dep() {
t '\n开始检测依赖...\n'
if [[ $(diff $dir_root/package.json ${dir_tmp}/qinglong-${primary_branch}/package.json) ]]; then
npm_install_2 "${dir_tmp}/qinglong-${primary_branch}"
fi
if [[ $exit_status -eq 0 ]]; then
t '\n依赖检测安装成功...\n'
t '更新包下载成功...\n'
if [[ "$needRestart" == 'true' ]]; then
reload_qinglong "system"
fi
else
t '\n依赖检测安装失败,请检查网络...\n'
fi
}
)
## 对比脚本
diff_scripts() {
@@ -429,23 +379,19 @@ gen_list_repo() {
cd ${repo_path}
local cmd="find ."
local index=0
if [[ $6 ]]; then
file_extensions="$6"
if [[ $file_extensions =~ "|" ]]; then
file_extensions=$(echo $file_extensions | sed 's/|/ /g')
local extension_string="${6:-$file_extensions}"
extension_string="${extension_string//|/ }"
extension_string="${extension_string//$'\n'/ }"
local -a extensions=() find_args=()
IFS=$' \t\n' read -r -a extensions <<< "$extension_string"
local extension
for extension in "${extensions[@]}"; do
if [[ ${#find_args[@]} -gt 0 ]]; then
find_args+=(-o)
fi
fi
for extension in $file_extensions; do
if [[ $index -eq 0 ]]; then
cmd="${cmd} -name \"*.${extension}\""
else
cmd="${cmd} -o -name \"*.${extension}\""
fi
let index+=1
find_args+=(-name "*.${extension}")
done
files=$(eval $cmd | sed 's/^..//')
files=$(find . "${find_args[@]}" | sed 's/^..//')
if [[ $path ]]; then
files=$(echo "$files" | egrep "$path")
fi
@@ -458,7 +404,7 @@ gen_list_repo() {
if [[ $dependence ]]; then
cd ${repo_path}
results=$(eval $cmd | sed 's/^..//' | egrep "$dependence")
results=$(find . "${find_args[@]}" | sed 's/^..//' | egrep "$dependence")
for _file in ${results}; do
file_path=$(dirname $_file)
make_dir "${dir_scripts}/${uniq_path}/${file_path}"
@@ -529,7 +475,7 @@ main() {
if [[ "$p1" == "log" ]]; then
show_service_logs "${@:2}"
return $?
return 0
fi
local log_dir="${p1}"
@@ -558,14 +504,17 @@ main() {
[[ $ID ]] && update_cron "\"$ID\"" "0" "$$" "$log_path" "$begin_timestamp"
local command_status=0
case $p1 in
update)
fix_config
local needRestart=${p2:-"true"}
eval update_qinglong $cmd
run_update_logged update_qinglong
command_status=$?
;;
reload)
eval reload_qinglong "$p2" $cmd
run_update_logged reload_qinglong "$p2"
command_status=$?
;;
extra)
eval run_extra_shell $cmd
@@ -624,9 +573,9 @@ main() {
if [[ "$p1" != "repo" ]] && [[ "$p1" != "raw" ]]; then
eval echo -e "\\\n\#\# 执行结束... $end_time 耗时 $diff_time 秒     " $cmd
fi
return "$command_status"
}
import_config "$@"
main "$@"
exit 0
exit $?
+8
View File
@@ -0,0 +1,8 @@
// The image updater and local CLI share one implementation and build contract.
const upgrade = require('../cli/src/internal/maintenance/upgradeArtifacts.cjs');
module.exports = upgrade;
if (require.main === module)
upgrade.main().catch((error) => {
console.error(`更新失败: ${error.message}`);
process.exitCode = 1;
});

Some files were not shown because too many files have changed in this diff Show More