Add data migration script and comprehensive migration guide

Co-authored-by: whyour <22700758+whyour@users.noreply.github.com>
2025-12-13 07:25:05 +08:00 · 2025-11-10 16:48:31 +00:00 · 2025-11-10 16:48:31 +00:00 · d42074f76a
commit d42074f76a
parent 5c798a0e93
3 changed files with 613 additions and 2 deletions
--- a/MIGRATION_GUIDE.md
+++ b/MIGRATION_GUIDE.md
@ -0,0 +1,252 @@
+# Multi-User Data Migration Guide
+
+This document explains how to migrate existing data to the multi-user system.
+
+## Overview
+
+When upgrading to the multi-user version of Qinglong, all existing data (cron tasks, environment variables, subscriptions, and dependencies) will be treated as "legacy data" that is accessible to all users. 
+
+To properly isolate data between users, you need to migrate existing data to specific user accounts.
+
+## Migration Script
+
+The `migrate-to-multiuser.js` script helps you assign existing legacy data to specific users.
+
+### Prerequisites
+
+- Node.js installed
+- Sequelize and dotenv packages (already included in the project)
+- At least one user created in the User Management interface
+
+### Usage
+
+#### 1. List All Users
+
+First, see all available users in your system:
+
+```bash
+node migrate-to-multiuser.js --list-users
+```
+
+Output example:
+```
+Users in the system:
+ID    Username    Role        Status
+--    --------    ----        ------
+1     admin       Admin       Enabled
+2     user1       User        Enabled
+3     user2       User        Enabled
+```
+
+#### 2. Preview Migration (Dry Run)
+
+Before making changes, preview what will be migrated:
+
+```bash
+node migrate-to-multiuser.js --userId=1 --dry-run
+```
+
+Or by username:
+
+```bash
+node migrate-to-multiuser.js --username=admin --dry-run
+```
+
+Output example:
+```
+Legacy Data Statistics:
+  Cron tasks: 15
+  Environment variables: 8
+  Subscriptions: 3
+  Dependencies: 5
+
+DRY RUN: No changes will be made.
+
+Would assign all legacy data to user ID 1
+```
+
+#### 3. Perform Migration
+
+Once you're ready, run the migration without `--dry-run`:
+
+**By User ID:**
+```bash
+node migrate-to-multiuser.js --userId=1
+```
+
+**By Username:**
+```bash
+node migrate-to-multiuser.js --username=admin
+```
+
+Output example:
+```
+Found user 'admin' with ID 1
+
+Legacy Data Statistics:
+  Cron tasks: 15
+  Environment variables: 8
+  Subscriptions: 3
+  Dependencies: 5
+
+Migrating data to user ID 1...
+✓ Migrated 15 cron tasks
+✓ Migrated 8 environment variables
+✓ Migrated 3 subscriptions
+✓ Migrated 5 dependencies
+
+✓ Migration completed successfully!
+```
+
+### Command Line Options
+
+| Option | Description |
+|--------|-------------|
+| `--userId=<id>` | Assign all legacy data to user with this ID |
+| `--username=<name>` | Assign all legacy data to user with this username |
+| `--list-users` | List all users in the system |
+| `--dry-run` | Show what would be changed without making changes |
+| `--help` | Show help message |
+
+## Migration Strategy
+
+### Scenario 1: Single User to Multi-User
+
+If you're upgrading from single-user to multi-user and want to keep all existing data under one admin account:
+
+1. Create an admin user in the User Management interface
+2. Run migration: `node migrate-to-multiuser.js --username=admin`
+
+### Scenario 2: Distribute Data to Multiple Users
+
+If you want to distribute existing data to different users:
+
+1. Create all necessary user accounts first
+2. Identify which data belongs to which user (you may need to do this manually by checking the database)
+3. For each user, manually update the `userId` field in the database tables (`Crontabs`, `Envs`, `Subscriptions`, `Dependences`)
+
+**SQL Example:**
+```sql
+-- Assign specific cron tasks to user ID 2
+UPDATE Crontabs 
+SET userId = 2 
+WHERE name LIKE '%user2%' AND userId IS NULL;
+
+-- Assign specific environment variables to user ID 2
+UPDATE Envs 
+SET userId = 2 
+WHERE name LIKE '%USER2%' AND userId IS NULL;
+```
+
+### Scenario 3: Keep as Shared Data
+
+If you want certain data to remain accessible to all users:
+
+- Simply don't run the migration script
+- Data with `userId = NULL` remains as "legacy data" accessible to everyone
+- This is useful for shared cron tasks or environment variables
+
+## Important Notes
+
+1. **Backup First**: Always backup your database before running migration scripts
+   ```bash
+   cp data/database.sqlite data/database.sqlite.backup
+   ```
+
+2. **Test in Dry Run**: Always use `--dry-run` first to see what will change
+
+3. **One-Time Operation**: The script only migrates data where `userId` is NULL
+   - Already migrated data won't be changed
+   - You can run it multiple times safely
+
+4. **Transaction Safety**: The migration uses database transactions
+   - If any error occurs, all changes are rolled back
+   - Your data remains safe
+
+5. **User Must Exist**: The target user must exist before migration
+   - Create users in the User Management interface first
+   - Use `--list-users` to verify users exist
+
+## Troubleshooting
+
+### Error: "User not found"
+
+**Problem:** The specified user doesn't exist in the database.
+
+**Solution:** 
+1. Run `node migrate-to-multiuser.js --list-users` to see available users
+2. Create the user in User Management interface if needed
+3. Use correct user ID or username
+
+### Error: "Database connection failed"
+
+**Problem:** Cannot connect to the database.
+
+**Solution:**
+1. Check that `data/database.sqlite` exists
+2. Verify database file permissions
+3. Check `QL_DATA_DIR` environment variable if using custom path
+
+### Error: "Migration failed"
+
+**Problem:** An error occurred during migration.
+
+**Solution:**
+1. Check the error message for details
+2. Verify database is not corrupted
+3. Restore from backup if needed
+4. Check database file permissions
+
+## Manual Migration
+
+If you prefer to migrate data manually using SQL:
+
+### Connect to Database
+```bash
+sqlite3 data/database.sqlite
+```
+
+### Check Legacy Data
+```sql
+-- Count legacy cron tasks
+SELECT COUNT(*) FROM Crontabs WHERE userId IS NULL;
+
+-- View legacy cron tasks
+SELECT id, name, command FROM Crontabs WHERE userId IS NULL;
+```
+
+### Migrate Data
+```sql
+-- Migrate all legacy data to user ID 1
+UPDATE Crontabs SET userId = 1 WHERE userId IS NULL;
+UPDATE Envs SET userId = 1 WHERE userId IS NULL;
+UPDATE Subscriptions SET userId = 1 WHERE userId IS NULL;
+UPDATE Dependences SET userId = 1 WHERE userId IS NULL;
+```
+
+### Verify Migration
+```sql
+-- Check if any legacy data remains
+SELECT 
+  (SELECT COUNT(*) FROM Crontabs WHERE userId IS NULL) as legacy_crons,
+  (SELECT COUNT(*) FROM Envs WHERE userId IS NULL) as legacy_envs,
+  (SELECT COUNT(*) FROM Subscriptions WHERE userId IS NULL) as legacy_subs,
+  (SELECT COUNT(*) FROM Dependences WHERE userId IS NULL) as legacy_deps;
+```
+
+## Support
+
+If you encounter issues with data migration:
+
+1. Check this guide for solutions
+2. Review the error messages carefully
+3. Ensure you have a recent backup
+4. Open an issue on GitHub with:
+   - Error messages
+   - Migration command used
+   - Database statistics (from dry run)
+
+## Related Documentation
+
+- [MULTI_USER_GUIDE.md](./MULTI_USER_GUIDE.md) - Complete multi-user feature guide
+- [README.md](./README.md) - Main project documentation
--- a/MULTI_USER_GUIDE.md
+++ b/MULTI_USER_GUIDE.md
@ -90,6 +90,48 @@ DELETE /api/user-management
 - Minimum password length is 6 characters
 - Strong passwords are recommended

+## 数据迁移 (Data Migration)
+
+### 迁移工具 (Migration Tool)
+
+项目提供了数据迁移脚本，可以将现有数据分配给特定用户。
+
+A migration script is provided to assign existing data to specific users.
+
+#### 使用方法 (Usage)
+
+1. **列出所有用户 (List all users)**
+```bash
+node migrate-to-multiuser.js --list-users
+```
+
+2. **预览迁移（不实际执行）(Dry run)**
+```bash
+node migrate-to-multiuser.js --userId=1 --dry-run
+```
+
+3. **将数据迁移到指定用户ID (Migrate to user ID)**
+```bash
+node migrate-to-multiuser.js --userId=1
+```
+
+4. **将数据迁移到指定用户名 (Migrate to username)**
+```bash
+node migrate-to-multiuser.js --username=admin
+```
+
+#### 注意事项 (Important Notes)
+
+- 迁移脚本只会处理 `userId` 为空的数据（遗留数据）
+- 已分配给用户的数据不会被修改
+- 建议先使用 `--dry-run` 预览变更
+- 迁移过程中如果出错会自动回滚
+
+- The script only migrates data where `userId` is NULL (legacy data)
+- Data already assigned to users will not be changed
+- It's recommended to use `--dry-run` first to preview changes
+- Changes are automatically rolled back if an error occurs
+
 ## 向后兼容 (Backward Compatibility)

 - 原有的单用户系统管理员账号继续有效
@ -103,10 +145,10 @@ DELETE /api/user-management

 1. **首次使用**：首次使用多用户功能时，建议先创建一个管理员账号作为备份
 2. **密码管理**：请妥善保管用户密码，忘记密码需要管理员重置
-3. **数据迁移**：如需将现有数据分配给特定用户，请联系管理员手动更新数据库
+3. **数据迁移**：使用提供的 `migrate-to-multiuser.js` 脚本将现有数据分配给特定用户
 4. **权限控制**：删除用户不会删除该用户的数据，数据会变为遗留数据

 1. **First Use**: When first using multi-user functionality, it's recommended to create an admin account as a backup
 2. **Password Management**: Please keep user passwords safe; forgotten passwords need admin reset
-3. **Data Migration**: To assign existing data to specific users, contact admin for manual database update
+3. **Data Migration**: Use the provided `migrate-to-multiuser.js` script to assign existing data to specific users
 4. **Permission Control**: Deleting a user doesn't delete their data; the data becomes legacy data
--- a/migrate-to-multiuser.js
+++ b/migrate-to-multiuser.js
@ -0,0 +1,317 @@
+#!/usr/bin/env node
+
+/**
+ * Multi-User Data Migration Script
+ * 
+ * This script migrates existing data (Cron, Env, Subscription, Dependence) 
+ * to be associated with specific users.
+ * 
+ * Usage:
+ *   node migrate-to-multiuser.js --userId=1
+ *   node migrate-to-multiuser.js --username=admin
+ *   node migrate-to-multiuser.js --list-users
+ * 
+ * Options:
+ *   --userId=<id>        Assign all legacy data to user with this ID
+ *   --username=<name>    Assign all legacy data to user with this username
+ *   --list-users         List all users in the system
+ *   --dry-run            Show what would be changed without making changes
+ *   --help               Show this help message
+ */
+
+const path = require('path');
+const fs = require('fs');
+const Sequelize = require('sequelize');
+
+// Load environment variables
+require('dotenv').config();
+
+// Configuration
+const config = {
+  dbPath: process.env.QL_DATA_DIR || path.join(__dirname, '../data'),
+  rootPath: __dirname,
+};
+
+// Initialize Sequelize
+const sequelize = new Sequelize({
+  dialect: 'sqlite',
+  storage: path.join(config.dbPath, 'database.sqlite'),
+  logging: false,
+});
+
+// Define models
+const UserModel = sequelize.define('User', {
+  username: Sequelize.STRING,
+  password: Sequelize.STRING,
+  role: Sequelize.NUMBER,
+  status: Sequelize.NUMBER,
+});
+
+const CrontabModel = sequelize.define('Crontab', {
+  name: Sequelize.STRING,
+  command: Sequelize.STRING,
+  schedule: Sequelize.STRING,
+  userId: Sequelize.NUMBER,
+});
+
+const EnvModel = sequelize.define('Env', {
+  name: Sequelize.STRING,
+  value: Sequelize.STRING,
+  userId: Sequelize.NUMBER,
+});
+
+const SubscriptionModel = sequelize.define('Subscription', {
+  name: Sequelize.STRING,
+  url: Sequelize.STRING,
+  userId: Sequelize.NUMBER,
+});
+
+const DependenceModel = sequelize.define('Dependence', {
+  name: Sequelize.STRING,
+  type: Sequelize.NUMBER,
+  userId: Sequelize.NUMBER,
+});
+
+// Parse command line arguments
+function parseArgs() {
+  const args = {
+    userId: null,
+    username: null,
+    listUsers: false,
+    dryRun: false,
+    help: false,
+  };
+
+  process.argv.slice(2).forEach(arg => {
+    if (arg.startsWith('--userId=')) {
+      args.userId = parseInt(arg.split('=')[1]);
+    } else if (arg.startsWith('--username=')) {
+      args.username = arg.split('=')[1];
+    } else if (arg === '--list-users') {
+      args.listUsers = true;
+    } else if (arg === '--dry-run') {
+      args.dryRun = true;
+    } else if (arg === '--help' || arg === '-h') {
+      args.help = true;
+    }
+  });
+
+  return args;
+}
+
+// Show help
+function showHelp() {
+  console.log(`
+Multi-User Data Migration Script
+
+This script migrates existing data (Cron, Env, Subscription, Dependence) 
+to be associated with specific users.
+
+Usage:
+  node migrate-to-multiuser.js --userId=1
+  node migrate-to-multiuser.js --username=admin
+  node migrate-to-multiuser.js --list-users
+
+Options:
+  --userId=<id>        Assign all legacy data to user with this ID
+  --username=<name>    Assign all legacy data to user with this username
+  --list-users         List all users in the system
+  --dry-run            Show what would be changed without making changes
+  --help               Show this help message
+
+Examples:
+  # List all users
+  node migrate-to-multiuser.js --list-users
+
+  # Migrate all data to user ID 1 (dry run)
+  node migrate-to-multiuser.js --userId=1 --dry-run
+
+  # Migrate all data to user 'admin'
+  node migrate-to-multiuser.js --username=admin
+
+Note: This script will only migrate data where userId is NULL or undefined.
+Data already assigned to users will not be changed.
+  `);
+}
+
+// List all users
+async function listUsers() {
+  const users = await UserModel.findAll();
+  
+  if (users.length === 0) {
+    console.log('\nNo users found in the database.');
+    console.log('Please create users first using the User Management interface.');
+    return;
+  }
+
+  console.log('\nUsers in the system:');
+  console.log('ID\tUsername\tRole\t\tStatus');
+  console.log('--\t--------\t----\t\t------');
+  
+  users.forEach(user => {
+    const role = user.role === 0 ? 'Admin' : 'User';
+    const status = user.status === 0 ? 'Enabled' : 'Disabled';
+    console.log(`${user.id}\t${user.username}\t\t${role}\t\t${status}`);
+  });
+  console.log('');
+}
+
+// Get statistics of legacy data
+async function getStatistics() {
+  const stats = {
+    crons: await CrontabModel.count({ where: { userId: null } }),
+    envs: await EnvModel.count({ where: { userId: null } }),
+    subscriptions: await SubscriptionModel.count({ where: { userId: null } }),
+    dependences: await DependenceModel.count({ where: { userId: null } }),
+  };
+  
+  return stats;
+}
+
+// Migrate data to a specific user
+async function migrateData(userId, dryRun = false) {
+  const stats = await getStatistics();
+  
+  console.log('\nLegacy Data Statistics:');
+  console.log(`  Cron tasks: ${stats.crons}`);
+  console.log(`  Environment variables: ${stats.envs}`);
+  console.log(`  Subscriptions: ${stats.subscriptions}`);
+  console.log(`  Dependencies: ${stats.dependences}`);
+  console.log('');
+
+  if (stats.crons + stats.envs + stats.subscriptions + stats.dependences === 0) {
+    console.log('No legacy data found. All data is already assigned to users.');
+    return;
+  }
+
+  if (dryRun) {
+    console.log('DRY RUN: No changes will be made.\n');
+    console.log(`Would assign all legacy data to user ID ${userId}`);
+    return;
+  }
+
+  console.log(`Migrating data to user ID ${userId}...`);
+
+  const transaction = await sequelize.transaction();
+  
+  try {
+    // Migrate crons
+    if (stats.crons > 0) {
+      await CrontabModel.update(
+        { userId },
+        { where: { userId: null }, transaction }
+      );
+      console.log(`✓ Migrated ${stats.crons} cron tasks`);
+    }
+
+    // Migrate envs
+    if (stats.envs > 0) {
+      await EnvModel.update(
+        { userId },
+        { where: { userId: null }, transaction }
+      );
+      console.log(`✓ Migrated ${stats.envs} environment variables`);
+    }
+
+    // Migrate subscriptions
+    if (stats.subscriptions > 0) {
+      await SubscriptionModel.update(
+        { userId },
+        { where: { userId: null }, transaction }
+      );
+      console.log(`✓ Migrated ${stats.subscriptions} subscriptions`);
+    }
+
+    // Migrate dependences
+    if (stats.dependences > 0) {
+      await DependenceModel.update(
+        { userId },
+        { where: { userId: null }, transaction }
+      );
+      console.log(`✓ Migrated ${stats.dependences} dependencies`);
+    }
+
+    await transaction.commit();
+    console.log('\n✓ Migration completed successfully!');
+  } catch (error) {
+    await transaction.rollback();
+    console.error('\n✗ Migration failed:', error.message);
+    throw error;
+  }
+}
+
+// Main function
+async function main() {
+  const args = parseArgs();
+
+  if (args.help) {
+    showHelp();
+    return;
+  }
+
+  try {
+    // Test database connection
+    await sequelize.authenticate();
+    console.log('Database connection established.');
+
+    if (args.listUsers) {
+      await listUsers();
+      return;
+    }
+
+    // Validate arguments
+    if (!args.userId && !args.username) {
+      console.error('\nError: You must specify either --userId or --username');
+      console.log('Use --help for usage information.');
+      process.exit(1);
+    }
+
+    // Get user ID
+    let userId = args.userId;
+    
+    if (args.username) {
+      const user = await UserModel.findOne({
+        where: { username: args.username }
+      });
+      
+      if (!user) {
+        console.error(`\nError: User '${args.username}' not found.`);
+        console.log('Use --list-users to see available users.');
+        process.exit(1);
+      }
+      
+      userId = user.id;
+      console.log(`Found user '${args.username}' with ID ${userId}`);
+    } else {
+      // Verify user exists
+      const user = await UserModel.findByPk(userId);
+      
+      if (!user) {
+        console.error(`\nError: User with ID ${userId} not found.`);
+        console.log('Use --list-users to see available users.');
+        process.exit(1);
+      }
+      
+      console.log(`Found user '${user.username}' with ID ${userId}`);
+    }
+
+    // Perform migration
+    await migrateData(userId, args.dryRun);
+
+  } catch (error) {
+    console.error('\nError:', error.message);
+    process.exit(1);
+  } finally {
+    await sequelize.close();
+  }
+}
+
+// Run the script
+if (require.main === module) {
+  main().catch(error => {
+    console.error('Fatal error:', error);
+    process.exit(1);
+  });
+}
+
+module.exports = { main, listUsers, migrateData, getStatistics };