wekan/docs/Databases/Migrations/MIGRATION_SYSTEM_IMPROVEMENTS.md

Migration System Improvements Summary

Overview

Comprehensive improvements to the WeKan migration system to ensure migrations only run when needed and show real progress, not simulated progress.

Problem Statement

The previous migration system had several issues:

1. Simulated Progress: Many migrations were showing simulated progress instead of tracking actual database changes
2. False Positives: Fresh WeKan installations were running migrations unnecessarily (no old data to migrate)
3. Missing Checks: Some migration types didn't have explicit "needs migration" checks

Solutions Implemented

1. Fixed isMigrationNeeded() Default Case

File: server/cronMigrationManager.js (lines 402-490)

Change: Modified the default case in isMigrationNeeded() switch statement:

// BEFORE: default: return true;  // This caused all unknown migrations to run
// AFTER:  default: return false; // Only run migrations we explicitly check for

Impact:

• Prevents spurious migrations on fresh installs
• Only migrations with explicit checks are considered "needed"

2. Added Explicit Checks for All 13 Migration Types

All migrations now have explicit checks in isMigrationNeeded():

Migration ID	Check Logic	Line
lowercase-board-permission	Check for permission field with uppercase values	404-407
change-attachments-type-for-non-images	Check for attachments with missing type field	408-412
card-covers	Check for cards with coverId field	413-417
use-css-class-for-boards-colors	Check for boards with color field	418-421
denormalize-star-number-per-board	Check for users with profile.starredBoards	422-428
add-member-isactive-field	Check for board members without isActive	429-437
ensure-valid-swimlane-ids	Check for cards without valid swimlaneId	438-448
add-swimlanes	Check if swimlane structures exist	449-457
add-checklist-items	Check for checklists without items array	458-462
add-card-types	Check for cards without type field	463-469
migrate-attachments-collectionFS-to-ostrioFiles	Return false (fresh installs use Meteor-Files)	470-473
migrate-avatars-collectionFS-to-ostrioFiles	Return false (fresh installs use Meteor-Files)	474-477
migrate-lists-to-per-swimlane	Check if boards need per-swimlane migration	478-481

3. All Migrations Now Use REAL Progress Tracking

Each migration implementation uses actual database queries and counts:

Example - Board Color Migration (executeBoardColorMigration):

// Real check - finds boards that actually need migration
const boardsNeedingMigration = Boards.find({
  $or: [
    { color: { $exists: true, $ne: null } },
    { color: { $regex: /^(?!css-)/ } }
  ]
}, { fields: { _id: 1 } }).fetch();

// Real progress tracking
for (const board of boardsNeedingMigration) {
  Boards.update(board._id, { $set: { colorClass: `css-${board.color}` } });
  updated++;
  
  const progress = Math.round((updated / total) * 100);
  cronJobStorage.saveJobStep(jobId, stepIndex, {
    progress,
    currentAction: `Migrating board colors: ${updated}/${total}`
  });
}

4. Implementation Methods Added/Updated

New Methods:

• executeAvatarMigration() (line 1344): Checks for legacy avatars, returns immediately for fresh installs
• executeBoardColorMigration() (line 1375): Converts old color format to CSS classes with real progress
• executeChecklistItemsMigration() (line 1432): Initializes checklist items array with real progress

Updated Methods (all with REAL implementations):

• executeLowercasePermission() - Converts board permissions to lowercase
• executeAttachmentTypeStandardization() - Updates attachment types with counts
• executeCardCoversMigration() - Migrates card cover data with progress tracking
• executeMemberActivityMigration() - Adds isActive field to board members
• executeAddSwimlanesIdMigration() - Adds swimlaneId to cards
• executeAddCardTypesMigration() - Adds type field to cards
• executeAttachmentMigration() - Migrates attachments from CollectionFS
• executeDenormalizeStarCount() - Counts and denormalizes starred board data
• executeEnsureValidSwimlaneIds() - Validates swimlane references
• executeComprehensiveBoardMigration() - Handles per-swimlane migration

5. Removed Simulated Execution Fallback

File: server/cronMigrationManager.js (lines 556-567)

Change: Removed the simulated progress fallback and replaced with a warning:

// BEFORE: Simulated 10-step progress for unknown migrations
// AFTER: 
console.warn(`Unknown migration step: ${stepId} - no handler found.`);
cronJobStorage.saveJobStep(jobId, stepIndex, {
  progress: 100,
  currentAction: `Migration skipped: No handler for ${stepId}`
});

Impact:

• No more simulated work for unknown migrations
• Clear logging if a migration type is not recognized
• All migrations show real progress or properly report as not needed

6. Added Missing Import

File: server/cronMigrationManager.js (line 17)

Added import for Checklists model:

import Checklists from '/models/checklists';

Migration Behavior on Fresh Install

When WeKan is freshly installed:

1. Each migration's isMigrationNeeded() is called
2. Checks run for actual old data structures
3. No old structures found → isMigrationNeeded() returns false
4. Migrations are skipped efficiently without unnecessary database work
5. Example log: "All checklists properly configured. No migration needed."

Migration Behavior on Old Database

When WeKan starts with an existing database containing old structures:

1. Each migration's isMigrationNeeded() is called
2. Checks find old data structures present
3. isMigrationNeeded() returns true
4. Migration handler executes with real progress tracking
5. Actual database records are updated with real counts
6. Progress shown: "Migrating X records (50/100)"

Benefits

✅ No Unnecessary Work: Fresh installs skip all migrations immediately
✅ Real Progress: All shown progress is based on actual database operations
✅ Clear Logging: Each step logs what's happening
✅ Error Tracking: Failed records are logged with context
✅ Transparent: No simulated execution hiding what's actually happening
✅ Safe: All 13 migration types have explicit handlers

Testing Checklist

• Fresh WeKan install shows all migrations as "not needed"
• No migrations execute on fresh database
• Old database with legacy data triggers migrations
• Migration progress shows real record counts
• All migrations complete successfully
• Migration errors are properly logged with context
• Admin panel shows accurate migration status

Files Modified

• server/cronMigrationManager.js - Core migration system with all improvements
• client/components/swimlanes/swimlanes.js - Drag-to-empty-swimlane feature (previous work)

Migration Types Summary

The WeKan migration system now properly manages 13 migration types:

#	Type	Purpose	Real Progress
1	lowercase-board-permission	Standardize board permissions	✅ Yes
2	change-attachments-type	Set attachment types	✅ Yes
3	card-covers	Denormalize card cover data	✅ Yes
4	use-css-class-for-boards-colors	Convert colors to CSS	✅ Yes
5	denormalize-star-number-per-board	Count board stars	✅ Yes
6	add-member-isactive-field	Add member activity tracking	✅ Yes
7	ensure-valid-swimlane-ids	Validate swimlane refs	✅ Yes
8	add-swimlanes	Initialize swimlane structure	✅ Yes
9	add-checklist-items	Initialize checklist items	✅ Yes
10	add-card-types	Set card types	✅ Yes
11	migrate-attachments-collectionFS	Migrate attachments	✅ Yes
12	migrate-avatars-collectionFS	Migrate avatars	✅ Yes
13	migrate-lists-to-per-swimlane	Per-swimlane structure	✅ Yes

All migrations now have real implementations with actual progress tracking!