Data Migration - Rails Engine

A mountable Rails engine for migrating data between Rails application environments with full audit trails, role-based access control, and dependency management.

• Role-Based Access Control - Admin, Operator, and Viewer roles
• Migration Plans - Reusable migration configurations with execution order
• Plan Configuration Import/Export - Share migration plans across environments as JSON
• Dynamic Filtering - ActiveRecord queries with runtime parameter substitution
• Dependency Management - Maintain referential integrity across related models
• Association Handling - Remap foreign keys and handle polymorphic associations
• Attachment Handling - Export/import Active Storage attachments with URL or raw data modes
• Background Processing - Sidekiq-based async exports and imports
• Audit Trail - Complete execution history and detailed record-level tracking
• Real-time Progress - ActionCable integration for live updates

gem "data_migration_for_rails", "~> 0.1", ">= 0.1.5"

bundle install
bundle exec rake data_migration_engine:install:migrations
bundle exec rake db:migrate

In config/routes.rb:

Rails.application.routes.draw do
  mount DataMigration::Engine, at: "/data_migration"
end

bundle exec rake data_migration:seed

This creates an admin user with independent authentication:

• Email: admin@datamigration.local
• Password: password

Change the password immediately after first login! Admin users can create additional users via the "Users" menu.

In config/application.rb:

config.active_job.queue_adapter = :sidekiq

Start services:

redis-server
bundle exec sidekiq

• Ruby 3.0+
• Rails 7.0+
• Redis
• Devise (for authentication)
• Pundit (for authorization)
• Active Storage (optional, for attachment handling)

Navigate to /data_migration/migration_plans and create a new plan.

Configure each step with:

• Model Name: Select from dropdown (shows fields and attachments)
• Sequence: Execution order
• Filter Query (optional): ActiveRecord query with optional placeholders

Examples:

where(active: true).limit(100)
where("created_at > ?", "{{cutoff_date}}")

• Column Overrides: Export association attributes
• Association ID Mappings: Remap foreign keys on import
• Dependee Attribute Mapping: Filter based on parent step's exported records
• Attachment Export Mode: Choose how to handle Active Storage attachments:
  • Ignore - Skip attachments
  • URL - Export attachment URLs (for cloud storage)
  • Raw Data - Export actual files in archive (for local storage)

Click Export, fill in any filter parameters, and download the .tar.gz archive.

On target environment, click Import, upload the archive, and select conflict resolution strategy.

Use {{placeholder}} syntax in filter queries. Values are collected at export time and validated before execution.

Set Depends On Step and Dependee Attribute Mapping to ensure child records only include those referencing exported parent records.

Example: Export only employees whose companies were exported.

Configure polymorphic associations with type-specific lookup attributes for proper ID remapping on import.

Share migration plan configurations across environments:

1. Export: Click "📋 Export Config" on any plan → Download JSON file
2. Import: Click "📋 Import Config" → Upload JSON → Plan and steps created/updated automatically

Perfect for deploying migration plans to production or sharing with team members.

• Pundit-based authorization on all actions
• Strong parameter validation
• Safe ActiveRecord query evaluation
• Multi-layer input validation
• Complete audit trail

Sidekiq not processing:

• Check Redis: redis-cli ping
• Verify config.active_job.queue_adapter = :sidekiq
• Check logs: tail -f log/sidekiq.log

Filter parameter errors:

• Ensure placeholders are inside quotes: "{{param}}" not {{param}}
• Provide all required parameter values before export

Model not found:

• Verify exact ActiveRecord class name (case-sensitive, singular form)
  • Example: Use "Company" not "company" or "Companies"
• Check lookup attributes exist on target models

This project is licensed under the GNU General Public License v3.0.

Copyright © 2025 Vaibhav Rokkam.