Debugging migrations

Executing and rolling back migrations

The contributed module, Migrate Tools, provides the drush migrate:import command which executes a collection or individual migration import. The command, drush migrate:rollback, can be used to undo a collection or individual migration so that it can be executed again after adjusting it.

If the execution of a migration fails, its state may continue to say "Importing". Running the migration in this state again will give you an error message "Migration xxx is busy with another operation". To fix this, you can stop and reset the migration with the drush migrate:reset-status command.  You may need to clear your cache if you have made changes to the migration Yaml.

View messages of a migration execution

When migrations are executed with Drush using contributed Migrate Tools module, the messages of a migration can be used with drush migrate:messages <migration> or alias drush mmsg <migration>.

You can also look for yourself in the migration's database tables. There are two relevant tables for each migration:

• The map table, named migrate_map_MIGRATION_NAME
• The message table, named migrate_message_MIGRATION_NAME

The two tables can be JOINed on the source_ids_hash field. In the map table, the source_row_status and rollback_action field values are constants defined in MigrateIdMapInterface:

Status values:

• 0: Imported
• 1: Needs update
• 2: Ignored
• 3: Failed

Rollback values:

• 0: Delete
• 1: Preserve

Printing debug information when using Drush

If you execute migrations using Drush, you can print debug information using drush_print() (Drush 8) or Drush::output()->writeln(...); (Drush 9+). This can be included for example in the prepareRow() method of the source plugin. 

Print source and pipeline data

var_dump callback

To investigate data within a migration, you can print data by combining the callback plugin and var_dump():

process:
  dump_sourcevar:
    plugin: callback
    callable: var_dump
    source: sourcevar 

You can use this to debug data inside of a pipeline too:

process:
  field_example:
    -
      plugin: callback
      callable: var_dump
      source: sourcevar
    -
      plugin: some_plugin

Migrate Devel

The Migrate Devel module provides two options you can use with the Migrate Tools drush command migrate:import (mim):

drush migrate:import <migration> --migrate-debug
drush migrate:import <migration> --migrate-debug-pre

• --migrate-debug: prints each row as it's migrated (MigrateEvents::POST_ROW_SAVE)
• --migrate-debug-pre: same idea, but before the process is run on the row (MigrateEvents::PRE_ROW_SAVE)

Migrate Sandbox!

The Migrate Sandbox! module provides a UI for developers to quickly and safely experiment with migrate process plugins and migrate process pipelines.

Utilizing XDebug

Various files are useful to start putting breakpoints in starting with core/modules/migrate/src/Plugin/Migration.php and core/modules/migrate/src/MigrateExecutable.php. For a list of fields and their values being pulled from the source plugin, put a breakpoint in the next() function in core/modules/migrate/src/Plugin/migrate/source/SourcePluginBase.php.

Analyzing why migration does not show up in drush migrate:status

This assumes that migration is done through a custom module via workflow https://cgit.drupalcode.org/migrate_plus/tree/migrate_example
An example migration definition is https://cgit.drupalcode.org/migrate_plus/tree/migrate_example/config/ins... (you must use migrate_plus.migration.  as prefix, not your_module.migration.)

drush en migrate_example -y
drush ms

Given a migration_id load its migration object and check messages why it failed to initialize.

$ drush php
Psy Shell v0.8.11 (PHP 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 — cli) by Justin Hileman
>>> $migration_id = 'beer_node';
>>> $manager = \Drupal::service('plugin.manager.migration');
>>> $plugins = $manager->createInstances([]);
>>> $migration = $plugins[$migration_id];
=> Drupal\migrate\Plugin\Migration {#11436
     +"class": "Drupal\migrate\Plugin\Migration",
     +"uuid": "2ce3ab55-bf95-4759-a2ee-630e5984f211",
     +"langcode": "en",
     +"status": true,
     +"_core": [
       "default_config_hash" => "B7O2KKDymRasREDBdEEavpADUN2SRyCkWY2ldOca2Xo",
     ],
     +"field_plugin_method": null,
     +"cck_plugin_method": null,
     +"migration_group": "beer",
   }
>>> $migration->getSourcePlugin()->checkRequirements();
Drupal\Component\Plugin\Exception\PluginNotFoundException with message 'The "beer_node" plugin does not exist.'

Other checks that shows a migration error.

>>> $map = $migration->getIdMap();
>>> $imported = $map->importedCount();
Drupal\Core\Database\ConnectionNotDefinedException with message 'The specified database connection is not defined: aa'

Debugging migration definition

$ drush php
Psy Shell v0.8.11 (PHP 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 — cli) by Justin Hileman
>>> $migration_id = 'beer_node';
>>> $manager = \Drupal::service('plugin.manager.migration');
>>> $plugins = $manager->createInstances([]);
>>> $migration = $plugins[$migration_id];
>>> $migration->getPluginDefinition();

Reinstalling blocked by configuration items

When you have changed your migration(s) and want to retest them sometimes you have to manually remove their configuration. Use drush config:delete your-migration-file-name-without-extension.

config_devel

Or let config_devel do the config reloads for you.

Cancelling failed migration

If migration fails, you might get the following error while trying to rerun the migration: 

 Migration MIGRATION_NAME is busy with another operation: Importing

Run the following command to break the migration lock if Migrate Tools or Migrate Run are installed:

$ drush migrate:reset-status MIGRATION_NAME       # or drush mrs MIGRATION_NAME

If Migrate Tools or Migrate Run are not installed, use:

$ drush php-eval "var_dump(Drupal::keyValue('migrate_status')->set('MIGRATION_NAME', 0));"
NULL

If you are getting the following error while trying to run command above:

\Drupal::$container is not initialized yet. \Drupal::setContainer() must be called with a real container.

you are probably running multisite website so don't forget to specify the website.

Limiting and individual migrations

The following commands require Migrate Tools to be installed: 

When developing large migrations, it is useful to limit the number of rows being imported. This speeds up the import/rollback cycle whilst refined field mappings, callbacks etc.

$ drush migrate:import --limit=10 MIGRATION_NAME 

The following commands allow an individual migration entity to be imported and rolled back for development purposes:

$ drush migrate:import --idlist=123 MIGRATION_NAME 
$ drush migrate:rollback --idlist=123 MIGRATION_NAME 

The id is usually the primary id of the entity being migrated, but more information can be found in the migrate_map_MIGRATION_NAME database tables, particularly looking at the sourceId column.