Migrating data from JSON, XML or SOAP source

The contributed Migrate Plus module provides capabilities for migrating data to Drupal from JSON, XML or SOAP source.

There is a complete example on how Commerce 2 Product Variations can be migrated to Drupal from an external source system that provides the source data in JSON format. The example uses the URL source plugin with HTTP Basic authentication provided by the Migrate Plus module. Migrate Plus also provides the HTTP fetcher and JSON parser plugins used in the example.

See also Pull Content From a Remote Drupal 8 Site Using Migrate and JSON API by Karen Stevenson (2016) and Migrating JSON files into Drupal by Mauricio Dinarte (2020).

Migrating JSON data

The json data parse provides a pager functionality to request multiple urls if required. Those are the available pager types:

Urls

The next urls are obtained from the response using the selector from configuration.

Configuration values:

• Selector ( required )

It finds the value in the response with the specified selector, and uses it as an URL for the next page to fetch.

Cursor

Obtains an integer value from the response using the selector from configuration. The next urls are generated with the same url, but adding a query attribute, with the key obtained using the "Key" configuration value.

Generates the next urls using a query parameter with the key configuration, and the value from the response using the Selected configuration.

Configuration values:

• Selector ( required )
• Key ( optional, defaults to cursor )

Page

Obtains an integer value from the response using the selector from configuration. It also obtains a max page value using the selector_max selector from configuration. The next urls are generated with the same url, but adding a query attribute, with the key obtained using the "Key" configuration value, and the scalar values from the current page to the selector_max one.

Configuration:

• Selector ( required )
• Key ( optional, defaults to 'page' )
• selector_max ( optional ).
   

An example of usage:

API URL: https://API-URL/api/products.json?page_no=1

'page_no' - is a query parameter responsible for paging.

API request returns the next data related to the pager:

• page_count: 26
• page_current: 1
   

For params above configuration looks the next:

pager:
  type: page
  selector: page_current
  key: 'page_no'
  selector_max: page_count

Paginator

Configuration:

• Selector ( optional )
• default_num_items ( required )
• page_key ( optional, defaults to 'page' )
• size_key ( optional, defaults to 'pagesize' )
• paginator_type ( optional, can be "page_number", or "starting_item". defaults to 'page_number' )

When the selector value is a scalar, it treats it as the number of items in the requests. If it equals to the one defined in default_num_items, it fetches the next page.

When the selector value is an array, it treats it as an array of rows. It fetches the next page until the selector data is an empty array, or something different than a scalar.

When the selector value is missing, it generates the next page url, and does an http request to check if the http response code is 200, or something different like a 404, 500, etc. If the response is a 200 http code, it adds it to the next request urls.

The urls are generated using the page_key, and the size_key query parameters.

See also the Migrate JSON Example module.

Migrating XML data

• Migrating XML files into Drupal by Mauricio Dinarte (2020)
• Stop Waiting for Feeds Module: How to Import RSS in Drupal by Campbell Vertesi (2017)

Migrating SOAP data

Migrate Plus comes with the migrate_example_advanced sub-module (still in progress) that provides an example for importing SOAP data in migrate_plus.migration.weather_soap.yml.

Help complementing this documentation guide by adding XML / SOAP examples to this page.