I have been working on a solution for transferring a large XML file into Azure SQL database, thus far unsuccessfully. The XML file is accessed via a fixed URL. I have no previous experience using SQL or Azure before this project, so I apologise if I've missed something obvious.
I setup the database with appropriate tables, I just need to populate the tables with my data.
The XML file comes in the following format, however there are several thousand jobs. The file, when downloaded, is approximately 1.2Mb however I want to access it from the URL not local computer.
<Response api-method="Current">
<Status>OK</Status>
<Jobs>
<Job>
<ID>N1234</ID>
<Name>Job name here</Name>
<Description/>Job description here</Description
<StartDate>2018-10-08T00:00:00</StartDate>
<DueDate>2018-10-21T00:00:00</DueDate>
<Manager>
<ID>12345</ID>
<Name>John Smith</Name>
</Manager>
</Job>
<Job>
Repeat above several thousand times
</Job>
</Jobs>
</Response>
So far my most successful method works using some built-in Azure Logic Apps. I am using HTML GET to load the XML data, then passing it into the Liquid app which converts XML into JSON. I then expect JSON format can be easily loaded into a database table. This Logic App successfully converts the XML to JSON, but only when I strip the file down to ~80 jobs. Hence I know that my logic app works and that the file size is my issue.
Does anyone have a suggestion for a better approach or a method of allowing larger files to be processed? I have also tried using SSMS, implementing T-SQL BulkInsert, OpenRowSet, etc but I've abandoned this approach for the time being.
Small selection of sources I've used while trying to find a solution:
- How can I to read a XML from a URL using T-SQL?
- Load data from XML file to SQL database in Azure using Logic Apps
- Countless MVA videos
Many thanks