I have a Text delimited file in my Azure data factory. I have to convert it to a json file. I also have a python code that converts my text delimited file to a json file. Now how do I integrate this python code in azure data factory. How to run this code from ADF?
1 Answer
You can use the Azure Data Factory Data Flow to do a lot of transforms like csv to JSON without Python (see this answer: Convert csv files,text files,pdf files into json using Azure Data Factory).
If you need the control Python offers, you can use Azure Batch to run your python file. In your python, you can grab the csv from a blob using blob_client.download_blob() to bring it down to a local file on the batch VM. Then you can load the file normally (pd.read_csv()) and do your transform. After you write your json file locally, you can upload back to the blob and then do whatever other ADF actions you want. I used this set of instructions to get started running Azure Batch python: https://learn.microsoft.com/en-us/azure/batch/quick-run-python
