0

I'm interested in pulling data from S3 using ADF. My intention is to grab the CSV data from S3, convert it to JSON and send it to a HTTP api using the REST target within ADF. I'm curious if it's possible to convert the data to JSON during the import/process within Azure?

Or am I forced to first import the data, store it in some storage service within Azure, convert it to JSON and then send it to the HTTP API. I want to avoid having to store it in some storage service within Azure. Is it possible to convert the CSV data to JSON without using some storage service within Azure?

Improve this question

1 Answer 1

Reset to default
1

Yes, it is very much possible using Azure Data Factory. You don't need to store the source data anywhere in Azure. Just directly load the it from Amazon S3, use the Azure Copy Activity to convert the CSV file to JSON and send to with the HTTP API.

Azure Data Factory connector allows to connect the AWS S3 using Linked Service. This official tutorial help help you to create a Linked Service which will connect your Azure Data Factory with Amazon S3. You will also create Dataset using DelimitedText option and Linked Service will be used with this dataset.

In copy activity, you need to select option File pattern as Array of objects in the sink tab to convert the CSV to JSON as shown below.

enter image description here

Once the data has been converted, you can use REST connector to copy at an endpoint. Refer Copy and transform data from and to a REST endpoint by using Azure Data Factory.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.