Azure Data Factory - Event based triggers on multiple files/blobs

Question

I am invoking an ADF V2 pipeline via an event based trigger when new files/blobs are created in a folder within a blob container.

Blob Container structure:

BlobContainer -> FolderName ->

-> File1.csv

-> File2.csv

-> File3.csv

I've created the trigger with below configuration:

Container Name: BlobContainer

Blob path begins with: FolderName/

Blob path ends with: .csv

Event Checked:Blob Created

Trigger Screenshot

Problem: Three csv files are created in the folder on ad hoc basis. The trigger that invokes the pipeline runs 3 times (probably because 3 blobs are created). The pipeline actually move the files in another blob container. So the 1st trigger run succeeds and remaining 2 fails because the files have been moved already. However how can I configure the trigger so that it only run once per folder even though 3 files are created within it?

Because the files are generated together, I am required to move them together into a new location using ADF.

Vijay Kumar Sharma · Accepted Answer · 2019-11-19 18:31:10Z

1

Your blobEventTrigger triggered the pipeline for each file, For it, you can use a 'lookup activity' which gets the filenames and then use filter activity, which filtered the required filename and gives the filterdItemCounts attribute that could be checked in the IF Activity. When there is no file the filterdItemCounts returns '0'and your pipeline not triggered.

Summary- Lookup Activity -> Filter Activity -> IF Activity -> Your Pipeline

answered Nov 19, 2019 at 18:31

Vijay Kumar Sharma

3971 gold badge4 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

skadam85 Over a year ago

I am also having the same question. How can Lookup activity get the filenames? Can you elaborate please?

Vijay Kumar Sharma Over a year ago

you can use "parameters": { "fileName": "@triggerBody().fileName"} Parameter in your trigger and this filename can be also used in the Lookup activity for your file.

Michal Over a year ago

I would suggest to take a look on similar issue stackoverflow.com/questions/64504159/…

Collectives™ on Stack Overflow

Azure Data Factory - Event based triggers on multiple files/blobs

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related