Each Monday, I have a number of files arrive in to a Sharepoint folder Folder consists of Folder/different clients. I have a python script (also stored on a different location in sharepoint) that transforms these files in the different clients folder and produces an output file.
All I want to do is have something, be it in data factory or any suggestion, to poll if the Sharepoint folder has a modified file, then trigger the python script to run.
Looking for ideas.
All I want to do is have something, be it in data factory or any suggestion, to poll if the Sharepoint folder has a modified file, then trigger the python script to run.
You can try the below possible workarounds in this scenario. Considering that you need a standard type, you can achieve the below using standard type of logic app.
Using Function app with Logic app:
If you could try to modify your code as per the function app, then you can opt for this approach. You can use When an item is created or modified trigger in the logic app with SharePoint connector and set the timer as per your requirement. Upon the trigger, call Azure function app with the desired authentication method.
Using ADF Custom activity with logic app:
This approach involves ADF Custom activity. So, it needs the python script file to be in Storage account.
If you don't want to modify your existing code, then you can try this approach. First have a logic app and copy the SharePoint python code file to an ADLS gen2 storage account. Or you can directly download and upload the same as it's only a one-time process.
Take a logic app with same When an item is created or modified trigger and then, take ADF connector with desired authentication method. Take the Create a pipeline run action.
In the ADF pipeline, add a Custom activity with the required python file selected.
You can go through this MS documentation to know how to configure a Custom activity.
After building the pipeline with custom activity, publish the pipeline. In the logic app, select the same pipeline.
In this step, you can pass the parameters like modified file name if there is a need. But you need to ensure that you define those parameters in the pipeline in first place.
