Mapping Dataflows iterate (avoiding Foreach loop)

Question

Within Mapping data flow (trying to avoid Foreach)

Is this possible with a structure like this container/directory/File

Land/’Folder1/20210815’/a.csv
Land/’Folder1/20210816’/b.csv
Land/’Folder2/20210815’/c.csv

I want to load Folder1 and subfiles and sink them to Base container in the data lake

Base/’Folder1/20210815’/a.csv
Base/’Folder1/20210816’/b.csv

In dataflow, I will add one derived column which is loaddate (or anything I want in dataflow).

All files contain the same csv columns structure column1,column2,column3,loaddate

I do not want to specify a date basically bulk load any csv under a subfolder

Thanks

Trent Tamura · Accepted Answer · 2021-08-16 19:45:43Z

1

Here is my example using a Copy Activity without using a For Each:

I have 4 files with this structure in Blob storage:

Folder1/20210815/Placeholder.csv
Folder1/20210815/Placeholder5.csv
Folder1/20210816/Placeholder3.csv
Folder2/20210816/Placeholder3.csv

Here is the copy data details:

Here are those files and as you can see the subfolders were copied over. This theoretically should work in Data Lake, but I can't be sure. It definitely works in Azure Blob Storage.

Here are the contents of the 20210815 Subfolder:

answered Aug 16, 2021 at 19:45

Trent Tamura

1,1557 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Trent Tamura Over a year ago

FYI, I did attempt a dataflow and could get the data over, but it would not copy the subfolders as well.

Collectives™ on Stack Overflow

Mapping Dataflows iterate (avoiding Foreach loop)

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related