Bash script to execute a python package on all files in a folder

Question

I am trying to run a python package called mETL from PuTTY and use it in all the files contained in a folder. The python package is mETL and I am using it to load data contained in 3 .csv files called upload-A.csv, upload-B.csv and upload-C.csv

Everything works perfectly when I do this process manually using the following commands:

metl -m migration.pickle -t new_migration.pickle -s folder_test/upload-A.csv config3.yml
metl -m migration.pickle -t new_migration.pickle -s folder_test/upload-B.csv config3.yml
metl -m migration.pickle -t new_migration.pickle -s folder_test/upload-C.csv config3.yml

All the data from every file is correctly uploaded or updated and the pickle files are updated accordingly.

But instead of doing this manually I want to make a loop that does this for all files contained in my 'folder_test/' folder, for which I tried the following Bash script:

folder_var=folder_test
for x in $folder_var
do
metl -m migration.pickle -t new_migration.pickle -s $x config3.yml
done

What happens after this is that the pickle files are created but no data is uploaded to the database.

can you format the commands that you are using in more readable way ,can't understand much from it. — g4ur4v
– g4ur4v, Commented Sep 10, 2014 at 13:18

g4ur4v · Accepted Answer · 2014-09-10 13:40:33Z

1

Try this

for x in folder_test/*
do
metl -m migration.pickle -t new_migration.pickle -s "${x}" config3.yml
done

edited Sep 10, 2014 at 13:40

answered Sep 10, 2014 at 13:21

g4ur4v

3,2985 gold badges35 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

glenn jackman Over a year ago

1) don't use ls, use for x in folder_test/*, 2) to protect filenames with spaces, make sure you quote "$x" wherever it's used

g4ur4v Over a year ago

Thanks for suggesting the changes.

Collectives™ on Stack Overflow

Bash script to execute a python package on all files in a folder

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related