execute list of python files in directory shell script

Question

My code:

#!/bin/bash

scriptspy =(
    '/scs/sp1.py',
    '/scs/sp2.py',
    '/scs/sp3.py',
    '/scs/sp4.py',
    '/scs/spweb.py',
    '/scs/sp11.py',
    '/scs/spservice.py',
    ....
)

for i in scriptspy;
  do python3.7 $i;
done

My goal is to run a list of files that have python within the directory and not need to run one by one. individually would be python3.7 sp1.py

Obs: I can't execute all .py files in the directory, I need to pass a list to the loop with their name.

to iterate over the array's elements you need for i in ${scriptspy[@]}. Your current code for i in scriptspy is iterating over a single string scriptspy. — builder-7000
– builder-7000, Commented Dec 17, 2019 at 20:03
@sergio The parameter expansion should be quoted. The array isn't properly defined, though, either: scriptspy=(/scs/sp1.py /scs/sp2.py /scs/sp3.py /scs/sp4.py). — chepner
– chepner, Commented Dec 17, 2019 at 20:15

wjandrea · Accepted Answer · 2019-12-17 20:09:23Z

2

Use a glob:

shopt -s nullglob
for i in /scs/sp[1-4].py; do
    python3.7 "$i"
done

I turn on nullglob here just in case the files don't exist.

As well, it's good practice to always quote variables.

BTW your array creation and dereferencing syntax is wrong. LMK if you want details.

edited Dec 17, 2019 at 20:09

answered Dec 17, 2019 at 19:36

wjandrea

33.8k10 gold badges69 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

12 Comments

JB_ Over a year ago

In my case I have some scripts inside the directory that I can't execute, because they are dependencies, so I need to pass a list with the name of all of them.

wjandrea Over a year ago

@JB_ Oh, then just use a more precise glob. I updated the answer. I thought you wanted to run all files with a .py extension.

JB_ Over a year ago

Can you do the reverse? Pass the files as I do not want you to run as an exception?

user1934428 Over a year ago

Because your glob pattern expands in alphanumeric order, and we don't know whether this is the order the OP wanted to have. In the concrete example he gave, he wanted for instance spweb to be executed before sp11, but after sp4. If you would modify your answer to for f in /scs/sp{[1-4],web,11,service}.py, it would be more obvious how the OP could keep his original order.

wjandrea Over a year ago

@user1934428 Ohh, I see what you mean. The question was edited after I posted my answer.

|

Gilles Quénot · Accepted Answer · 2019-12-17 20:10:34Z

1

Simply :

#!/bin/bash

printf '%s\n' *.py | xargs -n1 python3.7

edited Dec 17, 2019 at 20:10

answered Dec 17, 2019 at 19:58

Gilles Quénot

188k43 gold badges232 silver badges229 bronze badges

Comments

Mark Setchell · Accepted Answer · 2019-12-17 20:27:08Z

1

I would get them all done in parallel with GNU Parallel:

parallel python3.7 ::: sp[1-4].py

If you want to see what it would do, without actually doing anything, use:

parallel --dry-run ...

If you only want to run them sequentially, one after the other, use:

parallel -j1 ...

answered Dec 17, 2019 at 20:27

Mark Setchell

210k32 gold badges309 silver badges503 bronze badges

2 Comments

JB_ Over a year ago

the vps that will run has only one core, I can't use parallelism

Mark Setchell Over a year ago

It doesn't matter, GNU Parallel will detect that.

Shambu · Accepted Answer · 2019-12-17 20:12:16Z

-1

This should run python programs in parallel if shell script is not suppose to wait for all of them to be done

declare -a scriptspy=("/scs/sp1.py" "/scs/sp2.py", "/scs/sp3.py","/scs/sp4.py")

for i in "${scriptspy[@]}":
  do (python3.7 "$i" ; echo "script $i done") &
done

edited Dec 17, 2019 at 20:12

answered Dec 17, 2019 at 19:37

Shambu

2,9021 gold badge23 silver badges17 bronze badges

4 Comments

wjandrea Over a year ago

This won't run properly. You need to fix the array creation and dereferencing syntax first. LMK when you do and I'll reverse my downvote.

JB_ Over a year ago

But doesn't the loop wait for the python script to finish to start the other?

Shambu Over a year ago

Loop doesn't wait because you are running python process in back groud

JB_ Over a year ago

So what I want to do is not feasible, because it cannot run in parallel, only what is currently in the loop, otherwise it will use 100% machine resources.

Collectives™ on Stack Overflow

execute list of python files in directory shell script

4 Answers 4

12 Comments

Comments

2 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

12 Comments

Comments

2 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related