How to run dependent scripts in python in parallel in shell?

Question

I have 6 scripts called:

A_1.py, B_1.py, C_1.py
A_2.py, B_2.py, C_2.py

How do I run these script with the following dependent logic where C scripts should be run (in parallel) on B is complete while at the same time running A and B scripts in sequential order in shell?

When you have complex dependencies, and some programs could run in parallel, I would concider to use a simple Makefile and run the entire program with make -j4 or someting like that. — kvantour
– kvantour, Commented Jul 17, 2020 at 8:14

Mark Setchell · Accepted Answer · 2020-07-17 18:26:54Z

1

Unless I have misunderstood your table, it is no harder than this:

A_1.py   # Start A_1
B_1.py   # then B_1

# We can now start A_2 followed by B_2, along with C_1 in parallel    
{ A_2.py ; B_2.py; } &
C_1.py &

# Now wait for all of those to complete, and start C_2
wait
C_2.py

edited Jul 17, 2020 at 18:26

answered Jul 17, 2020 at 9:28

Mark Setchell

210k32 gold badges309 silver badges503 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

cesarteaser Over a year ago

Thank you, this works wonderfully. The only question I have is how would I manipulate the parallel execution portion if I wanted to run not only C_1 but also run C3 and C4 in that exact sequence in parallel with A2 and B2? Could you help please?

Mark Setchell Over a year ago

Questions, and answers, are free. Just ask another one, and you’ll get better attention from the whole Stack community.

Mark Setchell Over a year ago

Just change C_1.py to { C_1.py; C_3.py; C_4.py; } &

kvantour · Accepted Answer · 2020-07-17 08:29:48Z

I've never done this before, but I believe makefiles are the way forward:

A Makefile consists of a set of rules. A rule generally looks like this:

targets : prerequisities
   command
   command
   command

The targets are file names, separated by spaces. Typically, there is only one per rule. The commands are a series of steps typically used to make the target(s). These need to start with a tab character, not spaces. The prerequisites are also file names, separated by spaces. These files need to exist before the commands for the target are run.

So let's assume you want to run 5 commands: cmd1, cmd2, cmd3, cmd4, cmd5. Assume the following dependencies:

cmd1 and cmd3 have no dependencies
cmd2 depends on output from cmd1 and cmd4
cmd4 depends on the output of cmd1
cmd5 depends on output of cmd4 and cmd3.

In this case, the Makefile would look like:

all: cmd1.run cmd2.run cmd3.run cmd4.run cmd5.run

cmd1.run : 
   cmd1
   touch cmd1.run

cmd2.run : cmd1.run cmd4.run
   cmd2
   touch cmd2.run

cmd3.run :
   cmd3
   touch cmd3.run

cmd4.run : cmd1.run
   cmd4
   touch cmd4.run

cmd5.run : cmd3.run cmd4.run
   cmd5
   touch cmd5.run

Now you can just run your scripts with the simple command:

$ make

More details on the hocus-pocus: https://makefiletutorial.com/

Collectives™ on Stack Overflow

How to run dependent scripts in python in parallel in shell?

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related