3

I have a python script that takes several hours to complete. I usually run it as:

python myscript.py arg1 arg2 ... argN 2> log.err > log.out &

I'd like to have a shell script that runs several time the same python script (with different parameters). This script has to start the second python script ONLY when the first one has terminated and so on. I think I should use PID of the processes and iteratively check using sleep within some while loops. Any suggestions? The scripts are executed in a remote machine so I have to put them in background in order the guarantee they keep executing even when I log out.

5
  • where are you deriving the args from? Commented Aug 3, 2014 at 17:15
  • I set them manually. They are independent from on run to the other. Commented Aug 3, 2014 at 17:16
  • Why not use a bash script to just run each command and not put those commands in the background with &? You can always put that control script in the background instead. Commented Aug 3, 2014 at 17:17
  • Because the python scripts can't be run in parallel at the same time. Commented Aug 3, 2014 at 17:19
  • 3
    But a script that doesn't start them with & will run them sequentially. Commented Aug 3, 2014 at 17:20

4 Answers 4

5

Instead of backgrounding each invocation of python, use a script which runs all the jobs one by one and background the script. For example:

#!/bin/sh

python myscript.py arg1 arg2 ... argN 2> log.err > log.out
python myscript.py different_args
# etc.

Make the script executable (chmod +x script.sh) and run it from the shell like ./script.sh &


Perhaps you should take a look at a tool like screen, which will keep your jobs running after you log off. If you do screen ./script.sh, then detach (ctrl-a-d) and log off, your script will continue to run. Then you can log back in later and reattach with screen -r.

Sign up to request clarification or add additional context in comments.

7 Comments

The python scripts can't run at the same time. The have to be executed one by one sequentially
In my answer, that is exactly what will happen.
The problem is that I run them in a remote machine so they have to be in background to be running even when I log out
In that case, you should consider using the screen utility to keep an active session open when you log out, or run the entire script.sh in the background, as @TomFenech points out.
Then just run the global script with nohup ./script.sh &. Nohup avoid background script to be killed at end of interactive session.
|
1

For example, you can use cycle (make a script runner.sh with the next content:

n=0
while read -r line
do
    let n++
    echo python myscript.py $line #>log.$n.log 2>log.$n.err
done <<ARGS
a1 a2 a3 a4
b1 b2
c1 c2 c3 c4
ARGS

this will dry run your python script with args

a1 a2 a3 a4
b1 b2
c1 c2 c3 c4

sequentially, e.g. when the previous ends.

If you satisfied, remove the echo and the # from the line

    echo python myscript.py $line #>log.$n.log 2>log.$n.err

The log files will be numbered in sequence... 1..n.

Why do you want run the python script in the background? You always can send to background the above script and the whole sequence will be done in the bg.

If you need run it after the logout, use the above runner.sh as

chmod 755 runner.sh
nohup runner.sh
logout

or much better use the screen utility, and you can logout and login from another place and re-attach the session and so on... read man screen.

Comments

0

Make a bash script?

python first.py
python second.py

Comments

0

If you want one script to run directly after the other, simply drop the trailing &. The ampersand tells bash to run the process in the background, which allows you to run multiple processes in parallel (some in the background, and some in the foreground). If you want to run them sequentially, you should run them each in the foreground, like so:

python myscript.py argX1 argX2 ... argXN 2> logX.err > logX.out
python myscript.py argY1 argY2 ... argYN 2> logY.err > logY.out

This will run the script with the argx arguments, then, after that finishes, run the same script again with the argY arguments.

If you want the next execution to run only if the first run is successful, you can use the double ampersand (&&) like so (note that the trailing backslash (\) escapes the newline, so the entire thing is actually a single command):

python myscript.py argx1 argx2 ... argN 2> logx.err > logx.out && \
python myscript.py argy1 argy2 ... argN 2> logy.err > logy.out

You can chain more than two commands with double ampersands, but for longer scripts, you should consider using set -e at the beginning, which will cause bash to stop executing the script after the first error (see http://www.davidpashley.com/articles/writing-robust-shell-scripts/#id2382181)

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.