Kill python script that executes multiple subprocesses

Question

I have a python script that executes other python scripts using subprocess. My problem is that if and I want to termintate the full process, with for ex. crt-c, it kills only the current iteration but starts the execution of the following one. The main script:

for fold in range(10):
    comand = "python train_kfolds_pos.py {} {}".format(current_model,fold)
    print(comand)
    process = subprocess.Popen(comand, shell=True, stdout=subprocess.PIPE)
    process.wait()
    text = process.communicate()[0].decode("utf-8")
    print(text)

On Linux you can kill the parent process and its children by killing the process group. More info: unix.stackexchange.com/questions/14815/process-descendants — Alex V
– Alex V, Commented Nov 29, 2018 at 18:23

Jean-François Fabre · Accepted Answer · 2018-11-29 18:31:38Z

2

since you're not checking the return code of wait() the program continues with the loop, even if the sub-process is interrupted.

You should check it (and use a list of args/remove shell=True)

for fold in range(10):
    comand = ["python","train_kfolds_pos.py",current_model,str(fold)]
    print(comand)
    process = subprocess.Popen(comand, stdout=subprocess.PIPE)
    rc = process.wait()
    # quit loop if return code is not 0
    if rc:
       print("aborted, returncode {}".format(rc))
       break
    text = process.communicate()[0].decode("utf-8")
    print(text)

a side effect of this is that if the program terminates abnormally for some other reason, it stops too.

Aside: you could think about not calling a python subprocess but directly call a function within the python train_kfolds_pos.py script and parameters. Exceptions would propagate naturally (so would keyboard interrupt exceptions). Something like:

import train_kfolds_pos
for fold in range(10):
    train_kfolds_pos.compute(current_model,fold)

edited Nov 29, 2018 at 18:31

answered Nov 29, 2018 at 18:22

Jean-François Fabre♦

141k24 gold badges179 silver badges246 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Miguel Over a year ago

Ok, it worked. And thanks for the aside. I'll follow you advice.

Collectives™ on Stack Overflow

Kill python script that executes multiple subprocesses

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related