Exception Handling in Multiprocessing Python

Question

I am trying to execute a script using multiprocessing. But in case of errors the script is not throwing exception and pulling out of the processing. Is it possible to have an exception thrown and come out of the execution. I am not able to figure out how to do this since I am new to python.

Below is the script I have written :

import os 
import sys
import string
import multiprocessing as mp

path=sys.argv[1]

path_tpt=path+'/tpt_script'


""" Multiprocessing module to generate intermediate exports"""



total_tpt_file_list = []
for filename in os.listdir(path_tpt):
    total_tpt_file_list.append(os.path.join(path_tpt,filename))

total_tpt_file_list = list(filter(lambda x:x.endswith('.tpt'), total_tpt_file_list))
total_tpt_file_list = sorted(total_tpt_file_list)
print(total_tpt_file_list) 



def run_sed (file_list):
    x=file_list.rsplit("/", 2)
    y=x[0]
    file_name=x[2]
    print(file_name)
    path_ctl=str(y)+'/ctl_file'
    path_tpt=str(y)+'/tpt_script'
    path_log=str(y)+'/log'
    print("tbuild -f  "+ (file_list) + " -j  "+ 'tpt_chk_$$ '+ '> '+ os.path.join(path_log,file_name).split('.')[0]+'.log 2>>'+ os.path.join(path_log,file_name).split('.')[0]+'.log')
    status=os.system("tbuild -f  "+ (file_list) + " -j  "+ 'tpt_chk_$$ '+ '> '+ os.path.join(path_log,file_name).split('.')[0]+'.log 2>>'+ os.path.join(path_log,file_name).split('.')[0]+'.log')

try:    
    p = mp.Pool(processes=(mp.cpu_count()-1))
    total_file_list = p.map(run_sed,total_tpt_file_list)

finally:
    p.close()
    p.join()
    print("done")

Please let me know if anymore information is needed.

Thanks in Advance.

Where and what kind of exceptions are thrown in your script that you want to handle? Please be aware that we cannot run your script, and therefore don’t know how it and its auxiliary processes behave. — MisterMiyagi
– MisterMiyagi, Commented Feb 26, 2020 at 8:00

Macattack · Accepted Answer · 2020-02-26 07:30:36Z

1

Python exceptions will be propagated back up, given what I see, I'm guessing you're wanting errors from your os.system call to be raised. You'd need to do something like

if status != 0: 
    raise Exception("eeek")

What I really recommend, rather than using os.system is to use subprocess.check_output https://docs.python.org/3/library/subprocess.html#subprocess.check_output It will automatically check the status code of the subprocess call and raise any exception with the status code and the output text. So, your code would look something like:

    subprocess.check_output(["tbuild", "-f"] + file_list + ["-j", "tpt_chk_$$", ...

You may also want to return the output (return subprocess.check_output...)

answered Feb 26, 2020 at 7:30

Macattack

1,99010 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

brarjun Over a year ago

Will the script exit from execution as well?

Macattack Over a year ago

What do you mean by "script"? If a call to tbuild fails, it'll be what raises the exception so it'll already have stopped. If you don't catch the exception your python program will stop. Other subprocess calls to tbuild should exit when the python program stops I think.

brarjun Over a year ago

I am trying to run a multiprocessing script. I am just concerned , if exiting from the main process will also ensure that the child processes es are also exit-ed

Macattack Over a year ago

In my testing, the other processes did not stop. That's a whole separate problem if you want to solve that.

Collectives™ on Stack Overflow

Exception Handling in Multiprocessing Python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related