Python Multiprocessing TimeOut which sub function and inputs

Question

I have a process with a lot of sub functions runing with Multiprocessing pool.imap_unordered. Sometimes, the process may stuck. I currently managed that with a timeout as follow :

futures_res = pool.imap_unordered(ImageRequestedTypeGenerationWrapper, InputData.copy()) 

out1, out2, = futures_res.next(timeout=timeout * 60)

I would like to identify which sub function with which parameters failed. Could you advise a method ?

Elizabeth Koroleva · Accepted Answer · 2023-01-06 15:39:39Z

1

Try to wrap the call to the subfunction in a try-except block and to log any errors that occur. You can then check the logs to see which subfunction calls failed.

def process_subfunction(inputs):
    try:
        # Call subfunction with inputs
        result = subfunction(inputs)
    except Exception as e:
        # Log error and inputs
        logging.error(f"Error occurred while calling subfunction with inputs {inputs}: {e}")
        result = None
    return result

futures_res = pool.imap_unordered(process_subfunction, InputData.copy()) 

out1, out2, = futures_res.next(timeout=timeout * 60)

This will log any errors that occur when calling the subfunction, along with the inputs that were passed to the subfunction.

answered Jan 6, 2023 at 15:39

Elizabeth Koroleva

913 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python Multiprocessing TimeOut which sub function and inputs

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related