True Concurrency in Python

Question

I am relatively new to concurrency in Python and I am working on some code that has to call functions outside of my code. I cannot edit those functions but I need them to run concurrently. I've tried a few different solutions like Multiprocessing, Threading and AsyncIO. AsyncIO comes the closest to what I want if every function I was calling was defined with it, but they're not.

The functions I'm calling will block. Sometimes for 15-30 minutes. During that time, I need other functions doing other things. The code below illustrates my problem. If you run it you'll see that whether using Threads or Multiprocesses, the tasks always run serially. I need them to run simultaneous to each other. I get that the output blocks until the entire script runs, but the tasks themselves should not.

What am I missing? With so many choices for concurrency or at least apparent concurrency in Python, I would think this is easier than I'm finding it.

#!/usr/bin/python3

from datetime import datetime
from multiprocessing import Process
import sys
from threading import Thread
from time import sleep


def main():

    # Doing it with the multiprocess module
    print("Using MultiProcess:")
    useprocs()

    print("\nUsing Threading:")
    usethreads()


def useprocs():
    procs = []
    
    task1 = Process(target=blockingfunc('Task1'))
    task1.start()
    procs.append(task1)

    task2 = Process(target=blockingfunc('Tast2'))
    task2.start()
    procs.append(task2) 

    task1.join()
    task2.join()
    
    print('All processes completed')


def usethreads():
    threads = []

    task3 = Process(target=blockingfunc('Task3'))
    task3.start()
    threads.append(task3)

    task4 = Process(target=blockingfunc('Task4'))
    task4.start()
    threads.append(task4)   

    task3.join()
    task4.join()

    print('All threads completed')


def blockingfunc(taskname):
    now = datetime.now()
    current_time = now.strftime("%H:%M:%S")
    print(current_time, "Starting task: ", taskname)
    sleep(5)
    now = datetime.now()
    current_time = now.strftime("%H:%M:%S")
    print(current_time, taskname, "completed")


if __name__ == '__main__':
    try:
        main()
    except:
        sys.exit(1)

Tim Peters · Accepted Answer · 2021-05-05 21:55:04Z

2

Note that the program you posted imports Thread but never uses it.

More importantly, in a line like:

task1 = Process(target=blockingfunc('Task1'))

you're calling blockingfunc('Task1') and passing what it returns (None) as the value of the target argument. Not at all what you intended. What you intended:

task1 = Process(target=blockingfunc, args=['Task1'])

Then, as intended, blockingfunc isn't actually invoked before you call the start() method.

answered May 5, 2021 at 21:55

Tim Peters

71.4k14 gold badges133 silver badges140 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Jim Over a year ago

Buy this man a beer! I've been staring at this code for far too long and didn't even see that my usethreads function wasn't calling threads. More to the point, your explanation of the way Thread and Process take in the function to run and args made more sense in fewer words than all the web pages I've read about them so far. Thanks for the fast and concise response.

Collectives™ on Stack Overflow

True Concurrency in Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related