How to run multiple asynchronous processes in Python using multiprocessing?

Question

I need to run multiple background asynchronous functions, using multiprocessing. I have working Popen solution, but it looks a bit unnatural. Example:

from time import sleep
from multiprocessing import Process, Value
import subprocess

def worker_email(keyword):
    subprocess.Popen(["python", "mongoworker.py", str(keyword)])
    return True

keywords_list = ['apple', 'banana', 'orange', 'strawberry']

if __name__ == '__main__':
    for keyword in keywords_list:
        # Do work
        p = Process(target=worker_email, args=(keyword,))
        p.start()
        p.join()

If I try not to use Popen, like:

def worker_email(keyword):
    print('Before:' + keyword)
    sleep(10)
    print('After:' + keyword)
    return True

Functions run one-by-one, no async. So, how to run all functions at the same time without using Popen?

UPD: I'm using multiprocessing.Value to return results from Process, like:

def worker_email(keyword, func_result):
    sleep(10)
    print('Yo:' + keyword)
    func_result.value = 1
    return True

func_result = Value('i', 0)
p = Process(target=worker_email, args=(doc['check_id'],func_result))
p.start()
# Change status
if func_result.value == 1:
    stream.update_one({'_id': doc['_id']}, {"$set": {"status": True}}, upsert=False)

But it doesn't work without .join(). Any ideas how to make it work or similar way? :)

MegaIng · Accepted Answer · 2017-12-23 11:11:09Z

2

If you just remove the line p.join() it should work. You only need p.join if you want to wait for the process to finish before executing further. At the end of the Program python waits for all Process to finished before closing, so you don't need to worry about that.

edited Dec 23, 2017 at 11:11

answered Dec 23, 2017 at 10:42

MegaIng

7,9442 gold badges24 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

sortas Over a year ago

Can you you explain a little bit more? :) And do I need to use p.close() instead, like for Pools?

MegaIng Over a year ago

@sortas I edited the answer. No you don't need p.close(). Only Pools have this function.

sortas Over a year ago

One more detail: I'm trying to use shared Values to return result from process. Something like:

func_result = Value('i', 0)                     p = Process(target=worker_email, args=(doc['check_id'],func_result))                     p.start()                     # Обновляю статус задачи на "Выполнено" (если все ок) или не меняю статус и отправляю на повторое выполнение (если не ок)                     if func_result.value == 1:                         stream.update_one({'_id': doc['_id']}, {"$set": {"status": True}}, upsert=False)

But I can't do it without join now :) Any ideas?

sortas Over a year ago

Oh, didn't know how comments are formatted, sorry :)

sortas Over a year ago

Added to UPD. Hope you'll advice me smth :)

sortas · Accepted Answer · 2017-12-23 13:10:30Z

0

Solved problem with getting Process result by transferring result check and status update into worker function. Something like:

# Update task status if work is done
def update_status(task_id, func_result):
    # Connect to DB
    client = MongoClient('mongodb://localhost:27017/')
    db = client.admetric
    stream = db.stream

    # Update task status if OK
    if func_result:
        stream.update_one({'_id': task_id}, {"$set": {"status": True}}, upsert=False)

    # Close DB connection
    client.close()

# Do work
def yo_func(keyword):
    sleep(10)
    print('Yo:' + keyword)
    return True

# Worker function
def worker_email(keyword, task_id):
    update_status(task_id, yo_func(keyword))

answered Dec 23, 2017 at 13:10

sortas

1,7415 gold badges21 silver badges32 bronze badges

Collectives™ on Stack Overflow

How to run multiple asynchronous processes in Python using multiprocessing?

2 Answers 2

5 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related