Process a function on different arguments in parallel in Python

Question

This is my simple code where I want to run printRange() in parallel:

def printRange(lrange):
    print ("First is " + str(lrange[0]) + " and last is " + str(lrange[1]))


def runInParallel():
    ranges = [[0, 10], [10, 20], [20, 30]]
    // Call printRange in parallel with each sublist of ranges given as argument

My question is different from this SO question as here, the each process is hardcoded, started and finally joined. I want to run printRange() in parallel with say 100 other printRange() worker functions. Hardcoding each time is not feasible. How could this be done?

Parallelization is not panacea. There is overhead involved that is likely more expensive than this function, and because output is involved, there may be race conditions. You certainly can't expect the output to appear in the order implied by ranges, and in general you can't even assume that all the output from a single process will be written as a single unit. — chepner
– chepner, Commented Feb 20, 2019 at 12:58

balderman · Accepted Answer · 2019-02-20 12:58:50Z

12

Using multiprocessing

from multiprocessing import Pool


def print_range(lrange):
    print('First is {} and last is {}'.format(lrange[0], lrange[1]))


def run_in_parallel():
    ranges = [[0, 10], [10, 20], [20, 30]]
    pool = Pool(processes=len(ranges))
    pool.map(print_range, ranges)


if __name__ == '__main__':
    run_in_parallel()

Output:

First is 0 and last is 10
First is 10 and last is 20
First is 20 and last is 30

answered Feb 20, 2019 at 12:58

balderman

24k8 gold badges39 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Stef van der Zon · Accepted Answer · 2019-02-20 12:47:34Z

1

Something like this?

import threading

def printRange(lrange):
    print ("First is " + str(lrange[0]) + " and last is " + str(lrange[1]))

def runInParallel():
    ranges = [[0, 10], [10, 20], [20, 30]]
    for i in ranges:
        t = threading.Thread(target=printRange, args = [i])
        t.start()

runInParallel()

answered Feb 20, 2019 at 12:47

Stef van der Zon

6294 silver badges14 bronze badges

3 Comments

BlackBear Over a year ago

Threading is not parallel (although it gives the same headaches). Use multiprocessing instead. And you might want to add a join at the end

Stef van der Zon Over a year ago

I thought the only difference was that multiprocessing uses seperate memory space. Threading is also parallel. Easy example: use 5 threads on a time.sleep(1) code and it will take 1 second total.

BlackBear Over a year ago

Yes because the threads are sleeping :) try having the threads do some actual work, and in the meantime read about python's global interpreter lock: realpython.com/python-gil

Aekasitt Guruvanich · Accepted Answer · 2021-08-23 07:20:16Z

0

Using async/await for comparison

from asyncio import BaseEventLoop, get_event_loop, gather
from typing import List

async def print_range(lrange: List[int] = []):
  print('First is {} and last is {}'.format(lrange[0], lrange[1]))

def main():
  loop: BaseEventLoop = get_event_loop()
  lranges: list   = [[0, 10], [10, 20], [20, 30]]
  loop.run_until_complete(gather(*(print_range(_) for _ in lranges)))

if __name__ == '__main__':
  main()

answered Aug 23, 2021 at 7:20

Aekasitt Guruvanich

13 bronze badges

Collectives™ on Stack Overflow

Process a function on different arguments in parallel in Python

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related