How to parallelize a for loop in Python?

Question

Some posts about parallelizing for loop in Python already exist such as this one but I can't use them to deal with my issue. Let's take a simple example. I have three lists :

L1 = [1,2,3]
L2 = [3,4,5]
L3 = [5,6,7]

I would like to double each element of the list. I could do this :

for l in [L1,L2,L3] :
    for i in range(len(l)) :
        l[i] = l[i]*2

How please could I parallelize this code to transform L1, L2 and L3 in parallel ?

Note that this example is just to have a clear and easy to understand example, I know that it is not a good idea in reality to parallelize a quick code like that

Can you explain why you can't use the answers in the linked question? — pho
– pho, Commented Jun 7, 2022 at 12:28
Just because I don't understand how to apply the answer in the linked question to my example — Ewdlam
– Ewdlam, Commented Jun 7, 2022 at 12:29

Krisna Pranav · Accepted Answer · 2022-06-07 12:25:51Z

1

Using Asyncio:

import asyncio
import time
def background(f):
    def wrapped(*args, **kwargs):
        return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)

    return wrapped

@background
def your_function(argument):
    time.sleep(2)
    print('function finished for '+str(argument))


for i in range(10):
    your_function(i)


print('loop finished')

answered Jun 7, 2022 at 12:25

Krisna Pranav

792 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Artem Dumanov Over a year ago

Does that code utilize all the CPU cores even if called from the sync code initially?

patkru · Accepted Answer · 2022-06-07 12:39:51Z

1

Using multiprocessing:

from multiprocessing import Pool

L1 = [1,2,3]
L2 = [3,4,5]
L3 = [5,6,7]

def f(li):
    return [x * 2 for x in li]

if __name__ == '__main__':
    with Pool(4) as pool:
        print(pool.map(f, [L1, L2, L3]))

answered Jun 7, 2022 at 12:39

patkru

1301 silver badge7 bronze badges

Collectives™ on Stack Overflow

How to parallelize a for loop in Python?

2 Answers 2

Using Asyncio:

1 Comment

Using multiprocessing:

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Using Asyncio:

1 Comment

Using multiprocessing:

Comments

Your Answer

Sign up or log in

Post as a guest

Related