Is there any possibility to parallelize the following code in python? I was wondering how to convert this code with map and lambda functions..
values = (1,2,3,4,5 )
def op(x,y):
return x+y
[(i, j, op(i, j))
for i in values
for j in values
if i is not j]
You can parallelize the function op with multiprocessing and map:
from multiprocessing.dummy import Pool as ThreadPool
from itertools import permutations
pool = ThreadPool(4) # Number of threads
values = (1,2,3,4,5)
aux_val = [(i, j) for i,j in permutations(values,2)]
def op(tupx):
result = (tupx[0], tupx[1], tupx[0] + tupx[1])
return result
results = pool.map(op, aux_val)
Check this out:
from itertools import permutations
values = (1,2,3,4,5 )
[(i, j, i+j) for i, j in permutations(values, 2)]
It's in python's stdlib.
If you want run in parallel, check out this using python3:
import multiprocessing
from itertools import permutations
values = [1, 2, 3, 4, 5]
l = permutations(values, 2)
def f(x):
return x[0], x[1], x[0] + x[1]
with multiprocessing.Pool(5) as p:
data = p.map(f, l)
You can use asyncio. (Documentation can be found [here][1]). It is used as a foundation for multiple Python asynchronous frameworks that provide high-performance network and web-servers, database connection libraries, distributed task queues, etc. Plus it has both high-level and low-level APIs to accomodate any kind of problem.
import asyncio
def background(f):
def wrapped(*args, **kwargs):
return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)
return wrapped
@background
def your_function(argument):
#code
Now this function will be run in parallel whenever called without putting main program into wait state. You can use it to parallelize for loop as well. When called for a for loop, though loop is sequential but every iteration runs in parallel to the main program as soon as interpreter gets there.
For example for your specific case, you can do:
@background
def op(x,y):
time.sleep(1) # Added sleep to demonstrate parallelization
print(f"function called for {x=}, {y=}\n", end='')
return x, y, x+y
values = [1,2,3,4,5]
loop = asyncio.get_event_loop() # Have a new event loop
looper = asyncio.gather(*[op(i, j)
for i in values
for j in values
if i is not j]) # Run the loop
results = loop.run_until_complete(looper) # Wait until finish
print('Loop has finished and results are gathered!')
results
This produces following output:
function called for x=1, y=5
function called for x=1, y=2
function called for x=2, y=3
function called for x=2, y=1
function called for x=1, y=3
function called for x=1, y=4
function called for x=2, y=4
function called for x=2, y=5
function called for x=3, y=1
function called for x=3, y=4
function called for x=4, y=1
function called for x=3, y=5
function called for x=3, y=2
function called for x=4, y=5
function called for x=4, y=3
function called for x=4, y=2
function called for x=5, y=3
function called for x=5, y=4
function called for x=5, y=2
function called for x=5, y=1
Loop has finished and results are gathered!
[(1, 2, 3), (1, 3, 4), (1, 4, 5), (1, 5, 6), (2, 1, 3), (2, 3, 5), (2, 4, 6), (2, 5, 7), (3, 1, 4), (3, 2, 5), (3, 4, 7), (3, 5, 8), (4, 1, 5), (4, 2, 6), (4, 3, 7), (4, 5, 9), (5, 1, 6), (5, 2, 7), (5, 3, 8), (5, 4, 9)]
