Parallel programming in python (nested loop)

Question

I have the following code and since it is very slow for larger l i am asking myself if there is an easy possibility to parallelize this loop. I also tried to parallize it by my own, but because I never did it before I am not able to manage it. I would be happy if you can help me out

print('Create 2.0L%.3frec%.3f.npy' % (l, r))
for x1 in range(a**l):
    for x2 in range(a**l):
        for x3 in range(a**l):
            f11 = 0
            if numpy.ndarray.sum(numpy.absolute(numpy.subtract(ws[x1:x1+1], ws[x2:x2+1]))) == 0:
                f11 += 1
            if numpy.ndarray.sum(numpy.absolute(numpy.subtract(ws[x1:x1+1], ws[x3:x3+1]))) == 0:
                f11 += 1
            re[x1][x2][x3] = 1.0*r/(a**l-2)*(numpy.product(numpy.absolute(numpy.subtract((2*ws[x1:x1+1]+ws[x2:x2+1]+ws[x3:x3+1]), 2)))-f11)

            cv1 = numpy.ndarray.sum(numpy.absolute(numpy.subtract(ws[x1:x1+1], ws[x2:x2+1])))
            cv2 = numpy.ndarray.sum(numpy.absolute(numpy.subtract(ws[x1:x1+1], ws[x3:x3+1])))
            c2 = 0
            if cv1 == 0:
                c2 += 1
            if cv2 == 0:
                c2 += 1
            c2 *= 1.0*(1-r)/2
            re[x1][x2][x3] += c2
numpy.save('2.0L%.3frec%.3f' % (l, r), re)
print('Saved 2.0L%.3frec%.3f.npy' % (l, r))

So since all entries of re are independent of the others, there should be a way. I guess I would be helped if I know a solution how to parllize a python program like:

for x1 in range(a):
    for x2 in range(a):
        for x3 in range(a):
            re[x1][x2][x3] = 5*3

You need to give more background. What is the format of your data? Can you explain what the loop is supposed to do? — flyingmeatball
– flyingmeatball, Commented Apr 26, 2016 at 18:22
Won't ws[x1:x1+1] essentially mean ws[x1] and similarly for others? — Divakar
– Divakar, Commented Apr 27, 2016 at 3:47
@Divakar: You are right.. I just made it more complicated... — HighwayJohn
– HighwayJohn, Commented Apr 27, 2016 at 12:32

Forzaa · Accepted Answer · 2016-04-27 10:28:07Z

2

I don't fully understand what you are trying to calculate, but I'll give it a shot. For your latter question, you can do this as follows:

re = np.empty([a]*3) 
x = np.indices([a]*3)
re[x] = 5*3

You can vectorize your code as follows:

x = np.indices([a**l]*3)
cv1 = (ws[x[0]] == ws[x[1]]).astype(float)
cv2 = (ws[x[0]] == ws[x[2]]).astype(float)
f11 = cv1 + cv2
re = 1.0*r/(a**l-2)*(np.absolute(2*ws[x[0]]+ws[x[1]]+ws[x[2]]-2)-f11)    
re += f11*1.0*(1-r)/2

(I removed superfluous sums and products, and used the == operator to check for equal values, and used that c2 and f11 are the same thing). In theory this should do the same as your code.

edited Apr 27, 2016 at 10:28

answered Apr 27, 2016 at 8:57

Forzaa

1,5654 gold badges15 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

HighwayJohn Over a year ago

That looks really nice, I will try it out soon :)

HighwayJohn Over a year ago

Traceback (most recent call last): File "/2.3.1.py", line 113, in <module> wp[:,0] = numpy.tensordot(numpy.tensordot(re, wp[:,0], axes=1), wp[:,0], axes=1) File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/numpy/core/numeric.py", line 1246, in tensordot if as_[axes_a[k]] != bs[axes_b[k]]: IndexError: tuple index out of range I am getting this error now. Apparently the re tensor has now a different size?

Forzaa Over a year ago

It's shape should still be the same as before, namely (a**l,a**l,a**l).

HighwayJohn Over a year ago

Does it have to be re[x] ?

Forzaa Over a year ago

I tried it on an example a=2,l=3, r=0.3 and ws = np.ones(a**l). There it returns the same answer as your code (namely all entries of re are 0.7).

|

Collectives™ on Stack Overflow

Parallel programming in python (nested loop)

1 Answer 1

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related