Parallel array manipulations in numpy

Question

I have a code in which I need to handle some big numpy arrays. For example I have a 3D array A and I need to construct another 3d array B using the elements of A. However all the elements of B are independent of each other. Example:

for i in np.arange(Nx):
  for j in np.arange(Ny):
   for k in np.arange(Nz):
       B[i][j][k] = A[i+1][j][k]*np.sqrt(A[i][j-1][k-1])

So it will speed up immensely if I can construct the B array parallely. What is the simplest way to do this in python?

I also have similar matrix operations like normalizing each row of a 2D array. Example

for i in np.arange(Nx):
   f[i,:] = f[i,:]/np.linalg.norm(f[i,:])

This will also speed up if it runs parallely for each row. How can it be done?

possible duplicate of Numpy/Python: Array iteration without for-loop — YXD
– YXD, Commented Feb 20, 2013 at 17:17

YXD · Accepted Answer · 2013-02-20 17:31:39Z

2

You should look into Numpy's roll function. I think this is equivalent to your first block of code (though you need to decide what happens at the edges - roll "wraps around"):

B = np.roll(A,1,axis=0) * np.sqrt(np.roll(np.roll(A,-1,axis=1),-1,axis=2))

Another fairly horrible one-liner for your second case is:

f /= np.sqrt(np.sum(f**2, axis=1))[...,np.newaxis]

Explanation of this line:

We are first going to calculate the norm of each row. Let's

f = np.random.rand(5,6)

Square each element of f

f**2

Sum the squares along axis 1, which "flattens" out that axis.

np.sum(f**2, axis=1)

Take the square root of the sum of the squares.

np.sqrt(np.sum(f**2, axis=1))

We now have the norm of each row.

To divide each original row of f by this correctly we need to make use of the Numpy broadcasting rules to effectively add a dimension:

np.sqrt(np.sum(f**2, axis=1))[...,np.newaxis]

And finally we calculate our result

f /= np.sqrt(np.sum(f**2, axis=1))[...,np.newaxis]

edited Feb 20, 2013 at 17:31

answered Feb 20, 2013 at 17:21

YXD

32.6k15 gold badges79 silver badges117 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

lovespeed Over a year ago

In the actual code, the edges are well taken care of, but the construction of the B matrix is far more complicated, so the numpy.roll function makes things very messy and susceptible to errors. Isn't there a simple way to run different iterations of the for-loop independently?

YXD Over a year ago

Susceptible to errors in terms of programming errors or in terms of handling edges?

lovespeed Over a year ago

Programming errors. Also, I was looking for a generic way to parallelize iterations of for loops where each loop is independent.

YXD Over a year ago

I understand, but for me at least the generic way is to "line up" the elements with something like roll, maybe creating some intermediate arrays if it's getting messy. In terms of the full range of options to parallelize your code, you should take a look at this

Jaime · Accepted Answer · 2013-02-20 22:25:07Z

0

If you are taking good care of the edges, the standard way of going about your first vectorization would be something like this:

B = np.zeros(A.shape)
B[:-1, 1:, 1:] = A[1:, 1:, 1:] * np.sqrt(A[:-1, :-1, :-1])

You would then need to fill B[-1, :, :], B[:, 0, :] and B[:, :, 0] with appropriate values.

Extending this to other indices should be pretty straightforward.

answered Feb 20, 2013 at 22:25

Jaime

67.7k19 gold badges128 silver badges164 bronze badges

Comments

Jerry101 · Accepted Answer · 2016-08-27 00:00:32Z

0

To perform parallel processing in numpy, you should look at mpi4py. It's an MPI binding for Python. It allows distributed processing.

edited Aug 27, 2016 at 0:00

Jerry101

13.7k7 gold badges51 silver badges68 bronze badges

answered Feb 20, 2013 at 23:32

Charles Brunet

23.4k25 gold badges86 silver badges127 bronze badges

Collectives™ on Stack Overflow

Parallel array manipulations in numpy

3 Answers 3

4 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related