Moving results from parallel calculation to a final array in Python

Question

I have a parallelized python code that calculates the value of x[(1,i)] for o[(i)] with i in range (0, npt). As x[(1,i)] = x1[(i)] * x2[(i)], I calculate x1[(i)] and x2[(i)] separately and then multiply them together to get x[(1,i)].

import cmath, csv, sys, math, re
import numpy as np
import multiprocessing as mp

x1 = np.zeros(npt ,dtype=float)
x2 = np.zeros(npt ,dtype=float)

def chi2(i):
    print("\t wavelength", i+1," of ", npt)
    x1[(i)] = (some function of o[(i)]))

    for k in range(nk):
        for n in range(nb):
            for m in range(nb):
                for l in range(nb):
                        x2[(i)] = (x2[(i)] + (another function of o[(i)]))
    print(i,"x1:",x1[(i)])
    print(i,"x2:",x2[(i)])

    np.set_printoptions(precision=3)
    x[(1,i)] = x1[(i)] * x2[(i)]
    print(i,"x:",x[(1,i)])

    return x[(1,i)]

#-----------single process--------------
for i in range (npt):
    chi2(i)

#------------parallel processes-------------
#pool = mp.Pool(mp.cpu_count())
#pool.map(chi2,[i for i in range (npt)])
#pool.close() 
#-------------------------------------------

print("x:",x)

The output of the "parallel processes" code is as follows:

wavelength 1  of  6
wavelength 2  of  6
wavelength 3  of  6
wavelength 4  of  6
wavelength 6  of  6
wavelength 5  of  6
1 x1: 13.064431907056434
1 x2: -1.9906250877567282
1 x: -26.006385911476013
5 x1: 32.428696460232054
5 x2: -7.62814423558251
5 x: -247.37077397057408
2 x1: 16.848860458915905
2 x2: -2.8743277048490476
2 x: -48.429146412197625
3 x1: 21.301496841785333
0 x1: 9.893885346287407
3 x2: -4.053590423587782
0 x2: -1.339636506591729
3 x: -86.34754360594641
0 x: -13.254210001919562
4 x1: 26.47666689558421
4 x2: -5.606053928481043
4 x: -148.42962246307385
x: [[1030. 1130. 1230. 1330. 1430. 1530.]
 [   0.    0.    0.    0.    0.    0.]]

The parallelized code is able to obtain individual values of x1[(i)], x1[(i)], and x[(1,i)], but is unable to put these values into the full array print("x:",x).

Hence I tried using a single process, and the results are what I wanted:

         wavelength 1  of  6
0 x1: 9.893885346287407
0 x2: -1.339636506591729
0 x: -13.254210001919562
         wavelength 2  of  6
1 x1: 13.064431907056434
1 x2: -1.9906250877567282
1 x: -26.006385911476013
         wavelength 3  of  6
2 x1: 16.848860458915905
2 x2: -2.8743277048490476
2 x: -48.429146412197625
         wavelength 4  of  6
3 x1: 21.301496841785333
3 x2: -4.053590423587782
3 x: -86.34754360594641
         wavelength 5  of  6
4 x1: 26.47666689558421
4 x2: -5.606053928481043
4 x: -148.42962246307385
         wavelength 6  of  6
5 x1: 32.428696460232054
5 x2: -7.62814423558251
5 x: -247.37077397057408
x: [[1030.    1130.    1230.    1330.    1430.    1530.   ]
 [ -13.254  -26.006  -48.429  -86.348 -148.43  -247.371]]

Can anyone tell me what is preventing the parallelized code from getting values into x[(1,i)]?

When running with multiprocessing, global variables aren't shared between processes that way. Each process will have its own copy of the variable. That being said, you can use the return value of pool.map to get what you want — Tomer
– Tomer, Commented Aug 4, 2020 at 10:24
@Tomer do you mind writing an example for this? I'm not exactly sure how to use the return value of pool map as you described. Thanks! — Jacek
– Jacek, Commented Aug 4, 2020 at 10:34

Tomer · Accepted Answer · 2020-08-04 10:42:41Z

1

When running with multiprocessing, global variables aren't shared between processes that way. Each process will have its own copy of the variable.

Pool.map returns a list of the return values of each function for the list of the given inputs. You could do something like (instead of the pool.map line you use):

x[1] = pool.map(chi2,[i for i in range (npt)])

To use the return value of chi2 and insert it in the correct place in x. Note this uses the fact that you can assign a list of values into a column or row of a numpy array.

answered Aug 4, 2020 at 10:42

Tomer

6524 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Jacek Over a year ago

I see what you mean now. Originally I was worried that as the sequence of mp output is not in order, directing the results directly to x[1] will lead to wrongly placed values. Your last line helps clarify that. Many thanks!

Collectives™ on Stack Overflow

Moving results from parallel calculation to a final array in Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related