Shared memory in mpi4py

Question

I use a MPI (mpi4py) script (on a single node), which works with a very large object. In order to let all processes have access to the object, I distribute it through comm.bcast(). This copies the object to all processes and consumes a lot of memory, especially during the copying process. Therefore, I would like to share something like a pointer instead of the object itself. I found some features in memoryview useful to boost work with the object inside a process. Also the object's real memory address is accessible through the memoryview object string representation and can be distributed like this:

from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()

if rank:
    content_pointer = comm.bcast(root = 0)
    print(rank, content_pointer)
else:
    content = ''.join(['a' for i in range(100000000)]).encode()
    mv = memoryview(content)
    print(mv)
    comm.bcast(str(mv).split()[-1][: -1], root = 0)

This prints:

<memory at 0x7f362a405048>
1 0x7f362a405048
2 0x7f362a405048
...

That's why I believe that there must be a way to reconstitute the object in another process. However, I cannot find a clue in the documentation about how to do it.

In short, my question is: Is it possible to share an object between processes on the same node in mpi4py?

could you accept the new MPI 3.0 answer of JobJob as correct? — Robin De Schepper
– Robin De Schepper, Commented Jan 21, 2021 at 7:56

JobJob · Accepted Answer · 2018-02-08 05:30:37Z

20

Here's a simple example of shared memory using MPI, very slightly modified from https://groups.google.com/d/msg/mpi4py/Fme1n9niNwQ/lk3VJ54WAQAJ

You can run it with: mpirun -n 2 python3 shared_memory_test.py (assuming you saved it as shared_memory_test.py)

from mpi4py import MPI 
import numpy as np 

comm = MPI.COMM_WORLD 

# create a shared array of size 1000 elements of type double
size = 1000 
itemsize = MPI.DOUBLE.Get_size() 
if comm.Get_rank() == 0: 
    nbytes = size * itemsize 
else: 
    nbytes = 0

# on rank 0, create the shared block
# on rank 1 get a handle to it (known as a window in MPI speak)
win = MPI.Win.Allocate_shared(nbytes, itemsize, comm=comm) 

# create a numpy array whose data points to the shared mem
buf, itemsize = win.Shared_query(0) 
assert itemsize == MPI.DOUBLE.Get_size() 
ary = np.ndarray(buffer=buf, dtype='d', shape=(size,)) 

# in process rank 1:
# write the numbers 0.0,1.0,..,4.0 to the first 5 elements of the array
if comm.rank == 1: 
    ary[:5] = np.arange(5)

# wait in process rank 0 until process 1 has written to the array
comm.Barrier() 

# check that the array is actually shared and process 0 can see
# the changes made in the array by process 1
if comm.rank == 0: 
    print(ary[:10])

Should output this (printed from process rank 0):

[0. 1. 2. 3. 4. 0. 0. 0. 0. 0.]

edited Feb 8, 2018 at 5:30

answered Feb 7, 2018 at 13:32

JobJob

4,1973 gold badges36 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Zézouille Over a year ago

What about a multi-dimensional array, let's say np.zeros((50, 50, 50))?

JobJob Over a year ago

@Zézouille for a multi-dim array it's similar, you'd just make (approximately) the following changes: size = 1000 to shape = (50,50,50), size = np.prod(shape) and this line becomes ary = np.ndarray(buffer=buf, dtype='d', shape=shape) # instead of shape=(size,) and to get this particular example to run you'd also want to change ary[:5] = np.arange(5) to ary[:5, 1, 1] = np.arange(5) and print(ary[:10]) to print(ary[:10, 1, 1])

dreamcrash Over a year ago

@JobJob Do you have any idea if that would be possible with a dictionary?

haraldkl · Accepted Answer · 2015-09-09 21:43:12Z

1

I don't really know much about mpi4py, but this should not be possible from the MPI point of view. MPI stands for Message Passing Interface, which means exactly that: pass messages around between processes. You could try and use MPI One-sided communication to resemble something like a globally accessible memory, but otherwise process memory is unavailable to other processes.

If you need to rely on a large block of shared Memory, you need to utilize something like OpenMP or threads, which you absolutely could use on a single node. A hybrid parallelization with MPI and some shared memory parallelization would allow you to have one shared memory block per node, but still the option to utilize many nodes.

edited Sep 9, 2015 at 21:43

answered Sep 9, 2015 at 21:34

haraldkl

3,83928 silver badges44 bronze badges

2 Comments

Ben Thompson Over a year ago

This answer is incorrect in light of MPI 3.0 shared memory (software.intel.com/en-us/articles/…)

haraldkl Over a year ago

@BenThompson OK, you want me to delete it or change the answer? I have 0 experience with this new feature...

Collectives™ on Stack Overflow

Shared memory in mpi4py

2 Answers 2

3 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related