python multiprocessing shared memory array doesn't allow 2-d array

Question

I would like to store a 2-d array as shared memory array, so each process created by multiprocessing module can access the 2-d array. But it seems only vector is allowed. If I change the shape in the following example from (6,1) to (2,3), the example won't work and it says "TypeError: only length-1 arrays can be converted to Python scalars"

from multiprocessing import Process, Value, Array
import numpy as np

def f(n,a):
    a[:] = np.ones(shape=(6,1), dtype=float)[:]

if __name__ == '__main__':
    tmp =  np.zeros(shape=(6,1), dtype=float)
    print tmp

    arr = Array('f', tmp)

    p1 = Process(target=f, args=(1,arr))
    p1.start()
    p1.join()

    print 'done'
    print arr[:]

mgilson · Accepted Answer · 2014-04-30 14:53:16Z

3

This is because mutiprocessing.Array is a wrapper around python's array type, not numpy.ndarray and python's array type doesn't support multiple dimensionality. Specifically, look at the documentation for the initializer:

A new array whose items are restricted by typecode, and initialized from the optional initializer value, which must be a list, string, or iterable over elements of the appropriate type.

You've got an iterable (a multi-dimensional array), but it yields views/arrays, not elements of the appropriate type.

answered Apr 30, 2014 at 14:53

mgilson

312k70 gold badges656 silver badges722 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

read Read Over a year ago

Is there a quick way to convert numpy.ndarray to array?

mgilson Over a year ago

@readRead -- Only 1D numpy arrays can be converted to array -- and it makes a copy (not a view)

read Read Over a year ago

Thanks. I plan to do some matrix manipulation on the shared array in each process. If I just store 1-d array, it's not very convenient. How can I shared a 2-d numpy array across processes and avoid copy? maybe use something other than multiprocessing.Array?

mgilson Over a year ago

I'm not entirely convinced it's possible. As far as I see it, the best you can do is create a wrapper around the Array/array which provides a nicer interface.

read Read Over a year ago

If I create a wrapper around Array/array, which converts 1-d array to 2-d numpy.ndarray, does this involves extra copy?

|

Collectives™ on Stack Overflow

python multiprocessing shared memory array doesn't allow 2-d array

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related