Sharing a ctypes numpy array without lock when using multiprocessing

Question

I have a large array (~500k rows x 9 columns) which I would like to share when running a number of parallel processes using Python's multiprocessing module. I am using this SO answer to create my shared array and I understand from this SO answer that the array is locked. However in my case as I never concurrently write to the same row then a lock is superfluous and increases processing time.

When I specify lock=False however I get an error.

My code is this:

shared_array_base = multiprocessing.Array(ctypes.c_double, 90, lock=False)
shared_array = np.ctypeslib.as_array(shared_array_base.get_obj())
shared_array = shared_array.reshape(-1, 9)

And the error is this:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-15-d89681d70c37> in <module>()
      1 shared_array_base = multiprocessing.Array(ctypes.c_double, len(np.unique(value)) * 9, lock=False)
----> 2 shared_array = np.ctypeslib.as_array(shared_array_base.get_obj())
      3 shared_array = shared_array.reshape(-1, 9)

AttributeError: 'c_double_Array_4314834' object has no attribute 'get_obj'

My question is how can I share a numpy array that is not locked each time I write to it?

Community · Accepted Answer · 2017-05-23 11:54:22Z

3

Found the answer here thanks to HYRY

Stating lock=True returns a wrapped object:

multiprocessing.sharedctypes.SynchronizedArray

When lock=False returns a raw array which does not have the .get_obj() method

multiprocessing.sharedctypes.c_double_Array_10

Therefore code to create an unlocked array is this:

shared_array_base = multiprocessing.Array(ctypes.c_double, 90, lock=False)
shared_array = np.ctypeslib.as_array(shared_array_base)
shared_array = shared_array.reshape(-1, 9)

edited May 23, 2017 at 11:54

CommunityBot

11 silver badge

answered Aug 8, 2016 at 8:56

kungphil

1,8592 gold badges19 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Matthias Fripp Over a year ago

you could also try shared_array = np.frombuffer(shared_array_base).reshape(-1, 9)

Collectives™ on Stack Overflow

Sharing a ctypes numpy array without lock when using multiprocessing

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related