Numpy matrix of strings converted to shared array of strings creates type mismatch

Question

I am experimenting with multiprocessing in Python, however, I am having trouble with creating some shared memory. Take the following example that illustrates my problem:

In reference to the following (slightly different as he uses a matrix full of floats, but same principle), I want to convert a numpy matrix of strings into a shared memory space for processes to use. I have the following:

from ctypes import c_wchar_p
import numpy as np
from multiprocessing.sharedctypes import Array

input_array = np.array([['Red', 'Green', 'Blue', 'Yellow'],
                        ['Purple', 'Orange', 'Cyan', 'Pink']]).T

shared_memory = Array(c_wchar_p, input_array.size, lock=False) # Equivalent to just using a RawArray
np_wrapper = np.frombuffer(shared_memory, dtype='<U1').reshape(input_array.shape)
np.copyto(np_wrapper, input_array)
print(np_wrapper)

However, the np_wrapper only has the first character of each string:

[['R' 'P']
 ['G' 'O']
 ['B' 'C']
 ['Y' 'P']]

Things I have tried to rectify the problem:

I tried changing the dtype of the frombuffer function from <U1 to <U6, which is the dtype of the input_array. However, it throws the following exception:

ValueError: buffer size must be a multiple of element size

I tried using a dtype of int64 with the frombuffer function because my shared_memory array is of type c_wchar_p (i.e. string pointers) and I am on a 64-bit Windows 10 system. However, it throws the following exception:

ValueError: cannot reshape array of size 4 into shape (4,2)

I am extremely confused why my typing is wrong here. Does anyone have any insight on how to fix this problem?

In input_array, the strings are represented as U6 (24 bytes) items, all packed in the array's data_buffer. They don't reference strings elsewhere in memory, as they would in a list (or object dtype array). Check the input_array.itemsize. — hpaulj
– hpaulj, Commented Jun 13, 2018 at 20:21

hpaulj · Accepted Answer · 2018-06-13 20:28:19Z

1

It may help to understand what this array of strings contains:

In [643]: input_array = np.array([['Red', 'Green', 'Blue', 'Yellow'],
     ...:                         ['Purple', 'Orange', 'Cyan', 'Pink']]).T
     ...: 
     ...:                         
In [644]: input_array.size
Out[644]: 8
In [645]: input_array.itemsize
Out[645]: 24
In [646]: input_array.nbytes
Out[646]: 192

Because it is a transpose, the shape and strides differ from the input array, but the strings are in the original order.

In [647]: input_array.__array_interface__
Out[647]: 
{'data': (139792902236880, False),
 'strides': (24, 96),
 'descr': [('', '<U6')],
 'typestr': '<U6',
 'shape': (4, 2),
 'version': 3}

My guess is that Array should be defined with nbytes rather than size.

answered Jun 13, 2018 at 20:28

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Dannon Over a year ago

Yeah, I already figured out the solution, give me a second to post my solution (literally typing it as we speak)

Dannon · Accepted Answer · 2018-06-13 21:32:30Z

Preface

Before I detail my solution, I want to preface my answer with some helpful information. The function memoryview() in python proved to be extremely useful in getting the full picture. For example, run the following after specify the dtype of input_array as dtype='S6' (b/c less bytes to examine):

print(bytes(memoryview(input_array)))

Then following results are yielded:

b'Red\x00\x00\x00PurpleGreen\x00OrangeBlue\x00\x00Cyan\x00\x00YellowPink\x00\x00'

We can see by the following output that each entry in the input_array has a length of 6 bytes and is laid out in a contiguous block of memory. This tells us that our Numpy array is not just 8 pointers to strings out in memory.

Turning back to when the dtype was not specified, @hpaulj additionally provided even more helpful insight. With having read the dtype documentation, our array has type <U6, which translates to as follows:

<  -- Little-Endian (b/c I am on an Intel-based system)
U  -- Unicode String (Remember with 4 bytes per Unicode String)
6  -- 24 bytes per entry in the array.

Solution

TLDR; Here is the solution:

from ctypes import c_char
import numpy as np
from multiprocessing.sharedctypes import Array

input_array = np.array([['Red', 'Green', 'Blue', 'Yellow'],
                        ['Purple', 'Orange', 'Cyan', 'Pink']]).T

shared_memory = Array(c_char, input_array.size * input_array.itemsize, lock=False)
np_wrapper = np.frombuffer(shared_memory, dtype=input_array.dtype).reshape(input_array.shape)
np.copyto(np_wrapper, input_array)

print(shared_memory[:])
print(np_wrapper)

Solution Explanation:

The first incorrect aspect of the initial code was the typing information for the initial shared_memory array. Our Numpy array is not an array of pointers, but rather 8 strings compacted contiguously next to each other (with some padding dictated by the longest element). Therefore, using the type c_wchar_p (i.e. a string pointer) was not correct. I chose c_char over c_wchar because c_char is guaranteed to be one byte, where as c_wchar is not (see documentation for further details).

Next, one needs to specify the entire size of the shared memory. Because I chose c_char as my type, I will be specifying the number of bytes. The length is given by the following:

There are 8 elements (input_array.size) with each element contain 24 bytes (input_array.itemsize). Therefore, there are 8 * 24 = 192 bytes total in our shared memory.

Finally, when using the frombuffer function in Numpy, be sure to specify the correct dtype because this is how Numpy will be dividing up and interpreting the arbitrary bytes coming in. Just simply use the same dtype of the input_array to complete the translation.

And finally, once the copyto has commenced, the shared_memory will have been successfully configured!

Collectives™ on Stack Overflow

Numpy matrix of strings converted to shared array of strings creates type mismatch

2 Answers 2

1 Comment

Preface

Solution

TLDR; Here is the solution:

Solution Explanation:

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Preface

Solution

TLDR; Here is the solution:

Solution Explanation:

Comments

Your Answer

Sign up or log in

Post as a guest

Related