6

Suppose I have an __array_interface__ dictionary and I would like to create a numpy view of this data from the dictionary itself. For example:

buff = {'shape': (3, 3), 'data': (140546686381536, False), 'typestr': '<f8'}
view = np.array(buff, copy=False)

However, this does not work as np.array searches for either the buffer or array interface as attributes. The simple workaround could be the following:

class numpy_holder(object):
    pass

holder = numpy_holder()
holder.__array_interface__ = buff
view = np.array(holder, copy=False)

This seems a bit roundabout. Am I missing a straightforward way to do this?

9
  • Does your workaround work, or are you just speculating? Commented Sep 7, 2016 at 19:52
  • Why do you only have the the dict? Did you manually create it to expose an array? Commented Sep 7, 2016 at 19:55
  • I'd definitely just stick the dict on a wrapper object like you're doing here, though probably with an __init__ method. Commented Sep 7, 2016 at 19:59
  • @hpaulj No the above works fine. Commented Sep 7, 2016 at 21:11
  • @Dunes Im exposing a C++ Matrix class. Commented Sep 7, 2016 at 21:11

2 Answers 2

5

correction - with the right 'data' value your holder works in np.array:

np.array is definitely not going to work since it expects an iterable, some things like a list of lists, and parses the individual values.

There is a low level constructor, np.ndarray that takes a buffer parameter. And a np.frombuffer.

But my impression is that x.__array_interface__['data'][0] is a integer representation of the data buffer location, but not directly a pointer to the buffer. I've only used it to verify that a view shares the same databuffer, not to construct anything from it.

np.lib.stride_tricks.as_strided uses __array_interface__ for default stride and shape data, but gets the data from an array, not the __array_interface__ dictionary.

===========

An example of ndarray with a .data attribute:

In [303]: res
Out[303]: 
array([[ 0, 20, 50, 30],
       [ 0, 50, 50,  0],
       [ 0,  0, 75, 25]])
In [304]: res.__array_interface__
Out[304]: 
{'data': (178919136, False),
 'descr': [('', '<i4')],
 'shape': (3, 4),
 'strides': None,
 'typestr': '<i4',
 'version': 3}
In [305]: res.data
Out[305]: <memory at 0xb13ef72c>
In [306]: np.ndarray(buffer=res.data, shape=(4,3),dtype=int)
Out[306]: 
array([[ 0, 20, 50],
       [30,  0, 50],
       [50,  0,  0],
       [ 0, 75, 25]])
In [324]: np.frombuffer(res.data,dtype=int)
Out[324]: array([ 0, 20, 50, 30,  0, 50, 50,  0,  0,  0, 75, 25])

Both of these arrays are views.

OK, with your holder class, I can make the same thing, using this res.data as the data buffer. Your class creates an object exposing the array interface.

In [379]: holder=numpy_holder()
In [380]: buff={'data':res.data, 'shape':(4,3), 'typestr':'<i4'}
In [381]: holder.__array_interface__ = buff
In [382]: np.array(holder, copy=False)
Out[382]: 
array([[ 0, 20, 50],
       [30,  0, 50],
       [50,  0,  0],
       [ 0, 75, 25]])
Sign up to request clarification or add additional context in comments.

4 Comments

Re data[0] being representation of the buffer pointer. You're right, and may wish to include the numpy documentation __array_interface__ docs.scipy.org/doc/numpy/reference/arrays.interface.html
Also, using a view (rather than a copy) of the data pointed to by the pointer rather than Python's buffer interface could lead to memory leak issues. That is, the buffer interface allows other users to keep a buffer alive, should the original owner fall out of scope. (hope that makes sense :s)
What version of python/numpy is this? I get a type error for the buffer.
@Dunes Id like to point out that one of the very large upsides of this technique is that it is a view. This allows you to manipulate C++ data with python and NumPy. You cannot really get a memory leak, but you can get invalid access. This is usually handled by intelligent wrapping and smart pointers.
1

Here's another approach:

import numpy as np


def arr_from_ptr(pointer, typestr, shape, copy=False,
                 read_only_flag=False):
    """Generates numpy array from memory address
    https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.interface.html

    Parameters
    ----------
    pointer : int
        Memory address

    typestr : str
        A string providing the basic type of the homogenous array The
        basic string format consists of 3 parts: a character
        describing the byteorder of the data (<: little-endian, >:
        big-endian, |: not-relevant), a character code giving the
        basic type of the array, and an integer providing the number
        of bytes the type uses.

        The basic type character codes are:

        - t Bit field (following integer gives the number of bits in the bit field).
        - b Boolean (integer type where all values are only True or False)
        - i Integer
        - u Unsigned integer
        - f Floating point
        - c Complex floating point
        - m Timedelta
        - M Datetime
        - O Object (i.e. the memory contains a pointer to PyObject)
        - S String (fixed-length sequence of char)
        - U Unicode (fixed-length sequence of Py_UNICODE)
        - V Other (void * – each item is a fixed-size chunk of memory)

        See https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.interface.html#__array_interface__

    shape : tuple
        Shape of array.

    copy : bool
        Copy array.  Default False

    read_only_flag : bool
        Read only array.  Default False.
    """
    buff = {'data': (pointer, read_only_flag),
            'typestr': typestr,
            'shape': shape}

    class numpy_holder():
        pass

    holder = numpy_holder()
    holder.__array_interface__ = buff
    return np.array(holder, copy=copy)

Usage:

# create array
arr = np.ones(10)

# grab pointer from array
pointer, read_only_flag = arr.__array_interface__['data']

# constrct numpy array from an int pointer
arr_out = arr_from_ptr(pointer, '<f8', (10,))

# verify it's the same data
arr[0] = 0
assert np.allclose(arr, arr_out)

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.