Pass numpy array of list of integers in Cython method from Python

Question

I would like to pass the following array of list of integers (i.e., it's not an two dimensional array) to the Cython method from python code.

Python Sample Code:

import numpy as np
import result
a = np.array([[1], [2,3]])
process_result(a)

The output of a is array([list([1]), list([2, 3])], dtype=object)

Cython Sample Code:

def process_result(int[:,:] a):
    pass

The above code gives the following error:

ValueError: Buffer has wrong number of dimensions (expected 2, got 1)

I tried to pass a simple array instead of numpy I got the following error

a = [[1], [2,3]]
process_result(a)

TypeError: a bytes-like object is required, not 'list'

Kindly assist me how to pass the value of a into the Cython method process_result and whats the exact datatype needs to use to receive this value in Cython method.

How do you intend to process this array, especially the lists? At least in Python, there isn't much of an advantage to creating such array. A list of those lists is often easier, and faster, to process. Remember such a list, and array, just contains pointers to the lists, which reside elsewhere in memory. — hpaulj
– hpaulj, Commented Jun 12, 2018 at 5:01
Look at a. Is that a 2d array, or 1d? Count the [] or look at its shape. If that's what you want to pass to cython, why are you using [:,:] instead of [:]? You may need more experience with numpy in interpreted Python before moving on to cython. — hpaulj
– hpaulj, Commented Jun 12, 2018 at 5:18
@hpaulj - using of 1d [] is not an issue, but whats the datatype that I need to use. def process_result( ??? [:] a): — Sreedharan
– Sreedharan, Commented Jun 12, 2018 at 5:22
<void>[:] might work. cython.readthedocs.io/en/latest/src/userguide/…. a itemsize is 8, the items being pointers to the lists elsewhere in memory. Compare np.frombuffer(a,'int64') with id(a[0]). So any 8 byte dtype should be acceptable. — hpaulj
– hpaulj, Commented Jun 12, 2018 at 5:47

DavidW · Accepted Answer · 2018-06-12 16:01:18Z

I think you're using the wrong data-type. Instead of a numpy array of lists, you should be using a list of numpy arrays. There is very little benefit of using numpy arrays of Python objects (such as lists) - unlike numeric types they aren't stored particulatly efficiently, they aren't quick to do calculations on, and you can't accelerate them in Cython. Therefore the outermost level may as well be a normal Python list.

However, the inner levels all look to be homogenous arrays of integers, and so would be ideal candidates for Numpy arrays (especially if you want to process them in Cython).

Therefore, build your list as:

a = [ np.array([1],dtype=np.int), np.array([2,3],dtype=np.int) ]

(Or use tolist on a numpy array)

For your function you can define it like:

def process_result(list a):
    cdef int[:] item
    for item in a:
        #operations on the inner arrays are fast!
        pass

Here I've assumed that you most likely want to iterate over the list. Note that there's pretty little benefit in typing a to be list, so you could just leave it untyped (to accept any Python object) and then you could pass it other iterables too, like your original numpy array.

Sreedharan · Accepted Answer · 2018-06-12 12:00:03Z

0

Convert the array of list of integer to list of object (i.e., list of list of integers - its not a two dimensional array)

Python Code:

import numpy as np
import result
a = np.array([[1], [2,3]]).tolist()
process_result(a)

The output of a is [[1], [2,3]]

Cython Sample Code:

def process_result(list a):
    pass

Change the int[:, :] to list. It works fine.

Note: If anyone know the optimal answer kindly post it, It will be helpful.

answered Jun 12, 2018 at 12:00

Sreedharan

831 gold badge3 silver badges9 bronze badges

Collectives™ on Stack Overflow

Pass numpy array of list of integers in Cython method from Python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related