numpy dtype error - (structured array creation)

Question

I am having some trouble understanding why the following does not work:

np.dtype(dict(names="10", formats=np.float64))

I have been struggling with this because I would like to get the recfunctions function in numpy to work, but due to issues with the numpy.dtype, I haven't been successful. This is the error I am receiving at the moment:

dtype = np.dtype(dict(names=names, formats=formats))
ValueError: all items in the dictionary must have the same length.

I want to get a data structure that will contain a type of record array with multiple columns of data within each assigned field - similar to a dictionary where each value is a 2d array or several columns of data. Typically the data may end up being ~6 columns, ~2000 rows for each key or record, with ~200 records.

Here is what I have tried in a complete script: (although still giving the same error)

import numpy as np
from numpy.lib import recfunctions


# Just function to make random data
def make_data(i, j):
    # some arbitrary function to show that the number of columns may change, but rows stay the same length
    if i%3==0:
        data = np.array([[i for i in range(0,1150)]*t for t in range(0,3)])
    else:
        data = np.array([[i for i in range(0,1150)]*t for t in range(0,6)])
    return data

def data_struct(low_ij, high_ij):

    """
    Data Structure to contain several columns of data for different combined values between "low ij" and "high ij"

    Key: "(i, j)"
    Value: numpy ndarray (multidimensional)
    """

    for i in range(0,low_ij+1):
        for j in range(0,high_ij+1):
            # Get rid of some of the combinations
            # (unimportant)
            if(i<low_ij and j<low_ij):
                break
            elif(i<j):
                break

            # Combinations of interest to create structure
            else:
                names = str(i)+str(j)
                formats = np.float64
                data = np.array(make_data(i, j))
                try:
                    data_struct = recfunctions.append_fields(base=data_struct, names=names, data=data, dtypes=formats)
                # First loop will assign data_struct using this exception,
                # then proceed to use the try statement to add on the rest of the data
                except UnboundLocalError:
                    dtype = np.dtype(dict(names=names, formats=formats))
                    data_struct = np.array(data, dtype=dtype)

    return data_struct

hpaulj · Accepted Answer · 2015-01-14 22:46:06Z

Looks like you are trying to construct a structured array something like:

In [152]: names=['1','2','3','4']
In [153]: formats=[(float,2),(float,3),(float,2),(float,3)]
In [154]: dt=np.dtype({'names':names, 'formats':formats})
In [156]: ds=np.zeros(5, dtype=dt)

In [157]: ds
Out[157]: 
array([([0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0], [0.0, 0.0, 0.0]),
       ([0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0], [0.0, 0.0, 0.0]),
       ([0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0], [0.0, 0.0, 0.0]),
       ([0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0], [0.0, 0.0, 0.0]),
       ([0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0], [0.0, 0.0, 0.0])], 
      dtype=[('1', '<f8', (2,)), ('2', '<f8', (3,)), ('3', '<f8', (2,)), 
           ('4', '<f8', (3,))])
In [159]: ds['1']=np.arange(10).reshape(5,2)
In [160]: ds['2']=np.arange(15).reshape(5,3)

In other words, multiple fields, each with a different number of 'columns' (shape).

Here I create initialize the whole array, and then fill the fields individually. That seems to be the most straight forward way of creating complex structured arrays.

You are trying to build such an array incrementally, starting with one field, and adding new ones with recfunctions.append_fields

In [162]: i=1; 
   ds2 = np.array(np.arange(5),dtype=np.dtype({'names':[str(i)],'formats':[float]}))
In [164]: i+=1;
   ds2=recfunctions.append_fields(base=ds2,names=str(i),dtypes=float,
      data=np.arange(5), usemask=False,asrecarray=False)
In [165]: i+=1;
   ds2=recfunctions.append_fields(base=ds2,names=str(i),dtypes=float,
      data=np.arange(5), usemask=False,asrecarray=False)

In [166]: ds2
Out[166]: 
array(data = [(0.0, 0.0, 0.0) (1.0, 1.0, 1.0) (2.0, 2.0, 2.0) 
    (3.0, 3.0, 3.0) (4.0, 4.0, 4.0)], 
    dtype = [('1', '<f8'), ('2', '<f8'), ('3', '<f8')])

This works when the appended fields all have 1 'column'. With the masking they can even have different numbers of 'rows'. But when I try to vary the internal shape it has problems appending the field. marge_arrays isn't any more successful.

Even if we can get the incremental recfunctions approach to work, it probably will be slower than the initialize-and-fill approach. Even if you don't know the shape of each of the fields at the start, you could collect them all in a dictionary, and assemble the array from that. This kind of structured array isn't any more compact or efficient than a dictionary. It just makes certain styles of data access (across fields) more convenient.

gg349 · Accepted Answer · 2015-01-14 16:38:45Z

1

You must pass a list of values and a list of formats, not a single value and a single format. if you debug the issue, you will find that

type(names)   # result is <type 'str'>
type(formats) # result is <type 'type'>

It then happens that the dict is initialized to

{'formats': numpy.float64, 'names': '30'}

while each of formats and names should be a list.

EDIT: moreover, notice that formats should be a list of strings, like ['float64','u8'], etc..

edited Jan 14, 2015 at 16:38

answered Jan 14, 2015 at 16:31

gg349

22.8k5 gold badges58 silver badges65 bronze badges

Collectives™ on Stack Overflow

numpy dtype error - (structured array creation)

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related