Problem converting python list to np.array. Process is dropping sting type data

Question

My goal is to convert this list of strings to a Numpy Array.

I want to convert the first 2 columns to numerical data (integer)

list1 = [['380850', '625105', 'Dota 2'],
      ['354804', '846193', "PLAYERUNKNOWN'S BATTLEGROUNDS"],
      ['204354', '467109', 'Counter-Strike: Global Offensive']
     ]

dt = np.dtype('i,i,U')
cast_array = np.array([tuple(row) for row in sl], dtype=dt)
print(cast_array)

The result is ...

[OUT] [(380850, 625105, '') (354804, 846193, '') (204354, 467109, '')]

I am losing the string data. I am interested in

Understanding why the string data is getting dropped
Finding any solution that converts the first 2 columns to integer type in a numpy array

This answer gave me the approach but doesn't seem to work for strings

I agree, you can do it with Pandas. But all I was thinking there may be a performance improvement using the underlying numpy data structures and I am using this as a learning experience / test to see if it can be done this way — Axle Max
– Axle Max, Commented Oct 17, 2018 at 14:05
There'll certainly be some performance issues. But, as far as I'm aware, that is also the reason you need to specify the (maximum) string size: using a dynamic string size will lower performance (I guess it makes things more complicated to iterate over under the hood, in C). — 9769953
– 9769953, Commented Oct 17, 2018 at 14:08
Thank you. Your first comment had the answer. I have posted a solution in case it helps others in the future. — Axle Max
– Axle Max, Commented Oct 17, 2018 at 14:14
Arrghhh!! Now I can't slice the array by columns. Which was the main reason for converting in the first place. If I type cast_array[:, 0] I get an error because I have produced an array of tuples rather than a proper 2D np.array — Axle Max
– Axle Max, Commented Oct 17, 2018 at 14:30
A proper 2d array can't mix numeric a d string dtypes. What you get with dt is a structured array with fields, not columns. — hpaulj
– hpaulj, Commented Oct 17, 2018 at 15:12

Axle Max · Accepted Answer · 2018-10-17 14:13:32Z

0

Thanks to user: 9769953's comment above, this is the solution.

#when specifying strings you need to specify the length (derived from longest string in the list)
dtypestr = 'int, int, U' + str(max([len(i[2]) for i in plist1]))

cast_array = np.array([tuple(row) for row in plist1], dtype=dtypestr)

print(np.array(cast_array))

answered Oct 17, 2018 at 14:13

Axle Max

8253 gold badges17 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

B. M. · Accepted Answer · 2018-10-17 14:35:03Z

0

The simplest way to do that at high level is to use pandas, as said in comments, which will silently manage tricky problems :

In [64]: df=pd.DataFrame(list1)

In [65]: df2=df.apply(pd.to_numeric,errors='ignore')

In [66]: df2
Out[66]: 
        0       1                                 2
0  380850  625105                            Dota 2
1  354804  846193     PLAYERUNKNOWN'S BATTLEGROUNDS
2  204354  467109  Counter-Strike: Global Offensive

In [67]: df2.dtypes
Out[67]: 
0     int64
1     int64
2    object
dtype: object

df2.iloc[:,:2].values will be the numpy array, You can use all numpy accelerations on this part.

answered Oct 17, 2018 at 14:35

B. M.

18.7k2 gold badges40 silver badges56 bronze badges

Comments

Eric · Accepted Answer · 2018-10-17 16:15:01Z

0

Your dtype is not what you expect it to be - you're running into https://github.com/numpy/numpy/issues/8969:

>>> dt = np.dtype('i,i,U')
>>> dt
dtype([('f0', '<i4'), ('f1', '<i4'), ('f2', '<U')])
>>> dt['f2'].itemsize
0  # 0-length strings!

You need to either specify a maximum number of characters

>>> dt = np.dtype('i,i,16U')

Or use an object type to store variable length strings:

>>> dt = np.dtype('i,i,O')

answered Oct 17, 2018 at 16:15

Eric

98.1k54 gold badges257 silver badges389 bronze badges

Collectives™ on Stack Overflow

Problem converting python list to np.array. Process is dropping sting type data

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related