how to keep numpy array when saving pandas dataframe to csv

Question

I have a pandas.DataFrame with one of the columns as images. Each row of that column is an image as a 2d numpy.array. I saved the DataFrame to a csv file with pandas.DataFrame.to_csv(). However, when I open the csv file, the column becomes string instead of a numpy.array.

How can read the csv file and keep the numpy.array?

Stephen Rauch · Accepted Answer · 2017-03-13 03:43:45Z

16

To read the numpy.array from the csv file, you can provide a converter function to pandas.read_csv.

Code:

import ast
import numpy as np
def from_np_array(array_string):
    array_string = ','.join(array_string.replace('[ ', '[').split())
    return np.array(ast.literal_eval(array_string))

Test Code:

import numpy as np
import pandas as pd

image = np.array([[0.1, 0.2], [0.3, 0.4]])
df = pd.DataFrame(
    [['image name1', image],
     ['image name2', image],
     ],
    columns=['names', 'images']).set_index('names')
print(df)
df.to_csv('sample.csv')

df2 = pd.read_csv('sample.csv', converters={'images': from_np_array})
print(df2)

Results:

                               images
names                                
image name1  [[0.1, 0.2], [0.3, 0.4]]
image name2  [[0.1, 0.2], [0.3, 0.4]]

         names                    images
0  image name1  [[0.1, 0.2], [0.3, 0.4]]
1  image name2  [[0.1, 0.2], [0.3, 0.4]]

answered Mar 13, 2017 at 3:43

Stephen Rauch♦

50.1k32 gold badges118 silver badges143 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

zesla Over a year ago

Thanks a lot! what is the purpose of replace('[ ', '[')? @Stephen Rauch

zesla Over a year ago

A general question is: generally, should I store the image as flattened 1d array in dataframe? @Stephen Rauch

Stephen Rauch Over a year ago

The replace removed some spaces, because later I replace spaces with ',' and a comma there would have been a problem.

Stephen Rauch Over a year ago

I have not studied the csv parser in detail, so could not answer with authority as to the best format, but I can say that the line feeds in the non-flattened arrays are kinda weird.

Bikash Gyawali Over a year ago

And if you need to get hold of the array in images field of the dataframe df2 inside a python variable, use: my_images = np.array(df2['images'].values.tolist()).

|

Collectives™ on Stack Overflow

how to keep numpy array when saving pandas dataframe to csv

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related