Proper way of writing and reading Dataframe to file in Python

Question

I would like to write and later read a dataframe in Python.

df_final.to_csv(self.get_local_file_path(hash,dataset_name), sep='\t', encoding='utf8')
...
df_final = pd.read_table(self.get_local_file_path(hash,dataset_name), encoding='utf8',index_col=[0,1])

But then I get:

sys:1: DtypeWarning: Columns (7,17,28) have mixed types. Specify dtype option on import or set low_memory=False.

I found this question. Which in the bottom line says I should specify the field types when I read the file because "low_memory" is deprecated... I find it very inefficient.

Isn't there a simple way to write & later read a Dataframe? I don't care about the human-readability of the file.

Mike Müller · Accepted Answer · 2017-08-21 06:39:28Z

1

You can pickle your dataframe:

df_final.to_pickle(self.get_local_file_path(hash,dataset_name))

Read it back later:

df_final = pd.read_pickle(self.get_local_file_path(hash,dataset_name))

If your dataframe ist big and this gets to slow, you might have more luck using the HDF5 format:

df_final.to_hdf(self.get_local_file_path(hash,dataset_name))

Read it back later:

df_final = pd.read_hdf(self.get_local_file_path(hash,dataset_name))

You might need to install PyTables first.

Both ways store the data along with their types. Therefore, this should solve your problem.

edited Aug 21, 2017 at 6:39

answered Aug 21, 2017 at 6:30

Mike Müller

86k21 gold badges174 silver badges165 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Tim Seed · Accepted Answer · 2017-08-21 06:36:43Z

0

The warning is because Pandas has detected conflicting Data values in your Column. You can specify the datatypes in the DataFrame Constructor if you wish.

,dtype={'FIELD':int,'FIELD2':str}

Etc.

answered Aug 21, 2017 at 6:36

Tim Seed

5,2872 gold badges32 silver badges27 bronze badges

Collectives™ on Stack Overflow

Proper way of writing and reading Dataframe to file in Python

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related