How to use the appropriate encoding when reading csv in Pandas?

Question

I am trying to load datasets from FDIC. Every quarter FDIC releases a zip file that contains around 62 csv files with names like the following:

All_Reports_20080331_Assets and Liabilities.cvs,
All_Reports_20080331_Bank Assets Sold and Securitized.csv, 
etc

I have downloaded the all the files in a directory like the following:

C:\projects\FDIC\All_Reports_20080331

Once there are many zip files, from different quarters available, I am starting to prepare a structure for a loop that will run over many paths (each one representing a quarter with 62 csv files inside). Before getting into the loop, however, the upload is not working due to some utf_8 error.

import pandas as pd
path = r"C:\projects\FDIC\All_Reports_20080331"
file = r"\All_Reports_20080331_Assets and Liabilities.csv"
df_assets_&_liab = pd.read_csv(path+file)

gives me the following error:

'utf-8' codec can't decode byte 0xfc in position 5: invalid start byte

I tried to use a parameter in pandas.read_csv to "utf_8" but error message is the same.

Any idea on how to better load those files via panda? Thanks a lot!

ps: the forder with the 62 csv files can be found here: FDIC Website

giser_yugang · Accepted Answer · 2018-09-18 09:20:37Z

1

First look at the encoding format of the file.

import chardet
with open(path+file,"rb") as f:
    data = f.read()
    print(chardet.detect(data))

{'encoding': 'ISO-8859-1', 'confidence': 0.73, 'language': ''}

Then

df_assets_&_liab = pd.read_csv(path+file,encoding='ISO-8859-1')

answered Sep 18, 2018 at 9:20

giser_yugang

6,1864 gold badges24 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Guga Over a year ago

This worked perfectly, Yugang. Curiously It took 4 min to run over a 5 mb file. Thank you very much for such a answer. Much appreciated.

giser_yugang Over a year ago

@Guga Yes,it is slow as you say. You can also try cchardet that is high speed universal character encoding detector.

Guga Over a year ago

Does my question deserve a + ? Or was it too stupid ? :-)

Collectives™ on Stack Overflow

How to use the appropriate encoding when reading csv in Pandas?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related