Text data encoding error for csv format in Python

Question

I am trying to import the data and facing issue with the encoding.

Some times utf-8 works but not latin-1, similarly it depends on the type of data coming.

Different encoding used - latin-1, utf-8, windows-1252

Code -

pd.read_csv(dir_in+file_note,sep='|',
              low_memory=False,header=0,
              error_bad_lines=False,
              encoding = "windows-1252",
              warn_bad_lines=False)

Please guide on how to make the code dynamic so that if one gives error it should try the other one.

1) Priority one will be utf-8

2) Priority two will be latin-1

3) Priority three will be windows-1252

Try loading the csv with all 3 of the encodings you mentioned in that order & catch any errors that occur — rdas
– rdas, Commented Apr 1, 2020 at 18:59

incarnadine · Accepted Answer · 2020-04-01 19:01:26Z

1

Not very beautiful, but it works.

try:
  pd.read_csv(dir_in+file_note,sep='|',
                low_memory=False,header=0,
                error_bad_lines=False,
                encoding = "utf-8",
                warn_bad_lines=False)
except:
  try:
    pd.read_csv(dir_in+file_note,sep='|',
                  low_memory=False,header=0,
                  error_bad_lines=False,
                  encoding = "latin-1",
                  warn_bad_lines=False)
  except:
    pd.read_csv(dir_in+file_note,sep='|',
                  low_memory=False,header=0,
                  error_bad_lines=False,
                  encoding = "windows-1252",
                  warn_bad_lines=False)

answered Apr 1, 2020 at 19:01

incarnadine

7009 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Dr.Chuck Over a year ago

Thank you, Exactly what I was looking for. Really appreciate!

Kate · Accepted Answer · 2020-04-01 20:07:10Z

0

The question was asked on the github dev page:

Is there a way pandas can read a csv file and find out the encoding automatically . Or is there a fix to this? Maybe to be a feature (if it does not exist yet) in a future release?

One contributor says:

This seems out of scope for pandas. I'd recommend using a library like chardet to determine the encoding ahead of time.

answered Apr 1, 2020 at 20:07

Kate

1,8361 gold badge11 silver badges9 bronze badges

2 Comments

Dr.Chuck Over a year ago

Actually, I created the data using SPSS stream and choose the output format as utf-8. But when imported the data in notebook, it gives error. Will definitely explore chardet, thank you.

Kate Over a year ago

What kind of error exactly ? Please be more specific. If you always export the files in a specific encoding there should be no surprise and no need to guess the encoding.

Collectives™ on Stack Overflow

Text data encoding error for csv format in Python

2 Answers 2

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related