concatenating data frames python

Question

I need to concatenate 6 data frames and csv file names are stored as 'all_files'. I have the following code:

df_from_each_file = (pd.read_csv(f, encoding = 'utf-8') for f in all_files)
data = pd.concat(df_from_each_file, ignore_index=True)

It gives the following error:

First line works well, but second line gives this error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 1: invalid start byte

I tried different encodings but still gives the error. Do you guys have any idea?

Looks like you need to try more encodings. What about cp1252, cp1251? — koPytok
– koPytok, Commented Feb 19, 2019 at 19:15
Those give errors as well: 'charmap' codec can't decode byte 0x98 in position 1: character maps to <undefined> — user229519
– user229519, Commented Feb 19, 2019 at 19:16
@koPytok, I updated the question, the second line gives the error. — user229519
– user229519, Commented Feb 19, 2019 at 19:19
Did you try encoding='latin-1' ? Also, pd.concat needs axis=0 or 1. — razimbres
– razimbres, Commented Feb 19, 2019 at 19:25

razimbres · Accepted Answer · 2019-02-19 19:31:58Z

1

@user229519, use encoding='latin-1' for pd.read_csv and also axis=0 or 1 for pd.concat

answered Feb 19, 2019 at 19:31

razimbres

5,0855 gold badges29 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

WhiteHat · Accepted Answer · 2019-02-19 19:36:43Z

0

Maybe it's the separator, try either of these:

pd.read_csv(f,sep = ';')
pd.read_csv(f,sep = ',')

answered Feb 19, 2019 at 19:36

WhiteHat

1206 bronze badges