Python pandas read multiple csv file and turn into dataframe [duplicate]

Question

I tried to read multiple .csv files that have the same format from different folders. It turns out to be a list using .append and I tried to turn it into dataframe using .concat. But it didn't let me to do so. I also tried .os to read the data. It wouldn't work. Any suggestions?

test = []
train = []
for f in testdata:
    test.append(pd.read_csv(f, skiprows = 5, sep = ',', names = 
         ['time','avg_rss12','var_rss12','avg_rss13','var_rss13','avg_rss23','var_rss23']))
for f in traindata:
    train.append(pd.read_csv(f, skiprows = 5, sep = ',', names = 
         ['time','avg_rss12','var_rss12','avg_rss13','var_rss13','avg_rss23','var_rss23']))

What didn't work with pd.concat()? Could you please show what you tried, this is the correct approach to the problem if all of your frames are of the same schema. Did you specify which axis to perform the concatenation along? I think rows is the default axis. — Philip Ciunkiewicz
– Philip Ciunkiewicz, Commented Jul 1, 2020 at 22:00

gallen · Accepted Answer · 2020-07-01 22:00:46Z

1

You mention attempting pandas concat, but don't specify how it failed. It is the way you would want to go. Using just your test loop from the question:

import pandas as pd


frames = []
for f in testdata:
    df = pd.read_csv(f, skiprows=5, sep=',', names=['time','avg_rss12','var_rss12','avg_rss13','var_rss13','avg_rss23','var_rss23'])
    frames.append(df)
combined = pd.concat(frames)

What gets passed into concat() needs to be a list of DataFrame instances, which it will combine into a single frame.

answered Jul 1, 2020 at 22:00

gallen

1,2921 gold badge19 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python pandas read multiple csv file and turn into dataframe [duplicate]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related