How to concatenate a list of csv dataframe by for loop

Question

I have multiple csv files, and I'm trying to concatenate the desired columns for all csv files in the folder.

Here's my code:

import pandas as pd
import numpy as np
import os

path_dataset = r"C:\Users\KL"


def get_file(path_dataset):
    files = os.listdir(path_dataset)
    files.sort()
    file_list = []
    for file in files:
        path = path_dataset + "\\" + file

        if (file.startswith("OS")) and (file.endswith(".csv")):
            file_list.append(path)

    return file_list


read_columns = ["LX", "LY", "LZ", "LA"]

read_files = get_file(path_dataset)

for file in read_files:
    df = pd.read_csv(file, usecols=read_columns)
    all_df = [df]

Concat_table = pd.concat(all_df, axis=0)
Concat_table = Concat_table.sort_values(["LX", "LY", "LZ", "LA"])

Concat_table.to_csv(os.path.join(path_dataset, "Concate_all.csv"), index=False)

I was only able to read one file but not for all csv files. How can I solve this? Thank you.

Alex · Accepted Answer · 2021-07-28 10:05:58Z

3

You should initialise and append each DataFrame to the all_df list as you read them, then concat that list. This is the same as what you are doing in your get_file function.

all_df = []
for file in read_files:
    df = pd.read_csv(file, usecols=read_columns)
    all_df.append(df)

Concat_table = pd.concat(all_df)

answered Jul 28, 2021 at 10:05

Alex

7,1654 gold badges27 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to concatenate a list of csv dataframe by for loop

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related