Exporting Pandas output for multiple CSV files

Question

I have many CSV files under subdirectories in one folder. They all contain tweets and other metadata. I am interested in removing most of these metadata and keeping the tweets themselves and their time. I used glob to read the files, and the removing part seems to be working fine. However, I am not sure how to save the output so that all files are saved and with their original file name.

import pandas as pd
import glob
path = r'D:\tweets'
myfiles= glob.glob(r'D:\tweets\**\*.csv', recursive=True)
for f in myfiles:
    df = pd.read_csv(f)
df = df.drop(["name", "id","conversation_id","created_at","date"], axis=1)
df = df[df["language"].str.contains("bn|ca|ckbu|id||zh")==False]
df.to_csv("output_filename.csv", index=False, encoding='utf8')

Are their Indenting Problems in your question? If not, there isn't only the last file in D:\tweets is getting converted back into a csv. — Aryan Garg
– Aryan Garg, Commented Jun 13, 2021 at 14:26
The only indentation I have is in the sixth line (df = pd.read_csv(f)) — Muhsabrys
– Muhsabrys, Commented Jun 13, 2021 at 14:32
As you are processing each from the list myfiles, your code needs to be part of the for loop. i don't think you would need to overwrite the original file, then something like will help df.to_csv(os.path.splitext(f)[0]+"_transformed.csv") — simpleApp
– simpleApp, Commented Jun 13, 2021 at 14:46

ercnt · Accepted Answer · 2021-06-13 14:46:30Z

1

If you do it this way, it will overwrite the same file:

for f in myfiles:
    df = pd.read_csv(f)
    df = df.drop(["name", "id","conversation_id","created_at","date"], axis=1)
    df = df[df["language"].str.contains("bn|ca|ckbu|id||zh")==False]
    df.to_csv(f, index=False, encoding='utf8')

answered Jun 13, 2021 at 14:46

ercnt

111 silver badge1 bronze badge

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Exporting Pandas output for multiple CSV files

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related