Different file name when saving in loop python/pandas

Question

I use the following lines of code to loop over different files in a folder:

import os

files_in_folder_1 = [os.path.join(path1, f) for f in os.listdir(path1) if os.path.isfile(os.path.join(path1, f))]

files_in_folder_2 = [os.path.join(path2, f) for f in os.listdir(path2) if os.path.isfile(os.path.join(path2, f))]

for file1, file2 in zip(files_in_folder_1, files_in_folder_2):
    with open(file1) as f1, open(file2) as f2:

        dftask = pd.read_csv(file2)
        dfresource = pd.read_csv(file1)

At the end of all operations I want to save the files in another directory with the same filename. However how should I do that? I use this:

dftask.to_csv(r'path\file1.csv')
dfresource.to_csv(r'path\file2.csv')

However when using this line of code the csv. file is constantly overwritten inside the loop over all files.

What is the solution?

Padraic Cunningham · Accepted Answer · 2015-08-03 10:52:13Z

2

os.path.basename will give you the file name just join it to the new path and save it however you want:

new_dir = "path/to/dir/"
for file1, file2 in zip(files_in_folder_1, files_in_folder_2):
    dftask = pd.read_csv(file2)
    dfresource = pd.read_csv(file1)
    # work on df's  .......

    # save to new dir   
    dftask.to_csv(os.path.join(new_dir,os.path.basename(file2)))
    dfresource.to_csv(os.path.join(new_dir,os.path.basename(file1)))

If you are using file.open to open the files first you can get the name from the .name attribute:

new_dir = "path/to/dir/"
for file1, file2 in zip(files_in_folder_1, files_in_folder_2):
    with open(file1) as f1, open(file2) as f2:
        dftask = pd.read_csv(file2)
        dfresource = pd.read_csv(file1)

    dftask.to_csv(os.path.join(new_dir, file2.name))
    dfresource.to_csv(os.path.join(new_dir,file1.name))

edited Aug 3, 2015 at 10:52

answered Aug 3, 2015 at 10:29

Padraic Cunningham

181k30 gold badges264 silver badges327 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

F1990 Over a year ago

When I try this I get the following error: IOError: [Errno 22] invalid mode ('w') or filename

F1990 Over a year ago

This line: dftask.to_csv(os.path.join(new_dir,os.path.basename(file2)))

F1990 Over a year ago

filename: "r'path\\Tasks-Job--16kvZ-Feike_15min_data-201728.csv"

Padraic Cunningham Over a year ago

why is there an r there? Anyway what does print os.path.join(new_dir,os.path.basename(file2) output?

F1990 Over a year ago

I used this same 'r before when pointing to a directory, so I thaught it was necessary. But I tried it without the 'r and it returns the same error

|

zuku · Accepted Answer · 2015-08-03 10:10:31Z

0

As far as I understand your question you can use 'path\' + file1.name.split("/")[-1] to save all files with original names.

answered Aug 3, 2015 at 10:10

zuku

6679 silver badges25 bronze badges

Comments

shanmuga · Accepted Answer · 2015-08-03 10:11:35Z

With the above code when the loop exits file1 and file2 each will contain individual DataFrame from last iteration of the loop.

If you want to consolidate all the DataFrames should create list containing each of the individual DataFrames and concat them.

import os

files_in_folder_1 = [os.path.join(path1, f) for f in os.listdir(path1) if os.path.isfile(os.path.join(path1, f))]

files_in_folder_2 = [os.path.join(path2, f) for f in os.listdir(path2) if os.path.isfile(os.path.join(path2, f))]

dftask_list = []
dfresource_list = []
for file1, file2 in zip(files_in_folder_1, files_in_folder_2):
    with open(file1) as f1, open(file2) as f2:

        dftask_list.append(pd.read_csv(file2))
        dfresource_list.append(pd.read_csv(file1))

dftask = pd.concat(dftask_list)
dfresource = pd.concat(dfresource_list)

Note: You may need to reset index after this.

dftask = dftask.reset_index(drop=True)
dfresource = dfresource.reset_index(drop=True)

Anand S Kumar · Accepted Answer · 2015-08-03 10:14:02Z

0

You can use os.path.split() , the function returns a tuple, where the second element would be the filename . Example -

f1name = os.path.split(file1)[1]
f2name = os.path.split(file2)[1]

Then you can use os.path.join() to join it with the other directory and get the resultant path. Example -

file1newpath = os.path.join(otherdir, os.path.split(file1)[1])
file2newpath = os.path.join(otherdir, os.path.split(file2)[1])

Then you can use the above names to save the file -

dftask.to_csv(file1newpath)
dfresource.to_csv(file2newpath)

Demo for os.path.split() -

>>> import os.path
>>> os.path.split(r'C:\Users\temp\somedir\somefile.csv')[1]
'somefile.csv'

answered Aug 3, 2015 at 10:14

Anand S Kumar

91.4k18 gold badges196 silver badges179 bronze badges

Collectives™ on Stack Overflow

Different file name when saving in loop python/pandas

4 Answers 4

11 Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

11 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related