Merge multiple .txt/csv files in Python

Question

I have multiple .txt files in a directory and I want to merge them into one by importing in python. The catch here is that after the merge I want to convert it into one csv file on which the whole program is based.

So far I only had to input one .txt file and converted it into csv file by this code:

import io
bytes = open('XYZ.txt', 'rb').read()
df=pd.read_csv(io.StringIO(bytes.decode('utf-8')), sep='\t', parse_dates=['Time'] )
df.head()

Now I need to input multiple .txt files, merge them and then convert them into csv files. Any workaround?

you don't need pandas to do this if thats all its doing, csv module in the standard library would do fine — avigil
– avigil, Commented Jul 14, 2018 at 4:31
also those are commonly referred to as tsv files, so really your question is about converting tsv to csv? — avigil
– avigil, Commented Jul 14, 2018 at 4:39

Pang · Accepted Answer · 2018-07-14 03:46:45Z

3

If the headers are same then it should be as easy as this

import os
import io

merged_df = pd.DataFrame()
for file in os.listdir("PATH_OF_DIRECTORY"):
    if file.endswith(".txt"):
        bytes = open(file, 'rb').read()
        merged_df = merged_df.append(pd.read_csv(io.StringIO(
            bytes.decode('utf-8')), sep='\t', parse_dates=['Time']))

print(len(merged_df))

edited Jul 14, 2018 at 3:46

Pang

10.2k146 gold badges87 silver badges126 bronze badges

answered Jul 12, 2018 at 5:33

Aditya

3521 silver badge12 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Aditya Over a year ago

You can manipulate list of files by doing some string filter or regex operations.

TheVaibhavJ Over a year ago

Can you please tell how I can edit it so that it only takes .txt files?

TheVaibhavJ Over a year ago

There are 4 .txt files in the directory and it is only taking in one. Working same like the code which I wrote for reading 1 file

Aditya Over a year ago

Then make sure path is right and all the extensions are exactly ".txt" (matching the case)as this is very straightforward code.

TheVaibhavJ Over a year ago

Path is alright as it is reading one file. And the other files are also .txt. Not working. Weird. Thanks anyway man!

vrana95 · Accepted Answer · 2018-07-12 10:32:21Z

0

import glob
path="location/of/folder"
allFiles = glob.glob(path + "\\*.txt")

list_ = []
for file in allFiles:
    print(file)
    df = pd.read_csv(io.StringIO(file.decode('utf-8')), sep='\t', parse_dates=['Time'])
    list_.append(df)
combined_files = pd.concat(list_)

edited Jul 12, 2018 at 10:32

answered Jul 12, 2018 at 5:35

vrana95

5212 silver badges10 bronze badges

1 Comment

user9455968 Over a year ago

Could you please explain why you thing this is a solution? How is your code different from the one in the question?

Collectives™ on Stack Overflow

Merge multiple .txt/csv files in Python

2 Answers 2

5 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related