Combining multiple text files belonging to different groups using Python

Question

In my directory the following files (1_xxx.txt, 2_xxx.txt, 1_yyy.txt, 2_yyy.txt, 1_zzz.txt, 2_zzz.txt) exists. The contents of those files are shown below:

1_xxx.txt:
-114.265646442 34.0360392257
-112.977603537 31.6338662268
-117.239800991 36.1408246787
-114.716762067 32.958308901
-116.710069802 36.2660863375
-115.412539137 34.5790101356
-117.173651349 36.1032456689
-115.254332318 33.8689615728
-115.225643473 32.8079130497
-113.757416909 32.6491579487

2_xxx.txt:
-121.527298571 38.3074782763
-119.241009725 35.2597437123
-111.993090251 33.1087011262
-119.328464365 35.8944690935
-114.819870325 32.7076471384
-120.041889447 36.4080463723
-121.249592001 38.3951295581
-121.078565259 37.6730108558
-120.523147893 37.2889578323
-119.2383536 35.9028202963

1_yyy.txt:
-109.690156887 34.2072891001
-119.780672722 38.7665894396
-118.557741892 35.6314002547
-118.483411917 36.3579432166
-123.472136838 39.1714120111
-123.485136802 40.0894616596
-109.185105643 33.8647845733
-120.046426359 38.4660843951
-122.929234616 40.1186699391
-123.300682512 39.2757431576
2_yyy.txt:
-120.915282551 37.0468246029
-118.168309521 37.606220824
-111.172152572 35.5687631188
-110.999951025 34.8671827527
-120.375558342 37.7773687622
-121.028079242 36.5374775742
-118.53486589 36.7879815762
-115.771046166 39.1046390941
-117.618352132 39.3133019115
-110.163871705 34.6500104537

1_zzz.txt:
-117.442417987 34.0694542108
-118.624320171 34.3117074054
-111.915932786 33.6893480358
-118.214145399 34.0360392257
-122.189710383 37.6396159347
-122.413202409 37.9443375576
-115.524007077 32.9541312874
-117.735266836 33.9107314118
-110.840774505 32.3734158543
-122.399926026 37.7898915865

2_zzz.txt:
-106.544451063 31.5126888716
-112.728165588 32.3232796291
-117.793575105 34.8128904057
-116.464953895 32.3441697714
-116.206850112 34.2448798952
-121.758363934 37.9819048821
-113.317063698 33.5306154403
-115.999423067 31.4750816387
-115.257632657 37.8817248156
-117.558324417 37.4684639908

I want to combine those files matching xxx, yyy, and zzz, and write one separate file for each group:

files = glob.glob('*.txt')
my_classes = ['xxx', 'yyy', 'zzz']

files_dict ={}
for c in my_classes:
    files_dict[c] = [f for f in files if c in f]

all_lons, all_lats ={}, {}

for e, f in files_dict.iteritems():
    all_lons[e], all_lats[e] = [], []
    for x in f:
        fi = open(x, 'r')
        lines = fi.read().splitlines()
        lons = [l.split(' ')[0] for l in lines]
        lats = [l.split(' ')[1] for l in lines]
        all_lons[e].apend(lons)
        all_lats[e].apend(lats)
for g, h in all_lons.iteritems():
    for i, j in all_lats.iteritems():
        with open(g + '_final.txt', 'w') as fo:
            fo.write(str(h) + str(j) + '\n' )

Because of my limited knowledge of Python I could not do it. I am waiting to know best practices of solving it. The number of text files corresponding to each of my class(i.e., xxx, yyy, zzz) are more than two than shown in this example.

You seem to be doing a lot of unnecessary processing. Is your expected end result in the same format as the initial files? If so, maybe the shell solution can do the job. And if you need to use python, you can take the shell example as a starting point for how to do it. — Jacobo de Vera
– Jacobo de Vera, Commented Apr 24, 2014 at 13:09
@Jacobo de Vera yes, my expected end result is in the same format as the initial files. — user3235542
– user3235542, Commented Apr 24, 2014 at 13:11
Then in your python code you don't need separate containers for each columns, or splitting, you could even read all the file in one go with read() and then write it out with write(). — Jacobo de Vera
– Jacobo de Vera, Commented Apr 24, 2014 at 13:21

Jacobo de Vera · Accepted Answer · 2014-04-24 13:29:47Z

2

Since you are under windows, the shell option might not work for you unless you are working with a POSIX-like shell, so here is a similar way to it in python.

import glob
files = glob.glob('*.txt')
my_classes = ['xxx', 'yyy', 'zzz']

for cls in my_classes:
    files = glob.glob('*{}*.txt'.format(cls))
    if files:
        with open(cls, 'w') as fout:
            for file_name in files:
                with open(file_name, 'r') as fin:
                    fout.write(fin.read())

This is based on the fact that you really don't need to do any processing on the contents of each file other than putting them all together based on some keyword in the file name.

answered Apr 24, 2014 at 13:29

Jacobo de Vera

1,9512 gold badges16 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user3235542 Over a year ago

thanks for your answer, upvoted! based on your answer i have solved the problem as my answer to this question

user1907906 · Accepted Answer · 2014-04-24 13:01:10Z

0

Not Python but you can use the power of the shell:

for g in xxx yyy zzz
do
  cat *_$g.txt > $g.txt
done

This will compine all xxx files in the file xxx.txt etc.

answered Apr 24, 2014 at 13:01

user1907906

2 Comments

user3235542 Over a year ago

sorry where should i write your code in my pc, in cmd prompt?

Jacobo de Vera Over a year ago

You can do that, or you can write it in a file and run it with bash <file>

Nani · Accepted Answer · 2014-04-24 13:35:04Z

0

import glob

path = raw_input('Enter input path:')
files = glob.glob(path + '/*.*')
patterns = ['xxx', 'yyy', 'zzz']

for pattern in patterns :
    content = ''
    for filename in files :
        if pattern in filename :
            with open(filename) as fread:
                content = content + fread.read()            
    with open('output' + pattern +'.txt', 'w') as fwrite:
        fwrite.write(content)

answered Apr 24, 2014 at 13:35

Nani

13 bronze badges

Comments

user3235542 · Accepted Answer · 2014-04-24 13:40:38Z

0

import glob


files = glob.glob('*.txt')
my_classes = ['xxx', 'yyy', 'zzz']

files_dict ={}
for c in my_classes:
    files_dict[c] = [f for f in files if c in f]

for e, f in files_dict.iteritems():
    with open(e + '_final.txt', 'w') as fo:
        for x in f:
            with open (x, 'r') as fi:
                fo.write(fi.read())

answered Apr 24, 2014 at 13:40

user3235542

1 Comment

Jacobo de Vera Over a year ago

Great. Since you were looking for some best practice advise, I'd recommend to start with using more meaningful names for your variables. It will make it easier if, say, a couple of months from now, you need to modify something.

Collectives™ on Stack Overflow

Combining multiple text files belonging to different groups using Python

4 Answers 4

1 Comment

2 Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

2 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related