How can I merge two files in one using BASH or Python script?

Question

We assume that I have two files like this:

File1 :

F: user1 password1 
F: user2 password2 
F: user3 password3

File2 :

server1 24000
server2 24000
server3 24000
server4 24000

I want to combine them in order to get one file with this output file :

OuputFile :

C: server1 24000  user1 password1
C: server2 24000  user1 password1
C: server3 24000  user1 password1
C: server4 24000  user1 password1
C: server1 24000  user2 password2
C: server2 24000  user2 password2
C: server3 24000  user2 password2
C: server4 24000  user2 password2
C: server1 24000  user3 password3
C: server2 24000  user3 password3
C: server3 24000  user3 password3
C: server4 24000  user3 password3

So, in Windows, I made this batch file to get what I expected, but, I have no idea how to do it in BASH (Bourne-Again shell) or in Python script.

The Batch File :

@echo off
set file1=file1.txt
set file2=file2.txt
Set Output=Output_CCCam.cfg
If Exist %Output% Del %Output%
for /f "tokens=2 delims=:" %%a in ('Type "%file1%"') do (
    for /f "delims=" %%b in ('Type "%file2%"') do (
        >>%Output% echo C: %%b %%a
    )
)
Start "" Notepad %Output%

I removed the batch-file tag, as it is not applicable to your question. It refers to batch-files in the specific context of MS-DOS, Windows, or OS-2 operating systems. Please don't use tags just because they contain similar sounding names or phrases. Tags here have specific meanings. If you're not sure, read the description of the tag. If you're still not sure, don't use it; if it's necessary, someone will add it for you. — Ken White
– Ken White, Commented Apr 3, 2016 at 23:44
Do you have to do it in BASH or something like Python or AWK is acceptable? — pkacprzak
– pkacprzak, Commented Apr 3, 2016 at 23:49
@pkacprzak in python is acceptable ! Thank you ! i will add Python tag — Hackoo
– Hackoo, Commented Apr 3, 2016 at 23:50
cat in linux, and copy in windows can do the job I think — YOU
– YOU, Commented Apr 4, 2016 at 0:01

Benjamin W. · Accepted Answer · 2016-04-04 01:01:35Z

A Bash solution

#!/bin/bash

while IFS= read -r line1; do
    while IFS= read -r line2; do
        printf "C: %s  %s\n" "$line2" "${line1/#F: }"
    done < file2
done < file1

This loops over file1, and for each line of file1 loops over file2. The printf line assembles the output, and the parameter expansion for line removes the leading F: .

Result:

C: server1 24000  user1 password1
C: server2 24000  user1 password1
C: server3 24000  user1 password1
C: server4 24000  user1 password1
C: server1 24000  user2 password2
C: server2 24000  user2 password2
C: server3 24000  user2 password2
C: server4 24000  user2 password2
C: server1 24000  user3 password3
C: server2 24000  user3 password3
C: server3 24000  user3 password3
C: server4 24000  user3 password3

A solution with join and sed

This would work as well:

join -j 50 -o 2.1,1.1 -t '~' file1 file2 | sed s'/~F:/ /;s/^/C: /'

This is a slight abuse of join. -j 50 says to join on matching field number 50, which doesn't exist and is thus considered equal for all lines, resulting in the Cartesian product of the two files:

$ join -j 50 file1 file2
 F: user1 password1 server1 24000
 F: user1 password1 server2 24000
 F: user1 password1 server3 24000
 F: user1 password1 server4 24000
 F: user2 password2 server1 24000
 F: user2 password2 server2 24000
 F: user2 password2 server3 24000
 F: user2 password2 server4 24000
 F: user3 password3 server1 24000
 F: user3 password3 server2 24000
 F: user3 password3 server3 24000
 F: user3 password3 server4 24000

To get the lines into proper order, we specifiy the output format with -o 2.1,1,1. Because the default field delimiter is whitespace, we specify a character that is not contained in the input as the new delimiter with -t '~':

$ join -j 50 -o 2.1,1.1 -t '~' file1 file2
server1 24000~F: user1 password1
server2 24000~F: user1 password1
server3 24000~F: user1 password1
server4 24000~F: user1 password1
server1 24000~F: user2 password2
server2 24000~F: user2 password2
server3 24000~F: user2 password2
server4 24000~F: user2 password2
server1 24000~F: user3 password3
server2 24000~F: user3 password3
server3 24000~F: user3 password3
server4 24000~F: user3 password3

And finally, we replace ~F: with a space on each line and prepend C: using sed:

$ join -j 50 -o 2.1,1.1 -t '~' file1 file2 | sed 's/~F:/ /;s/^/C: /'
C: server1 24000  user1 password1
C: server2 24000  user1 password1
C: server3 24000  user1 password1
C: server4 24000  user1 password1
C: server1 24000  user2 password2
C: server2 24000  user2 password2
C: server3 24000  user2 password2
C: server4 24000  user2 password2
C: server1 24000  user3 password3
C: server2 24000  user3 password3
C: server3 24000  user3 password3
C: server4 24000  user3 password3

If the order of the lines doesn't matter, this can be slightly shortened to

$ join -j 50 file2 file1 | sed 's/F://;s/^/C:/'
C: server1 24000  user1 password1
C: server1 24000  user2 password2
C: server1 24000  user3 password3
C: server2 24000  user1 password1
C: server2 24000  user2 password2
C: server2 24000  user3 password3
C: server3 24000  user1 password1
C: server3 24000  user2 password2
C: server3 24000  user3 password3
C: server4 24000  user1 password1
C: server4 24000  user2 password2
C: server4 24000  user3 password3

5gon12eder · Accepted Answer · 2016-04-04 00:03:45Z

This solution might not win in terms of least characters to type but I think it will be pretty straight-forward to understand. I'm assuming that your files are reasonably small enough to easily fit into memory all at once.

#! /bin/bash

## File names of the files we want to join.

file_1st='file_1st.txt'
file_2nd='file_2nd.txt'

## Declare array variables to hold the lines of the data contained in the files.

declare -a file_data_1st
declare -a file_data_2nd

## Read both files into memory.  The `-t` option trims trailing newline
## characters.  The arrays will now contain the trimmed lies of each file.

mapfile -t file_data_1st < "${file_1st}"
mapfile -t file_data_2nd < "${file_2nd}"

## Now iterate over the lines of the first file and inside that loop over the
## lines of the second file.  Split both lines into white-space separated words
## and then re-assemble the output line as desired.  This is a little more
## general than actually needed here (you don't really have to split the lines
## from the second file.

for line_1st in "${file_data_1st[@]}"
do
    words_1st=(${line_1st})
    for line_2nd in "${file_data_2nd[@]}"
    do
        words_2nd=(${line_2nd})
        echo "C: ${words_2nd[0]} ${words_2nd[1]} ${words_1st[1]} ${words_1st[2]}"
    done
done

pkacprzak · Accepted Answer · 2016-04-04 00:11:59Z

1

In Python as I asked in the comment:

filename1 = 'file1.txt'
filename2 = 'file2.txt'

user_data = []
server_data = []

with open(filename1, 'r') as fp:
    user_data = map(lambda x: x.split()[1:], fp.readlines())

with open(filename2, 'r') as fp:
    server_data = map(lambda x: x.split(), fp.readlines())

output_filename = 'file3.txt'

with open(output_filename, 'w') as fp:
    for user_row in user_data:
        for server_row in server_data:
            fp.write("C: %s %s\n" % (" ".join(server_row), " ".join(user_row)))

answered Apr 4, 2016 at 0:11

pkacprzak

5,6131 gold badge20 silver badges37 bronze badges

Comments

Bernardo Sulzbach · Accepted Answer · 2016-04-04 01:37:12Z

I originally suggested using the paste tool. However, as pointed out by Benjamin W., the question wants permutations, although the word "combine" is used.

Paste cannot perform permutations alone, much less remove unwanted tokens as is only evident based on the snippets provided by the question author. Follows a Python 3 script that does what is asked.

#!/bin/python3


def merge_lines(line_list_a, line_list_b):
    # List comprehension could be shorter if smaller identifiers were used. However, I consider readability more important than small column limits.
    return [' '.join(['C:'] + line_of_b.split() + [' '] + line_of_a.split()[1:]) for line_of_a in line_list_a for line_of_b in line_list_b]


def main():
    with open('file1.txt') as file_1:
        with open('file2.txt') as file_2:
            with open('output.txt', 'w') as output_file:
                output_file.write('\n'.join((merge_lines(file_1.readlines(), file_2.readlines()))))
                output_file.write('\n')  # Python converts '\n' to the system's default line separator.

if __name__ == '__main__':
    main()

Collectives™ on Stack Overflow

How can I merge two files in one using BASH or Python script?

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related