python concatenate files using subprocess [duplicate]

Question

I have been searching and although I find long and complicated (many features I do not need) on how to just simply have python have linux use the cat function to concatenate files to a single file.

From my reading apparently subprocess is the way to do this. Here is what I have but obviously does not work :(

subprocess.call("cat", str(myfilelist[0]), str(myfilelist[1]), str(myfilelist[2]), str(myfilelist[3]), ">", "concatinatedfile.txt"])

the above assumes:

myfilelist[]

the above list has 4 filenames + paths as a list; for example one item in the list is "mypath/myfile1.txt"

I would take non subprocess (but simple) methods also

Joran Beasley · Accepted Answer · 2013-06-27 23:30:16Z

5

since > is a shell function you need to do shell=True

subprocess.call("echo hello world > some.txt",shell=True) ... works in windows at least

alternatively you can do somethign like

with open("redirected_output.txt") as f:
    subprocess.call("/home/bin/cat some_file.txt",stdout=f)

answered Jun 27, 2013 at 23:30

Joran Beasley

114k13 gold badges167 silver badges187 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Emanuele Paolini · Accepted Answer · 2013-06-27 23:27:08Z

4

If you want to use cat and redirection > you must call a shell, for example via system:

from os import system
system("cat a.txt b.txt > c.txt")

However you must pay attention to code injection.

answered Jun 27, 2013 at 23:27

Emanuele Paolini

10.2k5 gold badges45 silver badges69 bronze badges

3 Comments

StudentOfScience Over a year ago

coolz! this worked faster. so what is the difference of using subprocess then?

Joran Beasley Over a year ago

good answer(+1) i just did the same thing with subprocess

Emanuele Paolini Over a year ago

here you are running a shell so it is slower and more prone to security risks

Community · Accepted Answer · 2017-05-23 12:31:11Z

2

See this question. His solution seems to be concise and easy to understand, I'll post it here:

filenames = ['file1.txt', 'file2.txt', ...]
with open('path/to/output/file', 'w') as outfile:
    for fname in filenames:
        with open(fname) as infile:
            outfile.write(infile.read())

edited May 23, 2017 at 12:31

CommunityBot

11 silver badge

answered Jun 27, 2013 at 23:16

Stephan

18.4k8 gold badges38 silver badges61 bronze badges

4 Comments

StudentOfScience Over a year ago

Thanks Stephan. THis does work, and I had seen it. It is just that it is CPU intensive when you have text files each over 10k or more lines :( I wanted to use the faster cat function of linux.

Stephan Over a year ago

this seems to be some really optimized concatenation @StudentOfScience

StudentOfScience Over a year ago

I just ran it with my big files, took 1 min 34 sec. When I use cat manually (minus me typing the code) it take 24 sec. that is why. Not that I do not have an additional 1 min and 10 sec to give, but just as a learning opportunity to use subprocess, or python to linux I asked the question. Thx.

Stephan Over a year ago

this would help you with your learning opportunity to use subprocess for concatenation

Collectives™ on Stack Overflow

python concatenate files using subprocess [duplicate]

3 Answers 3

Comments

3 Comments

4 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

3 Comments

4 Comments

Linked

Related