merging big binary files using python 3

Question

How can I merge a few big files into 1 using python 3? it should work like the bash command

cat * > outfile

but it should work on Linux, Windows and OS X.

If I use

outfile = open("outfile", "wb")
for file in os.listdir():
    outfile.write(file.read())

it uses too much RAM

have you tried the "For large files" from here: stackoverflow.com/a/13613375/3430986 ? — orestisf
– orestisf, Commented Oct 31, 2014 at 18:29
It only works for text files <- open with 'rb' - the b flag is for binary files — inspectorG4dget
– inspectorG4dget, Commented Oct 31, 2014 at 18:32

Terry Jan Reedy · Accepted Answer · 2014-10-31 21:59:39Z

2

For large binary files, instead of reading lines, read chunks that are a multiple of the disk block size. Something like (untested)

BLOCKSIZE = 4096  # typical, I believe
BLOCKS = 1024  # somewhat arbitrary
chunk = BLOCKS * BLOCKSIZE
with open("outfile", "wb") as outfile:
    for fname in os.listdir():
        with open('fname', "rb") as infile
            outfile.write(infile.read(chunk))

answered Oct 31, 2014 at 21:59

Terry Jan Reedy

19.3k3 gold badges44 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

merging big binary files using python 3

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related