Unpacking and Converting Bytes -Python

Question

I am trying to unpack a file containing over 1 billion bytes that encode integers which are 4 bytes each. So every 4 bytes is a different integer. I obviously need to chunk my code for such a big file. I currently have the following:-

import os
z =os.path.getsize(x)
import struct
with open(x, "rb") as f:
    while True: 
        this_chunk = min(50000000, z)
        data = f.read(this_chunk)
        ints1 = struct.unpack("I" * (this_chunk //4) , data)
        if not data:
            break 
    print(ints1)

I get an error which reads:-

struct.error: unpack requires a bytes object of length 50000000

Could you please help me understand this error and how to fix it? Thank you!

Adam Hughes · Accepted Answer · 2017-07-20 14:01:51Z

1

You need to keep track of your chunks read. I'd recommend using expressive variables names instead of x and z. The main problem is on your last read, where you want to read the amount of sizeremaining, not a full chunk. Try this (untested)

filesize = os.path.getsize(x)
chunksread = 0
chunksize = 50000000
sizeremaining = filesize

with open(filename, "rb") as f:
    while sizeremaining > 0:
        this_chunk = min(chunksize, sizeremaining)
        data = f.read(this_chunk)
        ints1 = struct.unpack("I" * (this_chunk //4) , data)
        sizeremaining -= this_chunk
        if not data:
            break 
    print(ints1)

edited Jul 20, 2017 at 14:01

answered Jul 20, 2017 at 13:47

Adam Hughes

16.5k14 gold badges100 silver badges140 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Adam Hughes Over a year ago

Did that fix the issue?

Adam Hughes Over a year ago

Your min function is not correct. You should be comparing min(50000000) with the bytes remaining. Let me modify my answer

Adam Hughes Over a year ago

Ya, it should. What's it doing?

H. Minear Over a year ago

I just made an indentation error. Thank you for your help!

Collectives™ on Stack Overflow

Unpacking and Converting Bytes -Python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related