How to unpack from a binary file a byte array using Python?

Question

I'm giving myself a crash course in reading a binary file using Python. I'm new to both, so please bear with me.

The file format's documentation tells me that the first 16 bytes are a GUID and further reading tells me that this GUID is formatted thus:

typedef struct {
  unsigned long Data1;
  unsigned short Data2;
  unsigned short Data3;
  byte Data4[8];
} GUID, 
 UUID, 
 *PGUID;

I've got as far us being able to unpack the first three entries in the struct, but I'm getting stumped on #4. It's an array of 8 bytes I think but I'm not sure how to unpack it.

import struct

fp = open("./file.bin", mode='rb')

Data1 = struct.unpack('<L', fp.read(4)) # unsigned long, little-endian
Data2 = struct.unpack('<H', fp.read(2)) # unsigned short, little-endian 
Data3 = struct.unpack('<H', fp.read(2)) # unsigned short, little-endian
Data4 = struct.unpack('<s', bytearray(fp.read(8))) # byte array with 8 entries?

struct.error: unpack requires a bytes object of length 1

What am I doing wrong for Data4? (I'm using Python 3.2 BTW)

Data1 thru 3 are OK. If I use hex() on them I am getting the correct data that I'd expect to see (woohoo) I'm just failing over on the syntax of this byte array.

Edit: Answer

I'm reading a GUID as defined in MS-DTYP and this nailed it:

data = uuid.UUID(bytes_le=fp.read(16))

You're using struct in combination with bytearray, these are separate things. I'm not familiar with bytearray, but I'm pretty sure struct.unpack() won't take the object returned by bytearray — monkut
– monkut, Commented May 12, 2013 at 21:39

abarnert · Accepted Answer · 2013-05-12 22:03:11Z

14

If you want an 8-byte string, you need to put the number 8 in there:

struct.unpack('<8s', bytearray(fp.read(8)))

From the docs:

A format character may be preceded by an integral repeat count. For example, the format string '4h' means exactly the same as 'hhhh'.

…

For the 's' format character, the count is interpreted as the length of the bytes, not a repeat count like for the other format characters; for example, '10s' means a single 10-byte string, while '10c' means 10 characters. If a count is not given, it defaults to 1. For packing, the string is truncated or padded with null bytes as appropriate to make it fit. For unpacking, the resulting bytes object always has exactly the specified number of bytes. As a special case, '0s' means a single, empty string (while '0c' means 0 characters).

However, I'm not sure why you're doing this in the first place.

fp.read(8) gives you an 8-byte bytes object. You want an 8-byte bytes object. So, just do this:

Data4 = fp.read(8)

Converting the bytes to a bytearray has no effect except to make a mutable copy. Unpacking it just gives you back a copy of the same bytes you started with. So… why?

Well, actually, struct.unpack returns a tuple whose one value is a copy of the same bytes you started with, but you can do that with:

Data4 = (fp.read(8),)

Which raises the question of why you want four single-element tuples in the first place. You're going to be doing Data1[0], etc. all over the place for no good reason. Why not this?

Data1, Data2, Data3, Data4 = struct.unpack('<LHH8s', fp.read(16))

Of course if this is meant to read a UUID, it's always better to use the "batteries included" than to try to build your own batteries from nickel and cadmium ore. As icktoofay says, just use the uuid module:

data = uuid.UUID(bytes_le=fp.read(16))

But keep in mind that Python's uuid uses the 4-2-2-1-1-6 format, not the 4-2-2-8 format. If you really need exactly that format, you'll need to convert it, which means either struct or bit twiddling anyway. (Microsoft's GUID makes things even more fun by using a 4-2-2-2-6 format, which is not the same as either, and representing the first 3 in native-endian and the last two in big-endian, because they like to make things easier…)

edited May 12, 2013 at 22:03

answered May 12, 2013 at 21:42

abarnert

368k54 gold badges626 silver badges691 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

icktoofay Over a year ago

Additionally, the bytearray is not needed (although it won't hurt).

abarnert Over a year ago

@icktoofay: You're right… but the whole unpack is not needed either…

Darren Beale Over a year ago

Thanks for the all comments. very helpful and will assist as I now try to read the rest of the file :)

icktoofay · Accepted Answer · 2013-05-12 21:59:39Z

4

UUIDs are supported by Python with the uuid module. Do something like this:

import uuid

my_uuid = uuid.UUID(bytes_le=fp.read(16))

edited May 12, 2013 at 21:59

answered May 12, 2013 at 21:37

icktoofay

130k23 gold badges261 silver badges239 bronze badges

2 Comments

abarnert Over a year ago

bytes is in big-endian order; the OP's data (assuming he got the struct format right) is in little-endian. So, you need bytes_le here, not bytes.

abarnert Over a year ago

Also, it's worth noting that Python's UUID is 4-2-2-1-1-6, not 4-2-2-8. So, if you want to get the 4-2-2-8 format, you have to convert fields manually.

Collectives™ on Stack Overflow

How to unpack from a binary file a byte array using Python?

2 Answers 2

3 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related