fast binary data conversion in Python

Question

What is the fastest method for converting a binary data string to a numeric value in Python?

I am using struct.unpack_from(), but am hitting a performance limit.

Context: an incoming stream is mixed binary and ASCII data. The ASCII data conversion is done in C though ctypes. Implementing the unpacking in C through ctypes yielded similar performance to unpack. My guess is the call overhead was too much of a factor. I was hoping to find a native C-like coercion method (however un-Pythonic). Most likely all of this code will need to move to C.

The stream is in network byte order (big-endian) and the machine is little-endian. An example conversion would be:

import struct
network_stream = struct.pack('>I', 0x12345678)
(converted_int,) = struct.unpack_from('>I', network_stream, 0)

I am less concerned about handling the stream format, than the general case of binary conversion, and if there is even an alternative to unpack. For example, socket.ntohl() requires an int, and int() won't convert a binary data string.

Thanks for your suggestions!

What exactly are you unpacking? Is it simply a uniform array of numeric values? — Sven Marnach
– Sven Marnach, Commented Nov 16, 2011 at 0:35
Unfortunately it is a mixed stream of singletons, where the contents is indicated by a header. Most of the conversion is for 4B integers and floats. — CNK
– CNK, Commented Nov 16, 2011 at 0:42
Would Cython be useful here? You can wrap a C library, and do the tight looping in compiled code. — Thomas K
– Thomas K, Commented Nov 16, 2011 at 1:13
@CNK: "contents is indicated by a header" needs explanation with examples. How many different items per header? How many unique headers per data stream? Please explain "ASCII data conversion" -- is it NUL-terminated? Show your Python code. — John Machin
– John Machin, Commented Nov 16, 2011 at 1:40
@Triptych: I've added a simple case, float and longs are similar. — CNK
– CNK, Commented Nov 16, 2011 at 1:54

Michael Hoffman · Accepted Answer · 2011-11-16 02:15:32Z

2

The speed problem probably comes not in the implementation of struct.unpack_from() itself, but in everything else Python needs to do—dictionary lookups, create objects, call functions, and other tasks. You can speed things up ever so slightly by eliminating one of these dictionary lookups by importing unpack_from directly rather than getting it from the struct module each time:

$ python -m timeit -s "import struct; network_stream = struct.pack('>I', 0x12345678)" "(converted_int,) = struct.unpack_from('>I', network_stream, 0)" 
1000000 loops, best of 3: 0.277 usec per loop

$ python -m timeit -s "import struct; from struct import unpack_from; network_stream = struct.pack('>I', 0x12345678)" "(converted_int,) = unpack_from('>I', network_stream, 0)"
1000000 loops, best of 3: 0.258 usec per loop

However, if there needs to be a lot of parsing logic that necessitates unpacking one number at a time, and will keep you from unpacking a whole array of data in bulk, it doesn't matter what you call to do it for you. You are probably going to need to do this whole inner loop in a language with less overhead, such as C.

edited Nov 16, 2011 at 2:15

answered Nov 16, 2011 at 2:09

Michael Hoffman

34.8k7 gold badges68 silver badges91 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

CNK Over a year ago

Interesting thought. I actually used from struct import *. Converted this to mimic your example and also found a 5% performance difference over about 220M calls. Would never have expected that. Thanks!

Raymond Hettinger · Accepted Answer · 2011-11-16 02:02:58Z

2

Based on my experience, you are correct that the code will need to be moved to C. As you discovered the performance for the various tools for binary conversion (struct and ctypes for example) have roughly similar performance.

Cython is the easiest way to get generate a C extension for Python.

Another easy approach is to abandon CPython entirely in favor of pypy which can generate high quality, low-level code using its tracing JIT.

A more challenging but more direct approach is to write a plain C extension. This isn't fun but it isn't difficult.

answered Nov 16, 2011 at 2:02

Raymond Hettinger

229k67 gold badges405 silver badges504 bronze badges

2 Comments

CNK Over a year ago

Which of these methods would you estimate to have the lowest call overhead? Is the C extension going to be more efficient than using ctypes?

Raymond Hettinger Over a year ago

The C extension can always win because it has a chance to do many conversions at once (per returning the numbers in a list or in an iterator). The ctypes approach has a granularity of one call per conversion. In addition, you make not want all the data, so a C extension can skip unwanted values, saving many convert/allocate/incref/compare/jump/decref cycles.

Collectives™ on Stack Overflow

fast binary data conversion in Python

2 Answers 2

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related