Problem reading hex data from image - python automatically converts to a string

Question

I am reading in an image one byte at a time with with read(1), and appending it to a list. The image data is all hex data. When I print out the list with the print function it is in the format '\xd7'

['\xd7', '\xd7', '\xd7', '\xd7', '\xd7', '\xd7', '\xd7',...]

The problem is that now I need to perform some calculations on this hex data, however, it is in string format, and this '\xd' string format isn't supported by any of the int or hex conversion functions in python. They require a '0xd7' or just a 'd7'.

Thanks for the help

I don't know what you're trying to do, but are you aware of the Python Imaging Library? It may be easier than manipulating image data yourself. pythonware.com/products/pil — Thomas K
– Thomas K, Commented Nov 22, 2010 at 22:31

Thomas K · Accepted Answer · 2010-11-22 22:28:39Z

4

It's interpreting them as characters, so use ord to turn them into numbers. I.e. ord('\xd7') gives 215.

Also if you use Windows, or the program might have to run on Windows, make sure that you've got the file open in binary mode: open("imagefile.png","rb"). Makes no difference on other operating systems.

edited Nov 22, 2010 at 22:28

answered Nov 22, 2010 at 22:15

Thomas K

40.7k7 gold badges88 silver badges89 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

the Tin Man Over a year ago

b applies on Windows, but not on Linux or Mac.

Thomas K Over a year ago

Thanks, I didn't know that. All the same, it's good practice to write code that will work the same anywhere.

John Machin Over a year ago

@Greg: b is not necessary on *x boxes. However you should always use b so that your code is portable and the intent is clear anyone reading your code.

Chris Morgan · Accepted Answer · 2010-11-23 00:19:50Z

2

If you require 'd7' or '0xd7', rather than simply 0xd7 (viz, 215), hex() or '%x' are your friend.

>>> ord('\xd7')
215
>>> ord('\xd7') == 215 == 0xd7
True
>>> hex(ord('\xd7'))
'0xd7'
>>> '%x' % ord('\xd7')
'd7'

Also as observed in other answers, do make sure you open with the 'b' in the mode, otherwise it can get messed up, thinking it's UTF-8 or something like that, on certain sequences of bytes.

answered Nov 23, 2010 at 0:19

Chris Morgan

91.6k28 gold badges217 silver badges220 bronze badges

Comments

martineau · Accepted Answer · 2010-11-23 14:40:27Z

2

You could do something like this to get them into a numeric array:

import array

data = array.array('B') # array of unsigned bytes

with open("test.dat", 'rb') as input:
    data = input.read(100)
    data.fromstring(data)

print data
# array('B', [215, 215, 215, 215, 215, 215, 215])

edited Nov 23, 2010 at 14:40

answered Nov 22, 2010 at 23:11

martineau

124k29 gold badges181 silver badges319 bronze badges

2 Comments

Scott Griffiths Over a year ago

Though for Python 2.6+ using an array for byte data is over complicated - just use a bytearray: b = bytearray(input.read(100)), nothing to import and you can use plain indexing without ord. (Also best to avoid hiding the built in bytes type).

martineau Over a year ago

@Scott Griffiths: I initially wrote this using bytearray, but noticed they print like strings -- and decided the "complications" were worth it, plus the array module has been around for a long, long time so this will work with many more of the older versions of Python [still] out there.

the Tin Man · Accepted Answer · 2010-11-22 22:31:01Z

1

read() can take a size value larger than 1: read(1024) will read 1K worth of bytes from the stream. That will be a lot faster than reading a byte at a time and appending it to the previous bytes.

What are you trying to do when printing the data? See the byte values, or display the image?

The data isn't in "string format", it's just bytes, but when you print them the print routine will escape non-printing values into something that will mean more to human eyes and brains. If you want to see the values without the escaping you can iterate over the bytes and convert them to their hexadecimal values, or decimal, or binary - whatever works for you and your application. The string formatting mini-language will be a good starting place.

edited Nov 22, 2010 at 22:31

answered Nov 22, 2010 at 22:24

the Tin Man

161k44 gold badges222 silver badges308 bronze badges

Comments

luispedro · Accepted Answer · 2010-11-23 00:12:52Z

0

If you are doing image processing, then you probably want to look at numpy.

There are a few packages that will help you read your image into memory too (PIL is mentioned above, another is my own mahotas or scikits.image).

If the data is in a file as raw data an you know the dimensions, you can do the following

import numpy as np
img = np.empty( (n_rows, n_cols), dtype=np.uint8) # create an empty image
img.data[:] = input_file.read()

to get your data into img.

An introductory website for image processing in python is http://pythonvision.org.

answered Nov 23, 2010 at 0:12

luispedro

7,0345 gold badges37 silver badges46 bronze badges

Collectives™ on Stack Overflow

Problem reading hex data from image - python automatically converts to a string

5 Answers 5

3 Comments

Comments

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

3 Comments

Comments

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related