Why does indexing bytes in Python 3 return an int instead of bytes?

Question

Trying to understand why taking the index of a bytes object returns an int that you can't decode, but a slice returns a bytes object that you can. This seems un-intuitive. When you do the same operation with a string, taking an index at the string position still returns a string.

Working on the Cryptopals challenges, I'm trying to iterate over a byte array to do frequency analysis of an XORed string to count the occurrence of the number of plain text letters. I thought I could do the following, but I'm getting 'int' object has not attribute 'decode' error. From reading the Python docs, that makes sense, a byte array is a mutable sequence of integers, but when testing in the interpreter I was expecting different behavior.

str_a = bytearray(b'\x1b77316?x\x15\x1b\x7f+x413=x9x(7-6<x7>x:9;76')

for x in str_a:
    _ = x.decode('ascii').upper() 
    if _ in counts:
        counts[_] += 1

If I set a variable to a single byte, I can call decode() on it. I figured I could then iterate over all bytes in a byte string and decode in the same way (hence the loop above). But, since r[0] is an int, that doesn't work. But then if I take r[0:1], it does? I realize I could just call chr(r[0]), but I figured if r.decode() works, r[0].decode() should work too.

>>> r = b'A'
>>> type(r)
<class 'bytes'>
>>> r.decode('ascii')
'A'
>>> r[0:1]
b'A'
>>> r[0:1].decode('ascii')
'A'
>>> type(r[0])
<class 'int'>
>>> r[0]
65
>>> r[0].decode('ascii')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute 'decode'

String Example

>>> x = 'AB'
>>> type(x)
<class 'str'>
>>> x[0]
'A'
>>> type(x[0])
<class 'str'>

This appears to be one of Python3's least sensible design decisions. What happens if you want to do bit manipulation? You have to use C++, maybe. — user2138149
– user2138149, Commented Mar 24, 2021 at 16:19

jirassimok · Accepted Answer · 2020-05-10 16:17:03Z

3

The reason that bytes objects iterate as ints is because they are conceptually static arrays of bytes—integers from 0 to 255 (as per PEP 358)—not alternate-encoded strings. Alternate text encodings is a common use case, but reading and writing arbitrary binary data is no less important.

Specifically regarding str.encode and bytes.decode, it doesn't necessarily make sense to be able to call some_bytes[0].decode, because in many encodings, characters may be encoded as multiple bytes. For instance, b'a'.decode('utf-32') fails because UTF-32 uses four bytes per character.

The discussions around PEP 467, which proposed adding, among other things, bytes.iterbytes, might provide additional insight into why bytes behaves how it does.

answered May 10, 2020 at 16:17

jirassimok

4,3892 gold badges19 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

gr0k Over a year ago

Huh. That makes a ton of sense. I've only just started messing with it and handling data where a single byte is encoded. I hadn't considered it from the multi-byte perspective. Thanks!

Andrew · Accepted Answer · 2020-05-10 00:31:46Z

2

As you've found out the bytes iterator generates integers not characters. You need to use chr to convert the int value to an str value.

import collections
import string

str_a = b"\x1b77316?x\x15\x1b\x7f+x413=x9x(7-6<x7>x:9;76"
count = collections.defaultdict(int)

for value in str_a:
    value = chr(value).upper()
    if value in string.ascii_uppercase:
        count[value] += 1

answered May 10, 2020 at 0:31

Andrew

3961 silver badge5 bronze badges

2 Comments

gr0k Over a year ago

Yea, I guess I'm just confused as to why. Like, that seems counter-intuitive to what I'd expect. I'd figure if I'm working on a bytes array, the objects in it would be bytes, and I could call bytes methods on individual elements in them. This gets double confusing when you can do b'A'.decode(), and b'A'[0:1].decode(), but not b'A'[0].decode(). It seems like it "breaks" iterators

gr0k Over a year ago

PEP 358 wasn't helpful either. Guess it just is what it is. oh well

Collectives™ on Stack Overflow

Why does indexing bytes in Python 3 return an int instead of bytes?

2 Answers 2

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related