bytes.decode() in Python2 and Python3

Question

In the source code of sqlalchemy I see following

    val = cursor.fetchone()[0]
    if util.py3k and isinstance(val, bytes):
        val = val.decode()

Why we do decode only for Python3 and doesn't do it for Python2?

At first glance, val is immediately pass to a function that assumes a str value. The same code in Python 2, being more lax about the difference between str and unicode, may not care which val is. — chepner
– chepner, Commented Aug 12, 2019 at 15:11

CristiFati · Accepted Answer · 2022-08-22 19:14:35Z

5

In Python 3, "normal" strings are Unicode (as opposed to Python 2 where they are (Extended) ASCII (or ANSI)). According to [Python 3.Docs]: Unicode HOWTO - The String Type:

Since Python 3.0, the language’s str type contains Unicode characters, meaning any string created using "unicode rocks!", 'unicode rocks!', or the triple-quoted string syntax is stored as Unicode.

Example:

Python 3:

>>> import sys
>>> sys.version
'3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)]'
>>>
>>> b = b"abcd"
>>> s = "abcd"
>>> u = u"abcd"
>>>
>>> type(b), type(s), type(u)
(<class 'bytes'>, <class 'str'>, <class 'str'>)
>>>
>>> b.decode()
'abcd'
>>> s.decode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute 'decode'
>>> u.decode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute 'decode'

Python 2:

>>> import sys
>>> sys.version
'2.7.10 (default, Mar  8 2016, 15:02:46) [MSC v.1600 64 bit (AMD64)]'
>>>
>>> b = b"abcd"
>>> s = "abcd"
>>> u = u"abcd"
>>>
>>> type(b), type(s), type(u)
(<type 'str'>, <type 'str'>, <type 'unicode'>)
>>>
>>> b.decode()
u'abcd'
>>> s.decode()
u'abcd'
>>> u.decode()
u'abcd'

val will be further passed (to _parse_server_version) as a str. Since in Python 3, bytes and str differ, the conversion is performed.

You could also check [SO]: Passing utf-16 string to a Windows function (@CristiFati's answer).

edited Aug 22, 2022 at 19:14

answered Aug 12, 2019 at 16:27

CristiFati

41.5k9 gold badges68 silver badges115 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

tkazik Over a year ago

One of the nicest comparisons I have come across...great code samples!

mootmoot · Accepted Answer · 2019-08-13 10:51:46Z

1

You can check out a detail documentation of string encoding frustration here.

In short, since SQLAlchemy contains legacy API that parses the data into bytes data, the said statement is a simple way to migrate the string bytes data to Unicode in python 3.

edited Aug 13, 2019 at 10:51

answered Aug 12, 2019 at 15:44

mootmoot

13.2k5 gold badges49 silver badges44 bronze badges

Collectives™ on Stack Overflow

bytes.decode() in Python2 and Python3

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related