Code works in Python 2 but not Python3 TypeError: a bytes-like object is required, not 'str'

Question

The following code works in Python 2.7:

import os
import pickle

modelpath = "models/"

gmm_files = [os.path.join(modelpath,fname) for fname in 
          os.listdir(modelpath) if fname.endswith('.gmm')]

models    = [pickle.load(open(fname,'r')) for fname in gmm_files]

However, when I run the code in Python3, I get the following error from the last line:

TypeError: a bytes-like object is required, not 'str'

In order to get a better idea, I tried printing print([type(open(fname,'r')) for fname in gmm_files]) in both versions and found out that in python 2 the type is <type 'file'> and in Python 3 the type is <class '_io.TextIOWrapper'>.

I've checked out these stackoverflow questions but neither of them have helpful answers for this:

python 3.5: TypeError: a bytes-like object is required, not 'str' when writing to a file

Python sockets error TypeError: a bytes-like object is required, not 'str' with send function

UPDATE

A bunch of the answers here said to change open(fname, 'r')to open(fname, 'rb') but that just leads to another error: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc0 in position 0: ordinal not in range(128)

You need to open the file in binary mode, rather than (the default) text mode: open(fname,'rb') — Tom Dalton
– Tom Dalton, Commented Jun 8, 2018 at 15:12
I also suggest using the the pathlib library rather than os. In my opinion it is a lot nicer to work with. — Bram Vanroy
– Bram Vanroy, Commented Jun 8, 2018 at 15:14
Thanks everyone. @TomDalton I updated the question to show what happens when I implement changing r to rb — Philip7899
– Philip7899, Commented Jun 8, 2018 at 15:25
Are you trying to unpickle data in python 3.x that you pickled with 2.7? Can you post the full traceback of the error? — Tom Dalton
– Tom Dalton, Commented Jun 8, 2018 at 16:03
(It's probably worth asking this as a completely new question) — Tom Dalton
– Tom Dalton, Commented Jun 8, 2018 at 16:04

ForceBru · Accepted Answer · 2018-06-08 15:16:16Z

2

As the documentation for the pickle.load method says (emphasis mine):

The argument file must have two methods, a read() method that takes an integer argument, and a readline() method that requires no arguments. Both methods should return bytes.

open(stuff, 'r') will open the file for reading text, not raw bytes. Thus, open(stuff, 'r').read will return str, not bytes. To fix that, open the file in binary mode: open(stuff, 'rb').

answered Jun 8, 2018 at 15:16

ForceBru

45k10 gold badges71 silver badges104 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Philip7899 Over a year ago

Thanks, I tried this and got the error UnicodeDecodeError: 'ascii' codec can't decode byte 0xc0 in position 0: ordinal not in range(128)

ForceBru Over a year ago

@Philip7899, since you couldn't've gotten this error from opening the file or reading data from it (unless you typo'd somewhere), this means that you did manage to start the unpickling process, which faced some encoding issues. This, however, is a different problem and doesn't have to do with the original question. You should search for its cause in the pickled data.

Tom Dalton · Accepted Answer · 2018-06-08 15:16:23Z

2

Ref https://docs.python.org/3.6/library/pickle.html#pickle.load, the file-like object you pass to pickle.load needs to return binary data. Files are opened in text-mode by default, which is why you're seeing this error. If you open the file in binary mode (by adding 'b' to the mode), everything should be work.

E.g.

models = [pickle.load(open(fname, 'rb')) for fname in gmm_files]

answered Jun 8, 2018 at 15:16

Tom Dalton

6,19027 silver badges36 bronze badges

1 Comment

Philip7899 Over a year ago

Thanks. I tried this and get another error, which I have updated the question with. When I do that, i get the error UnicodeDecodeError: 'ascii' codec can't decode byte 0xc0 in position 0: ordinal not in range(128)

Collectives™ on Stack Overflow

Code works in Python 2 but not Python3 TypeError: a bytes-like object is required, not 'str'

2 Answers 2

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related