UnicodeEncodeError Converting Requests Text to JSON in Python

Question

I am trying to scrape website but with certain searches it gives me error:

File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 5399-5400: character maps to <undefined>

My Python script is:

import requests
import json

url = 'https://www.protocols.io/api/v3/protocols?filter=%20public%20&order_field=relevance&key=%20gel%22electrophoresis%20'

r = requests.get(url)
text = r.text 
jason = json.loads(text)
print (jason)

I have tried to change the encoding as suggested here by

text = text.encode('utf-8')

But still recieve the same error. It seems to be using the cp1252 encoding not utf-8?

Any suggestions will be greatly appreciated!

Update: This seems to be a issue in git bash or atom as works without error in Spyder or windows cmd

Edouard Yonga · Accepted Answer · 2020-05-20 10:29:55Z

1

Maybe is because you haven't imported sys,

like this:

import sys
import codecs

import requests
import json

url = 'https://www.protocols.io/api/v3/protocols?filter=%20public%20&order_field=relevance&key=%20gel%22electrophoresis%20'

r = requests.get(url)
text = r.text 
text = text.encode('utf-8') 
jason = json.loads(text)
print (jason)

answered May 20, 2020 at 10:29

Edouard Yonga

5175 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

JEJS Over a year ago

thanks but this doesn't seem to work for me. I still get the same error. What version of python are you using (I am on 3.7.4)

Edouard Yonga Over a year ago

python 3.6.4 i don't think is the version that disturbs How are you running the codes?

Edouard Yonga Over a year ago

i run mine on the terminal using this: python3 pytest.py

JEJS Over a year ago

I just updated to 3.8.3. I am running in atom text editor

JEJS Over a year ago

I ran on terminal but still have same error! Would you mind sharing your output?

Edouard Yonga · Accepted Answer · 2020-05-20 10:57:01Z

0

My OUTPUT: as you can see is a json of items

answered May 20, 2020 at 10:57

Edouard Yonga

5175 silver badges12 bronze badges

2 Comments

JEJS Over a year ago

Thank you. This is very wierd. I run in atom and terminal i get this error. I ran in Spyder (python 3.7) and it works, any ideas?

JEJS Over a year ago

I am going to leave it open because I think the original code works without importing sys or codecs, and i need a solution that works on terminal or atom. Thanks for the help though

Collectives™ on Stack Overflow

UnicodeEncodeError Converting Requests Text to JSON in Python

2 Answers 2

5 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related