Python JSON decoder error with unicode characters in request content

Question

Using requests library to execute http GET that return JSON response i'm getting this error when response string contains unicode char:

json.decoder.JSONDecodeError: Invalid control character at: line 1 column 20 (char 19)

Execute same http request with Postman the json output is:

{ "value": "VILLE D\u0019ANAUNIA" }

My python code is:

data = requests.get(uri, headers=HEADERS).text
json_data = json.loads(data)

Can I remove or replace all Unicode chars before executing conversion with json.loads(...)?

see if this one helps stackoverflow.com/questions/15321138/… or stackoverflow.com/questions/2234228/… — rgaduputi
– rgaduputi, Commented Feb 20, 2019 at 10:49
Possible duplicate of stackoverflow.com/questions/27955978/… — skaul05
– skaul05, Commented Feb 20, 2019 at 10:51
What is the raw response content (print(requests.get(uri, headers=HEADERS).content))? It could be caused by an encoding problem... — Serge Ballesta
– Serge Ballesta, Commented Feb 20, 2019 at 11:21

Serge Ballesta · Accepted Answer · 2019-02-20 11:32:55Z

3

It is likely to be caused by a RIGHT SINGLE QUOTATION MARK U+2019 (’). For reasons I cannot guess, the high order byte has been dropped leaving you with a control character which should be escaped in a correct JSON string.

So the correct way would be to control what exactly the API returns. If id does return a '\u0019' control character, you should contact the API owner because the problem should be there.

As a workaround, you can try to limit the problem for your processing by filtering out non ascii or control characters:

data = requests.get(uri, headers=HEADERS).text
data = ''.join((i for i in data if 0x20 <= ord(i) < 127))  # filter out unwanted chars
json_data = json.loads(data)

You should get {'value': 'VILLE DANAUNIA'}

Alternatively, you can replace all unwanted characters with spaces:

data = requests.get(uri, headers=HEADERS).text
data = ''.join((i if 0x20 <= ord(i) < 127 else ' ' for i in data))
json_data = json.loads(data)

You would get {'value': 'VILLE D ANAUNIA'}

answered Feb 20, 2019 at 11:32

Serge Ballesta

150k13 gold badges137 silver badges267 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

orion91 Over a year ago

Thank's for suggestion! For now i don't need replace unicode chars but maybe it could be useful in the future!

balderman · Accepted Answer · 2019-02-20 11:03:24Z

2

The code below works on python 2.7:

import json
d = json.loads('{ "value": "VILLE D\u0019ANAUNIA" }')
print(d)

The code below works on python 3.7:

import json
d = json.loads('{ "value": "VILLE D\u0019ANAUNIA" }', strict=False)
print(d)

Output:

{u'value': u'VILLE D\x19ANAUNIA'}

Another point is that requests get return the data as json:

r = requests.get('https://api.github.com/events')
r.json()

answered Feb 20, 2019 at 11:03

balderman

24k8 gold badges39 silver badges60 bronze badges

1 Comment

orion91 Over a year ago

With parameter strict=False no exception was throwed! Thank's

Collectives™ on Stack Overflow

Python JSON decoder error with unicode characters in request content

2 Answers 2

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related