Python - Issues with Unicode String from API Call

Question

I'm using Python to call an API that returns the last name of some soccer players. One of the players has a "ć" in his name.

When I call the endpoint, the name prints out with the unicode attached to it:

>>> last_name = (json.dumps(response["response"][2]["player"]["lastname"]))

>>> print(last_name)

"Mitrovi\u0107"

>>> print(type(last_name))

<class 'str'>

If I were to take copy and paste that output and put it in a variable on its own like so:

>>> print("Mitrovi\u0107")

Mitrović

>>> print(type("Mitrovi\u0107"))

<class 'str'>

Then it prints just fine?

What is wrong with the API endpoint call and the string that comes from it?

lenz · Accepted Answer · 2022-11-07 22:56:21Z

1

Well, you serialise the string with json.dumps() before printing it, that's why you get a different output. Compare the following:

>>> print("Mitrović")
Mitrović

and

>>> print(json.dumps("Mitrović"))
"Mitrovi\u0107"

The second command adds double quotes to the output and escapes non-ASCII chars, because that's how strings are encoded in JSON. So it's possible that response["response"][2]["player"]["lastname"] contains exactly what you want, but maybe you fooled yourself by wrapping it in json.dumps() before printing.

Note: don't confuse Python string literals and JSON serialisation of strings. They share some common features, but they aren't the same (eg. JSON strings can't be single-quoted), and they serve a different purpose (the first are for writing strings in source code, the second are for encoding data for sending it accross the network).

Another note: You can avoid most of the escaping with ensure_ascii=False in the json.dumps() call:

>>> print(json.dumps("Mitrović", ensure_ascii=False))
"Mitrović"

edited Nov 7, 2022 at 22:56

answered Nov 7, 2022 at 22:47

lenz

5,8585 gold badges27 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

sq89 Over a year ago

Thank you very much for your help, I didn't realize what json.dumps() was doing.

pbuck · Accepted Answer · 2022-11-07 22:31:38Z

1

Count the number of characters in your string & I'll bet you'll notice that the result of json is 13 characters:

"M-i-t-r-o-v-i-\-u-0-1-0-7", or "Mitrovi\\u0107"

When you copy "Mitrovi\u0107" you're coping 8 characters and the '\u0107' is a single unicode character.

That would suggest the endpoint is not sending properly json-escaped unicode, or somewhere in your doc you're reading it as ascii first. Carefully look at exactly what you're receiving.

answered Nov 7, 2022 at 22:31

pbuck

4,5902 gold badges28 silver badges37 bronze badges

1 Comment

sq89 Over a year ago

Ah, that makes sense as to why I saw the "\\" when trying to figure this out.

Collectives™ on Stack Overflow

Python - Issues with Unicode String from API Call

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related