Python Request binary content

Question

I'm trying to get JSON from a Google Trends URL, but I can't convert it to JSON because content goes as b''. How I can get this result as JSON?

My simple code:

import requests
r = requests.get('https://trends.google.ru/trends/api/stories/latest?hl=ru&tz=-180&cat=all&fi=15&fs=15&geo=RU&ri=300&rs=15&sort=0')
print(r.content)

r.content starts with:

b')]}\'\n{"featuredStoryIds":[],"trendingStoryIds":["RU_lnk_iJ8H1AAwAACP-M_ru","RU_lnk_7H7L0wAwAAAnHM_ru","RU_lnk_Q-IB1AAwAABChM_ru","RU_lnk_EErj0wAwAADzKM_ru","RU_lnk_VY2s0wAwAAD57M_ru","RU_lnk_sdUP1AAwAAC-sM_ru","RU_lnk_ILv60wAwAADa2M_ru","RU_lnk_O6j70wAwAADAyM_ru","RU_lnk_fVQS1AAwAABvMM_ru","RU_lnk_TJ8D1AAwAABP-M_ru","RU_lnk_I97F0wAwAADmvM_ru","RU_lnk_tCrq0wAwAABeSM_ru","RU_lnk_W8EA1AAwAABbpM_ru","RU_lnk_IYX90wAwAADc5M_ru","RU_lnk_bz4M1AAwAABjWM_ru","RU_lnk_EJ-...

Decoding this with the r.json() method fails:

simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

r.content is indeed the raw binary data. Have you looked at the response.json() method yet? What happens when you call that? — Martijn Pieters
– Martijn Pieters, Commented Jun 14, 2017 at 10:02
yep, simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0) — Konstantin Rusanov
– Konstantin Rusanov, Commented Jun 14, 2017 at 10:03

Martijn Pieters · Accepted Answer · 2017-06-14 10:10:27Z

3

You are contacting a Google service, and Google is prefixing JSON with some extra data to prevent JSON hijacking:

>>> import requests
>>> r = requests.get('https://trends.google.ru/trends/api/stories/latest?hl=ru&tz=-180&cat=all&fi=15&fs=15&geo=RU&ri=300&rs=15&sort=0')
>>> r.content[:10]
b')]}\'\n{"fea'

Note the )]}' and newline characters at the start.

You need to remove this extra data first and manually decode; there are no other newlines in the payload so we can just split on the newline:

import json

json_body = r.text.splitlines()[-1]
json_data = json.loads(json_body)

I used Response.text here to get decoded string data (the server sets the correct content type encoding in the headers).

This gives you a decoded dictionary:

>>> json_body = r.text.splitlines()[-1]
>>> json_data = json.loads(json_body)
>>> type(json_data)
<class 'dict'>
>>> sorted(json_data)
['date', 'featuredStoryIds', 'hideAllImages', 'storySummaries', 'trendingStoryIds']

edited Jun 14, 2017 at 10:10

answered Jun 14, 2017 at 10:05

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Konstantin Rusanov Over a year ago

TypeError: the JSON object must be str, not 'bytes'

Martijn Pieters Over a year ago

@KonstantinRusanov: ah, older Python 3 version, 3.6 accepts bytes. Will update.

Konstantin Rusanov Over a year ago

Perfect! Thanks a lot

Lena Weber · Accepted Answer · 2017-06-14 10:12:55Z

-2

Maybe try this it it might help:

 import requests
    r = requests.get('https://trends.google.ru/trends/api/stories/latest?hl=ru&tz=-180&cat=all&fi=15&fs=15&geo=RU&ri=300&rs=15&sort=0')
    page=r.status_code
    print page

answered Jun 14, 2017 at 10:12

Lena Weber

2722 silver badges4 bronze badges

Collectives™ on Stack Overflow

Python Request binary content

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related