python requests not returning json data

Question

I would like to get the json data from for instance https://app.weathercloud.net/d0838117883#current using python requests module. I tried:

    import re
    import requests
    
    device='0838117883'
    URL='https://app.weathercloud.net'
    URL1=URL+'/d'+device
    URL2=URL+'/device/stats'
    
    headers={'Content-Type':'text/plain; charset=UTF-8',
    'Referer':URL1,
    'User-Agent':'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/48.0.2564.82 Chrome/48.0.2564.82 Safari/537.36',
    'Accept':'application/json, text/javascript,*/*'}
    
    with requests.Session() as s:
        #get html from URL1 in order to get the CSRF token
        page = s.get(URL1)
        CSRF=re.findall('WEATHERCLOUD_CSRF_TOKEN:"(.*)"},',page.text)[0]
        #create parameters for URL2, in order to get the json file
        params={'code':device,'WEATHERCLOUD_CSRF_TOKEN':CSRF}
        page_stats=requests.get(URL2,params=params,headers=headers)
        print(page_stats.url)
        print(page_stats) #<Response [200]>
        print(page_stats.text) #empty
        print(page_stats.json()) #error

But the page_stats is empty. How can I get the stats data from weathercloud?

BlackMath · Accepted Answer · 2021-09-28 15:32:17Z

1

Inspecting the page with DevTools, you'll find a useful endpoint: https://app.weathercloud.net/device/stats

You can "replicate" the original web request made by your browser with requests library:

import requests

cookies = {
    'PHPSESSID': '************************',
   'WEATHERCLOUD_CSRF_TOKEN':'***********************',
    '_ga': '**********',
    '_gid': '**********',
    '__gads': 'ID=**********',
    'WeathercloudCookieAgreed': 'true',
    '_gat': '1',
    'WEATHERCLOUD_RECENT_ED3C8': '*****************',
}

headers = {
    'Connection': 'keep-alive',
    'sec-ch-ua': '^\\^Google',
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'X-Requested-With': 'XMLHttpRequest',
    'sec-ch-ua-mobile': '?0',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36',
    'sec-ch-ua-platform': '^\\^Windows^\\^',
    'Sec-Fetch-Site': 'same-origin',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Dest': 'empty',
    'Referer': 'https://app.weathercloud.net/d0838117883',
    'Accept-Language': 'it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7,es;q=0.6',
}

params = (
    ('code', '0838117883'),
    ('WEATHERCLOUD_CSRF_TOKEN', '****************'),
)

response = requests.get('https://app.weathercloud.net/device/stats', headers=headers, params=params, cookies=cookies)
# Serializing json
json_object = json.loads(response.text)

json Output:

{'last_update': 1632842172,
 'bar_current': [1632842172, 1006.2],
 'bar_day_max': [1632794772, 1013.4],
 'bar_day_min': [1632845772, 1006.2],
 'bar_month_max': [1632220572, 1028],
 'bar_month_min': [1632715572, 997.3],
 'bar_year_max': [1614418512, 1038.1],
 'bar_year_min': [1615434432, 988.1],
 'wdir_current': [1632842172, 180],
  ..............}

That's it.

edited Sep 28, 2021 at 15:32

answered Sep 28, 2021 at 14:36

BlackMath

1,8561 gold badge13 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

BlackMath Over a year ago

i edited my answer

charlie068 Over a year ago

Thank you! Works perfectly.

Collectives™ on Stack Overflow

python requests not returning json data

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related