Save data from API to csv in Python Pandas

Question

I need help with saving data I read with API key to csv. The code I have is below:

import requests
import pandas as pd
def get_precip(gooddate):
    urlstart = 'http://api.wunderground.com/api/API_KEY/history_'
    urlend = '/q/Switzerland/Zurich.json'
    url = urlstart + str(gooddate) + urlend
    data = requests.get(url).json()

    for summary in data['history']['dailysummary']:
        abc = ','.join((gooddate,summary['date']['year'],summary['date']['mon'],summary['date']['mday'],summary['precipm'], summary['maxtempm'], summary['meantempm'],summary['mintempm']))
        df = pd.DataFrame(data=abc)
        df.to_csv('/home/user/Desktop/2013_weather.csv', index=False)

if __name__ == "__main__":
    from datetime import date
    from dateutil.rrule import rrule, DAILY

    a = date(2013, 1, 1)
    b = date(2013, 12, 31)

    for dt in rrule(DAILY, dtstart=a, until=b):
        get_precip(dt.strftime("%Y%m%d"))

I'm sure that can't work this way, because it need to be saved into some list or dictionary before transform into dataframe, but not sure how to do that this time. If save it to the list it will give me just one row? Any help is welcomed. Thanks.

Take a look at the read_json function of pandas. It also accepts urls as input as stated in the docs. — error
– error, Commented Dec 19, 2017 at 13:05

jezrael · Accepted Answer · 2017-12-19 14:24:11Z

1

I think you can return tuples from get_precip, append them to list and use DataFrame constructor:

def get_precip(gooddate):
    urlstart = 'http://api.wunderground.com/api/API_KEY/history_'
    urlend = '/q/Switzerland/Zurich.json'
    url = urlstart + str(gooddate) + urlend
    data = requests.get(url).json()

    for summary in data['history']['dailysummary']:
        return (gooddate,summary['date']['year'],summary['date']['mon'],summary['date']['mday'],summary['precipm'], summary['maxtempm'], summary['meantempm'],summary['mintempm'])

if __name__ == "__main__":
    from datetime import date
    from dateutil.rrule import rrule, DAILY

    a = date(2013, 1, 1)
    b = date(2013, 12, 31)

    L = []
    for dt in rrule(DAILY, dtstart=a, until=b):
        tup = get_precip(dt.strftime("%Y%m%d"))
        L.append(tup)

what is same as:

    L = [get_precip(dt.strftime("%Y%m%d")) for dt in rrule(DAILY, dtstart=a, until=b)]

    cols = ['date','date.year','date.mon','date.mday','precipm','maxtempm', 
            'meantempm','mintempm']     
    df = pd.DataFrame(L, columns=cols)
    print (df.head())

           date date.year date.mon date.mday precipm maxtempm meantempm mintempm
    0  20130101      2013       01        01     0.0        7         2       -2
    1  20130102      2013       01        02     0.0        5         2       -3
    2  20130103      2013       01        03     0.0        4         0       -3
    3  20130104      2013       01        04     0.0        7         5        3

   df.to_csv('/home/user/Desktop/2013_weather.csv', index=False)

edited Dec 19, 2017 at 14:24

answered Dec 19, 2017 at 13:37

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

jovicbg Over a year ago

Thanks man, I think it's pratty much good, but I got an error : File "weather_req.py", line 35, in get_precip for summary in data['history']['dailysummary']: KeyError: 'history' . Do you know what that could be?

jezrael Over a year ago

Do you change API_KEY ?

jezrael Over a year ago

Hmmm, I think there should be reason API is not filtered only for some count of rows? I test it only with first 10, b = date(2013, 1, 10) and it working perfectly.

jovicbg Over a year ago

I think I find the problem. Zurich is provided, but some other cities that I need (Zurich was example) are not. Belgrade, Serbia (on this website is still Yugoslavia) is not, it doesn't have a history. Can you please try with Yugoslavia/Belgrade to be sure?

jezrael Over a year ago

Unfortunately now I am on phone only. So tomorow it will be possible only.

|

Collectives™ on Stack Overflow

Save data from API to csv in Python Pandas

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related