Using a function to create a dataframe from a webscrape

Question

I have a function that takes in longitude, latitude, and UNIX time format. And outputs a single row dataframe with weather-related columns

['time', 'summary', 'icon', 'precipIntensity', 'precipProbability','precipType', 'temperature', 'apparentTemperature', 'dewPoint','humidity', 'pressure', 'windSpeed', 'windBearing', 'cloudCover','uvIndex', 'visibility']

 def get_weather(latitude,longitude,unix):
        url = "https://dark-sky.p.rapidapi.com/"+latitude+','+longitude+','+unix
        headers = {
        'x-rapidapi-key': "xxxxxxxxxxxxxxMYKEYxxxxxxxxxxxxxxx",
        'x-rapidapi-host': "dark-sky.p.rapidapi.com"}
        response = requests.request("GET", url, headers=headers)
        data = response.json()
        weather = data['currently']
        weather = pd.DataFrame(weather, index=[0])

I would like to iterating through my dataset (10000 rows) and creating a new dataset with all the corresponding weather data for each row.

first create list with all rows (using append()) and later convert it to DataFrame — furas
– furas, Commented Dec 30, 2020 at 4:48

shravan · Accepted Answer · 2020-12-30 07:05:09Z

1

From what I understand, you have a dataset with information about latitude, longitude and unix and you want to iterate through that to create a new dataframe using the above function

Lets say your location dataframe is

location_df = pd.DataFrame([[10,10,5], [2,3,8], [9,9,10]],
columns=['lat','long','unix'])

To iterate through each row, use df.iterrows() and use append with ignore_index=True for auto-incrementing the index. In your case, suppose the function returns the weather dataframe, then:

precipitation_df = pd.DataFrame(columns=['precipIntensity', 'precipProbability','temp']) # assume 3 values returned
for index, row in location_df.iterrows():
    latitude = row['lat']
    longitude = row['long']
    unix = row['unix']
    precipitation_df = precipitation_df.append(get_weather(latitude,longitude,unix), ignore_index=True)

Would love to know if anyone has a more efficient approach.

edited Dec 30, 2020 at 7:05

answered Dec 30, 2020 at 6:59

shravan

614 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Using a function to create a dataframe from a webscrape

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related