Geocoding in Python

Question

I am using the geocoder package in Python to obtain the coordinates for a set of addresses (around 30k). I get the following error:

Status code Unknown from https://maps.googleapis.com/maps/api/geocode/json: ERROR - HTTPSConnectionPool(host='maps.googleapis.com', port=443): Max retries exceeded with url: /maps/api/geocode/json?address=Rancho+Palos+Verdes%2CCA%2CUS&bounds=&components=&region=&language= (Caused by ProxyError('Cannot connect to proxy.', timeout('timed out',)))

The number of times I receive the error reduces if I add the time.sleep(x) function but it significantly increases the time taken to execute the code. Is there a more efficient way to run the code?

Following is a snippet of the code:

for add in clean_address:
    g=geocoder.google(add)
    time.sleep(7)
    if(g.ok==True):
        lat.append(str(g.lat))
        lon.append(str(g.lng))
    if(g.ok==False):
        lat.append("")
        lon.append("")

Google's geocode API is only free up to a certain amount, you're probably exceeding that and getting rate-limited. If you want more you'll have to sign up and pay for your volume. — tzaman
– tzaman, Commented Dec 7, 2017 at 16:06
Is there a reason to use google api? you could use arcgis api that haven't use limitation: g=geocoder.arcgis(add) — Lupanoide
– Lupanoide, Commented Dec 7, 2017 at 16:07
@Lupanoide - Thanks for the recommending me ArcGIS. Its more accurate and faster in getting the coordinates for my 31k records. — Mazahir Bhagat
– Mazahir Bhagat, Commented Dec 8, 2017 at 17:41
@Lupanoide - could you post your comment as an answer so that I can accept it? — Mazahir Bhagat
– Mazahir Bhagat, Commented Feb 9, 2018 at 17:46

Lupanoide · Accepted Answer · 2018-03-06 13:24:14Z

3

Google API has a use limitation. However you could even use geocoder library that collects a lot of geocoding services. I suggest to you to use the ArcGis api that hasn't any limitation on usage and is very accurate. The usage is very simple:

g=geocoder.arcgis(add)
lat.append(g.x)
lon.append(g.y)

edited Mar 6, 2018 at 13:24

answered Feb 9, 2018 at 18:49

Lupanoide

3,24225 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Dennis M. Gray Over a year ago

arcgis still times out after several hundred records. I am a fair newbie at python and would like to know a method to catch this error and keep going

mormegil · Accepted Answer · 2020-10-03 03:44:00Z

2

This is an error originating in the Python requests library used by the geocoder package. The error itself is likely a symptom of exceeding a quota limit configured on your project's enabled API. There are short term quotas (per 100 seconds) and long-term quotas (per day) that you might be running into.

An improvement to automatically retry with exponential backoff may help if the error is due to short-term quota being exceeded. To do this, explicitly create a Session with a custom Retry, and use that session with the geocoder library:

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

...

# Retry for up to 65 seconds
ssn = requests.Session()
adp = HTTPAdapter(max_retries=Retry(total=16, backoff_factor=0.001))
ssn.mount('https://', adp)
ssn.mount('http://', adp)

...

geocoder.google(..., session=ssn)

edited Oct 3, 2020 at 3:44

answered Oct 2, 2020 at 20:29

mormegil

1,9001 gold badge20 silver badges22 bronze badges

1 Comment

David Miller Over a year ago

Great answer but a follow up: What is the way to add retry functionality in R? I have R code which is running into the same problem...

raratiru · Accepted Answer · 2017-12-07 19:53:57Z

1

Your addresses are 30K and the limit is 2,5K per day.

As Google quotes in the relevant docs:

To use the Google Maps Geocoding API, you must register your app project on the Google API Console and get a Google API key which you can add to your app or website.

After doing so, these are the limits:

Users of the standard API:

2,500 free requests per day, calculated as the sum of client-side and server-side queries.

50 requests per second, calculated as the sum of client-side and server-side queries.

edited Dec 7, 2017 at 19:53

answered Dec 7, 2017 at 16:10

raratiru

9,7864 gold badges83 silver badges119 bronze badges

3 Comments

raratiru Over a year ago

Is there a downvoting police in S.O.? I need to sue somebody.

MattR Over a year ago

no police. but i'll balance it out for you. I believe this answers the OP's question

Mad Physicist Over a year ago

The downvoters are the police (part of it anyway). Don't worry, the average statistics will work out in your favor. +1

Collectives™ on Stack Overflow

Geocoding in Python

3 Answers 3

1 Comment

1 Comment

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related