Geocoding, iterrows() and itertuples do not get the job done for a larger DataFrame

Question

Im trying to add coördinates to a set of addresses that are saved in an excel file using the google geocoder API. See code below:

for i, row in df.iterrows():

    #below combines the address columns together in one variable, to push to the geocoder API.
    apiAddress = str(df.at[i, 'adresse1']) + ',' + str(df.at[i, 'postnr']) + ',' + str(df.at[i, 'By']) 
    
    #below creates a dictionary with the API key and the address info, to push to the Geocoder API on each iteration
    parameters = {
        'key' : API_KEY,
        'address' : apiAddress
        }
    #response from the API, based on the input url + the dictionary above.
    response = requests.get(base_url, params = parameters).json() 
    #when you look at the response, it is given as a dictionary. with this command I access the geometry part of the dictionary.
    geometry = response['results'][0]['geometry']
    
    #within the geometry party of the dictionary given by the API, I access the lat and lng respectively.
    lat = geometry['location']['lat'] 
    lng = geometry['location']['lng']
    
    #here I append the lat / lng to a new column in the dataframe for each iteration.
    df.at[i, 'Geo_Lat_New'] = lat
    df.at[i, 'Geo_Lng_New'] = lng


#printing the first 10 rows.    
print(df.head(10))

the above code works perfectly fine for 20 addresses. But when I try to run it on the entire dataset of 90000 addresses; using iterrows() I get a IndexError:

  File "C:\Users\...", line 29, in <module>
    geometry = response['results'][0]['geometry']

IndexError: list index out of range

Using itertuples() instead, with:

for i, row in df.itertuples():

I get a ValueError:

  File "C:\Users\...", line 22, in <module>
    for i, row in df.itertuples():

ValueError: too many values to unpack (expected 2)

when I use:

for i in df.itertuples():

I get a complicated KeyError. That is to long to put here.

Any suggestions on how to properly add coördinates for each address in the entire dataframe?

Answer 1

Update, in the end I found out what the issue was. The google geocoding API only handles 50 request per second. Therefore I used to following code to take a 1 second break after every 49 requests:

if count == 49:
        print('Taking a 1 second break, total count is:', total_count)
        time.sleep(1)
        count = 0

Where count keeps count of the number of loops, as soon as it hits 49, the IF statement above is executed, taking a 1 second break and resetting the count back to zero.

Answer 2

Although you have already found the error - Google API limits the amount of requests that can be done - it isn't usually good practice to use for with pandas. Therefore, I would re write your code to take advantage of pd.DataFrame.apply.

def get_geometry(row: pd.Series, API_KEY: str, base_url: str, tries: int = 0):

    apiAddress = ",".join(row["adresse1"], row["postnr"], row["By"])

    parameters = {"key": API_KEY, "address": apiAddress}

    try:
        response = requests.get(base_url, params = parameters).json()
        geometry = response["results"][0]["geometry"]
    except IndexError: # reach limit
        # sleep to make the next 50 requests, but
        # beware that consistently reaching limits could
        # further limit sending requests.
        # this is why you might want to keep track of how
        # many tries you have already done, as to stop the process
        # if a threshold has been met
        if tries > 3: # tries > arbitrary threshold
            raise
        time.sleep(1)
        return get_geometry(row, API_KEY, base_url, tries + 1)
    else: 
        geometry = response["results"][0]["geometry"]
        return geometry["location"]["lat"], geometry["location"]["lng"]

# pass kwargs to apply function and iterate over every row
lat_lon = df.apply(get_geometry, API_KEY = API_KEY, base_url = base_url, axis = 1)

df["Geo_Lat_New"] = lat_lon.apply(lambda latlon: latlon[0])
df["Geo_Lng_New"] = lat_lon.apply(lambda latlon: latlon[1])