HItting query limit using Geocoding API from App Engine

1,189 views
Skip to first unread message

Joe Tyson

unread,
Apr 6, 2011, 12:25:26 PM4/6/11
to google-a...@googlegroups.com
Hello,

This seems to be more of a problem with the geocoding api than app engine, but I'm unable to get ahold of anyone on that team so I'll post here instead.

The new Geocoding API doesn't do any form of url signing or api keys for non-premier accounts and resorts to using IP address to control quotas. Since App Engine uses a shared IP pool, my app gets clumped in the quota limitations of other apps. This must be a somewhat frequent problem for other apps, since 90% of my geocoding attempts get an 'OVER_QUERY_LIMIT' message. These last few days, it has been closer to 100% at 500-600 queries spread out over an 15-20 hours.

I'll add that I don't think using a pool of IPs is at all the issue. Any large scale deployment would be hitting the Geocoding API from a pool of IPs, and if the pool were dedicated to a single service, the service would be circumventing the Geocoding API quotas anyway.

Is anyone else on the list having this issue? Have you been able to work with the geocoding team to get auth tokens that don't require having a premier account?

joe

nickmilon

unread,
Apr 6, 2011, 5:44:14 PM4/6/11
to Google App Engine
May be you can delegate the geocoding job to the client side using
js ?
also you can take a look here : http://gaengine.blogspot.com/2010/11/world-countries-and-ip-geocoding-api.html

Nick ;-)

Joe Tyson

unread,
Apr 6, 2011, 5:59:24 PM4/6/11
to google-a...@googlegroups.com
We are delegating some of that to the client now, but it is far
less than ideal. Our client is mobile, so every byte that we send
over can end up costing our users money (you would be surprised
how sensitive some people are to this).

Thanks for the resource, though. I wish country level would be
enough information for us :-(.

Doing some additional searching around, it looks this issue has
been bought up by some other users on the maps forum:
--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

Brandon Wirtz

unread,
Apr 6, 2011, 6:06:50 PM4/6/11
to google-a...@googlegroups.com

Use a Proxy.  Then you can come through your own IP-pool.

nickmilon

unread,
Apr 7, 2011, 6:20:40 PM4/7/11
to Google App Engine
IMHO a proxy will complicate things.
What about if GAE team gets in touch with maps V3 team and explain to
them the issue so may be they can rate limit all GAE originated appls
by app id which is a very secure method since app id can't be
hacked ?

@Joe sorry my api isn't that useful in your case.
Regards
Nick



On Apr 7, 1:06 am, "Brandon Wirtz" <drak...@digerat.com> wrote:
> Use a Proxy.  Then you can come through your own IP-pool.
>
> From: google-a...@googlegroups.com
> [mailto:google-a...@googlegroups.com] On Behalf Of Joe Tyson
> Sent: Wednesday, April 06, 2011 2:59 PM
> To: google-a...@googlegroups.com
> Subject: Re: [google-appengine] Re: HItting query limit using Geocoding API
> from App Engine
>
> We are delegating some of that to the client now, but it is far
>
> less than ideal. Our client is mobile, so every byte that we send
>
> over can end up costing our users money (you would be surprised
>
> how sensitive some people are to this).
>
> Thanks for the resource, though. I wish country level would be
>
> enough information for us :-(.
>
> Doing some additional searching around, it looks this issue has
>
> been bought up by some other users on the maps forum:
>
> http://goo.gl/RyJ9w
>
> On Wednesday, April 6, 2011 at 2:44 PM, nickmilon wrote:
>
> May be you can delegate the geocoding job to the client side using
> js ?
> also you can take a look here :http://gaengine.blogspot.com/2010/11/world-countries-and-ip-geocoding...
> ml
> For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en.

Brandon Wirtz

unread,
Apr 7, 2011, 7:29:10 PM4/7/11
to google-a...@googlegroups.com
"you get what you pay for" I mean the GeoCoding API, not GAE. I think the
more ideal scenario is GeoCoding and other Google API's should require
registration and authentication.

Similar to another thread about blocking all of GAE, in many cases GAE
doesn't work with Free API's because of the IP limits and the number of
people exploiting GAE as a Proxy.

I will be the first to admit that my first Application on GAE was a simple
URL proxy for gaining access to GAE's IP Address Pool. Us early grey hats
peed in the pool and ruined things. (yes I suck but I'm trying to suck
less).

Ikai Lan (Google)

unread,
Apr 8, 2011, 3:31:39 AM4/8/11
to Google App Engine
I'll admit that I don't understand the maps API to be an authority on this, but the last time I spoke to them, they mentioned some kind of a key you can get a hold of? With this key you can make signed requests that can be whitelisted. Otherwise, you're put into the general pool of IPs and subject to throttling.

Ikai Lan 
Developer Programs Engineer, Google App Engine

bFlood

unread,
Apr 8, 2011, 7:58:13 AM4/8/11
to Google App Engine

nickmilon

unread,
Apr 8, 2011, 6:30:54 PM4/8/11
to Google App Engine
@bFlood : Maps Premium is an expensive service used by not public-
facing, password protected Web sites, while what we are talking about
here is the free Google Maps service, where G is penalizing all Apps
running on top of GAE since they have to share the ip addresses pool
of GAE ip adresses.

@Ikai : what you write applies to old maps (V2) API where you can
obtain an application authorization key and applications are rate
limited based on this key, although I feel that some kind of ip based
limitations exists also.
Maps V3 which is the way to go especially for mobile appls, do not
require an application key, instead there are rate limitations based
just on originating IP addresses, this puts GAE based appls on a
disadvantage since we have to share this with all other GAE based
appls using the service.

Regards
Nick



On Apr 8, 2:58 pm, bFlood <bfl...@spatialdatalogic.com> wrote:
> Google Maps Premiumhttp://www.google.com/enterprise/earthmaps/maps.html
>
> Dev guide for Url signinghttp://code.google.com/apis/maps/documentation/premier/guide.html

Jeff Schnitzer

unread,
Apr 8, 2011, 8:32:21 PM4/8/11
to google-a...@googlegroups.com
On Thu, Apr 7, 2011 at 3:20 PM, nickmilon <nick...@gmail.com> wrote:
> IMHO a proxy will complicate things.
> What about if GAE team gets in touch with maps V3 team and explain to
> them the issue so may be they can rate limit all GAE originated appls
> by app id which is a very secure method since app id can't be
> hacked ?

Unfortunately this would incentivize the rest of the world to start
*adding* GAE headers to work around maps' ratelimits. But combined
with knowledge of the IP pool that appengine fetches from, it's a
great idea.

BTW, a proxy is pretty trivial to set up. And you can use it for
other services that have rate issues as well - I've run into several.

Jeff

nickmilon

unread,
Apr 9, 2011, 4:21:06 PM4/9/11
to Google App Engine
Jeff
You are right Google IP pool + app id is what is needed to make rate
limiting unhackable.


On Apr 9, 3:32 am, Jeff Schnitzer <j...@infohazard.org> wrote:

David Wursteisen

unread,
Jul 6, 2011, 2:50:39 AM7/6/11
to google-a...@googlegroups.com
is an issue was created for this problem ?

Maybe we can create one if not.

http://code.google.com/p/gmaps-api-issues/


2011/4/9 nickmilon <nick...@gmail.com>

Brandon Wirtz

unread,
Jul 6, 2011, 3:19:17 AM7/6/11
to google-a...@googlegroups.com

External Proxy, issue solved.  Not to be harsh, but all the API’s that are IP rate limited have this. 

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of David Wursteisen
Sent: Tuesday, July 05, 2011 11:51 PM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] Re: HItting query limit using Geocoding API from App Engine

 

is an issue was created for this problem ?

Brandon Wirtz

unread,
Jul 6, 2011, 4:44:00 AM7/6/11
to google-a...@googlegroups.com

Why is an external Proxy a bottle neck to scale? If there is an API limit then as long as your proxy can handle the volume of the limit then it is not a bottleneck.

 

We have talked before about the fact that you can extend the number of Geocodes by building your own database of results, and by using rounding to limit the coding to the accuracy you need.

 

If you need more scale Pony up for Eagle or Maxmind.

 

Again you are looking to solve a problem by making a service you depend on change their behavior.  This is the exact opposite of building for scale.  When you require others to change their behavior you put your fate in their hands.

 

-Brandon

 

 

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of David Wursteisen
Sent: Wednesday, July 06, 2011 1:24 AM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] Re: HItting query limit using Geocoding API from App Engine

 

I don't agree.

Used an external proxy will act as a bottleneck in a scallable system. And you'll have to manage this proxy too. (And I don't want to have to)

It's a good workaround, like use the V2 of Map API. But for me, it's not the right fix.

And I'll be happy with an "official" statement about this issue, even if it's a "won't fix"

Regards

2011/7/6 Brandon Wirtz <dra...@digerat.com>

David Wursteisen

unread,
Jul 6, 2011, 4:24:10 AM7/6/11
to google-a...@googlegroups.com
I don't agree.

Used an external proxy will act as a bottleneck in a scallable system. And you'll have to manage this proxy too. (And I don't want to have to)

It's a good workaround, like use the V2 of Map API. But for me, it's not the right fix.

And I'll be happy with an "official" statement about this issue, even if it's a "won't fix"

Regards

2011/7/6 Brandon Wirtz <dra...@digerat.com>

External Proxy, issue solved.  Not to be harsh, but all the API’s that are IP rate limited have this. 

Jeff Schnitzer

unread,
Jul 6, 2011, 6:26:07 AM7/6/11
to google-a...@googlegroups.com
Plus... if you're driving enough requests through even a single proxy
to swamp it, you're almost certainly going to bump into the geocode
rate limits again. The scalability limit isn't the proxy, it's the
rate limit.

In any case, this is probably something to take up with the Maps team.

Jeff

Jeremy McLain

unread,
Dec 17, 2013, 2:37:05 PM12/17/13
to google-a...@googlegroups.com
An issue has been opened.  Actually is a feature request: https://code.google.com/p/gmaps-api-issues/issues/detail?id=2844

Jeremy McLain

unread,
Dec 17, 2013, 5:17:50 PM12/17/13
to google-a...@googlegroups.com
This is driving me crazy.  I've tried three different proxies and I am still getting: "You have exceeded your daily request quota for this API."

Vinny P

unread,
Dec 17, 2013, 10:46:01 PM12/17/13
to google-a...@googlegroups.com
On Tue, Dec 17, 2013 at 4:17 PM, Jeremy McLain <gongch...@gmail.com> wrote:
This is driving me crazy.  I've tried three different proxies and I am still getting: "You have exceeded your daily request quota for this API."
 

What proxies are you using? Are these proxies giving you new IP addresses when you use them, or are they sharing a pool of IPs?

What you can try doing is using the Compute Engine API to open up a new VM and then proxy your requests through it (the machine will have its own IP, which hopefully will still have an empty Geocoding quota). 
 
 
-----------------
-Vinny P
Technology & Media Advisor
Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

bFlood

unread,
Dec 20, 2013, 10:08:38 AM12/20/13
to google-a...@googlegroups.com
how many requests are you making? the per IP limits are still pretty small...
Reply all
Reply to author
Forward
0 new messages