X-AppEngine-City

[Krishna Patel]

What's the most efficient way of determining city information since, at the moment, there is an X-AppEngine-Country but no X-AppEngine-City?

I was planning to get the GeoLite City database, which, uncompressed, is 30MB (not sure if it needs to be split to be under the new 30MB upload limit) and use a MapReduce job to load the database into entities.

Computationally expensive but is that the only way to do this?

[Ikai Lan (Google)]

Krishna,

It's a one time cost. How are you mapping the user to a city?

You might also want to look into other services like SimpleGeo.com or client side geocoding via Google Maps.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Jeff Schnitzer]

Unfortunately Google doesn't seem to have any products that do geoip
lookup anymore. There used to be a ClientLocation as part of the ajax
loader but the docs for that have been removed and the endpoints all
seem to return null now.

Using the HTML5 location API requires permission from the browser,
which isn't really practical for most apps.

We built an ip geolocation service on appengine using hessian and the
free maxmind db. The hard part was hacking around the 10M limit - but
that shouldn't be a problem with SDK 1.5.5 since the data file is
~27MB.

Some sort of finer-grained geoip lookup builtin to appengine would be
really, really handy.

Jeff

[Krishna Patel]

Ikai,

Uploading the data is a one-time cost, yes - but not an inexpensive one. And doing a city lookup upon each login is expensive because of the need to scan IP ranges in the Maxmind database. I assume this will require indexing but I don't know how expensive an IP/city lookup is going to be in terms of how many underlying datastore queries will need to be performed.

Third party apis are out of the question (this is a commercial app) and a professional geolocation license for Google Maps allegedly costs $10,000 - which is also out of reach.

For these reasons, it is disappointing that App Engine stopped short of implementing X-AppEngine-City - which would have saved countless developer hours and expense.

- Krishna

[Jeff Schnitzer]

On Thu, Oct 6, 2011 at 12:12 PM, Krishna Patel
<krishna...@gmail.com> wrote:
> Ikai,
> Uploading the data is a one-time cost, yes - but not an inexpensive one. And
> doing a city lookup upon each login is expensive because of the need to scan
> IP ranges in the Maxmind database. I assume this will require indexing but I
> don't know how expensive an IP/city lookup is going to be in terms of how
> many underlying datastore queries will need to be performed.

Maxmind's database doesn't work this way. It's a flat file that gets
pulled entirely into RAM. The code is horrific and it's nontrivial to
make it read from anything other than a filesystem. We had to hack it
heavily just to read from a three-way-split file.

The downside is that an app instance is slow to startup because it
needs to pull 30MB off of Google's incredibly slow filesystem. But
once it's running, lookups are pretty much instantaneous and
zero-cost.

Jeff

[Ronoaldo José de Lana Pereira]

Jeff,

Does your aproach plays well as a backend?

-Ronoaldo

[Jeff Schnitzer]

No point. It fits into a frontend instance - currently occupying around 108 MB.

I'd post my code that reconstitutes the three part file into one, but
it's pointless with the 30MB file limit raise around the corner.

Jeff

2011/10/6 Ronoaldo José de Lana Pereira <rper...@beneficiofacil.com.br>:

> Jeff,
>
> Does your aproach plays well as a backend?
>
> -Ronoaldo
>

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/tggusylTQdsJ.

[Krishna Patel]

Jeff,

Which code is horrific?

I was going to load the Maxmind database into custom entities - so I don't see why it's a big deal if it's tough to load a split file a single time. After it's loaded into custom entities all of my geo queries would run against the datastore. Maybe I'm misunderstanding you.

Krishna

[James Broberg]

Jeff,

This made me smile as we did the exact same thing. Split the maxmind
database in 3 and reconstituted the file in code.

We initialise the database in a warm up task so it gets loaded into
memory. It's quite speedy after that (but it does take up a chunk of
memory)

cheers

James

[Jeff Schnitzer]

This code is horrific: http://www.maxmind.com/app/java

It's horrific because doesn't allow for working with anything other
than a RandomAccessFile, which is the interface I had to mock in order
to get it working. It's horrific because it's full of magic numbers
and bizarre data structures without documentation.

Why do you want to run it agains the datastore? The free version fits
in frontend RAM. The "full" version will fit in backend RAM. Or run
it on a linode server. The idea of deciphering Maxmind's wacky data
structures gives me nightmares.

Jeff

[Jeff Schnitzer]

On Thu, Oct 6, 2011 at 9:17 PM, Juha K <juha.k...@gmail.com> wrote:
> How about using
> http://www.google.com/jsapi and google.loader.ClientLocation.address.city on
> the client side?

It doesn't work anymore. At least, it doesn't work for me here in the
Bay Area anymore. And Google has pulled down all documentation for
it, so presumably it has been deprecated.

Too bad, it's exactly what I want.

Jeff

[Juha K]

Yeah, I removed my post after 2 mins when I tested the API and realized that it returned null every time. You were faster :)

[Brandon Wirtz]

If someone tells me how to make money on it I will expose my API for doing
Geo-Location by IP. I never assumed anyone would pay anything... So I'm not
sure what is in it for me.

Also what level of detail do you want?

Jeff

--

[Krishna Patel]

Jeff,

I didn't even consider using their Java api. The idea of keeping the data in memory on App Engine is absurd. I'm surprised you thought that might work.

I'm following an approach similar to Ikai's in his blog post:

http://ikaisays.com/2010/08/11/using-the-app-engine-mapper-for-bulk-data-import/

As you can see, he parses the Maxmind file into City and Zip entities so that he can do lookups by Key.

Unfortunately I don't think this works to lookup a City from an IP address - because you have to scan a series of IP ranges in the underlying data (the data is not actually very complex).

Brandon, "what's in it for me" is a good question to ask - on both sides of the transaction. In my case, it's a commercial app so why would I rely on your service? Who are you and how do I know you will be around in 6 months or 6 years? I don't mind paying for a service but I would prefer to pay for a service with some weight behind it.

Juha, jsapi lookups are limited by daily quotas and questionable for a commercial app without paying an alleged $10,000 licensing fee for Google Maps Premium.

- Krishna

[Jeff Schnitzer]

On Fri, Oct 7, 2011 at 2:50 AM, Krishna Patel <krishna...@gmail.com> wrote:
> Jeff,
> I didn't even consider using their Java api. The idea of keeping the data in
> memory on App Engine is absurd. I'm surprised you thought that might work.

Uh... did you miss the part where I did this and it works great? And
James Broberg did the same?

Jeff

[Brandon Wirtz]

Fair enough…   I’m a data hoarder :-) and clients have been buying information for 18+ years.  I have never don the GEO-Ip as a service because the cost could never be competitive with Maxmind.  At somepoint max mind might offer a directly compatible/supported solution with AppEngine.  If I were a Developer Evangelist at Google this would be the kind of thing I would be working to make happen (hint, hint, nudge, nudge)

 

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Krishna Patel
Sent: Friday, October 07, 2011 2:50 AM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] X-AppEngine-City

 

Jeff,

--

[roberto.cr]

please star both Issues:

http://code.google.com/p/googleappengine/issues/detail?id=803&q=geolocation&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log

http://code.google.com/p/googleappengine/issues/detail?id=5227&q=geolocation&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log

I'd love to see this happen as well! Many apps need this!

On Oct 7, 4:37 pm, "Brandon Wirtz" <drak...@digerat.com> wrote:
> Fair enough.   I'm a data hoarder :-) and clients have been buying

> information for 18+ years.  I have never don the GEO-Ip as a service because
> the cost could never be competitive with Maxmind.  At somepoint max mind
> might offer a directly compatible/supported solution with AppEngine.  If I
> were a Developer Evangelist at Google this would be the kind of thing I
> would be working to make happen (hint, hint, nudge, nudge)
>
> From: google-a...@googlegroups.com
> [mailto:google-a...@googlegroups.com] On Behalf Of Krishna Patel
> Sent: Friday, October 07, 2011 2:50 AM
> To: google-a...@googlegroups.com
> Subject: Re: [google-appengine] X-AppEngine-City
>
> Jeff,
>
> I didn't even consider using their Java api. The idea of keeping the data in
> memory on App Engine is absurd. I'm surprised you thought that might work.
>
> I'm following an approach similar to Ikai's in his blog post:
>

> http://ikaisays.com/2010/08/11/using-the-app-engine-mapper-for-bulk-d...
> ort/

[Chris Collins]

Brandon,

How much would you charge to provide a REST api that takes in an IP address and spits out the 2-alpha country code (same as x-appengine-country) along with a best guess for the city name?

Also, would you be able to guarantee a reasonable time window (a few months?) before discontinuing your service - if you ever decided to?

- Chris

[Brandon Wirtz]

You have a sense of the volume?  Would you prefer a monthly Sub, or a per X IP’s looked up?

 

To be fast I’d likely have an always on instance. That is going to put the pricing at $10 a month minimum.

 

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Chris Collins
Sent: Friday, October 14, 2011 8:45 AM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] X-AppEngine-City

 

Brandon,

--

[Chris Collins]

It's a seasonal app(s). The volume would need to be about 5,000-10,000 requests per day for high season. On off-seasons (e.g. winter months), it would drop to around 100 requests per day.

What would be the difference in terms of monthly sub vs per-IP for that kind of volume?

[JH]

Have you seen:
http://freegeoip.net/static/index.html

or

http://ipinfodb.com/

On Oct 14, 12:54 pm, Chris Collins <xop...@gmail.com> wrote:
> It's a seasonal app(s). The volume would need to be about 5,000-10,000
> requests per day for high season. On off-seasons (e.g. winter months), it
> would drop to around 100 requests per day.
>
> What would be the difference in terms of monthly sub vs per-IP for that kind
> of volume?
>
>
>
>
>
>
>
> On Fri, Oct 14, 2011 at 6:04 PM, Brandon Wirtz <drak...@digerat.com> wrote:
> > You have a sense of the volume?  Would you prefer a monthly Sub, or a per X

> > IP’s looked up?****
>
> > ** **

>
> > To be fast I’d likely have an always on instance. That is going to put the

> > pricing at $10 a month minimum.****
>
> > ** **
>
> > ** **
>
> > *From:* google-a...@googlegroups.com [mailto:
> > google-a...@googlegroups.com] *On Behalf Of *Chris Collins
> > *Sent:* Friday, October 14, 2011 8:45 AM
>
> > *To:* google-a...@googlegroups.com
> > *Subject:* Re: [google-appengine] X-AppEngine-City****
>
> > ** **
>
> > Brandon,****
>
> > ** **

>
> > How much would you charge to provide a REST api that takes in an IP address
> > and spits out the 2-alpha country code (same as x-appengine-country) along

> > with a best guess for the city name?****
>
> > ** **

>
> > Also, would you be able to guarantee a reasonable time window (a few

> > months?) before discontinuing your service - if you ever decided to?****
>
> > ** **
>
> > - Chris****
>
> > On Fri, Oct 7, 2011 at 9:48 AM, Brandon Wirtz <drak...@digerat.com> wrote:
> > ****

>
> > If someone tells me how to make money on it I will expose my API for doing
> > Geo-Location by IP.  I never assumed anyone would pay anything... So I'm
> > not
> > sure what is in it for me.
>

> > Also what level of detail do you want?****

>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengi...@googlegroups.com.
> > For more options, visit this group at

> >http://groups.google.com/group/google-appengine?hl=en.****

[Chris Collins]

JH,

That's perfect! Thanks!

I guess I'll go with that. Sorry Brandon.

- Chris

[Brandon Wirtz]

Don’t be. I really didn’t want to have to support something ;-)

 

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Chris Collins

Sent: Friday, October 14, 2011 5:37 PM
To: google-a...@googlegroups.com

[GordonHo]

hi chris,

perhaps the maxmind webapi would be another solution.

i've not tried ipinfodb, but i am using maxmind on several projects (webapi and local db).

the webapi works quite well for me when caching results locally for a short time.

when using the webapi i've sometimes put in the local db as a backup solution in case the webapi times out etc.

cheers

[Chris Collins]

Thanks Gordon.

That looks even better. And the webapi is more affordable than I expected ($20 for 50,000 requests).

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/4s-jQZ3YBZYJ.

[Krishna Patel]

The Maxmind Webapi is working well for me now. But I'm curious whether App Engine will ever provide this out of the box.